software:matlab:mills

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
software:matlab:mills [2018-12-03 11:25] – [Matlab License Information] anitasoftware:matlab:mills [2019-08-29 16:05] (current) – [Matlab script] anita
Line 1: Line 1:
 +====== Matlab on Mills ======
  
 +
 +On Mills, MATLAB projects should be developed using a Desktop installation of MATLAB and then copied to the cluster
 +to be run in batch.  Here we
 +consider an extended MATLAB example involving two simple MATLAB functions, and two MATLAB scripts to execute
 +the first function in a loop, and to using the Parallel Toolbox.
 +
 +Details on how to run these two scripts in batch are given with the resulting output files.  There is also a
 +section with UNIX commands you can use to watch your jobs and gather [[#timings-and-core-count | timing and core count]] numbers.  
 +You will need to know how much memory and how many cores you should request for your jobs.
 +
 +Even though it easier to develop on a desktop, MATLAB can be run interactively on Farber.
 +Two interactive jobs are demonstrated.  One shows how to test the function by executing the function one time. A
 +second example shows an interactive session, which starts multiple MATLAB pool of workers to execute the function as a parallel toolbox loop, **''parfor''**.
 +The Parallel toolbox gives a faster time to completion, but with more memory and CPU resources consumed.
 +
 +Many MATLAB research projects fall in the the "high throughput computing" category.  One run can be done on the desktop, but it is desired complete 100s or 1000s of independent runs.  This greatly increases disk, memory and CPU requirements. 
 +Thus we have a
 +final example that gives the recommended workflow to scale your job to multiple nodes. The second MATLAB function has added logic to enabled it to be deployed as a compiled MATLAB code.  Then it is deployed as a single threaded grid engine array job.
 +
 +<note important>
 +The MATLAB distributed computing server (MDCS) is not installed on Mills.  This means jobs run with the Parallel Computing toolbox can only run on one node.  This limits both the size of the job and the number of workers you can use.  That is why
 +an array job of compiled MATLAB is recommended for large jobs.
 +</note>
 +
 +==== Matlab License Information ====
 +
 +Matlab licenses are pushed into consumable (global, per-job) integer complexes in Grid Engine and can be checked using 
 +
 +<code>
 +        qhost -h global -F
 +</code>
 +
 +to list number of unused license seats for each product.
 +
 +Below is an example representing a snapshot of unused licensed seats for Matlab products on the cluster.
 +<code>
 + [traine@mills ~]$ qhost -h global -F
 + HOSTNAME                ARCH         NCPU  LOAD  MEMTOT  MEMUSE  SWAPTO  SWAPUS
 + -------------------------------------------------------------------------------
 + global                  -               -     -       -       -       -       -
 +     gc:MLM.Compiler=50.000000
 +            gc:MLM.Aerospace_Blockset=1.000000
 +            gc:MLM.RTW_Embedded_Coder=2.000000
 +            gc:MLM.Robust_Toolbox=150.000000
 +            gc:MLM.Aerospace_Toolbox=1.000000
 +            gc:MLM.Identification_Toolbox=50.000000
 +            gc:MLM.XPC_Target=2.000000
 +            gc:MLM.Econometrics_Toolbox=1.000000
 +            gc:MLM.Real-Time_Workshop=2.000000
 +            gc:MLM.Fuzzy_Toolbox=50.000000
 +            gc:MLM.Video_and_Image_Blockset=1.000000
 +            gc:MLM.Neural_Network_Toolbox=50.000000
 +            gc:MLM.Fin_Instruments_Toolbox=1.000000
 +            gc:MLM.Optimization_Toolbox=44.000000
 +            gc:MLM.MATLAB_Coder=2.000000
 +            gc:MLM.MATLAB=204.000000
 +            gc:MLM.Database_Toolbox=1.000000
 +            gc:MLM.SIMULINK=100.000000
 +            gc:MLM.PDE_Toolbox=48.000000
 +            gc:MLM.GADS_Toolbox=1.000000
 +            gc:MLM.Symbolic_Toolbox=46.000000
 +            gc:MLM.Signal_Toolbox=146.000000
 +            gc:MLM.Financial_Toolbox=1.000000
 +            gc:MLM.Data_Acq_Toolbox=2.000000
 +            gc:MLM.Image_Acquisition_Toolbox=1.000000
 +            gc:MLM.Curve_Fitting_Toolbox=9.000000
 +            gc:MLM.Image_Toolbox=143.000000
 +            gc:MLM.Distrib_Computing_Toolbox=48.000000
 +            gc:MLM.OPC_Toolbox=1.000000
 +            gc:MLM.MPC_Toolbox=50.000000
 +            gc:MLM.Virtual_Reality_Toolbox=1.000000
 +            gc:MLM.Statistics_Toolbox=43.000000
 +            gc:MLM.Signal_Blocks=50.000000
 +            gc:MLM.Instr_Control_Toolbox=2.000000
 +            gc:MLM.MAP_Toolbox=12.000000
 +            gc:MLM.Communication_Toolbox=50.000000
 +            gc:MLM.Control_Toolbox=150.000000
 +            gc:MLM.Wavelet_Toolbox=1.000000
 +            gc:MLM.Bioinformatics_Toolbox=1.000000
 +     gc:MLM.Simulink_Control_Design=50.000000
 +     gc:MLM.Real-Time_Win_Target=1.000000
 +</code>
 +
 +Matlab jobs can be submitted to require a certain number of license seats to be available before a job will run. If there are inter-license dependencies for toolboxes, then you should specify all the licenses including Matlab and/or Simulink.
 +
 +For example, if a Matlab job requires the Financial toolbox, then you will also need to specify all the inter-related toolbox licenses required by the Financial toolbox such as the Statistics and Optimization toolboxes as well Matlab itself. See [[http://www.mathworks.com/products/availability|Mathworks System Requirements & Platform Availability by Product
 +]] for complete details.
 +
 +<code>
 + qsub -l MLM.MATLAB=1,MLM.Financial_Toolbox=1,MLM.Statistics_Toolbox=1,MLM.Optimization_Toolbox=1 ...
 +</code>
 +
 +Naturally, this isn't a to-the-moment mapping because the license server is not being queried constantly.  However, it's consumable, so it is keeping track of how many seats are unused every 6 minutes.
 +
 +This will be most helpful when submitting many Matlab jobs that require a toolbox with a low-seat count. They will wait for a toolbox seat to become available rather than trying to run and having many getting the "**License checkout failed**" message from MATLAB.
 +
 +===== Matlab function =====
 +
 +Sample Matlab function:
 +<file matlab maxEig.m>
 +%  maxEig   Maximum Eigenvalue of a random matrix
 +%     Input parameters
 +%       sd - seed for uniform random generator
 +%       dim - size of the square matrix (should be odd)
 +%     Output value
 +%       maxe - maximum real eigenvalue
 +function maxe = maxEig(sd,dim)
 +  rng(sd);
 +  ev = eig( randn(dim) );
 +  maxe = max( ev(imag(ev)==0) )
 +end
 +</file>
 +
 +
 +
 +The examples will be using a Matlab function to illustrate using Matlab in batch and interactively.  The function will be run on multiple cores using multiple computational threads, and 12 workers from a Matlab pool.  Finally it will be compiled and deployed.
 +
 +
 +==== Matlab script ====
 +First, write a Matlab script file. It should have a comment on the first line describing the purpose of the script and have the ''quit'' command on the last line. This script will call the [[#matlab-function|maxEig function]] 200 times and report the average:
 +
 +<file matlab script.m>
 +% script to run maxEig function 200 times
 +
 +count = 200;
 +dim = 5001;
 +sumMaxe = 0;
 +tic;
 +for i=1:count;
 +  sumMaxe = sumMaxe + maxEig(i,dim);
 +end;
 +toc
 +avgMaxEig = sumMaxe/count
 +
 +quit
 +</file>
 +
 +This is a detailed script example, which calls the maxEig function.  This example does no file I/O, all the I/O is to standard out.  In Matlab, assignments, not terminated by a semicolon, are display on the screen (standard out in batch).
 +
 +<note tip>
 +Several MATLAB commands could be added to the beginning of this script to set the maximum number of computational threads to the number of slots assigned to your job.  If the scheduler using CGROUPS to limit your job core count, then these commands are not necessary.
 +<code>
 +[compThreads,count]=sscanf(getenv('NSLOTS'),'%d');
 +if count == 1 
 +  warning('off','MATLAB:maxNumCompThreads:Deprecated');
 +  autoCompThreads = maxNumCompThreads(compThreads);
 +  disp(sprintf('NumCompThreads=%d, was %d',compThreads,autoCompThreads))
 +end
 +</code>
 +See [[maxNumCompThreadsGridEngine|Setting maximum number of computational threads]]</note>
 +
 +<note tip>
 +This script ends in a **__quit__** command (equivalent to MATLAB **__exit__**).  This is meant to be a complete script, which
 +terminates MATLAB when done.  If you run this from the bash command line with the ''-r script'' option, it will come back with a bash prompt when completed.  If this is run from a batch job, then you can do other commands in your batch script after the MATLAB script completes.
 +
 +Without the **__quit__** you will come back to the MATLAB prompt on completion for a interactive job.  If this is the last line of a batch queue script, then the only difference will be the MATLAB prompt ''>>'' at the very end of the output file.  MATLAB treats the end of batch script file the same as exiting the window, which is the preferred way to exit the MATLAB GUI.
 +</note>
 +
 +==== Grid Engine script ====
 +
 +Second, write a shell script file to set the Matlab environment and start Matlab running your script file. The following script file will set the Matlab environment and run the command in the [[#matlab-script|script.m]] file:
 +
 +<file bash batch-simple.qs>
 +#$ -N script-simple.m
 +#$ -m ea
 +#$ -M traine@gmail.com
 +#$ -pe threads 12
 +
 +vpkg_require matlab/r2013a
 +matlab -nodisplay -nojvm -r script
 +</file>
 +
 +<file bash batch-catcherr.qs>
 +#$ -N script-catcherr.m
 +#$ -m ea
 +#$ -M traine@gmail.com
 +#$ -pe threads 12
 +
 +vpkg_require matlab/r2013a
 +matlab -nodisplay -nojvm -r 'try; script; catch err; disp(err.getReport); end'
 +</file>
 +
 +Ths ''-nodisplay'' indicates no X11 graphics, which implies no introductory splash panel ''-nosplash''. The ''-nojvm'' indicates no Java. Java is needed for print graphics, but should be excluded for must computational jobs.
 +The ''-r'' is followed by a list of Matlab commands, enclosed in quotes.
 +
 +<note tip>**Errors in the Matlab script**
 +
 +This list of commands is the ''try/catch'' block enclose one command.  The command ''script'' will execute the lines in the ''script.m'' file.  The purpose of the ''try/catch'' is to handle the possibility that
 +there is an error in the script.  For some errors Matlab will display the error message and wait a respond.  Clearly not appropriate for a batch job.
 +</note>
 +
 +<note tip>**Graphics in the Matlab script**
 +
 +  * Do not include the ''-nojvm'' on the **matlab** command.
 +  * Do set paper dimensions and print each figure to a file.
 +
 +The text output will be included in the standard Grid Engine output file, but not any graphics.  All figures must be exported using the **print** command. Normally the **print** command will print on an 8 1/2 by 11 inch page with margins that are for a printed page of paper.  The size and margins will not work if you plan to include the figure in a paper or a web page.
 +
 +We suggest setting the current figure's ''PaperUnits'', ''PaperSize'' and ''PaperPosition''.  Matlab provides a handle to the current figure (**gcf**).  For example, the commands
 +
 +<code matlab>
 +  set(gcf,'PaperUnits','inches','PaperSize',[4,3],'PaperPosition',[0 0 4 3]);
 +  print('-dpng','-r100','maxe.png');
 +</code>
 +
 +will set the current figure to be 4 x 3 inches with no margins, and then print the figure as a 400x300 resolution ''png'' file.
 +</note>
 +
 +
 +
 +
 +==== Submit job ====
 +Third, from the directory with both ''script.m'' and either ''batch-simple.qs'' or ''batch-catcherr.qs'', submit the batch job with the command:
 +
 +<code>
 +qsub batch-simple.qs
 +</code>
 +
 +or
 +
 +<code>
 +qsub batch-catcherr.qs
 +</code>
 +
 +<note warning>See [[#matlab-license-information|Matlab licenses]] for details on specifying resources if there are limited number of license seats available for particular toolboxes.
 +
 +In this example you will need a license for the base Matlab, and the parallel toolbox needs one license.  We are using the default local scheduler which give you 12 workers on the same node with one license.
 +</note>
 +==== Wait for completion ====
 +Finally, wait for the mail notification, which will be sent to ''traine@gmail.com''.  When the job is done the output from the Matlab command will be in a file with the pattern - ''script.m.oJOBID'', where JOBID is the number assigned to your job.
 +
 +After waiting for about 5 hours, a message was receive with subject line "Grid Engine Job Scheduler":
 +
 +<code>
 +Job 236214 (script.m) Complete
 +User             = traine
 +Queue            = it_css.q+@n017
 +Host             = n017.mills.hpc.udel.edu
 +Start Time       = 03/27/2013 16:17:30
 +End Time         = 03/27/2013 21:11:46
 +User Time        = 1:10:47:28
 +System Time      = 00:30:04
 +Wallclock Time   = 04:54:16
 +CPU              = 1:11:17:33
 +Max vmem         = 1.976G
 +Exit Status      = 0
 +</code>
 +
 +==== Gather results ====
 +The results for Job 236214 are in the file
 +<code text>
 +Adding dependency `x11/RHEL6.1` to your environment
 +Adding package `matlab/r2013a` to your environment
 +
 +                            < M A T L A B (R) >
 +                  Copyright 1984-2013 The MathWorks, Inc.
 +                    R2013a (8.1.0.604) 64-bit (glnxa64)
 +                             February 15, 2013
 +
 + 
 +To get started, type one of these: helpwin, helpdesk, or demo.
 +For product information, visit www.mathworks.com.
 + 
 +
 +maxe =
 +
 +   70.0220
 +   
 +
 +           '''      //Skipping 198 similar displays of variablemaxe//
 +
 +maxe =
 +
 +   67.4221
 +
 +Elapsed time is 17651.125420 seconds.
 +
 +avgMaxEig =
 +
 +   69.5131
 +</code>
 +
 +==== Timing and core count ====
 +
 +While the batch job was running on node ''n017'', the top command was run to sample the resources being used by Matlab
 +every second  two times ''-b -d 1 -n 2''.  This ''-H'' option was used to display each individual thread, rather than a summery of all threads in a process.
 +
 +<code text>
 +$ ssh n017 top -H -b -d 1 -n 2 | egrep '(COMMAND|MATLAB)' | grep -v 'S  0'
 +  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 + 2158 traine    20   0 2011m 557m  54m R 101.6  0.9  30:19.69 MATLAB            
 + 2166 traine    20   0 2011m 557m  54m R 72.9  0.9  16:53.48 MATLAB             
 + 2167 traine    20   0 2011m 557m  54m R 72.9  0.9  16:53.28 MATLAB             
 + 2171 traine    20   0 2011m 557m  54m R 72.9  0.9  16:53.08 MATLAB             
 + 2173 traine    20   0 2011m 557m  54m R 72.9  0.9  16:53.12 MATLAB             
 + 2174 traine    20   0 2011m 557m  54m R 72.9  0.9  16:52.85 MATLAB             
 + 2175 traine    20   0 2011m 557m  54m R 72.9  0.9  16:52.98 MATLAB             
 + 2176 traine    20   0 2011m 557m  54m R 72.9  0.9  16:52.86 MATLAB             
 + 2168 traine    20   0 2011m 557m  54m R 70.9  0.9  16:52.88 MATLAB             
 + 2169 traine    20   0 2011m 557m  54m R 70.9  0.9  16:52.83 MATLAB             
 + 2170 traine    20   0 2011m 557m  54m R 70.9  0.9  16:52.93 MATLAB             
 + 2172 traine    20   0 2011m 557m  54m R 70.9  0.9  16:52.70 MATLAB             
 +  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 + 2158 traine    20   0 2011m 557m  54m R 99.2  0.9  30:20.71 MATLAB             
 + 2172 traine    20   0 2011m 557m  54m R 83.7  0.9  16:53.56 MATLAB             
 + 2166 traine    20   0 2011m 557m  54m R 82.7  0.9  16:54.33 MATLAB             
 + 2167 traine    20   0 2011m 557m  54m R 82.7  0.9  16:54.13 MATLAB             
 + 2168 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.73 MATLAB             
 + 2169 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.68 MATLAB             
 + 2170 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.78 MATLAB             
 + 2171 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.93 MATLAB             
 + 2173 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.97 MATLAB             
 + 2174 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.70 MATLAB             
 + 2176 traine    20   0 2011m 557m  54m R 82.7  0.9  16:53.71 MATLAB             
 + 2175 traine    20   0 2011m 557m  54m R 81.7  0.9  16:53.82 MATLAB      
 +</code>
 +
 +<code>
 +$ ssh n017 mpstat -P ALL 1 2
 +Linux 2.6.32-279.19.1.el6.x86_64 (n017) 03/27/2013 _x86_64_ (24 CPU)
 +
 +04:46:56 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
 +04:46:57 PM  all   47.61    0.00    0.96    0.00    0.00    0.00    0.00    0.00   51.44
 +04:46:57 PM    0    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM    1    8.08    0.00    0.00    0.00    0.00    0.00    0.00    0.00   91.92
 +04:46:57 PM    2   95.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM    4    0.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00   99.00
 +04:46:57 PM    5   95.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM    6   93.07    0.00    2.97    0.00    0.00    0.00    0.00    0.00    3.96
 +04:46:57 PM    7   16.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00   84.00
 +04:46:57 PM    8   75.25    0.00    0.00    0.00    0.00    0.00    0.00    0.00   24.75
 +04:46:57 PM    9   94.00    0.00    3.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM   10   95.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM   11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   13   95.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM   14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   15   95.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM   16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   17   94.95    0.00    2.02    0.00    0.00    0.00    0.00    0.00    3.03
 +04:46:57 PM   18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   19   96.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    3.00
 +04:46:57 PM   20    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   21   94.12    0.00    1.96    0.00    0.00    0.00    0.00    0.00    3.92
 +04:46:57 PM   22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:57 PM   23   95.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    3.00
 +
 +04:46:57 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest   %idle
 +04:46:58 PM  all   49.35    0.00    0.67    0.00    0.00    0.00    0.00    0.00   49.98
 +04:46:58 PM    0  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM    1    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM    2   98.99    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM    3    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM    4    0.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00   98.00
 +04:46:58 PM    5   98.99    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM    6  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM    7    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM    8    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM    9   98.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   10   98.02    0.00    1.98    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   11    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   12    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   13   98.99    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   14    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   15   99.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   16    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   17   99.00    0.00    1.00    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   18    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   19   98.99    0.00    1.01    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   20    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   21   98.00    0.00    2.00    0.00    0.00    0.00    0.00    0.00    0.00
 +04:46:58 PM   22    0.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00  100.00
 +04:46:58 PM   23   98.02    0.00    0.99    0.00    0.00    0.00    0.00    0.00    0.99
 +</code>
 +
 +==Conclusions==
 +The main computation task for this batch job is finding all 5001 eigenvalues.  This is done by the Matlab built-in function ''ev''.  The Matlab built-in matrix function call using MKL, which is a highly optimized multi-threaded library.  It uses what is called dynamic scheduling to using as many cores as needed for the application at hand.  In this snapshot it appears as if there is one main thread and 11 other threads that are called on for vector operations (BLAS).  The 11 threads have similar times that means they being using as data parallel operations (they are all doing the same operations on different parts of the vector.
 +
 +We have the correct Grid Engine options ''-pe threads 12'' for this case.
 +
 +[[software:matlab:mills:matlab-two-samenode|Two Matlab interactive jobs on same node]]
 +
 +===== Batch parallel example =====
 +
 +The Matlab parallel toolbox uses JVM to manage the workers and communicate while you are running.  You
 +need to setup the Matlab pools in your ''script''.
 +
 +==== Matlab script ====
 +Here are the slightly modified MATLAB script and qsub script files.  These can be used to do the same task with 12 Matlab workers.
 +
 +Add two ''matlabpool'' commands and change ''for'' ⇒ ''parfor''.
 +<file matlab pscript.m>
 +% script to run maxEig function 200 times
 +mypool=parpool(20);
 +
 +count = 200;
 +dim = 5001;
 +sumMaxe = 0;
 +tic;
 +parfor i=1:count;
 +  sumMaxe = sumMaxe + maxEig(i,dim);
 +end;
 +toc
 +avgMaxEig = sumMaxe/count
 +
 +delete(mypool);
 +quit
 +</file>
 +
 +==== Grid engine script ====
 +Take out ''-nojvm'', which is needed for the matlabpool.
 +<file bash pbatch.qs>
 +#$ -N pscript
 +#$ -m ea
 +#$ -M traine@gmail.com
 +#$ -l m_mem_free=3.1G
 +#$ -l MLM.Distrib_Computing_Toolbox=1
 +#$ -pe threads 20
 +
 +vpkg_require matlab/r2015a
 +
 +ls
 +matlab -nodisplay -r 'pscript'
 +</file>
 +
 +
 +===== Interactive example =====
 +
 +This example is based on being in your workgroup environment, cd'ing to your MATLAB project directory and starting an interactive session on a compute node.
 +==== Scheduling exclusive interactive job ====
 +
 +<code>
 +$ qlogin -l exclusive=1
 +Your job 2493 ("QLOGIN") has been submitted
 +waiting for interactive job to be scheduled ...
 +Your interactive job 2493 has been successfully scheduled.
 +Establishing /opt/shared/univa/local/qlogin_ssh session to host n036 ...
 +</code>
 +
 +==== Starting a command mode matlab session ====
 +
 +<code>
 +$ vpkg_require matlab/r2014b
 +Adding package `matlab/r2014b` to your environment
 +</code>
 +<code>
 +$ matlab -nodesktop -nosplash
 +MATLAB is selecting SOFTWARE OPENGL rendering.
 +
 +                                           < M A T L A B (R) >
 +                                 Copyright 1984-2014 The MathWorks, Inc.
 +                                 R2014b (8.4.0.150421) 64-bit (glnxa64)
 +                                           September 15, 2014
 +
 + 
 +To get started, type one of these: helpwin, helpdesk, or demo.
 +For product information, visit www.mathworks.com.
 +</code>
 +
 +
 +==== Using help as the first command ====
 +
 +<code>
 +>> help maxEig
 +   maxEig   Maximum Eigenvalue of a random matrix
 +      Input parameters
 +        sd - seed for uniform random generator
 +        dim - size of the square matrix (should be odd)
 +      Output value
 +        maxe - maximum real eigvalue
 +        
 +</code>
 +
 +==== Calling function once ====
 +
 +Use the tic and toc commands to report the elapsed time to generate the random matrix, find all eigenvalues and report the maximum real eigvalue.
 +
 +<code>
 +>> tic; maxEig(1,5001); toc
 +
 +maxe =
 +
 +   70.0220
 +
 +Elapsed time is 86.564581 seconds.
 +</code>
 +
 +==== Finishing up ====
 +
 +<code>
 +>> exit
 +$ exit
 +Connection to n036 closed.
 +/opt/shared/univa/local/qlogin_ssh exited with exit code 0
 +</code>
 +
 +===== Interactive parallel toolkit example =====
 +
 +When you plan to use the parallel toolbox, you should logon the compute node with the command
 +
 +  qlogin -pe threads 12
 +  
 +This will effectively reserve 12 slots for your Matlab workers.  The is the default, and maximum allowed value for the current version of Matlab (r2013a).
 +
 +Same example function called from 12 matlab workers (the maximum allowed for the local profile.)
 +Here we start 12 workers with the matlabpool, and then use parfor to send a different seed to each worker.  It is important that the workers do not do the same computations.  The output is from the workers, in the order they complete.  The order is not deterministic.
 +
 +It took about 150 seconds to start all 12 workers, and this is no reflected in the total elapsed time.
 +
 +<code>
 +>> matlabpool(12);
 +Starting matlabpool using the 'local' profile ... connected to 12 workers.
 +>> tic; parfor sd = 1:12; maxEig(sd,5001); end; toc
 +
 +maxe =
 +
 +   70.8182
 +
 +
 +maxe =
 +
 +   70.4677
 +
 +
 +maxe =
 +
 +   70.2538
 +
 +
 +maxe =
 +
 +   62.0506
 +
 +
 +maxe =
 +
 +   70.9143
 +
 +
 +maxe =
 +
 +   66.4947
 +
 +
 +maxe =
 +
 +   70.5141
 +
 +
 +maxe =
 +
 +   70.8709
 +
 +
 +maxe =
 +
 +   69.1104
 +
 +
 +maxe =
 +
 +   63.9266
 +
 +
 +maxe =
 +
 +   70.6306
 +
 +
 +maxe =
 +
 +   70.4674
 +
 +Elapsed time is 207.531791 seconds.
 +</code>
 +
 +
 +
 +===== MCR (Matlab Compiler Runtime) array job =====
 +
 +Most Matlab functions can be compiled using the Matlab Compiler (MCC) and then deployed to run on the compute nodes in the MATLAB Compiler Runtime (MCR).  The MCR is a prerequisite for deployment, and is installed on all the compute nodes. You must use VALET to set up the libraries you will need to run your function from the command line.  You do not need to use the shell (''.sh'' file) that the compiler creates.
 +
 +There are two ways to run compiled MATLAB jobs in a shared environment, such as Mills.
 +  - Compile to produce and executable that uses a single computational thread - MATLAB option '-singleCompThread'
 +  - Submit the job to use the nodes exclusively -  Grid engine option ''-l exclusive=1''
 +
 +You can run more jobs on each node when they compiled to use just one core (Single Comp Thread).  This will give
 +you higher throughput for an array job, but not higher performance.  
 +
 +==== Example function ====
 +
 +As an example, consider the Matlab function with one parameter:
 +<code>
 +function maxe = maxEig(lam)
 +if (isdeployed)
 +    lam=str2num(lam);
 +end
 +
 +dim=10000;
 +M=rand(dim);
 +D=diag(rand(dim,1));
 +
 +maxe = max(eig( M + lam*D ))
 +end
 +</code>
 +
 +<note>
 +The MATLAB function ''maxEig'' takes one numeric parameter as a argument (''lam''), constructs a random matrix in the form ''M + lam*D'' and then assigns ''maxe'' to the largest eigenvalue of the matrix.  
 +We should get different ''maxe'' values dependent on ''lam''.
 +</note>
 +
 +==== Example compiler commands ====
 +
 +We can convert this function into a single computational executable by using the Matlab compiler ''mcc''.  Type the commands
 +<code>
 +prog=maxEig
 +opt='-nojvm,-nodisplay,-singleCompThread'
 +version='r2012b'
 +
 +vpkg_require matlab/$version
 +mcc -R "$opt" -mv $prog.m
 +</code>
 +
 +<note tip>**Keep these commands in a file**: Even though this is just two commands, we recommend you keep these commands, including the shell assignment statements, as a record of the MATLAB version and options you used to create the executable ''maxEig''.  You will need to know these if you want to use the executable in a shell script.  You can source this file when you want to rebuild ''maxEig''</note>
 +
 +<note tip>**You can get mcc usage instructions with ''mcc -help''**: The string following the ''-R'' flag are the Matlab
 +options you want to use at run time.  The ''-m'' option tell mcc to build a standalone application to be deployed
 +using MCR.  The ''-v'' option is for verbose mode. </note>
 +
 +
 +==== Example queue script file ====
 +
 +The ''mcc'' command will give you a ''.sh'', which you can use to setup you environment and run the command.  This does not use VALET and does not have the grid engine commands in it.  We such you use the following ''.qs'' file as a example to modify for own job.
 +<code>
 +#$ -N maxEig
 +#$ -t 1-200
 +# 
 +# Parameter sweep array job to run the maxEig compiled MATLAB function with
 +#    lambda = 1,2. ... 200
 +#
 +date "+Start %s"
 +echo "Host $HOSTNAME"
 +
 +let lambda="$SGE_TASK_ID-1"
 +
 +source /opt/shared/valet/docs/valet.sh
 +vpkg_require mcr/r2012b-nojvm
 +export MCR_CACHE_ROOT=$TMPDIR
 +
 +./maxEig $lambda
 +
 +date "+Finish %s"
 +</code>
 +
 +The two ''date'' commands with record the start and finish time in seconds for each task.  These can be used to compute the elapsed time, and the echoed host name can be used to calculate the overlapping use of the computer nodes.
 +
 +==== Compiled MATLAB in owner queues ====
 +
 +To test the example compiled Matlab job on the three ''it_css'' owner queues, we first will modify the first few lines of queue script file:
 +<code>
 +#$ -t 1-6
 +#$ -l exclusive=1
 +</code>
 +This will do 6 tasks with exclusive access to the node. That is, each node will do 2 tasks consecutively.
 +
 +Next, we compile two copies of the program
 +  - multiply computational threads (default).
 +  - Single computional thread (-SingleCompThread)
 +
 +**Times for two runs**
 +
 +Multiple computational threads used a total of 16929.669 CPU seconds over 2226 seconds of elapsed time
 +on 3 nodes
 +^ Array job started Tue 26 Feb 2013 07:59:48 PM EST ^^^^^^
 +^              Node  ^^               Real Clock Time  ^^^       Ratio  ^
 +^    Name  ^   Count  ^      Min  ^      Max  ^  Average  ^   User/Real  ^
 +|    n015|       2|   1073.38|  1076.94|  1075.16 |     2.86154|
 +|    n016|       2|    737.28|   737.98|   737.63 |     3.68737|
 +|    n017|       2|    712.33|   715.36|   713.84 |     3.73799|
 +Single computational thread used a total of 8568.438 CPU seconds over 3351 seconds of elapsed time
 +on 3 nodes
 +^ Array job started Mon 25 Feb 2013 08:46:29 PM EST ^^^^^^
 +^              Node  ^^               Real Clock Time  ^^^       Ratio  ^
 +^    Name  ^   Count  ^      Min  ^      Max  ^  Average  ^   User/Real  ^
 +|    n015|       2|   1654.88|  1662.45|  1658.66 |     0.99852|
 +|    n016|       2|   1342.72|  1349.24|  1345.98 |     0.99821|
 +|    n017|       2|   1286.60|  1287.27|  1286.93 |     0.99807|
 +
 +**Predicted times for 200 tasks**
 +
 +In the multiple computational threads case, all the cores are being used by one Matlab job.  So for 200 jobs we can continue to run 3 at a time.  The median time for one job is 737.63, so the expected elapsed time for 200 tasks.
 +   Time 737.63 * 200 /3 = 49175.33 seconds > 13 hours
 +
 +In the multiple computational threads case, only one of the cores is being used.  So for 200 jobs we can remove the exclusive access and run 72 jobs at a time on the the 3 nodes. The median time for one job is 1345.98, so the expected elapsed time for 200 tasks.
 +   Time = 1345.98 * 200 /72 = 3738.83 seconds ≈ 1 hr
 +   
 +=== SGE array job started Fri 22 Feb 2013 11:57:07 AM EST ===
 +
 +Command to submit 200 jobs.
 +<code>qsub maxEig.qs</code>
 +
 +All 200 jobs used a total of 680420.993 CPU seconds over 10916 seconds of elapsed time
 +on 3 nodes:
 +^              Node  ^^               Real Clock Time  ^^^       Ratio  ^
 +^    Name  ^   Count  ^      Min  ^      Max  ^  Average  ^   User/Real  ^
 +|    n015|      62|   2500.36|  4268.85|  3478.68 |     0.97593|
 +|    n016|      70|   3042.86|  4010.68|  3504.29 |     0.98359|
 +|    n017|      68|   2957.71|  3993.39|  3441.66 |     0.97704|
 +
 +{{:clusters:mills:nostandby.png?640|}}
 +
 +==== Compiled MATLAB standby queue ====
 +=== SGE array job started Fri 22 Feb 2013 04:05:28 PM EST === 
 +Command to submit 200 jobs.
 +<code>qsub -l standby=1 maxEig.qs</code>
 +
 +All 200 jobs used a total of 613305.888 CPU seconds over 5388 seconds of elapsed time
 +on 9 nodes:
 +^              Node  ^^               Real Clock Time  ^^^       Ratio  ^
 +^    Name  ^   Count  ^      Min  ^      Max  ^  Average  ^   User/Real  ^
 +|    n073|      24|   2420.63|  3696.83|  3147.61 |     0.85789|
 +|    n075|      24|   3231.81|  3762.60|  3545.73 |     0.97684|
 +|    n092|      24|   3216.97|  3816.26|  3547.40 |     0.97047|
 +|    n133|      24|   3706.22|  5385.36|  5015.40 |     0.58127|
 +|    n161|      24|   3188.36|  3615.85|  3428.45 |     0.97021|
 +|    n162|      24|   2941.85|  3475.67|  3210.88 |     0.98208|
 +|    n167|      24|   3412.76|  4691.94|  4074.67 |     0.59991|
 +|    n179|      24|   3025.63|  3964.92|  3540.22 |     0.97072|
 +|    n187|       8|   1844.05|  2236.47|  2044.01 |     0.98612|
 +
 +{{:clusters:mills:standby.png?640|}}