Using the openmpi-psm.qs job script template found in /opt/shared/templates/gridengine/openmpi provides the best performance for a parallel MPI run for OpenFOAM applications. Once you have copied the template for your job script (e.g. qsub_sowfa.qs), then edit the following for your OpenFOAM application:

1. Modify NPROC to select the number of slots (cores); best results have been found specifying twice as many cores (a multiple of 24) and binding cores due to the shared FP unit (see steps 5 and 6).
2. Modify the email notification -m and -M according to the instructions if you want email to be sent at the end of the job, if the job is aborted, rescheduled, or suspended.
3. Setup your environment with your newly built OpenFOAM application by replacing the vpkg_require openmpi/1.4.4-gcc with your VALET package name (e.g. sowfa/2.0-OF2.1-gcc)
4. Setup the MPI executable and options (arguments) for your OpenFOAM application (e.g. ABLSolver -parallel > log.ABLSolver.3.${JOB_ID} 2>err.ABLSolver.3)
5. Remove # in front of #WANT_CPU_AFFINITY=YES
6. Remove # in front of #WANT_HALF_CORES_ONLY=YES

Now submit your job script using exclusive access to prevent other jobs from running on the same node(s)

qsub -l exclusive=1 qsub_sowfa.qs

 mca: base: component_find: unable to open /opt/shared/openfoam/2.1.
1-gcc/ThirdParty-2.1.1/platforms/linux64Gcc/openmpi-1.5.3/lib/openmpi/mca_plm_rs
hd: /opt/shared/openfoam/2.1.1-gcc/ThirdParty-2.1.1/platforms/linux64Gcc/openmpi
-1.5.3/lib/openmpi/mca_plm_rshd.so: cannot open shared object file: Permission d
enied (ignored)