technical:recipes:keras-in-virtualenv

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
technical:recipes:keras-in-virtualenv [2021-02-24 23:44] – freytechnical:recipes:keras-in-virtualenv [2021-08-06 13:00] (current) – frey
Line 1: Line 1:
 +====== Keras Python Virtual Environment ======
 +
 +This page documents the creation of a Python virtual environment (virtualenv) containing the Keras deep-learning suite on the Caviness HPC system.  It assumes that the user is adding the software to the workgroup storage.
 +
 +===== Prepare Workgroup Directory =====
 +
 +Prepare to add software in the standard sub-directories of the workgroup storage:
 +
 +<code bash>
 +[user@login01 ~]$ workgroup -g my_workgroup
 +[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/keras
 +[(my_workgroup:user)@login01 ~]$ mkdir --mode=2775 --parent ${WORKDIR}/sw/valet
 +</code>
 +
 +These commands create any missing directories.  All directories created will have group-write and -inherit permissions.
 +
 +===== Create Keras Virtualenv =====
 +
 +All versions of the Keras virtualenv will be stored in the common base directory, ''$WORKDIR/sw/keras''; each virtualenv must have a unique name that will become the VALET version of Keras.  In this tutorial, the latest version of Keras as of today (2020-02-04) will be installed and the SKLearn package will be added, as well.  An appropriate version would be ''<date>:sklearn'', or ''20200204:sklearn''.  That version can be translated to a VALET-friendly directory name:
 +
 +<code bash>
 +[(my_workgroup:user)@login01 ~]$ vpkg_id2path --version-id=20200204:sklearn
 +20200204-sklearn
 +[(my_workgroup:user)@login01 ~]$ mkdir --mode=3750 ${WORKDIR}/sw/keras/20200204-sklearn
 +</code>
 +
 +The Intel Python distribution will form the basis for the Keras virtualenv, so add it to the environment:
 +
 +<code bash>
 +[(my_workgroup:user)@login01 ~]$ vpkg_require intel-python/2019u5:python3
 +Adding package `intel-python/2019u5:python3` to your environment
 +(root) [(my_workgroup:user)@login01 ~]$
 +</code>
 +
 +Notice the prompt changed:  the text ''(root)'' now prefixes it, indicating the directory that contains the active Python virtualenv.
 +
 +The Python environment will include Keras, numpy, and pip:
 +
 +<code bash>
 +(root) [(my_workgroup:user)@login01 ~]$ conda create --prefix=${WORKDIR}/sw/keras/20200204-sklearn keras numpy pip
 +</code>
 +
 +The conda environment builder will take some time to locate all the necessary packages and their dependencies.  It will then ask for confirmation:
 +
 +<code bash>
 +Proceed ([y]/n)? y
 +</code>
 +
 +Answer "y" to create the virtualenv.  If successful, text will be displayed that mentions source'ing a file to activate the virtualenv:  refrain from doing that, and instead use VALET to manage the Keras virtualenv instances.  Rollback the ''intel-python'' environment changes before proceeding:
 +
 +<code bash>
 +(root) [(my_workgroup:user)@login01 ~]$ vpkg_rollback
 +[(my_workgroup:user)@login01 ~]$ 
 +</code>
 +
 +Notice the ''(root)'' has disappeared from the prompt, indicating that the baseline virtualenv has been deactivated.
 +===== VALET Package Definition =====
 +
 +Assuming the workgroup does //not// already have a Keras VALET package definition, the following text:
 +
 +<file keras.vpkg_yaml>
 +keras:
 +    prefix: /work/my_workgroup/sw/keras
 +    description: KERAS Python environments
 +    flags:
 +        - no-standard-paths
 +    actions:
 +        - action: source
 +          script:
 +              sh: intel-python.sh
 +          order: failure-first
 +          success: 0
 +    versions:
 +        "20200204:sklearn":
 +            description: 2020-02-04 conda build with SKLearn added
 +            dependencies:
 +                - intel-python/2019u5:python3
 +</file>
 +
 +would be added to ''${WORKDIR}/sw/valet/keras.vpkg_yaml''.  If that file already exists, add your new version at the same level as others:
 +
 +<file keras.vpkg_yaml>
 +keras:
 +    prefix: /work/my_workgroup/sw/keras
 +    description: KERAS Python environments
 +    flags:
 +        - no-standard-paths
 +    actions:
 +        - action: source
 +          script:
 +              sh: intel-python.sh
 +          order: failure-first
 +          success: 0
 +    versions:
 +        "20200204:sklearn":
 +            description: 2020-02-04 conda build with SKLearn added
 +            dependencies:
 +                - intel-python/2019u5:python3
 +        "20191218":
 +            description: 2019-12-18 conda build
 +            dependencies:
 +                - intel-python/2019u2:python3
 +        "20191112":
 +            description: 2019-11-12 conda build
 +            dependencies:
 +                - intel-python/2019u5:python3
 +</file>
 +
 +<note tip>On Caviness after a user has used the ''workgroup'' command, VALET searches for package definitions in ''${WORKDIR}/sw/valet'' by default.</note>
 +
 +===== Install SKLearn =====
 +
 +The SKLearn package is not present in the conda online repositories, but it can be installed using ''pip.''  First, activate the new Keras virtualenv:
 +
 +<code bash>
 +[(my_workgroup:user)@login01 ~]$ vpkg_require keras/20200204:sklearn
 +Adding dependency `intel-python/2019u5:python3` to your environment
 +Adding package `keras/20200204:sklearn` to your environment
 +(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 keras]$ which pip
 +/work/my_workgroup/sw/keras/20200204-sklearn/bin/pip
 +</code>
 +
 +A prefix has reappeared on the prompt — the path at which the new Keras virtualenv was created — and the ''pip'' command refers to an executable within that directory tree.  The virtualenv is ready to have SKLearn installed.
 +
 +<code bash>
 +(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ pip install sklearn
 +Collecting sklearn
 +  Downloading ..
 +    :
 +Successfully built sklearn
 +Installing collected packages: joblib, scikit-learn, sklearn
 +Successfully installed joblib-0.14.1 scikit-learn-0.22.1 sklearn-0.0
 +(/work/my_workgroup/sw/keras/20200204-sklearn) [(my_workgroup:user)@login01 ~]$ python3 -c "import sklearn;print(sklearn.__version__)"
 +0.22.1
 +</code>
 +
 +The Keras environment with SKLearn is now ready for use.
 +
 +===== Job Scripts =====
 +
 +Any job scripts you submit that want to run scripts using this virtualenv should include something like the following toward its end:
 +
 +<code>
 +#
 +# Setup Keras virtualenv:
 +#
 +vpkg_require keras/20200204:sklearn
 +
 +#
 +# Run a Python script in that virtualenv:
 +#
 +python3 my_keras_work.py
 +rc=$?
 +
 +#
 +# Do cleanup work, etc....
 +#
 +
 +#
 +# Exit with whatever exit code our Python script handed back:
 +#
 +exit $rc
 +</code>