abstract:farber:install_software

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
abstract:farber:install_software [2018-03-23 20:00] – [udbuild software deployment] sraskarabstract:farber:install_software [2018-09-10 13:27] (current) – [Filesystem] anita
Line 1: Line 1:
 +===== UDBUILD Software Deployment =====
 +The software installed and deployed on the Farber cluster each has it's own
 +methods for compiling and installing.  To manage this process, the HPC team
 +has a set of standards and technology used to reduce complexity and bring
 +consistency to the process.
  
 +Software is built, installed, and accessed using the
 +[[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey.  IT
 +developed a set of software helper functions which can be access using VALET
 +by importing the ''udbuild'' vpkg.
 +
 +This page describes the filesystem layout used by IT, and the anatomy of the
 +''udbuild'' file used to build and deploy software.  Throughout this process,
 +it is helpful to have an understanding of how to use
 +[[software:valet:valet|VALET]] to add and remove software packages from your
 +environment.
 +
 +==== Filesystem ====
 +Software is deployed to ''/opt/shared''.
 +The ''udbuild'' system defaults to this ''/opt/shared''
 +location.  However, this can be changed by setting the ''UDBUILD_HOME''
 +environment variable before initializing the ubuild environment.  A good
 +value for this environment variable for workgroup software is
 +''$WORKDIR/sw'', and for personal software installation is
 +''$HOME/sw''. Refer to [[:technical:developer:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage.
 +
 +Beneath this directory should be an ''attic'' sub-directory for downloaded
 +software bundles, optionally an ''add-ons'' directory for software with
 +optional add-ons, and one sub-directory for each package installed.
 +These sub-directories should always be in all lower-case letter.  One more
 +layer down should be a directory for each version of the software installed.
 +It is important understand that on a complex cluster like Farber, the same
 +release of a software package may have multiple installations due to various
 +compiler and dependency package requirements.  These directories are the
 +software installation roots.
 +
 +Underneath the installation root should be a directory called ''src'', which
 +is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
 +''lib'', ''share'', etc. directories neccessary for the final deployment.
 +
 +An illustrated example of the software directory structure is as such:
 +
 +  * opt
 +    * shared
 +      * atlas
 +         * 3.10.2
 +           * lib
 +           * include
 +           * src
 +             * ''udbuild''   <sup>// - build and install script for atlas 3.10.2//</sup>
 +         * 3.10.2-intel64
 +         * attic
 +      * python
 +         * 2.7.8
 +           * bin
 +           * include
 +           * lib
 +           * share
 +           * src
 +             * ''udbuild''   <sup>// - build and install script for python 2.7.8//</sup>
 +         * 3.2.5
 +         * add-ons
 +           * python2.7.8
 +             * numpy
 +               * 1.8.2
 +                 * bin
 +                 * lib
 +                 * src
 +                   * ''udbuild''  <sup>// - build and install script for the numpy 1.8.2 installed for python 2.7.8//</sup>
 +             * ipython
 +           * python3.2.5
 +         * attic
 +
 +
 +==== Building ====
 +When building software, the base directory structure (including the ''attic''
 +directory) should be created by you before proceeding further.
 +You should download the software source bundle into ''attic''.  Then,
 +unpack the software bundle and rename the directory to ''src'' as
 +above.  This provides consistency in finding the source bundle and the
 +''udbuild'' file.
 +
 +Examples of builds are provided below (after the udbuild function
 +documentation).
 +
 +=== udbuild functions ===
 +
 +== init_udbuildenv ==
 +This function initializes the udbuild environment. It ensures that you have
 +the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
 +not have VALET packages loaded before ''udbuild'' in your VALET history (these
 +might affect your build), sets compiler variables like CC, FC, etc., then
 +finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
 +command-line.  These command-line options affect ''init_udbuildenv'':
 +
 +  * none - This is equivalent to not supplying any parameters
 +  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
 +    appropriately for that python version's add-on installation path
 +  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
 +    appropriately for that R version's add-on installation path
 +  * Any other arguments are treated as the names of VALET packages which are \
 +    loaded and added to the ''VERSION'' environment variable.
 +
 +After all of this, your ''PREFIX'' variable will be set to
 +
 +  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
 +
 +== valet ==
 +This function takes either the name of a package (e.g. ''openmpi''), or a
 +package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
 +is a VALET package loaded to satisfy this dependency, and false otherwise.
 +
 +This function can be used along with any other shell constructs, such as
 +''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
 +
 +== version ==
 +This function takes a string and validates that it exists as a complete
 +entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.
 +
 +This function can be used along with any other shell constructs, such as
 +''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
 +
 +== ifvalet ==
 +This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
 +to make udbuild scripts simple to read and code.
 +
 +== ifversion ==
 +This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
 +to make udbuild scripts simple to read and code.
 +
 +== udbuildcapture ==
 +Put all screen output into a capture file.  The main purpose of this is to
 +log questions answered during an interactive isntall, to document what
 +choices were made.
 +
 +== udbuildmon ==
 +This script is helpful to be run during the install phase of a build, for
 +example:
 +
 +  udbuildmon make install
 +
 +It will log all ''open'' for write and ''mkdir'' system calls and log them to
 +a file named ''udbuildmon.log''.  You can use this log file to verify the
 +build did not write any files to unknown locations.  This function should not
 +be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
 +store this information in an ''install_manifest.txt'' file.
 +
 +=== udbuild script examples ===
 +
 +== simple ==
 +In this example, an easy-to-install software package called cmake is built
 +and isntalled.  It has no software dependencies, and uses the standard 
 +''configure'', ''make'', ''make install'' procedure used by very many
 +open source software packages.
 +
 +To prepare for this build, you would want to create the following directories:
 +
 +
 +<file sh udbuild>
 +#/bin/bash -l
 +
 +PKGNAME=cmake                       #These are required variables and must
 +VERSION=3.0.1                       #be set before calling 'init_udbuildenv'
 +
 +vpkg_devrequire udbuild             #Use VALET to load the udbuild environment
 +init_udbuildenv                     #Initialize the udbuild environment
 +
 +./configure --prefix=$PREFIX        #Run your normal configure, without
 +                                    #having to define your own PREFIX
 +                                    #variable, because 'init_udbuildenv' did
 +                                    #that for you.
 +
 +make                                #normal make commands
 +
 +udbuildmon make install             #wrap your 'make install' with the
 +                                    #'udbuildmon' function to log what files
 +                                    #and directories were changed.
 +</file>
 +
 +It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
 +because this instructs bash to setup the VALET system.
 +
 +== medium ==
 +<file sh udbuild>
 +#!/bin/bash -l
 +
 +PKGNAME=cdo
 +VERSION=1.6.4
 +
 +vpkg_devrequire udbuild
 +vpkg_devrequire netcdf/4.3.2 grib/1.12.3 udunits/2.2.17 proj/4 fftw/3
 +init_udbuildenv
 +
 +./configure --prefix="$PREFIX"                          \
 +                --with-szlib="$SZIP_PREFIX"             \
 +                --with-hdf5="$HDF5_PREFIX"              \
 +                --with-netcdf="$NETCDF_PREFIX"          \
 +                --with-grib_api="$GRIB_PREFIX"          \
 +                --with-proj="$PROJ_PREFIX"              \
 +                --with-udunits2="$UDUNITS_PREFIX"       \
 +                --with-threads=yes                      \
 +                --with-curl=yes                         \
 +                --with-libxml=yes
 +
 +make
 +
 +udbuildmon make install
 +</file>
 +
 +In this example, we use ''vpkg_devrequire'' to specify additional dependencies
 +needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
 +to ''/opt/shared/cdo/1.6.4''.
 +
 +== complex ==
 +<file sh udbuild>
 +#!/bin/bash -l
 +
 +PKGNAME=hdf4
 +VERSION=4.2.10
 +
 +vpkg_devrequire udbuild szip/2.1
 +init_udbuildenv intel64
 +
 +CFLAGS="$CFLAGS -fPIC"
 +CONFIGURE_FLAGS=''
 +
 +ifvalet intel64         'CFLAGS="$CFLAGS -qopt-jump-tables=large"'
 +
 +ifversion sansnetcdf    'CONFIGURE_FLAGS="--disable-netcdf"'
 +
 +# Make shared libraries (sans fortran support):
 +./configure --prefix="$PREFIX" --enable-shared --disable-fortran \
 +        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS
 +
 +udbuildmon make install
 +
 +make clean
 +
 +# Make fortran enabled HDF4:
 +
 +./configure --prefix="$PREFIX" --disable-shared --enable-fortran \
 +        --with-szlib=$SZIP_PREFIX $CONFIGURE_FLAGS
 +
 +udbuildmon make install
 +</file>
 +
 +In this more complicated example, we still need dependencies, but this time
 +one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
 +be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.
 +
 +Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
 +This example also illustrates how the ''VERSION'' string can be used.  Here,
 +we would set additional flags for the ''./configure'' script if the ''VERSION''
 +string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
 +to build multiple versions of a package, and with only minor changes near
 +the top of the script (namely to the ''VERSION'' variable and the
 +''init_udbuildenv'' command-line.
 +
 +Another interesting thing we do here is to make sure the installation is
 +as complete as possible.  HDF4 does not support shared object files for
 +fortran libraries.  So, first we build the shared objects which are possible,
 +then we enable fortran and ensure the full compliment of archive ''.a'' files
 +are present.
 +
 +== python ==
 +<file sh udbuild>
 +#!/bin/bash -l
 +
 +UDBUILD_HOME=$WORKDIR/sw
 +PKGNAME=netcdf
 +# Change this to the version of netcdf
 +VERSION=1.1.7.1
 +# Change this to the version of python
 +PY_VERS=2.7.8
 +
 +# Load the udbuild package
 +vpkg_devrequire udbuild
 +
 +# These are all the dependencies for netcdf specifying the
 +# particular python version based on PY_VERS environment variable
 +vpkg_devrequire python/$PY_VERS
 +vpkg_devrequire python-numpy/python$PY_VERS
 +vpkg_devrequire netcdf/4.3
 +
 +# This initializes the environment for the build based on the
 +# package name and version, and we use the install option
 +# for a python add-on (python-addon)
 +init_udbuildenv python-addon
 +
 +# These are environment variables needed for the install - hints for
 +# what might be needed can be found in the setup.cfg.template file
 +export HDF5_DIR=${HDF5_PREFIX}
 +export NETCDF4_DIR=${NETCDF_PREFIX}
 +export HDF4_DIR=${HDF4_PREFIX}
 +
 +# Standard install is 'python setup.py build' but we use $PYTHON to get
 +# the correct version of python we are building the add-on for as defined 
 +# above rather than a default system version
 +$PYTHON setup.py build
 +
 +$PYTHON setup.py install --prefix=$PREFIX
 +
 +</file>
 +
 +The python example above is used to install the netcdf library version 1.1.7.1 add-on for python version 2.7.8 in the shared research software directory ''$WORKDIR/sw'' following the directory setup described for [[#filesystem]] above. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes ''$PYTHON'' to get the correct version of python for the install. You can build and install the same version of netcdf 1.1.7.1 for a different version of python by simply changing ''PY_VERS'' to the correct version and rerunning your udbuild script. To use the python add-on, you can setup a VALET package.  Below is an example VALET 2.0 json package called ''python-netcdf'' based on netcdf version 1.1.7.1 installed for both python versions 2.7.8 and 3.2.5. You need to change ''<WORKGROUP_NAME>'' to match the corresponding shared research software directory workgroup name where you installed the python add-on software.
 +
 +<file sh python-netcdf.vpkg_json>
 +#
 +# This is the python-netcdf package for VALET 2.0
 +# supporting json package style.
 +#
 +{
 +  "python-netcdf": {
 +# Change to your install location (i.e. <WORKGROUP_NAME>)
 +    "prefix":           "/home/work/<WORKGROUP_NAME>/sw/python/add-ons",
 +    "url":              "https://pypi.python.org/pypi/netCDF4",
 +    "description":      "Python NetCDF",
 +
 +    "default-version":  "1.7.1.1-python2.7.8",
 +
 +    "standard-paths":   true,
 +    "development-env":  false,
 +
 +    "actions": [
 +      {
 +        "action":       "path-prepend",
 +        "variable":     "PYTHONPATH",
 +        "value":        "${VALET_PATH_PREFIX}/lib/python${PYTHON_VERSION_SHORT}/site-packages"
 +
 +      }
 +    ],
 +
 +    "versions": {
 +      "python2.7.8": {
 +        "alias-to":     "1.1.7.1-python2.7.8"
 +      },
 +      "python3.2.5": {
 +        "alias-to":     "1.1.7.1-python3.2.5"
 +      },
 +
 +      "1.1.7.1-python2.7.8": {
 +        "description":  "Version 1.1.7.1 and python 2.7.8",
 +        "prefix":       "python2.7.8/netcdf/1.1.7.1",
 +        "dependencies": [
 +                        "python/2.7.8",
 +                        "python-numpy/python2.7.8",
 +        "prefix":       "python2.7.8/netcdf/1.1.7.1",
 +        "dependencies": [
 +                        "python/2.7.8",
 +                        "python-numpy/python2.7.8",
 +                        "netcdf/4.3"
 +        ]
 +      },
 +      "1.1.7.1-python3.2.5": {
 +        "description":  "Version 1.1.7.1 and python 3.2.5",
 +        "prefix":       "python3.2.5/netcdf/1.1.7.1",
 +        "dependencies": [
 +                        "python/3.2.5",
 +                        "python-numpy/python3.2.5",
 +                        "netcdf/4.3"
 +        ]
 +      }
 +    }
 +  }
 +}
 +</file>