abstract:darwin:install_software:udbuild

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

abstract:darwin:install_software:udbuild [2021-04-22 11:33] – created anitaabstract:darwin:install_software:udbuild [2021-04-22 11:37] (current) – anita
Line 1: Line 1:
 +===== UDBUILD Software Deployment =====
 +The software installed and deployed on the DARWIN cluster each has it's own
 +methods for compiling and installing.  To manage this process, the HPC team
 +has a set of standards and technology used to reduce complexity and bring
 +consistency to the process.
  
 +Software is built, installed, and accessed using the
 +[[software:valet:valet|VALET]] system developed by Dr. Jeffrey Frey.  IT
 +developed a set of software helper functions which can be access using VALET
 +by importing the ''udbuild'' vpkg.
 +
 +This page describes the filesystem layout used by IT, and the anatomy of the
 +''udbuild'' file used to build and deploy software.  Throughout this process,
 +it is helpful to have an understanding of how to use
 +[[software:valet:valet|VALET]] to add and remove software packages from your
 +environment.
 +
 +==== Filesystem ====
 +Software is deployed to ''/opt/shared''.
 +The ''udbuild'' system defaults to this ''/opt/shared''
 +location.  However, this can be changed by setting the ''UDBUILD_HOME''
 +environment variable before initializing the ubuild environment.  A good
 +value for this environment variable for workgroup software is
 +''$WORKDIR/sw'', and for personal software installation is
 +''$HOME/sw''. Refer to [[:abstract:darwin:.install_software:workgroup-sw|workgroup software installs]] for help setting up your directories for workgroup storage.
 +
 +Beneath this directory should be an ''attic'' sub-directory for downloaded
 +software bundles, optionally an ''add-ons'' directory for software with
 +optional add-ons, and one sub-directory for each package installed.
 +These sub-directories should always be in all lower-case letter.  One more
 +layer down should be a directory for each version of the software installed.
 +It is important understand that on a complex cluster like DARWIN, the same
 +release of a software package may have multiple installations due to various
 +compiler and dependency package requirements.  These directories are the
 +software installation roots.
 +
 +Underneath the installation root should be a directory called ''src'', which
 +is the un-packed source bundle.  Next to ''src'' should be any ''bin'',
 +''lib'', ''share'', etc. directories neccessary for the final deployment.
 +
 +An illustrated example of the software directory structure is as such:
 +
 +  * opt
 +    * shared
 +      * atlas
 +         * 3.10.3
 +         * 3.10.3-intel
 +         * attic
 +           * ''udbuild''  <sup>// - build and install script for atlas//</sup>
 +      * python
 +         * 2.7.8
 +         * 3.2.5
 +         * add-ons
 +           * python2.7.15
 +             * mpi
 +               * 20180613
 +           * python3.2.5
 +         * attic
 +           * ''udbuild''   <sup>// - build and install script for python//</sup>
 +
 +==== Building ====
 +When building software, the base directory structure (including the ''attic''
 +directory) should be created by you before proceeding further.
 +You should download the software source bundle into ''attic''.  Then,
 +unpack the software bundle and rename the directory to ''src'' as
 +above.  This provides consistency in finding the source bundle and the
 +''udbuild'' file.
 +
 +Examples of builds are provided below (after the udbuild function
 +documentation).
 +
 +=== udbuild functions ===
 +
 +== init_udbuildenv ==
 +This function initializes the udbuild environment. It ensures that you have
 +the required ''PKGNAME'' and ''VERSION'' environment variables defined, you do
 +not have VALET packages loaded before ''udbuild'' in your VALET history (these
 +might affect your build), sets compiler variables like CC, FC, etc., then
 +finally sets your ''PREFIX'' and ''VERSION'' variables based on it's
 +command-line.  These command-line options affect ''init_udbuildenv'':
 +
 +  * none - This is equivalent to not supplying any parameters
 +  * python-addon - Ensure a python VALET package is loaded, and set PREFIX \
 +    appropriately for that python version's add-on installation path
 +  * r-addon - Ensure an R VALET package is loaded, and set PREFIX \
 +    appropriately for that R version's add-on installation path
 +  * node-addon - Ensure a Node.JS version is loaded and set the PREFIX \
 +    appropriately for that Node.JS versions' add-on installation path
 +  * Any other arguments are treated as the names of VALET packages which are \
 +    loaded and added to the ''VERSION'' environment variable.
 +
 +After all of this, your ''PREFIX'' variable will be set to
 +
 +  ${UDBUILD_HOME:-/opt/shared}/$PKGNAME/$VERSION
 +
 +== debug ==
 +Drop into a debug shell.  If the debug shell is exited cleanly, then the
 +udbuild script will continue from there.  This is a useful routine to use
 +when creating a udbuild script.  You may want to build the script based
 +on documentation, and run a ''debug'' function between a configure and
 +make step to verify the environment looks sane.  After the first successful
 +compile, you can then remove the debug line.
 +
 +== download ==
 +The ''download'' function takes a mandatory and optional argument.  The mandatory,
 +first, argument is the URL to download.  The resulting file will be named after
 +the last part of the URL unless the optional second argument is specified.  In this
 +case, the resulting file will be named after the second argument.
 +
 +If a file with the same name already exists, the download exits successfully without
 +doing anything.  If you wish to re-download the archive, delete or rename the existing
 +one.
 +
 +== unpack ==
 +The ''unpack'' function takes a mandatory and optional argument.  The mandatory,
 +first, argument is the name of an archive file (tar.gz, tar.bz2, tar.xz, zip, etc.)
 +to extract.  The archive will be unpacked into a directory named ''src'' under the
 +install prefix (versioned directory for installation) unless the optional second argument
 +is specified, then it will be used in place of the name ''src''.  This directory, and
 +its parents if necessary, will be created prior to extraction.  Source archives using
 +the ''tar'' format customarily have a single top-level directory entry which contains
 +the package name and version, this is automatically stripped from the extracted archive.
 +After completing the extraction process, the ''unpack'' function places the udbuild
 +script into the newly created directory to prepare the script for configure and make
 +steps.
 +
 +If the ''src'' (or alternately specified) directory exists, then the archive is not
 +extracted over it.  In this case, the function returns successfully without doing
 +anything.  If you wish to force a new extraction, remove or rename the existing
 +''src'' directory.
 +
 +== create_valet_template ==
 +Create a YAML based valet package file template and place it in the ''attic'' directory
 +if one can be found, and the same directory as the udbuild script if one cannot.  This
 +template is helpful, but usually cannot just be copied into place.  For example, it only
 +knows about the version of the software it is installing, and copying the file blindly would
 +remove entries for all other versions.  Furthermore, the "dependencies" entry is filled with
 +all loaded valet packages, even if they are dependencies of dependencies, and not needed
 +to be explicitly listed.
 +
 +== valet ==
 +This function takes either the name of a package (e.g. ''openmpi''), or a
 +package name/version pair (e.g. ''openmpi/1.8.2'') and return true if there
 +is a VALET package loaded to satisfy this dependency, and false otherwise.
 +
 +This function can be used along with any other shell constructs, such as
 +''if'' ... ''else'' ... ''fi'', to modify the behaviour of a build.
 +
 +== version ==
 +This function takes a string and validates that it exists as a complete
 +entry (i.e. starts, stops, or is bounded by hyphens) in the VERSION string.
 +
 +This function can be used along with any other shell constructs, such as
 +''if'' ... ''else'' ... ''fi'', to modify the behavior of a build.
 +
 +== package ==
 +This function takes a string and validates that it exists as part of
 +the final package name, which may include features.  This is useful for
 +matching features which are specified to configure a software build but
 +don't show up in the version string or require a valet package be
 +available, the string "threads" for OpenMP is a good example of using
 +this feature.
 +
 +== ifvalet ==
 +This function is shorthand for ''if valet "$1"; then shift; eval "$@"; fi''
 +to make udbuild scripts simple to read and code.
 +
 +== ifversion ==
 +This function is shorthand for ''if version "$1"; then shift; eval "$@"; fi''
 +to make udbuild scripts simple to read and code.
 +
 +== ifpackage ==
 +This function is shorthand for ''if package "$1"; then shift; eval "$@"; fi''
 +to make udbuild scripts simple to read and code.
 +
 +== udbuildcapture ==
 +Put all screen output into a capture file.  The main purpose of this is to
 +log questions answered during an interactive isntall, to document what
 +choices were made.
 +
 +== udbuildmon ==
 +This script is helpful to be run during the install phase of a build, for
 +example:
 +
 +  udbuildmon make install
 +
 +It will log all ''open'' for write and ''mkdir'' system calls and log them to
 +a file named ''udbuildmon.log''.  You can use this log file to verify the
 +build did not write any files to unknown locations.  This function should not
 +be necessary with [[http://www.cmake.org|cmake]] builds, as they normally
 +store this information in an ''install_manifest.txt'' file.
 +
 +== apath ==
 +Append a path to a variable and ensure that variable is exported.  The required
 +first argument is the environment variable name, all remaining arguments are
 +paths to append to the end of the environment variable.  A colon (:) character
 +is used as the delimiter, as is standard in path environment variables.
 +
 +== ppath ==
 +Prepend a path to a variable similar to ''apath'', but instead of adding the
 +path to the end, add it to the beginning.  Arguments are the same as ''apath''
 +
 +== rpath ==
 +Remove a path from an environment variable.  The required first argument is the
 +environment variable name.  All remaining arguments are removed from the
 +environment variable.  If an entry exists multiple times, all instances are
 +removed.
 +
 +== aflag ==
 +Append a flag to an environment variable.  The required first argument is the
 +environment variable name.  All remaining arguments are added to the environment
 +variable.  The ''aflag'' variable works under two contexts, which depend on the
 +status of the first argument.  If it is an already defined bash array variable
 +type, then the remaining arguments are added as new elements in the array.  In
 +all other cases, the remaining arguments are added to the string using a space
 +character as a delimiter.
 +
 +Using a bash array has the advantage of allowing flags which contain whitespace
 +characters.  If this is a requirement, the following steps should be undertaken:
 +
 +  CONFIG=()                            #Declare CONFIG as a bash array
 +  aflag CONFIG --prefix="$PREFIX"      #Specify flags, this one is regular
 +  aflag CONFIG --title="My Software"   #Specify flags, his one with spaces
 +  ./configure  "${CONFIG[@]}"          #bash syntax for arrays as arguments
 +
 +== pflag ==
 +Prepend flags to an environment variable.  This is the same as the ''aflag''
 +function, but it puts its arguments at the beginning of the variable.  Its
 +arguments are identical.
 +
 +== rflag ==
 +Remove a flag from an environment variable.  This works similar to the
 +''rpath'' variable and also supports bash arrays.
 +
 +== udexpect ==
 +This is a wrapper around the TCL ''expect'' utility to simplify the process of
 +answering questions for interactive builds.  This function accepts an expect
 +script as STDIN (the normal method is via HERE-DOC) and provides all the basics
 +of running ''expect''.  Some standard responses are provided to simplify the
 +process:
 +
 +  - enter - Send a carrige return as if the user pressed their "Enter" key
 +  - yes - Send the string "yes" as if the user typed "yes" and pressed "Enter"
 +  - no - Send "no" and press "Enter"
 +  - y - Send "y" and press "Enter"
 +  - n - Send "n" and press "Enter"
 +  - respond text - Send //text// and press "Enter"
 +  - keypress c - Send the character //c// and DO NOT press "Enter"
 +  - user - Prompt the person at the keyboard for a respone, and send it, press "Enter"
 +
 +== makeflags_set ==
 +Update a file (presumably a Makefile and specified as the first argument) which uses
 +the syntax "key=value" and update the value of the second argument to be that of the
 +third argument.  This is a simple helper function to make it simple to edit basic
 +information in a Makefile.
 +
 +== makeflags_prepend ==
 +Update a file similar to ''makeflags_set'', except prepend the third argument to the
 +existing value, instead of replacing it.
 +
 +== makeflags_append ==
 +Update a file similar to ''makeflags_set'', except append the third argument to the
 +existing value, instead of replacing it.
 +
 +=== udbuild script examples ===
 +
 +== simple ==
 +In this example, an easy-to-install software package called cmake is built
 +and isntalled.  It has no software dependencies, and uses the standard 
 +''configure'', ''make'', ''make install'' procedure used by very many
 +open source software packages.
 +
 +To prepare for this build, you would want to create the following directories:
 +
 +
 +<file sh udbuild>
 +#/bin/bash -l
 +
 +PKGNAME=cmake                       #These are required variables and must
 +VERSION=3.11.3                      #be set before calling 'init_udbuildenv'
 +                                    #Setting the "SITEURL" is not required, but is
 +                                    #helpful later.
 +SITEURL=https://cmake.org/files/v${VERSION%.*}
 +
 +PKGINFO='Cross-Platform Make'       #Helpful for the ''create_valet_template'' function,
 +URLINFO=http://www.cmake.org        #but not required at all
 +
 +vpkg_devrequire udbuild/2           #Use VALET to load the udbuild v2 environment
 +init_udbuildenv                     #Initialize the udbuild environment
 +
 +                                    #Download the source file if it doesn't already exist
 +download $SITEURL/$PKGNAME-$VERSION.tar.gz
 +unpack $PKGNAME-$VERSION.tar.gz     #Unpack the source file into $PREFIX/src and cd there
 +
 +create_valet_template               #Create a template file for valet in the attic (you
 +                                    #can't just copy this into place)
 +
 +./configure --prefix=$PREFIX        #Run your normal configure, without
 +                                    #having to define your own PREFIX
 +                                    #variable, because 'init_udbuildenv' did
 +                                    #that for you.
 +
 +make                                #normal make commands
 +
 +udbuildmon make install             #wrap your 'make install' with the
 +                                    #'udbuildmon' function to log what files
 +                                    #and directories were changed.
 +</file>
 +
 +It is imperitive to start udbuild scripts with the string ''#!/bin/bash -l''
 +because this instructs bash to setup the VALET system.
 +
 +== medium ==
 +<file sh udbuild>
 +#!/bin/bash -l
 +
 +PKGNAME=cdo
 +VERSION=1.9.4
 +SITEURL=https://code.mpimet.mpg.de/attachments/download/17374/
 +
 +vpkg_devrequire udbuild/2 eccodes/2.8.0:threads proj/5.1.0
 +vpkg_devrequire netcdf/4.6.1 udunits/2.2.26 fftw/3.3.8
 +init_udbuildenv
 +
 +create_valet_template
 +
 +download $SITEURL/$PKGNAME-$VERSION.tar.gz
 +unpack $PKGNAME-$VERSION.tar.gz
 +
 +aflag CONFIG --with-szlib="$SZIP_PREFIX"                
 +aflag CONFIG --with-hdf5="$HDF5_PREFIX"                 
 +aflag CONFIG --with-netcdf="$NETCDF_PREFIX"             
 +aflag CONFIG --with-eccodes="$ECCODES_PREFIX"   
 +aflag CONFIG --with-proj="$PROJ_PREFIX"                 
 +aflag CONFIG --with-udunits2="$UDUNITS_PREFIX"  
 +aflag CONFIG --with-threads=yes                         
 +aflag CONFIG --with-curl=yes                    
 +aflag CONFIG --with-libxml=yes
 +
 +./configure --prefix="$PREFIX" $CONFIG
 +
 +sed -i 's/#include <string>/#include <string.h>/' src/modules.cc
 +
 +make
 +
 +udbuildmon make install
 +</file>
 +
 +In this example, we use ''vpkg_devrequire'' to specify additional dependencies
 +needed to build the ''cdo'' package.  ''PREFIX'', however, will still be set
 +to ''/opt/shared/cdo/1.6.4''.
 +
 +== complex ==
 +<file sh udbuild>
 +#!/bin/bash -l
 +
 +PKGNAME=hdf4
 +VERSION=4.2.13
 +SITEURL=https://support.hdfgroup.org/ftp/HDF/HDF_Current/src
 +
 +PKGINFO='HDF4: Data Model, Library, & File Format'
 +URLINFO=http://www.hdfgroup.org/products/hdf4/
 +                                 
 +vpkg_devrequire udbuild szip/2.1.1
 +init_udbuildenv
 +
 +create_valet_template
 +                                     
 +download $SITEURL/hdf-$VERSION.tar.bz2
 +unpack hdf-$VERSION.tar.bz2
 +                 
 +aflag CFLAGS -fPIC                           
 +if valet intel; then   
 +        aflag CFLAGS -qopt-jump-tables=large
 +fi             
 +
 +aflag CONFIG --disable-netcdf
 +aflag CONFIG --with-szlib=$SZIP_PREFIX
 +                                     
 +# Make shared libraries (sans fortran support):
 +./configure --prefix="$PREFIX" --enable-shared --disable-fortran $CONFIG
 +        
 +make install
 +
 +make clean
 +
 +# Make fortran enabled HDF4:
 +./configure --prefix="$PREFIX" --disable-shared --enable-fortran $CONFIG
 +
 +make install
 +
 +</file>
 +
 +In this more complicated example, we still need dependencies, but this time
 +one of them will affect the ''PREFIX'' variable.  The Intel64 compiler will
 +be used, and PREFIX will be set to ''/opt/shared/hdf4/4.2.10-intel64''.
 +
 +Furthermore, specific ''CFLAGS'' changes will be made for this compiler.
 +This example also illustrates how the ''VERSION'' string can be used.  Here,
 +we would set additional flags for the ''./configure'' script if the ''VERSION''
 +string were set to ''4.2.10-sansnetcdf''.  These options allow one build file
 +to build multiple versions of a package, and with only minor changes near
 +the top of the script (namely to the ''VERSION'' variable and the
 +''init_udbuildenv'' command-line.
 +
 +Another interesting thing we do here is to make sure the installation is
 +as complete as possible.  HDF4 does not support shared object files for
 +fortran libraries.  So, first we build the shared objects which are possible,
 +then we enable fortran and ensure the full compliment of archive ''.a'' files
 +are present.
 +
 +== python ==
 +<file sh udbuild-cdf>
 +#!/bin/bash -l
 +
 +PKGNAME=cdf
 +VERSION=20180613
 +PY_VER=3.6.5
 +NETCDF_VER=4.6.1
 +
 +PKGINFO='Python NetCDF, HDF5, HDF4, and dependent Modules'
 +URLINFO=https://pypi.org/
 +
 +vpkg_devrequire udbuild python/$PY_VER netcdf/$NETCDF_VER
 +init_udbuildenv python-addon
 +
 +create_valet_template
 +
 +ppath PATH        "$PREFIX/bin"
 +apath LD_RUN_PATH "$PREFIX/lib"
 +
 +pip_install CDF cdflib netCDF4 h5netCDF wera2netcdf Puppy nco
 +pip_install python-hdf4 hdf5able hdf5pickle hdf5storage
 +pip_install dtt2hdf ascii2hdf5 h5json h5pyd hdf5_matlab_reader
 +pip_install HDFconvert mrr2c hdfdict LazyHDF5 simpletraj mdtraj
 +pip_install mriqc ncagg
 +pip_install 'MDAnalysis[analysis, AMBER]'
 +</file>
 +
 +The python example above is used to install the cdf and related modules as an add-on for python version 3.6.5. It is also considered a complex example since it displays the use of the option ''python-addon'' for ''init_udbuildenv'' to initialize the udbuild environment and ensure a python VALET package is loaded, sets ''PREFIX'' appropriately for python version's add-on installation path, and utilizes the udbuild ''pip_install'' function to call pip with using the correct ''PREFIX''. To use the python add-on module bundle, you can setup a VALET package.  Below is an example VALET yaml package created from the ''create_valet_template'' function.  It should be trimmed, as szip, hdf4, and hdf5 are all dependencies of netcdf.  The dependencies section should look like the ''vpkg_devrequire'' calls of the udbuild script.
 +
 +<file sh python-cdf.vpkg_yaml>
 +python-cdf:
 +  description:          Python NetCDF, HDF5, HDF4, and dependent Modules
 +  url:                  https://pypi.org/
 +  prefix:               /opt/shared/python/add-ons
 +  
 +  default-version:      "3.6.5:20180613"
 +  
 +  versions:
 +    "3.6.5:20180613":
 +      description:      cdf compiled with system compilers
 +      prefix:           python3.6.5/cdf/20180613
 +      dependencies:
 +        -               python/3.6.5
 +        -               szip/2.1.1
 +        -               hdf4/4.2.13
 +        -               hdf5/1.10.2
 +        -               netcdf/4.6.1
 +</file>