software:valet:02_packages

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
software:valet:02_packages [2023-12-21 10:42] – [Can I Add My Own Packages?] freysoftware:valet:02_packages [2023-12-21 10:44] (current) – [Can I Add My Own Packages?] frey
Line 1: Line 1:
 +====== Packages ====== 
  
 +{{:software:valet:valet-icon.png?128 |}}In VALET, a //package// is a software component, where "component" can mean an application, a code library, a set of man pages, or a combination of all these pieces.  For example, Mathematica would be an application:  all the end user needs is for the appropriate directory to be on his/her ''PATH'' so that typing ''mathematica'' starts the program((The ''mathematica'' startup script itself alters the environment further so that its own shared libraries are found properly, etc.)).  On the other hand, a program using OpenMPI for parallelism might need both the ''PATH'' and ''LD_LIBRARY_PATH'' configured in order to function.
 +
 +As newer versions of software are released, the older versions are usually not removed immediately when the new are installed on a system.  Users need time to adapt their workflow to changes in the software; occasionally the changes may be radical enough (e.g. Python 2 versus 3) that the workflow simply cannot be adapted to the new version.  A VALET //package// can have multiple //versions// associated with it.  Using the same examples as above, the system may have Mathematica 6, 7 and 8 available to users.  Likewise, OpenMPI 1.3.3 and 1.4.2 may be available, in each case in standard TCP and Myricom-enabled flavors.
 +
 +VALET uses a //versioned package id// similar to those used in "modules".  The //versioned package id// is a string containing two pieces, the package and the version, separated by a "/" character:
 +<code>
 +mathematica/6
 +mathematica/7
 +mathematica/8
 +openmpi/1.3.3
 +openmpi/1.3.3-mx
 +openmpi/1.4.2
 +openmpi/1.4.2-mx
 +</code>
 +A special form of //versioned package id// references whatever version of a package is indicated to be the default version:
 +<code>
 +mathematica/default
 +openmpi/default
 +</code>
 +The "/" and version information can also be omitted to indicate the default version of a package:
 +<code>
 +mathematica
 +openmpi
 +</code>
 +The default version is specified in each package's configuration file -- more on that later.
 +
 +===== What Packages Are Available? =====
 +
 +A list of available packages is displayed using the ''vpkg_list'' command:
 +
 +<code>
 +[user@host ~]$ vpkg_list
 +Available packages:
 +  openmpi
 +  pgi
 +  matlab
 +</code>
 +
 +For individual packages, a list of versions is displayed using ''vpkg_versions'':
 +
 +<code>
 +[user@host ~]$ vpkg_versions openmpi
 +Available versions in package (* = default version):
 +
 +[/opt/shared/valet/2.0/etc/openmpi.vpkg_json]
 +openmpi            Open MPI:  Message-Passing Interface
 +  gcc              alias to openmpi/1.8.2
 +  intel64          alias to openmpi/1.8.2-intel64
 +  1.8.2-intel64    Version 1.8.2, with Intel64(2015) compilers
 +  1.8.2-gcc-4.8.3  Version 1.8.2, with GCC(4.8.3) compilers
 +  1.8              alias to openmpi/1.8.2
 +* 1.8.2            Version 1.8.2, with GCC(system) compilers
 +</code>
 +
 +The "*" indicates which version is currently the default.
 +
 +A verbose description of a package or package version can be displayed using the ''vpkg_info'' command with a non-versioned package id (e.g. ''mathematica'') or a versioned package id (e.g. ''mathematica/6''):
 +
 +<code>
 +[user@host ~]$ vpkg_info openmpi/default
 +[openmpi/1.8.2] {
 +  Version 1.8.2, with GCC(system) compilers
 +  prefix: /opt/shared/openmpi/1.8.2
 +  affect dev env: <inherit>
 +  add std paths: <inherit>
 +}
 +
 +</code>
 +
 +===== Applying Environment Changes from Package(s) =====
 +
 +One or more packages are setup in the environment using the ''vpkg_require'' command:
 +
 +<code>
 +[user@host ~]$ vpkg_require openmpi/default
 +</code>
 +
 +Any problems encountered will result in no changes being made to the environment.  For example, trying to configure two versions of OpenMPI simultaneously yields:
 +
 +<code>
 +[user@host ~]$ vpkg_require openmpi/default openmpi/pgi-1.3.3
 +ERROR: unable to add versioned package: openmpi/pgi-1.3.3 conflicts with version: openmpi/pgi10-1.4.2-mx
 +</code>
 +
 +Problems may occur due to dependencies or incompatibilities that a version of a package has w.r.t. another package:
 +
 +<code>
 +[user@host ~]$ vpkg_require openmpi/pgi10-1.4.2 pgi/8
 +ERROR: unable to add versioned package: pgi/8 conflicts with version: pgi/10
 +[user@host ~]$ vpkg_require dummy gaussian
 +ERROR: incompatibility detected between a previously added package and gaussian/g03.c02
 +[user@host ~]$ vpkg_require gaussian dummy
 +ERROR: incompatibility detected between dummy/standard and gaussian/g03.c02
 +</code>
 +
 +===== Undoing Environment Changes =====
 +
 +VALET attempts to make a snapshot of your environment prior to each invocation of ''vpkg_require''.  To remove the changes made by your last use of ''vpkg_require'', use the ''vpkg_rollback'' command:
 +
 +<code>
 +[user@host ~]$ vpkg_rollback 
 +WARNING: no snapshots defined
 +</code>
 +
 +The above message is displayed because ''vpkg_require'' has not yet been called!  If you have called ''vpkg_require'' more than once, you can also rollback through more than just the last call to ''vpkg_require'' by supplying the number of rollbacks to attempt.  For example,
 +
 +<code>
 +[user@host ~]$ vpkg_rollback 2
 +</code>
 +
 +would perform two rollbacks.  To remove all changes made to your environment since the first use of ''vpkg_require'', use the ''all'' argument:
 +
 +<code>
 +[user@host ~]$ vpkg_rollback all
 +</code>
 +
 +===== What If I Forget These Commands? =====
 +
 +No problem!  Just type ''vpkg_help'' on the cluster for a summary of the commands.
 +
 +<code>
 +[user@host ~]$ vpkg_help
 +
 +  Package ids:
 + 
 +   A package is identified by an identifier and an optional version identifer separated
 +   by a forward slash:
 + 
 +     matlab                     Default version of matlab
 +     matlab/default             Default version of matlab
 +     matlab/R2007a              R2007a version of matlab
 + 
 +  Commands are:
 + 
 +   vpkg_cli                             Invoke the UDSEC command-line utility
 +   vpkg_list                            List the packages defined for UDSEC management
 +   vpkg_versions <pkgid>                List the versions available for a given package
 +   vpkg_info <pkgid>                    Show information for a package (or a specific
 +                                        version of a package)
 +   vpkg_require <pkgid> {<pkgid>}       Add configuration for the given packages
 +                                        to the environment
 +   vpkg_devrequire <pkgid> {<pkgid>}    Add configuration for the given packages
 +                                        to the environment; includes setup of the LDFLAGS
 +                                        and CPPFLAGS variables with library/header paths
 +                                        for the packages
 +   vpkg_rollback {#|all}                Attempt to restore the environment state that
 +                                        existed before the last vpkg_require.  A numerical
 +                                        argument does multiple rollback operations, e.g.
 +                                        "vpkg_rollback 2" will revert the last two calls
 +                                        of "vpkg_require".  The argument "all" reverts
 +                                        every call to "vpkg_require" made in the shell.
 +   vpkg_history                         Display a list of all versioned packages that
 +                                        have been added to the environment.
 +</code>
 +
 +===== Can I Add My Own Packages? =====
 +
 +Sure!  Version 1 of VALET exclusively used XML for specification of package files, and is documented in the next section.  For VALET 2, both XML and JSON formats are available.
 +
 +To augment the system-wide package files, create a ''.valet'' directory in your home directory and add your own package definition files inside it.  VALET automatically looks in ''~/.valet'' for package files.  VALET will also check any directories present in the ''VALET_PATH'' environment variable((In VALET 1 the ''VALET_SYSCONFDIR'' variable must be used, not ''VALET_PATH'')).  The ''VALET_PATH'' behaves like the ''PATH'' envrionment variable:
 +
 +<code sh>
 +  export VALET_PATH="${HOME}/valet/etc:${WORKDIR}/sw/valet"
 +</code>
 +
 +The paths in ''VALET_PATH'' will be consulted in the order they are specified, and VALET's default configuration directory is implicitly added to the end of the list of paths.  This means you can "override" packages that are defined elsewhere.  For example, if I add ''openmpi.vpkg'' to my ''~/.valet'' directory, then ''vpkg_require openmpi'' will use the definitions in ''~/.valet/openmpi.vpkg'' in preference to the ones defined by the system administrator in VALET's default configuration directory.
 +
 +----
 +
 +[[software:valet:valet|Prev: Intro]] | [[software:valet:03_packagefile|Next: XML Package Files]]