Differences

This shows you the differences between two versions of the page.

--- technical:recipes:jupyter-notebook [2024-08-02 14:32] – [Setting up SSH tunnel to connect to Jupyter Notebook server] thuachen
+++ technical:recipes:jupyter-notebook [2025-02-04 09:48] (current) – [VALET Package Definition] anita
@@ Line 1: / Line 1: @@
+====== Jupyter Notebook Python Virtual Environment======
+The following steps will walk you through setting up an Anaconda virtual environment with Python 3 and Jupyter Notebook. It will also cover the steps of requesting a compute note to run a Jupyter Notebook session on Caviness. Lastly, it will explain how to set up SSH connections to be able to connect to the Jupyter Notebook session running on a compute node. This setup will be done in the ''it_css'' workgroup directory. This will allow all users in the ''it_css'' workgroup to use the Jupyter Notebook virtual environment. If your workgroup has already created a Jupyter Notebook virtual environment and a corresponding VALET package, then proceed to [[technical:recipes:jupyter-notebook#using-jupyter-notebook-virtual-environment|using Jupyter Notebook Virtual Environment]].
+<note tip>These directions are geared towards setting up and running Jupyter Notebook on Caviness. However, these directions can easily be used on DARWIN, although references to directory paths and software versions will be different.</note>
+Before starting, make sure you set your workgroup and change to the directory where you would like to store your Jupyter Notebook virtual environment. In this example, a general directory ''anaconda-envs'' will be created for Anaconda virtual environments that can be shared with everyone in the workgroup ''it_css'' in the ''sw'' directory.
+<note warning>If you do not have a ''sw'' or ''sw/valet'' directory, please consult with your PI (stakeholder of the workgroup) on how to setup for your workgroup software installs [[abstract:caviness:install_software:workgroup-sw|Workgroup Directory]] on Caviness before proceeding with these instructions. On DARWIN, [[abstract:darwin:install_software:workgroup-sw|workgroup directories]] for ''sw'' and ''sw/valet'' are automatically created as part of the allocation.</note>
+<code bash>
+[traine@login00 ~]$ workgroup -g it_css
+[(it_css:traine)@login00 ~]$ cd $WORKDIR/sw
+[(it_css:traine)@login00 it_css]$ mkdir anaconda-envs
+[(it_css:traine)@login00 it_css]$ chmod 02775 anaconda-envs
+</code>
+===== Create Jupyter Notebook Virtual Environment  =====
+As stated above, in this example we will be using Anaconda version ''2024.02''. This was the latest version available on the Caviness Cluster in August 2024.
+<code bash>
+(it_css:traine)@login00 ~$ vpkg_require anaconda/2024.02
+Adding package `anaconda/2024.02` to your environment
+</code>
+After loading the Anaconda software, we will want create the Jupyter Notebook virtual environment. In this example we will call it ''jupyter-notebook'', but you are welcome to call it anything you would like. We will specify a version directory by date ''20240801'' for the installation of the August 2024 Jupyter Notebook virtual environment. This is needed for setting up a VALET package definition, and allows for multiple versions to be installed based on need. You will be asked to ''Proceed ([y]/n)?'' and you will want to choose ''y'' as this is advising you that prerequisite software is also going to be installed.
+<code bash>
+[(it_css:traine)@login00 ~]$ conda create --prefix $WORKDIR/sw/anaconda-envs/jupyter-notebook/20240801 -c jupyter python=3 jupyter
+Channels:
+ - jupyter
+ - conda-forge
+ - nodefaults
+Platform: linux-64
+Collecting package metadata (repodata.json): done
+Solving environment: done
+## Package Plan ##
+  environment location: /work/it_css/sw/anaconda-envs/jupyter-notebook/20240801
+  added / updated specs:
+    - jupyter
+    - python=3
+The following packages will be downloaded:
+    package                    |            build
+    ---------------------------|-----------------
+      :
+    jupyter-1.1.1              |     pyhd8ed1ab_1           9 KB  conda-forge
+    jupyter-lsp-2.2.5          |     pyhd8ed1ab_1          54 KB  conda-forge
+    jupyter_client-8.6.3       |     pyhd8ed1ab_1         104 KB  conda-forge
+    jupyter_console-6.6.3      |     pyhd8ed1ab_1          26 KB  conda-forge
+    jupyter_core-5.7.2         |     pyh31011fe_1          56 KB  conda-forge
+    jupyter_events-0.11.0      |     pyhd8ed1ab_0          22 KB  conda-forge
+    jupyter_server-2.15.0      |     pyhd8ed1ab_0         320 KB  conda-forge
+    jupyter_server_terminals-0.5.3|     pyhd8ed1ab_1          19 KB  conda-forge
+    jupyterlab-4.3.5           |     pyhd8ed1ab_0         7.3 MB  conda-forge
+    jupyterlab_pygments-0.3.0  |     pyhd8ed1ab_2          18 KB  conda-forge
+    jupyterlab_server-2.27.3   |     pyhd8ed1ab_1          48 KB  conda-forge
+    jupyterlab_widgets-3.0.13  |     pyhd8ed1ab_1         182 KB  conda-forge
+      :
+    yaml-0.2.5                 |       h7f98852_2          87 KB  conda-forge
+    zeromq-4.3.5               |       h3b0a872_7         328 KB  conda-forge
+    zipp-3.21.0                |     pyhd8ed1ab_1          21 KB  conda-forge
+    zstandard-0.23.0           |  py313h80202fe_1         414 KB  conda-forge
+    zstd-1.5.6                 |       ha6fb4c9_0         542 KB  conda-forge
+    ------------------------------------------------------------
+                                           Total:        87.4 MB
+The following NEW packages will be INSTALLED:
+    :
+  jupyter            conda-forge/noarch::jupyter-1.1.1-pyhd8ed1ab_1
+  jupyter-lsp        conda-forge/noarch::jupyter-lsp-2.2.5-pyhd8ed1ab_1
+  jupyter_client     conda-forge/noarch::jupyter_client-8.6.3-pyhd8ed1ab_1
+  jupyter_console    conda-forge/noarch::jupyter_console-6.6.3-pyhd8ed1ab_1
+  jupyter_core       conda-forge/noarch::jupyter_core-5.7.2-pyh31011fe_1
+  jupyter_events     conda-forge/noarch::jupyter_events-0.11.0-pyhd8ed1ab_0
+  jupyter_server     conda-forge/noarch::jupyter_server-2.15.0-pyhd8ed1ab_0
+  jupyter_server_te~ conda-forge/noarch::jupyter_server_terminals-0.5.3-pyhd8ed1ab_1
+  jupyterlab         conda-forge/noarch::jupyterlab-4.3.5-pyhd8ed1ab_0
+  jupyterlab_pygmen~ conda-forge/noarch::jupyterlab_pygments-0.3.0-pyhd8ed1ab_2
+  jupyterlab_server  conda-forge/noarch::jupyterlab_server-2.27.3-pyhd8ed1ab_1
+  jupyterlab_widgets conda-forge/noarch::jupyterlab_widgets-3.0.13-pyhd8ed1ab_1
+   :
+  yaml               conda-forge/linux-64::yaml-0.2.5-h7f98852_2
+  zeromq             conda-forge/linux-64::zeromq-4.3.5-h3b0a872_7
+  zipp               conda-forge/noarch::zipp-3.21.0-pyhd8ed1ab_1
+  zstandard          conda-forge/linux-64::zstandard-0.23.0-py313h80202fe_1
+  zstd               conda-forge/linux-64::zstd-1.5.6-ha6fb4c9_0
+Proceed ([y]/n)? y
+Downloading and Extracting Packages
+...
+...
+Preparing transaction: done
+Verifying transaction: done
+Executing transaction: done
+#
+# To activate this environment, use:
+# > conda activate /work/it_css/sw/anaconda-envs/jupyter-notebook/20240801
+#
+# To deactivate an active environment, use:
+# > conda deactivate
+#
+$
+</code>
+==== VALET Package Definition ====
+The new virtual environment can easily be added to your login shell and job runtime environments using VALET. Ensure you have a workgroup VALET package definition directory present:
+<code bash>
+[(it_css:traine)@login00 ~]$ ls -lad ${WORKDIR}/sw/valet
+[(it_css:traine)@login00 ~]$ echo ${WORKDIR}/sw/anaconda-envs/jupyter-notebook
+/work/it_css/sw/anaconda-envs/jupyter-notebook
+</code>
+Take note of the path echoed, then create a new file named ''${WORKDIR}/sw/valet/jupyter-notebook.vpkg_yaml'' and add the following text to it:
+<code yaml>
+jupyter-notebook:
+    prefix: /work/it_css/sw/anaconda-envs/jupyter-notebook
+    description: Jupyter notebook in Python
+    flags:
+        - no-standard-paths
+    actions:
+        - action: source
+          script:
+              sh: anaconda-activate-2024.sh
+          order: failure-first
+          success: 0
+    versions:
+          "20240801":
+              description: environment built August 1, 2024
+              dependencies:
+                  - anaconda/2024.02
+</code>
+<note>The ''prefix'' shown here ''/work/it_css/sw/anaconda-envs/jupyter-notebook'' will need to be changed based on your workgroup and directory names.</note>
+===== Using Jupyter Notebook Virtual Environment =====
+The versions of the virtual environment declared in the VALET package are listed using the ''vpkg_versions'' command:
+<code bash>
+[(it_css:traine)@login00 ~]$ vpkg_versions jupyter-notebook
+Available versions in package (* = default version):
+[/work/it_css/sw/valet/jupyter-notebook.vpkg_yaml]
+jupyter-notebook  Jupyter notebook in Python
+* 20240801        environment built August 1, 2024
+</code>
+Activating the virtual environment is accomplished using the ''vpkg_require'' command (in your login shell or inside job scripts):
+<code bash>
+[(it_css:traine)@login00 ~]$ vpkg_require jupyter-notebook/20240801
+Adding dependency `anaconda/2024.02` to your environment
+Adding package `jupyter-notebook/20240801` to your environment
+(/work/it_css/sw/anaconda-envs/jupyter-notebook/20240801) [(it_css:traine)@login00 valet]$
+</code>
+===== Running Jupyter Notebook on the Caviness Cluster =====
+Running Jupyter Notebook on Caviness, or any HPC cluster for that matter, takes some extra steps. You can simply install Jupyter Notebook on your personal laptop and start it up. On Caviness, you need to run the Jupyter Notebook on a compute node. The steps below will show you how to request an interactive compute node and use VALET to load the Jupyter Notebook virtual environment. After starting the virtual environment, we will run Jupyter Notebook with arguments that will allow for the session to be accessed via a tunnel connection on your local system.
+<note warning>On Caviness you're **required** to run Jupyter notebook on a **compute node**. If you run it on the login node, you could cause slowness or other issues with the login node, and IT might kill your Jupyter Notebook session without warning.</note>
+==== Requesting an interactive job ====
+In this example we will request an interactive job to connect us to a compute node with 2GB of memory for 1 hour on the ''it_css'' workgroup partition. Also it is important to pass the ''SLURM_EXPORT_ENV=NONE'' when requesting the interactive compute node. It will prevent issues with setting up a clean environment on the compute node.
+<code bash>
+[(it_css:traine)@login00 ~]$ SLURM_EXPORT_ENV=NONE salloc --mem=2G --time=1:00:00 --partition=it_css
+</code>
+==== Loading the Jupyter Notebook virtual environment ====
+After the interactive job has been established, it is time to load the Jupyter Notebook virtual environment with ''VALET''. Even though we did this earlier, it needs to be done again since we are now on a compute node.
+<code bash>
+[[traine@r00n50 1201]$ vpkg_require jupyter-notebook/20240801
+Adding dependency `anaconda/2024.02` to your environment
+Adding package `jupyter-notebook/20240801` to your environment
+(/work/it_css/sw/anaconda_envs/jupyter-notebook/20240801) [traine@r00n50 1201]$
+</code>
+<note tip>Once the Jupyter Notebook virtual environment has been loaded on the interactive compute node session, your prompt should look something like this:
+''(/work/it_css/sw/anaconda_envs/jupyter-notebook/20240801) [traine@r00n50 1201]$''
+</note>
+==== Starting Jupyter Notebook session ====
+When starting the Jupyter Notebook session, specific options are passed, which are used to set up the tunnel connection.
+<code bash>
+[traine@r04n68 1201]$ jupyter notebook --no-browser --ip=$(hostname -s)
+...
+...
+...
+[I 2024-08-01 15:16:27.747 LabApp] JupyterLab extension loaded from /opt/shared/anaconda/2024.02/lib/python3.11/site-packages/jupyterlab
+[I 2024-08-01 15:16:27.747 LabApp] JupyterLab application directory is /opt/shared/anaconda/2024.02/share/jupyter/lab
+[I 2024-08-01 15:16:27.748 LabApp] Extension Manager is 'pypi'.
+[I 2024-08-01 15:16:27.751 ServerApp] jupyterlab | extension was successfully loaded.
+[I 2024-08-01 15:16:27.756 ServerApp] notebook | extension was successfully loaded.
+[I 2024-08-01 15:16:27.757 ServerApp] panel.io.jupyter_server_extension | extension was successfully loaded.
+[I 2024-08-01 15:16:27.760 ServerApp] Serving notebooks from local directory: /home/3347
+[I 2024-08-01 15:16:27.760 ServerApp] Jupyter Server 2.10.0 is running at:
+[I 2024-08-01 15:16:27.761 ServerApp] http://r00n50:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
+[I 2024-08-01 15:16:27.761 ServerApp] http://127.0.0.1:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
+[I 2024-08-01 15:16:27.761 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
+[C 2024-08-01 15:16:27.765 ServerApp]
+    To access the server, open this file in a browser:
+        file:///home/1201/.local/share/jupyter/runtime/jpserver-20300-open.html
+    Or copy and paste one of these URLs:
+        http://r00n50:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
+        http://127.0.0.1:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
+</code>
+<note warning>
+The Jupyter Notebook server is running and there is no prompt. **Make sure you copy the line noted above and it will be used later by changing the compute node to ''localhost''**
+<code>
+http://localhost:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2
+</code>
+</note>
+==== Setting up SSH tunnel to connect to Jupyter Notebook server ====
+With the Jupyter Notebook server running on a compute node on Caviness, an SSH tunnel is needed to be able to make a connection and access the Jupyter Notebook server from a web browser on your local machine. This is done by opening a **second** SSH connection to Caviness. Follow the appropriate section below for Windows (PuTTY) or Terminal on a Linux/Mac laptop.
+<note warning>Your compute node name will likely be different than ''r00n50'', please make sure to change that accordingly. This SSH tunnel connection will have to remain open while you are using Jupyter Notebook. If it is closed or internet connectivity is lost, then your connection to Jupyter Notebook will also be lost.</note>
+==Windows (PuTTY)==
+Once you open the PuTTY, it will show the //Session// window. For the //Host Name(or IP Address)//, you will need to enter the ''<user-name>@caviness.hpc.udel.edu''.
+In addition to your standard connection PuTTY settings, you will need to set up the tunnel setting. This is easily done by loading an existing session you have saved for Caviness and then adding the tunnel settings based on the image below. The tunnel setting is found under the Category //Connection -> SSH -> Tunnels//
+{{:technical:recipes:putty-1.png?400|}}
+Add the **Source port** as ''8888'' and **Destination** as ''r00n50:8888'' as shown in the image above, then click on **Add**. The //Tunnel// settings are now available to your session, so click **Open** to connect. You may need to enter the password for the Caviness/DARWIN account in the prompted window if you do not have an existing session. These settings are not saved. However, it is likely the necessary information to set up the tunnel the next time will change anyway. Remember, this SSH tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.
+{{ :technical:recipes:jupyter-notebook-1.png?nolink&400 |}}
+==Linux/Mac==
+Open a new terminal session on your local machine. Set up a SSH Tunnel using the below ''ssh'' command.
+<code>
+$ ssh -L 8888:r00n50:8888 traine@caviness.hpc.udel.edu
+......................................................................
+    Caviness cluster (caviness.hpc.udel.edu)
+    This computer system is maintained by University of Delaware
+    IT.  Links to documentation and other online resources can be
+    found at:
+      http://docs.hpc.udel.edu/abstract/caviness/
+    For support, please contact consult@udel.edu
+......................................................................
+Last login: Thu Aug  1 12:28:19 2024
+[traine@login00 ~]$
+</code>
+==== Using your local browser to access Jupyter Notebook server  ====
+If everything this far has been set up correctly the final step is as easy as opening a web browser of choice on your local machine and entering the correct URL. If you followed the directions exactly, you can now use the URL from above changing the compute node to ''localhost'' in your local browser to connect to Jupyter Notebook server running on the compute node on Caviness.
+**Example URL:**
+''http://localhost:8888/tree?token=8a17fdf02d91c23270f796620adc9d15fb4c4d47dc705cd2''
+<note important>If you are not able to connect to the Jupyter Notebook session at this point, then you will need to review the prior steps and make sure that you have added and configured the SSH tunnel properly based on your compute node. Remember, the SSH tunnel connection will have to remain open the entire time while you are using Jupyter Notebook.  </note>