Local Python and PyTorch Install - Summit Nodes

PAGE UNDER CONSTRUCTION

Below is the process of obtaining and installing newer versions of Python than what is currently available through anaconda on the summit/dell nodes.

First, obtain a .tgz tarball of the python version you want from python.org, and put it in your home directory ~/ on any of the PHASTA group machines. You can do this in a multitude of ways; an easy way is to download the tarball your personal machine and scp'd it to your home directory on jumpgate:

scp -r local/machine/path/to/python.tgz username@jumpgate-phasta.colorado.edu:'~/.'

Next, log on to jumpgate and then start an interactive session on any of the summit nodes:

qsub -I -l select=1:ncpus=16:mpiprocs=1 -q workq

Once your interactive session is started, go to the directory where you scp'd the python tarball and unzip it with:

tar -xzf Python-3.your.version.tgz

We're now going to build and install a local version of python 3.your.version. First, make a new user specific 'local' dir if it doesn’t already exist (analogous to /local/ for the entire system, but this one lives in your home directory):

mkdir -p ~/local

Next, cd into the python dir that appeared when you unziped the tarball. We now need to configure the python build from within that directory:

./configure --prefix=$HOME/local --enable-optimizations

Note: --prefix needs to match where you created the user specific 'local' directory. Once that is complete, build python with:

make -j

Next, run the following to install the python executables and libraries in local.

make install

We now need to add ~/local to $PATH if it's not already there. Do this by adding export PATH=~/local/:$PATH somewhere that makes sense in your .bashrc file. This will put ~/local to the start of PATH whenever you log on. Note that if you have other software installed in '~/local', this will always access these '~/local' versions first when calling their executables.

You should now have a working python-3.your.version installation. Verify by running which python3 in your terminal. This should show the path to the new python installed in ~/local/bin/.

Next, you should create a virtual environment for the project you are currently working on. Any pip installs you do while in this virtual environment will be saved. A venv permits you to easily 'activate' that same environment setup whenever you return to working on that project with a new session on the summit nodes (interactive or batch). Return to your home directory, then run

</code>python3 -m venv NameOfYourEnv</code>

This will create a folder named 'MyEnv' which will store the virtual environment. You only need to do this once. Once you've created a venv you must activate it with:

source ~/NameOfYourEnv/bin/activate.

You should now see a (MyENV) tag at the start of your command line. If you ever need to deactivate this venv, simply run deactivate at any time.

While your venv of choice is active, you can pip install any of the python libraries you require, and the venv will save those installs for the next time you need to activate the venv.