Note: [in progress of updating… 28 May 2019]
Part 1: Network Python
As DoC machines are Ubuntu Linux, python should be preinstalled. Mostly likely location would be the
bh1511@kythnos:~$ which python /usr/bin/python bh1511@kythnos:~$ which pip /usr/local/bin/pip
This is the system python, calling
python straight away will call the binary
/usr/bin/python on that machine.
For Virtual Environments, see https://realpython.com/python-virtual-environments-a-primer/ or https://www.geeksforgeeks.org/python-virtual-environment/
Problems starts arising when virtual environments are created on the network (e.g. on
/vol) using the system python. It is not guaranteed that every base python is the same. Dependencies may be missing or compiled libraries may be slightly different. Many factors can come into play. The ideal way to circumvent this problem is to have a network python that is machine agnostic, and use that as the base python to create further virtual environments.
Any python would do, recommended to install intelpython2 or intelpython3. To install, download the intelpython linux package, extract it and run
setup_intel_python.sh. It installs IN PLACE, so the path where the extracted package lives is where it is installed. This should be on a networked location so it is accessibe from any cluster machine (e.g. /vol/medic01).
Source the intelpython virtual environment, this will override so that the networked intelpython becomes the default python.
bh1511@kythnos:~$ cd /vol/medic01/users/bh1511/_opt/intelpython3/ bh1511@kythnos:/vol/medic01/users/bh1511/_opt/intelpython3$ source bin/activate (base) bh1511@kythnos:/vol/medic01/users/bh1511/_opt/intelpython3$ (base) bh1511@kythnos:/vol/medic01/users/bh1511/_opt/intelpython3$ which python /vol/medic01/users/bh1511/_opt/intelpython3/bin/python (base) bh1511@kythnos:/vol/medic01/users/bh1511/_opt/intelpython3$ which pip /vol/medic01/users/bh1511/_opt/intelpython3/bin/pip (base) bh1511@kythnos:/vol/medic01/users/bh1511/_opt/intelpython3$ which conda /vol/medic01/users/bh1511/_opt/intelpython3/bin/conda
Part 2: Creating virtual environments with intelpython
Creating virtual environments with intelpython is a 4 step process:
- Source the intelpython root environment
- Create the virtual environment
- Deactivate the intelpython root environment
- Source the newly created virtualenv
virtualenv binary isn’t packaged by default with intelpython. Install by sourcing the root intelpython environment and run
conda install virtualenv. intelpython default package manager is conda.
# Sourcing the intelpython base virtual environment bh1511@kythnos:~$ source /vol/medic01/users/bh1511/_opt/intelpython3/bin/activate # Creating a virtual environment "myenv" (base) bh1511@kythnos:~$ virtualenv myenv Using base prefix '/vol/medic01/users/bh1511/_opt/intelpython3' New python executable in /homes/bh1511/myenv/bin/python copying /vol/medic01/users/bh1511/_opt/intelpython3/bin/python => /homes/bh1511/myenv/bin/python2 copying /vol/medic01/users/bh1511/_opt/intelpython3/bin/../lib/libpython3.6m.so.1.0 => /homes/bh1511/myenv/lib/libpython3.6m.so.1.0 Installing setuptools, pip, wheel...done. Overwriting /homes/bh1511/myenv/bin/activate.fish with new content # Deactivating intelpython base environment (base) bh1511@kythnos:~$ source deactivate # sourcing the newly created virtual environment bh1511@kythnos:~$ source myenv/bin/activate (myenv) bh1511@kythnos:~$ which python /homes/bh1511/myenv/bin/python (myenv) bh1511@kythnos:~$ which pip /homes/bh1511/myenv/bin/pip (myenv) bh1511@kythnos:~$
“myenv” here now depends on
/vol/medic01/users/bh1511/_opt/intelpython3, which is independent of any one specific machine.
Part 3: Remote deployment with PyCharm