Set-up your connection to hpc05 and explain usage

Set-up your connection to hpc05 and explain usage#

Requirements#

Test if you can ssh without a password from the computer you are running computations (typically IO Jupyterhub) to hpc05. This means that you should set up SSH key and authentication agent on this machine.

  • Ensure that SSH key exists at ~/.ssh/id_rsa, if not – user ssh-keygen to generate it.

  • The simpliest way to set up is using keychain utility. Run this on your local machine (or IO):

echo 'eval `keychain --eval id_rsa`' >> ~/.bash_profile; source ~/.bash_profile

(this will enable SSH agent).

  • Check that there is a record Host hpc05 in your ~/.ssh/config (locally) (see SSH tutorial step 3; if you are setting up a personal PC, you also need to add ProxyCommand ssh -W %h:%p tudelft entry there).

  • Finally, execute:

ssh-copy-id hpc05

(this will add your local SSH keys as trusted on hpc05).

Preparations#

Step 1. Set-up the .bash_profile and install python environment on the hpc05#

Follow the steps in this repo: init_hpc05

Step 2. Set-up a ipyparallel profile on hpc05 and make a connection.#

If not on io, install hpc05 with: conda install -c conda-forge hpc05

Open a notebook or terminal on io and run

import hpc05
hpc05.create_remote_pbs_profile()

If you get an error message read it. Probaly it will tell you to run this command on io:

rm -f ~/ssh-agent.socket; ssh-agent -a ~/ssh-agent.socket;export SSH_AUTH_SOCK=~/ssh-agent.socket; ssh-add

Then try hpc05.create_remote_pbs_profile() again.

Usage#

Warning

This ipyparallel-based workflow is deprecated. If you are learning parallel simulations now, you should use dask instead. Ask in ~coding for pointers.

Starting engines on hpc05 via io#

import hpc05
client, dview, lview = hpc05.start_remote_and_connect(100, folder='~/Work/your_folder_with_modules_on_hpc05/')

See an example: simple_example.md#

Kill your engines, three options:#

  • From your notebook run hpc05.kill_remote_ipcluster()

  • From your notebook run client.shutdown(hub=True)

  • Open a terminal at hpc05 and simply type del to kill all of your engines (and ipcontroller).

Keeping the io and hpc05 environments the same#

  • on hpc05 to get the latest python3 and dev environments (use files from Preparations Step 1.): bash ~/init_hpc05/install_conda.sh

  • on io to get the latest python3 environment: restart your server by clicking on the Stop My Server button

  • on io to get the latest dev environment, bash /environments/install_dev.sh