Introduction¶

Basic Information¶

Clusters nodes processors and cores¶

Like most high-performance computing facilities, NAISS mainly features clusters. A computer cluster is in the broad sense terminology for a supercomputer, consisting of a set of connected computers working together so that they can be viewed as a single system.

A node is the individual computer part of each cluster. Nodes are analogous to the computers we use everyday.

And each node in turn has processors made up of computing entities called cores.

Deciding whether you need HPC for your work¶

The key feature of our systems (or most HPC resources in general) is large-scale parallelization of computations. Our resources are used in a wide-range of scientific disciplines. If you want to find out if you work could benefit from our resources, check if:

your application require large computional resources
your application require large memory resources
your application require GPUs
your application can be parallelized

If you decide to use the resources, welcome on board! We provide you with:

supercomputer systems for large simulations and calculations
systems for processing data before and after simulations or calculations
software for simulation and modelling
short term storage for large volumes of data
assistance with using computing and storage resources
experts in different research fields to assist you with using and/or scaling software

Who may use NAISS HPC services¶

NAISS resources are available for both business and academic research. Academic researchers (from Sweden or Europe) may apply for free time/storage allocations. Swedish academic researchers and their collaborators will need to apply for an allocation of time/storage for their projects via NAISS.

Account and Time Allocation¶

Before starting using NAISS resources, you will need to get an account. Each user account must belong to one or more Time Allocations, since we allocate resources and manage job queueing based on the Time Allocation you belong to and not based on your individual user account. Each Time Allocation includes the following information:

list of users belonging to that Time Allocation
number of corehours allocated per 30 days period for all members of the Time Allocation
an expiration date for the Time Allocation
list of clusters available for running jobs

When you submit jobs in a cluster, you should belong to at least one Time Allocation, or the submission will fail. Using Time Allocations allows us to:

Prioritize your submitted jobs compared to other user’s jobs.
Keep track of compute-time used within the last 30 days by users and projects.

How much resources will be needed corehours¶

We allocate compute-time on our systems in corehours and you are granted an amount of corehours on a particular system. Corehours equals the number of cores used in how many hours. A time allocation gives you a certain amount of corehours evenly split over the months of the time allocation.

Basic Linux for new HPC users¶

Working with the resources requires a basic understanding of Linux systems. Some of our users are new to using Linux-based systems and have asked for introductory materials. Here is a collection for the basic command-line operations needed to get started.

Using commands in the shell¶

The shell is the program from which the user controls everything in a text-interface. When you login to a system remotely, you are already in the shell window of the system. If you login to your own system, you are probably on a graphical screen. From there, search for terminal or Ctrl+Alt+T to enter the shell terminal. In the shell, you can start typing commands to perform some action.

Useful Shell commands¶

Upon login we are greeted with the shell

ssh user@dardel.pdc.kth.se
Last login: Fri Aug 8 10:14:59 2017 from example.com
user@dardel-login-1:~> _

Bash Files and directories¶

Command pwd tells me where I am. After login I am in the “home”directory
```
pwd
/cfs/klemming/home/u/user
```

I can change the directory with cd

cd Private
/Private
pwd
/cfs/klemming/home/u/user/Private

I can go one level up with cd ..
I can return to my HOME folder with cd

List the contents with ls -l

user@machine:~/tmp/talks
ls -l
total 237
drwx------ 3 user csc-users 2048 Aug 17 15:21 img
-rw------- 1 user csc-users 18084 Aug 17 15:21 pdc-env.html
-rw------- 1 user csc-users 222051 Aug 17 15:22 remark-latest.min.js

Bash creating directories and files¶

We create a new directory called results and change to it
```
mkdir results
cd results
```
Creating and editing files
Textfiles can be edited on your local computer and then transferred.
Textfiles can also be edited locally using text editor like nano/emacs/vim.

Copying moving renaming and deleting¶

# copy file
cp draft.txt backup.txt
# recursively copy directory
cp -r results backup
# move rename file
mv draft.txt draft_2.txt
# move rename directory
mv results backup
# move directory one level up
mv results ..
# remove file
rm draft.txt
# remove directory and all its contents
rm -r results

File permissions with chmod¶

In Linux systems all files have a user, a group and a set of privileges which determines what resources a user can access. Every file has three different kind of access: read®, write(w) and execute(x), as well as three different kind of permissions depending on if the person is the owner(u=user) of the file, in the same group(g) or someone else(o=other).

chmod g+w file

Adds(+) write(w) permissions for group(g) to the file.

There is another way to set the permissions by using numbers. Assume that each permission equals the number listed below:

Number	Type
0	no permissions
1	execute
2	write
4	read

chmod 753 file

Gives the user the read, write and execute permission(4+2+1), whereas users in the same group get read and execute permissions (4+1) while others get write and execute permissions (2+1).

Access Control Lists¶

For more fine grained control of access to files, so called POSIX ACLs (Access Control Lists) can be used. An ACL allows the owner of a file or directory to control access to it on a user-by-user or group-by-group basis. To view and modify an ACL, the commands getfacl and setfacl are used.

getfacl¶

The command getfacl is used to get file access control lists (ACLs)

getfacl dir
# file  dir1
# owner  lama tst
# group  users
user::rwx
group::r-x
other::r-x

the directory dir1 is owned by the user lama-tst with permissions read®, write(w) and execute(x). The permissions of the group and others are set to r-x which means that the group and others can read and execute.

Note

Use option -c to skip the comment header.

setfacl¶

setfacl -m user:jon:rwx dir1
getfacl -c dir1
user::rwx
user:jon:rwx
group::r-x
mask::rwx
other::r-x

Now the user jon has been added and can access to dir1 with permission read®, write(w) and execute(x).

Note

The options -x is used to remove entries from the ACLs of file and -R to recurse into subdirectories.

Bash history and tab completion¶

history preserve commands used

history
689  cd ..
691  cd Documents/
692  cp -r introduction_PDC /cfs/klemming/home/h/hzazzi/Documents/Presentations
693  ssh dardel.pdc.kth.se
694  cd introduction_PDC/
695  ls -l
696  pwd
697  history

If I want to repeat…
```
!696
pwd
~/Documents/introduction_PDC
```
Use also the TAB key for completion
CTRL/R to search for previous commands
Arrows up/down to scroll for earlier commands

Bash finding things¶

Extract lines which contain an expression with grep

# extract all lines that contain searchme
grep searchme draft.txt

If you do not know what a UNIX command does, examine it with man
```
man [command]
```
Find files with find
```
find ~ | grep lostfile.txt
```
We can pipe commands and filter results with |
```
grep energy results.out | sort | uniq
```

Bash Redirecting output¶

Print content of a file to screen
```
cat test.out
```
Redirect output to a file
```
cat test.out > myfile.txt
```
Append output to a file
```
cat test.out >> myfile.txt
```

Bash Writing shell scripts¶

#! bin bash
# here we loop over all files that end with * out
for file in *.out; do
  echo $file
  cat $file
done

We make the script executable and then run it

# Make it executable
chmod u+x my_script
# run it
./my_script

Arguments to script can be passed by using $¶

File example

#! bin bash
echo "Hi" $1 $2

./myscript Nils Nilsson
Hi Nils Nilsson

$1..$X: First…Xth argument

To start executing such scripts, you would need to start with a text-editor. Choosing a text-editor is a matter of personal choice, with Vim and Emacs being traditional and popular programs. There are also many other new and interesting editors to choose from. Open your favorite text-editor and copy-paste the file example above and save with file as