You are here: Home › ATLAS-BFG › Software

Software

ATLAS Software

ATLAS is providing the complete software via CVMFS. In order to use the several software components add the following function to your .bashrc file:

function setupATLAS() {
    export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
    source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
}

 

Restart your shell or source .bashrc. Now you can type setupATLAS and all software components are listed:

<username>@<ui>:~$ setupATLAS
setupATLAS                        
 lsetup               lsetup <tool1> [ <tool2> ...] (see lsetup -h):
 lsetup agis          ATLAS Grid Information System
 lsetup asetup        (or asetup) to setup an Athena release
 lsetup atlantis      Atlantis: event display
 lsetup eiclient      Event Index 
 lsetup emi           EMI: grid middleware user interface 
 lsetup ganga         Ganga: job definition and management client
 lsetup lcgenv        lcgenv: setup tools from cvmfs SFT repository
 lsetup panda         Panda: Production ANd Distributed Analysis
 lsetup pod           Proof-on-Demand (obsolete)
 lsetup pyami         pyAMI: ATLAS Metadata Interface python client
 lsetup root          ROOT data processing framework
 lsetup rucio         distributed data management system client
 lsetup views         Set up a full LCG release
 lsetup xcache        XRootD local proxy cache
 lsetup xrootd        XRootD data access
advancedTools        advanced tools menu
diagnostics          diagnostic tools menu
helpMe               more help
printMenu            show this menu
showVersions         show versions of installed software

 

You can check the different available versions with:

showVersions

 

that's a lot of information, you can specify the program with:

<username>@<ui>:~$ showVersions root
root versions;
--> 5.34.10-i686-slc5-gcc43-opt
--> 5.34.10-x86_64-slc5-gcc43-opt
...
--> 6.14.04-x86_64-slc6-gcc62-opt               recommended-SL6 
... 
--> 6.18.04-x86_64-centos7-gcc8-opt             recommended      recommended-SL7
Type lsetup "root <value>" to use root

 

Setting up the default version is disabled for root. You will get the following error with a recommended command to run:

<username>@<ui>:~$ lsetup root
Error: You need to specify a root version.  The current recommendation is
         lsetup "root 6.18.04-x86_64-centos7-gcc8-opt" 
                  or 
         export ALRB_rootVersion=6.18.04-x86_64-centos7-gcc8-opt
         lsetup root
       Please consult your analysis group if you need a specific version for
       your work.  To see what versions are available, type
         showVersions root

 

There is even more help with:

<username>@<ui>:~$ helpMe

 

With CVMFS you are able to set up several compiler versions, ROOT, and any kind of ATLAS GRID TOOL.

 

Singularity

Sometimes you want to use software (like an up-to-date python version with machine learning libraries) which is not available on the BFG or CVMFS. But with the help of containers, you can create a virtual environment with all software needed for your work. Due to security reasons, Docker is not available on the BFG. But Singularity is installed on the BFG and can be used on the UIs and inside slurm jobs. The currently installed version of singularity is 3.5.3. 

The official documentation can be found at https://sylabs.io/guides/3.5/user-guide/. In the following, a few topics related to working on the BFG are covered.

Running a docker image

To run a docker image with singularity you need to prefix the URL of the container with 'docker://'. For example:

singularity run docker://godlovedc/lolcow

 

This will run this image: https://hub.docker.com/r/godlovedc/lolcow.

Singularity will convert the docker image into a SIF file. You can also do this by hand with

singularity build lolcow.sif docker://godlovedc/lolcow

 

Then you can just execute the image with

./lolcow.sif

 

To retrieve a docker container from the CERN GitLab registry you need to provide authentication credentials. You can do this by setting the SINGULARITY_DOCKER_USERNAME and  SINGULARITY_DOCKER_PASSWORD  environment variables. DO NOT use your normal CERN password for this but instead create a token with only read access to the container registry. You can create such a token at https://gitlab.cern.ch/profile/personal_access_tokens. Choose a name and only select read_registry. Make sure to save the token somewhere safe, as you won't be able to retrieve it again after GitLab shows it once to you. If you lose the token you can delete it create a new one. Then execute the following lines or add them to your  .bashrc.

export SINGULARITY_DOCKER_USERNAME=<cern-user-name>
export SINGULARITY_DOCKER_PASSWORD=<read-only-token>

 Note: After these environment variables are set you won't be able to retrieve docker containers located at https://hub.docker.com/.

Running an image inside a slurm job

There is no special configuration needed to run singularity in a slurm job. Just call the singularity command inside the job script. For the example above the job script could look like this:

#!/bin/bash

#SBATCH -t 00:05:00
#SBATCH -p nemo_vm_atlsch

singularity run docker://godlovedc/lolcow

Then submit the job with sbatch -J test myjob.sh, assuming the job script is named myjob.sh.

Building an image

For building singularity images you may need sudo permissions. This is not possible on the BFG, because users don't have these permissions. But you can build the singularity image on your local machine and then upload the image via scp to the BFG.

You can also build a docker image on your machine and then upload this docker image to the container registry in CERN GitLab. This can be done with

# build the image, Dockerfile is the same directory the command is executed
docker build -t gitlab-registry.cern.ch/<path-to-your-project> .

# login to container registry of CERN GitLab
docker login gitlab-registry.cern.ch

# upload iamge
docker push gitlab-registry.cern.ch/<path-to-your project>

 

You can also automate the building of the Docker image in the GitLab CI. This way you don't need to upload the image, which can take a lot of time if your upload bandwidth is small. Instructions for building a Docker image inside the CERN GitLab CI can be found here: https://gitlab.cern.ch/gitlabci-examples/build_docker_image.  

Building your own development tools

If you are using a program like vim or emacs for development you may have noticed that the versions installed on the BFG are quite old. Unfortunately, no newer versions are available in the official CentOS repositories. But you can compile newer versions of these tools by yourself.

We will try to provide all dependencies which are needed for compilation. If you don't find your favorite development tool on this list and are missing some dependencies please let the admins know.

In the instructions below the programs will be installed to the $HOME/.local directory. For easier use, you can either create an alias for each program or add the $HOME/.local/bin path to your PATH variable:

# put this into your .bashrc (or equivalent config file)

# option 1 - do this for each program you compile
alias my_program="$HOME/.local/bin/my_program"

# option 2
export PATH=$HOME/.local/bin:$PATH

 

vim

git clone https://github.com/vim/vim.git
cd vim
# feel free to add more flags (check ./configure --help)
./configure --prefix=$HOME/.local \
            --with-features=huge \ 
            --enable-python3interp=yes \
            --with-python3-config-dir=$(python3-config --configdir) \
            --enable-perlinterp=yes \
            --enable-multibyte \
            --enable-cscope  
make -j8
make install

emacs

This needs to be done one a filesystem which supports locking of files. Currently, not all user homes are located on such a filesystem. But you can do this in a workspace from /work.

git clone https://git.savannah.gnu.org/git/emacs.git
cd emacs
./autogen.sh
./configure --prefix=$HOME/.local --without-makeinfo --with-gnutls=no
make -j8
make install

neovim

This needs to be done one a filesytem which supports locking of files. Currently, not all user homes are located on such a filesystem. But you can do this in a workspace from /work.

git clone https://github.com/neovim/neovim.git
cd neovim
make -j8 CMAKE_INSTALL_PREFIX=$HOME/.local CMAKE_BUILD_TYPE=RelWithDebInfo
make install

tmux

This needs to be done one a filesytem which supports locking of files. Currently, not all user homes are located on such a filesystem. But you can do this in a workspace from /work.

git clone https://github.com/tmux/tmux.git
cd tmux
sh autogen.sh
./configure --prefix=$HOME/.local
make -j8
make install

clang

To compile clang you need a newer CMake and GCC version, which can be retrieved from cvmfs.

mkdir clang
cd clang
git clone https://github.com/llvm/llvm-project.git
cd llvm-project
setupATLAS
lsetup cmake
lsetup "lcgenv -p LCG_96b x86_64-centos7-gcc8-opt gcc"
cmake -S llvm -B build -G "Unix Makefiles" -DLLVM_ENABLE_PROJECTS="clang" -DCMAKE_INSTALL_PREFIX=$HOME/.local -DCMAKE_BUILD_TYPE=MinSizeRel -DLLVM_STATIC_LINK_CXX_STDLIB=ON ./llvm
cd build
make -j8
make install

nodejs/npm

To compile node gcc8 or later is needed. You can use a GCC version from cvmfs to compile the code.

git clone https://github.com/nodejs/node.git
cd node
setupATLAS
lsetup "lcgenv -p LCG_96b x86_64-centos7-gcc8-opt gcc"
./configure --prefix=$HOME/.local --partly-static
make -j8
make install