Instructions for Installing SNOMAN on Fedora 27

Here is how I installed SNOMAN on a recent Linux distribution. The first part of these instructions detail setting up a virtual machine which is how I installed it, but you obviously don't have to install it inside a virtual machine (if not you can skip to "Building Cernlib").

First, we have to install qemu: 

$ yum install qemu

Next, download the 64 bit Fedora 27 iso file: 

$ curl -O -L https://download.fedoraproject.org/pub/fedora/linux/releases/29/Server/x86_64/iso/Fedora-Server-dvd-x86_64-27-1.6.iso

and verify the checksums: 

$ md5sum Fedora-Server-dvd-x86_64-27-1.6.iso
f148145afbccebe532dc710362352052  Fedora-Server-dvd-x86_64-27-1.6.iso

Next, we create the disk image: 

$ qemu-img create -f raw sno.img 500G
$ qemu-system-x86_64 --enable-kvm -cdrom Fedora-Server-dvd-x86_64-27-1.6.iso -boot order=d -drive file=sno.img,format=raw -m 2G

I mostly selected the default settings, except modified the partitions so that the root partition used up the remaining ~492 GB of space.

You can now start the virtual machine with the following command: 

$ qemu-system-x86_64 --enable-kvm sno.img -m 2G -smp 4 -device e1000,netdev=net0 -netdev user,id=net0,hostfwd=tcp::2222-:22

And you can ssh it to it using the following command: 

$ ssh -p 2222 localhost

Building cernlib

First, log in as root, update everything, and install some basic utilities: 

# yum update
# yum install tcsh vim

Now, edit the files ~/.bashrc, ~/.cshrc, and ~/.tcshrc and comment out the aliases for rm, mv, and cp (otherwise the installation scripts used later will hang): 

# vim ~/.bashrc
# vim ~/.cshrc
# vim ~/.tcshrc

Then, log out and back in again and run the following commands: 

# yum groupinstall "Development Tools"
# yum install gcc-gfortran imake compat-gcc-34-g77 blas lapack blas-devel lapack-devel motif motif-devel
# mkdir -p /sno/cernlib
# cd /sno/cernlib
# curl -O -L http://www-zeuthen.desy.de/linear_collider/cernlib/new/cernlib-2005-all-new.tgz
# md5sum cernlib-2005-all-new.tgz 
a78d12f46cd8c571d3ab04c9e93981c1  cernlib-2005-all-new.tgz
# tar -xzvf cernlib-2005-all-new.tgz
# curl -O -L http://www-zeuthen.desy.de/linear_collider/cernlib/new/cernlib.2005.corr.2019.01.21.tgz
# md5sum cernlib.2005.corr.2019.01.21.tgz 
f837965eddb9c5c66f53cbdf6c9f96da  cernlib.2005.corr.2019.01.21.tgz
# cp cernlib.2005.corr.2019.01.21.tgz cernlib.2005.corr.tgz
# curl -O -L http://www-zeuthen.desy.de/linear_collider/cernlib/new/cernlib.2005.install.2019.01.21.tgz
# md5sum cernlib.2005.install.2019.01.21.tgz 
771a23ade034e004d69722cf98f5a341  cernlib.2005.install.2019.01.21.tgz
# tar -xzvf cernlib.2005.install.2019.01.21.tgz
# ./Install_cernlib

Now, to make sure everything built properly: 

# ls 2005/lib
cojets.dat    libfritiof.a    libkernlib.a        libpdflib.a
eurodec.dat   libgeant321.a   libkernlib-shift.a  libphotos202.a
flukaaf.dat   libgeant.a      liblepto651.a       libphtools.a
gxint321.f    libgraflib.a    liblepto.a          libpythia6205.a
gxint.f       libgrafX11.a    libmathlib.a        libpythia.a
isajet.dat    libherwig59.a   libpacklib.a        xsneut95.dat
libariadne.a  libherwig.a     libpacklib-shift.a
libcojets.a   libisajet758.a  libpawlib.a
libeurodec.a  libjetset74.a   libpdflib804.a

and make sure you see these files. If you don't see all the libraries, check the 2005/build/log folder and look for any errors.

We'll now create a few symlinks which will be useful later: 

# cd /sno/cernlib/2005/lib
# ln -s liblepto651.a liblepto.a
# ln -s libpythia635.a libpythia.a
# ln -s libpdflib804.a libpdflib.a

Building ROOT 

# yum install git make gcc-c++ gcc binutils libX11-devel libXpm-devel libXft-devel libXext-devel python2-devel fftw3 gsl-devel gsl cmake
# cd /sno
# curl -O -L https://root.cern.ch/download/root_v5.34.38.source.tar.gz
# md5sum root_v5.34.38.source.tar.gz 
c0e3a2ddd7dfab17db4a8e8dc042a579  root_v5.34.38.source.tar.gz
# tar -xzvf root_v5.34.38.source.tar.gz
# cd root
# ./configure
# make -j4

Building FFTW 

# mkdir -p /sno/fftw
# cd /sno/fftw
# curl -O -L http://fftw.org/fftw-2.1.5.tar.gz
# md5sum fftw-2.1.5.tar.gz 
8d16a84f3ca02a785ef9eb36249ba433  fftw-2.1.5.tar.gz
# tar -xzvf fftw-2.1.5.tar.gz
# cd fftw-2.1.5/
# ./configure --prefix=/sno/fftw
# make
# make install

Now, create the file /sno/env.sh and add to it: 

#!/bin/sh

export CERN=/sno/cernlib
export CERN_LEVEL=2005
export CERN_ROOT=$CERN/$CERN_LEVEL
export PATH=$CERN_ROOT/bin:$PATH

export ROOTSYS=/sno/root
export LD_LIBRARY_PATH=$ROOTSYS/lib:$LD_LIBRARY_PATH
export PATH=$ROOTSYS/bin:$PATH

export FFTW_DIR=/sno/fftw/lib

Building QSNO 

# source /sno/env.sh
# mkdir -p /sno/qsno
# cd /sno/qsno
# git clone https://github.com/snoanalysis/qsno_030.git qsno_030
# rsync -avzP nu:/sno/qsno/qsno_030_data.tar.gz .
# md5sum qsno_030_data.tar.gz 
3066f618ada78311649e60170b86319b  qsno_030_data.tar.gz
# tar -xzvf qsno_030_data.tar.gz

Now, we need to edit the file /sno/root/build/unix/makelib.sh and edit line 56 to use the full path to the root bin directory: 

  if [ "`/sno/root/bin/root-config --dicttype`" != "cint" ]; then

Then run the following command: 

# sed -i -e 's@-Llib@-L/sno/root/lib@g' /sno/root/build/unix/makelib.sh

Finally, edit the file /sno/root/config/Makefile.linuxx8664gcc and delete -Wl,--no-undefined from the LDFLAGS variable: 

LDFLAGS       = -m64 $(OPT) $(EXTRA_LDFLAGS) -Wl,--as-needed

# ./install_qsno libs

When prompted for QSNO_DATA type /sno/qsno/qsno_030_data. ROOTSYS should be /sno/root. And for SNO_ROOT you can just enter none. 

# ./configure
# source env.sh
# make

Now, update the /sno/env.sh file to look like this: 

#!/bin/sh

export CERN=/sno/cernlib
export CERN_LEVEL=2005
export CERN_ROOT=$CERN/$CERN_LEVEL
export PATH=$CERN_ROOT/bin:$PATH

export ROOTSYS=/sno/root
export LD_LIBRARY_PATH=$ROOTSYS/lib:$LD_LIBRARY_PATH
export PATH=$ROOTSYS/bin:$PATH

export FFTW_DIR=/sno/fftw/lib

export QSNO_ROOT=/sno/qsno/qsno_030
export QSNO_DATA=/sno/qsno/qsno_030_data
export QPHYSICS=$QSNO_ROOT/qphysics
export LD_LIBRARY_PATH=$QSNO_ROOT/lib:$LD_LIBRARY_PATH
export PATH=$QSNO_ROOT/bin:$PATH

Installing SNOMAN

Edit the /sno/env.sh file and add to it: 

export SNO_ROOT=/sno
export SNO_SNOMAN_VER=5_0294
export SNO_TOOLS=$SNO_ROOT/snoman/tools
export SNO_PLATFORM=LINUXAMD64
export SNO_EXTERNAL_LIBRARIES="-lXm -lXmu -lXt -lX11 -lm \
    -L$CERN_ROOT/lib -lgraflib \
    -lgrafX11 -lpawlib -lmathlib  -lpacklib \
    -llepto -lpythia -lfritiof -lpdflib -lkernlib \
    -Wl,-rpath -lgcc_s -lstdc++ \
    -L/sno/fftw/lib -lrfftw -lfftw"
export SNO_SNOMAN_EXTRACT_SW="(CPP_CODE,HIGH_MEM,GE_DEBUG,NO_DISCON_NPF,NO_DISCON_NPC,NO_DISCON_NPD,NO_DISCON_NPQ,NO_DISCON_NPW,NO_DISCON_ECA,NO_DISCON_PCA,NO_DISCON_OCA)"
export PATH=$SNO_ROOT/snoman/$SNO_SNOMAN_VER/code:$PATH

Note: We explicitly do not link against libjetset74 since it has some common block sizes which are different than those in libpythia and snoman.

Now, edit /sno/qsno/qsno_030/config/setup_qsno.csh and change: 

setenv SNO_ROOT /sno

Now, source this file 

# source /sno/env.sh

As root run the following commands: 

# mkdir -p /sno/snoman
# cd /sno/snoman
# git clone https://github.com/snoanalysis/snoman.git 5_0294
# git clone https://github.com/snoanalysis/snoman-tools.git tools
# cd tools
# ./install_snoman.scr 2>&1 | tee build_output.txt

If everything compiled OK you should see the message: 

SNOMAN linked successfully.

link_snoman complete


install_snoman complete

If not, have a look at build_output.txt and see if you can find the error: 

# cat build_output.txt | grep Error

Now, we need to download the database, and dqxx and anxx files: 

# cd /sno
# scp tlatorre@10.0.2.2:sno.tar.gz .
# md5sum sno.tar.gz
2d3c7db965bc1cfb6fcd4a5c86394698  sno.tar.gz
# tar -xzvf sno.tar.gz

and then go grab a cup of coffee because this takes a long time. This should untar the folders /sno/hepdb, /sno/mcprod/dqxx, and /sno/mcprod/anxx.

Now, edit the /sno/env.sh file and add: 

export CDSERV=/sno/hepdb
export SNO_CODE=$SNO_ROOT/snoman/$SNO_SNOMAN_VER/code
export SNO_PROD=$SNO_ROOT/snoman/$SNO_SNOMAN_VER/prod
export SNO_MCPROD=$SNO_ROOT/snoman/$SNO_SNOMAN_VER/mcprod

and source the file again.

You should now be able to run the SNOMAN demo: 

# snoman.exe

and then at the prompt type @run_demo.

Installing GENIE

First, we'll get the GENIE source files: 

# mkdir -p /sno/genie
# cd /sno/genie
# curl -O -L https://github.com/GENIE-MC/Generator/archive/R-2_12_8.tar.gz
# tar -xzvf R-2_12_8.tar.gz

Installing Pythia6 

# mkdir -p /sno/pythia6
# cd /sno/pythia6
# cp /sno/genie/Generator-R-2_12_8/src/scripts/build/ext/build_pythia6.sh .

Now, edit build_pythia6.sh and change the "wget" command to "wget --no-check-certificate": 

fetchit='wget --no-check-certificate '

Now, run the build pythia6 script: 

# ./build_pythia6.sh

Now edit /sno/env.sh and add the lines: 

export PYTHIA6=/sno/pythia6/v6_428/lib
export LD_LIBRARY_PATH=$PYTHIA6:$LD_LIBRARY_PATH

Installing LHAPDF 

# yum install python-devel
# mkdir -p /sno/lhapdf
# cd /sno/lhapdf
# curl -O -J -L https://lhapdf.hepforge.org/downloads/?f=lhapdf-5.9.1.tar.gz
# tar -xzvf lhapdf-5.9.1.tar.gz
# cd lhapdf-5.9.1
# ./configure --prefix=/sno/lhapdf
# make

Note: If you get an error when building lhapdf about a string literal then you need to change line 881 from: 

PyErr_Format(PyExc_RuntimeError, mesg);

to 

PyErr_Format(PyExc_RuntimeError, "%s", msg);

in lhapdf_wrap.cc.

Now, install it: 

# make install

Now, to install the data sets: 

# cd /sno/lhapdf
# rsync -avzP nu:/usr/share/lhapdf/PDFsets .

Now edit /sno/env.sh and add the lines: 

export LHAPATH=/sno/lhapdf/PDFsets
export LHAPDF_INC=/sno/lhapdf/include
export LHAPDF_LIB=/sno/lhapdf/lib
export LD_LIBRARY_PATH=$LHAPDF_LIB:$LD_LIBRARY_PATH

Bug Fix for gevgen_atmo

There appears to be a bug in the default version of gevgen_atmo which causes most of the interaction vertices to be centered in a 1 meter spherical ball at the center of the detector. This appears to be caused by a bad default value in gevgen_atmo. From the documentation in the gAtmoFlux class:

The position of each flux neutrino [going towards a detector centered at (0,0,0)] is generated uniformly on a plane that is perpendicular to a sphere of radius Rl at the point that is determined by the generated neutrino direction (theta,phi). The size of the area of that plane, where flux neutrinos are generated, is determined by the transverse radius Rt. You can tweak Rl, Rt to match the size of your detector.

However, by default gevgen_atmo sets the radius of this sphere to 1 meter by default which is too small.

To fix this, edit src/Apps/gAtmoEvGen.cxx and change line 472 to: 

atmo_flux_driver->SetRadii(10, 10);

Note: I don't understand exactly why there are two radii. They are labelled as the longitudinal and transverse radii.

Building GENIE

First, install a few more dependencies: 

# yum install log4cpp log4cpp-devel libxml2 libxml2-devel

Now, edit /sno/env.sh and add the lines: 

export LOG4CPP_INC=/usr/include/log4cpp
export LOG4CPP_LIB=/usr/lib64
export LD_LIBRARY_PATH=/usr/lib64:$LD_LIBRARY_PATH

export GENIE=/sno/genie/Generator-R-2_12_8
export LD_LIBRARY_PATH=$GENIE/lib:$LD_LIBRARY_PATH
export PATH=$GENIE/bin:$PATH

and source it.

Now, we need to rebuild ROOT with pythia6 support (not sure if this is actually needed): 

# cd /sno/root
# ./configure --enable-pythia6 --with-pythia6-libdir=$PYTHIA6 --enable-gsl-shared --enable-mathmore
# make

Edit src/make/Make.include and change line 275: 

CXXVRS      = $(shell $(CXX) -dumpfullversion -dumpversion)

Now, run the following commands to build GENIE: 

# cd /sno/genie/Generator-R-2_12_8
# ./configure --with-lhapdf-lib=$LHAPDF_LIB --with-lhapdf-inc=$LHAPDF_INC --with-log4cpp-inc=$LOG4CPP_INC --with-log4cpp-lib=$LOG4CPP_LIB --enable-atmo
# make 2>&1 | tee build_output.txt

To test that everything works you can run: 

# genie -b

Atmospheric Production 

# cd /sno
# git clone https://github.com/snoanalysis/atm-production
# cd atm-production/scripts/genie2mcpl
# make
# cd /sno
# rsync -avzP nu:/nfs/disk1/sno/sw/genie-xsec .

Setting up Autosno 

# yum install perl
# cd /sno
# git clone https://github.com/snoanalysis/autosno.git
# cd autosno/log
# rsync -avzP nu:/nfs/disk1/sno/autosno/log/Rlai_log.tar.gz .
# md5sum Rlai_log.tar.gz 
87989dbea5348eb04b689c3cff0d7fb3  Rlai_log.tar.gz
# tar -xzvf Rlai_log.tar.gz 
# cd /sno/autosno/state
# rsync -avzP nu:/nfs/disk1/sno/autosno/state/{Rlai,Runlogger,ANxx_Neutrino} .

Running Autosno 

# cd /sno/output
# ln -s ln -s /sno/mcprod/anxx anxx
# cd /sno/autosno
# ./bin/autosno -r 10000 -c genie
# ./bin/autosno -r 10000 -c genie_snoman

Resetting Autosno

Occasionally you want to completely reset autosno and start over from scratch. To do that just run the following commad: 

# cd /sno/autosno
# rm -rf state/MC_Atm_Nu_* log/MC_Atm_Nu_* working/* /sno/output/d2o-leta/mc_atmospherics/atmospheric_neutrinos* lock/* state/autosno*.state state/Dummy

Debugging SNOMAN

First, install gdb and libasan: 

# yum install gdb libasan
# dnf debuginfo-install glibc-2.26-30.fc27.x86_64

Next, edit /sno/qsno/qsno_030/config/Makefile.linuxx8664gcc and add "-fsanitize=address" to the following line: 

OPT            := -fsanitize=address -g -O0 -fpermissive

Next, edit /sno/root/config/Makefile.linuxx8664gcc and add "-fsanitize=address" to the following line: 

LDFLAGS       = -fsanitize=address -m64 $(OPT) $(EXTRA_LDFLAGS) -Wl,--as-needed

Now, rebuild QSNO: 

# cd /sno/qsno/qsno_030
# ./install_qsno libs 2>&1 | tee build_output.txt

Next, edit /sno/snoman/tools/get_platform_variables.scr and add "-fsanitize=address" to the cxx_compile_command, c_compile_command, compile_command, and link_command variables in the LINUXAMD64 case. 

# cd /sno/snoman/tools
# export SNO_DEBUG=YES
# ./install_snoman.scr 2>&1 | tee build_output.txt

Now, snoman should quit with an error if anything is overwriting the stack. It's also very useful to run snoman in gdb. In order to get asan to abort on error so you can inspect the stack you should run: 

# export ASAN_OPTIONS=abort_on_error=1

Running XSNOED 

# yum install xorg-x11-xinit xorg-x11-apps xorg-x11-server-source xorg-x11-drivers

Bugs fixed in SNOMAN

  1. fix a few bugs in eca_[pt]diagnostic.for
  2. tkn_get_token.for:91 fix a bug in tkn_get_token() which was reading past the end of the string
  3. hadron_read_particles.for:201 fix loop which went past end of string
  4. ftm_fit_event.for fix buffer overflow in ftm_fit_event.for
  5. muon_capture.for:73 fix buffer overflow in muon_capture.for
  6. vxscint.for:111 fix a buffer overflow in vxscint()
  7. hadron_calor.for Andy's bug fix for hadron_calor.for introduced between releases 5.0291 and 5.0294.
  8. hadron_decay.for:72 fix a buffer overflow in hadron_decay()
  9. ftg_fit_dir_ml.for fix a buffer overflow in ftg_fit_dir_ml()

Bugs fixed in QSNO

  1. gcc errors casting '\0' -> NULL
  2. fix stack buffer overflow in rropen()
  3. comment out a line in QTQIO::SetInteractions() to prevent a buffer overflow FIXME: not sure if this is correct
  4. fix stack buffer overflow in QNCDArray::Initialize()

Questions for Andy

  1. What is the genie-xsec directory? Why isn't it a part of atm-production?

Answer (from Andy):

Not included mainly because these precomputed cross section splines tend to be huge files, and are easily gotten from elsewehere. For standard GENIE configurations you get the splines from here: https://scisoft.fnal.gov/scisoft/packages/genie_xsec/. Or you can also make them yourself using the gmkspl utility (and perhaps gspl2root to convert from XML to TGraphs).

  1. Why does the GENIE module make a new pass when rerun? Seems like it needs to be run twice in order to copy the files correctly.

Answer: Was accidentally running genie with -m flag which forces reprocessing.

  1. Why don't I see any neutrino interactions in the neck when running GENIE?

I suspected it's because the world volume is set to the PSUP. Andy replied:

Ah, yes, it's probably the 840 cm radius world volume, as long as you see uniform vertices within that. (And it's easy enough to change the world volume you wanted to, in the ROOT script that makes the geometry [1]).

Best, Andy

[1] https://github.com/SNOanalysis/atm-production/blob/master/genie/root_geo/sno_geo.C