Research Blog
Welcome to my Research Blog.
This is mostly meant to document what I am working on for myself, and to communicate with my colleagues. It is likely filled with errors!
This project is maintained by ndrakos
I had set up AHF on Pleiades, as outlined in this post.
This is too slow on to run on the 1024 and 2048 simulations, so I am setting it up for MPI parallelization.
AHF can be recompiled after changing the flag in Makefile.config. I changed it from “Standard OpenMP” to “Standard MPI+OpenMP”
There are two extra parameters to set in the unput file:
LevelDomainDecomp: \(2**{\rm LevelDomainDecomp}\) on order \(B/r_{\rm vir, max}\), where \(B\) is the boxsize, and \(r_{\rm vir, max}\) is the virial radius of the most massive obect. I calculated this to be 5-6 for our problem. I am setting it to \(6\).
NcpuReading: number of CPUs reading data. I’m setting it to 8 for now.
Here is my updated job script:
#PBS -S /bin/csh
#PBS -j oe
#PBS -l select=1:ncpus=32:mpiprocs=8:ompthreads=4:mem=100GB
#PBS -l walltime=72:00:00
#PBS -q ldan
module load mpi-sgi/mpt
module load comp-intel/2018.3.222
setenv MPI_SHEPHERD true
cd .
export LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/nasa/pkgsrc/sles12/2016Q4/lib:/pleiades/u/ndrakos/install_to_here/gsl_in/lib
mpiexec -np 8 ./runAHF_wfirst1024.zsh
and the wrapper:
#!/bin/zsh -f
cd /pleiades/u/ndrakos/AHF/bin
minsnap=0
maxsnap=500
i=$minsnap
while ((i<=maxsnap))
do
mysnap=snapshot_$( printf '%03d' $i)
mpiexec -n 8 ./AHF /u/ndrakos/wfirst1024/HaloFinder/1024snap_inputs/${mysnap}.input
i=$((i+1))
done
The output files are split into 8 (since that was the number of MPI tasks), for each snapshot.
For the merger trees require you specify which files you are cross-correlating. From what I can tell, this means you have to cross-correlate each of the split files seperately.
This still isn’t that fast, and I am a bit worried about how it is handling the split files in the merger trees. I am thinking of just changing to using Rockstar. Rockstar is more popular, and there is plenty of documentation.