Research Blog
Welcome to my Research Blog.
This is mostly meant to document what I am working on for myself, and to communicate with my colleagues. It is likely filled with errors!
This project is maintained by ndrakos
Here are my notes on getting the SED assignment working on Pleiades.
First of all, I decided to run it on the Lou data analysis nodes instead of Pleaides, even though this means I can’t request as many processes.
I installed FSPS according to these instructions. I also included the roman filters, as detailed here
To install the python version, this required loading the python module, and remembering the --user flag during the python install step. At first I got an error of “Your FSPS version does not seem to be under git version control.” in the python installation… this was because when I ran the make file for the fsps code, I had the wrong path in LD_LIBRARY_PATH, which resulted in a bunch of “no version information available” warnings during installation.
Here is my job script:
#PBS -S /bin/csh
#PBS -j oe
#PBS -l select=1:ncpus=8:mem=10MB
#PBS -l walltime=0:30:00
#PBS -q ldan
module load mpi-sgi/mpt
module load comp-intel/2018.3.222
module load python3/3.7.0
setenv MPI_SHEPHERD true
setenv SPS_HOME /u/ndrakos/Software/fsps/
cd /u/ndrakos/MockCat/Analysis
mpiexec -np 8 python MakeCatalog.py > runMakeCatalog.out
I ran into a problem with reading in the pandas files on Pleiades (they were created in python 3.9, and Pleiades only has up to version 3.7). The error message was: “unsupported pickle protocol: 5”
Since it seems like saving pandas objects to pickle files can run into all sorts of compatibility issues, I decided to use csv files instead, e.g:
import pandas as pd
halocat = pd.read_pickle(myfile)
halocat.to_csv(filename)
and then to read in the csv file,
halocat = pd.read_csv(filename,index_col=0)
The pandas version on Pleiades is outdated, and the function to_numpy() does not exist.
I was having trouble updating the package on Pleiades, so instead I just replaced all uses of that panda function in the code. I.e., everywhere I had something like x = halocat.x.to_numpy() it became x = np.array(halocat.x). I don’t think this introduces any problems.