2x2Documentation_part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f33d0d0-3073-4ce1-93c6-05a148c201b7",
   "metadata": {},
   "source": [
    "## Running the 2x2 Simulation chain on Polaris using Singularity container"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ffd71cdb",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Obtain a debug node to do our work\n",
    "qsub -I -l select=1 -l filesystems=home:eagle -l walltime=1:00:00 -q debug -A ALCF_for_DUNE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75776b5a-b771-4bd8-af5a-287437c9b265",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Module load command for software loading\n",
    "module use /soft/spack/gcc/0.6.1/install/modulefiles/Core\n",
    "module load apptainer\n",
    "\n",
    "#load python\n",
    "module use /soft/modulefiles ; module load conda; conda activate base"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8ea8f62d-4fb2-4b97-bc51-4397dbb24d76",
   "metadata": {},
   "outputs": [],
   "source": [
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER_DIR='/grand/ALCF_for_DUNE/2x2_sim_data/containers'\n",
    "export SINGULARITY_BIND=$ARCUBE_DIR,/grand/ALCF_for_DUNE:/grand/ALCF_for_DUNE\n",
    "\n",
    "#Change this to your directory\n",
    "export ARCUBE_DIR='/home/aleena/2x2_sim_111424'\n",
    "\n",
    "#change the directory to where you want the output files\n",
    "export ARCUBE_OUTDIR_BASE='/grand/ALCF_for_DUNE/users/aleena/2x2_sim_111424/out'\n",
    "export ARCUBE_LOGDIR_BASE='/grand/ALCF_for_DUNE/users/aleena/2x2_sim_111424/log'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0489e60-7d0a-4f4e-b98d-60cfd32f804a",
   "metadata": {},
   "outputs": [],
   "source": [
    "#only once\n",
    "git clone -b MiniRun6 https://github.com/DUNE/2x2_sim\n",
    "cd 2x2_sim\n",
    "\n",
    "'''\n",
    "Make these changes in the files once you have it cloned (line with - is to be changed to the line with +)\n",
    "\n",
    "1. run-validation/run_validation.sh\n",
    "if [[ \"$ARCUBE_RUNTIME\" == \"NONE\" ]]; then\n",
    "-    module load python/3.11\n",
    "+    #module load python/3.11\n",
    "\n",
    "2. run-validation/install_validation.sh\n",
    "-module unload python 2>/dev/null\n",
    "-module load python/3.11\n",
    "+#module unload python 2>/dev/null\n",
    "+#module load python/3.11\n",
    "\n",
    "3. run-ndlar-flow/install_ndlar_flow.sh\n",
    "\n",
    "-module load python/3.11\n",
    "+# module load python/3.11\n",
    "\n",
    "Aleena: You should keep the venv so ignore the below lines on commenting out the venv\n", 
    "-python -m venv \"$venvName\"\n",
    "-source \"$venvName\"/bin/activate\n",
    "+#python -m venv \"$venvName\"\n",
    "+#source \"$venvName\"/bin/activate\n",
    "\n",
    "4. run-larnd-sim/install_larnd_sim.sh\n",
    "-module load cudatoolkit/11.7\n",
    "-module load python/3.11\n",
    "+# module load cudatoolkit/11.7\n",
    "+#module load python/3.11\n",
    "'''\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc9a03bd",
   "metadata": {},
   "source": [
    "#### IGNORE FOR MiniRun6 and beyond ###\n",
    "NDLAR flow installation: Only because of the bug with pylandau, can skip in later MiniRun versions\n",
    "\n",
    "Change pylandau package as it has some bugs (Solution provided by Matt Kramer)\n",
    "\n",
    "- cd run-ndlar-flow\n",
    "- Before installing the ndlar flow stage, comment out pip install -e . for ndlar_flow at the end of the install_ndlar_flow.sh\n",
    "- Run the install script\n",
    "- cd ndlar_flow directory inside run-ndlar-flow\n",
    "- source ../flow.venv/bin/activate\n",
    "- Replace `pylandau` with `pylandau @ git+https://github.com/cuddandr/pylandau.git#egg=pylandau` in ndlar_flow/setup.py \n",
    "- replace `pylandau` with `git+https://github.com/cuddandr/pylandau.git` in ndlar_flow/env-nompi.yaml and ndlar_flow/env.yaml (optional ndlar_flow/test/env-nompi.yaml and ndlar_flow/test/env.yaml)\n",
    "- Make sure you are inside run-ndlar-flow/ndlar_flow, pip install -e .\n",
    "- exit out of venv by `deactivate` command"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4de50123-d0a0-4a79-99ce-87932c30b95f",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#run this command to install tmp_bin\n",
    "mkdir -p tmp_bin\n",
    "cp /usr/bin/time tmp_bin/\n",
    "\n",
    "#if not installed already isntall ndlarflow\n",
    "cd run-ndlar-flow\n",
    "./install_ndlar_flow.sh\n",
    "cd ..\n",
    "#if not installed already isntall ndlarflow as above\n",
    "\n",
    "#install larnd-sim\n",
    "cd run-larnd-sim\n",
    "./install_larnd_sim.sh\n",
    "cd ..\n",
    "\n",
    "#install convert2h5\n",
    "export ARCUBE_CONTAINER='genie_edep.3_04_00.20230912.sif'\n",
    "cd run-convert2h5\n",
    "./install_convert2h5.sh\n",
    "cd ..\n",
    "\n",
    "#install validation\n",
    "cd run-validation\n",
    "./install_validation.sh\n",
    "cd ..\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cd396a3",
   "metadata": {},
   "source": [
    "#### IGNORE FOR MiniRun6 and beyond ###\n",
    "The below command is to run singularity container manually to execute our bash scripts. Ideally the run scripts (run_genie, run_edep_sim, etc) should take care of loading the container and then running the script but right now for some reason it keeps calling the singularity command in a loop (looking into this)\n",
    "\n",
    "> [!IMPORTANT]  \n",
    "> once the above problem is resolved, we dont need to wouldn't need to run the below command manually"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a469090",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "#### IGNORE FOR MiniRun6 and beyond ###\n",
    "#once the above problem is resolved, we dont need to wouldn't need to run the below command manually\n",
    "\n",
    "\n",
    "##THE ABOVE PROBLEM IS SOLVED, DO NOT NEED THESE COMMANDS\n",
    "export ARCUBE_CONTAINER='genie_edep.3_04_00.20230912.sif'\n",
    "singularity exec -B $ARCUBE_DIR,/grand/ALCF_for_DUNE:/grand/ALCF_for_DUNE $ARCUBE_CONTAINER_DIR/$ARCUBE_CONTAINER /bin/bash\n",
    "export ARCUBE_CONTAINER='NONE'\n",
    "export ARCUBE_RUNTIME='NONE'\n",
    "\n",
    "'''\n",
    "#IGNORE ABOVE COMMANDS, MAKE SURE POINT 5 OF CELL 4 IS MADE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "29337274-a528-47c0-b917-99fd2cb6ca4b",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "GENIE Stage-nu\n",
    "'''\n",
    "\n",
    "cd run-genie\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER='genie_edep.3_04_00.20230912.sif'\n",
    "export ARCUBE_DET_LOCATION='MiniRun5-Nu'\n",
    "export ARCUBE_OUT_NAME='genie.nu'\n",
    "export ARCUBE_DK2NU_DIR='/grand/ALCF_for_DUNE/users/fathima/2x2_sim_data/NuMI_dk2nu/newtarget-200kA_20220409'\n",
    "export ARCUBE_EXPOSURE='1E15'\n",
    "export ARCUBE_GEOM='geometry/Merged2x2MINERvA_v4/Merged2x2MINERvA_v4_noRock.gdml'\n",
    "export ARCUBE_TUNE='AR23_20i_00_000'\n",
    "export ARCUBE_XSEC_FILE='/grand/ALCF_for_DUNE/users/fathima/2x2_sim_data/gxspl-NUsmall.xml'\n",
    "export ARCUBE_RUN_OFFSET=0\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "\n",
    "./run_genie.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "875ea997-a34b-4bed-9a2b-11099571a737",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "GENIE Stage-rock\n",
    "'''\n",
    "\n",
    "cd run-genie\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER='genie_edep.3_04_00.20230912.sif'\n",
    "export ARCUBE_OUTDIR_BASE='/grand/ALCF_for_DUNE/users/fathima/out.test'\n",
    "export ARCUBE_LOGDIR_BASE='/grand/ALCF_for_DUNE/users/fathima/logs_cmd.test'\n",
    "export ARCUBE_DET_LOCATION='MiniRun5-Rock'\n",
    "export ARCUBE_OUT_NAME='genie.rock'\n",
    "export ARCUBE_DK2NU_DIR='/grand/ALCF_for_DUNE/users/fathima/2x2_sim_data/NuMI_dk2nu/newtarget-200kA_20220409'\n",
    "export ARCUBE_EXPOSURE='1E15'\n",
    "export ARCUBE_GEOM='geometry/Merged2x2MINERvA_v4/Merged2x2MINERvA_v4_justRock.gdml'\n",
    "export ARCUBE_TUNE='AR23_20i_00_000'\n",
    "export ARCUBE_XSEC_FILE='/grand/ALCF_for_DUNE/users/fathima/2x2_sim_data/gxspl-NUsmall.xml'\n",
    "export ARCUBE_RUN_OFFSET=1000000000\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_genie.sh\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d0f7b3b2",
   "metadata": {},
   "outputs": [],
   "source": [
    "#exit out of above container\n",
    "\n",
    "\n",
    "##THE PROBLEM IS SOLVED, DO NOT NEED THESE COMMANDS\n",
    "\n",
    "#load the below container for the next few steps\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "singularity exec -B $ARCUBE_DIR,/grand/ALCF_for_DUNE:/grand/ALCF_for_DUNE $ARCUBE_CONTAINER_DIR/$ARCUBE_CONTAINER /bin/bash\n",
    "export ARCUBE_CONTAINER='NONE'\n",
    "export ARCUBE_RUNTIME='NONE'\n",
    "\n",
    "#IGNORE ABOVE COMMANDS, MAKE SURE POINT 5 OF CELL 4 IS MADE"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "90d03a53-d04b-4160-b0e0-a2c3796b262b",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Edepsim Stage-nu\n",
    "'''\n",
    "\n",
    "cd run-edep-sim\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "export ARCUBE_GENIE_NAME='genie.nu'\n",
    "export ARCUBE_EDEP_MAC='macros/2x2_beam.mac'\n",
    "export ARCUBE_GEOM_EDEP='geometry/Merged2x2MINERvA_v4/Merged2x2MINERvA_v4_withRock.gdml'\n",
    "export ARCUBE_RUN_OFFSET=0\n",
    "export ARCUBE_OUT_NAME='edep.nu'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "\n",
    "./run_edep_sim.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "43e023ca-cb1d-43a6-9cc0-31707b8147bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Edepsim Stage-rock\n",
    "'''\n",
    "\n",
    "cd run-edep-sim\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "export ARCUBE_GENIE_NAME='genie.rock'\n",
    "export ARCUBE_EDEP_MAC='macros/2x2_beam.mac'\n",
    "export ARCUBE_GEOM_EDEP='geometry/Merged2x2MINERvA_v4/Merged2x2MINERvA_v4_withRock.gdml'\n",
    "export ARCUBE_RUN_OFFSET=1000000000\n",
    "export ARCUBE_OUT_NAME='edep.rock'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_edep_sim.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d85a6230-c3e8-47ed-943d-75aeb96ab475",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Hadd Stage-nu\n",
    "'''\n",
    "\n",
    "cd run-hadd\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "export ARCUBE_IN_NAME='edep.nu'\n",
    "#only one file produced, increase this to 10 to hadd together 10 1e15 files to form a 1e16 root file\n",
    "export ARCUBE_HADD_FACTOR=1\n",
    "export ARCUBE_OUT_NAME='edep.nu.hadd'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_hadd.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "384708e6-4e3b-47e5-a323-7f2305b03fbf",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "Hadd Stage-rock\n",
    "'''\n",
    "\n",
    "cd run-hadd\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "export ARCUBE_IN_NAME='edep.rock'\n",
    "#only one file produced, increase this to 10 to hadd together 10 1e15 files to form a 1e16 root file\n",
    "export ARCUBE_HADD_FACTOR=1\n",
    "export ARCUBE_OUT_NAME='edep.rock.hadd'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_hadd.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3d0b6c35",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "spill build\n",
    "'''\n",
    "\n",
    "cd run-spill-build\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "export ARCUBE_NU_NAME='edep.nu.hadd'\n",
    "#change pot to 1e16 if your hadd-ed files have 1e16 POT, in this tutorial only 1e15\n",
    "export ARCUBE_NU_POT=1E15\n",
    "export ARCUBE_ROCK_NAME='edep.rock.hadd'\n",
    "export ARCUBE_ROCK_POT=1E15\n",
    "export ARCUBE_OUT_NAME='Tutorial.spill'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_spill_build.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c06017d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "convert2h5\n",
    "'''\n",
    "\n",
    "cd run-convert2h5\n",
    "export ARCUBE_RUNTIME='SINGULARITY'\n",
    "# export ARCUBE_CONTAINER='genie_edep.3_04_00.20230912.sif'\n",
    "export ARCUBE_CONTAINER='sim2x2_ndlar011.sif'\n",
    "export ARCUBE_ACTIVE_VOLUME='volTPCActive'\n",
    "export ARCUBE_SPILL_NAME='spill'\n",
    "export ARCUBE_OUT_NAME='convert2h5'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_convert2h5.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ed172db",
   "metadata": {},
   "source": [
    "Change the location of LIGHT_LUT FILE in `run-larnd-sim/larnd-sim/larndsim/config/config.yaml` in 4 places  \n",
    "`/global/cfs/cdirs/dune/www/data/2x2/simulation/larndsim_data/light_LUT/lightLUT_Mod0.npz` -> `/grand/ALCF_for_DUNE/users/aleena/products/lightLUT_Mod0.npz`   \n",
    "`/global/cfs/cdirs/dune/www/data/2x2/simulation/larndsim_data/light_LUT/lightLUT_Mod123.npz` -> `/grand/ALCF_for_DUNE/users/aleena/products/lightLUT_Mod123.npz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2029f134",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "larnd\n",
    "'''\n",
    "\n",
    "#No container required for this stage onwards\n",
    "\n",
    "\n",
    "#------------------------------------------------------------------------------------------------------\n",
    "#Only if you havent gotten a node yet to workon, you will need to ask for compute nodes to work on GPU \n",
    "# and reload enviornment variables    \n",
    "\n",
    "#Obtain a debug node to do our work\n",
    "qsub -I -l select=1 -l filesystems=home:eagle -l walltime=1:00:00 -q debug -A ALCF_for_DUNE\n",
    "\n",
    "export ARCUBE_CONTAINER_DIR='/grand/ALCF_for_DUNE/2x2_sim_data/containers'\n",
    "#Change this to your directory\n",
    "export ARCUBE_DIR='/home/aleena/2x2_sim_111424/2x2_sim'\n",
    "#change the directory to where you want the output files\n",
    "export ARCUBE_OUTDIR_BASE='/grand/ALCF_for_DUNE/users/aleena/2x2_sim_111424/out'\n",
    "export ARCUBE_LOGDIR_BASE='/grand/ALCF_for_DUNE/users/aleena/2x2_sim_111424/log'\n",
    "\n",
    "#------------------------------------------------------------------------------------------------------\n",
    "\n",
    "#load conda and cuda\n",
    "module use /soft/modulefiles ; module load conda; conda activate base\n",
    "module load cudatoolkit-standalone/11.8.0\n",
    "\n",
    "cd run-larnd-sim\n",
    "\n",
    "export ARCUBE_RUNTIME='NONE'\n",
    "export ARCUBE_CONVERT2H5_NAME='convert2h5'\n",
    "export ARCUBE_OUT_NAME='larnd'\n",
    "export ARCUBE_LARNDSIM_CONFIG='2x2_old_response'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_larnd_sim.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dc73e4eb",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "NDLARFLOW\n",
    "'''\n",
    "\n",
    "#load conda and cuda\n",
    "module use /soft/modulefiles ; module load conda; conda activate base\n",
    "cd run-ndlar-flow\n",
    "\n",
    "\n",
    "export ARCUBE_RUNTIME='NONE'\n",
    "export ARCUBE_IN_NAME='larnd'\n",
    "export ARCUBE_OUT_NAME='flow'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_ndlar_flow.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4507696",
   "metadata": {},
   "outputs": [],
   "source": [
    "'''\n",
    "validation\n",
    "'''\n",
    "\n",
    "#load conda for python\n",
    "module use /soft/modulefiles ; module load conda; conda activate base\n",
    "\n",
    "#if validation.venv is not present due to failure of admin/install_everyhting, run install script manually\n",
    "\n",
    "#load conda and cuda\n",
    "cd run-validation\n",
    "\n",
    "export ARCUBE_RUNTIME='NONE'\n",
    "export ARCUBE_EDEP_NAME='convert2h5'\n",
    "export ARCUBE_LARND_NAME='larnd'\n",
    "export ARCUBE_FLOW_NAME='flow'\n",
    "export ARCUBE_OUT_NAME='plots'\n",
    "export ARCUBE_INDEX=0\n",
    "\n",
    "./run_validation.sh"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.19"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}