Chatbot parser generic output part 1 #676
Conversation
… successfully yet
… successfully yet
This file is just used to test some things locally and not part of the parser
…utorial are both read and saved properly
…docs into chatbot_parser
| @@ -0,0 +1,36 @@ | |||
| Getting an HPC Account | |||
| Getting ready to request an account | |||
| All users of AUGent can request | |||
There was a problem hiding this comment.
explain AUGent, or add the acrobym a few lines lower
| an | ||
| account on the HPC, which is part of the Flemish Supercomputing Centre (VSC). | ||
| See HPC policies for more information on who is entitled to an account. | ||
| The VSC, abbreviation of Flemish Supercomputer Centre, is a virtual |
There was a problem hiding this comment.
is this explained anywehre else? maybe add this the acronym is actually form dutch etc
| Brussels University Association, Antwerp University Association and the | ||
| University Colleges-Limburg. The VSC is funded by the Flemish | ||
| Government. | ||
| There are two methods for connecting to HPC-UGent infrastructure: |
There was a problem hiding this comment.
so much terminology is introduced here that it is not funny anymore. and that is also not relevant to getting an account. best to write up some separate section explaining what the vsc is, what augent is , what HPC_UGent is within all this. can be in an appendix for all i care
| Government. | ||
| There are two methods for connecting to HPC-UGent infrastructure: | ||
| - Using a terminal to connect via SSH. | ||
| - Using the web portal |
There was a problem hiding this comment.
can someone first handle the comments in #664 that are related ot the generated texts?
| $ ls /arcanine/scratch/gent/apps/AlphaFold | ||
| 20210812 20211201 20220701 20230310 | ||
| The directories located there indicate when the data was downloaded, so that this leaves room for providing updated datasets later. | ||
| As of writing this documentation the latest version is 20230310. |
There was a problem hiding this comment.
update this,"as of this writing" makes no sense and is confusing
| for example AlphaFold/2.3.1-foss-2022a-CUDA-11.7.0. | ||
| To run the job scripts you need to create a file named T1050.fasta with the following content: | ||
| >T1050 A7LXT1, Bacteroides Ovatus, 779 residues| | ||
| MASQSYLFKHLEVSDGLSNNSVNTIYKDRDGFMWFGTTTGLNRYDGYTFKIYQHAENEPGSLPDNYITDIVEMPDGRFWINTARGYVLFDKERDYFITDVTGFMKNLESWGVPEQVFVDREGNTWLSVAGEGCYRYKEGGKRLFFSYTEHSLPEYGVTQMAECSDGILLIYNTGLLVCLDRATLAIKWQSDEIKKYIPGGKTIELSLFVDRDNCIWAYSLMGIWAYDCGTKSWRTDLTGIWSSRPDVIIHAVAQDIEGRIWVGKDYDGIDVLEKETGKVTSLVAHDDNGRSLPHNTIYDLYADRDGVMWVGTYKKGVSYYSESIFKFNMYEWGDITCIEQADEDRLWLGTNDHGILLWNRSTGKAEPFWRDAEGQLPNPVVSMLKSKDGKLWVGTFNGGLYCMNGSQVRSYKEGTGNALASNNVWALVEDDKGRIWIASLGGGLQCLEPLSGTFETYTSNNSALLENNVTSLCWVDDNTLFFGTASQGVGTMDMRTREIKKIQGQSDSMKLSNDAVNHVYKDSRGLVWIATREGLNVYDTRRHMFLDLFPVVEAKGNFIAAITEDQERNMWVSTSRKVIRVTVASDGKGSYLFDSRAYNSEDGLQNCDFNQRSIKTLHNGIIAIGGLYGVNIFAPDHIRYNKMLPNVMFTGLSLFDEAVKVGQSYGGRVLIEKELNDVENVEFDYKQNIFSVSFASDNYNLPEKTQYMYKLEGFNNDWLTLPVGVHNVTFTNLAPGKYVLRVKAINSDGYVGIKEATLGIVVNPPFKLAAALQHHHHHH |
There was a problem hiding this comment.
this is garbage for the llm. we might need a way to exclude data from teh source, eg some comment that says: skip next X lines
| cd $WORKDIR | ||
| alphafold --fasta_paths=T1050.fasta --max_template_date=2020-05-14 --db_preset=full_dbs --output_dir=$PWD | ||
| echo "Output available in $WORKDIR" | ||
| In case of problems or questions, don't hesitate to contact use at <[email protected]>. |
| @@ -0,0 +1,42 @@ | |||
| Apptainer (formally known as Singularity) | |||
| What is Apptainer? | |||
| Apptainer is an open-source computer program that performs | |||
There was a problem hiding this comment.
is this offical description? who makes up this stuff?
| operating-system-level virtualization (also known as containerisation). | ||
| One of the main uses of Apptainer is to bring containers and | ||
| reproducibility to scientific computing and the high-performance | ||
| computing (HPC) world. Using Apptainer/Singularity containers, |
There was a problem hiding this comment.
remove the /Singularity. maybe add a small section or tip that it was used to be called singularity, and that you might still find refenrences to singularity
| design, and these complete environments can easily be copied and | ||
| executed on other platforms. | ||
| For more general information about the use of Apptainer, please see the | ||
| official documentation at https://apptainer.org/docs/. |
| is mainly done for performance reasons and to avoid that the use of | ||
| Apptainer impacts other users on the system. | ||
| The Apptainer/Singularity image file must be located on either one of | ||
| the scratch filesystems, the local disk of the workernode you are using |
There was a problem hiding this comment.
add clarification the vsc scratch is typically enough, and that local and esp /dev/shm are exception. do we explain the issues with using /dev/shm anywhere?
| $VSC_HOME, /apps or $VSC_DATA filesystems. | ||
| In addition, this implies that running containers images provided via a | ||
| URL (e.g., shub://... or docker://...) will not work. | ||
| If these limitations are a problem for you, please let us know via [email protected]. |
There was a problem hiding this comment.
if they are a problem, just copy the config file, and pass the modfied one via --conifg ... ;)
| Apptainer/Singularity image to a globally writable location, like | ||
| /tmp, or /local directories. Once the image is created, you should | ||
| move it to your desired destination. An example to make an | ||
| Apptainer/Singularity container image: |
| export APPTAINER_TMPDIR=/tmp/$USER/apptainer/tmpdir | ||
| # specified temp dir must exist, so create it | ||
| mkdir -p $APPTAINER_TMPDIR | ||
| # convert Docker container to Apptainer container image |
There was a problem hiding this comment.
maybe also mention spython tool somewhere, it can also help in the convertions
| We strongly recommend the use of Docker Hub, see | ||
| https://hub.docker.com/ for more information. | ||
| Execute our own script within our container | ||
| Copy testing image from /apps/gent/tutorials/Singularity to |
There was a problem hiding this comment.
rename this directory structure
| #PBS -l nodes=2:ppn=15 | ||
| #PBS -l walltime=12:00:00 | ||
| module load intel vsc-mympirun | ||
| mympirun --impi-fallback apptainer exec $VSC_SCRATCH/Debian8_UGentMPI.img ~/ring_debian |
There was a problem hiding this comment.
curious if this works because intelmpi has some form of apptinaer support builtin or not
| @@ -0,0 +1,39 @@ | |||
| Best Practices { #ch:best-practices} | |||
There was a problem hiding this comment.
cleanup these references from the txt, and make them work? (these look like old latex refs, not really markdown stuff)
| - Are there any errors in the script? | ||
| - Are the required modules loaded? | ||
| - Is the correct executable used? | ||
| 2. Check your computer requirements upfront, and request the correct |
There was a problem hiding this comment.
computer requirements? what are those?
| - Are the required modules loaded? | ||
| - Is the correct executable used? | ||
| 2. Check your computer requirements upfront, and request the correct | ||
| resources in your batch job script. |
There was a problem hiding this comment.
batch job script? try to keep consisten tterminology
| resources in your batch job script. | ||
| - Number of requested cores | ||
| - Amount of requested memory | ||
| - Requested network type |
There was a problem hiding this comment.
what does this mean? how can you request it?
| - Requested network type | ||
| 3. Check your jobs at runtime. You could login to the node and check | ||
| the proper execution of your jobs with, e.g., top or vmstat. | ||
| Alternatively you could run an interactive job (qsub -I). |
There was a problem hiding this comment.
clarifiy how this relates to jobs at runtime
| 4. Try to benchmark the software for scaling issues when using MPI or | ||
| for I/O issues. | ||
| 5. Use the scratch file system ($VSC_SCRATCH_NODE, which is mapped to | ||
| the local /tmp) whenever possible. Local disk I/O is always much |
There was a problem hiding this comment.
"local io always fatser", pffft, that is false
| the local /tmp) whenever possible. Local disk I/O is always much | ||
| faster as it does not have to use the network. | ||
| 6. When your job starts, it will log on to the compute node(s) and | ||
| start executing the commands in the job script. It will start in |
There was a problem hiding this comment.
will start ... is that correct?
| with cd $PBS_O_WORKDIR is the first thing which needs to be done. | ||
| You will have your default environment, so don't forget to load the | ||
| software with module load. | ||
| [//]: # (Do not worry, it will render with correct numbering in all cases.) |
| You will have your default environment, so don't forget to load the | ||
| software with module load. | ||
| [//]: # (Do not worry, it will render with correct numbering in all cases.) | ||
| 7. In case your job not running, use "checkjob". It will show why your |
| job is not yet running. Sometimes commands might timeout with an | ||
| overloaded scheduler. | ||
| 8. Submit your job and wait (be patient) ... | ||
| 9. Submit small jobs by grouping them together. See chapter Multi-job submission for |
| @@ -0,0 +1,39 @@ | |||
| Compiling and testing your software on the HPC | |||
| All nodes in the HPC cluster are running the "RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty)" | |||
There was a problem hiding this comment.
@wdpypere we need to take care of crap like this during the migration (or after migration is finished).
There was a problem hiding this comment.
we used to have a macros where we could fix all that in one place. But I will be updating docs yes.
| $ ls -l | ||
| total 512 | ||
| -rw-r--r-- 1 vsc40000 214 Sep 16 09:42 hello.c | ||
| -rw-r--r-- 1 vsc40000 130 Sep 16 11:39 hello.pbs* |
| In order to check all the available modules and their version numbers, | ||
| which are pre-installed on the HPC enter: | ||
| When your required application is not available on the HPC please contact | ||
| any HPC member. Be aware of potential "License Costs". "Open Source" |
| software projects. For larger projects it is more appropriate to use | ||
| makefiles or even an advanced build system like CMake. | ||
| All the HPC nodes run the same version of the Operating System, i.e. RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). So, | ||
| it is sufficient to compile your program on any compute node. Once you |
There was a problem hiding this comment.
really? because it will probably not work
No description provided.