Chatbot parser generic output part 1

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1.txt

@@ -0,0 +1,33 @@
+Frequently Asked Questions (FAQ)
+New users should consult the Introduction to HPC
+to get started, which is a great resource for learning the basics, troubleshooting, and looking up specifics.
+If you want to use software that's not yet installed on the HPC, send us a
+software installation request.
+Overview of HPC-UGent Tier-2 infrastructure
+Composing a job
+How many cores/nodes should I request?
+An important factor in this question is how well your task is being parallelized:
+does it actually run faster with more resources? You can test this yourself:
+start with 4 cores, then 8, then 16... The execution time should each time be reduced to
+around half of what it was before. You can also try this with full nodes: 1 node, 2 nodes.
+A rule of thumb is that you're around the limit when you double the resources but the
+execution time is still ~60-70% of what it was before. That's a signal to stop increasing the core count.
+See also: Running batch jobs.
+Which packages are available?
+When connected to the HPC, use the commands module avail [search_text] and module spider [module]
+to find installed modules and get information on them.
+Among others, many packages for both Python and R are readily available on the HPC.
+These aren't always easy to find, though, as we've bundled them together.
+Specifically, the module SciPy-bundle includes numpy, pandas, scipy and a few others.
+For R, the normal R module has many libraries included. The bundle R-bundle-Bioconductor
+contains more libraries.
+Use the command module spider [module] to find the specifics on these bundles.
+If the package or library you want is not available, send us a
+software installation request.
+How do I choose the job modules?
+Modules each come with a suffix that describes the toolchain used to install them.
+Examples:
+   AlphaFold/2.2.2-foss-2021a*
+   tqdm/4.61.2-GCCcore-10.3.0*
+   Python/3.9.5-GCCcore-10.3.0*
+   matplotlib/3.4.2-foss-2021a*

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1_metadata.json

@@ -0,0 +1,19 @@
+{
+    "main_title": "FAQ",
+    "subtitle": "How-do-I-choose-the-job-modules",
+    "source_file": "../../mkdocs/docs/HPC/FAQ.md",
+    "title_depth": 3,
+    "directory": "FAQ",
+    "links": {
+        "0": "https://www.ugent.be/hpc/en/training/2023/introhpcugent",
+        "1": "https://www.ugent.be/hpc/en/support/software-installation-request",
+        "2": "https://www.ugent.be/hpc/en/infrastructure",
+        "3": "https://docs.hpc.ugent.be/running_batch_jobs",
+        "4": "https://www.ugent.be/hpc/en/support/software-installation-request"
+    },
+    "parent_title": "",
+    "previous_title": null,
+    "next_title": "FAQ_paragraph_2",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/FAQ/#how-do-i-choose-the-job-modules"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2.txt

@@ -0,0 +1,34 @@
+Modules from the same toolchain always work together, and modules from a
+\*different version of the same toolchain\* never work together.
+The above set of modules works together: an overview of compatible toolchains can be found here:
+https://docs.easybuild.io/en/latest/Common-toolchains.html#overview-of-common-toolchains.
+You can use module avail [search_text] to see which versions on which toolchains are available to use.
+If you need something that's not available yet, you can request it through a
+software installation request.
+It is possible to use the modules without specifying a version or toolchain. However,
+this will probably cause incompatible modules to be loaded. Don't do it if you use multiple modules.
+Even if it works now, as more modules get installed on the HPC, your job can suddenly break.
+Troubleshooting
+My modules don't work together
+When incompatible modules are loaded, you might encounter an error like this:
+Lmod has detected the following error: A different version of the 'GCC' module
+is already loaded (see output of 'ml').
+You should load another foss module for that is compatible with the currently
+loaded version of GCC.
+Use ml spider foss to get an overview of the available versions.
+Modules from the same toolchain always work together, and modules from a
+_different version of the same toolchain_ never work together.
+An overview of compatible toolchains can be found here:
+https://docs.easybuild.io/en/latest/Common-toolchains.html#overview-of-common-toolchains.
+See also: How do I choose the job modules?
+My job takes longer than 72 hours
+The 72 hour walltime limit will not be extended. However, you can work around this barrier:
+* Check that all available resources are being used. See also:
+    * How many cores/nodes should I request?.
+    * My job is slow.
+    * My job isn't using any GPUs.
+* Use a faster cluster.
+* Divide the job into more parallel processes.
+* Divide the job into shorter processes, which you can submit as separate jobs.
+* Use the built-in checkpointing of your software.
+Job failed: SEGV Segmentation fault

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2_metadata.json

@@ -0,0 +1,20 @@
+{
+    "main_title": "FAQ",
+    "subtitle": "Job-failed-SEGV-Segmentation-fault",
+    "source_file": "../../mkdocs/docs/HPC/FAQ.md",
+    "title_depth": 3,
+    "directory": "FAQ",
+    "links": {
+        "0": "https://www.ugent.be/hpc/en/support/software-installation-request",
+        "1": "https://docs.hpc.ugent.be/FAQ/#how-do-i-choose-the-job-modules",
+        "2": "https://docs.hpc.ugent.be/FAQ/#how-many-coresnodes-should-i-request",
+        "3": "https://docs.hpc.ugent.be/FAQ/#my-job-runs-slower-than-i-expected",
+        "4": "https://docs.hpc.ugent.be/FAQ/#my-job-isnt-using-any-gpus",
+        "5": "https://www.ugent.be/hpc/en/infrastructure"
+    },
+    "parent_title": "",
+    "previous_title": "FAQ_paragraph_1",
+    "next_title": "FAQ_paragraph_3",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/FAQ/#job-failed-segv-segmentation-fault"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3.txt

@@ -0,0 +1,28 @@
+Any error mentioning SEGV or  Segmentation fault/violation has something to do with a memory error.
+If you weren't messing around with memory-unsafe applications or programming, your job probably hit its memory limit.
+When there's no memory amount specified in a job script, your job will get access to a proportional
+share of the total memory on the node: If you request a full node, all memory will be available.
+If you request 8 cores on a cluster where nodes have 2x18 cores, you will get 8/36 = 2/9
+of the total memory on the node.
+Try requesting a bit more memory than your proportional share, and see if that solves the issue.
+See also: Specifying memory requirements.
+My compilation/command fails on login node
+When logging in, you are using a connection to the login nodes. There are somewhat strict
+limitations on what you can do in those sessions: check out the output of ulimit -a.
+Specifically, the memory and the amount of processes you can use may present an issue.
+This is common with MATLAB compilation and Nextflow. An error caused by the login session
+limitations can look like this: Aborted (core dumped).
+It's easy to get around these limitations: start an interactive session on one of the clusters.
+Then, you are acting as a node on that cluster instead of a login node. Notably, the
+debug/interactive cluster will grant such a session immediately, while other clusters might make you wait a bit.
+Example command: ml swap cluster/donphan && qsub -I -l nodes=1:ppn=8
+See also: Running interactive jobs.
+My job isn't using any GPUs
+Only two clusters have GPUs. Check out the infrastructure overview,
+to see which one suits your needs. Make sure that you manually switch to the GPU cluster before you submit
+the job. Inside the job script, you need to explicitly request the GPUs:
+#PBS -l nodes=1:ppn=24:gpus=2
+Some software modules don't have GPU support, even when running on the GPU cluster. For example,
+when running module avail alphafold on the joltik cluster, you will find versions on both
+the foss toolchain and the fossCUDA toolchain. Of these, only the CUDA versions will
+use GPU power. When in doubt, CUDA means GPU support.

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3_metadata.json

@@ -0,0 +1,18 @@
+{
+    "main_title": "FAQ",
+    "subtitle": "My-job-isn't-using-any-GPUs",
+    "source_file": "../../mkdocs/docs/HPC/FAQ.md",
+    "title_depth": 3,
+    "directory": "FAQ",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#specifying-memory-requirements",
+        "1": "https://docs.hpc.ugent.be/interactive_debug",
+        "2": "https://docs.hpc.ugent.be/running_interactive_jobs",
+        "3": "https://www.ugent.be/hpc/en/infrastructure"
+    },
+    "parent_title": "",
+    "previous_title": "FAQ_paragraph_2",
+    "next_title": "FAQ_paragraph_4",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/FAQ/#my-job-isnt-using-any-gpus"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4.txt

@@ -0,0 +1,29 @@
+See also: HPC-UGent GPU clusters.
+My job runs slower than I expected
+There are a few possible causes why a job can perform worse than expected.
+Is your job using all the available cores you've requested? You can test this by increasing and
+decreasing the core amount: If the execution time stays the same, the job was not using all cores.
+Some workloads just don't scale well with more cores. If you expect the job to be very parallelizable
+and you encounter this problem, maybe you missed some settings that enable multicore execution.
+See also: How many cores/nodes should i request?
+Does your job have access to the GPUs you requested?
+See also: My job isn't using any GPUs
+Not all file locations perform the same. In particular, the $VSC_HOME and $VSC_DATA
+directories are, relatively, very slow to access. Your jobs should rather use the
+$VSC_SCRATCH directory, or other fast locations (depending on your needs), described
+in Where to store your data on the HPC.
+As an example how to do this: The job can copy the input to the scratch directory, then execute
+the computations, and lastly copy the output back to the data directory.
+Using the home and data directories is especially a problem when UGent isn't your home institution:
+your files may be stored, for example, in Leuven while you're running a job in Ghent.
+My MPI job fails
+Use mympirun in your job script instead of mpirun. It is a tool that makes sure everything
+gets set up correctly for the HPC infrastructure. You need to load it as a module in your
+job script: module load vsc-mympirun.
+To submit the job, use the qsub command rather than sbatch. Although both will submit a job,
+qsub will correctly interpret the #PBS parameters inside the job script. sbatch might not
+set the job environment up correctly for mympirun/OpenMPI.
+See also: Multi core jobs/Parallel Computing
+and Mympirun.
+mympirun seems to ignore its arguments
+For example, we have a simple script (./hello.sh):

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4_metadata.json

@@ -0,0 +1,20 @@
+{
+    "main_title": "FAQ",
+    "subtitle": "`mympirun`-seems-to-ignore-its-arguments",
+    "source_file": "../../mkdocs/docs/HPC/FAQ.md",
+    "title_depth": 3,
+    "directory": "FAQ",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/gpu_gent",
+        "1": "https://docs.hpc.ugent.be/FAQ/#how-many-coresnodes-should-i-request",
+        "2": "https://docs.hpc.ugent.be/FAQ/#my-job-isnt-using-any-gpus",
+        "3": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#where-to-store-your-data-on-the-hpc",
+        "4": "https://docs.hpc.ugent.be/multi_core_jobs",
+        "5": "https://docs.hpc.ugent.be/mympirun"
+    },
+    "parent_title": "",
+    "previous_title": "FAQ_paragraph_3",
+    "next_title": "FAQ_paragraph_5",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/FAQ/#mympirun-seems-to-ignore-its-arguments"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5.txt

@@ -0,0 +1,36 @@
+#!/bin/bash 
+echo "hello world"
+And we run it like mympirun ./hello.sh --output output.txt.
+To our surprise, this doesn't output to the file output.txt, but to
+standard out! This is because mympirun expects the program name and
+the arguments of the program to be its last arguments. Here, the
+--output output.txt arguments are passed to ./hello.sh instead of to
+mympirun. The correct way to run it is:
+mympirun --output output.txt ./hello.sh
+When will my job start?
+See the explanation about how jobs get prioritized in When will my job start. 
+Why do I get a "No space left on device" error, while I still have storage space left?
+When trying to create files, errors like this can occur:
+No space left on device
+The error "No space left on device" can mean two different things:
+- all available storage quota on the file system in question has been used;
+- the inode limit has been reached on that file system.
+An inode can be seen as a "file slot", meaning that when the limit is reached, no more additional files can be created.
+There is a standard inode limit in place that will be increased if needed. 
+The number of inodes used per file system can be checked on the VSC account page.
+Possible solutions to this problem include cleaning up unused files and directories or 
+compressing directories with a lot of files into zip- or tar-files.
+If the problem persists, feel free to contact support.
+Other
+Can I share my account with someone else?
+NO. You are not allowed to share your VSC account with anyone else, it is
+strictly personal. 
+See
+https://helpdesk.ugent.be/account/en/regels.php. 
+If you want to share data, there are alternatives (like a shared directories in VO
+space, see Virtual organisations).
+Can I share my data with other HPC users?
+Yes, you can use the chmod or setfacl commands to change permissions
+of files so other users can access the data. For example, the following
+command will enable a user named "otheruser" to read the file named
+dataset.txt. See

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5_metadata.json

@@ -0,0 +1,19 @@
+{
+    "main_title": "FAQ",
+    "subtitle": "Can-I-share-my-data-with-other-HPC-users",
+    "source_file": "../../mkdocs/docs/HPC/FAQ.md",
+    "title_depth": 3,
+    "directory": "FAQ",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/running_batch_jobs/#when-will-my-job-start",
+        "1": "https://account.vscentrum.be",
+        "2": "https://docs.hpc.ugent.be/linux-tutorial/manipulating_files_and_directories/#zipping-gzipgunzip-zipunzip",
+        "3": "https://docs.hpc.ugent.be/FAQ/#i-have-another-questionproblem",
+        "4": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations"
+    },
+    "parent_title": "",
+    "previous_title": "FAQ_paragraph_4",
+    "next_title": "FAQ_paragraph_6",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/FAQ/#can-i-share-my-data-with-other-hpc-users"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6.txt

@@ -0,0 +1,37 @@
+$ setfacl -m u:otheruser:r dataset.txt
+$ ls -l dataset.txt
+-rwxr-x---+ 2 vsc40000 mygroup      40 Apr 12 15:00 dataset.txt
+For more information about chmod or setfacl, see
+Linux tutorial.
+Can I use multiple different SSH key pairs to connect to my VSC account?
+Yes, and this is recommended when working from different computers.
+Please see Adding multiple SSH public keys on how to do this.
+I want to use software that is not available on the clusters yet
+Please fill out the details about the software and why you need it in
+this form:
+https://www.ugent.be/hpc/en/support/software-installation-request.
+When submitting the form, a mail will be sent to [email protected] containing all the
+provided information. The HPC team will look into your request as soon
+as possible you and contact you when the installation is done or if
+further information is required. 
+Is my connection compromised? Remote host identification has changed
+On Monday 25 April 2022, the login nodes received an update to RHEL8.
+This means that the host keys of those servers also changed. As a result,
+you could encounter the following warnings.
+MacOS & Linux (on Windows, only the second part is shown):
+@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
+@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
+IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
+Someone could be eavesdropping on you right now (man-in-the-middle attack)!
+It is also possible that a host key has just been changed.
+The fingerprint for the RSA key sent by the remote host is
+xx:xx:xx.
+Please contact your system administrator.
+Add correct host key in /home/hostname/.ssh/known_hosts to get rid of this message.
+Offending RSA key in /var/lib/sss/pubconf/known_hosts:1
+RSA host key for user has changed and you have requested strict checking.
+Host key verification failed.
+Please follow the instructions at migration to RHEL8
+to ensure it really is not a hacking attempt \- you will find the correct host key to compare.
+You will also find how to hide the warning.

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6_metadata.json

@@ -0,0 +1,17 @@
+{
+    "main_title": "FAQ",
+    "subtitle": "Is-my-connection-compromised-Remote-host-identification-has-changed",
+    "source_file": "../../mkdocs/docs/HPC/FAQ.md",
+    "title_depth": 3,
+    "directory": "FAQ",
+    "links": {
+        "0": "https://docs.hpc.ugent.be/linux-tutorial/manipulating_files_and_directories/#changing-permissions-chmod",
+        "1": "https://docs.hpc.ugent.be/account/#adding-multiple-ssh-public-keys-optional",
+        "2": "https://www.ugent.be/hpc/en/infrastructure/migration_to_rhel8"
+    },
+    "parent_title": "",
+    "previous_title": "FAQ_paragraph_5",
+    "next_title": "FAQ_paragraph_7",
+    "OS": "generic",
+    "reference_link": "https://docs.hpc.ugent.be/FAQ/#is-my-connection-compromised-remote-host-identification-has-changed"
+}

scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7.txt

@@ -0,0 +1,25 @@
+VO: how does it work?
+A Virtual Organisation consists of a number of members and moderators. A moderator can:
+*   Manage the VO members (but can't access/remove their data on the system).
+*   See how much storage each member has used, and set limits per member.
+*   Request additional storage for the VO.
+One person can only be part of one VO, be it as a member or moderator.
+It's possible to leave a VO and join another one. However, it's not
+recommended to keep switching between VO's (to supervise groups, for example).
+See also: Virtual Organisations.
+My UGent shared drives don't show up
+After mounting the UGent shared drives with kinit [email protected],
+you might not see an entry with your username when listing ls /UGent.
+This is normal: try ls /UGent/your_username or cd /UGent/your_username, and you should be able to access the drives.
+Be sure to use your UGent username and not your VSC username here.
+See also: Your UGent home drive and shares.
+My home directory is (almost) full, and I don't know why
+Your home directory might be full without looking like it due to hidden files.
+Hidden files and subdirectories have a name starting with a dot and do not show up when running ls.
+If you want to check where the storage in your home directory is used, you can make use of the du command to find out what the largest files and subdirectories are:
+du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G'
+The du command returns the size of every file and subdirectory in the $VSC_HOME directory. This output is then piped into an egrep to filter the lines to the ones that matter the most.
+The egrep command will only let entries that match with the specified regular expression [0-9]{3}M|[0-9]G through, which corresponds with files that consume more than 100 MB.
+How can I get more storage space?
+By default you get 3 GB of storage space for your home directory and 25 GB in your personal directories on both the data ($VSC_DATA) and scratch ($VSC_SCRATCH) filesystems.
+It is not possible to expand the storage quota for these personal directories.