diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1.txt new file mode 100644 index 00000000000..9ead695e388 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1.txt @@ -0,0 +1,33 @@ +Frequently Asked Questions (FAQ) +New users should consult the Introduction to HPC +to get started, which is a great resource for learning the basics, troubleshooting, and looking up specifics. +If you want to use software that's not yet installed on the HPC, send us a +software installation request. +Overview of HPC-UGent Tier-2 infrastructure +Composing a job +How many cores/nodes should I request? +An important factor in this question is how well your task is being parallelized: +does it actually run faster with more resources? You can test this yourself: +start with 4 cores, then 8, then 16... The execution time should each time be reduced to +around half of what it was before. You can also try this with full nodes: 1 node, 2 nodes. +A rule of thumb is that you're around the limit when you double the resources but the +execution time is still ~60-70% of what it was before. That's a signal to stop increasing the core count. +See also: Running batch jobs. +Which packages are available? +When connected to the HPC, use the commands module avail [search_text] and module spider [module] +to find installed modules and get information on them. +Among others, many packages for both Python and R are readily available on the HPC. +These aren't always easy to find, though, as we've bundled them together. +Specifically, the module SciPy-bundle includes numpy, pandas, scipy and a few others. +For R, the normal R module has many libraries included. The bundle R-bundle-Bioconductor +contains more libraries. +Use the command module spider [module] to find the specifics on these bundles. +If the package or library you want is not available, send us a +software installation request. +How do I choose the job modules? +Modules each come with a suffix that describes the toolchain used to install them. +Examples: + AlphaFold/2.2.2-foss-2021a* + tqdm/4.61.2-GCCcore-10.3.0* + Python/3.9.5-GCCcore-10.3.0* + matplotlib/3.4.2-foss-2021a* diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1_metadata.json new file mode 100644 index 00000000000..63406afa3fd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_1_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "FAQ", + "subtitle": "How-do-I-choose-the-job-modules", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://www.ugent.be/hpc/en/training/2023/introhpcugent", + "1": "https://www.ugent.be/hpc/en/support/software-installation-request", + "2": "https://www.ugent.be/hpc/en/infrastructure", + "3": "https://docs.hpc.ugent.be/running_batch_jobs", + "4": "https://www.ugent.be/hpc/en/support/software-installation-request" + }, + "parent_title": "", + "previous_title": null, + "next_title": "FAQ_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#how-do-i-choose-the-job-modules" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2.txt new file mode 100644 index 00000000000..0218d9926f1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2.txt @@ -0,0 +1,34 @@ +Modules from the same toolchain always work together, and modules from a +\*different version of the same toolchain\* never work together. +The above set of modules works together: an overview of compatible toolchains can be found here: +https://docs.easybuild.io/en/latest/Common-toolchains.html#overview-of-common-toolchains. +You can use module avail [search_text] to see which versions on which toolchains are available to use. +If you need something that's not available yet, you can request it through a +software installation request. +It is possible to use the modules without specifying a version or toolchain. However, +this will probably cause incompatible modules to be loaded. Don't do it if you use multiple modules. +Even if it works now, as more modules get installed on the HPC, your job can suddenly break. +Troubleshooting +My modules don't work together +When incompatible modules are loaded, you might encounter an error like this: +Lmod has detected the following error: A different version of the 'GCC' module +is already loaded (see output of 'ml'). +You should load another foss module for that is compatible with the currently +loaded version of GCC. +Use ml spider foss to get an overview of the available versions. +Modules from the same toolchain always work together, and modules from a +_different version of the same toolchain_ never work together. +An overview of compatible toolchains can be found here: +https://docs.easybuild.io/en/latest/Common-toolchains.html#overview-of-common-toolchains. +See also: How do I choose the job modules? +My job takes longer than 72 hours +The 72 hour walltime limit will not be extended. However, you can work around this barrier: +* Check that all available resources are being used. See also: + * How many cores/nodes should I request?. + * My job is slow. + * My job isn't using any GPUs. +* Use a faster cluster. +* Divide the job into more parallel processes. +* Divide the job into shorter processes, which you can submit as separate jobs. +* Use the built-in checkpointing of your software. +Job failed: SEGV Segmentation fault diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2_metadata.json new file mode 100644 index 00000000000..747a0de47bf --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_2_metadata.json @@ -0,0 +1,20 @@ +{ + "main_title": "FAQ", + "subtitle": "Job-failed-SEGV-Segmentation-fault", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://www.ugent.be/hpc/en/support/software-installation-request", + "1": "https://docs.hpc.ugent.be/FAQ/#how-do-i-choose-the-job-modules", + "2": "https://docs.hpc.ugent.be/FAQ/#how-many-coresnodes-should-i-request", + "3": "https://docs.hpc.ugent.be/FAQ/#my-job-runs-slower-than-i-expected", + "4": "https://docs.hpc.ugent.be/FAQ/#my-job-isnt-using-any-gpus", + "5": "https://www.ugent.be/hpc/en/infrastructure" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_1", + "next_title": "FAQ_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#job-failed-segv-segmentation-fault" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3.txt new file mode 100644 index 00000000000..9db33be16c5 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3.txt @@ -0,0 +1,28 @@ +Any error mentioning SEGV or Segmentation fault/violation has something to do with a memory error. +If you weren't messing around with memory-unsafe applications or programming, your job probably hit its memory limit. +When there's no memory amount specified in a job script, your job will get access to a proportional +share of the total memory on the node: If you request a full node, all memory will be available. +If you request 8 cores on a cluster where nodes have 2x18 cores, you will get 8/36 = 2/9 +of the total memory on the node. +Try requesting a bit more memory than your proportional share, and see if that solves the issue. +See also: Specifying memory requirements. +My compilation/command fails on login node +When logging in, you are using a connection to the login nodes. There are somewhat strict +limitations on what you can do in those sessions: check out the output of ulimit -a. +Specifically, the memory and the amount of processes you can use may present an issue. +This is common with MATLAB compilation and Nextflow. An error caused by the login session +limitations can look like this: Aborted (core dumped). +It's easy to get around these limitations: start an interactive session on one of the clusters. +Then, you are acting as a node on that cluster instead of a login node. Notably, the +debug/interactive cluster will grant such a session immediately, while other clusters might make you wait a bit. +Example command: ml swap cluster/donphan && qsub -I -l nodes=1:ppn=8 +See also: Running interactive jobs. +My job isn't using any GPUs +Only two clusters have GPUs. Check out the infrastructure overview, +to see which one suits your needs. Make sure that you manually switch to the GPU cluster before you submit +the job. Inside the job script, you need to explicitly request the GPUs: +#PBS -l nodes=1:ppn=24:gpus=2 +Some software modules don't have GPU support, even when running on the GPU cluster. For example, +when running module avail alphafold on the joltik cluster, you will find versions on both +the foss toolchain and the fossCUDA toolchain. Of these, only the CUDA versions will +use GPU power. When in doubt, CUDA means GPU support. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3_metadata.json new file mode 100644 index 00000000000..5a55993674a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_3_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "FAQ", + "subtitle": "My-job-isn't-using-any-GPUs", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#specifying-memory-requirements", + "1": "https://docs.hpc.ugent.be/interactive_debug", + "2": "https://docs.hpc.ugent.be/running_interactive_jobs", + "3": "https://www.ugent.be/hpc/en/infrastructure" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_2", + "next_title": "FAQ_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#my-job-isnt-using-any-gpus" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4.txt new file mode 100644 index 00000000000..93694817d0d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4.txt @@ -0,0 +1,29 @@ +See also: HPC-UGent GPU clusters. +My job runs slower than I expected +There are a few possible causes why a job can perform worse than expected. +Is your job using all the available cores you've requested? You can test this by increasing and +decreasing the core amount: If the execution time stays the same, the job was not using all cores. +Some workloads just don't scale well with more cores. If you expect the job to be very parallelizable +and you encounter this problem, maybe you missed some settings that enable multicore execution. +See also: How many cores/nodes should i request? +Does your job have access to the GPUs you requested? +See also: My job isn't using any GPUs +Not all file locations perform the same. In particular, the $VSC_HOME and $VSC_DATA +directories are, relatively, very slow to access. Your jobs should rather use the +$VSC_SCRATCH directory, or other fast locations (depending on your needs), described +in Where to store your data on the HPC. +As an example how to do this: The job can copy the input to the scratch directory, then execute +the computations, and lastly copy the output back to the data directory. +Using the home and data directories is especially a problem when UGent isn't your home institution: +your files may be stored, for example, in Leuven while you're running a job in Ghent. +My MPI job fails +Use mympirun in your job script instead of mpirun. It is a tool that makes sure everything +gets set up correctly for the HPC infrastructure. You need to load it as a module in your +job script: module load vsc-mympirun. +To submit the job, use the qsub command rather than sbatch. Although both will submit a job, +qsub will correctly interpret the #PBS parameters inside the job script. sbatch might not +set the job environment up correctly for mympirun/OpenMPI. +See also: Multi core jobs/Parallel Computing +and Mympirun. +mympirun seems to ignore its arguments +For example, we have a simple script (./hello.sh): diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4_metadata.json new file mode 100644 index 00000000000..609a3ca303f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_4_metadata.json @@ -0,0 +1,20 @@ +{ + "main_title": "FAQ", + "subtitle": "`mympirun`-seems-to-ignore-its-arguments", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://docs.hpc.ugent.be/gpu_gent", + "1": "https://docs.hpc.ugent.be/FAQ/#how-many-coresnodes-should-i-request", + "2": "https://docs.hpc.ugent.be/FAQ/#my-job-isnt-using-any-gpus", + "3": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#where-to-store-your-data-on-the-hpc", + "4": "https://docs.hpc.ugent.be/multi_core_jobs", + "5": "https://docs.hpc.ugent.be/mympirun" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_3", + "next_title": "FAQ_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#mympirun-seems-to-ignore-its-arguments" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5.txt new file mode 100644 index 00000000000..ac6b8aa113a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5.txt @@ -0,0 +1,36 @@ +#!/bin/bash +echo "hello world" +And we run it like mympirun ./hello.sh --output output.txt. +To our surprise, this doesn't output to the file output.txt, but to +standard out! This is because mympirun expects the program name and +the arguments of the program to be its last arguments. Here, the +--output output.txt arguments are passed to ./hello.sh instead of to +mympirun. The correct way to run it is: +mympirun --output output.txt ./hello.sh +When will my job start? +See the explanation about how jobs get prioritized in When will my job start. +Why do I get a "No space left on device" error, while I still have storage space left? +When trying to create files, errors like this can occur: +No space left on device +The error "No space left on device" can mean two different things: +- all available storage quota on the file system in question has been used; +- the inode limit has been reached on that file system. +An inode can be seen as a "file slot", meaning that when the limit is reached, no more additional files can be created. +There is a standard inode limit in place that will be increased if needed. +The number of inodes used per file system can be checked on the VSC account page. +Possible solutions to this problem include cleaning up unused files and directories or +compressing directories with a lot of files into zip- or tar-files. +If the problem persists, feel free to contact support. +Other +Can I share my account with someone else? +NO. You are not allowed to share your VSC account with anyone else, it is +strictly personal. +See +https://helpdesk.ugent.be/account/en/regels.php. +If you want to share data, there are alternatives (like a shared directories in VO +space, see Virtual organisations). +Can I share my data with other HPC users? +Yes, you can use the chmod or setfacl commands to change permissions +of files so other users can access the data. For example, the following +command will enable a user named "otheruser" to read the file named +dataset.txt. See diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5_metadata.json new file mode 100644 index 00000000000..3235cd89ff4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_5_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "FAQ", + "subtitle": "Can-I-share-my-data-with-other-HPC-users", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://docs.hpc.ugent.be/running_batch_jobs/#when-will-my-job-start", + "1": "https://account.vscentrum.be", + "2": "https://docs.hpc.ugent.be/linux-tutorial/manipulating_files_and_directories/#zipping-gzipgunzip-zipunzip", + "3": "https://docs.hpc.ugent.be/FAQ/#i-have-another-questionproblem", + "4": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_4", + "next_title": "FAQ_paragraph_6", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#can-i-share-my-data-with-other-hpc-users" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6.txt new file mode 100644 index 00000000000..011ad0ce463 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6.txt @@ -0,0 +1,37 @@ +$ setfacl -m u:otheruser:r dataset.txt +$ ls -l dataset.txt +-rwxr-x---+ 2 vsc40000 mygroup 40 Apr 12 15:00 dataset.txt +For more information about chmod or setfacl, see +Linux tutorial. +Can I use multiple different SSH key pairs to connect to my VSC account? +Yes, and this is recommended when working from different computers. +Please see Adding multiple SSH public keys on how to do this. +I want to use software that is not available on the clusters yet +Please fill out the details about the software and why you need it in +this form: +https://www.ugent.be/hpc/en/support/software-installation-request. +When submitting the form, a mail will be sent to hpc@ugent.be containing all the +provided information. The HPC team will look into your request as soon +as possible you and contact you when the installation is done or if +further information is required. +Is my connection compromised? Remote host identification has changed +On Monday 25 April 2022, the login nodes received an update to RHEL8. +This means that the host keys of those servers also changed. As a result, +you could encounter the following warnings. +MacOS & Linux (on Windows, only the second part is shown): +@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ +@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @ +@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ +IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY! +Someone could be eavesdropping on you right now (man-in-the-middle attack)! +It is also possible that a host key has just been changed. +The fingerprint for the RSA key sent by the remote host is +xx:xx:xx. +Please contact your system administrator. +Add correct host key in /home/hostname/.ssh/known_hosts to get rid of this message. +Offending RSA key in /var/lib/sss/pubconf/known_hosts:1 +RSA host key for user has changed and you have requested strict checking. +Host key verification failed. +Please follow the instructions at migration to RHEL8 +to ensure it really is not a hacking attempt \- you will find the correct host key to compare. +You will also find how to hide the warning. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6_metadata.json new file mode 100644 index 00000000000..6b646e83475 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_6_metadata.json @@ -0,0 +1,17 @@ +{ + "main_title": "FAQ", + "subtitle": "Is-my-connection-compromised-Remote-host-identification-has-changed", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial/manipulating_files_and_directories/#changing-permissions-chmod", + "1": "https://docs.hpc.ugent.be/account/#adding-multiple-ssh-public-keys-optional", + "2": "https://www.ugent.be/hpc/en/infrastructure/migration_to_rhel8" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_5", + "next_title": "FAQ_paragraph_7", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#is-my-connection-compromised-remote-host-identification-has-changed" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7.txt new file mode 100644 index 00000000000..8228e874f4b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7.txt @@ -0,0 +1,25 @@ +VO: how does it work? +A Virtual Organisation consists of a number of members and moderators. A moderator can: +* Manage the VO members (but can't access/remove their data on the system). +* See how much storage each member has used, and set limits per member. +* Request additional storage for the VO. +One person can only be part of one VO, be it as a member or moderator. +It's possible to leave a VO and join another one. However, it's not +recommended to keep switching between VO's (to supervise groups, for example). +See also: Virtual Organisations. +My UGent shared drives don't show up +After mounting the UGent shared drives with kinit your_email@ugent.be, +you might not see an entry with your username when listing ls /UGent. +This is normal: try ls /UGent/your_username or cd /UGent/your_username, and you should be able to access the drives. +Be sure to use your UGent username and not your VSC username here. +See also: Your UGent home drive and shares. +My home directory is (almost) full, and I don't know why +Your home directory might be full without looking like it due to hidden files. +Hidden files and subdirectories have a name starting with a dot and do not show up when running ls. +If you want to check where the storage in your home directory is used, you can make use of the du command to find out what the largest files and subdirectories are: +du -h --max-depth 1 $VSC_HOME | egrep '[0-9]{3}M|[0-9]G' +The du command returns the size of every file and subdirectory in the $VSC_HOME directory. This output is then piped into an egrep to filter the lines to the ones that matter the most. +The egrep command will only let entries that match with the specified regular expression [0-9]{3}M|[0-9]G through, which corresponds with files that consume more than 100 MB. +How can I get more storage space? +By default you get 3 GB of storage space for your home directory and 25 GB in your personal directories on both the data ($VSC_DATA) and scratch ($VSC_SCRATCH) filesystems. +It is not possible to expand the storage quota for these personal directories. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7_metadata.json new file mode 100644 index 00000000000..e3f76810c3d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_7_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "FAQ", + "subtitle": "How-can-I-get-more-storage-space", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations", + "1": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#your-ugent-home-drive-and-shares", + "2": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#check-your-quota", + "3": "https://docs.hpc.ugent.be/linux-tutorial/beyond_the_basics/#searching-file-contents-grep", + "4": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_6", + "next_title": "FAQ_paragraph_8", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#how-can-i-get-more-storage-space" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_8.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_8.txt new file mode 100644 index 00000000000..7819acaf2e6 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_8.txt @@ -0,0 +1,20 @@ +You can get more storage space through a Virtual Organisation (VO), +which will give you access to the additional directories in a subdirectory specific to that VO ($VSC_DATA_VO and $VSC_SCRATCH_VO). +The moderators of a VO can request more storage for their VO. +Why can't I use the sudo command? +When you attempt to use sudo, you will be prompted for a password. +However, you cannot enter a valid password because this feature is reserved exclusively for HPC administrators. +sudo is used to execute a command with administrator rights, which would allow you to make system-wide changes. +You are only able to run commands that make changes to the directories that your VSC account has access to, +like your home directory, your personal directories like $VSC_DATA and $VSC_SCRATCH, +or shared VO/group directories like $VSC_DATA_VO and $VSC_SCRATCH_VO. +A lot of tasks can be performed without sudo, including installing software in your own account. +Installing software +- If you know how to install the software without using sudo, you are welcome to proceed with the installation. +- If you are unsure how to install the software, you can submit a software installation request, and the HPC-UGent support team will handle the installation for you. +I have another question/problem +Who can I contact? +* General questions regarding HPC-UGent and VSC: +* HPC-UGent Tier-2: +* VSC Tier-1 compute: +* VSC Tier-1 cloud: \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_8_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_8_metadata.json new file mode 100644 index 00000000000..f290413808b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/FAQ/FAQ_paragraph_8_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "FAQ", + "subtitle": "I-have-another-questionproblem", + "source_file": "../../mkdocs/docs/HPC/FAQ.md", + "title_depth": 3, + "directory": "FAQ", + "links": { + "0": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#virtual-organisations", + "1": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#vo-directories", + "2": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#requesting-more-storage-space", + "3": "https://www.ugent.be/hpc/en/support/software-installation-request" + }, + "parent_title": "", + "previous_title": "FAQ_paragraph_7", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/FAQ/#i-have-another-questionproblem" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_1.txt new file mode 100644 index 00000000000..5cb90b96d22 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_1.txt @@ -0,0 +1,43 @@ +Hanythingondemand (HOD) +Hanythingondemand (or HOD for short) is a tool to run a Hadoop (Yarn) +cluster on a traditional HPC system. +Documentation +The official documentation for HOD version 3.0.0 and newer is available +at https://hod.readthedocs.org/en/latest/. The slides of the 2016 HOD +training session are available at +http://users.ugent.be/~kehoste/hod_20161024.pdf. +Using HOD +Before using HOD, you first need to load the hod module. We don't +specify a version here (this is an exception, for most other modules you +should, see Using explicit version numbers) because newer versions might include important bug fixes. +module load hod +Compatibility with login nodes +The hod modules are constructed such that they can be used on the HPC-UGent infrastructure +login nodes, regardless of which cluster module is loaded (this is not +the case for software installed via modules in general, see Running software that is incompatible with host). +As such, you should experience no problems if you swap to a different +cluster module before loading the hod module and subsequently running +|hod|. +For example, this will work as expected: +$ module swap cluster/donphan +$ module load hod +$ hod +hanythingondemand - Run services within an HPC cluster +usage: hod [subcommand options] +Available subcommands (one of these must be specified!): + batch Submit a job to spawn a cluster on a PBS job controller, run a job script, and tear down the cluster when it's done + clean Remove stale cluster info. +... +Note that also modules named hanythingondemand/* are available. These +should however not be used directly, since they may not be compatible +with the login nodes (depending on which cluster they were installed +for). +Standard HOD configuration +The hod module will also put a basic configuration in place for HOD, +by defining a couple of $HOD_* environment variables: +$ module load hod +$ env | grep HOD | sort +HOD_BATCH_HOD_MODULE=hanythingondemand/3.2.2-intel-2016b-Python-2.7.12 +HOD_BATCH_WORKDIR=$VSC_SCRATCH/hod +HOD_CREATE_HOD_MODULE=hanythingondemand/3.2.2-intel-2016b-Python-2.7.12 +HOD_CREATE_WORKDIR=$VSC_SCRATCH/hod diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_1_metadata.json new file mode 100644 index 00000000000..e451d3afb80 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_1_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "HOD", + "subtitle": "Standard-HOD-configuration", + "source_file": "../../mkdocs/docs/HPC/HOD.md", + "title_depth": 3, + "directory": "HOD", + "links": { + "0": "https://docs.hpc.ugent.be/running_batch_jobs/#using-explicit-version-numbers", + "1": "https://docs.hpc.ugent.be/troubleshooting/#running-software-that-is-incompatible-with-host" + }, + "parent_title": "", + "previous_title": null, + "next_title": "HOD_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/HOD/#standard-hod-configuration" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_2.txt new file mode 100644 index 00000000000..45df3717871 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_2.txt @@ -0,0 +1,34 @@ +By defining these environment variables, we avoid that you have to +specify --hod-module and --workdir when using hod batch or +hod create, since they are strictly required. +If you want to use a different parent working directory for HOD, it +suffices to either redefine $HOD_BATCH_WORKDIR and +$HOD_CREATE_WORKDIR, or to specify --workdir (which will override +the corresponding environment variable). +Changing the HOD module that is used by the HOD backend (i.e., using +--hod-module or redefining $HOD_*_HOD_MODULE) is strongly +discouraged. +Cleaning up +After HOD clusters terminate, their local working directory and cluster +information is typically not cleaned up automatically (for example, +because the job hosting an interactive HOD cluster submitted via +hod create runs out of walltime). +These HOD clusters will still show up in the output of hod list, and +will be marked as . +You should occasionally clean this up using hod clean: +$ module list +Currently Loaded Modulefiles: + 1) cluster/doduo(default) 2) pbs_python/4.6.0 3) vsc-base/2.4.2 4) hod/3.0.0-cli +$ hod list +Cluster label Job ID State Hosts +example1 123456 <job-not-found> <none> +$ hod clean +Removed cluster localworkdir directory /user/scratch/gent/vsc400/vsc40000/hod/hod/123456 for cluster labeled example1 +Removed cluster info directory /user/home/gent/vsc400/vsc40000/.config/hod.d/wordcount for cluster labeled example1 +$ module swap cluster/donphan +Cluster label Job ID State Hosts +example2 98765.master19.donphan.gent.vsc <job-not-found> <none> +$ hod clean +Removed cluster localworkdir directory /user/scratch/gent/vsc400/vsc40000/hod/hod/98765.master19.donphan.gent.vsc for cluster labeled example2 +Removed cluster info directory /user/home/gent/vsc400/vsc40000/.config/hod.d/wordcount for cluster labeled example2 +Note that only HOD clusters that were submitted to the currently loaded cluster module will be cleaned up. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_2_metadata.json new file mode 100644 index 00000000000..480afb3ceb6 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "HOD", + "subtitle": "Cleaning-up", + "source_file": "../../mkdocs/docs/HPC/HOD.md", + "title_depth": 3, + "directory": "HOD", + "parent_title": "", + "previous_title": "HOD_paragraph_1", + "next_title": "HOD_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/HOD/#cleaning-up" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_3.txt new file mode 100644 index 00000000000..d95838f55e2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_3.txt @@ -0,0 +1,9 @@ +Getting help +If you have any questions, or are experiencing problems using HOD, you +have a couple of options: +- Subscribe to the HOD mailing list via + https://lists.ugent.be/wws/info/hod, and contact the HOD users and + developers at hod@lists.ugent.be. +- Contact the HPC-UGent team via hpc@ugent.be +- Open an issue in the hanythingondemand GitHub repository, via + https://github.com/hpcugent/hanythingondemand/issues. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_3_metadata.json new file mode 100644 index 00000000000..1486348f570 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/HOD/HOD_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "HOD", + "subtitle": "Getting-help", + "source_file": "../../mkdocs/docs/HPC/HOD.md", + "title_depth": 2, + "directory": "HOD", + "parent_title": "", + "previous_title": "HOD_paragraph_2", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/HOD/#getting-help" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_1.txt new file mode 100644 index 00000000000..d7f8bf11a93 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_1.txt @@ -0,0 +1,36 @@ +Getting an HPC Account +Getting ready to request an account +All users of AUGent can request +an +account on the HPC, which is part of the Flemish Supercomputing Centre (VSC). +See HPC policies for more information on who is entitled to an account. +The VSC, abbreviation of Flemish Supercomputer Centre, is a virtual +supercomputer centre. It is a partnership between the five Flemish +associations: the Association KU Leuven, Ghent University Association, +Brussels University Association, Antwerp University Association and the +University Colleges-Limburg. The VSC is funded by the Flemish +Government. +There are two methods for connecting to HPC-UGent infrastructure: +- Using a terminal to connect via SSH. +- Using the web portal +The web portal offers a convenient way to upload files and gain shell access to the HPC-UGent infrastructure from a standard web browser (no software installation or configuration required). +If you would like use a terminal with SSH as this gives you more flexibility continue reading. +However if you prefer to use the web portal, you can skip ahead to the following section: Applying for the account. +Once you have successfully obtained an account, you can then delve into the details of utilizing the HPC-UGent web portal by reading Using the HPC-UGent web portal. +The HPC-UGent infrastructure clusters use public/private key pairs for user authentication +(rather than passwords). Technically, the private key is stored on your +local computer and always stays there; the public key is stored on the HPC. +Access to the HPC is granted to anyone who can prove to have access to the +corresponding private key on his local computer. +How do SSH keys work? +- an SSH public/private key pair can be seen as a lock and a key +- the SSH public key is equivalent with a lock: you give it to the + VSC and they put it on the door that gives access to your account. +- the SSH private key is like a physical key: you don't hand it out + to other people. +- anyone who has the key (and the optional password) can unlock the + door and log in to the account. +- the door to your VSC account is special: it can have multiple + locks (SSH public keys) attached to it, and you only need to open + one lock with the corresponding key (SSH private key) to open + the door (log in to the account). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_1_metadata.json new file mode 100644 index 00000000000..812fbbba4c2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/account/account_paragraph_1_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "account", + "subtitle": "How-do-SSH-keys-work", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "links": { + "0": "https://docs.hpc.ugent.be/sites/hpc_policies", + "1": "https://docs.hpc.ugent.be/web_portal", + "2": "https://docs.hpc.ugent.be/account/#applying-for-the-account", + "3": "https://docs.hpc.ugent.be/web_portal" + }, + "parent_title": "", + "previous_title": null, + "next_title": "account_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/account/#how-do-ssh-keys-work" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_1.txt new file mode 100644 index 00000000000..ec6f7d85f90 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_1.txt @@ -0,0 +1,32 @@ +What is AlphaFold? +AlphaFold is an AI system developed by DeepMind that predicts a protein’s 3D structure from its amino acid sequence. +It aims to achieve accuracy competitive with experimental methods. +See https://www.vscentrum.be/alphafold for more information and there you can also find a getting started video recording if you prefer that. +Documentation & extra material +This chapter focuses specifically on the use of AlphaFold on the HPC-UGent infrastructure. +It is intented to augment the existing AlphaFold documentation rather than replace it. +It is therefore recommended to first familiarize yourself with AlphaFold. The following resources can be helpful: +- AlphaFold website: https://alphafold.com/ +- AlphaFold repository: https://github.com/deepmind/alphafold/tree/main +- AlphaFold FAQ: https://alphafold.com/faq +- VSC webpage about AlphaFold: https://www.vscentrum.be/alphafold +- Introductory course on AlphaFold by VIB: https://elearning.vib.be/courses/alphafold +- "Getting Started with AlphaFold" presentation by Kenneth Hoste (HPC-UGent) + - recording available on YouTube + - slides available here (PDF) + - see also https://www.vscentrum.be/alphafold +Using AlphaFold on HPC-UGent infrastructure +Several different versions of AlphaFold are installed on both the CPU and GPU HPC-UGent Tier-2 clusters, see the output of module avail AlphaFold. +If you run this command on a GPU cluster, additional CUDA modules will show up: +$ module avail AlphaFold +------------ /apps/gent/RHEL8/cascadelake-volta-ib/modules/all ------------- + AlphaFold/2.0.0-fosscuda-2020b + AlphaFold/2.1.1-fosscuda-2020b + AlphaFold/2.1.2-foss-2021a-CUDA-11.3.1 + AlphaFold/2.2.2-foss-2021a-CUDA-11.3.1 + AlphaFold/2.3.0-foss-2021b-CUDA-11.4.1 + AlphaFold/2.3.1-foss-2022a-CUDA-11.7.0 +--------------- /apps/gent/RHEL8/cascadelake-ib/modules/all ---------------- + AlphaFold/2.0.0-foss-2020b AlphaFold/2.3.1-foss-2022a + AlphaFold/2.1.2-foss-2021a AlphaFold/2.3.4-foss-2022a-ColabFold (D) + AlphaFold/2.2.2-foss-2021a diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_1_metadata.json new file mode 100644 index 00000000000..c407aed2e0f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_1_metadata.json @@ -0,0 +1,17 @@ +{ + "main_title": "alphafold", + "subtitle": "Using-AlphaFold-on-HPC-UGent-infrastructure", + "source_file": "../../mkdocs/docs/HPC/alphafold.md", + "title_depth": 2, + "directory": "alphafold", + "links": { + "0": "https://www.youtube.com/watch?v=jP9Qg1yBGcs", + "1": "https://www.vscentrum.be/_files/ugd/5446c2_f19a8723f7f7460ebe990c28a53e56a2.pdf?index=true", + "2": "https://docs.hpc.ugent.be/gpu" + }, + "parent_title": "", + "previous_title": null, + "next_title": "alphafold_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/alphafold/#using-alphafold-on-hpc-ugent-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_2.txt new file mode 100644 index 00000000000..dc47707fe27 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_2.txt @@ -0,0 +1,32 @@ +To use AlphaFold, you should load a particular module, for example: +module load AlphaFold/2.3.1-foss-2022a-CUDA-11.7.0 + Tip "We strongly advise loading a specific version of an AlphaFold module, so you know exactly which version is being used." + Warning + + When using AlphaFold, you should submit jobs to a GPU cluster for better performance, see GPU clusters. + Later in this chapter, you will find a comparison between running AlphaFold on CPUs or GPUs. +Multiple revisions of the large database (~2.5TB) that is also required to run AlphaFold have been +made available on the HPC-UGent infrastructure in a central location (/arcanine/scratch/gent/apps/AlphaFold), +so you do not have to download it yourself. +$ ls /arcanine/scratch/gent/apps/AlphaFold +20210812 20211201 20220701 20230310 +The directories located there indicate when the data was downloaded, so that this leaves room for providing updated datasets later. +As of writing this documentation the latest version is 20230310. + Info + + The arcanine scratch shared filesystem is powered by fast SSD disks, + which is recommended for the AlphaFold data, because of random access I/O patterns. + See Pre-defined user directories to get more info about the arcanine filesystem. +The AlphaFold installations we provide have been modified a bit to facilitate the usage on HPC-UGent infrastructure. +Setting up the environment +The location to the AlphaFold data can be specified via the $ALPHAFOLD_DATA_DIR environment variable, so you should define this variable in your AlphaFold job script: +export ALPHAFOLD_DATA_DIR=/arcanine/scratch/gent/apps/AlphaFold/20230310 + Warning "Use newest version" + + Do not forget to replace 20230310 with a more up to date version if available. +Running AlphaFold +AlphaFold provides a script called run_alphafold.py +A symbolic link named alphafold that points to this script is included, +so you can just use alphafold instead of run_alphafold.py or python run_alphafold.py after loading the AlphaFold module. +The run_alphafold.py script has also been slightly modified such that defining the $ALPHAFOLD_DATA_DIR (see above) is sufficient to pick up all the data provided in that location, +so you don't need to use options like --data_dir to specify the location of the data. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_2_metadata.json new file mode 100644 index 00000000000..a134bec40f4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_2_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "alphafold", + "subtitle": "Running-AlphaFold", + "source_file": "../../mkdocs/docs/HPC/alphafold.md", + "title_depth": 3, + "directory": "alphafold", + "links": { + "0": "https://docs.hpc.ugent.be/gpu", + "1": "http://localhost:8000/HPC/Gent/running_jobs_with_input_output_data/#pre-defined-user-directories", + "2": "https://raw.githubusercontent.com/deepmind/alphafold/main/run_alphafold.py", + "3": "https://docs.hpc.ugent.be/alphafold/#setting-up-the-environment" + }, + "parent_title": "", + "previous_title": "alphafold_paragraph_1", + "next_title": "alphafold_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/alphafold/#running-alphafold" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_3.txt new file mode 100644 index 00000000000..8347a4e27fd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_3.txt @@ -0,0 +1,26 @@ +Similarly, the script was also tweaked such that the location to commands like hhblits,hhsearch,jackhmmer,kalign are already correctly set, so options like --hhblits_binary_path are not required. +For more information about the script and options see this section in the official README. + WARNING "READ README" + It is strongly advised to read the official README provided by DeepMind before continuing. + +Controlling core count for hhblits and jackhmmer +The Python scripts that are used to run hhblits and jackhmmer have been tweaked so you can control how many cores are used for these tools, +rather than hardcoding it to 4 and 8 cores, respectively. +Using the $ALPHAFOLD_HHBLITS_N_CPU environment variable, you can specify how many cores should be used for running hhblits; +the default of 4 cores will be used if $ALPHAFOLD_HHBLITS_N_CPU is not defined. +Likewise for jackhmmer, the core count can be controlled via $ALPHAFOLD_JACKHMMER_N_CPU. + Info + Tweaking this might not yield significant benefits, + as we have noticed that these tools may exhibit slower performance when utilizing more than 4/8 cores (though this behavior could vary based on the workload). +CPU/GPU comparison +The provided timings were obtained by executing the T1050.fasta example, as outlined in the Alphafold README. +For the corresponding jobscripts, they are available here. +Using --db_preset=full_dbs, the following runtime data was collected: +* CPU-only, on doduo, using 24 cores (1 node): 9h 9min +* CPU-only, on doduo, using 96 cores (1 full node): 12h 22min +* GPU on joltik, using 1 V100 GPU + 8 cores: 2h 20min +* GPU on joltik, using 2 V100 GPUs + 16 cores: 2h 16min +This highlights a couple of important attention points: +* Running AlphaFold on GPU is significantly faster than CPU-only (close to 4x faster for this particular example). + Using more CPU cores may lead to longer* runtimes, so be careful with using full nodes when running AlphaFold CPU-only. +* Using multiple GPUs results in barely any speedup (for this particular T1050.fasta example). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_3_metadata.json new file mode 100644 index 00000000000..2a0105633e7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_3_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "alphafold", + "subtitle": "CPUGPU-comparison", + "source_file": "../../mkdocs/docs/HPC/alphafold.md", + "title_depth": 3, + "directory": "alphafold", + "links": { + "0": "https://github.com/deepmind/alphafold/blob/main/README.md#running-alphafold", + "1": "https://github.com/deepmind/alphafold/blob/main/README.md", + "2": "https://github.com/deepmind/alphafold/blob/main/README.md", + "3": "https://github.com/deepmind/alphafold/blob/main/README.md", + "4": "https://docs.hpc.ugent.be/example-jobscripts" + }, + "parent_title": "", + "previous_title": "alphafold_paragraph_2", + "next_title": "alphafold_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/alphafold/#cpugpu-comparison" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_4.txt new file mode 100644 index 00000000000..521975f2e2a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_4.txt @@ -0,0 +1,33 @@ +With --db_preset=casp14, it is clearly more demanding: +* On doduo, with 24 cores (1 node): still running after 48h... +* On joltik, 1 V100 GPU + 8 cores: 4h 48min +This highlights the difference between CPU and GPU performance even more. +Example scenario +The following example comes from the official Examples section in the Alphafold README. +The run command is slightly different (see above: Running AlphaFold). +Do not forget to set up the environment (see above: Setting up the environment). +Folding a monomer +Say we have a monomer with the sequence . +Create a file monomer.fasta with the following content: +>sequence_name + +Then run the following command in the same directory: +alphafold + --fasta_paths=monomer.fasta \ + --max_template_date=2021-11-01 \ + --model_preset=monomer \ + --output_dir=. +See AlphaFold output, for information about the outputs. + Info + + For more scenarios see the example section in the official README. + +Example jobscripts +The following two example job scripts can be used as a starting point for running AlphaFold. +The main difference between using a GPU or CPU in a job script is what module to load. +For running AlphaFold on GPU, use an AlphaFold module that mentions CUDA (or cuda), +for example AlphaFold/2.3.1-foss-2022a-CUDA-11.7.0. +To run the job scripts you need to create a file named T1050.fasta with the following content: +>T1050 A7LXT1, Bacteroides Ovatus, 779 residues| +MASQSYLFKHLEVSDGLSNNSVNTIYKDRDGFMWFGTTTGLNRYDGYTFKIYQHAENEPGSLPDNYITDIVEMPDGRFWINTARGYVLFDKERDYFITDVTGFMKNLESWGVPEQVFVDREGNTWLSVAGEGCYRYKEGGKRLFFSYTEHSLPEYGVTQMAECSDGILLIYNTGLLVCLDRATLAIKWQSDEIKKYIPGGKTIELSLFVDRDNCIWAYSLMGIWAYDCGTKSWRTDLTGIWSSRPDVIIHAVAQDIEGRIWVGKDYDGIDVLEKETGKVTSLVAHDDNGRSLPHNTIYDLYADRDGVMWVGTYKKGVSYYSESIFKFNMYEWGDITCIEQADEDRLWLGTNDHGILLWNRSTGKAEPFWRDAEGQLPNPVVSMLKSKDGKLWVGTFNGGLYCMNGSQVRSYKEGTGNALASNNVWALVEDDKGRIWIASLGGGLQCLEPLSGTFETYTSNNSALLENNVTSLCWVDDNTLFFGTASQGVGTMDMRTREIKKIQGQSDSMKLSNDAVNHVYKDSRGLVWIATREGLNVYDTRRHMFLDLFPVVEAKGNFIAAITEDQERNMWVSTSRKVIRVTVASDGKGSYLFDSRAYNSEDGLQNCDFNQRSIKTLHNGIIAIGGLYGVNIFAPDHIRYNKMLPNVMFTGLSLFDEAVKVGQSYGGRVLIEKELNDVENVEFDYKQNIFSVSFASDNYNLPEKTQYMYKLEGFNNDWLTLPVGVHNVTFTNLAPGKYVLRVKAINSDGYVGIKEATLGIVVNPPFKLAAALQHHHHHH +source: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_4_metadata.json new file mode 100644 index 00000000000..e5ddf8f1e01 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_4_metadata.json @@ -0,0 +1,21 @@ +{ + "main_title": "alphafold", + "subtitle": "Example-jobscripts", + "source_file": "../../mkdocs/docs/HPC/alphafold.md", + "title_depth": 2, + "directory": "alphafold", + "links": { + "0": "https://github.com/deepmind/alphafold/blob/main/README.md#examples", + "1": "https://github.com/deepmind/alphafold/blob/main/README.md", + "2": "https://docs.hpc.ugent.be/running-alphafold", + "3": "https://docs.hpc.ugent.be/setting-up-the-environment", + "4": "https://github.com/deepmind/alphafold/blob/main/README.md#alphafold-output", + "5": "https://github.com/deepmind/alphafold/blob/main/README.md#examples", + "6": "https://github.com/deepmind/alphafold/blob/main/README.md" + }, + "parent_title": "", + "previous_title": "alphafold_paragraph_3", + "next_title": "alphafold_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/alphafold/#example-jobscripts" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_5.txt new file mode 100644 index 00000000000..d529a666c78 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_5.txt @@ -0,0 +1,33 @@ +Job script for running AlphaFold on GPU +Job script that runs AlphaFold on GPU using 1 V100 GPU + 8 cores. +Swap to the joltik GPU before submitting it: +module swap cluster/joltik +#!/bin/bash +#PBS -N AlphaFold-gpu-joltik +#PBS -l nodes=1:ppn=8,gpus=1 +#PBS -l walltime=10:0:0 +module load AlphaFold/2.3.1-foss-2022a-CUDA-11.7.0 +export ALPHAFOLD_DATA_DIR=/arcanine/scratch/gent/apps/AlphaFold/20230310 +WORKDIR=$VSC_SCRATCH/$PBS_JOBNAME-$PBS_JOBID +mkdir -p $WORKDIR +# download T1050.fasta via via https://www.predictioncenter.org/casp14/target.cgi?target=T1050&view=sequence +cp -a $PBS_O_WORKDIR/T1050.fasta $WORKDIR/ +cd $WORKDIR +alphafold --fasta_paths=T1050.fasta --max_template_date=2020-05-14 --db_preset=full_dbs --output_dir=$PWD +echo "Output available in $WORKDIR" +Job script for running AlphaFold CPU-only +Jobscript that runs AlphaFold on CPU using 24 cores on one node. +#!/bin/bash +#PBS -N AlphaFold-cpu-doduo +#PBS -l nodes=1:ppn=24 +#PBS -l walltime=72:0:0 +module load AlphaFold/2.3.1-foss-2022a +export ALPHAFOLD_DATA_DIR=/arcanine/scratch/gent/apps/AlphaFold/20230310 +WORKDIR=$VSC_SCRATCH/$PBS_JOBNAME-$PBS_JOBID +mkdir -p $WORKDIR +# download T1050.fasta via via https://www.predictioncenter.org/casp14/target.cgi?target=T1050&view=sequence +cp -a $PBS_O_WORKDIR/T1050.fasta $WORKDIR/ +cd $WORKDIR +alphafold --fasta_paths=T1050.fasta --max_template_date=2020-05-14 --db_preset=full_dbs --output_dir=$PWD +echo "Output available in $WORKDIR" +In case of problems or questions, don't hesitate to contact use at . \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_5_metadata.json new file mode 100644 index 00000000000..a44fef384f0 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/alphafold/alphafold_paragraph_5_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "alphafold", + "subtitle": "Job-script-for-running-AlphaFold-CPU-only", + "source_file": "../../mkdocs/docs/HPC/alphafold.md", + "title_depth": 3, + "directory": "alphafold", + "parent_title": "", + "previous_title": "alphafold_paragraph_4", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/alphafold/#job-script-for-running-alphafold-cpu-only" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_1.txt new file mode 100644 index 00000000000..0b035d5ab3b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_1.txt @@ -0,0 +1,42 @@ +Apptainer (formally known as Singularity) +What is Apptainer? +Apptainer is an open-source computer program that performs +operating-system-level virtualization (also known as containerisation). +One of the main uses of Apptainer is to bring containers and +reproducibility to scientific computing and the high-performance +computing (HPC) world. Using Apptainer/Singularity containers, +developers can work in reproducible environments of their choosing and +design, and these complete environments can easily be copied and +executed on other platforms. +For more general information about the use of Apptainer, please see the +official documentation at https://apptainer.org/docs/. +This documentation only covers aspects of using Apptainer on the +HPC-UGent infrastructure infrastructure. +Restrictions on image location +Some restrictions have been put in place on the use of Apptainer. This +is mainly done for performance reasons and to avoid that the use of +Apptainer impacts other users on the system. +The Apptainer/Singularity image file must be located on either one of +the scratch filesystems, the local disk of the workernode you are using +or /dev/shm. The centrally provided apptainer command will refuse to +run using images that are located elsewhere, in particular on the +$VSC_HOME, /apps or $VSC_DATA filesystems. +In addition, this implies that running containers images provided via a +URL (e.g., shub://... or docker://...) will not work. +If these limitations are a problem for you, please let us know via hpc@ugent.be. +Available filesystems +All HPC-UGent shared filesystems will be readily available in an +Apptainer/Singularity container, including the home, data and scratch +filesystems, and they will be accessible via the familiar $VSC_HOME, +$VSC_DATA* and $VSC_SCRATCH* environment variables. +Apptainer/Singularity Images +Creating Apptainer/Singularity images +Creating new Apptainer/Singularity images or converting Docker images, +by default, requires admin privileges, which is obviously not available +on the HPC-UGent infrastructure infrastructure. However, if you use the --fakeroot option, you +can make new Apptainer/Singularity images or convert Docker images. +Due to the nature of --fakeroot option, we recommend to write your +Apptainer/Singularity image to a globally writable location, like +/tmp, or /local directories. Once the image is created, you should +move it to your desired destination. An example to make an +Apptainer/Singularity container image: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_1_metadata.json new file mode 100644 index 00000000000..39678562b8b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "apptainer", + "subtitle": "Creating-ApptainerSingularity-images", + "source_file": "../../mkdocs/docs/HPC/apptainer.md", + "title_depth": 3, + "directory": "apptainer", + "parent_title": "", + "previous_title": null, + "next_title": "apptainer_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/apptainer/#creating-apptainersingularity-images" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_2.txt new file mode 100644 index 00000000000..ecfb6d41eb1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_2.txt @@ -0,0 +1,46 @@ +# avoid that Apptainer uses $HOME/.cache +export APPTAINER_CACHEDIR=/tmp/$USER/apptainer/cache +# instruct Apptainer to use temp dir on local filessytem +export APPTAINER_TMPDIR=/tmp/$USER/apptainer/tmpdir +# specified temp dir must exist, so create it +mkdir -p $APPTAINER_TMPDIR +# convert Docker container to Apptainer container image +apptainer build --fakeroot /tmp/$USER/tf.sif docker://nvcr.io/nvidia/tensorflow:21.10-tf1-py3 +# mv container image to $VSC_SCRATCH +mv /tmp/$USER/tf.sif $VSC_SCRATCH/tf.sif +Converting Docker images +For more information on converting existing Docker images to +Apptainer/Singularity images, see +https://apptainer.org/docs/user/main/docker_and_oci.html. +We strongly recommend the use of Docker Hub, see +https://hub.docker.com/ for more information. +Execute our own script within our container +Copy testing image from /apps/gent/tutorials/Singularity to +$VSC_SCRATCH: +cp /apps/gent/tutorials/Singularity/CentOS7_EasyBuild.img $VSC_SCRATCH/ +Create a job script like: +#!/bin/sh +#PBS -o apptainer.output +#PBS -e apptainer.error +#PBS -l nodes=1:ppn=1 +#PBS -l walltime=12:00:00 +apptainer exec $VSC_SCRATCH/CentOS7_EasyBuild.img ~/my_script.sh +Create an example myscript.sh: +#!/bin/bash +# prime factors +factor 1234567 +Tensorflow example +We already have a Tensorflow example image, but you can also convert the +Docker image (see https://hub.docker.com/r/tensorflow/tensorflow) to a +Apptainer/Singularity image yourself +Copy testing image from /apps/gent/tutorials to $VSC_SCRATCH: +cp /apps/gent/tutorials/Singularity/Ubuntu14.04_tensorflow.img $VSC_SCRATCH/ +#!/bin/sh +# +# +#PBS -o tensorflow.output +#PBS -e tensorflow.error +#PBS -l nodes=1:ppn=4 +#PBS -l walltime=12:00:00 +# +apptainer exec $VSC_SCRATCH/Ubuntu14.04_tensorflow.img python ~/linear_regression.py diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_2_metadata.json new file mode 100644 index 00000000000..ec6df548537 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "apptainer", + "subtitle": "Tensorflow-example", + "source_file": "../../mkdocs/docs/HPC/apptainer.md", + "title_depth": 2, + "directory": "apptainer", + "parent_title": "", + "previous_title": "apptainer_paragraph_1", + "next_title": "apptainer_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/apptainer/#tensorflow-example" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_3.txt new file mode 100644 index 00000000000..73860639fe2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_3.txt @@ -0,0 +1,34 @@ +You can download linear_regression.py from [the official Tensorflow +repository](https://github.com/tensorflow/tensorflow/blob/r1.12/tensorflow/examples/get_started/regression/linear_regression.py). +MPI example +It is also possible to execute MPI jobs within a container, but the +following requirements apply: +- Mellanox IB libraries must be available from the container (install + the infiniband-diags, libmlx5-1 and libmlx4-1 OS packages) +- Use modules within the container (install the environment-modules + or lmod package in your container) +- Load the required module(s) before apptainer execution. +- Set C_INCLUDE_PATH variable in your container if it is required + during compilation time + (export C_INCLUDE_PATH=/usr/include/x86_64-linux-gnu/:$C_INCLUDE_PATH + for Debian flavours) +Copy the testing image from /apps/gent/tutorials/Singularity to +$VSC_SCRATCH +cp /apps/gent/tutorials/Singularity/Debian8_UGentMPI.img $VSC_SCRATCH/ +For example to compile an [MPI +example](https://github.com/open-mpi/ompi/blob/master/examples/ring_c.c): +module load intel +apptainer shell $VSC_SCRATCH/Debian8_UGentMPI.img +export LANG=C +export C_INCLUDE_PATH=/usr/include/x86_64-linux-gnu/:$C_INCLUDE_PATH +mpiicc ompi/examples/ring_c.c -o ring_debian +exit +Example MPI job script: +#!/bin/sh +#PBS -N mpi +#PBS -o apptainermpi.output +#PBS -e apptainermpi.error +#PBS -l nodes=2:ppn=15 +#PBS -l walltime=12:00:00 +module load intel vsc-mympirun +mympirun --impi-fallback apptainer exec $VSC_SCRATCH/Debian8_UGentMPI.img ~/ring_debian diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_3_metadata.json new file mode 100644 index 00000000000..0db8305a661 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/apptainer/apptainer_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "apptainer", + "subtitle": "MPI-example", + "source_file": "../../mkdocs/docs/HPC/apptainer.md", + "title_depth": 2, + "directory": "apptainer", + "parent_title": "", + "previous_title": "apptainer_paragraph_2", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/apptainer/#mpi-example" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/best_practices/best_practices_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/best_practices/best_practices_paragraph_1.txt new file mode 100644 index 00000000000..c096d42572b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/best_practices/best_practices_paragraph_1.txt @@ -0,0 +1,39 @@ +Best Practices { #ch:best-practices} +General Best Practices { #sec:general-best-practices} +1. Before starting, you should always check: + - Are there any errors in the script? + - Are the required modules loaded? + - Is the correct executable used? +2. Check your computer requirements upfront, and request the correct + resources in your batch job script. + - Number of requested cores + - Amount of requested memory + - Requested network type +3. Check your jobs at runtime. You could login to the node and check + the proper execution of your jobs with, e.g., top or vmstat. + Alternatively you could run an interactive job (qsub -I). +4. Try to benchmark the software for scaling issues when using MPI or + for I/O issues. +5. Use the scratch file system ($VSC_SCRATCH_NODE, which is mapped to + the local /tmp) whenever possible. Local disk I/O is always much + faster as it does not have to use the network. +6. When your job starts, it will log on to the compute node(s) and + start executing the commands in the job script. It will start in + your home directory $VSC_HOME, so going to the current directory + with cd $PBS_O_WORKDIR is the first thing which needs to be done. + You will have your default environment, so don't forget to load the + software with module load. +[//]: # (Do not worry, it will render with correct numbering in all cases.) +7. In case your job not running, use "checkjob". It will show why your + job is not yet running. Sometimes commands might timeout with an + overloaded scheduler. +8. Submit your job and wait (be patient) ... +9. Submit small jobs by grouping them together. See chapter Multi-job submission for + how this is done. +10. The runtime is limited by the maximum walltime of the queues. +11. Requesting many processors could imply long queue times. It's + advised to only request the resources you'll be able to use. +12. For all multi-node jobs, please use a cluster that has an + "InfiniBand" interconnect network. +13. And above all, do not hesitate to contact the HPC staff at hpc@ugent.be. We're here + to help you. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/best_practices/best_practices_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/best_practices/best_practices_paragraph_1_metadata.json new file mode 100644 index 00000000000..72bc17a2ef1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/best_practices/best_practices_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "best_practices", + "subtitle": "General-Best-Practices", + "source_file": "../../mkdocs/docs/HPC/best_practices.md", + "title_depth": 2, + "directory": "best_practices", + "links": { + "0": "https://docs.hpc.ugent.be/multi_job_submission" + }, + "parent_title": "", + "previous_title": null, + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/best_practices/#general-best-practices" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt new file mode 100644 index 00000000000..5e9c1fc81b4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1.txt @@ -0,0 +1,39 @@ +Compiling and testing your software on the HPC +All nodes in the HPC cluster are running the "RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty)" +Operating system, which is a specific version of Red Hat Enterprise Linux. This means that all the +software programs +(executable) that the end-user wants to run on the HPC first must be +compiled for RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). It also means that you first have to install all the +required external software packages on the HPC. +Most commonly used compilers are already pre-installed on the HPC and can be +used straight away. Also, many popular external software packages, which +are regularly used in the scientific community, are also pre-installed. +Check the pre-installed software on the HPC +In order to check all the available modules and their version numbers, +which are pre-installed on the HPC enter: +When your required application is not available on the HPC please contact +any HPC member. Be aware of potential "License Costs". "Open Source" +software is often preferred. +Porting your code +To port a software-program is to translate it from the operating system in +which it was developed (e.g., Windows 7) to another operating system +(e.g., Red Hat Enterprise Linux on our HPC) so that it can be used there. Porting implies some +degree of effort, but not nearly as much as redeveloping the program in +the new environment. It all depends on how "portable" you wrote your +code. +In the simplest case the file or files may simply be copied from one +machine to the other. However, in many cases the software is installed +on a computer in a way, which depends upon its detailed hardware, +software, and setup, with device drivers for particular devices, using +installed operating system and supporting software components, and using +different directories. +In some cases software, usually described as "portable software" is +specifically designed to run on different computers with compatible +operating systems and processors without any machine-dependent +installation; it is sufficient to transfer specified directories and +their contents. Hardware- and software-specific information is often +stored in configuration files in specified locations (e.g., the registry +on machines running MS Windows). +Software, which is not portable in this sense, will have to be +transferred with modifications to support the environment on the +destination machine. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1_metadata.json new file mode 100644 index 00000000000..6ffb0d3529f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "compiling_your_software", + "subtitle": "Porting-your-code", + "source_file": "../../mkdocs/docs/HPC/compiling_your_software.md", + "title_depth": 2, + "directory": "compiling_your_software", + "parent_title": "", + "previous_title": null, + "next_title": "compiling_your_software_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/compiling_your_software/#porting-your-code" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_2.txt new file mode 100644 index 00000000000..224e61402ed --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_2.txt @@ -0,0 +1,42 @@ +Whilst programming, it would be wise to stick to certain standards +(e.g., ISO/ANSI/POSIX). This will ease the porting of your code to other +platforms. +Porting your code to the RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty) platform is the responsibility of the end-user. +Compiling and building on the HPC +Compiling refers to the process of translating code written in some +programming language, e.g., Fortran, C, or C++, to machine code. +Building is similar, but includes gluing together the machine code +resulting from different source files into an executable (or library). +The text below guides you through some basic problems typical for small +software projects. For larger projects it is more appropriate to use +makefiles or even an advanced build system like CMake. +All the HPC nodes run the same version of the Operating System, i.e. RHEL 8.8 (accelgor, doduo, donphan, gallade, joltik, skitty). So, +it is sufficient to compile your program on any compute node. Once you +have generated an executable with your compiler, this executable should +be able to run on any other compute-node. +A typical process looks like: +1. Copy your software to the login-node of the HPC +2. Start an interactive session on a compute node; +3. Compile it; +4. Test it locally; +5. Generate your job scripts; +6. Test it on the HPC +7. Run it (in parallel); +We assume you've copied your software to the HPC. The next step is to request +your private compute node. +$ qsub -I +qsub: waiting for job 123456 to start +Compiling a sequential program in C +Go to the examples for chapter +Compiling and testing your software on the HPC and load the +foss module: +cd ~/examples/Compiling-and-testing-your-software-on-the-HPC +module load foss +We now list the directory and explore the contents of the "hello.c" +program: +$ ls -l +total 512 +-rw-r--r-- 1 vsc40000 214 Sep 16 09:42 hello.c +-rw-r--r-- 1 vsc40000 130 Sep 16 11:39 hello.pbs* +-rw-r--r-- 1 vsc40000 359 Sep 16 13:55 mpihello.c +-rw-r--r-- 1 vsc40000 304 Sep 16 13:55 mpihello.pbs diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_2_metadata.json new file mode 100644 index 00000000000..4984907716e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_2_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "compiling_your_software", + "subtitle": "Compiling-a-sequential-program-in-C", + "source_file": "../../mkdocs/docs/HPC/compiling_your_software.md", + "title_depth": 3, + "directory": "compiling_your_software", + "links": { + "0": "https://docs.hpc.ugent.be/compiling_your_software/#compiling-and-building-on-the-hpc" + }, + "parent_title": "", + "previous_title": "compiling_your_software_paragraph_1", + "next_title": "compiling_your_software_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/compiling_your_software/#compiling-a-sequential-program-in-c" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_3.txt new file mode 100644 index 00000000000..4a667fca1d7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_3.txt @@ -0,0 +1,50 @@ +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: Print 500 numbers, whilst waiting 1 second in between + */ +#include "stdio.h" +int main( int argc, char *argv[] ) +{ + int i; + for (i=0; i<500; i++) + { + printf("Hello #%d\n", i); + fflush(stdout); + sleep(1); + } +} +The "hello.c" program is a simple source file, written in C. It'll print +500 times "Hello #<num>", and waits one second between 2 printouts. +We first need to compile this C-file into an executable with the +gcc-compiler. +First, check the command line options for *"gcc" (GNU C-Compiler)*, then +we compile. the O2 option enables a moderate level of optimization when compiling the code. +It instructs the compiler to optimize the code for better performance without significantly increasing compilation time. +Finally, list the contents of the directory again: +$ gcc -help +$ gcc -O2 -o hello hello.c +$ ls -l +total 512 +-rwxrwxr-x 1 vsc40000 7116 Sep 16 11:43 hello* +-rw-r--r-- 1 vsc40000 214 Sep 16 09:42 hello.c +-rwxr-xr-x 1 vsc40000 130 Sep 16 11:39 hello.pbs* +A new file "hello" has been created. Note that this file has "execute" +rights, i.e., it is an executable. More often than not, calling gcc -- +or any other compiler for that matter -- will provide you with a list of +errors and warnings referring to mistakes the programmer made, such as +typos, syntax errors. You will have to correct them first in order to +make the code compile. Warnings pinpoint less crucial issues that may +relate to performance problems, using unsafe or obsolete language +features, etc. It is good practice to remove all warnings from a +compilation process, even if they seem unimportant so that a code change +that produces a warning does not go unnoticed. +Let's test this program on the local compute node, which is at your +disposal after the qsub --I command: +$ ./hello +Hello #0 +Hello #1 +Hello #2 +Hello #3 +Hello #4 +... diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_3_metadata.json new file mode 100644 index 00000000000..8209e6c4ae4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "compiling_your_software", + "subtitle": "Compiling-a-sequential-program-in-C", + "source_file": "../../mkdocs/docs/HPC/compiling_your_software.md", + "title_depth": 3, + "directory": "compiling_your_software", + "parent_title": "", + "previous_title": "compiling_your_software_paragraph_2", + "next_title": "compiling_your_software_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/compiling_your_software/#compiling-a-sequential-program-in-c" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_4.txt new file mode 100644 index 00000000000..dfbb87a2e20 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_4.txt @@ -0,0 +1,52 @@ +It seems to work, now run it on the HPC +qsub hello.pbs +Compiling a parallel program in C/MPI +cd ~/examples/Compiling-and-testing-your-software-on-the-HPC +List the directory and explore the contents of the "mpihello.c" +program: +$ ls -l +total 512 +total 512 +-rw-r--r-- 1 vsc40000 214 Sep 16 09:42 hello.c +-rw-r--r-- 1 vsc40000 130 Sep 16 11:39 hello.pbs* +-rw-r--r-- 1 vsc40000 359 Sep 16 13:55 mpihello.c +-rw-r--r-- 1 vsc40000 304 Sep 16 13:55 mpihello.pbs +/* + * VSC : Flemish Supercomputing Centre + * Tutorial : Introduction to HPC + * Description: Example program, to compile with MPI + */ +#include +#include +main(int argc, char **argv) +{ + int node, i, j; + float f; + MPI_Init(&argc,&argv); + MPI_Comm_rank(MPI_COMM_WORLD, &node); + + printf("Hello World from Node %d.\n", node); + for (i=0; i<=100000; i++) + f=i*2.718281828*i+i+i*3.141592654; + MPI_Finalize(); +} +The "mpi_hello.c" program is a simple source file, written in C with MPI +library calls. +Then, check the command line options for *"mpicc" (GNU C-Compiler with +MPI extensions)*, then we compile and list the contents of the directory +again: +mpicc --help +mpicc -o mpihello mpihello.c +ls -l +A new file "hello" has been created. Note that this program has +"execute" rights. +Let's test this program on the "login" node first: +$ ./mpihello +Hello World from Node 0. +It seems to work, now run it on the HPC. +qsub mpihello.pbs +Compiling a parallel program in Intel Parallel Studio Cluster Edition +We will now compile the same program, but using the Intel Parallel +Studio Cluster Edition compilers. We stay in the examples directory for +this chapter: +cd ~/examples/Compiling-and-testing-your-software-on-the-HPC diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_4_metadata.json new file mode 100644 index 00000000000..d5416e5b7bc --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_4_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "compiling_your_software", + "subtitle": "Compiling-a-parallel-program-in-Intel-Parallel-Studio-Cluster-Edition", + "source_file": "../../mkdocs/docs/HPC/compiling_your_software.md", + "title_depth": 3, + "directory": "compiling_your_software", + "parent_title": "", + "previous_title": "compiling_your_software_paragraph_3", + "next_title": "compiling_your_software_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/compiling_your_software/#compiling-a-parallel-program-in-intel-parallel-studio-cluster-edition" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_5.txt new file mode 100644 index 00000000000..a94d986ff58 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_5.txt @@ -0,0 +1,28 @@ +We will compile this C/MPI -file into an executable with the Intel +Parallel Studio Cluster Edition. First, clear the modules (purge) and +then load the latest "intel" module: +module purge +module load intel +Then, compile and list the contents of the directory again. The Intel +equivalent of mpicc is mpiicc. +mpiicc -o mpihello mpihello.c +ls -l +Note that the old "mpihello" file has been overwritten. Let's test this +program on the "login" node first: +$ ./mpihello +Hello World from Node 0. +It seems to work, now run it on the HPC. +qsub mpihello.pbs +Note: The AUGent only has a license for the Intel Parallel Studio Cluster +Edition for a fixed number of users. As such, it might happen that you +have to wait a few minutes before a floating license becomes available +for your use. +Note: The Intel Parallel Studio Cluster Edition contains equivalent +compilers for all GNU compilers. Hereafter the overview for C, C++ and +Fortran compilers. +| | Sequential Program | | **Parallel Program (with MPI)** | | +|-------------|------------------------|-----------|---------------------------------|-----------| +| | GNU | Intel | GNU | Intel | +| C | gcc | icc | mpicc | mpiicc | +| **C++** | g++ | icpc | mpicxx | mpiicpc | +| Fortran | gfortran | ifort | mpif90 | mpiifort | \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_5_metadata.json new file mode 100644 index 00000000000..6ba11883f8b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/compiling_your_software/compiling_your_software_paragraph_5_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "compiling_your_software", + "subtitle": "Compiling-a-parallel-program-in-Intel-Parallel-Studio-Cluster-Edition", + "source_file": "../../mkdocs/docs/HPC/compiling_your_software.md", + "title_depth": 3, + "directory": "compiling_your_software", + "parent_title": "", + "previous_title": "compiling_your_software_paragraph_4", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/compiling_your_software/#compiling-a-parallel-program-in-intel-parallel-studio-cluster-edition" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1.txt new file mode 100644 index 00000000000..53bee49de3f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1.txt @@ -0,0 +1,39 @@ +Connecting to the HPC infrastructure +Before you can really start using the HPC clusters, there are several things +you need to do or know: +1. You need to log on to the cluster using an SSH client to one of + the login nodes or by using the HPC web portal. + This will give you command-line access. + A standard web browser like Firefox or Chrome for the web portal will suffice. +2. Before you can do some work, you'll have to transfer the files + that you need from your desktop computer to the cluster. At the end + of a job, you might want to transfer some files back. +3. Optionally, if you wish to use programs with a **graphical user + interface**, you will need an X-server on your client system and log + in to the login nodes with X-forwarding enabled. +4. Often several versions of software packages and libraries are + installed, so you need to select the ones you need. To manage + different versions efficiently, the VSC clusters use so-called + modules, so you will need to select and load the modules that + you need. +Connection restrictions +Since March 20th 2020, restrictions are in place that limit from where +you can connect to the VSC HPC infrastructure, in response to security +incidents involving several European HPC centres. +VSC login nodes are only directly accessible from within university +networks, and from (most) Belgian commercial internet providers. +All other IP domains are blocked by default. If you are connecting from +an IP address that is not allowed direct access, you have the following +options to get access to VSC login nodes: +- Use an VPN connection to connect to UGent the network (recommended). See https://helpdesk.ugent.be/vpn/en/ for more information. +- Whitelist your IP address automatically by accessing + https://firewall.vscentrum.be and log in with your UGent account. + - While this web connection is active new SSH sessions can be + started. + - Active SSH sessions will remain active even when this web page + is closed. +- Contact your HPC support team (via hpc@ugent.be) and ask them to whitelist your + IP range (e.g., for industry access, automated processes). +Trying to establish an SSH connection from an IP address that does not +adhere to these restrictions will result in an immediate failure to +connect, with an error message like: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1_metadata.json new file mode 100644 index 00000000000..450a04e1aa8 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Connection-restrictions", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "links": { + "0": "https://docs.hpc.ugent.be/web_portal" + }, + "parent_title": "", + "previous_title": null, + "next_title": "connecting_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/connecting/#connection-restrictions" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_4.txt new file mode 100644 index 00000000000..4c910d86d35 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_4.txt @@ -0,0 +1,35 @@ +This directory contains: +1. This HPC Tutorial (in either a Mac, Linux or Windows version). +2. An examples subdirectory, containing all the examples that you need in this + Tutorial, as well as examples that might be useful for your specific applications. +cd examples + tip + Typing cd ex followed by tab (the Tab-key) will generate the cd examples + command. Command-line completion (also tab completion) is a common feature of the bash command + line interpreter, in which the program automatically fills in partially + typed commands. + tip + For more exhaustive tutorials about Linux usage, see Appendix Useful Linux Commands +The first action is to copy the contents of the HPC examples directory to +your home directory, so that you have your own personal copy and that +you can start using the examples. The "-r" option of the copy command +will also copy the contents of the sub-directories "recursively". +cp -r /apps/gent/tutorials/Intro-HPC/examples ~/ +Go to your home directory, check your own private examples directory, ... and start working. +cd +ls -l +Upon connecting you will see a login message containing your last login time stamp and a basic overview of the current cluster utilisation. +Last login: Thu Mar 18 13:15:09 2021 from gligarha02.gastly.os + STEVIN HPC-UGent infrastructure status on Mon, 19 Feb 2024 10:00:01 + cluster - full - free - part - total - running - queued + nodes nodes free nodes jobs jobs + ------------------------------------------------------------------------- + skitty 39 0 26 68 1839 5588 + joltik 6 0 1 10 29 18 + doduo 22 0 75 128 1397 11933 + accelgor 4 3 2 9 18 1 + donphan 0 0 16 16 16 13 + gallade 2 0 5 16 19 136 +For a full view of the current loads and queues see: +https://hpc.ugent.be/clusterstate/ +Updates on current system status and planned maintenance can be found on https://www.ugent.be/hpc/en/infrastructure/status diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_4_metadata.json new file mode 100644 index 00000000000..faaf9fdf9c2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_4_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "links": { + "0": "https://docs.hpc.ugent.be/useful_linux_commands" + }, + "parent_title": "", + "previous_title": "connecting_paragraph_3", + "next_title": "connecting_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9.txt new file mode 100644 index 00000000000..dd4f3269fb5 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9.txt @@ -0,0 +1,11 @@ +If you want to find out which login host you are connected to, you can use the hostname command. +$ hostname +gligar07.gastly.os +$ ssh gligar08.gastly.os +$ hostname +gligar08.gastly.os +Rather than always starting a new session on the HPC, you can also use a terminal multiplexer like screen or tmux. +These can make sessions that 'survives' across disconnects. +You can find more information on how to use these tools here (or on other online sources): +- screen +- tmux \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9_metadata.json new file mode 100644 index 00000000000..7aafa76a65f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/connecting/connecting_paragraph_9_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "connecting", + "subtitle": "Changing-login-nodes", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "links": { + "0": "https://www.howtogeek.com/662422/how-to-use-linuxs-screen-command/", + "1": "https://www.howtogeek.com/671422/how-to-use-tmux-on-linux-and-why-its-better-than-screen/" + }, + "parent_title": "", + "previous_title": "connecting_paragraph_8", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/connecting/#changing-login-nodes" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/crontab/crontab_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/crontab/crontab_paragraph_1.txt new file mode 100644 index 00000000000..12d31c1677b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/crontab/crontab_paragraph_1.txt @@ -0,0 +1,28 @@ +Cron scripts +Cron scripts configuration +It is possible to run automated cron scripts as regular user on the +Ugent login nodes. Due to the high availability setup users should add +their cron scripts on the same login node to avoid any cron job script +duplication. +In order to create a new cron script first login to HPC-UGent login node +as usual with your vsc user's account (see section +Connecting). +Check if any cron script is already set in the current login node with: +crontab -l +At this point you can add/edit (with vi editor) any cron script +running the command: +crontab -e +Example cron job script + 15 5 * * * ~/runscript.sh >& ~/job.out +where runscript.sh has these lines in this example: +#!/bin/bash +module swap cluster/donphan +export SLURM_CLUSTERS="donphan" +/usr/libexec/jobcli/qsub ~/job_scripts/test.sh >& ~/job.out +In the previous example a cron script was set to be executed every day +at 5:15 am. More information about crontab and cron scheduling format at +https://www.redhat.com/sysadmin/automate-linux-tasks-cron. +Please note that you should login into the same login node to edit your +previously generated crontab tasks. If that is not the case you can +always jump from one login node to another with: +ssh gligar07 # or gligar08 diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/crontab/crontab_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/crontab/crontab_paragraph_1_metadata.json new file mode 100644 index 00000000000..e55974cb0c4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/crontab/crontab_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "crontab", + "subtitle": "Example-cron-job-script", + "source_file": "../../mkdocs/docs/HPC/crontab.md", + "title_depth": 4, + "directory": "crontab", + "links": { + "0": "https://docs.hpc.ugent.be/connecting/#connecting-to-the-hpc-infrastructure" + }, + "parent_title": "", + "previous_title": null, + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/crontab/#example-cron-job-script" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_1.txt new file mode 100644 index 00000000000..60ac3607af2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_1.txt @@ -0,0 +1,44 @@ +Easybuild +What is Easybuild? +You can use EasyBuild to build and install supported software in your +own VSC account, rather than requesting a central installation by the +HPC support team. +EasyBuild (https://easybuilders.github.io/easybuild) is the software +build and installation framework that was created by the HPC-UGent team, +and has recently been picked up by HPC sites around the world. It allows +you to manage (scientific) software on High Performance Computing (HPC) +systems in an efficient way. +When should I use Easybuild? +For general software installation requests, please see I want to use software that is not available on the clusters yet. However, there +might be reasons to install the software yourself: +- applying custom patches to the software that only you or your group + are using +- evaluating new software versions prior to requesting a central + software installation +- installing (very) old software versions that are no longer eligible + for central installation (on new clusters) +Configuring EasyBuild +Before you use EasyBuild, you need to configure it: +Path to sources +This is where EasyBuild can find software sources: +EASYBUILD_SOURCEPATH=$VSC_DATA/easybuild/sources:/apps/gent/source +- the first directory $VSC_DATA/easybuild/sources is where EasyBuild + will (try to) automatically download sources if they're not + available yet +- /apps/gent/source is the central "cache" for already downloaded + sources, and will be considered by EasyBuild before downloading + anything +Build directory +This directory is where EasyBuild will build software in. To have good +performance, this needs to be on a fast filesystem. +export EASYBUILD_BUILDPATH=${TMPDIR:-/tmp/$USER} +On cluster nodes, you can use the fast, in-memory /dev/shm/$USER +location as a build directory. +Software install location +This is where EasyBuild will install the software (and accompanying +modules) to. +For example, to let it use $VSC_DATA/easybuild, use: +export EASYBUILD_INSTALLPATH=$VSC_DATA/easybuild/$VSC_OS_LOCAL/$VSC_ARCH_LOCAL$VSC_ARCH_SUFFIX +Using the $VSC_OS_LOCAL, $VSC_ARCH and $VSC_ARCH_SUFFIX +environment variables ensures that your install software to a location +that is specific to the cluster you are building for. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_1_metadata.json new file mode 100644 index 00000000000..fee2b54bb4b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "easybuild", + "subtitle": "Software-install-location", + "source_file": "../../mkdocs/docs/HPC/easybuild.md", + "title_depth": 3, + "directory": "easybuild", + "links": { + "0": "https://docs.hpc.ugent.be/FAQ/#i-want-to-use-software-that-is-not-available-on-the-clusters-yet" + }, + "parent_title": "", + "previous_title": null, + "next_title": "easybuild_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/easybuild/#software-install-location" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_2.txt new file mode 100644 index 00000000000..9136e563891 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_2.txt @@ -0,0 +1,33 @@ +Make sure you do not build software on the login nodes, since the loaded cluster module determines the +location of the installed software. Software built on the login nodes +may not work on the cluster you want to use the software on (see also Running software that is incompatible with host). +To share custom software installations with members of your VO, replace +$VSC_DATA with $VSC_DATA_VO in the example above. +Using EasyBuild +Before using EasyBuild, you first need to load the EasyBuild module. +We don't specify a version here (this is an exception, for most other +modules you should see Using explicit version numbers) because newer versions might include important +bug fixes. +module load EasyBuild +Installing supported software +EasyBuild provides a large collection of readily available software +versions, combined with a particular toolchain version. Use the +--search (or -S) functionality to see which different 'easyconfigs' +(build recipes, see +http://easybuild.readthedocs.org/en/latest/Concepts_and_Terminology.html#easyconfig-files) are available: +$ eb -S example-1.2 +CFGS1=/apps/gent/CO7/sandybridge/software/EasyBuild/3.6.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.6.2-py2.7.egg/easybuild/easyconfigs + * $CFGS1/e/example/example-1.2.1-foss-a.eb + * $CFGS1/e/example/example-1.2.3-foss-b.eb + * $CFGS1/e/example/example-1.2.5-intel-a.eb +For readily available easyconfigs, just specify the name of the +easyconfig file to build and install the corresponding software package: +eb example-1.2.1-foss-2024a.eb --robot +Installing variants on supported software +To install small variants on supported software, e.g., a different +software version, or using a different compiler toolchain, use the +corresponding --try-X options: +To try to install example v1.2.6, based on the easyconfig file for +example v1.2.5: +eb example-1.2.5-intel-a.eb --try-software-version=1.2.6 +To try to install example v1.2.5 with a different compiler toolchain: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_2_metadata.json new file mode 100644 index 00000000000..5927768b87d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_2_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "easybuild", + "subtitle": "Installing-variants-on-supported-software", + "source_file": "../../mkdocs/docs/HPC/easybuild.md", + "title_depth": 3, + "directory": "easybuild", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#running-software-that-is-incompatible-with-host", + "1": "https://docs.hpc.ugent.be/running_batch_jobs/#using-explicit-version-numbers" + }, + "parent_title": "", + "previous_title": "easybuild_paragraph_1", + "next_title": "easybuild_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/easybuild/#installing-variants-on-supported-software" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_3.txt new file mode 100644 index 00000000000..148b0753508 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_3.txt @@ -0,0 +1,18 @@ +eb example-1.2.5-intel-a.eb --robot --try-toolchain=intel,b +Install other software +To install other, not yet supported, software, you will need to provide +the required easyconfig files yourself. See +https://easybuild.readthedocs.org/en/latest/Writing_easyconfig_files.html +for more information. +Using the installed modules +To use the modules you installed with EasyBuild, extend $MODULEPATH to +make them accessible for loading: +module use $EASYBUILD_INSTALLPATH/modules/all +It makes sense to put this module use command and all export +commands in your .bashrc login script. +That way, you don't have to type +these commands every time you want to use EasyBuild or you want to load +modules generated with EasyBuild. +See also [the section on .bashrc in +the "Beyond the basics" chapter of the intro to +Linux](../linux-tutorial/beyond_the_basics/#bashrc-login-script) diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_3_metadata.json new file mode 100644 index 00000000000..36db9a90c13 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/easybuild/easybuild_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "easybuild", + "subtitle": "Using-the-installed-modules", + "source_file": "../../mkdocs/docs/HPC/easybuild.md", + "title_depth": 2, + "directory": "easybuild", + "parent_title": "", + "previous_title": "easybuild_paragraph_2", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/easybuild/#using-the-installed-modules" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_1.txt new file mode 100644 index 00000000000..3cebfd9b571 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_1.txt @@ -0,0 +1,42 @@ +Fine-tuning Job Specifications +[//]: # (ch:fine-tuning-job-specifications) +As HPC system administrators, we often observe that the HPC resources are not +optimally (or wisely) used. For example, we regularly notice that +several cores on a computing node are not utilised, due to the fact that +one sequential program uses only one core on the node. Or users run I/O +intensive applications on nodes with "slow" network connections. +Users often tend to run their jobs without specifying specific PBS Job +parameters. As such, their job will automatically use the default +parameters, which are not necessarily (or rarely) the optimal ones. This +can slow down the run time of your application, but also block HPC resources +for other users. +Specifying the "optimal" Job Parameters requires some knowledge of your +application (e.g., how many parallel threads does my application use, +is there a lot of inter-process communication, how much memory does my +application need) and also some knowledge about the HPC infrastructure +(e.g., what kind of multi-core processors are available, which nodes +have InfiniBand). +There are plenty of monitoring tools on Linux available to the user, +which are useful to analyse your individual application. The HPC environment +as a whole often requires different techniques, metrics and time goals, +which are not discussed here. We will focus on tools that can help to +optimise your Job Specifications. +Determining the optimal computer resource specifications can be broken +down into different parts. The first is actually determining which +metrics are needed and then collecting that data from the hosts. Some of +the most commonly tracked metrics are CPU usage, memory consumption, +network bandwidth, and disk I/O stats. These provide different +indications of how well a system is performing, and may indicate where +there are potential problems or performance bottlenecks. Once the data +have actually been acquired, the second task is analysing the data and +adapting your PBS Job Specifications. +Another different task is to monitor the behaviour of an application at +run time and detect anomalies or unexpected behaviour. Linux provides a +large number of utilities to monitor the performance of its components. +This chapter shows you how to measure: +1. Walltime +2. Memory usage +3. CPU usage +4. Disk (storage) needs +5. Network bottlenecks +First, we allocate a compute node and move to our relevant directory: diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_1_metadata.json new file mode 100644 index 00000000000..ace4c7c9714 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Fine-tuning-Job-Specifications", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 1, + "directory": "fine_tuning_job_specifications", + "parent_title": "", + "previous_title": null, + "next_title": "fine_tuning_job_specifications_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#fine-tuning-job-specifications" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_2.txt new file mode 100644 index 00000000000..c34e4227576 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_2.txt @@ -0,0 +1,44 @@ +qsub -I +cd ~/examples/Fine-tuning-Job-Specifications +Specifying Walltime +[//]: # (sec:specifying-walltime-requirements) +One of the most important and also easiest parameters to measure is the +duration of your program. This information is needed to specify the +walltime. +The time utility executes and times your application. You can just add the time +command in front of your normal command line, including your command +line options. After your executable has finished, time writes the total time +elapsed, the time consumed by system overhead, and the time used to +execute your executable to the standard error stream. The calculated +times are reported in seconds. +Test the time command: +$ time sleep 75 +real 1m15.005s +user 0m0.001s +sys 0m0.002s +It is a good practice to correctly estimate and specify the run time +(duration) of an application. Of course, a margin of 10% to 20% can be +taken to be on the safe side. +It is also wise to check the walltime on different compute nodes or to +select the "slowest" compute node for your walltime tests. Your estimate +should be appropriate in case your application will run on the "slowest" +(oldest) compute nodes. +The walltime can be specified in a job scripts as: +#PBS -l walltime=3:00:00:00 +or on the command line +qsub -l walltime=3:00:00:00 +It is recommended to always specify the walltime for a job. +Specifying memory requirements +[//]: # (sec:specifying-memory-requirements) +In many situations, it is useful to monitor the amount of memory an +application is using. You need this information to determine the +characteristics of the required compute node, where that application +should run on. Estimating the amount of memory an application will use +during execution is often non-trivial, especially when one uses +third-party software. +Available Memory on the machine +The first point is to be aware of the available free memory in your +computer. The "free" command displays the total amount of free and +used physical and swap memory in the system, as well as the buffers used +by the kernel. We also use the options "-m" to see the results expressed +in Mega-Bytes and the "-t" option to get totals. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_2_metadata.json new file mode 100644 index 00000000000..39460947d0d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Available-Memory-on-the-machine", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 3, + "directory": "fine_tuning_job_specifications", + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_1", + "next_title": "fine_tuning_job_specifications_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#available-memory-on-the-machine" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_3.txt new file mode 100644 index 00000000000..6ca183af09a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_3.txt @@ -0,0 +1,49 @@ +$ free -m -t + total used free shared buffers cached +Mem: 16049 4772 11277 0 107 161 +-/+ buffers/cache: 4503 11546 +Swap: 16002 4185 11816 +Total: 32052 8957 23094 +Important is to note the total amount of memory available in the machine +(i.e., 16 GB in this example) and the amount of used and free memory +(i.e., 4.7 GB is used and another 11.2 GB is free here). +It is not a good practice to use swap-space for your computational +applications. A lot of "swapping" can increase the execution time of +your application tremendously. +On the UGent clusters, there is no swap space available for jobs, you +can only use physical memory, even though "free" will show swap. +Checking the memory consumption +To monitor the memory consumption of a running application, you can use the "top" or the "htop" command. +top +: provides an ongoing look at processor activity in real time. It + displays a listing of the most CPU-intensive tasks on the system, + and can provide an interactive interface for manipulating processes. + It can sort the tasks by memory usage, CPU usage and run time. +htop +: is similar to top, but shows the CPU-utilisation for all the CPUs in + the machine and allows to scroll the list vertically and + horizontally to see all processes and their full command lines. +Setting the memory parameter {: #pbs_mem } +Once you gathered a good idea of the overall memory consumption of your +application, you can define it in your job script. It is wise to foresee +a margin of about 10%. +The maximum amount of physical memory used by the job per node can be +specified in a job script as: +#PBS -l mem=4gb +or on the command line +qsub -l mem=4gb +Specifying processors requirements +Users are encouraged to fully utilise all the available cores on a +certain compute node. Once the required numbers of cores and nodes are +decently specified, it is also good practice to monitor the CPU +utilisation on these cores and to make sure that all the assigned nodes +are working at full load. +Number of processors +The number of core and nodes that a user shall request fully depends on +the architecture of the application. Developers design their +applications with a strategy for parallelization in mind. The +application can be designed for a certain fixed number or for a +configurable number of nodes and cores. It is wise to target a specific +set of compute nodes (e.g., Westmere, Harpertown) for your computing +work and then to configure your software to nicely fill up all +processors on these compute nodes. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_3_metadata.json new file mode 100644 index 00000000000..6eca439dcea --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_3_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Number-of-processors", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 3, + "directory": "fine_tuning_job_specifications", + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_2", + "next_title": "fine_tuning_job_specifications_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#number-of-processors" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_4.txt new file mode 100644 index 00000000000..7fdb9a8f6a7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_4.txt @@ -0,0 +1,60 @@ +The /proc/cpuinfo stores info about your CPU architecture like number +of CPUs, threads, cores, information about CPU caches, CPU family, model +and much more. So, if you want to detect how many cores are available on +a specific machine: +$ less /proc/cpuinfo +processor : 0 +vendor_id : GenuineIntel +cpu family : 6 +model : 23 +model name : Intel(R) Xeon(R) CPU E5420 @ 2.50GHz +stepping : 10 +cpu MHz : 2500.088 +cache size : 6144 KB +... +Or if you want to see it in a more readable format, execute: +$ grep processor /proc/cpuinfo +processor : 0 +processor : 1 +processor : 2 +processor : 3 +processor : 4 +processor : 5 +processor : 6 +processor : 7 + note + Unless you want information of the login nodes, you'll have to issue + these commands on one of the workernodes. This is most easily achieved + in an interactive job, see the chapter on Running interactive jobs. +In order to specify the number of nodes and the number of processors per +node in your job script, use: +#PBS -l nodes=N:ppn=M +or with equivalent parameters on the command line +qsub -l nodes=N:ppn=M +This specifies the number of nodes (nodes=N) and the number of +processors per node (ppn=M) that the job should use. PBS treats a +processor core as a processor, so a system with eight cores per compute +node can have ppn=8 as its maximum ppn request. +You can also use this statement in your job script: +#PBS -l nodes=N:ppn=all +to request all cores of a node, or +#PBS -l nodes=N:ppn=half +to request half of them. +Note that unless a job has some inherent parallelism of its own through +something like MPI or OpenMP, requesting more than a single processor on +a single node is usually wasteful and can impact the job start time. +Monitoring the CPU-utilisation +This could also be monitored with the htop command: +htop +Example output: + 1 [||| 11.0%] 5 [|| 3.0%] 9 [|| 3.0%] 13 [ 0.0%] + 2 [|||||100.0%] 6 [ 0.0%] 10 [ 0.0%] 14 [ 0.0%] + 3 [|| 4.9%] 7 [|| 9.1%] 11 [ 0.0%] 15 [ 0.0%] + 4 [|| 1.8%] 8 [ 0.0%] 12 [ 0.0%] 16 [ 0.0%] + Mem[|||||||||||||||||59211/64512MB] Tasks: 323, 932 thr; 2 running + Swp[|||||||||||| 7943/20479MB] Load average: 1.48 1.46 1.27 + Uptime: 211 days(!), 22:12:58 + PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command +22350 vsc00000 20 0 1729M 1071M 704 R 98.0 1.7 27:15.59 bwa index + 7703 root 0 -20 10.1G 1289M 70156 S 11.0 2.0 36h10:11 /usr/lpp/mmfs/bin +27905 vsc00000 20 0 123M 2800 1556 R 7.0 0.0 0:17.51 htop diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_4_metadata.json new file mode 100644 index 00000000000..fa856402fed --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_4_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Monitoring-the-CPU-utilisation", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 3, + "directory": "fine_tuning_job_specifications", + "links": { + "0": "https://docs.hpc.ugent.be/running_interactive_jobs" + }, + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_3", + "next_title": "fine_tuning_job_specifications_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#monitoring-the-cpu-utilisation" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_5.txt new file mode 100644 index 00000000000..20c96d20fc7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_5.txt @@ -0,0 +1,42 @@ +The advantage of htop is that it shows you the cpu utilisation for all +processors as well as the details per application. A nice exercise is to +start 4 instances of the "cpu_eat" program in 4 different terminals, and +inspect the cpu utilisation per processor with monitor and htop. +If htop reports that your program is taking 75% CPU on a certain processor, +it means that 75% of the samples taken by top found your process active +on the CPU. The rest of the time your application was in a wait. (It is +important to remember that a CPU is a discrete state machine. It really +can be at only 100%, executing an instruction, or at 0%, waiting for +something to do. There is no such thing as using 45% of a CPU. The CPU +percentage is a function of time.) However, it is likely that your +application's rest periods include waiting to be dispatched on a CPU and +not on external devices. That part of the wait percentage is then very +relevant to understanding your overall CPU usage pattern. +Fine-tuning your executable and/or job script +It is good practice to perform a number of run time stress tests, and to +check the CPU utilisation of your nodes. We (and all other users of the +HPC) would appreciate that you use the maximum of the CPU resources that +are assigned to you and make sure that there are no CPUs in your node +who are not utilised without reasons. +But how can you maximise? +1. Configure your software. (e.g., to exactly use the available amount + of processors in a node) +2. Develop your parallel program in a smart way. +3. Demand a specific type of compute node (e.g., Harpertown, Westmere), + which have a specific number of cores. +4. Correct your request for CPUs in your job script. +The system load +On top of the CPU utilisation, it is also important to check the system load. +The system load is a measure of the amount of computational work that a computer +system performs. +The system load is the number of applications running or waiting to run +on the compute node. In a system with for example four CPUs, a load +average of 3.61 would indicate that there were, on average, 3.61 +processes ready to run, and each one could be scheduled into a CPU. +The load averages differ from CPU percentage in two significant ways: +1. "load averages" measure the trend of processes waiting to be run + (and not only an instantaneous snapshot, as does CPU percentage); + and +2. "load averages" include all demand for all resources, e.g., CPU + and also I/O and network (and not only how much was active at the + time of measurement). diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_5_metadata.json new file mode 100644 index 00000000000..b0f65667858 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_5_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "The-system-load", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 2, + "directory": "fine_tuning_job_specifications", + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_4", + "next_title": "fine_tuning_job_specifications_paragraph_6", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#the-system-load" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_6.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_6.txt new file mode 100644 index 00000000000..b38d226db36 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_6.txt @@ -0,0 +1,38 @@ +Optimal load +What is the "optimal load" rule of thumb? +The load averages tell us whether our physical CPUs are over- or +under-utilised. The point of perfect utilisation, meaning that the CPUs are always busy and, yet, no +process ever waits for one, is **the +average matching the number of CPUs**. Your load should not exceed the number +of cores available. E.g., if there are four CPUs on a machine and the +reported one-minute load average is 4.00, the machine has been utilising +its processors perfectly for the last 60 seconds. The "100% utilisation" +mark is 1.0 on a single-core system, 2.0 on a dual-core, 4.0 on a +quad-core, etc. The optimal load shall be between 0.7 and 1.0 per +processor. +In general, the intuitive idea of load averages is the higher they rise +above the number of processors, the more processes are waiting and doing +nothing, and the lower they fall below the number of processors, the +more untapped CPU capacity there is. +Load averages do include any processes or threads waiting on I/O, +networking, databases or anything else not demanding the CPU. This means +that the optimal number of applications running on a system at the +same time, might be more than one per processor. +The "optimal number of applications" running on one machine at the same time depends on the type of +the applications that you are running. +1. When you are running computational intensive applications, one application per processor will generate + the optimal load. +2. For I/O intensive applications (e.g., applications which perform a lot of disk-I/O), a higher + number of applications can generate the optimal load. While some + applications are reading or writing data on disks, the processors + can serve other applications. +The optimal number of applications on a machine could be empirically +calculated by performing a number of stress tests, whilst checking the +highest throughput. There is however no manner in the HPC at the moment to +specify the maximum number of applications that shall run per core +dynamically. The HPC scheduler will not launch more than one process per +core. +The manner how the cores are spread out over CPUs does not matter for +what regards the load. Two quad-cores perform similar to four +dual-cores, and again perform similar to eight single-cores. It's all +eight cores for these purposes. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_6_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_6_metadata.json new file mode 100644 index 00000000000..4a3b26ca3c2 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_6_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Optimal-load", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 3, + "directory": "fine_tuning_job_specifications", + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_5", + "next_title": "fine_tuning_job_specifications_paragraph_7", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#optimal-load" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_7.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_7.txt new file mode 100644 index 00000000000..6959f61d299 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_7.txt @@ -0,0 +1,42 @@ +Monitoring the load +The load average represents the average system load over a period of time. It +conventionally appears in the form of three numbers, which represent the +system load during the last one-, five-, and fifteen-minute periods. +The uptime command will show us the average load +$ uptime +10:14:05 up 86 days, 12:01, 11 users, load average: 0.60, 0.41, 0.41 +Now, compile and start a few instances of the "eat_cpu" program in the background, +and check the effect on the load again: +$ gcc -O2 eat_cpu.c -o eat_cpu +$ ./eat_cpu& +$ ./eat_cpu& +$ ./eat_cpu& +$ uptime +10:14:42 up 86 days, 12:02, 11 users, load average: 2.60, 0.93, 0.58 +You can also read it in the htop command. +Fine-tuning your executable and/or job script +It is good practice to perform a number of run time stress tests, and to +check the system load of your nodes. We (and all other users of the HPC) +would appreciate that you use the maximum of the CPU resources that are +assigned to you and make sure that there are no CPUs in your node who +are not utilised without reasons. +But how can you maximise? +1. Profile your software to improve its performance. +2. Configure your software (e.g., to exactly use the available amount + of processors in a node). +3. Develop your parallel program in a smart way, so that it fully + utilises the available processors. +4. Demand a specific type of compute node (e.g., Harpertown, Westmere), + which have a specific number of cores. +5. Correct your request for CPUs in your job script. +And then check again. +Checking File sizes & Disk I/O +Monitoring File sizes during execution +Some programs generate intermediate or output files, the size of which +may also be a useful metric. +Remember that your available disk space on the HPC online storage is +limited, and that you have environment variables which point to these +directories available (i.e., *$VSC_DATA*, *$VSC_SCRATCH* and +*$VSC_DATA*). On top of those, you can also access some temporary +storage (i.e., the /tmp directory) on the compute node, which is defined +by the *$VSC_SCRATCH_NODE* environment variable. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_7_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_7_metadata.json new file mode 100644 index 00000000000..f5295cfe3ce --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_7_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Monitoring-File-sizes-during-execution", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 3, + "directory": "fine_tuning_job_specifications", + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_6", + "next_title": "fine_tuning_job_specifications_paragraph_8", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#monitoring-file-sizes-during-execution" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_8.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_8.txt new file mode 100644 index 00000000000..2d28c8b96a0 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_8.txt @@ -0,0 +1,33 @@ +It is important to be aware of the sizes of the file that will be +generated, as the available disk space for each user is limited. We +refer to section +How much disk space do I get? on Quotas to check your +quota and tools to find which files consumed the "quota". +Several actions can be taken, to avoid storage problems: +1. Be aware of all the files that are generated by your program. Also + check out the hidden files. +2. Check your quota consumption regularly. +3. Clean up your files regularly. +4. First work (i.e., read and write) with your big files in the local + /tmp directory. Once finished, you can move your files once to the + VSC_DATA directories. +5. Make sure your programs clean up their temporary files after + execution. +6. Move your output results to your own computer regularly. +7. Anyone can request more disk space to the HPC staff, but you will have + to duly justify your request. +Specifying network requirements +Users can examine their network activities with the htop command. When +your processors are 100% busy, but you see a lot of red bars and only +limited green bars in the htop screen, it is mostly an indication that +they lose a lot of time with inter-process communication. +Whenever your application utilises a lot of inter-process communication +(as is the case in most parallel programs), we strongly recommend to +request nodes with an "InfiniBand" network. The InfiniBand is a +specialised high bandwidth, low latency network that enables large +parallel jobs to run as efficiently as possible. +The parameter to add in your job script would be: +#PBS -l ib +If for some other reasons, a user is fine with the gigabit Ethernet +network, he can specify: +#PBS -l gbe diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_8_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_8_metadata.json new file mode 100644 index 00000000000..601614c5ccc --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/fine_tuning_job_specifications/fine_tuning_job_specifications_paragraph_8_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "fine_tuning_job_specifications", + "subtitle": "Specifying-network-requirements", + "source_file": "../../mkdocs/docs/HPC/fine_tuning_job_specifications.md", + "title_depth": 2, + "directory": "fine_tuning_job_specifications", + "links": { + "0": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#how-much-disk-space-do-i-get", + "1": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data/#quota" + }, + "parent_title": "", + "previous_title": "fine_tuning_job_specifications_paragraph_7", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/fine_tuning_job_specifications/#specifying-network-requirements" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_3.txt new file mode 100644 index 00000000000..57beab61703 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_3.txt @@ -0,0 +1,37 @@ +Our job script looks like this: +#!/bin/bash +module load TensorFlow/2.11.0-foss-2022a +python tensorflow_mnist.py +As you can see this job script will run the Python script named tensorflow_mnist.py. +The jobs you submit are per default executed on cluser/doduo, you can swap to another cluster by issuing the following command. +module swap cluster/donphan + Tip + + When submitting jobs with limited amount of resources, it is recommended to use the debug/interactive cluster: donphan. + To get a list of all clusters and their hardware, see https://www.ugent.be/hpc/en/infrastructure. +This job script can now be submitted to the cluster's job system for execution, using the qsub (queue submit) command: +$ qsub run.sh +123456 +This command returns a job identifier (123456) on the HPC cluster. This is a unique identifier for the job which can be used to monitor and manage your job. + Warning "Make sure you understand what the module command does" + + Note that the module commands only modify environment variables. For instance, running module swap cluster/donphan will update your shell environment so that qsub submits a job to the donphan cluster, + but our active shell session is still running on the login node. + + It is important to understand that while module commands affect your session environment, they do not change where the commands your are running are being executed: they will still be run on the login node you are on. + + When you submit a job script however, the commands in the job script will be run on a workernode of the cluster the job was submitted to (like donphan). +For detailed information about module commands, read the running batch jobs chapter. +Wait for job to be executed +Your job is put into a queue before being executed, so it may take a while before it actually starts. +(see when will my job start? for scheduling policy). +You can get an overview of the active jobs using the qstat command: +$ qstat +Job ID Name User Time Use S Queue +---------- ---------------- --------------- -------- - ------- +123456 run.sh vsc40000 0:00:00 Q donphan +Eventually, after entering qstat again you should see that your job has started running: +$ qstat +Job ID Name User Time Use S Queue +---------- ---------------- --------------- -------- - ------- +123456 run.sh vsc40000 0:00:01 R donphan diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_3_metadata.json new file mode 100644 index 00000000000..7b6b1c90dbd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_3_metadata.json @@ -0,0 +1,17 @@ +{ + "main_title": "getting_started", + "subtitle": "Wait-for-job-to-be-executed", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/interactive_debug/#interactive-and-debug-cluster", + "1": "https://docs.hpc.ugent.be/running_batch_jobs", + "2": "https://docs.hpc.ugent.be/running_batch_jobs/#when-will-my-job-start" + }, + "parent_title": "", + "previous_title": "getting_started_paragraph_2", + "next_title": "getting_started_paragraph_4", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/getting_started/#wait-for-job-to-be-executed" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_4.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_4.txt new file mode 100644 index 00000000000..24177a95b15 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_4.txt @@ -0,0 +1,32 @@ +If you don't see your job in the output of the qstat command anymore, your job has likely completed. +Read this section on how to interpret the output. +Inspect your results +When your job finishes it generates 2 output files: +- One for normal output messages (stdout output channel). +- One for warning and error messages (stderr output channel). +By default located in the directory where you issued qsub. + Info + For more information about the stdout and stderr output channels, see this section. +In our example when running ls in the current directory you should see 2 new files: + +- run.sh.o123456, containing normal output messages produced by job 123456; +- run.sh.e123456, containing errors and warnings produced by job 123456. + Info + + run.sh.e123456 should be empty (no errors or warnings). + Warning "Use your own job ID" + Replace 123456 with the jobid you got from the qstat command (see above) or simply look for added files in your current directory by running ls. +When examining the contents of run.sh.o123456 you will see something like this: +Downloading data from https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz +11493376/11490434 [==============================] - 1s 0us/step +Epoch 1/5 +1875/1875 [==============================] - 2s 823us/step - loss: 0.2960 - accuracy: 0.9133 +Epoch 2/5 +1875/1875 [==============================] - 1s 771us/step - loss: 0.1427 - accuracy: 0.9571 +Epoch 3/5 +1875/1875 [==============================] - 1s 767us/step - loss: 0.1070 - accuracy: 0.9675 +Epoch 4/5 +1875/1875 [==============================] - 1s 764us/step - loss: 0.0881 - accuracy: 0.9727 +Epoch 5/5 +1875/1875 [==============================] - 1s 764us/step - loss: 0.0741 - accuracy: 0.9768 +313/313 - 0s - loss: 0.0782 - accuracy: 0.9764 diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_4_metadata.json new file mode 100644 index 00000000000..22175edefcc --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_4_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "getting_started", + "subtitle": "Inspect-your-results", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/running_batch_jobs/#monitoring-and-managing-your-jobs", + "1": "https://docs.hpc.ugent.be/linux-tutorial/beyond_the_basics/#inputoutput" + }, + "parent_title": "", + "previous_title": "getting_started_paragraph_3", + "next_title": "getting_started_paragraph_5", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/getting_started/#inspect-your-results" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_5.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_5.txt new file mode 100644 index 00000000000..b8999963f1d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_5.txt @@ -0,0 +1,10 @@ +Hurray 🎉, we trained a deep learning model and achieved 97,64 percent accuracy. + Warning + When using TensorFlow specifically, you should actually submit jobs to a GPU cluster for better performance, see GPU clusters. + For the purpose of this example, we are running a very small TensorFlow workload on a CPU-only cluster. +Next steps +- Running interactive jobs +- Running jobs with input/output data +- Multi core jobs/Parallel Computing +- Interactive and debug cluster +For more examples see Program examples and Job script examples \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_5_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_5_metadata.json new file mode 100644 index 00000000000..fe8ab2642fd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/getting_started/getting_started_paragraph_5_metadata.json @@ -0,0 +1,21 @@ +{ + "main_title": "getting_started", + "subtitle": "Next-steps", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/gpu", + "1": "https://docs.hpc.ugent.be/running_interactive_jobs", + "2": "https://docs.hpc.ugent.be/running_jobs_with_input_output_data", + "3": "https://docs.hpc.ugent.be/multi_core_jobs", + "4": "https://docs.hpc.ugent.be/interactive_debug/#interactive-and-debug-cluster", + "5": "https://docs.hpc.ugent.be/program_examples", + "6": "https://docs.hpc.ugent.be/jobscript_examples" + }, + "parent_title": "", + "previous_title": "getting_started_paragraph_4", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/getting_started/#next-steps" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_1.txt new file mode 100644 index 00000000000..0a4b02b08dd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_1.txt @@ -0,0 +1,48 @@ +GPU clusters +Submitting jobs +To submit jobs to the joltik GPU cluster, where each node provides 4 +NVIDIA V100 GPUs (each with 32GB of GPU memory), use: +module swap cluster/joltik +To submit to the accelgor GPU cluster, where each node provides 4 +NVIDIA A100 GPUs (each with 80GB GPU memory), use: +module swap cluster/accelgor +Then use the familiar qsub, qstat, etc. commands, taking into +account the guidelines outlined in +section Requesting (GPU) resources. +Interactive jobs +To interactively experiment with GPUs, you can submit an interactive job +using qsub -I (and request one or more GPUs, see +section Requesting (GPU) resources). +Note that due to a bug in Slurm you will currently not be able to be +able to interactively use MPI software that requires access to the GPUs. +If you need this, please contact use via hpc@ugent.be. +Hardware +See https://www.ugent.be/hpc/en/infrastructure. +Requesting (GPU) resources +There are 2 main ways to ask for GPUs as part of a job: +- Either as a node property (similar to the number of cores per node + specified via ppn) using -l nodes=X:ppn=Y:gpus=Z (where the + ppn=Y is optional), or as a separate resource request (similar to + the amount of memory) via -l gpus=Z. Both notations give exactly + the same result. The -l gpus=Z is convenient if you only need one + node and you are fine with the default number of cores per GPU. The + -l nodes=...:gpus=Z notation is required if you want to run with + full control or in multinode cases like MPI jobs. If you do not + specify the number of GPUs by just using -l gpus, you get by + default 1 GPU. +- As a resource of its own, via --gpus X. In this case however, you + are not guaranteed that the GPUs are on the same node, so your + script or code must be able to deal with this. + diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_1_metadata.json new file mode 100644 index 00000000000..f426ff058e6 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_1_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "gpu", + "subtitle": "Requesting-(GPU)-resources", + "source_file": "../../mkdocs/docs/HPC/gpu.md", + "title_depth": 2, + "directory": "gpu", + "links": { + "0": "https://docs.hpc.ugent.be/gpu/#requesting-gpu-resources", + "1": "https://docs.hpc.ugent.be/gpu/#requesting-gpu-resources" + }, + "parent_title": "", + "previous_title": null, + "next_title": "gpu_paragraph_2", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/gpu/#requesting-gpu-resources" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_2.txt new file mode 100644 index 00000000000..240700d58b9 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_2.txt @@ -0,0 +1,35 @@ +Some background: +- The GPUs are constrained to the jobs (like the CPU cores), but do + not run in so-called "exclusive" mode. +- The GPUs run with the so-called "persistence daemon", so the GPUs is + not re-initialised between jobs. + + +Attention points +Some important attention points: +- For MPI jobs, we recommend the (new) wrapper mypmirun from the + vsc-mympirun module (pmi is the background mechanism to start + the MPI tasks, and is different from the usual mpirun that is used + by the mympirun wrapper). At some later point, we might promote + the mypmirun tool or rename it, to avoid the confusion in the + naming. +- Sharing GPUs requires MPS. The Slurm built-in MPS does not really do + want you want, so we will provide integration with mypmirun and + wurker. +- For parallel work, we are working on a wurker wrapper from the + vsc-mympirun module that supports GPU placement and MPS, without + any limitations wrt the requested resources (i.e. also support the + case where GPUs are spread heterogeneous over nodes from using the + --gpus Z option). +- Both mypmirun and wurker will try to do the most optimised + placement of cores and tasks, and will provide 1 (optimal) GPU per + task/MPI rank, and set one so-called visible device (i.e. + CUDA_VISIBLE_DEVICES only has 1 ID). The actual devices are not + constrained to the ranks, so you can access all devices requested in + the job. We know that at this moment, this is not working properly, but we are working on this. We advise against trying to fix this yourself. + + diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_2_metadata.json new file mode 100644 index 00000000000..b0d70feecff --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "gpu", + "subtitle": "Attention-points", + "source_file": "../../mkdocs/docs/HPC/gpu.md", + "title_depth": 2, + "directory": "gpu", + "parent_title": "", + "previous_title": "gpu_paragraph_1", + "next_title": "gpu_paragraph_3", + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/gpu/#attention-points" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_3.txt new file mode 100644 index 00000000000..1b92c087db3 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_3.txt @@ -0,0 +1,40 @@ +Software with GPU support +Use module avail to check for centrally installed software. +The subsections below only cover a couple of installed software +packages, more are available. +GROMACS +Please consult module avail GROMACS for a list of installed versions. +Horovod +Horovod can be used for (multi-node) multi-GPU TensorFlow/PyTorch +calculations. +Please consult module avail Horovod for a list of installed versions. +Horovod supports TensorFlow, Keras, PyTorch and MxNet (see +https://github.com/horovod/horovod#id9), but should be run as an MPI +application with mypmirun. (Horovod also provides its own wrapper +horovodrun, not sure if it handles placement and others correctly). +At least for simple TensorFlow benchmarks, it looks like Horovod is a +bit faster than usual autodetect multi-GPU TensorFlow without horovod, +but it comes at the cost of the code modifications to use horovod. + +PyTorch +Please consult module avail PyTorch for a list of installed versions. +TensorFlow +Please consult module avail TensorFlow for a list of installed +versions. +Note: for running TensorFlow calculations on multiple GPUs and/or on more than one workernode, use Horovod, see section Horovod. +Example TensorFlow job script +#!/bin/bash +#PBS -l walltime=5:0:0 +#PBS -l nodes=1:ppn=quarter:gpus=1 +module load TensorFlow/2.6.0-foss-2021a-CUDA-11.3.1 +cd $PBS_O_WORKDIR +python example.py + +AlphaFold +Please consult module avail AlphaFold for a list of installed +versions. +For more information on using AlphaFold, we strongly recommend the +VIB-UGent course available at +https://elearning.bits.vib.be/courses/alphafold. +Getting help +In case of questions or problems, please contact the HPC-UGent team via hpc@ugent.be, and clearly +indicate that your question relates to the joltik cluster by adding +[joltik] in the email subject. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_4_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_4_metadata.json new file mode 100644 index 00000000000..5a48827bb3b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/generic/gpu/gpu_paragraph_4_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "gpu", + "subtitle": "Getting-help", + "source_file": "../../mkdocs/docs/HPC/gpu.md", + "title_depth": 2, + "directory": "gpu", + "parent_title": "", + "previous_title": "gpu_paragraph_3", + "next_title": null, + "OS": "generic", + "reference_link": "https://docs.hpc.ugent.be/gpu/#getting-help" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_2.1.txt new file mode 100644 index 00000000000..3a282a73a15 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_2.1.txt @@ -0,0 +1,4 @@ +How do SSH keys work +Since all VSC clusters use Linux as their main operating system, you +will need to get acquainted with using the command-line interface and +using the terminal (see tutorial). \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..518f856303f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_2.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "How-do-SSH-keys-work", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial" + }, + "previous_title": "account_paragraph_1", + "next_title": "account_paragraph_3", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/account/#how-do-ssh-keys-work" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.1.txt new file mode 100644 index 00000000000..a48d7738003 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.1.txt @@ -0,0 +1,44 @@ +How do SSH keys work +Launch a terminal from your desktop's application menu and you will see +the bash shell. There are other shells, but most Linux distributions use +bash by default. +Test OpenSSH +Secure Shell (ssh) is a cryptographic network protocol for secure data +communication, remote command-line login, remote command execution, and +other secure network services between two networked computers. In short, +ssh provides a secure connection between 2 computers via insecure +channels (Network, Internet, telephone lines, ...). +"Secure" means that: +1. the User is authenticated to the System; and +2. the System is authenticated to the User; and +3. all data is encrypted during transfer. +OpenSSH is a FREE implementation of the SSH connectivity protocol. comes +with its own implementation of OpenSSH, so you don't need to install any +third-party software to use it. Just open a terminal window and jump in! +On all popular Linux distributions, the OpenSSH software is readily +available, and most often installed by default. You can check whether +the OpenSSH software is installed by opening a terminal and typing: +$ ssh -V +OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 +To access the clusters and transfer your files, you will use the +following commands: +1. ssh-keygen: to generate the SSH key pair (public + private key); +2. ssh: to open a shell on a remote machine; +3. sftp: a secure equivalent of ftp; +4. scp: a secure equivalent of the remote copy command rcp. +Generate a public/private key pair with OpenSSH +A key pair might already be present in the default location inside your +home directory. Therefore, we first check if a key is available with the +"list short" ("ls") command: +ls ~/.ssh +If a key-pair is already available, you would normally get: +authorized_keys id_rsa id_rsa.pub known_hosts +Otherwise, the command will show: +ls: .ssh: No such file or directory +You can recognise a public/private key pair when a pair of files has the +same name except for the extension ".pub" added to one of them. In this +particular case, the private key is "id_rsa" and public key is +"id_rsa.pub". You may have multiple keys (not necessarily in the +directory "~/.ssh") if you or your operating system requires this. Be +aware that your existing key pair might be too short, or not the right +type. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.1_metadata.json new file mode 100644 index 00000000000..d3ea56673f0 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Generate-a-publicprivate-key-pair-with-OpenSSH", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_paragraph_2", + "next_title": "account_linux_paragraph_3.2", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/account/#generate-a-publicprivate-key-pair-with-openssh" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.2.txt new file mode 100644 index 00000000000..ae6c282a95d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.2.txt @@ -0,0 +1,19 @@ +You will need to generate a new key pair, when: +1. you don't have a key pair yet +2. you forgot the passphrase protecting your private key +3. your private key was compromised +4. your key pair is too short or not the right type +For extra security, the private key itself can be encrypted using a +"passphrase", to prevent anyone from using your private key even when +they manage to copy it. You have to "unlock" the private key by typing +the passphrase. Be sure to never give away your private key, it is +private and should stay private. You should not even copy it to one of +your other machines, instead, you should create a new public/private key +pair for each machine. +ssh-keygen -t rsa -b 4096 +This will ask you for a file name to store the private and public key, +and a passphrase to protect your private key. It needs to be emphasised +that you really should choose the passphrase wisely! The system will ask +you for it every time you want to use the private key that is every time +you want to access the cluster or transfer your files. +Without your key pair, you won't be able to apply for a personal VSC account. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.2_metadata.json new file mode 100644 index 00000000000..118b5b02f2c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_3.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Generate-a-publicprivate-key-pair-with-OpenSSH", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_linux_paragraph_3.1", + "next_title": "account_paragraph_4", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/account/#generate-a-publicprivate-key-pair-with-openssh" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_4.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_4.1.txt new file mode 100644 index 00000000000..5072be0d303 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_4.1.txt @@ -0,0 +1,22 @@ +Using an SSH agent (optional) +Most recent Unix derivatives include by default an SSH agent ("gnome-keyring-daemon" in most cases) +to keep and manage the user SSH keys. If you use one of these derivatives you must include the new keys into +the SSH manager keyring to be able to connect to the HPC cluster. If +not, SSH client will display an error message (see Connecting) similar to this: +Agent admitted failure to sign using the key. +Permission denied (publickey,gssapi-keyex,gssapi-with-mic). +This could be fixed using the ssh-add command. You can include the new +private keys' identities in your keyring with: +ssh-add + tip + Without extra options ssh-add adds any key located at $HOME/.ssh + directory, but you can specify the private key location path as + argument, as example: ssh-add /path/to/my/id_rsa. +Check that your key is available from the keyring with: +ssh-add -l +After these changes the key agent will keep your SSH key to connect to +the clusters as usual. + tip + You should execute ssh-add command again if you generate a new SSH + key. +Visit https://wiki.gnome.org/Projects/GnomeKeyring/Ssh for more information. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_4.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_4.1_metadata.json new file mode 100644 index 00000000000..67ea12b9f7e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_4.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "Using-an-SSH-agent-(optional)", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/connecting" + }, + "previous_title": "account_paragraph_3", + "next_title": "account_paragraph_5", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/account/#using-an-ssh-agent-optional" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_5.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_5.1.txt new file mode 100644 index 00000000000..aa5284241c1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_5.1.txt @@ -0,0 +1,37 @@ +Welcome e mail +Applying for the account +Visit https://account.vscentrum.be/ +You will be redirected to our WAYF (Where Are You From) service where +you have to select your "Home Organisation". +Select "UGent" in the dropdown box and optionally select "Save my preference" +and "permanently". +Click "Confirm" +You will now be taken to the authentication page of your institute. +You will now have to log in with CAS using your UGent account. +You either have a login name of maximum 8 characters, or a (non-UGent) +email address if you are an external user. In case of problems with your +UGent password, please visit: https://password.ugent.be/. After +logging in, you may be requested to share your information. Click "Yes, +continue". +After you log in using your UGent login and password, you will be asked to +upload the file that contains your public key, i.e., the file +"id_rsa.pub" which you have generated earlier. Make sure that your +public key is actually accepted for upload, because if it is in a wrong +format, wrong type or too short, then it will be refused. +This file has been stored in the directory "~/.ssh/". +After you have uploaded your public key you will receive an e-mail with +a link to confirm your e-mail address. After confirming your e-mail +address the VSC staff will review and if applicable approve your +account. +Welcome e-mail +Within one day, you should receive a Welcome e-mail with your VSC +account details. +Dear (Username), +Your VSC-account has been approved by an administrator. +Your vsc-username is vsc40000 +Your account should be fully active within one hour. +To check or update your account information please visit +https://account.vscentrum.be/ +For further info please visit https://www.vscentrum.be/user-portal +Kind regards, +-- The VSC administrators \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_5.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_5.1_metadata.json new file mode 100644 index 00000000000..3cd42cf6af1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_5.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Welcome-e-mail", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_paragraph_4", + "next_title": "account_paragraph_6", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/account/#welcome-e-mail" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_6.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_6.1.txt new file mode 100644 index 00000000000..204ca799a10 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_6.1.txt @@ -0,0 +1,34 @@ +Computation Workflow on the HPC +Now, you can start using the HPC. You can always look up your VSC id later +by visiting https://account.vscentrum.be. +Adding multiple SSH public keys (optional) +In case you are connecting from different computers to the login nodes, +it is advised to use separate SSH public keys to do so. You should +follow these steps. +1. Create a new public/private SSH key pair from the new computer. + Repeat the process described in + section Generate a public/private key pair with OpenSSH. +2. Go to https://account.vscentrum.be/django/account/edit +3. Upload the new SSH public key using the Add public key section. Make sure that your + public key is actually saved, because a public key will be refused + if it is too short, wrong type, or in a wrong format. +4. (optional) If you lost your key, you can delete the old key on the + same page. You should keep at least one valid public SSH key in your + account. +5. Take into account that it will take some time before the new SSH + public key is active in your account on the system; waiting for + 15-30 minutes should be sufficient. +Computation Workflow on the HPC +A typical Computation workflow will be: +1. Connect to the HPC +2. Transfer your files to the HPC +3. Compile your code and test it +4. Create a job script +5. Submit your job +6. Wait while + 1. your job gets into the queue + 2. your job gets executed + 3. your job finishes +7. Move your results +We'll take you through the different tasks one by one in the following +chapters. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_6.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_6.1_metadata.json new file mode 100644 index 00000000000..e0293f7bc86 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/account/account_linux_paragraph_6.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "Computation-Workflow-on-the-HPC", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 2, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/account/#generate-a-publicprivate-key-pair-with-openssh" + }, + "previous_title": "account_paragraph_5", + "next_title": null, + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/account/#computation-workflow-on-the-hpc" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_2.1.txt new file mode 100644 index 00000000000..773d03f0689 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_2.1.txt @@ -0,0 +1,7 @@ +First Time connection to the HPC infrastructure +ssh_exchange_identification: read: Connection reset by peer +First Time connection to the HPC infrastructure +The remaining content in this chapter is primarily focused for people utilizing a terminal with SSH. +If you are instead using the web portal, the corresponding chapter might be more helpful: Using the HPC-UGent web portal. +If you have any issues connecting to the HPC after you've followed these +steps, see Issues connecting to login node to troubleshoot. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..a4f07157426 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_2.1_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "links": { + "0": "https://docs.hpc.ugent.be/web_portal", + "1": "https://docs.hpc.ugent.be/troubleshooting/#issues-connecting-to-login-node" + }, + "previous_title": "connecting_paragraph_1", + "next_title": "connecting_paragraph_3", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_3.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_3.1.txt new file mode 100644 index 00000000000..edcf3dcf60d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_3.1.txt @@ -0,0 +1,47 @@ +First Time connection to the HPC infrastructure +Connect +Open up a terminal and enter the following command to connect to the HPC. +ssh vsc40000@login.hpc.ugent.be +Here, user vsc40000 wants to make a connection to the "hpcugent" cluster at UGent via the login +node "login.hpc.ugent.be", so replace vsc40000 with your own VSC id in the above command. +The first time you make a connection to the login node, you will be +asked to verify the authenticity of the login node. Please check +Warning message when first connecting to new host on how to do this. +A possible error message you can get if you previously saved your +private key somewhere else than the default location +($HOME/.ssh/id_rsa): +Permission denied (publickey,gssapi-keyex,gssapi-with-mic). +In this case, use the -i option for the ssh command to specify the +location of your private key. For example: +ssh -i /home/example/my_keys +Congratulations, you're on the HPC infrastructure now! +To find out where you have landed you can print the current working directory: +$ pwd +/user/home/gent/vsc400/vsc40000 +Your new private home directory is "/user/home/gent/vsc400/vsc40000". Here you can create your own +subdirectory structure, copy and prepare your applications, compile and +test them and submit your jobs on the HPC. +$ cd /apps/gent/tutorials +$ ls +Intro-HPC/ +This directory currently contains all training material for the Introduction to the HPC. More +relevant training material to work with the HPC can always be added later in +this directory. +You can now explore the content of this directory with the "ls --l" (lists long) and the "cd" (change directory) commands: +As we are interested in the use of the HPC, move further to Intro-HPC and explore the +contents up to 2 levels deep: +$ cd Intro-HPC +$ tree -L 2 +. +'-- examples + |-- Compiling-and-testing-your-software-on-the-HPC + |-- Fine-tuning-Job-Specifications + |-- Multi-core-jobs-Parallel-Computing + |-- Multi-job-submission + |-- Program-examples + |-- Running-batch-jobs + |-- Running-jobs-with-input + |-- Running-jobs-with-input-output-data + |-- example.pbs + '-- example.sh +9 directories, 5 files \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_3.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_3.1_metadata.json new file mode 100644 index 00000000000..4d2bbae3565 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_3.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Connect", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "First-Time-connection-to-the-HPC-infrastructure", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#warning-message-when-first-connecting-to-new-host" + }, + "previous_title": "connecting_paragraph_2", + "next_title": "connecting_paragraph_4", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#connect" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_5.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_5.1.txt new file mode 100644 index 00000000000..8c2bd12d342 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_5.1.txt @@ -0,0 +1,61 @@ +First Time connection to the HPC infrastructure +You can exit the connection at anytime by entering: +$ exit +logout +Connection to login.hpc.ugent.be closed. + tip "tip: Setting your Language right" + You may encounter a warning message similar to the following one during connecting: + perl: warning: Setting locale failed. + perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LANG = (unset) + are supported and installed on your system. + perl: warning: Falling back to the standard locale ("C"). + or any other error message complaining about the locale. + This means that the correct "locale" has not yet been properly specified on your local machine. Try: + LANG= + LC_COLLATE="C" + LC_CTYPE="UTF-8" + LC_MESSAGES="C" + LC_MONETARY="C" + LC_NUMERIC="C" + LC_TIME="C" + LC_ALL= + A locale is a set of parameters that defines the user's language, country and + any special variant preferences that the user wants to see in their user + interface. Usually a locale identifier consists of at least a language + identifier and a region identifier. + Note + If you try to set a non-supported locale, then it will be automatically + set to the default. Currently the default is en_US.UFT-8 or en_US, + depending on whether your originally (non-supported) locale was UTF-8 or not. + Open the .bashrc on your local machine with your favourite editor and + add the following lines: + + $ nano ~/.bashrc + ... + export LANGUAGE="en_US.UTF-8" + export LC_ALL="en_US.UTF-8" + export LC_CTYPE="en_US.UTF-8" + export LANG="en_US.UTF-8" + ... + + tip "tip: vi" + To start entering text in vi: move to the place you want to start + entering text with the arrow keys and type "i" to switch to insert mode. You can easily exit vi by entering: ""ESC" :wq" + To exit vi without saving your changes, enter ""ESC":q!" + + + or alternatively (if you are not comfortable with the Linux editors), + again on your local machine: + + echo "export LANGUAGE=\"en_US.UTF-8\"" >> ~/.profile + echo "export LC_ALL=\"en_US.UTF-8\"" >> ~/.profile + echo "export LC_CTYPE=\"en_US.UTF-8\"" >> ~/.profile + echo "export LANG=\"en_US.UTF-8\"" >> ~/.profile + + You can now log out, open a new terminal/shell on your local machine and + reconnect to the login node, and you should not get these warnings anymore. + \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_5.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_5.1_metadata.json new file mode 100644 index 00000000000..8f814f7b838 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_5.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "previous_title": "connecting_paragraph_4", + "next_title": "connecting_paragraph_6", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_6.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_6.1.txt new file mode 100644 index 00000000000..d872c89a0f8 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_6.1.txt @@ -0,0 +1,6 @@ +Transfer Files to/from the HPC +Before you can do some work, you'll have to transfer the files you need from your desktop or department to the cluster. At the end of a job, you might want to transfer some files back. +The preferred way to transfer files is by using an scp or sftp via the +secure OpenSSH protocol. ships with an implementation of OpenSSH, so you +don't need to install any third-party software to use it. Just open a +terminal window and jump in! diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_6.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_6.1_metadata.json new file mode 100644 index 00000000000..43070f5ad83 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_6.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "Transfer-Files-tofrom-the-HPC", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "previous_title": "connecting_paragraph_5", + "next_title": "connecting_paragraph_7", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#transfer-files-tofrom-the-hpc" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.1.txt new file mode 100644 index 00000000000..319433fd88a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.1.txt @@ -0,0 +1,42 @@ +Transfer Files tofrom the HPC +Using scp +Secure copy or SCP is a tool (command) for securely transferring files between a local +host (= your computer) and a remote host (the HPC). It is based on the +Secure Shell (SSH) protocol. The scp command is the equivalent of the cp (i.e., +copy) command, but can copy files to or from remote machines. +It's easier to copy files directly to $VSC_DATA and $VSC_SCRATCH if +you have symlinks to them in your home directory. See +the chapter titled "Uploading/downloading/editing files", section "Symlinks for data/scratch" in the intro to Linux + for how to do this. +Open an additional terminal window and check that you're working on your +local machine. +$ hostname + +If you're still using the terminal that is connected to the HPC, close the +connection by typing "exit" in the terminal window. +For example, we will copy the (local) file "localfile.txt" to your +home directory on the HPC cluster. We first generate a small dummy +"localfile.txt", which contains the word "Hello". Use your own VSC +account, which is something like "vsc40000". Don't forget the colon (:) at the +end: if you forget it, it will just create a file named vsc40000@login.hpc.ugent.be on your +local filesystem. You can even specify where to save the file on the +remote filesystem by putting a path after the colon. +$ echo "Hello" > localfile.txt +$ ls -l +... +-rw-r--r-- 1 user staff 6 Sep 18 09:37 localfile.txt +$ scp localfile.txt vsc40000@login.hpc.ugent.be: +localfile.txt 100% 6 0.0KB/s 00:00 +Connect to the HPC via another terminal, print the working directory (to +make sure you're in the home directory) and check whether the file has +arrived: +$ pwd +/user/home/gent/vsc400/vsc40000 +$ ls -l +total 1536 +drwxrwxr-x 2 +drwxrwxr-x 2 +drwxrwxr-x 10 +-rw-r--r-- 1 +$ cat localfile.txt +Hello diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.1_metadata.json new file mode 100644 index 00000000000..5903f61e437 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Using-scp", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "links": { + "0": "https://docs.hpc.ugent.be/localhost:8000/Gent//intro-Linux/uploading_files/#symlinks-for-datascratch" + }, + "previous_title": "connecting_paragraph_6", + "next_title": "connecting_linux_paragraph_7.2", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#using-scp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.2.txt new file mode 100644 index 00000000000..16016dc219c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.2.txt @@ -0,0 +1,37 @@ +The scp command can also be used to copy files from the cluster to your +local machine. Let us copy the remote file "intro-HPC--Gent.pdf" from your "docs" +subdirectory on the cluster to your local computer. +First, we will confirm that the file is indeed in the "docs" +subdirectory. In the terminal on the login node, enter: +$ cd ~/docs +$ ls -l +total 1536 +-rw-r--r-- 1 vsc40000 Sep 11 09:53 intro-HPC--Gent.pdf +Now we will copy the file to the local machine. On the terminal on your +own local computer, enter: +$ scp vsc40000@login.hpc.ugent.be:./docs/intro-HPC--Gent.pdf . +intro-HPC--Gent.pdf 100% 725KB 724.6KB/s 00:01 +$ ls -l +total 899 +-rw-r--r-- 1 user staff 741995 Sep 18 09:53 +-rw-r--r-- 1 user staff 6 Sep 18 09:37 localfile.txt +The file has been copied from the HPC to your local computer. +It's also possible to copy entire directories (and their contents) with +the -r flag. For example, if we want to copy the local directory +dataset to $VSC_SCRATCH, we can use the following command (assuming +you've created the scratch symlink): +scp -r dataset vsc40000@login.hpc.ugent.be:scratch +If you don't use the -r option to copy a directory, you will run into +the following error: +$ scp dataset vsc40000@login.hpc.ugent.be:scratch +dataset: not a regular file +Using sftp +The SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP) is a network protocol that provides file access, file +transfer and file management functionalities over any reliable data +stream. It was designed as an extension of the Secure Shell protocol +(SSH) version 2.0. This protocol assumes that it is run over a secure +channel, such as SSH, that the server has already authenticated the +client, and that the identity of the client user is available to the +protocol. +The sftp is an equivalent of the ftp command, with the difference that +it uses the secure ssh protocol to connect to the clusters. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.2_metadata.json new file mode 100644 index 00000000000..a4cc992a338 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "Using-sftp", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "previous_title": "connecting_linux_paragraph_7.1", + "next_title": "connecting_linux_paragraph_7.3", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#using-sftp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.3.txt new file mode 100644 index 00000000000..1ef13b80c6f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.3.txt @@ -0,0 +1,18 @@ +One easy way of starting a sftp session is +sftp vsc40000@login.hpc.ugent.be +Typical and popular commands inside an sftp session are: +| | | +|:--------------------------|:-------------------------------------------------------------------------------------| +| cd ~/exmples/fibo | Move to the examples/fibo subdirectory on the (i.e., the HPC remote machine) | +| ls | Get a list of the files in the current directory on the HPC. | +| get fibo.py | Copy the file "fibo.py" from the HPC | +| get tutorial/HPC.pdf | Copy the file "HPC.pdf" from the HPC, which is in the "tutorial" subdirectory. | +| lcd test | Move to the "test" subdirectory on your local machine. | +| lcd .. | Move up one level in the local directory. | +| lls | Get local directory listing. | +| put test.py | Copy the local file test.py to the HPC. | +| put test1.py test2.py | Copy the local file test1.py to the and rename it to test2.py. | +| bye | Quit the sftp session | +| **mget *.cc** | Copy all the remote files with extension ".cc" to the local directory. | +| **mput *.h** | Copy all the local files with extension ".h" to the HPC. | +| | | diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.3_metadata.json new file mode 100644 index 00000000000..eb6c58af4de --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_7.3_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "connecting", + "subtitle": "Using-sftp", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "links": { + "0": "https://docs.hpc.ugent.be/", + "1": "https://docs.hpc.ugent.be/" + }, + "previous_title": "connecting_linux_paragraph_7.2", + "next_title": "connecting_paragraph_8", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#using-sftp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_8.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_8.1.txt new file mode 100644 index 00000000000..4ad49c5ff63 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_8.1.txt @@ -0,0 +1,17 @@ +Changing login nodes +Using a GUI +If you prefer a GUI to transfer files back and forth to the HPC, you can +use your file browser. Open your file browser and press +++"Ctrl"+"l"++ +This should open up a address bar where you can enter a URL. +Alternatively, look for the "connect to server" option in your file +browsers menu. +Enter: sftp://vsc40000@login.hpc.ugent.be/ and press enter. +You should now be able to browse files on the HPC in your file browser. +Fast file transfer for large datasets +See the section on rsync in chapter 5 of the Linux intro manual. +Changing login nodes +It can be useful to have control over which login node you are on. However, when you connect to the HPC (High-Performance Computing) system, you are directed to a random login node, which might not be the one where you already have an active session. To address this, there is a way to manually switch your active login node. +For instance, if you want to switch to the login node named gligar07.gastly.os, you can use the following command while you are connected to the gligar08.gastly.os login node on the HPC: +ssh gligar07.gastly.os +This is also possible the other way around. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_8.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_8.1_metadata.json new file mode 100644 index 00000000000..3bd72528191 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/connecting/connecting_linux_paragraph_8.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Using-a-GUI", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial/uploading_files/#copying-faster-with-rsync" + }, + "previous_title": "connecting_paragraph_7", + "next_title": "connecting_paragraph_9", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/connecting/#using-a-gui" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.1.txt new file mode 100644 index 00000000000..a4ec1c8370f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.1.txt @@ -0,0 +1,30 @@ +Getting Connected +Getting Started +Welcome to the "Getting Started" guide. This chapter will lead you through the initial steps of logging into the HPC-UGent infrastructure and submitting your very first job. We'll also walk you through the process step by step using a practical example. +In addition to this chapter, you might find the recording of the Introduction to HPC-UGent training session to be a useful resource. +Before proceeding, read the introduction to HPC to gain an understanding of the HPC-UGent infrastructure and related terminology. +Getting Access +To get access to the HPC-UGent infrastructure, visit Getting an HPC Account. +If you have not used Linux before, +now would be a good time to follow our Linux Tutorial. +A typical workflow looks like this: +1. Connect to the login nodes +2. Transfer your files to the HPC-UGent infrastructure +3. Optional: compile your code and test it +4. Create a job script and submit your job +5. Wait for job to be executed +6. Study the results generated by your jobs, either on the cluster or + after downloading them locally. +We will walk through an illustrative workload to get you started. In this example, our objective is to train a deep learning model for recognizing hand-written digits (MNIST dataset) using TensorFlow; +see the example scripts. +Getting Connected +There are two options to connect +- Using a terminal to connect via SSH (for power users) (see First Time connection to the HPC-UGent infrastructure) +- Using the web portal +Considering your operating system is *, +it is recommended to make use of the ssh command in a terminal to get the most flexibility. +Assuming you have already generated SSH keys in the previous step (Getting Access), and that they are in a default location, you should now be able to login by running the following command: +ssh vsc40000@login.hpc.ugent.be + Warning "User your own VSC account id" + + Replace vsc40000 with your VSC account id (see https://account.vscentrum.be) diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.1_metadata.json new file mode 100644 index 00000000000..02eec654266 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.1_metadata.json @@ -0,0 +1,23 @@ +{ + "main_title": "getting_started", + "subtitle": "Getting-Connected", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://www.ugent.be/hpc/en/training/introhpcugent-recording", + "1": "https://docs.hpc.ugent.be/introduction", + "2": "https://docs.hpc.ugent.be/account", + "3": "https://docs.hpc.ugent.be/linux-tutorial", + "4": "https://www.tensorflow.org/", + "5": "https://github.com/hpcugent/vsc_user_docs/tree/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist", + "6": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure", + "7": "https://docs.hpc.ugent.be/web_portal", + "8": "https://docs.hpc.ugent.be/getting_started/#getting-access" + }, + "previous_title": null, + "next_title": "getting_started_linux_paragraph_1.2", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/getting_started/#getting-connected" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.2.txt new file mode 100644 index 00000000000..d068845643c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.2.txt @@ -0,0 +1,2 @@ + Tip + You can also still use the web portal (see shell access on web portal) \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.2_metadata.json new file mode 100644 index 00000000000..d6309137812 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_1.2_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "getting_started", + "subtitle": "Getting-Connected", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/web_portal/#shell-access" + }, + "previous_title": "getting_started_linux_paragraph_1.1", + "next_title": "getting_started_paragraph_2", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/getting_started/#getting-connected" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_2.1.txt new file mode 100644 index 00000000000..438321b40a1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_2.1.txt @@ -0,0 +1,27 @@ +Submitting a job + Info + When having problems see the connection issues section on the troubleshooting page. +Transfer your files +Now that you can login, it is time to transfer files from your local computer to your home directory on the HPC-UGent infrastructure. +Download tensorflow_mnist.py +and run.sh example scripts to your computer (from here). +On your local machine you can run: +curl -OL https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/tensorflow_mnist.py +curl -OL https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/run.sh +Using the scp command, the files can be copied from your local host to your home directory (~) on the remote host (HPC). +scp tensorflow_mnist.py run.sh vsc40000login.hpc.ugent.be:~ +ssh vsc40000@login.hpc.ugent.be + Warning "User your own VSC account id" + + Replace vsc40000 with your VSC account id (see https://account.vscentrum.be) + Info + For more information about transfering files or scp, see tranfer files from/to hpc. +When running ls in your session on the HPC-UGent infrastructure, you should see the two files listed in your home directory (~): +$ ls ~ +run.sh tensorflow_mnist.py +When you do not see these files, make sure you uploaded the files to your home directory. +Submitting a job +Jobs are submitted and executed using job scripts. In our case run.sh can be used as a (very minimal) job script. +A job script is a shell script, a text file that specifies the resources, +the software that is used (via module load statements), +and the steps that should be executed to run the calculation. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..e9d9e24d26a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/linux/getting_started/getting_started_linux_paragraph_2.1_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "getting_started", + "subtitle": "Submitting-a-job", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#sec:connecting-issues", + "1": "https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/tensorflow_mnist.py", + "2": "https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/run.sh", + "3": "https://github.com/hpcugent/vsc_user_docs/tree/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist", + "4": "https://docs.hpc.ugent.be/connecting/#transfer-files-tofrom-the-hpc" + }, + "previous_title": "getting_started_paragraph_1", + "next_title": "getting_started_paragraph_3", + "OS": "linux", + "reference_link": "https://docs.hpc.ugent.be/Linux/getting_started/#submitting-a-job" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_2.1.txt new file mode 100644 index 00000000000..3a282a73a15 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_2.1.txt @@ -0,0 +1,4 @@ +How do SSH keys work +Since all VSC clusters use Linux as their main operating system, you +will need to get acquainted with using the command-line interface and +using the terminal (see tutorial). \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..d20916a48c8 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_2.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "How-do-SSH-keys-work", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial" + }, + "previous_title": "account_paragraph_1", + "next_title": "account_paragraph_3", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/account/#how-do-ssh-keys-work" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.1.txt new file mode 100644 index 00000000000..27bb9531822 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.1.txt @@ -0,0 +1,45 @@ +How do SSH keys work +To open a Terminal window in macOS, open the Finder and choose +*\>\> Applications \> Utilities \> Terminal* +Before requesting an account, you need to generate a pair of ssh keys. +One popular way to do this on is using the OpenSSH client included with , which you can then also use to log on to the clusters. +Test OpenSSH +Secure Shell (ssh) is a cryptographic network protocol for secure data +communication, remote command-line login, remote command execution, and +other secure network services between two networked computers. In short, +ssh provides a secure connection between 2 computers via insecure +channels (Network, Internet, telephone lines, ...). +"Secure" means that: +1. the User is authenticated to the System; and +2. the System is authenticated to the User; and +3. all data is encrypted during transfer. +OpenSSH is a FREE implementation of the SSH connectivity protocol. comes +with its own implementation of OpenSSH, so you don't need to install any +third-party software to use it. Just open a terminal window and jump in! +On all popular Linux distributions, the OpenSSH software is readily +available, and most often installed by default. You can check whether +the OpenSSH software is installed by opening a terminal and typing: +$ ssh -V +OpenSSH_7.4p1, OpenSSL 1.0.2k-fips 26 Jan 2017 +To access the clusters and transfer your files, you will use the +following commands: +1. ssh-keygen: to generate the SSH key pair (public + private key); +2. ssh: to open a shell on a remote machine; +3. sftp: a secure equivalent of ftp; +4. scp: a secure equivalent of the remote copy command rcp. +Generate a public/private key pair with OpenSSH +A key pair might already be present in the default location inside your +home directory. Therefore, we first check if a key is available with the +"list short" ("ls") command: +ls ~/.ssh +If a key-pair is already available, you would normally get: +authorized_keys id_rsa id_rsa.pub known_hosts +Otherwise, the command will show: +ls: .ssh: No such file or directory +You can recognise a public/private key pair when a pair of files has the +same name except for the extension ".pub" added to one of them. In this +particular case, the private key is "id_rsa" and public key is +"id_rsa.pub". You may have multiple keys (not necessarily in the +directory "~/.ssh") if you or your operating system requires this. Be +aware that your existing key pair might be too short, or not the right +type. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.1_metadata.json new file mode 100644 index 00000000000..7ecf0f36e1e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Generate-a-publicprivate-key-pair-with-OpenSSH", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_paragraph_2", + "next_title": "account_macos_paragraph_3.2", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/account/#generate-a-publicprivate-key-pair-with-openssh" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.2.txt new file mode 100644 index 00000000000..ae6c282a95d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.2.txt @@ -0,0 +1,19 @@ +You will need to generate a new key pair, when: +1. you don't have a key pair yet +2. you forgot the passphrase protecting your private key +3. your private key was compromised +4. your key pair is too short or not the right type +For extra security, the private key itself can be encrypted using a +"passphrase", to prevent anyone from using your private key even when +they manage to copy it. You have to "unlock" the private key by typing +the passphrase. Be sure to never give away your private key, it is +private and should stay private. You should not even copy it to one of +your other machines, instead, you should create a new public/private key +pair for each machine. +ssh-keygen -t rsa -b 4096 +This will ask you for a file name to store the private and public key, +and a passphrase to protect your private key. It needs to be emphasised +that you really should choose the passphrase wisely! The system will ask +you for it every time you want to use the private key that is every time +you want to access the cluster or transfer your files. +Without your key pair, you won't be able to apply for a personal VSC account. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.2_metadata.json new file mode 100644 index 00000000000..8509ef75bf3 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_3.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Generate-a-publicprivate-key-pair-with-OpenSSH", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_macos_paragraph_3.1", + "next_title": "account_paragraph_4", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/account/#generate-a-publicprivate-key-pair-with-openssh" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_4.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_4.1.txt new file mode 100644 index 00000000000..19d475c56ca --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_4.1.txt @@ -0,0 +1,21 @@ +Using an SSH agent (optional) +Most recent Unix derivatives include by default an SSH agent +to keep and manage the user SSH keys. If you use one of these derivatives you must include the new keys into +the SSH manager keyring to be able to connect to the HPC cluster. If +not, SSH client will display an error message (see Connecting) similar to this: +Agent admitted failure to sign using the key. +Permission denied (publickey,gssapi-keyex,gssapi-with-mic). +This could be fixed using the ssh-add command. You can include the new +private keys' identities in your keyring with: +ssh-add + tip + Without extra options ssh-add adds any key located at $HOME/.ssh + directory, but you can specify the private key location path as + argument, as example: ssh-add /path/to/my/id_rsa. +Check that your key is available from the keyring with: +ssh-add -l +After these changes the key agent will keep your SSH key to connect to +the clusters as usual. + tip + You should execute ssh-add command again if you generate a new SSH + key. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_4.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_4.1_metadata.json new file mode 100644 index 00000000000..8a289927685 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_4.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "Using-an-SSH-agent-(optional)", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/connecting" + }, + "previous_title": "account_paragraph_3", + "next_title": "account_paragraph_5", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/account/#using-an-ssh-agent-optional" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_5.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_5.1.txt new file mode 100644 index 00000000000..2dd8fceab42 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_5.1.txt @@ -0,0 +1,42 @@ +Welcome e mail +Applying for the account +Visit https://account.vscentrum.be/ +You will be redirected to our WAYF (Where Are You From) service where +you have to select your "Home Organisation". +Select "UGent" in the dropdown box and optionally select "Save my preference" +and "permanently". +Click "Confirm" +You will now be taken to the authentication page of your institute. +You will now have to log in with CAS using your UGent account. +You either have a login name of maximum 8 characters, or a (non-UGent) +email address if you are an external user. In case of problems with your +UGent password, please visit: https://password.ugent.be/. After +logging in, you may be requested to share your information. Click "Yes, +continue". +After you log in using your UGent login and password, you will be asked to +upload the file that contains your public key, i.e., the file +"id_rsa.pub" which you have generated earlier. Make sure that your +public key is actually accepted for upload, because if it is in a wrong +format, wrong type or too short, then it will be refused. +This file has been stored in the directory "~/.ssh/". + tip + As ".ssh" is an invisible directory, the Finder will not show it by + default. The easiest way to access the folder, is by pressing ++cmd+shift+g++ (or ++cmd+shift+"."++), + which will allow you to enter the name of a directory, which you would + like to open in Finder. Here, type "~/.ssh" and press enter. +After you have uploaded your public key you will receive an e-mail with +a link to confirm your e-mail address. After confirming your e-mail +address the VSC staff will review and if applicable approve your +account. +Welcome e-mail +Within one day, you should receive a Welcome e-mail with your VSC +account details. +Dear (Username), +Your VSC-account has been approved by an administrator. +Your vsc-username is vsc40000 +Your account should be fully active within one hour. +To check or update your account information please visit +https://account.vscentrum.be/ +For further info please visit https://www.vscentrum.be/user-portal +Kind regards, +-- The VSC administrators \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_5.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_5.1_metadata.json new file mode 100644 index 00000000000..7882c7ff73f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_5.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Welcome-e-mail", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_paragraph_4", + "next_title": "account_paragraph_6", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/account/#welcome-e-mail" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_6.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_6.1.txt new file mode 100644 index 00000000000..204ca799a10 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_6.1.txt @@ -0,0 +1,34 @@ +Computation Workflow on the HPC +Now, you can start using the HPC. You can always look up your VSC id later +by visiting https://account.vscentrum.be. +Adding multiple SSH public keys (optional) +In case you are connecting from different computers to the login nodes, +it is advised to use separate SSH public keys to do so. You should +follow these steps. +1. Create a new public/private SSH key pair from the new computer. + Repeat the process described in + section Generate a public/private key pair with OpenSSH. +2. Go to https://account.vscentrum.be/django/account/edit +3. Upload the new SSH public key using the Add public key section. Make sure that your + public key is actually saved, because a public key will be refused + if it is too short, wrong type, or in a wrong format. +4. (optional) If you lost your key, you can delete the old key on the + same page. You should keep at least one valid public SSH key in your + account. +5. Take into account that it will take some time before the new SSH + public key is active in your account on the system; waiting for + 15-30 minutes should be sufficient. +Computation Workflow on the HPC +A typical Computation workflow will be: +1. Connect to the HPC +2. Transfer your files to the HPC +3. Compile your code and test it +4. Create a job script +5. Submit your job +6. Wait while + 1. your job gets into the queue + 2. your job gets executed + 3. your job finishes +7. Move your results +We'll take you through the different tasks one by one in the following +chapters. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_6.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_6.1_metadata.json new file mode 100644 index 00000000000..96a0afcdcf6 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/account/account_macos_paragraph_6.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "Computation-Workflow-on-the-HPC", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 2, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/account/#generate-a-publicprivate-key-pair-with-openssh" + }, + "previous_title": "account_paragraph_5", + "next_title": null, + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/account/#computation-workflow-on-the-hpc" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_2.1.txt new file mode 100644 index 00000000000..773d03f0689 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_2.1.txt @@ -0,0 +1,7 @@ +First Time connection to the HPC infrastructure +ssh_exchange_identification: read: Connection reset by peer +First Time connection to the HPC infrastructure +The remaining content in this chapter is primarily focused for people utilizing a terminal with SSH. +If you are instead using the web portal, the corresponding chapter might be more helpful: Using the HPC-UGent web portal. +If you have any issues connecting to the HPC after you've followed these +steps, see Issues connecting to login node to troubleshoot. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..d1e758b8938 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_2.1_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "links": { + "0": "https://docs.hpc.ugent.be/web_portal", + "1": "https://docs.hpc.ugent.be/troubleshooting/#issues-connecting-to-login-node" + }, + "previous_title": "connecting_paragraph_1", + "next_title": "connecting_paragraph_3", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_3.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_3.1.txt new file mode 100644 index 00000000000..3afe2472575 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_3.1.txt @@ -0,0 +1,48 @@ +First Time connection to the HPC infrastructure +Connect +Open up a terminal and enter the following command to connect to the HPC. +You can open a terminal by navigation to Applications and then Utilities in the finder and open Terminal.app, or enter Terminal in Spotlight Search. +ssh vsc40000@login.hpc.ugent.be +Here, user vsc40000 wants to make a connection to the "hpcugent" cluster at UGent via the login +node "login.hpc.ugent.be", so replace vsc40000 with your own VSC id in the above command. +The first time you make a connection to the login node, you will be +asked to verify the authenticity of the login node. Please check +Warning message when first connecting to new host on how to do this. +A possible error message you can get if you previously saved your +private key somewhere else than the default location +($HOME/.ssh/id_rsa): +Permission denied (publickey,gssapi-keyex,gssapi-with-mic). +In this case, use the -i option for the ssh command to specify the +location of your private key. For example: +ssh -i /home/example/my_keys +Congratulations, you're on the HPC infrastructure now! +To find out where you have landed you can print the current working directory: +$ pwd +/user/home/gent/vsc400/vsc40000 +Your new private home directory is "/user/home/gent/vsc400/vsc40000". Here you can create your own +subdirectory structure, copy and prepare your applications, compile and +test them and submit your jobs on the HPC. +$ cd /apps/gent/tutorials +$ ls +Intro-HPC/ +This directory currently contains all training material for the Introduction to the HPC. More +relevant training material to work with the HPC can always be added later in +this directory. +You can now explore the content of this directory with the "ls --l" (lists long) and the "cd" (change directory) commands: +As we are interested in the use of the HPC, move further to Intro-HPC and explore the +contents up to 2 levels deep: +$ cd Intro-HPC +$ tree -L 2 +. +'-- examples + |-- Compiling-and-testing-your-software-on-the-HPC + |-- Fine-tuning-Job-Specifications + |-- Multi-core-jobs-Parallel-Computing + |-- Multi-job-submission + |-- Program-examples + |-- Running-batch-jobs + |-- Running-jobs-with-input + |-- Running-jobs-with-input-output-data + |-- example.pbs + '-- example.sh +9 directories, 5 files \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_3.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_3.1_metadata.json new file mode 100644 index 00000000000..81498072097 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_3.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Connect", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "First-Time-connection-to-the-HPC-infrastructure", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#warning-message-when-first-connecting-to-new-host" + }, + "previous_title": "connecting_paragraph_2", + "next_title": "connecting_paragraph_4", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#connect" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_5.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_5.1.txt new file mode 100644 index 00000000000..8c2bd12d342 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_5.1.txt @@ -0,0 +1,61 @@ +First Time connection to the HPC infrastructure +You can exit the connection at anytime by entering: +$ exit +logout +Connection to login.hpc.ugent.be closed. + tip "tip: Setting your Language right" + You may encounter a warning message similar to the following one during connecting: + perl: warning: Setting locale failed. + perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LANG = (unset) + are supported and installed on your system. + perl: warning: Falling back to the standard locale ("C"). + or any other error message complaining about the locale. + This means that the correct "locale" has not yet been properly specified on your local machine. Try: + LANG= + LC_COLLATE="C" + LC_CTYPE="UTF-8" + LC_MESSAGES="C" + LC_MONETARY="C" + LC_NUMERIC="C" + LC_TIME="C" + LC_ALL= + A locale is a set of parameters that defines the user's language, country and + any special variant preferences that the user wants to see in their user + interface. Usually a locale identifier consists of at least a language + identifier and a region identifier. + Note + If you try to set a non-supported locale, then it will be automatically + set to the default. Currently the default is en_US.UFT-8 or en_US, + depending on whether your originally (non-supported) locale was UTF-8 or not. + Open the .bashrc on your local machine with your favourite editor and + add the following lines: + + $ nano ~/.bashrc + ... + export LANGUAGE="en_US.UTF-8" + export LC_ALL="en_US.UTF-8" + export LC_CTYPE="en_US.UTF-8" + export LANG="en_US.UTF-8" + ... + + tip "tip: vi" + To start entering text in vi: move to the place you want to start + entering text with the arrow keys and type "i" to switch to insert mode. You can easily exit vi by entering: ""ESC" :wq" + To exit vi without saving your changes, enter ""ESC":q!" + + + or alternatively (if you are not comfortable with the Linux editors), + again on your local machine: + + echo "export LANGUAGE=\"en_US.UTF-8\"" >> ~/.profile + echo "export LC_ALL=\"en_US.UTF-8\"" >> ~/.profile + echo "export LC_CTYPE=\"en_US.UTF-8\"" >> ~/.profile + echo "export LANG=\"en_US.UTF-8\"" >> ~/.profile + + You can now log out, open a new terminal/shell on your local machine and + reconnect to the login node, and you should not get these warnings anymore. + \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_5.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_5.1_metadata.json new file mode 100644 index 00000000000..34db04e7ecf --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_5.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "previous_title": "connecting_paragraph_4", + "next_title": "connecting_paragraph_6", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_6.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_6.1.txt new file mode 100644 index 00000000000..d872c89a0f8 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_6.1.txt @@ -0,0 +1,6 @@ +Transfer Files to/from the HPC +Before you can do some work, you'll have to transfer the files you need from your desktop or department to the cluster. At the end of a job, you might want to transfer some files back. +The preferred way to transfer files is by using an scp or sftp via the +secure OpenSSH protocol. ships with an implementation of OpenSSH, so you +don't need to install any third-party software to use it. Just open a +terminal window and jump in! diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_6.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_6.1_metadata.json new file mode 100644 index 00000000000..ea88107f71a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_6.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "Transfer-Files-tofrom-the-HPC", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "previous_title": "connecting_paragraph_5", + "next_title": "connecting_paragraph_7", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#transfer-files-tofrom-the-hpc" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.1.txt new file mode 100644 index 00000000000..319433fd88a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.1.txt @@ -0,0 +1,42 @@ +Transfer Files tofrom the HPC +Using scp +Secure copy or SCP is a tool (command) for securely transferring files between a local +host (= your computer) and a remote host (the HPC). It is based on the +Secure Shell (SSH) protocol. The scp command is the equivalent of the cp (i.e., +copy) command, but can copy files to or from remote machines. +It's easier to copy files directly to $VSC_DATA and $VSC_SCRATCH if +you have symlinks to them in your home directory. See +the chapter titled "Uploading/downloading/editing files", section "Symlinks for data/scratch" in the intro to Linux + for how to do this. +Open an additional terminal window and check that you're working on your +local machine. +$ hostname + +If you're still using the terminal that is connected to the HPC, close the +connection by typing "exit" in the terminal window. +For example, we will copy the (local) file "localfile.txt" to your +home directory on the HPC cluster. We first generate a small dummy +"localfile.txt", which contains the word "Hello". Use your own VSC +account, which is something like "vsc40000". Don't forget the colon (:) at the +end: if you forget it, it will just create a file named vsc40000@login.hpc.ugent.be on your +local filesystem. You can even specify where to save the file on the +remote filesystem by putting a path after the colon. +$ echo "Hello" > localfile.txt +$ ls -l +... +-rw-r--r-- 1 user staff 6 Sep 18 09:37 localfile.txt +$ scp localfile.txt vsc40000@login.hpc.ugent.be: +localfile.txt 100% 6 0.0KB/s 00:00 +Connect to the HPC via another terminal, print the working directory (to +make sure you're in the home directory) and check whether the file has +arrived: +$ pwd +/user/home/gent/vsc400/vsc40000 +$ ls -l +total 1536 +drwxrwxr-x 2 +drwxrwxr-x 2 +drwxrwxr-x 10 +-rw-r--r-- 1 +$ cat localfile.txt +Hello diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.1_metadata.json new file mode 100644 index 00000000000..3df8e66ceb6 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Using-scp", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "links": { + "0": "https://docs.hpc.ugent.be/localhost:8000/Gent//intro-Linux/uploading_files/#symlinks-for-datascratch" + }, + "previous_title": "connecting_paragraph_6", + "next_title": "connecting_macos_paragraph_7.2", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#using-scp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.2.txt new file mode 100644 index 00000000000..16016dc219c --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.2.txt @@ -0,0 +1,37 @@ +The scp command can also be used to copy files from the cluster to your +local machine. Let us copy the remote file "intro-HPC--Gent.pdf" from your "docs" +subdirectory on the cluster to your local computer. +First, we will confirm that the file is indeed in the "docs" +subdirectory. In the terminal on the login node, enter: +$ cd ~/docs +$ ls -l +total 1536 +-rw-r--r-- 1 vsc40000 Sep 11 09:53 intro-HPC--Gent.pdf +Now we will copy the file to the local machine. On the terminal on your +own local computer, enter: +$ scp vsc40000@login.hpc.ugent.be:./docs/intro-HPC--Gent.pdf . +intro-HPC--Gent.pdf 100% 725KB 724.6KB/s 00:01 +$ ls -l +total 899 +-rw-r--r-- 1 user staff 741995 Sep 18 09:53 +-rw-r--r-- 1 user staff 6 Sep 18 09:37 localfile.txt +The file has been copied from the HPC to your local computer. +It's also possible to copy entire directories (and their contents) with +the -r flag. For example, if we want to copy the local directory +dataset to $VSC_SCRATCH, we can use the following command (assuming +you've created the scratch symlink): +scp -r dataset vsc40000@login.hpc.ugent.be:scratch +If you don't use the -r option to copy a directory, you will run into +the following error: +$ scp dataset vsc40000@login.hpc.ugent.be:scratch +dataset: not a regular file +Using sftp +The SSH File Transfer Protocol (also Secure File Transfer Protocol, or SFTP) is a network protocol that provides file access, file +transfer and file management functionalities over any reliable data +stream. It was designed as an extension of the Secure Shell protocol +(SSH) version 2.0. This protocol assumes that it is run over a secure +channel, such as SSH, that the server has already authenticated the +client, and that the identity of the client user is available to the +protocol. +The sftp is an equivalent of the ftp command, with the difference that +it uses the secure ssh protocol to connect to the clusters. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.2_metadata.json new file mode 100644 index 00000000000..acf4056f823 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "Using-sftp", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "previous_title": "connecting_macos_paragraph_7.1", + "next_title": "connecting_macos_paragraph_7.3", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#using-sftp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.3.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.3.txt new file mode 100644 index 00000000000..1ef13b80c6f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.3.txt @@ -0,0 +1,18 @@ +One easy way of starting a sftp session is +sftp vsc40000@login.hpc.ugent.be +Typical and popular commands inside an sftp session are: +| | | +|:--------------------------|:-------------------------------------------------------------------------------------| +| cd ~/exmples/fibo | Move to the examples/fibo subdirectory on the (i.e., the HPC remote machine) | +| ls | Get a list of the files in the current directory on the HPC. | +| get fibo.py | Copy the file "fibo.py" from the HPC | +| get tutorial/HPC.pdf | Copy the file "HPC.pdf" from the HPC, which is in the "tutorial" subdirectory. | +| lcd test | Move to the "test" subdirectory on your local machine. | +| lcd .. | Move up one level in the local directory. | +| lls | Get local directory listing. | +| put test.py | Copy the local file test.py to the HPC. | +| put test1.py test2.py | Copy the local file test1.py to the and rename it to test2.py. | +| bye | Quit the sftp session | +| **mget *.cc** | Copy all the remote files with extension ".cc" to the local directory. | +| **mput *.h** | Copy all the local files with extension ".h" to the HPC. | +| | | diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.3_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.3_metadata.json new file mode 100644 index 00000000000..70514bc20f5 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_7.3_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "connecting", + "subtitle": "Using-sftp", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "links": { + "0": "https://docs.hpc.ugent.be/", + "1": "https://docs.hpc.ugent.be/" + }, + "previous_title": "connecting_macos_paragraph_7.2", + "next_title": "connecting_paragraph_8", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#using-sftp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_8.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_8.1.txt new file mode 100644 index 00000000000..7fa726ab70e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_8.1.txt @@ -0,0 +1,25 @@ +Changing login nodes +Using a GUI (Cyberduck) +Cyberduck is a graphical alternative to the scp command. It can be +installed from https://cyberduck.io. +This is the one-time setup you will need to do before connecting: +1. After starting Cyberduck, the Bookmark tab will show up. To add a + new bookmark, click on the "+" sign on the bottom left of the + window. A new window will open. +2. In the drop-down menu on top, select "SFTP (SSH File Transfer Protocol)". +3. In the "Server" field, type in login.hpc.ugent.be. In the "Username" field, type in + your VSC account id (this looks like vsc40000). +4. Select the location of your SSH private key in the "SSH Private Key" field. +5. Finally, type in a name for the bookmark in the "Nickname" field and + close the window by pressing on the red circle in the top left + corner of the window. +To open the connection, click on the "Bookmarks" icon (which +resembles an open book) and double-click on the bookmark you just +created. +Fast file transfer for large datasets +See the section on rsync in chapter 5 of the Linux intro manual. +Changing login nodes +It can be useful to have control over which login node you are on. However, when you connect to the HPC (High-Performance Computing) system, you are directed to a random login node, which might not be the one where you already have an active session. To address this, there is a way to manually switch your active login node. +For instance, if you want to switch to the login node named gligar07.gastly.os, you can use the following command while you are connected to the gligar08.gastly.os login node on the HPC: +ssh gligar07.gastly.os +This is also possible the other way around. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_8.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_8.1_metadata.json new file mode 100644 index 00000000000..d4745200cb0 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/connecting/connecting_macos_paragraph_8.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Using-a-GUI-(Cyberduck)", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Transfer-Files-tofrom-the-HPC", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial/uploading_files/#copying-faster-with-rsync" + }, + "previous_title": "connecting_paragraph_7", + "next_title": "connecting_paragraph_9", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/connecting/#using-a-gui-cyberduck" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.1.txt new file mode 100644 index 00000000000..dfec88ca4e7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.1.txt @@ -0,0 +1,27 @@ +Getting Connected +Getting Started +Welcome to the "Getting Started" guide. This chapter will lead you through the initial steps of logging into the HPC-UGent infrastructure and submitting your very first job. We'll also walk you through the process step by step using a practical example. +In addition to this chapter, you might find the recording of the Introduction to HPC-UGent training session to be a useful resource. +Before proceeding, read the introduction to HPC to gain an understanding of the HPC-UGent infrastructure and related terminology. +Getting Access +To get access to the HPC-UGent infrastructure, visit Getting an HPC Account. +If you have not used Linux before, +now would be a good time to follow our Linux Tutorial. +A typical workflow looks like this: +1. Connect to the login nodes +2. Transfer your files to the HPC-UGent infrastructure +3. Optional: compile your code and test it +4. Create a job script and submit your job +5. Wait for job to be executed +6. Study the results generated by your jobs, either on the cluster or + after downloading them locally. +We will walk through an illustrative workload to get you started. In this example, our objective is to train a deep learning model for recognizing hand-written digits (MNIST dataset) using TensorFlow; +see the example scripts. +Getting Connected +There are two options to connect +- Using a terminal to connect via SSH (for power users) (see First Time connection to the HPC-UGent infrastructure) +- Using the web portal +Considering your operating system is *, it should be easy to make use of the ssh command in a terminal, but the web portal will work too. +The web portal offers a convenient way to upload files and gain shell access to the HPC-UGent infrastructure from a standard web browser (no software installation or configuration required). +See shell access when using the web portal, or +connection to the HPC-UGent infrastructure when using a terminal. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.1_metadata.json new file mode 100644 index 00000000000..a9c3e0c7726 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.1_metadata.json @@ -0,0 +1,25 @@ +{ + "main_title": "getting_started", + "subtitle": "Getting-Connected", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://www.ugent.be/hpc/en/training/introhpcugent-recording", + "1": "https://docs.hpc.ugent.be/introduction", + "2": "https://docs.hpc.ugent.be/account", + "3": "https://docs.hpc.ugent.be/linux-tutorial", + "4": "https://www.tensorflow.org/", + "5": "https://github.com/hpcugent/vsc_user_docs/tree/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist", + "6": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure", + "7": "https://docs.hpc.ugent.be/web_portal", + "8": "https://docs.hpc.ugent.be/web_portal", + "9": "https://docs.hpc.ugent.be/web_portal/#shell-access", + "10": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure" + }, + "previous_title": null, + "next_title": "getting_started_macos_paragraph_1.2", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/getting_started/#getting-connected" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.2.txt new file mode 100644 index 00000000000..99501b99809 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.2.txt @@ -0,0 +1 @@ +Make sure you can get to a shell access to the HPC-UGent infrastructure before proceeding with the next steps. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.2_metadata.json new file mode 100644 index 00000000000..3f34e4d8f83 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_1.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "getting_started", + "subtitle": "Getting-Connected", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "previous_title": "getting_started_macos_paragraph_1.1", + "next_title": "getting_started_paragraph_2", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/getting_started/#getting-connected" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_2.1.txt new file mode 100644 index 00000000000..438321b40a1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_2.1.txt @@ -0,0 +1,27 @@ +Submitting a job + Info + When having problems see the connection issues section on the troubleshooting page. +Transfer your files +Now that you can login, it is time to transfer files from your local computer to your home directory on the HPC-UGent infrastructure. +Download tensorflow_mnist.py +and run.sh example scripts to your computer (from here). +On your local machine you can run: +curl -OL https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/tensorflow_mnist.py +curl -OL https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/run.sh +Using the scp command, the files can be copied from your local host to your home directory (~) on the remote host (HPC). +scp tensorflow_mnist.py run.sh vsc40000login.hpc.ugent.be:~ +ssh vsc40000@login.hpc.ugent.be + Warning "User your own VSC account id" + + Replace vsc40000 with your VSC account id (see https://account.vscentrum.be) + Info + For more information about transfering files or scp, see tranfer files from/to hpc. +When running ls in your session on the HPC-UGent infrastructure, you should see the two files listed in your home directory (~): +$ ls ~ +run.sh tensorflow_mnist.py +When you do not see these files, make sure you uploaded the files to your home directory. +Submitting a job +Jobs are submitted and executed using job scripts. In our case run.sh can be used as a (very minimal) job script. +A job script is a shell script, a text file that specifies the resources, +the software that is used (via module load statements), +and the steps that should be executed to run the calculation. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..abfa4f07b2e --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/macos/getting_started/getting_started_macos_paragraph_2.1_metadata.json @@ -0,0 +1,19 @@ +{ + "main_title": "getting_started", + "subtitle": "Submitting-a-job", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#sec:connecting-issues", + "1": "https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/tensorflow_mnist.py", + "2": "https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/run.sh", + "3": "https://github.com/hpcugent/vsc_user_docs/tree/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist", + "4": "https://docs.hpc.ugent.be/connecting/#transfer-files-tofrom-the-hpc" + }, + "previous_title": "getting_started_paragraph_1", + "next_title": "getting_started_paragraph_3", + "OS": "macos", + "reference_link": "https://docs.hpc.ugent.be/macOS/getting_started/#submitting-a-job" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.1.txt new file mode 100644 index 00000000000..f47e95e91f0 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.1.txt @@ -0,0 +1,64 @@ +How do SSH keys work +Since all VSC clusters use Linux as their main operating system, you +will need to get acquainted with using the command-line interface and +using the terminal (see tutorial). +A typical Windows environment does not come with pre-installed software +to connect and run command-line executables on a HPC. Some tools need to be +installed on your Windows machine first, before we can start the actual +work. +Get PuTTY: A free telnet/SSH client +We recommend to use the PuTTY tools package, which is freely available. +You do not need to install PuTTY, you can download the PuTTY and +PuTTYgen executable and run it. This can be useful in situations where +you do not have the required permissions to install software on the +computer you are using. Alternatively, an installation package is also +available. +You can download PuTTY from the official address: +https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html. You +probably want the 64-bits version. If you can install software on your +computer, you can use the "Package files", if not, you can download and +use putty.exe and puttygen.exe in the "Alternative binary files" +section. +The PuTTY package consists of several components, but we'll only use +two: +1. PuTTY: the Telnet and SSH client itself (to login, see Open a terminal) +2. PuTTYgen: an RSA and DSA key generation utility (to generate a key pair, + see Generate a public/private key pair) +Generating a public/private key pair +Before requesting a VSC account, you need to generate a pair of ssh +keys. You need 2 keys, a public and a private key. You can visualise the +public key as a lock to which only you have the key (your private key). +You can send a copy of your lock to anyone without any problems, because +only you can open it, as long as you keep your private key secure. To +generate a public/private key pair, you can use the PuTTYgen key +generator. +Start PuTTYgen.exe it and follow these steps: +1. In "Parameters" (at the bottom of the window), choose "RSA" and set the number of + bits in the key to 4096. +2. Click on "Generate". To generate the key, you must move the mouse cursor over + the PuTTYgen window (this generates some random data that PuTTYgen + uses to generate the key pair). Once the key pair is generated, your + public key is shown in the field "Public key for pasting into OpenSSH authorized_keys file". +3. Next, it is advised to fill in the "Key comment" field to make it easier + identifiable afterwards. +4. Next, you should specify a passphrase in the "Key passphrase" field and retype it in + the "Confirm passphrase" field. Remember, the passphrase protects the private key against + unauthorised use, so it is best to choose one that is not too easy + to guess but that you can still remember. Using a passphrase is not + required, but we recommend you to use a good passphrase unless you + are certain that your computer's hard disk is encrypted with a + decent password. (If you are not sure your disk is encrypted, it + probably isn't.) +5. Save both the public and private keys in a folder on your personal + computer (We recommend to create and put them in the folder + "C:\\Users\\%USERNAME%\\AppData\\Local\\PuTTY\\.ssh") with the + buttons "Save public key" and "Save private key". We recommend using the name "id_rsa.pub" for the public key, and + "id_rsa.ppk" for the private key. +6. Finally, save an "OpenSSH" version of your private key (in + particular for later "X2Go" usage, see x2go) by entering the + "Conversions" menu and selecting "Export OpenSSH key" (do not select the + "force new file format" variant). Save the file in the same location + as in the previous step with filename "id_rsa". (If there is no + "Conversions" menu, you must update your "puttygen" version. If you + want to do this conversion afterwards, you can start with loading an + existing "id_rsa.ppk" and only do this conversions export.) diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..b17309fed2a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.1_metadata.json @@ -0,0 +1,18 @@ +{ + "main_title": "account", + "subtitle": "Generating-a-publicprivate-key-pair", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial", + "1": "https://docs.hpc.ugent.be/connecting/#open-a-terminal", + "2": "https://docs.hpc.ugent.be/account/#generating-a-publicprivate-key-pair", + "3": "https://docs.hpc.ugent.be/" + }, + "previous_title": "account_paragraph_1", + "next_title": "account_windows_paragraph_2.2", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/account/#generating-a-publicprivate-key-pair" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.2.txt new file mode 100644 index 00000000000..d0425d6738f --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.2.txt @@ -0,0 +1,2 @@ +If you use another program to generate a key pair, please remember that +they need to be in the OpenSSH format to access the HPC clusters. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.2_metadata.json new file mode 100644 index 00000000000..3df1f160c12 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_2.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Generating-a-publicprivate-key-pair", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_windows_paragraph_2.1", + "next_title": "account_paragraph_3", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/account/#generating-a-publicprivate-key-pair" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_4.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_4.1.txt new file mode 100644 index 00000000000..a70356b9172 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_4.1.txt @@ -0,0 +1,29 @@ +Using an SSH agent (optional) +It is possible to setup a SSH agent in Windows. This is an optional +configuration to help you to keep all your SSH keys (if you have +several) stored in the same key ring to avoid to type the SSH key +password each time. The SSH agent is also necessary to enable SSH hops +with key forwarding from Windows. +Pageant is the SSH authentication agent used in windows. This agent should be +available from the PuTTY installation package +https://www.chiark.greenend.org.uk/~sgtatham/putty/latest.html or as +stand alone binary package. +After the installation just start the Pageant application in Windows, +this will start the agent in background. The agent icon will be visible +from the Windows panel. +At this point the agent does not contain any private key. You should +include the private key(s) generated in the previous section Generating a public/private key pair. +1. Click on "Add key" +2. Select the private key file generated in Generating a public/private key pair ("id_rsa.ppk" by default). +3. Enter the same SSH key password used to generate the key. After this + step the new key will be included in Pageant to manage the SSH + connections. +4. You can see the SSH key(s) available in the key ring just clicking + on "View Keys". +5. You can change PuTTY setup to use the SSH agent. Open PuTTY and check + Connection > SSH > Auth > Allow agent forwarding. +Now you can connect to the login nodes as usual. The SSH agent will know +which SSH key should be used and you do not have to type the SSH +passwords each time, this task is done by Pageant agent automatically. +It is also possible to use WinSCP with Pageant, see +https://winscp.net/eng/docs/ui_pageant for more details. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_4.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_4.1_metadata.json new file mode 100644 index 00000000000..a5d3804cbcc --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_4.1_metadata.json @@ -0,0 +1,16 @@ +{ + "main_title": "account", + "subtitle": "Using-an-SSH-agent-(optional)", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/account/#generating-a-publicprivate-key-pair", + "1": "https://docs.hpc.ugent.be/account/#generating-a-publicprivate-key-pair" + }, + "previous_title": "account_paragraph_3", + "next_title": "account_paragraph_5", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/account/#using-an-ssh-agent-optional" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_5.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_5.1.txt new file mode 100644 index 00000000000..fbf46db59dd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_5.1.txt @@ -0,0 +1,38 @@ +Welcome e mail +Applying for the account +Visit https://account.vscentrum.be/ +You will be redirected to our WAYF (Where Are You From) service where +you have to select your "Home Organisation". +Select "UGent" in the dropdown box and optionally select "Save my preference" +and "permanently". +Click "Confirm" +You will now be taken to the authentication page of your institute. +You will now have to log in with CAS using your UGent account. +You either have a login name of maximum 8 characters, or a (non-UGent) +email address if you are an external user. In case of problems with your +UGent password, please visit: https://password.ugent.be/. After +logging in, you may be requested to share your information. Click "Yes, +continue". +After you log in using your UGent login and password, you will be asked to +upload the file that contains your public key, i.e., the file +"id_rsa.pub" which you have generated earlier. Make sure that your +public key is actually accepted for upload, because if it is in a wrong +format, wrong type or too short, then it will be refused. +This file should have been stored in the directory +"C:\\Users\\%USERNAME%\\AppData\\Local\\PuTTY\\.ssh" +After you have uploaded your public key you will receive an e-mail with +a link to confirm your e-mail address. After confirming your e-mail +address the VSC staff will review and if applicable approve your +account. +Welcome e-mail +Within one day, you should receive a Welcome e-mail with your VSC +account details. +Dear (Username), +Your VSC-account has been approved by an administrator. +Your vsc-username is vsc40000 +Your account should be fully active within one hour. +To check or update your account information please visit +https://account.vscentrum.be/ +For further info please visit https://www.vscentrum.be/user-portal +Kind regards, +-- The VSC administrators \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_5.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_5.1_metadata.json new file mode 100644 index 00000000000..ce6bf0bda3d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_5.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "account", + "subtitle": "Welcome-e-mail", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 3, + "directory": "account", + "parent_title": "account", + "previous_title": "account_paragraph_4", + "next_title": "account_paragraph_6", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/account/#welcome-e-mail" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_6.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_6.1.txt new file mode 100644 index 00000000000..dcb7ae5159d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_6.1.txt @@ -0,0 +1,34 @@ +Computation Workflow on the HPC +Now, you can start using the HPC. You can always look up your VSC id later +by visiting https://account.vscentrum.be. +Adding multiple SSH public keys (optional) +In case you are connecting from different computers to the login nodes, +it is advised to use separate SSH public keys to do so. You should +follow these steps. +1. Create a new public/private SSH key pair from Putty. Repeat the + process described in + section Generate a public/private key pair. +2. Go to https://account.vscentrum.be/django/account/edit +3. Upload the new SSH public key using the Add public key section. Make sure that your + public key is actually saved, because a public key will be refused + if it is too short, wrong type, or in a wrong format. +4. (optional) If you lost your key, you can delete the old key on the + same page. You should keep at least one valid public SSH key in your + account. +5. Take into account that it will take some time before the new SSH + public key is active in your account on the system; waiting for + 15-30 minutes should be sufficient. +Computation Workflow on the HPC +A typical Computation workflow will be: +1. Connect to the HPC +2. Transfer your files to the HPC +3. Compile your code and test it +4. Create a job script +5. Submit your job +6. Wait while + 1. your job gets into the queue + 2. your job gets executed + 3. your job finishes +7. Move your results +We'll take you through the different tasks one by one in the following +chapters. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_6.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_6.1_metadata.json new file mode 100644 index 00000000000..a7930a87bce --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/account/account_windows_paragraph_6.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "account", + "subtitle": "Computation-Workflow-on-the-HPC", + "source_file": "../../mkdocs/docs/HPC/account.md", + "title_depth": 2, + "directory": "account", + "parent_title": "account", + "links": { + "0": "https://docs.hpc.ugent.be/account/#generate-a-publicprivate-key-pair" + }, + "previous_title": "account_paragraph_5", + "next_title": null, + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/account/#computation-workflow-on-the-hpc" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_2.1.txt new file mode 100644 index 00000000000..7508c49411d --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_2.1.txt @@ -0,0 +1,48 @@ +First Time connection to the HPC infrastructure +ssh_exchange_identification: read: Connection reset by peer +First Time connection to the HPC infrastructure +The remaining content in this chapter is primarily focused for people utilizing a terminal with SSH. +If you are instead using the web portal, the corresponding chapter might be more helpful: Using the HPC-UGent web portal. +If you have any issues connecting to the HPC after you've followed these +steps, see Issues connecting to login node to troubleshoot. +Open a Terminal +You've generated a public/private key pair with PuTTYgen and have an +approved account on the VSC clusters. The next step is to setup the +connection to (one of) the HPC. +In the screenshots, we show the setup for user +"vsc20167" +to the HPC cluster via the login node "login.hpc.ugent.be". +1. Start the PuTTY executable putty.exe in your directory + C:\Program Files (x86)\PuTTY and the configuration screen will pop + up. As you will often use the PuTTY tool, we recommend adding a + shortcut on your desktop. +2. Within the category , in the field , enter the name of the + login node of the cluster (i.e., "login.hpc.ugent.be") you want to connect to. + +3. In the category "Connection > Data", in the field "Auto-login username", put in , which is your VSC + username that you have received by e-mail after your request was + approved. + +4. In the category "Connection > SSH > Auth", in the field "Private key file for authentication" click on "Browse" and select the private key + (i.e., "id_rsa.ppk") that you generated and saved above. +5. In the category "Connection > SSH > X11", click the "Enable X11 Forwarding" checkbox. +6. Now go back to , and fill in "hpcugent" in the "Saved Sessions" field and press "Save" to + store the session information. + +7. Now pressing "Open", will open a terminal window and asks for you + passphrase. + +8. If this is your first time connecting, you will be asked to verify + the authenticity of the login node. Please see + section Warning message when first connecting to new host + on how to do this. +9. After entering your correct passphrase, you will be connected to the + login-node of the HPC. +10. To check you can now "Print the Working Directory" (pwd) and check + the name of the computer, where you have logged in (hostname): + $ pwd + /user/home/gent/vsc400/vsc40000 + $ hostname -f + gligar07.gastly.os +11. For future PuTTY sessions, just select your saved session (i.e. "hpcugent") + from the list, "Load" it and press "Open". diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..dff7738ad08 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_2.1_metadata.json @@ -0,0 +1,17 @@ +{ + "main_title": "connecting", + "subtitle": "Open-a-Terminal", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "Connection-restrictions", + "links": { + "0": "https://docs.hpc.ugent.be/web_portal", + "1": "https://docs.hpc.ugent.be/troubleshooting/#issues-connecting-to-login-node", + "2": "https://docs.hpc.ugent.be/troubleshooting/#warning-message-when-first-connecting-to-new-host" + }, + "previous_title": "connecting_paragraph_1", + "next_title": "connecting_paragraph_3", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/connecting/#open-a-terminal" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_3.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_3.1.txt new file mode 100644 index 00000000000..aa43af81fbd --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_3.1.txt @@ -0,0 +1,32 @@ +First Time connection to the HPC infrastructure +Congratulations, you're on the HPC infrastructure now! +To find out where you have landed you can print the current working directory: +$ pwd +/user/home/gent/vsc400/vsc40000 +Your new private home directory is "/user/home/gent/vsc400/vsc40000". Here you can create your own +subdirectory structure, copy and prepare your applications, compile and +test them and submit your jobs on the HPC. +$ cd /apps/gent/tutorials +$ ls +Intro-HPC/ +This directory currently contains all training material for the Introduction to the HPC. More +relevant training material to work with the HPC can always be added later in +this directory. +You can now explore the content of this directory with the "ls --l" (lists long) and the "cd" (change directory) commands: +As we are interested in the use of the HPC, move further to Intro-HPC and explore the +contents up to 2 levels deep: +$ cd Intro-HPC +$ tree -L 2 +. +'-- examples + |-- Compiling-and-testing-your-software-on-the-HPC + |-- Fine-tuning-Job-Specifications + |-- Multi-core-jobs-Parallel-Computing + |-- Multi-job-submission + |-- Program-examples + |-- Running-batch-jobs + |-- Running-jobs-with-input + |-- Running-jobs-with-input-output-data + |-- example.pbs + '-- example.sh +9 directories, 5 files \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_3.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_3.1_metadata.json new file mode 100644 index 00000000000..0f620b2f326 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_3.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "previous_title": "connecting_paragraph_2", + "next_title": "connecting_paragraph_4", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_5.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_5.1.txt new file mode 100644 index 00000000000..7535a44a657 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_5.1.txt @@ -0,0 +1,33 @@ +First Time connection to the HPC infrastructure +You can exit the connection at anytime by entering: +$ exit +logout +Connection to login.hpc.ugent.be closed. + tip "tip: Setting your Language right" + You may encounter a warning message similar to the following one during connecting: + perl: warning: Setting locale failed. + perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = "UTF-8", + LANG = (unset) + are supported and installed on your system. + perl: warning: Falling back to the standard locale ("C"). + or any other error message complaining about the locale. + This means that the correct "locale" has not yet been properly specified on your local machine. Try: + LANG= + LC_COLLATE="C" + LC_CTYPE="UTF-8" + LC_MESSAGES="C" + LC_MONETARY="C" + LC_NUMERIC="C" + LC_TIME="C" + LC_ALL= + A locale is a set of parameters that defines the user's language, country and + any special variant preferences that the user wants to see in their user + interface. Usually a locale identifier consists of at least a language + identifier and a region identifier. + Note + If you try to set a non-supported locale, then it will be automatically + set to the default. Currently the default is en_US.UFT-8 or en_US, + depending on whether your originally (non-supported) locale was UTF-8 or not. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_5.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_5.1_metadata.json new file mode 100644 index 00000000000..f66c3ce38b5 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_5.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "First-Time-connection-to-the-HPC-infrastructure", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "previous_title": "connecting_paragraph_4", + "next_title": "connecting_paragraph_6", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/connecting/#first-time-connection-to-the-hpc-infrastructure" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.1.txt new file mode 100644 index 00000000000..f1012361c2a --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.1.txt @@ -0,0 +1,32 @@ +Transfer Files to/from the HPC +Before you can do some work, you'll have to transfer the files you need from your desktop or department to the cluster. At the end of a job, you might want to transfer some files back. +WinSCP +To transfer files to and from the cluster, we recommend the use of +WinSCP, a graphical file management tool which can transfer files using +secure protocols such as SFTP and SCP. WinSCP is freely available from +http://www.winscp.net. +To transfer your files using WinSCP, +1. Open the program +2. The "Login" menu is shown automatically (if it is closed, click "New Session" to open it again). Fill in the necessary fields under "Session" + 1. Click "New Site". + 2. Enter "login.hpc.ugent.be" in the "Host name" field. + 3. Enter your "vsc-account" in the "User name" field. + 4. Select "SCP" as the "file" protocol. + 5. Note that the password field remains empty. + + 6. Click "Advanced...". + 7. Click "SSH > Authentication". + 8. Select your private key in the field "Private key file". +3. Press the "Save" button, to save the session under "Session > Sites" for future access. +4. Finally, when clicking on "Login", you will be asked for your key passphrase. + +The first time you make a connection to the login node, a Security +Alert will appear and you will be asked to verify the authenticity of the +login node. +Make sure the fingerprint in the alert matches one of the following: +- ssh-rsa 2048 10:2f:31:21:04:75:cb:ed:67:e0:d5:0c:a1:5a:f4:78 +- ssh-rsa 2048 SHA256:W8Wz0/FkkCR2ulN7+w8tNI9M0viRgFr2YlHrhKD2Dd0 +- ssh-ed25519 255 19:28:76:94:52:9d:ff:7d:fb:8b:27:b6:d7:69:42:eb +- ssh-ed25519 256 SHA256:8AJg3lPN27y6i+um7rFx3xoy42U8ZgqNe4LsEycHILA +- ssh-ecdsa 256 e6:d2:9c:d8:e7:59:45:03:4a:1f:dc:96:62:29:9c:5f +- ssh-ecdsa 256 SHA256:C8TVx0w8UjGgCQfCmEUaOPxJGNMqv2PXLyBNODe5eOQ diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.1_metadata.json new file mode 100644 index 00000000000..8ea3fdc3107 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.1_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "WinSCP", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "First-Time-connection-to-the-HPC-infrastructure", + "previous_title": "connecting_paragraph_5", + "next_title": "connecting_windows_paragraph_6.2", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/connecting/#winscp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.2.txt new file mode 100644 index 00000000000..b52c614f263 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.2.txt @@ -0,0 +1,7 @@ +If it does, press Yes, if it doesn't, please contact hpc@ugent.be. +Note: it is possible that the ssh-ed25519 fingerprint starts with ssh-ed25519 255 +rather than ssh-ed25519 256 (or vice versa), depending on the PuTTY version you are using. +It is safe to ignore this 255 versus 256 difference, but the part after should be +identical. +Now, try out whether you can transfer an arbitrary file from your local +machine to the HPC and back. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.2_metadata.json new file mode 100644 index 00000000000..3da894d85e7 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_6.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "connecting", + "subtitle": "WinSCP", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 3, + "directory": "connecting", + "parent_title": "First-Time-connection-to-the-HPC-infrastructure", + "previous_title": "connecting_windows_paragraph_6.1", + "next_title": "connecting_paragraph_7", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/connecting/#winscp" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_8.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_8.1.txt new file mode 100644 index 00000000000..b16b119665b --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_8.1.txt @@ -0,0 +1,8 @@ +Changing login nodes +Fast file transfer for large datasets +See the section on rsync in chapter 5 of the Linux intro manual. +Changing login nodes +It can be useful to have control over which login node you are on. However, when you connect to the HPC (High-Performance Computing) system, you are directed to a random login node, which might not be the one where you already have an active session. To address this, there is a way to manually switch your active login node. +For instance, if you want to switch to the login node named gligar07.gastly.os, you can use the following command while you are connected to the gligar08.gastly.os login node on the HPC: +ssh gligar07.gastly.os +This is also possible the other way around. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_8.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_8.1_metadata.json new file mode 100644 index 00000000000..0ee95dd3ff4 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/connecting/connecting_windows_paragraph_8.1_metadata.json @@ -0,0 +1,15 @@ +{ + "main_title": "connecting", + "subtitle": "Changing-login-nodes", + "source_file": "../../mkdocs/docs/HPC/connecting.md", + "title_depth": 2, + "directory": "connecting", + "parent_title": "connecting", + "links": { + "0": "https://docs.hpc.ugent.be/linux-tutorial/uploading_files/#copying-faster-with-rsync" + }, + "previous_title": "connecting_paragraph_7", + "next_title": "connecting_paragraph_9", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/connecting/#changing-login-nodes" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.1.txt new file mode 100644 index 00000000000..eaf8a9391a1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.1.txt @@ -0,0 +1,27 @@ +Getting Connected +Getting Started +Welcome to the "Getting Started" guide. This chapter will lead you through the initial steps of logging into the HPC-UGent infrastructure and submitting your very first job. We'll also walk you through the process step by step using a practical example. +In addition to this chapter, you might find the recording of the Introduction to HPC-UGent training session to be a useful resource. +Before proceeding, read the introduction to HPC to gain an understanding of the HPC-UGent infrastructure and related terminology. +Getting Access +To get access to the HPC-UGent infrastructure, visit Getting an HPC Account. +If you have not used Linux before, +now would be a good time to follow our Linux Tutorial. +A typical workflow looks like this: +1. Connect to the login nodes +2. Transfer your files to the HPC-UGent infrastructure +3. Optional: compile your code and test it +4. Create a job script and submit your job +5. Wait for job to be executed +6. Study the results generated by your jobs, either on the cluster or + after downloading them locally. +We will walk through an illustrative workload to get you started. In this example, our objective is to train a deep learning model for recognizing hand-written digits (MNIST dataset) using TensorFlow; +see the example scripts. +Getting Connected +There are two options to connect +- Using a terminal to connect via SSH (for power users) (see First Time connection to the HPC-UGent infrastructure) +- Using the web portal +Considering your operating system is *, it is recommended to use the web portal. +The web portal offers a convenient way to upload files and gain shell access to the HPC-UGent infrastructure from a standard web browser (no software installation or configuration required). +See shell access when using the web portal, or +connection to the HPC-UGent infrastructure when using a terminal. diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.1_metadata.json new file mode 100644 index 00000000000..d17b01decef --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.1_metadata.json @@ -0,0 +1,25 @@ +{ + "main_title": "getting_started", + "subtitle": "Getting-Connected", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://www.ugent.be/hpc/en/training/introhpcugent-recording", + "1": "https://docs.hpc.ugent.be/introduction", + "2": "https://docs.hpc.ugent.be/account", + "3": "https://docs.hpc.ugent.be/linux-tutorial", + "4": "https://www.tensorflow.org/", + "5": "https://github.com/hpcugent/vsc_user_docs/tree/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist", + "6": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure", + "7": "https://docs.hpc.ugent.be/web_portal", + "8": "https://docs.hpc.ugent.be/web_portal", + "9": "https://docs.hpc.ugent.be/web_portal/#shell-access", + "10": "https://docs.hpc.ugent.be/connecting/#first-time-connection-to-the-hpc-infrastructure" + }, + "previous_title": null, + "next_title": "getting_started_windows_paragraph_1.2", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/getting_started/#getting-connected" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.2.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.2.txt new file mode 100644 index 00000000000..99501b99809 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.2.txt @@ -0,0 +1 @@ +Make sure you can get to a shell access to the HPC-UGent infrastructure before proceeding with the next steps. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.2_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.2_metadata.json new file mode 100644 index 00000000000..7c463f0f610 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_1.2_metadata.json @@ -0,0 +1,12 @@ +{ + "main_title": "getting_started", + "subtitle": "Getting-Connected", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "previous_title": "getting_started_windows_paragraph_1.1", + "next_title": "getting_started_paragraph_2", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/getting_started/#getting-connected" +} \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_2.1.txt b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_2.1.txt new file mode 100644 index 00000000000..5c2b11ef111 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_2.1.txt @@ -0,0 +1,21 @@ +Submitting a job + Info + When having problems see the connection issues section on the troubleshooting page. +Transfer your files +Now that you can login, it is time to transfer files from your local computer to your home directory on the HPC-UGent infrastructure. +Download tensorflow_mnist.py +and run.sh example scripts to your computer (from here). +The HPC-UGent web portal provides a file browser that allows uploading files. +For more information see the file browser section. +Upload both files (run.sh and tensorflow-mnist.py) to your home directory and go back to your shell. + Info + As an alternative, you can use WinSCP (see our section) +When running ls in your session on the HPC-UGent infrastructure, you should see the two files listed in your home directory (~): +$ ls ~ +run.sh tensorflow_mnist.py +When you do not see these files, make sure you uploaded the files to your home directory. +Submitting a job +Jobs are submitted and executed using job scripts. In our case run.sh can be used as a (very minimal) job script. +A job script is a shell script, a text file that specifies the resources, +the software that is used (via module load statements), +and the steps that should be executed to run the calculation. \ No newline at end of file diff --git a/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_2.1_metadata.json b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_2.1_metadata.json new file mode 100644 index 00000000000..e0616ff60e1 --- /dev/null +++ b/scripts/HPC_chatbot_preprocessor/parsed_mds/os_specific/windows/getting_started/getting_started_windows_paragraph_2.1_metadata.json @@ -0,0 +1,21 @@ +{ + "main_title": "getting_started", + "subtitle": "Submitting-a-job", + "source_file": "../../mkdocs/docs/HPC/getting_started.md", + "title_depth": 3, + "directory": "getting_started", + "parent_title": "getting_started", + "links": { + "0": "https://docs.hpc.ugent.be/troubleshooting/#sec:connecting-issues", + "1": "https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/tensorflow_mnist.py", + "2": "https://raw.githubusercontent.com/hpcugent/vsc_user_docs/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist/run.sh", + "3": "https://github.com/hpcugent/vsc_user_docs/tree/main/mkdocs/docs/HPC/examples/Getting_Started/tensorflow_mnist", + "4": "https://login.hpc.ugent.be", + "5": "https://docs.hpc.ugent.be/web_portal/#file-browser", + "6": "https://docs.hpc.ugent.be/connecting/#winscp" + }, + "previous_title": "getting_started_paragraph_1", + "next_title": "getting_started_paragraph_3", + "OS": "windows", + "reference_link": "https://docs.hpc.ugent.be/Windows/getting_started/#submitting-a-job" +} \ No newline at end of file diff --git a/scripts/README.md b/scripts/README.md index eed5a73e4d5..a88bd42cc46 100644 --- a/scripts/README.md +++ b/scripts/README.md @@ -1,3 +1,4 @@ Scripts that can be used to automatically generate markdown files, can be found here. -* [`available_software`](available_software): script to generate overview of available environment modules; \ No newline at end of file +* [`available_software`](available_software): script to generate overview of available environment modules; +* [`chatbot_preprocessor`](HPC_chatbot_preprocessor): script to generate input files for the chatbot; \ No newline at end of file