diff --git a/75/404.html b/75/404.html index 8167988633..016ff294b5 100644 --- a/75/404.html +++ b/75/404.html @@ -4,13 +4,13 @@ Page Not Found | Operating Systems - - + +
Skip to main content

Page Not Found

We could not find what you were looking for.

Please contact the owner of the site that linked you to the original URL and let them know their link is broken.

- - + + \ No newline at end of file diff --git a/75/Assignments/Asynchronous Web Server/index.html b/75/Assignments/Asynchronous Web Server/index.html index 57359592f6..63a8149bc4 100644 --- a/75/Assignments/Asynchronous Web Server/index.html +++ b/75/Assignments/Asynchronous Web Server/index.html @@ -4,12 +4,12 @@ Asynchronous Web Server | Operating Systems - - + +
-
Skip to main content

Asynchronous Web Server

Objectives​

  • Deepening the concepts related to working with sockets.
  • Developing skills in implementing and designing applications that use asynchronous operations and other advanced I/O operations.
  • Deepening the use of the API for advanced I/O operations in the Linux operating system.

Statement​

Implement a web server that uses the following advanced I/O operations:

  • Asynchronous operations on files
  • Non-blocking operations on sockets
  • Zero-copying
  • Multiplexing I/O operations

The server implements a limited functionality of the HTTP protocol: passing files to clients.

The web server will use the multiplexing API to wait for connections from clients - epoll. +

Asynchronous Web Server

Objectives​

  • Deepening the concepts related to working with sockets.
  • Developing skills in implementing and designing applications that use asynchronous operations and other advanced I/O operations.
  • Deepening the use of the API for advanced I/O operations in the Linux operating system.

Statement​

Implement a web server that uses the following advanced I/O operations:

  • Asynchronous operations on files
  • Non-blocking operations on sockets
  • Zero-copying
  • Multiplexing I/O operations

The server implements a limited functionality of the HTTP protocol: passing files to clients.

The web server will use the multiplexing API to wait for connections from clients - epoll. On the established connections, requests from clients will be received and then responses will be distributed to them.

The server will serve files from the AWS_DOCUMENT_ROOT directory, defined within the assignments' header. Files are only found in subdirectories AWS_DOCUMENT_ROOT/static/ and AWS_DOCUMENT_ROOT/dynamic/. The corresponding request paths will be, for example, AWS_DOCUMENT_ROOT/static/test.dat and AWS_DOCUMENT_ROOT/dynamic/test.dat. @@ -28,7 +28,7 @@ These folders are created and removed using the init and cleanup arguments to _test/run_test.sh.

Behind the Scenes​

Tests are basically unit tests.

Each test function follows the unit test patter: initialization, action, evaluation.

Each test starts the server, creates a given context, checks for validity and then terminates the server process.

Debugging​

Logs are collected in test.log and wget.log files.

Resources​

- - + + \ No newline at end of file diff --git a/75/Assignments/Asynchronous Web Server/src/http-parser/index.html b/75/Assignments/Asynchronous Web Server/src/http-parser/index.html index 9f86efd0ed..0b8b99a43b 100644 --- a/75/Assignments/Asynchronous Web Server/src/http-parser/index.html +++ b/75/Assignments/Asynchronous Web Server/src/http-parser/index.html @@ -4,8 +4,8 @@ HTTP Parser | Operating Systems - - + +
@@ -48,7 +48,7 @@ buffer to avoid copying memory around if this fits your application.

Reading headers may be a tricky task if you read/parse headers partially. Basically, you need to remember whether last header callback was field or value and apply following logic:

(on_header_field and on_header_value shortened to on_h_*)
 ------------------------ ------------ --------------------------------------------
| State (prev. callback) | Callback   | Description/action                         |
 ------------------------ ------------ --------------------------------------------
| nothing (first call)   | on_h_field | Allocate new buffer and copy callback data |
|                        |            | into it                                    |
 ------------------------ ------------ --------------------------------------------
| value                  | on_h_field | New header started.                        |
|                        |            | Copy current name,value buffers to headers |
|                        |            | list and allocate new buffer for new name  |
 ------------------------ ------------ --------------------------------------------
| field                  | on_h_field | Previous name continues. Reallocate name   |
|                        |            | buffer and append callback data to it      |
 ------------------------ ------------ --------------------------------------------
| field                  | on_h_value | Value for current header started. Allocate |
|                        |            | new buffer and copy callback data to it    |
 ------------------------ ------------ --------------------------------------------
| value                  | on_h_value | Value continues. Reallocate value buffer   |
|                        |            | and append callback data to it             |
 ------------------------ ------------ --------------------------------------------

See examples of reading in headers:

- - + + \ No newline at end of file diff --git a/75/Assignments/Memory Allocator/index.html b/75/Assignments/Memory Allocator/index.html index cf1a11cd1a..599cfceba7 100644 --- a/75/Assignments/Memory Allocator/index.html +++ b/75/Assignments/Memory Allocator/index.html @@ -4,12 +4,12 @@ Memory Allocator | Operating Systems - - + +
-
Skip to main content

Memory Allocator

Objectives​

  • Learn the basics of memory management by implementing minimal versions of malloc(), calloc(), realloc(), and free().
  • Accommodate with the memory management syscalls in Linux: brk(), mmap(), and munmap().
  • Understand the bottlenecks of memory allocation and how to reduce them.

Statement​

Build a minimalistic memory allocator that can be used to manually manage virtual memory. +

Memory Allocator

Objectives​

  • Learn the basics of memory management by implementing minimal versions of malloc(), calloc(), realloc(), and free().
  • Accommodate with the memory management syscalls in Linux: brk(), mmap(), and munmap().
  • Understand the bottlenecks of memory allocation and how to reduce them.

Statement​

Build a minimalistic memory allocator that can be used to manually manage virtual memory. The goal is to have a reliable library that accounts for explicit allocation, reallocation, and initialization of memory.

Support Code​

The support code consists of three directories:

  • src/ will contain your solution
  • tests/ contains the test suite and a Python script to verify your work
  • utils/ contains osmem.h that describes your library interface, block_meta.h which contains details of struct block_meta, and an implementation for printf() function that does NOT use the heap

The test suite consists of .c files that will be dynamically linked to your library, libosmem.so. You can find the sources in the tests/snippets/ directory. The results of the previous will also be stored in tests/snippets/ and the reference files are in the tests/ref/ directory.

The automated checking is performed using run_tests.py. @@ -51,8 +51,8 @@ For consistency, the heap start and addresses returned by mmap() are replaced with labels. Every other address is displayed as <label> + offset, where the label is the closest mapped address.

run_tests.py supports three modes:

  • verbose (-v), prints the output of the test
  • diff (-d), prints the diff between the output and the ref
  • memcheck (-m), prints the diff between the output and the ref and announces memory leaks

If you want to run a single test, you give its name or its path as arguments to run_tests.py:

student@os:~/.../mem-alloc/tests$ python3 run_tests.py test-all
OR
student@os:~/.../mem-alloc/tests$ python3 run_tests.py snippets/test-all

Debugging in VSCode​

If you are using Visual Studio Code, you can use the launch.json configurations to run tests.

Setup the breakpoints in the source files or the tests and go to Run and Debug (F5). Select Run test script and press F5. -This will enter a dialogue where you can choose which test to run.

You can find more on this in the official documentation: Debugging with VSCode.

If VSCode complains about MAP_ANON argument for mmap() change C_Cpp.default.cStandard option to gnu11.

Resources​

- - +This will enter a dialogue where you can choose which test to run.

You can find more on this in the official documentation: Debugging with VSCode.

If VSCode complains about MAP_ANON argument for mmap() change C_Cpp.default.cStandard option to gnu11.

Resources​

+ + \ No newline at end of file diff --git a/75/Assignments/Mini Libc/index.html b/75/Assignments/Mini Libc/index.html index 3bd757880a..bbfd6323ad 100644 --- a/75/Assignments/Mini Libc/index.html +++ b/75/Assignments/Mini Libc/index.html @@ -4,12 +4,12 @@ Mini-libc | Operating Systems - - + +
-
Skip to main content

Mini-libc

Objectives​

  • Learn about the structure and functionalities provided by the standard C library
  • Accommodate with the syscall interface in Linux
  • Gain a better understanding of strings and memory management functions
  • Learn how the standard C library provides support for low-level input/output operations

Statement​

Build a minimalistic*- [standard C library](https://en.wikipedia.org/wiki/C_standard_library) implementation for Linux systems (named mini-libc), that can be used as a replacement for the system libc*- (glibc in Linux). +

Mini-libc

Objectives​

  • Learn about the structure and functionalities provided by the standard C library
  • Accommodate with the syscall interface in Linux
  • Gain a better understanding of strings and memory management functions
  • Learn how the standard C library provides support for low-level input/output operations

Statement​

Build a minimalistic*- [standard C library](https://en.wikipedia.org/wiki/C_standard_library) implementation for Linux systems (named mini-libc), that can be used as a replacement for the system libc*- (glibc in Linux). The goal is to have a minimally functional libc with features such as string management, basic memory support and POSIX file I/O.

The implementation of mini-libc will be freestanding, i.e. it will not use any outside library calls. It will be implemented on top of the system call interface provided by Linux on an x86_64 architecture. Any function you require, that is typically part of libc, you will have to implement. @@ -27,7 +27,7 @@ Some tests don't fail because the missing implementation equates to the bad behavior being tested not happening.

Each test is worth a number of points. The total number of points is 900. The maximum grade is obtained by dividing the number of points to 10, for a maximum grade of 90.

A successful run will show the output:

student@so:~/.../assignments/mini-libc/tests$ make check
[...]
test_strcpy                      ........................ passed ...   9
test_strcpy_append               ........................ passed ...   9
test_strncpy                     ........................ passed ...   9
test_strncpy_cut                 ........................ passed ...   9
test_strcat                      ........................ passed ...   9
test_strcat_from_zero            ........................ passed ...   9
test_strcat_multiple             ........................ passed ...   9
test_strncat                     ........................ passed ...   9
test_strncat_cut                 ........................ passed ...   9
test_strcmp_equal                ........................ passed ...   9
test_strcmp_same_size_less       ........................ passed ...   1
test_strcmp_same_size_greater    ........................ passed ...   9
test_strcmp_diff_size_less       ........................ passed ...   1
test_strcmp_diff_size_greater    ........................ passed ...   9
test_strncmp_equal_size_equal    ........................ passed ...   9
test_strncmp_diff_contents_equal ........................ passed ...   9
test_strncmp_diff_size_equal     ........................ passed ...   9
test_strchr_exists               ........................ passed ...  11
test_strchr_exists_twice         ........................ passed ...   9
test_strchr_not_exists           ........................ passed ...   1
test_strrchr_exists              ........................ passed ...  11
test_strrchr_exists_twice        ........................ passed ...   9
test_strrchr_not_exists          ........................ passed ...   1
test_strstr_exists               ........................ passed ...  11
test_strstr_exists_twice         ........................ passed ...   9
test_strstr_not_exists           ........................ passed ...   1
test_strrstr_exists              ........................ passed ...  11
test_strrstr_exists_twice        ........................ passed ...   9
test_strrstr_not_exists          ........................ passed ...   1
test_memcpy                      ........................ passed ...  11
test_memcpy_part                 ........................ passed ...   9
test_memcmp_equal_size_equal     ........................ passed ...   9
test_memcmp_diff_contents_equal  ........................ passed ...   9
test_memcmp_diff_size_equal      ........................ passed ...   9
test_memset                      ........................ passed ...   9
test_memset_part                 ........................ passed ...   9
test_memmove_apart               ........................ passed ...   9
test_memmove_src_before_dst      ........................ passed ...   9
test_memmove_src_after_dst       ........................ passed ...   9
test_open_non_existent_file      ........................ passed ...   8
test_open_invalid_access_mode    ........................ passed ...   8
test_open_file_as_directory      ........................ passed ...   8
test_open_directory_for_writing  ........................ passed ...   8
test_open_force_invalid_creation ........................ passed ...   8
test_open_close_existent_file    ........................ passed ...   8
test_open_close_create_file      ........................ passed ...   8
test_open_read_write_only_mode   ........................ passed ...   8
test_open_write_read_only_mode   ........................ passed ...   8
test_lseek_invalid_fd            ........................ passed ...   8
test_lseek_invalid_whence        ........................ passed ...   8
test_lseek_invalid_offset        ........................ passed ...   8
test_lseek_set                   ........................ passed ...   8
test_lseek_cur                   ........................ passed ...   8
test_lseek_end                   ........................ passed ...   8
test_lseek_combined              ........................ passed ...   8
test_truncate_read_only_file     ........................ passed ...   8
test_truncate_invalid_size       ........................ passed ...   8
test_truncate_directory          ........................ passed ...   8
test_truncate_non_existent_file  ........................ passed ...   8
test_truncate_file               ........................ passed ...   8
test_ftruncate_read_only_file    ........................ passed ...   8
test_ftruncate_invalid_size      ........................ passed ...   8
test_ftruncate_directory         ........................ passed ...   8
test_ftruncate_bad_fd            ........................ passed ...   8
test_ftruncate_file              ........................ passed ...   8
test_stat_non_existent_file      ........................ passed ...   8
test_stat_regular_file           ........................ passed ...   8
test_fstat_bad_fd                ........................ passed ...   8
test_fstat_regular_file          ........................ passed ...   8
test_puts                        ........................ passed ...  15
test_open_close_create_file      ........................ passed ...  10
test_open_close_read_byte        ........................ passed ...  10
test_ftruncate                   ........................ passed ...  10
test_truncate                    ........................ passed ...  10
test_fstat                       ........................ passed ...  10
test_stat                        ........................ passed ...  10
test_sleep                       ........................ passed ...  20
test_nanosleep                   ........................ passed ...  20
test_mmap                        ........................ passed ...   8
test_mmap_bad_fd                 ........................ passed ...   8
test_mmap_bad_flags              ........................ passed ...   8
test_mremap                      ........................ passed ...   8
test_malloc                      ........................ passed ...   8
test_malloc_two                  ........................ passed ...   8
test_malloc_access               ........................ passed ...   8
test_malloc_memset               ........................ passed ...   8
test_malloc_memcpy               ........................ passed ...   8
test_calloc                      ........................ passed ...   8
test_realloc                     ........................ passed ...   8
test_realloc_access              ........................ passed ...   8
test_realloc_memset              ........................ passed ...   8
test_realloc_array               ........................ passed ...   8
test_malloc                      ........................ passed ...  10
test_multiple_malloc             ........................ passed ...  10
test_malloc_free                 ........................ passed ...  10
test_multiple_malloc_free        ........................ passed ...  10
test_malloc_free_sequence        ........................ passed ...  10
test_malloc_perm_ok              ........................ passed ...  10
test_malloc_perm_notok           ........................ passed ...  10
test_mmap                        ........................ passed ...  10
test_mmap_munmap                 ........................ passed ...  10
test_mmap_perm_ok                ........................ passed ...  10
test_mmap_perm_notok             ........................ passed ...  10
test_mmap_perm_none              ........................ passed ...  10

Total:                                                            90/100

Behind the Scenes​

For a fine grained approach, build tests and ignore errors (due to missing source code and header files) by using:

student@so:~/.../assignments/mini-libc/tests$ make -i

Then run the tests, either individually via executable files and scripts:

student@so:~/.../assignments/mini-libc/tests$ ./test_lseek.sh
test_lseek                       ........................ passed ...  10

student@so:~/.../assignments/mini-libc/tests$ ./test_memory
test_mmap                        ........................ passed ...   8
test_mmap_bad_fd                 ........................ passed ...   8
[...]

Or run them all via the run_all_tests.sh script:

student@so:~/.../assignments/mini-libc/tests$ ./run_all_tests.sh
test_strcpy                      ........................ passed ...   9
test_strcpy_append               ........................ passed ...   9
test_strncpy                     ........................ passed ...   9
[...]

Resources​

- - + + \ No newline at end of file diff --git a/75/Assignments/Mini Shell/index.html b/75/Assignments/Mini Shell/index.html index b501cc8378..11b6fc0102 100644 --- a/75/Assignments/Mini Shell/index.html +++ b/75/Assignments/Mini Shell/index.html @@ -4,12 +4,12 @@ Minishell | Operating Systems - - + +
-

Minishell

Objectives​

  • Learn how shells create new child processes and connect the I/O to the terminal.
  • Gain a better understanding of the fork() function wrapper.
  • Learn to correctly execute commands written by the user and treat errors.

Statement​

Introduction​

A shell is a command-line interpreter that provides a text-based user interface for operating systems. +

Minishell

Objectives​

  • Learn how shells create new child processes and connect the I/O to the terminal.
  • Gain a better understanding of the fork() function wrapper.
  • Learn to correctly execute commands written by the user and treat errors.

Statement​

Introduction​

A shell is a command-line interpreter that provides a text-based user interface for operating systems. Bash is both an interactive command language and a scripting language. It is used to interact with the file system, applications, operating system and more.

For this assignment you will build a Bash-like shell with minimal functionalities like traversing the file system, running applications, redirecting their output or piping the output from one application into the input of another. The details of the functionalities that must be implemented will be further explained.

Shell Functionalities​

Changing the Current Directory​

The shell will support a built-in command for navigating the file system, called cd. @@ -37,8 +37,8 @@ The maximum grade is obtained by dividing the number of points to 10, for a maximum grade of 9.00.

A successful test run will show the output:

student@so:~/.../assignment-mini-shell/tests$ make check
make[1]: Entering directory '...'
rm -f *~
[...]
01) Testing commands without arguments......................passed  [03/100]
02) Testing commands with arguments.........................passed  [02/100]
03) Testing simple redirect operators.......................passed  [05/100]
04) Testing append redirect operators.......................passed  [05/100]
05) Testing current directory...............................passed  [05/100]
06) Testing conditional operators...........................passed  [05/100]
07) Testing sequential commands.............................passed  [03/100]
08) Testing environment variables...........................passed  [05/100]
09) Testing single pipe.....................................passed  [05/100]
10) Testing multiple pipes..................................passed  [10/100]
11) Testing variables and redirect..........................passed  [05/100]
12) Testing overwritten variables...........................passed  [02/100]
13) Testing all operators...................................passed  [02/100]
14) Testing parallel operator...............................passed  [10/100]
15) Testing big file........................................passed  [05/100]
16) Testing sleep command...................................passed  [07/100]
17) Testing fscanf function.................................passed  [07/100]
18) Testing unknown command.................................passed  [04/100]

                                                            Total:   90/100

The actual tests are located in the inputs/ directory.

student@os:~/.../assignment-mini-shell/tests/$ ls -F _test/inputs
test_01.txt  test_03.txt  test_05.txt  test_07.txt  test_09.txt  test_11.txt  test_13.txt  test_15.txt  test_17.txt
test_02.txt  test_04.txt  test_06.txt  test_08.txt  test_10.txt  test_12.txt  test_14.txt  test_16.txt  test_18.txt

Running the Linters​

To run the linters, use the make lint command in the tests/ directory:

student@so:~/.../assignment-mini-shell/tests/$ make lint
[...]
cd .. && checkpatch.pl -f checker/*.sh tests/*.sh
[...]
cd .. && cpplint --recursive src/ tests/ checker/
[...]
cd .. && shellcheck checker/*.sh tests/*.sh

Note that the linters have to be installed on your system: checkpatch.pl, cpplint, shellcheck with certain configuration options.

Debugging​

To inspect the differences between the output of the mini-shell and the reference binary set DO_CLEANUP=no in tests/_test/run_test.sh. To see the results of the tests, you can check tests/_test/outputs/ directory.

Memory leaks​

To inspect the unreleased resources (memory leaks, file descriptors) set USE_VALGRIND=yes and DO_CLEANUP=no in tests/_test/run_test.sh. You can modify both the path to the Valgrind log file and the command parameters. -To see the results of the tests, you can check tests/_test/outputs/ directory.

- - +To see the results of the tests, you can check tests/_test/outputs/ directory.

+ + \ No newline at end of file diff --git a/75/Assignments/Mini Shell/util/parser/index.html b/75/Assignments/Mini Shell/util/parser/index.html index 81bc778317..6e1276ca60 100644 --- a/75/Assignments/Mini Shell/util/parser/index.html +++ b/75/Assignments/Mini Shell/util/parser/index.html @@ -4,8 +4,8 @@ Parser | Operating Systems - - + +
@@ -14,7 +14,7 @@ To use the parser, you need to link the object files parser.yy.o and parser.tab.o with your program.

Example​

Tests​

More tests can be found in the tests directory:

student@os:/.../minishell/util/parser$ cd tests

student@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>small_tests.out <small_tests.txt

student@os:/.../minishell/util/parser/tests$ cat small_tests.out
> mkdir tmp
Command successfully read!
command_t (
    scmd (
        simple_command_t (
            verb (
                'mkdir'
            )
            params (
                'tmp'
            )
        )
    )
)

> cd tmp
Command successfully read!
command_t (
    scmd (

...
student@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>ugly_tests.out <ugly_tests.txt

student@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>negative_tests.out <negative_tests.txt

Note​

The parser will fail with an error of unknown character if you use the Linux parser (which considers the end of line as \n) on Windows files (end of line as \r\n) because at the end of the lines (returned by getline()) there will be a \r followed by \n. The opposite works (Windows parser with Linux files). The test files use the Linux convention (\n).

Other information​

More information about the parser can be found in the file parser.h.

- - + + \ No newline at end of file diff --git a/75/Assignments/Parallel Firewall/index.html b/75/Assignments/Parallel Firewall/index.html new file mode 100644 index 0000000000..1eb5c67fba --- /dev/null +++ b/75/Assignments/Parallel Firewall/index.html @@ -0,0 +1,40 @@ + + + + + +Parallel Firewall | Operating Systems + + + + +
+
Skip to main content

Parallel Firewall

Objectives​

  • Learn how to design and implement parallel programs
  • Get experienced at utilizing the POSIX threading API
  • Learn how to convert a serial program into a parallel one

Statement​

A firewall is a program that checks network packets against a series of filters which provide a decision regarding dropping or allowing the packets to continue to their intended destination.

In a real setup, the network card will receive real packets (e.g. packets having Ethernet, IP headers plus payload) from the network and will send them to the firewall for processing. +The firewall will decide if the packets are to be dropped or not and, if not, passes them further.

In this assignment, instead of real network packets, we'll deal with made up packets consisting of a made up source (a number), a made up destination (also a number), a timestamp (also a number) and some payload. +And instead of the network card providing the packets, we'll have a producer thread creating these packets.

The created packets will be inserted into a circular buffer, out of which consumer threads (which implement the firewall logic) will take packets and process them in order to decide whether they advance to the destination.

The result of this processing is a log file in which the firewall will record the decision taken (PASS or DROP) for each packet, along with other information such as timestamp.

The purpose of this assignment is to:

  • implement the circular buffer, along with synchronization mechanisms for it to work in a multithreaded program

  • implement the consumer threads, which consume packets and process them

  • provide the log file containing the result of the packet processing

Support Code​

The support code consists of the directories:

  • src/ contains the skeleton for the parallelized firewall and the already implemented serial code in src/serial.c. +You will have to implement the missing parts marked as TODO

  • utils/ contains utility files used for debugging and logging.

  • tests/ contains tests used to validate and grade the assignment.

Implementation​

Firewall Threads​

In order to parallelize the firewall we have to distribute the packets to multiple threads. +The packets will be added to a shared data structure (visible to all threads) by a producer thread and processed by multiple consumer threads. +Each consumer thread picks a packet from the shared data structure, checks it against the filter function and writes the packet hash together with the drop/accept decision to a log file. +consumer threads stop waiting for new packets from the producer thread and exit when the producer thread closes the connection to the shared data structure.

The consumer threads must not do any form of busy waiting. +When there are new packets that need to be handled, the consumer threads must be notified. +Waiting in a while() loop or sleeping is not considered a valid synchronization mechanism and points will be deducted.

Implement the consumer related functions marked with TODO in the src/consumer.c file. +The number of consumer threads will be passed as the 3rd command-line argument

Ring Buffers​

A ring buffer (or a circular buffer) is a data structure that stores its elements in a circular fixed size array. +One of the advantages of using such a data structure as opposed to an array is that it acts as a FIFO, without the overhead of moving the elements to the left as they are consumed. +Thus, the shared ring buffer offers the following fields:

  • write_pos index in the buffer used by the producer thread for appending new packets.
  • read_pos index in the buffer used by the consumer threads to pick packets.
  • cap the size of the internal buffer.
  • data pointer to the internal buffer.

Apart from these fields you have to add synchronization primitives in order to allow multiple threads to access the ring buffer in a deterministic manner. +You can use mutexes, semaphores, conditional variables and other synchronization mechanisms offered by the pthread library.

You will have to implement the following interface for the ring buffer:

  • ring_buffer_init(): initialize the ring buffer (allocate memory and synchronization primitives).
  • ring_buffer_enqueue(): add elements to the ring buffer.
  • ring_buffer_dequeue(): remove elements from the ring buffer.
  • ring_buffer_destroy(): free up the memory used by the ring_buffer.
  • ring_buffer_stop(): finish up using the ring buffer for the calling thread.

Log File​

The output of the firewall will be a log file with the rows containing the firewall's decision, the hash of the packet and its timestamp. +The actual format can be found in the serial implementation (at src/serial.c).

When processing the packets in parallel the threads will finish up the work in a non deterministic order. +The packet processing functions are already implemented in src/packet.c

We would like the logs to be sorted by the packet timestamp, the order that they came in from the producer. +Thus, the consumers should insert the packet information to the log file such as the result is ordered by timestamp. +The printing format can be found in ./src/serial.c

The logs must be written to the file in ascending order during packet processing. +Sorting the log file after the consumer threads have finished processing is not considered a valid synchronization mechanism and points will be deducted.

Operations​

Building​

To build both the serial and the parallel versions, run make in the src/ directory:

student@so:~/.../content/assignments/parallel-firewall$ cd src/

student@so:~/.../assignments/parallel-firewall/src$ make

That will create the serial and firewall binaries.

Testing and Grading​

Testing is automated. +Tests are located in the tests/ directory.

To test and grade your assignment solution, enter the tests/ directory and run grade.sh.

student@so:~/.../content/assignments/parallel-firewall$ cd tests/
student@so:~/.../content/assignments/parallel-firewall/tests$ ./grade.sh

Note that this requires linters being available. +The easiest way to test the project is to use a Docker-based setup with everything installed and configured (see the README.checker.md file for instructions).

To create the tests, run:

student@so:~/.../content/assignments/parallel-firewall/tests$ make check

To remove the tests, run:

student@so:~/.../content/assignments/parallel-firewall/tests$ make distclean

When using grade.sh you will get a maximum of 90/100 points for general correctness and a maximum of 10/100 points for coding style.

Restrictions​

  • Threads must yield the cpu when waiting for empty/full buffers i.e. not doing busy waiting.
  • The logs must be written as they are processed and not after the processing is done, in ascending order by the timestamp.
  • The number of running threads must be at least num_consumers + 1, where num_consumers is the 3rd command-line argument of the firewall binary.

Grades​

  • 10 points are awarded for a single consumer solution that also implements the ring buffer
  • 50 points are awarded for a multi consumer solution
  • 30 points are awarded for a multi consumer solution that writes the logs in the sorted manner (bearing in mind the above restrictions)

Running the Checker​

Each test is worth a number of points. +The maximum grade is 90.

A successful run will show the output:

student@so:~/.../assignments/parallel-firewall/tests$ make check
[...]
Test [    10 packets, sort False, 1 thread ] ...................... passed ... 3
Test [ 1,000 packets, sort False, 1 thread ] ...................... passed ... 3
Test [20,000 packets, sort False, 1 thread ] ...................... passed ... 4
Test [    10 packets, sort True , 2 threads] ...................... passed ... 5
Test [    10 packets, sort True , 4 threads] ...................... passed ... 5
Test [   100 packets, sort True , 2 threads] ...................... passed ... 5
Test [   100 packets, sort True , 4 threads] ...................... passed ... 5
Test [ 1,000 packets, sort True , 2 threads] ...................... passed ... 5
Test [ 1,000 packets, sort True , 4 threads] ...................... passed ... 5
Test [10,000 packets, sort True , 2 threads] ...................... passed ... 5
Test [10,000 packets, sort True , 4 threads] ...................... passed ... 5
Test [20,000 packets, sort True , 2 threads] ...................... passed ... 5
Test [20,000 packets, sort True , 4 threads] ...................... passed ... 5
Test [ 1,000 packets, sort False, 4 threads] ...................... passed ... 5
Test [ 1,000 packets, sort False, 8 threads] ...................... passed ... 5
Test [10,000 packets, sort False, 4 threads] ...................... passed ... 5
Test [10,000 packets, sort False, 8 threads] ...................... passed ... 5
Test [20,000 packets, sort False, 4 threads] ...................... passed ... 5
Test [20,000 packets, sort False, 8 threads] ...................... passed ... 5

Checker:                                                                    90/100

Running the Linters​

To run the linters, use the make lint command in the tests/ directory:

student@so:~/.../assignments/parallel-firewall/tests$ make lint
[...]
cd .. && checkpatch.pl -f checker/*.sh tests/*.sh
[...]
cd .. && cpplint --recursive src/ tests/ checker/
[...]
cd .. && shellcheck checker/*.sh tests/*.sh

Note that the linters have to be installed on your system: checkpatch.pl, cpplint, shellcheck. +They also need to have certain configuration options. +It's easiest to run them in a Docker-based setup with everything configured.

Fine-Grained Testing​

Input tests cases are located in tests/in/ and are generated by the checker. +The expected results are generated by the checker while running the serial implementation. +If you want to run a single test, use the below commands while in the src/ directory:

student@so:~/.../assignments/parallel-firewall/src$ ./firewall ../tests/in/test_<num_packets>.in <output_file> <number_of_consumers>

Results provided by the serial and parallel implementation must be the same for the test to successfully pass.

+ + + + \ No newline at end of file diff --git a/75/Assignments/Parallel Graph/index.html b/75/Assignments/Parallel Graph/index.html deleted file mode 100644 index a506bc8655..0000000000 --- a/75/Assignments/Parallel Graph/index.html +++ /dev/null @@ -1,52 +0,0 @@ - - - - - -Parallel Graph | Operating Systems - - - - -
-
Skip to main content

Parallel Graph

Objectives​

  • Learn how to design and implement parallel programs
  • Gain skills in using synchronization primitives for parallel programs
  • Get a better understanding of the POSIX threading and synchronization API
  • Gain insight on the differences between serial and parallel programs

Statement​

Implement a generic thread pool, then use it to traverse a graph and compute the sum of the elements contained by the nodes. -You will be provided with a serial implementation of the graph traversal and with most of the data structures needed to implement the thread pool. -Your job is to write the thread pool routines and then use the thread pool to traverse the graph.

Support Code​

The support code consists of the directories:

  • src/ is the skeleton parallel graph implementation. -You will have to implement missing parts marked as TODO items.

  • utils/ utility files (used for debugging & logging)

  • tests/ are tests used to validate (and grade) the assignment.

Implementation​

Thread Pool Description​

A thread pool contains a given number of active threads that simply wait to be given specific tasks. -The threads are created when the thread pool is created. -Each thread continuously polls the task queue for available tasks. -Once tasks are put in the task queue, the threads poll tasks, and start running them. -A thread pool creates N threads upon its creation and does not destroy (join) them throughout its lifetime. -That way, the penalty of creating and destroying threads ad-hoc is avoided. -As such, you must implement the following functions (marked with TODO in the provided skeleton, in src/os_threadpool.c):

  • enqueue_task(): Enqueue task to the shared task queue. -Use synchronization.
  • dequeue_task(): Dequeue task from the shared task queue. -Use synchronization.
  • wait_for_completion(): Wait for all worker threads. -Use synchronization.
  • create_threadpool(): Create a new thread pool.
  • destroy_threadpool(): Destroy a thread pool. -Assume all threads have been joined.

You must also update the os_threadpool_t structure in src/os_threadpool.h with the required bits for synchronizing the parallel implementation.

Notice that the thread pool is completely independent of any given application. -Any function can be registered in the task queue.

Since the threads are polling the task queue indefinitely, you need to define a condition for them to stop once the graph has been traversed completely. -That is, the condition used by the wait_for_completion() function. -The recommended way is to note when no threads have any more work to do. -Since no thread is doing any work, no other task will be created.

Graph Traversal​

Once you have implemented the thread pool, you need to test it by doing a parallel traversal of all connected nodes in a graph. -A serial implementation for this algorithm is provided in src/serial.c. -To make use of the thread pool, you will need to create tasks that will be put in the task queue. -A task consists of 2 steps:

  1. Add the current node value to the overall sum.
  2. Create tasks and add them to the task queue for the neighbouring nodes.

Implement this in the src/parallel.c (see the TODO items). -You must implement the parallel and synchronized version of the process_node() function, also used in the serial implementation.

Synchronization​

For synchronization you can use mutexes, semaphores, spinlocks, condition variables - anything that grinds your gear. -However, you are not allowed to use hacks such as sleep(), printf() synchronization or adding superfluous computation.

Input Files​

Reading the graphs from the input files is being taken care of the functions implemented in src/os_graph.c. -A graph is represented in input files as follows:

  • First line contains 2 integers N and M: N - number of nodes, M - numbed or edges
  • Second line contains N integer numbers - the values of the nodes.
  • The next M lines contain each 2 integers that represent the source and the destination of an edge.

Data Structures​

Graph​

A graph is represented internally by the os_graph_t structure (see src/os_graph.h).

List​

A list is represented internally by the os_queue_t structure (see src/os_list.h). -You will use this list to implement the task queue.

Thread Pool​

A thread pool is represented internally by the os_threadpool_t structure (see src/os_threadpool.h). -The thread pool contains information about the task queue and the threads.

Requirements​

Your implementation needs to be contained in the src/os_threadpool.c, src/os_threadpool.h and src/parallel.c files. -Any other files that you are using will not be taken into account. -Any modifications that you are doing to the other files in the src/ directory will not be taken into account.

Operations​

Building​

To build both the serial and the parallel versions, run make in the src/ directory:

student@so:~/.../content/assignments/parallel-graph$ cd src/

student@so:~/.../assignments/parallel-graph/src$ make

That will create the serial and parallel binaries.

Testing and Grading​

Testing is automated. -Tests are located in the tests/ directory.

student@so:~/.../assignments/parallel-graph/tests$ ls -F
Makefile  checker.py  grade.sh@  in/

To test and grade your assignment solution, enter the tests/ directory and run grade.sh. -Note that this requires linters being available. -The easiest is to use a Docker-based setup with everything installed and configured. -When using grade.sh you will get grades for checking correctness (maximum 90 points) and for coding style (maxim 10 points). -A successful run will provide you an output ending with:

### GRADE


Checker:                                                         90/ 90
Style:                                                           10/ 10
Total:                                                          100/100


### STYLE SUMMARY


Running the Checker​

To run only the checker, use the make check command in the tests/ directory:

student@so:~/.../assignments/parallel-graph/tests$ make check
[...]
SRC_PATH=../src python checker.py
make[1]: Entering directory '...'
rm -f *~
[...]
TODO
test1.in                         ....................... failed ...   0.0
test2.in                         ....................... failed ...   0.0
test3.in                         ....................... failed ...   0.0
[...]

Total:                                                             0/100

Obviously, all tests will fail, as there is no implementation.

Each test is worth a number of points. -The maximum grade is 90.

A successful run will show the output:

student@so:~/.../assignments/parallel-graph/tests$ make check
[...]
SRC_PATH=../src python checker.py
test1.in                         ....................... passed ...   4.5
test2.in                         ....................... passed ...   4.5
test3.in                         ....................... passed ...   4.5
test4.in                         ....................... passed ...   4.5
test5.in                         ....................... passed ...   4.5
test6.in                         ....................... passed ...   4.5
test7.in                         ....................... passed ...   4.5
test8.in                         ....................... passed ...   4.5
test9.in                         ....................... passed ...   4.5
test10.in                        ....................... passed ...   4.5
test11.in                        ....................... passed ...   4.5
test12.in                        ....................... passed ...   4.5
test13.in                        ....................... passed ...   4.5
test14.in                        ....................... passed ...   4.5
test15.in                        ....................... passed ...   4.5
test16.in                        ....................... passed ...   4.5
test17.in                        ....................... passed ...   4.5
test18.in                        ....................... passed ...   4.5
test19.in                        ....................... passed ...   4.5
test20.in                        ....................... passed ...   4.5

Total:                                                              90/100

Running the Linters​

To run the linters, use the make lint command in the tests/ directory:

student@so:~/.../assignments/parallel-graph/tests$ make lint
[...]
cd .. && checkpatch.pl -f checker/*.sh tests/*.sh
[...]
cd .. && cpplint --recursive src/ tests/ checker/
[...]
cd .. && shellcheck checker/*.sh tests/*.sh

Note that the linters have to be installed on your system: checkpatch.pl, cpplint, shellcheck. -They also need to have certain configuration options. -It's easiest to run them in a Docker-based setup with everything configured.

Fine-Grained Testing​

Input tests cases are located in tests/in/. -If you want to run a single test, use commands such as below while in the src/ directory:

$./parallel ../tests/in/test5.in
-38

$ ./serial ../tests/in/test5.in
-38

Results provided by the serial and parallel implementation must be the same for the test to successfully pass.

- - - - \ No newline at end of file diff --git a/75/Assignments/index.html b/75/Assignments/index.html index 748a539e85..77b3b01c2d 100644 --- a/75/Assignments/index.html +++ b/75/Assignments/index.html @@ -4,13 +4,13 @@ Assignments | Operating Systems - - + +
-
Skip to main content

Assignments

- - +
Skip to main content

Assignments

+ + \ No newline at end of file diff --git a/75/Compute/Questions/apache2-strace/index.html b/75/Compute/Questions/apache2-strace/index.html index 2182014318..12d4538a8d 100644 --- a/75/Compute/Questions/apache2-strace/index.html +++ b/75/Compute/Questions/apache2-strace/index.html @@ -4,14 +4,14 @@ `apache2` Document Root | Operating Systems - - + +
Skip to main content

apache2 Document Root

Question Text​

What is the document root of the apache2 server?

Question Answers​

  • /etc/apache2
  • /usr/local/apache2/htdocs/

  • /var/www/html

  • /var/www/apache2/htdocs

Feedback​

In strace we see that the server opens the file /usr/local/apache2/htdocs/index.html. This means that the document root is /usr/local/apache2/htdocs/.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/cause-of-file-not-found-error/index.html b/75/Compute/Questions/cause-of-file-not-found-error/index.html index 5468ca7593..3dfc20fbf3 100644 --- a/75/Compute/Questions/cause-of-file-not-found-error/index.html +++ b/75/Compute/Questions/cause-of-file-not-found-error/index.html @@ -4,8 +4,8 @@ Cause of `FileNotFoundError` | Operating Systems - - + +
@@ -15,7 +15,7 @@ It's impossible to say what kind of data will be used by the first thread. In our case, the data is the file you give to the script as an argument. If scheduling the parent process or its running time takes long enough, the file may have been created by the time the parent needs it, but we can never be sure.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/child-faults-after-write/index.html b/75/Compute/Questions/child-faults-after-write/index.html index 4880634cd6..928d704f7e 100644 --- a/75/Compute/Questions/child-faults-after-write/index.html +++ b/75/Compute/Questions/child-faults-after-write/index.html @@ -4,13 +4,13 @@ Child Faults After Write | Operating Systems - - + +
Skip to main content

Child Faults After Write

Question Text​

What causes the page faults registered by the child after the fifth step?

Question Answers​

  • The child writes data to the frames it previously shared with its parent and the copy-on-write mechanism copies and remaps them before writing said data

  • Demand paging propagates the lazy allocation of pages from the parent to the child

  • Creating the child process inherently duplicates some frames

  • They are caused by the loader forking itself when creating the child process

  • They are caused by the bash process forking itself when creating the child process

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/coarse-vs-granular-critical-section/index.html b/75/Compute/Questions/coarse-vs-granular-critical-section/index.html index 791b4cb75f..bf4a0db4cb 100644 --- a/75/Compute/Questions/coarse-vs-granular-critical-section/index.html +++ b/75/Compute/Questions/coarse-vs-granular-critical-section/index.html @@ -4,8 +4,8 @@ Coarse vs Granular Critical Section | Operating Systems - - + +
@@ -14,7 +14,7 @@ The second thread then finds the mutex locked and enters the WAITING state. When the first thread finishes its loop, it calls unlock() and wakes up the second thread, which acquires the lock and starts its loop.

In the more granular example, in the worst case, the holder of the mutex can change at every step of the loop. This would mean 1 context switch per step per thread, i.e. 20 million context switches.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/create-sleepy-process-ending/index.html b/75/Compute/Questions/create-sleepy-process-ending/index.html index ade85a6a49..ed85858b86 100644 --- a/75/Compute/Questions/create-sleepy-process-ending/index.html +++ b/75/Compute/Questions/create-sleepy-process-ending/index.html @@ -4,14 +4,14 @@ `create_sleepy` Process Ending | Operating Systems - - + +
Skip to main content

create_sleepy Process Ending

Question Text​

Why does the create_sleepy process wait a very long time before ending? Use system's man page to find the answer.

Question Answers​

  • Because the code is unoptimized (the default optimisation level is -O0)

  • Because the operating system takes very long to finish the process

  • Because system returns when the command given to it (sleep 1000) ends
  • Because the CPU is very slow

Feedback​

The man page says it clearly:

system() returns after the command has been completed.

Therefore, in our case, it returns after sleep 1000 ends.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/exec-without-fork/index.html b/75/Compute/Questions/exec-without-fork/index.html index 91c371ccf9..3f49548445 100644 --- a/75/Compute/Questions/exec-without-fork/index.html +++ b/75/Compute/Questions/exec-without-fork/index.html @@ -4,15 +4,15 @@ exec()` Without `fork() | Operating Systems - - + +
Skip to main content

exec() Without fork()

Question Text​

We type the following command:

student@os:~$ ls

Why does the shell NOT simply call exec("/bin/ls")?

Bash exec

Question Answers​

  • Because /bin/ls may be buggy or vulnerable and compromise the shell
  • Because when ls ends, the shell is closed

  • Because the ls and bash processes must not share the same address space

  • Because bash needs to pass data to the ls process

Feedback​

If the shell simply execs "/bin/ls", the bash process is entirely replaced by the ls process. Therefore, when the ls process ends, the old bash process is no more and the terminal closes. This way, each command would close the terminal.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/fiber-strace/index.html b/75/Compute/Questions/fiber-strace/index.html index e1d6457865..331d8433fe 100644 --- a/75/Compute/Questions/fiber-strace/index.html +++ b/75/Compute/Questions/fiber-strace/index.html @@ -4,14 +4,14 @@ Fiber Strace | Operating Systems - - + +
Skip to main content

Fiber Strace

Question Text​

How many clone() system calls are performed when creating a fiber?

Question Answers​

  • none

  • one for each fiber

  • one for every 2 fibers

  • 2 for each fiber

Feedback​

Being user-level threads, the fibers aren't created by the operating system. The only system calls that you see used are mmap() and munmap(), used to create each fiber's stack.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/index.html b/75/Compute/Questions/index.html index 889eca36dd..a344c39b85 100644 --- a/75/Compute/Questions/index.html +++ b/75/Compute/Questions/index.html @@ -4,13 +4,13 @@ Questions | Operating Systems - - + +
Skip to main content

Questions

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/mini-shell-stops-after-command/index.html b/75/Compute/Questions/mini-shell-stops-after-command/index.html index 6d9023820c..f4f8b1a62b 100644 --- a/75/Compute/Questions/mini-shell-stops-after-command/index.html +++ b/75/Compute/Questions/mini-shell-stops-after-command/index.html @@ -4,8 +4,8 @@ Mini-shell Stops After Command | Operating Systems - - + +
@@ -13,7 +13,7 @@ So when you exec*("ls"), for example, the mini_shell process becomes ls. There is no more mini_shell past this point. So when ls ends, there is no mini_shell process to continue its execution anymore.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/mmap-cow-flag/index.html b/75/Compute/Questions/mmap-cow-flag/index.html index 10e084a6e1..b1ecb53555 100644 --- a/75/Compute/Questions/mmap-cow-flag/index.html +++ b/75/Compute/Questions/mmap-cow-flag/index.html @@ -4,13 +4,13 @@ Copy-on-write Flag for `mmap()` | Operating Systems - - + +
Skip to main content

Copy-on-write Flag for mmap()

Question Text​

From the description in its man page, what flag should we pass to mmap() in order to mark the mapped pages as copy-on-write?

Question Answers​

  • MAP_SHARED
  • MAP_PRIVATE

  • MAP_ANONYMOUS

  • MAP_POPULATE

Feedback​

Quoting the man page:

MAP_PRIVATE
              Create a private copy-on-write mapping.
- - + + \ No newline at end of file diff --git a/75/Compute/Questions/not-race-condition/index.html b/75/Compute/Questions/not-race-condition/index.html index c0d43269e2..6fd31f5e0d 100644 --- a/75/Compute/Questions/not-race-condition/index.html +++ b/75/Compute/Questions/not-race-condition/index.html @@ -4,14 +4,14 @@ Not Race Condition | Operating Systems - - + +
Skip to main content

Not Race Condition

Question Text​

What type of parallel actions never cause race conditions?

Question Answers​

  • Only reads

  • Reads and writes

  • Reads and executions
  • All parallel actions can cause race conditions

Feedback​

Race conditions only appear when a thread writes and another thread depends on the write / update of the other thread. Other types of access (read and execute) do not modify the data, thus they do not cause race conditions.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/notify-only-with-mutex/index.html b/75/Compute/Questions/notify-only-with-mutex/index.html index c0d65f8d49..f60d985bff 100644 --- a/75/Compute/Questions/notify-only-with-mutex/index.html +++ b/75/Compute/Questions/notify-only-with-mutex/index.html @@ -4,8 +4,8 @@ Both Condition and Mutex | Operating Systems - - + +
@@ -13,7 +13,7 @@ For this reason, it is unsafe to only use a mutex as a notification mechanism. In addition, a mutex cannot notify more than one thread at once, if we so desire. Mutexes are only meant to be used to isolate a critical section within the same thread.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/number-of-running-threads/index.html b/75/Compute/Questions/number-of-running-threads/index.html index d2a38116a4..9e552e1057 100644 --- a/75/Compute/Questions/number-of-running-threads/index.html +++ b/75/Compute/Questions/number-of-running-threads/index.html @@ -4,14 +4,14 @@ Number of Running Threads | Operating Systems - - + +
Skip to main content

Number of Running Threads

Question Text​

What is the maximum number of threads that can be in the RUNNING state at any time?

Question Answers​

  • All of them

  • Only one

  • It depends on the ISA

  • As many as the number of CPU cores

Feedback​

A RUNNING thread needs a CPU core on which to run. Therefore, we cannot have more threads in the RUNNING state than the number of CPU cores.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/number-of-running-ults/index.html b/75/Compute/Questions/number-of-running-ults/index.html index c58dc73346..748ed56a88 100644 --- a/75/Compute/Questions/number-of-running-ults/index.html +++ b/75/Compute/Questions/number-of-running-ults/index.html @@ -4,14 +4,14 @@ Number of RUNNING User-Level Threads | Operating Systems - - + +
Skip to main content

Number of RUNNING User-Level Threads

Question Text​

How many threads can be RUNNING simultaneously if we only create them using the API exposed by libult.so?

Question Answers​

  • Equal to the number of cores on the CPU
  • 1

  • None

  • 2: the main thread and another one for the created threads

Feedback​

Only kernel-level threads can run in parallel. Since all libult.so threads are user-level threads, they run within the same kernel-level thread, so only one of them can run at any time.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/parent-faults-before-fork/index.html b/75/Compute/Questions/parent-faults-before-fork/index.html index 8ed86b3457..910fc46333 100644 --- a/75/Compute/Questions/parent-faults-before-fork/index.html +++ b/75/Compute/Questions/parent-faults-before-fork/index.html @@ -4,13 +4,13 @@ Parent Faults before `fork()` | Operating Systems - - + +
Skip to main content

Parent Faults before fork()

Question Text​

What causes the page faults that occur between the first and second steps?

Question Answers​

  • Calling fork() duplicates the pages previously allocated by the parent
  • Demand paging makes the pages in the p array to be mapped to frames only when written

  • The OS duplicates the parent's pages in preparation for fork()

  • mmap() sets the pages to be mapped at a later time, decided by the OS

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/parent-of-sleep-processes/index.html b/75/Compute/Questions/parent-of-sleep-processes/index.html index 5d70abeabe..501398ecb3 100644 --- a/75/Compute/Questions/parent-of-sleep-processes/index.html +++ b/75/Compute/Questions/parent-of-sleep-processes/index.html @@ -4,8 +4,8 @@ Parent of `sleep` Processes | Operating Systems - - + +
@@ -13,7 +13,7 @@ Why?

Question Answers​

Feedback​

When a process dies without waiting for the termination of all its children, those processes are now orphans. Then the systemd process adopts those orphan processes by default. On older Linux systems, it was the init process who adopted orphans.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/process-creation/index.html b/75/Compute/Questions/process-creation/index.html index 74cc1bd5c7..2fd1596f70 100644 --- a/75/Compute/Questions/process-creation/index.html +++ b/75/Compute/Questions/process-creation/index.html @@ -4,13 +4,13 @@ Process Creation | Operating Systems - - + +
Skip to main content

Process Creation

Question Text​

At what stage is a new process created?

Question Answers​

  • At load time

  • At link time

  • At runtime

  • At compile time

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/processes-speedup/index.html b/75/Compute/Questions/processes-speedup/index.html index df6cabb7d4..74f77140fb 100644 --- a/75/Compute/Questions/processes-speedup/index.html +++ b/75/Compute/Questions/processes-speedup/index.html @@ -4,8 +4,8 @@ Processes Speedup | Operating Systems - - + +
@@ -16,7 +16,7 @@ Therefore there will always be parts of any given program that cannot be run in parallel. As a result, the speedup can never be equal to the number of processes between which we spread the workload.

It is possible to compute the speedup obtained from parallelising a portion of a given program. The formula is rather simple and is called Amdahl's law

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/sections-always-shared/index.html b/75/Compute/Questions/sections-always-shared/index.html index 1ed618ee1c..083e28b9e3 100644 --- a/75/Compute/Questions/sections-always-shared/index.html +++ b/75/Compute/Questions/sections-always-shared/index.html @@ -4,8 +4,8 @@ Always Shared Sections | Operating Systems - - + +
@@ -13,7 +13,7 @@ Pages are only copied when written to. If you never write data to a page, it will remain shared. You cannot write data to non-writable pages, such as those in the .text or .rodata sections.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/seg-fault-exit-code/index.html b/75/Compute/Questions/seg-fault-exit-code/index.html index d609cd0649..d1e6ba5eb6 100644 --- a/75/Compute/Questions/seg-fault-exit-code/index.html +++ b/75/Compute/Questions/seg-fault-exit-code/index.html @@ -4,8 +4,8 @@ Segfault Exit Code | Operating Systems - - + +
@@ -13,7 +13,7 @@ We can use the WIFSIGNALED() and WTERMSIG() macros. By doing so, we see the exit code of the faulty child process is 11. We can then use the kill -l command to view the code of each signal and SIGSEGV has the code 11.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/semaphore-equivalent/index.html b/75/Compute/Questions/semaphore-equivalent/index.html index e92aed6a17..39690c4cb7 100644 --- a/75/Compute/Questions/semaphore-equivalent/index.html +++ b/75/Compute/Questions/semaphore-equivalent/index.html @@ -4,14 +4,14 @@ Semaphore Equivalent | Operating Systems - - + +
Skip to main content

Semaphore Equivalent

Question Text​

From running and inspecting the code in support/apache2-simulator/apache2_simulator_semaphore.py, which of the following is an an equivalent to the value of the semaphore sem?

Question Answers​

  • The value of msg_mutex

  • The time a worker thread has to wait before running

  • The length of the messages list
  • The number of worker threads

Feedback​

sem is incremented (release()) upon adding a message to the messages list and decremented (acquire()) when removing a message from said list. So it's a rough equivalent to the length of this list.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/sleeping-on-a-fiber/index.html b/75/Compute/Questions/sleeping-on-a-fiber/index.html index 96c55c49fc..1da31048bc 100644 --- a/75/Compute/Questions/sleeping-on-a-fiber/index.html +++ b/75/Compute/Questions/sleeping-on-a-fiber/index.html @@ -4,14 +4,14 @@ Sleeping on a Fiber | Operating Systems - - + +
Skip to main content

Sleeping on a Fiber

Question Text​

What happens if a fiber calls sleep()?

Question Answers​

  • the whole kernel-level thread is blocked

  • only that fiber is blocked, and is scheduled out

  • nothing, a fiber can't sleep

  • the whole process sleeps - regardless of how many threads there are

Feedback​

The whole thread on which the fiber runs is blocked until the sleep() call is finished. For this reason, the fibers are best used with asynchronous operations, which you will explore in the following weeks.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/tcb-libult-unikraft/index.html b/75/Compute/Questions/tcb-libult-unikraft/index.html index 17ec87d11a..fcc262b874 100644 --- a/75/Compute/Questions/tcb-libult-unikraft/index.html +++ b/75/Compute/Questions/tcb-libult-unikraft/index.html @@ -4,8 +4,8 @@ Similarities Between the TCBs of `libult` and Unikraft | Operating Systems - - + +
@@ -14,7 +14,7 @@ argument and arg are pointers to the arguments of start_routine and entry, respectively. context and ctx are the contexts in which the new threads run. return_value and prv are both pointers to the values returned by the thread functions.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/thread-memory/index.html b/75/Compute/Questions/thread-memory/index.html index 242f0e35dc..f6933449fa 100644 --- a/75/Compute/Questions/thread-memory/index.html +++ b/75/Compute/Questions/thread-memory/index.html @@ -4,14 +4,14 @@ Thread Memory | Operating Systems - - + +
Skip to main content

Thread Memory

Question Text​

Is data used by a thread inaccessible from other threads?

Question Answers​

  • Yes, each thread has its own stack
  • No, each thread can access every address from the virtual address space

  • Only the heap is shared

  • Only the heap and the read-only zones are shared

Feedback​

Each thread has the same perspective on the system memory - it thinks it owns it all. Therefore, it can perform pointer arithmetic to access every memory address.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/threads-shared-data/index.html b/75/Compute/Questions/threads-shared-data/index.html index 7aa1c30ab2..32896fe411 100644 --- a/75/Compute/Questions/threads-shared-data/index.html +++ b/75/Compute/Questions/threads-shared-data/index.html @@ -4,15 +4,15 @@ Threads Shared Data | Operating Systems - - + +
Skip to main content

Threads Shared Data

Question Text​

Which section may store per-thread non-shared (variables) variables?

Question Answers​

  • On the heap

  • In the .rodata section

  • On the stack
  • In the .text section

Feedback​

Each thread has its own separate stack. Every other section is shared with the other threads. Data can also be stored in the .text section, albeit this is a bit unorthodox.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/time-slice-value/index.html b/75/Compute/Questions/time-slice-value/index.html index f632d6b511..fb78cec4fc 100644 --- a/75/Compute/Questions/time-slice-value/index.html +++ b/75/Compute/Questions/time-slice-value/index.html @@ -4,13 +4,13 @@ Time Slice Value | Operating Systems - - + +
Skip to main content

Time Slice Value

Question Text​

Using the man page, what is the time slice used by the scheduler in libult.so?

Question Answers​

  • 100 milliseconds

  • 10 microseconds

  • 100 microseconds

  • 10 milliseconds

Feedback​

The code we're interested in lies in the function init_profiling_timer():

const struct itimerval timer = {
    { 0, 10000 },
    { 0, 1 }  // arms the timer as soon as possible
};

The man page gives the following definition the struct itimerval:

struct itimerval {
    struct timeval it_interval; /* Interval for periodic timer */
    struct timeval it_value;    /* Time until next expiration */
};

struct timeval {
    time_t      tv_sec;         /* seconds */
    suseconds_t tv_usec;        /* microseconds */
};

So when constructing the timer variable, { 0, 10000 } means 0 seconds and 10000 microseconds, i.e. 0 seconds and 10 milliseconds.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/tls-synchronization/index.html b/75/Compute/Questions/tls-synchronization/index.html index be0c6da738..5693df4e20 100644 --- a/75/Compute/Questions/tls-synchronization/index.html +++ b/75/Compute/Questions/tls-synchronization/index.html @@ -4,14 +4,14 @@ TLS Synchronization | Operating Systems - - + +
Skip to main content

TLS Synchronization

Question Text​

Is placing var from support/race-condition/c/race_condition_tls.c in the TLS a valid form of synchronization?

Question Answers​

  • No, because the race condition remains. It just doesn't manifest itself anymore
  • No, because the threads now access different variables, not the same one

  • Yes, because we now remove the race condition

  • Yes, because now the result is correct

Feedback​

Synchronization means that both threads should access the same variable, whereas placing it in the TLS makes each of them access a copy of the variable.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/tls-var-copies/index.html b/75/Compute/Questions/tls-var-copies/index.html index 5e9b7fc1e8..77af67dd29 100644 --- a/75/Compute/Questions/tls-var-copies/index.html +++ b/75/Compute/Questions/tls-var-copies/index.html @@ -4,13 +4,13 @@ TLS `var` Copies | Operating Systems - - + +
Skip to main content

TLS var Copies

Question Text​

How many copies of the var variable from support/race-condition/c/race_condition_tls.c are there after each thread has modified it at leas once?

Question Answers​

  • 1

  • 2

  • 3
  • 5

Feedback​

There are 3 copies one for the main() thread, another one for increment_var() and the third for decrement_var().

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/type-of-scheduler-in-libult/index.html b/75/Compute/Questions/type-of-scheduler-in-libult/index.html index 7576a80f2b..2a50a3c91f 100644 --- a/75/Compute/Questions/type-of-scheduler-in-libult/index.html +++ b/75/Compute/Questions/type-of-scheduler-in-libult/index.html @@ -4,8 +4,8 @@ Type of Scheduler in `libult.so` | Operating Systems - - + +
@@ -13,7 +13,7 @@ Which type of scheduler does libult.so use?

Question Answers​

Feedback​

libult.so uses a preemptive scheduler. Its timer is initialised in the init_profiling_timer() function. The context switch is performed in the handle_sigprof() function.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/ult-thread-ids/index.html b/75/Compute/Questions/ult-thread-ids/index.html index 322dc5e2fb..044a595067 100644 --- a/75/Compute/Questions/ult-thread-ids/index.html +++ b/75/Compute/Questions/ult-thread-ids/index.html @@ -4,8 +4,8 @@ ULT Thread IDs | Operating Systems - - + +
@@ -15,7 +15,7 @@ This is needed in order to associate a ucontext_t with the main thread as well, so the main thread can also be run.

Feedback​

The threads_create() function calls init_first_context(), which, in turn, calls tcb_new(), thus creating the first context associated with the main thread (the one calling threads_create() the first time). Without this, the scheduler in libult.so wouldn't be able to run the main thread.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/who-calls-execve-parent/index.html b/75/Compute/Questions/who-calls-execve-parent/index.html index 21e21926bc..bb725b9efe 100644 --- a/75/Compute/Questions/who-calls-execve-parent/index.html +++ b/75/Compute/Questions/who-calls-execve-parent/index.html @@ -4,13 +4,13 @@ Who Calls `execve` in the Log of the Parent Process? | Operating Systems - - + +
Skip to main content

Who Calls execve in the Log of the Parent Process?

Question Text​

Which process calls execve("sleepy_creator", ["sleepy_creator"], ...), that you found in the log of the parent process?

Question Answers​

  • The kernel because that's where the loader is located

  • The loader because it is the loader who creates new processes

  • The C runtime because this is the C interpreter

  • bash because we run sleepy_creator from terminal, i.e. from bash

Feedback​

All processes spawned from the command-line are children of the current bash process.

- - + + \ No newline at end of file diff --git a/75/Compute/Questions/why-use-completed-queue/index.html b/75/Compute/Questions/why-use-completed-queue/index.html index 9b9f38c850..19c6934aa9 100644 --- a/75/Compute/Questions/why-use-completed-queue/index.html +++ b/75/Compute/Questions/why-use-completed-queue/index.html @@ -4,8 +4,8 @@ The Need for a COMPLETED Queue | Operating Systems - - + +
@@ -14,7 +14,7 @@ It is used by threads_create() to start executing the given function. This is a wrapper that calls the function associated with the thread (this->start_routine), saves its result and then calls threads_exit() to store this result in the COMPLETED queue.

- - + + \ No newline at end of file diff --git a/75/Compute/compute-overview/index.html b/75/Compute/compute-overview/index.html index 4550e26073..2c78b0c427 100644 --- a/75/Compute/compute-overview/index.html +++ b/75/Compute/compute-overview/index.html @@ -4,13 +4,13 @@ Compute | Operating Systems - - + +
Skip to main content
- - + + \ No newline at end of file diff --git a/75/Compute/index.html b/75/Compute/index.html index 5c67cf844f..9b66e243ab 100644 --- a/75/Compute/index.html +++ b/75/Compute/index.html @@ -4,13 +4,13 @@ Compute | Operating Systems - - + +
Skip to main content

Compute


tip

Focus the slides and press F for fullscreen viewing.

- - + + \ No newline at end of file diff --git a/75/Compute/lab6/index.html b/75/Compute/lab6/index.html index ec307a3ebf..4bb0713091 100644 --- a/75/Compute/lab6/index.html +++ b/75/Compute/lab6/index.html @@ -4,8 +4,8 @@ Lab 6 - Multiprocess and Multithread | Operating Systems - - + +
@@ -45,19 +45,7 @@ Keep in mind that you cannot change its definition. Bonus points if you do not use the thread's ID as the sleeping amount.

Task: Libraries for Parallel Processing​

In chapters/compute/threads/drills/tasks/sum-array/support/c/sum_array_threads.c we spawned threads "manually" by using the pthread_create() function. This is not a syscall, but a wrapper over the common syscall used by both fork() (which is also not a syscall) and pthread_create().

Still, pthread_create() is not yet a syscall. -In order to see what syscall pthread_create() uses, check out this section.

Most programming languages provide a more advanced API for handling parallel computation.

std.parallelism in D​

D language's standard library exposes the std.parallelism, which provides a series of parallel processing functions. -One such function is reduce(), which splits an array between a given number of threads and applies a given operation to these chunks. -In our case, the operation simply adds the elements to an accumulator: a + b. -Follow and run the code in sum-array/support/d/sum_array_threads_reduce.d.

The number of threads is used within a TaskPool. -This structure is a thread manager (not scheduler). -It silently creates the number of threads we request and then reduce() spreads its workload between these threads.

OpenMP for C​

Unlike D, C does not support parallel computation by design. -It needs a library to do advanced things, like reduce() from D. -We have chosen to use the OpenMP library for this. -Follow the code in support/sum-array/c/sum_array_threads_openmp.c.

The #pragma used in the code instructs the compiler to enable the omp module, and to parallelise the code. -In this case, we instruct the compiler to perform a reduce of the array, using the + operator, and to store the results in the result variable. -This reduction uses threads to calculate the sum, similar to sum_array_threads.c, but in a much more optimised form.

Now compile and run the sum_array_threads_openmp binary using 1, 2, 4, and 8 threads as before. -You'll see lower running times than sum_array_threads due to the highly-optimised code emitted by the compiler. -For this reason and because library functions are usually much better tested than your own code, it is always preferred to use a library function for a given task.

Array Sum in Python​

Let's first probe this by implementing two parallel versions of the code in sum-array/support/python/sum_array_sequential.py. +In order to see what syscall pthread_create() uses, check out this section.

Most programming languages provide a more advanced API for handling parallel computation.

Array Sum in Python​

Let's first probe this by implementing two parallel versions of the code in sum-array/support/python/sum_array_sequential.py. One version should use threads and the other should use processes. Run each of them using 1, 2, 4, and 8 threads / processes respectively and compare the running times. Notice that the running times of the multithreaded implementation do not decrease. @@ -177,7 +165,28 @@ This slight time difference is caused by process creation actions, which are costlier than thread creation actions. Because a process needs a separate virtual address space (VAS) and needs to duplicate some internal structures such as the file descriptor table and page table, it takes the operating system more time to create it than to create a thread. On the other hand, threads belonging to the same process share the same VAS and, implicitly, the same OS-internal structures. -Therefore, they are more lightweight than processes.

Guide: Threads and Processes: clone​

Let's go back to our initial demos that used threads and processes. +Therefore, they are more lightweight than processes.

std.parallelism in D​

D language's standard library exposes the std.parallelism, which provides a series of parallel processing functions. +One such function is reduce(), which splits an array between a given number of threads and applies a given operation to these chunks. +In our case, the operation simply adds the elements to an accumulator: a + b. +Follow and run the code in chapters/compute/threads/guides/sum-array-threads/support/d/sum_array_threads_reduce.d.

The number of threads is used within a TaskPool. +This structure is a thread manager (not scheduler). +It silently creates the number of threads we request and then reduce() spreads its workload between these threads.

Now that you've seen how parallelism works in D, go in chapters/compute/threads/guides/sum-array-threads/support/java/SumArrayThreads.java and follow the TODOs. +The code is similar to the one written in D, and it uses ThreadPoolExecutor. +More about that here. +To run the code use:

javac SumArrayThreads.java
java SumArrayThreads 4

4 is the number of threads used, but you can replace the value with a number less or equal than your available cores.

OpenMP for C​

Unlike D, C does not support parallel computation by design. +It needs a library to do advanced things, like reduce() from D. +We have chosen to use the OpenMP library for this. +Follow the code in chapters/compute/threads/guides/sum-array-threads/support/c/sum_array_threads_openmp.c.

The #pragma used in the code instructs the compiler to enable the omp module, and to parallelise the code. +In this case, we instruct the compiler to perform a reduce of the array, using the + operator, and to store the results in the result variable. +This reduction uses threads to calculate the sum, similar to summ_array_threads.c, but in a much more optimised form.

One of the advantages of OpenMP is that is relatively easy to use. +The syntax requires only a few additional lines of code and compiler options, thus converting sequential code into parallel code quickly. +For example, using #pragma omp parallel for, a developer can parallelize a for loop, enabling iterations to run across multiple threads.

OpenMP uses a shared-memory model, meaning all threads can access a common memory space. +This model is particularly useful for tasks that require frequent access to shared data, as it avoids the overhead of transferring data between threads. +However, shared memory can also introduce challenges, such as race conditions or synchronization issues, which can occur when multiple threads attempt to modify the same data simultaneously, but we'll talk about that later. +OpenMP offers constructs such as critical sections, atomic operations, and reductions to help manage these issues and ensure that parallel code executes safely and correctly.

Now compile and run the sum_array_threads_openmp binary using 1, 2, 4, and 8 threads as before. +You'll see lower running times than sum_array_threads due to the highly-optimised code emitted by the compiler. +For this reason and because library functions are usually much better tested than your own code, it is always preferred to use a library function for a given task.

For a challenge, enter chapters/compute/threads/guides/sum-array-threads/support/c/add_array_threads_openmp.c. +Use what you've learned from the previous exercise and add the value 100 to an array using OpenMP.

Guide: Threads and Processes: clone​

Let's go back to our initial demos that used threads and processes. We'll see that in order to create both threads and processes, the underlying Linux syscall is clone. For this, we'll run both sum_array_threads and sum_array_processes under strace. As we've already established, we're only interested in the clone syscall:

student@os:~/.../sum-array/support/c$ strace -e clone,clone3 ./sum_array_threads 2
clone(child_stack=0x7f60b56482b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819693], tls=0x7f60b5649640, child_tidptr=0x7f60b5649910) = 1819693
clone(child_stack=0x7f60b4e472b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819694], tls=0x7f60b4e48640, child_tidptr=0x7f60b4e48910) = 1819694

student@os:~/.../sum-array/support/c$ strace -e clone,clone3 ./sum_array_processes 2
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820599
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820600

We ran each program with an argument of 2, so we have 2 calls to clone. @@ -185,8 +194,8 @@ The relevant flags passed as arguments when creating threads are documented in clone's man page:

By contrast, when creating a new process, the arguments of the clone syscall are simpler (i.e. fewer flags are present). Remember that in both cases clone creates a new thread. -When creating a process, clone creates this new thread within a new separate address space.

- - +When creating a process, clone creates this new thread within a new separate address space.

+ + \ No newline at end of file diff --git a/75/Compute/lab7/index.html b/75/Compute/lab7/index.html index 8d7d369b90..185d985b9c 100644 --- a/75/Compute/lab7/index.html +++ b/75/Compute/lab7/index.html @@ -3,15 +3,13 @@ -Lab 7 - Copy-on-Write | Operating Systems - - +Lab 7 - Copy-on-Write | Operating Systems + +
-
Skip to main content

Lab 7 - Copy-on-Write

Task: Investigate apache2 Using strace​

Enter the chapters/compute/processes-threads-apache2/drills/tasks/apache2/support/ folder and go through the practice items below.

  1. Use strace to discover the server document root. -The document root is the path in the filesystem from where httpd serves all the files requested by the clients.

First, you will have to stop the running container using make stop, then restart it with make run-privileged.

  1. Use strace inside the container to attach to the worker processes (use the -p option for this). -You will also have to use the -f flag with strace, so that it will follow all the threads inside the processes.

  2. After you have attached successfully to all worker processes, use the curl command to send a request.

  3. Then check the strace output to see what files were opened by the server.

Quiz

If you're having difficulties solving this exercise, go through this reading material.

Task: Minor and Major Page Faults​

The code in chapters/compute/copy-on-write/drills/tasks/page-faults/support/page_faults.c generates some minor and major page faults. +

Lab 7 - Copy-on-Write

Task: Minor and Major Page Faults​

The code in chapters/compute/copy-on-write/drills/tasks/page-faults/support/page_faults.c generates some minor and major page faults. Open 2 terminals: one in which you will run the program, and one which will monitor the page faults of the program. In the monitoring terminal, run the following command:

watch -n 1 'ps -eo min_flt,maj_flt,cmd | grep ./page_faults | head -n 1'

Compile the program and run it in the other terminal. You must press enter one time, before the program will prompt you to press enter more times. @@ -31,7 +29,15 @@ These types of page faults happen in 2 situations:

  • a page that was swapped out (to the disk), due to lack of memory, is now accessed - this case is harder to show
  • the OS needs to read a file from the disk, because the file contents aren't present in the cache - the case we are showing now

Press enter to print the file contents. Note the second number go up in the monitoring terminal.

Comment the posix_fadvise() call, recompile the program, and run it again. You won't get any major page fault, because the file contents are cached by the OS, to avoid those page faults. -As a rule, the OS will avoid major page faults whenever possible, because they are very costly in terms of running time.

If you're having difficulties solving this exercise, go through this reading material.

Task: Shared Memory​

Navigate to the chapters/compute/copy-on-write/drills/tasks/shared-memory/ directory, run make skels to generate the support/ folder, enter the support/src/ folder, open shared_memory.c and go through the practice items below.

Use the support/tests/checker.sh script to check your solution.

./checker.sh
mmap ............................ passed ... 25
sem_wait ........................ passed ... 25
sem_post ........................ passed ... 25
match value ..................... passed ... 25
Total: 100 / 100

As you remember from the Data chapter, one way to allocate a given number of pages is to use the mmap() syscall.

Let's look at its man page, specifically at the flags argument. +As a rule, the OS will avoid major page faults whenever possible, because they are very costly in terms of running time.

If you're having difficulties solving this exercise, go through this reading material.

Task: Mini-shell​

As you might remember, to create a new process you need to use fork (or clone) and exec system calls. +If you don't, take a look at what happens under the hood when you use system.

Enter the chapters/compute/processes/drills/tasks/mini-shell directory, run make skels, open the support/src folder and go through the practice items below.

Use the tests/checker.sh script to check your solutions.

./checker.sh
mini_shell: ls ................ passed ... 50
mini_shell: pwd ................ passed ... 25
mini_shell: echo hello ................ passed ... 25
100 / 100

  1. With this knowledge in mind, let's implement our own mini-shell.

    Start from the skeleton code in mini_shell.c. +We're already running our Bash interpreter from the command-line, so there's no need to exec another Bash from it.

    Simply exec the command.

    Quiz

    So we need a way to "save" the mini_shell process before exec()-ing our command. +Find a way to do this.

    Hint: You can see what sleepy does and draw inspiration from there. +Use strace to also list the calls to clone() performed by sleepy or its children. +Remember what clone() is used for and use its parameters to deduce which of the two scenarios happens to sleepy.

Task: Investigate apache2 Using strace​

Enter the chapters/compute/processes-threads-apache2/drills/tasks/apache2/support/ folder and go through the practice items below.

  1. Use make run to start the container. +Use strace inside the container to discover the server document root. +The document root is the path in the filesystem from where httpd serves all the files requested by the clients.

First, you will have to stop the running container using make stop, then restart it with make run-privileged.

  1. Use strace inside the container to attach to the worker processes (use the -p option for this). +You will also have to use the -f flag with strace, so that it will follow all the threads inside the processes.

  2. After you have attached successfully to all worker processes, use the curl command to send a request.

  3. Then check the strace output to see what files were opened by the server.

Quiz

If you're having difficulties solving this exercise, go through this reading material.

Task: Shared Memory​

Navigate to the chapters/compute/copy-on-write/drills/tasks/shared-memory/ directory, run make skels to generate the support/ folder, enter the support/src/ folder, open shared_memory.c and go through the practice items below.

Use the support/tests/checker.sh script to check your solution.

./checker.sh
mmap ............................ passed ... 25
sem_wait ........................ passed ... 25
sem_post ........................ passed ... 25
match value ..................... passed ... 25
Total: 100 / 100

As you remember from the Data chapter, one way to allocate a given number of pages is to use the mmap() syscall.

Let's look at its man page, specifically at the flags argument. Its main purpose is to determine the way in which child processes interact with the mapped pages.

Quiz

Now let's test this flag, as well as its opposite: MAP_SHARED. Compile and run the code in shared-memory/support/src/shared_memory.c.

  1. See the value read by the parent is different from that written by the child. Modify the flags parameter of mmap() so they are the same.

  2. Create a semaphore in the shared page and use it to make the parent signal the child before it can exit. @@ -40,12 +46,7 @@ This only works between "related" processes. If you want to share a semaphore or other types of memory between any two processes, you need filesystem support. For this, you should use named semaphores, created using sem_open(). -You'll get more accustomed to such functions in the [Application Interaction chapter].

Task: Mini-shell​

As you might remember, to create a new process you need to use fork (or clone) and exec system calls. -If you don't, take a look at what happens under the hood when you use system.

Enter the chapters/compute/processes/drills/tasks/mini-shell directory, run make skels, open the support/src folder and go through the practice items below.

Use the tests/checker.sh script to check your solutions.

./checker.sh
mini_shell: ls ................ passed ... 50
mini_shell: pwd ................ passed ... 25
mini_shell: echo hello ................ passed ... 25
100 / 100

  1. With this knowledge in mind, let's implement our own mini-shell.

    Start from the skeleton code in mini_shell.c. -We're already running our Bash interpreter from the command-line, so there's no need to exec another Bash from it.

    Simply exec the command.

    Quiz

    So we need a way to "save" the mini_shell process before exec()-ing our command. -Find a way to do this.

    Hint: You can see what sleepy does and draw inspiration from there. -Use strace to also list the calls to clone() performed by sleepy or its children. -Remember what clone() is used for and use its parameters to deduce which of the two scenarios happens to sleepy.

Usage of Processes and Threads in apache2​

We'll take a look at how a real-world application - the apache2 HTTP server - makes use of processes and threads. +You'll get more accustomed to such functions in the [Application Interaction chapter].

Usage of Processes and Threads in apache2​

We'll take a look at how a real-world application - the apache2 HTTP server - makes use of processes and threads. Since the server must be able to handle multiple clients at the same time, it must therefore use some form of concurrency. When a new client arrives, the server offloads the work of interacting with that client to another process or thread.

The choice of whether to use multiple processes or threads is not baked into the code. Instead, apache2 provides a couple of modules called MPMs (Multi-Processing Modules). @@ -92,8 +93,8 @@ The first one still corresponds to the parent.

Quiz 2

Now it should be clear how demand paging differs from copy-on-write. Shared memory is a similar concept. It's a way of marking certain allocated pages so that copy-on-write is disabled. -As you may imagine, changes made by the parent to this memory are visible to the child and vice-versa.

- - +As you may imagine, changes made by the parent to this memory are visible to the child and vice-versa.

+ + \ No newline at end of file diff --git a/75/Compute/lab8/index.html b/75/Compute/lab8/index.html index c0cf9ebc28..81f0ed3e57 100644 --- a/75/Compute/lab8/index.html +++ b/75/Compute/lab8/index.html @@ -3,59 +3,14 @@ -Lab 8 - Syncronization | Operating Systems - - +Lab 8 - Syncronization | Operating Systems + +
-
Skip to main content

Lab 8 - Syncronization

Task: Race Conditions​

C - TLS on Demand​

The perspective of C towards TLS is the following: everything is shared by default. -This makes multithreading easier and more lightweight to implement than in other languages, like D, because synchronization is left entirely up to the developer, at the cost of potential unsafety.

Of course, we can specify that some data belongs to the TLS, by preceding the declaration of a variable with __thread keyword. -First, compile and run the code in chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition_tls.c a few times. -As expected, the result is different each time.

  1. Modify the declaration of var and add the __thread keyword to place the variable in the TLS of each thread. -Recompile and run the code a few more times. -You should see that in the end, var is 0.

    Quiz 1

    Quiz 2

  2. Print the address and value of var in each thread. -See that they differ.

  3. Modify the value of var in the main() function before calling pthread_create(). -Notice that the value doesn't propagate to the other threads. -This is because, upon creating a new thread, its TLS is initialised.

Atomic Assembly​

No, this section is not about nukes, sadly :(. -Instead, we aim to get accustomed to the way in which the x86 ISA provides atomic instructions.

This mechanism looks very simple. -It is but one instruction prefix: lock. -It is not an instruction with its own separate opcode, but a prefix that slightly modifies the opcode of the following instructions to make the CPU execute it atomically (i.e. with exclusive access to the data bus).

lock must only be place before an instruction that executes a read-modify-write action. -For example, we cannot place it before a mov instruction, as the action of a mov is simply read or write. -Instead, we can place it in front of an inc instruction if its operand is memory.

Go to chapters/compute/synchronization/drills/tasks/race-condition/support/. -Look at the code in asm/race_condition_lock.S. -It's an Assembly equivalent of the code you've already seen many times so far (such as c/race_condition.c). -Assemble and run it a few times. -Notice the different results you get.

Now add the lock prefix before inc and dec. -Reassemble and rerun the code. -And now we have synchronized the two threads by leveraging CPU support.

Task: Synchronization - Thread-Safe Data Structure​

Now it's time for a fully practical exercise. -Go to chapters/compute/synchronization/drills/tasks/threadsafe-data-struct/support/. -In the file clist.c you'll find a simple implementation of an array list. -Although correct, it is not (yet) thread-safe.

The code in test.c verifies its single-threaded correctness, while the one in test_parallel.c verifies it works properly with multiple threads. -Your task is to synchronize this data structure using whichever primitives you like. -Try to keep the implementation efficient. -Aim to decrease your running times as much as you can.

Task: Another Time Slice​

Enter the chapters/compute/scheduling/drills/tasks/libult/support/ folder and go through the practice items below.

  1. Modify the time slice set to the timer to 2 seconds. -Re-run the code in test_ult.c. -Notice that now no context switch happens between the 2 created threads because they end before the timer can fire.

  2. Now change the printer_thread() function in test_ult.c to make it run for more than 2 seconds. -See that now the prints from the two threads appear intermingled. -Add prints to the handle_sigprof() function in threads.c to see the context switch happen.

If you're having difficulties solving this exercise, go through this reading material.

Synchronization​

So far, we've used threads and processes without wondering how to "tell" them how to access shared data. -Moreover, in order to make threads wait for each other, we simply had the main thread wait for the others to finish all their work. -But what if we want one thread to wait until another one simply performs some specific action, after which it resumes its execution? -For this, we need to use some more complex synchronization mechanisms.

Race Conditions​

For example, what if one thread wants to increase a global variable while another one wants to decrease it? -Let's say the assembly code for increasing and decreasing the variable looks like the one in the snippet below.

increase:
    mov eax, [var]
    inc eax
    mov [var], eax

decrease:
    mov eax, [var]
    dec eax
    mov [var], eax

Imagine both threads executed mov eax, [var] at the same time. -Then each would independently increase its (non-shared) eax register. -In the end, the final value of var depends on which thread executes mov [var], eax last. -So it's kind of a reversed race. -The thread that runs the slowest "wins" this race and writes the final value of var. -But this is up to the scheduler and is non-deterministic. -Such undefined behaviours can cripple the execution of a program if var is some critical variable.

Let's see this bug in action. -Go to chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition.c, compile and run the code a few times. -It spawns to threads that do exactly what we've talked about so far: one thread increments var 10 million times, while the other decrements it 10 million times.

As you can see from running the program, the differences between subsequent runs can be substantial. -To fix this, we must ensure that only one thread can execute either var++ or var-- at any time. -We call these code sections critical sections. -A critical section is a piece of code that can only be executed by one thread at a time. -So we need some sort of mutual exclusion mechanism so that when one thread runs the critical section, the other has to wait before entering it. -This mechanism is called a mutex, whose name comes from "mutual exclusion".

Go to chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition_mutex.c and notice the differences between this code and the buggy one.

We now use a pthread_mutex_t variable, which we lock at the beginning of a critical section, and we unlock at the end. +

Lab 8 - Syncronization

Task: C: Race Conditions​

Go to chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition_mutex.c and notice the differences between this code and the buggy one. +We now use a pthread_mutex_t variable, which we lock at the beginning of a critical section, and we unlock at the end. Generally speaking, lock-ing a mutex makes a thread enter a critical section, while calling pthread_mutex_unlock() makes the thread leave said critical section. Therefore, as we said previously, the critical sections in our code are var-- and var++. Run the code multiple times to convince yourself that in the end, the value of var will always be 0.

Mutexes contain an internal variable which can be either 1 (locked) or 0 (unlocked). @@ -63,13 +18,13 @@ If it was 0, the thread sets it to 1 and proceeds to execute the critical section. Otherwise, it suspends its execution and waits until that variable is set to 0 again.

When calling pthread_mutex_unlock(), the internal variable is set to 0 and all waiting threads are woken up to try to acquire the mutex again. Be careful: It is generally considered unsafe and in many cases undefined behaviour to call pthread_mutex_unlock() from a different thread than the one that acquired the lock. -So the general workflow should look something like this:

within a single thread:
    pthread_mutex_lock(&mutex)
    // do atomic stuff
    pthread_mutex_unlock(&mutex)

Synchronization - Overhead​

There ain't no such thing as a free lunch

This saying is also true for multithreading. +So the general workflow should look something like this:

within a single thread:
    pthread_mutex_lock(&mutex)
    // do atomic stuff
    pthread_mutex_unlock(&mutex)

Synchronization - Overhead​

There ain't no such thing as a free lunch

This saying is also true for multithreading. Running threads in parallel is nice and efficient, but synchronization always comes with a penalty: overhead. Use the time command to record the running times of race_condition and race_condition_mutex. Notice that those of race_condition_mutex are larger than those of race_condition.

The cause of this is that now when one thread is executing the critical section, the other has to wait and do nothing. Waiting means changing its state from RUNNING to WAITING, which brings further overhead from the scheduler. -This latter overhead comes from the context switch that is necessary for a thread to switch its state from RUNNING to WAITING and back.

Practice: Wrap the Whole for Statements in Critical Sections​

Move the calls to pthread_mutex_lock() and pthread_mutex_unlock() outside the for statements so that the critical sections become the entire statement. -Measure the new time spent by the code and compare it with the execution times recorded when the critical sections were made up of only var-- and var++.

Quiz

Atomics​

So now we know how to use mutexes. +This latter overhead comes from the context switch that is necessary for a thread to switch its state from RUNNING to WAITING and back.

Task: Wrap the Whole for Statements in Critical Sections​

Navigate to the chapters/compute/synchronization/drills/tasks/wrap-the-for/ directory, run make skels and open the support/src directory.

Here you will find two source files:

  • race_condition_inner_mutex
  • race_condition_outer_mutex

Following the pattern used in race_condition_inner_mutex.c go in race_condition_outer_mutex.c move the calls to pthread_mutex_lock() and pthread_mutex_unlock() outside the for statements so that the critical sections become the entire statement. +To see the time difference between the two implementations all you have to do is enter tests/ and run the checker.

cd ../tests/
./checker.sh

You should see in the terminal:

Test passed

Quiz

Task: C: Atomics​

So now we know how to use mutexes. And we know that mutexes work by using an internal variable that can be either 1 (locked) or 0 (unlocked). But how does pthread_mutex_lock() actually set that variable to 1? How does it avoid a race condition in case another thread also wants to set it to 1?

We need a guarantee that anyone "touching" that variable does so "within its own critical section". @@ -82,18 +37,30 @@ In particular, this instruction is a prefix, called lock. It makes the instruction that follows it run atomically. The lock prefix ensures that the core performing the instruction has exclusive ownership of the cache line from where the data is transferred for the entire operation. -This is how the increment is made into an indivisible unit.

For example, inc dword [x] can be made atomic, like so: lock inc dword [x]. -You can play with the lock prefix in this race conditions task.

Compilers provide support for such hardware-level atomic operations. +This is how the increment is made into an indivisible unit.

For example, inc dword [x] can be made atomic, like so: lock inc dword [x].

Compilers provide support for such hardware-level atomic operations. GCC exposes built-ins such as __atomic_load(), __atomic_store(), __atomic_compare_exchange() and many others. -All of them rely on the mechanism described above.

Go to chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition_atomic.c and complete the function decrement_var(). +All of them rely on the mechanism described above.

Go to chapters/compute/synchronization/drills/tasks/race-condition-atomic/ and run:

make skels

Now enter chapters/compute/synchronization/drills/tasks/race-condition-atomic/support/src/race_condition_atomic.c and complete the function decrement_var(). Compile and run the code. -Now measure its running time against the mutex implementations. -It should be somewhere between race_condition and race_condition_mutex.

The C standard library also provides atomic data types. +Its running time should be somewhere between race_condition and race_condition_mutex.

The C standard library also provides atomic data types. Access to these variables can be done only by one thread at a time. -Go to chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition_atomic2.c, compile and run the code. -Now measure its running time against the other implementations. -Notice that the time is similar to race_condition_atomic.

So using the hardware support is more efficient, but it usually is leveraged only for simple, individual instructions, such as loads and stores. -And the fact that high-level languages also expose an API for atomic operations shows how useful these operations are for developers.

Semaphores​

Up to now, we've learned how to create critical sections that can be accessed by only one thread at a time. +Go to chapters/compute/synchronization/drills/tasks/race-condition-atomic/support/race_condition_atomic2.c, compile and run the code.

After both tasks are done, go in the checker folder and run it using the following commands:

cd ../tests/
./checker.sh

Notice that the time is similar to race_condition_atomic.

So using the hardware support is more efficient, but it usually is leveraged only for simple, individual instructions, such as loads and stores. +And the fact that high-level languages also expose an API for atomic operations shows how useful these operations are for developers.

Task: C - TLS on Demand​

The perspective of C towards TLS is the following: everything is shared by default. +This makes multithreading easier and more lightweight to implement than in other languages, like D, because synchronization is left entirely up to the developer, at the cost of potential unsafety.

Of course, we can specify that some data belongs to the TLS, by preceding the declaration of a variable with __thread keyword. +Enter chapters/compute/synchronization/drills/tasks/tls-on-demand/ and run make skels. +Now enter support/src and follow the TODOs.

  1. Create the declaration of var and add the __thread keyword to place the variable in the TLS of each thread. +Recompile and run the code a few more times. +You should see that in the end, var is 0.

  2. Print the address and value of var in each thread. +See that they differ.

  3. Modify the value of var in the main() function before calling pthread_create(). +Notice that the value doesn't propagate to the other threads. +This is because, upon creating a new thread, its TLS is initialised.

Quiz 1

Quiz 2

Task: Atomic Assembly​

No, this section is not about nukes, sadly :(. +Instead, we aim to get accustomed to the way in which the x86 ISA provides atomic instructions.

This mechanism looks very simple. +It is but one instruction prefix: lock. +It is not an instruction with its own separate opcode, but a prefix that slightly modifies the opcode of the following instructions to make the CPU execute it atomically (i.e. with exclusive access to the data bus).

lock must only be place before an instruction that executes a read-modify-write action. +For example, we cannot place it before a mov instruction, as the action of a mov is simply read or write. +Instead, we can place it in front of an inc instruction if its operand is memory.

Go in chapters/compute/synchronization/drills/tasks/atomic-assembly/ and run:

make skels

Look at the code in chapters/compute/synchronization/drills/tasks/atomic-assembly/support/src/race_condition_lock.asm. +It's an Assembly equivalent of the code you've already seen many times so far (such as chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition.c). +The 2 assembly functions (increment_var and decrement_var) are called by race_condition_lock_checker.c

Now add the lock prefix before dec. +Go in the tests/ folder and run the checker:

cd ../tests/
./checker.sh

You should see something like this, if done correctly:

Building the project...

nasm -f elf64 -o race_condition_lock.o race_condition_lock.asm
gcc -no-pie -o race_condition_lock_checker race_condition_lock_checker.c race_condition_lock.o -lpthread
Checking if var == 0...

Test passed

And now we have synchronized the two threads by leveraging CPU support.

Guide: apache2 Simulator - Semaphore​

Semaphores​

Up to now, we've learned how to create critical sections that can be accessed by only one thread at a time. These critical sections revolved around data. Whenever we define a critical section, there is some specific data to which we cannot allow parallel access. The reason why we can't allow it is, in general, data integrity, as we've seen in our examples in chapters/compute/synchronization/drills/tasks/race-condition/support/

But what if threads need to count? @@ -111,13 +78,56 @@ For simplicity's sake, you can view a semaphore as simply a mutex whose internal variable can take any value and acts like a counter. When a thread attempts to acquire() a semaphore, it will wait if this counter is less than or equal to 0. Otherwise, the thread decrements the internal counter and the function returns. -The opposite of acquire() is release(), which increases the internal counter by a given value (by default 1).

Conditions​

Another way we can implement our apache2 simulator is to use a condition variable. +The opposite of acquire() is release(), which increases the internal counter by a given value (by default 1).

Go to chapters/compute/synchronization/guides/apache2-simulator-semaphore/support/apache2_simulator_semaphore.py. +In the main() function we create a semaphore which we increment (release()) upon every new message. +Each thread decrements (acquire()) this semaphore to signal that it wants to retrieve a message from the list. +The retrieval means modifying a data structure, which is a critical section, so we use a separate mutex for this. +Otherwise, multiple threads could acquire the semaphore at the same time and try to modify the list at the same time. +Not good.

Locking this mutex (which in Python is called Lock) is done with the following statement: with msg_mutex: +This is a syntactic equivalent to:

event.acquire()
messages.append(msg)
event.release()

Since the length of the messages list is simply len(messages), it may seem a bit redundant to use a semaphore to store essentially the same value. +In the next section, we'll look at a more refined mechanism for our use case: condition variables.

Quiz

Task: apache2 Simulator - Condition​

Conditions​

Another way we can implement our apache2 simulator is to use a condition variable. This one is probably the most intuitive synchronization primitive. It's a means by which a thread can tell another one: "Hey, wake up, this happened!". So it's a way for threads to notify each other. For this reason, the main methods associated with conditions are notify() and wait(). As you might expect, they are complementary:

  • wait() puts the thread in the WAITING state until it's woken up by another one
  • notify() wakes up one or more wait()-ing threads. -If notify() is called before any thread has called wait(), the first thread that calls it will continue its execution unhindered.

Thread-Local Storage (TLS)​

First things first: what if we don't want data to be shared between threads? +If notify() is called before any thread has called wait(), the first thread that calls it will continue its execution unhindered.

Let's talk about a classic problem in synchronization and parallel computing: The producer-consumer problem +The problem states that one or more producers generate data and places it in a shared buffer (a queue for example), while one or more consumers take data from the buffer to further process it. +More about it in this video. +There are a few rules though, such as:

  • The producer must not insert data when the buffer is full.
  • The consumer must not retrieve data if the buffer is empty.
  • The producer and the consumer can't access the shared buffer at the same time.

Now enter chapters/compute/synchronization/drills/tasks/apache2-simulator-condition/ and run make skels. +Look at the code in chapters/compute/synchronization/drills/tasks/apache2-simulator/support/src/producer_consumer.py. +We have one producer and one consumer for simplicity. +Observe that the producer calls notify() once there is data available, and the consumer calls notify(), when data is read. +Notice that this call is preceded by an acquire() call, and succeeded by a release() call.

acquire() and release() are commonly associated with mutexes or semaphores. +What do they have to do with condition variables?

Well, a lock Condition variable also stores an inner lock (mutex). +It is this lock that we acquire() and release(). +In fact, the documentation states we should only call Condition methods with its inner lock taken.

Why is this necessary? +We don't want both the consumer and the producer to modify a buffer at the same time, this could lead to a race condition, especially if we have more producers and more consumers.

So now we know we cannot only use a mutex. +The mutex is used to access and modify the queue atomically. +Now, you might be thinking that this code causes a deadlock:

condition.acquire()
    with condition:
    ...
    condition.wait()

The thread gets the lock and then, if there is no data, it switches its state to WAITING. +A classic deadlock, right? +No. +wait() also releases the inner lock of the Condition and being woken up reacquires it. +Neat!

So now we have both synchronization and signalling. +This is what conditions are for, ultimately.

Open chapters/compute/synchronization/drills/tasks/apache2-simulator/support/src/apache2_simulator_condition.py and follow the TODOs. +The code is similar to apache2_simulator_semaphore.py, but this time we use condition variables as shown in producer_consumer.py.

Quiz

Synchronization​

So far, we've used threads and processes without wondering how to "tell" them how to access shared data. +Moreover, in order to make threads wait for each other, we simply had the main thread wait for the others to finish all their work. +But what if we want one thread to wait until another one simply performs some specific action, after which it resumes its execution? +For this, we need to use some more complex synchronization mechanisms.

Race Conditions​

For example, what if one thread wants to increase a global variable while another one wants to decrease it? +Let's say the assembly code for increasing and decreasing the variable looks like the one in the snippet below.

increase:
    mov eax, [var]
    inc eax
    mov [var], eax

decrease:
    mov eax, [var]
    dec eax
    mov [var], eax

Imagine both threads executed mov eax, [var] at the same time. +Then each would independently increase its (non-shared) eax register. +In the end, the final value of var depends on which thread executes mov [var], eax last. +So it's kind of a reversed race. +The thread that runs the slowest "wins" this race and writes the final value of var. +But this is up to the scheduler and is non-deterministic. +Such undefined behaviours can cripple the execution of a program if var is some critical variable.

Let's see this bug in action. +Go to chapters/compute/synchronization/drills/tasks/race-condition/support/c/race_condition.c, compile and run the code a few times. +It spawns to threads that do exactly what we've talked about so far: one thread increments var 10 million times, while the other decrements it 10 million times.

As you can see from running the program, the differences between subsequent runs can be substantial. +To fix this, we must ensure that only one thread can execute either var++ or var-- at any time. +We call these code sections critical sections. +A critical section is a piece of code that can only be executed by one thread at a time. +So we need some sort of mutual exclusion mechanism so that when one thread runs the critical section, the other has to wait before entering it. +This mechanism is called a mutex, whose name comes from "mutual exclusion".

Thread-Local Storage (TLS)​

First things first: what if we don't want data to be shared between threads? Are we condemned to have to worry about race conditions? Well, no.

To protect data from race conditions "by design", we can place in what's called Thread-Local Storage (TLS). As its name implies, this is a type of storage that is "owned" by individual threads, as opposed to being shared among all threads. @@ -205,8 +215,8 @@ It is only made up of 3 functions:

  • threads_create() creates a new ULT
  • threads_exit() moves the current ULT to the COMPLETED state
  • threads_join() waits for a given thread to end and saves its return value in the result argument

Look inside chapters/compute/scheduling/guides/libult/support/threads.c. Here you will find the 3 functions mentioned above.

The scheduler only uses 3 states: RUNNING, READY, COMPLETED.

Quiz

The threads in the READY and COMPLETED states are kept in 2 separate queues. When the scheduler wants to run a new thread, it retrieves it from the READY queue. -When a thread ends its execution, it is added to the COMPLETED queue, together with its return value.

Quiz

- - +When a thread ends its execution, it is added to the COMPLETED queue, together with its return value.

Quiz

+ + \ No newline at end of file diff --git a/75/Data/Questions/bypass-canary/index.html b/75/Data/Questions/bypass-canary/index.html index 6b526c265b..2cf6f3d8c6 100644 --- a/75/Data/Questions/bypass-canary/index.html +++ b/75/Data/Questions/bypass-canary/index.html @@ -4,13 +4,13 @@ Bypass Canary | Operating Systems - - + +
Skip to main content

Bypass Canary

Question​

If we enable ASLR, can we still exploit the stack_protector program?

  • no, because the address of pawned is going to be different for every run.

  • yes, because ASLR cannot work well when the canary is activated.

  • yes, because ASLR randomizes the start address of a section, but the offsets remain the same.
  • no, because the address to which addr points to is going to be random
- - + + \ No newline at end of file diff --git a/75/Data/Questions/half-page/index.html b/75/Data/Questions/half-page/index.html index 735c840a30..0bef721cac 100644 --- a/75/Data/Questions/half-page/index.html +++ b/75/Data/Questions/half-page/index.html @@ -4,15 +4,15 @@ Half Page | Operating Systems - - + +
Skip to main content

Half Page

Question Text​

char *p = malloc(2 * 1024);

What is a potential problem when allocating half a page?

Question Answers​

  • It will fragment the virtual memory because users should always request at least one page

  • Asking for less than a page might place the memory on two different pages

  • Writing to the other half may be allowed
  • Allocations smaller than one page should use the stack

Feedback​

The OS allocates memory in chunks of 4 KB (the page size). If the memory we allocate happens to be placed at the beginning of a page, we have permission to update the second half of this page despite not requesting it. This might be a problem because buffer overflows can pass unnoticed.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/index.html b/75/Data/Questions/index.html index b0d22beba3..fe24ce0ce3 100644 --- a/75/Data/Questions/index.html +++ b/75/Data/Questions/index.html @@ -4,13 +4,13 @@ Questions | Operating Systems - - + +
Skip to main content

Questions

- - + + \ No newline at end of file diff --git a/75/Data/Questions/malloc-brk/index.html b/75/Data/Questions/malloc-brk/index.html index 6710d51418..fbdb25c26d 100644 --- a/75/Data/Questions/malloc-brk/index.html +++ b/75/Data/Questions/malloc-brk/index.html @@ -4,14 +4,14 @@ malloc()` and `brk() | Operating Systems - - + +
Skip to main content

malloc() and brk()

Question Text​

When does malloc() use brk()?

Question Answers​

  • brk() is outdated, malloc() always uses mmap()
  • When it allocates a small chunk of memory

  • When it allocates an array

  • When it's working with dynamic libraries

Feedback​

malloc() uses both brk() and mmap(), but prefers brk() for small chunks of memory to keep granular allocations in a contiguous area. This way, free() does not necessarily return the memory to the OS as it might only mark the zone as "free" within libc's allocator and reuse it for later allocations.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/malloc-mmap/index.html b/75/Data/Questions/malloc-mmap/index.html index 72a9423e17..ef27c45dee 100644 --- a/75/Data/Questions/malloc-mmap/index.html +++ b/75/Data/Questions/malloc-mmap/index.html @@ -4,15 +4,15 @@ malloc()` and `mmap() | Operating Systems - - + +
Skip to main content

malloc() and mmap()

Question Text​

When does malloc() use mmap()?

Question Answers​

  • When it allocates read-only memory

  • When it allocates zeroed memory

  • When it allocates chunks of memory bigger than an internal threshold
  • When the heap is full

Feedback​

malloc uses both brk() and mmap(), but prefers mmap() for big chunks of memory (by default larger than 128 KB). This value can be altered using mallopt() with the param argument set to M_MMAP_THRESHOLD. These memory blocks are unlikely to be reused so they are not placed on heap to avoid memory fragmentation.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/memory-access/index.html b/75/Data/Questions/memory-access/index.html index 56b3d0368b..1bfd860348 100644 --- a/75/Data/Questions/memory-access/index.html +++ b/75/Data/Questions/memory-access/index.html @@ -4,8 +4,8 @@ Modify String | Operating Systems - - + +
@@ -13,7 +13,7 @@ One stores a constant pointer, cp, whereas the other stores the actual string. The compiler thinks that cp is able to modify the memory location that it points therefore it passes compilation. But at runtime a segmentation fault is issued because we are accessing data that is stored in read-only memory.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/memory-aslr/index.html b/75/Data/Questions/memory-aslr/index.html index d656f3fc67..61259e4a02 100644 --- a/75/Data/Questions/memory-aslr/index.html +++ b/75/Data/Questions/memory-aslr/index.html @@ -4,13 +4,13 @@ ASLR | Operating Systems - - + +
Skip to main content

ASLR

Question Text​

If we enable ASLR and run the bo_write_practice executable with the previously payload what vulnerabilities will we be able to still exploit?

Question Answers​

  • we can still jump to the secret_func

  • we can both still jump to the secret_func and overwrite the local variable

  • we can jump to a library function

  • we can still overwrite the local variable
- - + + \ No newline at end of file diff --git a/75/Data/Questions/memory-granularity/index.html b/75/Data/Questions/memory-granularity/index.html index 2e2b8ab746..aa668083f3 100644 --- a/75/Data/Questions/memory-granularity/index.html +++ b/75/Data/Questions/memory-granularity/index.html @@ -4,14 +4,14 @@ Memory Granularity | Operating Systems - - + +
Skip to main content

Memory Granularity

Question Text​

What is the granularity of the size of memory sections?

Question Answers​

  • 4 bytes

  • 4 MB

  • 4 KB
  • 1 KB

Feedback​

All sizes of memory areas are multiple of 4 KB (the page size). Also, all addresses start at multiple of 4 KB.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/memory-leaks/index.html b/75/Data/Questions/memory-leaks/index.html index 17a3cc7d29..35af8f8cb9 100644 --- a/75/Data/Questions/memory-leaks/index.html +++ b/75/Data/Questions/memory-leaks/index.html @@ -4,14 +4,14 @@ Memory Leaks | Operating Systems - - + +
Skip to main content

Memory Leaks

Question Text​

What happens to the leaked memory when the process finishes execution?

Question Answers​

  • It will be freed by the OS garbage collector

  • It remains unusable until the first restart

  • It is freed alongside all memory used by process
  • It will remain reachable until another process explicitly frees it

Feedback​

When a process ends, all its memory zones are freed by the OS to be reused. Leaking memory becomes a major problem in case of programs that run indefinitely, because the leaked memory will stack, causing a memory outage.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/memory-regions-vars/index.html b/75/Data/Questions/memory-regions-vars/index.html index 698eac8ab0..3125c9926b 100644 --- a/75/Data/Questions/memory-regions-vars/index.html +++ b/75/Data/Questions/memory-regions-vars/index.html @@ -4,8 +4,8 @@ Variables in memory regions | Operating Systems - - + +
@@ -14,7 +14,7 @@ Non-static local variables go on the.stack (c, k). malloc()'ed memory goes on the.heap. For k, the pointer is stored on the.stack, but the allocated memory, to which k points is stored on the.heap.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/memory-stack-protector/index.html b/75/Data/Questions/memory-stack-protector/index.html index 5b1d21fa3c..5bed40ad1c 100644 --- a/75/Data/Questions/memory-stack-protector/index.html +++ b/75/Data/Questions/memory-stack-protector/index.html @@ -4,8 +4,8 @@ Stack Protector | Operating Systems - - + +
@@ -13,7 +13,7 @@ Why is that?

Question Answers​

Feedback​

When using the canary, to minimize the damage a buffer overflow could cause, the buffers are always placed right below the canary. By doing so, a buffer overflow will not overwrite anything. However, it is still possible to overwrite other local buffers, provided that a function declares more than 1 array or if we use the pointer directly.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/mmap-file/index.html b/75/Data/Questions/mmap-file/index.html index 9ec14e6c2c..b0ada95f88 100644 --- a/75/Data/Questions/mmap-file/index.html +++ b/75/Data/Questions/mmap-file/index.html @@ -4,14 +4,14 @@ `mmap()` file | Operating Systems - - + +
Skip to main content

mmap() file

Question Text​

What is one advantage of mapping a file in memory?

Question Answers​

  • It reduces interaction with the disk

  • Consumes less memory

  • It is faster because it does not uses the file API

  • Allows all threads to use the same memory area

Feedback​

After mapping a file in memory, all changes will be visible only in memory. When removing the mapping or explicitly calling msync() the information from memory will be visible on disk.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/operators/index.html b/75/Data/Questions/operators/index.html index 13bb00af09..10d067881f 100644 --- a/75/Data/Questions/operators/index.html +++ b/75/Data/Questions/operators/index.html @@ -4,13 +4,13 @@ Operator Overloading | Operating Systems - - + +
Skip to main content

Operator Overloading

Question Text​

How many constructor calls, copy constructor calls, assignment operator calls and destructor calls does the following program issue?

Obj quiz(Obj o1, Obj o2)
{
    o2 = o1;
    return o2;
}
void main()
{
    Obj b = quiz(o1, o2);
}

Question Answers​

  • constructor calls = 0, copy constructor calls = 3, assignment operator calls = 1, destructor calls = 3

  • constructor calls = 1, copy constructor calls = 2, assignment operator calls = 1, destructor calls = 2

  • constructor calls = 0, copy constructor calls = 2, assignment operator calls = 1, destructor calls = 2

  • constructor calls = 0, copy constructor calls = 3, assignment operator calls = 1, destructor calls = 1

Feedback​

  • There are no constructor calls because there is no object construction when using int.
  • There are 3 copy constructor calls: for passing o1, for passing o2, and for returning o2.
  • There is 1 assignment operator call for o2 = o1.
  • There are 3 destructor calls, because each constructed object needs to be destroyed.
- - + + \ No newline at end of file diff --git a/75/Data/Questions/page-allocation/index.html b/75/Data/Questions/page-allocation/index.html index 3b4e426f98..e8f313c309 100644 --- a/75/Data/Questions/page-allocation/index.html +++ b/75/Data/Questions/page-allocation/index.html @@ -4,14 +4,14 @@ Page Allocation | Operating Systems - - + +
Skip to main content

Page Allocation

Question Text​

student@os:~/.../drills/tasks/static-dynamic/support$ size hello-static
text    data     bss     dec     hex filename
893333   20996    7128  921457   e0f71 hello-static

How many bytes should we add to the .data section to make its size 28 KB, instead of 24 KB?

Question Answers​

  • 1 KB

  • 4 KB

  • 3580 bytes

  • 3581 bytes

Feedback​

The total size must be 1 byte over the 24 KB threshold to cause a new page allocation. So in order to get that past the current size of 20996, we need 3581 bytes:

student@os:~$ echo "24 * 1024 + 1 - 20996" | bc
3581
- - + + \ No newline at end of file diff --git a/75/Data/Questions/stack-layout/index.html b/75/Data/Questions/stack-layout/index.html index 4a5284238a..1de2764969 100644 --- a/75/Data/Questions/stack-layout/index.html +++ b/75/Data/Questions/stack-layout/index.html @@ -4,13 +4,13 @@ Stack layout | Operating Systems - - + +
Skip to main content

Stack layout

Question Text​

What is the stack layout for the fun function in the bo_write.c program (starting from a high address)?

Question Answers​

  • return address, old rbp, maybe some padding, variable a, b[0], b[1], b[2]
  • return address, old rbp, maybe some padding, variable a, b[2], b[1], b[0]

  • return address, maybe some padding, variable a, b[0], b[1], b[2]

  • return address, old rbp, maybe some padding, b[0], b[1], b[2], variable a

Feedback​

Look at the assembly code and notice the exact layout.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/string-buff-over/index.html b/75/Data/Questions/string-buff-over/index.html index 245246cb94..b180931a3d 100644 --- a/75/Data/Questions/string-buff-over/index.html +++ b/75/Data/Questions/string-buff-over/index.html @@ -4,14 +4,14 @@ String Buffer Overflow | Operating Systems - - + +
Skip to main content

String Buffer Overflow

Question Text​

Why does the buffer overflow occur?

Question Answers​

  • the initial string, declared in main(), does not contain a terminating null byte.

  • the buffer is not large enough to store the copied bytes.

  • memcpy() skips the copying of terminating null bytes.

  • memcpy() copies 4 bytes, whereas the size of the string, including the terminating null byte, is 5.

Feedback​

The string soso has length equal to 4, however, 5 bytes are actually used to store it, including the terminating null byte. Even though the buffer declared in fun() is not large enough to store the 5 bytes, the underlying issue is that we copying just 4 bytes, thus skipping the null byte.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/string-strcpy/index.html b/75/Data/Questions/string-strcpy/index.html index 15eea80ef2..c7317a665a 100644 --- a/75/Data/Questions/string-strcpy/index.html +++ b/75/Data/Questions/string-strcpy/index.html @@ -4,13 +4,13 @@ Strcpy Buffer Overflow | Operating Systems - - + +
Skip to main content

Strcpy Buffer Overflow

Question Text​

Does any buffer overflow occur with the latest version of the program?

Question Answers​

  • no, because strcpy() was designed to correctly handle such situations.

  • no, because the string is correctly printed, i.e. no extra characters.

  • we cannot know

  • yes, strcpy() copies the entirety of the source, including the \0; since the size of dst is 4 the null byte overwrites the least significant byte of var

Feedback​

Print the contents of variable var.

- - + + \ No newline at end of file diff --git a/75/Data/Questions/valgrind-leaks/index.html b/75/Data/Questions/valgrind-leaks/index.html index a7e7f1de84..a7960b5c2d 100644 --- a/75/Data/Questions/valgrind-leaks/index.html +++ b/75/Data/Questions/valgrind-leaks/index.html @@ -4,15 +4,15 @@ Valgrind Leaks | Operating Systems - - + +
Skip to main content

Valgrind Leaks

Question Text​

struct student {
    char *name;
    int age;
}

struct student *s = malloc(sizeof(*s));
s->name = strdup("Reginald");
// ...
free(s);

What are the leaks in the above c program?

Question Answers​

  • There are no leaks

  • s->name is definitely lost

  • s->name is indirectly lost
  • s->name is still reachable

Feedback​

strdup() allocates memory for a string so the returned pointer must be freed. Freeing s will leave us unable to free s->name, so s->name is indirectly lost. Find more about valgrind leak categories here.

- - + + \ No newline at end of file diff --git a/75/Data/data-overview/index.html b/75/Data/data-overview/index.html index a9c706e87d..7651dac8e3 100644 --- a/75/Data/data-overview/index.html +++ b/75/Data/data-overview/index.html @@ -4,8 +4,8 @@ Data | Operating Systems - - + +
@@ -16,7 +16,7 @@ The programming language analyzes the use of these variables and outputs code that uses an interface provided by the operating system. This interface offers the possibility to allocate/deallocate different variables in certain memory regions. Next, the operating system manages the execution of the program and provides the actual physical addresses that are used to interact with the data.

Moreover, the operating system governs the competing access of multiple programs to memory, ensuring that a program does not have access to a different program's memory.

Contents​

  1. Working with Memory
  2. Process Memory
  3. Investigate Memory
  4. Memory Security
- - + + \ No newline at end of file diff --git a/75/Data/index.html b/75/Data/index.html index 143a06aaec..f25b7591a8 100644 --- a/75/Data/index.html +++ b/75/Data/index.html @@ -4,13 +4,13 @@ Data | Operating Systems - - + +
Skip to main content

Data


tip

Focus the slides and press F for fullscreen viewing.

- - + + \ No newline at end of file diff --git a/75/Data/lab3/index.html b/75/Data/lab3/index.html index e21f67f849..4454d64b13 100644 --- a/75/Data/lab3/index.html +++ b/75/Data/lab3/index.html @@ -4,8 +4,8 @@ Lab 3 - Memory | Operating Systems - - + +
@@ -146,7 +146,7 @@ Taking the address of a local, doing pointer arithmetic, reinterpret casts, calling non-@safe functions etc. are not allowed in @safe code. If any of these unsafe features are manually proven to be safe, the @trusted keyword may be used to disable the checks but still consider the code @safe. This is to allow writing system code, which by its nature is unsafe.

- - + + \ No newline at end of file diff --git a/75/Data/lab4/index.html b/75/Data/lab4/index.html index 45bbb95dbd..db0494bd1c 100644 --- a/75/Data/lab4/index.html +++ b/75/Data/lab4/index.html @@ -4,8 +4,8 @@ Lab 4 - Investigate Memory | Operating Systems - - + +
@@ -88,7 +88,7 @@ Note that the file path used for LD_PRELOAD may need to be updated, depending on your distribution:

student@os:~/.../memory-leak/support/src$ LD_PRELOAD=/lib/x86_64-linux-gnu/libc_malloc_debug.so.0 MALLOC_TRACE=mem.trace ./memory_leak_malloc
Linus Torvalds is 22 years old and likes Linux
Steve Jobs is 23 years old and likes macOS

Subsequently, we use the mtrace tool to show information about the leaked data:

student@os:~/.../memory-leak/support/src$ mtrace ./memory_leak_malloc mem.trace

Memory not freed:
-----------------
           Address     Size     Caller
0x000056506d8be6a0     0x94  at 0x56506c3777ec

The size (0x94) is the same value shown by Valgrind (148).

mtrace provides an outcome similar to Valgrind. Valgrind is however more powerful: it works on different types of memory (not only those allocated with malloc()) and it doesn't require access to the source code (and the compiler phase).

Practice​

  1. Print the size of the Student class and the struct student structure to see if it equates to the leak shown by Valgrind.

  2. Solve the memory leaks in both programs. Run the checker (./checker.sh in the memory-leak/support/tests/ folder) to check your results.

    Sample run:

student@so:~/.../support/tests/$ ./checker.sh
make: Entering directory '/home/student/operating-systems/chapters/data/investigate-memory/guides/memory-leak/support/src'
g++    -c -o memory_leak.o memory_leak.cpp
cc   memory_leak.o  -lstdc++ -o memory_leak
cc    -c -o memory_leak_malloc.o memory_leak_malloc.c
cc   memory_leak_malloc.o  -lstdc++ -o memory_leak_malloc
make: Leaving directory '/home/student/operating-systems/chapters/data/investigate-memory/guides/memory-leak/support/src'
-------------------------------------------------
Checking memory leaks for C executable: ../src/memory_leak
C executable is leak-free!

Points for ../src/memory_leak: 50/50
-------------------------------------------------
Checking memory leaks for C++ executable: ../src/memory_leak_malloc
C++ executable is leak-free!

Points for ../src/memory_leak_malloc: 50/50
-------------------------------------------------
Total Points: 100/100
- - + + \ No newline at end of file diff --git a/75/Data/lab5/index.html b/75/Data/lab5/index.html index bac3c716bc..cd226a48cb 100644 --- a/75/Data/lab5/index.html +++ b/75/Data/lab5/index.html @@ -4,8 +4,8 @@ Lab 5 - Memory Security | Operating Systems - - + +
@@ -66,7 +66,7 @@ Analyze the code, then compile it and run it.

  1. Try to find an input that alters the control flow of the program so that "Comm-link online" is printed. You are not allowed to modify the source file.

  2. Try to find an input that alters the control flow of the program so that "Channel open." is printed. You are not allowed to modify the source file.

Note: Addresses are 8 bytes long on 64 bit machines.

  1. Can you think of a different input that results in printing "Comm-link online"?
- - + + \ No newline at end of file diff --git a/75/Exams/2024 Summer/Syscall Tracing/syscall-tracing/index.html b/75/Exams/2024 Summer/Syscall Tracing/syscall-tracing/index.html index 6d0c88e88c..99afe8c750 100644 --- a/75/Exams/2024 Summer/Syscall Tracing/syscall-tracing/index.html +++ b/75/Exams/2024 Summer/Syscall Tracing/syscall-tracing/index.html @@ -4,8 +4,8 @@ Syscall Tracing | Operating Systems - - + +
@@ -16,7 +16,7 @@ If not directly, then through the interpreter.

  • Two consecutive runs of the application will generate the same sequence of system calls, possibly with some timing differences.

  • Generally, the application will consume more CPU time than the tracing tool.

  • Multiple instances of the tool process can run simultaneously to capture information from different applications.

  • The results displayed by the tool will differ significantly when the application runs on a bare-metal operating system compared to a virtual machine operating system.

  • The tool will capture page faults.

  • The tool cannot be used when the captured application communicates with another application.

  • The analyzed application can be statically or dynamically linked.

    • - - + + \ No newline at end of file diff --git a/75/Exams/2024 Summer/Sysinfo Library/sysinfo-library/index.html b/75/Exams/2024 Summer/Sysinfo Library/sysinfo-library/index.html index a47d774a48..fb631f8b28 100644 --- a/75/Exams/2024 Summer/Sysinfo Library/sysinfo-library/index.html +++ b/75/Exams/2024 Summer/Sysinfo Library/sysinfo-library/index.html @@ -4,8 +4,8 @@ Sysinfo Library | Operating Systems - - + +
    @@ -27,7 +27,7 @@ Here it can be argued that it might investigate the number of processes or occupied disk space. Points are also awarded if argued this way and the answer is "False".

  • The library can be provided as both a dynamically linked library and a statically linked library.

    • - - + + \ No newline at end of file diff --git a/75/Exams/2024 Summer/index.html b/75/Exams/2024 Summer/index.html index b0fd36a02c..240510f025 100644 --- a/75/Exams/2024 Summer/index.html +++ b/75/Exams/2024 Summer/index.html @@ -4,13 +4,13 @@ 2024 Summer | Operating Systems - - + +
    Skip to main content

    2024 Summer

    - - + + \ No newline at end of file diff --git a/75/Exams/Aggregator Application/aggregator-application/index.html b/75/Exams/Aggregator Application/aggregator-application/index.html index ae373f2c1c..cba3008c6a 100644 --- a/75/Exams/Aggregator Application/aggregator-application/index.html +++ b/75/Exams/Aggregator Application/aggregator-application/index.html @@ -4,8 +4,8 @@ Aggregator Application | Operating Systems - - + +
    @@ -25,7 +25,7 @@ Answer: True

  • The aggregator application's page table DOES NOT contain pages without write permission. Answer: False

  • The system can only function if the communication between the aggregator application and LLM sites is encrypted. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Application Investigator/application-investigator/index.html b/75/Exams/Application Investigator/application-investigator/index.html index 4f01e594d7..c4a4582317 100644 --- a/75/Exams/Application Investigator/application-investigator/index.html +++ b/75/Exams/Application Investigator/application-investigator/index.html @@ -4,8 +4,8 @@ Application Investigator | Operating Systems - - + +
    @@ -23,7 +23,7 @@ Answer: False

  • The resulting file from the memory dump is a binary file, not text. Answer: True

  • In principle, with certain adjustments, the application can work on both 32-bit and 64-bit systems. Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/Backup System/backup-system/index.html b/75/Exams/Backup System/backup-system/index.html index 24dda0bada..7c16d8e601 100644 --- a/75/Exams/Backup System/backup-system/index.html +++ b/75/Exams/Backup System/backup-system/index.html @@ -4,8 +4,8 @@ Backup System | Operating Systems - - + +
    @@ -14,7 +14,7 @@ We want to add a backup functionality to the program so that data from memory is periodically saved to disk. For data recovery, analysis data for the last N intervals is kept (we have N files of 4GB each, each file retains analysis data from a backup).

    All answers should be justified.

    Questions​

    1. How would you implement such a backup system?

    2. How do you determine the opportune moment for performing the backup?

    3. Do you use a separate process or thread for performing the backup? Justify your answer.

    4. How do you manage the files saved to disk?

    5. How do you analyze the overhead brought by the backup functionality?

    - - + + \ No newline at end of file diff --git a/75/Exams/Benchmarking Application/benchmarking-application/index.html b/75/Exams/Benchmarking Application/benchmarking-application/index.html index 84789629a7..ac5d8abaf5 100644 --- a/75/Exams/Benchmarking Application/benchmarking-application/index.html +++ b/75/Exams/Benchmarking Application/benchmarking-application/index.html @@ -4,8 +4,8 @@ Benchmarking Application | Operating Systems - - + +
    @@ -23,7 +23,7 @@ Answer: True

  • With the necessary modifications, the application can handle both HTTP and HTTPS connections. Answer: True

  • The benchmarking application can only measure open-source web applications + open-source web servers (with available sources). Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Blockchain System/blockchain-system/index.html b/75/Exams/Blockchain System/blockchain-system/index.html index b8af62973b..951c81de72 100644 --- a/75/Exams/Blockchain System/blockchain-system/index.html +++ b/75/Exams/Blockchain System/blockchain-system/index.html @@ -4,8 +4,8 @@ Blockchain System | Operating Systems - - + +
    @@ -29,7 +29,7 @@ Answer: False

  • An SSD (fast storage) is more useful for the frontend service than for the backend. Answer: False

  • The backend service uses a lot of RAM (almost the entire system memory). Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/Cloud System/cloud-system/index.html b/75/Exams/Cloud System/cloud-system/index.html index 25c8ffa4c3..8c3f8a33f3 100644 --- a/75/Exams/Cloud System/cloud-system/index.html +++ b/75/Exams/Cloud System/cloud-system/index.html @@ -4,8 +4,8 @@ Cloud System | Operating Systems - - + +
    @@ -14,7 +14,7 @@ Suspension can occur either in RAM or on disk.

    All answers should be justified.

    Questions​

    1. Describe the criteria based on which the dispatcher will send requests to the executors and how it obtains/measures the necessary metrics.

    2. What usage scenarios are suitable for such a cloud system? In these scenarios, when does the starting/resuming and stopping/suspending of a virtual machine occur?

    3. How does an executor node decide which method to use: stopping, suspending in RAM, or suspending on disk? What criteria does it use, and how does it use them in the decision-making process?

    4. How will you calculate the billing for the system's customers?

    - - + + \ No newline at end of file diff --git a/75/Exams/Database Application/database-application/index.html b/75/Exams/Database Application/database-application/index.html index 5436392bc8..589ab46336 100644 --- a/75/Exams/Database Application/database-application/index.html +++ b/75/Exams/Database Application/database-application/index.html @@ -4,8 +4,8 @@ Database Application | Operating Systems - - + +
    @@ -24,7 +24,7 @@ Answer: True

  • Memory within the container cannot be visible from outside the container, even by a privileged account. Answer: False

  • The application exclusively spends time in user mode, not in kernel mode. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Digital Forensics/digital-forensics/index.html b/75/Exams/Digital Forensics/digital-forensics/index.html index 27061e93f2..c3d02a484f 100644 --- a/75/Exams/Digital Forensics/digital-forensics/index.html +++ b/75/Exams/Digital Forensics/digital-forensics/index.html @@ -4,8 +4,8 @@ Digital Forensics | Operating Systems - - + +
    @@ -25,7 +25,7 @@ Answer: True

  • The application cannot realistically be used on a 64-bit system, only on a 32-bit system. Answer: False

  • As long as the operating system provides support, the application can work on both x86 and ARM processor architectures. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Distributed System/distributed-system/index.html b/75/Exams/Distributed System/distributed-system/index.html index e30bd9c6cb..40e41492fd 100644 --- a/75/Exams/Distributed System/distributed-system/index.html +++ b/75/Exams/Distributed System/distributed-system/index.html @@ -4,8 +4,8 @@ Distributed System | Operating Systems - - + +
    @@ -23,7 +23,7 @@ Answer: False

  • SSD storage is required for the frontend service. Answer: False

  • The backend service uses a lot of memory - almost the entire system memory. Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/Extending an App Manager/extending-an-app-manager/index.html b/75/Exams/Extending an App Manager/extending-an-app-manager/index.html index 826a705e26..d178a4eb12 100644 --- a/75/Exams/Extending an App Manager/extending-an-app-manager/index.html +++ b/75/Exams/Extending an App Manager/extending-an-app-manager/index.html @@ -4,8 +4,8 @@ Extending an App Manager | Operating Systems - - + +
    @@ -13,7 +13,7 @@ We want to add functionality to the application manager through which we can transfer a process managed by it from one system to another system managed by another application manager while the process is running. For simplicity, we consider that the process does not use files from the disk.

    All answers should be justified.

    Questions​

    1. How do you establish the connection and transfer between managers? Describe a minimal communication protocol.

    2. What process data needs to be transferred so that the process can continue running (from where it left off) on the new system?

    3. How do you minimize the time the process is stopped?

    - - + + \ No newline at end of file diff --git a/75/Exams/FaaS Application/faas-application/index.html b/75/Exams/FaaS Application/faas-application/index.html index fcc2b98b4b..49a6586ccc 100644 --- a/75/Exams/FaaS Application/faas-application/index.html +++ b/75/Exams/FaaS Application/faas-application/index.html @@ -4,8 +4,8 @@ FaaS Application | Operating Systems - - + +
    @@ -26,7 +26,7 @@ Answer: True

  • During the execution of the service, NO hardware interrupts are generated. Answer: False

  • The service can only operate on a Linux operating system. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/File Changes Notifier/file-changes-notifier/index.html b/75/Exams/File Changes Notifier/file-changes-notifier/index.html index 14f6d10539..26eda7ac8e 100644 --- a/75/Exams/File Changes Notifier/file-changes-notifier/index.html +++ b/75/Exams/File Changes Notifier/file-changes-notifier/index.html @@ -4,15 +4,15 @@ File Changes Notifier | Operating Systems - - + +
    Skip to main content

    File Changes Notifier

    Scenario​

    You are the administrator of a system and have multiple logging files from different applications at your disposal. You want to be notified every time new information appears in any of these files, and at the same time, you want to be able to add new monitoring targets throughout the runtime of this monitor.

    All answers should be justified.

    Question​

    Can you suggest a simple method or pseudocode to solve this dilemma? (hint: file operations).

    - - + + \ No newline at end of file diff --git a/75/Exams/Header Analysis Application/header-analysis-application/index.html b/75/Exams/Header Analysis Application/header-analysis-application/index.html index fd5d1b7d09..c4f9682b0b 100644 --- a/75/Exams/Header Analysis Application/header-analysis-application/index.html +++ b/75/Exams/Header Analysis Application/header-analysis-application/index.html @@ -4,14 +4,14 @@ Header Analysis Application | Operating Systems - - + +
    Skip to main content

    Header Analysis Application

    Scenario​

    We aim to implement a network packet header analysis application to detect suspicious behaviors from a network security standpoint. We assume the following steps are already implemented sequentially:

    • Packet headers are received through a socket.
    • Packet analysis is performed using functions already implemented in a shared library.
    • Data about suspicious packets (according to the analysis) are written to an application log file.

    All answers should be justified.

    Questions​

    1. What specific technologies and concepts do you use to optimize the application (multithreading, multiprocessing, I/O multiplexing, asynchronous operations)?

    2. How do you ensure scalability/performance of the application in case the number of packets in the network is very high (network flooding)?

    3. How do you ensure uniform data writing to the application analysis log file?

    - - + + \ No newline at end of file diff --git a/75/Exams/Industrial System/industrial-system/index.html b/75/Exams/Industrial System/industrial-system/index.html index 7432291b93..5543e65c43 100644 --- a/75/Exams/Industrial System/industrial-system/index.html +++ b/75/Exams/Industrial System/industrial-system/index.html @@ -4,8 +4,8 @@ Industrial System | Operating Systems - - + +
    @@ -26,7 +26,7 @@ Answer: True

  • Context switching is more costly for applications in wasmer. Answer: False

  • Applications in wasmer do not use the libc file access API but still benefit from the file cache system of the operating system. Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/Intrusion Detection System/intrusion-detection-system/index.html b/75/Exams/Intrusion Detection System/intrusion-detection-system/index.html index b37c37c5aa..9f5345d5bb 100644 --- a/75/Exams/Intrusion Detection System/intrusion-detection-system/index.html +++ b/75/Exams/Intrusion Detection System/intrusion-detection-system/index.html @@ -4,14 +4,14 @@ Intrusion Detection System | Operating Systems - - + +
    Skip to main content

    Intrusion Detection System

    Scenario​

    We want to build an intrusion detection system for applications serving different servers without having access to their source code. Assume that we have access to a knowledge base (in any form you prefer) describing the normal behavior of the application.

    All answers should be justified.

    Questions​

    1. What profiling and debugging utilities (perf, strace, valgrind, etc.) can help us verify whether, as a result of interaction between the server and a specific client, the client was able to exploit various vulnerabilities such as buffer overflow or code injection?

    2. Could you propose a simplistic algorithm that can determine the type of client behavior (normal or malicious) based on information gathered from the utility of your choice?

    - - + + \ No newline at end of file diff --git a/75/Exams/Library Warmer/library-warmer/index.html b/75/Exams/Library Warmer/library-warmer/index.html index a8c70af56b..997c0c6955 100644 --- a/75/Exams/Library Warmer/library-warmer/index.html +++ b/75/Exams/Library Warmer/library-warmer/index.html @@ -4,8 +4,8 @@ Library Warmer | Operating Systems - - + +
    @@ -16,7 +16,7 @@ What is the execution flow of the application?

  • What functionalities does the operating system need to expose in order to efficiently and effectively implement the heating application?

  • What configuration interface will the application expose to the user/system administrator? What will they be able to configure?

  • How will you evaluate the use of the heating application? What will you measure to demonstrate an improvement (or not) compared to the case where you do not use the application?

    • - - + + \ No newline at end of file diff --git a/75/Exams/Memory Deduplication/memory-deduplication/index.html b/75/Exams/Memory Deduplication/memory-deduplication/index.html index 27f5a64c74..da9fe3a9b1 100644 --- a/75/Exams/Memory Deduplication/memory-deduplication/index.html +++ b/75/Exams/Memory Deduplication/memory-deduplication/index.html @@ -4,8 +4,8 @@ Memory Deduplication | Operating Systems - - + +
    @@ -17,7 +17,7 @@ When and how will memory deduplication be performed with the support of the operating system?

  • How do we evaluate the functionality of memory deduplication? What metrics do we track? What scenarios will we use to monitor these metrics?

    • - - + + \ No newline at end of file diff --git a/75/Exams/Network Configurations Manager/network-configurations-manager/index.html b/75/Exams/Network Configurations Manager/network-configurations-manager/index.html index 0f9a5e7451..45a6ad21ef 100644 --- a/75/Exams/Network Configurations Manager/network-configurations-manager/index.html +++ b/75/Exams/Network Configurations Manager/network-configurations-manager/index.html @@ -4,8 +4,8 @@ Network Configurations Manager | Operating Systems - - + +
    @@ -24,7 +24,7 @@ Answer: True

  • In the application's address space, there are memory areas without write permissions. Answer: True

  • The application process will consistently have the highest priority. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Network Performance Utility/network-performance-utility/index.html b/75/Exams/Network Performance Utility/network-performance-utility/index.html index cca229691a..de7944841b 100644 --- a/75/Exams/Network Performance Utility/network-performance-utility/index.html +++ b/75/Exams/Network Performance Utility/network-performance-utility/index.html @@ -4,8 +4,8 @@ Network Performance Utility | Operating Systems - - + +
    @@ -24,7 +24,7 @@ Answer: True

  • The process started by the utility will use shared memory mechanisms. Answer: False

  • The process started by the utility is a component that requires additional security/isolation compared to other processes. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Nightly Builds System/nightly-builds-system/index.html b/75/Exams/Nightly Builds System/nightly-builds-system/index.html index 95bb5c703f..c35e714137 100644 --- a/75/Exams/Nightly Builds System/nightly-builds-system/index.html +++ b/75/Exams/Nightly Builds System/nightly-builds-system/index.html @@ -4,8 +4,8 @@ Nightly Builds System | Operating Systems - - + +
    @@ -29,7 +29,7 @@ Answer: True

  • The dispatcher component does not cause page faults. Answer: False

  • The dispatcher system can function as both a statically compiled and a dynamically compiled application. Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/Resource Monitor/resource-monitor/index.html b/75/Exams/Resource Monitor/resource-monitor/index.html index b8ca5f8b2c..0a882ba948 100644 --- a/75/Exams/Resource Monitor/resource-monitor/index.html +++ b/75/Exams/Resource Monitor/resource-monitor/index.html @@ -4,8 +4,8 @@ Resource Monitor | Operating Systems - - + +
    @@ -14,7 +14,7 @@ The master process is responsible for sending configurations to the agents and reporting resource (ab)use issues to the administrator or commanding actions for the agents to perform. The implementation of the agents should be as portable as possible to allow them to run on various platforms/operating systems.

    All answers should be justified.

    Questions​

    1. What resources/information will agents collect to report the overall system status? How will the configuration that the master process will transmit to the agents look like, schematically?

    2. Could you describe the design and implementation idea of an agent?

    3. Could you describe the design and implementation idea of the master process?

    4. Describe schematically the communication protocol between agent and master: what information is transferred, at what intervals, on what triggers, what happens when there are network issues, or when an agent or master fails?

    - - + + \ No newline at end of file diff --git a/75/Exams/Supervisor-type Service/supervisor-type-service/index.html b/75/Exams/Supervisor-type Service/supervisor-type-service/index.html index d1e324e362..2e912fd967 100644 --- a/75/Exams/Supervisor-type Service/supervisor-type-service/index.html +++ b/75/Exams/Supervisor-type Service/supervisor-type-service/index.html @@ -4,8 +4,8 @@ Supervisor-type Service | Operating Systems - - + +
    @@ -25,7 +25,7 @@ Answer: False

  • The service can configure the started processes to have standard descriptors redirected to a /dev/null-like input. Answer: True

  • The service can memory-map the configuration file. Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/System Process Monitoring Tool/system-process-monitoring-tool/index.html b/75/Exams/System Process Monitoring Tool/system-process-monitoring-tool/index.html index bdaea3b266..f38180c974 100644 --- a/75/Exams/System Process Monitoring Tool/system-process-monitoring-tool/index.html +++ b/75/Exams/System Process Monitoring Tool/system-process-monitoring-tool/index.html @@ -4,8 +4,8 @@ System Process Monitoring Tool | Operating Systems - - + +
    @@ -24,7 +24,7 @@ Answer: False

  • The application benefits from using the buffer cache. Answer: True

  • The application can be implemented without using synchronization primitives. Answer: True

    • - - + + \ No newline at end of file diff --git a/75/Exams/User-level Threading Library/user-level-threading-library/index.html b/75/Exams/User-level Threading Library/user-level-threading-library/index.html index c9c429a41c..c5f2cef635 100644 --- a/75/Exams/User-level Threading Library/user-level-threading-library/index.html +++ b/75/Exams/User-level Threading Library/user-level-threading-library/index.html @@ -4,8 +4,8 @@ User-level Threading Library | Operating Systems - - + +
    @@ -23,7 +23,7 @@ Answer: True

  • A program using the library will use only one core/processor. Answer: True

  • Programs using the library cannot use shared memory to communicate with other processes. Answer: False

    • - - + + \ No newline at end of file diff --git a/75/Exams/Web GUI 1/web-gui-1/index.html b/75/Exams/Web GUI 1/web-gui-1/index.html index a15ccc7384..25e98e13ee 100644 --- a/75/Exams/Web GUI 1/web-gui-1/index.html +++ b/75/Exams/Web GUI 1/web-gui-1/index.html @@ -4,8 +4,8 @@ Web GUI 1 | Operating Systems - - + +
    @@ -14,7 +14,7 @@ Describe the general components and their interaction. Applications should be able to access system resources.

  • We want to add the possibility of having threads in the application. Propose an architecture for this.

  • Propose a mechanism that will allow the use of async and await primitives for functions that interact with the operating system.

  • Answer one of the two options:

    • - - + + \ No newline at end of file diff --git a/75/Exams/Web GUI 2/web-gui-2/index.html b/75/Exams/Web GUI 2/web-gui-2/index.html index b556f8d217..db310becb9 100644 --- a/75/Exams/Web GUI 2/web-gui-2/index.html +++ b/75/Exams/Web GUI 2/web-gui-2/index.html @@ -4,8 +4,8 @@ Web GUI 2 | Operating Systems - - + +
    @@ -15,7 +15,7 @@ Take into account that the JavaScript engine (V8) can only execute JavaScript code on a single thread. You cannot modify the V8 engine.

  • Answer one of the following:

    • - - + + \ No newline at end of file diff --git a/75/Exams/index.html b/75/Exams/index.html index f8c657a13d..0063f335e2 100644 --- a/75/Exams/index.html +++ b/75/Exams/index.html @@ -4,13 +4,13 @@ Exams | Operating Systems - - + +
    Skip to main content

    Exams

    - - + + \ No newline at end of file diff --git a/75/Hackathons/Lambda Function Loader/index.html b/75/Hackathons/Lambda Function Loader/index.html index 47bd503934..bff6849948 100644 --- a/75/Hackathons/Lambda Function Loader/index.html +++ b/75/Hackathons/Lambda Function Loader/index.html @@ -4,8 +4,8 @@ Lambda Function Loader | Operating Systems - - + +
    @@ -42,7 +42,7 @@ Implement measures to prevent data leaks.
  • Conducting performance optimizations.
  • Establishing application logging.
  • Managing configuration files/options: Configure the server during startup based on specified options/configuration files such as maximum client count, socket type preferences, etc.
  • Real-time server monitoring and statistical extraction (e.g. client count, loaded libraries, memory usage, etc.).
  • Adapting the implementation to another programming language (the initial skeleton is in C, but the communication's nature allows implementation in any language meeting project requirements).
  • Generating a suitable response in cases where the requested function exceeds a TIMEOUT or performs actions leading to server shutdown (e.g. invalid memory access, exit/abort calls). The solution's complexity may vary based on how the client's connection termination is handled (whether an error message is conveyed) and how the cause of execution termination is identified.
    • - - + + \ No newline at end of file diff --git a/75/Hackathons/index.html b/75/Hackathons/index.html index c577d1bf03..bf3b8b5447 100644 --- a/75/Hackathons/index.html +++ b/75/Hackathons/index.html @@ -4,13 +4,13 @@ Hackathons | Operating Systems - - + +
    Skip to main content

    Hackathons

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/anonymous-pipes-limitation/index.html b/75/IO/Questions/anonymous-pipes-limitation/index.html index a811fdf5a4..6869edea8e 100644 --- a/75/IO/Questions/anonymous-pipes-limitation/index.html +++ b/75/IO/Questions/anonymous-pipes-limitation/index.html @@ -4,15 +4,15 @@ Limitation of Anonymous Pipes | Operating Systems - - + +
    Skip to main content

    Limitation of Anonymous Pipes

    Question Text​

    What of the following is the largest drawback of using anonymous pipes (created with pipe()) for inter-process communication?

    Question Answers​

    • they only allow unidirectional communication
    • they only allow IPC between "related" processes (parent - child - grandchild etc.)

    • if more processes use the same end of the same pipe, there may be race conditions

    • a pipe only has 2 file descriptors, but some processes may need more channels to communicate

    Feedback​

    Out of the answers above, the only limitation that cannot be overcome by using pipes alone is the requirement for the processes using them to be related. The parent must create the pipes so the children can inherit them. All other downsides can be overcome by using more pipes.

    The answer to this is to employ some filesystem support.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/bind-error-cause/index.html b/75/IO/Questions/bind-error-cause/index.html index 7c4b0a8d87..c4cb4c6b61 100644 --- a/75/IO/Questions/bind-error-cause/index.html +++ b/75/IO/Questions/bind-error-cause/index.html @@ -4,8 +4,8 @@ Cause of `bind()` Error | Operating Systems - - + +
    @@ -13,7 +13,7 @@ You will get an error. What is its cause?

    Question Answers​

    Feedback​

    One port may only be bound to one socket at a time. The fact that it's the same program (same source code) using it is irrelevant because they're different processes.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/client-server-sender-receiver/index.html b/75/IO/Questions/client-server-sender-receiver/index.html index 3694a32121..c409b9cf42 100644 --- a/75/IO/Questions/client-server-sender-receiver/index.html +++ b/75/IO/Questions/client-server-sender-receiver/index.html @@ -4,15 +4,15 @@ `sender.py` and `receiver.py` Client-Server Parallel | Operating Systems - - + +
    Skip to main content

    sender.py and receiver.py Client-Server Parallel

    Question Text​

    Drawing a parallel from the UDP examples in support/send-receive/, which of the sender and receiver is similar to the server and which is similar to the client?

    Question Answers​

    • both are similar to clients

    • both are similar to servers

    • receiver.py is similar to a server and sender.py is similar to a client
    • receiver.py is similar to a client and sender.py is similar to a server

    Feedback​

    receiver.py is the passive component. It has to be started first and then waits for sender.py (the client) to send data. Furthermore, you can only have one receiver.py running at the same time (remember the multiple bind() bug) and multiple sender.pys.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/fewer-than-2-copies/index.html b/75/IO/Questions/fewer-than-2-copies/index.html index 6757fe485b..31f6195544 100644 --- a/75/IO/Questions/fewer-than-2-copies/index.html +++ b/75/IO/Questions/fewer-than-2-copies/index.html @@ -4,8 +4,8 @@ Fewer than Two Copies | Operating Systems - - + +
    @@ -16,7 +16,7 @@ They are both connected to the CPU via the motherboard, so it's the CPU's job to do the transfer. For this, it needs a "temporary buffer". Then the NIC needs its own buffer because the speed of the network may be slower than the speed at which it receives data from the kernel, so it needs some memory where to place the "surplus" while waiting for the network to "clear".

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/file-handler-c/index.html b/75/IO/Questions/file-handler-c/index.html index 157e28a43a..761b5f2001 100644 --- a/75/IO/Questions/file-handler-c/index.html +++ b/75/IO/Questions/file-handler-c/index.html @@ -4,14 +4,14 @@ File handler in C | Operating Systems - - + +
    Skip to main content

    File handler in C

    Question Text​

    What is the type of the file handler in the C code located in chapters/io/file-descriptors/support/simple-file-operations/file_operations.c?

    Question Answers​

    • File
    • FILE *

    • FILE

    • void *

    • struct file

    Feedback​

    The file is opened using either of the following lines:

    f = fopen(file_name, "r");

    f = fopen(file_name, "w");

    The type of f is FILE *: FILE *f.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/firefox-tcp-udp/index.html b/75/IO/Questions/firefox-tcp-udp/index.html index 3a9886f12f..9b868eefdb 100644 --- a/75/IO/Questions/firefox-tcp-udp/index.html +++ b/75/IO/Questions/firefox-tcp-udp/index.html @@ -4,8 +4,8 @@ Firefox: TCP or UDP? | Operating Systems - - + +
    @@ -16,7 +16,7 @@ We don't update text-based content too often and since it needs to be precise, handling broken packets is important. On the other hand, a streaming-based site, such as YouTube sends data in real time and thus a few errors here and there can be omitted. So https://open-education-hub.github.io/operating-systems is going to be served via TCP, while YouTube videos via UDP.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/flush-libc-buffer/index.html b/75/IO/Questions/flush-libc-buffer/index.html index 2f17bab6f8..5d8c2ff7a4 100644 --- a/75/IO/Questions/flush-libc-buffer/index.html +++ b/75/IO/Questions/flush-libc-buffer/index.html @@ -4,14 +4,14 @@ Flush Libc Buffer | Operating Systems - - + +
    Skip to main content

    Flush Libc Buffer

    Question Text​

    Which of the following is a method of flushing libc's internal buffer?

    Question Answers​

    • print a \0 character
    • print a \n character

    • print a space character

    • print a \t character

    Feedback​

    Newlines (\n) force printf() to dump the internal buffer associated with the stdout FILE struct. If you place a \n character within one of the strings in support/buffering/printf_buffering.c, a write() syscall will be made right after it.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/fopen-syscall/index.html b/75/IO/Questions/fopen-syscall/index.html index 2aacab0edb..926802d99d 100644 --- a/75/IO/Questions/fopen-syscall/index.html +++ b/75/IO/Questions/fopen-syscall/index.html @@ -4,14 +4,14 @@ Syscall Used by `fopen()` | Operating Systems - - + +
    Skip to main content

    Syscall Used by fopen()

    Question Text​

    Use strace to determine the syscall called by fopen() to access the file. Which one is it?

    Question Answers​

    • read()
    • openat()

    • write()

    • fstat()

    Feedback​

    student@os:~/.../lab/support/simple-file-handling$ strace ./file_operations
    [...]
    openat(AT_FDCWD, "file.txt", O_RDONLY)  = 3
    fstat(3, {st_mode=S_IFREG|0664, st_size=11, ...}) = 0
    read(3, "C was here!", 4096)            = 11
    [...]

    So fopen()'s (main) underlying syscall is openat().

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/index.html b/75/IO/Questions/index.html index 65f0826e5c..aedaa6c335 100644 --- a/75/IO/Questions/index.html +++ b/75/IO/Questions/index.html @@ -4,13 +4,13 @@ Questions | Operating Systems - - + +
    Skip to main content

    Questions

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/local-io-errors/index.html b/75/IO/Questions/local-io-errors/index.html index 66a98fcc45..b9446e88ad 100644 --- a/75/IO/Questions/local-io-errors/index.html +++ b/75/IO/Questions/local-io-errors/index.html @@ -4,14 +4,14 @@ I/O Errors | Operating Systems - - + +
    Skip to main content

    I/O Errors

    Question Text​

    Which of the following types of errors are unlikely to occur during an I/O operation?

    Question Answers​

    • The current user does not have sufficient permissions to access a given file

    • There is not enough space left on the disk

    • The file offset has reached the end of the file when reading
    • Some data blocks in the filesystem are corrupted

    Feedback​

    We can always reposition the file offset within a given file with either a fseek() library call or an lseek() syscall, so this is not an error. The others are more difficult to manage, so can be regarded as errors.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/mmap-read-write-benchmark/index.html b/75/IO/Questions/mmap-read-write-benchmark/index.html index a3bffb41eb..90ca0671e7 100644 --- a/75/IO/Questions/mmap-read-write-benchmark/index.html +++ b/75/IO/Questions/mmap-read-write-benchmark/index.html @@ -4,8 +4,8 @@ `mmap()` vs `read()` and `write()` Benchmark | Operating Systems - - + +
    @@ -14,7 +14,7 @@ This depends on your storage device (SSD vs HDD) and its specific speed (like its RPM for an HDD). So the more conservative answer is to say that this depends on external aspects and that, in general, the 2 implementations are more or less equivalent.

    If you want to know why there isn't much of a difference between the 2 implementations, check out this explanation.

    However, some newer Linux systems use an updated cp that doesn't use read and write anymore, but instead uses zero-copy, by means of the copy_file_range() syscall. This implementation is likely going to be significantly faster than the one using mmap, as shown in the snippet below:

    student@os:/.../support/file-mappings$ ./benchmark_cp.sh
    make: Nothing to be done for 'all'.
    Benchmarking mmap_cp on a 1 GB file...

    real    0m1,443s
    user    0m0,429s
    sys     0m0,971s
    Benchmarking cp on a 1 GB file...

    real    0m0,821s
    user    0m0,001s
    sys     0m0,602s
    - - + + \ No newline at end of file diff --git a/75/IO/Questions/pipe-ends/index.html b/75/IO/Questions/pipe-ends/index.html index 4f36497c34..b785481d0e 100644 --- a/75/IO/Questions/pipe-ends/index.html +++ b/75/IO/Questions/pipe-ends/index.html @@ -4,8 +4,8 @@ Pipe Ends | Operating Systems - - + +
    @@ -16,7 +16,7 @@ Note each of these numbers is followed by a letter. As you might have guessed:

    Also, the man page is quite clear on this issue:

    pipefd[0] refers to the read end of the pipe. pipefd[1] refers to the write end of the pipe.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/receiver-socket-fd/index.html b/75/IO/Questions/receiver-socket-fd/index.html index 2d0a970556..23bbcf1476 100644 --- a/75/IO/Questions/receiver-socket-fd/index.html +++ b/75/IO/Questions/receiver-socket-fd/index.html @@ -4,13 +4,13 @@ Receiver Socked File Descriptor | Operating Systems - - + +
    Skip to main content

    Receiver Socked File Descriptor

    Question Text​

    What is the type of the file descriptor that corresponds to the socket created by support/send-receive/receiver.py?

    Question Answers​

    • only file descriptors that are linked to files have types

    • DIR

    • REG

    • CHR

    • IPv4

    Feedback​

    Running lsof yields the following output:

    student@os:~$ lsof -p 59681
    COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
    python3 59681  student  cwd    DIR    8,1        0  559322 /home/student/operating-systems-oer/content/chapters/io/lab/support/send-receive
    python3 59681  student  rtd    DIR  259,6     4096       2 /
    python3 59681  student  txt    REG  259,6  5502744 1835857 /usr/bin/python3.8
    python3 59681  student  mem    REG  259,6  8631488 1835827 /usr/lib/locale/locale-archive
    python3 59681  student  mem    REG  259,6   108936 1835887 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
    python3 59681  student  mem    REG  259,6   182560 1836149 /usr/lib/x86_64-linux-gnu/libexpat.so.1.6.11
    python3 59681  student  mem    REG  259,6  1369384 1857443 /usr/lib/x86_64-linux-gnu/libm-2.31.so
    python3 59681  student  mem    REG  259,6    14880 1857476 /usr/lib/x86_64-linux-gnu/libutil-2.31.so
    python3 59681  student  mem    REG  259,6    18848 1857439 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
    python3 59681  student  mem    REG  259,6   157224 1857471 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    python3 59681  student  mem    REG  259,6  2029592 1857435 /usr/lib/x86_64-linux-gnu/libc-2.31.so
    python3 59681  student  mem    REG  259,6    27002 2506848 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
    python3 59681  student  mem    REG  259,6   191504 1835092 /usr/lib/x86_64-linux-gnu/ld-2.31.so
    python3 59681  student    0u   CHR  136,1      0t0       4 /dev/pts/1
    python3 59681  student    1u   CHR  136,1      0t0       4 /dev/pts/1
    python3 59681  student    2u   CHR  136,1      0t0       4 /dev/pts/1
    python3 59681  student    3u  IPv4 588386      0t0     UDP localhost:5000

    The last line displays the socket:

    python3 59681  student    3u  IPv4 588386      0t0     UDP localhost:5000

    Its type is written on the the 5th column: IPv4 because it's a network socket.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/server-copies/index.html b/75/IO/Questions/server-copies/index.html index 021266beec..435d86879c 100644 --- a/75/IO/Questions/server-copies/index.html +++ b/75/IO/Questions/server-copies/index.html @@ -4,8 +4,8 @@ Client-Server Number of Copies | Operating Systems - - + +
    @@ -16,7 +16,7 @@ Following this step, the app will call send(), which will first copy the file to a buffer in the kernel. From this buffer, the kernel itself will copy the file to another buffer on the NIC (Network Interface Card). In total, there the file is copied 4 times, as outlined in the image below.

    Server Copies - Read-Send

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/stderr-fd/index.html b/75/IO/Questions/stderr-fd/index.html index 1ca945e8bc..d56c5e106c 100644 --- a/75/IO/Questions/stderr-fd/index.html +++ b/75/IO/Questions/stderr-fd/index.html @@ -4,15 +4,15 @@ File Descriptor of `stderr` | Operating Systems - - + +
    Skip to main content

    File Descriptor of stderr

    Question Text​

    Which file descriptor is associated by default to stderr?

    Question Answers​

    • it varies from process to process

    • it varies from one Linux distribution to another

    • stderr has no associated file descriptor

    • 2

    • 0

    • 1

    Feedback​

    You would type ls 2> /dev/null to ignore ls's errors. This equates to redirecting stderr to /dev/null. The number 2 in 2> hints that stderr is by default associated with file descriptor 2.

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/strace-printf/index.html b/75/IO/Questions/strace-printf/index.html index 5035f84a8e..73204deb78 100644 --- a/75/IO/Questions/strace-printf/index.html +++ b/75/IO/Questions/strace-printf/index.html @@ -4,13 +4,13 @@ `printf()` Under Strace | Operating Systems - - + +
    Skip to main content

    printf() Under Strace

    Question Text​

    How many calls to write() does the code in support/buffering/printf_buffering.c do?

    Question Answers​

    • none
    • 1

    • 6

    • 5

    Feedback​

    Just run the binary under strace:

    student@os:/.../support/buffering$ strace -e write ./printf_buffering
    write(1, "Dovahkiin, Dovahkiin Naal ok zin"..., 169Dovahkiin, Dovahkiin Naal ok zin los vahriin Wah dein vokul mahfaeraak ahst vaal! Ahrk fin norok paal graan Fod nust hon zindro zaan Dovahkiin, fah hin kogaan mu draal! ) = 169
    +++ exited with 0 +++

    There's one call to write().

    - - + + \ No newline at end of file diff --git a/75/IO/Questions/syscalls-cp/index.html b/75/IO/Questions/syscalls-cp/index.html index 9ae43a3fee..016b65ce49 100644 --- a/75/IO/Questions/syscalls-cp/index.html +++ b/75/IO/Questions/syscalls-cp/index.html @@ -4,14 +4,14 @@ Syscalls Used by `cp` | Operating Systems - - + +
    Skip to main content

    Syscalls Used by cp

    Question Text​

    What syscalls does cp use to copy files?

    Question Answers​

    • mmap()

    • read() and write()

    • copy_file_range()

    • a combination of read() - write() and mmap()

    Feedback​

    It suffices to use strace to see that cp uses read() and write().

    student@os:/.../support/file-mappings$ strace cp test-file.txt output.txt
    openat(AT_FDCWD, "test-file.txt", O_RDONLY) = 3
    fstat(3, {st_mode=S_IFREG|0664, st_size=1048576, ...}) = 0
    openat(AT_FDCWD, "output.txt", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
    fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
    [...]
    read(3, "@Y\344\0025\317\27\243\23\201:\27\342\356\240\345\331Nq\v/\36\244\200\301\247\3152\35WZ\337"..., 131072) = 131072
    write(4, "@Y\344\0025\317\27\243\23\201:\27\342\356\240\345\331Nq\v/\36\244\200\301\247\3152\35WZ\337"..., 131072) = 131072
    read(3, "\201\240J7x\275\257Z\343\334\307d<\321U\275\337\10\233j\222\313,##cQD\268e\324"..., 131072) = 131072
    write(4, "\201\240J7x\275\257Z\343\334\307d<\321U\275\337\10\233j\222\313,##cQD\268e\324"..., 131072) = 131072
    read(3, "\371\3244=\17\300L9\243\201\362\25\273\37\326\323\362\200\1T\310N\316\305\253\331\331Nt\346\3369"..., 131072) = 131072
    write(4, "\371\3244=\17\300L9\243\201\362\25\273\37\326\323\362\200\1T\310N\316\305\253\331\331Nt\346\3369"..., 131072) = 131072
    read(3, "\350\304\345f\16\305V\356\371\263?+\355{\16\235\344\310P4}\2043%\0052\345D\265\243t\354"..., 131072) = 131072
    write(4, "\350\304\345f\16\305V\356\371\263?+\355{\16\235\344\310P4}\2043%\0052\345D\265\243t\354"..., 131072) = 131072
    read(3, "\277\226\315\226\n\37\337;N*\211\352\254$\273\2\351\30a\254\ta\352R\25-\23\274\376zy\211"..., 131072) = 131072
    write(4, "\277\226\315\226\n\37\337;N*\211\352\254$\273\2\351\30a\254\ta\352R\25-\23\274\376zy\211"..., 131072) = 131072
    read(3, "}\245\356;\222\327\204\242u\26dy%\346\374\201ndT\226\233\3575\345\247\356\362\344\350\354\17\261"..., 131072) = 131072
    write(4, "}\245\356;\222\327\204\242u\26dy%\346\374\201ndT\226\233\3575\345\247\356\362\344\350\354\17\261"..., 131072) = 131072
    read(3, "\35\277\207\243~\355(i\351^\1\346\312V\232\204\32\230~\376\20\245\"\305\344d\304\304B\272\346\332"..., 131072) = 131072
    write(4, "\35\277\207\243~\355(i\351^\1\346\312V\232\204\32\230~\376\20\245\"\305\344d\304\304B\272\346\332"..., 131072) = 131072
    read(3, "\n)\334\275\331:R\236O\231\243\302\314\267\326\"\rY\262\21\374\305\275\3\332\23\345\16>\214\210\235"..., 131072) = 131072
    write(4, "\n)\334\275\331:R\236O\231\243\302\314\267\326\"\rY\262\21\374\305\275\3\332\23\345\16>\214\210\235"..., 131072) = 131072

    Alternatively, if your kernel version is 5.19 or newer, it's likely that cp will use copy_file_range():

    student@os:/.../support/file-mappings$ strace cp test-file.txt output.txt
    openat(AT_FDCWD, "test-file.txt", O_RDONLY) = 3
    newfstatat(3, "", {st_mode=S_IFREG|0664, st_size=1048576, ...}, AT_EMPTY_PATH) = 0
    openat(AT_FDCWD, "output.txt", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
    ioctl(4, BTRFS_IOC_CLONE or FICLONE, 3) = -1 EOPNOTSUPP (Operation not supported)
    newfstatat(4, "", {st_mode=S_IFREG|0644, st_size=0, ...}, AT_EMPTY_PATH) = 0
    fadvise64(3, 0, 0, POSIX_FADV_SEQUENTIAL) = 0
    copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 1048576
    copy_file_range(3, NULL, 4, NULL, 9223372035781033984, 0) = 0

    This syscall copies files inside the kernel without having the data pass through the user space. This procedure is called zero-copy.

    - - + + \ No newline at end of file diff --git a/75/IO/index.html b/75/IO/index.html index 417df67df3..f57f80a881 100644 --- a/75/IO/index.html +++ b/75/IO/index.html @@ -4,13 +4,13 @@ IO | Operating Systems - - + +
    Skip to main content

    IO


    tip

    Focus the slides and press F for fullscreen viewing.

    - - + + \ No newline at end of file diff --git a/75/IO/io-overview/index.html b/75/IO/io-overview/index.html index ea5a3dd9ab..e1ebb17cbb 100644 --- a/75/IO/io-overview/index.html +++ b/75/IO/io-overview/index.html @@ -4,8 +4,8 @@ I/O | Operating Systems - - + +
    @@ -22,7 +22,7 @@ If we want it to be responsive and to do something useful, most likely, the I/O is the key. So I/O is crucial for most applications, yet it is also the slowest...

    Sad Pepe

    But fear not! There are countless optimisations out there aimed precisely at bridging the speed gap between the memory and the disk.

    - - + + \ No newline at end of file diff --git a/75/IO/lab10/index.html b/75/IO/lab10/index.html index 4cc145a2f7..2814b3b0eb 100644 --- a/75/IO/lab10/index.html +++ b/75/IO/lab10/index.html @@ -4,8 +4,8 @@ Lab 10 - Inter-Process Communication | Operating Systems - - + +
    @@ -98,7 +98,7 @@ The first 1024 ports are reserved for well-known applications, ensuring consistency across networks. For example, SSH uses port 22 and Apache2 uses port 80 for both IPv4 and IPv6 addresses (look for rows starting with tcp for IPv4 and tcp6 for IPv6).

    Some user programs, like Firefox, establish multiple connections, often using both IPv4 and IPv6, with each connection assigned a unique port. Discord is another example, using TCP to handle text messages, images, videos, and other static content, while relying on UDP for real-time voice and video data during calls.

    Quiz: Why does Firefox uses both TCP and UDP?

    Conclusion​

    The difference between TCP and UDP can be summarised as follows:

    TCP vs UDP

    - - + + \ No newline at end of file diff --git a/75/IO/lab11/index.html b/75/IO/lab11/index.html index 1dba41d311..9edc67821a 100644 --- a/75/IO/lab11/index.html +++ b/75/IO/lab11/index.html @@ -4,8 +4,8 @@ Lab 11 - IO Optimizations | Operating Systems - - + +
    @@ -18,7 +18,7 @@ Test your client by running python server.py in one terminal and then ./client in another. If correctly implemented, you should be able to exchange messages as outlined above.

    Bonus Question: Why is it OK for the client to be implemented in C while the server is implemented in Python?

  • Open support/server.c and complete the TODOs to enable message exchange with the client. Test your server by running ./server in one terminal and then python client.py in another. -If implemented correctly, you should be able to exchange messages as outlined above.

    • If you're having difficulties solving this exercise, go through the sockets API and the client-server model.

    Task: Multiplexed Client Server​

    Navigate to chapters/io/optimizations/drills/tasks/client-server-epoll and run make to generate the support files.

    This task builds on the previous implementation of a client-server ordered communication. +If implemented correctly, you should be able to exchange messages as outlined above.

    If you're having difficulties solving this exercise, go through the sockets API and the client-server model.

    Task: Multiplexed Client Server​

    Navigate to chapters/io/optimizations/drills/tasks/multiplexed-client-server and run make to generate the support files.

    This task builds on the previous implementation of a client-server ordered communication. Previously, the client and server followed a strict, sequential communication pattern: each peer had to send a message and wait for a reply before proceeding.

    We plan to build a group chat where clients can send messages at any time, and each message is broadcast to all other connected clients. To accomplish this, we’ll implement I/O multiplexing mechanisms that notify us only when data is available on a file descriptor. This non-blocking approach allows for smooth, unhindered communication between clients.

    1. We’ll use the epoll interface to manage multiple file descriptors. @@ -96,7 +96,7 @@ The output might look like this:

      student@os:~/async/support$ time ./client_bench.sh 2000
      [...]
      root: Connected to localhost:2000
      root: Sending 30
      function(30): 1346269

      real    0m1.075s
      user    0m0.301s
      sys     0m0.029s

      Although the actual value may vary, you should observe a noticeable speed-up when testing the other two solutions.

    2. Next, benchmark the multiprocess and multithreaded alternatives to determine which offers the best performance improvement. To obtain more meaningful results, adjust parameters in client_bench.sh, such as increasing the number of clients or the Fibonacci value to compute.

    If you're having difficulties understanding the support code, go through this reading material. If you want to practice this yourself, go through the Async Server task.

    - - + + \ No newline at end of file diff --git a/75/IO/lab9/index.html b/75/IO/lab9/index.html index 0333e4063a..0f73ac00f8 100644 --- a/75/IO/lab9/index.html +++ b/75/IO/lab9/index.html @@ -4,12 +4,12 @@ Lab 9 - File Descriptors | Operating Systems - - + +
    -
    Skip to main content

    Lab 9 - File Descriptors

    Task: My cat​

    Navigate to file-descriptors/drills/tasks/my-cat/support/src and checkout my_cat.c. +

    Lab 9 - File Descriptors

    Task: My cat​

    Navigate to chapters/io/file-descriptors/drills/tasks/my-cat/support/src and checkout my_cat.c. We propose to implement the Linux command cat that reads one or more files, concatenates them (hence the name cat), and prints them to standard output.

    1. Implement rread() wrapper over read().

      read() system call does not guarantee that it will read the requested number of bytes in a single call. This happens when the file does not have enough bytes, or when read() is interrupted by a signal. rread() will handle these situations, ensuring that it reads either num_bytes or all available bytes.

    2. Implement wwrite() as a wrapper for write().

      The write() system call may not write the requested number of bytes in a single call. @@ -23,7 +23,7 @@ Ensure they have the same content with a simple diff: diff test-file.txt output.txt.

    3. Compare your implementation to the cp command. Run make large-file to generate a 1GB file with random data, and then run ./benchmark_cp.sh.

      Quiz: Debunk why cp is winning

      If you want a more generic answer, checkout this guide on mmap vs read()-write().

    4. This demo would not be complete without some live analysis. Uncomment the calls to wait_for_input() and rerun the program. -In another terminal, run cat /proc/$(pidof mmap_cp)/maps to see mapped files, and ps -o pid,vsz,rss <PID> to see how demand paging happens.

    Task: Anonymous Pipes Communication​

    Navigate to chapters/io/file-descriptors/drills/tasks/anon-pipes/support and run make. +In another terminal, run cat /proc/$(pidof mmap_cp)/maps to see mapped files, and ps -o pid,vsz,rss <PID> to see how demand paging happens.

    Task: Anonymous Pipes Communication​

    Navigate to chapters/io/ipc/drills/tasks/anon-pipes and run make to generate the support/ folder. In this exercise, you'll implement client-server communication between a parent and a child process using an anonymous pipe. The parent will act as the sender, while the child acts as the receiver, with both processes sharing messages through the pipe. Since pipes are unidirectional, each process should close the end of the pipe it does not use.

    1. Use the pipe() syscall to create the pipe. @@ -146,7 +146,7 @@ Most of the time, opendir() is used instead.

    In conclusion, the key difference between fopen() and open() is in the type of handler they return. The FILE structure from fopen() is suited only for regular files, while the file descriptor returned by open() is more flexible. The differences between the two handlers are explored in the file descriptors section.

    - - + + \ No newline at end of file diff --git a/75/Lecture/Application-Interaction/index.html b/75/Lecture/Application-Interaction/index.html index c837a44472..f71130ff15 100644 --- a/75/Lecture/Application-Interaction/index.html +++ b/75/Lecture/Application-Interaction/index.html @@ -4,13 +4,13 @@ Application-Interaction | Operating Systems - - + +

    Application-Interaction


    tip

    Focus the slides and press F for fullscreen viewing.

    - - + + \ No newline at end of file diff --git a/75/Lecture/IO/index.html b/75/Lecture/IO/index.html index 7ef9a37952..22fbba0a4b 100644 --- a/75/Lecture/IO/index.html +++ b/75/Lecture/IO/index.html @@ -4,13 +4,13 @@ IO | Operating Systems - - + +

    IO


    tip

    Focus the slides and press F for fullscreen viewing.

    - - + + \ No newline at end of file diff --git a/75/Lecture/index.html b/75/Lecture/index.html index 829e50e6b4..816ccd3423 100644 --- a/75/Lecture/index.html +++ b/75/Lecture/index.html @@ -4,13 +4,13 @@ Lecture | Operating Systems - - + +

    Lecture

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/dynamic-libraries/index.html b/75/Software Stack/Questions/dynamic-libraries/index.html index 85139eff4d..71f0e9ea18 100644 --- a/75/Software Stack/Questions/dynamic-libraries/index.html +++ b/75/Software Stack/Questions/dynamic-libraries/index.html @@ -4,13 +4,13 @@ Dynamic Libraries | Operating Systems - - + +

    Dynamic Libraries

    Question Text​

    Which of the following is a dynamic library?

    Question Answers​

    • libc.a
    • libc.so

    • libc.exe

    • libc.c

    Feedback​

    The .so (shared object) extension is used for dynamic libraries in Linux.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/index.html b/75/Software Stack/Questions/index.html index b4e7545a47..0d086c3d82 100644 --- a/75/Software Stack/Questions/index.html +++ b/75/Software Stack/Questions/index.html @@ -4,13 +4,13 @@ Questions | Operating Systems - - + +

    Questions

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/libc/index.html b/75/Software Stack/Questions/libc/index.html index 3c94ea978b..fcb21a2f39 100644 --- a/75/Software Stack/Questions/libc/index.html +++ b/75/Software Stack/Questions/libc/index.html @@ -4,13 +4,13 @@ Syscall Tool | Operating Systems - - + +

    Syscall Tool

    Question Text​

    Which of following is *not- and advantage of using libc for programs?

    Question Answers​

    • increased portability
    • reduced executable size

    • richer set of features

    • easier development

    Feedback​

    When using libc, because we add a new software component, the size of the resulting executable increases.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/libcall-syscall/index.html b/75/Software Stack/Questions/libcall-syscall/index.html index d06452d0ae..11014752cf 100644 --- a/75/Software Stack/Questions/libcall-syscall/index.html +++ b/75/Software Stack/Questions/libcall-syscall/index.html @@ -4,14 +4,14 @@ Libcall with Syscall | Operating Systems - - + +

    Libcall with Syscall

    Question Text​

    Which of the following library calls will for sure invoke a system call?

    Question Answers​

    • fopen()

    • fwrite()

    • printf()

    • strcpy()

    Feedback​

    fopen() requires opening a file and access to the operating system (for filesystem access). The others may not require a system call (strcpy()) or may use buffering to delay the invocation of a system call (fwrite(), printf()).

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/malloc/index.html b/75/Software Stack/Questions/malloc/index.html index 3e72f44ed4..0ce035b5bd 100644 --- a/75/Software Stack/Questions/malloc/index.html +++ b/75/Software Stack/Questions/malloc/index.html @@ -4,13 +4,13 @@ malloc() | Operating Systems - - + +

    malloc()

    Question Text​

    What system calls are invoked by the malloc() library call for Linux libc? (choose 2 answers)

    Question Answers​

    • brk

    • free

    • dup

    • mmap
    • copy

    Feedback​

    Depending on the allocation size, malloc() invokes brk or mmap.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/printf-syscall/index.html b/75/Software Stack/Questions/printf-syscall/index.html index 9dc577baab..b08b5d7f53 100644 --- a/75/Software Stack/Questions/printf-syscall/index.html +++ b/75/Software Stack/Questions/printf-syscall/index.html @@ -4,13 +4,13 @@ printf() System Call | Operating Systems - - + +

    printf() System Call

    Question Text​

    What system call does the printf() function invoke?

    Question Answers​

    • read
    • write

    • exec

    • exit

    Feedback​

    printf() invokes the write system call to print messages to standard output.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/printf-vs-write/index.html b/75/Software Stack/Questions/printf-vs-write/index.html index 9219a28854..7f60495384 100644 --- a/75/Software Stack/Questions/printf-vs-write/index.html +++ b/75/Software Stack/Questions/printf-vs-write/index.html @@ -4,14 +4,14 @@ printf() vs write | Operating Systems - - + +

    printf() vs write

    Question Text​

    What are features provided by printf() when compared to write? (choose 2 answers)

    Question Answers​

    • buffering

    • outputs to standard output

    • may write to file

    • does output formatting
    • can work with binary data

    Feedback​

    printf() can do buffering to reduce the number of system calls. Also, printf(), as it name suggests (the f suffix), does output formatting.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/python-tools/index.html b/75/Software Stack/Questions/python-tools/index.html index d204ec9845..25ba5061cd 100644 --- a/75/Software Stack/Questions/python-tools/index.html +++ b/75/Software Stack/Questions/python-tools/index.html @@ -4,14 +4,14 @@ Python Tools | Operating Systems - - + +

    Python Tools

    Question Text​

    A Python program is not a proper argument for ...

    Question Answers​

    • ltrace

    • strace

    • rm

    • ldd

    Feedback​

    Because a Python program is a script to be interpreted, it won't be usable by ldd. ldd is passed a binary executable to look for library dependencies.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/software/index.html b/75/Software Stack/Questions/software/index.html index ca44a27745..482ab31ca3 100644 --- a/75/Software Stack/Questions/software/index.html +++ b/75/Software Stack/Questions/software/index.html @@ -4,15 +4,15 @@ Software Properties | Operating Systems - - + +

    Software Properties

    Question Text​

    Which of the following is *not- an advantage of software when compared to hardware?

    Question Answers​

    • reusability
    • performance

    • portability

    • flexibility

    Feedback​

    Software is reusable and flexible, unlike physical hardware that's rigid and used-once. Software is also portable across different hardware (and software platforms). However, software relies on hardware for running, so software will never beat hardware in terms of sheer speed / performance.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/static-executables/index.html b/75/Software Stack/Questions/static-executables/index.html index ab9e6497e1..388da500ad 100644 --- a/75/Software Stack/Questions/static-executables/index.html +++ b/75/Software Stack/Questions/static-executables/index.html @@ -4,14 +4,14 @@ Static Executables | Operating Systems - - + +

    Static Executables

    Question Text​

    Which of the following tools has no influence over statically-linked executables? (choose 2 answers)

    Question Answers​

    • ltrace

    • strace

    • nm

    • ldd
    • rm

    Feedback​

    ltrace and ldd are used to capture connections to dynamic libraries. They have no effect on statically-linked executables.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/strcpy-syscall/index.html b/75/Software Stack/Questions/strcpy-syscall/index.html index 3cda814cc9..abedc31279 100644 --- a/75/Software Stack/Questions/strcpy-syscall/index.html +++ b/75/Software Stack/Questions/strcpy-syscall/index.html @@ -4,13 +4,13 @@ strcpy() System Call | Operating Systems - - + +

    strcpy() System Call

    Question Text​

    What system call does the strcpy() function invoke?

    Question Answers​

    • cpy

    • touch

    • memcpy

    • no system call

    Feedback​

    strcpy() doesn't invoke system calls, because it doesn't require any feature that is only provided by the operating system.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/syscall-id/index.html b/75/Software Stack/Questions/syscall-id/index.html index 7a094c2933..20529d2eb4 100644 --- a/75/Software Stack/Questions/syscall-id/index.html +++ b/75/Software Stack/Questions/syscall-id/index.html @@ -4,13 +4,13 @@ Syscall ID | Operating Systems - - + +

    Syscall ID

    Question Text​

    What register stores the system call ID on x86_64?

    Question Answers​

    • RIP

    • RSP

    • RAX
    • RDX

    Feedback​

    RAX is the register used for passing the syscall ID and the result code.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/syscall-numbers/index.html b/75/Software Stack/Questions/syscall-numbers/index.html index de22c43566..2c376f08da 100644 --- a/75/Software Stack/Questions/syscall-numbers/index.html +++ b/75/Software Stack/Questions/syscall-numbers/index.html @@ -4,13 +4,13 @@ Syscall Numbers | Operating Systems - - + +

    Syscall Numbers

    Question Text​

    What is the approximate number of system call numbers in Linux?

    Question Answers​

    • 3

    • 30

    • 300
    • 3000

    Feedback​

    As show here, they're about 300 system calls in Linux.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/syscall-tool/index.html b/75/Software Stack/Questions/syscall-tool/index.html index 516cf9f750..c1372b66ad 100644 --- a/75/Software Stack/Questions/syscall-tool/index.html +++ b/75/Software Stack/Questions/syscall-tool/index.html @@ -4,13 +4,13 @@ Syscall Tool | Operating Systems - - + +

    Syscall Tool

    Question Text​

    What tool do we use to capture system calls?

    Question Answers​

    • strace

    • make

    • gcc

    • ./exec

    Feedback​

    strace is used to trace system calls invoked by a running program.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Questions/syscall-wrapper/index.html b/75/Software Stack/Questions/syscall-wrapper/index.html index cf72058193..f0d3d086b7 100644 --- a/75/Software Stack/Questions/syscall-wrapper/index.html +++ b/75/Software Stack/Questions/syscall-wrapper/index.html @@ -4,13 +4,13 @@ Syscall Wrappers | Operating Systems - - + +

    Syscall Wrappers

    Question Text​

    What language do we use to invoke system calls?

    Question Answers​

    • assembly

    • C

    • C++

    • Go

    Feedback​

    System calls require setting of registers that can only be done in assembly language.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/Software-Stack/index.html b/75/Software Stack/Software-Stack/index.html index 8f26f270ef..9abf0c5be1 100644 --- a/75/Software Stack/Software-Stack/index.html +++ b/75/Software Stack/Software-Stack/index.html @@ -4,13 +4,13 @@ Software-Stack | Operating Systems - - + +

    Software-Stack


    tip

    Focus the slides and press F for fullscreen viewing.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/index.html b/75/Software Stack/index.html index 08f89e07ef..7614c34f6a 100644 --- a/75/Software Stack/index.html +++ b/75/Software Stack/index.html @@ -4,13 +4,13 @@ Software Stack | Operating Systems - - + +

    Software Stack

    - - + + \ No newline at end of file diff --git a/75/Software Stack/lab1/index.html b/75/Software Stack/lab1/index.html index 7e0a632266..e7d448c3ed 100644 --- a/75/Software Stack/lab1/index.html +++ b/75/Software Stack/lab1/index.html @@ -4,8 +4,8 @@ Lab 1 - Operating System Perspective | Operating Systems - - + +
    @@ -70,7 +70,7 @@ Actual system calls are made either when the standard C library buffer is full or when an fflush() library call is made.

    Note that on some systems, ltrace does not work*- as expected, due to now binding. To avoid this behaviour, you can force the lazy binding- (based on which ltrace is constructed to work). An example can be found in libcall-syscall/support/Makefile, however for system binaries, such as ls or pwd, the only alternative is to add the `-x ""` argument to force the command to trace all symbols in the symbol table:

    student@os:~$ ltrace -x "*" ls

    You can always choose what library functions ltrace is investigating, by replacing the wildcard with their name:

    student@os:~$ ltrace -x "malloc" -x "free" ls
    malloc@libc.so.6(5)                                                    = 0x55c42b2b8910
    free@libc.so.6(0x55c42b2b8910)                                         = <void>
    malloc@libc.so.6(120)                                                  = 0x55c42b2b8480
    malloc@libc.so.6(12)                                                   = 0x55c42b2b8910
    malloc@libc.so.6(776)                                                  = 0x55c42b2b8930
    malloc@libc.so.6(112)                                                  = 0x55c42b2b8c40
    malloc@libc.so.6(1336)                                                 = 0x55c42b2b8cc0
    malloc@libc.so.6(216)                                                  = 0x55c42b2b9200
    malloc@libc.so.6(432)                                                  = 0x55c42b2b92e0
    malloc@libc.so.6(104)                                                  = 0x55c42b2b94a0
    malloc@libc.so.6(88)                                                   = 0x55c42b2b9510
    malloc@libc.so.6(120)                                                  = 0x55c42b2b9570
    [...]

    If you would like to know more about lazy binding, now binding*- or PLT*- entries, check out this blog post.

    - - + + \ No newline at end of file diff --git a/75/Software Stack/lab2/index.html b/75/Software Stack/lab2/index.html index fb2cf9ff14..816f6ebf0e 100644 --- a/75/Software Stack/lab2/index.html +++ b/75/Software Stack/lab2/index.html @@ -4,8 +4,8 @@ Lab 2 - Library Perspective | Operating Systems - - + +
    @@ -101,7 +101,7 @@ Generally, additional software layers (libraries, language environments, interpreters) simplify development but decrease speed and efficiency.

    App Investigation​

    Let's spend some time investigating actual applications residing on the local system. For now, we know that applications are developed using high-level languages and then compiled or interpreted to use the lower-layer interfaces of the software stack, such as the system call API.

    Let's enter the chapters/software-stack/applications/drills/tasks/app-investigation/support/ folder and run the get_app_types.sh script:

    student@os:~/.../app-investigation/support/$ ./get_app_types.sh
    binary apps: 2223
    Perl apps: 256
    Shell apps: 454
    Python apps: 123
    Other apps: 27

    The script prints the types of the application executables in the system. The output will differ between systems, given each has particular types of applications installed.

    We list them by running the command inside the get_app_types.sh script:

    student@os:~/.../app-investigation/support/$ find /usr/bin /bin /usr/sbin /sbin -type f -exec file {} \;
    [...]
    /usr/bin/qpdldecode: ELF 64-bit LSB shared object, x86-64 [...]
    /usr/bin/mimeopen: Perl script text executable
    [...]

    As above, the output will differ between systems.

    So, depending on the developers' choice, applications may be:

    - - + + \ No newline at end of file diff --git a/75/Software Stack/software-stack-overview/index.html b/75/Software Stack/software-stack-overview/index.html index cb70337493..2394dff844 100644 --- a/75/Software Stack/software-stack-overview/index.html +++ b/75/Software Stack/software-stack-overview/index.html @@ -4,8 +4,8 @@ Software Stack | Operating Systems - - + +
    @@ -24,7 +24,7 @@ These generic interfaces "hides" possible differences in the even lower layers. This way, a software stack ensures portability across different other parts of software (and hardware as well). For example, the standard C library, that we will present shortly, ensures portability across different operating systems.

    Software Stack

    Contents​

    - - + + \ No newline at end of file diff --git a/75/assets/js/2117978d.fda07899.js b/75/assets/js/2117978d.ffe3f645.js similarity index 97% rename from 75/assets/js/2117978d.fda07899.js rename to 75/assets/js/2117978d.ffe3f645.js index 15ad69e785..c305ef8412 100644 --- a/75/assets/js/2117978d.fda07899.js +++ b/75/assets/js/2117978d.ffe3f645.js @@ -1 +1 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8640],{5680:(e,t,r)=>{r.d(t,{xA:()=>p,yg:()=>y});var n=r(6540);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function s(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var c=n.createContext({}),l=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):s(s({},t),e)),r},p=function(e){var t=l(e.components);return n.createElement(c.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},h=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,c=e.parentName,p=a(e,["components","mdxType","originalType","parentName"]),u=l(r),h=o,y=u["".concat(c,".").concat(h)]||u[h]||f[h]||i;return r?n.createElement(y,s(s({ref:t},p),{},{components:r})):n.createElement(y,s({ref:t},p))}));function y(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,s=new Array(i);s[0]=h;var a={};for(var c in t)hasOwnProperty.call(t,c)&&(a[c]=t[c]);a.originalType=e,a[u]="string"==typeof e?e:o,s[1]=a;for(var l=2;l{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>s,default:()=>f,frontMatter:()=>i,metadata:()=>a,toc:()=>l});var n=r(8168),o=(r(6540),r(5680));const i={},s="Fewer than Two Copies",a={unversionedId:"IO/Questions/fewer-than-2-copies",id:"IO/Questions/fewer-than-2-copies",title:"Fewer than Two Copies",description:"Question Text",source:"@site/docs/IO/Questions/fewer-than-2-copies.md",sourceDirName:"IO/Questions",slug:"/IO/Questions/fewer-than-2-copies",permalink:"/operating-systems/75/IO/Questions/fewer-than-2-copies",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Client-Server Number of Copies",permalink:"/operating-systems/75/IO/Questions/server-copies"},next:{title:"`printf()` Under Strace",permalink:"/operating-systems/75/IO/Questions/strace-printf"}},c={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:l},u="wrapper";function f(e){let{components:t,...i}=e;return(0,o.yg)(u,(0,n.A)({},p,i,{components:t,mdxType:"MDXLayout"}),(0,o.yg)("h1",{id:"fewer-than-two-copies"},"Fewer than Two Copies"),(0,o.yg)("h2",{id:"question-text"},"Question Text"),(0,o.yg)("p",null,"Can zero-copy be implemented so as to copy the file fewer than 2 times?"),(0,o.yg)("p",null,(0,o.yg)("img",{alt:"Zero-Copy",src:r(6317).A})),(0,o.yg)("h2",{id:"question-answers"},"Question Answers"),(0,o.yg)("ul",null,(0,o.yg)("li",{parentName:"ul"},"Yes, by copying the file straight from the disk to the network")),(0,o.yg)("ul",null,(0,o.yg)("li",{parentName:"ul"},"No")),(0,o.yg)("ul",null,(0,o.yg)("li",{parentName:"ul"},(0,o.yg)("p",{parentName:"li"},"Yes, by sending the file straight from the kernel buffer to the network")),(0,o.yg)("li",{parentName:"ul"},(0,o.yg)("p",{parentName:"li"},"Yes, by copying the file directly from the storage to the NIC's buffer"))),(0,o.yg)("h2",{id:"feedback"},"Feedback"),(0,o.yg)("p",null,'The truth is that we can\'t have fewer copies while using a server with a common PC-like architecture.\nThe disk is not directly connected to the internet, so the file cannot be sent directly from there.\nThe only place from where we can send data to the Web is the NIC.\nThen we need the intermediary storage in that kernel buffer because the disk and the NIC aren\'t directly connected.\nThey are both connected to the CPU via the motherboard, so it\'s the CPU\'s job to do the transfer.\nFor this, it needs a "temporary buffer".\nThen the NIC needs its own buffer because the speed of the network may be slower than the speed at which it receives data from the kernel, so it needs some memory where to place the "surplus" while waiting for the network to "clear".'))}f.isMDXComponent=!0},6317:(e,t,r)=>{r.d(t,{A:()=>n});const n=r.p+"assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg"}}]); \ No newline at end of file +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8640],{5680:(e,t,r)=>{r.d(t,{xA:()=>p,yg:()=>y});var n=r(6540);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function s(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var c=n.createContext({}),l=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):s(s({},t),e)),r},p=function(e){var t=l(e.components);return n.createElement(c.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},h=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,c=e.parentName,p=a(e,["components","mdxType","originalType","parentName"]),u=l(r),h=o,y=u["".concat(c,".").concat(h)]||u[h]||f[h]||i;return r?n.createElement(y,s(s({ref:t},p),{},{components:r})):n.createElement(y,s({ref:t},p))}));function y(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,s=new Array(i);s[0]=h;var a={};for(var c in t)hasOwnProperty.call(t,c)&&(a[c]=t[c]);a.originalType=e,a[u]="string"==typeof e?e:o,s[1]=a;for(var l=2;l{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>s,default:()=>f,frontMatter:()=>i,metadata:()=>a,toc:()=>l});var n=r(8168),o=(r(6540),r(5680));const i={},s="Fewer than Two Copies",a={unversionedId:"IO/Questions/fewer-than-2-copies",id:"IO/Questions/fewer-than-2-copies",title:"Fewer than Two Copies",description:"Question Text",source:"@site/docs/IO/Questions/fewer-than-2-copies.md",sourceDirName:"IO/Questions",slug:"/IO/Questions/fewer-than-2-copies",permalink:"/operating-systems/75/IO/Questions/fewer-than-2-copies",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Client-Server Number of Copies",permalink:"/operating-systems/75/IO/Questions/server-copies"},next:{title:"`printf()` Under Strace",permalink:"/operating-systems/75/IO/Questions/strace-printf"}},c={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:l},u="wrapper";function f(e){let{components:t,...i}=e;return(0,o.yg)(u,(0,n.A)({},p,i,{components:t,mdxType:"MDXLayout"}),(0,o.yg)("h1",{id:"fewer-than-two-copies"},"Fewer than Two Copies"),(0,o.yg)("h2",{id:"question-text"},"Question Text"),(0,o.yg)("p",null,"Can zero-copy be implemented so as to copy the file fewer than 2 times?"),(0,o.yg)("p",null,(0,o.yg)("img",{alt:"Zero-Copy",src:r(4378).A})),(0,o.yg)("h2",{id:"question-answers"},"Question Answers"),(0,o.yg)("ul",null,(0,o.yg)("li",{parentName:"ul"},"Yes, by copying the file straight from the disk to the network")),(0,o.yg)("ul",null,(0,o.yg)("li",{parentName:"ul"},"No")),(0,o.yg)("ul",null,(0,o.yg)("li",{parentName:"ul"},(0,o.yg)("p",{parentName:"li"},"Yes, by sending the file straight from the kernel buffer to the network")),(0,o.yg)("li",{parentName:"ul"},(0,o.yg)("p",{parentName:"li"},"Yes, by copying the file directly from the storage to the NIC's buffer"))),(0,o.yg)("h2",{id:"feedback"},"Feedback"),(0,o.yg)("p",null,'The truth is that we can\'t have fewer copies while using a server with a common PC-like architecture.\nThe disk is not directly connected to the internet, so the file cannot be sent directly from there.\nThe only place from where we can send data to the Web is the NIC.\nThen we need the intermediary storage in that kernel buffer because the disk and the NIC aren\'t directly connected.\nThey are both connected to the CPU via the motherboard, so it\'s the CPU\'s job to do the transfer.\nFor this, it needs a "temporary buffer".\nThen the NIC needs its own buffer because the speed of the network may be slower than the speed at which it receives data from the kernel, so it needs some memory where to place the "surplus" while waiting for the network to "clear".'))}f.isMDXComponent=!0},4378:(e,t,r)=>{r.d(t,{A:()=>n});const n=r.p+"assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/463aadd7.06309209.js b/75/assets/js/463aadd7.06309209.js new file mode 100644 index 0000000000..9ef4d1afe5 --- /dev/null +++ b/75/assets/js/463aadd7.06309209.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2213],{5680:(e,n,t)=>{t.d(n,{xA:()=>d,yg:()=>h});var a=t(6540);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function l(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var s=a.createContext({}),p=function(e){var n=a.useContext(s),t=n;return e&&(t="function"==typeof e?e(n):l(l({},n),e)),t},d=function(e){var n=p(e.components);return a.createElement(s.Provider,{value:n},e.children)},g="mdxType",u={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},m=a.forwardRef((function(e,n){var t=e.components,i=e.mdxType,r=e.originalType,s=e.parentName,d=o(e,["components","mdxType","originalType","parentName"]),g=p(t),m=i,h=g["".concat(s,".").concat(m)]||g[m]||u[m]||r;return t?a.createElement(h,l(l({ref:n},d),{},{components:t})):a.createElement(h,l({ref:n},d))}));function h(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var r=t.length,l=new Array(r);l[0]=m;var o={};for(var s in n)hasOwnProperty.call(n,s)&&(o[s]=n[s]);o.originalType=e,o[g]="string"==typeof e?e:i,l[1]=o;for(var p=2;p{t.r(n),t.d(n,{assets:()=>s,contentTitle:()=>l,default:()=>u,frontMatter:()=>r,metadata:()=>o,toc:()=>p});var a=t(8168),i=(t(6540),t(5680));const r={},l="Minishell",o={unversionedId:"Assignments/Mini Shell/README",id:"Assignments/Mini Shell/README",title:"Minishell",description:"Objectives",source:"@site/docs/Assignments/Mini Shell/README.md",sourceDirName:"Assignments/Mini Shell",slug:"/Assignments/Mini Shell/",permalink:"/operating-systems/75/Assignments/Mini Shell/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Parallel Firewall",permalink:"/operating-systems/75/Assignments/Parallel Firewall/"},next:{title:"Asynchronous Web Server",permalink:"/operating-systems/75/Assignments/Asynchronous Web Server/"}},s={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Introduction",id:"introduction",level:3},{value:"Shell Functionalities",id:"shell-functionalities",level:3},{value:"Changing the Current Directory",id:"changing-the-current-directory",level:4},{value:"Closing the Shell",id:"closing-the-shell",level:4},{value:"Running an Application",id:"running-an-application",level:4},{value:"Environment Variables",id:"environment-variables",level:4},{value:"Operators",id:"operators",level:4},{value:"Sequential Operator",id:"sequential-operator",level:5},{value:"Parallel Operator",id:"parallel-operator",level:5},{value:"Pipe Operator",id:"pipe-operator",level:5},{value:"Chain Operators for Conditional Execution",id:"chain-operators-for-conditional-execution",level:5},{value:"Operator Priority",id:"operator-priority",level:5},{value:"I/O Redirection",id:"io-redirection",level:4},{value:"Support Code",id:"support-code",level:2},{value:"Building mini-shell",id:"building-mini-shell",level:3},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Running the Checker",id:"running-the-checker",level:3},{value:"Running the Linters",id:"running-the-linters",level:3},{value:"Debugging",id:"debugging",level:3},{value:"Memory leaks",id:"memory-leaks",level:3}],d={toc:p},g="wrapper";function u(e){let{components:n,...t}=e;return(0,i.yg)(g,(0,a.A)({},d,t,{components:n,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"minishell"},"Minishell"),(0,i.yg)("h2",{id:"objectives"},"Objectives"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"Learn how shells create new child processes and connect the I/O to the terminal."),(0,i.yg)("li",{parentName:"ul"},"Gain a better understanding of the ",(0,i.yg)("inlineCode",{parentName:"li"},"fork()")," function wrapper."),(0,i.yg)("li",{parentName:"ul"},"Learn to correctly execute commands written by the user and treat errors.")),(0,i.yg)("h2",{id:"statement"},"Statement"),(0,i.yg)("h3",{id:"introduction"},"Introduction"),(0,i.yg)("p",null,"A shell is a command-line interpreter that provides a text-based user interface for operating systems.\nBash is both an interactive command language and a scripting language.\nIt is used to interact with the file system, applications, operating system and more."),(0,i.yg)("p",null,"For this assignment you will build a Bash-like shell with minimal functionalities like traversing the file system, running applications, redirecting their output or piping the output from one application into the input of another.\nThe details of the functionalities that must be implemented will be further explained."),(0,i.yg)("h3",{id:"shell-functionalities"},"Shell Functionalities"),(0,i.yg)("h4",{id:"changing-the-current-directory"},"Changing the Current Directory"),(0,i.yg)("p",null,"The shell will support a built-in command for navigating the file system, called ",(0,i.yg)("inlineCode",{parentName:"p"},"cd"),".\nTo implement this feature you will need to store the current directory path because the user can provide either relative or absolute paths as arguments to the ",(0,i.yg)("inlineCode",{parentName:"p"},"cd")," command."),(0,i.yg)("p",null,"The built-in ",(0,i.yg)("inlineCode",{parentName:"p"},"pwd")," command will show the current directory path."),(0,i.yg)("p",null,"Check the following examples below to understand these functionalities."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> pwd\n/home/student\n> cd operating-systems/assignments/minishell\n> pwd\n/home/student/operating-systems/assignments/minishell\n> cd inexitent\nno such file or directory\n> cd /usr/lib\n> pwd\n/usr/lib\n")),(0,i.yg)("blockquote",null,(0,i.yg)("p",{parentName:"blockquote"},(0,i.yg)("strong",{parentName:"p"},(0,i.yg)("em",{parentName:"strong"},"NOTE:"))," Using the ",(0,i.yg)("inlineCode",{parentName:"p"},"cd")," command without any arguments or with more than one argument doesn't affect the current directory path.\nMake sure this edge case is handled in a way that prevents crashes.")),(0,i.yg)("h4",{id:"closing-the-shell"},"Closing the Shell"),(0,i.yg)("p",null,"Inputting either ",(0,i.yg)("inlineCode",{parentName:"p"},"quit")," or ",(0,i.yg)("inlineCode",{parentName:"p"},"exit")," should close the minishell."),(0,i.yg)("h4",{id:"running-an-application"},"Running an Application"),(0,i.yg)("p",null,"Suppose you have an executable named ",(0,i.yg)("inlineCode",{parentName:"p"},"sum")," in the current directory.\nIt takes arbitrarily many numbers as arguments and prints their sum to ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),".\nThe following example shows how the minishell implemented by you should behave."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> ./sum 2 4 1\n7\n")),(0,i.yg)("p",null,"If the executable is located at the ",(0,i.yg)("inlineCode",{parentName:"p"},"/home/student/sum")," absolute path, the following example should also be valid."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> /home/student/sum 2 4 1\n7\n")),(0,i.yg)("p",null,"Each application will run in a separate child process of the minishell created using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/fork.2.html"},"fork"),"."),(0,i.yg)("h4",{id:"environment-variables"},"Environment Variables"),(0,i.yg)("p",null,"Your shell will support using environment variables.\nThe environment variables will be initially inherited from the ",(0,i.yg)("inlineCode",{parentName:"p"},"bash")," process that started your minishell application."),(0,i.yg)("p",null,"If an undefined variable is used, its value is the empty string: ",(0,i.yg)("inlineCode",{parentName:"p"},'""'),"."),(0,i.yg)("blockquote",null,(0,i.yg)("p",{parentName:"blockquote"},(0,i.yg)("strong",{parentName:"p"},(0,i.yg)("em",{parentName:"strong"},"NOTE:"))," The following examples contain comments which don't need to be supported by the minishell.\nThey are present here only to give a better understanding of the minishell's functionalities.")),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> NAME="John Doe" # Will assign the value "John Doe" to the NAME variable\n> AGE=27 # Will assign the value 27 to the AGE variable\n> ./identify $NAME $LOCATION $AGE # Will translate to ./identify "John Doe" "" 27 because $LOCATION is not defined\n')),(0,i.yg)("p",null,"A variable can be assigned to another variable."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> OLD_NAME=$NAME # Will assign the value of the NAME variable to OLD_NAME\n")),(0,i.yg)("h4",{id:"operators"},"Operators"),(0,i.yg)("h5",{id:"sequential-operator"},"Sequential Operator"),(0,i.yg)("p",null,"By using the ",(0,i.yg)("inlineCode",{parentName:"p"},";")," operator, you can chain multiple commands that will run sequentially, one after another.\nIn the command ",(0,i.yg)("inlineCode",{parentName:"p"},"expr1; expr2")," it is guaranteed that ",(0,i.yg)("inlineCode",{parentName:"p"},"expr1")," will finish before ",(0,i.yg)("inlineCode",{parentName:"p"},"expr2")," is be evaluated."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> echo "Hello"; echo "world!"; echo "Bye!"\nHello\nworld!\nBye!\n')),(0,i.yg)("h5",{id:"parallel-operator"},"Parallel Operator"),(0,i.yg)("p",null,"By using the ",(0,i.yg)("inlineCode",{parentName:"p"},"&")," operator you can chain multiple commands that will run in parallel.\nWhen running the command ",(0,i.yg)("inlineCode",{parentName:"p"},"expr1 & expr2"),", both expressions are evaluated at the same time (by different processes).\nThe order in which the two commands finish is not guaranteed."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> echo "Hello" & echo "world!" & echo "Bye!" # The words may be printed in any order\nworld!\nBye!\nHello\n')),(0,i.yg)("h5",{id:"pipe-operator"},"Pipe Operator"),(0,i.yg)("p",null,"With the ",(0,i.yg)("inlineCode",{parentName:"p"},"|")," operator you can chain multiple commands so that the standard output of the first command is redirected to the standard input of the second command."),(0,i.yg)("p",null,"Hint: Look into ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},"anonymous pipes")," and file descriptor inheritance while using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/fork.2.html"},"fork"),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> echo "Bye" # command outputs "Bye"\nBye\n> ./reverse_input\nHello # command reads input "Hello"\nolleH # outputs the reversed string "olleH"\n> echo "world" | ./reverse_input # the output generated by the echo command will be used as input for the reverse_input executable\ndlrow\n')),(0,i.yg)("h5",{id:"chain-operators-for-conditional-execution"},"Chain Operators for Conditional Execution"),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"&&")," operator allows chaining commands that are executed sequentially, from left to right.\nThe chain of execution stops at the first command ",(0,i.yg)("strong",{parentName:"p"},"that exits with an error (return code not 0)"),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"\n> echo "H" && echo "e" && echo "l" && ./throw_error && echo "l" && echo "o"\nH\ne\nl\nERROR: I always fail\n')),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"||")," operator allows chaining commands that are executed sequentially, from left to right.\nThe chain of execution stops at the first command ",(0,i.yg)("strong",{parentName:"p"},"that exits successfully (return code is 0)"),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"\n> ./throw_error || ./throw_error || echo "Hello" || echo "world!" || echo "Bye!"\nERROR: I always fail\nERROR: I always fail\nHello\n')),(0,i.yg)("h5",{id:"operator-priority"},"Operator Priority"),(0,i.yg)("p",null,"The priority of the available operators is the following.\nThe lower the number, the ",(0,i.yg)("strong",{parentName:"p"},"higher")," the priority:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},"Pipe operator (",(0,i.yg)("inlineCode",{parentName:"li"},"|"),")"),(0,i.yg)("li",{parentName:"ol"},"Conditional execution operators (",(0,i.yg)("inlineCode",{parentName:"li"},"&&")," or ",(0,i.yg)("inlineCode",{parentName:"li"},"||"),")"),(0,i.yg)("li",{parentName:"ol"},"Parallel operator (",(0,i.yg)("inlineCode",{parentName:"li"},"&"),")"),(0,i.yg)("li",{parentName:"ol"},"Sequential operator (",(0,i.yg)("inlineCode",{parentName:"li"},";"),")")),(0,i.yg)("h4",{id:"io-redirection"},"I/O Redirection"),(0,i.yg)("p",null,"The shell must support the following redirection options:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"< filename")," - redirects ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")," to standard input"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"> filename")," - redirects standard output to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"2> filename")," - redirects standard error to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"&> filename")," - redirects standard output and standard error to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},">> filename")," - redirects standard output to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")," in append mode"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"2>> filename")," - redirects standard error to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")," in append mode")),(0,i.yg)("p",null,"Hint: Look into ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},"open"),", ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},"dup2")," and ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/close.2.html"},"close"),"."),(0,i.yg)("h2",{id:"support-code"},"Support Code"),(0,i.yg)("p",null,"The support code consists of three directories:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"src/")," is the skeleton mini-shell implementation.\nYou will have to implement missing parts marked as ",(0,i.yg)("inlineCode",{parentName:"p"},"TODO")," items.")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"util/")," stores a parser to be used as support code for implementing the assignment.\nFor more information, you can check the ",(0,i.yg)("inlineCode",{parentName:"p"},"util/parser/README.md")," file.\nYou can use this parser or write your own.")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," are tests used to validate (and grade) the assignment."))),(0,i.yg)("h3",{id:"building-mini-shell"},"Building mini-shell"),(0,i.yg)("p",null,"To build mini-shell, run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," in the ",(0,i.yg)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell$ cd src/\n\nstudent@so:~/.../assignment-mini-shell/src$ make\n")),(0,i.yg)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,i.yg)("p",null,"The testing is automated.\nTests are located in the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests$ ls -F\nMakefile grade.sh* run_all.sh* _test/\n")),(0,i.yg)("p",null,"To test and grade your assignment solution, enter the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory and run ",(0,i.yg)("inlineCode",{parentName:"p"},"grade.sh"),".\nNote that this requires linters being available.\nThe easiest is to use a Docker-based setup with everything installed, as shown in the section ",(0,i.yg)("a",{parentName:"p",href:"#running-the-linters"},'"Running the Linters"'),".\nWhen using ",(0,i.yg)("inlineCode",{parentName:"p"},"grade.sh")," you will get grades for correctness (maximum ",(0,i.yg)("inlineCode",{parentName:"p"},"90")," points) and for coding style (maximum ",(0,i.yg)("inlineCode",{parentName:"p"},"10")," points).\nA successful run will provide you an output ending with:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"### GRADE\n\n\nChecker: 90/ 90\nStyle: 10/ 10\nTotal: 100/100\n\n\n### STYLE SUMMARY\n\n\n")),(0,i.yg)("h3",{id:"running-the-checker"},"Running the Checker"),(0,i.yg)("p",null,"To run the checker and everything else required, use the ",(0,i.yg)("inlineCode",{parentName:"p"},"make check")," command in the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests$ make check\nmake[1]: Entering directory '...'\nrm -f *~\n[...]\n16) Testing sleep command...................................failed [ 0/100]\n17) Testing fscanf function.................................failed [ 0/100]\n18) Testing unknown command.................................failed [ 0/100]\n\n Total: 0/100\n")),(0,i.yg)("p",null,"For starters, tests will fail."),(0,i.yg)("p",null,"Each test is worth a number of points.\nThe total number of points is ",(0,i.yg)("inlineCode",{parentName:"p"},"90"),".\nThe maximum grade is obtained by dividing the number of points to ",(0,i.yg)("inlineCode",{parentName:"p"},"10"),", for a maximum grade of ",(0,i.yg)("inlineCode",{parentName:"p"},"9.00"),"."),(0,i.yg)("p",null,"A successful test run will show the output:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests$ make check\nmake[1]: Entering directory '...'\nrm -f *~\n[...]\n01) Testing commands without arguments......................passed [03/100]\n02) Testing commands with arguments.........................passed [02/100]\n03) Testing simple redirect operators.......................passed [05/100]\n04) Testing append redirect operators.......................passed [05/100]\n05) Testing current directory...............................passed [05/100]\n06) Testing conditional operators...........................passed [05/100]\n07) Testing sequential commands.............................passed [03/100]\n08) Testing environment variables...........................passed [05/100]\n09) Testing single pipe.....................................passed [05/100]\n10) Testing multiple pipes..................................passed [10/100]\n11) Testing variables and redirect..........................passed [05/100]\n12) Testing overwritten variables...........................passed [02/100]\n13) Testing all operators...................................passed [02/100]\n14) Testing parallel operator...............................passed [10/100]\n15) Testing big file........................................passed [05/100]\n16) Testing sleep command...................................passed [07/100]\n17) Testing fscanf function.................................passed [07/100]\n18) Testing unknown command.................................passed [04/100]\n\n Total: 90/100\n")),(0,i.yg)("p",null,"The actual tests are located in the ",(0,i.yg)("inlineCode",{parentName:"p"},"inputs/")," directory."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../assignment-mini-shell/tests/$ ls -F _test/inputs\ntest_01.txt test_03.txt test_05.txt test_07.txt test_09.txt test_11.txt test_13.txt test_15.txt test_17.txt\ntest_02.txt test_04.txt test_06.txt test_08.txt test_10.txt test_12.txt test_14.txt test_16.txt test_18.txt\n")),(0,i.yg)("h3",{id:"running-the-linters"},"Running the Linters"),(0,i.yg)("p",null,"To run the linters, use the make lint command in the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests/$ make lint\n[...]\ncd .. && checkpatch.pl -f checker/*.sh tests/*.sh\n[...]\ncd .. && cpplint --recursive src/ tests/ checker/\n[...]\ncd .. && shellcheck checker/*.sh tests/*.sh\n")),(0,i.yg)("p",null,"Note that the linters have to be installed on your system: ",(0,i.yg)("a",{parentName:"p",href:"https://.com/torvalds/linux/blob/master/scripts/checkpatch.pl"},(0,i.yg)("inlineCode",{parentName:"a"},"checkpatch.pl")),", ",(0,i.yg)("a",{parentName:"p",href:"https://github.com/cpplint/cpplint"},(0,i.yg)("inlineCode",{parentName:"a"},"cpplint")),", ",(0,i.yg)("a",{parentName:"p",href:"https://www.shellcheck.net/"},(0,i.yg)("inlineCode",{parentName:"a"},"shellcheck"))," with certain configuration options."),(0,i.yg)("h3",{id:"debugging"},"Debugging"),(0,i.yg)("p",null,"To inspect the differences between the output of the mini-shell and the reference binary set ",(0,i.yg)("inlineCode",{parentName:"p"},"DO_CLEANUP=no")," in ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/run_test.sh"),".\nTo see the results of the tests, you can check ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/outputs/")," directory."),(0,i.yg)("h3",{id:"memory-leaks"},"Memory leaks"),(0,i.yg)("p",null,"To inspect the unreleased resources (memory leaks, file descriptors) set ",(0,i.yg)("inlineCode",{parentName:"p"},"USE_VALGRIND=yes")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"DO_CLEANUP=no")," in ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/run_test.sh"),".\nYou can modify both the path to the Valgrind log file and the command parameters.\nTo see the results of the tests, you can check ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/outputs/")," directory."))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/75/assets/js/463aadd7.e92b8873.js b/75/assets/js/463aadd7.e92b8873.js deleted file mode 100644 index 1167ab49d9..0000000000 --- a/75/assets/js/463aadd7.e92b8873.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2213],{5680:(e,n,t)=>{t.d(n,{xA:()=>d,yg:()=>h});var a=t(6540);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function l(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var s=a.createContext({}),p=function(e){var n=a.useContext(s),t=n;return e&&(t="function"==typeof e?e(n):l(l({},n),e)),t},d=function(e){var n=p(e.components);return a.createElement(s.Provider,{value:n},e.children)},g="mdxType",u={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},m=a.forwardRef((function(e,n){var t=e.components,i=e.mdxType,r=e.originalType,s=e.parentName,d=o(e,["components","mdxType","originalType","parentName"]),g=p(t),m=i,h=g["".concat(s,".").concat(m)]||g[m]||u[m]||r;return t?a.createElement(h,l(l({ref:n},d),{},{components:t})):a.createElement(h,l({ref:n},d))}));function h(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var r=t.length,l=new Array(r);l[0]=m;var o={};for(var s in n)hasOwnProperty.call(n,s)&&(o[s]=n[s]);o.originalType=e,o[g]="string"==typeof e?e:i,l[1]=o;for(var p=2;p{t.r(n),t.d(n,{assets:()=>s,contentTitle:()=>l,default:()=>u,frontMatter:()=>r,metadata:()=>o,toc:()=>p});var a=t(8168),i=(t(6540),t(5680));const r={},l="Minishell",o={unversionedId:"Assignments/Mini Shell/README",id:"Assignments/Mini Shell/README",title:"Minishell",description:"Objectives",source:"@site/docs/Assignments/Mini Shell/README.md",sourceDirName:"Assignments/Mini Shell",slug:"/Assignments/Mini Shell/",permalink:"/operating-systems/75/Assignments/Mini Shell/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Parallel Graph",permalink:"/operating-systems/75/Assignments/Parallel Graph/"},next:{title:"Asynchronous Web Server",permalink:"/operating-systems/75/Assignments/Asynchronous Web Server/"}},s={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Introduction",id:"introduction",level:3},{value:"Shell Functionalities",id:"shell-functionalities",level:3},{value:"Changing the Current Directory",id:"changing-the-current-directory",level:4},{value:"Closing the Shell",id:"closing-the-shell",level:4},{value:"Running an Application",id:"running-an-application",level:4},{value:"Environment Variables",id:"environment-variables",level:4},{value:"Operators",id:"operators",level:4},{value:"Sequential Operator",id:"sequential-operator",level:5},{value:"Parallel Operator",id:"parallel-operator",level:5},{value:"Pipe Operator",id:"pipe-operator",level:5},{value:"Chain Operators for Conditional Execution",id:"chain-operators-for-conditional-execution",level:5},{value:"Operator Priority",id:"operator-priority",level:5},{value:"I/O Redirection",id:"io-redirection",level:4},{value:"Support Code",id:"support-code",level:2},{value:"Building mini-shell",id:"building-mini-shell",level:3},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Running the Checker",id:"running-the-checker",level:3},{value:"Running the Linters",id:"running-the-linters",level:3},{value:"Debugging",id:"debugging",level:3},{value:"Memory leaks",id:"memory-leaks",level:3}],d={toc:p},g="wrapper";function u(e){let{components:n,...t}=e;return(0,i.yg)(g,(0,a.A)({},d,t,{components:n,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"minishell"},"Minishell"),(0,i.yg)("h2",{id:"objectives"},"Objectives"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"Learn how shells create new child processes and connect the I/O to the terminal."),(0,i.yg)("li",{parentName:"ul"},"Gain a better understanding of the ",(0,i.yg)("inlineCode",{parentName:"li"},"fork()")," function wrapper."),(0,i.yg)("li",{parentName:"ul"},"Learn to correctly execute commands written by the user and treat errors.")),(0,i.yg)("h2",{id:"statement"},"Statement"),(0,i.yg)("h3",{id:"introduction"},"Introduction"),(0,i.yg)("p",null,"A shell is a command-line interpreter that provides a text-based user interface for operating systems.\nBash is both an interactive command language and a scripting language.\nIt is used to interact with the file system, applications, operating system and more."),(0,i.yg)("p",null,"For this assignment you will build a Bash-like shell with minimal functionalities like traversing the file system, running applications, redirecting their output or piping the output from one application into the input of another.\nThe details of the functionalities that must be implemented will be further explained."),(0,i.yg)("h3",{id:"shell-functionalities"},"Shell Functionalities"),(0,i.yg)("h4",{id:"changing-the-current-directory"},"Changing the Current Directory"),(0,i.yg)("p",null,"The shell will support a built-in command for navigating the file system, called ",(0,i.yg)("inlineCode",{parentName:"p"},"cd"),".\nTo implement this feature you will need to store the current directory path because the user can provide either relative or absolute paths as arguments to the ",(0,i.yg)("inlineCode",{parentName:"p"},"cd")," command."),(0,i.yg)("p",null,"The built-in ",(0,i.yg)("inlineCode",{parentName:"p"},"pwd")," command will show the current directory path."),(0,i.yg)("p",null,"Check the following examples below to understand these functionalities."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> pwd\n/home/student\n> cd operating-systems/assignments/minishell\n> pwd\n/home/student/operating-systems/assignments/minishell\n> cd inexitent\nno such file or directory\n> cd /usr/lib\n> pwd\n/usr/lib\n")),(0,i.yg)("blockquote",null,(0,i.yg)("p",{parentName:"blockquote"},(0,i.yg)("strong",{parentName:"p"},(0,i.yg)("em",{parentName:"strong"},"NOTE:"))," Using the ",(0,i.yg)("inlineCode",{parentName:"p"},"cd")," command without any arguments or with more than one argument doesn't affect the current directory path.\nMake sure this edge case is handled in a way that prevents crashes.")),(0,i.yg)("h4",{id:"closing-the-shell"},"Closing the Shell"),(0,i.yg)("p",null,"Inputting either ",(0,i.yg)("inlineCode",{parentName:"p"},"quit")," or ",(0,i.yg)("inlineCode",{parentName:"p"},"exit")," should close the minishell."),(0,i.yg)("h4",{id:"running-an-application"},"Running an Application"),(0,i.yg)("p",null,"Suppose you have an executable named ",(0,i.yg)("inlineCode",{parentName:"p"},"sum")," in the current directory.\nIt takes arbitrarily many numbers as arguments and prints their sum to ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),".\nThe following example shows how the minishell implemented by you should behave."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> ./sum 2 4 1\n7\n")),(0,i.yg)("p",null,"If the executable is located at the ",(0,i.yg)("inlineCode",{parentName:"p"},"/home/student/sum")," absolute path, the following example should also be valid."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> /home/student/sum 2 4 1\n7\n")),(0,i.yg)("p",null,"Each application will run in a separate child process of the minishell created using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/fork.2.html"},"fork"),"."),(0,i.yg)("h4",{id:"environment-variables"},"Environment Variables"),(0,i.yg)("p",null,"Your shell will support using environment variables.\nThe environment variables will be initially inherited from the ",(0,i.yg)("inlineCode",{parentName:"p"},"bash")," process that started your minishell application."),(0,i.yg)("p",null,"If an undefined variable is used, its value is the empty string: ",(0,i.yg)("inlineCode",{parentName:"p"},'""'),"."),(0,i.yg)("blockquote",null,(0,i.yg)("p",{parentName:"blockquote"},(0,i.yg)("strong",{parentName:"p"},(0,i.yg)("em",{parentName:"strong"},"NOTE:"))," The following examples contain comments which don't need to be supported by the minishell.\nThey are present here only to give a better understanding of the minishell's functionalities.")),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> NAME="John Doe" # Will assign the value "John Doe" to the NAME variable\n> AGE=27 # Will assign the value 27 to the AGE variable\n> ./identify $NAME $LOCATION $AGE # Will translate to ./identify "John Doe" "" 27 because $LOCATION is not defined\n')),(0,i.yg)("p",null,"A variable can be assigned to another variable."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},"> OLD_NAME=$NAME # Will assign the value of the NAME variable to OLD_NAME\n")),(0,i.yg)("h4",{id:"operators"},"Operators"),(0,i.yg)("h5",{id:"sequential-operator"},"Sequential Operator"),(0,i.yg)("p",null,"By using the ",(0,i.yg)("inlineCode",{parentName:"p"},";")," operator, you can chain multiple commands that will run sequentially, one after another.\nIn the command ",(0,i.yg)("inlineCode",{parentName:"p"},"expr1; expr2")," it is guaranteed that ",(0,i.yg)("inlineCode",{parentName:"p"},"expr1")," will finish before ",(0,i.yg)("inlineCode",{parentName:"p"},"expr2")," is be evaluated."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> echo "Hello"; echo "world!"; echo "Bye!"\nHello\nworld!\nBye!\n')),(0,i.yg)("h5",{id:"parallel-operator"},"Parallel Operator"),(0,i.yg)("p",null,"By using the ",(0,i.yg)("inlineCode",{parentName:"p"},"&")," operator you can chain multiple commands that will run in parallel.\nWhen running the command ",(0,i.yg)("inlineCode",{parentName:"p"},"expr1 & expr2"),", both expressions are evaluated at the same time (by different processes).\nThe order in which the two commands finish is not guaranteed."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> echo "Hello" & echo "world!" & echo "Bye!" # The words may be printed in any order\nworld!\nBye!\nHello\n')),(0,i.yg)("h5",{id:"pipe-operator"},"Pipe Operator"),(0,i.yg)("p",null,"With the ",(0,i.yg)("inlineCode",{parentName:"p"},"|")," operator you can chain multiple commands so that the standard output of the first command is redirected to the standard input of the second command."),(0,i.yg)("p",null,"Hint: Look into ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},"anonymous pipes")," and file descriptor inheritance while using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/fork.2.html"},"fork"),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'> echo "Bye" # command outputs "Bye"\nBye\n> ./reverse_input\nHello # command reads input "Hello"\nolleH # outputs the reversed string "olleH"\n> echo "world" | ./reverse_input # the output generated by the echo command will be used as input for the reverse_input executable\ndlrow\n')),(0,i.yg)("h5",{id:"chain-operators-for-conditional-execution"},"Chain Operators for Conditional Execution"),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"&&")," operator allows chaining commands that are executed sequentially, from left to right.\nThe chain of execution stops at the first command ",(0,i.yg)("strong",{parentName:"p"},"that exits with an error (return code not 0)"),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"\n> echo "H" && echo "e" && echo "l" && ./throw_error && echo "l" && echo "o"\nH\ne\nl\nERROR: I always fail\n')),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"||")," operator allows chaining commands that are executed sequentially, from left to right.\nThe chain of execution stops at the first command ",(0,i.yg)("strong",{parentName:"p"},"that exits successfully (return code is 0)"),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-sh"},'# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"\n> ./throw_error || ./throw_error || echo "Hello" || echo "world!" || echo "Bye!"\nERROR: I always fail\nERROR: I always fail\nHello\n')),(0,i.yg)("h5",{id:"operator-priority"},"Operator Priority"),(0,i.yg)("p",null,"The priority of the available operators is the following.\nThe lower the number, the ",(0,i.yg)("strong",{parentName:"p"},"higher")," the priority:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},"Pipe operator (",(0,i.yg)("inlineCode",{parentName:"li"},"|"),")"),(0,i.yg)("li",{parentName:"ol"},"Conditional execution operators (",(0,i.yg)("inlineCode",{parentName:"li"},"&&")," or ",(0,i.yg)("inlineCode",{parentName:"li"},"||"),")"),(0,i.yg)("li",{parentName:"ol"},"Parallel operator (",(0,i.yg)("inlineCode",{parentName:"li"},"&"),")"),(0,i.yg)("li",{parentName:"ol"},"Sequential operator (",(0,i.yg)("inlineCode",{parentName:"li"},";"),")")),(0,i.yg)("h4",{id:"io-redirection"},"I/O Redirection"),(0,i.yg)("p",null,"The shell must support the following redirection options:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"< filename")," - redirects ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")," to standard input"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"> filename")," - redirects standard output to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"2> filename")," - redirects standard error to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"&> filename")," - redirects standard output and standard error to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},">> filename")," - redirects standard output to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")," in append mode"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"2>> filename")," - redirects standard error to ",(0,i.yg)("inlineCode",{parentName:"li"},"filename")," in append mode")),(0,i.yg)("p",null,"Hint: Look into ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},"open"),", ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},"dup2")," and ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/close.2.html"},"close"),"."),(0,i.yg)("h2",{id:"support-code"},"Support Code"),(0,i.yg)("p",null,"The support code consists of three directories:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"src/")," is the skeleton mini-shell implementation.\nYou will have to implement missing parts marked as ",(0,i.yg)("inlineCode",{parentName:"p"},"TODO")," items.")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"util/")," stores a parser to be used as support code for implementing the assignment.\nFor more information, you can check the ",(0,i.yg)("inlineCode",{parentName:"p"},"util/parser/README.md")," file.\nYou can use this parser or write your own.")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," are tests used to validate (and grade) the assignment."))),(0,i.yg)("h3",{id:"building-mini-shell"},"Building mini-shell"),(0,i.yg)("p",null,"To build mini-shell, run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," in the ",(0,i.yg)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell$ cd src/\n\nstudent@so:~/.../assignment-mini-shell/src$ make\n")),(0,i.yg)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,i.yg)("p",null,"The testing is automated.\nTests are located in the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests$ ls -F\nMakefile grade.sh* run_all.sh* _test/\n")),(0,i.yg)("p",null,"To test and grade your assignment solution, enter the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory and run ",(0,i.yg)("inlineCode",{parentName:"p"},"grade.sh"),".\nNote that this requires linters being available.\nThe easiest is to use a Docker-based setup with everything installed, as shown in the section ",(0,i.yg)("a",{parentName:"p",href:"#running-the-linters"},'"Running the Linters"'),".\nWhen using ",(0,i.yg)("inlineCode",{parentName:"p"},"grade.sh")," you will get grades for correctness (maximum ",(0,i.yg)("inlineCode",{parentName:"p"},"90")," points) and for coding style (maximum ",(0,i.yg)("inlineCode",{parentName:"p"},"10")," points).\nA successful run will provide you an output ending with:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"### GRADE\n\n\nChecker: 90/ 90\nStyle: 10/ 10\nTotal: 100/100\n\n\n### STYLE SUMMARY\n\n\n")),(0,i.yg)("h3",{id:"running-the-checker"},"Running the Checker"),(0,i.yg)("p",null,"To run the checker and everything else required, use the ",(0,i.yg)("inlineCode",{parentName:"p"},"make check")," command in the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests$ make check\nmake[1]: Entering directory '...'\nrm -f *~\n[...]\n16) Testing sleep command...................................failed [ 0/100]\n17) Testing fscanf function.................................failed [ 0/100]\n18) Testing unknown command.................................failed [ 0/100]\n\n Total: 0/100\n")),(0,i.yg)("p",null,"For starters, tests will fail."),(0,i.yg)("p",null,"Each test is worth a number of points.\nThe total number of points is ",(0,i.yg)("inlineCode",{parentName:"p"},"90"),".\nThe maximum grade is obtained by dividing the number of points to ",(0,i.yg)("inlineCode",{parentName:"p"},"10"),", for a maximum grade of ",(0,i.yg)("inlineCode",{parentName:"p"},"9.00"),"."),(0,i.yg)("p",null,"A successful test run will show the output:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests$ make check\nmake[1]: Entering directory '...'\nrm -f *~\n[...]\n01) Testing commands without arguments......................passed [03/100]\n02) Testing commands with arguments.........................passed [02/100]\n03) Testing simple redirect operators.......................passed [05/100]\n04) Testing append redirect operators.......................passed [05/100]\n05) Testing current directory...............................passed [05/100]\n06) Testing conditional operators...........................passed [05/100]\n07) Testing sequential commands.............................passed [03/100]\n08) Testing environment variables...........................passed [05/100]\n09) Testing single pipe.....................................passed [05/100]\n10) Testing multiple pipes..................................passed [10/100]\n11) Testing variables and redirect..........................passed [05/100]\n12) Testing overwritten variables...........................passed [02/100]\n13) Testing all operators...................................passed [02/100]\n14) Testing parallel operator...............................passed [10/100]\n15) Testing big file........................................passed [05/100]\n16) Testing sleep command...................................passed [07/100]\n17) Testing fscanf function.................................passed [07/100]\n18) Testing unknown command.................................passed [04/100]\n\n Total: 90/100\n")),(0,i.yg)("p",null,"The actual tests are located in the ",(0,i.yg)("inlineCode",{parentName:"p"},"inputs/")," directory."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../assignment-mini-shell/tests/$ ls -F _test/inputs\ntest_01.txt test_03.txt test_05.txt test_07.txt test_09.txt test_11.txt test_13.txt test_15.txt test_17.txt\ntest_02.txt test_04.txt test_06.txt test_08.txt test_10.txt test_12.txt test_14.txt test_16.txt test_18.txt\n")),(0,i.yg)("h3",{id:"running-the-linters"},"Running the Linters"),(0,i.yg)("p",null,"To run the linters, use the make lint command in the ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignment-mini-shell/tests/$ make lint\n[...]\ncd .. && checkpatch.pl -f checker/*.sh tests/*.sh\n[...]\ncd .. && cpplint --recursive src/ tests/ checker/\n[...]\ncd .. && shellcheck checker/*.sh tests/*.sh\n")),(0,i.yg)("p",null,"Note that the linters have to be installed on your system: ",(0,i.yg)("a",{parentName:"p",href:"https://.com/torvalds/linux/blob/master/scripts/checkpatch.pl"},(0,i.yg)("inlineCode",{parentName:"a"},"checkpatch.pl")),", ",(0,i.yg)("a",{parentName:"p",href:"https://github.com/cpplint/cpplint"},(0,i.yg)("inlineCode",{parentName:"a"},"cpplint")),", ",(0,i.yg)("a",{parentName:"p",href:"https://www.shellcheck.net/"},(0,i.yg)("inlineCode",{parentName:"a"},"shellcheck"))," with certain configuration options."),(0,i.yg)("h3",{id:"debugging"},"Debugging"),(0,i.yg)("p",null,"To inspect the differences between the output of the mini-shell and the reference binary set ",(0,i.yg)("inlineCode",{parentName:"p"},"DO_CLEANUP=no")," in ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/run_test.sh"),".\nTo see the results of the tests, you can check ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/outputs/")," directory."),(0,i.yg)("h3",{id:"memory-leaks"},"Memory leaks"),(0,i.yg)("p",null,"To inspect the unreleased resources (memory leaks, file descriptors) set ",(0,i.yg)("inlineCode",{parentName:"p"},"USE_VALGRIND=yes")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"DO_CLEANUP=no")," in ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/run_test.sh"),".\nYou can modify both the path to the Valgrind log file and the command parameters.\nTo see the results of the tests, you can check ",(0,i.yg)("inlineCode",{parentName:"p"},"tests/_test/outputs/")," directory."))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/75/assets/js/78478873.96384bbb.js b/75/assets/js/78478873.96384bbb.js new file mode 100644 index 0000000000..bb3ed45abd --- /dev/null +++ b/75/assets/js/78478873.96384bbb.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7075],{5680:(e,n,t)=>{t.d(n,{xA:()=>c,yg:()=>h});var a=t(6540);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function o(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var s=a.createContext({}),p=function(e){var n=a.useContext(s),t=n;return e&&(t="function"==typeof e?e(n):o(o({},n),e)),t},c=function(e){var n=p(e.components);return a.createElement(s.Provider,{value:n},e.children)},m="mdxType",g={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},d=a.forwardRef((function(e,n){var t=e.components,i=e.mdxType,r=e.originalType,s=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),m=p(t),d=i,h=m["".concat(s,".").concat(d)]||m[d]||g[d]||r;return t?a.createElement(h,o(o({ref:n},c),{},{components:t})):a.createElement(h,o({ref:n},c))}));function h(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var r=t.length,o=new Array(r);o[0]=d;var l={};for(var s in n)hasOwnProperty.call(n,s)&&(l[s]=n[s]);l.originalType=e,l[m]="string"==typeof e?e:i,o[1]=l;for(var p=2;p{t.r(n),t.d(n,{assets:()=>s,contentTitle:()=>o,default:()=>g,frontMatter:()=>r,metadata:()=>l,toc:()=>p});var a=t(8168),i=(t(6540),t(5680));const r={},o="Lab 11 - IO Optimizations",l={unversionedId:"IO/lab11",id:"IO/lab11",title:"Lab 11 - IO Optimizations",description:"Task: Ordered Client-Server Communication",source:"@site/docs/IO/lab11.md",sourceDirName:"IO",slug:"/IO/lab11",permalink:"/operating-systems/75/IO/lab11",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Lab 10 - Inter-Process Communication",permalink:"/operating-systems/75/IO/lab10"},next:{title:"Lecture",permalink:"/operating-systems/75/Lecture/"}},s={},p=[{value:"Task: Ordered Client-Server Communication",id:"task-ordered-client-server-communication",level:2},{value:"Task: Multiplexed Client Server",id:"task-multiplexed-client-server",level:2},{value:"Task: Async Server",id:"task-async-server",level:2},{value:"I/O Multiplexing",id:"io-multiplexing",level:2},{value:"The epoll API",id:"the-epoll-api",level:3},{value:"Asynchronous I/O",id:"asynchronous-io",level:2},{value:"Zero-Copy",id:"zero-copy",level:2},{value:"sendfile()",id:"sendfile",level:3},{value:"Guide: Benchmarking sendfile()",id:"guide-benchmarking-sendfile",level:2},{value:"Guide: Kernel Caching",id:"guide-kernel-caching",level:2},{value:"Caching in action",id:"caching-in-action",level:3},{value:"Guide: Async",id:"guide-async",level:2}],c={toc:p},m="wrapper";function g(e){let{components:n,...r}=e;return(0,i.yg)(m,(0,a.A)({},c,r,{components:n,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"lab-11---io-optimizations"},"Lab 11 - IO Optimizations"),(0,i.yg)("h2",{id:"task-ordered-client-server-communication"},"Task: Ordered Client-Server Communication"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/ipc/drills/tasks/client-server/")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," directory.\nThis exercise will guide you in creating a basic messaging protocol between a server and a client.\nAlthough in real-world applications a server typically handles multiple connections at once, here we focus on a single connection.\nHandling multiple connections is further explored in ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O multiplexing"),"."),(0,i.yg)("p",null,"Our application protocol is defined as follows:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("strong",{parentName:"li"},"server")," listens for connections on ",(0,i.yg)("strong",{parentName:"li"},"localhost")," and a specified ",(0,i.yg)("strong",{parentName:"li"},"port"),"."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("strong",{parentName:"li"},"client")," connects to ",(0,i.yg)("inlineCode",{parentName:"li"},"localhost:port"),"."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("strong",{parentName:"li"},"client")," sends a message, which the ",(0,i.yg)("strong",{parentName:"li"},"server")," then prints, responds to, and the ",(0,i.yg)("strong",{parentName:"li"},"client")," prints the reply.\nThis sequence repeats in a loop."),(0,i.yg)("li",{parentName:"ul"},"The communication ends when either the ",(0,i.yg)("strong",{parentName:"li"},"client")," or the ",(0,i.yg)("strong",{parentName:"li"},"server")," sends the message ",(0,i.yg)("inlineCode",{parentName:"li"},"exit"),".")),(0,i.yg)("p",null,"Since we are blocking on ",(0,i.yg)("inlineCode",{parentName:"p"},"recv()"),", the message order is fixed - the client ",(0,i.yg)("strong",{parentName:"p"},"must")," initiate communication.\nIn real-world applications, this constraint can be avoided with ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O multiplexing"),"."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"support/client.c")," and complete the TODOs to enable message exchange with the server.\nTest your client by running ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py")," in one terminal and then ",(0,i.yg)("inlineCode",{parentName:"p"},"./client")," in another.\nIf correctly implemented, you should be able to exchange messages as outlined above."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Bonus Question:")," Why is it OK for the client to be implemented in C while the server is implemented in Python?")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"support/server.c")," and complete the TODOs to enable message exchange with the client.\nTest your server by running ",(0,i.yg)("inlineCode",{parentName:"p"},"./server")," in one terminal and then ",(0,i.yg)("inlineCode",{parentName:"p"},"python client.py")," in another.\nIf implemented correctly, you should be able to exchange messages as outlined above."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab10#unix-sockets"},"the sockets API")," and ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab10#client-server-model"},"the client-server model"),"."),(0,i.yg)("h2",{id:"task-multiplexed-client-server"},"Task: Multiplexed Client Server"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/drills/tasks/multiplexed-client-server")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," files."),(0,i.yg)("p",null,"This task builds on the previous implementation of a ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-ordered-client-server-communication"},"client-server ordered communication"),".\nPreviously, the client and server followed a strict, sequential communication pattern: each peer had to send a message and wait for a reply before proceeding."),(0,i.yg)("p",null,"We plan to build a group chat where clients can send messages at any time, and each message is broadcast to all other connected clients.\nTo accomplish this, we\u2019ll implement I/O multiplexing mechanisms that notify us only when data is available on a file descriptor.\nThis non-blocking approach allows for smooth, unhindered communication between clients."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"We\u2019ll use the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"epoll"))," interface to manage multiple file descriptors.\nBegin by opening ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," and completing the TODOs.\nWe will define wrapper functions to ",(0,i.yg)("strong",{parentName:"p"},"add")," and ",(0,i.yg)("strong",{parentName:"p"},"remove")," file descriptors from the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance, making the main code more readable.\n",(0,i.yg)("strong",{parentName:"p"},"Note:")," Ensure that each wrapper returns the result of the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_ctl()")," for error handling.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Complete the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/client.c")," to enable multiplexing of the available file descriptors.\nThe file descriptors are ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," (for receiving user messages) and ",(0,i.yg)("inlineCode",{parentName:"p"},"sockfd")," (for communication with the server).\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_create()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()"),", and the wrappers defined in ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," to handle these descriptors without blocking.\nRemember to close the sockets before exiting."),(0,i.yg)("p",{parentName:"li"},"To test, start ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py")," in one terminal and run your client implementation in two separate terminals.\nIf successful, the clients should be able to communicate through the server.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Complete the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/server.c")," to multiplex I/O with clients.\nYou will need to create an ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance and dynamically ",(0,i.yg)("strong",{parentName:"p"},"add")," and ",(0,i.yg)("strong",{parentName:"p"},"remove")," clients as they connect and disconnect.\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_create()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()"),", and the wrappers defined in ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," to achieve this functionality.\n",(0,i.yg)("strong",{parentName:"p"},"Remember")," to remove the client sockets from the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance before closing them."),(0,i.yg)("p",{parentName:"li"},"To test your implementation, run ",(0,i.yg)("inlineCode",{parentName:"p"},"./server")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"./client")," (or ",(0,i.yg)("inlineCode",{parentName:"p"},"python client.py"),") in two separate terminals.\nIf everything works correctly, the clients should be able to communicate with each other via the server."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," API from ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O Multiplexing"),"."),(0,i.yg)("h2",{id:"task-async-server"},"Task: Async Server"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/drills/tasks/async-server")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," files.\nEnter ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make test-file.txt")," to generate the test file."),(0,i.yg)("p",null,"This task builds on the previous example of a ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-multiplexed-client-server"},"multiplexed client-server"),".\nThe server accepts connections from clients and downloads a file of ",(0,i.yg)("inlineCode",{parentName:"p"},"1 GB")," from each.\nAfter uploading the file, the clients close the connection."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"server.c")," and complete the TODOs in the main function to setup IO multiplexing using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"epoll")),".\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_create()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()"),", and the wrappers defined in ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," to handle descriptors without blocking.\n",(0,i.yg)("strong",{parentName:"p"},"Remember")," to remove the client sockets from the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance before closing them."),(0,i.yg)("p",{parentName:"li"},"To test, run ",(0,i.yg)("inlineCode",{parentName:"p"},"./server")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"./client")," in another terminal.s\nIf successful, the clients should print the upload progress.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"There is a problem with our current implementation.\nTry to start two clients at the same time - the first one will start uploading, and the second one will block at ",(0,i.yg)("inlineCode",{parentName:"p"},"connect()"),".\nThis happens because, even though we are multiplexing file descriptors on the server-side, it is busy with another client.\nTo account for this, complete the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"handle_client()")," to serve each client on a different process."),(0,i.yg)("p",{parentName:"li"},"To test, start ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py")," in one terminal and run your client implementation in two separate terminals.\nIf successful, the clients should be able to upload at the same time."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"Async I/O")," and ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O Multiplexing"),"."),(0,i.yg)("h2",{id:"io-multiplexing"},"I/O Multiplexing"),(0,i.yg)("p",null,"I/O multiplexing is the ability to serve multiple I/O channels (or anything that can be referenced via a file descriptor / handle) simultaneously.\nIf a given application, such a server, has multiple sockets on which it serves connection, it may be the case that operating on one socket blocks the server.\nOne solution is using asynchronous operations, with different backends.\nThe other solution is using I/O multiplexing."),(0,i.yg)("p",null,"The classical functions for I/O multiplexing are ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/select.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"select"))," and ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/poll.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"poll")),".\nDue to several limitations, modern operating systems provide advanced (non-portable) variants to these:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"Windows provides ",(0,i.yg)("a",{parentName:"li",href:"https://learn.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports"},"I/O completion ports (",(0,i.yg)("inlineCode",{parentName:"a"},"IOCP"),")"),"."),(0,i.yg)("li",{parentName:"ul"},"BSD provides ",(0,i.yg)("a",{parentName:"li",href:"https://www.freebsd.org/cgi/man.cgi?kqueue"},(0,i.yg)("inlineCode",{parentName:"a"},"kqueue")),"."),(0,i.yg)("li",{parentName:"ul"},"Linux provides ",(0,i.yg)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"epoll()")),".")),(0,i.yg)("blockquote",null,(0,i.yg)("p",{parentName:"blockquote"},(0,i.yg)("strong",{parentName:"p"},"Note")," that I/O multiplexing is orthogonal to ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"asynchronous I/O"),".\nYou could tie them together if the completion of the asynchronous operation sends a notification that can be handled via a file descriptor / handle.\nThis is the case with Windows asynchronous I/O (called ",(0,i.yg)("a",{parentName:"p",href:"https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o"},"overlapped I/O"),").")),(0,i.yg)("h3",{id:"the-epoll-api"},"The ",(0,i.yg)("inlineCode",{parentName:"h3"},"epoll")," API"),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," API allows user-space programs to efficiently monitor multiple file descriptors and be notified when one of them has data to read.\nIt provides a powerful, event-driven interface concentrated in three primary functions:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"int epoll_create1(int flags)"),": Creates an ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll")," instance.\nThe ",(0,i.yg)("inlineCode",{parentName:"li"},"flags")," argument specifies additional options for the instance.\nThe default value is ",(0,i.yg)("inlineCode",{parentName:"li"},"0"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout)"),": Waits for events on the monitored file descriptors.",(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"epfd"),": The file descriptor returned by ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll_create1()"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"events"),": An array of ",(0,i.yg)("inlineCode",{parentName:"li"},"struct epoll_event")," that will store the events that have occurred.\nIt only contains events that are ready (i.e., received data)."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"maxevents"),": The maximum number of events that can be stored in the ",(0,i.yg)("inlineCode",{parentName:"li"},"events")," array."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"timeout"),": The maximum time (in milliseconds) that ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll_wait()")," will block.\nA value of ",(0,i.yg)("inlineCode",{parentName:"li"},"-1")," means it will block indefinitely."))),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event)"),": Modifies the set of file descriptors monitored by ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll"),".",(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"epfd"),": The file descriptor returned by ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll_create1()"),"."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("inlineCode",{parentName:"li"},"op")," argument specifies the operation to perform, which can be:",(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"EPOLL_CTL_ADD"),": Adds a file descriptor to the monitoring list."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"EPOLL_CTL_MOD"),": Modifies an existing file descriptor\u2019s event list."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"EPOLL_CTL_DEL"),": Removes a file descriptor from the monitoring list."))),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("inlineCode",{parentName:"li"},"fd")," argument is the file descriptor to be added, modified, or removed."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("inlineCode",{parentName:"li"},"event")," argument is a pointer to a ",(0,i.yg)("inlineCode",{parentName:"li"},"struct epoll_event")," that defines the events associated with the file descriptor.")))),(0,i.yg)("p",null,"The ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/epoll_event.3type.html"},(0,i.yg)("inlineCode",{parentName:"a"},"struct epoll_event"))," is the core structure used to interact with ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll"),".\nIt is used to return events to user space after ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()")," is called and to pass parameters to ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_ctl()")," when modifying the set of monitored file descriptors.\nWhile the internal workings of ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," are complex, understanding how to use these functions and structures will cover most use cases."),(0,i.yg)("p",null,"Here is an example demonstrating how to use the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," interface:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'efd = epoll_create1(0)\nif (efd < 0) {...} // handle error\n\n// Add fd to monitored set\nstruct epoll_event ev;\nev.events = EPOLLIN; // monitor fd for reading\nev.data.fd = fd;\nrc = epoll_ctl(efd, EPOLL_CTL_ADD, fd, &ev);\nif (rc < 0) {...} // handle error\n\nstruct epoll_event events[10];\nn = epoll_wait(efd, events, 10, -1); // Wait indefinitely\nif (n < 0) {...} // handle error\n\n// Iterate through the events to get active file descriptors\nfor (int i = 0; i < n; i++)\n printf("%d received data\\n", events[i].data.fd);\n')),(0,i.yg)("p",null,"Test your ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," understanding by implementing ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-multiplexed-client-server"},"I/O multiplexing in a client-server app"),"."),(0,i.yg)("h2",{id:"asynchronous-io"},"Asynchronous I/O"),(0,i.yg)("p",null,"Asynchronous I/O (async I/O) provides an efficient way to handle input/output operations that are typically slower than CPU operations by allowing programs to continue executing while waiting for I/O operations to complete.\nHere\u2019s a breakdown of I/O operation types and how asynchronous I/O compares to synchronous operations:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Synchronous Blocking Operation"),":"),(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},"This is the simplest and most common form of I/O (e.g., ",(0,i.yg)("inlineCode",{parentName:"li"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"li"},"write()"),")."),(0,i.yg)("li",{parentName:"ul"},"It\u2019s ",(0,i.yg)("strong",{parentName:"li"},"synchronous"),", meaning the program waits for a response to proceed."),(0,i.yg)("li",{parentName:"ul"},"It\u2019s ",(0,i.yg)("strong",{parentName:"li"},"blocking"),", so if the requested data isn\u2019t available (e.g., no data in the buffer for ",(0,i.yg)("inlineCode",{parentName:"li"},"read()"),"), the program waits for the operation to finish."))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Synchronous Non-blocking Operation"),":"),(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},"This gives more control, especially in situations where waiting isn\u2019t ideal."),(0,i.yg)("li",{parentName:"ul"},"Opening a file using ",(0,i.yg)("inlineCode",{parentName:"li"},"open()")," alongside ",(0,i.yg)("inlineCode",{parentName:"li"},"O_NONBLOCK")," flag ensures the operation returns immediately instead of blocking."),(0,i.yg)("li",{parentName:"ul"},"If data isn\u2019t available right away, the operation notifies the program, which can try again later, avoiding unnecessary waiting."))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Asynchronous Operation"),":"),(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},"Here, the function call returns immediately, allowing the program to continue ",(0,i.yg)("strong",{parentName:"li"},"without waiting for the result"),"."),(0,i.yg)("li",{parentName:"ul"},"A notification or callback is sent when the operation completes, or the program can check its status periodically.")))),(0,i.yg)("p",null,"Keep in mind the async I/O is not the same thing as I/O multiplexing.\nWhile both techniques improve I/O efficiency, they\u2019re conceptually different:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Asynchronous I/O")," schedules operations concurrently, allowing the program to proceed without blocking."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("a",{parentName:"li",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},(0,i.yg)("strong",{parentName:"a"},"I/O Multiplexing"))," (e.g., ",(0,i.yg)("inlineCode",{parentName:"li"},"select()"),", ",(0,i.yg)("inlineCode",{parentName:"li"},"poll()"),", ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll()"),") monitors multiple channels simultaneously and informs the program when a channel has data ready.\nBlocking could still occur if the program cannot proceed without data from a channel.")),(0,i.yg)("p",null,"Think of them as ",(0,i.yg)("strong",{parentName:"p"},"complementary"),": multiplexing helps monitor multiple channels, while async I/O allows the program to do other things while waiting."),(0,i.yg)("p",null,"There are several ways asynchronous I/O can be implemented in practice:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Multiprocess Backend"),": Each request runs in a ",(0,i.yg)("strong",{parentName:"p"},"separate process"),", isolating tasks and preventing blocking between them."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-c"},"// Handle requests with processes\nvoid handle_client_proc(int client_sockfd) {\n pid_t pid = fork();\n if (pid < 0) // handle error\n\n if (pid == 0) { // Child process: handle client connection\n close(server_sockfd); // Close the server socket in the child\n\n {...} // compute and send answer\n\n close(client_sockfd); // Close client socket when done\n exit(0); // Exit child process\n }\n\n close(client_sockfd); // close client socket in parent\n}\n"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Multithreaded Backend"),": Each request runs in a ",(0,i.yg)("strong",{parentName:"p"},"separate thread"),", allowing concurrent operations within the same process."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-c"},"// Handle requests with threads\nvoid* handler(void* arg) {\n int client_sockfd = *(int*)arg;\n\n {...} // compute and send answer\n close(client_sockfd); // Close client socket when done\n\n return NULL;\n}\n\nvoid handle_client_thread(int sockfd) {\n int *sockfd_p = malloc(sizeof(int)); // use the heap to pass the address\n *sockfd_p = sockfd;\n\n int rc = pthread_create(&thread_id, NULL, handler, client_sock_ptr);\n if (rc < 0) // handle error\n pthread_detach(thread_id); // Let the thread clean up after itself\n}\n"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Event-based Backend"),": An action is scheduled with a ",(0,i.yg)("strong",{parentName:"p"},"callback"),", which is invoked upon completion, using event loops to manage tasks.\nA callback is simply a function pointer, allowing the system to execute the function later or when a specific event is triggered."))),(0,i.yg)("p",null,"Test your understanding by solving the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-async-server"},"Async Server task"),"."),(0,i.yg)("h2",{id:"zero-copy"},"Zero-Copy"),(0,i.yg)("p",null,"Imagine a server that responds with files that it stores locally.\nIts actions would be those highlighted in the image below:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},"Receive a new request and extract the filename"),(0,i.yg)("li",{parentName:"ol"},"Read the filename from the disk into memory"),(0,i.yg)("li",{parentName:"ol"},"Send the file from memory to the client")),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Client-Server Steps",src:t(1975).A})),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/server-copies"},"Quiz: How many copies does the OS make?")),(0,i.yg)("p",null,"As you might have guessed, 2 of these copies are useless.\nSince the app doesn't modify the file, there's no need for it to store the file in its own buffer.\nIt would be more efficient to use ",(0,i.yg)("strong",{parentName:"p"},"a single")," kernel buffer as intermediate storage between the disk and the NIC (Network Interface Card), as shown in the image below."),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Server Copies - Zero-Copy",src:t(6317).A})),(0,i.yg)("p",null,'For an easier comparison with the "classic" ',(0,i.yg)("inlineCode",{parentName:"p"},"read()")," + ",(0,i.yg)("inlineCode",{parentName:"p"},"send()")," model, here's the first version again:"),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Server Copies - Read-Send",src:t(7366).A})),(0,i.yg)("p",null,"It should be obvious that the former approach is more efficient than the latter."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/fewer-than-2-copies"},"Quiz: Almost zero copies")),(0,i.yg)("p",null,"These diagrams capture the essence of ",(0,i.yg)("strong",{parentName:"p"},"zero-copy"),": transferring data directly between kernel buffers, avoiding intermediate user-space buffers.\nThis approach is ideal for serving requests, whether forwarding data between clients or reading from disk.\nIt relies on the OS to retrieve and send data efficiently without extra copying steps."),(0,i.yg)("h3",{id:"sendfile"},(0,i.yg)("inlineCode",{parentName:"h3"},"sendfile()")),(0,i.yg)("p",null,"The syscall with which we can leverage ",(0,i.yg)("strong",{parentName:"p"},"zero-copy")," is called ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/sendfile.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"sendfile()")),".\nHere are some practical examples on how to use it:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'// file to file\nint in_fd = open("input_file.txt", O_RDONLY); // src\nint out_fd = open("output_socket", O_WRONLY); // dst\n\nssize_t bytes_sent = sendfile(out_fd, in_fd, &offset, 4096); // Transfer 4096 bytes\nif (bytes_sent < 0) {...} // handle error\n')),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'// file to network\nint in_fd = open("input_file.txt", O_RDONLY); // src\nint sockfd = socket(AF_INET, SOCK_STREAM, 0); // dst\n\nint rc = connect(sock_fd, &server_addr, sizeof(server_addr));\nif (rc < 0) {...} // handle error\n\nssize_t bytes_sent = sendfile(out_fd, in_fd, &offset, 4096); // Transfer 4096 bytes\nif (bytes_sent < 0) {...} // handle error\n')),(0,i.yg)("p",null,"You can read a slightly more detailed article about zero-copy ",(0,i.yg)("a",{parentName:"p",href:"https://developer.ibm.com/articles/j-zerocopy/"},"here"),"."),(0,i.yg)("h2",{id:"guide-benchmarking-sendfile"},"Guide: Benchmarking ",(0,i.yg)("inlineCode",{parentName:"h2"},"sendfile()")),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/guides/benchmarking-sendfile/support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to prepare the test file."),(0,i.yg)("p",null,(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," transfers data between two file descriptors directly within the kernel, bypassing user-space buffers.\nThis process is known as ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#zero-copy"},"zero-copy"),".\nHaving established that ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," is likely faster than traditional I/O operations, it's time to put this theory to the test!"),(0,i.yg)("p",null,"The code in ",(0,i.yg)("inlineCode",{parentName:"p"},"server.py")," creates two threads that behave nearly identically.\nOne listens on port ",(0,i.yg)("inlineCode",{parentName:"p"},"8081")," and handles connections using ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"send()"),", while the other listens on port ",(0,i.yg)("inlineCode",{parentName:"p"},"8082")," and uses ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()"),".\nStart the server in one terminal with ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py"),".\nIn a second terminal, run ",(0,i.yg)("inlineCode",{parentName:"p"},"benchmark_client.py read-send")," followed by ",(0,i.yg)("inlineCode",{parentName:"p"},"benchmark_client.py sendfile")," to compare the performance."),(0,i.yg)("p",null,"The results below are generic, and your outcomes may vary significantly depending on factors such as disk performance, network interface card (NIC), kernel version, Python version, and system load."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:/.../benchmarking-sendfile/support$ python3 benchmark_client.py read-send\nTime taken: 7.175773588009179 seconds\n\nstudent@os:/.../benchmarking-sendfile/support$ python3 benchmark_client.py sendfile\nTime taken: 3.71454380400246 seconds\n")),(0,i.yg)("p",null,"Using ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," ",(0,i.yg)("strong",{parentName:"p"},"halves the number of copies")," required, reducing it from 4 to 2.\nThis should translate to a roughly halved running time as well, making ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," a clear performance improvement over traditional methods."),(0,i.yg)("p",null,"You can explore another example of ",(0,i.yg)("strong",{parentName:"p"},"zero-copy")," in practice in this ",(0,i.yg)("a",{parentName:"p",href:"Questions/mmap-read-write-benchmark"},"Quiz: Why is ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")," faster than ",(0,i.yg)("inlineCode",{parentName:"a"},"mmap()"),"-based ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")),"."),(0,i.yg)("h2",{id:"guide-kernel-caching"},"Guide: Kernel Caching"),(0,i.yg)("p",null,"I/O is critical to system efficiency, but also often its weakest link.\nTechniques to improve I/O performance include ",(0,i.yg)("a",{parentName:"p",href:"guides/libc-FILE-struct/"},"buffering"),", ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#zero-copy"},"zero-copy"),", and ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"async I/O"),".\nAmong these, buffering is the most common and powerful."),(0,i.yg)("p",null,"Remember ",(0,i.yg)("inlineCode",{parentName:"p"},"buffering/support/benchmark_buffering.sh")," or ",(0,i.yg)("inlineCode",{parentName:"p"},"file-mappings/support/benchmark_cp.sh"),".\nThey both used this line:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-bash"},'sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"\n')),(0,i.yg)("p",null,"The line invalidates caches, forcing the OS to perform I/O operations directly from the disk.\nThis ensures that the scripts benchmark the C code's performance alone, without any speedup from cached data."),(0,i.yg)("p",null,"The kernel goes even further with ",(0,i.yg)("strong",{parentName:"p"},"buffering"),".\nThis time, it\u2019s at a level beyond syscalls like ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", using a strategy known as ",(0,i.yg)("strong",{parentName:"p"},"caching"),".\nWhile buffering helps with handling the ",(0,i.yg)("strong",{parentName:"p"},"next data")," efficiently by reading in advance or delaying writes, caching is about speeding up repeated access to the ",(0,i.yg)("strong",{parentName:"p"},"same data"),".\nJust as your browser caches frequently visited pages or your CPU caches recent addresses, the OS caches files that are accessed often, such as logs or configuration files."),(0,i.yg)("p",null,"When the OS encounters a file access, it stores portions of that file in memory so that subsequent requests can read or modify data from RAM rather than waiting on the slower disk.\nThis makes I/O faster."),(0,i.yg)("h3",{id:"caching-in-action"},"Caching in action"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/guides/kernel-caching/support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to create a large file that we'll use for benchmarking.\nWe have two scripts to benchmark the ",(0,i.yg)("inlineCode",{parentName:"p"},"cp")," command with and without caching:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:/.../kernel-caching/support$ ./benchmark_cp.sh\nmake: 'large-file.txt' is up to date.\nBenchmarking cp on a 1 GB file...\n\nreal 0m1.473s\nuser 0m0.001s\nsys 0m0.985s\n\nstudent@os:/.../kernel-caching/support$ ./benchmark_cp_allow_caching.sh\nmake: 'large-file.txt' is up to date.\nBenchmarking cp on a 1 GB file...\n\nreal 0m0.813s\nuser 0m0.000s\nsys 0m0.837s\n")),(0,i.yg)("p",null,"Each subsequent benchmark actually reads the data from the caches populated or refreshed by the previous one.\nSo running the script multiple times might improve the results."),(0,i.yg)("p",null,"You can use ",(0,i.yg)("inlineCode",{parentName:"p"},"free -h")," to view how much data your kernel is caching.\nLook at the ",(0,i.yg)("inlineCode",{parentName:"p"},"buff/cache")," column.\nOne possible output is shown below.\nIt says the OS is caching 7 GB of data."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ free -h\n total used free shared buff/cache available\nMem: 15Gi 8,1Gi 503Mi 691Mi 7,0Gi 6,5Gi\nSwap: 7,6Gi 234Mi 7,4Gi\n")),(0,i.yg)("h2",{id:"guide-async"},"Guide: Async"),(0,i.yg)("p",null,"Enter the ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/guide/async/")," folder for some implementations of a simple request-reply server in C.\nHere we have the implementation of a server that computes the n-th fibonacci number.\nThe server serves requests in different ways:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"synchronous")," server: ",(0,i.yg)("inlineCode",{parentName:"li"},"server.c")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"multiprocess")," server: ",(0,i.yg)("inlineCode",{parentName:"li"},"mp_server.c")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"multithreaded")," server: ",(0,i.yg)("inlineCode",{parentName:"li"},"mt_server.c"))),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Note:")," There is no asynchronous C variant, because of the unstable API of ",(0,i.yg)("a",{parentName:"p",href:"https://pagure.io/libaio"},(0,i.yg)("inlineCode",{parentName:"a"},"libaio"))," and ",(0,i.yg)("a",{parentName:"p",href:"https://unixism.net/loti/what_is_io_uring.html"},(0,i.yg)("inlineCode",{parentName:"a"},"io_uring")),"."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Benchmark the synchronous server to have a reference point.\nRun ",(0,i.yg)("inlineCode",{parentName:"p"},"./server 2999")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"time ./client_bench.sh 2999")," in another.\n",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh")," we'll run ",(0,i.yg)("inlineCode",{parentName:"p"},"8")," instances of ",(0,i.yg)("inlineCode",{parentName:"p"},"client.py")," that make a request."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/async/support$ time ./client_bench.sh 2000\n[...]\nroot: Connected to localhost:2000\nroot: Sending 30\nfunction(30): 1346269\n\nreal 0m1.075s\nuser 0m0.301s\nsys 0m0.029s\n")),(0,i.yg)("p",{parentName:"li"},"The value you obtain might be different, but you should observe a speed-up when benchmarking the other two solutions.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Benchmark the multiprocess and multithreaded alternatives and see which one got the best speed increase.\nYou can obtain more relevant values by tweaking parameters in ",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh"),".\nFor example, you could increase the ",(0,i.yg)("strong",{parentName:"p"},"number of clients")," or the ",(0,i.yg)("strong",{parentName:"p"},"fibonacci value to compute"),".")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Begin by benchmarking the synchronous server to establish a baseline.\nRun ",(0,i.yg)("inlineCode",{parentName:"p"},"./server 2999")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"time ./client_bench.sh 2999")," in another.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh")," script will initiate ",(0,i.yg)("inlineCode",{parentName:"p"},"8")," instances of ",(0,i.yg)("inlineCode",{parentName:"p"},"client.py"),", each making a request.\nThe output might look like this:"),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/async/support$ time ./client_bench.sh 2000\n[...]\nroot: Connected to localhost:2000\nroot: Sending 30\nfunction(30): 1346269\n\nreal 0m1.075s\nuser 0m0.301s\nsys 0m0.029s\n")),(0,i.yg)("p",{parentName:"li"},"Although the actual value may vary, you should observe a noticeable speed-up when testing the other two solutions.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Next, benchmark the multiprocess and multithreaded alternatives to determine which offers the best performance improvement.\nTo obtain more meaningful results, adjust parameters in ",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh"),", such as increasing the ",(0,i.yg)("strong",{parentName:"p"},"number of clients")," or the ",(0,i.yg)("strong",{parentName:"p"},"Fibonacci value to compute"),"."))),(0,i.yg)("p",null,"If you're having difficulties understanding the support code, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"this reading material"),".\nIf you want to practice this yourself, go through the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-async-server"},"Async Server task"),"."))}g.isMDXComponent=!0},1975:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg"},7366:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg"},6317:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/78478873.ca5213c8.js b/75/assets/js/78478873.ca5213c8.js deleted file mode 100644 index 0cd72fc645..0000000000 --- a/75/assets/js/78478873.ca5213c8.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7075],{5680:(e,n,t)=>{t.d(n,{xA:()=>c,yg:()=>h});var a=t(6540);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function o(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var s=a.createContext({}),p=function(e){var n=a.useContext(s),t=n;return e&&(t="function"==typeof e?e(n):o(o({},n),e)),t},c=function(e){var n=p(e.components);return a.createElement(s.Provider,{value:n},e.children)},m="mdxType",g={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},d=a.forwardRef((function(e,n){var t=e.components,i=e.mdxType,r=e.originalType,s=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),m=p(t),d=i,h=m["".concat(s,".").concat(d)]||m[d]||g[d]||r;return t?a.createElement(h,o(o({ref:n},c),{},{components:t})):a.createElement(h,o({ref:n},c))}));function h(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var r=t.length,o=new Array(r);o[0]=d;var l={};for(var s in n)hasOwnProperty.call(n,s)&&(l[s]=n[s]);l.originalType=e,l[m]="string"==typeof e?e:i,o[1]=l;for(var p=2;p{t.r(n),t.d(n,{assets:()=>s,contentTitle:()=>o,default:()=>g,frontMatter:()=>r,metadata:()=>l,toc:()=>p});var a=t(8168),i=(t(6540),t(5680));const r={},o="Lab 11 - IO Optimizations",l={unversionedId:"IO/lab11",id:"IO/lab11",title:"Lab 11 - IO Optimizations",description:"Task: Ordered Client-Server Communication",source:"@site/docs/IO/lab11.md",sourceDirName:"IO",slug:"/IO/lab11",permalink:"/operating-systems/75/IO/lab11",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Lab 10 - Inter-Process Communication",permalink:"/operating-systems/75/IO/lab10"},next:{title:"Lecture",permalink:"/operating-systems/75/Lecture/"}},s={},p=[{value:"Task: Ordered Client-Server Communication",id:"task-ordered-client-server-communication",level:2},{value:"Task: Multiplexed Client Server",id:"task-multiplexed-client-server",level:2},{value:"Task: Async Server",id:"task-async-server",level:2},{value:"I/O Multiplexing",id:"io-multiplexing",level:2},{value:"The epoll API",id:"the-epoll-api",level:3},{value:"Asynchronous I/O",id:"asynchronous-io",level:2},{value:"Zero-Copy",id:"zero-copy",level:2},{value:"sendfile()",id:"sendfile",level:3},{value:"Guide: Benchmarking sendfile()",id:"guide-benchmarking-sendfile",level:2},{value:"Guide: Kernel Caching",id:"guide-kernel-caching",level:2},{value:"Caching in action",id:"caching-in-action",level:3},{value:"Guide: Async",id:"guide-async",level:2}],c={toc:p},m="wrapper";function g(e){let{components:n,...r}=e;return(0,i.yg)(m,(0,a.A)({},c,r,{components:n,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"lab-11---io-optimizations"},"Lab 11 - IO Optimizations"),(0,i.yg)("h2",{id:"task-ordered-client-server-communication"},"Task: Ordered Client-Server Communication"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/ipc/drills/tasks/client-server/")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," directory.\nThis exercise will guide you in creating a basic messaging protocol between a server and a client.\nAlthough in real-world applications a server typically handles multiple connections at once, here we focus on a single connection.\nHandling multiple connections is further explored in ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O multiplexing"),"."),(0,i.yg)("p",null,"Our application protocol is defined as follows:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("strong",{parentName:"li"},"server")," listens for connections on ",(0,i.yg)("strong",{parentName:"li"},"localhost")," and a specified ",(0,i.yg)("strong",{parentName:"li"},"port"),"."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("strong",{parentName:"li"},"client")," connects to ",(0,i.yg)("inlineCode",{parentName:"li"},"localhost:port"),"."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("strong",{parentName:"li"},"client")," sends a message, which the ",(0,i.yg)("strong",{parentName:"li"},"server")," then prints, responds to, and the ",(0,i.yg)("strong",{parentName:"li"},"client")," prints the reply.\nThis sequence repeats in a loop."),(0,i.yg)("li",{parentName:"ul"},"The communication ends when either the ",(0,i.yg)("strong",{parentName:"li"},"client")," or the ",(0,i.yg)("strong",{parentName:"li"},"server")," sends the message ",(0,i.yg)("inlineCode",{parentName:"li"},"exit"),".")),(0,i.yg)("p",null,"Since we are blocking on ",(0,i.yg)("inlineCode",{parentName:"p"},"recv()"),", the message order is fixed - the client ",(0,i.yg)("strong",{parentName:"p"},"must")," initiate communication.\nIn real-world applications, this constraint can be avoided with ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O multiplexing"),"."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"support/client.c")," and complete the TODOs to enable message exchange with the server.\nTest your client by running ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py")," in one terminal and then ",(0,i.yg)("inlineCode",{parentName:"p"},"./client")," in another.\nIf correctly implemented, you should be able to exchange messages as outlined above."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Bonus Question:")," Why is it OK for the client to be implemented in C while the server is implemented in Python?")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"support/server.c")," and complete the TODOs to enable message exchange with the client.\nTest your server by running ",(0,i.yg)("inlineCode",{parentName:"p"},"./server")," in one terminal and then ",(0,i.yg)("inlineCode",{parentName:"p"},"python client.py")," in another.\nIf implemented correctly, you should be able to exchange messages as outlined above."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab10#unix-sockets"},"the sockets API")," and ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab10#client-server-model"},"the client-server model"),"."),(0,i.yg)("h2",{id:"task-multiplexed-client-server"},"Task: Multiplexed Client Server"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/drills/tasks/client-server-epoll")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," files."),(0,i.yg)("p",null,"This task builds on the previous implementation of a ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-ordered-client-server-communication"},"client-server ordered communication"),".\nPreviously, the client and server followed a strict, sequential communication pattern: each peer had to send a message and wait for a reply before proceeding."),(0,i.yg)("p",null,"We plan to build a group chat where clients can send messages at any time, and each message is broadcast to all other connected clients.\nTo accomplish this, we\u2019ll implement I/O multiplexing mechanisms that notify us only when data is available on a file descriptor.\nThis non-blocking approach allows for smooth, unhindered communication between clients."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"We\u2019ll use the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"epoll"))," interface to manage multiple file descriptors.\nBegin by opening ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," and completing the TODOs.\nWe will define wrapper functions to ",(0,i.yg)("strong",{parentName:"p"},"add")," and ",(0,i.yg)("strong",{parentName:"p"},"remove")," file descriptors from the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance, making the main code more readable.\n",(0,i.yg)("strong",{parentName:"p"},"Note:")," Ensure that each wrapper returns the result of the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_ctl()")," for error handling.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Complete the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/client.c")," to enable multiplexing of the available file descriptors.\nThe file descriptors are ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," (for receiving user messages) and ",(0,i.yg)("inlineCode",{parentName:"p"},"sockfd")," (for communication with the server).\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_create()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()"),", and the wrappers defined in ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," to handle these descriptors without blocking.\nRemember to close the sockets before exiting."),(0,i.yg)("p",{parentName:"li"},"To test, start ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py")," in one terminal and run your client implementation in two separate terminals.\nIf successful, the clients should be able to communicate through the server.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Complete the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/server.c")," to multiplex I/O with clients.\nYou will need to create an ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance and dynamically ",(0,i.yg)("strong",{parentName:"p"},"add")," and ",(0,i.yg)("strong",{parentName:"p"},"remove")," clients as they connect and disconnect.\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_create()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()"),", and the wrappers defined in ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," to achieve this functionality.\n",(0,i.yg)("strong",{parentName:"p"},"Remember")," to remove the client sockets from the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance before closing them."),(0,i.yg)("p",{parentName:"li"},"To test your implementation, run ",(0,i.yg)("inlineCode",{parentName:"p"},"./server")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"./client")," (or ",(0,i.yg)("inlineCode",{parentName:"p"},"python client.py"),") in two separate terminals.\nIf everything works correctly, the clients should be able to communicate with each other via the server."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," API from ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O Multiplexing"),"."),(0,i.yg)("h2",{id:"task-async-server"},"Task: Async Server"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/drills/tasks/async-server")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," files.\nEnter ",(0,i.yg)("inlineCode",{parentName:"p"},"support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make test-file.txt")," to generate the test file."),(0,i.yg)("p",null,"This task builds on the previous example of a ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-multiplexed-client-server"},"multiplexed client-server"),".\nThe server accepts connections from clients and downloads a file of ",(0,i.yg)("inlineCode",{parentName:"p"},"1 GB")," from each.\nAfter uploading the file, the clients close the connection."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"server.c")," and complete the TODOs in the main function to setup IO multiplexing using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"epoll")),".\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_create()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()"),", and the wrappers defined in ",(0,i.yg)("inlineCode",{parentName:"p"},"w_epoll.h")," to handle descriptors without blocking.\n",(0,i.yg)("strong",{parentName:"p"},"Remember")," to remove the client sockets from the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," instance before closing them."),(0,i.yg)("p",{parentName:"li"},"To test, run ",(0,i.yg)("inlineCode",{parentName:"p"},"./server")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"./client")," in another terminal.s\nIf successful, the clients should print the upload progress.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"There is a problem with our current implementation.\nTry to start two clients at the same time - the first one will start uploading, and the second one will block at ",(0,i.yg)("inlineCode",{parentName:"p"},"connect()"),".\nThis happens because, even though we are multiplexing file descriptors on the server-side, it is busy with another client.\nTo account for this, complete the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"handle_client()")," to serve each client on a different process."),(0,i.yg)("p",{parentName:"li"},"To test, start ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py")," in one terminal and run your client implementation in two separate terminals.\nIf successful, the clients should be able to upload at the same time."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"Async I/O")," and ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},"I/O Multiplexing"),"."),(0,i.yg)("h2",{id:"io-multiplexing"},"I/O Multiplexing"),(0,i.yg)("p",null,"I/O multiplexing is the ability to serve multiple I/O channels (or anything that can be referenced via a file descriptor / handle) simultaneously.\nIf a given application, such a server, has multiple sockets on which it serves connection, it may be the case that operating on one socket blocks the server.\nOne solution is using asynchronous operations, with different backends.\nThe other solution is using I/O multiplexing."),(0,i.yg)("p",null,"The classical functions for I/O multiplexing are ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/select.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"select"))," and ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/poll.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"poll")),".\nDue to several limitations, modern operating systems provide advanced (non-portable) variants to these:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"Windows provides ",(0,i.yg)("a",{parentName:"li",href:"https://learn.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports"},"I/O completion ports (",(0,i.yg)("inlineCode",{parentName:"a"},"IOCP"),")"),"."),(0,i.yg)("li",{parentName:"ul"},"BSD provides ",(0,i.yg)("a",{parentName:"li",href:"https://www.freebsd.org/cgi/man.cgi?kqueue"},(0,i.yg)("inlineCode",{parentName:"a"},"kqueue")),"."),(0,i.yg)("li",{parentName:"ul"},"Linux provides ",(0,i.yg)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"epoll()")),".")),(0,i.yg)("blockquote",null,(0,i.yg)("p",{parentName:"blockquote"},(0,i.yg)("strong",{parentName:"p"},"Note")," that I/O multiplexing is orthogonal to ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"asynchronous I/O"),".\nYou could tie them together if the completion of the asynchronous operation sends a notification that can be handled via a file descriptor / handle.\nThis is the case with Windows asynchronous I/O (called ",(0,i.yg)("a",{parentName:"p",href:"https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o"},"overlapped I/O"),").")),(0,i.yg)("h3",{id:"the-epoll-api"},"The ",(0,i.yg)("inlineCode",{parentName:"h3"},"epoll")," API"),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," API allows user-space programs to efficiently monitor multiple file descriptors and be notified when one of them has data to read.\nIt provides a powerful, event-driven interface concentrated in three primary functions:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"int epoll_create1(int flags)"),": Creates an ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll")," instance.\nThe ",(0,i.yg)("inlineCode",{parentName:"li"},"flags")," argument specifies additional options for the instance.\nThe default value is ",(0,i.yg)("inlineCode",{parentName:"li"},"0"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"int epoll_wait(int epfd, struct epoll_event *events, int maxevents, int timeout)"),": Waits for events on the monitored file descriptors.",(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"epfd"),": The file descriptor returned by ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll_create1()"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"events"),": An array of ",(0,i.yg)("inlineCode",{parentName:"li"},"struct epoll_event")," that will store the events that have occurred.\nIt only contains events that are ready (i.e., received data)."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"maxevents"),": The maximum number of events that can be stored in the ",(0,i.yg)("inlineCode",{parentName:"li"},"events")," array."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"timeout"),": The maximum time (in milliseconds) that ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll_wait()")," will block.\nA value of ",(0,i.yg)("inlineCode",{parentName:"li"},"-1")," means it will block indefinitely."))),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event)"),": Modifies the set of file descriptors monitored by ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll"),".",(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"epfd"),": The file descriptor returned by ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll_create1()"),"."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("inlineCode",{parentName:"li"},"op")," argument specifies the operation to perform, which can be:",(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"EPOLL_CTL_ADD"),": Adds a file descriptor to the monitoring list."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"EPOLL_CTL_MOD"),": Modifies an existing file descriptor\u2019s event list."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"EPOLL_CTL_DEL"),": Removes a file descriptor from the monitoring list."))),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("inlineCode",{parentName:"li"},"fd")," argument is the file descriptor to be added, modified, or removed."),(0,i.yg)("li",{parentName:"ul"},"The ",(0,i.yg)("inlineCode",{parentName:"li"},"event")," argument is a pointer to a ",(0,i.yg)("inlineCode",{parentName:"li"},"struct epoll_event")," that defines the events associated with the file descriptor.")))),(0,i.yg)("p",null,"The ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/epoll_event.3type.html"},(0,i.yg)("inlineCode",{parentName:"a"},"struct epoll_event"))," is the core structure used to interact with ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll"),".\nIt is used to return events to user space after ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_wait()")," is called and to pass parameters to ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll_ctl()")," when modifying the set of monitored file descriptors.\nWhile the internal workings of ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," are complex, understanding how to use these functions and structures will cover most use cases."),(0,i.yg)("p",null,"Here is an example demonstrating how to use the ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," interface:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'efd = epoll_create1(0)\nif (efd < 0) {...} // handle error\n\n// Add fd to monitored set\nstruct epoll_event ev;\nev.events = EPOLLIN; // monitor fd for reading\nev.data.fd = fd;\nrc = epoll_ctl(efd, EPOLL_CTL_ADD, fd, &ev);\nif (rc < 0) {...} // handle error\n\nstruct epoll_event events[10];\nn = epoll_wait(efd, events, 10, -1); // Wait indefinitely\nif (n < 0) {...} // handle error\n\n// Iterate through the events to get active file descriptors\nfor (int i = 0; i < n; i++)\n printf("%d received data\\n", events[i].data.fd);\n')),(0,i.yg)("p",null,"Test your ",(0,i.yg)("inlineCode",{parentName:"p"},"epoll")," understanding by implementing ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-multiplexed-client-server"},"I/O multiplexing in a client-server app"),"."),(0,i.yg)("h2",{id:"asynchronous-io"},"Asynchronous I/O"),(0,i.yg)("p",null,"Asynchronous I/O (async I/O) provides an efficient way to handle input/output operations that are typically slower than CPU operations by allowing programs to continue executing while waiting for I/O operations to complete.\nHere\u2019s a breakdown of I/O operation types and how asynchronous I/O compares to synchronous operations:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Synchronous Blocking Operation"),":"),(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},"This is the simplest and most common form of I/O (e.g., ",(0,i.yg)("inlineCode",{parentName:"li"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"li"},"write()"),")."),(0,i.yg)("li",{parentName:"ul"},"It\u2019s ",(0,i.yg)("strong",{parentName:"li"},"synchronous"),", meaning the program waits for a response to proceed."),(0,i.yg)("li",{parentName:"ul"},"It\u2019s ",(0,i.yg)("strong",{parentName:"li"},"blocking"),", so if the requested data isn\u2019t available (e.g., no data in the buffer for ",(0,i.yg)("inlineCode",{parentName:"li"},"read()"),"), the program waits for the operation to finish."))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Synchronous Non-blocking Operation"),":"),(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},"This gives more control, especially in situations where waiting isn\u2019t ideal."),(0,i.yg)("li",{parentName:"ul"},"Opening a file using ",(0,i.yg)("inlineCode",{parentName:"li"},"open()")," alongside ",(0,i.yg)("inlineCode",{parentName:"li"},"O_NONBLOCK")," flag ensures the operation returns immediately instead of blocking."),(0,i.yg)("li",{parentName:"ul"},"If data isn\u2019t available right away, the operation notifies the program, which can try again later, avoiding unnecessary waiting."))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Asynchronous Operation"),":"),(0,i.yg)("ul",{parentName:"li"},(0,i.yg)("li",{parentName:"ul"},"Here, the function call returns immediately, allowing the program to continue ",(0,i.yg)("strong",{parentName:"li"},"without waiting for the result"),"."),(0,i.yg)("li",{parentName:"ul"},"A notification or callback is sent when the operation completes, or the program can check its status periodically.")))),(0,i.yg)("p",null,"Keep in mind the async I/O is not the same thing as I/O multiplexing.\nWhile both techniques improve I/O efficiency, they\u2019re conceptually different:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Asynchronous I/O")," schedules operations concurrently, allowing the program to proceed without blocking."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("a",{parentName:"li",href:"/operating-systems/75/IO/lab11#i/o-multiplexing"},(0,i.yg)("strong",{parentName:"a"},"I/O Multiplexing"))," (e.g., ",(0,i.yg)("inlineCode",{parentName:"li"},"select()"),", ",(0,i.yg)("inlineCode",{parentName:"li"},"poll()"),", ",(0,i.yg)("inlineCode",{parentName:"li"},"epoll()"),") monitors multiple channels simultaneously and informs the program when a channel has data ready.\nBlocking could still occur if the program cannot proceed without data from a channel.")),(0,i.yg)("p",null,"Think of them as ",(0,i.yg)("strong",{parentName:"p"},"complementary"),": multiplexing helps monitor multiple channels, while async I/O allows the program to do other things while waiting."),(0,i.yg)("p",null,"There are several ways asynchronous I/O can be implemented in practice:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Multiprocess Backend"),": Each request runs in a ",(0,i.yg)("strong",{parentName:"p"},"separate process"),", isolating tasks and preventing blocking between them."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-c"},"// Handle requests with processes\nvoid handle_client_proc(int client_sockfd) {\n pid_t pid = fork();\n if (pid < 0) // handle error\n\n if (pid == 0) { // Child process: handle client connection\n close(server_sockfd); // Close the server socket in the child\n\n {...} // compute and send answer\n\n close(client_sockfd); // Close client socket when done\n exit(0); // Exit child process\n }\n\n close(client_sockfd); // close client socket in parent\n}\n"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Multithreaded Backend"),": Each request runs in a ",(0,i.yg)("strong",{parentName:"p"},"separate thread"),", allowing concurrent operations within the same process."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-c"},"// Handle requests with threads\nvoid* handler(void* arg) {\n int client_sockfd = *(int*)arg;\n\n {...} // compute and send answer\n close(client_sockfd); // Close client socket when done\n\n return NULL;\n}\n\nvoid handle_client_thread(int sockfd) {\n int *sockfd_p = malloc(sizeof(int)); // use the heap to pass the address\n *sockfd_p = sockfd;\n\n int rc = pthread_create(&thread_id, NULL, handler, client_sock_ptr);\n if (rc < 0) // handle error\n pthread_detach(thread_id); // Let the thread clean up after itself\n}\n"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Event-based Backend"),": An action is scheduled with a ",(0,i.yg)("strong",{parentName:"p"},"callback"),", which is invoked upon completion, using event loops to manage tasks.\nA callback is simply a function pointer, allowing the system to execute the function later or when a specific event is triggered."))),(0,i.yg)("p",null,"Test your understanding by solving the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-async-server"},"Async Server task"),"."),(0,i.yg)("h2",{id:"zero-copy"},"Zero-Copy"),(0,i.yg)("p",null,"Imagine a server that responds with files that it stores locally.\nIts actions would be those highlighted in the image below:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},"Receive a new request and extract the filename"),(0,i.yg)("li",{parentName:"ol"},"Read the filename from the disk into memory"),(0,i.yg)("li",{parentName:"ol"},"Send the file from memory to the client")),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Client-Server Steps",src:t(1975).A})),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/server-copies"},"Quiz: How many copies does the OS make?")),(0,i.yg)("p",null,"As you might have guessed, 2 of these copies are useless.\nSince the app doesn't modify the file, there's no need for it to store the file in its own buffer.\nIt would be more efficient to use ",(0,i.yg)("strong",{parentName:"p"},"a single")," kernel buffer as intermediate storage between the disk and the NIC (Network Interface Card), as shown in the image below."),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Server Copies - Zero-Copy",src:t(6317).A})),(0,i.yg)("p",null,'For an easier comparison with the "classic" ',(0,i.yg)("inlineCode",{parentName:"p"},"read()")," + ",(0,i.yg)("inlineCode",{parentName:"p"},"send()")," model, here's the first version again:"),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Server Copies - Read-Send",src:t(7366).A})),(0,i.yg)("p",null,"It should be obvious that the former approach is more efficient than the latter."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/fewer-than-2-copies"},"Quiz: Almost zero copies")),(0,i.yg)("p",null,"These diagrams capture the essence of ",(0,i.yg)("strong",{parentName:"p"},"zero-copy"),": transferring data directly between kernel buffers, avoiding intermediate user-space buffers.\nThis approach is ideal for serving requests, whether forwarding data between clients or reading from disk.\nIt relies on the OS to retrieve and send data efficiently without extra copying steps."),(0,i.yg)("h3",{id:"sendfile"},(0,i.yg)("inlineCode",{parentName:"h3"},"sendfile()")),(0,i.yg)("p",null,"The syscall with which we can leverage ",(0,i.yg)("strong",{parentName:"p"},"zero-copy")," is called ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/sendfile.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"sendfile()")),".\nHere are some practical examples on how to use it:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'// file to file\nint in_fd = open("input_file.txt", O_RDONLY); // src\nint out_fd = open("output_socket", O_WRONLY); // dst\n\nssize_t bytes_sent = sendfile(out_fd, in_fd, &offset, 4096); // Transfer 4096 bytes\nif (bytes_sent < 0) {...} // handle error\n')),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'// file to network\nint in_fd = open("input_file.txt", O_RDONLY); // src\nint sockfd = socket(AF_INET, SOCK_STREAM, 0); // dst\n\nint rc = connect(sock_fd, &server_addr, sizeof(server_addr));\nif (rc < 0) {...} // handle error\n\nssize_t bytes_sent = sendfile(out_fd, in_fd, &offset, 4096); // Transfer 4096 bytes\nif (bytes_sent < 0) {...} // handle error\n')),(0,i.yg)("p",null,"You can read a slightly more detailed article about zero-copy ",(0,i.yg)("a",{parentName:"p",href:"https://developer.ibm.com/articles/j-zerocopy/"},"here"),"."),(0,i.yg)("h2",{id:"guide-benchmarking-sendfile"},"Guide: Benchmarking ",(0,i.yg)("inlineCode",{parentName:"h2"},"sendfile()")),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/guides/benchmarking-sendfile/support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to prepare the test file."),(0,i.yg)("p",null,(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," transfers data between two file descriptors directly within the kernel, bypassing user-space buffers.\nThis process is known as ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#zero-copy"},"zero-copy"),".\nHaving established that ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," is likely faster than traditional I/O operations, it's time to put this theory to the test!"),(0,i.yg)("p",null,"The code in ",(0,i.yg)("inlineCode",{parentName:"p"},"server.py")," creates two threads that behave nearly identically.\nOne listens on port ",(0,i.yg)("inlineCode",{parentName:"p"},"8081")," and handles connections using ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"send()"),", while the other listens on port ",(0,i.yg)("inlineCode",{parentName:"p"},"8082")," and uses ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()"),".\nStart the server in one terminal with ",(0,i.yg)("inlineCode",{parentName:"p"},"python server.py"),".\nIn a second terminal, run ",(0,i.yg)("inlineCode",{parentName:"p"},"benchmark_client.py read-send")," followed by ",(0,i.yg)("inlineCode",{parentName:"p"},"benchmark_client.py sendfile")," to compare the performance."),(0,i.yg)("p",null,"The results below are generic, and your outcomes may vary significantly depending on factors such as disk performance, network interface card (NIC), kernel version, Python version, and system load."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:/.../benchmarking-sendfile/support$ python3 benchmark_client.py read-send\nTime taken: 7.175773588009179 seconds\n\nstudent@os:/.../benchmarking-sendfile/support$ python3 benchmark_client.py sendfile\nTime taken: 3.71454380400246 seconds\n")),(0,i.yg)("p",null,"Using ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," ",(0,i.yg)("strong",{parentName:"p"},"halves the number of copies")," required, reducing it from 4 to 2.\nThis should translate to a roughly halved running time as well, making ",(0,i.yg)("inlineCode",{parentName:"p"},"sendfile()")," a clear performance improvement over traditional methods."),(0,i.yg)("p",null,"You can explore another example of ",(0,i.yg)("strong",{parentName:"p"},"zero-copy")," in practice in this ",(0,i.yg)("a",{parentName:"p",href:"Questions/mmap-read-write-benchmark"},"Quiz: Why is ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")," faster than ",(0,i.yg)("inlineCode",{parentName:"a"},"mmap()"),"-based ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")),"."),(0,i.yg)("h2",{id:"guide-kernel-caching"},"Guide: Kernel Caching"),(0,i.yg)("p",null,"I/O is critical to system efficiency, but also often its weakest link.\nTechniques to improve I/O performance include ",(0,i.yg)("a",{parentName:"p",href:"guides/libc-FILE-struct/"},"buffering"),", ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#zero-copy"},"zero-copy"),", and ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"async I/O"),".\nAmong these, buffering is the most common and powerful."),(0,i.yg)("p",null,"Remember ",(0,i.yg)("inlineCode",{parentName:"p"},"buffering/support/benchmark_buffering.sh")," or ",(0,i.yg)("inlineCode",{parentName:"p"},"file-mappings/support/benchmark_cp.sh"),".\nThey both used this line:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-bash"},'sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"\n')),(0,i.yg)("p",null,"The line invalidates caches, forcing the OS to perform I/O operations directly from the disk.\nThis ensures that the scripts benchmark the C code's performance alone, without any speedup from cached data."),(0,i.yg)("p",null,"The kernel goes even further with ",(0,i.yg)("strong",{parentName:"p"},"buffering"),".\nThis time, it\u2019s at a level beyond syscalls like ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", using a strategy known as ",(0,i.yg)("strong",{parentName:"p"},"caching"),".\nWhile buffering helps with handling the ",(0,i.yg)("strong",{parentName:"p"},"next data")," efficiently by reading in advance or delaying writes, caching is about speeding up repeated access to the ",(0,i.yg)("strong",{parentName:"p"},"same data"),".\nJust as your browser caches frequently visited pages or your CPU caches recent addresses, the OS caches files that are accessed often, such as logs or configuration files."),(0,i.yg)("p",null,"When the OS encounters a file access, it stores portions of that file in memory so that subsequent requests can read or modify data from RAM rather than waiting on the slower disk.\nThis makes I/O faster."),(0,i.yg)("h3",{id:"caching-in-action"},"Caching in action"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/guides/kernel-caching/support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to create a large file that we'll use for benchmarking.\nWe have two scripts to benchmark the ",(0,i.yg)("inlineCode",{parentName:"p"},"cp")," command with and without caching:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:/.../kernel-caching/support$ ./benchmark_cp.sh\nmake: 'large-file.txt' is up to date.\nBenchmarking cp on a 1 GB file...\n\nreal 0m1.473s\nuser 0m0.001s\nsys 0m0.985s\n\nstudent@os:/.../kernel-caching/support$ ./benchmark_cp_allow_caching.sh\nmake: 'large-file.txt' is up to date.\nBenchmarking cp on a 1 GB file...\n\nreal 0m0.813s\nuser 0m0.000s\nsys 0m0.837s\n")),(0,i.yg)("p",null,"Each subsequent benchmark actually reads the data from the caches populated or refreshed by the previous one.\nSo running the script multiple times might improve the results."),(0,i.yg)("p",null,"You can use ",(0,i.yg)("inlineCode",{parentName:"p"},"free -h")," to view how much data your kernel is caching.\nLook at the ",(0,i.yg)("inlineCode",{parentName:"p"},"buff/cache")," column.\nOne possible output is shown below.\nIt says the OS is caching 7 GB of data."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ free -h\n total used free shared buff/cache available\nMem: 15Gi 8,1Gi 503Mi 691Mi 7,0Gi 6,5Gi\nSwap: 7,6Gi 234Mi 7,4Gi\n")),(0,i.yg)("h2",{id:"guide-async"},"Guide: Async"),(0,i.yg)("p",null,"Enter the ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/optimizations/guide/async/")," folder for some implementations of a simple request-reply server in C.\nHere we have the implementation of a server that computes the n-th fibonacci number.\nThe server serves requests in different ways:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"synchronous")," server: ",(0,i.yg)("inlineCode",{parentName:"li"},"server.c")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"multiprocess")," server: ",(0,i.yg)("inlineCode",{parentName:"li"},"mp_server.c")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"multithreaded")," server: ",(0,i.yg)("inlineCode",{parentName:"li"},"mt_server.c"))),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Note:")," There is no asynchronous C variant, because of the unstable API of ",(0,i.yg)("a",{parentName:"p",href:"https://pagure.io/libaio"},(0,i.yg)("inlineCode",{parentName:"a"},"libaio"))," and ",(0,i.yg)("a",{parentName:"p",href:"https://unixism.net/loti/what_is_io_uring.html"},(0,i.yg)("inlineCode",{parentName:"a"},"io_uring")),"."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Benchmark the synchronous server to have a reference point.\nRun ",(0,i.yg)("inlineCode",{parentName:"p"},"./server 2999")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"time ./client_bench.sh 2999")," in another.\n",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh")," we'll run ",(0,i.yg)("inlineCode",{parentName:"p"},"8")," instances of ",(0,i.yg)("inlineCode",{parentName:"p"},"client.py")," that make a request."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/async/support$ time ./client_bench.sh 2000\n[...]\nroot: Connected to localhost:2000\nroot: Sending 30\nfunction(30): 1346269\n\nreal 0m1.075s\nuser 0m0.301s\nsys 0m0.029s\n")),(0,i.yg)("p",{parentName:"li"},"The value you obtain might be different, but you should observe a speed-up when benchmarking the other two solutions.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Benchmark the multiprocess and multithreaded alternatives and see which one got the best speed increase.\nYou can obtain more relevant values by tweaking parameters in ",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh"),".\nFor example, you could increase the ",(0,i.yg)("strong",{parentName:"p"},"number of clients")," or the ",(0,i.yg)("strong",{parentName:"p"},"fibonacci value to compute"),".")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Begin by benchmarking the synchronous server to establish a baseline.\nRun ",(0,i.yg)("inlineCode",{parentName:"p"},"./server 2999")," in one terminal and ",(0,i.yg)("inlineCode",{parentName:"p"},"time ./client_bench.sh 2999")," in another.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh")," script will initiate ",(0,i.yg)("inlineCode",{parentName:"p"},"8")," instances of ",(0,i.yg)("inlineCode",{parentName:"p"},"client.py"),", each making a request.\nThe output might look like this:"),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/async/support$ time ./client_bench.sh 2000\n[...]\nroot: Connected to localhost:2000\nroot: Sending 30\nfunction(30): 1346269\n\nreal 0m1.075s\nuser 0m0.301s\nsys 0m0.029s\n")),(0,i.yg)("p",{parentName:"li"},"Although the actual value may vary, you should observe a noticeable speed-up when testing the other two solutions.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Next, benchmark the multiprocess and multithreaded alternatives to determine which offers the best performance improvement.\nTo obtain more meaningful results, adjust parameters in ",(0,i.yg)("inlineCode",{parentName:"p"},"client_bench.sh"),", such as increasing the ",(0,i.yg)("strong",{parentName:"p"},"number of clients")," or the ",(0,i.yg)("strong",{parentName:"p"},"Fibonacci value to compute"),"."))),(0,i.yg)("p",null,"If you're having difficulties understanding the support code, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#asynchronous-i/o"},"this reading material"),".\nIf you want to practice this yourself, go through the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab11#task-async-server"},"Async Server task"),"."))}g.isMDXComponent=!0},1975:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg"},7366:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg"},6317:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/8849ec36.8fd2dca6.js b/75/assets/js/8849ec36.8fd2dca6.js new file mode 100644 index 0000000000..bc97a4dcc7 --- /dev/null +++ b/75/assets/js/8849ec36.8fd2dca6.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9577],{5680:(e,a,t)=>{t.d(a,{xA:()=>h,yg:()=>u});var n=t(6540);function r(e,a,t){return a in e?Object.defineProperty(e,a,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[a]=t,e}function s(e,a){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);a&&(n=n.filter((function(a){return Object.getOwnPropertyDescriptor(e,a).enumerable}))),t.push.apply(t,n)}return t}function o(e){for(var a=1;a=0||(r[t]=e[t]);return r}(e,a);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(r[t]=e[t])}return r}var l=n.createContext({}),p=function(e){var a=n.useContext(l),t=a;return e&&(t="function"==typeof e?e(a):o(o({},a),e)),t},h=function(e){var a=p(e.components);return n.createElement(l.Provider,{value:a},e.children)},d="mdxType",c={inlineCode:"code",wrapper:function(e){var a=e.children;return n.createElement(n.Fragment,{},a)}},m=n.forwardRef((function(e,a){var t=e.components,r=e.mdxType,s=e.originalType,l=e.parentName,h=i(e,["components","mdxType","originalType","parentName"]),d=p(t),m=r,u=d["".concat(l,".").concat(m)]||d[m]||c[m]||s;return t?n.createElement(u,o(o({ref:a},h),{},{components:t})):n.createElement(u,o({ref:a},h))}));function u(e,a){var t=arguments,r=a&&a.mdxType;if("string"==typeof e||r){var s=t.length,o=new Array(s);o[0]=m;var i={};for(var l in a)hasOwnProperty.call(a,l)&&(i[l]=a[l]);i.originalType=e,i[d]="string"==typeof e?e:r,o[1]=i;for(var p=2;p{t.r(a),t.d(a,{assets:()=>l,contentTitle:()=>o,default:()=>c,frontMatter:()=>s,metadata:()=>i,toc:()=>p});var n=t(8168),r=(t(6540),t(5680));const s={},o="Lab 6 - Multiprocess and Multithread",i={unversionedId:"Compute/lab6",id:"Compute/lab6",title:"Lab 6 - Multiprocess and Multithread",description:"Task: Creating a process",source:"@site/docs/Compute/lab6.md",sourceDirName:"Compute",slug:"/Compute/lab6",permalink:"/operating-systems/75/Compute/lab6",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Not Race Condition",permalink:"/operating-systems/75/Compute/Questions/not-race-condition"},next:{title:"Lab 7 - Copy-on-Write",permalink:"/operating-systems/75/Compute/lab7"}},l={},p=[{value:"Task: Creating a process",id:"task-creating-a-process",level:2},{value:"Higher level - Python",id:"higher-level---python",level:3},{value:"Lower level - C",id:"lower-level---c",level:3},{value:"Task: Wait for Me",id:"task-wait-for-me",level:2},{value:"Task: Create Process",id:"task-create-process",level:2},{value:"Task: Multithreaded",id:"task-multithreaded",level:2},{value:"Task: Libraries for Parallel Processing",id:"task-libraries-for-parallel-processing",level:2},{value:"Array Sum in Python",id:"array-sum-in-python",level:3},{value:"Task: Wait for It",id:"task-wait-for-it",level:2},{value:"Memory Corruption",id:"memory-corruption",level:3},{value:"Hardware Perspective",id:"hardware-perspective",level:2},{value:"The Role of the Operating System",id:"the-role-of-the-operating-system",level:3},{value:"Processes",id:"processes",level:2},{value:"Fork",id:"fork",level:3},{value:"Threads",id:"threads",level:2},{value:"Threads vs Processes",id:"threads-vs-processes",level:3},{value:"Safety",id:"safety",level:4},{value:"Memory Layout of Multithreaded Programs",id:"memory-layout-of-multithreaded-programs",level:3},{value:"Guide: Baby steps - Python",id:"guide-baby-steps---python",level:2},{value:"Guide: Sum Array Processes",id:"guide-sum-array-processes",level:2},{value:"Sum of the Elements in an Array",id:"sum-of-the-elements-in-an-array",level:3},{value:"Spreading the Work Among Other Processes",id:"spreading-the-work-among-other-processes",level:3},{value:"Guide: system Dissected",id:"guide-system-dissected",level:2},{value:"Guide: Sum array Threads",id:"guide-sum-array-threads",level:2},{value:"Spreading the Work Among Other Threads",id:"spreading-the-work-among-other-threads",level:3},{value:"std.parallelism in D",id:"stdparallelism-in-d",level:3},{value:"OpenMP for C",id:"openmp-for-c",level:3},{value:"Guide: Threads and Processes: clone",id:"guide-threads-and-processes-clone",level:2}],h={toc:p},d="wrapper";function c(e){let{components:a,...s}=e;return(0,r.yg)(d,(0,n.A)({},h,s,{components:a,mdxType:"MDXLayout"}),(0,r.yg)("h1",{id:"lab-6---multiprocess-and-multithread"},"Lab 6 - Multiprocess and Multithread"),(0,r.yg)("h2",{id:"task-creating-a-process"},"Task: Creating a process"),(0,r.yg)("h3",{id:"higher-level---python"},"Higher level - Python"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/drills/tasks/sleepy")," directory, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", open the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/src")," folder and go through the practice items below."),(0,r.yg)("p",null,"Use the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/checker.sh")," script to check your solutions."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-bash"},"./checker.sh\nsleepy_creator ...................... passed ... 30\nsleepy_creator_wait ................. passed ... 30\nsleepy_creator_c .................... passed ... 40\n100 / 100\n")),(0,r.yg)("p",null,"Head over to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py"),"."),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Solve the ",(0,r.yg)("inlineCode",{parentName:"p"},"TODO"),": use ",(0,r.yg)("inlineCode",{parentName:"p"},"subprocess.Popen()")," to spawn 10 ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," processes."),(0,r.yg)("p",{parentName:"li"},"Start the script:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../tasks/sleepy/support$ python3 sleepy_creator.py\n")),(0,r.yg)("p",{parentName:"li"},"Look for the parent process:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep "python3 sleepy_creator.py")\n')),(0,r.yg)("p",{parentName:"li"},"It is a ",(0,r.yg)("inlineCode",{parentName:"p"},"python3")," process, as this is the interpreter that runs the script, but we call it the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py")," process for simplicity.\nNo output will be provided by the above command, as the parent process (",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py"),") dies before its child processes (the 10 ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," subprocesses) finish their execution.\nThe parent process of the newly created child processes is an ",(0,r.yg)("inlineCode",{parentName:"p"},"init"),"-like process: either ",(0,r.yg)("inlineCode",{parentName:"p"},"systemd"),"/",(0,r.yg)("inlineCode",{parentName:"p"},"init")," or another system process that adopts orphan processes.\nLook for the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep")," child processes using:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)\n PID PPID CMD\n4164 1680 sleep 1000\n4165 1680 sleep 1000\n4166 1680 sleep 1000\n4167 1680 sleep 1000\n4168 1680 sleep 1000\n4169 1680 sleep 1000\n4170 1680 sleep 1000\n4171 1680 sleep 1000\n4172 1680 sleep 1000\n4173 1680 sleep 1000\n")),(0,r.yg)("p",{parentName:"li"},"Notice that the child processes do not have ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py")," as a parent.\nWhat's more, as you saw above, ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py")," doesn't even exist anymore.\nThe child processes have been adopted by an ",(0,r.yg)("inlineCode",{parentName:"p"},"init"),"-like process (in the output above, that process has PID ",(0,r.yg)("inlineCode",{parentName:"p"},"1680")," - ",(0,r.yg)("inlineCode",{parentName:"p"},"PPID")," stands for ",(0,r.yg)("em",{parentName:"p"},"parent process ID"),")."),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("a",{parentName:"p",href:"Questions/parent-of-sleep-processes"},"Quiz"))),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Solve the ",(0,r.yg)("inlineCode",{parentName:"p"},"TODO"),": change the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py")," so that the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," processes remain the children of ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py"),".\nThis means that the parent / creator process must ",(0,r.yg)("strong",{parentName:"p"},"not")," exit until its children have finished their execution.\nIn other words, the parent / creator process must ",(0,r.yg)("strong",{parentName:"p"},"wait")," for the termination of its children.\nCheck out ",(0,r.yg)("a",{parentName:"p",href:"https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait"},(0,r.yg)("inlineCode",{parentName:"a"},"Popen.wait()"))," and add the code that makes the parent / creator process wait for its children.\nBefore anything, terminate the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep")," processes created above:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ pkill sleep\n")),(0,r.yg)("p",{parentName:"li"},"Start the program, again, as you did before:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../tasks/sleepy/support$ python3 sleepy_creator.py\n")),(0,r.yg)("p",{parentName:"li"},"On another terminal, verify that ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py")," remains the parent of the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep")," processes it creates:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)\n PID PPID CMD\n16107 9855 python3 sleepy_creator.py\n16108 16107 sleep 1000\n16109 16107 sleep 1000\n16110 16107 sleep 1000\n16111 16107 sleep 1000\n16112 16107 sleep 1000\n16113 16107 sleep 1000\n16114 16107 sleep 1000\n16115 16107 sleep 1000\n16116 16107 sleep 1000\n16117 16107 sleep 1000\n")),(0,r.yg)("p",{parentName:"li"},"Note that the parent process ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py")," (",(0,r.yg)("inlineCode",{parentName:"p"},"PID 16107"),") is still alive, and its child processes (the 10 ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000"),") have its ID as their ",(0,r.yg)("inlineCode",{parentName:"p"},"PPID"),".\nYou've successfully waited for the child processes to finish their execution."),(0,r.yg)("p",{parentName:"li"},"If you're having difficulties solving this exercise, go through ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-baby-steps---python"},"this")," reading material."))),(0,r.yg)("h3",{id:"lower-level---c"},"Lower level - C"),(0,r.yg)("p",null,"Now let's see how to create a child process in C.\nThere are multiple ways of doing this.\nFor now, we'll start with a higher-level approach."),(0,r.yg)("p",null,"Go to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.c")," and use ",(0,r.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/system.3.html"},(0,r.yg)("inlineCode",{parentName:"a"},"system"))," to create a ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," process."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/create-sleepy-process-ending"},"Quiz")),(0,r.yg)("p",null,"The ",(0,r.yg)("inlineCode",{parentName:"p"},"man")," page also mentions that ",(0,r.yg)("inlineCode",{parentName:"p"},"system")," calls ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"exec()")," to run the command it's given.\nIf you want to find out more about them, head over to the ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab7#task-mini-shell"},"Arena and create your own mini-shell"),"."),(0,r.yg)("h2",{id:"task-wait-for-me"},"Task: Wait for Me"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/drills/tasks/wait-for-me-processes/")," directory, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", open the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/src")," folder and go through the practice items below."),(0,r.yg)("p",null,"Use the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/checker.sh")," script to check your solutions."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-bash"},"wait_for_me_processes ...................... passed ... 100\n100 / 100\n")),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Run the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"wait_for_me_processes.py")," (e.g: ",(0,r.yg)("inlineCode",{parentName:"p"},"python3 wait_for_me_processes.py"),").\nThe parent process creates one child that writes and message to the given file.\nThen the parent reads that message.\nSimple enough, right?\nBut running the code raises a ",(0,r.yg)("inlineCode",{parentName:"p"},"FileNotFoundError"),".\nIf you inspect the file you gave the script as an argument, it does contain a string.\nWhat's going on?"),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("a",{parentName:"p",href:"Questions/cause-of-file-not-found-error"},"Quiz")),(0,r.yg)("p",{parentName:"li"},"In order to solve race conditions, we need ",(0,r.yg)("strong",{parentName:"p"},"synchronization"),".\nThis is a mechanism similar to a set of traffic lights in a crossroads.\nJust like traffic lights allow some cars to pass only after others have already passed, synchronization is a means for threads to communicate with each other and tell each other to access a resource or not."),(0,r.yg)("p",{parentName:"li"},"The most basic form of synchronization is ",(0,r.yg)("strong",{parentName:"p"},"waiting"),".\nConcretely, if the parent process ",(0,r.yg)("strong",{parentName:"p"},"waits")," for the child to end, we are sure the file is created and its contents are written.")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Use ",(0,r.yg)("inlineCode",{parentName:"p"},"join()")," to make the parent wait for its child before reading the file."))),(0,r.yg)("h2",{id:"task-create-process"},"Task: Create Process"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/drills/tasks/create-process/")," directory, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", open the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/src")," folder and go through the practice items below."),(0,r.yg)("p",null,"Use the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/checker.sh")," script to check your solutions."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-bash"},"./checker.sh\nexit_code22 ...................... passed ... 50\nsecond_fork ...................... passed ... 50\n100 / 100\n")),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Change the return value of the child process to 22 so that the value displayed by the parent is changed.")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Create a child process of the newly created child."))),(0,r.yg)("p",null,'Use a similar logic and a similar set of prints to those in the support code.\nTake a look at the printed PIDs.\nMake sure the PPID of the "grandchild" is the PID of the child, whose PPID is, in turn, the PID of the parent.'),(0,r.yg)("h2",{id:"task-multithreaded"},"Task: Multithreaded"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/multithreaded/")," folder, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", and go through the practice items below in the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/")," directory."),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Use the Makefile to compile ",(0,r.yg)("inlineCode",{parentName:"p"},"multithread.c"),", run it and follow the instructions."),(0,r.yg)("p",{parentName:"li"},"The aim of this task is to familiarize you with the ",(0,r.yg)("inlineCode",{parentName:"p"},"pthreads")," library.\nIn order to use it, you have to add ",(0,r.yg)("inlineCode",{parentName:"p"},"#include ")," in ",(0,r.yg)("inlineCode",{parentName:"p"},"multithreaded.c")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"-lpthread")," in the compiler options."),(0,r.yg)("p",{parentName:"li"},"The executable creates 5 threads besides the main thread, puts each of them to sleep for ",(0,r.yg)("strong",{parentName:"p"},"5 seconds"),", then waits for all of them to finish.\nGive it a run and notice that the total waiting time is around ",(0,r.yg)("strong",{parentName:"p"},"5 seconds")," since you started the last thread.\nThat is the whole point - they each run in parallel.")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Make each thread print its ID once it is done sleeping."),(0,r.yg)("p",{parentName:"li"},"Create a new function ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper2()")," identical to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper()")," to organize your work.\nSo far, the ",(0,r.yg)("inlineCode",{parentName:"p"},"data")," argument is unused (mind the ",(0,r.yg)("inlineCode",{parentName:"p"},"__unused")," attribute), so that is your starting point.\nYou cannot change ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper2()")," definition, since ",(0,r.yg)("inlineCode",{parentName:"p"},"pthreads_create()")," expects a pointer to a function that receives a ",(0,r.yg)("inlineCode",{parentName:"p"},"void *")," argument.\nWhat you can and should do is to pass a pointer to a ",(0,r.yg)("inlineCode",{parentName:"p"},"int")," as argument, and then cast ",(0,r.yg)("inlineCode",{parentName:"p"},"data")," to ",(0,r.yg)("inlineCode",{parentName:"p"},"int *")," inside ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper2()"),"."),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("strong",{parentName:"p"},"Note:")," Do not simply pass ",(0,r.yg)("inlineCode",{parentName:"p"},"&i")," as argument to the function.\nThis will make all threads to use the ",(0,r.yg)("strong",{parentName:"p"},"same integer")," as their ID."),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("strong",{parentName:"p"},"Note:")," Do not use global variables."),(0,r.yg)("p",{parentName:"li"},"If you get stuck you can google ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread example")," and you will probably stumble upon ",(0,r.yg)("a",{parentName:"p",href:"https://gist.github.com/ankurs/179778"},"this"),".")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"On top of printing its ID upon completion, make each thread sleep for a different amount of time."),(0,r.yg)("p",{parentName:"li"},"Create a new function ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper3()")," identical to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper()")," to organize your work.\nThe idea is to repeat what you did on the previous exercise and use the right argument for ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper3()"),".\nKeep in mind that you cannot change its definition.\nBonus points if you do not use the thread's ID as the sleeping amount."))),(0,r.yg)("h2",{id:"task-libraries-for-parallel-processing"},"Task: Libraries for Parallel Processing"),(0,r.yg)("p",null,"In ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/sum-array/support/c/sum_array_threads.c"),' we spawned threads "manually" by using the ',(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()")," function.\nThis is ",(0,r.yg)("strong",{parentName:"p"},"not")," a syscall, but a wrapper over the common syscall used by both ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," (which is also not a syscall) and ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()"),"."),(0,r.yg)("p",null,"Still, ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()")," is not yet a syscall.\nIn order to see what syscall ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()")," uses, check out ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-threads-and-processes-clone"},"this section"),"."),(0,r.yg)("p",null,"Most programming languages provide a more advanced API for handling parallel computation."),(0,r.yg)("h3",{id:"array-sum-in-python"},"Array Sum in Python"),(0,r.yg)("p",null,"Let's first probe this by implementing two parallel versions of the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sum-array/support/python/sum_array_sequential.py"),".\nOne version should use threads and the other should use processes.\nRun each of them using 1, 2, 4, and 8 threads / processes respectively and compare the running times.\nNotice that the running times of the multithreaded implementation do not decrease.\nThis is because the GIL makes it so that those threads that you create essentially run sequentially."),(0,r.yg)("p",null,"The GIL also makes it so that individual Python instructions are atomic.\nRun the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/synchronization/drills/tasks/race-condition/support/python/race_condition.py"),".\nEvery time, ",(0,r.yg)("inlineCode",{parentName:"p"},"var")," will be 0 because the GIL doesn't allow the two threads to run in parallel and reach the critical section at the same time.\nThis means that the instructions ",(0,r.yg)("inlineCode",{parentName:"p"},"var += 1")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"var -= 1")," become atomic."),(0,r.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-sum-array-threads"},"this")," reading material."),(0,r.yg)("h2",{id:"task-wait-for-it"},"Task: Wait for It"),(0,r.yg)("p",null,"The process that spawns all the others and subsequently calls ",(0,r.yg)("inlineCode",{parentName:"p"},"waitpid")," to wait for them to finish can also get their return codes.\nUpdate the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/sum-array-bugs/support/seg-fault/sum_array_processes.c")," and modify the call to ",(0,r.yg)("inlineCode",{parentName:"p"},"waitpid")," to obtain and investigate this return code.\nDisplay an appropriate message if one of the child processes returns an error."),(0,r.yg)("p",null,"Remember to use the appropriate ",(0,r.yg)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},"macros")," for handling the ",(0,r.yg)("inlineCode",{parentName:"p"},"status")," variable that is modified by ",(0,r.yg)("inlineCode",{parentName:"p"},"waitpid()"),", as it is a bit-field.\nWhen a process runs into a system error, it receives a signal.\nA signal is a means to interrupt the normal execution of a program from the outside.\nIt is associated with a number.\nUse ",(0,r.yg)("inlineCode",{parentName:"p"},"kill -l")," to find the full list of signals."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/seg-fault-exit-code"},"Quiz")),(0,r.yg)("p",null,"So up to this point we've seen that one advantage of processes is that they offer better safety than threads.\nBecause they use separate virtual address spaces, sibling processes are better isolated than threads.\nThus, an application that uses processes can be more robust to errors than if it were using threads."),(0,r.yg)("h3",{id:"memory-corruption"},"Memory Corruption"),(0,r.yg)("p",null,"Because they share the same address space, threads run the risk of corrupting each other's data.\nTake a look at the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sum-array-bugs/support/memory-corruption/python/"),".\nThe two programs only differ in how they spread their workload.\nOne uses threads while the other uses processes."),(0,r.yg)("p",null,"Run both programs with and without memory corruption.\nPass any value as a third argument to trigger the corruption."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sum-array-bugs/support/memory-corruption/python$ python3 memory_corruption_processes.py # no memory corruption\n[...]\n\nstudent@os:~/.../sum-array-bugs/support/memory-corruption/python$ python3 memory_corruption_processes.py 1 # do memory corruption\n[...]\n")),(0,r.yg)("p",null,"The one using threads will most likely print a negative sum, while the other displays the correct sum.\nThis happens because all threads refer to the same memory for the array ",(0,r.yg)("inlineCode",{parentName:"p"},"arr"),".\nWhat happens to the processes is a bit more complicated."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab7#copy-on-write"},"Later in this lab"),", we will see that initially, the page tables of all processes point to the same physical frames or ",(0,r.yg)("inlineCode",{parentName:"p"},"arr"),".\nWhen the malicious process tries to corrupt this array by ",(0,r.yg)("strong",{parentName:"p"},"writing data to it"),", the OS duplicates the original frames of ",(0,r.yg)("inlineCode",{parentName:"p"},"arr")," so that the malicious process writes the corrupted values to these new frames, while leaving the original ones untouched.\nThis mechanism is called ",(0,r.yg)("strong",{parentName:"p"},"Copy-on-Write")," and is an OS optimisation so that memory is shared between the parent and the child process, until one of them attempts to write to it.\nAt this point, this process receives its own separate copies of the previously shared frames."),(0,r.yg)("p",null,"Note that in order for the processes to share the ",(0,r.yg)("inlineCode",{parentName:"p"},"sums")," dictionary, it is not created as a regular dictionary, but using the ",(0,r.yg)("inlineCode",{parentName:"p"},"Manager")," module.\nThis module provides some special data structures that are allocated in ",(0,r.yg)("strong",{parentName:"p"},"shared memory")," so that all processes can access them.\nYou can learn more about shared memory and its various implementations ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab7#task-shared-memory"},"in this section"),"."),(0,r.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-sum-array-processes"},"this")," reading material."),(0,r.yg)("h2",{id:"hardware-perspective"},"Hardware Perspective"),(0,r.yg)("p",null,"The main criterion we use to rank CPUs is their ",(0,r.yg)("em",{parentName:"p"},"computation power"),", i.e. their ability to crunch numbers and do math.\nNumerous benchmarks exist out there, and they are publicly displayed on sites such as ",(0,r.yg)("a",{parentName:"p",href:"https://www.cpubenchmark.net/"},"CPUBenchmark"),"."),(0,r.yg)("p",null,"For example, a benchmark can measure the performance of the computer's CPU in a variety of scenarios:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"its ability to perform integer operations"),(0,r.yg)("li",{parentName:"ul"},"its speed in floating point arithmetic"),(0,r.yg)("li",{parentName:"ul"},"data encryption and compression"),(0,r.yg)("li",{parentName:"ul"},"sorting algorithms and others")),(0,r.yg)("p",null,"You can take a look at what exactly is measured using ",(0,r.yg)("a",{parentName:"p",href:"https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+Threadripper+PRO+5995WX"},"this link"),".\nIt displays the scores obtained by a high-end CPU.\nApart from the tests above, other benchmarks might focus on different performance metrics, such as branch prediction or prefetching."),(0,r.yg)("p",null,"Other approaches are less artificial, measuring performance on real-world applications such as compile times and performance in the latest (and most resource-demanding) video games.\nThe latter metric revolves around how many average FPS (frames per second) a given CPU is able to crank out in a specific video game.\nYou can find a lot of articles online on how CPU benchmarking is done."),(0,r.yg)("p",null,"Most benchmarks, unfortunately, are not open source, especially the more popular ones, such as ",(0,r.yg)("a",{parentName:"p",href:"https://browser.geekbench.com/processor-benchmarks"},"Geekbench 5"),".\nDespite this shortcoming, benchmarks are widely used to compare the performance of various computer ",(0,r.yg)("strong",{parentName:"p"},"hardware"),", CPUs included."),(0,r.yg)("h3",{id:"the-role-of-the-operating-system"},"The Role of the Operating System"),(0,r.yg)("p",null,'As you\'ve seen so far, the CPU provides the "muscle" required for fast computation, i.e. the highly optimised hardware and multiple ALUs, FPUs\nand cores necessary to perform those computations.\nHowever, it is the ',(0,r.yg)("strong",{parentName:"p"},"operating system"),' that provides the "brains" for this computation.\nSpecifically, modern CPUs have the capacity to run multiple tasks in parallel.\nBut they do not provide a means to decide which task to run at each moment.\nThe OS comes as an ',(0,r.yg)("em",{parentName:"p"},"orchestrator")," to ",(0,r.yg)("strong",{parentName:"p"},"schedule")," the way these tasks (that we will later call threads) are allowed to run and use the CPU's resources.\nThis way, the OS tells the CPU what code to run on each CPU core so that it reaches a good balance between high throughput (running many instructions) and fair access to CPU cores."),(0,r.yg)("p",null,"It is cumbersome for a user-level application to interact directly with the CPU.\nThe developer would have to write hardware-specific code, which is not scalable and is difficult to maintain.\nIn addition, doing so would leave it up to the developer to isolate their application from the others that are present on the system.\nThis leaves applications vulnerable to countless bugs and exploits."),(0,r.yg)("p",null,"To guard apps from these pitfalls, the OS comes and mediates interactions between regular programs and the CPU by providing a set of ",(0,r.yg)("strong",{parentName:"p"},"abstractions"),".\nThese abstractions offer a safe, uniform and also isolated way to leverage the CPU's resources, i.e. its cores.\nThere are 2 main abstractions: ",(0,r.yg)("strong",{parentName:"p"},"processes")," and ",(0,r.yg)("strong",{parentName:"p"},"threads"),"."),(0,r.yg)("p",null,(0,r.yg)("img",{alt:"Interaction between applications, OS and CPU",src:t(7032).A})),(0,r.yg)("p",null,"As we can see from the image above, an application can spawn one or more processes.\nEach of these is handled and maintained by the OS.\nSimilarly, each process can spawn however many threads, which are also managed by the OS.\nThe OS decides when and on what CPU core to make each thread run.\nThis is in line with the general interaction between an application and the hardware: it is always mediated by the OS."),(0,r.yg)("h2",{id:"processes"},"Processes"),(0,r.yg)("p",null,"A process is simply a running program.\nLet's take the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," command as a trivial example.\n",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," is a ",(0,r.yg)("strong",{parentName:"p"},"program")," on your system.\nIt has a binary file which you can find and inspect with the help of the ",(0,r.yg)("inlineCode",{parentName:"p"},"which")," command:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ which ls\n/usr/bin/ls\n\nstudent@os:~$ file /usr/bin/ls\n/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e3da6f0bc36b6398b8651bbc2e08831a21a90da, for GNU/Linux 3.2.0, stripped\n")),(0,r.yg)("p",null,"When you run it, the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," binary stored ",(0,r.yg)("strong",{parentName:"p"},"on the disk")," at ",(0,r.yg)("inlineCode",{parentName:"p"},"/usr/bin/ls")," is read by another application called the ",(0,r.yg)("strong",{parentName:"p"},"loader"),".\nThe loader spawns a ",(0,r.yg)("strong",{parentName:"p"},"process")," by copying some of the contents ",(0,r.yg)("inlineCode",{parentName:"p"},"/usr/bin/ls")," in memory (such as the ",(0,r.yg)("inlineCode",{parentName:"p"},".text"),", ",(0,r.yg)("inlineCode",{parentName:"p"},".rodata")," and ",(0,r.yg)("inlineCode",{parentName:"p"},".data")," sections).\nUsing ",(0,r.yg)("inlineCode",{parentName:"p"},"strace"),", we can see the ",(0,r.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/execve.2.html"},(0,r.yg)("inlineCode",{parentName:"a"},"execve"))," system call:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace -s 100 ls -a # -s 100 limits strings to 100 bytes instead of the default 32\nexecve("/usr/bin/ls", ["ls", "-a"], 0x7fffa7e0d008 /* 61 vars */) = 0\n[...]\nwrite(1, ". .. content\\tCONTRIBUTING.md COPYING.md .git .gitignore README.md REVIEWING.md\\n", 86. .. content CONTRIBUTING.md COPYING.md .git .gitignore README.md REVIEWING.md\n) = 86\nclose(1) = 0\nclose(2) = 0\nexit_group(0) = ?\n+++ exited with 0 +++\n')),(0,r.yg)("p",null,"Look at its parameters:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"the path to the ",(0,r.yg)("strong",{parentName:"li"},"program"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"/usr/bin/ls")),(0,r.yg)("li",{parentName:"ul"},"the list of arguments: ",(0,r.yg)("inlineCode",{parentName:"li"},'"ls", "-a"')),(0,r.yg)("li",{parentName:"ul"},"the environment variables: the rest of the syscall's arguments")),(0,r.yg)("p",null,(0,r.yg)("inlineCode",{parentName:"p"},"execve")," invokes the loader to load the VAS of the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," process ",(0,r.yg)("strong",{parentName:"p"},"by replacing that of the existing process"),".\nAll subsequent syscalls are performed by the newly spawned ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," process.\nWe will get into more details regarding ",(0,r.yg)("inlineCode",{parentName:"p"},"execve")," ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-system-dissected"},"towards the end of this lab"),"."),(0,r.yg)("p",null,(0,r.yg)("img",{alt:"Loading of `ls` Process",src:t(1286).A})),(0,r.yg)("h3",{id:"fork"},"Fork"),(0,r.yg)("p",null,"Up to now we've been creating processes using various high-level APIs, such as ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()"),", ",(0,r.yg)("inlineCode",{parentName:"p"},"Process()")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"system()"),".\nYes, despite being a C function, as you've seen from its man page, ",(0,r.yg)("inlineCode",{parentName:"p"},"system()")," itself calls 2 other functions: ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," to create a process and ",(0,r.yg)("inlineCode",{parentName:"p"},"execve()")," to execute the given command.\nAs you already know from the ",(0,r.yg)("a",{parentName:"p",href:"lab2.md#libraries-and-libc"},"Software Stack")," chapter, library functions may call one or more underlying system calls or other functions.\nNow we will move one step lower on the call stack and call ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," ourselves."),(0,r.yg)("p",null,(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," creates one child process that is ",(0,r.yg)("em",{parentName:"p"},"almost")," identical to its parent.\nWe say that ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," returns ",(0,r.yg)("strong",{parentName:"p"},"twice"),": once in the parent process and once more in the child process.\nThis means that after ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," returns, assuming no error has occurred, both the child and the parent resume execution from the same place: the instruction following the call to ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()"),".\nWhat's different between the two processes is the value returned by ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()"),":"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("strong",{parentName:"li"},"child process"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()")," returns 0"),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("strong",{parentName:"li"},"parent process"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()")," returns the PID of the child process (> 0)"),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("strong",{parentName:"li"},"on error"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()")," returns -1, only once, in the initial process")),(0,r.yg)("p",null,"Therefore, the typical code for handling a ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," is available in ",(0,r.yg)("inlineCode",{parentName:"p"},"create-process/support/fork.c"),".\nTake a look at it and then run it.\nNotice what each of the two processes prints:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"the PID of the child is also known by the parent"),(0,r.yg)("li",{parentName:"ul"},"the PPID of the child is the PID of the parent")),(0,r.yg)("p",null,"Unlike ",(0,r.yg)("inlineCode",{parentName:"p"},"system()"),", who also waits for its child, when using ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," we must do the waiting ourselves.\nIn order to wait for a process to end, we use the ",(0,r.yg)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},(0,r.yg)("inlineCode",{parentName:"a"},"waitpid()"))," syscall.\nIt places the exit code of the child process in the ",(0,r.yg)("inlineCode",{parentName:"p"},"status")," parameter.\nThis argument is actually a bit-field containing more information than merely the exit code.\nTo retrieve the exit code, we use the ",(0,r.yg)("inlineCode",{parentName:"p"},"WEXITSTATUS")," macro.\nKeep in mind that ",(0,r.yg)("inlineCode",{parentName:"p"},"WEXITSTATUS")," only makes sense if ",(0,r.yg)("inlineCode",{parentName:"p"},"WIFEXITED")," is true, i.e. if the child process finished on its own and wasn't killed by another one or by an illegal action (such as a segfault or illegal instruction) for example.\nOtherwise, ",(0,r.yg)("inlineCode",{parentName:"p"},"WEXITSTATUS")," will return something meaningless.\nYou can view the rest of the information stored in the ",(0,r.yg)("inlineCode",{parentName:"p"},"status")," bit-field ",(0,r.yg)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},"in the man page"),"."),(0,r.yg)("p",null,(0,r.yg)("strong",{parentName:"p"},"Moral of the story"),": Usually the execution flow is:"),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"fork()"),", followed by")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"wait()")," (called by the parent)")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"exit()"),", called by the child."))),(0,r.yg)("p",null,"The order of last 2 steps may be swapped."),(0,r.yg)("h2",{id:"threads"},"Threads"),(0,r.yg)("h3",{id:"threads-vs-processes"},"Threads vs Processes"),(0,r.yg)("p",null,"So why use the implementation that spawns more processes if it's slower than the one using threads?\nThe table below lists the differences between threads and processes.\nGenerally, if we only want to do some computing, we use threads.\nIf we need to drastically change the behaviour of the program, we need a new program altogether, or we need more than computing (e.g. communication on the network to create a computing cluster), we use processes."),(0,r.yg)("table",null,(0,r.yg)("thead",{parentName:"table"},(0,r.yg)("tr",{parentName:"thead"},(0,r.yg)("th",{parentName:"tr",align:"left"},"PROCESS"),(0,r.yg)("th",{parentName:"tr",align:"left"},"THREAD"))),(0,r.yg)("tbody",{parentName:"table"},(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"independent"),(0,r.yg)("td",{parentName:"tr",align:"left"},"part of a process")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"collection of threads"),(0,r.yg)("td",{parentName:"tr",align:"left"},"shares VAS with other threads")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"slower creation (new page table must be created)"),(0,r.yg)("td",{parentName:"tr",align:"left"},"faster creation")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"longer context switch duration (TLB must be flushed)"),(0,r.yg)("td",{parentName:"tr",align:"left"},"shorter context switch duration (part of the same process, so same TLB)")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"ending means ending all threads"),(0,r.yg)("td",{parentName:"tr",align:"left"},"other threads continue when finished")))),(0,r.yg)("h4",{id:"safety"},"Safety"),(0,r.yg)("p",null,"Compile and run the two programs in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/sum-array-bugs/support/seg-fault/"),", first with 2 processes and threads and then with 4.\nThey do the same thing as before: compute the sum of the elements in an array, but with a twist: each of them contains a bug causing a segfault.\nNotice that ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads"),' doesn\'t print anything with 4 threads, but merely a "Segmentation fault" message.\nOn the other hand, ',(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_processes")," prints a sum and a running time, albeit different from the sums we've seen so far."),(0,r.yg)("p",null,"The reason is that signals such as ",(0,r.yg)("inlineCode",{parentName:"p"},"SIGSEGV"),", which is used when a segmentation fault happens affect the entire process that handles them.\nTherefore, when we split our workload between several threads and one of them causes an error such as a segfault, that error is going to terminate the entire process.\nThe same thing happens when we use processes instead of threads: one process causes an error, which gets it killed, but the other processes continue their work unhindered.\nThis is why we end up with a lower sum in the end: because one process died too early and didn't manage to write the partial sum it had computed to the ",(0,r.yg)("inlineCode",{parentName:"p"},"results")," array."),(0,r.yg)("h3",{id:"memory-layout-of-multithreaded-programs"},"Memory Layout of Multithreaded Programs"),(0,r.yg)("p",null,"When a new thread is created, a new stack is allocated for a thread.\nThe default stack size if ",(0,r.yg)("inlineCode",{parentName:"p"},"8 MB")," / ",(0,r.yg)("inlineCode",{parentName:"p"},"8192 KB"),":"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ulimit -s\n8192\n")),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/multithreaded/support/")," directory to observe the update of the memory layout when creating new threads."),(0,r.yg)("p",null,"Build the ",(0,r.yg)("inlineCode",{parentName:"p"},"multithreaded")," executable:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../multithreaded/support$ make\n")),(0,r.yg)("p",null,"Start the program:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../multithreaded/support$ ./multithreaded\nPress key to start creating threads ...\n[...]\n")),(0,r.yg)("p",null,"And investigate it with ",(0,r.yg)("inlineCode",{parentName:"p"},"pmap")," on another console, while pressing a key to create new threads."),(0,r.yg)("p",null,"As you can see, there is a new ",(0,r.yg)("inlineCode",{parentName:"p"},"8192 KB")," area created for every thread, also increasing the total virtual size."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/thread-memory"},"Quiz")),(0,r.yg)("h2",{id:"guide-baby-steps---python"},"Guide: Baby steps - Python"),(0,r.yg)("p",null,"Run the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/guides/create-process/support/popen.py"),".\nIt simply spawns a new process running the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," command using ",(0,r.yg)("a",{parentName:"p",href:"https://docs.python.org/3/library/subprocess.html#subprocess.Popen"},(0,r.yg)("inlineCode",{parentName:"a"},"subprocess.Popen()")),".\nDo not worry about the huge list of arguments that ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()")," takes.\nThey are used for ",(0,r.yg)("strong",{parentName:"p"},"inter-process-communication"),".\nYou'll learn more about this in the ","[Application Interaction chapter]","."),(0,r.yg)("p",null,"Note that this usage of ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()")," is not entirely correct.\nYou'll discover why later, but for now focus on simply understanding how to use ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()")," on its own."),(0,r.yg)("p",null,"Now change the command to anything you want.\nAlso give it some arguments.\nFrom the outside, it's as if you were running these commands from the terminal."),(0,r.yg)("h2",{id:"guide-sum-array-processes"},"Guide: Sum Array Processes"),(0,r.yg)("h3",{id:"sum-of-the-elements-in-an-array"},"Sum of the Elements in an Array"),(0,r.yg)("p",null,"Let's assume we only have one process on our system, and that process knows how to add the numbers in an array.\nIt can use however many resources it wants, since there is no other process to contest it.\nIt would probably look like the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/guides/sum-array-processes/support/c/sum_array_sequential.c"),".\nThe program also measures the time spent computing the sum.\nLet's compile and run it:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sum-array/support/c$ ./sum_array_sequential\nArray sum is: 49945994146\nTime spent: 127 ms\n")),(0,r.yg)("p",null,"You will most likely get a different sum (because the array is made up of random numbers) and a different time than the ones shown above.\nThis is perfectly fine.\nUse these examples qualitatively, not quantitatively."),(0,r.yg)("h3",{id:"spreading-the-work-among-other-processes"},"Spreading the Work Among Other Processes"),(0,r.yg)("p",null,"Due to how it's implemented so far, our program only uses one of our CPU's cores.\nWe never tell it to distribute its workload to other cores.\nThis is wasteful as the rest of our cores remain unused:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ lscpu | grep ^CPU\\(s\\):\nCPU(s): 8\n")),(0,r.yg)("p",null,"We have 7 more cores waiting to add numbers in our array."),(0,r.yg)("p",null,(0,r.yg)("img",{alt:"What if we used 100% of the CPU?",src:t(7846).A,width:"541",height:"500"})),(0,r.yg)("p",null,"What if we use 7 more processes and spread the task of adding the numbers in this array between them?\nIf we split the array into several equal parts and designate a separate process to calculate the sum of each part, we should get a speedup because now the work performed by each individual process is reduced."),(0,r.yg)("p",null,"Let's take it methodically.\nCompile and run ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_processes.c")," using 1, 2, 4 and 8 processes respectively.\nIf your system only has 4 cores (",(0,r.yg)("a",{parentName:"p",href:"https://www.intel.com/content/www/us/en/gaming/resources/hyper-threading.html"},"hyperthreading")," included), limit your runs to 4 processes.\nNote the running times for each number of processes.\nWe expect the speedups compared to our reference run to be 1, 2, 4 and 8 respectively, right?"),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/processes-speedup"},"Quiz")),(0,r.yg)("p",null,"You most likely did get some speedup, especially when using 8 processes.\nNow we will try to improve this speedup by using ",(0,r.yg)("strong",{parentName:"p"},"threads")," instead."),(0,r.yg)("p",null,"Also notice that we're not using hundreds or thousands of processes.\nAssuming our system has 8 cores, only 8 ",(0,r.yg)("em",{parentName:"p"},"threads")," can run at the same time.\nIn general, ",(0,r.yg)("strong",{parentName:"p"},"the maximum number of threads that can run at the same time is equal to the number of cores"),".\nIn our example, each process only has one thread: its main thread.\nSo by consequence and by forcing the terminology (because it's the main thread of these processes that is running, not the processes themselves), we can only run in parallel a number of processes equal to at most the number of cores."),(0,r.yg)("h2",{id:"guide-system-dissected"},"Guide: ",(0,r.yg)("inlineCode",{parentName:"h2"},"system")," Dissected"),(0,r.yg)("p",null,"You already know that ",(0,r.yg)("inlineCode",{parentName:"p"},"system")," calls ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"execve()")," to create the new process.\nLet's see how and why.\nFirst, we run the following command to trace the ",(0,r.yg)("inlineCode",{parentName:"p"},"execve()")," syscalls used by ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator"),".\nWe'll leave ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," for later."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sleepy/support$ strace -e execve -ff -o syscalls ./sleepy_creator\n")),(0,r.yg)("p",null,"At this point, you will get two files whose names start with ",(0,r.yg)("inlineCode",{parentName:"p"},"syscalls"),", followed by some numbers.\nThose numbers are the PIDs of the parent and the child process.\nTherefore, the file with the higher number contains logs of the ",(0,r.yg)("inlineCode",{parentName:"p"},"execve")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," syscalls issued by the parent process, while\nthe other logs those two syscalls when made by the child process.\nLet's take a look at them.\nThe numbers below will differ from those on your system:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../sleepy/support:$ cat syscalls.2523393 # syscalls from parent process\nexecve("sleepy_creator", ["sleepy_creator"], 0x7ffd2c157758 /* 39 vars */) = 0\n--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2523394, si_uid=1052093, si_status=0, si_utime=0, si_stime=0} ---\n+++ exited with 0 +++\n\nstudent@os:~/.../sleepy/support:$ cat syscalls.2523394 # syscalls from child process\nexecve("/bin/sh", ["sh", "-c", "sleep 10"], 0x7ffd36253be8 /* 39 vars */) = 0\nexecve("/usr/bin/sleep", ["sleep", "10"], 0x560f41659d40 /* 38 vars */) = 0\n+++ exited with 0 +++\n')),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/who-calls-execve-parent"},"Quiz")),(0,r.yg)("p",null,"Now notice that the child process doesn't simply call ",(0,r.yg)("inlineCode",{parentName:"p"},'execve("/usr/bin/sleep" ...)'),".\nIt first changes its virtual address space (VAS) to that of a ",(0,r.yg)("inlineCode",{parentName:"p"},"bash")," process (",(0,r.yg)("inlineCode",{parentName:"p"},'execve("/bin/sh" ...)'),") and then that ",(0,r.yg)("inlineCode",{parentName:"p"},"bash")," process switches its VAS to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep"),".\nTherefore, calling ",(0,r.yg)("inlineCode",{parentName:"p"},"system()")," is equivalent to running ",(0,r.yg)("inlineCode",{parentName:"p"},"")," in the command-line."),(0,r.yg)("p",null,(0,r.yg)("strong",{parentName:"p"},"Moral of the story"),": When spawning a new command, the call order is:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"parent: ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()"),", ",(0,r.yg)("inlineCode",{parentName:"li"},"exec()"),", ",(0,r.yg)("inlineCode",{parentName:"li"},"wait()")),(0,r.yg)("li",{parentName:"ul"},"child: ",(0,r.yg)("inlineCode",{parentName:"li"},"exit()"))),(0,r.yg)("h2",{id:"guide-sum-array-threads"},"Guide: Sum array Threads"),(0,r.yg)("h3",{id:"spreading-the-work-among-other-threads"},"Spreading the Work Among Other Threads"),(0,r.yg)("p",null,"Compile the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/guides/sum-array-threads/support/c/sum_array_threads.c")," and run it using 1, 2, 4 and 8 threads as you did before.\nEach thread runs the ",(0,r.yg)("inlineCode",{parentName:"p"},"calculate_array_part_sum()")," function and then finishes.\nRunning times should be ",(0,r.yg)("em",{parentName:"p"},"slightly")," smaller than the implementation using processes.\nThis slight time difference is caused by process creation actions, which are costlier than thread creation actions.\nBecause a process needs a separate virtual address space (VAS) and needs to duplicate some internal structures such as the file descriptor table and page table, it takes the operating system more time to create it than to create a thread.\nOn the other hand, threads belonging to the same process share the same VAS and, implicitly, the same OS-internal structures.\nTherefore, they are more lightweight than processes."),(0,r.yg)("h3",{id:"stdparallelism-in-d"},(0,r.yg)("inlineCode",{parentName:"h3"},"std.parallelism")," in D"),(0,r.yg)("p",null,"D language's standard library exposes the ",(0,r.yg)("a",{parentName:"p",href:"https://dlang.org/phobos/std_parallelism.html"},(0,r.yg)("inlineCode",{parentName:"a"},"std.parallelism")),", which provides a series of parallel processing functions.\nOne such function is ",(0,r.yg)("inlineCode",{parentName:"p"},"reduce()"),", which splits an array between a given number of threads and applies a given operation to these chunks.\nIn our case, the operation simply adds the elements to an accumulator: ",(0,r.yg)("inlineCode",{parentName:"p"},"a + b"),".\nFollow and run the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/guides/sum-array-threads/support/d/sum_array_threads_reduce.d"),"."),(0,r.yg)("p",null,"The number of threads is used within a ",(0,r.yg)("a",{parentName:"p",href:"https://dlang.org/phobos/std_parallelism.html#.TaskPool"},(0,r.yg)("inlineCode",{parentName:"a"},"TaskPool")),".\nThis structure is a thread manager (not scheduler).\nIt silently creates the number of threads we request and then ",(0,r.yg)("inlineCode",{parentName:"p"},"reduce()")," spreads its workload between these threads."),(0,r.yg)("p",null,"Now that you've seen how parallelism works in D, go in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/guides/sum-array-threads/support/java/SumArrayThreads.java")," and follow the TODOs.\nThe code is similar to the one written in D, and it uses ",(0,r.yg)("inlineCode",{parentName:"p"},"ThreadPoolExecutor"),".\nMore about that ",(0,r.yg)("a",{parentName:"p",href:"https://www.baeldung.com/thread-pool-java-and-guava"},"here"),".\nTo run the code use:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-java"},"javac SumArrayThreads.java\njava SumArrayThreads 4\n")),(0,r.yg)("p",null,"4 is the number of threads used, but you can replace the value with a number less or equal than your available cores."),(0,r.yg)("h3",{id:"openmp-for-c"},"OpenMP for C"),(0,r.yg)("p",null,"Unlike D, C does not support parallel computation by design.\nIt needs a library to do advanced things, like ",(0,r.yg)("inlineCode",{parentName:"p"},"reduce()")," from D.\nWe have chosen to use the OpenMP library for this.\nFollow the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/guides/sum-array-threads/support/c/sum_array_threads_openmp.c"),"."),(0,r.yg)("p",null,"The ",(0,r.yg)("inlineCode",{parentName:"p"},"#pragma")," used in the code instructs the compiler to enable the ",(0,r.yg)("inlineCode",{parentName:"p"},"omp")," module, and to parallelise the code.\nIn this case, we instruct the compiler to perform a reduce of the array, using the ",(0,r.yg)("inlineCode",{parentName:"p"},"+")," operator, and to store the results in the ",(0,r.yg)("inlineCode",{parentName:"p"},"result")," variable.\nThis reduction uses threads to calculate the sum, similar to ",(0,r.yg)("inlineCode",{parentName:"p"},"summ_array_threads.c"),", but in a much more optimised form."),(0,r.yg)("p",null,"One of the advantages of OpenMP is that is relatively easy to use.\nThe syntax requires only a few additional lines of code and compiler options, thus converting sequential code into parallel code quickly.\nFor example, using ",(0,r.yg)("inlineCode",{parentName:"p"},"#pragma omp parallel for"),", a developer can parallelize a ",(0,r.yg)("inlineCode",{parentName:"p"},"for loop"),", enabling iterations to run across multiple threads."),(0,r.yg)("p",null,"OpenMP uses a ",(0,r.yg)("inlineCode",{parentName:"p"},"shared-memory model"),", meaning all threads can access a common memory space.\nThis model is particularly useful for tasks that require frequent access to shared data, as it avoids the overhead of transferring data between threads.\nHowever, shared memory can also introduce challenges, such as race conditions or synchronization issues, which can occur when multiple threads attempt to modify the same data simultaneously, but we'll talk about that later.\nOpenMP offers constructs such as critical sections, atomic operations, and reductions to help manage these issues and ensure that parallel code executes safely and correctly."),(0,r.yg)("p",null,"Now compile and run the ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads_openmp")," binary using 1, 2, 4, and 8 threads as before.\nYou'll see lower running times than ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads")," due to the highly-optimised code emitted by the compiler.\nFor this reason and because library functions are usually much better tested than your own code, it is always preferred to use a library function for a given task."),(0,r.yg)("p",null,"For a challenge, enter ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/guides/sum-array-threads/support/c/add_array_threads_openmp.c"),".\nUse what you've learned from the previous exercise and add the value 100 to an array using OpenMP."),(0,r.yg)("h2",{id:"guide-threads-and-processes-clone"},"Guide: Threads and Processes: ",(0,r.yg)("inlineCode",{parentName:"h2"},"clone")),(0,r.yg)("p",null,"Let's go back to our initial demos that used threads and processes.\nWe'll see that in order to create both threads and processes, the underlying Linux syscall is ",(0,r.yg)("inlineCode",{parentName:"p"},"clone"),".\nFor this, we'll run both ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_processes")," under ",(0,r.yg)("inlineCode",{parentName:"p"},"strace"),".\nAs we've already established, we're only interested in the ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," syscall:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sum-array/support/c$ strace -e clone,clone3 ./sum_array_threads 2\nclone(child_stack=0x7f60b56482b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819693], tls=0x7f60b5649640, child_tidptr=0x7f60b5649910) = 1819693\nclone(child_stack=0x7f60b4e472b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819694], tls=0x7f60b4e48640, child_tidptr=0x7f60b4e48910) = 1819694\n\nstudent@os:~/.../sum-array/support/c$ strace -e clone,clone3 ./sum_array_processes 2\nclone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820599\nclone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820600\n")),(0,r.yg)("p",null,"We ran each program with an argument of 2, so we have 2 calls to ",(0,r.yg)("inlineCode",{parentName:"p"},"clone"),".\nNotice that in the case of threads, the ",(0,r.yg)("inlineCode",{parentName:"p"},"clone3")," syscall receives more arguments.\nThe relevant flags passed as arguments when creating threads are documented in ",(0,r.yg)("a",{parentName:"p",href:"https://man.archlinux.org/man/clone3.2.en"},(0,r.yg)("inlineCode",{parentName:"a"},"clone"),"'s man page"),":"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"CLONE_VM"),": the child and the parent process share the same VAS"),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"CLONE_{FS,FILES,SIGHAND}"),": the new thread shares the filesystem information, file and signal handlers with the one that created it.\nThe syscall also receives valid pointers to the new thread's stack and TLS, i.e. the only parts of the VAS that are distinct between threads (although they are technically accessible from all threads).")),(0,r.yg)("p",null,"By contrast, when creating a new process, the arguments of the ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," syscall are simpler (i.e. fewer flags are present).\nRemember that in both cases ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," creates a new ",(0,r.yg)("strong",{parentName:"p"},"thread"),".\nWhen creating a process, ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," creates this new thread within a new separate address space."))}c.isMDXComponent=!0},7846:(e,a,t)=>{t.d(a,{A:()=>n});const n=t.p+"assets/images/100-percent-cpu-1138186529f154d864f643179e25cea1.jpeg"},7032:(e,a,t)=>{t.d(a,{A:()=>n});const n=t.p+"assets/images/app-os-cpu-interaction-ca7fbdbb7da380e0992c95467ef267ce.svg"},1286:(e,a,t)=>{t.d(a,{A:()=>n});const n=t.p+"assets/images/loading-of-ls-process-0dec67c0d0a826710e06f980224d5eb4.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/8849ec36.cdc50d1c.js b/75/assets/js/8849ec36.cdc50d1c.js deleted file mode 100644 index c442c85173..0000000000 --- a/75/assets/js/8849ec36.cdc50d1c.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9577],{5680:(e,t,a)=>{a.d(t,{xA:()=>h,yg:()=>u});var n=a(6540);function r(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function s(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,n)}return a}function o(e){for(var t=1;t=0||(r[a]=e[a]);return r}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(r[a]=e[a])}return r}var p=n.createContext({}),l=function(e){var t=n.useContext(p),a=t;return e&&(a="function"==typeof e?e(t):o(o({},t),e)),a},h=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},d="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var a=e.components,r=e.mdxType,s=e.originalType,p=e.parentName,h=i(e,["components","mdxType","originalType","parentName"]),d=l(a),m=r,u=d["".concat(p,".").concat(m)]||d[m]||c[m]||s;return a?n.createElement(u,o(o({ref:t},h),{},{components:a})):n.createElement(u,o({ref:t},h))}));function u(e,t){var a=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var s=a.length,o=new Array(s);o[0]=m;var i={};for(var p in t)hasOwnProperty.call(t,p)&&(i[p]=t[p]);i.originalType=e,i[d]="string"==typeof e?e:r,o[1]=i;for(var l=2;l{a.r(t),a.d(t,{assets:()=>p,contentTitle:()=>o,default:()=>c,frontMatter:()=>s,metadata:()=>i,toc:()=>l});var n=a(8168),r=(a(6540),a(5680));const s={},o="Lab 6 - Multiprocess and Multithread",i={unversionedId:"Compute/lab6",id:"Compute/lab6",title:"Lab 6 - Multiprocess and Multithread",description:"Task: Creating a process",source:"@site/docs/Compute/lab6.md",sourceDirName:"Compute",slug:"/Compute/lab6",permalink:"/operating-systems/75/Compute/lab6",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Not Race Condition",permalink:"/operating-systems/75/Compute/Questions/not-race-condition"},next:{title:"Lab 7 - Copy-on-Write",permalink:"/operating-systems/75/Compute/lab7"}},p={},l=[{value:"Task: Creating a process",id:"task-creating-a-process",level:2},{value:"Higher level - Python",id:"higher-level---python",level:3},{value:"Lower level - C",id:"lower-level---c",level:3},{value:"Task: Wait for Me",id:"task-wait-for-me",level:2},{value:"Task: Create Process",id:"task-create-process",level:2},{value:"Task: Multithreaded",id:"task-multithreaded",level:2},{value:"Task: Libraries for Parallel Processing",id:"task-libraries-for-parallel-processing",level:2},{value:"std.parallelism in D",id:"stdparallelism-in-d",level:3},{value:"OpenMP for C",id:"openmp-for-c",level:3},{value:"Array Sum in Python",id:"array-sum-in-python",level:3},{value:"Task: Wait for It",id:"task-wait-for-it",level:2},{value:"Memory Corruption",id:"memory-corruption",level:3},{value:"Hardware Perspective",id:"hardware-perspective",level:2},{value:"The Role of the Operating System",id:"the-role-of-the-operating-system",level:3},{value:"Processes",id:"processes",level:2},{value:"Fork",id:"fork",level:3},{value:"Threads",id:"threads",level:2},{value:"Threads vs Processes",id:"threads-vs-processes",level:3},{value:"Safety",id:"safety",level:4},{value:"Memory Layout of Multithreaded Programs",id:"memory-layout-of-multithreaded-programs",level:3},{value:"Guide: Baby steps - Python",id:"guide-baby-steps---python",level:2},{value:"Guide: Sum Array Processes",id:"guide-sum-array-processes",level:2},{value:"Sum of the Elements in an Array",id:"sum-of-the-elements-in-an-array",level:3},{value:"Spreading the Work Among Other Processes",id:"spreading-the-work-among-other-processes",level:3},{value:"Guide: system Dissected",id:"guide-system-dissected",level:2},{value:"Guide: Sum array Threads",id:"guide-sum-array-threads",level:2},{value:"Spreading the Work Among Other Threads",id:"spreading-the-work-among-other-threads",level:3},{value:"Guide: Threads and Processes: clone",id:"guide-threads-and-processes-clone",level:2}],h={toc:l},d="wrapper";function c(e){let{components:t,...s}=e;return(0,r.yg)(d,(0,n.A)({},h,s,{components:t,mdxType:"MDXLayout"}),(0,r.yg)("h1",{id:"lab-6---multiprocess-and-multithread"},"Lab 6 - Multiprocess and Multithread"),(0,r.yg)("h2",{id:"task-creating-a-process"},"Task: Creating a process"),(0,r.yg)("h3",{id:"higher-level---python"},"Higher level - Python"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/drills/tasks/sleepy")," directory, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", open the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/src")," folder and go through the practice items below."),(0,r.yg)("p",null,"Use the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/checker.sh")," script to check your solutions."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-bash"},"./checker.sh\nsleepy_creator ...................... passed ... 30\nsleepy_creator_wait ................. passed ... 30\nsleepy_creator_c .................... passed ... 40\n100 / 100\n")),(0,r.yg)("p",null,"Head over to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py"),"."),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Solve the ",(0,r.yg)("inlineCode",{parentName:"p"},"TODO"),": use ",(0,r.yg)("inlineCode",{parentName:"p"},"subprocess.Popen()")," to spawn 10 ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," processes."),(0,r.yg)("p",{parentName:"li"},"Start the script:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../tasks/sleepy/support$ python3 sleepy_creator.py\n")),(0,r.yg)("p",{parentName:"li"},"Look for the parent process:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep "python3 sleepy_creator.py")\n')),(0,r.yg)("p",{parentName:"li"},"It is a ",(0,r.yg)("inlineCode",{parentName:"p"},"python3")," process, as this is the interpreter that runs the script, but we call it the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py")," process for simplicity.\nNo output will be provided by the above command, as the parent process (",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py"),") dies before its child processes (the 10 ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," subprocesses) finish their execution.\nThe parent process of the newly created child processes is an ",(0,r.yg)("inlineCode",{parentName:"p"},"init"),"-like process: either ",(0,r.yg)("inlineCode",{parentName:"p"},"systemd"),"/",(0,r.yg)("inlineCode",{parentName:"p"},"init")," or another system process that adopts orphan processes.\nLook for the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep")," child processes using:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)\n PID PPID CMD\n4164 1680 sleep 1000\n4165 1680 sleep 1000\n4166 1680 sleep 1000\n4167 1680 sleep 1000\n4168 1680 sleep 1000\n4169 1680 sleep 1000\n4170 1680 sleep 1000\n4171 1680 sleep 1000\n4172 1680 sleep 1000\n4173 1680 sleep 1000\n")),(0,r.yg)("p",{parentName:"li"},"Notice that the child processes do not have ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py")," as a parent.\nWhat's more, as you saw above, ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.py")," doesn't even exist anymore.\nThe child processes have been adopted by an ",(0,r.yg)("inlineCode",{parentName:"p"},"init"),"-like process (in the output above, that process has PID ",(0,r.yg)("inlineCode",{parentName:"p"},"1680")," - ",(0,r.yg)("inlineCode",{parentName:"p"},"PPID")," stands for ",(0,r.yg)("em",{parentName:"p"},"parent process ID"),")."),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("a",{parentName:"p",href:"Questions/parent-of-sleep-processes"},"Quiz"))),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Solve the ",(0,r.yg)("inlineCode",{parentName:"p"},"TODO"),": change the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py")," so that the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," processes remain the children of ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py"),".\nThis means that the parent / creator process must ",(0,r.yg)("strong",{parentName:"p"},"not")," exit until its children have finished their execution.\nIn other words, the parent / creator process must ",(0,r.yg)("strong",{parentName:"p"},"wait")," for the termination of its children.\nCheck out ",(0,r.yg)("a",{parentName:"p",href:"https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait"},(0,r.yg)("inlineCode",{parentName:"a"},"Popen.wait()"))," and add the code that makes the parent / creator process wait for its children.\nBefore anything, terminate the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep")," processes created above:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ pkill sleep\n")),(0,r.yg)("p",{parentName:"li"},"Start the program, again, as you did before:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../tasks/sleepy/support$ python3 sleepy_creator.py\n")),(0,r.yg)("p",{parentName:"li"},"On another terminal, verify that ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py")," remains the parent of the ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep")," processes it creates:"),(0,r.yg)("pre",{parentName:"li"},(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)\n PID PPID CMD\n16107 9855 python3 sleepy_creator.py\n16108 16107 sleep 1000\n16109 16107 sleep 1000\n16110 16107 sleep 1000\n16111 16107 sleep 1000\n16112 16107 sleep 1000\n16113 16107 sleep 1000\n16114 16107 sleep 1000\n16115 16107 sleep 1000\n16116 16107 sleep 1000\n16117 16107 sleep 1000\n")),(0,r.yg)("p",{parentName:"li"},"Note that the parent process ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator_wait.py")," (",(0,r.yg)("inlineCode",{parentName:"p"},"PID 16107"),") is still alive, and its child processes (the 10 ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000"),") have its ID as their ",(0,r.yg)("inlineCode",{parentName:"p"},"PPID"),".\nYou've successfully waited for the child processes to finish their execution."),(0,r.yg)("p",{parentName:"li"},"If you're having difficulties solving this exercise, go through ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-baby-steps---python"},"this")," reading material."))),(0,r.yg)("h3",{id:"lower-level---c"},"Lower level - C"),(0,r.yg)("p",null,"Now let's see how to create a child process in C.\nThere are multiple ways of doing this.\nFor now, we'll start with a higher-level approach."),(0,r.yg)("p",null,"Go to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator.c")," and use ",(0,r.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/system.3.html"},(0,r.yg)("inlineCode",{parentName:"a"},"system"))," to create a ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep 1000")," process."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/create-sleepy-process-ending"},"Quiz")),(0,r.yg)("p",null,"The ",(0,r.yg)("inlineCode",{parentName:"p"},"man")," page also mentions that ",(0,r.yg)("inlineCode",{parentName:"p"},"system")," calls ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"exec()")," to run the command it's given.\nIf you want to find out more about them, head over to the ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab7#task-mini-shell"},"Arena and create your own mini-shell"),"."),(0,r.yg)("h2",{id:"task-wait-for-me"},"Task: Wait for Me"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/drills/tasks/wait-for-me-processes/")," directory, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", open the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/src")," folder and go through the practice items below."),(0,r.yg)("p",null,"Use the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/checker.sh")," script to check your solutions."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-bash"},"wait_for_me_processes ...................... passed ... 100\n100 / 100\n")),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Run the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"wait_for_me_processes.py")," (e.g: ",(0,r.yg)("inlineCode",{parentName:"p"},"python3 wait_for_me_processes.py"),").\nThe parent process creates one child that writes and message to the given file.\nThen the parent reads that message.\nSimple enough, right?\nBut running the code raises a ",(0,r.yg)("inlineCode",{parentName:"p"},"FileNotFoundError"),".\nIf you inspect the file you gave the script as an argument, it does contain a string.\nWhat's going on?"),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("a",{parentName:"p",href:"Questions/cause-of-file-not-found-error"},"Quiz")),(0,r.yg)("p",{parentName:"li"},"In order to solve race conditions, we need ",(0,r.yg)("strong",{parentName:"p"},"synchronization"),".\nThis is a mechanism similar to a set of traffic lights in a crossroads.\nJust like traffic lights allow some cars to pass only after others have already passed, synchronization is a means for threads to communicate with each other and tell each other to access a resource or not."),(0,r.yg)("p",{parentName:"li"},"The most basic form of synchronization is ",(0,r.yg)("strong",{parentName:"p"},"waiting"),".\nConcretely, if the parent process ",(0,r.yg)("strong",{parentName:"p"},"waits")," for the child to end, we are sure the file is created and its contents are written.")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Use ",(0,r.yg)("inlineCode",{parentName:"p"},"join()")," to make the parent wait for its child before reading the file."))),(0,r.yg)("h2",{id:"task-create-process"},"Task: Create Process"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/drills/tasks/create-process/")," directory, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", open the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/src")," folder and go through the practice items below."),(0,r.yg)("p",null,"Use the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/checker.sh")," script to check your solutions."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-bash"},"./checker.sh\nexit_code22 ...................... passed ... 50\nsecond_fork ...................... passed ... 50\n100 / 100\n")),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Change the return value of the child process to 22 so that the value displayed by the parent is changed.")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Create a child process of the newly created child."))),(0,r.yg)("p",null,'Use a similar logic and a similar set of prints to those in the support code.\nTake a look at the printed PIDs.\nMake sure the PPID of the "grandchild" is the PID of the child, whose PPID is, in turn, the PID of the parent.'),(0,r.yg)("h2",{id:"task-multithreaded"},"Task: Multithreaded"),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/multithreaded/")," folder, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make skels"),", and go through the practice items below in the ",(0,r.yg)("inlineCode",{parentName:"p"},"support/")," directory."),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Use the Makefile to compile ",(0,r.yg)("inlineCode",{parentName:"p"},"multithread.c"),", run it and follow the instructions."),(0,r.yg)("p",{parentName:"li"},"The aim of this task is to familiarize you with the ",(0,r.yg)("inlineCode",{parentName:"p"},"pthreads")," library.\nIn order to use it, you have to add ",(0,r.yg)("inlineCode",{parentName:"p"},"#include ")," in ",(0,r.yg)("inlineCode",{parentName:"p"},"multithreaded.c")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"-lpthread")," in the compiler options."),(0,r.yg)("p",{parentName:"li"},"The executable creates 5 threads besides the main thread, puts each of them to sleep for ",(0,r.yg)("strong",{parentName:"p"},"5 seconds"),", then waits for all of them to finish.\nGive it a run and notice that the total waiting time is around ",(0,r.yg)("strong",{parentName:"p"},"5 seconds")," since you started the last thread.\nThat is the whole point - they each run in parallel.")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"Make each thread print its ID once it is done sleeping."),(0,r.yg)("p",{parentName:"li"},"Create a new function ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper2()")," identical to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper()")," to organize your work.\nSo far, the ",(0,r.yg)("inlineCode",{parentName:"p"},"data")," argument is unused (mind the ",(0,r.yg)("inlineCode",{parentName:"p"},"__unused")," attribute), so that is your starting point.\nYou cannot change ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper2()")," definition, since ",(0,r.yg)("inlineCode",{parentName:"p"},"pthreads_create()")," expects a pointer to a function that receives a ",(0,r.yg)("inlineCode",{parentName:"p"},"void *")," argument.\nWhat you can and should do is to pass a pointer to a ",(0,r.yg)("inlineCode",{parentName:"p"},"int")," as argument, and then cast ",(0,r.yg)("inlineCode",{parentName:"p"},"data")," to ",(0,r.yg)("inlineCode",{parentName:"p"},"int *")," inside ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper2()"),"."),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("strong",{parentName:"p"},"Note:")," Do not simply pass ",(0,r.yg)("inlineCode",{parentName:"p"},"&i")," as argument to the function.\nThis will make all threads to use the ",(0,r.yg)("strong",{parentName:"p"},"same integer")," as their ID."),(0,r.yg)("p",{parentName:"li"},(0,r.yg)("strong",{parentName:"p"},"Note:")," Do not use global variables."),(0,r.yg)("p",{parentName:"li"},"If you get stuck you can google ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread example")," and you will probably stumble upon ",(0,r.yg)("a",{parentName:"p",href:"https://gist.github.com/ankurs/179778"},"this"),".")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},"On top of printing its ID upon completion, make each thread sleep for a different amount of time."),(0,r.yg)("p",{parentName:"li"},"Create a new function ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper3()")," identical to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper()")," to organize your work.\nThe idea is to repeat what you did on the previous exercise and use the right argument for ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep_wrapper3()"),".\nKeep in mind that you cannot change its definition.\nBonus points if you do not use the thread's ID as the sleeping amount."))),(0,r.yg)("h2",{id:"task-libraries-for-parallel-processing"},"Task: Libraries for Parallel Processing"),(0,r.yg)("p",null,"In ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/sum-array/support/c/sum_array_threads.c"),' we spawned threads "manually" by using the ',(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()")," function.\nThis is ",(0,r.yg)("strong",{parentName:"p"},"not")," a syscall, but a wrapper over the common syscall used by both ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," (which is also not a syscall) and ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()"),"."),(0,r.yg)("p",null,"Still, ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()")," is not yet a syscall.\nIn order to see what syscall ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread_create()")," uses, check out ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-threads-and-processes-clone"},"this section"),"."),(0,r.yg)("p",null,"Most programming languages provide a more advanced API for handling parallel computation."),(0,r.yg)("h3",{id:"stdparallelism-in-d"},(0,r.yg)("inlineCode",{parentName:"h3"},"std.parallelism")," in D"),(0,r.yg)("p",null,"D language's standard library exposes the ",(0,r.yg)("a",{parentName:"p",href:"https://dlang.org/phobos/std_parallelism.html"},(0,r.yg)("inlineCode",{parentName:"a"},"std.parallelism")),", which provides a series of parallel processing functions.\nOne such function is ",(0,r.yg)("inlineCode",{parentName:"p"},"reduce()"),", which splits an array between a given number of threads and applies a given operation to these chunks.\nIn our case, the operation simply adds the elements to an accumulator: ",(0,r.yg)("inlineCode",{parentName:"p"},"a + b"),".\nFollow and run the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sum-array/support/d/sum_array_threads_reduce.d"),"."),(0,r.yg)("p",null,"The number of threads is used within a ",(0,r.yg)("a",{parentName:"p",href:"https://dlang.org/phobos/std_parallelism.html#.TaskPool"},(0,r.yg)("inlineCode",{parentName:"a"},"TaskPool")),".\nThis structure is a thread manager (not scheduler).\nIt silently creates the number of threads we request and then ",(0,r.yg)("inlineCode",{parentName:"p"},"reduce()")," spreads its workload between these threads."),(0,r.yg)("h3",{id:"openmp-for-c"},"OpenMP for C"),(0,r.yg)("p",null,"Unlike D, C does not support parallel computation by design.\nIt needs a library to do advanced things, like ",(0,r.yg)("inlineCode",{parentName:"p"},"reduce()")," from D.\nWe have chosen to use the OpenMP library for this.\nFollow the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"support/sum-array/c/sum_array_threads_openmp.c"),"."),(0,r.yg)("p",null,"The ",(0,r.yg)("inlineCode",{parentName:"p"},"#pragma")," used in the code instructs the compiler to enable the ",(0,r.yg)("inlineCode",{parentName:"p"},"omp")," module, and to parallelise the code.\nIn this case, we instruct the compiler to perform a reduce of the array, using the ",(0,r.yg)("inlineCode",{parentName:"p"},"+")," operator, and to store the results in the ",(0,r.yg)("inlineCode",{parentName:"p"},"result")," variable.\nThis reduction uses threads to calculate the sum, similar to ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads.c"),", but in a much more optimised form."),(0,r.yg)("p",null,"Now compile and run the ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads_openmp")," binary using 1, 2, 4, and 8 threads as before.\nYou'll see lower running times than ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads")," due to the highly-optimised code emitted by the compiler.\nFor this reason and because library functions are usually much better tested than your own code, it is always preferred to use a library function for a given task."),(0,r.yg)("h3",{id:"array-sum-in-python"},"Array Sum in Python"),(0,r.yg)("p",null,"Let's first probe this by implementing two parallel versions of the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sum-array/support/python/sum_array_sequential.py"),".\nOne version should use threads and the other should use processes.\nRun each of them using 1, 2, 4, and 8 threads / processes respectively and compare the running times.\nNotice that the running times of the multithreaded implementation do not decrease.\nThis is because the GIL makes it so that those threads that you create essentially run sequentially."),(0,r.yg)("p",null,"The GIL also makes it so that individual Python instructions are atomic.\nRun the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/synchronization/drills/tasks/race-condition/support/python/race_condition.py"),".\nEvery time, ",(0,r.yg)("inlineCode",{parentName:"p"},"var")," will be 0 because the GIL doesn't allow the two threads to run in parallel and reach the critical section at the same time.\nThis means that the instructions ",(0,r.yg)("inlineCode",{parentName:"p"},"var += 1")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"var -= 1")," become atomic."),(0,r.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-sum-array-threads"},"this")," reading material."),(0,r.yg)("h2",{id:"task-wait-for-it"},"Task: Wait for It"),(0,r.yg)("p",null,"The process that spawns all the others and subsequently calls ",(0,r.yg)("inlineCode",{parentName:"p"},"waitpid")," to wait for them to finish can also get their return codes.\nUpdate the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/sum-array-bugs/support/seg-fault/sum_array_processes.c")," and modify the call to ",(0,r.yg)("inlineCode",{parentName:"p"},"waitpid")," to obtain and investigate this return code.\nDisplay an appropriate message if one of the child processes returns an error."),(0,r.yg)("p",null,"Remember to use the appropriate ",(0,r.yg)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},"macros")," for handling the ",(0,r.yg)("inlineCode",{parentName:"p"},"status")," variable that is modified by ",(0,r.yg)("inlineCode",{parentName:"p"},"waitpid()"),", as it is a bit-field.\nWhen a process runs into a system error, it receives a signal.\nA signal is a means to interrupt the normal execution of a program from the outside.\nIt is associated with a number.\nUse ",(0,r.yg)("inlineCode",{parentName:"p"},"kill -l")," to find the full list of signals."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/seg-fault-exit-code"},"Quiz")),(0,r.yg)("p",null,"So up to this point we've seen that one advantage of processes is that they offer better safety than threads.\nBecause they use separate virtual address spaces, sibling processes are better isolated than threads.\nThus, an application that uses processes can be more robust to errors than if it were using threads."),(0,r.yg)("h3",{id:"memory-corruption"},"Memory Corruption"),(0,r.yg)("p",null,"Because they share the same address space, threads run the risk of corrupting each other's data.\nTake a look at the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"sum-array-bugs/support/memory-corruption/python/"),".\nThe two programs only differ in how they spread their workload.\nOne uses threads while the other uses processes."),(0,r.yg)("p",null,"Run both programs with and without memory corruption.\nPass any value as a third argument to trigger the corruption."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sum-array-bugs/support/memory-corruption/python$ python3 memory_corruption_processes.py # no memory corruption\n[...]\n\nstudent@os:~/.../sum-array-bugs/support/memory-corruption/python$ python3 memory_corruption_processes.py 1 # do memory corruption\n[...]\n")),(0,r.yg)("p",null,"The one using threads will most likely print a negative sum, while the other displays the correct sum.\nThis happens because all threads refer to the same memory for the array ",(0,r.yg)("inlineCode",{parentName:"p"},"arr"),".\nWhat happens to the processes is a bit more complicated."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab7#copy-on-write"},"Later in this lab"),", we will see that initially, the page tables of all processes point to the same physical frames or ",(0,r.yg)("inlineCode",{parentName:"p"},"arr"),".\nWhen the malicious process tries to corrupt this array by ",(0,r.yg)("strong",{parentName:"p"},"writing data to it"),", the OS duplicates the original frames of ",(0,r.yg)("inlineCode",{parentName:"p"},"arr")," so that the malicious process writes the corrupted values to these new frames, while leaving the original ones untouched.\nThis mechanism is called ",(0,r.yg)("strong",{parentName:"p"},"Copy-on-Write")," and is an OS optimisation so that memory is shared between the parent and the child process, until one of them attempts to write to it.\nAt this point, this process receives its own separate copies of the previously shared frames."),(0,r.yg)("p",null,"Note that in order for the processes to share the ",(0,r.yg)("inlineCode",{parentName:"p"},"sums")," dictionary, it is not created as a regular dictionary, but using the ",(0,r.yg)("inlineCode",{parentName:"p"},"Manager")," module.\nThis module provides some special data structures that are allocated in ",(0,r.yg)("strong",{parentName:"p"},"shared memory")," so that all processes can access them.\nYou can learn more about shared memory and its various implementations ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab7#task-shared-memory"},"in this section"),"."),(0,r.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-sum-array-processes"},"this")," reading material."),(0,r.yg)("h2",{id:"hardware-perspective"},"Hardware Perspective"),(0,r.yg)("p",null,"The main criterion we use to rank CPUs is their ",(0,r.yg)("em",{parentName:"p"},"computation power"),", i.e. their ability to crunch numbers and do math.\nNumerous benchmarks exist out there, and they are publicly displayed on sites such as ",(0,r.yg)("a",{parentName:"p",href:"https://www.cpubenchmark.net/"},"CPUBenchmark"),"."),(0,r.yg)("p",null,"For example, a benchmark can measure the performance of the computer's CPU in a variety of scenarios:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"its ability to perform integer operations"),(0,r.yg)("li",{parentName:"ul"},"its speed in floating point arithmetic"),(0,r.yg)("li",{parentName:"ul"},"data encryption and compression"),(0,r.yg)("li",{parentName:"ul"},"sorting algorithms and others")),(0,r.yg)("p",null,"You can take a look at what exactly is measured using ",(0,r.yg)("a",{parentName:"p",href:"https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+Threadripper+PRO+5995WX"},"this link"),".\nIt displays the scores obtained by a high-end CPU.\nApart from the tests above, other benchmarks might focus on different performance metrics, such as branch prediction or prefetching."),(0,r.yg)("p",null,"Other approaches are less artificial, measuring performance on real-world applications such as compile times and performance in the latest (and most resource-demanding) video games.\nThe latter metric revolves around how many average FPS (frames per second) a given CPU is able to crank out in a specific video game.\nYou can find a lot of articles online on how CPU benchmarking is done."),(0,r.yg)("p",null,"Most benchmarks, unfortunately, are not open source, especially the more popular ones, such as ",(0,r.yg)("a",{parentName:"p",href:"https://browser.geekbench.com/processor-benchmarks"},"Geekbench 5"),".\nDespite this shortcoming, benchmarks are widely used to compare the performance of various computer ",(0,r.yg)("strong",{parentName:"p"},"hardware"),", CPUs included."),(0,r.yg)("h3",{id:"the-role-of-the-operating-system"},"The Role of the Operating System"),(0,r.yg)("p",null,'As you\'ve seen so far, the CPU provides the "muscle" required for fast computation, i.e. the highly optimised hardware and multiple ALUs, FPUs\nand cores necessary to perform those computations.\nHowever, it is the ',(0,r.yg)("strong",{parentName:"p"},"operating system"),' that provides the "brains" for this computation.\nSpecifically, modern CPUs have the capacity to run multiple tasks in parallel.\nBut they do not provide a means to decide which task to run at each moment.\nThe OS comes as an ',(0,r.yg)("em",{parentName:"p"},"orchestrator")," to ",(0,r.yg)("strong",{parentName:"p"},"schedule")," the way these tasks (that we will later call threads) are allowed to run and use the CPU's resources.\nThis way, the OS tells the CPU what code to run on each CPU core so that it reaches a good balance between high throughput (running many instructions) and fair access to CPU cores."),(0,r.yg)("p",null,"It is cumbersome for a user-level application to interact directly with the CPU.\nThe developer would have to write hardware-specific code, which is not scalable and is difficult to maintain.\nIn addition, doing so would leave it up to the developer to isolate their application from the others that are present on the system.\nThis leaves applications vulnerable to countless bugs and exploits."),(0,r.yg)("p",null,"To guard apps from these pitfalls, the OS comes and mediates interactions between regular programs and the CPU by providing a set of ",(0,r.yg)("strong",{parentName:"p"},"abstractions"),".\nThese abstractions offer a safe, uniform and also isolated way to leverage the CPU's resources, i.e. its cores.\nThere are 2 main abstractions: ",(0,r.yg)("strong",{parentName:"p"},"processes")," and ",(0,r.yg)("strong",{parentName:"p"},"threads"),"."),(0,r.yg)("p",null,(0,r.yg)("img",{alt:"Interaction between applications, OS and CPU",src:a(7032).A})),(0,r.yg)("p",null,"As we can see from the image above, an application can spawn one or more processes.\nEach of these is handled and maintained by the OS.\nSimilarly, each process can spawn however many threads, which are also managed by the OS.\nThe OS decides when and on what CPU core to make each thread run.\nThis is in line with the general interaction between an application and the hardware: it is always mediated by the OS."),(0,r.yg)("h2",{id:"processes"},"Processes"),(0,r.yg)("p",null,"A process is simply a running program.\nLet's take the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," command as a trivial example.\n",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," is a ",(0,r.yg)("strong",{parentName:"p"},"program")," on your system.\nIt has a binary file which you can find and inspect with the help of the ",(0,r.yg)("inlineCode",{parentName:"p"},"which")," command:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ which ls\n/usr/bin/ls\n\nstudent@os:~$ file /usr/bin/ls\n/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e3da6f0bc36b6398b8651bbc2e08831a21a90da, for GNU/Linux 3.2.0, stripped\n")),(0,r.yg)("p",null,"When you run it, the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," binary stored ",(0,r.yg)("strong",{parentName:"p"},"on the disk")," at ",(0,r.yg)("inlineCode",{parentName:"p"},"/usr/bin/ls")," is read by another application called the ",(0,r.yg)("strong",{parentName:"p"},"loader"),".\nThe loader spawns a ",(0,r.yg)("strong",{parentName:"p"},"process")," by copying some of the contents ",(0,r.yg)("inlineCode",{parentName:"p"},"/usr/bin/ls")," in memory (such as the ",(0,r.yg)("inlineCode",{parentName:"p"},".text"),", ",(0,r.yg)("inlineCode",{parentName:"p"},".rodata")," and ",(0,r.yg)("inlineCode",{parentName:"p"},".data")," sections).\nUsing ",(0,r.yg)("inlineCode",{parentName:"p"},"strace"),", we can see the ",(0,r.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/execve.2.html"},(0,r.yg)("inlineCode",{parentName:"a"},"execve"))," system call:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace -s 100 ls -a # -s 100 limits strings to 100 bytes instead of the default 32\nexecve("/usr/bin/ls", ["ls", "-a"], 0x7fffa7e0d008 /* 61 vars */) = 0\n[...]\nwrite(1, ". .. content\\tCONTRIBUTING.md COPYING.md .git .gitignore README.md REVIEWING.md\\n", 86. .. content CONTRIBUTING.md COPYING.md .git .gitignore README.md REVIEWING.md\n) = 86\nclose(1) = 0\nclose(2) = 0\nexit_group(0) = ?\n+++ exited with 0 +++\n')),(0,r.yg)("p",null,"Look at its parameters:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"the path to the ",(0,r.yg)("strong",{parentName:"li"},"program"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"/usr/bin/ls")),(0,r.yg)("li",{parentName:"ul"},"the list of arguments: ",(0,r.yg)("inlineCode",{parentName:"li"},'"ls", "-a"')),(0,r.yg)("li",{parentName:"ul"},"the environment variables: the rest of the syscall's arguments")),(0,r.yg)("p",null,(0,r.yg)("inlineCode",{parentName:"p"},"execve")," invokes the loader to load the VAS of the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," process ",(0,r.yg)("strong",{parentName:"p"},"by replacing that of the existing process"),".\nAll subsequent syscalls are performed by the newly spawned ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," process.\nWe will get into more details regarding ",(0,r.yg)("inlineCode",{parentName:"p"},"execve")," ",(0,r.yg)("a",{parentName:"p",href:"/operating-systems/75/Compute/lab6#guide-system-dissected"},"towards the end of this lab"),"."),(0,r.yg)("p",null,(0,r.yg)("img",{alt:"Loading of `ls` Process",src:a(1286).A})),(0,r.yg)("h3",{id:"fork"},"Fork"),(0,r.yg)("p",null,"Up to now we've been creating processes using various high-level APIs, such as ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()"),", ",(0,r.yg)("inlineCode",{parentName:"p"},"Process()")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"system()"),".\nYes, despite being a C function, as you've seen from its man page, ",(0,r.yg)("inlineCode",{parentName:"p"},"system()")," itself calls 2 other functions: ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," to create a process and ",(0,r.yg)("inlineCode",{parentName:"p"},"execve()")," to execute the given command.\nAs you already know from the ",(0,r.yg)("a",{parentName:"p",href:"lab2.md#libraries-and-libc"},"Software Stack")," chapter, library functions may call one or more underlying system calls or other functions.\nNow we will move one step lower on the call stack and call ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," ourselves."),(0,r.yg)("p",null,(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," creates one child process that is ",(0,r.yg)("em",{parentName:"p"},"almost")," identical to its parent.\nWe say that ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," returns ",(0,r.yg)("strong",{parentName:"p"},"twice"),": once in the parent process and once more in the child process.\nThis means that after ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," returns, assuming no error has occurred, both the child and the parent resume execution from the same place: the instruction following the call to ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()"),".\nWhat's different between the two processes is the value returned by ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()"),":"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("strong",{parentName:"li"},"child process"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()")," returns 0"),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("strong",{parentName:"li"},"parent process"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()")," returns the PID of the child process (> 0)"),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("strong",{parentName:"li"},"on error"),": ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()")," returns -1, only once, in the initial process")),(0,r.yg)("p",null,"Therefore, the typical code for handling a ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," is available in ",(0,r.yg)("inlineCode",{parentName:"p"},"create-process/support/fork.c"),".\nTake a look at it and then run it.\nNotice what each of the two processes prints:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"the PID of the child is also known by the parent"),(0,r.yg)("li",{parentName:"ul"},"the PPID of the child is the PID of the parent")),(0,r.yg)("p",null,"Unlike ",(0,r.yg)("inlineCode",{parentName:"p"},"system()"),", who also waits for its child, when using ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," we must do the waiting ourselves.\nIn order to wait for a process to end, we use the ",(0,r.yg)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},(0,r.yg)("inlineCode",{parentName:"a"},"waitpid()"))," syscall.\nIt places the exit code of the child process in the ",(0,r.yg)("inlineCode",{parentName:"p"},"status")," parameter.\nThis argument is actually a bit-field containing more information than merely the exit code.\nTo retrieve the exit code, we use the ",(0,r.yg)("inlineCode",{parentName:"p"},"WEXITSTATUS")," macro.\nKeep in mind that ",(0,r.yg)("inlineCode",{parentName:"p"},"WEXITSTATUS")," only makes sense if ",(0,r.yg)("inlineCode",{parentName:"p"},"WIFEXITED")," is true, i.e. if the child process finished on its own and wasn't killed by another one or by an illegal action (such as a segfault or illegal instruction) for example.\nOtherwise, ",(0,r.yg)("inlineCode",{parentName:"p"},"WEXITSTATUS")," will return something meaningless.\nYou can view the rest of the information stored in the ",(0,r.yg)("inlineCode",{parentName:"p"},"status")," bit-field ",(0,r.yg)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},"in the man page"),"."),(0,r.yg)("p",null,(0,r.yg)("strong",{parentName:"p"},"Moral of the story"),": Usually the execution flow is:"),(0,r.yg)("ol",null,(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"fork()"),", followed by")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"wait()")," (called by the parent)")),(0,r.yg)("li",{parentName:"ol"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"exit()"),", called by the child."))),(0,r.yg)("p",null,"The order of last 2 steps may be swapped."),(0,r.yg)("h2",{id:"threads"},"Threads"),(0,r.yg)("h3",{id:"threads-vs-processes"},"Threads vs Processes"),(0,r.yg)("p",null,"So why use the implementation that spawns more processes if it's slower than the one using threads?\nThe table below lists the differences between threads and processes.\nGenerally, if we only want to do some computing, we use threads.\nIf we need to drastically change the behaviour of the program, we need a new program altogether, or we need more than computing (e.g. communication on the network to create a computing cluster), we use processes."),(0,r.yg)("table",null,(0,r.yg)("thead",{parentName:"table"},(0,r.yg)("tr",{parentName:"thead"},(0,r.yg)("th",{parentName:"tr",align:"left"},"PROCESS"),(0,r.yg)("th",{parentName:"tr",align:"left"},"THREAD"))),(0,r.yg)("tbody",{parentName:"table"},(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"independent"),(0,r.yg)("td",{parentName:"tr",align:"left"},"part of a process")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"collection of threads"),(0,r.yg)("td",{parentName:"tr",align:"left"},"shares VAS with other threads")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"slower creation (new page table must be created)"),(0,r.yg)("td",{parentName:"tr",align:"left"},"faster creation")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"longer context switch duration (TLB must be flushed)"),(0,r.yg)("td",{parentName:"tr",align:"left"},"shorter context switch duration (part of the same process, so same TLB)")),(0,r.yg)("tr",{parentName:"tbody"},(0,r.yg)("td",{parentName:"tr",align:"left"},"ending means ending all threads"),(0,r.yg)("td",{parentName:"tr",align:"left"},"other threads continue when finished")))),(0,r.yg)("h4",{id:"safety"},"Safety"),(0,r.yg)("p",null,"Compile and run the two programs in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/sum-array-bugs/support/seg-fault/"),", first with 2 processes and threads and then with 4.\nThey do the same thing as before: compute the sum of the elements in an array, but with a twist: each of them contains a bug causing a segfault.\nNotice that ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads"),' doesn\'t print anything with 4 threads, but merely a "Segmentation fault" message.\nOn the other hand, ',(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_processes")," prints a sum and a running time, albeit different from the sums we've seen so far."),(0,r.yg)("p",null,"The reason is that signals such as ",(0,r.yg)("inlineCode",{parentName:"p"},"SIGSEGV"),", which is used when a segmentation fault happens affect the entire process that handles them.\nTherefore, when we split our workload between several threads and one of them causes an error such as a segfault, that error is going to terminate the entire process.\nThe same thing happens when we use processes instead of threads: one process causes an error, which gets it killed, but the other processes continue their work unhindered.\nThis is why we end up with a lower sum in the end: because one process died too early and didn't manage to write the partial sum it had computed to the ",(0,r.yg)("inlineCode",{parentName:"p"},"results")," array."),(0,r.yg)("h3",{id:"memory-layout-of-multithreaded-programs"},"Memory Layout of Multithreaded Programs"),(0,r.yg)("p",null,"When a new thread is created, a new stack is allocated for a thread.\nThe default stack size if ",(0,r.yg)("inlineCode",{parentName:"p"},"8 MB")," / ",(0,r.yg)("inlineCode",{parentName:"p"},"8192 KB"),":"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ulimit -s\n8192\n")),(0,r.yg)("p",null,"Enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/drills/tasks/multithreaded/support/")," directory to observe the update of the memory layout when creating new threads."),(0,r.yg)("p",null,"Build the ",(0,r.yg)("inlineCode",{parentName:"p"},"multithreaded")," executable:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../multithreaded/support$ make\n")),(0,r.yg)("p",null,"Start the program:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../multithreaded/support$ ./multithreaded\nPress key to start creating threads ...\n[...]\n")),(0,r.yg)("p",null,"And investigate it with ",(0,r.yg)("inlineCode",{parentName:"p"},"pmap")," on another console, while pressing a key to create new threads."),(0,r.yg)("p",null,"As you can see, there is a new ",(0,r.yg)("inlineCode",{parentName:"p"},"8192 KB")," area created for every thread, also increasing the total virtual size."),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/thread-memory"},"Quiz")),(0,r.yg)("h2",{id:"guide-baby-steps---python"},"Guide: Baby steps - Python"),(0,r.yg)("p",null,"Run the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/guides/create-process/support/popen.py"),".\nIt simply spawns a new process running the ",(0,r.yg)("inlineCode",{parentName:"p"},"ls")," command using ",(0,r.yg)("a",{parentName:"p",href:"https://docs.python.org/3/library/subprocess.html#subprocess.Popen"},(0,r.yg)("inlineCode",{parentName:"a"},"subprocess.Popen()")),".\nDo not worry about the huge list of arguments that ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()")," takes.\nThey are used for ",(0,r.yg)("strong",{parentName:"p"},"inter-process-communication"),".\nYou'll learn more about this in the ","[Application Interaction chapter]","."),(0,r.yg)("p",null,"Note that this usage of ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()")," is not entirely correct.\nYou'll discover why later, but for now focus on simply understanding how to use ",(0,r.yg)("inlineCode",{parentName:"p"},"Popen()")," on its own."),(0,r.yg)("p",null,"Now change the command to anything you want.\nAlso give it some arguments.\nFrom the outside, it's as if you were running these commands from the terminal."),(0,r.yg)("h2",{id:"guide-sum-array-processes"},"Guide: Sum Array Processes"),(0,r.yg)("h3",{id:"sum-of-the-elements-in-an-array"},"Sum of the Elements in an Array"),(0,r.yg)("p",null,"Let's assume we only have one process on our system, and that process knows how to add the numbers in an array.\nIt can use however many resources it wants, since there is no other process to contest it.\nIt would probably look like the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/processes/guides/sum-array-processes/support/c/sum_array_sequential.c"),".\nThe program also measures the time spent computing the sum.\nLet's compile and run it:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sum-array/support/c$ ./sum_array_sequential\nArray sum is: 49945994146\nTime spent: 127 ms\n")),(0,r.yg)("p",null,"You will most likely get a different sum (because the array is made up of random numbers) and a different time than the ones shown above.\nThis is perfectly fine.\nUse these examples qualitatively, not quantitatively."),(0,r.yg)("h3",{id:"spreading-the-work-among-other-processes"},"Spreading the Work Among Other Processes"),(0,r.yg)("p",null,"Due to how it's implemented so far, our program only uses one of our CPU's cores.\nWe never tell it to distribute its workload to other cores.\nThis is wasteful as the rest of our cores remain unused:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ lscpu | grep ^CPU\\(s\\):\nCPU(s): 8\n")),(0,r.yg)("p",null,"We have 7 more cores waiting to add numbers in our array."),(0,r.yg)("p",null,(0,r.yg)("img",{alt:"What if we used 100% of the CPU?",src:a(7846).A,width:"541",height:"500"})),(0,r.yg)("p",null,"What if we use 7 more processes and spread the task of adding the numbers in this array between them?\nIf we split the array into several equal parts and designate a separate process to calculate the sum of each part, we should get a speedup because now the work performed by each individual process is reduced."),(0,r.yg)("p",null,"Let's take it methodically.\nCompile and run ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_processes.c")," using 1, 2, 4 and 8 processes respectively.\nIf your system only has 4 cores (",(0,r.yg)("a",{parentName:"p",href:"https://www.intel.com/content/www/us/en/gaming/resources/hyper-threading.html"},"hyperthreading")," included), limit your runs to 4 processes.\nNote the running times for each number of processes.\nWe expect the speedups compared to our reference run to be 1, 2, 4 and 8 respectively, right?"),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/processes-speedup"},"Quiz")),(0,r.yg)("p",null,"You most likely did get some speedup, especially when using 8 processes.\nNow we will try to improve this speedup by using ",(0,r.yg)("strong",{parentName:"p"},"threads")," instead."),(0,r.yg)("p",null,"Also notice that we're not using hundreds or thousands of processes.\nAssuming our system has 8 cores, only 8 ",(0,r.yg)("em",{parentName:"p"},"threads")," can run at the same time.\nIn general, ",(0,r.yg)("strong",{parentName:"p"},"the maximum number of threads that can run at the same time is equal to the number of cores"),".\nIn our example, each process only has one thread: its main thread.\nSo by consequence and by forcing the terminology (because it's the main thread of these processes that is running, not the processes themselves), we can only run in parallel a number of processes equal to at most the number of cores."),(0,r.yg)("h2",{id:"guide-system-dissected"},"Guide: ",(0,r.yg)("inlineCode",{parentName:"h2"},"system")," Dissected"),(0,r.yg)("p",null,"You already know that ",(0,r.yg)("inlineCode",{parentName:"p"},"system")," calls ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"execve()")," to create the new process.\nLet's see how and why.\nFirst, we run the following command to trace the ",(0,r.yg)("inlineCode",{parentName:"p"},"execve()")," syscalls used by ",(0,r.yg)("inlineCode",{parentName:"p"},"sleepy_creator"),".\nWe'll leave ",(0,r.yg)("inlineCode",{parentName:"p"},"fork()")," for later."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sleepy/support$ strace -e execve -ff -o syscalls ./sleepy_creator\n")),(0,r.yg)("p",null,"At this point, you will get two files whose names start with ",(0,r.yg)("inlineCode",{parentName:"p"},"syscalls"),", followed by some numbers.\nThose numbers are the PIDs of the parent and the child process.\nTherefore, the file with the higher number contains logs of the ",(0,r.yg)("inlineCode",{parentName:"p"},"execve")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," syscalls issued by the parent process, while\nthe other logs those two syscalls when made by the child process.\nLet's take a look at them.\nThe numbers below will differ from those on your system:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../sleepy/support:$ cat syscalls.2523393 # syscalls from parent process\nexecve("sleepy_creator", ["sleepy_creator"], 0x7ffd2c157758 /* 39 vars */) = 0\n--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2523394, si_uid=1052093, si_status=0, si_utime=0, si_stime=0} ---\n+++ exited with 0 +++\n\nstudent@os:~/.../sleepy/support:$ cat syscalls.2523394 # syscalls from child process\nexecve("/bin/sh", ["sh", "-c", "sleep 10"], 0x7ffd36253be8 /* 39 vars */) = 0\nexecve("/usr/bin/sleep", ["sleep", "10"], 0x560f41659d40 /* 38 vars */) = 0\n+++ exited with 0 +++\n')),(0,r.yg)("p",null,(0,r.yg)("a",{parentName:"p",href:"Questions/who-calls-execve-parent"},"Quiz")),(0,r.yg)("p",null,"Now notice that the child process doesn't simply call ",(0,r.yg)("inlineCode",{parentName:"p"},'execve("/usr/bin/sleep" ...)'),".\nIt first changes its virtual address space (VAS) to that of a ",(0,r.yg)("inlineCode",{parentName:"p"},"bash")," process (",(0,r.yg)("inlineCode",{parentName:"p"},'execve("/bin/sh" ...)'),") and then that ",(0,r.yg)("inlineCode",{parentName:"p"},"bash")," process switches its VAS to ",(0,r.yg)("inlineCode",{parentName:"p"},"sleep"),".\nTherefore, calling ",(0,r.yg)("inlineCode",{parentName:"p"},"system()")," is equivalent to running ",(0,r.yg)("inlineCode",{parentName:"p"},"")," in the command-line."),(0,r.yg)("p",null,(0,r.yg)("strong",{parentName:"p"},"Moral of the story"),": When spawning a new command, the call order is:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"parent: ",(0,r.yg)("inlineCode",{parentName:"li"},"fork()"),", ",(0,r.yg)("inlineCode",{parentName:"li"},"exec()"),", ",(0,r.yg)("inlineCode",{parentName:"li"},"wait()")),(0,r.yg)("li",{parentName:"ul"},"child: ",(0,r.yg)("inlineCode",{parentName:"li"},"exit()"))),(0,r.yg)("h2",{id:"guide-sum-array-threads"},"Guide: Sum array Threads"),(0,r.yg)("h3",{id:"spreading-the-work-among-other-threads"},"Spreading the Work Among Other Threads"),(0,r.yg)("p",null,"Compile the code in ",(0,r.yg)("inlineCode",{parentName:"p"},"chapters/compute/threads/guides/sum-array-threads/support/c/sum_array_threads.c")," and run it using 1, 2, 4 and 8 threads as you did before.\nEach thread runs the ",(0,r.yg)("inlineCode",{parentName:"p"},"calculate_array_part_sum()")," function and then finishes.\nRunning times should be ",(0,r.yg)("em",{parentName:"p"},"slightly")," smaller than the implementation using processes.\nThis slight time difference is caused by process creation actions, which are costlier than thread creation actions.\nBecause a process needs a separate virtual address space (VAS) and needs to duplicate some internal structures such as the file descriptor table and page table, it takes the operating system more time to create it than to create a thread.\nOn the other hand, threads belonging to the same process share the same VAS and, implicitly, the same OS-internal structures.\nTherefore, they are more lightweight than processes."),(0,r.yg)("h2",{id:"guide-threads-and-processes-clone"},"Guide: Threads and Processes: ",(0,r.yg)("inlineCode",{parentName:"h2"},"clone")),(0,r.yg)("p",null,"Let's go back to our initial demos that used threads and processes.\nWe'll see that in order to create both threads and processes, the underlying Linux syscall is ",(0,r.yg)("inlineCode",{parentName:"p"},"clone"),".\nFor this, we'll run both ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_threads")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"sum_array_processes")," under ",(0,r.yg)("inlineCode",{parentName:"p"},"strace"),".\nAs we've already established, we're only interested in the ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," syscall:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../sum-array/support/c$ strace -e clone,clone3 ./sum_array_threads 2\nclone(child_stack=0x7f60b56482b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819693], tls=0x7f60b5649640, child_tidptr=0x7f60b5649910) = 1819693\nclone(child_stack=0x7f60b4e472b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819694], tls=0x7f60b4e48640, child_tidptr=0x7f60b4e48910) = 1819694\n\nstudent@os:~/.../sum-array/support/c$ strace -e clone,clone3 ./sum_array_processes 2\nclone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820599\nclone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820600\n")),(0,r.yg)("p",null,"We ran each program with an argument of 2, so we have 2 calls to ",(0,r.yg)("inlineCode",{parentName:"p"},"clone"),".\nNotice that in the case of threads, the ",(0,r.yg)("inlineCode",{parentName:"p"},"clone3")," syscall receives more arguments.\nThe relevant flags passed as arguments when creating threads are documented in ",(0,r.yg)("a",{parentName:"p",href:"https://man.archlinux.org/man/clone3.2.en"},(0,r.yg)("inlineCode",{parentName:"a"},"clone"),"'s man page"),":"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"CLONE_VM"),": the child and the parent process share the same VAS"),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"CLONE_{FS,FILES,SIGHAND}"),": the new thread shares the filesystem information, file and signal handlers with the one that created it.\nThe syscall also receives valid pointers to the new thread's stack and TLS, i.e. the only parts of the VAS that are distinct between threads (although they are technically accessible from all threads).")),(0,r.yg)("p",null,"By contrast, when creating a new process, the arguments of the ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," syscall are simpler (i.e. fewer flags are present).\nRemember that in both cases ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," creates a new ",(0,r.yg)("strong",{parentName:"p"},"thread"),".\nWhen creating a process, ",(0,r.yg)("inlineCode",{parentName:"p"},"clone")," creates this new thread within a new separate address space."))}c.isMDXComponent=!0},7846:(e,t,a)=>{a.d(t,{A:()=>n});const n=a.p+"assets/images/100-percent-cpu-1138186529f154d864f643179e25cea1.jpeg"},7032:(e,t,a)=>{a.d(t,{A:()=>n});const n=a.p+"assets/images/app-os-cpu-interaction-ca7fbdbb7da380e0992c95467ef267ce.svg"},1286:(e,t,a)=>{a.d(t,{A:()=>n});const n=a.p+"assets/images/loading-of-ls-process-0dec67c0d0a826710e06f980224d5eb4.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/88f87d1c.1d24f41e.js b/75/assets/js/88f87d1c.1d24f41e.js new file mode 100644 index 0000000000..c88a2b71c6 --- /dev/null +++ b/75/assets/js/88f87d1c.1d24f41e.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8224],{5680:(e,t,n)=>{n.d(t,{xA:()=>d,yg:()=>h});var a=n(6540);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function s(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var o=a.createContext({}),p=function(e){var t=a.useContext(o),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},d=function(e){var t=p(e.components);return a.createElement(o.Provider,{value:t},e.children)},c="mdxType",g={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,o=e.parentName,d=l(e,["components","mdxType","originalType","parentName"]),c=p(n),u=r,h=c["".concat(o,".").concat(u)]||c[u]||g[u]||i;return n?a.createElement(h,s(s({ref:t},d),{},{components:n})):a.createElement(h,s({ref:t},d))}));function h(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,s=new Array(i);s[0]=u;var l={};for(var o in t)hasOwnProperty.call(t,o)&&(l[o]=t[o]);l.originalType=e,l[c]="string"==typeof e?e:r,s[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>o,contentTitle:()=>s,default:()=>g,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var a=n(8168),r=(n(6540),n(5680));const i={},s="Parallel Firewall",l={unversionedId:"Assignments/Parallel Firewall/README",id:"Assignments/Parallel Firewall/README",title:"Parallel Firewall",description:"Objectives",source:"@site/docs/Assignments/Parallel Firewall/README.md",sourceDirName:"Assignments/Parallel Firewall",slug:"/Assignments/Parallel Firewall/",permalink:"/operating-systems/75/Assignments/Parallel Firewall/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Memory Allocator",permalink:"/operating-systems/75/Assignments/Memory Allocator/"},next:{title:"Minishell",permalink:"/operating-systems/75/Assignments/Mini Shell/"}},o={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Support Code",id:"support-code",level:2},{value:"Implementation",id:"implementation",level:2},{value:"Firewall Threads",id:"firewall-threads",level:3},{value:"Ring Buffers",id:"ring-buffers",level:3},{value:"Log File",id:"log-file",level:3},{value:"Operations",id:"operations",level:2},{value:"Building",id:"building",level:3},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Restrictions",id:"restrictions",level:3},{value:"Grades",id:"grades",level:3},{value:"Running the Checker",id:"running-the-checker",level:3},{value:"Running the Linters",id:"running-the-linters",level:3},{value:"Fine-Grained Testing",id:"fine-grained-testing",level:3}],d={toc:p},c="wrapper";function g(e){let{components:t,...n}=e;return(0,r.yg)(c,(0,a.A)({},d,n,{components:t,mdxType:"MDXLayout"}),(0,r.yg)("h1",{id:"parallel-firewall"},"Parallel Firewall"),(0,r.yg)("h2",{id:"objectives"},"Objectives"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"Learn how to design and implement parallel programs"),(0,r.yg)("li",{parentName:"ul"},"Get experienced at utilizing the POSIX threading API"),(0,r.yg)("li",{parentName:"ul"},"Learn how to convert a serial program into a parallel one")),(0,r.yg)("h2",{id:"statement"},"Statement"),(0,r.yg)("p",null,"A firewall is a program that checks network packets against a series of filters which provide a decision regarding dropping or allowing the packets to continue to their intended destination."),(0,r.yg)("p",null,(0,r.yg)("strong",{parentName:"p"},"In a real setup"),", the network card will receive real packets (e.g. packets having ",(0,r.yg)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/Ethernet_frame"},(0,r.yg)("inlineCode",{parentName:"a"},"Ethernet")),", ",(0,r.yg)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/IPv4"},(0,r.yg)("inlineCode",{parentName:"a"},"IP"))," headers plus payload) from the network and will send them to the firewall for processing.\nThe firewall will decide if the packets are to be dropped or not and, if not, passes them further."),(0,r.yg)("p",null,(0,r.yg)("strong",{parentName:"p"},"In this assignment"),", instead of real network packets, we'll deal with made up packets consisting of a made up source (a number), a made up destination (also a number), a timestamp (also a number) and some payload.\nAnd instead of the network card providing the packets, we'll have a ",(0,r.yg)("strong",{parentName:"p"},"producer thread")," creating these packets."),(0,r.yg)("p",null,"The created packets will be inserted into a ",(0,r.yg)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/Circular_buffer"},(0,r.yg)("inlineCode",{parentName:"a"},"circular buffer")),", out of which ",(0,r.yg)("strong",{parentName:"p"},"consumer threads")," (which implement the firewall logic) will take packets and process them in order to decide whether they advance to the destination."),(0,r.yg)("p",null,"The result of this processing is a log file in which the firewall will record the decision taken (PASS or DROP) for each packet, along with other information such as timestamp."),(0,r.yg)("p",null,"The purpose of this assignment is to:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("p",{parentName:"li"},"implement the circular buffer, along with synchronization mechanisms for it to work in a multithreaded program")),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("p",{parentName:"li"},"implement the consumer threads, which consume packets and process them")),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("p",{parentName:"li"},"provide the log file containing the result of the packet processing"))),(0,r.yg)("h2",{id:"support-code"},"Support Code"),(0,r.yg)("p",null,"The support code consists of the directories:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"src/")," contains the skeleton for the parallelized firewall and the already implemented serial code in ",(0,r.yg)("inlineCode",{parentName:"p"},"src/serial.c"),".\nYou will have to implement the missing parts marked as ",(0,r.yg)("inlineCode",{parentName:"p"},"TODO"))),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"utils/")," contains utility files used for debugging and logging.")),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("p",{parentName:"li"},(0,r.yg)("inlineCode",{parentName:"p"},"tests/")," contains tests used to validate and grade the assignment."))),(0,r.yg)("h2",{id:"implementation"},"Implementation"),(0,r.yg)("h3",{id:"firewall-threads"},"Firewall Threads"),(0,r.yg)("p",null,"In order to parallelize the firewall we have to distribute the packets to multiple threads.\nThe packets will be added to a shared data structure (visible to all threads) by a ",(0,r.yg)("inlineCode",{parentName:"p"},"producer")," thread and processed by multiple ",(0,r.yg)("inlineCode",{parentName:"p"},"consumer")," threads.\nEach ",(0,r.yg)("inlineCode",{parentName:"p"},"consumer")," thread picks a packet from the shared data structure, checks it against the filter function and writes the packet hash together with the drop/accept decision to a log file.\n",(0,r.yg)("inlineCode",{parentName:"p"},"consumer")," threads stop waiting for new packets from the ",(0,r.yg)("inlineCode",{parentName:"p"},"producer")," thread and exit when the ",(0,r.yg)("inlineCode",{parentName:"p"},"producer")," thread closes the connection to the shared data structure."),(0,r.yg)("p",null,"The ",(0,r.yg)("inlineCode",{parentName:"p"},"consumer")," threads ",(0,r.yg)("strong",{parentName:"p"},"must not do any form of busy waiting"),".\nWhen there are new packets that need to be handled, the ",(0,r.yg)("inlineCode",{parentName:"p"},"consumer")," threads must be ",(0,r.yg)("strong",{parentName:"p"},"notified"),".\n",(0,r.yg)("strong",{parentName:"p"},"Waiting in a ",(0,r.yg)("inlineCode",{parentName:"strong"},"while()")," loop or sleeping is not considered a valid synchronization mechanism and points will be deducted.")),(0,r.yg)("p",null,"Implement the ",(0,r.yg)("inlineCode",{parentName:"p"},"consumer")," related functions marked with ",(0,r.yg)("inlineCode",{parentName:"p"},"TODO")," in the ",(0,r.yg)("inlineCode",{parentName:"p"},"src/consumer.c")," file.\n",(0,r.yg)("strong",{parentName:"p"},"The number of consumer threads will be passed as the 3rd command-line argument")),(0,r.yg)("h3",{id:"ring-buffers"},"Ring Buffers"),(0,r.yg)("p",null,"A ring buffer (or a circular buffer) is a data structure that stores its elements in a circular fixed size array.\nOne of the advantages of using such a data structure as opposed to an array is that it acts as a FIFO, without the overhead of moving the elements to the left as they are consumed.\nThus, the shared ring buffer offers the following fields:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"write_pos")," index in the buffer used by the ",(0,r.yg)("inlineCode",{parentName:"li"},"producer")," thread for appending new packets."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"read_pos")," index in the buffer used by the ",(0,r.yg)("inlineCode",{parentName:"li"},"consumer")," threads to pick packets."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"cap")," the size of the internal buffer."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"data")," pointer to the internal buffer.")),(0,r.yg)("p",null,"Apart from these fields you have to add synchronization primitives in order to allow multiple threads to access the ring buffer in a deterministic manner.\nYou can use mutexes, semaphores, conditional variables and other synchronization mechanisms offered by the ",(0,r.yg)("inlineCode",{parentName:"p"},"pthread")," library."),(0,r.yg)("p",null,"You will have to implement the following interface for the ring buffer:"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"ring_buffer_init()"),": initialize the ring buffer (allocate memory and synchronization primitives)."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"ring_buffer_enqueue()"),": add elements to the ring buffer."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"ring_buffer_dequeue()"),": remove elements from the ring buffer."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"ring_buffer_destroy()"),": free up the memory used by the ring_buffer."),(0,r.yg)("li",{parentName:"ul"},(0,r.yg)("inlineCode",{parentName:"li"},"ring_buffer_stop()"),": finish up using the ring buffer for the calling thread.")),(0,r.yg)("h3",{id:"log-file"},"Log File"),(0,r.yg)("p",null,"The output of the firewall will be a log file with the rows containing the firewall's decision, the hash of the packet and its timestamp.\nThe actual format can be found in the serial implementation (at ",(0,r.yg)("inlineCode",{parentName:"p"},"src/serial.c"),")."),(0,r.yg)("p",null,"When processing the packets in parallel the threads will finish up the work in a non deterministic order.\nThe packet processing functions are already implemented in ",(0,r.yg)("inlineCode",{parentName:"p"},"src/packet.c")),(0,r.yg)("p",null,"We would like the logs to be sorted by the packet timestamp, the order that they came in from the producer.\nThus, the ",(0,r.yg)("inlineCode",{parentName:"p"},"consumers")," should insert the packet information to the log file such as the result is ordered by timestamp.\nThe printing format can be found in ",(0,r.yg)("inlineCode",{parentName:"p"},"./src/serial.c")),(0,r.yg)("p",null,"The logs must be written to the file in ascending order during packet processing.\n",(0,r.yg)("strong",{parentName:"p"},"Sorting the log file after the consumer threads have finished processing is not considered a valid synchronization mechanism and points will be deducted.")),(0,r.yg)("h2",{id:"operations"},"Operations"),(0,r.yg)("h3",{id:"building"},"Building"),(0,r.yg)("p",null,"To build both the serial and the parallel versions, run ",(0,r.yg)("inlineCode",{parentName:"p"},"make")," in the ",(0,r.yg)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/parallel-firewall$ cd src/\n\nstudent@so:~/.../assignments/parallel-firewall/src$ make\n")),(0,r.yg)("p",null,"That will create the ",(0,r.yg)("inlineCode",{parentName:"p"},"serial")," and ",(0,r.yg)("inlineCode",{parentName:"p"},"firewall")," binaries."),(0,r.yg)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,r.yg)("p",null,"Testing is automated.\nTests are located in the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,r.yg)("p",null,"To test and grade your assignment solution, enter the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/")," directory and run ",(0,r.yg)("inlineCode",{parentName:"p"},"grade.sh"),"."),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/parallel-firewall$ cd tests/\n")),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/parallel-firewall/tests$ ./grade.sh\n")),(0,r.yg)("p",null,"Note that this requires linters being available.\nThe easiest way to test the project is to use a Docker-based setup with everything installed and configured (see the ",(0,r.yg)("a",{parentName:"p",href:"README.checker.md"},"README.checker.md")," file for instructions)."),(0,r.yg)("p",null,"To create the tests, run:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/parallel-firewall/tests$ make check\n")),(0,r.yg)("p",null,"To remove the tests, run:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/parallel-firewall/tests$ make distclean\n")),(0,r.yg)("p",null,"When using ",(0,r.yg)("inlineCode",{parentName:"p"},"grade.sh")," you will get a maximum of 90/100 points for general correctness and a maximum of 10/100 points for coding style."),(0,r.yg)("h3",{id:"restrictions"},"Restrictions"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"Threads must yield the cpu when waiting for empty/full buffers i.e. not doing ",(0,r.yg)("inlineCode",{parentName:"li"},"busy waiting"),"."),(0,r.yg)("li",{parentName:"ul"},"The logs must be written as they are processed and not after the processing is done, in ascending order by the timestamp."),(0,r.yg)("li",{parentName:"ul"},"The number of running threads must be at least ",(0,r.yg)("inlineCode",{parentName:"li"},"num_consumers + 1"),", where ",(0,r.yg)("inlineCode",{parentName:"li"},"num_consumers")," is the 3rd command-line argument of the ",(0,r.yg)("inlineCode",{parentName:"li"},"firewall")," binary.")),(0,r.yg)("h3",{id:"grades"},"Grades"),(0,r.yg)("ul",null,(0,r.yg)("li",{parentName:"ul"},"10 points are awarded for a single consumer solution that also implements the ring buffer"),(0,r.yg)("li",{parentName:"ul"},"50 points are awarded for a multi consumer solution"),(0,r.yg)("li",{parentName:"ul"},"30 points are awarded for a multi consumer solution that writes the logs in the sorted manner (bearing in mind the above restrictions)")),(0,r.yg)("h3",{id:"running-the-checker"},"Running the Checker"),(0,r.yg)("p",null,"Each test is worth a number of points.\nThe maximum grade is ",(0,r.yg)("inlineCode",{parentName:"p"},"90"),"."),(0,r.yg)("p",null,"A successful run will show the output:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/parallel-firewall/tests$ make check\n[...]\nTest [ 10 packets, sort False, 1 thread ] ...................... passed ... 3\nTest [ 1,000 packets, sort False, 1 thread ] ...................... passed ... 3\nTest [20,000 packets, sort False, 1 thread ] ...................... passed ... 4\nTest [ 10 packets, sort True , 2 threads] ...................... passed ... 5\nTest [ 10 packets, sort True , 4 threads] ...................... passed ... 5\nTest [ 100 packets, sort True , 2 threads] ...................... passed ... 5\nTest [ 100 packets, sort True , 4 threads] ...................... passed ... 5\nTest [ 1,000 packets, sort True , 2 threads] ...................... passed ... 5\nTest [ 1,000 packets, sort True , 4 threads] ...................... passed ... 5\nTest [10,000 packets, sort True , 2 threads] ...................... passed ... 5\nTest [10,000 packets, sort True , 4 threads] ...................... passed ... 5\nTest [20,000 packets, sort True , 2 threads] ...................... passed ... 5\nTest [20,000 packets, sort True , 4 threads] ...................... passed ... 5\nTest [ 1,000 packets, sort False, 4 threads] ...................... passed ... 5\nTest [ 1,000 packets, sort False, 8 threads] ...................... passed ... 5\nTest [10,000 packets, sort False, 4 threads] ...................... passed ... 5\nTest [10,000 packets, sort False, 8 threads] ...................... passed ... 5\nTest [20,000 packets, sort False, 4 threads] ...................... passed ... 5\nTest [20,000 packets, sort False, 8 threads] ...................... passed ... 5\n\nChecker: 90/100\n")),(0,r.yg)("h3",{id:"running-the-linters"},"Running the Linters"),(0,r.yg)("p",null,"To run the linters, use the ",(0,r.yg)("inlineCode",{parentName:"p"},"make lint")," command in the ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/parallel-firewall/tests$ make lint\n[...]\ncd .. && checkpatch.pl -f checker/*.sh tests/*.sh\n[...]\ncd .. && cpplint --recursive src/ tests/ checker/\n[...]\ncd .. && shellcheck checker/*.sh tests/*.sh\n")),(0,r.yg)("p",null,"Note that the linters have to be installed on your system: ",(0,r.yg)("a",{parentName:"p",href:"https://.com/torvalds/linux/blob/master/scripts/checkpatch.pl"},(0,r.yg)("inlineCode",{parentName:"a"},"checkpatch.pl")),", ",(0,r.yg)("a",{parentName:"p",href:"https://github.com/cpplint/cpplint"},(0,r.yg)("inlineCode",{parentName:"a"},"cpplint")),", ",(0,r.yg)("a",{parentName:"p",href:"https://www.shellcheck.net/"},(0,r.yg)("inlineCode",{parentName:"a"},"shellcheck")),".\nThey also need to have certain configuration options.\nIt's easiest to run them in a Docker-based setup with everything configured."),(0,r.yg)("h3",{id:"fine-grained-testing"},"Fine-Grained Testing"),(0,r.yg)("p",null,"Input tests cases are located in ",(0,r.yg)("inlineCode",{parentName:"p"},"tests/in/")," and are generated by the checker.\nThe expected results are generated by the checker while running the serial implementation.\nIf you want to run a single test, use the below commands while in the ",(0,r.yg)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,r.yg)("pre",null,(0,r.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/parallel-firewall/src$ ./firewall ../tests/in/test_.in \n")),(0,r.yg)("p",null,"Results provided by the serial and parallel implementation must be the same for the test to successfully pass."))}g.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/75/assets/js/935f2afb.20a29b8d.js b/75/assets/js/935f2afb.20a29b8d.js deleted file mode 100644 index 8ba997de23..0000000000 --- a/75/assets/js/935f2afb.20a29b8d.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8581],{5610:e=>{e.exports=JSON.parse('{"pluginId":"default","version":"current","label":"Next","banner":null,"badge":false,"noIndex":false,"className":"docs-version-current","isLast":true,"docsSidebars":{"sidebar":[{"type":"link","label":"Introduction","href":"/operating-systems/75/","docId":"README"},{"type":"link","label":"Setting up the Lab Environment","href":"/operating-systems/75/lab-setup","docId":"lab-setup"},{"type":"category","label":"Software Stack","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/Software Stack/Software-Stack","docId":"Software Stack/Software-Stack"},{"type":"link","label":"Overview","href":"/operating-systems/75/Software Stack/software-stack-overview","docId":"Software Stack/software-stack-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Printf vs Write","href":"/operating-systems/75/Software Stack/Questions/printf-vs-write","docId":"Software Stack/Questions/printf-vs-write"},{"type":"link","label":"Syscall ID","href":"/operating-systems/75/Software Stack/Questions/syscall-id","docId":"Software Stack/Questions/syscall-id"},{"type":"link","label":"Python Tools","href":"/operating-systems/75/Software Stack/Questions/python-tools","docId":"Software Stack/Questions/python-tools"},{"type":"link","label":"Malloc","href":"/operating-systems/75/Software Stack/Questions/malloc","docId":"Software Stack/Questions/malloc"},{"type":"link","label":"Software","href":"/operating-systems/75/Software Stack/Questions/software","docId":"Software Stack/Questions/software"},{"type":"link","label":"Static Executables","href":"/operating-systems/75/Software Stack/Questions/static-executables","docId":"Software Stack/Questions/static-executables"},{"type":"link","label":"Syscall Numbers","href":"/operating-systems/75/Software Stack/Questions/syscall-numbers","docId":"Software Stack/Questions/syscall-numbers"},{"type":"link","label":"Printf Syscall","href":"/operating-systems/75/Software Stack/Questions/printf-syscall","docId":"Software Stack/Questions/printf-syscall"},{"type":"link","label":"Syscall Wrapper","href":"/operating-systems/75/Software Stack/Questions/syscall-wrapper","docId":"Software Stack/Questions/syscall-wrapper"},{"type":"link","label":"Syscall Tool","href":"/operating-systems/75/Software Stack/Questions/syscall-tool","docId":"Software Stack/Questions/syscall-tool"},{"type":"link","label":"Libcall Syscall","href":"/operating-systems/75/Software Stack/Questions/libcall-syscall","docId":"Software Stack/Questions/libcall-syscall"},{"type":"link","label":"Libc","href":"/operating-systems/75/Software Stack/Questions/libc","docId":"Software Stack/Questions/libc"},{"type":"link","label":"Dynamic Libraries","href":"/operating-systems/75/Software Stack/Questions/dynamic-libraries","docId":"Software Stack/Questions/dynamic-libraries"},{"type":"link","label":"Strcpy Syscall","href":"/operating-systems/75/Software Stack/Questions/strcpy-syscall","docId":"Software Stack/Questions/strcpy-syscall"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Software Stack/Questions/"},{"type":"link","label":"Lab 1 - Operating System Perspective","href":"/operating-systems/75/Software Stack/lab1","docId":"Software Stack/lab1"},{"type":"link","label":"Lab 2 - Library Perspective","href":"/operating-systems/75/Software Stack/lab2","docId":"Software Stack/lab2"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Software Stack/"},{"type":"category","label":"Data","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/Data/","docId":"Data/Data"},{"type":"link","label":"Overview","href":"/operating-systems/75/Data/data-overview","docId":"Data/data-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Memory Access","href":"/operating-systems/75/Data/Questions/memory-access","docId":"Data/Questions/memory-access"},{"type":"link","label":"Half page","href":"/operating-systems/75/Data/Questions/half-page","docId":"Data/Questions/half-page"},{"type":"link","label":"Malloc Brk","href":"/operating-systems/75/Data/Questions/malloc-brk","docId":"Data/Questions/malloc-brk"},{"type":"link","label":"Malloc Mmap","href":"/operating-systems/75/Data/Questions/malloc-mmap","docId":"Data/Questions/malloc-mmap"},{"type":"link","label":"Memory granularity","href":"/operating-systems/75/Data/Questions/memory-granularity","docId":"Data/Questions/memory-granularity"},{"type":"link","label":"Mmap file","href":"/operating-systems/75/Data/Questions/mmap-file","docId":"Data/Questions/mmap-file"},{"type":"link","label":"Page Allocation","href":"/operating-systems/75/Data/Questions/page-allocation","docId":"Data/Questions/page-allocation"},{"type":"link","label":"Operators","href":"/operating-systems/75/Data/Questions/operators","docId":"Data/Questions/operators"},{"type":"link","label":"Memory Leaks","href":"/operating-systems/75/Data/Questions/memory-leaks","docId":"Data/Questions/memory-leaks"},{"type":"link","label":"Valgrind Leaks","href":"/operating-systems/75/Data/Questions/valgrind-leaks","docId":"Data/Questions/valgrind-leaks"},{"type":"link","label":"Bypass Canary","href":"/operating-systems/75/Data/Questions/bypass-canary","docId":"Data/Questions/bypass-canary"},{"type":"link","label":"Memory Aslr","href":"/operating-systems/75/Data/Questions/memory-aslr","docId":"Data/Questions/memory-aslr"},{"type":"link","label":"Memory Regions Vars","href":"/operating-systems/75/Data/Questions/memory-regions-vars","docId":"Data/Questions/memory-regions-vars"},{"type":"link","label":"Memory Stack Protector","href":"/operating-systems/75/Data/Questions/memory-stack-protector","docId":"Data/Questions/memory-stack-protector"},{"type":"link","label":"Stack Layout","href":"/operating-systems/75/Data/Questions/stack-layout","docId":"Data/Questions/stack-layout"},{"type":"link","label":"String buff over","href":"/operating-systems/75/Data/Questions/string-buff-over","docId":"Data/Questions/string-buff-over"},{"type":"link","label":"String strcpy","href":"/operating-systems/75/Data/Questions/string-strcpy","docId":"Data/Questions/string-strcpy"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Data/Questions/"},{"type":"link","label":"Lab 3 - Memory","href":"/operating-systems/75/Data/lab3","docId":"Data/lab3"},{"type":"link","label":"Lab 4 - Investigate Memory","href":"/operating-systems/75/Data/lab4","docId":"Data/lab4"},{"type":"link","label":"Lab 5 - Memory Security","href":"/operating-systems/75/Data/lab5","docId":"Data/lab5"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Data/"},{"type":"category","label":"Compute","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/Compute/","docId":"Compute/Compute"},{"type":"link","label":"Overview","href":"/operating-systems/75/Compute/compute-overview","docId":"Compute/compute-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Parent Faults Before Fork","href":"/operating-systems/75/Compute/Questions/parent-faults-before-fork","docId":"Compute/Questions/parent-faults-before-fork"},{"type":"link","label":"Mmap Cow Flag","href":"/operating-systems/75/Compute/Questions/mmap-cow-flag","docId":"Compute/Questions/mmap-cow-flag"},{"type":"link","label":"Sections Always Shared","href":"/operating-systems/75/Compute/Questions/sections-always-shared","docId":"Compute/Questions/sections-always-shared"},{"type":"link","label":"Child Faults After Write","href":"/operating-systems/75/Compute/Questions/child-faults-after-write","docId":"Compute/Questions/child-faults-after-write"},{"type":"link","label":"Seg Fault Exit Code","href":"/operating-systems/75/Compute/Questions/seg-fault-exit-code","docId":"Compute/Questions/seg-fault-exit-code"},{"type":"link","label":"Threads Shared Data","href":"/operating-systems/75/Compute/Questions/threads-shared-data","docId":"Compute/Questions/threads-shared-data"},{"type":"link","label":"Thread Memory","href":"/operating-systems/75/Compute/Questions/thread-memory","docId":"Compute/Questions/thread-memory"},{"type":"link","label":"Apache2 Strace","href":"/operating-systems/75/Compute/Questions/apache2-strace","docId":"Compute/Questions/apache2-strace"},{"type":"link","label":"Mini Shell Stops After Command","href":"/operating-systems/75/Compute/Questions/mini-shell-stops-after-command","docId":"Compute/Questions/mini-shell-stops-after-command"},{"type":"link","label":"Cause of File Not Found Error","href":"/operating-systems/75/Compute/Questions/cause-of-file-not-found-error","docId":"Compute/Questions/cause-of-file-not-found-error"},{"type":"link","label":"Process Creation","href":"/operating-systems/75/Compute/Questions/process-creation","docId":"Compute/Questions/process-creation"},{"type":"link","label":"Who Calls Execve Parent","href":"/operating-systems/75/Compute/Questions/who-calls-execve-parent","docId":"Compute/Questions/who-calls-execve-parent"},{"type":"link","label":"Exec Without Fork","href":"/operating-systems/75/Compute/Questions/exec-without-fork","docId":"Compute/Questions/exec-without-fork"},{"type":"link","label":"Create Sleepy Process Ending","href":"/operating-systems/75/Compute/Questions/create-sleepy-process-ending","docId":"Compute/Questions/create-sleepy-process-ending"},{"type":"link","label":"Processes Speedup","href":"/operating-systems/75/Compute/Questions/processes-speedup","docId":"Compute/Questions/processes-speedup"},{"type":"link","label":"Parent of Sleep Processes","href":"/operating-systems/75/Compute/Questions/parent-of-sleep-processes","docId":"Compute/Questions/parent-of-sleep-processes"},{"type":"link","label":"Sleeping on a Fiber","href":"/operating-systems/75/Compute/Questions/sleeping-on-a-fiber","docId":"Compute/Questions/sleeping-on-a-fiber"},{"type":"link","label":"Fiber Strace","href":"/operating-systems/75/Compute/Questions/fiber-strace","docId":"Compute/Questions/fiber-strace"},{"type":"link","label":"Type of Scheduler in Libult","href":"/operating-systems/75/Compute/Questions/type-of-scheduler-in-libult","docId":"Compute/Questions/type-of-scheduler-in-libult"},{"type":"link","label":"Number of Running Ults","href":"/operating-systems/75/Compute/Questions/number-of-running-ults","docId":"Compute/Questions/number-of-running-ults"},{"type":"link","label":"Why Use Completed Queue","href":"/operating-systems/75/Compute/Questions/why-use-completed-queue","docId":"Compute/Questions/why-use-completed-queue"},{"type":"link","label":"Ult Thread IDs","href":"/operating-systems/75/Compute/Questions/ult-thread-ids","docId":"Compute/Questions/ult-thread-ids"},{"type":"link","label":"TCB Libult Unikraft","href":"/operating-systems/75/Compute/Questions/tcb-libult-unikraft","docId":"Compute/Questions/tcb-libult-unikraft"},{"type":"link","label":"Time Slice Value","href":"/operating-systems/75/Compute/Questions/time-slice-value","docId":"Compute/Questions/time-slice-value"},{"type":"link","label":"Number of Running Threads","href":"/operating-systems/75/Compute/Questions/number-of-running-threads","docId":"Compute/Questions/number-of-running-threads"},{"type":"link","label":"Notify Only With Mutex","href":"/operating-systems/75/Compute/Questions/notify-only-with-mutex","docId":"Compute/Questions/notify-only-with-mutex"},{"type":"link","label":"TLS Var Copies","href":"/operating-systems/75/Compute/Questions/tls-var-copies","docId":"Compute/Questions/tls-var-copies"},{"type":"link","label":"TLS Synchronization","href":"/operating-systems/75/Compute/Questions/tls-synchronization","docId":"Compute/Questions/tls-synchronization"},{"type":"link","label":"Semaphore Equivalent","href":"/operating-systems/75/Compute/Questions/semaphore-equivalent","docId":"Compute/Questions/semaphore-equivalent"},{"type":"link","label":"Coarse vs Granular Critical Section","href":"/operating-systems/75/Compute/Questions/coarse-vs-granular-critical-section","docId":"Compute/Questions/coarse-vs-granular-critical-section"},{"type":"link","label":"Not Race Condition","href":"/operating-systems/75/Compute/Questions/not-race-condition","docId":"Compute/Questions/not-race-condition"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Compute/Questions/"},{"type":"link","label":"Lab 6 - Multiprocess and Multithread","href":"/operating-systems/75/Compute/lab6","docId":"Compute/lab6"},{"type":"link","label":"Lab 7 - Copy-on-Write","href":"/operating-systems/75/Compute/lab7","docId":"Compute/lab7"},{"type":"link","label":"Lab 8 - Syncronization","href":"/operating-systems/75/Compute/lab8","docId":"Compute/lab8"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Compute/"},{"type":"category","label":"IO","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/IO/","docId":"IO/IO"},{"type":"link","label":"Overview","href":"/operating-systems/75/IO/io-overview","docId":"IO/io-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Bind Error Cause","href":"/operating-systems/75/IO/Questions/bind-error-cause","docId":"IO/Questions/bind-error-cause"},{"type":"link","label":"Anonymous Pipes Limitation","href":"/operating-systems/75/IO/Questions/anonymous-pipes-limitation","docId":"IO/Questions/anonymous-pipes-limitation"},{"type":"link","label":"Pipe Ends","href":"/operating-systems/75/IO/Questions/pipe-ends","docId":"IO/Questions/pipe-ends"},{"type":"link","label":"Receiver Socket FD","href":"/operating-systems/75/IO/Questions/receiver-socket-fd","docId":"IO/Questions/receiver-socket-fd"},{"type":"link","label":"Client Server Sender Receiver","href":"/operating-systems/75/IO/Questions/client-server-sender-receiver","docId":"IO/Questions/client-server-sender-receiver"},{"type":"link","label":"Firefox TCP UDP","href":"/operating-systems/75/IO/Questions/firefox-tcp-udp","docId":"IO/Questions/firefox-tcp-udp"},{"type":"link","label":"Server Copies","href":"/operating-systems/75/IO/Questions/server-copies","docId":"IO/Questions/server-copies"},{"type":"link","label":"Fewer Than 2 Copies","href":"/operating-systems/75/IO/Questions/fewer-than-2-copies","docId":"IO/Questions/fewer-than-2-copies"},{"type":"link","label":"Strace Printf","href":"/operating-systems/75/IO/Questions/strace-printf","docId":"IO/Questions/strace-printf"},{"type":"link","label":"Mmap Read Write Benchmark","href":"/operating-systems/75/IO/Questions/mmap-read-write-benchmark","docId":"IO/Questions/mmap-read-write-benchmark"},{"type":"link","label":"Syscalls CP","href":"/operating-systems/75/IO/Questions/syscalls-cp","docId":"IO/Questions/syscalls-cp"},{"type":"link","label":"File Handler C","href":"/operating-systems/75/IO/Questions/file-handler-c","docId":"IO/Questions/file-handler-c"},{"type":"link","label":"Flush Libc Buffer","href":"/operating-systems/75/IO/Questions/flush-libc-buffer","docId":"IO/Questions/flush-libc-buffer"},{"type":"link","label":"Stderr FD","href":"/operating-systems/75/IO/Questions/stderr-fd","docId":"IO/Questions/stderr-fd"},{"type":"link","label":"Local IO Errors","href":"/operating-systems/75/IO/Questions/local-io-errors","docId":"IO/Questions/local-io-errors"},{"type":"link","label":"Fopen Syscall","href":"/operating-systems/75/IO/Questions/fopen-syscall","docId":"IO/Questions/fopen-syscall"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/IO/Questions/"},{"type":"link","label":"Lab 9 - File Descriptors","href":"/operating-systems/75/IO/lab9","docId":"IO/lab9"},{"type":"link","label":"Lab 10 - Inter-Process Communication","href":"/operating-systems/75/IO/lab10","docId":"IO/lab10"},{"type":"link","label":"Lab 11 - IO Optimizations","href":"/operating-systems/75/IO/lab11","docId":"IO/lab11"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/IO/"},{"type":"category","label":"Lecture","items":[{"type":"link","label":"IO","href":"/operating-systems/75/Lecture/IO","docId":"Lecture/IO"},{"type":"link","label":"Application Interaction","href":"/operating-systems/75/Lecture/Application-Interaction","docId":"Lecture/Application-Interaction"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Lecture/"},{"type":"category","label":"Assignments","items":[{"type":"link","label":"Mini Libc","href":"/operating-systems/75/Assignments/Mini Libc/","docId":"Assignments/Mini Libc/README"},{"type":"link","label":"Memory Allocator","href":"/operating-systems/75/Assignments/Memory Allocator/","docId":"Assignments/Memory Allocator/README"},{"type":"link","label":"Parallel Graph","href":"/operating-systems/75/Assignments/Parallel Graph/","docId":"Assignments/Parallel Graph/README"},{"type":"link","label":"Mini Shell","href":"/operating-systems/75/Assignments/Mini Shell/","docId":"Assignments/Mini Shell/README"},{"type":"link","label":"Asynchronous Web Server","href":"/operating-systems/75/Assignments/Asynchronous Web Server/","docId":"Assignments/Asynchronous Web Server/README"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Assignments/"},{"type":"category","label":"Exams","items":[{"type":"category","label":"2024 Summer","items":[{"type":"link","label":"Syscall Tracing","href":"/operating-systems/75/Exams/2024 Summer/Syscall Tracing/syscall-tracing","docId":"Exams/2024 Summer/Syscall Tracing/syscall-tracing"},{"type":"link","label":"Sysinfo Library","href":"/operating-systems/75/Exams/2024 Summer/Sysinfo Library/sysinfo-library","docId":"Exams/2024 Summer/Sysinfo Library/sysinfo-library"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Exams/2024 Summer/"},{"type":"link","label":"Digital Forensics","href":"/operating-systems/75/Exams/Digital Forensics/digital-forensics","docId":"Exams/Digital Forensics/digital-forensics"},{"type":"link","label":"Distributed System","href":"/operating-systems/75/Exams/Distributed System/distributed-system","docId":"Exams/Distributed System/distributed-system"},{"type":"link","label":"Aggregator Application","href":"/operating-systems/75/Exams/Aggregator Application/aggregator-application","docId":"Exams/Aggregator Application/aggregator-application"},{"type":"link","label":"Benchmarking Application","href":"/operating-systems/75/Exams/Benchmarking Application/benchmarking-application","docId":"Exams/Benchmarking Application/benchmarking-application"},{"type":"link","label":"Database Application","href":"/operating-systems/75/Exams/Database Application/database-application","docId":"Exams/Database Application/database-application"},{"type":"link","label":"Application Investigator","href":"/operating-systems/75/Exams/Application Investigator/application-investigator","docId":"Exams/Application Investigator/application-investigator"},{"type":"link","label":"Network Performance Utility","href":"/operating-systems/75/Exams/Network Performance Utility/network-performance-utility","docId":"Exams/Network Performance Utility/network-performance-utility"},{"type":"link","label":"Nightly Builds System","href":"/operating-systems/75/Exams/Nightly Builds System/nightly-builds-system","docId":"Exams/Nightly Builds System/nightly-builds-system"},{"type":"link","label":"User-level Threading Library","href":"/operating-systems/75/Exams/User-level Threading Library/user-level-threading-library","docId":"Exams/User-level Threading Library/user-level-threading-library"},{"type":"link","label":"System Process Monitoring Tool","href":"/operating-systems/75/Exams/System Process Monitoring Tool/system-process-monitoring-tool","docId":"Exams/System Process Monitoring Tool/system-process-monitoring-tool"},{"type":"link","label":"Network Configurations Manager","href":"/operating-systems/75/Exams/Network Configurations Manager/network-configurations-manager","docId":"Exams/Network Configurations Manager/network-configurations-manager"},{"type":"link","label":"FaaS Application","href":"/operating-systems/75/Exams/FaaS Application/faas-application","docId":"Exams/FaaS Application/faas-application"},{"type":"link","label":"Industrial System","href":"/operating-systems/75/Exams/Industrial System/industrial-system","docId":"Exams/Industrial System/industrial-system"},{"type":"link","label":"Blockchain System","href":"/operating-systems/75/Exams/Blockchain System/blockchain-system","docId":"Exams/Blockchain System/blockchain-system"},{"type":"link","label":"Supervisor-type Service","href":"/operating-systems/75/Exams/Supervisor-type Service/supervisor-type-service","docId":"Exams/Supervisor-type Service/supervisor-type-service"},{"type":"link","label":"Header Analysis Application","href":"/operating-systems/75/Exams/Header Analysis Application/header-analysis-application","docId":"Exams/Header Analysis Application/header-analysis-application"},{"type":"link","label":"Extending an App Manager","href":"/operating-systems/75/Exams/Extending an App Manager/extending-an-app-manager","docId":"Exams/Extending an App Manager/extending-an-app-manager"},{"type":"link","label":"Backup System","href":"/operating-systems/75/Exams/Backup System/backup-system","docId":"Exams/Backup System/backup-system"},{"type":"link","label":"File Changes Notifier","href":"/operating-systems/75/Exams/File Changes Notifier/file-changes-notifier","docId":"Exams/File Changes Notifier/file-changes-notifier"},{"type":"link","label":"Intrusion Detection System","href":"/operating-systems/75/Exams/Intrusion Detection System/intrusion-detection-system","docId":"Exams/Intrusion Detection System/intrusion-detection-system"},{"type":"link","label":"Resource Monitor","href":"/operating-systems/75/Exams/Resource Monitor/resource-monitor","docId":"Exams/Resource Monitor/resource-monitor"},{"type":"link","label":"Library Warmer","href":"/operating-systems/75/Exams/Library Warmer/library-warmer","docId":"Exams/Library Warmer/library-warmer"},{"type":"link","label":"Web GUI 1","href":"/operating-systems/75/Exams/Web GUI 1/web-gui-1","docId":"Exams/Web GUI 1/web-gui-1"},{"type":"link","label":"Web GUI 2","href":"/operating-systems/75/Exams/Web GUI 2/web-gui-2","docId":"Exams/Web GUI 2/web-gui-2"},{"type":"link","label":"Memory Deduplication","href":"/operating-systems/75/Exams/Memory Deduplication/memory-deduplication","docId":"Exams/Memory Deduplication/memory-deduplication"},{"type":"link","label":"Cloud System","href":"/operating-systems/75/Exams/Cloud System/cloud-system","docId":"Exams/Cloud System/cloud-system"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Exams/"},{"type":"category","label":"Hackathons","items":[{"type":"link","label":"Lambda Function Loader","href":"/operating-systems/75/Hackathons/Lambda Function Loader/","docId":"Hackathons/Lambda Function Loader/README"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Hackathons/"},{"type":"link","label":"Rules and Grading","href":"/operating-systems/75/rules-and-grading","docId":"rules-and-grading"},{"type":"link","label":"Resources","href":"/operating-systems/75/resources","docId":"resources"}]},"docs":{"Assignments/Asynchronous Web Server/README":{"id":"Assignments/Asynchronous Web Server/README","title":"Asynchronous Web Server","description":"Objectives","sidebar":"sidebar"},"Assignments/Asynchronous Web Server/src/http-parser/README":{"id":"Assignments/Asynchronous Web Server/src/http-parser/README","title":"HTTP Parser","description":"This is a parser for HTTP messages written in C. It parses both requests and"},"Assignments/Memory Allocator/README":{"id":"Assignments/Memory Allocator/README","title":"Memory Allocator","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Libc/README":{"id":"Assignments/Mini Libc/README","title":"Mini-libc","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Shell/README":{"id":"Assignments/Mini Shell/README","title":"Minishell","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Shell/util/parser/README":{"id":"Assignments/Mini Shell/util/parser/README","title":"Parser","description":"The parser is made using Bison and Flex."},"Assignments/Parallel Graph/README":{"id":"Assignments/Parallel Graph/README","title":"Parallel Graph","description":"Objectives","sidebar":"sidebar"},"Compute/Compute":{"id":"Compute/Compute","title":"Compute","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Compute/compute-overview":{"id":"Compute/compute-overview","title":"Compute","description":"Contents","sidebar":"sidebar"},"Compute/lab6":{"id":"Compute/lab6","title":"Lab 6 - Multiprocess and Multithread","description":"Task: Creating a process","sidebar":"sidebar"},"Compute/lab7":{"id":"Compute/lab7","title":"Lab 7 - Copy-on-Write","description":"Task: Investigate apache2 Using strace","sidebar":"sidebar"},"Compute/lab8":{"id":"Compute/lab8","title":"Lab 8 - Syncronization","description":"Task: Race Conditions","sidebar":"sidebar"},"Compute/Questions/apache2-strace":{"id":"Compute/Questions/apache2-strace","title":"`apache2` Document Root","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/cause-of-file-not-found-error":{"id":"Compute/Questions/cause-of-file-not-found-error","title":"Cause of `FileNotFoundError`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/child-faults-after-write":{"id":"Compute/Questions/child-faults-after-write","title":"Child Faults After Write","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/coarse-vs-granular-critical-section":{"id":"Compute/Questions/coarse-vs-granular-critical-section","title":"Coarse vs Granular Critical Section","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/create-sleepy-process-ending":{"id":"Compute/Questions/create-sleepy-process-ending","title":"`create_sleepy` Process Ending","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/exec-without-fork":{"id":"Compute/Questions/exec-without-fork","title":"exec()` Without `fork()","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/fiber-strace":{"id":"Compute/Questions/fiber-strace","title":"Fiber Strace","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/mini-shell-stops-after-command":{"id":"Compute/Questions/mini-shell-stops-after-command","title":"Mini-shell Stops After Command","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/mmap-cow-flag":{"id":"Compute/Questions/mmap-cow-flag","title":"Copy-on-write Flag for `mmap()`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/not-race-condition":{"id":"Compute/Questions/not-race-condition","title":"Not Race Condition","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/notify-only-with-mutex":{"id":"Compute/Questions/notify-only-with-mutex","title":"Both Condition and Mutex","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/number-of-running-threads":{"id":"Compute/Questions/number-of-running-threads","title":"Number of Running Threads","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/number-of-running-ults":{"id":"Compute/Questions/number-of-running-ults","title":"Number of RUNNING User-Level Threads","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/parent-faults-before-fork":{"id":"Compute/Questions/parent-faults-before-fork","title":"Parent Faults before `fork()`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/parent-of-sleep-processes":{"id":"Compute/Questions/parent-of-sleep-processes","title":"Parent of `sleep` Processes","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/process-creation":{"id":"Compute/Questions/process-creation","title":"Process Creation","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/processes-speedup":{"id":"Compute/Questions/processes-speedup","title":"Processes Speedup","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/sections-always-shared":{"id":"Compute/Questions/sections-always-shared","title":"Always Shared Sections","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/seg-fault-exit-code":{"id":"Compute/Questions/seg-fault-exit-code","title":"Segfault Exit Code","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/semaphore-equivalent":{"id":"Compute/Questions/semaphore-equivalent","title":"Semaphore Equivalent","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/sleeping-on-a-fiber":{"id":"Compute/Questions/sleeping-on-a-fiber","title":"Sleeping on a Fiber","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/tcb-libult-unikraft":{"id":"Compute/Questions/tcb-libult-unikraft","title":"Similarities Between the TCBs of `libult` and Unikraft","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/thread-memory":{"id":"Compute/Questions/thread-memory","title":"Thread Memory","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/threads-shared-data":{"id":"Compute/Questions/threads-shared-data","title":"Threads Shared Data","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/time-slice-value":{"id":"Compute/Questions/time-slice-value","title":"Time Slice Value","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/tls-synchronization":{"id":"Compute/Questions/tls-synchronization","title":"TLS Synchronization","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/tls-var-copies":{"id":"Compute/Questions/tls-var-copies","title":"TLS `var` Copies","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/type-of-scheduler-in-libult":{"id":"Compute/Questions/type-of-scheduler-in-libult","title":"Type of Scheduler in `libult.so`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/ult-thread-ids":{"id":"Compute/Questions/ult-thread-ids","title":"ULT Thread IDs","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/who-calls-execve-parent":{"id":"Compute/Questions/who-calls-execve-parent","title":"Who Calls `execve` in the Log of the Parent Process?","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/why-use-completed-queue":{"id":"Compute/Questions/why-use-completed-queue","title":"The Need for a COMPLETED Queue","description":"Question Text","sidebar":"sidebar"},"Data/Data":{"id":"Data/Data","title":"Data","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Data/data-overview":{"id":"Data/data-overview","title":"Data","description":"Data represents information that is to be processed to produce a final result or more data.","sidebar":"sidebar"},"Data/lab3":{"id":"Data/lab3","title":"Lab 3 - Memory","description":"Task: Memory Access","sidebar":"sidebar"},"Data/lab4":{"id":"Data/lab4","title":"Lab 4 - Investigate Memory","description":"Task: Allocating and Deallocating Memory","sidebar":"sidebar"},"Data/lab5":{"id":"Data/lab5","title":"Lab 5 - Memory Security","description":"Task: Wild Pointer Arithmetic Info Leak","sidebar":"sidebar"},"Data/Questions/bypass-canary":{"id":"Data/Questions/bypass-canary","title":"Bypass Canary","description":"Question","sidebar":"sidebar"},"Data/Questions/half-page":{"id":"Data/Questions/half-page","title":"Half Page","description":"Question Text","sidebar":"sidebar"},"Data/Questions/malloc-brk":{"id":"Data/Questions/malloc-brk","title":"malloc()` and `brk()","description":"Question Text","sidebar":"sidebar"},"Data/Questions/malloc-mmap":{"id":"Data/Questions/malloc-mmap","title":"malloc()` and `mmap()","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-access":{"id":"Data/Questions/memory-access","title":"Modify String","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-aslr":{"id":"Data/Questions/memory-aslr","title":"ASLR","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-granularity":{"id":"Data/Questions/memory-granularity","title":"Memory Granularity","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-leaks":{"id":"Data/Questions/memory-leaks","title":"Memory Leaks","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-regions-vars":{"id":"Data/Questions/memory-regions-vars","title":"Variables in memory regions","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-stack-protector":{"id":"Data/Questions/memory-stack-protector","title":"Stack Protector","description":"Question Text","sidebar":"sidebar"},"Data/Questions/mmap-file":{"id":"Data/Questions/mmap-file","title":"`mmap()` file","description":"Question Text","sidebar":"sidebar"},"Data/Questions/operators":{"id":"Data/Questions/operators","title":"Operator Overloading","description":"Question Text","sidebar":"sidebar"},"Data/Questions/page-allocation":{"id":"Data/Questions/page-allocation","title":"Page Allocation","description":"Question Text","sidebar":"sidebar"},"Data/Questions/stack-layout":{"id":"Data/Questions/stack-layout","title":"Stack layout","description":"Question Text","sidebar":"sidebar"},"Data/Questions/string-buff-over":{"id":"Data/Questions/string-buff-over","title":"String Buffer Overflow","description":"Question Text","sidebar":"sidebar"},"Data/Questions/string-strcpy":{"id":"Data/Questions/string-strcpy","title":"Strcpy Buffer Overflow","description":"Question Text","sidebar":"sidebar"},"Data/Questions/valgrind-leaks":{"id":"Data/Questions/valgrind-leaks","title":"Valgrind Leaks","description":"Question Text","sidebar":"sidebar"},"Exams/2024 Summer/Syscall Tracing/syscall-tracing":{"id":"Exams/2024 Summer/Syscall Tracing/syscall-tracing","title":"Syscall Tracing","description":"Scenario","sidebar":"sidebar"},"Exams/2024 Summer/Sysinfo Library/sysinfo-library":{"id":"Exams/2024 Summer/Sysinfo Library/sysinfo-library","title":"Sysinfo Library","description":"Scenario","sidebar":"sidebar"},"Exams/Aggregator Application/aggregator-application":{"id":"Exams/Aggregator Application/aggregator-application","title":"Aggregator Application","description":"Scenario","sidebar":"sidebar"},"Exams/Application Investigator/application-investigator":{"id":"Exams/Application Investigator/application-investigator","title":"Application Investigator","description":"Scenario","sidebar":"sidebar"},"Exams/Backup System/backup-system":{"id":"Exams/Backup System/backup-system","title":"Backup System","description":"Scenario","sidebar":"sidebar"},"Exams/Benchmarking Application/benchmarking-application":{"id":"Exams/Benchmarking Application/benchmarking-application","title":"Benchmarking Application","description":"Scenario","sidebar":"sidebar"},"Exams/Blockchain System/blockchain-system":{"id":"Exams/Blockchain System/blockchain-system","title":"Blockchain System","description":"Scenario","sidebar":"sidebar"},"Exams/Cloud System/cloud-system":{"id":"Exams/Cloud System/cloud-system","title":"Cloud System","description":"Scenario","sidebar":"sidebar"},"Exams/Database Application/database-application":{"id":"Exams/Database Application/database-application","title":"Database Application","description":"Scenario","sidebar":"sidebar"},"Exams/Digital Forensics/digital-forensics":{"id":"Exams/Digital Forensics/digital-forensics","title":"Digital Forensics","description":"Scenario","sidebar":"sidebar"},"Exams/Distributed System/distributed-system":{"id":"Exams/Distributed System/distributed-system","title":"Distributed System","description":"Scenario","sidebar":"sidebar"},"Exams/Extending an App Manager/extending-an-app-manager":{"id":"Exams/Extending an App Manager/extending-an-app-manager","title":"Extending an App Manager","description":"Scenario","sidebar":"sidebar"},"Exams/FaaS Application/faas-application":{"id":"Exams/FaaS Application/faas-application","title":"FaaS Application","description":"Scenario","sidebar":"sidebar"},"Exams/File Changes Notifier/file-changes-notifier":{"id":"Exams/File Changes Notifier/file-changes-notifier","title":"File Changes Notifier","description":"Scenario","sidebar":"sidebar"},"Exams/Header Analysis Application/header-analysis-application":{"id":"Exams/Header Analysis Application/header-analysis-application","title":"Header Analysis Application","description":"Scenario","sidebar":"sidebar"},"Exams/Industrial System/industrial-system":{"id":"Exams/Industrial System/industrial-system","title":"Industrial System","description":"Scenario","sidebar":"sidebar"},"Exams/Intrusion Detection System/intrusion-detection-system":{"id":"Exams/Intrusion Detection System/intrusion-detection-system","title":"Intrusion Detection System","description":"Scenario","sidebar":"sidebar"},"Exams/Library Warmer/library-warmer":{"id":"Exams/Library Warmer/library-warmer","title":"Library Warmer","description":"Scenario","sidebar":"sidebar"},"Exams/Memory Deduplication/memory-deduplication":{"id":"Exams/Memory Deduplication/memory-deduplication","title":"Memory Deduplication","description":"Scenario","sidebar":"sidebar"},"Exams/Network Configurations Manager/network-configurations-manager":{"id":"Exams/Network Configurations Manager/network-configurations-manager","title":"Network Configurations Manager","description":"Scenario","sidebar":"sidebar"},"Exams/Network Performance Utility/network-performance-utility":{"id":"Exams/Network Performance Utility/network-performance-utility","title":"Network Performance Utility","description":"Scenario","sidebar":"sidebar"},"Exams/Nightly Builds System/nightly-builds-system":{"id":"Exams/Nightly Builds System/nightly-builds-system","title":"Nightly Builds System","description":"Scenario","sidebar":"sidebar"},"Exams/Resource Monitor/resource-monitor":{"id":"Exams/Resource Monitor/resource-monitor","title":"Resource Monitor","description":"Scenario","sidebar":"sidebar"},"Exams/Supervisor-type Service/supervisor-type-service":{"id":"Exams/Supervisor-type Service/supervisor-type-service","title":"Supervisor-type Service","description":"Scenario","sidebar":"sidebar"},"Exams/System Process Monitoring Tool/system-process-monitoring-tool":{"id":"Exams/System Process Monitoring Tool/system-process-monitoring-tool","title":"System Process Monitoring Tool","description":"Scenario","sidebar":"sidebar"},"Exams/User-level Threading Library/user-level-threading-library":{"id":"Exams/User-level Threading Library/user-level-threading-library","title":"User-level Threading Library","description":"Scenario","sidebar":"sidebar"},"Exams/Web GUI 1/web-gui-1":{"id":"Exams/Web GUI 1/web-gui-1","title":"Web GUI 1","description":"Scenario","sidebar":"sidebar"},"Exams/Web GUI 2/web-gui-2":{"id":"Exams/Web GUI 2/web-gui-2","title":"Web GUI 2","description":"Scenario","sidebar":"sidebar"},"Hackathons/Lambda Function Loader/README":{"id":"Hackathons/Lambda Function Loader/README","title":"Lambda Function Loader","description":"Application Development","sidebar":"sidebar"},"IO/IO":{"id":"IO/IO","title":"IO","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"IO/io-overview":{"id":"IO/io-overview","title":"I/O","description":"We know that a compute system is a collection of hardware and software that modifies data.","sidebar":"sidebar"},"IO/lab10":{"id":"IO/lab10","title":"Lab 10 - Inter-Process Communication","description":"Task: Named Pipes Communication","sidebar":"sidebar"},"IO/lab11":{"id":"IO/lab11","title":"Lab 11 - IO Optimizations","description":"Task: Ordered Client-Server Communication","sidebar":"sidebar"},"IO/lab9":{"id":"IO/lab9","title":"Lab 9 - File Descriptors","description":"Task: My cat","sidebar":"sidebar"},"IO/Questions/anonymous-pipes-limitation":{"id":"IO/Questions/anonymous-pipes-limitation","title":"Limitation of Anonymous Pipes","description":"Question Text","sidebar":"sidebar"},"IO/Questions/bind-error-cause":{"id":"IO/Questions/bind-error-cause","title":"Cause of `bind()` Error","description":"Question Text","sidebar":"sidebar"},"IO/Questions/client-server-sender-receiver":{"id":"IO/Questions/client-server-sender-receiver","title":"`sender.py` and `receiver.py` Client-Server Parallel","description":"Question Text","sidebar":"sidebar"},"IO/Questions/fewer-than-2-copies":{"id":"IO/Questions/fewer-than-2-copies","title":"Fewer than Two Copies","description":"Question Text","sidebar":"sidebar"},"IO/Questions/file-handler-c":{"id":"IO/Questions/file-handler-c","title":"File handler in C","description":"Question Text","sidebar":"sidebar"},"IO/Questions/firefox-tcp-udp":{"id":"IO/Questions/firefox-tcp-udp","title":"Firefox: TCP or UDP?","description":"Question Text","sidebar":"sidebar"},"IO/Questions/flush-libc-buffer":{"id":"IO/Questions/flush-libc-buffer","title":"Flush Libc Buffer","description":"Question Text","sidebar":"sidebar"},"IO/Questions/fopen-syscall":{"id":"IO/Questions/fopen-syscall","title":"Syscall Used by `fopen()`","description":"Question Text","sidebar":"sidebar"},"IO/Questions/local-io-errors":{"id":"IO/Questions/local-io-errors","title":"I/O Errors","description":"Question Text","sidebar":"sidebar"},"IO/Questions/mmap-read-write-benchmark":{"id":"IO/Questions/mmap-read-write-benchmark","title":"`mmap()` vs `read()` and `write()` Benchmark","description":"Question Text","sidebar":"sidebar"},"IO/Questions/pipe-ends":{"id":"IO/Questions/pipe-ends","title":"Pipe Ends","description":"Question Text","sidebar":"sidebar"},"IO/Questions/receiver-socket-fd":{"id":"IO/Questions/receiver-socket-fd","title":"Receiver Socked File Descriptor","description":"Question Text","sidebar":"sidebar"},"IO/Questions/server-copies":{"id":"IO/Questions/server-copies","title":"Client-Server Number of Copies","description":"Question Text","sidebar":"sidebar"},"IO/Questions/stderr-fd":{"id":"IO/Questions/stderr-fd","title":"File Descriptor of `stderr`","description":"Question Text","sidebar":"sidebar"},"IO/Questions/strace-printf":{"id":"IO/Questions/strace-printf","title":"`printf()` Under Strace","description":"Question Text","sidebar":"sidebar"},"IO/Questions/syscalls-cp":{"id":"IO/Questions/syscalls-cp","title":"Syscalls Used by `cp`","description":"Question Text","sidebar":"sidebar"},"lab-setup":{"id":"lab-setup","title":"Setting up the Lab Environment","description":"Prerequisites","sidebar":"sidebar"},"Lecture/Application-Interaction":{"id":"Lecture/Application-Interaction","title":"Application-Interaction","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Lecture/IO":{"id":"Lecture/IO","title":"IO","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"README":{"id":"README","title":"Operating Systems","description":"Welcome to the Operating Systems course, held by the National University of Science and Technology Politehnica Bucharest.","sidebar":"sidebar"},"resources":{"id":"resources","title":"Resources and Useful Links","description":"Need to Know","sidebar":"sidebar"},"rules-and-grading":{"id":"rules-and-grading","title":"Rules and Grading","description":"Grading","sidebar":"sidebar"},"Software Stack/lab1":{"id":"Software Stack/lab1","title":"Lab 1 - Operating System Perspective","description":"Task: System Calls","sidebar":"sidebar"},"Software Stack/lab2":{"id":"Software Stack/lab2","title":"Lab 2 - Library Perspective","description":"Task: Common Functions","sidebar":"sidebar"},"Software Stack/Questions/dynamic-libraries":{"id":"Software Stack/Questions/dynamic-libraries","title":"Dynamic Libraries","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/libc":{"id":"Software Stack/Questions/libc","title":"Syscall Tool","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/libcall-syscall":{"id":"Software Stack/Questions/libcall-syscall","title":"Libcall with Syscall","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/malloc":{"id":"Software Stack/Questions/malloc","title":"malloc()","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/printf-syscall":{"id":"Software Stack/Questions/printf-syscall","title":"printf() System Call","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/printf-vs-write":{"id":"Software Stack/Questions/printf-vs-write","title":"printf() vs write","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/python-tools":{"id":"Software Stack/Questions/python-tools","title":"Python Tools","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/software":{"id":"Software Stack/Questions/software","title":"Software Properties","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/static-executables":{"id":"Software Stack/Questions/static-executables","title":"Static Executables","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/strcpy-syscall":{"id":"Software Stack/Questions/strcpy-syscall","title":"strcpy() System Call","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-id":{"id":"Software Stack/Questions/syscall-id","title":"Syscall ID","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-numbers":{"id":"Software Stack/Questions/syscall-numbers","title":"Syscall Numbers","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-tool":{"id":"Software Stack/Questions/syscall-tool","title":"Syscall Tool","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-wrapper":{"id":"Software Stack/Questions/syscall-wrapper","title":"Syscall Wrappers","description":"Question Text","sidebar":"sidebar"},"Software Stack/Software-Stack":{"id":"Software Stack/Software-Stack","title":"Software-Stack","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Software Stack/software-stack-overview":{"id":"Software Stack/software-stack-overview","title":"Software Stack","description":"Software comprises of code and data that is loaded in memory and used by the CPU.","sidebar":"sidebar"}}}')}}]); \ No newline at end of file diff --git a/75/assets/js/935f2afb.e5312d42.js b/75/assets/js/935f2afb.e5312d42.js new file mode 100644 index 0000000000..cce3d4e1be --- /dev/null +++ b/75/assets/js/935f2afb.e5312d42.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8581],{5610:e=>{e.exports=JSON.parse('{"pluginId":"default","version":"current","label":"Next","banner":null,"badge":false,"noIndex":false,"className":"docs-version-current","isLast":true,"docsSidebars":{"sidebar":[{"type":"link","label":"Introduction","href":"/operating-systems/75/","docId":"README"},{"type":"link","label":"Setting up the Lab Environment","href":"/operating-systems/75/lab-setup","docId":"lab-setup"},{"type":"category","label":"Software Stack","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/Software Stack/Software-Stack","docId":"Software Stack/Software-Stack"},{"type":"link","label":"Overview","href":"/operating-systems/75/Software Stack/software-stack-overview","docId":"Software Stack/software-stack-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Printf vs Write","href":"/operating-systems/75/Software Stack/Questions/printf-vs-write","docId":"Software Stack/Questions/printf-vs-write"},{"type":"link","label":"Syscall ID","href":"/operating-systems/75/Software Stack/Questions/syscall-id","docId":"Software Stack/Questions/syscall-id"},{"type":"link","label":"Python Tools","href":"/operating-systems/75/Software Stack/Questions/python-tools","docId":"Software Stack/Questions/python-tools"},{"type":"link","label":"Malloc","href":"/operating-systems/75/Software Stack/Questions/malloc","docId":"Software Stack/Questions/malloc"},{"type":"link","label":"Software","href":"/operating-systems/75/Software Stack/Questions/software","docId":"Software Stack/Questions/software"},{"type":"link","label":"Static Executables","href":"/operating-systems/75/Software Stack/Questions/static-executables","docId":"Software Stack/Questions/static-executables"},{"type":"link","label":"Syscall Numbers","href":"/operating-systems/75/Software Stack/Questions/syscall-numbers","docId":"Software Stack/Questions/syscall-numbers"},{"type":"link","label":"Printf Syscall","href":"/operating-systems/75/Software Stack/Questions/printf-syscall","docId":"Software Stack/Questions/printf-syscall"},{"type":"link","label":"Syscall Wrapper","href":"/operating-systems/75/Software Stack/Questions/syscall-wrapper","docId":"Software Stack/Questions/syscall-wrapper"},{"type":"link","label":"Syscall Tool","href":"/operating-systems/75/Software Stack/Questions/syscall-tool","docId":"Software Stack/Questions/syscall-tool"},{"type":"link","label":"Libcall Syscall","href":"/operating-systems/75/Software Stack/Questions/libcall-syscall","docId":"Software Stack/Questions/libcall-syscall"},{"type":"link","label":"Libc","href":"/operating-systems/75/Software Stack/Questions/libc","docId":"Software Stack/Questions/libc"},{"type":"link","label":"Dynamic Libraries","href":"/operating-systems/75/Software Stack/Questions/dynamic-libraries","docId":"Software Stack/Questions/dynamic-libraries"},{"type":"link","label":"Strcpy Syscall","href":"/operating-systems/75/Software Stack/Questions/strcpy-syscall","docId":"Software Stack/Questions/strcpy-syscall"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Software Stack/Questions/"},{"type":"link","label":"Lab 1 - Operating System Perspective","href":"/operating-systems/75/Software Stack/lab1","docId":"Software Stack/lab1"},{"type":"link","label":"Lab 2 - Library Perspective","href":"/operating-systems/75/Software Stack/lab2","docId":"Software Stack/lab2"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Software Stack/"},{"type":"category","label":"Data","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/Data/","docId":"Data/Data"},{"type":"link","label":"Overview","href":"/operating-systems/75/Data/data-overview","docId":"Data/data-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Memory Access","href":"/operating-systems/75/Data/Questions/memory-access","docId":"Data/Questions/memory-access"},{"type":"link","label":"Half page","href":"/operating-systems/75/Data/Questions/half-page","docId":"Data/Questions/half-page"},{"type":"link","label":"Malloc Brk","href":"/operating-systems/75/Data/Questions/malloc-brk","docId":"Data/Questions/malloc-brk"},{"type":"link","label":"Malloc Mmap","href":"/operating-systems/75/Data/Questions/malloc-mmap","docId":"Data/Questions/malloc-mmap"},{"type":"link","label":"Memory granularity","href":"/operating-systems/75/Data/Questions/memory-granularity","docId":"Data/Questions/memory-granularity"},{"type":"link","label":"Mmap file","href":"/operating-systems/75/Data/Questions/mmap-file","docId":"Data/Questions/mmap-file"},{"type":"link","label":"Page Allocation","href":"/operating-systems/75/Data/Questions/page-allocation","docId":"Data/Questions/page-allocation"},{"type":"link","label":"Operators","href":"/operating-systems/75/Data/Questions/operators","docId":"Data/Questions/operators"},{"type":"link","label":"Memory Leaks","href":"/operating-systems/75/Data/Questions/memory-leaks","docId":"Data/Questions/memory-leaks"},{"type":"link","label":"Valgrind Leaks","href":"/operating-systems/75/Data/Questions/valgrind-leaks","docId":"Data/Questions/valgrind-leaks"},{"type":"link","label":"Bypass Canary","href":"/operating-systems/75/Data/Questions/bypass-canary","docId":"Data/Questions/bypass-canary"},{"type":"link","label":"Memory Aslr","href":"/operating-systems/75/Data/Questions/memory-aslr","docId":"Data/Questions/memory-aslr"},{"type":"link","label":"Memory Regions Vars","href":"/operating-systems/75/Data/Questions/memory-regions-vars","docId":"Data/Questions/memory-regions-vars"},{"type":"link","label":"Memory Stack Protector","href":"/operating-systems/75/Data/Questions/memory-stack-protector","docId":"Data/Questions/memory-stack-protector"},{"type":"link","label":"Stack Layout","href":"/operating-systems/75/Data/Questions/stack-layout","docId":"Data/Questions/stack-layout"},{"type":"link","label":"String buff over","href":"/operating-systems/75/Data/Questions/string-buff-over","docId":"Data/Questions/string-buff-over"},{"type":"link","label":"String strcpy","href":"/operating-systems/75/Data/Questions/string-strcpy","docId":"Data/Questions/string-strcpy"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Data/Questions/"},{"type":"link","label":"Lab 3 - Memory","href":"/operating-systems/75/Data/lab3","docId":"Data/lab3"},{"type":"link","label":"Lab 4 - Investigate Memory","href":"/operating-systems/75/Data/lab4","docId":"Data/lab4"},{"type":"link","label":"Lab 5 - Memory Security","href":"/operating-systems/75/Data/lab5","docId":"Data/lab5"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Data/"},{"type":"category","label":"Compute","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/Compute/","docId":"Compute/Compute"},{"type":"link","label":"Overview","href":"/operating-systems/75/Compute/compute-overview","docId":"Compute/compute-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Parent Faults Before Fork","href":"/operating-systems/75/Compute/Questions/parent-faults-before-fork","docId":"Compute/Questions/parent-faults-before-fork"},{"type":"link","label":"Mmap Cow Flag","href":"/operating-systems/75/Compute/Questions/mmap-cow-flag","docId":"Compute/Questions/mmap-cow-flag"},{"type":"link","label":"Sections Always Shared","href":"/operating-systems/75/Compute/Questions/sections-always-shared","docId":"Compute/Questions/sections-always-shared"},{"type":"link","label":"Child Faults After Write","href":"/operating-systems/75/Compute/Questions/child-faults-after-write","docId":"Compute/Questions/child-faults-after-write"},{"type":"link","label":"Seg Fault Exit Code","href":"/operating-systems/75/Compute/Questions/seg-fault-exit-code","docId":"Compute/Questions/seg-fault-exit-code"},{"type":"link","label":"Threads Shared Data","href":"/operating-systems/75/Compute/Questions/threads-shared-data","docId":"Compute/Questions/threads-shared-data"},{"type":"link","label":"Thread Memory","href":"/operating-systems/75/Compute/Questions/thread-memory","docId":"Compute/Questions/thread-memory"},{"type":"link","label":"Apache2 Strace","href":"/operating-systems/75/Compute/Questions/apache2-strace","docId":"Compute/Questions/apache2-strace"},{"type":"link","label":"Mini Shell Stops After Command","href":"/operating-systems/75/Compute/Questions/mini-shell-stops-after-command","docId":"Compute/Questions/mini-shell-stops-after-command"},{"type":"link","label":"Cause of File Not Found Error","href":"/operating-systems/75/Compute/Questions/cause-of-file-not-found-error","docId":"Compute/Questions/cause-of-file-not-found-error"},{"type":"link","label":"Process Creation","href":"/operating-systems/75/Compute/Questions/process-creation","docId":"Compute/Questions/process-creation"},{"type":"link","label":"Who Calls Execve Parent","href":"/operating-systems/75/Compute/Questions/who-calls-execve-parent","docId":"Compute/Questions/who-calls-execve-parent"},{"type":"link","label":"Exec Without Fork","href":"/operating-systems/75/Compute/Questions/exec-without-fork","docId":"Compute/Questions/exec-without-fork"},{"type":"link","label":"Create Sleepy Process Ending","href":"/operating-systems/75/Compute/Questions/create-sleepy-process-ending","docId":"Compute/Questions/create-sleepy-process-ending"},{"type":"link","label":"Processes Speedup","href":"/operating-systems/75/Compute/Questions/processes-speedup","docId":"Compute/Questions/processes-speedup"},{"type":"link","label":"Parent of Sleep Processes","href":"/operating-systems/75/Compute/Questions/parent-of-sleep-processes","docId":"Compute/Questions/parent-of-sleep-processes"},{"type":"link","label":"Sleeping on a Fiber","href":"/operating-systems/75/Compute/Questions/sleeping-on-a-fiber","docId":"Compute/Questions/sleeping-on-a-fiber"},{"type":"link","label":"Fiber Strace","href":"/operating-systems/75/Compute/Questions/fiber-strace","docId":"Compute/Questions/fiber-strace"},{"type":"link","label":"Type of Scheduler in Libult","href":"/operating-systems/75/Compute/Questions/type-of-scheduler-in-libult","docId":"Compute/Questions/type-of-scheduler-in-libult"},{"type":"link","label":"Number of Running Ults","href":"/operating-systems/75/Compute/Questions/number-of-running-ults","docId":"Compute/Questions/number-of-running-ults"},{"type":"link","label":"Why Use Completed Queue","href":"/operating-systems/75/Compute/Questions/why-use-completed-queue","docId":"Compute/Questions/why-use-completed-queue"},{"type":"link","label":"Ult Thread IDs","href":"/operating-systems/75/Compute/Questions/ult-thread-ids","docId":"Compute/Questions/ult-thread-ids"},{"type":"link","label":"TCB Libult Unikraft","href":"/operating-systems/75/Compute/Questions/tcb-libult-unikraft","docId":"Compute/Questions/tcb-libult-unikraft"},{"type":"link","label":"Time Slice Value","href":"/operating-systems/75/Compute/Questions/time-slice-value","docId":"Compute/Questions/time-slice-value"},{"type":"link","label":"Number of Running Threads","href":"/operating-systems/75/Compute/Questions/number-of-running-threads","docId":"Compute/Questions/number-of-running-threads"},{"type":"link","label":"Notify Only With Mutex","href":"/operating-systems/75/Compute/Questions/notify-only-with-mutex","docId":"Compute/Questions/notify-only-with-mutex"},{"type":"link","label":"TLS Var Copies","href":"/operating-systems/75/Compute/Questions/tls-var-copies","docId":"Compute/Questions/tls-var-copies"},{"type":"link","label":"TLS Synchronization","href":"/operating-systems/75/Compute/Questions/tls-synchronization","docId":"Compute/Questions/tls-synchronization"},{"type":"link","label":"Semaphore Equivalent","href":"/operating-systems/75/Compute/Questions/semaphore-equivalent","docId":"Compute/Questions/semaphore-equivalent"},{"type":"link","label":"Coarse vs Granular Critical Section","href":"/operating-systems/75/Compute/Questions/coarse-vs-granular-critical-section","docId":"Compute/Questions/coarse-vs-granular-critical-section"},{"type":"link","label":"Not Race Condition","href":"/operating-systems/75/Compute/Questions/not-race-condition","docId":"Compute/Questions/not-race-condition"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Compute/Questions/"},{"type":"link","label":"Lab 6 - Multiprocess and Multithread","href":"/operating-systems/75/Compute/lab6","docId":"Compute/lab6"},{"type":"link","label":"Lab 7 - Copy-on-Write","href":"/operating-systems/75/Compute/lab7","docId":"Compute/lab7"},{"type":"link","label":"Lab 8 - Syncronization","href":"/operating-systems/75/Compute/lab8","docId":"Compute/lab8"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Compute/"},{"type":"category","label":"IO","items":[{"type":"link","label":"Slides","href":"/operating-systems/75/IO/","docId":"IO/IO"},{"type":"link","label":"Overview","href":"/operating-systems/75/IO/io-overview","docId":"IO/io-overview"},{"type":"category","label":"Questions","items":[{"type":"link","label":"Bind Error Cause","href":"/operating-systems/75/IO/Questions/bind-error-cause","docId":"IO/Questions/bind-error-cause"},{"type":"link","label":"Anonymous Pipes Limitation","href":"/operating-systems/75/IO/Questions/anonymous-pipes-limitation","docId":"IO/Questions/anonymous-pipes-limitation"},{"type":"link","label":"Pipe Ends","href":"/operating-systems/75/IO/Questions/pipe-ends","docId":"IO/Questions/pipe-ends"},{"type":"link","label":"Receiver Socket FD","href":"/operating-systems/75/IO/Questions/receiver-socket-fd","docId":"IO/Questions/receiver-socket-fd"},{"type":"link","label":"Client Server Sender Receiver","href":"/operating-systems/75/IO/Questions/client-server-sender-receiver","docId":"IO/Questions/client-server-sender-receiver"},{"type":"link","label":"Firefox TCP UDP","href":"/operating-systems/75/IO/Questions/firefox-tcp-udp","docId":"IO/Questions/firefox-tcp-udp"},{"type":"link","label":"Server Copies","href":"/operating-systems/75/IO/Questions/server-copies","docId":"IO/Questions/server-copies"},{"type":"link","label":"Fewer Than 2 Copies","href":"/operating-systems/75/IO/Questions/fewer-than-2-copies","docId":"IO/Questions/fewer-than-2-copies"},{"type":"link","label":"Strace Printf","href":"/operating-systems/75/IO/Questions/strace-printf","docId":"IO/Questions/strace-printf"},{"type":"link","label":"Mmap Read Write Benchmark","href":"/operating-systems/75/IO/Questions/mmap-read-write-benchmark","docId":"IO/Questions/mmap-read-write-benchmark"},{"type":"link","label":"Syscalls CP","href":"/operating-systems/75/IO/Questions/syscalls-cp","docId":"IO/Questions/syscalls-cp"},{"type":"link","label":"File Handler C","href":"/operating-systems/75/IO/Questions/file-handler-c","docId":"IO/Questions/file-handler-c"},{"type":"link","label":"Flush Libc Buffer","href":"/operating-systems/75/IO/Questions/flush-libc-buffer","docId":"IO/Questions/flush-libc-buffer"},{"type":"link","label":"Stderr FD","href":"/operating-systems/75/IO/Questions/stderr-fd","docId":"IO/Questions/stderr-fd"},{"type":"link","label":"Local IO Errors","href":"/operating-systems/75/IO/Questions/local-io-errors","docId":"IO/Questions/local-io-errors"},{"type":"link","label":"Fopen Syscall","href":"/operating-systems/75/IO/Questions/fopen-syscall","docId":"IO/Questions/fopen-syscall"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/IO/Questions/"},{"type":"link","label":"Lab 9 - File Descriptors","href":"/operating-systems/75/IO/lab9","docId":"IO/lab9"},{"type":"link","label":"Lab 10 - Inter-Process Communication","href":"/operating-systems/75/IO/lab10","docId":"IO/lab10"},{"type":"link","label":"Lab 11 - IO Optimizations","href":"/operating-systems/75/IO/lab11","docId":"IO/lab11"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/IO/"},{"type":"category","label":"Lecture","items":[{"type":"link","label":"IO","href":"/operating-systems/75/Lecture/IO","docId":"Lecture/IO"},{"type":"link","label":"Application Interaction","href":"/operating-systems/75/Lecture/Application-Interaction","docId":"Lecture/Application-Interaction"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Lecture/"},{"type":"category","label":"Assignments","items":[{"type":"link","label":"Mini Libc","href":"/operating-systems/75/Assignments/Mini Libc/","docId":"Assignments/Mini Libc/README"},{"type":"link","label":"Memory Allocator","href":"/operating-systems/75/Assignments/Memory Allocator/","docId":"Assignments/Memory Allocator/README"},{"type":"link","label":"Parallel Firewall","href":"/operating-systems/75/Assignments/Parallel Firewall/","docId":"Assignments/Parallel Firewall/README"},{"type":"link","label":"Mini Shell","href":"/operating-systems/75/Assignments/Mini Shell/","docId":"Assignments/Mini Shell/README"},{"type":"link","label":"Asynchronous Web Server","href":"/operating-systems/75/Assignments/Asynchronous Web Server/","docId":"Assignments/Asynchronous Web Server/README"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Assignments/"},{"type":"category","label":"Exams","items":[{"type":"category","label":"2024 Summer","items":[{"type":"link","label":"Syscall Tracing","href":"/operating-systems/75/Exams/2024 Summer/Syscall Tracing/syscall-tracing","docId":"Exams/2024 Summer/Syscall Tracing/syscall-tracing"},{"type":"link","label":"Sysinfo Library","href":"/operating-systems/75/Exams/2024 Summer/Sysinfo Library/sysinfo-library","docId":"Exams/2024 Summer/Sysinfo Library/sysinfo-library"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Exams/2024 Summer/"},{"type":"link","label":"Digital Forensics","href":"/operating-systems/75/Exams/Digital Forensics/digital-forensics","docId":"Exams/Digital Forensics/digital-forensics"},{"type":"link","label":"Distributed System","href":"/operating-systems/75/Exams/Distributed System/distributed-system","docId":"Exams/Distributed System/distributed-system"},{"type":"link","label":"Aggregator Application","href":"/operating-systems/75/Exams/Aggregator Application/aggregator-application","docId":"Exams/Aggregator Application/aggregator-application"},{"type":"link","label":"Benchmarking Application","href":"/operating-systems/75/Exams/Benchmarking Application/benchmarking-application","docId":"Exams/Benchmarking Application/benchmarking-application"},{"type":"link","label":"Database Application","href":"/operating-systems/75/Exams/Database Application/database-application","docId":"Exams/Database Application/database-application"},{"type":"link","label":"Application Investigator","href":"/operating-systems/75/Exams/Application Investigator/application-investigator","docId":"Exams/Application Investigator/application-investigator"},{"type":"link","label":"Network Performance Utility","href":"/operating-systems/75/Exams/Network Performance Utility/network-performance-utility","docId":"Exams/Network Performance Utility/network-performance-utility"},{"type":"link","label":"Nightly Builds System","href":"/operating-systems/75/Exams/Nightly Builds System/nightly-builds-system","docId":"Exams/Nightly Builds System/nightly-builds-system"},{"type":"link","label":"User-level Threading Library","href":"/operating-systems/75/Exams/User-level Threading Library/user-level-threading-library","docId":"Exams/User-level Threading Library/user-level-threading-library"},{"type":"link","label":"System Process Monitoring Tool","href":"/operating-systems/75/Exams/System Process Monitoring Tool/system-process-monitoring-tool","docId":"Exams/System Process Monitoring Tool/system-process-monitoring-tool"},{"type":"link","label":"Network Configurations Manager","href":"/operating-systems/75/Exams/Network Configurations Manager/network-configurations-manager","docId":"Exams/Network Configurations Manager/network-configurations-manager"},{"type":"link","label":"FaaS Application","href":"/operating-systems/75/Exams/FaaS Application/faas-application","docId":"Exams/FaaS Application/faas-application"},{"type":"link","label":"Industrial System","href":"/operating-systems/75/Exams/Industrial System/industrial-system","docId":"Exams/Industrial System/industrial-system"},{"type":"link","label":"Blockchain System","href":"/operating-systems/75/Exams/Blockchain System/blockchain-system","docId":"Exams/Blockchain System/blockchain-system"},{"type":"link","label":"Supervisor-type Service","href":"/operating-systems/75/Exams/Supervisor-type Service/supervisor-type-service","docId":"Exams/Supervisor-type Service/supervisor-type-service"},{"type":"link","label":"Header Analysis Application","href":"/operating-systems/75/Exams/Header Analysis Application/header-analysis-application","docId":"Exams/Header Analysis Application/header-analysis-application"},{"type":"link","label":"Extending an App Manager","href":"/operating-systems/75/Exams/Extending an App Manager/extending-an-app-manager","docId":"Exams/Extending an App Manager/extending-an-app-manager"},{"type":"link","label":"Backup System","href":"/operating-systems/75/Exams/Backup System/backup-system","docId":"Exams/Backup System/backup-system"},{"type":"link","label":"File Changes Notifier","href":"/operating-systems/75/Exams/File Changes Notifier/file-changes-notifier","docId":"Exams/File Changes Notifier/file-changes-notifier"},{"type":"link","label":"Intrusion Detection System","href":"/operating-systems/75/Exams/Intrusion Detection System/intrusion-detection-system","docId":"Exams/Intrusion Detection System/intrusion-detection-system"},{"type":"link","label":"Resource Monitor","href":"/operating-systems/75/Exams/Resource Monitor/resource-monitor","docId":"Exams/Resource Monitor/resource-monitor"},{"type":"link","label":"Library Warmer","href":"/operating-systems/75/Exams/Library Warmer/library-warmer","docId":"Exams/Library Warmer/library-warmer"},{"type":"link","label":"Web GUI 1","href":"/operating-systems/75/Exams/Web GUI 1/web-gui-1","docId":"Exams/Web GUI 1/web-gui-1"},{"type":"link","label":"Web GUI 2","href":"/operating-systems/75/Exams/Web GUI 2/web-gui-2","docId":"Exams/Web GUI 2/web-gui-2"},{"type":"link","label":"Memory Deduplication","href":"/operating-systems/75/Exams/Memory Deduplication/memory-deduplication","docId":"Exams/Memory Deduplication/memory-deduplication"},{"type":"link","label":"Cloud System","href":"/operating-systems/75/Exams/Cloud System/cloud-system","docId":"Exams/Cloud System/cloud-system"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Exams/"},{"type":"category","label":"Hackathons","items":[{"type":"link","label":"Lambda Function Loader","href":"/operating-systems/75/Hackathons/Lambda Function Loader/","docId":"Hackathons/Lambda Function Loader/README"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/75/Hackathons/"},{"type":"link","label":"Rules and Grading","href":"/operating-systems/75/rules-and-grading","docId":"rules-and-grading"},{"type":"link","label":"Resources","href":"/operating-systems/75/resources","docId":"resources"}]},"docs":{"Assignments/Asynchronous Web Server/README":{"id":"Assignments/Asynchronous Web Server/README","title":"Asynchronous Web Server","description":"Objectives","sidebar":"sidebar"},"Assignments/Asynchronous Web Server/src/http-parser/README":{"id":"Assignments/Asynchronous Web Server/src/http-parser/README","title":"HTTP Parser","description":"This is a parser for HTTP messages written in C. It parses both requests and"},"Assignments/Memory Allocator/README":{"id":"Assignments/Memory Allocator/README","title":"Memory Allocator","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Libc/README":{"id":"Assignments/Mini Libc/README","title":"Mini-libc","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Shell/README":{"id":"Assignments/Mini Shell/README","title":"Minishell","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Shell/util/parser/README":{"id":"Assignments/Mini Shell/util/parser/README","title":"Parser","description":"The parser is made using Bison and Flex."},"Assignments/Parallel Firewall/README":{"id":"Assignments/Parallel Firewall/README","title":"Parallel Firewall","description":"Objectives","sidebar":"sidebar"},"Compute/Compute":{"id":"Compute/Compute","title":"Compute","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Compute/compute-overview":{"id":"Compute/compute-overview","title":"Compute","description":"Contents","sidebar":"sidebar"},"Compute/lab6":{"id":"Compute/lab6","title":"Lab 6 - Multiprocess and Multithread","description":"Task: Creating a process","sidebar":"sidebar"},"Compute/lab7":{"id":"Compute/lab7","title":"Lab 7 - Copy-on-Write","description":"Task: Minor and Major Page Faults","sidebar":"sidebar"},"Compute/lab8":{"id":"Compute/lab8","title":"Lab 8 - Syncronization","description":"Task Race Conditions","sidebar":"sidebar"},"Compute/Questions/apache2-strace":{"id":"Compute/Questions/apache2-strace","title":"`apache2` Document Root","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/cause-of-file-not-found-error":{"id":"Compute/Questions/cause-of-file-not-found-error","title":"Cause of `FileNotFoundError`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/child-faults-after-write":{"id":"Compute/Questions/child-faults-after-write","title":"Child Faults After Write","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/coarse-vs-granular-critical-section":{"id":"Compute/Questions/coarse-vs-granular-critical-section","title":"Coarse vs Granular Critical Section","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/create-sleepy-process-ending":{"id":"Compute/Questions/create-sleepy-process-ending","title":"`create_sleepy` Process Ending","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/exec-without-fork":{"id":"Compute/Questions/exec-without-fork","title":"exec()` Without `fork()","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/fiber-strace":{"id":"Compute/Questions/fiber-strace","title":"Fiber Strace","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/mini-shell-stops-after-command":{"id":"Compute/Questions/mini-shell-stops-after-command","title":"Mini-shell Stops After Command","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/mmap-cow-flag":{"id":"Compute/Questions/mmap-cow-flag","title":"Copy-on-write Flag for `mmap()`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/not-race-condition":{"id":"Compute/Questions/not-race-condition","title":"Not Race Condition","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/notify-only-with-mutex":{"id":"Compute/Questions/notify-only-with-mutex","title":"Both Condition and Mutex","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/number-of-running-threads":{"id":"Compute/Questions/number-of-running-threads","title":"Number of Running Threads","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/number-of-running-ults":{"id":"Compute/Questions/number-of-running-ults","title":"Number of RUNNING User-Level Threads","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/parent-faults-before-fork":{"id":"Compute/Questions/parent-faults-before-fork","title":"Parent Faults before `fork()`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/parent-of-sleep-processes":{"id":"Compute/Questions/parent-of-sleep-processes","title":"Parent of `sleep` Processes","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/process-creation":{"id":"Compute/Questions/process-creation","title":"Process Creation","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/processes-speedup":{"id":"Compute/Questions/processes-speedup","title":"Processes Speedup","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/sections-always-shared":{"id":"Compute/Questions/sections-always-shared","title":"Always Shared Sections","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/seg-fault-exit-code":{"id":"Compute/Questions/seg-fault-exit-code","title":"Segfault Exit Code","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/semaphore-equivalent":{"id":"Compute/Questions/semaphore-equivalent","title":"Semaphore Equivalent","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/sleeping-on-a-fiber":{"id":"Compute/Questions/sleeping-on-a-fiber","title":"Sleeping on a Fiber","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/tcb-libult-unikraft":{"id":"Compute/Questions/tcb-libult-unikraft","title":"Similarities Between the TCBs of `libult` and Unikraft","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/thread-memory":{"id":"Compute/Questions/thread-memory","title":"Thread Memory","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/threads-shared-data":{"id":"Compute/Questions/threads-shared-data","title":"Threads Shared Data","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/time-slice-value":{"id":"Compute/Questions/time-slice-value","title":"Time Slice Value","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/tls-synchronization":{"id":"Compute/Questions/tls-synchronization","title":"TLS Synchronization","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/tls-var-copies":{"id":"Compute/Questions/tls-var-copies","title":"TLS `var` Copies","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/type-of-scheduler-in-libult":{"id":"Compute/Questions/type-of-scheduler-in-libult","title":"Type of Scheduler in `libult.so`","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/ult-thread-ids":{"id":"Compute/Questions/ult-thread-ids","title":"ULT Thread IDs","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/who-calls-execve-parent":{"id":"Compute/Questions/who-calls-execve-parent","title":"Who Calls `execve` in the Log of the Parent Process?","description":"Question Text","sidebar":"sidebar"},"Compute/Questions/why-use-completed-queue":{"id":"Compute/Questions/why-use-completed-queue","title":"The Need for a COMPLETED Queue","description":"Question Text","sidebar":"sidebar"},"Data/Data":{"id":"Data/Data","title":"Data","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Data/data-overview":{"id":"Data/data-overview","title":"Data","description":"Data represents information that is to be processed to produce a final result or more data.","sidebar":"sidebar"},"Data/lab3":{"id":"Data/lab3","title":"Lab 3 - Memory","description":"Task: Memory Access","sidebar":"sidebar"},"Data/lab4":{"id":"Data/lab4","title":"Lab 4 - Investigate Memory","description":"Task: Allocating and Deallocating Memory","sidebar":"sidebar"},"Data/lab5":{"id":"Data/lab5","title":"Lab 5 - Memory Security","description":"Task: Wild Pointer Arithmetic Info Leak","sidebar":"sidebar"},"Data/Questions/bypass-canary":{"id":"Data/Questions/bypass-canary","title":"Bypass Canary","description":"Question","sidebar":"sidebar"},"Data/Questions/half-page":{"id":"Data/Questions/half-page","title":"Half Page","description":"Question Text","sidebar":"sidebar"},"Data/Questions/malloc-brk":{"id":"Data/Questions/malloc-brk","title":"malloc()` and `brk()","description":"Question Text","sidebar":"sidebar"},"Data/Questions/malloc-mmap":{"id":"Data/Questions/malloc-mmap","title":"malloc()` and `mmap()","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-access":{"id":"Data/Questions/memory-access","title":"Modify String","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-aslr":{"id":"Data/Questions/memory-aslr","title":"ASLR","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-granularity":{"id":"Data/Questions/memory-granularity","title":"Memory Granularity","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-leaks":{"id":"Data/Questions/memory-leaks","title":"Memory Leaks","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-regions-vars":{"id":"Data/Questions/memory-regions-vars","title":"Variables in memory regions","description":"Question Text","sidebar":"sidebar"},"Data/Questions/memory-stack-protector":{"id":"Data/Questions/memory-stack-protector","title":"Stack Protector","description":"Question Text","sidebar":"sidebar"},"Data/Questions/mmap-file":{"id":"Data/Questions/mmap-file","title":"`mmap()` file","description":"Question Text","sidebar":"sidebar"},"Data/Questions/operators":{"id":"Data/Questions/operators","title":"Operator Overloading","description":"Question Text","sidebar":"sidebar"},"Data/Questions/page-allocation":{"id":"Data/Questions/page-allocation","title":"Page Allocation","description":"Question Text","sidebar":"sidebar"},"Data/Questions/stack-layout":{"id":"Data/Questions/stack-layout","title":"Stack layout","description":"Question Text","sidebar":"sidebar"},"Data/Questions/string-buff-over":{"id":"Data/Questions/string-buff-over","title":"String Buffer Overflow","description":"Question Text","sidebar":"sidebar"},"Data/Questions/string-strcpy":{"id":"Data/Questions/string-strcpy","title":"Strcpy Buffer Overflow","description":"Question Text","sidebar":"sidebar"},"Data/Questions/valgrind-leaks":{"id":"Data/Questions/valgrind-leaks","title":"Valgrind Leaks","description":"Question Text","sidebar":"sidebar"},"Exams/2024 Summer/Syscall Tracing/syscall-tracing":{"id":"Exams/2024 Summer/Syscall Tracing/syscall-tracing","title":"Syscall Tracing","description":"Scenario","sidebar":"sidebar"},"Exams/2024 Summer/Sysinfo Library/sysinfo-library":{"id":"Exams/2024 Summer/Sysinfo Library/sysinfo-library","title":"Sysinfo Library","description":"Scenario","sidebar":"sidebar"},"Exams/Aggregator Application/aggregator-application":{"id":"Exams/Aggregator Application/aggregator-application","title":"Aggregator Application","description":"Scenario","sidebar":"sidebar"},"Exams/Application Investigator/application-investigator":{"id":"Exams/Application Investigator/application-investigator","title":"Application Investigator","description":"Scenario","sidebar":"sidebar"},"Exams/Backup System/backup-system":{"id":"Exams/Backup System/backup-system","title":"Backup System","description":"Scenario","sidebar":"sidebar"},"Exams/Benchmarking Application/benchmarking-application":{"id":"Exams/Benchmarking Application/benchmarking-application","title":"Benchmarking Application","description":"Scenario","sidebar":"sidebar"},"Exams/Blockchain System/blockchain-system":{"id":"Exams/Blockchain System/blockchain-system","title":"Blockchain System","description":"Scenario","sidebar":"sidebar"},"Exams/Cloud System/cloud-system":{"id":"Exams/Cloud System/cloud-system","title":"Cloud System","description":"Scenario","sidebar":"sidebar"},"Exams/Database Application/database-application":{"id":"Exams/Database Application/database-application","title":"Database Application","description":"Scenario","sidebar":"sidebar"},"Exams/Digital Forensics/digital-forensics":{"id":"Exams/Digital Forensics/digital-forensics","title":"Digital Forensics","description":"Scenario","sidebar":"sidebar"},"Exams/Distributed System/distributed-system":{"id":"Exams/Distributed System/distributed-system","title":"Distributed System","description":"Scenario","sidebar":"sidebar"},"Exams/Extending an App Manager/extending-an-app-manager":{"id":"Exams/Extending an App Manager/extending-an-app-manager","title":"Extending an App Manager","description":"Scenario","sidebar":"sidebar"},"Exams/FaaS Application/faas-application":{"id":"Exams/FaaS Application/faas-application","title":"FaaS Application","description":"Scenario","sidebar":"sidebar"},"Exams/File Changes Notifier/file-changes-notifier":{"id":"Exams/File Changes Notifier/file-changes-notifier","title":"File Changes Notifier","description":"Scenario","sidebar":"sidebar"},"Exams/Header Analysis Application/header-analysis-application":{"id":"Exams/Header Analysis Application/header-analysis-application","title":"Header Analysis Application","description":"Scenario","sidebar":"sidebar"},"Exams/Industrial System/industrial-system":{"id":"Exams/Industrial System/industrial-system","title":"Industrial System","description":"Scenario","sidebar":"sidebar"},"Exams/Intrusion Detection System/intrusion-detection-system":{"id":"Exams/Intrusion Detection System/intrusion-detection-system","title":"Intrusion Detection System","description":"Scenario","sidebar":"sidebar"},"Exams/Library Warmer/library-warmer":{"id":"Exams/Library Warmer/library-warmer","title":"Library Warmer","description":"Scenario","sidebar":"sidebar"},"Exams/Memory Deduplication/memory-deduplication":{"id":"Exams/Memory Deduplication/memory-deduplication","title":"Memory Deduplication","description":"Scenario","sidebar":"sidebar"},"Exams/Network Configurations Manager/network-configurations-manager":{"id":"Exams/Network Configurations Manager/network-configurations-manager","title":"Network Configurations Manager","description":"Scenario","sidebar":"sidebar"},"Exams/Network Performance Utility/network-performance-utility":{"id":"Exams/Network Performance Utility/network-performance-utility","title":"Network Performance Utility","description":"Scenario","sidebar":"sidebar"},"Exams/Nightly Builds System/nightly-builds-system":{"id":"Exams/Nightly Builds System/nightly-builds-system","title":"Nightly Builds System","description":"Scenario","sidebar":"sidebar"},"Exams/Resource Monitor/resource-monitor":{"id":"Exams/Resource Monitor/resource-monitor","title":"Resource Monitor","description":"Scenario","sidebar":"sidebar"},"Exams/Supervisor-type Service/supervisor-type-service":{"id":"Exams/Supervisor-type Service/supervisor-type-service","title":"Supervisor-type Service","description":"Scenario","sidebar":"sidebar"},"Exams/System Process Monitoring Tool/system-process-monitoring-tool":{"id":"Exams/System Process Monitoring Tool/system-process-monitoring-tool","title":"System Process Monitoring Tool","description":"Scenario","sidebar":"sidebar"},"Exams/User-level Threading Library/user-level-threading-library":{"id":"Exams/User-level Threading Library/user-level-threading-library","title":"User-level Threading Library","description":"Scenario","sidebar":"sidebar"},"Exams/Web GUI 1/web-gui-1":{"id":"Exams/Web GUI 1/web-gui-1","title":"Web GUI 1","description":"Scenario","sidebar":"sidebar"},"Exams/Web GUI 2/web-gui-2":{"id":"Exams/Web GUI 2/web-gui-2","title":"Web GUI 2","description":"Scenario","sidebar":"sidebar"},"Hackathons/Lambda Function Loader/README":{"id":"Hackathons/Lambda Function Loader/README","title":"Lambda Function Loader","description":"Application Development","sidebar":"sidebar"},"IO/IO":{"id":"IO/IO","title":"IO","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"IO/io-overview":{"id":"IO/io-overview","title":"I/O","description":"We know that a compute system is a collection of hardware and software that modifies data.","sidebar":"sidebar"},"IO/lab10":{"id":"IO/lab10","title":"Lab 10 - Inter-Process Communication","description":"Task: Named Pipes Communication","sidebar":"sidebar"},"IO/lab11":{"id":"IO/lab11","title":"Lab 11 - IO Optimizations","description":"Task: Ordered Client-Server Communication","sidebar":"sidebar"},"IO/lab9":{"id":"IO/lab9","title":"Lab 9 - File Descriptors","description":"Task: My cat","sidebar":"sidebar"},"IO/Questions/anonymous-pipes-limitation":{"id":"IO/Questions/anonymous-pipes-limitation","title":"Limitation of Anonymous Pipes","description":"Question Text","sidebar":"sidebar"},"IO/Questions/bind-error-cause":{"id":"IO/Questions/bind-error-cause","title":"Cause of `bind()` Error","description":"Question Text","sidebar":"sidebar"},"IO/Questions/client-server-sender-receiver":{"id":"IO/Questions/client-server-sender-receiver","title":"`sender.py` and `receiver.py` Client-Server Parallel","description":"Question Text","sidebar":"sidebar"},"IO/Questions/fewer-than-2-copies":{"id":"IO/Questions/fewer-than-2-copies","title":"Fewer than Two Copies","description":"Question Text","sidebar":"sidebar"},"IO/Questions/file-handler-c":{"id":"IO/Questions/file-handler-c","title":"File handler in C","description":"Question Text","sidebar":"sidebar"},"IO/Questions/firefox-tcp-udp":{"id":"IO/Questions/firefox-tcp-udp","title":"Firefox: TCP or UDP?","description":"Question Text","sidebar":"sidebar"},"IO/Questions/flush-libc-buffer":{"id":"IO/Questions/flush-libc-buffer","title":"Flush Libc Buffer","description":"Question Text","sidebar":"sidebar"},"IO/Questions/fopen-syscall":{"id":"IO/Questions/fopen-syscall","title":"Syscall Used by `fopen()`","description":"Question Text","sidebar":"sidebar"},"IO/Questions/local-io-errors":{"id":"IO/Questions/local-io-errors","title":"I/O Errors","description":"Question Text","sidebar":"sidebar"},"IO/Questions/mmap-read-write-benchmark":{"id":"IO/Questions/mmap-read-write-benchmark","title":"`mmap()` vs `read()` and `write()` Benchmark","description":"Question Text","sidebar":"sidebar"},"IO/Questions/pipe-ends":{"id":"IO/Questions/pipe-ends","title":"Pipe Ends","description":"Question Text","sidebar":"sidebar"},"IO/Questions/receiver-socket-fd":{"id":"IO/Questions/receiver-socket-fd","title":"Receiver Socked File Descriptor","description":"Question Text","sidebar":"sidebar"},"IO/Questions/server-copies":{"id":"IO/Questions/server-copies","title":"Client-Server Number of Copies","description":"Question Text","sidebar":"sidebar"},"IO/Questions/stderr-fd":{"id":"IO/Questions/stderr-fd","title":"File Descriptor of `stderr`","description":"Question Text","sidebar":"sidebar"},"IO/Questions/strace-printf":{"id":"IO/Questions/strace-printf","title":"`printf()` Under Strace","description":"Question Text","sidebar":"sidebar"},"IO/Questions/syscalls-cp":{"id":"IO/Questions/syscalls-cp","title":"Syscalls Used by `cp`","description":"Question Text","sidebar":"sidebar"},"lab-setup":{"id":"lab-setup","title":"Setting up the Lab Environment","description":"Prerequisites","sidebar":"sidebar"},"Lecture/Application-Interaction":{"id":"Lecture/Application-Interaction","title":"Application-Interaction","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Lecture/IO":{"id":"Lecture/IO","title":"IO","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"README":{"id":"README","title":"Operating Systems","description":"Welcome to the Operating Systems course, held by the National University of Science and Technology Politehnica Bucharest.","sidebar":"sidebar"},"resources":{"id":"resources","title":"Resources and Useful Links","description":"Need to Know","sidebar":"sidebar"},"rules-and-grading":{"id":"rules-and-grading","title":"Rules and Grading","description":"Grading","sidebar":"sidebar"},"Software Stack/lab1":{"id":"Software Stack/lab1","title":"Lab 1 - Operating System Perspective","description":"Task: System Calls","sidebar":"sidebar"},"Software Stack/lab2":{"id":"Software Stack/lab2","title":"Lab 2 - Library Perspective","description":"Task: Common Functions","sidebar":"sidebar"},"Software Stack/Questions/dynamic-libraries":{"id":"Software Stack/Questions/dynamic-libraries","title":"Dynamic Libraries","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/libc":{"id":"Software Stack/Questions/libc","title":"Syscall Tool","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/libcall-syscall":{"id":"Software Stack/Questions/libcall-syscall","title":"Libcall with Syscall","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/malloc":{"id":"Software Stack/Questions/malloc","title":"malloc()","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/printf-syscall":{"id":"Software Stack/Questions/printf-syscall","title":"printf() System Call","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/printf-vs-write":{"id":"Software Stack/Questions/printf-vs-write","title":"printf() vs write","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/python-tools":{"id":"Software Stack/Questions/python-tools","title":"Python Tools","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/software":{"id":"Software Stack/Questions/software","title":"Software Properties","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/static-executables":{"id":"Software Stack/Questions/static-executables","title":"Static Executables","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/strcpy-syscall":{"id":"Software Stack/Questions/strcpy-syscall","title":"strcpy() System Call","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-id":{"id":"Software Stack/Questions/syscall-id","title":"Syscall ID","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-numbers":{"id":"Software Stack/Questions/syscall-numbers","title":"Syscall Numbers","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-tool":{"id":"Software Stack/Questions/syscall-tool","title":"Syscall Tool","description":"Question Text","sidebar":"sidebar"},"Software Stack/Questions/syscall-wrapper":{"id":"Software Stack/Questions/syscall-wrapper","title":"Syscall Wrappers","description":"Question Text","sidebar":"sidebar"},"Software Stack/Software-Stack":{"id":"Software Stack/Software-Stack","title":"Software-Stack","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Software Stack/software-stack-overview":{"id":"Software Stack/software-stack-overview","title":"Software Stack","description":"Software comprises of code and data that is loaded in memory and used by the CPU.","sidebar":"sidebar"}}}')}}]); \ No newline at end of file diff --git a/75/assets/js/94849ae2.735f4013.js b/75/assets/js/94849ae2.735f4013.js new file mode 100644 index 0000000000..cd3ab0f550 --- /dev/null +++ b/75/assets/js/94849ae2.735f4013.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1055],{5680:(e,n,t)=>{t.d(n,{xA:()=>d,yg:()=>c});var a=t(6540);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function l(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var p=a.createContext({}),s=function(e){var n=a.useContext(p),t=n;return e&&(t="function"==typeof e?e(n):l(l({},n),e)),t},d=function(e){var n=s(e.components);return a.createElement(p.Provider,{value:n},e.children)},g="mdxType",m={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},y=a.forwardRef((function(e,n){var t=e.components,i=e.mdxType,r=e.originalType,p=e.parentName,d=o(e,["components","mdxType","originalType","parentName"]),g=s(t),y=i,c=g["".concat(p,".").concat(y)]||g[y]||m[y]||r;return t?a.createElement(c,l(l({ref:n},d),{},{components:t})):a.createElement(c,l({ref:n},d))}));function c(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var r=t.length,l=new Array(r);l[0]=y;var o={};for(var p in n)hasOwnProperty.call(n,p)&&(o[p]=n[p]);o.originalType=e,o[g]="string"==typeof e?e:i,l[1]=o;for(var s=2;s{t.r(n),t.d(n,{assets:()=>p,contentTitle:()=>l,default:()=>m,frontMatter:()=>r,metadata:()=>o,toc:()=>s});var a=t(8168),i=(t(6540),t(5680));const r={},l="Lab 9 - File Descriptors",o={unversionedId:"IO/lab9",id:"IO/lab9",title:"Lab 9 - File Descriptors",description:"Task: My cat",source:"@site/docs/IO/lab9.md",sourceDirName:"IO",slug:"/IO/lab9",permalink:"/operating-systems/75/IO/lab9",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Syscall Used by `fopen()`",permalink:"/operating-systems/75/IO/Questions/fopen-syscall"},next:{title:"Lab 10 - Inter-Process Communication",permalink:"/operating-systems/75/IO/lab10"}},p={},s=[{value:"Task: My cat",id:"task-my-cat",level:2},{value:"Task: Copy a File with mmap()",id:"task-copy-a-file-with-mmap",level:2},{value:"Task: Anonymous Pipes Communication",id:"task-anonymous-pipes-communication",level:2},{value:"File Descriptors",id:"file-descriptors",level:2},{value:"FILE Operations Explained",id:"file-operations-explained",level:3},{value:"File Descriptor Operations",id:"file-descriptor-operations",level:2},{value:"open()",id:"open",level:3},{value:"close()",id:"close",level:3},{value:"read() and write()",id:"read-and-write",level:3},{value:"lseek()",id:"lseek",level:3},{value:"Pipes",id:"pipes",level:2},{value:"Anonymous Pipes",id:"anonymous-pipes",level:3},{value:"Named Pipes (FIFOs)",id:"named-pipes-fifos",level:3},{value:"Redirections",id:"redirections",level:3},{value:"Guide: Simple File Operations",id:"guide-simple-file-operations",level:2},{value:"Guide: Redirections",id:"guide-redirections",level:2},{value:"dup()/dup2() - Atomic IO",id:"dupdup2---atomic-io",level:3},{value:"Guide: File Descriptor Table",id:"guide-file-descriptor-table",level:2},{value:"Guide: libc FILE struct",id:"guide-libc-file-struct",level:2},{value:"printf() Buffering",id:"printf-buffering",level:3},{value:"Guide: File Mappings",id:"guide-file-mappings",level:2},{value:"File I/O vs mmap()",id:"file-io-vs-mmap",level:3},{value:"Guide: Reading Linux Directories",id:"guide-reading-linux-directories",level:2}],d={toc:s},g="wrapper";function m(e){let{components:n,...r}=e;return(0,i.yg)(g,(0,a.A)({},d,r,{components:n,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"lab-9---file-descriptors"},"Lab 9 - File Descriptors"),(0,i.yg)("h2",{id:"task-my-cat"},"Task: My ",(0,i.yg)("inlineCode",{parentName:"h2"},"cat")),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/file-descriptors/drills/tasks/my-cat/support/src")," and checkout ",(0,i.yg)("inlineCode",{parentName:"p"},"my_cat.c"),".\nWe propose to implement the Linux command ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," that reads one or more files, ",(0,i.yg)("strong",{parentName:"p"},"concatenates")," them (hence the name ",(0,i.yg)("inlineCode",{parentName:"p"},"cat"),"), and prints them to standard output."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Implement ",(0,i.yg)("inlineCode",{parentName:"p"},"rread()")," wrapper over ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"read()")," system call does not guarantee that it will read the requested number of bytes in a single call.\nThis happens when the file does not have enough bytes, or when ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," is interrupted by a signal.\n",(0,i.yg)("inlineCode",{parentName:"p"},"rread()")," will handle these situations, ensuring that it reads either ",(0,i.yg)("inlineCode",{parentName:"p"},"num_bytes")," or all available bytes.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Implement ",(0,i.yg)("inlineCode",{parentName:"p"},"wwrite()")," as a wrapper for ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),"."),(0,i.yg)("p",{parentName:"li"},"The ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," system call may not write the requested number of bytes in a single call.\nThis happens if ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," is interrupted by a signal.\n",(0,i.yg)("inlineCode",{parentName:"p"},"wwrite()")," will guarantee that it wrote the full ",(0,i.yg)("inlineCode",{parentName:"p"},"num_bytes"),", retrying as necessary until all data is successfully written or an error occurs.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Implement ",(0,i.yg)("inlineCode",{parentName:"p"},"cat()"),"."),(0,i.yg)("p",{parentName:"li"},"Use ",(0,i.yg)("inlineCode",{parentName:"p"},"rread()")," to read an entire file and ",(0,i.yg)("inlineCode",{parentName:"p"},"wwrite()")," to write the contents to standard output.\nKeep in mind that the buffer size may not fit the entire file at once."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"this reading material"),"."),(0,i.yg)("h2",{id:"task-copy-a-file-with-mmap"},"Task: Copy a File with ",(0,i.yg)("inlineCode",{parentName:"h2"},"mmap()")),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/drills/tasks/mmap_cp")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate ",(0,i.yg)("inlineCode",{parentName:"p"},"support"),".\nAs you know ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," can map files in memory, perform operations on them, and then write them back to the disk.\nLet's check how well it performs by comparing it to the ",(0,i.yg)("inlineCode",{parentName:"p"},"cp")," command.\nThe benchmarking is automated by ",(0,i.yg)("inlineCode",{parentName:"p"},"benchmark_cp.sh")," so focus on completing ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap_cp.c")," for now."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/syscalls-cp"},"Quiz: Checkout what syscalls ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")," uses")),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap_cp.c")," and complete the TODOs to map the files in memory and copy the contents.\nDo not forget to clean up by unmapping and closing the files."),(0,i.yg)("p",{parentName:"li"},"To test, run ",(0,i.yg)("inlineCode",{parentName:"p"},"make test-file")," to generate a 1MB file with random data, and then run ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap_cp test-file output.txt"),".\nEnsure they have the same content with a simple ",(0,i.yg)("inlineCode",{parentName:"p"},"diff"),": ",(0,i.yg)("inlineCode",{parentName:"p"},"diff test-file.txt output.txt"),".")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Compare your implementation to the ",(0,i.yg)("inlineCode",{parentName:"p"},"cp")," command.\nRun ",(0,i.yg)("inlineCode",{parentName:"p"},"make large-file")," to generate a 1GB file with random data, and then run ",(0,i.yg)("inlineCode",{parentName:"p"},"./benchmark_cp.sh"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/mmap-read-write-benchmark"},"Quiz: Debunk why ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")," is winning")),(0,i.yg)("p",{parentName:"li"},"If you want a more generic answer, checkout this ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-mappings"},"guide on ",(0,i.yg)("inlineCode",{parentName:"a"},"mmap")," vs ",(0,i.yg)("inlineCode",{parentName:"a"},"read()-write()")),".")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"This demo would not be complete without some live analysis.\nUncomment the calls to ",(0,i.yg)("inlineCode",{parentName:"p"},"wait_for_input()")," and rerun the program.\nIn another terminal, run ",(0,i.yg)("inlineCode",{parentName:"p"},"cat /proc/$(pidof mmap_cp)/maps")," to see mapped files, and ",(0,i.yg)("inlineCode",{parentName:"p"},"ps -o pid,vsz,rss ")," to see how demand paging happens."))),(0,i.yg)("h2",{id:"task-anonymous-pipes-communication"},"Task: Anonymous Pipes Communication"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/ipc/drills/tasks/anon-pipes")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate the ",(0,i.yg)("inlineCode",{parentName:"p"},"support/")," folder.\nIn this exercise, you'll implement client-server communication between a parent and a child process using an anonymous pipe.\nThe parent will act as the sender, while the child acts as the receiver, with both processes sharing messages through the pipe.\nSince pipes are unidirectional, each process should close the end of the pipe it does not use."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Use the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/pipe.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"pipe()")," syscall")," to create the pipe.\nRemember, the first file descriptor (",(0,i.yg)("inlineCode",{parentName:"p"},"fds[0]"),") is the read end, and the second (",(0,i.yg)("inlineCode",{parentName:"p"},"fds[1]"),") is the write end, similar to how ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," are represented by file descriptors ",(0,i.yg)("inlineCode",{parentName:"p"},"0")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"1"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Hint:")," Use ",(0,i.yg)("inlineCode",{parentName:"p"},"exit")," to end the program."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/pipe-ends"},"Quiz: Discover why you cannot use either end of the pipe for reading or writing"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Solve the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"parent_loop")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"child_loop")," so that the application stops on ",(0,i.yg)("inlineCode",{parentName:"p"},"exit"),".\nEnsure each process closes the its pipe end before exiting to prevent indefinite blocking."),(0,i.yg)("blockquote",{parentName:"li"},(0,i.yg)("p",{parentName:"blockquote"},"Why is closing the pipe ends important?")),(0,i.yg)("p",{parentName:"li"},"The child process checks for the end of communication by reading from the pipe and checking for ",(0,i.yg)("inlineCode",{parentName:"p"},"EOF"),", which occurs when the write end is closed.\nWithout closing the write end, the child will block indefinitely in ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),".\nAs for the parent, it will block indefinitely in ",(0,i.yg)("inlineCode",{parentName:"p"},"wait()"),"."))),(0,i.yg)("h2",{id:"file-descriptors"},"File Descriptors"),(0,i.yg)("p",null,"You've most likely had to deal with files in the past.\nThe most common command that works with files is ",(0,i.yg)("inlineCode",{parentName:"p"},"cat"),".\nFor a quick refresher, let's write something to a file, and then read its contents."),(0,i.yg)("p",null,"You\u2019ve likely worked with files before;\nnow it\u2019s time to see what happens behind the scenes.\nThe most common way to read a file in Linux is by using the ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," command.\nFor a quick refresher, let\u2019s do a demo by writing some text to a file and then reading it back."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~/$ echo "OS Rullz!" # Print \'OS Rullz!\'\nOS Rullz!\nstudent@os:~/$ echo "OS Rullz!" > newfile.txt # redirect the output to newfile.txt\n## Let\'s check the contents of newfile.txt\nstudent@os:~/$ cat newfile.txt\nOS Rullz!\n')),(0,i.yg)("p",null,"If we were to implement this in C, we would use the ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure and write something like this:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'FILE *f = fopen("newfile.txt", "r");\nif (!f) {...} // handle error\n\nchar buf[1024];\nint rc = fread(buf, 1, sizeof(buf), f);\nif (rc < 0) {...} // handle error\n\nprintf("%s\\n", buf);\n')),(0,i.yg)("p",null,"For a complete example, check out this guide on ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-simple-file-operations"},"file operations in C, Python, and Java"),"."),(0,i.yg)("h3",{id:"file-operations-explained"},(0,i.yg)("inlineCode",{parentName:"h3"},"FILE")," Operations Explained"),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure is not the most straightforward method for performing file operations.\nIt is part of ",(0,i.yg)("inlineCode",{parentName:"p"},"libc")," and functions as a handler for working with files.\nThis is not particular to C, as most programming languages offer similar handlers."),(0,i.yg)("p",null,"Running ",(0,i.yg)("inlineCode",{parentName:"p"},"strace cat newfile.txt")," reveals that ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," (or ",(0,i.yg)("inlineCode",{parentName:"p"},"openat"),"), ",(0,i.yg)("inlineCode",{parentName:"p"},"fread()")," wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"fclose()")," wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"close()"),".\nAs you can see, the ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE"),"-related functions are just syscalls prefixed with ",(0,i.yg)("inlineCode",{parentName:"p"},"f-"),"."),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"th"},"FILE")," Operation"),(0,i.yg)("th",{parentName:"tr",align:null},"Syscall"),(0,i.yg)("th",{parentName:"tr",align:null},"Description"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fopen()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"open()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Opens a file and returns a file pointer.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fclose()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/close.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"close()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Closes the file associated with the pointer.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fread()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/read.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"read()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Reads data from the file into a buffer.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fwrite()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/write.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"write()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Writes data from a buffer to the file.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fseek()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/lseek.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"lseek()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Moves the file position indicator.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"truncate()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/ftruncate.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"ftruncate()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Truncates the file to a specified length.")))),(0,i.yg)("p",null,"The main distinction between ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," operations and their corresponding system calls is that the latter use a ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," to reference a file.\n",(0,i.yg)("strong",{parentName:"p"},"File descriptors")," are simply indexes into the process's ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table"),", which is the list of all currently open files for that process."),(0,i.yg)("p",null,"This concept is not entirely new, as each process has three default channels: ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"stderr"),".\nThese are, in fact, the first three entries in every process\u2019s ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table"),"."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/stderr-fd"},"Quiz: Test your intuition by finding the file descriptor of ",(0,i.yg)("inlineCode",{parentName:"a"},"stderr"))),(0,i.yg)("p",null,"Let's translate our previous example to illustrate how this change affects the implementation:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'int fd = open("newfile.txt", O_RDONLY)\nif (fd < 0) {...} // handle error\n\nchar buf[1024];\nint rc = read(fd, buf, sizeof(buf)); // Not complete, should\'ve used a while loop\nif (rc < 0) {...} // handle error\n\nbuf[rc] = \'\\0\'; // Null-terminate the buffer\nprintf("%s\\n", buf);\n')),(0,i.yg)("p",null,"To better understand the ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," API, you can either ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptor-operations"},"keep reading about file descriptor operations")," or checkout ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-reading-linux-directories"},"this guide on reading Linux directories"),"."),(0,i.yg)("p",null,"If you're interested in understanding how ",(0,i.yg)("inlineCode",{parentName:"p"},"libc")," utilizes file descriptors to simplify common operations, check out ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-libc-file-struct"},"this guide"),"."),(0,i.yg)("h2",{id:"file-descriptor-operations"},"File Descriptor Operations"),(0,i.yg)("p",null,"File descriptors are the primary means of referencing files in our system.\nThey are created, deleted, and manipulated through file interface operations, namely ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"lseek()"),".\nFrom a programmer's perspective, file descriptors are simply indexes into the process's ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table"),", which maintains a list of all currently open files for that process."),(0,i.yg)("p",null,"In this section, we will focus on how to utilize file descriptors to perform the same operations that ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," allows, and more.\nIf you want to delve deeper into file descriptors, we recommend exploring ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"this guide on the ",(0,i.yg)("strong",{parentName:"a"},"File Descriptor Table")),"."),(0,i.yg)("h3",{id:"open"},(0,i.yg)("inlineCode",{parentName:"h3"},"open()")),(0,i.yg)("p",null,"All processes start with three default file descriptors, inherited from the process's parent:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"stdin")," (standard input): 0"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"stdout")," (standard output): 1"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"stderr")," (standard error): 2")),(0,i.yg)("p",null,"To create new file descriptors (i.e. open new files), a process can use the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"open()"))," system call.\nIt receives the path to the file, some flags which are akin to the ",(0,i.yg)("inlineCode",{parentName:"p"},"mode")," string passed to ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()"),".\nAn optional ",(0,i.yg)("inlineCode",{parentName:"p"},"mode")," parameter that denotes the file's permissions if the ",(0,i.yg)("inlineCode",{parentName:"p"},"open")," must create it can also be provided.\nIf you use ",(0,i.yg)("inlineCode",{parentName:"p"},"O_CREAT"),", just remember to also pass ",(0,i.yg)("inlineCode",{parentName:"p"},"0644")," (",(0,i.yg)("inlineCode",{parentName:"p"},"rw-r--r--")," in octal, denoted by the first ",(0,i.yg)("inlineCode",{parentName:"p"},"0"),"), or permissions more restrictive."),(0,i.yg)("p",null,"Some other useful flags for ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," are:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"O_APPEND"),": place file cursor at the end"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"O_CLOEXEC"),": close the file descriptor when ",(0,i.yg)("inlineCode",{parentName:"li"},"exec()")," is called.\nThis is useful because child processes inherit the file descriptors, and this can lead to security problems."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"O_TRUNC"),": truncate the file to length 0.")),(0,i.yg)("h3",{id:"close"},(0,i.yg)("inlineCode",{parentName:"h3"},"close()")),(0,i.yg)("p",null,"Once you are done with a file descriptor you should call ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," to free its ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nThis is similar to how you free memory once you are done with it."),(0,i.yg)("h3",{id:"read-and-write"},(0,i.yg)("inlineCode",{parentName:"h3"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"h3"},"write()")),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"read_bytes = read(fd, buf, num_bytes);\nwritten_bytes = write(fd, buf, num_bytes);\n")),(0,i.yg)("p",null,"As you know, verifying the return code of system calls is the way to go in general.\nThis is even more apparent when dealing with I/O syscalls, namely ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", which return the number of bytes read or written."),(0,i.yg)("p",null,"Syscalls returning the number of bytes might seem redundant, but once you hear about partial I/O operations, it is of utmost importance.\nIf your process was interrupted by a signal while reading or writing, it is up to you to continue from where it left off."),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Remember: It is mandatory that we always use ",(0,i.yg)("inlineCode",{parentName:"strong"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"strong"},"write()")," inside ",(0,i.yg)("inlineCode",{parentName:"strong"},"while")," loops."),"\nHigher-level functions like ",(0,i.yg)("inlineCode",{parentName:"p"},"fread()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"fwrite()")," also use ",(0,i.yg)("inlineCode",{parentName:"p"},"while")," loops when calling ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," respectively.\nYou can practice this by ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#task-my-cat"},"implementing your own ",(0,i.yg)("inlineCode",{parentName:"a"},"cat")," command"),"."),(0,i.yg)("p",null,"In the following sections, we'll use file descriptors and ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," to interact with some inter-process-communication mechanisms, such as pipes."),(0,i.yg)("h3",{id:"lseek"},(0,i.yg)("inlineCode",{parentName:"h3"},"lseek()")),(0,i.yg)("p",null,"As you know, reading or writing from a file always continues from where it left off.\nMost of the time you would read from a file monotonically so it makes sense to keep the interface clean and handle bookkeeping in the back."),(0,i.yg)("p",null,"For cases when you selectively update the file or jump around fetching data, or making updates, we have ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/lseek.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"lseek")),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"off_t lseek(int fd, off_t offset, int whence);\n")),(0,i.yg)("p",null,"Its parameters are pretty intuitive: ",(0,i.yg)("inlineCode",{parentName:"p"},"fd")," stands for the file descriptor and ",(0,i.yg)("inlineCode",{parentName:"p"},"offset")," stands for the offset.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"whence")," directive explains what ",(0,i.yg)("inlineCode",{parentName:"p"},"offset")," is relative to, and has the following values:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"SEEK_SET"),": the file offset is set to offset bytes."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"SEEK_CUR"),": The file offset is set to its current location plus offset bytes."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"SEEK_END"),": the file offset is set to the size of the file plus offset bytes.")),(0,i.yg)("h2",{id:"pipes"},"Pipes"),(0,i.yg)("h3",{id:"anonymous-pipes"},"Anonymous Pipes"),(0,i.yg)("p",null,"In this session, we'll explore a new mean of Inter-Process Communication (IPC), namely ",(0,i.yg)("strong",{parentName:"p"},"the pipes"),".\nPipes are by no means something new, and you most probably played with them already in bash:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-bash"},"cat 'log_*.csv' | tr -s ' ' | cut -d ',' -f 2 | sort -u | head -n 10\n")),(0,i.yg)("p",null,"Using pipes (denoted as ",(0,i.yg)("inlineCode",{parentName:"p"},"|")," in the above example) enables linking the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," of multiple processes.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," of ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," is the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," of ",(0,i.yg)("inlineCode",{parentName:"p"},"tr"),", whose ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," is the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," of ",(0,i.yg)("inlineCode",{parentName:"p"},"cut"),' and so on.\nThis "chain" of commands looks like this:'),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Piped Commands",src:t(9234).A})),(0,i.yg)("p",null,"So here we have a ",(0,i.yg)("strong",{parentName:"p"},"unidirectional")," stream of data that starts from ",(0,i.yg)("inlineCode",{parentName:"p"},"cat"),", is modified by each new command, and then is passed to the next one.\nWe can tell from the image above that the communication channel between any 2 adjacent commands allows one process to write to it while the other reads from it.\nFor example, there is no need for ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," to read any of ",(0,i.yg)("inlineCode",{parentName:"p"},"tr"),"'s output, only vice versa."),(0,i.yg)("p",null,"In UNIX, the need for such a channel is fulfilled by the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"pipe()")," syscall"),".\nImagine there's a literal pipe between any 2 adjacent commands in the image above, where data is what flows through this pipe ",(0,i.yg)("strong",{parentName:"p"},"in only a single way"),"."),(0,i.yg)("p",null,"Such pipes are known as ",(0,i.yg)("strong",{parentName:"p"},"anonymous pipes")," because they don\u2019t have identifiers.\nThey are created by a parent process, which shares them with its children.\nData written to an anonymous pipe is stored in a kernel-managed circular buffer, where it\u2019s available for related-processes to read."),(0,i.yg)("p",null,"The following example showcases a typical workflow with anonymous pipes in Unix:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'##define EXIT_ON_COND(cond) do { if (cond) exit(EXIT_FAILURE); } while (0)\n\n// pipe_fd[0] -> for reading\n// pipe_fd[1] -> for writing\nint pipe_fd[2];\n\nEXIT_ON_COND(pipe(pipe_fd) < 0); // Create the pipe\n\nint pid = fork(); // Fork to create a child process\nEXIT_ON_COND(pid < 0); // Check for fork() failure\n\nif (pid == 0) { // Child process\n EXIT_ON_COND(close(pipe_fd[0]) != 0); // Close the read end\n EXIT_ON_COND(write(pipe_fd[1], "hi", 2) < 0); // Write "hi" to the pipe\n EXIT_ON_COND(close(pipe_fd[1]) != 0); // Close the write end\n} else { // Parent process\n char buf[BUFSIZ];\n\n EXIT_ON_COND(close(pipe_fd[1]) != 0); // Close the write end\n ssize_t n = read(pipe_fd[0], buf, sizeof(buf)); // Read data from the pipe into buf\n EXIT_ON_COND(n < 0); // Check for read() failure\n\n buf[n] = \'\\0\'; // Null-terminate the string\n printf("Received: %s\\n", buf); // Output the received message\n}\n')),(0,i.yg)("p",null,"In summary, the process creates the pipe and then calls ",(0,i.yg)("inlineCode",{parentName:"p"},"fork()")," to create a child process.\nBy default, the file descriptors created by ",(0,i.yg)("inlineCode",{parentName:"p"},"pipe()")," are shared with the child because the ",(0,i.yg)("em",{parentName:"p"},"(file descriptor table)")," is copied upon creation.\nTo better understand how this works, please refer to ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"this guide on the File Descriptor Table (FDT)"),"."),(0,i.yg)("p",null,"You can test your understanding of anonymous pipes by completing the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#task-anonymous-pipes-communication"},"Anonymous Pipes Communication task"),"."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/anonymous-pipes-limitation"},"Check your understanding by identifying the limitations of anonymous pipes")),(0,i.yg)("h3",{id:"named-pipes-fifos"},"Named Pipes (FIFOs)"),(0,i.yg)("p",null,"As we discussed, anonymous pipes are named so because they lack identifiers.\n",(0,i.yg)("strong",{parentName:"p"},"Named pipes")," address this limitation by creating a ",(0,i.yg)("em",{parentName:"p"},"special")," file on disk that serves as an identifier for the pipe."),(0,i.yg)("p",null,"You might think that interacting with a file would result in a performance loss compared to anonymous pipes, but this is not the case.\nThe FIFO file acts merely as ",(0,i.yg)("strong",{parentName:"p"},"a handler")," within the filesystem, which is used to write data to a buffer inside the kernel.\nThis buffer is responsible for holding the data that is passed between processes, not the filesystem itself."),(0,i.yg)("p",null,"Keep in mind that reading from and writing to a FIFO is not the same as interacting with a regular file - ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," will block if the pipe is empty and will return ",(0,i.yg)("inlineCode",{parentName:"p"},"EOF")," when the peer closes the pipe."),(0,i.yg)("p",null,"You can practice working with named pipes by completing the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab10#task-named-pipes-communication"},"Named Pipes Communication task"),"."),(0,i.yg)("h3",{id:"redirections"},"Redirections"),(0,i.yg)("p",null,"Although not directly related, redirections (e.g., ",(0,i.yg)("inlineCode",{parentName:"p"},"ls > file.txt"),") operate similarly to pipes.\nA process creates a new file descriptor, updates its ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),", and then creates the child process.\nYou can explore the similarities with pipes further in ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-redirections"},"this guide on redirections"),"."),(0,i.yg)("h2",{id:"guide-simple-file-operations"},"Guide: Simple File Operations"),(0,i.yg)("p",null,"To manipulate the file (read its contents, modify them, change its size etc.), each process must first get a ",(0,i.yg)("strong",{parentName:"p"},"handler")," to this file.\nThink of this handler as an object by which the process can identify and refer to the file."),(0,i.yg)("p",null,"Now take a look at the code examples in ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/guides/simple-file-operations/support"),".\nEach of them reads the contents of ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt"),", modifies them, and then reads the previously modified file again.\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to compile the C code, and ",(0,i.yg)("inlineCode",{parentName:"p"},"make java-file-operations")," to compile the Java code."),(0,i.yg)("p",null,"Now run the programs repeatedly in whatever order you wish:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../simple-file-operations/support$ python3 file_operations.py\nFile contents are: OS Rullz!\nWrote new data to file\nFile contents are: Python was here!\n\nstudent@os:~/.../simple-file-operations/support$ ./file_operations # from the C code\nFile contents are: Python was here!\nWrote new data to file\nFile contents are: C was here!\n\nstudent@os:~/.../simple-file-operations/support$ java FileOperations\nFile contents are: Python was here!\nWrote new data to file\nFile contents are: Java was here!\n")),(0,i.yg)("p",null,"Note that each piece of code creates a variable, which is then used as a handler."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/file-handler-c"},"Quiz")),(0,i.yg)("h2",{id:"guide-redirections"},"Guide: Redirections"),(0,i.yg)("p",null,"In the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"File Descriptors section"),", we mentioned redirections such as ",(0,i.yg)("inlineCode",{parentName:"p"},'echo "OS Rullz!" > newfile.txt'),".\nWe said ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt")," has to be opened at some point.\nLet\u2019s explore the relevant system calls (",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"openat()"),") to see this in action:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../guides/redirections$ strace -e trace=open,openat,execve,dup2 -f sh -c "ls > file.txt"\nexecve("/usr/bin/sh", ["sh", "-c", "ls > file.txt"], 0x7fffe1383e78 /* 36 vars */) = 0\nopenat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3\ndup2(3, 1) = 1\nstrace: Process 77547 attached\n[pid 77547] execve("/usr/bin/ls", ["ls"], 0x55ebb9b2dbf8 /* 36 vars */) = 0\n[...]\n')),(0,i.yg)("p",null,"Notice that we used ",(0,i.yg)("inlineCode",{parentName:"p"},"sh -c")," to run ",(0,i.yg)("inlineCode",{parentName:"p"},"ls > file.txt"),".\nRunning ",(0,i.yg)("inlineCode",{parentName:"p"},"strace -e trace=open,openat,execve,dup2 -f ls > file.txt")," would instead redirect the ",(0,i.yg)("inlineCode",{parentName:"p"},"strace")," output to ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt"),", hiding any system calls related to ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt"),".\nThis happens because, as we discussed earlier, redirection is transparent for the process being redirected.\nThe process still writes to its ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),", but ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," itself is now directed to the specified file."),(0,i.yg)("p",null,"Remember how processes are created using ",(0,i.yg)("inlineCode",{parentName:"p"},"fork()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"exec()"),", as shown in this diagram:"),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Launching a new command in Bash",src:t(5973).A})),(0,i.yg)("p",null,"In our case, the main process is ",(0,i.yg)("inlineCode",{parentName:"p"},'sh -c "ls > file.txt"'),".\nIn the ",(0,i.yg)("inlineCode",{parentName:"p"},"strace")," output, we see it opens ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt")," on file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"3"),", then uses ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"dup2(3, 1)"))," to redirect file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"1")," to the same ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nIt then ",(0,i.yg)("strong",{parentName:"p"},"forks")," a child process and calls ",(0,i.yg)("inlineCode",{parentName:"p"},"execve()"),"."),(0,i.yg)("p",null,(0,i.yg)("inlineCode",{parentName:"p"},"execve")," replaces the virtual address space (VAS) of the current process but retains the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"file descriptor table"),".\nThis preserve the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," of the parent process, thus the redirection to ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt")," remains effective in the new process as well."),(0,i.yg)("h3",{id:"dupdup2---atomic-io"},(0,i.yg)("inlineCode",{parentName:"h3"},"dup()/dup2()")," - Atomic IO"),(0,i.yg)("p",null,"If you're not familiar with the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"dup()")," syscall"),", it essentially creates a new file descriptor pointing to an existing ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nUnlike ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", as discussed in the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"file descriptor table guide"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"dup()")," doesn\u2019t create a fresh open file structure."),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2(old_fd, new_fd)")," variant closes ",(0,i.yg)("inlineCode",{parentName:"p"},"new_fd")," before making it point to the same open file structure as ",(0,i.yg)("inlineCode",{parentName:"p"},"old_fd"),".\nWhile this might seem like a combination of ",(0,i.yg)("inlineCode",{parentName:"p"},"close(new_fd)")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"open(old_fd)"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2()")," is actually atomic, which prevents race conditions."),(0,i.yg)("p",null,"To see why atomicity matters, review the code in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/redirect_parallel.c"),", compile it, and run it."),(0,i.yg)("p",null,"You\u2019ll find that ",(0,i.yg)("inlineCode",{parentName:"p"},"redirect_stderr_file.txt")," contains ",(0,i.yg)("inlineCode",{parentName:"p"},"Message for STDOUT"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"redirect_stdout_file.txt")," contains ",(0,i.yg)("inlineCode",{parentName:"p"},"Message for STDERR"),".\nInvestigate the code to understand where the race condition occurred."),(0,i.yg)("p",null,"While a ",(0,i.yg)("inlineCode",{parentName:"p"},"mutex")," around the ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," sequence could fix this, it can make the code cumbersome.\nInstead, follow the ",(0,i.yg)("inlineCode",{parentName:"p"},"FIXME")," comments for a more elegant solution using ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2()"),"."),(0,i.yg)("h2",{id:"guide-file-descriptor-table"},"Guide: File Descriptor Table"),(0,i.yg)("p",null,"Just as each process has its own Virtual Address Space for memory access, it also maintains its own ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table")," (FDT) for managing open files.\nIn this section we will explore how the process structures change when executing syscalls like ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"close()"),"."),(0,i.yg)("p",null,"Upon startup, every process has three ",(0,i.yg)("strong",{parentName:"p"},"file descriptors")," that correspond to standard input (",(0,i.yg)("inlineCode",{parentName:"p"},"stdin"),"), standard output (",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),"), and standard error (",(0,i.yg)("inlineCode",{parentName:"p"},"stderr"),").\nThese descriptors are inherited from the parent process and occupy the ",(0,i.yg)("strong",{parentName:"p"},"first three")," entries in the process's ",(0,i.yg)("strong",{parentName:"p"},"FDT"),"."),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"fd"),(0,i.yg)("th",{parentName:"tr",align:null},"Open File Struct"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"stdin"))),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"1"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"stdout"))),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"2"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"stderr"))))),(0,i.yg)("p",null,"Each entry ",(0,i.yg)("strong",{parentName:"p"},"points to")," an ",(0,i.yg)("strong",{parentName:"p"},"open file structure")," which stores data about the current session:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Permissions"),": define how the file can be accessed (read, write, or execute);\nthese are the options passed to ",(0,i.yg)("inlineCode",{parentName:"li"},"open()"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Offset"),": the current position inside the file from which the next read or write operation will occur."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Reference Count"),": The number of file descriptors referencing this open file structure."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Inode Pointer"),": A pointer to the inode structure that contains both the data and metadata associated with the file.")),(0,i.yg)("p",null,"These ",(0,i.yg)("strong",{parentName:"p"},"Open File Structures")," are held in the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table (OFT)"),", which is global across the system.\nWhenever a new file is opened, a new entry is created in this table."),(0,i.yg)("p",null,"To illustrate this, let's consider a code snippet and examine how the File Descriptor Table and Open File Table would appear.\nWe will focus on ",(0,i.yg)("inlineCode",{parentName:"p"},"fd"),", permissions, offset, and reference count, as the inode pointer is not relevant at this moment.\nFor simplicity, we'll also omit the standard file descriptors, as they remain unchanged."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'int fd = open("file.txt", O_RDONLY);\nint fd2 = open("file2.txt", O_WRONLY | O_APPEND);\nint fd3 = open("file.txt", O_RDWR);\n')),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"OFT index"),(0,i.yg)("th",{parentName:"tr",align:null},"Path"),(0,i.yg)("th",{parentName:"tr",align:null},"Perm"),(0,i.yg)("th",{parentName:"tr",align:null},"Off"),(0,i.yg)("th",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"th"},"RefCount")))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"...")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"123"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"r--"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"1")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"140"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file2.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"-w-"),(0,i.yg)("td",{parentName:"tr",align:null},"150"),(0,i.yg)("td",{parentName:"tr",align:null},"1")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"142"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"rw-"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"1")))),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"fd"),(0,i.yg)("th",{parentName:"tr",align:null},"Open File Struct (OFT index)"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"3"),(0,i.yg)("td",{parentName:"tr",align:null},"123")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"4"),(0,i.yg)("td",{parentName:"tr",align:null},"140")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"5"),(0,i.yg)("td",{parentName:"tr",align:null},"142")))),(0,i.yg)("p",null,"Let's discuss the changes from the OFT and FDT to understand what happened:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},'open("file.txt", O_RDONLY)')," created a new ",(0,i.yg)("strong",{parentName:"li"},"open file structure")," in the ",(0,i.yg)("strong",{parentName:"li"},"Open File Table")," for ",(0,i.yg)("inlineCode",{parentName:"li"},"file.txt"),".\nThe entry has read-only (",(0,i.yg)("inlineCode",{parentName:"li"},"O_RDONLY"),") permissions and offset ",(0,i.yg)("inlineCode",{parentName:"li"},"0"),", representing the start of the file.\nSubsequently, file descriptor ",(0,i.yg)("inlineCode",{parentName:"li"},"3")," was assigned to point to this OFT entry, and the reference counter was set to ",(0,i.yg)("inlineCode",{parentName:"li"},"1"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},'open("file2.txt", O_WRONLY)')," created a similar structure in the OFT for ",(0,i.yg)("inlineCode",{parentName:"li"},"file2.txt"),", but with write-only (",(0,i.yg)("inlineCode",{parentName:"li"},"O_WRONLY"),") permissions and an offset of ",(0,i.yg)("inlineCode",{parentName:"li"},"150"),", representing the end of the file (",(0,i.yg)("inlineCode",{parentName:"li"},"O_APPEND"),").\nIt then assigned this entry to file descriptor ",(0,i.yg)("inlineCode",{parentName:"li"},"4"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},'open("file.txt", O_RDWR)')," created a new ",(0,i.yg)("strong",{parentName:"li"},"open file structure")," for ",(0,i.yg)("inlineCode",{parentName:"li"},"file.txt")," and assigned it to file descriptor ",(0,i.yg)("inlineCode",{parentName:"li"},"5"),".")),(0,i.yg)("p",null,"At this point, one might wonder why the last ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," call didn't reuse the entry at file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"3")," and increase its reference counter instead.\nIt might seem logical, but doing so would lead to conflicts with the ",(0,i.yg)("strong",{parentName:"p"},"permissions")," and ",(0,i.yg)("strong",{parentName:"p"},"offset")," of the two open file structures.\n",(0,i.yg)("strong",{parentName:"p"},"Remember:")," each ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," call creates a new ",(0,i.yg)("strong",{parentName:"p"},"open file structure")," in the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table"),"."),(0,i.yg)("p",null,"This raises the question about the necessity for a reference counter.\nThe short answer is ",(0,i.yg)("inlineCode",{parentName:"p"},"dup()")," (or ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2()"),") syscall, which duplicates a file descriptor.\nLet's continue our previous example with the following snippet:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"// fd = 3 (from the previous snippet)\nint fd4 = dup(fd);\n")),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"OFT index"),(0,i.yg)("th",{parentName:"tr",align:null},"Path"),(0,i.yg)("th",{parentName:"tr",align:null},"Perm"),(0,i.yg)("th",{parentName:"tr",align:null},"Offset"),(0,i.yg)("th",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"th"},"RefCount")))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"...")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"123"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"r--"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"2")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"140"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file2.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"-w-"),(0,i.yg)("td",{parentName:"tr",align:null},"150"),(0,i.yg)("td",{parentName:"tr",align:null},"1")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"142"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"rw-"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"1")))),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"fd"),(0,i.yg)("th",{parentName:"tr",align:null},"Open File Struct (OFT index)"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"3"),(0,i.yg)("td",{parentName:"tr",align:null},"123")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"4"),(0,i.yg)("td",{parentName:"tr",align:null},"140")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"5"),(0,i.yg)("td",{parentName:"tr",align:null},"142")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"6"),(0,i.yg)("td",{parentName:"tr",align:null},"123")))),(0,i.yg)("p",null,"The call to ",(0,i.yg)("inlineCode",{parentName:"p"},"dup(fd)")," created a new file descriptor (",(0,i.yg)("inlineCode",{parentName:"p"},"6"),") that points to the same ",(0,i.yg)("strong",{parentName:"p"},"open file structure")," as its argument ",(0,i.yg)("inlineCode",{parentName:"p"},"fd")," (which equals ",(0,i.yg)("inlineCode",{parentName:"p"},"3")," in our example).\nThis operation also incremented the reference counter for the entry ",(0,i.yg)("inlineCode",{parentName:"p"},"123")," in the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table"),"."),(0,i.yg)("p",null,"As a result, operations performed on file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"3")," and file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"6")," are equivalent.\nFor instance, ",(0,i.yg)("inlineCode",{parentName:"p"},"read(3)")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"read(6)")," will both increment the shared file offset, while the offset of file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"5")," will remain unchanged.\nIf you want to see a concrete example of when duplicating file descriptors is useful, check out ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/guides/fd-table/support/redirect-stdout.c"),"."),(0,i.yg)("p",null,"Now that you know how to create entries in the ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table")," and the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table"),", it\u2019s important to understand how to remove them.\nCalling ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," will ",(0,i.yg)("strong",{parentName:"p"},"always")," free a file descriptor, but it will ",(0,i.yg)("strong",{parentName:"p"},"only decrement")," the reference counter of the ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nThe actual closing of the file occurs when the reference counter reaches ",(0,i.yg)("inlineCode",{parentName:"p"},"0"),"."),(0,i.yg)("h2",{id:"guide-libc-file-struct"},"Guide: libc ",(0,i.yg)("inlineCode",{parentName:"h2"},"FILE")," struct"),(0,i.yg)("p",null,"Now, we will take a short look at how the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"file descriptors")," are handled in libc.\nThe Software Stack chapter has taught us that applications generally interact with libraries which expose wrappers on top of syscalls.\nThe most important library in a POSIX system (such as Linux) is libc.\nAmong many others, it provides higher-level abstractions over I/O-related syscalls."),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Musl"),' (read just like "muscle") is a lightweight implementation of libc, which exposes the same API that you have used so far, while also being fit for embedded and OS development.\nFor example, ',(0,i.yg)("a",{parentName:"p",href:"https://unikraft.org/"},"Unikraft")," ",(0,i.yg)("a",{parentName:"p",href:"https://unikraft.org/docs/concepts/"},"unikernels")," may ",(0,i.yg)("a",{parentName:"p",href:"https://github.com/unikraft/lib-musl"},"use musl"),"."),(0,i.yg)("p",null,"First, it provides a ",(0,i.yg)("inlineCode",{parentName:"p"},"struct")," that groups together multiple data that is necessary when handling files.\nWe know from the example in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/simple-file-operations/file_operations.c")," that the file handler employed by libc is ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE *"),".\n",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," is just a ",(0,i.yg)("inlineCode",{parentName:"p"},"typedef")," for ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/internal/stdio_impl.h#L21"},(0,i.yg)("inlineCode",{parentName:"a"},"struct _IO_FILE")),".\nHere are the most important fields in ",(0,i.yg)("inlineCode",{parentName:"p"},"struct _IO_FILE"),":"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"struct _IO_FILE {\n int fd; /* File descriptor */\n\n unsigned flags; /* Flags with which `open()` was called */\n\n int mode; /* File permissions; passed to `open()` */\n\n off_t off; /* File offset from where to read / write */\n\n /**\n * Internal buffer used to make fewer costly `read()`/`write()`\n * syscalls.\n */\n unsigned char *buf;\n size_t buf_size;\n\n /* Pointers for reading and writing from/to the buffer defined above. */\n unsigned char *rpos, *rend;\n unsigned char *wend, *wpos;\n\n /* Function pointers to syscall wrappers. */\n size_t (*read)(FILE *, unsigned char *, size_t);\n size_t (*write)(FILE *, const unsigned char *, size_t);\n off_t (*seek)(FILE *, off_t, int);\n int (*close)(FILE *);\n\n /* Lock for concurrent file access. */\n volatile int lock;\n};\n")),(0,i.yg)("p",null,"As you might have imagined, this structure contains the underlying file descriptor, the ",(0,i.yg)("inlineCode",{parentName:"p"},"mode")," (read, write, truncate etc.) with which the file was opened, as well as the offset within the file from which the next read / write will start."),(0,i.yg)("p",null,"Libc also defines its own wrappers over commonly-used syscalls, such as ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"lseek()"),".\nThese syscalls themselves need to be implemented by the driver for each file system.\nThis is done by writing the required functions for each syscall and then populating ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L2093"},"this structure")," with pointers to them.\nYou will recognise quite a few syscalls: ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," etc."),(0,i.yg)("h3",{id:"printf-buffering"},(0,i.yg)("inlineCode",{parentName:"h3"},"printf()")," Buffering"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"buffering/support/printf_buffering.c"),".\nThose ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()")," calls obviously end up calling ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," at some point.\nRun the code under ",(0,i.yg)("inlineCode",{parentName:"p"},"strace"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/strace-printf"},"Quiz: What syscall does ",(0,i.yg)("inlineCode",{parentName:"a"},"printf")," use?")),(0,i.yg)("p",{parentName:"li"},"Since there is only one ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," syscall despite multiple calls to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()"),", it means that the strings given to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()")," as arguments are kept ",(0,i.yg)("em",{parentName:"p"},"somewhere")," until the syscall is made.\nThat ",(0,i.yg)("em",{parentName:"p"},"somewhere")," is precisely that buffer inside ",(0,i.yg)("inlineCode",{parentName:"p"},"struct _IO_FILE")," that we highlighted above.\nRemember that syscalls cause the system to change from user mode to kernel mode, which is time-consuming.\nInstead of performing one ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," syscall per call to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()"),", it is more efficient to copy the string passed to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()")," to an ",(0,i.yg)("strong",{parentName:"p"},"internal buffer")," inside libc (the ",(0,i.yg)("inlineCode",{parentName:"p"},"unsigned char *buf")," from above) and then at a given time (like when the buffer is full for example) ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," the whole buffer.\nThis results in far fewer ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," syscalls.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Now, it is interesting to see how we can force libc to dump that internal buffer.\nThe most direct way is by using the ",(0,i.yg)("inlineCode",{parentName:"p"},"fflush()")," library call, which is made for this exact purpose.\nBut we can be more subtle.\nAdd a ",(0,i.yg)("inlineCode",{parentName:"p"},"\\n")," in some of the strings printed in ",(0,i.yg)("inlineCode",{parentName:"p"},"buffering/support/printf_buffering.c"),".\nPlace them wherever you want (at the beginning, at the end, in the middle).\nRecompile the code and observe its change in behaviour under ",(0,i.yg)("inlineCode",{parentName:"p"},"strace"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/flush-libc-buffer"},"Quiz: How to get data out of ",(0,i.yg)("inlineCode",{parentName:"a"},"printf"),"'s buffer?")),(0,i.yg)("p",{parentName:"li"},"Now we know that I/O buffering ",(0,i.yg)("strong",{parentName:"p"},"does happen")," within libc.\nIf you need further convincing, check out the Musl implementation of ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L6"},(0,i.yg)("inlineCode",{parentName:"a"},"fread()")),", for example.\nIt first copies the ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L16"},"data previously saved in the internal buffer"),":"),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-c"},"if (f->rpos != f->rend) {\n /* First exhaust the buffer. */\n k = MIN(f->rend - f->rpos, l);\n memcpy(dest, f->rpos, k);\n f->rpos += k;\n dest += k;\n l -= k;\n}\n")),(0,i.yg)("p",{parentName:"li"},"Then, if more data is requested and the internal buffer isn't full, it refills it using ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L27"},"the internal ",(0,i.yg)("inlineCode",{parentName:"a"},"read()")," wrapper"),".\nThis wrapper also places the data inside the destination buffer."))),(0,i.yg)("h2",{id:"guide-file-mappings"},"Guide: File Mappings"),(0,i.yg)("p",null,"Mapping a file to the VAS of a process is similar to how shared libraries are loaded into the same VAS.\nIt's a fancier way of saying that the contents of a file are copied from a given offset within that file to a given address.\nWhat's nice about this is that the OS handles all offsets, addresses and memory allocations on its own, with a single highly versatile syscall: ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()"),"."),(0,i.yg)("p",null,"Let's run a ",(0,i.yg)("inlineCode",{parentName:"p"},"sleep")," process and inspect its memory zones:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ sleep 1000 & # start a `sleep` process in the background\n[1] 17579\n\nstudent@os:~$ cat /proc/$(pidof sleep)/maps\n55b7b646f000-55b7b6471000 r--p 00000000 103:07 6423964 /usr/bin/sleep\n55b7b6471000-55b7b6475000 r-xp 00002000 103:07 6423964 /usr/bin/sleep\n55b7b6475000-55b7b6477000 r--p 00006000 103:07 6423964 /usr/bin/sleep\n55b7b6478000-55b7b6479000 r--p 00008000 103:07 6423964 /usr/bin/sleep\n55b7b6479000-55b7b647a000 rw-p 00009000 103:07 6423964 /usr/bin/sleep\n55b7b677c000-55b7b679d000 rw-p 00000000 00:00 0 [heap]\n7fe442f61000-7fe44379d000 r--p 00000000 103:07 6423902 /usr/lib/locale/locale-archive\n7fe44379d000-7fe4437bf000 r--p 00000000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe4437bf000-7fe443937000 r-xp 00022000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe443937000-7fe443985000 r--p 0019a000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe443985000-7fe443989000 r--p 001e7000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe443989000-7fe44398b000 rw-p 001eb000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe44398b000-7fe443991000 rw-p 00000000 00:00 0\n7fe4439ad000-7fe4439ae000 r--p 00000000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439ae000-7fe4439d1000 r-xp 00001000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439d1000-7fe4439d9000 r--p 00024000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439da000-7fe4439db000 r--p 0002c000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439db000-7fe4439dc000 rw-p 0002d000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439dc000-7fe4439dd000 rw-p 00000000 00:00 0\n7ffd07aeb000-7ffd07b0c000 rw-p 00000000 00:00 0 [stack]\n7ffd07b8b000-7ffd07b8e000 r--p 00000000 00:00 0 [vvar]\n7ffd07b8e000-7ffd07b8f000 r-xp 00000000 00:00 0 [vdso]\nffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]\n")),(0,i.yg)("p",null,"In the output above, you can see that the ",(0,i.yg)("inlineCode",{parentName:"p"},".text"),", ",(0,i.yg)("inlineCode",{parentName:"p"},".rodata"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},".data")," sections for each dynamic library are mapped into the process\u2019s VAS, along with the sections of the main executable."),(0,i.yg)("p",null,"To understand how these mappings are created, let\u2019s explore a simpler example.\nBelow is an illustration of how ",(0,i.yg)("inlineCode",{parentName:"p"},"libc")," is loaded (or mapped) into the VAS of an ",(0,i.yg)("inlineCode",{parentName:"p"},"ls")," process."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace ls\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3\n[...]\nmmap(NULL, 2037344, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb313c9c000\nmmap(0x7fb313cbe000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7fb313cbe000\nmmap(0x7fb313e36000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19a000) = 0x7fb313e36000\nmmap(0x7fb313e84000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7fb313e84000\n')),(0,i.yg)("p",null,"For a quick recap on ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap(addr, length, prot, flags, fd, offset)"),", the fifth argument specifies the ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," to copy data from, while the sixth is the offset within the file from where to start copying."),(0,i.yg)("p",null,"In summary, when an executable runs, the loader uses ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," to reserve memory zones for its shared libraries.\nPerformance is not affected by this since pages are populated ",(0,i.yg)("strong",{parentName:"p"},"on-demand"),", when they\u2019re accessed for the first time."),(0,i.yg)("h3",{id:"file-io-vs-mmap"},"File I/O vs ",(0,i.yg)("inlineCode",{parentName:"h3"},"mmap()")),(0,i.yg)("p",null,"When it comes to dynamic libraries, ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap")," is unmatched.\nWith a single call, it handles ",(0,i.yg)("strong",{parentName:"p"},"address mapping"),", ",(0,i.yg)("strong",{parentName:"p"},"permission setting"),", and leverages demand paging to populate pages only when accessed.\nAdditionally, ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," fully supports ",(0,i.yg)("strong",{parentName:"p"},"copy-on-write (COW)"),", allowing libraries to share the same physical frames across multiple processes, which conserves memory and reduces load time."),(0,i.yg)("p",null,"In contrast, using ",(0,i.yg)("inlineCode",{parentName:"p"},"read")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write")," would require loading the entire library into physical memory for each process individually, missing out on both the ",(0,i.yg)("strong",{parentName:"p"},"copy-on-write")," and ",(0,i.yg)("strong",{parentName:"p"},"demand paging")," benefits."),(0,i.yg)("p",null,"For regular files, however, the choice isn\u2019t always straightforward.\nThe main sources of overhead for ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," include managing virtual memory mappings - which can lead to ",(0,i.yg)("strong",{parentName:"p"},"TLB flushes")," - and the cost of page faults due to ",(0,i.yg)("strong",{parentName:"p"},"demand paging"),"."),(0,i.yg)("p",null,"On the plus side, ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," excels with random access patterns, efficiently reusing mapped pages.\nIt is also great for operating large amounts of data, as it enables the kernel to automatically unload and reload pages as needed when memory when under memory pressure."),(0,i.yg)("p",null,"A concrete scenario where these downsides outweigh the benefits of ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," is one-time, sequential I/O.\nIf you\u2019re simply planning to read or write a file in one go, ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," are the way to go."),(0,i.yg)("h2",{id:"guide-reading-linux-directories"},"Guide: Reading Linux Directories"),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Everything in Linux is a file."),"\nThis statement says that the Linux OS treats every entry in a file system (regular file, directory, block device, char device, link, UNIX socket) as a file.\nThis unified approach simplifies file handling, allowing a single interface to interact with various types of entries.\nLet's see how this works in practice:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"guides/reading-linux-dirs/support/")," and checkout ",(0,i.yg)("inlineCode",{parentName:"p"},"dir_ops.c"),".\nThis code creates a directory ",(0,i.yg)("inlineCode",{parentName:"p"},"dir"),", if it does not exists, and attempts to open it the same way we would open a regular file.\nCompile and run the code."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../reading-linux-dirs/support$ ./dir_ops\n12:45:34 FATAL dir_ops.c:17: fopen: Is a directory\n")),(0,i.yg)("p",{parentName:"li"},"The error message is crystal clear: we cannot use ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," on directories.\nSo the ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure is unsuited for directories.\nTherefore, this handler is not generic enough for a regular Linux filesystem, and we have to use a lower-level function."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/fopen-syscall"},"Quiz - What syscall does fopen() use?"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Now that we know that ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," relies ",(0,i.yg)("inlineCode",{parentName:"p"},"openat()"),", let's try using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"open()")),", which wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"openat()")," but offers a simpler interface."),(0,i.yg)("p",{parentName:"li"},"Inspect, compile and run the code ",(0,i.yg)("inlineCode",{parentName:"p"},"dir_ops_syscalls.c"),"."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/...reading-linux-dirs/support$ ./dir_ops_syscalls\nDirectory file descriptor is: 3\n")),(0,i.yg)("p",{parentName:"li"},"This output proves that the ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," syscall is capable of also handling directories, so it's closer to what we want."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Note:")," that it is rather uncommon to use ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," for directories.\nMost of the time, ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/opendir.3.html"},(0,i.yg)("inlineCode",{parentName:"a"},"opendir()"))," is used instead."))),(0,i.yg)("p",null,"In conclusion, the key difference between ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," is in the type of handler they return.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure from ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," is suited only for regular files, while the ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," returned by ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," is more flexible.\nThe differences between the two handlers are explored in the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"file descriptors section"),"."))}m.isMDXComponent=!0},5973:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/fork-exec-e8ff2e7cb057592463ccc850bdaa0228.svg"},9234:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/piped-commands-118a36fba312c6bea5270dba74d653e2.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/94849ae2.78cc8b5c.js b/75/assets/js/94849ae2.78cc8b5c.js deleted file mode 100644 index cab3573981..0000000000 --- a/75/assets/js/94849ae2.78cc8b5c.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1055],{5680:(e,n,t)=>{t.d(n,{xA:()=>d,yg:()=>c});var a=t(6540);function i(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function l(e){for(var n=1;n=0||(i[t]=e[t]);return i}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var p=a.createContext({}),s=function(e){var n=a.useContext(p),t=n;return e&&(t="function"==typeof e?e(n):l(l({},n),e)),t},d=function(e){var n=s(e.components);return a.createElement(p.Provider,{value:n},e.children)},g="mdxType",m={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},y=a.forwardRef((function(e,n){var t=e.components,i=e.mdxType,r=e.originalType,p=e.parentName,d=o(e,["components","mdxType","originalType","parentName"]),g=s(t),y=i,c=g["".concat(p,".").concat(y)]||g[y]||m[y]||r;return t?a.createElement(c,l(l({ref:n},d),{},{components:t})):a.createElement(c,l({ref:n},d))}));function c(e,n){var t=arguments,i=n&&n.mdxType;if("string"==typeof e||i){var r=t.length,l=new Array(r);l[0]=y;var o={};for(var p in n)hasOwnProperty.call(n,p)&&(o[p]=n[p]);o.originalType=e,o[g]="string"==typeof e?e:i,l[1]=o;for(var s=2;s{t.r(n),t.d(n,{assets:()=>p,contentTitle:()=>l,default:()=>m,frontMatter:()=>r,metadata:()=>o,toc:()=>s});var a=t(8168),i=(t(6540),t(5680));const r={},l="Lab 9 - File Descriptors",o={unversionedId:"IO/lab9",id:"IO/lab9",title:"Lab 9 - File Descriptors",description:"Task: My cat",source:"@site/docs/IO/lab9.md",sourceDirName:"IO",slug:"/IO/lab9",permalink:"/operating-systems/75/IO/lab9",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Syscall Used by `fopen()`",permalink:"/operating-systems/75/IO/Questions/fopen-syscall"},next:{title:"Lab 10 - Inter-Process Communication",permalink:"/operating-systems/75/IO/lab10"}},p={},s=[{value:"Task: My cat",id:"task-my-cat",level:2},{value:"Task: Copy a File with mmap()",id:"task-copy-a-file-with-mmap",level:2},{value:"Task: Anonymous Pipes Communication",id:"task-anonymous-pipes-communication",level:2},{value:"File Descriptors",id:"file-descriptors",level:2},{value:"FILE Operations Explained",id:"file-operations-explained",level:3},{value:"File Descriptor Operations",id:"file-descriptor-operations",level:2},{value:"open()",id:"open",level:3},{value:"close()",id:"close",level:3},{value:"read() and write()",id:"read-and-write",level:3},{value:"lseek()",id:"lseek",level:3},{value:"Pipes",id:"pipes",level:2},{value:"Anonymous Pipes",id:"anonymous-pipes",level:3},{value:"Named Pipes (FIFOs)",id:"named-pipes-fifos",level:3},{value:"Redirections",id:"redirections",level:3},{value:"Guide: Simple File Operations",id:"guide-simple-file-operations",level:2},{value:"Guide: Redirections",id:"guide-redirections",level:2},{value:"dup()/dup2() - Atomic IO",id:"dupdup2---atomic-io",level:3},{value:"Guide: File Descriptor Table",id:"guide-file-descriptor-table",level:2},{value:"Guide: libc FILE struct",id:"guide-libc-file-struct",level:2},{value:"printf() Buffering",id:"printf-buffering",level:3},{value:"Guide: File Mappings",id:"guide-file-mappings",level:2},{value:"File I/O vs mmap()",id:"file-io-vs-mmap",level:3},{value:"Guide: Reading Linux Directories",id:"guide-reading-linux-directories",level:2}],d={toc:s},g="wrapper";function m(e){let{components:n,...r}=e;return(0,i.yg)(g,(0,a.A)({},d,r,{components:n,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"lab-9---file-descriptors"},"Lab 9 - File Descriptors"),(0,i.yg)("h2",{id:"task-my-cat"},"Task: My ",(0,i.yg)("inlineCode",{parentName:"h2"},"cat")),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/drills/tasks/my-cat/support/src")," and checkout ",(0,i.yg)("inlineCode",{parentName:"p"},"my_cat.c"),".\nWe propose to implement the Linux command ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," that reads one or more files, ",(0,i.yg)("strong",{parentName:"p"},"concatenates")," them (hence the name ",(0,i.yg)("inlineCode",{parentName:"p"},"cat"),"), and prints them to standard output."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Implement ",(0,i.yg)("inlineCode",{parentName:"p"},"rread()")," wrapper over ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("inlineCode",{parentName:"p"},"read()")," system call does not guarantee that it will read the requested number of bytes in a single call.\nThis happens when the file does not have enough bytes, or when ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," is interrupted by a signal.\n",(0,i.yg)("inlineCode",{parentName:"p"},"rread()")," will handle these situations, ensuring that it reads either ",(0,i.yg)("inlineCode",{parentName:"p"},"num_bytes")," or all available bytes.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Implement ",(0,i.yg)("inlineCode",{parentName:"p"},"wwrite()")," as a wrapper for ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),"."),(0,i.yg)("p",{parentName:"li"},"The ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," system call may not write the requested number of bytes in a single call.\nThis happens if ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," is interrupted by a signal.\n",(0,i.yg)("inlineCode",{parentName:"p"},"wwrite()")," will guarantee that it wrote the full ",(0,i.yg)("inlineCode",{parentName:"p"},"num_bytes"),", retrying as necessary until all data is successfully written or an error occurs.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Implement ",(0,i.yg)("inlineCode",{parentName:"p"},"cat()"),"."),(0,i.yg)("p",{parentName:"li"},"Use ",(0,i.yg)("inlineCode",{parentName:"p"},"rread()")," to read an entire file and ",(0,i.yg)("inlineCode",{parentName:"p"},"wwrite()")," to write the contents to standard output.\nKeep in mind that the buffer size may not fit the entire file at once."))),(0,i.yg)("p",null,"If you're having difficulties solving this exercise, go through ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"this reading material"),"."),(0,i.yg)("h2",{id:"task-copy-a-file-with-mmap"},"Task: Copy a File with ",(0,i.yg)("inlineCode",{parentName:"h2"},"mmap()")),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/drills/tasks/mmap_cp")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to generate ",(0,i.yg)("inlineCode",{parentName:"p"},"support"),".\nAs you know ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," can map files in memory, perform operations on them, and then write them back to the disk.\nLet's check how well it performs by comparing it to the ",(0,i.yg)("inlineCode",{parentName:"p"},"cp")," command.\nThe benchmarking is automated by ",(0,i.yg)("inlineCode",{parentName:"p"},"benchmark_cp.sh")," so focus on completing ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap_cp.c")," for now."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/syscalls-cp"},"Quiz: Checkout what syscalls ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")," uses")),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Open ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap_cp.c")," and complete the TODOs to map the files in memory and copy the contents.\nDo not forget to clean up by unmapping and closing the files."),(0,i.yg)("p",{parentName:"li"},"To test, run ",(0,i.yg)("inlineCode",{parentName:"p"},"make test-file")," to generate a 1MB file with random data, and then run ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap_cp test-file output.txt"),".\nEnsure they have the same content with a simple ",(0,i.yg)("inlineCode",{parentName:"p"},"diff"),": ",(0,i.yg)("inlineCode",{parentName:"p"},"diff test-file.txt output.txt"),".")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Compare your implementation to the ",(0,i.yg)("inlineCode",{parentName:"p"},"cp")," command.\nRun ",(0,i.yg)("inlineCode",{parentName:"p"},"make large-file")," to generate a 1GB file with random data, and then run ",(0,i.yg)("inlineCode",{parentName:"p"},"./benchmark_cp.sh"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/mmap-read-write-benchmark"},"Quiz: Debunk why ",(0,i.yg)("inlineCode",{parentName:"a"},"cp")," is winning")),(0,i.yg)("p",{parentName:"li"},"If you want a more generic answer, checkout this ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-mappings"},"guide on ",(0,i.yg)("inlineCode",{parentName:"a"},"mmap")," vs ",(0,i.yg)("inlineCode",{parentName:"a"},"read()-write()")),".")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"This demo would not be complete without some live analysis.\nUncomment the calls to ",(0,i.yg)("inlineCode",{parentName:"p"},"wait_for_input()")," and rerun the program.\nIn another terminal, run ",(0,i.yg)("inlineCode",{parentName:"p"},"cat /proc/$(pidof mmap_cp)/maps")," to see mapped files, and ",(0,i.yg)("inlineCode",{parentName:"p"},"ps -o pid,vsz,rss ")," to see how demand paging happens."))),(0,i.yg)("h2",{id:"task-anonymous-pipes-communication"},"Task: Anonymous Pipes Communication"),(0,i.yg)("p",null,"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"chapters/io/file-descriptors/drills/tasks/anon-pipes/support")," and run ",(0,i.yg)("inlineCode",{parentName:"p"},"make"),".\nIn this exercise, you'll implement client-server communication between a parent and a child process using an anonymous pipe.\nThe parent will act as the sender, while the child acts as the receiver, with both processes sharing messages through the pipe.\nSince pipes are unidirectional, each process should close the end of the pipe it does not use."),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Use the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/pipe.7.html"},(0,i.yg)("inlineCode",{parentName:"a"},"pipe()")," syscall")," to create the pipe.\nRemember, the first file descriptor (",(0,i.yg)("inlineCode",{parentName:"p"},"fds[0]"),") is the read end, and the second (",(0,i.yg)("inlineCode",{parentName:"p"},"fds[1]"),") is the write end, similar to how ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," are represented by file descriptors ",(0,i.yg)("inlineCode",{parentName:"p"},"0")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"1"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Hint:")," Use ",(0,i.yg)("inlineCode",{parentName:"p"},"exit")," to end the program."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/pipe-ends"},"Quiz: Discover why you cannot use either end of the pipe for reading or writing"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Solve the TODOs in ",(0,i.yg)("inlineCode",{parentName:"p"},"parent_loop")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"child_loop")," so that the application stops on ",(0,i.yg)("inlineCode",{parentName:"p"},"exit"),".\nEnsure each process closes the its pipe end before exiting to prevent indefinite blocking."),(0,i.yg)("blockquote",{parentName:"li"},(0,i.yg)("p",{parentName:"blockquote"},"Why is closing the pipe ends important?")),(0,i.yg)("p",{parentName:"li"},"The child process checks for the end of communication by reading from the pipe and checking for ",(0,i.yg)("inlineCode",{parentName:"p"},"EOF"),", which occurs when the write end is closed.\nWithout closing the write end, the child will block indefinitely in ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),".\nAs for the parent, it will block indefinitely in ",(0,i.yg)("inlineCode",{parentName:"p"},"wait()"),"."))),(0,i.yg)("h2",{id:"file-descriptors"},"File Descriptors"),(0,i.yg)("p",null,"You've most likely had to deal with files in the past.\nThe most common command that works with files is ",(0,i.yg)("inlineCode",{parentName:"p"},"cat"),".\nFor a quick refresher, let's write something to a file, and then read its contents."),(0,i.yg)("p",null,"You\u2019ve likely worked with files before;\nnow it\u2019s time to see what happens behind the scenes.\nThe most common way to read a file in Linux is by using the ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," command.\nFor a quick refresher, let\u2019s do a demo by writing some text to a file and then reading it back."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~/$ echo "OS Rullz!" # Print \'OS Rullz!\'\nOS Rullz!\nstudent@os:~/$ echo "OS Rullz!" > newfile.txt # redirect the output to newfile.txt\n## Let\'s check the contents of newfile.txt\nstudent@os:~/$ cat newfile.txt\nOS Rullz!\n')),(0,i.yg)("p",null,"If we were to implement this in C, we would use the ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure and write something like this:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'FILE *f = fopen("newfile.txt", "r");\nif (!f) {...} // handle error\n\nchar buf[1024];\nint rc = fread(buf, 1, sizeof(buf), f);\nif (rc < 0) {...} // handle error\n\nprintf("%s\\n", buf);\n')),(0,i.yg)("p",null,"For a complete example, check out this guide on ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-simple-file-operations"},"file operations in C, Python, and Java"),"."),(0,i.yg)("h3",{id:"file-operations-explained"},(0,i.yg)("inlineCode",{parentName:"h3"},"FILE")," Operations Explained"),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure is not the most straightforward method for performing file operations.\nIt is part of ",(0,i.yg)("inlineCode",{parentName:"p"},"libc")," and functions as a handler for working with files.\nThis is not particular to C, as most programming languages offer similar handlers."),(0,i.yg)("p",null,"Running ",(0,i.yg)("inlineCode",{parentName:"p"},"strace cat newfile.txt")," reveals that ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," (or ",(0,i.yg)("inlineCode",{parentName:"p"},"openat"),"), ",(0,i.yg)("inlineCode",{parentName:"p"},"fread()")," wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"fclose()")," wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"close()"),".\nAs you can see, the ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE"),"-related functions are just syscalls prefixed with ",(0,i.yg)("inlineCode",{parentName:"p"},"f-"),"."),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"th"},"FILE")," Operation"),(0,i.yg)("th",{parentName:"tr",align:null},"Syscall"),(0,i.yg)("th",{parentName:"tr",align:null},"Description"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fopen()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"open()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Opens a file and returns a file pointer.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fclose()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/close.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"close()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Closes the file associated with the pointer.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fread()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/read.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"read()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Reads data from the file into a buffer.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fwrite()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/write.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"write()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Writes data from a buffer to the file.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"fseek()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/lseek.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"lseek()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Moves the file position indicator.")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"truncate()")),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("a",{parentName:"td",href:"https://man7.org/linux/man-pages/man2/ftruncate.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"ftruncate()"))),(0,i.yg)("td",{parentName:"tr",align:null},"Truncates the file to a specified length.")))),(0,i.yg)("p",null,"The main distinction between ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," operations and their corresponding system calls is that the latter use a ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," to reference a file.\n",(0,i.yg)("strong",{parentName:"p"},"File descriptors")," are simply indexes into the process's ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table"),", which is the list of all currently open files for that process."),(0,i.yg)("p",null,"This concept is not entirely new, as each process has three default channels: ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"stderr"),".\nThese are, in fact, the first three entries in every process\u2019s ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table"),"."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/stderr-fd"},"Quiz: Test your intuition by finding the file descriptor of ",(0,i.yg)("inlineCode",{parentName:"a"},"stderr"))),(0,i.yg)("p",null,"Let's translate our previous example to illustrate how this change affects the implementation:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'int fd = open("newfile.txt", O_RDONLY)\nif (fd < 0) {...} // handle error\n\nchar buf[1024];\nint rc = read(fd, buf, sizeof(buf)); // Not complete, should\'ve used a while loop\nif (rc < 0) {...} // handle error\n\nbuf[rc] = \'\\0\'; // Null-terminate the buffer\nprintf("%s\\n", buf);\n')),(0,i.yg)("p",null,"To better understand the ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," API, you can either ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptor-operations"},"keep reading about file descriptor operations")," or checkout ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-reading-linux-directories"},"this guide on reading Linux directories"),"."),(0,i.yg)("p",null,"If you're interested in understanding how ",(0,i.yg)("inlineCode",{parentName:"p"},"libc")," utilizes file descriptors to simplify common operations, check out ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-libc-file-struct"},"this guide"),"."),(0,i.yg)("h2",{id:"file-descriptor-operations"},"File Descriptor Operations"),(0,i.yg)("p",null,"File descriptors are the primary means of referencing files in our system.\nThey are created, deleted, and manipulated through file interface operations, namely ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"lseek()"),".\nFrom a programmer's perspective, file descriptors are simply indexes into the process's ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table"),", which maintains a list of all currently open files for that process."),(0,i.yg)("p",null,"In this section, we will focus on how to utilize file descriptors to perform the same operations that ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," allows, and more.\nIf you want to delve deeper into file descriptors, we recommend exploring ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"this guide on the ",(0,i.yg)("strong",{parentName:"a"},"File Descriptor Table")),"."),(0,i.yg)("h3",{id:"open"},(0,i.yg)("inlineCode",{parentName:"h3"},"open()")),(0,i.yg)("p",null,"All processes start with three default file descriptors, inherited from the process's parent:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"stdin")," (standard input): 0"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"stdout")," (standard output): 1"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"stderr")," (standard error): 2")),(0,i.yg)("p",null,"To create new file descriptors (i.e. open new files), a process can use the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"open()"))," system call.\nIt receives the path to the file, some flags which are akin to the ",(0,i.yg)("inlineCode",{parentName:"p"},"mode")," string passed to ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()"),".\nAn optional ",(0,i.yg)("inlineCode",{parentName:"p"},"mode")," parameter that denotes the file's permissions if the ",(0,i.yg)("inlineCode",{parentName:"p"},"open")," must create it can also be provided.\nIf you use ",(0,i.yg)("inlineCode",{parentName:"p"},"O_CREAT"),", just remember to also pass ",(0,i.yg)("inlineCode",{parentName:"p"},"0644")," (",(0,i.yg)("inlineCode",{parentName:"p"},"rw-r--r--")," in octal, denoted by the first ",(0,i.yg)("inlineCode",{parentName:"p"},"0"),"), or permissions more restrictive."),(0,i.yg)("p",null,"Some other useful flags for ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," are:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"O_APPEND"),": place file cursor at the end"),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"O_CLOEXEC"),": close the file descriptor when ",(0,i.yg)("inlineCode",{parentName:"li"},"exec()")," is called.\nThis is useful because child processes inherit the file descriptors, and this can lead to security problems."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"O_TRUNC"),": truncate the file to length 0.")),(0,i.yg)("h3",{id:"close"},(0,i.yg)("inlineCode",{parentName:"h3"},"close()")),(0,i.yg)("p",null,"Once you are done with a file descriptor you should call ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," to free its ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nThis is similar to how you free memory once you are done with it."),(0,i.yg)("h3",{id:"read-and-write"},(0,i.yg)("inlineCode",{parentName:"h3"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"h3"},"write()")),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"read_bytes = read(fd, buf, num_bytes);\nwritten_bytes = write(fd, buf, num_bytes);\n")),(0,i.yg)("p",null,"As you know, verifying the return code of system calls is the way to go in general.\nThis is even more apparent when dealing with I/O syscalls, namely ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", which return the number of bytes read or written."),(0,i.yg)("p",null,"Syscalls returning the number of bytes might seem redundant, but once you hear about partial I/O operations, it is of utmost importance.\nIf your process was interrupted by a signal while reading or writing, it is up to you to continue from where it left off."),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Remember: It is mandatory that we always use ",(0,i.yg)("inlineCode",{parentName:"strong"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"strong"},"write()")," inside ",(0,i.yg)("inlineCode",{parentName:"strong"},"while")," loops."),"\nHigher-level functions like ",(0,i.yg)("inlineCode",{parentName:"p"},"fread()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"fwrite()")," also use ",(0,i.yg)("inlineCode",{parentName:"p"},"while")," loops when calling ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," respectively.\nYou can practice this by ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#task-my-cat"},"implementing your own ",(0,i.yg)("inlineCode",{parentName:"a"},"cat")," command"),"."),(0,i.yg)("p",null,"In the following sections, we'll use file descriptors and ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," to interact with some inter-process-communication mechanisms, such as pipes."),(0,i.yg)("h3",{id:"lseek"},(0,i.yg)("inlineCode",{parentName:"h3"},"lseek()")),(0,i.yg)("p",null,"As you know, reading or writing from a file always continues from where it left off.\nMost of the time you would read from a file monotonically so it makes sense to keep the interface clean and handle bookkeeping in the back."),(0,i.yg)("p",null,"For cases when you selectively update the file or jump around fetching data, or making updates, we have ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/lseek.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"lseek")),"."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"off_t lseek(int fd, off_t offset, int whence);\n")),(0,i.yg)("p",null,"Its parameters are pretty intuitive: ",(0,i.yg)("inlineCode",{parentName:"p"},"fd")," stands for the file descriptor and ",(0,i.yg)("inlineCode",{parentName:"p"},"offset")," stands for the offset.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"whence")," directive explains what ",(0,i.yg)("inlineCode",{parentName:"p"},"offset")," is relative to, and has the following values:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"SEEK_SET"),": the file offset is set to offset bytes."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"SEEK_CUR"),": The file offset is set to its current location plus offset bytes."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},"SEEK_END"),": the file offset is set to the size of the file plus offset bytes.")),(0,i.yg)("h2",{id:"pipes"},"Pipes"),(0,i.yg)("h3",{id:"anonymous-pipes"},"Anonymous Pipes"),(0,i.yg)("p",null,"In this session, we'll explore a new mean of Inter-Process Communication (IPC), namely ",(0,i.yg)("strong",{parentName:"p"},"the pipes"),".\nPipes are by no means something new, and you most probably played with them already in bash:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-bash"},"cat 'log_*.csv' | tr -s ' ' | cut -d ',' -f 2 | sort -u | head -n 10\n")),(0,i.yg)("p",null,"Using pipes (denoted as ",(0,i.yg)("inlineCode",{parentName:"p"},"|")," in the above example) enables linking the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," of multiple processes.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," of ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," is the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," of ",(0,i.yg)("inlineCode",{parentName:"p"},"tr"),", whose ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," is the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdin")," of ",(0,i.yg)("inlineCode",{parentName:"p"},"cut"),' and so on.\nThis "chain" of commands looks like this:'),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Piped Commands",src:t(9234).A})),(0,i.yg)("p",null,"So here we have a ",(0,i.yg)("strong",{parentName:"p"},"unidirectional")," stream of data that starts from ",(0,i.yg)("inlineCode",{parentName:"p"},"cat"),", is modified by each new command, and then is passed to the next one.\nWe can tell from the image above that the communication channel between any 2 adjacent commands allows one process to write to it while the other reads from it.\nFor example, there is no need for ",(0,i.yg)("inlineCode",{parentName:"p"},"cat")," to read any of ",(0,i.yg)("inlineCode",{parentName:"p"},"tr"),"'s output, only vice versa."),(0,i.yg)("p",null,"In UNIX, the need for such a channel is fulfilled by the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"pipe()")," syscall"),".\nImagine there's a literal pipe between any 2 adjacent commands in the image above, where data is what flows through this pipe ",(0,i.yg)("strong",{parentName:"p"},"in only a single way"),"."),(0,i.yg)("p",null,"Such pipes are known as ",(0,i.yg)("strong",{parentName:"p"},"anonymous pipes")," because they don\u2019t have identifiers.\nThey are created by a parent process, which shares them with its children.\nData written to an anonymous pipe is stored in a kernel-managed circular buffer, where it\u2019s available for related-processes to read."),(0,i.yg)("p",null,"The following example showcases a typical workflow with anonymous pipes in Unix:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'##define EXIT_ON_COND(cond) do { if (cond) exit(EXIT_FAILURE); } while (0)\n\n// pipe_fd[0] -> for reading\n// pipe_fd[1] -> for writing\nint pipe_fd[2];\n\nEXIT_ON_COND(pipe(pipe_fd) < 0); // Create the pipe\n\nint pid = fork(); // Fork to create a child process\nEXIT_ON_COND(pid < 0); // Check for fork() failure\n\nif (pid == 0) { // Child process\n EXIT_ON_COND(close(pipe_fd[0]) != 0); // Close the read end\n EXIT_ON_COND(write(pipe_fd[1], "hi", 2) < 0); // Write "hi" to the pipe\n EXIT_ON_COND(close(pipe_fd[1]) != 0); // Close the write end\n} else { // Parent process\n char buf[BUFSIZ];\n\n EXIT_ON_COND(close(pipe_fd[1]) != 0); // Close the write end\n ssize_t n = read(pipe_fd[0], buf, sizeof(buf)); // Read data from the pipe into buf\n EXIT_ON_COND(n < 0); // Check for read() failure\n\n buf[n] = \'\\0\'; // Null-terminate the string\n printf("Received: %s\\n", buf); // Output the received message\n}\n')),(0,i.yg)("p",null,"In summary, the process creates the pipe and then calls ",(0,i.yg)("inlineCode",{parentName:"p"},"fork()")," to create a child process.\nBy default, the file descriptors created by ",(0,i.yg)("inlineCode",{parentName:"p"},"pipe()")," are shared with the child because the ",(0,i.yg)("em",{parentName:"p"},"(file descriptor table)")," is copied upon creation.\nTo better understand how this works, please refer to ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"this guide on the File Descriptor Table (FDT)"),"."),(0,i.yg)("p",null,"You can test your understanding of anonymous pipes by completing the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#task-anonymous-pipes-communication"},"Anonymous Pipes Communication task"),"."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/anonymous-pipes-limitation"},"Check your understanding by identifying the limitations of anonymous pipes")),(0,i.yg)("h3",{id:"named-pipes-fifos"},"Named Pipes (FIFOs)"),(0,i.yg)("p",null,"As we discussed, anonymous pipes are named so because they lack identifiers.\n",(0,i.yg)("strong",{parentName:"p"},"Named pipes")," address this limitation by creating a ",(0,i.yg)("em",{parentName:"p"},"special")," file on disk that serves as an identifier for the pipe."),(0,i.yg)("p",null,"You might think that interacting with a file would result in a performance loss compared to anonymous pipes, but this is not the case.\nThe FIFO file acts merely as ",(0,i.yg)("strong",{parentName:"p"},"a handler")," within the filesystem, which is used to write data to a buffer inside the kernel.\nThis buffer is responsible for holding the data that is passed between processes, not the filesystem itself."),(0,i.yg)("p",null,"Keep in mind that reading from and writing to a FIFO is not the same as interacting with a regular file - ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," will block if the pipe is empty and will return ",(0,i.yg)("inlineCode",{parentName:"p"},"EOF")," when the peer closes the pipe."),(0,i.yg)("p",null,"You can practice working with named pipes by completing the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab10#task-named-pipes-communication"},"Named Pipes Communication task"),"."),(0,i.yg)("h3",{id:"redirections"},"Redirections"),(0,i.yg)("p",null,"Although not directly related, redirections (e.g., ",(0,i.yg)("inlineCode",{parentName:"p"},"ls > file.txt"),") operate similarly to pipes.\nA process creates a new file descriptor, updates its ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),", and then creates the child process.\nYou can explore the similarities with pipes further in ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-redirections"},"this guide on redirections"),"."),(0,i.yg)("h2",{id:"guide-simple-file-operations"},"Guide: Simple File Operations"),(0,i.yg)("p",null,"To manipulate the file (read its contents, modify them, change its size etc.), each process must first get a ",(0,i.yg)("strong",{parentName:"p"},"handler")," to this file.\nThink of this handler as an object by which the process can identify and refer to the file."),(0,i.yg)("p",null,"Now take a look at the code examples in ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/guides/simple-file-operations/support"),".\nEach of them reads the contents of ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt"),", modifies them, and then reads the previously modified file again.\nUse ",(0,i.yg)("inlineCode",{parentName:"p"},"make")," to compile the C code, and ",(0,i.yg)("inlineCode",{parentName:"p"},"make java-file-operations")," to compile the Java code."),(0,i.yg)("p",null,"Now run the programs repeatedly in whatever order you wish:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../simple-file-operations/support$ python3 file_operations.py\nFile contents are: OS Rullz!\nWrote new data to file\nFile contents are: Python was here!\n\nstudent@os:~/.../simple-file-operations/support$ ./file_operations # from the C code\nFile contents are: Python was here!\nWrote new data to file\nFile contents are: C was here!\n\nstudent@os:~/.../simple-file-operations/support$ java FileOperations\nFile contents are: Python was here!\nWrote new data to file\nFile contents are: Java was here!\n")),(0,i.yg)("p",null,"Note that each piece of code creates a variable, which is then used as a handler."),(0,i.yg)("p",null,(0,i.yg)("a",{parentName:"p",href:"Questions/file-handler-c"},"Quiz")),(0,i.yg)("h2",{id:"guide-redirections"},"Guide: Redirections"),(0,i.yg)("p",null,"In the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"File Descriptors section"),", we mentioned redirections such as ",(0,i.yg)("inlineCode",{parentName:"p"},'echo "OS Rullz!" > newfile.txt'),".\nWe said ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt")," has to be opened at some point.\nLet\u2019s explore the relevant system calls (",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"openat()"),") to see this in action:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../guides/redirections$ strace -e trace=open,openat,execve,dup2 -f sh -c "ls > file.txt"\nexecve("/usr/bin/sh", ["sh", "-c", "ls > file.txt"], 0x7fffe1383e78 /* 36 vars */) = 0\nopenat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3\ndup2(3, 1) = 1\nstrace: Process 77547 attached\n[pid 77547] execve("/usr/bin/ls", ["ls"], 0x55ebb9b2dbf8 /* 36 vars */) = 0\n[...]\n')),(0,i.yg)("p",null,"Notice that we used ",(0,i.yg)("inlineCode",{parentName:"p"},"sh -c")," to run ",(0,i.yg)("inlineCode",{parentName:"p"},"ls > file.txt"),".\nRunning ",(0,i.yg)("inlineCode",{parentName:"p"},"strace -e trace=open,openat,execve,dup2 -f ls > file.txt")," would instead redirect the ",(0,i.yg)("inlineCode",{parentName:"p"},"strace")," output to ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt"),", hiding any system calls related to ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt"),".\nThis happens because, as we discussed earlier, redirection is transparent for the process being redirected.\nThe process still writes to its ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),", but ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," itself is now directed to the specified file."),(0,i.yg)("p",null,"Remember how processes are created using ",(0,i.yg)("inlineCode",{parentName:"p"},"fork()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"exec()"),", as shown in this diagram:"),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Launching a new command in Bash",src:t(5973).A})),(0,i.yg)("p",null,"In our case, the main process is ",(0,i.yg)("inlineCode",{parentName:"p"},'sh -c "ls > file.txt"'),".\nIn the ",(0,i.yg)("inlineCode",{parentName:"p"},"strace")," output, we see it opens ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt")," on file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"3"),", then uses ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"dup2(3, 1)"))," to redirect file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"1")," to the same ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nIt then ",(0,i.yg)("strong",{parentName:"p"},"forks")," a child process and calls ",(0,i.yg)("inlineCode",{parentName:"p"},"execve()"),"."),(0,i.yg)("p",null,(0,i.yg)("inlineCode",{parentName:"p"},"execve")," replaces the virtual address space (VAS) of the current process but retains the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"file descriptor table"),".\nThis preserve the ",(0,i.yg)("inlineCode",{parentName:"p"},"stdout")," of the parent process, thus the redirection to ",(0,i.yg)("inlineCode",{parentName:"p"},"file.txt")," remains effective in the new process as well."),(0,i.yg)("h3",{id:"dupdup2---atomic-io"},(0,i.yg)("inlineCode",{parentName:"h3"},"dup()/dup2()")," - Atomic IO"),(0,i.yg)("p",null,"If you're not familiar with the ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"dup()")," syscall"),", it essentially creates a new file descriptor pointing to an existing ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nUnlike ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", as discussed in the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#guide-file-descriptor-table"},"file descriptor table guide"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"dup()")," doesn\u2019t create a fresh open file structure."),(0,i.yg)("p",null,"The ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2(old_fd, new_fd)")," variant closes ",(0,i.yg)("inlineCode",{parentName:"p"},"new_fd")," before making it point to the same open file structure as ",(0,i.yg)("inlineCode",{parentName:"p"},"old_fd"),".\nWhile this might seem like a combination of ",(0,i.yg)("inlineCode",{parentName:"p"},"close(new_fd)")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"open(old_fd)"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2()")," is actually atomic, which prevents race conditions."),(0,i.yg)("p",null,"To see why atomicity matters, review the code in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/redirect_parallel.c"),", compile it, and run it."),(0,i.yg)("p",null,"You\u2019ll find that ",(0,i.yg)("inlineCode",{parentName:"p"},"redirect_stderr_file.txt")," contains ",(0,i.yg)("inlineCode",{parentName:"p"},"Message for STDOUT"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"redirect_stdout_file.txt")," contains ",(0,i.yg)("inlineCode",{parentName:"p"},"Message for STDERR"),".\nInvestigate the code to understand where the race condition occurred."),(0,i.yg)("p",null,"While a ",(0,i.yg)("inlineCode",{parentName:"p"},"mutex")," around the ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," sequence could fix this, it can make the code cumbersome.\nInstead, follow the ",(0,i.yg)("inlineCode",{parentName:"p"},"FIXME")," comments for a more elegant solution using ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2()"),"."),(0,i.yg)("h2",{id:"guide-file-descriptor-table"},"Guide: File Descriptor Table"),(0,i.yg)("p",null,"Just as each process has its own Virtual Address Space for memory access, it also maintains its own ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table")," (FDT) for managing open files.\nIn this section we will explore how the process structures change when executing syscalls like ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},"close()"),"."),(0,i.yg)("p",null,"Upon startup, every process has three ",(0,i.yg)("strong",{parentName:"p"},"file descriptors")," that correspond to standard input (",(0,i.yg)("inlineCode",{parentName:"p"},"stdin"),"), standard output (",(0,i.yg)("inlineCode",{parentName:"p"},"stdout"),"), and standard error (",(0,i.yg)("inlineCode",{parentName:"p"},"stderr"),").\nThese descriptors are inherited from the parent process and occupy the ",(0,i.yg)("strong",{parentName:"p"},"first three")," entries in the process's ",(0,i.yg)("strong",{parentName:"p"},"FDT"),"."),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"fd"),(0,i.yg)("th",{parentName:"tr",align:null},"Open File Struct"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"stdin"))),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"1"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"stdout"))),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"2"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"stderr"))))),(0,i.yg)("p",null,"Each entry ",(0,i.yg)("strong",{parentName:"p"},"points to")," an ",(0,i.yg)("strong",{parentName:"p"},"open file structure")," which stores data about the current session:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Permissions"),": define how the file can be accessed (read, write, or execute);\nthese are the options passed to ",(0,i.yg)("inlineCode",{parentName:"li"},"open()"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Offset"),": the current position inside the file from which the next read or write operation will occur."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Reference Count"),": The number of file descriptors referencing this open file structure."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("strong",{parentName:"li"},"Inode Pointer"),": A pointer to the inode structure that contains both the data and metadata associated with the file.")),(0,i.yg)("p",null,"These ",(0,i.yg)("strong",{parentName:"p"},"Open File Structures")," are held in the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table (OFT)"),", which is global across the system.\nWhenever a new file is opened, a new entry is created in this table."),(0,i.yg)("p",null,"To illustrate this, let's consider a code snippet and examine how the File Descriptor Table and Open File Table would appear.\nWe will focus on ",(0,i.yg)("inlineCode",{parentName:"p"},"fd"),", permissions, offset, and reference count, as the inode pointer is not relevant at this moment.\nFor simplicity, we'll also omit the standard file descriptors, as they remain unchanged."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},'int fd = open("file.txt", O_RDONLY);\nint fd2 = open("file2.txt", O_WRONLY | O_APPEND);\nint fd3 = open("file.txt", O_RDWR);\n')),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"OFT index"),(0,i.yg)("th",{parentName:"tr",align:null},"Path"),(0,i.yg)("th",{parentName:"tr",align:null},"Perm"),(0,i.yg)("th",{parentName:"tr",align:null},"Off"),(0,i.yg)("th",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"th"},"RefCount")))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"...")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"123"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"r--"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"1")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"140"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file2.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"-w-"),(0,i.yg)("td",{parentName:"tr",align:null},"150"),(0,i.yg)("td",{parentName:"tr",align:null},"1")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"142"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"rw-"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"1")))),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"fd"),(0,i.yg)("th",{parentName:"tr",align:null},"Open File Struct (OFT index)"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"3"),(0,i.yg)("td",{parentName:"tr",align:null},"123")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"4"),(0,i.yg)("td",{parentName:"tr",align:null},"140")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"5"),(0,i.yg)("td",{parentName:"tr",align:null},"142")))),(0,i.yg)("p",null,"Let's discuss the changes from the OFT and FDT to understand what happened:"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},'open("file.txt", O_RDONLY)')," created a new ",(0,i.yg)("strong",{parentName:"li"},"open file structure")," in the ",(0,i.yg)("strong",{parentName:"li"},"Open File Table")," for ",(0,i.yg)("inlineCode",{parentName:"li"},"file.txt"),".\nThe entry has read-only (",(0,i.yg)("inlineCode",{parentName:"li"},"O_RDONLY"),") permissions and offset ",(0,i.yg)("inlineCode",{parentName:"li"},"0"),", representing the start of the file.\nSubsequently, file descriptor ",(0,i.yg)("inlineCode",{parentName:"li"},"3")," was assigned to point to this OFT entry, and the reference counter was set to ",(0,i.yg)("inlineCode",{parentName:"li"},"1"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},'open("file2.txt", O_WRONLY)')," created a similar structure in the OFT for ",(0,i.yg)("inlineCode",{parentName:"li"},"file2.txt"),", but with write-only (",(0,i.yg)("inlineCode",{parentName:"li"},"O_WRONLY"),") permissions and an offset of ",(0,i.yg)("inlineCode",{parentName:"li"},"150"),", representing the end of the file (",(0,i.yg)("inlineCode",{parentName:"li"},"O_APPEND"),").\nIt then assigned this entry to file descriptor ",(0,i.yg)("inlineCode",{parentName:"li"},"4"),"."),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("inlineCode",{parentName:"li"},'open("file.txt", O_RDWR)')," created a new ",(0,i.yg)("strong",{parentName:"li"},"open file structure")," for ",(0,i.yg)("inlineCode",{parentName:"li"},"file.txt")," and assigned it to file descriptor ",(0,i.yg)("inlineCode",{parentName:"li"},"5"),".")),(0,i.yg)("p",null,"At this point, one might wonder why the last ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," call didn't reuse the entry at file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"3")," and increase its reference counter instead.\nIt might seem logical, but doing so would lead to conflicts with the ",(0,i.yg)("strong",{parentName:"p"},"permissions")," and ",(0,i.yg)("strong",{parentName:"p"},"offset")," of the two open file structures.\n",(0,i.yg)("strong",{parentName:"p"},"Remember:")," each ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," call creates a new ",(0,i.yg)("strong",{parentName:"p"},"open file structure")," in the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table"),"."),(0,i.yg)("p",null,"This raises the question about the necessity for a reference counter.\nThe short answer is ",(0,i.yg)("inlineCode",{parentName:"p"},"dup()")," (or ",(0,i.yg)("inlineCode",{parentName:"p"},"dup2()"),") syscall, which duplicates a file descriptor.\nLet's continue our previous example with the following snippet:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"// fd = 3 (from the previous snippet)\nint fd4 = dup(fd);\n")),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"OFT index"),(0,i.yg)("th",{parentName:"tr",align:null},"Path"),(0,i.yg)("th",{parentName:"tr",align:null},"Perm"),(0,i.yg)("th",{parentName:"tr",align:null},"Offset"),(0,i.yg)("th",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"th"},"RefCount")))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"..."),(0,i.yg)("td",{parentName:"tr",align:null},"...")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"123"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"r--"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"2")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"140"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file2.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"-w-"),(0,i.yg)("td",{parentName:"tr",align:null},"150"),(0,i.yg)("td",{parentName:"tr",align:null},"1")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"142"),(0,i.yg)("td",{parentName:"tr",align:null},(0,i.yg)("inlineCode",{parentName:"td"},"file.txt")),(0,i.yg)("td",{parentName:"tr",align:null},"rw-"),(0,i.yg)("td",{parentName:"tr",align:null},"0"),(0,i.yg)("td",{parentName:"tr",align:null},"1")))),(0,i.yg)("table",null,(0,i.yg)("thead",{parentName:"table"},(0,i.yg)("tr",{parentName:"thead"},(0,i.yg)("th",{parentName:"tr",align:null},"fd"),(0,i.yg)("th",{parentName:"tr",align:null},"Open File Struct (OFT index)"))),(0,i.yg)("tbody",{parentName:"table"},(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"3"),(0,i.yg)("td",{parentName:"tr",align:null},"123")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"4"),(0,i.yg)("td",{parentName:"tr",align:null},"140")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"5"),(0,i.yg)("td",{parentName:"tr",align:null},"142")),(0,i.yg)("tr",{parentName:"tbody"},(0,i.yg)("td",{parentName:"tr",align:null},"6"),(0,i.yg)("td",{parentName:"tr",align:null},"123")))),(0,i.yg)("p",null,"The call to ",(0,i.yg)("inlineCode",{parentName:"p"},"dup(fd)")," created a new file descriptor (",(0,i.yg)("inlineCode",{parentName:"p"},"6"),") that points to the same ",(0,i.yg)("strong",{parentName:"p"},"open file structure")," as its argument ",(0,i.yg)("inlineCode",{parentName:"p"},"fd")," (which equals ",(0,i.yg)("inlineCode",{parentName:"p"},"3")," in our example).\nThis operation also incremented the reference counter for the entry ",(0,i.yg)("inlineCode",{parentName:"p"},"123")," in the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table"),"."),(0,i.yg)("p",null,"As a result, operations performed on file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"3")," and file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"6")," are equivalent.\nFor instance, ",(0,i.yg)("inlineCode",{parentName:"p"},"read(3)")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"read(6)")," will both increment the shared file offset, while the offset of file descriptor ",(0,i.yg)("inlineCode",{parentName:"p"},"5")," will remain unchanged.\nIf you want to see a concrete example of when duplicating file descriptors is useful, check out ",(0,i.yg)("inlineCode",{parentName:"p"},"file-descriptors/guides/fd-table/support/redirect-stdout.c"),"."),(0,i.yg)("p",null,"Now that you know how to create entries in the ",(0,i.yg)("strong",{parentName:"p"},"File Descriptor Table")," and the ",(0,i.yg)("strong",{parentName:"p"},"Open File Table"),", it\u2019s important to understand how to remove them.\nCalling ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," will ",(0,i.yg)("strong",{parentName:"p"},"always")," free a file descriptor, but it will ",(0,i.yg)("strong",{parentName:"p"},"only decrement")," the reference counter of the ",(0,i.yg)("strong",{parentName:"p"},"open file structure"),".\nThe actual closing of the file occurs when the reference counter reaches ",(0,i.yg)("inlineCode",{parentName:"p"},"0"),"."),(0,i.yg)("h2",{id:"guide-libc-file-struct"},"Guide: libc ",(0,i.yg)("inlineCode",{parentName:"h2"},"FILE")," struct"),(0,i.yg)("p",null,"Now, we will take a short look at how the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"file descriptors")," are handled in libc.\nThe Software Stack chapter has taught us that applications generally interact with libraries which expose wrappers on top of syscalls.\nThe most important library in a POSIX system (such as Linux) is libc.\nAmong many others, it provides higher-level abstractions over I/O-related syscalls."),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Musl"),' (read just like "muscle") is a lightweight implementation of libc, which exposes the same API that you have used so far, while also being fit for embedded and OS development.\nFor example, ',(0,i.yg)("a",{parentName:"p",href:"https://unikraft.org/"},"Unikraft")," ",(0,i.yg)("a",{parentName:"p",href:"https://unikraft.org/docs/concepts/"},"unikernels")," may ",(0,i.yg)("a",{parentName:"p",href:"https://github.com/unikraft/lib-musl"},"use musl"),"."),(0,i.yg)("p",null,"First, it provides a ",(0,i.yg)("inlineCode",{parentName:"p"},"struct")," that groups together multiple data that is necessary when handling files.\nWe know from the example in ",(0,i.yg)("inlineCode",{parentName:"p"},"support/simple-file-operations/file_operations.c")," that the file handler employed by libc is ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE *"),".\n",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," is just a ",(0,i.yg)("inlineCode",{parentName:"p"},"typedef")," for ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/internal/stdio_impl.h#L21"},(0,i.yg)("inlineCode",{parentName:"a"},"struct _IO_FILE")),".\nHere are the most important fields in ",(0,i.yg)("inlineCode",{parentName:"p"},"struct _IO_FILE"),":"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-c"},"struct _IO_FILE {\n int fd; /* File descriptor */\n\n unsigned flags; /* Flags with which `open()` was called */\n\n int mode; /* File permissions; passed to `open()` */\n\n off_t off; /* File offset from where to read / write */\n\n /**\n * Internal buffer used to make fewer costly `read()`/`write()`\n * syscalls.\n */\n unsigned char *buf;\n size_t buf_size;\n\n /* Pointers for reading and writing from/to the buffer defined above. */\n unsigned char *rpos, *rend;\n unsigned char *wend, *wpos;\n\n /* Function pointers to syscall wrappers. */\n size_t (*read)(FILE *, unsigned char *, size_t);\n size_t (*write)(FILE *, const unsigned char *, size_t);\n off_t (*seek)(FILE *, off_t, int);\n int (*close)(FILE *);\n\n /* Lock for concurrent file access. */\n volatile int lock;\n};\n")),(0,i.yg)("p",null,"As you might have imagined, this structure contains the underlying file descriptor, the ",(0,i.yg)("inlineCode",{parentName:"p"},"mode")," (read, write, truncate etc.) with which the file was opened, as well as the offset within the file from which the next read / write will start."),(0,i.yg)("p",null,"Libc also defines its own wrappers over commonly-used syscalls, such as ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"lseek()"),".\nThese syscalls themselves need to be implemented by the driver for each file system.\nThis is done by writing the required functions for each syscall and then populating ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L2093"},"this structure")," with pointers to them.\nYou will recognise quite a few syscalls: ",(0,i.yg)("inlineCode",{parentName:"p"},"open()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"close()")," ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"write()"),", ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," etc."),(0,i.yg)("h3",{id:"printf-buffering"},(0,i.yg)("inlineCode",{parentName:"h3"},"printf()")," Buffering"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"buffering/support/printf_buffering.c"),".\nThose ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()")," calls obviously end up calling ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," at some point.\nRun the code under ",(0,i.yg)("inlineCode",{parentName:"p"},"strace"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/strace-printf"},"Quiz: What syscall does ",(0,i.yg)("inlineCode",{parentName:"a"},"printf")," use?")),(0,i.yg)("p",{parentName:"li"},"Since there is only one ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," syscall despite multiple calls to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()"),", it means that the strings given to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()")," as arguments are kept ",(0,i.yg)("em",{parentName:"p"},"somewhere")," until the syscall is made.\nThat ",(0,i.yg)("em",{parentName:"p"},"somewhere")," is precisely that buffer inside ",(0,i.yg)("inlineCode",{parentName:"p"},"struct _IO_FILE")," that we highlighted above.\nRemember that syscalls cause the system to change from user mode to kernel mode, which is time-consuming.\nInstead of performing one ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," syscall per call to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()"),", it is more efficient to copy the string passed to ",(0,i.yg)("inlineCode",{parentName:"p"},"printf()")," to an ",(0,i.yg)("strong",{parentName:"p"},"internal buffer")," inside libc (the ",(0,i.yg)("inlineCode",{parentName:"p"},"unsigned char *buf")," from above) and then at a given time (like when the buffer is full for example) ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," the whole buffer.\nThis results in far fewer ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," syscalls.")),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Now, it is interesting to see how we can force libc to dump that internal buffer.\nThe most direct way is by using the ",(0,i.yg)("inlineCode",{parentName:"p"},"fflush()")," library call, which is made for this exact purpose.\nBut we can be more subtle.\nAdd a ",(0,i.yg)("inlineCode",{parentName:"p"},"\\n")," in some of the strings printed in ",(0,i.yg)("inlineCode",{parentName:"p"},"buffering/support/printf_buffering.c"),".\nPlace them wherever you want (at the beginning, at the end, in the middle).\nRecompile the code and observe its change in behaviour under ",(0,i.yg)("inlineCode",{parentName:"p"},"strace"),"."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/flush-libc-buffer"},"Quiz: How to get data out of ",(0,i.yg)("inlineCode",{parentName:"a"},"printf"),"'s buffer?")),(0,i.yg)("p",{parentName:"li"},"Now we know that I/O buffering ",(0,i.yg)("strong",{parentName:"p"},"does happen")," within libc.\nIf you need further convincing, check out the Musl implementation of ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L6"},(0,i.yg)("inlineCode",{parentName:"a"},"fread()")),", for example.\nIt first copies the ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L16"},"data previously saved in the internal buffer"),":"),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-c"},"if (f->rpos != f->rend) {\n /* First exhaust the buffer. */\n k = MIN(f->rend - f->rpos, l);\n memcpy(dest, f->rpos, k);\n f->rpos += k;\n dest += k;\n l -= k;\n}\n")),(0,i.yg)("p",{parentName:"li"},"Then, if more data is requested and the internal buffer isn't full, it refills it using ",(0,i.yg)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L27"},"the internal ",(0,i.yg)("inlineCode",{parentName:"a"},"read()")," wrapper"),".\nThis wrapper also places the data inside the destination buffer."))),(0,i.yg)("h2",{id:"guide-file-mappings"},"Guide: File Mappings"),(0,i.yg)("p",null,"Mapping a file to the VAS of a process is similar to how shared libraries are loaded into the same VAS.\nIt's a fancier way of saying that the contents of a file are copied from a given offset within that file to a given address.\nWhat's nice about this is that the OS handles all offsets, addresses and memory allocations on its own, with a single highly versatile syscall: ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()"),"."),(0,i.yg)("p",null,"Let's run a ",(0,i.yg)("inlineCode",{parentName:"p"},"sleep")," process and inspect its memory zones:"),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~$ sleep 1000 & # start a `sleep` process in the background\n[1] 17579\n\nstudent@os:~$ cat /proc/$(pidof sleep)/maps\n55b7b646f000-55b7b6471000 r--p 00000000 103:07 6423964 /usr/bin/sleep\n55b7b6471000-55b7b6475000 r-xp 00002000 103:07 6423964 /usr/bin/sleep\n55b7b6475000-55b7b6477000 r--p 00006000 103:07 6423964 /usr/bin/sleep\n55b7b6478000-55b7b6479000 r--p 00008000 103:07 6423964 /usr/bin/sleep\n55b7b6479000-55b7b647a000 rw-p 00009000 103:07 6423964 /usr/bin/sleep\n55b7b677c000-55b7b679d000 rw-p 00000000 00:00 0 [heap]\n7fe442f61000-7fe44379d000 r--p 00000000 103:07 6423902 /usr/lib/locale/locale-archive\n7fe44379d000-7fe4437bf000 r--p 00000000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe4437bf000-7fe443937000 r-xp 00022000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe443937000-7fe443985000 r--p 0019a000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe443985000-7fe443989000 r--p 001e7000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe443989000-7fe44398b000 rw-p 001eb000 103:07 6432810 /usr/lib/x86_64-linux-gnu/libc-2.31.so\n7fe44398b000-7fe443991000 rw-p 00000000 00:00 0\n7fe4439ad000-7fe4439ae000 r--p 00000000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439ae000-7fe4439d1000 r-xp 00001000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439d1000-7fe4439d9000 r--p 00024000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439da000-7fe4439db000 r--p 0002c000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439db000-7fe4439dc000 rw-p 0002d000 103:07 6429709 /usr/lib/x86_64-linux-gnu/ld-2.31.so\n7fe4439dc000-7fe4439dd000 rw-p 00000000 00:00 0\n7ffd07aeb000-7ffd07b0c000 rw-p 00000000 00:00 0 [stack]\n7ffd07b8b000-7ffd07b8e000 r--p 00000000 00:00 0 [vvar]\n7ffd07b8e000-7ffd07b8f000 r-xp 00000000 00:00 0 [vdso]\nffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0 [vsyscall]\n")),(0,i.yg)("p",null,"In the output above, you can see that the ",(0,i.yg)("inlineCode",{parentName:"p"},".text"),", ",(0,i.yg)("inlineCode",{parentName:"p"},".rodata"),", and ",(0,i.yg)("inlineCode",{parentName:"p"},".data")," sections for each dynamic library are mapped into the process\u2019s VAS, along with the sections of the main executable."),(0,i.yg)("p",null,"To understand how these mappings are created, let\u2019s explore a simpler example.\nBelow is an illustration of how ",(0,i.yg)("inlineCode",{parentName:"p"},"libc")," is loaded (or mapped) into the VAS of an ",(0,i.yg)("inlineCode",{parentName:"p"},"ls")," process."),(0,i.yg)("pre",null,(0,i.yg)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace ls\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3\n[...]\nmmap(NULL, 2037344, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb313c9c000\nmmap(0x7fb313cbe000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7fb313cbe000\nmmap(0x7fb313e36000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19a000) = 0x7fb313e36000\nmmap(0x7fb313e84000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7fb313e84000\n')),(0,i.yg)("p",null,"For a quick recap on ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap(addr, length, prot, flags, fd, offset)"),", the fifth argument specifies the ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," to copy data from, while the sixth is the offset within the file from where to start copying."),(0,i.yg)("p",null,"In summary, when an executable runs, the loader uses ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," to reserve memory zones for its shared libraries.\nPerformance is not affected by this since pages are populated ",(0,i.yg)("strong",{parentName:"p"},"on-demand"),", when they\u2019re accessed for the first time."),(0,i.yg)("h3",{id:"file-io-vs-mmap"},"File I/O vs ",(0,i.yg)("inlineCode",{parentName:"h3"},"mmap()")),(0,i.yg)("p",null,"When it comes to dynamic libraries, ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap")," is unmatched.\nWith a single call, it handles ",(0,i.yg)("strong",{parentName:"p"},"address mapping"),", ",(0,i.yg)("strong",{parentName:"p"},"permission setting"),", and leverages demand paging to populate pages only when accessed.\nAdditionally, ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," fully supports ",(0,i.yg)("strong",{parentName:"p"},"copy-on-write (COW)"),", allowing libraries to share the same physical frames across multiple processes, which conserves memory and reduces load time."),(0,i.yg)("p",null,"In contrast, using ",(0,i.yg)("inlineCode",{parentName:"p"},"read")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write")," would require loading the entire library into physical memory for each process individually, missing out on both the ",(0,i.yg)("strong",{parentName:"p"},"copy-on-write")," and ",(0,i.yg)("strong",{parentName:"p"},"demand paging")," benefits."),(0,i.yg)("p",null,"For regular files, however, the choice isn\u2019t always straightforward.\nThe main sources of overhead for ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," include managing virtual memory mappings - which can lead to ",(0,i.yg)("strong",{parentName:"p"},"TLB flushes")," - and the cost of page faults due to ",(0,i.yg)("strong",{parentName:"p"},"demand paging"),"."),(0,i.yg)("p",null,"On the plus side, ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," excels with random access patterns, efficiently reusing mapped pages.\nIt is also great for operating large amounts of data, as it enables the kernel to automatically unload and reload pages as needed when memory when under memory pressure."),(0,i.yg)("p",null,"A concrete scenario where these downsides outweigh the benefits of ",(0,i.yg)("inlineCode",{parentName:"p"},"mmap()")," is one-time, sequential I/O.\nIf you\u2019re simply planning to read or write a file in one go, ",(0,i.yg)("inlineCode",{parentName:"p"},"read()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"write()")," are the way to go."),(0,i.yg)("h2",{id:"guide-reading-linux-directories"},"Guide: Reading Linux Directories"),(0,i.yg)("p",null,(0,i.yg)("strong",{parentName:"p"},"Everything in Linux is a file."),"\nThis statement says that the Linux OS treats every entry in a file system (regular file, directory, block device, char device, link, UNIX socket) as a file.\nThis unified approach simplifies file handling, allowing a single interface to interact with various types of entries.\nLet's see how this works in practice:"),(0,i.yg)("ol",null,(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Navigate to ",(0,i.yg)("inlineCode",{parentName:"p"},"guides/reading-linux-dirs/support/")," and checkout ",(0,i.yg)("inlineCode",{parentName:"p"},"dir_ops.c"),".\nThis code creates a directory ",(0,i.yg)("inlineCode",{parentName:"p"},"dir"),", if it does not exists, and attempts to open it the same way we would open a regular file.\nCompile and run the code."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../reading-linux-dirs/support$ ./dir_ops\n12:45:34 FATAL dir_ops.c:17: fopen: Is a directory\n")),(0,i.yg)("p",{parentName:"li"},"The error message is crystal clear: we cannot use ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," on directories.\nSo the ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure is unsuited for directories.\nTherefore, this handler is not generic enough for a regular Linux filesystem, and we have to use a lower-level function."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("a",{parentName:"p",href:"Questions/fopen-syscall"},"Quiz - What syscall does fopen() use?"))),(0,i.yg)("li",{parentName:"ol"},(0,i.yg)("p",{parentName:"li"},"Now that we know that ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," relies ",(0,i.yg)("inlineCode",{parentName:"p"},"openat()"),", let's try using ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.yg)("inlineCode",{parentName:"a"},"open()")),", which wraps ",(0,i.yg)("inlineCode",{parentName:"p"},"openat()")," but offers a simpler interface."),(0,i.yg)("p",{parentName:"li"},"Inspect, compile and run the code ",(0,i.yg)("inlineCode",{parentName:"p"},"dir_ops_syscalls.c"),"."),(0,i.yg)("pre",{parentName:"li"},(0,i.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/...reading-linux-dirs/support$ ./dir_ops_syscalls\nDirectory file descriptor is: 3\n")),(0,i.yg)("p",{parentName:"li"},"This output proves that the ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," syscall is capable of also handling directories, so it's closer to what we want."),(0,i.yg)("p",{parentName:"li"},(0,i.yg)("strong",{parentName:"p"},"Note:")," that it is rather uncommon to use ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," for directories.\nMost of the time, ",(0,i.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/opendir.3.html"},(0,i.yg)("inlineCode",{parentName:"a"},"opendir()"))," is used instead."))),(0,i.yg)("p",null,"In conclusion, the key difference between ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," and ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," is in the type of handler they return.\nThe ",(0,i.yg)("inlineCode",{parentName:"p"},"FILE")," structure from ",(0,i.yg)("inlineCode",{parentName:"p"},"fopen()")," is suited only for regular files, while the ",(0,i.yg)("strong",{parentName:"p"},"file descriptor")," returned by ",(0,i.yg)("inlineCode",{parentName:"p"},"open()")," is more flexible.\nThe differences between the two handlers are explored in the ",(0,i.yg)("a",{parentName:"p",href:"/operating-systems/75/IO/lab9#file-descriptors"},"file descriptors section"),"."))}m.isMDXComponent=!0},5973:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/fork-exec-e8ff2e7cb057592463ccc850bdaa0228.svg"},9234:(e,n,t)=>{t.d(n,{A:()=>a});const a=t.p+"assets/images/piped-commands-118a36fba312c6bea5270dba74d653e2.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/99a00d4b.90cb41ee.js b/75/assets/js/99a00d4b.034b19e1.js similarity index 95% rename from 75/assets/js/99a00d4b.90cb41ee.js rename to 75/assets/js/99a00d4b.034b19e1.js index 0ef0ea13d2..dd62e6c387 100644 --- a/75/assets/js/99a00d4b.90cb41ee.js +++ b/75/assets/js/99a00d4b.034b19e1.js @@ -1 +1 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8192],{5680:(e,t,r)=>{r.d(t,{xA:()=>c,yg:()=>d});var n=r(6540);function i(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function s(e){for(var t=1;t=0||(i[r]=e[r]);return i}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(i[r]=e[r])}return i}var l=n.createContext({}),p=function(e){var t=n.useContext(l),r=t;return e&&(r="function"==typeof e?e(t):s(s({},t),e)),r},c=function(e){var t=p(e.components);return n.createElement(l.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},y=n.forwardRef((function(e,t){var r=e.components,i=e.mdxType,o=e.originalType,l=e.parentName,c=a(e,["components","mdxType","originalType","parentName"]),u=p(r),y=i,d=u["".concat(l,".").concat(y)]||u[y]||f[y]||o;return r?n.createElement(d,s(s({ref:t},c),{},{components:r})):n.createElement(d,s({ref:t},c))}));function d(e,t){var r=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var o=r.length,s=new Array(o);s[0]=y;var a={};for(var l in t)hasOwnProperty.call(t,l)&&(a[l]=t[l]);a.originalType=e,a[u]="string"==typeof e?e:i,s[1]=a;for(var p=2;p{r.r(t),r.d(t,{assets:()=>l,contentTitle:()=>s,default:()=>f,frontMatter:()=>o,metadata:()=>a,toc:()=>p});var n=r(8168),i=(r(6540),r(5680));const o={},s="Client-Server Number of Copies",a={unversionedId:"IO/Questions/server-copies",id:"IO/Questions/server-copies",title:"Client-Server Number of Copies",description:"Question Text",source:"@site/docs/IO/Questions/server-copies.md",sourceDirName:"IO/Questions",slug:"/IO/Questions/server-copies",permalink:"/operating-systems/75/IO/Questions/server-copies",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Firefox: TCP or UDP?",permalink:"/operating-systems/75/IO/Questions/firefox-tcp-udp"},next:{title:"Fewer than Two Copies",permalink:"/operating-systems/75/IO/Questions/fewer-than-2-copies"}},l={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},u="wrapper";function f(e){let{components:t,...o}=e;return(0,i.yg)(u,(0,n.A)({},c,o,{components:t,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"client-server-number-of-copies"},"Client-Server Number of Copies"),(0,i.yg)("h2",{id:"question-text"},"Question Text"),(0,i.yg)("p",null,"The server in the image below uses regular TCP sockets to handle the connection and ",(0,i.yg)("inlineCode",{parentName:"p"},"send()")," to send the data.\nHow many times are the contents of the file copied by the server while being sent to the client?"),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Client-Server Steps",src:r(1975).A})),(0,i.yg)("h2",{id:"question-answers"},"Question Answers"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},"2")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},"1"))),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"4")),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"3")),(0,i.yg)("h2",{id:"feedback"},"Feedback"),(0,i.yg)("p",null,"Remember double buffering!\nWhen the app calls ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", the server's kernel will first save the file to an internal buffer destined for reading.\nThen the app will copy the file to its own buffer.\nFollowing this step, the app will call ",(0,i.yg)("inlineCode",{parentName:"p"},"send()"),", which will first copy the file to a buffer in the kernel.\nFrom this buffer, the kernel itself will copy the file to another buffer on the NIC (Network Interface Card).\nIn total, there the file is copied ",(0,i.yg)("strong",{parentName:"p"},"4 times"),", as outlined in the image below."),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Server Copies - Read-Send",src:r(7366).A})))}f.isMDXComponent=!0},1975:(e,t,r)=>{r.d(t,{A:()=>n});const n=r.p+"assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg"},7366:(e,t,r)=>{r.d(t,{A:()=>n});const n=r.p+"assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg"}}]); \ No newline at end of file +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8192],{5680:(e,t,r)=>{r.d(t,{xA:()=>c,yg:()=>d});var n=r(6540);function i(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function s(e){for(var t=1;t=0||(i[r]=e[r]);return i}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(i[r]=e[r])}return i}var l=n.createContext({}),p=function(e){var t=n.useContext(l),r=t;return e&&(r="function"==typeof e?e(t):s(s({},t),e)),r},c=function(e){var t=p(e.components);return n.createElement(l.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},y=n.forwardRef((function(e,t){var r=e.components,i=e.mdxType,o=e.originalType,l=e.parentName,c=a(e,["components","mdxType","originalType","parentName"]),u=p(r),y=i,d=u["".concat(l,".").concat(y)]||u[y]||f[y]||o;return r?n.createElement(d,s(s({ref:t},c),{},{components:r})):n.createElement(d,s({ref:t},c))}));function d(e,t){var r=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var o=r.length,s=new Array(o);s[0]=y;var a={};for(var l in t)hasOwnProperty.call(t,l)&&(a[l]=t[l]);a.originalType=e,a[u]="string"==typeof e?e:i,s[1]=a;for(var p=2;p{r.r(t),r.d(t,{assets:()=>l,contentTitle:()=>s,default:()=>f,frontMatter:()=>o,metadata:()=>a,toc:()=>p});var n=r(8168),i=(r(6540),r(5680));const o={},s="Client-Server Number of Copies",a={unversionedId:"IO/Questions/server-copies",id:"IO/Questions/server-copies",title:"Client-Server Number of Copies",description:"Question Text",source:"@site/docs/IO/Questions/server-copies.md",sourceDirName:"IO/Questions",slug:"/IO/Questions/server-copies",permalink:"/operating-systems/75/IO/Questions/server-copies",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Firefox: TCP or UDP?",permalink:"/operating-systems/75/IO/Questions/firefox-tcp-udp"},next:{title:"Fewer than Two Copies",permalink:"/operating-systems/75/IO/Questions/fewer-than-2-copies"}},l={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},u="wrapper";function f(e){let{components:t,...o}=e;return(0,i.yg)(u,(0,n.A)({},c,o,{components:t,mdxType:"MDXLayout"}),(0,i.yg)("h1",{id:"client-server-number-of-copies"},"Client-Server Number of Copies"),(0,i.yg)("h2",{id:"question-text"},"Question Text"),(0,i.yg)("p",null,"The server in the image below uses regular TCP sockets to handle the connection and ",(0,i.yg)("inlineCode",{parentName:"p"},"send()")," to send the data.\nHow many times are the contents of the file copied by the server while being sent to the client?"),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Client-Server Steps",src:r(2818).A})),(0,i.yg)("h2",{id:"question-answers"},"Question Answers"),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},"2")),(0,i.yg)("li",{parentName:"ul"},(0,i.yg)("p",{parentName:"li"},"1"))),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"4")),(0,i.yg)("ul",null,(0,i.yg)("li",{parentName:"ul"},"3")),(0,i.yg)("h2",{id:"feedback"},"Feedback"),(0,i.yg)("p",null,"Remember double buffering!\nWhen the app calls ",(0,i.yg)("inlineCode",{parentName:"p"},"read()"),", the server's kernel will first save the file to an internal buffer destined for reading.\nThen the app will copy the file to its own buffer.\nFollowing this step, the app will call ",(0,i.yg)("inlineCode",{parentName:"p"},"send()"),", which will first copy the file to a buffer in the kernel.\nFrom this buffer, the kernel itself will copy the file to another buffer on the NIC (Network Interface Card).\nIn total, there the file is copied ",(0,i.yg)("strong",{parentName:"p"},"4 times"),", as outlined in the image below."),(0,i.yg)("p",null,(0,i.yg)("img",{alt:"Server Copies - Read-Send",src:r(9475).A})))}f.isMDXComponent=!0},2818:(e,t,r)=>{r.d(t,{A:()=>n});const n=r.p+"assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg"},9475:(e,t,r)=>{r.d(t,{A:()=>n});const n=r.p+"assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg"}}]); \ No newline at end of file diff --git a/75/assets/js/b407248a.032927c4.js b/75/assets/js/b407248a.032927c4.js deleted file mode 100644 index 5f60fcd1a9..0000000000 --- a/75/assets/js/b407248a.032927c4.js +++ /dev/null @@ -1 +0,0 @@ -"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2429],{5680:(e,t,n)=>{n.d(t,{xA:()=>c,yg:()=>u});var a=n(6540);function l(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function i(e){for(var t=1;t=0||(l[n]=e[n]);return l}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(l[n]=e[n])}return l}var r=a.createContext({}),p=function(e){var t=a.useContext(r),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(r.Provider,{value:t},e.children)},m="mdxType",g={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},d=a.forwardRef((function(e,t){var n=e.components,l=e.mdxType,o=e.originalType,r=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),m=p(n),d=l,u=m["".concat(r,".").concat(d)]||m[d]||g[d]||o;return n?a.createElement(u,i(i({ref:t},c),{},{components:n})):a.createElement(u,i({ref:t},c))}));function u(e,t){var n=arguments,l=t&&t.mdxType;if("string"==typeof e||l){var o=n.length,i=new Array(o);i[0]=d;var s={};for(var r in t)hasOwnProperty.call(t,r)&&(s[r]=t[r]);s.originalType=e,s[m]="string"==typeof e?e:l,i[1]=s;for(var p=2;p{n.r(t),n.d(t,{assets:()=>r,contentTitle:()=>i,default:()=>g,frontMatter:()=>o,metadata:()=>s,toc:()=>p});var a=n(8168),l=(n(6540),n(5680));const o={},i="Memory Allocator",s={unversionedId:"Assignments/Memory Allocator/README",id:"Assignments/Memory Allocator/README",title:"Memory Allocator",description:"Objectives",source:"@site/docs/Assignments/Memory Allocator/README.md",sourceDirName:"Assignments/Memory Allocator",slug:"/Assignments/Memory Allocator/",permalink:"/operating-systems/75/Assignments/Memory Allocator/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Mini-libc",permalink:"/operating-systems/75/Assignments/Mini Libc/"},next:{title:"Parallel Graph",permalink:"/operating-systems/75/Assignments/Parallel Graph/"}},r={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Support Code",id:"support-code",level:2},{value:"API",id:"api",level:2},{value:"Implementation",id:"implementation",level:2},{value:"Memory Alignment",id:"memory-alignment",level:3},{value:"Block Reuse",id:"block-reuse",level:3},{value:"struct block_meta",id:"struct-block_meta",level:4},{value:"Split Block",id:"split-block",level:4},{value:"Coalesce Blocks",id:"coalesce-blocks",level:4},{value:"Find Best Block",id:"find-best-block",level:4},{value:"Heap Preallocation",id:"heap-preallocation",level:3},{value:"Building Memory Allocator",id:"building-memory-allocator",level:2},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Running the Checker",id:"running-the-checker",level:3},{value:"Running the Linters",id:"running-the-linters",level:3},{value:"Debugging",id:"debugging",level:3},{value:"Debugging in VSCode",id:"debugging-in-vscode",level:3},{value:"Resources",id:"resources",level:2}],c={toc:p},m="wrapper";function g(e){let{components:t,...o}=e;return(0,l.yg)(m,(0,a.A)({},c,o,{components:t,mdxType:"MDXLayout"}),(0,l.yg)("h1",{id:"memory-allocator"},"Memory Allocator"),(0,l.yg)("h2",{id:"objectives"},"Objectives"),(0,l.yg)("ul",null,(0,l.yg)("li",{parentName:"ul"},"Learn the basics of memory management by implementing minimal versions of ",(0,l.yg)("inlineCode",{parentName:"li"},"malloc()"),", ",(0,l.yg)("inlineCode",{parentName:"li"},"calloc()"),", ",(0,l.yg)("inlineCode",{parentName:"li"},"realloc()"),", and ",(0,l.yg)("inlineCode",{parentName:"li"},"free()"),"."),(0,l.yg)("li",{parentName:"ul"},"Accommodate with the memory management syscalls in Linux: ",(0,l.yg)("inlineCode",{parentName:"li"},"brk()"),", ",(0,l.yg)("inlineCode",{parentName:"li"},"mmap()"),", and ",(0,l.yg)("inlineCode",{parentName:"li"},"munmap()"),"."),(0,l.yg)("li",{parentName:"ul"},"Understand the bottlenecks of memory allocation and how to reduce them.")),(0,l.yg)("h2",{id:"statement"},"Statement"),(0,l.yg)("p",null,"Build a minimalistic memory allocator that can be used to manually manage virtual memory.\nThe goal is to have a reliable library that accounts for explicit allocation, reallocation, and initialization of memory."),(0,l.yg)("h2",{id:"support-code"},"Support Code"),(0,l.yg)("p",null,"The support code consists of three directories:"),(0,l.yg)("ul",null,(0,l.yg)("li",{parentName:"ul"},(0,l.yg)("inlineCode",{parentName:"li"},"src/")," will contain your solution"),(0,l.yg)("li",{parentName:"ul"},(0,l.yg)("inlineCode",{parentName:"li"},"tests/")," contains the test suite and a Python script to verify your work"),(0,l.yg)("li",{parentName:"ul"},(0,l.yg)("inlineCode",{parentName:"li"},"utils/")," contains ",(0,l.yg)("inlineCode",{parentName:"li"},"osmem.h")," that describes your library interface, ",(0,l.yg)("inlineCode",{parentName:"li"},"block_meta.h")," which contains details of ",(0,l.yg)("inlineCode",{parentName:"li"},"struct block_meta"),", and an implementation for ",(0,l.yg)("inlineCode",{parentName:"li"},"printf()")," function that does ",(0,l.yg)("strong",{parentName:"li"},"NOT")," use the heap")),(0,l.yg)("p",null,"The test suite consists of ",(0,l.yg)("inlineCode",{parentName:"p"},".c")," files that will be dynamically linked to your library, ",(0,l.yg)("inlineCode",{parentName:"p"},"libosmem.so"),".\nYou can find the sources in the ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/snippets/")," directory.\nThe results of the previous will also be stored in ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/snippets/")," and the reference files are in the ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/ref/")," directory."),(0,l.yg)("p",null,"The automated checking is performed using ",(0,l.yg)("inlineCode",{parentName:"p"},"run_tests.py"),".\nIt runs each test and compares the syscalls made by the ",(0,l.yg)("inlineCode",{parentName:"p"},"os_*")," functions with the reference file, providing a diff if the test failed."),(0,l.yg)("h2",{id:"api"},"API"),(0,l.yg)("ol",null,(0,l.yg)("li",{parentName:"ol"},(0,l.yg)("p",{parentName:"li"},(0,l.yg)("inlineCode",{parentName:"p"},"void *os_malloc(size_t size)")),(0,l.yg)("p",{parentName:"li"},"Allocates ",(0,l.yg)("inlineCode",{parentName:"p"},"size")," bytes and returns a pointer to the allocated memory."),(0,l.yg)("p",{parentName:"li"},"Chunks of memory smaller than ",(0,l.yg)("inlineCode",{parentName:"p"},"MMAP_THRESHOLD")," are allocated with ",(0,l.yg)("inlineCode",{parentName:"p"},"brk()"),".\nBigger chunks are allocated using ",(0,l.yg)("inlineCode",{parentName:"p"},"mmap()"),".\nThe memory is uninitialized."),(0,l.yg)("ul",{parentName:"li"},(0,l.yg)("li",{parentName:"ul"},"Passing ",(0,l.yg)("inlineCode",{parentName:"li"},"0")," as ",(0,l.yg)("inlineCode",{parentName:"li"},"size")," will return ",(0,l.yg)("inlineCode",{parentName:"li"},"NULL"),"."))),(0,l.yg)("li",{parentName:"ol"},(0,l.yg)("p",{parentName:"li"},(0,l.yg)("inlineCode",{parentName:"p"},"void *os_calloc(size_t nmemb, size_t size)")),(0,l.yg)("p",{parentName:"li"},"Allocates memory for an array of ",(0,l.yg)("inlineCode",{parentName:"p"},"nmemb")," elements of ",(0,l.yg)("inlineCode",{parentName:"p"},"size")," bytes each and returns a pointer to the allocated memory."),(0,l.yg)("p",{parentName:"li"},"Chunks of memory smaller than ",(0,l.yg)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/getpagesize.2.html"},(0,l.yg)("inlineCode",{parentName:"a"},"page_size"))," are allocated with ",(0,l.yg)("inlineCode",{parentName:"p"},"brk()"),".\nBigger chunks are allocated using ",(0,l.yg)("inlineCode",{parentName:"p"},"mmap()"),".\nThe memory is set to zero."),(0,l.yg)("ul",{parentName:"li"},(0,l.yg)("li",{parentName:"ul"},"Passing ",(0,l.yg)("inlineCode",{parentName:"li"},"0")," as ",(0,l.yg)("inlineCode",{parentName:"li"},"nmemb")," or ",(0,l.yg)("inlineCode",{parentName:"li"},"size")," will return ",(0,l.yg)("inlineCode",{parentName:"li"},"NULL"),"."))),(0,l.yg)("li",{parentName:"ol"},(0,l.yg)("p",{parentName:"li"},(0,l.yg)("inlineCode",{parentName:"p"},"void *os_realloc(void *ptr, size_t size)")),(0,l.yg)("p",{parentName:"li"},"Changes the size of the memory block pointed to by ",(0,l.yg)("inlineCode",{parentName:"p"},"ptr")," to ",(0,l.yg)("inlineCode",{parentName:"p"},"size")," bytes.\nIf the size is smaller than the previously allocated size, the memory block will be truncated."),(0,l.yg)("p",{parentName:"li"},"If ",(0,l.yg)("inlineCode",{parentName:"p"},"ptr")," points to a block on heap, ",(0,l.yg)("inlineCode",{parentName:"p"},"os_realloc()")," will first try to expand the block, rather than moving it.\nOtherwise, the block will be reallocated and its contents copied."),(0,l.yg)("p",{parentName:"li"},"When attempting to expand a block followed by multiple free blocks, ",(0,l.yg)("inlineCode",{parentName:"p"},"os_realloc()")," will coalesce them one at a time and verify the condition for each.\nBlocks will remain coalesced even if the resulting block will not be big enough for the new size."),(0,l.yg)("p",{parentName:"li"},"Calling ",(0,l.yg)("inlineCode",{parentName:"p"},"os_realloc()")," on a block that has ",(0,l.yg)("inlineCode",{parentName:"p"},"STATUS_FREE")," should return ",(0,l.yg)("inlineCode",{parentName:"p"},"NULL"),".\nThis is a measure to prevent undefined behavior and make the implementation robust, it should not be considered a valid use case of ",(0,l.yg)("inlineCode",{parentName:"p"},"os_realloc()"),"."),(0,l.yg)("ul",{parentName:"li"},(0,l.yg)("li",{parentName:"ul"},"Passing ",(0,l.yg)("inlineCode",{parentName:"li"},"NULL")," as ",(0,l.yg)("inlineCode",{parentName:"li"},"ptr")," will have the same effect as ",(0,l.yg)("inlineCode",{parentName:"li"},"os_malloc(size)"),"."),(0,l.yg)("li",{parentName:"ul"},"Passing ",(0,l.yg)("inlineCode",{parentName:"li"},"0")," as ",(0,l.yg)("inlineCode",{parentName:"li"},"size")," will have the same effect as ",(0,l.yg)("inlineCode",{parentName:"li"},"os_free(ptr)"),"."))),(0,l.yg)("li",{parentName:"ol"},(0,l.yg)("p",{parentName:"li"},(0,l.yg)("inlineCode",{parentName:"p"},"void os_free(void *ptr)")),(0,l.yg)("p",{parentName:"li"},"Frees memory previously allocated by ",(0,l.yg)("inlineCode",{parentName:"p"},"os_malloc()"),", ",(0,l.yg)("inlineCode",{parentName:"p"},"os_calloc()")," or ",(0,l.yg)("inlineCode",{parentName:"p"},"os_realloc()"),"."),(0,l.yg)("p",{parentName:"li"},(0,l.yg)("inlineCode",{parentName:"p"},"os_free()")," will not return memory from the heap to the OS by calling ",(0,l.yg)("inlineCode",{parentName:"p"},"brk()"),", but rather mark it as free and reuse it in future allocations.\nIn the case of mapped memory blocks, ",(0,l.yg)("inlineCode",{parentName:"p"},"os_free()")," will call ",(0,l.yg)("inlineCode",{parentName:"p"},"munmap()"),".")),(0,l.yg)("li",{parentName:"ol"},(0,l.yg)("p",{parentName:"li"},"General"),(0,l.yg)("ul",{parentName:"li"},(0,l.yg)("li",{parentName:"ul"},"Allocations that increase the heap size will only expand the last block if it is free."),(0,l.yg)("li",{parentName:"ul"},"You are allowed to use ",(0,l.yg)("inlineCode",{parentName:"li"},"sbrk()")," instead of ",(0,l.yg)("inlineCode",{parentName:"li"},"brk()"),", in view of the fact that ",(0,l.yg)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man2/brk.2.html#NOTES"},"on Linux")," ",(0,l.yg)("inlineCode",{parentName:"li"},"sbrk()")," is implemented using the ",(0,l.yg)("inlineCode",{parentName:"li"},"brk()"),"."),(0,l.yg)("li",{parentName:"ul"},"Do ",(0,l.yg)("strong",{parentName:"li"},"NOT")," use ",(0,l.yg)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man2/mremap.2.html"},(0,l.yg)("inlineCode",{parentName:"a"},"mremap()"))),(0,l.yg)("li",{parentName:"ul"},"You must check the error code returned by every syscall.\nYou can use the ",(0,l.yg)("inlineCode",{parentName:"li"},"DIE()")," macro for this.")))),(0,l.yg)("h2",{id:"implementation"},"Implementation"),(0,l.yg)("p",null,"An efficient implementation must keep data aligned, keep track of memory blocks and reuse freed blocks.\nThis can be further improved by reducing the number of syscalls and block operations."),(0,l.yg)("h3",{id:"memory-alignment"},(0,l.yg)("a",{parentName:"h3",href:"https://stackoverflow.com/a/381368"},"Memory Alignment")),(0,l.yg)("p",null,"Allocated memory should be aligned (i.e. all addresses are multiple of a given size).\nThis is a space-time trade-off because memory blocks are padded so each can be read in one transaction.\nIt also allows for atomicity when interacting with a block of memory."),(0,l.yg)("p",null,"All memory allocations should be aligned to ",(0,l.yg)("strong",{parentName:"p"},"8 bytes")," as required by 64 bit systems."),(0,l.yg)("h3",{id:"block-reuse"},"Block Reuse"),(0,l.yg)("h4",{id:"struct-block_meta"},(0,l.yg)("inlineCode",{parentName:"h4"},"struct block_meta")),(0,l.yg)("p",null,"We will consider a ",(0,l.yg)("strong",{parentName:"p"},"block")," to be a continuous zone of memory, allocated and managed by our implementation.\nThe structure ",(0,l.yg)("inlineCode",{parentName:"p"},"block_meta")," will be used to manage the metadata of a block.\nEach allocated zone will comprise of a ",(0,l.yg)("inlineCode",{parentName:"p"},"block_meta")," structure placed at the start, followed by data (",(0,l.yg)("strong",{parentName:"p"},"payload"),").\nFor all functions, the returned address will be that of the ",(0,l.yg)("strong",{parentName:"p"},"payload")," (not of the ",(0,l.yg)("inlineCode",{parentName:"p"},"block_meta")," structure)."),(0,l.yg)("pre",null,(0,l.yg)("code",{parentName:"pre",className:"language-c"},"struct block_meta {\n size_t size;\n int status;\n struct block_meta *prev;\n struct block_meta *next;\n};\n")),(0,l.yg)("p",null,(0,l.yg)("em",{parentName:"p"},"Note"),": Both the ",(0,l.yg)("inlineCode",{parentName:"p"},"struct block_meta")," and the ",(0,l.yg)("strong",{parentName:"p"},"payload")," of a block should be aligned to ",(0,l.yg)("strong",{parentName:"p"},"8 bytes"),"."),(0,l.yg)("p",null,(0,l.yg)("em",{parentName:"p"},"Note"),": Most compilers will automatically pad the structure, but you should still align it for portability."),(0,l.yg)("p",null,(0,l.yg)("img",{alt:"memory-block",src:n(7734).A})),(0,l.yg)("h4",{id:"split-block"},"Split Block"),(0,l.yg)("p",null,"Reusing memory blocks improves the allocator's performance, but might lead to ",(0,l.yg)("a",{parentName:"p",href:"https://www.tutorialspoint.com/difference-between-internal-fragmentation-and-external-fragmentation#:~:text=What%20is%20Internal%20Fragmentation%3F"},"Internal Memory Fragmentation"),".\nThis happens when we allocate a size smaller than all available free blocks.\nIf we use one larger block the remaining size of that block will be wasted since it cannot be used for another allocation."),(0,l.yg)("p",null,"To avoid this, a block should be truncated to the required size and the remaining bytes should be used to create a new free block."),(0,l.yg)("p",null,(0,l.yg)("img",{alt:"Split Block",src:n(3089).A})),(0,l.yg)("p",null,"The resulting free block should be reusable.\nThe split will not be performed if the remaining size (after reserving space for ",(0,l.yg)("inlineCode",{parentName:"p"},"block_meta")," structure and payload) is not big enough to fit another block (",(0,l.yg)("inlineCode",{parentName:"p"},"block_meta")," structure and at least ",(0,l.yg)("strong",{parentName:"p"},"1 byte")," of usable memory)."),(0,l.yg)("p",null,(0,l.yg)("em",{parentName:"p"},"Note"),": Do not forget the alignment!"),(0,l.yg)("h4",{id:"coalesce-blocks"},"Coalesce Blocks"),(0,l.yg)("p",null,"There are cases when there is enough free memory for an allocation, but it is spread across multiple blocks that cannot be used.\nThis is called ",(0,l.yg)("a",{parentName:"p",href:"https://www.tutorialspoint.com/difference-between-internal-fragmentation-and-external-fragmentation#:~:text=What%20is%20External%20Fragmentation%3F"},"External Memory Fragmentation"),"."),(0,l.yg)("p",null,"One technique to reduce external memory fragmentation is ",(0,l.yg)("strong",{parentName:"p"},"block coalescing")," which implies merging adjacent free blocks to form a contiguous chunk."),(0,l.yg)("p",null,(0,l.yg)("img",{alt:"Coalesce Block Image",src:n(8277).A})),(0,l.yg)("p",null,"Coalescing will be used before searching for a block and in ",(0,l.yg)("inlineCode",{parentName:"p"},"os_realloc()")," to expand the current block when possible."),(0,l.yg)("p",null,(0,l.yg)("em",{parentName:"p"},"Note"),": You might still need to split the block after coalesce."),(0,l.yg)("h4",{id:"find-best-block"},"Find Best Block"),(0,l.yg)("p",null,"Our aim is to reuse a free block with a size closer to what we need in order to reduce the number of future operations on it.\nThis strategy is called ",(0,l.yg)("strong",{parentName:"p"},"find best"),".\nOn every allocation we need to search the whole list of blocks and choose the best fitting free block."),(0,l.yg)("p",null,"In practice, it also uses a list of free blocks to avoid parsing all blocks, but this is out of the scope of the assignment."),(0,l.yg)("p",null,(0,l.yg)("em",{parentName:"p"},"Note"),": For consistent results, coalesce all adjacent free blocks before searching."),(0,l.yg)("h3",{id:"heap-preallocation"},"Heap Preallocation"),(0,l.yg)("p",null,"Heap is used in most modern programs.\nThis hints at the possibility of preallocating a relatively big chunk of memory (i.e. ",(0,l.yg)("strong",{parentName:"p"},"128 kilobytes"),") when the heap is used for the first time.\nThis reduces the number of future ",(0,l.yg)("inlineCode",{parentName:"p"},"brk()")," syscalls."),(0,l.yg)("p",null,"For example, if we try to allocate 1000 bytes we should first allocate a block of 128 kilobytes and then split it.\nOn future small allocations, we should proceed to split the preallocated chunk."),(0,l.yg)("p",null,(0,l.yg)("em",{parentName:"p"},"Note"),": Heap preallocation happens only once."),(0,l.yg)("h2",{id:"building-memory-allocator"},"Building Memory Allocator"),(0,l.yg)("p",null,"To build ",(0,l.yg)("inlineCode",{parentName:"p"},"libosmem.so"),", run ",(0,l.yg)("inlineCode",{parentName:"p"},"make")," in the ",(0,l.yg)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,l.yg)("pre",null,(0,l.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../mem-alloc$ cd src/\nstudent@os:~/.../mem-alloc/src$ make\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o osmem.o osmem.c\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o ../utils/printf.o ../utils/printf.c\ngcc -shared -o libosmem.so osmem.o helpers.o ../utils/printf.o\n")),(0,l.yg)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,l.yg)("p",null,"Testing is automated.\nTests are located in the ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,l.yg)("pre",null,(0,l.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../mem-alloc/tests$ ls -F\nMakefile grade.sh@ ref/ run_tests.py snippets/\n")),(0,l.yg)("p",null,"To test and grade your assignment solution, enter the ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/")," directory and run ",(0,l.yg)("inlineCode",{parentName:"p"},"grade.sh"),".\nNote that this requires linters being available.\nThe easiest is to use a Docker-based setup with everything installed, as shown in the section ",(0,l.yg)("a",{parentName:"p",href:"#running-the-linters"},'"Running the Linters"'),".\nWhen using ",(0,l.yg)("inlineCode",{parentName:"p"},"grade.sh")," you will get grades for correctness (maximum ",(0,l.yg)("inlineCode",{parentName:"p"},"90")," points) and for coding style (maximum ",(0,l.yg)("inlineCode",{parentName:"p"},"10")," points).\nA successful run will provide you an output ending with:"),(0,l.yg)("pre",null,(0,l.yg)("code",{parentName:"pre",className:"language-console"},"### GRADE\n\n\nChecker: 90/ 90\nStyle: 10/ 10\nTotal: 100/100\n\n\n### STYLE SUMMARY\n\n\n")),(0,l.yg)("h3",{id:"running-the-checker"},"Running the Checker"),(0,l.yg)("p",null,"To run only the checker, use the ",(0,l.yg)("inlineCode",{parentName:"p"},"run_tests.py")," script from the ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,l.yg)("p",null,"Before running ",(0,l.yg)("inlineCode",{parentName:"p"},"run_tests.py"),", you first have to build ",(0,l.yg)("inlineCode",{parentName:"p"},"libosmem.so")," in the ",(0,l.yg)("inlineCode",{parentName:"p"},"src/")," directory and generate the test binaries in ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/snippets"),".\nYou can do so using the all-in-one ",(0,l.yg)("inlineCode",{parentName:"p"},"Makefile")," rule from ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/"),": ",(0,l.yg)("inlineCode",{parentName:"p"},"make check"),"."),(0,l.yg)("pre",null,(0,l.yg)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../mem-alloc$ cd tests/\nstudent@os:~/.../mem-alloc/tests$ make check\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o osmem.o osmem.c\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o helpers.o helpers.c\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o ../utils/printf.o ../utils/printf.c\n[...]\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-all snippets/test-all.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-arrays snippets/test-calloc-arrays.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-block-reuse snippets/test-calloc-block-reuse.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce-big snippets/test-calloc-coalesce-big.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce snippets/test-calloc-coalesce.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-expand-block snippets/test-calloc-expand-block.c -L../src -losmem\n[...]\ntest-malloc-no-preallocate ........................ passed ... 2\ntest-malloc-preallocate ........................ passed ... 3\ntest-malloc-arrays ........................ passed ... 5\ntest-malloc-block-reuse ........................ passed ... 3\ntest-malloc-expand-block ........................ passed ... 2\ntest-malloc-no-split ........................ passed ... 2\ntest-malloc-split-one-block ........................ passed ... 3\ntest-malloc-split-first ........................ passed ... 2\ntest-malloc-split-last ........................ passed ... 2\ntest-malloc-split-middle ........................ passed ... 3\ntest-malloc-split-vector ........................ passed ... 2\ntest-malloc-coalesce ........................ passed ... 3\ntest-malloc-coalesce-big ........................ passed ... 3\ntest-calloc-no-preallocate ........................ passed ... 1\ntest-calloc-preallocate ........................ passed ... 1\ntest-calloc-arrays ........................ passed ... 5\ntest-calloc-block-reuse ........................ passed ... 1\ntest-calloc-expand-block ........................ passed ... 1\ntest-calloc-no-split ........................ passed ... 1\ntest-calloc-split-one-block ........................ passed ... 1\ntest-calloc-split-first ........................ passed ... 1\ntest-calloc-split-last ........................ passed ... 1\ntest-calloc-split-middle ........................ passed ... 1\ntest-calloc-split-vector ........................ passed ... 2\ntest-calloc-coalesce ........................ passed ... 2\ntest-calloc-coalesce-big ........................ passed ... 2\ntest-realloc-no-preallocate ........................ passed ... 1\ntest-realloc-preallocate ........................ passed ... 1\ntest-realloc-arrays ........................ passed ... 3\ntest-realloc-block-reuse ........................ passed ... 3\ntest-realloc-expand-block ........................ passed ... 2\ntest-realloc-no-split ........................ passed ... 3\ntest-realloc-split-one-block ........................ passed ... 3\ntest-realloc-split-first ........................ passed ... 3\ntest-realloc-split-last ........................ passed ... 3\ntest-realloc-split-middle ........................ passed ... 2\ntest-realloc-split-vector ........................ passed ... 2\ntest-realloc-coalesce ........................ passed ... 3\ntest-realloc-coalesce-big ........................ passed ... 1\ntest-all ........................ passed ... 5\n\nTotal: 90/100\n")),(0,l.yg)("p",null,(0,l.yg)("strong",{parentName:"p"},"NOTE:")," By default, ",(0,l.yg)("inlineCode",{parentName:"p"},"run_tests.py")," checks for memory leaks, which can be time-consuming.\nTo speed up testing, use the ",(0,l.yg)("inlineCode",{parentName:"p"},"-d")," flag or ",(0,l.yg)("inlineCode",{parentName:"p"},"make check-fast")," to skip memory leak checks."),(0,l.yg)("h3",{id:"running-the-linters"},"Running the Linters"),(0,l.yg)("p",null,"To run the linters, use the ",(0,l.yg)("inlineCode",{parentName:"p"},"make lint")," command in the ",(0,l.yg)("inlineCode",{parentName:"p"},"tests/")," directory.\nNote that the linters have to be installed on your system: ",(0,l.yg)("a",{parentName:"p",href:"https://github.com/torvalds/linux/blob/master/scripts/checkpatch.pl"},(0,l.yg)("inlineCode",{parentName:"a"},"checkpatch.pl")),", ",(0,l.yg)("a",{parentName:"p",href:"https://github.com/cpplint/cpplint"},(0,l.yg)("inlineCode",{parentName:"a"},"cpplint")),", ",(0,l.yg)("a",{parentName:"p",href:"https://www.shellcheck.net/"},(0,l.yg)("inlineCode",{parentName:"a"},"shellcheck"))," with certain configuration options.\nIt's easiest to run them in a Docker-based setup with everything configured:"),(0,l.yg)("pre",null,(0,l.yg)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../mem-alloc/tests$ make lint\n[...]\ncd .. && checkpatch.pl -f checker/*.sh tests/*.sh\n[...]\ncd .. && cpplint --recursive src/ tests/ checker/\n[...]\ncd .. && shellcheck checker/*.sh tests/*.sh\n")),(0,l.yg)("h3",{id:"debugging"},"Debugging"),(0,l.yg)("p",null,(0,l.yg)("inlineCode",{parentName:"p"},"run_tests.py")," uses ",(0,l.yg)("inlineCode",{parentName:"p"},"ltrace")," to capture all the libcalls and syscalls performed."),(0,l.yg)("p",null,"The output of ",(0,l.yg)("inlineCode",{parentName:"p"},"ltrace")," is formatted to show only top level library calls and nested system calls.\nFor consistency, the heap start and addresses returned by ",(0,l.yg)("inlineCode",{parentName:"p"},"mmap()")," are replaced with labels.\nEvery other address is displayed as ",(0,l.yg)("inlineCode",{parentName:"p"},"