diff --git a/17/.nojekyll b/17/.nojekyll new file mode 100644 index 0000000000..e69de29bb2 diff --git a/17/404.html b/17/404.html new file mode 100644 index 0000000000..15c00437d0 --- /dev/null +++ b/17/404.html @@ -0,0 +1,16 @@ + + + + + +Page Not Found | Operating Systems + + + + +
+
Skip to main content

Page Not Found

We could not find what you were looking for.

Please contact the owner of the site that linked you to the original URL and let them know their link is broken.

+ + + + \ No newline at end of file diff --git a/17/Assignments/Asynchronous Web Server/index.html b/17/Assignments/Asynchronous Web Server/index.html new file mode 100644 index 0000000000..c73cc6aee6 --- /dev/null +++ b/17/Assignments/Asynchronous Web Server/index.html @@ -0,0 +1,34 @@ + + + + + +Asynchronous Web Server | Operating Systems + + + + +
+
Skip to main content

Asynchronous Web Server

Objectives​

  • Deepening the concepts related to working with sockets.
  • Developing skills in implementing and designing applications that use asynchronous operations and other advanced I/O operations.
  • Deepening the use of the API for advanced I/O operations in the Linux operating system.

Statement​

Implement a web server that uses the following advanced I/O operations:

  • Asynchronous operations on files
  • Non-blocking operations on sockets
  • Zero-copying
  • Multiplexing I/O operations

The server implements a limited functionality of the HTTP protocol: passing files to clients.

The web server will use the multiplexing API to wait for connections from clients - epoll. +On the established connections, requests from clients will be received and then responses will be distributed to them.

The server will serve files from the AWS_DOCUMENT_ROOT directory, defined within the assignments' header. +Files are only found in subdirectories AWS_DOCUMENT_ROOT/static/ and AWS_DOCUMENT_ROOT/dynamic/. +The corresponding request paths will be, for example, AWS_DOCUMENT_ROOT/static/test.dat and AWS_DOCUMENT_ROOT/dynamic/test.dat. +The file processing will be:

  • The files in the AWS_DOCUMENT_ROOT/static/ directory are static files that will be transmitted to clients using the zero-copying API - sendfile]
  • Files in the AWS_DOCUMENT_ROOT/dynamic/ directory are files that are supposed to require a server-side post-processing phase. These files will be read from disk using the asynchronous API and then pushed to the clients. Streaming will use non-blocking sockets (Linux)
  • An HTTP 404 message will be sent for invalid request paths

After transmitting a file, according to the HTTP protocol, the connection is closed.

Details and recommendations for the implementation​

  • Implementing the assignment requires having a state machine for each connection, which you periodically query and update as the transfer proceeds. +Check the connection_state data structure defined in the assignment header.
  • Find the connection data structure defined in the assignment header. +This can be used to keep track of an open connection.
  • Definitions of other useful macros and data structures can be found in the assignment header.
  • HTTP responses will have the code 200 for existing files and 404 for not existing files.
    • A valid response consists of the HTTP header, containing the related directives, two newlines (\r\n\r\n), followed by the actual content (the file).
    • Sample answers can be found in the parser test file or in the provided sample.
    • You can use predefined request directives such as Date, Last-Modified, etc.
      • The Content-Length directive must specify the size of the HTTP content (actual data) in bytes.
      • The Connection directive must be initialized to close.
  • The port on which the web server listens for connections is defined within the assignment header: the AWS_LISTEN_PORT macro.
  • The root directory relative to which the resources/files are searched is defined within the assignment header as the AWS_DOCUMENT_ROOT macro.

Support Code​

HTTP Parser​

The clients and server will communicate using the HTTP protocol. +For parsing HTTP requests from clients we recommend using this HTTP parser, also available in the assignments' http-parser. +You will need to use a callback to get the path to the local resource requested by the client. +Find a simplified example of using the parser in the samples directory.

API and Implementation Tasks​

The skel/aws.c file contains the code skelethon with several functions that have to be implemented. +Follow the TODO areas in the file to start your implementation.

It can be reorganized as desired, as long as all the requirements of the assignment are implemented.

Testing and Grading​

The testing is automated. +Tests are located in the tests/ directory.

To test your implementation, do the following steps:

  • Run the make command inside the skel/ directory and make sure it compiles with no errors and that the aws executable is generated.
  • Run the make check command in the tests/ directory.

There are 35 tests for this assignment, of which 13 are doubled by a memory leak check test. +A successful run looks as the following:

student@so:~/operating-systems/content/assignments/async-web-server/tests$ make check
make -C _test
make[1]: Entering directory '/home/student/operating-systems/content/assignments/async-web-server/tests/_test'
make[1]: Nothing to be done for 'all'.
make[1]: Leaving directory '/home/student/operating-systems/content/assignments/async-web-server/tests/_test'

                        = Testing - Asynchronous Web Server =

01) Test executable exists.............................................passed  [01/90]
02) Test executable runs...............................................passed  [01/90]
03) Test listening.....................................................passed  [01/90]
04) Test listening on port.............................................passed  [01/90]
05) Test accepts connections...........................................passed  [01/90]
06) Test accepts multiple connections..................................passed  [01/90]
07) Test epoll usage...................................................passed  [01/90]
08) Test disconnect....................................................passed  [01/90]
09) Test multiple disconnect...........................................passed  [01/90]
10) Test connect disconnect connect....................................passed  [01/90]
11) Test multiple connect disconnect connect...........................passed  [01/90]
12) Test unordered connect disconnect connect..........................passed  [01/90]
13) Test replies http request..........................................passed  [02/90]
13) Test replies http request - memcheck...............................passed  [01/90]
14) Test second replies http request...................................passed  [01/90]
15) Test sendfile usage................................................passed  [02/90]
16) Test small static file wget........................................passed  [02/90]
17) Test small static file wget cmp....................................passed  [04/90]
17) Test small static file wget cmp - memcheck.........................passed  [01/90]
18) Test large static file wget........................................passed  [02/90]
19) Test large static file wget cmp....................................passed  [04/90]
19) Test large static file wget cmp - memcheck.........................passed  [01/90]
20) Test bad static file 404...........................................passed  [02/90]
21) Test bad path 404..................................................passed  [02/90]
22) Test get one static file then another..............................passed  [02/90]
22) Test get one static file then another - memcheck...................passed  [01/90]
23) Test get two simultaneous static files.............................passed  [03/90]
23) Test get two simultaneous static files - memcheck..................passed  [01/90]
24) Test get multiple simultaneous static files........................passed  [04/90]
24) Test get multiple simultaneous static files - memcheck.............passed  [01/90]
25) Test io submit uses................................................passed  [02/90]
26) Test small dynamic file wget.......................................passed  [02/90]
27) Test small dynamic file wget cmp...................................passed  [04/90]
27) Test small dynamic file wget cmp - memcheck........................passed  [01/90]
28) Test large dynamic file wget.......................................passed  [02/90]
29) Test large dynamic file wget cmp...................................passed  [04/90]
29) Test large dynamic file wget cmp - memcheck........................passed  [01/90]
30) Test bad dynamic file 404..........................................passed  [02/90]
31) Test get one dynamic file then another.............................passed  [03/90]
31) Test get one dynamic file then another - memcheck..................passed  [01/90]
32) Test get two simultaneous dynamic files............................passed  [04/90]
32) Test get two simultaneous dynamic files - memcheck.................passed  [01/90]
33) Test get multiple simultaneous dynamic files.......................passed  [05/90]
33) Test get multiple simultaneous dynamic files - memcheck............passed  [01/90]
34) Test get two simultaneous static and dynamic files.................passed  [03/90]
34) Test get two simultaneous static and dynamic files - memcheck......passed  [01/90]
35) Test get multiple simultaneous static and dynamic files............passed  [04/90]
35) Test get multiple simultaneous static and dynamic files - memcheck.passed  [01/90]

                                                                       Total:  [90/100]

Individual tests can be run using the ./run_test.sh bash script as the following:

student@so:~/operating-systems/content/assignments/async-web-server/tests$ ./_test/run_test.sh 3
03) Test listening.....................................................passed  [01/90]

Where 3 is the test you want to run.

Some tests are doubled by a memory check test. +This will only run if the regular test passed. +For example, test 31 will output the following in case of success:

student@so:~/operating-systems/content/assignments/async-web-server/tests$ ./_test/run_test.sh 31
31) Test get one dynamic file then another.............................passed  [03/90]
31) Test get one dynamic file then another - memcheck..................passed  [01/90]

and one of the following in case of error:

# if the regular tests failed, the memory check tests is not performed
student@so:~/operating-systems/content/assignments/async-web-server/tests$ ./_test/run_test.sh 31
31) Test get one dynamic file then another.............................failed  [ 0/90]
31) Test get one dynamic file then another - memcheck..................passed  [01/90]

Note: The memcheck test for failed regular tests will not be taken into consideration for the final score. +This output will be fixed in the next commit.

Tests use the static/ and dynamic/ folders. +These folders are created and removed using the init and cleanup arguments to _test/run_test.sh.

Behind the Scenes​

Tests are basically unit tests.

Each test function follows the unit test patter: initialization, action, +evaluation.

Each test starts the server, creates a given context, checks for validity and +then terminates the server process.

Debugging​

Logs are collected in test.log and wget.log files.

Resources​

+ + + + \ No newline at end of file diff --git a/17/Assignments/Asynchronous Web Server/src/http-parser/index.html b/17/Assignments/Asynchronous Web Server/src/http-parser/index.html new file mode 100644 index 0000000000..3dc143d90a --- /dev/null +++ b/17/Assignments/Asynchronous Web Server/src/http-parser/index.html @@ -0,0 +1,54 @@ + + + + + +HTTP Parser | Operating Systems + + + + +
+
Skip to main content

HTTP Parser

This is a parser for HTTP messages written in C. It parses both requests and +responses. The parser is designed to be used in performance HTTP +applications. It does not make any syscalls nor allocations, it does not +buffer data, it can be interrupted at anytime. Depending on your +architecture, it only requires about 40 bytes of data per message +stream (in a web server that is per connection).

Features:

  • No dependencies
  • Handles persistent streams (keep-alive).
  • Decodes chunked encoding.
  • Upgrade support
  • Defends against buffer overflow attacks.

The parser extracts the following information from HTTP messages:

  • Header fields and values
  • Content-Length
  • Request method
  • Response status code
  • Transfer-Encoding
  • HTTP version
  • Request path, query string, fragment
  • Message body

Usage​

One http_parser object is used per TCP connection. Initialize the struct +using http_parser_init() and set the callbacks. That might look something +like this for a request parser:

http_parser_settings settings;
settings.on_path = my_path_callback;
settings.on_header_field = my_header_field_callback;
/* ... */

http_parser *parser = malloc(sizeof(http_parser));
http_parser_init(parser, HTTP_REQUEST);
parser->data = my_socket;

When data is received on the socket execute the parser and check for errors.

size_t len = 80*1024, nparsed;
char buf[len];
ssize_t recved;

recved = recv(fd, buf, len, 0);

if (recved < 0) {
  /* Handle error. */
}

/* Start up / continue the parser.
 * Note we pass recved==0 to signal that EOF has been recieved.
 */
nparsed = http_parser_execute(parser, &settings, buf, recved);

if (parser->upgrade) {
  /* handle new protocol */
} else if (nparsed != recved) {
  /* Handle error. Usually just close the connection. */
}

HTTP needs to know where the end of the stream is. For example, sometimes +servers send responses without Content-Length and expect the client to +consume input (for the body) until EOF. To tell http_parser about EOF, give +0 as the forth parameter to http_parser_execute(). Callbacks and errors +can still be encountered during an EOF, so one must still be prepared +to receive them.

Scalar valued message information such as status_code, method, and the +HTTP version are stored in the parser structure. This data is only +temporally stored in http_parser and gets reset on each new message. If +this information is needed later, copy it out of the structure during the +headers_complete callback.

The parser decodes the transfer-encoding for both requests and responses +transparently. That is, a chunked encoding is decoded before being sent to +the on_body callback.

The Special Problem of Upgrade​

HTTP supports upgrading the connection to a different protocol. An +increasingly common example of this is the Web Socket protocol which sends +a request like

    GET /demo HTTP/1.1
    Upgrade: WebSocket
    Connection: Upgrade
    Host: example.com
    Origin: http://example.com
    WebSocket-Protocol: sample

followed by non-HTTP data.

(See http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75 for more +information the Web Socket protocol.)

To support this, the parser will treat this as a normal HTTP message without a +body. Issuing both on_headers_complete and on_message_complete callbacks. However +http_parser_execute() will stop parsing at the end of the headers and return.

The user is expected to check if parser->upgrade has been set to 1 after +http_parser_execute() returns. Non-HTTP data begins at the buffer supplied +offset by the return value of http_parser_execute().

Callbacks​

During the http_parser_execute() call, the callbacks set in +http_parser_settings will be executed. The parser maintains state and +never looks behind, so buffering the data is not necessary. If you need to +save certain data for later usage, you can do that from the callbacks.

There are two types of callbacks:

  • notification typedef int (*http_cb) (http_parser*); +Callbacks: on_message_begin, on_headers_complete, on_message_complete.
  • data typedef int (*http_data_cb) (http_parser*, const char *at, size_t length); +Callbacks: (requests only) on_path, on_query_string, on_uri, on_fragment, +(common) on_header_field, on_header_value, on_body;

Callbacks must return 0 on success. Returning a non-zero value indicates +error to the parser, making it exit immediately.

In case you parse HTTP message in chunks (i.e. read() request line +from socket, parse, read half headers, parse, etc) your data callbacks +may be called more than once. Http-parser guarantees that data pointer is only +valid for the lifetime of callback. You can also read() into a heap allocated +buffer to avoid copying memory around if this fits your application.

Reading headers may be a tricky task if you read/parse headers partially. +Basically, you need to remember whether last header callback was field or value +and apply following logic:

(on_header_field and on_header_value shortened to on_h_*)
 ------------------------ ------------ --------------------------------------------
| State (prev. callback) | Callback   | Description/action                         |
 ------------------------ ------------ --------------------------------------------
| nothing (first call)   | on_h_field | Allocate new buffer and copy callback data |
|                        |            | into it                                    |
 ------------------------ ------------ --------------------------------------------
| value                  | on_h_field | New header started.                        |
|                        |            | Copy current name,value buffers to headers |
|                        |            | list and allocate new buffer for new name  |
 ------------------------ ------------ --------------------------------------------
| field                  | on_h_field | Previous name continues. Reallocate name   |
|                        |            | buffer and append callback data to it      |
 ------------------------ ------------ --------------------------------------------
| field                  | on_h_value | Value for current header started. Allocate |
|                        |            | new buffer and copy callback data to it    |
 ------------------------ ------------ --------------------------------------------
| value                  | on_h_value | Value continues. Reallocate value buffer   |
|                        |            | and append callback data to it             |
 ------------------------ ------------ --------------------------------------------

See examples of reading in headers:

+ + + + \ No newline at end of file diff --git a/17/Assignments/Memory Allocator/index.html b/17/Assignments/Memory Allocator/index.html new file mode 100644 index 0000000000..4580c158db --- /dev/null +++ b/17/Assignments/Memory Allocator/index.html @@ -0,0 +1,51 @@ + + + + + +Memory Allocator | Operating Systems + + + + +
+
Skip to main content

Memory Allocator

Objectives​

  • Learn the basics of memory management by implementing minimal versions of malloc(), calloc(), realloc(), and free().
  • Accommodate with the memory management syscalls in Linux: brk(), mmap(), and munmap().
  • Understand the bottlenecks of memory allocation and how to reduce them.

Statement​

Build a minimalistic memory allocator that can be used to manually manage virtual memory. +The goal is to have a reliable library that accounts for explicit allocation, reallocation, and initialization of memory.

Support Code​

The support code consists of three directories:

  • src/ will contain your solution
  • tests/ contains the test suite and a Python script to verify your work
  • utils/ contains osmem.h that describes your library interface, block_meta.h which contains details of struct block_meta, and an implementation for printf() function that does NOT use the heap

The test suite consists of .c files that will be dynamically linked to your library, libosmem.so. +You can find the sources in the tests/snippets/ directory. +The results of the previous will also be stored in tests/snippets/ and the reference files are in the tests/ref/ directory.

The automated checking is performed using run-tests.py. +It runs each test and compares the syscalls made by the os_* functions with the reference file, providing a diff if the test failed.

API​

  1. void *os_malloc(size_t size)

    Allocates size bytes and returns a pointer to the allocated memory.

    Chunks of memory smaller than MMAP_THRESHOLD are allocated with brk(). +Bigger chunks are allocated using mmap(). +The memory is uninitialized.

    • Passing 0 as size will return NULL.

  2. void *os_calloc(size_t nmemb, size_t size)

    Allocates memory for an array of nmemb elements of size bytes each and returns a pointer to the allocated memory.

    Chunks of memory smaller than page_size are allocated with brk(). +Bigger chunks are allocated using mmap(). +The memory is set to zero.

    • Passing 0 as nmemb or size will return NULL.

  3. void *os_realloc(void *ptr, size_t size)

    Changes the size of the memory block pointed to by ptr to size bytes. +If the size is smaller than the previously allocated size, the memory block will be truncated.

    If ptr points to a block on heap, os_realloc() will first try to expand the block, rather than moving it. +Otherwise, the block will be reallocated and its contents copied.

    When attempting to expand a block followed by multiple free blocks, os_realloc() will coalesce them one at a time and verify the condition for each. +Blocks will remain coalesced even if the resulting block will not be big enough for the new size.

    Calling os_realloc() on a block that has STATUS_FREE should return NULL. +This is a measure to prevent undefined behavior and make the implementation robust, it should not be considered a valid use case of os_realloc().

    • Passing NULL as ptr will have the same effect as os_malloc(size).
    • Passing 0 as size will have the same effect as os_free(ptr).

  4. void os_free(void *ptr)

    Frees memory previously allocated by os_malloc(), os_calloc() or os_realloc().

    os_free() will not return memory from the heap to the OS by calling brk(), but rather mark it as free and reuse it in future allocations. +In the case of mapped memory blocks, os_free() will call munmap().

  5. General

    • Allocations that increase the heap size will only expand the last block if it is free.
    • You are allowed to use sbrk() instead of brk(), in view of the fact that on Linux sbrk() is implemented using the brk().
    • Do NOT use mremap()
    • You must check the error code returned by every syscall. +You can use the DIE() macro for this.

Implementation​

An efficient implementation must keep data aligned, keep track of memory blocks and reuse freed blocks. +This can be further improved by reducing the number of syscalls and block operations.

Memory Alignment​

Allocated memory should be aligned (i.e. all addresses are multiple of a given size). +This is a space-time trade-off because memory blocks are padded so each can be read in one transaction. +It also allows for atomicity when interacting with a block of memory.

All memory allocations should be aligned to 8 bytes as required by 64 bit systems.

Block Reuse​

struct block_meta​

We will consider a block to be a continuous zone of memory, allocated and managed by our implementation. +The structure block_meta will be used to manage the metadata of a block. +Each allocated zone will comprise of a block_meta structure placed at the start, followed by data (payload). +For all functions, the returned address will be that of the payload (not of the block_meta structure).

struct block_meta {
    size_t size;
    int status;
    struct block_meta *prev;
    struct block_meta *next;
};

Note: Both the struct block_meta and the payload of a block should be aligned to 8 bytes.

Note: Most compilers will automatically pad the structure, but you should still align it for portability.

memory-block

Split Block​

Reusing memory blocks improves the allocator's performance, but might lead to Internal Memory Fragmentation. +This happens when we allocate a size smaller than all available free blocks. +If we use one larger block the remaining size of that block will be wasted since it cannot be used for another allocation.

To avoid this, a block should be truncated to the required size and the remaining bytes should be used to create a new free block.

Split Block

The resulting free block should be reusable. +The split will not be performed if the remaining size (after reserving space for block_meta structure and payload) is not big enough to fit another block (block_meta structure and at least 1 byte of usable memory).

Note: Do not forget the alignment!

Coalesce Blocks​

There are cases when there is enough free memory for an allocation, but it is spread across multiple blocks that cannot be used. +This is called External Memory Fragmentation.

One technique to reduce external memory fragmentation is block coalescing which implies merging adjacent free blocks to form a contiguous chunk.

Coalesce Block Image

Coalescing will be used before searching for a block and in os_realloc() to expand the current block when possible.

Note: You might still need to split the block after coalesce.

Find Best Block​

Our aim is to reuse a free block with a size closer to what we need in order to reduce the number of future operations on it. +This strategy is called find best. +On every allocation we need to search the whole list of blocks and choose the best fitting free block.

In practice, it also uses a list of free blocks to avoid parsing all blocks, but this is out of the scope of the assignment.

Note: For consistent results, coalesce all adjacent free blocks before searching.

Heap Preallocation​

Heap is used in most modern programs. +This hints at the possibility of preallocating a relatively big chunk of memory (i.e. 128 kilobytes) when the heap is used for the first time. +This reduces the number of future brk() syscalls.

For example, if we try to allocate 1000 bytes we should first allocate a block of 128 kilobytes and then split it. +On future small allocations, we should proceed to split the preallocated chunk.

Note: Heap preallocation happens only once.

Building Memory Allocator​

To build libosmem.so, run make in the src/ directory:

student@os:~/.../mem-alloc$ cd src/
student@os:~/.../mem-alloc/src$ make
gcc -fPIC -Wall -Wextra -g -I../utils  -c -o osmem.o osmem.c
gcc -fPIC -Wall -Wextra -g -I../utils  -c -o ../utils/printf.o ../utils/printf.c
gcc -shared -o libosmem.so osmem.o helpers.o ../utils/printf.o

Testing and Grading​

The testing is automated and performed with the run-tests.py script from the tests/ directory.

Before running run-tests.py, you first have to build libosmem.so in the src/ directory and generate the test binaries in tests/snippets. +You can do so using the all-in-one Makefile rule from tests/: make check.

student@os:~/.../mem-alloc$ cd tests/
student@os:~/.../mem-alloc/tests$ make check
gcc -fPIC -Wall -Wextra -g -I../utils  -c -o osmem.o osmem.c
gcc -fPIC -Wall -Wextra -g -I../utils  -c -o helpers.o helpers.c
gcc -fPIC -Wall -Wextra -g -I../utils  -c -o ../utils/printf.o ../utils/printf.c
[...]
gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-all snippets/test-all.c -L../src -losmem
gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-arrays snippets/test-calloc-arrays.c -L../src -losmem
gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-block-reuse snippets/test-calloc-block-reuse.c -L../src -losmem
gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce-big snippets/test-calloc-coalesce-big.c -L../src -losmem
gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce snippets/test-calloc-coalesce.c -L../src -losmem
gcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-expand-block snippets/test-calloc-expand-block.c -L../src -losmem
[...]
test-malloc-no-preallocate       ........................ passed ...   2
test-malloc-preallocate          ........................ passed ...   3
test-malloc-arrays               ........................ passed ...   5
test-malloc-block-reuse          ........................ passed ...   3
test-malloc-expand-block         ........................ passed ...   2
test-malloc-no-split             ........................ passed ...   2
test-malloc-split-one-block      ........................ passed ...   3
test-malloc-split-first          ........................ passed ...   2
test-malloc-split-last           ........................ passed ...   2
test-malloc-split-middle         ........................ passed ...   3
test-malloc-split-vector         ........................ passed ...   2
test-malloc-coalesce             ........................ passed ...   3
test-malloc-coalesce-big         ........................ passed ...   3
test-calloc-no-preallocate       ........................ passed ...   1
test-calloc-preallocate          ........................ passed ...   1
test-calloc-arrays               ........................ passed ...   5
test-calloc-block-reuse          ........................ passed ...   1
test-calloc-expand-block         ........................ passed ...   1
test-calloc-no-split             ........................ passed ...   1
test-calloc-split-one-block      ........................ passed ...   1
test-calloc-split-first          ........................ passed ...   1
test-calloc-split-last           ........................ passed ...   1
test-calloc-split-middle         ........................ passed ...   1
test-calloc-split-vector         ........................ passed ...   2
test-calloc-coalesce             ........................ passed ...   2
test-calloc-coalesce-big         ........................ passed ...   2
test-realloc-no-preallocate      ........................ passed ...   1
test-realloc-preallocate         ........................ passed ...   1
test-realloc-arrays              ........................ passed ...   3
test-realloc-block-reuse         ........................ passed ...   3
test-realloc-expand-block        ........................ passed ...   2
test-realloc-no-split            ........................ passed ...   3
test-realloc-split-one-block     ........................ passed ...   3
test-realloc-split-first         ........................ passed ...   3
test-realloc-split-last          ........................ passed ...   3
test-realloc-split-middle        ........................ passed ...   2
test-realloc-split-vector        ........................ passed ...   2
test-realloc-coalesce            ........................ passed ...   3
test-realloc-coalesce-big        ........................ passed ...   1
test-all                         ........................ passed ...   5

Total:                                                            90/100

NOTE: By default, run-test.py checks for memory leaks, which can be time-consuming. +To speed up testing, use the -d flag or make check-fast to skip memory leak checks, but remember to run make check before submitting your assignment to ensure it meets all criteria.

Debugging​

run-tests.py uses ltrace to capture all the libcalls and syscalls performed.

The output of ltrace is formatted to show only top level library calls and nested system calls. +For consistency, the heap start and addresses returned by mmap() are replaced with labels. +Every other address is displayed as <label> + offset, where the label is the closest mapped address.

run-tests.py supports three modes:

  • verbose (-v), prints the output of the test
  • diff (-d), prints the diff between the output and the ref
  • memcheck (-m), prints the diff between the output and the ref and announces memory leaks

If you want to run a single test, you give its name or its path as arguments to run-tests.py:

student@os:~/.../mem-alloc/tests$ python run-tests.py test-all
OR
student@os:~/.../mem-alloc/tests$ python run-tests.py snippets/test-all

Debugging in VSCode​

If you are using Visual Studio Code, you can use the launch.json configurations to run tests.

Setup the breakpoints in the source files or the tests and go to Run and Debug (F5). +Select Run test script and press F5. +This will enter a dialogue where you can choose which test to run.

You can find more on this in the official documentation: Debugging with VSCode.

If VSCode complains about MAP_ANON argument for mmap() change C_Cpp.default.cStandard option to gnu11.

Resources​

+ + + + \ No newline at end of file diff --git a/17/Assignments/Mini Libc/index.html b/17/Assignments/Mini Libc/index.html new file mode 100644 index 0000000000..e74cb984aa --- /dev/null +++ b/17/Assignments/Mini Libc/index.html @@ -0,0 +1,39 @@ + + + + + +Mini-libc | Operating Systems + + + + +
+
Skip to main content

Mini-libc

Objectives​

  • Learn about the structure and functionalities provided by the standard C library
  • Accommodate with the syscall interface in Linux
  • Gain a better understanding of strings and memory management functions
  • Learn how the standard C library provides support for low-level input/output operations

Statement​

Build a minimalistic standard C library implementation for Linux systems (named mini-libc), that can be used as a replacement for the system libc (glibc in Linux). +The goal is to have a minimally functional libc with features such as string management, basic memory support and POSIX file I/O.

The implementation of mini-libc will be freestanding, i.e. it will not use any outside library calls. +It will be implemented on top of the system call interface provided by Linux on an x86_64 architecture. +Any function you require, that is typically part of libc, you will have to implement. +You can reuse functions that you implement in other parts of the mini-libc.

In case you are using a macOS device with ARM64 / Aarch64, you will have to install an x86_64 virtual machine.

Support Code​

The support code consists of three directories:

  • src/ is the skeleton mini-libc implementation. +You will have to implement missing parts marked as TODO items.

  • samples/ stores use cases and tests of mini-libc.

  • tests/ are tests used to validate (and grade) the assignment.

System call invocation is done via the syscall() function defined in src/syscall.c. +That itself makes a call to the architecture-specific call in src/internal/arch/x86_64/syscall_arch.h; +hence the dependency on the x86_64 architecture.

API and Implementation Tasks​

The application programming interface (API) of the C standard library is declared in a number of header files. +Each header file contains one or more function declarations, data type definitions and macros. +For your minimal implementation, the following header files are of interest:

  • <string.h>: defines string-handling functions

    For this assignment, you will have to implement the following functions: strcpy(), strcat(), strlen(), strncpy(), strncat(), strcmp(), strncmp(), strstr(), strrstr(), memcpy(), memset(), memmove(), memcmp().

  • <stdio.h>: defines printing and I/O functions

    For this assignment, you will have to implement puts().

  • <unistd.h>, <sys/fcntl.h> and <sys/stat.h>: define I/O primitives

    For this assignment, you will have to implement the following functions: open(), close(), lseek(), stat(), fstat(), truncate(), ftruncate().

    You will also have to implement the nanosleep() and sleep() functions.

  • <stdlib.h> and <sys/mman.h> define memory allocation functions

    For this assignment, you will have to implement the following functions: malloc(), free(), calloc(), realloc(), realloc_array(), mmap(), mremap(), munmap().

    For managing memory areas, a basic list structure is provided in include/internal/mm/mem_list.h and mm/mem_list.c.

  • <errno.h> and errno.c: declare and define the integer variable errno, which is set by system calls and some library functions in the event of an error to indicate what went wrong.

Some tests do not build. +This is intentional. +You will have to add the missing features to make those tests compile, that is

  • the time.h header

  • the declaration and the implementation of puts()

  • the declaration and the implementation of nanosleep() and sleep()

  • the update of the libc Makefile to build the source code files implementing puts(), nanosleep() and sleep()

    ❗❗ Pay attention to which functions have to modify the errno variable.

Building mini-libc​

To build mini-libc, run make in the src/ directory:

student@so:~/.../content/assignments/mini-libc$ cd src/

student@so:~/.../assignments/mini-libc/src$ make

To build samples, enter the samples directory and run make:

student@so:~/.../content/assignments/mini-libc$ cd samples/

student@so:~/.../assignments/mini-libc/samples$ make

Testing and Grading​

The testing is automated. +Tests are located in the tests/ directory.

student@so:~/.../assignments/mini-libc/tests$ ls -F
Makefile       graded_test.inc.sh  run_all_tests.sh*   test_io_file_create.sh*  test_malloc_free.sh*           test_memory.c            test_mmap_perm_notok.sh*       test_nanosleep.sh*   test_stat.sh*
grade.sh*      io/                 test_fstat.sh*      test_io_file_delete.sh*  test_malloc_free_sequence.sh*  test_mmap.sh*            test_mmap_perm_ok.sh*          test_open_close.sh*  test_string.c
graded_test.c  memory/             test_ftruncate.sh*  test_lseek.sh*           test_malloc_perm_notok.sh*     test_mmap_munmap.sh*     test_multiple_malloc.sh*       test_puts.sh*        test_truncate.sh*
graded_test.h  process/            test_io.c           test_malloc.sh*          test_malloc_perm_ok.sh*        test_mmap_perm_none.sh*  test_multiple_malloc_free.sh*  test_sleep.sh*

To test and grade your assignment solution, enter the tests/ directory and run grade.sh. +Note that this requires linters being available. +The easiest is to use a Docker-based setup with everything installed, as shown in the section "Running the Linters". +When using grade.sh you will get grades for checking correctness (maximum 90 points) and for coding style (maxim 10 points). +A successful run will provide you an output ending with:

### GRADE


Checker:                                                         90/ 90
Style:                                                           10/ 10
Total:                                                          100/100


### STYLE SUMMARY


Running the Checker​

To run only the checker, use the make check command in the tests/ directory:

student@so:~/.../assignments/mini-libc/tests$ make check
make[1]: Entering directory '...'
rm -f *~
[...]
test_mmap_perm_ok                ........................ failed ...  0
test_mmap_perm_notok             ........................ failed ...  0
test_mmap_perm_none              ........................ failed ...  0

Total:                                                             0/100

Some files will fail to build, it's expected. +This is because there are missing files or missing functions that cause build errors. +You'll need to add those files and implement those functions for the build error to disappear.

Obviously, most tests will fail, as there is no implementation. +Some tests don't fail because the missing implementation equates to the bad behavior being tested not happening.

Each test is worth a number of points. +The total number of points is 900. +The maximum grade is obtained by dividing the number of points to 10, for a maximum grade of 90.

A successful run will show the output:

student@so:~/.../assignments/mini-libc/tests$ make check
[...]
test_strcpy                      ........................ passed ...   9
test_strcpy_append               ........................ passed ...   9
test_strncpy                     ........................ passed ...   9
test_strncpy_cut                 ........................ passed ...   9
test_strcat                      ........................ passed ...   9
test_strcat_from_zero            ........................ passed ...   9
test_strcat_multiple             ........................ passed ...   9
test_strncat                     ........................ passed ...   9
test_strncat_cut                 ........................ passed ...   9
test_strcmp_equal                ........................ passed ...   9
test_strcmp_same_size_less       ........................ passed ...   1
test_strcmp_same_size_greater    ........................ passed ...   9
test_strcmp_diff_size_less       ........................ passed ...   1
test_strcmp_diff_size_greater    ........................ passed ...   9
test_strncmp_equal_size_equal    ........................ passed ...   9
test_strncmp_diff_contents_equal ........................ passed ...   9
test_strncmp_diff_size_equal     ........................ passed ...   9
test_strchr_exists               ........................ passed ...  11
test_strchr_exists_twice         ........................ passed ...   9
test_strchr_not_exists           ........................ passed ...   1
test_strrchr_exists              ........................ passed ...  11
test_strrchr_exists_twice        ........................ passed ...   9
test_strrchr_not_exists          ........................ passed ...   1
test_strstr_exists               ........................ passed ...  11
test_strstr_exists_twice         ........................ passed ...   9
test_strstr_not_exists           ........................ passed ...   1
test_strrstr_exists              ........................ passed ...  11
test_strrstr_exists_twice        ........................ passed ...   9
test_strrstr_not_exists          ........................ passed ...   1
test_memcpy                      ........................ passed ...  11
test_memcpy_part                 ........................ passed ...   9
test_memcmp_equal_size_equal     ........................ passed ...   9
test_memcmp_diff_contents_equal  ........................ passed ...   9
test_memcmp_diff_size_equal      ........................ passed ...   9
test_memset                      ........................ passed ...   9
test_memset_part                 ........................ passed ...   9
test_memmove_apart               ........................ passed ...   9
test_memmove_src_before_dst      ........................ passed ...   9
test_memmove_src_after_dst       ........................ passed ...   9
test_open_non_existent_file      ........................ passed ...   8
test_open_invalid_access_mode    ........................ passed ...   8
test_open_file_as_directory      ........................ passed ...   8
test_open_directory_for_writing  ........................ passed ...   8
test_open_force_invalid_creation ........................ passed ...   8
test_open_close_existent_file    ........................ passed ...   8
test_open_close_create_file      ........................ passed ...   8
test_open_read_write_only_mode   ........................ passed ...   8
test_open_write_read_only_mode   ........................ passed ...   8
test_lseek_invalid_fd            ........................ passed ...   8
test_lseek_invalid_whence        ........................ passed ...   8
test_lseek_invalid_offset        ........................ passed ...   8
test_lseek_set                   ........................ passed ...   8
test_lseek_cur                   ........................ passed ...   8
test_lseek_end                   ........................ passed ...   8
test_lseek_combined              ........................ passed ...   8
test_truncate_read_only_file     ........................ passed ...   8
test_truncate_invalid_size       ........................ passed ...   8
test_truncate_directory          ........................ passed ...   8
test_truncate_non_existent_file  ........................ passed ...   8
test_truncate_file               ........................ passed ...   8
test_ftruncate_read_only_file    ........................ passed ...   8
test_ftruncate_invalid_size      ........................ passed ...   8
test_ftruncate_directory         ........................ passed ...   8
test_ftruncate_bad_fd            ........................ passed ...   8
test_ftruncate_file              ........................ passed ...   8
test_stat_non_existent_file      ........................ passed ...   8
test_stat_regular_file           ........................ passed ...   8
test_fstat_bad_fd                ........................ passed ...   8
test_fstat_regular_file          ........................ passed ...   8
test_puts                        ........................ passed ...  15
test_open_close_create_file      ........................ passed ...  10
test_open_close_read_byte        ........................ passed ...  10
test_ftruncate                   ........................ passed ...  10
test_truncate                    ........................ passed ...  10
test_fstat                       ........................ passed ...  10
test_stat                        ........................ passed ...  10
test_sleep                       ........................ passed ...  20
test_nanosleep                   ........................ passed ...  20
test_mmap                        ........................ passed ...   8
test_mmap_bad_fd                 ........................ passed ...   8
test_mmap_bad_flags              ........................ passed ...   8
test_mremap                      ........................ passed ...   8
test_malloc                      ........................ passed ...   8
test_malloc_two                  ........................ passed ...   8
test_malloc_access               ........................ passed ...   8
test_malloc_memset               ........................ passed ...   8
test_malloc_memcpy               ........................ passed ...   8
test_calloc                      ........................ passed ...   8
test_realloc                     ........................ passed ...   8
test_realloc_access              ........................ passed ...   8
test_realloc_memset              ........................ passed ...   8
test_realloc_array               ........................ passed ...   8
test_malloc                      ........................ passed ...  10
test_multiple_malloc             ........................ passed ...  10
test_malloc_free                 ........................ passed ...  10
test_multiple_malloc_free        ........................ passed ...  10
test_malloc_free_sequence        ........................ passed ...  10
test_malloc_perm_ok              ........................ passed ...  10
test_malloc_perm_notok           ........................ passed ...  10
test_mmap                        ........................ passed ...  10
test_mmap_munmap                 ........................ passed ...  10
test_mmap_perm_ok                ........................ passed ...  10
test_mmap_perm_notok             ........................ passed ...  10
test_mmap_perm_none              ........................ passed ...  10

Total:                                                            90/100

Running the Linters​

To run the linters, use the make linter command in the tests/ directory. +Note that the linters have to be installed on your system: checkpatch.pl, cpplint, shellcheck with certain configuration options. +It's easiest to run them in a Docker-based setup with everything configured:

student@so:~/.../assignments/mini-libc/tests$ make lint
[...]
cd .. && checkpatch.pl -f checker/*.sh tests/*.sh
[...]
cd .. && cpplint --recursive src/ tests/ checker/
[...]
cd .. && shellcheck checker/*.sh tests/*.sh

Behind the Scenes​

For a fine grained approach, build tests and ignore errors (due to missing source code and header files) by using:

student@so:~/.../assignments/mini-libc/tests$ make -i

Then run the tests, either individually via executable files and scripts:

student@so:~/.../assignments/mini-libc/tests$ ./test_lseek.sh
test_lseek                       ........................ passed ...  10

student@so:~/.../assignments/mini-libc/tests$ ./test_memory
test_mmap                        ........................ passed ...   8
test_mmap_bad_fd                 ........................ passed ...   8
[...]

Or run them all via the run_all_tests.sh script:

student@so:~/.../assignments/mini-libc/tests$ ./run_all_tests.sh
test_strcpy                      ........................ passed ...   9
test_strcpy_append               ........................ passed ...   9
test_strncpy                     ........................ passed ...   9
[...]

Resources​

+ + + + \ No newline at end of file diff --git a/17/Assignments/Mini Shell/index.html b/17/Assignments/Mini Shell/index.html new file mode 100644 index 0000000000..f58e0f6f1c --- /dev/null +++ b/17/Assignments/Mini Shell/index.html @@ -0,0 +1,37 @@ + + + + + +Minishell | Operating Systems + + + + +
+
Skip to main content

Minishell

Objectives​

  • Learn how shells create new child processes and connect the I/O to the terminal.
  • Gain a better understanding of the fork() function wrapper.
  • Learn to correctly execute commands written by the user and treat errors.

Statement​

Introduction​

A shell is a command-line interpreter that provides a text-based user interface for operating systems. +Bash is both an interactive command language and a scripting language. +It is used to interact with the file system, applications, operating system and more.

For this assignment you will build a Bash-like shell with minimal functionalities like traversing the file system, running applications, redirecting their output or piping the output from one application into the input of another. +The details of the functionalities that must be implemented will be further explained.

Shell Functionalities​

Changing the Current Directory​

The shell will support a built-in command for navigating the file system, called cd. +To implement this feature you will need to store the current directory path because the user can provide either relative or absolute paths as arguments to the cd command.

The built-in pwd command will show the current directory path.

Check the following examples below to understand these functionalities.

> pwd
/home/student
> cd operating-systems/assignments/minishell
> pwd
/home/student/operating-systems/assignments/minishell
> cd inexitent
no such file or directory
> cd /usr/lib
> pwd
/usr/lib

NOTE: Using the cd command without any arguments or with more than one argument doesn't affect the current directory path. +Make sure this edge case is handled in a way that prevents crashes.

Closing the Shell​

Inputting either quit or exit should close the minishell.

Running an Application​

Suppose you have an executable named sum in the current directory. +It takes arbitrarily many numbers as arguments and prints their sum to stdout. +The following example shows how the minishell implemented by you should behave.

> ./sum 2 4 1
7

If the executable is located at the /home/student/sum absolute path, the following example should also be valid.

> /home/student/sum 2 4 1
7

Each application will run in a separate child process of the minishell created using fork.

Environment Variables​

Your shell will support using environment variables. +The environment variables will be initially inherited from the bash process that started your minishell application.

If an undefined variable is used, its value is the empty string: "".

NOTE: The following examples contain comments which don't need to be supported by the minishell. +They are present here only to give a better understanding of the minishell's functionalities.

> NAME="John Doe"                    # Will assign the value "John Doe" to the NAME variable
> AGE=27                             # Will assign the value 27 to the AGE variable
> ./identify $NAME $LOCATION $AGE    # Will translate to ./identify "John Doe" "" 27 because $LOCATION is not defined

A variable can be assigned to another variable.

> OLD_NAME=$NAME    # Will assign the value of the NAME variable to OLD_NAME

Operators​

Sequential Operator​

By using the ; operator, you can chain multiple commands that will run sequentially, one after another. +In the command expr1; expr2 it is guaranteed that expr1 will finish before expr2 is be evaluated.

> echo "Hello"; echo "world!"; echo "Bye!"
Hello
world!
Bye!
Parallel Operator​

By using the & operator you can chain multiple commands that will run in parallel. +When running the command expr1 & expr2, both expressions are evaluated at the same time (by different processes). +The order in which the two commands finish is not guaranteed.

> echo "Hello" & echo "world!" & echo "Bye!"  # The words may be printed in any order
world!
Bye!
Hello
Pipe Operator​

With the | operator you can chain multiple commands so that the standard output of the first command is redirected to the standard input of the second command.

Hint: Look into anonymous pipes and file descriptor inheritance while using fork.

> echo "Bye"                      # command outputs "Bye"
Bye
> ./reverse_input
Hello                             # command reads input "Hello"
olleH                             # outputs the reversed string "olleH"
> echo "world" | ./reverse_input  # the output generated by the echo command will be used as input for the reverse_input executable
dlrow
Chain Operators for Conditional Execution​

The && operator allows chaining commands that are executed sequentially, from left to right. +The chain of execution stops at the first command that exits with an error (return code not 0).

# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"
> echo "H" && echo "e" && echo "l" && ./throw_error && echo "l" && echo "o"
H
e
l
ERROR: I always fail

The || operator allows chaining commands that are executed sequentially, from left to right. +The chain of execution stops at the first command that exits successfully (return code is 0).

# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"
> ./throw_error || ./throw_error || echo "Hello" || echo "world!" || echo "Bye!"
ERROR: I always fail
ERROR: I always fail
Hello
Operator Priority​

The priority of the available operators is the following. +The lower the number, the higher the priority:

  1. Pipe operator (|)
  2. Conditional execution operators (&& or ||)
  3. Parallel operator (&)
  4. Sequential operator (;)

I/O Redirection​

The shell must support the following redirection options:

  • < filename - redirects filename to standard input
  • > filename - redirects standard output to filename
  • 2> filename - redirects standard error to filename
  • &> filename - redirects standard output and standard error to filename
  • >> filename - redirects standard output to filename in append mode
  • 2>> filename - redirects standard error to filename in append mode

Hint: Look into open, dup2 and close.

Testing​

The testing is automated. +Tests are located in the inputs/ directory.

student@os:~/.../assignments/minishell/checker/_test/inputs$ ls -F
test_01.txt  test_03.txt  test_05.txt  test_07.txt  test_09.txt  test_11.txt  test_13.txt  test_15.txt  test_17.txt
test_02.txt  test_04.txt  test_06.txt  test_08.txt  test_10.txt  test_12.txt  test_14.txt  test_16.txt  test_18.txt

To execute tests you need to run:

student@os:~/.../assignments/minishell/checker$ ./run_all.sh

Debug​

To inspect the differences between the output of the mini-shell and the reference binary set DO_CLEANUP=no in _test/run_test.sh. +To see the results of the tests, you can check _test/outputs/ directory.

Memory leaks​

To inspect the unreleased resources (memory leaks, file descriptors) set USE_VALGRIND=yes and DO_CLEANUP=no in _test/run_test.sh. +You can modify both the path to the Valgrind log file and the command parameters. +To see the results of the tests, you can check _test/outputs/ directory.

Checkpatch​

checkpatch.pl is a script used in the development of the Linux kernel. +It is used to check patches that are submitted to the kernel mailing list for adherence to the coding style guidelines of the Linux kernel.

The script checks the code for common coding style issues, such as indentation, spacing, line length, function and variable naming conventions, and other formatting rules. +It also checks for some common errors, such as uninitialized variables, memory leaks, and other potential bugs.

You can download the checkpatch.pl script from the official Linux kernel repository.

Running the following command will show you linting warnings and errors:

./checkpatch.pl --no-tree --terse -f /path/to/your/code.c
+ + + + \ No newline at end of file diff --git a/17/Assignments/Mini Shell/util/parser/index.html b/17/Assignments/Mini Shell/util/parser/index.html new file mode 100644 index 0000000000..0e124caa98 --- /dev/null +++ b/17/Assignments/Mini Shell/util/parser/index.html @@ -0,0 +1,20 @@ + + + + + +Parser | Operating Systems + + + + +
+
Skip to main content

Parser

The parser is made using Bison and Flex.

The parser is used to parse the commands entered by the user, stored in structure command_t.

This structure is defined in the file parser.h and encapsulates a tree representation of the command.

You can use DisplayStructure.cpp to display the structure using various tests.

Also you can use CUseParser.c or UseParser.cpp to see how to use the parser in C or C++ and print the structure of the command.

Compile​

Run the following commands in the root of parser directory:

student@os:/.../minishell/util/parser$ make

Usage​

The parser is represented by two files:

  • parser.y - implementation of the parser
  • parser.l - implementation of the lexer

Build process​

The Makefile first generates the files parser.yy.c and parser.tab.c from parser.y and parser.l. +After that, it compiles the files parser.yy.c and parser.tab.c to generate the object files parser.yy.o and parser.tab.o. +To use the parser, you need to link the object files parser.yy.o and parser.tab.o with your program.

Example​

  • CUseParser.c - example of using the parser in C
  • UseParser.cpp - example of using the parser in C++
  • DisplayStructure.cpp - reads multiple commands and displays the structure of the resulting tree

Tests​

More tests can be found in the tests directory:

student@os:/.../minishell/util/parser$ cd tests

student@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>small_tests.out <small_tests.txt

student@os:/.../minishell/util/parser/tests$ cat small_tests.out
> mkdir tmp
Command successfully read!
command_t (
    scmd (
        simple_command_t (
            verb (
                'mkdir'
            )
            params (
                'tmp'
            )
        )
    )
)

> cd tmp
Command successfully read!
command_t (
    scmd (

...
student@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>ugly_tests.out <ugly_tests.txt

student@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>negative_tests.out <negative_tests.txt

Note​

The parser will fail with an error of unknown character if you use the Linux parser (which considers the end of line as \n) on Windows files (end of line as \r\n) because at the end of the lines (returned by getline()) there will be a \r followed by \n. +The opposite works (Windows parser with Linux files). +The test files use the Linux convention (\n).

Other information​

More information about the parser can be found in the file parser.h.

+ + + + \ No newline at end of file diff --git a/17/Assignments/Parallel Graph/index.html b/17/Assignments/Parallel Graph/index.html new file mode 100644 index 0000000000..0e3289a2a6 --- /dev/null +++ b/17/Assignments/Parallel Graph/index.html @@ -0,0 +1,40 @@ + + + + + +Parallel Graph | Operating Systems + + + + +
+
Skip to main content

Parallel Graph

For this assignment we will implement a generic thread pool, which we will then use to traverse a graph and compute the sum of the elements contained by the nodes. +You will be provided with a serial implementation of the graph traversal and with most of the data structures needed to implement the thread pool. +Your job is to write the thread pool routines and then use the thread pool to traverse the graph.

Thread Pool Description​

A thread pool contains a given number of active threads that simply wait to be given specific tasks. +The threads are created when the thread pool is created they poll a task queue until a task is available. +Once tasks are put in the task queue, the threads start running the task. +A thread pool creates N threads when the thread pool is created and does not destroy (join) them throughout the lifetime of the thread pool. +That way, the penalty of creating and destroying threads ad hoc is avoided. +As such, you must implement the following functions (marked with TODO in the provided skeleton):

  • task_create: creates an os_task_t that will be put in the task queue - a task consists of a function pointer and an argument.
  • add_task_in_queue: adds a given task in the thread pool's task queue.
  • get_task: get a task from the thread pool's task queue.
  • threadpool_create: allocate and initialize a new thread pool.
  • thread_loop_function: all the threads in the thread pool will execute this function - they all wait until a task is available in the task queue; once they grab a task they simply invoke the function that was provided to task_create.
  • threadpool_stop: stop all the threads from execution.

Notice that the thread pool is completely independent from any given application. +Any function can be registered in the task queue.

Graph Traversal​

Once you have implemented the thread pool, you need to test it by using it for computing the sum of all the nodes of a graph. +A serial implementation for this algorithm is provided in skep/serial.c +To make use of the thread pool, you will need to create tasks that will be put in the task queue. +A task consists of 2 steps:

  • adding the current node value to the overall sum.
  • creating tasks and adding them to the task queue for the neighbouring nodes.

Since the threads are polling the task queue indefinitely, you need to find a condition for the threads to stop once the graph has been traversed completely. +This condition should be implemented in a function that is passed to threadpool_stop. +threadpool_stop then needs to wait for the condition to be satisfied and then joins all the threads.

Synchronization​

For synchronization you can use mutexes, semaphores, spinlocks, condition variables - anything that grinds your gear. +However, you are not allowed to use hacks such as sleep, printf synchronization or adding superfluous computation.

Input Files​

Reading the graphs from the input files is being taken care of the functions implemented in src/os_graph.c. +A graph is represented in input files as follows:

  • first line contains 2 integers N and M: N - number of nodes, M - numbed or edges
  • second line contains N integer numbers - the values of the nodes
  • the next M lines contain each 2 integers that represent the source and the destination of an edge

Data Structures​

Graph​

A graph is represented internally as an os_graph_t (see src/os_graph.h).

List​

A list is represented internally as an os_queue_t (see src/os_list.h). +You will use this list to implement the task queue.

Thread pool​

A thread pool is represented internally as an os_threadpool_t (see src/os_threadpool.h) +The thread pool contains information about the task queue and the threads.

You are not allowed to modify these data structures. +However, you can create other data structures that leverage these ones.

Infrastructure​

Compilation​

To compile both the serial and the parallel version, enter the src/ directory and run:

make

That will create the serial and parallel binaries/

Testing​

Input tests cases are located in tests/in/. +The parallel and the serial version should provide the same results for the same input test case.

If you want manually run a single test, use commands such as below while in the src/ directory:

$./parallel ../tests/in/test5.in
-11

$ ./serial ../tests/in/test5.in
-11

Checker​

The testing is automated and performed with the checker.py script from the tests/ directory. +It's easiest to use the Makefile to run the tests:

$ make check
[...]
SRC_PATH=../src python checker.py
test1.in                         ....................... passed ...   4.5
test2.in                         ....................... passed ...   4.5
test3.in                         ....................... passed ...   4.5
test4.in                         ....................... passed ...   4.5
test5.in                         ....................... passed ...   4.5
test6.in                         ....................... passed ...   4.5
test7.in                         ....................... passed ...   4.5
test8.in                         ....................... passed ...   4.5
test9.in                         ....................... passed ...   4.5
test10.in                        ....................... passed ...   4.5
test11.in                        ....................... passed ...   4.5
test12.in                        ....................... passed ...   4.5
test13.in                        ....................... passed ...   4.5
test14.in                        ....................... passed ...   4.5
test15.in                        ....................... passed ...   4.5
test16.in                        ....................... passed ...   4.5
test17.in                        ....................... passed ...   4.5
test18.in                        ....................... passed ...   4.5
test19.in                        ....................... passed ...   4.5
test20.in                        ....................... passed ...   4.5

Total:                                                              90/100

It's recommended that you use the local Docker-based checker. +You would use the command:

./local.sh checker

to run the checker in a Docker-based environment that is identical to the one used for the official assignment evaluation.

Grading​

The grade that the checker outputs is not the final grade. +Your homework will be manually inspected and may suffer from penalties ranging from 1 to 100 points depending on the severity of the hack, including, but not limited to:

  • using a single mutex at the beginning of the traversal
  • not using the thread pool to solve the homework
  • inefficient usage of synchronization
  • incorrect graph traversal

Deployment​

Your implementation needs to be contained in the src/os_threadpool.c and src/os_parallel.c files. +Any other files that you are using will not be taken into account. +Any modifications that you are doing to the other files in the src/ directory will not be taken into account.

+ + + + \ No newline at end of file diff --git a/17/Assignments/index.html b/17/Assignments/index.html new file mode 100644 index 0000000000..13f2cc3b71 --- /dev/null +++ b/17/Assignments/index.html @@ -0,0 +1,16 @@ + + + + + +Assignments | Operating Systems + + + + +
+
Skip to main content

Assignments

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/arena/index.html b/17/Lab/Application Interaction/arena/index.html new file mode 100644 index 0000000000..9d9a8c3db1 --- /dev/null +++ b/17/Lab/Application Interaction/arena/index.html @@ -0,0 +1,53 @@ + + + + + +Arena | Operating Systems + + + + +
+
Skip to main content

Arena

Oneko​

An alternative to xeyes which allows us to observe Unix sockets is oneko. +Going through the same steps, we see that the application also create a Unix socket, then connects to the path @"/tmp/.X11-unix/X0".

student@os:~$ strace -e trace=socket,connect oneko
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X1"}, 20) = 0
--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---

When running oneko, what differs from xeyes is the SIGALRM signal. +This means that oneko uses a timer, which is periodically set, and then it expires only to be reset again. +The purpose of this timer is to slow down the cat.

Verifying the communication between the X server and oneko is easy. +We see that the cat follows our mouse cursor, behaving similarly to xeyes. +After running oneko under strace, we see the communication uses the UNIX socket created at the beginning:

strace -e 'trace=!poll' -e trace='socket,connect,recvmsg' oneko |& grep -v '\-1 EAGAIN'

Quiz

D-Bus​

Use the dbus python bindings to get the computer's battery level using a python script. +You can start from the documentation here. +You need to read the sections Connecting to the Bus, Proxy objects, and Interfaces and methods.

There's also a skeleton you can use in support/dbus/get_battery_level.py.

In summary, your script will start by connecting to the System Bus. +Then you'll use the get_object method to obtain a proxy object. +On this proxy object, you can actually do the method call as explained here:

To call a method, call the method of the same name on the proxy object, passing in the interface name via the dbus_interface keyword argument

So, if you want to call the method this.is.an.interface.method with the arguments A and B you can do it like this:

result = proxy.method(A, B, dbus_interface = "this.is.an.interface")

OS-Cloud: More Disk Customization​

You might have probably noticed that there are 2 types of disk customizations:

  • One type is for things that can be done without running the virtual machine. +If we only want to modify some files inside the disk filesystem, we can do so by mounting the disk. +This is done, for example, in the disk-templates/ubuntu_22.04/setup_root_password.sh script. +There we use nbd_connect_qcow2 + mount to mount the disk, then we modify the /etc/shadow file to change the root password.

  • The second case is for operations that must be done with the virtual machine running. +These are handled in the ubuntu_22_04_vm_prepare function: the virtual machine is first started (start_qemu_for_vm), then pexpect is used to interact with the virtual machine via the qemu serial console. +Here we do things like running ssh-keygen - a binary that is part of the disk filesystem, which depends on other parts of the operating system from the disk to be running. +Note that in ubuntu_22_04_vm_prepare, for convenience, we also do some customizations that fall into the first category (like modifying /etc/ssh/sshd_config).

Copy Additional Files to the Newly Created Disk​

This is a customization from the first category. +In disk-templates/ubuntu_22.04/files there is a file called 99-os-cloud-welcome (a script that prints a greeting message). +We want to copy this file to /etc/update-motd.d in our newly created disk, so that it will run whenever a user logs in.

To do this, you will create a script called copy_files.sh in disk-templates/ubuntu_22.04. +This script will receive a path to a qcow2 disk file as an argument, it will mount the disk, and then copy the file to the necessary location. +Then, in the create_disk_from_template function in disk.py you will call this script, similar with how the other scripts are called.

You can use disk-templates/ubuntu_22.04/setup_root_password.sh as an example.

SSH Key Setup​

We want to be able to log into the virtual machine using an ssh key, instead of the password 123456. +Notice that the vm_create API also accepts an ssh_key parameter. +Here, the user can provide an ssh public key, which the system will install in /root/.ssh/authorized_keys in the newly created virtual machine.

Your task is to implement this feature, as a customization from the second category (that is, implemented in the ubuntu_22_04_vm_prepare function). +The key will be accessible to the function as the ssh_pub_key parameter. +Then it's only a matter of writing the key to the appropriate place, using a command like echo key > /root/.ssh/authorized_keys. +Note that the /root/.ssh directory might not exist, so you need to create it as well.

After the feature is complete, you can test it using the keys in the support/os-cloud/keys directory. +This directory contains a pair of public-private keys. +The directory will also be mounted inside the os-cloud container in /keys.

You will create another virtual machine, passing the public key to vm_create:

student@os:~/.../support/os-cloud$ curl -H "Content-Type: application/json" \
    -d '{ "name": "my_vm2", "image": "ubuntu_22.04", "network": "default", "mem_size": "2G", "disk_size": "10G", "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8CDHgeE4NIIIih3wSz58GDkfPLUk2m9gmbZB1f6o8Lzawzb3HVFpslAUWK0f/Ymw9cloInpMo50gWMYFSyJ7ZrOWWak54BedpHDkFAxxy+JCE9b+pkKsrAT7wiir7gn2LHlhj55FLZkC9PpM9cBcrMfzlcP9Bf+2cnpDdINybSLmOUmrI23ANteM4lEVaa2yEbCaJk6dFB8+atz5zPjvVI0Hd+kJK7yJ0xV6Zc2ADle7TKW3dyiXOE9qFKe9933Rj7ocqNXCAO1cxUoJCVuVS7lh+1pSSPXLWLTOhVp/XiLGWVP6KRYmmn710MWKm9Kj1tPiGUphUraL20SJiRT6/ os-cloud-user"}' \
    localhost:5000/vm_create
{"id":2,"status":"ok"}

Obtain the IP address that was allocated to the new vm:

student@os:~/.../support/os-cloud$ curl -s -H "Content-Type: application/json" -d '{ "id": 2 }' localhost:5000/vm_info | jq .
{
  "disk_size": 10737418240,
  "id": 2,
  "ip": "192.168.0.3",
  "mem_size": 2147483648,
  "name": "my_vm2",
  "network": "default",
  "os": "ubuntu_22.04",
  "state": "RUNNING"
}

Then go inside the os-cloud container and ssh to the vm using the private key in /keys. +It should work without prompting for the password:

student@os:~/.../support/os-cloud$ docker-compose exec os-cloud bash
root@ac93d3d6cab2:/app# ssh -i /keys/ssh_key root@192.168.0.3
Welcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-56-generic x86_64)
[...]
Powered by OS Cloud
Last login: Mon Jan  2 19:34:53 2023 from 192.168.0.1
root@ubuntu:~#

OS-Cloud: Internet Access​

Notice that our virtual machines don't have internet access:

Powered by OS Cloud
Last login: Mon Jan  2 19:52:47 UTC 2023 on ttyS0
root@ubuntu:~# curl google.com
curl: (6) Could not resolve host: google.com

In this task, we want to fix this problem. +To do this, we must first understand how the networking for the virtual machines is done.

First, there is the concept of a network, which you saw in the previous section. +There is a network called default, with the address of 192.168.0.0/24. +All virtual machines are part of this network, that's why they were allocated ip addresses like 192.168.0.2.

Let's go inside the os-cloud container and take a look at the network interfaces:

$ docker-compose exec os-cloud bash
root@8333e5cefb0d:/app# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 8a:68:b7:5b:6b:45 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.1/16 scope global br0
       valid_lft forever preferred_lft forever
3: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether 8a:68:b7:5b:6b:45 brd ff:ff:ff:ff:ff:ff
4: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000
    link/ether fa:f8:7f:83:50:8f brd ff:ff:ff:ff:ff:ff
77: eth0@if78: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:16:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.22.0.3/16 brd 172.22.255.255 scope global eth0
       valid_lft forever preferred_lft forever

root@8333e5cefb0d:/app# ps -ef | grep qemu
root          19       8 29 09:15 ?        00:01:26 qemu-system-x86_64 -m 2048 -hda /vm-disks/1/disk.qcow2 -net nic,macaddr=52:54:00:12:34:00 -net tap,ifname=tap0,script=no -monitor telnet::10001,server,nowait -serial telnet::10002,server,nowait -nographic -enable-kvm
root          29       8 28 09:15 ?        00:01:24 qemu-system-x86_64 -m 2048 -hda /vm-disks/2/disk.qcow2 -net nic,macaddr=52:54:00:12:34:01 -net tap,ifname=tap1,script=no -monitor telnet::10003,server,nowait -serial telnet::10004,server,nowait -nographic -enable-kvm

Here we have 2 virtual machines running. +Each virtual machine uses a tap interface (the -net tap,ifname=tap0,script=no parameter for qemu). +This means that the ens0 interface inside the virtual machine corresponds to the tap0 interface outside the virtual machine. +All the tap interfaces are bridged together into the br0 bridge, which has the ip address 192.168.0.1. +Also, each virtual machine has the default gateway configured to be 192.168.0.1.

In summary, it looks something like this:

os-cloud

All the traffic coming from the virtual machines passes through the br0 interface. +So, in order to make the internet work, all we have to do is a simple NAT, with a command like:

root@8333e5cefb0d:/app# iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE

Now, the virtual machines should have internet access:

root@8333e5cefb0d:/app# ssh root@192.168.0.2
[...]
root@ubuntu:~# curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>

Now your task is to run the iptables command above automatically when the system starts, so that it's not necessary to run it manually like we did in the above example.

A good place to do this is in the create_one_network function in network.py. +There you can add another subprocess.run call to run iptables. +The 192.168.0.0/24 value should not be hardcoded, but you can take it from the ip_with_prefixlen member of the Net object.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/dbus/index.html b/17/Lab/Application Interaction/dbus/index.html new file mode 100644 index 0000000000..893bbac7ef --- /dev/null +++ b/17/Lab/Application Interaction/dbus/index.html @@ -0,0 +1,44 @@ + + + + + +D-Bus | Operating Systems + + + + +
+
Skip to main content

D-Bus

D-Bus is an Inter-Process Communication (IPC) mechanism that is commonly present on Linux. +It is particularly used by various components of the desktop environment (like GNOME) to communicate between one another, although the system itself is general-purpose and can be used in any other situations.

As the name suggests, the communication model is that of a bus: processes connect to the bus, then exchange messages with other processes through the bus. +The bus itself is implemented by the dbus-daemon, and there are in fact multiple buses: one system bus, accessible system-wide, and one or more session buses, each one corresponding to one user login session.

Every process that connects to D-Bus receives a unique connection name. +This name can be something human-readable, like org.freedesktop.Notifications, or some generated ID, like :1.63. +Once a process is connected, it can expose one or multiple objects. +An object has a path-like name, consisting of strings separated by a slash character (for example, /org/freedesktop/Notifications). +Each object contains one or more interfaces, which have the methods that can be called on that object.

D-Bus Inspection with D-Feet​

In order to better understand these concepts, we'll use a graphical tool (D-Feet) to inspect all the available D-Bus objects on our system.

Run D-Feet and select Session Bus from the top button:

dfeet-session-bus

On the left panel, we can see all the processes connected to D-Bus with their associated connection names. +Scroll down and find org.freedesktop.Notifications. +On the right side, expand /org/freedesktop/Notifications and then expand the org.freedesktop.Notifications interface. +The window should look like this:

dfeet-notifications

Some observations:

  • The bus communication happens over a Unix socket, with the path /run/user/1000/bus.

  • org.freedesktop.Notifications on the left panel is the connection name.

  • The process that has connected with this name is /usr/bin/gjs /usr/share/gnome-shell/org.gnome.Shell.Notifications and has the pid of 4373.

  • This process exposes one object: /org/freedesktop/Notifications. +Note that the object name is the same as the connection name, where the dots have been replaced with slashes. +This is not a requirement, as the objects exposed by a process can have any name.

  • The object has 4 interfaces: org.freedesktop.DBus.Introspectable, org.freedesktop.DBus.Peer, org.freedesktop.DBus.Properties and org.freedesktop.Notifications. +Note that the last one (org.freedesktop.Notifications) is the same as the connection name, but this again is just a coincidence, not a requirement.

  • The interface org.freedesktop.Notifications has some methods that can be called, such as Notify.

Calling D-Bus Methods​

The application behind org.freedesktop.Notifications is responsible with desktop notifications (the small bubbles of text that appear at the top of the screen when some event happens). +When an application wants to send a notification it needs to connect to D-Bus and call the Notify method from the org.freedesktop.Notifications interface.

In this example, we want to call the Notify method ourselves. +To do this, we must first understand the signature of this method:

Notify (String arg_0, UInt32 arg_1, String arg_2, String arg_3, String arg_4, Array of [String] arg_5, Dict of {String, Variant} arg_6, Int32 arg_7) ↦ (UInt32 arg_8)

This doesn't tell us much, but we can find more documentation here, since freedesktop is an open standard.

We'll set the arguments to the following (for our simple case, most of them will be unused):

  • app_name: ""

  • replaces_id: 0

  • app_icon: ""

  • summary: "This is the title"

  • body: "This is the content"

  • actions: []

  • hints: {}

  • expire_timeout: -1

Now the question is how to actually call the method. +Normally, we would have to write an application that connects to D-Bus and executes the call. +But for demonstrative purposes there are easier ways.

One way is directly from d-feet. +If we double-click on the Notify method in the right-side pane of d-feet, a window will open that allows us to call the method with any arguments that we want:

dfeet-execute-dialog

Then we click the Execute button and the notification will appear:

dfeet-execute-

Another way is from the command-line. There's the gdbus tool that can do this:

student@os:~$ gdbus call \
      --session \
      --dest org.freedesktop.Notifications \
      --object-path /org/freedesktop/Notifications \
      --method org.freedesktop.Notifications.Notify \
      -- \
      '""' \
      0 \
      '""' \
      "This is the title" \
      "This is the content" \
      [] \
      {} \
      -1

Let's see how it works:

gdbus-notify

You can also find this gdbus call in the support/dbus/send_notification.sh script.

Inspecting the Low-level Communication​

Let's run gdbus under strace to see what's happening behind the scenes. +Run the script in support/dbus/send_notification_strace.sh:

strace: Process 61888 attached
[pid 61887] socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 5
[pid 61887] connect(5, {sa_family=AF_UNIX, sun_path="/run/user/1000/bus"}, 110) = 0
[pid 61887] sendmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\0", iov_len=1}], msg_iovlen=1,
msg_control=[{cmsg_len=28, cmsg_level=SOL_SOCKET, cmsg_type=SCM_CREDENTIALS, cmsg_data={pid=61887,
uid=1000, gid=1000}}],
msg_controllen=32, msg_flags=0}, MSG_NOSIGNAL) = 1
strace: Process 61889 attached

[...]

[pid 61889] sendmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1T\0\0\0\3\0\0\0\237\0\0\0\1
\1o\0\36\0\0\0/org/freedesktop/Notifications\0\0\2\1s\0\35\0\0\0org.freedesktop.Notifications\0\0\0\6\1s\0\35
\0\0\0org.freedesktop.Notifications\0\0\0\10\1g\0\rsusssasa{sv}i\0\0\0\0\0\0\3\1s\0\6\0\0\0Notify\0\0\0\0\0\0
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\21\0\0\0This is the title\0\0\0\23\0\0\0This is the content\0\0\0\0\0\0\0\0\0
\0\0\0\0\377\377\377\377", iov_len=260}], msg_iovlen=1, msg_controllen=0,
 msg_flags=0}, MSG_NOSIGNAL) = 260
[pid 61889] recvmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\2\1\1\4\0\0\0\312\0\0\0.\0\0\0", iov_len=16}],
msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 16
[pid 61889] recvmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\6\1s\0\6\0\0\0:1.497\0\0\10\1g\0\1u\0\0\5\1u\0
\3\0\0\0\7\1s\0\5\0\0\0:1.49\0\0\0\36\0\0\0", iov_len=52}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_CMSG_CLOEXEC) = 52
(uint32 30,)
[pid 61889] +++ exited with 0 +++
[pid 61888] +++ exited with 0 +++
+++ exited with 0 +++

We see a Unix socket being created and a connection made to /run/user/1000/bus, as expected. +Then a series of messages are exchanged on the socket, which are part of the D-Bus protocol. +On a closer look, we can even identify some strings from our notification, like This is the title or This is the content:

[pid 61889] sendmsg(5, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\0\1T\0\0\0\3\0\0\0\237\0\0\0\1
\1o\0\36\0\0\0/org/freedesktop/Notifications\0\0\2\1s\0\35\0\0\0org.freedesktop.Notifications\0\0\0\6\1s\0\35
\0\0\0org.freedesktop.Notifications\0\0\0\10\1g\0\rsusssasa{sv}i\0\0\0\0\0\0\3\1s\0\6\0\0\0Notify\0\0\0\0\0\0
\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\21\0\0\0This is the title\0\0\0\23\0\0\0This is the content\0\0\0\0\0\0\0\0\0
\0\0\0\0\377\377\377\377", iov_len=260}], msg_iovlen=1, msg_controllen=0,

Practice​

Use D-Bus to find out the computer's battery level. +There is the org.freedesktop.UPower interface on the system bus that can provide this information.

The method you need to call is org.freedesktop.DBus.Properties.Get from the /org/freedesktop/UPower/devices/DisplayDevice object.

This method needs 2 arguments: an interface name and a property name.

Those should be org.freedesktop.UPower.Device and Percentage respectively.

Then input all of the above into a gdbus call, which, if everything is correct, should output the battery percentage level as a number between 0 and 100.

Note: if you are running on a desktop computer or inside a virtual machine, you will get the value 0.0, because those systems don't have a battery.

This task can also be solved in a python script. +Check out the details in the arena section.

Firefox​

Let's do the following experiment:

  • Open the Firefox browser

  • From a terminal run firefox www.google.com

firefox-url-open

Notice that the URL we passed in the command-line was opened in the existing Firefox window as a new tab. +Even though we started a separate Firefox process, which should have created a separate new window, this didn't actually happen. +Instead, the process that we started from the command-line exited immediately and the site was opened in the already running Firefox instance.

Without any precise knowledge about Firefox internals, we can guess that something like this happened:

  • The newly started Firefox process detected that another instance of Firefox is already running

  • The newly started Firefox process sent a message to the existing running process, requesting it to open a URL in a new tab

Since we're talking about message passing between 2 processes, there's a chance that maybe D-Bus was involved. +Let's check: we'll use a tool called dbus-monitor that will print all messages passed through D-Bus.

student@os:~$ dbus-monitor

Then, in another terminal, we'll run firefox www.google.com again.

Going back to the dbus-monitor output, we find the following:

...
method call time=1655809062.813923 sender=:1.757 -> destination=org.mozilla.firefox.ZGVmYXVsdC1yZWxlYXNl serial=2 path=/org/mozilla/firefox/Remote; interface=org.mozilla.firefox; member=OpenURL
   array of bytes [
      02 00 00 00 1a 00 00 00 2f 00 00 00 2f 68 6f 6d 65 2f 61 64 72 69 61 6e
      73 00 2f 6f 70 74 2f 66 69 72 65 66 6f 78 2f 66 69 72 65 66 6f 78 00 77
      77 77 2e 67 6f 6f 67 6c 65 2e 63 6f 6d 00
   ]

There was a D-Bus call to org.mozilla.firefox.ZGVmYXVsdC1yZWxlYXNl, on the object /org/mozilla/firefox/Remote, method OpenURL from the org.mozilla.firefox interface. +Indeed, we see that this object exists in d-feet as well:

dfeet-firefox

We can try to call the OpenURL method ourselves, directly from d-feet. +The method has only one argument of the type Array of [Byte]. +Although there's no documentation for it, we can use the same byte array that we saw in dbus-monitor:

   array of bytes [
      02 00 00 00 1a 00 00 00 2f 00 00 00 2f 68 6f 6d 65 2f 61 64 72 69 61 6e
      73 00 2f 6f 70 74 2f 66 69 72 65 66 6f 78 2f 66 69 72 65 66 6f 78 00 77
      77 77 2e 67 6f 6f 67 6c 65 2e 63 6f 6d 00
   ]

(Note that 77 77 77 2e 67 6f 6f 67 6c 65 2e 63 6f 6d at the end is the string www.google.com, so that's another confirmation that we're on the right track).

dfeet-url-open

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/index.html b/17/Lab/Application Interaction/index.html new file mode 100644 index 0000000000..2681a906f7 --- /dev/null +++ b/17/Lab/Application Interaction/index.html @@ -0,0 +1,16 @@ + + + + + +Application Interaction | Operating Systems + + + + +
+
Skip to main content

Application Interaction

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/os-cloud/index.html b/17/Lab/Application Interaction/os-cloud/index.html new file mode 100644 index 0000000000..232de6b9dc --- /dev/null +++ b/17/Lab/Application Interaction/os-cloud/index.html @@ -0,0 +1,63 @@ + + + + + +OS Cloud | Operating Systems + + + + +
+
Skip to main content

OS Cloud

In this section, we are going to build a "toy cloud" called OS Cloud. +Similar to a real cloud (like aws), OS Cloud will allow us to create and manage virtual machines, through an http API.

Containers vs VMs​

Containers are a lightweight virtualization technology that allows multiple isolated user-space instances to run on a single host operating system. +They are often compared to chroot because they both provide isolated environments for running applications.

Cgroups limit, account for, and isolate the resource usage (CPU, memory, disk I/O, network, etc.) of a collection of processes. +They can be used to enforce resource limits, prioritization, accounting, and control. +Namespaces isolate processes from each other by creating independent views of system resources. +There are different types of namespaces, such as user, PID, network, mount, IPC, and UTS. +You can read more about them here, here and a particularly good read about namespaces can be found here

Quiz

However, containers take this isolation a step further by using kernel features such as namespaces and cgroups to provide a more complete and secure isolation of resources.

Virtual machines, on the other hand, are a heavier form of virtualization that involves running a complete guest operating system on top of a host operating system using a hypervisor. +This allows multiple guest operating systems to run on a single physical machine, each with its own set of virtualized hardware resources.

VMs vs Containers

One key difference between containers and VMs is the level of abstraction. +Containers virtualize the operating system, allowing multiple containers to share the same kernel while providing the illusion of running on separate machines. +VMs virtualize the hardware, allowing multiple guest operating systems to run on the same physical machine while providing the illusion of running on separate physical hardware.

Another difference is the resource overhead. +Containers are generally more lightweight than VMs because they share the host kernel and do not require a separate guest operating system to be installed. +This means that containers can start up faster and use less memory than VMs.

Quiz

Containers​

Our app will make use of docker containers. +A container is an OS-level virtualization method in which a group of userspace processes are isolated from the rest of the system.

Take for example a database server. +Instead of running it directly on the host system, we'll run it in its own container. +This way, the server process will be isolated from other processes on the system. +It will also have its own filesystem.

Besides isolation, containers are also useful for portability. +Since a container comes with its own filesystem image, we can pack it together will all the dependencies, so that the app will run correctly no matter what packages are installed on the host system.

Finally, since our application will consist of more than 1 container, we'll also use docker-compose, which is a tool that helps us with running multi-container applications.

Prerequisites​

Make sure the following packages are installed:

sudo apt-get -y update; sudo apt-get -y install docker-compose jq

Also, make sure your user can run docker commands. +If not, maybe you need to add it to the docker group:

sudo usermod -aG docker student

Then, after relogin:

student@os:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

If you are running inside a virtual machine, you need to enable nested virtualization. +Example for vmware:

nested-virt-vmware

For virtualbox:

nested-virt-vbox

If the button is greyed out, try from the command line:

student@os:~$ VBoxManage  list vms
"USO 2022-2023" {042a5725-bfb7-4a46-9743-1164d3acac23}

student@os:~$ VBoxManage modifyvm {042a5725-bfb7-4a46-9743-1164d3acac23} --nested-hw-virt on

Initial Liftoff​

First, we need to do some initial setup:

student@os:~/.../support/os-cloud$ ./initial_setup.sh

Then go to support/os-cloud and run:

student@os:~/.../support/os-cloud$ ./setup_db.sh
Setting up db
Starting db server
Waiting for db server to start
...
Stopping db server
Restarting db server
Waiting for db server to start
Creating tables
Stopping db server

student@os:~/.../support/os-cloud$ docker-compose up --build

Now the http API will listen on port localhost:5000. Let's try:

student@os:~/.../support/os-cloud$ curl localhost:5000
Welcome to OS Cloud!

Let's check the running virtual machines:

student@os:~/.../support/os-cloud$ curl localhost:5000/vm_list
[]

We got an empty list, since there are no virtual machines yet. +Let's create one (the command will take about 1 minute to complete):

student@os:~/.../support/os-cloud$ curl -H "Content-Type: application/json" \
    -d '{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "2G", "disk_size": "10G"}' \
    localhost:5000/vm_create
{"id":1,"status":"ok"}

Quiz

Check the virtual machine list again:

student@os:~/.../support/os-cloud$ curl localhost:5000/vm_list
[{"id":1,"name":"my_vm"}]

We can also use the jq tool to pretty print the json outputs:

student@os:~/.../support/os-cloud$ curl -s localhost:5000/vm_list | jq .
[
  {
    "id": 1,
    "name": "my_vm"
  }
]

We see our newly created virtual machine. +Let's get some information about it:

student@os:~/.../support/os-cloud$ curl -s -H "Content-Type: application/json" -d '{ "id": 1 }' localhost:5000/vm_info | jq .
{
  "disk_size": 10737418240,
  "id": 1,
  "ip": "192.168.0.2",
  "mem_size": 2147483648,
  "name": "my_vm",
  "network": "default",
  "os": "ubuntu_22.04",
  "state": "RUNNING"
}

We recognize some parameters that we specified at creation time, like mem_size and disk_size. +Also, the IP address 192.168.0.2 has been allocated for our machine.

More Implementation Details​

The application consists of 2 containers:

  • db, which runs a MySQL database

  • os-cloud, which runs the web application and the virtual machines

Let's check them. +After running docker-compose up, in another terminal run docker-compose ps:

student@os:~/.../support/os-cloud$ docker-compose ps
       Name                      Command              State                    Ports
------------------------------------------------------------------------------------------------------
os-cloud_db_1         docker-entrypoint.sh mariadbd   Up      3306/tcp
os-cloud_os-cloud_1   python3 -u app.py               Up      0.0.0.0:5000->5000/tcp,:::5000->5000/tcp

Now let's move inside the os-cloud container:

student@os:~/.../support/os-cloud$ docker-compose exec os-cloud bash
root@89a986d2526e:/app#

Since the virtual machines run inside this container, we should expect to see the one that we created in the previous step.

root@89a986d2526e:/app# ps -ef | cat
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 09:02 ?        00:00:00 /sbin/docker-init -- python3 -u app.py
root           7       1  0 09:02 ?        00:00:00 python3 -u app.py
root          12       7  6 09:02 ?        00:00:41 qemu-system-x86_64 -enable-kvm -m 2048 -hda /vm-disks/1/disk.qcow2 -net nic,macaddr=52:54:00:12:34:00 -net tap,ifname=tap0,script=no -monitor telnet::10001,server,nowait -serial telnet::10002,server,nowait -nographic
root          27       0  0 09:11 pts/3    00:00:00 bash
root          35      27  0 09:13 pts/3    00:00:00 ps -ef

Indeed, a qemu-system-x86_64 process is there. +The vm should be accessible via ssh on the IP 192.168.0.2 with password 123456 (if you get connection refused here, you need to wait a bit more for the machine to boot):

root@adf6e0bf4e6e:/app# ssh root@192.168.0.2
The authenticity of host '192.168.0.2 (192.168.0.2)' can't be established.
ED25519 key fingerprint is SHA256:3Mfa1fB9y4knUDJWEmEOTz9dWOE7SVhnH/kCBJ15Y0E.
This key is not known by any other names
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
Warning: Permanently added '192.168.0.2' (ED25519) to the list of known hosts.
root@192.168.0.2's password:
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-40-generic x86_64)

...

Last login: Thu Nov 17 07:49:55 2022
root@ubuntu:~#

The vm is also accessible on the serial console (notice the -serial telnet::10002,server,nowait argument to qemu). +If we start a telnet connection on port 10002, qemu will show us the virtual machine's serial console (basically the output that we normally see when running a virtual machine in text mode)

root@adf6e0bf4e6e:/app# telnet localhost 10002
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.

ubuntu login: root
Password:
Welcome to Ubuntu 22.04 LTS (GNU/Linux 5.15.0-40-generic x86_64)

...

Last login: Thu Nov 17 07:50:11 UTC 2022 from 192.168.0.1 on pts/0
root@ubuntu:~#

To exit the serial console, press CTRL+], then type quit:

root@ubuntu:~#
telnet> quit
Connection closed.
root@adf6e0bf4e6e:/app#

(Even) More Implementation Details​

The architecture of the system can be summarized in the following diagram:

os-cloud

The os-cloud container is the core of the entire system. +It consists of a web application written in python using flask. +This web application exposes a virtual machine API that the user can interact with (like vm_create).

So, when we're calling curl like in the example above:

curl -H "Content-Type: application/json" \
    -d '{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "2G", "disk_size": "10G"}' \
    localhost:5000/vm_create

It will do an HTTP POST request (because of the -d parameter) to /vm_create. +The request will be handled by the api_vm_create function in app.py (because of the @app.route("/vm_create", methods=["POST"]) line).

Inside this function, we also have access to the request payload (the string that comes after -d in our curl call). +More specifically, request.json will parse this payload as a JSON object and hand it back to us as a python dictionary. +In this dictionary we'll find the parameters for our request, like name, image, network, and so on.

The function will then take the actions required to create the virtual machine: create the disk, start qemu, interact with the database, etc. +Finally, whatever is returned by the api_vm_create function will be received by the curl request as the HTTP response. +Here we also return JSON objects, like {"id":1,"status":"ok"}.

There are 3 objects used by the system:

  • vm - the actual virtual machine

  • disk - holds information about virtual machine disks

  • network - holds information about a network

Each of these objects are stored in a table in the database.

Let's check the database contents (take the password from the setup_db.sh file):

student@os:~/.../support/os-cloud$ docker-compose exec db mysql -u os-cloud -p os-cloud
Enter password:
...
MariaDB [os-cloud]> select * from vm;
+----+-------+---------+------------+------------+-------------------+------------+----------+-------------------+------------------+-------+
| id | name  | disk_id | mem_size   | network_id | tap_interface_idx | ip         | qemu_pid | qemu_monitor_port | qemu_serial_port | state |
+----+-------+---------+------------+------------+-------------------+------------+----------+-------------------+------------------+-------+
|  1 | my_vm |       1 | 2147483648 |          1 |                 0 | 3232235522 |       18 |             10001 |            10002 |     0 |
+----+-------+---------+------------+------------+-------------------+------------+----------+-------------------+------------------+-------+
1 row in set (0.001 sec)

MariaDB [os-cloud]> select * from disk;
+----+-------------+---------------+
| id | size        | template_name |
+----+-------------+---------------+
|  1 | 10737418240 | ubuntu_22.04  |
+----+-------------+---------------+
1 row in set (0.000 sec)

MariaDB [os-cloud]> select * from network;
+----+---------+----------------------+------------+------------+
| id | name    | bridge_interface_idx | ip         | mask       |
+----+---------+----------------------+------------+------------+
|  1 | default |                    0 | 3232235520 | 4294901760 |
+----+---------+----------------------+------------+------------+
1 row in set (0.000 sec)

Note: in real life, DON'T store passwords in text files inside a repository.

Some observations:

  • There is a default network already created. +That is why we specified "network": "default" in the vm creation parameters, and we see that the vm is assigned to this network (network_id is 1).

  • This network's ip address is 3232235520, which in hex is 0xC0A80000, that is, 192.168.0.0. +The netmask is 0xFFFF0000, or /16. +This explains why our vm received the ip address 192.168.0.2.

  • There is a disk with the size of 10GB, based on the ubuntu_22.04 template, exactly like we requested. +This disk is assigned to our vm (disk_id is 1). +The disk file will reside in support/os-cloud/vm-disks/1/disk.qcow2, or /vm-disks/1/disk.qcow2 inside the container.

Virtual Machine Creation​

Take a look at the vm_create function in support/os-cloud/os-cloud/vm.py. +The steps undertaken are roughly:

  1. some initial allocations: the virtual machine IP address, network interface, qemu ports, etc

  2. the virtual machine disk is created, based on the template specified by the user (like ubuntu_22.04)

  3. the virtual machine is started with this new disk, in order to do some more customizations (the ubuntu_22_04_vm_prepare function)

  4. the virtual machine is restarted again with the final disk in place

Disk Creation​

All the disk templates are in support/os-cloud/disk-templates. +This directory will be mounted in /disk-templates inside the container.

The first step of disk creation is to create a qcow2 disk file based on the template specified by the user (step 2 from the explanation above).

This is done in the create_disk_from_template function in support/os-cloud/os-cloud/disk.py. +The function will first create a disk object in the database, then it will call 2 shell scripts: create_disk_from_template.sh and setup_root_password.sh.

The second step is to start the virtual machine with this disk and do some customizations (step 3 from above).

This is done in the ubuntu_22_04_vm_prepare function in support/os-cloud/os-cloud/vm.py. +The code will connect to the vm's qemu serial console using pexpect. +Then it will use a series of expect_exact + sendline pairs to interact with the virtual machine, as if those commands were typed in the command-line.

Practice: Create a New Disk by Hand​

Let's replicate the above-mentioned steps and create a new disk ourselves.

First, we have to call the 2 scripts from the create_disk_from_template function:

student@os:~/.../support/os-cloud$ ./disk-templates/ubuntu_22.04/create_disk_from_template.sh ./disk-templates/ubuntu_22.04/ubuntu_22.04.qcow2 my-disk.qcow2 10737418240
Image resized.

student@os:~/.../support/os-cloud$ ls -lh my-disk.qcow2
-rw-r--r-- 1 student student 619M Nov 20 15:41 my-disk.qcow2

student@os:~/.../support/os-cloud$ sudo ./disk-templates/ubuntu_22.04/setup_root_password.sh my-disk.qcow2 123456

Now we can start a qemu instance using this disk:

student@os:~/.../support/os-cloud$ qemu-system-x86_64 -enable-kvm -m 2G -hda my-disk.qcow2 -nographic
...
Ubuntu 22.04 LTS ubuntu ttyS0

ubuntu login: root
Password:
...
root@ubuntu:~#

Here we can further run customization commands, like the ones in the ubuntu_22_04_vm_prepare function, or any other things that we want.

When we're done, we run the halt command:

root@ubuntu:~# halt
root@ubuntu:~#          Stopping Session 1 of User root...
[  OK  ] Removed slice Slice /system/modprobe.
[  OK  ] Stopped target Graphical Interface.
...
         Starting System Halt...
[   86.431398] reboot: System halted

When the System halted message is printed, press CTRL+A X to exit qemu (that is, press CTRL+A, release CTRL and A, press X).

Practice: Implement vm_stop​

The vm_stop command will stop a particular virtual machine, meaning it will stop the qemu process for that vm. +The implementation starts in api_vm_stop in app.py, which is the function that handles the http request for the stop operation. +Here you need to do the following:

  • extract the virtual machine id from the request

  • use the vm.vm_get function to convert this ID into a VM structure

  • call vm.vm_stop and pass the VM object to it

In vm.vm_stop:

  • call stop_qemu_for_vm

  • change the vm pid in the database to -1

  • change the vm state in the database to VM_STATE_STOPPED

After modifying the code, you should run docker-compose up --build again. +Also, if your database became inconsistent, you can clean it up by re-running the setup_db.sh script. +Then delete all vm disks with sudo rm -rf vm-disks/*.

With vm_stop implemented, the system should work like this:

student@os:~/.../support/os-cloud$ curl -s localhost:5000/vm_list | jq .
[
  {
    "id": 1,
    "name": "my_vm"
  }
]
student@os:~/.../support/os-cloud$ curl -H "Content-Type: application/json" -d '{ "id": 1}' localhost:5000/vm_scurl -s -H "Content-Type: application/json" -d '{ "id": 1 }' localhost:5000/vm_info | jq .
{
  "disk_size": 10737418240,
  "id": 1,
  "ip": "192.168.0.2",
  "mem_size": 2147483648,
  "name": "my_vm",
  "network": "default",
  "os": "ubuntu_22.04",
  "state": "RUNNING"
}

The vm is in the RUNNING state. +Now let's stop it:

student@os:~/.../support/os-cloud$ curl -H "Content-Type: application/json" -d '{ "id": 1}' localhost:5000/vm_stop
{"status":"ok"}
student@os:~/.../support/os-cloud$ curl -s -H "Content-Type: application/json" -d '{ "id": 1 }' localhost:5000/vm_info | jq .
{
  "disk_size": 10737418240,
  "id": 1,
  "ip": "192.168.0.2",
  "mem_size": 2147483648,
  "name": "my_vm",
  "network": "default",
  "os": "ubuntu_22.04",
  "state": "STOPPED"
}

Now the state is STOPPED. +Inside the container, the qemu process should be gone as well:

student@os:~/.../support/os-cloud$ docker-compose exec os-cloud bash
root@b0600eff8903:/app# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 10:00 ?        00:00:00 /sbin/docker-init -- python3 -u app.py
root           7       1  0 10:00 ?        00:00:00 python3 -u app.py
root          33       0  0 10:00 pts/3    00:00:00 bash
root          41      33  0 10:00 pts/3    00:00:00 ps -ef

Finally, the vm can be started again using vm_start:

student@os:~/.../support/os-cloud$ curl -H "Content-Type: application/json" -d '{ "id": 1}' localhost:5000/vm_start
{"status":"ok"}
+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/overview/index.html b/17/Lab/Application Interaction/overview/index.html new file mode 100644 index 0000000000..a5226ee5e7 --- /dev/null +++ b/17/Lab/Application Interaction/overview/index.html @@ -0,0 +1,17 @@ + + + + + +Application Interaction | Operating Systems + + + + +
+
Skip to main content

Application Interaction

In this chapter, you will discover various mechanisms through which applications on a system can interact. +You will see these mechanisms in action in existing applications, but also use them in some code of your own.

Setup​

Make sure the following packages are installed:

sudo apt-get -y update; sudo apt-get -y install net-tools x11-apps libssl-dev d-feet firefox

Contents​

  1. Time Server
  2. Password Cracker
  3. The X Window System
  4. D-Bus
  5. OS Cloud
  6. Arena
+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/password-cracker/index.html b/17/Lab/Application Interaction/password-cracker/index.html new file mode 100644 index 0000000000..1da1ab0ce7 --- /dev/null +++ b/17/Lab/Application Interaction/password-cracker/index.html @@ -0,0 +1,31 @@ + + + + + +Password Cracker | Operating Systems + + + + +
+
Skip to main content

Password Cracker

In this example, we will solve the following problem: given the sha512 hash of a password, we want to obtain the password that generated the hash.

Since a hash function is not reversible, one way to solve this problem is by brute-force: generate all possible word combinations, compute the hash for each word, and compare it with our desired hash value. +This is not feasible for long passwords, so for our example we will consider only passwords containing lowercase letters and having the length of 4.

In order to speed up the entire process, we want to parallelize the solution. +Instead of one process checking all combinations, we'll split the work among multiple processes or threads.

Multiprocess Version​

The code for this version is in support/password-cracker/password-cracker-multiprocess.c.

The idea is the following: we create 26 worker processes, where each process will consider passwords that start with one particular letter (the first process will brute-force passwords starting with a, the second with b, and so on).

Since we are using processes, which are naturally isolated, we need a method of communication. +The main process should be able to send data to the workers and read back results from them. +For this purpose we will use pipes: a pair of 2 pipes between the main process and each worker, one pipe for each direction of communication.

In summary, the flow will look like this:

  • main process

    • create worker processes, along with 2 pipes for each worker (one pipe for requests, one for results)

    • send the 'a' character to the first process request pipe, 'b' to the second, etc.

    • read the results from each result pipe

  • worker process

    • read one character from the request pipe

    • generate all words of length 4 that begin with that character

    • for each generated word, compute the sha512 hash and compare it with the desired hash

    • if there is a match, write it to the result pipe

Let's build and run the program:

student@os:~/.../support/password-cracker$ make
gcc -Wall -o password-cracker-multiprocess password-cracker-multiprocess.c -lcrypto
gcc -Wall -Wno-int-to-pointer-cast -Wno-pointer-to-int-cast -o password-cracker-multithread password-cracker-multithread.c -lcrypto -lpthread

student@os:~/.../support/password-cracker$ ./password-cracker-multiprocess
worker 7 found haxx

Practice​

Creating 26 processes is not very realistic, since it's unlikely that a usual machine has that many cores.

Modify the program so that it only creates 4 workers. +Each worker will receive 2 characters instead of one, defining an interval to search. +For example, the first worker will receive a and f, meaning it will brute-force passwords starting with a, b, c, d, e, or f, the second g - l, and so on.

Multithreaded Version​

Check out the code in support/password-cracker/password-cracker-multithread.c.

The core idea of the program is the same, but now we're using threads instead of processes.

This makes the communication easier: we'll use the thread function argument to send the first character of the password to each thread. +As for the result, each thread will return it as the return value of the thread function.

student@os:~/.../support/password-cracker$ ./password-cracker-multithread
worker 7 found haxx

Multiprocess Version in Python (1)​

Code in support/password-cracker/python/password-cracker-multiprocess-1.py.

This is the Python equivalent of the previous multiprocess version. The program structure is the same, but Python has a few nice features that make our life easier:

  • there is a Process object that takes a function argument and spawns a new process that begins execution from that function. +No need to call fork manually.

  • the Pipe object in Python is already bidirectional, unlike the OS pipes, which are unidirectional. +So we don't need to create 2 pipes for each direction.

  • we don't have to write the code that generates all the password combinations, itertools.product will do it for us

student@os:~/.../support/password-cracker$ python3 python/password-cracker-multiprocess-1.py
worker 7 found haxx

Multiprocess Version in Python (2)​

Code in support/password-cracker/python/password-cracker-multiprocess-2.py.

In this case, the code looks different than in the previous examples. +Now we are taking advantage of some Python constructs, namely process pools, which are a collection of worker processes.

A Pool object has, among others, a function called map. +map takes a function, together with an array of values, and applies this function on each value from the array. +At first glance, it might look like the usual map function, but with the key difference that the function application is done by the processes from the pool.

In other words, the work is distributed to the worker processes from the pool, and all the communication that we had to handle in the previous examples is done behind the scenes, greatly simplifying the code.

student@os:~/.../support/password-cracker$ python3 python/password-cracker-multiprocess-2.py
worker 7 found haxx

Practice​

Check that the worker function is indeed called from different worker processes. +One simple way to do this is to print out the current process ID at the beginning of the function. +To get the current process ID, use the getpid function from the os module.

Multithreaded Version in Python​

Code in support/password-cracker/python/password-cracker-multithread.py.

The Python equivalent of the previous multithreaded version.

student@os:~/.../support/password-cracker$ python3 python/password-cracker-multithread.py
worker 7 found haxx

This example is given only to provide an idea of how a multithreaded program is written. +Remember that CPU-bound threads in python don't actually run in parallel, due to the Global Interpreter Lock.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/quiz/cgroups-vs-namespaces/index.html b/17/Lab/Application Interaction/quiz/cgroups-vs-namespaces/index.html new file mode 100644 index 0000000000..b72917877d --- /dev/null +++ b/17/Lab/Application Interaction/quiz/cgroups-vs-namespaces/index.html @@ -0,0 +1,17 @@ + + + + + +Cgroups versus namespaces | Operating Systems + + + + +
+
Skip to main content

Cgroups versus namespaces

Question Text​

Which of the following affirmations about namespaces and cgroups is correct?

Question Answers​

  • Cgroups provide resource management and namespaces provide isolation and security.

  • Namespaces provide resource management and cgroups provide isolation and security.

  • Both provide resource management.

  • Both provide isolation and security.

Feedback​

They both serve different purposes, cgroups provide resource management and namespaces provide isolation and security. +Cgroups manage resources, while namespaces isolate and secure them.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/quiz/container-vs-vm/index.html b/17/Lab/Application Interaction/quiz/container-vs-vm/index.html new file mode 100644 index 0000000000..68206a7e58 --- /dev/null +++ b/17/Lab/Application Interaction/quiz/container-vs-vm/index.html @@ -0,0 +1,16 @@ + + + + + +Container versus VM | Operating Systems + + + + +
+
Skip to main content

Container versus VM

Question Text​

What is one advantage of using containers over VMs in regard to starting times and memory consumption?

Question Answers​

  • VMs can start up faster and use less memory than containers.
  • Containers can start up faster and use less memory than VMs.
  • The comparison cannot be made.

Feedback​

This means that they impose less resource overhead than virtual machines, which need to run a complete guest operating system on top of the host operating system using a hypervisor.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/quiz/time-server-interop/index.html b/17/Lab/Application Interaction/quiz/time-server-interop/index.html new file mode 100644 index 0000000000..066f2e62f3 --- /dev/null +++ b/17/Lab/Application Interaction/quiz/time-server-interop/index.html @@ -0,0 +1,18 @@ + + + + + +Time Server Interoperability | Operating Systems + + + + +
+
Skip to main content

Time Server Interoperability

Question Text​

Is the protocol between the python server and the python client the same? +Can we run the python server and connect to it via the C client?

Question Answers​

  • Yes
  • No, the protocols are different

Feedback​

By doing the same investigation on the python server we discover that the protocol is the same. +This means that we can run the python server and the C client, or the C server and python client, and everything will work:

student@os:~/.../support/time-server/python$ python3 server.py
student@os:~/.../support/time-server$ ./client 127.0.0.1 2000
The time is Thu Sep  1 11:48:03 2022
+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/quiz/time-server/index.html b/17/Lab/Application Interaction/quiz/time-server/index.html new file mode 100644 index 0000000000..50acd483b0 --- /dev/null +++ b/17/Lab/Application Interaction/quiz/time-server/index.html @@ -0,0 +1,16 @@ + + + + + +Time Server Protocol | Operating Systems + + + + +
+
Skip to main content

Time Server Protocol

Question Text​

What format does the message exchanged between the server and the client have?

Question Answers​

  • 4 byte length (little endian) followed by 8 byte timestamp (little endian)

  • 8 byte length (little endian) followed by 4 byte timestamp (little endian)

  • 4 byte length (big endian) followed by 8 byte timestamp (big endian)
  • 8 byte length (big endian) followed by 4 byte timestamp (big endian)

Feedback​

If we consider one output example:

00000000  00 00 00 08 00 00 00 00  63 1b a9 50              |........c..P|
0000000c

We can identify the 4 byte length in big endian (00 00 00 08), then the 8 byte timestamp (00 00 00 00 63 1b a9 50), also in big endian.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/quiz/timer/index.html b/17/Lab/Application Interaction/quiz/timer/index.html new file mode 100644 index 0000000000..bde3fd6fac --- /dev/null +++ b/17/Lab/Application Interaction/quiz/timer/index.html @@ -0,0 +1,16 @@ + + + + + +Oneko Timer | Operating Systems + + + + +
+
Skip to main content

Oneko Timer

Question Text​

What is the purpose of the timer used in the Oneko's implementantion

Question Answers​

  • Slow down the cat

  • Speed up the cat

  • It server no purpose

Feedback​

The purpose of the timer is to delay the cat. You can verify this by running

while true; do kill -14 $(pidof oneko)

You'll notice the cat is much faster in following your mouse.

Try and find other ways to "hack" Oneko to make it move faster.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/quiz/vm-creation/index.html b/17/Lab/Application Interaction/quiz/vm-creation/index.html new file mode 100644 index 0000000000..6b25123b16 --- /dev/null +++ b/17/Lab/Application Interaction/quiz/vm-creation/index.html @@ -0,0 +1,16 @@ + + + + + +VM Creation | Operating Systems + + + + +
+
Skip to main content

VM Creation

Question Text​

How do you create a new virtual machine with a memory of 3GB and disk size of 100 GB?

Question Answers​

  • By running curl -H "Content-Type: application/json" -d '{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "100G"}' localhost:5000/vm_create

  • By running curl -H "Content-Type: application/json" -d '{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "100G"}' localhost:5000/vm_delete

  • By running curl -H "Content-Type: application/json" -d '{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "101G"}' localhost:5000/vm_create

  • By running curl -H "Content-Type: application/json" -d '{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "6G", "disk_size": "1000G"}' localhost:5000/vm_delete

Feedback​

We need to uso curl with the right path localhost:5000/vm_create, specifying the right data { "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "100G"}.

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/time-server/index.html b/17/Lab/Application Interaction/time-server/index.html new file mode 100644 index 0000000000..c42aef1b8f --- /dev/null +++ b/17/Lab/Application Interaction/time-server/index.html @@ -0,0 +1,20 @@ + + + + + +Time Server | Operating Systems + + + + +
+
Skip to main content

Time Server

Check out the code in support/time-server/server.c and support/time-server/client.c.

This is a simple program consisting of a server and a client. +The server uses a tcp socket to wait for connections. +Once a client has connected, the server will send the current time to it. +The client will then print the received time to the console.

Let's build and run this example:

student@os:~/.../support/time-server$ make
gcc -Wall -o server server.c
gcc -Wall -o client client.c
student@os:~/.../support/time-server$ ./server

Then, in another terminal:

student@os:~/.../support/time-server$ ./client 127.0.0.1 2000
The time is Thu Sep  1 11:48:03 2022

Python Version​

In support/time-server/python we have the equivalent python implementation for both the server and client:

student@os:~/.../support/time-server/python$ python3 server.py
student@os:~/.../support/time-server/python$ python3 client.py 127.0.0.1 2000
The time is Thu Sep  1 11:58:01 2022

Practice​

Try to figure out the protocol used by the server and the client. +You can do this by reading the source code, corroborated with information obtained at runtime.

Run the server again (the version in C), but instead of running the client, let's run netcat and pipe the output to hexdump:

nc -d 127.0.0.1 2000 | hexdump -C

Quiz

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Application Interaction/x-window-system/index.html b/17/Lab/Application Interaction/x-window-system/index.html new file mode 100644 index 0000000000..ce9b4c73b7 --- /dev/null +++ b/17/Lab/Application Interaction/x-window-system/index.html @@ -0,0 +1,23 @@ + + + + + +The X Window System | Operating Systems + + + + +
+
Skip to main content

The X Window System

Unix-like systems that support a Graphical User Interface usually do this through the X Window System. +This system is implemented with a client-server model: the X Server is the component that controls the screen, keyboard, mouse, and other parts related to the GUI, while the X clients are the applications that want to use the graphical interface (like, for example, an internet browser).

The clients and the server communicate using a standardized protocol, and the system does not necessarily require the client and server to be on the same machine. +Although not so common nowadays, the X client can run on a different machine than the server, with the communication happening over the network. +But in the more usual case, when both the client and the server are on the same machine, modern implementations of the X Window System use a faster communication channel, like a Unix socket.

X Client and Server on the Same Machine​

Let's investigate the case when both the X client and X server run on the same machine. +First we'll take a look at the Unix sockets that are in listening mode.

student@os:~$ sudo netstat -xnlp | grep X11
unix  2      [ ACC ]     STREAM     LISTENING     29120    3472/Xorg            @/tmp/.X11-unix/X0
unix  2      [ ACC ]     STREAM     LISTENING     29121    3472/Xorg            /tmp/.X11-unix/X0

We observe the Xorg process (the X server) listening on a Unix socket with the path /tmp/.X11-unix/X0.

Now let's run an X client (that is, a GUI application) and check that it will indeed connect to this Unix socket. +A very simple example is the xeyes application:

student@os:~$ strace -e trace=socket,connect xeyes
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3
connect(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X0"}, 20) = 0

As expected, the application creates a Unix socket, then connects to the path @"/tmp/.X11-unix/X0".

Furthermore, let's confirm that there is actual communication taking place between xeyes and the X server. +We'll run xeyes again, and then we'll keep moving the mouse cursor around. +When the mouse is moved, the following events are taking place:

  • The X server captures the mouse movements (since the server is the one that controls the mouse)
  • The X server will pass these "mouse moved" events to the clients (including xeyes)
  • The client (xeyes) uses these events to update its window (changing the position of the pupils inside the eyes)

So, if we run xeyes under strace, we expect to see some communication on the Unix socket that is created at the beginning:

strace -e 'trace=!poll' -e trace='socket,connect,recvmsg' xeyes |& grep -v '\-1 EAGAIN'

strace-xeyes

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/arena/index.html b/17/Lab/Compute/arena/index.html new file mode 100644 index 0000000000..80fd407ec6 --- /dev/null +++ b/17/Lab/Compute/arena/index.html @@ -0,0 +1,136 @@ + + + + + +Arena | Operating Systems + + + + +
+
Skip to main content

Arena

Threads and Processes: clone​

Let's go back to our initial demos that used threads and processes. +We'll see that in order to create both threads and processes, the underlying Linux syscall is clone. +For this, we'll run both sum_array_threads and sum_array_processes under strace. +As we've already established, we're only interested in the clone syscall:

student@os:~/.../lab/support/sum-array/c$ strace -e clone ./sum_array_threads 2
clone(child_stack=0x7f60b56482b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819693], tls=0x7f60b5649640, child_tidptr=0x7f60b5649910) = 1819693
clone(child_stack=0x7f60b4e472b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819694], tls=0x7f60b4e48640, child_tidptr=0x7f60b4e48910) = 1819694

student@os:~/.../lab/support/sum-array/c$ strace -e clone ./sum_array_processes 2
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820599
clone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820600

We ran each program with an argument of 2, so we have 2 calls to clone. +Notice that in the case of threads, the clone syscall receives more arguments. +The relevant flags passed as arguments when creating threads are documented in clone's man page:

  • CLONE_VM: the child and the parent process share the same VAS
  • CLONE_{FS,FILES,SIGHAND}: the new thread shares the filesystem information, file and signal handlers with the one that created it +The syscall also receives valid pointers to the new thread's stack and TLS, i.e. the only parts of the VAS that are distinct between threads (although they are technically accessible from all threads).

By contrast, when creating a new process, the arguments of the clone syscall are simpler (i.e. fewer flags are present). +Remember that in both cases clone creates a new thread. +When creating a process, clone creates this new thread within a new separate address space.

Libraries for Parallel Processing​

In support/sum-array/c/sum_array_threads.c we spawned threads "manually" by using the pthread_create() function. +This is not a syscall, but a wrapper over the common syscall used by both fork() (which is also not a syscall) and pthread_create().

Still, pthread_create() is not yet a syscall. +In order to see what syscall pthread_create() uses, check out this section at the end of the lab.

Most programming languages provide a more advanced API for handling parallel computation.

std.parallelism in D​

D language's standard library exposes the std.parallelism, which provides a series of parallel processing functions. +One such function is reduce(), which splits an array between a given number of threads and applies a given operation to these chunks. +In our case, the operation simply adds the elements to an accumulator: a + b. +Follow and run the code in support/sum-array/d/sum_array_threads_reduce.d.

The number of threads is used within a TaskPool. +This structure is a thread manager (not scheduler). +It silently creates the number of threads we request and then reduce() spreads its workload between these threads.

OpenMP for C​

Unlike D, C does not support parallel computation by design. +It needs a library to do advanced things, like reduce() from D. +We have chosen to use the OpenMP library for this. +Follow the code in support/sum-array/c/sum_array_threads_openmp.c.

The #pragma used in the code instructs the compiler to enable the omp module, and to parallelise the code. +In this case, we instruct the compiler to perform a reduce of the array, using the + operator, and to store the results in the result variable. +This reduction uses threads to calculate the sum, similar to summ_array_threads.c, but in a much more optimised form.

Now compile and run the sum_array_threads_openmp binary using 1, 2, 4, and 8 threads as before. +You'll see lower running times than sum_array_threads due to the highly-optimised code emitted by the compiler. +For this reason and because library functions are usually much better tested than your own code, it is always preferred to use a library function for a given task.

Shared Memory​

As you remember from the Data chapter, one way to allocate a given number of pages is to use the mmap() syscall. +Let's look at its man page, specifically at the flags argument. +Its main purpose is to determine the way in which child processes interact with the mapped pages.

Quiz

Now let's test this flag, as well as its opposite: MAP_SHARED. +Compile and run the code in support/shared-memory/shared_memory.c.

  1. See the value read by the parent is different from that written by the child. +Modify the flags parameter of mmap() so they are the same.

  2. Create a semaphore in the shared page and use it to make the parent signal the child before it can exit. +Use the API defined in semaphore.h. +Be careful! +The value written and read previously by the child and the parent, respectively, must not change.

One way of creating a shared semaphore is to place it within a shared memory area, as we've just done. +This only works between "related" processes. +If you want to share a semaphore or other types of memory between any two processes, you need filesystem support. +For this, you should use named semaphores, created using sem_open(). +You'll get more accustomed to such functions in the Application Interaction chapter.

Mini-shell​

First Step: system Dissected​

You already know that system calls fork() and execve() to create the new process. +Let's see how and why. +First, we run the following command to trace the execve() syscalls used by sleepy_creator. +We'll leave fork() for later.

student@os:~/.../support/sleepy$ strace -e execve -ff -o syscalls ./sleepy_creator

At this point, you will get two files whose names start with syscalls, followed by some numbers. +Those numbers are the PIDs of the parent and the child process. +Therefore, the file with the higher number contains logs of the execve and clone syscalls issued by the parent process, while +the other logs those two syscalls when made by the child process. +Let's take a look at them. +The numbers below will differ from those on your system:

student@os:~/.../support/sleepy:$ cat syscalls.2523393  # syscalls from parent process
execve("sleepy_creator", ["sleepy_creator"], 0x7ffd2c157758 /* 39 vars */) = 0
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2523394, si_uid=1052093, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++

student@os:~/.../support/sleepy:$ cat syscalls.2523394  # syscalls from child process
execve("/bin/sh", ["sh", "-c", "sleep 10"], 0x7ffd36253be8 /* 39 vars */) = 0
execve("/usr/bin/sleep", ["sleep", "10"], 0x560f41659d40 /* 38 vars */) = 0
+++ exited with 0 +++

Quiz

Now notice that the child process doesn't simply call execve("/usr/bin/sleep" ...). +It first changes its virtual address space (VAS) to that of a bash process (execve("/bin/sh" ...)) and then that bash process switches its VAS to sleep. +Therefore, calling system(<some_command>) is equivalent to running <some_command> in the command-line.

With this knowledge in mind, let's implement our own mini-shell. +Start from the skeleton code in support/mini-shell/mini_shell.c. +We're already running our Bash interpreter from the command-line, so there's no need to exec another Bash from it. +Simply exec the command.

Quiz

So we need a way to "save" the mini_shell process before exec()-ing our command. +Find a way to do this.

Hint: You can see what sleepy does and draw inspiration from there. +Use strace to also list the calls to clone() performed by sleepy or its children. +Remember what clone() is used for and use its parameters to deduce which of the two scenarios happens to sleepy.

Moral of the story: We can add another step to the moral of our previous story. +When spawning a new command, the call order is:

  • parent: fork(), exec(), wait()
  • child: exit()

Command Executor in Another language​

Now implement the same functionality (a Bash command executor) in any other language, other than C/C++. +Use whatever API is provided by your language of choice for creating and waiting for processes.

The GIL​

Throughout this lab, you might have noticed that there were no thread exercises in Python. +If you did, you probably wondered why. +It's not because Python does not support threads, because it does, but because of a mechanism called the Global Interpreter Lock, or GIL. +As its name suggests, this is a lock implemented inside most commonly used Python interpreter (CPython), which only allows one thread to run at a given time. +As a consequence, multithreaded programs written in Python run concurrently, not in parallel. +For this reason, you will see no speedup even when you run an embarrassingly parallel code in parallel.

However, keep in mind that this drawback does not make threads useless in Python. +They are still useful and widely used when a process needs to perform many IO-bound tasks (i.e.: tasks that involve many file reads / writes or network requests). +Such tasks run many blocking syscalls that require the thread to switch from the RUNNING state to WAITING. +Doing so voluntarily makes threads viable because they rarely run for their entire time slice and spend most of the time waiting for data. +So it doesn't hurt them to run concurrently, instead of in parallel.

Do not make the confusion to believe threads in Python are user-level threads. +threading.Threads are kernel-level threads. +It's just that they are forced to run concurrently by the GIL.

Practice: Array Sum in Python​

Let's first probe this by implementing two parallel versions of the code in support/sum-array/python/sum_array_sequential.py. +One version should use threads and the other should use processes. +Run each of them using 1, 2, 4, and 8 threads / processes respectively and compare the running times. +Notice that the running times of the multithreaded implementation do not decrease. +This is because the GIL makes it so that those threads that you create essentially run sequentially.

The GIL also makes it so that individual Python instructions are atomic. +Run the code in support/race-condition/python/race_condition.py. +Every time, var will be 0 because the GIL doesn't allow the two threads to run in parallel and reach the critical section at the same time. +This means that the instructions var += 1 and var -= 1 become atomic.

But Why?​

Unlike Bigfoot, or the Loch Ness monster, we have proof that the GIL is real. +At first glance, this seems like a huge disadvantage. +Why force threads to run sequentially? +The answer has to do with memory management. +In the Data chapter, you learned that one way of managing memory is via garbage collection (GC). +In Python, the GC uses reference counting, i.e. each object also stores the number of live pointers to it (variables that reference it). +You can see that this number needs to be modified atomically by the interpreter to avoid race conditions. +This involves adding locks to all Python data structures. +This large number of locks doesn't scale for a language as large and open as Python. +In addition, it also introduces the risk of deadlocks. +You can read more on this topic in this article and if you think eliminating the GIL looks like an easy task, which should have been done long ago, check the requirements here. +They're not trivial to meet.

Single-threaded-ness is a common trope for interpreted languages to use some sort of GIL. +Ruby MRI, the reference Ruby interpreter, uses a similar mechanism, called the Global VM Lock. +JavaScript is even more straightforward: it is single-threaded by design, also for GC-related reasons. +It does, however, support asynchronous actions, but these are executed on the same thread as every other code. +This is implemented by placing each instruction on a call stack.

Atomic Assembly​

No, this section is not about nukes, sadly :(. +Instead, we aim to get accustomed to the way in which the x86 ISA provides atomic instructions.

This mechanism looks very simple. +It is but one instruction prefix: lock. +It is not an instruction with its own separate opcode, but a prefix that slightly modifies the opcode of the following instructions to make the CPU execute it atomically (i.e. with exclusive access to the data bus).

lock must only be place before an instruction that executes a read-modify-write action. +For example, we cannot place it before a mov instruction, as the action of a mov is simply read or write. +Instead, we can place it in front of an inc instruction if its operand is memory.

Look at the code in support/race-condition/asm/race_condition_lock.S. +It's an Assembly equivalent of the code you've already seen many times so far (such as support/race-condition/c/race_condition.c). +Assemble and run it a few times. +Notice the different results you get.

Now add the lock prefix before inc and dec. +Reassemble and rerun the code. +And now we have synchronized the two threads by leveraging CPU support.

Synchronization - Thread-Safe Data Structure​

Now it's time for a fully practical exercise. +Go to support/CLIST/. +In the file clist.c you'll find a simple implementation of an array list. +Although correct, it is not (yet) thread-safe.

The code in test.c verifies its single-threaded correctness, while the one in test_parallel.c verifies it works properly with multiple threads. +Your task is to synchronize this data structure using whichever primitives you like. +Try to keep the implementation efficient. +Aim to decrease your running times as much as you can.

Minor and Major Page Faults​

The code in support/page-faults/page_faults.c generates some minor and major page faults. +Open 2 terminals: one in which you will run the program, and one which will monitor the page faults of the program. +In the monitoring terminal, run the following command:

watch -n 1 'ps -eo min_flt,maj_flt,cmd | grep ./page_faults | head -n 1'

Compile the program and run it in the other terminal. +You must press enter one time, before the program will prompt you to press enter more times. +Watch the first number on the monitoring terminal; +it increases. +Those are the minor page faults.

Minor Page Faults​

A minor page fault is generated whenever a requested page is present in the physical memory, as a frame, but that frame isn't allocated to the process generating the request. +These types of page faults are the most common, and they happen when calling functions from dynamic libraries, allocating heap memory, loading programs, reading files that have been cached, and many more situations. +Now back to the program.

The monitoring command already starts with some minor page faults, generated when loading the program.

After pressing enter, the number increases, because a function from a dynamic library (libc) is fetched when the first printf() is executed. +Subsequent calls to functions that are in the same memory page as printf() won't generate other page faults.

After allocating the 100 Bytes, you might not see the number of page faults increase. +This is because the "bookkeeping" data allocated by malloc() was able to fit in an already mapped page. +The second allocation, the 1GB one, will always gnereate one minor page fault - for the bookkeeping data about the allocated memory zone. +Notice that not all the pages for the 1GB are allocated. +They are allocated - and generate page faults - when modified. +By now you should know that this mechanism is called copy-on-write.

Continue with pressing enter and observing the effects util you reach opening file.txt.

Note that neither opening a file, getting information about it, nor mapping it in memory using mmap(), generate page faults. +Also note the posix_fadvise() call after the one to fstat(). +With this call we force the OS to not cache the file, so we can generate a major page fault.

Major Page Faults​

Major page faults happen when a page is requested, but it isn't present in the physical memory. +These types of page faults happen in 2 situations:

  • a page that was swapped out (to the disk), due to lack of memory, is now accessed - this case is harder to show
  • the OS needs to read a file from the disk, because the file contents aren't present in the cache - the case we are showing now

Press enter to print the file contents. +Note the second number go up in the monitoring terminal.

Comment the posix_fadvise() call, recompile the program, and run it again. +You won't get any major page fault, because the file contents are cached by the OS, to avoid those page faults. +As a rule, the OS will avoid major page faults whenever possible, because they are very costly in terms of running time.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/copy-on-write/index.html b/17/Lab/Compute/copy-on-write/index.html new file mode 100644 index 0000000000..be9a0a3348 --- /dev/null +++ b/17/Lab/Compute/copy-on-write/index.html @@ -0,0 +1,39 @@ + + + + + +Copy-on-Write | Operating Systems + + + + +
+
Skip to main content

Copy-on-Write

So far, you know that the parent and child process have separate virtual address spaces. +But how are they created, namely how are they "separated"? +And what about the PAS (physical address space)? +Of course, we would like the stack of the parent, for example, to be physically distinct from that of the child, so they can execute different functions and use different local variables.

But should all memory sections from the PAS of the parent be distinct from that of the child? +What about some read-only memory sections, such as .text and .rodata? +And what about the heap, where the child may use some data previously written by the parent and then override it with its own data.

The answer to all of these questions is a core mechanism of multiprocess operating systems called Copy-on-Write. +It works according to one very simple principle:

The VAS of the child process initially points to the same PAS as that of the parent. +A (physical) frame is only duplicated by the child when it attempts to write data to it.

This ensures that read-only sections remain shared, while writable sections are shared as long as their contents remain unchanged. +When changes happen, the process making the change receives a unique frame as a modified copy of the original frame on demand.

In the image below, we have the state of the child and parent processes right after fork() returns in both of them. +See how each has its own VAS, both of them being mapped to (mostly) the same PAS.

Copy-on-Write

When one process writes data to a writeable page (in our case, the child writes to a heap page), the frame to which it corresponds is first duplicated. +Then the process' page table points the page to the newly copied frame, as you can see in the image below.

Copy-on-Write

Be careful! +Do not confuse copy-on-write with demand paging. +Remember from the Data chapter that demand paging means that when you allocate memory, the OS allocates virtual memory that remains unmapped to physical memory until it's used. +On the other hand, copy-on-write posits that the virtual memory is already mapped to some frames. +These frames are only duplicated when one of the processes attempts to write data to them.

Practice​

Now let's see the copy-on-write mechanism in practice. +Keep in mind that fork() is a function used to create a process.

Open two terminals (or better: use tmux). +In one of them, compile and run the code in support/fork-faults/fork_faults.c. +After each time you press Enter in the first terminal window, run the following command in the second window:

student@os:~/.../lab/support/fork-faults$ ps -o min_flt,maj_flt -p $(pidof fork_faults)

It will show you the number of minor and major page faults performed by the fork_faults process and its child.

To better understand minor and major page faults, go through the Minor and Major Page Faults excercise in Arena.

Quiz 1

Note that after fork()-ing, there is a second row in the output of ps. +That corresponds to the child process. +The first one still corresponds to the parent.

Quiz 2

Now it should be clear how demand paging differs from copy-on-write. +Shared memory is a similar concept. +It's a way of marking certain allocated pages so that copy-on-write is disabled. +As you may imagine, changes made by the parent to this memory are visible to the child and vice-versa. +You can learn more about it in its dedicated section in the Arena.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/hardware-perspective/index.html b/17/Lab/Compute/hardware-perspective/index.html new file mode 100644 index 0000000000..800606ce22 --- /dev/null +++ b/17/Lab/Compute/hardware-perspective/index.html @@ -0,0 +1,37 @@ + + + + + +Hardware Perspective | Operating Systems + + + + +
+
Skip to main content

Hardware Perspective

The main criterion we use to rank CPUs is their computation power, i.e. their ability to crunch numbers and do math. +Numerous benchmarks exist out there, and they are publicly displayed on sites such as CPUBenchmark.

For example, a benchmark can measure the performance of the computer's CPU in a variety of scenarios:

  • its ability to perform integer operations
  • its speed in floating point arithmetic
  • data encryption and compression
  • sorting algorithms and others

You can take a look at what exactly is measured using this link. +It displays the scores obtained by a high-end CPU. +Apart from the tests above, other benchmarks might focus on different performance metrics, such as branch prediction or prefetching.

Other approaches are less artificial, measuring performance on real-world applications such as compile times and performance in the latest (and most resource-demanding) video games. +The latter metric revolves around how many average FPS (frames per second) a given CPU is able to crank out in a specific video game. +This article goes into more detail regarding the methodology of running CPU benchmarks on real-world applications.

Most benchmarks, unfortunately, are not open source, especially the more popular ones, such as Geekbench 5. +Despite this shortcoming, benchmarks are widely used to compare the performance of various computer hardware, CPUs included.

The Role of the Operating System​

As you've seen so far, the CPU provides the "muscle" required for fast computation, i.e. the highly optimised hardware and multiple ALUs, FPUs +and cores necessary to perform those computations. +However, it is the operating system that provides the "brains" for this computation. +Specifically, modern CPUs have the capacity to run multiple tasks in parallel. +But they do not provide a means to decide which task to run at each moment. +The OS comes as an orchestrator to schedule the way these tasks (that we will later call threads) are allowed to run and use the CPU's resources. +This way, the OS tells the CPU what code to run on each CPU core so that it reaches a good balance between high throughput (running many instructions) and fair access to CPU cores.

It is cumbersome for a user-level application to interact directly with the CPU. +The developer would have to write hardware-specific code, which is not scalable and is difficult to maintain. +In addition, doing so would leave it up to the developer to isolate their application from the others that are present on the system. +This leaves applications vulnerable to countless bugs and exploits.

To guard apps from these pitfalls, the OS comes and mediates interactions between regular programs and the CPU by providing a set of abstractions. +These abstractions offer a safe, uniform and also isolated way to leverage the CPU's resources, i.e. its cores. +There are 2 main abstractions: processes and threads.

Interaction between applications, OS and CPU

As we can see from the image above, an application can spawn one or more processes. +Each of these is handled and maintained by the OS. +Similarly, each process can spawn however many threads, which are also managed by the OS. +The OS decides when and on what CPU core to make each thread run. +This is in line with the general interaction between an application and the hardware: it is always mediated by the OS.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/index.html b/17/Lab/Compute/index.html new file mode 100644 index 0000000000..a5509a6865 --- /dev/null +++ b/17/Lab/Compute/index.html @@ -0,0 +1,16 @@ + + + + + +Compute | Operating Systems + + + + +
+
Skip to main content

Compute

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/overview/index.html b/17/Lab/Compute/overview/index.html new file mode 100644 index 0000000000..5497a81a56 --- /dev/null +++ b/17/Lab/Compute/overview/index.html @@ -0,0 +1,16 @@ + + + + + +Compute | Operating Systems + + + + +
+
Skip to main content
+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/processes-threads-apache2/index.html b/17/Lab/Compute/processes-threads-apache2/index.html new file mode 100644 index 0000000000..0fd28ec0bd --- /dev/null +++ b/17/Lab/Compute/processes-threads-apache2/index.html @@ -0,0 +1,43 @@ + + + + + +Usage of Processes and Threads in `apache2` | Operating Systems + + + + +
+
Skip to main content

Usage of Processes and Threads in apache2

We'll take a look at how a real-world application - the apache2 HTTP server - makes use of processes and threads. +Since the server must be able to handle multiple clients at the same time, it must therefore use some form of concurrency. +When a new client arrives, the server offloads the work of interacting with that client to another process or thread.

The choice of whether to use multiple processes or threads is not baked into the code. +Instead, apache2 provides a couple of modules called MPMs (Multi-Processing Modules). +Each module implements a different concurrency model, and the users can pick whatever module best fits their needs by editing the server configuration files.

The most common MPMs are

  • prefork: there are multiple worker processes, each process is single-threaded and handles one client request at a time
  • worker: there are multiple worker processes, each process is multi-threaded, and each thread handles one client request at a time
  • event: same as worker but designed to better handle some particular use cases

In principle, prefork provides more stability and backwards compatibility, but it has a bigger overhead. +On the other hand, worker and event are more scalable, and thus able to handle more simultaneous connections, due to the usage of threads. +On modern systems, event is almost always the default.

apache2 Live Action​

Let's run an actual instance of apache2 and see how everything works. +Go to support/apache2 and run make run. +This will start a container with apache2 running inside.

Check that the server runs as expected:

student@os:~$ curl localhost:8080
<html><body><h1>It works!</h1></body></html>

Now go inside the container and take a look at running processes:

student@os:~/.../lab/support/apache2$ docker exec -it apache2-test bash

root@56b9a761d598:/usr/local/apache2# ps -ef
UID          PID    PPID  C STIME TTY          TIME CMD
root           1       0  0 20:38 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1  0 20:38 pts/0    00:00:00 httpd -DFOREGROUND
www-data      10       1  0 20:38 pts/0    00:00:00 httpd -DFOREGROUND
root          25       0  0 20:40 pts/1    00:00:00 bash
root          31      25  0 20:40 pts/1    00:00:00 ps -ef

We see 3 httpd processes. +The first one, running as root, is the main process, while the other 2 are the workers.

Let's confirm that we are using the event mpm:

root@56b9a761d598:/usr/local/apache2# grep mod_mpm conf/httpd.conf
LoadModule mpm_event_module modules/mod_mpm_event.so
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
LoadModule mpm_worker_module modules/mod_mpm_worker.so

The event mpm is enabled, so we expect each worker to be multithreaded. +Let's check:

root@56b9a761d598:/usr/local/apache2# ps -efL
UID          PID    PPID     LWP  C NLWP STIME TTY          TIME CMD
root           1       0       1  0    1 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1       8  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1      11  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1      12  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1      16  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1      17  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1      18  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       8       1      19  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1       9  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1      14  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1      15  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1      20  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1      21  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1      22  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data       9       1      23  0    7 20:56 pts/0    00:00:00 httpd -DFOREGROUND
root          24       0      24  1    1 20:56 pts/1    00:00:00 bash
root          30      24      30  0    1 20:56 pts/1    00:00:00 ps -efL

Indeed, each worker has 7 threads. +In fact, the number of threads per worker is configurable, as well as the number of initial workers.

When a new connection is created, it will be handled by whatever thread is available from any worker. +If all the threads are busy, then the server will spawn more worker processes (and therefore more threads), as long as the total number of threads is below some threshold, which is also configurable.

Let's see this dynamic scaling in action. +We need to create a number of simultaneous connections that is larger than the current number of threads. +There is a simple script in support/apache2/make_conn.py to do this:

student@os:~/.../lab/support/apache2$ python3 make_conn.py localhost 8080
Press ENTER to exit

The script has created 100 connections and will keep them open until we press Enter.

Now, in another terminal, let's check the situation inside the container:

student@os:~/.../lab/support/apache2$ docker exec -it apache2-test bash

root@56b9a761d598:/usr/local/apache2# ps -efL
UID          PID    PPID     LWP  C NLWP STIME TTY          TIME CMD
root           1       0       1  0    1 20:56 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      40  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      45  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      46  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      51  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      52  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      53  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      40       1      54  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      55  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      58  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      60  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      62  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      63  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      65  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data      55       1      66  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
[...]
www-data     109       1     109  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data     109       1     115  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data     109       1     116  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data     109       1     121  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data     109       1     122  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data     109       1     123  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
www-data     109       1     124  0    7 21:07 pts/0    00:00:00 httpd -DFOREGROUND
root         146       0     146  0    1 21:10 pts/1    00:00:00 bash
root         152     146     152  0    1 21:10 pts/1    00:00:00 ps -efL

We see a much larger number of threads, as expected.

Practice: Investigate apache2 Using strace​

Use strace to discover the server document root. +The document root is the path in the filesystem from where httpd serves all the files requested by the clients.

First, you will have to stop the running container using make stop, then restart it with make run-privileged.

Then you will use strace inside the container to attach to the worker processes (use the -p option for this). +You will also have to use the -f flag with strace, so that it will follow all the threads inside the processes.

After you have attached successfully to all worker processes, use the curl command to send a request, like the one in the beginning of this section.

Then check the strace output to see what files were opened by the server.

Quiz

Conclusion​

So far, you've probably seen that spawning a process can "use" a different program (hence the path in the args of system or Popen), but some languages such as Python allow you to spawn a process that executes a function from the same script. +A thread, however, can only start from a certain entry point within the current address space, as it is bound to the same process. +Concretely, a process is but a group of threads. +For this reason, when we talk about scheduling or synchronization, we talk about threads. +A thread is, thus, an abstraction of a task running on a CPU core. +A process is a logical group of such tasks.

We can sum up what we've learned so far by saying that processes are better used for separate, independent work, such as the different connections handled by a server. +Conversely, threads are better suited for replicated work: when the same task has to be performed on multiple cores. +However, replicated work can also be suited for processes. +Distributed applications, however, leverage different processes as this allows them to run on multiple physical machines at once. +This is required by the very large workloads such applications are commonly required to process.

These rules are not set in stone, though. +Like we saw in the apache2 example, the server uses multiple threads as well as multiple processes. +This provides a degree of stability - if one worker thread crashes, it will only crash the other threads belonging to the same process - while still taking advantage of the light resource usage inherent to threads.

These kinds of trade-offs are a normal part of the development of real-world applications.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/processes/index.html b/17/Lab/Compute/processes/index.html new file mode 100644 index 0000000000..c8f14d1c27 --- /dev/null +++ b/17/Lab/Compute/processes/index.html @@ -0,0 +1,90 @@ + + + + + +Processes | Operating Systems + + + + +
+
Skip to main content

Processes

A process is simply a running program. +Let's take the ls command as a trivial example. +ls is a program on your system. +It has a binary file which you can find and inspect with the help of the which command:

student@os:~$ which ls
/usr/bin/ls

student@os:~$ file /usr/bin/ls
/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e3da6f0bc36b6398b8651bbc2e08831a21a90da, for GNU/Linux 3.2.0, stripped

When you run it, the ls binary stored on the disk at /usr/bin/ls is read by another application called the loader. +The loader spawns a process by copying some of the contents /usr/bin/ls in memory (such as the .text, .rodata and .data sections). +Using strace, we can see the execve system call:

student@os:~$ strace -s 100 ls -a  # -s 100 limits strings to 100 bytes instead of the default 32
execve("/usr/bin/ls", ["ls", "-a"], 0x7fffa7e0d008 /* 61 vars */) = 0
[...]
write(1, ".  ..  content\tCONTRIBUTING.md  COPYING.md  .git  .gitignore  README.md  REVIEWING.md\n", 86.  ..  content   CONTRIBUTING.md  COPYING.md  .git  .gitignore  README.md  REVIEWING.md
) = 86
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
+++ exited with 0 +++

Look at its parameters:

  • the path to the program: /usr/bin/ls
  • the list of arguments: "ls", "-a"
  • the environment variables: the rest of the syscall's arguments

execve invokes the loader to load the VAS of the ls process by replacing that of the existing process. +All subsequent syscalls are performed by the newly spawned ls process. +We will get into more details regarding execve towards the end of this lab.

Loading of `ls` Process

Sum of the Elements in an Array​

Let's assume we only have one process on our system, and that process knows how to add the numbers in an array. +It can use however many resources it wants, since there is no other process to contest it. +It would probably look like the code in support/sum-array/c/sum_array_sequential.c. +The program also measures the time spent computing the sum. +Let's compile and run it:

student@os:~/.../lab/support/sum-array/c$ ./sum_array_sequential
Array sum is: 49945994146
Time spent: 127 ms

You will most likely get a different sum (because the array is made up of random numbers) and a different time than the ones shown above. +This is perfectly fine. +Use these examples qualitatively, not quantitatively.

Spreading the Work Among Other Processes​

Due to how it's implemented so far, our program only uses one of our CPU's cores. +We never tell it to distribute its workload to other cores. +This is wasteful as the rest of our cores remain unused:

student@os:~$ lscpu | grep ^CPU\(s\):
CPU(s):                          8

We have 7 more cores waiting to add numbers in our array.

What if we used 100% of the CPU?

What if we use 7 more processes and spread the task of adding the numbers in this array between them? +If we split the array into several equal parts and designate a separate process to calculate the sum of each part, we should get a speedup because now the work performed by each individual process is reduced.

Let's take it methodically. +Compile and run sum_array_processes.c using 1, 2, 4 and 8 processes respectively. +If your system only has 4 cores (hyperthreading included), limit your runs to 4 processes. +Note the running times for each number of processes. +We expect the speedups compared to our reference run to be 1, 2, 4 and 8 respectively, right?

Quiz

You most likely did get some speedup, especially when using 8 processes. +Now we will try to improve this speedup by using threads instead.

Also notice that we're not using hundreds or thousands of processes. +Assuming our system has 8 cores, only 8 threads (we'll see this later in the lab) can run at the same time. +In general, the maximum number of threads that can run at the same time is equal to the number of cores. +In our example, each process only has one thread: its main thread. +So by consequence and by forcing the terminology (because it's the main thread of these processes that is running, not the processes themselves), we can only run in parallel a number of processes equal to at most the number of cores.

Practice: Baby steps - Python​

Run the code in support/create-process/popen.py. +It simply spawns a new process running the ls command using subprocess.Popen(). +Do not worry about the huge list of arguments that Popen() takes. +They are used for inter-process-communication. +You'll learn more about this in the Application Interaction chapter.

Note that this usage of Popen() is not entirely correct. +You'll discover why in the next exercise, but for now focus on simply understanding how to use Popen() on its own.

Now change the command to anything you want. +Also give it some arguments. +From the outside, it's as if you were running these commands from the terminal.

Practice: High level - Python​

Head over to support/sleepy/sleepy_creator.py. +Use subprocess.Popen() to spawn 10 sleep 1000 processes.

  1. Solve TODO 1: use subprocess.Popen() to spawn 10 sleep 1000 processes.

    Start the script:

    student@os:~/.../lab/support/sleepy$ python3 sleepy_creator.py

    Look for the parent process:

    student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep "python3 sleepy_creator.py")

    It is a python3 process, as this is the interpreter that runs the script, but we call it the sleepy_creator.py process for simplicity. +No output will be provided by the above command, as the parent process (sleepy_creator.py) dies before its child processes (the 10 sleep 1000 subprocesses) finish their execution. +The parent process of the newly created child processes is an init-like process: either systemd/init or another system process that adopts orphan processes. +Look for the sleep child processes using:

    student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)
     PID    PPID         CMD
    4164    1680     sleep 1000
    4165    1680     sleep 1000
    4166    1680     sleep 1000
    4167    1680     sleep 1000
    4168    1680     sleep 1000
    4169    1680     sleep 1000
    4170    1680     sleep 1000
    4171    1680     sleep 1000
    4172    1680     sleep 1000
    4173    1680     sleep 1000

    Notice that the child processes do not have sleepy_creator.py as a parent. +What's more, as you saw above, sleepy_creator.py doesn't even exist anymore. +The child processes have been adopted by an init-like process (in the output above, that process has PID 1680 - PPID stands for parent process ID).

    Quiz

  2. Solve TODO 2: change the code in sleepy_creator.py so that the sleep 1000 processes remain the children of sleepy_creator.py. +This means that the parent / creator process must not exit until its children have finished their execution. +In other words, the parent / creator process must wait for the termination of its children. +Check out Popen.wait() and add the code that makes the parent / creator process wait for its children. +Before anything, terminate the sleep processes created above:

    student@os:~$ pkill sleep

    Start the program, again, as you did before:

    student@os:~/.../lab/support/sleepy$ python3 sleepy_creator.py

    On another terminal, verify that sleepy_creator.py remains the parent of the sleep processes it creates:

    student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)
     PID    PPID                CMD
    16107   9855         python3 sleepy_creator.py
    16108   16107           sleep 1000
    16109   16107           sleep 1000
    16110   16107           sleep 1000
    16111   16107           sleep 1000
    16112   16107           sleep 1000
    16113   16107           sleep 1000
    16114   16107           sleep 1000
    16115   16107           sleep 1000
    16116   16107           sleep 1000
    16117   16107           sleep 1000

    Note that the parent process sleepy_creator.py (PID 16107) is still alive, and its child processes (the 10 sleep 1000) have its ID as their PPID. +You've successfully waited for the child processes to finish their execution.

Practice: Lower level - C​

Now let's see how to create a child process in C. +There are multiple ways of doing this. +For now, we'll start with a higher-level approach.

Go to support/sleepy/sleepy_creator.c and use system to create a sleep 1000 process.

Quiz

The man page also mentions that system calls fork() and exec() to run the command it's given. +If you want to find out more about them, head over to the Arena and create your own mini-shell.

Practice: Wait for Me​

Run the code in support/wait-for-me/wait_for_me_processes.py. +The parent process creates one child that writes and message to the given file. +Then the parent reads that message. +Simple enough, right? +But running the code raises a FileNotFoundError. +If you inspect the file you gave the script as an argument, it does contain a string. +What's going on?

Quiz

In order to solve race conditions, we need synchronization. +This is a mechanism similar to a set of traffic lights in a crossroads. +Just like traffic lights allow some cars to pass only after others have already passed, synchronization is a means for threads to communicate with each other and tell each other to access a resource or not.

The most basic form of synchronization is waiting. +Concretely, if the parent process waits for the child to end, we are sure the file is created and its contents are written. +Use join() to make the parent wait for its child before reading the file.

Practice: fork()​

Up to now we've been creating processes using various high-level APIs, such as Popen(), Process() and system(). +Yes, despite being a C function, as you've seen from its man page, system() itself calls 2 other functions: fork() to create a process and execve() to execute the given command. +As you already know from the Software Stack chapter, library functions may call one or more underlying system calls or other functions. +Now we will move one step lower on the call stack and call fork() ourselves.

fork() creates one child process that is almost identical to its parent. +We say that fork() returns twice: once in the parent process and once more in the child process. +This means that after fork() returns, assuming no error has occurred, both the child and the parent resume execution from the same place: the instruction following the call to fork(). +What's different between the two processes is the value returned by fork():

  • child process: fork() returns 0
  • parent process: fork() returns the PID of the child process (> 0)
  • on error: fork() returns -1, only once, in the initial process

Therefore, the typical code for handling a fork() is available in support/create-process/fork.c. +Take a look at it and then run it. +Notice what each of the two processes prints:

  • the PID of the child is also known by the parent
  • the PPID of the child is the PID of the parent

Unlike system(), who also waits for its child, when using fork() we must do the waiting ourselves. +In order to wait for a process to end, we use the waitpid() syscall. +It places the exit code of the child process in the status parameter. +This argument is actually a bit-field containing more information than merely the exit code. +To retrieve the exit code, we use the WEXITSTATUS macro. +Keep in mind that WEXITSTATUS only makes sense if WIFEXITED is true, i.e. if the child process finished on its own and wasn't killed by another one or by an illegal action (such as a segfault or illegal instruction) for example. +Otherwise, WEXITSTATUS will return something meaningless. +You can view the rest of the information stored in the status bit-field in the man page.

Now modify the example to do the following:

  1. Change the return value of the child process so that the value displayed by the parent is changed.

  2. Create a child process of the newly created child. +Use a similar logic and a similar set of prints to those in the support code. +Take a look at the printed PIDs. +Make sure the PPID of the "grandchild" is the PID of the child, whose PPID is, in turn, the PID of the parent.

Moral of the story: Usually the execution flow is:

  1. fork(), followed by

  2. wait() (called by the parent)

  3. exit(), called by the child.

The order of last 2 steps may be swapped.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/apache2-strace/index.html b/17/Lab/Compute/quiz/apache2-strace/index.html new file mode 100644 index 0000000000..7e4e27325b --- /dev/null +++ b/17/Lab/Compute/quiz/apache2-strace/index.html @@ -0,0 +1,17 @@ + + + + + +`apache2` Document Root | Operating Systems + + + + +
+
Skip to main content

apache2 Document Root

Question Text​

What is the document root of the apache2 server?

Question Answers​

  • /etc/apache2
  • /usr/local/apache2/htdocs/

  • /var/www/html

  • /var/www/apache2/htdocs

Feedback​

In strace we see that the server opens the file /usr/local/apache2/htdocs/index.html. +This means that the document root is /usr/local/apache2/htdocs/.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/cause-of-file-not-found-error/index.html b/17/Lab/Compute/quiz/cause-of-file-not-found-error/index.html new file mode 100644 index 0000000000..f63ecac8a2 --- /dev/null +++ b/17/Lab/Compute/quiz/cause-of-file-not-found-error/index.html @@ -0,0 +1,21 @@ + + + + + +Cause of `FileNotFoundError` | Operating Systems + + + + +
+
Skip to main content

Cause of FileNotFoundError

Question Text​

What causes the FileNotFoundError when running support/wait-for-me/wait_for_me_processes.py?

Question Answers​

  • The parent process attempts to open the file before the child process has had the time to create it

  • There is a syntax error in the Python code

  • The mode with which to open the file is not specified in the parent process

  • The child process doesn't close the file

Feedback​

What you've just experienced is a race condition. +Race conditions are situations in which one thread uses data that may or may not have been previously modified by another thread. +Because scheduling is generally nondeterministic, this means that on some runs, the first thread may access unmodified data and modified on others. +It's impossible to say what kind of data will be used by the first thread. +In our case, the data is the file you give to the script as an argument. +If scheduling the parent process or its running time takes long enough, the file may have been created by the time the parent needs it, but we can never be sure.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/child-faults-after-write/index.html b/17/Lab/Compute/quiz/child-faults-after-write/index.html new file mode 100644 index 0000000000..6ed8e3c565 --- /dev/null +++ b/17/Lab/Compute/quiz/child-faults-after-write/index.html @@ -0,0 +1,16 @@ + + + + + +Child Faults After Write | Operating Systems + + + + +
+
Skip to main content

Child Faults After Write

Question Text​

What causes the page faults registered by the child after the fifth step?

Question Answers​

  • The child writes data to the frames it previously shared with its parent and the copy-on-write mechanism copies and remaps them before writing said data

  • Demand paging propagates the lazy allocation of pages from the parent to the child

  • Creating the child process inherently duplicates some frames

  • They are caused by the loader forking itself when creating the child process

  • They are caused by the bash process forking itself when creating the child process

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/coarse-vs-granular-critical-section/index.html b/17/Lab/Compute/quiz/coarse-vs-granular-critical-section/index.html new file mode 100644 index 0000000000..b86e2dd23e --- /dev/null +++ b/17/Lab/Compute/quiz/coarse-vs-granular-critical-section/index.html @@ -0,0 +1,20 @@ + + + + + +Coarse vs Granular Critical Section | Operating Systems + + + + +
+
Skip to main content

Coarse vs Granular Critical Section

Question Text​

Why does code with the the larger (coarser) critical section run faster than the one with the smaller (more granular) critical section?

Question Answers​

  • Because the more granular code causes more context switches, which are expensive

  • Because the coarser code can be better optimised by the compiler

  • Because the loops in the more granular code run for more steps

Feedback​

The larger critical sections only require 2 context switches. +The first thread that reaches the call to lock() acquires to the mutex and starts executing the whole of the for loop. +The second thread then finds the mutex locked and enters the WAITING state. +When the first thread finishes its loop, it calls unlock() and wakes up the second thread, which acquires the lock and starts its loop.

In the more granular example, in the worst case, the holder of the mutex can change at every step of the loop. +This would mean 1 context switch per step per thread, i.e. 20 million context switches.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/create-sleepy-process-ending/index.html b/17/Lab/Compute/quiz/create-sleepy-process-ending/index.html new file mode 100644 index 0000000000..c9b82ac07b --- /dev/null +++ b/17/Lab/Compute/quiz/create-sleepy-process-ending/index.html @@ -0,0 +1,17 @@ + + + + + +`create_sleepy` Process Ending | Operating Systems + + + + +
+
Skip to main content

create_sleepy Process Ending

Question Text​

Why does the create_sleepy process wait a very long time before ending? +Use system's man page to find the answer.

Question Answers​

  • Because the code is unoptimised (the default optimisation level is -O0)

  • Because the operating system takes very long to finish the process

  • Because system returns when the command given to it (sleep 1000) ends
  • Because the CPU is very slow

Feedback​

The man page says it clearly:

system() returns after the command has been completed.

Therefore, in our case, it returns after sleep 1000 ends.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/fiber-strace/index.html b/17/Lab/Compute/quiz/fiber-strace/index.html new file mode 100644 index 0000000000..9acab03aaf --- /dev/null +++ b/17/Lab/Compute/quiz/fiber-strace/index.html @@ -0,0 +1,17 @@ + + + + + +Fiber Strace | Operating Systems + + + + +
+
Skip to main content

Fiber Strace

Question Text​

How many clone() system calls are performed when creating a fiber?

Question Answers​

  • none

  • one for each fiber

  • one for every 2 fibers

  • 2 for each fiber

Feedback​

Being user-level threads, the fibers aren't created by the operating system. +The only system calls that you see used are mmap() and munmap(), used to create each fiber's stack.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/mini-shell-stops-after-command/index.html b/17/Lab/Compute/quiz/mini-shell-stops-after-command/index.html new file mode 100644 index 0000000000..afadb2e662 --- /dev/null +++ b/17/Lab/Compute/quiz/mini-shell-stops-after-command/index.html @@ -0,0 +1,19 @@ + + + + + +Mini-shell Stops After Command | Operating Systems + + + + +
+
Skip to main content

Mini-shell Stops After Command

Question Text​

Why does the mini_shell process stop after executing a single command?

Question Answers​

  • Because of an implementation error
  • Because the mini_shell process doesn't exist anymore

  • Because the OS sees that the command has ended and ends the mini_shell process as well

  • Because exec*() syscalls also kill the caller process when the callee ends

Feedback​

When you exec*() any binary, the VAS current process is replaced by that corresponding to that binary. +So when you exec*("ls"), for example, the mini_shell process becomes ls. +There is no more mini_shell past this point. +So when ls ends, there is no mini_shell process to continue its execution anymore.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/mmap-cow-flag/index.html b/17/Lab/Compute/quiz/mmap-cow-flag/index.html new file mode 100644 index 0000000000..c23c78b01f --- /dev/null +++ b/17/Lab/Compute/quiz/mmap-cow-flag/index.html @@ -0,0 +1,16 @@ + + + + + +Copy-on-write Flag for `mmap()` | Operating Systems + + + + +
+
Skip to main content

Copy-on-write Flag for mmap()

Question Text​

From the description in its man page, what flag should we pass to mmap() in order to mark the mapped pages as copy-on-write?

Question Answers​

  • MAP_SHARED
  • MAP_PRIVATE

  • MAP_ANONYMOUS

  • MAP_POPULATE

Feedback​

Quoting the man page:

MAP_PRIVATE
              Create a private copy-on-write mapping.
+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/notify-only-with-mutex/index.html b/17/Lab/Compute/quiz/notify-only-with-mutex/index.html new file mode 100644 index 0000000000..e8524836f1 --- /dev/null +++ b/17/Lab/Compute/quiz/notify-only-with-mutex/index.html @@ -0,0 +1,19 @@ + + + + + +Both Condition and Mutex | Operating Systems + + + + +
+
Skip to main content

Both Condition and Mutex

Question Text​

Can we only use a mutex when signalling an event from one thread to another in?

Question Answers​

  • No, because this would imply that the signalling thread would unlock() the mutex, that the signalled thread attempts to lock(), which is an undefined behaviour

  • No, because it will result in a deadlock where the both threads will be waiting for each other

  • Yes, because only one thread can modify the shared variables in order to maintain their integrity

  • Yes and this would yield better performance because the threads would only wait for one object: the mutex

Feedback​

In some implementations, such as POSIX threads (pthreads), calling unlock() from another thread than that which called lock() can result in an undefined behaviour. +For this reason, it is unsafe to only use a mutex as a notification mechanism. +In addition, a mutex cannot notify more than one thread at once, if we so desire. +Mutexes are only meant to be used to isolate a critical section within the same thread.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/number-of-running-ults/index.html b/17/Lab/Compute/quiz/number-of-running-ults/index.html new file mode 100644 index 0000000000..6704d96835 --- /dev/null +++ b/17/Lab/Compute/quiz/number-of-running-ults/index.html @@ -0,0 +1,17 @@ + + + + + +Number of RUNNING User-Level Threads | Operating Systems + + + + +
+
Skip to main content

Number of RUNNING User-Level Threads

Question Text​

How many threads can be RUNNING simultaneously if we only create them using the API exposed by libult.so?

Question Answers​

  • Equal to the number of cores on the CPU
  • 1

  • None

  • 2: the main thread and another one for the created threads

Feedback​

Only kernel-level threads can run in parallel. +Since all libult.so threads are user-level threads, they run within the same kernel-level thread, so only one of them can run at any time.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/parent-faults-before-fork/index.html b/17/Lab/Compute/quiz/parent-faults-before-fork/index.html new file mode 100644 index 0000000000..1e4bb17306 --- /dev/null +++ b/17/Lab/Compute/quiz/parent-faults-before-fork/index.html @@ -0,0 +1,16 @@ + + + + + +Parent Faults before `fork()` | Operating Systems + + + + +
+
Skip to main content

Parent Faults before fork()

Question Text​

What causes the page faults that occur between the first and second steps?

Question Answers​

  • Calling fork() duplicates the pages previously allocated by the parent
  • Demand paging makes the pages in the p array to be mapped to frames only when written

  • The OS duplicates the parent's pages in preparation for fork()

  • mmap() sets the pages to be mapped at a later time, decided by the OS

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/parent-of-sleep-processes/index.html b/17/Lab/Compute/quiz/parent-of-sleep-processes/index.html new file mode 100644 index 0000000000..3c3157a61b --- /dev/null +++ b/17/Lab/Compute/quiz/parent-of-sleep-processes/index.html @@ -0,0 +1,19 @@ + + + + + +Parent of `sleep` Processes | Operating Systems + + + + +
+
Skip to main content

Parent of sleep Processes

Question Text​

Who is the parent of the sleep processes? +Why?

Question Answers​

  • sleepy_creator.py because it is the one who created them

  • bash because it is sleepy_creator.py's parent and when a process dies, its parent adopts its orphan children

  • systemd because this is the default process that adopts orphans
  • systemd because it is sleepy_creator.py's parent and when a process dies, its parent adopts its orphan children

Feedback​

When a process dies without waiting for the termination of all its children, those processes are now orphans. +Then the systemd process adopts those orphan processes by default. +On older Linux systems, it was the init process who adopted orphans.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/processes-speedup/index.html b/17/Lab/Compute/quiz/processes-speedup/index.html new file mode 100644 index 0000000000..33f58a9bf4 --- /dev/null +++ b/17/Lab/Compute/quiz/processes-speedup/index.html @@ -0,0 +1,22 @@ + + + + + +Processes Speedup | Operating Systems + + + + +
+
Skip to main content

Processes Speedup

Question Text​

Why is the speedup from running the program in support/sum-array/d/sum_array_processes.d with 1, 2, 4 and 8 processes less than expected?

Question Answers​

  • Because the array is split into unequal parts
  • Because of the overhead introduced by the creation of the additional processes

  • Because the algorithm is incorrect

  • Because the operating systems runs all processes sequentially on the same core

Feedback​

Creating a new process involves an inherent overhead. +The OS calls the loader, launches the new process, then the parent process waits for it to finish, extracts its return value etc. +All this work together with creating the initial process has to be done by a single thread. +In addition, in real-world apps, other actions such as receiving data from the network or reading a file are inherently sequential. +Therefore there will always be parts of any given program that cannot be run in parallel. +As a result, the speedup can never be equal to the number of processes between which we spread the workload.

It is possible to compute the speedup obtained from parallelising a portion of a given program. +The formula is rather simple and is called Amdahl's law

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/seg-fault-exit-code/index.html b/17/Lab/Compute/quiz/seg-fault-exit-code/index.html new file mode 100644 index 0000000000..15b2666431 --- /dev/null +++ b/17/Lab/Compute/quiz/seg-fault-exit-code/index.html @@ -0,0 +1,19 @@ + + + + + +Seg Fault Exit Code | Operating Systems + + + + +
+
Skip to main content

Seg Fault Exit Code

Question Text​

What is the exit code of the faulty child process spawned by support/sum-array-bugs/seg-fault/sum_array_processes.d with more than 2 processes?

Question Answers​

  • 11 because this is the code of the SIGSEGV signal

  • 11 because this code is always returned when a process ends with an error

  • 11 because this is the value of the least significant 4 bytes of the partial array sum calculated by the process

  • 6 because the child process was aborted

Feedback​

We can obtain the number of the signal that killed a child process via the second argument of the waitpid syscall. +We can use the WIFSIGNALED() and WTERMSIG() marcros. +By doing so, we see the exit code of the faulty child process is 11. +We can then use the kill -l command to view the code of each signal and SIGSEGV has the code 11.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/semaphore-equivalent/index.html b/17/Lab/Compute/quiz/semaphore-equivalent/index.html new file mode 100644 index 0000000000..39bca2949b --- /dev/null +++ b/17/Lab/Compute/quiz/semaphore-equivalent/index.html @@ -0,0 +1,17 @@ + + + + + +Semaphore Equivalent | Operating Systems + + + + +
+
Skip to main content

Semaphore Equivalent

Question Text​

From running and inspecting the code in support/apache2-simulator/apache2_simulator_semaphore.py, which of the following is an an equivalent to the value of the semaphore sem?

Question Answers​

  • The value of msg_mutex

  • The time a worker thread has to wait before running

  • The length of the messages list
  • The number of worker threads

Feedback​

sem is incremented (release()) upon adding a message to the messages list and decremented (acquire()) when removing a message from said list. +So it's a rough equivalent to the length of this list.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/sleeping-on-a-fiber/index.html b/17/Lab/Compute/quiz/sleeping-on-a-fiber/index.html new file mode 100644 index 0000000000..faabb9ed29 --- /dev/null +++ b/17/Lab/Compute/quiz/sleeping-on-a-fiber/index.html @@ -0,0 +1,17 @@ + + + + + +Sleeping on a Fiber | Operating Systems + + + + +
+
Skip to main content

Sleeping on a Fiber

Question Text​

What happens if a fiber calls sleep()?

Question Answers​

  • the whole kernel-level thread is blocked

  • only that fiber is blocked, and is scheduled out

  • nothing, a fiber can't sleep

  • the whole process sleeps - regardless of how many threads there are

Feedback​

The whole thread on which the fiber runs is blocked until the sleep() call is finished. +For this reason, the fibers are best used with asynchronous operations, which you will explore in the following weeks.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/state-of-new-ult/index.html b/17/Lab/Compute/quiz/state-of-new-ult/index.html new file mode 100644 index 0000000000..5bab49f855 --- /dev/null +++ b/17/Lab/Compute/quiz/state-of-new-ult/index.html @@ -0,0 +1,17 @@ + + + + + +State of new ULT | Operating Systems + + + + +
+
Skip to main content

State of new ULT

Question Text​

What is the first state that is assigned to a newly created ULT?

Question Answers​

  • RUNNING
  • READY
  • COMPLETED

Feedback​

Inside the function threads_create(), we can see the following snippet queue_enqueue(ready, new). +Each new thread is thus added to the READY queue.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/tcb-libult-unikraft/index.html b/17/Lab/Compute/quiz/tcb-libult-unikraft/index.html new file mode 100644 index 0000000000..9ffe50ddf8 --- /dev/null +++ b/17/Lab/Compute/quiz/tcb-libult-unikraft/index.html @@ -0,0 +1,20 @@ + + + + + +Similarities Between the TCBs of `libult` and Unikraft | Operating Systems + + + + +
+
Skip to main content

Similarities Between the TCBs of libult and Unikraft

Question Text​

Which members of the TCBs in libult and Unikraft have similar meanings?

Question Answers​

  • start_routine and entry
  • id and name
  • context.uc_stack and stack

  • arguments and flags

  • has_dynamic_stack and detached

  • argument and arg

  • context and sched

  • context and tls

  • context and ctx

  • return_value and prv

Feedback​

start_routine and entry are the functions that run in the newly created threads. +context.uc_stack and stack are pointers to the stack of the newly created threads. +argument and arg are pointers to the arguments of start_routine and entry, respectively. +context and ctx are the contexts in which the new threads run. +return_value and prv are both pointers to the values returned by the thread functions.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/thread-memory/index.html b/17/Lab/Compute/quiz/thread-memory/index.html new file mode 100644 index 0000000000..601d1c2f3f --- /dev/null +++ b/17/Lab/Compute/quiz/thread-memory/index.html @@ -0,0 +1,17 @@ + + + + + +Thread Memory | Operating Systems + + + + +
+
Skip to main content

Thread Memory

Question Text​

Is data used by a thread inaccessible from other threads?

Question Answers​

  • Yes, each thread has its own stack
  • No, each thread can access every address from the virtual address space

  • Only the heap is shared

  • Only the heap and the read-only zones are shared

Feedback​

Each thread has the same perspective on the system memory - it thinks it owns it all. +Therefore, it can perform pointer arithmetic to access every memory address.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/time-slice-value/index.html b/17/Lab/Compute/quiz/time-slice-value/index.html new file mode 100644 index 0000000000..c720dcd183 --- /dev/null +++ b/17/Lab/Compute/quiz/time-slice-value/index.html @@ -0,0 +1,16 @@ + + + + + +Time Slice Value | Operating Systems + + + + +
+
Skip to main content

Time Slice Value

Question Text​

Using the man page, what is the time slice used by the scheduler in libult.so?

Question Answers​

  • 100 milliseconds

  • 10 microseconds

  • 100 microseconds

  • 10 milliseconds

Feedback​

The code we're interested in lies in the function init_profiling_timer():

const struct itimerval timer = {
    { 0, 10000 },
    { 0, 1 }  // arms the timer as soon as possible
};

The man page gives the following definition the struct itimerval:

struct itimerval {
    struct timeval it_interval; /* Interval for periodic timer */
    struct timeval it_value;    /* Time until next expiration */
};

struct timeval {
    time_t      tv_sec;         /* seconds */
    suseconds_t tv_usec;        /* microseconds */
};

So when constructing the timer variable, { 0, 10000 } means 0 seconds and 10000 microseconds, i.e. 0 seconds and 10 milliseconds.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/tls-synchronization/index.html b/17/Lab/Compute/quiz/tls-synchronization/index.html new file mode 100644 index 0000000000..89c65aeba4 --- /dev/null +++ b/17/Lab/Compute/quiz/tls-synchronization/index.html @@ -0,0 +1,17 @@ + + + + + +TLS Synchronization | Operating Systems + + + + +
+
Skip to main content

TLS Synchronization

Question Text​

Is placing var from support/race-condition/c/race_condition_tls.c in the TLS a valid form of synchronization?

Question Answers​

  • No, because the race condition remains. +It just doesn't manifest itself anymore
  • No, because the threads now access different variables, not the same one

  • Yes, because we now remove the race condition

  • Yes, because now the result is correct

Feedback​

Synchronization means that both threads should access the same variable, whereas placing it in the TLS makes each of them access a copy of the variable.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/tls-var-copies/index.html b/17/Lab/Compute/quiz/tls-var-copies/index.html new file mode 100644 index 0000000000..d3d01f1676 --- /dev/null +++ b/17/Lab/Compute/quiz/tls-var-copies/index.html @@ -0,0 +1,16 @@ + + + + + +TLS `var` Copies | Operating Systems + + + + +
+
Skip to main content

TLS var Copies

Question Text​

How many copies of the var variable from support/race-condition/c/race_condition_tls.c are there after each thread has modified it at leas once?

Question Answers​

  • 1

  • 2

  • 3
  • 5

Feedback​

There are 3 copies one for the main() thread, another one for increment_var() and the third for decrement_var().

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/type-of-scheduler-in-libult/index.html b/17/Lab/Compute/quiz/type-of-scheduler-in-libult/index.html new file mode 100644 index 0000000000..1f5ab1457b --- /dev/null +++ b/17/Lab/Compute/quiz/type-of-scheduler-in-libult/index.html @@ -0,0 +1,19 @@ + + + + + +Type of Scheduler in `libult.so` | Operating Systems + + + + +
+
Skip to main content

Type of Scheduler in libult.so

Question Text​

Inspect the code in support/libult/threads.c further. +Which type of scheduler does libult.so use?

Question Answers​

  • It uses a preemptive scheduler

  • It uses a cooperative scheduler

  • It uses both a cooperative and a preemptive scheduler

Feedback​

libult.so uses a preemptive scheduler. +Its timer is initialised in the init_profiling_timer() function. +The context switch is performed in the handle_sigprof() function.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/ult-thread-ids/index.html b/17/Lab/Compute/quiz/ult-thread-ids/index.html new file mode 100644 index 0000000000..201469bedb --- /dev/null +++ b/17/Lab/Compute/quiz/ult-thread-ids/index.html @@ -0,0 +1,21 @@ + + + + + +ULT Thread IDs | Operating Systems + + + + +
+
Skip to main content

ULT Thread IDs

Question Text​

Why do the thread IDs returned by threads_create() start from 2 and not 1? +Why is this necessary.

Question Answers​

  • Because ID 1 is associated with the main thread. +This is an implementation detail and can be omitted.
  • Because ID 1 belongs to the main thread. +This is needed in order to associate a ucontext_t with the main thread as well, so the main thread can also be run.

  • Because the underlying kernel thread is assigned ID 1. +This is mandatory in order for the OS's scheduler to run this thread.

  • Because libult.so first creates a pool of threads from which it threads_create() retrieves the threads it returns.

Feedback​

The threads_create() function calls init_first_context(), which, in turn, calls tcb_new(), thus creating the first context associated with the main thread (the one calling threads_create() the first time). +Without this, the scheduler in libult.so wouldn't be able to run the main thread.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/who-calls-execve-parent/index.html b/17/Lab/Compute/quiz/who-calls-execve-parent/index.html new file mode 100644 index 0000000000..5439947c0d --- /dev/null +++ b/17/Lab/Compute/quiz/who-calls-execve-parent/index.html @@ -0,0 +1,16 @@ + + + + + +Who Calls `execve` in the Log of the Parent Process? | Operating Systems + + + + +
+
Skip to main content

Who Calls execve in the Log of the Parent Process?

Question Text​

Which process calls execve("sleepy_creator", ["sleepy_creator"], ...), that you found in the log of the parent process?

Question Answers​

  • The kernel because that's where the loader is located

  • The loader because it is the loader who creates new processes

  • The C runtime because this is the C interpreter

  • bash because we run sleepy_creator from terminal, i.e. from bash

Feedback​

All processes spawned from the command-line are children of the current bash process.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/quiz/why-use-completed-queue/index.html b/17/Lab/Compute/quiz/why-use-completed-queue/index.html new file mode 100644 index 0000000000..693dab18a9 --- /dev/null +++ b/17/Lab/Compute/quiz/why-use-completed-queue/index.html @@ -0,0 +1,20 @@ + + + + + +The Need for a COMPLETED Queue | Operating Systems + + + + +
+
Skip to main content

The Need for a COMPLETED Queue

Question Text​

Why does the scheduler need a COMPLETED queue and not simply terminate one thread once its function finishes?

Question Answers​

  • The COMPLETED queue is an implementation preference. +The scheduler can expose the same functions without it

  • Because the OS's scheduler may kill the main kernel-level thread unless we keep the user-level thread in a queue

  • The COMPLETED queue is needed to save the value returned by the thread so that it can later be retrieved by threads_join().

Feedback​

Take a look at the handle_thread_start() function. +It is used by threads_create() to start executing the given function. +This is a wrapper that calls the function associated with the thread (this->start_routine), saves its result and then calls threads_exit() +to store this result in the COMPLETED queue.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/synchronization/index.html b/17/Lab/Compute/synchronization/index.html new file mode 100644 index 0000000000..c35359950c --- /dev/null +++ b/17/Lab/Compute/synchronization/index.html @@ -0,0 +1,140 @@ + + + + + +Synchronization | Operating Systems + + + + +
+
Skip to main content

Synchronization

So far, we've used threads and processes without wondering how to "tell" them how to access shared data. +Moreover, in order to make threads wait for each other, we simply had the main thread wait for the others to finish all their work. +But what if we want one thread to wait until another one simply performs some specific action, after which it resumes its execution? +For this, we need to use some more complex synchronization mechanisms.

Race Conditions​

For example, what if one thread wants to increase a global variable while another one wants to decrease it? +Let's say the assembly code for increasing and decreasing the variable looks like the one in the snippet below.

increase:
    mov eax, [var]
    inc eax
    mov [var], eax

decrease:
    mov eax, [var]
    dec eax
    mov [var], eax

Imagine both threads executed mov eax, [var] at the same time. +Then each would independently increase its (non-shared) eax register. +In the end, the final value of var depends on which thread executes mov [var], eax last. +So it's kind of a reversed race. +The thread that runs the slowest "wins" this race and writes the final value of var. +But this is up to the scheduler and is non-deterministic. +Such undefined behaviours can cripple the execution of a program if var is some critical variable.

Let's see this bug in action. +Go to support/race-condition/c/race_condition.c, compile and run the code a few times. +It spawns to threads that do exactly what we've talked about so far: one thread increments var 10 million times, while the other decrements it 10 million times.

As you can see from running the program, the differences between subsequent runs can be substantial. +To fix this, we must ensure that only one thread can execute either var++ or var-- at any time. +We call these code sections critical sections. +A critical section is a piece of code that can only be executed by one thread at a time. +So we need some sort of mutual exclusion mechanism so that when one thread runs the critical section, the other has to wait before entering it. +This mechanism is called a mutex, whose name comes from "mutual exclusion".

Go to support/race-condition/c/race_condition_mutex.c and notice the differences between this code and the buggy one. +We now use a pthread_mutex_t variable, which we lock at the beginning of a critical section, and we unlock at the end. +Generally speaking, lock-ing a mutex makes a thread enter a critical section, while calling pthread_mutex_unlock() makes the thread leave said critical section. +Therefore, as we said previously, the critical sections in our code are var-- and var++. +Run the code multiple times to convince yourself that in the end, the value of var will always be 0.

Mutexes contain an internal variable which can be either 1 (locked) or 0 (unlocked). +When a thread calls pthread_mutex_lock(), it attempts to set that variable to 1. +If it was 0, the thread sets it to 1 and proceeds to execute the critical section. +Otherwise, it suspends its execution and waits until that variable is set to 0 again.

When calling pthread_mute_unlock(), the internal variable is set to 0 and all waiting threads are woken up to try to acquire the mutex again. +Be careful: It is generally considered unsafe and in many cases undefined behaviour to call pthread_mutex_unlock() from a different thread than the one that acquired the lock. +So the general workflow should look something like this:

within a single thread:
    pthread_mutex_lock(&mutex)
    // do atomic stuff
    pthread_mutex_unlock(&mutex)

Synchronization - Overhead​

There ain't no such thing as a free lunch

This saying is also true for multithreading. +Running threads in parallel is nice and efficient, but synchronization always comes with a penalty: overhead. +Use the time command to record the running times of race_condition and race_condition_mutex. +Notice that those of race_condition_mutex are larger than those of race_condition.

The cause of this is that now when one thread is executing the critical section, the other has to wait and do nothing. +Waiting means changing its state from RUNNING to WAITING, which brings further overhead from the scheduler. +This latter overhead comes from the context switch that is necessary for a thread to switch its state from RUNNING to WAITING and back.

Practice: Wrap the Whole for Statements in Critical Sections​

Move the calls to pthread_mutex_lock() and pthread_mutex_unlock() outside the for statements so that the critical sections become the entire statement. +Measure the new time spent by the code and compare it with the execution times recorded when the critical sections were made up of only var-- and var++.

Quiz

Atomics​

So now we know how to use mutexes. +And we know that mutexes work by using an internal variable that can be either 1 (locked) or 0 (unlocked). +But how does pthread_mutex_lock() actually set that variable to 1? +How does it avoid a race condition in case another thread also wants to set it to 1?

We need a guarantee that anyone "touching" that variable does so "within its own critical section". +But now we need a critical section to implement a critical section... +To solve this circular problem, we make use of a very common Deus ex Machina: hardware support.

Modern processors are capable of atomically accessing data, either for reads or writes. +An atomic action is and indivisible sequence of operations that a thread runs without interference from others. +Concretely, before initiating an atomic transfer on one of its data buses, the CPU first makes sure all other transfers have ended, then locks the data bus by stalling all cores attempting to transfer data on it. +This way, one thread obtains exclusive access to the data bus while accessing data. +As a side note, the critical sections in support/race-condition/c/race_condition_mutex.c are also atomic once they are wrapped between calls to pthread_mutex_lock() and pthread_mutex_unlock().

As with every hardware feature, the x86 ISA exposes an instruction for atomic operations. +In particular, this instruction is a prefix, called lock. +It makes the instruction that follows it run atomically. +The lock prefix ensures that the core performing the instruction has exclusive ownership of the cache line from where the data is transferred for the entire operation. +This is how the increment is made into an indivisible unit.

For example, inc dword [x] can be made atomic, like so: lock inc dword [x]. +You can play with the lock prefix in the Arena.

Compilers provide support for such hardware-level atomic operations. +GCC exposes built-ins such as __atomic_load(), __atomic_store(), __atomic_compare_exchange() and many others. +All of them rely on the mechanism described above.

Go to support/race-condition/c/race_condition_atomic.c and complete the function decrement_var(). +Compile and run the code. +Now measure its running time against the mutex implementations. +It should be somewhere between race_condition.c and race_condition_mutex.c.

The C standard library also provides atomic data types. +Access to these variables can be done only by one thread at a time. +Go to support/race-condition/c/race_condition_atomic2.c, compile and run the code. +Now measure its running time against the other implementations. +Notice that the time is similar to race_condition_atomic.

So using the hardware support is more efficient, but it usually is leveraged only for simple, individual instructions, such as loads and stores. +And the fact that high-level languages also expose an API for atomic operations shows how useful these operations are for developers.

Semaphores​

Up to now, we've learned how to create critical sections that can be accessed by only one thread at a time. +These critical sections revolved around data. +Whenever we define a critical section, there is some specific data to which we cannot allow parallel access. +The reason why we can't allow it is, in general, data integrity, as we've seen in our examples in support/race-condition/

But what if threads need to count? +Counting is inherently thread-unsafe because it's a read-modify-write operation. +We read the counter, increment (modify) it and then write it back. +Think about our example with apache2 +Let's say a worker has created a pool of 3 threads. +They are not doing any work initially; +they are in the WAITING state. +As clients initiate connections, these threads are picked up and are used to serve at most 3 connections at a time. +But the number of connections may be arbitrarily large. +Therefore, we need a way to keep track of it. +When serving a client, a thread should decrement it to inform the others that a connection has been finished. +In short, we need a counter that the dispatcher increments and that worker threads decrement.

Such a counter could be implemented using a semaphore. +For simplicity's sake, you can view a semaphore as simply a mutex whose internal variable can take any value and acts like a counter. +When a thread attempts to acquire() a semaphore, it will wait if this counter is less than or equal to 0. +Otherwise, the thread decrements the internal counter and the function returns. +The opposite of acquire() is release(), which increases the internal counter by a given value (by default 1).

Practice: apache2 Simulator - Semaphore​

Go to support/apache2-simulator/apache2_simulator_semaphore.py. +In the main() function we create a semaphore which we increment (release()) upon every new message. +Each thread decrements (acquire()) this semaphore to signal that it wants to retrieve a message from the list. +The retrieval means modifying a data structure, which is a critical section, so we use a separate mutex for this. +Otherwise, multiple threads could acquire the semaphore at the same time and try to modify the list at the same time. +Not good.

Locking this mutex (which in Python is called Lock) is done with the following statement: with msg_mutex: +This is a syntactic equivalent to:

event.acquire()
messages.append(msg)
event.release()

Quiz

Since the length of the messages list is simply len(messages), it may seem a bit redundant to use a semaphore to store essentially the same value. +In the next section, we'll look at a more refined mechanism for our use case: condition variables.

Conditions​

Another way we can implement our apache2 simulator is to use a condition variable. +This one is probably the most intuitive synchronization primitive. +It's a means by which a thread can tell another one: "Hey, wake up, this happened!". +So it's a way for threads to notify each other. +For this reason, the main methods associated with conditions are notify() and wait(). +As you might expect, they are complementary:

  • wait() puts the thread in the WAITING state until it's woken up by another one
  • notify() wakes up one or more wait()-ing threads. +If notify() is called before any thread has called wait(), the first thread that calls it will continue its execution unhindered.

Practice: apache2 Simulator - Condition​

But this is not all, unfortunately. +Look at the code in support/apache2-simulator/apache2_simulator_condition.py. +See the main thread call notify() once it reads the message. +Notice that this call is preceded by an acquire() call, and succeeded by a release() call.

acquire() and release() are commonly associated with mutexes or semaphores. +What do they have to do with condition variables?

Well, a lock Condition variable also stores an inner lock (mutex). +It is this lock that we acquire() and release(). +In fact, the documentation states we should only call Condition methods with its inner lock taken.

Why is this necessary? +Take a look at the worker() function. +After wait()-ing (we'll explain the need for the loop in a bit), it extracts a message from the message queue. +This operation is not atomic, so it must be enclosed within a critical section. +Hence, the lock.

Quiz

So now we know we cannot only use a mutex. +The mutex is used to access and modify the messages list atomically. +Now, you might be thinking that this code causes a deadlock:

event.acquire()
while len(messages) == 0:
    event.wait()

The thread gets the lock and then, if there are no messages, it switches its state to WAITING. +A classic deadlock, right? +No. +wait() also releases the inner lock of the Condition and being woken up reacquires it. +Neat! +And the while loop that checks if there are any new messages is necessary because wait() can return after an arbitrary long time. +This is because the thread waiting for the event was notified to wake up, but another thread has woken up before it and started handling the event earlier by reacquiring the lock. +All the other threads that woke up, but can't acquire the lock, must be put back to wait. +This situation is called a spurious wakeup. +Therefore, it's necessary to check for messages again when waking up.

So now we have both synchronization and signalling. +This is what conditions are for, ultimately.

Now that you understand the concept of synchronization, you should apply it in a broader context. +In the Arena, you'll find an exercise asking you to make an existing array list implementation thread-safe. +Have fun!

Thread-Local Storage (TLS)​

First things first: what if we don't want data to be shared between threads? +Are we condemned to have to worry about race conditions? +Well, no.

To protect data from race conditions "by design", we can place in what's called Thread-Local Storage (TLS). +As its name implies, this is a type of storage that is "owned" by individual threads, as opposed to being shared among all threads. +Do not confuse it with copy-on-write. +TLS pages are always duplicated when creating a new thread and their contents are reinitialised.

Practice: C - TLS on Demand​

The perspective of C towards TLS is the following: everything is shared by default. +This makes multithreading easier and more lightweight to implement than in other languages, like D, because synchronization is left entirely up to the developer, at the cost of potential unsafety.

Of course, we can specify that some data belongs to the TLS, by preceding the declaration of a variable with __thread keyword. +First, compile and run the code in support/race-condition/c/race_condition_tls.c a few times. +As expected, the result is different each time.

  1. Modify the declaration of var and add the __thread keyword to place the variable in the TLS of each thread. +Recompile and run the code a few more times. +You should see that in the end, var is 0.

Quiz 1

Quiz 2

  1. Print the address and value of var in each thread. +See that they differ.

  2. Modify the value of var in the main() function before calling pthread_create(). +Notice that the value doesn't propagate to the other threads. +This is because, upon creating a new thread, its TLS is initialised.

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/threads/index.html b/17/Lab/Compute/threads/index.html new file mode 100644 index 0000000000..d7484dc713 --- /dev/null +++ b/17/Lab/Compute/threads/index.html @@ -0,0 +1,64 @@ + + + + + +Threads | Operating Systems + + + + +
+
Skip to main content

Threads

Spreading the Work Among Other Threads​

Compile the code in support/sum-array/c/sum_array_threads.c and run it using 1, 2, 4 and 8 threads as you did before. +Each thread runs the calculateArrayPartSum function and then finishes. +Running times should be slightly smaller than the implementation using processes. +This slight time difference is caused by process creation actions, which are costlier than thread creation actions. +Because a process needs a separate virtual address space (VAS) and needs to duplicate some internal structures such as the file descriptor table and page table, it takes the operating system more time to create it than to create a thread. +On the other hand, threads belonging to the same process share the same VAS and, implicitly, the same OS-internal structures. +Therefore, they are more lightweight than processes.

Practice: Wait for Me Once More​

Go to support/wait-for-me/wait_for_me_threads.c. +Spawn a thread that executes the negate_array() function. +For now, do not wait for it to finish; +simply start it.

Compile the code and run the resulting executable several times. +See that the negative numbers appear from different indices. +This is precisely the nondeterminism that we talked about in the previous section.

Now wait for that thread to finish and see that all the printed numbers are consistently negative.

As you can see, waiting is a very coarse form of synchronization. +If we only use waiting, we can expect no speedup as a result of parallelism, because one thread must finish completely before another can continue. +We will discuss more fine-grained synchronization mechanisms later in this lab.

Also, at this point, you might be wondering why this exercise is written in D, while the same exercise, but with processes was written in Python. +There is a very good reason for this and has to do with how threads are synchronized by default in Python. +You can find out what this is about in the Arena section, after you have completed the Synchronization section.

Threads vs Processes​

So why use the implementation that spawns more processes if it's slower than the one using threads? +The table below lists the differences between threads and processes. +Generally, if we only want to do some computing, we use threads. +If we need to drastically change the behaviour of the program, we need a new program altogether, or we need more than computing (e.g. communication on the network to create a computing cluster), we use processes.

PROCESSTHREAD
independentpart of a process
collection of threadsshares VAS with other threads
slower creation (new page table must be created)faster creation
longer context switch duration (TLB must be flushed)shorter context switch duration (part of the same process, so same TLB)
ending means ending all threadsother threads continue when finished

Safety​

Compile and run the two programs in support/sum-array-bugs/seg-fault/, first with 2 processes and threads and then with 4. +They do the same thing as before: compute the sum of the elements in an array, but with a twist: each of them contains a bug causing a segfault. +Notice that sum_array_threads doesn't print anything with 4 threads, but merely a "Segmentation fault" message. +On the other hand, sum_array_processes prints a sum and a running time, albeit different from the sums we've seen so far.

The reason is that signals such as SIGSEGV, which is used when a segmentation fault happens affect the entire process that handles them. +Therefore, when we split our workload between several threads and one of them causes an error such as a segfault, that error is going to terminate the entire process. +The same thing happens when we use processes instead of threads: one process causes an error, which gets it killed, but the other processes continue their work unhindered. +This is why we end up with a lower sum in the end: because one process died too early and didn't manage to write the partial sum it had computed to the results array.

Practice: Wait for It​

The process that spawns all the others and subsequently calls waitpid to wait for them to finish can also get their return codes. +Update the code in support/sum-array-bugs/seg-fault/sum_array_processes.c and modify the call to waitpid to obtain and investigate this return code. +Display an appropriate message if one of the child processes returns an error.

Remember to use the appropriate macros for handling the status variable that is modified by waitpid(), as it is a bit-field. +When a process runs into a system error, it receives a signal. +A signal is a means to interrupt the normal execution of a program from the outside. +It is associated with a number. +Use kill -l to find the full list of signals.

Quiz

So up to this point we've seen that one advantage of processes is that they offer better safety than threads. +Because they use separate virtual address spaces, sibling processes are better isolated than threads. +Thus, an application that uses processes can be more robust to errors than if it were using threads.

Memory Corruption​

Because they share the same address space, threads run the risk of corrupting each other's data. +Take a look at the code in support/sum-array-bugs/memory-corruption/python/. +The two programs only differ in how they spread their workload. +One uses threads while the other uses processes.

Run both programs with and without memory corruption. +Pass any value as a third argument to trigger the corruption.

student@os:~/.../sum-array-bugs/memory-corruption/python$ python3 memory_corruption_processes.py <number_of_processes>  # no memory corruption
[...]

student@os:~/.../sum-array-bugs/memory-corruption/python$ python3 memory_corruption_processes.py <number_of_processes> 1  # do memory corruption
[...]

The one using threads will most likely print a negative sum, while the other displays the correct sum. +This happens because all threads refer to the same memory for the array arr. +What happens to the processes is a bit more complicated.

Later in this lab, we will see that initially, the page tables of all processes point to the same physical frames or arr. +When the malicious process tries to corrupt this array by writing data to it, the OS duplicates the original frames of arr so that the malicious process writes the corrupted values to these new frames, while leaving the original ones untouched. +This mechanism is called Copy-on-Write and is an OS optimisation so that memory is shared between the parent and the child process, until one of them attempts to write to it. +At this point, this process receives its own separate copies of the previously shared frames.

Note that in order for the processes to share the sums dictionary, it is not created as a regular dictionary, but using the Manager module. +This module provides some special data structures that are allocated in shared memory so that all processes can access them. +You can learn more about shared memory and its various implementations in the Arena section.

Memory Layout of Multithreaded Programs​

When a new thread is created, a new stack is allocated for a thread. +The default stack size if 8 MB / 8192 KB:

student@os:~$ ulimit -s
8192

Enter the support/multithreaded/ directory to observe the update of the memory layout when creating new threads.

Build the multithreaded executable:

student@os:~/.../lab/support/multithreaded$ make

Start the program:

student@os:~/.../lab/support/multithreaded$ ./multithreaded
Press key to start creating threads ...
[...]

And investigate it with pmap on another console, while pressing a key to create new threads.

As you can see, there is a new 8192 KB area created for every thread, also increasing the total virtual size.

Practice​

  1. Build the multithreaded program as a static executable by adding LDFLAGS = -static to the Makefile: +Run it. +You can check the executable is statically linked by executing the command ldd multithreaded. +Notice the same effect of the thread creation on the process memory layout: the creation of a new stack of 8192 KB.

  2. Make a program in another language of your choice that creates threads. +Investigate it with pmap.

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Compute/user-level-threads/index.html b/17/Lab/Compute/user-level-threads/index.html new file mode 100644 index 0000000000..11d02ff1d1 --- /dev/null +++ b/17/Lab/Compute/user-level-threads/index.html @@ -0,0 +1,50 @@ + + + + + +User-Level Threads | Operating Systems + + + + +
+
Skip to main content

User-Level Threads

User-level threads differ from the threads you are used to (kernel-level threads, those created by pthread_create). +This kind of threads are scheduled by an user-level scheduler, and can run on the same kernel-level thread. +From now on, we will reffer to user-level threads as fibers, and kernel-level threads as simply threads.

We will use the fiber implementation from libboost. +This implementation uses a cooperative scheduler on each thread, meaning that each fiber has to yield, in order for other fiber to be executed. +We will also use C++, and the standard thread implementation.

Prerequisites​

Unless you are using the OS docker image, you will need to install cmake and libboost. +You can do this with the following command:

student@os:~$ sudo apt-get install cmake libboost-context-dev libboost-fiber-dev

Creation​

Follow the support/user-level-threads/simple.cc implementation. +It creates NUM_FIBERS fibers, that each prints "Hello World". +To compile and run the program, do the following steps:

student@os:~/.../lab/support/user-level-threads$ mkdir build/
student@os:~/.../lab/support/user-level-threads$ cd build/
student@os:~/.../lab/support/user-level-threads$ cmake -S .. -B .
student@os:~/.../lab/support/user-level-threads$ make
student@os:~/.../lab/support/user-level-threads$ ./simple

The cmake step must be executed only once. +After modifying the source files, it is enough to run make.

Practice: Sleeper Fiber​

Add in support/user-level-threads/simple.cc a fiber that sleeps for 5 seconds, before the other ones are created. +What happens? +Answer in this quiz.

No system calls​

Use strace to find calls to clone() in the execution of simple. +Can you find any? +Provide your answer in this quiz +Remember that clone() is the system call used to create kernel-level threads, as pointed out here.

Synchronization​

By default, the fibers that run on the same thread are synchronized - no race-conditions can occur. +This is illustrated by the support/user-level-threads/sum.cc implementation.

The user can, however, implement further synchronization, by using the yield() call, or classic synchronization methods, like mutexes, barriers and condition variables.

Yielding​

As the scheduler is cooperative, each fiber can yield (or not), to allow another fiber to run. +Follow the support/user-level-threads/yield_launch.cc implementation and run it. +Note the boost::fibers::launch::dispatch parameter provided to the fiber constructor. +It notifies the scheduler to start the fibre as soon as it is created. +In order to explain the output, we must consider that the fibers are created by a main fiber, that is scheduled along with the others, in this case.

Practice​

Modify the launch parameter into boost::fibers::launch::post, compile and notice the differences. +The post parameter notifies the scheduler not to start the fibers imediately, but rather place them into an execution queue. +Their execution will start after the main fiber calls the join() function.

Barriers​

Follow the support/user-level-threads/yield_barrier.cc implementation. +It uses a barrier to achieve the same result as the previos implementation, that used post as the launch parameter.

Interaction Between Threads and Fibers​

As we mentioned before, multiple fibers can run on the same thread, and a scheduler is implemented on each thread. +By default, the scheduling algorithm is round_robin. +It runs the fibers, in the order of their creation, until they yield or finish their work. +If a fiber yields, it is placed at the back of the round-robin queue. +Using this scheduler, each thread only uses its fibers; +if one thread has more work to do than another, bad luck. +This may lead to starvation.

But there are other scheduler implementations, such as shared_work and work_stealing. +Follow the support/user-level-threads/threads_and_fibers.cc implementation. +It creates multiple fibers and threads, and uses the shared_work scheduler to balance the workload between the threads. +Each main fiber, from each thread, is suspended until all worker fibers have completed their work, using a condition variable.

cnd_count.wait( lk, [](){ return 0 == fiber_count; } );

The program also uses thread local storage and fiber local storage to store the ID of each thread / fiber.

Now change the shared_work scheduler into the work_stealing one. +It takes a parameter, the number of threads that will use that scheduler.

Compile, rerun and note the differences. +The work_stealing scheduler, as the name suggests, will "steal" fibers from other schedulers. +So, if the shared_work scheduler tried to balance the available work between the available threads, the work_stealing one will focus on having as many threads as possible on 100% workload. +Vary the number of threads and fibers, and the workload (maybe put each fibre to do some computational-intensive work), and observe the results.

C++ unique_lock​

unique_lock is a type of mutex that is unlocked automatically when the end of its scope is reached (end of function or bracket-pair).

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/arena/index.html b/17/Lab/Data/arena/index.html new file mode 100644 index 0000000000..66e5bb1db6 --- /dev/null +++ b/17/Lab/Data/arena/index.html @@ -0,0 +1,53 @@ + + + + + +Arena | Operating Systems + + + + +
+
Skip to main content

Arena

Challenge tasks

Memory Support​

Manual memory management (MMM) is one of the most difficult tasks. +Even experienced programmers make mistakes when tackling such a complicated endeavor. +As a consequence, the programming world has been migrating towards languages that offer automatic memory management (AMM). +AMM programming languages typically offer a garbage collector that tracks down the usage of objects and frees memory once no references exist to a given object. +As a consequence, garbage collected programming languages are easier to use and safer. +However, this comes with a cost: the garbage collector, in most cases, requires a significant amount of resources to run. +Therefore, for performance-critical systems, MMM is still the preferred solution.

A middle-ground between programming languages that have AMM (Java, Python, Swift, D) and those that do not (C, C++) is represented by those languages that do not have built-in AMM but offer the possibility to implement it as a library solution (C++, D). +Concretely, these languages offer lightweight library solutions to optimally track down the lifetime of an object. +This is done by using reference counted objects.

Reference Counting​

Reference counting is a technique of tracking the lifetime of an object by counting how many references to an object exist. +As long as at least one reference exists, the object cannot be destroyed. +Once no reference to a given object exists, it can be safely destroyed. +Reference counted is typically implemented by storing a count with the actual payload of the object. +Every time a new reference to the object is created, the reference count is incremented. +Every time a reference expires, the reference is decremented.

The operations that trigger a reference increment are:

  • initializing an object from another object.
  • assigning an object to another object.

The operations that trigger a reference decrement are:

  • the lifetime of an object expires

Modern programming languages offer the possibility to specify what code should be run in each of these situations, therefore enabling the implementation of referenced counted data structures. +As such, copy constructors may be used to automatically initialize an object from another object, assignment operators may be used to assign an object to another object and destructors may be used to destroy objects.

Operator overloading​

Navigate to the support/reference-counting directory. +Analyze the operators.d file. +A struct is defined that also implements 4 special functions: a constructor, a copy constructor, an assignment operator and a destructor. +Each of these special functions may be called automatically by the compiler:

  • the constructor is called automatically whenever an object is initialized with a field of a type that corresponds to the constructor parameter type.
  • the copy constructor is called automatically when an object is initialized from an object of the same type.
  • the assignment operator is called automatically when an object is assigned an object of the same type.
  • the destructor is called automatically whenever an object goes out of scope.

Note: the difference between initialization and assignment is that the initialization occurs when an object is being declared and occurs a single time (Obj o1 = 1), whereas assignement is decoupled from the declaration site and may occur multiple times (provided that the variable is mutable).

Compile and run the program in operators.d. +Notice how the different special functions are automatically called. +Considering the definition of Obj from the file operators.d, answer the following Quiz.

Practice​

Navigate to the support/reference-counting directory. +Analyze the refcount_skel.d. +A reference counted int array is implemented, however, some bits are missing. +Run the code, try to understand what happens.

The constructor allocates memory for the array, whereas the destructor deallocates it. +Compile and run the code. +Notice how the array's memory is automatically managed.

  1. Uncomment the following line in the main function (//test1()). +Run the code. +What happens? +Why?

  2. The reference counted array does not implement the copy constructor. +Comment the version(none) annotation for the copy constructor and implement the logic so that the reference counted array is correct. +When an object is initialized from another object, we need to appropriately set the fields and then increment the reference count. +Once you have completed this exercise, make sure the output is correct and that the reference counted array is not freed too early.

  3. Uncomment the following line in the main function (//test2()). +Run the code. +What happens? +Why? +Use GDB to find out.

  4. The reference counted array does not implement the assignment operator. +Comment the version(none) annotation for the assignment operator and implement the logic so that the reference counted array is correct. +When an object is assigned to another object, we need to first decrement the count for the object that is being assigned to, then fill the fields similarly to the copy constructor case and lastly increment the count for the assigned object. +After completing the exercise, make sure that the memory is properly managed.

  5. Play with your reference counted array and create different scenarios to test its limits.

Use-After-Free​

TODO

Code Injection​

TODO

Quiz​

TODO

DEP​

Practice​

TODO

Code Reuse​

TODO

Quiz​

TODO

Practice​

TODO

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/index.html b/17/Lab/Data/index.html new file mode 100644 index 0000000000..75c34da6c1 --- /dev/null +++ b/17/Lab/Data/index.html @@ -0,0 +1,16 @@ + + + + + +Data | Operating Systems + + + + +
+
Skip to main content

Data

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/investigate-memory/index.html b/17/Lab/Data/investigate-memory/index.html new file mode 100644 index 0000000000..a00633f894 --- /dev/null +++ b/17/Lab/Data/investigate-memory/index.html @@ -0,0 +1,50 @@ + + + + + +Investigate Memory Actions | Operating Systems + + + + +
+
Skip to main content

Investigate Memory Actions

Memory actions generally mean:

  • memory access: read, write or execute
  • memory allocation
  • memory deallocation

By far, the most important actions are allocation and deallocation. +Because, if not done right, these can get to memory loss and poor memory use.

Memory loss generally happens in the form of memory leaks.

Memory Leaks​

A memory leak occurs when we lose reference to a memory area. +That is, a pointer used to point to a memory area. +And then it's pointing to a new memory area and the old memory area is lost.

Enter the support/memory-leak/ folder. +It stores two files showing memory leaks:

  • one in C++: memory_leak.cpp
  • one in C: memory_leak_malloc

Let's build and run the two executables:

student@os:~/.../lab/support/memory-leak$ make
g++    -c -o memory_leak.o memory_leak.cpp
cc   memory_leak.o  -lstdc++ -o memory_leak
cc    -c -o memory_leak_malloc.o memory_leak_malloc.c
cc   memory_leak_malloc.o  -lstdc++ -o memory_leak_malloc

Running them yields similar output:

student@os:~/.../lab/support/memory-leak$ ./memory_leak
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS
student@os:~/.../lab/support/memory-leak$ ./memory_leak_malloc
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS

We investigate the memory leaks of the two programs by using Valgrind:

student@os:~/.../lab/support/memory-leak$ valgrind ./memory_leak
==22362== Memcheck, a memory error detector
==22362== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==22362== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==22362== Command: ./memory_leak
==22362==
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS
==22362==
==22362== HEAP SUMMARY:
==22362==     in use at exit: 72 bytes in 1 blocks
==22362==   total heap usage: 4 allocs, 3 frees, 73,872 bytes allocated
==22362==
==22362== LEAK SUMMARY:
==22362==    definitely lost: 72 bytes in 1 blocks
==22362==    indirectly lost: 0 bytes in 0 blocks
==22362==      possibly lost: 0 bytes in 0 blocks
==22362==    still reachable: 0 bytes in 0 blocks
==22362==         suppressed: 0 bytes in 0 blocks
==22362== Rerun with --leak-check=full to see details of leaked memory
==22362==
==22362== For counts of detected and suppressed errors, rerun with: -v
==22362== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

student@os:~/.../lab/support/memory-leak$ valgrind ./memory_leak_malloc
==22369== Memcheck, a memory error detector
==22369== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==22369== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==22369== Command: ./memory_leak_malloc
==22369==
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS
==22369==
==22369== HEAP SUMMARY:
==22369==     in use at exit: 148 bytes in 1 blocks
==22369==   total heap usage: 3 allocs, 2 frees, 1,320 bytes allocated
==22369==
==22369== LEAK SUMMARY:
==22369==    definitely lost: 148 bytes in 1 blocks
==22369==    indirectly lost: 0 bytes in 0 blocks
==22369==      possibly lost: 0 bytes in 0 blocks
==22369==    still reachable: 0 bytes in 0 blocks
==22369==         suppressed: 0 bytes in 0 blocks
==22369== Rerun with --leak-check=full to see details of leaked memory
==22369==
==22369== For counts of detected and suppressed errors, rerun with: -v
==22369== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

As we are doing allocations that are not freed, this results in memory leaks.

For malloc()-based programs, we can use mtrace() feature and mtrace command to verify proper allocations with malloc() and deallocations with free(). +We call mtrace() in the program (in memory_leak_malloc.c) to enable malloc() and free() checking.

To use mtrace() we define the MALLOC_TRACE environment variable. +We probably also require to preload the libc malloc debugging library, so we use LD_PRELOAD for that. +Note that the file path used for LD_PRELOAD may need to be updated, depending on your distribution:

student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/lib/x86_64-linux-gnu/libc_malloc_debug.so.0 MALLOC_TRACE=mem.trace ./memory_leak_malloc
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS

Subsequently, we use the mtrace tool to show information about the leaked data:

student@os:~/.../lab/support/memory-leak$ mtrace ./memory_leak_malloc mem.trace

Memory not freed:
-----------------
           Address     Size     Caller
0x000056506d8be6a0     0x94  at 0x56506c3777ec

The size (0x94) is the same value shown by Valgrind (148).

mtrace provides an outcome similar to Valgrind. +Valgrind is however more powerful: it works on different types of memory (not only those allocated with malloc()) and it doesn't require access to the source code (and the compiler phase).

Practice​

  1. Print the size of the Student class and the struct student structure to see if it equates to the leak shown by Valgrind.

  2. Solve the memory leaks in both programs. +Validate with Valgrind.

Quiz

Memory Actions (and Leaks) in Existing Programs​

We can use Valgrind to investigate existing programs in the system. +This tells us whether they possess memory leaks:

student@os:~/.../lab/support/memory-leak$ valgrind ls
==24669== Memcheck, a memory error detector
==24669== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24669== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==24669== Command: ls
==24669==
Makefile  memory_leak  memory_leak.cpp  memory_leak_malloc  memory_leak_malloc.c  memory_leak_malloc.o  memory_leak.o
==24669==
==24669== HEAP SUMMARY:
==24669==     in use at exit: 21,696 bytes in 14 blocks
==24669==   total heap usage: 51 allocs, 37 frees, 61,331 bytes allocated
==24669==
==24669== LEAK SUMMARY:
==24669==    definitely lost: 0 bytes in 0 blocks
==24669==    indirectly lost: 0 bytes in 0 blocks
==24669==      possibly lost: 0 bytes in 0 blocks
==24669==    still reachable: 21,696 bytes in 14 blocks
==24669==         suppressed: 0 bytes in 0 blocks
==24669== Rerun with --leak-check=full to see details of leaked memory
==24669==
==24669== For counts of detected and suppressed errors, rerun with: -v
==24669== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

student@os:~/.../lab/support/memory-leak$ valgrind ps
==24671== Memcheck, a memory error detector
==24671== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24671== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==24671== Command: ps
==24671==
  PID TTY          TIME CMD
24671 pts/22   00:00:00 memcheck-amd64-
26732 pts/22   00:00:01 bash
==24671==
==24671== HEAP SUMMARY:
==24671==     in use at exit: 264,929 bytes in 25 blocks
==24671==   total heap usage: 692 allocs, 667 frees, 334,268 bytes allocated
==24671==
==24671== LEAK SUMMARY:
==24671==    definitely lost: 0 bytes in 0 blocks
==24671==    indirectly lost: 0 bytes in 0 blocks
==24671==      possibly lost: 0 bytes in 0 blocks
==24671==    still reachable: 264,929 bytes in 25 blocks
==24671==         suppressed: 0 bytes in 0 blocks
==24671== Rerun with --leak-check=full to see details of leaked memory
==24671==
==24671== For counts of detected and suppressed errors, rerun with: -v
==24671== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

student@os:~/.../lab/support/memory-leak$ valgrind bash -c 'echo "ha"'
==24675== Memcheck, a memory error detector
==24675== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.
==24675== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info
==24675== Command: bash -c echo\ "ha"
==24675==
ha
==24675==
==24675== HEAP SUMMARY:
==24675==     in use at exit: 43,056 bytes in 672 blocks
==24675==   total heap usage: 774 allocs, 102 frees, 51,405 bytes allocated
==24675==
==24675== LEAK SUMMARY:
==24675==    definitely lost: 12 bytes in 1 blocks
==24675==    indirectly lost: 0 bytes in 0 blocks
==24675==      possibly lost: 0 bytes in 0 blocks
==24675==    still reachable: 43,044 bytes in 671 blocks
==24675==         suppressed: 0 bytes in 0 blocks
==24675== Rerun with --leak-check=full to see details of leaked memory
==24675==
==24675== For counts of detected and suppressed errors, rerun with: -v
==24675== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

We can see that ls and ps don't have memory leaks. +However, the shell (Bash) shows a memory leak of 12 bytes (on the test system). +This may be a false positive or the subject of an actual investigation.

Note that the still reachable section of the output refers to memory that wasn't freed, but still has pointers referring to it. +A true memory leak occurs when no pointers refer any memory area.

Practice​

  1. Investigate 2-3 other executables in the system using Valgrind.

  2. Use ltrace to list malloc() and free() calls made by the investigated system executables.

Note that, as explained in the Software Stack lab, on some systems, ltrace does not accurately show the output, due to now binding. +Fear not, you can always check the library calls with a more verbose and harder to parse ltrace command:

student@os:~$ ltrace -x "*"

Quiz

jemalloc​

jemalloc is a featureful allocator that is intended to replace the standard allocator in the standard C library (libc). +jemalloc provides replacements for the general malloc() and free() functions, and also provides a custom API targeted for performance tuning.

As documented, there are multiple ways to use jemalloc, the easiest of which is to use the LD_PRELOAD environment variable and preload the library and hook into malloc() and free() function calls.

First install jemalloc on our system. +On your typical Ubuntu / Debian-based system, use apt:

student@os:~/.../data/lab/content$ sudo apt -y install libjemalloc-dev

Note that this installs the distribution package, not the latest one (that may provide more features).

With this in place, we can use jemalloc against our pre-built executables or system executables (such as ls, ps). +We can test it against the executable files from support/memory-leak/:

student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./memory_leak_malloc
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS

jemalloc can use the MALLOC_CONF environment variable for a diverse set of configurations. +For example, by using stats_print:true we print out information regarding the use of the library functions:

student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF="stats_print:true" ./memory_leak_malloc
Andrei Popescu is 22 years old and likes Linux
Ioana David is 23 years old and likes macOS
___ Begin jemalloc statistics ___
Version: 3.6.0-11
Assertions disabled
Run-time option settings:
  opt.abort: false
  opt.lg_chunk: 22
  opt.dss: "secondary"
  opt.narenas: 32
  opt.lg_dirty_mult: 3
  opt.stats_print: true
  opt.junk: false
  opt.quarantine: 0
  opt.redzone: false
[...]
dirty pages: 26:0 active:dirty, 0 sweeps, 0 madvises, 0 purged
            allocated      nmalloc      ndalloc    nrequests
small:          72672          114            0            3
large:          32768            1            0            1
total:         105440          115            0            4
active:        106496
mapped:       4194304
[...]

jemalloc doesn't work against system executables using preloading, likely because of security options disabling the use of the library:

student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF="stats_print:true" /bin/ls
Makefile  memory_leak  memory_leak.cpp  memory_leak_malloc  memory_leak_malloc.c  memory_leak_malloc.o  memory_leak.o

student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF="stats_print:true" /bin/ps
  PID TTY          TIME CMD
 1581 pts/22   00:00:00 ps
26732 pts/22   00:00:01 bash

malloc in Musl​

Each libc (or memory allocator such as jemalloc) uses their own implementation of malloc(), free() and other functions. +Musl libc is a lightweight standard C library that provides compatible features with the more heavyweights GNU libc.

Take a look through implementation of malloc() and free() in Musl libc. +See all three implementations for malloc():

  • the one in lite_malloc.c
  • the one in mallocng/malloc.c
  • the one in oldmalloc/malloc

See also the implementation of free(). +And the implementation of calloc().

You needn't spend too much time browsing the implementation of these functions, just having a broad understanding of how they work.

App Investigation: Deluge​

Deluge is a Bittorrent client written in Python.

We want to locate places that allocate memory in Deluge (in Python). +This generally means locating instantiation of classes.

Let's clone the source code:

student@os:~/.../data/lab/support$ git clone https://github.com/deluge-torrent/deluge
Cloning into 'deluge'...
[...]

student@os:~/.../data/lab/support$ cd deluge/

student@os:~/.../lab/support/deluge$ ls
AUTHORS       deluge      docs             gen_web_gettext.py  MANIFEST.in       msgfmt.py  pyproject.toml  requirements-dev.txt    requirements.txt  setup.py  version.py
CHANGELOG.md  DEPENDS.md  generate_pot.py  LICENSE             minify_web_js.py  packaging  README.md       requirements-tests.txt  setup.cfg         tox.ini

And enter the deluge/core/ subdirectory:

student@os:~/.../lab/support/deluge$ cd deluge/core/

student@os:~/.../deluge/deluge/core$ ls
alertmanager.py  core.py          daemon.py        filtermanager.py  pluginmanager.py       rpcserver.py       torrent.py
authmanager.py   daemon_entry.py  eventmanager.py  __init__.py       preferencesmanager.py  torrentmanager.py

Most files in the subdirectory have a class defined. +We can search for instantiations of that class using grep:

student@os:~/.../deluge/deluge/core$ grep -rn 'Torrent('
torrentmanager.py:644:        torrent = Torrent(handle, options, state, filename, magne

student@os:~/.../deluge/deluge/core$ grep -rn 'TorrentManager('
core.py:139:        self.torrentmanager = TorrentManager()
torrentmanager.py:135:class TorrentManager(component.Component):

This gives us an overview of when memory is allocated in Deluge / Python.

Practice​

  1. Investigate the lines shown to contain instantiations of classes. +Explore the source code and understand their placements in the source code.

  2. Find out other classes and search for their instantiation in the source code.

App Investigation: Servo​

Servo is a browser engine written in Rust that provides reusable components to implement web standards.

We do not clone the repository, since it's very large.

We find information about allocator used, by accessing the components/allocator/ in its source code. +In Cargo.toml we see that it requires jemalloc for non-Windows implementations and the standard Windows API (called heapapi) for Windows:

[...]
[lib]
path = "lib.rs"

[target.'cfg(not(windows))'.dependencies]
jemalloc-sys = { version = "0.3.2" }

[target.'cfg(windows)'.dependencies]
winapi = { version = "0.3", features = ["heapapi"] }
ust: https://github.com/servo/servo

In lib.rs, in GlobalAlloc:alloc() we see it is using the mallocx custom function from jemalloc(). +See the initialization of ffi.

See the use of the allocator in the Cargo.toml file in the net component. +Search for the alloc string.

Practice​

  1. Look for uses of the allocator in other components of Servo.

Investigation: Alocator in the D Programming Language​

Phobos is the standard library that comes with the D programming language compiler.

Let's clone the source code:

student@os:~/.../data/lab/support$ git clone https://github.com/dlang/phobos
[...]

student@os:~/.../data/lab/support$ cd phobos/

student@os:~/.../lab/support/phobos$ ls
azure-pipelines.yml  changelog  CODEOWNERS  CONTRIBUTING.md  dub.sdl  etc  index.dd  LICENSE_1_0.txt  posix.mak  project.ddoc  README.md  std  test  unittest.d  win32.mak  win64.mak

And enter std/experimental/allocator/ to browse information about the allocator:

student@os:~/.../lab/support/phobos$ cd std/experimental/allocator/

student@os:~/.../std/experimental/allocator$ ls
building_blocks  common.d  gc_allocator.d  mallocator.d  mmap_allocator.d  package.d  showcase.d  typed.d

We then do a search of the allocate( string to find instances of allocation calls:

student@os:~/.../std/experimental/allocator$ grep -r 'allocate('
[...]

We see that there are definitions of the function (as expected) as part of ...allocator files: mallocator.d, gc_allocator.d, mmap_allocator.d. +Browse the functions and look for implementations of the allocate() function.

Practice​

  1. Do a similar search and then source code browsing for the deallocate() function.

App Investigation: Git​

Git is among the most used source code management system, powering development infrastructures such as GitHub, GitLab and Bitbucket.

Let's clone the repository. +Note that it is about 200MB large:

student@os:~/.../data/lab/support$ git clone https://github.com/git/git
[...]

student@os:~/.../data/lab/support$ cd git/

We look of uses of malloc():

student@os:~/.../lab/support/git$ grep -r 'malloc(' .

We see there are multiple calls to the xmalloc() function, which is likely a wrapper for malloc(). +We search for the definition of xmalloc():

student@os:~/.../lab/support/git$ grep -rn 'xmalloc(' . | grep -v ';'
./commit.c:188:                 graft = xmalloc(st_add(sizeof(*graft),
./add-interactive.c:157:        list->sorted.items = xmalloc(st_mult(sizeof(*list->sorted.items),
./Documentation/RelNotes/2.24.0.txt:91: * xmalloc() used to have a mechanism to ditch memory and address
./Documentation/RelNotes/2.24.0.txt:210:   xmalloc() wrapper, as the rest of the system, for consistency.
./Documentation/RelNotes/2.34.0.txt:230: * mmap() imitation used to call xmalloc() that dies upon malloc()
./Documentation/RelNotes/2.33.1.txt:44: * mmap() imitation used to call xmalloc() that dies upon malloc()
./diffcore-delta.c:56:  new_spanhash = xmalloc(st_add(sizeof(*orig),
./diffcore-delta.c:135: hash = xmalloc(st_add(sizeof(*hash),
./kwset.c:41:/* adapter for `xmalloc()`, which takes `size_t`, not `long` */
./builtin/fast-import.c:461:    b = xmalloc(sizeof(struct object_entry_pool)
./hashmap.h:311: * your structure was allocated with xmalloc(), you can just free(3) it,
./xdiff/xdiff.h:122:#define xdl_malloc(x) xmalloc(x)
./wrapper.c:45:static void *do_xmalloc(size_t size, int gentle)
./wrapper.c:70:void *xmalloc(size_t size)
./contrib/credential/wincred/git-credential-wincred.c:26:static void *xmalloc(size_t size)
Binary file ./.git/objects/pack/pack-c587b9f11a82bc4d49848d74132e60ea4dbeb177.pack matches
./git-compat-util.h:1046:# define xalloca(size)      (xmalloc(size))
./git-compat-util.h:1086:#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))
./read-cache.c:3768:    ieot = xmalloc(sizeof(struct index_entry_offset_table)

Line ./wrapper.c:70 is the one with the definition of the xmalloc() function. +It makes a call of the do_xmalloc() function, that makes extra checks. +Also, if the XMALLOC_POISON macro is defined, all the allocated data is overwritten with a "poison" value (0xA5). +This is useful for early detection of memory-related issues, although, evidently, it adds overhead.

We can look for parts of the source code with the largest number of uses of xmalloc():

student@os:~/.../lab/support/git$ grep -rc 'xmalloc(' . | grep -v ':0' | sort -n -t ':' -k 20
[...]
./compat/mingw.c:6
./submodule-config.c:6
./merge-recursive.c:7

We can look into the merge-recursive.c file for uses of the xmalloc() function.

Practice​

  1. Do the same actions as above for the mmap() and xmmap() function calls.

    Note that these are not memory allocation calls, since a valid fd file argument is passed. +These are file mapping calls, that we will talk more as part of the I/O chapter.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/memory-security/index.html b/17/Lab/Data/memory-security/index.html new file mode 100644 index 0000000000..9a251b3221 --- /dev/null +++ b/17/Lab/Data/memory-security/index.html @@ -0,0 +1,65 @@ + + + + + +Memory Security | Operating Systems + + + + +
+
Skip to main content

Memory Security

Memory security is one of the most important aspects in today's computer systems. +Its main objectives are to avoid the development of vulnerable code and prevent attackers from exploiting existing memory vulnerabilities. +Memory protection techniques are implemented at all levels of memory abstraction: hardware, operating system and programming language. +In addition, third-party software tools may be used to statically and dynamically examine a program to identify vulnerabilities when working with memory.

In this section, we will focus on the main memory vulnerabilities and how to exploit them. +We will use the C programming language for presentation purposes, however, these examples may be reproduced in any language that implements arrays as pointers without bounds and allows the liberal use of pointers.

Wild Pointer Arithmetic Info Leak​

In C, once a pointer has been initialized with a valid address it can potentially access any location of the process address space. +This is problematic for situations where the binary contains sensitive information. +Take for example a server that authenticates users: if the password is stored in some buffer, then if a pointer is wrongfully used, it can end up leaking the password.

Navigate to the support/memory-security/ directory. +Open and analyze the buff_leak.c file.

Answer this quiz

Practice​

The pointer p points to the stack, however, we can modify any variable that is declared in the program through p. +All we need to know is the offset of the other memory locations that we wish to access. +Run the program and try to input the correct offsets to modify variables from different regions of our program. +Once a correct offset is given as input, the program will output a validation message.

Note that adding or subtracting user provided values to pointers enables an attacker to observe a program's entire memory!

Buffer Overflow Info Leak​

Since arrays decay to pointers in C, the same effect may be obtained by using them.

Navigate to the support/memory-security/ directory. +Open and analyze the array_leak.c file.

Compile and run the program. +Can you extract any information from the output? +Can you identify the return address of the main function?

Note: You can use objdump -d -M intel array_leak to check the stack layout. +Note: Depending on the environment the layout may differ.

Next, open and analyze the string_leak.c file. +Compile and run the program. +To better understand the output, use xxd to interpret the output as hexadecimal bytes:

student@os:~/.../lab/support/memory-security$ ./string_leak | xxd

Note: 73 and 6f are the ascii values of s and o

Answer this quiz

Practice​

In file string_leak.c replace the usage of memcpy with strcpy. +Do not modify anything else (including the size of the buffer). +As the name suggests, strcpy() is specialized for string copies, therefore we don need to specify how much we want to copy. +What is the result? +Is the result correct? +Explain the result.

Answer this quiz

Buffer Overflow Overwrite​

Up until this point we have seen how we can exploit a buffer to leak information. +However, this vulnerability may be used most of the time to overwrite data.

Take for example the code in support/memory-security/bo_write.c. +Compile and run the code. +What happens? +Why?

Answer this quiz

Practice​

Open the support/memory-security/bo_write_practice.c file. +Analyze the code, then compile it and run it.

  1. Try to find an input that alters the control flow of the program so that "Comm-link online" is printed. +You are not allowed to modify the source file.

  2. Try to find an input that alters the control flow of the program so that "Channel open." is printed. +You are not allowed to modify the source file.

Note: Addresses are 8 bytes long on 64 bit machines.

  1. Can you think of a different input that results in printing "Comm-link online"?

ASLR​

Address Space Layout Randomization (ASLR) is a protection technique employed by operating systems to make it harder for attackers to identify code locations to jump to. +Essentially, every program section (including shared library code) is mapped at a random virtual address every time the program is run. +That way, it is harder for the attacker to exploit a buffer overflow: the address of a potential code target is different with each run.

To enable randomization, there are 2 steps that need to be taken:

  1. Enable ASLR on the operating system. +This enables the operating system to map library code at different virtual addresses in a program.

  2. Compile our code as Position Independent Code (PIC). +This instructs the compiler to generate code such that it does not use absolute addresses when calling functions or accessing variables. +As a consequence, the code may be loaded at any given address in memory.

On most Linux systems, ASLR is enabled by default. +Modern compilers generate position independent code by default.

Practice​

Use the Makefile.aslr file to compile the support/memory-security/aslr.c file:

student@os:~/.../lab/support/memory-security$ make -f Makefile.aslr

By default, ASLR and PIC are enabled. +Observe the results. +Next, we disable ASLR:

student@os:~/.../lab/support/memory-security$ echo 0 | sudo tee /proc/sys/kernel/randomize_va_space

Even though the code is compiled with PIC, both library and user functions have the same address between runs. +Re-enable ASLR:

student@os:~/.../lab/support/memory-security$ echo 2 | sudo tee /proc/sys/kernel/randomize_va_space

Disable PIC by uncommenting the -fno-PIC and LDFLAGS lines.

We observe that for randomization to work, we need to instruct the OS to randomize the program sections and the compiler to generate code that is position independent.

Answer this quiz

Stack Protector​

As observed in the previous quiz, ASLR prevents an attacker (most of the times) from discovering a reliable address where to jump to, however, it does not prevent the occurrence of a buffer overflow. +An effective strategy to protect against buffer overflow is represented by the stack protector (or stack canary). +Between the end of the buffer and the return address (below the saved base pointer rbp), a random value is placed on the stack. +Before the function returns, the value is checked: if the initial value was modified, then an exception is issued and the program is aborted.

Stack canaries are enabled by default, however, for learning purposes we have disabled it by passing -fno-stack-protector to the compiler.

Practice​

Comment the -fno-stack-protector switch from the support/memory-security/Makefile, recompile and run the bo_practice_write executable. +Examine the binary with objdump and identify the instructions that set and test the canary. +Observe what happens when a buffer overflow occurs.

Answer this quiz

Bypassing the Stack Protector​

The stack protector is generally placed immediately after the old rbp. +With this information we can craft various exploits to bypass it:

  • Leak the canary and do not modify it. +Reading the canary, as long as we do not modify it is not going to cause any disturbances. +If we have the canary, we can just include it in our payload and hapily overwrite the return address.

  • If we have access to a pointer that we can modify appropriately, we can just jump over the canary and directly modify the return address.

  • In case we have multiple buffers defined in the same stack frame, we can simply overwrite data in a buffer that is placed above the buffer we are exploiting without the stack protector intervention.

Practice​

Inspect the support/memory-security/stack_protector.c source file. +Compile the program and examine the object code. +Try to identify the canary value. +Using the addr variable, write 2 scanf instructions: one that overwrites the canary with the corect value and one that overwrites the return address with the address of function pawned. +In case of a successful exploit a video will be offered as reward.

Answer this quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/overview/index.html b/17/Lab/Data/overview/index.html new file mode 100644 index 0000000000..620dacf746 --- /dev/null +++ b/17/Lab/Data/overview/index.html @@ -0,0 +1,22 @@ + + + + + +Data | Operating Systems + + + + +
+
Skip to main content

Data

Data represents information that is to be processed to produce a final result or more data. +Computers store, retrieve and compute data. +This process involves 4 entities: the programmer, the programming language, the operating system and the hardware.

From a programmer's perspective, data is represented by the variables. +These are declared and utilized depending on the rules of the programming language that is employed. +The programming language analyzes the use of these variables and outputs code that uses an interface provided by the operating system. +This interface offers the possibility to allocate/deallocate different variables in certain memory regions. +Next, the operating system manages the execution of the program and provides the actual physical addresses that are used to interact with the data.

Moreover, the operating system governs the competing access of multiple programs to memory, ensuring that a program does not have access to a different program's memory.

Contents​

  1. Working with Memory
  2. Process Memory
  3. Investigate Memory
  4. Memory Security
  5. Arena
+ + + + \ No newline at end of file diff --git a/17/Lab/Data/process-memory/index.html b/17/Lab/Data/process-memory/index.html new file mode 100644 index 0000000000..01f0e17d5c --- /dev/null +++ b/17/Lab/Data/process-memory/index.html @@ -0,0 +1,73 @@ + + + + + +Process Memory | Operating Systems + + + + +
+
Skip to main content

Process Memory

Memory Regions​

To better manage a program's memory, the operating systems creates an address space for each process. +The address space is compartmentalized in multiple areas, each with its own role. +Memory addresses use different permissions to decide what actions are allowed.

Let's investigate the memory areas of a given process. +We use pmap to see the memory layout of a running process. +The command below shows the memory layout of the current shell process:

student@os:~$ pmap -p $$
1127:   /bin/bash
000055fb4d77d000   1040K r-x-- /bin/bash
000055fb4da80000     16K r---- /bin/bash
000055fb4da84000     36K rw--- /bin/bash
000055fb4da8d000     40K rw---   [ anon ]
000055fb4e9bb000   1604K rw---   [ anon ]
00007f8fcf670000   4480K r---- /usr/lib/locale/locale-archive
00007f8fcfad0000     44K r-x-- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfadb000   2044K ----- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfcda000      4K r---- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfcdb000      4K rw--- /lib/x86_64-linux-gnu/libnss_files-2.27.so
00007f8fcfcdc000     24K rw---   [ anon ]
00007f8fcfce2000     92K r-x-- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfcf9000   2044K ----- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfef8000      4K r---- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfef9000      4K rw--- /lib/x86_64-linux-gnu/libnsl-2.27.so
00007f8fcfefa000      8K rw---   [ anon ]
00007f8fcfefc000     44K r-x-- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fcff07000   2044K ----- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fd0106000      4K r---- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fd0107000      4K rw--- /lib/x86_64-linux-gnu/libnss_nis-2.27.so
00007f8fd0108000     32K r-x-- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0110000   2048K ----- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0310000      4K r---- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0311000      4K rw--- /lib/x86_64-linux-gnu/libnss_compat-2.27.so
00007f8fd0312000   1948K r-x-- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd04f9000   2048K ----- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd06f9000     16K r---- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd06fd000      8K rw--- /lib/x86_64-linux-gnu/libc-2.27.so
00007f8fd06ff000     16K rw---   [ anon ]
00007f8fd0703000     12K r-x-- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0706000   2044K ----- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0905000      4K r---- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0906000      4K rw--- /lib/x86_64-linux-gnu/libdl-2.27.so
00007f8fd0907000    148K r-x-- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd092c000   2048K ----- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd0b2c000     16K r---- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd0b30000      4K rw--- /lib/x86_64-linux-gnu/libtinfo.so.5.9
00007f8fd0b31000    164K r-x-- /lib/x86_64-linux-gnu/ld-2.27.so
00007f8fd0d24000     20K rw---   [ anon ]
00007f8fd0d53000     28K r--s- /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
00007f8fd0d5a000      4K r---- /lib/x86_64-linux-gnu/ld-2.27.so
00007f8fd0d5b000      4K rw--- /lib/x86_64-linux-gnu/ld-2.27.so
00007f8fd0d5c000      4K rw---   [ anon ]
00007ffff002f000    132K rw---   [ stack ]
00007ffff00c5000     12K r----   [ anon ]
00007ffff00c8000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total            24364K

Information will differ among different systems.

See the different regions:

  • the first region, with r-x permissions is the .text (code) area
  • the second region, with r-- premissions is the .rodata area
  • the third region, with rw- permissions is the .data area, for initialized global variables
  • the fourth region, with rw- permissions is the .bss area
  • the fifth region, with the rw- permissions is the dynamic data memory area, also known as heap
  • there are multiple dynamic libraries mapped in the virtual address space of the process, each library with their own regions
  • there is a [stack] memory region, with rw- permissions

pmap also shows the total amount of virtual memory available to the process (24364K), as a total of the sizes of the regions. +Note that this is virtual memory, not actual physical memory used by the process. +For the process investigated above (with the 1127 pid) we could use the command below to show the total virtual size and physical size (also called resident set size):

student@os:~$ ps -o pid,rss,vsz -p $$
  PID   RSS    VSZ
 1127  1968  24364

The resident size is 1968K, much smaller than the virtual size.

Note how each region has a size multiple of 4K, this has to do with the memory granularity. +The operating system allocates memory in chunks of a predefined size (in our case 4K) called pages.

Quiz

Practice​

Enter the support/memory-areas/ directory. +We investigate other programs.

  1. The hello.c program prints out a message and then sleeps. +Build it:

    student@os:~/.../lab/support/memory-areas$ make

    then run it (it will block):

    student@os:~/.../lab/support/memory-areas$ ./hello
    Hello, world!

    In another terminal, use the command below to show the memory areas of the process:

    student@os:~/.../lab/support/memory-areas$ pmap $(pidof hello)
    8220:   ./hello
    000055c0bef4b000      8K r-x-- hello
    000055c0bf14c000      4K r---- hello
    000055c0bf14d000      4K rw--- hello
    000055c0bf454000    132K rw---   [ anon ]
    00007f2a9e4a5000   1948K r-x-- libc-2.27.so
    00007f2a9e68c000   2048K ----- libc-2.27.so
    00007f2a9e88c000     16K r---- libc-2.27.so
    00007f2a9e890000      8K rw--- libc-2.27.so
    00007f2a9e892000     16K rw---   [ anon ]
    00007f2a9e896000    164K r-x-- ld-2.27.so
    00007f2a9ea8c000      8K rw---   [ anon ]
    00007f2a9eabf000      4K r---- ld-2.27.so
    00007f2a9eac0000      4K rw--- ld-2.27.so
    00007f2a9eac1000      4K rw---   [ anon ]
    00007ffee6471000    132K rw---   [ stack ]
    00007ffee6596000     12K r----   [ anon ]
    00007ffee6599000      4K r-x--   [ anon ]
    ffffffffff600000      4K --x--   [ anon ]
     total             4520K

    The output is similar, but with fewer dynamic libraries than bash, since they are not used by the program.

  2. Make a program in another language of your choice that prints Hello, world! and sleeps and investigate it with pmap. +Note that in the case of interpreted languages (Python, Lua, Perl, Ruby, PHP, JavaScript etc.) you have to investigate the interpreter process.

Memory Layout of Statically-Linked and Dynamically-Linked Executables​

We want to see the difference in memory layout between the statically-linked and dynamically-linked executables.

Enter the support/static-dynamic/ directory and build the statically-linked and dynamically-linked executables hello-static and hello-dynamic:

student@os:~/.../lab/support/static-dynamic$ make

Now, by running the two programs and inspecting them with pmap on another terminal, we get the output:

student@os:~/.../lab/support/static-dynamic$ pmap $(pidof hello-static)
9714:   ./hello-static
0000000000400000    876K r-x-- hello-static
00000000006db000     24K rw--- hello-static
00000000006e1000      4K rw---   [ anon ]
00000000017b5000    140K rw---   [ anon ]
00007ffc6f1d6000    132K rw---   [ stack ]
00007ffc6f1f9000     12K r----   [ anon ]
00007ffc6f1fc000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             1196K

student@os:~/.../lab/support/static-dynamic$ pmap $(pidof hello-dynamic)
9753:   ./hello-dynamic
00005566e757f000      8K r-x-- hello-dynamic
00005566e7780000      4K r---- hello-dynamic
00005566e7781000      4K rw--- hello-dynamic
00005566e8894000    132K rw---   [ anon ]
00007fd434eb8000   1948K r-x-- libc-2.27.so
00007fd43509f000   2048K ----- libc-2.27.so
00007fd43529f000     16K r---- libc-2.27.so
00007fd4352a3000      8K rw--- libc-2.27.so
00007fd4352a5000     16K rw---   [ anon ]
00007fd4352a9000    164K r-x-- ld-2.27.so
00007fd43549f000      8K rw---   [ anon ]
00007fd4354d2000      4K r---- ld-2.27.so
00007fd4354d3000      4K rw--- ld-2.27.so
00007fd4354d4000      4K rw---   [ anon ]
00007ffe497ba000    132K rw---   [ stack ]
00007ffe497e3000     12K r----   [ anon ]
00007ffe497e6000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4520K

For the static executable, we can see there are no areas for dynamic libraries. +And the .rodata section has been coalesced in the .text area.

We can see the size of each section in the two executables by using the size command:

student@os:~/.../lab/support/static-dynamic$ size hello-static
text    data     bss     dec     hex filename
893333   20996    7128  921457   e0f71 hello-static

student@os:~/.../lab/support/static-dynamic$ size hello-dynamic
text    data     bss     dec     hex filename
4598     736     824    6158    180e hello-dynamic

Quiz​

Based on the information above, answer this quiz.

Practice​

  1. Let's investigate another static executable / process.

    If not already installed, install the busybox-static package on your system. +On Debian/Ubuntu systems, use:

    student@os:~$ sudo apt install busybox-static

    Start a process using:

    student@os:~$ busybox sleep 1000

    Investigate the process using pmap and the executable using size.

Modifying Memory Region Size​

We want to observe the update in size of memory regions for different instructions used in a program.

Enter the support/modify-areas/ directory. +Browse the contents of the hello.c file; +it is an update to the hello.c file in the memory-areas/ directory. +Build the executable:

student@os:~/.../lab/support/modify-areas$ make

Use size to view the difference between the new executable and the one in the memory-areas/ directory:

student@os:~/.../lab/support/modify-areas$ size hello
   text    data     bss     dec     hex filename
  13131   17128   33592   63851    f96b hello

student@os:~/.../lab/support/modify-areas$ size ../memory-areas/hello
   text    data     bss     dec     hex filename
   4598     736     824    6158    180e ../memory-areas/hello

Explain the differences.

Then use the pmap to watch the memory areas of the resulting processes from the two different executables. +We will see something like this for the new executable:

student@os:~/.../lab/support/modify-areas$ pmap $(pidof hello)
18254:   ./hello
000055beff4d0000     16K r-x-- hello
000055beff6d3000      4K r---- hello
000055beff6d4000     20K rw--- hello
000055beff6d9000     32K rw---   [ anon ]
000055beffb99000    324K rw---   [ anon ]
00007f7b6c2e6000   1948K r-x-- libc-2.27.so
00007f7b6c4cd000   2048K ----- libc-2.27.so
00007f7b6c6cd000     16K r---- libc-2.27.so
00007f7b6c6d1000      8K rw--- libc-2.27.so
00007f7b6c6d3000     16K rw---   [ anon ]
00007f7b6c6d7000    164K r-x-- ld-2.27.so
00007f7b6c8cd000      8K rw---   [ anon ]
00007f7b6c900000      4K r---- ld-2.27.so
00007f7b6c901000      4K rw--- ld-2.27.so
00007f7b6c902000      4K rw---   [ anon ]
00007ffe2b196000    204K rw---   [ stack ]
00007ffe2b1d8000     12K r----   [ anon ]
00007ffe2b1db000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4840K

We notice the size increase of text, data, bss, heap and stack sections.

Practice​

  1. Comment out different parts of the hello.c program to notice differences in only specific areas (text, data, bss, heap, stack).

  2. Use a different argument (order) for the call to the alloc_stack() function. +See how it affects the stack size during runtime (investigate with pmap).

  3. Do a static build of hello.c and check the size of the memory areas, both statically and dynamically.

  4. The extend_mem_area.py Python script allocates a new string at each step by merging the two previous versions. +Start the program and investigate the resulting process at each allocation step. +Notice which memory area is updated and explain why.

Quiz

Allocating and Deallocating Memory​

Memory areas in a process address space are static or dynamic. +Static memory areas are known at the beginning of process lifetime (i.e. at load-time), while dynamic memory areas are managed at runtime.

.text, .rodata, .data, .bss are allocated at load-time and have a predefined size. +The stack and the heap and memory mappings are allocated at runtime and have a variable size. +For those, we say we use runtime allocation and deallocation.

Memory allocation is implicit for the stack and explicit for the heap. +That is, we don't make a particular call to allocate data on the stack; +the compiler generates the code that the operating system uses to increase the stack when required. +For the heap, we use the malloc() and free() calls to explicitly allocate and deallocate memory.

Omitting to deallocate memory results in memory leaks that hurt the resource use in the system. +Because of this, some language runtimes employ a garbage collector that automatically frees unused memory areas. +More than that, some languages (think of Python) provide no explicit means to allocate memory: you just define and use data.

Let's enter the support/alloc_size/ directory. +Browse the alloc_size.c file. +Build it:

student@os:~/.../lab/support/alloc_size$ make

Now see the update in the process layout, by running the program in one console:

student@os:~/.../lab/support/alloc_size$ ./alloc_size
Press key to allocate ...
[...]

And investigating it with pmap on another console:

student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)
21107:   ./alloc_size
000055de9d173000      8K r-x-- alloc_size
000055de9d374000      4K r---- alloc_size
000055de9d375000      4K rw--- alloc_size
000055de9deea000    132K rw---   [ anon ]
00007f1ea4fd4000   1948K r-x-- libc-2.27.so
00007f1ea51bb000   2048K ----- libc-2.27.so
00007f1ea53bb000     16K r---- libc-2.27.so
00007f1ea53bf000      8K rw--- libc-2.27.so
00007f1ea53c1000     16K rw---   [ anon ]
00007f1ea53c5000    164K r-x-- ld-2.27.so
00007f1ea55bb000      8K rw---   [ anon ]
00007f1ea55ee000      4K r---- ld-2.27.so
00007f1ea55ef000      4K rw--- ld-2.27.so
00007f1ea55f0000      4K rw---   [ anon ]
00007ffcf28e9000    132K rw---   [ stack ]
00007ffcf29be000     12K r----   [ anon ]
00007ffcf29c1000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4520K

student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)
21107:   ./alloc_size
000055de9d173000      8K r-x-- alloc_size
000055de9d374000      4K r---- alloc_size
000055de9d375000      4K rw--- alloc_size
000055de9deea000    452K rw---   [ anon ]
00007f1ea4fd4000   1948K r-x-- libc-2.27.so
00007f1ea51bb000   2048K ----- libc-2.27.so
00007f1ea53bb000     16K r---- libc-2.27.so
00007f1ea53bf000      8K rw--- libc-2.27.so
00007f1ea53c1000     16K rw---   [ anon ]
00007f1ea53c5000    164K r-x-- ld-2.27.so
00007f1ea55bb000      8K rw---   [ anon ]
00007f1ea55ee000      4K r---- ld-2.27.so
00007f1ea55ef000      4K rw--- ld-2.27.so
00007f1ea55f0000      4K rw---   [ anon ]
00007ffcf28e9000    132K rw---   [ stack ]
00007ffcf29be000     12K r----   [ anon ]
00007ffcf29c1000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4840K

student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)
21107:   ./alloc_size
000055de9d173000      8K r-x-- alloc_size
000055de9d374000      4K r---- alloc_size
000055de9d375000      4K rw--- alloc_size
000055de9deea000    420K rw---   [ anon ]
00007f1ea4fd4000   1948K r-x-- libc-2.27.so
00007f1ea51bb000   2048K ----- libc-2.27.so
00007f1ea53bb000     16K r---- libc-2.27.so
00007f1ea53bf000      8K rw--- libc-2.27.so
00007f1ea53c1000     16K rw---   [ anon ]
00007f1ea53c5000    164K r-x-- ld-2.27.so
00007f1ea55bb000      8K rw---   [ anon ]
00007f1ea55ee000      4K r---- ld-2.27.so
00007f1ea55ef000      4K rw--- ld-2.27.so
00007f1ea55f0000      4K rw---   [ anon ]
00007ffcf28e9000    132K rw---   [ stack ]
00007ffcf29be000     12K r----   [ anon ]
00007ffcf29c1000      4K r-x--   [ anon ]
ffffffffff600000      4K --x--   [ anon ]
 total             4808K

The three runs above of the pmap command occur before the allocation, after allocation and before deallocation and after deallocation. +Notice the update toe the 4th section, the heap.

Now, let's see what happens behind the scenes. +Run the executable under ltrace and strace:

student@os:~/.../lab/support/alloc_size$ ltrace ./alloc_size
malloc(32768)                                                                                                    = 0x55e33f490b10
printf("New allocation at %p\n", 0x55e33f490b10New allocation at 0x55e33f490b10
)                                                                 = 33
[...]
free(0x55e33f490b10)                                                                                             = <void>
[...]

student@os:~/.../lab/support/alloc_size$ strace ./alloc_size
[...]
write(1, "New allocation at 0x55ab98acfaf0"..., 33New allocation at 0x55ab98acfaf0
) = 33
write(1, "New allocation at 0x55ab98ad7b00"..., 33New allocation at 0x55ab98ad7b00
) = 33
brk(0x55ab98b08000)                     = 0x55ab98b08000
write(1, "New allocation at 0x55ab98adfb10"..., 33New allocation at 0x55ab98adfb10
) = 33
write(1, "Press key to deallocate ...", 27Press key to deallocate ...) = 27
read(0,
"\n", 1024)                     = 1
brk(0x55ab98b00000)                     = 0x55ab98b00000
[...]

The resulting output above shows us the following:

  • malloc() and free() library calls both map to the brk syscall, a syscall that updates the end of the heap (called program break).
  • Multiple malloc() calls map to a single brk syscall for efficiency. +brk is called to preallocate a larger chunk of memory that malloc will then use.

Update the ALLOC_SIZE_KB macro in the alloc_size.c file to 256. +Rebuild the program and rerun it under ltrace and strace:

student@os:~/.../lab/support/alloc_size$ ltrace ./alloc_size
[...]
malloc(262144)                                                                                                   = 0x7f4c016a9010
[...]
free(0x7f4c016a9010)                                                                                             = <void>
[...]

student@os:~/.../lab/support/alloc_size$ strace ./alloc_size
[...]
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feee19f2000
write(1, "New allocation at 0x7feee19f2010"..., 33New allocation at 0x7feee19f2010
) = 33
mmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feee19b1000
write(1, "New allocation at 0x7feee19b1010"..., 33New allocation at 0x7feee19b1010
) = 33
write(1, "Press key to deallocate ...", 27Press key to deallocate ...) = 27
read(0,
"\n", 1024)                     = 1
munmap(0x7feee19b1000, 266240)          = 0
[...]

For the new allocation size, notice that the remarks above don't hold:

  • malloc() now invokes the mmap syscall, while free() invokes the munmap syscall.
  • Each malloc() calls results in a separate mmap syscall.

This is a behavior of the malloc() in libc, documented in the manual page. +A variable MALLOC_THRESHOLD holds the size after which mmap is used, instead of brk. +This is based on a heuristic of using the heap or some other area in the process address space.

Practice​

  1. Use pmap to analyze the process address space for ALLOC_SIZE_KB initialized to 256. +Notice the new memory areas and the difference between the use of mmap syscall and brk syscall.

  2. Use valgrind on the resulting executable, and notice there are memory leaks. +They are quite obvious due to the lack of proper freeing. +Solve the leaks.

  3. Use valgrind on different executables in the system (in /bin/, /usr/bin/) and see if they have memory leaks.

Quiz

Quiz

Memory Mapping​

The mmap syscall is used to allocate memory as anonymous mapping, that is reserving memory in the process address space. +An alternate use is for mapping files in the memory address space. +Mapping of files is done by the loader for executables and libraries. +That is why, in the output of pmap, there is a column with a filename.

Mapping of a file results in getting a pointer to its contents and then using that pointer. +This way, reading and writing to a file is an exercise of pointer copying, instead of the use of read / write-like system calls.

In the support/copy/ folder, there are two source code files and two scripts:

  • read_write_copy.c implements copying with read / write syscalls
  • mmap_copy.c implements copying using mmap
  • generate.sh script generates the input file in.dat
  • benchmark_cp.sh script runs the two executables mmap_copy and read_write_copy

Open the two source code files and investigate them. +You will notice that the open() system call has the following prototype int open(const char *pathname, int flags). +The argument flags must include one of the following access modes: O_RDONLY, O_WRONLY, or O_RDWR - indicating that the file is opened in read-only, write-only, or read/write mode. +You can add an additional flag - O_CREAT - that will create a new file with pathname if the file does not already exist. +This is only the case when opening the file for writing (O_WRONLY or O_RDWR). +If O_CREAT is set, a third argument mode_t mode is required for the open() syscall. +The mode argument specifies the permissions of the newly created file. +For example:

// If DST_FILENAME exists it will be open in read/write mode and truncated to length 0
// If DST_FILENAME does not exist, a file at the path DST_FILENAME will be create with 644 permissions
dst_fd = open(DST_FILENAME, O_RDWR | O_CREAT | O_TRUNC, 0644);

Let's generate the input file:

student@os:~/.../lab/support/copy$ ./generate.sh

and let's build the two executable files:

student@os:~/.../lab/support/copy$ make

Run the benchmark_cp.sh script:

Run the script in your local environment, not in the Docker container. +Docker does not have permission to write to /proc/sys/vm/drop_caches file.

student@os:~/.../lab/support/copy$ ./benchmark_cp.sh
Benchmarking mmap_copy on in.dat
time passed 54015 microseconds

Benchmarking read_write_copy on in.dat
time passed 42011 microseconds

Run the script a few more times. +As you can see, there isn't much of a difference between the two approaches. +Although we would have expected the use of multiple system calls to cause overhead, it's too little compared to the memory copying overhead.

If you inspect benchmark_cp.sh, you will notice a weird-looking command sh -c "sync; echo 3 > /proc/sys/vm/drop_caches". +This is used to disable a memory optimization that the kernel does. +It's called "buffer cache" and it's a mechanism by which the kernel caches data blocks from recently accessed files in memory. +You will get more detailed information about this in the I/O chapter.

Browse the two source code files (mmap_copy.c and read_write_copy.c) for a glimpse on how the two types of copies are implemented.

Quiz

Practice​

  1. Use a different value for BUFSIZE and see if that affects the comparison between the two executables.

  2. Add a sleep() call to the mmap_copy.c file after the files were mapped. +Rebuild the program and run it. +On a different console, use pmap to view the two new memory regions that were added to the process, by mapping the in.dat and out.dat files.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/bypass-canary/index.html b/17/Lab/Data/quiz/bypass-canary/index.html new file mode 100644 index 0000000000..71b535169e --- /dev/null +++ b/17/Lab/Data/quiz/bypass-canary/index.html @@ -0,0 +1,16 @@ + + + + + +Bypass Canary | Operating Systems + + + + +
+
Skip to main content

Bypass Canary

Question​

If we enable ASLR, can we still exploit the stack_protector program?

  • no, because the address of pawned is going to be different for every run.

  • yes, because ASLR cannot work well when the canary is activated.

  • yes, because ASLR randomizes the start address of a section, but the offsets remain the same.
  • no, because the address to which addr points to is going to be random
+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/half-page/index.html b/17/Lab/Data/quiz/half-page/index.html new file mode 100644 index 0000000000..00ac97d810 --- /dev/null +++ b/17/Lab/Data/quiz/half-page/index.html @@ -0,0 +1,18 @@ + + + + + +Half Page | Operating Systems + + + + +
+
Skip to main content

Half Page

Question Text​

char *p = malloc(2 * 1024);

What is a potential problem when allocating half a page?

Question Answers​

  • It will fragment the virtual memory because users should always request at least one page

  • Asking for less than a page might place the memory on two different pages

  • Writing to the other half may be allowed
  • Allocations smaller than one page should use the stack

Feedback​

The OS allocates memory in chunks of 4 KB (the page size). +If the memory we allocate happens to be placed at the beginning of a page, we have permission to update the second half of this page despite not requesting it. +This might be a problem because buffer overflows can pass unnoticed.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/malloc-brk/index.html b/17/Lab/Data/quiz/malloc-brk/index.html new file mode 100644 index 0000000000..b2417b121a --- /dev/null +++ b/17/Lab/Data/quiz/malloc-brk/index.html @@ -0,0 +1,17 @@ + + + + + +Malloc `brk()` | Operating Systems + + + + +
+
Skip to main content

Malloc brk()

Question Text​

When does malloc() use brk()?

Question Answers​

  • brk() is outdated, malloc() always uses mmap()
  • When it allocates a small chunk of memory

  • When it allocates an array

  • When it's working with dynamic libraries

Feedback​

malloc() uses both brk() and mmap(), but preffers brk() for small chunks of memory to keep granular allocations in a contiguous area. +This way, free() does not necessarily return the memory to the OS as it might only mark the zone as "free" within libc's allocator and reuse it for later allocations.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/malloc-mmap/index.html b/17/Lab/Data/quiz/malloc-mmap/index.html new file mode 100644 index 0000000000..cb2a4d7572 --- /dev/null +++ b/17/Lab/Data/quiz/malloc-mmap/index.html @@ -0,0 +1,18 @@ + + + + + +Malloc `mmap()` | Operating Systems + + + + +
+
Skip to main content

Malloc mmap()

Question Text​

When does malloc() use mmap()?

Question Answers​

  • When it allocates read-only memory

  • When it allocates zeroed memory

  • When it allocates chunks of memory bigger than an internal threshold
  • When the heap is full

Feedback​

malloc uses both brk() and mmap(), but prefers mmap() for big chunks of memory (by default larger than 128 KB). +This value can be altered using mallopt() with the param argument set to M_MMAP_THRESHOLD. +These memroy blocks are unlikely to be reused so they are not placed on heap to avoid memory fragmentation.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/memory-access/index.html b/17/Lab/Data/quiz/memory-access/index.html new file mode 100644 index 0000000000..543b6b45ad --- /dev/null +++ b/17/Lab/Data/quiz/memory-access/index.html @@ -0,0 +1,19 @@ + + + + + +Modify String | Operating Systems + + + + +
+
Skip to main content

Modify String

Question Text​

What happens if we introduce the code cp[2] = 't' in the program located in the memory-access/mem_access.c file?

Question Answers​

  • Compile time error because we are trying to modify a const pointer.

  • Compile time error because we are trying to modify a const value.

  • Segmentation fault at runtime we are trying to modify read-only memory.
  • Program compiles and runs succesfully.

Feedback​

The declaration char *const cp = "ConstLeString" actually defines 2 memory locations. +One stores a constant pointer, cp, whereas the other stores the actual string. +The compiler thinks that cp is able to modify the memory location that it points therefore it passes compilation. +But at runtime a segmentation fault is issued because we are accessing data that is stored in read-only memory.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/memory-aslr/index.html b/17/Lab/Data/quiz/memory-aslr/index.html new file mode 100644 index 0000000000..d0872905c3 --- /dev/null +++ b/17/Lab/Data/quiz/memory-aslr/index.html @@ -0,0 +1,16 @@ + + + + + +ASLR | Operating Systems + + + + +
+
Skip to main content

ASLR

Question Text​

If we enable ASLR and run the bo_write_practice executable with the previously payload what vulnerabilities will we be able to still exploit?

Question Answers​

  • we can still jump to the secret_func

  • we can both still jump to the secret_func and overwrite the local variable

  • we can jump to a library function

  • we can still overwrite the local variable
+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/memory-granularity/index.html b/17/Lab/Data/quiz/memory-granularity/index.html new file mode 100644 index 0000000000..f51410b1a9 --- /dev/null +++ b/17/Lab/Data/quiz/memory-granularity/index.html @@ -0,0 +1,17 @@ + + + + + +Memory Granularity | Operating Systems + + + + +
+
Skip to main content

Memory Granularity

Question Text​

What is the granularity of the size of memory sections?

Question Answers​

  • 4 bytes

  • 4 MB

  • 4 KB
  • 1 KB

Feedback​

All sizes of memory areas are multiple of 4 KB (the page size). +Also, all addresses start at multiple of 4 KB.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/memory-leaks/index.html b/17/Lab/Data/quiz/memory-leaks/index.html new file mode 100644 index 0000000000..42249101a4 --- /dev/null +++ b/17/Lab/Data/quiz/memory-leaks/index.html @@ -0,0 +1,17 @@ + + + + + +Memory Leaks | Operating Systems + + + + +
+
Skip to main content

Memory Leaks

Question Text​

What happens to the leaked memory when the process finishes execution?

Question Answers​

  • It will be freed by the OS garbage collector

  • It remains unusable until the first restart

  • It is freed alongside all memory used by process
  • It will remain reachable untill another process explicitely frees it

Feedback​

When a process ends, all its memory zones are freed by the OS to be reused. +Leaking memory becomes a major problem in case of programs that run indefinitely, because the leaked memory will stack, causing a memory outage.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/memory-regions-vars/index.html b/17/Lab/Data/quiz/memory-regions-vars/index.html new file mode 100644 index 0000000000..b039bafd90 --- /dev/null +++ b/17/Lab/Data/quiz/memory-regions-vars/index.html @@ -0,0 +1,20 @@ + + + + + +Variables in memory regions | Operating Systems + + + + +
+
Skip to main content

Variables in memory regions

Question Text​

In what memory regions are the a, b, c, k varaables from support/memory-security/buff_leak.c stored?

Question Answers​

  • a - .data, b - .bss, c - .stack, k - .heap
  • a - .data, b - .bss, c - .stack, k - .stack

  • a - .data, b - .data, c - .stack, k - .heap

  • a - .bss, b - .bss, c - .stack, k - .heap

Feedback​

Global initialized variables go to .data (a). +Global uninitialized variables go to .bss (b). +Non-static local variables go on the.stack (c, k). +malloc()'ed memory goes on the.heap. +For k, the pointer is stored on the.stack, but the allocated memory, to which k points is stored on the.heap.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/memory-stack-protector/index.html b/17/Lab/Data/quiz/memory-stack-protector/index.html new file mode 100644 index 0000000000..96bef2d243 --- /dev/null +++ b/17/Lab/Data/quiz/memory-stack-protector/index.html @@ -0,0 +1,19 @@ + + + + + +Stack Protector | Operating Systems + + + + +
+
Skip to main content

Stack Protector

Question Text​

For the support/memory-security/bo_write_practice executable we are not able overwrite the s variable no matter the input. +Why is that?

Question Answers​

  • s needs to be declared after buffer.
  • when using the stack canary, the buffer is always placed right below it.

  • due to ASLR, the address of s is random.

  • the stack canary makes it impossible to overwrite local variables.

Feedback​

When using the canary, to minimize the damage a buffer overflow could cause, the buffers are always placed right below the canary. +By doing so, a buffer overflow will not overwrite anything. +However, it is still possible to overwrite other local buffers, provided that a function declares more than 1 array or if we use the pointer directly.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/mmap-file/index.html b/17/Lab/Data/quiz/mmap-file/index.html new file mode 100644 index 0000000000..3639b1bce6 --- /dev/null +++ b/17/Lab/Data/quiz/mmap-file/index.html @@ -0,0 +1,17 @@ + + + + + +`mmap()` file | Operating Systems + + + + +
+
Skip to main content

mmap() file

Question Text​

What is one advantage of mapping a file in memory?

Question Answers​

  • It reduces interaction with the disk

  • Consumes less memory

  • It is faster because it does not uses the file API

  • Allows all threads to use the same memory area

Feedback​

After mapping a file in memory, all changes will be visible only in memory. +When removing the mapping or explicitely calling msync() the information from memory will be visible on disk.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/operators/index.html b/17/Lab/Data/quiz/operators/index.html new file mode 100644 index 0000000000..d980fb91ba --- /dev/null +++ b/17/Lab/Data/quiz/operators/index.html @@ -0,0 +1,16 @@ + + + + + +Operator Overloading | Operating Systems + + + + +
+
Skip to main content

Operator Overloading

Question Text​

How many constructor calls, copy constructor calls, assignment operator calls and destructor calls does the following program issue?

Obj quiz(Obj o1, Obj o2)
{
    o2 = o1;
    return o2;
}
void main()
{
    Obj b = quiz(o1, o2);
}

Question Answers​

  • constructor calls = 0, copy constructor calls = 3, assignment operator calls = 1, destructor calls = 3

  • constructor calls = 1, copy constructor calls = 2, assignment operator calls = 1, destructor calls = 2

  • constructor calls = 0, copy constructor calls = 2, assignment operator calls = 1, destructor calls = 2

  • constructor calls = 0, copy constructor calls = 3, assignment operator calls = 1, destructor calls = 1

Feedback​

  • There are no constructor calls because there is no object construction when using int.

  • There are 3 copy constructor calls: for passing o1, for passing o2, and for returning o2.

  • There is 1 assignment operator call for o2 = o1.

  • There are 3 destructor calls, because each constructed object needs to be destroyed.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/page-allocation/index.html b/17/Lab/Data/quiz/page-allocation/index.html new file mode 100644 index 0000000000..491b3a6fd4 --- /dev/null +++ b/17/Lab/Data/quiz/page-allocation/index.html @@ -0,0 +1,17 @@ + + + + + +Page Allocation | Operating Systems + + + + +
+
Skip to main content

Page Allocation

Question Text​

student@os:~/.../lab/support/static-dynamic$ size hello-static
text    data     bss     dec     hex filename
893333   20996    7128  921457   e0f71 hello-static

How many bytes should we add to the .data section to make its size 28 KB, instead of 24 KB?

Question Answers​

  • 1 KB

  • 4 KB

  • 3580 bytes

  • 3581 bytes

Feedback​

The total size must be 1 byte over the 24 KB threshold to cause a new page allocation. +So in order to get that past the current size of 20996, we need 3581 bytes:

student@os:~$ echo "24 * 1024 + 1 - 20996" | bc
3581
+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/stack-layout/index.html b/17/Lab/Data/quiz/stack-layout/index.html new file mode 100644 index 0000000000..0b3f9a05b8 --- /dev/null +++ b/17/Lab/Data/quiz/stack-layout/index.html @@ -0,0 +1,16 @@ + + + + + +Stack layout | Operating Systems + + + + +
+
Skip to main content

Stack layout

Question Text​

What is the stack layout for the fun function in the bo_write.c program (starting from a high address)?

Question Answers​

  • return address, old rbp, maybe some padding, variable a, b[0], b[1], b[2]
  • return address, old rbp, maybe some padding, variable a, b[2], b[1], b[0]

  • return address, maybe some padding, variable a, b[0], b[1], b[2]

  • return address, old rbp, maybe some padding, b[0], b[1], b[2], variable a

Feedback​

Look at the assembly code and notice the exact layout.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/string-buff-over/index.html b/17/Lab/Data/quiz/string-buff-over/index.html new file mode 100644 index 0000000000..31b729a263 --- /dev/null +++ b/17/Lab/Data/quiz/string-buff-over/index.html @@ -0,0 +1,17 @@ + + + + + +String Buffer Overflow | Operating Systems + + + + +
+
Skip to main content

String Buffer Overflow

Question Text​

Why does the buffer overflow occur?

Question Answers​

  • the initial string, declared in main(), does not contain a terminating null byte.

  • the buffer is not large enough to store the copied bytes.

  • memcpy() skips the copying of terminating null bytes.

  • memcpy() copies 4 bytes, whereas the size of the string, including the terminating null byte, is 5.

Feedback​

The string "soso" has length equal to 4, however, 5 bytes are actually used to store it, including the terminating null byte. +Even though the buffer declared in fun() is not large enough to store the 5 bytes, the underlying issue is that we copying just 4 bytes, thus skipping the null byte.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/string-strcpy/index.html b/17/Lab/Data/quiz/string-strcpy/index.html new file mode 100644 index 0000000000..b7e0c87d0c --- /dev/null +++ b/17/Lab/Data/quiz/string-strcpy/index.html @@ -0,0 +1,16 @@ + + + + + +Strcpy Buffer Overflow | Operating Systems + + + + +
+
Skip to main content

Strcpy Buffer Overflow

Question Text​

Does any buffer overflow occur with the latest version of the program?

Question Answers​

  • no, because strcpy() was designed to correctly handle such situations.

  • no, because the string is correctly printed, i.e. no extra characters.

  • we cannot know

  • yes, strcpy() copies the entirety of the source, including the \0; since the size of dst is 4 the null byte overwrites the least significant byte of var

Feedback​

Print the contents of variable var.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/quiz/valgrind-leaks/index.html b/17/Lab/Data/quiz/valgrind-leaks/index.html new file mode 100644 index 0000000000..671a99ed3e --- /dev/null +++ b/17/Lab/Data/quiz/valgrind-leaks/index.html @@ -0,0 +1,18 @@ + + + + + +Valgrind Leaks | Operating Systems + + + + +
+
Skip to main content

Valgrind Leaks

Question Text​

struct student {
    char *name;
    int age;
}

struct student *s = malloc(sizeof(*s));
s->name = strdup("Reginald");
// ...
free(s);

What are the leaks in the above c program?

Question Answers​

  • There are no leaks

  • s->name is definitely lost

  • s->name is indirectly lost
  • s->name is still reachable

Feedback​

strdup() allocates memory for a string so the returned pointer must be freed. +Freeing s will leave us unable to free s->name, so s->name is indirectly lost. +Find more about valgrind leak categories here.

+ + + + \ No newline at end of file diff --git a/17/Lab/Data/working-memory/index.html b/17/Lab/Data/working-memory/index.html new file mode 100644 index 0000000000..88821f4f45 --- /dev/null +++ b/17/Lab/Data/working-memory/index.html @@ -0,0 +1,81 @@ + + + + + +Working with Memory | Operating Systems + + + + +
+
Skip to main content

Working with Memory

As previously stated, from a programmer's perspective, memory is abstracted into variables. +This hides most of the lower level abstractions. +Each variable is characterized by an address (or location in memory), type and access rights. +Some languages require that the developer spells out these attributes explicitly (statically typed languages - notable examples: C\C++, D, Java) whereas others deduce them by analyzing the context (dynamically typed languages - notable examples: Python, JavaScript). +Nevertheless, the language compiler needs to handle this information and, based on it, generate code that manages memory correctly and efficiently.

Memory Access​

Accessing memory is defined by reading or writing values to or from a variable. +From a programmer's perspective, this looks pretty straightforward:

int main(void)
{
    int a;               // declare variable
    a = 42;              // write 42 to variable a
    printf("%d\n", a);   // read variable a and print its contents

    return 0;
}

However, from a lower level perspective, there are other attributes that need to be taken care of. +For instance, variable a needs to have a correspondent area that is reserved in memory. +That specific chunk of memory is described by an address and a size. +The address for a is automatically generated by going through multiple layers of abstractions, but the size is spelled out indirectly by the programmer by using the keyword int. +Another aspect is represented by the access rights for a specific memory area. +In our example, a is defined as being plain mutable, however, it is possible to declare constant variables which are stored in memory location with no writing rights.

Using the above information, the compiler and the operating system co-work to allocate memory that can represent the contents of the variable.

No matter what sort of language you are using, statically or dynamically typed, a variable is always described by the (address, size, access rights) triplet. +By using this triplet, the content of a variable is stored, retrieved or rewritten.

Practice​

Navigate to the support/memory-access/ directory. +Inspect the mem_access.c source file.

  1. Describe each variable by completing its (address, size, access rights) tuple.

  2. Try to modify the ca, cp and cp2 variables by assigning some other value to them. +Explain the behavior.

Quiz​

Quiz

Memory Protection​

Memory contents (both code and data) are separated into sections or zones. +This makes it easier to manage. +More than that, it allows different zones to have different permissions. +This follows the principle of least privilege where only required permissions are part of a given section.

Code is usually placed in a section (.text) with read and execute permissions; +no write permissions. +Variables are placed in different sections (.data, .bss, stack, heap) with read and write permissions; +no execute permissions.

Let's navigate to the support/memory-protection/ directory and inspect the mem_prot.c source file. +The file uses different access types for the data variable and the do_nothing function.

Build it:

student@os:~/.../lab/support/memory-protection$ make
gcc -g -Wall -Wextra -Werror -I../../../../../common/makefile/../utils -I../../../../../common/makefile/../utils/log  -c -o mem_prot.o mem_prot.c
gcc mem_prot.o ../../../../../common/makefile/../utils/log/log.o  -o mem_prot

student@os:~/.../lab/support/memory-protection$ ./mem_prot
reading from .data section
writing to .data section
reading from .text section
executing .text section

All current actions in the program are valid.

Let's uncomment each commented line in the program and try again:

student@os:~/.../lab/support/memory-protection$ ./mem_prot
reading from .data section
writing to .data section
reading from .text section
executing .text section
executing .data section
Segmentation fault (core dumped)

We now receive the dreaded Segmentation fault message when we try to access a memory section with wrong permissions.

Permissions come into play when we control the memory address via pointers. +But even for programming languages that don't offer pointers (such as Python) issues may still arise.

In the str.py file, we look to modify str[1], but this fails:

student@os:~/.../lab/support/memory-protection$ ./str.py
n, 110, n
Traceback (most recent call last):
  File "./str.py", line 5, in <module>
    str[1] = 'z'
TypeError: 'str' object does not support item assignment

This fails because strings are, in Python, immutable. +Once a string is being created, it can not be modified; +you have to create a new string.

Practice​

Go to the support/memory-protection/ folder and solve the practice items below.

  1. Add a variable named ro that you define as const. +The variable will be placed on a read-only section (.rodata) such as that write and execution access would result in Segmentation fault.

    Access the ro variable and show that, indeed, for write and execution access, Segmentation fault is issued.

Memory Allocation Strategy​

Navigate to the support/memory-alloc/ directory. +It contains 3 implementations of the same program in different languages: C, Python and D. +The program creates a list of entries, each entry storing a name and an id. +The purpose of this exercise is to present the different strategies that programming languages adopt to manage memory.

C​

The C implementation manages the memory manually. +You can observe that all allocations are performed via malloc and the memory is freed using free. +Arrays can be defined as static (on the stack) or dynamic (a pointer to some heap memory). +Stack memory doesn't need to be freed, hence static arrays are automatically deallocated. +Heap memory, however, is managed by the user, therefore it is the burden of the programmer to find the optimal memory strategy. +This offers the advantage that you can fine tune the memory usage depending on your application, but this comes with a cost: more often than not, managing memory is a highly complex error-prone task.

Python​

The Python implementation of the program has no notion of memory allocation. +It simply defines variables and the garbage collector takes care of allocating and deallocating memory. +Notice how the destructor is called automatically at some point when the garbage collector deems that the list is not used anymore. +Garbage collection lifts the burden of memory management from the user, however, it may be unsuitable for certain scenarios. +For example, real-time applications that need to take action immediately once a certain event occurs cannot use a garbage collector (GC). +That is because the GC usually stops the application to free dead objects.

D​

The previous 2 examples have showcased extreme situations: fully manual vs fully automatic memory management. +In D, both worlds are combined: variables may be allocated manually on the stack/heap or allocated via the garbage collector (for brevity, malloc-based allocation is not presented in this example). +Arrays that are allocated on the stack behave the same as in C, whereas arrays allocated with the garbage collector mimic Python lists. +Classes are also garbage collected.

Memory Vulnerabilities​

The purpose of this exercise is to provide examples on how memory corruption may occur and what are the safety guards implemented by different programming languages.

Navigate to the support/memory-vuln/ directory. +It features 3 files, each showcasing what happens in case of actions that may lead to memory corruption.

C​

The C implementation showcases some of the design flaws of the language can lead to memory corruption.

The first example demonstrates how a pointer to an expired stack frame may be leaked to an outer scope. +The C language does not implement any guards against such behavior, although data flow analysis could be used to detect such cases.

The second example highlights the fact that C does not check any bounds when performing array operations. +This leads to all sorts of undefined behavior. +In this scenario, some random memory is overwritten with 5. +The third example exhibits a manifestation of the previous design flaw, where the return address of the main function is overwritten with 0, thus leading to a segmentation fault.

Although today it seems obvious that such behavior should not be accepted, we should take into account that the context in which the C language was created was entirely different from today. +At that time the resource constraints - DRAM memory was around a few KBs, operating systems were in their infancy, branch predictors did not exist etc. - were overwhelming. +Moreover, security was not a concern because the internet basically did not exist. +As a consequence, the language was not developed with memory safety in mind.

Python​

Technically, it is not possible to do any memory corruption in Python (that is, if you avoid calling C functions from it). +Pointers do not formally exist, and any kind of array access is checked to be within its bounds. +The example simply showcases what happens when an out-of-bounds access is performed - an IndexError is thrown and execution halts.

D​

The D implementation uses almost the same code as the C implementation, but suffers from minor syntax modifications. +In essence, the two implement the same logic. +When compiling this code, it can be observed that the D compiler notices at compile time that an out-of-bounds access is performed. +This makes sense, since a static array cannot modify its length and therefore the compiler has all the information to spot the mistake. +The only way to make the code compile is to comment the faulting lines or to replace the out-of-bounds index with a correct one. +After doing so, the program compiles and we can see that memory corruption occurs. +However, D also has safety checks, however, these are not performed by default. +To enable such checks, the user must annotate a function with the @safe keyword:

int* bad() @safe

By doing so, the mechanical checks are enabled and a new set of criteria needs to be followed for the code to be accepted. +Taking the address of a local, doing pointer arithmetic, reinterpret casts, calling non-@safe functions etc. are not allowed in @safe code. +If any of these unsafe features are manually proven to be safe, the @trusted keyword may be used to disable the checks but still consider the code @safe. +This is to allow writing system code, which by its nature is unsafe.

Memory Corruption​

For this practice item, you will need to identify the programming mistake that makes it possible to corrupt memory.

Navigate to the support/memory-corruption folder. +Inspect the source file segfault.c.

  1. What does the program do? (this could be a quiz in the final form)
  2. Compile and run it. +What happens?
  3. Debug the program and find the line that causes the segfault. +Note: Although using printf() calls is a viable option, we strongly suggest you use GDB.
  4. Fix the program.
  5. Analyze the corresponding Python and D implementation.

What is the expected result in each case? +Why? +Run the programs and see what happens.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/arena/index.html b/17/Lab/IO/arena/index.html new file mode 100644 index 0000000000..00af7312af --- /dev/null +++ b/17/Lab/IO/arena/index.html @@ -0,0 +1,35 @@ + + + + + +Arena | Operating Systems + + + + +
+
Skip to main content

Arena

Open File Structure in the Kernel​

The "open file" struct in the Linux kernel is called struct file +Its most important fields are:

struct file {
        struct path                     f_path;
        /* Identifier within the filesystem. */
        struct inode                    *f_inode;

        /**
         * Contains pointers to functions that implement operations that
         * correspond to syscalls, such as `read()`, `write()`, `lseek()` etc.
         */
        const struct file_operations    *f_op;

        /**
         * Reference count. A `struct file` is deallocated when this reaches 0,
         * i.e. nobody uses it anymore.
         */
        atomic_long_t                   f_count;

        /* Those passed to `open()`. */
        unsigned int                    f_flags;
        fmode_t                         f_mode;

        /* Cursor from where reads/writes are made */
        loff_t                          f_pos;
        /* To allow `f_pos` to be modified atomically. */
        struct mutex                    f_pos_lock;
}

As mentioned above, struct file_operations contains function pointers that well-known syscalls such as read() end up calling. +Each filesystem needs to define its own implementations of these functions. +Some of the most widely known file_operations are listed below. +By now, you should recognise most of them:

struct file_operations {
        loff_t (*llseek) (struct file *, loff_t, int);
        ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);
        ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);
        int (*mmap) (struct file *, struct vm_area_struct *);
        unsigned long mmap_supported_flags;
        int (*open) (struct inode *, struct file *);
        int (*flush) (struct file *, fl_owner_t id);
        int (*fsync) (struct file *, loff_t, loff_t, int datasync);
        int (*fasync) (int, struct file *, int);
}

Mini-shell with Blackjack and Pipes​

OK, there will be no Blackjack... +for now at least. +But there will be pipes. +Navigate back to support/mini-shell/mini_shell.c and add support for piping 2 commands together like this:

> cat bosses.txt | head -n 5
Darkeater Midir
Slave Knight Gael
Nameless King
Dancer Of The Boreal Valley
Ancient Dragon

To Drop or Not to Drop?​

Remember support/buffering/benchmark_buffering.sh or support/file-mappings/benchmark_cp.sh. +They both used this line:

sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"

Note that sync has a man page and it partially explains what's going on:

The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes. This improves performance

So the kernel does even more buffering! +But this time, it's not at the syscall level, like with read() and write(). +And it's used a bit differently.

While buffering is a means of either receiving data in advance (for reading) or committing it retroactively (for writing) to speed up subsequent syscalls that use the next data, caching is a means of speeding up calls that use the same data. +Just like your browser caches the pages you visit so you refresh them faster or your CPU caches your most recently accessed addresses, so does your OS with your files.

Some files are read more often than others: logs, some configs etc. +Upon encountering a first request (read / write) to a file, the kernel stores chunks of them in its memory so that subsequent requests can receive / modify the data in the RAM rather than waiting for the slow disk. +This makes I/O faster.

The line from which this discussion started invalidates those caches and forces the OS to perform I/O operations "the slow way" by interrogating the disk. +The scripts use it to benchmark only the C code, not the OS.

To see just how much faster this type of caching is, navigate to support/buffering/benchmark_buffering.sh once again and comment-out the line with sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches". +Now run the script a few times and compare the results. +You should see some drastic improvements in the running times, such as:

student@os:/.../support/file-mappings$ ./benchmark_cp.sh
make: Nothing to be done for 'all'.
Benchmarking mmap_cp on a 1 GB file...

real    0m13,299s
user    0m0,522s
sys     0m1,695s
Benchmarking cp on a 1 GB file...

real    0m10,921s
user    0m0,013s
sys     0m1,301s


student@os:/.../support/file-mappings$ ./benchmark_cp.sh
make: Nothing to be done for 'all'.
Benchmarking mmap_cp on a 1 GB file...

real    0m1,286s
user    0m0,174s
sys     0m0,763s
Benchmarking cp on a 1 GB file...

real    0m5,411s
user    0m0,012s
sys     0m1,201s

Each subsequent benchmark actually reads the data from the caches populated or refreshed by the previous one.

You can use free -h to view how much data your kernel is caching. +Look at the buff/cache column. +One possible output is shown below. +It says the OS is caching 7 GB of data.

student@os:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:           15Gi       8,1Gi       503Mi       691Mi       7,0Gi       6,5Gi
Swap:         7,6Gi       234Mi       7,4Gi
+ + + + \ No newline at end of file diff --git a/17/Lab/IO/async-io/index.html b/17/Lab/IO/async-io/index.html new file mode 100644 index 0000000000..a7ebc381fe --- /dev/null +++ b/17/Lab/IO/async-io/index.html @@ -0,0 +1,41 @@ + + + + + +Asynchronous I/O | Operating Systems + + + + +
+
Skip to main content

Asynchronous I/O

When doing I/O, the major issue we are facing is that I/O operations are typically much slower than CPU operations. +Because of that, in a typical synchronous scenario, a lot of time may be spent waiting for an I/O operation to be complete.

Because of that, we have other types of I/O operations.

The simplest type of I/O operating is a synchronous, blocking operation. +In this operation type, we call a given function (such as read() or write()) and wait until it completes.

Another type of operation is a non-blocking operation. +In this type of operation, if data isn't available (in case of read()) or can not be sent (in case of write()), the given call will not block, it will simply return with a mark to try again later on.

Another type of operation is an asynchronous operation. +In the case of an asynchronous operation, the given function returns immediately and the program continues its execution. +Once the operation succeeds, a notification may be sent out to the program. +Or the program can periodically check the state of the operation.

Apart from these operating types, there is the option to do I/O multiplexing, i.e. the ability to tackle multiple channels simultaneously. +This is useful in case of servers, that get a large number of requests and have to iterate through them.

In case of asynchronous I/O, the "backend" used to implement the operations may differ:

  1. It can be a multiprocess backend, where each action / request is passed to a given process.

  2. It can be a multithreaded backend, where each action / request is passed to a given thread.

  3. It can be an event-based backend, where an action is scheduled together with a callback that is invoked when an action ends.

Practice​

Enter the support/async/ folder for some implementations of a simple request-reply server in Python or in C. +The server gets requests and serves them in different ways: synchronous, multiprocess-based, multi-threading-based, asynchronous.

We use two implementations, in Python and in C.

  1. For the Python implementation, enter the python/ subdirectory. +Take a look at the implementation of different servers:

    • synchronous server: server.py
    • multiprocess backend: mp_server.py
    • multithreaded backend: mt_server.py
    • asynchronous API backend: async_server.py (requires Python >= 3.7) and async_server_3.6.py (works for Python 3.6)

    Let's do a benchmarking session of each server type. +For this we will:

    1. Start the server on one console.

    2. Use the client_bench.sh script to benchmark the server. +client_bench.sh is a simple script that starts multiple clients to connect to the servers, by running client.py. +It runs NUM_CLIENT instances of client.py to trigger actions in the remote server.

      Note that you may be required to run servers on different ports in case of an Address already in use error. +If that is the case, you will also need to run the client_bench.sh script.

    To start the server, run each of these commands (one at a time to test the respective server type):

    student@os:/.../support/async/python$ ./server.py 2999

    student@os:/.../support/async/python$ ./mp_server.py 2999

    student@os:/.../support/async/python$ ./mt_server.py 2999

    student@os:/.../support/async/python$ ./async_server_3.6.py 2999

    For each server, in a different console, we can test to see how well it behaves by running:

    student@os:/.../support/async/python$ time ./client_bench.sh

    You will see a time duration difference between mp_server.py and the others, mp_server.py runs requests faster. +This is because the multiprocess model works OK for a CPU-intensive server such as this.

    But not on threading, because threading suffers from the use of GIL (Global Interpreter Lock), that prevents multithreaded programs from running Python code simultaneously.

  2. For the C implementation, enter the c/ subdirectory. +Take a look at the implementation of different servers:

    • synchronous server: server.c
    • multiprocess backend: mp_server.c
    • multithreaded backend: mt_server.c

    There is no asynchronous C variant, because of the unstable API of libaio and io_uring.

    First, use make to build the servers. +Then, go through the same steps as above:

    1. Start the server on one console.

    2. Use the client_bench.sh script (in the ../python/ directory) to benchmark the server.

    Same as with Python, to start the server, run each of these commands (one at a time to test the respective server type):

    student@os:/.../support/async/c$ ./server 2999

    student@os:/.../support/async/c$ ./mp_server 2999

    student@os:/.../support/async/c$ ./mt_server 2999

    For each server, in a different console, we can test to see how well it behaves by running:

    student@os:/.../support/async/python$ time client_bench.sh

    We draw 2 conclusions from using the C variant:

    1. C is faster than Python
    2. The thread variant is indeed parallel, as now the GIL is no longer involved.

Remarks​

Asynchronous operations, as with others, provide an API, as is the case with the Python API or the libaio and io_uring. +The backend of these operations may be a thread-based one in the library providing the API, or it may rely on support from the operating system. +When aiming for performance, asynchronous I/O operations are part of the game. +And it's very useful having a good understanding of what's happening behind the scenes.

For example, for the Python async_server_3.6.py server, a message asyncio: Using selector: EpollSelector is provided. +This means that the backend relies on the use of the epoll() function that's part of the I/O Multiplexing section.

Also, for Python, the use of the GIL may be an issue when the operations are CPU intensive.

This last point deserves a discussion. +It's rare that servers or programs using asynchronous operations are CPU intensive. +It's more likely that they are I/O intensive, and the challenge is avoiding blocking points in multiple I/O channels; +not avoiding doing a lot of processing, as is the case with the fibonacci() function. +In that particular case, having thread-based asynchronous I/O and the GIL will be a good option, as you rely on the thread scheduler to be able to serve multiple I/O channels simultaneously. +This later approach is called I/O multiplexing, discussed in the next section.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/beyond-network-sockets/index.html b/17/Lab/IO/beyond-network-sockets/index.html new file mode 100644 index 0000000000..619df668e9 --- /dev/null +++ b/17/Lab/IO/beyond-network-sockets/index.html @@ -0,0 +1,37 @@ + + + + + +Beyond Network Sockets | Operating Systems + + + + +
+
Skip to main content

Beyond Network Sockets

Up until this point, we first learned how to use the Berkeley Sockets API, then we learned about the client-server model, based on this API. +So now we know that sockets offer a ubiquitous interface for inter-process communication, which is great. +A program written in Python can easily send data to another one written in C, D, Java, Haskell, you name it. +However, in the section dedicated to networking, we saw that it takes a whole stack of protocols to send this message from one process to the other. +As you might imagine, this is much slower even than local I/O using files.

So far we've only used sockets for local communication, but in practice it is a bit counterproductive to use network sockets for local IPC due to their high latency. +Wouldn't it be great if we had a way to use the sockets API for local IPC without having to deal with this increased latency? +Well, there is a way and it's called UNIX sockets.

UNIX Sockets​

UNIX sockets are created using the socket() syscall and are bound TO A FILE instead of an IP and port using bind(). +You may already see a similarity with named pipes. +Just like them, UNIX sockets don't work by writing data to the file (that would be slow), but instead the kernel retains the data they send internally so that send() and recv() can read it from the kernel's storage. +You can use read() and write() to read/write data from/to both network and UNIX sockets as well, by the way. +The differences between using send()/recv() or write()/read() are rather subtle and are described in this Stack Overflow thread.

UNIX sockets are a feature of POSIX-compliant operating systems (e.g. Linux, macOS) and are not available on non-POSIX operating systems, such as Microsoft Windows. +However, there are third-party libraries providing similar features to UNIX sockets in non-POSIX systems, but they might not have the same performance and reliability.

Practice: Receive from UNIX Socket​

Navigate to support/receive-challenges/receive_unix_socket.c. +Don't write any code yet. +Let's compare UNIX sockets with network sockets first:

  • To create them, give socket() the PF_UNIX/AF_UNIX instead of PF_INET/AF_INET. +The latter are used to create network sockets. +You can use PF_* or AF_* interchangeably. +They mean "protocol families" and "address families", respectively, but they are essentially the same thing. +But apart from this, UNIX sockets can be set to work in both TCP (SOCK_STREAM) and UDP (SOCK_DGRAM) modes.

  • The bind() call now takes a path as argument, as specified in the UNIX socket man page.

And this is it. +Everything else is the same.

Now fill in the TODOs in receive_unix_socket.c. +Use the example at the bottom of the UNIX socket man page if you get stuck at bind(). +When you're done, compile the code and then run send_unix_socket. +If you did this task correctly, receive_unix_socket should display the flag.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/client-server-model/index.html b/17/Lab/IO/client-server-model/index.html new file mode 100644 index 0000000000..06dda3611e --- /dev/null +++ b/17/Lab/IO/client-server-model/index.html @@ -0,0 +1,46 @@ + + + + + +Client-Server Model | Operating Systems + + + + +
+
Skip to main content

Client-Server Model

Up to now, we've avoided code snippets using TCP. +Not anymore. +Keep in mind that all socket functions in the Berkeley API have very detailed and informative man pages. +For example, here are the man pages for sendto() and recvfrom() that are used with UDP sockets.

Going back to our initial example with how the browser gets data from https://open-education-hub.github.io/operating-systems/, by now we know the preferred method for transferring text is TCP.

Unlike UDP, TCP is connection-oriented. +This is why, in the example with the browser, we kept using the word connection. +What's different from UDP is that this connection is bidirectional, so we can both send() and receive (recv()) data from it. +Notice that the syscalls have changed. +We were using sendto() and recvfrom() for UDP, and now we're using send() and recv() for TCP. +And yes, despite the fact that we're using Python, these are syscalls. +You saw them in C when you solved the challenge.

Server vs Client​

When discussing servers and clients, the server is the passive actor. +It may have some data and wait for clients to ask for this data. +Or it may require some data and wait for clients to send it. +Either way, it is listening for connections.

The client is the active actor, being the one who initiates the connection.

Quiz

Establishing the Connection​

There is a series of steps that a client and most importantly a server must take to establish a TCP connection.

Steps Taken by the Server​

  1. Call socket() to create a socket.

  2. bind() this socket to an IP and a port. +Up to now, this is nothing new. +However, with TCP sockets, bind() doesn't automatically make them listen for connections.

  3. Call listen() to make the socket ready to receive connections.

  4. Call accept() to set up one connection initiated by a client. +From now, the connection is established. +accept() returns a new socket, which will be further used for communication between the server and the client.

  5. Exchange messages with the client. +The server can both read messages from the client and send responses back using the same socket returned by accept(). +When done, close() the socket returned by accept and repeat from step 4.

Steps Taken by the Client​

  1. Call socket() to create a socket.

  2. Use connect() to... you guessed it: connect to the server. +This step is like an "answer" to step 4 from the server. +We can say that the server accept()s a connect() request from the client.

  3. Exchange messages with the server. +The client can both send messages to the server and read responses from it using the same socket created during step 1.

Below is an image summarising the steps above:

Steps to Establish a Connection

Practice: Client​

Navigate to support/client-server/. +Here you will find a minimalistic server implementation in server.py.

  1. Read the code and identify the steps outlined above.

  2. Now fill in the code in client.c or client.py to be able to send messages to the server and receive responses from it. +Run multiple clients.

  3. Run the code, then run lsof and netstat in other terminals to identify the file descriptor corresponding to the listening file descriptor.

Practice: Just a Little Bit More Deluge​

We've already said that Deluge uses an abstraction over TCP to handle socket operations, so we don't have the luxury of seeing it perform remote I/O "manually". +However, there are a few instances where Deluge uses socket operations itself, mostly for testing purposes.

Deluge saves its PIDs (it can spawn multiple processes) and ports in a file. +Find the is_daemon_running() method. +This method uses the aforementioned file to check if a given process is Deluge or some other process. +To do this, it connect()s to that process's socket. +If it can't, then that process is not Deluge. +Otherwise, it is Deluge and that connection is immediately closed :)) +This may sound like a stupid way of checking whether a process is Deluge or not, but if it's stupid and it works, then it's not stupid!

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/file-descriptors/index.html b/17/Lab/IO/file-descriptors/index.html new file mode 100644 index 0000000000..57e8961374 --- /dev/null +++ b/17/Lab/IO/file-descriptors/index.html @@ -0,0 +1,80 @@ + + + + + +File Descriptors | Operating Systems + + + + +
+
Skip to main content

File Descriptors

After running the code in the "File Handlers" section, you saw that open() returns a number. +This number is a file descriptor. +Run the code above multiple times. +You'll always get file descriptor 3. +We can already tell these numbers have some meaning and aren't just some seemingly random numbers, like PIDs were.

You've most likely used file descriptors before without knowing. +Remember the following concepts:

  • stdin (standard input)
  • stdout (standard output)
  • stderr (standard error)

When you wanted to run a shell command and ignore its errors, you wrote something like:

student@os:~$ ls 2> /dev/null

The command above uses 2> /dev/null to redirect stderr to /dev/null.

Quiz

Good, so we know that stderr is file descriptor 2. +The code in support/file-descriptors/open_directory.c opened the directory as file descriptor 3. +So what are file descriptors 0 and 1? +They are stdin and stdout, respectively.

Now we know that the basic I/O streams of a process correspond to specific file descriptors. +But what is a file descriptor exactly? +From the application's point of view, it is a positive integer acting as a unique identifier for an I/O channel, such as a file. +To the operating system, a file descriptor is an index. +Each process has what's called a file descriptor table, which is just a fancy name for an array of pointers. +A file descriptor is an index in this array. +Each element in this array points towards a structure in the kernel's memory that represents an I/O resource. +This structure is called the open file structure.

To illustrate all this information, look at the image below. +All data structures shown below are stored in the Kernel's memory area and are transparent to the user space. +And remember that the file descriptor table is bound to the process, so all its threads share the same file descriptors.

File Descriptors

To find out more about the contents of these structures, check out this section in the Arena

Creating New File Descriptors​

We already know that each process gets 3 file descriptors "by default":

  • stdin (standard input): 0
  • stdout (standard output): 1
  • stderr (standard error): 2

To create new file descriptors (i.e. open new files), a process can use the open() system call. +It receives the path to the file, some flags which are akin to the mode string passed to fopen(). +An optional mode parameter that denotes the file's permissions if the open must create it can also be provided. +We'll revisit open()'s mode parameter in the future.

From now, you should consult open()'s man page whenever you encounter a new argument to this syscall. +Navigate to support/file-descriptors/open_read_write.c. +The function open_file_for_reading() calls open() with the O_RDONLY flag, which is equivalent to opening the file with fopen() and setting "r" as the mode. +This means read-only. +Note that the file descriptor we get is 3, just like before.

Then read_from_file() reads bytes_to_read bytes from this file. +For this, it uses the read() syscall. +It returns the number of bytes that were read from the file.

Notice that the code doesn't simply call read(fd, buff, bytes_to_read). +Instead, it uses a while loop to read data from the file.

The return value of read() may be less than bytes_to_read if there are not enough bytes available or if the operation is interrupted by a signal. +A return value between 0 and bytes_to_read is not enough to decide whether we should stop reading. +To determine this, we make another read() call, which will return 0 if the cursor is already at EOF (end of file).

The same goes for write(): its return value may differ from the intended number of bytes to write. +Partial writes should also be handled at the application level, and the way to do this is by using loops.

Remember: +It is mandatory that we always use read() and write() inside while loops. +Higher-level functions like fread() and fwrite() also use while loops when calling read() and write() respectively.

Practice: Open a File for Writing​

Follow the code in open_file_for_reading() and fill in the function open_file_for_writing(). +The file write_file.txt doesn't exist. +open() should create it. +Use the open()'s man page to find the flags you require. +Do you require anything else?

At this point, depending on what flags you passed to open(), a few things might happen. +Work your way through the errors until successfully create and open the file. +The open() syscall should return file descriptor 4.

Now verify the file. +This part may be different on your system. +Delete write_file.txt and rerun open_read_write a few times. +Each time, check the permissions of write_file.txt. +They may be different between runs, or they may always be the same. +Anyway, they are likely not the default permissions for a regular file (rw-r--r--). +It is not mandatory that you get the same output as below.

student@os:~/.../lab/support/file-descriptors$ ls -l
total 11
drwxrwxr-x 1 student student 4096 Nov 20 18:26 ./
drwxrwxr-x 1 student student    0 Nov 20 14:11 ../
-rw-rw-r-- 1 student student   46 Nov 20 17:27 .gitignore
-rw-rw-r-- 1 student student  125 Nov 20 18:26 Makefile
-rw-rw-r-- 1 student student  396 Nov 20 11:27 open_directory.c
-rw-rw-r-- 1 student student 2210 Nov 20 17:24 open_read_write.c
-rw-rw-r-- 1 student student  34 Nov 20 18:26 read_file.txt
---------- 1 student student  45 Nov 20 18:26 write_file.txt

Quiz

Remember: +It is mandatory that we pass a mode argument to open() when using the O_CREAT flag.

Now pass some sensible mode argument to open(), such as 0644 (for rw-r--r-- permissions).

Practice: Write to the File​

Follow the example in read_from_file() to fill in write_to_file(). +Remember to use a loop to make sure your data is fully written to the file.

  1. Use cat write_file.txt to check if the file contains the right data.

  2. Use open_file_for_reading() and read_from_file() to reopen write_file.txt for reading and read the bytes you've just written to it.

Replace or Truncate?​

Practice: Write Once More​

Change the message written to write_file.txt by support/file-descriptors/open_read_write.c to a shorter one. +It is important that the new message be shorter than the first one. +Now recompile the code, then run it, and then inspect the contents of the write_file.txt file.

If the new message were "Something short", the contents of write_file.txt should be:

student@os:~/.../lab/support/file-descriptors$ cat ../../support/file-descriptors/write_file.txt
Something shorte_file.txt: What's up, Doc?

Note that the final bytes of the previous text remain unchanged. +The new message was simply written on top of the old one.

Now let's do a quick test. +We haven't talked about how redirections work in the terminal (we'll get there, step by step), but you can imagine that if you type ls > file.txt, file.txt has to be opened at some point. +Let's write data to a file twice and observe the behaviour:

student@os:~/.../lab/support/file-descriptors$ ls -l > file.txt

student@os:~/.../lab/support/file-descriptors$ cat file.txt
total 6
-rw-rw-r-- 1 student student    0 Nov 20 21:11 file.txt
-rw-rw-r-- 1 student student  125 Nov 20 18:26 Makefile
-rw-rw-r-- 1 student student  396 Nov 20 21:10 open_directory.c
-rw-rw-r-- 1 student student 2300 Nov 20 20:51 open_read_write.c
-rw-rw-r-- 1 student student   34 Nov 20 18:26 read_file.txt
-rw-r--r-- 1 student student   45 Nov 20 20:56 write_file.txt

student@os:~/.../lab/support/file-descriptors$ ls > file.txt

student@os:~/.../lab/support/file-descriptors$ cat file.txt
file.txt
Makefile
open_directory.c
open_read_write.c
read_file.txt
write_file.txt

The second output is shorter than the first, yet the first output is no longer present in the file after the second ls. +How come? +Well, the reason is another flag being passed to open(): O_TRUNC. +At this point, you should be accustomed to looking for this flag in open()'s man page. +Go ahead and do it.

Quiz 1

Quiz 2

Practice: Close'em All​

Just like you use open() to create new file descriptors, you can use close() to destroy them. +This clears and frees the open file structure to which that entry in the file descriptor table is pointing. +Use close() on the file descriptors you've opened so far in support/file-descriptors/open_read_write.c.

Note that you don't have to close file descriptors 0, 1 and 2 manually. +The standard streams are meant to stay alive throughout the lifetime of the process. +Just like calling free() on a malloc()-ed pointer, calling close() is not really necessary. +When a process terminates, the OS closes all its file descriptors the same way it frees all its memory.

And keeping this comparison with malloc() and free(), closing file descriptors is important when they are created inside a loop, as the file descriptor table's size is limited.

File Handling Conclusion: libc vs syscalls​

Up to now, we can draw some parallels between fopen() and open(). +While fopen() allows the usage of high-level functions such as fread() and fwrite(), which, among other things, use while loops to ensure they always read the required number of bytes, the libc-specific API is not generic enough.

In the following sections, we'll use file descriptors and read() and write() to interact with some inter-process-communication mechanisms, such as pipes.

The table below shows the higher level API provided by libc and the syscalls it relies on. +As usual, use the man pages when in doubt about either of them.

libcsyscall
fopen()open()
fread()read()
fwrite()write()
fseek()lseek()
fclose()close()

So for most equivalents, just remove the leading f when moving from the libc function to the underlying syscall.

For a quick recap of the flags we've discussed so far, take a look at the following table. +But don't bother memorising it. +You can find it any time in by typing man fopen in your terminal.

fopen() modeopen() flag
"r"O_RDONLY
"w"O_WRONLY │ O_CREAT │ O_TRUNC
"a"O_WRONLY │ O_CREAT │ O_APPEND
"r+"O_RDWR
"w+"O_RDWR │ O_CREAT │ O_TRUNC
"a+"O_RDWR │ O_CREAT │ O_APPEND
+ + + + \ No newline at end of file diff --git a/17/Lab/IO/file-handlers/index.html b/17/Lab/IO/file-handlers/index.html new file mode 100644 index 0000000000..4d16b88ae1 --- /dev/null +++ b/17/Lab/IO/file-handlers/index.html @@ -0,0 +1,40 @@ + + + + + +File Handling | Operating Systems + + + + +
+
Skip to main content

File Handling

You've most likely had to deal with files in the past. +Go to support/simple-file-operations and run a most basic command: +cat file.txt:

student@os:~/.../lab/support/simple-file-operations$ cat file.txt
OS Rullz!

But how does cat actually "reach" the file, then read its contents, then print them to standard output? +This is part of what we're going to learn.

Reaching the File​

To manipulate the file (read its contents, modify them, change its size etc.), each process must first get a handler to this file. +Think of this handler as an object by which the process can identify and refer to the file.

Now take a look at the code examples in support/simple-file-operations. +Each of them reads the contents of file.txt, modifies them, and then reads the previously modified file again. +Use make to compile the C code, and make java-file-operations to compile the Java code.

Now run the programs repeatedly in whatever order you wish:

student@os:~/.../lab/support/simple-file-operations$ python3 file_operations.py
File contents are: OS Rullz!
Wrote new data to file
File contents are: Python was here!

student@os:~/.../lab/support/simple-file-operations$ ./file_operations  # from the C code
File contents are: Python was here!
Wrote new data to file
File contents are: C was here!

student@os:~/.../lab/support/simple-file-operations$ java FileOperations
File contents are: Python was here!
Wrote new data to file
File contents are: Java was here!

Note that each piece of code creates a variable, which is then used as a handler.

Quiz

Limitations of High-level File Handlers​

Everything in Linux is a file. +This statement says that the Linux OS treats every entry in a file system (regular file, directory, block device, char device, link, UNIX socket) as a file. +This is very convenient for creating a unified interface that deals with all these files. +We would like our file handlers to also be able to handle all types of files.

Let's try this. +Navigate to support/simple-file-operations/directory_operations.c. +Read the code. +It does something very simple: +it attempts to open the dir directory the same way file_operations.c tried to open file.txt. +Compile and run the code.

student@os:~/.../lab/support/simple-file-operations$ ./directory_operations
18:18:11 FATAL directory_operations.c:14: fopen: Is a directory

The error message is crystal clear: we cannot use fopen() on directories. +So the FILE structure is unsuited for directories. +Therefore, this handler is not generic enough for a regular Linux filesystem.

To get to the directory, we need to use a lower-level function. +Let's take a look at the syscall used by fopen().

Quiz

We will use a simpler syscall, called open(). +In fact, open() is a wrapper over openat(). +Navigate to support/file-descriptors/directory_open.c. +Inspect, compile and run the code.

student@os:~/.../lab/support/file-descriptors$ ./open_directory
Directory file descriptor is: 3

We can now see that the open() syscall is capable of also handling directories, so it's closer to what we want. +Note that it is rather uncommon to use open() for directories. +Most of the time, opendir() is used instead. +But what does it return? +Find out in the "File Descriptors" section.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/file-mappings/index.html b/17/Lab/IO/file-mappings/index.html new file mode 100644 index 0000000000..1e4dd40464 --- /dev/null +++ b/17/Lab/IO/file-mappings/index.html @@ -0,0 +1,45 @@ + + + + + +File Mappings | Operating Systems + + + + +
+
Skip to main content

File Mappings

Mapping a file to the VAS of a process is similar to how shared libraries are loaded into the same VAS. +It's a fancier way of saying that the contents of a file are copied from a given offset within that file to a given address. +What's nice about this is that the OS handles all offsets, addresses and memory allocations on its own, with a single highly versatile syscall: mmap().

The Return of mmap()​

Remember that the .text, .rodata and .data sections of libraries are present in the VAS of any Linux process:

student@os:~$ sleep 1000 &  # start a `sleep` process in the background
[1] 17579

student@os:~$ cat /proc/$(pidof sleep)/maps
55b7b646f000-55b7b6471000 r--p 00000000 103:07 6423964                   /usr/bin/sleep
55b7b6471000-55b7b6475000 r-xp 00002000 103:07 6423964                   /usr/bin/sleep
55b7b6475000-55b7b6477000 r--p 00006000 103:07 6423964                   /usr/bin/sleep
55b7b6478000-55b7b6479000 r--p 00008000 103:07 6423964                   /usr/bin/sleep
55b7b6479000-55b7b647a000 rw-p 00009000 103:07 6423964                   /usr/bin/sleep
55b7b677c000-55b7b679d000 rw-p 00000000 00:00 0                          [heap]
7fe442f61000-7fe44379d000 r--p 00000000 103:07 6423902                   /usr/lib/locale/locale-archive
7fe44379d000-7fe4437bf000 r--p 00000000 103:07 6432810                   /usr/lib/x86_64-linux-gnu/libc-2.31.so
7fe4437bf000-7fe443937000 r-xp 00022000 103:07 6432810                   /usr/lib/x86_64-linux-gnu/libc-2.31.so
7fe443937000-7fe443985000 r--p 0019a000 103:07 6432810                   /usr/lib/x86_64-linux-gnu/libc-2.31.so
7fe443985000-7fe443989000 r--p 001e7000 103:07 6432810                   /usr/lib/x86_64-linux-gnu/libc-2.31.so
7fe443989000-7fe44398b000 rw-p 001eb000 103:07 6432810                   /usr/lib/x86_64-linux-gnu/libc-2.31.so
7fe44398b000-7fe443991000 rw-p 00000000 00:00 0
7fe4439ad000-7fe4439ae000 r--p 00000000 103:07 6429709                   /usr/lib/x86_64-linux-gnu/ld-2.31.so
7fe4439ae000-7fe4439d1000 r-xp 00001000 103:07 6429709                   /usr/lib/x86_64-linux-gnu/ld-2.31.so
7fe4439d1000-7fe4439d9000 r--p 00024000 103:07 6429709                   /usr/lib/x86_64-linux-gnu/ld-2.31.so
7fe4439da000-7fe4439db000 r--p 0002c000 103:07 6429709                   /usr/lib/x86_64-linux-gnu/ld-2.31.so
7fe4439db000-7fe4439dc000 rw-p 0002d000 103:07 6429709                   /usr/lib/x86_64-linux-gnu/ld-2.31.so
7fe4439dc000-7fe4439dd000 rw-p 00000000 00:00 0
7ffd07aeb000-7ffd07b0c000 rw-p 00000000 00:00 0                          [stack]
7ffd07b8b000-7ffd07b8e000 r--p 00000000 00:00 0                          [vvar]
7ffd07b8e000-7ffd07b8f000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 --xp 00000000 00:00 0                  [vsyscall]

How does the content of those files get there? +Below you can see how libc is loaded, i.e. mapped into the VAS of an ls process.

student@os:~$ strace ls
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
[...]
mmap(NULL, 2037344, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fb313c9c000
mmap(0x7fb313cbe000, 1540096, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x22000) = 0x7fb313cbe000
mmap(0x7fb313e36000, 319488, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x19a000) = 0x7fb313e36000
mmap(0x7fb313e84000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7fb313e84000

For a quick reminder about mmap(), its 5th argument is the file descriptor from where we want to copy the data to the RAM. +The 6th argument is the offset within the file from where to start copying.

What About read()?​

Let's get the elephant out of the room: why use mmap() instead of read() for dynamic libraries?

The short answer: for efficiency and convenience.

read() is a rather blunt approach for loading libraries into memory, as it does not take advantage of their particularities. +There are two main points to consider: how is memory accessed and when is it modified. +Dynamic libraries are mostly non-writable (.data section being the exception) and arbitrarily accessed. +Reading and storing a library whose content might end up mostly unused does not sound like the right thing to do.

On the other hand, mmap() makes use of demand paging. +As a reminder, demand paging is a technique where memory is allocated, but each memory page is loaded when needed. +Furthermore, mmap() will check if the library was already loaded into memory, and if so, it will not use additional memory, thus reducing the startup time. +As you remember from the Compute chapter, processes sharing memory will lead to copy-on-write, but this is far better than creating copies of the non-writable sections as read() does.

Even without considering any OS tweaks, using read() to load dynamic libraries would require extra work. +This is because allocating memory at the correct address would still require the use of mmap(). +Additionally, setting the appropriate permissions to avoid security issues would require the use of mprotect(). +Therefore, using mmap() alone is a more convenient way for loading dynamic libraries.

Practice: Copy a File​

If mmap() is good for copying files, let's use it like this. +Navigate to support/file-mappings/mmap_cp.c. +It copies the contents of a file to another by mmap()-ing each of them to its VAS and then simply copying the bytes from one mapping to another as if copying the contents of 2 arrays.

  1. Fill in the TODOs so that the program correctly copies the contents of the input file to the output file. +Use make test-file to generate a 1MB file with random data. +You can use this for debugging and diff to test whether the input and output files differ (they shouldn't):
student@os:/.../support/file-mappings$ ./mmap_cp test-file.txt output.txt

student@os:/.../support/file-mappings$ diff test-file.txt output.txt

student@os:/.../support/file-mappings$
  1. Uncomment the calls to wait_for_input() and rerun the program. +While the program is waiting, open another terminal and run cat /proc/$(pidof mmap_cp)/maps and see the mapped files.

Practice: I Am Speed​

Now we can copy files using mmap(). +The code is rather short and convenient to write. +Its disadvantage is that we have to mmap() the 2 files entirely of the VAS of the process. +While this may be alright for small files, for larger ones we simply may not have enough RAM. +What if we had to copy a 500GB file?

Let's look at what the cp tool uses for copying.

Quiz 1

OK, so we have a competition: our mmap_cp versus cp. +Run the script benchmark_cp.sh to measure which of the 2 implementations is faster.

Quiz 2

Now take a look at the benchmark_cp.sh script. +You might get a little confused about the following command: echo 3 > /proc/sys/vm/drop_caches. +Head over to this section in the Arena to find out what it's about.

Conclusion​

So by using mmap(), we pay the price for loading the file into memory once, but then all read()s, write()s and especially seek()s will be faster. +This is akin to treating the file as a regular byte array. +So file mappings are especially useful for randomly accessing file data. +Accessing the i-th byte in a file becomes the same as mapping[i], which is obviously more efficient than calling lseek(). +File mappings are also useful when we have to overwrite existing data multiple times or when we read certain chunks multiple times.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/index.html b/17/Lab/IO/index.html new file mode 100644 index 0000000000..6f2c9e7bae --- /dev/null +++ b/17/Lab/IO/index.html @@ -0,0 +1,16 @@ + + + + + +IO | Operating Systems + + + + +
+
Skip to main content

IO

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/io-internals/index.html b/17/Lab/IO/io-internals/index.html new file mode 100644 index 0000000000..7da6e2be24 --- /dev/null +++ b/17/Lab/IO/io-internals/index.html @@ -0,0 +1,64 @@ + + + + + +I/O Internals | Operating Systems + + + + +
+
Skip to main content

I/O Internals

Now, we will take a short look at how the file descriptors you've just learnt about are handled in libc. +The Software Stack chapter has taught us that applications generally interact with libraries which expose wrappers on top of syscalls. +The most important library in a POSIX system (such as Linux) is libc. +Among many others, it provides higher-level abstractions over I/O-related syscalls.

Musl (read just like "muscle") is a lightweight implementation of libc, which exposes the same API that you have used so far, while also being fit for embedded and OS development. +For example, Unikraft unikernels may use musl.

First, it provides a struct that groups together multiple data that is necessary when handling files. +We know from the example in support/simple-file-operations/file_operations.c that the file handler employed by libc is FILE *. +FILE is just a typedef for struct _IO_FILE. +Here are the most important fields in struct _IO_FILE:

struct _IO_FILE {
        int fd;         /* File descriptor */

        unsigned flags; /* Flags with which `open()` was called */

        int mode;       /* File permissions; passed to `open()` */

        off_t off;      /* File offset from where to read / write */

        /**
         * Internal buffer used to make fewer costly `read()`/`write()`
         * syscalls.
         */
        unsigned char *buf;
        size_t buf_size;

        /* Pointers for reading and writing from/to the buffer defined above. */
        unsigned char *rpos, *rend;
        unsigned char *wend, *wpos;

        /* Function pointers to syscall wrappers. */
        size_t (*read)(FILE *, unsigned char *, size_t);
        size_t (*write)(FILE *, const unsigned char *, size_t);
        off_t (*seek)(FILE *, off_t, int);
        int (*close)(FILE *);

        /* Lock for concurrent file access. */
        volatile int lock;
};

As you might have imagined, this structure contains the underlying file descriptor, the mode (read, write, truncate etc.) with which the file was opened, as well as the offset within the file from which the next read / write will start.

Libc also defines its own wrappers over commonly-used syscalls, such as read(), write(), close() and lseek(). +These syscalls themselves need to be implemented by the driver for each file system. +This is done by writing the required functions for each syscall and then populating this structure with pointers to them. +You will recognise quite a few syscalls: open(), close() read(), write(), mmap() etc.

IO Optimisations​

You saw this hierarchy during the Data lecture:

Memory Hierarchy

It says that while the disk can store lots of data, it does so at the cost of speed. +When we say speed, we mean the rate at which we can read/write data from/to the disk, i.e. the maximum number of bytes transferred per unit of time. +This means that read() and write() syscalls (or their various corresponding library calls) are slow and cause performance bottlenecks. +More often than not, it is the I/O component that drags down the performance of an otherwise fast application. +And what's worse, the further the "destination" of the I/O operation (file on the disk or host on the Web) is, the more time it takes to transfer data to and from it.

On the other hand, as we've already established, the I/O component defines how we interact with an app. +If we want it to be responsive and to do something useful, most likely, the I/O is the key.

So I/O is crucial for most applications, yet it is also the slowest...

Sad Pepe

But fear not! +There are countless optimisations out there aimed precisely at bridging the speed gap between the memory and the disk.

I/O Buffering​

Going back to our initial example with struct _IO_FILE from Musl, we can see some more fields:

unsigned char *buf;
size_t buf_size;
unsigned char *rpos, *rend;
unsigned char *wend, *wpos;

Given the number unsigned char * fields we have in this structure, it seems there is some heavy buffering going on. +But what is it?

Practice: printf() Buffering​

  1. Navigate to support/buffering/printf_buffering.c. +Those printf() calls obviously end up calling write() at some point. +Run the code under strace.

Quiz

Since there is only one write() syscall despite multiple calls to printf(), it means that the strings given to printf() as arguments are kept somewhere until the syscall is made. +That somewhere is precisely that buffer inside struct _IO_FILE that we highlighted above. +Remember that syscalls cause the system to change from user mode to kernel mode, which is time-consuming. +Instead of performing one write() syscall per call to printf(), it is more efficient to copy the string passed to printf() to an internal buffer inside libc (the unsigned char *buf from above) and then at a given time (like when the buffer is full for example) write() the whole buffer. +This results in far fewer write() syscalls.

  1. Now, it is interesting to see how we can force libc to dump that internal buffer. +The most direct way is by using the fflush() library call, which is made for this exact purpose. +But we can be more subtle. +Add a \n in some of the strings printed in support/buffering/printf_buffering.c. +Place them wherever you want (at the beginning, at the end, in the middle). +Recompile the code and observe its change in behaviour under strace.

Quiz

Now we know that I/O buffering does happen within libc. +If you need further convincing, check out the Musl implementation of fread(), for example. +It first copies the data previously saved in the internal buffer:

if (f->rpos != f->rend) {
        /* First exhaust the buffer. */
        k = MIN(f->rend - f->rpos, l);
        memcpy(dest, f->rpos, k);
        f->rpos += k;
        dest += k;
        l -= k;
}

Then, if more data is requested and the internal buffer isn't full, it refills it using the internal read() wrapper. +This wrapper also places the data inside the destination buffer.

Practice: Buffering Performance​

Up to now, it's pretty clear what I/O buffering is about. +Let's see what kind of a performance increase it brings. +We'll look at an extreme example. +Navigate to support/buffering/no_buffering.c. +This code either writes or reads the contents of a file one byte at a time using a syscall for each of them.

Use the benchmark_buffering.sh script to compare the no_buffering and libc_buffering implementations. +Below are some possible results. +Yours are likely going to be different:

student@os:/.../support/buffering$
======== Testing no_buffering ========
Testing no_buffering read...
Read 1048576 bytes from test-file.txt in 717 ms
Testing no_buffering write...
Wrote 1048576 bytes to test-file.txt in 19632 ms
======== Testing libc_buffering ========
Testing libc_buffering read...
Read 1048576 bytes from test-file.txt in 14 ms
Testing libc_buffering write...
Wrote 1048576 bytes to test-file.txt in 38 ms

So buffering brings a 98% improvement for reading and a 99.8% improvement for writing! +This is massive! +Yes, this is an extreme example, but it goes a long way to show how powerful I/O buffering can be. +Now, the question is, "Can YOU do better?"

Practice: DIY Buffering​

  1. Navigate to support/buffering/diy_buffering.c. +diy_fread() already defines a minimalistic implementation of fread(). +Use it as a starting point to implement diy_write() as an implementation of fwrite().

  2. Run benchmark_buffering.sh to compare the performance of your implementation with that of libc. +Did you beat it?

Conclusion​

I/O Buffering is a ubiquitous optimisation in all libc implementations. +It is so powerful that the kernel uses it as well. +This means that the kernel also reads more bytes than it's requested and stores the remainder in an internal buffer, just like libc. +This concept is known as double buffering. +In the following section, we will see how to make use of this internal buffering to optimise network requests.

Notice that the script in support/buffering/benchmark_buffering.sh also uses echo 3 > /proc/sys/vm/drop_caches. +That section in the Arena that we mentioned earlier is becoming even more interesting.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/io-multiplexing/index.html b/17/Lab/IO/io-multiplexing/index.html new file mode 100644 index 0000000000..5a6b99c6ad --- /dev/null +++ b/17/Lab/IO/io-multiplexing/index.html @@ -0,0 +1,29 @@ + + + + + +I/O Multiplexing | Operating Systems + + + + +
+
Skip to main content

I/O Multiplexing

I/O multiplexing is the ability to serve multiple I/O channels (or anything that can be referenced via a file descriptor / handle) simultaneously. +If a given application, such a server, has multiple sockets on which it serves connection, it may be the case that operating on one socket blocks the server. +One solution is using asynchronous operations, with different backends. +The other solution is using I/O multiplexing.

With I/O multiplexing, the API provides a structure / an array to list all used channels. +And then it provides an API to operate on that channel. +And a (blocking) function to retrieve the channel that is ready for interaction (or channels). +So instead of waiting for a given channel (while others may be ready), you now simultaneously wait on all channels, and the ready channel is returned by the function.

The classical functions for I/O multiplexing are select and poll. +Due to several limitations, modern operating systems provide advanced (non-portable) variants to these:

Note that I/O multiplexing is orthogonal to asynchronous operations. +You could tie them together if the completion of the asynchronous operation sends a notification that can be handled via a file descriptor / handle. +This is the case with Windows asynchronous I/O (called overlapped I/O).

Practice​

Enter the multiplex/ directory. +See the implementation of an epoll()-based server in epoll_echo_server.c.

  1. Build the server:

    student@os:~/.../lab/support/multiplex$ make
    gcc -g -Wall -Wextra -I../../../../../common/makefile/..  -c -o epoll_echo_server.o epoll_echo_server.c
    gcc -g -Wall -Wextra -I../../../../../common/makefile/..  -c -o ../../../../../common/makefile/../utils/log/log.o ../../../../../common/makefile/../utils/log/log.c
    gcc -g -Wall -Wextra -I../../../../../common/makefile/..  -c -o ../../../../../common/utils/sock/sock_util.o ../../../../../common/utils/sock/sock_util.c
    gcc epoll_echo_server.o ../../../../../common/makefile/../utils/log/log.o ../../../../../common/utils/sock/sock_util.o  -o epoll_echo_server

    And run it:

    student@os:~/.../lab/support/multiplex$ ./epoll_echo_server
    08:41:07 INFO  epoll_echo_server.c:252: Server waiting for connections on port 42424

    It listens for connection on port 42424.

    Connect using netcat on another console and send messages:

    $ nc localhost 42424
    aaa
    aaa
    bbb
    bbb

    Keep the connection open and, on the third console, initiate another netcat connection. +The server is now multiplexing both connections.

  2. Create a script and / or a program to exercise the server. +Create many connections to the server and continuously send messages to the server. +See it multiplex the I/O channels (one for each connection - actually two: one for receiving and one for sending).

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/local-io-in-action/index.html b/17/Lab/IO/local-io-in-action/index.html new file mode 100644 index 0000000000..b9a357b79b --- /dev/null +++ b/17/Lab/IO/local-io-in-action/index.html @@ -0,0 +1,38 @@ + + + + + +Local I/O in Action | Operating Systems + + + + +
+
Skip to main content

Local I/O in Action

Most of the time, file handling is a simple operation from the perspective of the application. +We've already looked at the Deluge Bittorrent client written during the Data lab. +We'll examine it once more, but this time from the perspective of its I/O operations. +Being a Bittorrent client, we expect it to deal with I/O in 4 ways. +There are 2 "local" types of operations:

  • reading local files from the disk to be sent to other clients

  • writing the previously downloaded files to the disk

Simple, right? +Now we'll only look at the "local" part: reading and writing data to/from the disk. +The remaining 2 actions are about network-related operations.

  • requesting files from other stations on the Web. +Those files are then retained in Deluge's memory while they're being downloaded

  • sending (parts of) files to other clients. +Those files are first read from the disk, retained in memory for a while, and then sent over the internet.

Networking is complementary to local I/O and we'll discuss it starting from a future section.

Local I/O in Deluge​

If you haven't done so in the Data lab, clone the Deluge repository.

Writing Data to Files​

Now find the write_torrentfile() function in the repository.

It defines an inner function called write_file(), which does the actual writing. +Notice that it's very similar to the example we saw in support/simple-file-operations/file_operations.py:

with open(filepath, 'wb') as save_file:
    save_file.write(filedump)

The code above is the same as:

save_file = open(filepath, 'wb')
save_file.write(filedump)
save_file.close()

The with keyword is just a context manager that makes sure save_file is also closed when its "body" finished. +Writing the data is done simply by calling the write() function. +As you've probably figured out, there's an entire software stack beneath this write() function that sends the data first to libc and then to the kernel. +Furthermore, the kernel itself has its own separate stack for handling I/O requests.

Software Stacks Everywhere

Error Handling​

What is noteworthy about this short snippet is the (slightly) bigger picture - the error handling:

try:
    with open(filepath, 'wb') as save_file:
        save_file.write(filedump)
except OSError as ex:
    log.error('Unable to save torrent file to: %s', ex)

This is similar to the DIE() macro you have seen (and used, right?) throughout the labs, but less brutal. +While the DIE() macro kills the program upon encountering an error, Deluge's try - except approach simply logs an error and continues. +As you might imagine, there is no silver bullet when it comes to error handling. +What is important is that errors are somehow accounted for and not ignored.

This is especially true for I/O-related errors. +Because I/O handling means dealing with peripherals or devices outside the typical CPU-RAM interaction. +The more devices, the higher the complexity. +The higher the complexity, the higher the likelihood of errors.

Quiz

Reading Data from Files​

Now find the load_resume_data_file() function in the Deluge source code. +It is used when (re)starting to seed existing files to peers.

The function looks through all possible copies and backups of the file and loads them to a "torrent" format given by libtorrent. +For this, it calls lt.bdecode().

As before, error handling is important:

try:
    with open(_filepath, 'rb') as _file:
        resume_data = lt.bdecode(_file.read())
except (OSError, EOFError, RuntimeError) as ex:
    if self.torrents:
        log.warning('Unable to load %s: %s', _filepath, ex)
    resume_data = None

So now we know how Deluge handles local files. +But we still don't know how it gets those files from peers. +We'll find out how in the following sections.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/networking-101/index.html b/17/Lab/IO/networking-101/index.html new file mode 100644 index 0000000000..085bf78eee --- /dev/null +++ b/17/Lab/IO/networking-101/index.html @@ -0,0 +1,52 @@ + + + + + +Networking 101 | Operating Systems + + + + +
+
Skip to main content

Networking 101

In this section, we will briefly explore how networking works in general, from the perspective of the application. +Understanding the details of it, however, is beyond the scope of this course.

The main protocols used by applications are User Datagram Protocol (UDP) and Transmission Control Protocol (TCP). +We can specify this protocol when creating a socket.

UDP​

UDP is the simpler of the two protocols above. +It simply states that the sender should pass the data over to the receiver specified by an IP and a port. +The scripts in support/send-receive/ both used UDP. +You can tell by the fact they both created their sockets like so:

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

The socket.SOCK_DGRAM argument stands for "datagram" and specifies UDP.

It doesn't care whether the receiver has got all the data, whether it was corrupted or dropped altogether by some router along the way.

To prove this, rerun support/send-receive/receiver.py, then run support/send-receive/sender.py, type "exit" and then run sender.py again. +You'll see no error from the sender because whether the message reaches its destination or not is not important for UDP.

So far UDP might seem rather useless. +It doesn't confirm whether a message was received correctly or not, so why use it? +Well, exactly because its mechanism is so simple, UDP is fast. +Therefore, it is used by many real-time services, such as for streaming or video calls where if a frame drops or is incorrect, it doesn't really matter that much since it will immediately be replaced with the next frame, thus masking the error.

TCP​

TCP is the polar opposite of UDP. +TCP makes sure the data is given to the application correctly by performing error checks on the receiving end and then retransmitting any incorrect or missing messages. +For this reason, TCP is good for applications that require precise data, such as banking applications, static images or text. +The cost of correctness, however, is transmission speed.

Practice: Encapsulation Example: Deluge Revived​

Quiz

You haven't forgotten about our favourite Bittorrent clint, Deluge, have you? +It implements its own protocol for transferring data. +It is built on top of TCP using the ITCPTransport interface.

  1. In the Deluge codebase, find the class DelugeTransferProtocol and read the docstring comment about the format of its messages.

  2. Follow the code in the transfer_message() method of the DelugeTransferProtocol class and see the message variable conform to the message format you've just read about. +self.transport simply refers to the underlying TCP interface. +So self.transport.write() invokes the whole socket creation boilerplate necessary to send data via TCP. +We'll discuss it in the next section.

This is what protocols are mostly about. +They describe which fields go where within a message and how they should be interpreted. +For example, this protocol says that the body (the data itself) should be compressed before being sent. +This makes a lot of sense, as sending less data over the network results in lower latencies. +However, if encryption is too time-consuming, it may not be worth it. +There are compromises to be made everywhere.

Local TCP and UDP Services​

To get a full list of all network-handling processes in your system together with the protocols they're using, we can use the netstat with the -tuanp arguments. +-tuanp is short for -t -u -a -n -p, which stand for:

  • -t: list processes using the TCP protocol
  • -u: list processes using the UDP protocol
  • -a: list both servers and clients
  • -n: list IPs in numeric format
  • -p: show the PID and name of each program
student@os:~$ sudo netstat -tunp
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      1057/sshd: /usr/sbi
tcp        0      0 127.0.0.1:6463          0.0.0.0:*               LISTEN      3261/Discord --type
tcp        0      0 192.168.100.2:51738     162.159.128.235:443     ESTABLISHED 3110/Discord --type
tcp        0      0 192.168.100.2:43694     162.159.129.235:443     ESTABLISHED 3110/Discord --type
tcp        0      0 192.168.100.2:56230     54.230.159.113:443      ESTABLISHED 9154/firefox
tcp        0      0 192.168.100.2:38096     34.107.141.31:443       ESTABLISHED 9154/firefox
tcp        0      0 192.168.100.2:42462     34.117.237.239:443      ESTABLISHED 9154/firefox
tcp        0      0 192.168.100.2:41128     162.159.135.234:443     ESTABLISHED 3110/Discord --type
tcp6       0      0 :::80                   :::*                    LISTEN      1114/apache2
tcp6       0      0 :::22                   :::*                    LISTEN      1057/sshd: /usr/sbi
tcp6       0      0 2a02:2f0a:c10f:97:55754 2a02:2f0c:dff0:b::1:443 ESTABLISHED 9154/firefox
tcp6       0      0 2a02:2f0a:c10f:97:55750 2a02:2f0c:dff0:b::1:443 ESTABLISHED 9154/firefox
udp        0      0 0.0.0.0:56585           0.0.0.0:*                           3261/Discord --type
udp        0      0 0.0.0.0:42629           0.0.0.0:*                           3261/Discord --type
udp6       0      0 :::52070                :::*                                9154/firefox
udp6       0      0 :::38542                :::*                                9154/firefox

Obviously, your output will differ from the one above. +First look at the fourth column. +It shows the local address and port used by the process. +As we've already established, SSH uses port 22. +Apache2 uses port 80 for both IPv4 and IPv6 addresses (look for rows starting with tcp for IPv4 and tcp6 for IPv6). +Port 80 is used for the HTTP protocol.

Moving on to some user programs, Firefox runs multiple connections using both IPv4 and IPv6 and a different port for each connection.

Quiz

Discord does the same things as Firefox. +It uses TCP to send text messages, memes, videos and static content in general. +And at the same time, it uses UDP to exchange voice and video data during calls.

Practice: Run netstat Yourself​

  1. Run netstat on your own host machine. +Identify processes that use TCP and UDP and try to figure out what they might be using these protocols for. +Look for browsers (different sites and types of content), servers such as Apache2 or Nginx, online games etc. +Discuss your findings with your classmates and teacher(s).

  2. Now run receiver.py and then run netstat in another terminal to see your receiver process. +What arguments do you need to pass to netstat to see receiver.py? +Do you need the whole -tuapn? +No.

Conclusion​

The difference between TCP and UDP can be summarised as follows:

TCP vs UDP

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/overview/index.html b/17/Lab/IO/overview/index.html new file mode 100644 index 0000000000..a04719f95b --- /dev/null +++ b/17/Lab/IO/overview/index.html @@ -0,0 +1,22 @@ + + + + + +I/O | Operating Systems + + + + +
+
Skip to main content

I/O

We know that a compute system is a collection of hardware and software that modifies data. +This data has to come from somewhere. +This somewhere is always outside the compute system: +files, network packets, radio signals, sensor data.

A compute system without output is nearly useless. +It will always run the same code on the same data and, thus, produce the same result. +This may be useful in some narrow cases, such as calculating the decimals of Pi. +However, for more real-world-facing applications such as web servers, operating systems and databases inputs and outputs are mandatory.

The most simplistic representation of a compute system is a black box that receives some input and delivers some output.

Compute System - Oversimplified

In this session, we will look into how a compute system interacts with the outside world to get and produce these inputs and outputs.

  1. File Handlers
  2. File Descriptors
  3. Redirections
  4. Pipes
  5. Local I/O in Action
  6. Remote I/O
  7. Networking 101
  8. Client-Server Model
  9. Beyond Network Sockets
  10. File Mappings
  11. IO Internals
  12. Zero-copy
  13. Asynchronous I/O
  14. I/O Multiplexing
  15. Arena
+ + + + \ No newline at end of file diff --git a/17/Lab/IO/pipes/index.html b/17/Lab/IO/pipes/index.html new file mode 100644 index 0000000000..78c1bd2569 --- /dev/null +++ b/17/Lab/IO/pipes/index.html @@ -0,0 +1,70 @@ + + + + + +Pipes | Operating Systems + + + + +
+
Skip to main content

Pipes

When it comes to inter-process communication, so far we know that 2 different processes can mmap() the same file and use that as some sort of shared memory, but this requires writing data to the disk which is slow. +Then we know they can wait()/waitpid() for each other to finish, or better yet, use shared semaphores or mutexes, but these mechanisms aren't good at passing data between processes. +So our goals for this session are to learn how to use an IPC (inter-process communication) mechanism that:

  • allows transfers between processes, not notifications

  • is faster than reading and writing from/to files

Anonymous Pipes - pipe()​

Have you ever wondered how Bash handles redirecting the stdout of a command to the stdin of the next command in one-liners such as:

cat 'log_*.csv' | tr -s ' ' | cut -d ',' -f 2 | sort -u | head -n 10

The stdout of cat is the stdin of tr, whose stdout is the stdin of cut and so on. +This "chain" of commands looks like this:

Piped Commands

So here we have a unidirectional stream of data that starts from cat, is modified by each new command, and then is passed to the next one. +We can tell from the image above that the communication channel between any 2 adjacent commands allows one process to write to it while the other reads from it. +For example, there is no need for cat to read any of tr's output, only vice versa.

Therefore, this communication channel needs 2 ends: +one for reading (from which commands get their input) and another for writing (to which commands write their output). +In UNIX, the need for such a channel is fulfilled by the pipe() syscall. +Imagine there's a literal pipe between any 2 adjacent commands in the image above, where data is what flows through this pipe in only a single way. +This is why the | operator in Bash is called pipe and why the syscall is also named pipe().

This type of pipe is also called an anonymous pipe, because it cannot be identified using a name (i.e. it is not backed by any file). +The data written to it is kept in a circular buffer inside the kernel from where it can be then read by the child process. +This is faster than writing data to a file, so we achieve both our initial goals.

Practice: Find the Right Hole File Descriptor​

Navigate to support/pipes/anonymous_pipe.c. +Compile and run the code. +In another terminal, use lsof to see:

  • the file descriptors opened by the parent process between the creation of the pipe and the call to fork()

  • the file descriptors opened by the child process

Quiz

A simple way to memorise which pipe end is which is to think about stdin and stdout, respectively. +stdin is file descriptor 0 and is mostly for reading and pipedes[0] is also for reading. +Conversely, stdout is file descriptor 1 and is meant for writing, just like pipedes[1]. +Now you won't confuse them.

Practice: Inheritance​

An important thing to take note of before we actually use pipes is that file descriptors are inherited by the child process from the parent. +So if the parent opens some file descriptors (like, say, for a pipe), the child will also be able to use them. +Don't believe us?

Modify the code in support/pipes/anonymous_pipes.c and call wait_for_input() from the child process. +Then use lsof again with the PID of the child process to make sure file descriptors 3 and 4 are still open.

Practice: Now We Pipe​

Now comes the moment you've most likely been waiting for. +The code in support/pipes/anonymous_pipes.c wants to create something like a client-server dynamic between the parent and the child. +The parent reads data from stdin and sends it to the child via the pipe they share. +The client (parent) ends communication when you type "exit".

You can comment out the calls to wait_for_input() if you find them annoying.

Sample output:

student@os:~/.../support/pipes$ ./anonymous_pipe
pipedes[0] = 3; pipedes[1] = 4
 * pipe created
 -- Press ENTER to continue ...
echo
[Child received]: echo
echo
[Child received]: echo
to pipe, or not to pipe
[Child received]: to pipe, or not to pipe

Practice: Receive Pipes​

Use your knowledge of pipes to solve a CTF challenge. +Navigate to support/receive-challenges and look into the files receive_pipe.c and send_fd_4.c. +Modify receive_pipe.c so that it creates a pipe, then spawns a child process. +The child will redirect file descriptor 4 to stdout and then execlp() send_fd_4. +send_fd_4 writes the flag to file descriptor 4 (pipefd[1]), so the parent process needs to read it from pipedefd[0].

Once you do this, note that file descriptors are also maintained after calling exec() to run a completely new program.

Now, if you want to use pipes even more, go over to the Arena and add support for pipes to the mini-shell you've previously worked on.

Anonymous Pipes: Conclusion​

Anonymous pipes give allow us to efficiently transmit data between 2 processes, as no disk access is required. +However, they still have one major limitation.

Quiz

The answer to this is to employ some filesystem support.

Named Pipes - mkfifo()​

We will give up on some performance by writing data to a file, but the reading and writing to the file must behave just like using a named pipe:

  • reading doesn't block while there's still data in the pipe
  • reading from an empty pipe stalls the current thread until data becomes available. +This is one of the cases where read() might not return as many bytes as we requested. +Remember: +always use loops with read() and write()

Because this pipe uses a file, which must have a name, it is called a named pipe.

Practice: From the CLI​

First, let's use named pipes from the terminal. +Use the mkfifo command to create one such file:

student@os:~$ mkfifo fifo

student@os:~$ ls -l fifo
prw-rw-r-- 1 student student 0 Nov 22 23:20 fifo|

The fact that pipes are also called FIFOs should come as no surprise. +They act like queues/FIFOs:

  • you add data to one end (push/enqueue)
  • you extract data from the other (pop/dequeue)

Also note the p at the beginning of the output above. +It symbolises the type of this file: +a named pipe.

Now let's use it. +Open 2 terminals.

  1. In the first one, use cat to read from fifo. +Note that the command is blocked because the pipe is empty, so cat has nothing to read. +Then, in the second terminal, write some message to fifo using echo. +In the first terminal, you should see that cat has finished, and the message has appeared there.

  2. Now do it the other way around: +first echo some string into the pipe and then read it with cat. +Note that now the echo command is blocked. +Now cat should end immediately, and the string should appear because we have already placed some data in the pipe. +Also, the previous echo should finish now.

Remember: +Reading from a pipe blocks while the pipe is empty. +Writing to a pipe blocks until it is empty.

Practice: From the Code - Receive FIFO​

The libc function with which we can create named pipes is... +mkfifo(). +You weren't expecting this, were you? +Its underlying syscall is mknod(), which simply creates a file of whatever type we specify, but that's besides the point.

Navigate to support/receive-challenges/ and look into receive_fifo.c and send_fifo.c. +Follow the TODOs in the former file to read the flag that the latter writes into the named pipe. +Note that after you create the named pipe, you have to read from it like you would from any regular file.

Named Pipes: Conclusion​

It is nice to remember that named pipes sacrifice little to no performance when compared to anonymous pipes. +While it may seem that the data being passed through them is written to the disk, then read and overwritten, this is not the case. +The FIFO file is just a handler within the filesystem that is used to write data to a buffer inside the kernel. +This buffer holds the data that is passed between processes, not the filesystem. +So we still don't break our 2 desires from the beginning of this section: +to allow data transfer and to do so efficiently.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/anonymous-pipes-limitation/index.html b/17/Lab/IO/quiz/anonymous-pipes-limitation/index.html new file mode 100644 index 0000000000..7abca642df --- /dev/null +++ b/17/Lab/IO/quiz/anonymous-pipes-limitation/index.html @@ -0,0 +1,18 @@ + + + + + +Limitation of Anonymous Pipes | Operating Systems + + + + +
+
Skip to main content

Limitation of Anonymous Pipes

Question Text​

What of the following is the largest drawback of using anonymous pipes (created with pipe()) for inter-process communication?

Question Answers​

  • they only allow unidirectional communication
  • they only allow IPC between "related" processes (parent - child - grandchild etc.)

  • if more processes use the same end of the same pipe, there may be race conditions

  • a pipe only has 2 file descriptors, but some processes may need more channels to communicate

Feedback​

Out of the answers above, the only limitation that cannot be overcome by using pipes alone is the requirement for the processes using them to be related. +The parent must create the pipes so the children can inherit them. +All other downsides can be overcome by using more pipes.

The answer to this is to employ some filesystem support.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/bind-error-cause/index.html b/17/Lab/IO/quiz/bind-error-cause/index.html new file mode 100644 index 0000000000..9af461780e --- /dev/null +++ b/17/Lab/IO/quiz/bind-error-cause/index.html @@ -0,0 +1,19 @@ + + + + + +Cause of `bind()` Error | Operating Systems + + + + +
+
Skip to main content

Cause of bind() Error

Question Text​

While receiver.py is still running, run it again from another terminal. +You will get an error. +What is its cause?

Question Answers​

  • the IP 127.0.0.1 is already used by receive.py
  • the port 5000 is already used (by receive.py)

  • a port may not be used multiple times by the same process

  • the socket was not created correctly

Feedback​

One port may only be bound to one socket at a time. +The fact that it's the same program (same source code) using it is irrelevant because they're different processes.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/client-server-sender-receiver/index.html b/17/Lab/IO/quiz/client-server-sender-receiver/index.html new file mode 100644 index 0000000000..3a9778f3ac --- /dev/null +++ b/17/Lab/IO/quiz/client-server-sender-receiver/index.html @@ -0,0 +1,18 @@ + + + + + +`sender.py` and `receiver.py` Client-Server Parallel | Operating Systems + + + + +
+
Skip to main content

sender.py and receiver.py Client-Server Parallel

Question Text​

Drawing a parallel from the UDP examples in support/send-receive/, which of the sender and receiver is similar to the server and which is similar to the client?

Question Answers​

  • both are similar to clients

  • both are similar to servers

  • receiver.py is similar to a server and sender.py is similar to a client
  • receiver.py is similar to a client and sender.py is similar to a server

Feedback​

receiver.py is the passive component. +It has to be started first and then waits for sender.py (the client) to send data. +Furthermore, you can only have one receiver.py running at the same time (remember the multiple bind() bug) and multiple sender.pys.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/deluge-tcp-udp/index.html b/17/Lab/IO/quiz/deluge-tcp-udp/index.html new file mode 100644 index 0000000000..abae0ccfd6 --- /dev/null +++ b/17/Lab/IO/quiz/deluge-tcp-udp/index.html @@ -0,0 +1,20 @@ + + + + + +Deluge: TCP or UDP | Operating Systems + + + + +
+
Skip to main content

Deluge: TCP or UDP

Question Text​

Should Deluge use UDP or TCP to transfer torrent files?

Question Answers​

  • It should use UDP for faster file transfers
  • It should use TCP to guarantee the integrity of the transferred files

Feedback​

Speed is nice to have. +Correctness is mandatory in most scenarios, including this one. +The only situation when correctness may be overlooked is when some given data will be quckly replaced by some other data. +But files are persistent. +If you download a video game from as a torrent (we've all done that), you want to keep it for a while and first and foremost, it has to work properly, i.e. not be corrupt.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/execve/index.html b/17/Lab/IO/quiz/execve/index.html new file mode 100644 index 0000000000..c0419d057e --- /dev/null +++ b/17/Lab/IO/quiz/execve/index.html @@ -0,0 +1,19 @@ + + + + + +Effect of `execve()` Syscall | Operating Systems + + + + +
+
Skip to main content

Effect of execve() Syscall

Question Text​

What is the effect of the execve() syscall?

Question Answers​

  • it spawns a new process as the child of the current one

  • it executes a given shell command

  • it replaces the VAS of the current process with that of the file given as argument
  • it spawns a new shell and executes the given command

Feedback​

The man page says it all:

execve() executes the program referred to by pathname. This +causes the program that is currently being run by the calling +process to be replaced with a new program, with newly initialized +stack, heap, and (initialized and uninitialized) data segments.

Simply put, we can say that execve() replaces the VAS of the current process with that of the program given as argument.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/fewer-than-2-copies/index.html b/17/Lab/IO/quiz/fewer-than-2-copies/index.html new file mode 100644 index 0000000000..32b58d2561 --- /dev/null +++ b/17/Lab/IO/quiz/fewer-than-2-copies/index.html @@ -0,0 +1,22 @@ + + + + + +Fewer than Two Copies | Operating Systems + + + + +
+
Skip to main content

Fewer than Two Copies

Question Text​

Can zero-copy be implemented so as to copy the file fewer than 2 times?

Zero-Copy

Question Answers​

  • Yes, by copying the file straight from the disk to the network
  • No

  • Yes, by sending the file straight from the kernel buffer to the network

  • Yes, by copying the file directly from the storage to the NIC's buffer

Feedback​

The truth is that we can't have fewer copies while using a server with a common PC-like architecture. +The disk is not directly connected to the internet, so the file cannot be sent directly from there. +The only place from where we can send data to the Web is the NIC. +Then we need the intermediary storage in that kernel buffer because the disk and the NIC aren't dirrectly connected. +They are both connected to the CPU via the motherboard, so it's the CPU's job to do the transfer. +For this, it needs a "temporary buffer". +Then the NIC needs its own buffer because the speed of the network may be slower than the speed at which it receives data from the kernel, so it needs some memory where to place the "surplus" while waiting for the network to "clear".

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/file-handler-c/index.html b/17/Lab/IO/quiz/file-handler-c/index.html new file mode 100644 index 0000000000..08371a6f44 --- /dev/null +++ b/17/Lab/IO/quiz/file-handler-c/index.html @@ -0,0 +1,17 @@ + + + + + +File handler in C | Operating Systems + + + + +
+
Skip to main content

File handler in C

Question Text​

What is the type of the file handler in the C code located in support/simple-file-operations/file_operations.c?

Question Answers​

  • File
  • FILE *

  • FILE

  • void *

  • struct file

Feedaback​

The file is opened using either of the following lines:

f = fopen(file_name, "r");

f = fopen(file_name, "w");

The type of f is FILE *: +FILE *f.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/firefox-tcp-udp/index.html b/17/Lab/IO/quiz/firefox-tcp-udp/index.html new file mode 100644 index 0000000000..f5b3a361c8 --- /dev/null +++ b/17/Lab/IO/quiz/firefox-tcp-udp/index.html @@ -0,0 +1,22 @@ + + + + + +Firefox: TCP or UDP? | Operating Systems + + + + +
+
Skip to main content

Firefox: TCP or UDP?

Question Text​

If the user requests a text-based web page (such as our Operating Systems course), should the browser transfer the content using TCP or UDP? +What about video content, such as YouTube?

Question Answers​

  • TCP and UDP, respectively

  • UDP and TCP, respectively

  • both connections should use TCP

  • both connections should use UDP

Feedback​

The "TCP vs UDP" question boils down to 2 things:

  • is the data updated in real-time (multiple times per second)?
  • can we afford a few errors / missing messages?

If the answers to both questions is "Yes", then we should use UDP. +If they're "No", we should use TCP. +However, if the answer to one question is "Yes" and the other one is "No", then it gets complicated.

Luckily, in our cases, the answers are quite clear. +We don't update text-based content too often and since it needs to be precise, handling broken packets is important. +On the other hand, a streaming-based site, such as YouTube sends data in real time and thus a few errors here and there can be omitted. +So https://open-education-hub.github.io/operating-systems is going to be served via TCP, while YouTube videos via UDP.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/flush-libc-buffer/index.html b/17/Lab/IO/quiz/flush-libc-buffer/index.html new file mode 100644 index 0000000000..3531289e5d --- /dev/null +++ b/17/Lab/IO/quiz/flush-libc-buffer/index.html @@ -0,0 +1,17 @@ + + + + + +Flush Libc Buffer | Operating Systems + + + + +
+
Skip to main content

Flush Libc Buffer

Question Text​

Which of the following is a method of flushing libc's internal buffer?

Question Answers​

  • print a \0 character
  • print a \n character

  • print a space character

  • print a \t character

Feedback​

Newlines (\n) force printf() to dump the internal buffer associated with the stdout FILE struct. +If you place a \n character within one of the strings in support/buffering/printf_buffering.c, a write() syscall will be made right after it.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/fopen-syscall/index.html b/17/Lab/IO/quiz/fopen-syscall/index.html new file mode 100644 index 0000000000..876da3cc9d --- /dev/null +++ b/17/Lab/IO/quiz/fopen-syscall/index.html @@ -0,0 +1,17 @@ + + + + + +Syscall Used by `fopen()` | Operating Systems + + + + +
+
Skip to main content

Syscall Used by fopen()

Question Text​

Use strace to determine the syscall called by fopen() to access the file. +Which one is it?

Question Answers​

  • read()
  • openat()

  • write()

  • fstat()

Feedaback​

student@os:~/.../lab/support/simple-file-handling$ strace ./file_operations
[...]
openat(AT_FDCWD, "file.txt", O_RDONLY)  = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=11, ...}) = 0
read(3, "C was here!", 4096)            = 11
[...]

So fopen()'s (main) underlying syscall is openat().

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/fopen-w/index.html b/17/Lab/IO/quiz/fopen-w/index.html new file mode 100644 index 0000000000..4e60a73775 --- /dev/null +++ b/17/Lab/IO/quiz/fopen-w/index.html @@ -0,0 +1,17 @@ + + + + + +open()` equivalent of `fopen(..., "w") | Operating Systems + + + + +
+
Skip to main content

open() equivalent of fopen(..., "w")

Question Text​

Use strace on the code in support/simple-file-operations/file_operations.c to find the flags used by openat() when calling fopen(file_name, "w"). +First, try to make an educated guess and only then verify your answer by running strace.

Question Answers​

  • O_WRONLY | O_CREAT | O_TRUNC

  • O_WRONLY | O_CREAT

  • O_WRONLY

  • O_WRONLY | O_TRUNC

Feedback​

student@os:~/.../lab/support/simple-file-operations$ strace ./file_operations
[...]
openat(AT_FDCWD, "file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
[...]
+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/local-io-errors/index.html b/17/Lab/IO/quiz/local-io-errors/index.html new file mode 100644 index 0000000000..ecb4053ca1 --- /dev/null +++ b/17/Lab/IO/quiz/local-io-errors/index.html @@ -0,0 +1,17 @@ + + + + + +I/O Errors | Operating Systems + + + + +
+
Skip to main content

I/O Errors

Question Text​

Which of the following types of errors are unlikely to occur during an I/O operation?

Question Answers​

  • The current user does not have sufficient permisions to access a given file

  • There is not enough space left on the disk

  • The file offset has reached the end of the file when reading
  • Some data blocks in the filesystem are corrupted

Feedback​

We can always reposition the file offset within a given file with either a fseek() library call or an lseek() syscall, so this is not an error. +The others are more difficult to manage, so can be regarded as errors.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/mmap-read-write-benchmark/index.html b/17/Lab/IO/quiz/mmap-read-write-benchmark/index.html new file mode 100644 index 0000000000..387f18591e --- /dev/null +++ b/17/Lab/IO/quiz/mmap-read-write-benchmark/index.html @@ -0,0 +1,19 @@ + + + + + +`mmap()` vs `read()` and `write()` Benchmark | Operating Systems + + + + +
+
Skip to main content

mmap() vs read() and write() Benchmark

Question Text​

According to the times shown by benchmark_cp.sh, which of the two implementations is faster?

Question Answers​

  • the one using mmap()

  • the one using read() and write()

  • they are roughly equivalent

Feedback​

In our case, running the script a few times results in the following running times:

student@os:/.../support/file-mappings$ ./benchmark_cp.sh
make: Nothing to be done for 'all'.
Benchmarking mmap_cp on a 1GB file...

real    0m30,597s
user    0m0,569s
sys 0m2,286s
Benchmarking cp on a 1 GB file...

real    0m36,012s
user    0m0,039s
sys 0m2,469s


student@os:/.../support/file-mappings$ ./benchmark_cp.sh
make: Nothing to be done for 'all'.
Benchmarking mmap_cp on a 1 GB file...

real    0m27,803s
user    0m0,590s
sys 0m2,114s
Benchmarking cp on a 1 GB file...

real    0m35,607s
user    0m0,033s
sys 0m2,564s

So it seems that using mmap() rather than read() and write() yields about a 15% increase in performance. +However, you might get different results on your system. +This depends on your storage device (SSD vs HDD) and its specific speed (like its RPM for an HDD). +So the more conservative answer is to say that this depends on external aspects and that, in general, the 2 implementations are more or less equivalent.

If you want to know why there isn't much of a difference between the 2 implementations, check out this explanation.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/o-trunc/index.html b/17/Lab/IO/quiz/o-trunc/index.html new file mode 100644 index 0000000000..4205cfd53e --- /dev/null +++ b/17/Lab/IO/quiz/o-trunc/index.html @@ -0,0 +1,16 @@ + + + + + +`O_TRUNC` Flag Behaviour | Operating Systems + + + + +
+
Skip to main content

O_TRUNC Flag Behaviour

Question Text​

How does the O_TRUNC flag change the behaviour of open()?

Question Answers​

  • the previous content of the file is deleted

  • new data will be appended to the file

  • newly written data will be ignored

Feedback​

The man page provides us with unlimited wisdon:

If the file already exists and is a regular file and the access mode allows writing (i.e., is O_RDWR or O_WRONLY) it will be truncated to length 0.

Setting the length of the file to 0 is equivalent to deleting the previous content of the file.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/pipe-ends/index.html b/17/Lab/IO/quiz/pipe-ends/index.html new file mode 100644 index 0000000000..0fb8ca8fd1 --- /dev/null +++ b/17/Lab/IO/quiz/pipe-ends/index.html @@ -0,0 +1,22 @@ + + + + + +Pipe Ends | Operating Systems + + + + +
+
Skip to main content

Pipe Ends

Question Text​

Which end of a pipe created by pipe() is for reading and which one is for writing?

Question Answers​

  • pipefds[0] is for reading; +pipefds[1] is for writing

  • pipefds[0] is for writing; +pipefds[1] is for reading

  • both heads are for reading and writing;

Feedback​

Running the binary first tells us which file descriptor is which:

student@os:~/.../lab/support/pipes$ ./anonymous_pipes
pipedes[0] = 3; pipedes[1] = 4
 * pipe created
 -- Press ENTER to continue ...

Then lsof gives us the complete answer:

student@os:~/.../lab/support/pipes$ lsof -w -p $(pidof anonymous_pipes)
anonymous 22474  student  cwd    DIR    8,1      504  296964 /media/student/2TB/Chestii/Poli/Asistent/SO/operating-systems-oer/content/chapters/io/lab/support/pipes
anonymous 22474  student  rtd    DIR  259,6     4096       2 /
anonymous 22474  student  txt    REG    8,1    26712  296968 /media/student/2TB/Chestii/Poli/Asistent/SO/operating-systems-oer/content/chapters/io/lab/support/pipes/anonymous_pipes
anonymous 22474  student  mem    REG  259,6  2029592 1857435 /usr/lib/x86_64-linux-gnu/libc-2.31.so
anonymous 22474  student  mem    REG  259,6   191504 1835092 /usr/lib/x86_64-linux-gnu/ld-2.31.so
anonymous 22474  student    0u   CHR  136,0      0t0       3 /dev/pts/0
anonymous 22474  student    1u   CHR  136,0      0t0       3 /dev/pts/0
anonymous 22474  student    2u   CHR  136,0      0t0       3 /dev/pts/0
anonymous 22474  student    3r  FIFO   0,13      0t0  252007 pipe
anonymous 22474  student    4w  FIFO   0,13      0t0  252007 pipe

The last 2 lines are the 2 ends of the pipe: +3 and 4. +Note each of these numbers is followed by a letter. +As you might have guessed:

  • r means "read"
  • w means "write"
  • u means both "read" and "write"

Also, the man page is quite clear on this issue:

pipefd[0] refers to the read end of the pipe. pipefd[1] refers +to the write end of the pipe.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/prints-work-no-stdio/index.html b/17/Lab/IO/quiz/prints-work-no-stdio/index.html new file mode 100644 index 0000000000..c011fae3ab --- /dev/null +++ b/17/Lab/IO/quiz/prints-work-no-stdio/index.html @@ -0,0 +1,17 @@ + + + + + +Prints Working after Closing `stdio` | Operating Systems + + + + +
+
Skip to main content

Prints Working after Closing stdio

Question Text​

Why does support/redirect/redirect.c, still print messages to the console after closing file descriptor 1 (stdout)?

Question Answers​

  • because wait_for_input() calls fprintf(stderr, ...), which prints to stderr (file descriptor 2)

  • because the default file descriptors cannot be "truly" closed

  • because the other two default file descriptors are still linked to the console

  • because the wait_for_input() function started printed before closing the stdout file descriptor

Feedback​

If you look at wait_for_input() closely, you'll notice it calls fprintf(stderr, ...). +stderr is liked to file descriptor 2, which is left unchanged so we can still write data to it.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/receiver-socket-fd/index.html b/17/Lab/IO/quiz/receiver-socket-fd/index.html new file mode 100644 index 0000000000..f79f12282c --- /dev/null +++ b/17/Lab/IO/quiz/receiver-socket-fd/index.html @@ -0,0 +1,16 @@ + + + + + +Receiver Socked File Descriptor | Operating Systems + + + + +
+
Skip to main content

Receiver Socked File Descriptor

Question Text​

What is the type of the file descriptor that corresponds to the socket created by support/send-receive/receiver.py?

Question Answers​

  • only file descriptors that are linked to files have types

  • DIR

  • REG

  • CHR

  • IPv4

Feedback​

Running lsof yields the following output:

student@os:~$ lsof -p 59681
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
python3 59681  student  cwd    DIR    8,1        0  559322 /home/student/operating-systems-oer/content/chapters/io/lab/support/send-receive
python3 59681  student  rtd    DIR  259,6     4096       2 /
python3 59681  student  txt    REG  259,6  5502744 1835857 /usr/bin/python3.8
python3 59681  student  mem    REG  259,6  8631488 1835827 /usr/lib/locale/locale-archive
python3 59681  student  mem    REG  259,6   108936 1835887 /usr/lib/x86_64-linux-gnu/libz.so.1.2.11
python3 59681  student  mem    REG  259,6   182560 1836149 /usr/lib/x86_64-linux-gnu/libexpat.so.1.6.11
python3 59681  student  mem    REG  259,6  1369384 1857443 /usr/lib/x86_64-linux-gnu/libm-2.31.so
python3 59681  student  mem    REG  259,6    14880 1857476 /usr/lib/x86_64-linux-gnu/libutil-2.31.so
python3 59681  student  mem    REG  259,6    18848 1857439 /usr/lib/x86_64-linux-gnu/libdl-2.31.so
python3 59681  student  mem    REG  259,6   157224 1857471 /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
python3 59681  student  mem    REG  259,6  2029592 1857435 /usr/lib/x86_64-linux-gnu/libc-2.31.so
python3 59681  student  mem    REG  259,6    27002 2506848 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
python3 59681  student  mem    REG  259,6   191504 1835092 /usr/lib/x86_64-linux-gnu/ld-2.31.so
python3 59681  student    0u   CHR  136,1      0t0       4 /dev/pts/1
python3 59681  student    1u   CHR  136,1      0t0       4 /dev/pts/1
python3 59681  student    2u   CHR  136,1      0t0       4 /dev/pts/1
python3 59681  student    3u  IPv4 588386      0t0     UDP localhost:5000

The last line displays the socket:

python3 59681  student    3u  IPv4 588386      0t0     UDP localhost:5000

Its type is written on the the 5th column: IPv4 because it's a network socket.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/server-copies/index.html b/17/Lab/IO/quiz/server-copies/index.html new file mode 100644 index 0000000000..e293e6c53d --- /dev/null +++ b/17/Lab/IO/quiz/server-copies/index.html @@ -0,0 +1,22 @@ + + + + + +Client-Server Number of Copies | Operating Systems + + + + +
+
Skip to main content

Client-Server Number of Copies

Question Text​

The server in the image below uses regular TCP sockets to handle the connection and send() to send the data. +How many times are the contents of the file copied by the server while being sent to the client?

Client-Server Steps

Question Answers​

  • 2

  • 1

  • 4
  • 3

Feedback​

Rembember double buffering! +When the app calls read(), the server's kernel will first save the file to an internal buffer destined for reading. +Then the app will copy the file to its own buffer. +Following this step, the app will call send(), which will first copy the file to a buffer in the kernel. +From this buffer, the kernel itself will copy the file to another buffer on the NIC (Network Interface Card). +In total, there the file is copied 4 times, as outlined in the image below.

Server Copies - Read-Send

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/stderr-fd/index.html b/17/Lab/IO/quiz/stderr-fd/index.html new file mode 100644 index 0000000000..86af3a60cc --- /dev/null +++ b/17/Lab/IO/quiz/stderr-fd/index.html @@ -0,0 +1,18 @@ + + + + + +File Descriptor of `stderr` | Operating Systems + + + + +
+
Skip to main content

File Descriptor of stderr

Question Text​

Which file descriptor is associated by default to stderr?

Question Answers​

  • it varies from process to process

  • it varies from one Linux distribution to another

  • stderr has no associated file descriptor

  • 2

  • 0

  • 1

Feedaback​

You would type ls 2> /dev/null to ignore ls's errors. +This equates to redirecting stderr to /dev/null. +The number 2 in 2> hints that stderr is by default associated with file descriptor 2.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/strace-printf/index.html b/17/Lab/IO/quiz/strace-printf/index.html new file mode 100644 index 0000000000..5af011b28d --- /dev/null +++ b/17/Lab/IO/quiz/strace-printf/index.html @@ -0,0 +1,16 @@ + + + + + +`printf()` Under Strace | Operating Systems + + + + +
+
Skip to main content

printf() Under Strace

Question Text​

How many calls to write() does the code in support/buffering/printf_buffering.c do?

Question Answers​

  • none
  • 1

  • 6

  • 5

Feedback​

Just run the binary under strace:

student@os:/.../support/buffering$ strace -e write ./printf_buffering
write(1, "Dovahkiin, Dovahkiin Naal ok zin"..., 169Dovahkiin, Dovahkiin Naal ok zin los vahriin Wah dein vokul mahfaeraak ahst vaal! Ahrk fin norok paal graan Fod nust hon zindro zaan Dovahkiin, fah hin kogaan mu draal! ) = 169
+++ exited with 0 +++

There's one call to write().

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/syscalls-cp/index.html b/17/Lab/IO/quiz/syscalls-cp/index.html new file mode 100644 index 0000000000..62cc423100 --- /dev/null +++ b/17/Lab/IO/quiz/syscalls-cp/index.html @@ -0,0 +1,16 @@ + + + + + +Syscalls Used by `cp` | Operating Systems + + + + +
+
Skip to main content

Syscalls Used by cp

Question Text​

What syscalls does cp use to copy files?

Question Answers​

  • mmap()
  • read() and write()
  • a combination of read() - write() and mmap()

Feedback​

It suffices to use strace to see that cp uses read() and write().

student@os:/.../support/file-mappings$ strace cp test-file.txt output.txt
openat(AT_FDCWD, "test-file.txt", O_RDONLY) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=1048576, ...}) = 0
openat(AT_FDCWD, "output.txt", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4
fstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
[...]
read(3, "@Y\344\0025\317\27\243\23\201:\27\342\356\240\345\331Nq\v/\36\244\200\301\247\3152\35WZ\337"..., 131072) = 131072
write(4, "@Y\344\0025\317\27\243\23\201:\27\342\356\240\345\331Nq\v/\36\244\200\301\247\3152\35WZ\337"..., 131072) = 131072
read(3, "\201\240J7x\275\257Z\343\334\307d<\321U\275\337\10\233j\222\313,##cQD\268e\324"..., 131072) = 131072
write(4, "\201\240J7x\275\257Z\343\334\307d<\321U\275\337\10\233j\222\313,##cQD\268e\324"..., 131072) = 131072
read(3, "\371\3244=\17\300L9\243\201\362\25\273\37\326\323\362\200\1T\310N\316\305\253\331\331Nt\346\3369"..., 131072) = 131072
write(4, "\371\3244=\17\300L9\243\201\362\25\273\37\326\323\362\200\1T\310N\316\305\253\331\331Nt\346\3369"..., 131072) = 131072
read(3, "\350\304\345f\16\305V\356\371\263?+\355{\16\235\344\310P4}\2043%\0052\345D\265\243t\354"..., 131072) = 131072
write(4, "\350\304\345f\16\305V\356\371\263?+\355{\16\235\344\310P4}\2043%\0052\345D\265\243t\354"..., 131072) = 131072
read(3, "\277\226\315\226\n\37\337;N*\211\352\254$\273\2\351\30a\254\ta\352R\25-\23\274\376zy\211"..., 131072) = 131072
write(4, "\277\226\315\226\n\37\337;N*\211\352\254$\273\2\351\30a\254\ta\352R\25-\23\274\376zy\211"..., 131072) = 131072
read(3, "}\245\356;\222\327\204\242u\26dy%\346\374\201ndT\226\233\3575\345\247\356\362\344\350\354\17\261"..., 131072) = 131072
write(4, "}\245\356;\222\327\204\242u\26dy%\346\374\201ndT\226\233\3575\345\247\356\362\344\350\354\17\261"..., 131072) = 131072
read(3, "\35\277\207\243~\355(i\351^\1\346\312V\232\204\32\230~\376\20\245\"\305\344d\304\304B\272\346\332"..., 131072) = 131072
write(4, "\35\277\207\243~\355(i\351^\1\346\312V\232\204\32\230~\376\20\245\"\305\344d\304\304B\272\346\332"..., 131072) = 131072
read(3, "\n)\334\275\331:R\236O\231\243\302\314\267\326\"\rY\262\21\374\305\275\3\332\23\345\16>\214\210\235"..., 131072) = 131072
write(4, "\n)\334\275\331:R\236O\231\243\302\314\267\326\"\rY\262\21\374\305\275\3\332\23\345\16>\214\210\235"..., 131072) = 131072
+ + + + \ No newline at end of file diff --git a/17/Lab/IO/quiz/write-file-permissions/index.html b/17/Lab/IO/quiz/write-file-permissions/index.html new file mode 100644 index 0000000000..f9ec27b290 --- /dev/null +++ b/17/Lab/IO/quiz/write-file-permissions/index.html @@ -0,0 +1,24 @@ + + + + + +`write_file.txt` Permissions | Operating Systems + + + + +
+
Skip to main content

write_file.txt Permissions

Question Text​

Assume write_file.txt is opened like this: +open("write_file.txt", O_WRONLY | O_CREAT). +What might cause it to have different permissions that read_file.txt?

Question Answers​

  • an undefined behaviour in the kernel

  • using the O_WRONLY flag

  • not passing a mode argument to open(), so a random set of permissions is used.
  • a filesystem error.

Feedaback​

Quoting from open()'s man page, regarding the O_CREAT flag:

The mode argument must be supplied if O_CREAT or O_TMPFILE is +specified in flags; if it is not supplied, some arbitrary +bytes from the stack will be applied as the file mode.

The wording "arbitrary bytes from the stack" is a bit obsolete. +To demistify the quote above, let's consider that open() is a function call. +In fact, libc defines wrapper functions on top of all system calls and our code actually calls those wrappers, so our assumption is not far-fetched.

To call open() correctly, the coulde would look something like this:

open("write_file.txt", O_WRONLY | O_CREAT, 0644).

0644 is the octal representation of rw-r--r--. +The approximate Assembly code for this function call would look like this:

mov rdi, path_to_file   ; first argument: path

mov rsi, O_WRONLY
or rsi, O_CREAT         ; second argument: flags

mov rdx, 0644           ; third argument: mode

call open

When we don't specifty a mode argument, the open() function simply takes this value from rdx as it was before the function call. +This is an undefined behaviour because this register might have been modified either way by the program before calling open().

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/redirections/index.html b/17/Lab/IO/redirections/index.html new file mode 100644 index 0000000000..97bb15986f --- /dev/null +++ b/17/Lab/IO/redirections/index.html @@ -0,0 +1,60 @@ + + + + + +Redirections | Operating Systems + + + + +
+
Skip to main content

Redirections

In the File Descriptors section, we mentioned redirections such as ls > file.txt. +We said file.txt has to be opened at some point. +Let's check that. +We'll use strace, obviously, to look for open() and openat() syscalls.

student@os:~/.../lab/support/simple-file-handling$ strace -e open,openat ls > file.txt
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3

This looks strange. +Where is the call to openat(AT_FDCWD, "file.txt", ...)? +Well, if we look at the full strace output, the first call is execve():

student@os:~/.../lab/support/simple-file-handling$ strace ls > file.txt
execve("/usr/bin/ls", ["ls"], 0x7ffe550d59e0 /* 60 vars */) = 0
[...]

Quiz

So the openat() syscalls we saw earlier come from the ls process. +Remember how launching a command works in Bash:

  1. The Bash process fork()s itself
  2. The child (still a Bash process) then calls execve()
  3. The parent (the original Bash process) calls waitpid() to wait for the new process to end.

Launching a new command in Bash

So we can deduce that file.txt is opened by the child process before calling execve(). +Note that despite replacing the VAS of the current process, execve() does NOT replace its file descriptor table. +So whatever files were opened by the original process remain open within the new one. +This behaviour is the basis of pipes (the | that you use in Bash to use the output of a command as the input of another) and redirections (>, < and 2>).

Practice: Naive Redirection​

But before diving into reimplementing shell functionalities, let's look at a simpler example. +Navigate to support/redirect/redirect.c. +The code makes a naive attempt at redirecting the newly opened file to stdout. +It simply closes stdout first so that when open() returns the lowest available file descriptor, that value will be 1, which is STDOUT_FILENO.

Note there's a difference between stdout and STDOUT_FILENO. +While stdout is of type FILE * and is meant to be used with libc functions such as fread(), STDOUT_FILENO is the default file descriptor for the standard output, which almost always is 1. +So STDOUT_FILENO is an int type with the value 1. +Don't confuse them!

Compile and run the code without modifying it. +It pauses after each file descriptor operation. +In another terminal (or in another tmux window), run the following each time you press Enter in the first terminal/window: lsof -p $(pidof redirect) or try watch-ing it.

lsof displays the files opened by the given process. +From the output below, we see that these files are mainly files (opened as file descriptors) and libraries, which are memory mapped (mem). +On the third column, you can see the file descriptor corresponding to each file.

student@os:~/.../lab/support/redirect$ lsof -w -p $(pidof redirect)  # before any file operations
COMMAND    PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
redirect 44303  student  cwd    DIR    8,1     4096  299870 /home/student/operating-systems-oer/content/chapters/io/lab/support/redirect
redirect 44303  student  rtd    DIR  259,6     4096       2 /
redirect 44303  student  txt    REG    8,1    25848  299929 /home/student/operating-systems-oer/content/chapters/io/lab/support/redirect/redirect
redirect 44303  student  mem    REG  259,6  2029592 1857435 /usr/lib/x86_64-linux-gnu/libc-2.31.so
redirect 44303  student  mem    REG  259,6   191504 1835092 /usr/lib/x86_64-linux-gnu/ld-2.31.so
redirect 44303  student    0u   CHR  136,0      0t0       3 /dev/pts/0
redirect 44303  student    1u   CHR  136,0      0t0       3 /dev/pts/0
redirect 44303  student    2u   CHR  136,0      0t0       3 /dev/pts/0

Notice that all 3 default file descriptors are first liked to /dev/pts/0. +It may be different on your machine, but most likely it will be /dev/pts/<some_number>. +This is a character device that signifies your current (pseudo)terminal.

student@os:~/.../lab/support/redirect$ lsof -w -p $(pidof redirect)  # after closing `STDOUT_FILENO`
redirect 46906  student    0u   CHR  136,0      0t0       3 /dev/pts/0
redirect 46906  student    2u   CHR  136,0      0t0       3 /dev/pts/0

See that file descriptor 1 (stdout) has "disappeared". +Now the second entry in the process's FD table is free.

student@os:~/.../lab/support/redirect$ lsof -w -p $(pidof redirect)  # after opening `redirect_file.txt`
redirect 46906  student    0u   CHR  136,0      0t0       3 /dev/pts/0
redirect 46906  student    1w   REG    8,1        0  299958 /.../lab/support/redirect/redirect_file.txt
redirect 46906  student    2u   CHR  136,0      0t0       3 /dev/pts/0

open() has assigned the newly opened file to the lowest file descriptor available, which is 1, the former stdout. +printf() writes its output to stdout, which in this case is redirect_file.txt. +Now inspect the contents of this file to make sure that string was written there.

Quiz

Practice: Thread-unsafe Redirection​

This is all fine, but it doesn't allow us to copy file descriptors. +We can only replace an existing file descriptor with another one. +To be able to do replacements, we can use the dup() syscall. +It simply creates a new file descriptor that refers to the same open file struct as the one given to it as an argument. +Both file descriptors remain active after calling dup().

Change the do_redirect() function in support/redirect/redirect.c to employ this new logic. +It should follow the steps below. +Track them using lsof, just like before.

Step 1:

  • 0 -> stdin
  • 1 -> stdout
  • 2 -> stderr

Step 2 - after open("redirect_file.txt", ...):

  • 0 -> stdin
  • 1 -> stdout
  • 2 -> stderr
  • 3 -> redirect_file.txt

Step 3 - after close(STDOUT_FILENO):

  • 0 -> stdin
  • 2 -> stderr
  • 3 -> redirect_file.txt

Step 4 - after dup(3). +Note that now both 1 and 3 are linked to redirect_file.txt, so we managed to successfully copy file descriptor 3.

  • 0 -> stdin
  • 1 -> redirect_file.txt
  • 2 -> stderr
  • 3 -> redirect_file.txt

Step 5 - after close(3). +We don't need file descriptor 3 at this point anymore.

  • 0 -> stdin
  • 1 -> redirect_file.txt
  • 2 -> stderr

Practice: Safe Redirection​

dup() is all fine and dandy, but what if 2 threads use the steps above concurrently? +Because steps 3 and 4 don't happen atomically, they risk having their results inverted. +Take a look at support/redirect/redirect_parallel.c. +Compile and run the code, then inspect the resulting files. +You'll notice they contain opposing strings:

student@os:~/.../lab/support/redirect$ cat redirect_stderr_file.txt
Message for STDOUT

student@os:~/.../lab/support/redirect$ cat redirect_stdout_file.txt
Message for STDERR

What happens is that thread 1 is forced to call close(STDOUT_FILENO), then thread 2 calls close(STDERR_FILENO). +So far, this is not problematic. +But then thread 2 continues to dup() redirect_stderr.txt into the lowest file descriptor available, which is 1 (STDOUT_FILENO). +Then thread 1 resumes to dup() redirect_stdout.txt into file descriptor 2 (STDERR_FILENO). +Thus we end up redirecting stdout and stderr to the opposite files than those we intended.

To fix this, we need to call an atomic syscall, called dup2(). +It receives 2 file descriptors (dup2(src_fd, dst_fd)) and its actions are equivalent to the following, but performed atomically:

close(dst_fd);
dup(src_fd);  // This places `src_fd` into the previous `dst_fd`

  1. Modify support/redirect/redirect_parallel.c and change the calls to close() and dup() to calls to dup2() and check the contents of the resulting files to see they're correct.

  2. Now go back to support/redirect/redirect.c and refactor the code in do_redirect() to use dup2() as well.

Practice: Mini-shell Reloaded​

Remember the mini-shell you implemented in the Arena of the previous lab. +It is capable of fork()-ing itself and execvp()-ing commands, just like Bash. +We can now extend it to allow redirecting stdout to a file.

Use what you've learnt so far in this section to allow this mini-shell to redirect the output of its commands to files. +Redirection is performed just like in bash via >.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/remote-io/index.html b/17/Lab/IO/remote-io/index.html new file mode 100644 index 0000000000..d8514b38de --- /dev/null +++ b/17/Lab/IO/remote-io/index.html @@ -0,0 +1,86 @@ + + + + + +Remote I/O | Operating Systems + + + + +
+
Skip to main content

Remote I/O

In the previous sections, we started looking into how applications interact with the outside world. +However, so far this "outside world" has only meant local files and other local processes. +But what about files located on other computers? +What about "talking to" processes running in other parts of the world?

One Browser - Many Connections​

What happens when we request a web page? +A simple example is http://example.com/. +When the browser displays this web page, it first downloads it and then renders it. +The web page comes as a file called index.html. +We can roughly redo its steps like this:

student@os:~$ wget http://example.com/  # download index.html
wget http://example.com/
--2022-12-02 15:53:31--  http://example.com/
Resolving example.com (example.com)... 2606:2800:220:1:248:1893:25c8:1946, 93.184.216.34
Connecting to example.com (example.com)|2606:2800:220:1:248:1893:25c8:1946|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1256 (1,2K) [text/html]
Saving to: ‘index.html’

index.html                                                           100%[====================================================================================================================================================================>]   1,23K  --.-KB/s    in 0s

2022-12-02 15:53:31 (248 MB/s) - ‘index.html’ saved [1256/1256]

Then we can view the HTML contents of the file:

student@os:~$ cat index.html
<!doctype html>
<html>
<head>
    <title>Example Domain</title>

    <meta charset="utf-8" />
    <meta http-equiv="Content-type" content="text/html; charset=utf-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1" />
    <style type="text/css">
    body {
        background-color: #f0f0f2;
        margin: 0;
        padding: 0;
        font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;

    }
    div {
        width: 600px;
        margin: 5em auto;
        padding: 2em;
        background-color: #fdfdff;
        border-radius: 0.5em;
        box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
    }
    a:link, a:visited {
        color: #38488f;
        text-decoration: none;
    }
    @media (max-width: 700px) {
        div {
            margin: 0 auto;
            width: auto;
        }
    }
    </style>
</head>

<body>
<div>
    <h1>Example Domain</h1>
    <p>This domain is for use in illustrative examples in documents. You may use this
    domain in literature without prior coordination or asking for permission.</p>
    <p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>

And, finally, we can render the index.html file in the browser like so:

student@os:~$ xdg-open index.html  # xdg-open invokes the OS's default program for opening HTML files

After running the command above, look at the URL in the browser. +It's not http://example.com/ anymore, but the path to your local index.html.

So now you've switched from doing remote I/O back to local I/O. +Deluge does the same thing: it performs remote I/O operations to get files locally so that you, the user, can do local I/O with it. +Remote and local I/O are not by any means separated. +Rather, they are strongly intertwined.

Connection​

Focusing on how the index.html file is sent over the Web, we first need to establish the 2 endpoints. +We have a sender: the http://example.com/ server. +Then we have a receiver: our host. +How do the 2 endpoints know each other?

Recap: IPs​

When someone asks you who you are on the internet, the answer is obvious: your IP Address. +IP Addresses are 32-bit (or 128-bit for IPv6) numbers that identify hosts on the web. +To find the IPv4 and IPv6 addresses of a host given by a URL, you can use the host command:

student@os:~$ host example.com
example.com has address 93.184.216.34
example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946
example.com mail is handled by 0 .

So we can imagine our browser identifies http://example.com/ by its IP address, say 93.184.216.34. +Similarly, the server also knows our host's IP address. +Each of them uses the other's IP address to locate their peer and communicate with them.

But what if we shift our example to another site: https://open-education-hub.github.io/operating-systems/? +Let's say we open 2 tabs:

Now our browser needs to know what to do with data coming from two sources. +In addition, the server also needs to maintain information about our 2 tabs so it can send the correct data to each of them. +Therefore, each tab establishes a different connection to the server. +All communication between the tab and the site occurs within this connection.

Now the question is: how can we maintain 2 connections between 2 endpoints only identified by IPs? +... and the answer is that we can't. +If the browser and the server were to only use IP addresses, they wouldn't be able to differentiate between the 2 connections mentioned above. +We need something more: ports

Further than IPs: Ports​

A port is simply a number assigned to uniquely identify a connection from a host to another and to direct the data that's transferred between them. +This way, we can create multiple connections between the same 2 hosts. +Port numbers are encoded on 16 bits and thus range from $0$ to $2^{16} - 1$, i.e. from $0$ to $65535$.

The first 1024 ports are reserved for well-known system services, such as SSH (which uses port 22). +These services run using certain communication protocols. +A communication protocol is a set of rules that allow 2 or more hosts to communicate a certain way. +These rules include, but are not limited to:

  • the format of each message: fields, sizes, maximum lengths etc.
  • the order in which messages are sent
  • the behaviour of the endpoints with respect to these messages

So the correct way of saying it isn't that the SSH process / service runs on port 22, but rather that the SSH protocol runs on port 22.

Our browser also uses ports to distinguish between different connections. +And so does the github.io server: it uses one port for sending the "File Descriptors" page and another for the "File Handling" page. +The image below shows how multiple tabs to the same site can be handled. +The port numbers are chosen randomly. +They may have any value higher than 1023.

Browser Tabs and Ports

So it should be clear now that a connection is uniquely identified by an IP and a port.

API - Hail Berkeley Sockets​

Up to now we've described how sites work in general. +Before we can implement something of this sort ourselves, we need to understand the API. +Unlike other APIs such as syscalls, which differ between OSs (Unix vs Windows for example), the one we're about to learn is almost universally adopted across OSs and programming languages. +It's called the Berkeley Sockets API. +And with this, we've just introduced a new word: socket.

Not this Socket

No, not this type of socket... +But our socket is somewhat similar in concept. +Just like wall sockets allow us to plug into the electric grid, network sockets allow us to "plug" into the Web. +Remember file handlers? +You should. +File handlers are objects with which we can interact with files. +Similarly, sockets are handlers that provide an abstraction for a connection to another process, running either on a remote machine or on the same host.

Sender and Receiver​

We'll start with 2 programs: a sender and a receiver. +Navigate to support/send-receive/ and take a look at both sender.py and receiver.py.

The sender reads data from the keyboard and sends it to the receiver. +The receiver reads data continuously. +Upon receiving the message "exit", it closes. +Otherwise, it prints whatever it receives to stdout. +This detail about how to handle a message containing "exit" may be regarded as a communication protocol established between the sender and the receiver.

Now open 2 terminals (or use tmux). +First run receiver.py in one terminal. +Looking at its code, the receiver does 2 things. +It creates a socket:

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)

We'll explain the arguments in the next section. +One thing to note here is that sockets are file descriptors too.

The server displays its PID. +Give it as an argument to lsof, like you did in the section on Redirections, to visualise the file descriptors opened by receiver.py.

Quiz 1

After creating the socket, the receiver exposes itself as "listening" for connections on IP 127.0.0.1 and on port 5000. +This means that it is now ready and waiting for other processes to send messages to it.

Quiz 2

Remember: +bind()-ing to a certain port locks (like a mutex) it for the current process. +No other socket may bind() to this port until the initial socket bound to it is close()d.

Now run sender.py and type some messages. +See them appear in the terminal where receiver.py is running. +The sender creates a socket and then sends messages directly to IP 127.0.0.1 and port 5000 using sendto(). +Without stopping the first sender.py create another one in another terminal. +Type messages to both senders and see all of them displayed by receiver.py along with the addresses and ports from where they came.

In the end, both sender.py and receiver.py close() their sockets. +You can see this in analogy to regular file descriptors.

So we can use sendto() and recvfrom() to send and receive messages via sockets. +This is a very simple communication model where the receiver is like a "sink" that receives messages from anywhere. +As you can see, it has no control over who sends data to it. +To get a high-level understanding of how these messages are passed and what other models exist, head over to the next section.

Practice​

  1. Use the C sockets API to replicate the behavior of sender.py and receiver.py.

    Start from the skeleton in support/send-receive. +The workflow is the same: define the endpoint using IP (localhost) and port (5000), then communicate using sendto() and recvfrom(). +We recommend starting with sender.c and test it using receiver.py, then continue with receiver.c and test it using sender.py. +If everything is right, each sender should be able to communicate with each receiver.

  2. Use the API you've just learned about to fill in the TODOs in support/receive-challenges/receive_net_dgram_socket.c.

    This is like receiver.py. +For it to run properly, you should compile it using make, then run it and after that run send_net_dgram_socket. +If you solved the challenge correctly, receive_net_dgram_socket should display the flag.

+ + + + \ No newline at end of file diff --git a/17/Lab/IO/zero-copy/index.html b/17/Lab/IO/zero-copy/index.html new file mode 100644 index 0000000000..8e19d26a94 --- /dev/null +++ b/17/Lab/IO/zero-copy/index.html @@ -0,0 +1,33 @@ + + + + + +Zero-Copy | Operating Systems + + + + +
+
Skip to main content

Zero-Copy

Imagine a server that responds with files that it stores locally. +Its actions would be those highlighted in the image below:

  1. Receive a new request and extract the filename
  2. Read the filename from the disk into memory
  3. Send the file from memory to the client

Client-Server Steps

The quiz below is tricky, yet very important. +Do NOT skip it in order for this section to make sense!

Quiz

As you might have guessed, 2 of these copies are useless. +Since the app doesn't modify the file, there's no need for it to store the file in its own buffer. +It would be more efficient to use a single kernel buffer as intermediate storage between the disk and the NIC, as shown in the image below.

Server Copies - Zero-Copy

For an easier comparison with the "classic" read() + send() model, here's the first version again:

Server Copies - Read-Send

It should be obvious that the former approach is more efficient than the latter. +The syscall with which we can leverage zero-copy is called sendfile().

Practice: Measure It​

So we have all the reasons to believe zero-copy is the faster of the two approaches we know. +But belief alone is meaningless. +Let's test it!

First, look at the code in support/zero-copy/server.py. +It spawns 2 threads. +One of them listens on port 8081 and serves connections via read() and send(). +The other listens on port 8082 and serves connections via sendfile(). +As you can see, the difference between them is minimal.

First generate the test file using the Makefile. +Then start the server in one terminal. +Now, in another one, use benchmark_client.py to benchmark both implementations. +Below are some generic results. +Yours might differ by quite a lot, as they depend on your disk, your NIC, your kernel, your Python version, the load on your system etc.

student@os:/.../support/zero-copy$ python3 benchmark_client.py read-send
Time taken: 7.175773588009179 seconds

student@os:/.../support/zero-copy$ python3 benchmark_client.py sendfile
Time taken: 3.71454380400246 seconds

This is quite good! +Using sendfile() halves the number of copies needed from 4 to 2. +Thus, it makes sense for the running time to roughly halve as well.

Quiz

You can read a slightly more detailed article about zero-copy here.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/app-investigate/index.html b/17/Lab/Software Stack/app-investigate/index.html new file mode 100644 index 0000000000..cc6849d15c --- /dev/null +++ b/17/Lab/Software Stack/app-investigate/index.html @@ -0,0 +1,21 @@ + + + + + +App Investigation | Operating Systems + + + + +
+
Skip to main content

App Investigation

Let's spend some time investigating actual applications residing on the local system. +For now, we know that applications are developed using high-level languages and then compiled or interpreted to use the lower-layer interfaces of the software stack, such as the system call API.

Let's enter the support/app-investigate/ folder and run the get_app_types.sh script:

student@os:~/.../lab/support/app-investigate$ ./get_app_types.sh
binary apps: 2223
Perl apps: 256
Shell apps: 454
Python apps: 123
Other apps: 27

The script prints the types of the application executables in the system. +The output will differ between systems, given each has particular types of applications installed.

We list them by running the command inside the get_app_types.sh script:

student@os:~/.../lab/support/app-investigate$ find /usr/bin /bin /usr/sbin /sbin -type f -exec file {} \;
[...]
/usr/bin/qpdldecode: ELF 64-bit LSB shared object, x86-64 [...]
/usr/bin/mimeopen: Perl script text executable
[...]

As above, the output will differ between systems.

So, depending on the developers' choice, applications may be:

  • compiled into executables, from compiled languages such as C, C++, Go, Rust, D
  • developed as scripts, from interpreted languages such as Python, Perl, JavaScript

Practice​

Enter the support/app-investigate/ folder and go through the practice items below. +Select a binary executable application and a scripted application from those listed above.

  1. Use ldd on the two applications. +Notice the resulting messages and explain the results.

  2. Use ltrace and strace on the two applications. +Follow the library calls and the system calls done by each application.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/arena/index.html b/17/Lab/Software Stack/arena/index.html new file mode 100644 index 0000000000..ccc1a705cb --- /dev/null +++ b/17/Lab/Software Stack/arena/index.html @@ -0,0 +1,55 @@ + + + + + +Arena | Operating Systems + + + + +
+
Skip to main content

Arena

Go through the practice items below to hone your skills in working with layers of the software stack.

System Calls​

Enter the support/basic-syscall/ folder and go through the practice items below. +If you get stuck, take a sneak peek at the solutions in the solution/basic-syscall/ folder.

For debugging, use strace to trace the system calls from your program and make sure the arguments are set right.

  1. Update the hello.asm and / or hello.s files to pause the execution of the program before the exit system call.

    You need to make the sys_pause system call, with no arguments. +Find its ID here.

  2. Update the hello.asm and / or hello.s files to read a message from standard input and print it to standard output.

    You'll need to define a buffer in the data or bss section. +Use the read system call to read data in the buffer. +The return value of read (placed in the rax register) is the number of bytes read. +Use that value as the 3rd argument or write, i.e. the number of bytes printed.

    Find the ID of the read system call here. +To find out more about its arguments, see its man page. +Standard input descriptor is 0.

  3. Difficult: Port the initial program to ARM on 64 bits (also called aarch64).

    Use the skeleton files in the arm/ folder. +Find information about the aarch64 system calls here.

  4. Create your own program, written in assembly, doing some system calls you want to learn more about. +Some system calls you could try: open, rename, mkdir. +Create a Makefile for that program. +Run the resulting program with strace to see the actual system calls being made (and their arguments).

System Call Wrappers​

Enter the support/syscall-wrapper/ folder and go through the practice items below. +If you get stuck, take a sneak peek at the solutions in the solution/syscall-wrapper/ folder.

  1. Update the files in the syscall-wrapper/ folder to make the getpid system call available as a wrapper. +Create a function with the signature unsigned int itoa(int n, char *a) that converts an integer to a string. +It returns the number of digits in the string. +For example, it will convert the number 1234 to the string "1234" string (NULL-terminated, 5 bytes long); +the return value is 4 (the number of digits of the "1234" string).

    Then make the call to getpid; +it gets no arguments and returns an integer (the PID - process ID of the current process).

Common Functions​

Enter the support/common-functions/ folder and go through the practice items below. +If you get stuck, take a sneak peek at the solutions in the solution/common-functions/ folder.

  1. Update the putchar() function in main_printf.c to implement a buffered functionality of printf(). +Characters passed via the putchar() call will be stored in a predefined static global buffer. +The write() call will be invoked when a newline is encountered or when the buffer is full. +This results in a reduced number of write system calls. +Use strace to confirm the reduction of the number of write system calls.

  2. Update the main_printf.c file to also feature a flush() function that forces the flushing of the static global buffer and a write system call. +Make calls to printf() and flush() to validate the implementation. +Use strace to inspect the write system calls invoked by printf() and flush().

Libraries and libc​

Enter the support/libc/ folder and go through the practice items below. +If you get stuck, take a sneak peek at the solutions in the solution/libc/ folder.

  1. Inside the vendetta.c file make a call open("a.txt", O_RDWR | O_CREAT, 0644) to open / create the a.txt file. +Make sure you include all required headers. +Check the system call being made.

    Make an fopen() with the proper arguments that gets as close as possible to the open() call, i.e. the system call arguments are as close as possible.

  2. Inside the vendetta.c file make a call to sin() function (for sine). +Compute sin(0) and sin(PI/2).

High-Level Languages​

Enter the support/high-level-lang/ folder and go through the practice items below. +If you get stuck, take a sneak peek at the solutions in the solution/high-level-lang/ folder.

  1. Create programs in C, Python and Go that compute the N-th Fibonacci number. +N is passed as a command-line argument.

    Use ltrace and strace to compute the number of library calls and system calls. +Use perf to measure the running time.

    Compare the values of the three programs.

  2. Create programs in C, Python and Go that copy a source file into a destination file. +Both files are passed as the two command-line arguments for the program. +Sample run:

    student@so:~$ cp src dest

    Use ltrace and strace to compute the number of library calls and system calls. +Use perf to measure the running time. +Use source files of different sizes. +Compare the outputs of these commands on the three programs.

App Investigation​

Enter the support/app-investigation/ folder and go through the practice items below.

  1. Check to see whether there are statically-linked application executables in the system. +The file command tells if the file passed as argument is a statically-linked executable. +If you can't find one, install the busybox-static package.

  2. Look into what busybox is and explain why it's custom to have it as statically-linked executable.

  3. Run ldd, nm, strace, ltrace on a statically-linked application executable. +Explain the results.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/basic-syscall/index.html b/17/Lab/Software Stack/basic-syscall/index.html new file mode 100644 index 0000000000..08a3f623bf --- /dev/null +++ b/17/Lab/Software Stack/basic-syscall/index.html @@ -0,0 +1,33 @@ + + + + + +Analyzing the Software Stack | Operating Systems + + + + +
+
Skip to main content

Analyzing the Software Stack

To get a better grasp on how the software stack works, let's do a bottom-up approach: +we build and run different programs, that start off by using the system call API (the lowest layer in the software stack) and progressively use higher layers.

System Calls Explained​

A system call, or syscall for short, is a method used by applications to communicate with the operating system's kernel.

The need for syscalls is tied to the modern operating systems model of conceptually separating into kernel space and user space.

The kernel space manages the hardware resources such as CPU, I/O devices, disk or memory. +Moreover, the kernel also provides an interface for the user space applications to interact with the hardware.

The user space is where you are running your applications and processes. +From the user space, we cannot directly access the hardware or perform privileged operations. +You need to use syscalls to perform privileged operations such as accessing the hardware.

Below, you can see some examples of system calls and what resource they request from the kernel:

  • brk() is used to allocate memory

  • open() is used to access the file system and open a specific file

  • write() is used to access the file system and modify the contents of a specific file

System Call API Explained

Basic System Calls​

The support/basic-syscall/ folder stores the implementation of a simple program in assembly language for the x86_64 (64 bit) architecture. +The program invokes two system calls: write and exit. +The program is duplicated in two files using the two x86 assembly language syntaxes: the Intel / NASM syntax (hello.asm) and the AT&T / GAS syntax (hello.s).

The implementation follows the x86_64 Linux calling convention:

  • system call ID is passed in the rax register
  • system call arguments are passed, in order, in the rdi, rsi, rdx, r10, r8, r9 registers

Let's build and run the two programs:

student@os:~/.../lab/support/basic-syscall$ ls
hello.asm  hello.s  Makefile

student@os:~/.../lab/support/basic-syscall$ make
nasm -f elf64 -o hello-nasm.o hello.asm
cc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none  hello-nasm.o   -o hello-nasm
gcc -c -o hello-gas.o hello.s
cc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none  hello-gas.o   -o hello-gas

student@os:~/.../lab/support/basic-syscall$ ls
hello.asm  hello-gas  hello-gas.o  hello-nasm  hello-nasm.o  hello.s  Makefile

student@os:~/.../lab/support/basic-syscall$ ./hello-nasm
Hello, world!
student@os:~/.../lab/support/basic-syscall$ ./hello-gas
Hello, world!

The two programs end up printing the Hello, world! message at standard output by issuing the write system call. +Then they complete their work by issuing the exit system call.

The write system call writes a buffer to the file referred by the first argument, which is the file descriptor. +File descriptors are going to be studied in-depth in future chapters. +For now, it is enough for you to know that they are integers that behave like file handlers. +The 3 most common file descriptors are:

  • 0 references the standard input (stdin)

  • 1 references the standard output (stdout)

  • 2 references the standard error (stderr)

Use man 2 write and man 3 exit to get a detailed understanding of the syntax and use of the two system calls. +You can also check the online man pages: write, exit

We use strace to inspect system calls issued by a program:

student@os:~/.../lab/support/basic-syscall$ strace ./hello-nasm
execve("./hello-nasm", ["./hello-nasm"], 0x7ffc4e175f00 /* 63 vars */) = 0
write(1, "Hello, world!\n", 14Hello, world!
)         = 14
exit(0)                                 = ?
+++ exited with 0 +++

There are three system calls captured by strace:

  • execve: this is issued by the shell to create the new process; +you'll find out more about execve in the "Compute" chapter
  • write: called by the program to print Hello, world! to standard output
  • exit: to exit the program

This is the most basic program for doing system calls. +Given that system calls require a specific calling convention, their invocation can only be done in assembly language. +Obviously, this is not portable (specific to a given CPU architecture, x86_64 in our case) and too verbose and difficult to maintain. +For portability and maintainability, we require a higher level language, such as C. +In order to use C, we need function wrappers around system calls.

Practice​

Update the hello.asm and / or hello.s files to print both Hello, world! and Bye, world!. +This means adding another write system call.

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/common-functions/index.html b/17/Lab/Software Stack/common-functions/index.html new file mode 100644 index 0000000000..ea3a41d3bf --- /dev/null +++ b/17/Lab/Software Stack/common-functions/index.html @@ -0,0 +1,33 @@ + + + + + +Common Functions | Operating Systems + + + + +
+
Skip to main content

Common Functions

By using wrapper calls, we are able to write our programs in C. +However, we still need to implement common functions for string management, working with I/O, working with memory.

The simple attempt is to implement these functions (printf() or strcpy() or malloc()) once in a C source code file and then reuse them when needed. +This saves us time (we don't have to reimplement) and allows us to constantly improve one implementation constantly; +there will only be one implementation that we update to increase its safety, efficiency or performance.

The support/common-functions/ folder stores the implementation of string management functions, in string.c and string.h and of printing functions in printf.c and printf.h. +The printf implementation is this one.

There are two programs: main_string.c showcases string management functions, main_printf.c showcases the printf() function.

main_string.c depends on the string.h and string.c files that implement the strlen() and strcpy() functions. +We print messages using the write() system call wrapper implemented in syscall.s

Let's build and run the program:

student@os:~/.../lab/support/common-functions$ make main_string
gcc -fno-stack-protector   -c -o main_string.o main_string.c
gcc -fno-stack-protector   -c -o string.o string.c
nasm -f elf64 -o syscall.o syscall.s
gcc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none  main_string.o string.o syscall.o   -o main_string

student@os:~/.../lab/support/common-functions$ ./main_string
Destination string is: warhammer40k

student@os:~/.../lab/support/common-functions$ strace ./main_string
execve("./main_string", ["./main_string"], 0x7ffd544d0a70 /* 63 vars */) = 0
write(1, "Destination string is: ", 23Destination string is: ) = 23
write(1, "warhammer40k\n", 13warhammer40k
)          = 13
exit(0)                                 = ?
+++ exited with 0 +++

When using strace we see that only the write() system call wrapper triggers a system call. +There are no system calls triggered by strlen() and strcpy() as can be seen in their implementation.

In addition, main_printf.c depends on the printf.h and printf.c files that implement the printf() function. +There is a requirement to implement the _putchar() function; +we implement it in the main_printf.c file using the write() syscall call wrapper. +The main() function main_printf.c file contains all the string and printing calls. +printf() offers a more powerful printing interface, allowing us to print addresses and integers.

Let's build and run the program:

student@os:~/.../lab/support/common-functions$ make main_printf
gcc -fno-stack-protector   -c -o printf.o printf.c
gcc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none  main_printf.o printf.o string.o syscall.o   -o main_printf

student@os:~/.../lab/support/common-functions$ ./main_printf
[before] src is at 00000000004026A0, len is 12, content: "warhammer40k"
[before] dest is at 0000000000603000, len is 0, content: ""
copying src to dest
[after] src is at 00000000004026A0, len is 12, content: "warhammer40k"
[after] dest is at 0000000000603000, len is 12, content: "warhammer40k"

student@os:~/.../lab/support/common-functions$ strace ./main_printf
execve("./main_printf", ["./main_printf"], 0x7ffcaaa1d660 /* 63 vars */) = 0
write(1, "[", 1[)                        = 1
write(1, "b", 1b)                        = 1
write(1, "e", 1e)                        = 1
write(1, "f", 1f)                        = 1
write(1, "o", 1o)                        = 1
write(1, "r", 1r)                        = 1
write(1, "e", 1e)                        = 1
write(1, "]", 1])                        = 1
write(1, " ", 1 )                        = 1
write(1, "s", 1s)                        = 1
write(1, "r", 1r)                        = 1
[...]

We see that we have greater printing flexibility with the printf() function. +However, one downside of the current implementation is that it makes a system call for each character. +This is inefficient and could be improved by printing a whole string.

Practice​

Enter the support/common-functions/ folder and go through the practice items below.

  1. Update string.c and string.h to make available the strcat() function. +Call that function in main_string.c and print the result.

  2. Update the main_printf.c file to use the implementation of sprintf() to collect information to be printed inside a buffer. +Call the write() function to print the information. +The printf() function will no longer be called. +This results in a single write system call.

Using previously implemented functions allows us to more efficiently write new programs. +These functions provide us with extensive features that we use in our programs.

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/high-level-lang/index.html b/17/Lab/Software Stack/high-level-lang/index.html new file mode 100644 index 0000000000..2cd64431aa --- /dev/null +++ b/17/Lab/Software Stack/high-level-lang/index.html @@ -0,0 +1,29 @@ + + + + + +High-Level Languages | Operating Systems + + + + +
+
Skip to main content

High-Level Languages

Using the standard C library (libc) frees the programmer from the cumbersome steps of invoking system calls and reimplementing common features. +Still, for improved development time and safety, other programming languages can be used, such as Rust, Python, JavaScript. +Most (if not all) of these high-level programming languages still make use of the standard C library. +Such that a call to a function in Python would end-up making a call to a function in the standard C library.

The support/high-level-lang/ folder stores the implementation of a simple "Hello, World!"-printing program in Python. +We simply invoke the python interpreter to run the program:

student@os:~/.../lab/support/high-level-lang$ python hello.py
Hello, world!

We count the number of functions called from the standard C library and the number of system calls:

student@os:~/.../lab/support/high-level-lang$ ltrace -l 'libc*' python hello.py 2> libc.out
Hello, world!

student@os:~/.../lab/support/high-level-lang$ wc -l libc.out
50469 out

student@os:~/.../lab/support/high-level-lang$ strace python hello.py 2> syscall.out
Hello, world!

student@os:~/.../lab/support/high-level-lang$ wc -l syscall.out
948 syscall.out

The dynamic standard C library (libc.so.6) is a dependency of the Python interpreter (/usr/bin/python3):

student@os:~/.../lab/support/high-level-lang$ ldd /usr/bin/python3
[...]
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa6fd6d0000)
[...]

We can see the complexity of invoking the Python interpreter, resulting in more the 50,000 of library calls being made. +This means added overhead versus a simple C function. +However, this also means faster development in the Python programming language. +Each new layer in the software stack simplifies development but adds overhead.

We can use perf to compare the running time between the Python and a C "Hello, World!"-printing programs:

student@os:~/.../lab/support/high-level-lang$ sudo perf stat ../static-dynamic/hello
Hello, World!

 Performance counter stats for '../static-dynamic/hello':

              0.46 msec task-clock                #    0.559 CPUs utilized
                 0      context-switches          #    0.000 K/sec
                 0      cpu-migrations            #    0.000 K/sec
                52      page-faults               #    0.114 M/sec
           859,341      cycles                    #    1.882 GHz
           713,395      instructions              #    0.83  insn per cycle
           141,710      branches                  #  310.393 M/sec
             6,208      branch-misses             #    4.38% of all branches

       0.000816974 seconds time elapsed

       0.000872000 seconds user
       0.000000000 seconds sys

student@os:~/.../lab/support/high-level-lang$ sudo perf stat python hello.py
Hello, world!

 Performance counter stats for 'python hello.py':

             69.39 msec task-clock                #    0.992 CPUs utilized
                 2      context-switches          #    0.029 K/sec
                 0      cpu-migrations            #    0.000 K/sec
             1,115      page-faults               #    0.016 M/sec
        74,405,125      cycles                    #    1.072 GHz
        84,957,056      instructions              #    1.14  insn per cycle
        18,574,724      branches                  #  267.689 M/sec
           759,104      branch-misses             #    4.09% of all branches

       0.069981351 seconds time elapsed

       0.054376000 seconds user
       0.015536000 seconds sys

We can see that on all metrics, the running of the Python program is less efficient than the running of the C program. +The Python code takes 69 milliseconds, whereas the C code runs in less than 1 millisecond.

When deciding what programming language and what libraries and software components to use, you have to balance requirements for fast development and increased safety (inherent to higher-level programming languages) with requirements for speed or efficiency (common to lower-level programming languages such as C). +Newer modern programming languages such as Go, Rust, D aim to add the benefits of high-level programming languages and keep efficiency close to the C programming language. +Generally, additional software layers (libraries, language environments, interpreters) simplify development but decrease speed and efficiency.

Practice​

Enter the support/high-level-lang/ folder and go through the practice items below.

  1. Use make to create the hello executable from the hello.go file (a Go "Hello, World!"-printing program). +Use ltrace and strace to compute the number of library calls and system calls. +Use perf to measure the running time.

    Compare the values with those from the "Hello, World!"-printing programs in C and Python.

  2. Create a "Hello, World!"-printing program in a programming language of your choice (other than C, Python and Go). +Find the values above (library calls, system calls and running time).

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/index.html b/17/Lab/Software Stack/index.html new file mode 100644 index 0000000000..9a0c9d421c --- /dev/null +++ b/17/Lab/Software Stack/index.html @@ -0,0 +1,16 @@ + + + + + +Software Stack | Operating Systems + + + + +
+
Skip to main content

Software Stack

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/libc/index.html b/17/Lab/Software Stack/libc/index.html new file mode 100644 index 0000000000..fbdb5d5934 --- /dev/null +++ b/17/Lab/Software Stack/libc/index.html @@ -0,0 +1,43 @@ + + + + + +Libraries and libc | Operating Systems + + + + +
+
Skip to main content

Libraries and libc

Once we have common functions implemented, we can reuse them at any time. +The main unit for software reusability is the library. +In short, a library is a common machine code that can be linked against different other software components. +Each time we want to use the printf() function or the strlen() function, we don't need to reimplement them. +We also don't need to use existing source code files, rebuild them and reuse them. +We (re)use existing machine code in libraries.

A library is a collection of object files that export given data structures and functions to be used by other programs. +We create a program, we compile and then we link it against the library for all the features it provides.

The most important library in modern operating systems is the standard C library, also called libc. +This is the library providing system call wrappers and basic functionality for input-output, string management, memory management. +By default, a program is always linked with the standard C library. +In the examples above, we've explicitly disabled the use of the standard C library with the help of the -nostdlib linker option.

By using the standard C library, it's much easier to create new programs. +You call existing functionality in the library and implement only features particular to your program.

The support/libc/ folder stores the implementation of programs using the standard C library: hello.c, main_string.c and main_printf.c. +These programs are almost identical to those used in the past sections:

  • hello.c is similar to the programs in solution/basic-syscall/ and solution/syscall-wrapper/
  • main_string.c and main_printf.c are similar to the programs in solution/common-functions/

Let's build and run them:

student@os:~/.../lab/support/libc$ ls
hello  hello.c  hello.o  main_printf  main_printf.c  main_printf.o  main_string  main_string.c  main_string.o  Makefile

student@os:~/.../lab/support/libc$ make clean
rm -f hello hello.o
rm -f main_printf main_printf.o
rm -f main_string main_string.o

student@os:~/.../lab/support/libc$ ls
hello.c  main_printf.c  main_string.c  Makefile

student@os:~/.../lab/support/libc$ make
cc -Wall   -c -o hello.o hello.c
cc -static  hello.o   -o hello
cc -Wall   -c -o main_printf.o main_printf.c
cc -static  main_printf.o   -o main_printf
cc -Wall   -c -o main_string.o main_string.c
cc -static  main_string.o   -o main_string

student@os:~/.../lab/support/libc$ ls
hello  hello.c  hello.o  main_printf  main_printf.c  main_printf.o  main_string  main_string.c  main_string.o  Makefile

student@os:~/.../lab/support/libc$ ./hello
Hello, world!
Bye, world!
aaa
aaa
^C

student@os:~/.../lab/support/libc$ ./main_string
Destination string is: warhammer40k

student@os:~/.../lab/support/libc$ ./main_printf
[before] src is at 0x492308, len is 12, content: "warhammer40k"
[before] dest is at 0x6bb340, len is 0, content: ""
copying src to dest
[after] src is at 0x492308, len is 12, content: "warhammer40k"
[after] dest is at 0x6bb340, len is 12, content: "warhammer40k"
abc

The behavior / output is similar to the ones in the previous sections:

student@os:~/.../lab/support/libc$ ../../solution/basic-syscall/hello-nasm
Hello, world!
Bye, world!
aaa
aaa
^C

student@os:~/.../lab/support/libc$ ../../solution/common-functions/main_string
Destination string is: warhammer40k

student@os:~/.../lab/support/libc$ ../../solution/common-functions/main_printf
[before] src is at 0000000000402680, len is 12, content: "warhammer40k"
[before] dest is at 0000000000604000, len is 0, content: ""
copying src to dest
[after] src is at 0000000000402680, len is 12, content: "warhammer40k"
[after] dest is at 0000000000604000, len is 12, content: "warhammer40k"
abc

We can inspect the system calls made to check the similarities. +For example, for the main_printf program we get the outputs:

student@os:~/.../lab/support/libc$ strace ./main_printf
execve("./main_printf", ["./main_printf"], 0x7fff7b38c240 /* 66 vars */) = 0
brk(NULL)                               = 0x15af000
brk(0x15b01c0)                          = 0x15b01c0
arch_prctl(ARCH_SET_FS, 0x15af880)      = 0
uname({sysname="Linux", nodename="[...]", ...}) = 0
readlink("/proc/self/exe", "[...]/operating"..., 4096) = 105
brk(0x15d11c0)                          = 0x15d11c0
brk(0x15d2000)                          = 0x15d2000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 18), ...}) = 0
write(1, "[before] src is at 0x492308, len"..., 64[before] src is at 0x492308, len is 12, content: "warhammer40k"
) = 64
write(1, "[before] dest is at 0x6bb340, le"..., 52[before] dest is at 0x6bb340, len is 0, content: ""
) = 52
write(1, "copying src to dest\n", 20copying src to dest
)   = 20
write(1, "[after] src is at 0x492308, len "..., 63[after] src is at 0x492308, len is 12, content: "warhammer40k"
) = 63
write(1, "[after] dest is at 0x6bb340, len"..., 64[after] dest is at 0x6bb340, len is 12, content: "warhammer40k"
) = 64
write(1, "ab", 2ab)                       = 2
write(1, "c\n", 2c
)                      = 2
exit_group(0)                           = ?
+++ exited with 0 +++

student@os:~/.../lab/support/libc$ strace ../../solution/common-functions/main_printf
execve("../../solution/common-functions/main_printf", ["../../solution/common-functions/"...], 0x7ffe204eec00 /* 66 vars */) = 0
write(1, "[before] src is at 0000000000402"..., 72[before] src is at 0000000000402680, len is 12, content: "warhammer40k"
) = 72
write(1, "[before] dest is at 000000000060"..., 60[before] dest is at 0000000000604000, len is 0, content: ""
) = 60
write(1, "copying src to dest\n", 20copying src to dest
)   = 20
write(1, "[after] src is at 00000000004026"..., 71[after] src is at 0000000000402680, len is 12, content: "warhammer40k"
) = 71
write(1, "[after] dest is at 0000000000604"..., 72[after] dest is at 0000000000604000, len is 12, content: "warhammer40k"
) = 72
write(1, "ab", 2ab)                       = 2
write(1, "c\n", 2c
)                      = 2
exit(0)                                 = ?
+++ exited with 0 +++

The output is similar, with differences at the beginning and the end of the system call trace. +In the case of the libc-built program, a series of additional system calls (brk, arch_prctl, uname etc.) are made. +Also, there is an implicit call to exit_group instead of an explicit one to exit in the non-libc case. +These are initialization and cleanup routines that are implicitly added when using the standard C library. +They are generally used for setting and cleaning up the stack, environment variables and other pieces of information required by the program or the standard C library itself.

We could argue that the initialization steps incur overhead, and that's a downside of using the standard C library. +However, these initialization steps are required for almost all programs. +And, given that almost all programs make use of the basic features of the standard C library, libc is almost always used. +We can say the above were exceptions to the rule, where we didn't make use of the standard C library.

Summarizing, the advantages and disadvantages of using the standard C library are:

  • (+) easier development: do calls to existing functions already implemented in the standard C library; +default build and link flags
  • (+) portability: if the system provides a standard C library, one calls the library functions that will then interact with the lower-layer API
  • (+) implicit initialization and cleanup: no need for you do explicitly create them
  • (-) usually larger in size (static) executables
  • (-) a level of overhead as the standard C library wraps system calls
  • (-) potential security issues: a larger set of (potentially vulnerable) functions are presented by the standard C library

Practice​

Enter the support/libc/ folder and go through the practice items below.

  1. Use malloc() and free() functions in the memory.c program. +Make your own use of the allocated memory.

    It's very easy to use memory management functions with the libc. +The alternative (without the libc) would be more cumbersome.

    Use different values for malloc(), i.e. the allocation size. +Use strace to check the system calls invoked by malloc() and free(). +You'll see that, depending on the size, the brk or mmap / munmap system calls are invoked. +And for certain calls to malloc() / free() no syscall is happening. +You'll find more about them in the Data chapter.

  2. Create your own C program with calls to the standard C library in vendetta.c. +Be as creative as you can about the types of functions being made.

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/libcall-syscall/index.html b/17/Lab/Software Stack/libcall-syscall/index.html new file mode 100644 index 0000000000..9484ef69d1 --- /dev/null +++ b/17/Lab/Software Stack/libcall-syscall/index.html @@ -0,0 +1,25 @@ + + + + + +Library calls vs system calls | Operating Systems + + + + +
+
Skip to main content

Library calls vs system calls

The standard C library has primarily two uses:

  1. wrapping system calls into easier to use C-style library calls, such as open(), write(), read()
  2. adding common functionality required for our program, such as string management (strcpy), memory management (malloc()) or formatted I/O (printf())

The first use means a 1-to-1 mapping between library calls and system calls: one library call means one system call. +The second group doesn't have a standard mapping. +A library call could be mapped to no system calls, one system call, two or more system calls, or it may depend (a system call may or may not happen).

The support/libcall-syscall/ folder stores the implementation of a simple program that makes different library calls. +Let's build the program and then trace the library calls (with ltrace) and the system calls (with strace):

student@os:~/.../lab/support/libcall-syscall$ make
cc -Wall   -c -o call.o call.c
cc   call.o   -o call
cc -Wall   -c -o call2.o call2.c
cc   call2.o   -o call2

student@os:~/.../lab/support/libcall-syscall$ ltrace ./call
fopen("a.txt", "wt")                                                                                             = 0x556d57679260
strlen("Hello, world!\n")                                                                                        = 14
fwrite("Hello, world!\n", 1, 14, 0x556d57679260)                                                                 = 14
strlen("Bye, world!\n")                                                                                          = 12
fwrite("Bye, world!\n", 1, 12, 0x556d57679260)                                                                   = 12
fflush(0x556d57679260)                                                                                           = 0
+++ exited (status 0) +++

student@os:~/.../lab/support/libcall-syscall$ strace ./call
[...]
openat(AT_FDCWD, "a.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fstat(3, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0
write(3, "Hello, world!\nBye, world!\n", 26) = 26
exit_group(0)                           = ?
+++ exited with 0 +++

We have the following mappings:

  • The fopen() library call invokes the openat and the fstat system calls.
  • The fwrite() library call invokes no system calls.
  • The strlen() library call invokes no system calls.
  • The fflush() library call invokes the write system call.

This all seems to make sense. +The main reason for fwrite() not making any system calls is the use of a standard C library buffer. +Calls the fwrite() end up writing to that buffer to reduce the number of system calls. +Actual system calls are made either when the standard C library buffer is full or when an fflush() library call is made.

Note that on some systems, ltrace does not work as expected, due to now binding. +To avoid this behaviour, you can force the lazy binding (based on which ltrace is constructed to work). +An example can be found in support/libcall-syscall/Makefile, however for system binaries, such as ls or pwd, the only alternative is to add the -x "*" argument to force the command to trace all symbols in the symbol table:

student@os:~$ ltrace -x "*" ls

You can always choose what library functions ltrace is investigating, by replacing the wildcard with their name:

student@os:~$ ltrace -x "malloc" -x "free" ls
malloc@libc.so.6(5)                                                    = 0x55c42b2b8910
free@libc.so.6(0x55c42b2b8910)                                         = <void>
malloc@libc.so.6(120)                                                  = 0x55c42b2b8480
malloc@libc.so.6(12)                                                   = 0x55c42b2b8910
malloc@libc.so.6(776)                                                  = 0x55c42b2b8930
malloc@libc.so.6(112)                                                  = 0x55c42b2b8c40
malloc@libc.so.6(1336)                                                 = 0x55c42b2b8cc0
malloc@libc.so.6(216)                                                  = 0x55c42b2b9200
malloc@libc.so.6(432)                                                  = 0x55c42b2b92e0
malloc@libc.so.6(104)                                                  = 0x55c42b2b94a0
malloc@libc.so.6(88)                                                   = 0x55c42b2b9510
malloc@libc.so.6(120)                                                  = 0x55c42b2b9570
[...]

If you would like to know more about lazy binding, now binding or PLT entries, check out this blog post.

Practice​

Enter the support/libcall-syscall/ folder and go through the practice items below.

  1. Check library calls and system calls for the call2.c file. +Use ltrace and strace.

    Find explanations for the calls being made and the library call to system call mapping.

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/modern-sw-stack/index.html b/17/Lab/Software Stack/modern-sw-stack/index.html new file mode 100644 index 0000000000..3a8db9ff83 --- /dev/null +++ b/17/Lab/Software Stack/modern-sw-stack/index.html @@ -0,0 +1,25 @@ + + + + + +Modern Software Stacks | Operating Systems + + + + +
+
Skip to main content

Modern Software Stacks

Most modern computing systems use a software stack such as the one in the figure below:

Modern Software Stack

This modern software stack allows fast development and provides a rich set of applications to the user.

The basic software component is the operating system (OS) (technically the operating system kernel). +The OS provides the fundamental primitives to interact with hardware (read and write data) and to manage the running of applications (such as memory allocation, thread creation, scheduling). +These primitives form the system call API or system API. +An item in the system call API, i.e. the equivalent of a function call that triggers the execution of a functionality in the operating system, is a system call.

The system call API is well-defined, stable and complete: it exposes the entire functionality of the operating system and hardware. +However, it is also minimalistic with respect to features, and it provides a low-level (close to hardware) specification, making it cumbersome to use and not portable.

Due to the downsides of the system call API, a basic library, the standard C library (also called libc), is built on top of it. +Because the system call API uses an OS-specific calling convention, the standard C library typically wraps each system call into an equivalent function call, following a portable calling convention. +More than these wrappers, the standard C library provides its own API that is typically portable. +Part of the API exposed by the standard C library is the standard C API, also called ANSI C or ISO C; +this API is typically portable across all platforms (operating systems and hardware). +This API, going beyond system call wrappers, has several advantages:

  • portability: irrespective of the underlying operating system (and system call API), the API is the same
  • extensive features: string management, I/O formatting
  • possibility of increased efficiency with techniques such as buffering, as we show later
+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/overview/index.html b/17/Lab/Software Stack/overview/index.html new file mode 100644 index 0000000000..6d41329afd --- /dev/null +++ b/17/Lab/Software Stack/overview/index.html @@ -0,0 +1,30 @@ + + + + + +Software Stack | Operating Systems + + + + +
+
Skip to main content

Software Stack

Software comprises of code and data that is loaded in memory and used by the CPU. +Code means instructions that are to be fetched by the CPU, decoded and executed. +This is called machine code, i.e. binary instructions that are understood by the CPU.

Hardware and Software

So, when compared to hardware, software is highly flexible. +We can tie together specific instructions to handle a given task and run them on hardware (CPU, memory, I/O). +Different pieces of these instructions solve different tasks and run on the same hardware. +Moreover, these pieces of instructions can be duplicated and run on different pieces of hardware, thus providing software reusability. +All we are left with is creating those pieces of instructions, also called programs.

In summary, software has intrinsic characteristics:

  • flexibility: We can (easily) create new pieces of software. +Little is required, we don't need raw materials as in the case of hardware or housing or transportation.
  • reusability: Software can be easily copied to new systems and provide the same benefits there.

Other characteristics are important to have, as they make life easier for both users and developers of software:

  • portability: This is the ability to build and run the same program on different computing platforms. +This allows a developer to write the application code once and then run it everywhere.
  • fast development: We want developers to be able to write code faster, using higher-level programming languages.

The last two characteristics rely on two items:

  • higher-level programming languages: As discussed above, a compiler will take a higher-level program and transform it into binary code for different computing platforms, thus providing portability. +Also, it's easier to read (comprehend) and write (develop) source code in a higher-level programming language, thus providing fast development.

  • software stacks: A software stack is the layering of software such that each lower layer provides a set of features that the higher layer can directly use. +This means that there is no need for the higher layer to reimplement those features; +this provides fast development: focus on only the newer / required parts of software.

    Also, each lower layer provides a generic interface to the higher layer. +These generic interfaces "hides" possible differences in the even lower layers. +This way, a software stack ensures portability across different other parts of software (and hardware as well). +For example, the standard C library, that we will present shortly, ensures portability across different operating systems.

Software Stack

Quiz

Contents​

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/common-functions/index.html b/17/Lab/Software Stack/quiz/common-functions/index.html new file mode 100644 index 0000000000..404bd6166f --- /dev/null +++ b/17/Lab/Software Stack/quiz/common-functions/index.html @@ -0,0 +1,17 @@ + + + + + +Common Functions | Operating Systems + + + + +
+
Skip to main content

Common Functions

printf() System Call​

Question Text​

What system call does the printf() function invoke?

Question Answers​

  • read
  • write

  • exec

  • exit

Feedback​

printf() invokes the write system call to print messages to standard output.

strcpy() System Call​

Question Text​

What system call does the strcpy() function invoke?

Question Answers​

  • cpy

  • touch

  • memcpy

  • no system call

Feedback​

strcpy() doesn't invoke system calls, because it doesn't require any feature that is only provided by the operating system

printf() vs write​

Question Text​

What are features provided by printf() when compared to write? (choose 2 answers)

Question Answers​

  • buffering

  • outputs to standard output

  • may write to file

  • does output formatting
  • can work with binary data

Feedback​

printf() can do buffering to reduce the number of system calls. +Also, printf(), as it name suggests (the f suffix), does output formatting.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/high-level-lang/index.html b/17/Lab/Software Stack/quiz/high-level-lang/index.html new file mode 100644 index 0000000000..74eac2697a --- /dev/null +++ b/17/Lab/Software Stack/quiz/high-level-lang/index.html @@ -0,0 +1,17 @@ + + + + + +Python Tools | Operating Systems + + + + +
+
Skip to main content

Python Tools

Question Text​

A Python program is not a proper argument for ...

Question Answers​

  • ltrace

  • strace

  • rm

  • ldd

Feedback​

Because a Python program is a script to be interpreted, it won't be usable by ldd. +ldd is passed a binary executable to look for library dependencies.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/libc/index.html b/17/Lab/Software Stack/quiz/libc/index.html new file mode 100644 index 0000000000..26b7d394ff --- /dev/null +++ b/17/Lab/Software Stack/quiz/libc/index.html @@ -0,0 +1,16 @@ + + + + + +libc | Operating Systems + + + + +
+
Skip to main content

libc

malloc()​

Question Text​

What system calls are invoked by the malloc() library call for Linux libc? (choose 2 answers)

Question Answers​

  • brk

  • free

  • dup

  • mmap
  • copy

Feedback​

Depending on the allocation size, malloc() invokes brk or mmap.

Syscall Tool​

Question Text​

Which of following is not and advantage of using libc for programs?

Question Answers​

  • increased portability
  • reduced executable size

  • richer set of features

  • easier development

Feedback​

When using libc, because we add a new software component, the size of the resulting executable increases.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/libcall-syscall/index.html b/17/Lab/Software Stack/quiz/libcall-syscall/index.html new file mode 100644 index 0000000000..4739a90760 --- /dev/null +++ b/17/Lab/Software Stack/quiz/libcall-syscall/index.html @@ -0,0 +1,17 @@ + + + + + +Libcall with Syscall | Operating Systems + + + + +
+
Skip to main content

Libcall with Syscall

Question Text​

Which of the following library calls will for sure invoke a system call?

Question Answers​

  • fopen()

  • fwrite()

  • printf()

  • strcpy()

Feedback​

fopen() requires opening a file and access to the operating system (for filesystem access). +The others may not require a system call (strcpy()) or may use buffering to delay the invocation of a system call (fwrite(), printf()).

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/libs/index.html b/17/Lab/Software Stack/quiz/libs/index.html new file mode 100644 index 0000000000..87a2c1fa16 --- /dev/null +++ b/17/Lab/Software Stack/quiz/libs/index.html @@ -0,0 +1,17 @@ + + + + + +libs | Operating Systems + + + + +
+
Skip to main content

libs

Static Executables​

Question Text​

Which of the following tools has no influence over statically-linked executables? (choose 2 answers)

Question Answers​

  • ltrace

  • strace

  • nm

  • ldd
  • rm

Feedback​

ltrace and ldd are used to capture connections to dynamic libraries. +They have no effect on statically-linked executables.

Dynamic Libraries​

Question Text​

Which of the following is a dynamic library?

Question Answers​

  • libc.a
  • libc.so

  • libc.exe

  • libc.c

Feedback​

The .so (shared object) extension is used for dynamic libraries in Linux.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/software/index.html b/17/Lab/Software Stack/quiz/software/index.html new file mode 100644 index 0000000000..d0c0776f3a --- /dev/null +++ b/17/Lab/Software Stack/quiz/software/index.html @@ -0,0 +1,18 @@ + + + + + +Software Properties | Operating Systems + + + + +
+
Skip to main content

Software Properties

Question Text​

Which of the following is not an advantage of software when compared to hardware?

Question Answers​

  • reusability
  • performance

  • portability

  • flexibility

Feedback​

Software is reusable and flexible, unlike physical hardware that's rigid and used-once. +Software is also portable across different hardware (and software platforms). +However, software relies on hardware for running, so software will never beat hardware in terms of sheer speed / performance.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/syscall-wrapper/index.html b/17/Lab/Software Stack/quiz/syscall-wrapper/index.html new file mode 100644 index 0000000000..a8efcfe46f --- /dev/null +++ b/17/Lab/Software Stack/quiz/syscall-wrapper/index.html @@ -0,0 +1,16 @@ + + + + + +Syscall Wrappers | Operating Systems + + + + +
+
Skip to main content

Syscall Wrappers

Question Text​

What language do we use to invoke system calls?

Question Answers​

  • assembly

  • C

  • C++

  • Go

Feedback​

System calls require setting of registers that can only be done in assembly language.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/quiz/syscalls/index.html b/17/Lab/Software Stack/quiz/syscalls/index.html new file mode 100644 index 0000000000..62be63efc6 --- /dev/null +++ b/17/Lab/Software Stack/quiz/syscalls/index.html @@ -0,0 +1,16 @@ + + + + + +Syscalls | Operating Systems + + + + +
+
Skip to main content

Syscalls

Syscall ID​

Question Text​

What register stores the system call ID on x86_64?

Question Answers​

  • RIP

  • RSP

  • RAX
  • RDX

Feedback​

RAX is the register used for passing the syscall ID and the result code.

Syscall Tool​

Question Text​

What tool do we use to capture system calls?

Question Answers​

  • strace

  • make

  • gcc

  • ./exec

Feedback​

strace is used to trace system calls invoked by a running program.

Syscall Numbers​

Question Text​

What is the approximate number of system call numbers in Linux?

Question Answers​

  • 3

  • 30

  • 300
  • 3000

Feedback​

As show here, they're about 300 system calls in Linux.

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/static-dynamic/index.html b/17/Lab/Software Stack/static-dynamic/index.html new file mode 100644 index 0000000000..836da91c2f --- /dev/null +++ b/17/Lab/Software Stack/static-dynamic/index.html @@ -0,0 +1,20 @@ + + + + + +Statically-linked and Dynamically-linked Libraries | Operating Systems + + + + +
+
Skip to main content

Statically-linked and Dynamically-linked Libraries

Libraries can be statically-linked or dynamically-linked, creating statically-linked executables and dynamically-linked executables. +Typically, the executables found in modern operating systems are dynamically-linked, given their reduced size and ability to share libraries at runtime.

The support/static-dynamic/ folder stores the implementation of a simple "Hello, World!"-printing program that uses both static and dynamic linking of libraries. +Let's build and run the two executables:

student@os:~/.../lab/support/static-dynamic$ ls
hello.c  Makefile

student@os:~/.../lab/support/static-dynamic$ make
cc -Wall   -c -o hello.o hello.c
cc   hello.o   -o hello
cc -static -o hello_static hello.o

student@os:~/.../lab/support/static-dynamic$ ls -lh
total 852K
-rwxrwxr-x 1 razvan razvan 8.2K Aug  2 15:53 hello
-rw-rw-r-- 1 razvan razvan   76 Aug  2 15:51 hello.c
-rw-rw-r-- 1 razvan razvan 1.6K Aug  2 15:53 hello.o
-rwxrwxr-x 1 razvan razvan 827K Aug  2 15:53 hello_static
-rw-rw-r-- 1 razvan razvan  237 Aug  2 15:53 Makefile

student@os:~/.../lab/support/static-dynamic$ ./hello
Hello, World!

student@os:~/.../lab/support/static-dynamic$ ./hello_static
Hello, World!

The two executables (hello and hello_static) behave similarly, despite having vastly different sizes (8.2K vs. 827K - 100 times larger).

We use nm and ldd to catch differences between the two types of resulting executables:

student@os:~/.../lab/support/static-dynamic$ ldd hello
        linux-vdso.so.1 (0x00007ffc8d9b2000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f10d1d88000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f10d237b000)

student@os:~/.../lab/support/static-dynamic$ ldd hello_static
        not a dynamic executable

student@os:~/.../lab/support/static-dynamic$ nm hello | wc -l
33

student@os:~/.../lab/support/static-dynamic$ nm hello_static | wc -l
1674

The dynamic executable references the dynamically-linked libc library (/lib/x86_64-linux-gnu/libc.so.6), while the statically-linked executable has no references. +Also, given the statically-linked executable integrated entire parts of statically-linked libraries, there are many more symbols than in the case of a dynamically-linked executable (1674 vs. 33).

We can use strace to see that there are differences in the preparatory system calls for each type of executables. +For the dynamically-linked executable, the dynamically-linked library (/lib/x86_64-linux-gnu/libc.so.6) is opened during runtime:

student@os:~/.../lab/support/static-dynamic$ strace ./hello
execve("./hello", ["./hello"], 0x7ffc409c6640 /* 66 vars */) = 0
brk(NULL)                               = 0x55a72eda6000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=198014, ...}) = 0
mmap(NULL, 198014, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f3136a41000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240\35\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2030928, ...}) = 0
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f3136a3f000
mmap(NULL, 4131552, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f3136458000
mprotect(0x7f313663f000, 2097152, PROT_NONE) = 0
mmap(0x7f313683f000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e7000) = 0x7f313683f000
mmap(0x7f3136845000, 15072, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f3136845000
close(3)                                = 0
arch_prctl(ARCH_SET_FS, 0x7f3136a404c0) = 0
mprotect(0x7f313683f000, 16384, PROT_READ) = 0
mprotect(0x55a72d1bb000, 4096, PROT_READ) = 0
mprotect(0x7f3136a72000, 4096, PROT_READ) = 0
munmap(0x7f3136a41000, 198014)          = 0
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 18), ...}) = 0
brk(NULL)                               = 0x55a72eda6000
brk(0x55a72edc7000)                     = 0x55a72edc7000
write(1, "Hello, World!\n", 14Hello, World!
)         = 14
exit_group(0)                           = ?
+++ exited with 0 +++

student@os:~/.../lab/support/static-dynamic$ strace ./hello_static
execve("./hello_static", ["./hello_static"], 0x7ffc9fd45400 /* 66 vars */) = 0
brk(NULL)                               = 0xff8000
brk(0xff91c0)                           = 0xff91c0
arch_prctl(ARCH_SET_FS, 0xff8880)       = 0
uname({sysname="Linux", nodename="yggdrasil", ...}) = 0
readlink("/proc/self/exe", "/home/razvan/school/so/operating"..., 4096) = 116
brk(0x101a1c0)                          = 0x101a1c0
brk(0x101b000)                          = 0x101b000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
fstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 18), ...}) = 0
write(1, "Hello, World!\n", 14Hello, World!
)         = 14
exit_group(0)                           = ?
+++ exited with 0 +++

Similarly, we can investigate a system executable (/bin/ls) to see that indeed all referenced dynamically-linked libraries are opened (via the openat system call) at runtime:

student@os:~/.../lab/support/static-dynamic$ ldd $(which ls)
    linux-vdso.so.1 (0x00007ffc3bdf3000)
    libselinux.so.1 => /lib/x86_64-linux-gnu/libselinux.so.1 (0x00007f092bd88000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f092b997000)
    libpcre.so.3 => /lib/x86_64-linux-gnu/libpcre.so.3 (0x00007f092b726000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f092b522000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f092c1d2000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f092b303000)

student@os:~/.../lab/support/static-dynamic$ strace -e openat ls
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre.so.3", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
openat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
community  docs  _index.html  search.md
+++ exited with 0 +++

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/Software Stack/syscall-wrapper/index.html b/17/Lab/Software Stack/syscall-wrapper/index.html new file mode 100644 index 0000000000..2eb3f6acbf --- /dev/null +++ b/17/Lab/Software Stack/syscall-wrapper/index.html @@ -0,0 +1,25 @@ + + + + + +System Call Wrappers | Operating Systems + + + + +
+
Skip to main content

System Call Wrappers

The support/syscall-wrapper/ folder stores the implementation of a simple program written in C (main.c) that calls the write() and exit() functions. +The functions are defined in syscall.asm as wrappers around corresponding system calls. +Each function invokes the corresponding system call using the specific system call ID and the arguments provided for the function call.

The implementation of the two wrapper functions in syscall.asm is very simple, as the function arguments are passed in the same registers required by the system call. +This is because of the overlap of the first three registers for the x86_64 Linux function calling convention and the x86_64 Linux system call convention.

syscall.h contains the declaration of the two functions and is included in main.c. +This way, C programs can be written that make function calls that end up making system calls.

Let's build, run and trace system calls for the program:

student@os:~/.../lab/support/syscall-wrapper$ ls
main.c  Makefile  syscall.h  syscall.s

student@os:~/.../lab/support/syscall-wrapper$ make
gcc -c -o main.o main.c
nasm -f elf64 -o syscall.o syscall.s
cc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none  main.o syscall.o   -o main

student@os:~/.../lab/support/syscall-wrapper$ ls
main  main.c  main.o  Makefile  syscall.h  syscall.o  syscall.s

student@os:~/.../software-stack/lab/syscall-wrapper$ ./main
Hello, world!

student@os:~/.../lab/support/syscall-wrapper$ strace ./main
execve("./main", ["./main"], 0x7ffee60fb590 /* 63 vars */) = 0
write(1, "Hello, world!\n", 14Hello, world!
)         = 14
exit(0)                                 = ?
+++ exited with 0 +++

The trace is similar to the previous example, showing the write and exit system calls.

By creating system call wrappers as C functions, we are now relieved of the burden of writing assembly language code. +Of course, there has to be an initial implementation of wrapper functions written in assembly language; +but, after that, we can use C only.

Practice​

Update the files in the support/syscall-wrapper/ folder to make read system call available as a wrapper. +Make a call to the read system call to read data from standard input in a buffer. +Then call write() to print data from that buffer.

Note that the read system call returns the number of bytes read. +Use that as the argument to the subsequent write call that prints read data.

We can see that it's easier to have wrapper calls and write most of the code in C than in assembly language.

Quiz

+ + + + \ No newline at end of file diff --git a/17/Lab/index.html b/17/Lab/index.html new file mode 100644 index 0000000000..3b274c9961 --- /dev/null +++ b/17/Lab/index.html @@ -0,0 +1,16 @@ + + + + + +Lab | Operating Systems + + + + +
+
Skip to main content

Lab

+ + + + \ No newline at end of file diff --git a/17/Lab/lab-setup/index.html b/17/Lab/lab-setup/index.html new file mode 100644 index 0000000000..6701a3ffcd --- /dev/null +++ b/17/Lab/lab-setup/index.html @@ -0,0 +1,17 @@ + + + + + +Setting up the Lab Environment | Operating Systems + + + + +
+
Skip to main content

Setting up the Lab Environment

If you have already cloned the repository, make sure it is updated:

student@os:~$ cd operating-systems

student@os:~/operating-systems$ git pull --rebase

The command may fail if you have uncommitted changes. +If that is the case, stash your changes, retry, and pop the stash:

student@os:~/operating-systems$ git stash

student@os:~/operating-systems$ git pull --rebase

student@os:~/operating-systems$ git stash pop

If you haven't already cloned the repository, do so and then enter the repository:

student@os:~$ git clone https://github.com/cs-pub-ro/operating-systems

student@os:~$ cd operating-systems

Navigate to a chapter's lab directory:

student@os:~/operating-systems$ cd content/chapters/<chapter-name>/lab/

The possible options are: software-stack, data, compute, io and app-interact.

If you're using the OS-runner Docker container, you can use the following shortcuts:

go-ss - changes directory to Software Stack lab

go-data - changes directory to Data lab

go-compute - changes directory to Compute lab

go-io - changes directory to IO lab

go-appInt - changes directory to App Interaction lab

+ + + + \ No newline at end of file diff --git a/17/Lecture/Application-Interaction/index.html b/17/Lecture/Application-Interaction/index.html new file mode 100644 index 0000000000..7fe69dddfa --- /dev/null +++ b/17/Lecture/Application-Interaction/index.html @@ -0,0 +1,16 @@ + + + + + +Application-Interaction | Operating Systems + + + + +
+
Skip to main content

Application-Interaction


tip

Focus the slides and press F for fullscreen viewing.

+ + + + \ No newline at end of file diff --git a/17/Lecture/Compute/index.html b/17/Lecture/Compute/index.html new file mode 100644 index 0000000000..d199d464b8 --- /dev/null +++ b/17/Lecture/Compute/index.html @@ -0,0 +1,16 @@ + + + + + +Compute | Operating Systems + + + + +
+
Skip to main content

Compute


tip

Focus the slides and press F for fullscreen viewing.

+ + + + \ No newline at end of file diff --git a/17/Lecture/Data/index.html b/17/Lecture/Data/index.html new file mode 100644 index 0000000000..ae9afe9965 --- /dev/null +++ b/17/Lecture/Data/index.html @@ -0,0 +1,16 @@ + + + + + +Data | Operating Systems + + + + +
+
Skip to main content

Data


tip

Focus the slides and press F for fullscreen viewing.

+ + + + \ No newline at end of file diff --git a/17/Lecture/IO/index.html b/17/Lecture/IO/index.html new file mode 100644 index 0000000000..c1be85d8c1 --- /dev/null +++ b/17/Lecture/IO/index.html @@ -0,0 +1,16 @@ + + + + + +IO | Operating Systems + + + + +
+
Skip to main content

IO


tip

Focus the slides and press F for fullscreen viewing.

+ + + + \ No newline at end of file diff --git a/17/Lecture/Software-Stack/index.html b/17/Lecture/Software-Stack/index.html new file mode 100644 index 0000000000..26ea6cad1e --- /dev/null +++ b/17/Lecture/Software-Stack/index.html @@ -0,0 +1,16 @@ + + + + + +Software-Stack | Operating Systems + + + + +
+
Skip to main content

Software-Stack


tip

Focus the slides and press F for fullscreen viewing.

+ + + + \ No newline at end of file diff --git a/17/Lecture/index.html b/17/Lecture/index.html new file mode 100644 index 0000000000..f254e40a35 --- /dev/null +++ b/17/Lecture/index.html @@ -0,0 +1,16 @@ + + + + + +Lecture | Operating Systems + + + + +
+
Skip to main content

Lecture

+ + + + \ No newline at end of file diff --git a/17/assets/css/styles.17f22a47.css b/17/assets/css/styles.17f22a47.css new file mode 100644 index 0000000000..0544710002 --- /dev/null +++ b/17/assets/css/styles.17f22a47.css @@ -0,0 +1 @@ +.col,.container{padding:0 var(--ifm-spacing-horizontal);width:100%}.markdown>h2,.markdown>h3,.markdown>h4,.markdown>h5,.markdown>h6{margin-bottom:calc(var(--ifm-heading-vertical-rhythm-bottom)*var(--ifm-leading))}.markdown li,body{word-wrap:break-word}body,ol ol,ol ul,ul ol,ul ul{margin:0}pre,table{overflow:auto}blockquote,pre{margin:0 0 var(--ifm-spacing-vertical)}.breadcrumbs__link,.button{transition-timing-function:var(--ifm-transition-timing-default)}.button,code{vertical-align:middle}.button--outline.button--active,.button--outline:active,.button--outline:hover,:root{--ifm-button-color:var(--ifm-font-color-base-inverse)}.menu__link:hover,a{transition:color var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.navbar--dark,:root{--ifm-navbar-link-hover-color:var(--ifm-color-primary)}.menu,.navbar-sidebar{overflow-x:hidden}:root,html[data-theme=dark]{--ifm-color-emphasis-500:var(--ifm-color-gray-500)}.toggleButton_gllP,html{-webkit-tap-highlight-color:transparent}.clean-list,.containsTaskList_mC6p,.details_lb9f>summary,.dropdown__menu,.menu__list{list-style:none}:root{--ifm-color-scheme:light;--ifm-dark-value:10%;--ifm-darker-value:15%;--ifm-darkest-value:30%;--ifm-light-value:15%;--ifm-lighter-value:30%;--ifm-lightest-value:50%;--ifm-contrast-background-value:90%;--ifm-contrast-foreground-value:70%;--ifm-contrast-background-dark-value:70%;--ifm-contrast-foreground-dark-value:90%;--ifm-color-primary:#3578e5;--ifm-color-secondary:#ebedf0;--ifm-color-success:#00a400;--ifm-color-info:#54c7ec;--ifm-color-warning:#ffba00;--ifm-color-danger:#fa383e;--ifm-color-primary-dark:#306cce;--ifm-color-primary-darker:#2d66c3;--ifm-color-primary-darkest:#2554a0;--ifm-color-primary-light:#538ce9;--ifm-color-primary-lighter:#72a1ed;--ifm-color-primary-lightest:#9abcf2;--ifm-color-primary-contrast-background:#ebf2fc;--ifm-color-primary-contrast-foreground:#102445;--ifm-color-secondary-dark:#d4d5d8;--ifm-color-secondary-darker:#c8c9cc;--ifm-color-secondary-darkest:#a4a6a8;--ifm-color-secondary-light:#eef0f2;--ifm-color-secondary-lighter:#f1f2f5;--ifm-color-secondary-lightest:#f5f6f8;--ifm-color-secondary-contrast-background:#fdfdfe;--ifm-color-secondary-contrast-foreground:#474748;--ifm-color-success-dark:#009400;--ifm-color-success-darker:#008b00;--ifm-color-success-darkest:#007300;--ifm-color-success-light:#26b226;--ifm-color-success-lighter:#4dbf4d;--ifm-color-success-lightest:#80d280;--ifm-color-success-contrast-background:#e6f6e6;--ifm-color-success-contrast-foreground:#003100;--ifm-color-info-dark:#4cb3d4;--ifm-color-info-darker:#47a9c9;--ifm-color-info-darkest:#3b8ba5;--ifm-color-info-light:#6ecfef;--ifm-color-info-lighter:#87d8f2;--ifm-color-info-lightest:#aae3f6;--ifm-color-info-contrast-background:#eef9fd;--ifm-color-info-contrast-foreground:#193c47;--ifm-color-warning-dark:#e6a700;--ifm-color-warning-darker:#d99e00;--ifm-color-warning-darkest:#b38200;--ifm-color-warning-light:#ffc426;--ifm-color-warning-lighter:#ffcf4d;--ifm-color-warning-lightest:#ffdd80;--ifm-color-warning-contrast-background:#fff8e6;--ifm-color-warning-contrast-foreground:#4d3800;--ifm-color-danger-dark:#e13238;--ifm-color-danger-darker:#d53035;--ifm-color-danger-darkest:#af272b;--ifm-color-danger-light:#fb565b;--ifm-color-danger-lighter:#fb7478;--ifm-color-danger-lightest:#fd9c9f;--ifm-color-danger-contrast-background:#ffebec;--ifm-color-danger-contrast-foreground:#4b1113;--ifm-color-white:#fff;--ifm-color-black:#000;--ifm-color-gray-0:var(--ifm-color-white);--ifm-color-gray-100:#f5f6f7;--ifm-color-gray-200:#ebedf0;--ifm-color-gray-300:#dadde1;--ifm-color-gray-400:#ccd0d5;--ifm-color-gray-500:#bec3c9;--ifm-color-gray-600:#8d949e;--ifm-color-gray-700:#606770;--ifm-color-gray-800:#444950;--ifm-color-gray-900:#1c1e21;--ifm-color-gray-1000:var(--ifm-color-black);--ifm-color-emphasis-0:var(--ifm-color-gray-0);--ifm-color-emphasis-100:var(--ifm-color-gray-100);--ifm-color-emphasis-200:var(--ifm-color-gray-200);--ifm-color-emphasis-300:var(--ifm-color-gray-300);--ifm-color-emphasis-400:var(--ifm-color-gray-400);--ifm-color-emphasis-600:var(--ifm-color-gray-600);--ifm-color-emphasis-700:var(--ifm-color-gray-700);--ifm-color-emphasis-800:var(--ifm-color-gray-800);--ifm-color-emphasis-900:var(--ifm-color-gray-900);--ifm-color-emphasis-1000:var(--ifm-color-gray-1000);--ifm-color-content:var(--ifm-color-emphasis-900);--ifm-color-content-inverse:var(--ifm-color-emphasis-0);--ifm-color-content-secondary:#525860;--ifm-background-color:#0000;--ifm-background-surface-color:var(--ifm-color-content-inverse);--ifm-global-border-width:1px;--ifm-global-radius:0.4rem;--ifm-hover-overlay:#0000000d;--ifm-font-color-base:var(--ifm-color-content);--ifm-font-color-base-inverse:var(--ifm-color-content-inverse);--ifm-font-color-secondary:var(--ifm-color-content-secondary);--ifm-font-family-base:system-ui,-apple-system,Segoe UI,Roboto,Ubuntu,Cantarell,Noto Sans,sans-serif,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";--ifm-font-family-monospace:SFMono-Regular,Menlo,Monaco,Consolas,"Liberation Mono","Courier New",monospace;--ifm-font-size-base:100%;--ifm-font-weight-light:300;--ifm-font-weight-normal:400;--ifm-font-weight-semibold:500;--ifm-font-weight-bold:700;--ifm-font-weight-base:var(--ifm-font-weight-normal);--ifm-line-height-base:1.65;--ifm-global-spacing:1rem;--ifm-spacing-vertical:var(--ifm-global-spacing);--ifm-spacing-horizontal:var(--ifm-global-spacing);--ifm-transition-fast:200ms;--ifm-transition-slow:400ms;--ifm-transition-timing-default:cubic-bezier(0.08,0.52,0.52,1);--ifm-global-shadow-lw:0 1px 2px 0 #0000001a;--ifm-global-shadow-md:0 5px 40px #0003;--ifm-global-shadow-tl:0 12px 28px 0 #0003,0 2px 4px 0 #0000001a;--ifm-z-index-dropdown:100;--ifm-z-index-fixed:200;--ifm-z-index-overlay:400;--ifm-container-width:1140px;--ifm-container-width-xl:1320px;--ifm-code-background:#f6f7f8;--ifm-code-border-radius:var(--ifm-global-radius);--ifm-code-font-size:90%;--ifm-code-padding-horizontal:0.1rem;--ifm-code-padding-vertical:0.1rem;--ifm-pre-background:var(--ifm-code-background);--ifm-pre-border-radius:var(--ifm-code-border-radius);--ifm-pre-color:inherit;--ifm-pre-line-height:1.45;--ifm-pre-padding:1rem;--ifm-heading-color:inherit;--ifm-heading-margin-top:0;--ifm-heading-margin-bottom:var(--ifm-spacing-vertical);--ifm-heading-font-family:var(--ifm-font-family-base);--ifm-heading-font-weight:var(--ifm-font-weight-bold);--ifm-heading-line-height:1.25;--ifm-h1-font-size:2rem;--ifm-h2-font-size:1.5rem;--ifm-h3-font-size:1.25rem;--ifm-h4-font-size:1rem;--ifm-h5-font-size:0.875rem;--ifm-h6-font-size:0.85rem;--ifm-image-alignment-padding:1.25rem;--ifm-leading-desktop:1.25;--ifm-leading:calc(var(--ifm-leading-desktop)*1rem);--ifm-list-left-padding:2rem;--ifm-list-margin:1rem;--ifm-list-item-margin:0.25rem;--ifm-list-paragraph-margin:1rem;--ifm-table-cell-padding:0.75rem;--ifm-table-background:#0000;--ifm-table-stripe-background:#00000008;--ifm-table-border-width:1px;--ifm-table-border-color:var(--ifm-color-emphasis-300);--ifm-table-head-background:inherit;--ifm-table-head-color:inherit;--ifm-table-head-font-weight:var(--ifm-font-weight-bold);--ifm-table-cell-color:inherit;--ifm-link-color:var(--ifm-color-primary);--ifm-link-decoration:none;--ifm-link-hover-color:var(--ifm-link-color);--ifm-link-hover-decoration:underline;--ifm-paragraph-margin-bottom:var(--ifm-leading);--ifm-blockquote-font-size:var(--ifm-font-size-base);--ifm-blockquote-border-left-width:2px;--ifm-blockquote-padding-horizontal:var(--ifm-spacing-horizontal);--ifm-blockquote-padding-vertical:0;--ifm-blockquote-shadow:none;--ifm-blockquote-color:var(--ifm-color-emphasis-800);--ifm-blockquote-border-color:var(--ifm-color-emphasis-300);--ifm-hr-background-color:var(--ifm-color-emphasis-500);--ifm-hr-height:1px;--ifm-hr-margin-vertical:1.5rem;--ifm-scrollbar-size:7px;--ifm-scrollbar-track-background-color:#f1f1f1;--ifm-scrollbar-thumb-background-color:silver;--ifm-scrollbar-thumb-hover-background-color:#a7a7a7;--ifm-alert-background-color:inherit;--ifm-alert-border-color:inherit;--ifm-alert-border-radius:var(--ifm-global-radius);--ifm-alert-border-width:0px;--ifm-alert-border-left-width:5px;--ifm-alert-color:var(--ifm-font-color-base);--ifm-alert-padding-horizontal:var(--ifm-spacing-horizontal);--ifm-alert-padding-vertical:var(--ifm-spacing-vertical);--ifm-alert-shadow:var(--ifm-global-shadow-lw);--ifm-avatar-intro-margin:1rem;--ifm-avatar-intro-alignment:inherit;--ifm-avatar-photo-size:3rem;--ifm-badge-background-color:inherit;--ifm-badge-border-color:inherit;--ifm-badge-border-radius:var(--ifm-global-radius);--ifm-badge-border-width:var(--ifm-global-border-width);--ifm-badge-color:var(--ifm-color-white);--ifm-badge-padding-horizontal:calc(var(--ifm-spacing-horizontal)*0.5);--ifm-badge-padding-vertical:calc(var(--ifm-spacing-vertical)*0.25);--ifm-breadcrumb-border-radius:1.5rem;--ifm-breadcrumb-spacing:0.5rem;--ifm-breadcrumb-color-active:var(--ifm-color-primary);--ifm-breadcrumb-item-background-active:var(--ifm-hover-overlay);--ifm-breadcrumb-padding-horizontal:0.8rem;--ifm-breadcrumb-padding-vertical:0.4rem;--ifm-breadcrumb-size-multiplier:1;--ifm-breadcrumb-separator:url('data:image/svg+xml;utf8,');--ifm-breadcrumb-separator-filter:none;--ifm-breadcrumb-separator-size:0.5rem;--ifm-breadcrumb-separator-size-multiplier:1.25;--ifm-button-background-color:inherit;--ifm-button-border-color:var(--ifm-button-background-color);--ifm-button-border-width:var(--ifm-global-border-width);--ifm-button-font-weight:var(--ifm-font-weight-bold);--ifm-button-padding-horizontal:1.5rem;--ifm-button-padding-vertical:0.375rem;--ifm-button-size-multiplier:1;--ifm-button-transition-duration:var(--ifm-transition-fast);--ifm-button-border-radius:calc(var(--ifm-global-radius)*var(--ifm-button-size-multiplier));--ifm-button-group-spacing:2px;--ifm-card-background-color:var(--ifm-background-surface-color);--ifm-card-border-radius:calc(var(--ifm-global-radius)*2);--ifm-card-horizontal-spacing:var(--ifm-global-spacing);--ifm-card-vertical-spacing:var(--ifm-global-spacing);--ifm-toc-border-color:var(--ifm-color-emphasis-300);--ifm-toc-link-color:var(--ifm-color-content-secondary);--ifm-toc-padding-vertical:0.5rem;--ifm-toc-padding-horizontal:0.5rem;--ifm-dropdown-background-color:var(--ifm-background-surface-color);--ifm-dropdown-font-weight:var(--ifm-font-weight-semibold);--ifm-dropdown-link-color:var(--ifm-font-color-base);--ifm-dropdown-hover-background-color:var(--ifm-hover-overlay);--ifm-footer-background-color:var(--ifm-color-emphasis-100);--ifm-footer-color:inherit;--ifm-footer-link-color:var(--ifm-color-emphasis-700);--ifm-footer-link-hover-color:var(--ifm-color-primary);--ifm-footer-link-horizontal-spacing:0.5rem;--ifm-footer-padding-horizontal:calc(var(--ifm-spacing-horizontal)*2);--ifm-footer-padding-vertical:calc(var(--ifm-spacing-vertical)*2);--ifm-footer-title-color:inherit;--ifm-footer-logo-max-width:min(30rem,90vw);--ifm-hero-background-color:var(--ifm-background-surface-color);--ifm-hero-text-color:var(--ifm-color-emphasis-800);--ifm-menu-color:var(--ifm-color-emphasis-700);--ifm-menu-color-active:var(--ifm-color-primary);--ifm-menu-color-background-active:var(--ifm-hover-overlay);--ifm-menu-color-background-hover:var(--ifm-hover-overlay);--ifm-menu-link-padding-horizontal:0.75rem;--ifm-menu-link-padding-vertical:0.375rem;--ifm-menu-link-sublist-icon:url('data:image/svg+xml;utf8,');--ifm-menu-link-sublist-icon-filter:none;--ifm-navbar-background-color:var(--ifm-background-surface-color);--ifm-navbar-height:3.75rem;--ifm-navbar-item-padding-horizontal:0.75rem;--ifm-navbar-item-padding-vertical:0.25rem;--ifm-navbar-link-color:var(--ifm-font-color-base);--ifm-navbar-link-active-color:var(--ifm-link-color);--ifm-navbar-padding-horizontal:var(--ifm-spacing-horizontal);--ifm-navbar-padding-vertical:calc(var(--ifm-spacing-vertical)*0.5);--ifm-navbar-shadow:var(--ifm-global-shadow-lw);--ifm-navbar-search-input-background-color:var(--ifm-color-emphasis-200);--ifm-navbar-search-input-color:var(--ifm-color-emphasis-800);--ifm-navbar-search-input-placeholder-color:var(--ifm-color-emphasis-500);--ifm-navbar-search-input-icon:url('data:image/svg+xml;utf8,');--ifm-navbar-sidebar-width:83vw;--ifm-pagination-border-radius:var(--ifm-global-radius);--ifm-pagination-color-active:var(--ifm-color-primary);--ifm-pagination-font-size:1rem;--ifm-pagination-item-active-background:var(--ifm-hover-overlay);--ifm-pagination-page-spacing:0.2em;--ifm-pagination-padding-horizontal:calc(var(--ifm-spacing-horizontal)*1);--ifm-pagination-padding-vertical:calc(var(--ifm-spacing-vertical)*0.25);--ifm-pagination-nav-border-radius:var(--ifm-global-radius);--ifm-pagination-nav-color-hover:var(--ifm-color-primary);--ifm-pills-color-active:var(--ifm-color-primary);--ifm-pills-color-background-active:var(--ifm-hover-overlay);--ifm-pills-spacing:0.125rem;--ifm-tabs-color:var(--ifm-font-color-secondary);--ifm-tabs-color-active:var(--ifm-color-primary);--ifm-tabs-color-active-border:var(--ifm-tabs-color-active);--ifm-tabs-padding-horizontal:1rem;--ifm-tabs-padding-vertical:1rem;--docusaurus-progress-bar-color:var(--ifm-color-primary);--ifm-color-primary:#2e8555;--ifm-color-primary-dark:#29784c;--ifm-color-primary-darker:#277148;--ifm-color-primary-darkest:#205d3b;--ifm-color-primary-light:#33925d;--ifm-color-primary-lighter:#359962;--ifm-color-primary-lightest:#3cad6e;--ifm-code-font-size:95%;--docusaurus-highlighted-code-line-bg:#0000001a;--docusaurus-announcement-bar-height:auto;--docusaurus-tag-list-border:var(--ifm-color-emphasis-300);--docusaurus-collapse-button-bg:#0000;--docusaurus-collapse-button-bg-hover:#0000001a;--doc-sidebar-width:300px;--doc-sidebar-hidden-width:30px}.badge--danger,.badge--info,.badge--primary,.badge--secondary,.badge--success,.badge--warning{--ifm-badge-border-color:var(--ifm-badge-background-color)}.button--link,.button--outline{--ifm-button-background-color:#0000}*{box-sizing:border-box}html{-webkit-font-smoothing:antialiased;-webkit-text-size-adjust:100%;text-size-adjust:100%;background-color:var(--ifm-background-color);color:var(--ifm-font-color-base);color-scheme:var(--ifm-color-scheme);font:var(--ifm-font-size-base)/var(--ifm-line-height-base) var(--ifm-font-family-base);text-rendering:optimizelegibility}iframe{border:0;color-scheme:auto}.container{margin:0 auto;max-width:var(--ifm-container-width)}.container--fluid{max-width:inherit}.row{display:flex;flex-wrap:wrap;margin:0 calc(var(--ifm-spacing-horizontal)*-1)}.list_eTzJ article:last-child,.margin-bottom--none,.margin-vert--none,.markdown>:last-child{margin-bottom:0!important}.margin-top--none,.margin-vert--none{margin-top:0!important}.row--no-gutters{margin-left:0;margin-right:0}.margin-horiz--none,.margin-right--none{margin-right:0!important}.row--no-gutters>.col{padding-left:0;padding-right:0}.row--align-top{align-items:flex-start}.row--align-bottom{align-items:flex-end}.menuExternalLink_NmtK,.row--align-center{align-items:center}.row--align-stretch{align-items:stretch}.row--align-baseline{align-items:baseline}.col{--ifm-col-width:100%;flex:1 0;margin-left:0;max-width:var(--ifm-col-width)}.padding-bottom--none,.padding-vert--none{padding-bottom:0!important}.padding-top--none,.padding-vert--none{padding-top:0!important}.padding-horiz--none,.padding-left--none{padding-left:0!important}.padding-horiz--none,.padding-right--none{padding-right:0!important}.col[class*=col--]{flex:0 0 var(--ifm-col-width)}.col--1{--ifm-col-width:8.33333%}.col--offset-1{margin-left:8.33333%}.col--2{--ifm-col-width:16.66667%}.col--offset-2{margin-left:16.66667%}.col--3{--ifm-col-width:25%}.col--offset-3{margin-left:25%}.col--4{--ifm-col-width:33.33333%}.col--offset-4{margin-left:33.33333%}.col--5{--ifm-col-width:41.66667%}.col--offset-5{margin-left:41.66667%}.col--6{--ifm-col-width:50%}.col--offset-6{margin-left:50%}.col--7{--ifm-col-width:58.33333%}.col--offset-7{margin-left:58.33333%}.col--8{--ifm-col-width:66.66667%}.col--offset-8{margin-left:66.66667%}.col--9{--ifm-col-width:75%}.col--offset-9{margin-left:75%}.col--10{--ifm-col-width:83.33333%}.col--offset-10{margin-left:83.33333%}.col--11{--ifm-col-width:91.66667%}.col--offset-11{margin-left:91.66667%}.col--12{--ifm-col-width:100%}.col--offset-12{margin-left:100%}.margin-horiz--none,.margin-left--none{margin-left:0!important}.margin--none{margin:0!important}.margin-bottom--xs,.margin-vert--xs{margin-bottom:.25rem!important}.margin-top--xs,.margin-vert--xs{margin-top:.25rem!important}.margin-horiz--xs,.margin-left--xs{margin-left:.25rem!important}.margin-horiz--xs,.margin-right--xs{margin-right:.25rem!important}.margin--xs{margin:.25rem!important}.margin-bottom--sm,.margin-vert--sm{margin-bottom:.5rem!important}.margin-top--sm,.margin-vert--sm{margin-top:.5rem!important}.margin-horiz--sm,.margin-left--sm{margin-left:.5rem!important}.margin-horiz--sm,.margin-right--sm{margin-right:.5rem!important}.margin--sm{margin:.5rem!important}.margin-bottom--md,.margin-vert--md{margin-bottom:1rem!important}.margin-top--md,.margin-vert--md{margin-top:1rem!important}.margin-horiz--md,.margin-left--md{margin-left:1rem!important}.margin-horiz--md,.margin-right--md{margin-right:1rem!important}.margin--md{margin:1rem!important}.margin-bottom--lg,.margin-vert--lg{margin-bottom:2rem!important}.margin-top--lg,.margin-vert--lg{margin-top:2rem!important}.margin-horiz--lg,.margin-left--lg{margin-left:2rem!important}.margin-horiz--lg,.margin-right--lg{margin-right:2rem!important}.margin--lg{margin:2rem!important}.margin-bottom--xl,.margin-vert--xl{margin-bottom:5rem!important}.margin-top--xl,.margin-vert--xl{margin-top:5rem!important}.margin-horiz--xl,.margin-left--xl{margin-left:5rem!important}.margin-horiz--xl,.margin-right--xl{margin-right:5rem!important}.margin--xl{margin:5rem!important}.padding--none{padding:0!important}.padding-bottom--xs,.padding-vert--xs{padding-bottom:.25rem!important}.padding-top--xs,.padding-vert--xs{padding-top:.25rem!important}.padding-horiz--xs,.padding-left--xs{padding-left:.25rem!important}.padding-horiz--xs,.padding-right--xs{padding-right:.25rem!important}.padding--xs{padding:.25rem!important}.padding-bottom--sm,.padding-vert--sm{padding-bottom:.5rem!important}.padding-top--sm,.padding-vert--sm{padding-top:.5rem!important}.padding-horiz--sm,.padding-left--sm{padding-left:.5rem!important}.padding-horiz--sm,.padding-right--sm{padding-right:.5rem!important}.padding--sm{padding:.5rem!important}.padding-bottom--md,.padding-vert--md{padding-bottom:1rem!important}.padding-top--md,.padding-vert--md{padding-top:1rem!important}.padding-horiz--md,.padding-left--md{padding-left:1rem!important}.padding-horiz--md,.padding-right--md{padding-right:1rem!important}.padding--md{padding:1rem!important}.padding-bottom--lg,.padding-vert--lg{padding-bottom:2rem!important}.padding-top--lg,.padding-vert--lg{padding-top:2rem!important}.padding-horiz--lg,.padding-left--lg{padding-left:2rem!important}.padding-horiz--lg,.padding-right--lg{padding-right:2rem!important}.padding--lg{padding:2rem!important}.padding-bottom--xl,.padding-vert--xl{padding-bottom:5rem!important}.padding-top--xl,.padding-vert--xl{padding-top:5rem!important}.padding-horiz--xl,.padding-left--xl{padding-left:5rem!important}.padding-horiz--xl,.padding-right--xl{padding-right:5rem!important}.padding--xl{padding:5rem!important}code{background-color:var(--ifm-code-background);border:.1rem solid #0000001a;border-radius:var(--ifm-code-border-radius);font-family:var(--ifm-font-family-monospace);font-size:var(--ifm-code-font-size);padding:var(--ifm-code-padding-vertical) var(--ifm-code-padding-horizontal)}a code{color:inherit}pre{background-color:var(--ifm-pre-background);border-radius:var(--ifm-pre-border-radius);color:var(--ifm-pre-color);font:var(--ifm-code-font-size)/var(--ifm-pre-line-height) var(--ifm-font-family-monospace);padding:var(--ifm-pre-padding)}pre code{background-color:initial;border:none;font-size:100%;line-height:inherit;padding:0}kbd{background-color:var(--ifm-color-emphasis-0);border:1px solid var(--ifm-color-emphasis-400);border-radius:.2rem;box-shadow:inset 0 -1px 0 var(--ifm-color-emphasis-400);color:var(--ifm-color-emphasis-800);font:80% var(--ifm-font-family-monospace);padding:.15rem .3rem}h1,h2,h3,h4,h5,h6{color:var(--ifm-heading-color);font-family:var(--ifm-heading-font-family);font-weight:var(--ifm-heading-font-weight);line-height:var(--ifm-heading-line-height);margin:var(--ifm-heading-margin-top) 0 var(--ifm-heading-margin-bottom) 0}h1{font-size:var(--ifm-h1-font-size)}h2{font-size:var(--ifm-h2-font-size)}h3{font-size:var(--ifm-h3-font-size)}h4{font-size:var(--ifm-h4-font-size)}h5{font-size:var(--ifm-h5-font-size)}h6{font-size:var(--ifm-h6-font-size)}img{max-width:100%}img[align=right]{padding-left:var(--image-alignment-padding)}img[align=left]{padding-right:var(--image-alignment-padding)}.markdown{--ifm-h1-vertical-rhythm-top:3;--ifm-h2-vertical-rhythm-top:2;--ifm-h3-vertical-rhythm-top:1.5;--ifm-heading-vertical-rhythm-top:1.25;--ifm-h1-vertical-rhythm-bottom:1.25;--ifm-heading-vertical-rhythm-bottom:1}.markdown:after,.markdown:before{content:"";display:table}.markdown:after{clear:both}.markdown h1:first-child{--ifm-h1-font-size:3rem;margin-bottom:calc(var(--ifm-h1-vertical-rhythm-bottom)*var(--ifm-leading))}.markdown>h2{--ifm-h2-font-size:2rem;margin-top:calc(var(--ifm-h2-vertical-rhythm-top)*var(--ifm-leading))}.markdown>h3{--ifm-h3-font-size:1.5rem;margin-top:calc(var(--ifm-h3-vertical-rhythm-top)*var(--ifm-leading))}.markdown>h4,.markdown>h5,.markdown>h6{margin-top:calc(var(--ifm-heading-vertical-rhythm-top)*var(--ifm-leading))}.markdown>p,.markdown>pre,.markdown>ul{margin-bottom:var(--ifm-leading)}.markdown li>p{margin-top:var(--ifm-list-paragraph-margin)}.markdown li+li{margin-top:var(--ifm-list-item-margin)}ol,ul{margin:0 0 var(--ifm-list-margin);padding-left:var(--ifm-list-left-padding)}ol ol,ul ol{list-style-type:lower-roman}ol ol ol,ol ul ol,ul ol ol,ul ul ol{list-style-type:lower-alpha}table{border-collapse:collapse;display:block;margin-bottom:var(--ifm-spacing-vertical)}table thead tr{border-bottom:2px solid var(--ifm-table-border-color)}table thead,table tr:nth-child(2n){background-color:var(--ifm-table-stripe-background)}table tr{background-color:var(--ifm-table-background);border-top:var(--ifm-table-border-width) solid var(--ifm-table-border-color)}table td,table th{border:var(--ifm-table-border-width) solid var(--ifm-table-border-color);padding:var(--ifm-table-cell-padding)}table th{background-color:var(--ifm-table-head-background);color:var(--ifm-table-head-color);font-weight:var(--ifm-table-head-font-weight)}table td{color:var(--ifm-table-cell-color)}strong{font-weight:var(--ifm-font-weight-bold)}a{color:var(--ifm-link-color);text-decoration:var(--ifm-link-decoration)}a:hover{color:var(--ifm-link-hover-color);text-decoration:var(--ifm-link-hover-decoration)}.button:hover,.text--no-decoration,.text--no-decoration:hover,a:not([href]){text-decoration:none}p{margin:0 0 var(--ifm-paragraph-margin-bottom)}blockquote{border-left:var(--ifm-blockquote-border-left-width) solid var(--ifm-blockquote-border-color);box-shadow:var(--ifm-blockquote-shadow);color:var(--ifm-blockquote-color);font-size:var(--ifm-blockquote-font-size);padding:var(--ifm-blockquote-padding-vertical) var(--ifm-blockquote-padding-horizontal)}blockquote>:first-child{margin-top:0}blockquote>:last-child{margin-bottom:0}hr{background-color:var(--ifm-hr-background-color);border:0;height:var(--ifm-hr-height);margin:var(--ifm-hr-margin-vertical) 0}.shadow--lw{box-shadow:var(--ifm-global-shadow-lw)!important}.shadow--md{box-shadow:var(--ifm-global-shadow-md)!important}.shadow--tl{box-shadow:var(--ifm-global-shadow-tl)!important}.text--primary,.wordWrapButtonEnabled_EoeP .wordWrapButtonIcon_Bwma{color:var(--ifm-color-primary)}.text--secondary{color:var(--ifm-color-secondary)}.text--success{color:var(--ifm-color-success)}.text--info{color:var(--ifm-color-info)}.text--warning{color:var(--ifm-color-warning)}.text--danger{color:var(--ifm-color-danger)}.text--center{text-align:center}.text--left{text-align:left}.text--justify{text-align:justify}.text--right{text-align:right}.text--capitalize{text-transform:capitalize}.text--lowercase{text-transform:lowercase}.admonitionHeading_tbUL,.alert__heading,.text--uppercase{text-transform:uppercase}.text--light{font-weight:var(--ifm-font-weight-light)}.text--normal{font-weight:var(--ifm-font-weight-normal)}.text--semibold{font-weight:var(--ifm-font-weight-semibold)}.text--bold{font-weight:var(--ifm-font-weight-bold)}.text--italic{font-style:italic}.text--truncate{overflow:hidden;text-overflow:ellipsis;white-space:nowrap}.text--break{word-wrap:break-word!important;word-break:break-word!important}.clean-btn{background:none;border:none;color:inherit;cursor:pointer;font-family:inherit;padding:0}.alert,.alert .close{color:var(--ifm-alert-foreground-color)}.clean-list{padding-left:0}.alert--primary{--ifm-alert-background-color:var(--ifm-color-primary-contrast-background);--ifm-alert-background-color-highlight:#3578e526;--ifm-alert-foreground-color:var(--ifm-color-primary-contrast-foreground);--ifm-alert-border-color:var(--ifm-color-primary-dark)}.alert--secondary{--ifm-alert-background-color:var(--ifm-color-secondary-contrast-background);--ifm-alert-background-color-highlight:#ebedf026;--ifm-alert-foreground-color:var(--ifm-color-secondary-contrast-foreground);--ifm-alert-border-color:var(--ifm-color-secondary-dark)}.alert--success{--ifm-alert-background-color:var(--ifm-color-success-contrast-background);--ifm-alert-background-color-highlight:#00a40026;--ifm-alert-foreground-color:var(--ifm-color-success-contrast-foreground);--ifm-alert-border-color:var(--ifm-color-success-dark)}.alert--info{--ifm-alert-background-color:var(--ifm-color-info-contrast-background);--ifm-alert-background-color-highlight:#54c7ec26;--ifm-alert-foreground-color:var(--ifm-color-info-contrast-foreground);--ifm-alert-border-color:var(--ifm-color-info-dark)}.alert--warning{--ifm-alert-background-color:var(--ifm-color-warning-contrast-background);--ifm-alert-background-color-highlight:#ffba0026;--ifm-alert-foreground-color:var(--ifm-color-warning-contrast-foreground);--ifm-alert-border-color:var(--ifm-color-warning-dark)}.alert--danger{--ifm-alert-background-color:var(--ifm-color-danger-contrast-background);--ifm-alert-background-color-highlight:#fa383e26;--ifm-alert-foreground-color:var(--ifm-color-danger-contrast-foreground);--ifm-alert-border-color:var(--ifm-color-danger-dark)}.alert{--ifm-code-background:var(--ifm-alert-background-color-highlight);--ifm-link-color:var(--ifm-alert-foreground-color);--ifm-link-hover-color:var(--ifm-alert-foreground-color);--ifm-link-decoration:underline;--ifm-tabs-color:var(--ifm-alert-foreground-color);--ifm-tabs-color-active:var(--ifm-alert-foreground-color);--ifm-tabs-color-active-border:var(--ifm-alert-border-color);background-color:var(--ifm-alert-background-color);border:var(--ifm-alert-border-width) solid var(--ifm-alert-border-color);border-left-width:var(--ifm-alert-border-left-width);border-radius:var(--ifm-alert-border-radius);box-shadow:var(--ifm-alert-shadow);padding:var(--ifm-alert-padding-vertical) var(--ifm-alert-padding-horizontal)}.alert__heading{align-items:center;display:flex;font:700 var(--ifm-h5-font-size)/var(--ifm-heading-line-height) var(--ifm-heading-font-family);margin-bottom:.5rem}.alert__icon{display:inline-flex;margin-right:.4em}.alert__icon svg{fill:var(--ifm-alert-foreground-color);stroke:var(--ifm-alert-foreground-color);stroke-width:0}.alert .close{margin:calc(var(--ifm-alert-padding-vertical)*-1) calc(var(--ifm-alert-padding-horizontal)*-1) 0 0;opacity:.75}.alert .close:focus,.alert .close:hover{opacity:1}.alert a{text-decoration-color:var(--ifm-alert-border-color)}.alert a:hover{text-decoration-thickness:2px}.avatar{column-gap:var(--ifm-avatar-intro-margin);display:flex}.avatar__photo{border-radius:50%;display:block;height:var(--ifm-avatar-photo-size);overflow:hidden;width:var(--ifm-avatar-photo-size)}.card--full-height,.navbar__logo img,body,html{height:100%}.avatar__photo--sm{--ifm-avatar-photo-size:2rem}.avatar__photo--lg{--ifm-avatar-photo-size:4rem}.avatar__photo--xl{--ifm-avatar-photo-size:6rem}.avatar__intro{display:flex;flex:1 1;flex-direction:column;justify-content:center;text-align:var(--ifm-avatar-intro-alignment)}.badge,.breadcrumbs__item,.breadcrumbs__link,.button,.dropdown>.navbar__link:after{display:inline-block}.avatar__name{font:700 var(--ifm-h4-font-size)/var(--ifm-heading-line-height) var(--ifm-font-family-base)}.avatar__subtitle{margin-top:.25rem}.avatar--vertical{--ifm-avatar-intro-alignment:center;--ifm-avatar-intro-margin:0.5rem;align-items:center;flex-direction:column}.badge{background-color:var(--ifm-badge-background-color);border:var(--ifm-badge-border-width) solid var(--ifm-badge-border-color);border-radius:var(--ifm-badge-border-radius);color:var(--ifm-badge-color);font-size:75%;font-weight:var(--ifm-font-weight-bold);line-height:1;padding:var(--ifm-badge-padding-vertical) var(--ifm-badge-padding-horizontal)}.badge--primary{--ifm-badge-background-color:var(--ifm-color-primary)}.badge--secondary{--ifm-badge-background-color:var(--ifm-color-secondary);color:var(--ifm-color-black)}.breadcrumbs__link,.button.button--secondary.button--outline:not(.button--active):not(:hover){color:var(--ifm-font-color-base)}.badge--success{--ifm-badge-background-color:var(--ifm-color-success)}.badge--info{--ifm-badge-background-color:var(--ifm-color-info)}.badge--warning{--ifm-badge-background-color:var(--ifm-color-warning)}.badge--danger{--ifm-badge-background-color:var(--ifm-color-danger)}.breadcrumbs{margin-bottom:0;padding-left:0}.breadcrumbs__item:not(:last-child):after{background:var(--ifm-breadcrumb-separator) center;content:" ";display:inline-block;filter:var(--ifm-breadcrumb-separator-filter);height:calc(var(--ifm-breadcrumb-separator-size)*var(--ifm-breadcrumb-size-multiplier)*var(--ifm-breadcrumb-separator-size-multiplier));margin:0 var(--ifm-breadcrumb-spacing);opacity:.5;width:calc(var(--ifm-breadcrumb-separator-size)*var(--ifm-breadcrumb-size-multiplier)*var(--ifm-breadcrumb-separator-size-multiplier))}.breadcrumbs__item--active .breadcrumbs__link{background:var(--ifm-breadcrumb-item-background-active);color:var(--ifm-breadcrumb-color-active)}.breadcrumbs__link{border-radius:var(--ifm-breadcrumb-border-radius);font-size:calc(1rem*var(--ifm-breadcrumb-size-multiplier));padding:calc(var(--ifm-breadcrumb-padding-vertical)*var(--ifm-breadcrumb-size-multiplier)) calc(var(--ifm-breadcrumb-padding-horizontal)*var(--ifm-breadcrumb-size-multiplier));transition-duration:var(--ifm-transition-fast);transition-property:background,color}.breadcrumbs__link:any-link:hover,.breadcrumbs__link:link:hover,.breadcrumbs__link:visited:hover,area.breadcrumbs__link[href]:hover{background:var(--ifm-breadcrumb-item-background-active);text-decoration:none}.breadcrumbs--sm{--ifm-breadcrumb-size-multiplier:0.8}.breadcrumbs--lg{--ifm-breadcrumb-size-multiplier:1.2}.button{background-color:var(--ifm-button-background-color);border:var(--ifm-button-border-width) solid var(--ifm-button-border-color);border-radius:var(--ifm-button-border-radius);cursor:pointer;font-size:calc(.875rem*var(--ifm-button-size-multiplier));font-weight:var(--ifm-button-font-weight);line-height:1.5;padding:calc(var(--ifm-button-padding-vertical)*var(--ifm-button-size-multiplier)) calc(var(--ifm-button-padding-horizontal)*var(--ifm-button-size-multiplier));text-align:center;transition-duration:var(--ifm-button-transition-duration);transition-property:color,background,border-color;-webkit-user-select:none;user-select:none;white-space:nowrap}.button,.button:hover{color:var(--ifm-button-color)}.button--outline{--ifm-button-color:var(--ifm-button-border-color)}.button--outline:hover{--ifm-button-background-color:var(--ifm-button-border-color)}.button--link{--ifm-button-border-color:#0000;color:var(--ifm-link-color);text-decoration:var(--ifm-link-decoration)}.button--link.button--active,.button--link:active,.button--link:hover{color:var(--ifm-link-hover-color);text-decoration:var(--ifm-link-hover-decoration)}.button.disabled,.button:disabled,.button[disabled]{opacity:.65;pointer-events:none}.button--sm{--ifm-button-size-multiplier:0.8}.button--lg{--ifm-button-size-multiplier:1.35}.button--block{display:block;width:100%}.button.button--secondary{color:var(--ifm-color-gray-900)}:where(.button--primary){--ifm-button-background-color:var(--ifm-color-primary);--ifm-button-border-color:var(--ifm-color-primary)}:where(.button--primary):not(.button--outline):hover{--ifm-button-background-color:var(--ifm-color-primary-dark);--ifm-button-border-color:var(--ifm-color-primary-dark)}.button--primary.button--active,.button--primary:active{--ifm-button-background-color:var(--ifm-color-primary-darker);--ifm-button-border-color:var(--ifm-color-primary-darker)}:where(.button--secondary){--ifm-button-background-color:var(--ifm-color-secondary);--ifm-button-border-color:var(--ifm-color-secondary)}:where(.button--secondary):not(.button--outline):hover{--ifm-button-background-color:var(--ifm-color-secondary-dark);--ifm-button-border-color:var(--ifm-color-secondary-dark)}.button--secondary.button--active,.button--secondary:active{--ifm-button-background-color:var(--ifm-color-secondary-darker);--ifm-button-border-color:var(--ifm-color-secondary-darker)}:where(.button--success){--ifm-button-background-color:var(--ifm-color-success);--ifm-button-border-color:var(--ifm-color-success)}:where(.button--success):not(.button--outline):hover{--ifm-button-background-color:var(--ifm-color-success-dark);--ifm-button-border-color:var(--ifm-color-success-dark)}.button--success.button--active,.button--success:active{--ifm-button-background-color:var(--ifm-color-success-darker);--ifm-button-border-color:var(--ifm-color-success-darker)}:where(.button--info){--ifm-button-background-color:var(--ifm-color-info);--ifm-button-border-color:var(--ifm-color-info)}:where(.button--info):not(.button--outline):hover{--ifm-button-background-color:var(--ifm-color-info-dark);--ifm-button-border-color:var(--ifm-color-info-dark)}.button--info.button--active,.button--info:active{--ifm-button-background-color:var(--ifm-color-info-darker);--ifm-button-border-color:var(--ifm-color-info-darker)}:where(.button--warning){--ifm-button-background-color:var(--ifm-color-warning);--ifm-button-border-color:var(--ifm-color-warning)}:where(.button--warning):not(.button--outline):hover{--ifm-button-background-color:var(--ifm-color-warning-dark);--ifm-button-border-color:var(--ifm-color-warning-dark)}.button--warning.button--active,.button--warning:active{--ifm-button-background-color:var(--ifm-color-warning-darker);--ifm-button-border-color:var(--ifm-color-warning-darker)}:where(.button--danger){--ifm-button-background-color:var(--ifm-color-danger);--ifm-button-border-color:var(--ifm-color-danger)}:where(.button--danger):not(.button--outline):hover{--ifm-button-background-color:var(--ifm-color-danger-dark);--ifm-button-border-color:var(--ifm-color-danger-dark)}.button--danger.button--active,.button--danger:active{--ifm-button-background-color:var(--ifm-color-danger-darker);--ifm-button-border-color:var(--ifm-color-danger-darker)}.button-group{display:inline-flex;gap:var(--ifm-button-group-spacing)}.button-group>.button:not(:first-child){border-bottom-left-radius:0;border-top-left-radius:0}.button-group>.button:not(:last-child){border-bottom-right-radius:0;border-top-right-radius:0}.button-group--block{display:flex;justify-content:stretch}.button-group--block>.button{flex-grow:1}.card{background-color:var(--ifm-card-background-color);border-radius:var(--ifm-card-border-radius);box-shadow:var(--ifm-global-shadow-lw);display:flex;flex-direction:column;overflow:hidden}.card__image{padding-top:var(--ifm-card-vertical-spacing)}.card__image:first-child{padding-top:0}.card__body,.card__footer,.card__header{padding:var(--ifm-card-vertical-spacing) var(--ifm-card-horizontal-spacing)}.card__body:not(:last-child),.card__footer:not(:last-child),.card__header:not(:last-child){padding-bottom:0}.card__body>:last-child,.card__footer>:last-child,.card__header>:last-child{margin-bottom:0}.card__footer{margin-top:auto}.table-of-contents{font-size:.8rem;margin-bottom:0;padding:var(--ifm-toc-padding-vertical) 0}.table-of-contents,.table-of-contents ul{list-style:none;padding-left:var(--ifm-toc-padding-horizontal)}.table-of-contents li{margin:var(--ifm-toc-padding-vertical) var(--ifm-toc-padding-horizontal)}.table-of-contents__left-border{border-left:1px solid var(--ifm-toc-border-color)}.table-of-contents__link{color:var(--ifm-toc-link-color);display:block}.table-of-contents__link--active,.table-of-contents__link--active code,.table-of-contents__link:hover,.table-of-contents__link:hover code{color:var(--ifm-color-primary);text-decoration:none}.close{color:var(--ifm-color-black);float:right;font-size:1.5rem;font-weight:var(--ifm-font-weight-bold);line-height:1;opacity:.5;padding:1rem;transition:opacity var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.close:hover{opacity:.7}.close:focus,.theme-code-block-highlighted-line .codeLineNumber_Tfdd:before{opacity:.8}.dropdown{display:inline-flex;font-weight:var(--ifm-dropdown-font-weight);position:relative;vertical-align:top}.dropdown--hoverable:hover .dropdown__menu,.dropdown--show .dropdown__menu{opacity:1;pointer-events:all;transform:translateY(-1px);visibility:visible}#nprogress,.dropdown__menu,.navbar__item.dropdown .navbar__link:not([href]){pointer-events:none}.dropdown--right .dropdown__menu{left:inherit;right:0}.dropdown--nocaret .navbar__link:after{content:none!important}.dropdown__menu{background-color:var(--ifm-dropdown-background-color);border-radius:var(--ifm-global-radius);box-shadow:var(--ifm-global-shadow-md);left:0;max-height:80vh;min-width:10rem;opacity:0;overflow-y:auto;padding:.5rem;position:absolute;top:calc(100% - var(--ifm-navbar-item-padding-vertical) + .3rem);transform:translateY(-.625rem);transition-duration:var(--ifm-transition-fast);transition-property:opacity,transform,visibility;transition-timing-function:var(--ifm-transition-timing-default);visibility:hidden;z-index:var(--ifm-z-index-dropdown)}.menu__caret,.menu__link,.menu__list-item-collapsible{border-radius:.25rem;transition:background var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.dropdown__link{border-radius:.25rem;color:var(--ifm-dropdown-link-color);display:block;font-size:.875rem;margin-top:.2rem;padding:.25rem .5rem;white-space:nowrap}.dropdown__link--active,.dropdown__link:hover{background-color:var(--ifm-dropdown-hover-background-color);color:var(--ifm-dropdown-link-color);text-decoration:none}.dropdown__link--active,.dropdown__link--active:hover{--ifm-dropdown-link-color:var(--ifm-link-color)}.dropdown>.navbar__link:after{border-color:currentcolor #0000;border-style:solid;border-width:.4em .4em 0;content:"";margin-left:.3em;position:relative;top:2px;transform:translateY(-50%)}.footer{background-color:var(--ifm-footer-background-color);color:var(--ifm-footer-color);padding:var(--ifm-footer-padding-vertical) var(--ifm-footer-padding-horizontal)}.footer--dark{--ifm-footer-background-color:#303846;--ifm-footer-color:var(--ifm-footer-link-color);--ifm-footer-link-color:var(--ifm-color-secondary);--ifm-footer-title-color:var(--ifm-color-white)}.footer__links{margin-bottom:1rem}.footer__link-item{color:var(--ifm-footer-link-color);line-height:2}.footer__link-item:hover{color:var(--ifm-footer-link-hover-color)}.footer__link-separator{margin:0 var(--ifm-footer-link-horizontal-spacing)}.footer__logo{margin-top:1rem;max-width:var(--ifm-footer-logo-max-width)}.footer__title{color:var(--ifm-footer-title-color);font:700 var(--ifm-h4-font-size)/var(--ifm-heading-line-height) var(--ifm-font-family-base);margin-bottom:var(--ifm-heading-margin-bottom)}.menu,.navbar__link{font-weight:var(--ifm-font-weight-semibold)}.docItemContainer_Djhp article>:first-child,.docItemContainer_Djhp header+*,.footer__item{margin-top:0}.admonitionContent_S0QG>:last-child,.cardContainer_fWXF :last-child,.collapsibleContent_i85q>:last-child,.footer__items{margin-bottom:0}.codeBlockStandalone_MEMb,[type=checkbox]{padding:0}.hero{align-items:center;background-color:var(--ifm-hero-background-color);color:var(--ifm-hero-text-color);display:flex;padding:4rem 2rem}.hero--primary{--ifm-hero-background-color:var(--ifm-color-primary);--ifm-hero-text-color:var(--ifm-font-color-base-inverse)}.hero--dark{--ifm-hero-background-color:#303846;--ifm-hero-text-color:var(--ifm-color-white)}.hero__title{font-size:3rem}.hero__subtitle{font-size:1.5rem}.menu__list{margin:0;padding-left:0}.menu__caret,.menu__link{padding:var(--ifm-menu-link-padding-vertical) var(--ifm-menu-link-padding-horizontal)}.menu__list .menu__list{flex:0 0 100%;margin-top:.25rem;padding-left:var(--ifm-menu-link-padding-horizontal)}.menu__list-item:not(:first-child){margin-top:.25rem}.menu__list-item--collapsed .menu__list{height:0;overflow:hidden}.details_lb9f[data-collapsed=false].isBrowser_bmU9>summary:before,.details_lb9f[open]:not(.isBrowser_bmU9)>summary:before,.menu__list-item--collapsed .menu__caret:before,.menu__list-item--collapsed .menu__link--sublist:after{transform:rotate(90deg)}.menu__list-item-collapsible{display:flex;flex-wrap:wrap;position:relative}.menu__caret:hover,.menu__link:hover,.menu__list-item-collapsible--active,.menu__list-item-collapsible:hover{background:var(--ifm-menu-color-background-hover)}.menu__list-item-collapsible .menu__link--active,.menu__list-item-collapsible .menu__link:hover{background:none!important}.menu__caret,.menu__link{align-items:center;display:flex}.menu__link{color:var(--ifm-menu-color);flex:1;line-height:1.25}.menu__link:hover{color:var(--ifm-menu-color);text-decoration:none}.menu__caret:before,.menu__link--sublist-caret:after{height:1.25rem;transform:rotate(180deg);transition:transform var(--ifm-transition-fast) linear;width:1.25rem;filter:var(--ifm-menu-link-sublist-icon-filter);content:""}.menu__link--sublist-caret:after{background:var(--ifm-menu-link-sublist-icon) 50%/2rem 2rem;margin-left:auto;min-width:1.25rem}.menu__link--active,.menu__link--active:hover{color:var(--ifm-menu-color-active)}.navbar__brand,.navbar__link{color:var(--ifm-navbar-link-color)}.menu__link--active:not(.menu__link--sublist){background-color:var(--ifm-menu-color-background-active)}.menu__caret:before{background:var(--ifm-menu-link-sublist-icon) 50%/2rem 2rem}.navbar--dark,html[data-theme=dark]{--ifm-menu-link-sublist-icon-filter:invert(100%) sepia(94%) saturate(17%) hue-rotate(223deg) brightness(104%) contrast(98%)}.navbar{background-color:var(--ifm-navbar-background-color);box-shadow:var(--ifm-navbar-shadow);height:var(--ifm-navbar-height);padding:var(--ifm-navbar-padding-vertical) var(--ifm-navbar-padding-horizontal)}.docsWrapper_BCFX,.navbar,.navbar>.container,.navbar>.container-fluid{display:flex}.navbar--fixed-top{position:sticky;top:0;z-index:var(--ifm-z-index-fixed)}.navbar-sidebar,.navbar-sidebar__backdrop{bottom:0;opacity:0;position:fixed;transition-timing-function:ease-in-out;left:0;top:0;visibility:hidden}.announcementBar_mb4j,.skipToContent_fXgn{z-index:calc(var(--ifm-z-index-fixed) + 1)}.navbar__inner{display:flex;flex-wrap:wrap;justify-content:space-between;width:100%}.navbar__brand{align-items:center;display:flex;margin-right:1rem;min-width:0}.navbar__brand:hover{color:var(--ifm-navbar-link-hover-color);text-decoration:none}.announcementBarContent_xLdY,.navbar__title{flex:1 1 auto}.navbar__toggle{display:none;margin-right:.5rem}.navbar__logo{flex:0 0 auto;height:2rem;margin-right:.5rem}.navbar__items{align-items:center;display:flex;flex:1;min-width:0}.navbar__items--center{flex:0 0 auto}.navbar__items--center .navbar__brand{margin:0}.navbar__items--center+.navbar__items--right{flex:1}.navbar__items--right{flex:0 0 auto;justify-content:flex-end}.navbar__items--right>:last-child{padding-right:0}.navbar__item{display:inline-block;padding:var(--ifm-navbar-item-padding-vertical) var(--ifm-navbar-item-padding-horizontal)}.navbar__link--active,.navbar__link:hover{color:var(--ifm-navbar-link-hover-color);text-decoration:none}.navbar--dark,.navbar--primary{--ifm-menu-color:var(--ifm-color-gray-300);--ifm-navbar-link-color:var(--ifm-color-gray-100);--ifm-navbar-search-input-background-color:#ffffff1a;--ifm-navbar-search-input-placeholder-color:#ffffff80;color:var(--ifm-color-white)}.navbar--dark{--ifm-navbar-background-color:#242526;--ifm-menu-color-background-active:#ffffff0d;--ifm-navbar-search-input-color:var(--ifm-color-white)}.navbar--primary{--ifm-navbar-background-color:var(--ifm-color-primary);--ifm-navbar-link-hover-color:var(--ifm-color-white);--ifm-menu-color-active:var(--ifm-color-white);--ifm-navbar-search-input-color:var(--ifm-color-emphasis-500)}.navbar__search-input{-webkit-appearance:none;appearance:none;background:var(--ifm-navbar-search-input-background-color) var(--ifm-navbar-search-input-icon) no-repeat .75rem center/1rem 1rem;border:none;border-radius:2rem;color:var(--ifm-navbar-search-input-color);cursor:text;display:inline-block;font-size:.9rem;height:2rem;padding:0 .5rem 0 2.25rem;width:12.5rem}.navbar__search-input::placeholder{color:var(--ifm-navbar-search-input-placeholder-color)}.navbar-sidebar{background-color:var(--ifm-navbar-background-color);box-shadow:var(--ifm-global-shadow-md);transform:translate3d(-100%,0,0);transition-duration:.25s;transition-property:opacity,visibility,transform;width:var(--ifm-navbar-sidebar-width)}.navbar-sidebar--show .navbar-sidebar,.navbar-sidebar__items{transform:translateZ(0)}.navbar-sidebar--show .navbar-sidebar,.navbar-sidebar--show .navbar-sidebar__backdrop{opacity:1;visibility:visible}.navbar-sidebar__backdrop{background-color:#0009;right:0;transition-duration:.1s;transition-property:opacity,visibility}.navbar-sidebar__brand{align-items:center;box-shadow:var(--ifm-navbar-shadow);display:flex;flex:1;height:var(--ifm-navbar-height);padding:var(--ifm-navbar-padding-vertical) var(--ifm-navbar-padding-horizontal)}.navbar-sidebar__items{display:flex;height:calc(100% - var(--ifm-navbar-height));transition:transform var(--ifm-transition-fast) ease-in-out}.navbar-sidebar__items--show-secondary{transform:translate3d(calc((var(--ifm-navbar-sidebar-width))*-1),0,0)}.navbar-sidebar__item{flex-shrink:0;padding:.5rem;width:calc(var(--ifm-navbar-sidebar-width))}.navbar-sidebar__back{background:var(--ifm-menu-color-background-active);font-size:15px;font-weight:var(--ifm-button-font-weight);margin:0 0 .2rem -.5rem;padding:.6rem 1.5rem;position:relative;text-align:left;top:-.5rem;width:calc(100% + 1rem)}.navbar-sidebar__close{display:flex;margin-left:auto}.pagination{column-gap:var(--ifm-pagination-page-spacing);display:flex;font-size:var(--ifm-pagination-font-size);padding-left:0}.pagination--sm{--ifm-pagination-font-size:0.8rem;--ifm-pagination-padding-horizontal:0.8rem;--ifm-pagination-padding-vertical:0.2rem}.pagination--lg{--ifm-pagination-font-size:1.2rem;--ifm-pagination-padding-horizontal:1.2rem;--ifm-pagination-padding-vertical:0.3rem}.pagination__item{display:inline-flex}.pagination__item>span{padding:var(--ifm-pagination-padding-vertical)}.pagination__item--active .pagination__link{color:var(--ifm-pagination-color-active)}.pagination__item--active .pagination__link,.pagination__item:not(.pagination__item--active):hover .pagination__link{background:var(--ifm-pagination-item-active-background)}.pagination__item--disabled,.pagination__item[disabled]{opacity:.25;pointer-events:none}.pagination__link{border-radius:var(--ifm-pagination-border-radius);color:var(--ifm-font-color-base);display:inline-block;padding:var(--ifm-pagination-padding-vertical) var(--ifm-pagination-padding-horizontal);transition:background var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.pagination__link:hover{text-decoration:none}.pagination-nav{grid-gap:var(--ifm-spacing-horizontal);display:grid;gap:var(--ifm-spacing-horizontal);grid-template-columns:repeat(2,1fr)}.pagination-nav__link{border:1px solid var(--ifm-color-emphasis-300);border-radius:var(--ifm-pagination-nav-border-radius);display:block;height:100%;line-height:var(--ifm-heading-line-height);padding:var(--ifm-global-spacing);transition:border-color var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.pagination-nav__link:hover{border-color:var(--ifm-pagination-nav-color-hover);text-decoration:none}.pagination-nav__link--next{grid-column:2/3;text-align:right}.pagination-nav__label{font-size:var(--ifm-h4-font-size);font-weight:var(--ifm-heading-font-weight);word-break:break-word}.pagination-nav__link--prev .pagination-nav__label:before{content:"« "}.pagination-nav__link--next .pagination-nav__label:after{content:" »"}.pagination-nav__sublabel{color:var(--ifm-color-content-secondary);font-size:var(--ifm-h5-font-size);font-weight:var(--ifm-font-weight-semibold);margin-bottom:.25rem}.pills__item,.tabs{font-weight:var(--ifm-font-weight-bold)}.pills{display:flex;gap:var(--ifm-pills-spacing);padding-left:0}.pills__item{border-radius:.5rem;cursor:pointer;display:inline-block;padding:.25rem 1rem;transition:background var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.tabs,:not(.containsTaskList_mC6p>li)>.containsTaskList_mC6p{padding-left:0}.pills__item--active{color:var(--ifm-pills-color-active)}.pills__item--active,.pills__item:not(.pills__item--active):hover{background:var(--ifm-pills-color-background-active)}.pills--block{justify-content:stretch}.pills--block .pills__item{flex-grow:1;text-align:center}.tabs{color:var(--ifm-tabs-color);display:flex;margin-bottom:0;overflow-x:auto}.tabs__item{border-bottom:3px solid #0000;border-radius:var(--ifm-global-radius);cursor:pointer;display:inline-flex;padding:var(--ifm-tabs-padding-vertical) var(--ifm-tabs-padding-horizontal);transition:background-color var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.tabs__item--active{border-bottom-color:var(--ifm-tabs-color-active-border);border-bottom-left-radius:0;border-bottom-right-radius:0;color:var(--ifm-tabs-color-active)}.tabs__item:hover{background-color:var(--ifm-hover-overlay)}.tabs--block{justify-content:stretch}.tabs--block .tabs__item{flex-grow:1;justify-content:center}html[data-theme=dark]{--ifm-color-scheme:dark;--ifm-color-emphasis-0:var(--ifm-color-gray-1000);--ifm-color-emphasis-100:var(--ifm-color-gray-900);--ifm-color-emphasis-200:var(--ifm-color-gray-800);--ifm-color-emphasis-300:var(--ifm-color-gray-700);--ifm-color-emphasis-400:var(--ifm-color-gray-600);--ifm-color-emphasis-600:var(--ifm-color-gray-400);--ifm-color-emphasis-700:var(--ifm-color-gray-300);--ifm-color-emphasis-800:var(--ifm-color-gray-200);--ifm-color-emphasis-900:var(--ifm-color-gray-100);--ifm-color-emphasis-1000:var(--ifm-color-gray-0);--ifm-background-color:#1b1b1d;--ifm-background-surface-color:#242526;--ifm-hover-overlay:#ffffff0d;--ifm-color-content:#e3e3e3;--ifm-color-content-secondary:#fff;--ifm-breadcrumb-separator-filter:invert(64%) sepia(11%) saturate(0%) hue-rotate(149deg) brightness(99%) contrast(95%);--ifm-code-background:#ffffff1a;--ifm-scrollbar-track-background-color:#444;--ifm-scrollbar-thumb-background-color:#686868;--ifm-scrollbar-thumb-hover-background-color:#7a7a7a;--ifm-table-stripe-background:#ffffff12;--ifm-toc-border-color:var(--ifm-color-emphasis-200);--ifm-color-primary-contrast-background:#102445;--ifm-color-primary-contrast-foreground:#ebf2fc;--ifm-color-secondary-contrast-background:#474748;--ifm-color-secondary-contrast-foreground:#fdfdfe;--ifm-color-success-contrast-background:#003100;--ifm-color-success-contrast-foreground:#e6f6e6;--ifm-color-info-contrast-background:#193c47;--ifm-color-info-contrast-foreground:#eef9fd;--ifm-color-warning-contrast-background:#4d3800;--ifm-color-warning-contrast-foreground:#fff8e6;--ifm-color-danger-contrast-background:#4b1113;--ifm-color-danger-contrast-foreground:#ffebec}#nprogress .bar{background:var(--docusaurus-progress-bar-color);height:2px;left:0;position:fixed;top:0;width:100%;z-index:1031}#nprogress .peg{box-shadow:0 0 10px var(--docusaurus-progress-bar-color),0 0 5px var(--docusaurus-progress-bar-color);height:100%;opacity:1;position:absolute;right:0;transform:rotate(3deg) translateY(-4px);width:100px}[data-theme=dark]{--ifm-color-primary:#25c2a0;--ifm-color-primary-dark:#21af90;--ifm-color-primary-darker:#1fa588;--ifm-color-primary-darkest:#1a8870;--ifm-color-primary-light:#29d5b0;--ifm-color-primary-lighter:#32d8b4;--ifm-color-primary-lightest:#4fddbf;--docusaurus-highlighted-code-line-bg:#0000004d}body:not(.navigation-with-keyboard) :not(input):focus{outline:0}#docusaurus-base-url-issue-banner-container,.collapseSidebarButton_PEFL,.docSidebarContainer_b6E3,.sidebarLogo_isFc,.themedImage_ToTc,[data-theme=dark] .lightToggleIcon_pyhR,[data-theme=light] .darkToggleIcon_wfgR,html[data-announcement-bar-initially-dismissed=true] .announcementBar_mb4j{display:none}.skipToContent_fXgn{background-color:var(--ifm-background-surface-color);color:var(--ifm-color-emphasis-900);left:100%;padding:calc(var(--ifm-global-spacing)/2) var(--ifm-global-spacing);position:fixed;top:1rem}.skipToContent_fXgn:focus{box-shadow:var(--ifm-global-shadow-md);left:1rem}.closeButton_CVFx{line-height:0;padding:0}.content_knG7{font-size:85%;padding:5px 0;text-align:center}.content_knG7 a{color:inherit;text-decoration:underline}.announcementBar_mb4j{align-items:center;background-color:var(--ifm-color-white);box-shadow:var(--ifm-global-shadow-lw);color:var(--ifm-color-black);display:flex;height:var(--docusaurus-announcement-bar-height)}.announcementBarPlaceholder_vyr4{flex:0 0 10px}.announcementBarClose_gvF7{align-self:stretch;flex:0 0 30px}.toggle_vylO{height:2rem;width:2rem}.toggleButton_gllP{align-items:center;border-radius:50%;display:flex;height:100%;justify-content:center;transition:background var(--ifm-transition-fast);width:100%}.toggleButton_gllP:hover{background:var(--ifm-color-emphasis-200)}.toggleButtonDisabled_aARS{cursor:not-allowed}[data-theme=dark] .themedImage--dark_i4oU,[data-theme=light] .themedImage--light_HNdA{display:initial}.iconExternalLink_nPIU{margin-left:.3rem}.iconLanguage_nlXk{margin-right:5px;vertical-align:text-bottom}.navbarHideable_m1mJ{transition:transform var(--ifm-transition-fast) ease}.navbarHidden_jGov{transform:translate3d(0,calc(-100% - 2px),0)}.footerLogoLink_BH7S{opacity:.5;transition:opacity var(--ifm-transition-fast) var(--ifm-transition-timing-default)}.footerLogoLink_BH7S:hover,.hash-link:focus,:hover>.hash-link{opacity:1}.mainWrapper_z2l0{flex:1 0 auto}.docusaurus-mt-lg{margin-top:3rem}#__docusaurus{display:flex;flex-direction:column;min-height:100%}.cardContainer_fWXF{--ifm-link-color:var(--ifm-color-emphasis-800);--ifm-link-hover-color:var(--ifm-color-emphasis-700);--ifm-link-hover-decoration:none;border:1px solid var(--ifm-color-emphasis-200);box-shadow:0 1.5px 3px 0 #00000026;transition:all var(--ifm-transition-fast) ease;transition-property:border,box-shadow}.cardContainer_fWXF:hover{border-color:var(--ifm-color-primary);box-shadow:0 3px 6px 0 #0003}.cardTitle_rnsV{font-size:1.2rem}.cardDescription_PWke{font-size:.8rem}.iconEdit_Z9Sw{margin-right:.3em;vertical-align:sub}.tag_zVej{border:1px solid var(--docusaurus-tag-list-border);transition:border var(--ifm-transition-fast)}.tag_zVej:hover{--docusaurus-tag-list-border:var(--ifm-link-color);text-decoration:none}.tagRegular_sFm0{border-radius:.5rem;font-size:90%;padding:.2rem .5rem .3rem}.tagWithCount_h2kH{align-items:center;border-left:0;display:flex;padding:0 .5rem 0 1rem;position:relative}.tagWithCount_h2kH:after,.tagWithCount_h2kH:before{border:1px solid var(--docusaurus-tag-list-border);content:"";position:absolute;top:50%;transition:inherit}.tagWithCount_h2kH:before{border-bottom:0;border-right:0;height:1.18rem;right:100%;transform:translate(50%,-50%) rotate(-45deg);width:1.18rem}.tagWithCount_h2kH:after{border-radius:50%;height:.5rem;left:0;transform:translateY(-50%);width:.5rem}.tagWithCount_h2kH span{background:var(--ifm-color-secondary);border-radius:var(--ifm-global-radius);color:var(--ifm-color-black);font-size:.7rem;line-height:1.2;margin-left:.3rem;padding:.1rem .4rem}.tags_jXut{display:inline}.tag_QGVx{display:inline-block;margin:0 .4rem .5rem 0}.lastUpdated_vwxv{font-size:smaller;font-style:italic;margin-top:.2rem}.tocCollapsibleButton_TO0P{align-items:center;display:flex;font-size:inherit;justify-content:space-between;padding:.4rem .8rem;width:100%}.tocCollapsibleButton_TO0P:after{background:var(--ifm-menu-link-sublist-icon) 50% 50%/2rem 2rem no-repeat;content:"";filter:var(--ifm-menu-link-sublist-icon-filter);height:1.25rem;transform:rotate(180deg);transition:transform var(--ifm-transition-fast);width:1.25rem}.tocCollapsibleButtonExpanded_MG3E:after,.tocCollapsibleExpanded_sAul{transform:none}.tocCollapsible_ETCw{background-color:var(--ifm-menu-color-background-active);border-radius:var(--ifm-global-radius);margin:1rem 0}.tocCollapsibleContent_vkbj>ul{border-left:none;border-top:1px solid var(--ifm-color-emphasis-300);font-size:15px;padding:.2rem 0}.tocCollapsibleContent_vkbj ul li{margin:.4rem .8rem}.tocCollapsibleContent_vkbj a{display:block}.backToTopButton_sjWU{background-color:var(--ifm-color-emphasis-200);border-radius:50%;bottom:1.3rem;box-shadow:var(--ifm-global-shadow-lw);height:3rem;opacity:0;position:fixed;right:1.3rem;transform:scale(0);transition:all var(--ifm-transition-fast) var(--ifm-transition-timing-default);visibility:hidden;width:3rem;z-index:calc(var(--ifm-z-index-fixed) - 1)}.buttonGroup__atx button,.codeBlockContainer_Ckt0{background:var(--prism-background-color);color:var(--prism-color)}.backToTopButton_sjWU:after{background-color:var(--ifm-color-emphasis-1000);content:" ";display:inline-block;height:100%;-webkit-mask:var(--ifm-menu-link-sublist-icon) 50%/2rem 2rem no-repeat;mask:var(--ifm-menu-link-sublist-icon) 50%/2rem 2rem no-repeat;width:100%}.backToTopButtonShow_xfvO{opacity:1;transform:scale(1);visibility:visible}[data-theme=dark]:root{--docusaurus-collapse-button-bg:#ffffff0d;--docusaurus-collapse-button-bg-hover:#ffffff1a}.docMainContainer_gTbr,.docPage__5DB{display:flex;width:100%}.codeBlockContainer_Ckt0{border-radius:var(--ifm-code-border-radius);box-shadow:var(--ifm-global-shadow-lw);margin-bottom:var(--ifm-leading)}.codeBlockContent_biex{border-radius:inherit;direction:ltr;position:relative}.codeBlockTitle_Ktv7{border-bottom:1px solid var(--ifm-color-emphasis-300);border-top-left-radius:inherit;border-top-right-radius:inherit;font-size:var(--ifm-code-font-size);font-weight:500;padding:.75rem var(--ifm-pre-padding)}.codeBlock_bY9V{--ifm-pre-background:var(--prism-background-color);margin:0;padding:0}.codeBlockTitle_Ktv7+.codeBlockContent_biex .codeBlock_bY9V{border-top-left-radius:0;border-top-right-radius:0}.codeBlockLines_e6Vv{float:left;font:inherit;min-width:100%;padding:var(--ifm-pre-padding)}.codeBlockLinesWithNumbering_o6Pm{display:table;padding:var(--ifm-pre-padding) 0}.buttonGroup__atx{column-gap:.2rem;display:flex;position:absolute;right:calc(var(--ifm-pre-padding)/2);top:calc(var(--ifm-pre-padding)/2)}.buttonGroup__atx button{align-items:center;border:1px solid var(--ifm-color-emphasis-300);border-radius:var(--ifm-global-radius);display:flex;line-height:0;opacity:0;padding:.4rem;transition:opacity .2s ease-in-out}.buttonGroup__atx button:focus-visible,.buttonGroup__atx button:hover{opacity:1!important}.theme-code-block:hover .buttonGroup__atx button{opacity:.4}:where(:root){--docusaurus-highlighted-code-line-bg:#484d5b}:where([data-theme=dark]){--docusaurus-highlighted-code-line-bg:#646464}.theme-code-block-highlighted-line{background-color:var(--docusaurus-highlighted-code-line-bg);display:block;margin:0 calc(var(--ifm-pre-padding)*-1);padding:0 var(--ifm-pre-padding)}.codeLine_lJS_{counter-increment:a;display:table-row}.codeLineNumber_Tfdd{background:var(--ifm-pre-background);display:table-cell;left:0;overflow-wrap:normal;padding:0 var(--ifm-pre-padding);position:sticky;text-align:right;width:1%}.codeLineNumber_Tfdd:before{content:counter(a);opacity:.4}.codeLineContent_feaV{padding-right:var(--ifm-pre-padding)}.theme-code-block:hover .copyButtonCopied_obH4{opacity:1!important}.copyButtonIcons_eSgA{height:1.125rem;position:relative;width:1.125rem}.copyButtonIcon_y97N,.copyButtonSuccessIcon_LjdS{fill:currentColor;height:inherit;left:0;opacity:inherit;position:absolute;top:0;transition:.15s;width:inherit}.copyButtonSuccessIcon_LjdS{color:#00d600;left:50%;opacity:0;top:50%;transform:translate(-50%,-50%) scale(.33)}.copyButtonCopied_obH4 .copyButtonIcon_y97N{opacity:0;transform:scale(.33)}.copyButtonCopied_obH4 .copyButtonSuccessIcon_LjdS{opacity:1;transform:translate(-50%,-50%) scale(1);transition-delay:75ms}.wordWrapButtonIcon_Bwma{height:1.2rem;width:1.2rem}.details_lb9f{--docusaurus-details-summary-arrow-size:0.38rem;--docusaurus-details-transition:transform 200ms ease;--docusaurus-details-decoration-color:grey}.details_lb9f>summary{cursor:pointer;padding-left:1rem;position:relative}.details_lb9f>summary::-webkit-details-marker{display:none}.details_lb9f>summary:before{border-color:#0000 #0000 #0000 var(--docusaurus-details-decoration-color);border-style:solid;border-width:var(--docusaurus-details-summary-arrow-size);content:"";left:0;position:absolute;top:.45rem;transform:rotate(0);transform-origin:calc(var(--docusaurus-details-summary-arrow-size)/2) 50%;transition:var(--docusaurus-details-transition)}.collapsibleContent_i85q{border-top:1px solid var(--docusaurus-details-decoration-color);margin-top:1rem;padding-top:1rem}.details_b_Ee{--docusaurus-details-decoration-color:var(--ifm-alert-border-color);--docusaurus-details-transition:transform var(--ifm-transition-fast) ease;border:1px solid var(--ifm-alert-border-color);margin:0 0 var(--ifm-spacing-vertical)}.anchorWithStickyNavbar_LWe7{scroll-margin-top:calc(var(--ifm-navbar-height) + .5rem)}.anchorWithHideOnScrollNavbar_WYt5{scroll-margin-top:.5rem}.hash-link{opacity:0;padding-left:.5rem;transition:opacity var(--ifm-transition-fast);-webkit-user-select:none;user-select:none}.hash-link:before{content:"#"}.img_ev3q{height:auto}.tableOfContents_bqdL{max-height:calc(100vh - var(--ifm-navbar-height) - 2rem);overflow-y:auto;position:sticky;top:calc(var(--ifm-navbar-height) + 1rem)}.admonition_LlT9{margin-bottom:1em}.admonitionHeading_tbUL{font:var(--ifm-heading-font-weight) var(--ifm-h5-font-size)/var(--ifm-heading-line-height) var(--ifm-heading-font-family);margin-bottom:.3rem}.admonitionHeading_tbUL code{text-transform:none}.admonitionIcon_kALy{display:inline-block;margin-right:.4em;vertical-align:middle}.admonitionIcon_kALy svg{fill:var(--ifm-alert-foreground-color);display:inline-block;height:1.6em;width:1.6em}.breadcrumbsContainer_Z_bl{--ifm-breadcrumb-size-multiplier:0.8;margin-bottom:.8rem}.breadcrumbHomeIcon_OVgt{height:1.1rem;position:relative;top:1px;vertical-align:top;width:1.1rem}.title_kItE{--ifm-h1-font-size:3rem;margin-bottom:calc(var(--ifm-leading)*1.25)}.mdxPageWrapper_j9I6{justify-content:center}@media (min-width:997px){.collapseSidebarButton_PEFL,.expandButton_m80_{background-color:var(--docusaurus-collapse-button-bg);position:sticky}:root{--docusaurus-announcement-bar-height:30px}.announcementBarClose_gvF7,.announcementBarPlaceholder_vyr4{flex-basis:50px}.searchBox_ZlJk{padding:var(--ifm-navbar-item-padding-vertical) var(--ifm-navbar-item-padding-horizontal)}.lastUpdated_vwxv{text-align:right}.tocMobile_ITEo{display:none}.collapseSidebarButton_PEFL{border:1px solid var(--ifm-toc-border-color);border-radius:0;bottom:0;display:block!important;height:40px}.collapseSidebarButtonIcon_kv0_{margin-top:4px;transform:rotate(180deg)}.expandButtonIcon_BlDH,[dir=rtl] .collapseSidebarButtonIcon_kv0_{transform:rotate(0)}.collapseSidebarButton_PEFL:focus,.collapseSidebarButton_PEFL:hover,.expandButton_m80_:focus,.expandButton_m80_:hover{background-color:var(--docusaurus-collapse-button-bg-hover)}.menuHtmlItem_M9Kj{padding:var(--ifm-menu-link-padding-vertical) var(--ifm-menu-link-padding-horizontal)}.menu_SIkG{flex-grow:1;padding:.5rem}@supports (scrollbar-gutter:stable){.menu_SIkG{padding:.5rem 0 .5rem .5rem;scrollbar-gutter:stable}}.menuWithAnnouncementBar_GW3s{margin-bottom:var(--docusaurus-announcement-bar-height)}.sidebar_njMd{display:flex;flex-direction:column;height:100%;max-height:100vh;padding-top:var(--ifm-navbar-height);position:sticky;top:0;transition:opacity 50ms;width:var(--doc-sidebar-width)}.sidebarWithHideableNavbar_wUlq{padding-top:0}.sidebarHidden_VK0M{height:0;opacity:0;overflow:hidden;visibility:hidden}.sidebarLogo_isFc{align-items:center;color:inherit!important;display:flex!important;margin:0 var(--ifm-navbar-padding-horizontal);max-height:var(--ifm-navbar-height);min-height:var(--ifm-navbar-height);text-decoration:none!important}.sidebarLogo_isFc img{height:2rem;margin-right:.5rem}.expandButton_m80_{align-items:center;display:flex;height:100%;justify-content:center;max-height:100vh;top:0;transition:background-color var(--ifm-transition-fast) ease}[dir=rtl] .expandButtonIcon_BlDH{transform:rotate(180deg)}.docSidebarContainer_b6E3{border-right:1px solid var(--ifm-toc-border-color);-webkit-clip-path:inset(0);clip-path:inset(0);display:block;margin-top:calc(var(--ifm-navbar-height)*-1);transition:width var(--ifm-transition-fast) ease;width:var(--doc-sidebar-width);will-change:width}.docSidebarContainerHidden_b3ry{cursor:pointer;width:var(--doc-sidebar-hidden-width)}.docMainContainer_gTbr{flex-grow:1;max-width:calc(100% - var(--doc-sidebar-width))}.docMainContainerEnhanced_Uz_u{max-width:calc(100% - var(--doc-sidebar-hidden-width))}.docItemWrapperEnhanced_czyv{max-width:calc(var(--ifm-container-width) + var(--doc-sidebar-width))!important}.docItemCol_VOVn,.generatedIndexPage_vN6x{max-width:75%!important}.list_eTzJ article:nth-last-child(-n+2){margin-bottom:0!important}}@media (min-width:1440px){.container{max-width:var(--ifm-container-width-xl)}}@media (max-width:996px){.col{--ifm-col-width:100%;flex-basis:var(--ifm-col-width);margin-left:0}.footer{--ifm-footer-padding-horizontal:0}.colorModeToggle_DEke,.footer__link-separator,.navbar__item,.tableOfContents_bqdL{display:none}.footer__col{margin-bottom:calc(var(--ifm-spacing-vertical)*3)}.footer__link-item{display:block}.hero{padding-left:0;padding-right:0}.navbar>.container,.navbar>.container-fluid{padding:0}.navbar__toggle{display:inherit}.navbar__search-input{width:9rem}.pills--block,.tabs--block{flex-direction:column}.searchBox_ZlJk{position:absolute;right:var(--ifm-navbar-padding-horizontal)}.docItemContainer_F8PC{padding:0 .3rem}}@media (max-width:576px){.markdown h1:first-child{--ifm-h1-font-size:2rem}.markdown>h2{--ifm-h2-font-size:1.5rem}.markdown>h3{--ifm-h3-font-size:1.25rem}}@media (hover:hover){.backToTopButton_sjWU:hover{background-color:var(--ifm-color-emphasis-300)}}@media (pointer:fine){.thin-scrollbar{scrollbar-width:thin}.thin-scrollbar::-webkit-scrollbar{height:var(--ifm-scrollbar-size);width:var(--ifm-scrollbar-size)}.thin-scrollbar::-webkit-scrollbar-track{background:var(--ifm-scrollbar-track-background-color);border-radius:10px}.thin-scrollbar::-webkit-scrollbar-thumb{background:var(--ifm-scrollbar-thumb-background-color);border-radius:10px}.thin-scrollbar::-webkit-scrollbar-thumb:hover{background:var(--ifm-scrollbar-thumb-hover-background-color)}}@media print{.announcementBar_mb4j,.footer,.menu,.navbar,.pagination-nav,.table-of-contents,.tocMobile_ITEo{display:none}.tabs{page-break-inside:avoid}.codeBlockLines_e6Vv{white-space:pre-wrap}} \ No newline at end of file diff --git a/17/assets/images/100-percent-cpu-1138186529f154d864f643179e25cea1.jpeg b/17/assets/images/100-percent-cpu-1138186529f154d864f643179e25cea1.jpeg new file mode 100644 index 0000000000..65cbcef814 Binary files /dev/null and b/17/assets/images/100-percent-cpu-1138186529f154d864f643179e25cea1.jpeg differ diff --git a/17/assets/images/app-os-cpu-interaction-ca7fbdbb7da380e0992c95467ef267ce.svg b/17/assets/images/app-os-cpu-interaction-ca7fbdbb7da380e0992c95467ef267ce.svg new file mode 100644 index 0000000000..72136f39ac --- /dev/null +++ b/17/assets/images/app-os-cpu-interaction-ca7fbdbb7da380e0992c95467ef267ce.svg @@ -0,0 +1,4 @@ + + + +
Process 1
Process 1
Thread 1
Thread 1
Thread 2
Thread 2
Thread 3%3CmxGraphModel%3E%3Croot%3E%3CmxCell%20id%3D%220%22%2F%3E%3CmxCell%20id%3D%221%22%20parent%3D%220%22%2F%3E%3CmxCell%20id%3D%222%22%20value%3D%22Thread%201%22%20style%3D%22text%3Bhtml%3D1%3BstrokeColor%3Dnone%3BfillColor%3Dnone%3Balign%3Dcenter%3BverticalAlign%3Dmiddle%3BwhiteSpace%3Dwrap%3Brounded%3D0%3BfontSize%3D20%3B%22%20vertex%3D%221%22%20parent%3D%221%22%3E%3CmxGeometry%20x%3D%22105%22%20y%3D%22135%22%20width%3D%2290%22%20height%3D%2230%22%20as%3D%22geometry%22%2F%3E%3C%2FmxCell%3E%3C%2Froot%3E%3C%2FmxGraphModel%3E
Thread 3%...
Thread 4
Thread 4
CPU
CPU
Core 1
Core 1
Core 2
Core 2
Core 3
Core 3
Core 4
Core 4
Core 5
Core 5
Core 6
Core 6
Core 7
Core 7
Core 8
Core 8
Process 2
Process 2
Thread 1
Thread 1
Thread 2
Thread 2
Operating System
Operating System
Application
Application
Text
Text
User
Space
User...
Kernel
Space
Kernel...
Text
Text
Hardware
HardwareText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/browser-tabs-2e3b83e1acd44607136a81a1a6346e56.svg b/17/assets/images/browser-tabs-2e3b83e1acd44607136a81a1a6346e56.svg new file mode 100644 index 0000000000..022f129c14 --- /dev/null +++ b/17/assets/images/browser-tabs-2e3b83e1acd44607136a81a1a6346e56.svg @@ -0,0 +1,4 @@ + + + +
Tab 1: File Descriptors
Tab 1: File Descripto...
Tab 1: File Handling
Tab 1: File Handling
Port: 34789
Port: 34789
Port: 25678
Port: 25678
github.io Server
github.io Server
Port: 31234
Port: 31234
Port: 32987
Port: 32987Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg b/17/assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg new file mode 100644 index 0000000000..2d0a8c1cb5 --- /dev/null +++ b/17/assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg @@ -0,0 +1,4 @@ + + + +
Server Storage
Server...
Web Server
Web Se...
Client
Client
2. Read file contents
2. Read file...
1. Request
1. Request
3. Response
3. ResponseText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/coalesce-blocks-d56dae67c5b407080ddab19c660ca56d.svg b/17/assets/images/coalesce-blocks-d56dae67c5b407080ddab19c660ca56d.svg new file mode 100644 index 0000000000..f1614bc936 --- /dev/null +++ b/17/assets/images/coalesce-blocks-d56dae67c5b407080ddab19c660ca56d.svg @@ -0,0 +1,4 @@ + + + +Coalesce Blocks
struct block_meta
struct block_...
payload
payload
struct block_meta
struct block_...
padding
padding
payload
payload
padding
padding
struct block_meta
struct block_...
payload
payload
padding
paddingText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/connection-establishment-05ca0959a59f3204b866855631182875.svg b/17/assets/images/connection-establishment-05ca0959a59f3204b866855631182875.svg new file mode 100644 index 0000000000..1c0c5f184d --- /dev/null +++ b/17/assets/images/connection-establishment-05ca0959a59f3204b866855631182875.svg @@ -0,0 +1,4 @@ + + + +
Server
Server
Client
Client
socket()
socket()
bind()
bind()
listen()
listen()
accept()
accept()
recv()
recv()
send()
send()
close()
close()
socket()
socket()
connect()
connect()
send()
send()
recv()
recv()
close()
close()
Time
Time
Connection Established
Connection...Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/containers-vs-vms-75f454982e671d86b4d127b6bed5db07.svg b/17/assets/images/containers-vs-vms-75f454982e671d86b4d127b6bed5db07.svg new file mode 100644 index 0000000000..ab2950e90d --- /dev/null +++ b/17/assets/images/containers-vs-vms-75f454982e671d86b4d127b6bed5db07.svg @@ -0,0 +1,4 @@ + + + +
Infrastructure
Infrastructure
Infrastructure
Infrastructure
Operating System
Operating System
Container Engine
Container Engine
Hypervisor
Hypervisor
Bins/Lib
Bins/Lib
Bins/Lib
Bins/Lib
Bins/Lib
Bins/Lib
App 1
App 1
App  3
App  3
App 2
App 2
Bins/Lib
Bins/Lib
Bins/Lib
Bins/Lib
Bins/Lib
Bins/Lib
App 1
App 1
App  3
App  3
App 2
App 2
Guest
OS
Guest...
Guest
OS
Guest...
Guest
OS
Guest...
Machine Virtualuzation
Machine Virtualuzation
Containers
ContainersText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/copy-on-write-final-2dfe1835636c0b38b11fed42b5b690d2.svg b/17/assets/images/copy-on-write-final-2dfe1835636c0b38b11fed42b5b690d2.svg new file mode 100644 index 0000000000..33e8245fa5 --- /dev/null +++ b/17/assets/images/copy-on-write-final-2dfe1835636c0b38b11fed42b5b690d2.svg @@ -0,0 +1,4 @@ + + + +
Frame A
Frame A
Frame B
Frame B
Frame C
Frame C
Physical Memory
Physical Memory
Code Page
Code Page
.rodata Page
.rodata Page
Heap Page
Heap Page
After
Memory
Write
After...
Parent Process
Page Table
Parent Process...
Child Process Page Table
Child Process...
Code Page
Code Page
.rodata Page
.rodata Page
Heap Page
Heap Page
r--
r--
rw-
rw-
r-x
r-x
Copy of Frame C
Copy of Frame C Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/copy-on-write-initial-a3673d26b2087aaacf630bc556e0a6a8.svg b/17/assets/images/copy-on-write-initial-a3673d26b2087aaacf630bc556e0a6a8.svg new file mode 100644 index 0000000000..0387aa5ed9 --- /dev/null +++ b/17/assets/images/copy-on-write-initial-a3673d26b2087aaacf630bc556e0a6a8.svg @@ -0,0 +1,4 @@ + + + +
Frame B
Frame B
Frame C
Frame C
Physical Memory
Physical Memory
.rodata Page
.rodata Page
Heap Page
Heap Page
Parent Process Page Table
Parent Process...
Child Process Page Table
Child Process...
Heap Page
Heap Page
fork()
fork()
r--
r--
rw-
rw-
r-x
r-x
Code Page
Code Page
Frame A
Frame A
Code Page
Code Page
.rodata Page
.rodata PageText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/dfeet_execute-0e80376026099d72795e2e52e08d754b.png b/17/assets/images/dfeet_execute-0e80376026099d72795e2e52e08d754b.png new file mode 100644 index 0000000000..359bf48f11 Binary files /dev/null and b/17/assets/images/dfeet_execute-0e80376026099d72795e2e52e08d754b.png differ diff --git a/17/assets/images/dfeet_execute-e3d7472f8f70d73deabbaa7d217c10bb.gif b/17/assets/images/dfeet_execute-e3d7472f8f70d73deabbaa7d217c10bb.gif new file mode 100644 index 0000000000..e6c5d698d4 Binary files /dev/null and b/17/assets/images/dfeet_execute-e3d7472f8f70d73deabbaa7d217c10bb.gif differ diff --git a/17/assets/images/dfeet_firefox-f29826be8e314acb7f0047d5b5b86431.png b/17/assets/images/dfeet_firefox-f29826be8e314acb7f0047d5b5b86431.png new file mode 100644 index 0000000000..60d6597d62 Binary files /dev/null and b/17/assets/images/dfeet_firefox-f29826be8e314acb7f0047d5b5b86431.png differ diff --git a/17/assets/images/dfeet_notifications-80931f45466247c5cd1d6164d83f97d8.png b/17/assets/images/dfeet_notifications-80931f45466247c5cd1d6164d83f97d8.png new file mode 100644 index 0000000000..3de86f7d94 Binary files /dev/null and b/17/assets/images/dfeet_notifications-80931f45466247c5cd1d6164d83f97d8.png differ diff --git a/17/assets/images/dfeet_session_bus-1772a989fbb5f4e3c0f5d0569771e54b.png b/17/assets/images/dfeet_session_bus-1772a989fbb5f4e3c0f5d0569771e54b.png new file mode 100644 index 0000000000..e92ed1a568 Binary files /dev/null and b/17/assets/images/dfeet_session_bus-1772a989fbb5f4e3c0f5d0569771e54b.png differ diff --git a/17/assets/images/dfeet_url_open-99b78cbaf472fad6a0c5dd4d550b4a55.gif b/17/assets/images/dfeet_url_open-99b78cbaf472fad6a0c5dd4d550b4a55.gif new file mode 100644 index 0000000000..38755c56d9 Binary files /dev/null and b/17/assets/images/dfeet_url_open-99b78cbaf472fad6a0c5dd4d550b4a55.gif differ diff --git a/17/assets/images/file-descriptors-d19424f0a417ecd1c032f98a1969ad75.svg b/17/assets/images/file-descriptors-d19424f0a417ecd1c032f98a1969ad75.svg new file mode 100644 index 0000000000..a898a2d374 --- /dev/null +++ b/17/assets/images/file-descriptors-d19424f0a417ecd1c032f98a1969ad75.svg @@ -0,0 +1,4 @@ + + + +
FD Table
FD Table
0
0
1
1
2
2
3
3
stdin
stdin
stdout
stdout
stderr
stderr
file 1
file 1
4
4
file 2
file 2
...
...
Keyboard
Keyboard
Display
Display
/home/student/file1
/home/student/file1
/path/to/file2
/path/to/file2
Open File Structs
Open File Struct...Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/firefox_url_open-85c4cc365a96e2d43b66891dc60c0911.gif b/17/assets/images/firefox_url_open-85c4cc365a96e2d43b66891dc60c0911.gif new file mode 100644 index 0000000000..04223ca5c8 Binary files /dev/null and b/17/assets/images/firefox_url_open-85c4cc365a96e2d43b66891dc60c0911.gif differ diff --git a/17/assets/images/fork-exec-e8ff2e7cb057592463ccc850bdaa0228.svg b/17/assets/images/fork-exec-e8ff2e7cb057592463ccc850bdaa0228.svg new file mode 100644 index 0000000000..0ca4dd5063 --- /dev/null +++ b/17/assets/images/fork-exec-e8ff2e7cb057592463ccc850bdaa0228.svg @@ -0,0 +1,4 @@ + + + +
bash
bash
bash
bash
bash
bash
bash
bash
ls
ls
fork() = 0
fork() = 0
fork() > 0
fork() > 0
exec("/bin/ls")
exec("/bin/l...
waitpid()
waitpid()
student@os:~$ ls
student@os:~$ ls
Time
TimeText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/gdbus_notify-ea95f7a06a7878b31ad48bbdfaba30bb.gif b/17/assets/images/gdbus_notify-ea95f7a06a7878b31ad48bbdfaba30bb.gif new file mode 100644 index 0000000000..bdf940b39a Binary files /dev/null and b/17/assets/images/gdbus_notify-ea95f7a06a7878b31ad48bbdfaba30bb.gif differ diff --git a/17/assets/images/hardware-software-98463c6b754c2d0e75ccc886451176e6.svg b/17/assets/images/hardware-software-98463c6b754c2d0e75ccc886451176e6.svg new file mode 100644 index 0000000000..c633ef61ae --- /dev/null +++ b/17/assets/images/hardware-software-98463c6b754c2d0e75ccc886451176e6.svg @@ -0,0 +1,4 @@ + + + +
Game
Game
Document
Document
Shopping
Shopping
Video
Video
News
News
Chat
Chat
what we have
what we have
what we want
what we want
hardware (devices)
hardware (devices)
features
features
software
software
have-to-want
have-to-wantText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/loading-of-ls-process-0dec67c0d0a826710e06f980224d5eb4.svg b/17/assets/images/loading-of-ls-process-0dec67c0d0a826710e06f980224d5eb4.svg new file mode 100644 index 0000000000..8f19e1ea37 --- /dev/null +++ b/17/assets/images/loading-of-ls-process-0dec67c0d0a826710e06f980224d5eb4.svg @@ -0,0 +1,4 @@ + + + +
Loader
code
Loader...
execve("ls", ...)
execve("ls", ...)
Read ELF file /bin/ls
Read ELF file /bin/...
Setup VAS of ls process
Setup VAS of ls pro...
Start main thread of ls process
Start main thread o...
...
...
Initial
process
Initial...
ls
process
ls...Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/memory-block-27bcad65f280db00766c7a9824c624f2.svg b/17/assets/images/memory-block-27bcad65f280db00766c7a9824c624f2.svg new file mode 100644 index 0000000000..d051bd1c59 --- /dev/null +++ b/17/assets/images/memory-block-27bcad65f280db00766c7a9824c624f2.svg @@ -0,0 +1,4 @@ + + + +Memory Block
0x00
0x00
struct block_meta
struct block_meta
payload
payload
padding
padding
padding
padding
0x08
0x08
0x10
0x10
0x18
0x18
0x20
0x20
0x28
0x28Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/modern-sw-stack-4a2427d07a59c3a6599305b8eedc43dd.svg b/17/assets/images/modern-sw-stack-4a2427d07a59c3a6599305b8eedc43dd.svg new file mode 100644 index 0000000000..6c46692395 --- /dev/null +++ b/17/assets/images/modern-sw-stack-4a2427d07a59c3a6599305b8eedc43dd.svg @@ -0,0 +1,4 @@ + + + +
Library call
Library call
Application
Application
Library
Library
Operating System
Operating System
Hardware
Hardware
Library API
Library API
System Call API
System Call API
System call
System call
Hardware
HardwareText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/nested_virt_vbox-4c9c494a233277658f851662ca2dc082.png b/17/assets/images/nested_virt_vbox-4c9c494a233277658f851662ca2dc082.png new file mode 100644 index 0000000000..90d059b49f Binary files /dev/null and b/17/assets/images/nested_virt_vbox-4c9c494a233277658f851662ca2dc082.png differ diff --git a/17/assets/images/nested_virt_vmware-1b3a6b314c74c626bc006936abb01bf7.png b/17/assets/images/nested_virt_vmware-1b3a6b314c74c626bc006936abb01bf7.png new file mode 100644 index 0000000000..a10a51bc1b Binary files /dev/null and b/17/assets/images/nested_virt_vmware-1b3a6b314c74c626bc006936abb01bf7.png differ diff --git a/17/assets/images/os_cloud-ff20c47681f663aa8430106ab44e0cd5.svg b/17/assets/images/os_cloud-ff20c47681f663aa8430106ab44e0cd5.svg new file mode 100644 index 0000000000..708471638b --- /dev/null +++ b/17/assets/images/os_cloud-ff20c47681f663aa8430106ab44e0cd5.svg @@ -0,0 +1,4 @@ + + + +
DB
DB
OS-Cloud
OS-Cloud
OS-Cloud API
OS-Cloud API
VM
VM
VM
VM
VM
VM
User
User
QEMU
QEMU
QEMU
QEMU
QEMU
QEMUText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/os_cloud_networking-c27fac0c5e0a3100a184bf283728b467.svg b/17/assets/images/os_cloud_networking-c27fac0c5e0a3100a184bf283728b467.svg new file mode 100644 index 0000000000..2851677b4c --- /dev/null +++ b/17/assets/images/os_cloud_networking-c27fac0c5e0a3100a184bf283728b467.svg @@ -0,0 +1,4 @@ + + + +
ens0
ens0
ens0
ens0
tap0
tap0
tap1
tap1
192.168.0.2
192.168.0.2
192.168.0.3
192.168.0.3
br0
192.168.0.1
br0...
VM 1
VM 1
VM 2
VM 2Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/piped-commands-118a36fba312c6bea5270dba74d653e2.svg b/17/assets/images/piped-commands-118a36fba312c6bea5270dba74d653e2.svg new file mode 100644 index 0000000000..3810db9150 --- /dev/null +++ b/17/assets/images/piped-commands-118a36fba312c6bea5270dba74d653e2.svg @@ -0,0 +1,4 @@ + + + +
input
input
input
input
input
input
cat
cat
tr
tr
cut
cut
output
output
output
output
output
output
...
...Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/sad-pepe-93384f7b72f31ab085cdf22a743d769d.png b/17/assets/images/sad-pepe-93384f7b72f31ab085cdf22a743d769d.png new file mode 100644 index 0000000000..9b82b8f535 Binary files /dev/null and b/17/assets/images/sad-pepe-93384f7b72f31ab085cdf22a743d769d.png differ diff --git a/17/assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg b/17/assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg new file mode 100644 index 0000000000..aa6f37b0ac --- /dev/null +++ b/17/assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg @@ -0,0 +1,4 @@ + + + +
Read Buffer
Read Buffer
Write Buffer
Write Buffer
App Buffer
App Buffer
User Space
User Space
Kernel Space
Kernel Space
Server Storage
Server S...
NIC
NIC
NIC Buffer
NIC Buffer
Internet
Internet
read()
read()
send()
send()Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg b/17/assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg new file mode 100644 index 0000000000..13fb1fbbfe --- /dev/null +++ b/17/assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg @@ -0,0 +1,4 @@ + + + +
Buffer
Buffer
User Space
User Space
Kernel Space
Kernel Space
Server Storage
Server S...
NIC
NIC
NIC Buffer
NIC Buffer
Internet
Internet
sendfile()
sendfile()Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/software-stack-ffefaf1db325c303dbc9909b608b58a5.svg b/17/assets/images/software-stack-ffefaf1db325c303dbc9909b608b58a5.svg new file mode 100644 index 0000000000..ea9bb256f8 --- /dev/null +++ b/17/assets/images/software-stack-ffefaf1db325c303dbc9909b608b58a5.svg @@ -0,0 +1,4 @@ + + + +
glib, libpng, libxml, protobuf, APR, lib-7zip
glib, libpng, libxml, protobuf, APR, lib-7zip
Python, Ruby, lib-druntime, V8, JRE
Python, Ruby, lib-druntime, V8, JRE
libc, Musl, pthread, libdl, kernel32.dll
libc, Musl, pthread, libdl, kernel32.dll
Linux, NT kernel, Darwin / XNU
Linux, NT kernel, Darwin / XNU
Firefox, Vim, VS Code, Dota 2, Nginx
Firefox, Vim, VS Code, Dota 2, Nginx
x86, ARM, MIPS, RISC-V
x86, ARM, MIPS, RISC-V ApplicationsSupport LibrariesLanguage EnvironmentsSystem LibrariesOperating SystemHardwareText is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/software-stacks-everywhere-3f33e8d01bec7787cdaad5752d91a4b4.jpeg b/17/assets/images/software-stacks-everywhere-3f33e8d01bec7787cdaad5752d91a4b4.jpeg new file mode 100644 index 0000000000..238f37c09e Binary files /dev/null and b/17/assets/images/software-stacks-everywhere-3f33e8d01bec7787cdaad5752d91a4b4.jpeg differ diff --git a/17/assets/images/split-block-0e690e7363ed4af850595e15c0f8cd7e.svg b/17/assets/images/split-block-0e690e7363ed4af850595e15c0f8cd7e.svg new file mode 100644 index 0000000000..50905a9366 --- /dev/null +++ b/17/assets/images/split-block-0e690e7363ed4af850595e15c0f8cd7e.svg @@ -0,0 +1,4 @@ + + + +Split Block
unused
unused
struct block_meta
struct block_...
payload
payload
padding
padding
free space
free space
struct block_meta
struct block_...
payload
payload
padding
padding
struct block_meta
struct block_...Text is not SVG - cannot display \ No newline at end of file diff --git a/17/assets/images/standard-us-power-outlet-c4b65ad6ff6a97ce0b898b5ca79be88f.jpg b/17/assets/images/standard-us-power-outlet-c4b65ad6ff6a97ce0b898b5ca79be88f.jpg new file mode 100644 index 0000000000..926c7b17fb Binary files /dev/null and b/17/assets/images/standard-us-power-outlet-c4b65ad6ff6a97ce0b898b5ca79be88f.jpg differ diff --git a/17/assets/images/strace_xeyes-f055c379e7028e39de13f0d039171718.gif b/17/assets/images/strace_xeyes-f055c379e7028e39de13f0d039171718.gif new file mode 100644 index 0000000000..ac92770719 Binary files /dev/null and b/17/assets/images/strace_xeyes-f055c379e7028e39de13f0d039171718.gif differ diff --git a/17/assets/images/syscall-explained-c884ed051906c31957f8621ae87d3b2f.svg b/17/assets/images/syscall-explained-c884ed051906c31957f8621ae87d3b2f.svg new file mode 100644 index 0000000000..1e03169ca3 --- /dev/null +++ b/17/assets/images/syscall-explained-c884ed051906c31957f8621ae87d3b2f.svg @@ -0,0 +1,867 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/17/assets/images/tcp-udp-simplified-7bb728409d8a83b05d5d574df21d9ec4.png b/17/assets/images/tcp-udp-simplified-7bb728409d8a83b05d5d574df21d9ec4.png new file mode 100644 index 0000000000..654c52e3fc Binary files /dev/null and b/17/assets/images/tcp-udp-simplified-7bb728409d8a83b05d5d574df21d9ec4.png differ diff --git a/17/assets/js/007cf997.ee6e5852.js b/17/assets/js/007cf997.ee6e5852.js new file mode 100644 index 0000000000..cc89040625 --- /dev/null +++ b/17/assets/js/007cf997.ee6e5852.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8631],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>d});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var s=n.createContext({}),l=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},u=function(e){var t=l(e.components);return n.createElement(s.Provider,{value:t},e.children)},p="mdxType",h={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,s=e.parentName,u=c(e,["components","mdxType","originalType","parentName"]),p=l(r),m=a,d=p["".concat(s,".").concat(m)]||p[m]||h[m]||o;return r?n.createElement(d,i(i({ref:t},u),{},{components:r})):n.createElement(d,i({ref:t},u))}));function d(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=m;var c={};for(var s in t)hasOwnProperty.call(t,s)&&(c[s]=t[s]);c.originalType=e,c[p]="string"==typeof e?e:a,i[1]=c;for(var l=2;l{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>i,default:()=>h,frontMatter:()=>o,metadata:()=>c,toc:()=>l});var n=r(7462),a=(r(7294),r(3905));const o={},i="Coarse vs Granular Critical Section",c={unversionedId:"Lab/Compute/quiz/coarse-vs-granular-critical-section",id:"Lab/Compute/quiz/coarse-vs-granular-critical-section",title:"Coarse vs Granular Critical Section",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/coarse-vs-granular-critical-section.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/coarse-vs-granular-critical-section",permalink:"/operating-systems/17/Lab/Compute/quiz/coarse-vs-granular-critical-section",draft:!1,tags:[],version:"current",frontMatter:{}},s={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:l},p="wrapper";function h(e){let{components:t,...r}=e;return(0,a.kt)(p,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"coarse-vs-granular-critical-section"},"Coarse vs Granular Critical Section"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Why does code with the the larger (coarser) critical section run faster than the one with the smaller (more granular) critical section?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because the more granular code causes more context switches, which are expensive")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the coarser code can be better optimised by the compiler")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the loops in the more granular code run for more steps"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"The larger critical sections only require ",(0,a.kt)("strong",{parentName:"p"},"2 context switches"),".\nThe first thread that reaches the call to ",(0,a.kt)("inlineCode",{parentName:"p"},"lock()")," acquires to the mutex and starts executing the whole of the ",(0,a.kt)("inlineCode",{parentName:"p"},"for")," loop.\nThe second thread then finds the mutex ",(0,a.kt)("em",{parentName:"p"},"locked")," and enters the WAITING state.\nWhen the first thread finishes its loop, it calls ",(0,a.kt)("inlineCode",{parentName:"p"},"unlock()")," and wakes up the second thread, which acquires the lock and starts its loop."),(0,a.kt)("p",null,"In the more granular example, in the worst case, the holder of the mutex can change at every step of the loop.\nThis would mean 1 context switch per step per thread, i.e. ",(0,a.kt)("strong",{parentName:"p"},"20 million context switches"),"."))}h.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0275a3b2.dbdf66c4.js b/17/assets/js/0275a3b2.dbdf66c4.js new file mode 100644 index 0000000000..6a7702482b --- /dev/null +++ b/17/assets/js/0275a3b2.dbdf66c4.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5782],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>f});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var s=r.createContext({}),p=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(s.Provider,{value:t},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,s=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),m=p(n),d=o,f=m["".concat(s,".").concat(d)]||m[d]||u[d]||a;return n?r.createElement(f,i(i({ref:t},c),{},{components:n})):r.createElement(f,i({ref:t},c))}));function f(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=d;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[m]="string"==typeof e?e:o,i[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>i,default:()=>u,frontMatter:()=>a,metadata:()=>l,toc:()=>p});var r=n(7462),o=(n(7294),n(3905));const a={},i="Mini-shell Stops After Command",l={unversionedId:"Lab/Compute/quiz/mini-shell-stops-after-command",id:"Lab/Compute/quiz/mini-shell-stops-after-command",title:"Mini-shell Stops After Command",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/mini-shell-stops-after-command.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/mini-shell-stops-after-command",permalink:"/operating-systems/17/Lab/Compute/quiz/mini-shell-stops-after-command",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},m="wrapper";function u(e){let{components:t,...n}=e;return(0,o.kt)(m,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"mini-shell-stops-after-command"},"Mini-shell Stops After Command"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Why does the ",(0,o.kt)("inlineCode",{parentName:"p"},"mini_shell")," process stop after executing a single command?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"Because of an implementation error")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"Because the ",(0,o.kt)("inlineCode",{parentName:"li"},"mini_shell")," process doesn't exist anymore")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Because the OS sees that the command has ended and ends the ",(0,o.kt)("inlineCode",{parentName:"p"},"mini_shell")," process as well")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Because ",(0,o.kt)("inlineCode",{parentName:"p"},"exec*()")," syscalls also kill the caller process when the callee ends"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"When you ",(0,o.kt)("inlineCode",{parentName:"p"},"exec*()")," any binary, the VAS current process is ",(0,o.kt)("strong",{parentName:"p"},"replaced")," by that corresponding to that binary.\nSo when you ",(0,o.kt)("inlineCode",{parentName:"p"},'exec*("ls")'),", for example, the ",(0,o.kt)("inlineCode",{parentName:"p"},"mini_shell")," process ",(0,o.kt)("em",{parentName:"p"},"becomes")," ",(0,o.kt)("inlineCode",{parentName:"p"},"ls"),".\nThere is no more ",(0,o.kt)("inlineCode",{parentName:"p"},"mini_shell")," past this point.\nSo when ",(0,o.kt)("inlineCode",{parentName:"p"},"ls")," ends, there is no ",(0,o.kt)("inlineCode",{parentName:"p"},"mini_shell")," process to continue its execution anymore."))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0277d735.917106af.js b/17/assets/js/0277d735.917106af.js new file mode 100644 index 0000000000..09d03cd37c --- /dev/null +++ b/17/assets/js/0277d735.917106af.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1018],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>d});var r=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function l(e){for(var t=1;t=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var u=r.createContext({}),c=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},p=function(e){var t=c(e.components);return r.createElement(u.Provider,{value:t},e.children)},f="mdxType",s={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},b=r.forwardRef((function(e,t){var n=e.components,i=e.mdxType,a=e.originalType,u=e.parentName,p=o(e,["components","mdxType","originalType","parentName"]),f=c(n),b=i,d=f["".concat(u,".").concat(b)]||f[b]||s[b]||a;return n?r.createElement(d,l(l({ref:t},p),{},{components:n})):r.createElement(d,l({ref:t},p))}));function d(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var a=n.length,l=new Array(a);l[0]=b;var o={};for(var u in t)hasOwnProperty.call(t,u)&&(o[u]=t[u]);o.originalType=e,o[f]="string"==typeof e?e:i,l[1]=o;for(var c=2;c{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>l,default:()=>s,frontMatter:()=>a,metadata:()=>o,toc:()=>c});var r=n(7462),i=(n(7294),n(3905));const a={},l="Flush Libc Buffer",o={unversionedId:"Lab/IO/quiz/flush-libc-buffer",id:"Lab/IO/quiz/flush-libc-buffer",title:"Flush Libc Buffer",description:"Question Text",source:"@site/docs/Lab/IO/quiz/flush-libc-buffer.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/flush-libc-buffer",permalink:"/operating-systems/17/Lab/IO/quiz/flush-libc-buffer",draft:!1,tags:[],version:"current",frontMatter:{}},u={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:c},f="wrapper";function s(e){let{components:t,...n}=e;return(0,i.kt)(f,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"flush-libc-buffer"},"Flush Libc Buffer"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Which of the following is a method of flushing libc's internal buffer?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"print a ",(0,i.kt)("inlineCode",{parentName:"li"},"\\0")," character")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"print a ",(0,i.kt)("inlineCode",{parentName:"li"},"\\n")," character")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"print a space character")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"print a ",(0,i.kt)("inlineCode",{parentName:"p"},"\\t")," character"))),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,"Newlines (",(0,i.kt)("inlineCode",{parentName:"p"},"\\n"),") force ",(0,i.kt)("inlineCode",{parentName:"p"},"printf()")," to dump the internal buffer associated with the ",(0,i.kt)("inlineCode",{parentName:"p"},"stdout")," ",(0,i.kt)("inlineCode",{parentName:"p"},"FILE")," ",(0,i.kt)("inlineCode",{parentName:"p"},"struct"),".\nIf you place a ",(0,i.kt)("inlineCode",{parentName:"p"},"\\n")," character within one of the strings in ",(0,i.kt)("inlineCode",{parentName:"p"},"support/buffering/printf_buffering.c"),", a ",(0,i.kt)("inlineCode",{parentName:"p"},"write()")," syscall will be made right after it."))}s.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/03dc0c9b.4f4b4f3b.js b/17/assets/js/03dc0c9b.4f4b4f3b.js new file mode 100644 index 0000000000..0ce9745415 --- /dev/null +++ b/17/assets/js/03dc0c9b.4f4b4f3b.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[214],{3905:(e,n,a)=>{a.d(n,{Zo:()=>c,kt:()=>u});var t=a(7294);function l(e,n,a){return n in e?Object.defineProperty(e,n,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[n]=a,e}function i(e,n){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var t=Object.getOwnPropertySymbols(e);n&&(t=t.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),a.push.apply(a,t)}return a}function o(e){for(var n=1;n=0||(l[a]=e[a]);return l}(e,n);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(t=0;t=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(l[a]=e[a])}return l}var s=t.createContext({}),p=function(e){var n=t.useContext(s),a=n;return e&&(a="function"==typeof e?e(n):o(o({},n),e)),a},c=function(e){var n=p(e.components);return t.createElement(s.Provider,{value:n},e.children)},d="mdxType",m={inlineCode:"code",wrapper:function(e){var n=e.children;return t.createElement(t.Fragment,{},n)}},f=t.forwardRef((function(e,n){var a=e.components,l=e.mdxType,i=e.originalType,s=e.parentName,c=r(e,["components","mdxType","originalType","parentName"]),d=p(a),f=l,u=d["".concat(s,".").concat(f)]||d[f]||m[f]||i;return a?t.createElement(u,o(o({ref:n},c),{},{components:a})):t.createElement(u,o({ref:n},c))}));function u(e,n){var a=arguments,l=n&&n.mdxType;if("string"==typeof e||l){var i=a.length,o=new Array(i);o[0]=f;var r={};for(var s in n)hasOwnProperty.call(n,s)&&(r[s]=n[s]);r.originalType=e,r[d]="string"==typeof e?e:l,o[1]=r;for(var p=2;p{a.r(n),a.d(n,{assets:()=>s,contentTitle:()=>o,default:()=>m,frontMatter:()=>i,metadata:()=>r,toc:()=>p});var t=a(7462),l=(a(7294),a(3905));const i={},o="Process Memory",r={unversionedId:"Lab/Data/process-memory",id:"Lab/Data/process-memory",title:"Process Memory",description:"Memory Regions",source:"@site/docs/Lab/Data/process-memory.md",sourceDirName:"Lab/Data",slug:"/Lab/Data/process-memory",permalink:"/operating-systems/17/Lab/Data/process-memory",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Working with Memory",permalink:"/operating-systems/17/Lab/Data/working-memory"},next:{title:"Investigate Memory Actions",permalink:"/operating-systems/17/Lab/Data/investigate-memory"}},s={},p=[{value:"Memory Regions",id:"memory-regions",level:2},{value:"Practice",id:"practice",level:3},{value:"Memory Layout of Statically-Linked and Dynamically-Linked Executables",id:"memory-layout-of-statically-linked-and-dynamically-linked-executables",level:2},{value:"Quiz",id:"quiz",level:3},{value:"Practice",id:"practice-1",level:3},{value:"Modifying Memory Region Size",id:"modifying-memory-region-size",level:2},{value:"Practice",id:"practice-2",level:3},{value:"Allocating and Deallocating Memory",id:"allocating-and-deallocating-memory",level:2},{value:"Practice",id:"practice-3",level:3},{value:"Memory Mapping",id:"memory-mapping",level:2},{value:"Practice",id:"practice-4",level:3}],c={toc:p},d="wrapper";function m(e){let{components:n,...a}=e;return(0,l.kt)(d,(0,t.Z)({},c,a,{components:n,mdxType:"MDXLayout"}),(0,l.kt)("h1",{id:"process-memory"},"Process Memory"),(0,l.kt)("h2",{id:"memory-regions"},"Memory Regions"),(0,l.kt)("p",null,"To better manage a program's memory, the operating systems creates an address space for each process.\nThe address space is compartmentalized in multiple areas, each with its own role.\nMemory addresses use different permissions to decide what actions are allowed."),(0,l.kt)("p",null,"Let's investigate the memory areas of a given process.\nWe use ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," to see the memory layout of a running process.\nThe command below shows the memory layout of the current shell process:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ pmap -p $$\n1127: /bin/bash\n000055fb4d77d000 1040K r-x-- /bin/bash\n000055fb4da80000 16K r---- /bin/bash\n000055fb4da84000 36K rw--- /bin/bash\n000055fb4da8d000 40K rw--- [ anon ]\n000055fb4e9bb000 1604K rw--- [ anon ]\n00007f8fcf670000 4480K r---- /usr/lib/locale/locale-archive\n00007f8fcfad0000 44K r-x-- /lib/x86_64-linux-gnu/libnss_files-2.27.so\n00007f8fcfadb000 2044K ----- /lib/x86_64-linux-gnu/libnss_files-2.27.so\n00007f8fcfcda000 4K r---- /lib/x86_64-linux-gnu/libnss_files-2.27.so\n00007f8fcfcdb000 4K rw--- /lib/x86_64-linux-gnu/libnss_files-2.27.so\n00007f8fcfcdc000 24K rw--- [ anon ]\n00007f8fcfce2000 92K r-x-- /lib/x86_64-linux-gnu/libnsl-2.27.so\n00007f8fcfcf9000 2044K ----- /lib/x86_64-linux-gnu/libnsl-2.27.so\n00007f8fcfef8000 4K r---- /lib/x86_64-linux-gnu/libnsl-2.27.so\n00007f8fcfef9000 4K rw--- /lib/x86_64-linux-gnu/libnsl-2.27.so\n00007f8fcfefa000 8K rw--- [ anon ]\n00007f8fcfefc000 44K r-x-- /lib/x86_64-linux-gnu/libnss_nis-2.27.so\n00007f8fcff07000 2044K ----- /lib/x86_64-linux-gnu/libnss_nis-2.27.so\n00007f8fd0106000 4K r---- /lib/x86_64-linux-gnu/libnss_nis-2.27.so\n00007f8fd0107000 4K rw--- /lib/x86_64-linux-gnu/libnss_nis-2.27.so\n00007f8fd0108000 32K r-x-- /lib/x86_64-linux-gnu/libnss_compat-2.27.so\n00007f8fd0110000 2048K ----- /lib/x86_64-linux-gnu/libnss_compat-2.27.so\n00007f8fd0310000 4K r---- /lib/x86_64-linux-gnu/libnss_compat-2.27.so\n00007f8fd0311000 4K rw--- /lib/x86_64-linux-gnu/libnss_compat-2.27.so\n00007f8fd0312000 1948K r-x-- /lib/x86_64-linux-gnu/libc-2.27.so\n00007f8fd04f9000 2048K ----- /lib/x86_64-linux-gnu/libc-2.27.so\n00007f8fd06f9000 16K r---- /lib/x86_64-linux-gnu/libc-2.27.so\n00007f8fd06fd000 8K rw--- /lib/x86_64-linux-gnu/libc-2.27.so\n00007f8fd06ff000 16K rw--- [ anon ]\n00007f8fd0703000 12K r-x-- /lib/x86_64-linux-gnu/libdl-2.27.so\n00007f8fd0706000 2044K ----- /lib/x86_64-linux-gnu/libdl-2.27.so\n00007f8fd0905000 4K r---- /lib/x86_64-linux-gnu/libdl-2.27.so\n00007f8fd0906000 4K rw--- /lib/x86_64-linux-gnu/libdl-2.27.so\n00007f8fd0907000 148K r-x-- /lib/x86_64-linux-gnu/libtinfo.so.5.9\n00007f8fd092c000 2048K ----- /lib/x86_64-linux-gnu/libtinfo.so.5.9\n00007f8fd0b2c000 16K r---- /lib/x86_64-linux-gnu/libtinfo.so.5.9\n00007f8fd0b30000 4K rw--- /lib/x86_64-linux-gnu/libtinfo.so.5.9\n00007f8fd0b31000 164K r-x-- /lib/x86_64-linux-gnu/ld-2.27.so\n00007f8fd0d24000 20K rw--- [ anon ]\n00007f8fd0d53000 28K r--s- /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache\n00007f8fd0d5a000 4K r---- /lib/x86_64-linux-gnu/ld-2.27.so\n00007f8fd0d5b000 4K rw--- /lib/x86_64-linux-gnu/ld-2.27.so\n00007f8fd0d5c000 4K rw--- [ anon ]\n00007ffff002f000 132K rw--- [ stack ]\n00007ffff00c5000 12K r---- [ anon ]\n00007ffff00c8000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 24364K\n")),(0,l.kt)("p",null,"Information will differ among different systems."),(0,l.kt)("p",null,"See the different regions:"),(0,l.kt)("ul",null,(0,l.kt)("li",{parentName:"ul"},"the first region, with ",(0,l.kt)("inlineCode",{parentName:"li"},"r-x")," permissions is the ",(0,l.kt)("inlineCode",{parentName:"li"},".text")," (code) area"),(0,l.kt)("li",{parentName:"ul"},"the second region, with ",(0,l.kt)("inlineCode",{parentName:"li"},"r--")," premissions is the ",(0,l.kt)("inlineCode",{parentName:"li"},".rodata")," area"),(0,l.kt)("li",{parentName:"ul"},"the third region, with ",(0,l.kt)("inlineCode",{parentName:"li"},"rw-")," permissions is the ",(0,l.kt)("inlineCode",{parentName:"li"},".data")," area, for initialized global variables"),(0,l.kt)("li",{parentName:"ul"},"the fourth region, with ",(0,l.kt)("inlineCode",{parentName:"li"},"rw-")," permissions is the ",(0,l.kt)("inlineCode",{parentName:"li"},".bss")," area"),(0,l.kt)("li",{parentName:"ul"},"the fifth region, with the ",(0,l.kt)("inlineCode",{parentName:"li"},"rw-")," permissions is the dynamic data memory area, also known as heap"),(0,l.kt)("li",{parentName:"ul"},"there are multiple dynamic libraries mapped in the virtual address space of the process, each library with their own regions"),(0,l.kt)("li",{parentName:"ul"},"there is a ",(0,l.kt)("inlineCode",{parentName:"li"},"[stack]")," memory region, with ",(0,l.kt)("inlineCode",{parentName:"li"},"rw-")," permissions")),(0,l.kt)("p",null,(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," also shows the total amount of virtual memory available to the process (",(0,l.kt)("inlineCode",{parentName:"p"},"24364K"),"), as a total of the sizes of the regions.\nNote that this is virtual memory, not actual physical memory used by the process.\nFor the process investigated above (with the ",(0,l.kt)("inlineCode",{parentName:"p"},"1127")," pid) we could use the command below to show the total virtual size and physical size (also called ",(0,l.kt)("em",{parentName:"p"},"resident set size"),"):"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -o pid,rss,vsz -p $$\n PID RSS VSZ\n 1127 1968 24364\n")),(0,l.kt)("p",null,"The resident size is ",(0,l.kt)("inlineCode",{parentName:"p"},"1968K"),", much smaller than the virtual size."),(0,l.kt)("p",null,"Note how each region has a size multiple of ",(0,l.kt)("inlineCode",{parentName:"p"},"4K"),", this has to do with the memory granularity.\nThe operating system allocates memory in chunks of a predefined size (in our case ",(0,l.kt)("inlineCode",{parentName:"p"},"4K"),") called pages."),(0,l.kt)("p",null,(0,l.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/half-page"},"Quiz")),(0,l.kt)("h3",{id:"practice"},"Practice"),(0,l.kt)("p",null,"Enter the ",(0,l.kt)("inlineCode",{parentName:"p"},"support/memory-areas/")," directory.\nWe investigate other programs."),(0,l.kt)("ol",null,(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"The ",(0,l.kt)("inlineCode",{parentName:"p"},"hello.c")," program prints out a message and then sleeps.\nBuild it:"),(0,l.kt)("pre",{parentName:"li"},(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-areas$ make\n")),(0,l.kt)("p",{parentName:"li"},"then run it (it will block):"),(0,l.kt)("pre",{parentName:"li"},(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-areas$ ./hello\nHello, world!\n")),(0,l.kt)("p",{parentName:"li"},"In another terminal, use the command below to show the memory areas of the process:"),(0,l.kt)("pre",{parentName:"li"},(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-areas$ pmap $(pidof hello)\n8220: ./hello\n000055c0bef4b000 8K r-x-- hello\n000055c0bf14c000 4K r---- hello\n000055c0bf14d000 4K rw--- hello\n000055c0bf454000 132K rw--- [ anon ]\n00007f2a9e4a5000 1948K r-x-- libc-2.27.so\n00007f2a9e68c000 2048K ----- libc-2.27.so\n00007f2a9e88c000 16K r---- libc-2.27.so\n00007f2a9e890000 8K rw--- libc-2.27.so\n00007f2a9e892000 16K rw--- [ anon ]\n00007f2a9e896000 164K r-x-- ld-2.27.so\n00007f2a9ea8c000 8K rw--- [ anon ]\n00007f2a9eabf000 4K r---- ld-2.27.so\n00007f2a9eac0000 4K rw--- ld-2.27.so\n00007f2a9eac1000 4K rw--- [ anon ]\n00007ffee6471000 132K rw--- [ stack ]\n00007ffee6596000 12K r---- [ anon ]\n00007ffee6599000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 4520K\n")),(0,l.kt)("p",{parentName:"li"},"The output is similar, but with fewer dynamic libraries than ",(0,l.kt)("inlineCode",{parentName:"p"},"bash"),", since they are not used by the program.")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Make a program in another language of your choice that prints ",(0,l.kt)("inlineCode",{parentName:"p"},"Hello, world!")," and sleeps and investigate it with ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap"),".\nNote that in the case of interpreted languages (Python, Lua, Perl, Ruby, PHP, JavaScript etc.) you have to investigate the interpreter process."))),(0,l.kt)("h2",{id:"memory-layout-of-statically-linked-and-dynamically-linked-executables"},"Memory Layout of Statically-Linked and Dynamically-Linked Executables"),(0,l.kt)("p",null,"We want to see the difference in memory layout between the statically-linked and dynamically-linked executables."),(0,l.kt)("p",null,"Enter the ",(0,l.kt)("inlineCode",{parentName:"p"},"support/static-dynamic/")," directory and build the statically-linked and dynamically-linked executables ",(0,l.kt)("inlineCode",{parentName:"p"},"hello-static")," and ",(0,l.kt)("inlineCode",{parentName:"p"},"hello-dynamic"),":"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/static-dynamic$ make\n")),(0,l.kt)("p",null,"Now, by running the two programs and inspecting them with ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," on another terminal, we get the output:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/static-dynamic$ pmap $(pidof hello-static)\n9714: ./hello-static\n0000000000400000 876K r-x-- hello-static\n00000000006db000 24K rw--- hello-static\n00000000006e1000 4K rw--- [ anon ]\n00000000017b5000 140K rw--- [ anon ]\n00007ffc6f1d6000 132K rw--- [ stack ]\n00007ffc6f1f9000 12K r---- [ anon ]\n00007ffc6f1fc000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 1196K\n\nstudent@os:~/.../lab/support/static-dynamic$ pmap $(pidof hello-dynamic)\n9753: ./hello-dynamic\n00005566e757f000 8K r-x-- hello-dynamic\n00005566e7780000 4K r---- hello-dynamic\n00005566e7781000 4K rw--- hello-dynamic\n00005566e8894000 132K rw--- [ anon ]\n00007fd434eb8000 1948K r-x-- libc-2.27.so\n00007fd43509f000 2048K ----- libc-2.27.so\n00007fd43529f000 16K r---- libc-2.27.so\n00007fd4352a3000 8K rw--- libc-2.27.so\n00007fd4352a5000 16K rw--- [ anon ]\n00007fd4352a9000 164K r-x-- ld-2.27.so\n00007fd43549f000 8K rw--- [ anon ]\n00007fd4354d2000 4K r---- ld-2.27.so\n00007fd4354d3000 4K rw--- ld-2.27.so\n00007fd4354d4000 4K rw--- [ anon ]\n00007ffe497ba000 132K rw--- [ stack ]\n00007ffe497e3000 12K r---- [ anon ]\n00007ffe497e6000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 4520K\n")),(0,l.kt)("p",null,"For the static executable, we can see there are no areas for dynamic libraries.\nAnd the ",(0,l.kt)("inlineCode",{parentName:"p"},".rodata")," section has been coalesced in the ",(0,l.kt)("inlineCode",{parentName:"p"},".text")," area."),(0,l.kt)("p",null,"We can see the size of each section in the two executables by using the ",(0,l.kt)("inlineCode",{parentName:"p"},"size")," command:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/static-dynamic$ size hello-static\ntext data bss dec hex filename\n893333 20996 7128 921457 e0f71 hello-static\n\nstudent@os:~/.../lab/support/static-dynamic$ size hello-dynamic\ntext data bss dec hex filename\n4598 736 824 6158 180e hello-dynamic\n")),(0,l.kt)("h3",{id:"quiz"},"Quiz"),(0,l.kt)("p",null,"Based on the information above, answer ",(0,l.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/memory-granularity"},"this quiz"),"."),(0,l.kt)("h3",{id:"practice-1"},"Practice"),(0,l.kt)("ol",null,(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Let's investigate another static executable / process."),(0,l.kt)("p",{parentName:"li"},"If not already installed, install the ",(0,l.kt)("inlineCode",{parentName:"p"},"busybox-static")," package on your system.\nOn Debian/Ubuntu systems, use:"),(0,l.kt)("pre",{parentName:"li"},(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ sudo apt install busybox-static\n")),(0,l.kt)("p",{parentName:"li"},"Start a process using:"),(0,l.kt)("pre",{parentName:"li"},(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ busybox sleep 1000\n")),(0,l.kt)("p",{parentName:"li"},"Investigate the process using ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," and the executable using ",(0,l.kt)("inlineCode",{parentName:"p"},"size"),"."))),(0,l.kt)("h2",{id:"modifying-memory-region-size"},"Modifying Memory Region Size"),(0,l.kt)("p",null,"We want to observe the update in size of memory regions for different instructions used in a program."),(0,l.kt)("p",null,"Enter the ",(0,l.kt)("inlineCode",{parentName:"p"},"support/modify-areas/")," directory.\nBrowse the contents of the ",(0,l.kt)("inlineCode",{parentName:"p"},"hello.c")," file;\nit is an update to the ",(0,l.kt)("inlineCode",{parentName:"p"},"hello.c")," file in the ",(0,l.kt)("inlineCode",{parentName:"p"},"memory-areas/")," directory.\nBuild the executable:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/modify-areas$ make\n")),(0,l.kt)("p",null,"Use ",(0,l.kt)("inlineCode",{parentName:"p"},"size")," to view the difference between the new executable and the one in the ",(0,l.kt)("inlineCode",{parentName:"p"},"memory-areas/")," directory:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/modify-areas$ size hello\n text data bss dec hex filename\n 13131 17128 33592 63851 f96b hello\n\nstudent@os:~/.../lab/support/modify-areas$ size ../memory-areas/hello\n text data bss dec hex filename\n 4598 736 824 6158 180e ../memory-areas/hello\n")),(0,l.kt)("p",null,"Explain the differences."),(0,l.kt)("p",null,"Then use the ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," to watch the memory areas of the resulting processes from the two different executables.\nWe will see something like this for the new executable:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/modify-areas$ pmap $(pidof hello)\n18254: ./hello\n000055beff4d0000 16K r-x-- hello\n000055beff6d3000 4K r---- hello\n000055beff6d4000 20K rw--- hello\n000055beff6d9000 32K rw--- [ anon ]\n000055beffb99000 324K rw--- [ anon ]\n00007f7b6c2e6000 1948K r-x-- libc-2.27.so\n00007f7b6c4cd000 2048K ----- libc-2.27.so\n00007f7b6c6cd000 16K r---- libc-2.27.so\n00007f7b6c6d1000 8K rw--- libc-2.27.so\n00007f7b6c6d3000 16K rw--- [ anon ]\n00007f7b6c6d7000 164K r-x-- ld-2.27.so\n00007f7b6c8cd000 8K rw--- [ anon ]\n00007f7b6c900000 4K r---- ld-2.27.so\n00007f7b6c901000 4K rw--- ld-2.27.so\n00007f7b6c902000 4K rw--- [ anon ]\n00007ffe2b196000 204K rw--- [ stack ]\n00007ffe2b1d8000 12K r---- [ anon ]\n00007ffe2b1db000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 4840K\n")),(0,l.kt)("p",null,"We notice the size increase of text, data, bss, heap and stack sections."),(0,l.kt)("h3",{id:"practice-2"},"Practice"),(0,l.kt)("ol",null,(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Comment out different parts of the ",(0,l.kt)("inlineCode",{parentName:"p"},"hello.c")," program to notice differences in only specific areas (text, data, bss, heap, stack).")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Use a different argument (",(0,l.kt)("inlineCode",{parentName:"p"},"order"),") for the call to the ",(0,l.kt)("inlineCode",{parentName:"p"},"alloc_stack()")," function.\nSee how it affects the stack size during runtime (investigate with ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap"),").")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Do a static build of ",(0,l.kt)("inlineCode",{parentName:"p"},"hello.c")," and check the size of the memory areas, both statically and dynamically.")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"The ",(0,l.kt)("inlineCode",{parentName:"p"},"extend_mem_area.py")," Python script allocates a new string at each step by merging the two previous versions.\nStart the program and investigate the resulting process at each allocation step.\nNotice which memory area is updated and explain why."))),(0,l.kt)("p",null,(0,l.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/page-allocation"},"Quiz")),(0,l.kt)("h2",{id:"allocating-and-deallocating-memory"},"Allocating and Deallocating Memory"),(0,l.kt)("p",null,"Memory areas in a process address space are static or dynamic.\nStatic memory areas are known at the beginning of process lifetime (i.e. at load-time), while dynamic memory areas are managed at runtime."),(0,l.kt)("p",null,(0,l.kt)("inlineCode",{parentName:"p"},".text"),", ",(0,l.kt)("inlineCode",{parentName:"p"},".rodata"),", ",(0,l.kt)("inlineCode",{parentName:"p"},".data"),", ",(0,l.kt)("inlineCode",{parentName:"p"},".bss")," are allocated at load-time and have a predefined size.\nThe stack and the heap and memory mappings are allocated at runtime and have a variable size.\nFor those, we say we use runtime allocation and deallocation."),(0,l.kt)("p",null,"Memory allocation is implicit for the stack and explicit for the heap.\nThat is, we don't make a particular call to allocate data on the stack;\nthe compiler generates the code that the operating system uses to increase the stack when required.\nFor the heap, we use the ",(0,l.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,l.kt)("inlineCode",{parentName:"p"},"free()")," calls to explicitly allocate and deallocate memory."),(0,l.kt)("p",null,"Omitting to deallocate memory results in memory leaks that hurt the resource use in the system.\nBecause of this, some language runtimes employ a garbage collector that automatically frees unused memory areas.\nMore than that, some languages (think of Python) provide no explicit means to allocate memory: you just define and use data."),(0,l.kt)("p",null,"Let's enter the ",(0,l.kt)("inlineCode",{parentName:"p"},"support/alloc_size/")," directory.\nBrowse the ",(0,l.kt)("inlineCode",{parentName:"p"},"alloc_size.c")," file.\nBuild it:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/alloc_size$ make\n")),(0,l.kt)("p",null,"Now see the update in the process layout, by running the program in one console:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/alloc_size$ ./alloc_size\nPress key to allocate ...\n[...]\n")),(0,l.kt)("p",null,"And investigating it with ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," on another console:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)\n21107: ./alloc_size\n000055de9d173000 8K r-x-- alloc_size\n000055de9d374000 4K r---- alloc_size\n000055de9d375000 4K rw--- alloc_size\n000055de9deea000 132K rw--- [ anon ]\n00007f1ea4fd4000 1948K r-x-- libc-2.27.so\n00007f1ea51bb000 2048K ----- libc-2.27.so\n00007f1ea53bb000 16K r---- libc-2.27.so\n00007f1ea53bf000 8K rw--- libc-2.27.so\n00007f1ea53c1000 16K rw--- [ anon ]\n00007f1ea53c5000 164K r-x-- ld-2.27.so\n00007f1ea55bb000 8K rw--- [ anon ]\n00007f1ea55ee000 4K r---- ld-2.27.so\n00007f1ea55ef000 4K rw--- ld-2.27.so\n00007f1ea55f0000 4K rw--- [ anon ]\n00007ffcf28e9000 132K rw--- [ stack ]\n00007ffcf29be000 12K r---- [ anon ]\n00007ffcf29c1000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 4520K\n\nstudent@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)\n21107: ./alloc_size\n000055de9d173000 8K r-x-- alloc_size\n000055de9d374000 4K r---- alloc_size\n000055de9d375000 4K rw--- alloc_size\n000055de9deea000 452K rw--- [ anon ]\n00007f1ea4fd4000 1948K r-x-- libc-2.27.so\n00007f1ea51bb000 2048K ----- libc-2.27.so\n00007f1ea53bb000 16K r---- libc-2.27.so\n00007f1ea53bf000 8K rw--- libc-2.27.so\n00007f1ea53c1000 16K rw--- [ anon ]\n00007f1ea53c5000 164K r-x-- ld-2.27.so\n00007f1ea55bb000 8K rw--- [ anon ]\n00007f1ea55ee000 4K r---- ld-2.27.so\n00007f1ea55ef000 4K rw--- ld-2.27.so\n00007f1ea55f0000 4K rw--- [ anon ]\n00007ffcf28e9000 132K rw--- [ stack ]\n00007ffcf29be000 12K r---- [ anon ]\n00007ffcf29c1000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 4840K\n\nstudent@os:~/.../lab/support/alloc_size$ pmap $(pidof alloc_size)\n21107: ./alloc_size\n000055de9d173000 8K r-x-- alloc_size\n000055de9d374000 4K r---- alloc_size\n000055de9d375000 4K rw--- alloc_size\n000055de9deea000 420K rw--- [ anon ]\n00007f1ea4fd4000 1948K r-x-- libc-2.27.so\n00007f1ea51bb000 2048K ----- libc-2.27.so\n00007f1ea53bb000 16K r---- libc-2.27.so\n00007f1ea53bf000 8K rw--- libc-2.27.so\n00007f1ea53c1000 16K rw--- [ anon ]\n00007f1ea53c5000 164K r-x-- ld-2.27.so\n00007f1ea55bb000 8K rw--- [ anon ]\n00007f1ea55ee000 4K r---- ld-2.27.so\n00007f1ea55ef000 4K rw--- ld-2.27.so\n00007f1ea55f0000 4K rw--- [ anon ]\n00007ffcf28e9000 132K rw--- [ stack ]\n00007ffcf29be000 12K r---- [ anon ]\n00007ffcf29c1000 4K r-x-- [ anon ]\nffffffffff600000 4K --x-- [ anon ]\n total 4808K\n")),(0,l.kt)("p",null,"The three runs above of the ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," command occur before the allocation, after allocation and before deallocation and after deallocation.\nNotice the update toe the 4th section, the heap."),(0,l.kt)("p",null,"Now, let's see what happens behind the scenes.\nRun the executable under ",(0,l.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,l.kt)("inlineCode",{parentName:"p"},"strace"),":"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/alloc_size$ ltrace ./alloc_size\nmalloc(32768) = 0x55e33f490b10\nprintf("New allocation at %p\\n", 0x55e33f490b10New allocation at 0x55e33f490b10\n) = 33\n[...]\nfree(0x55e33f490b10) = \n[...]\n\nstudent@os:~/.../lab/support/alloc_size$ strace ./alloc_size\n[...]\nwrite(1, "New allocation at 0x55ab98acfaf0"..., 33New allocation at 0x55ab98acfaf0\n) = 33\nwrite(1, "New allocation at 0x55ab98ad7b00"..., 33New allocation at 0x55ab98ad7b00\n) = 33\nbrk(0x55ab98b08000) = 0x55ab98b08000\nwrite(1, "New allocation at 0x55ab98adfb10"..., 33New allocation at 0x55ab98adfb10\n) = 33\nwrite(1, "Press key to deallocate ...", 27Press key to deallocate ...) = 27\nread(0,\n"\\n", 1024) = 1\nbrk(0x55ab98b00000) = 0x55ab98b00000\n[...]\n')),(0,l.kt)("p",null,"The resulting output above shows us the following:"),(0,l.kt)("ul",null,(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"malloc()")," and ",(0,l.kt)("inlineCode",{parentName:"li"},"free()")," library calls both map to the ",(0,l.kt)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man2/sbrk.2.html"},(0,l.kt)("inlineCode",{parentName:"a"},"brk")," syscall"),", a syscall that updates the end of the heap (called ",(0,l.kt)("strong",{parentName:"li"},"program break"),")."),(0,l.kt)("li",{parentName:"ul"},"Multiple ",(0,l.kt)("inlineCode",{parentName:"li"},"malloc()")," calls map to a single ",(0,l.kt)("inlineCode",{parentName:"li"},"brk")," syscall for efficiency.\n",(0,l.kt)("inlineCode",{parentName:"li"},"brk")," is called to preallocate a larger chunk of memory that ",(0,l.kt)("inlineCode",{parentName:"li"},"malloc")," will then use.")),(0,l.kt)("p",null,"Update the ",(0,l.kt)("inlineCode",{parentName:"p"},"ALLOC_SIZE_KB")," macro in the ",(0,l.kt)("inlineCode",{parentName:"p"},"alloc_size.c")," file to ",(0,l.kt)("inlineCode",{parentName:"p"},"256"),".\nRebuild the program and rerun it under ",(0,l.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,l.kt)("inlineCode",{parentName:"p"},"strace"),":"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/alloc_size$ ltrace ./alloc_size\n[...]\nmalloc(262144) = 0x7f4c016a9010\n[...]\nfree(0x7f4c016a9010) = \n[...]\n\nstudent@os:~/.../lab/support/alloc_size$ strace ./alloc_size\n[...]\nmmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feee19f2000\nwrite(1, "New allocation at 0x7feee19f2010"..., 33New allocation at 0x7feee19f2010\n) = 33\nmmap(NULL, 266240, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7feee19b1000\nwrite(1, "New allocation at 0x7feee19b1010"..., 33New allocation at 0x7feee19b1010\n) = 33\nwrite(1, "Press key to deallocate ...", 27Press key to deallocate ...) = 27\nread(0,\n"\\n", 1024) = 1\nmunmap(0x7feee19b1000, 266240) = 0\n[...]\n')),(0,l.kt)("p",null,"For the new allocation size, notice that the remarks above don't hold:"),(0,l.kt)("ul",null,(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"malloc()")," now invokes the ",(0,l.kt)("inlineCode",{parentName:"li"},"mmap")," syscall, while ",(0,l.kt)("inlineCode",{parentName:"li"},"free()")," invokes the ",(0,l.kt)("inlineCode",{parentName:"li"},"munmap")," syscall."),(0,l.kt)("li",{parentName:"ul"},"Each ",(0,l.kt)("inlineCode",{parentName:"li"},"malloc()")," calls results in a separate ",(0,l.kt)("inlineCode",{parentName:"li"},"mmap")," syscall.")),(0,l.kt)("p",null,"This is a behavior of the ",(0,l.kt)("inlineCode",{parentName:"p"},"malloc()")," in libc, documented in the ",(0,l.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/malloc.3.html#NOTES"},"manual page"),".\nA variable ",(0,l.kt)("inlineCode",{parentName:"p"},"MALLOC_THRESHOLD")," holds the size after which ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap")," is used, instead of ",(0,l.kt)("inlineCode",{parentName:"p"},"brk"),".\nThis is based on a heuristic of using the heap or some other area in the process address space."),(0,l.kt)("h3",{id:"practice-3"},"Practice"),(0,l.kt)("ol",null,(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Use ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," to analyze the process address space for ",(0,l.kt)("inlineCode",{parentName:"p"},"ALLOC_SIZE_KB")," initialized to ",(0,l.kt)("inlineCode",{parentName:"p"},"256"),".\nNotice the new memory areas and the difference between the use of ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap")," syscall and ",(0,l.kt)("inlineCode",{parentName:"p"},"brk")," syscall.")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Use ",(0,l.kt)("inlineCode",{parentName:"p"},"valgrind")," on the resulting executable, and notice there are memory leaks.\nThey are quite obvious due to the lack of proper freeing.\nSolve the leaks.")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Use ",(0,l.kt)("inlineCode",{parentName:"p"},"valgrind")," on different executables in the system (in ",(0,l.kt)("inlineCode",{parentName:"p"},"/bin/"),", ",(0,l.kt)("inlineCode",{parentName:"p"},"/usr/bin/"),") and see if they have memory leaks."))),(0,l.kt)("p",null,(0,l.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/malloc-brk"},"Quiz")),(0,l.kt)("p",null,(0,l.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/malloc-mmap"},"Quiz")),(0,l.kt)("h2",{id:"memory-mapping"},"Memory Mapping"),(0,l.kt)("p",null,"The ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap")," syscall is used to allocate memory as ",(0,l.kt)("em",{parentName:"p"},"anonymous mapping"),", that is reserving memory in the process address space.\nAn alternate use is for mapping files in the memory address space.\nMapping of files is done by the loader for executables and libraries.\nThat is why, in the output of ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap"),", there is a column with a filename."),(0,l.kt)("p",null,"Mapping of a file results in getting a pointer to its contents and then using that pointer.\nThis way, reading and writing to a file is an exercise of pointer copying, instead of the use of ",(0,l.kt)("inlineCode",{parentName:"p"},"read")," / ",(0,l.kt)("inlineCode",{parentName:"p"},"write"),"-like system calls."),(0,l.kt)("p",null,"In the ",(0,l.kt)("inlineCode",{parentName:"p"},"support/copy/")," folder, there are two source code files and two scripts:"),(0,l.kt)("ul",null,(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"read_write_copy.c")," implements copying with ",(0,l.kt)("inlineCode",{parentName:"li"},"read")," / ",(0,l.kt)("inlineCode",{parentName:"li"},"write")," syscalls"),(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"mmap_copy.c")," implements copying using ",(0,l.kt)("inlineCode",{parentName:"li"},"mmap")),(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"generate.sh")," script generates the input file ",(0,l.kt)("inlineCode",{parentName:"li"},"in.dat")),(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"benchmark_cp.sh")," script runs the two executables ",(0,l.kt)("inlineCode",{parentName:"li"},"mmap_copy")," and ",(0,l.kt)("inlineCode",{parentName:"li"},"read_write_copy"))),(0,l.kt)("p",null,"Open the two source code files and investigate them.\nYou will notice that the ",(0,l.kt)("inlineCode",{parentName:"p"},"open()")," system call has the following prototype ",(0,l.kt)("inlineCode",{parentName:"p"},"int open(const char *pathname, int flags)"),".\nThe argument ",(0,l.kt)("inlineCode",{parentName:"p"},"flags")," must include one of the following access modes: ",(0,l.kt)("inlineCode",{parentName:"p"},"O_RDONLY"),", ",(0,l.kt)("inlineCode",{parentName:"p"},"O_WRONLY"),", or ",(0,l.kt)("inlineCode",{parentName:"p"},"O_RDWR")," - indicating that the file is opened in read-only, write-only, or read/write mode.\nYou can add an additional flag - ",(0,l.kt)("inlineCode",{parentName:"p"},"O_CREAT")," - that will create a new file with ",(0,l.kt)("inlineCode",{parentName:"p"},"pathname")," if the file does not already exist.\nThis is only the case when opening the file for writing (",(0,l.kt)("inlineCode",{parentName:"p"},"O_WRONLY")," or ",(0,l.kt)("inlineCode",{parentName:"p"},"O_RDWR"),").\nIf ",(0,l.kt)("inlineCode",{parentName:"p"},"O_CREAT")," is set, a third argument ",(0,l.kt)("inlineCode",{parentName:"p"},"mode_t mode")," is required for the ",(0,l.kt)("inlineCode",{parentName:"p"},"open()")," syscall.\nThe ",(0,l.kt)("inlineCode",{parentName:"p"},"mode")," argument specifies the permissions of the newly created file.\nFor example:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-c"},"// If DST_FILENAME exists it will be open in read/write mode and truncated to length 0\n// If DST_FILENAME does not exist, a file at the path DST_FILENAME will be create with 644 permissions\ndst_fd = open(DST_FILENAME, O_RDWR | O_CREAT | O_TRUNC, 0644);\n")),(0,l.kt)("p",null,"Let's generate the input file:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/copy$ ./generate.sh\n")),(0,l.kt)("p",null,"and let's build the two executable files:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/copy$ make\n")),(0,l.kt)("p",null,"Run the ",(0,l.kt)("inlineCode",{parentName:"p"},"benchmark_cp.sh")," script:"),(0,l.kt)("p",null,"Run the script in your local environment, not in the Docker container.\nDocker does not have permission to write to ",(0,l.kt)("inlineCode",{parentName:"p"},"/proc/sys/vm/drop_caches")," file."),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/copy$ ./benchmark_cp.sh\nBenchmarking mmap_copy on in.dat\ntime passed 54015 microseconds\n\nBenchmarking read_write_copy on in.dat\ntime passed 42011 microseconds\n")),(0,l.kt)("p",null,"Run the script a few more times.\nAs you can see, there isn't much of a difference between the two approaches.\nAlthough we would have expected the use of multiple system calls to cause overhead, it's too little compared to the memory copying overhead."),(0,l.kt)("p",null,"If you inspect ",(0,l.kt)("inlineCode",{parentName:"p"},"benchmark_cp.sh"),", you will notice a weird-looking command ",(0,l.kt)("inlineCode",{parentName:"p"},'sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"'),".\nThis is used to disable a memory optimization that the kernel does.\nIt's called \"buffer cache\" and it's a mechanism by which the kernel caches data blocks from recently accessed files in memory.\nYou will get more detailed information about this in the I/O chapter."),(0,l.kt)("p",null,"Browse the two source code files (",(0,l.kt)("inlineCode",{parentName:"p"},"mmap_copy.c")," and ",(0,l.kt)("inlineCode",{parentName:"p"},"read_write_copy.c"),") for a glimpse on how the two types of copies are implemented."),(0,l.kt)("p",null,(0,l.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/mmap-file"},"Quiz")),(0,l.kt)("h3",{id:"practice-4"},"Practice"),(0,l.kt)("ol",null,(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Use a different value for ",(0,l.kt)("inlineCode",{parentName:"p"},"BUFSIZE")," and see if that affects the comparison between the two executables.")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"Add a ",(0,l.kt)("inlineCode",{parentName:"p"},"sleep()")," call to the ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap_copy.c")," file ",(0,l.kt)("strong",{parentName:"p"},"after")," the files were mapped.\nRebuild the program and run it.\nOn a different console, use ",(0,l.kt)("inlineCode",{parentName:"p"},"pmap")," to view the two new memory regions that were added to the process, by mapping the ",(0,l.kt)("inlineCode",{parentName:"p"},"in.dat")," and ",(0,l.kt)("inlineCode",{parentName:"p"},"out.dat")," files."))))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0438abee.bae04586.js b/17/assets/js/0438abee.bae04586.js new file mode 100644 index 0000000000..aede9a644c --- /dev/null +++ b/17/assets/js/0438abee.bae04586.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2815],{3905:(e,t,n)=>{n.d(t,{Zo:()=>d,kt:()=>h});var i=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function r(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);t&&(i=i.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,i)}return n}function l(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var p=i.createContext({}),s=function(e){var t=i.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},d=function(e){var t=s(e.components);return i.createElement(p.Provider,{value:t},e.children)},c="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return i.createElement(i.Fragment,{},t)}},m=i.forwardRef((function(e,t){var n=e.components,a=e.mdxType,r=e.originalType,p=e.parentName,d=o(e,["components","mdxType","originalType","parentName"]),c=s(n),m=a,h=c["".concat(p,".").concat(m)]||c[m]||u[m]||r;return n?i.createElement(h,l(l({ref:t},d),{},{components:n})):i.createElement(h,l({ref:t},d))}));function h(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var r=n.length,l=new Array(r);l[0]=m;var o={};for(var p in t)hasOwnProperty.call(t,p)&&(o[p]=t[p]);o.originalType=e,o[c]="string"==typeof e?e:a,l[1]=o;for(var s=2;s{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>l,default:()=>u,frontMatter:()=>r,metadata:()=>o,toc:()=>s});var i=n(7462),a=(n(7294),n(3905));const r={},l="Redirections",o={unversionedId:"Lab/IO/redirections",id:"Lab/IO/redirections",title:"Redirections",description:"In the File Descriptors section, we mentioned redirections such as ls > file.txt.",source:"@site/docs/Lab/IO/redirections.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/redirections",permalink:"/operating-systems/17/Lab/IO/redirections",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"File Descriptors",permalink:"/operating-systems/17/Lab/IO/file-descriptors"},next:{title:"Pipes",permalink:"/operating-systems/17/Lab/IO/pipes"}},p={},s=[{value:"Practice: Naive Redirection",id:"practice-naive-redirection",level:2},{value:"Practice: Thread-unsafe Redirection",id:"practice-thread-unsafe-redirection",level:2},{value:"Practice: Safe Redirection",id:"practice-safe-redirection",level:2},{value:"Practice: Mini-shell Reloaded",id:"practice-mini-shell-reloaded",level:2}],d={toc:s},c="wrapper";function u(e){let{components:t,...r}=e;return(0,a.kt)(c,(0,i.Z)({},d,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"redirections"},"Redirections"),(0,a.kt)("p",null,"In the ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/file-descriptors"},"File Descriptors section"),", we mentioned redirections such as ",(0,a.kt)("inlineCode",{parentName:"p"},"ls > file.txt"),".\nWe said ",(0,a.kt)("inlineCode",{parentName:"p"},"file.txt")," has to be opened at some point.\nLet's check that.\nWe'll use ",(0,a.kt)("inlineCode",{parentName:"p"},"strace"),", obviously, to look for ",(0,a.kt)("inlineCode",{parentName:"p"},"open()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"openat()")," syscalls."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/simple-file-handling$ strace -e open,openat ls > file.txt\nopenat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libselinux.so.1", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpcre2-8.so.0", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libdl.so.2", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/lib/x86_64-linux-gnu/libpthread.so.0", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/proc/filesystems", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, "/usr/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3\nopenat(AT_FDCWD, ".", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3\n')),(0,a.kt)("p",null,"This looks strange.\nWhere is the call to ",(0,a.kt)("inlineCode",{parentName:"p"},'openat(AT_FDCWD, "file.txt", ...)'),"?\nWell, if we look at the full ",(0,a.kt)("inlineCode",{parentName:"p"},"strace")," output, the first call is ",(0,a.kt)("inlineCode",{parentName:"p"},"execve()"),":"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/simple-file-handling$ strace ls > file.txt\nexecve("/usr/bin/ls", ["ls"], 0x7ffe550d59e0 /* 60 vars */) = 0\n[...]\n')),(0,a.kt)("p",null,(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/execve"},"Quiz")),(0,a.kt)("p",null,"So the ",(0,a.kt)("inlineCode",{parentName:"p"},"openat()")," syscalls we saw earlier come from the ",(0,a.kt)("inlineCode",{parentName:"p"},"ls")," process.\nRemember how launching a command works in Bash:"),(0,a.kt)("ol",null,(0,a.kt)("li",{parentName:"ol"},"The Bash process ",(0,a.kt)("inlineCode",{parentName:"li"},"fork()"),"s itself"),(0,a.kt)("li",{parentName:"ol"},"The child (still a Bash process) then calls ",(0,a.kt)("inlineCode",{parentName:"li"},"execve()")),(0,a.kt)("li",{parentName:"ol"},"The parent (the original Bash process) calls ",(0,a.kt)("inlineCode",{parentName:"li"},"waitpid()")," to wait for the new process to end.")),(0,a.kt)("p",null,(0,a.kt)("img",{alt:"Launching a new command in Bash",src:n(6766).Z,width:"483",height:"403"})),(0,a.kt)("p",null,"So we can deduce that ",(0,a.kt)("inlineCode",{parentName:"p"},"file.txt")," is opened by the child process ",(0,a.kt)("strong",{parentName:"p"},"before")," calling ",(0,a.kt)("inlineCode",{parentName:"p"},"execve()"),".\nNote that despite replacing the VAS of the current process, ",(0,a.kt)("inlineCode",{parentName:"p"},"execve()")," does ",(0,a.kt)("strong",{parentName:"p"},"NOT")," replace its file descriptor table.\nSo whatever files were opened by the original process remain open within the new one.\nThis behaviour is the basis of pipes (the ",(0,a.kt)("inlineCode",{parentName:"p"},"|")," that you use in Bash to use the output of a command as the input of another) and redirections (",(0,a.kt)("inlineCode",{parentName:"p"},">"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"<")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"2>"),")."),(0,a.kt)("h2",{id:"practice-naive-redirection"},"Practice: Naive Redirection"),(0,a.kt)("p",null,"But before diving into reimplementing shell functionalities, let's look at a simpler example.\nNavigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/redirect/redirect.c"),".\nThe code makes a naive attempt at redirecting the newly opened file to ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout"),".\nIt simply closes ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," first so that when ",(0,a.kt)("inlineCode",{parentName:"p"},"open()")," returns ",(0,a.kt)("strong",{parentName:"p"},"the lowest available file descriptor"),", that value will be 1, which is ",(0,a.kt)("inlineCode",{parentName:"p"},"STDOUT_FILENO"),"."),(0,a.kt)("p",null,"Note there's a difference between ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"STDOUT_FILENO"),".\nWhile ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," is of type ",(0,a.kt)("inlineCode",{parentName:"p"},"FILE *")," and is meant to be used with libc functions such as ",(0,a.kt)("inlineCode",{parentName:"p"},"fread()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"STDOUT_FILENO")," is the default file descriptor for the standard output, which almost always is 1.\nSo ",(0,a.kt)("inlineCode",{parentName:"p"},"STDOUT_FILENO")," is an ",(0,a.kt)("inlineCode",{parentName:"p"},"int")," type with the value 1.\nDon't confuse them!"),(0,a.kt)("p",null,"Compile and run the code without modifying it.\nIt pauses after each file descriptor operation.\nIn another terminal (or in another ",(0,a.kt)("inlineCode",{parentName:"p"},"tmux")," window), run the following each time you press Enter in the first terminal/window: ",(0,a.kt)("inlineCode",{parentName:"p"},"lsof -p $(pidof redirect)")," or try ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man1/watch.1.html"},(0,a.kt)("inlineCode",{parentName:"a"},"watch")),"-ing it."),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"lsof")," displays the files opened by the given process.\nFrom the output below, we see that these files are mainly files (opened as file descriptors) and libraries, which are memory mapped (",(0,a.kt)("inlineCode",{parentName:"p"},"mem"),").\nOn the third column, you can see the file descriptor corresponding to each file."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/redirect$ lsof -w -p $(pidof redirect) # before any file operations\nCOMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME\nredirect 44303 student cwd DIR 8,1 4096 299870 /home/student/operating-systems-oer/content/chapters/io/lab/support/redirect\nredirect 44303 student rtd DIR 259,6 4096 2 /\nredirect 44303 student txt REG 8,1 25848 299929 /home/student/operating-systems-oer/content/chapters/io/lab/support/redirect/redirect\nredirect 44303 student mem REG 259,6 2029592 1857435 /usr/lib/x86_64-linux-gnu/libc-2.31.so\nredirect 44303 student mem REG 259,6 191504 1835092 /usr/lib/x86_64-linux-gnu/ld-2.31.so\nredirect 44303 student 0u CHR 136,0 0t0 3 /dev/pts/0\nredirect 44303 student 1u CHR 136,0 0t0 3 /dev/pts/0\nredirect 44303 student 2u CHR 136,0 0t0 3 /dev/pts/0\n")),(0,a.kt)("p",null,"Notice that all 3 default file descriptors are first liked to ",(0,a.kt)("inlineCode",{parentName:"p"},"/dev/pts/0"),".\nIt may be different on your machine, but most likely it will be ",(0,a.kt)("inlineCode",{parentName:"p"},"/dev/pts/"),".\nThis is a ",(0,a.kt)("a",{parentName:"p",href:"./devices.md"},"character device")," that signifies your current (pseudo)terminal."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/redirect$ lsof -w -p $(pidof redirect) # after closing `STDOUT_FILENO`\nredirect 46906 student 0u CHR 136,0 0t0 3 /dev/pts/0\nredirect 46906 student 2u CHR 136,0 0t0 3 /dev/pts/0\n")),(0,a.kt)("p",null,"See that file descriptor 1 (",(0,a.kt)("inlineCode",{parentName:"p"},"stdout"),') has "disappeared".\nNow the second entry in the process\'s FD table is free.'),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/redirect$ lsof -w -p $(pidof redirect) # after opening `redirect_file.txt`\nredirect 46906 student 0u CHR 136,0 0t0 3 /dev/pts/0\nredirect 46906 student 1w REG 8,1 0 299958 /.../lab/support/redirect/redirect_file.txt\nredirect 46906 student 2u CHR 136,0 0t0 3 /dev/pts/0\n")),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"open()")," has assigned the newly opened file to the lowest file descriptor available, which is 1, the former ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout"),".\n",(0,a.kt)("inlineCode",{parentName:"p"},"printf()")," writes its output to ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout"),", which in this case is ",(0,a.kt)("inlineCode",{parentName:"p"},"redirect_file.txt"),".\nNow inspect the contents of this file to make sure that string was written there."),(0,a.kt)("p",null,(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/prints-work-no-stdio"},"Quiz")),(0,a.kt)("h2",{id:"practice-thread-unsafe-redirection"},"Practice: Thread-unsafe Redirection"),(0,a.kt)("p",null,"This is all fine, but it doesn't allow us to ",(0,a.kt)("strong",{parentName:"p"},"copy")," file descriptors.\nWe can only replace an existing file descriptor with another one.\nTo be able to do replacements, we can use the ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},(0,a.kt)("inlineCode",{parentName:"a"},"dup()"))," syscall.\nIt simply creates a new file descriptor that refers to the same open file ",(0,a.kt)("inlineCode",{parentName:"p"},"struct")," as the one given to it as an argument.\nBoth file descriptors remain active after calling ",(0,a.kt)("inlineCode",{parentName:"p"},"dup()"),"."),(0,a.kt)("p",null,"Change the ",(0,a.kt)("inlineCode",{parentName:"p"},"do_redirect()")," function in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/redirect/redirect.c")," to employ this new logic.\nIt should follow the steps below.\nTrack them using ",(0,a.kt)("inlineCode",{parentName:"p"},"lsof"),", just like before."),(0,a.kt)("p",null,"Step 1:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"0 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdin")),(0,a.kt)("li",{parentName:"ul"},"1 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdout")),(0,a.kt)("li",{parentName:"ul"},"2 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stderr"))),(0,a.kt)("p",null,"Step 2 - after ",(0,a.kt)("inlineCode",{parentName:"p"},'open("redirect_file.txt", ...)'),":"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"0 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdin")),(0,a.kt)("li",{parentName:"ul"},"1 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdout")),(0,a.kt)("li",{parentName:"ul"},"2 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stderr")),(0,a.kt)("li",{parentName:"ul"},"3 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"redirect_file.txt"))),(0,a.kt)("p",null,"Step 3 - after ",(0,a.kt)("inlineCode",{parentName:"p"},"close(STDOUT_FILENO)"),":"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"0 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdin")),(0,a.kt)("li",{parentName:"ul"},"2 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stderr")),(0,a.kt)("li",{parentName:"ul"},"3 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"redirect_file.txt"))),(0,a.kt)("p",null,"Step 4 - after ",(0,a.kt)("inlineCode",{parentName:"p"},"dup(3)"),".\nNote that now both 1 and 3 are linked to ",(0,a.kt)("inlineCode",{parentName:"p"},"redirect_file.txt"),", so we managed to successfully copy file descriptor 3."),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"0 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdin")),(0,a.kt)("li",{parentName:"ul"},"1 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"redirect_file.txt")),(0,a.kt)("li",{parentName:"ul"},"2 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stderr")),(0,a.kt)("li",{parentName:"ul"},"3 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"redirect_file.txt"))),(0,a.kt)("p",null,"Step 5 - after ",(0,a.kt)("inlineCode",{parentName:"p"},"close(3)"),".\nWe don't need file descriptor 3 at this point anymore."),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"0 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stdin")),(0,a.kt)("li",{parentName:"ul"},"1 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"redirect_file.txt")),(0,a.kt)("li",{parentName:"ul"},"2 -> ",(0,a.kt)("inlineCode",{parentName:"li"},"stderr"))),(0,a.kt)("h2",{id:"practice-safe-redirection"},"Practice: Safe Redirection"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"dup()")," is all fine and dandy, but what if 2 threads use the steps above concurrently?\nBecause steps 3 and 4 don't happen atomically, they risk having their results inverted.\nTake a look at ",(0,a.kt)("inlineCode",{parentName:"p"},"support/redirect/redirect_parallel.c"),".\nCompile and run the code, then inspect the resulting files.\nYou'll notice they contain opposing strings:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/redirect$ cat redirect_stderr_file.txt\nMessage for STDOUT\n\nstudent@os:~/.../lab/support/redirect$ cat redirect_stdout_file.txt\nMessage for STDERR\n")),(0,a.kt)("p",null,"What happens is that thread 1 is forced to call ",(0,a.kt)("inlineCode",{parentName:"p"},"close(STDOUT_FILENO)"),", then thread 2 calls ",(0,a.kt)("inlineCode",{parentName:"p"},"close(STDERR_FILENO)"),".\nSo far, this is not problematic.\nBut then thread 2 continues to ",(0,a.kt)("inlineCode",{parentName:"p"},"dup()")," ",(0,a.kt)("inlineCode",{parentName:"p"},"redirect_stderr.txt")," into the lowest file descriptor available, which is 1 (",(0,a.kt)("inlineCode",{parentName:"p"},"STDOUT_FILENO"),").\nThen thread 1 resumes to ",(0,a.kt)("inlineCode",{parentName:"p"},"dup()")," ",(0,a.kt)("inlineCode",{parentName:"p"},"redirect_stdout.txt")," into file descriptor 2 (",(0,a.kt)("inlineCode",{parentName:"p"},"STDERR_FILENO"),").\nThus we end up redirecting ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"stderr")," to the opposite files than those we intended."),(0,a.kt)("p",null,"To fix this, we need to call an ",(0,a.kt)("strong",{parentName:"p"},"atomic")," syscall, called ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},(0,a.kt)("inlineCode",{parentName:"a"},"dup2()")),".\nIt receives 2 file descriptors (",(0,a.kt)("inlineCode",{parentName:"p"},"dup2(src_fd, dst_fd)"),") and its actions are equivalent to the following, but performed atomically:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},"close(dst_fd);\ndup(src_fd); // This places `src_fd` into the previous `dst_fd`\n")),(0,a.kt)("ol",null,(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Modify ",(0,a.kt)("inlineCode",{parentName:"p"},"support/redirect/redirect_parallel.c")," and change the calls to ",(0,a.kt)("inlineCode",{parentName:"p"},"close()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"dup()")," to calls to ",(0,a.kt)("inlineCode",{parentName:"p"},"dup2()")," and check the contents of the resulting files to see they're correct.")),(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Now go back to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/redirect/redirect.c")," and refactor the code in ",(0,a.kt)("inlineCode",{parentName:"p"},"do_redirect()")," to use ",(0,a.kt)("inlineCode",{parentName:"p"},"dup2()")," as well."))),(0,a.kt)("h2",{id:"practice-mini-shell-reloaded"},"Practice: Mini-shell Reloaded"),(0,a.kt)("p",null,"Remember the mini-shell you implemented in the Arena of the previous lab.\nIt is capable of ",(0,a.kt)("inlineCode",{parentName:"p"},"fork()"),"-ing itself and ",(0,a.kt)("inlineCode",{parentName:"p"},"execvp()"),"-ing commands, just like Bash.\nWe can now extend it to allow redirecting ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," to a file."),(0,a.kt)("p",null,"Use what you've learnt so far in this section to allow this mini-shell to redirect the output of its commands to files.\nRedirection is performed just like in bash via ",(0,a.kt)("inlineCode",{parentName:"p"},">"),"."))}u.isMDXComponent=!0},6766:(e,t,n)=>{n.d(t,{Z:()=>i});const i=n.p+"assets/images/fork-exec-e8ff2e7cb057592463ccc850bdaa0228.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/04d8a21d.9483629a.js b/17/assets/js/04d8a21d.9483629a.js new file mode 100644 index 0000000000..28a9890379 --- /dev/null +++ b/17/assets/js/04d8a21d.9483629a.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[983],{3905:(e,t,n)=>{n.d(t,{Zo:()=>s,kt:()=>k});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function l(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),u=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},s=function(e){var t=u(e.components);return r.createElement(c.Provider,{value:t},e.children)},p="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},m=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,l=e.originalType,c=e.parentName,s=o(e,["components","mdxType","originalType","parentName"]),p=u(n),m=a,k=p["".concat(c,".").concat(m)]||p[m]||d[m]||l;return n?r.createElement(k,i(i({ref:t},s),{},{components:n})):r.createElement(k,i({ref:t},s))}));function k(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var l=n.length,i=new Array(l);i[0]=m;var o={};for(var c in t)hasOwnProperty.call(t,c)&&(o[c]=t[c]);o.originalType=e,o[p]="string"==typeof e?e:a,i[1]=o;for(var u=2;u{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>d,frontMatter:()=>l,metadata:()=>o,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const l={},i="libc",o={unversionedId:"Lab/Software Stack/quiz/libc",id:"Lab/Software Stack/quiz/libc",title:"libc",description:"malloc()",source:"@site/docs/Lab/Software Stack/quiz/libc.md",sourceDirName:"Lab/Software Stack/quiz",slug:"/Lab/Software Stack/quiz/libc",permalink:"/operating-systems/17/Lab/Software Stack/quiz/libc",draft:!1,tags:[],version:"current",frontMatter:{}},c={},u=[{value:"malloc()",id:"malloc",level:2},{value:"Question Text",id:"question-text",level:3},{value:"Question Answers",id:"question-answers",level:3},{value:"Feedback",id:"feedback",level:3},{value:"Syscall Tool",id:"syscall-tool",level:2},{value:"Question Text",id:"question-text-1",level:3},{value:"Question Answers",id:"question-answers-1",level:3},{value:"Feedback",id:"feedback-1",level:3}],s={toc:u},p="wrapper";function d(e){let{components:t,...n}=e;return(0,a.kt)(p,(0,r.Z)({},s,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"libc"},"libc"),(0,a.kt)("h2",{id:"malloc"},(0,a.kt)("inlineCode",{parentName:"h2"},"malloc()")),(0,a.kt)("h3",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What system calls are invoked by the ",(0,a.kt)("inlineCode",{parentName:"p"},"malloc()")," library call for Linux libc? (choose 2 answers)"),(0,a.kt)("h3",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"brk"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"free"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"dup")))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"mmap"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"copy"))),(0,a.kt)("h3",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Depending on the allocation size, ",(0,a.kt)("inlineCode",{parentName:"p"},"malloc()")," invokes ",(0,a.kt)("inlineCode",{parentName:"p"},"brk")," or ",(0,a.kt)("inlineCode",{parentName:"p"},"mmap"),"."),(0,a.kt)("h2",{id:"syscall-tool"},"Syscall Tool"),(0,a.kt)("h3",{id:"question-text-1"},"Question Text"),(0,a.kt)("p",null,"Which of following is ",(0,a.kt)("strong",{parentName:"p"},"not")," and advantage of using libc for programs?"),(0,a.kt)("h3",{id:"question-answers-1"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"increased portability")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"reduced executable size")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"richer set of features")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"easier development"))),(0,a.kt)("h3",{id:"feedback-1"},"Feedback"),(0,a.kt)("p",null,"When using libc, because we add a new software component, the size of the resulting executable increases."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/092edce8.9cfa9d9f.js b/17/assets/js/092edce8.9cfa9d9f.js new file mode 100644 index 0000000000..d91951036c --- /dev/null +++ b/17/assets/js/092edce8.9cfa9d9f.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9837],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>m});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function o(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var s=n.createContext({}),d=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):o(o({},t),e)),r},p=function(e){var t=d(e.components);return n.createElement(s.Provider,{value:t},e.children)},u="mdxType",h={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},c=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,i=e.originalType,s=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),u=d(r),c=a,m=u["".concat(s,".").concat(c)]||u[c]||h[c]||i;return r?n.createElement(m,o(o({ref:t},p),{},{components:r})):n.createElement(m,o({ref:t},p))}));function m(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=r.length,o=new Array(i);o[0]=c;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[u]="string"==typeof e?e:a,o[1]=l;for(var d=2;d{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>h,frontMatter:()=>i,metadata:()=>l,toc:()=>d});var n=r(7462),a=(r(7294),r(3905));const i={},o="User-Level Threads",l={unversionedId:"Lab/Compute/user-level-threads",id:"Lab/Compute/user-level-threads",title:"User-Level Threads",description:"User-level threads differ from the threads you are used to (kernel-level threads, those created by pthread_create).",source:"@site/docs/Lab/Compute/user-level-threads.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/user-level-threads",permalink:"/operating-systems/17/Lab/Compute/user-level-threads",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Synchronization",permalink:"/operating-systems/17/Lab/Compute/synchronization"},next:{title:"Arena",permalink:"/operating-systems/17/Lab/Compute/arena"}},s={},d=[{value:"Prerequisites",id:"prerequisites",level:2},{value:"Creation",id:"creation",level:2},{value:"Practice: Sleeper Fiber",id:"practice-sleeper-fiber",level:3},{value:"No system calls",id:"no-system-calls",level:2},{value:"Synchronization",id:"synchronization",level:2},{value:"Yielding",id:"yielding",level:3},{value:"Practice",id:"practice",level:4},{value:"Barriers",id:"barriers",level:3},{value:"Interaction Between Threads and Fibers",id:"interaction-between-threads-and-fibers",level:2},{value:"C++ unique_lock",id:"c-unique_lock",level:3}],p={toc:d},u="wrapper";function h(e){let{components:t,...r}=e;return(0,a.kt)(u,(0,n.Z)({},p,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"user-level-threads"},"User-Level Threads"),(0,a.kt)("p",null,"User-level threads differ from the threads you are used to (kernel-level threads, those created by pthread_create).\nThis kind of threads are scheduled by an user-level scheduler, and can run on the same kernel-level thread.\nFrom now on, we will reffer to user-level threads as fibers, and kernel-level threads as simply threads."),(0,a.kt)("p",null,"We will use the fiber implementation from libboost.\nThis implementation uses a cooperative scheduler on each thread, meaning that each fiber has to yield, in order for other fiber to be executed.\nWe will also use C++, and the standard ",(0,a.kt)("inlineCode",{parentName:"p"},"thread")," implementation."),(0,a.kt)("h2",{id:"prerequisites"},"Prerequisites"),(0,a.kt)("p",null,"Unless you are using the OS docker image, you will need to install ",(0,a.kt)("inlineCode",{parentName:"p"},"cmake")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"libboost"),".\nYou can do this with the following command:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ sudo apt-get install cmake libboost-context-dev libboost-fiber-dev\n")),(0,a.kt)("h2",{id:"creation"},"Creation"),(0,a.kt)("p",null,"Follow the ",(0,a.kt)("inlineCode",{parentName:"p"},"support/user-level-threads/simple.cc")," implementation.\nIt creates ",(0,a.kt)("inlineCode",{parentName:"p"},"NUM_FIBERS"),' fibers, that each prints "Hello World".\nTo compile and run the program, do the following steps:'),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/user-level-threads$ mkdir build/\nstudent@os:~/.../lab/support/user-level-threads$ cd build/\nstudent@os:~/.../lab/support/user-level-threads$ cmake -S .. -B .\nstudent@os:~/.../lab/support/user-level-threads$ make\nstudent@os:~/.../lab/support/user-level-threads$ ./simple\n")),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"cmake")," step must be executed only once.\nAfter modifying the source files, it is enough to run ",(0,a.kt)("inlineCode",{parentName:"p"},"make"),"."),(0,a.kt)("h3",{id:"practice-sleeper-fiber"},"Practice: Sleeper Fiber"),(0,a.kt)("p",null,"Add in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/user-level-threads/simple.cc")," a fiber that sleeps for 5 seconds, before the other ones are created.\nWhat happens?\nAnswer in this ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/sleeping-on-a-fiber"},"quiz"),"."),(0,a.kt)("h2",{id:"no-system-calls"},"No system calls"),(0,a.kt)("p",null,"Use ",(0,a.kt)("inlineCode",{parentName:"p"},"strace")," to find calls to ",(0,a.kt)("inlineCode",{parentName:"p"},"clone()")," in the execution of ",(0,a.kt)("inlineCode",{parentName:"p"},"simple"),".\nCan you find any?\nProvide your answer in this ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/fiber-strace"},"quiz"),"\nRemember that ",(0,a.kt)("inlineCode",{parentName:"p"},"clone()")," is the system call used to create ",(0,a.kt)("strong",{parentName:"p"},"kernel-level")," threads, as pointed out ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#threads-and-processes-clone"},"here"),"."),(0,a.kt)("h2",{id:"synchronization"},"Synchronization"),(0,a.kt)("p",null,"By default, the fibers that run on the same thread are synchronized - no race-conditions can occur.\nThis is illustrated by the ",(0,a.kt)("inlineCode",{parentName:"p"},"support/user-level-threads/sum.cc")," implementation."),(0,a.kt)("p",null,"The user can, however, implement further synchronization, by using the ",(0,a.kt)("inlineCode",{parentName:"p"},"yield()")," call, or classic synchronization methods, like mutexes, barriers and condition variables."),(0,a.kt)("h3",{id:"yielding"},"Yielding"),(0,a.kt)("p",null,"As the scheduler is cooperative, each fiber can yield (or not), to allow another fiber to run.\nFollow the ",(0,a.kt)("inlineCode",{parentName:"p"},"support/user-level-threads/yield_launch.cc")," implementation and run it.\nNote the ",(0,a.kt)("inlineCode",{parentName:"p"},"boost::fibers::launch::dispatch")," parameter provided to the fiber constructor.\nIt notifies the scheduler to start the fibre as soon as it is created.\nIn order to explain the output, we must consider that the fibers are created by a ",(0,a.kt)("strong",{parentName:"p"},"main fiber"),", that is scheduled along with the others, in this case."),(0,a.kt)("h4",{id:"practice"},"Practice"),(0,a.kt)("p",null,"Modify the launch parameter into ",(0,a.kt)("inlineCode",{parentName:"p"},"boost::fibers::launch::post"),", compile and notice the differences.\nThe ",(0,a.kt)("inlineCode",{parentName:"p"},"post")," parameter notifies the scheduler not to start the fibers imediately, but rather place them into an execution queue.\nTheir execution will start after the main fiber calls the ",(0,a.kt)("inlineCode",{parentName:"p"},"join()")," function."),(0,a.kt)("h3",{id:"barriers"},"Barriers"),(0,a.kt)("p",null,"Follow the ",(0,a.kt)("inlineCode",{parentName:"p"},"support/user-level-threads/yield_barrier.cc")," implementation.\nIt uses a barrier to achieve the same result as the previos implementation, that used ",(0,a.kt)("inlineCode",{parentName:"p"},"post")," as the launch parameter."),(0,a.kt)("h2",{id:"interaction-between-threads-and-fibers"},"Interaction Between Threads and Fibers"),(0,a.kt)("p",null,"As we mentioned before, multiple fibers can run on the same thread, and a scheduler is implemented on each thread.\nBy default, the scheduling algorithm is ",(0,a.kt)("a",{parentName:"p",href:"https://www.guru99.com/round-robin-scheduling-example.html"},(0,a.kt)("inlineCode",{parentName:"a"},"round_robin")),".\nIt runs the fibers, in the order of their creation, until they yield or finish their work.\nIf a fiber yields, it is placed at the back of the round-robin queue.\nUsing this scheduler, each thread only uses its fibers;\nif one thread has more work to do than another, bad luck.\nThis may lead to starvation."),(0,a.kt)("p",null,"But there are other scheduler implementations, such as ",(0,a.kt)("inlineCode",{parentName:"p"},"shared_work")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"work_stealing"),".\nFollow the ",(0,a.kt)("inlineCode",{parentName:"p"},"support/user-level-threads/threads_and_fibers.cc")," implementation.\nIt creates multiple fibers and threads, and uses the ",(0,a.kt)("inlineCode",{parentName:"p"},"shared_work")," scheduler to balance the workload between the threads.\nEach main fiber, from each thread, is suspended until all worker fibers have completed their work, using a condition variable."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-cpp"},"cnd_count.wait( lk, [](){ return 0 == fiber_count; } );\n")),(0,a.kt)("p",null,"The program also uses ",(0,a.kt)("inlineCode",{parentName:"p"},"thread local storage")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"fiber local storage")," to store the ID of each thread / fiber."),(0,a.kt)("p",null,"Now change the ",(0,a.kt)("inlineCode",{parentName:"p"},"shared_work")," scheduler into the ",(0,a.kt)("inlineCode",{parentName:"p"},"work_stealing")," one.\nIt takes a parameter, the number of threads that will use that scheduler."),(0,a.kt)("p",null,"Compile, rerun and note the differences.\nThe ",(0,a.kt)("inlineCode",{parentName:"p"},"work_stealing"),' scheduler, as the name suggests, will "steal" fibers from other schedulers.\nSo, if the ',(0,a.kt)("inlineCode",{parentName:"p"},"shared_work")," scheduler tried to balance the available work between the available threads, the ",(0,a.kt)("inlineCode",{parentName:"p"},"work_stealing")," one will focus on having as many threads as possible on 100% workload.\nVary the number of threads and fibers, and the workload (maybe put each fibre to do some computational-intensive work), and observe the results."),(0,a.kt)("h3",{id:"c-unique_lock"},"C++ unique_lock"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"unique_lock")," is a type of mutex that is unlocked automatically when the end of its scope is reached (end of function or bracket-pair)."))}h.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/09e11287.7699b2ae.js b/17/assets/js/09e11287.7699b2ae.js new file mode 100644 index 0000000000..afd2017bd1 --- /dev/null +++ b/17/assets/js/09e11287.7699b2ae.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7914],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>u});var a=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function r(e){for(var t=1;t=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var l=a.createContext({}),c=function(e){var t=a.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):r(r({},t),e)),n},p=function(e){var t=c(e.components);return a.createElement(l.Provider,{value:t},e.children)},h="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,i=e.mdxType,o=e.originalType,l=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),h=c(n),m=i,u=h["".concat(l,".").concat(m)]||h[m]||d[m]||o;return n?a.createElement(u,r(r({ref:t},p),{},{components:n})):a.createElement(u,r({ref:t},p))}));function u(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var o=n.length,r=new Array(o);r[0]=m;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[h]="string"==typeof e?e:i,r[1]=s;for(var c=2;c{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>r,default:()=>d,frontMatter:()=>o,metadata:()=>s,toc:()=>c});var a=n(7462),i=(n(7294),n(3905));const o={},r="Synchronization",s={unversionedId:"Lab/Compute/synchronization",id:"Lab/Compute/synchronization",title:"Synchronization",description:'So far, we\'ve used threads and processes without wondering how to "tell" them how to access shared data.',source:"@site/docs/Lab/Compute/synchronization.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/synchronization",permalink:"/operating-systems/17/Lab/Compute/synchronization",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Copy-on-Write",permalink:"/operating-systems/17/Lab/Compute/copy-on-write"},next:{title:"User-Level Threads",permalink:"/operating-systems/17/Lab/Compute/user-level-threads"}},l={},c=[{value:"Race Conditions",id:"race-conditions",level:2},{value:"Synchronization - Overhead",id:"synchronization---overhead",level:3},{value:"Practice: Wrap the Whole for Statements in Critical Sections",id:"practice-wrap-the-whole-for-statements-in-critical-sections",level:3},{value:"Atomics",id:"atomics",level:2},{value:"Semaphores",id:"semaphores",level:2},{value:"Practice: apache2 Simulator - Semaphore",id:"practice-apache2-simulator---semaphore",level:3},{value:"Conditions",id:"conditions",level:2},{value:"Practice: apache2 Simulator - Condition",id:"practice-apache2-simulator---condition",level:3},{value:"Thread-Local Storage (TLS)",id:"thread-local-storage-tls",level:2},{value:"Practice: C - TLS on Demand",id:"practice-c---tls-on-demand",level:3}],p={toc:c},h="wrapper";function d(e){let{components:t,...n}=e;return(0,i.kt)(h,(0,a.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"synchronization"},"Synchronization"),(0,i.kt)("p",null,'So far, we\'ve used threads and processes without wondering how to "tell" them how to access shared data.\nMoreover, in order to make threads wait for each other, we simply had the main thread wait for the others to finish all their work.\nBut what if we want one thread to wait until another one simply performs some specific action, after which it resumes its execution?\nFor this, we need to use some more complex synchronization mechanisms.'),(0,i.kt)("h2",{id:"race-conditions"},"Race Conditions"),(0,i.kt)("p",null,"For example, what if one thread wants to increase a global variable while another one wants to decrease it?\nLet's say the assembly code for increasing and decreasing the variable looks like the one in the snippet below."),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-asm"},"increase:\n mov eax, [var]\n inc eax\n mov [var], eax\n\ndecrease:\n mov eax, [var]\n dec eax\n mov [var], eax\n")),(0,i.kt)("p",null,"Imagine both threads executed ",(0,i.kt)("inlineCode",{parentName:"p"},"mov eax, [var]")," at the same time.\nThen each would independently increase its (",(0,i.kt)("strong",{parentName:"p"},"non-shared"),") ",(0,i.kt)("inlineCode",{parentName:"p"},"eax")," register.\nIn the end, the final value of ",(0,i.kt)("inlineCode",{parentName:"p"},"var")," depends on which thread executes ",(0,i.kt)("inlineCode",{parentName:"p"},"mov [var], eax")," ",(0,i.kt)("em",{parentName:"p"},"last"),'.\nSo it\'s kind of a reversed race.\nThe thread that runs the slowest "wins" this race and writes the final value of ',(0,i.kt)("inlineCode",{parentName:"p"},"var"),".\nBut this is up to the scheduler and is non-deterministic.\nSuch undefined behaviours can cripple the execution of a program if ",(0,i.kt)("inlineCode",{parentName:"p"},"var")," is some critical variable."),(0,i.kt)("p",null,"Let's see this bug in action.\nGo to ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition.c"),", compile and run the code a few times.\nIt spawns to threads that do exactly what we've talked about so far: one thread increments ",(0,i.kt)("inlineCode",{parentName:"p"},"var")," 10 million times, while the other decrements it 10 million times."),(0,i.kt)("p",null,"As you can see from running the program, the differences between subsequent runs can be substantial.\nTo fix this, we must ensure that ",(0,i.kt)("strong",{parentName:"p"},"only one thread")," can execute either ",(0,i.kt)("inlineCode",{parentName:"p"},"var++")," or ",(0,i.kt)("inlineCode",{parentName:"p"},"var--")," at any time.\nWe call these code sections ",(0,i.kt)("strong",{parentName:"p"},"critical sections"),".\nA critical section is a piece of code that can only be executed by ",(0,i.kt)("strong",{parentName:"p"},"one thread")," at a time.\nSo we need some sort of ",(0,i.kt)("em",{parentName:"p"},"mutual exclusion mechanism")," so that when one thread runs the critical section, the other has to ",(0,i.kt)("strong",{parentName:"p"},"wait")," before entering it.\nThis mechanism is called a ",(0,i.kt)("strong",{parentName:"p"},"mutex"),', whose name comes from "mutual exclusion".'),(0,i.kt)("p",null,"Go to ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition_mutex.c")," and notice the differences between this code and the buggy one.\nWe now use a ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_t")," variable, which we ",(0,i.kt)("inlineCode",{parentName:"p"},"lock")," at the beginning of a critical section, and we ",(0,i.kt)("inlineCode",{parentName:"p"},"unlock")," at the end.\nGenerally speaking, ",(0,i.kt)("inlineCode",{parentName:"p"},"lock"),"-ing a mutex makes a thread enter a critical section, while calling ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_unlock()")," makes the thread leave said critical section.\nTherefore, as we said previously, the critical sections in our code are ",(0,i.kt)("inlineCode",{parentName:"p"},"var--")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"var++"),".\nRun the code multiple times to convince yourself that in the end, the value of ",(0,i.kt)("inlineCode",{parentName:"p"},"var")," will always be 0."),(0,i.kt)("p",null,"Mutexes contain an internal variable which can be either 1 (locked) or 0 (unlocked).\nWhen a thread calls ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_lock()"),", it attempts to set that variable to 1.\nIf it was 0, the thread sets it to 1 and proceeds to execute the critical section.\nOtherwise, it ",(0,i.kt)("strong",{parentName:"p"},"suspends its execution")," and waits until that variable is set to 0 again."),(0,i.kt)("p",null,"When calling ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mute_unlock()"),", the internal variable is set to 0 and all waiting threads are woken up to try to acquire the mutex again.\n",(0,i.kt)("strong",{parentName:"p"},"Be careful:")," It is generally considered unsafe and ",(0,i.kt)("a",{parentName:"p",href:"https://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutex_lock.html"},"in many cases undefined behaviour")," to call ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_unlock()")," from a different thread than the one that acquired the lock.\nSo the general workflow should look something like this:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-text"},"within a single thread:\n pthread_mutex_lock(&mutex)\n // do atomic stuff\n pthread_mutex_unlock(&mutex)\n")),(0,i.kt)("h3",{id:"synchronization---overhead"},"Synchronization - Overhead"),(0,i.kt)("blockquote",null,(0,i.kt)("p",{parentName:"blockquote"},"There ain't no such thing as a free lunch")),(0,i.kt)("p",null,"This saying is also true for multithreading.\nRunning threads in parallel is nice and efficient, but synchronization always comes with a penalty: overhead.\nUse the ",(0,i.kt)("inlineCode",{parentName:"p"},"time")," command to record the running times of ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition_mutex"),".\nNotice that those of ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition_mutex")," are larger than those of ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition"),"."),(0,i.kt)("p",null,"The cause of this is that now when one thread is executing the critical section, the other has to wait and do nothing.\nWaiting means changing its state from RUNNING to WAITING, which brings further overhead from the scheduler.\nThis latter overhead comes from the ",(0,i.kt)("strong",{parentName:"p"},"context switch")," that is necessary for a thread to switch its state from RUNNING to WAITING and back."),(0,i.kt)("h3",{id:"practice-wrap-the-whole-for-statements-in-critical-sections"},"Practice: Wrap the Whole ",(0,i.kt)("inlineCode",{parentName:"h3"},"for")," Statements in Critical Sections"),(0,i.kt)("p",null,"Move the calls to ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_lock()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_unlock()")," outside the ",(0,i.kt)("inlineCode",{parentName:"p"},"for")," statements so that the critical sections become the entire statement.\nMeasure the new time spent by the code and compare it with the execution times recorded when the critical sections were made up of only ",(0,i.kt)("inlineCode",{parentName:"p"},"var--")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"var++"),"."),(0,i.kt)("p",null,(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/coarse-vs-granular-critical-section"},"Quiz")),(0,i.kt)("h2",{id:"atomics"},"Atomics"),(0,i.kt)("p",null,"So now we know how to use mutexes.\nAnd we know that mutexes work by using an internal variable that can be either 1 (locked) or 0 (unlocked).\nBut how does ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_lock()")," actually set that variable to 1?\nHow does it avoid a race condition in case another thread also wants to set it to 1?"),(0,i.kt)("p",null,'We need a guarantee that anyone "touching" that variable does so "within its own critical section".\nBut now we need a critical section to implement a critical section...\nTo solve this circular problem, we make use of a very common ',(0,i.kt)("em",{parentName:"p"},"Deus ex Machina"),": ",(0,i.kt)("strong",{parentName:"p"},"hardware support"),"."),(0,i.kt)("p",null,"Modern processors are capable of ",(0,i.kt)("em",{parentName:"p"},"atomically")," accessing data, either for reads or writes.\nAn atomic action is and indivisible sequence of operations that a thread runs without interference from others.\nConcretely, before initiating an atomic transfer on one of its data buses, the CPU first makes sure all other transfers have ended, then ",(0,i.kt)("strong",{parentName:"p"},"locks")," the data bus by stalling all cores attempting to transfer data on it.\nThis way, one thread obtains ",(0,i.kt)("strong",{parentName:"p"},"exclusive")," access to the data bus while accessing data.\nAs a side note, the critical sections in ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition_mutex.c")," are also atomic once they are wrapped between calls to ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_lock()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_mutex_unlock()"),"."),(0,i.kt)("p",null,"As with every hardware feature, the ",(0,i.kt)("inlineCode",{parentName:"p"},"x86")," ISA exposes an instruction for atomic operations.\nIn particular, this instruction is a ",(0,i.kt)("strong",{parentName:"p"},"prefix"),", called ",(0,i.kt)("inlineCode",{parentName:"p"},"lock"),".\nIt makes the instruction that follows it run atomically.\nThe ",(0,i.kt)("inlineCode",{parentName:"p"},"lock")," prefix ensures that the core performing the instruction has exclusive ownership of the cache line from where the data is transferred for the entire operation.\nThis is how the increment is made into an indivisible unit."),(0,i.kt)("p",null,"For example, ",(0,i.kt)("inlineCode",{parentName:"p"},"inc dword [x]")," can be made atomic, like so: ",(0,i.kt)("inlineCode",{parentName:"p"},"lock inc dword [x]"),".\nYou can play with the ",(0,i.kt)("inlineCode",{parentName:"p"},"lock")," prefix ",(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#atomic-assembly"},"in the Arena"),"."),(0,i.kt)("p",null,"Compilers provide support for such hardware-level atomic operations.\nGCC exposes ",(0,i.kt)("a",{parentName:"p",href:"https://gcc.gnu.org/onlinedocs/gcc/_005f_005fatomic-Builtins.html"},"built-ins")," such as ",(0,i.kt)("inlineCode",{parentName:"p"},"__atomic_load()"),", ",(0,i.kt)("inlineCode",{parentName:"p"},"__atomic_store()"),", ",(0,i.kt)("inlineCode",{parentName:"p"},"__atomic_compare_exchange()")," and many others.\nAll of them rely on the mechanism described above."),(0,i.kt)("p",null,"Go to ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition_atomic.c")," and complete the function ",(0,i.kt)("inlineCode",{parentName:"p"},"decrement_var()"),".\nCompile and run the code.\nNow measure its running time against the mutex implementations.\nIt should be somewhere between ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition.c")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition_mutex.c"),"."),(0,i.kt)("p",null,"The C standard library also provides atomic data types.\nAccess to these variables can be done only by one thread at a time.\nGo to ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition_atomic2.c"),", compile and run the code.\nNow measure its running time against the other implementations.\nNotice that the time is similar to ",(0,i.kt)("inlineCode",{parentName:"p"},"race_condition_atomic"),"."),(0,i.kt)("p",null,"So using the hardware support is more efficient, but it usually is leveraged only for simple, individual instructions, such as loads and stores.\nAnd the fact that high-level languages also expose an API for atomic operations shows how useful these operations are for developers."),(0,i.kt)("h2",{id:"semaphores"},"Semaphores"),(0,i.kt)("p",null,"Up to now, we've learned how to create critical sections that can be accessed by ",(0,i.kt)("strong",{parentName:"p"},"only one thread")," at a time.\nThese critical sections revolved around ",(0,i.kt)("strong",{parentName:"p"},"data"),".\nWhenever we define a critical section, there is some specific data to which we cannot allow parallel access.\nThe reason why we can't allow it is, in general, data integrity, as we've seen in our examples in ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/")),(0,i.kt)("p",null,"But what if threads need to count?\nCounting is inherently thread-unsafe because it's a ",(0,i.kt)("em",{parentName:"p"},"read-modify-write")," operation.\nWe read the counter, increment (modify) it and then write it back.\nThink about our example with ",(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/processes-threads-apache2"},(0,i.kt)("inlineCode",{parentName:"a"},"apache2")),"\nLet's say a ",(0,i.kt)("inlineCode",{parentName:"p"},"worker")," has created a ",(0,i.kt)("em",{parentName:"p"},"pool")," of 3 threads.\nThey are not doing any work initially;\nthey are in the WAITING state.\nAs clients initiate connections, these threads are picked up and are used to serve ",(0,i.kt)("strong",{parentName:"p"},"at most 3")," connections at a time.\nBut the number of connections may be arbitrarily large.\nTherefore, we need a way to keep track of it.\nWhen serving a client, a thread should decrement it to inform the others that a connection has been finished.\nIn short, we need a counter that the dispatcher increments and that worker threads decrement."),(0,i.kt)("p",null,"Such a counter could be implemented using a ",(0,i.kt)("strong",{parentName:"p"},"semaphore"),".\nFor simplicity's sake, you can view a semaphore as simply a mutex whose internal variable can take any value and acts like a counter.\nWhen a thread attempts to ",(0,i.kt)("inlineCode",{parentName:"p"},"acquire()")," a semaphore, it will wait if this counter is less than or equal to 0.\nOtherwise, the thread ",(0,i.kt)("strong",{parentName:"p"},"decrements")," the internal counter and the function returns.\nThe opposite of ",(0,i.kt)("inlineCode",{parentName:"p"},"acquire()")," is ",(0,i.kt)("inlineCode",{parentName:"p"},"release()"),", which increases the internal counter by a given value (by default 1)."),(0,i.kt)("h3",{id:"practice-apache2-simulator---semaphore"},"Practice: ",(0,i.kt)("inlineCode",{parentName:"h3"},"apache2")," Simulator - Semaphore"),(0,i.kt)("p",null,"Go to ",(0,i.kt)("inlineCode",{parentName:"p"},"support/apache2-simulator/apache2_simulator_semaphore.py"),".\nIn the ",(0,i.kt)("inlineCode",{parentName:"p"},"main()")," function we create a semaphore which we increment (",(0,i.kt)("inlineCode",{parentName:"p"},"release()"),") upon every new message.\nEach thread decrements (",(0,i.kt)("inlineCode",{parentName:"p"},"acquire()"),") this semaphore to signal that it wants to retrieve a message from the list.\nThe retrieval means modifying a data structure, which is a critical section, so we use a ",(0,i.kt)("strong",{parentName:"p"},"separate")," mutex for this.\nOtherwise, multiple threads could acquire the semaphore at the same time and try to modify the list at the same time.\nNot good."),(0,i.kt)("p",null,"Locking this mutex (which in Python is called ",(0,i.kt)("inlineCode",{parentName:"p"},"Lock"),") is done with the following statement: ",(0,i.kt)("inlineCode",{parentName:"p"},"with msg_mutex:"),"\nThis is a syntactic equivalent to:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-Python"},"event.acquire()\nmessages.append(msg)\nevent.release()\n")),(0,i.kt)("p",null,(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/semaphore-equivalent"},"Quiz")),(0,i.kt)("p",null,"Since the length of the ",(0,i.kt)("inlineCode",{parentName:"p"},"messages")," list is simply ",(0,i.kt)("inlineCode",{parentName:"p"},"len(messages)"),", it may seem a bit redundant to use a semaphore to store essentially the same value.\nIn the next section, we'll look at a more refined mechanism for our use case: ",(0,i.kt)("em",{parentName:"p"},"condition variables"),"."),(0,i.kt)("h2",{id:"conditions"},"Conditions"),(0,i.kt)("p",null,"Another way we can implement our ",(0,i.kt)("inlineCode",{parentName:"p"},"apache2")," simulator is to use a condition variable.\nThis one is probably the most intuitive synchronization primitive.\nIt's a means by which a thread can tell another one: \"Hey, wake up, ",(0,i.kt)("em",{parentName:"p"},"this")," happened!\".\nSo it's a way for threads to notify each other.\nFor this reason, the main methods associated with conditions are ",(0,i.kt)("inlineCode",{parentName:"p"},"notify()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"wait()"),".\nAs you might expect, they are complementary:"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"wait()")," puts the thread in the WAITING state until it's woken up by another one"),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"notify()")," wakes up one or more ",(0,i.kt)("inlineCode",{parentName:"li"},"wait()"),"-ing threads.\nIf ",(0,i.kt)("inlineCode",{parentName:"li"},"notify()")," is called before any thread has called ",(0,i.kt)("inlineCode",{parentName:"li"},"wait()"),", the first thread that calls it will continue its execution unhindered.")),(0,i.kt)("h3",{id:"practice-apache2-simulator---condition"},"Practice: ",(0,i.kt)("inlineCode",{parentName:"h3"},"apache2")," Simulator - Condition"),(0,i.kt)("p",null,"But this is not all, unfortunately.\nLook at the code in ",(0,i.kt)("inlineCode",{parentName:"p"},"support/apache2-simulator/apache2_simulator_condition.py"),".\nSee the main thread call ",(0,i.kt)("inlineCode",{parentName:"p"},"notify()")," once it reads the message.\nNotice that this call is preceded by an ",(0,i.kt)("inlineCode",{parentName:"p"},"acquire()")," call, and succeeded by a ",(0,i.kt)("inlineCode",{parentName:"p"},"release()")," call."),(0,i.kt)("p",null,(0,i.kt)("inlineCode",{parentName:"p"},"acquire()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"release()")," are commonly associated with mutexes or semaphores.\nWhat do they have to do with condition variables?"),(0,i.kt)("p",null,"Well, a lock ",(0,i.kt)("inlineCode",{parentName:"p"},"Condition")," variable also stores an inner lock (mutex).\nIt is this lock that we ",(0,i.kt)("inlineCode",{parentName:"p"},"acquire()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"release()"),".\nIn fact, the ",(0,i.kt)("a",{parentName:"p",href:"https://docs.python.org/3/library/threading.html#condition-objects"},"documentation")," states we should only call ",(0,i.kt)("inlineCode",{parentName:"p"},"Condition")," methods with its inner lock taken."),(0,i.kt)("p",null,"Why is this necessary?\nTake a look at the ",(0,i.kt)("inlineCode",{parentName:"p"},"worker()")," function.\nAfter ",(0,i.kt)("inlineCode",{parentName:"p"},"wait()"),"-ing (we'll explain the need for the loop in a bit), it extracts a message from the message queue.\nThis operation is ",(0,i.kt)("strong",{parentName:"p"},"not")," atomic, so it must be enclosed within a critical section.\nHence, the lock."),(0,i.kt)("p",null,(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/notify-only-with-mutex"},"Quiz")),(0,i.kt)("p",null,"So now we know we cannot only use a mutex.\nThe mutex is used to access and modify the ",(0,i.kt)("inlineCode",{parentName:"p"},"messages")," list atomically.\nNow, you might be thinking that this code causes a deadlock:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-Python"},"event.acquire()\nwhile len(messages) == 0:\n event.wait()\n")),(0,i.kt)("p",null,"The thread gets the lock and then, if there are no messages, it switches its state to WAITING.\nA classic deadlock, right?\nNo.\n",(0,i.kt)("inlineCode",{parentName:"p"},"wait()")," also releases the inner lock of the ",(0,i.kt)("inlineCode",{parentName:"p"},"Condition")," and being woken up reacquires it.\nNeat!\nAnd the ",(0,i.kt)("inlineCode",{parentName:"p"},"while")," loop that checks if there are any new messages is necessary because ",(0,i.kt)("inlineCode",{parentName:"p"},"wait()")," can return after an arbitrary long time.\nThis is because the thread waiting for the event was notified to wake up, but another thread has woken up before it and started handling the event earlier by reacquiring the lock.\nAll the other threads that woke up, but can't acquire the lock, must be put back to wait.\nThis situation is called a ",(0,i.kt)("strong",{parentName:"p"},"spurious wakeup"),".\nTherefore, it's necessary to check for messages again when waking up."),(0,i.kt)("p",null,"So now we have both synchronization ",(0,i.kt)("strong",{parentName:"p"},"and")," signalling.\nThis is what conditions are for, ultimately."),(0,i.kt)("p",null,"Now that you understand the concept of synchronization, you should apply it in a broader context.\n",(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#synchronization---thread-safe-data-structure"},"In the Arena"),", you'll find an exercise asking you to make an existing array list implementation thread-safe.\nHave fun!"),(0,i.kt)("h2",{id:"thread-local-storage-tls"},"Thread-Local Storage (TLS)"),(0,i.kt)("p",null,"First things first: what if we don't want data to be shared between threads?\nAre we condemned to have to worry about race conditions?\nWell, no."),(0,i.kt)("p",null,'To protect data from race conditions "by design", we can place in what\'s called ',(0,i.kt)("strong",{parentName:"p"},"Thread-Local Storage (TLS)"),'.\nAs its name implies, this is a type of storage that is "owned" by individual threads, as opposed to being shared among all threads.\n',(0,i.kt)("strong",{parentName:"p"},"Do not confuse it with copy-on-write"),".\nTLS pages are always duplicated when creating a new thread and their contents are reinitialised."),(0,i.kt)("h3",{id:"practice-c---tls-on-demand"},"Practice: C - TLS on Demand"),(0,i.kt)("p",null,"The perspective of C towards TLS is the following: everything is shared by default.\nThis makes multithreading easier and more lightweight to implement than in other languages, like D, because synchronization is left entirely up to the developer, at the cost of potential unsafety."),(0,i.kt)("p",null,"Of course, we can specify that some data belongs to the TLS, by preceding the declaration of a variable with ",(0,i.kt)("inlineCode",{parentName:"p"},"__thread")," keyword.\nFirst, compile and run the code in ",(0,i.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition_tls.c")," a few times.\nAs expected, the result is different each time."),(0,i.kt)("ol",null,(0,i.kt)("li",{parentName:"ol"},"Modify the declaration of ",(0,i.kt)("inlineCode",{parentName:"li"},"var")," and add the ",(0,i.kt)("inlineCode",{parentName:"li"},"__thread")," keyword to place the variable in the TLS of each thread.\nRecompile and run the code a few more times.\nYou should see that in the end, ",(0,i.kt)("inlineCode",{parentName:"li"},"var")," is 0.")),(0,i.kt)("p",null,(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/tls-synchronization"},"Quiz 1")),(0,i.kt)("p",null,(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/tls-var-copies"},"Quiz 2")),(0,i.kt)("ol",null,(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("p",{parentName:"li"},"Print the address and value of ",(0,i.kt)("inlineCode",{parentName:"p"},"var")," in each thread.\nSee that they differ.")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("p",{parentName:"li"},"Modify the value of ",(0,i.kt)("inlineCode",{parentName:"p"},"var")," in the ",(0,i.kt)("inlineCode",{parentName:"p"},"main()")," function before calling ",(0,i.kt)("inlineCode",{parentName:"p"},"pthread_create()"),".\nNotice that the value doesn't propagate to the other threads.\nThis is because, upon creating a new thread, its TLS is initialised."))))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0bbd68f9.4541a23b.js b/17/assets/js/0bbd68f9.4541a23b.js new file mode 100644 index 0000000000..dedac745dd --- /dev/null +++ b/17/assets/js/0bbd68f9.4541a23b.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9041],{3905:(e,t,n)=>{n.d(t,{Zo:()=>s,kt:()=>k});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function l(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),u=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},s=function(e){var t=u(e.components);return r.createElement(c.Provider,{value:t},e.children)},p="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,s=i(e,["components","mdxType","originalType","parentName"]),p=u(n),d=a,k=p["".concat(c,".").concat(d)]||p[d]||m[d]||o;return n?r.createElement(k,l(l({ref:t},s),{},{components:n})):r.createElement(k,l({ref:t},s))}));function k(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,l=new Array(o);l[0]=d;var i={};for(var c in t)hasOwnProperty.call(t,c)&&(i[c]=t[c]);i.originalType=e,i[p]="string"==typeof e?e:a,l[1]=i;for(var u=2;u{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>l,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const o={},l="Malloc `brk()`",i={unversionedId:"Lab/Data/quiz/malloc-brk",id:"Lab/Data/quiz/malloc-brk",title:"Malloc `brk()`",description:"Question Text",source:"@site/docs/Lab/Data/quiz/malloc-brk.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/malloc-brk",permalink:"/operating-systems/17/Lab/Data/quiz/malloc-brk",draft:!1,tags:[],version:"current",frontMatter:{}},c={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],s={toc:u},p="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(p,(0,r.Z)({},s,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"malloc-brk"},"Malloc ",(0,a.kt)("inlineCode",{parentName:"h1"},"brk()")),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"When does ",(0,a.kt)("inlineCode",{parentName:"p"},"malloc()")," use ",(0,a.kt)("inlineCode",{parentName:"p"},"brk()"),"?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"brk()")," is outdated, ",(0,a.kt)("inlineCode",{parentName:"li"},"malloc()")," always uses ",(0,a.kt)("inlineCode",{parentName:"li"},"mmap()"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"When it allocates a small chunk of memory")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"When it allocates an array")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"When it's working with dynamic libraries"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"malloc()")," uses both ",(0,a.kt)("inlineCode",{parentName:"p"},"brk()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"mmap()"),", but preffers ",(0,a.kt)("inlineCode",{parentName:"p"},"brk()")," for small chunks of memory to keep granular allocations in a contiguous area.\nThis way, ",(0,a.kt)("inlineCode",{parentName:"p"},"free()"),' does not necessarily return the memory to the OS as it might only mark the zone as "free" within ',(0,a.kt)("inlineCode",{parentName:"p"},"libc"),"'s allocator and reuse it for later allocations."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0c19000f.abb0179c.js b/17/assets/js/0c19000f.abb0179c.js new file mode 100644 index 0000000000..5a791c4264 --- /dev/null +++ b/17/assets/js/0c19000f.abb0179c.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3068],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>y});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function s(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var i=r.createContext({}),l=function(e){var t=r.useContext(i),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},c=function(e){var t=l(e.components);return r.createElement(i.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,i=e.parentName,c=p(e,["components","mdxType","originalType","parentName"]),u=l(n),d=a,y=u["".concat(i,".").concat(d)]||u[d]||m[d]||o;return n?r.createElement(y,s(s({ref:t},c),{},{components:n})):r.createElement(y,s({ref:t},c))}));function y(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,s=new Array(o);s[0]=d;var p={};for(var i in t)hasOwnProperty.call(t,i)&&(p[i]=t[i]);p.originalType=e,p[u]="string"==typeof e?e:a,s[1]=p;for(var l=2;l{n.r(t),n.d(t,{assets:()=>i,contentTitle:()=>s,default:()=>m,frontMatter:()=>o,metadata:()=>p,toc:()=>l});var r=n(7462),a=(n(7294),n(3905));const o={},s="Setting up the Lab Environment",p={unversionedId:"Lab/lab-setup",id:"Lab/lab-setup",title:"Setting up the Lab Environment",description:"If you have already cloned the repository, make sure it is updated:",source:"@site/docs/Lab/lab-setup.md",sourceDirName:"Lab",slug:"/Lab/lab-setup",permalink:"/operating-systems/17/Lab/lab-setup",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Lab",permalink:"/operating-systems/17/Lab/"},next:{title:"Software Stack",permalink:"/operating-systems/17/Lab/Software Stack/"}},i={},l=[],c={toc:l},u="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(u,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"setting-up-the-lab-environment"},"Setting up the Lab Environment"),(0,a.kt)("p",null,"If you have already cloned the repository, make sure it is updated:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ cd operating-systems\n\nstudent@os:~/operating-systems$ git pull --rebase\n")),(0,a.kt)("p",null,"The command may fail if you have uncommitted changes.\nIf that is the case, stash your changes, retry, and pop the stash:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/operating-systems$ git stash\n\nstudent@os:~/operating-systems$ git pull --rebase\n\nstudent@os:~/operating-systems$ git stash pop\n")),(0,a.kt)("p",null,"If you haven't already cloned the repository, do so and then enter the repository:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ git clone https://github.com/cs-pub-ro/operating-systems\n\nstudent@os:~$ cd operating-systems\n")),(0,a.kt)("p",null,"Navigate to a chapter's lab directory:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/operating-systems$ cd content/chapters//lab/\n")),(0,a.kt)("p",null,"The possible options are: ",(0,a.kt)("inlineCode",{parentName:"p"},"software-stack"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"data"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"compute"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"io")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"app-interact"),"."),(0,a.kt)("p",null,"If you're using the OS-runner Docker container, you can use the following shortcuts:"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"go-ss")," - changes directory to Software Stack lab"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"go-data")," - changes directory to Data lab"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"go-compute")," - changes directory to Compute lab"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"go-io")," - changes directory to IO lab"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"go-appInt")," - changes directory to App Interaction lab"))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0f890721.ed25aa74.js b/17/assets/js/0f890721.ed25aa74.js new file mode 100644 index 0000000000..3e4ff13bd2 --- /dev/null +++ b/17/assets/js/0f890721.ed25aa74.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8211],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>d});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function i(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var p=a.createContext({}),l=function(e){var t=a.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=l(e.components);return a.createElement(p.Provider,{value:t},e.children)},h="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,p=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),h=l(n),u=r,d=h["".concat(p,".").concat(u)]||h[u]||m[u]||o;return n?a.createElement(d,i(i({ref:t},c),{},{components:n})):a.createElement(d,i({ref:t},c))}));function d(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,i=new Array(o);i[0]=u;var s={};for(var p in t)hasOwnProperty.call(t,p)&&(s[p]=t[p]);s.originalType=e,s[h]="string"==typeof e?e:r,i[1]=s;for(var l=2;l{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>s,toc:()=>l});var a=n(7462),r=(n(7294),n(3905));const o={},i="Asynchronous I/O",s={unversionedId:"Lab/IO/async-io",id:"Lab/IO/async-io",title:"Asynchronous I/O",description:"When doing I/O, the major issue we are facing is that I/O operations are typically much slower than CPU operations.",source:"@site/docs/Lab/IO/async-io.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/async-io",permalink:"/operating-systems/17/Lab/IO/async-io",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Zero-Copy",permalink:"/operating-systems/17/Lab/IO/zero-copy"},next:{title:"I/O Multiplexing",permalink:"/operating-systems/17/Lab/IO/io-multiplexing"}},p={},l=[{value:"Practice",id:"practice",level:2},{value:"Remarks",id:"remarks",level:2}],c={toc:l},h="wrapper";function m(e){let{components:t,...n}=e;return(0,r.kt)(h,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"asynchronous-io"},"Asynchronous I/O"),(0,r.kt)("p",null,"When doing I/O, the major issue we are facing is that I/O operations are typically much slower than CPU operations.\nBecause of that, in a typical synchronous scenario, a lot of time may be spent waiting for an I/O operation to be complete."),(0,r.kt)("p",null,"Because of that, we have other types of I/O operations."),(0,r.kt)("p",null,"The simplest type of I/O operating is a synchronous, blocking operation.\nIn this operation type, we call a given function (such as ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"write()"),") and wait until it completes."),(0,r.kt)("p",null,"Another type of operation is a non-blocking operation.\nIn this type of operation, if data isn't available (in case of ",(0,r.kt)("inlineCode",{parentName:"p"},"read()"),") or can not be sent (in case of ",(0,r.kt)("inlineCode",{parentName:"p"},"write()"),"), the given call will not block, it will simply return with a mark to try again later on."),(0,r.kt)("p",null,"Another type of operation is an asynchronous operation.\nIn the case of an asynchronous operation, the given function returns immediately and the program continues its execution.\nOnce the operation succeeds, a notification may be sent out to the program.\nOr the program can periodically check the state of the operation."),(0,r.kt)("p",null,"Apart from these operating types, there is the option to do I/O multiplexing, i.e. the ability to tackle multiple channels simultaneously.\nThis is useful in case of servers, that get a large number of requests and have to iterate through them."),(0,r.kt)("p",null,'In case of asynchronous I/O, the "backend" used to implement the operations may differ:'),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"It can be a multiprocess backend, where each action / request is passed to a given process.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"It can be a multithreaded backend, where each action / request is passed to a given thread.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"It can be an event-based backend, where an action is scheduled together with a callback that is invoked when an action ends."))),(0,r.kt)("h2",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/async/")," folder for some implementations of a simple request-reply server in Python or in C.\nThe server gets requests and serves them in different ways: synchronous, multiprocess-based, multi-threading-based, asynchronous."),(0,r.kt)("p",null,"We use two implementations, in Python and in C."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"For the Python implementation, enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"python/")," subdirectory.\nTake a look at the implementation of different servers:"),(0,r.kt)("ul",{parentName:"li"},(0,r.kt)("li",{parentName:"ul"},"synchronous server: ",(0,r.kt)("inlineCode",{parentName:"li"},"server.py")),(0,r.kt)("li",{parentName:"ul"},"multiprocess backend: ",(0,r.kt)("inlineCode",{parentName:"li"},"mp_server.py")),(0,r.kt)("li",{parentName:"ul"},"multithreaded backend: ",(0,r.kt)("inlineCode",{parentName:"li"},"mt_server.py")),(0,r.kt)("li",{parentName:"ul"},"asynchronous API backend: ",(0,r.kt)("inlineCode",{parentName:"li"},"async_server.py")," (requires Python >= 3.7) and ",(0,r.kt)("inlineCode",{parentName:"li"},"async_server_3.6.py")," (works for Python 3.6)")),(0,r.kt)("p",{parentName:"li"},"Let's do a benchmarking session of each server type.\nFor this we will:"),(0,r.kt)("ol",{parentName:"li"},(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Start the server on one console.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use the ",(0,r.kt)("inlineCode",{parentName:"p"},"client_bench.sh")," script to benchmark the server.\n",(0,r.kt)("inlineCode",{parentName:"p"},"client_bench.sh")," is a simple script that starts multiple clients to connect to the servers, by running ",(0,r.kt)("inlineCode",{parentName:"p"},"client.py"),".\nIt runs ",(0,r.kt)("inlineCode",{parentName:"p"},"NUM_CLIENT")," instances of ",(0,r.kt)("inlineCode",{parentName:"p"},"client.py")," to trigger actions in the remote server."),(0,r.kt)("p",{parentName:"li"},(0,r.kt)("strong",{parentName:"p"},"Note")," that you may be required to run servers on different ports in case of an ",(0,r.kt)("inlineCode",{parentName:"p"},"Address already in use error"),".\nIf that is the case, you will also need to run the ",(0,r.kt)("inlineCode",{parentName:"p"},"client_bench.sh")," script."))),(0,r.kt)("p",{parentName:"li"},"To start the server, run each of these commands (one at a time to test the respective server type):"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/async/python$ ./server.py 2999\n\nstudent@os:/.../support/async/python$ ./mp_server.py 2999\n\nstudent@os:/.../support/async/python$ ./mt_server.py 2999\n\nstudent@os:/.../support/async/python$ ./async_server_3.6.py 2999\n")),(0,r.kt)("p",{parentName:"li"},"For each server, in a different console, we can test to see how well it behaves by running:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/async/python$ time ./client_bench.sh\n")),(0,r.kt)("p",{parentName:"li"},"You will see a time duration difference between ",(0,r.kt)("inlineCode",{parentName:"p"},"mp_server.py")," and the others, ",(0,r.kt)("inlineCode",{parentName:"p"},"mp_server.py")," runs requests faster.\nThis is because the multiprocess model works OK for a CPU-intensive server such as this."),(0,r.kt)("p",{parentName:"li"},"But not on threading, because threading suffers from the use of ",(0,r.kt)("a",{parentName:"p",href:"https://realpython.com/python-gil/"},"GIL (",(0,r.kt)("em",{parentName:"a"},"Global Interpreter Lock"),")"),", that prevents multithreaded programs from running Python code simultaneously.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"For the C implementation, enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"c/")," subdirectory.\nTake a look at the implementation of different servers:"),(0,r.kt)("ul",{parentName:"li"},(0,r.kt)("li",{parentName:"ul"},"synchronous server: ",(0,r.kt)("inlineCode",{parentName:"li"},"server.c")),(0,r.kt)("li",{parentName:"ul"},"multiprocess backend: ",(0,r.kt)("inlineCode",{parentName:"li"},"mp_server.c")),(0,r.kt)("li",{parentName:"ul"},"multithreaded backend: ",(0,r.kt)("inlineCode",{parentName:"li"},"mt_server.c"))),(0,r.kt)("p",{parentName:"li"},"There is no asynchronous C variant, because of the unstable API of ",(0,r.kt)("a",{parentName:"p",href:"https://pagure.io/libaio"},(0,r.kt)("inlineCode",{parentName:"a"},"libaio"))," and ",(0,r.kt)("a",{parentName:"p",href:"https://unixism.net/loti/what_is_io_uring.html"},(0,r.kt)("inlineCode",{parentName:"a"},"io_uring")),"."),(0,r.kt)("p",{parentName:"li"},"First, use ",(0,r.kt)("inlineCode",{parentName:"p"},"make")," to build the servers.\nThen, go through the same steps as above:"),(0,r.kt)("ol",{parentName:"li"},(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Start the server on one console.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use the ",(0,r.kt)("inlineCode",{parentName:"p"},"client_bench.sh")," script (in the ",(0,r.kt)("inlineCode",{parentName:"p"},"../python/")," directory) to benchmark the server."))),(0,r.kt)("p",{parentName:"li"},"Same as with Python, to start the server, run each of these commands (one at a time to test the respective server type):"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/async/c$ ./server 2999\n\nstudent@os:/.../support/async/c$ ./mp_server 2999\n\nstudent@os:/.../support/async/c$ ./mt_server 2999\n")),(0,r.kt)("p",{parentName:"li"},"For each server, in a different console, we can test to see how well it behaves by running:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/async/python$ time client_bench.sh\n")),(0,r.kt)("p",{parentName:"li"},"We draw 2 conclusions from using the C variant:"),(0,r.kt)("ol",{parentName:"li"},(0,r.kt)("li",{parentName:"ol"},"C is faster than Python"),(0,r.kt)("li",{parentName:"ol"},"The thread variant is indeed parallel, as now the GIL is no longer involved.")))),(0,r.kt)("h2",{id:"remarks"},"Remarks"),(0,r.kt)("p",null,"Asynchronous operations, as with others, provide an API, as is the case with the Python API or the ",(0,r.kt)("a",{parentName:"p",href:"https://pagure.io/libaio"},(0,r.kt)("inlineCode",{parentName:"a"},"libaio"))," and ",(0,r.kt)("a",{parentName:"p",href:"https://unixism.net/loti/what_is_io_uring.html"},(0,r.kt)("inlineCode",{parentName:"a"},"io_uring")),".\nThe backend of these operations may be a thread-based one in the library providing the API, or it may rely on support from the operating system.\nWhen aiming for performance, asynchronous I/O operations are part of the game.\nAnd it's very useful having a good understanding of what's happening behind the scenes."),(0,r.kt)("p",null,"For example, for the Python ",(0,r.kt)("inlineCode",{parentName:"p"},"async_server_3.6.py")," server, a message ",(0,r.kt)("inlineCode",{parentName:"p"},"asyncio: Using selector: EpollSelector")," is provided.\nThis means that the backend relies on the use of the ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,r.kt)("inlineCode",{parentName:"a"},"epoll()")," function")," that's part of the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/io-multiplexing"},"I/O Multiplexing section"),"."),(0,r.kt)("p",null,"Also, for Python, the use of the GIL may be an issue when the operations are CPU intensive."),(0,r.kt)("p",null,"This last point deserves a discussion.\nIt's rare that servers or programs using asynchronous operations are CPU intensive.\nIt's more likely that they are I/O intensive, and the challenge is avoiding blocking points in multiple I/O channels;\nnot avoiding doing a lot of processing, as is the case with the ",(0,r.kt)("inlineCode",{parentName:"p"},"fibonacci()")," function.\nIn that particular case, having thread-based asynchronous I/O and the GIL will be a good option, as you rely on the thread scheduler to be able to serve multiple I/O channels simultaneously.\nThis later approach is called I/O multiplexing, discussed in ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/io-multiplexing"},"the next section"),"."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/0fa0bebd.fd189ff9.js b/17/assets/js/0fa0bebd.fd189ff9.js new file mode 100644 index 0000000000..0aa599e5b7 --- /dev/null +++ b/17/assets/js/0fa0bebd.fd189ff9.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1967],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>m});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function o(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var s=n.createContext({}),p=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):o(o({},t),e)),r},u=function(e){var t=p(e.components);return n.createElement(s.Provider,{value:t},e.children)},c="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,i=e.originalType,s=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),c=p(r),f=a,m=c["".concat(s,".").concat(f)]||c[f]||d[f]||i;return r?n.createElement(m,o(o({ref:t},u),{},{components:r})):n.createElement(m,o({ref:t},u))}));function m(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=r.length,o=new Array(i);o[0]=f;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[c]="string"==typeof e?e:a,o[1]=l;for(var p=2;p{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>d,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var n=r(7462),a=(r(7294),r(3905));const i={},o="File Descriptor of `stderr`",l={unversionedId:"Lab/IO/quiz/stderr-fd",id:"Lab/IO/quiz/stderr-fd",title:"File Descriptor of `stderr`",description:"Question Text",source:"@site/docs/Lab/IO/quiz/stderr-fd.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/stderr-fd",permalink:"/operating-systems/17/Lab/IO/quiz/stderr-fd",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedaback",id:"feedaback",level:2}],u={toc:p},c="wrapper";function d(e){let{components:t,...r}=e;return(0,a.kt)(c,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"file-descriptor-of-stderr"},"File Descriptor of ",(0,a.kt)("inlineCode",{parentName:"h1"},"stderr")),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Which file descriptor is associated by default to ",(0,a.kt)("inlineCode",{parentName:"p"},"stderr"),"?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"it varies from process to process")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"it varies from one Linux distribution to another")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"stderr")," has no associated file descriptor"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"2")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"0")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"1"))),(0,a.kt)("h2",{id:"feedaback"},"Feedaback"),(0,a.kt)("p",null,"You would type ",(0,a.kt)("inlineCode",{parentName:"p"},"ls 2> /dev/null")," to ignore ",(0,a.kt)("inlineCode",{parentName:"p"},"ls"),"'s errors.\nThis equates to ",(0,a.kt)("strong",{parentName:"p"},"redirecting")," ",(0,a.kt)("inlineCode",{parentName:"p"},"stderr")," to ",(0,a.kt)("inlineCode",{parentName:"p"},"/dev/null"),".\nThe number 2 in ",(0,a.kt)("inlineCode",{parentName:"p"},"2>")," hints that ",(0,a.kt)("inlineCode",{parentName:"p"},"stderr")," is by default associated with file descriptor 2."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/144ccec8.a808b6b9.js b/17/assets/js/144ccec8.a808b6b9.js new file mode 100644 index 0000000000..b42247c6d2 --- /dev/null +++ b/17/assets/js/144ccec8.a808b6b9.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5737],{3905:(e,t,n)=>{n.d(t,{Zo:()=>m,kt:()=>k});var a=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function l(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function r(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var s=a.createContext({}),p=function(e){var t=a.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):r(r({},t),e)),n},m=function(e){var t=p(e.components);return a.createElement(s.Provider,{value:t},e.children)},c="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var n=e.components,o=e.mdxType,l=e.originalType,s=e.parentName,m=i(e,["components","mdxType","originalType","parentName"]),c=p(n),u=o,k=c["".concat(s,".").concat(u)]||c[u]||d[u]||l;return n?a.createElement(k,r(r({ref:t},m),{},{components:n})):a.createElement(k,r({ref:t},m))}));function k(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var l=n.length,r=new Array(l);r[0]=u;var i={};for(var s in t)hasOwnProperty.call(t,s)&&(i[s]=t[s]);i.originalType=e,i[c]="string"==typeof e?e:o,r[1]=i;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>r,default:()=>d,frontMatter:()=>l,metadata:()=>i,toc:()=>p});var a=n(7462),o=(n(7294),n(3905));const l={},r="Investigate Memory Actions",i={unversionedId:"Lab/Data/investigate-memory",id:"Lab/Data/investigate-memory",title:"Investigate Memory Actions",description:"Memory actions generally mean:",source:"@site/docs/Lab/Data/investigate-memory.md",sourceDirName:"Lab/Data",slug:"/Lab/Data/investigate-memory",permalink:"/operating-systems/17/Lab/Data/investigate-memory",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Process Memory",permalink:"/operating-systems/17/Lab/Data/process-memory"},next:{title:"Memory Security",permalink:"/operating-systems/17/Lab/Data/memory-security"}},s={},p=[{value:"Memory Leaks",id:"memory-leaks",level:2},{value:"Practice",id:"practice",level:3},{value:"Memory Actions (and Leaks) in Existing Programs",id:"memory-actions-and-leaks-in-existing-programs",level:2},{value:"Practice",id:"practice-1",level:3},{value:"jemalloc",id:"jemalloc",level:2},{value:"malloc in Musl",id:"malloc-in-musl",level:2},{value:"App Investigation: Deluge",id:"app-investigation-deluge",level:2},{value:"Practice",id:"practice-2",level:3},{value:"App Investigation: Servo",id:"app-investigation-servo",level:2},{value:"Practice",id:"practice-3",level:3},{value:"Investigation: Alocator in the D Programming Language",id:"investigation-alocator-in-the-d-programming-language",level:2},{value:"Practice",id:"practice-4",level:3},{value:"App Investigation: Git",id:"app-investigation-git",level:2},{value:"Practice",id:"practice-5",level:3}],m={toc:p},c="wrapper";function d(e){let{components:t,...n}=e;return(0,o.kt)(c,(0,a.Z)({},m,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"investigate-memory-actions"},"Investigate Memory Actions"),(0,o.kt)("p",null,"Memory actions generally mean:"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"memory access: read, write or execute"),(0,o.kt)("li",{parentName:"ul"},"memory allocation"),(0,o.kt)("li",{parentName:"ul"},"memory deallocation")),(0,o.kt)("p",null,"By far, the most important actions are allocation and deallocation.\nBecause, if not done right, these can get to memory loss and poor memory use."),(0,o.kt)("p",null,"Memory loss generally happens in the form of memory leaks."),(0,o.kt)("h2",{id:"memory-leaks"},"Memory Leaks"),(0,o.kt)("p",null,"A memory leak occurs when we lose reference to a memory area.\nThat is, a pointer used to point to a memory area.\nAnd then it's pointing to a new memory area and the old memory area is lost."),(0,o.kt)("p",null,"Enter the ",(0,o.kt)("inlineCode",{parentName:"p"},"support/memory-leak/")," folder.\nIt stores two files showing memory leaks:"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"one in C++: ",(0,o.kt)("inlineCode",{parentName:"li"},"memory_leak.cpp")),(0,o.kt)("li",{parentName:"ul"},"one in C: ",(0,o.kt)("inlineCode",{parentName:"li"},"memory_leak_malloc"))),(0,o.kt)("p",null,"Let's build and run the two executables:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ make\ng++ -c -o memory_leak.o memory_leak.cpp\ncc memory_leak.o -lstdc++ -o memory_leak\ncc -c -o memory_leak_malloc.o memory_leak_malloc.c\ncc memory_leak_malloc.o -lstdc++ -o memory_leak_malloc\n")),(0,o.kt)("p",null,"Running them yields similar output:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ ./memory_leak\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\nstudent@os:~/.../lab/support/memory-leak$ ./memory_leak_malloc\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\n")),(0,o.kt)("p",null,"We investigate the memory leaks of the two programs by using ",(0,o.kt)("a",{parentName:"p",href:"https://valgrind.org/"},"Valgrind"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ valgrind ./memory_leak\n==22362== Memcheck, a memory error detector\n==22362== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.\n==22362== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info\n==22362== Command: ./memory_leak\n==22362==\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\n==22362==\n==22362== HEAP SUMMARY:\n==22362== in use at exit: 72 bytes in 1 blocks\n==22362== total heap usage: 4 allocs, 3 frees, 73,872 bytes allocated\n==22362==\n==22362== LEAK SUMMARY:\n==22362== definitely lost: 72 bytes in 1 blocks\n==22362== indirectly lost: 0 bytes in 0 blocks\n==22362== possibly lost: 0 bytes in 0 blocks\n==22362== still reachable: 0 bytes in 0 blocks\n==22362== suppressed: 0 bytes in 0 blocks\n==22362== Rerun with --leak-check=full to see details of leaked memory\n==22362==\n==22362== For counts of detected and suppressed errors, rerun with: -v\n==22362== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)\n\nstudent@os:~/.../lab/support/memory-leak$ valgrind ./memory_leak_malloc\n==22369== Memcheck, a memory error detector\n==22369== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.\n==22369== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info\n==22369== Command: ./memory_leak_malloc\n==22369==\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\n==22369==\n==22369== HEAP SUMMARY:\n==22369== in use at exit: 148 bytes in 1 blocks\n==22369== total heap usage: 3 allocs, 2 frees, 1,320 bytes allocated\n==22369==\n==22369== LEAK SUMMARY:\n==22369== definitely lost: 148 bytes in 1 blocks\n==22369== indirectly lost: 0 bytes in 0 blocks\n==22369== possibly lost: 0 bytes in 0 blocks\n==22369== still reachable: 0 bytes in 0 blocks\n==22369== suppressed: 0 bytes in 0 blocks\n==22369== Rerun with --leak-check=full to see details of leaked memory\n==22369==\n==22369== For counts of detected and suppressed errors, rerun with: -v\n==22369== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)\n")),(0,o.kt)("p",null,"As we are doing allocations that are not freed, this results in memory leaks."),(0,o.kt)("p",null,"For ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()"),"-based programs, we can use ",(0,o.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/mtrace.3.html"},(0,o.kt)("inlineCode",{parentName:"a"},"mtrace()")," feature")," and ",(0,o.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man1/mtrace.1.html"},(0,o.kt)("inlineCode",{parentName:"a"},"mtrace")," command")," to verify proper allocations with ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()")," and deallocations with ",(0,o.kt)("inlineCode",{parentName:"p"},"free()"),".\nWe call ",(0,o.kt)("inlineCode",{parentName:"p"},"mtrace()")," in the program (in ",(0,o.kt)("inlineCode",{parentName:"p"},"memory_leak_malloc.c"),") to enable ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"free()")," checking."),(0,o.kt)("p",null,"To use ",(0,o.kt)("inlineCode",{parentName:"p"},"mtrace()")," we define the ",(0,o.kt)("inlineCode",{parentName:"p"},"MALLOC_TRACE")," environment variable.\nWe probably also require to preload the libc malloc debugging library, so we use ",(0,o.kt)("inlineCode",{parentName:"p"},"LD_PRELOAD")," for that.\nNote that the file path used for ",(0,o.kt)("inlineCode",{parentName:"p"},"LD_PRELOAD")," may need to be updated, depending on your distribution:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/lib/x86_64-linux-gnu/libc_malloc_debug.so.0 MALLOC_TRACE=mem.trace ./memory_leak_malloc\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\n")),(0,o.kt)("p",null,"Subsequently, we use the ",(0,o.kt)("inlineCode",{parentName:"p"},"mtrace")," tool to show information about the leaked data:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ mtrace ./memory_leak_malloc mem.trace\n\nMemory not freed:\n-----------------\n Address Size Caller\n0x000056506d8be6a0 0x94 at 0x56506c3777ec\n")),(0,o.kt)("p",null,"The size (",(0,o.kt)("inlineCode",{parentName:"p"},"0x94"),") is the same value shown by Valgrind (",(0,o.kt)("inlineCode",{parentName:"p"},"148"),")."),(0,o.kt)("p",null,(0,o.kt)("inlineCode",{parentName:"p"},"mtrace")," provides an outcome similar to Valgrind.\nValgrind is however more powerful: it works on different types of memory (not only those allocated with ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()"),") and it doesn't require access to the source code (and the compiler phase)."),(0,o.kt)("h3",{id:"practice"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Print the size of the ",(0,o.kt)("inlineCode",{parentName:"p"},"Student")," class and the ",(0,o.kt)("inlineCode",{parentName:"p"},"struct student")," structure to see if it equates to the leak shown by Valgrind.")),(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Solve the memory leaks in both programs.\nValidate with Valgrind."))),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/valgrind-leaks"},"Quiz")),(0,o.kt)("h2",{id:"memory-actions-and-leaks-in-existing-programs"},"Memory Actions (and Leaks) in Existing Programs"),(0,o.kt)("p",null,"We can use Valgrind to investigate existing programs in the system.\nThis tells us whether they possess memory leaks:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ valgrind ls\n==24669== Memcheck, a memory error detector\n==24669== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.\n==24669== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info\n==24669== Command: ls\n==24669==\nMakefile memory_leak memory_leak.cpp memory_leak_malloc memory_leak_malloc.c memory_leak_malloc.o memory_leak.o\n==24669==\n==24669== HEAP SUMMARY:\n==24669== in use at exit: 21,696 bytes in 14 blocks\n==24669== total heap usage: 51 allocs, 37 frees, 61,331 bytes allocated\n==24669==\n==24669== LEAK SUMMARY:\n==24669== definitely lost: 0 bytes in 0 blocks\n==24669== indirectly lost: 0 bytes in 0 blocks\n==24669== possibly lost: 0 bytes in 0 blocks\n==24669== still reachable: 21,696 bytes in 14 blocks\n==24669== suppressed: 0 bytes in 0 blocks\n==24669== Rerun with --leak-check=full to see details of leaked memory\n==24669==\n==24669== For counts of detected and suppressed errors, rerun with: -v\n==24669== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)\n\nstudent@os:~/.../lab/support/memory-leak$ valgrind ps\n==24671== Memcheck, a memory error detector\n==24671== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.\n==24671== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info\n==24671== Command: ps\n==24671==\n PID TTY TIME CMD\n24671 pts/22 00:00:00 memcheck-amd64-\n26732 pts/22 00:00:01 bash\n==24671==\n==24671== HEAP SUMMARY:\n==24671== in use at exit: 264,929 bytes in 25 blocks\n==24671== total heap usage: 692 allocs, 667 frees, 334,268 bytes allocated\n==24671==\n==24671== LEAK SUMMARY:\n==24671== definitely lost: 0 bytes in 0 blocks\n==24671== indirectly lost: 0 bytes in 0 blocks\n==24671== possibly lost: 0 bytes in 0 blocks\n==24671== still reachable: 264,929 bytes in 25 blocks\n==24671== suppressed: 0 bytes in 0 blocks\n==24671== Rerun with --leak-check=full to see details of leaked memory\n==24671==\n==24671== For counts of detected and suppressed errors, rerun with: -v\n==24671== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)\n\nstudent@os:~/.../lab/support/memory-leak$ valgrind bash -c 'echo \"ha\"'\n==24675== Memcheck, a memory error detector\n==24675== Copyright (C) 2002-2017, and GNU GPL'd, by Julian Seward et al.\n==24675== Using Valgrind-3.13.0 and LibVEX; rerun with -h for copyright info\n==24675== Command: bash -c echo\\ \"ha\"\n==24675==\nha\n==24675==\n==24675== HEAP SUMMARY:\n==24675== in use at exit: 43,056 bytes in 672 blocks\n==24675== total heap usage: 774 allocs, 102 frees, 51,405 bytes allocated\n==24675==\n==24675== LEAK SUMMARY:\n==24675== definitely lost: 12 bytes in 1 blocks\n==24675== indirectly lost: 0 bytes in 0 blocks\n==24675== possibly lost: 0 bytes in 0 blocks\n==24675== still reachable: 43,044 bytes in 671 blocks\n==24675== suppressed: 0 bytes in 0 blocks\n==24675== Rerun with --leak-check=full to see details of leaked memory\n==24675==\n==24675== For counts of detected and suppressed errors, rerun with: -v\n==24675== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)\n")),(0,o.kt)("p",null,"We can see that ",(0,o.kt)("inlineCode",{parentName:"p"},"ls")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"ps")," don't have memory leaks.\nHowever, the shell (Bash) shows a memory leak of 12 bytes (on the test system).\nThis may be a false positive or the subject of an actual investigation."),(0,o.kt)("p",null,"Note that the ",(0,o.kt)("inlineCode",{parentName:"p"},"still reachable")," section of the output refers to memory that wasn't freed, but still has pointers referring to it.\nA true memory leak occurs when no pointers refer any memory area."),(0,o.kt)("h3",{id:"practice-1"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Investigate 2-3 other executables in the system using Valgrind.")),(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Use ",(0,o.kt)("inlineCode",{parentName:"p"},"ltrace")," to list ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"free()")," calls made by the investigated system executables."))),(0,o.kt)("p",null,"Note that, as explained in the ",(0,o.kt)("a",{parentName:"p",href:"https://open-education-hub.github.io/operating-systems/Lab/Software%20Stack/libcall-syscall"},"Software Stack lab"),", on some systems, ",(0,o.kt)("inlineCode",{parentName:"p"},"ltrace")," does not accurately show the output, due to ",(0,o.kt)("em",{parentName:"p"},"now binding"),".\nFear not, you can always check the library calls with a more verbose and harder to parse ",(0,o.kt)("inlineCode",{parentName:"p"},"ltrace")," command:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~$ ltrace -x "*"\n')),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/memory-leaks"},"Quiz")),(0,o.kt)("h2",{id:"jemalloc"},"jemalloc"),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"http://jemalloc.net/"},"jemalloc")," is a featureful allocator that is intended to replace the standard allocator in the standard C library (libc).\njemalloc provides replacements for the general ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"free()")," functions, and also provides a custom API targeted for performance tuning."),(0,o.kt)("p",null,"As ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/jemalloc/jemalloc/wiki/Getting-Started"},"documented"),", there are multiple ways to use ",(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc"),", the easiest of which is to use the ",(0,o.kt)("inlineCode",{parentName:"p"},"LD_PRELOAD")," environment variable and preload the library and hook into ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"free()")," function calls."),(0,o.kt)("p",null,"First install ",(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc")," on our system.\nOn your typical Ubuntu / Debian-based system, use ",(0,o.kt)("inlineCode",{parentName:"p"},"apt"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../data/lab/content$ sudo apt -y install libjemalloc-dev\n")),(0,o.kt)("p",null,"Note that this installs the distribution package, not the latest one (that may provide more features)."),(0,o.kt)("p",null,"With this in place, we can use ",(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc")," against our pre-built executables or system executables (such as ",(0,o.kt)("inlineCode",{parentName:"p"},"ls"),", ",(0,o.kt)("inlineCode",{parentName:"p"},"ps"),").\nWe can test it against the executable files from ",(0,o.kt)("inlineCode",{parentName:"p"},"support/memory-leak/"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so ./memory_leak_malloc\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\n")),(0,o.kt)("p",null,(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc")," can use the ",(0,o.kt)("inlineCode",{parentName:"p"},"MALLOC_CONF")," environment variable for a ",(0,o.kt)("a",{parentName:"p",href:"https://www.freebsd.org/cgi/man.cgi?query=malloc.conf"},"diverse set of configurations"),".\nFor example, by using ",(0,o.kt)("inlineCode",{parentName:"p"},"stats_print:true")," we print out information regarding the use of the library functions:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF="stats_print:true" ./memory_leak_malloc\nAndrei Popescu is 22 years old and likes Linux\nIoana David is 23 years old and likes macOS\n___ Begin jemalloc statistics ___\nVersion: 3.6.0-11\nAssertions disabled\nRun-time option settings:\n opt.abort: false\n opt.lg_chunk: 22\n opt.dss: "secondary"\n opt.narenas: 32\n opt.lg_dirty_mult: 3\n opt.stats_print: true\n opt.junk: false\n opt.quarantine: 0\n opt.redzone: false\n[...]\ndirty pages: 26:0 active:dirty, 0 sweeps, 0 madvises, 0 purged\n allocated nmalloc ndalloc nrequests\nsmall: 72672 114 0 3\nlarge: 32768 1 0 1\ntotal: 105440 115 0 4\nactive: 106496\nmapped: 4194304\n[...]\n')),(0,o.kt)("p",null,(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc")," doesn't work against system executables using preloading, likely because of security options disabling the use of the library:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF="stats_print:true" /bin/ls\nMakefile memory_leak memory_leak.cpp memory_leak_malloc memory_leak_malloc.c memory_leak_malloc.o memory_leak.o\n\nstudent@os:~/.../lab/support/memory-leak$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so MALLOC_CONF="stats_print:true" /bin/ps\n PID TTY TIME CMD\n 1581 pts/22 00:00:00 ps\n26732 pts/22 00:00:01 bash\n')),(0,o.kt)("h2",{id:"malloc-in-musl"},"malloc in Musl"),(0,o.kt)("p",null,"Each libc (or memory allocator such as ",(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc"),") uses their own implementation of ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()"),", ",(0,o.kt)("inlineCode",{parentName:"p"},"free()")," and other functions.\n",(0,o.kt)("a",{parentName:"p",href:"https://musl.libc.org/"},"Musl libc")," is a lightweight standard C library that provides compatible features with the more heavyweights ",(0,o.kt)("a",{parentName:"p",href:"https://www.gnu.org/software/libc/"},"GNU libc"),"."),(0,o.kt)("p",null,"Take a look through implementation of ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"free()")," in ",(0,o.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/malloc"},"Musl libc"),".\nSee all three implementations for ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()"),":"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"the one in ",(0,o.kt)("inlineCode",{parentName:"li"},"lite_malloc.c")),(0,o.kt)("li",{parentName:"ul"},"the one in ",(0,o.kt)("inlineCode",{parentName:"li"},"mallocng/malloc.c")),(0,o.kt)("li",{parentName:"ul"},"the one in ",(0,o.kt)("inlineCode",{parentName:"li"},"oldmalloc/malloc"))),(0,o.kt)("p",null,"See also ",(0,o.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/malloc/mallocng/free.c#L101"},"the implementation of ",(0,o.kt)("inlineCode",{parentName:"a"},"free()")),".\nAnd ",(0,o.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/malloc/calloc.c#L33"},"the implementation of ",(0,o.kt)("inlineCode",{parentName:"a"},"calloc()")),"."),(0,o.kt)("p",null,"You needn't spend too much time browsing the implementation of these functions, just having a broad understanding of how they work."),(0,o.kt)("h2",{id:"app-investigation-deluge"},"App Investigation: Deluge"),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"https://www.deluge-torrent.org/"},"Deluge")," is a Bittorrent client written in Python."),(0,o.kt)("p",null,"We want to locate places that allocate memory in Deluge (in Python).\nThis generally means locating instantiation of classes."),(0,o.kt)("p",null,"Let's clone the ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/deluge-torrent/deluge"},"source code"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../data/lab/support$ git clone https://github.com/deluge-torrent/deluge\nCloning into 'deluge'...\n[...]\n\nstudent@os:~/.../data/lab/support$ cd deluge/\n\nstudent@os:~/.../lab/support/deluge$ ls\nAUTHORS deluge docs gen_web_gettext.py MANIFEST.in msgfmt.py pyproject.toml requirements-dev.txt requirements.txt setup.py version.py\nCHANGELOG.md DEPENDS.md generate_pot.py LICENSE minify_web_js.py packaging README.md requirements-tests.txt setup.cfg tox.ini\n")),(0,o.kt)("p",null,"And enter the ",(0,o.kt)("inlineCode",{parentName:"p"},"deluge/core/")," subdirectory:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/deluge$ cd deluge/core/\n\nstudent@os:~/.../deluge/deluge/core$ ls\nalertmanager.py core.py daemon.py filtermanager.py pluginmanager.py rpcserver.py torrent.py\nauthmanager.py daemon_entry.py eventmanager.py __init__.py preferencesmanager.py torrentmanager.py\n")),(0,o.kt)("p",null,"Most files in the subdirectory have a class defined.\nWe can search for instantiations of that class using ",(0,o.kt)("inlineCode",{parentName:"p"},"grep"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../deluge/deluge/core$ grep -rn 'Torrent('\ntorrentmanager.py:644: torrent = Torrent(handle, options, state, filename, magne\n\nstudent@os:~/.../deluge/deluge/core$ grep -rn 'TorrentManager('\ncore.py:139: self.torrentmanager = TorrentManager()\ntorrentmanager.py:135:class TorrentManager(component.Component):\n")),(0,o.kt)("p",null,"This gives us an overview of when memory is allocated in Deluge / Python."),(0,o.kt)("h3",{id:"practice-2"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Investigate the lines shown to contain instantiations of classes.\nExplore the source code and understand their placements in the source code.")),(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Find out other classes and search for their instantiation in the source code."))),(0,o.kt)("h2",{id:"app-investigation-servo"},"App Investigation: Servo"),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"https://servo.org/"},"Servo")," is a browser engine written in Rust that provides reusable components to implement web standards."),(0,o.kt)("p",null,"We do not clone the repository, since it's very large."),(0,o.kt)("p",null,"We find information about allocator used, by accessing the ",(0,o.kt)("inlineCode",{parentName:"p"},"components/allocator/")," in ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/servo/servo/tree/master/components/allocator"},"its source code"),".\nIn ",(0,o.kt)("inlineCode",{parentName:"p"},"Cargo.toml")," we see that it requires ",(0,o.kt)("inlineCode",{parentName:"p"},"jemalloc")," for non-Windows implementations and the standard Windows API (called ",(0,o.kt)("inlineCode",{parentName:"p"},"heapapi"),") for Windows:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-text"},'[...]\n[lib]\npath = "lib.rs"\n\n[target.\'cfg(not(windows))\'.dependencies]\njemalloc-sys = { version = "0.3.2" }\n\n[target.\'cfg(windows)\'.dependencies]\nwinapi = { version = "0.3", features = ["heapapi"] }\nust: https://github.com/servo/servo\n')),(0,o.kt)("p",null,"In ",(0,o.kt)("inlineCode",{parentName:"p"},"lib.rs"),", in ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/servo/servo/blob/master/components/allocator/lib.rs#L70"},(0,o.kt)("inlineCode",{parentName:"a"},"GlobalAlloc:alloc()"))," we see it is using ",(0,o.kt)("a",{parentName:"p",href:"https://jemalloc.net/jemalloc.3.html"},"the ",(0,o.kt)("inlineCode",{parentName:"a"},"mallocx")," custom function from ",(0,o.kt)("inlineCode",{parentName:"a"},"jemalloc()")),".\nSee ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/servo/servo/blob/master/components/allocator/lib.rs#L17"},"the initialization of ",(0,o.kt)("inlineCode",{parentName:"a"},"ffi")),"."),(0,o.kt)("p",null,"See the use of the allocator in the ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/servo/servo/blob/master/components/net/Cargo.toml"},(0,o.kt)("inlineCode",{parentName:"a"},"Cargo.toml")," file in the ",(0,o.kt)("inlineCode",{parentName:"a"},"net")," component"),".\nSearch for the ",(0,o.kt)("em",{parentName:"p"},"alloc")," string."),(0,o.kt)("h3",{id:"practice-3"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},"Look for uses of the allocator in other components of Servo.")),(0,o.kt)("h2",{id:"investigation-alocator-in-the-d-programming-language"},"Investigation: Alocator in the D Programming Language"),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"https://github.com/dlang/phobos"},"Phobos")," is the standard library that comes with the D programming language compiler."),(0,o.kt)("p",null,"Let's clone the source code:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../data/lab/support$ git clone https://github.com/dlang/phobos\n[...]\n\nstudent@os:~/.../data/lab/support$ cd phobos/\n\nstudent@os:~/.../lab/support/phobos$ ls\nazure-pipelines.yml changelog CODEOWNERS CONTRIBUTING.md dub.sdl etc index.dd LICENSE_1_0.txt posix.mak project.ddoc README.md std test unittest.d win32.mak win64.mak\n")),(0,o.kt)("p",null,"And enter ",(0,o.kt)("inlineCode",{parentName:"p"},"std/experimental/allocator/")," to browse information about the allocator:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/phobos$ cd std/experimental/allocator/\n\nstudent@os:~/.../std/experimental/allocator$ ls\nbuilding_blocks common.d gc_allocator.d mallocator.d mmap_allocator.d package.d showcase.d typed.d\n")),(0,o.kt)("p",null,"We then do a search of the ",(0,o.kt)("inlineCode",{parentName:"p"},"allocate(")," string to find instances of allocation calls:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../std/experimental/allocator$ grep -r 'allocate('\n[...]\n")),(0,o.kt)("p",null,"We see that there are definitions of the function (as expected) as part of ",(0,o.kt)("inlineCode",{parentName:"p"},"...allocator")," files: ",(0,o.kt)("inlineCode",{parentName:"p"},"mallocator.d"),", ",(0,o.kt)("inlineCode",{parentName:"p"},"gc_allocator.d"),", ",(0,o.kt)("inlineCode",{parentName:"p"},"mmap_allocator.d"),".\nBrowse the functions and look for implementations of the ",(0,o.kt)("inlineCode",{parentName:"p"},"allocate()")," function."),(0,o.kt)("h3",{id:"practice-4"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},"Do a similar search and then source code browsing for the ",(0,o.kt)("inlineCode",{parentName:"li"},"deallocate()")," function.")),(0,o.kt)("h2",{id:"app-investigation-git"},"App Investigation: Git"),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"https://git-scm.com/"},"Git")," is among the most used source code management system, powering development infrastructures such as ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/"},"GitHub"),", ",(0,o.kt)("a",{parentName:"p",href:"https://about.gitlab.com/"},"GitLab")," and ",(0,o.kt)("a",{parentName:"p",href:"https://bitbucket.org/"},"Bitbucket"),"."),(0,o.kt)("p",null,"Let's clone ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/git/git"},"the repository"),".\nNote that it is about 200MB large:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../data/lab/support$ git clone https://github.com/git/git\n[...]\n\nstudent@os:~/.../data/lab/support$ cd git/\n")),(0,o.kt)("p",null,"We look of uses of ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/git$ grep -r 'malloc(' .\n")),(0,o.kt)("p",null,"We see there are multiple calls to the ",(0,o.kt)("inlineCode",{parentName:"p"},"xmalloc()")," function, which is likely a wrapper for ",(0,o.kt)("inlineCode",{parentName:"p"},"malloc()"),".\nWe search for the definition of ",(0,o.kt)("inlineCode",{parentName:"p"},"xmalloc()"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/git$ grep -rn 'xmalloc(' . | grep -v ';'\n./commit.c:188: graft = xmalloc(st_add(sizeof(*graft),\n./add-interactive.c:157: list->sorted.items = xmalloc(st_mult(sizeof(*list->sorted.items),\n./Documentation/RelNotes/2.24.0.txt:91: * xmalloc() used to have a mechanism to ditch memory and address\n./Documentation/RelNotes/2.24.0.txt:210: xmalloc() wrapper, as the rest of the system, for consistency.\n./Documentation/RelNotes/2.34.0.txt:230: * mmap() imitation used to call xmalloc() that dies upon malloc()\n./Documentation/RelNotes/2.33.1.txt:44: * mmap() imitation used to call xmalloc() that dies upon malloc()\n./diffcore-delta.c:56: new_spanhash = xmalloc(st_add(sizeof(*orig),\n./diffcore-delta.c:135: hash = xmalloc(st_add(sizeof(*hash),\n./kwset.c:41:/* adapter for `xmalloc()`, which takes `size_t`, not `long` */\n./builtin/fast-import.c:461: b = xmalloc(sizeof(struct object_entry_pool)\n./hashmap.h:311: * your structure was allocated with xmalloc(), you can just free(3) it,\n./xdiff/xdiff.h:122:#define xdl_malloc(x) xmalloc(x)\n./wrapper.c:45:static void *do_xmalloc(size_t size, int gentle)\n./wrapper.c:70:void *xmalloc(size_t size)\n./contrib/credential/wincred/git-credential-wincred.c:26:static void *xmalloc(size_t size)\nBinary file ./.git/objects/pack/pack-c587b9f11a82bc4d49848d74132e60ea4dbeb177.pack matches\n./git-compat-util.h:1046:# define xalloca(size) (xmalloc(size))\n./git-compat-util.h:1086:#define ALLOC_ARRAY(x, alloc) (x) = xmalloc(st_mult(sizeof(*(x)), (alloc)))\n./read-cache.c:3768: ieot = xmalloc(sizeof(struct index_entry_offset_table)\n")),(0,o.kt)("p",null,"Line ",(0,o.kt)("inlineCode",{parentName:"p"},"./wrapper.c:70")," is the one with the definition of the ",(0,o.kt)("inlineCode",{parentName:"p"},"xmalloc()")," function.\nIt makes a call of the ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/git/git/blob/master/wrapper.c#L45"},(0,o.kt)("inlineCode",{parentName:"a"},"do_xmalloc()")," function"),", that makes extra checks.\nAlso, if the ",(0,o.kt)("inlineCode",{parentName:"p"},"XMALLOC_POISON"),' macro is defined, all the allocated data is overwritten with a "poison" value (',(0,o.kt)("inlineCode",{parentName:"p"},"0xA5"),").\nThis is useful for early detection of memory-related issues, although, evidently, it adds overhead."),(0,o.kt)("p",null,"We can look for parts of the source code with the largest number of uses of ",(0,o.kt)("inlineCode",{parentName:"p"},"xmalloc()"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/git$ grep -rc 'xmalloc(' . | grep -v ':0' | sort -n -t ':' -k 20\n[...]\n./compat/mingw.c:6\n./submodule-config.c:6\n./merge-recursive.c:7\n")),(0,o.kt)("p",null,"We can look into the ",(0,o.kt)("a",{parentName:"p",href:"https://github.com/git/git/blob/master/merge-recursive.c"},(0,o.kt)("inlineCode",{parentName:"a"},"merge-recursive.c")," file")," for uses of the ",(0,o.kt)("inlineCode",{parentName:"p"},"xmalloc()")," function."),(0,o.kt)("h3",{id:"practice-5"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Do the same actions as above for the ",(0,o.kt)("inlineCode",{parentName:"p"},"mmap()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"xmmap()")," function calls."),(0,o.kt)("p",{parentName:"li"},"Note that these are not memory allocation calls, since a valid ",(0,o.kt)("inlineCode",{parentName:"p"},"fd")," file argument is passed.\nThese are file mapping calls, that we will talk more as part of the I/O chapter."))))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/14580385.be011ac3.js b/17/assets/js/14580385.be011ac3.js new file mode 100644 index 0000000000..1b90dca5a6 --- /dev/null +++ b/17/assets/js/14580385.be011ac3.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6632],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>h});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function s(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var l=r.createContext({}),p=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(l.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,l=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),u=p(n),f=a,h=u["".concat(l,".").concat(f)]||u[f]||m[f]||o;return n?r.createElement(h,s(s({ref:t},c),{},{components:n})):r.createElement(h,s({ref:t},c))}));function h(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,s=new Array(o);s[0]=f;var i={};for(var l in t)hasOwnProperty.call(t,l)&&(i[l]=t[l]);i.originalType=e,i[u]="string"==typeof e?e:a,s[1]=i;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>s,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const o={},s="Arena",i={unversionedId:"Lab/IO/arena",id:"Lab/IO/arena",title:"Arena",description:"Open File Structure in the Kernel",source:"@site/docs/Lab/IO/arena.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/arena",permalink:"/operating-systems/17/Lab/IO/arena",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"I/O Multiplexing",permalink:"/operating-systems/17/Lab/IO/io-multiplexing"},next:{title:"Application Interaction",permalink:"/operating-systems/17/Lab/Application Interaction/"}},l={},p=[{value:"Open File Structure in the Kernel",id:"open-file-structure-in-the-kernel",level:2},{value:"Mini-shell with Blackjack and Pipes",id:"mini-shell-with-blackjack-and-pipes",level:2},{value:"To Drop or Not to Drop?",id:"to-drop-or-not-to-drop",level:2}],c={toc:p},u="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(u,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"arena"},"Arena"),(0,a.kt)("h2",{id:"open-file-structure-in-the-kernel"},"Open File Structure in the Kernel"),(0,a.kt)("p",null,'The "open file" ',(0,a.kt)("inlineCode",{parentName:"p"},"struct")," in the Linux kernel is called ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L940"},(0,a.kt)("inlineCode",{parentName:"a"},"struct file")),"\nIts most important fields are:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},"struct file {\n struct path f_path;\n /* Identifier within the filesystem. */\n struct inode *f_inode;\n\n /**\n * Contains pointers to functions that implement operations that\n * correspond to syscalls, such as `read()`, `write()`, `lseek()` etc.\n */\n const struct file_operations *f_op;\n\n /**\n * Reference count. A `struct file` is deallocated when this reaches 0,\n * i.e. nobody uses it anymore.\n */\n atomic_long_t f_count;\n\n /* Those passed to `open()`. */\n unsigned int f_flags;\n fmode_t f_mode;\n\n /* Cursor from where reads/writes are made */\n loff_t f_pos;\n /* To allow `f_pos` to be modified atomically. */\n struct mutex f_pos_lock;\n}\n")),(0,a.kt)("p",null,"As mentioned above, ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L2093"},(0,a.kt)("inlineCode",{parentName:"a"},"struct file_operations"))," contains function pointers that well-known syscalls such as ",(0,a.kt)("inlineCode",{parentName:"p"},"read()")," end up calling.\nEach filesystem needs to define its own implementations of these functions.\nSome of the most widely known ",(0,a.kt)("inlineCode",{parentName:"p"},"file_operations")," are listed below.\nBy now, you should recognise most of them:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},"struct file_operations {\n loff_t (*llseek) (struct file *, loff_t, int);\n ssize_t (*read) (struct file *, char __user *, size_t, loff_t *);\n ssize_t (*write) (struct file *, const char __user *, size_t, loff_t *);\n int (*mmap) (struct file *, struct vm_area_struct *);\n unsigned long mmap_supported_flags;\n int (*open) (struct inode *, struct file *);\n int (*flush) (struct file *, fl_owner_t id);\n int (*fsync) (struct file *, loff_t, loff_t, int datasync);\n int (*fasync) (int, struct file *, int);\n}\n")),(0,a.kt)("h2",{id:"mini-shell-with-blackjack-and-pipes"},"Mini-shell with Blackjack and Pipes"),(0,a.kt)("p",null,"OK, there will be no Blackjack...\nfor now at least.\nBut there ",(0,a.kt)("strong",{parentName:"p"},"will")," be pipes.\nNavigate back to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/mini-shell/mini_shell.c")," and add support for piping 2 commands together like this:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"> cat bosses.txt | head -n 5\nDarkeater Midir\nSlave Knight Gael\nNameless King\nDancer Of The Boreal Valley\nAncient Dragon\n")),(0,a.kt)("h2",{id:"to-drop-or-not-to-drop"},"To Drop or Not to Drop?"),(0,a.kt)("p",null,"Remember ",(0,a.kt)("inlineCode",{parentName:"p"},"support/buffering/benchmark_buffering.sh")," or ",(0,a.kt)("inlineCode",{parentName:"p"},"support/file-mappings/benchmark_cp.sh"),".\nThey both used this line:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-bash"},'sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"\n')),(0,a.kt)("p",null,"Note that ",(0,a.kt)("inlineCode",{parentName:"p"},"sync")," has a ",(0,a.kt)("a",{parentName:"p",href:"https://linux.die.net/man/8/sync"},"man page")," and it partially explains what's going on:"),(0,a.kt)("blockquote",null,(0,a.kt)("p",{parentName:"blockquote"},"The kernel keeps data in memory to avoid doing (relatively slow) disk reads and writes. This improves performance")),(0,a.kt)("p",null,"So the kernel does ",(0,a.kt)("strong",{parentName:"p"},"even more ",(0,a.kt)("a",{parentName:"strong",href:"/operating-systems/17/Lab/IO/io-internals#io-buffering"},"buffering"),"!"),"\nBut this time, it's not at the syscall level, like with ",(0,a.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"write()"),".\nAnd it's used a bit differently."),(0,a.kt)("p",null,"While buffering is a means of either receiving data in advance (for reading) or committing it retroactively (for writing) to speed up subsequent syscalls that use the ",(0,a.kt)("strong",{parentName:"p"},"next data"),", caching is a means of speeding up calls that use the ",(0,a.kt)("strong",{parentName:"p"},"same data"),".\nJust like your browser caches the pages you visit so you refresh them faster or your CPU caches your most recently accessed addresses, so does your OS ",(0,a.kt)("strong",{parentName:"p"},"with your files"),"."),(0,a.kt)("p",null,"Some files are read more often than others: logs, some configs etc.\nUpon encountering a first request (read / write) to a file, the kernel stores chunks of them in its memory so that subsequent requests can receive / modify the data in the RAM rather than waiting for the slow disk.\nThis makes I/O faster."),(0,a.kt)("p",null,'The line from which this discussion started invalidates those caches and forces the OS to perform I/O operations "the slow way" by interrogating the disk.\nThe scripts use it to benchmark only the C code, not the OS.'),(0,a.kt)("p",null,"To see just how much faster this type of caching is, navigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/buffering/benchmark_buffering.sh")," once again and comment-out the line with ",(0,a.kt)("inlineCode",{parentName:"p"},'sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"'),".\nNow run the script ",(0,a.kt)("strong",{parentName:"p"},"a few times")," and compare the results.\nYou should see some drastic improvements in the running times, such as:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/file-mappings$ ./benchmark_cp.sh\nmake: Nothing to be done for 'all'.\nBenchmarking mmap_cp on a 1 GB file...\n\nreal 0m13,299s\nuser 0m0,522s\nsys 0m1,695s\nBenchmarking cp on a 1 GB file...\n\nreal 0m10,921s\nuser 0m0,013s\nsys 0m1,301s\n\n\nstudent@os:/.../support/file-mappings$ ./benchmark_cp.sh\nmake: Nothing to be done for 'all'.\nBenchmarking mmap_cp on a 1 GB file...\n\nreal 0m1,286s\nuser 0m0,174s\nsys 0m0,763s\nBenchmarking cp on a 1 GB file...\n\nreal 0m5,411s\nuser 0m0,012s\nsys 0m1,201s\n")),(0,a.kt)("p",null,"Each subsequent benchmark actually reads the data from the caches populated or refreshed by the previous one."),(0,a.kt)("p",null,"You can use ",(0,a.kt)("inlineCode",{parentName:"p"},"free -h")," to view how much data your kernel is caching.\nLook at the ",(0,a.kt)("inlineCode",{parentName:"p"},"buff/cache")," column.\nOne possible output is shown below.\nIt says the OS is caching 7 GB of data."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ free -h\n total used free shared buff/cache available\nMem: 15Gi 8,1Gi 503Mi 691Mi 7,0Gi 6,5Gi\nSwap: 7,6Gi 234Mi 7,4Gi\n")))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/14eb3368.f8efd3cc.js b/17/assets/js/14eb3368.f8efd3cc.js new file mode 100644 index 0000000000..3e6187c83d --- /dev/null +++ b/17/assets/js/14eb3368.f8efd3cc.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9817],{1986:(e,t,a)=>{a.d(t,{Z:()=>p});var n=a(7462),r=a(7294),i=a(6010),l=a(5281),s=a(2802),c=a(8596),o=a(9960),m=a(4996),d=a(5999);function u(e){return r.createElement("svg",(0,n.Z)({viewBox:"0 0 24 24"},e),r.createElement("path",{d:"M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z",fill:"currentColor"}))}const h={breadcrumbsContainer:"breadcrumbsContainer_Z_bl",breadcrumbHomeIcon:"breadcrumbHomeIcon_OVgt"};function b(e){let{children:t,href:a,isLast:n}=e;const i="breadcrumbs__link";return n?r.createElement("span",{className:i,itemProp:"name"},t):a?r.createElement(o.Z,{className:i,href:a,itemProp:"item"},r.createElement("span",{itemProp:"name"},t)):r.createElement("span",{className:i},t)}function v(e){let{children:t,active:a,index:l,addMicrodata:s}=e;return r.createElement("li",(0,n.Z)({},s&&{itemScope:!0,itemProp:"itemListElement",itemType:"https://schema.org/ListItem"},{className:(0,i.Z)("breadcrumbs__item",{"breadcrumbs__item--active":a})}),t,r.createElement("meta",{itemProp:"position",content:String(l+1)}))}function g(){const e=(0,m.Z)("/");return r.createElement("li",{className:"breadcrumbs__item"},r.createElement(o.Z,{"aria-label":(0,d.I)({id:"theme.docs.breadcrumbs.home",message:"Home page",description:"The ARIA label for the home page in the breadcrumbs"}),className:(0,i.Z)("breadcrumbs__link",h.breadcrumbsItemLink),href:e},r.createElement(u,{className:h.breadcrumbHomeIcon})))}function p(){const e=(0,s.s1)(),t=(0,c.Ns)();return e?r.createElement("nav",{className:(0,i.Z)(l.k.docs.docBreadcrumbs,h.breadcrumbsContainer),"aria-label":(0,d.I)({id:"theme.docs.breadcrumbs.navAriaLabel",message:"Breadcrumbs",description:"The ARIA label for the breadcrumbs"})},r.createElement("ul",{className:"breadcrumbs",itemScope:!0,itemType:"https://schema.org/BreadcrumbList"},t&&r.createElement(g,null),e.map(((t,a)=>{const n=a===e.length-1;return r.createElement(v,{key:a,active:n,index:a,addMicrodata:!!t.href},r.createElement(b,{href:t.href,isLast:n},t.label))})))):null}},4228:(e,t,a)=>{a.r(t),a.d(t,{default:()=>I});var n=a(7294),r=a(1944),i=a(2802),l=a(4996),s=a(6010),c=a(9960),o=a(3919),m=a(5999);const d={cardContainer:"cardContainer_fWXF",cardTitle:"cardTitle_rnsV",cardDescription:"cardDescription_PWke"};function u(e){let{href:t,children:a}=e;return n.createElement(c.Z,{href:t,className:(0,s.Z)("card padding--lg",d.cardContainer)},a)}function h(e){let{href:t,icon:a,title:r,description:i}=e;return n.createElement(u,{href:t},n.createElement("h2",{className:(0,s.Z)("text--truncate",d.cardTitle),title:r},a," ",r),i&&n.createElement("p",{className:(0,s.Z)("text--truncate",d.cardDescription),title:i},i))}function b(e){let{item:t}=e;const a=(0,i.Wl)(t);return a?n.createElement(h,{href:a,icon:"\ud83d\uddc3\ufe0f",title:t.label,description:(0,m.I)({message:"{count} items",id:"theme.docs.DocCard.categoryDescription",description:"The default description for a category card in the generated index about how many items this category includes"},{count:t.items.length})}):null}function v(e){let{item:t}=e;const a=(0,o.Z)(t.href)?"\ud83d\udcc4\ufe0f":"\ud83d\udd17",r=(0,i.xz)(t.docId??void 0);return n.createElement(h,{href:t.href,icon:a,title:t.label,description:r?.description})}function g(e){let{item:t}=e;switch(t.type){case"link":return n.createElement(v,{item:t});case"category":return n.createElement(b,{item:t});default:throw new Error(`unknown item type ${JSON.stringify(t)}`)}}function p(e){let{className:t}=e;const a=(0,i.jA)();return n.createElement(E,{items:a.items,className:t})}function E(e){const{items:t,className:a}=e;if(!t)return n.createElement(p,e);const r=(0,i.MN)(t);return n.createElement("section",{className:(0,s.Z)("row",a)},r.map(((e,t)=>n.createElement("article",{key:t,className:"col col--6 margin-bottom--lg"},n.createElement(g,{item:e})))))}var f=a(4966),N=a(3120),Z=a(4364),k=a(1986),L=a(2503);const _={generatedIndexPage:"generatedIndexPage_vN6x",list:"list_eTzJ",title:"title_kItE"};function T(e){let{categoryGeneratedIndex:t}=e;return n.createElement(r.d,{title:t.title,description:t.description,keywords:t.keywords,image:(0,l.Z)(t.image)})}function x(e){let{categoryGeneratedIndex:t}=e;const a=(0,i.jA)();return n.createElement("div",{className:_.generatedIndexPage},n.createElement(N.Z,null),n.createElement(k.Z,null),n.createElement(Z.Z,null),n.createElement("header",null,n.createElement(L.Z,{as:"h1",className:_.title},t.title),t.description&&n.createElement("p",null,t.description)),n.createElement("article",{className:"margin-top--lg"},n.createElement(E,{items:a.items,className:_.list})),n.createElement("footer",{className:"margin-top--lg"},n.createElement(f.Z,{previous:t.navigation.previous,next:t.navigation.next})))}function I(e){return n.createElement(n.Fragment,null,n.createElement(T,e),n.createElement(x,e))}},4966:(e,t,a)=>{a.d(t,{Z:()=>o});var n=a(7462),r=a(7294),i=a(5999),l=a(6010),s=a(9960);function c(e){const{permalink:t,title:a,subLabel:n,isNext:i}=e;return r.createElement(s.Z,{className:(0,l.Z)("pagination-nav__link",i?"pagination-nav__link--next":"pagination-nav__link--prev"),to:t},n&&r.createElement("div",{className:"pagination-nav__sublabel"},n),r.createElement("div",{className:"pagination-nav__label"},a))}function o(e){const{previous:t,next:a}=e;return r.createElement("nav",{className:"pagination-nav docusaurus-mt-lg","aria-label":(0,i.I)({id:"theme.docs.paginator.navAriaLabel",message:"Docs pages navigation",description:"The ARIA label for the docs pagination"})},t&&r.createElement(c,(0,n.Z)({},t,{subLabel:r.createElement(i.Z,{id:"theme.docs.paginator.previous",description:"The label used to navigate to the previous doc"},"Previous")})),a&&r.createElement(c,(0,n.Z)({},a,{subLabel:r.createElement(i.Z,{id:"theme.docs.paginator.next",description:"The label used to navigate to the next doc"},"Next"),isNext:!0})))}},4364:(e,t,a)=>{a.d(t,{Z:()=>c});var n=a(7294),r=a(6010),i=a(5999),l=a(5281),s=a(4477);function c(e){let{className:t}=e;const a=(0,s.E)();return a.badge?n.createElement("span",{className:(0,r.Z)(t,l.k.docs.docVersionBadge,"badge badge--secondary")},n.createElement(i.Z,{id:"theme.docs.versionBadge.label",values:{versionLabel:a.label}},"Version: {versionLabel}")):null}},3120:(e,t,a)=>{a.d(t,{Z:()=>g});var n=a(7294),r=a(6010),i=a(2263),l=a(9960),s=a(5999),c=a(143),o=a(5281),m=a(373),d=a(4477);const u={unreleased:function(e){let{siteTitle:t,versionMetadata:a}=e;return n.createElement(s.Z,{id:"theme.docs.versions.unreleasedVersionLabel",description:"The label used to tell the user that he's browsing an unreleased doc version",values:{siteTitle:t,versionLabel:n.createElement("b",null,a.label)}},"This is unreleased documentation for {siteTitle} {versionLabel} version.")},unmaintained:function(e){let{siteTitle:t,versionMetadata:a}=e;return n.createElement(s.Z,{id:"theme.docs.versions.unmaintainedVersionLabel",description:"The label used to tell the user that he's browsing an unmaintained doc version",values:{siteTitle:t,versionLabel:n.createElement("b",null,a.label)}},"This is documentation for {siteTitle} {versionLabel}, which is no longer actively maintained.")}};function h(e){const t=u[e.versionMetadata.banner];return n.createElement(t,e)}function b(e){let{versionLabel:t,to:a,onClick:r}=e;return n.createElement(s.Z,{id:"theme.docs.versions.latestVersionSuggestionLabel",description:"The label used to tell the user to check the latest version",values:{versionLabel:t,latestVersionLink:n.createElement("b",null,n.createElement(l.Z,{to:a,onClick:r},n.createElement(s.Z,{id:"theme.docs.versions.latestVersionLinkLabel",description:"The label used for the latest version suggestion link label"},"latest version")))}},"For up-to-date documentation, see the {latestVersionLink} ({versionLabel}).")}function v(e){let{className:t,versionMetadata:a}=e;const{siteConfig:{title:l}}=(0,i.Z)(),{pluginId:s}=(0,c.gA)({failfast:!0}),{savePreferredVersionName:d}=(0,m.J)(s),{latestDocSuggestion:u,latestVersionSuggestion:v}=(0,c.Jo)(s),g=u??(p=v).docs.find((e=>e.id===p.mainDocId));var p;return n.createElement("div",{className:(0,r.Z)(t,o.k.docs.docVersionBanner,"alert alert--warning margin-bottom--md"),role:"alert"},n.createElement("div",null,n.createElement(h,{siteTitle:l,versionMetadata:a})),n.createElement("div",{className:"margin-top--md"},n.createElement(b,{versionLabel:v.label,to:g.path,onClick:()=>d(v.name)})))}function g(e){let{className:t}=e;const a=(0,d.E)();return a.banner?n.createElement(v,{className:t,versionMetadata:a}):null}},2503:(e,t,a)=>{a.d(t,{Z:()=>o});var n=a(7462),r=a(7294),i=a(6010),l=a(5999),s=a(6668);const c={anchorWithStickyNavbar:"anchorWithStickyNavbar_LWe7",anchorWithHideOnScrollNavbar:"anchorWithHideOnScrollNavbar_WYt5"};function o(e){let{as:t,id:a,...o}=e;const{navbar:{hideOnScroll:m}}=(0,s.L)();return"h1"!==t&&a?r.createElement(t,(0,n.Z)({},o,{className:(0,i.Z)("anchor",m?c.anchorWithHideOnScrollNavbar:c.anchorWithStickyNavbar),id:a}),o.children,r.createElement("a",{className:"hash-link",href:`#${a}`,title:(0,l.I)({id:"theme.common.headingLinkTitle",message:"Direct link to heading",description:"Title for link to heading"})},"\u200b")):r.createElement(t,(0,n.Z)({},o,{id:void 0}))}}}]); \ No newline at end of file diff --git a/17/assets/js/14ee73ae.43bb5f71.js b/17/assets/js/14ee73ae.43bb5f71.js new file mode 100644 index 0000000000..b5a7e08c41 --- /dev/null +++ b/17/assets/js/14ee73ae.43bb5f71.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2202],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>h});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var u=r.createContext({}),l=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},p=function(e){var t=l(e.components);return r.createElement(u.Provider,{value:t},e.children)},c="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,u=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),c=l(n),f=o,h=c["".concat(u,".").concat(f)]||c[f]||d[f]||a;return n?r.createElement(h,i(i({ref:t},p),{},{components:n})):r.createElement(h,i({ref:t},p))}));function h(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=f;var s={};for(var u in t)hasOwnProperty.call(t,u)&&(s[u]=t[u]);s.originalType=e,s[c]="string"==typeof e?e:o,i[1]=s;for(var l=2;l{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>d,frontMatter:()=>a,metadata:()=>s,toc:()=>l});var r=n(7462),o=(n(7294),n(3905));const a={},i="Firefox: TCP or UDP?",s={unversionedId:"Lab/IO/quiz/firefox-tcp-udp",id:"Lab/IO/quiz/firefox-tcp-udp",title:"Firefox: TCP or UDP?",description:"Question Text",source:"@site/docs/Lab/IO/quiz/firefox-tcp-udp.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/firefox-tcp-udp",permalink:"/operating-systems/17/Lab/IO/quiz/firefox-tcp-udp",draft:!1,tags:[],version:"current",frontMatter:{}},u={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:l},c="wrapper";function d(e){let{components:t,...n}=e;return(0,o.kt)(c,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"firefox-tcp-or-udp"},"Firefox: TCP or UDP?"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"If the user requests a text-based web page (such as ",(0,o.kt)("a",{parentName:"p",href:"https://open-education-hub.github.io/operating-systems/"},"our Operating Systems course"),"), should the browser transfer the content using TCP or UDP?\nWhat about video content, such as YouTube?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"TCP and UDP, respectively")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"UDP and TCP, respectively")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"both connections should use TCP")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"both connections should use UDP"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,'The "TCP vs UDP" question boils down to 2 things:'),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"is the data updated in real-time (multiple times per second)?"),(0,o.kt)("li",{parentName:"ul"},"can we afford a few errors / missing messages?")),(0,o.kt)("p",null,'If the answers to both questions is "Yes", then we should use UDP.\nIf they\'re "No", we should use TCP.\nHowever, if the answer to one question is "Yes" and the other one is "No", then it gets complicated.'),(0,o.kt)("p",null,"Luckily, in our cases, the answers are quite clear.\nWe don't update text-based content too often and since it needs to be precise, handling broken packets is important.\nOn the other hand, a streaming-based site, such as YouTube sends data in real time and thus a few errors here and there can be omitted.\nSo ",(0,o.kt)("a",{parentName:"p",href:"https://open-education-hub.github.io/operating-systems"},"https://open-education-hub.github.io/operating-systems")," is going to be served via TCP, while YouTube videos via UDP."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/1548d5c7.aa6dd86c.js b/17/assets/js/1548d5c7.aa6dd86c.js new file mode 100644 index 0000000000..9b5f6f3ed4 --- /dev/null +++ b/17/assets/js/1548d5c7.aa6dd86c.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4272],{3905:(e,t,a)=>{a.d(t,{Zo:()=>s,kt:()=>p});var l=a(7294);function i(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function I(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);t&&(l=l.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,l)}return a}function n(e){for(var t=1;t=0||(i[a]=e[a]);return i}(e,t);if(Object.getOwnPropertySymbols){var I=Object.getOwnPropertySymbols(e);for(l=0;l=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(i[a]=e[a])}return i}var d=l.createContext({}),c=function(e){var t=l.useContext(d),a=t;return e&&(a="function"==typeof e?e(t):n(n({},t),e)),a},s=function(e){var t=c(e.components);return l.createElement(d.Provider,{value:t},e.children)},Z="mdxType",r={inlineCode:"code",wrapper:function(e){var t=e.children;return l.createElement(l.Fragment,{},t)}},o=l.forwardRef((function(e,t){var a=e.components,i=e.mdxType,I=e.originalType,d=e.parentName,s=m(e,["components","mdxType","originalType","parentName"]),Z=c(a),o=i,p=Z["".concat(d,".").concat(o)]||Z[o]||r[o]||I;return a?l.createElement(p,n(n({ref:t},s),{},{components:a})):l.createElement(p,n({ref:t},s))}));function p(e,t){var a=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var I=a.length,n=new Array(I);n[0]=o;var m={};for(var d in t)hasOwnProperty.call(t,d)&&(m[d]=t[d]);m.originalType=e,m[Z]="string"==typeof e?e:i,n[1]=m;for(var c=2;c{a.r(t),a.d(t,{assets:()=>d,contentTitle:()=>n,default:()=>r,frontMatter:()=>I,metadata:()=>m,toc:()=>c});var l=a(7462),i=(a(7294),a(3905));const I={},n="I/O",m={unversionedId:"Lab/IO/overview",id:"Lab/IO/overview",title:"I/O",description:"We know that a compute system is a collection of hardware and software that modifies data.",source:"@site/docs/Lab/IO/overview.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/overview",permalink:"/operating-systems/17/Lab/IO/overview",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"IO",permalink:"/operating-systems/17/Lab/IO/"},next:{title:"File Handling",permalink:"/operating-systems/17/Lab/IO/file-handlers"}},d={},c=[],s={toc:c},Z="wrapper";function r(e){let{components:t,...I}=e;return(0,i.kt)(Z,(0,l.Z)({},s,I,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"io"},"I/O"),(0,i.kt)("p",null,"We know that a compute system is a collection of hardware and software that modifies data.\nThis data has to come from ",(0,i.kt)("em",{parentName:"p"},"somewhere"),".\nThis ",(0,i.kt)("em",{parentName:"p"},"somewhere")," is always outside the compute system:\nfiles, network packets, radio signals, sensor data."),(0,i.kt)("p",null,"A compute system without output is nearly useless.\nIt will always run the same code on the same data and, thus, produce the same result.\nThis may be useful in some narrow cases, such as calculating the decimals of Pi.\nHowever, for more real-world-facing applications such as web servers, operating systems and databases inputs and outputs are mandatory."),(0,i.kt)("p",null,"The most simplistic representation of a compute system is a black box that receives some input and delivers some output."),(0,i.kt)("p",null,(0,i.kt)("img",{alt:"Compute System - Oversimplified",src:a(1259).Z,width:"352",height:"102"})),(0,i.kt)("p",null,"In this session, we will look into how a compute system interacts with the outside world to get and produce these inputs and outputs."),(0,i.kt)("ol",null,(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/file-handlers"},"File Handlers")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/file-descriptors"},"File Descriptors")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/redirections"},"Redirections")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/pipes"},"Pipes")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/local-io-in-action"},"Local I/O in Action")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/remote-io"},"Remote I/O")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/networking-101"},"Networking 101")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/client-server-model"},"Client-Server Model")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/beyond-network-sockets"},"Beyond Network Sockets")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/file-mappings"},"File Mappings")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/io-internals"},"IO Internals")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/zero-copy"},"Zero-copy")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/async-io"},"Asynchronous I/O")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/io-multiplexing"},"I/O Multiplexing")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/IO/arena"},"Arena"))))}r.isMDXComponent=!0},1259:(e,t,a)=>{a.d(t,{Z:()=>l});const l="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCEtLSBEbyBub3QgZWRpdCB0aGlzIGZpbGUgd2l0aCBlZGl0b3JzIG90aGVyIHRoYW4gZGlhZ3JhbXMubmV0IC0tPgo8IURPQ1RZUEUgc3ZnIFBVQkxJQyAiLS8vVzNDLy9EVEQgU1ZHIDEuMS8vRU4iICJodHRwOi8vd3d3LnczLm9yZy9HcmFwaGljcy9TVkcvMS4xL0RURC9zdmcxMS5kdGQiPgo8c3ZnIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgc3R5bGU9ImJhY2tncm91bmQtY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsiIHhtbG5zOnhsaW5rPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hsaW5rIiB2ZXJzaW9uPSIxLjEiIHdpZHRoPSIzNTJweCIgaGVpZ2h0PSIxMDJweCIgdmlld0JveD0iLTAuNSAtMC41IDM1MiAxMDIiIGNvbnRlbnQ9IiZsdDtteGZpbGUgaG9zdD0mcXVvdDthcHAuZGlhZ3JhbXMubmV0JnF1b3Q7IG1vZGlmaWVkPSZxdW90OzIwMjItMTEtMjJUMTQ6MzI6MTAuODkxWiZxdW90OyBhZ2VudD0mcXVvdDs1LjAgKFgxMSkmcXVvdDsgZXRhZz0mcXVvdDt3QTNVVEJxazR5R1l1OXpXWm01cCZxdW90OyB2ZXJzaW9uPSZxdW90OzIwLjUuMyZxdW90OyB0eXBlPSZxdW90O2RldmljZSZxdW90OyZndDsmbHQ7ZGlhZ3JhbSBpZD0mcXVvdDtUS093Yk5SN2pvNm8tNS1Ja0NMaSZxdW90OyBuYW1lPSZxdW90O1BhZ2UtMSZxdW90OyZndDs3WmZMY3Bzd0ZJYWZobVU2Z0hCdEwyMmN0RjMwTXZHaWJuWWFkQUthQ3NRSUVVT2V2cElSVjVGSjdZazdXWlR4Z3ZOTEhLVHovWkt3ZzhLMCtpUndubnpsQkpqanU2UnkwTTd4ZlMvd2ZVZi9YRkkzeXNwM0d5RVdsSmhPdmJDbnoyREV0bHRKQ1JTampwSnpKbWsrRmlPZVpSREprWWFGNE1keHQwZk94bS9OY1F5V3NJOHdzOVdmbE1qRXpHTGg5dnBub0hIU3Z0bHpUVXVLMjg1R0tCSk0rSEVnb1ZzSGhZSnoyZHlsVlFoTUY2K3RTL1BjM1F1dDNjQUVaUEp2SHRqc25oL3V4U0hKdnhFM2hQdkZ6VDQ1M0t5YUxFK1lsV2JDWnJDeWJpc2dlSmtSMEVrOEIyMlBDWld3ejNHa1c0K0t1ZElTbVRMVFhFakJmM2VWVW5QY1B2Sk1HcXhxZ0UwY2NzYkZLVHR5VDVmV0tXTUQvZTUwZFJuYmxveG5LdFBXakJxRWhPckZjbmhka1pVN2dhY2dSYTI2bUFjNmh4bGplbXNUSDN2TXFNV2NEQkczaExHeFZ0emw3cXV2Ymd5QU0yQjRGb3lRcDNrcFFZbjd1cENRL21NNEhZUUpuQTdhRzNCQS9waUQ3OW9jUEgrR3crcGFHUHpYMXdSa1pLTTNGeFZGREJjRmpjYVY3ckc0WjFUWjVxTnFLT3FEU1hNS2Z1bmd3NklOZDlXd2NWZWJxQmt3RUd0dm0xQlJteWtXTWNqWFRHblRHOUNaV3lTdEpvQmhTWi9HdzVnalp0N3dnMU0xd0g2UkxpZm1DQ2JRQzE2S0NNeFR3ODF2a2loWWpCTUYwMFhjMU1GS2RESlFOKzNMUFlYZWthY3FLZzhtcDc0Zk9FcEZ2YUYwY0w2ZkdpRHYxVS9CZXVJbnRGWXp2OHhSMDFSVGExN1pVWUhscUMrWk9pc3NXNm1kV2M0ZEE1TVRkYmpER3drekdtZmFqWW94S0gycjkzbXFQbzAycGlHbGhPalh6SjQ3WTd0ZS9sMXdoYU5lMmNrNllqN09tQTVkNjRSWld1eStsL0kvdkZsNDFqcWJnYmQ4RzNncTdEL0htM1hhLzZsQnQzOEEmbHQ7L2RpYWdyYW0mZ3Q7Jmx0Oy9teGZpbGUmZ3Q7Ij48ZGVmcy8+PGc+PHJlY3QgeD0iMSIgeT0iMSIgd2lkdGg9IjM1MCIgaGVpZ2h0PSIxMDAiIHJ4PSIxNSIgcnk9IjE1IiBmaWxsPSIjZmZmZmZmIiBzdHJva2U9Im5vbmUiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48cmVjdCB4PSIxMTEiIHk9IjExIiB3aWR0aD0iMTIwIiBoZWlnaHQ9IjgwIiByeD0iMTIiIHJ5PSIxMiIgZmlsbD0iIzAwMDAwMCIgc3Ryb2tlPSJyZ2IoMCwgMCwgMCkiIHN0cm9rZS13aWR0aD0iMiIgcG9pbnRlci1ldmVudHM9ImFsbCIvPjxnIHRyYW5zZm9ybT0idHJhbnNsYXRlKC0wLjUgLTAuNSkiPjxzd2l0Y2g+PGZvcmVpZ25PYmplY3Qgc3R5bGU9Im92ZXJmbG93OiB2aXNpYmxlOyB0ZXh0LWFsaWduOiBsZWZ0OyIgcG9pbnRlci1ldmVudHM9Im5vbmUiIHdpZHRoPSIxMDAlIiBoZWlnaHQ9IjEwMCUiIHJlcXVpcmVkRmVhdHVyZXM9Imh0dHA6Ly93d3cudzMub3JnL1RSL1NWRzExL2ZlYXR1cmUjRXh0ZW5zaWJpbGl0eSI+PGRpdiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMTk5OS94aHRtbCIgc3R5bGU9ImRpc3BsYXk6IGZsZXg7IGFsaWduLWl0ZW1zOiB1bnNhZmUgY2VudGVyOyBqdXN0aWZ5LWNvbnRlbnQ6IHVuc2FmZSBjZW50ZXI7IHdpZHRoOiAxMThweDsgaGVpZ2h0OiAxcHg7IHBhZGRpbmctdG9wOiA1MXB4OyBtYXJnaW4tbGVmdDogMTEycHg7Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyBmb250LXNpemU6IDBweDsgdGV4dC1hbGlnbjogY2VudGVyOyIgZGF0YS1kcmF3aW8tY29sb3JzPSJjb2xvcjogI0ZGRkZGRjsgIj48ZGl2IHN0eWxlPSJkaXNwbGF5OiBpbmxpbmUtYmxvY2s7IGZvbnQtc2l6ZTogMjBweDsgZm9udC1mYW1pbHk6IEhlbHZldGljYTsgY29sb3I6IHJnYigyNTUsIDI1NSwgMjU1KTsgbGluZS1oZWlnaHQ6IDEuMjsgcG9pbnRlci1ldmVudHM6IGFsbDsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgb3ZlcmZsb3ctd3JhcDogbm9ybWFsOyI+Q29tcHV0ZSBTeXN0ZW08L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iMTcxIiB5PSI1NyIgZmlsbD0iI0ZGRkZGRiIgZm9udC1mYW1pbHk9IkhlbHZldGljYSIgZm9udC1zaXplPSIyMHB4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5Db21wdXRlIFN5c3QuLi48L3RleHQ+PC9zd2l0Y2g+PC9nPjxwYXRoIGQ9Ik0gNjEgNTEgTCAxMDIuNzYgNTEiIGZpbGw9Im5vbmUiIHN0cm9rZT0icmdiKDAsIDAsIDApIiBzdHJva2Utd2lkdGg9IjIiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9InN0cm9rZSIvPjxwYXRoIGQ9Ik0gMTA4Ljc2IDUxIEwgMTAwLjc2IDU1IEwgMTAyLjc2IDUxIEwgMTAwLjc2IDQ3IFoiIGZpbGw9InJnYigwLCAwLCAwKSIgc3Ryb2tlPSJyZ2IoMCwgMCwgMCkiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PHBhdGggZD0iTSAyMzEgNTEgTCAyNzIuNzYgNTEiIGZpbGw9Im5vbmUiIHN0cm9rZT0icmdiKDAsIDAsIDApIiBzdHJva2Utd2lkdGg9IjIiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9InN0cm9rZSIvPjxwYXRoIGQ9Ik0gMjc4Ljc2IDUxIEwgMjcwLjc2IDU1IEwgMjcyLjc2IDUxIEwgMjcwLjc2IDQ3IFoiIGZpbGw9InJnYigwLCAwLCAwKSIgc3Ryb2tlPSJyZ2IoMCwgMCwgMCkiIHN0cm9rZS13aWR0aD0iMiIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PHJlY3QgeD0iMSIgeT0iMzYiIHdpZHRoPSI2MCIgaGVpZ2h0PSIzMCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJub25lIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTAuNSAtMC41KSI+PHN3aXRjaD48Zm9yZWlnbk9iamVjdCBzdHlsZT0ib3ZlcmZsb3c6IHZpc2libGU7IHRleHQtYWxpZ246IGxlZnQ7IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTogZmxleDsgYWxpZ24taXRlbXM6IHVuc2FmZSBjZW50ZXI7IGp1c3RpZnktY29udGVudDogdW5zYWZlIGNlbnRlcjsgd2lkdGg6IDU4cHg7IGhlaWdodDogMXB4OyBwYWRkaW5nLXRvcDogNTFweDsgbWFyZ2luLWxlZnQ6IDJweDsiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7IGZvbnQtc2l6ZTogMHB4OyB0ZXh0LWFsaWduOiBjZW50ZXI7IiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiAjMDAwMDAwOyAiPjxkaXYgc3R5bGU9ImRpc3BsYXk6IGlubGluZS1ibG9jazsgZm9udC1zaXplOiAyMHB4OyBmb250LWZhbWlseTogSGVsdmV0aWNhOyBjb2xvcjogcmdiKDAsIDAsIDApOyBsaW5lLWhlaWdodDogMS4yOyBwb2ludGVyLWV2ZW50czogYWxsOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyBvdmVyZmxvdy13cmFwOiBub3JtYWw7Ij5JbnB1dDwvZGl2PjwvZGl2PjwvZGl2PjwvZm9yZWlnbk9iamVjdD48dGV4dCB4PSIzMSIgeT0iNTciIGZpbGw9IiMwMDAwMDAiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2EiIGZvbnQtc2l6ZT0iMjBweCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+SW5wdXQ8L3RleHQ+PC9zd2l0Y2g+PC9nPjxyZWN0IHg9IjI4MSIgeT0iMzYiIHdpZHRoPSI3MCIgaGVpZ2h0PSIzMCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJub25lIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTAuNSAtMC41KSI+PHN3aXRjaD48Zm9yZWlnbk9iamVjdCBzdHlsZT0ib3ZlcmZsb3c6IHZpc2libGU7IHRleHQtYWxpZ246IGxlZnQ7IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTogZmxleDsgYWxpZ24taXRlbXM6IHVuc2FmZSBjZW50ZXI7IGp1c3RpZnktY29udGVudDogdW5zYWZlIGNlbnRlcjsgd2lkdGg6IDY4cHg7IGhlaWdodDogMXB4OyBwYWRkaW5nLXRvcDogNTFweDsgbWFyZ2luLWxlZnQ6IDI4MnB4OyI+PGRpdiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsgZm9udC1zaXplOiAwcHg7IHRleHQtYWxpZ246IGNlbnRlcjsiIGRhdGEtZHJhd2lvLWNvbG9ycz0iY29sb3I6ICMwMDAwMDA7ICI+PGRpdiBzdHlsZT0iZGlzcGxheTogaW5saW5lLWJsb2NrOyBmb250LXNpemU6IDIwcHg7IGZvbnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGxpbmUtaGVpZ2h0OiAxLjI7IHBvaW50ZXItZXZlbnRzOiBhbGw7IHdoaXRlLXNwYWNlOiBub3JtYWw7IG92ZXJmbG93LXdyYXA6IG5vcm1hbDsiPk91dHB1dDwvZGl2PjwvZGl2PjwvZGl2PjwvZm9yZWlnbk9iamVjdD48dGV4dCB4PSIzMTYiIHk9IjU3IiBmaWxsPSIjMDAwMDAwIiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjIwcHgiIHRleHQtYW5jaG9yPSJtaWRkbGUiPk91dHB1dDwvdGV4dD48L3N3aXRjaD48L2c+PC9nPjxzd2l0Y2g+PGcgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ii8+PGEgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoMCwtNSkiIHhsaW5rOmhyZWY9Imh0dHBzOi8vd3d3LmRpYWdyYW1zLm5ldC9kb2MvZmFxL3N2Zy1leHBvcnQtdGV4dC1wcm9ibGVtcyIgdGFyZ2V0PSJfYmxhbmsiPjx0ZXh0IHRleHQtYW5jaG9yPSJtaWRkbGUiIGZvbnQtc2l6ZT0iMTBweCIgeD0iNTAlIiB5PSIxMDAlIj5UZXh0IGlzIG5vdCBTVkcgLSBjYW5ub3QgZGlzcGxheTwvdGV4dD48L2E+PC9zd2l0Y2g+PC9zdmc+"}}]); \ No newline at end of file diff --git a/17/assets/js/15e6e258.17f6806f.js b/17/assets/js/15e6e258.17f6806f.js new file mode 100644 index 0000000000..a2655213d9 --- /dev/null +++ b/17/assets/js/15e6e258.17f6806f.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7482],{3321:e=>{e.exports=JSON.parse('{"title":"Lab","slug":"/Lab/","permalink":"/operating-systems/17/Lab/","navigation":{"previous":{"title":"Application-Interaction","permalink":"/operating-systems/17/Lecture/Application-Interaction"},"next":{"title":"Setting up the Lab Environment","permalink":"/operating-systems/17/Lab/lab-setup"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/15f618ea.88780c01.js b/17/assets/js/15f618ea.88780c01.js new file mode 100644 index 0000000000..cdb6444fc8 --- /dev/null +++ b/17/assets/js/15f618ea.88780c01.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5139],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>u});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function i(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var l=a.createContext({}),p=function(e){var t=a.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(l.Provider,{value:t},e.children)},d="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},k=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,l=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),d=p(n),k=r,u=d["".concat(l,".").concat(k)]||d[k]||m[k]||o;return n?a.createElement(u,i(i({ref:t},c),{},{components:n})):a.createElement(u,i({ref:t},c))}));function u(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,i=new Array(o);i[0]=k;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[d]="string"==typeof e?e:r,i[1]=s;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>s,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const o={},i="Beyond Network Sockets",s={unversionedId:"Lab/IO/beyond-network-sockets",id:"Lab/IO/beyond-network-sockets",title:"Beyond Network Sockets",description:"Up until this point, we first learned how to use the Berkeley Sockets API, then we learned about the client-server model, based on this API.",source:"@site/docs/Lab/IO/beyond-network-sockets.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/beyond-network-sockets",permalink:"/operating-systems/17/Lab/IO/beyond-network-sockets",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Client-Server Model",permalink:"/operating-systems/17/Lab/IO/client-server-model"},next:{title:"File Mappings",permalink:"/operating-systems/17/Lab/IO/file-mappings"}},l={},p=[{value:"UNIX Sockets",id:"unix-sockets",level:2},{value:"Practice: Receive from UNIX Socket",id:"practice-receive-from-unix-socket",level:3}],c={toc:p},d="wrapper";function m(e){let{components:t,...n}=e;return(0,r.kt)(d,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"beyond-network-sockets"},"Beyond Network Sockets"),(0,r.kt)("p",null,"Up until this point, we first learned how to use the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/remote-io#api---hail-berkeley-sockets"},"Berkeley Sockets API"),", then we learned about the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/client-server-model"},"client-server model"),", based on this API.\nSo now we know that sockets offer a ubiquitous interface for inter-process communication, which is great.\nA program written in Python can easily send data to another one written in C, D, Java, Haskell, you name it.\nHowever, in the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/networking-101"},"section dedicated to networking"),", we saw that it takes a whole stack of protocols to send this message from one process to the other.\nAs you might imagine, this is ",(0,r.kt)("strong",{parentName:"p"},"much slower even than local I/O using files"),"."),(0,r.kt)("p",null,"So far we've only used sockets for local communication, but in practice it is a bit counterproductive to use network sockets for local IPC due to their high latency.\nWouldn't it be great if we had a way to use the sockets API for local IPC without having to deal with this increased latency?\nWell, there is a way and it's called ",(0,r.kt)("strong",{parentName:"p"},"UNIX sockets"),"."),(0,r.kt)("h2",{id:"unix-sockets"},"UNIX Sockets"),(0,r.kt)("p",null,"UNIX sockets are created using the ",(0,r.kt)("inlineCode",{parentName:"p"},"socket()")," syscall and are bound ",(0,r.kt)("strong",{parentName:"p"},"TO A FILE")," instead of an IP and port using ",(0,r.kt)("inlineCode",{parentName:"p"},"bind()"),".\nYou may already see a similarity with ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/pipes#named-pipes---mkfifo"},"named pipes"),".\nJust like them, UNIX sockets don't work by writing data to the file (that would be slow), but instead the kernel retains the data they send internally so that ",(0,r.kt)("inlineCode",{parentName:"p"},"send()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"recv()")," can read it from the kernel's storage.\nYou can use ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," to read/write data from/to both network and UNIX sockets as well, by the way.\nThe differences between using ",(0,r.kt)("inlineCode",{parentName:"p"},"send()"),"/",(0,r.kt)("inlineCode",{parentName:"p"},"recv()")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"write()"),"/",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," are rather subtle and are described in ",(0,r.kt)("a",{parentName:"p",href:"https://stackoverflow.com/questions/1790750/what-is-the-difference-between-read-and-recv-and-between-send-and-write"},"this Stack Overflow thread"),"."),(0,r.kt)("p",null,"UNIX sockets are a feature of POSIX-compliant operating systems (e.g. Linux, macOS) and are not available on non-POSIX operating systems, such as Microsoft Windows.\nHowever, there are ",(0,r.kt)("a",{parentName:"p",href:"https://crates.io/crates/uds_windows"},"third-party libraries")," providing similar features to UNIX sockets in non-POSIX systems, but they might not have the same performance and reliability."),(0,r.kt)("h3",{id:"practice-receive-from-unix-socket"},"Practice: Receive from UNIX Socket"),(0,r.kt)("p",null,"Navigate to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/receive-challenges/receive_unix_socket.c"),".\nDon't write any code yet.\nLet's compare UNIX sockets with network sockets first:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"To create them, give ",(0,r.kt)("inlineCode",{parentName:"p"},"socket()")," the ",(0,r.kt)("inlineCode",{parentName:"p"},"PF_UNIX"),"/",(0,r.kt)("inlineCode",{parentName:"p"},"AF_UNIX")," instead of ",(0,r.kt)("inlineCode",{parentName:"p"},"PF_INET"),"/",(0,r.kt)("inlineCode",{parentName:"p"},"AF_INET"),".\nThe latter are used to create network sockets.\nYou can use ",(0,r.kt)("inlineCode",{parentName:"p"},"PF_*")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"AF_*"),' interchangeably.\nThey mean "protocol families" and "address families", respectively, but they are ',(0,r.kt)("a",{parentName:"p",href:"https://stackoverflow.com/a/6737450"},"essentially the same thing"),".\nBut apart from this, UNIX sockets can be set to work in both TCP (",(0,r.kt)("inlineCode",{parentName:"p"},"SOCK_STREAM"),") and UDP (",(0,r.kt)("inlineCode",{parentName:"p"},"SOCK_DGRAM"),") modes.")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"The ",(0,r.kt)("inlineCode",{parentName:"p"},"bind()")," call now takes a path as argument, as specified ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/unix.7.html"},"in the UNIX socket man page"),"."))),(0,r.kt)("p",null,"And this is it.\nEverything else is the same."),(0,r.kt)("p",null,"Now fill in the ",(0,r.kt)("inlineCode",{parentName:"p"},"TODO"),"s in ",(0,r.kt)("inlineCode",{parentName:"p"},"receive_unix_socket.c"),".\nUse the example at the bottom of ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/unix.7.html"},"the UNIX socket man page")," if you get stuck at ",(0,r.kt)("inlineCode",{parentName:"p"},"bind()"),".\nWhen you're done, compile the code and then run ",(0,r.kt)("inlineCode",{parentName:"p"},"send_unix_socket"),".\nIf you did this task correctly, ",(0,r.kt)("inlineCode",{parentName:"p"},"receive_unix_socket")," should display the flag."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/17896441.4e93f2ea.js b/17/assets/js/17896441.4e93f2ea.js new file mode 100644 index 0000000000..90f97a3bd0 --- /dev/null +++ b/17/assets/js/17896441.4e93f2ea.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7918],{1986:(e,t,a)=>{a.d(t,{Z:()=>v});var n=a(7462),l=a(7294),r=a(6010),o=a(5281),s=a(2802),c=a(8596),i=a(9960),d=a(4996),m=a(5999);function u(e){return l.createElement("svg",(0,n.Z)({viewBox:"0 0 24 24"},e),l.createElement("path",{d:"M10 19v-5h4v5c0 .55.45 1 1 1h3c.55 0 1-.45 1-1v-7h1.7c.46 0 .68-.57.33-.87L12.67 3.6c-.38-.34-.96-.34-1.34 0l-8.36 7.53c-.34.3-.13.87.33.87H5v7c0 .55.45 1 1 1h3c.55 0 1-.45 1-1z",fill:"currentColor"}))}const b={breadcrumbsContainer:"breadcrumbsContainer_Z_bl",breadcrumbHomeIcon:"breadcrumbHomeIcon_OVgt"};function p(e){let{children:t,href:a,isLast:n}=e;const r="breadcrumbs__link";return n?l.createElement("span",{className:r,itemProp:"name"},t):a?l.createElement(i.Z,{className:r,href:a,itemProp:"item"},l.createElement("span",{itemProp:"name"},t)):l.createElement("span",{className:r},t)}function E(e){let{children:t,active:a,index:o,addMicrodata:s}=e;return l.createElement("li",(0,n.Z)({},s&&{itemScope:!0,itemProp:"itemListElement",itemType:"https://schema.org/ListItem"},{className:(0,r.Z)("breadcrumbs__item",{"breadcrumbs__item--active":a})}),t,l.createElement("meta",{itemProp:"position",content:String(o+1)}))}function h(){const e=(0,d.Z)("/");return l.createElement("li",{className:"breadcrumbs__item"},l.createElement(i.Z,{"aria-label":(0,m.I)({id:"theme.docs.breadcrumbs.home",message:"Home page",description:"The ARIA label for the home page in the breadcrumbs"}),className:(0,r.Z)("breadcrumbs__link",b.breadcrumbsItemLink),href:e},l.createElement(u,{className:b.breadcrumbHomeIcon})))}function v(){const e=(0,s.s1)(),t=(0,c.Ns)();return e?l.createElement("nav",{className:(0,r.Z)(o.k.docs.docBreadcrumbs,b.breadcrumbsContainer),"aria-label":(0,m.I)({id:"theme.docs.breadcrumbs.navAriaLabel",message:"Breadcrumbs",description:"The ARIA label for the breadcrumbs"})},l.createElement("ul",{className:"breadcrumbs",itemScope:!0,itemType:"https://schema.org/BreadcrumbList"},t&&l.createElement(h,null),e.map(((t,a)=>{const n=a===e.length-1;return l.createElement(E,{key:a,active:n,index:a,addMicrodata:!!t.href},l.createElement(p,{href:t.href,isLast:n},t.label))})))):null}},230:(e,t,a)=>{a.r(t),a.d(t,{default:()=>X});var n=a(7294),l=a(1944),r=a(902);const o=n.createContext(null);function s(e){let{children:t,content:a}=e;const l=function(e){return(0,n.useMemo)((()=>({metadata:e.metadata,frontMatter:e.frontMatter,assets:e.assets,contentTitle:e.contentTitle,toc:e.toc})),[e])}(a);return n.createElement(o.Provider,{value:l},t)}function c(){const e=(0,n.useContext)(o);if(null===e)throw new r.i6("DocProvider");return e}function i(){const{metadata:e,frontMatter:t,assets:a}=c();return n.createElement(l.d,{title:e.title,description:e.description,keywords:t.keywords,image:a.image??t.image})}var d=a(6010),m=a(7524),u=a(4966);function b(){const{metadata:e}=c();return n.createElement(u.Z,{previous:e.previous,next:e.next})}var p=a(3120),E=a(4364),h=a(5281),v=a(5999);function g(e){let{lastUpdatedAt:t,formattedLastUpdatedAt:a}=e;return n.createElement(v.Z,{id:"theme.lastUpdated.atDate",description:"The words used to describe on which date a page has been last updated",values:{date:n.createElement("b",null,n.createElement("time",{dateTime:new Date(1e3*t).toISOString()},a))}}," on {date}")}function f(e){let{lastUpdatedBy:t}=e;return n.createElement(v.Z,{id:"theme.lastUpdated.byUser",description:"The words used to describe by who the page has been last updated",values:{user:n.createElement("b",null,t)}}," by {user}")}function Z(e){let{lastUpdatedAt:t,formattedLastUpdatedAt:a,lastUpdatedBy:l}=e;return n.createElement("span",{className:h.k.common.lastUpdated},n.createElement(v.Z,{id:"theme.lastUpdated.lastUpdatedAtBy",description:"The sentence used to display when a page has been last updated, and by who",values:{atDate:t&&a?n.createElement(g,{lastUpdatedAt:t,formattedLastUpdatedAt:a}):"",byUser:l?n.createElement(f,{lastUpdatedBy:l}):""}},"Last updated{atDate}{byUser}"),!1)}var _=a(7462);const N={iconEdit:"iconEdit_Z9Sw"};function k(e){let{className:t,...a}=e;return n.createElement("svg",(0,_.Z)({fill:"currentColor",height:"20",width:"20",viewBox:"0 0 40 40",className:(0,d.Z)(N.iconEdit,t),"aria-hidden":"true"},a),n.createElement("g",null,n.createElement("path",{d:"m34.5 11.7l-3 3.1-6.3-6.3 3.1-3q0.5-0.5 1.2-0.5t1.1 0.5l3.9 3.9q0.5 0.4 0.5 1.1t-0.5 1.2z m-29.5 17.1l18.4-18.5 6.3 6.3-18.4 18.4h-6.3v-6.2z"})))}function L(e){let{editUrl:t}=e;return n.createElement("a",{href:t,target:"_blank",rel:"noreferrer noopener",className:h.k.common.editThisPage},n.createElement(k,null),n.createElement(v.Z,{id:"theme.common.editThisPage",description:"The link label to edit the current page"},"Edit this page"))}var C=a(9960);const T={tag:"tag_zVej",tagRegular:"tagRegular_sFm0",tagWithCount:"tagWithCount_h2kH"};function U(e){let{permalink:t,label:a,count:l}=e;return n.createElement(C.Z,{href:t,className:(0,d.Z)(T.tag,l?T.tagWithCount:T.tagRegular)},a,l&&n.createElement("span",null,l))}const w={tags:"tags_jXut",tag:"tag_QGVx"};function x(e){let{tags:t}=e;return n.createElement(n.Fragment,null,n.createElement("b",null,n.createElement(v.Z,{id:"theme.tags.tagsListLabel",description:"The label alongside a tag list"},"Tags:")),n.createElement("ul",{className:(0,d.Z)(w.tags,"padding--none","margin-left--sm")},t.map((e=>{let{label:t,permalink:a}=e;return n.createElement("li",{key:a,className:w.tag},n.createElement(U,{label:t,permalink:a}))}))))}const y={lastUpdated:"lastUpdated_vwxv"};function A(e){return n.createElement("div",{className:(0,d.Z)(h.k.docs.docFooterTagsRow,"row margin-bottom--sm")},n.createElement("div",{className:"col"},n.createElement(x,e)))}function M(e){let{editUrl:t,lastUpdatedAt:a,lastUpdatedBy:l,formattedLastUpdatedAt:r}=e;return n.createElement("div",{className:(0,d.Z)(h.k.docs.docFooterEditMetaRow,"row")},n.createElement("div",{className:"col"},t&&n.createElement(L,{editUrl:t})),n.createElement("div",{className:(0,d.Z)("col",y.lastUpdated)},(a||l)&&n.createElement(Z,{lastUpdatedAt:a,formattedLastUpdatedAt:r,lastUpdatedBy:l})))}function B(){const{metadata:e}=c(),{editUrl:t,lastUpdatedAt:a,formattedLastUpdatedAt:l,lastUpdatedBy:r,tags:o}=e,s=o.length>0,i=!!(t||a||r);return s||i?n.createElement("footer",{className:(0,d.Z)(h.k.docs.docFooter,"docusaurus-mt-lg")},s&&n.createElement(A,{tags:o}),i&&n.createElement(M,{editUrl:t,lastUpdatedAt:a,lastUpdatedBy:r,formattedLastUpdatedAt:l})):null}var I=a(6043),V=a(3743);const H={tocCollapsibleButton:"tocCollapsibleButton_TO0P",tocCollapsibleButtonExpanded:"tocCollapsibleButtonExpanded_MG3E"};function P(e){let{collapsed:t,...a}=e;return n.createElement("button",(0,_.Z)({type:"button"},a,{className:(0,d.Z)("clean-btn",H.tocCollapsibleButton,!t&&H.tocCollapsibleButtonExpanded,a.className)}),n.createElement(v.Z,{id:"theme.TOCCollapsible.toggleButtonLabel",description:"The label used by the button on the collapsible TOC component"},"On this page"))}const D={tocCollapsible:"tocCollapsible_ETCw",tocCollapsibleContent:"tocCollapsibleContent_vkbj",tocCollapsibleExpanded:"tocCollapsibleExpanded_sAul"};function S(e){let{toc:t,className:a,minHeadingLevel:l,maxHeadingLevel:r}=e;const{collapsed:o,toggleCollapsed:s}=(0,I.u)({initialState:!0});return n.createElement("div",{className:(0,d.Z)(D.tocCollapsible,!o&&D.tocCollapsibleExpanded,a)},n.createElement(P,{collapsed:o,onClick:s}),n.createElement(I.z,{lazy:!0,className:D.tocCollapsibleContent,collapsed:o},n.createElement(V.Z,{toc:t,minHeadingLevel:l,maxHeadingLevel:r})))}const R={tocMobile:"tocMobile_ITEo"};function F(){const{toc:e,frontMatter:t}=c();return n.createElement(S,{toc:e,minHeadingLevel:t.toc_min_heading_level,maxHeadingLevel:t.toc_max_heading_level,className:(0,d.Z)(h.k.docs.docTocMobile,R.tocMobile)})}var O=a(9407);function z(){const{toc:e,frontMatter:t}=c();return n.createElement(O.Z,{toc:e,minHeadingLevel:t.toc_min_heading_level,maxHeadingLevel:t.toc_max_heading_level,className:h.k.docs.docTocDesktop})}var j=a(2503),G=a(210);function W(e){let{children:t}=e;const a=function(){const{metadata:e,frontMatter:t,contentTitle:a}=c();return t.hide_title||void 0!==a?null:e.title}();return n.createElement("div",{className:(0,d.Z)(h.k.docs.docMarkdown,"markdown")},a&&n.createElement("header",null,n.createElement(j.Z,{as:"h1"},a)),n.createElement(G.Z,null,t))}var q=a(1986);const J={docItemContainer:"docItemContainer_Djhp",docItemCol:"docItemCol_VOVn"};function Q(e){let{children:t}=e;const a=function(){const{frontMatter:e,toc:t}=c(),a=(0,m.i)(),l=e.hide_table_of_contents,r=!l&&t.length>0;return{hidden:l,mobile:r?n.createElement(F,null):void 0,desktop:!r||"desktop"!==a&&"ssr"!==a?void 0:n.createElement(z,null)}}();return n.createElement("div",{className:"row"},n.createElement("div",{className:(0,d.Z)("col",!a.hidden&&J.docItemCol)},n.createElement(p.Z,null),n.createElement("div",{className:J.docItemContainer},n.createElement("article",null,n.createElement(q.Z,null),n.createElement(E.Z,null),a.mobile,n.createElement(W,null,t),n.createElement(B,null)),n.createElement(b,null))),a.desktop&&n.createElement("div",{className:"col col--3"},a.desktop))}function X(e){const t=`docs-doc-id-${e.content.metadata.unversionedId}`,a=e.content;return n.createElement(s,{content:e.content},n.createElement(l.FG,{className:t},n.createElement(i,null),n.createElement(Q,null,n.createElement(a,null))))}},4966:(e,t,a)=>{a.d(t,{Z:()=>i});var n=a(7462),l=a(7294),r=a(5999),o=a(6010),s=a(9960);function c(e){const{permalink:t,title:a,subLabel:n,isNext:r}=e;return l.createElement(s.Z,{className:(0,o.Z)("pagination-nav__link",r?"pagination-nav__link--next":"pagination-nav__link--prev"),to:t},n&&l.createElement("div",{className:"pagination-nav__sublabel"},n),l.createElement("div",{className:"pagination-nav__label"},a))}function i(e){const{previous:t,next:a}=e;return l.createElement("nav",{className:"pagination-nav docusaurus-mt-lg","aria-label":(0,r.I)({id:"theme.docs.paginator.navAriaLabel",message:"Docs pages navigation",description:"The ARIA label for the docs pagination"})},t&&l.createElement(c,(0,n.Z)({},t,{subLabel:l.createElement(r.Z,{id:"theme.docs.paginator.previous",description:"The label used to navigate to the previous doc"},"Previous")})),a&&l.createElement(c,(0,n.Z)({},a,{subLabel:l.createElement(r.Z,{id:"theme.docs.paginator.next",description:"The label used to navigate to the next doc"},"Next"),isNext:!0})))}},4364:(e,t,a)=>{a.d(t,{Z:()=>c});var n=a(7294),l=a(6010),r=a(5999),o=a(5281),s=a(4477);function c(e){let{className:t}=e;const a=(0,s.E)();return a.badge?n.createElement("span",{className:(0,l.Z)(t,o.k.docs.docVersionBadge,"badge badge--secondary")},n.createElement(r.Z,{id:"theme.docs.versionBadge.label",values:{versionLabel:a.label}},"Version: {versionLabel}")):null}},3120:(e,t,a)=>{a.d(t,{Z:()=>h});var n=a(7294),l=a(6010),r=a(2263),o=a(9960),s=a(5999),c=a(143),i=a(5281),d=a(373),m=a(4477);const u={unreleased:function(e){let{siteTitle:t,versionMetadata:a}=e;return n.createElement(s.Z,{id:"theme.docs.versions.unreleasedVersionLabel",description:"The label used to tell the user that he's browsing an unreleased doc version",values:{siteTitle:t,versionLabel:n.createElement("b",null,a.label)}},"This is unreleased documentation for {siteTitle} {versionLabel} version.")},unmaintained:function(e){let{siteTitle:t,versionMetadata:a}=e;return n.createElement(s.Z,{id:"theme.docs.versions.unmaintainedVersionLabel",description:"The label used to tell the user that he's browsing an unmaintained doc version",values:{siteTitle:t,versionLabel:n.createElement("b",null,a.label)}},"This is documentation for {siteTitle} {versionLabel}, which is no longer actively maintained.")}};function b(e){const t=u[e.versionMetadata.banner];return n.createElement(t,e)}function p(e){let{versionLabel:t,to:a,onClick:l}=e;return n.createElement(s.Z,{id:"theme.docs.versions.latestVersionSuggestionLabel",description:"The label used to tell the user to check the latest version",values:{versionLabel:t,latestVersionLink:n.createElement("b",null,n.createElement(o.Z,{to:a,onClick:l},n.createElement(s.Z,{id:"theme.docs.versions.latestVersionLinkLabel",description:"The label used for the latest version suggestion link label"},"latest version")))}},"For up-to-date documentation, see the {latestVersionLink} ({versionLabel}).")}function E(e){let{className:t,versionMetadata:a}=e;const{siteConfig:{title:o}}=(0,r.Z)(),{pluginId:s}=(0,c.gA)({failfast:!0}),{savePreferredVersionName:m}=(0,d.J)(s),{latestDocSuggestion:u,latestVersionSuggestion:E}=(0,c.Jo)(s),h=u??(v=E).docs.find((e=>e.id===v.mainDocId));var v;return n.createElement("div",{className:(0,l.Z)(t,i.k.docs.docVersionBanner,"alert alert--warning margin-bottom--md"),role:"alert"},n.createElement("div",null,n.createElement(b,{siteTitle:o,versionMetadata:a})),n.createElement("div",{className:"margin-top--md"},n.createElement(p,{versionLabel:E.label,to:h.path,onClick:()=>m(E.name)})))}function h(e){let{className:t}=e;const a=(0,m.E)();return a.banner?n.createElement(E,{className:t,versionMetadata:a}):null}}}]); \ No newline at end of file diff --git a/17/assets/js/18331331.91bdb813.js b/17/assets/js/18331331.91bdb813.js new file mode 100644 index 0000000000..9422340547 --- /dev/null +++ b/17/assets/js/18331331.91bdb813.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2055],{3769:s=>{s.exports=JSON.parse('{"name":"docusaurus-plugin-content-docs","id":"default"}')}}]); \ No newline at end of file diff --git a/17/assets/js/1891e676.27e878e3.js b/17/assets/js/1891e676.27e878e3.js new file mode 100644 index 0000000000..9d9cf0f0f3 --- /dev/null +++ b/17/assets/js/1891e676.27e878e3.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7915],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>m});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var s=n.createContext({}),c=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},u=function(e){var t=c(e.components);return n.createElement(s.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,s=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),p=c(r),d=o,m=p["".concat(s,".").concat(d)]||p[d]||f[d]||a;return r?n.createElement(m,i(i({ref:t},u),{},{components:r})):n.createElement(m,i({ref:t},u))}));function m(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,i=new Array(a);i[0]=d;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[p]="string"==typeof e?e:o,i[1]=l;for(var c=2;c{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>i,default:()=>f,frontMatter:()=>a,metadata:()=>l,toc:()=>c});var n=r(7462),o=(r(7294),r(3905));const a={},i="I/O Errors",l={unversionedId:"Lab/IO/quiz/local-io-errors",id:"Lab/IO/quiz/local-io-errors",title:"I/O Errors",description:"Question Text",source:"@site/docs/Lab/IO/quiz/local-io-errors.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/local-io-errors",permalink:"/operating-systems/17/Lab/IO/quiz/local-io-errors",draft:!1,tags:[],version:"current",frontMatter:{}},s={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:c},p="wrapper";function f(e){let{components:t,...r}=e;return(0,o.kt)(p,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"io-errors"},"I/O Errors"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Which of the following types of errors are ",(0,o.kt)("strong",{parentName:"p"},"unlikely")," to occur during an I/O operation?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"The current user does not have sufficient permisions to access a given file")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"There is not enough space left on the disk"))),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"The file offset has reached the end of the file when reading")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"Some data blocks in the filesystem are corrupted")),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"We can always reposition the file offset within a given file with either a ",(0,o.kt)("inlineCode",{parentName:"p"},"fseek()")," library call or an ",(0,o.kt)("inlineCode",{parentName:"p"},"lseek()")," syscall, so this is not an error.\nThe others are more difficult to manage, so can be regarded as errors."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/1b1148a8.f9727c57.js b/17/assets/js/1b1148a8.f9727c57.js new file mode 100644 index 0000000000..c6f6d14f61 --- /dev/null +++ b/17/assets/js/1b1148a8.f9727c57.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5616],{3905:(e,t,a)=>{a.d(t,{Zo:()=>p,kt:()=>u});var r=a(7294);function n(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function o(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,r)}return a}function s(e){for(var t=1;t=0||(n[a]=e[a]);return n}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(n[a]=e[a])}return n}var l=r.createContext({}),c=function(e){var t=r.useContext(l),a=t;return e&&(a="function"==typeof e?e(t):s(s({},t),e)),a},p=function(e){var t=c(e.components);return r.createElement(l.Provider,{value:t},e.children)},f="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var a=e.components,n=e.mdxType,o=e.originalType,l=e.parentName,p=i(e,["components","mdxType","originalType","parentName"]),f=c(a),d=n,u=f["".concat(l,".").concat(d)]||f[d]||m[d]||o;return a?r.createElement(u,s(s({ref:t},p),{},{components:a})):r.createElement(u,s({ref:t},p))}));function u(e,t){var a=arguments,n=t&&t.mdxType;if("string"==typeof e||n){var o=a.length,s=new Array(o);s[0]=d;var i={};for(var l in t)hasOwnProperty.call(t,l)&&(i[l]=t[l]);i.originalType=e,i[f]="string"==typeof e?e:n,s[1]=i;for(var c=2;c{a.r(t),a.d(t,{assets:()=>l,contentTitle:()=>s,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>c});var r=a(7462),n=(a(7294),a(3905));const o={},s="Software Stack",i={unversionedId:"Lab/Software Stack/overview",id:"Lab/Software Stack/overview",title:"Software Stack",description:"Software comprises of code and data that is loaded in memory and used by the CPU.",source:"@site/docs/Lab/Software Stack/overview.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/overview",permalink:"/operating-systems/17/Lab/Software Stack/overview",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Software Stack",permalink:"/operating-systems/17/Lab/Software Stack/"},next:{title:"Modern Software Stacks",permalink:"/operating-systems/17/Lab/Software Stack/modern-sw-stack"}},l={},c=[{value:"Contents",id:"contents",level:2}],p={toc:c},f="wrapper";function m(e){let{components:t,...o}=e;return(0,n.kt)(f,(0,r.Z)({},p,o,{components:t,mdxType:"MDXLayout"}),(0,n.kt)("h1",{id:"software-stack"},"Software Stack"),(0,n.kt)("p",null,"Software comprises of code and data that is loaded in memory and used by the CPU.\nCode means instructions that are to be fetched by the CPU, decoded and executed.\nThis is called ",(0,n.kt)("strong",{parentName:"p"},"machine code"),", i.e. binary instructions that are understood by the CPU."),(0,n.kt)("p",null,(0,n.kt)("img",{alt:"Hardware and Software",src:a(4084).Z,width:"754",height:"329"})),(0,n.kt)("p",null,"So, when compared to hardware, ",(0,n.kt)("strong",{parentName:"p"},"software is highly flexible"),".\nWe can tie together specific instructions to handle a given task and run them on hardware (CPU, memory, I/O).\nDifferent pieces of these instructions solve different tasks and run on the same hardware.\nMoreover, these pieces of instructions can be duplicated and run on different pieces of hardware, thus providing ",(0,n.kt)("strong",{parentName:"p"},"software reusability"),".\nAll we are left with is creating those pieces of instructions, also called programs."),(0,n.kt)("p",null,"In summary, software has intrinsic characteristics:"),(0,n.kt)("ul",null,(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("strong",{parentName:"li"},"flexibility"),": We can (easily) create new pieces of software.\nLittle is required, we don't need raw materials as in the case of hardware or housing or transportation."),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("strong",{parentName:"li"},"reusability"),": Software can be easily copied to new systems and provide the same benefits there.")),(0,n.kt)("p",null,"Other characteristics are important to have, as they make life easier for both users and developers of software:"),(0,n.kt)("ul",null,(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("strong",{parentName:"li"},"portability"),": This is the ability to build and run the same program on different computing platforms.\nThis allows a developer to write the application code once and then run it everywhere."),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("strong",{parentName:"li"},"fast development"),": We want developers to be able to write code faster, using higher-level programming languages.")),(0,n.kt)("p",null,"The last two characteristics rely on two items:"),(0,n.kt)("ul",null,(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("p",{parentName:"li"},(0,n.kt)("strong",{parentName:"p"},"higher-level programming languages"),": As discussed above, a compiler will take a higher-level program and transform it into binary code for different computing platforms, thus providing portability.\nAlso, it's easier to read (comprehend) and write (develop) source code in a higher-level programming language, thus providing fast development.")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("p",{parentName:"li"},(0,n.kt)("strong",{parentName:"p"},"software stacks"),": A software stack is the layering of software such that each lower layer provides a set of features that the higher layer can directly use.\nThis means that there is no need for the higher layer to reimplement those features;\nthis provides fast development: focus on only the newer / required parts of software."),(0,n.kt)("p",{parentName:"li"},'Also, each lower layer provides a generic interface to the higher layer.\nThese generic interfaces "hides" possible differences in the even lower layers.\nThis way, a software stack ensures portability across different other parts of software (and hardware as well).\nFor example, the standard C library, that we will present shortly, ensures portability across different operating systems.'))),(0,n.kt)("p",null,(0,n.kt)("img",{alt:"Software Stack",src:a(3504).Z,width:"627",height:"405"})),(0,n.kt)("p",null,(0,n.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Software%20Stack/quiz/software"},"Quiz")),(0,n.kt)("h2",{id:"contents"},"Contents"),(0,n.kt)("ul",null,(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/overview"},"Overview")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/modern-sw-stack"},"Modern Software Stacks")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/basic-syscall"},"Basic System Calls")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/syscall-wrapper"},"System Call Wrapper")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/common-functions"},"Common Functions")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/libc"},"Libraries and libc")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/static-dynamic"},"Statically-linked and Dynamically-linked Libraries")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/libcall-syscall"},"Library calls vs system calls")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/high-level-lang"},"High-Level Languages")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/app-investigate"},"App Investigation")),(0,n.kt)("li",{parentName:"ul"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Software%20Stack/arena"},"Arena"))))}m.isMDXComponent=!0},4084:(e,t,a)=>{a.d(t,{Z:()=>r});const r=a.p+"assets/images/hardware-software-98463c6b754c2d0e75ccc886451176e6.svg"},3504:(e,t,a)=>{a.d(t,{Z:()=>r});const r=a.p+"assets/images/software-stack-ffefaf1db325c303dbc9909b608b58a5.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/1be78505.f8b9d3b1.js b/17/assets/js/1be78505.f8b9d3b1.js new file mode 100644 index 0000000000..57742abe43 --- /dev/null +++ b/17/assets/js/1be78505.f8b9d3b1.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9514,4972],{9963:(e,t,n)=>{n.r(t),n.d(t,{default:()=>fe});var a=n(7294),l=n(6010),o=n(1944),r=n(5281),c=n(3320),i=n(2802),s=n(4477),d=n(1116),m=n(215),u=n(5999),b=n(2466),p=n(5936);const h={backToTopButton:"backToTopButton_sjWU",backToTopButtonShow:"backToTopButtonShow_xfvO"};function E(){const{shown:e,scrollToTop:t}=function(e){let{threshold:t}=e;const[n,l]=(0,a.useState)(!1),o=(0,a.useRef)(!1),{startScroll:r,cancelScroll:c}=(0,b.Ct)();return(0,b.RF)(((e,n)=>{let{scrollY:a}=e;const r=n?.scrollY;r&&(o.current?o.current=!1:a>=r?(c(),l(!1)):a{e.location.hash&&(o.current=!0,l(!1))})),{shown:n,scrollToTop:()=>r(0)}}({threshold:300});return a.createElement("button",{"aria-label":(0,u.I)({id:"theme.BackToTopButton.buttonAriaLabel",message:"Scroll back to top",description:"The ARIA label for the back to top button"}),className:(0,l.Z)("clean-btn",r.k.common.backToTopButton,h.backToTopButton,e&&h.backToTopButtonShow),type:"button",onClick:t})}var f=n(6550),g=n(7524),k=n(6668),_=n(1327),v=n(7462);function C(e){return a.createElement("svg",(0,v.Z)({width:"20",height:"20","aria-hidden":"true"},e),a.createElement("g",{fill:"#7a7a7a"},a.createElement("path",{d:"M9.992 10.023c0 .2-.062.399-.172.547l-4.996 7.492a.982.982 0 01-.828.454H1c-.55 0-1-.453-1-1 0-.2.059-.403.168-.551l4.629-6.942L.168 3.078A.939.939 0 010 2.528c0-.548.45-.997 1-.997h2.996c.352 0 .649.18.828.45L9.82 9.472c.11.148.172.347.172.55zm0 0"}),a.createElement("path",{d:"M19.98 10.023c0 .2-.058.399-.168.547l-4.996 7.492a.987.987 0 01-.828.454h-3c-.547 0-.996-.453-.996-1 0-.2.059-.403.168-.551l4.625-6.942-4.625-6.945a.939.939 0 01-.168-.55 1 1 0 01.996-.997h3c.348 0 .649.18.828.45l4.996 7.492c.11.148.168.347.168.55zm0 0"})))}const S={collapseSidebarButton:"collapseSidebarButton_PEFL",collapseSidebarButtonIcon:"collapseSidebarButtonIcon_kv0_"};function I(e){let{onClick:t}=e;return a.createElement("button",{type:"button",title:(0,u.I)({id:"theme.docs.sidebar.collapseButtonTitle",message:"Collapse sidebar",description:"The title attribute for collapse button of doc sidebar"}),"aria-label":(0,u.I)({id:"theme.docs.sidebar.collapseButtonAriaLabel",message:"Collapse sidebar",description:"The title attribute for collapse button of doc sidebar"}),className:(0,l.Z)("button button--secondary button--outline",S.collapseSidebarButton),onClick:t},a.createElement(C,{className:S.collapseSidebarButtonIcon}))}var N=n(9689),T=n(902);const Z=Symbol("EmptyContext"),x=a.createContext(Z);function B(e){let{children:t}=e;const[n,l]=(0,a.useState)(null),o=(0,a.useMemo)((()=>({expandedItem:n,setExpandedItem:l})),[n]);return a.createElement(x.Provider,{value:o},t)}var y=n(6043),L=n(8596),w=n(9960),A=n(2389);function M(e){let{categoryLabel:t,onClick:n}=e;return a.createElement("button",{"aria-label":(0,u.I)({id:"theme.DocSidebarItem.toggleCollapsedCategoryAriaLabel",message:"Toggle the collapsible sidebar category '{label}'",description:"The ARIA label to toggle the collapsible sidebar category"},{label:t}),type:"button",className:"clean-btn menu__caret",onClick:n})}function F(e){let{item:t,onItemClick:n,activePath:o,level:c,index:s,...d}=e;const{items:m,label:u,collapsible:b,className:p,href:h}=t,{docs:{sidebar:{autoCollapseCategories:E}}}=(0,k.L)(),f=function(e){const t=(0,A.Z)();return(0,a.useMemo)((()=>e.href?e.href:!t&&e.collapsible?(0,i.Wl)(e):void 0),[e,t])}(t),g=(0,i._F)(t,o),_=(0,L.Mg)(h,o),{collapsed:C,setCollapsed:S}=(0,y.u)({initialState:()=>!!b&&(!g&&t.collapsed)}),{expandedItem:I,setExpandedItem:N}=function(){const e=(0,a.useContext)(x);if(e===Z)throw new T.i6("DocSidebarItemsExpandedStateProvider");return e}(),B=function(e){void 0===e&&(e=!C),N(e?null:s),S(e)};return function(e){let{isActive:t,collapsed:n,updateCollapsed:l}=e;const o=(0,T.D9)(t);(0,a.useEffect)((()=>{t&&!o&&n&&l(!1)}),[t,o,n,l])}({isActive:g,collapsed:C,updateCollapsed:B}),(0,a.useEffect)((()=>{b&&null!=I&&I!==s&&E&&S(!0)}),[b,I,s,S,E]),a.createElement("li",{className:(0,l.Z)(r.k.docs.docSidebarItemCategory,r.k.docs.docSidebarItemCategoryLevel(c),"menu__list-item",{"menu__list-item--collapsed":C},p)},a.createElement("div",{className:(0,l.Z)("menu__list-item-collapsible",{"menu__list-item-collapsible--active":_})},a.createElement(w.Z,(0,v.Z)({className:(0,l.Z)("menu__link",{"menu__link--sublist":b,"menu__link--sublist-caret":!h&&b,"menu__link--active":g}),onClick:b?e=>{n?.(t),h?B(!1):(e.preventDefault(),B())}:()=>{n?.(t)},"aria-current":_?"page":void 0,"aria-expanded":b?!C:void 0,href:b?f??"#":f},d),u),h&&b&&a.createElement(M,{categoryLabel:u,onClick:e=>{e.preventDefault(),B()}})),a.createElement(y.z,{lazy:!0,as:"ul",className:"menu__list",collapsed:C},a.createElement(V,{items:m,tabIndex:C?-1:0,onItemClick:n,activePath:o,level:c+1})))}var H=n(3919),P=n(9471);const W={menuExternalLink:"menuExternalLink_NmtK"};function D(e){let{item:t,onItemClick:n,activePath:o,level:c,index:s,...d}=e;const{href:m,label:u,className:b,autoAddBaseUrl:p}=t,h=(0,i._F)(t,o),E=(0,H.Z)(m);return a.createElement("li",{className:(0,l.Z)(r.k.docs.docSidebarItemLink,r.k.docs.docSidebarItemLinkLevel(c),"menu__list-item",b),key:u},a.createElement(w.Z,(0,v.Z)({className:(0,l.Z)("menu__link",!E&&W.menuExternalLink,{"menu__link--active":h}),autoAddBaseUrl:p,"aria-current":h?"page":void 0,to:m},E&&{onClick:n?()=>n(t):void 0},d),u,!E&&a.createElement(P.Z,null)))}const R={menuHtmlItem:"menuHtmlItem_M9Kj"};function z(e){let{item:t,level:n,index:o}=e;const{value:c,defaultStyle:i,className:s}=t;return a.createElement("li",{className:(0,l.Z)(r.k.docs.docSidebarItemLink,r.k.docs.docSidebarItemLinkLevel(n),i&&[R.menuHtmlItem,"menu__list-item"],s),key:o,dangerouslySetInnerHTML:{__html:c}})}function U(e){let{item:t,...n}=e;switch(t.type){case"category":return a.createElement(F,(0,v.Z)({item:t},n));case"html":return a.createElement(z,(0,v.Z)({item:t},n));default:return a.createElement(D,(0,v.Z)({item:t},n))}}function K(e){let{items:t,...n}=e;return a.createElement(B,null,t.map(((e,t)=>a.createElement(U,(0,v.Z)({key:t,item:e,index:t},n)))))}const V=(0,a.memo)(K),j={menu:"menu_SIkG",menuWithAnnouncementBar:"menuWithAnnouncementBar_GW3s"};function G(e){let{path:t,sidebar:n,className:o}=e;const c=function(){const{isActive:e}=(0,N.nT)(),[t,n]=(0,a.useState)(e);return(0,b.RF)((t=>{let{scrollY:a}=t;e&&n(0===a)}),[e]),e&&t}();return a.createElement("nav",{className:(0,l.Z)("menu thin-scrollbar",j.menu,c&&j.menuWithAnnouncementBar,o)},a.createElement("ul",{className:(0,l.Z)(r.k.docs.docSidebarMenu,"menu__list")},a.createElement(V,{items:n,activePath:t,level:1})))}const Y="sidebar_njMd",q="sidebarWithHideableNavbar_wUlq",O="sidebarHidden_VK0M",X="sidebarLogo_isFc";function J(e){let{path:t,sidebar:n,onCollapse:o,isHidden:r}=e;const{navbar:{hideOnScroll:c},docs:{sidebar:{hideable:i}}}=(0,k.L)();return a.createElement("div",{className:(0,l.Z)(Y,c&&q,r&&O)},c&&a.createElement(_.Z,{tabIndex:-1,className:X}),a.createElement(G,{path:t,sidebar:n}),i&&a.createElement(I,{onClick:o}))}const Q=a.memo(J);var $=n(3102),ee=n(2961);const te=e=>{let{sidebar:t,path:n}=e;const o=(0,ee.e)();return a.createElement("ul",{className:(0,l.Z)(r.k.docs.docSidebarMenu,"menu__list")},a.createElement(V,{items:t,activePath:n,onItemClick:e=>{"category"===e.type&&e.href&&o.toggle(),"link"===e.type&&o.toggle()},level:1}))};function ne(e){return a.createElement($.Zo,{component:te,props:e})}const ae=a.memo(ne);function le(e){const t=(0,g.i)(),n="desktop"===t||"ssr"===t,l="mobile"===t;return a.createElement(a.Fragment,null,n&&a.createElement(Q,e),l&&a.createElement(ae,e))}const oe={expandButton:"expandButton_m80_",expandButtonIcon:"expandButtonIcon_BlDH"};function re(e){let{toggleSidebar:t}=e;return a.createElement("div",{className:oe.expandButton,title:(0,u.I)({id:"theme.docs.sidebar.expandButtonTitle",message:"Expand sidebar",description:"The ARIA label and title attribute for expand button of doc sidebar"}),"aria-label":(0,u.I)({id:"theme.docs.sidebar.expandButtonAriaLabel",message:"Expand sidebar",description:"The ARIA label and title attribute for expand button of doc sidebar"}),tabIndex:0,role:"button",onKeyDown:t,onClick:t},a.createElement(C,{className:oe.expandButtonIcon}))}const ce={docSidebarContainer:"docSidebarContainer_b6E3",docSidebarContainerHidden:"docSidebarContainerHidden_b3ry"};function ie(e){let{children:t}=e;const n=(0,d.V)();return a.createElement(a.Fragment,{key:n?.name??"noSidebar"},t)}function se(e){let{sidebar:t,hiddenSidebarContainer:n,setHiddenSidebarContainer:o}=e;const{pathname:c}=(0,f.TH)(),[i,s]=(0,a.useState)(!1),d=(0,a.useCallback)((()=>{i&&s(!1),o((e=>!e))}),[o,i]);return a.createElement("aside",{className:(0,l.Z)(r.k.docs.docSidebarContainer,ce.docSidebarContainer,n&&ce.docSidebarContainerHidden),onTransitionEnd:e=>{e.currentTarget.classList.contains(ce.docSidebarContainer)&&n&&s(!0)}},a.createElement(ie,null,a.createElement(le,{sidebar:t,path:c,onCollapse:d,isHidden:i})),i&&a.createElement(re,{toggleSidebar:d}))}const de={docMainContainer:"docMainContainer_gTbr",docMainContainerEnhanced:"docMainContainerEnhanced_Uz_u",docItemWrapperEnhanced:"docItemWrapperEnhanced_czyv"};function me(e){let{hiddenSidebarContainer:t,children:n}=e;const o=(0,d.V)();return a.createElement("main",{className:(0,l.Z)(de.docMainContainer,(t||!o)&&de.docMainContainerEnhanced)},a.createElement("div",{className:(0,l.Z)("container padding-top--md padding-bottom--lg",de.docItemWrapper,t&&de.docItemWrapperEnhanced)},n))}const ue={docPage:"docPage__5DB",docsWrapper:"docsWrapper_BCFX"};function be(e){let{children:t}=e;const n=(0,d.V)(),[l,o]=(0,a.useState)(!1);return a.createElement(m.Z,{wrapperClassName:ue.docsWrapper},a.createElement(E,null),a.createElement("div",{className:ue.docPage},n&&a.createElement(se,{sidebar:n.items,hiddenSidebarContainer:l,setHiddenSidebarContainer:o}),a.createElement(me,{hiddenSidebarContainer:l},t)))}var pe=n(4972),he=n(197);function Ee(e){const{versionMetadata:t}=e;return a.createElement(a.Fragment,null,a.createElement(he.Z,{version:t.version,tag:(0,c.os)(t.pluginId,t.version)}),a.createElement(o.d,null,t.noIndex&&a.createElement("meta",{name:"robots",content:"noindex, nofollow"})))}function fe(e){const{versionMetadata:t}=e,n=(0,i.hI)(e);if(!n)return a.createElement(pe.default,null);const{docElement:c,sidebarName:m,sidebarItems:u}=n;return a.createElement(a.Fragment,null,a.createElement(Ee,e),a.createElement(o.FG,{className:(0,l.Z)(r.k.wrapper.docsPages,r.k.page.docsDocPage,e.versionMetadata.className)},a.createElement(s.q,{version:t},a.createElement(d.b,{name:m,items:u},a.createElement(be,null,c)))))}},4972:(e,t,n)=>{n.r(t),n.d(t,{default:()=>c});var a=n(7294),l=n(5999),o=n(1944),r=n(215);function c(){return a.createElement(a.Fragment,null,a.createElement(o.d,{title:(0,l.I)({id:"theme.NotFound.title",message:"Page Not Found"})}),a.createElement(r.Z,null,a.createElement("main",{className:"container margin-vert--xl"},a.createElement("div",{className:"row"},a.createElement("div",{className:"col col--6 col--offset-3"},a.createElement("h1",{className:"hero__title"},a.createElement(l.Z,{id:"theme.NotFound.title",description:"The title of the 404 page"},"Page Not Found")),a.createElement("p",null,a.createElement(l.Z,{id:"theme.NotFound.p1",description:"The first paragraph of the 404 page"},"We could not find what you were looking for.")),a.createElement("p",null,a.createElement(l.Z,{id:"theme.NotFound.p2",description:"The 2nd paragraph of the 404 page"},"Please contact the owner of the site that linked you to the original URL and let them know their link is broken.")))))))}}}]); \ No newline at end of file diff --git a/17/assets/js/1dccb566.b9ac2ee1.js b/17/assets/js/1dccb566.b9ac2ee1.js new file mode 100644 index 0000000000..0b6a4e057a --- /dev/null +++ b/17/assets/js/1dccb566.b9ac2ee1.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2279],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function l(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var p=r.createContext({}),u=function(e){var t=r.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},c=function(e){var t=u(e.components);return r.createElement(p.Provider,{value:t},e.children)},s="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,p=e.parentName,c=o(e,["components","mdxType","originalType","parentName"]),s=u(n),d=a,m=s["".concat(p,".").concat(d)]||s[d]||f[d]||i;return n?r.createElement(m,l(l({ref:t},c),{},{components:n})):r.createElement(m,l({ref:t},c))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,l=new Array(i);l[0]=d;var o={};for(var p in t)hasOwnProperty.call(t,p)&&(o[p]=t[p]);o.originalType=e,o[s]="string"==typeof e?e:a,l[1]=o;for(var u=2;u{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>l,default:()=>f,frontMatter:()=>i,metadata:()=>o,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const i={},l="File handler in C",o={unversionedId:"Lab/IO/quiz/file-handler-c",id:"Lab/IO/quiz/file-handler-c",title:"File handler in C",description:"Question Text",source:"@site/docs/Lab/IO/quiz/file-handler-c.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/file-handler-c",permalink:"/operating-systems/17/Lab/IO/quiz/file-handler-c",draft:!1,tags:[],version:"current",frontMatter:{}},p={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedaback",id:"feedaback",level:2}],c={toc:u},s="wrapper";function f(e){let{components:t,...n}=e;return(0,a.kt)(s,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"file-handler-in-c"},"File handler in C"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What is the type of the file handler in the C code located in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/simple-file-operations/file_operations.c"),"?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"File"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"FILE *"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"FILE"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"void *"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"struct file")))),(0,a.kt)("h2",{id:"feedaback"},"Feedaback"),(0,a.kt)("p",null,"The file is opened using either of the following lines:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},'f = fopen(file_name, "r");\n\nf = fopen(file_name, "w");\n')),(0,a.kt)("p",null,"The type of ",(0,a.kt)("inlineCode",{parentName:"p"},"f")," is ",(0,a.kt)("inlineCode",{parentName:"p"},"FILE *"),":\n",(0,a.kt)("inlineCode",{parentName:"p"},"FILE *f"),"."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/1e0b7734.0d01aea4.js b/17/assets/js/1e0b7734.0d01aea4.js new file mode 100644 index 0000000000..7bbcd5089d --- /dev/null +++ b/17/assets/js/1e0b7734.0d01aea4.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9152],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>f});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function l(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var s=n.createContext({}),i=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):l(l({},t),e)),r},p=function(e){var t=i(e.components);return n.createElement(s.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,s=e.parentName,p=c(e,["components","mdxType","originalType","parentName"]),u=i(r),d=a,f=u["".concat(s,".").concat(d)]||u[d]||m[d]||o;return r?n.createElement(f,l(l({ref:t},p),{},{components:r})):n.createElement(f,l({ref:t},p))}));function f(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,l=new Array(o);l[0]=d;var c={};for(var s in t)hasOwnProperty.call(t,s)&&(c[s]=t[s]);c.originalType=e,c[u]="string"==typeof e?e:a,l[1]=c;for(var i=2;i{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>l,default:()=>m,frontMatter:()=>o,metadata:()=>c,toc:()=>i});var n=r(7462),a=(r(7294),r(3905));const o={},l="Operator Overloading",c={unversionedId:"Lab/Data/quiz/operators",id:"Lab/Data/quiz/operators",title:"Operator Overloading",description:"Question Text",source:"@site/docs/Lab/Data/quiz/operators.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/operators",permalink:"/operating-systems/17/Lab/Data/quiz/operators",draft:!1,tags:[],version:"current",frontMatter:{}},s={},i=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:i},u="wrapper";function m(e){let{components:t,...r}=e;return(0,a.kt)(u,(0,n.Z)({},p,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"operator-overloading"},"Operator Overloading"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"How many constructor calls, copy constructor calls, assignment operator calls and destructor calls does the following program issue?"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-d"},"Obj quiz(Obj o1, Obj o2)\n{\n o2 = o1;\n return o2;\n}\nvoid main()\n{\n Obj b = quiz(o1, o2);\n}\n")),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"constructor calls = 0, copy constructor calls = 3, assignment operator calls = 1, destructor calls = 3")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"constructor calls = 1, copy constructor calls = 2, assignment operator calls = 1, destructor calls = 2")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"constructor calls = 0, copy constructor calls = 2, assignment operator calls = 1, destructor calls = 2")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"constructor calls = 0, copy constructor calls = 3, assignment operator calls = 1, destructor calls = 1"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"There are no constructor calls because there is no object construction when using ",(0,a.kt)("inlineCode",{parentName:"p"},"int"),".")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"There are 3 copy constructor calls: for passing ",(0,a.kt)("inlineCode",{parentName:"p"},"o1"),", for passing ",(0,a.kt)("inlineCode",{parentName:"p"},"o2"),", and for returning ",(0,a.kt)("inlineCode",{parentName:"p"},"o2"),".")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"There is 1 assignment operator call for ",(0,a.kt)("inlineCode",{parentName:"p"},"o2 = o1"),".")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"There are 3 destructor calls, because each constructed object needs to be destroyed."))))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/1ef36804.d9d216f5.js b/17/assets/js/1ef36804.d9d216f5.js new file mode 100644 index 0000000000..347176f70a --- /dev/null +++ b/17/assets/js/1ef36804.d9d216f5.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2764],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>d});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var c=n.createContext({}),l=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},u=function(e){var t=l(e.components);return n.createElement(c.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},h=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,c=e.parentName,u=s(e,["components","mdxType","originalType","parentName"]),p=l(r),h=o,d=p["".concat(c,".").concat(h)]||p[h]||f[h]||a;return r?n.createElement(d,i(i({ref:t},u),{},{components:r})):n.createElement(d,i({ref:t},u))}));function d(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,i=new Array(a);i[0]=h;var s={};for(var c in t)hasOwnProperty.call(t,c)&&(s[c]=t[c]);s.originalType=e,s[p]="string"==typeof e?e:o,i[1]=s;for(var l=2;l{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>f,frontMatter:()=>a,metadata:()=>s,toc:()=>l});var n=r(7462),o=(r(7294),r(3905));const a={},i="Fewer than Two Copies",s={unversionedId:"Lab/IO/quiz/fewer-than-2-copies",id:"Lab/IO/quiz/fewer-than-2-copies",title:"Fewer than Two Copies",description:"Question Text",source:"@site/docs/Lab/IO/quiz/fewer-than-2-copies.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/fewer-than-2-copies",permalink:"/operating-systems/17/Lab/IO/quiz/fewer-than-2-copies",draft:!1,tags:[],version:"current",frontMatter:{}},c={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:l},p="wrapper";function f(e){let{components:t,...a}=e;return(0,o.kt)(p,(0,n.Z)({},u,a,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"fewer-than-two-copies"},"Fewer than Two Copies"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Can zero-copy be implemented so as to copy the file fewer than 2 times?"),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"Zero-Copy",src:r(8409).Z,width:"562",height:"472"})),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"Yes, by copying the file straight from the disk to the network")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"No")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Yes, by sending the file straight from the kernel buffer to the network")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Yes, by copying the file directly from the storage to the NIC's buffer"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,'The truth is that we can\'t have fewer copies while using a server with a common PC-like architecture.\nThe disk is not directly connected to the internet, so the file cannot be sent directly from there.\nThe only place from where we can send data to the Web is the NIC.\nThen we need the intermediary storage in that kernel buffer because the disk and the NIC aren\'t dirrectly connected.\nThey are both connected to the CPU via the motherboard, so it\'s the CPU\'s job to do the transfer.\nFor this, it needs a "temporary buffer".\nThen the NIC needs its own buffer because the speed of the network may be slower than the speed at which it receives data from the kernel, so it needs some memory where to place the "surplus" while waiting for the network to "clear".'))}f.isMDXComponent=!0},8409:(e,t,r)=>{r.d(t,{Z:()=>n});const n=r.p+"assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/1f391b9e.b2d53f92.js b/17/assets/js/1f391b9e.b2d53f92.js new file mode 100644 index 0000000000..12662c9be5 --- /dev/null +++ b/17/assets/js/1f391b9e.b2d53f92.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3085],{4247:(e,t,a)=>{a.r(t),a.d(t,{default:()=>d});var l=a(7294),n=a(6010),c=a(1944),r=a(5281),m=a(215),s=a(210),i=a(9407);const o={mdxPageWrapper:"mdxPageWrapper_j9I6"};function d(e){const{content:t}=e,{metadata:{title:a,description:d,frontMatter:p}}=t,{wrapperClassName:g,hide_table_of_contents:u}=p;return l.createElement(c.FG,{className:(0,n.Z)(g??r.k.wrapper.mdxPages,r.k.page.mdxPage)},l.createElement(c.d,{title:a,description:d}),l.createElement(m.Z,null,l.createElement("main",{className:"container container--fluid margin-vert--lg"},l.createElement("div",{className:(0,n.Z)("row",o.mdxPageWrapper)},l.createElement("div",{className:(0,n.Z)("col",!u&&"col--8")},l.createElement("article",null,l.createElement(s.Z,null,l.createElement(t,null)))),!u&&t.toc.length>0&&l.createElement("div",{className:"col col--2"},l.createElement(i.Z,{toc:t.toc,minHeadingLevel:p.toc_min_heading_level,maxHeadingLevel:p.toc_max_heading_level}))))))}}}]); \ No newline at end of file diff --git a/17/assets/js/1ffc5c5f.dac88b05.js b/17/assets/js/1ffc5c5f.dac88b05.js new file mode 100644 index 0000000000..d165ffae3c --- /dev/null +++ b/17/assets/js/1ffc5c5f.dac88b05.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4379],{3905:(e,t,a)=>{a.d(t,{Zo:()=>d,kt:()=>m});var n=a(7294);function r(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function s(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,n)}return a}function o(e){for(var t=1;t=0||(r[a]=e[a]);return r}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(r[a]=e[a])}return r}var i=n.createContext({}),l=function(e){var t=n.useContext(i),a=t;return e&&(a="function"==typeof e?e(t):o(o({},t),e)),a},d=function(e){var t=l(e.components);return n.createElement(i.Provider,{value:t},e.children)},c="mdxType",h={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},u=n.forwardRef((function(e,t){var a=e.components,r=e.mdxType,s=e.originalType,i=e.parentName,d=p(e,["components","mdxType","originalType","parentName"]),c=l(a),u=r,m=c["".concat(i,".").concat(u)]||c[u]||h[u]||s;return a?n.createElement(m,o(o({ref:t},d),{},{components:a})):n.createElement(m,o({ref:t},d))}));function m(e,t){var a=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var s=a.length,o=new Array(s);o[0]=u;var p={};for(var i in t)hasOwnProperty.call(t,i)&&(p[i]=t[i]);p.originalType=e,p[c]="string"==typeof e?e:r,o[1]=p;for(var l=2;l{a.r(t),a.d(t,{assets:()=>i,contentTitle:()=>o,default:()=>h,frontMatter:()=>s,metadata:()=>p,toc:()=>l});var n=a(7462),r=(a(7294),a(3905));const s={},o="Usage of Processes and Threads in `apache2`",p={unversionedId:"Lab/Compute/processes-threads-apache2",id:"Lab/Compute/processes-threads-apache2",title:"Usage of Processes and Threads in `apache2`",description:"We'll take a look at how a real-world application - the apache2 HTTP server - makes use of processes and threads.",source:"@site/docs/Lab/Compute/processes-threads-apache2.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/processes-threads-apache2",permalink:"/operating-systems/17/Lab/Compute/processes-threads-apache2",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Threads",permalink:"/operating-systems/17/Lab/Compute/threads"},next:{title:"Copy-on-Write",permalink:"/operating-systems/17/Lab/Compute/copy-on-write"}},i={},l=[{value:"apache2 Live Action",id:"apache2-live-action",level:2},{value:"Practice: Investigate apache2 Using strace",id:"practice-investigate-apache2-using-strace",level:2},{value:"Conclusion",id:"conclusion",level:2}],d={toc:l},c="wrapper";function h(e){let{components:t,...a}=e;return(0,r.kt)(c,(0,n.Z)({},d,a,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"usage-of-processes-and-threads-in-apache2"},"Usage of Processes and Threads in ",(0,r.kt)("inlineCode",{parentName:"h1"},"apache2")),(0,r.kt)("p",null,"We'll take a look at how a real-world application - the ",(0,r.kt)("inlineCode",{parentName:"p"},"apache2")," HTTP server - makes use of processes and threads.\nSince the server must be able to handle multiple clients at the same time, it must therefore use some form of concurrency.\nWhen a new client arrives, the server offloads the work of interacting with that client to another process or thread."),(0,r.kt)("p",null,"The choice of whether to use multiple processes or threads is not baked into the code.\nInstead, ",(0,r.kt)("inlineCode",{parentName:"p"},"apache2")," provides a couple of modules called MPMs (Multi-Processing Modules).\nEach module implements a different concurrency model, and the users can pick whatever module best fits their needs by editing the server configuration files."),(0,r.kt)("p",null,"The most common MPMs are"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"prefork"),": there are multiple worker processes, each process is single-threaded and handles one client request at a time"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"worker"),": there are multiple worker processes, each process is multi-threaded, and each thread handles one client request at a time"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"event"),": same as ",(0,r.kt)("inlineCode",{parentName:"li"},"worker")," but designed to better handle some particular use cases")),(0,r.kt)("p",null,"In principle, ",(0,r.kt)("inlineCode",{parentName:"p"},"prefork")," provides more stability and backwards compatibility, but it has a bigger overhead.\nOn the other hand, ",(0,r.kt)("inlineCode",{parentName:"p"},"worker")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"event")," are more scalable, and thus able to handle more simultaneous connections, due to the usage of threads.\nOn modern systems, ",(0,r.kt)("inlineCode",{parentName:"p"},"event")," is almost always the default."),(0,r.kt)("h2",{id:"apache2-live-action"},(0,r.kt)("inlineCode",{parentName:"h2"},"apache2")," Live Action"),(0,r.kt)("p",null,"Let's run an actual instance of ",(0,r.kt)("inlineCode",{parentName:"p"},"apache2")," and see how everything works.\nGo to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/apache2")," and run ",(0,r.kt)("inlineCode",{parentName:"p"},"make run"),".\nThis will start a container with ",(0,r.kt)("inlineCode",{parentName:"p"},"apache2")," running inside."),(0,r.kt)("p",null,"Check that the server runs as expected:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ curl localhost:8080\n

It works!

\n")),(0,r.kt)("p",null,"Now go inside the container and take a look at running processes:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/apache2$ docker exec -it apache2-test bash\n\nroot@56b9a761d598:/usr/local/apache2# ps -ef\nUID PID PPID C STIME TTY TIME CMD\nroot 1 0 0 20:38 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 0 20:38 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 10 1 0 20:38 pts/0 00:00:00 httpd -DFOREGROUND\nroot 25 0 0 20:40 pts/1 00:00:00 bash\nroot 31 25 0 20:40 pts/1 00:00:00 ps -ef\n")),(0,r.kt)("p",null,"We see 3 ",(0,r.kt)("inlineCode",{parentName:"p"},"httpd")," processes.\nThe first one, running as root, is the main process, while the other 2 are the workers."),(0,r.kt)("p",null,"Let's confirm that we are using the ",(0,r.kt)("inlineCode",{parentName:"p"},"event")," mpm:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"root@56b9a761d598:/usr/local/apache2# grep mod_mpm conf/httpd.conf\nLoadModule mpm_event_module modules/mod_mpm_event.so\nLoadModule mpm_prefork_module modules/mod_mpm_prefork.so\nLoadModule mpm_worker_module modules/mod_mpm_worker.so\n")),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"event")," mpm is enabled, so we expect each worker to be multithreaded.\nLet's check:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"root@56b9a761d598:/usr/local/apache2# ps -efL\nUID PID PPID LWP C NLWP STIME TTY TIME CMD\nroot 1 0 1 0 1 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 8 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 11 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 12 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 16 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 17 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 18 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 8 1 19 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 9 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 14 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 15 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 20 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 21 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 22 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 9 1 23 0 7 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nroot 24 0 24 1 1 20:56 pts/1 00:00:00 bash\nroot 30 24 30 0 1 20:56 pts/1 00:00:00 ps -efL\n")),(0,r.kt)("p",null,"Indeed, each worker has 7 threads.\nIn fact, the number of threads per worker is configurable, as well as the number of initial workers."),(0,r.kt)("p",null,"When a new connection is created, it will be handled by whatever thread is available from any worker.\nIf all the threads are busy, then the server will spawn more worker processes (and therefore more threads), as long as the total number of threads is below some threshold, which is also configurable."),(0,r.kt)("p",null,"Let's see this dynamic scaling in action.\nWe need to create a number of simultaneous connections that is larger than the current number of threads.\nThere is a simple script in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/apache2/make_conn.py")," to do this:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/apache2$ python3 make_conn.py localhost 8080\nPress ENTER to exit\n")),(0,r.kt)("p",null,"The script has created 100 connections and will keep them open until we press Enter."),(0,r.kt)("p",null,"Now, in another terminal, let's check the situation inside the container:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/apache2$ docker exec -it apache2-test bash\n\nroot@56b9a761d598:/usr/local/apache2# ps -efL\nUID PID PPID LWP C NLWP STIME TTY TIME CMD\nroot 1 0 1 0 1 20:56 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 40 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 45 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 46 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 51 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 52 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 53 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 40 1 54 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 55 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 58 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 60 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 62 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 63 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 65 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 55 1 66 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\n[...]\nwww-data 109 1 109 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 109 1 115 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 109 1 116 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 109 1 121 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 109 1 122 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 109 1 123 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nwww-data 109 1 124 0 7 21:07 pts/0 00:00:00 httpd -DFOREGROUND\nroot 146 0 146 0 1 21:10 pts/1 00:00:00 bash\nroot 152 146 152 0 1 21:10 pts/1 00:00:00 ps -efL\n")),(0,r.kt)("p",null,"We see a much larger number of threads, as expected."),(0,r.kt)("h2",{id:"practice-investigate-apache2-using-strace"},"Practice: Investigate ",(0,r.kt)("inlineCode",{parentName:"h2"},"apache2")," Using ",(0,r.kt)("inlineCode",{parentName:"h2"},"strace")),(0,r.kt)("p",null,"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to discover the server document root.\nThe document root is the path in the filesystem from where ",(0,r.kt)("inlineCode",{parentName:"p"},"httpd")," serves all the files requested by the clients."),(0,r.kt)("p",null,"First, you will have to stop the running container using ",(0,r.kt)("inlineCode",{parentName:"p"},"make stop"),", then restart it with ",(0,r.kt)("inlineCode",{parentName:"p"},"make run-privileged"),"."),(0,r.kt)("p",null,"Then you will use ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," inside the container to attach to the worker processes (use the ",(0,r.kt)("inlineCode",{parentName:"p"},"-p")," option for this).\nYou will also have to use the ",(0,r.kt)("inlineCode",{parentName:"p"},"-f")," flag with ",(0,r.kt)("inlineCode",{parentName:"p"},"strace"),", so that it will follow all the threads inside the processes."),(0,r.kt)("p",null,"After you have attached successfully to all worker processes, use the ",(0,r.kt)("inlineCode",{parentName:"p"},"curl")," command to send a request, like the one in the beginning of this section."),(0,r.kt)("p",null,"Then check the ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," output to see what files were opened by the server."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/apache2-strace"},"Quiz")),(0,r.kt)("h2",{id:"conclusion"},"Conclusion"),(0,r.kt)("p",null,'So far, you\'ve probably seen that spawning a process can "use" a different program (hence the path in the args of ',(0,r.kt)("inlineCode",{parentName:"p"},"system")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"Popen"),"), but some languages such as Python allow you to spawn a process that executes a function from the same script.\nA thread, however, can only start from a certain entry point ",(0,r.kt)("strong",{parentName:"p"},"within the current address space"),", as it is bound to the same process.\nConcretely, a process is but a group of threads.\nFor this reason, when we talk about scheduling or synchronization, we talk about threads.\nA thread is, thus, an abstraction of a task running on a CPU core.\nA process is a logical group of such tasks."),(0,r.kt)("p",null,"We can sum up what we've learned so far by saying that processes are better used for separate, independent work, such as the different connections handled by a server.\nConversely, threads are better suited for replicated work: when the same task has to be performed on multiple cores.\nHowever, replicated work can also be suited for processes.\nDistributed applications, however, leverage different processes as this allows them to run on multiple physical machines at once.\nThis is required by the very large workloads such applications are commonly required to process."),(0,r.kt)("p",null,"These rules are not set in stone, though.\nLike we saw in the ",(0,r.kt)("inlineCode",{parentName:"p"},"apache2")," example, the server uses multiple threads as well as multiple processes.\nThis provides a degree of stability - if one worker thread crashes, it will only crash the other threads belonging to the same process - while still taking advantage of the light resource usage inherent to threads."),(0,r.kt)("p",null,"These kinds of trade-offs are a normal part of the development of real-world applications."))}h.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/232c749b.2bec8249.js b/17/assets/js/232c749b.2bec8249.js new file mode 100644 index 0000000000..4e46419ed8 --- /dev/null +++ b/17/assets/js/232c749b.2bec8249.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5899],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function l(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var s=r.createContext({}),p=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(s.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,l=e.originalType,s=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),u=p(n),d=a,m=u["".concat(s,".").concat(d)]||u[d]||f[d]||l;return n?r.createElement(m,o(o({ref:t},c),{},{components:n})):r.createElement(m,o({ref:t},c))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var l=n.length,o=new Array(l);o[0]=d;var i={};for(var s in t)hasOwnProperty.call(t,s)&&(i[s]=t[s]);i.originalType=e,i[u]="string"==typeof e?e:a,o[1]=i;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>f,frontMatter:()=>l,metadata:()=>i,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const l={},o="Syscall Used by `fopen()`",i={unversionedId:"Lab/IO/quiz/fopen-syscall",id:"Lab/IO/quiz/fopen-syscall",title:"Syscall Used by `fopen()`",description:"Question Text",source:"@site/docs/Lab/IO/quiz/fopen-syscall.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/fopen-syscall",permalink:"/operating-systems/17/Lab/IO/quiz/fopen-syscall",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedaback",id:"feedaback",level:2}],c={toc:p},u="wrapper";function f(e){let{components:t,...n}=e;return(0,a.kt)(u,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"syscall-used-by-fopen"},"Syscall Used by ",(0,a.kt)("inlineCode",{parentName:"h1"},"fopen()")),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Use ",(0,a.kt)("inlineCode",{parentName:"p"},"strace")," to determine the syscall called by ",(0,a.kt)("inlineCode",{parentName:"p"},"fopen()")," to access the file.\nWhich one is it?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"read()"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"openat()"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"write()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"fstat()")))),(0,a.kt)("h2",{id:"feedaback"},"Feedaback"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/simple-file-handling$ strace ./file_operations\n[...]\nopenat(AT_FDCWD, "file.txt", O_RDONLY) = 3\nfstat(3, {st_mode=S_IFREG|0664, st_size=11, ...}) = 0\nread(3, "C was here!", 4096) = 11\n[...]\n')),(0,a.kt)("p",null,"So ",(0,a.kt)("inlineCode",{parentName:"p"},"fopen()"),"'s (main) underlying syscall is ",(0,a.kt)("inlineCode",{parentName:"p"},"openat()"),"."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/2330fd66.bba33578.js b/17/assets/js/2330fd66.bba33578.js new file mode 100644 index 0000000000..b6a2b62517 --- /dev/null +++ b/17/assets/js/2330fd66.bba33578.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1487],{3905:(e,n,t)=>{t.d(n,{Zo:()=>p,kt:()=>g});var a=t(7294);function r(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function l(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);n&&(a=a.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,a)}return t}function o(e){for(var n=1;n=0||(r[t]=e[t]);return r}(e,n);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(r[t]=e[t])}return r}var i=a.createContext({}),c=function(e){var n=a.useContext(i),t=n;return e&&(t="function"==typeof e?e(n):o(o({},n),e)),t},p=function(e){var n=c(e.components);return a.createElement(i.Provider,{value:n},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var n=e.children;return a.createElement(a.Fragment,{},n)}},h=a.forwardRef((function(e,n){var t=e.components,r=e.mdxType,l=e.originalType,i=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),m=c(t),h=r,g=m["".concat(i,".").concat(h)]||m[h]||u[h]||l;return t?a.createElement(g,o(o({ref:n},p),{},{components:t})):a.createElement(g,o({ref:n},p))}));function g(e,n){var t=arguments,r=n&&n.mdxType;if("string"==typeof e||r){var l=t.length,o=new Array(l);o[0]=h;var s={};for(var i in n)hasOwnProperty.call(n,i)&&(s[i]=n[i]);s.originalType=e,s[m]="string"==typeof e?e:r,o[1]=s;for(var c=2;c{t.r(n),t.d(n,{assets:()=>i,contentTitle:()=>o,default:()=>u,frontMatter:()=>l,metadata:()=>s,toc:()=>c});var a=t(7462),r=(t(7294),t(3905));const l={},o="High-Level Languages",s={unversionedId:"Lab/Software Stack/high-level-lang",id:"Lab/Software Stack/high-level-lang",title:"High-Level Languages",description:"Using the standard C library (libc) frees the programmer from the cumbersome steps of invoking system calls and reimplementing common features.",source:"@site/docs/Lab/Software Stack/high-level-lang.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/high-level-lang",permalink:"/operating-systems/17/Lab/Software Stack/high-level-lang",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Library calls vs system calls",permalink:"/operating-systems/17/Lab/Software Stack/libcall-syscall"},next:{title:"App Investigation",permalink:"/operating-systems/17/Lab/Software Stack/app-investigate"}},i={},c=[{value:"Practice",id:"practice",level:2}],p={toc:c},m="wrapper";function u(e){let{components:n,...t}=e;return(0,r.kt)(m,(0,a.Z)({},p,t,{components:n,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"high-level-languages"},"High-Level Languages"),(0,r.kt)("p",null,"Using the standard C library (libc) frees the programmer from the cumbersome steps of invoking system calls and reimplementing common features.\nStill, for improved development time and safety, other programming languages can be used, such as Rust, Python, JavaScript.\nMost (if not all) of these high-level programming languages still make use of the standard C library.\nSuch that a call to a function in Python would end-up making a call to a function in the standard C library."),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"support/high-level-lang/"),' folder stores the implementation of a simple "Hello, World!"-printing program in Python.\nWe simply invoke the ',(0,r.kt)("inlineCode",{parentName:"p"},"python")," interpreter to run the program:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/high-level-lang$ python hello.py\nHello, world!\n")),(0,r.kt)("p",null,"We count the number of functions called from the standard C library and the number of system calls:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/high-level-lang$ ltrace -l 'libc*' python hello.py 2> libc.out\nHello, world!\n\nstudent@os:~/.../lab/support/high-level-lang$ wc -l libc.out\n50469 out\n\nstudent@os:~/.../lab/support/high-level-lang$ strace python hello.py 2> syscall.out\nHello, world!\n\nstudent@os:~/.../lab/support/high-level-lang$ wc -l syscall.out\n948 syscall.out\n")),(0,r.kt)("p",null,"The dynamic standard C library (",(0,r.kt)("inlineCode",{parentName:"p"},"libc.so.6"),") is a dependency of the Python interpreter (",(0,r.kt)("inlineCode",{parentName:"p"},"/usr/bin/python3"),"):"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/high-level-lang$ ldd /usr/bin/python3\n[...]\n libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fa6fd6d0000)\n[...]\n")),(0,r.kt)("p",null,"We can see the complexity of invoking the Python interpreter, resulting in more the 50,000 of library calls being made.\nThis means added overhead versus a simple C function.\nHowever, this also means faster development in the Python programming language.\nEach new layer in the software stack simplifies development but adds overhead."),(0,r.kt)("p",null,"We can use ",(0,r.kt)("inlineCode",{parentName:"p"},"perf"),' to compare the running time between the Python and a C "Hello, World!"-printing programs:'),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/high-level-lang$ sudo perf stat ../static-dynamic/hello\nHello, World!\n\n Performance counter stats for '../static-dynamic/hello':\n\n 0.46 msec task-clock # 0.559 CPUs utilized\n 0 context-switches # 0.000 K/sec\n 0 cpu-migrations # 0.000 K/sec\n 52 page-faults # 0.114 M/sec\n 859,341 cycles # 1.882 GHz\n 713,395 instructions # 0.83 insn per cycle\n 141,710 branches # 310.393 M/sec\n 6,208 branch-misses # 4.38% of all branches\n\n 0.000816974 seconds time elapsed\n\n 0.000872000 seconds user\n 0.000000000 seconds sys\n\nstudent@os:~/.../lab/support/high-level-lang$ sudo perf stat python hello.py\nHello, world!\n\n Performance counter stats for 'python hello.py':\n\n 69.39 msec task-clock # 0.992 CPUs utilized\n 2 context-switches # 0.029 K/sec\n 0 cpu-migrations # 0.000 K/sec\n 1,115 page-faults # 0.016 M/sec\n 74,405,125 cycles # 1.072 GHz\n 84,957,056 instructions # 1.14 insn per cycle\n 18,574,724 branches # 267.689 M/sec\n 759,104 branch-misses # 4.09% of all branches\n\n 0.069981351 seconds time elapsed\n\n 0.054376000 seconds user\n 0.015536000 seconds sys\n")),(0,r.kt)("p",null,"We can see that on all metrics, the running of the Python program is less efficient than the running of the C program.\nThe Python code takes ",(0,r.kt)("inlineCode",{parentName:"p"},"69")," milliseconds, whereas the C code runs in less than ",(0,r.kt)("inlineCode",{parentName:"p"},"1")," millisecond."),(0,r.kt)("p",null,"When deciding what programming language and what libraries and software components to use, you have to balance requirements for fast development and increased safety (inherent to higher-level programming languages) with requirements for speed or efficiency (common to lower-level programming languages such as C).\nNewer modern programming languages such as Go, Rust, D aim to add the benefits of high-level programming languages and keep efficiency close to the C programming language.\nGenerally, additional software layers (libraries, language environments, interpreters) simplify development but decrease speed and efficiency."),(0,r.kt)("h2",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/high-level-lang/")," folder and go through the practice items below."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"make")," to create the ",(0,r.kt)("inlineCode",{parentName:"p"},"hello")," executable from the ",(0,r.kt)("inlineCode",{parentName:"p"},"hello.go"),' file (a Go "Hello, World!"-printing program).\nUse ',(0,r.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to compute the number of library calls and system calls.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"perf")," to measure the running time."),(0,r.kt)("p",{parentName:"li"},'Compare the values with those from the "Hello, World!"-printing programs in C and Python.')),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},'Create a "Hello, World!"-printing program in a programming language of your choice (other than C, Python and Go).\nFind the values above (library calls, system calls and running time).'))),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Software%20Stack/quiz/high-level-lang"},"Quiz")))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/23374ca6.0de54ebe.js b/17/assets/js/23374ca6.0de54ebe.js new file mode 100644 index 0000000000..fcf1ec0586 --- /dev/null +++ b/17/assets/js/23374ca6.0de54ebe.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8421],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>m});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var u=n.createContext({}),l=function(e){var t=n.useContext(u),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},s=function(e){var t=l(e.components);return n.createElement(u.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},y=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,u=e.parentName,s=c(e,["components","mdxType","originalType","parentName"]),p=l(r),y=o,m=p["".concat(u,".").concat(y)]||p[y]||f[y]||a;return r?n.createElement(m,i(i({ref:t},s),{},{components:r})):n.createElement(m,i({ref:t},s))}));function m(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,i=new Array(a);i[0]=y;var c={};for(var u in t)hasOwnProperty.call(t,u)&&(c[u]=t[u]);c.originalType=e,c[p]="string"==typeof e?e:o,i[1]=c;for(var l=2;l{r.r(t),r.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>f,frontMatter:()=>a,metadata:()=>c,toc:()=>l});var n=r(7462),o=(r(7294),r(3905));const a={slug:"/"},i="Intro",c={unversionedId:"README",id:"README",title:"Intro",description:"This is a landing page for your course.",source:"@site/docs/README.md",sourceDirName:".",slug:"/",permalink:"/operating-systems/17/",draft:!1,tags:[],version:"current",frontMatter:{slug:"/"},sidebar:"sidebar",next:{title:"Lecture",permalink:"/operating-systems/17/Lecture/"}},u={},l=[],s={toc:l},p="wrapper";function f(e){let{components:t,...r}=e;return(0,o.kt)(p,(0,n.Z)({},s,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"intro"},"Intro"),(0,o.kt)("p",null,"This is a landing page for your course."),(0,o.kt)("p",null,"Here you will add infomation about your course that a student might want to know at first glance."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/2641b427.60e47a37.js b/17/assets/js/2641b427.60e47a37.js new file mode 100644 index 0000000000..2ac530a7a5 --- /dev/null +++ b/17/assets/js/2641b427.60e47a37.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8986],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>u});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function s(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function o(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var p=a.createContext({}),l=function(e){var t=a.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},c=function(e){var t=l(e.components);return a.createElement(p.Provider,{value:t},e.children)},m="mdxType",h={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},d=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,s=e.originalType,p=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),m=l(n),d=r,u=m["".concat(p,".").concat(d)]||m[d]||h[d]||s;return n?a.createElement(u,o(o({ref:t},c),{},{components:n})):a.createElement(u,o({ref:t},c))}));function u(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var s=n.length,o=new Array(s);o[0]=d;var i={};for(var p in t)hasOwnProperty.call(t,p)&&(i[p]=t[p]);i.originalType=e,i[m]="string"==typeof e?e:r,o[1]=i;for(var l=2;l{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>o,default:()=>h,frontMatter:()=>s,metadata:()=>i,toc:()=>l});var a=n(7462),r=(n(7294),n(3905));const s={},o="Processes",i={unversionedId:"Lab/Compute/processes",id:"Lab/Compute/processes",title:"Processes",description:"A process is simply a running program.",source:"@site/docs/Lab/Compute/processes.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/processes",permalink:"/operating-systems/17/Lab/Compute/processes",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Hardware Perspective",permalink:"/operating-systems/17/Lab/Compute/hardware-perspective"},next:{title:"Threads",permalink:"/operating-systems/17/Lab/Compute/threads"}},p={},l=[{value:"Sum of the Elements in an Array",id:"sum-of-the-elements-in-an-array",level:2},{value:"Spreading the Work Among Other Processes",id:"spreading-the-work-among-other-processes",level:2},{value:"Practice: Baby steps - Python",id:"practice-baby-steps---python",level:3},{value:"Practice: High level - Python",id:"practice-high-level---python",level:3},{value:"Practice: Lower level - C",id:"practice-lower-level---c",level:3},{value:"Practice: Wait for Me",id:"practice-wait-for-me",level:3},{value:"Practice: fork()",id:"practice-fork",level:3}],c={toc:l},m="wrapper";function h(e){let{components:t,...s}=e;return(0,r.kt)(m,(0,a.Z)({},c,s,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"processes"},"Processes"),(0,r.kt)("p",null,"A process is simply a running program.\nLet's take the ",(0,r.kt)("inlineCode",{parentName:"p"},"ls")," command as a trivial example.\n",(0,r.kt)("inlineCode",{parentName:"p"},"ls")," is a ",(0,r.kt)("strong",{parentName:"p"},"program")," on your system.\nIt has a binary file which you can find and inspect with the help of the ",(0,r.kt)("inlineCode",{parentName:"p"},"which")," command:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ which ls\n/usr/bin/ls\n\nstudent@os:~$ file /usr/bin/ls\n/usr/bin/ls: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=6e3da6f0bc36b6398b8651bbc2e08831a21a90da, for GNU/Linux 3.2.0, stripped\n")),(0,r.kt)("p",null,"When you run it, the ",(0,r.kt)("inlineCode",{parentName:"p"},"ls")," binary stored ",(0,r.kt)("strong",{parentName:"p"},"on the disk")," at ",(0,r.kt)("inlineCode",{parentName:"p"},"/usr/bin/ls")," is read by another application called the ",(0,r.kt)("strong",{parentName:"p"},"loader"),".\nThe loader spawns a ",(0,r.kt)("strong",{parentName:"p"},"process")," by copying some of the contents ",(0,r.kt)("inlineCode",{parentName:"p"},"/usr/bin/ls")," in memory (such as the ",(0,r.kt)("inlineCode",{parentName:"p"},".text"),", ",(0,r.kt)("inlineCode",{parentName:"p"},".rodata")," and ",(0,r.kt)("inlineCode",{parentName:"p"},".data")," sections).\nUsing ",(0,r.kt)("inlineCode",{parentName:"p"},"strace"),", we can see the ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/execve.2.html"},(0,r.kt)("inlineCode",{parentName:"a"},"execve"))," system call:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace -s 100 ls -a # -s 100 limits strings to 100 bytes instead of the default 32\nexecve("/usr/bin/ls", ["ls", "-a"], 0x7fffa7e0d008 /* 61 vars */) = 0\n[...]\nwrite(1, ". .. content\\tCONTRIBUTING.md COPYING.md .git .gitignore README.md REVIEWING.md\\n", 86. .. content CONTRIBUTING.md COPYING.md .git .gitignore README.md REVIEWING.md\n) = 86\nclose(1) = 0\nclose(2) = 0\nexit_group(0) = ?\n+++ exited with 0 +++\n')),(0,r.kt)("p",null,"Look at its parameters:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"the path to the ",(0,r.kt)("strong",{parentName:"li"},"program"),": ",(0,r.kt)("inlineCode",{parentName:"li"},"/usr/bin/ls")),(0,r.kt)("li",{parentName:"ul"},"the list of arguments: ",(0,r.kt)("inlineCode",{parentName:"li"},'"ls", "-a"')),(0,r.kt)("li",{parentName:"ul"},"the environment variables: the rest of the syscall's arguments")),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"execve")," invokes the loader to load the VAS of the ",(0,r.kt)("inlineCode",{parentName:"p"},"ls")," process ",(0,r.kt)("strong",{parentName:"p"},"by replacing that of the existing process"),".\nAll subsequent syscalls are performed by the newly spawned ",(0,r.kt)("inlineCode",{parentName:"p"},"ls")," process.\nWe will get into more details regarding ",(0,r.kt)("inlineCode",{parentName:"p"},"execve")," ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#first-step-system-dissected"},"towards the end of this lab"),"."),(0,r.kt)("p",null,(0,r.kt)("img",{alt:"Loading of `ls` Process",src:n(3046).Z,width:"432",height:"462"})),(0,r.kt)("h2",{id:"sum-of-the-elements-in-an-array"},"Sum of the Elements in an Array"),(0,r.kt)("p",null,"Let's assume we only have one process on our system, and that process knows how to add the numbers in an array.\nIt can use however many resources it wants, since there is no other process to contest it.\nIt would probably look like the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sum-array/c/sum_array_sequential.c"),".\nThe program also measures the time spent computing the sum.\nLet's compile and run it:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/sum-array/c$ ./sum_array_sequential\nArray sum is: 49945994146\nTime spent: 127 ms\n")),(0,r.kt)("p",null,"You will most likely get a different sum (because the array is made up of random numbers) and a different time than the ones shown above.\nThis is perfectly fine.\nUse these examples qualitatively, not quantitatively."),(0,r.kt)("h2",{id:"spreading-the-work-among-other-processes"},"Spreading the Work Among Other Processes"),(0,r.kt)("p",null,"Due to how it's implemented so far, our program only uses one of our CPU's cores.\nWe never tell it to distribute its workload to other cores.\nThis is wasteful as the rest of our cores remain unused:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ lscpu | grep ^CPU\\(s\\):\nCPU(s): 8\n")),(0,r.kt)("p",null,"We have 7 more cores waiting to add numbers in our array."),(0,r.kt)("p",null,(0,r.kt)("img",{alt:"What if we used 100% of the CPU?",src:n(9886).Z,width:"541",height:"500"})),(0,r.kt)("p",null,"What if we use 7 more processes and spread the task of adding the numbers in this array between them?\nIf we split the array into several equal parts and designate a separate process to calculate the sum of each part, we should get a speedup because now the work performed by each individual process is reduced."),(0,r.kt)("p",null,"Let's take it methodically.\nCompile and run ",(0,r.kt)("inlineCode",{parentName:"p"},"sum_array_processes.c")," using 1, 2, 4 and 8 processes respectively.\nIf your system only has 4 cores (",(0,r.kt)("a",{parentName:"p",href:"https://www.intel.com/content/www/us/en/gaming/resources/hyper-threading.html"},"hyperthreading")," included), limit your runs to 4 processes.\nNote the running times for each number of processes.\nWe expect the speedups compared to our reference run to be 1, 2, 4 and 8 respectively, right?"),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/processes-speedup"},"Quiz")),(0,r.kt)("p",null,"You most likely did get some speedup, especially when using 8 processes.\nNow we will try to improve this speedup by using ",(0,r.kt)("strong",{parentName:"p"},"threads")," instead."),(0,r.kt)("p",null,"Also notice that we're not using hundreds or thousands of processes.\nAssuming our system has 8 cores, only 8 ",(0,r.kt)("em",{parentName:"p"},"threads")," (we'll see this later in the lab) can run at the same time.\nIn general, ",(0,r.kt)("strong",{parentName:"p"},"the maximum number of threads that can run at the same time is equal to the number of cores"),".\nIn our example, each process only has one thread: its main thread.\nSo by consequence and by forcing the terminology (because it's the main thread of these processes that is running, not the processes themselves), we can only run in parallel a number of processes equal to at most the number of cores."),(0,r.kt)("h3",{id:"practice-baby-steps---python"},"Practice: Baby steps - Python"),(0,r.kt)("p",null,"Run the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/create-process/popen.py"),".\nIt simply spawns a new process running the ",(0,r.kt)("inlineCode",{parentName:"p"},"ls")," command using ",(0,r.kt)("a",{parentName:"p",href:"https://docs.python.org/3/library/subprocess.html#subprocess.Popen"},(0,r.kt)("inlineCode",{parentName:"a"},"subprocess.Popen()")),".\nDo not worry about the huge list of arguments that ",(0,r.kt)("inlineCode",{parentName:"p"},"Popen()")," takes.\nThey are used for ",(0,r.kt)("strong",{parentName:"p"},"inter-process-communication"),".\nYou'll learn more about this in the ",(0,r.kt)("a",{parentName:"p",href:"../../../app-interact/"},"Application Interaction chapter"),"."),(0,r.kt)("p",null,"Note that this usage of ",(0,r.kt)("inlineCode",{parentName:"p"},"Popen()")," is not entirely correct.\nYou'll discover why in the next exercise, but for now focus on simply understanding how to use ",(0,r.kt)("inlineCode",{parentName:"p"},"Popen()")," on its own."),(0,r.kt)("p",null,"Now change the command to anything you want.\nAlso give it some arguments.\nFrom the outside, it's as if you were running these commands from the terminal."),(0,r.kt)("h3",{id:"practice-high-level---python"},"Practice: High level - Python"),(0,r.kt)("p",null,"Head over to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sleepy/sleepy_creator.py"),".\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"subprocess.Popen()")," to spawn 10 ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep 1000")," processes."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Solve ",(0,r.kt)("inlineCode",{parentName:"p"},"TODO 1"),": use ",(0,r.kt)("inlineCode",{parentName:"p"},"subprocess.Popen()")," to spawn 10 ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep 1000")," processes."),(0,r.kt)("p",{parentName:"li"},"Start the script:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/sleepy$ python3 sleepy_creator.py\n")),(0,r.kt)("p",{parentName:"li"},"Look for the parent process:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep "python3 sleepy_creator.py")\n')),(0,r.kt)("p",{parentName:"li"},"It is a ",(0,r.kt)("inlineCode",{parentName:"p"},"python3")," process, as this is the interpreter that runs the script, but we call it the ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," process for simplicity.\nNo output will be provided by the above command, as the parent process (",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py"),") dies before its child processes (the 10 ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep 1000")," subprocesses) finish their execution.\nThe parent process of the newly created child processes is an ",(0,r.kt)("inlineCode",{parentName:"p"},"init"),"-like process: either ",(0,r.kt)("inlineCode",{parentName:"p"},"systemd"),"/",(0,r.kt)("inlineCode",{parentName:"p"},"init")," or another system process that adopts orphan processes.\nLook for the ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep")," child processes using:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)\n PID PPID CMD\n4164 1680 sleep 1000\n4165 1680 sleep 1000\n4166 1680 sleep 1000\n4167 1680 sleep 1000\n4168 1680 sleep 1000\n4169 1680 sleep 1000\n4170 1680 sleep 1000\n4171 1680 sleep 1000\n4172 1680 sleep 1000\n4173 1680 sleep 1000\n")),(0,r.kt)("p",{parentName:"li"},"Notice that the child processes do not have ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," as a parent.\nWhat's more, as you saw above, ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," doesn't even exist anymore.\nThe child processes have been adopted by an ",(0,r.kt)("inlineCode",{parentName:"p"},"init"),"-like process (in the output above, that process has PID ",(0,r.kt)("inlineCode",{parentName:"p"},"1680")," - ",(0,r.kt)("inlineCode",{parentName:"p"},"PPID")," stands for ",(0,r.kt)("em",{parentName:"p"},"parent process ID"),")."),(0,r.kt)("p",{parentName:"li"},(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/parent-of-sleep-processes"},"Quiz"))),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Solve ",(0,r.kt)("inlineCode",{parentName:"p"},"TODO 2"),": change the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," so that the ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep 1000")," processes remain the children of ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py"),".\nThis means that the parent / creator process must ",(0,r.kt)("strong",{parentName:"p"},"not")," exit until its children have finished their execution.\nIn other words, the parent / creator process must ",(0,r.kt)("strong",{parentName:"p"},"wait")," for the termination of its children.\nCheck out ",(0,r.kt)("a",{parentName:"p",href:"https://docs.python.org/3/library/subprocess.html#subprocess.Popen.wait"},(0,r.kt)("inlineCode",{parentName:"a"},"Popen.wait()"))," and add the code that makes the parent / creator process wait for its children.\nBefore anything, terminate the ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep")," processes created above:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ pkill sleep\n")),(0,r.kt)("p",{parentName:"li"},"Start the program, again, as you did before:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/sleepy$ python3 sleepy_creator.py\n")),(0,r.kt)("p",{parentName:"li"},"On another terminal, verify that ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," remains the parent of the ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep")," processes it creates:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ps -e -H -o pid,ppid,cmd | (head -1; grep sleep)\n PID PPID CMD\n16107 9855 python3 sleepy_creator.py\n16108 16107 sleep 1000\n16109 16107 sleep 1000\n16110 16107 sleep 1000\n16111 16107 sleep 1000\n16112 16107 sleep 1000\n16113 16107 sleep 1000\n16114 16107 sleep 1000\n16115 16107 sleep 1000\n16116 16107 sleep 1000\n16117 16107 sleep 1000\n")),(0,r.kt)("p",{parentName:"li"},"Note that the parent process ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," (",(0,r.kt)("inlineCode",{parentName:"p"},"PID 16107"),") is still alive, and its child processes (the 10 ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep 1000"),") have its ID as their ",(0,r.kt)("inlineCode",{parentName:"p"},"PPID"),".\nYou've successfully waited for the child processes to finish their execution."))),(0,r.kt)("h3",{id:"practice-lower-level---c"},"Practice: Lower level - C"),(0,r.kt)("p",null,"Now let's see how to create a child process in C.\nThere are multiple ways of doing this.\nFor now, we'll start with a higher-level approach."),(0,r.kt)("p",null,"Go to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sleepy/sleepy_creator.c")," and use ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/system.3.html"},(0,r.kt)("inlineCode",{parentName:"a"},"system"))," to create a ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep 1000")," process."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/create-sleepy-process-ending"},"Quiz")),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"man")," page also mentions that ",(0,r.kt)("inlineCode",{parentName:"p"},"system")," calls ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"exec()")," to run the command it's given.\nIf you want to find out more about them, head over to the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#mini-shell"},"Arena and create your own mini-shell"),"."),(0,r.kt)("h3",{id:"practice-wait-for-me"},"Practice: Wait for Me"),(0,r.kt)("p",null,"Run the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/wait-for-me/wait_for_me_processes.py"),".\nThe parent process creates one child that writes and message to the given file.\nThen the parent reads that message.\nSimple enough, right?\nBut running the code raises a ",(0,r.kt)("inlineCode",{parentName:"p"},"FileNotFoundError"),".\nIf you inspect the file you gave the script as an argument, it does contain a string.\nWhat's going on?"),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/cause-of-file-not-found-error"},"Quiz")),(0,r.kt)("p",null,"In order to solve race conditions, we need ",(0,r.kt)("strong",{parentName:"p"},"synchronization"),".\nThis is a mechanism similar to a set of traffic lights in a crossroads.\nJust like traffic lights allow some cars to pass only after others have already passed, synchronization is a means for threads to communicate with each other and tell each other to access a resource or not."),(0,r.kt)("p",null,"The most basic form of synchronization is ",(0,r.kt)("strong",{parentName:"p"},"waiting"),".\nConcretely, if the parent process ",(0,r.kt)("strong",{parentName:"p"},"waits")," for the child to end, we are sure the file is created and its contents are written.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"join()")," to make the parent wait for its child before reading the file."),(0,r.kt)("h3",{id:"practice-fork"},"Practice: ",(0,r.kt)("inlineCode",{parentName:"h3"},"fork()")),(0,r.kt)("p",null,"Up to now we've been creating processes using various high-level APIs, such as ",(0,r.kt)("inlineCode",{parentName:"p"},"Popen()"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"Process()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"system()"),".\nYes, despite being a C function, as you've seen from its man page, ",(0,r.kt)("inlineCode",{parentName:"p"},"system()")," itself calls 2 other functions: ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," to create a process and ",(0,r.kt)("inlineCode",{parentName:"p"},"execve()")," to execute the given command.\nAs you already know from the ",(0,r.kt)("a",{parentName:"p",href:"../../../software-stack/"},"Software Stack")," chapter, library functions may call one or more underlying system calls or other functions.\nNow we will move one step lower on the call stack and call ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," ourselves."),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," creates one child process that is ",(0,r.kt)("em",{parentName:"p"},"almost")," identical to its parent.\nWe say that ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," returns ",(0,r.kt)("strong",{parentName:"p"},"twice"),": once in the parent process and once more in the child process.\nThis means that after ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," returns, assuming no error has occurred, both the child and the parent resume execution from the same place: the instruction following the call to ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()"),".\nWhat's different between the two processes is the value returned by ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()"),":"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("strong",{parentName:"li"},"child process"),": ",(0,r.kt)("inlineCode",{parentName:"li"},"fork()")," returns 0"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("strong",{parentName:"li"},"parent process"),": ",(0,r.kt)("inlineCode",{parentName:"li"},"fork()")," returns the PID of the child process (> 0)"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("strong",{parentName:"li"},"on error"),": ",(0,r.kt)("inlineCode",{parentName:"li"},"fork()")," returns -1, only once, in the initial process")),(0,r.kt)("p",null,"Therefore, the typical code for handling a ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," is available in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/create-process/fork.c"),".\nTake a look at it and then run it.\nNotice what each of the two processes prints:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"the PID of the child is also known by the parent"),(0,r.kt)("li",{parentName:"ul"},"the PPID of the child is the PID of the parent")),(0,r.kt)("p",null,"Unlike ",(0,r.kt)("inlineCode",{parentName:"p"},"system()"),", who also waits for its child, when using ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," we must do the waiting ourselves.\nIn order to wait for a process to end, we use the ",(0,r.kt)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},(0,r.kt)("inlineCode",{parentName:"a"},"waitpid()"))," syscall.\nIt places the exit code of the child process in the ",(0,r.kt)("inlineCode",{parentName:"p"},"status")," parameter.\nThis argument is actually a bit-field containing more information than merely the exit code.\nTo retrieve the exit code, we use the ",(0,r.kt)("inlineCode",{parentName:"p"},"WEXITSTATUS")," macro.\nKeep in mind that ",(0,r.kt)("inlineCode",{parentName:"p"},"WEXITSTATUS")," only makes sense if ",(0,r.kt)("inlineCode",{parentName:"p"},"WIFEXITED")," is true, i.e. if the child process finished on its own and wasn't killed by another one or by an illegal action (such as a segfault or illegal instruction) for example.\nOtherwise, ",(0,r.kt)("inlineCode",{parentName:"p"},"WEXITSTATUS")," will return something meaningless.\nYou can view the rest of the information stored in the ",(0,r.kt)("inlineCode",{parentName:"p"},"status")," bit-field ",(0,r.kt)("a",{parentName:"p",href:"https://linux.die.net/man/2/waitpid"},"in the man page"),"."),(0,r.kt)("p",null,"Now modify the example to do the following:"),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Change the return value of the child process so that the value displayed by the parent is changed.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},'Create a child process of the newly created child.\nUse a similar logic and a similar set of prints to those in the support code.\nTake a look at the printed PIDs.\nMake sure the PPID of the "grandchild" is the PID of the child, whose PPID is, in turn, the PID of the parent.'))),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Moral of the story"),": Usually the execution flow is:"),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"fork()"),", followed by")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"wait()")," (called by the parent)")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"exit()"),", called by the child."))),(0,r.kt)("p",null,"The order of last 2 steps may be swapped."))}h.isMDXComponent=!0},9886:(e,t,n)=>{n.d(t,{Z:()=>a});const a=n.p+"assets/images/100-percent-cpu-1138186529f154d864f643179e25cea1.jpeg"},3046:(e,t,n)=>{n.d(t,{Z:()=>a});const a=n.p+"assets/images/loading-of-ls-process-0dec67c0d0a826710e06f980224d5eb4.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/2666.5efb0c03.js b/17/assets/js/2666.5efb0c03.js new file mode 100644 index 0000000000..f82f4669e5 --- /dev/null +++ b/17/assets/js/2666.5efb0c03.js @@ -0,0 +1 @@ +(self.webpackChunkso=self.webpackChunkso||[]).push([[2666],{3905:(e,t,n)=>{"use strict";n.d(t,{Zo:()=>u,kt:()=>f});var o=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);t&&(o=o.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,o)}return n}function c(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(o=0;o=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var i=o.createContext({}),s=function(e){var t=o.useContext(i),n=t;return e&&(n="function"==typeof e?e(t):c(c({},t),e)),n},u=function(e){var t=s(e.components);return o.createElement(i.Provider,{value:t},e.children)},m="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return o.createElement(o.Fragment,{},t)}},p=o.forwardRef((function(e,t){var n=e.components,r=e.mdxType,a=e.originalType,i=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),m=s(n),p=r,f=m["".concat(i,".").concat(p)]||m[p]||d[p]||a;return n?o.createElement(f,c(c({ref:t},u),{},{components:n})):o.createElement(f,c({ref:t},u))}));function f(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var a=n.length,c=new Array(a);c[0]=p;var l={};for(var i in t)hasOwnProperty.call(t,i)&&(l[i]=t[i]);l.originalType=e,l[m]="string"==typeof e?e:r,c[1]=l;for(var s=2;s{"use strict";n.d(t,{Z:()=>s});var o=n(7462),r=n(7294),a=n(6010),c=n(5999),l=n(6668);const i={anchorWithStickyNavbar:"anchorWithStickyNavbar_LWe7",anchorWithHideOnScrollNavbar:"anchorWithHideOnScrollNavbar_WYt5"};function s(e){let{as:t,id:n,...s}=e;const{navbar:{hideOnScroll:u}}=(0,l.L)();return"h1"!==t&&n?r.createElement(t,(0,o.Z)({},s,{className:(0,a.Z)("anchor",u?i.anchorWithHideOnScrollNavbar:i.anchorWithStickyNavbar),id:n}),s.children,r.createElement("a",{className:"hash-link",href:`#${n}`,title:(0,c.I)({id:"theme.common.headingLinkTitle",message:"Direct link to heading",description:"Title for link to heading"})},"\u200b")):r.createElement(t,(0,o.Z)({},s,{id:void 0}))}},210:(e,t,n)=>{"use strict";n.d(t,{Z:()=>pe});var o=n(7294),r=n(3905),a=n(7462),c=n(5742);var l=n(2389),i=n(6010),s=n(2949),u=n(6668);function m(){const{prism:e}=(0,u.L)(),{colorMode:t}=(0,s.I)(),n=e.theme,o=e.darkTheme||n;return"dark"===t?o:n}var d=n(5281),p=n(7594),f=n.n(p);const h=/title=(?["'])(?.*?)\1/,g=/\{(?<range>[\d,-]+)\}/,y={js:{start:"\\/\\/",end:""},jsBlock:{start:"\\/\\*",end:"\\*\\/"},jsx:{start:"\\{\\s*\\/\\*",end:"\\*\\/\\s*\\}"},bash:{start:"#",end:""},html:{start:"\x3c!--",end:"--\x3e"}};function v(e,t){const n=e.map((e=>{const{start:n,end:o}=y[e];return`(?:${n}\\s*(${t.flatMap((e=>[e.line,e.block?.start,e.block?.end].filter(Boolean))).join("|")})\\s*${o})`})).join("|");return new RegExp(`^\\s*(?:${n})\\s*$`)}function b(e,t){let n=e.replace(/\n$/,"");const{language:o,magicComments:r,metastring:a}=t;if(a&&g.test(a)){const e=a.match(g).groups.range;if(0===r.length)throw new Error(`A highlight range has been given in code block's metastring (\`\`\` ${a}), but no magic comment config is available. Docusaurus applies the first magic comment entry's className for metastring ranges.`);const t=r[0].className,o=f()(e).filter((e=>e>0)).map((e=>[e-1,[t]]));return{lineClassNames:Object.fromEntries(o),code:n}}if(void 0===o)return{lineClassNames:{},code:n};const c=function(e,t){switch(e){case"js":case"javascript":case"ts":case"typescript":return v(["js","jsBlock"],t);case"jsx":case"tsx":return v(["js","jsBlock","jsx"],t);case"html":return v(["js","jsBlock","html"],t);case"python":case"py":case"bash":return v(["bash"],t);case"markdown":case"md":return v(["html","jsx","bash"],t);default:return v(Object.keys(y),t)}}(o,r),l=n.split("\n"),i=Object.fromEntries(r.map((e=>[e.className,{start:0,range:""}]))),s=Object.fromEntries(r.filter((e=>e.line)).map((e=>{let{className:t,line:n}=e;return[n,t]}))),u=Object.fromEntries(r.filter((e=>e.block)).map((e=>{let{className:t,block:n}=e;return[n.start,t]}))),m=Object.fromEntries(r.filter((e=>e.block)).map((e=>{let{className:t,block:n}=e;return[n.end,t]})));for(let p=0;p<l.length;){const e=l[p].match(c);if(!e){p+=1;continue}const t=e.slice(1).find((e=>void 0!==e));s[t]?i[s[t]].range+=`${p},`:u[t]?i[u[t]].start=p:m[t]&&(i[m[t]].range+=`${i[m[t]].start}-${p-1},`),l.splice(p,1)}n=l.join("\n");const d={};return Object.entries(i).forEach((e=>{let[t,{range:n}]=e;f()(n).forEach((e=>{d[e]??=[],d[e].push(t)}))})),{lineClassNames:d,code:n}}const E={codeBlockContainer:"codeBlockContainer_Ckt0"};function k(e){let{as:t,...n}=e;const r=function(e){const t={color:"--prism-color",backgroundColor:"--prism-background-color"},n={};return Object.entries(e.plain).forEach((e=>{let[o,r]=e;const a=t[o];a&&"string"==typeof r&&(n[a]=r)})),n}(m());return o.createElement(t,(0,a.Z)({},n,{style:r,className:(0,i.Z)(n.className,E.codeBlockContainer,d.k.common.codeBlock)}))}const N={codeBlockContent:"codeBlockContent_biex",codeBlockTitle:"codeBlockTitle_Ktv7",codeBlock:"codeBlock_bY9V",codeBlockStandalone:"codeBlockStandalone_MEMb",codeBlockLines:"codeBlockLines_e6Vv",codeBlockLinesWithNumbering:"codeBlockLinesWithNumbering_o6Pm",buttonGroup:"buttonGroup__atx"};function C(e){let{children:t,className:n}=e;return o.createElement(k,{as:"pre",tabIndex:0,className:(0,i.Z)(N.codeBlockStandalone,"thin-scrollbar",n)},o.createElement("code",{className:N.codeBlockLines},t))}var L=n(902);const w={attributes:!0,characterData:!0,childList:!0,subtree:!0};function B(e,t){const[n,r]=(0,o.useState)(),a=(0,o.useCallback)((()=>{r(e.current?.closest("[role=tabpanel][hidden]"))}),[e,r]);(0,o.useEffect)((()=>{a()}),[a]),function(e,t,n){void 0===n&&(n=w);const r=(0,L.zX)(t),a=(0,L.Ql)(n);(0,o.useEffect)((()=>{const t=new MutationObserver(r);return e&&t.observe(e,a),()=>t.disconnect()}),[e,r,a])}(n,(e=>{e.forEach((e=>{"attributes"===e.type&&"hidden"===e.attributeName&&(t(),a())}))}),{attributes:!0,characterData:!1,childList:!1,subtree:!1})}const x={plain:{backgroundColor:"#2a2734",color:"#9a86fd"},styles:[{types:["comment","prolog","doctype","cdata","punctuation"],style:{color:"#6c6783"}},{types:["namespace"],style:{opacity:.7}},{types:["tag","operator","number"],style:{color:"#e09142"}},{types:["property","function"],style:{color:"#9a86fd"}},{types:["tag-id","selector","atrule-id"],style:{color:"#eeebff"}},{types:["attr-name"],style:{color:"#c4b9fe"}},{types:["boolean","string","entity","url","attr-value","keyword","control","directive","unit","statement","regex","atrule","placeholder","variable"],style:{color:"#ffcc99"}},{types:["deleted"],style:{textDecorationLine:"line-through"}},{types:["inserted"],style:{textDecorationLine:"underline"}},{types:["italic"],style:{fontStyle:"italic"}},{types:["important","bold"],style:{fontWeight:"bold"}},{types:["important"],style:{color:"#c4b9fe"}}]};var T={Prism:n(7410).Z,theme:x};function O(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function Z(){return Z=Object.assign||function(e){for(var t=1;t<arguments.length;t++){var n=arguments[t];for(var o in n)Object.prototype.hasOwnProperty.call(n,o)&&(e[o]=n[o])}return e},Z.apply(this,arguments)}var j=/\r\n|\r|\n/,H=function(e){0===e.length?e.push({types:["plain"],content:"\n",empty:!0}):1===e.length&&""===e[0].content&&(e[0].content="\n",e[0].empty=!0)},_=function(e,t){var n=e.length;return n>0&&e[n-1]===t?e:e.concat(t)};function S(e,t){var n={};for(var o in e)Object.prototype.hasOwnProperty.call(e,o)&&-1===t.indexOf(o)&&(n[o]=e[o]);return n}var A=function(e){function t(){for(var t=this,n=[],o=arguments.length;o--;)n[o]=arguments[o];e.apply(this,n),O(this,"getThemeDict",(function(e){if(void 0!==t.themeDict&&e.theme===t.prevTheme&&e.language===t.prevLanguage)return t.themeDict;t.prevTheme=e.theme,t.prevLanguage=e.language;var n=e.theme?function(e,t){var n=e.plain,o=Object.create(null),r=e.styles.reduce((function(e,n){var o=n.languages,r=n.style;return o&&!o.includes(t)||n.types.forEach((function(t){var n=Z({},e[t],r);e[t]=n})),e}),o);return r.root=n,r.plain=Z({},n,{backgroundColor:null}),r}(e.theme,e.language):void 0;return t.themeDict=n})),O(this,"getLineProps",(function(e){var n=e.key,o=e.className,r=e.style,a=Z({},S(e,["key","className","style","line"]),{className:"token-line",style:void 0,key:void 0}),c=t.getThemeDict(t.props);return void 0!==c&&(a.style=c.plain),void 0!==r&&(a.style=void 0!==a.style?Z({},a.style,r):r),void 0!==n&&(a.key=n),o&&(a.className+=" "+o),a})),O(this,"getStyleForToken",(function(e){var n=e.types,o=e.empty,r=n.length,a=t.getThemeDict(t.props);if(void 0!==a){if(1===r&&"plain"===n[0])return o?{display:"inline-block"}:void 0;if(1===r&&!o)return a[n[0]];var c=o?{display:"inline-block"}:{},l=n.map((function(e){return a[e]}));return Object.assign.apply(Object,[c].concat(l))}})),O(this,"getTokenProps",(function(e){var n=e.key,o=e.className,r=e.style,a=e.token,c=Z({},S(e,["key","className","style","token"]),{className:"token "+a.types.join(" "),children:a.content,style:t.getStyleForToken(a),key:void 0});return void 0!==r&&(c.style=void 0!==c.style?Z({},c.style,r):r),void 0!==n&&(c.key=n),o&&(c.className+=" "+o),c})),O(this,"tokenize",(function(e,t,n,o){var r={code:t,grammar:n,language:o,tokens:[]};e.hooks.run("before-tokenize",r);var a=r.tokens=e.tokenize(r.code,r.grammar,r.language);return e.hooks.run("after-tokenize",r),a}))}return e&&(t.__proto__=e),t.prototype=Object.create(e&&e.prototype),t.prototype.constructor=t,t.prototype.render=function(){var e=this.props,t=e.Prism,n=e.language,o=e.code,r=e.children,a=this.getThemeDict(this.props),c=t.languages[n];return r({tokens:function(e){for(var t=[[]],n=[e],o=[0],r=[e.length],a=0,c=0,l=[],i=[l];c>-1;){for(;(a=o[c]++)<r[c];){var s=void 0,u=t[c],m=n[c][a];if("string"==typeof m?(u=c>0?u:["plain"],s=m):(u=_(u,m.type),m.alias&&(u=_(u,m.alias)),s=m.content),"string"==typeof s){var d=s.split(j),p=d.length;l.push({types:u,content:d[0]});for(var f=1;f<p;f++)H(l),i.push(l=[]),l.push({types:u,content:d[f]})}else c++,t.push(u),n.push(s),o.push(0),r.push(s.length)}c--,t.pop(),n.pop(),o.pop(),r.pop()}return H(l),i}(void 0!==c?this.tokenize(t,o,c,n):[o]),className:"prism-code language-"+n,style:void 0!==a?a.root:{},getLineProps:this.getLineProps,getTokenProps:this.getTokenProps})},t}(o.Component);const I=A,P={codeLine:"codeLine_lJS_",codeLineNumber:"codeLineNumber_Tfdd",codeLineContent:"codeLineContent_feaV"};function z(e){let{line:t,classNames:n,showLineNumbers:r,getLineProps:c,getTokenProps:l}=e;1===t.length&&"\n"===t[0].content&&(t[0].content="");const s=c({line:t,className:(0,i.Z)(n,r&&P.codeLine)}),u=t.map(((e,t)=>o.createElement("span",(0,a.Z)({key:t},l({token:e,key:t})))));return o.createElement("span",s,r?o.createElement(o.Fragment,null,o.createElement("span",{className:P.codeLineNumber}),o.createElement("span",{className:P.codeLineContent},u)):o.createElement(o.Fragment,null,u,o.createElement("br",null)))}var M=n(5999);const D={copyButtonCopied:"copyButtonCopied_obH4",copyButtonIcons:"copyButtonIcons_eSgA",copyButtonIcon:"copyButtonIcon_y97N",copyButtonSuccessIcon:"copyButtonSuccessIcon_LjdS"};function W(e){let{code:t,className:n}=e;const[r,a]=(0,o.useState)(!1),c=(0,o.useRef)(void 0),l=(0,o.useCallback)((()=>{!function(e,t){let{target:n=document.body}=void 0===t?{}:t;if("string"!=typeof e)throw new TypeError(`Expected parameter \`text\` to be a \`string\`, got \`${typeof e}\`.`);const o=document.createElement("textarea"),r=document.activeElement;o.value=e,o.setAttribute("readonly",""),o.style.contain="strict",o.style.position="absolute",o.style.left="-9999px",o.style.fontSize="12pt";const a=document.getSelection(),c=a.rangeCount>0&&a.getRangeAt(0);n.append(o),o.select(),o.selectionStart=0,o.selectionEnd=e.length;let l=!1;try{l=document.execCommand("copy")}catch{}o.remove(),c&&(a.removeAllRanges(),a.addRange(c)),r&&r.focus()}(t),a(!0),c.current=window.setTimeout((()=>{a(!1)}),1e3)}),[t]);return(0,o.useEffect)((()=>()=>window.clearTimeout(c.current)),[]),o.createElement("button",{type:"button","aria-label":r?(0,M.I)({id:"theme.CodeBlock.copied",message:"Copied",description:"The copied button label on code blocks"}):(0,M.I)({id:"theme.CodeBlock.copyButtonAriaLabel",message:"Copy code to clipboard",description:"The ARIA label for copy code blocks button"}),title:(0,M.I)({id:"theme.CodeBlock.copy",message:"Copy",description:"The copy button label on code blocks"}),className:(0,i.Z)("clean-btn",n,D.copyButton,r&&D.copyButtonCopied),onClick:l},o.createElement("span",{className:D.copyButtonIcons,"aria-hidden":"true"},o.createElement("svg",{className:D.copyButtonIcon,viewBox:"0 0 24 24"},o.createElement("path",{d:"M19,21H8V7H19M19,5H8A2,2 0 0,0 6,7V21A2,2 0 0,0 8,23H19A2,2 0 0,0 21,21V7A2,2 0 0,0 19,5M16,1H4A2,2 0 0,0 2,3V17H4V3H16V1Z"})),o.createElement("svg",{className:D.copyButtonSuccessIcon,viewBox:"0 0 24 24"},o.createElement("path",{d:"M21,7L9,19L3.5,13.5L4.91,12.09L9,16.17L19.59,5.59L21,7Z"}))))}const R={wordWrapButtonIcon:"wordWrapButtonIcon_Bwma",wordWrapButtonEnabled:"wordWrapButtonEnabled_EoeP"};function $(e){let{className:t,onClick:n,isEnabled:r}=e;const a=(0,M.I)({id:"theme.CodeBlock.wordWrapToggle",message:"Toggle word wrap",description:"The title attribute for toggle word wrapping button of code block lines"});return o.createElement("button",{type:"button",onClick:n,className:(0,i.Z)("clean-btn",t,r&&R.wordWrapButtonEnabled),"aria-label":a,title:a},o.createElement("svg",{className:R.wordWrapButtonIcon,viewBox:"0 0 24 24","aria-hidden":"true"},o.createElement("path",{fill:"currentColor",d:"M4 19h6v-2H4v2zM20 5H4v2h16V5zm-3 6H4v2h13.25c1.1 0 2 .9 2 2s-.9 2-2 2H15v-2l-3 3l3 3v-2h2c2.21 0 4-1.79 4-4s-1.79-4-4-4z"})))}function V(e){let{children:t,className:n="",metastring:r,title:c,showLineNumbers:l,language:s}=e;const{prism:{defaultLanguage:d,magicComments:p}}=(0,u.L)(),f=s??function(e){const t=e.split(" ").find((e=>e.startsWith("language-")));return t?.replace(/language-/,"")}(n)??d,g=m(),y=function(){const[e,t]=(0,o.useState)(!1),[n,r]=(0,o.useState)(!1),a=(0,o.useRef)(null),c=(0,o.useCallback)((()=>{const n=a.current.querySelector("code");e?n.removeAttribute("style"):(n.style.whiteSpace="pre-wrap",n.style.overflowWrap="anywhere"),t((e=>!e))}),[a,e]),l=(0,o.useCallback)((()=>{const{scrollWidth:e,clientWidth:t}=a.current,n=e>t||a.current.querySelector("code").hasAttribute("style");r(n)}),[a]);return B(a,l),(0,o.useEffect)((()=>{l()}),[e,l]),(0,o.useEffect)((()=>(window.addEventListener("resize",l,{passive:!0}),()=>{window.removeEventListener("resize",l)})),[l]),{codeBlockRef:a,isEnabled:e,isCodeScrollable:n,toggle:c}}(),v=function(e){return e?.match(h)?.groups.title??""}(r)||c,{lineClassNames:E,code:C}=b(t,{metastring:r,language:f,magicComments:p}),L=l??function(e){return Boolean(e?.includes("showLineNumbers"))}(r);return o.createElement(k,{as:"div",className:(0,i.Z)(n,f&&!n.includes(`language-${f}`)&&`language-${f}`)},v&&o.createElement("div",{className:N.codeBlockTitle},v),o.createElement("div",{className:N.codeBlockContent},o.createElement(I,(0,a.Z)({},T,{theme:g,code:C,language:f??"text"}),(e=>{let{className:t,tokens:n,getLineProps:r,getTokenProps:a}=e;return o.createElement("pre",{tabIndex:0,ref:y.codeBlockRef,className:(0,i.Z)(t,N.codeBlock,"thin-scrollbar")},o.createElement("code",{className:(0,i.Z)(N.codeBlockLines,L&&N.codeBlockLinesWithNumbering)},n.map(((e,t)=>o.createElement(z,{key:t,line:e,getLineProps:r,getTokenProps:a,classNames:E[t],showLineNumbers:L})))))})),o.createElement("div",{className:N.buttonGroup},(y.isEnabled||y.isCodeScrollable)&&o.createElement($,{className:N.codeButton,onClick:()=>y.toggle(),isEnabled:y.isEnabled}),o.createElement(W,{className:N.codeButton,code:C}))))}function F(e){let{children:t,...n}=e;const r=(0,l.Z)(),c=function(e){return o.Children.toArray(e).some((e=>(0,o.isValidElement)(e)))?e:Array.isArray(e)?e.join(""):e}(t),i="string"==typeof c?V:C;return o.createElement(i,(0,a.Z)({key:String(r)},n),c)}var q=n(9960);var U=n(6043);const G={details:"details_lb9f",isBrowser:"isBrowser_bmU9",collapsibleContent:"collapsibleContent_i85q"};function Y(e){return!!e&&("SUMMARY"===e.tagName||Y(e.parentElement))}function Q(e,t){return!!e&&(e===t||Q(e.parentElement,t))}function X(e){let{summary:t,children:n,...r}=e;const c=(0,l.Z)(),s=(0,o.useRef)(null),{collapsed:u,setCollapsed:m}=(0,U.u)({initialState:!r.open}),[d,p]=(0,o.useState)(r.open);return o.createElement("details",(0,a.Z)({},r,{ref:s,open:d,"data-collapsed":u,className:(0,i.Z)(G.details,c&&G.isBrowser,r.className),onMouseDown:e=>{Y(e.target)&&e.detail>1&&e.preventDefault()},onClick:e=>{e.stopPropagation();const t=e.target;Y(t)&&Q(t,s.current)&&(e.preventDefault(),u?(m(!1),p(!0)):m(!0))}}),t??o.createElement("summary",null,"Details"),o.createElement(U.z,{lazy:!1,collapsed:u,disableSSRStyle:!0,onCollapseTransitionEnd:e=>{m(e),p(!e)}},o.createElement("div",{className:G.collapsibleContent},n)))}const J={details:"details_b_Ee"},K="alert alert--info";function ee(e){let{...t}=e;return o.createElement(X,(0,a.Z)({},t,{className:(0,i.Z)(K,J.details,t.className)}))}var te=n(2503);function ne(e){return o.createElement(te.Z,e)}const oe={containsTaskList:"containsTaskList_mC6p"};const re={img:"img_ev3q"};const ae="admonition_LlT9",ce="admonitionHeading_tbUL",le="admonitionIcon_kALy",ie="admonitionContent_S0QG";const se={note:{infimaClassName:"secondary",iconComponent:function(){return o.createElement("svg",{viewBox:"0 0 14 16"},o.createElement("path",{fillRule:"evenodd",d:"M6.3 5.69a.942.942 0 0 1-.28-.7c0-.28.09-.52.28-.7.19-.18.42-.28.7-.28.28 0 .52.09.7.28.18.19.28.42.28.7 0 .28-.09.52-.28.7a1 1 0 0 1-.7.3c-.28 0-.52-.11-.7-.3zM8 7.99c-.02-.25-.11-.48-.31-.69-.2-.19-.42-.3-.69-.31H6c-.27.02-.48.13-.69.31-.2.2-.3.44-.31.69h1v3c.02.27.11.5.31.69.2.2.42.31.69.31h1c.27 0 .48-.11.69-.31.2-.19.3-.42.31-.69H8V7.98v.01zM7 2.3c-3.14 0-5.7 2.54-5.7 5.68 0 3.14 2.56 5.7 5.7 5.7s5.7-2.55 5.7-5.7c0-3.15-2.56-5.69-5.7-5.69v.01zM7 .98c3.86 0 7 3.14 7 7s-3.14 7-7 7-7-3.12-7-7 3.14-7 7-7z"}))},label:o.createElement(M.Z,{id:"theme.admonition.note",description:"The default label used for the Note admonition (:::note)"},"note")},tip:{infimaClassName:"success",iconComponent:function(){return o.createElement("svg",{viewBox:"0 0 12 16"},o.createElement("path",{fillRule:"evenodd",d:"M6.5 0C3.48 0 1 2.19 1 5c0 .92.55 2.25 1 3 1.34 2.25 1.78 2.78 2 4v1h5v-1c.22-1.22.66-1.75 2-4 .45-.75 1-2.08 1-3 0-2.81-2.48-5-5.5-5zm3.64 7.48c-.25.44-.47.8-.67 1.11-.86 1.41-1.25 2.06-1.45 3.23-.02.05-.02.11-.02.17H5c0-.06 0-.13-.02-.17-.2-1.17-.59-1.83-1.45-3.23-.2-.31-.42-.67-.67-1.11C2.44 6.78 2 5.65 2 5c0-2.2 2.02-4 4.5-4 1.22 0 2.36.42 3.22 1.19C10.55 2.94 11 3.94 11 5c0 .66-.44 1.78-.86 2.48zM4 14h5c-.23 1.14-1.3 2-2.5 2s-2.27-.86-2.5-2z"}))},label:o.createElement(M.Z,{id:"theme.admonition.tip",description:"The default label used for the Tip admonition (:::tip)"},"tip")},danger:{infimaClassName:"danger",iconComponent:function(){return o.createElement("svg",{viewBox:"0 0 12 16"},o.createElement("path",{fillRule:"evenodd",d:"M5.05.31c.81 2.17.41 3.38-.52 4.31C3.55 5.67 1.98 6.45.9 7.98c-1.45 2.05-1.7 6.53 3.53 7.7-2.2-1.16-2.67-4.52-.3-6.61-.61 2.03.53 3.33 1.94 2.86 1.39-.47 2.3.53 2.27 1.67-.02.78-.31 1.44-1.13 1.81 3.42-.59 4.78-3.42 4.78-5.56 0-2.84-2.53-3.22-1.25-5.61-1.52.13-2.03 1.13-1.89 2.75.09 1.08-1.02 1.8-1.86 1.33-.67-.41-.66-1.19-.06-1.78C8.18 5.31 8.68 2.45 5.05.32L5.03.3l.02.01z"}))},label:o.createElement(M.Z,{id:"theme.admonition.danger",description:"The default label used for the Danger admonition (:::danger)"},"danger")},info:{infimaClassName:"info",iconComponent:function(){return o.createElement("svg",{viewBox:"0 0 14 16"},o.createElement("path",{fillRule:"evenodd",d:"M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"}))},label:o.createElement(M.Z,{id:"theme.admonition.info",description:"The default label used for the Info admonition (:::info)"},"info")},caution:{infimaClassName:"warning",iconComponent:function(){return o.createElement("svg",{viewBox:"0 0 16 16"},o.createElement("path",{fillRule:"evenodd",d:"M8.893 1.5c-.183-.31-.52-.5-.887-.5s-.703.19-.886.5L.138 13.499a.98.98 0 0 0 0 1.001c.193.31.53.501.886.501h13.964c.367 0 .704-.19.877-.5a1.03 1.03 0 0 0 .01-1.002L8.893 1.5zm.133 11.497H6.987v-2.003h2.039v2.003zm0-3.004H6.987V5.987h2.039v4.006z"}))},label:o.createElement(M.Z,{id:"theme.admonition.caution",description:"The default label used for the Caution admonition (:::caution)"},"caution")}},ue={secondary:"note",important:"info",success:"tip",warning:"danger"};function me(e){const{mdxAdmonitionTitle:t,rest:n}=function(e){const t=o.Children.toArray(e),n=t.find((e=>o.isValidElement(e)&&"mdxAdmonitionTitle"===e.props?.mdxType)),r=o.createElement(o.Fragment,null,t.filter((e=>e!==n)));return{mdxAdmonitionTitle:n,rest:r}}(e.children);return{...e,title:e.title??t,children:n}}const de={head:function(e){const t=o.Children.map(e.children,(e=>o.isValidElement(e)?function(e){if(e.props?.mdxType&&e.props.originalType){const{mdxType:t,originalType:n,...r}=e.props;return o.createElement(e.props.originalType,r)}return e}(e):e));return o.createElement(c.Z,e,t)},code:function(e){const t=["a","b","big","i","span","em","strong","sup","sub","small"];return o.Children.toArray(e.children).every((e=>"string"==typeof e&&!e.includes("\n")||(0,o.isValidElement)(e)&&t.includes(e.props?.mdxType)))?o.createElement("code",e):o.createElement(F,e)},a:function(e){return o.createElement(q.Z,e)},pre:function(e){return o.createElement(F,(0,o.isValidElement)(e.children)&&"code"===e.children.props?.originalType?e.children.props:{...e})},details:function(e){const t=o.Children.toArray(e.children),n=t.find((e=>o.isValidElement(e)&&"summary"===e.props?.mdxType)),r=o.createElement(o.Fragment,null,t.filter((e=>e!==n)));return o.createElement(ee,(0,a.Z)({},e,{summary:n}),r)},ul:function(e){return o.createElement("ul",(0,a.Z)({},e,{className:(t=e.className,(0,i.Z)(t,t?.includes("contains-task-list")&&oe.containsTaskList))}));var t},img:function(e){return o.createElement("img",(0,a.Z)({loading:"lazy"},e,{className:(t=e.className,(0,i.Z)(t,re.img))}));var t},h1:e=>o.createElement(ne,(0,a.Z)({as:"h1"},e)),h2:e=>o.createElement(ne,(0,a.Z)({as:"h2"},e)),h3:e=>o.createElement(ne,(0,a.Z)({as:"h3"},e)),h4:e=>o.createElement(ne,(0,a.Z)({as:"h4"},e)),h5:e=>o.createElement(ne,(0,a.Z)({as:"h5"},e)),h6:e=>o.createElement(ne,(0,a.Z)({as:"h6"},e)),admonition:function(e){const{children:t,type:n,title:r,icon:a}=me(e),c=function(e){const t=ue[e]??e,n=se[t];return n||(console.warn(`No admonition config found for admonition type "${t}". Using Info as fallback.`),se.info)}(n),l=r??c.label,{iconComponent:s}=c,u=a??o.createElement(s,null);return o.createElement("div",{className:(0,i.Z)(d.k.common.admonition,d.k.common.admonitionType(e.type),"alert",`alert--${c.infimaClassName}`,ae)},o.createElement("div",{className:ce},o.createElement("span",{className:le},u),l),o.createElement("div",{className:ie},t))}};function pe(e){let{children:t}=e;return o.createElement(r.Zo,{components:de},t)}},9407:(e,t,n)=>{"use strict";n.d(t,{Z:()=>u});var o=n(7462),r=n(7294),a=n(6010),c=n(3743);const l={tableOfContents:"tableOfContents_bqdL",docItemContainer:"docItemContainer_F8PC"},i="table-of-contents__link toc-highlight",s="table-of-contents__link--active";function u(e){let{className:t,...n}=e;return r.createElement("div",{className:(0,a.Z)(l.tableOfContents,"thin-scrollbar",t)},r.createElement(c.Z,(0,o.Z)({},n,{linkClassName:i,linkActiveClassName:s})))}},3743:(e,t,n)=>{"use strict";n.d(t,{Z:()=>f});var o=n(7462),r=n(7294),a=n(6668);function c(e){const t=e.map((e=>({...e,parentIndex:-1,children:[]}))),n=Array(7).fill(-1);t.forEach(((e,t)=>{const o=n.slice(2,e.level);e.parentIndex=Math.max(...o),n[e.level]=t}));const o=[];return t.forEach((e=>{const{parentIndex:n,...r}=e;n>=0?t[n].children.push(r):o.push(r)})),o}function l(e){let{toc:t,minHeadingLevel:n,maxHeadingLevel:o}=e;return t.flatMap((e=>{const t=l({toc:e.children,minHeadingLevel:n,maxHeadingLevel:o});return function(e){return e.level>=n&&e.level<=o}(e)?[{...e,children:t}]:t}))}function i(e){const t=e.getBoundingClientRect();return t.top===t.bottom?i(e.parentNode):t}function s(e,t){let{anchorTopOffset:n}=t;const o=e.find((e=>i(e).top>=n));if(o){return function(e){return e.top>0&&e.bottom<window.innerHeight/2}(i(o))?o:e[e.indexOf(o)-1]??null}return e[e.length-1]??null}function u(){const e=(0,r.useRef)(0),{navbar:{hideOnScroll:t}}=(0,a.L)();return(0,r.useEffect)((()=>{e.current=t?0:document.querySelector(".navbar").clientHeight}),[t]),e}function m(e){const t=(0,r.useRef)(void 0),n=u();(0,r.useEffect)((()=>{if(!e)return()=>{};const{linkClassName:o,linkActiveClassName:r,minHeadingLevel:a,maxHeadingLevel:c}=e;function l(){const e=function(e){return Array.from(document.getElementsByClassName(e))}(o),l=function(e){let{minHeadingLevel:t,maxHeadingLevel:n}=e;const o=[];for(let r=t;r<=n;r+=1)o.push(`h${r}.anchor`);return Array.from(document.querySelectorAll(o.join()))}({minHeadingLevel:a,maxHeadingLevel:c}),i=s(l,{anchorTopOffset:n.current}),u=e.find((e=>i&&i.id===function(e){return decodeURIComponent(e.href.substring(e.href.indexOf("#")+1))}(e)));e.forEach((e=>{!function(e,n){n?(t.current&&t.current!==e&&t.current.classList.remove(r),e.classList.add(r),t.current=e):e.classList.remove(r)}(e,e===u)}))}return document.addEventListener("scroll",l),document.addEventListener("resize",l),l(),()=>{document.removeEventListener("scroll",l),document.removeEventListener("resize",l)}}),[e,n])}function d(e){let{toc:t,className:n,linkClassName:o,isChild:a}=e;return t.length?r.createElement("ul",{className:a?void 0:n},t.map((e=>r.createElement("li",{key:e.id},r.createElement("a",{href:`#${e.id}`,className:o??void 0,dangerouslySetInnerHTML:{__html:e.value}}),r.createElement(d,{isChild:!0,toc:e.children,className:n,linkClassName:o}))))):null}const p=r.memo(d);function f(e){let{toc:t,className:n="table-of-contents table-of-contents__left-border",linkClassName:i="table-of-contents__link",linkActiveClassName:s,minHeadingLevel:u,maxHeadingLevel:d,...f}=e;const h=(0,a.L)(),g=u??h.tableOfContents.minHeadingLevel,y=d??h.tableOfContents.maxHeadingLevel,v=function(e){let{toc:t,minHeadingLevel:n,maxHeadingLevel:o}=e;return(0,r.useMemo)((()=>l({toc:c(t),minHeadingLevel:n,maxHeadingLevel:o})),[t,n,o])}({toc:t,minHeadingLevel:g,maxHeadingLevel:y});return m((0,r.useMemo)((()=>{if(i&&s)return{linkClassName:i,linkActiveClassName:s,minHeadingLevel:g,maxHeadingLevel:y}}),[i,s,g,y])),r.createElement(p,(0,o.Z)({toc:v,className:n,linkClassName:i},f))}},7594:(e,t)=>{function n(e){let t,n=[];for(let o of e.split(",").map((e=>e.trim())))if(/^-?\d+$/.test(o))n.push(parseInt(o,10));else if(t=o.match(/^(-?\d+)(-|\.\.\.?|\u2025|\u2026|\u22EF)(-?\d+)$/)){let[e,o,r,a]=t;if(o&&a){o=parseInt(o),a=parseInt(a);const e=o<a?1:-1;"-"!==r&&".."!==r&&"\u2025"!==r||(a+=e);for(let t=o;t!==a;t+=e)n.push(t)}}return n}t.default=n,e.exports=n}}]); \ No newline at end of file diff --git a/17/assets/js/267b5415.e2568bf5.js b/17/assets/js/267b5415.e2568bf5.js new file mode 100644 index 0000000000..132021760d --- /dev/null +++ b/17/assets/js/267b5415.e2568bf5.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8411],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>k});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function l(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){r(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function o(e,t){if(null==e)return{};var n,a,r=function(e,t){if(null==e)return{};var n,a,r={},i=Object.keys(e);for(a=0;a<i.length;a++)n=i[a],t.indexOf(n)>=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a<i.length;a++)n=i[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var p=a.createContext({}),u=function(e){var t=a.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},c=function(e){var t=u(e.components);return a.createElement(p.Provider,{value:t},e.children)},d="mdxType",s={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,p=e.parentName,c=o(e,["components","mdxType","originalType","parentName"]),d=u(n),m=r,k=d["".concat(p,".").concat(m)]||d[m]||s[m]||i;return n?a.createElement(k,l(l({ref:t},c),{},{components:n})):a.createElement(k,l({ref:t},c))}));function k(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,l=new Array(i);l[0]=m;var o={};for(var p in t)hasOwnProperty.call(t,p)&&(o[p]=t[p]);o.originalType=e,o[d]="string"==typeof e?e:r,l[1]=o;for(var u=2;u<i;u++)l[u]=n[u];return a.createElement.apply(null,l)}return a.createElement.apply(null,n)}m.displayName="MDXCreateElement"},7046:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>l,default:()=>s,frontMatter:()=>i,metadata:()=>o,toc:()=>u});var a=n(7462),r=(n(7294),n(3905));const i={},l="Similarities Between the TCBs of `libult` and Unikraft",o={unversionedId:"Lab/Compute/quiz/tcb-libult-unikraft",id:"Lab/Compute/quiz/tcb-libult-unikraft",title:"Similarities Between the TCBs of `libult` and Unikraft",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/tcb-libult-unikraft.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/tcb-libult-unikraft",permalink:"/operating-systems/17/Lab/Compute/quiz/tcb-libult-unikraft",draft:!1,tags:[],version:"current",frontMatter:{}},p={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:u},d="wrapper";function s(e){let{components:t,...n}=e;return(0,r.kt)(d,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"similarities-between-the-tcbs-of-libult-and-unikraft"},"Similarities Between the TCBs of ",(0,r.kt)("inlineCode",{parentName:"h1"},"libult")," and Unikraft"),(0,r.kt)("h2",{id:"question-text"},"Question Text"),(0,r.kt)("p",null,"Which members of the TCBs in ",(0,r.kt)("inlineCode",{parentName:"p"},"libult")," and Unikraft have similar meanings?"),(0,r.kt)("h2",{id:"question-answers"},"Question Answers"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"start_routine")," and ",(0,r.kt)("inlineCode",{parentName:"li"},"entry"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"id")," and ",(0,r.kt)("inlineCode",{parentName:"li"},"name"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"context.uc_stack")," and ",(0,r.kt)("inlineCode",{parentName:"li"},"stack"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"arguments")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"flags"))),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"has_dynamic_stack")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"detached")))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"argument")," and ",(0,r.kt)("inlineCode",{parentName:"li"},"arg"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"context")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"sched"))),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"context")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"tls")))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"context")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"ctx"))),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("inlineCode",{parentName:"p"},"return_value")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"prv")))),(0,r.kt)("h2",{id:"feedback"},"Feedback"),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"start_routine")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"entry")," are the functions that run in the newly created threads.\n",(0,r.kt)("inlineCode",{parentName:"p"},"context.uc_stack")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"stack")," are pointers to the stack of the newly created threads.\n",(0,r.kt)("inlineCode",{parentName:"p"},"argument")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"arg")," are pointers to the arguments of ",(0,r.kt)("inlineCode",{parentName:"p"},"start_routine")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"entry"),", respectively.\n",(0,r.kt)("inlineCode",{parentName:"p"},"context")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"ctx")," are the contexts in which the new threads run.\n",(0,r.kt)("inlineCode",{parentName:"p"},"return_value")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"prv")," are both pointers to the values returned by the thread functions."))}s.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/2828cd91.9ddde4d3.js b/17/assets/js/2828cd91.9ddde4d3.js new file mode 100644 index 0000000000..4ed4ffcce7 --- /dev/null +++ b/17/assets/js/2828cd91.9ddde4d3.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2845],{3905:(e,t,r)=>{r.d(t,{Zo:()=>c,kt:()=>m});var n=r(7294);function i(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?o(Object(r),!0).forEach((function(t){i(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):o(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function l(e,t){if(null==e)return{};var r,n,i=function(e,t){if(null==e)return{};var r,n,i={},o=Object.keys(e);for(n=0;n<o.length;n++)r=o[n],t.indexOf(r)>=0||(i[r]=e[r]);return i}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n<o.length;n++)r=o[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(i[r]=e[r])}return i}var s=n.createContext({}),p=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},c=function(e){var t=p(e.components);return n.createElement(s.Provider,{value:t},e.children)},u="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,i=e.mdxType,o=e.originalType,s=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),u=p(r),f=i,m=u["".concat(s,".").concat(f)]||u[f]||d[f]||o;return r?n.createElement(m,a(a({ref:t},c),{},{components:r})):n.createElement(m,a({ref:t},c))}));function m(e,t){var r=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var o=r.length,a=new Array(o);a[0]=f;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[u]="string"==typeof e?e:i,a[1]=l;for(var p=2;p<o;p++)a[p]=r[p];return n.createElement.apply(null,a)}return n.createElement.apply(null,r)}f.displayName="MDXCreateElement"},3894:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>a,default:()=>d,frontMatter:()=>o,metadata:()=>l,toc:()=>p});var n=r(7462),i=(r(7294),r(3905));const o={},a="Prints Working after Closing `stdio`",l={unversionedId:"Lab/IO/quiz/prints-work-no-stdio",id:"Lab/IO/quiz/prints-work-no-stdio",title:"Prints Working after Closing `stdio`",description:"Question Text",source:"@site/docs/Lab/IO/quiz/prints-work-no-stdio.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/prints-work-no-stdio",permalink:"/operating-systems/17/Lab/IO/quiz/prints-work-no-stdio",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},u="wrapper";function d(e){let{components:t,...r}=e;return(0,i.kt)(u,(0,n.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"prints-working-after-closing-stdio"},"Prints Working after Closing ",(0,i.kt)("inlineCode",{parentName:"h1"},"stdio")),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Why does ",(0,i.kt)("inlineCode",{parentName:"p"},"support/redirect/redirect.c"),", still print messages to the console after closing file descriptor 1 (",(0,i.kt)("inlineCode",{parentName:"p"},"stdout"),")?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"because ",(0,i.kt)("inlineCode",{parentName:"li"},"wait_for_input()")," calls ",(0,i.kt)("inlineCode",{parentName:"li"},"fprintf(stderr, ...)"),", which prints to ",(0,i.kt)("inlineCode",{parentName:"li"},"stderr")," (file descriptor 2)")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},'because the default file descriptors cannot be "truly" closed')),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"because the other two default file descriptors are still linked to the console")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"because the ",(0,i.kt)("inlineCode",{parentName:"p"},"wait_for_input()")," function started printed before closing the ",(0,i.kt)("inlineCode",{parentName:"p"},"stdout")," file descriptor"))),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,"If you look at ",(0,i.kt)("inlineCode",{parentName:"p"},"wait_for_input()")," closely, you'll notice it calls ",(0,i.kt)("inlineCode",{parentName:"p"},"fprintf(stderr, ...)"),".\n",(0,i.kt)("inlineCode",{parentName:"p"},"stderr")," is liked to file descriptor 2, which is left unchanged so we can still write data to it."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/2bde47d5.136c0499.js b/17/assets/js/2bde47d5.136c0499.js new file mode 100644 index 0000000000..6ebdfa8dad --- /dev/null +++ b/17/assets/js/2bde47d5.136c0499.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4006],{3905:(e,t,n)=>{n.d(t,{Zo:()=>s,kt:()=>f});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function l(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){r(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function i(e,t){if(null==e)return{};var n,a,r=function(e,t){if(null==e)return{};var n,a,r={},o=Object.keys(e);for(a=0;a<o.length;a++)n=o[a],t.indexOf(n)>=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a<o.length;a++)n=o[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var c=a.createContext({}),p=function(e){var t=a.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},s=function(e){var t=p(e.components);return a.createElement(c.Provider,{value:t},e.children)},u="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,c=e.parentName,s=i(e,["components","mdxType","originalType","parentName"]),u=p(n),m=r,f=u["".concat(c,".").concat(m)]||u[m]||d[m]||o;return n?a.createElement(f,l(l({ref:t},s),{},{components:n})):a.createElement(f,l({ref:t},s))}));function f(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,l=new Array(o);l[0]=m;var i={};for(var c in t)hasOwnProperty.call(t,c)&&(i[c]=t[c]);i.originalType=e,i[u]="string"==typeof e?e:r,l[1]=i;for(var p=2;p<o;p++)l[p]=n[p];return a.createElement.apply(null,l)}return a.createElement.apply(null,n)}m.displayName="MDXCreateElement"},3995:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>l,default:()=>d,frontMatter:()=>o,metadata:()=>i,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const o={},l="Page Allocation",i={unversionedId:"Lab/Data/quiz/page-allocation",id:"Lab/Data/quiz/page-allocation",title:"Page Allocation",description:"Question Text",source:"@site/docs/Lab/Data/quiz/page-allocation.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/page-allocation",permalink:"/operating-systems/17/Lab/Data/quiz/page-allocation",draft:!1,tags:[],version:"current",frontMatter:{}},c={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],s={toc:p},u="wrapper";function d(e){let{components:t,...n}=e;return(0,r.kt)(u,(0,a.Z)({},s,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"page-allocation"},"Page Allocation"),(0,r.kt)("h2",{id:"question-text"},"Question Text"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/static-dynamic$ size hello-static\ntext data bss dec hex filename\n893333 20996 7128 921457 e0f71 hello-static\n")),(0,r.kt)("p",null,"How many bytes should we add to the ",(0,r.kt)("inlineCode",{parentName:"p"},".data")," section to make its size ",(0,r.kt)("inlineCode",{parentName:"p"},"28 KB"),", instead of ",(0,r.kt)("inlineCode",{parentName:"p"},"24 KB"),"?"),(0,r.kt)("h2",{id:"question-answers"},"Question Answers"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"1 KB")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"4 KB")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"3580 bytes"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"3581 bytes")),(0,r.kt)("h2",{id:"feedback"},"Feedback"),(0,r.kt)("p",null,"The total size must be ",(0,r.kt)("inlineCode",{parentName:"p"},"1")," byte over the ",(0,r.kt)("inlineCode",{parentName:"p"},"24 KB")," threshold to cause a new page allocation.\nSo in order to get that past the current size of ",(0,r.kt)("inlineCode",{parentName:"p"},"20996"),", we need ",(0,r.kt)("inlineCode",{parentName:"p"},"3581")," bytes:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~$ echo "24 * 1024 + 1 - 20996" | bc\n3581\n')))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/2c270c24.c3b8bb98.js b/17/assets/js/2c270c24.c3b8bb98.js new file mode 100644 index 0000000000..896d4a00cc --- /dev/null +++ b/17/assets/js/2c270c24.c3b8bb98.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3540],{3905:(e,t,a)=>{a.d(t,{Zo:()=>h,kt:()=>u});var n=a(7294);function r(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function o(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,n)}return a}function i(e){for(var t=1;t<arguments.length;t++){var a=null!=arguments[t]?arguments[t]:{};t%2?o(Object(a),!0).forEach((function(t){r(e,t,a[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(a)):o(Object(a)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(a,t))}))}return e}function s(e,t){if(null==e)return{};var a,n,r=function(e,t){if(null==e)return{};var a,n,r={},o=Object.keys(e);for(n=0;n<o.length;n++)a=o[n],t.indexOf(a)>=0||(r[a]=e[a]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n<o.length;n++)a=o[n],t.indexOf(a)>=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(r[a]=e[a])}return r}var l=n.createContext({}),p=function(e){var t=n.useContext(l),a=t;return e&&(a="function"==typeof e?e(t):i(i({},t),e)),a},h=function(e){var t=p(e.components);return n.createElement(l.Provider,{value:t},e.children)},m="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},c=n.forwardRef((function(e,t){var a=e.components,r=e.mdxType,o=e.originalType,l=e.parentName,h=s(e,["components","mdxType","originalType","parentName"]),m=p(a),c=r,u=m["".concat(l,".").concat(c)]||m[c]||d[c]||o;return a?n.createElement(u,i(i({ref:t},h),{},{components:a})):n.createElement(u,i({ref:t},h))}));function u(e,t){var a=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=a.length,i=new Array(o);i[0]=c;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[m]="string"==typeof e?e:r,i[1]=s;for(var p=2;p<o;p++)i[p]=a[p];return n.createElement.apply(null,i)}return n.createElement.apply(null,a)}c.displayName="MDXCreateElement"},4591:(e,t,a)=>{a.r(t),a.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>d,frontMatter:()=>o,metadata:()=>s,toc:()=>p});var n=a(7462),r=(a(7294),a(3905));const o={},i="Arena",s={unversionedId:"Lab/Compute/arena",id:"Lab/Compute/arena",title:"Arena",description:"Threads and Processes: clone",source:"@site/docs/Lab/Compute/arena.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/arena",permalink:"/operating-systems/17/Lab/Compute/arena",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"User-Level Threads",permalink:"/operating-systems/17/Lab/Compute/user-level-threads"},next:{title:"IO",permalink:"/operating-systems/17/Lab/IO/"}},l={},p=[{value:"Threads and Processes: <code>clone</code>",id:"threads-and-processes-clone",level:2},{value:"Libraries for Parallel Processing",id:"libraries-for-parallel-processing",level:2},{value:"<code>std.parallelism</code> in D",id:"stdparallelism-in-d",level:3},{value:"OpenMP for C",id:"openmp-for-c",level:3},{value:"Shared Memory",id:"shared-memory",level:2},{value:"Mini-shell",id:"mini-shell",level:2},{value:"First Step: <code>system</code> Dissected",id:"first-step-system-dissected",level:3},{value:"Command Executor in Another language",id:"command-executor-in-another-language",level:3},{value:"The GIL",id:"the-gil",level:2},{value:"Practice: Array Sum in Python",id:"practice-array-sum-in-python",level:3},{value:"But Why?",id:"but-why",level:3},{value:"Atomic Assembly",id:"atomic-assembly",level:2},{value:"Synchronization - Thread-Safe Data Structure",id:"synchronization---thread-safe-data-structure",level:2},{value:"Minor and Major Page Faults",id:"minor-and-major-page-faults",level:2},{value:"Minor Page Faults",id:"minor-page-faults",level:3},{value:"Major Page Faults",id:"major-page-faults",level:3}],h={toc:p},m="wrapper";function d(e){let{components:t,...a}=e;return(0,r.kt)(m,(0,n.Z)({},h,a,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"arena"},"Arena"),(0,r.kt)("h2",{id:"threads-and-processes-clone"},"Threads and Processes: ",(0,r.kt)("inlineCode",{parentName:"h2"},"clone")),(0,r.kt)("p",null,"Let's go back to our initial demos that used threads and processes.\nWe'll see that in order to create both threads and processes, the underlying Linux syscall is ",(0,r.kt)("inlineCode",{parentName:"p"},"clone"),".\nFor this, we'll run both ",(0,r.kt)("inlineCode",{parentName:"p"},"sum_array_threads")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"sum_array_processes")," under ",(0,r.kt)("inlineCode",{parentName:"p"},"strace"),".\nAs we've already established, we're only interested in the ",(0,r.kt)("inlineCode",{parentName:"p"},"clone")," syscall:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/sum-array/c$ strace -e clone ./sum_array_threads 2\nclone(child_stack=0x7f60b56482b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819693], tls=0x7f60b5649640, child_tidptr=0x7f60b5649910) = 1819693\nclone(child_stack=0x7f60b4e472b0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tid=[1819694], tls=0x7f60b4e48640, child_tidptr=0x7f60b4e48910) = 1819694\n\nstudent@os:~/.../lab/support/sum-array/c$ strace -e clone ./sum_array_processes 2\nclone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820599\nclone(child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f7a4e346650) = 1820600\n")),(0,r.kt)("p",null,"We ran each program with an argument of 2, so we have 2 calls to ",(0,r.kt)("inlineCode",{parentName:"p"},"clone"),".\nNotice that in the case of threads, the ",(0,r.kt)("inlineCode",{parentName:"p"},"clone")," syscall receives more arguments.\nThe relevant flags passed as arguments when creating threads are documented in ",(0,r.kt)("a",{parentName:"p",href:"https://man.archlinux.org/man/clone3.2.en"},(0,r.kt)("inlineCode",{parentName:"a"},"clone"),"'s man page"),":"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"CLONE_VM"),": the child and the parent process share the same VAS"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"CLONE_{FS,FILES,SIGHAND}"),": the new thread shares the filesystem information, file and signal handlers with the one that created it\nThe syscall also receives valid pointers to the new thread's stack and TLS, i.e. the only parts of the VAS that are distinct between threads (although they are technically accessible from all threads).")),(0,r.kt)("p",null,"By contrast, when creating a new process, the arguments of the ",(0,r.kt)("inlineCode",{parentName:"p"},"clone")," syscall are simpler (i.e. fewer flags are present).\nRemember that in both cases ",(0,r.kt)("inlineCode",{parentName:"p"},"clone")," creates a new ",(0,r.kt)("strong",{parentName:"p"},"thread"),".\nWhen creating a process, ",(0,r.kt)("inlineCode",{parentName:"p"},"clone")," creates this new thread within a new separate address space."),(0,r.kt)("h2",{id:"libraries-for-parallel-processing"},"Libraries for Parallel Processing"),(0,r.kt)("p",null,"In ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sum-array/c/sum_array_threads.c"),' we spawned threads "manually" by using the ',(0,r.kt)("inlineCode",{parentName:"p"},"pthread_create()")," function.\nThis is ",(0,r.kt)("strong",{parentName:"p"},"not")," a syscall, but a wrapper over the common syscall used by both ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," (which is also not a syscall) and ",(0,r.kt)("inlineCode",{parentName:"p"},"pthread_create()"),"."),(0,r.kt)("p",null,"Still, ",(0,r.kt)("inlineCode",{parentName:"p"},"pthread_create()")," is not yet a syscall.\nIn order to see what syscall ",(0,r.kt)("inlineCode",{parentName:"p"},"pthread_create()")," uses, check out ",(0,r.kt)("a",{parentName:"p",href:"#threads-and-processes-clone"},"this section at the end of the lab"),"."),(0,r.kt)("p",null,"Most programming languages provide a more advanced API for handling parallel computation."),(0,r.kt)("h3",{id:"stdparallelism-in-d"},(0,r.kt)("inlineCode",{parentName:"h3"},"std.parallelism")," in D"),(0,r.kt)("p",null,"D language's standard library exposes the ",(0,r.kt)("a",{parentName:"p",href:"https://dlang.org/phobos/std_parallelism.html"},(0,r.kt)("inlineCode",{parentName:"a"},"std.parallelism")),", which provides a series of parallel processing functions.\nOne such function is ",(0,r.kt)("inlineCode",{parentName:"p"},"reduce()"),", which splits an array between a given number of threads and applies a given operation to these chunks.\nIn our case, the operation simply adds the elements to an accumulator: ",(0,r.kt)("inlineCode",{parentName:"p"},"a + b"),".\nFollow and run the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sum-array/d/sum_array_threads_reduce.d"),"."),(0,r.kt)("p",null,"The number of threads is used within a ",(0,r.kt)("a",{parentName:"p",href:"https://dlang.org/phobos/std_parallelism.html#.TaskPool"},(0,r.kt)("inlineCode",{parentName:"a"},"TaskPool")),".\nThis structure is a thread manager (not scheduler).\nIt silently creates the number of threads we request and then ",(0,r.kt)("inlineCode",{parentName:"p"},"reduce()")," spreads its workload between these threads."),(0,r.kt)("h3",{id:"openmp-for-c"},"OpenMP for C"),(0,r.kt)("p",null,"Unlike D, C does not support parallel computation by design.\nIt needs a library to do advanced things, like ",(0,r.kt)("inlineCode",{parentName:"p"},"reduce()")," from D.\nWe have chosen to use the OpenMP library for this.\nFollow the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sum-array/c/sum_array_threads_openmp.c"),"."),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"#pragma")," used in the code instructs the compiler to enable the ",(0,r.kt)("inlineCode",{parentName:"p"},"omp")," module, and to parallelise the code.\nIn this case, we instruct the compiler to perform a reduce of the array, using the ",(0,r.kt)("inlineCode",{parentName:"p"},"+")," operator, and to store the results in the ",(0,r.kt)("inlineCode",{parentName:"p"},"result")," variable.\nThis reduction uses threads to calculate the sum, similar to ",(0,r.kt)("inlineCode",{parentName:"p"},"summ_array_threads.c"),", but in a much more optimised form."),(0,r.kt)("p",null,"Now compile and run the ",(0,r.kt)("inlineCode",{parentName:"p"},"sum_array_threads_openmp")," binary using 1, 2, 4, and 8 threads as before.\nYou'll see lower running times than ",(0,r.kt)("inlineCode",{parentName:"p"},"sum_array_threads")," due to the highly-optimised code emitted by the compiler.\nFor this reason and because library functions are usually much better tested than your own code, it is always preferred to use a library function for a given task."),(0,r.kt)("h2",{id:"shared-memory"},"Shared Memory"),(0,r.kt)("p",null,"As you remember from the ",(0,r.kt)("a",{parentName:"p",href:"../../../data/"},"Data chapter"),", one way to allocate a given number of pages is to use the ",(0,r.kt)("inlineCode",{parentName:"p"},"mmap()")," syscall.\nLet's look at its ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/mmap.2.html"},"man page"),", specifically at the ",(0,r.kt)("inlineCode",{parentName:"p"},"flags")," argument.\nIts main purpose is to determine the way in which child processes interact with the mapped pages."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/mmap-cow-flag"},"Quiz")),(0,r.kt)("p",null,"Now let's test this flag, as well as its opposite: ",(0,r.kt)("inlineCode",{parentName:"p"},"MAP_SHARED"),".\nCompile and run the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/shared-memory/shared_memory.c"),"."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"See the value read by the parent is different from that written by the child.\nModify the ",(0,r.kt)("inlineCode",{parentName:"p"},"flags")," parameter of ",(0,r.kt)("inlineCode",{parentName:"p"},"mmap()")," so they are the same.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Create a semaphore in the shared page and use it to make the parent signal the child before it can exit.\nUse the API defined in ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man0/semaphore.h.0p.html"},(0,r.kt)("inlineCode",{parentName:"a"},"semaphore.h")),".\n",(0,r.kt)("strong",{parentName:"p"},"Be careful!"),"\nThe value written and read previously by the child and the parent, respectively, must not change."))),(0,r.kt)("p",null,'One way of creating a shared semaphore is to place it within a shared memory area, as we\'ve just done.\nThis only works between "related" processes.\nIf you want to share a semaphore or other types of memory between any two processes, you need filesystem support.\nFor this, you should use ',(0,r.kt)("strong",{parentName:"p"},"named semaphores"),", created using ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/sem_open.3.html"},(0,r.kt)("inlineCode",{parentName:"a"},"sem_open()")),".\nYou'll get more accustomed to such functions in the ",(0,r.kt)("a",{parentName:"p",href:"../../../app-interact/"},"Application Interaction chapter"),"."),(0,r.kt)("h2",{id:"mini-shell"},"Mini-shell"),(0,r.kt)("h3",{id:"first-step-system-dissected"},"First Step: ",(0,r.kt)("inlineCode",{parentName:"h3"},"system")," Dissected"),(0,r.kt)("p",null,"You already know that ",(0,r.kt)("inlineCode",{parentName:"p"},"system")," calls ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"execve()")," to create the new process.\nLet's see how and why.\nFirst, we run the following command to trace the ",(0,r.kt)("inlineCode",{parentName:"p"},"execve()")," syscalls used by ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy_creator"),".\nWe'll leave ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," for later."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/sleepy$ strace -e execve -ff -o syscalls ./sleepy_creator\n")),(0,r.kt)("p",null,"At this point, you will get two files whose names start with ",(0,r.kt)("inlineCode",{parentName:"p"},"syscalls"),", followed by some numbers.\nThose numbers are the PIDs of the parent and the child process.\nTherefore, the file with the higher number contains logs of the ",(0,r.kt)("inlineCode",{parentName:"p"},"execve")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"clone")," syscalls issued by the parent process, while\nthe other logs those two syscalls when made by the child process.\nLet's take a look at them.\nThe numbers below will differ from those on your system:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../support/sleepy:$ cat syscalls.2523393 # syscalls from parent process\nexecve("sleepy_creator", ["sleepy_creator"], 0x7ffd2c157758 /* 39 vars */) = 0\n--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=2523394, si_uid=1052093, si_status=0, si_utime=0, si_stime=0} ---\n+++ exited with 0 +++\n\nstudent@os:~/.../support/sleepy:$ cat syscalls.2523394 # syscalls from child process\nexecve("/bin/sh", ["sh", "-c", "sleep 10"], 0x7ffd36253be8 /* 39 vars */) = 0\nexecve("/usr/bin/sleep", ["sleep", "10"], 0x560f41659d40 /* 38 vars */) = 0\n+++ exited with 0 +++\n')),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/who-calls-execve-parent"},"Quiz")),(0,r.kt)("p",null,"Now notice that the child process doesn't simply call ",(0,r.kt)("inlineCode",{parentName:"p"},'execve("/usr/bin/sleep" ...)'),".\nIt first changes its virtual address space (VAS) to that of a ",(0,r.kt)("inlineCode",{parentName:"p"},"bash")," process (",(0,r.kt)("inlineCode",{parentName:"p"},'execve("/bin/sh" ...)'),") and then that ",(0,r.kt)("inlineCode",{parentName:"p"},"bash")," process switches its VAS to ",(0,r.kt)("inlineCode",{parentName:"p"},"sleep"),".\nTherefore, calling ",(0,r.kt)("inlineCode",{parentName:"p"},"system(<some_command>)")," is equivalent to running ",(0,r.kt)("inlineCode",{parentName:"p"},"<some_command>")," in the command-line."),(0,r.kt)("p",null,"With this knowledge in mind, let's implement our own mini-shell.\nStart from the skeleton code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/mini-shell/mini_shell.c"),".\nWe're already running our Bash interpreter from the command-line, so there's no need to ",(0,r.kt)("inlineCode",{parentName:"p"},"exec")," another Bash from it.\nSimply ",(0,r.kt)("inlineCode",{parentName:"p"},"exec")," the command."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/mini-shell-stops-after-command"},"Quiz")),(0,r.kt)("p",null,'So we need a way to "save" the ',(0,r.kt)("inlineCode",{parentName:"p"},"mini_shell")," process before ",(0,r.kt)("inlineCode",{parentName:"p"},"exec()"),"-ing our command.\nFind a way to do this."),(0,r.kt)("blockquote",null,(0,r.kt)("p",{parentName:"blockquote"},(0,r.kt)("strong",{parentName:"p"},"Hint"),": You can see what ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy")," does and draw inspiration from there.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to also list the calls to ",(0,r.kt)("inlineCode",{parentName:"p"},"clone()")," performed by ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy")," or its children.\n",(0,r.kt)("a",{parentName:"p",href:"#threads-and-processes-clone"},"Remember")," what ",(0,r.kt)("inlineCode",{parentName:"p"},"clone()")," is used for and use its parameters to deduce which of the two scenarios happens to ",(0,r.kt)("inlineCode",{parentName:"p"},"sleepy"),".")),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Moral of the story"),": We can add another step to the moral of ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/processes#practice-fork"},"our previous story"),".\nWhen spawning a new command, the call order is:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"parent: ",(0,r.kt)("inlineCode",{parentName:"li"},"fork()"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"exec()"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"wait()")),(0,r.kt)("li",{parentName:"ul"},"child: ",(0,r.kt)("inlineCode",{parentName:"li"},"exit()"))),(0,r.kt)("h3",{id:"command-executor-in-another-language"},"Command Executor in Another language"),(0,r.kt)("p",null,"Now implement the same functionality (a Bash command executor) in any other language, other than C/C++.\nUse whatever API is provided by your language of choice for creating and waiting for processes."),(0,r.kt)("h2",{id:"the-gil"},"The GIL"),(0,r.kt)("p",null,"Throughout this lab, you might have noticed that there were no thread exercises ",(0,r.kt)("em",{parentName:"p"},"in Python"),".\nIf you did, you probably wondered why.\nIt's not because Python does not support threads, because it does, but because of a mechanism called the ",(0,r.kt)("strong",{parentName:"p"},"Global Interpreter Lock"),", or GIL.\nAs its name suggests, this is a lock implemented inside most commonly used Python interpreter (CPython), which only allows ",(0,r.kt)("strong",{parentName:"p"},"one")," thread to run at a given time.\nAs a consequence, multithreaded programs written in Python run ",(0,r.kt)("strong",{parentName:"p"},"concurrently"),", not in parallel.\nFor this reason, you will see no speedup even when you run an embarrassingly parallel code in parallel."),(0,r.kt)("p",null,"However, keep in mind that this drawback does not make threads useless in Python.\nThey are still useful and widely used when a process needs to perform many IO-bound tasks (i.e.: tasks that involve many file reads / writes or network requests).\nSuch tasks run many blocking syscalls that require the thread to switch from the RUNNING state to WAITING.\nDoing so voluntarily makes threads viable because they rarely run for their entire time slice and spend most of the time waiting for data.\nSo it doesn't hurt them to run concurrently, instead of in parallel."),(0,r.kt)("p",null,"Do not make the confusion to believe threads in Python are ",(0,r.kt)("a",{parentName:"p",href:"./scheduling.md#user-level-vs-kernel-level-threads"},"user-level threads"),".\n",(0,r.kt)("a",{parentName:"p",href:"https://docs.python.org/3/library/threading.html#threading.Thread"},(0,r.kt)("inlineCode",{parentName:"a"},"threading.Thread")),"s are kernel-level threads.\nIt's just that they are forced to run concurrently by the GIL."),(0,r.kt)("h3",{id:"practice-array-sum-in-python"},"Practice: Array Sum in Python"),(0,r.kt)("p",null,"Let's first probe this by implementing two parallel versions of the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/sum-array/python/sum_array_sequential.py"),".\nOne version should use threads and the other should use processes.\nRun each of them using 1, 2, 4, and 8 threads / processes respectively and compare the running times.\nNotice that the running times of the multithreaded implementation do not decrease.\nThis is because the GIL makes it so that those threads that you create essentially run sequentially."),(0,r.kt)("p",null,"The GIL also makes it so that individual Python instructions are atomic.\nRun the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/race-condition/python/race_condition.py"),".\nEvery time, ",(0,r.kt)("inlineCode",{parentName:"p"},"var")," will be 0 because the GIL doesn't allow the two threads to run in parallel and reach the critical section at the same time.\nThis means that the instructions ",(0,r.kt)("inlineCode",{parentName:"p"},"var += 1")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"var -= 1")," become atomic."),(0,r.kt)("h3",{id:"but-why"},"But Why?"),(0,r.kt)("p",null,"Unlike Bigfoot, or the Loch Ness monster, we have proof that the GIL is real.\nAt first glance, this seems like a huge disadvantage.\nWhy force threads to run sequentially?\nThe answer has to do with memory management.\nIn the ",(0,r.kt)("a",{parentName:"p",href:"../../../data"},"Data chapter"),", you learned that one way of managing memory is via ",(0,r.kt)("em",{parentName:"p"},"garbage collection")," (GC).\nIn Python, the GC uses reference counting, i.e. each object also stores the number of live pointers to it (variables that reference it).\nYou can see that this number needs to be modified atomically by the interpreter to avoid race conditions.\nThis involves adding locks to ",(0,r.kt)("strong",{parentName:"p"},"all")," Python data structures.\nThis large number of locks doesn't scale for a language as large and open as Python.\nIn addition, it also introduces the risk of ",(0,r.kt)("em",{parentName:"p"},"deadlocks"),".\nYou can read more on this topic ",(0,r.kt)("a",{parentName:"p",href:"https://realpython.com/python-gil/"},"in this article")," and if you think eliminating the GIL looks like an easy task, which should have been done long ago, check the requirements ",(0,r.kt)("a",{parentName:"p",href:"https://wiki.python.org/moin/GlobalInterpreterLock"},"here"),".\nThey're not trivial to meet."),(0,r.kt)("p",null,"Single-threaded-ness is a common trope for interpreted languages to use some sort of GIL.\n",(0,r.kt)("a",{parentName:"p",href:"https://git.ruby-lang.org/ruby.git"},"Ruby MRI, the reference Ruby interpreter"),", uses a similar mechanism, called the ",(0,r.kt)("a",{parentName:"p",href:"https://ivoanjo.me/blog/2022/07/17/tracing-ruby-global-vm-lock/"},"Global VM Lock"),".\nJavaScript is even more straightforward: it is single-threaded by design, also for GC-related reasons.\nIt does, however, support asynchronous actions, but these are executed on the same thread as every other code.\nThis is implemented by placing each instruction on a ",(0,r.kt)("a",{parentName:"p",href:"https://medium.com/swlh/what-does-it-mean-by-javascript-is-single-threaded-language-f4130645d8a9"},"call stack"),"."),(0,r.kt)("h2",{id:"atomic-assembly"},"Atomic Assembly"),(0,r.kt)("p",null,"No, this section is not about nukes, sadly :(.\nInstead, we aim to get accustomed to the way in which the x86 ISA provides atomic instructions."),(0,r.kt)("p",null,"This mechanism looks very simple.\nIt is but ",(0,r.kt)("strong",{parentName:"p"},"one instruction prefix"),": ",(0,r.kt)("inlineCode",{parentName:"p"},"lock"),".\nIt is not an instruction with its own separate opcode, but a prefix that slightly modifies the opcode of the following instructions to make the CPU execute it atomically (i.e. with exclusive access to the data bus)."),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"lock")," must only be place before an instruction that executes a ",(0,r.kt)("em",{parentName:"p"},"read-modify-write")," action.\nFor example, we cannot place it before a ",(0,r.kt)("inlineCode",{parentName:"p"},"mov")," instruction, as the action of a ",(0,r.kt)("inlineCode",{parentName:"p"},"mov")," is simply ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"write"),".\nInstead, we can place it in front of an ",(0,r.kt)("inlineCode",{parentName:"p"},"inc")," instruction if its operand is memory."),(0,r.kt)("p",null,"Look at the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/race-condition/asm/race_condition_lock.S"),".\nIt's an Assembly equivalent of the code you've already seen many times so far (such as ",(0,r.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition.c"),").\nAssemble and run it a few times.\nNotice the different results you get."),(0,r.kt)("p",null,"Now add the ",(0,r.kt)("inlineCode",{parentName:"p"},"lock")," prefix before ",(0,r.kt)("inlineCode",{parentName:"p"},"inc")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"dec"),".\nReassemble and rerun the code.\nAnd now we have synchronized the two threads by leveraging CPU support."),(0,r.kt)("h2",{id:"synchronization---thread-safe-data-structure"},"Synchronization - Thread-Safe Data Structure"),(0,r.kt)("p",null,"Now it's time for a fully practical exercise.\nGo to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/CLIST/"),".\nIn the file ",(0,r.kt)("inlineCode",{parentName:"p"},"clist.c")," you'll find a simple implementation of an array list.\nAlthough correct, it is not (yet) thread-safe."),(0,r.kt)("p",null,"The code in ",(0,r.kt)("inlineCode",{parentName:"p"},"test.c")," verifies its single-threaded correctness, while the one in ",(0,r.kt)("inlineCode",{parentName:"p"},"test_parallel.c")," verifies it works properly with multiple threads.\nYour task is to synchronize this data structure using whichever primitives you like.\nTry to keep the implementation efficient.\nAim to decrease your running times as much as you can."),(0,r.kt)("h2",{id:"minor-and-major-page-faults"},"Minor and Major Page Faults"),(0,r.kt)("p",null,"The code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/page-faults/page_faults.c")," generates some minor and major page faults.\nOpen 2 terminals: one in which you will run the program, and one which will monitor the page faults of the program.\nIn the monitoring terminal, run the following command:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"watch -n 1 'ps -eo min_flt,maj_flt,cmd | grep ./page_faults | head -n 1'\n")),(0,r.kt)("p",null,"Compile the program and run it in the other terminal.\nYou must press ",(0,r.kt)("inlineCode",{parentName:"p"},"enter")," one time, before the program will prompt you to press ",(0,r.kt)("inlineCode",{parentName:"p"},"enter")," more times.\nWatch the first number on the monitoring terminal;\nit increases.\nThose are the minor page faults."),(0,r.kt)("h3",{id:"minor-page-faults"},"Minor Page Faults"),(0,r.kt)("p",null,"A minor page fault is generated whenever a requested page is present in the physical memory, as a frame, but that frame isn't allocated to the process generating the request.\nThese types of page faults are the most common, and they happen when calling functions from dynamic libraries, allocating heap memory, loading programs, reading files that have been cached, and many more situations.\nNow back to the program."),(0,r.kt)("p",null,"The monitoring command already starts with some minor page faults, generated when loading the program."),(0,r.kt)("p",null,"After pressing ",(0,r.kt)("inlineCode",{parentName:"p"},"enter"),", the number increases, because a function from a dynamic library (libc) is fetched when the first ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," is executed.\nSubsequent calls to functions that are in the same memory page as ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," won't generate other page faults."),(0,r.kt)("p",null,'After allocating the 100 Bytes, you might not see the number of page faults increase.\nThis is because the "bookkeeping" data allocated by ',(0,r.kt)("inlineCode",{parentName:"p"},"malloc()")," was able to fit in an already mapped page.\nThe second allocation, the 1GB one, will always gnereate one minor page fault - for the bookkeeping data about the allocated memory zone.\nNotice that not all the pages for the 1GB are allocated.\nThey are allocated - and generate page faults - when modified.\nBy now you should know that this mechanism is called ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/copy-on-write"},"copy-on-write"),"."),(0,r.kt)("p",null,"Continue with pressing ",(0,r.kt)("inlineCode",{parentName:"p"},"enter")," and observing the effects util you reach opening ",(0,r.kt)("inlineCode",{parentName:"p"},"file.txt"),"."),(0,r.kt)("p",null,"Note that neither opening a file, getting information about it, nor mapping it in memory using ",(0,r.kt)("inlineCode",{parentName:"p"},"mmap()"),", generate page faults.\nAlso note the ",(0,r.kt)("inlineCode",{parentName:"p"},"posix_fadvise()")," call after the one to ",(0,r.kt)("inlineCode",{parentName:"p"},"fstat()"),".\nWith this call we force the OS to not cache the file, so we can generate a ",(0,r.kt)("strong",{parentName:"p"},"major page fault"),"."),(0,r.kt)("h3",{id:"major-page-faults"},"Major Page Faults"),(0,r.kt)("p",null,"Major page faults happen when a page is requested, but it isn't present in the physical memory.\nThese types of page faults happen in 2 situations:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"a page that was swapped out (to the disk), due to lack of memory, is now accessed - this case is harder to show"),(0,r.kt)("li",{parentName:"ul"},"the OS needs to read a file from the disk, because the file contents aren't present in the cache - the case we are showing now")),(0,r.kt)("p",null,"Press ",(0,r.kt)("inlineCode",{parentName:"p"},"enter")," to print the file contents.\nNote the second number go up in the monitoring terminal."),(0,r.kt)("p",null,"Comment the ",(0,r.kt)("inlineCode",{parentName:"p"},"posix_fadvise()")," call, recompile the program, and run it again.\nYou won't get any major page fault, because the file contents are cached by the OS, to avoid those page faults.\nAs a rule, the OS will avoid major page faults whenever possible, because they are very costly in terms of running time."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/2e030903.b906e43b.js b/17/assets/js/2e030903.b906e43b.js new file mode 100644 index 0000000000..8f01211dd2 --- /dev/null +++ b/17/assets/js/2e030903.b906e43b.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2661],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>d});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?o(Object(r),!0).forEach((function(t){a(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):o(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function l(e,t){if(null==e)return{};var r,n,a=function(e,t){if(null==e)return{};var r,n,a={},o=Object.keys(e);for(n=0;n<o.length;n++)r=o[n],t.indexOf(r)>=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n<o.length;n++)r=o[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var c=n.createContext({}),s=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},u=function(e){var t=s(e.components);return n.createElement(c.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),p=s(r),m=a,d=p["".concat(c,".").concat(m)]||p[m]||f[m]||o;return r?n.createElement(d,i(i({ref:t},u),{},{components:r})):n.createElement(d,i({ref:t},u))}));function d(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=m;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[p]="string"==typeof e?e:a,i[1]=l;for(var s=2;s<o;s++)i[s]=r[s];return n.createElement.apply(null,i)}return n.createElement.apply(null,r)}m.displayName="MDXCreateElement"},8580:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>f,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var n=r(7462),a=(r(7294),r(3905));const o={},i="Stack Protector",l={unversionedId:"Lab/Data/quiz/memory-stack-protector",id:"Lab/Data/quiz/memory-stack-protector",title:"Stack Protector",description:"Question Text",source:"@site/docs/Lab/Data/quiz/memory-stack-protector.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/memory-stack-protector",permalink:"/operating-systems/17/Lab/Data/quiz/memory-stack-protector",draft:!1,tags:[],version:"current",frontMatter:{}},c={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:s},p="wrapper";function f(e){let{components:t,...r}=e;return(0,a.kt)(p,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"stack-protector"},"Stack Protector"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"For the ",(0,a.kt)("inlineCode",{parentName:"p"},"support/memory-security/bo_write_practice")," executable we are not able overwrite the ",(0,a.kt)("inlineCode",{parentName:"p"},"s")," variable no matter the input.\nWhy is that?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"s")," needs to be declared after ",(0,a.kt)("inlineCode",{parentName:"li"},"buffer"),".")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"when using the stack canary, the buffer is always placed right below it.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"due to ASLR, the address of ",(0,a.kt)("inlineCode",{parentName:"p"},"s")," is random.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the stack canary makes it impossible to overwrite local variables."))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"When using the canary, to minimize the damage a buffer overflow could cause, the buffers are always placed right below the canary.\nBy doing so, a buffer overflow will not overwrite anything.\nHowever, it is still possible to overwrite other local buffers, provided that a function declares more than 1 array or if we use the pointer directly."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/31a6b50d.3bab6d53.js b/17/assets/js/31a6b50d.3bab6d53.js new file mode 100644 index 0000000000..3d27a133d9 --- /dev/null +++ b/17/assets/js/31a6b50d.3bab6d53.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3675],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>d});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function s(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},o=Object.keys(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),l=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},p=function(e){var t=l(e.components);return r.createElement(c.Provider,{value:t},e.children)},m="mdxType",h={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},u=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),m=l(n),u=a,d=m["".concat(c,".").concat(u)]||m[u]||h[u]||o;return n?r.createElement(d,i(i({ref:t},p),{},{components:n})):r.createElement(d,i({ref:t},p))}));function d(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,i=new Array(o);i[0]=u;var s={};for(var c in t)hasOwnProperty.call(t,c)&&(s[c]=t[c]);s.originalType=e,s[m]="string"==typeof e?e:a,i[1]=s;for(var l=2;l<o;l++)i[l]=n[l];return r.createElement.apply(null,i)}return r.createElement.apply(null,n)}u.displayName="MDXCreateElement"},4284:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>h,frontMatter:()=>o,metadata:()=>s,toc:()=>l});var r=n(7462),a=(n(7294),n(3905));const o={},i="The X Window System",s={unversionedId:"Lab/Application Interaction/x-window-system",id:"Lab/Application Interaction/x-window-system",title:"The X Window System",description:"Unix-like systems that support a Graphical User Interface usually do this through the X Window System.",source:"@site/docs/Lab/Application Interaction/x-window-system.md",sourceDirName:"Lab/Application Interaction",slug:"/Lab/Application Interaction/x-window-system",permalink:"/operating-systems/17/Lab/Application Interaction/x-window-system",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Password Cracker",permalink:"/operating-systems/17/Lab/Application Interaction/password-cracker"},next:{title:"D-Bus",permalink:"/operating-systems/17/Lab/Application Interaction/dbus"}},c={},l=[{value:"X Client and Server on the Same Machine",id:"x-client-and-server-on-the-same-machine",level:2}],p={toc:l},m="wrapper";function h(e){let{components:t,...o}=e;return(0,a.kt)(m,(0,r.Z)({},p,o,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"the-x-window-system"},"The X Window System"),(0,a.kt)("p",null,"Unix-like systems that support a Graphical User Interface usually do this through the X Window System.\nThis system is implemented with a client-server model: the X Server is the component that controls the screen, keyboard, mouse, and other parts related to the GUI, while the X clients are the applications that want to use the graphical interface (like, for example, an internet browser)."),(0,a.kt)("p",null,"The clients and the server communicate using a standardized protocol, and the system does not necessarily require the client and server to be on the same machine.\nAlthough not so common nowadays, the X client can run on a different machine than the server, with the communication happening over the network.\nBut in the more usual case, when both the client and the server are on the same machine, modern implementations of the X Window System use a faster communication channel, like a Unix socket."),(0,a.kt)("h2",{id:"x-client-and-server-on-the-same-machine"},"X Client and Server on the Same Machine"),(0,a.kt)("p",null,"Let's investigate the case when both the X client and X server run on the same machine.\nFirst we'll take a look at the Unix sockets that are in listening mode."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ sudo netstat -xnlp | grep X11\nunix 2 [ ACC ] STREAM LISTENING 29120 3472/Xorg @/tmp/.X11-unix/X0\nunix 2 [ ACC ] STREAM LISTENING 29121 3472/Xorg /tmp/.X11-unix/X0\n")),(0,a.kt)("p",null,"We observe the ",(0,a.kt)("inlineCode",{parentName:"p"},"Xorg")," process (the X server) listening on a Unix socket with the path ",(0,a.kt)("inlineCode",{parentName:"p"},"/tmp/.X11-unix/X0"),"."),(0,a.kt)("p",null,"Now let's run an X client (that is, a GUI application) and check that it will indeed connect to this Unix socket.\nA very simple example is the ",(0,a.kt)("inlineCode",{parentName:"p"},"xeyes")," application:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace -e trace=socket,connect xeyes\nsocket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3\nconnect(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X0"}, 20) = 0\n')),(0,a.kt)("p",null,"As expected, the application creates a Unix socket, then connects to the path ",(0,a.kt)("inlineCode",{parentName:"p"},'@"/tmp/.X11-unix/X0"'),"."),(0,a.kt)("p",null,"Furthermore, let's confirm that there is actual communication taking place between ",(0,a.kt)("inlineCode",{parentName:"p"},"xeyes")," and the X server.\nWe'll run ",(0,a.kt)("inlineCode",{parentName:"p"},"xeyes")," again, and then we'll keep moving the mouse cursor around.\nWhen the mouse is moved, the following events are taking place:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The X server captures the mouse movements (since the server is the one that controls the mouse)"),(0,a.kt)("li",{parentName:"ul"},'The X server will pass these "mouse moved" events to the clients (including xeyes)'),(0,a.kt)("li",{parentName:"ul"},"The client (xeyes) uses these events to update its window (changing the position of the pupils inside the eyes)")),(0,a.kt)("p",null,"So, if we run ",(0,a.kt)("inlineCode",{parentName:"p"},"xeyes")," under ",(0,a.kt)("inlineCode",{parentName:"p"},"strace"),", we expect to see some communication on the Unix socket that is created at the beginning:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"strace -e 'trace=!poll' -e trace='socket,connect,recvmsg' xeyes |& grep -v '\\-1 EAGAIN'\n")),(0,a.kt)("p",null,(0,a.kt)("img",{alt:"strace-xeyes",src:n(7504).Z,width:"1920",height:"1080"})))}h.isMDXComponent=!0},7504:(e,t,n)=>{n.d(t,{Z:()=>r});const r=n.p+"assets/images/strace_xeyes-f055c379e7028e39de13f0d039171718.gif"}}]); \ No newline at end of file diff --git a/17/assets/js/32839693.e1596f9a.js b/17/assets/js/32839693.e1596f9a.js new file mode 100644 index 0000000000..a2f5cae528 --- /dev/null +++ b/17/assets/js/32839693.e1596f9a.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3063],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function u(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function i(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},o=Object.keys(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var l=r.createContext({}),s=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):u(u({},t),e)),n},c=function(e){var t=s(e.components);return r.createElement(l.Provider,{value:t},e.children)},p="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},h=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,l=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),p=s(n),h=a,m=p["".concat(l,".").concat(h)]||p[h]||d[h]||o;return n?r.createElement(m,u(u({ref:t},c),{},{components:n})):r.createElement(m,u({ref:t},c))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,u=new Array(o);u[0]=h;var i={};for(var l in t)hasOwnProperty.call(t,l)&&(i[l]=t[l]);i.originalType=e,i[p]="string"==typeof e?e:a,u[1]=i;for(var s=2;s<o;s++)u[s]=n[s];return r.createElement.apply(null,u)}return r.createElement.apply(null,n)}h.displayName="MDXCreateElement"},4838:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>u,default:()=>d,frontMatter:()=>o,metadata:()=>i,toc:()=>s});var r=n(7462),a=(n(7294),n(3905));const o={},u="The Need for a COMPLETED Queue",i={unversionedId:"Lab/Compute/quiz/why-use-completed-queue",id:"Lab/Compute/quiz/why-use-completed-queue",title:"The Need for a COMPLETED Queue",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/why-use-completed-queue.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/why-use-completed-queue",permalink:"/operating-systems/17/Lab/Compute/quiz/why-use-completed-queue",draft:!1,tags:[],version:"current",frontMatter:{}},l={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:s},p="wrapper";function d(e){let{components:t,...n}=e;return(0,a.kt)(p,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"the-need-for-a-completed-queue"},"The Need for a COMPLETED Queue"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Why does the scheduler need a COMPLETED queue and not simply terminate one thread once its function finishes?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"The COMPLETED queue is an implementation preference.\nThe scheduler can expose the same functions without it")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the OS's scheduler may kill the main kernel-level thread unless we keep the user-level thread in a queue"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The COMPLETED queue is needed to save the value returned by the thread so that it can later be retrieved by ",(0,a.kt)("inlineCode",{parentName:"li"},"threads_join()"),".")),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Take a look at the ",(0,a.kt)("inlineCode",{parentName:"p"},"handle_thread_start()")," function.\nIt is used by ",(0,a.kt)("inlineCode",{parentName:"p"},"threads_create()")," to start executing the given function.\nThis is a wrapper that calls the function associated with the thread (",(0,a.kt)("inlineCode",{parentName:"p"},"this->start_routine"),"), saves its result and then calls ",(0,a.kt)("inlineCode",{parentName:"p"},"threads_exit()"),"\nto store this result in the COMPLETED queue."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/34746f8c.a1d84fcf.js b/17/assets/js/34746f8c.a1d84fcf.js new file mode 100644 index 0000000000..527e2f3458 --- /dev/null +++ b/17/assets/js/34746f8c.a1d84fcf.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5330],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function s(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},i=Object.keys(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var l=r.createContext({}),u=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},c=function(e){var t=u(e.components);return r.createElement(l.Provider,{value:t},e.children)},d="mdxType",p={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},h=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,l=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),d=u(n),h=a,m=d["".concat(l,".").concat(h)]||d[h]||p[h]||i;return n?r.createElement(m,o(o({ref:t},c),{},{components:n})):r.createElement(m,o({ref:t},c))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,o=new Array(i);o[0]=h;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[d]="string"==typeof e?e:a,o[1]=s;for(var u=2;u<i;u++)o[u]=n[u];return r.createElement.apply(null,o)}return r.createElement.apply(null,n)}h.displayName="MDXCreateElement"},4976:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>o,default:()=>p,frontMatter:()=>i,metadata:()=>s,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const i={},o="ULT Thread IDs",s={unversionedId:"Lab/Compute/quiz/ult-thread-ids",id:"Lab/Compute/quiz/ult-thread-ids",title:"ULT Thread IDs",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/ult-thread-ids.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/ult-thread-ids",permalink:"/operating-systems/17/Lab/Compute/quiz/ult-thread-ids",draft:!1,tags:[],version:"current",frontMatter:{}},l={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:u},d="wrapper";function p(e){let{components:t,...n}=e;return(0,a.kt)(d,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"ult-thread-ids"},"ULT Thread IDs"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Why do the thread IDs returned by ",(0,a.kt)("inlineCode",{parentName:"p"},"threads_create()")," start from 2 and not 1?\nWhy is this necessary."),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because ID 1 is associated with the main thread.\nThis is an implementation detail and can be omitted.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because ID 1 belongs to the main thread.\nThis is needed in order to associate a ",(0,a.kt)("inlineCode",{parentName:"li"},"ucontext_t")," with the main thread as well, so the main thread can also be run.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the underlying kernel thread is assigned ID 1.\nThis is mandatory in order for the OS's scheduler to run this thread.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because ",(0,a.kt)("inlineCode",{parentName:"p"},"libult.so")," first creates a pool of threads from which it ",(0,a.kt)("inlineCode",{parentName:"p"},"threads_create()")," retrieves the threads it returns."))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"threads_create()")," function calls ",(0,a.kt)("inlineCode",{parentName:"p"},"init_first_context()"),", which, in turn, calls ",(0,a.kt)("inlineCode",{parentName:"p"},"tcb_new()"),", thus creating the first context associated with the main thread (the one calling ",(0,a.kt)("inlineCode",{parentName:"p"},"threads_create()")," the first time).\nWithout this, the scheduler in ",(0,a.kt)("inlineCode",{parentName:"p"},"libult.so")," wouldn't be able to run the main thread."))}p.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/35edd654.2244c528.js b/17/assets/js/35edd654.2244c528.js new file mode 100644 index 0000000000..a1d8901cc9 --- /dev/null +++ b/17/assets/js/35edd654.2244c528.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9671],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>f});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function l(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?l(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):l(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function o(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},l=Object.keys(e);for(r=0;r<l.length;r++)n=l[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(r=0;r<l.length;r++)n=l[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var s=r.createContext({}),u=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},p=function(e){var t=u(e.components);return r.createElement(s.Provider,{value:t},e.children)},c="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},m=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,l=e.originalType,s=e.parentName,p=o(e,["components","mdxType","originalType","parentName"]),c=u(n),m=a,f=c["".concat(s,".").concat(m)]||c[m]||d[m]||l;return n?r.createElement(f,i(i({ref:t},p),{},{components:n})):r.createElement(f,i({ref:t},p))}));function f(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var l=n.length,i=new Array(l);i[0]=m;var o={};for(var s in t)hasOwnProperty.call(t,s)&&(o[s]=t[s]);o.originalType=e,o[c]="string"==typeof e?e:a,i[1]=o;for(var u=2;u<l;u++)i[u]=n[u];return r.createElement.apply(null,i)}return r.createElement.apply(null,n)}m.displayName="MDXCreateElement"},4570:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>i,default:()=>d,frontMatter:()=>l,metadata:()=>o,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const l={},i="Valgrind Leaks",o={unversionedId:"Lab/Data/quiz/valgrind-leaks",id:"Lab/Data/quiz/valgrind-leaks",title:"Valgrind Leaks",description:"Question Text",source:"@site/docs/Lab/Data/quiz/valgrind-leaks.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/valgrind-leaks",permalink:"/operating-systems/17/Lab/Data/quiz/valgrind-leaks",draft:!1,tags:[],version:"current",frontMatter:{}},s={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:u},c="wrapper";function d(e){let{components:t,...n}=e;return(0,a.kt)(c,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"valgrind-leaks"},"Valgrind Leaks"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},'struct student {\n char *name;\n int age;\n}\n\nstruct student *s = malloc(sizeof(*s));\ns->name = strdup("Reginald");\n// ...\nfree(s);\n')),(0,a.kt)("p",null,"What are the leaks in the above c program?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"There are no leaks")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"s->name")," is ",(0,a.kt)("strong",{parentName:"p"},"definitely lost")))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"s->name")," is ",(0,a.kt)("strong",{parentName:"li"},"indirectly lost"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"s->name")," is ",(0,a.kt)("strong",{parentName:"li"},"still reachable"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"strdup()")," allocates memory for a string so the returned pointer must be freed.\nFreeing ",(0,a.kt)("inlineCode",{parentName:"p"},"s")," will leave us unable to free ",(0,a.kt)("inlineCode",{parentName:"p"},"s->name"),", so ",(0,a.kt)("inlineCode",{parentName:"p"},"s->name")," is ",(0,a.kt)("strong",{parentName:"p"},"indirectly lost"),".\nFind more about valgrind leak categories ",(0,a.kt)("a",{parentName:"p",href:"https://valgrind.org/docs/manual/faq.html#faq.deflost"},"here"),"."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/374a2386.1339cda0.js b/17/assets/js/374a2386.1339cda0.js new file mode 100644 index 0000000000..90e2357137 --- /dev/null +++ b/17/assets/js/374a2386.1339cda0.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1765],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>y});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function a(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){o(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function s(e,t){if(null==e)return{};var n,r,o=function(e,t){if(null==e)return{};var n,r,o={},i=Object.keys(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var l=r.createContext({}),p=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},u=function(e){var t=p(e.components);return r.createElement(l.Provider,{value:t},e.children)},c="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,i=e.originalType,l=e.parentName,u=s(e,["components","mdxType","originalType","parentName"]),c=p(n),f=o,y=c["".concat(l,".").concat(f)]||c[f]||m[f]||i;return n?r.createElement(y,a(a({ref:t},u),{},{components:n})):r.createElement(y,a({ref:t},u))}));function y(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=n.length,a=new Array(i);a[0]=f;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[c]="string"==typeof e?e:o,a[1]=s;for(var p=2;p<i;p++)a[p]=n[p];return r.createElement.apply(null,a)}return r.createElement.apply(null,n)}f.displayName="MDXCreateElement"},4458:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>a,default:()=>m,frontMatter:()=>i,metadata:()=>s,toc:()=>p});var r=n(7462),o=(n(7294),n(3905));const i={},a="Limitation of Anonymous Pipes",s={unversionedId:"Lab/IO/quiz/anonymous-pipes-limitation",id:"Lab/IO/quiz/anonymous-pipes-limitation",title:"Limitation of Anonymous Pipes",description:"Question Text",source:"@site/docs/Lab/IO/quiz/anonymous-pipes-limitation.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/anonymous-pipes-limitation",permalink:"/operating-systems/17/Lab/IO/quiz/anonymous-pipes-limitation",draft:!1,tags:[],version:"current",frontMatter:{}},l={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:p},c="wrapper";function m(e){let{components:t,...n}=e;return(0,o.kt)(c,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"limitation-of-anonymous-pipes"},"Limitation of Anonymous Pipes"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"What of the following is the largest drawback of using anonymous pipes (created with ",(0,o.kt)("inlineCode",{parentName:"p"},"pipe()"),") for inter-process communication?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"they only allow unidirectional communication")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},'they only allow IPC between "related" processes (parent - child - grandchild etc.)')),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"if more processes use the same end of the same pipe, there may be race conditions")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"a pipe only has 2 file descriptors, but some processes may need more channels to communicate"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"Out of the answers above, the only limitation that cannot be overcome by using pipes alone is the requirement for the processes using them to be related.\nThe parent must create the pipes so the children can inherit them.\nAll other downsides can be overcome by using more pipes."),(0,o.kt)("p",null,"The answer to this is to employ ",(0,o.kt)("em",{parentName:"p"},"some")," filesystem support."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/392776e2.fe335ad7.js b/17/assets/js/392776e2.fe335ad7.js new file mode 100644 index 0000000000..fccbe92352 --- /dev/null +++ b/17/assets/js/392776e2.fe335ad7.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8351],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>f});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function l(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){r(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function i(e,t){if(null==e)return{};var n,a,r=function(e,t){if(null==e)return{};var n,a,r={},o=Object.keys(e);for(a=0;a<o.length;a++)n=o[a],t.indexOf(n)>=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a<o.length;a++)n=o[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var p=a.createContext({}),m=function(e){var t=a.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},u=function(e){var t=m(e.components);return a.createElement(p.Provider,{value:t},e.children)},c="mdxType",s={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},d=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,p=e.parentName,u=i(e,["components","mdxType","originalType","parentName"]),c=m(n),d=r,f=c["".concat(p,".").concat(d)]||c[d]||s[d]||o;return n?a.createElement(f,l(l({ref:t},u),{},{components:n})):a.createElement(f,l({ref:t},u))}));function f(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,l=new Array(o);l[0]=d;var i={};for(var p in t)hasOwnProperty.call(t,p)&&(i[p]=t[p]);i.originalType=e,i[c]="string"==typeof e?e:r,l[1]=i;for(var m=2;m<o;m++)l[m]=n[m];return a.createElement.apply(null,l)}return a.createElement.apply(null,n)}d.displayName="MDXCreateElement"},1201:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>l,default:()=>s,frontMatter:()=>o,metadata:()=>i,toc:()=>m});var a=n(7462),r=(n(7294),n(3905));const o={},l="Malloc `mmap()`",i={unversionedId:"Lab/Data/quiz/malloc-mmap",id:"Lab/Data/quiz/malloc-mmap",title:"Malloc `mmap()`",description:"Question Text",source:"@site/docs/Lab/Data/quiz/malloc-mmap.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/malloc-mmap",permalink:"/operating-systems/17/Lab/Data/quiz/malloc-mmap",draft:!1,tags:[],version:"current",frontMatter:{}},p={},m=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:m},c="wrapper";function s(e){let{components:t,...n}=e;return(0,r.kt)(c,(0,a.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"malloc-mmap"},"Malloc ",(0,r.kt)("inlineCode",{parentName:"h1"},"mmap()")),(0,r.kt)("h2",{id:"question-text"},"Question Text"),(0,r.kt)("p",null,"When does ",(0,r.kt)("inlineCode",{parentName:"p"},"malloc()")," use ",(0,r.kt)("inlineCode",{parentName:"p"},"mmap()"),"?"),(0,r.kt)("h2",{id:"question-answers"},"Question Answers"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"When it allocates read-only memory")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"When it allocates zeroed memory"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"When it allocates chunks of memory bigger than an internal threshold")),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"When the heap is full")),(0,r.kt)("h2",{id:"feedback"},"Feedback"),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"malloc")," uses both ",(0,r.kt)("inlineCode",{parentName:"p"},"brk()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"mmap()"),", but prefers ",(0,r.kt)("inlineCode",{parentName:"p"},"mmap()")," for big chunks of memory (by default larger than 128 KB).\nThis value can be altered using ",(0,r.kt)("a",{parentName:"p",href:"https://linux.die.net/man/3/mallopt"},(0,r.kt)("inlineCode",{parentName:"a"},"mallopt()"))," with the ",(0,r.kt)("inlineCode",{parentName:"p"},"param")," argument set to ",(0,r.kt)("inlineCode",{parentName:"p"},"M_MMAP_THRESHOLD"),".\nThese memroy blocks are unlikely to be reused so they are not placed on heap to avoid memory fragmentation."))}s.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/393be207.71906d0c.js b/17/assets/js/393be207.71906d0c.js new file mode 100644 index 0000000000..3187463a02 --- /dev/null +++ b/17/assets/js/393be207.71906d0c.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7414],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>d});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function p(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?a(Object(r),!0).forEach((function(t){o(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):a(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function c(e,t){if(null==e)return{};var r,n,o=function(e,t){if(null==e)return{};var r,n,o={},a=Object.keys(e);for(n=0;n<a.length;n++)r=a[n],t.indexOf(r)>=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n<a.length;n++)r=a[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var l=n.createContext({}),i=function(e){var t=n.useContext(l),r=t;return e&&(r="function"==typeof e?e(t):p(p({},t),e)),r},u=function(e){var t=i(e.components);return n.createElement(l.Provider,{value:t},e.children)},s="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,l=e.parentName,u=c(e,["components","mdxType","originalType","parentName"]),s=i(r),m=o,d=s["".concat(l,".").concat(m)]||s[m]||f[m]||a;return r?n.createElement(d,p(p({ref:t},u),{},{components:r})):n.createElement(d,p({ref:t},u))}));function d(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,p=new Array(a);p[0]=m;var c={};for(var l in t)hasOwnProperty.call(t,l)&&(c[l]=t[l]);c.originalType=e,c[s]="string"==typeof e?e:o,p[1]=c;for(var i=2;i<a;i++)p[i]=r[i];return n.createElement.apply(null,p)}return n.createElement.apply(null,r)}m.displayName="MDXCreateElement"},9286:(e,t,r)=>{r.r(t),r.d(t,{contentTitle:()=>p,default:()=>s,frontMatter:()=>a,metadata:()=>c,toc:()=>l});var n=r(7462),o=(r(7294),r(3905));const a={title:"Markdown page example"},p="Markdown page example",c={type:"mdx",permalink:"/operating-systems/17/markdown-page",source:"@site/src/pages/markdown-page.md",title:"Markdown page example",description:"You don't need React to write simple standalone pages.",frontMatter:{title:"Markdown page example"}},l=[],i={toc:l},u="wrapper";function s(e){let{components:t,...r}=e;return(0,o.kt)(u,(0,n.Z)({},i,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"markdown-page-example"},"Markdown page example"),(0,o.kt)("p",null,"You don't need React to write simple standalone pages."))}s.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/3c30233f.1e5021d2.js b/17/assets/js/3c30233f.1e5021d2.js new file mode 100644 index 0000000000..dde8203b52 --- /dev/null +++ b/17/assets/js/3c30233f.1e5021d2.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1901],{3905:(e,t,r)=>{r.d(t,{Zo:()=>l,kt:()=>d});var a=r(7294);function n(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,a)}return r}function s(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?o(Object(r),!0).forEach((function(t){n(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):o(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function i(e,t){if(null==e)return{};var r,a,n=function(e,t){if(null==e)return{};var r,a,n={},o=Object.keys(e);for(a=0;a<o.length;a++)r=o[a],t.indexOf(r)>=0||(n[r]=e[r]);return n}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a<o.length;a++)r=o[a],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(n[r]=e[r])}return n}var p=a.createContext({}),c=function(e){var t=a.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):s(s({},t),e)),r},l=function(e){var t=c(e.components);return a.createElement(p.Provider,{value:t},e.children)},h="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var r=e.components,n=e.mdxType,o=e.originalType,p=e.parentName,l=i(e,["components","mdxType","originalType","parentName"]),h=c(r),u=n,d=h["".concat(p,".").concat(u)]||h[u]||m[u]||o;return r?a.createElement(d,s(s({ref:t},l),{},{components:r})):a.createElement(d,s({ref:t},l))}));function d(e,t){var r=arguments,n=t&&t.mdxType;if("string"==typeof e||n){var o=r.length,s=new Array(o);s[0]=u;var i={};for(var p in t)hasOwnProperty.call(t,p)&&(i[p]=t[p]);i.originalType=e,i[h]="string"==typeof e?e:n,s[1]=i;for(var c=2;c<o;c++)s[c]=r[c];return a.createElement.apply(null,s)}return a.createElement.apply(null,r)}u.displayName="MDXCreateElement"},6904:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>s,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>c});var a=r(7462),n=(r(7294),r(3905));const o={},s="Hardware Perspective",i={unversionedId:"Lab/Compute/hardware-perspective",id:"Lab/Compute/hardware-perspective",title:"Hardware Perspective",description:"The main criterion we use to rank CPUs is their computation power, i.e. their ability to crunch numbers and do math.",source:"@site/docs/Lab/Compute/hardware-perspective.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/hardware-perspective",permalink:"/operating-systems/17/Lab/Compute/hardware-perspective",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Compute",permalink:"/operating-systems/17/Lab/Compute/overview"},next:{title:"Processes",permalink:"/operating-systems/17/Lab/Compute/processes"}},p={},c=[{value:"The Role of the Operating System",id:"the-role-of-the-operating-system",level:2}],l={toc:c},h="wrapper";function m(e){let{components:t,...o}=e;return(0,n.kt)(h,(0,a.Z)({},l,o,{components:t,mdxType:"MDXLayout"}),(0,n.kt)("h1",{id:"hardware-perspective"},"Hardware Perspective"),(0,n.kt)("p",null,"The main criterion we use to rank CPUs is their ",(0,n.kt)("em",{parentName:"p"},"computation power"),", i.e. their ability to crunch numbers and do math.\nNumerous benchmarks exist out there, and they are publicly displayed on sites such as ",(0,n.kt)("a",{parentName:"p",href:"https://www.cpubenchmark.net/"},"CPUBenchmark"),"."),(0,n.kt)("p",null,"For example, a benchmark can measure the performance of the computer's CPU in a variety of scenarios:"),(0,n.kt)("ul",null,(0,n.kt)("li",{parentName:"ul"},"its ability to perform integer operations"),(0,n.kt)("li",{parentName:"ul"},"its speed in floating point arithmetic"),(0,n.kt)("li",{parentName:"ul"},"data encryption and compression"),(0,n.kt)("li",{parentName:"ul"},"sorting algorithms and others")),(0,n.kt)("p",null,"You can take a look at what exactly is measured using ",(0,n.kt)("a",{parentName:"p",href:"https://www.cpubenchmark.net/cpu.php?cpu=AMD+Ryzen+Threadripper+PRO+5995WX"},"this link"),".\nIt displays the scores obtained by a high-end CPU.\nApart from the tests above, other benchmarks might focus on different performance metrics, such as branch prediction or prefetching."),(0,n.kt)("p",null,"Other approaches are less artificial, measuring performance on real-world applications such as compile times and performance in the latest (and most resource-demanding) video games.\nThe latter metric revolves around how many average FPS (frames per second) a given CPU is able to crank out in a specific video game.\n",(0,n.kt)("a",{parentName:"p",href:"https://www.gamersnexus.net/guides/3577-cpu-test-methodology-unveil-for-2020-compile-gaming-more"},"This article")," goes into more detail regarding the methodology of running CPU benchmarks on real-world applications."),(0,n.kt)("p",null,"Most benchmarks, unfortunately, are not open source, especially the more popular ones, such as ",(0,n.kt)("a",{parentName:"p",href:"https://browser.geekbench.com/processor-benchmarks"},"Geekbench 5"),".\nDespite this shortcoming, benchmarks are widely used to compare the performance of various computer ",(0,n.kt)("strong",{parentName:"p"},"hardware"),", CPUs included."),(0,n.kt)("h2",{id:"the-role-of-the-operating-system"},"The Role of the Operating System"),(0,n.kt)("p",null,'As you\'ve seen so far, the CPU provides the "muscle" required for fast computation, i.e. the highly optimised hardware and multiple ALUs, FPUs\nand cores necessary to perform those computations.\nHowever, it is the ',(0,n.kt)("strong",{parentName:"p"},"operating system"),' that provides the "brains" for this computation.\nSpecifically, modern CPUs have the capacity to run multiple tasks in parallel.\nBut they do not provide a means to decide which task to run at each moment.\nThe OS comes as an ',(0,n.kt)("em",{parentName:"p"},"orchestrator")," to ",(0,n.kt)("strong",{parentName:"p"},"schedule")," the way these tasks (that we will later call threads) are allowed to run and use the CPU's resources.\nThis way, the OS tells the CPU what code to run on each CPU core so that it reaches a good balance between high throughput (running many instructions) and fair access to CPU cores."),(0,n.kt)("p",null,"It is cumbersome for a user-level application to interact directly with the CPU.\nThe developer would have to write hardware-specific code, which is not scalable and is difficult to maintain.\nIn addition, doing so would leave it up to the developer to isolate their application from the others that are present on the system.\nThis leaves applications vulnerable to countless bugs and exploits."),(0,n.kt)("p",null,"To guard apps from these pitfalls, the OS comes and mediates interactions between regular programs and the CPU by providing a set of ",(0,n.kt)("strong",{parentName:"p"},"abstractions"),".\nThese abstractions offer a safe, uniform and also isolated way to leverage the CPU's resources, i.e. its cores.\nThere are 2 main abstractions: ",(0,n.kt)("strong",{parentName:"p"},"processes")," and ",(0,n.kt)("strong",{parentName:"p"},"threads"),"."),(0,n.kt)("p",null,(0,n.kt)("img",{alt:"Interaction between applications, OS and CPU",src:r(2163).Z,width:"772",height:"772"})),(0,n.kt)("p",null,"As we can see from the image above, an application can spawn one or more processes.\nEach of these is handled and maintained by the OS.\nSimilarly, each process can spawn however many threads, which are also managed by the OS.\nThe OS decides when and on what CPU core to make each thread run.\nThis is in line with the general interaction between an application and the hardware: it is always mediated by the OS."))}m.isMDXComponent=!0},2163:(e,t,r)=>{r.d(t,{Z:()=>a});const a=r.p+"assets/images/app-os-cpu-interaction-ca7fbdbb7da380e0992c95467ef267ce.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/3e17b32d.5b7054e2.js b/17/assets/js/3e17b32d.5b7054e2.js new file mode 100644 index 0000000000..fb7aecba96 --- /dev/null +++ b/17/assets/js/3e17b32d.5b7054e2.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8650],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>d});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?o(Object(r),!0).forEach((function(t){a(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):o(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function l(e,t){if(null==e)return{};var r,n,a=function(e,t){if(null==e)return{};var r,n,a={},o=Object.keys(e);for(n=0;n<o.length;n++)r=o[n],t.indexOf(r)>=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n<o.length;n++)r=o[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var c=n.createContext({}),s=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},u=function(e){var t=s(e.components);return n.createElement(c.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},y=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),p=s(r),y=a,d=p["".concat(c,".").concat(y)]||p[y]||f[y]||o;return r?n.createElement(d,i(i({ref:t},u),{},{components:r})):n.createElement(d,i({ref:t},u))}));function d(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=y;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[p]="string"==typeof e?e:a,i[1]=l;for(var s=2;s<o;s++)i[s]=r[s];return n.createElement.apply(null,i)}return n.createElement.apply(null,r)}y.displayName="MDXCreateElement"},626:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>f,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var n=r(7462),a=(r(7294),r(3905));const o={},i="Strcpy Buffer Overflow",l={unversionedId:"Lab/Data/quiz/string-strcpy",id:"Lab/Data/quiz/string-strcpy",title:"Strcpy Buffer Overflow",description:"Question Text",source:"@site/docs/Lab/Data/quiz/string-strcpy.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/string-strcpy",permalink:"/operating-systems/17/Lab/Data/quiz/string-strcpy",draft:!1,tags:[],version:"current",frontMatter:{}},c={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:s},p="wrapper";function f(e){let{components:t,...r}=e;return(0,a.kt)(p,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"strcpy-buffer-overflow"},"Strcpy Buffer Overflow"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Does any buffer overflow occur with the latest version of the program?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"no, because ",(0,a.kt)("inlineCode",{parentName:"p"},"strcpy()")," was designed to correctly handle such situations.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"no, because the string is correctly printed, i.e. no extra characters.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"we cannot know"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"yes, ",(0,a.kt)("inlineCode",{parentName:"li"},"strcpy()")," copies the entirety of the source, including the ",(0,a.kt)("inlineCode",{parentName:"li"},"\\0"),"; since the size of ",(0,a.kt)("inlineCode",{parentName:"li"},"dst")," is 4 the null byte overwrites the least significant byte of ",(0,a.kt)("inlineCode",{parentName:"li"},"var"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Print the contents of variable ",(0,a.kt)("inlineCode",{parentName:"p"},"var"),"."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/3e3393a3.a0490608.js b/17/assets/js/3e3393a3.a0490608.js new file mode 100644 index 0000000000..084b62060b --- /dev/null +++ b/17/assets/js/3e3393a3.a0490608.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4881],{3905:(e,t,r)=>{r.d(t,{Zo:()=>c,kt:()=>m});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function o(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?i(Object(r),!0).forEach((function(t){a(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):i(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function l(e,t){if(null==e)return{};var r,n,a=function(e,t){if(null==e)return{};var r,n,a={},i=Object.keys(e);for(n=0;n<i.length;n++)r=i[n],t.indexOf(r)>=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n<i.length;n++)r=i[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var s=n.createContext({}),u=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):o(o({},t),e)),r},c=function(e){var t=u(e.components);return n.createElement(s.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,i=e.originalType,s=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),p=u(r),d=a,m=p["".concat(s,".").concat(d)]||p[d]||f[d]||i;return r?n.createElement(m,o(o({ref:t},c),{},{components:r})):n.createElement(m,o({ref:t},c))}));function m(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=r.length,o=new Array(i);o[0]=d;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[p]="string"==typeof e?e:a,o[1]=l;for(var u=2;u<i;u++)o[u]=r[u];return n.createElement.apply(null,o)}return n.createElement.apply(null,r)}d.displayName="MDXCreateElement"},211:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>f,frontMatter:()=>i,metadata:()=>l,toc:()=>u});var n=r(7462),a=(r(7294),r(3905));const i={},o="Child Faults After Write",l={unversionedId:"Lab/Compute/quiz/child-faults-after-write",id:"Lab/Compute/quiz/child-faults-after-write",title:"Child Faults After Write",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/child-faults-after-write.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/child-faults-after-write",permalink:"/operating-systems/17/Lab/Compute/quiz/child-faults-after-write",draft:!1,tags:[],version:"current",frontMatter:{}},s={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2}],c={toc:u},p="wrapper";function f(e){let{components:t,...r}=e;return(0,a.kt)(p,(0,n.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"child-faults-after-write"},"Child Faults After Write"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What causes the page faults registered by the child after the fifth step?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The child writes data to the frames it previously shared with its parent and the copy-on-write mechanism copies and remaps them before writing said data")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Demand paging propagates the lazy allocation of pages from the parent to the child")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Creating the child process inherently duplicates some frames")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"They are caused by the loader forking itself when creating the child process")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"They are caused by the ",(0,a.kt)("inlineCode",{parentName:"p"},"bash")," process forking itself when creating the child process"))))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/3e478533.8d22d392.js b/17/assets/js/3e478533.8d22d392.js new file mode 100644 index 0000000000..bb52aebc88 --- /dev/null +++ b/17/assets/js/3e478533.8d22d392.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7327],{3905:(e,t,n)=>{n.d(t,{Zo:()=>m,kt:()=>_});var s=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);t&&(s=s.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,s)}return n}function r(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function l(e,t){if(null==e)return{};var n,s,a=function(e,t){if(null==e)return{};var n,s,a={},i=Object.keys(e);for(s=0;s<i.length;s++)n=i[s],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(s=0;s<i.length;s++)n=i[s],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var p=s.createContext({}),o=function(e){var t=s.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):r(r({},t),e)),n},m=function(e){var t=o(e.components);return s.createElement(p.Provider,{value:t},e.children)},d="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return s.createElement(s.Fragment,{},t)}},u=s.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,p=e.parentName,m=l(e,["components","mdxType","originalType","parentName"]),d=o(n),u=a,_=d["".concat(p,".").concat(u)]||d[u]||c[u]||i;return n?s.createElement(_,r(r({ref:t},m),{},{components:n})):s.createElement(_,r({ref:t},m))}));function _(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,r=new Array(i);r[0]=u;var l={};for(var p in t)hasOwnProperty.call(t,p)&&(l[p]=t[p]);l.originalType=e,l[d]="string"==typeof e?e:a,r[1]=l;for(var o=2;o<i;o++)r[o]=n[o];return s.createElement.apply(null,r)}return s.createElement.apply(null,n)}u.displayName="MDXCreateElement"},6370:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>r,default:()=>c,frontMatter:()=>i,metadata:()=>l,toc:()=>o});var s=n(7462),a=(n(7294),n(3905));const i={},r="Mini-libc",l={unversionedId:"Assignments/Mini Libc/README",id:"Assignments/Mini Libc/README",title:"Mini-libc",description:"Objectives",source:"@site/docs/Assignments/Mini Libc/README.md",sourceDirName:"Assignments/Mini Libc",slug:"/Assignments/Mini Libc/",permalink:"/operating-systems/17/Assignments/Mini Libc/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Assignments",permalink:"/operating-systems/17/Assignments/"},next:{title:"Memory Allocator",permalink:"/operating-systems/17/Assignments/Memory Allocator/"}},p={},o=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Support Code",id:"support-code",level:2},{value:"API and Implementation Tasks",id:"api-and-implementation-tasks",level:3},{value:"Building mini-libc",id:"building-mini-libc",level:3},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Running the Checker",id:"running-the-checker",level:3},{value:"Running the Linters",id:"running-the-linters",level:3},{value:"Behind the Scenes",id:"behind-the-scenes",level:3},{value:"Resources",id:"resources",level:2}],m={toc:o},d="wrapper";function c(e){let{components:t,...n}=e;return(0,a.kt)(d,(0,s.Z)({},m,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"mini-libc"},"Mini-libc"),(0,a.kt)("h2",{id:"objectives"},"Objectives"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Learn about the structure and functionalities provided by the standard C library"),(0,a.kt)("li",{parentName:"ul"},"Accommodate with the syscall interface in Linux"),(0,a.kt)("li",{parentName:"ul"},"Gain a better understanding of strings and memory management functions"),(0,a.kt)("li",{parentName:"ul"},"Learn how the standard C library provides support for low-level input/output operations")),(0,a.kt)("h2",{id:"statement"},"Statement"),(0,a.kt)("p",null,"Build a ",(0,a.kt)("strong",{parentName:"p"},"minimalistic")," ",(0,a.kt)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/C_standard_library"},(0,a.kt)("strong",{parentName:"a"},"standard C library"))," implementation for Linux systems (named ",(0,a.kt)("strong",{parentName:"p"},"mini-libc"),"), that can be used as a replacement for the ",(0,a.kt)("strong",{parentName:"p"},"system libc")," (",(0,a.kt)("a",{parentName:"p",href:"https://www.gnu.org/software/libc/"},"glibc")," in Linux).\nThe goal is to have a minimally functional libc with features such as string management, basic memory support and POSIX file I/O."),(0,a.kt)("p",null,"The implementation of mini-libc will be ",(0,a.kt)("strong",{parentName:"p"},"freestanding"),", i.e. it will not use any outside library calls.\nIt will be implemented on top of the system call interface provided by Linux on an ",(0,a.kt)("inlineCode",{parentName:"p"},"x86_64")," architecture.\nAny function you require, that is typically part of libc, you will have to implement.\nYou can reuse functions that you implement in other parts of the mini-libc."),(0,a.kt)("p",null,"In case you are using a macOS device with ARM64 / Aarch64, you will have to install an ",(0,a.kt)("inlineCode",{parentName:"p"},"x86_64")," virtual machine."),(0,a.kt)("h2",{id:"support-code"},"Support Code"),(0,a.kt)("p",null,"The support code consists of three directories:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"src/")," is the skeleton mini-libc implementation.\nYou will have to implement missing parts marked as ",(0,a.kt)("inlineCode",{parentName:"p"},"TODO")," items.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"samples/")," stores use cases and tests of mini-libc.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"tests/")," are tests used to validate (and grade) the assignment."))),(0,a.kt)("p",null,"System call invocation is done via the ",(0,a.kt)("inlineCode",{parentName:"p"},"syscall()")," function defined in ",(0,a.kt)("inlineCode",{parentName:"p"},"src/syscall.c"),".\nThat itself makes a call to the architecture-specific call in ",(0,a.kt)("inlineCode",{parentName:"p"},"src/internal/arch/x86_64/syscall_arch.h"),";\nhence the dependency on the ",(0,a.kt)("inlineCode",{parentName:"p"},"x86_64")," architecture."),(0,a.kt)("h3",{id:"api-and-implementation-tasks"},"API and Implementation Tasks"),(0,a.kt)("p",null,"The application programming interface (API) of the C standard library is declared in a number of header files.\nEach header file contains one or more function declarations, data type definitions and macros.\nFor your minimal implementation, the following header files are of interest:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"<string.h>"),": defines string-handling functions"),(0,a.kt)("p",{parentName:"li"},"For this assignment, you will have to implement the following functions: ",(0,a.kt)("inlineCode",{parentName:"p"},"strcpy()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strcat()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strlen()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strncpy()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strncat()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strcmp()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strncmp()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strstr()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"strrstr()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"memcpy()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"memset()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"memmove()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"memcmp()"),".")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"<stdio.h>"),": defines printing and I/O functions"),(0,a.kt)("p",{parentName:"li"},"For this assignment, you will have to implement ",(0,a.kt)("inlineCode",{parentName:"p"},"puts()"),".")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"<unistd.h>"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"<sys/fcntl.h>")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"<sys/stat.h>"),": define I/O primitives"),(0,a.kt)("p",{parentName:"li"},"For this assignment, you will have to implement the following functions: ",(0,a.kt)("inlineCode",{parentName:"p"},"open()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"close()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"lseek()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"stat()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"fstat()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"truncate()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"ftruncate()"),"."),(0,a.kt)("p",{parentName:"li"},"You will also have to implement the ",(0,a.kt)("inlineCode",{parentName:"p"},"nanosleep()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sleep()")," functions.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"<stdlib.h"),"> and ",(0,a.kt)("inlineCode",{parentName:"p"},"<sys/mman.h>")," define memory allocation functions"),(0,a.kt)("p",{parentName:"li"},"For this assignment, you will have to implement the following functions: ",(0,a.kt)("inlineCode",{parentName:"p"},"malloc()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"free()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"calloc()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"realloc()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"realloc_array()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"mmap()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"mremap()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"munmap()"),"."),(0,a.kt)("p",{parentName:"li"},"For managing memory areas, a basic list structure is provided in ",(0,a.kt)("inlineCode",{parentName:"p"},"include/internal/mm/mem_list.h")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"mm/mem_list.c"),".")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"<errno.h>")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"errno.c"),": declare and define the integer variable ",(0,a.kt)("inlineCode",{parentName:"p"},"errno"),", which is set by system calls and some library functions in the event of an error to indicate what went wrong."))),(0,a.kt)("p",null,"Some tests do not build.\nThis is intentional.\nYou will have to add the missing features to make those tests compile, that is"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the ",(0,a.kt)("inlineCode",{parentName:"p"},"time.h")," header")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the declaration and the implementation of ",(0,a.kt)("inlineCode",{parentName:"p"},"puts()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the declaration and the implementation of ",(0,a.kt)("inlineCode",{parentName:"p"},"nanosleep()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sleep()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the update of the libc ",(0,a.kt)("inlineCode",{parentName:"p"},"Makefile")," to build the source code files implementing ",(0,a.kt)("inlineCode",{parentName:"p"},"puts()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"nanosleep()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"sleep()")),(0,a.kt)("p",{parentName:"li"},"\u2757\u2757 ",(0,a.kt)("strong",{parentName:"p"},"Pay attention")," to which functions have to modify the ",(0,a.kt)("inlineCode",{parentName:"p"},"errno")," variable."))),(0,a.kt)("h3",{id:"building-mini-libc"},"Building mini-libc"),(0,a.kt)("p",null,"To build mini-libc, run ",(0,a.kt)("inlineCode",{parentName:"p"},"make")," in the ",(0,a.kt)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/mini-libc$ cd src/\n\nstudent@so:~/.../assignments/mini-libc/src$ make\n")),(0,a.kt)("p",null,"To build samples, enter the ",(0,a.kt)("inlineCode",{parentName:"p"},"samples")," directory and run ",(0,a.kt)("inlineCode",{parentName:"p"},"make"),":"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../content/assignments/mini-libc$ cd samples/\n\nstudent@so:~/.../assignments/mini-libc/samples$ make\n")),(0,a.kt)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,a.kt)("p",null,"The testing is automated.\nTests are located in the ",(0,a.kt)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ ls -F\nMakefile graded_test.inc.sh run_all_tests.sh* test_io_file_create.sh* test_malloc_free.sh* test_memory.c test_mmap_perm_notok.sh* test_nanosleep.sh* test_stat.sh*\ngrade.sh* io/ test_fstat.sh* test_io_file_delete.sh* test_malloc_free_sequence.sh* test_mmap.sh* test_mmap_perm_ok.sh* test_open_close.sh* test_string.c\ngraded_test.c memory/ test_ftruncate.sh* test_lseek.sh* test_malloc_perm_notok.sh* test_mmap_munmap.sh* test_multiple_malloc.sh* test_puts.sh* test_truncate.sh*\ngraded_test.h process/ test_io.c test_malloc.sh* test_malloc_perm_ok.sh* test_mmap_perm_none.sh* test_multiple_malloc_free.sh* test_sleep.sh*\n")),(0,a.kt)("p",null,"To test and grade your assignment solution, enter the ",(0,a.kt)("inlineCode",{parentName:"p"},"tests/")," directory and run ",(0,a.kt)("inlineCode",{parentName:"p"},"grade.sh"),".\nNote that this requires linters being available.\nThe easiest is to use a Docker-based setup with everything installed, as shown in the section ",(0,a.kt)("a",{parentName:"p",href:"#running-the-linters"},'"Running the Linters"'),".\nWhen using ",(0,a.kt)("inlineCode",{parentName:"p"},"grade.sh")," you will get grades for checking correctness (maximum ",(0,a.kt)("inlineCode",{parentName:"p"},"90")," points) and for coding style (maxim ",(0,a.kt)("inlineCode",{parentName:"p"},"10")," points).\nA successful run will provide you an output ending with:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"### GRADE\n\n\nChecker: 90/ 90\nStyle: 10/ 10\nTotal: 100/100\n\n\n### STYLE SUMMARY\n\n\n")),(0,a.kt)("h3",{id:"running-the-checker"},"Running the Checker"),(0,a.kt)("p",null,"To run only the checker, use the ",(0,a.kt)("inlineCode",{parentName:"p"},"make check")," command in the ",(0,a.kt)("inlineCode",{parentName:"p"},"tests/")," directory:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ make check\nmake[1]: Entering directory '...'\nrm -f *~\n[...]\ntest_mmap_perm_ok ........................ failed ... 0\ntest_mmap_perm_notok ........................ failed ... 0\ntest_mmap_perm_none ........................ failed ... 0\n\nTotal: 0/100\n")),(0,a.kt)("p",null,"Some files will fail to build, it's expected.\nThis is because there are missing files or missing functions that cause build errors.\nYou'll need to add those files and implement those functions for the build error to disappear."),(0,a.kt)("p",null,"Obviously, most tests will fail, as there is no implementation.\nSome tests don't fail because the missing implementation equates to the bad behavior being tested not happening."),(0,a.kt)("p",null,"Each test is worth a number of points.\nThe total number of points is ",(0,a.kt)("inlineCode",{parentName:"p"},"900"),".\nThe maximum grade is obtained by dividing the number of points to ",(0,a.kt)("inlineCode",{parentName:"p"},"10"),", for a maximum grade of ",(0,a.kt)("inlineCode",{parentName:"p"},"90"),"."),(0,a.kt)("p",null,"A successful run will show the output:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ make check\n[...]\ntest_strcpy ........................ passed ... 9\ntest_strcpy_append ........................ passed ... 9\ntest_strncpy ........................ passed ... 9\ntest_strncpy_cut ........................ passed ... 9\ntest_strcat ........................ passed ... 9\ntest_strcat_from_zero ........................ passed ... 9\ntest_strcat_multiple ........................ passed ... 9\ntest_strncat ........................ passed ... 9\ntest_strncat_cut ........................ passed ... 9\ntest_strcmp_equal ........................ passed ... 9\ntest_strcmp_same_size_less ........................ passed ... 1\ntest_strcmp_same_size_greater ........................ passed ... 9\ntest_strcmp_diff_size_less ........................ passed ... 1\ntest_strcmp_diff_size_greater ........................ passed ... 9\ntest_strncmp_equal_size_equal ........................ passed ... 9\ntest_strncmp_diff_contents_equal ........................ passed ... 9\ntest_strncmp_diff_size_equal ........................ passed ... 9\ntest_strchr_exists ........................ passed ... 11\ntest_strchr_exists_twice ........................ passed ... 9\ntest_strchr_not_exists ........................ passed ... 1\ntest_strrchr_exists ........................ passed ... 11\ntest_strrchr_exists_twice ........................ passed ... 9\ntest_strrchr_not_exists ........................ passed ... 1\ntest_strstr_exists ........................ passed ... 11\ntest_strstr_exists_twice ........................ passed ... 9\ntest_strstr_not_exists ........................ passed ... 1\ntest_strrstr_exists ........................ passed ... 11\ntest_strrstr_exists_twice ........................ passed ... 9\ntest_strrstr_not_exists ........................ passed ... 1\ntest_memcpy ........................ passed ... 11\ntest_memcpy_part ........................ passed ... 9\ntest_memcmp_equal_size_equal ........................ passed ... 9\ntest_memcmp_diff_contents_equal ........................ passed ... 9\ntest_memcmp_diff_size_equal ........................ passed ... 9\ntest_memset ........................ passed ... 9\ntest_memset_part ........................ passed ... 9\ntest_memmove_apart ........................ passed ... 9\ntest_memmove_src_before_dst ........................ passed ... 9\ntest_memmove_src_after_dst ........................ passed ... 9\ntest_open_non_existent_file ........................ passed ... 8\ntest_open_invalid_access_mode ........................ passed ... 8\ntest_open_file_as_directory ........................ passed ... 8\ntest_open_directory_for_writing ........................ passed ... 8\ntest_open_force_invalid_creation ........................ passed ... 8\ntest_open_close_existent_file ........................ passed ... 8\ntest_open_close_create_file ........................ passed ... 8\ntest_open_read_write_only_mode ........................ passed ... 8\ntest_open_write_read_only_mode ........................ passed ... 8\ntest_lseek_invalid_fd ........................ passed ... 8\ntest_lseek_invalid_whence ........................ passed ... 8\ntest_lseek_invalid_offset ........................ passed ... 8\ntest_lseek_set ........................ passed ... 8\ntest_lseek_cur ........................ passed ... 8\ntest_lseek_end ........................ passed ... 8\ntest_lseek_combined ........................ passed ... 8\ntest_truncate_read_only_file ........................ passed ... 8\ntest_truncate_invalid_size ........................ passed ... 8\ntest_truncate_directory ........................ passed ... 8\ntest_truncate_non_existent_file ........................ passed ... 8\ntest_truncate_file ........................ passed ... 8\ntest_ftruncate_read_only_file ........................ passed ... 8\ntest_ftruncate_invalid_size ........................ passed ... 8\ntest_ftruncate_directory ........................ passed ... 8\ntest_ftruncate_bad_fd ........................ passed ... 8\ntest_ftruncate_file ........................ passed ... 8\ntest_stat_non_existent_file ........................ passed ... 8\ntest_stat_regular_file ........................ passed ... 8\ntest_fstat_bad_fd ........................ passed ... 8\ntest_fstat_regular_file ........................ passed ... 8\ntest_puts ........................ passed ... 15\ntest_open_close_create_file ........................ passed ... 10\ntest_open_close_read_byte ........................ passed ... 10\ntest_ftruncate ........................ passed ... 10\ntest_truncate ........................ passed ... 10\ntest_fstat ........................ passed ... 10\ntest_stat ........................ passed ... 10\ntest_sleep ........................ passed ... 20\ntest_nanosleep ........................ passed ... 20\ntest_mmap ........................ passed ... 8\ntest_mmap_bad_fd ........................ passed ... 8\ntest_mmap_bad_flags ........................ passed ... 8\ntest_mremap ........................ passed ... 8\ntest_malloc ........................ passed ... 8\ntest_malloc_two ........................ passed ... 8\ntest_malloc_access ........................ passed ... 8\ntest_malloc_memset ........................ passed ... 8\ntest_malloc_memcpy ........................ passed ... 8\ntest_calloc ........................ passed ... 8\ntest_realloc ........................ passed ... 8\ntest_realloc_access ........................ passed ... 8\ntest_realloc_memset ........................ passed ... 8\ntest_realloc_array ........................ passed ... 8\ntest_malloc ........................ passed ... 10\ntest_multiple_malloc ........................ passed ... 10\ntest_malloc_free ........................ passed ... 10\ntest_multiple_malloc_free ........................ passed ... 10\ntest_malloc_free_sequence ........................ passed ... 10\ntest_malloc_perm_ok ........................ passed ... 10\ntest_malloc_perm_notok ........................ passed ... 10\ntest_mmap ........................ passed ... 10\ntest_mmap_munmap ........................ passed ... 10\ntest_mmap_perm_ok ........................ passed ... 10\ntest_mmap_perm_notok ........................ passed ... 10\ntest_mmap_perm_none ........................ passed ... 10\n\nTotal: 90/100\n")),(0,a.kt)("h3",{id:"running-the-linters"},"Running the Linters"),(0,a.kt)("p",null,"To run the linters, use the ",(0,a.kt)("inlineCode",{parentName:"p"},"make linter")," command in the ",(0,a.kt)("inlineCode",{parentName:"p"},"tests/")," directory.\nNote that the linters have to be installed on your system: ",(0,a.kt)("a",{parentName:"p",href:"https://.com/torvalds/linux/blob/master/scripts/checkpatch.pl"},(0,a.kt)("inlineCode",{parentName:"a"},"checkpatch.pl")),", ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/cpplint/cpplint"},(0,a.kt)("inlineCode",{parentName:"a"},"cpplint")),", ",(0,a.kt)("a",{parentName:"p",href:"https://www.shellcheck.net/"},(0,a.kt)("inlineCode",{parentName:"a"},"shellcheck"))," with certain configuration options.\nIt's easiest to run them in a Docker-based setup with everything configured:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ make lint\n[...]\ncd .. && checkpatch.pl -f checker/*.sh tests/*.sh\n[...]\ncd .. && cpplint --recursive src/ tests/ checker/\n[...]\ncd .. && shellcheck checker/*.sh tests/*.sh\n")),(0,a.kt)("h3",{id:"behind-the-scenes"},"Behind the Scenes"),(0,a.kt)("p",null,"For a fine grained approach, build tests and ignore errors (due to missing source code and header files) by using:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ make -i\n")),(0,a.kt)("p",null,"Then run the tests, either individually via executable files and scripts:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ ./test_lseek.sh\ntest_lseek ........................ passed ... 10\n\nstudent@so:~/.../assignments/mini-libc/tests$ ./test_memory\ntest_mmap ........................ passed ... 8\ntest_mmap_bad_fd ........................ passed ... 8\n[...]\n")),(0,a.kt)("p",null,"Or run them all via the ",(0,a.kt)("inlineCode",{parentName:"p"},"run_all_tests.sh")," script:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~/.../assignments/mini-libc/tests$ ./run_all_tests.sh\ntest_strcpy ........................ passed ... 9\ntest_strcpy_append ........................ passed ... 9\ntest_strncpy ........................ passed ... 9\n[...]\n")),(0,a.kt)("h2",{id:"resources"},"Resources"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("a",{parentName:"p",href:"https://www.gnu.org/software/libc/manual/html_mono/libc.html"},"GNU libc manual"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/latest/source"},"musl implementation of the standard C library for Linux-based systems"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Syscall interface in Linux - ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/index.html"},"Linux man pages online"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/glibc/latest/source"},"glibc implementation of the standard C library for Linux-based systems")))))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/432299a8.8dd11b20.js b/17/assets/js/432299a8.8dd11b20.js new file mode 100644 index 0000000000..5831bd8dc9 --- /dev/null +++ b/17/assets/js/432299a8.8dd11b20.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9859],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>f});var a=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function r(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function o(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?r(Object(n),!0).forEach((function(t){i(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):r(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function l(e,t){if(null==e)return{};var n,a,i=function(e,t){if(null==e)return{};var n,a,i={},r=Object.keys(e);for(a=0;a<r.length;a++)n=r[a],t.indexOf(n)>=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a<r.length;a++)n=r[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var s=a.createContext({}),c=function(e){var t=a.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},p=function(e){var t=c(e.components);return a.createElement(s.Provider,{value:t},e.children)},m="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var n=e.components,i=e.mdxType,r=e.originalType,s=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),m=c(n),u=i,f=m["".concat(s,".").concat(u)]||m[u]||d[u]||r;return n?a.createElement(f,o(o({ref:t},p),{},{components:n})):a.createElement(f,o({ref:t},p))}));function f(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var r=n.length,o=new Array(r);o[0]=u;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[m]="string"==typeof e?e:i,o[1]=l;for(var c=2;c<r;c++)o[c]=n[c];return a.createElement.apply(null,o)}return a.createElement.apply(null,n)}u.displayName="MDXCreateElement"},7939:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>d,frontMatter:()=>r,metadata:()=>l,toc:()=>c});var a=n(7462),i=(n(7294),n(3905));const r={},o="Libraries and libc",l={unversionedId:"Lab/Software Stack/libc",id:"Lab/Software Stack/libc",title:"Libraries and libc",description:"Once we have common functions implemented, we can reuse them at any time.",source:"@site/docs/Lab/Software Stack/libc.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/libc",permalink:"/operating-systems/17/Lab/Software Stack/libc",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Common Functions",permalink:"/operating-systems/17/Lab/Software Stack/common-functions"},next:{title:"Statically-linked and Dynamically-linked Libraries",permalink:"/operating-systems/17/Lab/Software Stack/static-dynamic"}},s={},c=[{value:"Practice",id:"practice",level:2}],p={toc:c},m="wrapper";function d(e){let{components:t,...n}=e;return(0,i.kt)(m,(0,a.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"libraries-and-libc"},"Libraries and libc"),(0,i.kt)("p",null,"Once we have common functions implemented, we can reuse them at any time.\nThe main unit for software reusability is the ",(0,i.kt)("strong",{parentName:"p"},"library"),".\nIn short, a library is a common machine code that can be linked against different other software components.\nEach time we want to use the ",(0,i.kt)("inlineCode",{parentName:"p"},"printf()")," function or the ",(0,i.kt)("inlineCode",{parentName:"p"},"strlen()")," function, we don't need to reimplement them.\nWe also don't need to use existing source code files, rebuild them and reuse them.\nWe (re)use existing machine code in libraries."),(0,i.kt)("p",null,"A library is a collection of object files that export given data structures and functions to be used by other programs.\nWe create a program, we compile and then we link it against the library for all the features it provides."),(0,i.kt)("p",null,"The most important library in modern operating systems is the ",(0,i.kt)("strong",{parentName:"p"},"standard C library"),", also called ",(0,i.kt)("strong",{parentName:"p"},"libc"),".\nThis is the library providing system call wrappers and basic functionality for input-output, string management, memory management.\nBy default, a program is always linked with the standard C library.\nIn the examples above, we've explicitly disabled the use of the standard C library with the help of the ",(0,i.kt)("inlineCode",{parentName:"p"},"-nostdlib")," linker option."),(0,i.kt)("p",null,"By using the standard C library, it's much easier to create new programs.\nYou call existing functionality in the library and implement only features particular to your program."),(0,i.kt)("p",null,"The ",(0,i.kt)("inlineCode",{parentName:"p"},"support/libc/")," folder stores the implementation of programs using the standard C library: ",(0,i.kt)("inlineCode",{parentName:"p"},"hello.c"),", ",(0,i.kt)("inlineCode",{parentName:"p"},"main_string.c")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"main_printf.c"),".\nThese programs are almost identical to those used in the past sections:"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"hello.c")," is similar to the programs in ",(0,i.kt)("inlineCode",{parentName:"li"},"solution/basic-syscall/")," and ",(0,i.kt)("inlineCode",{parentName:"li"},"solution/syscall-wrapper/")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"main_string.c")," and ",(0,i.kt)("inlineCode",{parentName:"li"},"main_printf.c")," are similar to the programs in ",(0,i.kt)("inlineCode",{parentName:"li"},"solution/common-functions/"))),(0,i.kt)("p",null,"Let's build and run them:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/libc$ ls\nhello hello.c hello.o main_printf main_printf.c main_printf.o main_string main_string.c main_string.o Makefile\n\nstudent@os:~/.../lab/support/libc$ make clean\nrm -f hello hello.o\nrm -f main_printf main_printf.o\nrm -f main_string main_string.o\n\nstudent@os:~/.../lab/support/libc$ ls\nhello.c main_printf.c main_string.c Makefile\n\nstudent@os:~/.../lab/support/libc$ make\ncc -Wall -c -o hello.o hello.c\ncc -static hello.o -o hello\ncc -Wall -c -o main_printf.o main_printf.c\ncc -static main_printf.o -o main_printf\ncc -Wall -c -o main_string.o main_string.c\ncc -static main_string.o -o main_string\n\nstudent@os:~/.../lab/support/libc$ ls\nhello hello.c hello.o main_printf main_printf.c main_printf.o main_string main_string.c main_string.o Makefile\n\nstudent@os:~/.../lab/support/libc$ ./hello\nHello, world!\nBye, world!\naaa\naaa\n^C\n\nstudent@os:~/.../lab/support/libc$ ./main_string\nDestination string is: warhammer40k\n\nstudent@os:~/.../lab/support/libc$ ./main_printf\n[before] src is at 0x492308, len is 12, content: "warhammer40k"\n[before] dest is at 0x6bb340, len is 0, content: ""\ncopying src to dest\n[after] src is at 0x492308, len is 12, content: "warhammer40k"\n[after] dest is at 0x6bb340, len is 12, content: "warhammer40k"\nabc\n')),(0,i.kt)("p",null,"The behavior / output is similar to the ones in the previous sections:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/libc$ ../../solution/basic-syscall/hello-nasm\nHello, world!\nBye, world!\naaa\naaa\n^C\n\nstudent@os:~/.../lab/support/libc$ ../../solution/common-functions/main_string\nDestination string is: warhammer40k\n\nstudent@os:~/.../lab/support/libc$ ../../solution/common-functions/main_printf\n[before] src is at 0000000000402680, len is 12, content: "warhammer40k"\n[before] dest is at 0000000000604000, len is 0, content: ""\ncopying src to dest\n[after] src is at 0000000000402680, len is 12, content: "warhammer40k"\n[after] dest is at 0000000000604000, len is 12, content: "warhammer40k"\nabc\n')),(0,i.kt)("p",null,"We can inspect the system calls made to check the similarities.\nFor example, for the ",(0,i.kt)("inlineCode",{parentName:"p"},"main_printf")," program we get the outputs:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/libc$ strace ./main_printf\nexecve("./main_printf", ["./main_printf"], 0x7fff7b38c240 /* 66 vars */) = 0\nbrk(NULL) = 0x15af000\nbrk(0x15b01c0) = 0x15b01c0\narch_prctl(ARCH_SET_FS, 0x15af880) = 0\nuname({sysname="Linux", nodename="[...]", ...}) = 0\nreadlink("/proc/self/exe", "[...]/operating"..., 4096) = 105\nbrk(0x15d11c0) = 0x15d11c0\nbrk(0x15d2000) = 0x15d2000\naccess("/etc/ld.so.nohwcap", F_OK) = -1 ENOENT (No such file or directory)\nfstat(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 18), ...}) = 0\nwrite(1, "[before] src is at 0x492308, len"..., 64[before] src is at 0x492308, len is 12, content: "warhammer40k"\n) = 64\nwrite(1, "[before] dest is at 0x6bb340, le"..., 52[before] dest is at 0x6bb340, len is 0, content: ""\n) = 52\nwrite(1, "copying src to dest\\n", 20copying src to dest\n) = 20\nwrite(1, "[after] src is at 0x492308, len "..., 63[after] src is at 0x492308, len is 12, content: "warhammer40k"\n) = 63\nwrite(1, "[after] dest is at 0x6bb340, len"..., 64[after] dest is at 0x6bb340, len is 12, content: "warhammer40k"\n) = 64\nwrite(1, "ab", 2ab) = 2\nwrite(1, "c\\n", 2c\n) = 2\nexit_group(0) = ?\n+++ exited with 0 +++\n\nstudent@os:~/.../lab/support/libc$ strace ../../solution/common-functions/main_printf\nexecve("../../solution/common-functions/main_printf", ["../../solution/common-functions/"...], 0x7ffe204eec00 /* 66 vars */) = 0\nwrite(1, "[before] src is at 0000000000402"..., 72[before] src is at 0000000000402680, len is 12, content: "warhammer40k"\n) = 72\nwrite(1, "[before] dest is at 000000000060"..., 60[before] dest is at 0000000000604000, len is 0, content: ""\n) = 60\nwrite(1, "copying src to dest\\n", 20copying src to dest\n) = 20\nwrite(1, "[after] src is at 00000000004026"..., 71[after] src is at 0000000000402680, len is 12, content: "warhammer40k"\n) = 71\nwrite(1, "[after] dest is at 0000000000604"..., 72[after] dest is at 0000000000604000, len is 12, content: "warhammer40k"\n) = 72\nwrite(1, "ab", 2ab) = 2\nwrite(1, "c\\n", 2c\n) = 2\nexit(0) = ?\n+++ exited with 0 +++\n')),(0,i.kt)("p",null,"The output is similar, with differences at the beginning and the end of the system call trace.\nIn the case of the libc-built program, a series of additional system calls (",(0,i.kt)("inlineCode",{parentName:"p"},"brk"),", ",(0,i.kt)("inlineCode",{parentName:"p"},"arch_prctl"),", ",(0,i.kt)("inlineCode",{parentName:"p"},"uname")," etc.) are made.\nAlso, there is an implicit call to ",(0,i.kt)("inlineCode",{parentName:"p"},"exit_group")," instead of an explicit one to ",(0,i.kt)("inlineCode",{parentName:"p"},"exit")," in the non-libc case.\nThese are initialization and cleanup routines that are implicitly added when using the standard C library.\nThey are generally used for setting and cleaning up the stack, environment variables and other pieces of information required by the program or the standard C library itself."),(0,i.kt)("p",null,"We could argue that the initialization steps incur overhead, and that's a downside of using the standard C library.\nHowever, these initialization steps are required for almost all programs.\nAnd, given that almost all programs make use of the basic features of the standard C library, libc is almost always used.\nWe can say the above were exceptions to the rule, where we didn't make use of the standard C library."),(0,i.kt)("p",null,"Summarizing, the advantages and disadvantages of using the standard C library are:"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"(+) easier development: do calls to existing functions already implemented in the standard C library;\ndefault build and link flags"),(0,i.kt)("li",{parentName:"ul"},"(+) portability: if the system provides a standard C library, one calls the library functions that will then interact with the lower-layer API"),(0,i.kt)("li",{parentName:"ul"},"(+) implicit initialization and cleanup: no need for you do explicitly create them"),(0,i.kt)("li",{parentName:"ul"},"(-) usually larger in size (static) executables"),(0,i.kt)("li",{parentName:"ul"},"(-) a level of overhead as the standard C library wraps system calls"),(0,i.kt)("li",{parentName:"ul"},"(-) potential security issues: a larger set of (potentially vulnerable) functions are presented by the standard C library")),(0,i.kt)("h2",{id:"practice"},"Practice"),(0,i.kt)("p",null,"Enter the ",(0,i.kt)("inlineCode",{parentName:"p"},"support/libc/")," folder and go through the practice items below."),(0,i.kt)("ol",null,(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("p",{parentName:"li"},"Use ",(0,i.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"free()")," functions in the ",(0,i.kt)("inlineCode",{parentName:"p"},"memory.c")," program.\nMake your own use of the allocated memory."),(0,i.kt)("p",{parentName:"li"},"It's very easy to use memory management functions with the libc.\nThe alternative (without the libc) would be more cumbersome."),(0,i.kt)("p",{parentName:"li"},"Use different values for ",(0,i.kt)("inlineCode",{parentName:"p"},"malloc()"),", i.e. the allocation size.\nUse ",(0,i.kt)("inlineCode",{parentName:"p"},"strace")," to check the system calls invoked by ",(0,i.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,i.kt)("inlineCode",{parentName:"p"},"free()"),".\nYou'll see that, depending on the size, the ",(0,i.kt)("inlineCode",{parentName:"p"},"brk")," or ",(0,i.kt)("inlineCode",{parentName:"p"},"mmap")," / ",(0,i.kt)("inlineCode",{parentName:"p"},"munmap")," system calls are invoked.\nAnd for certain calls to ",(0,i.kt)("inlineCode",{parentName:"p"},"malloc()")," / ",(0,i.kt)("inlineCode",{parentName:"p"},"free()")," no syscall is happening.\nYou'll find more about them in the ",(0,i.kt)("a",{parentName:"p",href:"../../../data/lab"},"Data chapter"),".")),(0,i.kt)("li",{parentName:"ol"},(0,i.kt)("p",{parentName:"li"},"Create your own C program with calls to the standard C library in ",(0,i.kt)("inlineCode",{parentName:"p"},"vendetta.c"),".\nBe as creative as you can about the types of functions being made."))),(0,i.kt)("p",null,(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Software%20Stack/quiz/libc"},"Quiz")))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/463aadd7.d12faab0.js b/17/assets/js/463aadd7.d12faab0.js new file mode 100644 index 0000000000..29130d4635 --- /dev/null +++ b/17/assets/js/463aadd7.d12faab0.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[13],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>d});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function l(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){r(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function o(e,t){if(null==e)return{};var n,a,r=function(e,t){if(null==e)return{};var n,a,r={},i=Object.keys(e);for(a=0;a<i.length;a++)n=i[a],t.indexOf(n)>=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a<i.length;a++)n=i[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var s=a.createContext({}),p=function(e){var t=a.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},u=function(e){var t=p(e.components);return a.createElement(s.Provider,{value:t},e.children)},h="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,s=e.parentName,u=o(e,["components","mdxType","originalType","parentName"]),h=p(n),m=r,d=h["".concat(s,".").concat(m)]||h[m]||c[m]||i;return n?a.createElement(d,l(l({ref:t},u),{},{components:n})):a.createElement(d,l({ref:t},u))}));function d(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,l=new Array(i);l[0]=m;var o={};for(var s in t)hasOwnProperty.call(t,s)&&(o[s]=t[s]);o.originalType=e,o[h]="string"==typeof e?e:r,l[1]=o;for(var p=2;p<i;p++)l[p]=n[p];return a.createElement.apply(null,l)}return a.createElement.apply(null,n)}m.displayName="MDXCreateElement"},5611:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>l,default:()=>c,frontMatter:()=>i,metadata:()=>o,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const i={},l="Minishell",o={unversionedId:"Assignments/Mini Shell/README",id:"Assignments/Mini Shell/README",title:"Minishell",description:"Objectives",source:"@site/docs/Assignments/Mini Shell/README.md",sourceDirName:"Assignments/Mini Shell",slug:"/Assignments/Mini Shell/",permalink:"/operating-systems/17/Assignments/Mini Shell/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Parallel Graph",permalink:"/operating-systems/17/Assignments/Parallel Graph/"},next:{title:"Asynchronous Web Server",permalink:"/operating-systems/17/Assignments/Asynchronous Web Server/"}},s={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Introduction",id:"introduction",level:3},{value:"Shell Functionalities",id:"shell-functionalities",level:3},{value:"Changing the Current Directory",id:"changing-the-current-directory",level:4},{value:"Closing the Shell",id:"closing-the-shell",level:4},{value:"Running an Application",id:"running-an-application",level:4},{value:"Environment Variables",id:"environment-variables",level:4},{value:"Operators",id:"operators",level:4},{value:"Sequential Operator",id:"sequential-operator",level:5},{value:"Parallel Operator",id:"parallel-operator",level:5},{value:"Pipe Operator",id:"pipe-operator",level:5},{value:"Chain Operators for Conditional Execution",id:"chain-operators-for-conditional-execution",level:5},{value:"Operator Priority",id:"operator-priority",level:5},{value:"I/O Redirection",id:"io-redirection",level:4},{value:"Testing",id:"testing",level:2},{value:"Debug",id:"debug",level:3},{value:"Memory leaks",id:"memory-leaks",level:3},{value:"Checkpatch",id:"checkpatch",level:3}],u={toc:p},h="wrapper";function c(e){let{components:t,...n}=e;return(0,r.kt)(h,(0,a.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"minishell"},"Minishell"),(0,r.kt)("h2",{id:"objectives"},"Objectives"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"Learn how shells create new child processes and connect the I/O to the terminal."),(0,r.kt)("li",{parentName:"ul"},"Gain a better understanding of the ",(0,r.kt)("inlineCode",{parentName:"li"},"fork()")," function wrapper."),(0,r.kt)("li",{parentName:"ul"},"Learn to correctly execute commands written by the user and treat errors.")),(0,r.kt)("h2",{id:"statement"},"Statement"),(0,r.kt)("h3",{id:"introduction"},"Introduction"),(0,r.kt)("p",null,"A shell is a command-line interpreter that provides a text-based user interface for operating systems.\nBash is both an interactive command language and a scripting language.\nIt is used to interact with the file system, applications, operating system and more."),(0,r.kt)("p",null,"For this assignment you will build a Bash-like shell with minimal functionalities like traversing the file system, running applications, redirecting their output or piping the output from one application into the input of another.\nThe details of the functionalities that must be implemented will be further explained."),(0,r.kt)("h3",{id:"shell-functionalities"},"Shell Functionalities"),(0,r.kt)("h4",{id:"changing-the-current-directory"},"Changing the Current Directory"),(0,r.kt)("p",null,"The shell will support a built-in command for navigating the file system, called ",(0,r.kt)("inlineCode",{parentName:"p"},"cd"),".\nTo implement this feature you will need to store the current directory path because the user can provide either relative or absolute paths as arguments to the ",(0,r.kt)("inlineCode",{parentName:"p"},"cd")," command."),(0,r.kt)("p",null,"The built-in ",(0,r.kt)("inlineCode",{parentName:"p"},"pwd")," command will show the current directory path."),(0,r.kt)("p",null,"Check the following examples below to understand these functionalities."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"> pwd\n/home/student\n> cd operating-systems/assignments/minishell\n> pwd\n/home/student/operating-systems/assignments/minishell\n> cd inexitent\nno such file or directory\n> cd /usr/lib\n> pwd\n/usr/lib\n")),(0,r.kt)("blockquote",null,(0,r.kt)("p",{parentName:"blockquote"},(0,r.kt)("strong",{parentName:"p"},(0,r.kt)("em",{parentName:"strong"},"NOTE:"))," Using the ",(0,r.kt)("inlineCode",{parentName:"p"},"cd")," command without any arguments or with more than one argument doesn't affect the current directory path.\nMake sure this edge case is handled in a way that prevents crashes.")),(0,r.kt)("h4",{id:"closing-the-shell"},"Closing the Shell"),(0,r.kt)("p",null,"Inputting either ",(0,r.kt)("inlineCode",{parentName:"p"},"quit")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"exit")," should close the minishell."),(0,r.kt)("h4",{id:"running-an-application"},"Running an Application"),(0,r.kt)("p",null,"Suppose you have an executable named ",(0,r.kt)("inlineCode",{parentName:"p"},"sum")," in the current directory.\nIt takes arbitrarily many numbers as arguments and prints their sum to ",(0,r.kt)("inlineCode",{parentName:"p"},"stdout"),".\nThe following example shows how the minishell implemented by you should behave."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"> ./sum 2 4 1\n7\n")),(0,r.kt)("p",null,"If the executable is located at the ",(0,r.kt)("inlineCode",{parentName:"p"},"/home/student/sum")," absolute path, the following example should also be valid."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"> /home/student/sum 2 4 1\n7\n")),(0,r.kt)("p",null,"Each application will run in a separate child process of the minishell created using ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/fork.2.html"},"fork"),"."),(0,r.kt)("h4",{id:"environment-variables"},"Environment Variables"),(0,r.kt)("p",null,"Your shell will support using environment variables.\nThe environment variables will be initially inherited from the ",(0,r.kt)("inlineCode",{parentName:"p"},"bash")," process that started your minishell application."),(0,r.kt)("p",null,"If an undefined variable is used, its value is the empty string: ",(0,r.kt)("inlineCode",{parentName:"p"},'""'),"."),(0,r.kt)("blockquote",null,(0,r.kt)("p",{parentName:"blockquote"},(0,r.kt)("strong",{parentName:"p"},(0,r.kt)("em",{parentName:"strong"},"NOTE:"))," The following examples contain comments which don't need to be supported by the minishell.\nThey are present here only to give a better understanding of the minishell's functionalities.")),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},'> NAME="John Doe" # Will assign the value "John Doe" to the NAME variable\n> AGE=27 # Will assign the value 27 to the AGE variable\n> ./identify $NAME $LOCATION $AGE # Will translate to ./identify "John Doe" "" 27 because $LOCATION is not defined\n')),(0,r.kt)("p",null,"A variable can be assigned to another variable."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"> OLD_NAME=$NAME # Will assign the value of the NAME variable to OLD_NAME\n")),(0,r.kt)("h4",{id:"operators"},"Operators"),(0,r.kt)("h5",{id:"sequential-operator"},"Sequential Operator"),(0,r.kt)("p",null,"By using the ",(0,r.kt)("inlineCode",{parentName:"p"},";")," operator, you can chain multiple commands that will run sequentially, one after another.\nIn the command ",(0,r.kt)("inlineCode",{parentName:"p"},"expr1; expr2")," it is guaranteed that ",(0,r.kt)("inlineCode",{parentName:"p"},"expr1")," will finish before ",(0,r.kt)("inlineCode",{parentName:"p"},"expr2")," is be evaluated."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},'> echo "Hello"; echo "world!"; echo "Bye!"\nHello\nworld!\nBye!\n')),(0,r.kt)("h5",{id:"parallel-operator"},"Parallel Operator"),(0,r.kt)("p",null,"By using the ",(0,r.kt)("inlineCode",{parentName:"p"},"&")," operator you can chain multiple commands that will run in parallel.\nWhen running the command ",(0,r.kt)("inlineCode",{parentName:"p"},"expr1 & expr2"),", both expressions are evaluated at the same time (by different processes).\nThe order in which the two commands finish is not guaranteed."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},'> echo "Hello" & echo "world!" & echo "Bye!" # The words may be printed in any order\nworld!\nBye!\nHello\n')),(0,r.kt)("h5",{id:"pipe-operator"},"Pipe Operator"),(0,r.kt)("p",null,"With the ",(0,r.kt)("inlineCode",{parentName:"p"},"|")," operator you can chain multiple commands so that the standard output of the first command is redirected to the standard input of the second command."),(0,r.kt)("p",null,"Hint: Look into ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},"anonymous pipes")," and file descriptor inheritance while using ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/fork.2.html"},"fork"),"."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},'> echo "Bye" # command outputs "Bye"\nBye\n> ./reverse_input\nHello # command reads input "Hello"\nolleH # outputs the reversed string "olleH"\n> echo "world" | ./reverse_input # the output generated by the echo command will be used as input for the reverse_input executable\ndlrow\n')),(0,r.kt)("h5",{id:"chain-operators-for-conditional-execution"},"Chain Operators for Conditional Execution"),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"&&")," operator allows chaining commands that are executed sequentially, from left to right.\nThe chain of execution stops at the first command ",(0,r.kt)("strong",{parentName:"p"},"that exits with an error (return code not 0)"),"."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},'# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"\n> echo "H" && echo "e" && echo "l" && ./throw_error && echo "l" && echo "o"\nH\ne\nl\nERROR: I always fail\n')),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"||")," operator allows chaining commands that are executed sequentially, from left to right.\nThe chain of execution stops at the first command ",(0,r.kt)("strong",{parentName:"p"},"that exits successfully (return code is 0)"),"."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},'# throw_error always exits with a return code different than 0 and outputs to stderr "ERROR: I always fail"\n> ./throw_error || ./throw_error || echo "Hello" || echo "world!" || echo "Bye!"\nERROR: I always fail\nERROR: I always fail\nHello\n')),(0,r.kt)("h5",{id:"operator-priority"},"Operator Priority"),(0,r.kt)("p",null,"The priority of the available operators is the following.\nThe lower the number, the ",(0,r.kt)("strong",{parentName:"p"},"higher")," the priority:"),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},"Pipe operator (",(0,r.kt)("inlineCode",{parentName:"li"},"|"),")"),(0,r.kt)("li",{parentName:"ol"},"Conditional execution operators (",(0,r.kt)("inlineCode",{parentName:"li"},"&&")," or ",(0,r.kt)("inlineCode",{parentName:"li"},"||"),")"),(0,r.kt)("li",{parentName:"ol"},"Parallel operator (",(0,r.kt)("inlineCode",{parentName:"li"},"&"),")"),(0,r.kt)("li",{parentName:"ol"},"Sequential operator (",(0,r.kt)("inlineCode",{parentName:"li"},";"),")")),(0,r.kt)("h4",{id:"io-redirection"},"I/O Redirection"),(0,r.kt)("p",null,"The shell must support the following redirection options:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"< filename")," - redirects ",(0,r.kt)("inlineCode",{parentName:"li"},"filename")," to standard input"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"> filename")," - redirects standard output to ",(0,r.kt)("inlineCode",{parentName:"li"},"filename")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"2> filename")," - redirects standard error to ",(0,r.kt)("inlineCode",{parentName:"li"},"filename")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"&> filename")," - redirects standard output and standard error to ",(0,r.kt)("inlineCode",{parentName:"li"},"filename")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},">> filename")," - redirects standard output to ",(0,r.kt)("inlineCode",{parentName:"li"},"filename")," in append mode"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"2>> filename")," - redirects standard error to ",(0,r.kt)("inlineCode",{parentName:"li"},"filename")," in append mode")),(0,r.kt)("p",null,"Hint: Look into ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},"open"),", ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/dup.2.html"},"dup2")," and ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/close.2.html"},"close"),"."),(0,r.kt)("h2",{id:"testing"},"Testing"),(0,r.kt)("p",null,"The testing is automated.\nTests are located in the ",(0,r.kt)("inlineCode",{parentName:"p"},"inputs/")," directory."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../assignments/minishell/checker/_test/inputs$ ls -F\ntest_01.txt test_03.txt test_05.txt test_07.txt test_09.txt test_11.txt test_13.txt test_15.txt test_17.txt\ntest_02.txt test_04.txt test_06.txt test_08.txt test_10.txt test_12.txt test_14.txt test_16.txt test_18.txt\n")),(0,r.kt)("p",null,"To execute tests you need to run:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../assignments/minishell/checker$ ./run_all.sh\n")),(0,r.kt)("h3",{id:"debug"},"Debug"),(0,r.kt)("p",null,"To inspect the differences between the output of the mini-shell and the reference binary set ",(0,r.kt)("inlineCode",{parentName:"p"},"DO_CLEANUP=no")," in ",(0,r.kt)("inlineCode",{parentName:"p"},"_test/run_test.sh"),".\nTo see the results of the tests, you can check ",(0,r.kt)("inlineCode",{parentName:"p"},"_test/outputs/")," directory."),(0,r.kt)("h3",{id:"memory-leaks"},"Memory leaks"),(0,r.kt)("p",null,"To inspect the unreleased resources (memory leaks, file descriptors) set ",(0,r.kt)("inlineCode",{parentName:"p"},"USE_VALGRIND=yes")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"DO_CLEANUP=no")," in ",(0,r.kt)("inlineCode",{parentName:"p"},"_test/run_test.sh"),".\nYou can modify both the path to the Valgrind log file and the command parameters.\nTo see the results of the tests, you can check ",(0,r.kt)("inlineCode",{parentName:"p"},"_test/outputs/")," directory."),(0,r.kt)("h3",{id:"checkpatch"},"Checkpatch"),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"checkpatch.pl")," is a script used in the development of the Linux kernel.\nIt is used to check patches that are submitted to the kernel mailing list for adherence to the coding style guidelines of the Linux kernel."),(0,r.kt)("p",null,"The script checks the code for common coding style issues, such as indentation, spacing, line length, function and variable naming conventions, and other formatting rules.\nIt also checks for some common errors, such as uninitialized variables, memory leaks, and other potential bugs."),(0,r.kt)("p",null,"You can ",(0,r.kt)("a",{parentName:"p",href:"https://github.com/torvalds/linux/blob/master/scripts/checkpatch.pl"},"download")," the ",(0,r.kt)("inlineCode",{parentName:"p"},"checkpatch.pl")," script from the official Linux kernel repository."),(0,r.kt)("p",null,"Running the following command will show you linting warnings and errors:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-sh"},"./checkpatch.pl --no-tree --terse -f /path/to/your/code.c\n")))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/487d1817.acb67fdf.js b/17/assets/js/487d1817.acb67fdf.js new file mode 100644 index 0000000000..9bf2c01575 --- /dev/null +++ b/17/assets/js/487d1817.acb67fdf.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4126],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function l(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function i(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},o=Object.keys(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var p=r.createContext({}),c=function(e){var t=r.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},u=function(e){var t=c(e.components);return r.createElement(p.Provider,{value:t},e.children)},s="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,p=e.parentName,u=i(e,["components","mdxType","originalType","parentName"]),s=c(n),d=a,m=s["".concat(p,".").concat(d)]||s[d]||f[d]||o;return n?r.createElement(m,l(l({ref:t},u),{},{components:n})):r.createElement(m,l({ref:t},u))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,l=new Array(o);l[0]=d;var i={};for(var p in t)hasOwnProperty.call(t,p)&&(i[p]=t[p]);i.originalType=e,i[s]="string"==typeof e?e:a,l[1]=i;for(var c=2;c<o;c++)l[c]=n[c];return r.createElement.apply(null,l)}return r.createElement.apply(null,n)}d.displayName="MDXCreateElement"},2974:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>l,default:()=>f,frontMatter:()=>o,metadata:()=>i,toc:()=>c});var r=n(7462),a=(n(7294),n(3905));const o={},l="Python Tools",i={unversionedId:"Lab/Software Stack/quiz/high-level-lang",id:"Lab/Software Stack/quiz/high-level-lang",title:"Python Tools",description:"Question Text",source:"@site/docs/Lab/Software Stack/quiz/high-level-lang.md",sourceDirName:"Lab/Software Stack/quiz",slug:"/Lab/Software Stack/quiz/high-level-lang",permalink:"/operating-systems/17/Lab/Software Stack/quiz/high-level-lang",draft:!1,tags:[],version:"current",frontMatter:{}},p={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:c},s="wrapper";function f(e){let{components:t,...n}=e;return(0,a.kt)(s,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"python-tools"},"Python Tools"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"A Python program is not a proper argument for ..."),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"ltrace"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"strace"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"rm")))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"ldd"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Because a Python program is a script to be interpreted, it won't be usable by ",(0,a.kt)("inlineCode",{parentName:"p"},"ldd"),".\n",(0,a.kt)("inlineCode",{parentName:"p"},"ldd")," is passed a binary executable to look for library dependencies."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/4931bb70.a37f64db.js b/17/assets/js/4931bb70.a37f64db.js new file mode 100644 index 0000000000..4b04073234 --- /dev/null +++ b/17/assets/js/4931bb70.a37f64db.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3867],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function l(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function o(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},i=Object.keys(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),s=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},u=function(e){var t=s(e.components);return r.createElement(c.Provider,{value:t},e.children)},p="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},k=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,c=e.parentName,u=o(e,["components","mdxType","originalType","parentName"]),p=s(n),k=a,m=p["".concat(c,".").concat(k)]||p[k]||d[k]||i;return n?r.createElement(m,l(l({ref:t},u),{},{components:n})):r.createElement(m,l({ref:t},u))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,l=new Array(i);l[0]=k;var o={};for(var c in t)hasOwnProperty.call(t,c)&&(o[c]=t[c]);o.originalType=e,o[p]="string"==typeof e?e:a,l[1]=o;for(var s=2;s<i;s++)l[s]=n[s];return r.createElement.apply(null,l)}return r.createElement.apply(null,n)}k.displayName="MDXCreateElement"},6020:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>l,default:()=>d,frontMatter:()=>i,metadata:()=>o,toc:()=>s});var r=n(7462),a=(n(7294),n(3905));const i={},l="libs",o={unversionedId:"Lab/Software Stack/quiz/libs",id:"Lab/Software Stack/quiz/libs",title:"libs",description:"Static Executables",source:"@site/docs/Lab/Software Stack/quiz/libs.md",sourceDirName:"Lab/Software Stack/quiz",slug:"/Lab/Software Stack/quiz/libs",permalink:"/operating-systems/17/Lab/Software Stack/quiz/libs",draft:!1,tags:[],version:"current",frontMatter:{}},c={},s=[{value:"Static Executables",id:"static-executables",level:2},{value:"Question Text",id:"question-text",level:3},{value:"Question Answers",id:"question-answers",level:3},{value:"Feedback",id:"feedback",level:3},{value:"Dynamic Libraries",id:"dynamic-libraries",level:2},{value:"Question Text",id:"question-text-1",level:3},{value:"Question Answers",id:"question-answers-1",level:3},{value:"Feedback",id:"feedback-1",level:3}],u={toc:s},p="wrapper";function d(e){let{components:t,...n}=e;return(0,a.kt)(p,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"libs"},"libs"),(0,a.kt)("h2",{id:"static-executables"},"Static Executables"),(0,a.kt)("h3",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Which of the following tools has no influence over statically-linked executables? (choose 2 answers)"),(0,a.kt)("h3",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"ltrace"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"strace"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"nm")))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"ldd"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"rm"))),(0,a.kt)("h3",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"ldd")," are used to capture connections to dynamic libraries.\nThey have no effect on statically-linked executables."),(0,a.kt)("h2",{id:"dynamic-libraries"},"Dynamic Libraries"),(0,a.kt)("h3",{id:"question-text-1"},"Question Text"),(0,a.kt)("p",null,"Which of the following is a dynamic library?"),(0,a.kt)("h3",{id:"question-answers-1"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"libc.a"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"libc.so"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"libc.exe"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"libc.c")))),(0,a.kt)("h3",{id:"feedback-1"},"Feedback"),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},".so")," (",(0,a.kt)("em",{parentName:"p"},"shared object"),") extension is used for dynamic libraries in Linux."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/4972.d4c24351.js b/17/assets/js/4972.d4c24351.js new file mode 100644 index 0000000000..96214b2d14 --- /dev/null +++ b/17/assets/js/4972.d4c24351.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4972],{4972:(e,t,n)=>{n.r(t),n.d(t,{default:()=>i});var a=n(7294),l=n(5999),o=n(1944),r=n(215);function i(){return a.createElement(a.Fragment,null,a.createElement(o.d,{title:(0,l.I)({id:"theme.NotFound.title",message:"Page Not Found"})}),a.createElement(r.Z,null,a.createElement("main",{className:"container margin-vert--xl"},a.createElement("div",{className:"row"},a.createElement("div",{className:"col col--6 col--offset-3"},a.createElement("h1",{className:"hero__title"},a.createElement(l.Z,{id:"theme.NotFound.title",description:"The title of the 404 page"},"Page Not Found")),a.createElement("p",null,a.createElement(l.Z,{id:"theme.NotFound.p1",description:"The first paragraph of the 404 page"},"We could not find what you were looking for.")),a.createElement("p",null,a.createElement(l.Z,{id:"theme.NotFound.p2",description:"The 2nd paragraph of the 404 page"},"Please contact the owner of the site that linked you to the original URL and let them know their link is broken.")))))))}}}]); \ No newline at end of file diff --git a/17/assets/js/49cee3f0.b07449a8.js b/17/assets/js/49cee3f0.b07449a8.js new file mode 100644 index 0000000000..81e6844501 --- /dev/null +++ b/17/assets/js/49cee3f0.b07449a8.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9856],{3905:(e,t,a)=>{a.d(t,{Zo:()=>p,kt:()=>g});var n=a(7294);function r(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function o(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,n)}return a}function l(e){for(var t=1;t<arguments.length;t++){var a=null!=arguments[t]?arguments[t]:{};t%2?o(Object(a),!0).forEach((function(t){r(e,t,a[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(a)):o(Object(a)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(a,t))}))}return e}function i(e,t){if(null==e)return{};var a,n,r=function(e,t){if(null==e)return{};var a,n,r={},o=Object.keys(e);for(n=0;n<o.length;n++)a=o[n],t.indexOf(a)>=0||(r[a]=e[a]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n<o.length;n++)a=o[n],t.indexOf(a)>=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(r[a]=e[a])}return r}var s=n.createContext({}),u=function(e){var t=n.useContext(s),a=t;return e&&(a="function"==typeof e?e(t):l(l({},t),e)),a},p=function(e){var t=u(e.components);return n.createElement(s.Provider,{value:t},e.children)},c="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var a=e.components,r=e.mdxType,o=e.originalType,s=e.parentName,p=i(e,["components","mdxType","originalType","parentName"]),c=u(a),m=r,g=c["".concat(s,".").concat(m)]||c[m]||f[m]||o;return a?n.createElement(g,l(l({ref:t},p),{},{components:a})):n.createElement(g,l({ref:t},p))}));function g(e,t){var a=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=a.length,l=new Array(o);l[0]=m;var i={};for(var s in t)hasOwnProperty.call(t,s)&&(i[s]=t[s]);i.originalType=e,i[c]="string"==typeof e?e:r,l[1]=i;for(var u=2;u<o;u++)l[u]=a[u];return n.createElement.apply(null,l)}return n.createElement.apply(null,a)}m.displayName="MDXCreateElement"},9298:(e,t,a)=>{a.r(t),a.d(t,{assets:()=>s,contentTitle:()=>l,default:()=>f,frontMatter:()=>o,metadata:()=>i,toc:()=>u});var n=a(7462),r=(a(7294),a(3905));const o={},l="Half Page",i={unversionedId:"Lab/Data/quiz/half-page",id:"Lab/Data/quiz/half-page",title:"Half Page",description:"Question Text",source:"@site/docs/Lab/Data/quiz/half-page.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/half-page",permalink:"/operating-systems/17/Lab/Data/quiz/half-page",draft:!1,tags:[],version:"current",frontMatter:{}},s={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:u},c="wrapper";function f(e){let{components:t,...a}=e;return(0,r.kt)(c,(0,n.Z)({},p,a,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"half-page"},"Half Page"),(0,r.kt)("h2",{id:"question-text"},"Question Text"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-c"},"char *p = malloc(2 * 1024);\n")),(0,r.kt)("p",null,"What is a potential problem when allocating half a page?"),(0,r.kt)("h2",{id:"question-answers"},"Question Answers"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"It will fragment the virtual memory because users should always request at least one page")),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"Asking for less than a page might place the memory on two different pages"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"Writing to the other half may be allowed")),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"Allocations smaller than one page should use the stack")),(0,r.kt)("h2",{id:"feedback"},"Feedback"),(0,r.kt)("p",null,"The OS allocates memory in chunks of 4 KB (the page size).\nIf the memory we allocate happens to be placed at the beginning of a page, we have permission to update the second half of this page despite not requesting it.\nThis might be a problem because buffer overflows can pass unnoticed."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/4c165629.a25ff5f8.js b/17/assets/js/4c165629.a25ff5f8.js new file mode 100644 index 0000000000..b9b91d317d --- /dev/null +++ b/17/assets/js/4c165629.a25ff5f8.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9724],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>f});var r=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?a(Object(n),!0).forEach((function(t){i(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):a(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function s(e,t){if(null==e)return{};var n,r,i=function(e,t){if(null==e)return{};var n,r,i={},a=Object.keys(e);for(r=0;r<a.length;r++)n=a[r],t.indexOf(n)>=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r<a.length;r++)n=a[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var p=r.createContext({}),l=function(e){var t=r.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},u=function(e){var t=l(e.components);return r.createElement(p.Provider,{value:t},e.children)},d="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},m=r.forwardRef((function(e,t){var n=e.components,i=e.mdxType,a=e.originalType,p=e.parentName,u=s(e,["components","mdxType","originalType","parentName"]),d=l(n),m=i,f=d["".concat(p,".").concat(m)]||d[m]||c[m]||a;return n?r.createElement(f,o(o({ref:t},u),{},{components:n})):r.createElement(f,o({ref:t},u))}));function f(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var a=n.length,o=new Array(a);o[0]=m;var s={};for(var p in t)hasOwnProperty.call(t,p)&&(s[p]=t[p]);s.originalType=e,s[d]="string"==typeof e?e:i,o[1]=s;for(var l=2;l<a;l++)o[l]=n[l];return r.createElement.apply(null,o)}return r.createElement.apply(null,n)}m.displayName="MDXCreateElement"},1094:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>o,default:()=>c,frontMatter:()=>a,metadata:()=>s,toc:()=>l});var r=n(7462),i=(n(7294),n(3905));const a={},o="Pipe Ends",s={unversionedId:"Lab/IO/quiz/pipe-ends",id:"Lab/IO/quiz/pipe-ends",title:"Pipe Ends",description:"Question Text",source:"@site/docs/Lab/IO/quiz/pipe-ends.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/pipe-ends",permalink:"/operating-systems/17/Lab/IO/quiz/pipe-ends",draft:!1,tags:[],version:"current",frontMatter:{}},p={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:l},d="wrapper";function c(e){let{components:t,...n}=e;return(0,i.kt)(d,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"pipe-ends"},"Pipe Ends"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Which end of a pipe created by ",(0,i.kt)("inlineCode",{parentName:"p"},"pipe()")," is for reading and which one is for writing?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"pipefds[0]")," is for reading;\n",(0,i.kt)("inlineCode",{parentName:"li"},"pipefds[1]")," is for writing")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},(0,i.kt)("inlineCode",{parentName:"p"},"pipefds[0]")," is for writing;\n",(0,i.kt)("inlineCode",{parentName:"p"},"pipefds[1]")," is for reading")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"both heads are for reading and writing;"))),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,"Running the binary first tells us which file descriptor is which:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/pipes$ ./anonymous_pipes\npipedes[0] = 3; pipedes[1] = 4\n * pipe created\n -- Press ENTER to continue ...\n")),(0,i.kt)("p",null,"Then ",(0,i.kt)("inlineCode",{parentName:"p"},"lsof")," gives us the complete answer:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/pipes$ lsof -w -p $(pidof anonymous_pipes)\nanonymous 22474 student cwd DIR 8,1 504 296964 /media/student/2TB/Chestii/Poli/Asistent/SO/operating-systems-oer/content/chapters/io/lab/support/pipes\nanonymous 22474 student rtd DIR 259,6 4096 2 /\nanonymous 22474 student txt REG 8,1 26712 296968 /media/student/2TB/Chestii/Poli/Asistent/SO/operating-systems-oer/content/chapters/io/lab/support/pipes/anonymous_pipes\nanonymous 22474 student mem REG 259,6 2029592 1857435 /usr/lib/x86_64-linux-gnu/libc-2.31.so\nanonymous 22474 student mem REG 259,6 191504 1835092 /usr/lib/x86_64-linux-gnu/ld-2.31.so\nanonymous 22474 student 0u CHR 136,0 0t0 3 /dev/pts/0\nanonymous 22474 student 1u CHR 136,0 0t0 3 /dev/pts/0\nanonymous 22474 student 2u CHR 136,0 0t0 3 /dev/pts/0\nanonymous 22474 student 3r FIFO 0,13 0t0 252007 pipe\nanonymous 22474 student 4w FIFO 0,13 0t0 252007 pipe\n")),(0,i.kt)("p",null,"The last 2 lines are the 2 ends of the pipe:\n3 and 4.\nNote each of these numbers is followed by a letter.\nAs you might have guessed:"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"r"),' means "read"'),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"w"),' means "write"'),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"u"),' means both "read" and "write"')),(0,i.kt)("p",null,"Also, the ",(0,i.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},"man page")," is quite clear on this issue:"),(0,i.kt)("blockquote",null,(0,i.kt)("p",{parentName:"blockquote"},(0,i.kt)("inlineCode",{parentName:"p"},"pipefd[0]")," refers to the read end of the pipe. ",(0,i.kt)("inlineCode",{parentName:"p"},"pipefd[1]")," refers\nto the write end of the pipe.")))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/4c341edc.ee92d376.js b/17/assets/js/4c341edc.ee92d376.js new file mode 100644 index 0000000000..afbaf91f98 --- /dev/null +++ b/17/assets/js/4c341edc.ee92d376.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8245],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>f});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function s(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function l(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},i=Object.keys(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r<i.length;r++)n=i[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var o=r.createContext({}),p=function(e){var t=r.useContext(o),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},u=function(e){var t=p(e.components);return r.createElement(o.Provider,{value:t},e.children)},m="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,o=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),m=p(n),d=a,f=m["".concat(o,".").concat(d)]||m[d]||c[d]||i;return n?r.createElement(f,s(s({ref:t},u),{},{components:n})):r.createElement(f,s({ref:t},u))}));function f(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,s=new Array(i);s[0]=d;var l={};for(var o in t)hasOwnProperty.call(t,o)&&(l[o]=t[o]);l.originalType=e,l[m]="string"==typeof e?e:a,s[1]=l;for(var p=2;p<i;p++)s[p]=n[p];return r.createElement.apply(null,s)}return r.createElement.apply(null,n)}d.displayName="MDXCreateElement"},6449:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>o,contentTitle:()=>s,default:()=>c,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const i={},s="Parser",l={unversionedId:"Assignments/Mini Shell/util/parser/README",id:"Assignments/Mini Shell/util/parser/README",title:"Parser",description:"The parser is made using Bison and Flex.",source:"@site/docs/Assignments/Mini Shell/util/parser/README.md",sourceDirName:"Assignments/Mini Shell/util/parser",slug:"/Assignments/Mini Shell/util/parser/",permalink:"/operating-systems/17/Assignments/Mini Shell/util/parser/",draft:!1,tags:[],version:"current",frontMatter:{}},o={},p=[{value:"Compile",id:"compile",level:2},{value:"Usage",id:"usage",level:2},{value:"Build process",id:"build-process",level:3},{value:"Example",id:"example",level:3},{value:"Tests",id:"tests",level:3},{value:"Note",id:"note",level:4},{value:"Other information",id:"other-information",level:3}],u={toc:p},m="wrapper";function c(e){let{components:t,...n}=e;return(0,a.kt)(m,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"parser"},"Parser"),(0,a.kt)("p",null,"The parser is made using ",(0,a.kt)("a",{parentName:"p",href:"http://www.gnu.org/software/bison/"},"Bison")," and ",(0,a.kt)("a",{parentName:"p",href:"http://flex.sourceforge.net/"},"Flex"),"."),(0,a.kt)("p",null,"The parser is used to parse the commands entered by the user, stored in structure ",(0,a.kt)("inlineCode",{parentName:"p"},"command_t"),"."),(0,a.kt)("p",null,"This structure is defined in the file ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.h")," and encapsulates a tree representation of the command."),(0,a.kt)("p",null,"You can use ",(0,a.kt)("inlineCode",{parentName:"p"},"DisplayStructure.cpp")," to display the structure using various ",(0,a.kt)("a",{parentName:"p",href:"#tests"},"tests"),"."),(0,a.kt)("p",null,"Also you can use ",(0,a.kt)("inlineCode",{parentName:"p"},"CUseParser.c")," or ",(0,a.kt)("inlineCode",{parentName:"p"},"UseParser.cpp")," to see how to use the parser in C or C++ and print the structure of the command."),(0,a.kt)("h2",{id:"compile"},"Compile"),(0,a.kt)("p",null,"Run the following commands in the root of parser directory:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../minishell/util/parser$ make\n")),(0,a.kt)("h2",{id:"usage"},"Usage"),(0,a.kt)("p",null,"The parser is represented by two files:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"parser.y")," - implementation of the parser"),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"parser.l")," - implementation of the lexer")),(0,a.kt)("h3",{id:"build-process"},"Build process"),(0,a.kt)("p",null,"The Makefile first generates the files ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.yy.c")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.tab.c")," from ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.y")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.l"),".\nAfter that, it compiles the files ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.yy.c")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.tab.c")," to generate the object files ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.yy.o")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.tab.o"),".\nTo use the parser, you need to link the object files ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.yy.o")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.tab.o")," with your program."),(0,a.kt)("h3",{id:"example"},"Example"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"CUseParser.c")," - example of using the parser in C"),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"UseParser.cpp")," - example of using the parser in C++"),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"DisplayStructure.cpp")," - reads multiple commands and displays the structure of the resulting tree")),(0,a.kt)("h3",{id:"tests"},"Tests"),(0,a.kt)("p",null,"More tests can be found in the ",(0,a.kt)("inlineCode",{parentName:"p"},"tests")," directory:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../minishell/util/parser$ cd tests\n\nstudent@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>small_tests.out <small_tests.txt\n\nstudent@os:/.../minishell/util/parser/tests$ cat small_tests.out\n> mkdir tmp\nCommand successfully read!\ncommand_t (\n scmd (\n simple_command_t (\n verb (\n 'mkdir'\n )\n params (\n 'tmp'\n )\n )\n )\n)\n\n> cd tmp\nCommand successfully read!\ncommand_t (\n scmd (\n\n...\nstudent@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>ugly_tests.out <ugly_tests.txt\n\nstudent@os:/.../minishell/util/parser/tests$ ../DisplayStructure &>negative_tests.out <negative_tests.txt\n")),(0,a.kt)("h4",{id:"note"},"Note"),(0,a.kt)("p",null,"The parser will fail with an error of unknown character if you use the Linux parser (which considers the end of line as ",(0,a.kt)("inlineCode",{parentName:"p"},"\\n"),") on Windows files (end of line as ",(0,a.kt)("inlineCode",{parentName:"p"},"\\r\\n"),") because at the end of the lines (returned by ",(0,a.kt)("inlineCode",{parentName:"p"},"getline()"),") there will be a ",(0,a.kt)("inlineCode",{parentName:"p"},"\\r")," followed by ",(0,a.kt)("inlineCode",{parentName:"p"},"\\n"),".\nThe opposite works (Windows parser with Linux files).\nThe test files use the Linux convention (",(0,a.kt)("inlineCode",{parentName:"p"},"\\n"),")."),(0,a.kt)("h3",{id:"other-information"},"Other information"),(0,a.kt)("p",null,"More information about the parser can be found in the file ",(0,a.kt)("inlineCode",{parentName:"p"},"parser.h"),"."))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/4e29cae1.ea2942a3.js b/17/assets/js/4e29cae1.ea2942a3.js new file mode 100644 index 0000000000..ad149ede8c --- /dev/null +++ b/17/assets/js/4e29cae1.ea2942a3.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[937],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>h});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function s(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function o(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?s(Object(r),!0).forEach((function(t){a(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):s(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function i(e,t){if(null==e)return{};var r,n,a=function(e,t){if(null==e)return{};var r,n,a={},s=Object.keys(e);for(n=0;n<s.length;n++)r=s[n],t.indexOf(r)>=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(n=0;n<s.length;n++)r=s[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var p=n.createContext({}),l=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):o(o({},t),e)),r},u=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},c="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,s=e.originalType,p=e.parentName,u=i(e,["components","mdxType","originalType","parentName"]),c=l(r),m=a,h=c["".concat(p,".").concat(m)]||c[m]||d[m]||s;return r?n.createElement(h,o(o({ref:t},u),{},{components:r})):n.createElement(h,o({ref:t},u))}));function h(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var s=r.length,o=new Array(s);o[0]=m;var i={};for(var p in t)hasOwnProperty.call(t,p)&&(i[p]=t[p]);i.originalType=e,i[c]="string"==typeof e?e:a,o[1]=i;for(var l=2;l<s;l++)o[l]=r[l];return n.createElement.apply(null,o)}return n.createElement.apply(null,r)}m.displayName="MDXCreateElement"},5593:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>o,default:()=>d,frontMatter:()=>s,metadata:()=>i,toc:()=>l});var n=r(7462),a=(r(7294),r(3905));const s={},o="Processes Speedup",i={unversionedId:"Lab/Compute/quiz/processes-speedup",id:"Lab/Compute/quiz/processes-speedup",title:"Processes Speedup",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/processes-speedup.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/processes-speedup",permalink:"/operating-systems/17/Lab/Compute/quiz/processes-speedup",draft:!1,tags:[],version:"current",frontMatter:{}},p={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:l},c="wrapper";function d(e){let{components:t,...r}=e;return(0,a.kt)(c,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"processes-speedup"},"Processes Speedup"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Why is the speedup from running the program in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/sum-array/d/sum_array_processes.d")," with 1, 2, 4 and 8 processes less than expected?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because the array is split into unequal parts")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because of the overhead introduced by the creation of the additional processes")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the algorithm is incorrect")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the operating systems runs all processes sequentially on the same core"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Creating a new process involves an inherent overhead.\nThe OS calls the loader, launches the new process, then the parent process waits for it to finish, extracts its return value etc.\nAll this work together with creating the initial process has to be done by a single thread.\nIn addition, in real-world apps, other actions such as receiving data from the network or reading a file are inherently ",(0,a.kt)("strong",{parentName:"p"},"sequential"),".\nTherefore there will always be parts of any given program that cannot be run in parallel.\nAs a result, the speedup can never be equal to the number of processes between which we spread the workload."),(0,a.kt)("p",null,"It is possible to compute the speedup obtained from parallelising a portion of a given program.\nThe formula is rather simple and is called ",(0,a.kt)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/Amdahl%27s_law"},"Amdahl's law")))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/50ed9da3.75d4fd7d.js b/17/assets/js/50ed9da3.75d4fd7d.js new file mode 100644 index 0000000000..75f1cf775b --- /dev/null +++ b/17/assets/js/50ed9da3.75d4fd7d.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8448],{3905:(e,t,n)=>{n.d(t,{Zo:()=>s,kt:()=>f});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?o(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):o(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function l(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},o=Object.keys(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r<o.length;r++)n=o[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var u=r.createContext({}),p=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},s=function(e){var t=p(e.components);return r.createElement(u.Provider,{value:t},e.children)},c="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,u=e.parentName,s=l(e,["components","mdxType","originalType","parentName"]),c=p(n),d=a,f=c["".concat(u,".").concat(d)]||c[d]||m[d]||o;return n?r.createElement(f,i(i({ref:t},s),{},{components:n})):r.createElement(f,i({ref:t},s))}));function f(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,i=new Array(o);i[0]=d;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[c]="string"==typeof e?e:a,i[1]=l;for(var p=2;p<o;p++)i[p]=n[p];return r.createElement.apply(null,i)}return r.createElement.apply(null,n)}d.displayName="MDXCreateElement"},2870:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>l,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const o={},i="Semaphore Equivalent",l={unversionedId:"Lab/Compute/quiz/semaphore-equivalent",id:"Lab/Compute/quiz/semaphore-equivalent",title:"Semaphore Equivalent",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/semaphore-equivalent.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/semaphore-equivalent",permalink:"/operating-systems/17/Lab/Compute/quiz/semaphore-equivalent",draft:!1,tags:[],version:"current",frontMatter:{}},u={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],s={toc:p},c="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(c,(0,r.Z)({},s,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"semaphore-equivalent"},"Semaphore Equivalent"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"From running and inspecting the code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/apache2-simulator/apache2_simulator_semaphore.py"),", which of the following is an an equivalent to the value of the semaphore ",(0,a.kt)("inlineCode",{parentName:"p"},"sem"),"?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"The value of ",(0,a.kt)("inlineCode",{parentName:"p"},"msg_mutex"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"The time a worker thread has to wait before running"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The length of the ",(0,a.kt)("inlineCode",{parentName:"li"},"messages")," list")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The number of worker threads")),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"sem")," is incremented (",(0,a.kt)("inlineCode",{parentName:"p"},"release()"),") upon adding a message to the ",(0,a.kt)("inlineCode",{parentName:"p"},"messages")," list and decremented (",(0,a.kt)("inlineCode",{parentName:"p"},"acquire()"),") when removing a message from said list.\nSo it's a rough equivalent to the length of this list."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/56fa2c9d.3f08ea95.js b/17/assets/js/56fa2c9d.3f08ea95.js new file mode 100644 index 0000000000..690cc5ee6e --- /dev/null +++ b/17/assets/js/56fa2c9d.3f08ea95.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4980],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>y});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function l(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?l(Object(n),!0).forEach((function(t){a(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):l(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function o(e,t){if(null==e)return{};var n,r,a=function(e,t){if(null==e)return{};var n,r,a={},l=Object.keys(e);for(r=0;r<l.length;r++)n=l[r],t.indexOf(n)>=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(r=0;r<l.length;r++)n=l[r],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),s=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},p=function(e){var t=s(e.components);return r.createElement(c.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},m=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,l=e.originalType,c=e.parentName,p=o(e,["components","mdxType","originalType","parentName"]),u=s(n),m=a,y=u["".concat(c,".").concat(m)]||u[m]||f[m]||l;return n?r.createElement(y,i(i({ref:t},p),{},{components:n})):r.createElement(y,i({ref:t},p))}));function y(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var l=n.length,i=new Array(l);i[0]=m;var o={};for(var c in t)hasOwnProperty.call(t,c)&&(o[c]=t[c]);o.originalType=e,o[u]="string"==typeof e?e:a,i[1]=o;for(var s=2;s<l;s++)i[s]=n[s];return r.createElement.apply(null,i)}return r.createElement.apply(null,n)}m.displayName="MDXCreateElement"},8086:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>f,frontMatter:()=>l,metadata:()=>o,toc:()=>s});var r=n(7462),a=(n(7294),n(3905));const l={},i="Libcall with Syscall",o={unversionedId:"Lab/Software Stack/quiz/libcall-syscall",id:"Lab/Software Stack/quiz/libcall-syscall",title:"Libcall with Syscall",description:"Question Text",source:"@site/docs/Lab/Software Stack/quiz/libcall-syscall.md",sourceDirName:"Lab/Software Stack/quiz",slug:"/Lab/Software Stack/quiz/libcall-syscall",permalink:"/operating-systems/17/Lab/Software Stack/quiz/libcall-syscall",draft:!1,tags:[],version:"current",frontMatter:{}},c={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:s},u="wrapper";function f(e){let{components:t,...n}=e;return(0,a.kt)(u,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"libcall-with-syscall"},"Libcall with Syscall"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Which of the following library calls will for sure invoke a system call?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"fopen()"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"fwrite()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"printf()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"strcpy()")))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,(0,a.kt)("inlineCode",{parentName:"p"},"fopen()")," requires opening a file and access to the operating system (for filesystem access).\nThe others may not require a system call (",(0,a.kt)("inlineCode",{parentName:"p"},"strcpy()"),") or may use buffering to delay the invocation of a system call (",(0,a.kt)("inlineCode",{parentName:"p"},"fwrite()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"printf()"),")."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/577dcf4c.89c8d9db.js b/17/assets/js/577dcf4c.89c8d9db.js new file mode 100644 index 0000000000..20e1fa68aa --- /dev/null +++ b/17/assets/js/577dcf4c.89c8d9db.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[556],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>d});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t<arguments.length;t++){var r=null!=arguments[t]?arguments[t]:{};t%2?i(Object(r),!0).forEach((function(t){o(e,t,r[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(r)):i(Object(r)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(r,t))}))}return e}function l(e,t){if(null==e)return{};var r,n,o=function(e,t){if(null==e)return{};var r,n,o={},i=Object.keys(e);for(n=0;n<i.length;n++)r=i[n],t.indexOf(r)>=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n<i.length;n++)r=i[n],t.indexOf(r)>=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var c=n.createContext({}),p=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},s=function(e){var t=p(e.components);return n.createElement(c.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},y=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,c=e.parentName,s=l(e,["components","mdxType","originalType","parentName"]),u=p(r),y=o,d=u["".concat(c,".").concat(y)]||u[y]||m[y]||i;return r?n.createElement(d,a(a({ref:t},s),{},{components:r})):n.createElement(d,a({ref:t},s))}));function d(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,a=new Array(i);a[0]=y;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[u]="string"==typeof e?e:o,a[1]=l;for(var p=2;p<i;p++)a[p]=r[p];return n.createElement.apply(null,a)}return n.createElement.apply(null,r)}y.displayName="MDXCreateElement"},9577:(e,t,r)=>{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>a,default:()=>m,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var n=r(7462),o=(r(7294),r(3905));const i={},a="Time Server Interoperability",l={unversionedId:"Lab/Application Interaction/quiz/time-server-interop",id:"Lab/Application Interaction/quiz/time-server-interop",title:"Time Server Interoperability",description:"Question Text",source:"@site/docs/Lab/Application Interaction/quiz/time-server-interop.md",sourceDirName:"Lab/Application Interaction/quiz",slug:"/Lab/Application Interaction/quiz/time-server-interop",permalink:"/operating-systems/17/Lab/Application Interaction/quiz/time-server-interop",draft:!1,tags:[],version:"current",frontMatter:{}},c={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],s={toc:p},u="wrapper";function m(e){let{components:t,...r}=e;return(0,o.kt)(u,(0,n.Z)({},s,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"time-server-interoperability"},"Time Server Interoperability"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Is the protocol between the python server and the python client the same?\nCan we run the python server and connect to it via the C client?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"Yes")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"No, the protocols are different")),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"By doing the same investigation on the python server we discover that the protocol is the same.\nThis means that we can run the python server and the C client, or the C server and python client, and everything will work:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/time-server/python$ python3 server.py\n")),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/time-server$ ./client 127.0.0.1 2000\nThe time is Thu Sep 1 11:48:03 2022\n")))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/5894dc17.a8fe4d6b.js b/17/assets/js/5894dc17.a8fe4d6b.js new file mode 100644 index 0000000000..7a1d0f47b1 --- /dev/null +++ b/17/assets/js/5894dc17.a8fe4d6b.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8322],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>h});var a=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function r(e){for(var t=1;t<arguments.length;t++){var n=null!=arguments[t]?arguments[t]:{};t%2?i(Object(n),!0).forEach((function(t){o(e,t,n[t])})):Object.getOwnPropertyDescriptors?Object.defineProperties(e,Object.getOwnPropertyDescriptors(n)):i(Object(n)).forEach((function(t){Object.defineProperty(e,t,Object.getOwnPropertyDescriptor(n,t))}))}return e}function s(e,t){if(null==e)return{};var n,a,o=function(e,t){if(null==e)return{};var n,a,o={},i=Object.keys(e);for(a=0;a<i.length;a++)n=i[a],t.indexOf(n)>=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a<i.length;a++)n=i[a],t.indexOf(n)>=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var l=a.createContext({}),p=function(e){var t=a.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):r(r({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(l.Provider,{value:t},e.children)},d="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,o=e.mdxType,i=e.originalType,l=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),d=p(n),m=o,h=d["".concat(l,".").concat(m)]||d[m]||u[m]||i;return n?a.createElement(h,r(r({ref:t},c),{},{components:n})):a.createElement(h,r({ref:t},c))}));function h(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=n.length,r=new Array(i);r[0]=m;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[d]="string"==typeof e?e:o,r[1]=s;for(var p=2;p<i;p++)r[p]=n[p];return a.createElement.apply(null,r)}return a.createElement.apply(null,n)}m.displayName="MDXCreateElement"},3489:(e,t,n)=>{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>r,default:()=>u,frontMatter:()=>i,metadata:()=>s,toc:()=>p});var a=n(7462),o=(n(7294),n(3905));const i={},r="Arena",s={unversionedId:"Lab/Application Interaction/arena",id:"Lab/Application Interaction/arena",title:"Arena",description:"Oneko",source:"@site/docs/Lab/Application Interaction/arena.md",sourceDirName:"Lab/Application Interaction",slug:"/Lab/Application Interaction/arena",permalink:"/operating-systems/17/Lab/Application Interaction/arena",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"OS Cloud",permalink:"/operating-systems/17/Lab/Application Interaction/os-cloud"},next:{title:"Assignments",permalink:"/operating-systems/17/Assignments/"}},l={},p=[{value:"Oneko",id:"oneko",level:2},{value:"D-Bus",id:"d-bus",level:2},{value:"OS-Cloud: More Disk Customization",id:"os-cloud-more-disk-customization",level:2},{value:"Copy Additional Files to the Newly Created Disk",id:"copy-additional-files-to-the-newly-created-disk",level:3},{value:"SSH Key Setup",id:"ssh-key-setup",level:3},{value:"OS-Cloud: Internet Access",id:"os-cloud-internet-access",level:2}],c={toc:p},d="wrapper";function u(e){let{components:t,...i}=e;return(0,o.kt)(d,(0,a.Z)({},c,i,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"arena"},"Arena"),(0,o.kt)("h2",{id:"oneko"},"Oneko"),(0,o.kt)("p",null,"An alternative to ",(0,o.kt)("inlineCode",{parentName:"p"},"xeyes")," which allows us to observe Unix sockets is ",(0,o.kt)("inlineCode",{parentName:"p"},"oneko"),".\nGoing through the same steps, we see that the application also create a Unix socket, then connects to the path ",(0,o.kt)("inlineCode",{parentName:"p"},'@"/tmp/.X11-unix/X0"'),"."),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~$ strace -e trace=socket,connect oneko\nsocket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC, 0) = 3\nconnect(3, {sa_family=AF_UNIX, sun_path=@"/tmp/.X11-unix/X1"}, 20) = 0\n--- SIGALRM {si_signo=SIGALRM, si_code=SI_KERNEL} ---\n')),(0,o.kt)("p",null,"When running ",(0,o.kt)("inlineCode",{parentName:"p"},"oneko"),", what differs from ",(0,o.kt)("inlineCode",{parentName:"p"},"xeyes")," is the ",(0,o.kt)("inlineCode",{parentName:"p"},"SIGALRM")," signal.\nThis means that ",(0,o.kt)("inlineCode",{parentName:"p"},"oneko")," uses a timer, which is periodically set, and then it expires only to be reset again.\nThe purpose of this timer is to slow down the cat."),(0,o.kt)("p",null,"Verifying the communication between the X server and ",(0,o.kt)("inlineCode",{parentName:"p"},"oneko")," is easy.\nWe see that the cat follows our mouse cursor, behaving similarly to ",(0,o.kt)("inlineCode",{parentName:"p"},"xeyes"),".\nAfter running ",(0,o.kt)("inlineCode",{parentName:"p"},"oneko")," under ",(0,o.kt)("inlineCode",{parentName:"p"},"strace"),", we see the communication uses the UNIX socket created at the beginning:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"strace -e 'trace=!poll' -e trace='socket,connect,recvmsg' oneko |& grep -v '\\-1 EAGAIN'\n")),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Application%20Interaction/quiz/timer"},"Quiz")),(0,o.kt)("h2",{id:"d-bus"},"D-Bus"),(0,o.kt)("p",null,"Use the ",(0,o.kt)("inlineCode",{parentName:"p"},"dbus")," python bindings to get the computer's battery level using a python script.\nYou can start from the documentation ",(0,o.kt)("a",{parentName:"p",href:"https://dbus.freedesktop.org/doc/dbus-python/tutorial.html#"},"here"),".\nYou need to read the sections ",(0,o.kt)("inlineCode",{parentName:"p"},"Connecting to the Bus"),", ",(0,o.kt)("inlineCode",{parentName:"p"},"Proxy objects"),", and ",(0,o.kt)("inlineCode",{parentName:"p"},"Interfaces and methods"),"."),(0,o.kt)("p",null,"There's also a skeleton you can use in ",(0,o.kt)("inlineCode",{parentName:"p"},"support/dbus/get_battery_level.py"),"."),(0,o.kt)("p",null,"In summary, your script will start by connecting to the ",(0,o.kt)("inlineCode",{parentName:"p"},"System Bus"),".\nThen you'll use the ",(0,o.kt)("inlineCode",{parentName:"p"},"get_object")," method to obtain a proxy object.\nOn this proxy object, you can actually do the method call as explained ",(0,o.kt)("a",{parentName:"p",href:"https://dbus.freedesktop.org/doc/dbus-python/tutorial.html#interfaces-and-methods"},"here"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-text"},"To call a method, call the method of the same name on the proxy object, passing in the interface name via the dbus_interface keyword argument\n")),(0,o.kt)("p",null,"So, if you want to call the method ",(0,o.kt)("inlineCode",{parentName:"p"},"this.is.an.interface.method")," with the arguments ",(0,o.kt)("inlineCode",{parentName:"p"},"A")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"B")," you can do it like this:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-python"},'result = proxy.method(A, B, dbus_interface = "this.is.an.interface")\n')),(0,o.kt)("h2",{id:"os-cloud-more-disk-customization"},"OS-Cloud: More Disk Customization"),(0,o.kt)("p",null,"You might have probably noticed that there are 2 types of disk customizations:"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"One type is for things that can be done without running the virtual machine.\nIf we only want to modify some files inside the disk filesystem, we can do so by mounting the disk.\nThis is done, for example, in the ",(0,o.kt)("inlineCode",{parentName:"p"},"disk-templates/ubuntu_22.04/setup_root_password.sh")," script.\nThere we use ",(0,o.kt)("inlineCode",{parentName:"p"},"nbd_connect_qcow2")," + ",(0,o.kt)("inlineCode",{parentName:"p"},"mount")," to mount the disk, then we modify the ",(0,o.kt)("inlineCode",{parentName:"p"},"/etc/shadow")," file to change the root password.")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"The second case is for operations that must be done with the virtual machine running.\nThese are handled in the ",(0,o.kt)("inlineCode",{parentName:"p"},"ubuntu_22_04_vm_prepare")," function: the virtual machine is first started (",(0,o.kt)("inlineCode",{parentName:"p"},"start_qemu_for_vm"),"), then ",(0,o.kt)("inlineCode",{parentName:"p"},"pexpect")," is used to interact with the virtual machine via the ",(0,o.kt)("inlineCode",{parentName:"p"},"qemu")," serial console.\nHere we do things like running ",(0,o.kt)("inlineCode",{parentName:"p"},"ssh-keygen")," - a binary that is part of the disk filesystem, which depends on other parts of the operating system from the disk to be running.\nNote that in ",(0,o.kt)("inlineCode",{parentName:"p"},"ubuntu_22_04_vm_prepare"),", for convenience, we also do some customizations that fall into the first category (like modifying ",(0,o.kt)("inlineCode",{parentName:"p"},"/etc/ssh/sshd_config"),")."))),(0,o.kt)("h3",{id:"copy-additional-files-to-the-newly-created-disk"},"Copy Additional Files to the Newly Created Disk"),(0,o.kt)("p",null,"This is a customization from the first category.\nIn ",(0,o.kt)("inlineCode",{parentName:"p"},"disk-templates/ubuntu_22.04/files")," there is a file called ",(0,o.kt)("inlineCode",{parentName:"p"},"99-os-cloud-welcome")," (a script that prints a greeting message).\nWe want to copy this file to ",(0,o.kt)("inlineCode",{parentName:"p"},"/etc/update-motd.d")," in our newly created disk, so that it will run whenever a user logs in."),(0,o.kt)("p",null,"To do this, you will create a script called ",(0,o.kt)("inlineCode",{parentName:"p"},"copy_files.sh")," in ",(0,o.kt)("inlineCode",{parentName:"p"},"disk-templates/ubuntu_22.04"),".\nThis script will receive a path to a ",(0,o.kt)("inlineCode",{parentName:"p"},"qcow2")," disk file as an argument, it will mount the disk, and then copy the file to the necessary location.\nThen, in the ",(0,o.kt)("inlineCode",{parentName:"p"},"create_disk_from_template")," function in ",(0,o.kt)("inlineCode",{parentName:"p"},"disk.py")," you will call this script, similar with how the other scripts are called."),(0,o.kt)("p",null,"You can use ",(0,o.kt)("inlineCode",{parentName:"p"},"disk-templates/ubuntu_22.04/setup_root_password.sh")," as an example."),(0,o.kt)("h3",{id:"ssh-key-setup"},"SSH Key Setup"),(0,o.kt)("p",null,"We want to be able to log into the virtual machine using an ssh key, instead of the password ",(0,o.kt)("inlineCode",{parentName:"p"},"123456"),".\nNotice that the ",(0,o.kt)("inlineCode",{parentName:"p"},"vm_create")," API also accepts an ",(0,o.kt)("inlineCode",{parentName:"p"},"ssh_key")," parameter.\nHere, the user can provide an ssh public key, which the system will install in ",(0,o.kt)("inlineCode",{parentName:"p"},"/root/.ssh/authorized_keys")," in the newly created virtual machine."),(0,o.kt)("p",null,"Your task is to implement this feature, as a customization from the second category (that is, implemented in the ",(0,o.kt)("inlineCode",{parentName:"p"},"ubuntu_22_04_vm_prepare")," function).\nThe key will be accessible to the function as the ",(0,o.kt)("inlineCode",{parentName:"p"},"ssh_pub_key")," parameter.\nThen it's only a matter of writing the key to the appropriate place, using a command like ",(0,o.kt)("inlineCode",{parentName:"p"},"echo key > /root/.ssh/authorized_keys"),".\nNote that the ",(0,o.kt)("inlineCode",{parentName:"p"},"/root/.ssh")," directory might not exist, so you need to create it as well."),(0,o.kt)("p",null,"After the feature is complete, you can test it using the keys in the ",(0,o.kt)("inlineCode",{parentName:"p"},"support/os-cloud/keys")," directory.\nThis directory contains a pair of public-private keys.\nThe directory will also be mounted inside the ",(0,o.kt)("inlineCode",{parentName:"p"},"os-cloud")," container in ",(0,o.kt)("inlineCode",{parentName:"p"},"/keys"),"."),(0,o.kt)("p",null,"You will create another virtual machine, passing the public key to ",(0,o.kt)("inlineCode",{parentName:"p"},"vm_create"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../support/os-cloud$ curl -H "Content-Type: application/json" \\\n -d \'{ "name": "my_vm2", "image": "ubuntu_22.04", "network": "default", "mem_size": "2G", "disk_size": "10G", "ssh_key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC8CDHgeE4NIIIih3wSz58GDkfPLUk2m9gmbZB1f6o8Lzawzb3HVFpslAUWK0f/Ymw9cloInpMo50gWMYFSyJ7ZrOWWak54BedpHDkFAxxy+JCE9b+pkKsrAT7wiir7gn2LHlhj55FLZkC9PpM9cBcrMfzlcP9Bf+2cnpDdINybSLmOUmrI23ANteM4lEVaa2yEbCaJk6dFB8+atz5zPjvVI0Hd+kJK7yJ0xV6Zc2ADle7TKW3dyiXOE9qFKe9933Rj7ocqNXCAO1cxUoJCVuVS7lh+1pSSPXLWLTOhVp/XiLGWVP6KRYmmn710MWKm9Kj1tPiGUphUraL20SJiRT6/ os-cloud-user"}\' \\\n localhost:5000/vm_create\n{"id":2,"status":"ok"}\n')),(0,o.kt)("p",null,"Obtain the IP address that was allocated to the new vm:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../support/os-cloud$ curl -s -H "Content-Type: application/json" -d \'{ "id": 2 }\' localhost:5000/vm_info | jq .\n{\n "disk_size": 10737418240,\n "id": 2,\n "ip": "192.168.0.3",\n "mem_size": 2147483648,\n "name": "my_vm2",\n "network": "default",\n "os": "ubuntu_22.04",\n "state": "RUNNING"\n}\n')),(0,o.kt)("p",null,"Then go inside the ",(0,o.kt)("inlineCode",{parentName:"p"},"os-cloud")," container and ssh to the vm using the private key in ",(0,o.kt)("inlineCode",{parentName:"p"},"/keys"),".\nIt should work without prompting for the password:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/os-cloud$ docker-compose exec os-cloud bash\nroot@ac93d3d6cab2:/app# ssh -i /keys/ssh_key root@192.168.0.3\nWelcome to Ubuntu 22.04.1 LTS (GNU/Linux 5.15.0-56-generic x86_64)\n[...]\nPowered by OS Cloud\nLast login: Mon Jan 2 19:34:53 2023 from 192.168.0.1\nroot@ubuntu:~#\n")),(0,o.kt)("h2",{id:"os-cloud-internet-access"},"OS-Cloud: Internet Access"),(0,o.kt)("p",null,"Notice that our virtual machines don't have internet access:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"Powered by OS Cloud\nLast login: Mon Jan 2 19:52:47 UTC 2023 on ttyS0\nroot@ubuntu:~# curl google.com\ncurl: (6) Could not resolve host: google.com\n")),(0,o.kt)("p",null,"In this task, we want to fix this problem.\nTo do this, we must first understand how the networking for the virtual machines is done."),(0,o.kt)("p",null,"First, there is the concept of a ",(0,o.kt)("inlineCode",{parentName:"p"},"network"),", which you saw in the previous section.\nThere is a network called ",(0,o.kt)("inlineCode",{parentName:"p"},"default"),", with the address of ",(0,o.kt)("inlineCode",{parentName:"p"},"192.168.0.0/24"),".\nAll virtual machines are part of this network, that's why they were allocated ip addresses like ",(0,o.kt)("inlineCode",{parentName:"p"},"192.168.0.2"),"."),(0,o.kt)("p",null,"Let's go inside the ",(0,o.kt)("inlineCode",{parentName:"p"},"os-cloud")," container and take a look at the network interfaces:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"$ docker-compose exec os-cloud bash\nroot@8333e5cefb0d:/app# ip a\n1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000\n link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00\n inet 127.0.0.1/8 scope host lo\n valid_lft forever preferred_lft forever\n2: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000\n link/ether 8a:68:b7:5b:6b:45 brd ff:ff:ff:ff:ff:ff\n inet 192.168.0.1/16 scope global br0\n valid_lft forever preferred_lft forever\n3: tap0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000\n link/ether 8a:68:b7:5b:6b:45 brd ff:ff:ff:ff:ff:ff\n4: tap1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master br0 state UP group default qlen 1000\n link/ether fa:f8:7f:83:50:8f brd ff:ff:ff:ff:ff:ff\n77: eth0@if78: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default\n link/ether 02:42:ac:16:00:03 brd ff:ff:ff:ff:ff:ff link-netnsid 0\n inet 172.22.0.3/16 brd 172.22.255.255 scope global eth0\n valid_lft forever preferred_lft forever\n\nroot@8333e5cefb0d:/app# ps -ef | grep qemu\nroot 19 8 29 09:15 ? 00:01:26 qemu-system-x86_64 -m 2048 -hda /vm-disks/1/disk.qcow2 -net nic,macaddr=52:54:00:12:34:00 -net tap,ifname=tap0,script=no -monitor telnet::10001,server,nowait -serial telnet::10002,server,nowait -nographic -enable-kvm\nroot 29 8 28 09:15 ? 00:01:24 qemu-system-x86_64 -m 2048 -hda /vm-disks/2/disk.qcow2 -net nic,macaddr=52:54:00:12:34:01 -net tap,ifname=tap1,script=no -monitor telnet::10003,server,nowait -serial telnet::10004,server,nowait -nographic -enable-kvm\n")),(0,o.kt)("p",null,"Here we have 2 virtual machines running.\nEach virtual machine uses a ",(0,o.kt)("inlineCode",{parentName:"p"},"tap")," interface (the ",(0,o.kt)("inlineCode",{parentName:"p"},"-net tap,ifname=tap0,script=no")," parameter for ",(0,o.kt)("inlineCode",{parentName:"p"},"qemu"),").\nThis means that the ",(0,o.kt)("inlineCode",{parentName:"p"},"ens0")," interface inside the virtual machine corresponds to the ",(0,o.kt)("inlineCode",{parentName:"p"},"tap0")," interface outside the virtual machine.\nAll the tap interfaces are bridged together into the ",(0,o.kt)("inlineCode",{parentName:"p"},"br0")," bridge, which has the ip address ",(0,o.kt)("inlineCode",{parentName:"p"},"192.168.0.1"),".\nAlso, each virtual machine has the default gateway configured to be ",(0,o.kt)("inlineCode",{parentName:"p"},"192.168.0.1"),"."),(0,o.kt)("p",null,"In summary, it looks something like this:"),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"os-cloud",src:n(7804).Z,width:"322",height:"342"})),(0,o.kt)("p",null,"All the traffic coming from the virtual machines passes through the ",(0,o.kt)("inlineCode",{parentName:"p"},"br0")," interface.\nSo, in order to make the internet work, all we have to do is a simple ",(0,o.kt)("inlineCode",{parentName:"p"},"NAT"),", with a command like:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"root@8333e5cefb0d:/app# iptables -t nat -A POSTROUTING -s 192.168.0.0/24 -j MASQUERADE\n")),(0,o.kt)("p",null,"Now, the virtual machines should have internet access:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},'root@8333e5cefb0d:/app# ssh root@192.168.0.2\n[...]\nroot@ubuntu:~# curl google.com\n<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">\n<TITLE>301 Moved\n

301 Moved

\nThe document has moved\nhere.\n\n')),(0,o.kt)("p",null,"Now your task is to run the ",(0,o.kt)("inlineCode",{parentName:"p"},"iptables")," command above automatically when the system starts, so that it's not necessary to run it manually like we did in the above example."),(0,o.kt)("p",null,"A good place to do this is in the ",(0,o.kt)("inlineCode",{parentName:"p"},"create_one_network")," function in ",(0,o.kt)("inlineCode",{parentName:"p"},"network.py"),".\nThere you can add another ",(0,o.kt)("inlineCode",{parentName:"p"},"subprocess.run")," call to run ",(0,o.kt)("inlineCode",{parentName:"p"},"iptables"),".\nThe ",(0,o.kt)("inlineCode",{parentName:"p"},"192.168.0.0/24")," value should not be hardcoded, but you can take it from the ",(0,o.kt)("inlineCode",{parentName:"p"},"ip_with_prefixlen")," member of the ",(0,o.kt)("inlineCode",{parentName:"p"},"Net")," object."))}u.isMDXComponent=!0},7804:(e,t,n)=>{n.d(t,{Z:()=>a});const a=n.p+"assets/images/os_cloud_networking-c27fac0c5e0a3100a184bf283728b467.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/59897f9f.f6861d98.js b/17/assets/js/59897f9f.f6861d98.js new file mode 100644 index 0000000000..f12da3c29a --- /dev/null +++ b/17/assets/js/59897f9f.f6861d98.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4207],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>m});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var u=r.createContext({}),c=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},p=function(e){var t=c(e.components);return r.createElement(u.Provider,{value:t},e.children)},s="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,u=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),s=c(n),d=o,m=s["".concat(u,".").concat(d)]||s[d]||f[d]||a;return n?r.createElement(m,i(i({ref:t},p),{},{components:n})):r.createElement(m,i({ref:t},p))}));function m(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=d;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[s]="string"==typeof e?e:o,i[1]=l;for(var c=2;c{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>f,frontMatter:()=>a,metadata:()=>l,toc:()=>c});var r=n(7462),o=(n(7294),n(3905));const a={},i="`O_TRUNC` Flag Behaviour",l={unversionedId:"Lab/IO/quiz/o-trunc",id:"Lab/IO/quiz/o-trunc",title:"`O_TRUNC` Flag Behaviour",description:"Question Text",source:"@site/docs/Lab/IO/quiz/o-trunc.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/o-trunc",permalink:"/operating-systems/17/Lab/IO/quiz/o-trunc",draft:!1,tags:[],version:"current",frontMatter:{}},u={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:c},s="wrapper";function f(e){let{components:t,...n}=e;return(0,o.kt)(s,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"o_trunc-flag-behaviour"},(0,o.kt)("inlineCode",{parentName:"h1"},"O_TRUNC")," Flag Behaviour"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"How does the ",(0,o.kt)("inlineCode",{parentName:"p"},"O_TRUNC")," flag change the behaviour of ",(0,o.kt)("inlineCode",{parentName:"p"},"open()"),"?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"the previous content of the file is deleted")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"new data will be appended to the file")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"newly written data will be ignored"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"The man page provides us with unlimited wisdon:"),(0,o.kt)("blockquote",null,(0,o.kt)("p",{parentName:"blockquote"},"If the file already exists and is a regular file and the access mode allows writing (i.e., is ",(0,o.kt)("inlineCode",{parentName:"p"},"O_RDWR")," or ",(0,o.kt)("inlineCode",{parentName:"p"},"O_WRONLY"),") it will be truncated to length 0.")),(0,o.kt)("p",null,"Setting the length of the file to 0 is equivalent to deleting the previous content of the file."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/59ad4d8f.04cbb511.js b/17/assets/js/59ad4d8f.04cbb511.js new file mode 100644 index 0000000000..d642dd19bb --- /dev/null +++ b/17/assets/js/59ad4d8f.04cbb511.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4461],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>d});var r=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var s=r.createContext({}),p=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},u=function(e){var t=p(e.components);return r.createElement(s.Provider,{value:t},e.children)},m="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,i=e.mdxType,a=e.originalType,s=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),m=p(n),f=i,d=m["".concat(s,".").concat(f)]||m[f]||c[f]||a;return n?r.createElement(d,o(o({ref:t},u),{},{components:n})):r.createElement(d,o({ref:t},u))}));function d(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var a=n.length,o=new Array(a);o[0]=f;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[m]="string"==typeof e?e:i,o[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>c,frontMatter:()=>a,metadata:()=>l,toc:()=>p});var r=n(7462),i=(n(7294),n(3905));const a={},o="`write_file.txt` Permissions",l={unversionedId:"Lab/IO/quiz/write-file-permissions",id:"Lab/IO/quiz/write-file-permissions",title:"`write_file.txt` Permissions",description:"Question Text",source:"@site/docs/Lab/IO/quiz/write-file-permissions.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/write-file-permissions",permalink:"/operating-systems/17/Lab/IO/quiz/write-file-permissions",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedaback",id:"feedaback",level:2}],u={toc:p},m="wrapper";function c(e){let{components:t,...n}=e;return(0,i.kt)(m,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"write_filetxt-permissions"},(0,i.kt)("inlineCode",{parentName:"h1"},"write_file.txt")," Permissions"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Assume ",(0,i.kt)("inlineCode",{parentName:"p"},"write_file.txt")," is opened like this:\n",(0,i.kt)("inlineCode",{parentName:"p"},'open("write_file.txt", O_WRONLY | O_CREAT)'),".\nWhat might cause it to have different permissions that ",(0,i.kt)("inlineCode",{parentName:"p"},"read_file.txt"),"?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"an undefined behaviour in the kernel")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"using the ",(0,i.kt)("inlineCode",{parentName:"p"},"O_WRONLY")," flag"))),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"not passing a ",(0,i.kt)("inlineCode",{parentName:"li"},"mode")," argument to ",(0,i.kt)("inlineCode",{parentName:"li"},"open()"),", so a random set of permissions is used.")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"a filesystem error.")),(0,i.kt)("h2",{id:"feedaback"},"Feedaback"),(0,i.kt)("p",null,"Quoting from ",(0,i.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,i.kt)("inlineCode",{parentName:"a"},"open()"),"'s man page"),", regarding the ",(0,i.kt)("inlineCode",{parentName:"p"},"O_CREAT")," flag:"),(0,i.kt)("blockquote",null,(0,i.kt)("p",{parentName:"blockquote"},"The mode argument must be supplied if O_CREAT or O_TMPFILE is\nspecified in flags; if it is not supplied, some arbitrary\nbytes from the stack will be applied as the file mode.")),(0,i.kt)("p",null,'The wording "arbitrary bytes from the stack" is a bit obsolete.\nTo demistify the quote above, let\'s consider that ',(0,i.kt)("inlineCode",{parentName:"p"},"open()")," is a function call.\nIn fact, ",(0,i.kt)("inlineCode",{parentName:"p"},"libc")," defines wrapper functions on top of all system calls and our code actually calls those wrappers, so our assumption is not far-fetched."),(0,i.kt)("p",null,"To call ",(0,i.kt)("inlineCode",{parentName:"p"},"open()")," correctly, the coulde would look something like this:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-c"},'open("write_file.txt", O_WRONLY | O_CREAT, 0644).\n')),(0,i.kt)("p",null,(0,i.kt)("inlineCode",{parentName:"p"},"0644")," is the octal representation of ",(0,i.kt)("inlineCode",{parentName:"p"},"rw-r--r--"),".\nThe approximate Assembly code for this function call would look like this:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-asm"},"mov rdi, path_to_file ; first argument: path\n\nmov rsi, O_WRONLY\nor rsi, O_CREAT ; second argument: flags\n\nmov rdx, 0644 ; third argument: mode\n\ncall open\n")),(0,i.kt)("p",null,"When we don't specifty a ",(0,i.kt)("inlineCode",{parentName:"p"},"mode")," argument, the ",(0,i.kt)("inlineCode",{parentName:"p"},"open()")," function simply takes this value from ",(0,i.kt)("inlineCode",{parentName:"p"},"rdx")," as it was before the function call.\nThis is an undefined behaviour because this register might have been modified either way by the program before calling ",(0,i.kt)("inlineCode",{parentName:"p"},"open()"),"."))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/5b157b4b.46b893ce.js b/17/assets/js/5b157b4b.46b893ce.js new file mode 100644 index 0000000000..739d608096 --- /dev/null +++ b/17/assets/js/5b157b4b.46b893ce.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9701],{3905:(e,t,a)=>{a.d(t,{Zo:()=>c,kt:()=>d});var n=a(7294);function r(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function o(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,n)}return a}function i(e){for(var t=1;t=0||(r[a]=e[a]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(r[a]=e[a])}return r}var p=n.createContext({}),l=function(e){var t=n.useContext(p),a=t;return e&&(a="function"==typeof e?e(t):i(i({},t),e)),a},c=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},m="mdxType",h={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},u=n.forwardRef((function(e,t){var a=e.components,r=e.mdxType,o=e.originalType,p=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),m=l(a),u=r,d=m["".concat(p,".").concat(u)]||m[u]||h[u]||o;return a?n.createElement(d,i(i({ref:t},c),{},{components:a})):n.createElement(d,i({ref:t},c))}));function d(e,t){var a=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=a.length,i=new Array(o);i[0]=u;var s={};for(var p in t)hasOwnProperty.call(t,p)&&(s[p]=t[p]);s.originalType=e,s[m]="string"==typeof e?e:r,i[1]=s;for(var l=2;l{a.r(t),a.d(t,{assets:()=>p,contentTitle:()=>i,default:()=>h,frontMatter:()=>o,metadata:()=>s,toc:()=>l});var n=a(7462),r=(a(7294),a(3905));const o={},i="Copy-on-Write",s={unversionedId:"Lab/Compute/copy-on-write",id:"Lab/Compute/copy-on-write",title:"Copy-on-Write",description:"So far, you know that the parent and child process have separate virtual address spaces.",source:"@site/docs/Lab/Compute/copy-on-write.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/copy-on-write",permalink:"/operating-systems/17/Lab/Compute/copy-on-write",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Usage of Processes and Threads in `apache2`",permalink:"/operating-systems/17/Lab/Compute/processes-threads-apache2"},next:{title:"Synchronization",permalink:"/operating-systems/17/Lab/Compute/synchronization"}},p={},l=[{value:"Practice",id:"practice",level:2}],c={toc:l},m="wrapper";function h(e){let{components:t,...o}=e;return(0,r.kt)(m,(0,n.Z)({},c,o,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"copy-on-write"},"Copy-on-Write"),(0,r.kt)("p",null,'So far, you know that the parent and child process have separate virtual address spaces.\nBut how are they created, namely how are they "separated"?\nAnd what about the ',(0,r.kt)("strong",{parentName:"p"},"PAS (physical address space)"),"?\nOf course, we would like the stack of the parent, for example, to be physically distinct from that of the child, so they can execute different functions and use different local variables."),(0,r.kt)("p",null,"But should ",(0,r.kt)("strong",{parentName:"p"},"all")," memory sections from the PAS of the parent be distinct from that of the child?\nWhat about some read-only memory sections, such as ",(0,r.kt)("inlineCode",{parentName:"p"},".text")," and ",(0,r.kt)("inlineCode",{parentName:"p"},".rodata"),"?\nAnd what about the heap, where the child ",(0,r.kt)("em",{parentName:"p"},"may")," use some data previously written by the parent and then override it with its own data."),(0,r.kt)("p",null,"The answer to all of these questions is a core mechanism of multiprocess operating systems called ",(0,r.kt)("strong",{parentName:"p"},"Copy-on-Write"),".\nIt works according to one very simple principle:"),(0,r.kt)("blockquote",null,(0,r.kt)("p",{parentName:"blockquote"},"The VAS of the child process initially points to the same PAS as that of the parent.\nA (physical) frame is only duplicated by the child when it attempts to ",(0,r.kt)("strong",{parentName:"p"},"write")," data to it.")),(0,r.kt)("p",null,"This ensures that read-only sections remain shared, while writable sections are shared as long as their contents remain unchanged.\nWhen changes happen, the process making the change receives a unique frame as a modified copy of the original frame ",(0,r.kt)("em",{parentName:"p"},"on demand"),"."),(0,r.kt)("p",null,"In the image below, we have the state of the child and parent processes right after ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," returns in both of them.\nSee how each has its own VAS, both of them being mapped to (mostly) the same PAS."),(0,r.kt)("p",null,(0,r.kt)("img",{alt:"Copy-on-Write",src:a(2873).Z,width:"782",height:"402"})),(0,r.kt)("p",null,"When one process writes data to a writeable page (in our case, the child writes to a heap page), the frame to which it corresponds is first duplicated.\nThen the process' page table points the page to the newly copied frame, as you can see in the image below."),(0,r.kt)("p",null,(0,r.kt)("img",{alt:"Copy-on-Write",src:a(7635).Z,width:"821",height:"363"})),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Be careful!"),"\nDo not confuse ",(0,r.kt)("strong",{parentName:"p"},"copy-on-write")," with ",(0,r.kt)("strong",{parentName:"p"},"demand paging"),".\nRemember from the ",(0,r.kt)("a",{parentName:"p",href:"../../../data/"},"Data chapter")," that ",(0,r.kt)("strong",{parentName:"p"},"demand paging")," means that when you allocate memory, the OS allocates virtual memory that remains unmapped to physical memory until it's used.\nOn the other hand, ",(0,r.kt)("strong",{parentName:"p"},"copy-on-write")," posits that the virtual memory is already mapped to some frames.\nThese frames are only duplicated when one of the processes attempts to write data to them."),(0,r.kt)("h2",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Now let's see the copy-on-write mechanism in practice.\nKeep in mind that ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()")," is a function used to create a process."),(0,r.kt)("p",null,"Open two terminals (or better: use ",(0,r.kt)("a",{parentName:"p",href:"https://github.com/tmux/tmux/wiki"},(0,r.kt)("inlineCode",{parentName:"a"},"tmux")),").\nIn one of them, compile and run the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/fork-faults/fork_faults.c"),".\nAfter each time you press ",(0,r.kt)("inlineCode",{parentName:"p"},"Enter")," in the first terminal window, run the following command in the second window:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/fork-faults$ ps -o min_flt,maj_flt -p $(pidof fork_faults)\n")),(0,r.kt)("p",null,"It will show you the number of minor and major page faults performed by the ",(0,r.kt)("inlineCode",{parentName:"p"},"fork_faults")," process and its child."),(0,r.kt)("p",null,"To better understand minor and major page faults, go through the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#minor-and-major-page-faults"},"Minor and Major Page Faults")," excercise in Arena."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/parent-faults-before-fork"},"Quiz 1")),(0,r.kt)("p",null,"Note that after ",(0,r.kt)("inlineCode",{parentName:"p"},"fork()"),"-ing, there is a second row in the output of ",(0,r.kt)("inlineCode",{parentName:"p"},"ps"),".\nThat corresponds to the child process.\nThe first one still corresponds to the parent."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/quiz/child-faults-after-write"},"Quiz 2")),(0,r.kt)("p",null,"Now it should be clear how demand paging differs from copy-on-write.\nShared memory is a similar concept.\nIt's a way of marking certain allocated pages so that copy-on-write is disabled.\nAs you may imagine, changes made by the parent to this memory are visible to the child and vice-versa.\nYou can learn more about it in ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Compute/arena#shared-memory"},"its dedicated section in the Arena"),"."))}h.isMDXComponent=!0},7635:(e,t,a)=>{a.d(t,{Z:()=>n});const n=a.p+"assets/images/copy-on-write-final-2dfe1835636c0b38b11fed42b5b690d2.svg"},2873:(e,t,a)=>{a.d(t,{Z:()=>n});const n=a.p+"assets/images/copy-on-write-initial-a3673d26b2087aaacf630bc556e0a6a8.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/5c4ba3bf.4f6b1a74.js b/17/assets/js/5c4ba3bf.4f6b1a74.js new file mode 100644 index 0000000000..532430f603 --- /dev/null +++ b/17/assets/js/5c4ba3bf.4f6b1a74.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8459],{3905:(e,n,t)=>{t.d(n,{Zo:()=>c,kt:()=>d});var i=t(7294);function r(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function a(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);n&&(i=i.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,i)}return t}function o(e){for(var n=1;n=0||(r[t]=e[t]);return r}(e,n);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(r[t]=e[t])}return r}var p=i.createContext({}),l=function(e){var n=i.useContext(p),t=n;return e&&(t="function"==typeof e?e(n):o(o({},n),e)),t},c=function(e){var n=l(e.components);return i.createElement(p.Provider,{value:n},e.children)},m="mdxType",f={inlineCode:"code",wrapper:function(e){var n=e.children;return i.createElement(i.Fragment,{},n)}},u=i.forwardRef((function(e,n){var t=e.components,r=e.mdxType,a=e.originalType,p=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),m=l(t),u=r,d=m["".concat(p,".").concat(u)]||m[u]||f[u]||a;return t?i.createElement(d,o(o({ref:n},c),{},{components:t})):i.createElement(d,o({ref:n},c))}));function d(e,n){var t=arguments,r=n&&n.mdxType;if("string"==typeof e||r){var a=t.length,o=new Array(a);o[0]=u;var s={};for(var p in n)hasOwnProperty.call(n,p)&&(s[p]=n[p]);s.originalType=e,s[m]="string"==typeof e?e:r,o[1]=s;for(var l=2;l{t.r(n),t.d(n,{assets:()=>p,contentTitle:()=>o,default:()=>f,frontMatter:()=>a,metadata:()=>s,toc:()=>l});var i=t(7462),r=(t(7294),t(3905));const a={},o="Common Functions",s={unversionedId:"Lab/Software Stack/common-functions",id:"Lab/Software Stack/common-functions",title:"Common Functions",description:"By using wrapper calls, we are able to write our programs in C.",source:"@site/docs/Lab/Software Stack/common-functions.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/common-functions",permalink:"/operating-systems/17/Lab/Software Stack/common-functions",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"System Call Wrappers",permalink:"/operating-systems/17/Lab/Software Stack/syscall-wrapper"},next:{title:"Libraries and libc",permalink:"/operating-systems/17/Lab/Software Stack/libc"}},p={},l=[{value:"Practice",id:"practice",level:2}],c={toc:l},m="wrapper";function f(e){let{components:n,...t}=e;return(0,r.kt)(m,(0,i.Z)({},c,t,{components:n,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"common-functions"},"Common Functions"),(0,r.kt)("p",null,"By using wrapper calls, we are able to write our programs in C.\nHowever, we still need to implement common functions for string management, working with I/O, working with memory."),(0,r.kt)("p",null,"The simple attempt is to implement these functions (",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"strcpy()")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"malloc()"),") once in a C source code file and then reuse them when needed.\nThis saves us time (we don't have to reimplement) and allows us to constantly improve one implementation constantly;\nthere will only be one implementation that we update to increase its safety, efficiency or performance."),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"support/common-functions/")," folder stores the implementation of string management functions, in ",(0,r.kt)("inlineCode",{parentName:"p"},"string.c")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"string.h")," and of printing functions in ",(0,r.kt)("inlineCode",{parentName:"p"},"printf.c")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"printf.h"),".\nThe ",(0,r.kt)("inlineCode",{parentName:"p"},"printf")," implementation is ",(0,r.kt)("a",{parentName:"p",href:"https://github.com/mpaland/printf"},"this one"),"."),(0,r.kt)("p",null,"There are two programs: ",(0,r.kt)("inlineCode",{parentName:"p"},"main_string.c")," showcases string management functions, ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," showcases the ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," function."),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"main_string.c")," depends on the ",(0,r.kt)("inlineCode",{parentName:"p"},"string.h")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"string.c")," files that implement the ",(0,r.kt)("inlineCode",{parentName:"p"},"strlen()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"strcpy()")," functions.\nWe print messages using the ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," system call wrapper implemented in ",(0,r.kt)("inlineCode",{parentName:"p"},"syscall.s")),(0,r.kt)("p",null,"Let's build and run the program:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/common-functions$ make main_string\ngcc -fno-stack-protector -c -o main_string.o main_string.c\ngcc -fno-stack-protector -c -o string.o string.c\nnasm -f elf64 -o syscall.o syscall.s\ngcc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none main_string.o string.o syscall.o -o main_string\n\nstudent@os:~/.../lab/support/common-functions$ ./main_string\nDestination string is: warhammer40k\n\nstudent@os:~/.../lab/support/common-functions$ strace ./main_string\nexecve("./main_string", ["./main_string"], 0x7ffd544d0a70 /* 63 vars */) = 0\nwrite(1, "Destination string is: ", 23Destination string is: ) = 23\nwrite(1, "warhammer40k\\n", 13warhammer40k\n) = 13\nexit(0) = ?\n+++ exited with 0 +++\n')),(0,r.kt)("p",null,"When using ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," we see that only the ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," system call wrapper triggers a system call.\nThere are no system calls triggered by ",(0,r.kt)("inlineCode",{parentName:"p"},"strlen()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"strcpy()")," as can be seen in their implementation."),(0,r.kt)("p",null,"In addition, ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," depends on the ",(0,r.kt)("inlineCode",{parentName:"p"},"printf.h")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"printf.c")," files that implement the ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," function.\nThere is a requirement to implement the ",(0,r.kt)("inlineCode",{parentName:"p"},"_putchar()")," function;\nwe implement it in the ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," file using the ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," syscall call wrapper.\nThe ",(0,r.kt)("inlineCode",{parentName:"p"},"main()")," function ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," file contains all the string and printing calls.\n",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," offers a more powerful printing interface, allowing us to print addresses and integers."),(0,r.kt)("p",null,"Let's build and run the program:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/common-functions$ make main_printf\ngcc -fno-stack-protector -c -o printf.o printf.c\ngcc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none main_printf.o printf.o string.o syscall.o -o main_printf\n\nstudent@os:~/.../lab/support/common-functions$ ./main_printf\n[before] src is at 00000000004026A0, len is 12, content: "warhammer40k"\n[before] dest is at 0000000000603000, len is 0, content: ""\ncopying src to dest\n[after] src is at 00000000004026A0, len is 12, content: "warhammer40k"\n[after] dest is at 0000000000603000, len is 12, content: "warhammer40k"\n\nstudent@os:~/.../lab/support/common-functions$ strace ./main_printf\nexecve("./main_printf", ["./main_printf"], 0x7ffcaaa1d660 /* 63 vars */) = 0\nwrite(1, "[", 1[) = 1\nwrite(1, "b", 1b) = 1\nwrite(1, "e", 1e) = 1\nwrite(1, "f", 1f) = 1\nwrite(1, "o", 1o) = 1\nwrite(1, "r", 1r) = 1\nwrite(1, "e", 1e) = 1\nwrite(1, "]", 1]) = 1\nwrite(1, " ", 1 ) = 1\nwrite(1, "s", 1s) = 1\nwrite(1, "r", 1r) = 1\n[...]\n')),(0,r.kt)("p",null,"We see that we have greater printing flexibility with the ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," function.\nHowever, one downside of the current implementation is that it makes a system call for each character.\nThis is inefficient and could be improved by printing a whole string."),(0,r.kt)("h2",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/common-functions/")," folder and go through the practice items below."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update ",(0,r.kt)("inlineCode",{parentName:"p"},"string.c")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"string.h")," to make available the ",(0,r.kt)("inlineCode",{parentName:"p"},"strcat()")," function.\nCall that function in ",(0,r.kt)("inlineCode",{parentName:"p"},"main_string.c")," and print the result.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update the ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," file to use the implementation of ",(0,r.kt)("inlineCode",{parentName:"p"},"sprintf()")," to collect information to be printed inside a buffer.\nCall the ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," function to print the information.\nThe ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," function will no longer be called.\nThis results in a single ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," system call."))),(0,r.kt)("p",null,"Using previously implemented functions allows us to more efficiently write new programs.\nThese functions provide us with extensive features that we use in our programs."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Software%20Stack/quiz/common-functions"},"Quiz")))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/5d4cc23a.14cfd612.js b/17/assets/js/5d4cc23a.14cfd612.js new file mode 100644 index 0000000000..0d0fa1d833 --- /dev/null +++ b/17/assets/js/5d4cc23a.14cfd612.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2216],{1982:a=>{a.exports=JSON.parse('{"title":"Data","slug":"/Lab/Data/","permalink":"/operating-systems/17/Lab/Data/","navigation":{"previous":{"title":"Arena","permalink":"/operating-systems/17/Lab/Software Stack/arena"},"next":{"title":"Data","permalink":"/operating-systems/17/Lab/Data/overview"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/5d7f8186.d039043f.js b/17/assets/js/5d7f8186.d039043f.js new file mode 100644 index 0000000000..6b0752ea5e --- /dev/null +++ b/17/assets/js/5d7f8186.d039043f.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4426],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function a(e){for(var t=1;t=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var u=r.createContext({}),s=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},c=function(e){var t=s(e.components);return r.createElement(u.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},b=r.forwardRef((function(e,t){var n=e.components,i=e.mdxType,o=e.originalType,u=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),p=s(n),b=i,m=p["".concat(u,".").concat(b)]||p[b]||f[b]||o;return n?r.createElement(m,a(a({ref:t},c),{},{components:n})):r.createElement(m,a({ref:t},c))}));function m(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var o=n.length,a=new Array(o);a[0]=b;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[p]="string"==typeof e?e:i,a[1]=l;for(var s=2;s{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>a,default:()=>f,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var r=n(7462),i=(n(7294),n(3905));const o={},a="String Buffer Overflow",l={unversionedId:"Lab/Data/quiz/string-buff-over",id:"Lab/Data/quiz/string-buff-over",title:"String Buffer Overflow",description:"Question Text",source:"@site/docs/Lab/Data/quiz/string-buff-over.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/string-buff-over",permalink:"/operating-systems/17/Lab/Data/quiz/string-buff-over",draft:!1,tags:[],version:"current",frontMatter:{}},u={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:s},p="wrapper";function f(e){let{components:t,...n}=e;return(0,i.kt)(p,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"string-buffer-overflow"},"String Buffer Overflow"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Why does the buffer overflow occur?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"the initial string, declared in ",(0,i.kt)("inlineCode",{parentName:"p"},"main()"),", does not contain a terminating null byte.")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"the buffer is not large enough to store the copied bytes.")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},(0,i.kt)("inlineCode",{parentName:"p"},"memcpy()")," skips the copying of terminating null bytes."))),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"memcpy()")," copies 4 bytes, whereas the size of the string, including the terminating null byte, is 5.")),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,'The string "soso" has length equal to 4, however, 5 bytes are actually used to store it, including the terminating null byte.\nEven though the buffer declared in ',(0,i.kt)("inlineCode",{parentName:"p"},"fun()")," is not large enough to store the 5 bytes, the underlying issue is that we copying just 4 bytes, thus skipping the null byte."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/5e10bb01.37e6c382.js b/17/assets/js/5e10bb01.37e6c382.js new file mode 100644 index 0000000000..3408e7f479 --- /dev/null +++ b/17/assets/js/5e10bb01.37e6c382.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6901],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>d});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function s(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function l(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var o=a.createContext({}),p=function(e){var t=a.useContext(o),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(o.Provider,{value:t},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},f=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,s=e.originalType,o=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),m=p(n),f=r,d=m["".concat(o,".").concat(f)]||m[f]||u[f]||s;return n?a.createElement(d,l(l({ref:t},c),{},{components:n})):a.createElement(d,l({ref:t},c))}));function d(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var s=n.length,l=new Array(s);l[0]=f;var i={};for(var o in t)hasOwnProperty.call(t,o)&&(i[o]=t[o]);i.originalType=e,i[m]="string"==typeof e?e:r,l[1]=i;for(var p=2;p{n.r(t),n.d(t,{assets:()=>o,contentTitle:()=>l,default:()=>u,frontMatter:()=>s,metadata:()=>i,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const s={},l="System Call Wrappers",i={unversionedId:"Lab/Software Stack/syscall-wrapper",id:"Lab/Software Stack/syscall-wrapper",title:"System Call Wrappers",description:"The support/syscall-wrapper/ folder stores the implementation of a simple program written in C (main.c) that calls the write() and exit() functions.",source:"@site/docs/Lab/Software Stack/syscall-wrapper.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/syscall-wrapper",permalink:"/operating-systems/17/Lab/Software Stack/syscall-wrapper",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Analyzing the Software Stack",permalink:"/operating-systems/17/Lab/Software Stack/basic-syscall"},next:{title:"Common Functions",permalink:"/operating-systems/17/Lab/Software Stack/common-functions"}},o={},p=[{value:"Practice",id:"practice",level:2}],c={toc:p},m="wrapper";function u(e){let{components:t,...n}=e;return(0,r.kt)(m,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"system-call-wrappers"},"System Call Wrappers"),(0,r.kt)("p",null,"The ",(0,r.kt)("inlineCode",{parentName:"p"},"support/syscall-wrapper/")," folder stores the implementation of a simple program written in C (",(0,r.kt)("inlineCode",{parentName:"p"},"main.c"),") that calls the ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"exit()")," functions.\nThe functions are defined in ",(0,r.kt)("inlineCode",{parentName:"p"},"syscall.asm")," as wrappers around corresponding system calls.\nEach function invokes the corresponding system call using the specific system call ID and the arguments provided for the function call."),(0,r.kt)("p",null,"The implementation of the two wrapper functions in ",(0,r.kt)("inlineCode",{parentName:"p"},"syscall.asm")," is very simple, as the function arguments are passed in the same registers required by the system call.\nThis is because of the overlap of the first three registers for the ",(0,r.kt)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/X86_calling_conventions#System_V_AMD64_ABI"},"x86_64 Linux function calling convention")," and the ",(0,r.kt)("a",{parentName:"p",href:"https://x64.syscall.sh/"},"x86_64 Linux system call convention"),"."),(0,r.kt)("p",null,(0,r.kt)("inlineCode",{parentName:"p"},"syscall.h")," contains the declaration of the two functions and is included in ",(0,r.kt)("inlineCode",{parentName:"p"},"main.c"),".\nThis way, C programs can be written that make function calls that end up making system calls."),(0,r.kt)("p",null,"Let's build, run and trace system calls for the program:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},'student@os:~/.../lab/support/syscall-wrapper$ ls\nmain.c Makefile syscall.h syscall.s\n\nstudent@os:~/.../lab/support/syscall-wrapper$ make\ngcc -c -o main.o main.c\nnasm -f elf64 -o syscall.o syscall.s\ncc -nostdlib -no-pie -Wl,--entry=main -Wl,--build-id=none main.o syscall.o -o main\n\nstudent@os:~/.../lab/support/syscall-wrapper$ ls\nmain main.c main.o Makefile syscall.h syscall.o syscall.s\n\nstudent@os:~/.../software-stack/lab/syscall-wrapper$ ./main\nHello, world!\n\nstudent@os:~/.../lab/support/syscall-wrapper$ strace ./main\nexecve("./main", ["./main"], 0x7ffee60fb590 /* 63 vars */) = 0\nwrite(1, "Hello, world!\\n", 14Hello, world!\n) = 14\nexit(0) = ?\n+++ exited with 0 +++\n')),(0,r.kt)("p",null,"The trace is similar to the previous example, showing the ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"exit")," system calls."),(0,r.kt)("p",null,"By creating system call wrappers as C functions, we are now relieved of the burden of writing assembly language code.\nOf course, there has to be an initial implementation of wrapper functions written in assembly language;\nbut, after that, we can use C only."),(0,r.kt)("h2",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Update the files in the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/syscall-wrapper/")," folder to make ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," system call available as a wrapper.\nMake a call to the ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," system call to read data from standard input in a buffer.\nThen call ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," to print data from that buffer."),(0,r.kt)("p",null,"Note that the ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," system call returns the number of bytes read.\nUse that as the argument to the subsequent ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," call that prints read data."),(0,r.kt)("p",null,"We can see that it's easier to have wrapper calls and write most of the code in C than in assembly language."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Software%20Stack/quiz/syscall-wrapper"},"Quiz")))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/5ebe6805.255a0b51.js b/17/assets/js/5ebe6805.255a0b51.js new file mode 100644 index 0000000000..902f8762c1 --- /dev/null +++ b/17/assets/js/5ebe6805.255a0b51.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6078],{3905:(e,t,r)=>{r.d(t,{Zo:()=>l,kt:()=>d});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var p=n.createContext({}),s=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},l=function(e){var t=s(e.components);return n.createElement(p.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,p=e.parentName,l=c(e,["components","mdxType","originalType","parentName"]),u=s(r),m=o,d=u["".concat(p,".").concat(m)]||u[m]||f[m]||i;return r?n.createElement(d,a(a({ref:t},l),{},{components:r})):n.createElement(d,a({ref:t},l))}));function d(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,a=new Array(i);a[0]=m;var c={};for(var p in t)hasOwnProperty.call(t,p)&&(c[p]=t[p]);c.originalType=e,c[u]="string"==typeof e?e:o,a[1]=c;for(var s=2;s{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>c,default:()=>m,frontMatter:()=>a,metadata:()=>p,toc:()=>l});var n=r(7462),o=(r(7294),r(3905)),i=r(4996);const a={title:"Compute"},c=void 0,p={unversionedId:"Lecture/Compute",id:"Lecture/Compute",title:"Compute",description:"Focus the slides and press F for fullscreen viewing.",source:"@site/docs/Lecture/Compute.mdx",sourceDirName:"Lecture",slug:"/Lecture/Compute",permalink:"/operating-systems/17/Lecture/Compute",draft:!1,tags:[],version:"current",frontMatter:{title:"Compute"},sidebar:"sidebar",previous:{title:"Data",permalink:"/operating-systems/17/Lecture/Data"},next:{title:"IO",permalink:"/operating-systems/17/Lecture/IO"}},s={},l=[],u={toc:l},f="wrapper";function m(e){let{components:t,...r}=e;return(0,o.kt)(f,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("div",{style:{display:"flex",width:"100%",height:"100%",flexDirection:"row"}},(0,o.kt)("iframe",{style:{flexGrow:1,border:"none",margin:0,padding:0},width:"100%",height:"500px",src:(0,i.Z)("/slides/Compute/index.html")})),(0,o.kt)("br",null),(0,o.kt)("admonition",{type:"tip"},(0,o.kt)("p",{parentName:"admonition"},"Focus the slides and press ",(0,o.kt)("strong",{parentName:"p"},"F")," for fullscreen viewing.")))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/60905a9d.251622e7.js b/17/assets/js/60905a9d.251622e7.js new file mode 100644 index 0000000000..37e97ecf59 --- /dev/null +++ b/17/assets/js/60905a9d.251622e7.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1746],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var u=r.createContext({}),p=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(u.Provider,{value:t},e.children)},s="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,u=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),s=p(n),d=a,m=s["".concat(u,".").concat(d)]||s[d]||f[d]||i;return n?r.createElement(m,o(o({ref:t},c),{},{components:n})):r.createElement(m,o({ref:t},c))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,o=new Array(i);o[0]=d;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[s]="string"==typeof e?e:a,o[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>o,default:()=>f,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const i={},o="`printf()` Under Strace",l={unversionedId:"Lab/IO/quiz/strace-printf",id:"Lab/IO/quiz/strace-printf",title:"`printf()` Under Strace",description:"Question Text",source:"@site/docs/Lab/IO/quiz/strace-printf.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/strace-printf",permalink:"/operating-systems/17/Lab/IO/quiz/strace-printf",draft:!1,tags:[],version:"current",frontMatter:{}},u={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},s="wrapper";function f(e){let{components:t,...n}=e;return(0,a.kt)(s,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"printf-under-strace"},(0,a.kt)("inlineCode",{parentName:"h1"},"printf()")," Under Strace"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"How many calls to ",(0,a.kt)("inlineCode",{parentName:"p"},"write()")," does the code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/buffering/printf_buffering.c")," do?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"none")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"1")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"6")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"5"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Just run the binary under ",(0,a.kt)("inlineCode",{parentName:"p"},"strace"),":"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},'student@os:/.../support/buffering$ strace -e write ./printf_buffering\nwrite(1, "Dovahkiin, Dovahkiin Naal ok zin"..., 169Dovahkiin, Dovahkiin Naal ok zin los vahriin Wah dein vokul mahfaeraak ahst vaal! Ahrk fin norok paal graan Fod nust hon zindro zaan Dovahkiin, fah hin kogaan mu draal! ) = 169\n+++ exited with 0 +++\n')),(0,a.kt)("p",null,"There's ",(0,a.kt)("strong",{parentName:"p"},"one")," call to ",(0,a.kt)("inlineCode",{parentName:"p"},"write()"),"."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/625a19b4.17d4b584.js b/17/assets/js/625a19b4.17d4b584.js new file mode 100644 index 0000000000..17a7758902 --- /dev/null +++ b/17/assets/js/625a19b4.17d4b584.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5430],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>b});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var c=n.createContext({}),u=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},s=function(e){var t=u(e.components);return n.createElement(c.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,s=l(e,["components","mdxType","originalType","parentName"]),p=u(r),m=a,b=p["".concat(c,".").concat(m)]||p[m]||f[m]||o;return r?n.createElement(b,i(i({ref:t},s),{},{components:r})):n.createElement(b,i({ref:t},s))}));function b(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=m;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[p]="string"==typeof e?e:a,i[1]=l;for(var u=2;u{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>f,frontMatter:()=>o,metadata:()=>l,toc:()=>u});var n=r(7462),a=(r(7294),r(3905));const o={},i="Fiber Strace",l={unversionedId:"Lab/Compute/quiz/fiber-strace",id:"Lab/Compute/quiz/fiber-strace",title:"Fiber Strace",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/fiber-strace.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/fiber-strace",permalink:"/operating-systems/17/Lab/Compute/quiz/fiber-strace",draft:!1,tags:[],version:"current",frontMatter:{}},c={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],s={toc:u},p="wrapper";function f(e){let{components:t,...r}=e;return(0,a.kt)(p,(0,n.Z)({},s,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"fiber-strace"},"Fiber Strace"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"How many ",(0,a.kt)("inlineCode",{parentName:"p"},"clone()")," system calls are performed when creating a fiber?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"none")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"one for each fiber")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"one for every 2 fibers")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"2 for each fiber"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"Being ",(0,a.kt)("strong",{parentName:"p"},"user-level threads"),", the fibers aren't created by the operating system.\nThe only system calls that you see used are ",(0,a.kt)("inlineCode",{parentName:"p"},"mmap()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"munmap()"),", used to create each fiber's stack."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/62a2c47b.cacc82c7.js b/17/assets/js/62a2c47b.cacc82c7.js new file mode 100644 index 0000000000..e5306a0159 --- /dev/null +++ b/17/assets/js/62a2c47b.cacc82c7.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6429],{3905:(e,t,n)=>{n.d(t,{Zo:()=>m,kt:()=>k});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function l(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var p=a.createContext({}),s=function(e){var t=a.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},m=function(e){var t=s(e.components);return a.createElement(p.Provider,{value:t},e.children)},d="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,p=e.parentName,m=o(e,["components","mdxType","originalType","parentName"]),d=s(n),u=r,k=d["".concat(p,".").concat(u)]||d[u]||c[u]||i;return n?a.createElement(k,l(l({ref:t},m),{},{components:n})):a.createElement(k,l({ref:t},m))}));function k(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,l=new Array(i);l[0]=u;var o={};for(var p in t)hasOwnProperty.call(t,p)&&(o[p]=t[p]);o.originalType=e,o[d]="string"==typeof e?e:r,l[1]=o;for(var s=2;s{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>l,default:()=>c,frontMatter:()=>i,metadata:()=>o,toc:()=>s});var a=n(7462),r=(n(7294),n(3905));const i={},l="Arena",o={unversionedId:"Lab/Software Stack/arena",id:"Lab/Software Stack/arena",title:"Arena",description:"Go through the practice items below to hone your skills in working with layers of the software stack.",source:"@site/docs/Lab/Software Stack/arena.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/arena",permalink:"/operating-systems/17/Lab/Software Stack/arena",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"App Investigation",permalink:"/operating-systems/17/Lab/Software Stack/app-investigate"},next:{title:"Data",permalink:"/operating-systems/17/Lab/Data/"}},p={},s=[{value:"System Calls",id:"system-calls",level:2},{value:"System Call Wrappers",id:"system-call-wrappers",level:2},{value:"Common Functions",id:"common-functions",level:2},{value:"Libraries and libc",id:"libraries-and-libc",level:2},{value:"High-Level Languages",id:"high-level-languages",level:2},{value:"App Investigation",id:"app-investigation",level:2}],m={toc:s},d="wrapper";function c(e){let{components:t,...n}=e;return(0,r.kt)(d,(0,a.Z)({},m,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"arena"},"Arena"),(0,r.kt)("p",null,"Go through the practice items below to hone your skills in working with layers of the software stack."),(0,r.kt)("h2",{id:"system-calls"},"System Calls"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/basic-syscall/")," folder and go through the practice items below.\nIf you get stuck, take a sneak peek at the solutions in the ",(0,r.kt)("inlineCode",{parentName:"p"},"solution/basic-syscall/")," folder."),(0,r.kt)("p",null,"For debugging, use ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to trace the system calls from your program and make sure the arguments are set right."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update the ",(0,r.kt)("inlineCode",{parentName:"p"},"hello.asm")," and / or ",(0,r.kt)("inlineCode",{parentName:"p"},"hello.s")," files to pause the execution of the program before the ",(0,r.kt)("inlineCode",{parentName:"p"},"exit")," system call."),(0,r.kt)("p",{parentName:"li"},"You need to make the ",(0,r.kt)("inlineCode",{parentName:"p"},"sys_pause")," system call, with no arguments.\nFind its ID ",(0,r.kt)("a",{parentName:"p",href:"https://blog.rchapman.org/posts/Linux_System_Call_Table_for_x86_64/"},"here"),".")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update the ",(0,r.kt)("inlineCode",{parentName:"p"},"hello.asm")," and / or ",(0,r.kt)("inlineCode",{parentName:"p"},"hello.s")," files to read a message from standard input and print it to standard output."),(0,r.kt)("p",{parentName:"li"},"You'll need to define a buffer in the ",(0,r.kt)("inlineCode",{parentName:"p"},"data")," or ",(0,r.kt)("inlineCode",{parentName:"p"},"bss")," section.\nUse the ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," system call to read data in the buffer.\nThe return value of ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," (placed in the ",(0,r.kt)("inlineCode",{parentName:"p"},"rax")," register) is the number of bytes read.\nUse that value as the 3rd argument or ",(0,r.kt)("inlineCode",{parentName:"p"},"write"),", i.e. the number of bytes printed."),(0,r.kt)("p",{parentName:"li"},"Find the ID of the ",(0,r.kt)("inlineCode",{parentName:"p"},"read")," system call ",(0,r.kt)("a",{parentName:"p",href:"https://x64.syscall.sh/"},"here"),".\nTo find out more about its arguments, see ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/read.2.html"},"its man page"),".\nStandard input descriptor is ",(0,r.kt)("inlineCode",{parentName:"p"},"0"),".")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},(0,r.kt)("strong",{parentName:"p"},"Difficult"),": Port the initial program to ARM on 64 bits (also called ",(0,r.kt)("strong",{parentName:"p"},"aarch64"),")."),(0,r.kt)("p",{parentName:"li"},"Use the skeleton files in the ",(0,r.kt)("inlineCode",{parentName:"p"},"arm/")," folder.\nFind information about the aarch64 system calls ",(0,r.kt)("a",{parentName:"p",href:"https://arm64.syscall.sh/"},"here"),".")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Create your own program, written in assembly, doing some system calls you want to learn more about.\nSome system calls you could try: ",(0,r.kt)("inlineCode",{parentName:"p"},"open"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"rename"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"mkdir"),".\nCreate a Makefile for that program.\nRun the resulting program with ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to see the actual system calls being made (and their arguments)."))),(0,r.kt)("h2",{id:"system-call-wrappers"},"System Call Wrappers"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/syscall-wrapper/")," folder and go through the practice items below.\nIf you get stuck, take a sneak peek at the solutions in the ",(0,r.kt)("inlineCode",{parentName:"p"},"solution/syscall-wrapper/")," folder."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update the files in the ",(0,r.kt)("inlineCode",{parentName:"p"},"syscall-wrapper/")," folder to make the ",(0,r.kt)("inlineCode",{parentName:"p"},"getpid")," system call available as a wrapper.\nCreate a function with the signature ",(0,r.kt)("inlineCode",{parentName:"p"},"unsigned int itoa(int n, char *a)")," that converts an integer to a string.\nIt returns the number of digits in the string.\nFor example, it will convert the number ",(0,r.kt)("inlineCode",{parentName:"p"},"1234")," to the string ",(0,r.kt)("inlineCode",{parentName:"p"},'"1234"')," string (",(0,r.kt)("inlineCode",{parentName:"p"},"NULL"),"-terminated, 5 bytes long);\nthe return value is ",(0,r.kt)("inlineCode",{parentName:"p"},"4")," (the number of digits of the ",(0,r.kt)("inlineCode",{parentName:"p"},'"1234"')," string)."),(0,r.kt)("p",{parentName:"li"},"Then make the call to ",(0,r.kt)("inlineCode",{parentName:"p"},"getpid"),";\nit gets no arguments and returns an integer (the PID - ",(0,r.kt)("em",{parentName:"p"},"process ID")," of the current process)."))),(0,r.kt)("h2",{id:"common-functions"},"Common Functions"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/common-functions/")," folder and go through the practice items below.\nIf you get stuck, take a sneak peek at the solutions in the ",(0,r.kt)("inlineCode",{parentName:"p"},"solution/common-functions/")," folder."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update the ",(0,r.kt)("inlineCode",{parentName:"p"},"putchar()")," function in ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," to implement a buffered functionality of ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()"),".\nCharacters passed via the ",(0,r.kt)("inlineCode",{parentName:"p"},"putchar()")," call will be stored in a predefined static global buffer.\nThe ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," call will be invoked when a newline is encountered or when the buffer is full.\nThis results in a reduced number of ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," system calls.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to confirm the reduction of the number of ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," system calls.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Update the ",(0,r.kt)("inlineCode",{parentName:"p"},"main_printf.c")," file to also feature a ",(0,r.kt)("inlineCode",{parentName:"p"},"flush()")," function that forces the flushing of the static global buffer and a ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," system call.\nMake calls to ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"flush()")," to validate the implementation.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to inspect the ",(0,r.kt)("inlineCode",{parentName:"p"},"write")," system calls invoked by ",(0,r.kt)("inlineCode",{parentName:"p"},"printf()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"flush()"),"."))),(0,r.kt)("h2",{id:"libraries-and-libc"},"Libraries and libc"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/libc/")," folder and go through the practice items below.\nIf you get stuck, take a sneak peek at the solutions in the ",(0,r.kt)("inlineCode",{parentName:"p"},"solution/libc/")," folder."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Inside the ",(0,r.kt)("inlineCode",{parentName:"p"},"vendetta.c")," file make a call ",(0,r.kt)("inlineCode",{parentName:"p"},'open("a.txt", O_RDWR | O_CREAT, 0644)')," to open / create the ",(0,r.kt)("inlineCode",{parentName:"p"},"a.txt")," file.\nMake sure you include all required headers.\nCheck the system call being made."),(0,r.kt)("p",{parentName:"li"},"Make an ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()")," with the proper arguments that gets as close as possible to the ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," call, i.e. the system call arguments are as close as possible.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Inside the ",(0,r.kt)("inlineCode",{parentName:"p"},"vendetta.c")," file make a call to ",(0,r.kt)("inlineCode",{parentName:"p"},"sin()")," function (for sine).\nCompute ",(0,r.kt)("inlineCode",{parentName:"p"},"sin(0)")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"sin(PI/2)"),"."))),(0,r.kt)("h2",{id:"high-level-languages"},"High-Level Languages"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/high-level-lang/")," folder and go through the practice items below.\nIf you get stuck, take a sneak peek at the solutions in the ",(0,r.kt)("inlineCode",{parentName:"p"},"solution/high-level-lang/")," folder."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Create programs in C, Python and Go that compute the N-th Fibonacci number.\n",(0,r.kt)("inlineCode",{parentName:"p"},"N")," is passed as a command-line argument."),(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to compute the number of library calls and system calls.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"perf")," to measure the running time."),(0,r.kt)("p",{parentName:"li"},"Compare the values of the three programs.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Create programs in C, Python and Go that copy a source file into a destination file.\nBoth files are passed as the two command-line arguments for the program.\nSample run:"),(0,r.kt)("pre",{parentName:"li"},(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@so:~$ cp src dest\n")),(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," to compute the number of library calls and system calls.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"perf")," to measure the running time.\nUse source files of different sizes.\nCompare the outputs of these commands on the three programs."))),(0,r.kt)("h2",{id:"app-investigation"},"App Investigation"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/app-investigation/")," folder and go through the practice items below."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Check to see whether there are statically-linked application executables in the system.\nThe ",(0,r.kt)("inlineCode",{parentName:"p"},"file")," command tells if the file passed as argument is a statically-linked executable.\nIf you can't find one, install the ",(0,r.kt)("inlineCode",{parentName:"p"},"busybox-static")," package.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Look into what ",(0,r.kt)("a",{parentName:"p",href:"https://busybox.net/"},"busybox")," is and explain why it's custom to have it as statically-linked executable.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Run ",(0,r.kt)("inlineCode",{parentName:"p"},"ldd"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"nm"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"strace"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"ltrace")," on a statically-linked application executable.\nExplain the results."))))}c.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/65687b6f.5dad0b0b.js b/17/assets/js/65687b6f.5dad0b0b.js new file mode 100644 index 0000000000..95b834dfce --- /dev/null +++ b/17/assets/js/65687b6f.5dad0b0b.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7156],{3905:(e,t,r)=>{r.d(t,{Zo:()=>c,kt:()=>f});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var u=n.createContext({}),s=function(e){var t=n.useContext(u),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},c=function(e){var t=s(e.components);return n.createElement(u.Provider,{value:t},e.children)},p="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,u=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),p=s(r),m=a,f=p["".concat(u,".").concat(m)]||p[m]||d[m]||o;return r?n.createElement(f,i(i({ref:t},c),{},{components:r})):n.createElement(f,i({ref:t},c))}));function f(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=m;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[p]="string"==typeof e?e:a,i[1]=l;for(var s=2;s{r.r(t),r.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>d,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var n=r(7462),a=(r(7294),r(3905));const o={},i="Cause of `bind()` Error",l={unversionedId:"Lab/IO/quiz/bind-error-cause",id:"Lab/IO/quiz/bind-error-cause",title:"Cause of `bind()` Error",description:"Question Text",source:"@site/docs/Lab/IO/quiz/bind-error-cause.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/bind-error-cause",permalink:"/operating-systems/17/Lab/IO/quiz/bind-error-cause",draft:!1,tags:[],version:"current",frontMatter:{}},u={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:s},p="wrapper";function d(e){let{components:t,...r}=e;return(0,a.kt)(p,(0,n.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"cause-of-bind-error"},"Cause of ",(0,a.kt)("inlineCode",{parentName:"h1"},"bind()")," Error"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"While ",(0,a.kt)("inlineCode",{parentName:"p"},"receiver.py")," is still running, run it again from another terminal.\nYou will get an error.\nWhat is its cause?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"the IP ",(0,a.kt)("inlineCode",{parentName:"li"},"127.0.0.1")," is already used by ",(0,a.kt)("inlineCode",{parentName:"li"},"receive.py"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"the port 5000 is already used (by ",(0,a.kt)("inlineCode",{parentName:"li"},"receive.py"),")")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"a port may not be used multiple times by the same process")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the socket was not created correctly"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"One port may only be bound to ",(0,a.kt)("strong",{parentName:"p"},"one socket")," at a time.\nThe fact that it's the same program (same source code) using it is irrelevant because they're different processes."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/65ab3714.ff8d3217.js b/17/assets/js/65ab3714.ff8d3217.js new file mode 100644 index 0000000000..631c5363a0 --- /dev/null +++ b/17/assets/js/65ab3714.ff8d3217.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6397],{3905:(e,t,n)=>{n.d(t,{Zo:()=>l,kt:()=>v});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),u=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},l=function(e){var t=u(e.components);return r.createElement(c.Provider,{value:t},e.children)},p="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,l=s(e,["components","mdxType","originalType","parentName"]),p=u(n),f=a,v=p["".concat(c,".").concat(f)]||p[f]||m[f]||o;return n?r.createElement(v,i(i({ref:t},l),{},{components:n})):r.createElement(v,i({ref:t},l))}));function v(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,i=new Array(o);i[0]=f;var s={};for(var c in t)hasOwnProperty.call(t,c)&&(s[c]=t[c]);s.originalType=e,s[p]="string"==typeof e?e:a,i[1]=s;for(var u=2;u{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>s,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const o={},i="Container versus VM",s={unversionedId:"Lab/Application Interaction/quiz/container-vs-vm",id:"Lab/Application Interaction/quiz/container-vs-vm",title:"Container versus VM",description:"Question Text",source:"@site/docs/Lab/Application Interaction/quiz/container-vs-vm.md",sourceDirName:"Lab/Application Interaction/quiz",slug:"/Lab/Application Interaction/quiz/container-vs-vm",permalink:"/operating-systems/17/Lab/Application Interaction/quiz/container-vs-vm",draft:!1,tags:[],version:"current",frontMatter:{}},c={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],l={toc:u},p="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(p,(0,r.Z)({},l,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"container-versus-vm"},"Container versus VM"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What is one advantage of using containers over VMs in regard to starting times and memory consumption?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"VMs can start up faster and use less memory than containers.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Containers can start up faster and use less memory than VMs.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The comparison cannot be made.")),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"This means that they impose less resource overhead than virtual machines, which need to run a complete guest operating system on top of the host operating system using a hypervisor."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/67019f13.b6cd6079.js b/17/assets/js/67019f13.b6cd6079.js new file mode 100644 index 0000000000..8efb687abb --- /dev/null +++ b/17/assets/js/67019f13.b6cd6079.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2145],{2114:t=>{t.exports=JSON.parse('{"title":"Application Interaction","slug":"/Lab/Application Interaction/","permalink":"/operating-systems/17/Lab/Application Interaction/","navigation":{"previous":{"title":"Arena","permalink":"/operating-systems/17/Lab/IO/arena"},"next":{"title":"Application Interaction","permalink":"/operating-systems/17/Lab/Application Interaction/overview"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/686c5edf.1a64eab4.js b/17/assets/js/686c5edf.1a64eab4.js new file mode 100644 index 0000000000..32a0073e99 --- /dev/null +++ b/17/assets/js/686c5edf.1a64eab4.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6394],{2441:e=>{e.exports=JSON.parse('{"title":"IO","slug":"/Lab/IO/","permalink":"/operating-systems/17/Lab/IO/","navigation":{"previous":{"title":"Arena","permalink":"/operating-systems/17/Lab/Compute/arena"},"next":{"title":"I/O","permalink":"/operating-systems/17/Lab/IO/overview"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/68eeb637.fac04e81.js b/17/assets/js/68eeb637.fac04e81.js new file mode 100644 index 0000000000..64f21ea672 --- /dev/null +++ b/17/assets/js/68eeb637.fac04e81.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3742],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>m});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function l(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var i=n.createContext({}),c=function(e){var t=n.useContext(i),r=t;return e&&(r="function"==typeof e?e(t):l(l({},t),e)),r},p=function(e){var t=c(e.components);return n.createElement(i.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},y=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,i=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),u=c(r),y=a,m=u["".concat(i,".").concat(y)]||u[y]||f[y]||o;return r?n.createElement(m,l(l({ref:t},p),{},{components:r})):n.createElement(m,l({ref:t},p))}));function m(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,l=new Array(o);l[0]=y;var s={};for(var i in t)hasOwnProperty.call(t,i)&&(s[i]=t[i]);s.originalType=e,s[u]="string"==typeof e?e:a,l[1]=s;for(var c=2;c{r.r(t),r.d(t,{assets:()=>i,contentTitle:()=>l,default:()=>f,frontMatter:()=>o,metadata:()=>s,toc:()=>c});var n=r(7462),a=(r(7294),r(3905));const o={},l="Syscall Wrappers",s={unversionedId:"Lab/Software Stack/quiz/syscall-wrapper",id:"Lab/Software Stack/quiz/syscall-wrapper",title:"Syscall Wrappers",description:"Question Text",source:"@site/docs/Lab/Software Stack/quiz/syscall-wrapper.md",sourceDirName:"Lab/Software Stack/quiz",slug:"/Lab/Software Stack/quiz/syscall-wrapper",permalink:"/operating-systems/17/Lab/Software Stack/quiz/syscall-wrapper",draft:!1,tags:[],version:"current",frontMatter:{}},i={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:c},u="wrapper";function f(e){let{components:t,...r}=e;return(0,a.kt)(u,(0,n.Z)({},p,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"syscall-wrappers"},"Syscall Wrappers"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What language do we use to invoke system calls?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"assembly")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"C")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"C++")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Go"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"System calls require setting of registers that can only be done in assembly language."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/692011d4.d0f70f48.js b/17/assets/js/692011d4.d0f70f48.js new file mode 100644 index 0000000000..5d20727de4 --- /dev/null +++ b/17/assets/js/692011d4.d0f70f48.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4350],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>f});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function p(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var s=a.createContext({}),l=function(e){var t=a.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):p(p({},t),e)),n},c=function(e){var t=l(e.components);return a.createElement(s.Provider,{value:t},e.children)},u="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,s=e.parentName,c=o(e,["components","mdxType","originalType","parentName"]),u=l(n),m=r,f=u["".concat(s,".").concat(m)]||u[m]||d[m]||i;return n?a.createElement(f,p(p({ref:t},c),{},{components:n})):a.createElement(f,p({ref:t},c))}));function f(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,p=new Array(i);p[0]=m;var o={};for(var s in t)hasOwnProperty.call(t,s)&&(o[s]=t[s]);o.originalType=e,o[u]="string"==typeof e?e:r,p[1]=o;for(var l=2;l{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>p,default:()=>d,frontMatter:()=>i,metadata:()=>o,toc:()=>l});var a=n(7462),r=(n(7294),n(3905));const i={},p="App Investigation",o={unversionedId:"Lab/Software Stack/app-investigate",id:"Lab/Software Stack/app-investigate",title:"App Investigation",description:"Let's spend some time investigating actual applications residing on the local system.",source:"@site/docs/Lab/Software Stack/app-investigate.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/app-investigate",permalink:"/operating-systems/17/Lab/Software Stack/app-investigate",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"High-Level Languages",permalink:"/operating-systems/17/Lab/Software Stack/high-level-lang"},next:{title:"Arena",permalink:"/operating-systems/17/Lab/Software Stack/arena"}},s={},l=[{value:"Practice",id:"practice",level:2}],c={toc:l},u="wrapper";function d(e){let{components:t,...n}=e;return(0,r.kt)(u,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"app-investigation"},"App Investigation"),(0,r.kt)("p",null,"Let's spend some time investigating actual applications residing on the local system.\nFor now, we know that applications are developed using high-level languages and then compiled or interpreted to use the lower-layer interfaces of the software stack, such as the system call API."),(0,r.kt)("p",null,"Let's enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/app-investigate/")," folder and run the ",(0,r.kt)("inlineCode",{parentName:"p"},"get_app_types.sh")," script:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/app-investigate$ ./get_app_types.sh\nbinary apps: 2223\nPerl apps: 256\nShell apps: 454\nPython apps: 123\nOther apps: 27\n")),(0,r.kt)("p",null,"The script prints the types of the application executables in the system.\nThe output will differ between systems, given each has particular types of applications installed."),(0,r.kt)("p",null,"We list them by running the command inside the ",(0,r.kt)("inlineCode",{parentName:"p"},"get_app_types.sh")," script:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/app-investigate$ find /usr/bin /bin /usr/sbin /sbin -type f -exec file {} \\;\n[...]\n/usr/bin/qpdldecode: ELF 64-bit LSB shared object, x86-64 [...]\n/usr/bin/mimeopen: Perl script text executable\n[...]\n")),(0,r.kt)("p",null,"As above, the output will differ between systems."),(0,r.kt)("p",null,"So, depending on the developers' choice, applications may be:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"compiled into executables, from compiled languages such as C, C++, Go, Rust, D"),(0,r.kt)("li",{parentName:"ul"},"developed as scripts, from interpreted languages such as Python, Perl, JavaScript")),(0,r.kt)("h2",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Enter the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/app-investigate/")," folder and go through the practice items below.\nSelect a binary executable application and a scripted application from those listed above."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"ldd")," on the two applications.\nNotice the resulting messages and explain the results.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"ltrace")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"strace")," on the two applications.\nFollow the library calls and the system calls done by each application."))))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/6b326dfe.a9207055.js b/17/assets/js/6b326dfe.a9207055.js new file mode 100644 index 0000000000..b7ba9d7e84 --- /dev/null +++ b/17/assets/js/6b326dfe.a9207055.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[24],{5745:s=>{s.exports=JSON.parse('{"name":"docusaurus-plugin-content-pages","id":"default"}')}}]); \ No newline at end of file diff --git a/17/assets/js/6bda9475.1ffe9302.js b/17/assets/js/6bda9475.1ffe9302.js new file mode 100644 index 0000000000..040230130c --- /dev/null +++ b/17/assets/js/6bda9475.1ffe9302.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9984],{3905:(e,t,r)=>{r.d(t,{Zo:()=>m,kt:()=>f});var a=r(7294);function n(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,a)}return r}function i(e){for(var t=1;t=0||(n[r]=e[r]);return n}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(n[r]=e[r])}return n}var p=a.createContext({}),l=function(e){var t=a.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},m=function(e){var t=l(e.components);return a.createElement(p.Provider,{value:t},e.children)},c="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},y=a.forwardRef((function(e,t){var r=e.components,n=e.mdxType,o=e.originalType,p=e.parentName,m=s(e,["components","mdxType","originalType","parentName"]),c=l(r),y=n,f=c["".concat(p,".").concat(y)]||c[y]||u[y]||o;return r?a.createElement(f,i(i({ref:t},m),{},{components:r})):a.createElement(f,i({ref:t},m))}));function f(e,t){var r=arguments,n=t&&t.mdxType;if("string"==typeof e||n){var o=r.length,i=new Array(o);i[0]=y;var s={};for(var p in t)hasOwnProperty.call(t,p)&&(s[p]=t[p]);s.originalType=e,s[c]="string"==typeof e?e:n,i[1]=s;for(var l=2;l{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>i,default:()=>u,frontMatter:()=>o,metadata:()=>s,toc:()=>l});var a=r(7462),n=(r(7294),r(3905));const o={},i="Data",s={unversionedId:"Lab/Data/overview",id:"Lab/Data/overview",title:"Data",description:"Data represents information that is to be processed to produce a final result or more data.",source:"@site/docs/Lab/Data/overview.md",sourceDirName:"Lab/Data",slug:"/Lab/Data/overview",permalink:"/operating-systems/17/Lab/Data/overview",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Data",permalink:"/operating-systems/17/Lab/Data/"},next:{title:"Working with Memory",permalink:"/operating-systems/17/Lab/Data/working-memory"}},p={},l=[{value:"Contents",id:"contents",level:2}],m={toc:l},c="wrapper";function u(e){let{components:t,...r}=e;return(0,n.kt)(c,(0,a.Z)({},m,r,{components:t,mdxType:"MDXLayout"}),(0,n.kt)("h1",{id:"data"},"Data"),(0,n.kt)("p",null,"Data represents information that is to be processed to produce a final result or more data.\nComputers store, retrieve and compute data.\nThis process involves 4 entities: the programmer, the programming language, the operating system and the hardware."),(0,n.kt)("p",null,"From a programmer's perspective, data is represented by the variables.\nThese are declared and utilized depending on the rules of the programming language that is employed.\nThe programming language analyzes the use of these variables and outputs code that uses an interface provided by the operating system.\nThis interface offers the possibility to allocate/deallocate different variables in certain memory regions.\nNext, the operating system manages the execution of the program and provides the actual physical addresses that are used to interact with the data."),(0,n.kt)("p",null,"Moreover, the operating system governs the competing access of multiple programs to memory, ensuring that a program does not have access to a different program's memory."),(0,n.kt)("h2",{id:"contents"},"Contents"),(0,n.kt)("ol",null,(0,n.kt)("li",{parentName:"ol"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Data/working-memory"},"Working with Memory")),(0,n.kt)("li",{parentName:"ol"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Data/process-memory"},"Process Memory")),(0,n.kt)("li",{parentName:"ol"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Data/investigate-memory"},"Investigate Memory")),(0,n.kt)("li",{parentName:"ol"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Data/memory-security"},"Memory Security")),(0,n.kt)("li",{parentName:"ol"},(0,n.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Data/arena"},"Arena"))))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/6e6dee7f.58a2f5b0.js b/17/assets/js/6e6dee7f.58a2f5b0.js new file mode 100644 index 0000000000..9cbda6300d --- /dev/null +++ b/17/assets/js/6e6dee7f.58a2f5b0.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2838],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function s(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function a(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var i=r.createContext({}),l=function(e){var t=r.useContext(i),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},c=function(e){var t=l(e.components);return r.createElement(i.Provider,{value:t},e.children)},u="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,s=e.originalType,i=e.parentName,c=p(e,["components","mdxType","originalType","parentName"]),u=l(n),f=o,m=u["".concat(i,".").concat(f)]||u[f]||d[f]||s;return n?r.createElement(m,a(a({ref:t},c),{},{components:n})):r.createElement(m,a({ref:t},c))}));function m(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var s=n.length,a=new Array(s);a[0]=f;var p={};for(var i in t)hasOwnProperty.call(t,i)&&(p[i]=t[i]);p.originalType=e,p[u]="string"==typeof e?e:o,a[1]=p;for(var l=2;l{n.r(t),n.d(t,{assets:()=>i,contentTitle:()=>a,default:()=>d,frontMatter:()=>s,metadata:()=>p,toc:()=>l});var r=n(7462),o=(n(7294),n(3905));const s={},a="Parent of `sleep` Processes",p={unversionedId:"Lab/Compute/quiz/parent-of-sleep-processes",id:"Lab/Compute/quiz/parent-of-sleep-processes",title:"Parent of `sleep` Processes",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/parent-of-sleep-processes.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/parent-of-sleep-processes",permalink:"/operating-systems/17/Lab/Compute/quiz/parent-of-sleep-processes",draft:!1,tags:[],version:"current",frontMatter:{}},i={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:l},u="wrapper";function d(e){let{components:t,...n}=e;return(0,o.kt)(u,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"parent-of-sleep-processes"},"Parent of ",(0,o.kt)("inlineCode",{parentName:"h1"},"sleep")," Processes"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Who is the parent of the ",(0,o.kt)("inlineCode",{parentName:"p"},"sleep")," processes?\nWhy?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},(0,o.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py")," because it is the one who created them")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},(0,o.kt)("inlineCode",{parentName:"p"},"bash")," because it is ",(0,o.kt)("inlineCode",{parentName:"p"},"sleepy_creator.py"),"'s parent and when a process dies, its parent adopts its orphan children"))),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"systemd")," because this is the default process that adopts orphans")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"systemd")," because it is ",(0,o.kt)("inlineCode",{parentName:"li"},"sleepy_creator.py"),"'s parent and when a process dies, its parent adopts its orphan children")),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"When a process dies without waiting for the termination of all its children, those processes are now orphans.\nThen the ",(0,o.kt)("inlineCode",{parentName:"p"},"systemd")," process adopts those orphan processes by default.\nOn older Linux systems, it was the ",(0,o.kt)("inlineCode",{parentName:"p"},"init")," process who adopted orphans."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/6e77dc18.f47ee164.js b/17/assets/js/6e77dc18.f47ee164.js new file mode 100644 index 0000000000..7e3271c9c5 --- /dev/null +++ b/17/assets/js/6e77dc18.f47ee164.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5220],{3905:(e,t,r)=>{r.d(t,{Zo:()=>c,kt:()=>m});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var p=n.createContext({}),l=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},c=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},h="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},u=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,p=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),h=l(r),u=a,m=h["".concat(p,".").concat(u)]||h[u]||d[u]||o;return r?n.createElement(m,i(i({ref:t},c),{},{components:r})):n.createElement(m,i({ref:t},c))}));function m(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=u;var s={};for(var p in t)hasOwnProperty.call(t,p)&&(s[p]=t[p]);s.originalType=e,s[h]="string"==typeof e?e:a,i[1]=s;for(var l=2;l{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>i,default:()=>d,frontMatter:()=>o,metadata:()=>s,toc:()=>l});var n=r(7462),a=(r(7294),r(3905));const o={},i="Password Cracker",s={unversionedId:"Lab/Application Interaction/password-cracker",id:"Lab/Application Interaction/password-cracker",title:"Password Cracker",description:"In this example, we will solve the following problem: given the sha512 hash of a password, we want to obtain the password that generated the hash.",source:"@site/docs/Lab/Application Interaction/password-cracker.md",sourceDirName:"Lab/Application Interaction",slug:"/Lab/Application Interaction/password-cracker",permalink:"/operating-systems/17/Lab/Application Interaction/password-cracker",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Time Server",permalink:"/operating-systems/17/Lab/Application Interaction/time-server"},next:{title:"The X Window System",permalink:"/operating-systems/17/Lab/Application Interaction/x-window-system"}},p={},l=[{value:"Multiprocess Version",id:"multiprocess-version",level:2},{value:"Practice",id:"practice",level:3},{value:"Multithreaded Version",id:"multithreaded-version",level:2},{value:"Multiprocess Version in Python (1)",id:"multiprocess-version-in-python-1",level:2},{value:"Multiprocess Version in Python (2)",id:"multiprocess-version-in-python-2",level:2},{value:"Practice",id:"practice-1",level:3},{value:"Multithreaded Version in Python",id:"multithreaded-version-in-python",level:2}],c={toc:l},h="wrapper";function d(e){let{components:t,...r}=e;return(0,a.kt)(h,(0,n.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"password-cracker"},"Password Cracker"),(0,a.kt)("p",null,"In this example, we will solve the following problem: given the sha512 hash of a password, we want to obtain the password that generated the hash."),(0,a.kt)("p",null,"Since a hash function is not reversible, one way to solve this problem is by brute-force: generate all possible word combinations, compute the hash for each word, and compare it with our desired hash value.\nThis is not feasible for long passwords, so for our example we will consider only passwords containing lowercase letters and having the length of 4."),(0,a.kt)("p",null,"In order to speed up the entire process, we want to parallelize the solution.\nInstead of one process checking all combinations, we'll split the work among multiple processes or threads."),(0,a.kt)("h2",{id:"multiprocess-version"},"Multiprocess Version"),(0,a.kt)("p",null,"The code for this version is in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/password-cracker/password-cracker-multiprocess.c"),"."),(0,a.kt)("p",null,"The idea is the following: we create 26 worker processes, where each process will consider passwords that start with one particular letter (the first process will brute-force passwords starting with ",(0,a.kt)("inlineCode",{parentName:"p"},"a"),", the second with ",(0,a.kt)("inlineCode",{parentName:"p"},"b"),", and so on)."),(0,a.kt)("p",null,"Since we are using processes, which are naturally isolated, we need a method of communication.\nThe main process should be able to send data to the workers and read back results from them.\nFor this purpose we will use pipes: a pair of 2 pipes between the main process and each worker, one pipe for each direction of communication."),(0,a.kt)("p",null,"In summary, the flow will look like this:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"main process"),(0,a.kt)("ul",{parentName:"li"},(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"create worker processes, along with 2 pipes for each worker (one pipe for requests, one for results)")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"send the 'a' character to the first process request pipe, 'b' to the second, etc.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"read the results from each result pipe")))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"worker process"),(0,a.kt)("ul",{parentName:"li"},(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"read one character from the request pipe")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"generate all words of length 4 that begin with that character")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"for each generated word, compute the sha512 hash and compare it with the desired hash")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"if there is a match, write it to the result pipe"))))),(0,a.kt)("p",null,"Let's build and run the program:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/password-cracker$ make\ngcc -Wall -o password-cracker-multiprocess password-cracker-multiprocess.c -lcrypto\ngcc -Wall -Wno-int-to-pointer-cast -Wno-pointer-to-int-cast -o password-cracker-multithread password-cracker-multithread.c -lcrypto -lpthread\n\nstudent@os:~/.../support/password-cracker$ ./password-cracker-multiprocess\nworker 7 found haxx\n")),(0,a.kt)("h3",{id:"practice"},"Practice"),(0,a.kt)("p",null,"Creating 26 processes is not very realistic, since it's unlikely that a usual machine has that many cores."),(0,a.kt)("p",null,"Modify the program so that it only creates 4 workers.\nEach worker will receive 2 characters instead of one, defining an interval to search.\nFor example, the first worker will receive ",(0,a.kt)("inlineCode",{parentName:"p"},"a")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"f"),", meaning it will brute-force passwords starting with ",(0,a.kt)("inlineCode",{parentName:"p"},"a"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"b"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"c"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"d"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"e"),", or ",(0,a.kt)("inlineCode",{parentName:"p"},"f"),", the second ",(0,a.kt)("inlineCode",{parentName:"p"},"g")," - ",(0,a.kt)("inlineCode",{parentName:"p"},"l"),", and so on."),(0,a.kt)("h2",{id:"multithreaded-version"},"Multithreaded Version"),(0,a.kt)("p",null,"Check out the code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/password-cracker/password-cracker-multithread.c"),"."),(0,a.kt)("p",null,"The core idea of the program is the same, but now we're using threads instead of processes."),(0,a.kt)("p",null,"This makes the communication easier: we'll use the thread function argument to send the first character of the password to each thread.\nAs for the result, each thread will return it as the return value of the thread function."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/password-cracker$ ./password-cracker-multithread\nworker 7 found haxx\n")),(0,a.kt)("h2",{id:"multiprocess-version-in-python-1"},"Multiprocess Version in Python (1)"),(0,a.kt)("p",null,"Code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/password-cracker/python/password-cracker-multiprocess-1.py"),"."),(0,a.kt)("p",null,"This is the Python equivalent of the previous multiprocess version. The program structure is the same, but Python has a few nice features that make our life easier:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"there is a ",(0,a.kt)("inlineCode",{parentName:"p"},"Process")," object that takes a function argument and spawns a new process that begins execution from that function.\nNo need to call ",(0,a.kt)("inlineCode",{parentName:"p"},"fork")," manually.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the ",(0,a.kt)("inlineCode",{parentName:"p"},"Pipe")," object in Python is already bidirectional, unlike the OS pipes, which are unidirectional.\nSo we don't need to create 2 pipes for each direction.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"we don't have to write the code that generates all the password combinations, ",(0,a.kt)("inlineCode",{parentName:"p"},"itertools.product")," will do it for us"))),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/password-cracker$ python3 python/password-cracker-multiprocess-1.py\nworker 7 found haxx\n")),(0,a.kt)("h2",{id:"multiprocess-version-in-python-2"},"Multiprocess Version in Python (2)"),(0,a.kt)("p",null,"Code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/password-cracker/python/password-cracker-multiprocess-2.py"),"."),(0,a.kt)("p",null,"In this case, the code looks different than in the previous examples.\nNow we are taking advantage of some Python constructs, namely ",(0,a.kt)("inlineCode",{parentName:"p"},"process pools"),", which are a collection of worker processes."),(0,a.kt)("p",null,"A ",(0,a.kt)("inlineCode",{parentName:"p"},"Pool")," object has, among others, a function called ",(0,a.kt)("inlineCode",{parentName:"p"},"map"),".\n",(0,a.kt)("inlineCode",{parentName:"p"},"map")," takes a function, together with an array of values, and applies this function on each value from the array.\nAt first glance, it might look like the usual ",(0,a.kt)("inlineCode",{parentName:"p"},"map")," function, but with the key difference that the function application is done by the processes from the pool."),(0,a.kt)("p",null,"In other words, the work is distributed to the worker processes from the pool, and all the communication that we had to handle in the previous examples is done behind the scenes, greatly simplifying the code."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/password-cracker$ python3 python/password-cracker-multiprocess-2.py\nworker 7 found haxx\n")),(0,a.kt)("h3",{id:"practice-1"},"Practice"),(0,a.kt)("p",null,"Check that the ",(0,a.kt)("inlineCode",{parentName:"p"},"worker")," function is indeed called from different worker processes.\nOne simple way to do this is to print out the current process ID at the beginning of the function.\nTo get the current process ID, use the ",(0,a.kt)("inlineCode",{parentName:"p"},"getpid")," function from the ",(0,a.kt)("inlineCode",{parentName:"p"},"os")," module."),(0,a.kt)("h2",{id:"multithreaded-version-in-python"},"Multithreaded Version in Python"),(0,a.kt)("p",null,"Code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/password-cracker/python/password-cracker-multithread.py"),"."),(0,a.kt)("p",null,"The Python equivalent of the previous multithreaded version."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/password-cracker$ python3 python/password-cracker-multithread.py\nworker 7 found haxx\n")),(0,a.kt)("p",null,"This example is given only to provide an idea of how a multithreaded program is written.\nRemember that CPU-bound threads in python don't actually run in parallel, due to the Global Interpreter Lock."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/6e95b748.4ae10100.js b/17/assets/js/6e95b748.4ae10100.js new file mode 100644 index 0000000000..dbc37f964a --- /dev/null +++ b/17/assets/js/6e95b748.4ae10100.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3470],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>b});var r=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var c=r.createContext({}),p=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},u=function(e){var t=p(e.components);return r.createElement(c.Provider,{value:t},e.children)},s="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,i=e.mdxType,a=e.originalType,c=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),s=p(n),d=i,b=s["".concat(c,".").concat(d)]||s[d]||m[d]||a;return n?r.createElement(b,o(o({ref:t},u),{},{components:n})):r.createElement(b,o({ref:t},u))}));function b(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var a=n.length,o=new Array(a);o[0]=d;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[s]="string"==typeof e?e:i,o[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>o,default:()=>m,frontMatter:()=>a,metadata:()=>l,toc:()=>p});var r=n(7462),i=(n(7294),n(3905));const a={},o="Time Server Protocol",l={unversionedId:"Lab/Application Interaction/quiz/time-server",id:"Lab/Application Interaction/quiz/time-server",title:"Time Server Protocol",description:"Question Text",source:"@site/docs/Lab/Application Interaction/quiz/time-server.md",sourceDirName:"Lab/Application Interaction/quiz",slug:"/Lab/Application Interaction/quiz/time-server",permalink:"/operating-systems/17/Lab/Application Interaction/quiz/time-server",draft:!1,tags:[],version:"current",frontMatter:{}},c={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:p},s="wrapper";function m(e){let{components:t,...n}=e;return(0,i.kt)(s,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"time-server-protocol"},"Time Server Protocol"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"What format does the message exchanged between the server and the client have?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"4 byte length (little endian) followed by 8 byte timestamp (little endian)")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"8 byte length (little endian) followed by 4 byte timestamp (little endian)"))),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"4 byte length (big endian) followed by 8 byte timestamp (big endian)")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"8 byte length (big endian) followed by 4 byte timestamp (big endian)")),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,"If we consider one output example:"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-console"},"00000000 00 00 00 08 00 00 00 00 63 1b a9 50 |........c..P|\n0000000c\n")),(0,i.kt)("p",null,"We can identify the 4 byte length in big endian (",(0,i.kt)("inlineCode",{parentName:"p"},"00 00 00 08"),"), then the 8 byte timestamp (",(0,i.kt)("inlineCode",{parentName:"p"},"00 00 00 00 63 1b a9 50"),"), also in big endian."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/70ad9ac2.59b9a25c.js b/17/assets/js/70ad9ac2.59b9a25c.js new file mode 100644 index 0000000000..29b2912aea --- /dev/null +++ b/17/assets/js/70ad9ac2.59b9a25c.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9173],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>f});var r=n(7294);function i(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function l(e){for(var t=1;t=0||(i[n]=e[n]);return i}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(i[n]=e[n])}return i}var s=r.createContext({}),c=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):l(l({},t),e)),n},u=function(e){var t=c(e.components);return r.createElement(s.Provider,{value:t},e.children)},p="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,i=e.mdxType,a=e.originalType,s=e.parentName,u=o(e,["components","mdxType","originalType","parentName"]),p=c(n),d=i,f=p["".concat(s,".").concat(d)]||p[d]||m[d]||a;return n?r.createElement(f,l(l({ref:t},u),{},{components:n})):r.createElement(f,l({ref:t},u))}));function f(e,t){var n=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var a=n.length,l=new Array(a);l[0]=d;var o={};for(var s in t)hasOwnProperty.call(t,s)&&(o[s]=t[s]);o.originalType=e,o[p]="string"==typeof e?e:i,l[1]=o;for(var c=2;c{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>l,default:()=>m,frontMatter:()=>a,metadata:()=>o,toc:()=>c});var r=n(7462),i=(n(7294),n(3905));const a={},l="Time Slice Value",o={unversionedId:"Lab/Compute/quiz/time-slice-value",id:"Lab/Compute/quiz/time-slice-value",title:"Time Slice Value",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/time-slice-value.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/time-slice-value",permalink:"/operating-systems/17/Lab/Compute/quiz/time-slice-value",draft:!1,tags:[],version:"current",frontMatter:{}},s={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:c},p="wrapper";function m(e){let{components:t,...n}=e;return(0,i.kt)(p,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"time-slice-value"},"Time Slice Value"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Using the ",(0,i.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/setitimer.2.html"},"man page"),", what is the time slice used by the scheduler in ",(0,i.kt)("inlineCode",{parentName:"p"},"libult.so"),"?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"100 milliseconds")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"10 microseconds")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"100 microseconds"))),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"10 milliseconds")),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,"The code we're interested in lies in the function ",(0,i.kt)("inlineCode",{parentName:"p"},"init_profiling_timer()"),":"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-C"},"const struct itimerval timer = {\n { 0, 10000 },\n { 0, 1 } // arms the timer as soon as possible\n};\n")),(0,i.kt)("p",null,"The ",(0,i.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/setitimer.2.html"},"man page")," gives the following definition the ",(0,i.kt)("inlineCode",{parentName:"p"},"struct itimerval"),":"),(0,i.kt)("pre",null,(0,i.kt)("code",{parentName:"pre",className:"language-C"},"struct itimerval {\n struct timeval it_interval; /* Interval for periodic timer */\n struct timeval it_value; /* Time until next expiration */\n};\n\nstruct timeval {\n time_t tv_sec; /* seconds */\n suseconds_t tv_usec; /* microseconds */\n};\n")),(0,i.kt)("p",null,"So when constructing the ",(0,i.kt)("inlineCode",{parentName:"p"},"timer")," variable, ",(0,i.kt)("inlineCode",{parentName:"p"},"{ 0, 10000 }")," means 0 seconds and 10000 microseconds, i.e. 0 seconds and 10 milliseconds."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/7e98c1d8.fa314f46.js b/17/assets/js/7e98c1d8.fa314f46.js new file mode 100644 index 0000000000..74ec0248b7 --- /dev/null +++ b/17/assets/js/7e98c1d8.fa314f46.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[5475],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var l=r.createContext({}),s=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=s(e.components);return r.createElement(l.Provider,{value:t},e.children)},p="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,l=e.parentName,c=u(e,["components","mdxType","originalType","parentName"]),p=s(n),d=o,m=p["".concat(l,".").concat(d)]||p[d]||f[d]||a;return n?r.createElement(m,i(i({ref:t},c),{},{components:n})):r.createElement(m,i({ref:t},c))}));function m(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=d;var u={};for(var l in t)hasOwnProperty.call(t,l)&&(u[l]=t[l]);u.originalType=e,u[p]="string"==typeof e?e:o,i[1]=u;for(var s=2;s{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>f,frontMatter:()=>a,metadata:()=>u,toc:()=>s});var r=n(7462),o=(n(7294),n(3905));const a={},i="State of new ULT",u={unversionedId:"Lab/Compute/quiz/state-of-new-ult",id:"Lab/Compute/quiz/state-of-new-ult",title:"State of new ULT",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/state-of-new-ult.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/state-of-new-ult",permalink:"/operating-systems/17/Lab/Compute/quiz/state-of-new-ult",draft:!1,tags:[],version:"current",frontMatter:{}},l={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:s},p="wrapper";function f(e){let{components:t,...n}=e;return(0,o.kt)(p,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"state-of-new-ult"},"State of new ULT"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"What is the first state that is assigned to a newly created ULT?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"RUNNING"),(0,o.kt)("li",{parentName:"ul"},"READY"),(0,o.kt)("li",{parentName:"ul"},"COMPLETED")),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"Inside the function ",(0,o.kt)("inlineCode",{parentName:"p"},"threads_create()"),", we can see the following snippet ",(0,o.kt)("inlineCode",{parentName:"p"},"queue_enqueue(ready, new)"),".\nEach new thread is thus added to the READY queue."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/7f023807.73c1f7db.js b/17/assets/js/7f023807.73c1f7db.js new file mode 100644 index 0000000000..1ea14ae04c --- /dev/null +++ b/17/assets/js/7f023807.73c1f7db.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[820],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>m});var n=r(7294);function i(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t=0||(i[r]=e[r]);return i}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(i[r]=e[r])}return i}var s=n.createContext({}),c=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},p=function(e){var t=c(e.components);return n.createElement(s.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,i=e.mdxType,o=e.originalType,s=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),u=c(r),d=i,m=u["".concat(s,".").concat(d)]||u[d]||f[d]||o;return r?n.createElement(m,a(a({ref:t},p),{},{components:r})):n.createElement(m,a({ref:t},p))}));function m(e,t){var r=arguments,i=t&&t.mdxType;if("string"==typeof e||i){var o=r.length,a=new Array(o);a[0]=d;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[u]="string"==typeof e?e:i,a[1]=l;for(var c=2;c{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>a,default:()=>f,frontMatter:()=>o,metadata:()=>l,toc:()=>c});var n=r(7462),i=(r(7294),r(3905));const o={},a="Client-Server Number of Copies",l={unversionedId:"Lab/IO/quiz/server-copies",id:"Lab/IO/quiz/server-copies",title:"Client-Server Number of Copies",description:"Question Text",source:"@site/docs/Lab/IO/quiz/server-copies.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/server-copies",permalink:"/operating-systems/17/Lab/IO/quiz/server-copies",draft:!1,tags:[],version:"current",frontMatter:{}},s={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:c},u="wrapper";function f(e){let{components:t,...o}=e;return(0,i.kt)(u,(0,n.Z)({},p,o,{components:t,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"client-server-number-of-copies"},"Client-Server Number of Copies"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"The server in the image below uses regular TCP sockets to handle the connection and ",(0,i.kt)("inlineCode",{parentName:"p"},"send()")," to send the data.\nHow many times are the contents of the file copied by the server while being sent to the client?"),(0,i.kt)("p",null,(0,i.kt)("img",{alt:"Client-Server Steps",src:r(2287).Z,width:"502",height:"752"})),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"2")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"1"))),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"4")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},"3")),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,"Rembember double buffering!\nWhen the app calls ",(0,i.kt)("inlineCode",{parentName:"p"},"read()"),", the server's kernel will first save the file to an internal buffer destined for reading.\nThen the app will copy the file to its own buffer.\nFollowing this step, the app will call ",(0,i.kt)("inlineCode",{parentName:"p"},"send()"),", which will first copy the file to a buffer in the kernel.\nFrom this buffer, the kernel itself will copy the file to another buffer on the NIC (Network Interface Card).\nIn total, there the file is copied ",(0,i.kt)("strong",{parentName:"p"},"4 times"),", as outlined in the image below."),(0,i.kt)("p",null,(0,i.kt)("img",{alt:"Server Copies - Read-Send",src:r(4432).Z,width:"572",height:"652"})))}f.isMDXComponent=!0},2287:(e,t,r)=>{r.d(t,{Z:()=>n});const n=r.p+"assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg"},4432:(e,t,r)=>{r.d(t,{Z:()=>n});const n=r.p+"assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/81b85611.4f6edd13.js b/17/assets/js/81b85611.4f6edd13.js new file mode 100644 index 0000000000..9950a55bbd --- /dev/null +++ b/17/assets/js/81b85611.4f6edd13.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4222],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>f});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function o(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var s=a.createContext({}),p=function(e){var t=a.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(s.Provider,{value:t},e.children)},d="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},h=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,s=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),d=p(n),h=r,f=d["".concat(s,".").concat(h)]||d[h]||u[h]||i;return n?a.createElement(f,o(o({ref:t},c),{},{components:n})):a.createElement(f,o({ref:t},c))}));function f(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,o=new Array(i);o[0]=h;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[d]="string"==typeof e?e:r,o[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>u,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const i={},o="File Handling",l={unversionedId:"Lab/IO/file-handlers",id:"Lab/IO/file-handlers",title:"File Handling",description:"You've most likely had to deal with files in the past.",source:"@site/docs/Lab/IO/file-handlers.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/file-handlers",permalink:"/operating-systems/17/Lab/IO/file-handlers",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"I/O",permalink:"/operating-systems/17/Lab/IO/overview"},next:{title:"File Descriptors",permalink:"/operating-systems/17/Lab/IO/file-descriptors"}},s={},p=[{value:"Reaching the File",id:"reaching-the-file",level:2},{value:"Limitations of High-level File Handlers",id:"limitations-of-high-level-file-handlers",level:3}],c={toc:p},d="wrapper";function u(e){let{components:t,...n}=e;return(0,r.kt)(d,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"file-handling"},"File Handling"),(0,r.kt)("p",null,"You've most likely had to deal with files in the past.\nGo to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/simple-file-operations")," and run a most basic command:\n",(0,r.kt)("inlineCode",{parentName:"p"},"cat file.txt"),":"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/simple-file-operations$ cat file.txt\nOS Rullz!\n")),(0,r.kt)("p",null,"But how does ",(0,r.kt)("inlineCode",{parentName:"p"},"cat"),' actually "reach" the file, then read its contents, then print them to standard output?\nThis is part of what we\'re going to learn.'),(0,r.kt)("h2",{id:"reaching-the-file"},"Reaching the File"),(0,r.kt)("p",null,"To manipulate the file (read its contents, modify them, change its size etc.), each process must first get a ",(0,r.kt)("strong",{parentName:"p"},"handler")," to this file.\nThink of this handler as an object by which the process can identify and refer to the file."),(0,r.kt)("p",null,"Now take a look at the code examples in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/simple-file-operations"),".\nEach of them reads the contents of ",(0,r.kt)("inlineCode",{parentName:"p"},"file.txt"),", modifies them, and then reads the previously modified file again.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"make")," to compile the C code, and ",(0,r.kt)("inlineCode",{parentName:"p"},"make java-file-operations")," to compile the Java code."),(0,r.kt)("p",null,"Now run the programs repeatedly in whatever order you wish:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/simple-file-operations$ python3 file_operations.py\nFile contents are: OS Rullz!\nWrote new data to file\nFile contents are: Python was here!\n\nstudent@os:~/.../lab/support/simple-file-operations$ ./file_operations # from the C code\nFile contents are: Python was here!\nWrote new data to file\nFile contents are: C was here!\n\nstudent@os:~/.../lab/support/simple-file-operations$ java FileOperations\nFile contents are: Python was here!\nWrote new data to file\nFile contents are: Java was here!\n")),(0,r.kt)("p",null,"Note that each piece of code creates a variable, which is then used as a handler."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/file-handler-c"},"Quiz")),(0,r.kt)("h3",{id:"limitations-of-high-level-file-handlers"},"Limitations of High-level File Handlers"),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Everything in Linux is a file."),"\nThis statement says that the Linux OS treats every entry in a file system (regular file, directory, block device, char device, link, UNIX socket) as a file.\nThis is very convenient for creating a unified interface that deals with all these files.\nWe would like our file handlers to also be able to handle all types of files."),(0,r.kt)("p",null,"Let's try this.\nNavigate to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/simple-file-operations/directory_operations.c"),".\nRead the code.\nIt does something very simple:\nit attempts to open the ",(0,r.kt)("inlineCode",{parentName:"p"},"dir")," directory the same way ",(0,r.kt)("inlineCode",{parentName:"p"},"file_operations.c")," tried to open ",(0,r.kt)("inlineCode",{parentName:"p"},"file.txt"),".\nCompile and run the code."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/simple-file-operations$ ./directory_operations\n18:18:11 FATAL directory_operations.c:14: fopen: Is a directory\n")),(0,r.kt)("p",null,"The error message is crystal clear: we cannot use ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()")," on directories.\nSo the ",(0,r.kt)("inlineCode",{parentName:"p"},"FILE")," structure is unsuited for directories.\nTherefore, this handler is not generic enough for a regular Linux filesystem."),(0,r.kt)("p",null,"To get to the directory, we need to use a lower-level function.\nLet's take a look at the syscall used by ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()"),"."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/fopen-syscall"},"Quiz")),(0,r.kt)("p",null,"We will use a simpler syscall, called ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,r.kt)("inlineCode",{parentName:"a"},"open()")),".\nIn fact, ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," is a wrapper over ",(0,r.kt)("inlineCode",{parentName:"p"},"openat()"),".\nNavigate to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/file-descriptors/directory_open.c"),".\nInspect, compile and run the code."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/file-descriptors$ ./open_directory\nDirectory file descriptor is: 3\n")),(0,r.kt)("p",null,"We can now see that the ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," syscall is capable of also handling directories, so it's closer to what we want.\nNote that it is rather uncommon to use ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," for directories.\nMost of the time, ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/opendir.3.html"},(0,r.kt)("inlineCode",{parentName:"a"},"opendir()"))," is used instead.\nBut what does it return?\nFind out in ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/file-descriptors"},'the "File Descriptors" section'),"."))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/8209a29d.ae76223f.js b/17/assets/js/8209a29d.ae76223f.js new file mode 100644 index 0000000000..afcbc4c289 --- /dev/null +++ b/17/assets/js/8209a29d.ae76223f.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6074],{3905:(e,t,n)=>{n.d(t,{Zo:()=>d,kt:()=>u});var i=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);t&&(i=i.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,i)}return n}function r(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(i=0;i=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var s=i.createContext({}),l=function(e){var t=i.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):r(r({},t),e)),n},d=function(e){var t=l(e.components);return i.createElement(s.Provider,{value:t},e.children)},c="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return i.createElement(i.Fragment,{},t)}},h=i.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,s=e.parentName,d=p(e,["components","mdxType","originalType","parentName"]),c=l(n),h=a,u=c["".concat(s,".").concat(h)]||c[h]||m[h]||o;return n?i.createElement(u,r(r({ref:t},d),{},{components:n})):i.createElement(u,r({ref:t},d))}));function u(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,r=new Array(o);r[0]=h;var p={};for(var s in t)hasOwnProperty.call(t,s)&&(p[s]=t[s]);p.originalType=e,p[c]="string"==typeof e?e:a,r[1]=p;for(var l=2;l{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>r,default:()=>m,frontMatter:()=>o,metadata:()=>p,toc:()=>l});var i=n(7462),a=(n(7294),n(3905));const o={},r="Pipes",p={unversionedId:"Lab/IO/pipes",id:"Lab/IO/pipes",title:"Pipes",description:"When it comes to inter-process communication, so far we know that 2 different processes can mmap() the same file and use that as some sort of shared memory, but this requires writing data to the disk which is slow.",source:"@site/docs/Lab/IO/pipes.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/pipes",permalink:"/operating-systems/17/Lab/IO/pipes",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Redirections",permalink:"/operating-systems/17/Lab/IO/redirections"},next:{title:"Local I/O in Action",permalink:"/operating-systems/17/Lab/IO/local-io-in-action"}},s={},l=[{value:"Anonymous Pipes - pipe()",id:"anonymous-pipes---pipe",level:2},{value:"Practice: Find the Right Hole File Descriptor",id:"practice-find-the-right-hole-file-descriptor",level:2},{value:"Practice: Inheritance",id:"practice-inheritance",level:3},{value:"Practice: Now We Pipe",id:"practice-now-we-pipe",level:3},{value:"Practice: Receive Pipes",id:"practice-receive-pipes",level:3},{value:"Anonymous Pipes: Conclusion",id:"anonymous-pipes-conclusion",level:3},{value:"Named Pipes - mkfifo()",id:"named-pipes---mkfifo",level:2},{value:"Practice: From the CLI",id:"practice-from-the-cli",level:3},{value:"Practice: From the Code - Receive FIFO",id:"practice-from-the-code---receive-fifo",level:3},{value:"Named Pipes: Conclusion",id:"named-pipes-conclusion",level:3}],d={toc:l},c="wrapper";function m(e){let{components:t,...o}=e;return(0,a.kt)(c,(0,i.Z)({},d,o,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"pipes"},"Pipes"),(0,a.kt)("p",null,"When it comes to inter-process communication, so far we know that 2 different processes can ",(0,a.kt)("inlineCode",{parentName:"p"},"mmap()")," the same file and use that as some sort of shared memory, but this requires writing data to the disk which is slow.\nThen we know they can ",(0,a.kt)("inlineCode",{parentName:"p"},"wait()"),"/",(0,a.kt)("inlineCode",{parentName:"p"},"waitpid()")," for each other to finish, or better yet, use shared semaphores or mutexes, but these mechanisms aren't good at passing data between processes.\nSo our goals for this session are to learn how to use an IPC (inter-process communication) mechanism that:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"allows transfers between processes, not notifications")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"is faster than reading and writing from/to files"))),(0,a.kt)("h2",{id:"anonymous-pipes---pipe"},"Anonymous Pipes - ",(0,a.kt)("inlineCode",{parentName:"h2"},"pipe()")),(0,a.kt)("p",null,"Have you ever wondered how Bash handles redirecting the ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," of a command to the ",(0,a.kt)("inlineCode",{parentName:"p"},"stdin")," of the next command in one-liners such as:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-bash"},"cat 'log_*.csv' | tr -s ' ' | cut -d ',' -f 2 | sort -u | head -n 10\n")),(0,a.kt)("p",null,"The ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," of ",(0,a.kt)("inlineCode",{parentName:"p"},"cat")," is the ",(0,a.kt)("inlineCode",{parentName:"p"},"stdin")," of ",(0,a.kt)("inlineCode",{parentName:"p"},"tr"),", whose ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," is the ",(0,a.kt)("inlineCode",{parentName:"p"},"stdin")," of ",(0,a.kt)("inlineCode",{parentName:"p"},"cut"),' and so on.\nThis "chain" of commands looks like this:'),(0,a.kt)("p",null,(0,a.kt)("img",{alt:"Piped Commands",src:n(9199).Z,width:"762",height:"82"})),(0,a.kt)("p",null,"So here we have a ",(0,a.kt)("strong",{parentName:"p"},"unidirectional")," stream of data that starts from ",(0,a.kt)("inlineCode",{parentName:"p"},"cat"),", is modified by each new command, and then is passed to the next one.\nWe can tell from the image above that the communication channel between any 2 adjacent commands allows one process to write to it while the other reads from it.\nFor example, there is no need for ",(0,a.kt)("inlineCode",{parentName:"p"},"cat")," to read any of ",(0,a.kt)("inlineCode",{parentName:"p"},"tr"),"'s output, only vice versa."),(0,a.kt)("p",null,"Therefore, this communication channel needs 2 ends:\none for reading (from which commands get their input) and another for writing (to which commands write their output).\nIn UNIX, the need for such a channel is fulfilled by the ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/pipe.2.html"},(0,a.kt)("inlineCode",{parentName:"a"},"pipe()")," syscall"),".\nImagine there's a literal pipe between any 2 adjacent commands in the image above, where data is what flows through this pipe ",(0,a.kt)("strong",{parentName:"p"},"in only a single way"),".\nThis is why the ",(0,a.kt)("inlineCode",{parentName:"p"},"|")," operator in Bash is called pipe and why the syscall is also named ",(0,a.kt)("inlineCode",{parentName:"p"},"pipe()"),"."),(0,a.kt)("p",null,"This type of pipe is also called an ",(0,a.kt)("strong",{parentName:"p"},"anonymous pipe"),", because it cannot be identified using a name (i.e. it is not backed by any file).\nThe data written to it is kept in a circular buffer inside the kernel from where it can be then read by the child process.\nThis is faster than writing data to a file, so we achieve both our ",(0,a.kt)("a",{parentName:"p",href:"#pipes"},"initial goals"),"."),(0,a.kt)("h2",{id:"practice-find-the-right-hole-file-descriptor"},"Practice: Find the Right ",(0,a.kt)("del",{parentName:"h2"},"Hole")," File Descriptor"),(0,a.kt)("p",null,"Navigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/pipes/anonymous_pipe.c"),".\nCompile and run the code.\nIn another terminal, use ",(0,a.kt)("inlineCode",{parentName:"p"},"lsof")," to see:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the file descriptors opened by the parent process between the creation of the pipe and the call to ",(0,a.kt)("inlineCode",{parentName:"p"},"fork()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"the file descriptors opened by the child process"))),(0,a.kt)("p",null,(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/pipe-ends"},"Quiz")),(0,a.kt)("p",null,"A simple way to memorise which pipe end is which is to think about ",(0,a.kt)("inlineCode",{parentName:"p"},"stdin")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout"),", respectively.\n",(0,a.kt)("inlineCode",{parentName:"p"},"stdin")," is file descriptor 0 and is mostly for reading and ",(0,a.kt)("inlineCode",{parentName:"p"},"pipedes[0]")," is also for reading.\nConversely, ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," is file descriptor 1 and is meant for writing, just like ",(0,a.kt)("inlineCode",{parentName:"p"},"pipedes[1]"),".\nNow you won't confuse them."),(0,a.kt)("h3",{id:"practice-inheritance"},"Practice: Inheritance"),(0,a.kt)("p",null,"An important thing to take note of before we actually use pipes is that file descriptors are ",(0,a.kt)("strong",{parentName:"p"},"inherited")," by the child process from the parent.\nSo if the parent opens some file descriptors (like, say, for a pipe), the child will also be able to use them.\nDon't believe us?"),(0,a.kt)("p",null,"Modify the code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/pipes/anonymous_pipes.c")," and call ",(0,a.kt)("inlineCode",{parentName:"p"},"wait_for_input()")," from the child process.\nThen use ",(0,a.kt)("inlineCode",{parentName:"p"},"lsof")," again with the PID of the child process to make sure file descriptors 3 and 4 are still open."),(0,a.kt)("h3",{id:"practice-now-we-pipe"},"Practice: Now We Pipe"),(0,a.kt)("p",null,"Now comes the moment you've most likely been waiting for.\nThe code in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/pipes/anonymous_pipes.c")," wants to create something like a client-server dynamic between the parent and the child.\nThe parent reads data from ",(0,a.kt)("inlineCode",{parentName:"p"},"stdin")," and sends it to the child via the pipe they share.\nThe client (parent) ends communication when you type ",(0,a.kt)("inlineCode",{parentName:"p"},'"exit"'),"."),(0,a.kt)("p",null,"You can comment out the calls to ",(0,a.kt)("inlineCode",{parentName:"p"},"wait_for_input()")," if you find them annoying."),(0,a.kt)("p",null,"Sample output:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/pipes$ ./anonymous_pipe\npipedes[0] = 3; pipedes[1] = 4\n * pipe created\n -- Press ENTER to continue ...\necho\n[Child received]: echo\necho\n[Child received]: echo\nto pipe, or not to pipe\n[Child received]: to pipe, or not to pipe\n")),(0,a.kt)("h3",{id:"practice-receive-pipes"},"Practice: Receive Pipes"),(0,a.kt)("p",null,"Use your knowledge of pipes to solve a CTF challenge.\nNavigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/receive-challenges")," and look into the files ",(0,a.kt)("inlineCode",{parentName:"p"},"receive_pipe.c")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"send_fd_4.c"),".\nModify ",(0,a.kt)("inlineCode",{parentName:"p"},"receive_pipe.c")," so that it creates a pipe, then spawns a child process.\nThe child will redirect file descriptor 4 to ",(0,a.kt)("inlineCode",{parentName:"p"},"stdout")," and then ",(0,a.kt)("inlineCode",{parentName:"p"},"execlp()")," ",(0,a.kt)("inlineCode",{parentName:"p"},"send_fd_4"),".\n",(0,a.kt)("inlineCode",{parentName:"p"},"send_fd_4")," writes the flag to file descriptor 4 (",(0,a.kt)("inlineCode",{parentName:"p"},"pipefd[1]"),"), so the parent process needs to read it from ",(0,a.kt)("inlineCode",{parentName:"p"},"pipedefd[0]"),"."),(0,a.kt)("p",null,"Once you do this, note that file descriptors are also maintained after calling ",(0,a.kt)("inlineCode",{parentName:"p"},"exec()")," to run a completely new program."),(0,a.kt)("p",null,"Now, if you want to use pipes even more, go over to ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/arena#mini-shell-with-blackjack-and-pipes"},"the Arena")," and add support for pipes to the mini-shell you've previously worked on."),(0,a.kt)("h3",{id:"anonymous-pipes-conclusion"},"Anonymous Pipes: Conclusion"),(0,a.kt)("p",null,"Anonymous pipes give allow us to efficiently transmit data between 2 processes, as no disk access is required.\nHowever, they still have one major limitation."),(0,a.kt)("p",null,(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/anonymous-pipes-limitation"},"Quiz")),(0,a.kt)("p",null,"The answer to this is to employ ",(0,a.kt)("em",{parentName:"p"},"some")," filesystem support."),(0,a.kt)("h2",{id:"named-pipes---mkfifo"},"Named Pipes - ",(0,a.kt)("inlineCode",{parentName:"h2"},"mkfifo()")),(0,a.kt)("p",null,"We will give up on some performance by writing data to a file, but the reading and writing to the file must behave just like using a named pipe:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"reading doesn't block while there's still data in the pipe"),(0,a.kt)("li",{parentName:"ul"},"reading from an empty pipe stalls the current thread until data becomes available.\nThis is one of the cases where ",(0,a.kt)("inlineCode",{parentName:"li"},"read()")," might not return as many bytes as we requested.\nRemember:\nalways use loops with ",(0,a.kt)("inlineCode",{parentName:"li"},"read()")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"write()"))),(0,a.kt)("p",null,"Because this pipe uses a file, which must have a name, it is called a ",(0,a.kt)("strong",{parentName:"p"},"named pipe"),"."),(0,a.kt)("h3",{id:"practice-from-the-cli"},"Practice: From the CLI"),(0,a.kt)("p",null,"First, let's use named pipes from the terminal.\nUse the ",(0,a.kt)("inlineCode",{parentName:"p"},"mkfifo")," command to create one such file:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ mkfifo fifo\n\nstudent@os:~$ ls -l fifo\nprw-rw-r-- 1 student student 0 Nov 22 23:20 fifo|\n")),(0,a.kt)("p",null,"The fact that pipes are also called FIFOs should come as no surprise.\nThey act like queues/FIFOs:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"you add data to one end (push/enqueue)"),(0,a.kt)("li",{parentName:"ul"},"you extract data from the other (pop/dequeue)")),(0,a.kt)("p",null,"Also note the ",(0,a.kt)("inlineCode",{parentName:"p"},"p")," at the beginning of the output above.\nIt symbolises the type of this file:\na named ",(0,a.kt)("strong",{parentName:"p"},"pipe"),"."),(0,a.kt)("p",null,"Now let's use it.\nOpen 2 terminals."),(0,a.kt)("ol",null,(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"In the first one, use ",(0,a.kt)("inlineCode",{parentName:"p"},"cat")," to read from ",(0,a.kt)("inlineCode",{parentName:"p"},"fifo"),".\nNote that the command is blocked because the pipe is empty, so ",(0,a.kt)("inlineCode",{parentName:"p"},"cat")," has nothing to read.\nThen, in the second terminal, write some message to ",(0,a.kt)("inlineCode",{parentName:"p"},"fifo")," using ",(0,a.kt)("inlineCode",{parentName:"p"},"echo"),".\nIn the first terminal, you should see that ",(0,a.kt)("inlineCode",{parentName:"p"},"cat")," has finished, and the message has appeared there.")),(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Now do it the other way around:\nfirst ",(0,a.kt)("inlineCode",{parentName:"p"},"echo")," some string into the pipe and ",(0,a.kt)("strong",{parentName:"p"},"then")," read it with ",(0,a.kt)("inlineCode",{parentName:"p"},"cat"),".\nNote that now the ",(0,a.kt)("inlineCode",{parentName:"p"},"echo")," command is blocked.\nNow ",(0,a.kt)("inlineCode",{parentName:"p"},"cat")," should end immediately, and the string should appear because we have already placed some data in the pipe.\nAlso, the previous ",(0,a.kt)("inlineCode",{parentName:"p"},"echo")," should finish now."))),(0,a.kt)("p",null,(0,a.kt)("strong",{parentName:"p"},"Remember:"),"\n",(0,a.kt)("strong",{parentName:"p"},"Reading from a pipe blocks while the pipe is empty."),"\n",(0,a.kt)("strong",{parentName:"p"},"Writing to a pipe blocks until it is empty.")),(0,a.kt)("h3",{id:"practice-from-the-code---receive-fifo"},"Practice: From the Code - Receive FIFO"),(0,a.kt)("p",null,"The libc function with which we can create named pipes is...\n",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/mkfifo.3.html"},(0,a.kt)("inlineCode",{parentName:"a"},"mkfifo()")),".\nYou weren't expecting this, were you?\nIts underlying syscall is ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/mknodat.2.html"},(0,a.kt)("inlineCode",{parentName:"a"},"mknod()")),", which simply creates a file of whatever type we specify, but that's besides the point."),(0,a.kt)("p",null,"Navigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/receive-challenges/")," and look into ",(0,a.kt)("inlineCode",{parentName:"p"},"receive_fifo.c")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"send_fifo.c"),".\nFollow the ",(0,a.kt)("inlineCode",{parentName:"p"},"TODO"),"s in the former file to read the flag that the latter writes into the named pipe.\nNote that after you create the named pipe, you have to read from it like you would from any regular file."),(0,a.kt)("h3",{id:"named-pipes-conclusion"},"Named Pipes: Conclusion"),(0,a.kt)("p",null,"It is nice to remember that named pipes sacrifice little to no performance when compared to anonymous pipes.\nWhile it may seem that the data being passed through them is written to the disk, then read and overwritten, this is not the case.\nThe FIFO file is just a handler within the filesystem that is used to write data to a buffer inside the kernel.\nThis buffer holds the data that is passed between processes, not the filesystem.\nSo we still don't break our 2 desires from the beginning of this section:\nto allow data transfer and to do so efficiently."))}m.isMDXComponent=!0},9199:(e,t,n)=>{n.d(t,{Z:()=>i});const i=n.p+"assets/images/piped-commands-118a36fba312c6bea5270dba74d653e2.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/83016638.fa6fb28f.js b/17/assets/js/83016638.fa6fb28f.js new file mode 100644 index 0000000000..d7fe276514 --- /dev/null +++ b/17/assets/js/83016638.fa6fb28f.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8755],{2172:e=>{e.exports=JSON.parse('{"title":"Software Stack","slug":"/Lab/Software Stack/","permalink":"/operating-systems/17/Lab/Software Stack/","navigation":{"previous":{"title":"Setting up the Lab Environment","permalink":"/operating-systems/17/Lab/lab-setup"},"next":{"title":"Software Stack","permalink":"/operating-systems/17/Lab/Software Stack/overview"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/87bfa638.c07f8ced.js b/17/assets/js/87bfa638.c07f8ced.js new file mode 100644 index 0000000000..819fe59660 --- /dev/null +++ b/17/assets/js/87bfa638.c07f8ced.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7986],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>m});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var s=n.createContext({}),l=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},p=function(e){var t=l(e.components);return n.createElement(s.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,s=e.parentName,p=c(e,["components","mdxType","originalType","parentName"]),u=l(r),d=o,m=u["".concat(s,".").concat(d)]||u[d]||f[d]||a;return r?n.createElement(m,i(i({ref:t},p),{},{components:r})):n.createElement(m,i({ref:t},p))}));function m(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,i=new Array(a);i[0]=d;var c={};for(var s in t)hasOwnProperty.call(t,s)&&(c[s]=t[s]);c.originalType=e,c[u]="string"==typeof e?e:o,i[1]=c;for(var l=2;l{r.r(t),r.d(t,{assets:()=>l,contentTitle:()=>c,default:()=>d,frontMatter:()=>i,metadata:()=>s,toc:()=>p});var n=r(7462),o=(r(7294),r(3905)),a=r(4996);const i={title:"Software-Stack"},c=void 0,s={unversionedId:"Lecture/Software-Stack",id:"Lecture/Software-Stack",title:"Software-Stack",description:"Focus the slides and press F for fullscreen viewing.",source:"@site/docs/Lecture/Software-Stack.mdx",sourceDirName:"Lecture",slug:"/Lecture/Software-Stack",permalink:"/operating-systems/17/Lecture/Software-Stack",draft:!1,tags:[],version:"current",frontMatter:{title:"Software-Stack"},sidebar:"sidebar",previous:{title:"Lecture",permalink:"/operating-systems/17/Lecture/"},next:{title:"Data",permalink:"/operating-systems/17/Lecture/Data"}},l={},p=[],u={toc:p},f="wrapper";function d(e){let{components:t,...r}=e;return(0,o.kt)(f,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("div",{style:{display:"flex",width:"100%",height:"100%",flexDirection:"row"}},(0,o.kt)("iframe",{style:{flexGrow:1,border:"none",margin:0,padding:0},width:"100%",height:"500px",src:(0,a.Z)("/slides/Software-Stack/index.html")})),(0,o.kt)("br",null),(0,o.kt)("admonition",{type:"tip"},(0,o.kt)("p",{parentName:"admonition"},"Focus the slides and press ",(0,o.kt)("strong",{parentName:"p"},"F")," for fullscreen viewing.")))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/89d3ad7d.7a961b19.js b/17/assets/js/89d3ad7d.7a961b19.js new file mode 100644 index 0000000000..65a3630742 --- /dev/null +++ b/17/assets/js/89d3ad7d.7a961b19.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4148],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>m});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var s=r.createContext({}),c=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},p=function(e){var t=c(e.components);return r.createElement(s.Provider,{value:t},e.children)},u="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,s=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),u=c(n),f=a,m=u["".concat(s,".").concat(f)]||u[f]||d[f]||i;return n?r.createElement(m,o(o({ref:t},p),{},{components:n})):r.createElement(m,o({ref:t},p))}));function m(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,o=new Array(i);o[0]=f;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[u]="string"==typeof e?e:a,o[1]=l;for(var c=2;c{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>d,frontMatter:()=>i,metadata:()=>l,toc:()=>c});var r=n(7462),a=(n(7294),n(3905));const i={},o="Syscalls Used by `cp`",l={unversionedId:"Lab/IO/quiz/syscalls-cp",id:"Lab/IO/quiz/syscalls-cp",title:"Syscalls Used by `cp`",description:"Question Text",source:"@site/docs/Lab/IO/quiz/syscalls-cp.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/syscalls-cp",permalink:"/operating-systems/17/Lab/IO/quiz/syscalls-cp",draft:!1,tags:[],version:"current",frontMatter:{}},s={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:c},u="wrapper";function d(e){let{components:t,...n}=e;return(0,a.kt)(u,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"syscalls-used-by-cp"},"Syscalls Used by ",(0,a.kt)("inlineCode",{parentName:"h1"},"cp")),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What syscalls does ",(0,a.kt)("inlineCode",{parentName:"p"},"cp")," use to copy files?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"mmap()"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"read()")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"write()"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"a combination of ",(0,a.kt)("inlineCode",{parentName:"li"},"read()")," - ",(0,a.kt)("inlineCode",{parentName:"li"},"write()")," and ",(0,a.kt)("inlineCode",{parentName:"li"},"mmap()"))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"It suffices to use ",(0,a.kt)("inlineCode",{parentName:"p"},"strace")," to see that ",(0,a.kt)("inlineCode",{parentName:"p"},"cp")," uses ",(0,a.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"write()"),"."),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},'student@os:/.../support/file-mappings$ strace cp test-file.txt output.txt\nopenat(AT_FDCWD, "test-file.txt", O_RDONLY) = 3\nfstat(3, {st_mode=S_IFREG|0664, st_size=1048576, ...}) = 0\nopenat(AT_FDCWD, "output.txt", O_WRONLY|O_CREAT|O_EXCL, 0664) = 4\nfstat(4, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0\n[...]\nread(3, "@Y\\344\\0025\\317\\27\\243\\23\\201:\\27\\342\\356\\240\\345\\331Nq\\v/\\36\\244\\200\\301\\247\\3152\\35WZ\\337"..., 131072) = 131072\nwrite(4, "@Y\\344\\0025\\317\\27\\243\\23\\201:\\27\\342\\356\\240\\345\\331Nq\\v/\\36\\244\\200\\301\\247\\3152\\35WZ\\337"..., 131072) = 131072\nread(3, "\\201\\240J7x\\275\\257Z\\343\\334\\307d<\\321U\\275\\337\\10\\233j\\222\\313,##cQD\\268e\\324"..., 131072) = 131072\nwrite(4, "\\201\\240J7x\\275\\257Z\\343\\334\\307d<\\321U\\275\\337\\10\\233j\\222\\313,##cQD\\268e\\324"..., 131072) = 131072\nread(3, "\\371\\3244=\\17\\300L9\\243\\201\\362\\25\\273\\37\\326\\323\\362\\200\\1T\\310N\\316\\305\\253\\331\\331Nt\\346\\3369"..., 131072) = 131072\nwrite(4, "\\371\\3244=\\17\\300L9\\243\\201\\362\\25\\273\\37\\326\\323\\362\\200\\1T\\310N\\316\\305\\253\\331\\331Nt\\346\\3369"..., 131072) = 131072\nread(3, "\\350\\304\\345f\\16\\305V\\356\\371\\263?+\\355{\\16\\235\\344\\310P4}\\2043%\\0052\\345D\\265\\243t\\354"..., 131072) = 131072\nwrite(4, "\\350\\304\\345f\\16\\305V\\356\\371\\263?+\\355{\\16\\235\\344\\310P4}\\2043%\\0052\\345D\\265\\243t\\354"..., 131072) = 131072\nread(3, "\\277\\226\\315\\226\\n\\37\\337;N*\\211\\352\\254$\\273\\2\\351\\30a\\254\\ta\\352R\\25-\\23\\274\\376zy\\211"..., 131072) = 131072\nwrite(4, "\\277\\226\\315\\226\\n\\37\\337;N*\\211\\352\\254$\\273\\2\\351\\30a\\254\\ta\\352R\\25-\\23\\274\\376zy\\211"..., 131072) = 131072\nread(3, "}\\245\\356;\\222\\327\\204\\242u\\26dy%\\346\\374\\201ndT\\226\\233\\3575\\345\\247\\356\\362\\344\\350\\354\\17\\261"..., 131072) = 131072\nwrite(4, "}\\245\\356;\\222\\327\\204\\242u\\26dy%\\346\\374\\201ndT\\226\\233\\3575\\345\\247\\356\\362\\344\\350\\354\\17\\261"..., 131072) = 131072\nread(3, "\\35\\277\\207\\243~\\355(i\\351^\\1\\346\\312V\\232\\204\\32\\230~\\376\\20\\245\\"\\305\\344d\\304\\304B\\272\\346\\332"..., 131072) = 131072\nwrite(4, "\\35\\277\\207\\243~\\355(i\\351^\\1\\346\\312V\\232\\204\\32\\230~\\376\\20\\245\\"\\305\\344d\\304\\304B\\272\\346\\332"..., 131072) = 131072\nread(3, "\\n)\\334\\275\\331:R\\236O\\231\\243\\302\\314\\267\\326\\"\\rY\\262\\21\\374\\305\\275\\3\\332\\23\\345\\16>\\214\\210\\235"..., 131072) = 131072\nwrite(4, "\\n)\\334\\275\\331:R\\236O\\231\\243\\302\\314\\267\\326\\"\\rY\\262\\21\\374\\305\\275\\3\\332\\23\\345\\16>\\214\\210\\235"..., 131072) = 131072\n')))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/8ca589d3.c1ceb508.js b/17/assets/js/8ca589d3.c1ceb508.js new file mode 100644 index 0000000000..d92c6bf907 --- /dev/null +++ b/17/assets/js/8ca589d3.c1ceb508.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2827],{3905:(e,t,i)=>{i.d(t,{Zo:()=>p,kt:()=>b});var n=i(7294);function a(e,t,i){return t in e?Object.defineProperty(e,t,{value:i,enumerable:!0,configurable:!0,writable:!0}):e[t]=i,e}function l(e,t){var i=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),i.push.apply(i,n)}return i}function r(e){for(var t=1;t=0||(a[i]=e[i]);return a}(e,t);if(Object.getOwnPropertySymbols){var l=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,i)&&(a[i]=e[i])}return a}var o=n.createContext({}),d=function(e){var t=n.useContext(o),i=t;return e&&(i="function"==typeof e?e(t):r(r({},t),e)),i},p=function(e){var t=d(e.components);return n.createElement(o.Provider,{value:t},e.children)},c="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},I=n.forwardRef((function(e,t){var i=e.components,a=e.mdxType,l=e.originalType,o=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),c=d(i),I=a,b=c["".concat(o,".").concat(I)]||c[I]||m[I]||l;return i?n.createElement(b,r(r({ref:t},p),{},{components:i})):n.createElement(b,r({ref:t},p))}));function b(e,t){var i=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var l=i.length,r=new Array(l);r[0]=I;var s={};for(var o in t)hasOwnProperty.call(t,o)&&(s[o]=t[o]);s.originalType=e,s[c]="string"==typeof e?e:a,r[1]=s;for(var d=2;d{i.r(t),i.d(t,{assets:()=>o,contentTitle:()=>r,default:()=>m,frontMatter:()=>l,metadata:()=>s,toc:()=>d});var n=i(7462),a=(i(7294),i(3905));const l={},r="I/O Internals",s={unversionedId:"Lab/IO/io-internals",id:"Lab/IO/io-internals",title:"I/O Internals",description:"Now, we will take a short look at how the file descriptors you've just learnt about are handled in libc.",source:"@site/docs/Lab/IO/io-internals.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/io-internals",permalink:"/operating-systems/17/Lab/IO/io-internals",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"File Mappings",permalink:"/operating-systems/17/Lab/IO/file-mappings"},next:{title:"Zero-Copy",permalink:"/operating-systems/17/Lab/IO/zero-copy"}},o={},d=[{value:"IO Optimisations",id:"io-optimisations",level:2},{value:"I/O Buffering",id:"io-buffering",level:3},{value:"Practice: printf() Buffering",id:"practice-printf-buffering",level:4},{value:"Practice: Buffering Performance",id:"practice-buffering-performance",level:4},{value:"Practice: DIY Buffering",id:"practice-diy-buffering",level:3},{value:"Conclusion",id:"conclusion",level:2}],p={toc:d},c="wrapper";function m(e){let{components:t,...l}=e;return(0,a.kt)(c,(0,n.Z)({},p,l,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"io-internals"},"I/O Internals"),(0,a.kt)("p",null,"Now, we will take a short look at how the file descriptors you've just learnt about are handled in libc.\nThe Software Stack chapter has taught us that applications generally interact with libraries which expose wrappers on top of syscalls.\nThe most important library in a POSIX system (such as Linux) is libc.\nAmong many others, it provides higher-level abstractions over I/O-related syscalls."),(0,a.kt)("p",null,(0,a.kt)("strong",{parentName:"p"},"Musl"),' (read just like "muscle") is a lightweight implementation of libc, which exposes the same API that you have used so far, while also being fit for embedded and OS development.\nFor example, ',(0,a.kt)("a",{parentName:"p",href:"https://unikraft.org/"},"Unikraft")," ",(0,a.kt)("a",{parentName:"p",href:"https://unikraft.org/docs/concepts/"},"unikernels")," may ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/unikraft/lib-musl"},"use musl"),"."),(0,a.kt)("p",null,"First, it provides a ",(0,a.kt)("inlineCode",{parentName:"p"},"struct")," that groups together multiple data that is necessary when handling files.\nWe know from the example in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/simple-file-operations/file_operations.c")," that the file handler employed by libc is ",(0,a.kt)("inlineCode",{parentName:"p"},"FILE *"),".\n",(0,a.kt)("inlineCode",{parentName:"p"},"FILE")," is just a ",(0,a.kt)("inlineCode",{parentName:"p"},"typedef")," for ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/internal/stdio_impl.h#L21"},(0,a.kt)("inlineCode",{parentName:"a"},"struct _IO_FILE")),".\nHere are the most important fields in ",(0,a.kt)("inlineCode",{parentName:"p"},"struct _IO_FILE"),":"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},"struct _IO_FILE {\n int fd; /* File descriptor */\n\n unsigned flags; /* Flags with which `open()` was called */\n\n int mode; /* File permissions; passed to `open()` */\n\n off_t off; /* File offset from where to read / write */\n\n /**\n * Internal buffer used to make fewer costly `read()`/`write()`\n * syscalls.\n */\n unsigned char *buf;\n size_t buf_size;\n\n /* Pointers for reading and writing from/to the buffer defined above. */\n unsigned char *rpos, *rend;\n unsigned char *wend, *wpos;\n\n /* Function pointers to syscall wrappers. */\n size_t (*read)(FILE *, unsigned char *, size_t);\n size_t (*write)(FILE *, const unsigned char *, size_t);\n off_t (*seek)(FILE *, off_t, int);\n int (*close)(FILE *);\n\n /* Lock for concurrent file access. */\n volatile int lock;\n};\n")),(0,a.kt)("p",null,"As you might have imagined, this structure contains the underlying file descriptor, the ",(0,a.kt)("inlineCode",{parentName:"p"},"mode")," (read, write, truncate etc.) with which the file was opened, as well as the offset within the file from which the next read / write will start."),(0,a.kt)("p",null,"Libc also defines its own wrappers over commonly-used syscalls, such as ",(0,a.kt)("inlineCode",{parentName:"p"},"read()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"write()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"close()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"lseek()"),".\nThese syscalls themselves need to be implemented by the driver for each file system.\nThis is done by writing the required functions for each syscall and then populating ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/linux/v6.0.9/source/include/linux/fs.h#L2093"},"this structure")," with pointers to them.\nYou will recognise quite a few syscalls: ",(0,a.kt)("inlineCode",{parentName:"p"},"open()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"close()")," ",(0,a.kt)("inlineCode",{parentName:"p"},"read()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"write()"),", ",(0,a.kt)("inlineCode",{parentName:"p"},"mmap()")," etc."),(0,a.kt)("h2",{id:"io-optimisations"},"IO Optimisations"),(0,a.kt)("p",null,"You saw this hierarchy during the Data lecture:"),(0,a.kt)("p",null,(0,a.kt)("img",{alt:"Memory Hierarchy",src:i(4132).Z,width:"711",height:"371"})),(0,a.kt)("p",null,"It says that while the disk can store lots of data, it does so at the cost of speed.\nWhen we say speed, we mean the rate at which we can read/write data from/to the disk, i.e. the maximum number of bytes transferred per unit of time.\nThis means that ",(0,a.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"write()"),' syscalls (or their various corresponding library calls) are slow and cause performance bottlenecks.\nMore often than not, it is the I/O component that drags down the performance of an otherwise fast application.\nAnd what\'s worse, the further the "destination" of the I/O operation (file on the disk or host on the Web) is, the more time it takes to transfer data to and from it.'),(0,a.kt)("p",null,"On the other hand, as we've already established, the I/O component defines how we interact with an app.\nIf we want it to be responsive and to do something useful, most likely, the I/O is the key."),(0,a.kt)("p",null,"So I/O is crucial for most applications, yet it is also the slowest..."),(0,a.kt)("p",null,(0,a.kt)("img",{alt:"Sad Pepe",src:i(6585).Z,width:"382",height:"362"})),(0,a.kt)("p",null,"But fear not!\nThere are countless optimisations out there aimed precisely at bridging the speed gap between the memory and the disk."),(0,a.kt)("h3",{id:"io-buffering"},"I/O Buffering"),(0,a.kt)("p",null,"Going back to our initial example with ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/internal/stdio_impl.h#L21"},(0,a.kt)("inlineCode",{parentName:"a"},"struct _IO_FILE"))," from Musl, we can see some more fields:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},"unsigned char *buf;\nsize_t buf_size;\nunsigned char *rpos, *rend;\nunsigned char *wend, *wpos;\n")),(0,a.kt)("p",null,"Given the number ",(0,a.kt)("inlineCode",{parentName:"p"},"unsigned char *")," fields we have in this structure, it seems there is some heavy ",(0,a.kt)("em",{parentName:"p"},"buffering")," going on.\nBut what is it?"),(0,a.kt)("h4",{id:"practice-printf-buffering"},"Practice: ",(0,a.kt)("inlineCode",{parentName:"h4"},"printf()")," Buffering"),(0,a.kt)("ol",null,(0,a.kt)("li",{parentName:"ol"},"Navigate to ",(0,a.kt)("inlineCode",{parentName:"li"},"support/buffering/printf_buffering.c"),".\nThose ",(0,a.kt)("inlineCode",{parentName:"li"},"printf()")," calls obviously end up calling ",(0,a.kt)("inlineCode",{parentName:"li"},"write()")," at some point.\nRun the code under ",(0,a.kt)("inlineCode",{parentName:"li"},"strace"),".")),(0,a.kt)("p",null,(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/strace-printf"},"Quiz")),(0,a.kt)("p",null,"Since there is only one ",(0,a.kt)("inlineCode",{parentName:"p"},"write()")," syscall despite multiple calls to ",(0,a.kt)("inlineCode",{parentName:"p"},"printf()"),", it means that the strings given to ",(0,a.kt)("inlineCode",{parentName:"p"},"printf()")," as arguments are kept ",(0,a.kt)("em",{parentName:"p"},"somewhere")," until the syscall is made.\nThat ",(0,a.kt)("em",{parentName:"p"},"somewhere")," is precisely that buffer inside ",(0,a.kt)("inlineCode",{parentName:"p"},"struct _IO_FILE")," that we highlighted above.\nRemember that syscalls cause the system to change from user mode to kernel mode, which is time-consuming.\nInstead of performing one ",(0,a.kt)("inlineCode",{parentName:"p"},"write()")," syscall per call to ",(0,a.kt)("inlineCode",{parentName:"p"},"printf()"),", it is more efficient to copy the string passed to ",(0,a.kt)("inlineCode",{parentName:"p"},"printf()")," to an ",(0,a.kt)("strong",{parentName:"p"},"internal buffer")," inside libc (the ",(0,a.kt)("inlineCode",{parentName:"p"},"unsigned char *buf")," from above) and then at a given time (like when the buffer is full for example) ",(0,a.kt)("inlineCode",{parentName:"p"},"write()")," the whole buffer.\nThis results in far fewer ",(0,a.kt)("inlineCode",{parentName:"p"},"write()")," syscalls."),(0,a.kt)("ol",{start:2},(0,a.kt)("li",{parentName:"ol"},"Now, it is interesting to see how we can force libc to dump that internal buffer.\nThe most direct way is by using the ",(0,a.kt)("inlineCode",{parentName:"li"},"fflush()")," library call, which is made for this exact purpose.\nBut we can be more subtle.\nAdd a ",(0,a.kt)("inlineCode",{parentName:"li"},"\\n")," in some of the strings printed in ",(0,a.kt)("inlineCode",{parentName:"li"},"support/buffering/printf_buffering.c"),".\nPlace them wherever you want (at the beginning, at the end, in the middle).\nRecompile the code and observe its change in behaviour under ",(0,a.kt)("inlineCode",{parentName:"li"},"strace"),".")),(0,a.kt)("p",null,(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/flush-libc-buffer"},"Quiz")),(0,a.kt)("p",null,"Now we know that I/O buffering ",(0,a.kt)("strong",{parentName:"p"},"does happen")," within libc.\nIf you need further convincing, check out the Musl implementation of ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L6"},(0,a.kt)("inlineCode",{parentName:"a"},"fread()")),", for example.\nIt first copies the ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L16"},"data previously saved in the internal buffer"),":"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-c"},"if (f->rpos != f->rend) {\n /* First exhaust the buffer. */\n k = MIN(f->rend - f->rpos, l);\n memcpy(dest, f->rpos, k);\n f->rpos += k;\n dest += k;\n l -= k;\n}\n")),(0,a.kt)("p",null,"Then, if more data is requested and the internal buffer isn't full, it refills it using ",(0,a.kt)("a",{parentName:"p",href:"https://elixir.bootlin.com/musl/v1.2.3/source/src/stdio/fread.c#L27"},"the internal ",(0,a.kt)("inlineCode",{parentName:"a"},"read()")," wrapper"),".\nThis wrapper also places the data inside the destination buffer."),(0,a.kt)("h4",{id:"practice-buffering-performance"},"Practice: Buffering Performance"),(0,a.kt)("p",null,"Up to now, it's pretty clear what I/O buffering is about.\nLet's see what kind of a performance increase it brings.\nWe'll look at an extreme example.\nNavigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/buffering/no_buffering.c"),".\nThis code either writes or reads the contents of a file ",(0,a.kt)("strong",{parentName:"p"},"one byte at a time")," using a syscall for each of them."),(0,a.kt)("p",null,"Use the ",(0,a.kt)("inlineCode",{parentName:"p"},"benchmark_buffering.sh")," script to compare the ",(0,a.kt)("inlineCode",{parentName:"p"},"no_buffering")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"libc_buffering")," implementations.\nBelow are some possible results.\nYours are likely going to be different:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/buffering$\n======== Testing no_buffering ========\nTesting no_buffering read...\nRead 1048576 bytes from test-file.txt in 717 ms\nTesting no_buffering write...\nWrote 1048576 bytes to test-file.txt in 19632 ms\n======== Testing libc_buffering ========\nTesting libc_buffering read...\nRead 1048576 bytes from test-file.txt in 14 ms\nTesting libc_buffering write...\nWrote 1048576 bytes to test-file.txt in 38 ms\n")),(0,a.kt)("p",null,"So buffering brings a ",(0,a.kt)("strong",{parentName:"p"},"98%")," improvement for reading and a ",(0,a.kt)("strong",{parentName:"p"},"99.8%"),' improvement for writing!\nThis is massive!\nYes, this is an extreme example, but it goes a long way to show how powerful I/O buffering can be.\nNow, the question is, "Can YOU do better?"'),(0,a.kt)("h3",{id:"practice-diy-buffering"},"Practice: DIY Buffering"),(0,a.kt)("ol",null,(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Navigate to ",(0,a.kt)("inlineCode",{parentName:"p"},"support/buffering/diy_buffering.c"),".\n",(0,a.kt)("inlineCode",{parentName:"p"},"diy_fread()")," already defines a minimalistic implementation of ",(0,a.kt)("inlineCode",{parentName:"p"},"fread()"),".\nUse it as a starting point to implement ",(0,a.kt)("inlineCode",{parentName:"p"},"diy_write()")," as an implementation of ",(0,a.kt)("inlineCode",{parentName:"p"},"fwrite()"),".")),(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Run ",(0,a.kt)("inlineCode",{parentName:"p"},"benchmark_buffering.sh")," to compare the performance of your implementation with that of libc.\nDid you beat it?"))),(0,a.kt)("h2",{id:"conclusion"},"Conclusion"),(0,a.kt)("p",null,"I/O Buffering is a ubiquitous optimisation in all libc implementations.\nIt is so powerful that the kernel uses it as well.\nThis means that the kernel also reads more bytes than it's requested and stores the remainder in an internal buffer, just like libc.\nThis concept is known as ",(0,a.kt)("strong",{parentName:"p"},"double buffering"),".\nIn the ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/zero-copy"},"following section"),", we will see how to make use of this internal buffering to optimise network requests."),(0,a.kt)("p",null,"Notice that the script in ",(0,a.kt)("inlineCode",{parentName:"p"},"support/buffering/benchmark_buffering.sh")," also uses ",(0,a.kt)("inlineCode",{parentName:"p"},"echo 3 > /proc/sys/vm/drop_caches"),".\nThat section ",(0,a.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/arena#to-drop-or-not-to-drop"},"in the Arena")," that we mentioned earlier is becoming even more interesting."))}m.isMDXComponent=!0},4132:(e,t,i)=>{i.d(t,{Z:()=>n});const n="data:image/svg+xml;base64,PD94bWwgdmVyc2lvbj0iMS4wIiBlbmNvZGluZz0iVVRGLTgiPz4KPCEtLSBEbyBub3QgZWRpdCB0aGlzIGZpbGUgd2l0aCBlZGl0b3JzIG90aGVyIHRoYW4gZGlhZ3JhbXMubmV0IC0tPgo8IURPQ1RZUEUgc3ZnIFBVQkxJQyAiLS8vVzNDLy9EVEQgU1ZHIDEuMS8vRU4iICJodHRwOi8vd3d3LnczLm9yZy9HcmFwaGljcy9TVkcvMS4xL0RURC9zdmcxMS5kdGQiPgo8c3ZnIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyIgeG1sbnM6eGxpbms9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGxpbmsiIHZlcnNpb249IjEuMSIgd2lkdGg9IjcxMXB4IiBoZWlnaHQ9IjM3MXB4IiB2aWV3Qm94PSItMC41IC0wLjUgNzExIDM3MSIgY29udGVudD0iJmx0O214ZmlsZSBob3N0PSZxdW90O2FwcC5kaWFncmFtcy5uZXQmcXVvdDsgbW9kaWZpZWQ9JnF1b3Q7MjAyMi0xMi0wOFQwOTowMzozNy44MjhaJnF1b3Q7IGFnZW50PSZxdW90OzUuMCAoWDExKSZxdW90OyBldGFnPSZxdW90O3JWbUViSzdSbGM2VW1rU2RVRFNzJnF1b3Q7IHZlcnNpb249JnF1b3Q7MjAuNi4wJnF1b3Q7IHR5cGU9JnF1b3Q7ZGV2aWNlJnF1b3Q7Jmd0OyZsdDtkaWFncmFtIGlkPSZxdW90OzFOb3R2clVOZU4td2JBQWhaRFlYJnF1b3Q7IG5hbWU9JnF1b3Q7UGFnZS0xJnF1b3Q7Jmd0OzdaaGRiOW93RklaL1RhVHRnc21KQ1drdklaU2hUVWhWVVRYdUpqZDJFNnRKbkRtbXdINzk3TVRPTndLdFVGWGF1Q0Y1YlIvYjcrT0RUMnRCUDlsLzVTaUxWZ3lUMkhJQTNsdHdiam55NHpueVN5bUhVcmx4UUNtRW5PSlNzbXRoVFg4VExacHVXNHBKM3Vvb0dJc0Z6ZHBpd05LVUJLS2xJYzdacnQzdG1jWHRXVE1Va3A2d0RsRGNWMzlRTENLOUN4ZlUrcExRTURJejIwQzNKTWgwMWtJZUljeDJEUW5lV2REbmpJbnlLZG43SkZibUdWL0tjWXNqcmRYQ09FbkZPUU95N3orLzNhWGVaamQ3ZWR4Nm04VnFFM2tqSGVVVnhWdTlZYjFZY1RBT2NMWk5NVkZCYkF2T2RoRVZaSjJoUUxYdUpIT3BSU0tKZFhNdU9Ic2hQb3NabDByS1V0bHRwdWNnWEpEOTBjWGJsU1h5TEJHV0VNRVBzb3NlTUlMYXhZTTVIdnA5VjBQeHpKbUpHa0NncDBXa0QwSll4YTY5a2cvYXJtSHJIa2ZaUEhOdm95QzBJNWd0bHhrR3dhQjFrMWhPTzN0bWNsTk5EeWUvdHN3MGpQTGlqRTlsQjJuQXZyREh0TXVuVUg4WGdaNk04RUJDbWd2Q2M5TWlWL3pVN1MyMWNtWWp2eDJrT2ZKanRYZ2F4d2FzNVVBSExoWmdXczNUUXprQS9DaGRhSGZvdW4yNnRqTkFkM0l0dU01UnVKWHRQZ29pb2hLZEpFek5PQVRtTFFRVVMvMTdLTGQrQmhFQWJtOEJ1QXdSeDIwVHFYNlBHMFNjOFhzU2dhZUpyQkJOWlplVkJnSStQVXhYbno4QW1JWDhRSGdoTUtBRHh1dURnZThLWm53YXpKckk2eG1qWWg4MW5TWGllSVJwL21JNWNtYXdYcy9MQnlLQ0x4OEJtKzhEY0tsOHNqdjVCR0VmMi9oZHNibW5iMzZTNHFrcW9lUmJFS004cDBIYjBkcCtjSmEvUjMwa3VGV0U5VjFzdU9RT21HUTBUbUlrNkd1N2RCdHlUczl3ejJoeFV4dElUaHZTMk9tWW43TXRENGdlMVN5MVRnU3k3VTRnZ1hoSVJDOVFBYkxhOXQrem5aeE95WHZDbnhsUFVDcXo1WnhVaytkY3RPbWptSWFwT2hxU0pwRXBNMVBaUUdYbFBOVU5DY1ZZRFo5eEltc2U5RlNFVWljalUvc3VuSEJubGp0WHNiYUNsWFhSOFNxeW1aMUc2aHk1Q3lScXQ5Q3NMcmxtS1RLVXFQQmFpZXI5VDlTckplckVPeEhveW9sNmN6cFJmU1J2TkNyT0t6RC9sU3l0S2h6RGJTaEx3V1d5Vkw3V2Y2U1g0T3QvZGNDN1B3PT0mbHQ7L2RpYWdyYW0mZ3Q7Jmx0Oy9teGZpbGUmZ3Q7Ij48ZGVmcy8+PGc+PHJlY3QgeD0iMCIgeT0iMCIgd2lkdGg9IjcxMCIgaGVpZ2h0PSIzNzAiIHJ4PSI1NS41IiByeT0iNTUuNSIgZmlsbD0icmdiKDI1NSwgMjU1LCAyNTUpIiBzdHJva2U9Im5vbmUiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48cmVjdCB4PSIzNDAiIHk9IjUwIiB3aWR0aD0iMTIwIiBoZWlnaHQ9IjYwIiByeD0iOSIgcnk9IjkiIGZpbGw9IiMyM2ZmMGEiIHN0cm9rZT0icmdiKDAsIDAsIDApIiBzdHJva2Utd2lkdGg9IjQiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtMC41IC0wLjUpIj48c3dpdGNoPjxmb3JlaWduT2JqZWN0IHN0eWxlPSJvdmVyZmxvdzogdmlzaWJsZTsgdGV4dC1hbGlnbjogbGVmdDsiIHBvaW50ZXItZXZlbnRzPSJub25lIiB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OiBmbGV4OyBhbGlnbi1pdGVtczogdW5zYWZlIGNlbnRlcjsganVzdGlmeS1jb250ZW50OiB1bnNhZmUgY2VudGVyOyB3aWR0aDogMTE4cHg7IGhlaWdodDogMXB4OyBwYWRkaW5nLXRvcDogODBweDsgbWFyZ2luLWxlZnQ6IDM0MXB4OyI+PGRpdiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsgZm9udC1zaXplOiAwcHg7IHRleHQtYWxpZ246IGNlbnRlcjsiIGRhdGEtZHJhd2lvLWNvbG9ycz0iY29sb3I6IHJnYigwLCAwLCAwKTsgIj48ZGl2IHN0eWxlPSJkaXNwbGF5OiBpbmxpbmUtYmxvY2s7IGZvbnQtc2l6ZTogMTJweDsgZm9udC1mYW1pbHk6IEhlbHZldGljYTsgY29sb3I6IHJnYigwLCAwLCAwKTsgbGluZS1oZWlnaHQ6IDEuMjsgcG9pbnRlci1ldmVudHM6IGFsbDsgd2hpdGUtc3BhY2U6IG5vcm1hbDsgb3ZlcmZsb3ctd3JhcDogbm9ybWFsOyI+PGZvbnQgc3R5bGU9ImZvbnQtc2l6ZTogMjBweDsiPjxiPlJlZ2lzdGVyczwvYj48L2ZvbnQ+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjQwMCIgeT0iODQiIGZpbGw9InJnYigwLCAwLCAwKSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSIgZm9udC1zaXplPSIxMnB4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5SZWdpc3RlcnM8L3RleHQ+PC9zd2l0Y2g+PC9nPjxyZWN0IHg9IjI4MCIgeT0iMTEwIiB3aWR0aD0iMjQwIiBoZWlnaHQ9IjYwIiByeD0iOSIgcnk9IjkiIGZpbGw9IiMwMDk5MDAiIHN0cm9rZT0icmdiKDAsIDAsIDApIiBzdHJva2Utd2lkdGg9IjQiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtMC41IC0wLjUpIj48c3dpdGNoPjxmb3JlaWduT2JqZWN0IHN0eWxlPSJvdmVyZmxvdzogdmlzaWJsZTsgdGV4dC1hbGlnbjogbGVmdDsiIHBvaW50ZXItZXZlbnRzPSJub25lIiB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OiBmbGV4OyBhbGlnbi1pdGVtczogdW5zYWZlIGNlbnRlcjsganVzdGlmeS1jb250ZW50OiB1bnNhZmUgY2VudGVyOyB3aWR0aDogMjM4cHg7IGhlaWdodDogMXB4OyBwYWRkaW5nLXRvcDogMTQwcHg7IG1hcmdpbi1sZWZ0OiAyODFweDsiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7IGZvbnQtc2l6ZTogMHB4OyB0ZXh0LWFsaWduOiBjZW50ZXI7IiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiByZ2IoMCwgMCwgMCk7ICI+PGRpdiBzdHlsZT0iZGlzcGxheTogaW5saW5lLWJsb2NrOyBmb250LXNpemU6IDIwcHg7IGZvbnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGxpbmUtaGVpZ2h0OiAxLjI7IHBvaW50ZXItZXZlbnRzOiBhbGw7IHdoaXRlLXNwYWNlOiBub3JtYWw7IG92ZXJmbG93LXdyYXA6IG5vcm1hbDsiPjxiPkNhY2hlIG1lbW9yeTwvYj48L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNDAwIiB5PSIxNDYiIGZpbGw9InJnYigwLCAwLCAwKSIgZm9udC1mYW1pbHk9IkhlbHZldGljYSIgZm9udC1zaXplPSIyMHB4IiB0ZXh0LWFuY2hvcj0ibWlkZGxlIj5DYWNoZSBtZW1vcnk8L3RleHQ+PC9zd2l0Y2g+PC9nPjxyZWN0IHg9IjIzMCIgeT0iMTcwIiB3aWR0aD0iMzQwIiBoZWlnaHQ9IjYwIiByeD0iOSIgcnk9IjkiIGZpbGw9IiNmZmZmMzMiIHN0cm9rZT0icmdiKDAsIDAsIDApIiBzdHJva2Utd2lkdGg9IjQiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtMC41IC0wLjUpIj48c3dpdGNoPjxmb3JlaWduT2JqZWN0IHN0eWxlPSJvdmVyZmxvdzogdmlzaWJsZTsgdGV4dC1hbGlnbjogbGVmdDsiIHBvaW50ZXItZXZlbnRzPSJub25lIiB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OiBmbGV4OyBhbGlnbi1pdGVtczogdW5zYWZlIGNlbnRlcjsganVzdGlmeS1jb250ZW50OiB1bnNhZmUgY2VudGVyOyB3aWR0aDogMzM4cHg7IGhlaWdodDogMXB4OyBwYWRkaW5nLXRvcDogMjAwcHg7IG1hcmdpbi1sZWZ0OiAyMzFweDsiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7IGZvbnQtc2l6ZTogMHB4OyB0ZXh0LWFsaWduOiBjZW50ZXI7IiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiByZ2IoMCwgMCwgMCk7ICI+PGRpdiBzdHlsZT0iZGlzcGxheTogaW5saW5lLWJsb2NrOyBmb250LXNpemU6IDIwcHg7IGZvbnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGxpbmUtaGVpZ2h0OiAxLjI7IHBvaW50ZXItZXZlbnRzOiBhbGw7IHdoaXRlLXNwYWNlOiBub3JtYWw7IG92ZXJmbG93LXdyYXA6IG5vcm1hbDsiPjxiPk1haW4gTWVtb3J5IChSQU0pPC9iPjwvZGl2PjwvZGl2PjwvZGl2PjwvZm9yZWlnbk9iamVjdD48dGV4dCB4PSI0MDAiIHk9IjIwNiIgZmlsbD0icmdiKDAsIDAsIDApIiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjIwcHgiIHRleHQtYW5jaG9yPSJtaWRkbGUiPk1haW4gTWVtb3J5IChSQU0pPC90ZXh0Pjwvc3dpdGNoPjwvZz48cmVjdCB4PSIxODAiIHk9IjIzMCIgd2lkdGg9IjQ0MCIgaGVpZ2h0PSI2MCIgcng9IjkiIHJ5PSI5IiBmaWxsPSIjY2MwMDAwIiBzdHJva2U9InJnYigwLCAwLCAwKSIgc3Ryb2tlLXdpZHRoPSI0IiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTAuNSAtMC41KSI+PHN3aXRjaD48Zm9yZWlnbk9iamVjdCBzdHlsZT0ib3ZlcmZsb3c6IHZpc2libGU7IHRleHQtYWxpZ246IGxlZnQ7IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTogZmxleDsgYWxpZ24taXRlbXM6IHVuc2FmZSBjZW50ZXI7IGp1c3RpZnktY29udGVudDogdW5zYWZlIGNlbnRlcjsgd2lkdGg6IDQzOHB4OyBoZWlnaHQ6IDFweDsgcGFkZGluZy10b3A6IDI2MHB4OyBtYXJnaW4tbGVmdDogMTgxcHg7Ij48ZGl2IHN0eWxlPSJib3gtc2l6aW5nOiBib3JkZXItYm94OyBmb250LXNpemU6IDBweDsgdGV4dC1hbGlnbjogY2VudGVyOyIgZGF0YS1kcmF3aW8tY29sb3JzPSJjb2xvcjogcmdiKDAsIDAsIDApOyAiPjxkaXYgc3R5bGU9ImRpc3BsYXk6IGlubGluZS1ibG9jazsgZm9udC1zaXplOiAyMHB4OyBmb250LWZhbWlseTogSGVsdmV0aWNhOyBjb2xvcjogcmdiKDAsIDAsIDApOyBsaW5lLWhlaWdodDogMS4yOyBwb2ludGVyLWV2ZW50czogYWxsOyB3aGl0ZS1zcGFjZTogbm9ybWFsOyBvdmVyZmxvdy13cmFwOiBub3JtYWw7Ij48Yj5TZWNvbmRhcnkgTWVtb3J5IChIYXJkLWRpc2ssIFNTRCwgZXRjLik8L2I+PC9kaXY+PC9kaXY+PC9kaXY+PC9mb3JlaWduT2JqZWN0Pjx0ZXh0IHg9IjQwMCIgeT0iMjY2IiBmaWxsPSJyZ2IoMCwgMCwgMCkiIGZvbnQtZmFtaWx5PSJIZWx2ZXRpY2EiIGZvbnQtc2l6ZT0iMjBweCIgdGV4dC1hbmNob3I9Im1pZGRsZSI+U2Vjb25kYXJ5IE1lbW9yeSAoSGFyZC1kaXNrLCBTU0QsIGV0Yy4pPC90ZXh0Pjwvc3dpdGNoPjwvZz48cGF0aCBkPSJNIDE1MCAzMjAgTCAxNTAgMjEuOTciIGZpbGw9Im5vbmUiIHN0cm9rZT0icmdiKDAsIDAsIDApIiBzdHJva2Utd2lkdGg9IjQiIHN0cm9rZS1taXRlcmxpbWl0PSIxMCIgcG9pbnRlci1ldmVudHM9InN0cm9rZSIvPjxwYXRoIGQ9Ik0gMTUwIDE0LjQ3IEwgMTU1IDI0LjQ3IEwgMTUwIDIxLjk3IEwgMTQ1IDI0LjQ3IFoiIGZpbGw9InJnYigwLCAwLCAwKSIgc3Ryb2tlPSJyZ2IoMCwgMCwgMCkiIHN0cm9rZS13aWR0aD0iNCIgc3Ryb2tlLW1pdGVybGltaXQ9IjEwIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PHJlY3QgeD0iMCIgeT0iMTQwIiB3aWR0aD0iMTQwIiBoZWlnaHQ9IjMwIiBmaWxsPSJub25lIiBzdHJva2U9Im5vbmUiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48ZyB0cmFuc2Zvcm09InRyYW5zbGF0ZSgtMC41IC0wLjUpIj48c3dpdGNoPjxmb3JlaWduT2JqZWN0IHN0eWxlPSJvdmVyZmxvdzogdmlzaWJsZTsgdGV4dC1hbGlnbjogbGVmdDsiIHBvaW50ZXItZXZlbnRzPSJub25lIiB3aWR0aD0iMTAwJSIgaGVpZ2h0PSIxMDAlIiByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiPjxkaXYgeG1sbnM9Imh0dHA6Ly93d3cudzMub3JnLzE5OTkveGh0bWwiIHN0eWxlPSJkaXNwbGF5OiBmbGV4OyBhbGlnbi1pdGVtczogdW5zYWZlIGNlbnRlcjsganVzdGlmeS1jb250ZW50OiB1bnNhZmUgY2VudGVyOyB3aWR0aDogMXB4OyBoZWlnaHQ6IDFweDsgcGFkZGluZy10b3A6IDE1NXB4OyBtYXJnaW4tbGVmdDogNzBweDsiPjxkaXYgc3R5bGU9ImJveC1zaXppbmc6IGJvcmRlci1ib3g7IGZvbnQtc2l6ZTogMHB4OyB0ZXh0LWFsaWduOiBjZW50ZXI7IiBkYXRhLWRyYXdpby1jb2xvcnM9ImNvbG9yOiByZ2IoMCwgMCwgMCk7ICI+PGRpdiBzdHlsZT0iZGlzcGxheTogaW5saW5lLWJsb2NrOyBmb250LXNpemU6IDIwcHg7IGZvbnQtZmFtaWx5OiBIZWx2ZXRpY2E7IGNvbG9yOiByZ2IoMCwgMCwgMCk7IGxpbmUtaGVpZ2h0OiAxLjI7IHBvaW50ZXItZXZlbnRzOiBhbGw7IHdoaXRlLXNwYWNlOiBub3dyYXA7Ij48Yj5QZXJmb3JtYW5jZTwvYj48L2Rpdj48L2Rpdj48L2Rpdj48L2ZvcmVpZ25PYmplY3Q+PHRleHQgeD0iNzAiIHk9IjE2MSIgZmlsbD0icmdiKDAsIDAsIDApIiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjIwcHgiIHRleHQtYW5jaG9yPSJtaWRkbGUiPlBlcmZvcm1hbmNlPC90ZXh0Pjwvc3dpdGNoPjwvZz48cGF0aCBkPSJNIDE1MCAzMjAgTCA2ODguMDMgMzIwIiBmaWxsPSJub25lIiBzdHJva2U9InJnYigwLCAwLCAwKSIgc3Ryb2tlLXdpZHRoPSI0IiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJzdHJva2UiLz48cGF0aCBkPSJNIDY5NS41MyAzMjAgTCA2ODUuNTMgMzI1IEwgNjg4LjAzIDMyMCBMIDY4NS41MyAzMTUgWiIgZmlsbD0icmdiKDAsIDAsIDApIiBzdHJva2U9InJnYigwLCAwLCAwKSIgc3Ryb2tlLXdpZHRoPSI0IiBzdHJva2UtbWl0ZXJsaW1pdD0iMTAiIHBvaW50ZXItZXZlbnRzPSJhbGwiLz48cmVjdCB4PSIzNzAiIHk9IjM0MCIgd2lkdGg9IjEwMCIgaGVpZ2h0PSIzMCIgZmlsbD0ibm9uZSIgc3Ryb2tlPSJub25lIiBwb2ludGVyLWV2ZW50cz0iYWxsIi8+PGcgdHJhbnNmb3JtPSJ0cmFuc2xhdGUoLTAuNSAtMC41KSI+PHN3aXRjaD48Zm9yZWlnbk9iamVjdCBzdHlsZT0ib3ZlcmZsb3c6IHZpc2libGU7IHRleHQtYWxpZ246IGxlZnQ7IiBwb2ludGVyLWV2ZW50cz0ibm9uZSIgd2lkdGg9IjEwMCUiIGhlaWdodD0iMTAwJSIgcmVxdWlyZWRGZWF0dXJlcz0iaHR0cDovL3d3dy53My5vcmcvVFIvU1ZHMTEvZmVhdHVyZSNFeHRlbnNpYmlsaXR5Ij48ZGl2IHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5L3hodG1sIiBzdHlsZT0iZGlzcGxheTogZmxleDsgYWxpZ24taXRlbXM6IHVuc2FmZSBjZW50ZXI7IGp1c3RpZnktY29udGVudDogdW5zYWZlIGNlbnRlcjsgd2lkdGg6IDFweDsgaGVpZ2h0OiAxcHg7IHBhZGRpbmctdG9wOiAzNTVweDsgbWFyZ2luLWxlZnQ6IDQyMHB4OyI+PGRpdiBzdHlsZT0iYm94LXNpemluZzogYm9yZGVyLWJveDsgZm9udC1zaXplOiAwcHg7IHRleHQtYWxpZ246IGNlbnRlcjsiIGRhdGEtZHJhd2lvLWNvbG9ycz0iY29sb3I6IHJnYigwLCAwLCAwKTsgIj48ZGl2IHN0eWxlPSJkaXNwbGF5OiBpbmxpbmUtYmxvY2s7IGZvbnQtc2l6ZTogMjBweDsgZm9udC1mYW1pbHk6IEhlbHZldGljYTsgY29sb3I6IHJnYigwLCAwLCAwKTsgbGluZS1oZWlnaHQ6IDEuMjsgcG9pbnRlci1ldmVudHM6IGFsbDsgd2hpdGUtc3BhY2U6IG5vd3JhcDsiPjxiPkNhcGFjaXR5PC9iPjwvZGl2PjwvZGl2PjwvZGl2PjwvZm9yZWlnbk9iamVjdD48dGV4dCB4PSI0MjAiIHk9IjM2MSIgZmlsbD0icmdiKDAsIDAsIDApIiBmb250LWZhbWlseT0iSGVsdmV0aWNhIiBmb250LXNpemU9IjIwcHgiIHRleHQtYW5jaG9yPSJtaWRkbGUiPkNhcGFjaXR5PC90ZXh0Pjwvc3dpdGNoPjwvZz48L2c+PHN3aXRjaD48ZyByZXF1aXJlZEZlYXR1cmVzPSJodHRwOi8vd3d3LnczLm9yZy9UUi9TVkcxMS9mZWF0dXJlI0V4dGVuc2liaWxpdHkiLz48YSB0cmFuc2Zvcm09InRyYW5zbGF0ZSgwLC01KSIgeGxpbms6aHJlZj0iaHR0cHM6Ly93d3cuZGlhZ3JhbXMubmV0L2RvYy9mYXEvc3ZnLWV4cG9ydC10ZXh0LXByb2JsZW1zIiB0YXJnZXQ9Il9ibGFuayI+PHRleHQgdGV4dC1hbmNob3I9Im1pZGRsZSIgZm9udC1zaXplPSIxMHB4IiB4PSI1MCUiIHk9IjEwMCUiPlRleHQgaXMgbm90IFNWRyAtIGNhbm5vdCBkaXNwbGF5PC90ZXh0PjwvYT48L3N3aXRjaD48L3N2Zz4="},6585:(e,t,i)=>{i.d(t,{Z:()=>n});const n=i.p+"assets/images/sad-pepe-93384f7b72f31ab085cdf22a743d769d.png"}}]); \ No newline at end of file diff --git a/17/assets/js/8d5b82ca.0974ffe7.js b/17/assets/js/8d5b82ca.0974ffe7.js new file mode 100644 index 0000000000..da6c1edb70 --- /dev/null +++ b/17/assets/js/8d5b82ca.0974ffe7.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7674],{3905:(e,t,r)=>{r.d(t,{Zo:()=>c,kt:()=>y});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function p(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var s=n.createContext({}),l=function(e){var t=n.useContext(s),r=t;return e&&(r="function"==typeof e?e(t):p(p({},t),e)),r},c=function(e){var t=l(e.components);return n.createElement(s.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,s=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),u=l(r),f=a,y=u["".concat(s,".").concat(f)]||u[f]||m[f]||o;return r?n.createElement(y,p(p({ref:t},c),{},{components:r})):n.createElement(y,p({ref:t},c))}));function y(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,p=new Array(o);p[0]=f;var i={};for(var s in t)hasOwnProperty.call(t,s)&&(i[s]=t[s]);i.originalType=e,i[u]="string"==typeof e?e:a,p[1]=i;for(var l=2;l{r.r(t),r.d(t,{assets:()=>s,contentTitle:()=>p,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>l});var n=r(7462),a=(r(7294),r(3905));const o={},p="Compute",i={unversionedId:"Lab/Compute/overview",id:"Lab/Compute/overview",title:"Compute",description:"Contents",source:"@site/docs/Lab/Compute/overview.md",sourceDirName:"Lab/Compute",slug:"/Lab/Compute/overview",permalink:"/operating-systems/17/Lab/Compute/overview",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Compute",permalink:"/operating-systems/17/Lab/Compute/"},next:{title:"Hardware Perspective",permalink:"/operating-systems/17/Lab/Compute/hardware-perspective"}},s={},l=[{value:"Contents",id:"contents",level:2}],c={toc:l},u="wrapper";function m(e){let{components:t,...r}=e;return(0,a.kt)(u,(0,n.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"compute"},"Compute"),(0,a.kt)("h2",{id:"contents"},"Contents"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/hardware-perspective"},"Hardware Perspective")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/processes"},"Processes")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/threads"},"Threads")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/processes-threads-apache2"},"Processes and Threads in ",(0,a.kt)("inlineCode",{parentName:"a"},"apache2"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/copy-on-write"},"Copy-on-Write")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/synchronization"},"Synchronization")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/user-level-threads"},"User Level Threads")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("a",{parentName:"li",href:"/operating-systems/17/Lab/Compute/arena"},"Arena"))))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/8de34314.080a5f50.js b/17/assets/js/8de34314.080a5f50.js new file mode 100644 index 0000000000..61683939b3 --- /dev/null +++ b/17/assets/js/8de34314.080a5f50.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1758],{3905:(e,t,n)=>{n.d(t,{Zo:()=>d,kt:()=>h});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function o(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var s=a.createContext({}),p=function(e){var t=a.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},d=function(e){var t=p(e.components);return a.createElement(s.Provider,{value:t},e.children)},m="mdxType",c={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},u=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,s=e.parentName,d=l(e,["components","mdxType","originalType","parentName"]),m=p(n),u=r,h=m["".concat(s,".").concat(u)]||m[u]||c[u]||i;return n?a.createElement(h,o(o({ref:t},d),{},{components:n})):a.createElement(h,o({ref:t},d))}));function h(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,o=new Array(i);o[0]=u;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[m]="string"==typeof e?e:r,o[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>o,default:()=>c,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const i={},o="File Descriptors",l={unversionedId:"Lab/IO/file-descriptors",id:"Lab/IO/file-descriptors",title:"File Descriptors",description:'After running the code in the "File Handlers" section, you saw that open() returns a number.',source:"@site/docs/Lab/IO/file-descriptors.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/file-descriptors",permalink:"/operating-systems/17/Lab/IO/file-descriptors",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"File Handling",permalink:"/operating-systems/17/Lab/IO/file-handlers"},next:{title:"Redirections",permalink:"/operating-systems/17/Lab/IO/redirections"}},s={},p=[{value:"Creating New File Descriptors",id:"creating-new-file-descriptors",level:2},{value:"Practice: Open a File for Writing",id:"practice-open-a-file-for-writing",level:3},{value:"Practice: Write to the File",id:"practice-write-to-the-file",level:3},{value:"Replace or Truncate?",id:"replace-or-truncate",level:2},{value:"Practice: Write Once More",id:"practice-write-once-more",level:3},{value:"Practice: Close'em All",id:"practice-closeem-all",level:3},{value:"File Handling Conclusion: libc vs syscalls",id:"file-handling-conclusion-libc-vs-syscalls",level:2}],d={toc:p},m="wrapper";function c(e){let{components:t,...i}=e;return(0,r.kt)(m,(0,a.Z)({},d,i,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"file-descriptors"},"File Descriptors"),(0,r.kt)("p",null,"After running the code in the ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/file-handlers"},'"File Handlers" section'),", you saw that ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," returns a ",(0,r.kt)("strong",{parentName:"p"},"number"),".\nThis number is a ",(0,r.kt)("strong",{parentName:"p"},"file descriptor"),".\nRun the code above multiple times.\nYou'll always get file descriptor 3.\nWe can already tell these numbers have some meaning and aren't just some seemingly random numbers, like PIDs were."),(0,r.kt)("p",null,"You've most likely used file descriptors before without knowing.\nRemember the following concepts:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"stdin")," (standard input)"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"stdout")," (standard output)"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"stderr")," (standard error)")),(0,r.kt)("p",null,"When you wanted to run a shell command and ignore its errors, you wrote something like:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ ls 2> /dev/null\n")),(0,r.kt)("p",null,"The command above uses ",(0,r.kt)("inlineCode",{parentName:"p"},"2> /dev/null")," to ",(0,r.kt)("strong",{parentName:"p"},"redirect")," ",(0,r.kt)("inlineCode",{parentName:"p"},"stderr")," to /dev/null."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/stderr-fd"},"Quiz")),(0,r.kt)("p",null,"Good, so we know that ",(0,r.kt)("inlineCode",{parentName:"p"},"stderr")," is file descriptor 2.\nThe code in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/file-descriptors/open_directory.c")," opened the directory as file descriptor 3.\nSo what are file descriptors 0 and 1?\nThey are ",(0,r.kt)("inlineCode",{parentName:"p"},"stdin")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"stdout"),", respectively."),(0,r.kt)("p",null,"Now we know that the basic I/O streams of a process correspond to specific file descriptors.\nBut what is a file descriptor exactly?\nFrom the application's point of view, it is a positive integer acting as a unique identifier for an I/O channel, such as a file.\nTo the operating system, a file descriptor is an index.\nEach process has what's called a ",(0,r.kt)("strong",{parentName:"p"},"file descriptor table"),", which is just a fancy name for an array of pointers.\nA file descriptor is an index in this array.\nEach element in this array points towards a structure in the kernel's memory that represents an I/O resource.\nThis structure is called the ",(0,r.kt)("strong",{parentName:"p"},"open file structure"),"."),(0,r.kt)("p",null,"To illustrate all this information, look at the image below.\nAll data structures shown below are stored in the Kernel's memory area and are transparent to the user space.\nAnd remember that the file descriptor table is bound to the process, so all its threads share the same file descriptors."),(0,r.kt)("p",null,(0,r.kt)("img",{alt:"File Descriptors",src:n(7367).Z,width:"462",height:"242"})),(0,r.kt)("p",null,"To find out more about the contents of these structures, check out ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/arena#open-file-structure-in-the-kernel"},"this section in the Arena")),(0,r.kt)("h2",{id:"creating-new-file-descriptors"},"Creating New File Descriptors"),(0,r.kt)("p",null,'We already know that each process gets 3 file descriptors "by default":'),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"stdin")," (standard input): 0"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"stdout")," (standard output): 1"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("inlineCode",{parentName:"li"},"stderr")," (standard error): 2")),(0,r.kt)("p",null,"To create new file descriptors (i.e. open new files), a process can use the ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,r.kt)("inlineCode",{parentName:"a"},"open()"))," system call.\nIt receives the path to the file, some flags which are akin to the ",(0,r.kt)("inlineCode",{parentName:"p"},"mode")," string passed to ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()"),".\nAn optional ",(0,r.kt)("inlineCode",{parentName:"p"},"mode")," parameter that denotes the file's permissions if the ",(0,r.kt)("inlineCode",{parentName:"p"},"open")," must create it can also be provided.\nWe'll revisit ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),"'s ",(0,r.kt)("inlineCode",{parentName:"p"},"mode")," parameter in the future."),(0,r.kt)("p",null,"From now, you should consult ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/open.2.html"},(0,r.kt)("inlineCode",{parentName:"a"},"open()"),"'s man page")," whenever you encounter a new argument to this syscall.\nNavigate to ",(0,r.kt)("inlineCode",{parentName:"p"},"support/file-descriptors/open_read_write.c"),".\nThe function ",(0,r.kt)("inlineCode",{parentName:"p"},"open_file_for_reading()")," calls ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," with the ",(0,r.kt)("inlineCode",{parentName:"p"},"O_RDONLY")," flag, which is equivalent to opening the file with ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()")," and setting ",(0,r.kt)("inlineCode",{parentName:"p"},'"r"')," as the ",(0,r.kt)("inlineCode",{parentName:"p"},"mode"),".\nThis means read-only.\nNote that the file descriptor we get is 3, just like before."),(0,r.kt)("p",null,"Then ",(0,r.kt)("inlineCode",{parentName:"p"},"read_from_file()")," reads ",(0,r.kt)("inlineCode",{parentName:"p"},"bytes_to_read")," bytes from this file.\nFor this, it uses the ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," syscall.\nIt returns the number of bytes that were read from the file."),(0,r.kt)("p",null,"Notice that the code doesn't simply call ",(0,r.kt)("inlineCode",{parentName:"p"},"read(fd, buff, bytes_to_read)"),".\nInstead, it uses a ",(0,r.kt)("inlineCode",{parentName:"p"},"while")," loop to read data from the file."),(0,r.kt)("p",null,"The ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/read.2.html#RETURN_VALUE"},"return value of ",(0,r.kt)("inlineCode",{parentName:"a"},"read()"))," may be less than ",(0,r.kt)("inlineCode",{parentName:"p"},"bytes_to_read")," if there are not enough bytes available or if the operation is interrupted by a signal.\nA return value between ",(0,r.kt)("strong",{parentName:"p"},"0")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"bytes_to_read")," is not enough to decide whether we should stop reading.\nTo determine this, we make another ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," call, which will return ",(0,r.kt)("strong",{parentName:"p"},"0")," if the cursor is already at ",(0,r.kt)("strong",{parentName:"p"},"EOF")," (end of file)."),(0,r.kt)("p",null,"The same goes for ",(0,r.kt)("inlineCode",{parentName:"p"},"write()"),": its return value may differ from the intended number of bytes to write.\nPartial writes should also be handled at the application level, and the way to do this is by using loops."),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Remember:"),"\n",(0,r.kt)("strong",{parentName:"p"},"It is mandatory that we always use ",(0,r.kt)("inlineCode",{parentName:"strong"},"read()")," and ",(0,r.kt)("inlineCode",{parentName:"strong"},"write()")," inside ",(0,r.kt)("inlineCode",{parentName:"strong"},"while")," loops."),"\nHigher-level functions like ",(0,r.kt)("inlineCode",{parentName:"p"},"fread()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"fwrite()")," also use ",(0,r.kt)("inlineCode",{parentName:"p"},"while")," loops when calling ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," respectively."),(0,r.kt)("h3",{id:"practice-open-a-file-for-writing"},"Practice: Open a File for Writing"),(0,r.kt)("p",null,"Follow the code in ",(0,r.kt)("inlineCode",{parentName:"p"},"open_file_for_reading()")," and fill in the function ",(0,r.kt)("inlineCode",{parentName:"p"},"open_file_for_writing()"),".\nThe file ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt")," doesn't exist.\n",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," should create it.\nUse the ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),"'s man page to find the flags you require.\nDo you require anything ",(0,r.kt)("em",{parentName:"p"},"else"),"?"),(0,r.kt)("p",null,"At this point, depending on what flags you passed to ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),", a few things might happen.\nWork your way through the errors until successfully create and open the file.\nThe ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," syscall should return file descriptor 4."),(0,r.kt)("p",null,"Now verify the file.\nThis part may be different on your system.\nDelete ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt")," and rerun ",(0,r.kt)("inlineCode",{parentName:"p"},"open_read_write")," a few times.\nEach time, check the permissions of ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt"),".\nThey may be different between runs, or they may always be the same.\nAnyway, they are likely not the default permissions for a regular file (",(0,r.kt)("inlineCode",{parentName:"p"},"rw-r--r--"),").\nIt is not mandatory that you get the same output as below."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/file-descriptors$ ls -l\ntotal 11\ndrwxrwxr-x 1 student student 4096 Nov 20 18:26 ./\ndrwxrwxr-x 1 student student 0 Nov 20 14:11 ../\n-rw-rw-r-- 1 student student 46 Nov 20 17:27 .gitignore\n-rw-rw-r-- 1 student student 125 Nov 20 18:26 Makefile\n-rw-rw-r-- 1 student student 396 Nov 20 11:27 open_directory.c\n-rw-rw-r-- 1 student student 2210 Nov 20 17:24 open_read_write.c\n-rw-rw-r-- 1 student student 34 Nov 20 18:26 read_file.txt\n---------- 1 student student 45 Nov 20 18:26 write_file.txt\n")),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/write-file-permissions"},"Quiz")),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Remember:"),"\n",(0,r.kt)("strong",{parentName:"p"},"It is mandatory that we pass a ",(0,r.kt)("inlineCode",{parentName:"strong"},"mode")," argument to ",(0,r.kt)("inlineCode",{parentName:"strong"},"open()")," when using the ",(0,r.kt)("inlineCode",{parentName:"strong"},"O_CREAT")," flag.")),(0,r.kt)("p",null,"Now pass some sensible ",(0,r.kt)("inlineCode",{parentName:"p"},"mode")," argument to ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),", such as ",(0,r.kt)("inlineCode",{parentName:"p"},"0644")," (for ",(0,r.kt)("inlineCode",{parentName:"p"},"rw-r--r--")," permissions)."),(0,r.kt)("h3",{id:"practice-write-to-the-file"},"Practice: Write to the File"),(0,r.kt)("p",null,"Follow the example in ",(0,r.kt)("inlineCode",{parentName:"p"},"read_from_file()")," to fill in ",(0,r.kt)("inlineCode",{parentName:"p"},"write_to_file()"),".\nRemember to use a loop to make sure your data is fully written to the file."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"cat write_file.txt")," to check if the file contains the right data.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Use ",(0,r.kt)("inlineCode",{parentName:"p"},"open_file_for_reading()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"read_from_file()")," to reopen ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt")," for reading and read the bytes you've just written to it."))),(0,r.kt)("h2",{id:"replace-or-truncate"},"Replace or Truncate?"),(0,r.kt)("h3",{id:"practice-write-once-more"},"Practice: Write Once More"),(0,r.kt)("p",null,"Change the message written to ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt")," by ",(0,r.kt)("inlineCode",{parentName:"p"},"support/file-descriptors/open_read_write.c")," ",(0,r.kt)("strong",{parentName:"p"},"to a shorter one"),".\nIt is important that the new message be shorter than the first one.\nNow recompile the code, then run it, and then inspect the contents of the ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt")," file."),(0,r.kt)("p",null,"If the new message were ",(0,r.kt)("inlineCode",{parentName:"p"},'"Something short"'),", the contents of ",(0,r.kt)("inlineCode",{parentName:"p"},"write_file.txt")," should be:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/file-descriptors$ cat ../../support/file-descriptors/write_file.txt\nSomething shorte_file.txt: What's up, Doc?\n")),(0,r.kt)("p",null,"Note that the final bytes of the previous text remain unchanged.\nThe new message was simply written ",(0,r.kt)("strong",{parentName:"p"},"on top")," of the old one."),(0,r.kt)("p",null,"Now let's do a quick test.\nWe haven't talked about how redirections work in the terminal (we'll get there, step by step), but you can imagine that if you type ",(0,r.kt)("inlineCode",{parentName:"p"},"ls > file.txt"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"file.txt")," has to be opened at some point.\nLet's write data to a file twice and observe the behaviour:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/file-descriptors$ ls -l > file.txt\n\nstudent@os:~/.../lab/support/file-descriptors$ cat file.txt\ntotal 6\n-rw-rw-r-- 1 student student 0 Nov 20 21:11 file.txt\n-rw-rw-r-- 1 student student 125 Nov 20 18:26 Makefile\n-rw-rw-r-- 1 student student 396 Nov 20 21:10 open_directory.c\n-rw-rw-r-- 1 student student 2300 Nov 20 20:51 open_read_write.c\n-rw-rw-r-- 1 student student 34 Nov 20 18:26 read_file.txt\n-rw-r--r-- 1 student student 45 Nov 20 20:56 write_file.txt\n\nstudent@os:~/.../lab/support/file-descriptors$ ls > file.txt\n\nstudent@os:~/.../lab/support/file-descriptors$ cat file.txt\nfile.txt\nMakefile\nopen_directory.c\nopen_read_write.c\nread_file.txt\nwrite_file.txt\n")),(0,r.kt)("p",null,"The second output is shorter than the first, yet the first output is no longer present in the file after the second ",(0,r.kt)("inlineCode",{parentName:"p"},"ls"),".\nHow come?\nWell, the reason is another flag being passed to ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),": ",(0,r.kt)("inlineCode",{parentName:"p"},"O_TRUNC"),".\nAt this point, you should be accustomed to looking for this flag in ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),"'s man page.\nGo ahead and do it."),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/o-trunc"},"Quiz 1")),(0,r.kt)("p",null,(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/fopen-w"},"Quiz 2")),(0,r.kt)("h3",{id:"practice-closeem-all"},"Practice: Close'em All"),(0,r.kt)("p",null,"Just like you use ",(0,r.kt)("inlineCode",{parentName:"p"},"open()")," to create new file descriptors, you can use ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/close.2.html"},(0,r.kt)("inlineCode",{parentName:"a"},"close()"))," to destroy them.\nThis clears and frees the open file structure to which that entry in the file descriptor table is pointing.\nUse ",(0,r.kt)("inlineCode",{parentName:"p"},"close()")," on the file descriptors you've opened so far in ",(0,r.kt)("inlineCode",{parentName:"p"},"support/file-descriptors/open_read_write.c"),"."),(0,r.kt)("p",null,"Note that you don't have to close file descriptors 0, 1 and 2 manually.\nThe standard streams are meant to stay alive throughout the lifetime of the process.\nJust like calling ",(0,r.kt)("inlineCode",{parentName:"p"},"free()")," on a ",(0,r.kt)("inlineCode",{parentName:"p"},"malloc()"),"-ed pointer, calling ",(0,r.kt)("inlineCode",{parentName:"p"},"close()")," is not really necessary.\nWhen a process terminates, the OS closes all its file descriptors the same way it frees all its memory."),(0,r.kt)("p",null,"And keeping this comparison with ",(0,r.kt)("inlineCode",{parentName:"p"},"malloc()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"free()"),", closing file descriptors is important when they are created inside a loop, as the file descriptor table's size is limited."),(0,r.kt)("h2",{id:"file-handling-conclusion-libc-vs-syscalls"},"File Handling Conclusion: libc vs syscalls"),(0,r.kt)("p",null,"Up to now, we can draw some parallels between ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"open()"),".\nWhile ",(0,r.kt)("inlineCode",{parentName:"p"},"fopen()")," allows the usage of high-level functions such as ",(0,r.kt)("inlineCode",{parentName:"p"},"fread()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"fwrite()"),", which, among other things, use ",(0,r.kt)("inlineCode",{parentName:"p"},"while")," loops to ensure they always read the required number of bytes, the libc-specific API is not generic enough."),(0,r.kt)("p",null,"In the following sections, we'll use file descriptors and ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,r.kt)("inlineCode",{parentName:"p"},"write()")," to interact with some inter-process-communication mechanisms, such as pipes."),(0,r.kt)("p",null,"The table below shows the higher level API provided by libc and the syscalls it relies on.\nAs usual, use the ",(0,r.kt)("inlineCode",{parentName:"p"},"man")," pages when in doubt about either of them."),(0,r.kt)("table",null,(0,r.kt)("thead",{parentName:"table"},(0,r.kt)("tr",{parentName:"thead"},(0,r.kt)("th",{parentName:"tr",align:"center"},"libc"),(0,r.kt)("th",{parentName:"tr",align:"center"},"syscall"))),(0,r.kt)("tbody",{parentName:"table"},(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"fopen()")),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"open()"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"fread()")),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"read()"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"fwrite()")),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"write()"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"fseek()")),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"lseek()"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"fclose()")),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"close()"))))),(0,r.kt)("p",null,"So for most equivalents, just remove the leading ",(0,r.kt)("inlineCode",{parentName:"p"},"f")," when moving from the libc function to the underlying syscall."),(0,r.kt)("p",null,"For a quick recap of the flags we've discussed so far, take a look at the following table.\nBut don't bother memorising it.\nYou can find it any time in by typing ",(0,r.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/fopen.3.html"},(0,r.kt)("inlineCode",{parentName:"a"},"man fopen"))," in your terminal."),(0,r.kt)("table",null,(0,r.kt)("thead",{parentName:"table"},(0,r.kt)("tr",{parentName:"thead"},(0,r.kt)("th",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"th"},"fopen()")," mode"),(0,r.kt)("th",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"th"},"open()")," flag"))),(0,r.kt)("tbody",{parentName:"table"},(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},'"r"')),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"O_RDONLY"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},'"w"')),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"O_WRONLY \u2502 O_CREAT \u2502 O_TRUNC"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},'"a"')),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"O_WRONLY \u2502 O_CREAT \u2502 O_APPEND"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},'"r+"')),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"O_RDWR"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},'"w+"')),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"O_RDWR \u2502 O_CREAT \u2502 O_TRUNC"))),(0,r.kt)("tr",{parentName:"tbody"},(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},'"a+"')),(0,r.kt)("td",{parentName:"tr",align:"center"},(0,r.kt)("inlineCode",{parentName:"td"},"O_RDWR \u2502 O_CREAT \u2502 O_APPEND"))))))}c.isMDXComponent=!0},7367:(e,t,n)=>{n.d(t,{Z:()=>a});const a=n.p+"assets/images/file-descriptors-d19424f0a417ecd1c032f98a1969ad75.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/8ec866b0.1cec016d.js b/17/assets/js/8ec866b0.1cec016d.js new file mode 100644 index 0000000000..910315f778 --- /dev/null +++ b/17/assets/js/8ec866b0.1cec016d.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6723],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>h});var s=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);t&&(s=s.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,s)}return n}function r(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(s=0;s=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var o=s.createContext({}),p=function(e){var t=s.useContext(o),n=t;return e&&(n="function"==typeof e?e(t):r(r({},t),e)),n},c=function(e){var t=p(e.components);return s.createElement(o.Provider,{value:t},e.children)},m="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return s.createElement(s.Fragment,{},t)}},u=s.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,o=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),m=p(n),u=a,h=m["".concat(o,".").concat(u)]||m[u]||d[u]||i;return n?s.createElement(h,r(r({ref:t},c),{},{components:n})):s.createElement(h,r({ref:t},c))}));function h(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,r=new Array(i);r[0]=u;var l={};for(var o in t)hasOwnProperty.call(t,o)&&(l[o]=t[o]);l.originalType=e,l[m]="string"==typeof e?e:a,r[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>o,contentTitle:()=>r,default:()=>d,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var s=n(7462),a=(n(7294),n(3905));const i={},r="Asynchronous Web Server",l={unversionedId:"Assignments/Asynchronous Web Server/README",id:"Assignments/Asynchronous Web Server/README",title:"Asynchronous Web Server",description:"Objectives",source:"@site/docs/Assignments/Asynchronous Web Server/README.md",sourceDirName:"Assignments/Asynchronous Web Server",slug:"/Assignments/Asynchronous Web Server/",permalink:"/operating-systems/17/Assignments/Asynchronous Web Server/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Minishell",permalink:"/operating-systems/17/Assignments/Mini Shell/"},next:{title:"Rules and Grading",permalink:"/operating-systems/17/rules-and-grading"}},o={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Details and recommendations for the implementation",id:"details-and-recommendations-for-the-implementation",level:3},{value:"Support Code",id:"support-code",level:2},{value:"HTTP Parser",id:"http-parser",level:3},{value:"API and Implementation Tasks",id:"api-and-implementation-tasks",level:3},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Behind the Scenes",id:"behind-the-scenes",level:3},{value:"Debugging",id:"debugging",level:3},{value:"Resources",id:"resources",level:2}],c={toc:p},m="wrapper";function d(e){let{components:t,...n}=e;return(0,a.kt)(m,(0,s.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"asynchronous-web-server"},"Asynchronous Web Server"),(0,a.kt)("h2",{id:"objectives"},"Objectives"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Deepening the concepts related to working with sockets."),(0,a.kt)("li",{parentName:"ul"},"Developing skills in implementing and designing applications that use asynchronous operations and other advanced I/O operations."),(0,a.kt)("li",{parentName:"ul"},"Deepening the use of the API for advanced I/O operations in the Linux operating system.")),(0,a.kt)("h2",{id:"statement"},"Statement"),(0,a.kt)("p",null,"Implement a web server that uses the following advanced I/O operations:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Asynchronous operations on files"),(0,a.kt)("li",{parentName:"ul"},"Non-blocking operations on sockets"),(0,a.kt)("li",{parentName:"ul"},"Zero-copying"),(0,a.kt)("li",{parentName:"ul"},"Multiplexing I/O operations")),(0,a.kt)("p",null,"The server implements a limited functionality of the HTTP protocol: passing files to clients."),(0,a.kt)("p",null,"The web server will use the multiplexing API to wait for connections from clients - ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},"epoll"),".\nOn the established connections, requests from clients will be received and then responses will be distributed to them."),(0,a.kt)("p",null,"The server will serve files from the ",(0,a.kt)("inlineCode",{parentName:"p"},"AWS_DOCUMENT_ROOT")," directory, defined within the assignments' ",(0,a.kt)("a",{parentName:"p",href:"skel/aws.h"},"header"),".\nFiles are only found in subdirectories ",(0,a.kt)("inlineCode",{parentName:"p"},"AWS_DOCUMENT_ROOT/static/")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"AWS_DOCUMENT_ROOT/dynamic/"),".\nThe corresponding request paths will be, for example, ",(0,a.kt)("inlineCode",{parentName:"p"},"AWS_DOCUMENT_ROOT/static/test.dat")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"AWS_DOCUMENT_ROOT/dynamic/test.dat"),".\nThe file processing will be:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"The files in the ",(0,a.kt)("inlineCode",{parentName:"li"},"AWS_DOCUMENT_ROOT/static/")," directory are static files that will be transmitted to clients using the zero-copying API - ",(0,a.kt)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man2/sendfile.2.html"},"sendfile"),"]"),(0,a.kt)("li",{parentName:"ul"},"Files in the ",(0,a.kt)("inlineCode",{parentName:"li"},"AWS_DOCUMENT_ROOT/dynamic/")," directory are files that are supposed to require a server-side post-processing phase. These files will be read from disk using the asynchronous API and then pushed to the clients. Streaming will use non-blocking sockets (Linux)"),(0,a.kt)("li",{parentName:"ul"},"An ",(0,a.kt)("a",{parentName:"li",href:"https://en.wikipedia.org/wiki/HTTP_404"},"HTTP 404")," message will be sent for invalid request paths")),(0,a.kt)("p",null,"After transmitting a file, according to the HTTP protocol, the connection is closed."),(0,a.kt)("h3",{id:"details-and-recommendations-for-the-implementation"},"Details and recommendations for the implementation"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Implementing the assignment requires having a state machine for each connection, which you periodically query and update as the transfer proceeds.\nCheck the ",(0,a.kt)("inlineCode",{parentName:"li"},"connection_state")," data structure defined in the ",(0,a.kt)("a",{parentName:"li",href:"skel/awh.h"},"assignment header"),"."),(0,a.kt)("li",{parentName:"ul"},"Find the ",(0,a.kt)("inlineCode",{parentName:"li"},"connection")," data structure defined in the ",(0,a.kt)("a",{parentName:"li",href:"skel/awh.h"},"assignment header"),".\nThis can be used to keep track of an open connection."),(0,a.kt)("li",{parentName:"ul"},"Definitions of other useful macros and data structures can be found in the assignment header."),(0,a.kt)("li",{parentName:"ul"},"HTTP responses will have the code ",(0,a.kt)("inlineCode",{parentName:"li"},"200")," for existing files and ",(0,a.kt)("inlineCode",{parentName:"li"},"404")," for not existing files.",(0,a.kt)("ul",{parentName:"li"},(0,a.kt)("li",{parentName:"ul"},"A valid response consists of the HTTP header, containing the related directives, two newlines (",(0,a.kt)("inlineCode",{parentName:"li"},"\\r\\n\\r\\n"),"), followed by the actual content (the file)."),(0,a.kt)("li",{parentName:"ul"},"Sample answers can be found in the parser test file or in the provided sample."),(0,a.kt)("li",{parentName:"ul"},"You can use predefined request directives such as ",(0,a.kt)("inlineCode",{parentName:"li"},"Date"),", ",(0,a.kt)("inlineCode",{parentName:"li"},"Last-Modified"),", etc.",(0,a.kt)("ul",{parentName:"li"},(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"Content-Length")," directive ",(0,a.kt)("strong",{parentName:"li"},"must")," specify the size of the HTTP content (actual data) in bytes."),(0,a.kt)("li",{parentName:"ul"},"The ",(0,a.kt)("inlineCode",{parentName:"li"},"Connection")," directive ",(0,a.kt)("strong",{parentName:"li"},"must")," be initialized to ",(0,a.kt)("inlineCode",{parentName:"li"},"close"),"."))))),(0,a.kt)("li",{parentName:"ul"},"The port on which the web server listens for connections is defined within the assignment header: the ",(0,a.kt)("inlineCode",{parentName:"li"},"AWS_LISTEN_PORT")," macro."),(0,a.kt)("li",{parentName:"ul"},"The root directory relative to which the resources/files are searched is defined within the assignment header as the ",(0,a.kt)("inlineCode",{parentName:"li"},"AWS_DOCUMENT_ROOT")," macro.")),(0,a.kt)("h2",{id:"support-code"},"Support Code"),(0,a.kt)("h3",{id:"http-parser"},"HTTP Parser"),(0,a.kt)("p",null,"The clients and server will communicate using the HTTP protocol.\nFor parsing HTTP requests from clients we recommend using ",(0,a.kt)("a",{parentName:"p",href:"https://github.com/nodejs/http-parser"},"this HTTP parser"),", also available in the assignments' ",(0,a.kt)("a",{parentName:"p",href:"skel/http-parser"},"http-parser"),".\nYou will need to use a callback to get the path to the local resource requested by the client.\nFind a simplified example of using the parser in the ",(0,a.kt)("a",{parentName:"p",href:"skel/http-parser/samples/"},"samples directory"),"."),(0,a.kt)("h3",{id:"api-and-implementation-tasks"},"API and Implementation Tasks"),(0,a.kt)("p",null,"The ",(0,a.kt)("a",{parentName:"p",href:"skel/aws.c"},"skel/aws.c")," file contains the code skelethon with several functions that have to be implemented.\nFollow the ",(0,a.kt)("inlineCode",{parentName:"p"},"TODO")," areas in the file to start your implementation."),(0,a.kt)("blockquote",null,(0,a.kt)("p",{parentName:"blockquote"},"It can be reorganized as desired, as long as all the requirements of the assignment are implemented.")),(0,a.kt)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,a.kt)("p",null,"The testing is automated.\nTests are located in the ",(0,a.kt)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,a.kt)("p",null,"To test your implementation, do the following steps:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Run the ",(0,a.kt)("inlineCode",{parentName:"li"},"make")," command inside the ",(0,a.kt)("inlineCode",{parentName:"li"},"skel/")," directory and make sure it compiles with no errors and that the ",(0,a.kt)("inlineCode",{parentName:"li"},"aws")," executable is generated."),(0,a.kt)("li",{parentName:"ul"},"Run the ",(0,a.kt)("inlineCode",{parentName:"li"},"make check")," command in the ",(0,a.kt)("inlineCode",{parentName:"li"},"tests/")," directory.")),(0,a.kt)("p",null,"There are 35 tests for this assignment, of which 13 are doubled by a memory leak check test.\nA successful run looks as the following:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre"},"student@so:~/operating-systems/content/assignments/async-web-server/tests$ make check\nmake -C _test\nmake[1]: Entering directory '/home/student/operating-systems/content/assignments/async-web-server/tests/_test'\nmake[1]: Nothing to be done for 'all'.\nmake[1]: Leaving directory '/home/student/operating-systems/content/assignments/async-web-server/tests/_test'\n\n = Testing - Asynchronous Web Server =\n\n01) Test executable exists.............................................passed [01/90]\n02) Test executable runs...............................................passed [01/90]\n03) Test listening.....................................................passed [01/90]\n04) Test listening on port.............................................passed [01/90]\n05) Test accepts connections...........................................passed [01/90]\n06) Test accepts multiple connections..................................passed [01/90]\n07) Test epoll usage...................................................passed [01/90]\n08) Test disconnect....................................................passed [01/90]\n09) Test multiple disconnect...........................................passed [01/90]\n10) Test connect disconnect connect....................................passed [01/90]\n11) Test multiple connect disconnect connect...........................passed [01/90]\n12) Test unordered connect disconnect connect..........................passed [01/90]\n13) Test replies http request..........................................passed [02/90]\n13) Test replies http request - memcheck...............................passed [01/90]\n14) Test second replies http request...................................passed [01/90]\n15) Test sendfile usage................................................passed [02/90]\n16) Test small static file wget........................................passed [02/90]\n17) Test small static file wget cmp....................................passed [04/90]\n17) Test small static file wget cmp - memcheck.........................passed [01/90]\n18) Test large static file wget........................................passed [02/90]\n19) Test large static file wget cmp....................................passed [04/90]\n19) Test large static file wget cmp - memcheck.........................passed [01/90]\n20) Test bad static file 404...........................................passed [02/90]\n21) Test bad path 404..................................................passed [02/90]\n22) Test get one static file then another..............................passed [02/90]\n22) Test get one static file then another - memcheck...................passed [01/90]\n23) Test get two simultaneous static files.............................passed [03/90]\n23) Test get two simultaneous static files - memcheck..................passed [01/90]\n24) Test get multiple simultaneous static files........................passed [04/90]\n24) Test get multiple simultaneous static files - memcheck.............passed [01/90]\n25) Test io submit uses................................................passed [02/90]\n26) Test small dynamic file wget.......................................passed [02/90]\n27) Test small dynamic file wget cmp...................................passed [04/90]\n27) Test small dynamic file wget cmp - memcheck........................passed [01/90]\n28) Test large dynamic file wget.......................................passed [02/90]\n29) Test large dynamic file wget cmp...................................passed [04/90]\n29) Test large dynamic file wget cmp - memcheck........................passed [01/90]\n30) Test bad dynamic file 404..........................................passed [02/90]\n31) Test get one dynamic file then another.............................passed [03/90]\n31) Test get one dynamic file then another - memcheck..................passed [01/90]\n32) Test get two simultaneous dynamic files............................passed [04/90]\n32) Test get two simultaneous dynamic files - memcheck.................passed [01/90]\n33) Test get multiple simultaneous dynamic files.......................passed [05/90]\n33) Test get multiple simultaneous dynamic files - memcheck............passed [01/90]\n34) Test get two simultaneous static and dynamic files.................passed [03/90]\n34) Test get two simultaneous static and dynamic files - memcheck......passed [01/90]\n35) Test get multiple simultaneous static and dynamic files............passed [04/90]\n35) Test get multiple simultaneous static and dynamic files - memcheck.passed [01/90]\n\n Total: [90/100]\n")),(0,a.kt)("p",null,"Individual tests can be run using the ",(0,a.kt)("inlineCode",{parentName:"p"},"./run_test.sh")," bash script as the following:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre"},"student@so:~/operating-systems/content/assignments/async-web-server/tests$ ./_test/run_test.sh 3\n03) Test listening.....................................................passed [01/90]\n\n")),(0,a.kt)("p",null,"Where ",(0,a.kt)("inlineCode",{parentName:"p"},"3")," is the test you want to run."),(0,a.kt)("p",null,"Some tests are doubled by a memory check test.\nThis will only run if the regular test passed.\nFor example, test 31 will output the following in case of success:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre"},"student@so:~/operating-systems/content/assignments/async-web-server/tests$ ./_test/run_test.sh 31\n31) Test get one dynamic file then another.............................passed [03/90]\n31) Test get one dynamic file then another - memcheck..................passed [01/90]\n")),(0,a.kt)("p",null,"and one of the following in case of error:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre"},"# if the regular tests failed, the memory check tests is not performed\nstudent@so:~/operating-systems/content/assignments/async-web-server/tests$ ./_test/run_test.sh 31\n31) Test get one dynamic file then another.............................failed [ 0/90]\n31) Test get one dynamic file then another - memcheck..................passed [01/90]\n")),(0,a.kt)("blockquote",null,(0,a.kt)("p",{parentName:"blockquote"},"Note: The memcheck test for failed regular tests will not be taken into consideration for the final score.\nThis output will be fixed in the next commit.")),(0,a.kt)("p",null,"Tests use the ",(0,a.kt)("inlineCode",{parentName:"p"},"static/")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"dynamic/")," folders.\nThese folders are created and removed using the ",(0,a.kt)("inlineCode",{parentName:"p"},"init")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"cleanup")," arguments to ",(0,a.kt)("inlineCode",{parentName:"p"},"_test/run_test.sh"),"."),(0,a.kt)("h3",{id:"behind-the-scenes"},"Behind the Scenes"),(0,a.kt)("p",null,"Tests are basically unit tests."),(0,a.kt)("p",null,"Each test function follows the unit test patter: initialization, action,\nevaluation."),(0,a.kt)("p",null,"Each test starts the server, creates a given context, checks for validity and\nthen terminates the server process."),(0,a.kt)("h3",{id:"debugging"},"Debugging"),(0,a.kt)("p",null,"Logs are collected in ",(0,a.kt)("inlineCode",{parentName:"p"},"test.log")," and ",(0,a.kt)("inlineCode",{parentName:"p"},"wget.log")," files."),(0,a.kt)("h2",{id:"resources"},"Resources"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/sendfile.2.html"},"sendfile"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/io_setup.2.html"},"io_setup & friends"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},"epoll")))))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/8f35a413.4821de54.js b/17/assets/js/8f35a413.4821de54.js new file mode 100644 index 0000000000..31b48f9d5e --- /dev/null +++ b/17/assets/js/8f35a413.4821de54.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2713],{3905:(e,t,n)=>{n.d(t,{Zo:()=>s,kt:()=>b});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function o(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var p=a.createContext({}),u=function(e){var t=a.useContext(p),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},s=function(e){var t=u(e.components);return a.createElement(p.Provider,{value:t},e.children)},c="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,i=e.originalType,p=e.parentName,s=l(e,["components","mdxType","originalType","parentName"]),c=u(n),m=r,b=c["".concat(p,".").concat(m)]||c[m]||d[m]||i;return n?a.createElement(b,o(o({ref:t},s),{},{components:n})):a.createElement(b,o({ref:t},s))}));function b(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var i=n.length,o=new Array(i);o[0]=m;var l={};for(var p in t)hasOwnProperty.call(t,p)&&(l[p]=t[p]);l.originalType=e,l[c]="string"==typeof e?e:r,o[1]=l;for(var u=2;u{n.r(t),n.d(t,{assets:()=>p,contentTitle:()=>o,default:()=>d,frontMatter:()=>i,metadata:()=>l,toc:()=>u});var a=n(7462),r=(n(7294),n(3905));const i={},o="Stack layout",l={unversionedId:"Lab/Data/quiz/stack-layout",id:"Lab/Data/quiz/stack-layout",title:"Stack layout",description:"Question Text",source:"@site/docs/Lab/Data/quiz/stack-layout.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/stack-layout",permalink:"/operating-systems/17/Lab/Data/quiz/stack-layout",draft:!1,tags:[],version:"current",frontMatter:{}},p={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],s={toc:u},c="wrapper";function d(e){let{components:t,...n}=e;return(0,r.kt)(c,(0,a.Z)({},s,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"stack-layout"},"Stack layout"),(0,r.kt)("h2",{id:"question-text"},"Question Text"),(0,r.kt)("p",null,"What is the stack layout for the ",(0,r.kt)("inlineCode",{parentName:"p"},"fun")," function in the ",(0,r.kt)("inlineCode",{parentName:"p"},"bo_write.c")," program (starting from a high address)?"),(0,r.kt)("h2",{id:"question-answers"},"Question Answers"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"return address, old ",(0,r.kt)("inlineCode",{parentName:"li"},"rbp"),", maybe some padding, variable ",(0,r.kt)("inlineCode",{parentName:"li"},"a"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"b[0]"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"b[1]"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"b[2]"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"return address, old ",(0,r.kt)("inlineCode",{parentName:"li"},"rbp"),", maybe some padding, variable ",(0,r.kt)("inlineCode",{parentName:"li"},"a"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"b[2]"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"b[1]"),", ",(0,r.kt)("inlineCode",{parentName:"li"},"b[0]"))),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"return address, maybe some padding, variable ",(0,r.kt)("inlineCode",{parentName:"p"},"a"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"b[0]"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"b[1]"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"b[2]"))),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("p",{parentName:"li"},"return address, old ",(0,r.kt)("inlineCode",{parentName:"p"},"rbp"),", maybe some padding, ",(0,r.kt)("inlineCode",{parentName:"p"},"b[0]"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"b[1]"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"b[2]"),", variable ",(0,r.kt)("inlineCode",{parentName:"p"},"a")))),(0,r.kt)("h2",{id:"feedback"},"Feedback"),(0,r.kt)("p",null,"Look at the assembly code and notice the exact layout."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/918b3e21.70516993.js b/17/assets/js/918b3e21.70516993.js new file mode 100644 index 0000000000..dbe0de457c --- /dev/null +++ b/17/assets/js/918b3e21.70516993.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1786],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>m});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var p=n.createContext({}),l=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},s=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,p=e.parentName,s=c(e,["components","mdxType","originalType","parentName"]),u=l(r),d=o,m=u["".concat(p,".").concat(d)]||u[d]||f[d]||i;return r?n.createElement(m,a(a({ref:t},s),{},{components:r})):n.createElement(m,a({ref:t},s))}));function m(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,a=new Array(i);a[0]=d;var c={};for(var p in t)hasOwnProperty.call(t,p)&&(c[p]=t[p]);c.originalType=e,c[u]="string"==typeof e?e:o,a[1]=c;for(var l=2;l{r.r(t),r.d(t,{assets:()=>l,contentTitle:()=>c,default:()=>d,frontMatter:()=>a,metadata:()=>p,toc:()=>s});var n=r(7462),o=(r(7294),r(3905)),i=r(4996);const a={title:"Application-Interaction"},c=void 0,p={unversionedId:"Lecture/Application-Interaction",id:"Lecture/Application-Interaction",title:"Application-Interaction",description:"Focus the slides and press F for fullscreen viewing.",source:"@site/docs/Lecture/Application-Interaction.mdx",sourceDirName:"Lecture",slug:"/Lecture/Application-Interaction",permalink:"/operating-systems/17/Lecture/Application-Interaction",draft:!1,tags:[],version:"current",frontMatter:{title:"Application-Interaction"},sidebar:"sidebar",previous:{title:"IO",permalink:"/operating-systems/17/Lecture/IO"},next:{title:"Lab",permalink:"/operating-systems/17/Lab/"}},l={},s=[],u={toc:s},f="wrapper";function d(e){let{components:t,...r}=e;return(0,o.kt)(f,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("div",{style:{display:"flex",width:"100%",height:"100%",flexDirection:"row"}},(0,o.kt)("iframe",{style:{flexGrow:1,border:"none",margin:0,padding:0},width:"100%",height:"500px",src:(0,i.Z)("/slides/Application-Interaction/index.html")})),(0,o.kt)("br",null),(0,o.kt)("admonition",{type:"tip"},(0,o.kt)("p",{parentName:"admonition"},"Focus the slides and press ",(0,o.kt)("strong",{parentName:"p"},"F")," for fullscreen viewing.")))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/91ac0728.dbf52636.js b/17/assets/js/91ac0728.dbf52636.js new file mode 100644 index 0000000000..44bfa2608a --- /dev/null +++ b/17/assets/js/91ac0728.dbf52636.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1157],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>d});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function i(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function a(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var p=n.createContext({}),l=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):a(a({},t),e)),r},s=function(e){var t=l(e.components);return n.createElement(p.Provider,{value:t},e.children)},u="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,i=e.originalType,p=e.parentName,s=c(e,["components","mdxType","originalType","parentName"]),u=l(r),m=o,d=u["".concat(p,".").concat(m)]||u[m]||f[m]||i;return r?n.createElement(d,a(a({ref:t},s),{},{components:r})):n.createElement(d,a({ref:t},s))}));function d(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=r.length,a=new Array(i);a[0]=m;var c={};for(var p in t)hasOwnProperty.call(t,p)&&(c[p]=t[p]);c.originalType=e,c[u]="string"==typeof e?e:o,a[1]=c;for(var l=2;l{r.r(t),r.d(t,{assets:()=>l,contentTitle:()=>c,default:()=>m,frontMatter:()=>a,metadata:()=>p,toc:()=>s});var n=r(7462),o=(r(7294),r(3905)),i=r(4996);const a={title:"IO"},c=void 0,p={unversionedId:"Lecture/IO",id:"Lecture/IO",title:"IO",description:"Focus the slides and press F for fullscreen viewing.",source:"@site/docs/Lecture/IO.mdx",sourceDirName:"Lecture",slug:"/Lecture/IO",permalink:"/operating-systems/17/Lecture/IO",draft:!1,tags:[],version:"current",frontMatter:{title:"IO"},sidebar:"sidebar",previous:{title:"Compute",permalink:"/operating-systems/17/Lecture/Compute"},next:{title:"Application-Interaction",permalink:"/operating-systems/17/Lecture/Application-Interaction"}},l={},s=[],u={toc:s},f="wrapper";function m(e){let{components:t,...r}=e;return(0,o.kt)(f,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("div",{style:{display:"flex",width:"100%",height:"100%",flexDirection:"row"}},(0,o.kt)("iframe",{style:{flexGrow:1,border:"none",margin:0,padding:0},width:"100%",height:"500px",src:(0,i.Z)("/slides/IO/index.html")})),(0,o.kt)("br",null),(0,o.kt)("admonition",{type:"tip"},(0,o.kt)("p",{parentName:"admonition"},"Focus the slides and press ",(0,o.kt)("strong",{parentName:"p"},"F")," for fullscreen viewing.")))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/924f80ad.29b21edc.js b/17/assets/js/924f80ad.29b21edc.js new file mode 100644 index 0000000000..8d7acaf86a --- /dev/null +++ b/17/assets/js/924f80ad.29b21edc.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7089],{3905:(e,t,r)=>{r.d(t,{Zo:()=>u,kt:()=>f});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function s(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var c=n.createContext({}),p=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):s(s({},t),e)),r},u=function(e){var t=p(e.components);return n.createElement(c.Provider,{value:t},e.children)},l="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},d=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,u=i(e,["components","mdxType","originalType","parentName"]),l=p(r),d=a,f=l["".concat(c,".").concat(d)]||l[d]||m[d]||o;return r?n.createElement(f,s(s({ref:t},u),{},{components:r})):n.createElement(f,s({ref:t},u))}));function f(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,s=new Array(o);s[0]=d;var i={};for(var c in t)hasOwnProperty.call(t,c)&&(i[c]=t[c]);i.originalType=e,i[l]="string"==typeof e?e:a,s[1]=i;for(var p=2;p{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>s,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>p});var n=r(7462),a=(r(7294),r(3905));const o={},s="Cgroups versus namespaces",i={unversionedId:"Lab/Application Interaction/quiz/cgroups-vs-namespaces",id:"Lab/Application Interaction/quiz/cgroups-vs-namespaces",title:"Cgroups versus namespaces",description:"Question Text",source:"@site/docs/Lab/Application Interaction/quiz/cgroups-vs-namespaces.md",sourceDirName:"Lab/Application Interaction/quiz",slug:"/Lab/Application Interaction/quiz/cgroups-vs-namespaces",permalink:"/operating-systems/17/Lab/Application Interaction/quiz/cgroups-vs-namespaces",draft:!1,tags:[],version:"current",frontMatter:{}},c={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:p},l="wrapper";function m(e){let{components:t,...r}=e;return(0,a.kt)(l,(0,n.Z)({},u,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"cgroups-versus-namespaces"},"Cgroups versus namespaces"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Which of the following affirmations about namespaces and cgroups is correct?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Cgroups provide resource management and namespaces provide isolation and security.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Namespaces provide resource management and cgroups provide isolation and security.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Both provide resource management.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Both provide isolation and security."))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"They both serve different purposes, cgroups provide resource management and namespaces provide isolation and security.\nCgroups manage resources, while namespaces isolate and secure them."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/926d7fb7.b879ae45.js b/17/assets/js/926d7fb7.b879ae45.js new file mode 100644 index 0000000000..344b1e796a --- /dev/null +++ b/17/assets/js/926d7fb7.b879ae45.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8051],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>f});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function o(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var c=r.createContext({}),u=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):o(o({},t),e)),n},p=function(e){var t=u(e.components);return r.createElement(c.Provider,{value:t},e.children)},s="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,i=e.originalType,c=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),s=u(n),d=a,f=s["".concat(c,".").concat(d)]||s[d]||m[d]||i;return n?r.createElement(f,o(o({ref:t},p),{},{components:n})):r.createElement(f,o({ref:t},p))}));function f(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var i=n.length,o=new Array(i);o[0]=d;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[s]="string"==typeof e?e:a,o[1]=l;for(var u=2;u{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>o,default:()=>m,frontMatter:()=>i,metadata:()=>l,toc:()=>u});var r=n(7462),a=(n(7294),n(3905));const i={},o="VM Creation",l={unversionedId:"Lab/Application Interaction/quiz/vm-creation",id:"Lab/Application Interaction/quiz/vm-creation",title:"VM Creation",description:"Question Text",source:"@site/docs/Lab/Application Interaction/quiz/vm-creation.md",sourceDirName:"Lab/Application Interaction/quiz",slug:"/Lab/Application Interaction/quiz/vm-creation",permalink:"/operating-systems/17/Lab/Application Interaction/quiz/vm-creation",draft:!1,tags:[],version:"current",frontMatter:{}},c={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:u},s="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(s,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"vm-creation"},"VM Creation"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"How do you create a new virtual machine with a memory of 3GB and disk size of 100 GB?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"By running ",(0,a.kt)("inlineCode",{parentName:"li"},'curl -H "Content-Type: application/json" -d \'{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "100G"}\' localhost:5000/vm_create'))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"By running ",(0,a.kt)("inlineCode",{parentName:"p"},'curl -H "Content-Type: application/json" -d \'{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "100G"}\' localhost:5000/vm_delete'))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"By running ",(0,a.kt)("inlineCode",{parentName:"p"},'curl -H "Content-Type: application/json" -d \'{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "101G"}\' localhost:5000/vm_create'))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"By running ",(0,a.kt)("inlineCode",{parentName:"p"},'curl -H "Content-Type: application/json" -d \'{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "6G", "disk_size": "1000G"}\' localhost:5000/vm_delete')))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"We need to uso ",(0,a.kt)("inlineCode",{parentName:"p"},"curl")," with the right path ",(0,a.kt)("inlineCode",{parentName:"p"},"localhost:5000/vm_create"),", specifying the right data ",(0,a.kt)("inlineCode",{parentName:"p"},'{ "name": "my_vm", "image": "ubuntu_22.04", "network": "default", "mem_size": "3G", "disk_size": "100G"}'),"."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/928d0c11.8b988545.js b/17/assets/js/928d0c11.8b988545.js new file mode 100644 index 0000000000..eaadd4607f --- /dev/null +++ b/17/assets/js/928d0c11.8b988545.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9944],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>d});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function a(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var s=r.createContext({}),p=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},u=function(e){var t=p(e.components);return r.createElement(s.Provider,{value:t},e.children)},c="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},b=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,i=e.originalType,s=e.parentName,u=l(e,["components","mdxType","originalType","parentName"]),c=p(n),b=o,d=c["".concat(s,".").concat(b)]||c[b]||f[b]||i;return n?r.createElement(d,a(a({ref:t},u),{},{components:n})):r.createElement(d,a({ref:t},u))}));function d(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=n.length,a=new Array(i);a[0]=b;var l={};for(var s in t)hasOwnProperty.call(t,s)&&(l[s]=t[s]);l.originalType=e,l[c]="string"==typeof e?e:o,a[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>a,default:()=>f,frontMatter:()=>i,metadata:()=>l,toc:()=>p});var r=n(7462),o=(n(7294),n(3905));const i={},a="Sleeping on a Fiber",l={unversionedId:"Lab/Compute/quiz/sleeping-on-a-fiber",id:"Lab/Compute/quiz/sleeping-on-a-fiber",title:"Sleeping on a Fiber",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/sleeping-on-a-fiber.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/sleeping-on-a-fiber",permalink:"/operating-systems/17/Lab/Compute/quiz/sleeping-on-a-fiber",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:p},c="wrapper";function f(e){let{components:t,...n}=e;return(0,o.kt)(c,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"sleeping-on-a-fiber"},"Sleeping on a Fiber"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"What happens if a fiber calls ",(0,o.kt)("inlineCode",{parentName:"p"},"sleep()"),"?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"the whole kernel-level thread is blocked")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"only that fiber is blocked, and is scheduled out")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"nothing, a fiber can't sleep")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"the whole process sleeps - regardless of how many threads there are"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"The whole thread on which the fiber runs is blocked until the ",(0,o.kt)("inlineCode",{parentName:"p"},"sleep()")," call is finished.\nFor this reason, the fibers are best used with asynchronous operations, which you will explore in the following weeks."))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/935f2afb.76dc9ddc.js b/17/assets/js/935f2afb.76dc9ddc.js new file mode 100644 index 0000000000..a1cead1160 --- /dev/null +++ b/17/assets/js/935f2afb.76dc9ddc.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[53],{1109:e=>{e.exports=JSON.parse('{"pluginId":"default","version":"current","label":"Next","banner":null,"badge":false,"noIndex":false,"className":"docs-version-current","isLast":true,"docsSidebars":{"sidebar":[{"type":"link","label":"Introduction","href":"/operating-systems/17/","docId":"README"},{"type":"category","label":"Lecture","items":[{"type":"link","label":"Software Stack","href":"/operating-systems/17/Lecture/Software-Stack","docId":"Lecture/Software-Stack"},{"type":"link","label":"Data","href":"/operating-systems/17/Lecture/Data","docId":"Lecture/Data"},{"type":"link","label":"Compute","href":"/operating-systems/17/Lecture/Compute","docId":"Lecture/Compute"},{"type":"link","label":"IO","href":"/operating-systems/17/Lecture/IO","docId":"Lecture/IO"},{"type":"link","label":"Application Interaction","href":"/operating-systems/17/Lecture/Application-Interaction","docId":"Lecture/Application-Interaction"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lecture/"},{"type":"category","label":"Lab","items":[{"type":"link","label":"Setting up the Lab Environment","href":"/operating-systems/17/Lab/lab-setup","docId":"Lab/lab-setup"},{"type":"category","label":"Software Stack","items":[{"type":"link","label":"Overview","href":"/operating-systems/17/Lab/Software Stack/overview","docId":"Lab/Software Stack/overview"},{"type":"link","label":"Modern Software Stacks","href":"/operating-systems/17/Lab/Software Stack/modern-sw-stack","docId":"Lab/Software Stack/modern-sw-stack"},{"type":"link","label":"Basic System Calls","href":"/operating-systems/17/Lab/Software Stack/basic-syscall","docId":"Lab/Software Stack/basic-syscall"},{"type":"link","label":"System Call Wrapper","href":"/operating-systems/17/Lab/Software Stack/syscall-wrapper","docId":"Lab/Software Stack/syscall-wrapper"},{"type":"link","label":"Common Functions","href":"/operating-systems/17/Lab/Software Stack/common-functions","docId":"Lab/Software Stack/common-functions"},{"type":"link","label":"Libc","href":"/operating-systems/17/Lab/Software Stack/libc","docId":"Lab/Software Stack/libc"},{"type":"link","label":"Static-dynamic","href":"/operating-systems/17/Lab/Software Stack/static-dynamic","docId":"Lab/Software Stack/static-dynamic"},{"type":"link","label":"Libcall-Syscall","href":"/operating-systems/17/Lab/Software Stack/libcall-syscall","docId":"Lab/Software Stack/libcall-syscall"},{"type":"link","label":"High-Level Languages","href":"/operating-systems/17/Lab/Software Stack/high-level-lang","docId":"Lab/Software Stack/high-level-lang"},{"type":"link","label":"App Investigation","href":"/operating-systems/17/Lab/Software Stack/app-investigate","docId":"Lab/Software Stack/app-investigate"},{"type":"link","label":"Arena","href":"/operating-systems/17/Lab/Software Stack/arena","docId":"Lab/Software Stack/arena"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lab/Software Stack/"},{"type":"category","label":"Data","items":[{"type":"link","label":"Overview","href":"/operating-systems/17/Lab/Data/overview","docId":"Lab/Data/overview"},{"type":"link","label":"Working with Memory","href":"/operating-systems/17/Lab/Data/working-memory","docId":"Lab/Data/working-memory"},{"type":"link","label":"Process Memory","href":"/operating-systems/17/Lab/Data/process-memory","docId":"Lab/Data/process-memory"},{"type":"link","label":"Investigate Memory","href":"/operating-systems/17/Lab/Data/investigate-memory","docId":"Lab/Data/investigate-memory"},{"type":"link","label":"Memory Security","href":"/operating-systems/17/Lab/Data/memory-security","docId":"Lab/Data/memory-security"},{"type":"link","label":"Arena","href":"/operating-systems/17/Lab/Data/arena","docId":"Lab/Data/arena"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lab/Data/"},{"type":"category","label":"Compute","items":[{"type":"link","label":"Overview","href":"/operating-systems/17/Lab/Compute/overview","docId":"Lab/Compute/overview"},{"type":"link","label":"Hardware Perspective","href":"/operating-systems/17/Lab/Compute/hardware-perspective","docId":"Lab/Compute/hardware-perspective"},{"type":"link","label":"Processes","href":"/operating-systems/17/Lab/Compute/processes","docId":"Lab/Compute/processes"},{"type":"link","label":"Threads","href":"/operating-systems/17/Lab/Compute/threads","docId":"Lab/Compute/threads"},{"type":"link","label":"Processes-threads-apache2","href":"/operating-systems/17/Lab/Compute/processes-threads-apache2","docId":"Lab/Compute/processes-threads-apache2"},{"type":"link","label":"Copy-on-Write","href":"/operating-systems/17/Lab/Compute/copy-on-write","docId":"Lab/Compute/copy-on-write"},{"type":"link","label":"Synchronization","href":"/operating-systems/17/Lab/Compute/synchronization","docId":"Lab/Compute/synchronization"},{"type":"link","label":"User-Level Threads","href":"/operating-systems/17/Lab/Compute/user-level-threads","docId":"Lab/Compute/user-level-threads"},{"type":"link","label":"Arena","href":"/operating-systems/17/Lab/Compute/arena","docId":"Lab/Compute/arena"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lab/Compute/"},{"type":"category","label":"IO","items":[{"type":"link","label":"Overview","href":"/operating-systems/17/Lab/IO/overview","docId":"Lab/IO/overview"},{"type":"link","label":"File Handlers","href":"/operating-systems/17/Lab/IO/file-handlers","docId":"Lab/IO/file-handlers"},{"type":"link","label":"File Descriptors","href":"/operating-systems/17/Lab/IO/file-descriptors","docId":"Lab/IO/file-descriptors"},{"type":"link","label":"Redirections","href":"/operating-systems/17/Lab/IO/redirections","docId":"Lab/IO/redirections"},{"type":"link","label":"Pipes","href":"/operating-systems/17/Lab/IO/pipes","docId":"Lab/IO/pipes"},{"type":"link","label":"Local IO in Action","href":"/operating-systems/17/Lab/IO/local-io-in-action","docId":"Lab/IO/local-io-in-action"},{"type":"link","label":"Remote IO","href":"/operating-systems/17/Lab/IO/remote-io","docId":"Lab/IO/remote-io"},{"type":"link","label":"Networking 101","href":"/operating-systems/17/Lab/IO/networking-101","docId":"Lab/IO/networking-101"},{"type":"link","label":"Client-Server Model","href":"/operating-systems/17/Lab/IO/client-server-model","docId":"Lab/IO/client-server-model"},{"type":"link","label":"Beyond Network Sockets","href":"/operating-systems/17/Lab/IO/beyond-network-sockets","docId":"Lab/IO/beyond-network-sockets"},{"type":"link","label":"File Mappings","href":"/operating-systems/17/Lab/IO/file-mappings","docId":"Lab/IO/file-mappings"},{"type":"link","label":"IO Internals","href":"/operating-systems/17/Lab/IO/io-internals","docId":"Lab/IO/io-internals"},{"type":"link","label":"Zero-Copy","href":"/operating-systems/17/Lab/IO/zero-copy","docId":"Lab/IO/zero-copy"},{"type":"link","label":"Asynchronous IO","href":"/operating-systems/17/Lab/IO/async-io","docId":"Lab/IO/async-io"},{"type":"link","label":"IO Multiplexing","href":"/operating-systems/17/Lab/IO/io-multiplexing","docId":"Lab/IO/io-multiplexing"},{"type":"link","label":"Arena","href":"/operating-systems/17/Lab/IO/arena","docId":"Lab/IO/arena"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lab/IO/"},{"type":"category","label":"Application Interaction","items":[{"type":"link","label":"Overview","href":"/operating-systems/17/Lab/Application Interaction/overview","docId":"Lab/Application Interaction/overview"},{"type":"link","label":"Time Server","href":"/operating-systems/17/Lab/Application Interaction/time-server","docId":"Lab/Application Interaction/time-server"},{"type":"link","label":"Password Cracker","href":"/operating-systems/17/Lab/Application Interaction/password-cracker","docId":"Lab/Application Interaction/password-cracker"},{"type":"link","label":"The X Window System","href":"/operating-systems/17/Lab/Application Interaction/x-window-system","docId":"Lab/Application Interaction/x-window-system"},{"type":"link","label":"D-Bus","href":"/operating-systems/17/Lab/Application Interaction/dbus","docId":"Lab/Application Interaction/dbus"},{"type":"link","label":"OS Cloud","href":"/operating-systems/17/Lab/Application Interaction/os-cloud","docId":"Lab/Application Interaction/os-cloud"},{"type":"link","label":"Arena","href":"/operating-systems/17/Lab/Application Interaction/arena","docId":"Lab/Application Interaction/arena"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lab/Application Interaction/"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Lab/"},{"type":"category","label":"Assignments","items":[{"type":"link","label":"Mini Libc","href":"/operating-systems/17/Assignments/Mini Libc/","docId":"Assignments/Mini Libc/README"},{"type":"link","label":"Memory Allocator","href":"/operating-systems/17/Assignments/Memory Allocator/","docId":"Assignments/Memory Allocator/README"},{"type":"link","label":"Parallel Graph","href":"/operating-systems/17/Assignments/Parallel Graph/","docId":"Assignments/Parallel Graph/README"},{"type":"link","label":"Mini Shell","href":"/operating-systems/17/Assignments/Mini Shell/","docId":"Assignments/Mini Shell/README"},{"type":"link","label":"Asynchronous Web Server","href":"/operating-systems/17/Assignments/Asynchronous Web Server/","docId":"Assignments/Asynchronous Web Server/README"}],"collapsed":true,"collapsible":true,"href":"/operating-systems/17/Assignments/"},{"type":"link","label":"Rules and Grading","href":"/operating-systems/17/rules-and-grading","docId":"rules-and-grading"}]},"docs":{"Assignments/Asynchronous Web Server/README":{"id":"Assignments/Asynchronous Web Server/README","title":"Asynchronous Web Server","description":"Objectives","sidebar":"sidebar"},"Assignments/Asynchronous Web Server/src/http-parser/README":{"id":"Assignments/Asynchronous Web Server/src/http-parser/README","title":"HTTP Parser","description":"This is a parser for HTTP messages written in C. It parses both requests and"},"Assignments/Memory Allocator/README":{"id":"Assignments/Memory Allocator/README","title":"Memory Allocator","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Libc/README":{"id":"Assignments/Mini Libc/README","title":"Mini-libc","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Shell/README":{"id":"Assignments/Mini Shell/README","title":"Minishell","description":"Objectives","sidebar":"sidebar"},"Assignments/Mini Shell/util/parser/README":{"id":"Assignments/Mini Shell/util/parser/README","title":"Parser","description":"The parser is made using Bison and Flex."},"Assignments/Parallel Graph/README":{"id":"Assignments/Parallel Graph/README","title":"Parallel Graph","description":"For this assignment we will implement a generic thread pool, which we will then use to traverse a graph and compute the sum of the elements contained by the nodes.","sidebar":"sidebar"},"Lab/Application Interaction/arena":{"id":"Lab/Application Interaction/arena","title":"Arena","description":"Oneko","sidebar":"sidebar"},"Lab/Application Interaction/dbus":{"id":"Lab/Application Interaction/dbus","title":"D-Bus","description":"D-Bus is an Inter-Process Communication (IPC) mechanism that is commonly present on Linux.","sidebar":"sidebar"},"Lab/Application Interaction/os-cloud":{"id":"Lab/Application Interaction/os-cloud","title":"OS Cloud","description":"In this section, we are going to build a \\"toy cloud\\" called OS Cloud.","sidebar":"sidebar"},"Lab/Application Interaction/overview":{"id":"Lab/Application Interaction/overview","title":"Application Interaction","description":"In this chapter, you will discover various mechanisms through which applications on a system can interact.","sidebar":"sidebar"},"Lab/Application Interaction/password-cracker":{"id":"Lab/Application Interaction/password-cracker","title":"Password Cracker","description":"In this example, we will solve the following problem: given the sha512 hash of a password, we want to obtain the password that generated the hash.","sidebar":"sidebar"},"Lab/Application Interaction/quiz/cgroups-vs-namespaces":{"id":"Lab/Application Interaction/quiz/cgroups-vs-namespaces","title":"Cgroups versus namespaces","description":"Question Text"},"Lab/Application Interaction/quiz/container-vs-vm":{"id":"Lab/Application Interaction/quiz/container-vs-vm","title":"Container versus VM","description":"Question Text"},"Lab/Application Interaction/quiz/time-server":{"id":"Lab/Application Interaction/quiz/time-server","title":"Time Server Protocol","description":"Question Text"},"Lab/Application Interaction/quiz/time-server-interop":{"id":"Lab/Application Interaction/quiz/time-server-interop","title":"Time Server Interoperability","description":"Question Text"},"Lab/Application Interaction/quiz/timer":{"id":"Lab/Application Interaction/quiz/timer","title":"Oneko Timer","description":"Question Text"},"Lab/Application Interaction/quiz/vm-creation":{"id":"Lab/Application Interaction/quiz/vm-creation","title":"VM Creation","description":"Question Text"},"Lab/Application Interaction/time-server":{"id":"Lab/Application Interaction/time-server","title":"Time Server","description":"Check out the code in support/time-server/server.c and support/time-server/client.c.","sidebar":"sidebar"},"Lab/Application Interaction/x-window-system":{"id":"Lab/Application Interaction/x-window-system","title":"The X Window System","description":"Unix-like systems that support a Graphical User Interface usually do this through the X Window System.","sidebar":"sidebar"},"Lab/Compute/arena":{"id":"Lab/Compute/arena","title":"Arena","description":"Threads and Processes: clone","sidebar":"sidebar"},"Lab/Compute/copy-on-write":{"id":"Lab/Compute/copy-on-write","title":"Copy-on-Write","description":"So far, you know that the parent and child process have separate virtual address spaces.","sidebar":"sidebar"},"Lab/Compute/hardware-perspective":{"id":"Lab/Compute/hardware-perspective","title":"Hardware Perspective","description":"The main criterion we use to rank CPUs is their computation power, i.e. their ability to crunch numbers and do math.","sidebar":"sidebar"},"Lab/Compute/overview":{"id":"Lab/Compute/overview","title":"Compute","description":"Contents","sidebar":"sidebar"},"Lab/Compute/processes":{"id":"Lab/Compute/processes","title":"Processes","description":"A process is simply a running program.","sidebar":"sidebar"},"Lab/Compute/processes-threads-apache2":{"id":"Lab/Compute/processes-threads-apache2","title":"Usage of Processes and Threads in `apache2`","description":"We\'ll take a look at how a real-world application - the apache2 HTTP server - makes use of processes and threads.","sidebar":"sidebar"},"Lab/Compute/quiz/apache2-strace":{"id":"Lab/Compute/quiz/apache2-strace","title":"`apache2` Document Root","description":"Question Text"},"Lab/Compute/quiz/cause-of-file-not-found-error":{"id":"Lab/Compute/quiz/cause-of-file-not-found-error","title":"Cause of `FileNotFoundError`","description":"Question Text"},"Lab/Compute/quiz/child-faults-after-write":{"id":"Lab/Compute/quiz/child-faults-after-write","title":"Child Faults After Write","description":"Question Text"},"Lab/Compute/quiz/coarse-vs-granular-critical-section":{"id":"Lab/Compute/quiz/coarse-vs-granular-critical-section","title":"Coarse vs Granular Critical Section","description":"Question Text"},"Lab/Compute/quiz/create-sleepy-process-ending":{"id":"Lab/Compute/quiz/create-sleepy-process-ending","title":"`create_sleepy` Process Ending","description":"Question Text"},"Lab/Compute/quiz/fiber-strace":{"id":"Lab/Compute/quiz/fiber-strace","title":"Fiber Strace","description":"Question Text"},"Lab/Compute/quiz/mini-shell-stops-after-command":{"id":"Lab/Compute/quiz/mini-shell-stops-after-command","title":"Mini-shell Stops After Command","description":"Question Text"},"Lab/Compute/quiz/mmap-cow-flag":{"id":"Lab/Compute/quiz/mmap-cow-flag","title":"Copy-on-write Flag for `mmap()`","description":"Question Text"},"Lab/Compute/quiz/notify-only-with-mutex":{"id":"Lab/Compute/quiz/notify-only-with-mutex","title":"Both Condition and Mutex","description":"Question Text"},"Lab/Compute/quiz/number-of-running-ults":{"id":"Lab/Compute/quiz/number-of-running-ults","title":"Number of RUNNING User-Level Threads","description":"Question Text"},"Lab/Compute/quiz/parent-faults-before-fork":{"id":"Lab/Compute/quiz/parent-faults-before-fork","title":"Parent Faults before `fork()`","description":"Question Text"},"Lab/Compute/quiz/parent-of-sleep-processes":{"id":"Lab/Compute/quiz/parent-of-sleep-processes","title":"Parent of `sleep` Processes","description":"Question Text"},"Lab/Compute/quiz/processes-speedup":{"id":"Lab/Compute/quiz/processes-speedup","title":"Processes Speedup","description":"Question Text"},"Lab/Compute/quiz/seg-fault-exit-code":{"id":"Lab/Compute/quiz/seg-fault-exit-code","title":"Seg Fault Exit Code","description":"Question Text"},"Lab/Compute/quiz/semaphore-equivalent":{"id":"Lab/Compute/quiz/semaphore-equivalent","title":"Semaphore Equivalent","description":"Question Text"},"Lab/Compute/quiz/sleeping-on-a-fiber":{"id":"Lab/Compute/quiz/sleeping-on-a-fiber","title":"Sleeping on a Fiber","description":"Question Text"},"Lab/Compute/quiz/state-of-new-ult":{"id":"Lab/Compute/quiz/state-of-new-ult","title":"State of new ULT","description":"Question Text"},"Lab/Compute/quiz/tcb-libult-unikraft":{"id":"Lab/Compute/quiz/tcb-libult-unikraft","title":"Similarities Between the TCBs of `libult` and Unikraft","description":"Question Text"},"Lab/Compute/quiz/thread-memory":{"id":"Lab/Compute/quiz/thread-memory","title":"Thread Memory","description":"Question Text"},"Lab/Compute/quiz/time-slice-value":{"id":"Lab/Compute/quiz/time-slice-value","title":"Time Slice Value","description":"Question Text"},"Lab/Compute/quiz/tls-synchronization":{"id":"Lab/Compute/quiz/tls-synchronization","title":"TLS Synchronization","description":"Question Text"},"Lab/Compute/quiz/tls-var-copies":{"id":"Lab/Compute/quiz/tls-var-copies","title":"TLS `var` Copies","description":"Question Text"},"Lab/Compute/quiz/type-of-scheduler-in-libult":{"id":"Lab/Compute/quiz/type-of-scheduler-in-libult","title":"Type of Scheduler in `libult.so`","description":"Question Text"},"Lab/Compute/quiz/ult-thread-ids":{"id":"Lab/Compute/quiz/ult-thread-ids","title":"ULT Thread IDs","description":"Question Text"},"Lab/Compute/quiz/who-calls-execve-parent":{"id":"Lab/Compute/quiz/who-calls-execve-parent","title":"Who Calls `execve` in the Log of the Parent Process?","description":"Question Text"},"Lab/Compute/quiz/why-use-completed-queue":{"id":"Lab/Compute/quiz/why-use-completed-queue","title":"The Need for a COMPLETED Queue","description":"Question Text"},"Lab/Compute/synchronization":{"id":"Lab/Compute/synchronization","title":"Synchronization","description":"So far, we\'ve used threads and processes without wondering how to \\"tell\\" them how to access shared data.","sidebar":"sidebar"},"Lab/Compute/threads":{"id":"Lab/Compute/threads","title":"Threads","description":"Spreading the Work Among Other Threads","sidebar":"sidebar"},"Lab/Compute/user-level-threads":{"id":"Lab/Compute/user-level-threads","title":"User-Level Threads","description":"User-level threads differ from the threads you are used to (kernel-level threads, those created by pthread_create).","sidebar":"sidebar"},"Lab/Data/arena":{"id":"Lab/Data/arena","title":"Arena","description":"Challenge tasks","sidebar":"sidebar"},"Lab/Data/investigate-memory":{"id":"Lab/Data/investigate-memory","title":"Investigate Memory Actions","description":"Memory actions generally mean:","sidebar":"sidebar"},"Lab/Data/memory-security":{"id":"Lab/Data/memory-security","title":"Memory Security","description":"Memory security is one of the most important aspects in today\'s computer systems.","sidebar":"sidebar"},"Lab/Data/overview":{"id":"Lab/Data/overview","title":"Data","description":"Data represents information that is to be processed to produce a final result or more data.","sidebar":"sidebar"},"Lab/Data/process-memory":{"id":"Lab/Data/process-memory","title":"Process Memory","description":"Memory Regions","sidebar":"sidebar"},"Lab/Data/quiz/bypass-canary":{"id":"Lab/Data/quiz/bypass-canary","title":"Bypass Canary","description":"Question"},"Lab/Data/quiz/half-page":{"id":"Lab/Data/quiz/half-page","title":"Half Page","description":"Question Text"},"Lab/Data/quiz/malloc-brk":{"id":"Lab/Data/quiz/malloc-brk","title":"Malloc `brk()`","description":"Question Text"},"Lab/Data/quiz/malloc-mmap":{"id":"Lab/Data/quiz/malloc-mmap","title":"Malloc `mmap()`","description":"Question Text"},"Lab/Data/quiz/memory-access":{"id":"Lab/Data/quiz/memory-access","title":"Modify String","description":"Question Text"},"Lab/Data/quiz/memory-aslr":{"id":"Lab/Data/quiz/memory-aslr","title":"ASLR","description":"Question Text"},"Lab/Data/quiz/memory-granularity":{"id":"Lab/Data/quiz/memory-granularity","title":"Memory Granularity","description":"Question Text"},"Lab/Data/quiz/memory-leaks":{"id":"Lab/Data/quiz/memory-leaks","title":"Memory Leaks","description":"Question Text"},"Lab/Data/quiz/memory-regions-vars":{"id":"Lab/Data/quiz/memory-regions-vars","title":"Variables in memory regions","description":"Question Text"},"Lab/Data/quiz/memory-stack-protector":{"id":"Lab/Data/quiz/memory-stack-protector","title":"Stack Protector","description":"Question Text"},"Lab/Data/quiz/mmap-file":{"id":"Lab/Data/quiz/mmap-file","title":"`mmap()` file","description":"Question Text"},"Lab/Data/quiz/operators":{"id":"Lab/Data/quiz/operators","title":"Operator Overloading","description":"Question Text"},"Lab/Data/quiz/page-allocation":{"id":"Lab/Data/quiz/page-allocation","title":"Page Allocation","description":"Question Text"},"Lab/Data/quiz/stack-layout":{"id":"Lab/Data/quiz/stack-layout","title":"Stack layout","description":"Question Text"},"Lab/Data/quiz/string-buff-over":{"id":"Lab/Data/quiz/string-buff-over","title":"String Buffer Overflow","description":"Question Text"},"Lab/Data/quiz/string-strcpy":{"id":"Lab/Data/quiz/string-strcpy","title":"Strcpy Buffer Overflow","description":"Question Text"},"Lab/Data/quiz/valgrind-leaks":{"id":"Lab/Data/quiz/valgrind-leaks","title":"Valgrind Leaks","description":"Question Text"},"Lab/Data/working-memory":{"id":"Lab/Data/working-memory","title":"Working with Memory","description":"As previously stated, from a programmer\'s perspective, memory is abstracted into variables.","sidebar":"sidebar"},"Lab/IO/arena":{"id":"Lab/IO/arena","title":"Arena","description":"Open File Structure in the Kernel","sidebar":"sidebar"},"Lab/IO/async-io":{"id":"Lab/IO/async-io","title":"Asynchronous I/O","description":"When doing I/O, the major issue we are facing is that I/O operations are typically much slower than CPU operations.","sidebar":"sidebar"},"Lab/IO/beyond-network-sockets":{"id":"Lab/IO/beyond-network-sockets","title":"Beyond Network Sockets","description":"Up until this point, we first learned how to use the Berkeley Sockets API, then we learned about the client-server model, based on this API.","sidebar":"sidebar"},"Lab/IO/client-server-model":{"id":"Lab/IO/client-server-model","title":"Client-Server Model","description":"Up to now, we\'ve avoided code snippets using TCP.","sidebar":"sidebar"},"Lab/IO/file-descriptors":{"id":"Lab/IO/file-descriptors","title":"File Descriptors","description":"After running the code in the \\"File Handlers\\" section, you saw that open() returns a number.","sidebar":"sidebar"},"Lab/IO/file-handlers":{"id":"Lab/IO/file-handlers","title":"File Handling","description":"You\'ve most likely had to deal with files in the past.","sidebar":"sidebar"},"Lab/IO/file-mappings":{"id":"Lab/IO/file-mappings","title":"File Mappings","description":"Mapping a file to the VAS of a process is similar to how shared libraries are loaded into the same VAS.","sidebar":"sidebar"},"Lab/IO/io-internals":{"id":"Lab/IO/io-internals","title":"I/O Internals","description":"Now, we will take a short look at how the file descriptors you\'ve just learnt about are handled in libc.","sidebar":"sidebar"},"Lab/IO/io-multiplexing":{"id":"Lab/IO/io-multiplexing","title":"I/O Multiplexing","description":"I/O multiplexing is the ability to serve multiple I/O channels (or anything that can be referenced via a file descriptor / handle) simultaneously.","sidebar":"sidebar"},"Lab/IO/local-io-in-action":{"id":"Lab/IO/local-io-in-action","title":"Local I/O in Action","description":"Most of the time, file handling is a simple operation from the perspective of the application.","sidebar":"sidebar"},"Lab/IO/networking-101":{"id":"Lab/IO/networking-101","title":"Networking 101","description":"In this section, we will briefly explore how networking works in general, from the perspective of the application.","sidebar":"sidebar"},"Lab/IO/overview":{"id":"Lab/IO/overview","title":"I/O","description":"We know that a compute system is a collection of hardware and software that modifies data.","sidebar":"sidebar"},"Lab/IO/pipes":{"id":"Lab/IO/pipes","title":"Pipes","description":"When it comes to inter-process communication, so far we know that 2 different processes can mmap() the same file and use that as some sort of shared memory, but this requires writing data to the disk which is slow.","sidebar":"sidebar"},"Lab/IO/quiz/anonymous-pipes-limitation":{"id":"Lab/IO/quiz/anonymous-pipes-limitation","title":"Limitation of Anonymous Pipes","description":"Question Text"},"Lab/IO/quiz/bind-error-cause":{"id":"Lab/IO/quiz/bind-error-cause","title":"Cause of `bind()` Error","description":"Question Text"},"Lab/IO/quiz/client-server-sender-receiver":{"id":"Lab/IO/quiz/client-server-sender-receiver","title":"`sender.py` and `receiver.py` Client-Server Parallel","description":"Question Text"},"Lab/IO/quiz/deluge-tcp-udp":{"id":"Lab/IO/quiz/deluge-tcp-udp","title":"Deluge: TCP or UDP","description":"Question Text"},"Lab/IO/quiz/execve":{"id":"Lab/IO/quiz/execve","title":"Effect of `execve()` Syscall","description":"Question Text"},"Lab/IO/quiz/fewer-than-2-copies":{"id":"Lab/IO/quiz/fewer-than-2-copies","title":"Fewer than Two Copies","description":"Question Text"},"Lab/IO/quiz/file-handler-c":{"id":"Lab/IO/quiz/file-handler-c","title":"File handler in C","description":"Question Text"},"Lab/IO/quiz/firefox-tcp-udp":{"id":"Lab/IO/quiz/firefox-tcp-udp","title":"Firefox: TCP or UDP?","description":"Question Text"},"Lab/IO/quiz/flush-libc-buffer":{"id":"Lab/IO/quiz/flush-libc-buffer","title":"Flush Libc Buffer","description":"Question Text"},"Lab/IO/quiz/fopen-syscall":{"id":"Lab/IO/quiz/fopen-syscall","title":"Syscall Used by `fopen()`","description":"Question Text"},"Lab/IO/quiz/fopen-w":{"id":"Lab/IO/quiz/fopen-w","title":"open()` equivalent of `fopen(..., \\"w\\")","description":"Question Text"},"Lab/IO/quiz/local-io-errors":{"id":"Lab/IO/quiz/local-io-errors","title":"I/O Errors","description":"Question Text"},"Lab/IO/quiz/mmap-read-write-benchmark":{"id":"Lab/IO/quiz/mmap-read-write-benchmark","title":"`mmap()` vs `read()` and `write()` Benchmark","description":"Question Text"},"Lab/IO/quiz/o-trunc":{"id":"Lab/IO/quiz/o-trunc","title":"`O_TRUNC` Flag Behaviour","description":"Question Text"},"Lab/IO/quiz/pipe-ends":{"id":"Lab/IO/quiz/pipe-ends","title":"Pipe Ends","description":"Question Text"},"Lab/IO/quiz/prints-work-no-stdio":{"id":"Lab/IO/quiz/prints-work-no-stdio","title":"Prints Working after Closing `stdio`","description":"Question Text"},"Lab/IO/quiz/receiver-socket-fd":{"id":"Lab/IO/quiz/receiver-socket-fd","title":"Receiver Socked File Descriptor","description":"Question Text"},"Lab/IO/quiz/server-copies":{"id":"Lab/IO/quiz/server-copies","title":"Client-Server Number of Copies","description":"Question Text"},"Lab/IO/quiz/stderr-fd":{"id":"Lab/IO/quiz/stderr-fd","title":"File Descriptor of `stderr`","description":"Question Text"},"Lab/IO/quiz/strace-printf":{"id":"Lab/IO/quiz/strace-printf","title":"`printf()` Under Strace","description":"Question Text"},"Lab/IO/quiz/syscalls-cp":{"id":"Lab/IO/quiz/syscalls-cp","title":"Syscalls Used by `cp`","description":"Question Text"},"Lab/IO/quiz/write-file-permissions":{"id":"Lab/IO/quiz/write-file-permissions","title":"`write_file.txt` Permissions","description":"Question Text"},"Lab/IO/redirections":{"id":"Lab/IO/redirections","title":"Redirections","description":"In the File Descriptors section, we mentioned redirections such as ls > file.txt.","sidebar":"sidebar"},"Lab/IO/remote-io":{"id":"Lab/IO/remote-io","title":"Remote I/O","description":"In the previous sections, we started looking into how applications interact with the outside world.","sidebar":"sidebar"},"Lab/IO/zero-copy":{"id":"Lab/IO/zero-copy","title":"Zero-Copy","description":"Imagine a server that responds with files that it stores locally.","sidebar":"sidebar"},"Lab/lab-setup":{"id":"Lab/lab-setup","title":"Setting up the Lab Environment","description":"If you have already cloned the repository, make sure it is updated:","sidebar":"sidebar"},"Lab/Software Stack/app-investigate":{"id":"Lab/Software Stack/app-investigate","title":"App Investigation","description":"Let\'s spend some time investigating actual applications residing on the local system.","sidebar":"sidebar"},"Lab/Software Stack/arena":{"id":"Lab/Software Stack/arena","title":"Arena","description":"Go through the practice items below to hone your skills in working with layers of the software stack.","sidebar":"sidebar"},"Lab/Software Stack/basic-syscall":{"id":"Lab/Software Stack/basic-syscall","title":"Analyzing the Software Stack","description":"To get a better grasp on how the software stack works, let\'s do a bottom-up approach:","sidebar":"sidebar"},"Lab/Software Stack/common-functions":{"id":"Lab/Software Stack/common-functions","title":"Common Functions","description":"By using wrapper calls, we are able to write our programs in C.","sidebar":"sidebar"},"Lab/Software Stack/high-level-lang":{"id":"Lab/Software Stack/high-level-lang","title":"High-Level Languages","description":"Using the standard C library (libc) frees the programmer from the cumbersome steps of invoking system calls and reimplementing common features.","sidebar":"sidebar"},"Lab/Software Stack/libc":{"id":"Lab/Software Stack/libc","title":"Libraries and libc","description":"Once we have common functions implemented, we can reuse them at any time.","sidebar":"sidebar"},"Lab/Software Stack/libcall-syscall":{"id":"Lab/Software Stack/libcall-syscall","title":"Library calls vs system calls","description":"The standard C library has primarily two uses:","sidebar":"sidebar"},"Lab/Software Stack/modern-sw-stack":{"id":"Lab/Software Stack/modern-sw-stack","title":"Modern Software Stacks","description":"Most modern computing systems use a software stack such as the one in the figure below:","sidebar":"sidebar"},"Lab/Software Stack/overview":{"id":"Lab/Software Stack/overview","title":"Software Stack","description":"Software comprises of code and data that is loaded in memory and used by the CPU.","sidebar":"sidebar"},"Lab/Software Stack/quiz/common-functions":{"id":"Lab/Software Stack/quiz/common-functions","title":"Common Functions","description":"printf() System Call"},"Lab/Software Stack/quiz/high-level-lang":{"id":"Lab/Software Stack/quiz/high-level-lang","title":"Python Tools","description":"Question Text"},"Lab/Software Stack/quiz/libc":{"id":"Lab/Software Stack/quiz/libc","title":"libc","description":"malloc()"},"Lab/Software Stack/quiz/libcall-syscall":{"id":"Lab/Software Stack/quiz/libcall-syscall","title":"Libcall with Syscall","description":"Question Text"},"Lab/Software Stack/quiz/libs":{"id":"Lab/Software Stack/quiz/libs","title":"libs","description":"Static Executables"},"Lab/Software Stack/quiz/software":{"id":"Lab/Software Stack/quiz/software","title":"Software Properties","description":"Question Text"},"Lab/Software Stack/quiz/syscall-wrapper":{"id":"Lab/Software Stack/quiz/syscall-wrapper","title":"Syscall Wrappers","description":"Question Text"},"Lab/Software Stack/quiz/syscalls":{"id":"Lab/Software Stack/quiz/syscalls","title":"Syscalls","description":"Syscall ID"},"Lab/Software Stack/static-dynamic":{"id":"Lab/Software Stack/static-dynamic","title":"Statically-linked and Dynamically-linked Libraries","description":"Libraries can be statically-linked or dynamically-linked, creating statically-linked executables and dynamically-linked executables.","sidebar":"sidebar"},"Lab/Software Stack/syscall-wrapper":{"id":"Lab/Software Stack/syscall-wrapper","title":"System Call Wrappers","description":"The support/syscall-wrapper/ folder stores the implementation of a simple program written in C (main.c) that calls the write() and exit() functions.","sidebar":"sidebar"},"Lecture/Application-Interaction":{"id":"Lecture/Application-Interaction","title":"Application-Interaction","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Lecture/Compute":{"id":"Lecture/Compute","title":"Compute","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Lecture/Data":{"id":"Lecture/Data","title":"Data","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Lecture/IO":{"id":"Lecture/IO","title":"IO","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"Lecture/Software-Stack":{"id":"Lecture/Software-Stack","title":"Software-Stack","description":"Focus the slides and press F for fullscreen viewing.","sidebar":"sidebar"},"README":{"id":"README","title":"Intro","description":"This is a landing page for your course.","sidebar":"sidebar"},"rules-and-grading":{"id":"rules-and-grading","title":"Rules and Grading","description":"Grading","sidebar":"sidebar"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/94458749.3b4f7c7a.js b/17/assets/js/94458749.3b4f7c7a.js new file mode 100644 index 0000000000..cd9512156b --- /dev/null +++ b/17/assets/js/94458749.3b4f7c7a.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6512],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>h});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function s(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var l=r.createContext({}),p=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(l.Provider,{value:t},e.children)},d="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},m=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,l=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),d=p(n),m=o,h=d["".concat(l,".").concat(m)]||d[m]||u[m]||a;return n?r.createElement(h,s(s({ref:t},c),{},{components:n})):r.createElement(h,s({ref:t},c))}));function h(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,s=new Array(a);s[0]=m;var i={};for(var l in t)hasOwnProperty.call(t,l)&&(i[l]=t[l]);i.originalType=e,i[d]="string"==typeof e?e:o,s[1]=i;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>s,default:()=>u,frontMatter:()=>a,metadata:()=>i,toc:()=>p});var r=n(7462),o=(n(7294),n(3905));const a={},s="Networking 101",i={unversionedId:"Lab/IO/networking-101",id:"Lab/IO/networking-101",title:"Networking 101",description:"In this section, we will briefly explore how networking works in general, from the perspective of the application.",source:"@site/docs/Lab/IO/networking-101.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/networking-101",permalink:"/operating-systems/17/Lab/IO/networking-101",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Remote I/O",permalink:"/operating-systems/17/Lab/IO/remote-io"},next:{title:"Client-Server Model",permalink:"/operating-systems/17/Lab/IO/client-server-model"}},l={},p=[{value:"UDP",id:"udp",level:2},{value:"TCP",id:"tcp",level:2},{value:"Practice: Encapsulation Example: Deluge Revived",id:"practice-encapsulation-example-deluge-revived",level:3},{value:"Local TCP and UDP Services",id:"local-tcp-and-udp-services",level:2},{value:"Practice: Run netstat Yourself",id:"practice-run-netstat-yourself",level:3},{value:"Conclusion",id:"conclusion",level:3}],c={toc:p},d="wrapper";function u(e){let{components:t,...a}=e;return(0,o.kt)(d,(0,r.Z)({},c,a,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"networking-101"},"Networking 101"),(0,o.kt)("p",null,"In this section, we will ",(0,o.kt)("strong",{parentName:"p"},"briefly")," explore how networking works in general, from the perspective of the application.\nUnderstanding the details of it, however, is beyond the scope of this course."),(0,o.kt)("p",null,"The main ",(0,o.kt)("strong",{parentName:"p"},"protocols")," used by applications are ",(0,o.kt)("strong",{parentName:"p"},"User Datagram Protocol (UDP)")," and ",(0,o.kt)("strong",{parentName:"p"},"Transmission Control Protocol (TCP)"),".\nWe can specify this protocol when creating a socket."),(0,o.kt)("h2",{id:"udp"},"UDP"),(0,o.kt)("p",null,"UDP is the simpler of the two protocols above.\nIt simply states that the sender should pass the data over to the receiver specified by an IP and a port.\nThe scripts in ",(0,o.kt)("inlineCode",{parentName:"p"},"support/send-receive/")," both used UDP.\nYou can tell by the fact they both created their sockets like so:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-Python"},"sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n")),(0,o.kt)("p",null,"The ",(0,o.kt)("inlineCode",{parentName:"p"},"socket.SOCK_DGRAM"),' argument stands for "datagram" and specifies UDP.'),(0,o.kt)("p",null,"It doesn't care whether the receiver has got all the data, whether it was corrupted or dropped altogether by some router along the way."),(0,o.kt)("p",null,"To prove this, rerun ",(0,o.kt)("inlineCode",{parentName:"p"},"support/send-receive/receiver.py"),", then run ",(0,o.kt)("inlineCode",{parentName:"p"},"support/send-receive/sender.py"),", type ",(0,o.kt)("inlineCode",{parentName:"p"},'"exit"')," and then run ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py")," again.\nYou'll see no error from the sender because whether the message reaches its destination or not is not important for UDP."),(0,o.kt)("p",null,"So far UDP might seem rather useless.\nIt doesn't confirm whether a message was received correctly or not, so why use it?\nWell, exactly because its mechanism is so simple, UDP is ",(0,o.kt)("strong",{parentName:"p"},"fast"),".\nTherefore, it is used by many ",(0,o.kt)("strong",{parentName:"p"},"real-time services"),", such as for streaming or video calls where if a frame drops or is incorrect, it doesn't really matter that much since it will immediately be replaced with the next frame, thus masking the error."),(0,o.kt)("h2",{id:"tcp"},"TCP"),(0,o.kt)("p",null,"TCP is the polar opposite of UDP.\nTCP makes sure the data is given to the application correctly by performing error checks on the receiving end and then retransmitting any incorrect or missing messages.\nFor this reason, TCP is good for applications that require precise data, such as banking applications, static images or text.\nThe cost of correctness, however, is transmission speed."),(0,o.kt)("h3",{id:"practice-encapsulation-example-deluge-revived"},"Practice: Encapsulation Example: Deluge Revived"),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/deluge-tcp-udp"},"Quiz")),(0,o.kt)("p",null,"You haven't forgotten about our favourite Bittorrent clint, ",(0,o.kt)("a",{parentName:"p",href:"https://deluge-torrent.org/"},"Deluge"),", have you?\nIt implements its own protocol for transferring data.\nIt is built on top of TCP using the ",(0,o.kt)("a",{parentName:"p",href:"https://twisted.org/documents/18.7.0/api/twisted.internet.interfaces.ITCPTransport.html"},(0,o.kt)("inlineCode",{parentName:"a"},"ITCPTransport")," interface"),"."),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"In the Deluge codebase, find the class ",(0,o.kt)("inlineCode",{parentName:"p"},"DelugeTransferProtocol")," and read the docstring comment about the format of its messages.")),(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Follow the code in the ",(0,o.kt)("inlineCode",{parentName:"p"},"transfer_message()")," method of the ",(0,o.kt)("inlineCode",{parentName:"p"},"DelugeTransferProtocol")," class and see the ",(0,o.kt)("inlineCode",{parentName:"p"},"message")," variable conform to the message format you've just read about.\n",(0,o.kt)("inlineCode",{parentName:"p"},"self.transport")," simply refers to the underlying TCP interface.\nSo ",(0,o.kt)("inlineCode",{parentName:"p"},"self.transport.write()")," invokes the whole socket creation boilerplate necessary to send data via TCP.\nWe'll discuss it in the ",(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/client-server-model#establishing-the-connection"},"next section"),"."))),(0,o.kt)("p",null,"This is what protocols are mostly about.\nThey describe which fields go where within a message and how they should be interpreted.\nFor example, this protocol says that the ",(0,o.kt)("inlineCode",{parentName:"p"},"body")," (the data itself) should be compressed before being sent.\nThis makes a lot of sense, as sending less data over the network results in lower latencies.\nHowever, if encryption is too time-consuming, it may not be worth it.\nThere are compromises to be made everywhere."),(0,o.kt)("h2",{id:"local-tcp-and-udp-services"},"Local TCP and UDP Services"),(0,o.kt)("p",null,"To get a full list of all network-handling processes in your system together with the protocols they're using, we can use the ",(0,o.kt)("inlineCode",{parentName:"p"},"netstat")," with the ",(0,o.kt)("inlineCode",{parentName:"p"},"-tuanp")," arguments.\n",(0,o.kt)("inlineCode",{parentName:"p"},"-tuanp")," is short for ",(0,o.kt)("inlineCode",{parentName:"p"},"-t -u -a -n -p"),", which stand for:"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"-t"),": list processes using the TCP protocol"),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"-u"),": list processes using the UDP protocol"),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"-a"),": list both servers and clients"),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"-n"),": list IPs in numeric format"),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("inlineCode",{parentName:"li"},"-p"),": show the PID and name of each program")),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ sudo netstat -tunp\nActive Internet connections (w/o servers)\nProto Recv-Q Send-Q Local Address Foreign Address State PID/Program name\ntcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1057/sshd: /usr/sbi\ntcp 0 0 127.0.0.1:6463 0.0.0.0:* LISTEN 3261/Discord --type\ntcp 0 0 192.168.100.2:51738 162.159.128.235:443 ESTABLISHED 3110/Discord --type\ntcp 0 0 192.168.100.2:43694 162.159.129.235:443 ESTABLISHED 3110/Discord --type\ntcp 0 0 192.168.100.2:56230 54.230.159.113:443 ESTABLISHED 9154/firefox\ntcp 0 0 192.168.100.2:38096 34.107.141.31:443 ESTABLISHED 9154/firefox\ntcp 0 0 192.168.100.2:42462 34.117.237.239:443 ESTABLISHED 9154/firefox\ntcp 0 0 192.168.100.2:41128 162.159.135.234:443 ESTABLISHED 3110/Discord --type\ntcp6 0 0 :::80 :::* LISTEN 1114/apache2\ntcp6 0 0 :::22 :::* LISTEN 1057/sshd: /usr/sbi\ntcp6 0 0 2a02:2f0a:c10f:97:55754 2a02:2f0c:dff0:b::1:443 ESTABLISHED 9154/firefox\ntcp6 0 0 2a02:2f0a:c10f:97:55750 2a02:2f0c:dff0:b::1:443 ESTABLISHED 9154/firefox\nudp 0 0 0.0.0.0:56585 0.0.0.0:* 3261/Discord --type\nudp 0 0 0.0.0.0:42629 0.0.0.0:* 3261/Discord --type\nudp6 0 0 :::52070 :::* 9154/firefox\nudp6 0 0 :::38542 :::* 9154/firefox\n")),(0,o.kt)("p",null,"Obviously, your output will differ from the one above.\nFirst look at the fourth column.\nIt shows the local address and port used by the process.\nAs we've already established, SSH uses port 22.\nApache2 uses port 80 for both IPv4 and IPv6 addresses (look for rows starting with ",(0,o.kt)("inlineCode",{parentName:"p"},"tcp")," for IPv4 and ",(0,o.kt)("inlineCode",{parentName:"p"},"tcp6")," for IPv6).\nPort 80 is used for the HTTP protocol."),(0,o.kt)("p",null,"Moving on to some user programs, Firefox runs multiple connections using both IPv4 and IPv6 and a different port for each connection."),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/firefox-tcp-udp"},"Quiz")),(0,o.kt)("p",null,"Discord does the same things as Firefox.\nIt uses TCP to send text messages, memes, videos and static content in general.\nAnd at the same time, it uses UDP to exchange voice and video data during calls."),(0,o.kt)("h3",{id:"practice-run-netstat-yourself"},"Practice: Run ",(0,o.kt)("inlineCode",{parentName:"h3"},"netstat")," Yourself"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Run ",(0,o.kt)("inlineCode",{parentName:"p"},"netstat")," on your own host machine.\nIdentify processes that use TCP and UDP and try to figure out what they might be using these protocols for.\nLook for browsers (different sites and types of content), servers such as Apache2 or Nginx, online games etc.\nDiscuss your findings with your classmates and teacher(s).")),(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Now run ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py")," and then run ",(0,o.kt)("inlineCode",{parentName:"p"},"netstat")," in another terminal to see your receiver process.\nWhat arguments do you need to pass to ",(0,o.kt)("inlineCode",{parentName:"p"},"netstat")," to see ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py"),"?\nDo you need the whole ",(0,o.kt)("inlineCode",{parentName:"p"},"-tuapn"),"?\nNo."))),(0,o.kt)("h3",{id:"conclusion"},"Conclusion"),(0,o.kt)("p",null,"The difference between TCP and UDP can be summarised as follows:"),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"TCP vs UDP",src:n(5373).Z,width:"460",height:"386"})))}u.isMDXComponent=!0},5373:(e,t,n)=>{n.d(t,{Z:()=>r});const r=n.p+"assets/images/tcp-udp-simplified-7bb728409d8a83b05d5d574df21d9ec4.png"}}]); \ No newline at end of file diff --git a/17/assets/js/95637cbd.6edcb571.js b/17/assets/js/95637cbd.6edcb571.js new file mode 100644 index 0000000000..1ff40db6f1 --- /dev/null +++ b/17/assets/js/95637cbd.6edcb571.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[842],{3905:(e,t,r)=>{r.d(t,{Zo:()=>c,kt:()=>y});var n=r(7294);function o(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function a(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(o[r]=e[r]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(o[r]=e[r])}return o}var l=n.createContext({}),s=function(e){var t=n.useContext(l),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},c=function(e){var t=s(e.components);return n.createElement(l.Provider,{value:t},e.children)},p="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,o=e.mdxType,a=e.originalType,l=e.parentName,c=u(e,["components","mdxType","originalType","parentName"]),p=s(r),f=o,y=p["".concat(l,".").concat(f)]||p[f]||d[f]||a;return r?n.createElement(y,i(i({ref:t},c),{},{components:r})):n.createElement(y,i({ref:t},c))}));function y(e,t){var r=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=r.length,i=new Array(a);i[0]=f;var u={};for(var l in t)hasOwnProperty.call(t,l)&&(u[l]=t[l]);u.originalType=e,u[p]="string"==typeof e?e:o,i[1]=u;for(var s=2;s{r.r(t),r.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>d,frontMatter:()=>a,metadata:()=>u,toc:()=>s});var n=r(7462),o=(r(7294),r(3905));const a={},i="Deluge: TCP or UDP",u={unversionedId:"Lab/IO/quiz/deluge-tcp-udp",id:"Lab/IO/quiz/deluge-tcp-udp",title:"Deluge: TCP or UDP",description:"Question Text",source:"@site/docs/Lab/IO/quiz/deluge-tcp-udp.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/deluge-tcp-udp",permalink:"/operating-systems/17/Lab/IO/quiz/deluge-tcp-udp",draft:!1,tags:[],version:"current",frontMatter:{}},l={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:s},p="wrapper";function d(e){let{components:t,...r}=e;return(0,o.kt)(p,(0,n.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"deluge-tcp-or-udp"},"Deluge: TCP or UDP"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Should Deluge use UDP or TCP to transfer torrent files?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"It should use UDP for faster file transfers")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"It should use TCP to guarantee the integrity of the transferred files")),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"Speed is nice to have.\nCorrectness is mandatory in most scenarios, including this one.\nThe only situation when correctness may be overlooked is when some given data will be quckly replaced by some other data.\nBut files are persistent.\nIf you download a video game from as a torrent (we've all done that), you want to keep it for a while and first and foremost, it has to work properly, i.e. not be corrupt."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/96a31749.b8b527d6.js b/17/assets/js/96a31749.b8b527d6.js new file mode 100644 index 0000000000..f847e07d02 --- /dev/null +++ b/17/assets/js/96a31749.b8b527d6.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[488],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>y});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var u=n.createContext({}),c=function(e){var t=n.useContext(u),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},p=function(e){var t=c(e.components);return n.createElement(u.Provider,{value:t},e.children)},s="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,u=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),s=c(r),f=a,y=s["".concat(u,".").concat(f)]||s[f]||m[f]||o;return r?n.createElement(y,i(i({ref:t},p),{},{components:r})):n.createElement(y,i({ref:t},p))}));function y(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=f;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[s]="string"==typeof e?e:a,i[1]=l;for(var c=2;c{r.r(t),r.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>l,toc:()=>c});var n=r(7462),a=(r(7294),r(3905));const o={},i="ASLR",l={unversionedId:"Lab/Data/quiz/memory-aslr",id:"Lab/Data/quiz/memory-aslr",title:"ASLR",description:"Question Text",source:"@site/docs/Lab/Data/quiz/memory-aslr.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/memory-aslr",permalink:"/operating-systems/17/Lab/Data/quiz/memory-aslr",draft:!1,tags:[],version:"current",frontMatter:{}},u={},c=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2}],p={toc:c},s="wrapper";function m(e){let{components:t,...r}=e;return(0,a.kt)(s,(0,n.Z)({},p,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"aslr"},"ASLR"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"If we enable ASLR and run the ",(0,a.kt)("inlineCode",{parentName:"p"},"bo_write_practice")," executable with the previously payload what vulnerabilities will we be able to still exploit?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"we can still jump to the ",(0,a.kt)("inlineCode",{parentName:"p"},"secret_func"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"we can both still jump to the ",(0,a.kt)("inlineCode",{parentName:"p"},"secret_func")," and overwrite the local variable")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"we can jump to a library function"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"we can still overwrite the local variable")))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/97f8cc4f.32ce1587.js b/17/assets/js/97f8cc4f.32ce1587.js new file mode 100644 index 0000000000..93cb615300 --- /dev/null +++ b/17/assets/js/97f8cc4f.32ce1587.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9125],{7039:e=>{e.exports=JSON.parse('{"title":"Lecture","slug":"/Lecture/","permalink":"/operating-systems/17/Lecture/","navigation":{"previous":{"title":"Intro","permalink":"/operating-systems/17/"},"next":{"title":"Software-Stack","permalink":"/operating-systems/17/Lecture/Software-Stack"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/97fe4cb3.df65cbc4.js b/17/assets/js/97fe4cb3.df65cbc4.js new file mode 100644 index 0000000000..d12f81bb84 --- /dev/null +++ b/17/assets/js/97fe4cb3.df65cbc4.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9995],{3905:(e,r,t)=>{t.d(r,{Zo:()=>c,kt:()=>v});var n=t(7294);function i(e,r,t){return r in e?Object.defineProperty(e,r,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[r]=t,e}function a(e,r){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);r&&(n=n.filter((function(r){return Object.getOwnPropertyDescriptor(e,r).enumerable}))),t.push.apply(t,n)}return t}function l(e){for(var r=1;r=0||(i[t]=e[t]);return i}(e,r);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(i[t]=e[t])}return i}var s=n.createContext({}),p=function(e){var r=n.useContext(s),t=r;return e&&(t="function"==typeof e?e(r):l(l({},r),e)),t},c=function(e){var r=p(e.components);return n.createElement(s.Provider,{value:r},e.children)},u="mdxType",d={inlineCode:"code",wrapper:function(e){var r=e.children;return n.createElement(n.Fragment,{},r)}},m=n.forwardRef((function(e,r){var t=e.components,i=e.mdxType,a=e.originalType,s=e.parentName,c=o(e,["components","mdxType","originalType","parentName"]),u=p(t),m=i,v=u["".concat(s,".").concat(m)]||u[m]||d[m]||a;return t?n.createElement(v,l(l({ref:r},c),{},{components:t})):n.createElement(v,l({ref:r},c))}));function v(e,r){var t=arguments,i=r&&r.mdxType;if("string"==typeof e||i){var a=t.length,l=new Array(a);l[0]=m;var o={};for(var s in r)hasOwnProperty.call(r,s)&&(o[s]=r[s]);o.originalType=e,o[u]="string"==typeof e?e:i,l[1]=o;for(var p=2;p{t.r(r),t.d(r,{assets:()=>s,contentTitle:()=>l,default:()=>d,frontMatter:()=>a,metadata:()=>o,toc:()=>p});var n=t(7462),i=(t(7294),t(3905));const a={},l="`sender.py` and `receiver.py` Client-Server Parallel",o={unversionedId:"Lab/IO/quiz/client-server-sender-receiver",id:"Lab/IO/quiz/client-server-sender-receiver",title:"`sender.py` and `receiver.py` Client-Server Parallel",description:"Question Text",source:"@site/docs/Lab/IO/quiz/client-server-sender-receiver.md",sourceDirName:"Lab/IO/quiz",slug:"/Lab/IO/quiz/client-server-sender-receiver",permalink:"/operating-systems/17/Lab/IO/quiz/client-server-sender-receiver",draft:!1,tags:[],version:"current",frontMatter:{}},s={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},u="wrapper";function d(e){let{components:r,...t}=e;return(0,i.kt)(u,(0,n.Z)({},c,t,{components:r,mdxType:"MDXLayout"}),(0,i.kt)("h1",{id:"senderpy-and-receiverpy-client-server-parallel"},(0,i.kt)("inlineCode",{parentName:"h1"},"sender.py")," and ",(0,i.kt)("inlineCode",{parentName:"h1"},"receiver.py")," Client-Server Parallel"),(0,i.kt)("h2",{id:"question-text"},"Question Text"),(0,i.kt)("p",null,"Drawing a parallel from the UDP examples in ",(0,i.kt)("inlineCode",{parentName:"p"},"support/send-receive/"),", which of the sender and receiver is similar to the server and which is similar to the client?"),(0,i.kt)("h2",{id:"question-answers"},"Question Answers"),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"both are similar to clients")),(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("p",{parentName:"li"},"both are similar to servers"))),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"receiver.py")," is similar to a server and ",(0,i.kt)("inlineCode",{parentName:"li"},"sender.py")," is similar to a client")),(0,i.kt)("ul",null,(0,i.kt)("li",{parentName:"ul"},(0,i.kt)("inlineCode",{parentName:"li"},"receiver.py")," is similar to a client and ",(0,i.kt)("inlineCode",{parentName:"li"},"sender.py")," is similar to a server")),(0,i.kt)("h2",{id:"feedback"},"Feedback"),(0,i.kt)("p",null,(0,i.kt)("inlineCode",{parentName:"p"},"receiver.py")," is the passive component.\nIt has to be started first and then waits for ",(0,i.kt)("inlineCode",{parentName:"p"},"sender.py")," (the client) to send data.\nFurthermore, you can only have ",(0,i.kt)("strong",{parentName:"p"},"one")," ",(0,i.kt)("inlineCode",{parentName:"p"},"receiver.py")," running at the same time (remember the ",(0,i.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/bind-error-cause"},"multiple ",(0,i.kt)("inlineCode",{parentName:"a"},"bind()")," bug"),") and multiple ",(0,i.kt)("inlineCode",{parentName:"p"},"sender.py"),"s."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/98c71577.9f7b031e.js b/17/assets/js/98c71577.9f7b031e.js new file mode 100644 index 0000000000..a642d84bde --- /dev/null +++ b/17/assets/js/98c71577.9f7b031e.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[675],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>d});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var l=r.createContext({}),p=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(l.Provider,{value:t},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},h=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,l=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),m=p(n),h=o,d=m["".concat(l,".").concat(h)]||m[h]||u[h]||a;return n?r.createElement(d,i(i({ref:t},c),{},{components:n})):r.createElement(d,i({ref:t},c))}));function d(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=h;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[m]="string"==typeof e?e:o,i[1]=s;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>u,frontMatter:()=>a,metadata:()=>s,toc:()=>p});var r=n(7462),o=(n(7294),n(3905));const a={},i="Zero-Copy",s={unversionedId:"Lab/IO/zero-copy",id:"Lab/IO/zero-copy",title:"Zero-Copy",description:"Imagine a server that responds with files that it stores locally.",source:"@site/docs/Lab/IO/zero-copy.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/zero-copy",permalink:"/operating-systems/17/Lab/IO/zero-copy",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"I/O Internals",permalink:"/operating-systems/17/Lab/IO/io-internals"},next:{title:"Asynchronous I/O",permalink:"/operating-systems/17/Lab/IO/async-io"}},l={},p=[{value:"Practice: Measure It",id:"practice-measure-it",level:2}],c={toc:p},m="wrapper";function u(e){let{components:t,...a}=e;return(0,o.kt)(m,(0,r.Z)({},c,a,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"zero-copy"},"Zero-Copy"),(0,o.kt)("p",null,"Imagine a server that responds with files that it stores locally.\nIts actions would be those highlighted in the image below:"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},"Receive a new request and extract the filename"),(0,o.kt)("li",{parentName:"ol"},"Read the filename from the disk into memory"),(0,o.kt)("li",{parentName:"ol"},"Send the file from memory to the client")),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"Client-Server Steps",src:n(2287).Z,width:"502",height:"752"})),(0,o.kt)("p",null,(0,o.kt)("strong",{parentName:"p"},"The quiz below is tricky, yet very important."),"\n",(0,o.kt)("strong",{parentName:"p"},"Do NOT skip it in order for this section to make sense!")),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/server-copies"},"Quiz")),(0,o.kt)("p",null,"As you might have guessed, 2 of these copies are useless.\nSince the app doesn't modify the file, there's no need for it to store the file in its own buffer.\nIt would be more efficient to use ",(0,o.kt)("strong",{parentName:"p"},"a single")," kernel buffer as intermediate storage between the disk and the NIC, as shown in the image below."),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"Server Copies - Zero-Copy",src:n(8409).Z,width:"562",height:"472"})),(0,o.kt)("p",null,'For an easier comparison with the "classic" ',(0,o.kt)("inlineCode",{parentName:"p"},"read()")," + ",(0,o.kt)("inlineCode",{parentName:"p"},"send()")," model, here's the first version again:"),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"Server Copies - Read-Send",src:n(4432).Z,width:"572",height:"652"})),(0,o.kt)("p",null,"It should be obvious that the former approach is more efficient than the latter.\nThe syscall with which we can leverage ",(0,o.kt)("strong",{parentName:"p"},"zero-copy")," is called ",(0,o.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/sendfile.2.html"},(0,o.kt)("inlineCode",{parentName:"a"},"sendfile()")),"."),(0,o.kt)("h2",{id:"practice-measure-it"},"Practice: Measure It"),(0,o.kt)("p",null,"So we have all the reasons to believe zero-copy is the faster of the two approaches we know.\nBut belief alone is meaningless.\nLet's test it!"),(0,o.kt)("p",null,"First, look at the code in ",(0,o.kt)("inlineCode",{parentName:"p"},"support/zero-copy/server.py"),".\nIt spawns 2 threads.\nOne of them listens on port 8081 and serves connections via ",(0,o.kt)("inlineCode",{parentName:"p"},"read()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"send()"),".\nThe other listens on port 8082 and serves connections via ",(0,o.kt)("inlineCode",{parentName:"p"},"sendfile()"),".\nAs you can see, the difference between them is minimal."),(0,o.kt)("p",null,"First generate the test file using the Makefile.\nThen start the server in one terminal.\nNow, in another one, use ",(0,o.kt)("inlineCode",{parentName:"p"},"benchmark_client.py")," to benchmark both implementations.\nBelow are some generic results.\nYours might differ by quite a lot, as they depend on your disk, your NIC, your kernel, your Python version, the load on your system etc."),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:/.../support/zero-copy$ python3 benchmark_client.py read-send\nTime taken: 7.175773588009179 seconds\n\nstudent@os:/.../support/zero-copy$ python3 benchmark_client.py sendfile\nTime taken: 3.71454380400246 seconds\n")),(0,o.kt)("p",null,"This is quite good!\nUsing ",(0,o.kt)("inlineCode",{parentName:"p"},"sendfile()")," halves the number of copies needed from 4 to 2.\nThus, it makes sense for the running time to ",(0,o.kt)("em",{parentName:"p"},"roughly")," halve as well."),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/fewer-than-2-copies"},"Quiz")),(0,o.kt)("p",null,"You can read a slightly more detailed article about zero-copy ",(0,o.kt)("a",{parentName:"p",href:"https://developer.ibm.com/articles/j-zerocopy/"},"here"),"."))}u.isMDXComponent=!0},2287:(e,t,n)=>{n.d(t,{Z:()=>r});const r=n.p+"assets/images/client-server-file-c21c08a102e6557188be7f080092a12c.svg"},4432:(e,t,n)=>{n.d(t,{Z:()=>r});const r=n.p+"assets/images/server-copies-normal-7e82d53d42a478d0313cb85917335f94.svg"},8409:(e,t,n)=>{n.d(t,{Z:()=>r});const r=n.p+"assets/images/server-copies-zero-copy-fc1fa1195f2444d92486d7d63dfc81a3.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/997e6ee2.97f3e7c5.js b/17/assets/js/997e6ee2.97f3e7c5.js new file mode 100644 index 0000000000..9306c77d3b --- /dev/null +++ b/17/assets/js/997e6ee2.97f3e7c5.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1329],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>h});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var u=r.createContext({}),s=function(e){var t=r.useContext(u),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=s(e.components);return r.createElement(u.Provider,{value:t},e.children)},p="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},m=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,u=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),p=s(n),m=o,h=p["".concat(u,".").concat(m)]||p[m]||d[m]||a;return n?r.createElement(h,i(i({ref:t},c),{},{components:n})):r.createElement(h,i({ref:t},c))}));function h(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=m;var l={};for(var u in t)hasOwnProperty.call(t,u)&&(l[u]=t[u]);l.originalType=e,l[p]="string"==typeof e?e:o,i[1]=l;for(var s=2;s{n.r(t),n.d(t,{assets:()=>u,contentTitle:()=>i,default:()=>d,frontMatter:()=>a,metadata:()=>l,toc:()=>s});var r=n(7462),o=(n(7294),n(3905));const a={},i="Both Condition and Mutex",l={unversionedId:"Lab/Compute/quiz/notify-only-with-mutex",id:"Lab/Compute/quiz/notify-only-with-mutex",title:"Both Condition and Mutex",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/notify-only-with-mutex.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/notify-only-with-mutex",permalink:"/operating-systems/17/Lab/Compute/quiz/notify-only-with-mutex",draft:!1,tags:[],version:"current",frontMatter:{}},u={},s=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:s},p="wrapper";function d(e){let{components:t,...n}=e;return(0,o.kt)(p,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"both-condition-and-mutex"},"Both Condition and Mutex"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Can we only use a mutex when signalling an event from one thread to another in?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"No, because this would imply that the signalling thread would ",(0,o.kt)("inlineCode",{parentName:"li"},"unlock()")," the mutex, that the signalled thread attempts to ",(0,o.kt)("inlineCode",{parentName:"li"},"lock()"),", which is an undefined behaviour")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"No, because it will result in a deadlock where the both threads will be waiting for each other")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Yes, because only one thread can modify the shared variables in order to maintain their integrity")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Yes and this would yield better performance because the threads would only wait for one object: the mutex"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"In some implementations, such as ",(0,o.kt)("a",{parentName:"p",href:"https://pubs.opengroup.org/onlinepubs/9699919799/functions/pthread_mutex_lock.html"},"POSIX threads (pthreads)"),", calling ",(0,o.kt)("inlineCode",{parentName:"p"},"unlock()")," from another thread than that which called ",(0,o.kt)("inlineCode",{parentName:"p"},"lock()")," can result in an undefined behaviour.\nFor this reason, it is unsafe to only use a mutex as a notification mechanism.\nIn addition, a mutex cannot notify more than one thread at once, if we so desire.\nMutexes are only meant to be used to isolate a critical section within the same thread."))}d.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/9d5f5dc2.939cc8c8.js b/17/assets/js/9d5f5dc2.939cc8c8.js new file mode 100644 index 0000000000..5db99c41fe --- /dev/null +++ b/17/assets/js/9d5f5dc2.939cc8c8.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8293],{7941:e=>{e.exports=JSON.parse('{"title":"Compute","slug":"/Lab/Compute/","permalink":"/operating-systems/17/Lab/Compute/","navigation":{"previous":{"title":"Arena","permalink":"/operating-systems/17/Lab/Data/arena"},"next":{"title":"Compute","permalink":"/operating-systems/17/Lab/Compute/overview"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/9dd394c3.7cfa561b.js b/17/assets/js/9dd394c3.7cfa561b.js new file mode 100644 index 0000000000..9d54507365 --- /dev/null +++ b/17/assets/js/9dd394c3.7cfa561b.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[9836],{3905:(e,t,a)=>{a.d(t,{Zo:()=>p,kt:()=>d});var r=a(7294);function n(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function s(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,r)}return a}function o(e){for(var t=1;t=0||(n[a]=e[a]);return n}(e,t);if(Object.getOwnPropertySymbols){var s=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(n[a]=e[a])}return n}var l=r.createContext({}),c=function(e){var t=r.useContext(l),a=t;return e&&(a="function"==typeof e?e(t):o(o({},t),e)),a},p=function(e){var t=c(e.components);return r.createElement(l.Provider,{value:t},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var a=e.components,n=e.mdxType,s=e.originalType,l=e.parentName,p=i(e,["components","mdxType","originalType","parentName"]),m=c(a),f=n,d=m["".concat(l,".").concat(f)]||m[f]||u[f]||s;return a?r.createElement(d,o(o({ref:t},p),{},{components:a})):r.createElement(d,o({ref:t},p))}));function d(e,t){var a=arguments,n=t&&t.mdxType;if("string"==typeof e||n){var s=a.length,o=new Array(s);o[0]=f;var i={};for(var l in t)hasOwnProperty.call(t,l)&&(i[l]=t[l]);i.originalType=e,i[m]="string"==typeof e?e:n,o[1]=i;for(var c=2;c{a.r(t),a.d(t,{assets:()=>l,contentTitle:()=>o,default:()=>u,frontMatter:()=>s,metadata:()=>i,toc:()=>c});var r=a(7462),n=(a(7294),a(3905));const s={},o="Modern Software Stacks",i={unversionedId:"Lab/Software Stack/modern-sw-stack",id:"Lab/Software Stack/modern-sw-stack",title:"Modern Software Stacks",description:"Most modern computing systems use a software stack such as the one in the figure below:",source:"@site/docs/Lab/Software Stack/modern-sw-stack.md",sourceDirName:"Lab/Software Stack",slug:"/Lab/Software Stack/modern-sw-stack",permalink:"/operating-systems/17/Lab/Software Stack/modern-sw-stack",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Software Stack",permalink:"/operating-systems/17/Lab/Software Stack/overview"},next:{title:"Analyzing the Software Stack",permalink:"/operating-systems/17/Lab/Software Stack/basic-syscall"}},l={},c=[],p={toc:c},m="wrapper";function u(e){let{components:t,...s}=e;return(0,n.kt)(m,(0,r.Z)({},p,s,{components:t,mdxType:"MDXLayout"}),(0,n.kt)("h1",{id:"modern-software-stacks"},"Modern Software Stacks"),(0,n.kt)("p",null,"Most modern computing systems use a software stack such as the one in the figure below:"),(0,n.kt)("p",null,(0,n.kt)("img",{alt:"Modern Software Stack",src:a(2148).Z,width:"542",height:"512"})),(0,n.kt)("p",null,"This modern software stack allows fast development and provides a rich set of applications to the user."),(0,n.kt)("p",null,"The basic software component is the ",(0,n.kt)("strong",{parentName:"p"},"operating system")," (OS) (technically the operating system ",(0,n.kt)("strong",{parentName:"p"},"kernel"),").\nThe OS provides the fundamental primitives to interact with hardware (read and write data) and to manage the running of applications (such as memory allocation, thread creation, scheduling).\nThese primitives form the ",(0,n.kt)("strong",{parentName:"p"},"system call API")," or ",(0,n.kt)("strong",{parentName:"p"},"system API"),".\nAn item in the system call API, i.e. the equivalent of a function call that triggers the execution of a functionality in the operating system, is a ",(0,n.kt)("strong",{parentName:"p"},"system call"),"."),(0,n.kt)("p",null,"The system call API is well-defined, stable and complete: it exposes the entire functionality of the operating system and hardware.\nHowever, it is also minimalistic with respect to features, and it provides a low-level (close to hardware) specification, making it cumbersome to use and ",(0,n.kt)("strong",{parentName:"p"},"not portable"),"."),(0,n.kt)("p",null,"Due to the downsides of the system call API, a basic library, the ",(0,n.kt)("strong",{parentName:"p"},"standard C library")," (also called ",(0,n.kt)("strong",{parentName:"p"},"libc"),"), is built on top of it.\nBecause the system call API uses an OS-specific calling convention, the standard C library typically wraps each system call into an equivalent function call, following a portable calling convention.\nMore than these wrappers, the standard C library provides its own API that is typically portable.\nPart of the API exposed by the standard C library is the ",(0,n.kt)("strong",{parentName:"p"},"standard C API"),", also called ",(0,n.kt)("strong",{parentName:"p"},"ANSI C")," or ",(0,n.kt)("strong",{parentName:"p"},"ISO C"),";\nthis API is typically portable across all platforms (operating systems and hardware).\nThis API, going beyond system call wrappers, has several advantages:"),(0,n.kt)("ul",null,(0,n.kt)("li",{parentName:"ul"},"portability: irrespective of the underlying operating system (and system call API), the API is the same"),(0,n.kt)("li",{parentName:"ul"},"extensive features: string management, I/O formatting"),(0,n.kt)("li",{parentName:"ul"},"possibility of increased efficiency with techniques such as buffering, as we show later")))}u.isMDXComponent=!0},2148:(e,t,a)=>{a.d(t,{Z:()=>r});const r=a.p+"assets/images/modern-sw-stack-4a2427d07a59c3a6599305b8eedc43dd.svg"}}]); \ No newline at end of file diff --git a/17/assets/js/9ef964a6.e5708931.js b/17/assets/js/9ef964a6.e5708931.js new file mode 100644 index 0000000000..3f03d4a9ae --- /dev/null +++ b/17/assets/js/9ef964a6.e5708931.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4772],{3737:s=>{s.exports=JSON.parse('{"title":"Assignments","slug":"/Assignments/","permalink":"/operating-systems/17/Assignments/","navigation":{"previous":{"title":"Arena","permalink":"/operating-systems/17/Lab/Application Interaction/arena"},"next":{"title":"Mini-libc","permalink":"/operating-systems/17/Assignments/Mini Libc/"}}}')}}]); \ No newline at end of file diff --git a/17/assets/js/9f99d3fc.aa170994.js b/17/assets/js/9f99d3fc.aa170994.js new file mode 100644 index 0000000000..dd7ec6d715 --- /dev/null +++ b/17/assets/js/9f99d3fc.aa170994.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[8788],{3905:(e,t,r)=>{r.d(t,{Zo:()=>s,kt:()=>d});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var p=n.createContext({}),u=function(e){var t=n.useContext(p),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},s=function(e){var t=u(e.components);return n.createElement(p.Provider,{value:t},e.children)},c="mdxType",f={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},m=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,p=e.parentName,s=l(e,["components","mdxType","originalType","parentName"]),c=u(r),m=a,d=c["".concat(p,".").concat(m)]||c[m]||f[m]||o;return r?n.createElement(d,i(i({ref:t},s),{},{components:r})):n.createElement(d,i({ref:t},s))}));function d(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=m;var l={};for(var p in t)hasOwnProperty.call(t,p)&&(l[p]=t[p]);l.originalType=e,l[c]="string"==typeof e?e:a,i[1]=l;for(var u=2;u{r.r(t),r.d(t,{assets:()=>p,contentTitle:()=>i,default:()=>f,frontMatter:()=>o,metadata:()=>l,toc:()=>u});var n=r(7462),a=(r(7294),r(3905));const o={},i="Parent Faults before `fork()`",l={unversionedId:"Lab/Compute/quiz/parent-faults-before-fork",id:"Lab/Compute/quiz/parent-faults-before-fork",title:"Parent Faults before `fork()`",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/parent-faults-before-fork.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/parent-faults-before-fork",permalink:"/operating-systems/17/Lab/Compute/quiz/parent-faults-before-fork",draft:!1,tags:[],version:"current",frontMatter:{}},p={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2}],s={toc:u},c="wrapper";function f(e){let{components:t,...r}=e;return(0,a.kt)(c,(0,n.Z)({},s,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"parent-faults-before-fork"},"Parent Faults before ",(0,a.kt)("inlineCode",{parentName:"h1"},"fork()")),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What causes the page faults that occur between the first and second steps?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Calling ",(0,a.kt)("inlineCode",{parentName:"li"},"fork()")," duplicates the pages previously allocated by the parent")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Demand paging makes the pages in the ",(0,a.kt)("inlineCode",{parentName:"li"},"p")," array to be mapped to frames only when written")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"The OS duplicates the parent's pages in preparation for ",(0,a.kt)("inlineCode",{parentName:"p"},"fork()"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"mmap()")," sets the pages to be mapped at a later time, decided by the OS"))))}f.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/a0ac5395.a2bcd064.js b/17/assets/js/a0ac5395.a2bcd064.js new file mode 100644 index 0000000000..ac04d61cc8 --- /dev/null +++ b/17/assets/js/a0ac5395.a2bcd064.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3788],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>f});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var l=r.createContext({}),p=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},u=function(e){var t=p(e.components);return r.createElement(l.Provider,{value:t},e.children)},s="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,l=e.parentName,u=c(e,["components","mdxType","originalType","parentName"]),s=p(n),d=a,f=s["".concat(l,".").concat(d)]||s[d]||m[d]||o;return n?r.createElement(f,i(i({ref:t},u),{},{components:n})):r.createElement(f,i({ref:t},u))}));function f(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,i=new Array(o);i[0]=d;var c={};for(var l in t)hasOwnProperty.call(t,l)&&(c[l]=t[l]);c.originalType=e,c[s]="string"==typeof e?e:a,i[1]=c;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>c,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const o={},i="`apache2` Document Root",c={unversionedId:"Lab/Compute/quiz/apache2-strace",id:"Lab/Compute/quiz/apache2-strace",title:"`apache2` Document Root",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/apache2-strace.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/apache2-strace",permalink:"/operating-systems/17/Lab/Compute/quiz/apache2-strace",draft:!1,tags:[],version:"current",frontMatter:{}},l={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:p},s="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(s,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"apache2-document-root"},(0,a.kt)("inlineCode",{parentName:"h1"},"apache2")," Document Root"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"What is the document root of the ",(0,a.kt)("inlineCode",{parentName:"p"},"apache2")," server?"),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"/etc/apache2"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("inlineCode",{parentName:"li"},"/usr/local/apache2/htdocs/"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"/var/www/html"))),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},(0,a.kt)("inlineCode",{parentName:"p"},"/var/www/apache2/htdocs")))),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"In ",(0,a.kt)("inlineCode",{parentName:"p"},"strace")," we see that the server opens the file ",(0,a.kt)("inlineCode",{parentName:"p"},"/usr/local/apache2/htdocs/index.html"),".\nThis means that the document root is ",(0,a.kt)("inlineCode",{parentName:"p"},"/usr/local/apache2/htdocs/"),"."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/a191e07d.8d34275d.js b/17/assets/js/a191e07d.8d34275d.js new file mode 100644 index 0000000000..05439d95c7 --- /dev/null +++ b/17/assets/js/a191e07d.8d34275d.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[7841],{3905:(e,t,r)=>{r.d(t,{Zo:()=>p,kt:()=>m});var n=r(7294);function a(e,t,r){return t in e?Object.defineProperty(e,t,{value:r,enumerable:!0,configurable:!0,writable:!0}):e[t]=r,e}function o(e,t){var r=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),r.push.apply(r,n)}return r}function i(e){for(var t=1;t=0||(a[r]=e[r]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,r)&&(a[r]=e[r])}return a}var c=n.createContext({}),l=function(e){var t=n.useContext(c),r=t;return e&&(r="function"==typeof e?e(t):i(i({},t),e)),r},p=function(e){var t=l(e.components);return n.createElement(c.Provider,{value:t},e.children)},u="mdxType",y={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},f=n.forwardRef((function(e,t){var r=e.components,a=e.mdxType,o=e.originalType,c=e.parentName,p=s(e,["components","mdxType","originalType","parentName"]),u=l(r),f=a,m=u["".concat(c,".").concat(f)]||u[f]||y[f]||o;return r?n.createElement(m,i(i({ref:t},p),{},{components:r})):n.createElement(m,i({ref:t},p))}));function m(e,t){var r=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=r.length,i=new Array(o);i[0]=f;var s={};for(var c in t)hasOwnProperty.call(t,c)&&(s[c]=t[c]);s.originalType=e,s[u]="string"==typeof e?e:a,i[1]=s;for(var l=2;l{r.r(t),r.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>y,frontMatter:()=>o,metadata:()=>s,toc:()=>l});var n=r(7462),a=(r(7294),r(3905));const o={},i="Bypass Canary",s={unversionedId:"Lab/Data/quiz/bypass-canary",id:"Lab/Data/quiz/bypass-canary",title:"Bypass Canary",description:"Question",source:"@site/docs/Lab/Data/quiz/bypass-canary.md",sourceDirName:"Lab/Data/quiz",slug:"/Lab/Data/quiz/bypass-canary",permalink:"/operating-systems/17/Lab/Data/quiz/bypass-canary",draft:!1,tags:[],version:"current",frontMatter:{}},c={},l=[{value:"Question",id:"question",level:2}],p={toc:l},u="wrapper";function y(e){let{components:t,...r}=e;return(0,a.kt)(u,(0,n.Z)({},p,r,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"bypass-canary"},"Bypass Canary"),(0,a.kt)("h2",{id:"question"},"Question"),(0,a.kt)("p",null,"If we enable ASLR, can we still exploit the ",(0,a.kt)("inlineCode",{parentName:"p"},"stack_protector")," program?"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"no, because the address of ",(0,a.kt)("inlineCode",{parentName:"p"},"pawned")," is going to be different for every run.")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"yes, because ASLR cannot work well when the canary is activated."))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"yes, because ASLR randomizes the start address of a section, but the offsets remain the same.")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"no, because the address to which ",(0,a.kt)("inlineCode",{parentName:"li"},"addr")," points to is going to be random")))}y.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/a210c167.3981e880.js b/17/assets/js/a210c167.3981e880.js new file mode 100644 index 0000000000..1685b9e7cb --- /dev/null +++ b/17/assets/js/a210c167.3981e880.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4471],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>m});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function s(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var i=a.createContext({}),p=function(e){var t=a.useContext(i),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(i.Provider,{value:t},e.children)},d="mdxType",u={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},h=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,i=e.parentName,c=l(e,["components","mdxType","originalType","parentName"]),d=p(n),h=r,m=d["".concat(i,".").concat(h)]||d[h]||u[h]||o;return n?a.createElement(m,s(s({ref:t},c),{},{components:n})):a.createElement(m,s({ref:t},c))}));function m(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,s=new Array(o);s[0]=h;var l={};for(var i in t)hasOwnProperty.call(t,i)&&(l[i]=t[i]);l.originalType=e,l[d]="string"==typeof e?e:r,s[1]=l;for(var p=2;p{n.r(t),n.d(t,{assets:()=>i,contentTitle:()=>s,default:()=>u,frontMatter:()=>o,metadata:()=>l,toc:()=>p});var a=n(7462),r=(n(7294),n(3905));const o={},s="HTTP Parser",l={unversionedId:"Assignments/Asynchronous Web Server/src/http-parser/README",id:"Assignments/Asynchronous Web Server/src/http-parser/README",title:"HTTP Parser",description:"This is a parser for HTTP messages written in C. It parses both requests and",source:"@site/docs/Assignments/Asynchronous Web Server/src/http-parser/README.md",sourceDirName:"Assignments/Asynchronous Web Server/src/http-parser",slug:"/Assignments/Asynchronous Web Server/src/http-parser/",permalink:"/operating-systems/17/Assignments/Asynchronous Web Server/src/http-parser/",draft:!1,tags:[],version:"current",frontMatter:{}},i={},p=[{value:"Usage",id:"usage",level:2},{value:"The Special Problem of Upgrade",id:"the-special-problem-of-upgrade",level:2},{value:"Callbacks",id:"callbacks",level:2}],c={toc:p},d="wrapper";function u(e){let{components:t,...n}=e;return(0,r.kt)(d,(0,a.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"http-parser"},"HTTP Parser"),(0,r.kt)("p",null,"This is a parser for HTTP messages written in C. It parses both requests and\nresponses. The parser is designed to be used in performance HTTP\napplications. It does not make any syscalls nor allocations, it does not\nbuffer data, it can be interrupted at anytime. Depending on your\narchitecture, it only requires about 40 bytes of data per message\nstream (in a web server that is per connection)."),(0,r.kt)("p",null,"Features:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"No dependencies"),(0,r.kt)("li",{parentName:"ul"},"Handles persistent streams (keep-alive)."),(0,r.kt)("li",{parentName:"ul"},"Decodes chunked encoding."),(0,r.kt)("li",{parentName:"ul"},"Upgrade support"),(0,r.kt)("li",{parentName:"ul"},"Defends against buffer overflow attacks.")),(0,r.kt)("p",null,"The parser extracts the following information from HTTP messages:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"Header fields and values"),(0,r.kt)("li",{parentName:"ul"},"Content-Length"),(0,r.kt)("li",{parentName:"ul"},"Request method"),(0,r.kt)("li",{parentName:"ul"},"Response status code"),(0,r.kt)("li",{parentName:"ul"},"Transfer-Encoding"),(0,r.kt)("li",{parentName:"ul"},"HTTP version"),(0,r.kt)("li",{parentName:"ul"},"Request path, query string, fragment"),(0,r.kt)("li",{parentName:"ul"},"Message body")),(0,r.kt)("h2",{id:"usage"},"Usage"),(0,r.kt)("p",null,"One ",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser")," object is used per TCP connection. Initialize the struct\nusing ",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser_init()")," and set the callbacks. That might look something\nlike this for a request parser:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre"},"http_parser_settings settings;\nsettings.on_path = my_path_callback;\nsettings.on_header_field = my_header_field_callback;\n/* ... */\n\nhttp_parser *parser = malloc(sizeof(http_parser));\nhttp_parser_init(parser, HTTP_REQUEST);\nparser->data = my_socket;\n")),(0,r.kt)("p",null,"When data is received on the socket execute the parser and check for errors."),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre"},"size_t len = 80*1024, nparsed;\nchar buf[len];\nssize_t recved;\n\nrecved = recv(fd, buf, len, 0);\n\nif (recved < 0) {\n /* Handle error. */\n}\n\n/* Start up / continue the parser.\n * Note we pass recved==0 to signal that EOF has been recieved.\n */\nnparsed = http_parser_execute(parser, &settings, buf, recved);\n\nif (parser->upgrade) {\n /* handle new protocol */\n} else if (nparsed != recved) {\n /* Handle error. Usually just close the connection. */\n}\n")),(0,r.kt)("p",null,"HTTP needs to know where the end of the stream is. For example, sometimes\nservers send responses without Content-Length and expect the client to\nconsume input (for the body) until EOF. To tell http_parser about EOF, give\n",(0,r.kt)("inlineCode",{parentName:"p"},"0")," as the forth parameter to ",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser_execute()"),". Callbacks and errors\ncan still be encountered during an EOF, so one must still be prepared\nto receive them."),(0,r.kt)("p",null,"Scalar valued message information such as ",(0,r.kt)("inlineCode",{parentName:"p"},"status_code"),", ",(0,r.kt)("inlineCode",{parentName:"p"},"method"),", and the\nHTTP version are stored in the parser structure. This data is only\ntemporally stored in ",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser")," and gets reset on each new message. If\nthis information is needed later, copy it out of the structure during the\n",(0,r.kt)("inlineCode",{parentName:"p"},"headers_complete")," callback."),(0,r.kt)("p",null,"The parser decodes the transfer-encoding for both requests and responses\ntransparently. That is, a chunked encoding is decoded before being sent to\nthe on_body callback."),(0,r.kt)("h2",{id:"the-special-problem-of-upgrade"},"The Special Problem of Upgrade"),(0,r.kt)("p",null,"HTTP supports upgrading the connection to a different protocol. An\nincreasingly common example of this is the Web Socket protocol which sends\na request like"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre"}," GET /demo HTTP/1.1\n Upgrade: WebSocket\n Connection: Upgrade\n Host: example.com\n Origin: http://example.com\n WebSocket-Protocol: sample\n")),(0,r.kt)("p",null,"followed by non-HTTP data."),(0,r.kt)("p",null,"(See ",(0,r.kt)("a",{parentName:"p",href:"http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75"},"http://tools.ietf.org/html/draft-hixie-thewebsocketprotocol-75")," for more\ninformation the Web Socket protocol.)"),(0,r.kt)("p",null,"To support this, the parser will treat this as a normal HTTP message without a\nbody. Issuing both on_headers_complete and on_message_complete callbacks. However\nhttp_parser_execute() will stop parsing at the end of the headers and return."),(0,r.kt)("p",null,"The user is expected to check if ",(0,r.kt)("inlineCode",{parentName:"p"},"parser->upgrade")," has been set to 1 after\n",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser_execute()")," returns. Non-HTTP data begins at the buffer supplied\noffset by the return value of ",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser_execute()"),"."),(0,r.kt)("h2",{id:"callbacks"},"Callbacks"),(0,r.kt)("p",null,"During the ",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser_execute()")," call, the callbacks set in\n",(0,r.kt)("inlineCode",{parentName:"p"},"http_parser_settings")," will be executed. The parser maintains state and\nnever looks behind, so buffering the data is not necessary. If you need to\nsave certain data for later usage, you can do that from the callbacks."),(0,r.kt)("p",null,"There are two types of callbacks:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"notification ",(0,r.kt)("inlineCode",{parentName:"li"},"typedef int (*http_cb) (http_parser*);"),"\nCallbacks: on_message_begin, on_headers_complete, on_message_complete."),(0,r.kt)("li",{parentName:"ul"},"data ",(0,r.kt)("inlineCode",{parentName:"li"},"typedef int (*http_data_cb) (http_parser*, const char *at, size_t length);"),"\nCallbacks: (requests only) on_path, on_query_string, on_uri, on_fragment,\n(common) on_header_field, on_header_value, on_body;")),(0,r.kt)("p",null,"Callbacks must return 0 on success. Returning a non-zero value indicates\nerror to the parser, making it exit immediately."),(0,r.kt)("p",null,"In case you parse HTTP message in chunks (i.e. ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," request line\nfrom socket, parse, read half headers, parse, etc) your data callbacks\nmay be called more than once. Http-parser guarantees that data pointer is only\nvalid for the lifetime of callback. You can also ",(0,r.kt)("inlineCode",{parentName:"p"},"read()")," into a heap allocated\nbuffer to avoid copying memory around if this fits your application."),(0,r.kt)("p",null,"Reading headers may be a tricky task if you read/parse headers partially.\nBasically, you need to remember whether last header callback was field or value\nand apply following logic:"),(0,r.kt)("pre",null,(0,r.kt)("code",{parentName:"pre"},"(on_header_field and on_header_value shortened to on_h_*)\n ------------------------ ------------ --------------------------------------------\n| State (prev. callback) | Callback | Description/action |\n ------------------------ ------------ --------------------------------------------\n| nothing (first call) | on_h_field | Allocate new buffer and copy callback data |\n| | | into it |\n ------------------------ ------------ --------------------------------------------\n| value | on_h_field | New header started. |\n| | | Copy current name,value buffers to headers |\n| | | list and allocate new buffer for new name |\n ------------------------ ------------ --------------------------------------------\n| field | on_h_field | Previous name continues. Reallocate name |\n| | | buffer and append callback data to it |\n ------------------------ ------------ --------------------------------------------\n| field | on_h_value | Value for current header started. Allocate |\n| | | new buffer and copy callback data to it |\n ------------------------ ------------ --------------------------------------------\n| value | on_h_value | Value continues. Reallocate value buffer |\n| | | and append callback data to it |\n ------------------------ ------------ --------------------------------------------\n")),(0,r.kt)("p",null,"See examples of reading in headers:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("a",{parentName:"li",href:"http://gist.github.com/155877"},"partial example")," in C"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("a",{parentName:"li",href:"http://github.com/ry/http-parser/blob/37a0ff8928fb0d83cec0d0d8909c5a4abcd221af/test.c#L403"},"from http-parser tests")," in C"),(0,r.kt)("li",{parentName:"ul"},(0,r.kt)("a",{parentName:"li",href:"http://github.com/ry/node/blob/842eaf446d2fdcb33b296c67c911c32a0dabc747/src/http.js#L284"},"from Node library")," in Javascript")))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/a315d4d8.4163140d.js b/17/assets/js/a315d4d8.4163140d.js new file mode 100644 index 0000000000..444aa8ad81 --- /dev/null +++ b/17/assets/js/a315d4d8.4163140d.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3425],{3905:(e,t,n)=>{n.d(t,{Zo:()=>l,kt:()=>v});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function a(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var c=r.createContext({}),s=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},l=function(e){var t=s(e.components);return r.createElement(c.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,i=e.originalType,c=e.parentName,l=p(e,["components","mdxType","originalType","parentName"]),u=s(n),d=o,v=u["".concat(c,".").concat(d)]||u[d]||m[d]||i;return n?r.createElement(v,a(a({ref:t},l),{},{components:n})):r.createElement(v,a({ref:t},l))}));function v(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=n.length,a=new Array(i);a[0]=d;var p={};for(var c in t)hasOwnProperty.call(t,c)&&(p[c]=t[c]);p.originalType=e,p[u]="string"==typeof e?e:o,a[1]=p;for(var s=2;s{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>a,default:()=>m,frontMatter:()=>i,metadata:()=>p,toc:()=>s});var r=n(7462),o=(n(7294),n(3905));const i={},a="Time Server",p={unversionedId:"Lab/Application Interaction/time-server",id:"Lab/Application Interaction/time-server",title:"Time Server",description:"Check out the code in support/time-server/server.c and support/time-server/client.c.",source:"@site/docs/Lab/Application Interaction/time-server.md",sourceDirName:"Lab/Application Interaction",slug:"/Lab/Application Interaction/time-server",permalink:"/operating-systems/17/Lab/Application Interaction/time-server",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Application Interaction",permalink:"/operating-systems/17/Lab/Application Interaction/overview"},next:{title:"Password Cracker",permalink:"/operating-systems/17/Lab/Application Interaction/password-cracker"}},c={},s=[{value:"Python Version",id:"python-version",level:2},{value:"Practice",id:"practice",level:3}],l={toc:s},u="wrapper";function m(e){let{components:t,...n}=e;return(0,o.kt)(u,(0,r.Z)({},l,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"time-server"},"Time Server"),(0,o.kt)("p",null,"Check out the code in ",(0,o.kt)("inlineCode",{parentName:"p"},"support/time-server/server.c")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"support/time-server/client.c"),"."),(0,o.kt)("p",null,"This is a simple program consisting of a server and a client.\nThe server uses a tcp socket to wait for connections.\nOnce a client has connected, the server will send the current time to it.\nThe client will then print the received time to the console."),(0,o.kt)("p",null,"Let's build and run this example:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/time-server$ make\ngcc -Wall -o server server.c\ngcc -Wall -o client client.c\nstudent@os:~/.../support/time-server$ ./server\n")),(0,o.kt)("p",null,"Then, in another terminal:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/time-server$ ./client 127.0.0.1 2000\nThe time is Thu Sep 1 11:48:03 2022\n")),(0,o.kt)("h2",{id:"python-version"},"Python Version"),(0,o.kt)("p",null,"In ",(0,o.kt)("inlineCode",{parentName:"p"},"support/time-server/python")," we have the equivalent python implementation for both the server and client:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/time-server/python$ python3 server.py\n")),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../support/time-server/python$ python3 client.py 127.0.0.1 2000\nThe time is Thu Sep 1 11:58:01 2022\n")),(0,o.kt)("h3",{id:"practice"},"Practice"),(0,o.kt)("p",null,"Try to figure out the protocol used by the server and the client.\nYou can do this by reading the source code, corroborated with information obtained at runtime."),(0,o.kt)("p",null,"Run the server again (the version in C), but instead of running the client, let's run ",(0,o.kt)("inlineCode",{parentName:"p"},"netcat")," and pipe the output to ",(0,o.kt)("inlineCode",{parentName:"p"},"hexdump"),":"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"nc -d 127.0.0.1 2000 | hexdump -C\n")),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Application%20Interaction/quiz/time-server"},"Quiz")),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Application%20Interaction/quiz/time-server-interop"},"Quiz")))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/a7bc80c4.d8275609.js b/17/assets/js/a7bc80c4.d8275609.js new file mode 100644 index 0000000000..50922a1362 --- /dev/null +++ b/17/assets/js/a7bc80c4.d8275609.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3320],{3905:(e,n,t)=>{t.d(n,{Zo:()=>p,kt:()=>d});var o=t(7294);function a(e,n,t){return n in e?Object.defineProperty(e,n,{value:t,enumerable:!0,configurable:!0,writable:!0}):e[n]=t,e}function r(e,n){var t=Object.keys(e);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);n&&(o=o.filter((function(n){return Object.getOwnPropertyDescriptor(e,n).enumerable}))),t.push.apply(t,o)}return t}function i(e){for(var n=1;n=0||(a[t]=e[t]);return a}(e,n);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(o=0;o=0||Object.prototype.propertyIsEnumerable.call(e,t)&&(a[t]=e[t])}return a}var s=o.createContext({}),c=function(e){var n=o.useContext(s),t=n;return e&&(t="function"==typeof e?e(n):i(i({},n),e)),t},p=function(e){var n=c(e.components);return o.createElement(s.Provider,{value:n},e.children)},m="mdxType",u={inlineCode:"code",wrapper:function(e){var n=e.children;return o.createElement(o.Fragment,{},n)}},h=o.forwardRef((function(e,n){var t=e.components,a=e.mdxType,r=e.originalType,s=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),m=c(t),h=a,d=m["".concat(s,".").concat(h)]||m[h]||u[h]||r;return t?o.createElement(d,i(i({ref:n},p),{},{components:t})):o.createElement(d,i({ref:n},p))}));function d(e,n){var t=arguments,a=n&&n.mdxType;if("string"==typeof e||a){var r=t.length,i=new Array(r);i[0]=h;var l={};for(var s in n)hasOwnProperty.call(n,s)&&(l[s]=n[s]);l.originalType=e,l[m]="string"==typeof e?e:a,i[1]=l;for(var c=2;c{t.r(n),t.d(n,{assets:()=>s,contentTitle:()=>i,default:()=>u,frontMatter:()=>r,metadata:()=>l,toc:()=>c});var o=t(7462),a=(t(7294),t(3905));const r={},i="I/O Multiplexing",l={unversionedId:"Lab/IO/io-multiplexing",id:"Lab/IO/io-multiplexing",title:"I/O Multiplexing",description:"I/O multiplexing is the ability to serve multiple I/O channels (or anything that can be referenced via a file descriptor / handle) simultaneously.",source:"@site/docs/Lab/IO/io-multiplexing.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/io-multiplexing",permalink:"/operating-systems/17/Lab/IO/io-multiplexing",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Asynchronous I/O",permalink:"/operating-systems/17/Lab/IO/async-io"},next:{title:"Arena",permalink:"/operating-systems/17/Lab/IO/arena"}},s={},c=[{value:"Practice",id:"practice",level:2}],p={toc:c},m="wrapper";function u(e){let{components:n,...t}=e;return(0,a.kt)(m,(0,o.Z)({},p,t,{components:n,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"io-multiplexing"},"I/O Multiplexing"),(0,a.kt)("p",null,"I/O multiplexing is the ability to serve multiple I/O channels (or anything that can be referenced via a file descriptor / handle) simultaneously.\nIf a given application, such a server, has multiple sockets on which it serves connection, it may be the case that operating on one socket blocks the server.\nOne solution is using asynchronous operations, with different backends.\nThe other solution is using I/O multiplexing."),(0,a.kt)("p",null,"With I/O multiplexing, the API provides a structure / an array to list all used channels.\nAnd then it provides an API to operate on that channel.\nAnd a (blocking) function to retrieve the channel that is ready for interaction (or channels).\nSo instead of waiting for a given channel (while others may be ready), you now simultaneously wait on all channels, and the ready channel is returned by the function."),(0,a.kt)("p",null,"The classical functions for I/O multiplexing are ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/select.2.html"},(0,a.kt)("inlineCode",{parentName:"a"},"select"))," and ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/poll.2.html"},(0,a.kt)("inlineCode",{parentName:"a"},"poll")),".\nDue to several limitations, modern operating systems provide advanced (non-portable) variants to these:"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Windows provides ",(0,a.kt)("a",{parentName:"li",href:"https://learn.microsoft.com/en-us/windows/win32/fileio/i-o-completion-ports"},"I/O completion ports (IOCP)"),"."),(0,a.kt)("li",{parentName:"ul"},"BSD provides ",(0,a.kt)("a",{parentName:"li",href:"https://www.freebsd.org/cgi/man.cgi?kqueue"},(0,a.kt)("inlineCode",{parentName:"a"},"kqueue")),"."),(0,a.kt)("li",{parentName:"ul"},"Linux provides ",(0,a.kt)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man7/epoll.7.html"},(0,a.kt)("inlineCode",{parentName:"a"},"epoll()")),".")),(0,a.kt)("p",null,(0,a.kt)("strong",{parentName:"p"},"Note")," that I/O multiplexing is orthogonal to asynchronous operations.\nYou could tie them together if the completion of the asynchronous operation sends a notification that can be handled via a file descriptor / handle.\nThis is the case with Windows asynchronous I/O (called ",(0,a.kt)("a",{parentName:"p",href:"https://learn.microsoft.com/en-us/windows/win32/fileio/synchronous-and-asynchronous-i-o"},"overlapped I/O"),")."),(0,a.kt)("h2",{id:"practice"},"Practice"),(0,a.kt)("p",null,"Enter the ",(0,a.kt)("inlineCode",{parentName:"p"},"multiplex/")," directory.\nSee the implementation of an ",(0,a.kt)("inlineCode",{parentName:"p"},"epoll()"),"-based server in ",(0,a.kt)("inlineCode",{parentName:"p"},"epoll_echo_server.c"),"."),(0,a.kt)("ol",null,(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Build the server:"),(0,a.kt)("pre",{parentName:"li"},(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/multiplex$ make\ngcc -g -Wall -Wextra -I../../../../../common/makefile/.. -c -o epoll_echo_server.o epoll_echo_server.c\ngcc -g -Wall -Wextra -I../../../../../common/makefile/.. -c -o ../../../../../common/makefile/../utils/log/log.o ../../../../../common/makefile/../utils/log/log.c\ngcc -g -Wall -Wextra -I../../../../../common/makefile/.. -c -o ../../../../../common/utils/sock/sock_util.o ../../../../../common/utils/sock/sock_util.c\ngcc epoll_echo_server.o ../../../../../common/makefile/../utils/log/log.o ../../../../../common/utils/sock/sock_util.o -o epoll_echo_server\n")),(0,a.kt)("p",{parentName:"li"},"And run it:"),(0,a.kt)("pre",{parentName:"li"},(0,a.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../lab/support/multiplex$ ./epoll_echo_server\n08:41:07 INFO epoll_echo_server.c:252: Server waiting for connections on port 42424\n")),(0,a.kt)("p",{parentName:"li"},"It listens for connection on port ",(0,a.kt)("inlineCode",{parentName:"p"},"42424"),"."),(0,a.kt)("p",{parentName:"li"},"Connect using ",(0,a.kt)("inlineCode",{parentName:"p"},"netcat")," on another console and send messages:"),(0,a.kt)("pre",{parentName:"li"},(0,a.kt)("code",{parentName:"pre",className:"language-console"},"$ nc localhost 42424\naaa\naaa\nbbb\nbbb\n")),(0,a.kt)("p",{parentName:"li"},"Keep the connection open and, on the third console, initiate another ",(0,a.kt)("inlineCode",{parentName:"p"},"netcat")," connection.\nThe server is now multiplexing both connections.")),(0,a.kt)("li",{parentName:"ol"},(0,a.kt)("p",{parentName:"li"},"Create a script and / or a program to exercise the server.\nCreate many connections to the server and continuously send messages to the server.\nSee it multiplex the I/O channels (one for each connection - actually two: one for receiving and one for sending)."))))}u.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/a8d01eab.86e2b472.js b/17/assets/js/a8d01eab.86e2b472.js new file mode 100644 index 0000000000..7a3839f985 --- /dev/null +++ b/17/assets/js/a8d01eab.86e2b472.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3711],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>d});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function i(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function a(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var i=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var c=r.createContext({}),u=function(e){var t=r.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):a(a({},t),e)),n},p=function(e){var t=u(e.components);return r.createElement(c.Provider,{value:t},e.children)},s="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,i=e.originalType,c=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),s=u(n),f=o,d=s["".concat(c,".").concat(f)]||s[f]||m[f]||i;return n?r.createElement(d,a(a({ref:t},p),{},{components:n})):r.createElement(d,a({ref:t},p))}));function d(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var i=n.length,a=new Array(i);a[0]=f;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[s]="string"==typeof e?e:o,a[1]=l;for(var u=2;u{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>a,default:()=>m,frontMatter:()=>i,metadata:()=>l,toc:()=>u});var r=n(7462),o=(n(7294),n(3905));const i={},a="Oneko Timer",l={unversionedId:"Lab/Application Interaction/quiz/timer",id:"Lab/Application Interaction/quiz/timer",title:"Oneko Timer",description:"Question Text",source:"@site/docs/Lab/Application Interaction/quiz/timer.md",sourceDirName:"Lab/Application Interaction/quiz",slug:"/Lab/Application Interaction/quiz/timer",permalink:"/operating-systems/17/Lab/Application Interaction/quiz/timer",draft:!1,tags:[],version:"current",frontMatter:{}},c={},u=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],p={toc:u},s="wrapper";function m(e){let{components:t,...n}=e;return(0,o.kt)(s,(0,r.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"oneko-timer"},"Oneko Timer"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"What is the purpose of the timer used in the Oneko's implementantion"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"Slow down the cat")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Speed up the cat")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"It server no purpose"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"The purpose of the timer is to delay the cat. You can verify this by running"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"while true; do kill -14 $(pidof oneko)\n")),(0,o.kt)("p",null,"You'll notice the cat is much faster in following your mouse."),(0,o.kt)("p",null,'Try and find other ways to "hack" Oneko to make it move faster.'))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/ab4aa734.6a48b1ad.js b/17/assets/js/ab4aa734.6a48b1ad.js new file mode 100644 index 0000000000..468c962134 --- /dev/null +++ b/17/assets/js/ab4aa734.6a48b1ad.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[3851],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>u});var a=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function r(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var l=a.createContext({}),p=function(e){var t=a.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},c=function(e){var t=p(e.components);return a.createElement(l.Provider,{value:t},e.children)},h="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},m=a.forwardRef((function(e,t){var n=e.components,o=e.mdxType,r=e.originalType,l=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),h=p(n),m=o,u=h["".concat(l,".").concat(m)]||h[m]||d[m]||r;return n?a.createElement(u,i(i({ref:t},c),{},{components:n})):a.createElement(u,i({ref:t},c))}));function u(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var r=n.length,i=new Array(r);i[0]=m;var s={};for(var l in t)hasOwnProperty.call(t,l)&&(s[l]=t[l]);s.originalType=e,s[h]="string"==typeof e?e:o,i[1]=s;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>i,default:()=>d,frontMatter:()=>r,metadata:()=>s,toc:()=>p});var a=n(7462),o=(n(7294),n(3905));const r={},i="Remote I/O",s={unversionedId:"Lab/IO/remote-io",id:"Lab/IO/remote-io",title:"Remote I/O",description:"In the previous sections, we started looking into how applications interact with the outside world.",source:"@site/docs/Lab/IO/remote-io.md",sourceDirName:"Lab/IO",slug:"/Lab/IO/remote-io",permalink:"/operating-systems/17/Lab/IO/remote-io",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Local I/O in Action",permalink:"/operating-systems/17/Lab/IO/local-io-in-action"},next:{title:"Networking 101",permalink:"/operating-systems/17/Lab/IO/networking-101"}},l={},p=[{value:"One Browser - Many Connections",id:"one-browser---many-connections",level:2},{value:"Connection",id:"connection",level:2},{value:"Recap: IPs",id:"recap-ips",level:3},{value:"Further than IPs: Ports",id:"further-than-ips-ports",level:3},{value:"API - Hail Berkeley Sockets",id:"api---hail-berkeley-sockets",level:2},{value:"Sender and Receiver",id:"sender-and-receiver",level:3},{value:"Practice",id:"practice",level:3}],c={toc:p},h="wrapper";function d(e){let{components:t,...r}=e;return(0,o.kt)(h,(0,a.Z)({},c,r,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"remote-io"},"Remote I/O"),(0,o.kt)("p",null,'In the previous sections, we started looking into how applications interact with the outside world.\nHowever, so far this "outside world" has only meant ',(0,o.kt)("strong",{parentName:"p"},"local")," files and other ",(0,o.kt)("strong",{parentName:"p"},"local"),' processes.\nBut what about files located on other computers?\nWhat about "talking to" processes running in other parts of the world?'),(0,o.kt)("h2",{id:"one-browser---many-connections"},"One Browser - Many Connections"),(0,o.kt)("p",null,"What happens when we request a web page?\nA simple example is ",(0,o.kt)("a",{parentName:"p",href:"http://example.com/"},"http://example.com/"),".\nWhen the browser displays this web page, it first ",(0,o.kt)("strong",{parentName:"p"},"downloads")," it and then ",(0,o.kt)("strong",{parentName:"p"},"renders")," it.\nThe web page comes as a file called ",(0,o.kt)("inlineCode",{parentName:"p"},"index.html"),".\nWe can roughly redo its steps like this:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ wget http://example.com/ # download index.html\nwget http://example.com/\n--2022-12-02 15:53:31-- http://example.com/\nResolving example.com (example.com)... 2606:2800:220:1:248:1893:25c8:1946, 93.184.216.34\nConnecting to example.com (example.com)|2606:2800:220:1:248:1893:25c8:1946|:80... connected.\nHTTP request sent, awaiting response... 200 OK\nLength: 1256 (1,2K) [text/html]\nSaving to: \u2018index.html\u2019\n\nindex.html 100%[====================================================================================================================================================================>] 1,23K --.-KB/s in 0s\n\n2022-12-02 15:53:31 (248 MB/s) - \u2018index.html\u2019 saved [1256/1256]\n")),(0,o.kt)("p",null,"Then we can view the HTML contents of the file:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-html"},'student@os:~$ cat index.html\n\n\n\n Example Domain\n\n \n \n \n \n\n\n\n
\n

Example Domain

\n

This domain is for use in illustrative examples in documents. You may use this\n domain in literature without prior coordination or asking for permission.

\n

More information...

\n
\n\n\n')),(0,o.kt)("p",null,"And, finally, we can render the ",(0,o.kt)("inlineCode",{parentName:"p"},"index.html")," file in the browser like so:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ xdg-open index.html # xdg-open invokes the OS's default program for opening HTML files\n")),(0,o.kt)("p",null,"After running the command above, look at the URL in the browser.\nIt's not ",(0,o.kt)("a",{parentName:"p",href:"http://example.com/"},"http://example.com/")," anymore, but the path to your ",(0,o.kt)("strong",{parentName:"p"},"local")," ",(0,o.kt)("inlineCode",{parentName:"p"},"index.html"),"."),(0,o.kt)("p",null,"So now you've switched from doing ",(0,o.kt)("strong",{parentName:"p"},"remote")," I/O back to ",(0,o.kt)("strong",{parentName:"p"},"local")," I/O.\nDeluge does the same thing: it performs ",(0,o.kt)("strong",{parentName:"p"},"remote")," I/O operations to get files locally so that you, the user, can do ",(0,o.kt)("strong",{parentName:"p"},"local")," I/O with it.\nRemote and local I/O are not by any means separated.\nRather, they are strongly intertwined."),(0,o.kt)("h2",{id:"connection"},"Connection"),(0,o.kt)("p",null,"Focusing on how the ",(0,o.kt)("inlineCode",{parentName:"p"},"index.html")," file is sent over the Web, we first need to establish the 2 ",(0,o.kt)("strong",{parentName:"p"},"endpoints"),".\nWe have a ",(0,o.kt)("strong",{parentName:"p"},"sender"),": the ",(0,o.kt)("a",{parentName:"p",href:"http://example.com/"},"http://example.com/")," server.\nThen we have a ",(0,o.kt)("strong",{parentName:"p"},"receiver"),": our host.\nHow do the 2 endpoints know each other?"),(0,o.kt)("h3",{id:"recap-ips"},"Recap: IPs"),(0,o.kt)("p",null,"When someone asks you who you are on the internet, the answer is obvious: your ",(0,o.kt)("strong",{parentName:"p"},"IP Address"),".\nIP Addresses are 32-bit (or 128-bit for IPv6) numbers that identify hosts on the web.\nTo find the IPv4 and IPv6 addresses of a host given by a URL, you can use the ",(0,o.kt)("inlineCode",{parentName:"p"},"host")," command:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~$ host example.com\nexample.com has address 93.184.216.34\nexample.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946\nexample.com mail is handled by 0 .\n")),(0,o.kt)("p",null,"So we can imagine our browser identifies ",(0,o.kt)("a",{parentName:"p",href:"http://example.com/"},"http://example.com/")," by its IP address, say ",(0,o.kt)("inlineCode",{parentName:"p"},"93.184.216.34"),".\nSimilarly, the server also knows our host's IP address.\nEach of them uses the other's IP address to locate their peer and communicate with them."),(0,o.kt)("p",null,"But what if we shift our example to another site: ",(0,o.kt)("a",{parentName:"p",href:"https://open-education-hub.github.io/operating-systems/"},"https://open-education-hub.github.io/operating-systems/"),"?\nLet's say we open 2 tabs:"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"one to the ",(0,o.kt)("a",{parentName:"li",href:"https://open-education-hub.github.io/operating-systems/Lab/IO/file-descriptors"},"File Descriptors section")," of this lab"),(0,o.kt)("li",{parentName:"ul"},"another one to the ",(0,o.kt)("a",{parentName:"li",href:"https://open-education-hub.github.io/operating-systems/Lab/IO/file-handlers"},"File Handling section"))),(0,o.kt)("p",null,"Now our browser needs to ",(0,o.kt)("strong",{parentName:"p"},"know")," what to do with data coming from two sources.\nIn addition, the server also needs to maintain information about our 2 tabs so it can send the correct data to each of them.\nTherefore, each tab establishes a different ",(0,o.kt)("strong",{parentName:"p"},"connection")," to the server.\nAll communication between the tab and the site occurs within this connection."),(0,o.kt)("p",null,"Now the question is: ",(0,o.kt)("em",{parentName:"p"},"how can we maintain 2 connections between 2 endpoints only identified by IPs?"),"\n... and the answer is that we can't.\nIf the browser and the server were to only use IP addresses, they wouldn't be able to differentiate between the 2 connections mentioned above.\nWe need something more: ",(0,o.kt)("strong",{parentName:"p"},"ports")),(0,o.kt)("h3",{id:"further-than-ips-ports"},"Further than IPs: Ports"),(0,o.kt)("p",null,"A port is simply a ",(0,o.kt)("strong",{parentName:"p"},"number")," assigned to uniquely identify a connection from a host to another and to direct the data that's transferred between them.\nThis way, we can create multiple connections between the same 2 hosts.\nPort numbers are encoded on 16 bits and thus range from $0$ to $2^{16} - 1$, i.e. from $0$ to $65535$."),(0,o.kt)("p",null,"The first 1024 ports are reserved for well-known system services, such as SSH (which uses port 22).\nThese services run using certain ",(0,o.kt)("strong",{parentName:"p"},"communication protocols"),".\nA communication protocol is a set of rules that allow 2 or more hosts to communicate a certain way.\nThese rules include, but are not limited to:"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"the format of each message: fields, sizes, maximum lengths etc."),(0,o.kt)("li",{parentName:"ul"},"the order in which messages are sent"),(0,o.kt)("li",{parentName:"ul"},"the behaviour of the endpoints with respect to these messages")),(0,o.kt)("p",null,"So the correct way of saying it isn't that the SSH process / service runs on port 22, but rather that ",(0,o.kt)("strong",{parentName:"p"},"the SSH protocol runs on port 22"),"."),(0,o.kt)("p",null,"Our browser also uses ports to distinguish between different connections.\nAnd so does the ",(0,o.kt)("inlineCode",{parentName:"p"},"github.io"),' server: it uses one port for sending the "File Descriptors" page and another for the "File Handling" page.\nThe image below shows how multiple tabs to the same site can be handled.\nThe port numbers are chosen randomly.\nThey may have any value higher than 1023.'),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"Browser Tabs and Ports",src:n(1565).Z,width:"552",height:"352"})),(0,o.kt)("p",null,"So it should be clear now that a connection is uniquely identified by ",(0,o.kt)("strong",{parentName:"p"},"an IP and a port"),"."),(0,o.kt)("h2",{id:"api---hail-berkeley-sockets"},"API - Hail Berkeley Sockets"),(0,o.kt)("p",null,"Up to now we've described how sites work in general.\nBefore we can implement something of this sort ourselves, we need to understand the API.\nUnlike other APIs such as syscalls, which differ between OSs (Unix vs Windows for example), the one we're about to learn is almost universally adopted across OSs and programming languages.\nIt's called the ",(0,o.kt)("a",{parentName:"p",href:"https://en.wikipedia.org/wiki/Berkeley_sockets"},"Berkeley Sockets API"),".\nAnd with this, we've just introduced a new word: ",(0,o.kt)("strong",{parentName:"p"},"socket"),"."),(0,o.kt)("p",null,(0,o.kt)("img",{alt:"Not this Socket",src:n(1014).Z,width:"533",height:"800"})),(0,o.kt)("p",null,"No, not this type of socket...\nBut our socket is somewhat similar ",(0,o.kt)("em",{parentName:"p"},"in concept"),'.\nJust like wall sockets allow us to plug into the electric grid, network sockets allow us to "plug" into the Web.\nRemember ',(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/file-handlers"},"file handlers"),"?\nYou should.\nFile handlers are objects with which we can interact with files.\nSimilarly, sockets are handlers that provide an abstraction for a connection to another process, running either on a remote machine or on the same host."),(0,o.kt)("h3",{id:"sender-and-receiver"},"Sender and Receiver"),(0,o.kt)("p",null,"We'll start with 2 programs: a sender and a receiver.\nNavigate to ",(0,o.kt)("inlineCode",{parentName:"p"},"support/send-receive/")," and take a look at both ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py"),"."),(0,o.kt)("p",null,"The sender reads data from the keyboard and sends it to the receiver.\nThe receiver reads data continuously.\nUpon receiving the message ",(0,o.kt)("inlineCode",{parentName:"p"},'"exit"'),", it closes.\nOtherwise, it prints whatever it receives to ",(0,o.kt)("inlineCode",{parentName:"p"},"stdout"),".\nThis detail about how to handle a message containing ",(0,o.kt)("inlineCode",{parentName:"p"},'"exit"')," may be regarded as a ",(0,o.kt)("a",{parentName:"p",href:"#further-than-ips-ports"},"communication protocol")," established between the sender and the receiver."),(0,o.kt)("p",null,"Now open 2 terminals (or use ",(0,o.kt)("a",{parentName:"p",href:"https://tmuxcheatsheet.com/"},"tmux"),").\nFirst run ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py")," in one terminal.\nLooking at its code, the receiver does 2 things.\nIt creates a socket:"),(0,o.kt)("pre",null,(0,o.kt)("code",{parentName:"pre",className:"language-Python"},"sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)\n")),(0,o.kt)("p",null,"We'll explain the arguments in the ",(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/networking-101#udp"},"next section"),".\nOne thing to note here is that ",(0,o.kt)("strong",{parentName:"p"},"sockets are file descriptors too.")),(0,o.kt)("p",null,"The server displays its PID.\nGive it as an argument to ",(0,o.kt)("inlineCode",{parentName:"p"},"lsof"),", like you did in ",(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/redirections"},"the section on Redirections"),", to visualise the file descriptors opened by ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py"),"."),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/receiver-socket-fd"},"Quiz 1")),(0,o.kt)("p",null,'After creating the socket, the receiver exposes itself as "listening" for connections on IP ',(0,o.kt)("inlineCode",{parentName:"p"},"127.0.0.1")," and on port 5000.\nThis means that it is now ready and waiting for other processes to send messages to it."),(0,o.kt)("p",null,(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/quiz/bind-error-cause"},"Quiz 2")),(0,o.kt)("p",null,(0,o.kt)("strong",{parentName:"p"},"Remember:"),"\n",(0,o.kt)("strong",{parentName:"p"},(0,o.kt)("inlineCode",{parentName:"strong"},"bind()"),"-ing to a certain port locks (like a mutex) it for the current process."),"\n",(0,o.kt)("strong",{parentName:"p"},"No other socket may ",(0,o.kt)("inlineCode",{parentName:"strong"},"bind()")," to this port until the initial socket bound to it is ",(0,o.kt)("inlineCode",{parentName:"strong"},"close()"),"d.")),(0,o.kt)("p",null,"Now run ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py")," and type some messages.\nSee them appear in the terminal where ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py")," is running.\nThe sender creates a socket and then sends messages directly to IP ",(0,o.kt)("inlineCode",{parentName:"p"},"127.0.0.1")," and port 5000 using ",(0,o.kt)("inlineCode",{parentName:"p"},"sendto()"),".\nWithout stopping the first ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py")," create another one in another terminal.\nType messages to both senders and see all of them displayed by ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py")," along with the addresses and ports from where they came."),(0,o.kt)("p",null,"In the end, both ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py")," ",(0,o.kt)("inlineCode",{parentName:"p"},"close()")," their sockets.\nYou can see this in analogy to regular file descriptors."),(0,o.kt)("p",null,"So we can use ",(0,o.kt)("inlineCode",{parentName:"p"},"sendto()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"recvfrom()"),' to send and receive messages via sockets.\nThis is a very simple communication model where the receiver is like a "sink" that receives messages from anywhere.\nAs you can see, it has no control over who sends data to it.\nTo get a high-level understanding of how these messages are passed and what other models exist, head over to the ',(0,o.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/IO/networking-101"},"next section"),"."),(0,o.kt)("h3",{id:"practice"},"Practice"),(0,o.kt)("ol",null,(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Use the C sockets API to replicate the behavior of ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py"),"."),(0,o.kt)("p",{parentName:"li"},"Start from the skeleton in ",(0,o.kt)("inlineCode",{parentName:"p"},"support/send-receive"),".\nThe workflow is the same: define the endpoint using IP (",(0,o.kt)("inlineCode",{parentName:"p"},"localhost"),") and port (",(0,o.kt)("inlineCode",{parentName:"p"},"5000"),"), then communicate using ",(0,o.kt)("inlineCode",{parentName:"p"},"sendto()")," and ",(0,o.kt)("inlineCode",{parentName:"p"},"recvfrom()"),".\nWe recommend starting with ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.c")," and test it using ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py"),", then continue with ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.c")," and test it using ",(0,o.kt)("inlineCode",{parentName:"p"},"sender.py"),".\nIf everything is right, each sender should be able to communicate with each receiver.")),(0,o.kt)("li",{parentName:"ol"},(0,o.kt)("p",{parentName:"li"},"Use the API you've just learned about to fill in the ",(0,o.kt)("inlineCode",{parentName:"p"},"TODO"),"s in ",(0,o.kt)("inlineCode",{parentName:"p"},"support/receive-challenges/receive_net_dgram_socket.c"),"."),(0,o.kt)("p",{parentName:"li"},"This is like ",(0,o.kt)("inlineCode",{parentName:"p"},"receiver.py"),".\nFor it to run properly, you should compile it using ",(0,o.kt)("inlineCode",{parentName:"p"},"make"),", then run it and after that run ",(0,o.kt)("inlineCode",{parentName:"p"},"send_net_dgram_socket"),".\nIf you solved the challenge correctly, ",(0,o.kt)("inlineCode",{parentName:"p"},"receive_net_dgram_socket")," should display the flag."))))}d.isMDXComponent=!0},1565:(e,t,n)=>{n.d(t,{Z:()=>a});const a=n.p+"assets/images/browser-tabs-2e3b83e1acd44607136a81a1a6346e56.svg"},1014:(e,t,n)=>{n.d(t,{Z:()=>a});const a=n.p+"assets/images/standard-us-power-outlet-c4b65ad6ff6a97ce0b898b5ca79be88f.jpg"}}]); \ No newline at end of file diff --git a/17/assets/js/ac5c3a27.3de06132.js b/17/assets/js/ac5c3a27.3de06132.js new file mode 100644 index 0000000000..38582aed30 --- /dev/null +++ b/17/assets/js/ac5c3a27.3de06132.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[2456],{3905:(e,t,n)=>{n.d(t,{Zo:()=>c,kt:()=>y});var r=n(7294);function a(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function s(e){for(var t=1;t=0||(a[n]=e[n]);return a}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(a[n]=e[n])}return a}var l=r.createContext({}),p=function(e){var t=r.useContext(l),n=t;return e&&(n="function"==typeof e?e(t):s(s({},t),e)),n},c=function(e){var t=p(e.components);return r.createElement(l.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},d=r.forwardRef((function(e,t){var n=e.components,a=e.mdxType,o=e.originalType,l=e.parentName,c=i(e,["components","mdxType","originalType","parentName"]),u=p(n),d=a,y=u["".concat(l,".").concat(d)]||u[d]||m[d]||o;return n?r.createElement(y,s(s({ref:t},c),{},{components:n})):r.createElement(y,s({ref:t},c))}));function y(e,t){var n=arguments,a=t&&t.mdxType;if("string"==typeof e||a){var o=n.length,s=new Array(o);s[0]=d;var i={};for(var l in t)hasOwnProperty.call(t,l)&&(i[l]=t[l]);i.originalType=e,i[u]="string"==typeof e?e:a,s[1]=i;for(var p=2;p{n.r(t),n.d(t,{assets:()=>l,contentTitle:()=>s,default:()=>m,frontMatter:()=>o,metadata:()=>i,toc:()=>p});var r=n(7462),a=(n(7294),n(3905));const o={},s="`create_sleepy` Process Ending",i={unversionedId:"Lab/Compute/quiz/create-sleepy-process-ending",id:"Lab/Compute/quiz/create-sleepy-process-ending",title:"`create_sleepy` Process Ending",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/create-sleepy-process-ending.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/create-sleepy-process-ending",permalink:"/operating-systems/17/Lab/Compute/quiz/create-sleepy-process-ending",draft:!1,tags:[],version:"current",frontMatter:{}},l={},p=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],c={toc:p},u="wrapper";function m(e){let{components:t,...n}=e;return(0,a.kt)(u,(0,r.Z)({},c,n,{components:t,mdxType:"MDXLayout"}),(0,a.kt)("h1",{id:"create_sleepy-process-ending"},(0,a.kt)("inlineCode",{parentName:"h1"},"create_sleepy")," Process Ending"),(0,a.kt)("h2",{id:"question-text"},"Question Text"),(0,a.kt)("p",null,"Why does the ",(0,a.kt)("inlineCode",{parentName:"p"},"create_sleepy")," process wait a very long time before ending?\nUse ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/system.3.html"},(0,a.kt)("inlineCode",{parentName:"a"},"system"),"'s man page")," to find the answer."),(0,a.kt)("h2",{id:"question-answers"},"Question Answers"),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the code is unoptimised (the default optimisation level is ",(0,a.kt)("inlineCode",{parentName:"p"},"-O0"),")")),(0,a.kt)("li",{parentName:"ul"},(0,a.kt)("p",{parentName:"li"},"Because the operating system takes very long to finish the process"))),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because ",(0,a.kt)("inlineCode",{parentName:"li"},"system")," returns when the command given to it (",(0,a.kt)("inlineCode",{parentName:"li"},"sleep 1000"),") ends")),(0,a.kt)("ul",null,(0,a.kt)("li",{parentName:"ul"},"Because the CPU is very slow")),(0,a.kt)("h2",{id:"feedback"},"Feedback"),(0,a.kt)("p",null,"The ",(0,a.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man3/system.3.html"},"man page")," says it clearly:"),(0,a.kt)("pre",null,(0,a.kt)("code",{parentName:"pre",className:"language-text"},"system() returns after the command has been completed.\n")),(0,a.kt)("p",null,"Therefore, in our case, it returns after ",(0,a.kt)("inlineCode",{parentName:"p"},"sleep 1000")," ends."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/ade537b6.876be72a.js b/17/assets/js/ade537b6.876be72a.js new file mode 100644 index 0000000000..30ea9b60f6 --- /dev/null +++ b/17/assets/js/ade537b6.876be72a.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[4470],{3905:(e,t,n)=>{n.d(t,{Zo:()=>u,kt:()=>d});var r=n(7294);function o(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function a(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var r=Object.getOwnPropertySymbols(e);t&&(r=r.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,r)}return n}function i(e){for(var t=1;t=0||(o[n]=e[n]);return o}(e,t);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);for(r=0;r=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(o[n]=e[n])}return o}var s=r.createContext({}),l=function(e){var t=r.useContext(s),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},u=function(e){var t=l(e.components);return r.createElement(s.Provider,{value:t},e.children)},p="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return r.createElement(r.Fragment,{},t)}},f=r.forwardRef((function(e,t){var n=e.components,o=e.mdxType,a=e.originalType,s=e.parentName,u=c(e,["components","mdxType","originalType","parentName"]),p=l(n),f=o,d=p["".concat(s,".").concat(f)]||p[f]||m[f]||a;return n?r.createElement(d,i(i({ref:t},u),{},{components:n})):r.createElement(d,i({ref:t},u))}));function d(e,t){var n=arguments,o=t&&t.mdxType;if("string"==typeof e||o){var a=n.length,i=new Array(a);i[0]=f;var c={};for(var s in t)hasOwnProperty.call(t,s)&&(c[s]=t[s]);c.originalType=e,c[p]="string"==typeof e?e:o,i[1]=c;for(var l=2;l{n.r(t),n.d(t,{assets:()=>s,contentTitle:()=>i,default:()=>m,frontMatter:()=>a,metadata:()=>c,toc:()=>l});var r=n(7462),o=(n(7294),n(3905));const a={},i="TLS Synchronization",c={unversionedId:"Lab/Compute/quiz/tls-synchronization",id:"Lab/Compute/quiz/tls-synchronization",title:"TLS Synchronization",description:"Question Text",source:"@site/docs/Lab/Compute/quiz/tls-synchronization.md",sourceDirName:"Lab/Compute/quiz",slug:"/Lab/Compute/quiz/tls-synchronization",permalink:"/operating-systems/17/Lab/Compute/quiz/tls-synchronization",draft:!1,tags:[],version:"current",frontMatter:{}},s={},l=[{value:"Question Text",id:"question-text",level:2},{value:"Question Answers",id:"question-answers",level:2},{value:"Feedback",id:"feedback",level:2}],u={toc:l},p="wrapper";function m(e){let{components:t,...n}=e;return(0,o.kt)(p,(0,r.Z)({},u,n,{components:t,mdxType:"MDXLayout"}),(0,o.kt)("h1",{id:"tls-synchronization"},"TLS Synchronization"),(0,o.kt)("h2",{id:"question-text"},"Question Text"),(0,o.kt)("p",null,"Is placing ",(0,o.kt)("inlineCode",{parentName:"p"},"var")," from ",(0,o.kt)("inlineCode",{parentName:"p"},"support/race-condition/c/race_condition_tls.c")," in the TLS a valid form of synchronization?"),(0,o.kt)("h2",{id:"question-answers"},"Question Answers"),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"No, because the race condition remains.\nIt just doesn't manifest itself anymore")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},"No, because the threads now access different variables, not the same one")),(0,o.kt)("ul",null,(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Yes, because we now remove the race condition")),(0,o.kt)("li",{parentName:"ul"},(0,o.kt)("p",{parentName:"li"},"Yes, because now the result is correct"))),(0,o.kt)("h2",{id:"feedback"},"Feedback"),(0,o.kt)("p",null,"Synchronization means that both threads should access the same variable, whereas placing it in the TLS makes each of them access a copy of the variable."))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/aecc3d85.999b135a.js b/17/assets/js/aecc3d85.999b135a.js new file mode 100644 index 0000000000..fa3b6c5b70 --- /dev/null +++ b/17/assets/js/aecc3d85.999b135a.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[6492],{3905:(e,t,n)=>{n.d(t,{Zo:()=>p,kt:()=>h});var a=n(7294);function r(e,t,n){return t in e?Object.defineProperty(e,t,{value:n,enumerable:!0,configurable:!0,writable:!0}):e[t]=n,e}function o(e,t){var n=Object.keys(e);if(Object.getOwnPropertySymbols){var a=Object.getOwnPropertySymbols(e);t&&(a=a.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),n.push.apply(n,a)}return n}function i(e){for(var t=1;t=0||(r[n]=e[n]);return r}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(a=0;a=0||Object.prototype.propertyIsEnumerable.call(e,n)&&(r[n]=e[n])}return r}var c=a.createContext({}),s=function(e){var t=a.useContext(c),n=t;return e&&(n="function"==typeof e?e(t):i(i({},t),e)),n},p=function(e){var t=s(e.components);return a.createElement(c.Provider,{value:t},e.children)},u="mdxType",m={inlineCode:"code",wrapper:function(e){var t=e.children;return a.createElement(a.Fragment,{},t)}},d=a.forwardRef((function(e,t){var n=e.components,r=e.mdxType,o=e.originalType,c=e.parentName,p=l(e,["components","mdxType","originalType","parentName"]),u=s(n),d=r,h=u["".concat(c,".").concat(d)]||u[d]||m[d]||o;return n?a.createElement(h,i(i({ref:t},p),{},{components:n})):a.createElement(h,i({ref:t},p))}));function h(e,t){var n=arguments,r=t&&t.mdxType;if("string"==typeof e||r){var o=n.length,i=new Array(o);i[0]=d;var l={};for(var c in t)hasOwnProperty.call(t,c)&&(l[c]=t[c]);l.originalType=e,l[u]="string"==typeof e?e:r,i[1]=l;for(var s=2;s{n.r(t),n.d(t,{assets:()=>c,contentTitle:()=>i,default:()=>m,frontMatter:()=>o,metadata:()=>l,toc:()=>s});var a=n(7462),r=(n(7294),n(3905));const o={},i="Arena",l={unversionedId:"Lab/Data/arena",id:"Lab/Data/arena",title:"Arena",description:"Challenge tasks",source:"@site/docs/Lab/Data/arena.md",sourceDirName:"Lab/Data",slug:"/Lab/Data/arena",permalink:"/operating-systems/17/Lab/Data/arena",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Memory Security",permalink:"/operating-systems/17/Lab/Data/memory-security"},next:{title:"Compute",permalink:"/operating-systems/17/Lab/Compute/"}},c={},s=[{value:"Memory Support",id:"memory-support",level:2},{value:"Reference Counting",id:"reference-counting",level:3},{value:"Operator overloading",id:"operator-overloading",level:3},{value:"Practice",id:"practice",level:4},{value:"Use-After-Free",id:"use-after-free",level:2},{value:"Code Injection",id:"code-injection",level:2},{value:"Quiz",id:"quiz",level:3},{value:"DEP",id:"dep",level:2},{value:"Practice",id:"practice-1",level:3},{value:"Code Reuse",id:"code-reuse",level:2},{value:"Quiz",id:"quiz-1",level:3},{value:"Practice",id:"practice-2",level:3}],p={toc:s},u="wrapper";function m(e){let{components:t,...n}=e;return(0,r.kt)(u,(0,a.Z)({},p,n,{components:t,mdxType:"MDXLayout"}),(0,r.kt)("h1",{id:"arena"},"Arena"),(0,r.kt)("p",null,"Challenge tasks"),(0,r.kt)("h2",{id:"memory-support"},"Memory Support"),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Manual memory management")," (MMM) is one of the most difficult tasks.\nEven experienced programmers make mistakes when tackling such a complicated endeavor.\nAs a consequence, the programming world has been migrating towards languages that offer automatic memory management (AMM).\nAMM programming languages typically offer a garbage collector that tracks down the usage of objects and frees memory once no references exist to a given object.\nAs a consequence, garbage collected programming languages are easier to use and safer.\nHowever, this comes with a cost: the garbage collector, in most cases, requires a significant amount of resources to run.\nTherefore, for performance-critical systems, MMM is still the preferred solution."),(0,r.kt)("p",null,"A middle-ground between programming languages that have AMM (Java, Python, Swift, D) and those that do not (C, C++) is represented by those languages that do not have built-in AMM but offer the possibility to implement it as a library solution (C++, D).\nConcretely, these languages offer lightweight library solutions to optimally track down the lifetime of an object.\nThis is done by using reference counted objects."),(0,r.kt)("h3",{id:"reference-counting"},"Reference Counting"),(0,r.kt)("p",null,"Reference counting is a technique of tracking the lifetime of an object by counting how many references to an object exist.\nAs long as at least one reference exists, the object cannot be destroyed.\nOnce no reference to a given object exists, it can be safely destroyed.\nReference counted is typically implemented by storing a count with the actual payload of the object.\nEvery time a new reference to the object is created, the reference count is incremented.\nEvery time a reference expires, the reference is decremented."),(0,r.kt)("p",null,"The operations that trigger a reference increment are:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"initializing an object from another object."),(0,r.kt)("li",{parentName:"ul"},"assigning an object to another object.")),(0,r.kt)("p",null,"The operations that trigger a reference decrement are:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"the lifetime of an object expires")),(0,r.kt)("p",null,"Modern programming languages offer the possibility to specify what code should be run in each of these situations, therefore enabling the implementation of referenced counted data structures.\nAs such, copy constructors may be used to automatically initialize an object from another object, assignment operators may be used to assign an object to another object and destructors may be used to destroy objects."),(0,r.kt)("h3",{id:"operator-overloading"},"Operator overloading"),(0,r.kt)("p",null,"Navigate to the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/reference-counting")," directory.\nAnalyze the ",(0,r.kt)("inlineCode",{parentName:"p"},"operators.d")," file.\nA ",(0,r.kt)("inlineCode",{parentName:"p"},"struct")," is defined that also implements 4 special functions: a constructor, a copy constructor, an assignment operator and a destructor.\nEach of these special functions may be called automatically by the compiler:"),(0,r.kt)("ul",null,(0,r.kt)("li",{parentName:"ul"},"the constructor is called automatically whenever an object is initialized with a field of a type that corresponds to the constructor parameter type."),(0,r.kt)("li",{parentName:"ul"},"the copy constructor is called automatically when an object is initialized from an object of the same type."),(0,r.kt)("li",{parentName:"ul"},"the assignment operator is called automatically when an object is assigned an object of the same type."),(0,r.kt)("li",{parentName:"ul"},"the destructor is called automatically whenever an object goes out of scope.")),(0,r.kt)("p",null,(0,r.kt)("strong",{parentName:"p"},"Note: the difference between initialization and assignment is that the initialization occurs when an object is being declared and occurs a single time (",(0,r.kt)("inlineCode",{parentName:"strong"},"Obj o1 = 1"),"), whereas assignement is decoupled from the declaration site and may occur multiple times (provided that the variable is mutable).")),(0,r.kt)("p",null,"Compile and run the program in ",(0,r.kt)("inlineCode",{parentName:"p"},"operators.d"),".\nNotice how the different special functions are automatically called.\nConsidering the definition of ",(0,r.kt)("inlineCode",{parentName:"p"},"Obj")," from the file ",(0,r.kt)("inlineCode",{parentName:"p"},"operators.d"),", answer the following ",(0,r.kt)("a",{parentName:"p",href:"/operating-systems/17/Lab/Data/quiz/operators"},"Quiz"),"."),(0,r.kt)("h4",{id:"practice"},"Practice"),(0,r.kt)("p",null,"Navigate to the ",(0,r.kt)("inlineCode",{parentName:"p"},"support/reference-counting")," directory.\nAnalyze the ",(0,r.kt)("inlineCode",{parentName:"p"},"refcount_skel.d"),".\nA reference counted ",(0,r.kt)("inlineCode",{parentName:"p"},"int")," array is implemented, however, some bits are missing.\nRun the code, try to understand what happens."),(0,r.kt)("p",null,"The constructor allocates memory for the array, whereas the destructor deallocates it.\nCompile and run the code.\nNotice how the array's memory is automatically managed."),(0,r.kt)("ol",null,(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Uncomment the following line in the ",(0,r.kt)("inlineCode",{parentName:"p"},"main")," function (",(0,r.kt)("inlineCode",{parentName:"p"},"//test1()"),").\nRun the code.\nWhat happens?\nWhy?")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"The reference counted array does not implement the copy constructor.\nComment the ",(0,r.kt)("inlineCode",{parentName:"p"},"version(none)")," annotation for the copy constructor and implement the logic so that the reference counted array is correct.\nWhen an object is initialized from another object, we need to appropriately set the fields and then increment the reference count.\nOnce you have completed this exercise, make sure the output is correct and that the reference counted array is not freed too early.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Uncomment the following line in the ",(0,r.kt)("inlineCode",{parentName:"p"},"main")," function (",(0,r.kt)("inlineCode",{parentName:"p"},"//test2()"),").\nRun the code.\nWhat happens?\nWhy?\nUse GDB to find out.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"The reference counted array does not implement the assignment operator.\nComment the ",(0,r.kt)("inlineCode",{parentName:"p"},"version(none)")," annotation for the assignment operator and implement the logic so that the reference counted array is correct.\nWhen an object is assigned to another object, we need to first decrement the count for the object that is being assigned to, then fill the fields similarly to the copy constructor case and lastly increment the count for the assigned object.\nAfter completing the exercise, make sure that the memory is properly managed.")),(0,r.kt)("li",{parentName:"ol"},(0,r.kt)("p",{parentName:"li"},"Play with your reference counted array and create different scenarios to test its limits."))),(0,r.kt)("h2",{id:"use-after-free"},"Use-After-Free"),(0,r.kt)("p",null,"TODO"),(0,r.kt)("h2",{id:"code-injection"},"Code Injection"),(0,r.kt)("p",null,"TODO"),(0,r.kt)("h3",{id:"quiz"},"Quiz"),(0,r.kt)("p",null,"TODO"),(0,r.kt)("h2",{id:"dep"},"DEP"),(0,r.kt)("h3",{id:"practice-1"},"Practice"),(0,r.kt)("p",null,"TODO"),(0,r.kt)("h2",{id:"code-reuse"},"Code Reuse"),(0,r.kt)("p",null,"TODO"),(0,r.kt)("h3",{id:"quiz-1"},"Quiz"),(0,r.kt)("p",null,"TODO"),(0,r.kt)("h3",{id:"practice-2"},"Practice"),(0,r.kt)("p",null,"TODO"))}m.isMDXComponent=!0}}]); \ No newline at end of file diff --git a/17/assets/js/b407248a.19f91a4e.js b/17/assets/js/b407248a.19f91a4e.js new file mode 100644 index 0000000000..893be8c5d1 --- /dev/null +++ b/17/assets/js/b407248a.19f91a4e.js @@ -0,0 +1 @@ +"use strict";(self.webpackChunkso=self.webpackChunkso||[]).push([[1101],{3905:(e,t,a)=>{a.d(t,{Zo:()=>c,kt:()=>k});var n=a(7294);function l(e,t,a){return t in e?Object.defineProperty(e,t,{value:a,enumerable:!0,configurable:!0,writable:!0}):e[t]=a,e}function o(e,t){var a=Object.keys(e);if(Object.getOwnPropertySymbols){var n=Object.getOwnPropertySymbols(e);t&&(n=n.filter((function(t){return Object.getOwnPropertyDescriptor(e,t).enumerable}))),a.push.apply(a,n)}return a}function i(e){for(var t=1;t=0||(l[a]=e[a]);return l}(e,t);if(Object.getOwnPropertySymbols){var o=Object.getOwnPropertySymbols(e);for(n=0;n=0||Object.prototype.propertyIsEnumerable.call(e,a)&&(l[a]=e[a])}return l}var r=n.createContext({}),p=function(e){var t=n.useContext(r),a=t;return e&&(a="function"==typeof e?e(t):i(i({},t),e)),a},c=function(e){var t=p(e.components);return n.createElement(r.Provider,{value:t},e.children)},m="mdxType",d={inlineCode:"code",wrapper:function(e){var t=e.children;return n.createElement(n.Fragment,{},t)}},u=n.forwardRef((function(e,t){var a=e.components,l=e.mdxType,o=e.originalType,r=e.parentName,c=s(e,["components","mdxType","originalType","parentName"]),m=p(a),u=l,k=m["".concat(r,".").concat(u)]||m[u]||d[u]||o;return a?n.createElement(k,i(i({ref:t},c),{},{components:a})):n.createElement(k,i({ref:t},c))}));function k(e,t){var a=arguments,l=t&&t.mdxType;if("string"==typeof e||l){var o=a.length,i=new Array(o);i[0]=u;var s={};for(var r in t)hasOwnProperty.call(t,r)&&(s[r]=t[r]);s.originalType=e,s[m]="string"==typeof e?e:l,i[1]=s;for(var p=2;p{a.r(t),a.d(t,{assets:()=>r,contentTitle:()=>i,default:()=>d,frontMatter:()=>o,metadata:()=>s,toc:()=>p});var n=a(7462),l=(a(7294),a(3905));const o={},i="Memory Allocator",s={unversionedId:"Assignments/Memory Allocator/README",id:"Assignments/Memory Allocator/README",title:"Memory Allocator",description:"Objectives",source:"@site/docs/Assignments/Memory Allocator/README.md",sourceDirName:"Assignments/Memory Allocator",slug:"/Assignments/Memory Allocator/",permalink:"/operating-systems/17/Assignments/Memory Allocator/",draft:!1,tags:[],version:"current",frontMatter:{},sidebar:"sidebar",previous:{title:"Mini-libc",permalink:"/operating-systems/17/Assignments/Mini Libc/"},next:{title:"Parallel Graph",permalink:"/operating-systems/17/Assignments/Parallel Graph/"}},r={},p=[{value:"Objectives",id:"objectives",level:2},{value:"Statement",id:"statement",level:2},{value:"Support Code",id:"support-code",level:2},{value:"API",id:"api",level:2},{value:"Implementation",id:"implementation",level:2},{value:"Memory Alignment",id:"memory-alignment",level:3},{value:"Block Reuse",id:"block-reuse",level:3},{value:"struct block_meta",id:"struct-block_meta",level:4},{value:"Split Block",id:"split-block",level:4},{value:"Coalesce Blocks",id:"coalesce-blocks",level:4},{value:"Find Best Block",id:"find-best-block",level:4},{value:"Heap Preallocation",id:"heap-preallocation",level:3},{value:"Building Memory Allocator",id:"building-memory-allocator",level:2},{value:"Testing and Grading",id:"testing-and-grading",level:2},{value:"Debugging",id:"debugging",level:3},{value:"Debugging in VSCode",id:"debugging-in-vscode",level:3},{value:"Resources",id:"resources",level:2}],c={toc:p},m="wrapper";function d(e){let{components:t,...o}=e;return(0,l.kt)(m,(0,n.Z)({},c,o,{components:t,mdxType:"MDXLayout"}),(0,l.kt)("h1",{id:"memory-allocator"},"Memory Allocator"),(0,l.kt)("h2",{id:"objectives"},"Objectives"),(0,l.kt)("ul",null,(0,l.kt)("li",{parentName:"ul"},"Learn the basics of memory management by implementing minimal versions of ",(0,l.kt)("inlineCode",{parentName:"li"},"malloc()"),", ",(0,l.kt)("inlineCode",{parentName:"li"},"calloc()"),", ",(0,l.kt)("inlineCode",{parentName:"li"},"realloc()"),", and ",(0,l.kt)("inlineCode",{parentName:"li"},"free()"),"."),(0,l.kt)("li",{parentName:"ul"},"Accommodate with the memory management syscalls in Linux: ",(0,l.kt)("inlineCode",{parentName:"li"},"brk()"),", ",(0,l.kt)("inlineCode",{parentName:"li"},"mmap()"),", and ",(0,l.kt)("inlineCode",{parentName:"li"},"munmap()"),"."),(0,l.kt)("li",{parentName:"ul"},"Understand the bottlenecks of memory allocation and how to reduce them.")),(0,l.kt)("h2",{id:"statement"},"Statement"),(0,l.kt)("p",null,"Build a minimalistic memory allocator that can be used to manually manage virtual memory.\nThe goal is to have a reliable library that accounts for explicit allocation, reallocation, and initialization of memory."),(0,l.kt)("h2",{id:"support-code"},"Support Code"),(0,l.kt)("p",null,"The support code consists of three directories:"),(0,l.kt)("ul",null,(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"src/")," will contain your solution"),(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"tests/")," contains the test suite and a Python script to verify your work"),(0,l.kt)("li",{parentName:"ul"},(0,l.kt)("inlineCode",{parentName:"li"},"utils/")," contains ",(0,l.kt)("inlineCode",{parentName:"li"},"osmem.h")," that describes your library interface, ",(0,l.kt)("inlineCode",{parentName:"li"},"block_meta.h")," which contains details of ",(0,l.kt)("inlineCode",{parentName:"li"},"struct block_meta"),", and an implementation for ",(0,l.kt)("inlineCode",{parentName:"li"},"printf()")," function that does ",(0,l.kt)("strong",{parentName:"li"},"NOT")," use the heap")),(0,l.kt)("p",null,"The test suite consists of ",(0,l.kt)("inlineCode",{parentName:"p"},".c")," files that will be dynamically linked to your library, ",(0,l.kt)("inlineCode",{parentName:"p"},"libosmem.so"),".\nYou can find the sources in the ",(0,l.kt)("inlineCode",{parentName:"p"},"tests/snippets/")," directory.\nThe results of the previous will also be stored in ",(0,l.kt)("inlineCode",{parentName:"p"},"tests/snippets/")," and the reference files are in the ",(0,l.kt)("inlineCode",{parentName:"p"},"tests/ref/")," directory."),(0,l.kt)("p",null,"The automated checking is performed using ",(0,l.kt)("inlineCode",{parentName:"p"},"run-tests.py"),".\nIt runs each test and compares the syscalls made by the ",(0,l.kt)("inlineCode",{parentName:"p"},"os_*")," functions with the reference file, providing a diff if the test failed."),(0,l.kt)("h2",{id:"api"},"API"),(0,l.kt)("ol",null,(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},(0,l.kt)("inlineCode",{parentName:"p"},"void *os_malloc(size_t size)")),(0,l.kt)("p",{parentName:"li"},"Allocates ",(0,l.kt)("inlineCode",{parentName:"p"},"size")," bytes and returns a pointer to the allocated memory."),(0,l.kt)("p",{parentName:"li"},"Chunks of memory smaller than ",(0,l.kt)("inlineCode",{parentName:"p"},"MMAP_THRESHOLD")," are allocated with ",(0,l.kt)("inlineCode",{parentName:"p"},"brk()"),".\nBigger chunks are allocated using ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap()"),".\nThe memory is uninitialized."),(0,l.kt)("ul",{parentName:"li"},(0,l.kt)("li",{parentName:"ul"},"Passing ",(0,l.kt)("inlineCode",{parentName:"li"},"0")," as ",(0,l.kt)("inlineCode",{parentName:"li"},"size")," will return ",(0,l.kt)("inlineCode",{parentName:"li"},"NULL"),"."))),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},(0,l.kt)("inlineCode",{parentName:"p"},"void *os_calloc(size_t nmemb, size_t size)")),(0,l.kt)("p",{parentName:"li"},"Allocates memory for an array of ",(0,l.kt)("inlineCode",{parentName:"p"},"nmemb")," elements of ",(0,l.kt)("inlineCode",{parentName:"p"},"size")," bytes each and returns a pointer to the allocated memory."),(0,l.kt)("p",{parentName:"li"},"Chunks of memory smaller than ",(0,l.kt)("a",{parentName:"p",href:"https://man7.org/linux/man-pages/man2/getpagesize.2.html"},(0,l.kt)("inlineCode",{parentName:"a"},"page_size"))," are allocated with ",(0,l.kt)("inlineCode",{parentName:"p"},"brk()"),".\nBigger chunks are allocated using ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap()"),".\nThe memory is set to zero."),(0,l.kt)("ul",{parentName:"li"},(0,l.kt)("li",{parentName:"ul"},"Passing ",(0,l.kt)("inlineCode",{parentName:"li"},"0")," as ",(0,l.kt)("inlineCode",{parentName:"li"},"nmemb")," or ",(0,l.kt)("inlineCode",{parentName:"li"},"size")," will return ",(0,l.kt)("inlineCode",{parentName:"li"},"NULL"),"."))),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},(0,l.kt)("inlineCode",{parentName:"p"},"void *os_realloc(void *ptr, size_t size)")),(0,l.kt)("p",{parentName:"li"},"Changes the size of the memory block pointed to by ",(0,l.kt)("inlineCode",{parentName:"p"},"ptr")," to ",(0,l.kt)("inlineCode",{parentName:"p"},"size")," bytes.\nIf the size is smaller than the previously allocated size, the memory block will be truncated."),(0,l.kt)("p",{parentName:"li"},"If ",(0,l.kt)("inlineCode",{parentName:"p"},"ptr")," points to a block on heap, ",(0,l.kt)("inlineCode",{parentName:"p"},"os_realloc()")," will first try to expand the block, rather than moving it.\nOtherwise, the block will be reallocated and its contents copied."),(0,l.kt)("p",{parentName:"li"},"When attempting to expand a block followed by multiple free blocks, ",(0,l.kt)("inlineCode",{parentName:"p"},"os_realloc()")," will coalesce them one at a time and verify the condition for each.\nBlocks will remain coalesced even if the resulting block will not be big enough for the new size."),(0,l.kt)("p",{parentName:"li"},"Calling ",(0,l.kt)("inlineCode",{parentName:"p"},"os_realloc()")," on a block that has ",(0,l.kt)("inlineCode",{parentName:"p"},"STATUS_FREE")," should return ",(0,l.kt)("inlineCode",{parentName:"p"},"NULL"),".\nThis is a measure to prevent undefined behavior and make the implementation robust, it should not be considered a valid use case of ",(0,l.kt)("inlineCode",{parentName:"p"},"os_realloc()"),"."),(0,l.kt)("ul",{parentName:"li"},(0,l.kt)("li",{parentName:"ul"},"Passing ",(0,l.kt)("inlineCode",{parentName:"li"},"NULL")," as ",(0,l.kt)("inlineCode",{parentName:"li"},"ptr")," will have the same effect as ",(0,l.kt)("inlineCode",{parentName:"li"},"os_malloc(size)"),"."),(0,l.kt)("li",{parentName:"ul"},"Passing ",(0,l.kt)("inlineCode",{parentName:"li"},"0")," as ",(0,l.kt)("inlineCode",{parentName:"li"},"size")," will have the same effect as ",(0,l.kt)("inlineCode",{parentName:"li"},"os_free(ptr)"),"."))),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},(0,l.kt)("inlineCode",{parentName:"p"},"void os_free(void *ptr)")),(0,l.kt)("p",{parentName:"li"},"Frees memory previously allocated by ",(0,l.kt)("inlineCode",{parentName:"p"},"os_malloc()"),", ",(0,l.kt)("inlineCode",{parentName:"p"},"os_calloc()")," or ",(0,l.kt)("inlineCode",{parentName:"p"},"os_realloc()"),"."),(0,l.kt)("p",{parentName:"li"},(0,l.kt)("inlineCode",{parentName:"p"},"os_free()")," will not return memory from the heap to the OS by calling ",(0,l.kt)("inlineCode",{parentName:"p"},"brk()"),", but rather mark it as free and reuse it in future allocations.\nIn the case of mapped memory blocks, ",(0,l.kt)("inlineCode",{parentName:"p"},"os_free()")," will call ",(0,l.kt)("inlineCode",{parentName:"p"},"munmap()"),".")),(0,l.kt)("li",{parentName:"ol"},(0,l.kt)("p",{parentName:"li"},"General"),(0,l.kt)("ul",{parentName:"li"},(0,l.kt)("li",{parentName:"ul"},"Allocations that increase the heap size will only expand the last block if it is free."),(0,l.kt)("li",{parentName:"ul"},"You are allowed to use ",(0,l.kt)("inlineCode",{parentName:"li"},"sbrk()")," instead of ",(0,l.kt)("inlineCode",{parentName:"li"},"brk()"),", in view of the fact that ",(0,l.kt)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man2/brk.2.html#NOTES"},"on Linux")," ",(0,l.kt)("inlineCode",{parentName:"li"},"sbrk()")," is implemented using the ",(0,l.kt)("inlineCode",{parentName:"li"},"brk()"),"."),(0,l.kt)("li",{parentName:"ul"},"Do ",(0,l.kt)("strong",{parentName:"li"},"NOT")," use ",(0,l.kt)("a",{parentName:"li",href:"https://man7.org/linux/man-pages/man2/mremap.2.html"},(0,l.kt)("inlineCode",{parentName:"a"},"mremap()"))),(0,l.kt)("li",{parentName:"ul"},"You must check the error code returned by every syscall.\nYou can use the ",(0,l.kt)("inlineCode",{parentName:"li"},"DIE()")," macro for this.")))),(0,l.kt)("h2",{id:"implementation"},"Implementation"),(0,l.kt)("p",null,"An efficient implementation must keep data aligned, keep track of memory blocks and reuse freed blocks.\nThis can be further improved by reducing the number of syscalls and block operations."),(0,l.kt)("h3",{id:"memory-alignment"},(0,l.kt)("a",{parentName:"h3",href:"https://stackoverflow.com/a/381368"},"Memory Alignment")),(0,l.kt)("p",null,"Allocated memory should be aligned (i.e. all addresses are multiple of a given size).\nThis is a space-time trade-off because memory blocks are padded so each can be read in one transaction.\nIt also allows for atomicity when interacting with a block of memory."),(0,l.kt)("p",null,"All memory allocations should be aligned to ",(0,l.kt)("strong",{parentName:"p"},"8 bytes")," as required by 64 bit systems."),(0,l.kt)("h3",{id:"block-reuse"},"Block Reuse"),(0,l.kt)("h4",{id:"struct-block_meta"},(0,l.kt)("inlineCode",{parentName:"h4"},"struct block_meta")),(0,l.kt)("p",null,"We will consider a ",(0,l.kt)("strong",{parentName:"p"},"block")," to be a continuous zone of memory, allocated and managed by our implementation.\nThe structure ",(0,l.kt)("inlineCode",{parentName:"p"},"block_meta")," will be used to manage the metadata of a block.\nEach allocated zone will comprise of a ",(0,l.kt)("inlineCode",{parentName:"p"},"block_meta")," structure placed at the start, followed by data (",(0,l.kt)("strong",{parentName:"p"},"payload"),").\nFor all functions, the returned address will be that of the ",(0,l.kt)("strong",{parentName:"p"},"payload")," (not of the ",(0,l.kt)("inlineCode",{parentName:"p"},"block_meta")," structure)."),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-C"},"struct block_meta {\n size_t size;\n int status;\n struct block_meta *prev;\n struct block_meta *next;\n};\n")),(0,l.kt)("p",null,(0,l.kt)("em",{parentName:"p"},"Note"),": Both the ",(0,l.kt)("inlineCode",{parentName:"p"},"struct block_meta")," and the ",(0,l.kt)("strong",{parentName:"p"},"payload")," of a block should be aligned to ",(0,l.kt)("strong",{parentName:"p"},"8 bytes"),"."),(0,l.kt)("p",null,(0,l.kt)("em",{parentName:"p"},"Note"),": Most compilers will automatically pad the structure, but you should still align it for portability."),(0,l.kt)("p",null,(0,l.kt)("img",{alt:"memory-block",src:a(7666).Z,width:"641",height:"191"})),(0,l.kt)("h4",{id:"split-block"},"Split Block"),(0,l.kt)("p",null,"Reusing memory blocks improves the allocator's performance, but might lead to ",(0,l.kt)("a",{parentName:"p",href:"https://www.tutorialspoint.com/difference-between-internal-fragmentation-and-external-fragmentation#:~:text=What%20is%20Internal%20Fragmentation%3F"},"Internal Memory Fragmentation"),".\nThis happens when we allocate a size smaller than all available free blocks.\nIf we use one larger block the remaining size of that block will be wasted since it cannot be used for another allocation."),(0,l.kt)("p",null,"To avoid this, a block should be truncated to the required size and the remaining bytes should be used to create a new free block."),(0,l.kt)("p",null,(0,l.kt)("img",{alt:"Split Block",src:a(99).Z,width:"662",height:"282"})),(0,l.kt)("p",null,"The resulting free block should be reusable.\nThe split will not be performed if the remaining size (after reserving space for ",(0,l.kt)("inlineCode",{parentName:"p"},"block_meta")," structure and payload) is not big enough to fit another block (",(0,l.kt)("inlineCode",{parentName:"p"},"block_meta")," structure and at least ",(0,l.kt)("strong",{parentName:"p"},"1 byte")," of usable memory)."),(0,l.kt)("p",null,(0,l.kt)("em",{parentName:"p"},"Note"),": Do not forget the alignment!"),(0,l.kt)("h4",{id:"coalesce-blocks"},"Coalesce Blocks"),(0,l.kt)("p",null,"There are cases when there is enough free memory for an allocation, but it is spread across multiple blocks that cannot be used.\nThis is called ",(0,l.kt)("a",{parentName:"p",href:"https://www.tutorialspoint.com/difference-between-internal-fragmentation-and-external-fragmentation#:~:text=What%20is%20External%20Fragmentation%3F"},"External Memory Fragmentation"),"."),(0,l.kt)("p",null,"One technique to reduce external memory fragmentation is ",(0,l.kt)("strong",{parentName:"p"},"block coalescing")," which implies merging adjacent free blocks to form a contiguous chunk."),(0,l.kt)("p",null,(0,l.kt)("img",{alt:"Coalesce Block Image",src:a(6062).Z,width:"661",height:"281"})),(0,l.kt)("p",null,"Coalescing will be used before searching for a block and in ",(0,l.kt)("inlineCode",{parentName:"p"},"os_realloc()")," to expand the current block when possible."),(0,l.kt)("p",null,(0,l.kt)("em",{parentName:"p"},"Note"),": You might still need to split the block after coalesce."),(0,l.kt)("h4",{id:"find-best-block"},"Find Best Block"),(0,l.kt)("p",null,"Our aim is to reuse a free block with a size closer to what we need in order to reduce the number of future operations on it.\nThis strategy is called ",(0,l.kt)("strong",{parentName:"p"},"find best"),".\nOn every allocation we need to search the whole list of blocks and choose the best fitting free block."),(0,l.kt)("p",null,"In practice, it also uses a list of free blocks to avoid parsing all blocks, but this is out of the scope of the assignment."),(0,l.kt)("p",null,(0,l.kt)("em",{parentName:"p"},"Note"),": For consistent results, coalesce all adjacent free blocks before searching."),(0,l.kt)("h3",{id:"heap-preallocation"},"Heap Preallocation"),(0,l.kt)("p",null,"Heap is used in most modern programs.\nThis hints at the possibility of preallocating a relatively big chunk of memory (i.e. ",(0,l.kt)("strong",{parentName:"p"},"128 kilobytes"),") when the heap is used for the first time.\nThis reduces the number of future ",(0,l.kt)("inlineCode",{parentName:"p"},"brk()")," syscalls."),(0,l.kt)("p",null,"For example, if we try to allocate 1000 bytes we should first allocate a block of 128 kilobytes and then split it.\nOn future small allocations, we should proceed to split the preallocated chunk."),(0,l.kt)("p",null,(0,l.kt)("em",{parentName:"p"},"Note"),": Heap preallocation happens only once."),(0,l.kt)("h2",{id:"building-memory-allocator"},"Building Memory Allocator"),(0,l.kt)("p",null,"To build ",(0,l.kt)("inlineCode",{parentName:"p"},"libosmem.so"),", run ",(0,l.kt)("inlineCode",{parentName:"p"},"make")," in the ",(0,l.kt)("inlineCode",{parentName:"p"},"src/")," directory:"),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../mem-alloc$ cd src/\nstudent@os:~/.../mem-alloc/src$ make\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o osmem.o osmem.c\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o ../utils/printf.o ../utils/printf.c\ngcc -shared -o libosmem.so osmem.o helpers.o ../utils/printf.o\n")),(0,l.kt)("h2",{id:"testing-and-grading"},"Testing and Grading"),(0,l.kt)("p",null,"The testing is automated and performed with the ",(0,l.kt)("inlineCode",{parentName:"p"},"run-tests.py")," script from the ",(0,l.kt)("inlineCode",{parentName:"p"},"tests/")," directory."),(0,l.kt)("p",null,"Before running ",(0,l.kt)("inlineCode",{parentName:"p"},"run-tests.py"),", you first have to build ",(0,l.kt)("inlineCode",{parentName:"p"},"libosmem.so")," in the ",(0,l.kt)("inlineCode",{parentName:"p"},"src/")," directory and generate the test binaries in ",(0,l.kt)("inlineCode",{parentName:"p"},"tests/snippets"),".\nYou can do so using the all-in-one ",(0,l.kt)("inlineCode",{parentName:"p"},"Makefile")," rule from ",(0,l.kt)("inlineCode",{parentName:"p"},"tests/"),": ",(0,l.kt)("inlineCode",{parentName:"p"},"make check"),"."),(0,l.kt)("pre",null,(0,l.kt)("code",{parentName:"pre",className:"language-console"},"student@os:~/.../mem-alloc$ cd tests/\nstudent@os:~/.../mem-alloc/tests$ make check\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o osmem.o osmem.c\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o helpers.o helpers.c\ngcc -fPIC -Wall -Wextra -g -I../utils -c -o ../utils/printf.o ../utils/printf.c\n[...]\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-all snippets/test-all.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-arrays snippets/test-calloc-arrays.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-block-reuse snippets/test-calloc-block-reuse.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce-big snippets/test-calloc-coalesce-big.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-coalesce snippets/test-calloc-coalesce.c -L../src -losmem\ngcc -I../utils -fPIC -Wall -Wextra -g -o snippets/test-calloc-expand-block snippets/test-calloc-expand-block.c -L../src -losmem\n[...]\ntest-malloc-no-preallocate ........................ passed ... 2\ntest-malloc-preallocate ........................ passed ... 3\ntest-malloc-arrays ........................ passed ... 5\ntest-malloc-block-reuse ........................ passed ... 3\ntest-malloc-expand-block ........................ passed ... 2\ntest-malloc-no-split ........................ passed ... 2\ntest-malloc-split-one-block ........................ passed ... 3\ntest-malloc-split-first ........................ passed ... 2\ntest-malloc-split-last ........................ passed ... 2\ntest-malloc-split-middle ........................ passed ... 3\ntest-malloc-split-vector ........................ passed ... 2\ntest-malloc-coalesce ........................ passed ... 3\ntest-malloc-coalesce-big ........................ passed ... 3\ntest-calloc-no-preallocate ........................ passed ... 1\ntest-calloc-preallocate ........................ passed ... 1\ntest-calloc-arrays ........................ passed ... 5\ntest-calloc-block-reuse ........................ passed ... 1\ntest-calloc-expand-block ........................ passed ... 1\ntest-calloc-no-split ........................ passed ... 1\ntest-calloc-split-one-block ........................ passed ... 1\ntest-calloc-split-first ........................ passed ... 1\ntest-calloc-split-last ........................ passed ... 1\ntest-calloc-split-middle ........................ passed ... 1\ntest-calloc-split-vector ........................ passed ... 2\ntest-calloc-coalesce ........................ passed ... 2\ntest-calloc-coalesce-big ........................ passed ... 2\ntest-realloc-no-preallocate ........................ passed ... 1\ntest-realloc-preallocate ........................ passed ... 1\ntest-realloc-arrays ........................ passed ... 3\ntest-realloc-block-reuse ........................ passed ... 3\ntest-realloc-expand-block ........................ passed ... 2\ntest-realloc-no-split ........................ passed ... 3\ntest-realloc-split-one-block ........................ passed ... 3\ntest-realloc-split-first ........................ passed ... 3\ntest-realloc-split-last ........................ passed ... 3\ntest-realloc-split-middle ........................ passed ... 2\ntest-realloc-split-vector ........................ passed ... 2\ntest-realloc-coalesce ........................ passed ... 3\ntest-realloc-coalesce-big ........................ passed ... 1\ntest-all ........................ passed ... 5\n\nTotal: 90/100\n")),(0,l.kt)("p",null,(0,l.kt)("strong",{parentName:"p"},"NOTE:")," By default, ",(0,l.kt)("inlineCode",{parentName:"p"},"run-test.py")," checks for memory leaks, which can be time-consuming.\nTo speed up testing, use the ",(0,l.kt)("inlineCode",{parentName:"p"},"-d")," flag or ",(0,l.kt)("inlineCode",{parentName:"p"},"make check-fast")," to skip memory leak checks, but remember to run ",(0,l.kt)("inlineCode",{parentName:"p"},"make check")," before submitting your assignment to ensure it meets all criteria."),(0,l.kt)("h3",{id:"debugging"},"Debugging"),(0,l.kt)("p",null,(0,l.kt)("inlineCode",{parentName:"p"},"run-tests.py")," uses ",(0,l.kt)("inlineCode",{parentName:"p"},"ltrace")," to capture all the libcalls and syscalls performed."),(0,l.kt)("p",null,"The output of ",(0,l.kt)("inlineCode",{parentName:"p"},"ltrace")," is formatted to show only top level library calls and nested system calls.\nFor consistency, the heap start and addresses returned by ",(0,l.kt)("inlineCode",{parentName:"p"},"mmap()")," are replaced with labels.\nEvery other address is displayed as ",(0,l.kt)("inlineCode",{parentName:"p"},"