Merge pull request #1386 from trapexit/docs

Rework mkdocs based documentation

mkdocs/README.md

@@ -3,13 +3,14 @@
 ## Getting started
 
 ```bash
-python3 -m venv env
-source env/bin/activate
-pip3 install -r requirements.txt
+python3 -m venv venv
+source venv/bin/activate
+pip3 install --no-cache-dir mkdocs mkdocs-material pymdown-extensions
 mkdocs serve
 ```
 
 ## References
 
-- https://squidfunk.github.io/mkdocs-material/
-- https://www.mkdocs.org/
+* https://www.mkdocs.org/
+* https://squidfunk.github.io/mkdocs-material/
+

mkdocs/docs/pages/documentation/benchmarking.md → mkdocs/docs/benchmarking.md

@@ -1,4 +1,4 @@
-# BENCHMARKING
+# Benchmarking
 
 Filesystems are complicated. They do many things and many of those are
 interconnected. Additionally, the OS, drivers, hardware, etc. can all

mkdocs/docs/config/branches.md

@@ -0,0 +1,54 @@
+# branches
+
+The 'branches' argument is a colon (':') delimited list of paths to be
+pooled together. It does not matter if the paths are on the same or
+different filesystems nor does it matter the filesystem type (within
+reason). Used and available space will not be duplicated for paths on
+the same filesystem and any features which aren't supported by the
+underlying filesystem (such as file attributes or extended attributes)
+will return the appropriate errors.
+
+Branches currently have two options which can be set. A type which
+impacts whether or not the branch is included in a policy calculation
+and a individual minfreespace value. The values are set by prepending
+an `=` at the end of a branch designation and using commas as
+delimiters. Example: `/mnt/drive=RW,1234`
+
+### branch mode
+
+- RW: (read/write) - Default behavior. Will be eligible in all policy
+  categories.
+- RO: (read-only) - Will be excluded from `create` and `action`
+  policies. Same as a read-only mounted filesystem would be (though
+  faster to process).
+- NC: (no-create) - Will be excluded from `create` policies. You can't
+  create on that branch but you can change or delete.
+
+### minfreespace
+
+Same purpose and syntax as the global option but specific to the
+branch. If not set the global value is used.
+
+### globbing
+
+To make it easier to include multiple branches mergerfs supports
+[globbing](http://linux.die.net/man/7/glob). **The globbing tokens
+MUST be escaped when using via the shell else the shell itself will
+apply the glob itself.**
+
+```
+# mergerfs /mnt/hdd\*:/mnt/ssd /media
+```
+
+The above line will use all mount points in /mnt prefixed with **hdd**
+and **ssd**.
+
+To have the pool mounted at boot or otherwise accessible from related
+tools use `/etc/fstab`.
+
+```
+# <file system>        <mount point>  <type>    <options>             <dump>  <pass>
+/mnt/hdd*:/mnt/ssd    /media          mergerfs  minfreespace=16G      0       0
+```
+
+**NOTE:** The globbing is done at mount or when updated using the runtime API. If a new directory is added matching the glob after the fact it will not be automatically included.

mkdocs/docs/config/cache.md

@@ -0,0 +1,187 @@
+# caching
+
+## cache.files
+
+Controls how [page caching](https://en.wikipedia.org/wiki/Page_cache)
+works for mergerfs itself. Not the underlying filesystems.
+
+* `cache.files=off`: Disables page caching for mergerfs.
+* `cache.files=partial`: Enables page caching. Files are cached
+  while open.
+* `cache.files=full`: Enables page caching. Files are cached across
+  opens.
+* `cache.files=auto-full`: Enables page caching. Files are cached
+  across opens if mtime and size are unchanged since previous open.
+* `cache.files=per-process`: Enable page caching (equivalent to
+  `cache.files=partial`) only for processes whose 'comm' name matches
+  one of the values defined in cache.files.process-names. If the name
+  does not match the file open is equivalent to `cache.files=off`.
+
+
+Generally, enabling the page cache actually *harms*
+performance[^1]. In part because it can lead to buffer bloat due to
+the kernel caching both the underlying filesystem's file content as
+well as the file through mergerfs. However, if you want to confirm
+performance differences it is recommended that you perform some
+benchmark to confirm which option works best for your setup.
+
+Why then would you want to enable page caching if it consumes ~2x the
+RAM as normal and is on average slower? Because it is the only way to
+support
+[mmap](https://man7.org/linux/man-pages/man2/mmap.2.html). `mmap` is a
+way for programs to treat a file as if it is a contiguous RAM buffer
+which is regularly used by a number of programs such as those that
+leverage **sqlite3**. Despite `mmap` not being supported by all
+filesystems it is unfortunately common for software to not have an
+option to use regular file IO instead of `mmap`.
+
+The good thing is that in Linux v6.6[^2] and above FUSE can now
+transparently enable page caching when mmap is requested. This means
+it should be safe to set `cache.files=off`. However, on Linux v6.5 and
+below you will need to configure `cache.files` as you need.
+
+
+[^1]: This is not unique to mergerfs and affects all FUSE
+    filesystems. It is something that the FUSE community hopes to
+    investigate at some point but as of early 2025 there are a number
+    of major reworking going on with FUSE which needs to be finished
+    first.
+[^2]: [https://kernelnewbies.org/Linux_6.6#FUSE](https://kernelnewbies.org/Linux_6.6#FUSE)
+
+
+## cache.entry
+
+* `cache.entry=UINT`: Sets the number of seconds to cache
+  entry queries. Defaults to `1`.
+
+The kernel must ask mergerfs about the existence of files. The entry
+cache caches that those details which limits the number of requests
+sent to mergerfs.
+
+The risk of setting this value, as with most any cache, is related to
+[out-of-band](https://en.wikipedia.org/wiki/Out-of-band) changes. If
+the filesystems are changed outside mergerfs there is a risk of files
+which have been removed continuing to show as available. It will fail
+gracefully if a phantom file is actioned on in some way so there is
+little risk in setting the value much higher. Especially if there are
+no out-of-band changes.
+
+
+## cache.negative_entry
+
+* `cache.negative_entry=UINT`: Sets the number of seconds to cache
+  negative entry queries. Defaults to `1`.
+
+This is a cache for negative entry query responses. Such as when a
+file which does not exist is referenced.
+
+The risk of setting this value, as with most any cache, is related to
+[out-of-band](https://en.wikipedia.org/wiki/Out-of-band) changes. If
+the filesystems are changed outside mergerfs there is a risk of files
+which have been added outside mergerfs not appearing correctly till
+the cache entry times out if there had been a request for the same
+name within mergerfs which didn't exist. This is mostly an
+inconvenience.
+
+
+## cache.attr
+
+* `cache.attr=UINT`: Sets the number of seconds to cache file
+  attributes. Defaults to `1`.
+
+This is a cache for file attributes and metadata such as that which is
+collected by the
+[stat](https://man7.org/linux/man-pages/man2/stat.2.html) system call
+which is used when you run commands such as `find` or `ls -lh`. 
+
+As with other caches the risk of enabling the attribute cache is if
+changes are made to the file out-of-band there could be
+inconsistencies between the actual file and the cached details which
+could result in different issues depending on how the data is used. If
+the simultaneous writing of a file from inside and outside is unlikely
+then you should be safe. That said any simultaneous, uncoordinated
+manipulation of a file can lead to unexpected results.
+
+
+## cache.statfs
+
+* `cache.statfs=UINT`: Sets the number of seconds to cache `statfs`
+  calls used by policies. Defaults to `0`.
+
+A number of policies require looking up the available space of the
+branches being considered. This is accomplished by calling
+[statfs](https://man7.org/linux/man-pages/man2/statfs.2.html). This
+call however is a bit expensive so this cache reduces the overhead by
+limiting how often the calls are actually made.
+
+This will mean that if the available space of branches changed
+somewhat rapidly there is a risk of `create` or `mkdir` calls made
+within the timeout period ending up on the same branch. This however
+should even itself out over time.
+
+
+## cache.symlinks
+
+* `cache.symlinks=BOOL`: Enable kernel caching of symlink
+  values. Defaults to `false`.
+
+As of Linux v4.20 there is an ability to cache the value of symlinks
+so that the kernel does not need to make a request to mergerfs every
+single time a
+[readlink](https://man7.org/linux/man-pages/man2/readlink.2.html)
+request is made. While not a common usage pattern, if software very
+regularly queries symlink values, the use of this cache could
+significantly improve performance.
+
+mergerfs will not error if the kernel used does not support symlink
+caching.
+
+As with other caches the main risk in enabling it is if you are
+manipulating symlinks from both within and without the mergerfs
+mount. Should the value be changed outside of mergerfs then it will
+not be reflected in the mergerfs mount till the cached value is
+invalidated.
+
+
+## cache.readdir
+
+* `cache.readdir=BOOL`: Enable kernel caching of readdir
+  results. Defaults to `false`.
+
+As of Linux v4.20 it supports readdir caching. This can have a
+significant impact on directory traversal. Especially when combined
+with entry (`cache.entry`) and attribute (`cache.attr`) caching. If
+the kernel doesn't support readdir caching setting the option to true
+has no effect. This option is configurable at runtime via xattr
+user.mergerfs.cache.readdir.
+
+## cache.writeback
+
+* `cache.writeback=BOOL`: Enable writeback cache. Defaults to `false`.
+
+When `cache.files` is enabled the default is for it to perform
+writethrough caching. This behavior won't help improve performance as
+each write still goes one for one through the filesystem. By enabling
+the FUSE writeback cache small writes *may* be aggregated by the
+kernel and then sent to mergerfs as one larger request. This can
+greatly improve the throughput for apps which write to files
+inefficiently. The amount the kernel can aggregate is limited by the
+size of a FUSE message. Read the fuse_msg_size section for more
+details.
+
+There is a side effect as a result of enabling writeback
+caching. Underlying files won't ever be opened with O_APPEND or
+O_WRONLY. The former because the kernel then manages append mode and
+the latter because the kernel may request file data from mergerfs to
+populate the write cache. The O_APPEND change means that if a file is
+changed outside of mergerfs it could lead to corruption as the kernel
+won't know the end of the file has changed. That said any time you use
+caching you should keep from writing to the same file outside of
+mergerfs at the same time.
+
+Note that if an application is properly sizing writes then writeback
+caching will have little or no effect. It will only help with writes
+of sizes below the FUSE message size (128K on older kernels, 1M on
+newer). Even then its effectiveness might not be great. Given the side
+effects of enabling this feature it is recommended that its benefits
+be proved out with benchmarks.

mkdocs/docs/config/deprecated_options.md

@@ -0,0 +1,21 @@
+# Deprecated Options
+
+These are old, deprecated options which may no longer have any
+function or have been replaced.
+
+* **direct_io**: Bypass page cache. Use `cache.files=off`
+  instead.
+* **kernel_cache**: Do not invalidate data cache on file open. Use
+  `cache.files=full` instead.
+* **auto_cache**: Invalidate data cache if file mtime or
+  size change. Use `cache.files=auto-full` instead. (default: false)
+* **async_read**: Perform reads asynchronously. Use
+  `async_read=true` instead.
+* **sync_read**: Perform reads synchronously. Use
+  `async_read=false` instead.
+* **splice_read**: Does nothing.
+* **splice_write**: Does nothing.
+* **splice_move**: Does nothing.
+* **allow_other**: mergerfs v2.35.0 and above sets this FUSE option
+  automatically if running as root.
+* **use_ino**: Effectively replaced with `inodecalc`.

mkdocs/docs/config/export-support.md

@@ -0,0 +1,14 @@
+# export-support
+
+* `export-support=true|false`
+* Defaults to `true`.
+
+In theory, this flag should not be exposed to the end user. It is a
+low-level FUSE flag which indicates whether or not the kernel can send
+certain kinds of messages to it for the purposes of using it with
+NFS. mergerfs does support these messages but due to bugs and quirks
+found in the kernel and mergerfs this option is provided just in case
+it is needed for debugging.
+
+Given that this flag is set when the FUSE connection is first
+initiated it is not possible to change during run time.

mkdocs/docs/config/flush-on-close.md

@@ -0,0 +1,21 @@
+# flush-on-close
+
+By default, FUSE would issue a flush before the release of a file
+descriptor. This was considered a bit aggressive and a feature added
+to give the FUSE server the ability to choose when that happens.
+
+* `flush-on-close=always`
+* `flush-on-close=never`
+* `flush-on-close=opened-for-write`
+* Defaults to `opened-for-write`.
+
+For now it defaults to `opened-for-write` which is less aggressive
+than the behavior before this feature was added. It should not be a
+problem because the flush is really only relevant when a file is
+written to. Given flush is irrelevant for many filesystems in the
+future a branch specific flag may be added so only files opened on a
+specific branch would be flushed on close.
+
+## References
+
+* [https://lkml.kernel.org/linux-fsdevel/[email protected]/T/](https://lkml.kernel.org/linux-fsdevel/[email protected]/T/)

mkdocs/docs/config/follow-symlinks.md

@@ -0,0 +1,23 @@
+# follow-symlinks
+
+This feature, when enabled, will cause symlinks to be interpreted by
+mergerfs as their target.
+
+When there is a getattr/stat request for a file mergerfs will check if
+the file is a symlink and depending on the `follow-symlinks` setting
+will replace the information about the symlink with that of that which
+it points to.
+
+When unlink'ing or rmdir'ing the followed symlink it will remove the
+symlink itself and not that which it points to.
+
+* `follow-symlinks=never`: Behave as normal. Symlinks are treated as such.
+* `follow-symlinks=directory`: Resolve symlinks only which point to directories.
+* `follow-symlinks=regular`: Resolve symlinks only which point to regular files.
+* `follow-symlinks=all`: Resolve all symlinks to that which they point
+  to. Symlinks which do not point to anything are left as is.
+* Defaults to `never`.
+
+**WARNING:** This feature should be considered experimental. There
+might be edge cases yet found. If you find any odd behaviors please
+file a ticket on [github](https://github.com/trapexit/mergerfs/issues/new?assignees=&labels=bug%2C+investigating&projects=&template=bug_report.md&title=).

mkdocs/docs/config/func_readdir.md

@@ -0,0 +1,17 @@
+# func.readdir
+
+examples: `func.readdir=seq`, `func.readdir=cor:4`
+
+`readdir` has policies to control how it reads directory content.
+
+| Policy | Description |
+| ------ | ----------- |
+| seq    | "sequential" : Iterate sequentially over branches in the order defined in `branches`. This is the default and traditional behavior found prior to the readdir policy introduction. This will be increasingly slower as more branches are added to the pool. Especially if needing to wait for drives to spin up or network filesystems to respond. |
+| cosr   | "concurrent open, sequential read" : Concurrently open branch directories using a thread pool and process them in the order defined in `branches`. This keeps memory and CPU usage low while also reducing the time spent waiting on branches to respond. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cosr:N` where `N` is the number of threads. |
+| cor    | "concurrent open and read" : Concurrently open branch directories and immediately start reading their contents using a thread pool. This will result in slightly higher memory and CPU usage but reduced latency. Particularly when using higher latency / slower speed network filesystem branches. Unlike `seq` and `cosr` the order of files could change due the async nature of the thread pool. This should not be a problem since the order of files listed in not guaranteed. Number of threads defaults to the number of logical cores. Can be overwritten via the syntax `func.readdir=cor:N` where `N` is the number of threads. |
+
+Keep in mind that `readdir` mostly just provides a list of file names
+in a directory and possibly some basic metadata about said files. To
+know details about the files, as one would see from commands like
+`find` or `ls`, it is required to call `stat` on the file which is
+controlled by `fuse.getattr`.