ignore_failures -> ignore_failure, IgnoreFailure moved to core, lots …

…of docs enhancements

README.md

@@ -159,7 +159,7 @@ fs.builder(state="exists", path="testfile", mode="a=rX", group="sean")
 Produces the following status output:
 
 ```
-=> run(command=rm -rf testdir, shell=True, ignore_failures=False, change=True)
+=> run(command=rm -rf testdir, shell=True, ignore_failure=False, change=True)
 ==> mkdir(path=testdir, mode=a=rX,u+w, parents=True)
 ==# chmod(path=testdir, mode=493)
 => builder(path=testdir, mode=a=rX,u+w, state=directory)

docs/getting_started.md

@@ -49,16 +49,16 @@ You can create a test "skeleton" playbook called "my-test-playbook" by running `
 new-uplaybook my-test-playbook`:
 
     $ up new-uplaybook my-test-playbook
-    =! exists(dst=my-test-playbook, ignore_failures=True) (failure ignored)
+    =! exists(dst=my-test-playbook, ignore_failure=True) (failure ignored)
     => mkdir(dst=my-test-playbook, parents=True)
     =# cd(dst=my-test-playbook)
     => cp(dst=playbook, src=playbook.j2, template=True, template_filenames=True, recursive=True) (Contents)
     =# cd(dst=/home/sean/projects/uplaybook)
     >> *** Starting handler: git_init
     =# cd(dst=my-test-playbook)
-    => run(command=git init, shell=True, ignore_failures=False, change=True)
+    => run(command=git init, shell=True, ignore_failure=False, change=True)
     Initialized empty Git repository in /home/sean/projects/uplaybook/my-test-playbook/.git/
-    => run(command=git add ., shell=True, ignore_failures=False, change=True)
+    => run(command=git add ., shell=True, ignore_failure=False, change=True)
     =# cd(dst=/home/sean/projects/uplaybook)
     >> *** Done with handlers
 

docs/playbooks/basics.md

@@ -33,6 +33,8 @@ fs.mkdir(project_name).notify(initialize_project)
 This playbook creates a `my-test-project` playbook, and puts an empty `README` file in it,
 then initializes git.
 
+## Being Declarative
+
 The `notify()` makes this playbook declarative.  A `notify()` sets up a function (known as
 a "handler") to be called later, **only when a task makes a change**.  uPlaybook Tasks
 will determine if they change the system (in this case, if the directory already exists,
@@ -42,17 +44,17 @@ directory is created.
 This is a useful trait of a playbook because you don't want to overwrite the `README`, or
 re-run the `git` commands if the project has already been created.
 
-Output:
+Output of the above playbook, if run twice, is:
 
 ```bash
 $ up my-test-playbook
 => mkdir(dst=my-test-project, parents=True)
 >> *** Starting handler: initialize_project
 =# cd(dst=my-test-project)
 => mkfile(dst=README)
-=> run(command=git init, shell=True, ignore_failures=False, change=True)
+=> run(command=git init, shell=True, ignore_failure=False, change=True)
 Initialized empty Git repository in /home/sean/projects/uplaybook/my-test-project/.git/
-=> run(command=git add ., shell=True, ignore_failures=False, change=True)
+=> run(command=git add ., shell=True, ignore_failure=False, change=True)
 =# cd(dst=/home/sean/projects/uplaybook)
 >> *** Done with handlers
 
@@ -64,4 +66,163 @@ $ up my-test-playbook
 *** RECAP:  total=2 changed=0 failure=0
 ```
 
+Note the first run creates the directory, creates the README, and runs git.  The second
+run skips the `mkdir` (that's what the "=#" denotes: no change was made), and because of
+that it does not run the handler.
+
+## Calling Tasks
+
+The most basic component of playbooks is calling tasks, such as `fs.mkdir` or
+`core.run` above.  `core` and `fs` are uPlaybook modules of "core functionality" and
+"filesystem tasks" respectively.
+
+These tasks are the heart of uPlaybook.  All uPlaybook tasks are declarative, as described
+[above](#being-declarative).
+
+## Task Return()s
+
+Tasks return a object called `Return()`.  This has some notable features:
+
+- It has a `notify()` method which registers a [handler](#handlers) if the task determines
+  it has changed the system.
+- It can be checked to see if the task failed.  This only applies to tasks that can ignore
+  failures, like [core.run](../tasks/core.md#uplaybook.core.run).  For example: `if
+  not core.run("false", ignore_failur=False):`
+- Some tasks can be used as context managers, see [fs.cd](../tasks/fs.md#uplaybook.fs.cd)
+- Capture output: the `output` attribute stores output of the task, see
+  [core.run](../tasks/core.md#uplaybook.core.run) for an example.
+- Extra data: The `extra` attribute stores additional information the task may return, see
+  for example (fs.stat)[/tasks/fs#uplaybook.fs.stat] stores information about the file in
+  `extra`.
+
+## Extra Return Data
+
+Some tasks return extra data in the `extra` attribute of the return object.  For example:
+
+```python
+stats = fs.stat("{{project_dir}}/README")
+print(f"Permissions: {stats.extra.perms:o}")
+if stats.S_ISDIR:
+    core.fail(msg="The README is a directory, that's unexpected!")
+```
+
+## Ignoring Failures
+
+Some tasks, such as [core.run](../tasks/core.md#uplaybook.core.run), take an `ignore_failure`
+option for one-shot failure ignoring.
+
+There is also an "IgnoreFailure" context manager to ignore failures for a block of
+tasks:
+
+```python
+with core.IgnoreFailures():
+    core.run("false")
+    if not mkdir("/root/fail"):
+        print("You are not root")
+```
+
+## Getting Help
+
+The `up` command-line can be used to get documentation on the uPlaybook tasks with the
+`--up-doc` argument.  For example:
+
+    up --up-doc fs
+    [Displays a list of tasks in the "fs" module]
+    up --up-doc core
+    [Displays a list of tasks in the "core" module]
+    up --up-doc core.run
+    [Dislays documentation for the "core.run" task]
+
+## Handlers
+
+An idea taken from Ansible, handlers are functions that are called only if changes are
+made to the system.  They are deferred, either until the end of the playbook run, or until
+(core.flush_handlers)[../tasks/core#uplaybook.core.handlers] is called.
+
+They are deferred so that multiple tasks can all register handlers, but only run them once
+rather than running multiple times.  For example, if you are installing multiple Apache
+modules, and writing several configuration files, these all may "notify" the
+"restart_apache" handler, but only run the handler once:
+
+```python
+def restart_apache():
+    core.run("systemctl restart apache2")
+core.run("apt -y install apache2", creates="/etc/apache2").notify(restart_apache)
+fs.cp(src="site1.conf.j2", dst="/etc/apache2/sites-enabled/site1.conf").notify(restart_apache)
+fs.cp(src="site2.conf.j2", dst="/etc/apache2/sites-enabled/site2.conf").notify(restart_apache)
+fs.cp(src="site3.conf.j2", dst="/etc/apache2/sites-enabled/site3.conf").notify(restart_apache)
+core.flush_handlers()
+#  ensure apache is running
+run("wget -O /dev/null http://localhost/")
+```
+
+## Arguments
+
+Playbooks can include arguments and options for customizing the playbook run.  For
+example:
+
+    core.playbook_args(
+            core.Argument(name="playbook_name",
+                          description="Name of playbook to create, creates directory of this name."),
+            core.Argument(name="git", default=False, type="bool",
+                          description="Initialize git (only for directory-basd playbooks)."),
+            core.Argument(name="single-file", default=False, type="bool",
+                          description="Create a single-file uplaybook rather than a directory."),
+            core.Argument(name="force", default=False, type="bool",
+                          description="Reset the playbook back to the default if it "
+                          "already exists (default is to abort if playbook already exists)."),
+            )
+
+This set up an argument of "playbook_name" and options of "--git", "--single-file", and
+"--force".
+
+These can be accessed as `ARGS.playbook_name`, `ARGS.git`, `ARGS.single_file`, etc...
+
+!!! Note "Note on dash in name"
+
+    A dash in the argument name is converted to an underscore in the `ARGS` list.
+
+See [core.Argument](../tasks/core#uplaybook.core.Argument) for full documentation.
+
+## Item Lists
+
+Another idea taken from Ansible is looping over items.  See
+[core.Item](../tasks/core#uplaybook.core.Item) and
+[fs.builder](../tasks/fs#uplaybook.fs.builder) for some examples on how to effectively use
+item lists.
+
+Example:
+
+    for item in [
+            core.Item(dst="foo", action="directory", owner="nobody"),
+            core.Item(dst="bar", action="exists"),
+            core.Item(dst="/etc/apache2/sites-enabled/foo", notify=restart_apache),
+            ]:
+        fs.builder(**item)
+
+fs.builder is an incredibly powerful paradigm for managing the state on files and
+directories on a system.
+
+## Keeping Playbooks Declarative
+
+You have the flexibility to determine whether to make your playbooks declarative or not.
+
+The benefits of declarative playbooks are that they can be updated and re-run to update
+the system configuration, for "configuration as code" usage.  For example: you could have
+a playbook that sets up your user environment, or configures a web server.  Rather than
+updating the configurations directly, if your configuration is a playbook you can update
+the playbook and then run it on system reinstallation, or across a cluster of systems.
+
+However, as you have the full power of Python at your command, you need to be aware of
+whether you are trying to make a declarative playbook or not.
+
+For example, a playbook that creates scaffolding for a new project may not be something
+you can re-run.  Since scaffolding is a starting point for user customization, it may not
+be possible or reasonable to re-run the playbook at a later time.  In this case, you may
+wish to detect a re-run, say by checking if the project directory already exists, and
+abort the run.
+
+To make a declarative playbook, you need to ensure that all steps of the playbook,
+including Python code, is repeatable when re-run.
+
 <!-- vim: set tw=90: -->

examples/new-uplaybook/playbook

@@ -17,7 +17,8 @@ core.playbook_args(
         core.Argument(name="single-file", default=False, type="bool",
                       description="Create a single-file uplaybook rather than a directory."),
         core.Argument(name="force", default=False, type="bool",
-                      description="Reset the playbook back to the default if it already exists (default is to abort if playbook already exists)."),
+                      description="Reset the playbook back to the default if it already "
+                      "exists (default is to abort if playbook already exists)."),
         )
 
 if ARGS.git and ARGS.single_file:

mkdocs.yml

@@ -14,12 +14,13 @@ extra_css:
 extra_javascript:
 - assets/versions.js
 markdown_extensions:
+- toc:
+    permalink: true
+    toc_depth: 3
 - markdown_include.include
 - codehilite:
     css_class: highlight
 - admonition
-- toc:
-    permalink: true
 - pymdownx.superfences
 repo_url: https://github.com/linsomniac/uplaybook
 site_name: uPlaybook - Declarative System/Project Setup

tests/test_basics/basics.pb

@@ -11,7 +11,7 @@ with fs.cd(dst="testdir"):
 core.run(command="date")
 r = core.run(command="true")
 assert r
-r = core.run(command="false", ignore_failures=True)
+r = core.run(command="false", ignore_failure=True)
 assert not r
 
 var = "bar"

uplaybook/__init__.py

@@ -17,15 +17,3 @@
 
 up_context = internals.up_context
 ARGS = up_context.context["ARGS"]
-
-
-class IgnoreFailure:
-    """A context-manager to ignore failures in wrapped tasks"""
-
-    def __enter__(self):
-        up_context.ignore_failure_count += 1
-        return self
-
-    def __exit__(self, exc_type, exc_value, traceback) -> None:
-        up_context.ignore_failure_count -= 1
-        assert up_context.ignore_failure_count >= 0

uplaybook/core.py

@@ -26,6 +26,26 @@
 import re
 
 
+class IgnoreFailure:
+    """A context-manager to ignore failures in wrapped tasks.
+
+    Example:
+
+        with core.IgnoreFailures():
+            core.run("false")
+            if not core.mkdir("/root/failure"):
+                print("You are not root")
+    """
+
+    def __enter__(self):
+        up_context.ignore_failure_count += 1
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback) -> None:
+        up_context.ignore_failure_count -= 1
+        assert up_context.ignore_failure_count >= 0
+
+
 class Item(dict):
     """
     An (ansible-like) item for processing in a playbook (a file, directory, user...)
@@ -173,7 +193,7 @@ def render(s: TemplateStr) -> str:
 def run(
     command: TemplateStr,
     shell: bool = True,
-    ignore_failures: bool = False,
+    ignore_failure: bool = False,
     change: bool = True,
     creates: Optional[TemplateStr] = None,
 ) -> object:
@@ -186,7 +206,7 @@ def run(
         shell: If False, run `command` without a shell.  Safer.  Default is True:
              allows shell processing of `command` for things like output
              redirection, wildcard expansion, pipelines, etc. (optional, bool)
-        ignore_failures: If True, do not treat non-0 return code as a fatal failure.
+        ignore_failure: If True, do not treat non-0 return code as a fatal failure.
              This allows testing of return code within playbook.  (optional, bool)
         change: By default, all shell commands are assumed to have caused a change
              to the system and will trigger notifications.  If False, this `command`
@@ -209,7 +229,7 @@ def run(
     print(f"Current date/time: {{r.output}}")
     print(f"Return code: {{r.extra.returncode}}")
 
-    if core.run(command="grep -q ^user: /etc/passwd", ignore_failures=True, change=False):
+    if core.run(command="grep -q ^user: /etc/passwd", ignore_failure=True, change=False):
         print("User exists")
     ```
 
@@ -233,9 +253,9 @@ def run(
         failure=failure,
         output=p.stdout.rstrip(),
         extra=extra,
-        ignore_failure=ignore_failures,
+        ignore_failure=ignore_failure,
         raise_exc=Failure(f"Exit code {p.returncode}")
-        if failure and not ignore_failures
+        if failure and not ignore_failure
         else None,
     )
 
@@ -511,7 +531,7 @@ def grep(
     path: TemplateStr,
     search: TemplateStr,
     regex: bool = True,
-    ignore_failures: bool = True,
+    ignore_failure: bool = True,
 ) -> object:
     """
     Look for `search` in the file `path`
@@ -520,7 +540,7 @@ def grep(
         path: File location to look for a match in. (templateable)
         search: The string (or regex) to look for. (templateable)
         regex: Do a regex search, if False do a simple string search. (bool, default=True)
-        ignore_failures: If True, do not treat file absence as a fatal failure.
+        ignore_failure: If True, do not treat file absence as a fatal failure.
              (optional, bool, default=True)
 
     Examples:
@@ -546,8 +566,8 @@ def grep(
     return Return(
         changed=False,
         failure=True,
-        ignore_failure=ignore_failures,
-        raise_exc=Failure("No match found") if not ignore_failures else None,
+        ignore_failure=ignore_failure,
+        raise_exc=Failure("No match found") if not ignore_failure else None,
     )
 
 

uplaybook/fs.py

@@ -707,14 +707,14 @@ def builder(
 @template_args
 def exists(
     dst: TemplateStr,
-    ignore_failures: bool = True,
+    ignore_failure: bool = True,
 ) -> object:
     """
     Does `dst` exist?
 
     Args:
         dst: File location to see if it exists. (templateable).
-        ignore_failures: If True, do not treat file absence as a fatal failure.
+        ignore_failure: If True, do not treat file absence as a fatal failure.
              (optional, bool, default=True)
 
     Examples:
@@ -733,8 +733,8 @@ def exists(
     return Return(
         changed=False,
         failure=True,
-        ignore_failure=ignore_failures,
+        ignore_failure=ignore_failure,
         raise_exc=Failure(f"File does not exist: {dst}")
-        if not ignore_failures
+        if not ignore_failure
         else None,
     )