diff --git a/README.md b/README.md index bdbf286..b4aa0a0 100644 --- a/README.md +++ b/README.md @@ -159,7 +159,7 @@ fs.builder(state="exists", path="testfile", mode="a=rX", group="sean") Produces the following status output: ``` -=> run(command=rm -rf testdir, shell=True, ignore_failures=False, change=True) +=> run(command=rm -rf testdir, shell=True, ignore_failure=False, change=True) ==> mkdir(path=testdir, mode=a=rX,u+w, parents=True) ==# chmod(path=testdir, mode=493) => builder(path=testdir, mode=a=rX,u+w, state=directory) diff --git a/docs/getting_started.md b/docs/getting_started.md index 8b11d45..662cab9 100644 --- a/docs/getting_started.md +++ b/docs/getting_started.md @@ -49,16 +49,16 @@ You can create a test "skeleton" playbook called "my-test-playbook" by running ` new-uplaybook my-test-playbook`: $ up new-uplaybook my-test-playbook - =! exists(dst=my-test-playbook, ignore_failures=True) (failure ignored) + =! exists(dst=my-test-playbook, ignore_failure=True) (failure ignored) => mkdir(dst=my-test-playbook, parents=True) =# cd(dst=my-test-playbook) => cp(dst=playbook, src=playbook.j2, template=True, template_filenames=True, recursive=True) (Contents) =# cd(dst=/home/sean/projects/uplaybook) >> *** Starting handler: git_init =# cd(dst=my-test-playbook) - => run(command=git init, shell=True, ignore_failures=False, change=True) + => run(command=git init, shell=True, ignore_failure=False, change=True) Initialized empty Git repository in /home/sean/projects/uplaybook/my-test-playbook/.git/ - => run(command=git add ., shell=True, ignore_failures=False, change=True) + => run(command=git add ., shell=True, ignore_failure=False, change=True) =# cd(dst=/home/sean/projects/uplaybook) >> *** Done with handlers diff --git a/docs/playbooks/basics.md b/docs/playbooks/basics.md index 63c3200..e39e386 100644 --- a/docs/playbooks/basics.md +++ b/docs/playbooks/basics.md @@ -33,6 +33,8 @@ fs.mkdir(project_name).notify(initialize_project) This playbook creates a `my-test-project` playbook, and puts an empty `README` file in it, then initializes git. +## Being Declarative + The `notify()` makes this playbook declarative. A `notify()` sets up a function (known as a "handler") to be called later, **only when a task makes a change**. uPlaybook Tasks will determine if they change the system (in this case, if the directory already exists, @@ -42,7 +44,7 @@ directory is created. This is a useful trait of a playbook because you don't want to overwrite the `README`, or re-run the `git` commands if the project has already been created. -Output: +Output of the above playbook, if run twice, is: ```bash $ up my-test-playbook @@ -50,9 +52,9 @@ $ up my-test-playbook >> *** Starting handler: initialize_project =# cd(dst=my-test-project) => mkfile(dst=README) -=> run(command=git init, shell=True, ignore_failures=False, change=True) +=> run(command=git init, shell=True, ignore_failure=False, change=True) Initialized empty Git repository in /home/sean/projects/uplaybook/my-test-project/.git/ -=> run(command=git add ., shell=True, ignore_failures=False, change=True) +=> run(command=git add ., shell=True, ignore_failure=False, change=True) =# cd(dst=/home/sean/projects/uplaybook) >> *** Done with handlers @@ -64,4 +66,163 @@ $ up my-test-playbook *** RECAP: total=2 changed=0 failure=0 ``` +Note the first run creates the directory, creates the README, and runs git. The second +run skips the `mkdir` (that's what the "=#" denotes: no change was made), and because of +that it does not run the handler. + +## Calling Tasks + +The most basic component of playbooks is calling tasks, such as `fs.mkdir` or +`core.run` above. `core` and `fs` are uPlaybook modules of "core functionality" and +"filesystem tasks" respectively. + +These tasks are the heart of uPlaybook. All uPlaybook tasks are declarative, as described +[above](#being-declarative). + +## Task Return()s + +Tasks return a object called `Return()`. This has some notable features: + +- It has a `notify()` method which registers a [handler](#handlers) if the task determines + it has changed the system. +- It can be checked to see if the task failed. This only applies to tasks that can ignore + failures, like [core.run](../tasks/core.md#uplaybook.core.run). For example: `if + not core.run("false", ignore_failur=False):` +- Some tasks can be used as context managers, see [fs.cd](../tasks/fs.md#uplaybook.fs.cd) +- Capture output: the `output` attribute stores output of the task, see + [core.run](../tasks/core.md#uplaybook.core.run) for an example. +- Extra data: The `extra` attribute stores additional information the task may return, see + for example (fs.stat)[/tasks/fs#uplaybook.fs.stat] stores information about the file in + `extra`. + +## Extra Return Data + +Some tasks return extra data in the `extra` attribute of the return object. For example: + +```python +stats = fs.stat("{{project_dir}}/README") +print(f"Permissions: {stats.extra.perms:o}") +if stats.S_ISDIR: + core.fail(msg="The README is a directory, that's unexpected!") +``` + +## Ignoring Failures + +Some tasks, such as [core.run](../tasks/core.md#uplaybook.core.run), take an `ignore_failure` +option for one-shot failure ignoring. + +There is also an "IgnoreFailure" context manager to ignore failures for a block of +tasks: + +```python +with core.IgnoreFailures(): + core.run("false") + if not mkdir("/root/fail"): + print("You are not root") +``` + +## Getting Help + +The `up` command-line can be used to get documentation on the uPlaybook tasks with the +`--up-doc` argument. For example: + + up --up-doc fs + [Displays a list of tasks in the "fs" module] + up --up-doc core + [Displays a list of tasks in the "core" module] + up --up-doc core.run + [Dislays documentation for the "core.run" task] + +## Handlers + +An idea taken from Ansible, handlers are functions that are called only if changes are +made to the system. They are deferred, either until the end of the playbook run, or until +(core.flush_handlers)[../tasks/core#uplaybook.core.handlers] is called. + +They are deferred so that multiple tasks can all register handlers, but only run them once +rather than running multiple times. For example, if you are installing multiple Apache +modules, and writing several configuration files, these all may "notify" the +"restart_apache" handler, but only run the handler once: + +```python +def restart_apache(): + core.run("systemctl restart apache2") +core.run("apt -y install apache2", creates="/etc/apache2").notify(restart_apache) +fs.cp(src="site1.conf.j2", dst="/etc/apache2/sites-enabled/site1.conf").notify(restart_apache) +fs.cp(src="site2.conf.j2", dst="/etc/apache2/sites-enabled/site2.conf").notify(restart_apache) +fs.cp(src="site3.conf.j2", dst="/etc/apache2/sites-enabled/site3.conf").notify(restart_apache) +core.flush_handlers() +# ensure apache is running +run("wget -O /dev/null http://localhost/") +``` + +## Arguments + +Playbooks can include arguments and options for customizing the playbook run. For +example: + + core.playbook_args( + core.Argument(name="playbook_name", + description="Name of playbook to create, creates directory of this name."), + core.Argument(name="git", default=False, type="bool", + description="Initialize git (only for directory-basd playbooks)."), + core.Argument(name="single-file", default=False, type="bool", + description="Create a single-file uplaybook rather than a directory."), + core.Argument(name="force", default=False, type="bool", + description="Reset the playbook back to the default if it " + "already exists (default is to abort if playbook already exists)."), + ) + +This set up an argument of "playbook_name" and options of "--git", "--single-file", and +"--force". + +These can be accessed as `ARGS.playbook_name`, `ARGS.git`, `ARGS.single_file`, etc... + +!!! Note "Note on dash in name" + + A dash in the argument name is converted to an underscore in the `ARGS` list. + +See [core.Argument](../tasks/core#uplaybook.core.Argument) for full documentation. + +## Item Lists + +Another idea taken from Ansible is looping over items. See +[core.Item](../tasks/core#uplaybook.core.Item) and +[fs.builder](../tasks/fs#uplaybook.fs.builder) for some examples on how to effectively use +item lists. + +Example: + + for item in [ + core.Item(dst="foo", action="directory", owner="nobody"), + core.Item(dst="bar", action="exists"), + core.Item(dst="/etc/apache2/sites-enabled/foo", notify=restart_apache), + ]: + fs.builder(**item) + +fs.builder is an incredibly powerful paradigm for managing the state on files and +directories on a system. + +## Keeping Playbooks Declarative + +You have the flexibility to determine whether to make your playbooks declarative or not. + +The benefits of declarative playbooks are that they can be updated and re-run to update +the system configuration, for "configuration as code" usage. For example: you could have +a playbook that sets up your user environment, or configures a web server. Rather than +updating the configurations directly, if your configuration is a playbook you can update +the playbook and then run it on system reinstallation, or across a cluster of systems. + +However, as you have the full power of Python at your command, you need to be aware of +whether you are trying to make a declarative playbook or not. + +For example, a playbook that creates scaffolding for a new project may not be something +you can re-run. Since scaffolding is a starting point for user customization, it may not +be possible or reasonable to re-run the playbook at a later time. In this case, you may +wish to detect a re-run, say by checking if the project directory already exists, and +abort the run. + +To make a declarative playbook, you need to ensure that all steps of the playbook, +including Python code, is repeatable when re-run. + diff --git a/examples/new-uplaybook/playbook b/examples/new-uplaybook/playbook index 329a9b5..ab2fb5f 100644 --- a/examples/new-uplaybook/playbook +++ b/examples/new-uplaybook/playbook @@ -17,7 +17,8 @@ core.playbook_args( core.Argument(name="single-file", default=False, type="bool", description="Create a single-file uplaybook rather than a directory."), core.Argument(name="force", default=False, type="bool", - description="Reset the playbook back to the default if it already exists (default is to abort if playbook already exists)."), + description="Reset the playbook back to the default if it already " + "exists (default is to abort if playbook already exists)."), ) if ARGS.git and ARGS.single_file: diff --git a/mkdocs.yml b/mkdocs.yml index 2bc0430..5f09fb5 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -14,12 +14,13 @@ extra_css: extra_javascript: - assets/versions.js markdown_extensions: +- toc: + permalink: true + toc_depth: 3 - markdown_include.include - codehilite: css_class: highlight - admonition -- toc: - permalink: true - pymdownx.superfences repo_url: https://github.com/linsomniac/uplaybook site_name: uPlaybook - Declarative System/Project Setup diff --git a/tests/test_basics/basics.pb b/tests/test_basics/basics.pb index a612353..2350ebb 100644 --- a/tests/test_basics/basics.pb +++ b/tests/test_basics/basics.pb @@ -11,7 +11,7 @@ with fs.cd(dst="testdir"): core.run(command="date") r = core.run(command="true") assert r -r = core.run(command="false", ignore_failures=True) +r = core.run(command="false", ignore_failure=True) assert not r var = "bar" diff --git a/uplaybook/__init__.py b/uplaybook/__init__.py index a788c63..461bde9 100644 --- a/uplaybook/__init__.py +++ b/uplaybook/__init__.py @@ -17,15 +17,3 @@ up_context = internals.up_context ARGS = up_context.context["ARGS"] - - -class IgnoreFailure: - """A context-manager to ignore failures in wrapped tasks""" - - def __enter__(self): - up_context.ignore_failure_count += 1 - return self - - def __exit__(self, exc_type, exc_value, traceback) -> None: - up_context.ignore_failure_count -= 1 - assert up_context.ignore_failure_count >= 0 diff --git a/uplaybook/core.py b/uplaybook/core.py index 36e2edf..568f66a 100644 --- a/uplaybook/core.py +++ b/uplaybook/core.py @@ -26,6 +26,26 @@ import re +class IgnoreFailure: + """A context-manager to ignore failures in wrapped tasks. + + Example: + + with core.IgnoreFailures(): + core.run("false") + if not core.mkdir("/root/failure"): + print("You are not root") + """ + + def __enter__(self): + up_context.ignore_failure_count += 1 + return self + + def __exit__(self, exc_type, exc_value, traceback) -> None: + up_context.ignore_failure_count -= 1 + assert up_context.ignore_failure_count >= 0 + + class Item(dict): """ An (ansible-like) item for processing in a playbook (a file, directory, user...) @@ -173,7 +193,7 @@ def render(s: TemplateStr) -> str: def run( command: TemplateStr, shell: bool = True, - ignore_failures: bool = False, + ignore_failure: bool = False, change: bool = True, creates: Optional[TemplateStr] = None, ) -> object: @@ -186,7 +206,7 @@ def run( shell: If False, run `command` without a shell. Safer. Default is True: allows shell processing of `command` for things like output redirection, wildcard expansion, pipelines, etc. (optional, bool) - ignore_failures: If True, do not treat non-0 return code as a fatal failure. + ignore_failure: If True, do not treat non-0 return code as a fatal failure. This allows testing of return code within playbook. (optional, bool) change: By default, all shell commands are assumed to have caused a change to the system and will trigger notifications. If False, this `command` @@ -209,7 +229,7 @@ def run( print(f"Current date/time: {{r.output}}") print(f"Return code: {{r.extra.returncode}}") - if core.run(command="grep -q ^user: /etc/passwd", ignore_failures=True, change=False): + if core.run(command="grep -q ^user: /etc/passwd", ignore_failure=True, change=False): print("User exists") ``` @@ -233,9 +253,9 @@ def run( failure=failure, output=p.stdout.rstrip(), extra=extra, - ignore_failure=ignore_failures, + ignore_failure=ignore_failure, raise_exc=Failure(f"Exit code {p.returncode}") - if failure and not ignore_failures + if failure and not ignore_failure else None, ) @@ -511,7 +531,7 @@ def grep( path: TemplateStr, search: TemplateStr, regex: bool = True, - ignore_failures: bool = True, + ignore_failure: bool = True, ) -> object: """ Look for `search` in the file `path` @@ -520,7 +540,7 @@ def grep( path: File location to look for a match in. (templateable) search: The string (or regex) to look for. (templateable) regex: Do a regex search, if False do a simple string search. (bool, default=True) - ignore_failures: If True, do not treat file absence as a fatal failure. + ignore_failure: If True, do not treat file absence as a fatal failure. (optional, bool, default=True) Examples: @@ -546,8 +566,8 @@ def grep( return Return( changed=False, failure=True, - ignore_failure=ignore_failures, - raise_exc=Failure("No match found") if not ignore_failures else None, + ignore_failure=ignore_failure, + raise_exc=Failure("No match found") if not ignore_failure else None, ) diff --git a/uplaybook/fs.py b/uplaybook/fs.py index ad6c60a..0c00ab6 100644 --- a/uplaybook/fs.py +++ b/uplaybook/fs.py @@ -707,14 +707,14 @@ def builder( @template_args def exists( dst: TemplateStr, - ignore_failures: bool = True, + ignore_failure: bool = True, ) -> object: """ Does `dst` exist? Args: dst: File location to see if it exists. (templateable). - ignore_failures: If True, do not treat file absence as a fatal failure. + ignore_failure: If True, do not treat file absence as a fatal failure. (optional, bool, default=True) Examples: @@ -733,8 +733,8 @@ def exists( return Return( changed=False, failure=True, - ignore_failure=ignore_failures, + ignore_failure=ignore_failure, raise_exc=Failure(f"File does not exist: {dst}") - if not ignore_failures + if not ignore_failure else None, )