-
Notifications
You must be signed in to change notification settings - Fork 12
Welcome
d3 is a package deployment and patch management system for OS X that enhances the Casper Suite from JAMF Software. It was created by Pixar Animation Studios.
d3 adds these capabilities and more to Casper's package handling:
- Automatic software updates on clients when new versions are released on the server
- Pre-release piloting of new packages
- Customizable slideshow presented during logout/reboot installs
- Installs and uninstalls are conditional on the exit status of pre-flight scripts
- Packages can be expired (auto-uninstalled) after a period of disuse
- Both the client and admin tools are command-line only and fully scriptable
- Admin command-line options allow integration with developer workflows and package-retrieval tools
d3 is written in Ruby and available as a github project and a Ruby gem called 'depot3'. It interfaces with Casper mostly via it's REST API using the JSS Ruby module. It also accesses the JSS's backend MySQL database directly to provide enhanced features.
d3 was written to meet Pixar's needs, in our relatively simple Casper environment. As such it might not be appropriate for all Casper users.
Our needs included:
- Fully usable from the command-line and scriptable.
- Ease-of-use for software developers to package, pilot, and deploy their wares.
- Automatic installation of software based on tags or group membership
- Automatic updating of installed software.
- The ability to pilot pre-release software, and disable auto-updating during pilot.
- Automatic removal of unused software
That said, part of the reason we've made it open-source is with the hopes that others will be able to improve it, making it useful for a wider range of Casper users.
d3 makes no attempt to automatically search and retrieve newly-available patches or updates from the internet. It works with packages you provide or create (d3 can build them for you!)
An eventual goal is to integrate a tool like AutoPkg for this purpose.
d3 will not work with Cloud instances of the JSS. It requires MySQL access to the JSS database.
It also requires that the the master distribution point be a regular fileshare distribution point, not a JDS. There is minimal support for a cloud-based distribution point for failover access to package installers (the Casper REST API has no access to cloud distribution point data)
Nearly 30 years ago, Pixar's original NFS-based software deployment system for Unix workstations was called "depot". When it had outgrown itself, a replacement based on RPM packages was created and called "depot2", or "d2" for short. When Mac OS X arrived at Pixar in 2002, the Mac team adopted many of the Linux team's existing tools, including d2.
d2 was useful as far it went, but Macs aren't Linux, and d2 couldn't handle the installation of .pkg's, especially those requiring a reboot. To deal with those, Pixar's Mac team created PuppyTime! - a system that could install .pkgs from a server, and if they needed a reboot, would queue them to be installed at the next logout.
During logout installation, something had to be displayed on the screen, or the users would think the machine was crashed, and might force-reboot. So puppytime would display a slideshow of cute puppies to let the users know what was up, and placate their mood.
By 2008 the original developer of d2 had left, and d2's sustainability for the Macs was waning. Also, the world of Mac Sys Admins was changing to become what it is today. In 2009 the Pixar Mac team starting looking at third-party tools that might replace d2.
Nothing seemed to be an exact fit for our needs, but the Casper Suite from JAMF Software seemed promising, and offered other tools very similar to our own home-grown Mac infrastructure. We realized that with a little customization, Casper could provide the features we wanted. The first version of d3 was created in 2010 to add those features.
The second version of d3, which utilised the newly-released Casper REST API, was presented at the 2012 JAMF Nation User Conference in Minneapolis. One of the first questions from the audience was "Is it open-sourced?", to which we had to say no.
Since then, work has been progressing on the third version of d3 with a goal of enhancing its features and refactoring the code-base for eventual open-source release. The first step towards that goal was the 2014 release of the ruby-jss, a Ruby module which provides quick-and-easy access to the Casper REST API.
ruby-jss was created specifically as the API connection for d3, but has become useful in it's own right for all of our access to the API. d3 itself took another 18 months before the first upload to github.
d3 is made of several parts.
The d3 command is the heart of the system. It provides:
- Pre-release piloting of new packages
- Manual package installation, uninstallation
- Queuing of packages requiring reboot
- Removal or installation of packages in the logout/reboot queue
- Automatic scoped installations
- Automatic updates
- Automatic expiration of unused packages
- Various kinds of reporting (packages available, packages installed, packages being piloted, etc)
The d3admin command allows administrators and developers to:
- Add packages to d3 for piloting
- Release pilot packages for deployment
- Edit package attributes and settings
- Delete packages
- View reports about packages & installations
- Interactively prompt for all options or...
- Accept all options from the commandline, for integration with development workflows.
Puppytime handles the installation of packages in the logout queue. It is triggered by a Casper logout policy, and displays a slideshow to the user while the installs are happening. After installation, the machine reboots in a customizable manner. The default images are a selection of cute puppies, but any images can be used.
Currently d3helper provides a method to display a notification to users that something has been added to the logout queue, and they should log out soon. Any other notification method can be used instead, as the notification is handled by a Casper policy. It also can be used to populate extention-attributes that allow enhanced reporting about client installs. d3helper may provide other uses in the future.
This background process is needed if you opt to use the expiration feature of d3. It registers with the OS for low-level notifications and simply records a timestamp in a plist every time any app comes to the foreground in the GUI. When 'd3 sync' is running, installed packages that are expirable are checked agains the plists to see if they've been in the foreground recently enough, and if not, they are uninstalled.
The core funtionality of d3 is in the D3 module, which makes it straightforward to write additional tools that interact with d3. See http://www.rubydoc.info/gems/depot3 for the module documentation.
The ruby-jss project was the first part of d3 to be released. It provides simplified access to the JSS REST API and underlies all of d3.
The following terms are fundamental to understanding any discussion of d3.
A basename in d3 is a word used to identify all packages that install any version of the same thing. Every package has a basename, and will share it with other packages that install the same thing. For example "transmogrifier" could refer to all packages in d3 that install any version of Transmogrifier.app.
When a basename alone is used to specify a package, it always refers to the currently live package (see Status below) for the basename.
Every package also has a version, which is the version of the thing installed by the package. A package that installs Transmogrifier might install version 2.1.2, while another package that installs Transmogrifier might install version 2.2.3. Versions can include any alpha-numeric characters, e.g. "12.5a3", but they can't include spaces, which will be converted to underscores.
A package's revision is an integer representing how many times a single version of a basename has been added to d3.
When Transmogrifier 2.2.3 is added to d3 for the first time, it's revision will be 1. Later, a problem may be discovered with the .pkg itself, and a new .pkg for version 2.2.3 might be created. When added to d3, the fixed .pkg will have revision 2.
The edition is a combination of basename, version, and revision of a package, joined by hyphens. It is a unique identifier for each package in d3.
For example, there might be three editions of the basename 'transmogrifier' in d3 at the same time: transmogrifier-2.1.2-1, transmogrifier-2.1.2-2, and transmogrifier-2.2.3-1. When specifying any package that isn't live (see Status below), the edition is used to identify it.
Only one edition of a basename can be installed at a time.
A Package in d3 is a package in Casper. It can be used as any other package in Casper, in policies and Casper Remote (but be careful with that!). When a package is added to d3 (possibly by importing from an existing Casper package), extra data about it is stored in the JSS, which allows for d3's enhanced features. Throughout this documentation the word "package" usually refers to a thing stored on the server. See Receipt below.
A receipt is the on-disk representation of a d3 Package installed onto a client computer. Receipts have basenames, editions, and status, just like packages. Only one receipt per basename can be installed at a time. d3 receipts are stored separately from JAMF receipts and Apple .pkg receipts. Throughout this documentation the word "package" usually refers to a thing stored on the server, "receipt" refers to a thing installed on a client computer.
As a package spends time in d3, its status will change. Here are the statuses and a brief description of each. Receipts update their status from their matching package at every sync. For details about the signifigance of each status, see the full discussion of packages and receipts.
When a package is first added to d3, it's considered to be 'in pilot'. Pilot packages are not yet approved for general deployment, but can be manually installed (i.e. 'piloted') on individual machines for testing purposes.
Once a package has been piloted and is ready for general deployment, it is released by making it 'live'.
There can only be one live package per basename at a time, and using a basename to refer to a package will always yield the currently live one (if any).
A package that was once live, but has been superceded by a newer edition of it's basename being made live.
A package that is older than the currently live one for its basename, but was never made live itself.
A package that is still listed in d3, but has been removed from Casper. Or, a receipt who's the matching package has status 'missing', or no longer exists in d3.
Some packages require a reboot, so auto-installing them can be problematic.
When d3 installs a package that needs a reboot, it doesn't actually install it, but adds its edition to the "puppy queue" - a list of packages awaiting logout before they are installed.
When a package is added to the puppy queue, a policy is potentially triggered which can notify the user to log out as soon as possible.
When logout happens, a logout policy is execute which runs puppytime.
Puppytime examines the queue, and installs any packages listed. While doing so, it displays a slideshow - originally of cute puppies - to let the user know the machine isn't hung.
The default images that come with d3 are still puppies, but you can use any images you'd like.
When the installs are complete, puppytime will either:
- reboot the machine with
shutdown -r now
or
- run a policy which will reboot the machine (and possibly do other things first)
Check out some basic vocabulary for learning how d3 uses words like 'sync', 'live', 'edition' and so on.
For documentation of d3, see the Client page and the d3 help output.
-
Install the currently live edition of a package
% sudo d3 install transmogrifier Installing: transmogrifier-2.0-4 (live) Done installing transmogrifier-2.0-4 -
Install an un-released edition for piloting. In this case, the pilot package is set to remove the previously installed edition before installing.
% sudo d3 install transmogrifier-2.0-5 Updating transmogrifier-2.0-4(live) to transmogrifier-2.0-5(pilot) Uninstalling transmogrifier-2.0-4 Done, uninstalled transmogrifier-2.0-4 Installing: transmogrifier-2.0-5 (pilot) Done installing transmogrifier-2.0-4 -
List currently installed packages
% sudo d3 list-installed #------' ' next, 'b' prev, 'q' exit, 'h' help ------ # All packages installed by d3 (not-uninstallable<, frozen^) # Basename Vers-Rev Status Installed By #---------------------------------------------------------------------------- boxedit 3.0.18-3 live 2015-05-20 auto-installed caspersuite 9.82-1 live 2016-02-10 auto-installed chrisltest 3.0-6 live 2016-04-02 chrisl cocoadialog 3.0b7-1 live 2012-09-04 auto-installed transmogrifier 2.0-5 pilot 2015-03-08 chrislMost d3 actions have short versions, such as 'li' for 'list-installed', and 'i' for 'install'.
When d3 and d3admin display lists and reports that are longer than your terminal window, the output is piped thru
less. -
Perform all automated tasks
% sudo d3 sync Starting sync Updating receipts Checking for invalid puppies in the queue Checking for updates to installed packages Updating transmogrifier-2.0-5 (pilot) to transmogrifier-2.0-6 (live) Checking for new packages to auto-install Starting expiration check Finished syncSyncing is usually done at regular intervals by a launchd job or Casper policy. However it can be run manually at any time to get a client immediately up-to-date. The sync command performs these tasks:
- Update the local receipts with any relevant changes from the server packages
- Remove any pending logout installs from the puppy queue if the package is no longer available.
- Auto-update any installed packages if an update has been made live (and the receipt isn't frozen)
- Auto-install new packages that are in-scope
- Expire any packages that haven't been brought to the foreground within the expiration period.
For documentation of d3admin, see the Admin page and the d3admin help output.
-
Add a new package to d3 interactively
This will present a menu of options for defining the new package. If a previous edition is already in d3, it will be used for the default values of the new one.
% d3admin add transmogrifier --walkthru ------------------------------------ Adding pilot d3 package 'transmogrifier-2.0-7' with values inherited from 'transmogrifier-2.0-6' ------------------------------------ 1) Basename: transmogrifier 2) Version: 2.0 3) Revision: 7 4) JSS Package Name: transmogrifier-2.0-7.pkg 5) Description: ---- This is a descriptive description it describes this package in great depth. ---- 6) Dist. Point Filename: transmogrifier-2.0-7.pkg 7) Category: testing 8) Limited to OS's: >=10.10.x 9) Limited to CPU type: none 10) Needs Reboot: false 11) Uninstallable: true 12) Uninstalls older installs: true 13) Installation prohibited by processes matching: Safari 14) Auto installed for groups: standard 15) Not installed for groups: byod 16) Pre-install script: transmogrifier-install 17) Post-install script: transmogrifier-install 18) Pre-uninstall script: transmogrifier-uninstall 19) Post-uninstall script: transmogrifier-uninstall 20) Expration: 30 21) Expration Path: /Applications/Transmogrifier.app/Contents/MacOS/transmogrifier 22) Source path: /Users/Shared/Transmogrifier.pkg Which to change? (1-22, 'x' = done, ^c=cancel): x ***************************************** Ahoy there! You are about to: Create a new package 'transmogrifier-2.0-7' in d3 with settings shown above. ***************************************** Are you SURE? (y/n): y Saving new pilot transmogrifier-2.0-7to the server... Indexing... Uploading to the Master Distribution Point... Done! To pilot it, run 'sudo d3 install transmogrifier-2.0-7' on a test machine. To make it live, run 'd3admin live transmogrifier-2.0-7' on your machine. -
Add a new package to d3 with command-line options (split to multi-line for clarity)
Not all options need to be provided. Those not provided will either inherit from the previous edition or use default values.
% d3admin add transmogrifier \ --source-path /Users/Shared/Transmogrifier.pkg \ --description 'this is a new description' \ --pre-install 'better pre-install script' \ --auto-groups standard \ --excl-groups special-macs \ --oses '>=10.10.1' \ --remove-first \ --category test-apps \ --expiration 30 \ --expiration-path /Applications/Transmogrifier.app/Contents/MacOS/transmogrifier -
Make a package live
% d3admin live transmogrifier-2.0-7 ***************************************** Ahoy there! You are about to: Make transmogrifier-2.0-7 live for basename 'chrisltest' ***************************************** Are you SURE? (y/n): y Done 'transmogrifier-2.0-7' is now live. Starting auto-clean of old packages for 'chrisltest' Keeping 3 deprecated packages. Keeping most recent pilot packages as skipped. Finished auto-clean of old packages for 'chrisltest' Done! New installs of basename 'transmogrifier' will get 2.0-7 Existing installs will be updated at the next d3 sync.Once live, the package will install automatically when a client runs
d3 syncif:- An older edition of the same basename is installed (auto updating)
- The client is in one of the "auto-install" computer groups listed for the package. (scoped auto-installs)
-
Edit the settings of an existing package interactively
d3admin edit transmogrifier-20-7 -wThis creates a menu similar to the above. -w is the short version of --walkthru.
Edits can be made non-interactively by providing all new values on the commandline.
-
searching packages on the server
The 'search' action lists packages on the server. Without any targets, it lists all packages.
% d3admin search transmogrifier # All 'transmogrifier' packages in d3 # Edition Status Added By Released By #----------------------------------------------------------------------- transmogrifier-1.6-1 deprecated 2015-10-08 barney 2015-10-08 barney transmogrifier-1.7-1 deprecated 2015-11-08 barney 2015-11-08 fred transmogrifier-1.8-2 deprecated 2015-12-14 wilma 2015-12-20 wilma transmogrifier-2.0-1 skipped 2016-03-01 barney - - transmogrifier-2.0-2 skipped 2016-03-03 barney - - transmogrifier-2.0-3 skipped 2016-03-04 barney - - transmogrifier-2.0-4 skipped 2016-03-04 barney - - transmogrifier-2.0-5 skipped 2016-03-09 chrisl - - transmogrifier-2.0-6 skipped 2016-03-09 chrisl - - transmogrifier-2.0-7 live 2016-03-10 chrisl 2016-03-16 chrisl transmogrifier-2.1-1 pilot 2016-03-29 wilma - - transmogrifier-2.1-2 pilot 2016-03-31 wilma - - -
report receipts on client computers
The 'report' action will generate reports about what's installed on clients, as of the last 'jamf recon'.
% d3admin report transmogrifier #------' ' next, 'b' prev, 'q' exit, 'h' help ------ # All computers with 'transmogrifier' receipts # Computer User Edition Status As_of Frozen Installed By #---------------------------------------------------------------------------------------------- ale fred transmogrifier-2.0-7 live 2016-04-01 - 2016-03-16 auto-installed wine wilma transmogrifier-1.8-2 deprecated 2016-04-01 frozen 2015-12-20 auto-installed whiskey barney transmogrifier-2.1-1 pilot 2016-04-01 - 2016-03-29 wilma tequila betty transmogrifier-2.1-2 pilot 2016-04-01 - 2016-03-31 wilma gin pebbles transmogrifier-345-3 live 2016-03-31 - 2016-03-16 metro vodka bambam transmogrifier-345-3 live 2016-04-01 - 2016-03-16 auto-installed rum dino transmogrifier-2.0-7 live 2016-04-01 - 2016-03-16 auto-installed [...]
As with d3, all actions and options for d3admin have short versions, and lists & reports are displayed with less if they are longer than your terminal window.
Hopefully the documentation is enough to get you going. I'm happy to provide as much assistance as I can, but please be patient if I'm unable to respond quickly. The #d3 channel in Slack exists for general discussion, questions, and mutual assistance. Remember, this is the first version released outside of our environment.
- Macadmins Slack Channel (@glenfarclas17)
Copyright 2016 Pixar
Licensed under the Apache License, Version 2.0 (the "Apache License") with the following modification; you may not use d3 except in compliance with the Apache License and the following modification to it:
Section 6. Trademarks. is deleted and replaced with:
6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor and its affiliates, except as required to comply with Section 4(c) of the License and to reproduce the content of the NOTICE file.
You may obtain a copy of the Apache License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software distributed under the Apache License with the above modification is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the Apache License for the specific language governing permissions and limitations under the Apache License.