Installing & Automating AutoPkg with a Launch Daemon

UPDATE (1/12/15): Since AutoPkgr now supports recipe overrides I recommend using AutoPkgr over my headless script.

---

AutoPkg is a great tool for downloading and packaging software for distribution. It also integrates nicely with Munki. After testing AutoPkg I looked into methods to further automate the updating process. I came across Sean Kaiser’s blog detailing how he created a script and triggered it using a launch daemon. I took his script as an example and began to build my own, with a few extra tweaks.

My AutoPkg Wrapper triggers AutoPkg to run once daily at 8:30am, dynamically checking for updates to user recipe overrides and sending an email in the event of a new package.

Without further ado I will walk you through setting up AutoPkg and automating it with my AutoPkg Wrapper.

Note: If you haven’t already, it is important to install Munki and AutoPkg.

Configuring AutoPkg

First we need to add the main recipe repository to AutoPkg, we do this by opening Terminal and typing:

autopkg repo-add http://github.com/autopkg/recipes.git

Note: AutoPkg requires Git to be installed, if it is not installed you will be prompted to install it from Apple Software Update.

View fullsize

Next we need to set the location of our Munki repository:

defaults write com.github.autopkg MUNKI_REPO "/PATH/TO/MUNKI_REPO"

AutoPkg uses recipe overrides to override default or unspecified recipe attributes. My Munki repositories follow a consistent naming convention and therefore I create recipe overrides for every AutoPkg recipe used. Below are the criteria for packages added to my Munki repositories:

• Packages are placed directly into pkgs and not into subdirectories.
• Packages are named in lowercase (e.g. Firefox.dmg > firefox.dmg).
• Packages do not contain the developer’s name (e.g. googlechrome.dmg > chrome.dmg).
• Packages are initially added to the development catalog.
• The pkginfo display_name attribute is set to match the package name with proper case and spaces (e.g. flashplayer.dmg > Flash Player).
• The unattended_installs pkginfo attribute is disabled.
• The developer and category pkginfo attributes are set.

Optionally you can download and install my collection of AutoPkg recipe overrides from GitHub:

git clone https://github.com/Error-freeIT/AutoPkg-Recipe-Overrides.git ~/Library/AutoPkg/RecipeOverrides

With AutoPkg configured let’s test a recipe:

autopkg run -v AdobeFlashPlayer.munki

View fullsize

Automating AutoPkg

If that worked it’s time to automate AutoPkg, download and run the autopkgwrapper installer:

git clone https://github.com/Error-freeIT/AutoPkg-Wrapper.git /tmp/autopkgwrapper && cd /tmp/autopkgwrapper && sudo ./install.sh

The install script requires administrator privileges and therefore will prompt for a password, as it copies the script and launch daemon into place and opens the script in Nano for configuration.

Update the ACCOUNT_NAME value to match the account name (a.k.a. username) containing the recipe overrides and update the EMAIL_FROM and EMAIL_TO addresses.

Note: To save changes in the Nano text editor press control + X, type y and hit return.

View fullsize

That’s it! Once set up you will be emailed when new items are added to the Munki repository. Standard workflow would then involve testing the new software and if the new software is stable, simply add it to your production catalog.

Troubleshooting The AutoPkg Wrapper

This AutoPkg Wrapper was intended to only be run as a launch daemon (by root) to manually run the AutoPkg Wrapper type:

sudo "/Library/Scripts/AutoPkg Wrapper/autopkgwrapper.sh"

View fullsize

You can update your email settings by typing:

sudo nano "/Library/Scripts/AutoPkg Wrapper/autopkgwrapper.sh"

By default the installed launch daemon is set to run daily at 8:30am, if you want to further customise how often the script is run I recommend editing the launch daemon with Lingon X.

DeployStudio Rsync Backup & Restore User Data

The MacBook Air is a great laptop and thanks to its SSD (Solid-state Drive), it provides impressive read and write speeds. The only problem is that SSD storage is expensive and the approach of storing user data on a separate partition becomes unfeasible with the limited space. One site running a fleet of 64GB MacBook Airs found themselves manually backing up user data before reimaging, then manually restoring the user's data.

To help improve their workflow, I investigated methods of backing up user data during reimaging with DeployStudio. I came across the BackupRestore scripts by Rusty Myers and they looked great for backing up and restoring over a gigabit network (up to 125 MB/s). However, since all reimaging at this site occurs in the I.T. Office, I wanted to make use of the jaw-dropping speeds of USB 3.0 (up to 525 MB/s) and Thunderbolt (up to 1.25 GB/s). Of course, real world transfer rates are less due to disk read/write speeds, but it’s still a substantial improvement over gigabit Ethernet speeds.

My solution involves two scripts, a backup script ran just before DeployStudio reimages the internal disk and a restore script executed on first boot.

To further speed up the backup/restore process, unwanted data is skipped with rsync’s --include-from option, allowing pattern filtering of the items backed up and restored. A good example use is skipping the restore of ~/Library/Caches directories.

Setup

Setting up the required workflows is straightforward and takes approximately 10 minutes. To try it out simply follow the steps below.

Download backup.sh, restore.sh, backup_filter.txt and restore_filter.txt from GitHub here.

Copy backup.sh and restore.sh into your DeployStudio Scripts directory.

View fullsize

Copy backup_filter.txt and restore_filter.txt into your DeployStudio Files directory.

View fullsize

Open DeployStudio and create a workflow called 'Backup', add a ‘Generic’ script task and select backup.sh from the ‘Command’ drop-down list.

View fullsize

Create another workflow called ‘Restore’ and add a file copy task. Set the ‘Target volume’ to ‘Macintosh HD’, select ‘restore_filter.txt’ from the ‘File’ drop-down list and set the ‘Path’ to ‘/Library/Scripts’. Add a ‘Generic’ script task and select ‘restore.sh’ from the ‘Command’ drop-down list and check ‘Postponed execution (command will be launched on first boot).’

Create a third workflow called ‘Backup + Reimage + Restore’ and drag over three ‘Workflow’ tasks. As you can probably guess, the first ‘Workflow’ task is set to ‘Backup’. The second is your standard workflow for reimaging and the last is ‘Restore’.

Duplicate this workflow, rename it to ‘Reimage + Restore’ and remove the first backup ‘Workflow’ task. This workflow is used to restore a backup to a different Mac.

Lastly, uncheck the Publish checkboxes next to your reimage and ‘Restore’ workflows.

Time To Test

Before proceeding make sure you have a backup of any important data!

Format a USB/Thunderbolt/FireWire external disk, with the name 'Backups' and connect it to a Mac you wish to reimage.

View fullsize

NetBoot into DeployStudio and test out the new workflows.

Rapid Backup & Restore

Unfortunately, todays MacBook Airs only have one Thunderbolt port and during reimaging, that port is taken up by a Thunderbolt to gigabit Ethernet adapter. Although untested, I believe a Belkin Thunderbolt Express Dock would allow NetBooting into DeployStudio while attaching a Thunderbolt SSD.

Mac Migration

To migrate a user from one Mac to another, first run the the Backup workflow to take a backup of the source Mac. Connect the external disk to another Mac you will see a directory with the serial number of the source Mac. Simply rename the directory to the serial number of the destination Mac. Connect the backup disk to the destination Mac and then run a ‘Reimage + Restore’ workflow on the new Mac.

It is important to note that the restore script does not recreate user accounts, it just transfers user home directories. This is by design as the creation of the mobile account occurs when a user logs into the Mac for the first time.

Backup Archive

After successfully restoring a backup, it is moved into /Volumes/Backups/Restored/. This allows for recovery of any files that may have been skipped during restore, due to restore_filter.txt rules.

View fullsize

Meet Munki - Managed Mac Software Deployment

Recently I have come across a number of Mac schools and businesses running Munki, an open source software distribution tool. I have only studied it for a couple weeks but I can already see why so many Mac sites are using it. This post covers the concept of Munki, my best practices and a list of common commands.

The Concept

To put it simply, a Munki server maintains a repository of Mac software. Users running the Munki client application (a.k.a. Managed Software Update) are notified of new software and are then prompted to install it.

Munki can deploy newer versions of currently installed applications (e.g. Google Chrome), Apple software updates (e.g. OS X 10.9.3 Combo) or even custom in-house packages (e.g. ConfigurePrinters.pkg). Munki can also detect if an application has been deleted and will reinstall it.

The Structure

A Munki repository is hosted on an internal web server and consists of four directories:

pkgs: This contains all available software, generally as dmg, iso or pkg files.

View fullsize

pkgsinfo: This contains a plist file for each software in the pkgs folder. It’s common to edit these plist files to add software to different catalogues, set software dependencies or edit the minimum OS requirements. The Munki server reads all the plists in pkgsinfo and compiles catalogues from the data. Clients do not access this directory.

View fullsize

catalogs: This is another directory of plists. My Munki server has two catalogues: ‘development’ and ‘production.’ Catalogues are useful for sandboxing new versions of software until they are deemed ready for widespread deployment. Munki automatically generates a catalogue named ‘all’. This catalogue should not be included in any manifest.

View fullsize

manifests: This directory contains more plist files, controlling which software is deployed to which clients as each client is set to download a specific manifest (e.g testing, staff_laptop, etc.). A manifest plist lists software (without versions) to install and/or remove. You can also add the optional_items key in a manifest plist and specify a list of optional software choices. This provides a self-service interface, allowing users to choose from the list of optional software.

View fullsize

My Best Practices

Naming Conventions
To make maintaining the Munki repository more intuitive, I have taken a different approach to naming my catalogues and manifests that differs from norm found on the Munki wiki. I have chosen not to have a ‘testing’ catalogue, only ‘development’ and ‘production’ catalogues. I do however have a manifest called ‘testing’ that accesses the ‘development’ catalogue.

Munki imports new .dmg files with the same filename. To keep the repository tidy, I recommend renaming files before importing. The filename should be in lowercase with no spaces, followed by a dash and the softwares version number (e.g. googlechrome-2.1.4.dmg).

Granular Software Control
In my previous workplace, every Mac ran a SOE (Standard Operating Environment) suite of software. Depending on the location (e.g. Art, Music, etc.) and user type (e.g. staff or student), specific extra software (e.g. Photoshop, Sibelius, etc.) was installed. On top of that there were faculty that required access to software normally only provided to Art or Music computer labs. Since Munki clients can only check a single manifest, I dealt with this limitation by creating a structure of manifests included in other manifests.

It seems confusing at first, but once the infrastructure is set up, assigning new software to all relevant clients is simple. The green bubbles are manifests that clients check, we normally do not add any software directly to these. Yellow are purely for merging manifests, again nothing should be added to these. Blue are core attributes (e.g. laptop, staff, science, etc) software is assigned to these.

For example; Skype is an application only installed on boarding house student machines. To add Skype to the boarders manifest I run:

/usr/local/munki/manifestutil
add-pkg Skype --manifest boarder

The boarder’s MacBooks are configured to check the ‘student_laptop_boarder’ manifest and therefore are notified of any changes made to ‘soe,’ ‘student,’ ‘laptop’ and ‘boarder’ manifests.

Since Skype was added to the ‘boarder’ manifest, Managed Software Update lists Skype as well as VLC media player from the ‘soe’ manifest as available software.

View fullsize

Please note: Manifests directly accessed by clients require a catalogue (e.g. ‘production’) to be specified, however, included manifests that are not directly accessed by clients do not require a catalogue to be set.

Updating Software

In this example I have two versions of Skype, 6.15 and 6.17 in my repository. Version 6.15 has already been tested and added to the ‘production’ catalogue, however, the newer version 6.17 still needs to be tested and was added to the ‘development’ catalogue on import. After thoroughly testing Skype version 6.17, I would simply edit the associated Skype-6.17.plist file in pkgsinfo, replacing 'development' with ‘production’ in the 'catalogs' section.

View fullsize

To update the ‘production’ catalogue with this change I run:

/usr/local/munki/makecatalogs

View fullsize

The new version of Skype is now part of the ‘production’ catalogue and made available to clients.

Common Commands

Server Commands
Display Munki server configuration:

defaults read ~/Library/Preferences/com.googlecode.munki.munkiimport

Import a new application or package:

sudo /usr/local/munki/munkiimport /PATH/TO/SOFTWARE

Rebuild catalogues from pkgsinfo data:

/usr/local/munki/makecatalogs

Enter manifest utility interactive mode:

/usr/local/munki/manifestutil

Common manifest utility interactive mode commands:

list-manifests
new-manifest MANIFESTNAME
display-manifest MANIFESTNAME
list-catalogs
list-catalog-items CATALOGNAME
add-catalog CATALOGNAME --manifest MANIFESTNAME
add-pkg CATALOGITEM --manifest MANIFESTNAME

Client Commands
Set Munki software repo URL:

sudo defaults write /Library/Preferences/ManagedInstalls SoftwareRepoURL "http://swupate.local/munki_repo"

Set ClientIdentifier (manifest):

sudo defaults write /Library/Preferences/ManagedInstalls ClientIdentifier "student_laptop_boarder"

View client configuration:

defaults read /Library/Preferences/ManagedInstalls

Check for available updates from Terminal:

sudo /usr/local/munki/managedsoftwareupdate 

After checking for updates, install any discovered updates:

sudo /usr/local/munki/managedsoftwareupdate —-installonly

Download and install Apple software updates (does not require admin rights):

sudo defaults write /Library/Preferences/ManagedInstalls InstallAppleSoftwareUpdates -bool true

Show the user which software will be removed (managed uninstalls):

sudo defaults write /Library/Preferences/ManagedInstalls ShowRemovalDetail -bool true

Disable Munki installing software at the login window (useful for laptop users):

sudo defaults write /Library/Preferences/ManagedInstalls SuppressAutoInstall -bool true

Disable Munki installing software while user is logged in (useful for computer labs):

sudo defaults write /Library/Preferences/ManagedInstalls SuppressUserNotification -bool true

Turn Your Mac into a PHP Web Server In Under 10 Seconds

In almost no time at all this post will have your Mac up and running as an Apache 2 web server with PHP support. It's great for those that want to set-up a local web development environment to code that quick web app idea.

I have tested the following steps on Mac OS X Lion (10.7), OS X Mountain Lion (10.8) and OS X Mavericks (10.9).

Step 1: Enable PHP in the Apache configuration file

Open Terminal (/Applications/Utilities/Terminal) and type: 

sudo nano /etc/apache2/httpd.conf

Hit return and type your password. Don’t worry if it looks like you are not typing anything, it's just Terminal hiding your password input for added security.

To search the document press control + w, type 'php' and hit return.

You'll need to uncomment (remove the leading #) the php5_module.

To save changes press control + x, type y and hit return.

Step 2: Start up Apache

Type: 

sudo apachectl restart

Step 3: Test it out

Open Safari and type 'localhost' into the address bar. You should see:

This is the default web page that comes with Apache.

Open Finder and click 'Go' → 'Go to Folder…'

Type “/Library/WebServer/Documents/“ and click 'Go.'

Tip: You can press ‘tab’ to autocomplete folder names.

 /Library/WebServer/Documents/

This index.html file is the “This works!” web page you saw in the last step. Go ahead and edit index.html in your favourite text editor. I like to use Coda 2 (bit pricey) for my web development projects, but you could use any text editor. Before Coda 2 I was using Smultron and Notepad++, which are much cheaper. Once you have made a change refresh the page in Safari to see your changes.

[SOLVED] "This copy of the Install OS X Mavericks application can't be verified. It may have been corrupted or tampered with during downloading."

2018 UPDATE: This issue also occurs when attempting to install macOS High Sierra, with a slightly different error message: “This copy of the Install macOS High Sierra application is damaged, and can’t be used to install macOS.”

While trying to do a clean install of Mavericks onto a MacBook Air I received the error: "This copy of the install OS X Mavericks application can't be verified. It may have been corrupted or tampered with during downloading."

View fullsize

I found the error wasn't due to a corrupted Mavericks installer, but an incorrect date setting, caused by a flat battery. Since I had already blown away the previous Mac OS X install I needed to fix the time while booted from the Mavericks installer.

If you're experiencing this issue, here's what to do:

Boot into the Mavericks Installer. You can learn how to make bootable Mavericks USB here.

Open Terminal (Utilities → Terminal)

Check what the date is currently set to by typing "date". If it's incorrect type:

	date mmddHHMMYYYY

	
	Substitute

	mm = current month (e.g. February = 02)

	dd = current date (e.g 31)

	HH = current hour (e.g. 6pm = 18)

	MM = current minute (e.g. 05)

	YYYY = current year (e.g. 2013)
	

Hit return and quit Terminal. Now attempt the Mavericks install again, if it fails again it's not a date issue and I would try downloading and recreating your Mavericks installer.

Creating a Bootable OS X/macOS USB Installer

Since the release of Mac OS X 10.6 (Snow Leopard) I've carried around a bootable USB on my keychain. It has made my life a lot easier reinstalling OS X/macOS without an Internet connection and troubleshooting Macs without a recovery partition. I also much prefer clean installs of the operating system, using this method vs updating from a previous version of OS X/macOS.

This post will go through the process I use to create a bootable OS X/macOS installer.

Requirements

• At least an 8GB USB, you can also use a FireWire or Thunderbolt drive.

• A Mac running 10.6.8 (Snow Leopard) or newer.

• A fast Internet connection to download the OS X/macOS installer.

Step 1: Download OS X/macOS

If you haven’t already you can download the desired Install macOS application from the Mac App Store. Here are some direct links:

• macOS Catalina (10.15)

• macOS Big Sur (11)

Once it's downloaded the installer automatically opens, simply quit it.

Step 2: Prepare the USB

We are about to erase everything on the USB, make sure you have copied off anything you wish to keep. 

Open Disk Utility (/Applications/Utilities/Disk Utility), select your USB drive from the list and click the ‘Erase’ tab.

Set format to: Mac OS Extended (Journaled)

Set name to either: Mavericks, Yosemite, El Capitan, Sierra, High Sierra, Mojave, Catalina, Big Sur

Click ‘Erase...’ and 'Erase' again, then enter your password if required. 

Step 3: Copy over the files

Open Terminal  (/Applications/Utilities/Terminal) and paste the appropriate command.

For Mavericks (10.9):

sudo /Applications/Install\ OS\ X\ Mavericks.app/Contents/Resources/createinstallmedia --volume /Volumes/Mavericks --applicationpath /Applications/Install\ OS\ X\ Mavericks.app --nointeraction

For Yosemite (10.10):

sudo /Applications/Install\ OS\ X\ Yosemite.app/Contents/Resources/createinstallmedia --volume /Volumes/Yosemite --applicationpath /Applications/Install\ OS\ X\ Yosemite.app --nointeraction

For El Capitan (10.11):

sudo /Applications/Install\ OS\ X\ El\ Capitan.app/Contents/Resources/createinstallmedia --volume /Volumes/El\ Capitan --applicationpath /Applications/Install\ OS\ X\ El\ Capitan.app --nointeraction

For Sierra (10.12):

sudo /Applications/Install\ macOS\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/Sierra --applicationpath /Applications/Install\ macOS\ Sierra.app --nointeraction

For High Sierra (10.13):

sudo /Applications/Install\ macOS\ High\ Sierra.app/Contents/Resources/createinstallmedia --volume /Volumes/High\ Sierra --applicationpath /Applications/Install\ macOS\ High\ Sierra.app --nointeraction

 For Mojave (10.14):

sudo /Applications/Install\ macOS\ Mojave.app/Contents/Resources/createinstallmedia --volume /Volumes/Mojave --applicationpath /Applications/Install\ macOS\ Mojave.app --nointeraction --downloadassets

For Catalina (10.15):

sudo /Applications/Install\ macOS\ Catalina.app/Contents/Resources/createinstallmedia --volume /Volumes/Catalina --nointeraction --downloadassets

For Big Sur (11):

sudo /Applications/Install\ macOS\ Big\ Sur.app/Contents/Resources/createinstallmedia --volume /Volumes/Big\ Sur --nointeraction --downloadassets

Hit return, type in your password, then hit return again. Don’t worry if it looks like you are not typing anything, it's just Terminal hiding your password input for added security. Depending on your USB, the copy process takes approximately 20 minutes to finish.

That's it! You should now have a bootable OS X USB installer. Go test it out by plugging it into a Mac that is turned off, pressing the power button and then holding down the 'option' key.

I’d like to thank tywebb13  for sharing the above Terminal command on the MacRumors forums.

If you need any help feel free to leave a comment.