Software Patching System – Part II
In a previous post, I discussed the important requirements for a software patching system. This post will describe a practical implementation of such a patching system. We use this design in Tonido. Finally, we will look at how this design fulfills some of those requirements.
Tonido is modular and consists of the Tonido core framework and individual applications. Each Tonido application is independent of other apps and can be installed, updated, deleted without any impact on the rest of the system.
Therefore, the patching system is also organized along these lines. The Tonido core and the individual apps are treated as separate patchable components. A component is a loose collection of files that are connected together for a single purpose. For the purpose of patching, these files *should* be updated together.
Every component contains a description file called manifest.xml. A manifest file is a listing of the files in that module.
The manifest file also contains a version number, various meta information and a URL for downloading a remote manifest.
See below for an example:
The meta section contains the BaseURL and other information relevant to that module. The files section contains a listing of all the files including their checksums and whether they are executable. Finally, the signature data section contains a signature of the file.
Once the patcher wants to do an update for that component, it decides to do the following:
Step 1: It opens the local manifest file, reads the remote URL and connects to that URL via HTTP. The remote manifest is located at that URL.
Step 2: The patcher then first verifies the remote manifest is valid by looking at the signature information. Each Tonido client contains the public key. By reading the manifest, it can check whether the manifest was actually generated by us or by someone else. Once things are ok, it will then proceed to perform verification of each file in that file listing.
Step 3: For each file in the file listing, it will download the appropriate file from the server via HTTP if
- The file is not available locally
- The checksum of the file doesn’t match the value in the remote manifest
Once step 3 completes, it will “replace” the active files by the downloaded ones and save the remote manifest as the local manifest. The patching is complete. Nothing very complicated, but works well over the hundreds of times we patched Tonido during our development and alpha.
I simplified the process quite a bit, but this should be enough to get started. Additionally, here are some gotchas.
- Most of the time, you cannot simply “replace” the active files, because you might be trying to replace the running program itself and that is a big no-no. Instead what you need to do is have a small helper application that the main application will launch and exit. The helper will start up, looks and checks if something needs replacing, does the swap, launches the main application and exits. Voila! you have replaced a running process.
- Instead of downloading full files via HTTP, to save bandwidth, you can store compressed files on the HTTP server which you can unpack after you download. This gives you almost 1/3rd size of the original file. We use .gz compression.
- Creating manifests by hand is laborious. You can automate the process by having a small tool which when run over a directory will create the manifest automatically. We do this process during every Tonido build, so that if we decide to ‘promote’ the build online, we just have to copy the manifest files and the compressed files to the HTTP server directory and we are done.
Now, let’s look at the technical requirements from our first post to see if how well our requirements fit in.
Works (or fails gracefully) in non-admin mode
Tonido installs to locations that the patcher can write to without requiring admin privileges. Therefore Tonido runs, patches itself as a non-admin user.
Supports incremental patching
Tonido uses a significant amount of thirdparty files that don’t change much between builds. Instead of downloading the full installer every single time, the patching system only downloads affected files. This saves bandwidth, reduces time for download and is less work for the user.
Tonido’s patching system uses public/private key ecosystem to ensure manifest files cannot be faked. Even if an attacker somehow was able to divert a client to his own server and download his own files, he cannot sign the manifest files. So when the remote file is downloaded it will still be discarded.
Easy to administer and manage software patches
Tonido’s patching system is almost zero work for us. Every build, we create all the required files for the patch. If the build ever makes it to release, we simply copy over the manifest and the files to the server. Nothing much to to do. Also, in this design there is only one single “latest” version. So irrespective of whatever version someone is running at, when they do an update they only update to the latest version. There are no huge number of versions that we need to track and maintain. Finally, all patching is done over HTTP, therefore a Apache Server and a fast connection is all you need. No special server software.
To summarize, a patching system like one used in Tonido can help software be up to date easily without being a pain to maintain. Obviously more optimizations are possible, but we instead took the approach of making it robust and simple first rather than optimized and more complex.
We expect to go to Tonido beta over the next week, and it would be interesting to see how well our patching system holds up under much more load.