Topic

To improve resurrect support such that it is more feasable to redistribute cache tarballs in such situations as a homogenous lab environment or on an iso.

Problem

The current resurrect functionality isnt really defined or well supported, nor does it truely address the issues required for the iso. Currently the iso just has some quick hacks to accomplish what it needs. It would be better if sorcery produced good cache tarballs and resurrected them in a useful and more defined way. Also, resurrect does not work well with install_root and others.

There are two components to this project, one is improved cache tarballs. The other is unpacking them intelligently, obviously the quality of the resurrection is directly dependent on the quality of the cache tarball.

Introduction and terms

Here I will try and explain some concepts I'll use freely below, these are here as reference and to make the following sections more concise.

  • install log: this is the file that lists all the files a spell owns
  • md5sum log: this file is a subset of the install log and shows md5sums for spell files

See install_root_and_others or this basic overview:

  • INSTALL_ROOT: Regular files (generated by a spell) are installed relative to this directory, its an adjustment of / to the value of $INSTALL_ROOT
  • STATE_ROOT: Sorcery's files are stored relative to this directory, it may be within $INSTALL_ROOT, equal to it, or outside of it.
  • TRACK_ROOT: $INSTALL_ROOT and $STATE_ROOT are adjusted to this in the install log

Log formats: the install/md5sum logs can be in one of three formats, the different formats refer to how INSTALL_ROOT, TRACK_ROOT, and STATE_ROOT have been applied.

  • "filterable" INSTALL_ROOT and STATE_ROOT have been stripped off the front of all filenames, this allows the filter() function to be used
  • "log" This is the /on disk/ format of the logs, paths are relative to TRACK_ROOT
  • "root" Files are listed as they are installed on the filesystem, in other words, with INSTALL_ROOT or STATE_ROOT pre-pended. You can do a test -e on these files and it should work.

In the standard usage all three *_ROOT values are empty and the above formats are indistinguishable. The logic for translating between formats is in libtrack::log_adjuster(), the finer points that are irrelevant to this document are documented there.

In this document I will talk about three disjoint sets of files:

  • state-data is made of sorcery's files, this is currently everything under /var/log/sorcery and /var/state/sorcery (in filterable form).

  • config-data is described by the /var/lib/sorcery/configs file plus the spell specific version thereof. See File_Filters

  • regular-data is everything else

Proposed Spec

Improved tarball generation

With the InstalledGrimoire (now known as the tablet) project underway there will be more state data stored in the cache tarballs. Also, there will be a new filter to describe files as config files, this allows more files to be tracked than with excluded/protected, but we must be careful no to blatently trample on them.

Cache tarballs will be archived relative to install_root and state_root. This, combined with the tablet, enables any (new) cache tarball, regardless of its INSTALL/STATE/TRACK_ROOT'ness to be resurrected into a different set of INSTALL/STATE/TRACK ROOTS.

Cache generation must therefore occur in two phases, the first phase generates a tar containing just the state files, the second appends the data files to the archive and it is then compressed. The reason for packing state data before regular data is two-fold. State data is generally small, so there is little penalty for building it first. Secondly it makes it easier to peek into a compressed tarball to figure out what parameters it was built with. Doing so is currently beyond the scope of this project/document, but is a future design goal.

Unpacking the files

When people say resurrect, this is generally what they mean. The basic overview is:

  • unpack the tarball somewhere safe
  • run a custom spell file for any pre-installation commands (currently this is called PRE_RESURRECT)
  • copy regular files to their locations
  • copy config files to their locations but dont trample on existing modifications
  • run a custom spell file for any post-installation commands (currently this is called POST_RESURRECT)
  • resolve state data

Note that there are spell plugin files that can be run before and after the installation phase. Their names are currently subject to change as this project has not been completed in full yet. One other scheme has emerged which is to have FINAL be run instead of POST_RESURRECT but define a magic variable so FINAL files can tell if they are being casted or resurrected in case there are different things to do.

Since cache tarballs are built with relative paths, they can be unpacked into a unique directory as a safe sandbox. From this point the files can be copied over to the real system. Note that this process must respect INSTALL_ROOT, TRACK_ROOT and STATE_ROOT of the local system, and that the install log is generated relative to the cache tarball's TRACK_ROOT, a naive copy of state data WILL NOT WORK. Instead the list of files is generated, copied to install_root and a new install log is to be generated. The tablet must also be installed and some values must be updated: pointers to install/md5/compile logs must be rebuilt, the "roots" file must be updated to indicate the roots the existing installation was made with.

Config files are installed according to the following algorithm: 

   if the file does not exist, just copy the new one
   elif the file exists and is identical to the new one, do nothing
   else the file exists and differs from the new one then
     if it is owned by the old spell with a valid md5
       replace the file
     elif it is owned by the old spell with an invalid md5 (admin changed it)
       ask if the user wants to change it
     else the file might be owned by another spell, or might be an alien
       ask if the user wants to change it

Current the user query involves the following choices:

  1. trash $to and install over it
  2. backup $to to $to.$savetime, install the new file in its place
  3. leave $to in its place, copy the new file to $to.$savetime
  4. do nothing
  5. see a diff between $to and the new file

$to is the destination file, $savetime is a timestamp.

Comments below this line please:

The see a diff between $to and the new file is good, but I'd like to also see (if an archive was made of the old one) a diff of the old spell's original file and the current modifications and the option to take this diff and apply it to the new config and see the output if patch succeeds, otherwise have the patch appended to the file with >>>> <<<< markers and let the sysadmin resolve them. It sure would make upgrading config files a snap, and I've seen FreeBSD do something similar with their portupgrade script.

-- Seth

Sometimes you may not be updating, in which case the "old spell's original file" is the file we're copying in, there is also the case of not knowing who really had the original file if it belongs to another spell (say several spells have owned the spell at one time or another).

-- Andrew

Improved_Resurrect (last edited 2008-09-22 23:35:07 by localhost)