DRAFT

This will probably never get finished due to a lack of time

I tried evaluating trac, but it's a PITA to make work. Apparently some sort of python modules for svn are missing, tends to make it break :/

This process needs work

The current process is not intuitive enough. Things could be made simpler. I'm proposing we try to use the states bugzilla has in it rather than try to use flags to define states.

Current process

This page describes the process of getting bug fixes into stable and stable-rc grimoires between regular integrations. All of this assumes there's a bug report for it on http://bugs.sourcemage.org. If such a bug report doesn't exist, e.g. because a guru just fixed something in test grimoire without a bug report, a bug should be created for the integration. No integration at all should happen to stable-rc or stable grimoire between release cycles without going through this process.

This same process should also be used for security fixes.

  • the bug "Version" field should be set to the highest grimoire it applies to, stable being the highest
  • if the bug applies to test grimoire, it gets fixed there
  • when a bug is fixed in test, the "fixed in lesser branch" flag is set to "+". Then the "integrate to $BRANCH grimoire" flags should be set to "?" for the branches the fix should get integrated to.

  • one of the gatekeepers (currently Eric Sandall, Seth Woolley, Arwed v. Merkatz, Jeremy Blosser (as assistant for Eric Sandall) and Robin Cook (as assistant for Arwed v. Merkatz)) approves or denies the request(s) for integration, with an explanation added at least for the deny case. If the requester for an integration is a gatekeeper himself, a different gatekeeper has to approve the request.

  • once a request for integration is approved (flag set to "+"), it can be integrated to the branch for which it was approved. This integration can be done by anyone with stable/stable-rc grimoire access.
  • the bug is resolved "FIXED" once it is integrated to the branch that the Version field specifies.

Note: When setting the "fixed in lesser branch" flag, add the p4 change number/git change id of the change fixing the bug as a comment. That way it's easy for the gatekeepers to evaluate the change for integration and to do the actual integration.

How it would happen Ideally

  1. New Bug is filed
    • Things the Filer should enter
      • Version found in (Stable, test, stable-rc)
      • Arch of the computer the bug showed up on
      • Meaningful Summary
      • Description
      • attach compile logs/install logs if applicable
  2. Someone notices the bug and decides to work on it
    • notify everyone that they're working on the bug
  3. A solution to the bug is found and submitted in test
  4. IF THE BUG IS CRITICAL The solution is integrated down into stable-rc
  5. IF THE BUG IS CRITICAL the solution is integrated from stable-rc into stable
  6. the reporter is notified that the bug is fixed
  7. the reporter verifies that the bug is indeed fixed
  8. Complete!

Fitting it to bugzilla

Currently bugzilla doesn't have 8 states. It's only got 3 or 4 (Useful ones anyway). New -> Assigned -> Resolved -> Verified -> Closed. This doesn't quite fit the states where it's in multiple grimoires. Currently, we've tried to solve this lack of states problem with flags. Unfortunately, with bugzilla flags aren't really designed to define state changes. They don't show up obviously in searches and listings. Due to the fact that we'd have to modify bugzilla for our own internal use, I'll outline how to make bugzilla fit our paradigm a bit better without modifications first. This example applies to a critically broken item, but it will also apply to enhancements and the like just you won't go through the integrations until the normal release cycle.

Critical bug found in stable

I'll list the state changes the problem should go through as it's being worked, until delivery.

New bug is filed

Someone has filed a bug. They put in that the bug is in stable! OH NOES! They've got an x86 so it's not an architecture oddity. They file it with a BLOCKER priority, because the problem completely prevents something from building.

Someone noticies the bug and decides to work on it

Here's where it gets somewhat convoluted. It's only one step here, and it makes it significantly easier to manage, rather than change versions, make new bugs. A new bug is filed by the person that want's to start working on the bug. They put a suitable description "Integrate fix for bug #<previous> to stable-rc" The first bug is then made to depend on the new one. That way an integration to stable-rc must happen before the fix can be applied to stable. Then we do the same thing again for test. "Fix for bug #<first bug> in test" and the "integrate to stable-rc" bug depends on this. Now we've established the hierarchy of how our fixes get applied. All these bugs should have the same priority as the first one. Same Arch, and Same component, the only difference will be the Version. I propose doing this so that the version field remaions immutable during the fixing process. I feel that it's too easy for the bug to get lost if versions are changed. It's much simpler to cascade down through the dependencies. So the hierarchy has been established. The person doing the fix assigns the "Fix in test" bug to themselves, whereas the others remain new, because they've not been worked on yet. This person now happily hacks away.

A solution to the bug is found!

W00t! Day of days! There's a git commit id with logic behind it that will solve world hunger. Or something. Anyway, regardless of the power behind the fix, the "Fix in test" bug can now be marked RESOLVED FIXED, because it is! It makes sense. It can be easily and simply queried.

Integrate the bug to stable-rc

Okay now it gets interesting. The person evaluating the fix to place it into stable-rc (a gatekeeper) will verify that the fix in stable really does solve the problem. Then they will mark that bug VERIFIED. They then ASSIGN the "Integrate to stable-rc" bug to the "stable-rc integrators." They cherry-pick down, and resolve conflicts and test the fix (if it's non trivial, test the integration), then submit their stuff to the stable-rc git. Now the stable-rc bug can be marked as RESOLVED FIXED, because it's fixed! Again, using the states here makes it easy to query, and simple to understand.

Integrate the bug to stable

Same basic process as integrating to stable-rc. The "Integrate to stable-rc" bug is marked VERIFIED if the fix does fix the problem in stable. Then the original bug, the one filed by the user against stable, is ASSIGNED to the stable gatekeeper that will be performing the operation. They'll then do the integration, resolve conflicts, and test the fix (if non trivial; Ideally, we'd test every fix, but we don't have the resources for that.) Once the integration is done and clean, they'll submit it and mark that bug RESOLVED FIXED. Now the fix is in the grimoire. I think that a new stable should be rolled at this point so that we get the fix, which has now been verified by 3 people, can get to the end user.

The bug reporter verifies the fix

Ideally, the reporter of the bug would come back and let us know that this really did fix their problem. They would verify that the bug is indeed fixed, and mark the bug VERIFIED FIXED. Unfortunately we don't get this feedback very often. Perhaps with this simpler, albeit more noisy, process we'll get more feedback. :)

The bug is finished

The user has verified that the fixed worked. It's done. CLOSED

Alternative Resolutions

I want to cover what would happen if we didn't apply a fix. If it's RESOLVED one of INVALID, WORKSFORME, WONTFIX. LATER isn't really a good option, and in the authors opinion, shouldn't be used.

RESOLVED INVALID

The bug basically dies at the "Someone noticies it" stage. No new bugs are created, it's dealt with in one bug. Because it's an invalid problem. There's no reason to request integrations, because there's no problem :)

RESOLVED WORKSFORME

Unfortunately, with Source Mage there's no possible way to test every environment possible for building spells. Sometimes people just have wierd computers. If the developer has tested the spell and cannot duplicate the problem, then its not a problem that can be fixed. More information can be solicited, obviously, perhaps remote debugging and such. WORKSFORME is not a trivial thing to apply to a bug, because we're basically telling the user "Tough shit, it works here" albeit more tactfully

RESOLVED WONTFIX

Sometimes there's just crazy requests. This is simply something we will not fix. It will also die at the "Someone noticies it" stage.

A normal Stable release cycle

Here's the interesting part. If bugs are filed in the above listed fashion, it's quite simple to ensure that the appropriate things are fixed before a release. And which bugs are fixed and can be cherry picked into stable-rc and stable. It provides a very simple and obvious method to track requested modifications down from test to stable. In my feeble little mind this is how I see the stable release happening:

  1. A set time has been reached, so a new stable grimoire is requested.
    • Either due to lots of bugs fixed, or lots of new versions
  2. Bugzilla is checked for bugs that need to be in the next stable
    • Precise criteria for that is beyond the scope of this document
  3. People are tasked to get those done for the release.

  4. Those are finished, and integrated along their normal process (outlined above :)

  5. Everything from stable-rc is integrated with stable. TaDa new stable!

    • Bugs that haven't been worked on won't need to change, as the bug hasn't been fixed up the grimoire tree.

  6. Everything from test is integrated into stable-rc. TaDa new stable-rc!

  7. Bugs that are in various stages of development get bumped up one.
    • If a bug was fixed in test, but not integrated into stable-rc yet, it just was!
    • If a bug was integrated into stable-rc but not yet into stable, it just was!

StairStep

Here will be how the stair step of bugs needs to happen when a stable release occurrs. All the bugs that are assigned to test need to be assigned to stable-rc and all the bugs assigned to stable-rc need to be assigned to stable. Hopefully we don't have any stable bugs at this point. I'll cover what we should do about that later. So following will be a step-by-step for doing the stairstep of bugs.

  1. Query up all the bugs belonging to codex and filed against stable-rc (This is assuming all the bugs for stable have either been closed, or dealt with)
  2. Do the Change multiple bugs at once.
  3. Change the version to stable (You can only do this if you work on one Product at a time so the prometheus bugs will have to be done seperately, but the procedure will be the same)
  4. Add some comment (Bug integrated into stable. Stable 0.7 release cycle)
  5. Click submit
  6. Repeat, but for test into stable-rc

Dealing with bugs against stable

There's two things that can be done here.

  1. Delay the release until bugs are fixed
  2. They're not important enough to delay the release

This process affects the StairStep process. You don't want to pull bugs down from stable-rc that still exist in stable. The integrator will have to do different things based on what's been done. It's simpler from a bug management perspective to delay the release until the bugs are taken care of. And if we allocate our resources correctly that shouldn't be much of a problem. We just need to push our developers a bit to take care of the bugs in stable. In the author's permission, it's the best solution. However, if we must leave bugs laying around, it gets more complicated.

Bug exists, has been cloned, but no work done AT ALL

This is the simplest case. Don't StairStep any of the bugs in that chain. The bug exists in all the same places as it did before, because no work was done. No code is integrated.

Bug exists, has been cloned, and some work has been done

There's a few ways to deal with this one. The easiest solution would be to treat the bug like it's an individual integration and pull it all the way down. However, there may be too many bugs to do this for. So, the integrator has to pay attention and marke bugs fixed as things get pulled down. Or the integrator can choose not to pull those fixes down. Not a very viable option though. So the only other viable alternative is to change the state of bugs that have fixes in various stages. If a bug is fixed in test, but not in stable-rc, then when the code is integrated, it's now fixed in stable-rc. The bug against test now becomes verified. It follows the normal integrate an individual bug chain to stable, but doesn't follow completely through. To save time for the integrator, the bug isn't pushed all the way to stable, only one step.

Use Cases

Follows will be a few use cases when using this process.

Regressions

Say perhaps there's a bug filed against stable-rc and the bug is cloned to test. No work has been done. A stable release is about to happen. There is no bug in stable currently. So when the integration from stable-rc is pulled into stable, it will introduce a new bug in stable. There's a few options regarding what to do here. In the authors opinion, we should delay the stable release and bump up the priority of these bugs, if feasable (the bug can be fixed within a reasonable amount of time.) This would prevent regressions and is probably the easiest way to manage it. However, if such things cannot be done, a new bug can be filed against stable, with and have it depend on the original bug, which was filed against stable-rc. This way the bug can still be tracked down to stable.

Real-life Release, with versions even

I'll go through how I would produce a new stable release. Versions for this example are:

  • stable - 0.4
  • stable-rc 0.5
  • test 0.6 (I think test will remain immuatble, as in test will not change versions, it'll be 'test')

Integrating stable-rc (0.5) to stable (0.4)

TODO: Define this without changing versions on any bugs. We want to keep our history.

Integrating Test to Stable-RC-0.6

TODO: Define this without changing versions on any bugs. We want to keep our history.

Test

The only bugs left assigned to this version should be the ones that are still open and have had no work done. There should be no bugs here with resolutions, unless it's RESOLVED INVALID, WONTFIX or whatever would cause the bug to die.

Summary

There are obvious advantages to organizing our bugs this way. It makes it easy to use the "Change lots of bugs at once" feature of bugzilla when a release cycle happens. It provides us with an extremely easy way to follow a bugs progress and gives us enough states to ensure that we can track the bug through all it's stages of fixed-ness. Also it's simple. The only difficult part is creating and linking the bugs together, which could potentially be automated. It also makes it very easy to move bugs along with the release cycle integrations. Which I believe we're currently doing with comments ("This bug exists in stable-rc 0.5" ).

NewBugProcess (last edited 2008-09-22 23:34:51 by localhost)