Author: Andrew Stitt (afrayedknot) To future sorcery leads: this is your handbook, none of this has to be set in stone if you feel strongly enough that the rules I've put in place should be ammended it is yours to change, I just ask that you understand the reasoning behind those rules.
Table of Contents
Section 1
- Introduction
- My view of sorcery
- Finding Things To Do
- Source Repository
Section 2
- How to do development work
- Coding Guidelines
- Improving some bit of code
- Fixing a little bug
- Fixing a big bug
- Creating a new module
- Removing functions
Section 3
- Before you submit...
- Testing
- *Function Testing
- *Unit Testing
- *System Testing
- Changelog
- Communication
- When mistakes happen
- How to fix it
- Consequences
Section 4
- Understanding sorcery
SECTION 1
INTRODUCTION
If you are reading this, you're either already a sorcery team memeber and I've told you to read this, you're thinking about being on the team, or you're just curious about what it means to be a sorcery hacker. So I'll try to explain what being a sorcery hacker means, and the responsibility that comes along with it. If you're new to smgl and want to do development work, I would suggest starting somewhere other than sorcery.
But before I even get to that, if you don't read anything beyond this sentence, the most important part of being on the team, above all else, and if done properly would make the act of me writing this entire handbook less of a necessity, is COMMUNICATION; you MUST communicate with your sorcery team leader, you NEED to be able to communicate with your fellow team memebers. <end sentance>
If you can accomplish this one thing I will be FAR more forgiving of any mistakes you make, because at least I'll know what you were doing, or at least what you were trying to do. Theres also a good chance that in the act of communicating with me we've gotten more angles on a problem worked out, and thus theres less chance of problems later, not only that, but if I help you solve something or at least say "yes good idea!" I then take some ownership and understanding in that work and what you were thinking when you did it and therefore problems with it are now, in part, also my problem, not just yours. Which in the end makes it a lot harder for me to chew you out for doing something dumb 
Besides, you don't want me to be angry, you wouldn't like me when I'm angry.
MY VIEW OF SORCERY
You should read this, even if you think I'm full of shit, you may gain some perspective on why I will act the way I do with sorcery related business.
Sorcery, in my opinion, is the most difficult and complex piece of the Source Mage GNU/Linux Distribution (SMGL). Not to belittle the huge amounts of work that have been put in by the other teams on the grimoire, the installer and the other bits and pieces of SMGL. Our work on sorcery is WORTHLESS without the grimoire that our guru's have been thanklessly working on day in and day out, and similarly without an installer, we would have no user-base, and I'll let you extrapolate from there.
Sorcery does, however, take the cake in terms of complexity, and the frequency of execution versus lines of code.
What does that mean? It means that your code will get run more often than the code in any spell (on average). Things like the libhash and libcodex provide a significant portion of sorcery's backbone, and need to be solid, lest the whole of sorcery comes crashing down.
That means that when you're code breaks, there is a much higher chance of it effecting most SMGL users. That means that your code needs to work, it means that when you change the behavior of a function you need to be aware of when and how its going to be called, it means that when you mess up and submit "rm -rf ${var} *" instead of "rm -rf ${var}*" people lose their data, people lose their time, people go to some other less broken distribution.
Everything you do, reflects on the entire SMGL community as a whole.
A guru can submit a broken spell and it can sit idle in the grimoire for months before anyone breaks their box with it. We don't get that option. Chances are, everyday, somewhere in the world, someone is running your code, not only that, they are trusting their box, their data and their time with it.
People depend on the validity of your code.
Working on sorcery is not as easy as writing a spell where you have certain well defined places to put code. You have to know how to program, you have to be able to see the big picture, your code has to be maintainable and readable.
As a sorcery team member you must be able to work with the rest of the team, admit your mistakes, learn from them, understand how important sorcery is and you must accept and follow the guidelines from this handbook, or at least tell me why you can't.
If you think something I've said is incorrect, tell me.
FINDING THINGS TO DO
First of all, reading this guidebook is a good place to start. After that understanding sorcery is another good thing to do. Finally reading the Advanced Bash Scripting Guide would be good too.
Now in terms of real sorcery work, theres a sorcery wishlist. On that wishlist is a list of current projects, future projects, and easy to fix bugs.
I'd like to see all those easy to fix bugs go away first. Then I'd like to see projects worked on.
Some of those projects are writing all new code (like a new tool or something). Other projects are fixing up or re-writing some existing bunch of code.
No matter what you're doing, before you start submitting code, and preferably before you write a whole lot, come talk with me so we can work out a design, document it if need be. Then you can go code 
Why do it this way? Well its easy to slide crappy changes in to the codebase, its also easy to tangle the codebase up. I dont like either of those things, and I may not be very nice to you when I notice. It also give me a chance to know what you're doing, and helps ensure that the code we are adding is well thought out and is thus, less likely to break somewhere.
Communication is the most important part of being a sorcery team member.
Again, if you have trouble doing this, sorcery may not be the best team for you to be on, go work on some spells they're much more laid back about working on your own thing, besides, they have more bugs and more lines of code that will need tending to.
SOURCE REPOSITORY
We keep our code in a source repository, why we do this should be mostly obvious, but if not it helps keep revision history and it allows users to collaborate on a shared project.
Currently we use perforce. I would like that to change someday to subversion or some other equally functional and free SCM. Perforce, while a great tool. is not free, and it troubles a number of people that we are a project commited to free software and yet in the background we dont use free software. Not everyone wants to use perforce because it is non-free. Anyways the debate has been long and difficult. Attempts were made at a dual revision system with CVS and perforce talking to each other. Unfortunatly that has proved difficult, and IMO is a rather silly waste of time. Not to mention that CVS is not a very good fit for grimoire related work. Subversion however, is. Of course sorcery could live in most SCM's but we dont really want to have seperate homes for sorcery and the grimoire on a permanent basis. Anyways...changes may occur in the coming months with where sorcery is held. There may be a sudden shift to subversion for sorcery as a testing ground before moving the grimoire over. I'll update this section as needed.
So back to the point. For the average non-big-project type of work, you'll need to have the devel branch of sorcery in your client view (perforce speak for checked out). That would be under //sgl/sorcery/devel/... Now the usual commands would be "p4 sync" "p4 edit" "p4 revert" "p4 diff" and "p4 submit". As a sorcery team memeber you will be given read/write access to the devel branch, and you won't need to worry too much about the other details.
Now for example we collaborate and decide on re-writing some big piece of sorcery, or we're adding some new, potentially dangerous tool. In order to work on this tool we'll have to adjust some library apis, and cause other general havoc. Not only that we dont want these changes to be part of devel for a little while. SO you can do one of two things, keep all the files open and unsubmitted in a perforce sandbox. Then after you've finished everything submit it all at once.
The problem with this method is that 1) you dont get any revision control while you work on it, so if you screw up you can revert easily, and 2) no one else can help make changes. If this is a major ordeal you may want to work on it with others, and the only way to do that is to just duck around perforce entirely. So you're basically defeating the purpose of having revision control at all.
The other more preferable method is having a project branch whereby devel is branched off into a seperate code stream while we break everything and put it back together again. All the while perforce is helping keep things in order, which is why we use it after all. So, thats what we can do now. We can potentially branch devel into //sgl/sorcery/proj/gaze-rewrite/... or something then pull new changes from devel into the branch and eventually push all the changes back.
The commands for that will be here later.
The other aspect of this is that at release time I branch devel into "test" then immediatly into "stable". After that I can branch it into a frozen release branch. So we can keep each release frozen in time somewhere to look at. If there are bugs that come up in that release we can fix them there, in the release branch (or make a new one), then push all those changes back into devel if need-be.
All the details of this you dont really need to know, but when I figure it all out myself I'll write some more about it.
---
SECTION 2
This section well tell you how to do work and make your sorcery leader happy.
HOW TO DO DEVELOPMENT WORK
you write code, and then you do stuff
CODING GUIDELINES
I'm not big on exact code formatting and things like that, so these are only guidelines, not standards. Its probably a good idea to follow the indenting depth thats already in the file.
These two guidelines are not optional, you must follow them:
Use spaces not tabs, too many people don't understand that a tab is the same width as 8 spaces and instead have their editor set to 4 spaces or something else, then they end up with spaces and tabs mixed up, and of course everything looks fine to them in their happy world of 4 space tabs, but guess what, it looks like crap to the rest of us.
All lines need to be wrapped to 80 characters or less, no excuses, or at least not any excuses that I've already heard.
I don't like the use of this construction:
if foo && bar && baz && then success else failure fi
if is not the right construct for that sort of thing. Call me old fashioned but I like putting stuff in { } braces, which will run the code in the current shell. This is not to be confused with ( ), which will run the code in a subshell, and thus incur a fork() and not preserve variables. You can use a || between two blocks of code this way if you want, or maybe its time for some functions, use your judgement.
functions should be named using the function keyword
function foobar() { ... }
instead of without
foobar() { ... }
its not just a matter of taste and consistancy. It also makes grepping for functions easier and more importantly, enables their documentation through bashdoc.
Sometimes you end up with a small if statement like this if [ stuff ] ; then other stuff fi It can make things more readable if you do this instead [ stuff ] && other stuff Use your judgement.
Read the advanced bash scripting guide, keep a copy of it under your pillow. Pop quizes will be given daily
IMPROVING SOME BIT OF CODE
So in the never ending quest to understand sorcery better you come across some function that either has a bug in it, or can be improved.
The first question you should ask is, why is it this way? Is there something you haven't discovered yet, maybe there is, maybe there isn't. If there is, it ought to be documented or perhaps fixed (possibly in other functions).
The next question you should ask is, if I fix this, will it still work at least as well as it used to? If there is a function header explaining what the function does, it should adhere to that first and foremost, and callers of the function should respect that. If a caller is not respecting the documented specification (and perhaps the function itself has been hacked up to work around that), then it (and possibly the function itself) need to be fixed.
Ask yourself if you can fix the surrounding code as well. Perhaps some interface or module was poorly designed. You can then turn this into more of a project, of renovating this particular module.
Finally, you ought to talk to me about what your doing, both for my knowledge and sanity, and for quality control purposes.
FIXING A LITTLE BUG
If you're picking up one of the "easy to fix" bugs on the wishlist then chances are I've already taken a look at what you'd need to do and/or it wont turn into a project. But the guidelines about about fixing some bit of code still apply.
Finally, be sure to let me know what you're doing, assign the bug to yourself, and let me know what changes you plan on making.
FIXING A BIG BUG
This would be a "project grade" sort of thing. This will require either writing a lot of new code or fixing/re-writing existing code, or both. Don't take up a project on your first day.
As always, assign the bug to yourself, let me know what your doing, discuss with me your design with me, document it. So on and so forth. I hope you're noticing a theme here.
This should lead directly into the creating a new module section. Hopefully the module you are making/rewriting will address the problems that are occuring, thats kind of the point isn't it?
CREATING A NEW MODULE
So you've decided to take on writing a new module. Good job. This section is also going to cover re-writing a module, which is almost the same thing, because the reason you're re-writing it is because its current design/interface sucks.
So lets jump right in. A module is going to have some external functions that make up its "api". So for the libdownload library that is download_files, and get_files_and_urls. One sets the current spell, then runs get_files_and_urls, which then generates a stream of output containing a filename and some urls that it might be at. This output can then be redirected into download_files, which itself has parameters to control how it will act. Alternatively one can build their own list of file + urls and send them into download_files.
So this huge api consists of basically one real function, that provides the be-it-all, end-all interface for downloading anything and everything in sorcery (yes). The other is a helper routine for looking into spells. That helper routine is also used for some other information gathering. In anycase where download_files ends up going before we start actually /downloading/ stuff, or at least poping out the other side into another library is basically the internals. This stuff should be basically only called within the module. Otherwise it would be part of the api! Bash doesn't give us a lot of protection from random people using internal functions, so you'll have to mark the functions as such, and hope for the best. If we can all just get along and stick with our api rather than needlessly hacking through every possible entry point in a module we can avoid a lot of code tangling.
For your module you'll want to come up with something that fits what you're working on. Typically the external stuff should be simple, but flexible. Talk to me if you need help.
Now for the guts, continuing with our libdownload example. The download_files function guarentees its caller that it will do everything possible to get the files its been given. These actions include looking on disk, which is an internal function, with a few other library calls. It then makes use of the liburl library (where libdownload comes out the other side). liburl provides a url_download function, which given some list of urls will do everything it can to optimize around those urls and download one of them. in libdownload we have a concept of leap forward and fall back urls, which are tried before and after the passed in urls.
One important thing to note is that liburl has /no/ idea what a leap forward of fall back url is. To it, a url is a url is a url. OTOH libdownload has /no/ idea what a url actually is, its just passing them along. This is a seperation of responsibilities. Each module does one well defined thing, and does it really well.
So in your quest to make your new module, come up with what your input api's are going to be and what output api's you'll need to connect to.
CODE LIBRARIES
Whats a code library? Well its not quite a module. Its more of a collection of helper routines. These routines can modify data, for example they could filter out directories from a list of files/symlinks and directories. Or provide some useful lower level function, or perhaps do something more than one module needs. The distinction is subtle, but one way to tell is how much code is underneath them. Please be aware of this distinction.
STACKS
What is a stack? This is kind of a borrowed term from networking. In networking you have multiple protocol layers. Indulge me with this overly simplified example. Say you want to look at a web page, so your browser makes an http request to some server, that http request gets bundled into a tcp packet, which gets bundled into an ip packet which then goes out your (for example) ethernet card, and is thus sent out as an arp packet (or whatever they're called at that point). The packet then gets partially unpacked and repacked as it traverses different types of networks. It eventually gets to the server whose network card gives its operating system back the ip packet, which is then unpacked to reveal the tcp packet, which is then put in order with other tcp packets, and eventually forms to http request.
The thing to remember here is that you have all these layers, none of whom care about the one above or below them. The http request doesn't really care what type of nic you have, or the server has. tcp doesn't care if the data is http or smtp. All the layers stack together, one on top of the other.
We do some of the same stuff with modules. Going back to my libdownload example, summon makes a download request to download_files, download_files then pokes around making requests to liburl. liburl then starts having url_handlers look at urls and then run some command (like wget). ALl these layers for a "stack" if you will. Scribe and sorcery (with their update commands) make use of this downloading stack if you will. They analogous to the http, ftp and smtp in relation to tcp.
Parts of sorcery are, or should be designed this way. It helps keep code seperated. If we can keep everyone but libdownload from using liburl then thats fine by me. The less tangling and fewer connections between boxes the better.
---
SECTION 3
So now that you've written some code are you going to just submit it for peer review? Thats the open source way isnt it? Well sort of. It never hurts to do at least /some/ testing. Here I'll outline some stuff that might help you test things.
BEFORE YOU SUBMIT
I /always/ run p4 diff before submitting. It will give you all the changes you've made and lets you know exactly what changes your making. Its a sanity check. If you're removing functions or changing an api (within sorcery) make sure the callers are all updated (and that I know, because that shouldn't be happening too often).
TESTING
Okay so testing. That humbling un-exciting thing. I was at a lug meeting recently with a job opening for a test automator. Basically writing code to run a test so some moron doesnt have to do it by hand. Standing next to me was some other guy offering a job doing "development" work. Nevermind my opinion of what the company was trying to do, I got a whole lot of "oh, nevermind" sorts of answers, or "thats really lame and un-sexy" sorts of looks/responses. But this other guy had a crowd of people around him. Why? because developing is COOL and testing is LAME and for un-experienced or stupid people.
Well, don't let those people fool you. I can guarantee you that their code won't work without testing. Basically NO ONE's code will work flawlessly without testing. Sure you can make little changes and wave your hands. But I doubt a single one of those people who crowded around the offering of a sexy development job truely realizes how lousy their code probably is. Yes I'm cynical. Yes I've seen lots of bad code. And yes I've seen people who don't realize their code sucks, and are so ignorant that they can't be bothered to test it. Dont be on of those people. If 75% of the code you wrote was to test the other 25%, rather than the other way around, your 25% would be worth keeping (assuming you fixed the problems you found, and that your testing code was reasonable). The other party's code would be far less worthwhile and far more likely to have bugs.
Unfortunatly testing isn't something one can just do. You have to apply yourself and think about all the possible inputs and situations your code can get into.
Lets start from the bottom.
- Function Testing
You wrote some function. It has a comment explaining what the parameters are, what goes in stdin if anything, and what comes out stdout.
First of all your function should match the comment. But now how to actually test this one function.
You write a simple script that calls it in different ways (as opposed to unit/system testing where you use it in its environment).
So for download_files I wrote a script like this:
#!/bin/bash . /etc/sorcery/config SPELL=tree run_details get_files_and_urls|download_files
Amazing huh? I also did similar stuff to test just get_files_and_urls. I also made up some sample inputs for download_files and echo'ed it into the function rather than using another routine.
The . /etc/sorcery/config just loads all the sorcery libraries and stuff.
I of course also tried different combinations of parameters and so forth. After doing all that different parts of the code were exercised. Problems were found and fixed. All without having to do complicated testing with the rest of sorcery.
I also did similar things with each of the internal routines, as they were written.
One final note, its "okay" to "fake" calls to other modules or functions if you're trying to test something thats higher up in the code path. So I may have written a fake url_download that just faked downloading by touching a file in the right place. You get the idea hopefully.
You certainly don't have to be so rigorous, this is a technique I use that may be helpful for big complicated things.
- Unit Testing
This is almost exactly the same as function testing, but you're testing an entire subsystem. (such as the download stack). The distinction is that you're specificically testing the unit as a whole.
- System Testing
This is where you bundle all that stuff up and work with the system as a whole. So you start casting lots of stuff or summon things and make sure things work as a whole.
It goes without saying that all of this requires you to use your imagination and try to find problems in your code and make sure you've thought about everything.
CHANGELOG
We have a changelog. Its pretty straightforward what to do here.
COMMUNICATION
If you plan on changing a whole lot of stuff, or re-writing a module, talk to me your fellow team members and all that good stuff. We need to know what you're doing.
WHEN MISTAKES HAPPEN
Mistakes happen. Hopefully it was found in devel, not stable. But its your responsibility to help fix the problem. I dont really like having to go back and clean up your mistakes. As a team member you should be mature enough to own up to your mistakes, fix them and learn from them. I make mistakes. I had a bug that left .fifo's all over /tmp. So I fixed it and learned to pay more attention to stuff like that.
HOW TO REVERT CODE
So I dont know exactly how to do this in perforce, but basically you sync up the files to somewhere before the change, then p4 edit and submit. You'll be asked to resolve, you of course do not accept your previous change, but most of the others (unless they're related). There is kind of a short window for reverting. But I thought Id at least leave a place to mention it.
CONSEQUENCES
So if you're finding that you cant seem to "get it right". And you're always submitting things that aren't working. I ask you to consider 1) working on something simpler, maybe no more big project stuff, only smaller stuff I delegate to you, or 2) working on a lower impact part of the project. Say grimoire or cauldron.
---
SECTION 4
Someday we'll have a Sorcery Architecture document. Which will explain how it all works and connects...but not today.