The sad evolution of wikis

I set up my first wiki in 2001, in preparation for hiring our very first software developer employee. Until then, software development at NITI had been handled entirely by me and the other technical co-founder, dcoombs. And since we knew everything already, documentation was unnecessary. When it came to hire a new person, documentation suddenly became very necessary. I had to do a brain dump, and a wiki seemed like a great solution. The original NitWiki was born.

The first NitWiki ran on zwiki. Then we migrated to WikkiTikkiTavi, which we hacked up resulting in what we called HackyTavi, which we then cleaned up and renamed to GracefulTavi. (Note: the original WikkiTikkiTavi is very nice and rather graceful. It was just HackyTavi that wasn't graceful. No offense intended.)

GracefulTavi performed wonderfully; at NITI's peak, we had more than 30 internal developers, plus all our technical support staff, using it daily. All our technical documentation was in there, and it was easy to read, easy to find, easy to link to, and (perhaps most importantly) easy to write. There was also tons of non-technical stuff; all sorts of company culture, office games, product plans, newsletters, and everything else got dumped into the wiki. The wiki was the default place to dump stuff, and it was all full-text-indexable and made it easy for anyone to provide feedback and fix typos.

It's the most fun I've ever had doing documentation. Even more fun than GourD, which was fun too in its own way. But that's another story.

GracefulTavi had a long and happy life; it was still in use in late 2006 when I left, and I suspect it's still alive at IBM (which acquired NITI) even today, at least as a read-only reference material.

As great as it was, however, I don't think a NitWiki would be easy to make successful today. At least, I've seen people try, and it doesn't really work out. One of the problems is Wikipedia Mentality.

Wikipedia changed a few fundamental ideas about wikis that had been taken for granted:

• Wikipedia articles are supposed to look finished and professional.
• Thus the concept of "ruining" a Wikipedia article actually exists, where it's much fuzzier in normal wikis (modulo spammers of course).
• WikiWords are replaced with full-English phrases as hyperlinks.
• Articles are separated from Discussion pages.
• A mandatory "neutral point of view" (NPOV) replaces freedom and discussion.
• Some articles/topics are considered "not appropriate" for wikipedia and get deleted.

These differences weren't driven home to me until a recent discussion on the git mailing list about what sorts of pages are appropriate or not and who should or should not have admin access to those pages.

"Oh," I thought to myself. "So that's why wikis aren't wikis anymore."

The Wikipedia changes make sense; absolutely. They're great changes. Without them, Wikipedia almost certainly would have failed. But if you make those changes, you don't end up with a NitWiki, you end up with a Wikipedia.

Wikipedia-style wikis are written primarily for outsiders. There's a community, but it doesn't come through in the articles; if you want the community, you have to go elsewhere, to the User or Discussion pages. And of course, the User and Discussion pages aren't part of the product; they're the necessary junk produced as a side effect of producing the product. In Wikipedia, the community is overhead.

Same with the Git wiki. It's treated as a product - a set of documentation. Anything that's not "an information source about git" is unwelcome. Which is fine, I guess, and git already has an active mailing list where the community hangs out (as you can see from the vigorous discussion inside that thread I linked to), so there's no reason to have a community in the wiki. Having two community hangouts might actually be detrimental.

But I miss having a community wiki. The very original wiki, at c2.com Wiki, is such a wiki, and it's a beautiful thing. I was never a member of that wiki, but it's a fascinating place to lurk nonetheless. And our interns at NITI used to call NitWiki the "electric babysitter."

The c2 wikiers also apparently have their own version of this rant that I'm writing and you're currently reading. Theirs is called WikiIsNotWikipedia.

So yeah, a "real community wiki" is nothing like Wikipedia. That's not so hard. We can just start a new un-wikipedia-like wiki and just do it the "old way." It wasn't very hard to teach people to use a wiki who had never used a wiki and who were used to writing Microsoft Word documents for everything. Just stay away from MediaWiki (which strongly encourages the Wikipedia way of doing things) and you'll be fine.

Unfortunately, outside of wikis, the world itself has also been busy evolving, and in a way I don't know how to deal with. The new problem is: teams are the wrong size now. Thirty dedicated, full-time programmers was an ideal number for NitWiki. But who has 30 dedicated, full-time programmers now? I mean really dedicated? You can be a very successful Internet startup with way less than 30 developers. Maybe you need only two. Maybe those two aren't even working full time on the one project. And two people don't need a wiki.

That lack of dedication - lack of single-minded attention of one team to one goal - is a serious problem in building a project-oriented wiki. A lot of developers at NITI lived in the wiki; that was where they went when they were bored (instead of to a news site). It was where they went to ask a question, or answer a question, or design a new module, or plan a party. And as the wiki expanded, more and more WikiWords that you invented - planning to fill it in next - turned out to already exist, and to already have the content you meant to write. The WikiWord effect is kind of magical that way. It's really fun and rewarding when it happens.

But when there are too many wikis - more than one is too many - it stops happening. Instead, you get the opposite effect, and it's frustrating: you know you wrote a WikiWord page that describes a particular concept... but that page isn't in the current wiki, it's in some other wiki. So you use the WikiWord, but it comes up as missing. Argh. That defeats the whole point. You might as well be writing emails or Word documents or Wikipedia articles.

We even had this problem at NITI, and it was a bad one. In fact, we had two wikis: the internal-only NitWiki, and the public now-defunct OpenNit (open.nit.ca) wiki. In theory, we were supposed to put all our public-facing stuff in OpenNit, and all the private stuff in NitWiki. But it didn't work because you never knew quite what would be "private" until it happened. What if you're discussing something in public, but you want to add a related comment about some private project we're working on that we haven't announced yet? It kills the discussion. Or more likely, it results in all discussions just being done on the internal-only wiki, which is mostly what happened.

Internally, we had some really great documentation. But the documentation of our open source stuff was mostly garbage, because it was mostly on the internal-only site and nobody internally visited the public one.

We never solved this problem. I wish we had; our NitWiki was fun enough that I would have liked to share it with more people. Of course, people would have been more afraid to post if they had to do it in front of everyone on the Internet, so making it public never would have worked.

How do you create a vibrant community, but allow for private topics and discussion, but allow for public topics and discussion, and allow me to work for more than one company at a time with multiple private discussions, and have my WikiWords always end up pointing where they're supposed to?

I have no idea.

Dear Internet: please invent the solution to that for me. Thanks.

P.S. You don't get any bonus points for saying, "The answer is Google; just blog about it, don't worry about hyperlinks, and let Google full-text index the rest." That answer is funny, but it doesn't work for my private stuff.

Crystal ball

I now have had my foggy crystal ball for quite a long time. Its predictions are invariably gloomy and usually correct, but I am quite used to that and they won't keep me from giving you a few suggestions, even if it is merely an exercise in futility whose only effect is to make you feel guilty.

-- E.W. Dijkstra

bup backups are 2.7 times smaller than rsnapshot

Zoran has been doing some interesting tests of bup space efficiency and import performance vs. good old rsnapshot.

Lately, he's been working on an rsnapshot-to-bup importer tool. According to his most recent message, after importing the incremental backups from two servers, the bup repo was 4.6GB vs. the original rsnapshot disk usage of 12.6GB. That's 2.7 times smaller.

YMMV. Results may not be typical. Always consult a physician.

And Also...

As of last night, you can also restore your bup backups. Yeah, I know. It's that awesome.

::li

Browsing Google Groups Now Requires You to Log In

Someone emailed me to point out that the bup-list archive link in my previous post doesn't work unless you're logged in with a Google Account.

I know this used to work, once upon a time; it's not a restricted mailing list. And the Do I need a Google Account help page is the same as ever, ie. it says you can search, browse, and read public groups without an account. There are also instructions for how to join one of their lists via email without using the web interface, and I'm pretty sure that still works.

But if you go to groups.google.com (even the home page) and you're not logged in, you get bounced to the login screen.

But it's even more insidious...

Testing this was harder than I expected. In Google Chrome, the archive page above still worked after I logged out!! It even worked when I opened a "New Incognito Window," which supposedly eats all your cookies and guarantees you some form of privacy.

But when I got to the same page in Firefox or Safari - in which I haven't logged into Google for a long time - I do indeed get the login page.

So now we have two bugs:

• Google Groups isn't as open as (I assume) they actually intended to be because they didn't test carefully enough;
• Incognito Mode in Google Chrome (6.0.472.55/MacOS) doesn't successfully hide my identity.

And maybe a third, if you're generously assuming it's classified as a bug down at Google:

• Logging out of Google doesn't remove all my cookies, so they can still associate me with my account after I log out.

Metacommentary

I didn't bother posting this to Google's support mailing lists, because their support is so terrible that everyone knows it's a waste of time. This posting probably won't accomplish anything either, but at least I'll feel like someone heard me.

I did google around a bit to see if anyone had mentioned the problem, but I didn't see any matching results. Please let me know if I'm totally crazy and this problem doesn't really exist; but test carefully, as leftover cookies might be letting you see the site anyway.

Update 2010/09/09: Aha, not quite as scary as I was worried it might be: when I clear my Firefox cookies, it lets me access Google Groups without signing in. That suggests that it's actually old Google login credentials that are the problem; they cause it to jump to the login screen instead of just letting you through. Still a bug in Google Groups, but at least it means Chrome's private browsing seems to be okay after all.

Poetry Adaptation

A dark cave. In the middle, a caldron boiling. Thunder.

Enter the three Witches.

1 WITCH. Thrice the fishy fish hath twined.
2 WITCH. Thrice and once, the fam'ly dined.
3 WITCH. Mother cries: 'tis time! 'tis time!
1 WITCH. Round about the kitchen go;
   In the poison'd cookies throw.
   Snails, who without a thought,
   Captured slime that hell hath wrought;
   Swelter'd steam go up the flue,
   Boil thou first the Jana goo!
ALL. Weather, weather, boils and feather;
   Conjure Jana from the nether!

Witches ad-lib more two more verses with imaginary spell ingredients. Man, that's a lot of work.

MacFluffin enters, visibly upset.

MacFluffin. Ay! For towhich thou hast wrought herminity upon thy shoustest?

Witches cackle, then disappear in a puff of smoke, which goes up the flue.

MacFluffin. Aught! Wherefore this always happens?

Traffic shaping on a sheevaplug

In March 2009 I ordered two $100 Sheevaplugs (sometimes misspelled as Shivaplugs, hi Google). These are ARM-based Linux-capable "plug computers." Where by "plug computer" we mean that it's about the same size as a wall wart, and includes the AC/DC converter internally, so you don't need a separate stupid power adapter unit.

The devices are quite amazing for the price: gigabit ethernet (I don't know if they ever actually saturate it, though), 512 MB of flash, 512 MB of RAM, 1.2 GHz ARM processor, USB host and target ports, and an SD card slot. With Ubuntu pre-installed. And although you're definitely not getting as much "per GHz" as you would on a modern x86 processor, they're still surprisingly fast. Seems like Pentium-level performance, at least.

I ordered two of them because I figured I was going to do some experimenting, and since there was (is?) a 6-week lead time on ordering them, I figured it would be good to have a second one around in case I toasted the first one.

As it turned out, it's pretty hard to permanently break things; the only real way I can find to do it is to corrupt U-Boot, which in itself is kind of hard to do and I haven't managed it yet. And even if you do, the USB target port can be used as a serial console and as a JTAG port, which means you can reprogram the boot flash from any USB host computer, like, say, my laptop.

I haven't tried the JTAG yet, because I haven't needed to. But I believe the program necessary is called openocd. See this Sheevaplug-specific openocd tutorial. There is also something called a "sheevaplug installer," which purports to automate this process (it seems to use openocd internally), but such platform-specific tools are probably a bad idea. Hardware people seem to have a nasty propensity to make one-off solutions to their problems - after all, every circuit layout is unavoidably a one-off solution, so you get into the habit. Someday, there will be a post-Sheeva-plug, and we don't want to start a whole new software project with an installer for that. Do we? Well, someone will probably do it anyway. In any case, if you want to maximize your benefit-to-effort ratio, learn how to use openocd and forget about the "Sheevaplug installer."

If you haven't guessed yet, this article mostly exists to appease Google with a bunch of useful links, plus to write down the current overflowing contents of my web browser tabs in case I need the information again in the future, which I intend to do.

By the way, you can't order a Sheevaplug directly from Marvell (the chipset maker and the designer of the Sheevaplug reference design). The supplier I used was Globalscale Technologies, which seems to be rather clearly based in China, and has the customer service to prove it. There's probably a business model out there just for a trivial company who will stock these things and make a non-ridiculously-horrible web order form so it doesn't take 6 weeks to order one.

Now, the original Sheevaplug (still the only model of Sheevaplug, as far as I know, as of this writing) is a bit limited. The most terrible limitation is that it only has one network port and no wireless, which means, essentially, that the only thing you can possibly do with it is make a fileserver, because routing is impossible. Not surprisingly, the first and only Sheeva-based products (technically, that means using the Sheevaplug reference design, which uses the Marvell Fereoceon CPU core, part of the Marvell "Orion" (or is it "Kirkwood"?) product line) were all crappy fileservers. Essentially, Ubuntu + samba + terrible web UI + no attention to backups + low performance. Uninspiring.

Meanwhile, these systems are physically way more powerful than the original Weaver we made in 1999, and are also 1/10th the price, albeit software-wise much more crude than good old Weaver. The same old story: hardware improves, software backslides to keep up. But I digress.

Now, this Sheevaplug model was designed by Marvell to inspire developers (and hit the magic $99 price point!), and it's quite nicely done. Then companies like Globalscale manufacture it so Marvell doesn't have to. But interestingly, Globalscale has also done their own bit of innovation on top of the reference design. For example, their Guruplug Plus, which is essentially a Sheevaplug with 802.11g, Bluetooth, 2 x Gigabit ethernet, 2 x USB, an eSATA port, and a somewhat higher price tag ($129 instead of $99 USD). They also make a more expensive model with video out. Oh, and also it reputedly has terrible heat dissipation problems.

It's not surprising that the Guruplug has heat problems: while the Sheevaplug was designed with Western Hardware Designer Principles (make it look decent, not kill people, and largely useless), the Guruplug seems to have been designed using Chinese Hardware Designer Principles (make it amazingly cheap, flashy, potentially useful, and don't worry about those pesky safety underwriters). I'm not sure which I'd rather have. FWIW, I've run my "original" Sheevaplug full blast for long periods of time and it never gets any more than slightly warm, so any heat problems of the Guruplug are definitely not shared by the Sheeva design.

Nevertheless, everything factual (as opposed to opinionated) below applies to the Sheevaplug and maybe the Guruplug, but maybe not. I don't have a Guruplug.

Connecting to the serial console

Before you start any serious hacking on a Sheevaplug, you really want to get the serial console working. That's the only way to get to talk to U-boot, which you will want to do if you need to replace the kernel, boot from an external disk, etc.

The serial console uses the USB "target" port on the Sheevaplug. That is, you plug it into your USB "host" (eg. a laptop) and the Sheevaplug acts like a USB device, in this case a serial port or modem. (There's also a USB "host" port that works the other way around.) It even comes with a convenient cable. So you plug one end into your laptop, and the other end into the Sheevaplug.

In my case, I have a Macbook, which of course doesn't have the right driver for screwing around with Sheevaplugs, because such screwage is obviously not Apple Approved. Luckily though, I do all my real work in a Linux virtual machine running under Parallels (4.0.3848, in case you care).

One of the miracles of USB is that it's so simple - it's just a really fast serial port, after all - that it's trivially easy to forward it from your physical machine to your virtual one. So it's very easy to control the Sheevaplug from my virtual Linux box as if it was physically attached, and sure enough, it works perfectly.

The driver you need is called ftdi_sio. I had a pretty old kernel installed on my laptop (2.6.24), so the ftdi_sio driver didn't know the Sheevaplug's device ids. Luckily, Linux is designed for people like me, so it was easy to force the issue:

    modprobe ftdi_sio vendor=0x9e88 product=0x9e8f

(I got the above magic incantation from Nozell.com.)

If you have a 2.6.24 kernel like me, you can then talk to /dev/ttyUSB1, 8N1, 115200. /dev/ttyUSB0 also exists, but that seems to be a bug since it doesn't do anything. I used minicom at first, but then got tired of it and wrote port.py, a totally trivial terminal connector that lets your terminal emulate VT100 the way God intended, rather than faking it up (badly) by hand. To use it, you'll also need a copy of the awesome options.py from bup, downloadable at that link for your convenience. You just run it like this:

    port.py /dev/ttyUSB1 115200

Since my next step was to download and compile a new kernel for my Sheevaplug, I figured I might as well upgrade my laptop's Linux kernel at the same time. I picked Linus's standard, unadulterated 2.6.34 kernel. Two items of good news with this version: you no longer need the special vendor/product codes for the ftdi driver, and the mysterious broken /dev/ttyUSB0 no longer appears. (Which means the working /dev/ttyUSB1 is now renamed to /dev/ttyUSB0, so don't be surprised.)

Using a Sheevaplug as a router/firewall/shaper

I said earlier that a Sheevaplug only has one ethernet port, which makes it useless for anything but a fileserver. That was a slight lie; in fact, because it has a USB port, you can add whatever you want. I added an external ethernet device, the Trendnet TU-ET100C. It cost me about $40 at the overpriced local computer store on an island in the middle of nowhere, which isn't bad. It works with the Linux-USB "pegasus" ethernet driver, and it's fine.

Interestingly, either the hardware or the driver for the Trendnet turns out not to support automatic crossover cable detection/emulation. It's been so long since I ran into this problem that I'd forgotten what a hard-to-diagnose pain in the neck it is when you run into it. Someone put a lot of work into fixing that problem, and we as a society have already forgotten all about it.

Let's all just have a moment of silence to celebrate automatic ethernet port sensing.

...

Right, so, anyway, this USB device doesn't have automatic port sensing. Which is really not a big deal, once you know about it, but it sure confused me for about half an hour because I didn't have a crossover cable. In case you're wondering, the onboard ethernet on the Sheevaplug does do port sensing. Nice job, guys. So I just swapped ports and my problems were solved.

Okay, so, after all that, we now have a $140 device ($100 box + $40 ethernet) that can do routing. Right? Well, no, because the kernel is missing all the happy fun bits like iptables and iproute2. We need to upgrade the kernel.

Building the new kernel

Ah, but which kernel? On my x86 laptop, I used Linus's latest stable Linux kernel (2.6.34), but I gather that's not the best thing to do for ARM platforms right now. All the latest Marvell patches have been merged into Linus's 2.6.35-rc1, but I know better than to use a Linus -rc1 release. Instead, I tracked down Marvell's repo and used their "stable-2.6.31" branch, which corresponds to git commit 17e9ccfb3d083d9ad566f867d47de4cf37511cce, aka v2.6.31.1-277-g17e9ccf. It seems to work fine. By the way, here's my kernel .config file. It probably has some extra stuff enabled that's not necessary, but it works for me, and it works with my USB ethernet adapter.

For reference, the interesting git repositories are: - Linus: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git - Marvell: git://git.marvell.com/orion.git

Supposedly you can build the kernel with a cross-compiler. I didn't bother trying to do this. I just installed gcc and friends on the provided Ubuntu distribution and built it there. Warning: it took a long time. This processor isn't exactly running at warp speed, but worse, I was building it all on the cheapest 8GB SD card I could get at Future Shop a year and a half ago. Result: slow disk writes. Very slow. But I couldn't use a USB disk because my USB port was taken up by the USB ethernet, and I didn't want to blow more money on a USB hub.

plugcomputer.org has a very nice article with all you need to know to build a kernel for the Sheevaplug. Most of what you need to know is what I just wrote, plus that you need to 'make uImage' to get a U-Boot image, and the 'make uImage' process requires you to have a 'mkimage' program that comes with U-Boot. They have links in that article, but the particular one I used (a pre-compiled binary for x86) was one I downloaded from a random place on the net. I can't find it anymore, so in the interest of sharing the love, I have mirrored mkimage.gz here. gunzip it, and rename it to /usr/local/bin/mkimage and you'll be fine.

Fixing broken Ubuntu

Now that you've upgraded a bunch of packages in order to install the compiler, you'll probably notice that your system is completely scrambled and won't let you log in anymore. The bad news is, this sucks. The good news is, maybe you read this far in advance so you're still logged in so you can fix it. The magic incantation to fix it is:

    passwd -d root
    passwd -d root
    passwd root

Somehow, the passwords that got written into /etc/shadow didn't get encrypted properly, and some kind of Ubuntu upgrade "fixes" it, but makes future password checks fail. It's also suspicious that you have to delete the password twice, and that only changing the password doesn't work. I don't want to know. The /etc/shadow file has always been a mystery to me. Thanks, anyway, Ubuntu.

In case you managed to log out and now can't back log in, all is not lost. You need to run this series of commands in U-Boot to get into an emergency recovery mode:

    setenv bootargs console=ttyS0,115200
       mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs)
       rw root=/dev/mtdblock1 init=/bin/bash
    saveenv
    reset

(You have to use init=/bin/bash. Using the normal sysvinit "emergency" command doesn't work, because it makes you enter your password, which has always been a pretty stupid idea. Because if you can type "emergency" you can also type "init=/bin/bash", so there's no added security, just more hassle. It used to not be stupid and just happily logged you in. But I digress.)

And then run the 'passwd' commands above, sync, and reboot.

Booting the new kernel

You can't, because you have to upgrade U-Boot first. Something about the bootloader operation in the more recent kernels is different from the experimental bootloader operation in the experimental Sheevaplug development kit, so it won't work by default.

Upgrading U-Boot

Luckily this is pretty easy. I found some nice instructions for upgrading U-Boot as part of a document about installing Debian on a Sheevaplug.

The easiest way to do this part is to use a TFTP server. Any TFTP server will do fine here, since you'll only do it once. However, WvTFTPd (wvtftp) is still the world's fastest TFTP server, and by comparison all the other ones are... "retarded," in the literal sense of not using negative latency.

WvTFTP really is at least 3-5x faster (and even more on a network with significant packet drops) than a typical TFTP server. Seriously. If you're netbooting embedded devices and you're still using a TFTP server that sucks, you're wasting your time. Stop doing that.

Okay, now we really boot the new kernel

Just follow the instructions and you'll be fine. However, I'm terrible at following instructions without knowing what I'm doing, so I didn't follow them and got myself into a bit of a mess which required a certain amount of Googling to get myself out of.

Here are the really important points that you shouldn't forget:

First, the new version of U-Boot supports booting both old-style kernels (like the one it came with) and new-style ones (like the current release kernels). Strangely, it defaults to the former, not the latter. To switch it into a mode capable of booting new-style kernels, you have to do (from the U-boot serial console) setenv mainlineLinux yes and saveenv and then reboot.

Note note note! This may sound obvious in retrospect, but once you've done this, booting your old kernel will stop working, because it's not a mainline kernel. Right? Well, it's obvious if you remember, hours of panic later, that you've run this command in the first place. Ha ha. Yeah. Anyway, if you want to boot your old kernel again, just run setenv mainlineLinux no and saveenv and then reboot.

The other thing you need to do is tell Linux what kind of hardware you're running on. Once upon a time, before the invention of so called plug-and-play, Linux used to "probe" for hardware devices automatically, so you didn't have to do such silly things. Linux had the very best hardware probing anywhere I've ever seen. I personally spent hours optimizing my Linux ISA arcnet driver's probing routines, and I personally solved the mystery of the "IRQ autoprobing in kernel modules causes a crash" in Linux 1.1.x or 1.2.x or so. But those days are long gone. Nowadays, we're back to specifying the hardware platform by hand, as if we were running Windows 3.0, which by the way is at least 10x smaller than the Sheevaplug's default Ubuntu install and it has a GUI. Hardware improves, software backslides. But I digress.

If you have an original Sheevaplug like mine, the magic incantation (at the U-boot console) is:

    setenv arcNumber 2097
    saveenv
    reset

Yes, you do have to saveenv. You might also have to reset. Don't be like me, and think you can "try a temporary experiment" by setting the environment variable and not saving it in the hopes it won't screw up. It will fail.

Note note note! Unlike the mainlineLinux option, setting the arcNumber option does not mess up your old kernel. That's because mainlineLinux is interpreted by U-Boot (I assume), which does something different depending if it's a new or old kernel. But arcNumber is interpreted by Linux, and the old Linux didn't know to look for it. (This also explains why you have to saveenv; Linux won't be able to find it if it's not saved.)

Once again, I refer you to the Debian Sheevaplug instructions for how to get your kernel booted, depending on what device you want to boot from.

Note note note! I don't recommend installing the new kernel into the "boot flash" memory segment until you're really sure it works, because that will overwrite the old kernel, and then you might be stuck. For experimenting, boot from an SD card or TFTP instead.

I personally recommend starting your experiments with TFTP, since that's the quickest way to change out your kernel while you're experimenting. I went from there to using an SD card, which gave me more disk space for messing around. Both boot methods are described at the Debian Sheevaplug link above.

Recovering after you've mangled all your U-Boot settings

If you're like me, you'll inevitably end up getting into a situation where you really wish you could just get back to your old kernel and tweak some more stuff from a nice, safe, userland Linux process. Here's the magic U-Boot incantation that works for me:

    setenv bootargs console=ttyS0,115200
        mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) rw
        root=/dev/mtdblock1
    setenv bootargs_root root=/dev/mtdblock2 ro
    setenv bootcmd 'nand read.e 0x800000 0x100000 0x400000; bootm 0x800000'
    setenv cesvcid ULULULULULULPPULULULULULDA
    setenv console console=ttyS0,115200
    setenv mainlineLinux no
    setenv run_diag no
    saveenv
    reset

Not so shockingly, the "resetenv" command resets the environment to "factory defaults," but apparently the guys down at the factory do not like to have their Sheevaplugs in a state that is actually bootable. Ha ha. Sucker.

Actually routing those packets

After all that, you're finally ready to set up your Sheevaplug to be a router. The good news: now that you have the @$!@#!@ thing booted, it's just like any old Linux box. iptables and iproute2 work exactly like they would on any Ubuntu system. Which is to say, they're amazingly obtuse and complex, and in the case of iproute2, the documentation outright lies.

I do have it working now, if that's any consolation.

Some other time, I'll write about my amazing adventures with token buckets, stochastic fair queuing, traffic shaping vs. policing, random early detection, explicit congestion notification (oh man, ECN performance actually is amazing when you have a router that uses it!) and how it really is possible to have a "usable" 256k Internet link even when you're downloading 10 files simultaneously and your provider thought a 5 second downstream DSL queue length was a good idea and you technically can't even shape downstream traffic.

(I put "usable" in quotes because 256k is getting to be pretty unusable nowadays even when it's not loaded down. Over time, you lose sight of the fact that that the 6 megabit link back at home really is 23 times faster than a "perfectly acceptable" 256 kbit link used to be. But the speed mostly doesn't bother me; it's the terrible queuing. Example: before I installed the shaping/policing, leaving Google Reader open in the background totally killed all other traffic. And it's mostly just downloading text!)

Too bad Linux's rather awesome RED traffic shaper doesn't work on downstream traffic, which means you can't use ECN on downstream traffic, so you end up dropping lots of packets all over the place. And too bad that if you use the "intermediate device" method (basically, egress queuing on the packets going to your internal network) you add tons of latency - of course.

But I think I know how to fix it. More later.