Traffic shaping on a sheevaplug

In March 2009 I ordered two $100 Sheevaplugs (sometimes misspelled as Shivaplugs, hi Google). These are ARM-based Linux-capable "plug computers." Where by "plug computer" we mean that it's about the same size as a wall wart, and includes the AC/DC converter internally, so you don't need a separate stupid power adapter unit.

The devices are quite amazing for the price: gigabit ethernet (I don't know if they ever actually saturate it, though), 512 MB of flash, 512 MB of RAM, 1.2 GHz ARM processor, USB host and target ports, and an SD card slot. With Ubuntu pre-installed. And although you're definitely not getting as much "per GHz" as you would on a modern x86 processor, they're still surprisingly fast. Seems like Pentium-level performance, at least.

I ordered two of them because I figured I was going to do some experimenting, and since there was (is?) a 6-week lead time on ordering them, I figured it would be good to have a second one around in case I toasted the first one.

As it turned out, it's pretty hard to permanently break things; the only real way I can find to do it is to corrupt U-Boot, which in itself is kind of hard to do and I haven't managed it yet. And even if you do, the USB target port can be used as a serial console and as a JTAG port, which means you can reprogram the boot flash from any USB host computer, like, say, my laptop.

I haven't tried the JTAG yet, because I haven't needed to. But I believe the program necessary is called openocd. See this Sheevaplug-specific openocd tutorial. There is also something called a "sheevaplug installer," which purports to automate this process (it seems to use openocd internally), but such platform-specific tools are probably a bad idea. Hardware people seem to have a nasty propensity to make one-off solutions to their problems - after all, every circuit layout is unavoidably a one-off solution, so you get into the habit. Someday, there will be a post-Sheeva-plug, and we don't want to start a whole new software project with an installer for that. Do we? Well, someone will probably do it anyway. In any case, if you want to maximize your benefit-to-effort ratio, learn how to use openocd and forget about the "Sheevaplug installer."

If you haven't guessed yet, this article mostly exists to appease Google with a bunch of useful links, plus to write down the current overflowing contents of my web browser tabs in case I need the information again in the future, which I intend to do.

By the way, you can't order a Sheevaplug directly from Marvell (the chipset maker and the designer of the Sheevaplug reference design). The supplier I used was Globalscale Technologies, which seems to be rather clearly based in China, and has the customer service to prove it. There's probably a business model out there just for a trivial company who will stock these things and make a non-ridiculously-horrible web order form so it doesn't take 6 weeks to order one.

Now, the original Sheevaplug (still the only model of Sheevaplug, as far as I know, as of this writing) is a bit limited. The most terrible limitation is that it only has one network port and no wireless, which means, essentially, that the only thing you can possibly do with it is make a fileserver, because routing is impossible. Not surprisingly, the first and only Sheeva-based products (technically, that means using the Sheevaplug reference design, which uses the Marvell Fereoceon CPU core, part of the Marvell "Orion" (or is it "Kirkwood"?) product line) were all crappy fileservers. Essentially, Ubuntu + samba + terrible web UI + no attention to backups + low performance. Uninspiring.

Meanwhile, these systems are physically way more powerful than the original Weaver we made in 1999, and are also 1/10th the price, albeit software-wise much more crude than good old Weaver. The same old story: hardware improves, software backslides to keep up. But I digress.

Now, this Sheevaplug model was designed by Marvell to inspire developers (and hit the magic $99 price point!), and it's quite nicely done. Then companies like Globalscale manufacture it so Marvell doesn't have to. But interestingly, Globalscale has also done their own bit of innovation on top of the reference design. For example, their Guruplug Plus, which is essentially a Sheevaplug with 802.11g, Bluetooth, 2 x Gigabit ethernet, 2 x USB, an eSATA port, and a somewhat higher price tag ($129 instead of $99 USD). They also make a more expensive model with video out. Oh, and also it reputedly has terrible heat dissipation problems.

It's not surprising that the Guruplug has heat problems: while the Sheevaplug was designed with Western Hardware Designer Principles (make it look decent, not kill people, and largely useless), the Guruplug seems to have been designed using Chinese Hardware Designer Principles (make it amazingly cheap, flashy, potentially useful, and don't worry about those pesky safety underwriters). I'm not sure which I'd rather have. FWIW, I've run my "original" Sheevaplug full blast for long periods of time and it never gets any more than slightly warm, so any heat problems of the Guruplug are definitely not shared by the Sheeva design.

Nevertheless, everything factual (as opposed to opinionated) below applies to the Sheevaplug and maybe the Guruplug, but maybe not. I don't have a Guruplug.

Connecting to the serial console

Before you start any serious hacking on a Sheevaplug, you really want to get the serial console working. That's the only way to get to talk to U-boot, which you will want to do if you need to replace the kernel, boot from an external disk, etc.

The serial console uses the USB "target" port on the Sheevaplug. That is, you plug it into your USB "host" (eg. a laptop) and the Sheevaplug acts like a USB device, in this case a serial port or modem. (There's also a USB "host" port that works the other way around.) It even comes with a convenient cable. So you plug one end into your laptop, and the other end into the Sheevaplug.

In my case, I have a Macbook, which of course doesn't have the right driver for screwing around with Sheevaplugs, because such screwage is obviously not Apple Approved. Luckily though, I do all my real work in a Linux virtual machine running under Parallels (4.0.3848, in case you care).

One of the miracles of USB is that it's so simple - it's just a really fast serial port, after all - that it's trivially easy to forward it from your physical machine to your virtual one. So it's very easy to control the Sheevaplug from my virtual Linux box as if it was physically attached, and sure enough, it works perfectly.

The driver you need is called ftdi_sio. I had a pretty old kernel installed on my laptop (2.6.24), so the ftdi_sio driver didn't know the Sheevaplug's device ids. Luckily, Linux is designed for people like me, so it was easy to force the issue:

    modprobe ftdi_sio vendor=0x9e88 product=0x9e8f

(I got the above magic incantation from Nozell.com.)

If you have a 2.6.24 kernel like me, you can then talk to /dev/ttyUSB1, 8N1, 115200. /dev/ttyUSB0 also exists, but that seems to be a bug since it doesn't do anything. I used minicom at first, but then got tired of it and wrote port.py, a totally trivial terminal connector that lets your terminal emulate VT100 the way God intended, rather than faking it up (badly) by hand. To use it, you'll also need a copy of the awesome options.py from bup, downloadable at that link for your convenience. You just run it like this:

    port.py /dev/ttyUSB1 115200

Since my next step was to download and compile a new kernel for my Sheevaplug, I figured I might as well upgrade my laptop's Linux kernel at the same time. I picked Linus's standard, unadulterated 2.6.34 kernel. Two items of good news with this version: you no longer need the special vendor/product codes for the ftdi driver, and the mysterious broken /dev/ttyUSB0 no longer appears. (Which means the working /dev/ttyUSB1 is now renamed to /dev/ttyUSB0, so don't be surprised.)

Using a Sheevaplug as a router/firewall/shaper

I said earlier that a Sheevaplug only has one ethernet port, which makes it useless for anything but a fileserver. That was a slight lie; in fact, because it has a USB port, you can add whatever you want. I added an external ethernet device, the Trendnet TU-ET100C. It cost me about $40 at the overpriced local computer store on an island in the middle of nowhere, which isn't bad. It works with the Linux-USB "pegasus" ethernet driver, and it's fine.

Interestingly, either the hardware or the driver for the Trendnet turns out not to support automatic crossover cable detection/emulation. It's been so long since I ran into this problem that I'd forgotten what a hard-to-diagnose pain in the neck it is when you run into it. Someone put a lot of work into fixing that problem, and we as a society have already forgotten all about it.

Let's all just have a moment of silence to celebrate automatic ethernet port sensing.

...

Right, so, anyway, this USB device doesn't have automatic port sensing. Which is really not a big deal, once you know about it, but it sure confused me for about half an hour because I didn't have a crossover cable. In case you're wondering, the onboard ethernet on the Sheevaplug does do port sensing. Nice job, guys. So I just swapped ports and my problems were solved.

Okay, so, after all that, we now have a $140 device ($100 box + $40 ethernet) that can do routing. Right? Well, no, because the kernel is missing all the happy fun bits like iptables and iproute2. We need to upgrade the kernel.

Building the new kernel

Ah, but which kernel? On my x86 laptop, I used Linus's latest stable Linux kernel (2.6.34), but I gather that's not the best thing to do for ARM platforms right now. All the latest Marvell patches have been merged into Linus's 2.6.35-rc1, but I know better than to use a Linus -rc1 release. Instead, I tracked down Marvell's repo and used their "stable-2.6.31" branch, which corresponds to git commit 17e9ccfb3d083d9ad566f867d47de4cf37511cce, aka v2.6.31.1-277-g17e9ccf. It seems to work fine. By the way, here's my kernel .config file. It probably has some extra stuff enabled that's not necessary, but it works for me, and it works with my USB ethernet adapter.

For reference, the interesting git repositories are: - Linus: git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git - Marvell: git://git.marvell.com/orion.git

Supposedly you can build the kernel with a cross-compiler. I didn't bother trying to do this. I just installed gcc and friends on the provided Ubuntu distribution and built it there. Warning: it took a long time. This processor isn't exactly running at warp speed, but worse, I was building it all on the cheapest 8GB SD card I could get at Future Shop a year and a half ago. Result: slow disk writes. Very slow. But I couldn't use a USB disk because my USB port was taken up by the USB ethernet, and I didn't want to blow more money on a USB hub.

plugcomputer.org has a very nice article with all you need to know to build a kernel for the Sheevaplug. Most of what you need to know is what I just wrote, plus that you need to 'make uImage' to get a U-Boot image, and the 'make uImage' process requires you to have a 'mkimage' program that comes with U-Boot. They have links in that article, but the particular one I used (a pre-compiled binary for x86) was one I downloaded from a random place on the net. I can't find it anymore, so in the interest of sharing the love, I have mirrored mkimage.gz here. gunzip it, and rename it to /usr/local/bin/mkimage and you'll be fine.

Fixing broken Ubuntu

Now that you've upgraded a bunch of packages in order to install the compiler, you'll probably notice that your system is completely scrambled and won't let you log in anymore. The bad news is, this sucks. The good news is, maybe you read this far in advance so you're still logged in so you can fix it. The magic incantation to fix it is:

    passwd -d root
    passwd -d root
    passwd root

Somehow, the passwords that got written into /etc/shadow didn't get encrypted properly, and some kind of Ubuntu upgrade "fixes" it, but makes future password checks fail. It's also suspicious that you have to delete the password twice, and that only changing the password doesn't work. I don't want to know. The /etc/shadow file has always been a mystery to me. Thanks, anyway, Ubuntu.

In case you managed to log out and now can't back log in, all is not lost. You need to run this series of commands in U-Boot to get into an emergency recovery mode:

    setenv bootargs console=ttyS0,115200
       mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs)
       rw root=/dev/mtdblock1 init=/bin/bash
    saveenv
    reset

(You have to use init=/bin/bash. Using the normal sysvinit "emergency" command doesn't work, because it makes you enter your password, which has always been a pretty stupid idea. Because if you can type "emergency" you can also type "init=/bin/bash", so there's no added security, just more hassle. It used to not be stupid and just happily logged you in. But I digress.)

And then run the 'passwd' commands above, sync, and reboot.

Booting the new kernel

You can't, because you have to upgrade U-Boot first. Something about the bootloader operation in the more recent kernels is different from the experimental bootloader operation in the experimental Sheevaplug development kit, so it won't work by default.

Upgrading U-Boot

Luckily this is pretty easy. I found some nice instructions for upgrading U-Boot as part of a document about installing Debian on a Sheevaplug.

The easiest way to do this part is to use a TFTP server. Any TFTP server will do fine here, since you'll only do it once. However, WvTFTPd (wvtftp) is still the world's fastest TFTP server, and by comparison all the other ones are... "retarded," in the literal sense of not using negative latency.

WvTFTP really is at least 3-5x faster (and even more on a network with significant packet drops) than a typical TFTP server. Seriously. If you're netbooting embedded devices and you're still using a TFTP server that sucks, you're wasting your time. Stop doing that.

Okay, now we really boot the new kernel

Just follow the instructions and you'll be fine. However, I'm terrible at following instructions without knowing what I'm doing, so I didn't follow them and got myself into a bit of a mess which required a certain amount of Googling to get myself out of.

Here are the really important points that you shouldn't forget:

First, the new version of U-Boot supports booting both old-style kernels (like the one it came with) and new-style ones (like the current release kernels). Strangely, it defaults to the former, not the latter. To switch it into a mode capable of booting new-style kernels, you have to do (from the U-boot serial console) setenv mainlineLinux yes and saveenv and then reboot.

Note note note! This may sound obvious in retrospect, but once you've done this, booting your old kernel will stop working, because it's not a mainline kernel. Right? Well, it's obvious if you remember, hours of panic later, that you've run this command in the first place. Ha ha. Yeah. Anyway, if you want to boot your old kernel again, just run setenv mainlineLinux no and saveenv and then reboot.

The other thing you need to do is tell Linux what kind of hardware you're running on. Once upon a time, before the invention of so called plug-and-play, Linux used to "probe" for hardware devices automatically, so you didn't have to do such silly things. Linux had the very best hardware probing anywhere I've ever seen. I personally spent hours optimizing my Linux ISA arcnet driver's probing routines, and I personally solved the mystery of the "IRQ autoprobing in kernel modules causes a crash" in Linux 1.1.x or 1.2.x or so. But those days are long gone. Nowadays, we're back to specifying the hardware platform by hand, as if we were running Windows 3.0, which by the way is at least 10x smaller than the Sheevaplug's default Ubuntu install and it has a GUI. Hardware improves, software backslides. But I digress.

If you have an original Sheevaplug like mine, the magic incantation (at the U-boot console) is:

    setenv arcNumber 2097
    saveenv
    reset

Yes, you do have to saveenv. You might also have to reset. Don't be like me, and think you can "try a temporary experiment" by setting the environment variable and not saving it in the hopes it won't screw up. It will fail.

Note note note! Unlike the mainlineLinux option, setting the arcNumber option does not mess up your old kernel. That's because mainlineLinux is interpreted by U-Boot (I assume), which does something different depending if it's a new or old kernel. But arcNumber is interpreted by Linux, and the old Linux didn't know to look for it. (This also explains why you have to saveenv; Linux won't be able to find it if it's not saved.)

Once again, I refer you to the Debian Sheevaplug instructions for how to get your kernel booted, depending on what device you want to boot from.

Note note note! I don't recommend installing the new kernel into the "boot flash" memory segment until you're really sure it works, because that will overwrite the old kernel, and then you might be stuck. For experimenting, boot from an SD card or TFTP instead.

I personally recommend starting your experiments with TFTP, since that's the quickest way to change out your kernel while you're experimenting. I went from there to using an SD card, which gave me more disk space for messing around. Both boot methods are described at the Debian Sheevaplug link above.

Recovering after you've mangled all your U-Boot settings

If you're like me, you'll inevitably end up getting into a situation where you really wish you could just get back to your old kernel and tweak some more stuff from a nice, safe, userland Linux process. Here's the magic U-Boot incantation that works for me:

    setenv bootargs console=ttyS0,115200
        mtdparts=nand_mtd:0x400000@0x100000(uImage),0x1fb00000@0x500000(rootfs) rw
        root=/dev/mtdblock1
    setenv bootargs_root root=/dev/mtdblock2 ro
    setenv bootcmd 'nand read.e 0x800000 0x100000 0x400000; bootm 0x800000'
    setenv cesvcid ULULULULULULPPULULULULULDA
    setenv console console=ttyS0,115200
    setenv mainlineLinux no
    setenv run_diag no
    saveenv
    reset

Not so shockingly, the "resetenv" command resets the environment to "factory defaults," but apparently the guys down at the factory do not like to have their Sheevaplugs in a state that is actually bootable. Ha ha. Sucker.

Actually routing those packets

After all that, you're finally ready to set up your Sheevaplug to be a router. The good news: now that you have the @$!@#!@ thing booted, it's just like any old Linux box. iptables and iproute2 work exactly like they would on any Ubuntu system. Which is to say, they're amazingly obtuse and complex, and in the case of iproute2, the documentation outright lies.

I do have it working now, if that's any consolation.

Some other time, I'll write about my amazing adventures with token buckets, stochastic fair queuing, traffic shaping vs. policing, random early detection, explicit congestion notification (oh man, ECN performance actually is amazing when you have a router that uses it!) and how it really is possible to have a "usable" 256k Internet link even when you're downloading 10 files simultaneously and your provider thought a 5 second downstream DSL queue length was a good idea and you technically can't even shape downstream traffic.

(I put "usable" in quotes because 256k is getting to be pretty unusable nowadays even when it's not loaded down. Over time, you lose sight of the fact that that the 6 megabit link back at home really is 23 times faster than a "perfectly acceptable" 256 kbit link used to be. But the speed mostly doesn't bother me; it's the terrible queuing. Example: before I installed the shaping/policing, leaving Google Reader open in the background totally killed all other traffic. And it's mostly just downloading text!)

Too bad Linux's rather awesome RED traffic shaper doesn't work on downstream traffic, which means you can't use ECN on downstream traffic, so you end up dropping lots of packets all over the place. And too bad that if you use the "intermediate device" method (basically, egress queuing on the packets going to your internal network) you add tons of latency - of course.

But I think I know how to fix it. More later.