On Montreal Highway Construction

Last week I tried to pick someone up at the Montreal airport and got stuck in the most ridiculous traffic jam I've ever been part of, taking 2 hours to travel 4 kilometers.

pmccurdy sent me a story from the Montreal Gazette explaining the situation. Tip to Montreal highway industry: signs that tell people before getting to the insane traffic jam that you have pre-scheduled an insane traffic jam for the next 30 days, so they can maybe take an alternate route, would be more than welcome.

My favourite comment on the article was a response to someone who compared Montreal's construction industry to that of a third-world country. The response was:

In a real top notch third world city, people wouldn't have to walk to the airport. There would be mopeds and donkeys and bicycle rikshaws etc.

You know, if someone had passed me on a donkey, I think it would have cheered me up.

Call to Arms (DRAFT)

Something something
Something something
AND WE'RE NOT GOING TO TAKE IT ANYMORE!

Crowd: YEEEAAH!

(Cheering and riots ensue)

Avery on Entrepreneurship

[...] horrible bloody death [...] in way over our heads [...] slash and burn [...] And when someday they finally find your rotting corpse amidst the carnage, nobody's going to care if your underwear was clean or you weighed 5 pounds over optimal or you had a Positive Outlook on Life.

-- Me

Okay, it was a bit of a digression.

Who's the host, and who's the guest?

So there's this big announcement today that Microsoft Open Sourced some Linux drivers: specifically the drivers that make Linux run well under Microsoft's hypervisor.

Great, right? This is the new, open minded Microsoft, right?

Well, sort of. It's also the same old Microsoft. While we're all caught up in making Linux better, they're focusing on the real question about virtualization: which one is the host, and which one is the guest?

Because the answer to this question defines whether Microsoft gets to keep their monopoly or not in the long run. Cool, huh?

Look at it this way: right now, 100% of the Windows apps I need to run run just fine on Windows XP. And Windows XP runs just fine under virtualization. In fact, my Linux box + Virtualbox + Windows XP runs faster than Windows Vista. (And Windows 7 is dodging the question by running XP in virtualization anyway.)

In fact, because my virtual hardware can stay the same forever even if my physical hardware gets upgraded, a virtual Windows XP actually has a potentially longer lifespan than a physical one. After all, I can still run Windows 98 under Win4lin under kvm and it'll work forever.

Microsoft's vendor lock-in is all about the fact that you need the new version of Windows to run your apps. They've already started pressuring vendors to stop making XP drivers for their new devices - and it'll get much worse with Windows 7.

But with virtualization, maybe you'll just keep your copy of XP - just about everyone in Canada and the U.S. has one or more legal copies of XP at this point - and run it under whatever OS you want. Suddenly, no more vendor lock in. You can choose whatever OS you want to run on your PC - or Mac - and you don't lose. In fact, you could probably make a business model out of just making a stripped-down Linux kernel designed just to run a virtualized XP, only with updated drivers.

So the question, then is: which OS will you choose, now that you have the choice?

Well, which one is better for virtualizing all the OSes I might need to run?

As of the recent announcement, Windows 7 can run Windows and Linux VMs really fast. Linux can run Linux VMs really fast, but it can only run Windows at "normal speed." (Xen actually built accelerated drivers for Windows too back when it was a research project - but Microsoft cut them off eventually, IIRC.)

You know what? Most people will probably choose Windows 7 as their host anyway, just out of habit. But habits can change over time. Microsoft needs you to have a better reason than just that.

Asynchronizing the Synchronous

We've been working on some further speed optimizations for our EQL Data plugin for Microsoft Access. This involves rewriting some parts (which were previously in VBA) into C++.

Now, to do this, I decided it would be nicest if we could write a simple command-line tool first, and then afterwards, adapt it into a GUI plugin. That makes testing easier and makes the code nice and linear, at least at first. So that's what I did.

The next steps was to convert it to something more appropriate for a GUI. What's the #1 concern of GUI code? It has to be asynchronous: you're not allowed to block for too long, or else the system seems unresponsive. Usually this means either a) using threads, or b) rewriting your program to use a state machine so you can give up your time slice occasionally, or more realistically c) both, because using a thread gives the best interactive performance but you still have to be able to abort them safely.

So I figured we'd go with (b) and then (c). Then, I thought, do I really have to actually change the code at all? I should be able to make my synchronous function effectively asynchronous without any changes, using a trick like this:

#include "wvcont.h"

typedef void LogFunc(const char *msg);

static void *long_running_function(LogFunc *log)
{
    char msg[100];
    for (int i = 1; i <= 5; i++)
    {
        sprintf(msg, "Hello world! %d\n", i);
        log(msg);
    }
    return (void *)1;
}

static void do_log(const char *msg)
{
    fprintf(stderr, "%s", msg);
    WvCont::yield(0);
}

int main()
{
    WvCont c(wv::bind(long_running_function, do_log));
    while (!c())
        fprintf(stderr, "If you see this, yielding worked.\n");
    return 0;
}

Now, using this trick here isn't really necessary; I have total control over long_running_func() and I could just break it into more pieces, or make a class, or whatever. But the result is actually pretty elegant.

Not always a waste of time

In fact, however, when I first used this trick, it was the only way to get the results I wanted. That was a few years ago, when I was working on the Nitix ExchangeIt plugin for Outlook. (I can't link to that, because it effectively no longer exists now that IBM bought my company. If you google for ExchangeIt, you'll mostly find that the name has been recycled by other people. The one on Google Code is definitely not it.)

The problem we ran into in ExchangeIt was with the ITnef::ExtractProps function (if I remember right). This function is kind of neat: you give it an IStream (an object implementing a simple streaming read interface) and it automatically reads the stream, decodes the results, turns them into a MAPI message, and returns when it's done.

Great, right? You just wrap whatever input stream you have (in our case, it was an auto-zlib-decompressing, SSL-enabled TCP socket) in an IStream, pass the IStream to ExtractProps, and off it goes ... except it's synchronous. While you wait for ExtractProps to finish, your thread is stuck. And because it's a network connection, it might be stuck for a long time.

To resolve the problem, we used the same trick as above: our IStream implementation actually called WvCont::yield() whenever ExtractProps asked it to Read(). The program's mainloop, then, could resume the WvCont only when data was ready on the stream; the rest of the time, the MAPI function was just waiting patiently to be resumed.

All this with no threads, and thus no locking, no race conditions, and no kernel-level task switching overhead.

(WvCont is part of the WvStreams library that I helped to write. Apologies go to mag and drheld, who tried to use a predecessor to WvCont when writing WvFuse/FunFS, before I had managed to make it awesome.)

::nv

No, you don't get to ruin my stuff anymore

I first wrote about the knowledgebase project I created, code-named GourD, almost exactly three years ago. At the time, it was one of my favourite achievements, aside from my company's main product itself.

...

Things have changed since then. What they've done to the product is kind of terrible.¹ Worse, they paid my friends (former co-workers) to do it. But that's another story.

The short version is that, through this exciting learning experience, I now know the thing I hate most of all in the whole world. I hate it when people take something that has the potential to be good, and they make it bad. But most of all, I hate it when they do that to me.

I'm sure half my readers will unsubscribe just because I say this, but: one of my favourite books of all time is The Fountainhead.² You have either already read it or you never will, so I don't feel bad in ruining the story for you. The climax is when the hero blows up a building he designed so that it can't be ruined in the name of popularity/fashion/etc. It's all a bit overdramatic - or so I thought at the time.

Today I found myself wishing that I had just blown it all up rather than selling it, and to hell with the consequences.³

I mean, really, truly, honestly. It would have been a global net improvement. The world really would be no worse off, and I wouldn't have to be ashamed to be associated with it anymore.⁴ Yeah, they'd probably sue the pants off me, and (unlike in the book) it's extremely unlikely I'd get a sympathetic judge. But who cares? At least that's one less steaming pile of crap in the world.

That was a really interesting feeling for me. Normally, I pride myself on producing stuff that will help the most people. If we have to mangle the software a bit in the name of marketability or so we can get it into the Giant Megacorp Marketing Engine, so that 10x more people will hopefully get to use it, isn't that better overall? What's the big deal?

Good question.

The answer comes down to integrity; I shouldn't have to write good software or have lots of people use it. If I do it right, I should write good software and lots of people will use it. Yeah, I know, that's not as easy as it sounds.

But from now on, it's good software⁵ first, people using it second. Sorry, potential customers. If you want crap, there are plenty of megacorps who will sell it to you. You don't need to buy it from me.

The rights to ruin my work are no longer for sale. You can fork and ruin my open source stuff if you want, but my version will still be mine.

Footnotes

¹ Check out the part that says "Version 5.0x and earlier don't have this problem." That's because they switched the DNS server from (the secure and well-written) djbdns to (timelessly crappy) BIND for some sort of legal reasons. Shortly afterwards, say hello to a worldwide DNS poisoning attack. Bad luck? It's not bad luck when you bring it on yourself. djbdns and qmail: still no security holes. In history. Ever. (Correction: djbdns did finally have a security hole in March 2009, although it affected almost nobody.)

² It's by Ayn Rand. So sue me. I'm always confused by people who hate Ayn Rand on principle despite the fact that either a) they didn't read her books at all, or b) they didn't understand them, and thus think she's in favour of extremist crazy libertarian capitalism. The Fountainhead includes a greedy guy who sells his soul for money and ends up committing suicide because it turns out money doesn't buy happiness; meanwhile, the story's hero lives in poverty, never becomes a success, is widely despised by most members of his profession, literally blows up his own construction rather than let it be ruined by its buyers, and ends up comparatively happy (sort of) despite having tanked his career. But yeah, sure, this is the philosophical basis for libertarianism.

³ Of course, in real life, both for software and for buildings, it's not so gratifying. Blowing stuff up doesn't accomplish anything. The character in the book is an architect - a word we stole for software, though for him it was physical buildings - and you can blow up software or buildings all you want. People can just make more copies of it, because they still have the blueprints. Architecture is just knowledge. Short of a Library of Alexandria scale event, knowledge just doesn't get destroyed so easily.

⁴ I don't want to give the impression that everything pre-acquisition was absolutely perfect and rosy. I suck too, but at least it's an honest sort of suck. The changes that have been made to the current system are ones that no sane person could possibly believe are an improvement. It's just the usual megacorp story: bureaucracy, make-work, and lawyers.

⁵ I've been working on a rewrite of GourD as open source. The new version is already as awesome as GourD in most ways, and in some ways even better. I'll write more about this when it's ready. If you really want, you can try a live version of it, albeit sparsely populated so far, in the EQL Data Knowledgebase. The source code for that, plus newer (not yet production) features can be found in my ekb source code on GitHub.