bup 0.01: It backs things up

I just spent a few days of my Christmas vacation writing a new program, bup.

bup is a program that backs things up. It's short for "backup." Can you believe that nobody else has named an open source program "bup" after all this time? Me neither. It also has almost no other meanings.

Despite its unassuming name, bup is pretty cool. To give you an idea of just how cool it is, I wrote you this poem:

Bup is teh awesome
What rhymes with awesome?
I guess maybe possum
But that's irrelevant.

Hmm. Did that help? Maybe prose is more useful after all.

Reasons bup is awesome

bup has a few advantages over other backup software:

• It uses a rolling checksum algorithm (similar to rsync) to split large files into chunks. The most useful result of this is you can backup huge virtual machine (VM) disk images, databases, and XML files incrementally, even though they're typically all in one huge file, and not use tons of disk space for multiple versions.
• It uses the packfile format from git, so you can access the stored data even if you don't like bup's user interface.
• Unlike git, it writes packfiles directly (instead of having a separate garbage collection / repacking stage) so it's fast even with gratuitously huge amounts of data.
• Data is "automagically" shared between incremental backups without having to know which backup is based on which other one - even if the backups are made from two different computers that don't even know about each other. You just tell bup to back stuff up, and it saves only the minimum amount of data needed.
• Even when a backup is incremental, you don't have to worry about restoring the full backup, then each of the incrementals in turn; an incremental backup acts as if it's a full backup, it just takes less disk space.
• It's written in python (with some C parts to make it faster) so it's easy to extend and maintain.

Super quick example

(The README actually has a more detailed example.)

Try making a remote backup:

    tar -cvf - /etc | bup split -r myserver: -n my-etc -vv

Try restoring your backup:

    bup join -r myserver: my-etc | tar -tf -

(On myserver) look at how much disk space your backup took:

    du -s ~/.bup

Make another backup (yes, that's exactly the same command):

    tar -cvf - /etc | bup split -r myserver: -n my-etc -vv

Look how little extra space your second backup used on top of the first:

    du -s ~/.bup

Restore your first backup over again (the ~1 is git notation for "one older than the most recent"):

    bup join -r myserver: local-etc~1 | tar -tf -

What's next?

I have lots of plans for this lovely program, in the event that I actually get time to implement them. But if you think it's cool, please feel free to git clone it, hack away, and send some patches! Read the README for a list of some deficiencies in the current release.

I'm sure there are also more deficiencies that I don't know about, of course.

(Previous poetry-related adventures.)

Update (2010/01/05): Commentary at ycombinator news and reddit programming. To answer the most common question: it's different from most of those other apps you mention because: a) bup backs up really huge files rather than silently ignoring them or running out of memory; b) bup is a backend, not a GUI, while most of those apps are GUIs (which could use bup as a backend if they wanted); c) bup stores its backups in big packfiles, rather than a one-file-per-file model, and thus can be much faster (but 0.01 isn't optimized yet).

Update 2 (2010/01/05): By popular demand (well, nonzero demand from the populace, anyway), I've created a mailing list. You can subscribe by sending an email to bup-list+subscribe@googlegroups.com (note the weird + character in the email address).

Dealing with "tumultuous" economic times

Canadian Economics Superhero Larry Smith has a new article out, addressing burning economic question questions such as:

• Why is Silicon Valley so shortsighted?
• Why do so many people write iPhone apps and so few write medical software?
• How to deal with a shortage of engineers.
• Volume of information vs. value of information.

All great stuff as always. You should read it.

The Google Phone

...even Microsoft never was brazen enough to pull something like this. Even Microsoft had some tiny bit of shame. Google is a different beast altogether. They're like nothing anyone has ever seen in our business. Not only are they not ashamed - they think they're the good guys!

-- Fake Steve Jobs

Dot dot dot.

Proroguing parliament

"Prorogation" is the term we use to describe the end of any session of parliament in Canada.

Or as the Canadian Parliament web site says, "Each session of a Parliament ends with the prorogation of Parliament by the Governor General, on the advice of the Prime Minister."

Since Canada's confederation in 1867 (143 years ago), parliament has prorogued 105 times. That's an average of about 1.4 years per session (some are longer, some are shorter).

See the Government of Canada's complete list of prorogations.

Use of the term "despotic" in this context makes you automatically a lunatic.

That is all.

More on prorogation

In response to my previous post about prorogation, someone emailed me this comment:

My participation in our democracy is limited to voting when there's an election and mostly ignoring everything else. [...] Despite that, I can tell from the rumbling that there's something unusual about this particular prorogation. If it was just normal boring governing, nobody would be talking about it.

I actually think this comment is very insightful, because it gets right to the heart of the issue I was trying to address. Other people sent responses that were more like, "But what gives them the right to manipulate it in such an evil way?" It's almost the same question, but those comments were not insightful. They missed the fact that there may not be any manipulation at all, and therefore turned it from a question of fact into a question of opinion.

I'm not a professional journalist. Yesterday's attempt at factual reporting took all the restraint I had. So this post will surely not meet my own high standards for journalistic integrity. I'm just some guy on the Internet. You've been warned.

How I came to know what "prorogation" means

Full disclosure: I personally don't like the Conservatives. I think if they had a majority government, Canada would be worse off. I think the fact that Harper does most of his public relations through a "spokesperson" is a total embarrassment. I also think our current batch of other political leaders, with the possible exception of Gilles Duceppe, are even worse, and I sorely miss the days when Jean Chretien used to beat people up with statues. But I wouldn't vote for Ignatieff, precisely because he pulls the current kind of crap. I wish I still lived in Quebec so I'd at least have a party worth voting for.

That's the perspective with which, not knowing anything about the current prorogation debate (or even what "prorogation" means), I returned to Canada from my vacation in Mexico and was met by an angry Internet mob complaining about our upcoming dictatorship.

See above: "My participation in our democracy is limited to voting when there's an election and mostly ignoring everything else." Me too. You know why? Because I think the system works. That's the beauty of representative democracy. But I figured, okay guys, I'm pretty smart, I can figure this stuff out. If there's really a dictatorship coming, I want to be on the winning team. So I thought I'd better look into it.

The actual mob of complainers were no help. They all figured that someone else knew what was bad about what was going on, or else they figured that just suspending parliament at all made the government a dictatorship.

This left me to do my own "research" (a word that is in quotes because I did all this "research" in bed using my laptop).

First stop: The National Post (via Google), in an article titled, "Thousands turn out at rallies to protest proroguing of Parliament."

Intermission: A note on how to read political news
All news sources are biased. The first thing you have to do is identify the bias. Both the National Post and the Globe and Mail are Liberal-friendly and anti-Conservative. How can you tell? Just read any headline about politics and watch the trend. Other tips: 1) all quotes from politicians are weasel words; don't trust them. 2) using quotations from any individual allow the newspaper to avoid fact checking; it is always a fact that "person X said Y," no matter how false Y may be. 3) in constructions like "estimates pegged the turnout at more than 3000 people" note that nobody in particular is being quoted; they are reporting that some random person estimated more than 3000 people. Do not fool yourself into thinking that they don't pull tricks like this. They have to research, write, edit, and publish a fat sheaf of paper like the National Post every single day. Corners will be cut.

...shockingly, the National Post headline is phrased in an anti-government way (since the government is Conservative). Reading on, we see lots of quotes from political leaders (ignore them; rule 1). The whole article is also really a big quote from an angry mob of protesters (no fact checking was done; rule 2), since it merely reports that there was an angry mob, not that what the mob was angry about even exists. And we don't know if the mob was really 10 people or a million people (rule 3).

So that article, although it used many words, was in fact 100% pure unenlightening.

Nevertheless, I could feel myself thinking anti-Conservative thoughts despite the total lack of facts. I thought I'd better go find something biased in the opposite direction so that I could balance things out a little. I realized that I couldn't think of any actual Canadian mainstream newspapers that are Conservative-friendly; I'm probably forgetting something obvious.

So I resorted to blogs, which made things easier. Next stop: Alberta Ardvark (via Google). I have to admit that I assumed they were Conservative-friendly just because they're in Alberta, which means I'm a racist. However, I was not disappointed.

That article was the usual politico-blogger nonsense, giving handy advice on how to win a political argument not by arguing about the issue, but instead by turning the conversation around to character assassination whenever possible. (In this case, it's about the fact that Ignatieff supported prorogation last year, so he's not allowed to be against this particular prorogation this year; that makes him inconsistent, therefore a liar, so it doesn't matter what he says, etc.) Nicely done as always, Internet. But there was one intriguing quote: "...the ones who have been convinced by the media that this prorogation is not a routine event..."

Wait... routine event? Surely this is Conservative propaganda. I was intrigued, so I followed the link.

It points out that Chretien prorogued parliament 4 times. And Pierre Elliot Trudeau, supposed Canadian hero... 11 times?! Holy crap! Who's the evil one around here, again?

My brain's magical pattern-detectors then kicked in and I thought: hey, wait a minute. Those prorogation counts seem to be roughly proportional to the amount of time a particular Prime Minister was in office. Perhaps there's a pattern here.

And then, at the bottom of the article: "In our 143 years of existence as Canada, Parliament has been prorogued 105 times."

Oh dear.

Maybe I'd better go learn what prorogation is. The answer is: it's an all-too-fancy word for the end of every parliamentary session.

And it's a word I didn't know the meaning of until now. A word that every news article and blog entry I've read so far has not bothered to define. A word that, in some tenses, has the word "rogue" in it.

Okay, this sucks. Reading biased articles in a search for truth is getting me nowhere. Is there not any news source that will just give me the facts and not try to spin it the way they want? Well, no, I guess there isn't. But there's something close: the CBC.

Because it would be weird if they didn't, the above-linked CBC article has quotes from politicians; try to ignore them (rule 1). If you fail to do so, you will discover that the first apparent use of the word "despotic" in this context was by Ralph Goodale, Liberal House Leader, whom you should not vote for because he is thus automatically a lunatic.

However, the non-politician-quote parts of the article seem to be well balanced and, notably, identify several reasons why Harper might have wanted to prorogue government right now as opposed to some other time.

So what have we learned? First, that prorogation of parliament is totally normal, and that the length of the just-ended session isn't even unusually short; and second, that Harper might very well have chosen this particular date in order to benefit himself or his party. Gasp! Let us look at these possible reasons in more detail.

The CBC's suggested reasons for the current prorogation

"Muzzle parliamentarians amid controversy over the Afghan detainees affair." Don't know about you, but parliament has been shut down for a whole month already and I haven't noticed any of those parliamentarians not talking. I wish. But maybe you have a point; when parliament next starts up, I bet the opposition parties will have completely forgotten about the whole thing, despite the obvious political leverage they could gain from bringing it up. Harper has totally outfoxed them on this one.

"To consult with Canadians, stakeholders and businesses as it moves into the 'next phase' of its economic action plan amid signs of economic recovery." Well, I guess theoretically, if you're in parliament you don't have as much time for consulting with Canadians. But don't we have Royal Commissions for that or something? Maybe they just wanted a longer Christmas holiday. (Aside: the reason the prorogation "doesn't start until January 25th" is that they've all been on holidays since sometime in December. Seriously.)

"Strategically, prorogation also prevents question period criticisms from the opposition parties during the Olympics." Hey, not bad. Avoid the bad PR for Canada from discussing our idiotic foreign affairs policies at the same time as we're in the global spotlight. Critically, this allows Mr. Harper, who (let's be honest) doesn't look all that lovable, to hide in the cellar for the whole time the Olympics are on, letting someone cuter represent us to the world. This seems to be a wise strategy no matter which side of the fence you're on.

"By proroguing Parliament, he is unilaterally making a decision to stop any kind of disclosure from happening." As if information can't be disclosed just because nobody's making any laws right now. Remember: parliament is the legislative branch of the government. It's for making new laws. No other part of the government is suspended just because parliament is. (Note: see update below.)

"Gilles Duceppe wrote that prorogation has become 'a tradition for Harper.'" Duceppe has an awesome sense of humour. I had to read this one a few times before I realized that he managed to give them a sound bite while simultaneously making fun of the fact that prorogation is totally normal, ie. a "tradition."

"By the time Parliament resumes, Harper would have had time to ask Jean to name five new senators, which would give the Conservatives a majority on the newly formed Senate committees and greater control for passing their own legislation." (Notably: nobody was quoted saying this. CBC had to look it up on their own.) "Soudas confirmed the prime minister will seek to fill the Senate vacancies between now and March." This one is actually a great example of a real political reason to prorogue parliament; to get more control of the senate in time for the next session. But the Canadian senate system is designed (on purpose) to work like that. That's why the current senate is mostly Liberal even though our elected representatives are mostly non-Liberal. Senators are appointed for life, at which time the Prime Minister selects new ones. No surprises here.

(In case you don't like that system: the only party in favour of senate reform in Canada is the Conservatives. They'd rather you could elect your senators. How "anti-democratic" of them. I actually think such reform would be a change for the worse, but that's just my opinion.)

"Shortly after Soudas' announcement, the government sent out an email saying it would reintroduce, in original form, the consumer safety bill and the anti-drug-crime law that the Tories claimed the Liberals 'gutted' in the Senate." This shows significant political maneuvering. However, bills take multiple rounds through both houses before they (might) get passed anyway, so this isn't as bad as it sounds. The "gutted" version might never have been passed anyway. Also, interestingly, this was pointed out in an email from a Conservative MP. Apparently they don't think it's evil. At least not evil enough to cover up.

Conclusions

Guys, I did my homework. But I'm just not seeing it. The actual facts are:

1) Prorogation is perfectly normal and the recent parliamentary session wasn't abnormally short.

2) We won't have any more new laws getting made for a month or so longer than usual. (Remember: they were on vacation until January 25th anyway.) But being unable to do stuff doesn't make you a despot, it makes you a eunuch.

3) If Harper is really evil, the first thing to happen in the new parliament in March is that there will be a vote of non-confidence followed by an election. If this doesn't happen, it's because the angry non-Conservative parties didn't actually believe he was evil either.

4) There are some valid political reasons why it's better for the Conservatives if they prorogue parliament right now instead of later. However, they aren't very exciting reasons.

5) There is at least one actual reason (Harper is scary-looking and the Olympics are coming) that it's better for Canada if they prorogue parliament right now.

6) All mainstream media that I read - which was quite a bit - failed to properly define the term "prorogation" or to mention that it's perfectly normal. This seems a rather critical thing to know. Its omission suggests to me that they're trying to make news out of non-news.

Epilogue

"If it was just normal boring governing, nobody would be talking about it."

Unfortunately not true.

That's textbook mob mentality: he must be guilty, because otherwise my friends wouldn't be burning down his house.

The only cure for mob mentality is thinking for yourself.

Updates

Some helpful people have emailed me to clarify or correct or question various parts of the above.

Information release on the Afghan torture investigation: make no mistake, this investigation, and the demand for release of information related to it, is very important. It will also be delayed (for about one month) because of prorogation. Because of "parliamentary immunity," the interesting testimony won't be released during the delay. However, you need to think about two key points: first, will the end result of the investigation be any different because of a one-month delay? And second: will the Conservatives benefit because of the delay? Keep in mind that if the results came out now, they would be largely overshadowed by news about the Olympics. If they come out later, the Olympics will be over, the opposition parties could force an election, and the results would be headlining right as we're thinking about who to vote for. And yet the Conservatives have chosen the latter, not the former.

Precise timing of prorogation: Several people responded by claiming that it's not prorogation that's the problem, it's the particular timing of Harper's use of prorogation. This is an insidious line of argument because it's impossible to disprove; if Harper had prorogued parliament back on the 9th of September (9/9/9 is the British equivalent of 9/1/1), you could have accused him of using numerology to choose prorogation dates, and it would be impossible to refute that claim, even if it had been a perfectly sensible date to end the parliamentary session. Thus, to demonstrate any wrongdoing, you really have to be more specific about why the current timing is so evil. I have discussed several possible reasons above. Please feel free to suggest more. But "the timing is evil!" is not specific enough.

With that in mind, however, a lot of bad statistics are being spread with regard to the lengths of various parliamentary sessions. Here are all of the sitting days per session since 1980:

Turner and Trudeau (Liberal): 591, 116
Mulroney (Conservative): 308, 389, 11, 308, 271
Chretien (Liberal): 283, 164, 243, 133, 211, 143 (avg: 196)
Martin (Liberal): 55, 159
Harper (Conservative): 175, 117, 13, 128 (avg. excluding outlier: 140)

Eyeballing it, Harper's numbers are very slightly lower than typical (except for last year's prorogation, which was indeed an interesting event). Ignoring the outlier (13), Harper's average is just 29% lower than Chretien's. However, Harper has also managed to hold together the longest-running (by a large margin) minority government in Canadian history. I find it unsurprising that the effort required to do so would result in somewhat shorter sessions.