Assorted Writings

2009-03-01 »

Variation on a theme: maxtime

Previously, I explained my simple runlock program. This time, a slightly different example: maxtime.

The purpose of maxtime is to automatically kill a process if it runs for more than a certain number of seconds. This is useful in gitbuilder to handle cases when your build process gets permanently stuck and never exits (such as if you have unit tests that go into an infinite loop or start waiting for user input). Last time, when discussing runlock, I mentioned that it was incredibly evil to simply break a lock because it's "old"; maxtime solves this correctly, by properly killing the program that owns the lock.

Here's the interesting part of maxtime:

    my $pid = fork();
    if ($pid) {
        # parent
        local $SIG{INT} = sub { kill 2, -$pid; };
        local $SIG{TERM} = sub { kill 15, -$pid; };
        local $SIG{ALRM} = sub {
		kill 15, -$pid; sleep 1;
		kill 9, -$pid; exit 98; };
        alarm $maxtime;
        my $newpid = waitpid($pid, 0);
        my $ret = $?;
        if ($newpid != $pid) {
            die("waitpid returned '$newpid', expected '$pid'\n");
        }	
        exit $ret >> 8;
    } else {
        # child
        setpgrp(0,0);
        exec(@ARGV);
    }

To the untrained observer, this looks a lot like the fork/exec chunk of runlock. But it's not. There are some really critical differences here.

The most basic difference is we're now calling alarm() to request an alarm signal after the timeout. When we receive the signal, we make sure the process is good and dead.

Another difference is that we kill "-$pid" in maxtime, versus "$pid" in runlock. That takes advantage of a little-understood Unix feature called process groups. Have you ever noticed how when you run a command in the background, then terminate it, your shell can reliably kill it and all its subprocesses? Like this:

    $ (sleep 100 & sleep 200 & sleep 300) &
    [1] 16130
    
    $ kill %1

In the shell, there's an important difference between killing %1 versus killing 16060. If you kill %1 in the above example, you actually send signal 15 (SIGTERM) to "process -16060", also known as "process group 16060." You'll explicitly terminate all the processes, not just the parent one. Watch:

    $ (sleep 100 & sleep 200 & sleep 300) &
    [1] 16130

    $ ps xjf
     PPID   PID  PGID   SID STAT COMMAND
    16123 16124 16124 31457 S     \_ sh
    16124 16130 16130 31457 S         \_ sh
    16130 16131 16130 31457 S         |   \_ sleep 100
    16130 16132 16130 31457 S         |   \_ sleep 200
    16130 16133 16130 31457 S         |   \_ sleep 300
    16124 16134 16134 31457 R+        \_ ps xjf

    $ kill 16130
    $ ps xjf
     PPID   PID  PGID   SID STAT COMMAND
    16123 16124 16124 31457 S     \_ sh
    16124 16135 16135 31457 R+        \_ ps xjf
        1 16133 16130 31457 S    sleep 300
        1 16132 16130 31457 S    sleep 200
        1 16131 16130 31457 S    sleep 100

    $ kill -15 -16130
    $ ps xjf
     PPID   PID  PGID   SID STAT COMMAND
    16123 16124 16124 31457 S     \_ sh
    16124 16136 16136 31457 R+        \_ ps xjf

Notice that after killing the primary subprocess (known as the "process group leader"), the child process's "sleep" calls are still running happily in the background. That's no good! But you can see that they all have the PGID (process group id) of the primary subprocess, 16130. If we kill the process group using -16130, the entire set of processes will die.

This is what we want with maxtime, of course. If the problem really is some unit tests that are frozen in the background, you can't guarantee that the process group leader will clean them up correctly; after all, he's obviously buggy, and that's why you're killing him.

One last note: how do you create a process group in the first place? Normally, when you fork() a process, the new process is in the same process group as its parent. (It wouldn't be much of a group if every new process got a new one!)

That's the purpose of the setpgrp(0,0) call⁽¹⁾ right before running exec(). This creates a new process group for the current process, and gives it the same number as the process itself. (That way, the parent will know what the pgid is.) Your shell does this automatically whenever you run a command interactively.

Okay, so what's that SID field, then?

The SID is the "session" ID, and it's related to your terminal. Session IDs are even more complicated, because of the restrictive rules on when you are/are not allowed to create a new session, who controls the session, and so on. Basically, the session is what's responsible for such handy features as this:

    $ cat &
    [1] 23298

    $ true
    [1]+  Stopped(SIGTTIN)        cat

Only the current foreground process in a particular session is allowed to receive input from your terminal, or things would get pretty confusing. Also, when you press CTRL-C, it sends the SIGINT signal only to the current foreground process. To read more about sessions, try "man 2 setsid".

Footnote

⁽¹⁾ Note that setpgrp(0,0) in perl is equivalent to setpgrp(void) in C, which is the same as setpgid(0,0) in C. I'm not sure why perl chose to name their system calls like that. Thanks to Aaron Davies for pointing out that my article mixed the two meanings.

March 3, 2009 14:18

2009-03-03 »

Avery @ DemoCampGuelph 8 tomorrow

In case you're in the neighbourhood, I'm currently signed up to present EQL=Data at DemoCampGuelph8 in Guelph, Ontario, tomorrow afternoon.

Because of the magical organization structures that create the VariousCamps, just the fact that I'm signed up and got accepted doesn't necessarily that I'll end up presenting. But we can hope.

See you there!

March 3, 2009 22:56

2009-03-05 »

Public service announcement: everyday vs. every day

everyday: An adjective that means something is normal or typical. (Merriam-Webster) For example, "everyday low prices," or "everyday average guy."

every day: A set of days when something might occur, that is, all of them, generally used as an adverb. For example, "We have low prices every day," or "Every day, I'm an average guy."

Hint for using adverbs: you need a verb before you can add to it.

Unfortunately, my dictionary doesn't have "every day" in it because it's just a normal combination of two words. But if you look up "everyday," you will find that things like "low prices everyday" or "people do this everyday" don't work.

See also my hybrid personality/vocabulary quiz.

March 3, 2009 17:02

2009-03-07 »

On brilliant solutions to boring problems

"Linus has a brilliant technical mind, but unlike most brilliant people, he does not have tunnel vision that steers him towards problems that are already framed as "interesting" problems in an existing theoretical framework."

--Someone on news.ycombinator.com

March 7, 2009 04:10

2009-03-10 »

Git's DAG is just a bunch of branches

Eric Sink, a generally smart guy whose postings I've been reading for some time, has recently been working on a series of articles about Git.⁽¹⁾

Now, I think Eric is awesome. He writes generally good stuff, and he runs a bootstrapped software company, and people seem to like him, and he's a respectable guy. I hope to be as smart and successful as Eric some day.

I also respect that a lot of rather crazy people have noticed that he's being kind of hard on Git in his article series, and have been flaming the crap out of him.⁽²⁾

But.

But the last couple of articles have gotten a little... erroneous. First of all, let's acknowledge some things he says that are totally true: most users of version control do like the linear model just fine. The linear model does solve lots of problems. Git's user interface is incredibly obnoxious and confusing.⁽³⁾ Being able to rewrite revision history and make partial checkins is amazingly dangerous, and fun, and startlingly easy to horrendously misuse. And if you work for a Big Faceless Software Corporation, may God help you if your co-workers get their hands on such tools.

If you really feel you have to flame Eric, please don't flame him for saying any of those things. Stop for a minute, think about it, and realize he's actually right.

It's just that he's only telling half the story.

Anyway, I might not have bothered about this at all, except for his recent comments about the DAG. He claims that allowing developers to attach new commits anywhere they want in the tree is this terrible, confusing thing that nobody will understand and which can release untold pain and suffering upon the masses.

...but Git's DAG (directed acyclic graph) is just a bunch of branches. I admit it, I've never used SourceGear Vault, but... it has branches, right? Aren't branches just arbitrary points in a tree structure where development went off in two directions? Kind of... a fork? Where your "straight line" development, you know... branched?

The magical thing about Git's DAG is not that it branches and has lots of different leaf nodes. If your version control system does anything right, it can do that. Geez, RCS and CVS can do that just fine. (Ironically, Subversion can't, but I'll claim that it's the crazy one. Its every-branch-is-a-subdirectory model is certainly extremely rare.)

No, what makes the DAG in Git special is the way it joins the branches up again. The only difference between a tree structure and a more general DAG is that DAGs can join back up again. That's all. Before DVCSes came along, you could branch, but merging was a synthetic operation. All you could do was make more and more branches, and then, if you really wanted, find a way to copy the changes you made on one branch into another branch, which we called merging. After that, you could try to ignore the old branch, or even "delete" it (for varying definitions of "delete" depending on your VCS). But once you branched, that branch was its own completely independent set of history... forever.

The only thing special about Git's DAG is that you can join branches back together again. The merging operation is the same, Git just stores the fact that it happened. Merges in Git reduce confusion, because once you've merged, you'll never accidentally re-merge the same changes, and you never have to remember by hand which changes are which.

In short, I don't know anyone who has been confused by branching and merging in Git, at least no more than they've been confused by branching and merging in any other VCS. Confused by Git's user interface, sure, absolutely, every single person, every single time, until (if that person is very lucky) it finally clicks.

But the DAG concept, mind blowingly confusing? No. It's not even new enough to be confusing.

Footnote

⁽¹⁾ How much do you want to bet that the eventual conclusion to his series will be the announcement that the new version of his product now supports distributed version control, but is more user friendly than Git? Hey, go for it.

⁽²⁾ I've been thinking of writing an article that I'd call, "The Free Software Community is a Bunch of Weenies," in which I point out that back in the good old days, the only way free software weenies would flame you en masse is if you said something nice about Microsoft. Now you can get flamed for just about anything. Like when ext2resize ate all my data, for example, and it was obviously my fault. It appears that most Linux users now use Linux just so they can flame people, which makes me sad.

⁽³⁾ When I said "Git is the next Unix," I meant it in the bad ways as well as the good. Unix is 1) crazy, 2) not mainstream, 3) appealing to geeks, and 4) apparently unkillable. I expect Git will fill the same role in version control systems. Those who do not understand Git are doomed to reinvent it... poorly.

March 10, 2009 04:14

2009-03-16 »

Twitter and me

I have been using twitter for a couple of months now, and I've come to some simple conclusions.

There are, as far as I can see, only two good reasons to use twitter: to "monitor the zeitgeist in real time," as you might say, or to see who's talking about you.

Those two systems are completely different.

In the first case, you run one of the many twitter clients, subscribe to a lot of people's feeds, and just let the random 140-character-or-less messages flash on your screen, embedding themselves into your subconscious. I think this could be very valuable for, say, people interested in marketing to some combination of geeks and teenage girls, which seem to be twitter's primary audiences. I'm not (entirely) kidding; if you're into market research, I think this twitter stuff could actually be useful to you.

The name "twitter" comes from this idea. The original concept of "twitter" comes from radio. If you mistune your radio completely, you get noise; if you mistune it by just a little, you get something as unintelligible as noise, but which actually contains a lot of signal. You just can't make out what the signal is... but if you listen just a little bit closer, then maybe...

I tried that method for about three days. Then I turned it off; it's not for me. The way I see it, background twitter uses up about one slot in your brain. Rumour has it that the average human mind can hold around 7 things in it at once; some people, maybe only 5 or 6, and if you're super-smart, maybe 8 or 9, but 7 is about normal. So if you're normally a 7, twitter makes you about a 6. It effectively lowers your ability to think; in exchange, you get more raw information, and maybe you don't need to think as much.

And like I said, that works fine for some people. But if you're a programmer, you need to get into The Zone. The Zone is that state where all the slots in your mind are fully engaged in solving one specific problem. If you waste one of those slots, you're not just 6/7ths as effective; you could be 1/10 or 1/30 as effective. Effective zoning is where the rumoured 10x difference between "great" and "good" programmers comes from.

So anyway, that's not for me.

But I still use twitter for reason #2: tracking keywords. This is awesome, and takes very little work, and it doesn't have to interrupt you. Just go to search.twitter.com and sign up for some rss feeds that track whatever words you think are interesting, like your name, or the name of your company, or some products you made / care about / etc.

Like magic, you'll suddenly be tied into a real-time feed from perfect strangers telling you how people feel about those topics at that exact moment. If you chose your topics wisely, this can be extremely useful.

For example, I randomly mentioned on twitter that I was frustrated with signing up for Microsoft Bizspark, because you had to go through a value-added reseller, and they all kept wanting to add value ("come meet with us!") and I didn't really need any added value, I needed free freakin' MSDN. Well, it turns out someone (actually a few people) at Microsoft had a twitter search for anyone using the word "Bizspark." They had me signed up for Bizspark within five minutes. I don't know if twitter is gimmicky, or a big fad, or the next big thing, or what, but I love those guys now. It seems like an awfully cheap and easy way to do the same for your own customers too.

Random suggestion to Google

Wouldn't it be awesome if there was a "Google Alerts" for the whole web, not just news? I'm almost never in the news, and the companies I work with are often too small or early stage to be in the news. But people still mention them. I don't really need twitter at all; I just want to know when people are talking about certain things.

March 17, 2009 22:12

2009-03-18 »

Note: Startups change frequently

...you don't get a memo that tells you that things have changed. If you did, it would read something like this: "Dear Eric, thank you for your service to this company. Unfortunately, the job you have been doing is no longer available, and the company you used to work for no longer exists. However, we are pleased to offer you a new job at an entirely new company, that happens to contain all the same people as before. This new job began months ago, and you are already failing at it. Luckily, all the strategies you've developed that made you successful at the old company are entirely obsolete. Best of luck!"

-- Eric Ries

There's lots of other great stuff on Eric's site.

Thanks to pphaneuf for the original link.

March 17, 2009 23:05

2009-03-20 »

The ternary operator in python

You might have heard of the so-called "ternary operator" in C. Or maybe you haven't; someone asked me about it as one of those useless skill-testing questions (which actually just checks trivia knowledge, not skill) at my first ever job interview, and I hadn't heard that term before.

Before that fateful interview,⁽¹⁾ I used to just call it the "question-mark colon" operator. It is, as far as I know, the only operator in C that really does take three parameters. Or you could argue that it's really two binary operators stuck together. You know what? Nobody really cares. I'm just trying to talk about the questionmark colon in as many different ways as I can so that Google can find this entry in case someone needs it later.

Phew. So as I was saying, the question colon operator looks something like this:

    int x = really ? 10000 : -1;

Or, if you're dramatic and participating in an obfuscated code contest, you could write this:

    int x = really?! no: !yes; maybe. ok();

Okay, whatever, I have a cold and I'm delerious. I'm pretty sure you can make that compile. But for the love of God, don't.

The point of the ternary operator is it checks the truthiness of the first parameter, and if it's true, returns the second parameter. Otherwise, it returns the third parameter. It's a short way of saying "if really then x = 10000 else x = -1", and it's useful in a number of space-saving contexts.

This is all fine. Lots of languages, in the name of being "friendlier," omitted the concept of a ternary operator, because it's scary the first time you see one, and people need to be coddled and protected lest they actually learn something. Perl is not one of those languages, for example.

But python is. They have resisted including a ternary operator. Looking for evidence of one online will result in people angrily telling you the ternary operator is scary and confusing and should be avoided at all costs. Geez, write nice-looking code already!

Well, I'm sorry, but combined with the way python forces me to put even trivial if statements on more than one line, this is a pretty insulting thing to leave out.

Now, if you've been following the not-so-recent python news like I have been not-so-recently, you might know that python 2.5 added a handy feature that sort of works like the ternary operator:

    x = 10000 if really else -1

To that I say... WHAT THE HECK IS WRONG WITH YOU PEOPLE? Did you really just put the condition in the middle of a sentence? Did you really just define "if" as having a higher precedence than "="?

I'm sure the python people thought they were very clever when they came up with this; they were, I assume, copying the feature in perl that lets you say things like "$x = 5 unless $really" and so on. But they've missed the point; perl can have a feature like that because sometimes it makes sense. Nobody tries to claim that this is the way to do an inline conditional in perl; they have the real ?: operator and this one, and several other weird things. And you can always use "if" instead of "unless."

Phew.

Like I said, I'm delerious. Please forgive my rambling.

All this was just to say that:

There was already a perfectly good ternary operator in python

Oh.

Somehow, in all my searching, I didn't find anybody who told me this; in fact, I stole it from perl too.

    x = really and 10000 or -1

Why does that work?

Well, in fact, it's kind of neat. In C, it wouldn't work, because while C's type coercion rules are almost the greatest type coercion rules ever invented by programmers, they missed one trick. In C, the result of a boolean operator (like || or &&) is always a boolean, either true or false. So if you say "0 || 5", you get 1, which is the canonical value for truth.⁽²⁾

In perl, they realized that wasn't quite optimal; any non-zero value is true, after all, so there's no reason to give one such special status. So in perl, they decided that "0 || 5" would be 5. If you check to see if 5 is true, it is, so the net result is the same. The same, that is, until you write something like this:

    $x = $really && 10000 || -1;

If $really is true (which means nonzero), then you need to evaluate the other side of the && operation to make sure that the whole thing is true. Since the other side is 10000, which is true, then the whole thing is true, and perl just leaves behind the most recent number it used, which was 10000. On the other hand, if $really is false, then the overall statement can only be true if the || clause is true. In fact, we don't even care if the || clause is true, but it'll be the last thing perl evaluates, so whether it's true or not, that value is returned in the false case. So this is pretty much the same as writing "$x = $really ? 10000 : -1".

BEWARE. Do not write "$x = $really and 10000 or -1;" in perl. It doesn't do what you think it does, because the "and" operator has lower precedence than the "=" operator, while the "&&" operator has higher precedence. perl kind of missed the sanity boat on that one.

Anyway, the python designers, liking minimal syntax and maximal type dynamicness, stole this feature, so you can write:

    x = really and 10000 or -1

And it'll work. Note that in python, the "and" operator has higher precedence than "=", as it should, unlike in perl.

There's only one little catch: the "true" side has to evaluate to true. This won't work:⁽³⁾

    x = really and 0 or 1

Because even if "really" is true, you'll get 1 instead of 0. Oops. Luckily this comes up virtually not at all in real life, since if you wanted to write the above, you would have just written "x = not really". Which is much funnier.

Side note

Visual Basic doesn't have expression short circuiting, which is only one of many reasons why this trick won't work there. (VB.Net does, but you have to use new words, not "or" or "and.")

Footnote

⁽¹⁾ I got the job anyway, because I redeemed myself by knowing that "volatile" can apply either to the pointer or what it points at. Try not to think about that too much. It hurts.

⁽²⁾ Except in the Unix shell, where 0 is the canonical value for truth. That actually makes a lot of sense, since there's generally only one way to be right, but lots of ways to be wrong. But 0 just doesn't feel true enough.

⁽³⁾ I'm still waiting for perl6, so I can write "$x = $really and (0 is true) or (1 is false);" Of course, I won't need to, because they have a ternary operator.

March 20, 2009 17:59

2009-03-23 »

The Facebook Redesign

I keep reading people complaining about the Big Facebook Redesign that everyone apparently hates. It's funny that I know this, because I don't even use Facebook.

But I do know one thing: what they did was kind of dumb. Not the specific redesign, but the way they delivered it.

I know this because I've seen it done better. When gmail changed their look and moved things around a year or so ago, they left a link at the top of the screen that says "Old Version." If you hate the new look, you can just go back to the old version as long as you like. And then Google can track exactly how many people actually prefer the old version, without resorting to highly biased polls or open letters or email counters or guessing.

And you know what? For months, I did use the Old Version mode in gmail. The new one, possibly combined with Firefox 3 Beta bugs at the time, had screwed up the keyboard focus for me, and keyboard mode is critical to my happiness. (You do know about gmail's keyboard mode, right? It's so great!)

Still, I knew the new version had some new features I'd like, so I filed a bug report about the keyboard stuff (which presumably disappeared into /dev/null), mostly used Old Version mode, and switched back and forth occasionally to see if my bug was fixed. Eventually, the bugs got fixed, and I didn't need Old Version mode anymore.

As far as user experience goes - other than the /dev/null bug tracking system - this was top notch. They offer me something new and different. I don't like it as much as the old one, but that's okay, because I can just use the old one and tell them what I don't like. They monitor (presumably) what people like and don't like, and track how many people are using the old version - hopefully a declining number over time.

Nobody has to get angry or emotional. The complainers never get critical mass, because whenever they decide to write an open letter or stage a protest, someone inevitably (and rightly!) says, "Just click Old Version and quiet down, already." People who have honest problems don't suffer, Google gets unbiased, accurate statistics about who likes the old vs. new version, and people who don't like the new colours (or whatever) have as much time as they want to get acclimatized to it, without getting defensive, until they forget why they hated the new colours so much in the first place.

The new Google.com favicon is total crap, though. What were they thinking? I wish I could Old Version that.

March 20, 2009 17:59

2009-03-26 »

Larry Smith has a blog

Canadian economics superhero Larry Smith now has a blog.

It says on it, "This site is intended for the exclusive use of Larry Smith's current and former students. Its use by other persons is expressly prohibited." Now, I don't know much about the legal enforceability of click-thru licenses, but I'll trust you'll comply with his wishes.

I think the few articles he's already posted are great, although you might have a hard time understanding (or believing) them if you haven't taken his economics classes. Or if you aren't Canadian.

While in university, I took Econ 101 (microeconomics) twice: once with a "normal" professor, and once with Larry. The two classes were almost entirely unlike each other, in very interesting ways.

Previously.

March 25, 2009 21:18