Building your own online transit planner with a bike and a GPS

Fellow Navarra member William Lachance has been featured in an awesome article in The Coast about hbus.ca, his new online public transit planner for the city of Halifax.

The catch? The Halifax public transit commission doesn't want to have anything to do with it. They've spent over a year and untold amounts of money trying to do it themselves. And because they're not helping, Will had to travel around Halifax by himself, marking down stop numbers, times, and GPS locations, and feeding them into his database. The resulting database is incomplete, but more complete than you might expect, given that he estimates the data collection was only about 30 hours of work. (Halifax has a population of only about 360,000 people.)

Now, in my experience, once you've made someone look stupid, as Will has unfortunately done with the transit people in Halifax, they'll get even more defensive. That means they'll do everything possible to avoid ever admitting that they've wasted their money and produced results inferior to results you would have given them for free, even if it's clear to everyone that this is the case. In sales, a defensive customer is a dead end, and although I wish Will the best of luck, I expect not much progress to happen.

...in Halifax, that is. On the other hand, there are a tonne of municipalities with exactly the same problem. Sure, Google Transit software exists and does trip planning - if you send them your route information in the right format. But most cities and towns don't have their route information in the right format, and getting it right is seen as prohibitively expensive or risky. Maybe Will has stumbled on an innovative way to solve that problem: get students, volunteers, community members, or any random end users to just bike (or bus!) around collecting the information from its existing sources. If you can make it easy enough for lots of people to collect and enter the data into your system, and then find ways to cross-check the inputs for errors, this is something that could benefit people in a lot more places than just Halifax.

iPhone app, anyone?

::nv,li

I tried Joel's salary scale method, and the results were...

...mixed. ;)

Joel Spolsky recently published an article about the public salary scale system at Fog Creek Software.

At it happens, I've been following Joel since he first wrote about that system back in 2002 or so, and my software company at the time used a salary system modeled after his. Here's my story.

The NITI Employee Level System

Joel has plenty of rationale for his system. You might agree with him or not, but we only needed one reason to adopt it. Programmers love algorithms but are terrible negotiators, so turning your salary calculation into an algorithm is a receipe for instantly higher employee satisfaction and less Stupid Management Crap. Right? As I was in the Stupid Management Crapper at the time, this seemed like the perfect plan for me.

And you know what? It worked.

"Worked," in this case, meant that people generally didn't complain about their salaries, and especially about their salaries relative to other people. Whenever people hear about Joel's scheme, they always bring up an odd concern: that this only changes the basis of salary negotiation, so instead of "negotiating your salary," you'll be "negotiating your employee level." Well... yes! That's exactly what you want! Money is stupid and meaningless. Your employee level is one step closer to the quality of your output, and as a Manager who is trying to not be Crappy, I'll be happy to discuss your output and how to improve it, and by extension, how to get you to the next level.

Another concern people have is that a public level scale will encourage people to start comparing themselves to others and getting jealous. Well... yes! That's exactly what they did. Or rather, they started comparing themselves to others, with an emotional response that had to be carefully managed.

When you start talking about the relative value of one person versus another, it has serious emotional and social ramifications. You have to manage it carefully so it doesn't go wildly off course.

That's true even if you don't have a public scale. If you want to learn about jealousy, try paying different salaries to different people at the same qualification level, just because one guy negotiates better and you think you can get away with it. Trust me, someday someone will find out, and it won't be pretty.

But back to jealousy. The problem of jealousy was foremost in my mind when we brought in the new system. Speaking of which, bringing such a system into an established company with traditional secret salaries is tricky!

It was important to set the tone from the start. The way we did this was to take a survey of everyone: we asked each person to name two people in the company that they thought should be at a higher level than them ("betters"), and one person that should be at the same level as them ("peers"). Note that we didn't ask anyone to name someone who should be at a lower level; that's bad form, plus the math would give us that answer anyway.

We did a quick topological sort (after dealing with a couple of inevitable loops in the graph), and what did we find? The results were, in fact, extremely interesting. There were almost no disagreements about relative level. In fact, as I recall, there were only two people who "mistakenly" chose someone below their level to be in the "betters" pile; those two people were the ones who ended up at the maximum level, so as it turns out, they had no choice. (They each chose each other and one other person. As statistics go, this experiment was pretty convincing.)

Yes, yes, but I can tell you're still waiting for the jealousy.

So all those people who named people as "better" to them. They were jealous, right? I mean, it's sort of like signing your own execution order or voting in favour of a tax hike; you know that you're basically giving away money by playing this game. Someone you selected is going to end up getting paid more than you.

But amazingly, that wasn't a problem. We were a company where people respected each other, and we played the ranking game on the basis of respect. If you respect someone enough to identify them as someone who's better than you, you're not going to be jealous of them. You're going to try to be more like them, so that one day you can be as good as them, and then (incidentally) get paid as much as them.⁽⁰⁾

It worked. Human productivity, to a surprisingly large extent, seems to thrive on low self esteem. Feeling like there's something to live up to makes you work harder.⁽¹⁾

There was a flip side to this, however. We had spent so much time making sure that people at lower levels wouldn't be jealous that we forgot what it would be like for people at the highest levels. In fact, for us, that was the biggest problem of all with the public salary scale: managing the people at the top.

It's kind of like giving out an "employee of the month" award; the results of those awards are demonstrably negative. It might make the person at the top feel important - although usually it makes them feel singled out, vulnerable, and embarrassed - but it devalues the idea of hard work. You're at the top, now what?

This was really hard. Think about career growth. For people at lower levels, we could (and did) encourage everyone to choose a mentor or role model; someone at a higher level who had a skill or skills that they could try to improve on.⁽²⁾ People at the top, however, had no such easy goals. What do you need to do to progress to the next level? I don't know. What does the next level even look like? I don't know. What does the next level get paid? I don't know.

We only had one developer ever resign while working for me. He was at the highest level; he quit because he felt he wasn't being challenged and thus didn't feel as productive as he thought he should be.⁽³⁾ He also thought the level system was stupid; I remember that he was quite insistent that he shouldn't be ranked so highly, and I reminded him that literally everyone else in the development group thought otherwise, including me. And I honestly didn't know what he could do to feel more productive.

So anyway, jealousy. Don't worry about jealousy. As long as you're ranking fairly - and don't worry, with a public scale, people will gladly tell you when you aren't - then jealousy is not your problem. Your problem is finding ways to help your best people stay happy and achieve levels you haven't invented yet. That problem is the one that defeated me, but I'd rather have that one than any other.

And then we stopped

Eventually, we ended up switching away from the public salary scale back to an old-fashioned private one. We did it for lots of reasons, and we learned a lot from that process too.

One reason is that I was working pretty hard to make myself irrelevant; I'm a company starter, not really an empire builder, and it was important to find someone more suitable to manage a large number of people. We found someone like that, but he didn't like the public level scale; he was more of a traditionalist.

Also, the rest of the company (ie. all the non-engineers in our head office) did not have a public salary scale. Joel has it easy; it sounds like almost everyone who works for him is a programmer. But how do you compare someone in sales, or marketing, or tech support, and rank them on the same salary scale as a programmer? Can you even put tech support people on a scale? And salespeople prefer commission, is that a scale or is it different? Anyway, they weren't too happy with our programmer salary scale, largely because they were getting paid less than our programmers. Not that they expected to be paid as much as our programmers, but... what if I'm the only one? What if everyone else in my whole department, all with private salaries, is getting paid way more than me? I mean, everyone in programming, even the junior developers, gets more than me. Am I a chump?

In other words, partially public salary scales are not a good idea.

And combined with all that, finances started getting tight. That meant we couldn't scale up the salary scale effectively - not even to handle the effects of inflation. When you don't have the money to pay someone at the level everyone knows they're worth, what can you do? If it were up to me, I might have thought hard about really serious action, including layoffs. After all, wouldn't it be far better to lose someone from the bottom level than to make someone at the top feel that they were being treated unfairly?

But it wasn't up to me, and it might have been a bad idea anyway. We took the easier route: we took the salaries private again. I was busy in other departments after that, but my understanding is it went exactly as you'd expect. The people who threatened to quit got big raises to try to retain them; the people who didn't threaten to quit didn't see any raises at all.

Eventually half the department resigned, and the other half was forced to leave when we closed our Montreal R&D office. And that was that.

(Lest you think this is an entirely unhappy ending, several of our former developers have gone on to work at Google, and others have founded companies like Navarra, EQL Data, hbus.ca, Carillon Information Security, and more. NITI itself was sold to IBM last year and is now known as Lotus Foundations.)

Epilogue

I'm now working at a totally different software company that's a spinoff of a bank. The bank is expertly managed, and filled with highly competent people across all roles. But I don't know if those people are getting paid fairly, unfairly, or exorbitantly: the salaries are kept private.

And you know what? It works fine.

You can argue about salary scales all day, but the fact is, none of it actually matters. What matters is that people are happy with their job, and they feel like they're being treated fairly. The best way to achieve that is to treat people fairly. If you don't have anything to hide, then you don't have to worry about whether it's hidden or not. Just get your job done.

Footnotes

⁽⁰⁾ Of course, by then, you hope they'll have been promoted. Nobody actually wants to catch up with their heroes. It's depressing.

⁽¹⁾ I personally wasn't assigned a level, unless you counted the fictional "level 13." Let's face it, there was just no way to do that objectively. Even if there were, it would have been entirely counterproductive if I was objectively less productive than someone else (which I quite possibly was), and thus got assigned a lower level. What better way to destroy someone's faith in a company's leadership than to tell them outright that the best people aren't the ones in charge?

⁽²⁾ Our employee self-evaluation was a 2-page form with essay questions. Some people said it took them all weekend to fill it out, because the questions were really hard and they had to think a lot. As much as our evaluation scheme was unpopular with some people, I remain really proud of it. Your skill level, what you want to improve, who you can learn from, and why, are all serious questions that you should take a couple of days to think about every now and then. I filled out the whole form for myself in half an hour, and I'm proud of that too.

⁽³⁾ He was right.

::li

Avery confronts the tough technical terms

People seem to be overthinking things a bit. Allow me.

Web 2.0: a new, friendlier web, where user-friendly grey text is used instead of ugly black text. Also, we use subtle background shading instead of lines to separate sections on a page.

And:

Cloud computing: running programs over the network on computers that belong to someone else.

Hope this helps.

Of Lotus Foundations, Microsoft Server Foundation, and real-life straw men

You've probably never heard of Windows Server 2008 Foundation, the new product (released April 1st, 2009, how ironic) with a name suspiciously similar to Lotus Foundations, which you have probably also never heard of.

Now, I happened to know about the latter, because it's actually the new name for Nitix, the product I spent about 10 years of my life building. We sold the company to IBM in 2008. (Note: I don't work there anymore, so nothing I say here represents the company, and certainly doesn't represent IBM.)

On some level, it's a huge compliment that Microsoft has acknowledged all our hard work by releasing a whole new product to compete with it. But on the other hand, when Steve Ballmer himself describes this competitive new product as "low-cost, low-price, low-functionality," you have to wonder if the compliment is a bit... backhanded.

Hey, wait a minute. Did I just say that Ballmer described their product as "low-functionality?" What about IBM's product? Did someone get the message backwards?

No, not at all. And this is the genius part.

Microsoft doesn't want you to buy, like, or even respect Windows Server 2008 Foundation. They want you to think it's a cheap, crappy alternative to Serious Windows that nobody would buy unless they were thoroughly desperate. Or as Lisa Szpunar says, "When I read the above list a bell goes off in my head that says nonprofit."

And those other wannabe Foundations products like IBM's? Well. I mean, we wouldn't want to say anything bad about them. That might get us sued. It's kind of an ugly market segment, though, mostly filled with poor, unlucky saps, mostly nonprofits, who are just going to get screwed by paying too little for their computer infrastructure. You're not one of them, are you? Oh, that's too bad.

Windows Server 2008 Foundation is like a straw man argument made real. It exists primarily as a marketing black hole; easy to criticize, and all too easy to become associated with.

(Thanks to Bilal Jaffery for the posting that led me to this.)

Larry Smith on Depressions and Recessions

The growth of women in the paid labour force is a radical change, as is the fact the employment losses are presently a largely male problem.
-- Larry Smith

A new alternative to git submodules: git subtree

Since I first started using git, I've been annoyed by its support (or lack thereof) for "submodules" - the ability to embed the source code for a library, for example, into the source code for your project, and branch/merge them all as a group.

I do this all the time. As you might have guessed from reading this journal, I have worked on a huge number of different things, sometimes many of them in the same day. It turns out that a lot of these projects can benefit from sharing code with each other.

Now, git does have some rudimentary support for submodules through the git submodule command, but it's extremely complicated and error prone, particularly when it comes to switching branches and renaming directories. git checkout branchname is no longer enough to correctly change from one branch to another; git commit && git push is no longer enough to be sure all your changes have made it safely to a shared repository. Perhaps these problems will all be resolved someday, but it isn't looking like that day will be soon.

The problem is that git submodule is designed around a different workflow than I use for myself. If we were building, for example, a "superproject" that contains tons of tools from other sources, and the source code of those tools was huge and rapidly-changing, and the result was a repository 1GB in size or more, then it might make sense to slice and dice your repository and to update the sub-repositories as seldom as possible. Well, maybe - after all, your git history isn't very useful if you can't actually check out some of the old revisions - but apparently some people like it that way. I don't.

What I prefer is to have one big repository with all the stuff my application needs. Of course, what I also need is to allow a bunch of additional small repositories, one for each subproject, so that all my other projects that depend on those subprojects can easily pull from them.

The first hint of a solution to this is what's called the "subtree merge" strategy in git. If you drop a copy of your library in the mylib/ folder in your project, then you can git pull -s subtree from your library in the future, and it'll "automagically" end up in that subdirectory where it belongs. So merging stuff into your superproject from a particular subproject, although it's a little known feature, is basically a solved problem.

The real problem comes when you want to get those changes back out again. Say you make some changes to the library to support your project, and commit them into your project. (Most libraries I write are written at the same time as my applications, because I only develop features as I need them. YAGNI, after all.) Now you need to take those changes out of your application's project and put them in the primary library's project.

If you use git submodule, this is easy, because they were never really merged into your application project in the first place. You were forced to tediously maintain separate projects for each library all along. But if you use a subtree merge, you're in trouble! The two histories are all intermingled, and there's no way to extract one from the other.

...until now. My new git subtree command lets you easily split out the history of a subdirectory, and auto-join it with the original subproject's history so that you can easily push or pull just that piece into the subproject. I call this command git subtree split, the companion to git merge -s subtree.

As an added convenience, there are add, merge, and pull subcommands that make the other common subtree operations a little easier to remember.

Example

Here's an example you can try with the git.git repository. Once upon a time, the gitweb project was maintained separately from git in its own project. They merged it (commit 11e0ef3) into git at one point (commit 0a8f4f0), from which time it was maintained as part of git. Now imagine we wanted to re-extract the gitweb changes from git.git back into the original gitweb project so it could be maintained separately again. Here's what we'd do (you can try this at home if you download git-subtree):

    git clone git://git2.kernel.org/pub/scm/git/git.git
    newtree=$(git subtree split --prefix=gitweb --annotate='(split) ' \
           0a8f4f0^.. --onto=1130ef3 --rejoin)
    git branch latest_gitweb $newtree
    gitk latest_gitweb

If gitweb had originally been merged using 'git subtree add' (or a previous split had been done with --rejoin specified), then you could have left out the weird "0a8f4f0^.." and "--onto=1130ef3" parameters. In fact, after the first split --rejoin, you can incrementally get new changes in the future without remembering any commit ids:

    git subtree split --prefix=gitweb --annotate='(split) ' --rejoin

Current Status

git-subtree is now working, and you can get it (standalone) from its repository at github. It works fine with at least git 1.5.4 and newer, and possibly even earlier versions. I've been using it in some of my own projects. I've also submitted it to the git maintainers, so hopefully git-subtree (or something similar) will make it into future versions of git someday.

Please feel free to send me any questions/comments about git-subtree. It's my first major contribution to git (other than fixing some bugs in git-svn), and I'd like it to be extremely awesome, if possible.

Bonus

Note that, unlike git submodule, git subtree doesn't change the way people using your project need to work. As far as they're concerned, it's just one big project; nobody has to run (or install) git subtree unless they want to. It can just be the responsibility of a single person to extract the subproject history and upload it to the subproject repository, if you want.

::nv,li