SimSWE part 2: The perils of multitasking

I updated my SWE simulator[1] from a few weeks ago. This time, instead of PMs changing their minds about features, we have PMs and execs who choose not to make up their minds at all, letting launch requirements get bloated.

These four charts have the same eng team, same set of features, same pattern of bugs getting filed, same work rate, etc. The only difference is how many "features" (one "bug" can be tagged as required for one or more features) are required for a particular milestone. In this version of the simulator, when we hit a given milestone (reach ~0 bugs in the related features), we launch the product with those features intact. From then on, we have to maintain the product (bugs in already-launched features take precedence) as well as trying to develop new features, all with the same team. Thus, each subsequent feature takes longer to launch, and eventually new features can't launch at all without expanding the team size. (This is why products need bigger teams as they launch more features. In turn, that's why they need more and more revenue.)

When a feature launches, we consider that to be "value" provided to customers. It's a rough proxy for revenue, or increased customer retention, or whatever. Providing value takes time: if I buy a product, I don't get value for it instantaneously. I expect it to keep delivering value as time passes. So the height of the green line is the instantaneous value being delivered (in this simulation, it's simply proportional to number of features shipped). The green area under the curve (the integral of instantaneous product value over time) is the total value that has been delivered so far.

Using this (admittedly oversimplified) model, for this hypothetical product, shipping two features at a time would deliver 4x as much value, after 1200 days, as shipping 5 features at a time. With the same engineering team and the same features! In short, that's what makes SpaceX efficient and others inefficient. SpaceX might work engineers a little harder, but that's not the real benefit. The real benefit is that SpaceX has a clear idea of what really, really needs to happen next, and they deliver it incrementally. Others, mostly, don't.

(You might wonder what happens if we extend the plot to, say, 2000 days. Eventually, the team launches all the features it's capable of maintaining, and just keeps churning away at bugs. The value ratio between chart#2 and chart#3 keeps decreasing, but the absolute difference never goes away. Also note that value delivery usually drops off at a fixed point in the future, when your feature isn't needed anymore (eg. someone else builds a better one). So you can't just extend the time horizon forever to try to make it look better.)

Not shown: the benefits of being able to change your direction, based on customer feedback, after launch #1. The assumption that we are building the same features, but in a different order, is not very realistic. In reality, what you learn after launch #1 is so valuable that you end up changing product direction quite a lot, so early time spent working on features for future milestones is largely wasted.

(The team that completely fails to focus, chart #4, never launches anything because there are so many moving parts that new bugs are found faster than you can fix them. The usual response is to just launch something crappy and plan to fix it in v2. Notice how different that is from the first three charts, where we make no quality sacrifices at all, for a limited subset of functionality, and still launch much earlier. And then we "unrealistically" commit to fixing the bugs in already-launched features before launching new features. You could say that's the difference between many Apple products and other products.)

[1] SWE simulator v1