apenwarr apenwarr - NITLog https://apenwarr.ca/log/ en-ca PyNITLog http://blogs.law.harvard.edu/tech/rss NPS, the good parts Tue, 05 Dec 2023 05:01:12 +0000 https://apenwarr.ca/log/20231204 https://apenwarr.ca/log/20231204 <p>The Net Promoter Score (NPS) is a statistically questionable way to turn a set of 10-point ratings into a single number you can compare with other NPSes. That's not the good part.</p> <p><b>Humans</b></p> <p>To understand the good parts, first we have to start with humans. Humans have emotions, and those emotions are what they mostly use when asked to rate things on a 10-point scale.</p> <p>Almost exactly twenty years ago, I wrote about sitting on a plane next to a <a href="/log/20031227">musician who told me about music album reviews</a>. The worst rating an artist can receive, he said, is a lukewarm one. If people think your music is neutral, it means you didn't make them feel anything at all. You failed. Someone might buy music that reviewers hate, or buy music that people love, but they aren't really that interested in music that is just kinda meh. They listen to music because they want to feel something.</p> <p>(At the time I contrasted that with tech reviews in computer magazines (remember those?), and how negative ratings were the worst thing for a tech product, so magazines never produced them, lest they get fewer free samples. All these years later, journalism is dead but we're still debating the ethics of game companies sponsoring Twitch streams. You can bet there's no sponsored game that gets an actively negative review during 5+ hours of gameplay and still gets more money from that sponsor. If artists just want you to feel something, but no vendor will pay for a game review that says it sucks, I wonder what that says about video game companies and art?)</p> <p>Anyway, when you ask regular humans, who are not being sponsored, to rate things on a 10-point scale, they will rate based on their emotions. Most of the ratings will be just kinda meh, because most products are, if we're honest, just kinda meh. I go through most of my days using a variety of products and services that do not, on any more than the rarest basis, elicit any emotion at all. Mostly I don't notice those. I notice when I have experiences that are surprisingly good, or (less surprisingly but still notably) bad. Or, I notice when one of the services in any of those three categories asks me to rate them on a 10-point scale.</p> <p><b>The moment</b></p> <p>The moment when they ask me is important. Many products and services are just kinda invisibly meh, most of the time, so perhaps I'd give them a meh rating. But if my bluetooth headphones are currently failing to connect, or I just had to use an airline's online international check-in system and it once again rejected my passport for no reason, then maybe my score will be extra low. Or if Apple releases a new laptop that finally brings back a non-sucky keyboard after making laptops with sucky keyboards for literally years because of some obscure internal political battle, maybe I'll give a high rating for a while.</p> <p>If you're a person who likes manipulating ratings, you'll figure out what moments are best for asking for the rating you want. But let's assume you're above that sort of thing, because that's not one of the good parts.</p> <p><b>The calibration</b></p> <p>Just now I said that if I'm using an invisible meh product or service, I would rate it with a meh rating. But that's not true in real life, because even though I was having no emotion about, say, Google Meet during a call, perhaps when they ask me (after every...single...call) how it was, that makes me feel an emotion after all. Maybe that emotion is "leave me alone, you ask me this way too often." Or maybe I've learned that if I pick anything other than five stars, I get a clicky multi-tab questionnaire that I don't have time to answer, so I almost always pick five stars unless the experience was <em>so</em> bad that I feel it's worth an extra minute because I simply need to tell the unresponsive and uncaring machine how I really feel.</p> <p>Google Meet never gets a meh rating. It's designed not to. In Google Meet, meh gets five stars.</p> <p>Or maybe I bought something from Amazon and it came with a thank-you card begging for a 5-star rating (this happens). Or a restaurant offers free stuff if I leave a 5-star rating and prove it (this happens). Or I ride in an Uber and there's a sign on the back seat talking about how they really need a 5-star rating because this job is essential so they can support their family and too many 4-star ratings get them disqualified (this happens, though apparently not at UberEats). Okay. As one of my high school teachers, Physics I think, once said, "A's don't cost me anything. What grade do you want?" (He was that kind of teacher. I learned a lot.)</p> <p>I'm not a professional reviewer. Almost nobody you ask is a professional reviewer. Most people don't actually care; they have no basis for comparison; just about anything will influence their score. They will not feel badly about this. They're just trying to exit your stupid popup interruption as quickly as possible, and half the time they would have mashed the X button instead but you hid it, so they mashed this one instead. People's answers will be... untrustworthy at best.</p> <p>That's not the good part.</p> <p><b>And yet</b></p> <p>And yet. As in so many things, randomness tends to average out, <a href="https://en.wikipedia.org/wiki/Central_limit_theorem">probably into a Gaussian distribution, says the Central Limit Theorem</a>.</p> <p>The Central Limit Theorem is the fun-destroying reason that you can't just average 10-point ratings or star ratings and get something useful: most scores are meh, a few are extra bad, a few are extra good, and the next thing you know, every Uber driver is a 4.997. Or you can <a href="https://xkcd.com/325/">ship a bobcat one in 30 times</a> and still get 97% positive feedback.</p> <p>There's some deep truth hidden in NPS calculations: that meh ratings mean nothing, that the frequency of strong emotions matters a lot, and that deliriously happy moments don't average out disastrous ones.</p> <p>Deming might call this <a href="/log/20161226">the continuous region and the "special causes"</a> (outliers). NPS is all about counting outliers, and averages don't work on outliers.</p> <p><b>The degrees of meh</b></p> <p>Just kidding, there are no degrees of meh. If you're not feeling anything, you're just not. You're not feeling more nothing, or less nothing.</p> <p>One of my friends used to say, on a scale of 6 to 9, how good is this? It was a joke about how nobody ever gives a score less than 6 out of 10, and nothing ever deserves a 10. It was one of those jokes that was never funny because they always had to explain it. But they seemed to enjoy explaining it, and after hearing the explanation the first several times, that part was kinda funny. Anyway, if you took the 6-to-9 instructions seriously, you'd end up rating almost everything between 7 and 8, just to save room for something unimaginably bad or unimaginably good, just like you did with 1-to-10, so it didn't help at all.</p> <p>And so, the NPS people say, rather than changing the scale, let's just define meaningful regions in the existing scale. Only very angry people use scores like 1-6. Only very happy people use scores like 9 or 10. And if you're not one of those you're meh. It doesn't matter how meh. And in fact, it doesn't matter much whether you're "5 angry" or "1 angry"; that says more about your internal rating system than about the degree of what you experienced. Similarly with 9 vs 10; it seems like you're quite happy. Let's not split hairs.</p> <p>So with NPS we take a 10-point scale and turn it into a 3-point scale. The exact opposite of my old friend: you know people misuse the 10-point scale, but instead of giving them a new 3-point scale to misuse, you just postprocess the 10-point scale to clean it up. And now we have a 3-point scale with 3 meaningful points. That's a good part.</p> <p><b>Evangelism</b></p> <p>So then what? Average out the measurements on the newly calibrated 1-2-3 scale, right?</p> <p>Still no. It turns out there are three kinds of people: the ones so mad they will tell everyone how mad they are about your thing; the ones who don't care and will never think about you again if they can avoid it; and the ones who had such an over-the-top amazing experience that they will tell everyone how happy they are about your thing.</p> <p>NPS says, you really care about the 1s and the 3s, but averaging them makes no sense. And the 2s have no effect on anything, so you can just leave them out.</p> <p>Cool, right?</p> <p>Pretty cool. Unfortunately, that's still two valuable numbers but we promised you one single score. So NPS says, let's subtract them! Yay! Okay, no. That's not the good part.</p> <p><b>The threefold path</b></p> <p>I like to look at it this way instead. First of all, we have computers now, we're not tracking ratings on one of those 1980s desktop bookkeeping printer-calculators, you don't have to make every analysis into one single all-encompassing number.</p> <p>Postprocessing a 10-point scale into a 3-point one, that seems pretty smart. But you have to stop there. Maybe you now have three separate aggregate numbers. That's tough, I'm sorry. Here's a nickel, kid, go sell your personal information in exchange for a spreadsheet app. (I don't know what you'll do with the nickel. Anyway I don't need it. Here. Go.)</p> <p>Each of those three rating types gives you something different you can do in response:</p> <ul> <li> <p>The <b>ones</b> had a very bad experience, which is hopefully an outlier, unless you're Comcast or the New York Times subscription department. Normally you want to get rid of every bad experience. The absence of awful isn't greatness, it's just meh, but meh is infinitely better than awful. Eliminating negative outliers is a whole job. It's a job filled with Deming's special causes. It's hard, and it requires creativity, but it really matters.</p> </li> <li> <p>The <b>twos</b> had a meh experience. This is, most commonly, the majority. But perhaps they could have had a better experience. Perhaps even a great one? Deming would say you can and should work to improve the average experience and reduce the standard deviation. That's the dream; heck, what if the average experience could be an amazing one? That's rarely achieved, but a few products achieve it, especially luxury brands. And maybe that Broadway show, Hamilton? I don't know, I couldn't get tickets, because everyone said it was great so it was always sold out and I guess that's my point.</p> <p>If getting the average up to three is too hard or will take too long (and it will take a long time!), you could still try to at least randomly turn a few of them into threes. For example, they say users who have a great customer support experience often rate a product more highly than the ones who never needed to contact support at all, because the support interaction made the company feel more personal. Maybe you can't afford to interact with everyone, but if you have to interact anyway, perhaps you can use that chance to make it great instead of meh.</p> </li> <li> <p>The <b>threes</b> already had an amazing experience. Nothing to do, right? No! These are the people who are, or who can become, your superfan evangelists. Sometimes that happens on its own, but often people don't know where to put that excess positive energy. You can help them. Pop stars and fashion brands know all about this; get some true believers really excited about your product, and the impact is huge. This is a completely different job than turning ones into twos, or twos into threes.</p> </li> </ul> <p><b>What not to do</b></p> <p>Those are all good parts. Let's ignore that unfortunately they aren't part of NPS at all and we've strayed way off topic.</p> <p>From here, there are several additional things you can do, but it turns out you shouldn't.</p> <p><b>Don't compare scores with other products.</b> I guarantee you, your methodology isn't the same as theirs. The slightest change in timing or presentation will change the score in incomparable ways. You just can't. I'm sorry.</p> <p><b>Don't reward your team based on aggregate ratings.</b> They will find a way to change the ratings. Trust me, it's too easy.</p> <p><b>Don't average or difference the bad with the great.</b> The two groups have nothing to do with each other, require completely different responses (usually from different teams), and are often very small. They're outliers after all. They're by definition not the mainstream. Outlier data is very noisy and each terrible experience is different from the others; each deliriously happy experience is special. As the famous writer said, <a href="https://en.wikipedia.org/wiki/Anna_Karenina_principle">all meh families are alike</a>.</p> <p><b>Don't fret about which "standard" rating ranges translate to bad-meh-good.</b> Your particular survey or product will have the bad outliers, the big centre, and the great outliers. Run your survey enough and you'll be able to find them.</p> <p><b>Don't call it NPS.</b> NPS nowadays has a bad reputation. Nobody can really explain the bad reputation; I've asked. But they've all heard it's bad and wrong and misguided and unscientific and "not real statistics" and gives wrong answers and leads to bad incentives. You don't want that stigma attached to your survey mechanic. But if you call it a <em>satisfaction survey</em> on a 10-point or 5-point scale, tada, clear skies and lush green fields ahead.</p> <p><b>Bonus advice</b></p> <p>Perhaps the neatest thing about NPS is how much information you can get from just one simple question that can be answered with the same effort it takes to dismiss a popup.</p> <p>I joked about Google Meet earlier, but I wasn't really kidding; after having a few meetings, if I had learned that I could just rank from 1 to 5 stars and then <em>not</em> get guilted for giving anything other than 5, I would do it. It would be great science and pretty unobtrusive. As it is, I lie instead. (I don't even skip, because it's faster to get back to the menu by lying than by skipping.)</p> <p>While we're here, only the weirdest people want to answer a survey that says it will take "just 5 minutes" or "just 30 seconds." I don't have 30 seconds, I'm busy being mad/meh/excited about your product, I have other things to do! But I can click just one single star rating, as long as I'm 100% confident that the survey will go the heck away after that. (And don't even get me started about the extra layer in "Can we ask you a few simple questions about our website? Yes or no")</p> <p>Also, don't be the survey that promises one question and then asks "just one more question." Be the survey that gets a reputation for really truly asking that one question. Then ask it, optionally, in more places and more often. A good role model is those knowledgebases where every article offers just thumbs up or thumbs down (or the default of no click, which means meh). That way you can legitimately look at aggregates or even the same person's answers over time, at different points in the app, after they have different parts of the experience. And you can compare scores at the same point after you update the experience.</p> <p>But for heaven's sake, not by just averaging them.</p> Interesting Fri, 06 Oct 2023 20:59:31 +0000 https://apenwarr.ca/log/20231006 https://apenwarr.ca/log/20231006 <p>A few conversations last week made me realize I use the word “interesting” in an unusual way.</p> <p>I rely heavily on mental models. Of course, everyone <em>relies</em> on mental models. But I do it intentionally and I push it extra hard.</p> <p>What I mean by that is, when I’m making predictions about what will happen next, I mostly don’t look around me and make a judgement based on my immediate surroundings. Instead, I look at what I see, try to match it to something inside my mental model, and then let the mental model extrapolate what “should” happen from there.</p> <p>If this sounds predictably error prone: yes. It is.</p> <p>But it’s also powerful, when used the right way, which I try to do. Here’s my system.</p> <p><b>Confirmation bias</b></p> <p>First of all, let’s acknowledge the problem with mental models: confirmation bias. Confirmation bias is the tendency of all people, including me and you, to consciously or subconsciously look for evidence to support what we already believe to be true, and try to ignore or reject evidence that disagrees with our beliefs.</p> <p>This is just something your brain does. If you believe you’re exempt from this, you’re wrong, and dangerously so. Confirmation bias gives you more certainty where certainty is not necessarily warranted, and we all act on that unwarranted certainty sometimes.</p> <p>On the one hand, we would all collapse from stress and probably die from bear attacks if we didn’t maintain some amount of certainty, even if it’s certainty about wrong things. But on the other hand, certainty about wrong things is pretty inefficient.</p> <p>There’s a word for the feeling of stress when your brain is working hard to ignore or reject evidence against your beliefs: cognitive dissonance. Certain Internet Dingbats have recently made entire careers talking about how to build and exploit cognitive dissonance, so I’ll try to change the subject quickly, but I’ll say this: cognitive dissonance is bad… if you don’t realize you’re having it.</p> <p>But your own cognitive dissonance is <em>amazingly useful</em> if you notice the feeling and use it as a tool.</p> <p><b>The search for dissonance</b></p> <p>Whether you like it or not, your brain is going to be working full time, on automatic pilot, in the background, looking for evidence to support your beliefs. But you know that; at least, you know it now because I just told you. You can be aware of this effect, but you can’t prevent it, which is annoying.</p> <p>But you can try to compensate for it. What that means is using the part of your brain you have control over — the supposedly rational part — to look for the opposite: things that don’t match what you believe.</p> <p>To take a slight detour, what’s the relationship between your beliefs and your mental model? For the purposes of this discussion, I’m going to say that mental models are a <em>system for generating beliefs.</em> Beliefs are the output of mental models. And there’s a feedback loop: beliefs are also the things you generalize in order to produce your mental model. (Self-proclaimed ”Bayesians” will know what I’m talking about here.)</p> <p>So let’s put it this way: your mental model, combined with current observations, produce your set of beliefs about the world and about what will happen next.</p> <p>Now, what happens if what you expected to happen next, doesn’t happen? Or something happens that was entirely unexpected? Or even, what if someone tells you you’re wrong and they expect something else to happen?</p> <p>Those situations are some of the most useful ones in the world. They’re what I mean by <em>interesting</em>. </p> <p><b>The “aha” moment</b></p> <ul> <i>The most exciting phrase to hear in science, the one that heralds new discoveries, is not “Eureka!” (I found it!) but “That’s funny…”</i> <br> &nbsp;&nbsp;&nbsp;&nbsp;— <a href="https://quoteinvestigator.com/2015/03/02/eureka-funny/">possibly</a> Isaac Asimov </ul> <p>When you encounter evidence that your mental model mismatches someone else’s model, that’s an exciting opportunity to compare and figure out which one of you is wrong (or both). Not everybody is super excited about doing that with you, so you have to be be respectful. But the most important people to surround yourself with, at least for mental model purposes, are the ones who will talk it through with you.</p> <p>Or, if you get really lucky, your predictions turn out to be demonstrably concretely wrong. That’s an even bigger opportunity, because now you get to figure out what part of your mental model is mistaken, and you don’t have to negotiate with a possibly-unwilling partner in order to do it. It’s you against reality. It’s science: you had a hypothesis, you did an experiment, your hypothesis was proven wrong. Neat! Now we’re getting somewhere.</p> <p>What follows is then the often-tedious process of figuring out what actual thing was wrong with your model, updating the model, generating new outputs that presumably match your current observations, and then generating new hypotheses that you can try out to see if the new model works better more generally.</p> <p>For physicists, this whole process can sometimes take decades and require building multiple supercolliders. For most of us, it often takes less time than that, so we should count ourselves fortunate even if sometimes we get frustrated.</p> <p>The reason we update our model, of course, is that most of the time, the update changes a lot more predictions than just the one you’re working with right now. Turning observations back into generalizable mental models allows you to learn things you’ve never been taught; perhaps things nobody has ever learned before. That’s a superpower.</p> <p><b>Proceeding under uncertainty</b></p> <p>But we still have a problem: that pesky slowness. Observing outcomes, updating models, generating new hypotheses, and repeating the loop, although productive, can be very time consuming. My guess is that’s why we didn’t evolve to do that loop most of the time. Analysis paralysis is no good when a tiger is chasing you and you’re worried your preconceived notion that it wants to eat you may or may not be correct.</p> <p>Let’s tie this back to business for a moment.</p> <p>You have evidence that your mental model about your business is not correct. For example, let’s say you have two teams of people, both very smart and well-informed, who believe conflicting things about what you should do next. That’s <em>interesting</em>, because first of all, your mental model is that these two groups of people are very smart and make right decisions almost all the time, or you wouldn’t have hired them. How can two conflicting things be the right decision? They probably can’t. That means we have a few possibilities:</p> <ol> <li>The first group is right</li> <li>The second group is right</li> <li>Both groups are wrong</li> <li>The appearance of conflict is actually not correct, because you missed something critical</li> </ol> <p>There is also often a fifth possibility:</p> <ul> <li>Okay, it’s probably one of the first four but I don’t have time to figure that out right now</li> </ul> <p>In that case, there’s various wisdom out there involving <a href="https://www.inc.com/jeff-haden/amazon-founder-jeff-bezos-this-is-how-successful-people-make-such-smart-decisions.html">one- vs two-way doors</a>, and oxen pulling in different directions, and so on. But it comes down to this: almost always, it’s better to get everyone aligned to the same direction, even if it’s a somewhat wrong direction, than to have different people going in different directions.</p> <p>To be honest, I quite dislike it when that’s necessary. But sometimes it is, and you might as well accept it in the short term.</p> <p>The way I make myself feel better about it is to choose the path that will allow us to learn as much as possible, as quickly as possible, in order to update our mental models as quickly as possible (without doing <em>too</em> much damage) so we have fewer of these situations in the future. In other words, yes, we “bias toward action” — but maybe more of a “bias toward learning.” And even after the action has started, we don’t stop trying to figure out the truth.</p> <p><b>Being wrong</b></p> <p>Leaving aside many philosophers’ objections to the idea that “the truth” exists, I think we can all agree that being wrong is pretty uncomfortable. Partly that’s cognitive dissonance again, and partly it’s just being embarrassed in front of your peers. But for me, what matters more is the objective operational expense of the bad decisions we make by being wrong.</p> <p>You know what’s even worse (and more embarrassing, and more expensive) than being wrong? Being wrong for <em>even longer</em> because we ignored the evidence in front of our eyes.</p> <p>You might have to talk yourself into this point of view. For many of us, admitting wrongness hurts more than continuing wrongness. But if you can pull off that change in perspective, you’ll be able to do things few other people can.</p> <p><b>Bonus: Strong opinions held weakly</b></p> <p>Like many young naive nerds, when I first heard of the idea of “strong opinions held weakly,” I thought it was a pretty good idea. At least, clearly more productive than weak opinions held weakly (which are fine if you want to keep your job), or weak opinions held strongly (which usually keep you out of the spotlight).</p> <p>The real competitor to strong opinions held weakly is, of course, strong opinions held strongly. We’ve all met those people. They are supremely confident and inspiring, until they inspire everyone to jump off a cliff with them.</p> <p>Strong opinions held weakly, on the other hand, is really an invitation to debate. If you disagree with me, why not try to convince me otherwise? Let the best idea win.</p> <p>After some decades of experience with this approach, however, I eventually learned that the problem with this framing is the word “debate.” Everyone has a mental model, but not everyone wants to debate it. And if you’re really good at debating — the thing they teach you to be, in debate club or whatever — then you learn how to “win” debates without uncovering actual truth.</p> <p>Some days it feels like most of the Internet today is people “debating” their weakly-held strong beliefs and pulling out every rhetorical trick they can find, in order to “win” some kind of low-stakes war of opinion where there was no right answer in the first place.</p> <p>Anyway, I don’t recommend it, it’s kind of a waste of time. The people who want to hang out with you at the debate club are the people who already, secretly, have the same mental models as you in all the ways that matter.</p> <p>What’s really useful, and way harder, is to find the people who are not interested in debating you at all, and figure out why.</p> Tech debt metaphor maximalism Tue, 11 Jul 2023 03:12:47 +0000 https://apenwarr.ca/log/20230605 https://apenwarr.ca/log/20230605 <p>I really like the "tech debt" metaphor. A lot of people don't, but I think that's because they either don't extend the metaphor far enough, or because they don't properly understand financial debt.</p> <p>So let's talk about debt!</p> <p><b>Consumer debt vs capital investment</b></p> <p>Back in school my professor, <a href="http://lwsmith.ca/">Canadian economics superhero Larry Smith</a>, explained debt this way (paraphrased): debt is stupid if it's for instant gratification that you pay for later, with interest. But debt is great if it means you can make more money than the interest payments.</p> <p>A family that takes on high-interest credit card debt for a visit to Disneyland is wasting money. If you think you can pay it off in a year, you'll pay 20%-ish interest for that year for no reason. You can instead save up for a year and get the same gratification next year without the 20% surcharge.</p> <p>But if you want to buy a $500k machine that will earn your factory an additional $1M/year in revenue, it would be foolish <em>not</em> to buy it now, even with 20% interest ($100k/year). That's a profit of $900k in just the first year! (excluding depreciation)</p> <p>There's a reason profitable companies with CFOs take on debt, and often the total debt increases rather than decreases over time. They're not idiots. They're making a rational choice that's win-win for everyone. (The company earns more money faster, the banks earn interest, the interest gets paid out to consumers' deposit accounts.)</p> <p>Debt is bad when you take out the wrong kind, or you mismanage it, or it has weird strings attached (hello Venture Debt that requires you to put all your savings in <a href="https://www.washingtonpost.com/business/2023/03/15/svb-billions-uninsured-assets-companies/">one underinsured place</a>). But done right, debt is a way to move faster instead of slower.</p> <p><b>High-interest vs low-interest debt</b></p> <p>For a consumer, the highest interest rates are for "store" credit cards, the kinds issued by Best Buy or Macy's or whatever that only work in that one store. They aren't as picky about risk (thus have more defaults) because it's the ultimate loyalty programme: it gets people to spend more at their store instead of other stores, in some cases because it's the only place that would issue those people debt in the first place.</p> <p>The second-highest interest rate is on a general-purpose credit card like Visa or Mastercard. They can get away with high interest rates because they're also the payment system and so they're very convenient.</p> <p>(Incidentally, when I looked at the stats a decade or so ago, in Canada credit cards make <em>most</em> of their income on payment fees because Canadians are annoyingly persistent about paying off their cards; in the US it's the opposite. The rumours are true: Canadians really are more cautious about spending.)</p> <p>If you have a good credit rating, you can get better interest rates on a bank-issued "line of credit" (LOC) (lower interest rate, but less convenient than a card). In Canada, one reason many people pay off their credit card each month is simply that they transfer the balance to a lower-interest LOC.</p> <p>Even lower interest rates can be obtained if you're willing to provide collateral: most obviously, the equity in your home. This greatly reduces the risk for the lender because they can repossess and then resell your home if you don't pay up. Which is pretty good for them even if you don't pay, but what's better is it makes you much more likely to pay rather than lose your home.</p> <p>Some people argue that you should almost never plan to pay off your mortgage: typical mortgage interest rates are lower than the rates you'd get long-term from investing in the S&amp;P. The advice that you should "always buy the biggest home you can afford" is often perversely accurate, especially if you believe property values will keep going up. And subject to your risk tolerance and lock-in preferences.</p> <p>What's the pattern here? Just this: high-interest debt is quick and convenient but you should pay it off quickly. Sometimes you pay it off just by converting to longer-term lower-rate debt. Sometimes debt is collateralized and sometimes it isn't.</p> <p><b>High-interest and low-interest tech debt</b></p> <p>Bringing that back to tech debt: a simple kind of high-interest short-term debt would be committing code without tests or documentation. Yay, it works, ship it! And truthfully, maybe you should, because the revenue (and customer feedback) you get from shipping fast can outweigh how much more bug-prone you made the code in the short term.</p> <p>But like all high-interest debt, you should plan to pay it back fast. Tech debt generally manifests as a slowdown in your development velocity (ie. overhead on everything else you do), which means fewer features launched in the medium-long term, which means less revenue and customer feedback.</p> <p>Whoa, weird, right? This short-term high-interest debt both <em>increases</em> revenue and feedback rate, and <em>decreases</em> it. Why?</p> <ul> <li> <p>If you take a single pull request (PR) that adds a new feature, and launch it without tests or documentation, you will definitely get the benefits of that PR sooner.</p> </li> <li> <p>Every PR you try to write after that, before adding the tests and docs (ie. repaying the debt) will be slower because you risk creating undetected bugs or running into undocumented edge cases.</p> </li> <li> <p>If you take a long time to pay off the debt, the slowdown in future launches will outweigh the speedup from the first launch.</p> </li> </ul> <p>This is exactly how CFOs manage corporate financial debt. Debt is a drain on your revenues; the thing you did to incur the debt is a boost to your revenues; if you take too long to pay back the debt, it's an overall loss.</p> <p>CFOs can calculate that. Engineers don't like to. (Partly because tech debt is less quantifiable. And partly because engineers are the sort of people who pay off their loans sooner than they mathematically should, as a matter of principle.)</p> <p><b>Debt ceilings</b></p> <p>The US government has imposed a <a href="https://www.reuters.com/world/us/biden-signs-bill-lifting-us-debt-limit-2023-06-03/">famously ill-advised debt ceiling</a> on itself, that mainly serves to cause drama and create a great place to push through unrelated riders that nobody will read, because the bill to raise the debt ceiling will always pass.</p> <p>Real-life debt ceilings are defined by your creditworthiness: banks simply will not lend you more money if you've got so much outstanding debt that they don't believe you can handle the interest payments. That's your credit limit, or the largest mortgage they'll let you have.</p> <p>Banks take a systematic approach to calculating the debt ceiling for each client. How much can we lend you so that you take out the biggest loan you possibly can, thus paying as much interest as possible, without starving to death or (even worse) missing more than two consecutive payments? Also, morbidly but honestly, since debts are generally not passed down to your descendants, they would like you to be able to just barely pay it all off (perhaps by selling off all your assets) right before you kick the bucket.</p> <p>They can math this, they're good at it. Remember, they don't want you to pay it off early. If you have leftover money you might use it to pay down your debt. That's no good, because less debt means lower interest payments. They'd rather you incur even more debt, then use that leftover monthly income even for bigger interest payments. That's when you're trapped.</p> <p>The equivalent in tech debt is when you are so far behind that you can barely keep the system running with no improvements at all; the perfect balance. If things get worse over time, you're underwater and will eventually fail. But if you reach this zen state of perfect equilibrium, you can keep going forever, running in place. That's your tech debt ceiling.</p> <p>Unlike the banking world, I can't think of a way to anthropomorphize a villain who wants you to go that far into debt. Maybe the CEO? I guess maybe someone who is trying to juice revenues for a well-timed acquisition. Private Equity firms also specialize in maximizing both financial and technical debt so they can extract the assets while your company slowly dies.</p> <p>Anyway, both in finance and tech, you want to stay well away from your credit limit.</p> <p><b>Debt to income ratios</b></p> <p>There are many imperfect rules of thumb for how much debt is healthy. (Remember, some debt is very often healthy, and only people who don't understand debt rush to pay it all off as fast as they can.)</p> <p>One measure is the debt to income ratio (or for governments, the debt to GDP ratio). The problem with debt-to-income is debt and income are two different things. The first produces a mostly-predictable repayment cost spread over an undefined period of time; the other is a possibly-fast-changing benefit measured annually. One is an amount, the other is a rate.</p> <p>It would be better to measure interest payments as a fraction of revenue. At least that encompasses the distinction between high-interest and low-interest loans. And it compares two cashflow rates rather than the nonsense comparison of a balance sheet measure vs a cashflow measure. Banks love interest-to-income ratios; that's why your income level has such a big impact on your debt ceiling.</p> <p>In the tech world, the interest-to-income equivalent is how much time you spend dealing with overhead compared to building new revenue-generating features. Again, getting to zero overhead is probably not worth it. I like this <a href="https://xkcd.com/1205/">xkcd explanation</a> of what is and is not worth the time:</p> <p><img src="https://imgs.xkcd.com/comics/is_it_worth_the_time.png"></p> <p>Tech debt, in its simplest form, is the time you didn't spend making tasks more efficient. When you think of it that way, it's obvious that zero tech debt is a silly choice.</p> <p>(Note that the interest-to-income ratio in this formulation has nothing to do with financial income. "Tech income" in our metaphor is feature development time, where "tech debt" is what eats up your development time.)</p> <p>(Also note that by this definiton, nowadays tech stacks are so big, complex, and irritable that every project starts with a giant pile of someone else's tech debt on day 1. Enjoy!)</p> <p><b>Debt to equity ratios</b></p> <p>Interest-to-income ratios compare two items from your cashflow statement. Debt-to-equity ratios compare two items from your balance sheet. Which means they, too, are at least not nonsense.</p> <p>"Equity" is unfortunately a lot fuzzier than income. How much is your company worth? Or your product? The potential value of a factory isn't just the value of the machines inside it; it's the amortized income stream you (or a buyer) could get from continuing to operate that factory. Which means it includes the built-up human and business expertise needed to operate the factory.</p> <p>And of course, software is even worse; as many of us know but few businesspeople admit, the value of proprietary software without the people is zero. This is why you hear about acqui-hires (humans create value even if they might quit tomorrow) but never about acqui-codes (code without humans is worthless).</p> <p>Anyway, for a software company the "equity" comes from a variety of factors. In the startup world, Venture Capitalists are -- and I know this is depressing -- the best we have for valuing company equity. They are, of course, not very good at it, but they make it up in volume. As software companies get more mature, valuation becomes more quantifiable and comes back to expectations for the future cashflow statement.</p> <p>Venture Debt is typically weighted heavily on equity (expected future value) and somewhat less on revenue (ability to pay the interest).</p> <p>As the company builds up assets and shows faster growth, the assumed equity value gets bigger and bigger. In the financial world, that means people are willing to issue more debt.</p> <p>(Over in the consumer world: your home is equity. That's why you can get a huge mortgage on a house but your unsecured loan limit is much smaller. So Venture Debt is like a mortgage.)</p> <p>Anyway, back to tech debt: the debt-to-equity ratio is how much tech debt you've taken on compared to the accumulated value, and future growth rate, of your product quality. If your product is acquiring lots of customers fast, you can afford to take on more tech debt so you can acquire more customers even faster.</p> <p>What's weirder is that as the absolute value of product equity increases, you can take on a larger and larger absolute value of tech debt.</p> <p>That feels unexpected. If we're doing so well, why would we want to take on <em>more</em> tech debt? But think of it this way: if your product (thus company) are really growing that fast, you will have more people to pay down the tech debt next year than you do now. In theory, you could even take on so much tech debt this year that your current team can't even pay the interest...</p> <p>...which brings us to leverage. And risk.</p> <p><b>Leverage risk</b></p> <p>Earlier in this article, I mentioned the popular (and surprisingly, often correct!) idea that you should "buy the biggest house you can afford." Why would I want a bigger house? My house is fine. I have a big enough house. How is this good advice?</p> <p>The answer is the amazing multiplying power of leverage.</p> <p>Let's say housing goes up at 5%/year. (I wish it didn't because this rate is fabulously unsustainable. But bear with me.) And let's say you have $100k in savings and $100k in annual income.</p> <p>You could pay cash and buy a house for $100k. Woo hoo, no mortgage! And it'll go up in value by about $5k/year, which is not bad I guess.</p> <p>Or, you could buy a $200k house: a $100k down payment and a $100k mortgage at, say, 3% (fairly common back in 2021), which means $3k/year in interest. But your $200k house goes up by 5% = $10k/year. Now you have an annual gain of $10k - $3k = $7k, much more than the $5k you were making before, with the same money. Sweet!</p> <p>But don't stop there. If the bank will let you get away with it, why not a $1M house with a $100k down payment? That's $1M x 5% = +$50k/year in value, and $900k x 3% = $27k in interest, so a solid $23k in annual (unrealized) capital gain. From the same initial bank balance! Omg we're printing money.</p> <p>(Obviously we're omitting maintenance costs and property tax here. Forgive me. On the other hand, presumably you're getting intangible value from living in a much bigger and fancier house. $AAPL shares don't have skylights and rumpus rooms and that weird statue in bedroom number seven.)</p> <p>What's the catch? Well, the catch is massively increasing risk.</p> <p>Let's say you lose your job and can't afford interest payments. If you bought your $100k house with no mortgage, you're in luck: that house is yours, free and clear. You might not have food but you have a place to live.</p> <p>If you bought the $1M house and have $900k worth of mortgage payments to keep up, you're screwed. Get another job or get ready to move out and disrupt your family and change everything about your standard of living, up to and possibly including bankruptcy, which we'll get to in a bit.</p> <p>Similarly, let's imagine that your property value stops increasing, or (less common in the US for stupid reasons, but common everywhere else) mortage rates go up. The leverage effect multiplies your potential losses just like it multiplies your potential gains.</p> <p>Back to tech debt. What's the analogy?</p> <p>Remember that idea I had above, of incurring extra tech debt this year to keep the revenue growth rolling, and then planning to pay it off next year with the newer and bigger team? Yeah, that actually works... if you keep growing. If you estimated your tech debt interest rate correctly. If that future team materializes. (If you can even motivate that future team to work on tech debt.) If you're rational, next year, about whether you borrow more or not.</p> <p>That thing I said about the perfect equilibrium running-in-place state, when you spend all your time just keeping the machine operating and you have no time to make it better. How do so many companies get themselves into that state? In a word, leverage. They guessed wrong. The growth rate fell off, the new team members didn't materialize or didn't ramp up fast enough.</p> <p>And if you go past equilibrium, you get the worst case: your tech debt interest is greater than your tech production (income). Things get worse and worse and you enter the downward spiral. This is where desperation sets in. The only remaining option is <strike>bankruptcy</strike> Tech Debt Refinancing.</p> <p><b>Refinancing</b></p> <p>Most people who can't afford the interest on their loans don't declare bankruptcy. The step before that is to make an arrangement with your creditors to lower your interest payments. Why would they accept such an agreement? Because if they don't, you'll declare bankruptcy, which is annoying for you but hugely unprofitable for them.</p> <p>The tech metaphor for refinancing is <em>premature deprecation</em>. Yes, people love both service A and service B. Yes, we are even running both services at financial breakeven. But they are slipping, slipping, getting a little worse every month and digging into a hole that I can't escape. In order to pull out of this, I have to stop my payments on A so I can pay back more of B; by then A will be unrecoverably broken. But at least B will live on, to fight another day.</p> <p>Companies do this all the time. Even at huge profitable companies, in some corners you'll occasionally find an understaffed project sliding deeper and deeper into tech debt. Users may still love it, and it may even be net profitable, but not profitable enough to pay for the additional engineering time to dig it out. Such a project is destined to die, and the only question is when. The answer is "whenever some executive finally notices."</p> <p><b>Bankruptcy</b></p> <p>The tech bankruptcy metaphor is an easy one: if refinancing doesn't work and your tech debt continues to spiral downward, sooner or later your finances will follow. When you run out of money you declare bankruptcy; what's interesting is your tech debt disappears at the same time your financial debt does.</p> <p>This is a really important point. You can incur all the tech debt in the world, and while your company is still operating, you at least have some chance of someday paying it back. When your company finally dies, you will find yourself off the hook; the tech debt never needs to be repaid.</p> <p>Okay, for those of us grinding away at code all day, perhaps that sounds perversely refreshing. But it explains lots of corporate behaviour. The more desperate a company gets, the less they care about tech debt. <em>Anything</em> to turn a profit. They're not wrong to do so, but you can see how the downward spiral begins to spiral downward. The more tech debt you incur, the slower your development goes, and the harder it is to do something productive that might make you profitable. You might still pull it off! But your luck will get progressively worse.</p> <p>The reverse is also true. When your company is doing well, you have time to pay back tech debt, or at least to control precisely how much debt you take on and when. To maintain your interest-to-income ratio or debt-to-equity ratio at a reasonable level.</p> <p>When you see a company managing their tech debt carefully, you see a company that is planning for the long term rather than a quick exit. Again, that doesn't mean paying it all back. It means being careful.</p> <p><b>Student loans that are non-dischargeable in bankruptcy</b></p> <p>Since we're here anyway talking about finance, let's talk about the idiotic US government policy of guaranteeing student loans, but also not allowing people to discharge those loans (ie. zero them out) in bankruptcy.</p> <p>What's the effect of this? Well, of course, banks are extremely eager to give these loans out to anybody, at any scale, as fast as they can, because they can't lose. They have all the equity of the US government to back them up. The debt-to-equity ratio is effectively zero.</p> <p>And of course, people who don't understand finance (which they don't teach you until university; catch-22!) take on lots of these loans in the hope of making money in the future.</p> <p>Since anyone who wants to go to university can get a student loan, American universities keep raising their rates until they find the maximum amount that lenders are willing to lend (unlimited!) or foolish borrowers are willing to borrow in the name of the American Dream (so far we haven't found the limit).</p> <p>Where was I? Oh right, tech metaphors.</p> <p>Well, there are two parts here. First, unlimited access to money. Well, the tech world has had plenty of that, prior to the 2022 crash anyway. The result is they hired way too many engineers (students) who did a lot of dumb stuff (going to school) and incurred a lot of tech debt (student loans) that they promised to pay back later when their team got bigger (they earned their Bachelor's degree and got a job), which unfortunately didn't materialize. Oops. They are worse off than if they had skipped all that.</p> <p>Second, inability to discharge the debt in bankruptcy. Okay, you got me. Maybe we've come to the end of our analogy. Maybe US government policies actually, and this is quite an achievement, manage to be even dumber than tech company management. In this one way. Maybe.</p> <p>OR MAYBE YOU <a href="/log/20091224">OPEN SOURCED WVDIAL</a> AND PEOPLE STILL EMAIL YOU FOR HELP DECADES AFTER YOUR FIRST STARTUP IS LONG GONE.</p> <p>Um, sorry for that outburst. I have no idea where that came from.</p> <p><b>Bonus note: bug bankruptcy</b></p> <p>While we're here exploring financial metaphors, I might as well say something about bug bankruptcy. Although I <a href="/log/20171213">have been known to make fun of bug bankruptcy</a>, it too is an excellent metaphor, but only if you take it far enough.</p> <p>For those who haven't heard of this concept, bug bankruptcy happens when your bug tracking database is so full of bugs that you give up and delete them all and start over ("declare bankruptcy").</p> <p>Like financial bankruptcy, it is very tempting: I have this big pile of bills. Gosh, it is a big pile. Downright daunting, if we're honest. Chances are, if I opened all these bills, I would find out that I owe more money than I have, and moreover, next month a bunch more bills will come and I won't be able to pay them either and this is hopeless. That would be stressful. My solution, therefore, is to throw all the bills in the dumpster, call up my friendly neighbourhood bankruptcy trustee, and conveniently discharge all my debt once and for all.</p> <p>Right?</p> <p>Well, not so fast, buddy. Bankruptcy has consequences. First of all, it's kind of annoying to arrange legally. Secondly, it sits on your financial records for like 7 years afterwards, during which time probably nobody will be willing to issue you any loans, because you're empirically the kind of person who does not pay back their loans.</p> <p>And that, my friends, is also how bug bankruptcy works. Although the process for declaring it is easier -- no lawyers or trustees required! -- the long-term destruction of trust is real. If you run a project in which a lot of people spent a bunch of effort filing and investigating bugs (ie. lent you their time in the hope that you'll pay it back by fixing the bugs later), and you just close them all wholesale, you can expect that those people will eventually stop filing bugs. Which, you know, admittedly feels better, just like the hydro company not sending you bills anymore feels better until winter comes and your heater doesn't work and you can't figure out why and you eventually remember "oh, I think someone said this might happen but I forget the details."</p> <p>Anyway, yes, you can do it. But refinancing is better.</p> <p><b>Email bankruptcy</b></p> <p>Email bankruptcy is similar to bug bankruptcy, with one important distinction: nobody ever expected you to answer your email anyway. I'm honestly not sure why people keep sending them.</p> <p>ESPECIALLY EMAILS ABOUT WVDIAL where does that voice keep coming from</p> Systems design 2: What we hope we know Tue, 11 Jul 2023 03:12:47 +0000 https://apenwarr.ca/log/20230415 https://apenwarr.ca/log/20230415 <p>Someone asked if I could write about the rise of AI and Large Language Models (LLMs) and what I think that means for the future of people, technology, society, and so on. Although that's a fun topic, it left me with two problems: I know approximately nothing about AI, and predicting the future is hard even for people who know what they're talking about.</p> <p>Let's try something else instead. I'll tell you a bunch of things I do know that are somehow related to the topic, and then you can predict the future yourself.</p> <p><b>Magic</b></p> <p>I think magic gets a bad reputation for no good reason.</p> <p>First of all, you might be thinking: magic doesn't actually exist. I assure you that it does. We just need to agree on a definition. For our purposes, let's define magic as: something you know is there, but you can't explain.</p> <blockquote> <p><i>Any sufficiently advanced technology is indistinguishable from magic.</i><br /> &mdash; Arthur C. Clarke</p> </blockquote> <p>One outcome of this definition is that something which is mundane and obvious to one person can be magic to another. Many of us understand this concept unconsciously; outside of storybooks, we more often say something "feels like" magic than we say it "is" magic. Magic is a feeling. Sometimes it's a pleasant feeling, when things go better than they should for reasons we don't understand. Sometimes it's an annoying feeling, when something works differently than expected and you really want to know why.</p> <p>People often say <a href="https://tailscale.com/">Tailscale</a> feels like magic. This is not a coincidence. I've never seen anyone say ChatGPT feels like magic. That makes me curious.</p> <p><b>Magical thinking</b></p> <p>On the other hand, people who believe AIs and specifically LLMs possess "intelligence" are often accused of "magical thinking." Unlike magic itself, magical thinking is always used derisively. Since we now know what magic means, we know what magical thinking means: a tendency to interpret something as magic instead of trying to understand it. A tendency to care about outcomes rather than mechanisms. The underlying assumption, when someone says you're a victim of magical thinking, is that if you understood the mechanisms, you could make better predictions.</p> <p>When it comes to AI, I doubt it.</p> <p>The mechanisms used in AI systems are pretty simple. But at a large scale, combined cleverly, they create amazingly complex emergent outcomes far beyond what we put in.</p> <p>Emergent outcomes defy expectations. Understanding how transistors work doesn't help you at all to explain why Siri sucks. Understanding semi-permeable cell membranes doesn't help much in figuring out what's the deal with frogs. Mechanisms are not the right level of abstraction at all. You can't get there from here.</p> <p>Magical thinking, it turns out, is absolutely essential to understanding any emergent system. You have to believe in magic to understand anything truly complex.</p> <p>You see, magical thinking is just another way to say Systems Design.</p> <p><b>Emergent complexity</b></p> <p>I don't want to go too far into emergent complexity, but I think it's worth a detour since many of us have not thought much about it. Let me link you to three things you might want to read more about.</p> <p>First and most newsworthy at the moment, there's the recently discovered aperiodic monotile:</p> <div align="center"> <img src="https://apenwarr.ca/img/monotile.png"><p> <a href="https://cs.uwaterloo.ca/~csk/hat/">Monotile image</a><br> by Smith, Myers, Kaplan, and Goodman-Strauss, 2023 </div> <p>The monotile is a surprisingly simple shape that, when tiled compactly across a plane of any size, creates a never-repeating pattern. It was hard to discover but it's easy to use, and creates endlessly variable, endlessly complex output from a very simple input.</p> <p>Secondly, I really enjoyed <a href="https://www.amazon.com/Infinite-Staircase-Universe-Ethics-Mortality/dp/1950665984">The Infinite Staircase</a> by Geoffrey Moore. It's a philosophy book, but it will never be accepted as serious philosophy because it's not written the right way. That said, it draws a map from entropy, to life, to genetics, to memetics, showing how at each step along the ladder, emergent complexity unexpectedly produces a new level that has fundamentally different characteristics from the earlier one. It's a bit of a slog to read but it says things I've never seen anywhere else. Moore even offers a solution to the mind/body duality problem. If you like systems, I think you'll like it.</p> <p>Thirdly, the book <a href="https://en.wikipedia.org/wiki/A_New_Kind_of_Science">A New Kind of Science</a> by Stephen Wolfram has lots and lots of examples of emergent complexity, starting with simple finite state automatons of the sort you might recognize from Conway's Game of Life (though even simpler). For various reasons, the book got a bad reputation and the author appears to be widely disliked. Part of the book's bad reputation is that it claims to describe "science" but was self-published and not peer reviewed, completely unlike science. True, but it's shortsighted to discount the content because of that.</p> <p>The book also made a lot of people mad by saying certain important empirical observations in physics and biology can't be reduced to a math formula, but <em>can</em> be reduced to simple iteration rules. The reasons people got mad about that seem to be:</p> <ul> <li> <p>an iteration rule is technically a math formula;</p> </li> <li> <p>just using iterations instead of formulas hardly justifies calling for a "New Kind of Science" as if there were something wrong with the old kind of science;</p> </li> <li> <p>scientists <em>absolutely bloody despise</em> emergent complexity → systems design → magical thinking.</p> </li> </ul> <p>Science is the opposite of magical thinking. By definition. Right?</p> <p><b>Hypotheses</b></p> <p>A friend's favourite book growing up was <a href="https://www.amazon.com/Zen-Art-Motorcycle-Maintenance-Inquiry/dp/0060589469">Zen and the Art of Motorcycle Maintenance</a>. It's an unusual novel that is not especially about motorcycle maintenance, although actually it does contain quite a lot of motorcycle maintenance. It's worth reading, if you haven't, and even more worth reading if you're no longer in high school because I think some of the topics are deeper than they appear at first.</p> <p>Here's one of my highlights:</p> <blockquote> <p><i>Part Three, that part of formal scientific method called experimentation, is sometimes thought of by romantics as all of science itself because that’s the only part with much visual surface. They see lots of test tubes and bizarre equipment and people running around making discoveries. They do not see the experiment as part of a larger intellectual process and so they often confuse experiments with demonstrations, which look the same. A man conducting a gee-whiz science show with fifty thousand dollars’ worth of Frankenstein equipment is not doing anything scientific if he knows beforehand what the results of his efforts are going to be. A motorcycle mechanic, on the other hand, who honks the horn to see if the battery works is informally conducting a true scientific experiment. He is testing a hypothesis by putting the question to nature.</p> <p>The formation of hypotheses is the most mysterious of all the categories of scientific method. Where they come from, no one knows. A person is sitting somewhere, minding his own business, and suddenly—flash!—he understands something he didn’t understand before. Until it’s tested the hypothesis isn’t truth. For the tests aren’t its source. Its source is somewhere else.</p> <p>A lesser scientist than Einstein might have said, “But scientific knowledge comes from nature. Nature provides the hypotheses.” But Einstein understood that nature does not. Nature provides only experimental data.</i></p> <p>-- <a href="https://www.amazon.com/Zen-Art-Motorcycle-Maintenance-Inquiry/dp/0060589469">Zen and the Art of Motorcycle Maintenance</a></p> </blockquote> <p>I love this observation: much of science is straightforward, logical, almost rote. It's easy to ask questions; toddlers do it. It's not too hard to hire grad student lab assistants to execute experiments. It's relatively easy for an analyst or statistician to look at a pile of observations from an experiment and draw conclusions.</p> <p>There's just one really hard step, the middle step: coming up with <em>testable</em> hypotheses. By testable, we mean, we can design an experiment that is actually possible to execute, that will tell us if the hypothesis is true or not, hopefully leading toward answering the original question. Testable hypotheses are, I've heard, where string theory falls flat. We have lots of theories, lots of hypotheses, and billions of dollars to build supercolliders, but we are surprisingly short of things we are able to test for, in order to make the next leap forward.</p> <p>The book asks, where do hypotheses come from?</p> <p>Science is supposed to be logical. Almost all the steps are logical. But coming up with testable hypotheses is infuriatingly intuitive. Hypotheses don't arise automatically from a question. Even hypotheses that are obvious are often untestable, or not obviously testable.</p> <p>Science training doesn't teach us where hypotheses come from. It assumes they're already there. We spend forever talking about how to run experiments in a valid way and to not bias our observations and to make our results repeatable, but we spend almost no time talking about why we test the things we test in the first place. That's because the answer is embarrassing: nobody knows. The best testable hypotheses come to you in the shower or in a dream or when your grad students are drunk at the bar commiserating with their friends about the tedious lab experiments you assigned because they are so straightforward they don't warrant your attention.</p> <p>Hypotheses are magic. Scientists hate magic.</p> <p><b>Engineering</b></p> <p>But enough about science. Let's talk about applied science: engineering. Engineering is delightful because it doesn't require hypotheses. We simply take the completed science, the outcome of which has produced facts rather than guesses, and then we use our newfound knowledge to build stuff. The most logical and methodical thing in the world. Awesome.</p> <p>Right?</p> <p>Well, hold on.</p> <p>In the first year of my engineering programme back in university, there was a class called Introduction to Engineering. Now, first of all, that's a bad sign, because it was a semester-long course and they obviously were able to fill it, so perhaps engineering isn't quite as simple as it sounds. Admittedly, much of the course involved drafting (for some reason) and lab safety training (for good reasons), but I've forgotten most of that by now. What I do remember was a simple experiment the professor had all of us do.</p> <p>He handed out a bunch of paperclips to everyone in the class. Our job was to take each paperclip and bend the outer arm back and forth until it snapped, then record how many bends each one took. After doing that for about five minutes, we each drew a histogram of our own paperclips, then combined the results for the entire class's collection of paperclips into one big histogram.</p> <p>If you know engineering, you know what we got: a big Gaussian distribution (bell curve). In a sample set that large, a few paperclips snapped within just one or two bends. More lasted for three. A few amazingly resilient high-performers lasted for 20 or more bends ("the long tail"). And so on.</p> <p>At that point in our educations most of us had seen a Gaussian distribution at least once, in some math class where we'd been taught about standard deviations or whatever, without any real understanding. The paperclip experiment was kind of cool because it made the Gaussian distribution feel a lot more real than it did in math formulas. But still, we wondered what any of this had to do with engineering.</p> <p>I will forever be haunted by the professor's answer (paraphrased, of course):</p> <blockquote> <p><i>Nobody in the world knows how to build a paperclip that will never break. We could build one that bends a thousand times, or a million times, but not one that can bend forever. And nobody builds a paperclip that can bend a thousand times, because it would be more expensive than a regular paperclip and nobody needs it.</p> <p>Engineering isn't about building a paperclip that will never break, it's about building a paperclip that will bend enough times to get the job done, at a reasonable price, in sufficient quantities, out of attainable materials, on schedule.</p> <p>Engineering is knowing that no matter how hard you try, some fraction of your paperclips will snap after only one bend, and that's not your fault, that's how reality works, and it's your job to accept that and know exactly what fraction that is and design around it, because if you do engineering wrong, people are going to die. But what's worse, even if you do engineering right, sometimes people might die. As an engineer you are absolutely going to make tradeoffs in which you make things cheaper in exchange for a higher probability that people will die, because the only alternative is not making things at all.</p> <p>In the real world, the failure rate is never zero, even if you do your job perfectly.</i><br /> -- My engineering professor </p> </blockquote> <p>And after that he shared a different anecdote:</p> <blockquote> <p><i>I know some of you were top of your class in high school. Maybe you're used to getting 100% on math tests. Well, this is engineering, not math. If you graduate at the top of your engineering class, we should fail you. It means you didn't learn engineering. You wasted your time. Unless you're going to grad school, nobody in the world cares if you got an 80% or a 99%. Do as little work as you can, to learn most of what we're teaching and graduate with a passable grade and get your money's worth. That's engineering.</i><br /> -- My engineering professor </p> </blockquote> <p>That is also, it was frequently pointed out at the time, the difference between engineering and computer science.</p> <p>(I'm proud to say I successfully did not graduate at the top of my engineering class.)</p> <p><b>Software engineering</b></p> <p>Back in the 1990s when I was learning these things, there was an ongoing vigorous debate about whether software development was or could ever be a form of engineering. Most definitions of engineering were not as edgy as my professor's; engineering definitions mostly revolved around accountability, quality, regulation, ethics. And yes, whose fault it is when people die because of what you made.</p> <p>I know many people reading this weren't even alive in the 1990s, or not programming professionally, or perhaps they just don't remember because it was a long time ago. But let me tell you, things used to be very different back then! Things like automated tests were nearly nonexistent; they had barely been invented. Computer scientists still thought correctness proofs were the way to go as long as you had a Sufficiently Smart Compiler. The standard way to write commercial software was to throw stuff together, then a "quality assurance" team would try running it, and it wouldn't work, and they'd tell you so and sometimes you'd fix it (often breaking something else) and sometimes there was a deadline so you'd ship it, bugs and all, and all this was normal.</p> <p>I mean, it's normal now too. But now we have automated tests. Sometimes.</p> <p>Although much software development today is still not software engineering, some software development today <em>is</em> software engineering. Here are some signs of engineering that you can look for:</p> <ul> <li>Monitoring and tracking error rates</li> <li>SLOs and SLAs and uptime targets</li> <li>Distributed system designs that assume and work around the fact that every component will fail</li> <li>Long-time-period bug burndown charts</li> <li>Continuous improvement and <a href="https://lostgarden.home.blog/2008/05/20/improving-bug-triage-with-user-pain/">user pain</a> tracking</li> <li>Well-tested "unhappy paths" such as degraded operation or inter-region migrations</li> </ul> <p>In short, in software engineering, we acknowledge that failures happen and we measure them, characterize them, and compensate for them. We don't aim for perfection.</p> <p>Software development that isn't engineering is almost the same: failures still happen, of course. Perfection is still not achieved, of course. But only engineers call that success.</p> <p><b>Brute force and cleverness</b></p> <p>There are two ways to solve an engineering problem: the "brute force" way and the clever way.</p> <p>Brute force is the easiest one to describe. You just do something (say graph traversal) in the obvious way, and if that's too slow, you buy more CPUs or bandwidth or whatever and parallelize it harder until the solution comes through within an acceptable amount of time. It costs more, of course, but computers are getting pretty cheap compared to programmer time, so often, the brute force approach is better in all relevant dimensions.</p> <p>The best thing about brute force solutions is you don't need very fancy engineers to do it. You don't need fancy algorithms. You don't need the latest research. You just do the dumbest thing that can possibly work and you throw a lot of money and electricity at it. It's the ultimate successful engineering tradeoff.</p> <p>There's only one catch: sometimes brute force simply cannot get you what you want.</p> <blockquote> <p><i>We can solve any problem by introducing an extra level of indirection… except for the problem of too many levels of indirection.</i><br /> — <a href="https://en.wikipedia.org/wiki/Fundamental_theorem_of_software_engineering#:~:text=%22We%20can%20solve%20any%20problem,expanded%20by%20the%20humorous%20clause%20%22%E2%80%A6">possibly David J. Wheeler</a><br /> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;via Butler Lampson,<br /> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;via Andrew Koenig,<br /> &nbsp;&nbsp;&nbsp;&nbsp;&nbsp;via Peter McCurdy</p> </blockquote> <p>Here's a simple example: if I want to transfer a terabyte of data in less time, I can increase my network throughput. Throughput is an eminently brute-forceable problem. Just run more fibers and/or put fancier routers on each end. You can, in theory, with enough money, use parallelism to get as much aggregate throughput as you want, without limit. Amazing!</p> <p>But the overall outcome has limits imposed by latency. Let's say I get myself 100 terabytes/sec of throughput; my single terabyte of data uses only 0.01 seconds, or 10 milliseconds, of capacity. That's pretty fast! And if I want it faster, just get me 1000 terabytes/sec and it'll only use 1 millisecond, and so on.</p> <p>But that 1 millisecond is not the only thing that matters. If the other end is 100 milliseconds away at the speed of light, then the total transfer time is 101 milliseconds (and 100 milliseconds more to wait for the acknowledgement back!), and brute force will at best save you a fraction of the one millisecond, not any of the 100 milliseconds of latency.</p> <p>Web developers know about this problem: even on the fastest link, eliminating round trips greatly speeds up page loads. Without this, typical page load times stop improving after about 50 Mbps because they become primarily latency-limited.</p> <p>Throughput can always be added with brute force. Cutting latency always requires cleverness.</p> <p><b>Negative latency</b></p> <p>Speaking from a systems design point of view, we say that all real-world systems are "causal": that is, outputs are produced after inputs, never before. As a result, every component you add to a flow can only add latency, never reduce it.</p> <p><img src="https://apenwarr.ca/img/system-1.png"></p> <p>In a boxes-and-arrows network diagram, it's easy to imagine adding more brute force throughput: just add more boxes and arrows, operating in parallel, and add a split/merge step at the beginning and end. Adding boxes is easy. That's brute force.</p> <p><img src="https://apenwarr.ca/img/system-2.png"></p> <p>But the only way to make latency go down, causal systems tell us, is to either remove boxes or reduce the latency added by the boxes.</p> <p>This is often possible, certainly. On a web page, incur fewer round trips. In a router, find ways to speed up the modulation, demodulation, and switching layers. On the Internet, find a more direct route. In a virtual reality headset, eliminate extra frames of buffering or put the compositor closer to the position sensors.</p> <p><img src="https://apenwarr.ca/img/system-3.png"></p> <p>All these things are much harder than just adding more links; all of them require "better" engineering rather than more engineering; all of them have fundamental limits on how much improvement is available at all. It's hard work making causal systems faster.</p> <p>Now, here's the bad news: systems designers can violate causality.</p> <p>Scientists Do Not Like This.</p> <p>Engineers are not so thrilled either.</p> <p>You merely need to accurately predict <strike>the next word</strike> future requests, so that when someone later asks you to do work, it's already done.</p> <p><img src="https://apenwarr.ca/img/system-4.png"></p> <div align=right>The cache adds 3ms of latency to a system that used to take 120ms. But sometimes it lets the <i>overall</i> system finish in 13ms: 107ms faster than the system without the cache. Thus, <i>adding</i> the cache has <i>subtracted</i> 107ms of latency.</div> <p>The result is probabilistic. If you guess wrong, the predictor box slightly increases latency (by having to look up the request and then not find it, before forwarding it on). But if you guess <em>right</em>, you can massively reduce latency, down to nearly nothing. And even better, the more money you throw at your predictor, the more predictions you can run pre-emptively (a technique known as "prefetching"). Eventually one of them has to be right. Right?</p> <p>Well, no, not in any non-trivial cases. (A trivial case would be, say, a web service that increments every time you call it. A Sufficiently Smart Predictor could be right every time and never have to wait for the request. Some people call this Edge Computing.)</p> <p>(By the way, any cache counts as a predictor, even if it doesn't prefetch. A cache <em>predicts</em> that you will need its answers again later so it keeps some of them around and hopes for the best, still reducing latency on average.)</p> <p>Anyway, predictors violate causality, depending on your frame of reference for causality. But they can't do it reliably. They only work when they get lucky. And how often they get lucky depends on the quality of&mdash;oh no&mdash;their <em>hypotheses</em> about what you will need next.</p> <p>You remember where hypotheses come from, right? Magic.</p> <p>All caches are magic. Knowing their mechanism is not enough to predict their outcome.</p> <p>(By the way, this is one of the reasons that Cache Invalidation is one of the "two hard problems in computer science.")</p> <p><b>Insight</b></p> <p>In my last year of high school, the student sitting next to me asked my English teacher why their essay only got a B while mine got an A+. The teacher said: the difference is... insight. Read Avery's essay. It says things I've never heard before. You want to do that. To get an A+, write something insightful.</p> <p>My classmate was, naturally, nonplussed. I still remember this as some of the least actionable advice I've ever heard. Be more insightful? Sure, I'll get right on that.</p> <p>(By an odd coincidence my computer at the time, my first ever Linux PC, was already named <tt>insight</tt> because I thought it sounded cool. I migrated that hostname from one home-built PC to another for several years afterward, Ship of Theseus style, so that no matter how tired and uncreative I might feel, I would always have at least one <tt>insight</tt>.)</p> <p>Anyway, you guessed it. Insight is magic.</p> <p><b>Conciseness</b></p> <p>You will have noticed by now that this article is long. As I've gotten older, my articles seem to have gotten longer. I'm not entirely sure why that is. I'm guessing it's not especially caused by an Abundance of Insight.</p> <blockquote> <p><i>I apologize for such a long letter - I didn't have time to write a short one.</i><br /> — Blaise Pascal <strike>Mark Twain</strike></p> </blockquote> <p>To be fair, however, I think there's at least some insight hidden away in here.</p> <p>But let's say we wanted to distill this post down to something equally useful but shorter and easier to absorb. That leads us to an important question. Is shortening articles <em>brute force</em>, or is it <em>clever</em>?</p> <p>I think the answer is complicated. Anyone can summarize an article; grade schoolers do it (with varying degrees of success) in their book reports. Very bad computer software has been writing auto-abstracts poorly for years. Cole's Notes charges good money for their service. ChatGPT summarizes stuff quite well for a computer, thank you.</p> <blockquote> <p><i>Perfection is achieved, not when there is nothing more to add, but when there is nothing left to take away.</i><br /> ― Antoine de Saint-Exupéry</p> </blockquote> <p>So summarization, or conciseness, or maybe we call it information compression, can be done with little to no insight at all. Perhaps to do it <em>better</em> requires some insight: which parts are worth highlighting, and which are worth leaving out? How do we take even one sentence and say it with fewer words? Exactly which parts of Macbeth should we include because they are going to be on the test? These are hard jobs that require some kind of skill.</p> <p>Or maybe we need to draw a distinction between <em>producing</em> insight and <em>recognizing</em> insight. After all, the good parts of this essay are the insightful parts; anything you already knew can be left out. Something you didn't already know, I bet you can recognize with less effort than it took for me to write it.</p> <blockquote> <p><i>I don't know much about art, but I know what I like.</i><br /> — apparently <a href="https://idiomorigins.org/origin/i-dont-know-much-about-artmusicetc-but-i-know-what-i-like">nobody knows who said this</a></p> </blockquote> <p>Perhaps summarization is not the ability to produce insight, but rather the ability to recognize what other people will experience as insight.</p> <p><b>But Avery, you promised me some opinions about AI</b></p> <p>No I didn't.</p> <p>But let's tie all this together. Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it &ndash; a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.</p> <p>It's quite good at summarizing. I don't know how good. I wonder if there's a way to quantify that.</p> <p>Summarizing <em>well</em> requires the ability to recognize and highlight insight. I don't know if it's good at that. I think it might be.</p> <p>When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights &ndash; things the reader has never heard before &ndash; on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic. Someone else's insight, delivered at the right moment, is as good as an entirely new insight.</p> <p>But can LLM AIs produce entirely new insights? I don't know. Good question.</p> <p>Can I? I don't know that either. This whole article was just a series of quotes and book recommendations.</p> <p>Where do insights come from, anyway?</p> <p>They don't teach that in science class.</p> <p><b>Epilogue</b></p> <style> .spark ~ * { display: none; } .spark ~ #spork9 ~ * { display: block; } </style> <script> function highlight(n) { for (let i = 1; i <= 9; i++) { let q = '#spork' + i + ' ~ *'; let ee = document.querySelectorAll(q); ee.forEach((e) => { if (i == n || i == 9) e.style.display = 'block'; else e.style.display = 'none'; }); } } </script> <p>I asked an AI to improve this article. With apologies to Samuel Beckett, here is ChatGPT's account, <a href="javascript:highlight(1)">expurgated</a>, <a href="javascript:highlight(2)">accelerated</a>, <a href="javascript:highlight(3)">improved</a> and <a href="javascript:highlight(4)">reduced</a>, respectively.</p> <div class='spark'></div> <div id='spork1' class='spork'></div> <hr> <p><b>ChatGPT: Clean up this article to give it a coherent thesis and a clear beginning, middle, and end.</b></p> <p>The article "Systems Design 2: What We Hope We Know" by Paul Graham is a thought-provoking exploration of the rise of AI and Large Language Models (LLMs) and their impact on the future of people, technology, and society. However, the author admits that predicting the future is difficult, even for experts, and that they know very little about AI. Instead of discussing AI, the author explores the concept of magic, magical thinking, and systems design, and how they are all interconnected.</p> <p>The author begins by defining magic as something that you know is there, but you cannot explain. They argue that magic gets a bad reputation for no good reason and that many of us understand the concept of magic unconsciously. The author notes that magical thinking is a tendency to interpret something as magic instead of trying to understand it. It is a tendency to care about outcomes rather than mechanisms. The author points out that understanding emergent complexity requires a belief in magic, or what is referred to as magical thinking. They explain that emergent outcomes defy expectations, and understanding how they work requires a deeper level of abstraction that cannot always be explained through mechanisms.</p> <p>The author offers a few examples of emergent complexity, such as the monotile, which is a surprisingly simple shape that, when tiled across a plane of any size, creates a never-repeating pattern. The author also recommends two books, "The Infinite Staircase" by Geoffrey Moore and "A New Kind of Science" by Stephen Wolfram, that delve into emergent complexity and offer insights into how complex systems work.</p> <p>The article then explores the formation of hypotheses in scientific method and how it is the most mysterious of all the categories. The author notes that hypotheses are the foundation of scientific inquiry, and yet, science training does not teach us where hypotheses come from or how to form them. They argue that hypotheses are magic, and scientists hate magic. The author points out that the best testable hypotheses come spontaneously, and that hypotheses don't arise automatically from a question.</p> <p>The article then differentiates between science and engineering and discusses how engineers accept failures and make tradeoffs. They explain that engineering is about building things that work well enough and that engineering is knowing that no matter how hard you try, some fraction of your creations will fail, and that's how reality works. The author highlights the importance of software engineering and discusses how some software development today is software engineering. They point out that engineers who can accept failures and make tradeoffs are the ones who will succeed in the future.</p> <p>The author then goes into the brute force and clever way to solve engineering problems and how cutting latency always requires cleverness. They explain that brute force is the easiest way to solve engineering problems, but that sometimes it simply cannot get you what you want. The author notes that cutting latency always requires cleverness, and that engineers who can do this are the ones who will succeed in the future.</p> <p>In conclusion, the author argues that understanding emergent complexity requires a belief in magic or magical thinking. They also highlight the importance of engineering and how it accepts failures and makes tradeoffs. The article concludes by pointing out that cutting latency always requires cleverness, and that engineers who can do this are the ones who will succeed in the future. The article is a fascinating exploration of the interconnectedness of magic, emergent complexity, systems design, and engineering, and how they all play a role in shaping the future of technology and society.</p> <div id='spork2' class='spork'></div> <hr> <p><b>ChatGPT: Tell me the most important insights in this article.</b></p> <ul> <li>Magic is something we know is there but can't explain, and it is essential to understanding any emergent system.</li> <li>Magical thinking is a tendency to interpret something as magic instead of trying to understand it, and it is often used derisively.</li> <li>The mechanisms used in AI systems are pretty simple, but at a large scale, combined cleverly, they create amazingly complex emergent outcomes far beyond what we put in.</li> <li>Engineering is knowing that no matter how hard you try, some fraction of your paperclips will snap after only one bend, and that's not your fault, that's how reality works, and it's your job to accept that and know exactly what fraction that is and design around it, because if you do engineering wrong, people are going to die.</li> <li>There are two ways to solve an engineering problem: the "brute force" way and the clever way.</li> <li>The only way to make latency go down is to either remove boxes or reduce the latency added by the boxes.</li> </ul> <div id='spork3' class='spork'></div> <hr> <p><b>ChatGPT: Say something insightful about this topic that is not already in the article, and which the audience is not expected to know already.</b></p> <p>One thing worth noting is that the rise of AI and LLMs has brought about both excitement and concerns. On one hand, these technologies have the potential to revolutionize various industries and improve our lives in ways we can't even imagine yet. On the other hand, there are concerns about the ethical implications of AI, such as job displacement and bias in decision-making. It's important for us to continue to monitor and evaluate the development and implementation of these technologies to ensure that they benefit society as a whole, rather than just a select few. Additionally, it's worth noting that while AI and LLMs are impressive and powerful tools, they are not a panacea for all problems. It's important to recognize the limitations and potential pitfalls of these technologies, and to use them in conjunction with human intelligence and expertise to achieve the best outcomes.</p> <div id='spork4' class='spork'></div> <hr> <p><b>ChatGPT: This article is too boring to read. Summarize it in two sentences.</b></p> <p>The author discusses the rise of AI and Large Language Models (LLMs) and how they are often accused of "magical thinking". He gives examples of how magical thinking is essential to understanding emergent systems and how engineering is about building products that are good enough to get the job done, not perfect.</p> <div id='spork9'></div> The Gift of It's Your Problem Now Sat, 01 Jan 2022 13:32:51 +0000 https://apenwarr.ca/log/20211229 https://apenwarr.ca/log/20211229 <p>Recently a security hole in a certain open source Java library resulted in a worldwide emergency kerfuffle as, say, 40% of the possibly hundreds of millions of worldwide deployments of this library needed to be updated in a hurry. (The other 60% also needed to be updated in a hurry, but won't be until they facilitate some ransomware, which is pretty normal for these situations.)</p> <p>I have a 20+ year history of poking fun at Java in this space, and it pains me to stop now. But the truth is: this could have happened to anyone.</p> <p>What happened was:</p> <ul> <li>Someone wrote a library they thought was neat</li> <li>They decided to share it with the world for free</li> <li>Millions of people liked it and used it everywhere</li> <li>Some contributors contributed some good ideas and, in this case, at least one bad idea</li> <li>Out of a sense of stewardship, they decided to maintain backward compatibility with the bad idea</li> <li>The bad idea turned out to have one or more security flaws that affected all the users</li> <li>The stewards responded quickly with a fix</li> </ul> <p>From this, if you believe the Internet Consensus, we can conclude that open source doesn't work, people don't get paid enough, capitalism is a sham, billionaires are built on the backs of the proletariat, your $50 Patreon donation makes a real difference, and Blorkchain Would Have Solved This.</p> <p>(Miraculously the Internet Consensus is always the same both before and after these kinds of events. In engineering we call this a "non-causal system" because the outputs are produced before the inputs.)</p> <p>Nevertheless, I have been dared to take my take on the issue. It, too, was the same before and after, but the difference is I didn't write it down until now, which makes this journal a causal system. You could probably write an interesting philosophical paper about observations of a non-causal system nevertheless being causal, but mercifully, I will not.</p> <p><b>Free Software is Communism</b></p> <p>So anyway, meandering gently toward the point, let's go back in time to the original Free Software movement. Long ago, before the average reader of this sentence was born, a person whose name is now unpopular was at a university, where they had a printer, and the printer firmware was buggy. This person firmly believed they could quickly fix the printer firmware if only they had the source code. (In the spirit of every "I could do this better in a weekend" story, I'm not sure whether we ever confirmed if this was true. In any case printer firmware is still buggy.)</p> <p>As a result, they started a nonprofit organization to rewrite all of Unix, which the printer did not run and which therefore would not solve any of the original problem, but was a pretty cool project nonetheless and was much more fun than the original problem, and the rest was history.</p> <p>This story archetype is the Hero's Journey that inspires all software development:</p> <ul> <li>I have a problem</li> <li>I do not know how to solve that problem</li> <li>But I strongly believe, without evidence, that I can solve a generalized version of that problem if I form a large enough team and work at it for 35 years(*)</li> <li>We are now accepting donations</li> </ul> <p>(*) initial estimate is often less than 35 years</p> <p>Now, you have probably heard all this before, and if you're a software developer you have probably lived it. This part is not really in question. The burning question for us today, as we enjoy the (hopefully) peak of late-stage capitalism, is: ...<em>but where will the donations come from?</em></p> <p><b>Authoritarianism</b></p> <p>Before we get back onto communism, let me draw an important distinction. Most communist governments in history ended up being authoritarian systems, which is to say, top-down control. Ironically, the people at the top seem to have more power than the people at the bottom, which at first seems like the antithesis of communism. This is not the place to claim an understanding of why that always seems to happen. But one has to acknowledge a pattern when one sees it.</p> <p>On the other hand, it's easy to find examples of authoritarianism outside communism. Our world is filled with top-down control systems. Many corporations are in many ways, top-down controlled. The US system of government is increasingly top-down controlled (ie. authoritarian), despite the many safety measures introduced early to try to prevent that.</p> <p>When politicians rail against <em>communism</em> it is because they don't want you to notice the ever-growing <em>non-communist authoritarianism</em>.</p> <p>Authoritarianism is self-reinforcing. Once some people or groups start having more power, they tend to use that power to adjust or capture the rules of the system so they can accumulate more power, and so on. Sometimes this is peacefully reversible, and sometimes it eventually leads to uprisings and revolutions.</p> <p>People like to write about facism and communism as if they are opposite ends of some spectrum, but that's not really true in the most important sense. Fascism blatantly, and communism accidentally but consistently, leads to authoritarianism. And authoritarianism is the problem.</p> <p>Authoritarianism is about <em>taking things from me.</em> Communism, in its noncorporeal theoretical form, is about <em>giving things away.</em></p> <p>I read a book once which argued that the problem with modern political discourse is it pits the "I don't want things taken from me" (liberty!) people against the "XYZ is a human right" (entitlement!) people. And that a better way to frame the cultural argument is "XYZ is my responsibility to society."</p> <p>As a simple example, "Internet access is a human right," is just a sneaky way of saying "someone should give people free Internet." Who is someone? It's left unspecified, which is skipping over the entire mechanism by which we deliver the Internet. It's much more revealing to write, "To live in a healthy society, it's our responsibility to make sure every person has Internet access." Suddenly, oh, crap. The someone is me!</p> <p>Healthy society is created through constant effort, by all of us, as a gift to our fellow members. It's not extracted from us as a mandatory payment to our overlords who will do all the work.</p> <p>If there's one thing we know for sure about overlords, it's that they never do all the work.</p> <p>Free software is a gift.</p> <p><b>I would like to inquire about the return policy</b></p> <p>Here's the thing about gifts: the sender chooses them, not the recipient. We can have norms around what gifts are appropriate, and agreements to not over-spend, and wishlists, and so on. But I won't always get the exact gift I want. Sometimes I didn't even want a gift. Sometimes the gift interprets JNDI strings in my log messages and executes random code from my LDAP server. This is the nature of gifts.</p> <p>On the other hand, the best gifts are the things I never would have bought for myself, because they seemed too expensive or I didn't even realize I would like them or they were too much work to obtain, or because someone hand-made them just for me. These feel like luxuries of the sort capitalism cannot produce, because deciding, going out, and buying something for myself isn't luxury, it's everyday. It's lonely. It's a negotiation. It's limited by my own lack of creativity.</p> <p>The best part of free software is it sometimes produces stuff you never would have been willing to pay to develop (Linux), and sometimes at quality levels too high to be rational for the market to provide (sqlite).</p> <p>The worst part of free software is you get what you get, and the developers don't have to listen to you. (And as a developer, the gift recipients aren't always so grateful either.)</p> <p><b>Paying for gifts</b></p> <p>...does not work.</p> <p>You don't say to someone, "here's $100, maybe this time get me a gift worth $100 more than you'd regularly spend." It's kind of insulting. It still probably won't get you exactly the thing you wanted. Actually, the other person might just pocket the $100 and run off with it.</p> <p>We already have a way for you to spend $100 to get the thing you want. It's a market. A market works fine for that. It's not very inspiring, but most of the time it's quite efficient. Even gift-givers will often buy things on the same market, but with a different selection criteria, thus adding value of their own.</p> <p>When you try to pay for gifts, it turns the whole gift process into a transaction. It stops being a gift. It becomes an inefficient, misdesigned, awkward market.</p> <p>There's research showing that, for example, financial compensation in a job is more likely a demotivator than a motivator (ie. if you pay me too little, I'll work less hard or quit, but if you double my pay, it won't double my output). If you tie cash compensation to specific metrics, people will game the metrics and usually do an overall worse job. If you pay someone for doing you a favour, they are less likely to repeat the favour. Gifts are inherently socially and emotionally meaningful. Ruin the giftiness, and you ruin the intangible rewards.</p> <p>So it is with free software. You literally cannot pay for it. If you do, it becomes something else.</p> <p>This is why we have things like the Linux Foundation, where the idea is you can give a gift because you appreciate and want to support Linux (and ideally you are a rich megacorporation so your gift is very big), but it dilutes the influence of that money through an organization that supposedly will not try to influence the gift of Linux that was already happening. You end up with multiple gift flows in different directions. Money goes here, code goes there. They are interdependent - maybe if one flow slows down the other flow will also slow down - but not directly tied. It's a delicate balance. People who keep receiving Christmas gifts but never give any might eventually stop receiving them. But might not.</p> <p>Anyway, gifts will not get you 24-hour guaranteed response times to security incidents.</p> <p>Gifts won't get you guaranteed high quality code reviews.</p> <p>Gifts will not, for heaven's sake, prevent developers from implementing bad ideas occasionally that turn into security holes. Nothing will. Have you met developers?</p> <p><b>Open source</b></p> <p>I've avoided the term "open source" so far because it means something different from the original idea of Free Software.</p> <p>Open source was, as I understand it, coined to explain what happened when Netscape originally opened their Mozilla source code, back at the end of the 1990s. That was not a gift. That was a transaction. Or at least, it was intended to be.</p> <p>The promise of open source was:</p> <ul> <li>You, the company, can still mostly control your project</li> <li>Customers will still pay you to add new features</li> <li>Actually customers might pay other people to add new features, but you can still capitalize on it because you get their code too</li> <li>Linux distributions only package open source code so you'll onboard more customers more easily this way</li> <li>You can distance yourself from this anti-capitalist gift-giving philosophical stuff that makes investors nervous</li> <li>Plus a bunch of people will look at the code and find bugs for you for free!</li> </ul> <p>Maybe this sounds cynical, but capitalists are cynical, and you know what? It worked! Okay, not for Netscape Corporation (sorry), but for a lot of other people since then.</p> <p>It also failed a lot of people. Many developers and companies have been disappointed to learn that just uploading your code to github doesn't make a community of developers appear. (It does make it more likely that AWS will fork your product and make more money from it than you do.) Code reviews are famously rare even in security-critical projects. Supply chain issues are rampant.</p> <p>In fact, we've now gotten to the point where some people hesitate to give away their source code, mainly because of this confusion of gifts and customers. If I spend some spare time hacking something together on a weekend and give it away, that's a gift. If you yell at me for making it, that makes giving less fun, and I will spend fewer weekends making gifts.</p> <p>Whereas when a company has a product and open sources it and you complain, that's customers giving valuable feedback and it's worth money to learn from them and service them, because you eventually earn money in exchange (through whatever business model they've established). No gift necessary.</p> <p>Call it cynical or call it a win/win relationship. But it's not a gift.</p> <p><b>The startup ecosystem</b></p> <p>Since the creation of the open source designation 20+ years ago, software startups have taken off more than ever. I attribute this to a combination of factors:</p> <ul> <li>Cloud computing has made it vastly cheaper to get started</li> <li>Incubators like YCombinator have industrialized the process of assembling and running a small software company</li> <li>Megacorps have become exponentially richer but no more creative, so they need to acquire or acqui-hire those startups faster and faster in order to grow.</li> </ul> <p>Although a lot of startups open source their code, and they all depend heavily on open source ecosystems, the startup world's motivations are <em>amazingly different</em> from the free software and open source worlds.</p> <p>Gifts exist in the startup world. They are things like "we were both in YCombinator so I will intro you to this investor I like" or "I got extremely rich so let me invest in your startup and incidentally I get a lottery ticket for becoming even more rich." These absolutely are still gifts. They each strengthen social ties. The startup world is a society, and the society is built up from these gifts. It's a society that largely ignores the trials and tribulations of anyone who isn't a rich software engineer insider, but history has hosted many past societies of that sort and it takes a long time to build and deploy enough guillotines, and anyway they are having fun and producing a lot and surely that counts for something.</p> <p>If free software gifts are communism and open source is cynically capitalist exploitation, then startups may be, weirdly, the democratization of capitalism.</p> <p>Hear me out. Big companies don't care what you think; you can't pay them enough to care. Gift givers care only a little what you think; if they gave you what you wanted, it wouldn't be a gift. But startups, well, there are a lot of them and their mantras are "do things that don't scale" and "focus on the customer" and "build rapid feedback loops." What that spells for you is a whole bunch of people who want to give you what you want, in exchange for money, and who are excited to amortize the costs of that over all the other customers who want the same thing.</p> <p>It's kind of exciting, conceptually, and more self-optimizing than untuned gift giving, and so it's not super surprising to me that it has started to eclipse the earlier concepts of free software and open source. More and more "open" projects are backed by small companies, who have financial incentives to make their users happy because some of the users turn into paying customers. They'll even provide the uptime SLAs and security fix turnaround guarantees you wanted so much. Our company, <a href="https://tailscale.com/company/">Tailscale</a>, is unabashedly one of those. Nothing to be ashamed of there. The system works.</p> <p>What doesn't work is assuming those startup mechanics apply to everyone out there who gives you a software gift. Not every project on github is the same.</p> <p>Not everyone has the same motivations.</p> <p>Giving them money won't change their motivations.</p> <p>Trying to pay them or regulate them taints the gift.</p> <p>If you wanted to pay someone to fix some software, you didn't want a gift. You wanted a company.</p> <p>But if there is no company and someone gave you something anyway? Say thanks.</p> <p><b>Epilogue</b></p> <p>This isn't where evolution stops. There's a lot more to say about how SaaS taints the unwritten agreement of open source (because you don't have to give back your changes to the code), and how startups tend to go bankrupt and their tech dies with them, and how the best developers are not good at starting companies (no matter how much easier it has become), and how acquiring a startup usually destroys all the stuff they innovated, and how open source is often used as a way to exfiltrate past those kinds of disasters, and how simultaneously, whole promising branches of the "gift economy" structure have never been explored. But that's enough for today. Maybe another time.</p> 100 years of whatever this will be Thu, 02 Dec 2021 12:38:46 +0000 https://apenwarr.ca/log/20211201 https://apenwarr.ca/log/20211201 <p>What if all these weird tech trends actually add up to something?</p> <p>Last time, we explored why various bits of trendy technology are, in my opinion, simply never going to be able to achieve their goals. But we ended on a hopeful(?) note: maybe that doesn't matter. Maybe the fact that people really, really, really want it, is enough.</p> <p>Since writing that, I've been thinking about it more.</p> <p>I think we are all gradually becoming more aware of patterns, of major things wrong with our society. They echo some patterns we've been seeing for decades now. The patterns go far beyond tech, extending into economics and politics and culture. There's a growing feeling of malaise many of us feel:</p> <ul> <li>Rich, powerful, greedy people and corporations just get richer, more powerful, and more greedy.</li> <li>Everyone seems to increasingly be in it for themselves, not for society.</li> <li>Or, people who <em>are</em> in it for society tend to lose or to get screwed until they give up.</li> <li>Artists really <em>don't</em> get enough of a reward for all the benefit they provide.</li> <li>Big banks and big governments really <em>do</em> nonspecifically just suck a lot.</li> <li>The gap between the haves and have-nots keeps widening.</li> <li>You can't hope to run an Internet service unless you pay out a fraction to one of the Big Cloud Providers, just like you couldn't run software without paying IBM and then Microsoft, back in those days.</li> <li>Bloody egress fees, man. What a racket.</li> <li>Your phone can run mapreduce jobs 10x-100x faster than your timeshared cloud instance that costs more. Plus it has a GPU.</li> <li>One SSD in a Macbook is ~1000x faster than the default disk in an EC2 instance.</li> <li>Software stacks, governments, and financial systems: they all keep getting more and more bloated and complex while somehow delivering less per dollar, gigahertz, gigabyte, or watt.</li> <li>Computers are so hard to run now, that we are supposed to give up and pay a subscription to someone - well, actually to every software microvendor - to do it for us.</li> <li>We even pay 30% margins to App Stores mainly so they can <em>not</em> let us download apps that are "too dangerous."</li> <li>IT security has become literally impossible: if you install all the patches, you get SolarWinds-style supply chain malware delivered to you automatically. If you don't install the patches, well, that's worse. Either way, enjoy your ransomware.</li> <li>Software intercompatibility is trending toward zero. Text chat apps are literally the easiest thing in the world to imagine making compatible - they just send very short strings, very rarely, to very small networks of people! But I use at least 7 separate ones because every vendor wants their own stupid castle and won't share. Don't even get me started about books or video.</li> <li>The most reasonable daycare and public transit in the Bay Area is available only with your Big Tech Employee ID card.</li> <li>Everything about modern business is designed to funnel money, faster and faster, to a few people who have demonstrated they can be productive. This totally works, up to a point. But we've now reached the extreme corner cases of capitalism. Winning money is surely a motivator, but that motivation goes down the more you have. Eventually it simply stops mattering at all. Capitalism has become a "success disaster."</li> </ul> <p>Writing all this down, you know what? I'm kind of mad about it too. Not so mad that I'll go chasing obviously-ill-fated scurrilous rainbow financial instruments. But there's something here that needs solving. If I'm not solving it, or part of it, or at least trying, then I'm... wasting my time. Who cares about money? This is a systemic train wreck, well underway.</p> <p>We have, in Western society, managed to simultaneously botch the dreams of democracy, capitalism, social coherence, and techno-utopianism, <em>all at once</em>. It's embarrassing actually. I am embarrassed. You should be embarrassed.</p> <p><b>"Decentralization"</b></p> <p>I'm a networking person and a systems person, so please forgive me if I talk about all this through my favourite lens. Societies, governments, economies, social networks, and scalable computing all have something in common: <strong>they are all distributed systems.</strong></p> <p>And.</p> <p>And everyone.</p> <p><em>Everyone</em> seems to have an <em>increasingly horrifically misguided idea</em> of how distributed systems work.</p> <p>There is of course the most obvious horrifically misguided recently-popular "decentralized" system, whose name shall not be spoken in this essay. Instead let's back up to something older and better understood: markets. The fundamental mechanism of the capitalist model.</p> <p>Markets are great! They work! Centrally planning a whole society clearly does not work (demonstrated, bloodily, several times). Centrally planning corporations seems to work, up to a certain size. Connecting those corporations together using markets is the most efficient option we've found so far.</p> <p>But there's a catch. People like to use the term <em>free market</em> to describe the optimal market system, but that's pretty lousy terminology. The truth is, functioning markets are not "free" at all. They are regulated. Unregulated markets rapidly devolve into monopolies, oligopolies, monopsonies, and, if things get <em>really</em> bad, libertarianism. Once you arrive there, every thread ends up with people posting about "a monopoly on the use of force" and "paying taxes at gunpoint" and "I'll run my <em>own</em> fire department" and things that "end at the tip of the other person's nose," and all useful discourse terminates forevermore.</p> <p>The job of market regulation - fundamentally a restriction on your freedom - is to prevent all that bad stuff. Markets work well as long as they're in, as we call it in engineering, the "continuous control region," that is, the part far away from any weird outliers. You need no participant in the market to have too much power. You need downside protection (bankruptcy, social safety net, insurance). You need fair enforcement of contracts (which is different from <em>literal</em> enforcement of contracts).</p> <p>And yet: markets are distributed systems.</p> <p>Even though there are, in fact, very strict regulators and regulations, I can still enter into a contract with you <em>without ever telling anyone</em>. I can buy something from you, in cash, and nobody needs to know. (Tax authorities merely <em>want</em> to know, and anyway, notifying them is asynchronous and lossy.) Prices are set through peer-to-peer negotiation and supply and demand, almost automatically, through what some call an "invisible hand." It's really neat.</p> <p>As long as we're in the continuous control region.</p> <p>As long as the regulators are doing their job.</p> <p>Here's what everyone peddling the new trendy systems is so desperately trying to forget, that makes all of them absurdly expensive and destined to fail, even if the things we want from them are beautiful and desirable and well worth working on. Here is the very bad news:</p> <p><b>Regulation is a centralized function.</b></p> <p>The job of regulation is to <em>stop distributed systems from going awry</em>.</p> <p>Because distributed systems <em>always</em> go awry.</p> <p>If you design a distributed control system to stop a distributed system from going awry, it might even work. It'll be unnecessarily expensive and complex, but it might work... until the control system itself, inevitably, goes awry.</p> <p>I find myself linking to this article way too much lately, but here it is again: <a href="https://www.jofreeman.com/joreen/tyranny.htm">The Tyranny of Structurelessness</a> by Jo Freeman. You should read it. The summary is that in any system, if you don't have an explicit hierarchy, then you have an implicit one.</p> <p>Despite my ongoing best efforts, I have never seen any exception to this rule.</p> <p>Even the fanciest pantsed distributed databases, with all the Rafts and Paxoses and red/greens and active/passives and Byzantine generals and dining philosophers and CAP theorems, are subject to this. You can do a bunch of math to <em>absolutely prove beyond a shadow of a doubt</em> that your database is completely distributed and has no single points of failure. There are papers that do this. You can do it too. Go ahead. I'll wait.</p> <p>&lt;several PhDs later&gt;</p> <p>Okay, great. Now skip paying your AWS bill for a few months.</p> <p>Whoops, there's a hierarchy after all!</p> <p><b>You can stay in denial, or you can get serious.</b></p> <p>Western society, economics, capitalism, finance, government, the tech sector, the cloud. They are all distributed systems already. They are all in severe distress. Things are going very bad very quickly. It will get worse. Major rework is needed. We all feel it.</p> <p>We are not doing the rework.</p> <p>We are chasing rainbows.</p> <p>We don't need deregulation. We need better designed regulation.</p> <p>The major rework we need isn't some math theory, some kind of Paxos for Capitalism, or Paxos for Government. The sad, boring fact is that no fundamental advances in math or computer science are needed to solve these problems.</p> <p>All we need is to build distributed systems <em>that work</em>. That means decentralized bulk activity, hierarchical regulation.</p> <p>As a society, we are so much richer, so much luckier, than we have ever been.</p> <p>It's all so much easier, and harder, than they've been telling you.</p> <p>Let's build what we already know is right.</p> 10 years of... whatever this has been Wed, 17 Nov 2021 22:38:25 +0000 https://apenwarr.ca/log/20211117 https://apenwarr.ca/log/20211117 <p>I guess I know something about train wrecks.</p> <p>One night when I was 10 years old, me and my mom were driving home. We came to a train crossing outside of town. There was another car stopped right on the tracks, stalled. A lady was inside, trying to get her car to start. It didn’t.</p> <p>Train crossings are bumpy, cars were worse then, it was a long time ago, I don’t know, I don’t remember clearly. Anyway, it was cold out and most people didn’t have cell phones yet, so when the car wouldn’t start and it was too heavy to push, there wasn’t much to be done. My mom convinced her to get the heck away from the tracks and come sit in our car to warm up and make a plan. We heard the whistle of an arriving train. And I made what I now consider one of the (several) biggest mistakes of my life: I closed my eyes.</p> <p>It was only a few seconds later when I realized OH MY GOD WHAT WAS I THINKING I COULD HAVE WATCHED A TRAIN DESTROY A CAR RIGHT IN FRONT OF ME!!! But I wasn’t brave enough, I panicked, I closed my eyes, and you know what? The train wreck happened anyway. I just didn’t get to see it.</p> <p>It was in the local newspaper a couple days later. The newspaper said the car ran into the train, and not the other way around. I was boggled. I learned later that this was my first, surprisingly on-the-nose, encounter with the <a href="https://www.epsilontheory.com/gell-mann-amnesia/">Gell-Mann Amnesia Effect</a>. (To this day, I still believe some of the things I read. I have no idea why.)</p> <p>What’s the point of this story? That the train crash still happens, whether or not you’re watching. And everything you've read about it is probably wrong. And I’m glad my mom helped that lady get out of the way.</p> <p>Anyway that’s why I don't mute blockchain-related keywords on twitter.</p> <p><strong>The blockchain train crash</strong></p> <p>Ten years(!) have passed since I wrote <a href="/log/20110508">Why bitcoin will fail</a>. And yet, here we are, still talking about bitcoin. Did it fail?</p> <p>According to the many cryptobots who pester me, apparently not. They still gleefully repost my old article periodically, pointing out that <em>at the time, bitcoins were worth maybe three dollars, and now they're worth infinity dollars, and wow, that apenperson sure must feel dumb for not HODLING BIGTIME back when they had the chance, lol.</em></p> <p>Do I feel dumb? Well, hmm. It’s complicated. Everything I predicted seems to have come true. If your definition of “failure” is “not achieving any of the stated goals,” then I guess bitcoin has profoundly... not succeeded. But that doesn’t really tell the whole story, does it? A proper failure would be in the past tense by now.</p> <p>What I do know is I’ve learned some stuff in 10 years.</p> <p><strong>What I got right</strong></p> <p>But first, let’s review the claims I made in the original article:</p> <ul> <li><strong>If you like bitcoin, you must think the gold standard was a good idea.</strong><br /> To create gold currency, you do pointless busywork (“mining”). Gold is a stupid inconvenient currency that’s worse than paper.<br /> Printing and destroying money is a key economic tool.</li> </ul> <p>Yup. Over the years we’ve seen an ongoing, embarrassing overlap between “goldbug” zealots and bitcoin zealots. The busywork mining has gotten absurdly more expensive than it was in 2011, and somehow is now a significant fraction of worldwide energy usage (what. the. heck), and various <a href="https://e360.yale.edu/digest/bitcoin-miners-resurrect-fossil-fuel-power-plant-drawing-backlash-from-environmentalists">blockchains’ environmental impact</a> is now the most common argument people use against them.</p> <p>Beyond my imagination, bitcoin has achieved the unlikely goal of being even <em>less</em> convenient than gold for actually buying things (the job of a currency). The exchange rate of bitcoin is almost completely a random walk, impossible for anyone to manage (unlike a regular currency), and much worse than even gold.</p> <ul> <li><strong>Even if it was a good idea, governments would squash it.</strong><br /> The only reason they haven’t is it’s too small to matter.</li> </ul> <p>Yes and yes.</p> <p>Congratulations, we’ve now seen the bitcoin movement get big enough to matter! There’s a corresponding increase in regulation, from SEC investigations, to outright banning in some countries, to the IRS wanting to tax you on it, to anti-terrorist financing and KYC rules. Each new regulation removes yet another supposed advantage of using something other than cash.</p> <p>Also, it’s now obvious that use of bitcoin (and related blockchains) for payments is almost entirely scams and illegal stuff. This agrees with my prediction, but in a way I didn’t expect. It turns out to be maybe tautological. Like the old American saying, “If you outlaw guns, then only outlaws will have guns,” you could argue that we have now regulated bitcoin so much that only criminals can productively use bitcoin.</p> <p>But it's grown enough to now be producing the largest (and ongoing!) ransomware outbreak the world has ever seen, so, you win, I guess.</p> <ul> <li><strong>The whole technological basis is flawed.</strong><br /> The weak link is not SHA256, it’s the rest of the cryptosystem.</li> </ul> <p>Yes, in multitudes.</p> <p>We’ve seen forked chains, theft, mysteriously reversed transactions, 51% attacks.</p> <p>It turned out bitcoin is irreconcilably privacy-destroying. (Law enforcement teams say thanks!) This was originally billed as a feature until some drug dealers got caught. The feature, or maybe bug, can’t be fixed without changing the system, which can’t be done without getting everyone to upgrade.</p> <p>But ha, it's a decentralized system. Since nobody could figure out how to get everyone to upgrade a decentralized system all at once, it was more profitable to instead spin up zillions of new blockchains, each with its own systemic flaws, but all sharing the one big systemic flaw: it’s an ownerless distributed cryptosystem, so when each fatal flaw is inevitably revealed, nobody can fix it.</p> <p>On top of the technical problems, there were social problems. Jo Freeman's <a href="https://www.jofreeman.com/joreen/tyranny.htm">Tyranny of Structurelessness</a> showed up here, as it does whenever you try to pretend you have no social control hierarchy. We learned that the people who write the code, and the people who have the biggest mining rigs, and the people who operate exchanges, and the people who go on <a href="https://boingboing.net/2018/12/05/blockchain-cruise.html">expensive and very shady cruises to hobnob with the cabal</a>, and <a href="https://www.bbc.com/news/technology-58719884">something about North Korea</a>, and basically everyone who is <em>not you</em>, all have disproportionate control over what happens with this “decentralized” “currency.” And this is equally true in all the other “decentralized” chains invented to either scam you or solve technical problems in the original, or both.</p> <p>For heaven's sake, people, it's software. You built a system, or series of systems, that will fail in completely predictable ways, forever, if you didn't get the software perfectly right the first time. What did you think would happen.</p> <ul> <li><strong>It doesn’t work offline.</strong><br /> Paper money does.</li> </ul> <p>Still true. On the other hand, the global expansion of cellular data availability has been relentless and perhaps this never did matter.</p> <p><strong>What I got wrong</strong></p> <p>Okay, well. The title.</p> <p>“Why bitcoin will fail” wasn’t right. It would have been better to call it “Why bitcoin <em>should</em> fail,” because it really should have! But it didn’t, at least not yet, at least in the minds of its ever-growing user base. I feel like this is important.</p> <p>A few years ago I learned the investor variant of <a href="https://en.wikipedia.org/wiki/Sturgeon%27s_law">Sturgeon’s Law</a>. Here’s what a VC told me: 90% of new things will fail. Therefore you can just predict every new thing will fail, and 90% of the time you’ll be right. That’s a pretty good way to feel good about yourself, but it’s not very useful. Anybody can do that. Instead, can you pick the 10% that will succeed?</p> <p>Even though I accurately predicted a bunch of things about bitcoin that wouldn’t work, I didn’t predict <em>all the other things about bitcoin that wouldn't work.</em> Maybe that seems like splitting hairs, but it isn’t. If you miss the reasons something won’t work, then you might just as easily miss the reasons why it will work. It suggests that you don’t know what you’re talking about.</p> <p>Here are some reasons I missed for why bitcoin (and blockchains generally) didn’t and still don't work, for anything productive:</p> <ul> <li> <p><strong>Scams.</strong> Lots and lots of scams. Blockchains became the center of gravity of almost all scams on the Internet. I don’t know what kind of achievement that is, exactly, but it’s sure something.</p> </li> <li> <p><strong>Citizens moving money out of authoritarian regimes.</strong> This is, by definition, illegal, but is it a net benefit to society? I don’t know. Maybe sometimes.</p> </li> <li> <p><strong>Other kinds of organized crime and trafficking.</strong> I don’t know what fraction of money laundering nowadays goes through blockchains. Maybe it’s still a small percentage. But it seems to be a growing percentage. </p> </li> <li> <p><strong>More and more blockchains.</strong> There are so many of them now (see “scams”, above), claiming to do all sorts of things. None of them do. But somehow even bitcoin is still alive, even though a whole ecosystem of derivative junk has sprouted trying to compete with it. </p> </li> <li> <p><strong>Corrupt or collapsed exchanges.</strong> I predicted technical problems, but most of the failures we’ve seen have been simple, old fashioned grifters and incompetents. Indeed, the failures of this new financial system are just like the historical failures of old financial systems, albeit with faster iterations. Some people are excited about how much faster we can make more expensive mistakes now. I'm not so sure. </p> </li> <li> <p><strong>Gambling and speculation.</strong> I wrote the whole article expecting bitcoin to fail at being a <em>currency</em>, but that charade ended almost immediately. What exists now is an expensive, power-hungry, distributed, online gambling system. The house still always wins, but it’s not totally clear who the house is, which is how the house likes it. Gambling has always been fundamentally a drain on society (a “tax on the uneducated,” someone once told me), but it’s always very popular anyway. Bitcoin is casino chips. Casino chips aren’t currency, but they don't “fail” either.</p> </li> </ul> <p>Despite all that - and I didn't even need to exaggerate! - bitcoin has still not failed, if failure means it’s gone. It's very much still around.</p> <p>That’s because I forgot one essential reason bitcoin has survived:</p> <p><strong>Because people really, really, really want it to.</strong></p> <p>If there’s one lesson I learned over and over in the last ten years, that’s it. Projects don’t survive merely because they are good ideas; many good ideas get cancelled or outcompeted. Ask anyone who works at an overfunded tech company.</p> <p>Similarly, movements don’t die just because they are, in every conceivable way, stupid. Projects live or die because of the energy people do or do not continue to put into them.</p> <p>…</p> <p>Here's a metaphor. Blockchains today are like… XML in the early 2000s. A long-burning overcomplicated trash fire that a bunch of large, cash-rich, unscrupulous, consultant-filled mega-organizations foisted on us for years and years, that we all now pretend we weren’t dumb enough to fall for. Remember SOAP? Remember when we tried to make HTML4 into XHTML? Ew, no.</p> <p>The thing is, a ton of major tech infrastructure spending was somehow justified in the name of XML. A lot of computer systems, especially in the financial and manufacturing industries, got a long-needed overhaul. Fixed-width COBOL databases couldn't do XML, so bam, gotta replace some fixed-width COBOL databases (or at least slap a new API translator in front). The XML part sucked and still sucks and we’ll be dealing with it for decades.</p> <p>But is that really so bad, in the name of progress?</p> <p><strong>Epilogue</strong></p> <p>It's been ten years, and it all went pretty badly, so let me make a new prediction.</p> <p>A lot of stuff will get redesigned in the name of blockchains. Like XML, the blockchains will always make it worse, but if carefully managed, maybe not <em>too much</em> worse. Something good will eventually come out of it, by pure random chance, because of all those massive rewrites. Blockchains will take credit for it, like XML took credit for it. And then we'll finally move on to the next thing.</p> <p>Nowadays if someone picks XML for a new design, we look at them like they’re from another planet. So, too, it will be with decentralized consensus blockchains, someday.</p> <p>So, too, it was for the <a href="https://en.wikipedia.org/wiki/Gold_standard">gold standard for international trade</a>.</p> <p>But it took more than 50 years.</p>