and "balanced"
Everything here is my opinion. I do not speak for your employer.
April 2013
June 2013

2013-05-03 »

One of the scariest yet most informative periods of my life was a brief stint doing some digital control systems for mechanical devices.  In university we learned about this stuff as Control Theory, PID controllers, etc.

Approximately the first thing you learn about control theory is the concept of "overshoot."  Say you have a robot, and it's too far to the left, so you power up the motor full blast to try to move it to the right.  Then, once it gets there, you turn off the motor.  That's too late; you'll probably end up flying past where you wanted to be, so you have to go back the other way, probably fly past again, etc.  That's the diagram on the right.

If instead, you run the motor a little slower - or, as they soon taught us, run it "proportionally" to the distance from the target (that's the "P" in "PID controller"), you can adjust it to avoid overshoot, getting a result more like the one on the left.  In most cases, the diagram on the left is better because it's less crazy, even though you don't get to the destination right away.

Now let's relate that back to software engineering.  Let's say you've got a release pipeline: dev->staging->dogfood->canary->prod.  For simplicity, let's assume you're doing it like Chrome, and each phase is running a slightly older version of the software than the previous, because versions progress through the phases linearly (probably getting patched along the way).

Now let's say that this week, you've got a set of new cool stuff in dev that you want to get to customers (prod) as soon as possible.

The default option is to keep running the pipeline as is.  That gives you something like the diagram on the left: you don't get to the goal line (the dotted line in the picture) very fast, but along the way everything is sane and predictable and once you finally get there, you're there.

An alternative is to make an exception and disrupt your pipeline.  Take the version from dev, and ram it through the pipeline as fast as you can, fixing bugs as fast as you can along the way.  This will get you to the goal - released features - much faster.  But it will probably look a bit like the diagram on the right; you won't exactly hit your goal, and your release process might start making weird patterns.  Maybe you have to revert to a previous one.  Once you've reverted, maybe you'll roll forward again, but you're even more desperate this time, so maybe you overshoot one more time.  And so on.

Once you're into this overshoot/undershoot pattern, it's actually hard to get back out.  And moreover, if you've overshot really badly, then now you're probably really desperate, and you might overshoot a second time.  Sometimes it can take longer to finally settle on the goal with this overshoot pattern than in a slow-stabilization pattern.  By then, you might have a new input (ie. new crazy-important features) before you even finish bouncing from the last one.  (In the diagram they both stabilize after about the same delay, but either graph could actually be longer or shorter than the other depending on the amount of over/under compensation you use.)

For a software release manager, the P knob in the PID controller is the ratio between number of changes and length of time in your release cycle, aka the "time pressure" on your release.

Another thing I learned in my control systems classes was: there are two ways to build a control system.  Many traditional PID controllers are "designed" by calculating some basic ratios to start with, and then twiddling knobs until  the results come out pretty decently, then locking in those settings.  That's physical-world engineering for you.  It sounds silly, but it works.

The other way is to do some really complicated math and come up with the theoretically perfect compensator (which is probably not a simple PID controller).  That's either fantasy or literal rocket science, depending whether you're good at it or not.  Most of us are not. :)

You fail the class if you just turn the P knob all the way up to max because you're in a hurry to get there as fast as possible.

2013-05-07 »

"Specifically, the Bernstein survey found that a whopping 98 percent of Kansas City residents in the fiberhoods were aware of Google Fiber. Slightly more than half of the respondents, or 52 percent, said they would 'definitely or probably buy' the service, while another 25 percent said they might purchase it. Only 19 percent said they would definitely or probably not buy Google Fiber."


2013-05-08 »

Overheard in a discussion of wifi interference from walls:

"It's pretty baffling."

No pun intended, I'm sure.

2013-05-09 »

Things I hate:

When you load a text/plain into a web browser, press ctrl-s to save it, and it decides to re-request the data from the server because it's supposedly uncacheable.  I don't care if it's uncacheable.  It's in my browser frame.  I can see it.  I can scroll up and down through it.  I can cut and paste it.  Why are you reloading it?

2013-05-13 »

Newsflash: Recent ops emails about "bouncing PONs" not as exciting as expected.

2013-05-17 »

The notification dropdown drives user engagement.

That is, it doesn't work, so I have to open the app and scroll around to find out what actually happened.

2013-05-20 »

Although this is characterized as "hard nosed negotiation," I think it's important to note that he didn't really have to do anything underhanded, and he wasn't really taking anything from anyone.  The reason Steve Jobs "won" that negotiation is that he had a well thought out plan that was good for everyone, and he was able to explain that to the people he was negotiating with.


The only really "hard nosed" point is that when News Corp asked him to (essentially) give them more money, he said no, the deal is already good enough.  It's so good you don't even want to work with anybody else, so why are we having this conversation?

That's not reality distortion, because it was true.  There really was nobody else at the time, except Amazon, and they were already working with Amazon, and this deal only asked for the rights to sell books at an even higher price.

When companies suck at negotiating, it's because they fail to create the kind of situation where the other people need what we're offering.  It doesn't matter how good a negotiator you are, smart people will see through a bad offer.

2013-05-21 »

"it’s been called a Segway for your face"

2013-05-22 »

For next time someone tells you you need to integrate with your company's version of something instead of the Popular version.

2. Avoid Locality Bias in Product Development
You're part of Google, there's corporate pressure (and perceived quick wins) to work with other Google products. Blogger, Orkut, OpenSocial, Google Video, Picasa. Remember those? We were growing so quickly that if YouTube integrated or heavily promoted them, they could probably hit their quarterly growth target just from our marketing efforts. But guess what, in many cases that wasn't where our growth was coming from. It was coming from Facebook, Twitter, Buzzfeed, Tumblr and the curatorial blogosphere. So that's where we focused. Be where our users are and grow on the back of those ecosystems. Were some of them Google competitors? Heck, some of them were YOUTUBE competitors but overall the goal was to sew ourselves into the fabric of the web. Boy did I take arrows in the back (I still remember Jeff Huber saving me from a few of them) but at the time, it was the right choice. YouTube's partnership with Apple to be a default app on the original iPhone not only helped us make the jump to mobile but it ensured that every other carrier wanted YouTube on their phones too.


2013-05-23 »

Goodbye Sprint Nexus S, hello iPhone 4S 64GB (ie. the last, best one that doesn't have a stupid incompatible new connector).

Some first impressions:
- Hey, cool, the keyboard can keep up with my typing!
- Animations aren't jerky!
- Everything resynced perfectly from my old iPod touch, just by plugging in the USB cable to iTunes for a while.  Including wifi passwords.
- It fits all my music!  And no built-in products are trying to convince me to "stream" music while on the subway!
- Whatever its flaws, Apple Maps shows more street labels on the same zoom level in downtown NYC than Google Maps does, which is what I wanted.  Its UI is also clutter-free and doesn't try to guilt me into logging in.
- You might think the new iOS6 "compass" app is dumb, but it is exactly what I frequently want when getting off the subway in New York.  I don't want a freaking rotating map, I want to know which direction I'm walking. It's super awesome that they added this.
- I have a gnubby now so I won't miss the bluetooth OTP.  (This was the main reason I didn't switch sooner.)
- The Settings app contains the settings for all your programs, all in one place.
- The Settings app has really well thought-out "Notifications" and "Privacy" pages that reflect across all apps.
- Google Voice integration is, er, nonexistent. :(  (For bonus points: voice.google.com refuses to let you add a phone number from a mobile web browser! At least on Android you can "Request Desktop Version" and fake it.)
- iOS "Messages" is what other IM apps wish they were (including the vendor lock-in)

I need this phone to last me 3 years.  That's the runway I want to give whoever is next in line to make a phone that I don't hate.  Unfortunately, I have little faith in present-day Apple to do so.  I'm worried that the uninspiring iPhone 5 and the totally unnecessary new Macbook power connectors are only the beginning of the downward slide.

2013-05-25 »

In reflection of e-mail retention policies, I bring you this quote I kept from a long-ago Wired article:

"Thinking is based on selection and weeding out; remembering everything is strangely similar to forgetting everything.  Maybe most things that people do shouldn't be remembered. Maybe forgetting is good."  - Rob Jellinghaus, of the Xanadu Project


(Ironically, this quote would not have been retained by most email retention policies. :))

2013-05-26 »

return Integer.valueOf(o2.size()).compareTo(o1.size());

2013-05-27 »

Nice!  Now people who write articles for the ACM can not only not get paid, but they have the option to pay the ACM so they can keep the copyright on their own work.

Well, that sounds like exactly how things should work.

If Monty Python wrote licensing agreements.

2013-05-28 »

"I have to tell you, it's really easy to build in Kansas," Medin said. "It's really easy to build in Missouri. And it's easy to build in Texas. Actions have consequences."


2013-05-29 »

It took my entire life so far.  But we can mark this year as the year it finally became useful that tar archives were designed to be read linearly from spools of magnetic tape.

That is, assuming you can model the internet as a giant spool of magnetic tape.

2013-05-30 »

Welcome to Halfassville.  Population: everyone who makes bootloaders.

2013-05-31 »

No H1B for me this year :(

This could be serious.  If I don't get it sorted out sometime in the next 6-8 years, I could be forced to go back home a few hours north where we have sensible healthcare and better festivals, and work from there instead.  You can imagine my concern.

April 2013
June 2013

I'm CEO at Tailscale, where we make network problems disappear.

Why would you follow me on twitter? Use RSS.

apenwarr on gmail.com