Real-life Buffer Underruns

Okay, this is about a week old but I'm so impressed by it that I'll post it anyway. While I was visiting my family last week, I finally learned how water pressure tanks work.

Background info: my family lives outside of town, and gets their water from a well. Wells are in the ground, and typically the water would rather be in the ground than come up to your house and shoot out of your taps. In a city, they make the water come out by pressurizing the water at the Central Water Pressurizing Authority (or whatever), but out in the country, you're left to your own devices.

What you can do, and some people do this but it's bad, is to just turn on your pump whenever you turn on the tap. The pump works like an airplane propeller, sucking the water from the ground (or a holding tank, or whatever) and pushing it through your pipes into your tap. The problem: this means your pump has to turn on and off an awful lot, and response to water requests is rather sluggish.

The alternative is a pressure tank, in which you pump water into the tank, increasing the pressure so that the water wants to get out. The more water you add, the more it wants to get out. Now here's the catch: why does adding water make it want to get out more? What is "water pressure" exactly? Since liquids can't be compressed, how can adding more liquid increase the pressure?

My dad finally told me the answer to this last week. (It turns out he's always known this, but I never asked. Go figure.) The answer is: the pressure tank is actually a sealed tank full of air, and the water goes into an expandable baggie inside that. An empty tank has some amount of air pressure (with one valve for adding/removing air) and an empty baggie attached to the in/out water pipes.

Adding water to the tank decreases the volume of air, but not the amount of air. Thus, because of everyone's favourite chemistry formula, PV=nRT, the air pressure increases. The air wants to expand and push the water out of the tank, and the more water you add, the more the air pressure increases - and thus, the more the water pressure increases. The "water pressure" that you can measure on the water outflow pipe is exactly equal to the air pressure in the tank.

Why do we care? Well, this means that in fact most water pressure tanks are more than 50% empty even when they're "full." They're mostly air. And if you tune them wrong, it's even worse. So don't expect that your 65 gallon water pressure tank will ever have 65 gallons of water in it, or you'll seriously confuse yourself.

And now we get into my area of interest, namely networking. What's the point of the tank again? To make it so the pump doesn't have to instantly respond to changes in water requests. Also to make it so that, even if the well runs dry temporarily, there'll still be water available to service our requests. But standard tank tuning algorithms (which my dad also knew, because he read the instructions for his tank) involve setting up the tank to activate the pump when the tank is almost empty and deactivate the pump when the tank is almost full (where "full" is using our new definition, meaning still more than half empty). In programming, we call these two points the "low water mark" (LWM) and the "high water mark" (HWM), respectively (snicker).

Problem is, if the well runs dry while you're nearing the low water mark, you have almost nothing left in your buffer, er, tank, to service the requests. So, for maximum recoverability, you want the low water mark to be as high as possible. But this decreases the "block size" of demands on the service provider, er, pump. Where with the tank almost empty you could say "send me 30 gallons of water", now you have to say "send me 2 gallons of water" much more often. In the typical heavily optimized case, your tank doesn't buy you anything because the LWM=HWM, just like with no tank at all. You'll only notice the improvement if the service disappears altogether for a while.

Of course, you're also receiving water that's 30 gallons old - your tank has increased the staleness of the data, er, water, that you receive.

Okay, the other problem, and this is the point of the story: the reason my dad bought the tank in the first place. The well was running dry pretty often this winter, since it was a pretty dry year. To improve the situation, my dad thought it would make sense to buy a larger tank (buffer), so that if there was a temporary high demand for water followed by a long period of low demand, the tank could fill up and not stress the well too much. Now, this didn't really work, mostly because of the HWM/LWM tuning problem above, and partly because no buffer will save you if the long-term average bandwidth of the network is less than average demand. (Water caching is gross.)

In fact, the larger tank made things worse once spring came and the water was abundant again: because the transaction size (HWM minus LWM) was much larger than with the old tank, it would run the pump for longer in one shot. Unfortunately, HWM-LWM for the tank is more water than would fit in the entire well, so you're guaranteed to underrun your buffer every time you go to the network for more service, thus causing TCP to start limiting your bandwidth and... oh wait, I'm mixing my metaphors again. Anyway, the tank made it appear that there still wasn't enough bandwidth, even when the congestion went away.

The Moral of the Story

Adding buffers doesn't fix anything unless you tune them properly. And they sometimes make your system unusable.