Death by Abstraction

I put in some work this weekend trying to make UniConf maximally palatable to the KDE project. Since UniConf does pretty much everything that can be done, or else can rather easily be extended to do so, you might think this would be an easy job. Not exactly, because of one tricky bit: performance.

UniConf uses a rather neat abstraction, introduced to UniConf by Jeff Brown a couple of years ago, called a "generator." It's one minor step up from a filesystem in a Unix VFS. Because of the slightly more/less rigid (depending how you look at it) rules a generator has to follow, UniConf makes it easier to stack layers of generators together to achieve what you want, and so people do. For example, there's an ini-file generator, a tcp client-server generator, a D-BUS cilent-server generator, at least two kinds of generic caching generators, a list-of-generators generator, and so on. As of yesterday, there's also an automount generator (take that, autofs!) and a filesystem one-file-per-key generator, the two of which work together (with the ini-file generator) in a slightly complicated way to handle reading/writing your ~/.kde/share/config directory.

The power of the UniConf abstraction is that you can easily rearrange these individual parts to do something similar but different: for example, if you wrote an xml-file generator, you could combine it with the same one-file-per-key and automounter generators to produce something that can read/write your ~/.gconf directory instead. That's a lot less work than writing a tree-of-xml-files configuration system from scratch, and you benefit from other people having debugged the other generators you're using.

But in a system this general, performance can be a problem. In my initial simple-minded experiment, switching to UniConf took the start time of "kwrite" from 1.0s to 25s. Okay, so that was a bit extreme, and with minor tweaking I got it back down to 3.0s. But still: that's 2/3 of kwrite's startup time just fiddling with config values. Obviously not okay.

Now, what's amazing about Jeff's abstraction is that it's so complete - the slowness is really not the fault of the abstraction layer at all, because the way it works, the front end matches up almost exactly with the backend; that is, the API you call is pretty much the same as the API the generators each implement. So you could write a very fast generator if you wanted - optimally fast, because you can throw away all the layering and just implement the API calls directly. It's just that, because it's so easy to combine pre-existing modules instead of writing something from scratch, most UniConf setups end up being pretty badly non-optimal.

Here's the real killer example: profiling my UniConf-enabled kwrite shows that much of the time is actually spent converting UniConf keys to and from KDE QEntryMap objects. (KConfig keeps a QMap of all the groups/entries in the tree.) To make it "optimally" fast, I should really store all my config entries directly in a QMap, rather than storing them in a UniConf-style data structure and then copying them to a QMap. But there's clearly no way to do that outside of rewriting the pre-existing UniConf generators.

Since writing a completely special "KDE generator" wouldn't be "unified" enough for me, I'm going to instead try to optimize the other generators more until the performance is acceptable. The profiler shows some pretty obvious starting points. Flexible and fast - not the easy approach, but maybe the most fun. Wish me luck.