Monday, January 30, 2006

ldb take 2

in my last blog entry i mentioned working with ldb for kde configuration. there was, as expected, a good amount of feedback most of which was negative. i thought i'd take a moment to address a few of the issues raised.

first the good news: kconfig was designed to have multiple back-ends available. i'm adding ldb/tdb as a possible backend; an env var controls which to use for now (makes it easy for testing things =). i see no reason to get rid of INI style configs altogether: applications still just see KConfig objects (so it matters not to them), and for other subsystems such as the services resources and application menu entry objects it continues to make sense to distribute those as INI style files. add to that the issues of migration from kde3->kde4 and the braindamage of NFS file locking and it just becomes pretty obvious that keeping INI-style config files as a backend is a no-brainer. that much was obvious to me from the start.

so there is no need to hyperventilate over INI files going away .....

some were concerned about storing the data in a binary format. people often go running and screaming from binary formats, and for some good reasons: binary formats have been home to some of the worst implementations of these things ever (e.g. the windows registry), and binary formats are usually impervious to text editors.

personally, i've worked with unix(-y) systems for years and really like a lot of things in them; being able to edit things with a text editor is among those things. if ldb didn't provide facilities for that, i wouldn't be interested in it.

but i do i believe it's important to separate failure from design; just because a particular airplane design tends to crash a lot, it doesn't mean that fixed wing aircraft are a bad idea when it comes to flight. so pointing to the windows registry (or gconf, if that's your axe to grind) and howling is not the most adroit means of evaluation here IMHO.

if you wish to edit your data via a text editor, you can certainly do just that with ldb using ldbedit (which launches $EDITOR).

and unlike the windows registry with its alice-in-wonderland-like maze of obscure entries with mysterious values, i'm keeping the same naming conventions so we can see "kickerrc -> general -> AutoHideDelay: 0" and with the beauty of kconfigxt even be able to see what the zero value means, what the range of allowed values is, etc...

and of course you can also export the db to plain text. that's how ldbedit works, actually.

but what about stability? well, ldb/tdb is being used by samba4 which in turn is being (or will be) used in some pretty heavy duty installations. when it comes to single sign on, printer access and file server stability failure isn't exactly an option nor is slowness. i'd also note that INI-style files aren't impervious to corruption either (as i noted, i just had a bug report on kicker about this exact problem last week), though they can keep it compartmentalized due to separate files.

so, how about the speed? that's the issue i'm working on right now. after years of using the same $KDEHOME, i have 181289 entries in 14066 groups spread across 386 files. cp'ing the rc files takes ~0.045s. migrating those settings over takes ~105s on my machine of which all but ~6s is spent in lbd_add, which amounts ~142 groups/second. kconfig's INI backend does it ~18s, or ~781 groups/s.

i need to do some more testing with realistic usage patterns and ensure i'm not using the API wrong and therefore tripping it up ... i also need to test read with appropriate indexes more before i can comment on those things.

12 comments:

Peter said...

I appreciate that perhaps the knee-jerk reaction to your proposal seems a little silly, but I believe that there are good reasons for it above and beyond, "Eww, that looks like the Windows registry!"

One of the big advantages of the "many text files" config backend is that it's very, very robust. Even if you have a complete power failure at the end of a session when several config files are being written out, and you have to fsck your non-journalled filesystem and lose chunks of the config data, the chances are that it'll only affect a few applications. With all the config data stored in one big ldb file, it's possible that you'll be hunting around for your last backup just in order to be able to start KDE in a sane state.

Moving on to the text export: I'm sure that it's fairly straight forward to write a tool that exports gconf data to a text file too; in fact, I suspect that one already exists somewhere. However, the problem is that you either have to work out what the query you need to extract the bit of configuration you're looking for is, which could well be non-trivial, or you have to hunt through a file containing the configuration data for every KDE application.

On the other hand, it could well be a very useful feature for KDE to be able to use an LDAP server on a remote system for its configuration storage.

Maybe the answer to this particular argument would be to implement the new backend, but include an painless way to switch from ldb to plain text files (or other backends) and back again (and by painless I don't mean a DCOP or DBUS command line with 20 options).

Janne said...

OK, I'm no Einstein, so bear with me :).

Basically, ldb would be a configuration-system for KDE. It's currently used by Samba4. It's LDAP-like, which means (among other things) that it could offer centralized configs for numerous workstations.

Now, could such system be used as a mixed central/distributed system? I mean, some settings would be handled by the central ldb-server, while some would be handled by the local ldb? What I have in mind is that the admin could use ldb://server-name to change the settings on the central server, and those changes would then be reflected on workstations as well. Or, the admin could have a case where a used calls him and tells that his workstation is doing some weird things. The admin could then type ldb://workstation-name, and check the local settings on that machine, and change them as needed. He could also do comparisons to some reference-settings and see what changes have been made.

If the user nukes his HD, the admin could re-image a new machine for the user, along with his files AND his settings, since the global and local settings were automatically backed up.

In short: the admin could change global and local settings of all machines in the network from his own workstation. The users would still have their personal settings (that were created over the course of dayes/weeks/months/years) in their new machine, even if their old machine got nuked.

In addition, there could be different settings-groups for different "class" of users (for example, "engineers", "secretaries" and so forth). Moving users between those groups would also change their settigns accordingly (this could be overridden if needed). ldb would also tell which apps would be on the workstation. So if someone moves from "secretaries" to "engineers" (for the sake of example) he would notice that he now has bunch of developement-tools on his machine, besides the regural office-suite. New users would automatically receive his workstation, as configured by the ldb-server. No manual tweaking needed.

If this system would offer ANYTHING like that, I want it. And I want it since yesterday. if I would have something like that today, I would be a bit more sane than I am, and I would have a bunch of happy users :).

Or, it could be that my mental-image of this system is completely wrong, and in that case you are free to disregard it.

Anonymous said...

Regarding elektra (issue started at previous blog comment)... It's not the truth anymore that you have to use single file for one value. You can use now different backends: for example Berkeley DB or ini files if you want.
Elektra is simple, elegant and light solution. It should be also possible with it to get notification about configuration changes. They think also about configuration options stacking what would be really great!

Philip Rodrigues said...

AIUI, KConfig can use different backends (INI files, ldb, etc), it's just that no one really got round to writing one. So, if you'd like to see elektra as an option for KDE, take a look at the KConfig API docs and get coding :-)

Troy Unrau said...

I fully support your endeavour to create a modern configuration system, as long as it's still easy to edit from the commandline (which ldbedit seems to do reasonably).

Additionally, some apps need an overhaul on where they store their information as well, since they are sometimes stored in non-obvious locations.

Anyway, regarding the backup problem, one can simply have a kde process that makes a copy of the single config file every few days, giving a very nice rollback/recovery feature with very little overhead. This I would appreciate.

Janne said...

Yes, you could make backups of the config-files. But I feel that having a central-repository of the settings in LDAP (or something similar)-server would just be more elegant solution. I think it would make central management of the backups and settings that much more easier.

peter said...

I respectfully suggest look at Elektra again -- its inventor got a real pummelling because people thought his project was like Windows registry, just as you are now being unfairly criticized.

Here are three key facts worth repeating... its config is available at boot time _before_ the filesystem is loaded, uses simple text/xml editor _or_ C API, backwardly compatible with most ini/conf files.

Finally it has a healthy project going, is well documented and good to go. Give Elektra a fair shake!

Aaron J. Seigo said...

elektra is a non-starter because it offers none of what i really want (network capabilities, e.g. ldap integration) and lots of what i don't (a bazillion files to stat)

ldb could just as easily become a "universal storage mechanism" and help solve that part of things.

> that it's very, very robust

and very very slow, esp on cold boot.

> With all the config data stored
> in one big ldb file, it's
> possible that you'll be hunting
> around for your last backup

dubious. with transactions that fsync() it's pretty damn safe

> the problem is that you either
> have to work out what the query
> you need to extract the bit of
> configuration you're looking for
> is

it's pretty much the same as the text file hierarchy: kde -> appname -> groupname. if you just do "kde" you'll get everything; if you do "cn=appname,cn=kde" you get just that apps configs. and that's with the generic ldbsearch. hacking up something specific to kde would be fast.

and remember, it's not just "export to text" but also "load the text editor, and update the database on exit"

> include an painless way to
> switch from ldb to plain text
> files

i've already written said app. it's very trivial, and sports both a GUI and console-only mode.

Aaron J. Seigo said...

@janne:
> Now, could such system be used
> as a mixed central/distributed
> system?

you've just hit exactly why i'm doing this. nice one =)

Aaron J. Seigo said...

@anonymous, regarding elektra:

> You can use now different
> backends: for example Berkeley
> DB or ini files if you want.
> Elektra is simple, elegant and
> light solution. It should be
> also possible with it to get
> notification about configuration
> changes. They think also about
> configuration options stacking
> what would be really great!

so with a few added features (e.g. options stacking) we'd have exactly what we have now with kconfig but with a few extra back ends? sorry, but this isn't very inspiring.

i don't need a new way to store config locally, i want a way to easily create distributed configs and collapse system config bundles into single files (for the obvious stat-mania reasons)

does elektra give me that?

one of the nice things with LDB is that if we decide on a common place to put configs, there could be one system-wide desktop.tdb file and in there top levels such as:

cn=xdg
cn=kde
cn=gnome

so we don't lose much / anything there either. (in fact, i've already started putting everything under a cn=kde =)

Jamie McCracken said...

one of the nice things with LDB is that if we decide on a common place to put configs, there could be one system-wide desktop.tdb file and in there top levels such as:

cn=xdg
cn=kde
cn=gnome


okay that looks nice!

On to more practical measures:

If we were to go down the route of using ldb/tdb as a freedesktop thing then we would need a few issues clearing up:

1) Can you confirm if ldb/tdb are fully threadsafe (perhaps you can ask tridge this?)

2) Gnome and some others would want a daemon for notifications and obviously if we are using LDAP as an ldb backend then we definitely dont want 20+ individual LDAP connections which would be insane when a daemon with one or two LDAP connections would be much faster on startup (authenticating each client would be a big bottleneck as well as consuming excessive resources on the LDAP server).

Seriously I would be interested in assiting any freedesktop adoption providing (1) above turns out positive. A common config system even if its just a common backend like ldb would be a lot better than nothing.

Aaron J. Seigo said...

> Can you confirm if ldb/tdb are
> fully threadsafe

tridge emailed me about this today actually; they haven't put mutexes in ldb/tdb at this point because they don't need them, but he noted that if we needed it to be thread safe they'd do so.

so the answer is "not right now, but it will be soon."

> Gnome and some others would want
> a daemon for notifications and
> obviously if we are using LDAP
> as an ldb backend then we
> definitely dont want 20+
> individual LDAP connections

that's one of the things i really like about ldb: you don't need to use an ldap server. an ldap server is, simply, just too slow, settings daemon or not.

with ldb, once replication is in, you can store your settings centrally in an ldap database (or ldb/tdb store, for that matter) and sync them out over the network.

even right now you could store the configs in an ldap database and push them out as ldif's to ldb.

so a gconf ldb backend wouldn't require an ldap server and shouldn't incur extra overhead for them. =)

and since it's multi-proc safe, both gconf-daemon and kconfig could be accessing the same tdb file via ldb and life would be merry.