Project Unity:

I have to admit, I've been thinking about something like this for a long time, but I didn't quite know how to do it. I think that, with the help of UniConf, I've finally figured it out.

Motivation:

Hugely annoying problem with Unix that prevents it from being popular: configuring it is insane. There are options, options everywhere, and no two config files are 100% syntax-compatible. Some config files are actually scripts. Some config files are optional. Some config files inherit from other ones. Some are in the user's home directory, while others are in /etc. For the most part, the FilesystemHierarchyStandard? has at least made it so no more config files are in /var or /usr anymore. But it's a big mess, and because of the mess, it's really hard to write a comprehensive GUI for the configuration unless you cheat massively like we did in Weaver?.

Extremely cool useful Unix feature for experts: config files are freeform text, there are tons of options you can tweak, and you're not restricted to a particular file format that might not be easily able to represent what you want. Some config files are actually scripts, which you can tweak to dynamically generate config information when you need it.

Unfortunately, the above annoying problem and the above most useful feature happen to be the same thing, which sort of prevents us from making much progress.

Oh, while we're here, there are some things that it seems no system does, but I'd like anyway: defaults for all apps configurable by the admin but overridable by the user. Network-wide per-user settings (I want my bookmarks to be the same wherever I login!). Default settings for various applications inherited from global "useful random information" like email domain, hostname, local IP, etc.

Now, for the very restricted environment of Weaver?, UniConf and LifeCluster? will solve all our problems, since we don't need all that much end-user config flexibility anyway (Weaver? just always does the "right" thing without asking). But what about ExpressionDesktop? and Twister?? These are full Unix systems where people can install all kinds of apps that we weren't expecting - we can't run the whole as a dictatorship, like we do with Weaver?. We need to support, well, total anarchy. And no, I don't mean chaos :) But we want to have a sort of organized mess that's not too much work to administer. In other words, a cross between Unix and Windows: a mess (Unix), but not too hard to administer (Windows).

Quick requirements:

  • Keep compatibility with old apps and allow partial conversion. We're certainly not going to ever convert everything to the new scheme.

  • Keep the flexibility allowed by Unix-style "scripty" and auto-generated config files.

  • Where possible, let apps retain compatibility with their old-style config file formats.

  • I think it's okay to require application code to be modified to support the new system, but if so, the API has to be supremely simple and the library we link to has to be really small, really portable, compatible with nearly every language (including C, C++, and shell), and should have an optional "just use a stupid name=value text file" dumb mode. Basically, apps still should work even if the end-user doesn't want to change anything.

Basic design:

  • Have a system-wide UniConfDaemon using UniConf to store all config values for the whole system. Make sure we allow UniConfPlugins for reading/writing specially-formatted config files. done

  • Have a special UniConfPlugin for executing scripts to get config objects (UniShellGen?) which should make sure the config stays nice and flexible (just like the current, non-unified, system).

  • Write a hyper-stupid client library using Unix sockets and a simple binary protocol that can do basic UniConf operations using the UniConfDaemon: get, set, pre-get (pre-cache a value for later get), notify (tell me when an object changes). Put all the brains (such as caching and asynchronous stuff) in the UniConfDaemon so we don't have to relink apps in case it changes. done

  • Write an even stupider client library that has the same API but doesn't use the UniConfDaemon, in case people don't want to run a UniConfDaemon. In this case, it should just read name=value pairs from a text file, or something. This library should be so small that it can be included in the source for every uniconf-enabled app, in case the rest of the stuff (like UniConfDaemon) isn't installed.

Initial targets for conversion:

  • We can modify KDE's KConf and Gnome's GConf to handle the uniconf-library. Apparently they both support special "config backends" that should help us. Plus, we can probably then get rid of their horrible desktop-specific config daemons.

  • Do WinE? too, mainly because its config file code sucks so badly and I hate it.

  • Servers could also benefit: Apache, MySQL?, OpenLdap? (heh heh), Samba, NetAtalk?, all DanBernstein? tools, all WvStreams programs (of course), apt/dpkg and anything using DebConf?, etc.

  • Window managers, web browsers, etc.

For extra motivation, see also:

WvConfScripts? (self-generating config files!), OtherUsesForWvHConf?, WvHConfAllowsLooselyCoupledComponents?

-- apenwarr

dgtaylor? (2002/09/18): Well, a configuration system like this would certainly be welcomed by many many people... (maybe I could get my parents to finally stop using WinDoh?'s)...

Random ponderings though:

  • For system wide configurations, if this is in a single file, won't it get quite... huge depending on how many applications there are installed which make use of this system... Would it be of benefit to split the config settings into directories maybe? I.E. system/(appname).cfg, users/(uname)/(appname).cfg

  • Would the GUI for modifying individual config files be supplied by a plugin / view?

  • Other idea from curiosity: If a program didn't want to change it's style of config file, etc, would we make it possible for them to write a plugin to ProjectUnity which would open up a more specialized editor for their config files?

    • pphaneuf (2002/09/19): No GUI, at least at first. ProjectUnity is mostly an API, think "Unix VFS". There should be a "default" file format, but you should be able to "mount" other config files (or not quite config "files", think LDAP) on the tree. So if you have a program that doesn't want to change its config file format, it should provide a "config backend" for its chosen file format (I'm quite partial to libPropList? myself) and mount its config file somewhere in the tree using that backend.

      Once the config file is mounted, it can be changed using the ProjectUnity API, so if you have a GUI configuration editor, there's no need for a specialized editor, the API abstracts it all away from you.

      The "mounting config files" feature is also how you can get away from a single huge config file.

    • jbrown (2002/09/24): Beware abstractions that hide too much detail. We don't want to give end users a new Regedit tool. We will need to think about how to present configuration elements in more useful ways to users than that. Dan has a point here, because there would be little point in all of the complexity if ProjectUnity were only of use to automated tools and such. We must look into how to expose high level configurable entities in some user friendly way. This is precisely the same problem faced by the Gnome and KDE desktop teams. Having an abstract back end is all nice and fine, but we also need a front end.

dgtaylor? (2002/09/24): Ok, the following is a general attempt to lay out a plan here, but please correct my thinking / add anything here ![to begin with, I'm looking at only unix sockets].

  • a daemon process is created, binding itself to some unix socket ![any preferences as to what the filename for the WvUnixAddr? is?] done ![just need to know preferred file name if any exist]

  • Reads in initial config settings ![for now, one giant config file with all settings... later a file which indicates where individual settings are stored for various applications, etc.] Hmm... this should probably be done before we start listening for requests though... done ![once we have an actual config file instead of using tests/test.ini this will be much better]

  • once we're listening on the unix socket, a request will come in, for arguments sake in the format: get ![setting key]. done

  • the response from the get will then be written back out to the stream it came in on. Also in the "get" code, will need to register the request(er)(or)? to receive notifications in case the setting changes. Suggestions? Just need to do notifications now.

  • to do a set, the request would likely come in something along the lines of: SETK [key] [value], which would mark the key as being dirty and trigger the callback and notify any app which had requested the key or children below the key... done

Does that sound about right?

jbrown (2002/09/24): I think a somewhat more general hierarchical solution will be better. Taking a page from Microsoft's wildly successful registry (the API and implementation might suck, but the ideas are sound), break the settings down first by scope, then by application. ie. Some analogue to HKLM and HKLU possibly with a shortcut for the current user. Since we'll want to manage information about hundreds of users, applications, and other things using one common API, it pays to be a bit more general. Since we'll want to administrate multiple boxes at once, we want to set things up so that all of the machines can be mounted at once without naming clashes. Also we should be using the Unix hierarchical pathname conventions because of their inherent goodness. Bonus: for initial testing you can just map the requests down to the file system.

So try get /machines/mydesktop/system/...\n for system settings, get /machines/mydesktop/app/...\n for local application settings, get /users/myusername/app/...\n for user application settings, get /currentuser/...\n as a shortcut for current settings, get /currentmachine/...\n as a shortcut for local machine settings. Then set somekey=<typecode><length><binary value>\n. The \n here can help with error checking. Heck, it would probably even be better if we presented the length field with the key name too, so that keys can contain arbitrary characters (some back-ends might want keys containing =) without having to resort to escaping them. Dunno really...

Keep in mind we need to be able to pass around binary values too (for compatibility if nothing else). To keep things simple, values should be tagged with a type code and length and transferred in a well-defined binary network order. The server doesn't need to know anything at all about the contents of these values. Let the client-side API provide a nice interface that hides the transport protocol. As for the types, we should have Unicode strings (could just be UTF8), integral values (32bit), and byte arrays, as a bare minimum. The keys should also be Unicode (UTF8 will do here as well). I say Unicode because that offers us the most flexibility at little cost.

You should be able to write a simple proof-of-concept daemon in no time if you just dump the values straight out to the file system as ordinary files, and use symlinks to make any useful shortcuts.

Just my two cent's worth. Feel free to whack me on the head and call me a pompous ass.

apenwarr (2002/09/25): I Like Text. Thus, I suggest encoding the protocol using TclStyleLists (which can encode anything including nul bytes, although my current implementation can't do nuls :)). Also, everybody seems to just love integers and byte arrays: screw them. Everything is a string. If it works for 99% of Unix, it'll work fine for us. The easiest-to-manage number encoding nowadays is ASCII (with bonus points for the fact that it has no maximum or minimum value or floating-point precision).

Other than that, yes, hierarchy is the whole point of UniConf, so we don't want to flatten it for no reason. ProjectUnity should be very careful about defining exactly how we arrange the hierarchy, though; what you want should be doable, and more flexibly, using UniConfInheritance? instead.

dgtaylor? (2002/09/26): When should the daemon save the config info if it's been modified? On close yes.. but I'd also think it should be on a timed interval too... unless we save every time a key is changed... which is probably too expensive to do. Any idea what interval? or just after a key change?

  • apenwarr (2002/09/30): every few minutes, or on a polite shutdown of the daemon. We can make the exact time delay configurable, of course.

dgtaylor? (2002/10/02): Avery, I know you explained this to me originally, but it has become a tad convoluted in my mind. How are notifications supposed to work? Is the recipient supposed to register to recieve a notification, or if the app asks for a key, is it supposed to be notified if that key ever changes?

  • pphaneuf (2002/10/03): You have to register to get notification.

    • dgtaylor? (2002/10/03): Actually Avery answered this in person.. the notification is just a forget message to the client, so you are registered when you do a get.

      • pphaneuf (2002/10/03): Ah, so that we can cache stuff, and registration for notification is thus done on the client side (when you get the "forget" message, you can get the updated key and notify the registered list). Nice!

        • dgtaylor? (2002/10/03): Isn't it though? Although I don't know if I like having a forgetful application ;)

          jbrown (2002/10/04): Still not convinced this is a good idea... One client reading a million keys (think doing a Find in regedit) will cause the daemon to have to maintain a list of all of those keys to send out later notifications... The client should at least be able to tell the daemon that it would be wasting its time.

          • dgtaylor? (20002/10/04): Jeff, at a later point, the idea is if an application is getting a lot of keys under a certain tree, we'll just notify them if anything under that tree has changed.

  • pphaneuf (2002/10/04): Maybe a separate "GETCACHE" command? The normal GET command wouldn't add the key to the list of notification, and the GETCACHE would. If the client caches a key he got using GET, he's silly and should be shot.

dgtaylor? (2002/10/04): Ok, since this has been checked in now, here's a glimpse into the daemon's inner workings:

  1. The daemon starts, mounting the initial config file "uniconf.ini" into the / directory of the configuration tree.

  2. The daemon opens a WvUnixSocket? connection to /tmp/uniconf/uniconfsocket

  3. The daemon opens a WvTCPListener? to localhost, port 4111

  4. Now we just sit pretty and wait for incoming transactions.

The daemon isn't quite perfect... for instance the following transaction blows up miserably and I've yet to figure out the reason (I know it's in wvtcl_getword.. just not sure how to fix it): 

{get /chickens/bob get {wacky
test
section/  goose  }
set /chickens/nob narf get /chickens/bob}

  • apenwarr (2002/10/15): Err, why is that even supposed to work? Do we have some misplaced newlines somewhere? (everywhere?)

    • dgtaylor? (2002/10/15): No.. but at the same time it shouldn't crash / lockup the daemon. :P which it doesn't anymore.

Also, currently the daemon expects to load all keys from uniconf.ini, but this will be easy to change in the future using internal commands, and I have a rough idea of how to do this easily, which I'm sketching out in my little black book.

Comments / suggestions / critiques / foolishness?

}:D>

dgtaylor? (2002/10/15): Problem with unregistering callbacks after sending the FGET message.

The Client sends: set /chickens/bob foo

The Daemon receives and performs the set.

The Daemon now proceeds to go through everyone that performed a get on /chickens/bob, sending them the FGET /chickens/bob message.

Now, unless I'm mistaken, would it not be wise to delete the notifications for the clients we just sent the FGET message? Otherwise anytime anyone does a set after this, we will get redundant FGET messages...

  • apenwarr (2002/10/15): Can't you just do a UniConfEvents?::del() for the appropriate callback, userdata, and key? Presumably you know what those three values are, since you previously did an add() call with the exact same values.

    • dgtaylor? (2002/10/16): Here's the problem with the del. I cannot do the event.del command during the execution of the callback, since that is what was causing my seg fault errors from before. The main reason for this (I think) is because the do_callbacks() function uses an iterator to go through all of the events. Any other suggestions?

      • apenwarr (2002/10/16): Well, we could fix the iterator; having it not actually call the callback until doing i.next() would allow you to unregister a particular event from inside that event, although it would (oddly) prevent you from unregistering the next event while inside an event :)

      Other possibilities: allow creation of "one-shot" events, which automatically get removed after they run; or, have an event "deletion queue" which keeps a list of things to delete after all the callbacks have been run.

      The next()-before-calling-callback is the most efficient (has no performance hit at all :)) but has somewhat odd/hacky behaviour. Probably the one-shot events are the best way to go, since they might be useful elsewhere and they have only a small performance hit. Hmm. Deletion queues are the most flexible, but cost the most.

      Not sure which we should do. My initial leaning is toward one-shot events.

      • dgtaylor? (2002/10/18): Re: One shot events. Should all UniConfEvents? be oneshot events?

        • apenwarr (2002/10/18): definitely not.

          • dgtaylor? (2002/10/21): didn't think so. just checking. :) --ppatters