The short answer is this: more than they used to be .. but with a lot of room for improvement.
We have a growing number of libraries that only depend on non-GUI parts of Qt such as KDECore, Solid and Attica to name three off the top of my head. We have more than only rely on Qt and no other KDE libraries. This is rather better than the "big glop" of libraries in kdelibs in KDE 3.x days. Are we getting the most out of this? Sadly, not really.
Here are three avenues of thought for those of us involved with writing, maintaining and/or packaging KDE's libraries to consider that could help improve things:
Packaging With More Modularity
While most of our downstream packagers tend to split up applications into separate packages, KDE's libraries tend to come in big chunks that reflect the source packages. On OpenSuse, the distribution I'm using, there are two main packages: libkde4 and kdepimlibs4. The both contain large collections of libraries that are vaguely related to each other. The downside to this is that if a Qt app developer wants to use Solid, they end up having to pull in all of libkde4.
Open questions:
- What can KDE library developers due to improve this situation through build system or module layouts?
- What information or other help for packager would help drive more modular packaging?
- What libraries are most worth modularizing into their own packagers for application developers?
Identifying and Justifying Dependencies
Even if libraries were split out more efficiently, there are some interesting limits to that right now due to dependencies between our libraries. In the "pre-4" days we really didn't pay much attention to inter-library dependencies, nor did we have many good reasons to do so. Today, with a more modular Qt, a growing Qt app ecosystem and targets like mobile, we now do have some pretty good reasons to do so.
Let's take libsyndication as an example. libsyndication is the library used by applications such as Akregrator and Plasma DataEngines to fetch and process RSS feeds. It depends on KIO, a library which pulls in a fair number of dependencies with it. If we examine what it uses KIO for, we find out that KIO is used in just one file: dataretreiver.cpp. Within that file, it is used for one purpose only: to fetch feeds over the network, usually (exclusively?) it seems via http. It would be possible to use QNetworkAccess for this instead, which would use KIO in a KDE Workspace, turning KIO into a runtime dependency driven implicitly by the user. Without those few lines of KIO usage, libsyndication would only require KDECore, QtCore, QtNetwork, QtDBus and zlib.
Combined with more modular packaging, this would open libsyndication to a much broader Qt app development audience, with no loss of functionality. Of course, libsyndication is just one example (it came up in an email conversation with a Qt and Gtk+ app developer last night, actually). How many other KDE libraries have similar unnecessary dependencies?
We don't want to sacrifice integration or functionality, but there seems to be a lot of possibility here. How many can we identify? How many identifications can we turn into realized improvements?
To Split, To Aggregate?
Another avenue for exploration is the lines along which our libraries are split. In working towards the first version of KDE Platform 4, we re-arranged the classes in KDECore and KDEUi so that KDECore had not GUI dependencies. This brought a very nice division of tasks between the two. There is so much more opportunity for this, however.
KDEUi itself is an interesting, though highly complex, example: it has all sorts of useful UI bits in it, some of which are applicable to QtQuick applications and some which only make sense to QWidget based ones. Is this an opportunity?
Nepomuk libraries in kdelibs combine both data and user interface elements: does it make sense to split that into two, so more Qt apps could take advantage of Nepomuk with fewer dependencies?
How about libplasma, which has a number of both data and user interface elements in it? It's still an unresolved question as to whether we'd gain anything from splitting out the data elements (DataEngine, AbstractRunner, etc) from the user interface bits, or along which lines such a split would actually happen. It might make more sense to cleave it into "QtQuick-appropriate" and "QGraphicsWidget" parts.
This becomes a balancing act: too many libraries and load times go up; too many libraries and maintenance of the code as well as of packaging and building goes up; too few libraries and the audience decreases due to lack of clear focus and odd dependency chains.
... and more?
There are indeed more topics to mull over, such as the relationship between kde-runtime and kdelibs. What goes into kde-runtime and what goes into kdelibs is pretty well defined these days: build dependencies go into kdelibs, runtime dependencies go into kde-runtime. However, when it comes to dependencies and use cases between these two modules, what then?
Or how about the icons situation: we have a large chunk of very high quality icons .. but do we need to install them all together, or can they be modularized as well, e.g. into "in the icon spec", "also needed by kdelibs", "needed in the workspace" and "used by specific apps"?
How do we best prep the KDE Platform for QtQuick? Several parts of that answer were worked on at the recent KDE Mobile developer sprint, and we have code to show for it. Where do we go from there?
If you are a library developer, a packager or an app developer: what topics related to modularity can you think of?
These are the kinds of questions that swirl around the exciting life and times of a KDE library developer. It's the kind of querying we'll press upon ourselves in the amazing alpine environment of Switzerland's Valais region in June. Our work in 2011 will be tinted by the answers and ideas that sprout out of these lines of thought.
If you've ever pondered getting your hands dirty and your feet wet (or other body-in-substance metaphors ;) with KDE platform and library development, there's probably never been a better time to do so. For the curious, we coordinate most day-to-day development of the libraries on kde-core-devel@kde.org, use the "kdelibs" group on reviewboard.kde.org for patches and hang out in #kde-devel on irc.freenode.net.
For those of us already foot, knee, waist or neck deep in the code, we owe it to our selves, our code and our users to think about the above and take action to push the state of our art. It's a path to more users of our libraries (which also means more contributors!) as well as a way to embrace more of the Qt ecosystem and in turn be embraced right back.
11 comments:
Thank you, it is a very interesting subject. I have several thoughts. This is a bit long of a comment so I split it into two parts:
You are focusing on the libraries, but I think the same thinking can be applied to applications as well. Trying wherever possible to make applications more modular and reusable could be very beneficial. I do not mean integrating them into kdelibs or kdebase-runtime necessarily, I know there is concern that this is being done too much already. Rather, I am suggesting having them, wherever possible, be split it into user-visible interfaces and modular non-user-visible libraries or plug-ins that can be re-used by other applications on a case-by-case basis.
Two examples that immediately come to mind are ark and k3b. Both, at there core, manipulate files. Ark moves them in and out of archives, while k3b does the same for optical media. This is exactly the sort of thing kio is designed for. So rather than being single, monolithic applications, I think it would make sense to split them into a GUI and kio slaves. These kio slaves could then be utilized by other applications. For instance a video editing program like kdenlive, rather than sending its data to the k3b application and using its GUI to burn them, could handle all the burning inside its own GUI, hooking into the k3b kio slave behind the scenes. A vector graphics program could hook into the ark kio slave for saving a compressed svg rather than having to handle compressed files itself. And, of course, obvious uses like integrating simple archive or optical disc handling directly into the file browser or even the save/open dialog would also be a benefit.
Another example are kipi plugins for online photo sites. These currently use their own GUIs, rather than being KIO slaves that allow online photos to be handled right in your file browser.
Your example of plasma dataengines is another great one for this. Up to this point dataengines have been largely described solely in terms of plasma, being a plasma tool designed for plasma and intended for use in plasma. This may not be how it is intended, but that is how it comes across at least to me. However, I think there are a lot of potential uses for an easily extendable, scriptable, generic data handling library inside KDE. I think that if it was split off from plasma and promoted as a KDE-wide library for use in any sort of applications, I think we would see a lot of interesting and imaginative uses for such a system. Some potential examples just off the top of my head are: online leaderboards for games, simple akonadi calendars from websites using regex matching, updates on near-earth asteroids, supernova, or extrasolar planets in kstars, and volcanic eruptions in marble. I think these sorts of things would be well-suited to being implemented as dataengines.
Along similar lines, Akonadi is designed for PIM data -- email, calendars, etc. However, it has proven to also be mostly well-suited to financial data, so some KDE finance applications are using it. But, as far as I can tell, rather than using it directly and extending it to their own needs, they are making their own independent implementation. There are other places where Akonadi would also be useful. One example is education, where KDE is on the cusp of making a big push it seems. Things like handling class schedules, grades, students lists, attendance, and assignments seems to have a lot of similarity to handling PIM data. Along similar lines, scientific research requires tracking publications, grants, subjects, paperwork, and supplies. Medical information systems for handling patient data is another example. But akonadi is being billed as solely a PIM manager, and is tightly linked with KDE PIM applications, rather than being billed as a general information manager that can be used in a wide variety of roles in a wide variety of contexts. I think setting it up and promoting it as a more general tool would once again encourage people to find powerful and unexpected uses for it.
So I think the issue is not only how we split up the libraries, it is also how we make it easy for one application to take advantage of the work done in another application. We already have frameworks like kio that allow for what normally require separate GUIs to be integrated directly into the user's normal work-flow. So trying to make it easy to take advantage of these frameworks, easy to extend them for new and unexpected uses, and encouraging developers to use them wherever possible rather than making their own implementations, I think would encourage others to take advantage of that work rather than duplicating the effort in their own applications.
Thanks for bringing this topic up again :)
Some while ago I tried to use KHotnewstuff in my crossplattform Qt program. Its only dependencies were Qt, QtScript (including Qt bindings) and cmake.
As the program should be easily compileable on Windows, MacOS and of course linux I tried to find a way not having to include the whole kdelibs. Sadly I ended up ripping apart kdelibs and picking just the stuff I needed - and the port of Khotnewstuff itself didn't succeed... Kio I think was the biggest dependency - but there was kdecore and some other stuff... redistributable libraries for windows and macos would make our lifes much easier - libraries installable just like the Qt framework to immediatley be able to start developing.
Downloading 200mb kdelibs, compiling those and then only use 10 classes at maximum from it is not very efficient...
I don't know how much the situation has changed with Khotnewstuff3 - it has a new dependency on libattica I've heard - maybe I should try this one out...
I had hoped it to be easier to integrate DLC in my program using hotnewstuff...
@toddrme2178: "I think the same thinking can be applied to applications as well."
i agreed, however this is a completely different set of questions of a very different kind.
they are both about modularity, but the "how" and the "why" is very different for the two things. :)
definitely a discussion to pursue elsewhere (and some are: there was a recent discussion on the silk mailing list about generic re-usable KJob-based services, very much like Plasma::Service, in fact)
@friesoft: yes, that's a good example of the challenges faced. it needs to be balanced against creating 100s of tiny libraries that just becomes a new kind of mess, but i think we have a ton of low(ish) hanging fruit to tackle before we have to worry about that kind of issue :)
From my pov, libplasma could have benefits in separating it into ui and 'core'.
Runners proved to be useful in quite a few places (and I guess the same would go dataengines).
So, while runners could be used for a better help system, smart urlbar in konq, smart completion in konsole, etc. having to link all QGV stuff into non QGV apps is a waste.
Cheerio!
Have you written "big glop on purpose? ;) I can't remember finding any spelling or grammatical error since I've started reading Planet KDE, and that was, like, at the time KDE 4.0 was born :)
Ah, and to remain on topic, I'd like to add this:
What about the proposed KDElibs - Qt merge? The one Cornelius Schumacher suggested long time ago and the one that caused those long and sometimes hot, but still productive discussions on the kdelibs-devel mailing list.
I am not sure the issues are entirely separate.
First, they are not currently platforms, but the whole idea is to make them a new type of platform that KDE applications can rely on.
Second, the non-gui-parts that get split off becomes libraries. However, they are libraries that live outside of the normal core library directories. Is this where we want them? Should they be moved to kdebase-runtime, or should they be kept with their application, or should both be handled separately?
Finally, these libraries would depend on the libraries you are discussing, and could be used in ways very different than how the platforms are used by existing applications. This could have a major impact on how you lay out the
So I think you cannot make an informed decision about how the kdelibs and kbase-runtime libraries are going to be modularized and organized unless you have some idea how these libraries are going to be used and organized, and vice versus.
It also has impacts on plans for updating existing libraries. If a goal is to move most file-oriented operations from applications into kio, this will need to be taken into account in the kio updates you mentioned.
So I know that it is not currently the same issue, but depending on how you choose to organize things it could be, and it is prerequisite to planning the changes you are discussing.
And as for something like akonadi and dataengines, you mentioned dataengines yourself, and akonadi is in many ways quite similar in the scope of its use, so I think discussing those is well within the scope of your post.
So I know these are not part of the KDE platform, but the whole idea is to possibly either make them part of the platform, or redefine what we mean by the platform. So this most may be more relevant to your platform sprint post, but I hadn't seen that when I wrote this. Would you prefer I cross-post it there? Ijavascript:void(0) do feel it is appropriate for that discussion, if not this one, for the reasons I outlined.
I am not familiar with KDE from a developer point of view, but have some thoughts on this.
When splitting libraries the question is whether part x will be useful without part y. If you can think of a other uses for part x the split is a good idea. If the UI is mixed in to a library you really have a problem if you want to use it elsewhere because as far as I know user interfaces tend to carry a lot of extra baggage. I think it's always good to take the interface out of the picture for that reason, unless the library deals specifically with interfaces.
When a library is used in many places it becomes difficult to modify the design because it will cause breakage. In that sense increasing adoption can be a double edged sword. That said a library with a clear purpose and a good design is definitely illegible for exposure to a larger audience.
I Would definitely be in favour of higher levels of modularity, even if it incurs a performance hit. Better design allows you to optimize smaller bits of a software eco-system more easily, which could negate the performance hit caused by splitting everything up into smaller parts.
With regards to mobile development I would be quite unhappy if one KDE application pulled in 200mb of libraries, considering the high cost of data in many countries such as mine.
It seems like Qt4 is going to be around for a long while, as Nokia is just getting started with it in their devices. It's unlikely we'll see a Qt5 anytime soon.
I understand KDE's commitment to binary compatibility, but what about a planned breakage a few versions ahead? A time to really revamp things and refresh the layout of the project, by KDE 4.10 for example.
In general, it seems that since KDE 4.0, much has changed in KDE land. Things are becoming more modular; the platform has become more complex; there's now a rather vague idea of the 'software collection' and with that, external usage of KDELibs has been decreasing dramatically. Things are changing.
It seems like KDE might need to adapt to these changes by doing some somewhat drastic reorganization of the code and the project's components. And I can't help but think this will require breaking BC. On the other hand, it's clear that version 4 is just really getting started and has quite a bit more shelf life left in it, so I do not think a KDE 5 is warranted.
So what about a compromise -- drastic changes in KDE 4.10. What are your thoughts?
I think it is very important to get the simple things right.
For instance, while Solid depends only on QtCore, I would be surprised if it didn't depend on cmake modules from kdelibs. Of course a compile-time dependency technically shouldn't be a problem, but it is. QtRuby was sort-of forked because someone couldn't figure out how to build it without KDE.
We've now resolved this by completely splitting up kdebindings and having everything be in its own Git repo.
Maybe such a radical solution isn't necessary for kdelibs modularization. But "build-time independence" would certainly be on the top of the list to make modular libraries actually appear to be modular. "clone/checkout-time independence" like what KDE Bindings has perhaps not necessary if there is plenty of education to overcome the obvious assumption that if several libraries are grouped together in one repo that they actually require each other.
Post a Comment