I've been pondering the question of "when should we backport features?" It's fashionable right now to backport features in Plasma (and, I imagine, other applications) from the development track to stable releases and then include those stable releases plus the backported code from unstable branches to the general user base.
It has been shown that this leads to bugs since the features backported are not always completed or been properly QA'd (that's the meaning of a "development branch"). It also diminishes the goals of the upstream development team by changing the timing and combination of features provided (which can have morale diminishing effects of "stealing their thunder"). It also exposes users to unnecessary risk: what happens if we change how the configuration is stored and retrieved for any given feature? What happens if we change how the feature works, or even withdraw the feature from a release due to scheduling?
There is a reason we have release cycles, and backporting here and there circumvents all of those reasons. It's a bit ironic that some doing the backporting are also involved in planning releases and understand those reasons, implying that it's good enough for upstream but not downstream.
Of course Free software is great because it lets you do whatever you want (within the bounds of that license). Some KDE distributors see good reasons in doing these backports: they want to support KDE 4.1 for a longer period of time and their user base demands a certain feature that is only available in the next release after 4.1. Or they don't have flexible release schedules and so can't wait for 4.2 or they have a policy of not offering upgrades such as 4.2 for a release based on 4.1 (a form of self-imposed inflexibility). Or they simply want to show off as being "better than the other KDE teams" while others then prove they can "keep up". There are numerous reasons for backporting, some better than others.
It remains a judgment call in all cases, however. I urge those doing the backporting to use their judgment wisely and consider both their users and upstream developers in the process.
One unfortunate result of this backporting fad is that every time we do anything interesting in Plasma these days we get requests from someone to backport it, as if that was always easy and came with no downside. It's getting a bit ridiculous and completely runs counter to the idea of release and testing processes, and puts us in the Plasma team in an uncomfortable position of saying "No" to questions (and people) we should never have to be confronted with.
There is a middle ground, a balance point to be found. Right now I don't think we're there, though with a bit of thought and care we can reach it.
What do you think is a good reason or motivation to backport features that are in a development branch back to a stable branch?
Subscribe to:
Post Comments (Atom)
22 comments:
Thats is so true..... Thats what also put's some KDE Distributions like say Kubuntu, that don't do much or any back-porting in a unfairly bad position if compared to openSuSE, from a non-knowing, don't care user's Point of view of course..
If to say, that building a Distribution is a game, then back-porting KDE's features is defently a foul or cheat, and unfair to other player's....
KDE 3 feature regressions.
disclaimer: I'm on the openSUSE KDE team.
As distributions we played the 'early adopters only' card for KDE 4.0. Now users want to be able to move to 4.1 but demand some more features from KDE 3. Some could be worked around by starting KDE 3 apps; many distros don't have the luxury of being able to have multiple KDE versions installed in parallel; Plasma regressions cannot be avoided in this way.
Feature regressions are especially significant for the Autumn 2008 round of distro releases because we're caught between the unstoppable migration to KDE 4, and 4.2 in January will definitely provide the missing features. Our release dates are determined by a number of factors that prevent skewing until 4.2 is out. And software version upgrades to released distributions are impossible to maintain.
The features that we do backport are done deliberately and responsibly - we know the risks you describe and how to mitigate them, and since they are feature regressions we know how they should behave and are able to QA them. There is no irony in the overlap between our roles in the KDE release process and distro development - as distros we assign extra resources to rush a feature, but this necessarily happens outside the KDE release process.
@Bille: i understand your motivations (as a distro); i don't find the release date fixations to be increasingly damaging (and blame it mostly on the 6 month fetishists), but also understand the rational for them.
facts remain, however, that people are backporting bugs, that users are now asking for ever more backportings (distros are teaching them the wrong expectations!) and that the distros are exposing their users to unusual and highly dubious risks.
there probably is no single "right" answer for all forms of "should we backport features $FOO", and i think i address that in this blog entry.
my hope is that downstreams can take the matter a bit more seriously than they are right now.
and it's not an OpenSuse issue: this backporting phenom is happening in Kubuntu, Fedora, Mandriva and doubtless others.
"The features that we do backport are done deliberately and responsibly"
deliberately, absolutely. i would debate the responsibly part based on what i've seen happen already.
"we know the risks you describe and how to mitigate them, "
how do you mitigate the "we are going to change the configuration system" risk? (to name just one)
"as distros we assign extra resources to rush a feature"
it's not about amount of resources; if it were then putting 2x the # of people on a project would reduce the time needed by half. that's not how software dev works. you know that, i know that.
so while i understand why backporting of some features is happening, it would be great to see the unhealthy rationalizations be curbed a bit.
i'm not 100% against backporting, i'm just looking to return the pendulum back towards center a bit here.
disclaimer: I'm an openSUSE KDE user. ;)
I can very much imagine the backporting of regressions from KDE3. openSUSE uses the KDE Factory for putting a most up to date KDE 4.1.x on my system. However, that is Factory and a choice by me. As far as I know, the stable version on openSUSE 11 is still 4.0.4 (or does this also get backports from 4.1.x?), is it not? So those using KDE 4.1.x from openSUSE at the moment made a choice to run possibly unstable software. Correct me if I'm wrong on part of this, though. :)
It's not only about regressions compared to KDE3 - look at how all of Mandriva, Fedora, Kubuntu and openSUSE have backported the Plasma tooltip manager from KDE 4.2 simply because this feature dissappeared from KDE 4.0 to 4.1.
@perena: this is about 4.1 in stable releases with features backported from 4.2, but it was equally true about 4.0.
@beineri: yes, there were some casualties from the QGraphicsView changes between Qt 4.3 and Qt 4.4; were the tooltips that critical, though? dunno. were they easy and safe to backport? almost certainly. what if that effort went into improving the 4.2 experience instead, though?
it isn't an easy set of decisions. =)
@parena
**As far as I know, the stable version on openSUSE 11 is still 4.0.4 (or does this also get backports from 4.1.x?), is it not?**
exact.. The "stable" openSuSE 11.0 KDE 4.0.4 uses a lot of back-ported features from kde4.1
As you see, users think those features are stable and belong to the regular release-branch, because they name it 4.0.4
It would make more sence if Distributions that do a lot of back-porting to name it different, as it is not a KDE 4.0.4 only release, maybe something like (KDE 4.0.4 / 4.X DEV )
Well, I am a Kubuntu 8.10 user, and I admit I thought a couple of times that backporting things from 4.2 to 4.1 would be exciting (thinking, for example, about the integration between Nepomuk and FolderView, that is on trunk/ now, at least for what I know), without the necessity to wait for 3 months.
At the same time, I know that if something appears on 4.2 trunk it may be using new functions from 4.2 libraries, so the backporting may remain a dream...
As Aaron, I see the reasons for the distros; their users demand something that kde4 still doesn't have. So these distros try to give it to them.
While this is very noble of those distros I don't think they notice the pain it causes upstream. The users that run KDE4 from such a distro will blame the project for the bad QA and only secondairy blame their distro, if at all.
So when one packager in the comments said "we know the risks you describe and how to mitigate them", I'm wondering if he took the damage done to the upstream project into account.
As a software developer I am wondering about the claim that there is an unstoppable migration to kde4. Isn't that caused by these same packagers that keep on claiming kde4 is good-enough for all their users. While we very well know that it has yet to reach feature parity of 3.5. So its certainly good for a large set of users, but not everyone.
The reason I'm bringing this up is that I feel that these distros are just continuing their mistake of giving their users what they ask, even if they can't. The effect we are seeing is the difficult position they are in now and the effect of backporting on the whole ecosystem of free software is yet to be seen. But I doubt its a positive one.
Amen. Everyone would be better off (if-at-all-possible) avoiding the short-term temptation of local (ie, distro-only) backporting.
If those same folks are that serious about wanting or needing a feature, devote the time, energy, resources to aid in it's (imo proper) upstream development. And, in that effort, some features could be more safely backported to stable branches.
It's a hard sell, particularly because the latter part (upstream backporting) may require more flexibility in policy, but I'd venture there is some middle ground to be found.
Simple solution:
Release a little later.
That will give distros less time to backport.
A very related issue:
The newest fad is to release distros with beta versions of Amarok 2. Mandriva even released with some SVN version. OpenSUSE is also set to release with a beta I think, according to the Ars article I read. It is partly our fault, we were planning on having Amarok 2 out sooner. But a major feature (almost rewritten) release being a bit late is hardly a unforeseeable circumstance.
I suppose they have limited live CD space so they don't want to include Amarok 1.4. But its still just shows how little these distros care for producing a quality product, at least for their KDE users. Amarok 2 is my main player, but its still fairly crashy and not intended for a user not interested in helping us test but just wants to play some music.
I haven't switched my girlfriend from Amarok 1.4. So why the hell would Mandriva think its OK to switch to Amarok 2?
@ian:
If you don't know, don't assume the worst. openSUSE 11.1 will ship with 1.4.10 as well as whatever the latest beta is we can get our hands on as late as possible. They are co-installable, so your gf won't lose her music.
@aseigo:
mitigating config changes: update scripts. It takes work, but it's just donkey work to transform one set of configs to another.
resourcing backports: i didn't claim we could halve development time by putting people on a task, but we can make well-understood and scope-limited features work adequately for our needs.
and in case anyone's not clear: wherever possible, we do contribute directly to upstream development directly. But sometimes that just doesn't fit our schedules. It gets pushed up later.
@bille what's the default?
If Amarok 1.4.10 is the default, I don't mind at all if they ship Amarok 2. Amarok 2 is good stuff, but I think its still only for users who consciously choose to use it.
@jscurtu:
> then back-porting KDE's features is defently a foul or cheat, and unfair to other player's....
Please don't overdramatize. Our backports are all public one way or the other (source packages, packaging SCMs, openSUSE even has a branch in upstream SVN for their Plasma backports), nothing prevents other distributions from using them. (And yes, Kubuntu also has some backports, and even some patches which aren't upstream at all, such as support for multimedia keys in KMix.)
@bille:
I agree with most of what you're saying, in particular you mentioned the #1 reason for backporting:
> KDE 3 feature regressions.
Users see missing features in KDE 4 as a bug, they don't care that it is because the code was completely rewritten, all they see is that the feature used to be there and no longer is, and they expect that "bug" to get fixed in bugfix releases. There's a big disconnect there between KDE policies (which see these things as features, to be added only in new feature releases) and user expectations.
There is, however, one statement I have to disagree with:
> And software version upgrades to released distributions are impossible to maintain.
This is not quite true, we upgraded Fedora 9 from 4.0 to 4.1 with very few problems (one of which was the tooltip regression, which got fixed by, you guessed it, a backport) and we will try to do the same with 4.2 for Fedora 10 and possibly 9. Fedora also did this in the past: Fedora Core 4 originally shipped with KDE 3.4.0, it got upgraded all the way to 3.5.3 (one of the upgrades was from 3.4.2 to 3.5.0). Of course we judged the move from KDE 3 to 4 as too big for an upgrade, but that was about the only one.
@aaron:
> deliberately, absolutely. i would debate the responsibly part based on what i've seen happen already.
What? The bugs in the first versions of the autohide backport? Those never made it into any openSUSE or Fedora release (only Factory/Rawhide and openSUSE's repository of backports from Factory), the worst one (100% CPU, due to me backporting a bugfix without the regression fix for it, oops...) was exactly one day in Fedora Rawhide and never anywhere in openSUSE. Now they're all fixed.
> how do you mitigate the "we are going to change the configuration system" risk? (to name just one)
As Bille says, we can easily add a custom kconf_update script if the need arises. That's what kconf_update is for.
> what if that effort went into improving the 4.2 experience instead, though?
We will start working on improving the 4.2 experience once you stop working on it and start focusing on 4.3, and then we'll have the same discussion all over again... Unfortunately, as upstream developers, you will always be one branch ahead of what we ship (and thus care about the most), it's pretty much unavoidable. Development is about making the next release great, packaging is about making the current release great.
@jscurtu:
> It would make more sence if Distributions that do a lot of back-porting to name it different, as it is not a KDE 4.0.4 only release, maybe something like (KDE 4.0.4 / 4.X DEV )
Well, putting 2 version numbers on it would be really confusing, and if we named it 4.1 (for 4.0+backports) or 4.2 (for 4.1+backports), we'd be making a promise to people we obviously can't keep. The fact is, almost all the code we ship is still the base version, the backports are fairly small parts, even in relation to Plasma (well, it may be arguable for openSUSE, but in Fedora we only use a subset of their backports), but even more so in relation to all of KDE.
@thomas:
> While this is very noble of those distros I don't think they notice the pain it causes upstream. The users that run KDE4 from such a distro will blame the project for the bad QA and only secondairy blame their distro, if at all.
Is this really that big a problem? Do you have any numbers of how many bugs these backports actually introduce? I'm not convinced that it's really a significant amount. We do do QA on our backports and we don't push them out if they're obviously buggy! Of course there may be bugs our testing misses, nobody is infallible, but please don't accuse us of doing "bad QA". For example, I was the first one to notice the infamous 100% CPU bug in my autohide bugfix (ironically) backports, I did test it myself. And we also had at least one Rawhide tester who found it in the one day it was in Rawhide. Rawhide testers know the risk of testing Rawhide, and they're invaluable for QA. In several cases, Rawhide users testing backports of features actually uncovered bugs in upstream software, which could then be fixed in time for both the first upstream release with the feature and the first Fedora release with the same feature. But they also know that a lot of the bugs in Rawhide are our fault, not upstream's.
@Rex:
> It's a hard sell, particularly because the latter part (upstream backporting) may require more flexibility in policy, but I'd venture there is some middle ground to be found.
I think you nailed one of the main problems there. For example, when all the major distros backport a feature (such as the Plasma tooltips from 4.2 to 4.1), what sense does it make not to include that feature in upstream 4.1.x releases? It would make it much easier to share bugfixes if they turn out to be needed (though in the case of the tooltip backports I'm not aware of any bugs we'd have to fix), possibly also performance optimizations (I know some optimizations for the tooltips went into 4.2 after I did my backport), and it would make sure the upstream 4.1.x releases actually reflect what most users will be getting (because, let's face it, the vast majority of KDE users will get it from one of the distros which carry the tooltip manager backport).
Really, writing off a backport as "distro-only" stops making sense when all the major distros ship that particular backport, it would make much more sense to have it upstream.
IMHO the "no new features" policy for the bugfix releases really ought to be relaxed, especially for feature regressions compared to earlier releases (especially earlier 4.x releases, but 3.5 should also be considered).
@kragil:
> Simple solution:
> Release a little later.
> That will give distros less time to backport.
That won't change much. For one, not all distros release at the same time, and secondly at least Fedora will also add backports in updates, so there's no way to time your feature additions to prevent them from landing in Fedora. As far as I know, KDEmod also pulls in backports at any time.
Another thing I should add: There's also some amount of cooperation between distros on those backports. It could probably be better (there's some amount of competition or rivalry going on, unfortunately, so most of the exchanges happen by pulling, not pushing as they ideally would), but for example the autohide stuff was originally done by openSUSE, then I pulled it into Fedora and backported a set of bugfixes, now those bugfixes made it back into plasma-4.1-openSUSE.
I think building some forum to exchange our backports in could also help making the current situation a bit less of a mess. But ideally they'd just land in upstream bugfix releases!
As for the Amarok 2 issue, this is a combination of 2 things:
* schedule slippage: Unfortunately, distributions also have schedules, we need to decide on which version of Amarok to ship well in advance (well, reverting would be possible in extreme cases, but now that Fedora 10 is frozen, it's too late to do it in Fedora 10, and it's also something we don't like to do because reverting has its own problems), we can't just upgrade it at the last moment, because it needs to be tested in Rawhide (or Factory or Cooker or sid or whatever). So if you promise you'll have Amarok 2 done at a certain point in time and then you can't keep that promise, you put us in a difficult position.
* one-sided communication: All we (and even more importantly, our users!) get to see is blog posts about how great Amarok 2 already is, the message that it's "not intended for a user not interested in helping [you] test but just wants to play some music" gets completely drowned in all the hype (with the result that our users will complain if we don't ship it, especially in Fedora which primarily targets early adopters).
Oh, some stuff in the original post I forgot to reply to:
> What happens if we change how the feature works,
Then the changed way is (hopefully ;-) ) an improvement, so I don't see the problem, a later update (either an updated backport or the inevitable version upgrade) will have your improvements.
For example, we shipped openSUSE's hack to move plasmoids on panels in Fedora's KDE 4.0 packages, but we were quite happy to be able to replace it with your much more usable solution in 4.1. (This is not a criticism of the openSUSE patch though, it served us well in the absence of a better solution! But things are rarely perfect, and improvements are always welcome.)
> or even withdraw the feature from a release due to scheduling?
Then we can just keep the patch (but do expect us to complain about a feature our users requested so much that we decided to backport it getting dropped).
> There is a reason we have release cycles, and backporting here and there circumvents all of those reasons. It's a bit ironic that some doing the backporting are also involved in planning releases and understand those reasons, implying that it's good enough for upstream but not downstream.
That argument might work for the likes of openSUSE, but here in Fedora we can and do push new features as updates to a released distribution, so the strictness of KDE's "bugfixes only" policy for release branches is foreign to us. The KDE branch closest to how we're used to working in Fedora is/was the 3.5 branch. (I wrote "is/was" because it has cooled down a lot these days, as obviously most development effort goes into 4.x now (and that's a good thing).)
The back port rule:
Its up to you, the requester, to back-port. The feature already exists, so if it is so simple for the developer to do, it should be simple for you to do. And hey, it's a learning experience that might get you knowledgeable enough to add your own features and fixes...
@b10663r: Sorry, but that's just nonsense. Backports are usually requested by users who don't even know how to compile KDE, let alone patch it. And even if they could manage to do it, no thanks, a backport done by a user with no coding and packaging experience is very likely to be full of bugs! Backporting should only be done by people with at least a minimum of experience with the programming language used.
and what about release more often (3 month release) ?
KDE have now good fundations and perhaps we dont need so long cycle for now ?
@kevin k.
> Is this really that big a problem?
This is one of those questions I'm not entirely sure how to answer; to me the question feels like similar to "are kids really smaller than adults[1]?".
Anyone with experience in managing a software release knows the answer to your question and the fact that just about all software out there has a stable-release branch with a feature freeze on it should be enough to proof my point.
If you don't share that experience, allow me to explain my view and expertise (of some 20 years in the business); yes its a big problem. We freeze the introduction of features months ahead of release to find and fix bugs in them because there always are lots of um. Bugfixing will continue for months after the release even. Even the most experienced developer will introduce lots of bugs and miss tiny usecases where his code will be buggy.
So basically you are taking a feature 6 months before its owners find it release-worthy and you give it to users. Not even users that signed up for a known-buggy version, no all users.
Thats 6 months of bugfixes you are missing, at least half of that time even before the developer thinks his code is good enough for general use.
Stating that you backport the bugfixes too is first of all not reality (not all bugs are *in* the feature itself, good luck finding them all), second distros *do* release this unfinished software to real users before the authors finally do their release.
Bottom line is that the distros that do backporting are giving developers a lot of pain and this pain will most likely have the effect that more developers will start doing more closed development keeping those features out of the public trees to avoid them being shipped as 'done' by distros.
ps. no idea which distro you work on. This certainly is not anything against any distro or person specific.
1) kudos for getting the movie reference.
I think you are vastly overestimating the amount of bugs these small, targeted backports introduce. It's not like we ship the whole KDE 4.2 prereleases, or even the whole Plasma trunk! What we're doing is to take one frequently-requested feature, isolate the changesets (not just the one which introduced it, but also any followups with bugfixes), backport them to the released version, test the result for at least 1-2 weeks (which is a lot for a single feature!), keep tracking the trunk for bugfixes in that time, and only push the result when we're satisfied with it, which is when we don't notice any bug.
Now of course I'm no dreamer, I know that it's unlikely there is really no bug at all, we can't notice everything. However, in distributions, our goal is to make our users happy. We were able to ship panel autohiding in a Fedora 9 upgrade, which is something a lot of people requested, and we got zero complaints about bugs introduced by that update. In other words, if there are any bugs, our users are not seeing them (nor did we, or we wouldn't have pushed it in the first place!).
And our experience is that users will tend to report all bugs in our distro bug tracker (even those which are not our fault). But if you're seeing any bugs in bugs.kde.org which you suspect are caused by our backports, then please by all means CC me on them or even assign them to me (name dot surname at chello.at). If I can confirm it is my fault, I will take the blame for them and explain what is going on. If you are aware of any relevant bugfixes which I'm likely to miss, you're also welcome to bring them to my attention by e-mail (and I'll do my best to forward the information to the openSUSE and KDEmod folks, I realize that communication between distros has been lackluster at times, we should do better!).
And according to the KDE 4.2 schedule, we're only offering the feature 3 months early, not 6 months. We also plan to push 4.2 as an update when it's out, which would mean that at the very latest, all the upstream bugfixes from now to the 4.2.0 release will get pushed out then (in about 3.5 months - we do need about 2 weeks of testing for major upgrades too!).
Developing in private is going to really hurt community involvement, I think you'd only hurt yourselves that way. And the end result would probably be that instead of backports, the distros would ship ugly buggy quick hacks to implement the features their users need. (For an example of what kind of stuff we have to ship if there's no code from upstream to do what is needed, see the "move plasmoids on panels" patch which was in the openSUSE and Fedora KDE 4.0 packages.)
As for which distro I work on, I'm a volunteer for Fedora.
Post a Comment