Preview
Open Original
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 20:09:34 UTC
Message-ID: <fa.v22Gl2Sq+O+kAPIbhi3/O6Q7Mg0@ifi.uio.no>
On Tue, 12 Feb 2008, J. Bruce Fields wrote:
>
> But the "author" is still preserved, right? Why do you need the
> committer name to be preserved? (I'm not denying that there could be
> reasons, I'm just curious what they are.)
It's not that the committer should be preserved, but:
- the chain from author -> committer should be visible in the
Signed-off-by: lines.
If you reba...From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 20:09:34 UTC
Message-ID: <fa.v22Gl2Sq+O+kAPIbhi3/O6Q7Mg0@ifi.uio.no>
On Tue, 12 Feb 2008, J. Bruce Fields wrote:
>
> But the "author" is still preserved, right? Why do you need the
> committer name to be preserved? (I'm not denying that there could be
> reasons, I'm just curious what they are.)
It's not that the committer should be preserved, but:
- the chain from author -> committer should be visible in the
Signed-off-by: lines.
If you rebase somebody elses tree, you screw that up. You need to add
your sign-off, since now *you* are the new committer, and *you* took
somebody elses work!
- you should respect the down-stream developer, and if that downstream
developer continues to work with his branch or works with other people,
you shouldn't screw that up!
Both of those basically say that you should never rebase somebody elses
work. You can use rebase to rebase your *own* work on top of somebody
elses thing (since that doesn't change the sign-off chain, and you still
respect the downstream developers development model)!
But of course, if you rebase, you should respect the wishes of the
up-stream developer too. I don't do rebases. So if you asked me to pull,
the stuff I pulled can never be rebased, because it just *is* in my tree.
Put another way: think of the absolute *chaos* that would happen if I were
to rebase instead of just merging. Every time I pull from you I'd
invalidate your whole tree, and you'd have to re-generate. It gets
unmaintainable very quickly.
And that's actually ignoring a real issue: stability of commits. The nice
thing about stable commit naming is that all bug-reports from other people
that told where the bug happened are basically 100% trust-worthy and the
code is 100% reproducible not just for you, but for everybody else.
In other words, you really shouldn't rebase stuff that has been exposed
anywhere outside of your own private tree. But *within* your own private
tree, and within the commits that have never seen the light of day,
rebasing is fine.
(And yes, there are exceptions. If it's a clear "throw-away tree" all the
rules go out the window, of course, as long as everybody involved *knows*
it's a throw-away tree, and know that if they pull it they have to
synchronise 100% with you - so within a very tight-knit case or within a
very specific small detail that is actively being worked on, those rebases
with cleanups make tons of sense).
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 01:33:08 UTC
Message-ID: <fa.Z/Ob0SLov7zP7PKFbkR8yBXnHog@ifi.uio.no>
On Tue, 12 Feb 2008, David Miller wrote:
>
> > Put another way: think of the absolute *chaos* that would happen if I were
> > to rebase instead of just merging. Every time I pull from you I'd
> > invalidate your whole tree, and you'd have to re-generate. It gets
> > unmaintainable very quickly.
>
> I actually wouldn't mind that, the first thing I do when sending a
> pull request is I stop putting things into my tree and as soon as the
> recipient pulls I wipe out my tree and clone a fresh copy of their's.
You *really* don't see the problem here?
> I really like that mode of operation.
*YOU* like it, because it never generates any issues for *you*. You're the
top in your heap, and the people above you don't do that insane thing, so
you get all of the advantages, with none of the downsides. Of *course* you
like it.
But as people have pointed out, it generates issues for the people under
you! If I did it, the people who now complain about networking would
not just be a couple, it would be everybody. Nobody could depend on
anything out there, because everything would have to rebase.
You just don't see the problems, because the only person above you isn't
crazy enough to do what you propose. You also don't do ten merges a day of
subsystems you don't know.
The importance of merging (rather, not screwing up history in general)
becomes really obvious when things go tits-up. Then they go tits-up
*without* screwing up the history of the trees that were hopefully tested
individually.
If you re-base things that others developed, you lose that. Imagine if I
merged first Greg's tree (by rebasing), and then there was some
fundamental thing that didn't cause a conflict, but just made something
not work, when I rebased yours on top. Think about what happens.
Now I've merged (say) 1500 networking-related commits by rebasing, but
because I rebased on top of Greg's tree that I had also rebased,
absolutely *none* of that has been tested in any shape of form. I'd not
use most of the things I pulled, so I'd never see it, I'd just push out
something that was very different from *both* trees I pulled, with no way
to really blame the merge - because it doesn't even exist.
So as a result, some *random* commit that was actually fine on its own has
now become a bug, just because it was re-written.
You don't see the problem here?
Yes, this is the *crap* you do all the time. You don't see the problems as
much, because you merge probably only about a tenth of the volume I merge,
and you can keep track of the subsystem more. But even though you don't
have nearly the same kinds of problems, people have complained about your
process.
So there's a real reason why we strive to *not* rewrite history. Rewriting
history silently turns tested code into totally untested code, with
absolutely no indication left to say that it now is untested.
You can limit the damage by keeping it to a single subsystem and by
serializing that subsystem by - for example - talking it over amongst
yourself what order you do things in, and yes, most of the time rewriting
doesn't hurt anything at all, but I guarantee you that it's a big mistake
to do, and the mistake gets bigger the more _independent_ people you have
involved.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 01:59:40 UTC
Message-ID: <fa.mmT3DcI20pDu2TzHf84+doLhilQ@ifi.uio.no>
On Tue, 12 Feb 2008, David Miller wrote:
>
> Now how do I remove a bogus commit for a tree that I've already pushed
> out and published for other people, without any record of it appearing
> in the GIT tree any more?
So, the answer is: if others have actually pulled, it's simply not
possible.
There simply is no way to undo history that isn't local any more. You just
*have* to revert, or you need to find every single person that pulled
(directly or indirectly) and ask them to undo all the work they did on top
of your bad commit.
This is not git-related, btw. That's just how the world works. It's a bit
like the internet - if you said something stupid on #IRC and it made it to
bash.org, there's not a whole lot you can do to "undo" your stupidity. You
can only hope to do better in the future and assume people forget.
> How do I insert build fixes into existing changesets so that the tree
> is more bisectable?
Just delay pushing out. There really is _zero_ downside to this. None at
all. There are only upsides.
> If Jeff merged in a tree that introduced a ton of whitespace errors
> git is complaing about, is there a way I can fixup that changeset
> in-place? (or should I tell Jeff to start adhering to GIT's whitespace
> warning messages when he applies patches?)
Umm. Git doesn't complain about whitespace _except_ when applying patches.
So if you don't rebase his work, you'll never see the whitespace warnings
either!
Of course, we'd probably wish that Jeff cared about the whitespace
warnings and pushed back on them, but the fact is, that warning isn't
meant for you - because by the time you pull Jeff's tree, it's simply not
your issue any more. Jeff already applied it. Either he's your trusted
lietenant or he's not.
Quite frankly, to me it sounds like you're not ready to "let go" and trust
the people under you. Trust me, it's worth it. It's why your life is easy:
I have let go and I trust you.
Also, I'd *much* rather have a few problems in the tree than have people
screw up history in order to hide them. Sure, we want to keep things
bisectable, but quite frankly, if you do a reasonable job and compile the
kernel before you push out, it will be "mostly bisectable".
And yes, mistakes happen. Mistakes will *always* happen. It's ok. Relax.
Let me put it another way: You're _both_ going to be *much* better off
pushing back on Jeff, telling him that "I can't pull from you because your
tree is ugly and doesn't compile", than taking his tree and rebasing it.
Remember? I used to do that all the time. I berated the ACPI people for
creating monster trees that were horrible and contained fifteen merges and
two real commits. I didn't try to clean it up for them, I just told them
what the problem was, and you know what? The ACPI tree is one of the
cleanest ones out there now!
So in short:
- clean trees and bisectability are all wonderful things. No doubt about
that at all.
- but if getting there means that you lose a lot of _other_ wonderful
things (like being able to trust history, and the people being under
your watchful eyes having to deal with you re-writing their trees),
we'd be much better off taking the occasional commit that fixes things
up _after_ the fact rather than before!
- and you actually can help fix your issues by doing some simple things
*before* pushing out, rather than push out immediately. IOW, do your
whitespace sanity fixes, your compile checks etc early, and don't push
out until after you've done them.
Hmm?
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 02:20:14 UTC
Message-ID: <fa.c9IkgAsU/RxSF7B7AdO8v1Ku1xg@ifi.uio.no>
On Tue, 12 Feb 2008, Andrew Morton wrote:
>
> So it would not be efficient for David to do all this queue-cleaning
> *prior* to putting the tree into linux-next, because more stuff will pop up
> anyway.
Well, what others have done is to have special "temporary branches".
This is what git itself does, for example. The "pu" branch in git is used
for experimental stuff, and it's _declared_ to be rebased, redone, and
generally just unsafe at any moment.
So it is easy to have a special "testing" branch that is just declared to
be unsafe. Make Linux-next pull that testing branch - it will pollute the
Linux-next tree (and anybody else who just wants to see what the current
state is), but since those are re-generatd from scratch every day
_anyway_, so who cares?
But don't make it something people pull by mistake (ie never call it
"master", and when mentioning it in some email message, always mention the
fact that it's not a stable branch, and never ask anybody to pull it
without making it very clear that it's just for testing, not for real
merging).
So git does have support for those things. They are very much "secondary"
branches (any tree they are pulled into will itself become "poisoned" and
unstable), but it's easy enough to have something like that for testing
purposes. And if it all tests out fine, you can just move it as-is into
the "real" branch if you want to.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 05:45:22 UTC
Message-ID: <fa.aDGitWhzDbPN7o4q+UuSjj41eZ8@ifi.uio.no>
On Tue, 12 Feb 2008, J. Bruce Fields wrote:
> > So as a result, some *random* commit that was actually fine on its own has
> > now become a bug, just because it was re-written.
>
> If there was a "fundamental thing that didn't cause a conflict", then
> the two trees in question probably didn't touch the same code, so would
> probably merge cleanly, for the same reason that one rebased onto the
> other cleanly. But depending on what the "fundamental thing" was, the
> merge might still introduce the same bug, right?
Absolutely. But if you do a true merge, the bug is clearly in the merge
(automatedly clean or not), and the blame is there too. IOW, you can blame
me for screwing up. Now, I will say "oh, me bad, I didn't realize how
subtle the interaction was", so it's not like I'll be all that contrite,
but at least it's obvious where the blame lies.
In contrast, when you rebase, the same problem happens, but now a totally
innocent commit is blamed just because it happened to no longer work in
the location it was not tested in. The person who wrote that commit, the
people who tested it and said it works, all that work is now basically
worthless: the testing was done with another version, the original patch
is bad, and the history and _reason_ for it being bad has been lost.
And there's literally nothing left to indicate the fact that the patch and
the testing _used_ to be perfectly valid.
That may not sound like such a big deal, but what does that make of code
review and tested-by, and the like? It just makes a mockery of trying to
do a good job testing any sub-trees, when you know that eventually it will
all quite possibly be pointless, and the fact that maybe the networking
tree was tested exhaustively is all totally moot, because in the end the
stuff that hit the main tree is something else altogether?
I don't know about you, but I'd personally be really disappointed if it
happened to me, and I felt that I did a really good job as a
submaintainer. I'd also feel that the source control management sucked.
Contrast that to the case where somebody simply does a merge error. The
original work doesn't lose it's validity - so the original maintainer
hasn't lost anything. And quite frankly, even the person who "screwed up"
with the merge hasn't really done anything bad: these things _do_ happen.
So bugs happen; not big deal. But the fact that the bugs are correctly
attributed - or rather, not mis-attributed to somebody blameless - that
_is_ a big deal.
It's not like I will guarantee that all my manual merges are always 100%
correct, much less try to guarantee that no subtle merge issue can make
things not work even if it all merged totally cleanly. That isn't my
point. And others will make merge mistakes too. But the people they merged
from will not be blamed.
So just the fact that the right commit gets blamed when somebody does a
"git bisect" is I think a big issue. It's just fundamentally more fair to
everybody. And it means that the people who push their work to me can
really choose to stand behind it, knowing that whatever happens, their
work won't get diluted by bad luck or others' incompetence.
And no, maybe most people don't feel things like that matters. But I do
think it's important.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 17:10:53 UTC
Message-ID: <fa.wB4gyi77c52JUMRBUzDezTF9cvs@ifi.uio.no>
On Tue, 12 Feb 2008, Jeff Garzik wrote:
> David Miller wrote:
> > This is why, with the networking, we've just tossed all of the network
> > driver stuff in there too. I can rebase freely, remove changesets,
> > rework them, etc. and this causes a very low amount of pain for Jeff
> > Garzik and John Linville.
>
> Rebasing is always a pain, and John and I both agreed the other day that you
> do it too often.
I do think that some people rebase too often (and that David is in that
number), often with no real discernible reason. I think rebasing is great
when you are doing active development (ie you're really acting in "quilt
mode") and I actually think git could and should integrate more of the
queues modes, but I don't think it should be a default action for an
up-stream developer.
I also don't think rebasing helps the particular problem under discussion
(ie conflicts due to having to sort out dependencies between different
trees), and in some ways hurts it.
One thing that I personally react to is that
- I think sysfs and the device model layer has had too much churn, and
I'm unhappy that people seem to expect that to continue.
[ NOTE!! I'm picking on the device model/sysfs stuff here not because
it's the only one, but because it's the obvious and good example. I
do think we have other cases of the same thing. ]
Really. I do agree that we need to fix up bad designs, but I disagree
violently with the notion that this should be seen as some ongoing
thing. The API churn should absolutely *not* be seen as a constant
pain, and if it is (and it clearly is) then I think the people involved
should start off not by asking "how can we synchronize", but looking a
bit deeper and saying "what are we doing wrong?"
It may well be that part of the problem is that the people causing the
churn don't realize the downsides of the pain they are causing, because
THEY aren't the generally ones that see it!
For example, it's easy for Greg to change his driver core, and he can
obviously synchronize with himself in the other trees (because his left
hand is hopefully somewhat aware of what his right hand is doing), so I
suspect Greg simply doesn't see the pain that much. So Greg thinks that
the solution is to just have me merge his changes early, and the pain
is all gone as far as he is concerned.
- That said, I'm also a bit unhappy about the fact you think all merging
has to go through my tree and has to be visible during the two-week
merge period. Quite frankly, I think that you guys could - and should -
just try to sort API changes out more actively against each other, and
if you can't, then that's a problem too.
In other words, please do use the distributed nature of git to your
advantage, when there are things you guys know you need to sort out.
So there are two separate and totally independent issues here.
One is that I suspect some people are a bit too willing to do cleanup for
its own sake, and do not realize that backwards compatibility does
actually help too, and that "better solutions" are sometimes worse than
"keep things stable". We should always *allow* major breakage when
necessary, but I think the threshold for them should be higher than I
think it currently is.
The other is that once somebody says "ok, I *really* need to cause this
breakage, because there's a major bug or we need it for fundamental reason
XYZ", then that person should
(a) create a base tree with _just_ that fundamental infrastructure change,
and make sure that base branch is so obviously good that there is no
question about merging it.
(b) tell other people about the reason for the infrastructure change, and
simply allow others to merge it. You don't have to wait for *me* to
open the merge window, you need to make sure that the people that get
impacted most can continue development!
This is where "rebases really are bad" comes in. When the above sequence
happens, the fundamental infrastructure change obviously does need to be
solid and not shifting under from other people who end up merging it. I do
not want to see five different copies of the fundamental change either
because the original source fixed it up and rebased it, or because the
people who merged it rebased _their_ trees and rebased the fundamental
change in the process.
Can that (b) be my tree? Sure. That's been the common case, and I'll
happily continue it, of course, so I'm not arguing for that to go away.
Merging is my job, I'll do it. But when the merge window is a problem, my
merge window should *not* hold up people from using the distributed nature
of git for their advantage.
But yes, obviously when doing cross-merges, you'd better be really
*really* sure that the base is solid and will get merged. But let's face
it, all the really core maintainers should damn well know that by now:
you've all worked with me for years, so you should be able to trivially be
able to tell whether you *might* need to worry about something, and when
it's a slam dunk.
And it's the "it's a slam dunk" cases that I think are (a) the common ones
and (b) the ones where you can just do cross-merges to satisfy each others
needs.
Hmm? Does that sound palatable to people?
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 18:50:15 UTC
Message-ID: <fa.tPCENopPlFqcYCCZZdpMRn5fp5k@ifi.uio.no>
On Tue, 12 Feb 2008, James Bottomley wrote:
>
> Hm ... I think net is a counter example to this. Rebases certainly work
> for them.
They consider themselves to be "one tree" and are thus largely a totally
different issue than the one discussed here.
Also, I actually flamed David a lot last round over his rebasing of
netfilter etc. It was no acceptably done. That network tree was all
screwed up, with the git committer information not matching the signed-off
path etc.
If you do cross-tree rebasing, you need to consider it 100% equivalent to
just passing patches around in emails. Because it really is.
> Yes ... I don't do that ... Like I said, I only rebase for an actual
> conflict.
And this is how things should work.
> Well, it came at me because Jens was rebasing the block tree as he
> worked through issues in the two branches I was based on.
Yes, and I am in no way saying that the core driver model has been the
only problem spot.
And also, I do not like "hard rules". Every rule always has an exception,
and sometimes a rebase-based strategy can be the right thing even across
trees.
But you're all ignoring my fundamental objection: you're talking as if
cross-tree fundamental API changes should be the "norm", and that we
should try to solve the workflow issues that stem from that. And I'm
saying that I think we should try to FIX the issue, and make sure that
it simply *isn't* the norm.
In other words, I'm perfectly happy to be an a*hole and tell people that I
simply won't merge things that cause undue API churn at all, and that were
not thought out sufficiently.
We've had too many issues like that (SG chaining, iommu, driver core, not
to mention the upheavals in x86) lately, but realistically, which
subsystem remains a problem for the future? And maybe the correct thing to
do is to just say "enough!".
I'm perfectly happy being hardnosed and saying "nope, that's crap, it
doesn't matter if the code is slightly better if you cause those kinds of
issues".
The thing is, sometimes the answer really *is* "Don't do that then!". If
our model becomes bogged up by cross-subsystem serialization issues, that
means that we (a) spend too much time handling the fall-out from that and
(b) too little time just refining the details.
And quite frankly, while "good core architecture" matters a lot, in the
end, "getting all the boring small things right" matters even more! Big
architectural churn that cleans up and fixes the "big issues" is totally
anti-productive if it means that we don't look at the boring small detail
stuff.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 19:00:06 UTC
Message-ID: <fa.O5Ko/o53kZW8No0KtLGVLnQnl70@ifi.uio.no>
On Tue, 12 Feb 2008, Linus Torvalds wrote:
>
> In other words, I'm perfectly happy to be an a*hole and tell people that I
> simply won't merge things that cause undue API churn at all, and that were
> not thought out sufficiently.
.. btw: I'd need to know this in advance. I usually don't see the problem
until it's too late.
And this is very much an area where "Linux-next" can help: if some
subsystem causes problems in Linux-next for other maintainers, I really
think it shouldn't just be a matter of "drop the git tree that didn't
merge cleanly", but it should literally be a question of "maybe we should
drop the _earlier_ git tree that caused the later one not to merge
cleanly".
In other words, maybe things like core block layer changes or device model
changes should be *last* in the merge-list (or if first, also be first to
be dropped if they cause merge errors downstream!).
That way, infrastructure changes that screw up others can only happen if
the maintainer actively works with the others to make sure it works even
before it would ever merge into Linux-next successfully.
That may sound odd, but it actually matches what I personally believe in:
we have more driver code and other "outlying" things than we have core
things, and most of our problems come from that - so we should prioritize
*those* things, not the "fundmantal core changes".
So how about making that the default situation: drivers and other outliers
merge first. If fundamental API changes happen, they merge last, and if
their maintainers can't make it in time in the merge window, they just get
dropped.
That sure as hell would put the pain on API changes solidly where it
belongs.
Linus
From: Al Viro <viro@ZenIV.linux.org.uk>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 19:42:18 UTC
Message-ID: <fa.JMzhKPk03DyNKlAqGka9bVU4eJc@ifi.uio.no>
On Tue, Feb 12, 2008 at 10:59:00AM -0800, Linus Torvalds wrote:
>
>
> On Tue, 12 Feb 2008, Linus Torvalds wrote:
> >
> > In other words, I'm perfectly happy to be an a*hole and tell people that I
> > simply won't merge things that cause undue API churn at all, and that were
> > not thought out sufficiently.
>
> .. btw: I'd need to know this in advance. I usually don't see the problem
> until it's too late.
We could simply decide that API changes affecting more than one subsystem
Must Be Serialized(tm). Explicitly. As in "any such change is posted
and discussed in advance, order of merges decided upon and we have merge
window for one decided set of API changes + fallout *ONLY*". With merge
window in question normally taking a few days.
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 00:54:43 UTC
Message-ID: <fa.IC9dUVprMCmwGxD9ARtqf79gXWg@ifi.uio.no>
On Tue, 12 Feb 2008, David Miller wrote:
> From: Linus Torvalds <torvalds@linux-foundation.org>
> Date: Tue, 12 Feb 2008 10:59:00 -0800 (PST)
>
> > That sure as hell would put the pain on API changes solidly where it
> > belongs.
>
> If a person does a driver API change and does all the work to sweep
> the entire tree updating all the drivers, doesn't it penalize that
> person a bit much to stick a new driver in front of that work?
If that API change doesn't conflict with the work that hundreds of other
people do, it's obviously not a problem whichever way it goes.
And if the API change *does* cause conflicts, then yes, the onus of fixing
those conflicts (again) goes to the person who changed the API. Everybody
else did everything right.
> People write code on top of infrastructure, both new and old, not the
> other way around. At least to me, that seems how the merging ought to
> work too.
You think that infrastructure is more important than outlying code. But
you do that only because you write the infrastructure, not because you
have any logical reason to think so.
The fact is, that "outlying code" is where we have all the bulk of the
code, and it's also where we have all those developers who aren't on the
"inside track". So we should help the outliers, not the core code.
And very fundamentally, API changes are to be discouraged. If we make them
harder to do and make people think twice (and occasionally say "not worth
it"), that sounds like a damn good thing to me.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 18:27:57 UTC
Message-ID: <fa.O90JgBQCp/TUnVaFgERSAuRdpos@ifi.uio.no>
On Tue, 12 Feb 2008, Greg KH wrote:
>
> I may be a bit defensive here, but I hope that all of the recent
> kobject/kset/driver core changes have been done with the thought of
> "what are we doing wrong".
.. but are we expecting it to be finished?
That's the point.
This whole "Linux-next" discussion so far has almost been predicated on
the whole assumption that this is an on-going concern. And it really
should NOT be.
If it's an on-going concern, we need to tackle *that* issue, not the issue
that cross-subsystem merges are hard. They simply seem to happen too much.
In other words, I'm not AT ALL interested in the merges we've already
done. That's over and done with, and we'll never ever do those merges
again. Who cares? I don't.
I'm purely and _only_ interested in the merges of the future. You don't
need to be defensive about the things that led up to this discussion, I'm
more hoping that we can aim at fixing the problem at the source, rather
than trying to work around it.
We simply shouldn't have all that many conflicts. We've had *way* too many
of them lately, and I think it's because people have felt it wasn't too
painful.
Put another way: back when we worked with just patches, we avoided renames
like hell, and we also tried to simply even re-architect the whole tree so
that you didn't have so many patch conflicts. One main reason as far as I
was concerned for things like per-directory Kconfig files and the whole
initcall() stuff was the fact that the old single Kconfig file and the old
crazy init/main.c file were total *nightmares* when it came to conflict
resolution.
So we split things up more, and we didn't do renames (or were very careful
about it). We avoided the things that caused pain.
I think we need to remember that: yes, we'll always have to have ways to
fix the pain that does happen, but even more importantly, we should strive
for models where it doesn't happen in the first place!
And simply avoiding cross-subsystem API changes unless there is a major
*MAJOR* reason for them is the obvious thing to do. Simply face the fact
that even in open source there are major reasons to stay with an old
interface even if it's not optimal.
We absolutely MUST NOT have the mindset that "cross-subsystem conflicts
happen all the time".
That was my point.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 21:37:45 UTC
Message-ID: <fa.ggMNyLiHQvj9t+pqn0dUeUOq3cw@ifi.uio.no>
On Tue, 12 Feb 2008, Greg KH wrote:
>
> Yes, I agree, there are lots of examples of this, but the overall
> majority are reviewed by 2 people at least (or sure as hell should be,
> maybe we need to bring into existance the "reviewed-by" marking to
> ensure this.)
Well, I don't really "review" any patches that come through Andrew. What I
do is:
- global search-and-replace Andrew's "acked-by:" with one that is both
him and me (that way I make sure that I _only_ sign off on patches that
he has signed off on!)
- look through all the commit *messages* (but not patches). This
sometimes involves also editing up grammar etc - some of those messages
just make me wince - but it also tends to include things like adding
commit one-liner information if only a git commit ID is mentioned etc.
- and only for areas that I feel competent in, I look at the patches too.
So, to take an example, when Andrew passes on uml patches that only touch
arch/um and include/asm-um, my sign-off does not mean *any* kind of review
at all. It's purely a sign that it's passed the sign-off requirements
properly.
When it comes to VM issues or other things, things are different, and I
actually review the patch (and occasionally send it back with "nope, I'm
not applying this"). But for stuff that comes through Andrew, that's
probably less than a quarter of the patches. And I don't mark the ones
I've reviewed specially in any way.
And I suspect I'm not at all alone in this. People simply have maintainers
they trust (and _need_ to trust in order to not become a bottleneck).
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 19:56:44 UTC
Message-ID: <fa.tF0tt1NIFfPFS+8Nr1rluZvMYeE@ifi.uio.no>
On Tue, 12 Feb 2008, Greg KH wrote:
>
> > That's the point.
>
> Not it isn't. To quote you a number of years ago:
> "Linux is evolution, not intelligent design"
Umm. Have you read a lot of books on evolution?
It doesn't sound like you have.
The fact is, evolution often does odd (and "suboptimal") things exactly
because it does incremental changes that DO NOT BREAK at any point.
The examples are legion. The mammalian eye has the retina "backwards",
with the blind spot appearing because the fundmanetal infrastructure (the
optical nerves) actually being in *front* of the light sensor and needing
a hole in the retina to get the information (and blood flow) to go to the
brain!
In other words, exactly *because* evolution requires "bisectability" (any
non-viable point in between is a dead end by definition) and does things
incrementally, it doesn't do big flips. It fixes the problems on an
incremental scale both when it comes to the details and when it comes to
both "details" (actual protein-coding genes that code directly for some
expression) and "infrastructure" (homeobox and non-coding genes).
So quite frankly, you're the "intelligent designer" here. You're the one
who seems to claim that we need those leaps of faith and wild jumps.
> Oh, it's been painful at times, but they are, overall, very rare.
No, overall, they have *not* been rare lately. We've had them all over.
And not just the one introduced by you.
> If you look at the rate of change we are currently running at, it's
> amazing that we do not get _more_ of these kinds of problems.
I don't think that's a valid argument.
Sure, we have lots of changes, but 99.9% of them have no cross-subsystem
effect what-so-ever.
> > And simply avoiding cross-subsystem API changes unless there is a major
> > *MAJOR* reason for them is the obvious thing to do. Simply face the fact
> > that even in open source there are major reasons to stay with an old
> > interface even if it's not optimal.
>
> I strongly disagree here. We lived with that kset/ktype crap for years,
> and I finally broke down and cleaned it up, simplifying things, removing
> code, making the kernel smaller, leaner, and easier for others to change
> and use in the future. With your statement, such a change should have
> never taken place as it what we had at the time was "not optimal", but
> good enough to live with.
You didn't listen at all.
I said that the threshold should be high, not that it should be
impossible. I also said that we should strive for making it unnecessary to
have the painful total synchronization points.
The fact is, we *have* been able to do things like this gradually and
well, without introducing breakage. Take the VM changes, for example:
those were pretty damn fundamental, where we've changed the calling
convention totally for fault handling.
But that thing was done without at any point really seriously breaking
code. It involved adding the new interface, and letting the old one live
in parallel.
The last remnant of the old "nopage()" interface still exists, but I think
right now it's only used by DRM.
Did it require the drivers to be updated? Yes. But it did NOT require the
total synchronization, because it still worked with the old interface.
> But they do happen about once or twice a kernel release, just by virtue
> of the way things need to happen.
And I violently disagree.
It should not be "once of twice a kernel release".
It should be "once or twice a year" that you hit a flag-day issue. The
rest of the time you should be able to do it without breakage. It's
doable. You just HAVEN'T EVEN TRIED, and seem to be actively against even
doing so.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Tue, 12 Feb 2008 20:33:36 UTC
Message-ID: <fa.iLIKP6VrU4asedw/gKFaznDACmY@ifi.uio.no>
On Tue, 12 Feb 2008, Russell King wrote:
>
> 3. rebase the branch on top of the conflicting change, throw out the
> patches which prove to be a problem and ask the original author of
> those patches to fix them up for the conflicting change. The result
> is a completely bisectable tree.
>
> (3) is the solution which I chose, and it worked _extremely_ well.
Don't get me wrong at all. Rebasing is fine for stuff you have committed
yourself (which I assume was the case here).
Rebasing is also a fine conflict resolution strategy when you try to
basically turn a "big and complex one-time merge conflict" into "multiple
much smaller ones by doing them one commit at a time".
But what rebasing is _not_ is a fine "default strategy", especially if
other people are depending on you.
> (3) is effectively what akpm does with his tree - when a patch conflicts
> with other changes, he throws the changes out and bangs peoples heads
> together to get a new set of patches generated which work together.
Right. And it's one of the fundamental differences between git and a patch
queue.
Patch queues are very flexible, but they simply don't scale. They don't
scale in history (ie you cannot sanely keep track of multiple queues as
they grow in the long run - you need a way to regularly "freeze" things
into a release tar-ball or something like that). But they also don't scale
with number of users - even just read-only ones.
The latter example is something we see in -mm right now. When people
report problems against -mm, it's usually an all-or-nothing thing
(so-and-so mm release doesn't work), but even when people sometimes bisect
to a specific point in mm, it's not "stable" in the sense that that point
may not make any sense in a subsequent -mm queue version.
And all of these issues are not about "-mm" per se, but are about patch
queues in general. And "rebase" turns a git repository effectively to a
patch queue too, with all the same downsides.
And I don't think patch queues are evil: I use git rebase all the time
myself (not for the kernel, but for git, where I'm not the top-level
thing). But I'd not rebase commits from other peoples trees: I rebase
commits that I applied as patches (ie the *author* may be somebody else,
but the committer is me!) or that I committed myself, and that I haven't
pushed out.
Note that difference between "committer" and "author". There's nothing
wrong with rebasing commits that are _authored_ by other people. But there
absolutely is something wrong rebasing commits that are _commmitted_ by
somebody else, unless you two are best buddies and know and trust each
other and are ok with each other messing up the commits (and use some
external method of serialization, so that you don't step on each others
toes).
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 00:50:59 UTC
Message-ID: <fa.JnpcI8eCDHCJ3EdwW0QnDmaYg9M@ifi.uio.no>
On Tue, 12 Feb 2008, Greg KH wrote:
>
> Perhaps you need to switch to using quilt. This is the main reason why
> I use it.
Btw, on that note: if some quilt user can send an "annotated history file"
of their quilt usage, it's something that git really can do, and I'll see
if I can merge (or rather, coax Junio to merge) the relevant part of stgit
to make it possible to just basically get "quilt behaviour" for the parts
of a git tree that you haven't pushed out yet.
A pure patch-stack will be faster at that thing than git would be (it's
simply easier to just track patches), but on the other hand, using git
would get some other advantages outside of the integration issue (eg the
cherry-pick thing really is a proper three-way merge, not just an "apply
patch", so it can do better).
It wasn't the original goal of git, but not only are really doing all the
series management anyway (that is largely what "rebase" is all about,
after all), but the git goals have obviously expanded over time too.
Linus
From: Theodore Tso <tytso@MIT.EDU>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 02:18:13 UTC
Message-ID: <fa.Du4ACaRbg9JHkAf8E3ul8VyS1FY@ifi.uio.no>
On Tue, Feb 12, 2008 at 04:49:46PM -0800, Linus Torvalds wrote:
> On Tue, 12 Feb 2008, Greg KH wrote:
> >
> > Perhaps you need to switch to using quilt. This is the main reason why
> > I use it.
>
> Btw, on that note: if some quilt user can send an "annotated history file"
> of their quilt usage, it's something that git really can do, and I'll see
> if I can merge (or rather, coax Junio to merge) the relevant part of stgit
> to make it possible to just basically get "quilt behaviour" for the parts
> of a git tree that you haven't pushed out yet.
So this is what I do for ext4 development. We maintain a quilt series
in git, which is located here at: http://repo.or.cz/w/ext4-patch-queue.git
A number of ext4 developers have write access to commit into that
tree, and we coordinate amongst ourselves and on
linux-ext4@vger.kernel.org. I tend to suck it into git using the
"guilt" package, and do periodic merge testing with a number of git
queues to detect potential merge conflicts. Not as many as James
does, but I may start doing more of that once I steal his scripts. :-)
The patch queue also gets automatic testing on a number different
platforms; for that reason the series files comments which version of
the kernel it was last based off of, so the ABAT system can know what
version of the kernel to use as the base of the quilt series.
I do a fair amount of QA, including copy editing and in some cases
rewriting the patch descriptions (which are often pretty vile, due to
a number of the ext4 developers not being native English speakers; not
their fault, but more than once I've had no idea what the patch
description is trying to say until I read through the patch very
closely, which is also good for me to do from a code QA point of view :-).
Periodically, the patch queue gets pushed into the ext4.git tree and
as a patch series on ftp.kernel.org.
I've never been very happy with stgit because of past experiences
which has scarred me when it got get confused and lost my entire patch
series (this was before git reflogs, so recovery was.... interesting).
There's always been something deeply comforting about having the ASCII
patch series since it's easy to back it up and know you're in no
danger of losing everything in case of a bug. Also, having the patch
series stored in ASCII as a quilt stack means that we can store the
quilt stack itself in git, and with repo.or.cz it allows us to have
multiple write access to the shared quilt stack, while still giving us
the off-line access advantages of git. (Yes, I've spent plane rides
rewriting patch descriptions. :-)
The other advantage of storing the patch stack as a an ASCII quilt
series is we have a history of changes of the patches, which we don't
necessarily have if you just use stgit to rewrite the patch. So we
have the best of both worlds; what gets checked into Linus's tree is a
clean patch series, but we keep some history of different versions of
a patch over time in the ext4-patch-queue git repository. (I wish we
had better changelog comments there too, but I'll take what I can
get.)
- Ted
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 00:45:49 UTC
Message-ID: <fa.glojDm0g6N8ZuMLqSeaPpM0ue1Y@ifi.uio.no>
On Tue, 12 Feb 2008, David Miller wrote:
>
> At 1500 changesets, a merge conflict shows up about once
> every day or two as 2.6.N nears it's release into final
> as bug fixes trickle in.
>
> I find using GIT to fixup merge errors on a tree of that
> scale to be really painful. And it only fixes up the final
> result in a merge changeset.
Heh. I've had the reverse situation: "git rebase" often results in *more*
conflicts than "git merge" (ie "pull").
But one issue is also that when conflicts happen, different people are
used to different things. I'm particularly used to merge-type conflicts,
and in fact there's some fairly advanced support in git for helping
resolve them that people who *don't* do merge-level conflict resolution
may not even be aware of.
In particular, if you want to try it, do something that conflicts and then
do
gitk --merge
to see what the conflict is all about. That is just fancy shorthand for
gitk HEAD...MERGE_HEAD -- <list of conflicting files>
so what it does is to show only the relevant history (the three dots means
that it's a _symmetric_ set difference from HEAD to MERGE_HEAD) for the
merge, and only for the particular files that had conflicts!
This often means that even when you merge a thousand commits (or the thing
you merge *into* has thousands of commits since the merge base, which is
the common case for me), you actually only see a couple of commits - only
the ones that actually modified the conflicting files!
(If you have many files that conflict, you can further narrow it down to
just one at a time by explicitly listing the file/directory you want to
work on, ie do "gitk --merge <pathname-here>").
> Let me give you a good example, just yesterday I had to rebase
> my net-2.6 tree a few times. It's because I put a patch in there
> that the person said was OK but it broke the build. There is
> zero reason for me to push that tree to Linus with the bad
> commit and the revert, it's just noise and makes the tree harder
> for people to review.
Actually, better than rebase in that situation is to just remove the bad
commit. Yes, you'd use "rebase" for it, but you'd use it not to move the
whole series to a newer place, you'd use it just to rebase the commits
*after* the commit you remove.
This is something where I actually think git could and should do better:
git has the capability to act as more of a "quilt replacement", but
because it wasn't part of the original design, we never actually exposed
the simple queue management commands to do this (stgit does things like
that, though).
So if you haven't pushed out, right now you'd have to do this stupid
thing:
[ start (and activate) a 'fixup' branch at X ]
git checkout -b fixup X
[ edit edit edit to fix it up ]
..
[ commit the fixed state ]
git commit --amend
[ go back to the old broken state ]
git checkout master
[ now, rebase 'master' on top of the fix ]
git rebase fixup
[ ok, done, forget the fixup branch ]
git branch -d fixup
and I don't discourage this kind of behaviour at all, but it is only good
iff:
- you have not pushed things out (obviously), so nobody ever even notices
that you've fixed up stuff
- you haven't pulled anything from outside (so you aren't trying to
rebase other peoples commits).
If you *really* want to try to do this even across merges you've done,
there is fancy a "--preserve-merges" thing that you can try to use
(needs "-i" to work, but "-i" is often cool for other reasons too!)
Basically, what I'm trying to say is that "git rebase" can be used in
fancy ways to do things that people outside your repository will never
even *know* were done. It's only when outsiders can see the effects of git
rebase that you're in trouble!
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 01:42:16 UTC
Message-ID: <fa.C5IXM8SF+98RrpTneVRavo/uFYQ@ifi.uio.no>
On Tue, 12 Feb 2008, David Miller wrote:
>
> But as soon as I've applied any patches to my tree I've "pushed out".
> So this scheme doesn't work for me. The first thing I do when I have
> changes to apply is clone a tree locally and on master.kernel.org,
> then I apply that first patch locally and push it out to master.
I actually suggest you literally delay your push-out.
I don't generally delay things by a lot, but I tend to try to at least do
a compile in between pushing out - and even if I've pulled something in
between the thing that broke, I'll just "git reset --hard" to a working
state if something broke, and just re-pull instead of even trying to
rebase or anything like that.
(IOW, I often find it much easier to just start over and re-do than
actually doing a rebase).
I don't do it all the time, by any means, but there's really no huge
reason to push out all the time. And that's doubly true for subsystem
maintainers. Quite often, the right thing to do is to only push out when
you are ready to do the "please pull" message.
> What would be really cool is if you could do the rebase thing, push
> that to a remote tree you were already pushing into and others could
> pull from that and all the right things happen.
It would also be really cool if Claudia Schiffer had decided that hiding
under my desk is a good idea.
IOW, you really haven't though that through. That is how TLA and darcs
worked, and it's a total disaster.
Trust me, you don't know how good you have it.
Linus
From: Linus Torvalds <torvalds@linux-foundation.org>
Newsgroups: fa.linux.kernel
Subject: Re: Announce: Linux-next (Or Andrew's dream :-))
Date: Wed, 13 Feb 2008 02:36:09 UTC
Message-ID: <fa.xfKJNMzGfeSGbxC5KNaMjgCqyqs@ifi.uio.