[coyotos-dev] Can we really think at a new OS design nowadays ?

[Jonathan S. Shapiro]

Oh this is *really* annoying -- either I didn't get the earlier two
posts (by Guillaume, Benno, and Neal), or I deleted them by mistake.
Bother. Fortunately all of them are in the archives.

I am unclear about Benno's note. This didn't end up in the coyotos-dev
archives. Did it go to some other list and get cross-posted here? If so,
could someone reply to this and post a link to it?



> On Sat Aug 05, 2006 at 19:50:05 +0200, Guillaume FORTAINE wrote:
> >
> >Abstraction will be the solution,

Yes, but there is a longer discussion here. I will take this up in a
separate thread.

> Doesn't the research of Engler show us that abstraction in operating
> system design is bad?

I assume that you are referring to the Exokernel work -- Engler has done
a lot of good work in several fields. In my opinion, Exokernel was an
important research exploration, because it showed how *not* to advance
the state of the art in secure and robust systems.

The most important question to ask is: what are the primary design
objectives for the system? Historically, they have been performance and
scalability (in that order). But this focus has changed. Two things are
now true:

  1. CPU's now spend the vast majority of their time idle because
     memory is not fast enough.
  2. This delay is due to *data* latency. Instruction fetch delays
     do exist, but they arise from changes in control flow.

What is interesting about this is that application-level performance
issues are driven much more by data organization than by choice of
language or operating system. Another implication is that the cost of
abstraction (which was never as large as people claimed) is less and
less important, while the benefits of abstraction are as important as
ever.

What is really interesting (to me) about Exokernel is that *because* all
of the abstraction boundaries were removed, it is impossible to build
engineerable systems on top of it. Exokernel is a system where 5-10
experts can build something that runs like a bat out of hell. But the
thing they are building better be pretty small and better not need any
maintenance, because the total world pool of experts capable of building
that software may be as large as 12. That is: exokernel doesn't support
the *human* requirements of large scale engineering. One of those
requirements is abstraction boundaries.

EROS and Coyotos achieve 90%+ (closer to 95%) of Exokernel performance,
but we preserve the abstraction boundaries. I need 10 experts to build
the core system, but after that it is a system that normal mortals can
develop on.

And before you say that giving up 10% is fatal, stop to consider that
Windows XP gives up 50% and NOBODY CARES.

So: in my opinion the engineering priorities today are engineerability
(by which I mean both maintainability and scalability of human teams),
robustness, performance, and security (in that order). I personally
would put security higher in the list, but the customers do not.

On the subject of eliminating abstraction, Neal wrote (in reference to an Exokernel paper):
> > I draw from this that eliminating abstractions simply to eliminate
> > abstractions misses an important point: some polices introduced via
> > mechanism have negligible negative impact on performance and some such
> > policies actually have positive impact (e.g. code simplification).
> > This observation supports Liedtke in his challenge of the Exokernel
> > architecture in "On u-Kernel Construction:"

I agree with him entirely.

On Mon, 2006-08-07 at 03:48 -0400, Haplo wrote:
> Coyotos ... is just an optimized and improved version of  
> keykos. .. Linux hasn't really changed much in years... 
> Commercial OS vendors [... are ...]
> forced to reach back in time and pick out an OS/KOS that sucks less  
> than their current design enough so to allow them to continue their  
> trash pattern.

I'm happy with this up to the part about "trash patterns". The sad fact
is that commercial behavior isn't stupid. It is a rational response to
the reality of market conditions. As engineers and designers it is
disgusting. Sadly, disgusting is what wins in the market. Richard
Gabriel, who was one of the key people at Lucent, once wrote a pretty
good essay on this:

  http://www.dreamsongs.com/WIB.html

[There is more to that thread, so if you are interested, see also

  http://www.dreamsongs.com/WorseIsBetter.html]

I don't agree with a lot of what Richard wrote. I think that the
explanation of what happened is quite different from the one that he
offers, and I'm about to write an essay about it, but the thing to note
is that (1) this problem has been around for a long time and (2) "the
right thing" probably isn't right when it is carried too far -- you need
to know where and how to compromise.

> ...Sadly, Coyotos is unlikely to find it's way into  
> a commercial OS any time soon, simply because of the amount of work  
> required for that and the fact that nobody outside of the most elite  
> and well educated users care.

I think you mean "desktop OS". There is a big difference between "not in
a desktop OS" and "not in a commercial OS". My company is currently
waiting on two contract decision outcomes. If *either* contract comes
through, Coyotos will deploy commercially much quicker than you might
expect.


shap