[coyotos-dev] Activations

[Jonathan S. Shapiro]

On Wed, 2007-05-23 at 08:51 -0400, Christopher Nelson wrote:
>         > If you have to have additional storage for each activation
>         handler
>         > per-thread, then it seems like the only additional overhead
>         for the 
>         > extern handlers it the address space mapping.
>         
>         I'm very confused as to why you keep coming back to this. In
>         the
>         external case, the handler is using *identically* the same
>         address space
>         as the faulting process. 
> 
> Ah hah.  Thus part of my confusion.  It was my impression that an
> external fault handler was a classic heavy process with it's own
> private address space.  You and I are probably using process and
> thread in almost but not quite compatible ways.  Regardless of how the
> instruction streams get scheduled, if they share the same address
> space I tend to think of them as threads, while threads that don't
> share the same address space are processes.  

I am glad we have arrived here, because it is helpful to understand root
causes in these confusions -- they help me describe better in the
future. Indeed, I suspect that the thread/process term problem is the
root cause.

Let me first distinguish between user-mode threads (uthreads) and
kernel-mode threads (kthreads). The kernel is completely ignorant of
user-mode threads. There are many implementations, but uthreads are
ultimately implemented on top of kthreads.

The kernel is aware of kthreads and is responsible for scheduling them.
In the present conversation we have been discussing kthreads.

In most operating systems, kthreads were retrofitted onto an existing
process abstraction. In these systems, the original process abstraction
had exactly one kthread and one address space. When kthreads were
grafted in, the Process abstraction morphed into the resource container,
and the kthread became the unit of kernel-scheduled execution.

But if you look more closely, you will conclude that the real problem
was something else. The real problem was that address spaces were not a
first-class abstraction. That is: you could not have an address space
existing independent of a Process.

In EROS and Coyotos, address spaces are first-class. A Process names
(via a capability) its address space. Two processes can name the same
(i.e. identically the same) address space. If they do, they exhibit the
behavior of kthreads. Because of this, we do not distinguish between
processes and kthreads.

The "heaviness" of a classic "heavy process" comes from the fact that
its various resource bundles are not first class. When you look at the
clone() system call in Linux, you will conclude that it created a
variable-weight process abstraction. This is very close to what
EROS/Coyotos did from the beginning.


> ----Thinking about what you wrote, would it be generally fair to say
> that an external fault handler is only external to the execution
> context of the instruction stream it is handling faults for, whereas
> an activation handler merges into and splits from the same context as
> the instruction stream it handles faults for? 

I am not certain how you are using the term "execution context" here,
and since the definition of execution context lies at the root of our
confusion, I hesitate to answer in a way that will mislead us further.

I *think* that your answer oversimplifies matters. The problem is the
assumption that execution contexts are of uniform weight. I prefer to
think of an execution context as a tuple of the form

   (regSet, aSpace, fltHandler, ...)

In abstract, any element of this tuple can be altered independent of the
others, and any element (excluding the register sets) can be identically
shared by more than one execution context. The difference between a
classic heavyweight process and an EROS/Coyotos process is that in the
classic heavyweight process many of the resource bindings of the
execution context are "unboxed" (i.e. structurally a part of the
execution context). This prevents multiple contexts from sharing these
resources, and increases the cost of context creation (leading to the
"heaviness" in "heavyweight processes"). In EROS/Coyotos most of them
are by-reference and first-class. This allows them to be shared by
multiple execution contexts and lowers the cost of context fabrication,
giving a more continuous range of choices between lighter and heavier
context creations according to the purposes the developer is trying to
achieve.

The reason that the register set cannot sensibly be "by reference" has
to do with the hardware structure. The hardware load-store model assumes
that threads have independent register sets. If we tried to change that,
we would have to re-define load and store, and we wouldn't be in the OS
business anymore. Hmm. To be excruciatingly precise, it *can* be "by
reference", but it must be 1:1 with the execution context, so there
isn't much point to splitting it out.

In activations, we have

  (threadRegSet, aSpace, ...)  <- the thread context
  (actRegSet, aSpace, ...)     <- the activation context

That is, only the register set changes. Actually, this is not true. The
register set switch is implemented in a very clever way such that the
register set *contents* get saved but the kernel register set *resource*
is re-used. It is fairly accurate to view this as a kernel-supported
continuation switch. Normally, we treat the (PC,SP) pair as part of the
register set. To explain continuations accurately we need to split them
out, and then we will have:

  (regSet, thrdContinuation, aSpace, ...)  <- the thread context
  (regSet, actContinuation,  aSpace, ...)  <- the handler context

That is, the mechanism is fast because it consists entirely of a
kernel-implemented continuation switch, which is really (in minimal
form) just an update to three registers (PC, SP, scratch).

The difference between this an an external handler sharing the same
address space may be described by:

  (thrdRegSet, thrdContinuation, aSpace, ...)  <- the thread context
  (extRegSet,  extContinuation,  aSpace, ...)  <- the handler context

This seems big because the introduction of a new "regSet" requires a new
kernel process object, but for reasons we have already discussed the
comparative space cost actually leans in favor of the kernel process
object. This is offset by the added latency of the additional U->S->U
crossing.

Either those tuples now have things really confused or they helped. Let
me know which. :-)

Part of my reason for hesitatingMy main reason for hesitating is that
this is not the *only* possible configuration for an external fault
handler. It is also possible to configure one as:

  (thrdRegSet, thrdContinuation, thrdSpace, ...)  <- the thread context
  (extRegSet,  extContinuation,  extSpace, ...)  <- the handler context

That is: a fault handler that runs in a completely independent address
space. Such a handler shares no elements in common with the managed
thread of control, and it is truly a "heavyweight" process in the sense
that you mean.



Does any of this help?