[coyotos-dev] Major change to IPC specification

[Jonathan S. Shapiro]

For the last several days I have been working on the Coyotos IPC
implementation. The conclusion is that the current specification is too
@)&%@##))&@$ complicated. The problem, at its essence, is that there are
too many conditionals in the path, and too many places where things have
to be checked in the cross-process case.

This violates a basic maxim:

  Keep simple things simple, keep common things fast, keep complex
  things possible.

Here are some examples of problems in the current specification:

 1. The interpretation of typed items requires far too much
    processing (too many data-dependent branches).
 2. The scatter/gather functionality dominates the interface.
    Strings are rare in any form, and scatter/gather almost
    never happens if there are a sufficient number of direct
    data words.
 3. The interpretation of capitem_t requires excessive case
    analysis.
 4. The UPCB turned out to be a total disaster.

Collectively, all of this means that any hope of sensible
registerization in the current path is pretty much Bitched All To Hell
(TM). That is: we're taking a bath on this thing. :-) It reminds me of
the current ads for Comcast internet (for those of you in Europe:
comcast is a cable television supplier here) where two turtles are
talking about how much they like DSL because they really prefer for
things to be slow.

>From a prioritization standpoint, my views are:

  1. The "all data words in registers, 0 or 1 capabilities"
     case is the most important one.
  2. The "all capabilities in registers" case comes next.
  3. The "single string" case is the third priority.
  4. The "capabilities from/to memory" case is important
     because the runtime system needs to be able to implement
     a capability stack. That said, this case must not inject
     too much complexity into the main invocation path (which it is
     currently doing).
  5. Scatter/Gather is one of those seductive ideas that
     should be deferred for now. We do know of some use cases,
     but none that are urgent today. A key point is that
     effective use of scatter/gather requires some extensions
     to the CapIDL tools, and that isn't on the short list
     of priorities.

If we did not have to cope with the stunningly elegant artifact that is
IA-32, I would declare that [4] was not critical either. It can be done
with extra same-process syscalls, and on all other architectures those
calls are very cheap. Unfortunately, IA-32 is something we all have to
live with in our lives, and even with the new SYSCALL/SYSENTER
instructions, the cost of the extra User/Supervisor crossings is
considerable.

The best solution we have come up with is a "boxcar" technique. To give
credit where it is due, either Neal Walfield or Marcus Brinkmann
suggested that I consider this a long time ago. At the time, It thought
that it would not be called for, and we did not go into specifics. I was
wrong. [For those of you keeping score, that marks the third time that I
have ever been wrong, and this sentence marks the fourth time... :-)]

Observe that if you are taking any sensibly small number of arguments
from the stack, it really doesn't matter if they begin at offset 0 or
offset N from the stack pointer. The offset must be word aligned, and
you don't want to run off of the single-byte offset encoding if you can
avoid it. What *does* matter is whether the parts of the stack you touch
are cache line friendly.

Given this, we are going to completely revise the invocation
specification. An invocation now consists of FOUR phases:

   1. Load Multiple Capabilities. This transfers four capability
      arguments from memory **or registers** to sequential capability
      registers. The first target register number must be a multiple of
      four (CR4, CR8, ...).

      If CR0 is specified as the first target register, this phase
      is skipped.

   2. The send phase. For capabilities, this only implements
      register to register transfers. CR0 is hardwired to Null
      when used as a source, and is not modified when used as a
      target.

      If the send phase specifies that the last data word to be
      transferred is zero (which is the control word), the
      send phase is skipped.

   3. The receive phase, which may be skipped by setting the SR
      bit of the control word to 1.

   4. Store Multiple Capabilities. This transfers four capability
      arguments from sequential registers to memory **or registers**.
      The first source register number must be a multiple of 4
      (CR4, CR8, ...).

      If CR0 is specified as the first source register, this
      phase is skipped.

The stack frame for the system call contains all of the locations
necessary to describe all four phases. In many cases there will be
chunks of this frame that are completely unreferenced by the actual
system call path.

Somewhat surprisingly, encoding this whole thing requires the same
number of words as the previous specification.

There is a corner case: note that if the invoked capability and all four
capability arguments are sourced from memory, the LMC phase is
insufficient to fetch all four capabilities. In this case, a separate
system call to load the fifth capability into a capability register is
required. We think this is probably not a significant concern. If it
turns out to be important, we'll add another phase.



Jonathan