From shap at eros-os.org  Thu Dec  1 15:11:46 2011
From: shap at eros-os.org (Jonathan S. Shapiro)
Date: Thu, 1 Dec 2011 15:11:46 -0800
Subject: [coyotos-dev] Re-examining my concerns with L4 "map"
Message-ID: <CAAP=3QNd9dxsuYYK9QsijbTcWsvBcS-H5EfU7GG3910fhv2p7w@mail.gmail.com>

Historically, I have raised three objections to the L4 "map" operation:


   1. Sending a mapping induces hierarchy, which means that the receiver
   loses their mapping when the sender exits. I believe that a "map
   co-equally" operation is desired. I also believe that the existing L4 data
   structures could probably be adapted to support it.
   2. I have claimed that hierarchical unmap is the wrong conceptual model,
   stating that I strongly prefer the "revoke" model.
   3. Support for "unmap" requires that the "map" operation create mapping
   database entries that are later referenced by "unmap". These are
   allocated from kernel storage, which can be exhausted. There is no
   architectural bound on the length of a chain of maps.

The first issue is noise. It may require a new design of the mapping
database, but it doesn't present any great conceptual leap of faith.

The second issue may be nothing more than bias on my part.

The third issue has been my main concern. What I want to do here is compare
it to what happens in Coyotos and try to assess clearly whether the concern
is warranted.

In the map operation, the potential concern is in two parts:

   1. The operation can fail due to storage exhaustion. Exhaustion is an
   inherent problem in any in-memory system. The concern here is that the
   allocation happens in an implicit way within the call. If a map operation
   allocates *exactly one* or *at most one* mapping database entry, then I
   think we're fine, and the problem more or less reduces to a quota
   management issue. One which I don't know how to solve off the top of my
   head, but I haven't put any energy into solving it yet.
   2. If "map shared" is introduced, there is a question of whose pool of
   storage (sender or recipient) provides the mapping database entries. I'm
   not sure what the "right" answer is here.

Coyotos (and EROS, and ...) takes a different approach. We allocate either
per-object or per-capability storage (depending on which implementation you
are examining) that allows us to revoke all outstanding capabilities. We
don't do the revocation hierarchically at the kernel level. As a matter of
style, we tend to introduce capability indirection (through some variant of
forwarding node) in those cases where a later hierarchical revocation is
desired.

So what are the real differences here?

   1. Allocation of a hierarchy node in Coyotos is explicit, and is subject
   to quota restrictions. Allocation of mapping database nodes is implicit,
   and (so far as I know) is *not* subject to resource limits.
   2. L4 is able to make very efficient use of the hardware mapping
   structures as "normative" storage. My impression is that the total storage
   per capability in Fiasco.OC (i.e. the size of the capability plus metadata
   plus the mapping database entry) is *significantly* less, on average,
   than the total size in Coyotos.
   3. Because the software mapping structures (GPTs) in Coyotos/EROS are
   normative, the hardware structures and the dependency tables can be
   reconstructed on demand. This means that they can be managed from a fixed
   kernel resource pool that is dedicated to mapping management. The *
   problem* with this story is that it breaks down in hard real-time
   systems; in many cases the *need* to reconstruct the hardware mapping
   structures means that you have already exceeded the tolerable variance for
   the application in question.
   4. When mappings are hierarchical, page fault recovery necessarily
   involves messages between an earlier sender and the currently faulting
   receiver. There is a problem with this:
      - If the most recently preceding sender has "dropped out", then the
      receiver may not have a capability to the process that can handle the
      fault. Who gets contacted in this case, and how? Current L4 environments
      typically push this problem to some form of "area manager". In order for
      the area manager to resolve the problem, the sender must have previously
      established a record of intent to send. This has the feel [to me] of a
      duplication of mechanism. Very possibly because I do not
understand how the
      pattern works correctly.
      - This turns out to be a common case, because a sender may have a
      "hole" in the middle of mapped region that is inherited by the receiver.
      When referenced, those holes need to be populated by somebody, which
      invokes the out-of-band area recovery mechanism. I should be clear,
      however, that I keep looking at this pattern expecting to see a
first-class
      notion of memory object somewhere, which isn't the L4 way of
doing things.
      This may very well be a case where my expectations are stopping me from
      seeing why the L4 way is a reasonable approach.
   5. Coyotos "revoke" is constant-time. L4 "unmap" latency is a function
   of the number of mappings. I *suspect* that there are ways to re-arrange
   the L4 unmap operation such that the unmap latency gets rearranged into
   constant-sized chunks (or near enough).

When these issues are looked at as a whole, two points jump out at me:

   1. Allocating resource without a limit on allocation is a bad idea. It's
   clearly fixable, and I know that some versions of L4 have looked at various
   forms of "resource heap" by which applications are made to supply the
   required backing store for this resource and others. I'm not up to date on
   the status of that work. Once a resource limit mechanism is present, the
   absence of a depth limit ceases to be an issue, and in fact becomes an
   advantage.
   2. The amount of memory required in Coyotos to represent mappings is
   significant, and it's likely a problem in small, in-memory system scenarios.

A third point - or more accurately, a question - is whether some notion of
first-class memory objects is desirable. I definitely tend to lean in that
direction, but that doesn't make it the right place to start.

So much for collecting my thoughts. Let's see if anything falls out of the
trees in the back of my head over the next few days on this.


Jonathan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.coyotos.org/pipermail/coyotos-dev/attachments/20111201/8bab5edb/attachment.html