[coyotos-dev] Re-thinking FCRBs

[Marcus Brinkmann]

At Tue, 20 Feb 2007 12:33:14 -0500,
"Jonathan S. Shapiro" <shap at eros-os.com> wrote:
> 
> On Tue, 2007-02-20 at 15:23 +0100, Marcus Brinkmann wrote:
> > At Thu, 15 Feb 2007 14:15:19 -0500,
> > "Jonathan S. Shapiro" <shap at eros-os.com> wrote:
> > > ** So what about asynchronous *sends* ??
> > > 
> > > If we need a mechanism for asynchronous completion, what about a
> > > mechanism for asynchronous sends?
> > > 
> > > My answer is "no". If the use of asynchronous receive is properly
> > > engineered, sends will not block. If they do you have a systemic issue
> > > and my sense is that you want to see it rather than obscure it. I could
> > > be completely wrong on this, and I'ld appreciate input on this point.
> > 
> > I have only come up with one use case for asynchronous send.  It's the
> > case where one does not want to allocate a stall queue in the server
> > for "blocking" RPCs like reading from an empty pipe (which is a
> > read-only operation and presumably should not allocate server
> > resources).  In principle, it is not clear why the server should need
> > to allocate a queue for such client operations, as there already is a
> > kernel queue that could be used for this purpose. However, pushing a
> > client back to a send() operation on an end point that the server is
> > not listening on (until the operation can potentially be completed)
> > makes send() a blocking operation.
> 
> I'm confused, because there is no mechanism for a server to allocate a
> stall queue at all.

Maybe I should not have used the term "stall queue".
 
> In the read on empty pipe case, where server does not listen actively,
> sender will block.
> 
> The motivation for asynchronous send in this case would be to allow the
> *sender* to avoid blocking, or to enqueue multiple operations.
> 
> My feeling is: if the server is correctly implemented for its
> requirements this becomes a non-issue (because the server will arrange
> to be listening). If it is not, you have bigger problems than
> asynchronous send can solve for you.

The sender will be listening, but it won't be able to fulfill the
request, because the pipe is empty.  What happens next?

1) The server puts the request (or rather: a continuation for it) on a
   queue (that's what I meant by "stall queue") and reply later.  This
   requires a list element to be allocated, which consumes server
   resources.

or

2) The server arranges for a new capability to be made available to
   the client directly or indirectly, on which the request is made
   (again) explicitely or implicitely, and the server will not listen
   on this capability until data becomes available, for example using
   the "push-back" algorithm that I developed in Jan 2006 when I was
   at your place.  I think there is a similar concept in the EROS
   design (where it is done implicitely in the kernel), and there also
   is something remotely similar in L4 X.2 draft specification
   ("propagation", but there the propagator has to enter a send phase,
   and the calling client state is just modified "as if", so it's not
   the same thing).

In the first case, the client is relieved but the server is burdened
by allocations proportionally to the number of outstanding blocking
client requests.

In the second case the server only needs to allocate resources
proportional to the objects it serves, not to the number of
outstanding client requests.  But the price is that clients will block
in the send phase, which means that you can not use this method to
support select() from multiple file descriptors, for example.

The issue here is how to support/implement asynchronous blocking
operations.  One can also try to argue the case away by doing one or
more of the following:

1) One can try to restrict the overall system design to cases where
   such servers only serve a single client, and run on client provided
   resources totally.

or

2) One can simply accept that servers do some amount of memory
   allocation even on read-only requests, crossing fingers.

Thanks,
Marcus