[coyotos-dev] Re-thinking FCRBs

[Marcus Brinkmann]

At Thu, 22 Feb 2007 06:38:03 -0500,
"Jonathan S. Shapiro" <shap at eros-os.com> wrote:
> 
> On Wed, 2007-02-21 at 00:17 +0100, Marcus Brinkmann wrote:
> > The sender will be listening, but it won't be able to fulfill the
> > request, because the pipe is empty.  What happens next?
> 
> I'm still confused about this, because I'm not clear what you mean by
> "sender" here. If "sender" is *writing* to the pipe, an empty pipe is
> not an issue. I *think* you mean that "sender" is a client of the pipe
> server that is performing a pipe read operation?

D'oh.  I meant to say "server" here.  I did refer to a client of the
pipe server performing a pipe read operation though, yes.
 
> > 1) The server puts the request (or rather: a continuation for it) on a
> >    queue (that's what I meant by "stall queue") and reply later.  This
> >    requires a list element to be allocated, which consumes server
> >    resources.
> 
> This is possible, but as of Coyotos 0.4 no list element is required if
> the SQ bit is used. Assume that the client is willing to block (NB=0).
> Assume further that there are two endpoints EP.read and EP.write. When
> there is room in the pipe buffer, the server receives actively on
> EP.write. When the pipe is full, the server receives actively on
> EP.read. Both endpoints have the SQ bit set.
>
> When the server is not receiving on EP.read, the reading client will
> block within the kernel. This puts it on a kernel-maintained stall
> queue. The link pointers are already built in to the process structure
> of the client. The pipe server will still receive an activation telling
> it that there is a pending request on EP.read.

Yes, this is how the protocol I developed in Jan 2006 worked, although
I had three endpoints (a third endpoint that serves for any other
control requests and also to retrieve the EP.write and EP.read
capabilities so they do not have to be communicated as a set).
 
> This design works, but from the "how to emulate unix" perspective, there
> are two problems with this design:
> 
>   1. Multiple capabilities are required for a single UNIX fd.
>   2. It doesn't address ioctl() operations well, because these get
>      blocked along with the reads and writes.

The "control" capability addresses these issues.
 
> In order to solve these problems the pipe server will need to accept the
> requests, and then you do need to enqueue them. At that point, one of
> three things needs to happen:

I don't think the issue is the control operations, because those can
happen on yet another endpoint.  The real issue is that we can not
allow the client to block without accepting the requirement for the
client to create one real kernel thread for each file descriptor in a
select (ie: sensing 0 byte read/write) operation.

>   1. The server keeps a list (requires allocating storage). This is your
>      case [1].
> 
>   2. The server advises the client to retry at a different capability
>      in order to block the client. This is your case [2].
> 
>   3. Some operation is available that allows the server to enqueue the
>      client elsewhere in such a way that it's message will be re-sent.
> 
>      This is the EROS 'retry' operation. It's very hard to use this
>      correctly and it is not virtualizable.

I suppose you refer to interposition where the reply from the upstream
service should not go directly to the client.  Interesting test case,
it's definitely a trouble-maker and seems to suggest that (1) above is
the generic answer here.  It seems to be related to the difference
between tail-recursion, which can be turned into an iteration in
constant space, and real recursion which requires linear space with
depth of recursion.

Thanks,
Marcus