[coyotos-dev] Why we have FCRBs

[Jonathan S. Shapiro]

FCRBs really only have one motivating case. Two applications, but both
are really demonstrating the same issue: sometimes you need a mechanism
for preemption.

If you are going to do preemption, activations are pretty clearly the
right way to do it.

Once I was doing preemption anyway, I decided to look at whether
messaging and activations could be integrated. In the absence of
capability arguments I think they can, but in the presence of capability
arguments the need to cross the kernel boundary multiple times
eliminates nearly all of the advantage.

The motivating case is illustrated by ring buffers: you have one
application (e.g. an ethernet driver) that is inserting state into a
ring buffer. You have a second application (e.g. a TCP/IP stack) that is
*reading* from the ring buffer. The ethernet can drop packets, but it
can never be blocked by the TCP/IP stack.

Inserting the packet into the ring buffer (or dropping it) is not the
problem. The problem is that *after* the packet is inserted you need to
notify the receiver that there is more work to be done. It is tempting
to think that this could be done with a shared memory word, but there
are update races in such a design. The race case is the following:

   1. Enet delivers a packet
   2. TCP/IP enters "receive next message" state
   3. Enet updates shared word.
   4. TCP/IP never notices that new packets are available, remains
      blocked.

The essential problem is the race between the receiver blocking and the
sender notifying. This can *only* be solved with a system call.

It is not enough for the ethernet to send a "will not wait" message.
That case can still have the following modified race:

   1. Enet delivers a packet
   2. TCP/IP checks word, finds no data, decides to sleep.
   3. Enet updates shared word.
   4. Enet sends will-not-wait message, which is dropped because
      TCP/IP stack is not yet available.
   5. TCP/IP never notices that new packets are available,
      remains blocked.

The simplest solution would be to add a modifier to the receive
operation, saying "return immediately if memory word X is non-zero".
This would be very simple to do, but it does not generalize to more than
one sender unless all senders are fully trusted. If a shared update word
is used for all senders, a hostile sender could disrupt a good sender by
zeroing the word.

A solution that we considered in EROS was to introduce an "notification
word" into the process state. If the notification word is non-zero when
a process becomes available, a notification message is immediately
synthesized by the kernel and delivered to the application. I will
describe in my next note the problems with the notification word idea.

-- 
Jonathan S. Shapiro, Ph.D.
Managing Director
The EROS Group, LLC
+1 443 927 1719 x5100