[coyotos-dev] Need for revised invocation

[Charles Landau]

At 11:53 AM -0500 2/6/06, Jonathan S. Shapiro wrote:
>From the perspective of mechanism, kernel-implemented preemption and
>user-implemented preemption are essentially similar. In the kernel
>thread case, the thread is halted by the kernel and the kernel resumes
>some other thread. Similarly, in the user-mode thread case, the
>activation handler interrupts one user-mode thread and may elect to
>resume a second one. In the kernel case, the kernel is aware of
>preemption but the application need not be. In the activation handler
>case, the activation handler is aware of preemption but the user-mode
>"worker" threads need not be.

If the "worker" threads are not to be aware of preemption, they need 
the full state of a kernel thread, so you haven't gained much over 
kernel threads. Conversely, if you are willing to expose scheduling 
artifacts, you can implement less-than-full threads (for example, 
without floating-point state) either as user threads or kernel 
threads. (When loading less-than-full kernel threads, you may need to 
clear the unsaved state to avoid leaking secrets.)

>Because the mechanisms are essentially identical, I do not believe that
>the user-mode case should be viewed as qualitatively different from the
>kernel-mode case. Concurrency is certainly tricky and error prone, but
>this is orthogonal to whether the concurrency is managed at kernel level
>or user level. It seems to me that any argument concerning preemption
>must apply co-equally in both cases.

In the case of kernel-managed threads, there is a very clear line 
between that which is aware of the details of CPU scheduling (the 
kernel) and that which is not aware (all user code). In the case of 
user-managed threads, there needs to be an equally clear line. At the 
least, I think you should invent a term for "that code which is not 
aware of preemption" so there is at least a psychological separation.

User-managed threads are inferior to kernel-managed threads because 
the managing code is not protected from the code being managed. But I 
suppose that is simply a case of trading off protection for fewer 
context switches, which we designers do all the time.

>I believe we agree that in the absence of a compelling requirement,
>awareness of preemption should exist only within the runtime system.
>What we are really debating is whether the line between application and
>runtime should be at the user/supervisor boundary or someplace else.
>There are compelling performance arguments that it should not be
>restricted to the user/supervisor boundary.
>
>Unfortunately, there exist applications with a compelling requirement:
>multimedia applications. These applications absolutely MUST know when
>certain preemptive events occur -- notably the start of their guaranteed
>scheduling slot. They need to know this because they require the ability
>to synchronize their processing behavior to external events. A second
>example is high-speed network stacks. In both cases, the incoming event
>will be "posted" to an application-level event loop for later
>processing. The main difference vs. pure event-driven processing is that
>a global variable will get set by the runtime indicating that a
>high-priority event has arrived, and some of the event handlers may poll
>this variable and voluntarily abandon their work.

As I've said, I think a kernel-implemented helper domain could set 
such a global variable. I'm not yet convinced that all the options 
for avoiding user-level scheduling have been exhausted. If they have, 
it should be possible to construct a proof that the requirements 
cannot be met otherwise.

The big issue is, who is in charge of allocating the CPU resource? It 
is simplest to centralize this function (in the kernel). Of course 
simplest isn't always adequate. If you believe it is necessary to 
migrate some of this function down into user code (and I'm still not 
convinced), you'll have to impart some knowledge of global scheduling 
conditions into the user code. For example, the multimedia event 
handlers might not want to abandon their work if there is an idle CPU 
available. I'm quite sure you've thought about these issues, but I 
haven't seen a credible design yet. (I haven't yet reviewed the 
recent ukernel spec.)

(To illustrate with an example; a single cache can improve 
performance, but caches at two levels that do not know about each 
other can make performance worse.)

Coyotos is going much further in supporting real-time requirements 
than KeyKOS/EROS/CapROS did. It's certainly an interesting research 
direction.