[coyotos-dev] SMP Issue: load correct kernel stack on trap

[Jeroen Visser]

Hello Shap et al,

If any processor takes a trap in SMP mode it needs to load the correct
per-CPU kernel stack_hi address.

With two indirections we find the per-CPU stack base in
curProc->onCPU->stack (address of the bottom of the per-CPU stack). We
need to load %esp with the top of the stack so we need one addition as
well.

I currently use the patch pasted below. It is not complete (notably
other code paths need fixing).

Questions:

- Can %ebx be clobbered safely?

- Any specific reason we keep stack_lo(cpu) instead of stack_hi(cpu)
in the CPU structure? (Saving the addition.)

- If this operation needs to be faster (does it?) will you store
stack_hi(cpu) directly in proc instead of indirecting trough
proc->onCPU?

-JCV



diff -r b68e1214e2d2 src/sys/arch/i386/kernel/interrupt.S
--- a/src/sys/arch/i386/kernel/interrupt.S      Sun Oct 21 12:54:43 2007 -0400
+++ b/src/sys/arch/i386/kernel/interrupt.S      Sun Oct 21 17:01:19 2007 -0400
@@ -236,16 +236,25 @@ L_no_led:
        mov     %ds,FIX_OFF_DS(%ebp)
        mov     %es,FIX_OFF_ES(%ebp)

-       /* Load up the kernel segment regster values */
+       /* Load up the kernel segment register values */
        mov     $sel_KernelData,%ax
        mov     %ax,%ds
        mov     %ax,%es

-       /* Re-load kernel stack */
-       movl    $EXT(kstack_hi),%esp
-
-       pushl   %ebp    /* pointer to save area */
+       /* Re-load per-CPU kernel stack.
+        * %ebp == sa, %eax, %ebx will be clobbered */
+       movl    %ebp, %eax
        subl    $PR_OFF_FIXREGS,%ebp
+       /* %ebp == proc, %eax == sa */
+
+       movl    PR_OFF_ONCPU(%ebp), %ebx
+       /* %ebx == my_cpu */
+
+       movl    CPU_OFF_STACK(%ebx), %esp
+       addl    $KSTACK_SIZE, %esp
+       /* %esp == my_cpu.stack + KSTACK_SIZE */
+
+       pushl   %eax    /* pointer to save area */
        pushl   %ebp    /* pointer to process structure */

        /* Call generic user mode interrupt/trap handler: */
@@ -299,6 +308,7 @@ L_kernel_iret:

        .text
 ENTRY(sched_low_level_yield)
+       /* Discard kernel stack contents and dispatch */
        movl    $EXT(kstack_hi),%esp
        call    EXT(sched_dispatch_something)