[coyotos-dev] SMP: Dropping the low map

[Jonathan S. Shapiro]

On Mon, 2007-10-22 at 21:56 -0400, Jeroen Visser wrote:
> Shap,
> 
> I re-enabled dropping of the low map in boot.S exactly as you had it before.
> 
> In order to boot up an AP I now allocate two AP bootstrap pages in low memory.
> 
> The first page contains bootstrap code, bootstrap data area and
> bootstrap stack as before.
> 
> In the second page we store a copy of the cpu0_KernPDPT (or PTE
> equivalent). We then manually copy IA32_PAE[3] to IA32_PAE[0] (or PTE
> equivalent) to create a workable bootstrap map with a low alias....

It seems to me that there are two possible approaches for handling the
low-memory mapping:

1. Drop the low map at the end of arch_init().
2. Drop the low map early, but re-establish it temporarily during AP
   spin-up.

Until very recently, the code used approach [2], and it should not be a
problem to go back to that approach. The reason we switched to [1] was
(a) it caught some bugs, (b) we had not yet looked at AP spin-up
requirements, and (c) it was easier to explain to an assurance
evaluator.

>From the standpoint of constructive paranoia, I think that [2] is
probably the best approach here. From a bug prevention perspective I
think that it is a real exposure to leave the low mapping lying around
when we aren't actively using it.

So:

I have just introduced hwmap_enable_low_map() and
hwmap_disable_low_map(). You should double check the implementations,
because they are untested (and unused at the moment) but I think they
are correct.

I have adjusted init.c to reserve physical page 1 when SMP support is
compiled in. This is where the ap_boot code should get copied.

I also integrated a bunch of your AP boot code as apboot.S. This is not
yet called, and you should check it for errors. It is a bit different
from your approach. You will also need to check cpu_finish_ap_setup().


With these additions, there may now be a bug in the interrupt handling
code. Specifically, we need to figure out how to correctly handled the
"unmasked" bit for CPU-local interrupts in the interrupt vector
structure. My suspicion is that the right thing to do is to remove it,
or possible to handle CPU-local interrupts as a special case.

Anyway, take a look at this partial integration and see if you can get
the lapic IPI stuff shoveled in.

Also, my apologies. I initially thought only to integrate some of the
code you had provided in your patch to reduce your effort, and I got
sucked in to changing the code as I went. this was not, perhaps, the
best approach.
-- 
Jonathan S. Shapiro, Ph.D.
Managing Director
The EROS Group, LLC
www.coyotos.org, www.eros-os.org