[coyotos-dev] Fixing the kernel heap size

[Charles Landau]

At 7:12 PM -0500 12/19/07, Jonathan S. Shapiro wrote:
>The Coldfire port has brought an issue into sharp relief:

I'm keenly interested, since CapROS has the same issue, though 
perhaps not as severely.

>For device memory, the requirement on the heap is that each page of
>device memory requires an associated "struct Page" overhead structure in
>the kernel heap.

It's worth questioning this assumption.

Consider that device memory does not require many of the things in 
the struct Page: it's never on the list of free RAM, it never needs 
to be pinned, it never gets aged or paged, and its OID is a simple 
function of the physical address. (I'm using CapROS concepts; 
hopefully most apply to Coyotos.)

If you can do without revoking the keys to this memory, you can do 
without the allocation count and perhaps keyring.

In CapROS a page can sometimes give rise to a page table. You could 
restrict the use of device memory so this doesn't happen.

I don't see much left in the struct PageHeader that is needed.

>At present, these "struct Page" objects are dynamically
>allocated by growing (but never shrinking) the kernel heap at run time.

It would be elegant to allocate the struct Page objects from the 
device memory itself, but I'm sure there are good reasons why you 
can't do this.

>There seem to be three classes of solution:
>
>   1. Have a tunable number of "reserved" device page frames. Allocate
>      the "struct page" structures for these early. If you run out,
>      the attempt to map the device memory fails and it is necessary to
>      re-tune and re-boot.
>
>   2. Separate the kernel heap into two parts. One is fixed at boot time
>      by pre-sizing. The second is dynamically growable but falls outside
>      of the direct-mapped kernel heap. This is, in essence, a hybrid
>      strategy.

I prefer this option because it doesn't require tuning.

>   3. Conservatively reserve excess kernel heap space so that you
>      always have enough, then provide a mechanism to "release" this
>      space after the device config has been determined.

This seems incompatible with hot-plugged devices.

>      This could be combined with [1].

I suppose that's how you handle hot-plugged devices.