[coyotos-dev] Fixing the kernel heap size

[Jonathan S. Shapiro]

The Coldfire port has brought an issue into sharp relief: it is highly
desirable to set a relatively HARD upper bound on the kernel heap.

Issue: on Coldfire, and possibly other embedded targets, we want to
direct-map the kernel to reduce TLB pressure. On other targets, we want
to use large page mappings for the kernel region. The underlying problem
is that the kernel heap is allocated simultaneously from two spaces:

  - The virtual address space of the kernel, which MUST be contiguous.
  - The physical memory space of the underlying memory, which MUST be
    contiguous on some platforms, and on others wants to be made up
    of large page mappings.

In order to do either direct mapping or large page mapping, it is
desirable to ensure that the majority of heap VAs are of the form

	VA = PA + constant.

For the most part, the kernel's heap requirements are known at boot
time. Given the size of physical memory, we can estimate the size
requirement on the heap exactly, with ONE exception: device memory. The
problem with device memory is that the drivers aren't in the kernel so
the kernel has no way to know what memory will appear from attached
cards.

For device memory, the requirement on the heap is that each page of
device memory requires an associated "struct Page" overhead structure in
the kernel heap. At present, these "struct Page" objects are dynamically
allocated by growing (but never shrinking) the kernel heap at run time.

The ultimate problem with this heap growth is that it relies on the MMU,
and for targets with soft-managed TLBs we really want to avoid pressure
from kernel mappings (thus the incentive to direct map the kernel).

A further issue is that post-boot heap growth shrinks the number of
available object frames that are allocatable by the space bank. This
inadvertently violates the kernel/spacebank contract.

There seem to be three classes of solution:

  1. Have a tunable number of "reserved" device page frames. Allocate
     the "struct page" structures for these early. If you run out,
     the attempt to map the device memory fails and it is necessary to
     re-tune and re-boot.

  2. Separate the kernel heap into two parts. One is fixed at boot time
     by pre-sizing. The second is dynamically growable but falls outside
     of the direct-mapped kernel heap. This is, in essence, a hybrid
     strategy.

  3. Conservatively reserve excess kernel heap space so that you
     always have enough, then provide a mechanism to "release" this
     space after the device config has been determined. This could be
     combined with [1].

I personally prefer strategies [1,3] The problem with it will mainly
arise when people stick modern gaming graphics cards into their PCI
slots and chew up some huge part of the memory map with them.  The
*minimum* requirement is that we be able to configure enough devices to
tell the user that we need to re-tune and reboot.

Do people have opinions about this?


shap