[coyotos-dev] Fixing the kernel heap size

[Godfrey Vassallo]

I too prefer [1,3] for an embedded target. Additionally, building/tuning target specific configurations for embedded systems is not that big a deal. The embedded system configuration once established may remain unchanged for as much as a decade. Then it is usually replaced by a totally new model. This, at least, has been my experience with medical systems (MRI and RF knife) and military systems.
 
Many of the embedded system that I have worked with had very low memory requirements (~ 1 : 4 Kbytes and a few control and status registers). However, some of these devices (mostly vme bus as I recall) were inflexible relative to their address requirements and conflicts were always a pain to resolve. 
 
More recently, PCI bus devices usually provided a memory window whose size can be configured. Access to different portions of the device's memory sometimes required moving the window, which was accomplished through a dialog between the host based device driver and the device based firmware. In practice your 1G example may be easier to manage by placing a maximum window size (e.g. 2M) where possible on the device memory. This is somewhat in keeping with Landau's suggestion to restrict the use of device memory.
 
Charles Landau's suggestion to question the "struct Page" object relative to device memory is worth consideration. Is it possible to store a much larger page size in the struct and to compute the tlb data required? At first blush that may be workable for s/w managed tlb's but will present a problem for h/w managed tlb's. 
 
Godfrey

________________________________

From: coyotos-dev-bounces at smtp.coyotos.org on behalf of Jonathan S. Shapiro
Sent: Wed 12/19/2007 7:12 PM
To: Coyotos development discussions
Subject: [coyotos-dev] Fixing the kernel heap size



The Coldfire port has brought an issue into sharp relief: it is highly
desirable to set a relatively HARD upper bound on the kernel heap.

Issue: on Coldfire, and possibly other embedded targets, we want to
direct-map the kernel to reduce TLB pressure. On other targets, we want
to use large page mappings for the kernel region. The underlying problem
is that the kernel heap is allocated simultaneously from two spaces:

  - The virtual address space of the kernel, which MUST be contiguous.
  - The physical memory space of the underlying memory, which MUST be
    contiguous on some platforms, and on others wants to be made up
    of large page mappings.

In order to do either direct mapping or large page mapping, it is
desirable to ensure that the majority of heap VAs are of the form

        VA = PA + constant.

For the most part, the kernel's heap requirements are known at boot
time. Given the size of physical memory, we can estimate the size
requirement on the heap exactly, with ONE exception: device memory. The
problem with device memory is that the drivers aren't in the kernel so
the kernel has no way to know what memory will appear from attached
cards.

For device memory, the requirement on the heap is that each page of
device memory requires an associated "struct Page" overhead structure in
the kernel heap. At present, these "struct Page" objects are dynamically
allocated by growing (but never shrinking) the kernel heap at run time.

The ultimate problem with this heap growth is that it relies on the MMU,
and for targets with soft-managed TLBs we really want to avoid pressure
from kernel mappings (thus the incentive to direct map the kernel).

A further issue is that post-boot heap growth shrinks the number of
available object frames that are allocatable by the space bank. This
inadvertently violates the kernel/spacebank contract.

There seem to be three classes of solution:

  1. Have a tunable number of "reserved" device page frames. Allocate
     the "struct page" structures for these early. If you run out,
     the attempt to map the device memory fails and it is necessary to
     re-tune and re-boot.

  2. Separate the kernel heap into two parts. One is fixed at boot time
     by pre-sizing. The second is dynamically growable but falls outside
     of the direct-mapped kernel heap. This is, in essence, a hybrid
     strategy.

  3. Conservatively reserve excess kernel heap space so that you
     always have enough, then provide a mechanism to "release" this
     space after the device config has been determined. This could be
     combined with [1].

I personally prefer strategies [1,3] The problem with it will mainly
arise when people stick modern gaming graphics cards into their PCI
slots and chew up some huge part of the memory map with them.  The
*minimum* requirement is that we be able to configure enough devices to
tell the user that we need to re-tune and reboot.

Do people have opinions about this?


shap

_______________________________________________
coyotos-dev mailing list
coyotos-dev at smtp.coyotos.org
http://www.coyotos.org/mailman/listinfo/coyotos-dev


-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/ms-tnef
Size: 7374 bytes
Desc: not available
Url : http://www.coyotos.org/pipermail/coyotos-dev/attachments/20071220/f2bfb95a/attachment-0001.bin