[coyotos-dev] Fixing the kernel heap size

[Jonathan S. Shapiro]

On Thu, 2007-12-20 at 14:53 +0000, Sam Mason wrote:
> On Thu, Dec 20, 2007 at 06:25:49AM -0500, Godfrey Vassallo wrote:
> > More recently, PCI bus devices usually provided a memory window whose
> > size can be configured. Access to different portions of the device's
> > memory sometimes required moving the window, which was accomplished
> > through a dialog between the host based device driver and the device
> > based firmware. In practice your 1G example may be easier to manage
> > by placing a maximum window size (e.g. 2M) where possible on the
> > device memory. This is somewhat in keeping with Landau's suggestion to
> > restrict the use of device memory.
> 
> I'm confused.  I thought Shap's problem was with storing the kernel
> related structures associated with device memory, not with the kernel
> attempting to map the the actual device's memory itself.  My current
> mental model of how it works is that the device memory would be mapped
> into the virtual address space of the driver, but wouldn't be touched by
> the kernel at all.

Your model is correct. The problem in the kernel is storing the related
per-frame structures that are necessary for that mapping to occur.

> I can't imagine these growing without bound.

There is a bound, because ultimately the machine only implements so much
physical address range. The problem is that the growth ratio is 1G of
card memory currently requires 2M of management structures. That ratio
is too large for prior reservation to be a good solution on small memory
systems.

>   Having never had to look
> after a system with hot-pluggable memory/PCI cards I'm not sure how much
> this is ever actually done in practice.

On small iron it isn't done, and I can live with "need to reconfigure
and reboot after card swap" on small iron. On large iron, it *is* done,
but it is reasonable to ask how much we need to plan for. Also, on large
systems the main hot-swap item is hot-swap general purpose memory, and
in that case the necessary structures can come out of the memory itself.
This reduces the problem back to things like video cards, and I believe
that the reboot solution is acceptable for that case.

>   I can't imagine devices getting
> moved around much, if at all, after system install time.  I've also just
> thought that the maximum number of CPUs supported by the kernel is fixed
> at build time, it therefore doesn't seem insane to fix the number of
> usable memory-mapped devices at boot time.