[coyotos-dev] IPC Redesign

[Jonathan S. Shapiro]

On Fri, 2007-06-22 at 13:54 -0400, Jonathan S. Shapiro wrote:

> Yes, it can be spilled. Yes it increases the cost. No, it definitely is
> NOT a very small cost. On Pentium Pro, the cost jumps from 2.48us to
> 3.68us when you put a single byte into the indirect string.
> 
> This does not seem very big, but note that the trend in microprocessors
> is for this cost to get worse, not better, over time.

Sorry. I should have added that at 2.48us the performance is already at
the ragged edge of unacceptable for benchmarking purposes.

Also, there is a more subtle issue for read/write: if the length is
pushed into the indirect string, then the receiver must generally make
an extra string copy in order to (de)marshal the indirect string
properly. This extra string copy adds enough performance cost to the
file object that it cannot compete on equal terms with a conventional
file system implementation. The problem is not the presence of the extra
8 bytes; it is the need to do an additional copy of the *data* block
within the file system during the READ/WRITE operations, and the cache
misses that this additional copy entails (and *that* cost is simply
*huge*).

Note that this extra (de)marshal cost is not measured by
microbenchmarks. It can therefore come as a rude surprise when the
macrobenchmarks are run (as it did for us).

So I don't agree with Charlie that the "spill over" solution is
acceptable. Based on the macrobenchmark data for EROS, there is very
little question that we *need* a larger direct string pool (in EROS
terms: a larger number of words that are transferred with register
transfer semantics). Fortunately, this isn't all that hard to do.