[coyotos-dev] Test infrastructure for bignum

Jonathan S. Shapiro shap at eros-os.com
Mon May 5 06:56:34 CDT 2008


On Sat, 2008-05-03 at 19:31 +0200, Thomas Stratmann wrote:

Several responses:


ON IOSTREAM

> Specifically, my point of criticism is that it depends on the stream 
> being set to not ignore whitespace. I think it should work in both 
> cases, and the error handling is essentially nonexistent. I would try to 
> make this better (but please confirm before).

So first, it is very likely that the current iostream functions are not
right. They were implemented in haste.

But second, the correct iostream functions should behave like the
equivalent ones for normal integers. This is easy enough to fix, but my
question then becomes: what is the correct behavior?

I would be very happy to update the iostream functions; I just need to
know what they are supposed to do.

But see below. I think that it is important to fix the iostream
interface, but this is probably not critical for most of the test cases.

ON TESTING

Some further history here. One of the intentions for BigNum is that it
will become the BigNum implementation used in the BitC runtime. One goal
of the implementation is to ensure that all of the data that encodes a
BigNum can be reached without additional pointer indirections. This is
why, for example, we use a struct of the form {length, wordvec[]}, where
the wordvec does not have an additional indirection. This implementation
choice was also informed by some work I did on TinyScheme at one point.

I think that part of what was in the back of my mind at the time was
that BitC would have an interactive mode, and that we would be able to
do a lot of the testing in BitC.

In hindsight, this was a little foolish, because the BitC bignum
implementation needs to be written in BitC eventually.

However, the current BigNum implementation is used in the mkimage
utility, and mkimage is a fairly general-purpose interpreter, even
incorporating first-class functions.

So this raises a question in my mind about how to test BigNum. For
low-level tests we obviously want to be in C, but for high-level tests
would it make sense to implement some very simple interpreter?
 
> > We prefer to drive our test process from make rather than bash. We have
> > some infrastructure for this in the bitc tree; I can migrate it into the
> > coyotos tree for you if you like.
> > 
> 
> What exactly would you move over? testit.sh would definitely be helpful.

Yes. testit.sh was mainly what I had in mind.

> As a matter of fact I'm already using it, but since I'm behind the 
> BitC<->hg gateway the file history would probably be lost if you 
> committed my changes.

I'm not sure what you mean but "the BitC<->hg gateway" here. First, I'm
not sure what you mean by a "gateway" in this case, and second, the
official home of the BigNum implementation is in the ccs-xenv tree of
the Coyotos code base.

If what you are saying is that you have a set of changes that you want
to ship back for integration, there is a way to email them to me using
mercurial. Is that what you need?

> Ok. To get anything working I will prepare the "alien steal build 
> infrastructure" thing and try to keep it in separate orthogonal branches 
> (this is part of the headache I had -- my hg structure is overly 
> complex). Before part of this goes into the tree I would need 
> significant assistance.

I think that the right thing to do in the short term is to add "bntest"
to the subdirs list in the parent directory, let the build in the
libsherpa directory proceed as-is, and then have the "make all" target
in the bntest directory simply link the libsherpa.a directly by reaching
over into the libsherpa directory.

The catch here is that this needs to be suppressed when cross-building,
and I will need to look in to how to do that.

> One reason why having tests NEVER being triggered automatically (i.e. on 
> a normal build on platforms that allow it) is BAD is that people will 
> just forget about them.

I agree.

> I just looked into the bntest dir (which probably means BigNum test) and 
> figured that I couldn't get anything to work in there. Maybe this is one 
> of these "never looked at and forgot" things... Could you try to 
> reproduce this? Just make bntest...

It has drifted. I will get what is there cleaned up.

> Also, I'm curious about what is actually supposed to be tested (testing 
> implemented) in there, as I couldn't make any sense out of it (partly 
> because it wouldn't compile).

What is supposed to happen in bntest is that it builds a small
calculator program which can then be fed test cases. Other things in
addition to the calculator need to be added, but the idea was to avoid
needing to compile a billion small test cases by building an
interpreter.

> > I am not sure why a maintainer test should be different from a
> > regression test. Can you give an example?
> Not really. It's a bit like this:
> If a regression test fails (which means a bug is triggered which has 
> been fixed before), this should alert the entire project so that the 
> patch that is to blame is taken back.
> If a maintainer test fails, api behaviour might still be ok for some 
> double fault reason, but the maintainer must look into why this is the 
> case (and either fix or adjust the test).

Hmm. I would suggest a different distinction. A maintainer test is
entitled to bypass the API when checking its result. For example,
maintainer tests are entitled to know about internal implementation
details. Regression tests are not.


shap



More information about the coyotos-dev mailing list