[coyotos-dev] Test infrastructure for bignum

[Thomas Stratmann]

Thanks for your clarifications. Bottom line for me is that we have to 
try to avoid duplicate work.

Jonathan S. Shapiro wrote:
> On Sat, 2008-05-03 at 19:31 +0200, Thomas Stratmann wrote:
> 
> Several responses:
> 
> 
> ON IOSTREAM
> 
---snip---
> So first, it is very likely that the current iostream functions are not
> right. They were implemented in haste.
> 
> But second, the correct iostream functions should behave like the
> equivalent ones for normal integers. This is easy enough to fix, but my
> question then becomes: what is the correct behavior?
> 
> I would be very happy to update the iostream functions; I just need to
> know what they are supposed to do.
> 
> But see below. I think that it is important to fix the iostream
> interface, but this is probably not critical for most of the test cases.

I'd do that when I find the time, but I believe that having tests at all 
should have higher priority. My approach then would be to search the web 
for other places where people feed an istream to a "complex" data structure.

Could you comment the istream stuff out and recompile to ensure it is 
currently not in use? (Or do you know this to be the case?)

> 
> ON TESTING
> 
> Some further history here. One of the intentions for BigNum is that it
> will become the BigNum implementation used in the BitC runtime. One goal
> of the implementation is to ensure that all of the data that encodes a
> BigNum can be reached without additional pointer indirections. This is
> why, for example, we use a struct of the form {length, wordvec[]}, where
> the wordvec does not have an additional indirection. This implementation
> choice was also informed by some work I did on TinyScheme at one point.
> 
> I think that part of what was in the back of my mind at the time was
> that BitC would have an interactive mode, and that we would be able to
> do a lot of the testing in BitC.
> 
> In hindsight, this was a little foolish, because the BitC bignum
> implementation needs to be written in BitC eventually.
> 
> However, the current BigNum implementation is used in the mkimage
> utility, and mkimage is a fairly general-purpose interpreter, even
> incorporating first-class functions.
> 
> So this raises a question in my mind about how to test BigNum. For
> low-level tests we obviously want to be in C, but for high-level tests
> would it make sense to implement some very simple interpreter?
Again, and for the case anyone else finds our BigNum useful, there 
should be API tests. There MIGHT me lower level tests (see the comment 
at the very end of this mail). There is no higher level to BigNum, only 
to tools using it. These should have their own tests.

But see my longer comment below on how I implement my tests.

>  
>>> We prefer to drive our test process from make rather than bash. We have
>>> some infrastructure for this in the bitc tree; I can migrate it into the
>>> coyotos tree for you if you like.
>>>
>> What exactly would you move over? testit.sh would definitely be helpful.
> 
> Yes. testit.sh was mainly what I had in mind.
> 
>> As a matter of fact I'm already using it, but since I'm behind the 
>> BitC<->hg gateway the file history would probably be lost if you 
>> committed my changes.
> 
> I'm not sure what you mean but "the BitC<->hg gateway" here. First, I'm
> not sure what you mean by a "gateway" in this case, and second, the
> official home of the BigNum implementation is in the ccs-xenv tree of
> the Coyotos code base.

I just assumed that you keep the sources in... ah never mind. We both 
speak hg, there is no problem.

> 
> If what you are saying is that you have a set of changes that you want
> to ship back for integration, there is a way to email them to me using
> mercurial. Is that what you need?

Since I'm using some weird hack to "steal the buildsystem" (calling 
configure from a subdir in order to produce a Makefile out of my 
Makefile.in and to be able to link against libsherpa), I would send you 
my patches privately.
I will send stuff over the list when I find a bug.

> 
>> Ok. To get anything working I will prepare the "alien steal build 
>> infrastructure" thing and try to keep it in separate orthogonal branches 
>> (this is part of the headache I had -- my hg structure is overly 
>> complex). Before part of this goes into the tree I would need 
>> significant assistance.
> 
> I think that the right thing to do in the short term is to add "bntest"
> to the subdirs list in the parent directory, let the build in the
> libsherpa directory proceed as-is, and then have the "make all" target
> in the bntest directory simply link the libsherpa.a directly by reaching
> over into the libsherpa directory.
> 
> The catch here is that this needs to be suppressed when cross-building,
> and I will need to look in to how to do that.
I am keeping several repos, in order to keep my buildsystem hacks and 
the actual testing apart. Just let me know when you change things, and I 
will adapt my approach. Or I can send you my stuff prematurely so you 
can see what it would require.

> 
>> One reason why having tests NEVER being triggered automatically (i.e. on 
>> a normal build on platforms that allow it) is BAD is that people will 
>> just forget about them.
> 
> I agree.
> 
>> I just looked into the bntest dir (which probably means BigNum test) and 
>> figured that I couldn't get anything to work in there. Maybe this is one 
>> of these "never looked at and forgot" things... Could you try to 
>> reproduce this? Just make bntest...
> 
> It has drifted. I will get what is there cleaned up.
> 
>> Also, I'm curious about what is actually supposed to be tested (testing 
>> implemented) in there, as I couldn't make any sense out of it (partly 
>> because it wouldn't compile).
> 
> What is supposed to happen in bntest is that it builds a small
> calculator program which can then be fed test cases. Other things in
> addition to the calculator need to be added, but the idea was to avoid
> needing to compile a billion small test cases by building an
> interpreter.
Ah. So this means that we have two approaches here which go into totally 
different directions.

I am currently creating the "billion tests" solution (i expect billions 
to be <60). I already have make rules to easily setup further tests just 
by including a source, an input, an expected output and adding a word to 
a Makefile. The first tests, checking a combination of 
constructor-from-string with base (8, 10, 16) and without base (check 
for leadin "0x" etc.) and the ostream interface, already work, except 
for the fact that I screwed up my fixture data. Further addition of 
arithmetic ops tests should then be a matter of (working) days.

If we ever need anything as complicated as "make sure exception xy is 
thrown" this should be easily possible inside this framework, too.

I want these tests so that I can rearrange code without having to be 
afraid of breaking things. Do we really need a dedicated BigNum 
calculator program? If we stuff the tests in their proper subdir and 
comment them properly we should be fine.

> 
>>> I am not sure why a maintainer test should be different from a
>>> regression test. Can you give an example?
>> Not really. It's a bit like this:
>> If a regression test fails (which means a bug is triggered which has 
>> been fixed before), this should alert the entire project so that the 
>> patch that is to blame is taken back.
>> If a maintainer test fails, api behaviour might still be ok for some 
>> double fault reason, but the maintainer must look into why this is the 
>> case (and either fix or adjust the test).
> 
> Hmm. I would suggest a different distinction. A maintainer test is
> entitled to bypass the API when checking its result. For example,
> maintainer tests are entitled to know about internal implementation
> details. Regression tests are not.
Sounds good. So that testing for getDigit() and theDigit() would be 
maintainer tests (both are private members).