[coyotos-dev] Test infrastructure for bignum

[Jonathan S. Shapiro]

Thomas:

A couple of answers here.

First, there isn't any serious test infrastructure at the moment. This
is the intended purpose of the bntest/ subdirectory, but it isn't really
being used effectively.

We prefer to drive our test process from make rather than bash. We have
some infrastructure for this in the bitc tree; I can migrate it into the
coyotos tree for you if you like.

On Fri, 2008-03-07 at 21:45 +0100, Thomas Stratmann wrote:
> I was about to move code around in bignum to make it more maintainable, 
> but I realized that it would not be a good idea to do so without testing 
> for expected behaviour prior and after the changes.
> 
> Writing tests for bignum is also a good thing in itself for several 
> reasons. One ist that, at least for now, it replaces API documentation.

That is regrettably correct. I need to spend some time with doxygen in
this tree.

> I would like to know if there is already any support for testing 
> purposes, any kind of infrastructure, inside the build system. Any 
> code-place I can look at? 

Only the bntest directory.

> Currently, I have an ugly hack that "borrows" 
> some configure and makefile stuff from parent folders -- I'd prefer 
> things to flow in the other direction: something like "make test" that 
> traverses the whole system.

Yes. That would be an excellent idea. The reason we have not done this
is that it is very hard to build a test harness this way for an embedded
system. And I understand that we must not allow that to be an excuse,
and we need to address this.

There is also an issue in ccs-xenv that this is an "alien" subtree, so
integrating test procedures with the rest of the tree is very hard. It
may be the case that the coytools/ test system will not be the same as
the test system for other tools.

> What tools can I use to trigger/prepare the tests? Is bash ok? Some kind 
> of scripting will be necessary to feed the test programs with numeric 
> input, in order to avoid modifying source code or compiling with -DTEST17.

In the bitc tree, we have a shell script called "testit.sh":


http://dev.eros-os.com/hg/bitc/file/8fd77903b72b/src/bitcc-bootstrap/tests/testit.sh

You can see use of this in a makefile at:

http://dev.eros-os.com/hg/bitc/file/8fd77903b72b/src/bitcc-bootstrap/tests/IA32/Makefile


Regrettably, it seems that Swaroop did not continue this practice
elsewhere, and I will ask him to fix that.

I am not saying that you must use this script. I am suggesting that it
may provide a useful framework.

> Another thing to consider is directory structure: a good testing habit 
> is to differentiate between tests triggering old bugs ("regression"), 
> tests meant to replace api documentation and tests for the aid of the 
> maintainer to keep things working the way he/she assumes to do.

I agree with the main point, but I disagree on one specific point: tests
are not a replacement for API documentation. API documentation should be
fixed when it is determined to be deficient.

I am not sure why a maintainer test should be different from a
regression test. Can you give an example?