Re: Call for contributions: test suite

From: Lorenzo <>
Date: Mon, 30 Mar 2015 00:51:23 +0200

On 03/29/2015 11:23 PM, Laurent Bercot wrote:
> On 2015-03-28 15:36, Lorenzo wrote:
>> It's been a month, and no replies... but hey, "please contact me",
>> does it mean there's been some progress in private discussions?
> I wish. But unfortunately, it looks like testing is as unappealing
> to other people as it is to me. :)
> (Not that I blame anyone. I can't be bothered to write tests for
> my own freakin' projects, so why would it be surprising that nobody
> wants to write tests for a project they don't even own ? It's really
> a case of "there are all kind of fetishes in the world and I'm hoping
> to find somebody with a compatible one". Which is a numbers game,
> and we don't have the numbers yet.)
>> If not, I've spent a few hours looking at possible candidates that
>> match your criteria(*), and some candidates actually survived.
> Awesome! I'm all ears.
>> Please tell me that someone mailed you in private anyway :)
> The silence in my mailbox was deafening. The community is still too
> small so the odds are against me. :)
> I hope to make the community larger with the projects I'm currently
> working on; I can still keep things under control until then at least.
A part of me still wishes I never sent that email, you know... ;p

So, I went to check the test suites I sort of knew (and hated), than
those from, then
wikipedia (it's 1e9 projects, which helps explaining why I was
aggressively trying to cut the numbers down).

Oh, and since command line utilities are weird to test, add or

Having no extra dependencies cut many of them; having to be included in
the tarball cut most of those left.
That being said, I regret not keeping a log of reasons why I rejected a
project or another - the list is long, it's possible that I just skipped
some of them and I don't know.

Anyway, here's the projects that survived till the end.
TLDR: I vote for tap - who said that nothing good came out of perl?


Should be small and simple enough XD


Just look at the header and read the examples, it's simplicity itself.
Then look at test_crash(), add another test after it and see what
happens - neat! So neat it rules out seemingly promising projects like
Wikipedia says it's used in the APR; this is no longer true since ????,
but at least proves it has real world usage.

The main downside is that it's only for C code, so testing a binary by
running it from C is awkward.

Examples of projects I discarded because I feared people would think
"ugh, too much to learn - someone else will write the tests anyway".
Especially cu shows that I probably set the bar too high.


Remember that text is a universal interface? they do.
TAP is a *very* simple textual protocol to say "test 1 ok, test 2
botched, test 3 ok (extra details)" etc.

My favorite library is - it's <500loc,
looks very sensible, and the author recommends just copying tap.[ch] in
your projects.

+ it's a __protocol__, so there's a lot of documentation; try running
"perldoc Test::Tutorial", the perl api is... well, perl, but the
concepts are very sound, explained clearly and with the assumption of a
reader scared of testing.

+ very sensible: eg it helps you maintain todo lists. Really, look at

- need a program ("consumer") to analyse the output of the testsuite
+ can be very minimal; if tests exit nonzero on failure, just a "for t
in $tests ; do $t||exit; done" is enough to get you started; even the
autotools felt they could write a consumer using the shell...
+ can do fancy stuff, like running tests in parallel etc - see eg the
commands suggested in git's t/README
- I didn't look for a consumer as minimal as libtap

+ MAIN SELLING POINT: not restricted to C, can be used to test
commandline utilities - see the git testsuite, it's *overkill* for us,
but gives you an idea of how well tap can scale if you really need it

+ test gets a SIGSEGV or similar => the consumer notices

+ superlightweight, you can even run the tests and analyze them on
another machine if you want to
        - git produces ~500KB of output- a few minutes even with 9600 baud

Test coverage - aka, how I miss "go test -cover"...
Gcov is gcc only, it that ok? Do you have any platform where it's not
the main compiler?

Sorry that this message is more similar to a braindump than a proper
email, it's almost 1am here and DST started today.
Next time I'll be fully awake!
Received on Sun Mar 29 2015 - 22:51:23 UTC

This archive was generated by hypermail 2.3.0 : Sun May 09 2021 - 19:38:49 UTC