Test cases make bad law

Testing seems to be the topic du jour this weekend… A few remarks.

I don’t think this is a discussion that should be framed as an argument between pro-test and anti-test factions. In fact, I’m not even sure the latter group really exists. Yes, some modules could be better at adding tests on a regular basis, but I don’t really see people arguing that testing sucks and we should just stop doing it. What I *do* see are concerns about the degree of testing that should be required. That’s an interesting discussion, and should be held without any implication that supporting anything less than CMM Level 5 is akin to supporting terrorism. (Or the reverse, oops.)

I think a lot of the uncertainty about testing comes from the fact that, until recently, there was almost no automated testing. And so we’re going through a period of growing pains where the project figures out how to handle things. The existing policy (for Toolkit and Browser), which is basically “everything should have a test”, has been a good starting point. It’s simple, is mostly the right thing to do, and is a solid kick-in-the-pants to help sidestep the initial inertia to change.

But there are principles we shouldn’t lose sight of… Tests are a means to and end. They have both costs and benefits. And we need to balance these (and a multitude of other factors) when deciding the degree to which something needs tested. That’s not to say we should only aspire to half-assed testing, but neither should we become so risk-adverse that testing requirements halt progress. [Note: being on the verge of a release, where being hyper risk-adverse is a good thing, makes this a complicated discussion!]

Now, switching gears to the issue of tests and new contributors:

I don’t think new contributors should just get a free-pass when it comes to testing. Tests are an important part of good software engineering, and they’re important to the Mozilla project. However, I do think that we can do things to aid newcomers and make the process easier… Ensuring we have good documentation on writing and using tests helps everyone. Module owners and active contributors can work to ensure there are existing tests that newcomers can easily emulate and modify. The scope of required testing can be trimmed to just the essentials. We can be polite, but firm, on requirements without being “jerks”. And so on.

This issue is probably somewhat self-limiting, because the scale of testing should generally correlate with the complexity of the patch. Newcomers are more likely to be doing simpler patches, ergo the testing should be simpler. But there will be tricky cases where simple changes end up being complex to test… Good judgement and balance should be applied, as I argued above. For example, if the existing code is frail and known to be regression prone, tests are unavoidable. If the code is solid and the change well-understood, then making an exception for minimal testing can be reasonable. And while automated tests are strongly preferred, other forms of testing might be acceptable an alternative.