When I was developing large apps for other people to use, one of my recurring nightmares was that old bugs, once fixed, would return in a future version and make me look unprofessional.
Consequently, one of my most important tools was an ever-growing suite of test scenarios which I ran my app through before issuing a new release. I also asked my beta testers to run these scenarios where practical.
It seems to me that if my app were a programming language, this sort of suite would be particularly easy to produce.
No doubt RS have an extensive testing suite but they don't make it public for users to test. Also, I sometimes wonder if they share my nightmare.
I keep wondering about starting an open-source project which would host tests of current and previous bugs.
- Have I the time and enthusiasm to produce the original framework?
- Do I know enough about the current bugs?
- Would someone else be more appropriate as the project owner?
Of course, there was Joe Strout's excellent ROTOR which ran several tests that could be automated and which many of us ran on each new beta and release for a while but those tests all seem to pretty much pass these days (except the encoding of external files) and it has fallen out of sight.
My preference now would be for something that runs unit tests where possible but which also includes tests which required user-involvement.
There would be a project (or projects) with windows describing what the user needed to do and what result was being looked for.
This would allow tests to be run on as many platforms as possible by beta testers, new users and anyone who starts using the latest version.
Hopefully, if there were user reports quoting problems found by the testing suite, RS would be able to check it out for themselves.
It seems to me that this would be a more positive step than just producing a list of bugs.
Perhaps I'll manage to get this started at some point. Or perhaps someone else will come up with the idea and the enthusiasm first and run with it.