What’s the goal of your bloody test suite?
Thanks for reading this article.
Feel free to recommend it or to add a comment.
Should you have any doubts about Software Testing, please contact me: I will be glad to help you.
On the other hand, if you want to get notified about my blog posts, please sign up through the BLOG > SUBSCRIBE TO THE BLOG NEWSLETTER menu.
Thank you.
There are some data points missing in what you are proposing:
1) how comprehensive is your test suite?
2) is this a test for a single bug or for a more comprehensive suite? In other words, is it a quick production fix or a larger product release?
3) do some of tests fail on a regular basis and if so are these the same ones that are failing now? Did expect them to fail or not?
4) Have you done an analysis on the regularly failing tests and you are comfortable enough that they are still failing? Examples: stress tests that are meant to fail, tests for functionality that is not yet available.
The questions above are pointing to an important parameter: there are several contexts that need to be considered, in some of them failing is ok in others it is not.
Thanks for your comment, Tom.
What you mention is important, of course.
My point was intentionally simpler, though, since it seems to me that too many professionals are worryingly missing the basics of Software Testing.
I love this analogy.
Blood tests are designed to test against a specific issue or highlight a potential issue. I feel this matches the nature of scripted testing which are designed to test against a specific issue and also highlight areas of concern. Additional investigation is needed to find the actual bug.
A scripted test rarely finds a bug but might point the tester to the area of concern. Additional exploratory testing is needed to pinpoint the actual issue.
However, a blood test is usually run because a doctor has a suspicion of something wrong. Sometimes regression software tests are run to make sure something is right. If we knew it was right, we wouldn’t need to run it.
Thanks for your comment, Louise.
To me, the goal of running both regression tests and blood tests is basically the same: to discover, as soon as possible, if there’s something wrong (within your system/body) which needs your attention, or, as you said, “to make sure something is right. If we knew it was right, we wouldn’t need to run it”…
All in all, I hope you agree that the goal of running them shouldn’t be to simply have all the tests pass.
I wouldn’t call it a goal, but an expectation. I would expect regression tests to pass.
I see your point, Louise.
The thing is, some people seem to confuse expectations with goals, to such an extent that they end up putting all their effort in meeting those expectations instead of trying to discover if there’s something wrong somewhere.
And that’s a huge problem.
Here’s a thought I just had. It shouldn’t be the testers goal for the test to pass, it should be the teams goal.
Devs and testers should work together to make the test pass. The tester provides the developers with information about why the test is failing. The developer uses this information to improve the application so the test passes.
This is a little like how a doctor orders a blood test and passes the information to a specialist who treats the patient so the blood test passes next time.
I’m afraid we aren’t on the same wavelength, Louise.
As a matter of fact, I’ve seen too many “professionals” focusing so much on making the tests pass, that they completely forgot that the goal of running those tests was to quickly identify potential areas of concern.
By the way, I would say that, when a doctor orders a blood test, it’s not because they just want it to pass next time: I hope they want the patient to feel better. So, if the blood test passes, but the patient is still sick, I would expect them to abstain from celebrations…
I think that we’re on different wavelengths regarding the interpretation on an analogy not necessarily how we run tests.
When a test fails, we should work together with developers to fix the issue so that it passes next time.
However just because one test passes doesn’t mean there are no issues. It just means that particular issue has been fixed. Testing needs to continue to find what these other issues are.
Exactly: ” just because one test passes doesn’t mean there are no issues”.
What’s more, even when a test passes, you can’t forget that it might be wrong…