Under control

From Arbitrary Technical Decision To Bug: It’s Just a Matter of Time…
A miniseries on software testing, legacy code and the impact of wrong assumptions or arbitrary technical decisions on software products.

As I explained within the previous article of this miniseries on software testing, bugs on production are not very common when you work on medical devices.

However, any complaint coming from the field —that is, from hospitals or medical labs— is taken very seriously: for starters, an investigation needs to be conducted.

So, some years ago, while serving as a software tester on medical devices, one day was I given the task of trying to reproduce an issue reported by a lab technician.

He was complaining about the fact a quality control, that is a kind of job run on a specific benchmark material (instead of blood) in order to somehow calibrate and certify the accuracy of the assay itself, had not automatically executed at the expected time.

Which was really surprising, especially taking into account that both the programmer and the tester historically in charge of that functional area for many years were very experienced.
Now, I don’t remember if the experienced tester was busy with something else or on holiday at the time, but I do perfectly recall the skepticism in the programmer’s face.
He likely thought it must have been a user error. Hopefully, a not reproducible issue…

Being a strong believer that something which happened to a customer (in production environment or in the field) tends to be reproducible by definition, I enthusiastically took on that challenge.

Anyway, I quickly realized that the issue was not easy at all to reproduce.
What’s more, the complaint was affected by a lot of noise: several conjectures about the reason of the issue made by the lab technician did nothing but distract me from trying to figure out what the cause of the problem really was.

So, one of the first and most important things I learnt from that investigation1 was that separating facts from conjectures is crucial.

"Just the Facts" by Bart Heird

After cancelling all that noise, things started being clearer.
Several changes in settings and quality control executions later, persistence eventually paid off: at last, I was able to reproduce the issue.
At that point, getting the attention of the programmer was a smooth task.
It was time for him to carefully inspect the code in order to understand what the hell was happening there.

It turned out that, many years before, he had made a technical decision regarding that functionality: he had introduced a buffer (of thirty minutes) to make sure the quality control job runs were properly scheduled without overlapping.

A probably reasonable (yet arbitrary) technical decision nobody was aware of, basically because it was not documented anywhere and the programmer had probably forgot about it immediately after making it.

So, even though, in that case, we decided not to change the code —it would have been too risky: something that we had to explain to the customer, by the way—, there was definitely a bug there: a documentation one. Yet still a bug.

A bug which provoked a customer complaint and costed us a considerable amount of investigation work many years later.

So, we updated both the internal documentation and the user manual, of course.
But, more importantly, we learned the hard way that arbitrary (or not documented) technical decisions (especially those ones made many years back on legacy code) will surely convert into bugs: it’s just a matter of time…

But how about your company?
Is everything under control or might there be a hidden buffer somewhere?
Are you deliberately challenging your assumptions or are you just waiting for the moment your arbitrary technical decisions have a dramatic impact on your software products?
Just asking…

1 It must be said that it was the first time for me to investigate a complaint. Anyway, since dealing with the second one (which was also affected by a lot of noise), some months later, only took me 20% of the time the first one did, I guess I can claim I’m a quick learner.

my profile picture

Thanks for reading this article.
Feel free to recommend it or to add a comment.

Should you have any doubts about Software Testing, contact me: I will be glad to help you.

On the other hand, if you want to get notified about my blog posts, sign up through the BLOG > SUBSCRIBE TO THE BLOG NEWSLETTER menu.
Thank you.

2 thoughts on “Under control

Comments are closed.