I was going to talk more about Continuous Integration, but I received a comment on LinkedIn regarding my post on balancing safety and speed that made me think, so I’m passing my thinking on to you.

Paul writes:

I think that Jeff is overlooking the fact that medical devices require special test systems that can control the input and monitor and verify the output. As such, these are expensive and time-consuming to develop and time-consuming to keep the tests up to date with the software changes. I’ve seen several medical device companies that use them and I’ve created them myself. So, the automated build/test process would have to take away these expensive test systems from the developers that use them, or the company would have to build several of these and dedicate them to the automated execution. Unfortunately, the development can change in a single commit that breaks some or all of the automated tests. In short, automated daily builds are practical (and common) in medical device industry but automated daily verification is not practical.

And my reply to him:

Thanks for commenting, Paul!

I agree that automating final verification may not generally be practical, though I’m interested in pushing that boundary to see how far it can be taken. And yes, automated Hardware-in-the-Loop testing is extremely expensive and time-consuming to set up, and is often too brittle to justify the investment.

What I am advocating is more comprehensive unit tests, and integration tests in simulation. My assertion is that we often rely too much on manual system-level testing (or one-off manual subsystem testing) to verify application logic that in fact can be decoupled from the hardware.

I’m very interested in discussing this more with you, and getting your perspective on what kinds of tests actually add value instead of just adding work.

Paul is talking about the difficulties in trying to automate “System Testing”, where the medical device is tested as a system. Since a device has hardware, you need to feed real, physical signals into that hardware and measure the real, physical effects that the medical device produces. He’s right. It’s often just not practical.

But, the difficulties come from interacting with hardware. Relatively little of the software actually interacts with the hardware. With a proper Hardware Abstraction Layer (HAL), you can isolate your application from hardware completely, making it (relatively) easy to run your application, or at least parts of it, on a regular desktop computer.

Here’s the point: I think we can greatly increase software quality (and increase development speed as well) by doing more unit testing and integration testing. Too many development teams only do system testing, and that’s a shame.

Happy developing!