Tag Archives: code coverage

Embedded Software Testing

Posted on by
Reply

Software Testing

I recently met with an Australian Software Development company, PepperStack, and we got onto the subject of Software Testing. As someone who began their career as an Electronics Hardware Engineer, one of the things I learnt was that you have to test thoroughly to be sure everything is working as it should be. With Electronics, if you make a mistake with an Engineering Calculation you can easily destroy things. This is sometimes referred to as “letting the smoke out”. So it was good to meet with others who believe in the same level of rigorous software unit, module and system testing that we do.

 

 

Some Engineering Humour

Which reminds me of a joke I once heard:

 

There are three Engineers in a car going for a drive. The first is a Mechanical Engineer, the second an Electronics Engineer and the third is a Software Engineer. Fortunately the Mechanical Engineer is driving because the brakes fail and they are going downhill. The Mechanical Engineer eventually brings the car safely to a halt and gets out to examine the hydraulic systems. The Electronics Engineer gets out and checks and body computer, ABS system and the power train CAN bus. The Software Engineer stays in the car and when queried about it says that they should all just get back in the car and see if it happens again!

 

Now don’t get me wrong, I’m not having a go at Software Engineers. The process of finding and eliminating faults is a very important part of the development cycle and is something that needs up front thinking and not just responding to symptoms. And the more complex or sophisticated a system is, that harder it is to fully test every possible response to every possible stimuli and after a certain point it becomes impractical to have 100% Test Coverage (every line of code has been executed through all of the possible states). The reason this is a bigger problem with Software Development is that the flexibility of software means that it is inherently complex and it takes skill and planning to manage that complexity so it is testable.

 

So here is the issue. More than any other discipline, faults can be experienced by an end user of a product under a situation or scenario you could not have proactively tested against before release. There are many potential reasons for this including:

  • change of hardware or operating system environment
  • new standards or protocols
  • the sheer number of potential combinations of drivers, peripherals, software and users
  • the product being used for a purpose it wasn’t originally designed for
  • gamma ray corruption of a memory location – I am getting esoteric now but in some areas like avionics and space this is a big threat

So how do you reduce the likelihood of these problems occurring?

 

 

Improving Software Quality

With many new products having Electronics and Embedded Software and the Software Development requiring 80% of the effort, it is important to delivery it as quickly and fault free as you can. The main weapons in your Software Quality arsenal have been known about for a long time but are, in our experience, just not used. These are:

  • Architectural Design – work out how the data and execution flow will happen and how you will manage the constraints
  • Functional Decomposition – divide and conquer but with an emphasis on how each module fits into the system and how the interfaces work in detail
  • Error handling – who will decide what to do with response codes – again this is data and execution flow and part of the architecture. In many cases exception management is at least 50% of the project.
  • Have an Integration Test Plan – some thing that proves the data and execution flow matches the architectural design. Too often “it builds” seems to be good enough here.
  • Unit Test modules – so you remove all the issues before adding them to the integration
  • Do the Integration Tests before you try system testing
  • Design modules so you can integrate them as shells then add functionality down the track
  • Have NVM and configuration data available at the beginning of the project and not as an after thought at the end
  • Have a System Test Plan and use it
  • Use some of the good practices of Test Driven Development – run the tests every time you change the code
  • Have a rationale for what level of Code Coverage you can accept
  • Have a rationale for what level of Churn you can accept – Churn is the percentage of the lines of code that have changed in the past time period. Usually either a week or month depending on the size of the project.
  • Use automated software quality tools. For instance we use both PC-Lint and RSM to automated many software quality metrics which saves a lot of time in Code Reviews
  • Use Code Reviews, also known as Software Peer Review. It really does save time.

Next I plan to look at what you can learn about software testing from a Squash Racquet.

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile. This post is Copyright © 2010 Successful Endeavours Pty Ltd.

Reducing Electronics and Embedded Software Product Development Costs

Posted on by
Reply

First some basic statistics that made me think about this issue a bit more:

  • Software typically consumes 80% of the development budget. Digital Avionics Handbook and Embedded.com
  • 80% of software projects are unsuccessful IBM

 

So working from the Pareto Principle it is clear that product development success and cost can be most improved by addressing the Software Development component. In my recent post on Reducing Electronics Manufacturing Parts Cost I argued that increasing the software component can reduce the hardware costs. Which is a great idea as long as it doesn’t introduce an even more expensive problem.

 

I agree with Jack Ganssle in his article looking at tools where he points out that software quality tools are often not budgeted for yet will find many classes of defect quickly and at a significantly lower cost than the test and debugging effort required to find them after integration with the rest of the project. Or put another way, the cheapest way to get rid of bugs is not to introduce them in the first place – Lean Coding.

 

Since we mainly develop in C and C++, this is what we do to ensure we minimise software development cost and overruns:

 

Static analysis and code reviews

We use static analysis and code quality tools such as PC-Lint and RSM and integrate them into our editors and IDEs so we can run the tests are part of our build or at the very least with a single click covering either the current file or the current project. These tools find flaws you are hard pressed to identify by visual inspection and I believe they pay for themselves within a month of purchasing them. They can also enforce coding standards. Another great benefit is that when you do a code walk through and review, you are not looking for these classes of faults explicitly because you know the toolset will find them for you. So the first thing you do is run the tests and focus on anything found there.

 

Code reviews save money. Every issue identified in a code review is an issue you don’t have to debug later on. And another person is going to look at your code without the same assumptions you would so they will see the things you miss. It just makes sense to do it. Software debugging is more expensive than coding so not bugging in the first place is good budget management.

 

Unit testing

Next, we unit test. A huge benefit of this is that you have to think about test and it makes you think about error handling in the design phase. Many problems in implementing embedded systems come from not handling errors consistently. Sometimes they aren’t handled at all! Someone else once suggested that software developers were the most optimistic people on the market – you can tell this is true by looking at how they handle exceptions! I’m not sure who said it so if you know then post a comment and I’ll credit them and provide a link too if you have one.

 

Integration testing

Integration testing itself does not have to be overly complex. You want to know that things work and it is often easier to write a cut down system to manage the test process. This way you are proving that each subsystem is present and correct before doing the full scale system test. This is an area that often gets overcomplicated. Don;t try and do more here than you have to.

 

Oh, and by the way, just because something builds don’t mean it passes the integration test. Some things to cover are:

  • software manifest – do I have the right version of each module?
  • data flow – do the higher level calls get at the right data lower down?
  • exceptions – do error returns get passed back?
  • exceptions again – if you raise exceptions, do they get acted on?
  • communications – does it communicate?
  • IO – are they mapped to the right pins and peripherals?

Simulation

For some systems or subsystems we write fully fledged PC mocks around the code and ensure it handles all the parameter and error cases correctly and that all the functions are correctly implemented. This is a form of integration testing that proves the software component of the system is doing what it is meant to but goes a lot further to fully excercise part of it. And since 80% of the problems come from software this is a very effective way of reducing bugs and difficult to track down system defects that are expensive on time and resources to cover in real time operating tests.

 

To do this, you have to abstract the interface so the code can run in the embedded version or the PC version without any changes. This is easy to do if you think about it in advance.

 

One word of caution; the PC has a lot more resources and clock speed available compared to a smaller embedded system so this is not a substitue for testing on the real hardware to ensure execution latency is acceptable.

 

And for the purposes of this post, the PC could just as easily be a Linux or Mac system. The point is to use the higher level system to efficiently and fully test the embedded software module so you save time and money later on in the project. And let’s face it, who like to be under unnecessary pressure at the back end of an embedded software project?

 

System testing

If you think in advance about how to most easily implement the system testing then you can save a lot here as well. We put effort into deciding how the do the test process at the architecture design phase so that we have the data flow required to actually do the test. This can be as simple as having some extra parameters or calls available to be able to inspect the state of the system and the communications facilities to get at this data. Where possible 100% parameter range testing and 100% code coverage testing is very desirable. One thing this means is that you had better think about how you will create each error condition that must be handled!

 

Low Cost Software Development

Low Cost Electronics Manufacture relies on Low Cost Software Development. So make it a priority. The Pareto Principle says that it is the most important thing to get right.

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.