Squash Lessons for Engineering

Squash Lessons for Engineering

The picture in today’s post comes courtesy of Dr Marc Dussault, The Exponential Growth Strategist. At his recent Exponential Business Building Bootcamp, he demonstrated how a Squash Racquet gets broken from repeated use.

 

Broken Squash Racquet

 

So what does this have to do with Engineering? Glad you asked.

 

First, I have to explain the demonstration. Marc showed that it takes a very large amount of force to break the Squash Racquet. He really applied himself to the destructive task and it took a few minutes of escalating Squash Racquet abuse before it finally succumbed and broke. Some of us in the front of the room could tell just how much it required to break the Squash Racquet. However the Squash Racket already had a crack, so Marc knew where to apply the force in order to break it. The picture above is the final outcome. Without the crack being obvious, it would have been almost impossible to have broken the Squash Racquet using just randomly applied force.

 

Marc then explained that way the Squash Racquet became cracked in the first place, was by it being consistently scraped along the wall as he retrieved the ball from shots along the wall. Marc is an outstanding competitive squash player and currently ranks as World # 18! So he knows his stuff when it comes to squash. You can read more about this at his Mindset Of A Champion blog.

 

So if you know what to look for, you can monitor the thinning of the racquet and get an idea of when and where it might fail. If you don’t know what to look for, then the failure will be unexpected.

 

Software Testing and Software Engineering

A lot of Software Testing can suffer from the same problem. If you already know where the weakness will be and how to spot it, then finding a bug is easy. You can set up the scenario, monitor for the symptom and confirm the failure. Or, if you have enough resources you can go the brute force approach and just break it through the persistent use of randomly directed and escalated force of testing. However very products are simple enough and very few companies are large enough to have that level of resource and to solve the problem this way. So for the rest of us, the other 99.995%, a more intelligent approach is needed.

 

Since you don’t know where and when it will fail, it is best to remove failure causes from the beginning. This is where Software Engineering come is. Software Engineering is not just coding. Coding is the production end of the Software Engineering process. Software Engineering is about designing the system so you have defined the components so they are each fully testable in their own right. Then you can apply processes like Unit Testing to ensure they are fully functional as stand alone pieces of software. You can then perform Integration Testing to ensure that software added to the system correctly handles both the Execution Flow, also known as Control Flow, and Data Flow required including error and Exception Handling. The result is that you build up a fully working and correctly executing system quickly and with great confidence. It isn’t a magic bullet but it is close to it.

 

As was famously quipped by Edsger Dijkstra, “If Debugging is the process of removing bugs, then programming must be the process of putting them in”.

 

So if you put less bugs in, you have less debugging to do. And that saves time and removes future time bombs. Because the chance that you find them all is zero percent. And you can’t create a system that is 100% testable by brute force means. So you have to go about it smarter. It will save time, money and improve the business outcome now and into the future.

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile. This post is Copyright © 2010 Successful Endeavours Pty Ltd.

Reducing Electronics and Embedded Software Product Development Costs

First some basic statistics that made me think about this issue a bit more:

  • Software development is responsible for 80% of the delays and complications associated with designing a new product. Source Embedded Forcast
  • 80% of embedded projects are delivered late. Source Embedded.com
  • Software typically consumes 80% of the development budget. Digital Avionics Handbook and Embedded.com
  • 80% of software projects are unsuccessful IBM

That is a lot of 80% figures associated with the software component of product development.

 

So working from the Pareto Principle it is clear that product development success and cost can be most improved by addressing the Software Development component. In my recent post on Reducing Electronics Manufacturing Parts Cost I argued that increasing the software component can reduce the hardware costs. Which is a great idea as long as it doesn’t introduce an even more expensive problem.

 

I agree with Jack Ganssle in his article looking at tools where he points out that software quality tools are often not budgetted for yet will find many classes of defect quickly and at a significantly lower cost than the test and debugging effort required to find them after integration with the rest of the project. Or put another way, the cheapest way to get rid of bugs is not to introduce them in the first place – Lean Coding.

 

Since we mainly develop in C and C++, this is what we do to ensure we minimise software development cost and overruns:

 

Static analysis and code reviews

We use static analysis and code quality tools such as PC-Lint and RSM and integrate them into our editors and IDEs so we can run the tests are part of our build or at the very least with a single click covering either the current file or the current project. These tools find flaws you are hard pressed to identify by visual inpection and I believe they pay for themselves within a month of purchasing them. They can also enforce coding standards. Another great benefit is that when you do a code walkthrough and review, you are not looking for these classes of faults explicitly because you know the toolset will find them for you. So the first thing you do is run the tests and focus on anything found there.

 

Code reviews save money. Every issue identified in a code review is an issue you don’t have to debug later on. And another person is going to look at your code without the same assumptions you would so they will see the things you miss. It just makes sense to do it. Software debugging is more expensive than coding so not bugging in the first place is good budget management.

 

Smart Bear Software have an excellent whitepaper you can download for free that covers best practices of peer code review and if this is a new idea to you, then I strongly recommend you get the whitepaper as they have distilled a lifetimes worth of learnign in this area into a concise and easily implementable strategy to improve code quality.

 

Unit testing

Next, we unit test. A huge benefit of this is that you have to think about test and it makes you think about error handling in the design phase. Many problems in implementing embedded systems come from not handling errors consistently. Sometimes they aren’t handled at all! In Failure is an option this gets explored a little. Someone else once suggested that software developers were the most optimistic people on the market – you can tell this is true by looking at how they handle exceptions! I’m not sure who said it so if you know then post a comment and I’ll credit them and provide a link too if you have one.

 

Integration testing

Integration testing itself does not have to be overly complex. You want to know that things work and it is often easier to write a cut down system to manage the test process. This way you are proving that each susbsystem is present and correct before doing the full scale system test. This is an area that often gets overcomplicated. Don;t try and do more here than you have to.

 

Oh, and by the way, just because something builds don’t mean it passes the integration test. Some things to cover are:

  • software manifest – do I have the right version of each module?
  • data flow – do the higher level calls get at the right data lower down?
  • exceptions – do error returns get passed back?
  • exceptions again – if you raise exceptions, do they get acted on?
  • communications – does it communicate?
  • IO – are they mapped to the right pins and peripherals?

Simulation

For some systems or subsystems we write fully fledged PC mocks around the code and ensure it handles all the parameter and error cases correctly and that all the functions are correctly implemented. This is a form of integration testing that proves the software component of the system is doing what it is meant to but goes a lot further to fully excercise part of it. And since 80% of the problems come from software this is a very effective way of reducing bugs and difficult to track down system defects that are expensive on time and resources to cover in real time operating tests.

 

To do this, you have to abstract the interface so the code can run in the embedded version or the PC version without any changes. This is easy to do if you think about it in advance.

 

One word of caution; the PC has a lot more resources and clock speed available compared to a smaller embedded system so this is not a substitue for testing on the real hardware to ensure execution latency is acceptable.

 

And for the purposes of this post, the PC could just as easily be a Linux or Mac system. The point is to use the higher level system to efficiently and fully test the embedded software module so you save time and money later on in the project. And let’s face it, who like to be under unnecessary pressure at the back end of an embedded software project?

 

System testing

If you think in advance about how to most easily implement the system testing then you can save a lot here as well. We put effort into deciding how the do the test process at the architecture design phase so that we have the data flow required to actually do the test. This can be as simple as having some extra parameters or calls available to be able to inspect the state of the system and the communications facilities to get at this data. Where possible 100% parameter range testing and 100% code coverage testing is very desirable. One thing this means is that you had better think about how you will create each error condition that must be handled!

 

Low Cost Software Development

Low Cost Electronics Manufacture relies on Low Cost Software Development. So make it a priority. The Pareto Principle says that it is the most important thing to get right.

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.

 

First some basic statistics that made me think about this issue a bit more:

  • Software development is responsible for 80% of the delays and complications associated with designing a new product. Source Embedded Forcast
  • 80% of embedded projects are delivered late. Source Embedded.com
  • Software typically consumes 80% of the development budget. Digital Avionics Handbook and Embedded.com
  • 80% of software projects are unsuccessful IBM

That is a lot of 80% figures associated with the software component of product development.

 

So working from the Pareto Principle it is clear that product development success and cost can be most improved by addressing the Software Development component. In my recent post on Reducing Electronics Manufacturing Parts Cost I argued that increasing the software component can reduce the hardware costs. Which is a great idea as long as it doesn’t introduce an even more expensive problem.

 

I agree with Jack Ganssle in his article looking at tools where he points out that software quality tools are often not budgetted for yet will find many classes of defect quickly and at a significantly lower cost than the test and debugging effort required to find them after integration with the rest of the project. Or put another way, the cheapest way to get rid of bugs is not to introduce them in the first place – Lean Coding.

 

Since we mainly develop in C and C++, this is what we do to ensure we minimise software development cost and overruns:

 

Static analysis and code reviews

We use static analysis and code quality tools such as PC-Lint and RSM and integrate them into our editors and IDEs so we can run the tests are part of our build or at the very least with a single click covering either the current file or the current project. These tools find flaws you are hard pressed to identify by visual inpection and I believe they pay for themselves within a month of purchasing them. They can also enforce coding standards. Another great benefit is that when you do a code walkthrough and review, you are not looking for these classes of faults explicitly because you know the toolset will find them for you. So the first thing you do is run the tests and focus on anything found there.

 

Code reviews save money. Every issue identified in a code review is an issue you don’t have to debug later on. And another person is going to look at your code without the same assumptions you would so they will see the things you miss. It just makes sense to do it. Software debugging is more expensive than coding so not bugging in the first place is good budget management.

 

Smart Bear Software have an excellent whitepaper you can download for free that covers best practices of peer code review and if this is a new idea to you, then I strongly recommend you get the whitepaper as they have distilled a lifetimes worth of learnign in this area into a concise and easily implementable strategy to improve code quality.

 

Unit testing

Next, we unit test. A huge benefit of this is that you have to think about test and it makes you think about error handling in the design phase. Many problems in implementing embedded systems come from not handling errors consistently. Sometimes they aren’t handled at all! In Failure is an option this gets explored a little. Someone else once suggested that software developers were the most optimistic people on the market – you can tell this is true by looking at how they handle exceptions! I’m not sure who said it so if you know then post a comment and I’ll credit them and provide a link too if you have one.

 

Integration testing

Integration testing itself does not have to be overly complex. You want to know that things work and it is often easier to write a cut down system to manage the test process. This way you are proving that each susbsystem is present and correct before doing the full scale system test. This is an area that often gets overcomplicated. Don;t try and do more here than you have to.

 

Oh, and by the way, just because something builds don’t mean it passes the integration test. Some things to cover are:

  • software manifest – do I have the right version of each module?
  • data flow – do the higher level calls get at the right data lower down?
  • exceptions – do error returns get passed back?
  • exceptions again – if you raise exceptions, do they get acted on?
  • communications – does it communicate?
  • IO – are they mapped to the right pins and peripherals?

Simulation

For some systems or subsystems we write fully fledged PC mocks around the code and ensure it handles all the parameter and error cases correctly and that all the functions are correctly implemented. This is a form of integration testing that proves the software component of the system is doing what it is meant to but goes a lot further to fully excercise part of it. And since 80% of the problems come from software this is a very effective way of reducing bugs and difficult to track down system defects that are expensive on time and resources to cover in real time operating tests.

 

To do this, you have to abstract the interface so the code can run in the embedded version or the PC version without any changes. This is easy to do if you think about it in advance.

 

One word of caution; the PC has a lot more resources and clock speed available compared to a smaller embedded system so this is not a substitue for testing on the real hardware to ensure execution latency is acceptable.

 

And for the purposes of this post, the PC could just as easily be a Linux or Mac system. The point is to use the higher level system to efficiently and fully test the embedded software module so you save time and money later on in the project. And let’s face it, who like to be under unnecessary pressure at the back end of an embedded software project?

 

System testing

If you think in advance about how to most easily implement the system testing then you can save a lot here as well. We put effort into deciding how the do the test process at the architecture design phase so that we have the data flow required to actually do the test. This can be as simple as having some extra parameters or calls available to be able to inspect the state of the system and the communications facilities to get at this data. Where possible 100% parameter range testing and 100% code coverage testing is very desirable. One thing this means is that you had better think about how you will create each error condition that must be handled!

 

Low Cost Software Development

Low Cost Electronics Manufacture relies on Low Cost Software Development. So make it a priority. The Pareto Principle says that it is the most important thing to get right.

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.