Improving Product Development Outcomes

In this post we will look at the Product Development Process and how to get improved outcomes. But first here is a fun graphic made from our logo.

Successful Endeavours - Making Electronics and Embedded Software Work

Successful Endeavours – Making Electronics and Embedded Software Work

Product Development Process

The Product Development Process is intended to reliably deliver new products for manufacture or distribution. This is a critical component of a Product Strategy where you are creating the product rather than sourcing it from a supplier. So you would think that it should be a highly optimised, well oiled machine that reliably delivers successful products. Alas that is not always the case. With 30 years of experience in Developing Products for a wide range of industries I have seen my share of projects handled well and not so well. Here are some general principles I have gleaned from my experience in Successful Product Development Projects:

  • Risks must be identified and managed. Track them and eliminate them as soon as possible.
  • Anything clever or tricky needs to be checked by someone else.
  • Everything else also gets checked. Design reviews, code walk-throughs and prototypes save time, money and heart ache later on.
  • Hold the timeline. Foster an attitude that slippage is not acceptable.
  • Test and check everything.
  • It’s not finished until no-one has to do another thing to it.

So six core principles. They are inter related of cousre. Let’s look at how these work out in practice.

 

Successful Product Development Principles

Lets look at how each of these priciples can be used to improve the likelihood of a Successful Product Development Project.

Risk ManagementRiskManagement

Risk Management is an old idea. Not surprising since risks have always existed. Did you know that during the Manhattan Project it was determined that there was a chance that a fission bomb could ignite the whole atmosphere ? Having got contradictory reports the argument was eventually settled by a report showing that although it was possible, it was unlikely. How comfortable would you feel running that risk ? Fortunately the average Development Project is dealing with much more mundane risks such as achieving Technical Requirements such as:

  • Power Consumption
  • Unit Manufacturing Cost
  • Performance Criteria

But the approach is still the same:

  • Identify the risk
  • Work out how to ameliorate the risk – reduce it – or eliminate it
  • Do tests to confirm the risk has been dealt with
  • Iterate until it is no longer a risk

Review the clever bits

Test Everything - Clever Design Needs Test

Test Everything – Clever Design Needs Test

Where possible, any particularly clever or tricky areas of the project need to be reviewed by someone not involved in the everyday work of the project. This is primarily to ensure that assumptions are challenged. If you can’t get an outsider to do the review, use a process like Six Thinking Hats by Edward De Bono which can allow team members to step outside their emotional and assumptive predispositions. Unchallenged assumptions are unmanaged risks.

 

Review the rest of the project

Test Everything

Review Everything

The astute amongst would have noticed that I am proposing everything gets reviewed. But the tricky bits get extra review. This section is for the regular bits. Reviews are an essential tool to find mistakes early and eliminate problems down the track. You don’t have to solve a problem you don’t have. Or as Jack Ganssle famously quipped “Skip Bugging To Speed Delivery“. The whole article refers to using Code Review and Design Review to find problems early and fix them so they don’t become much bigger problems later on. Imagine a scenario where a Software Bug causes an electric motor to try and spin backward every now and again and then corrected itself almost immediately. You would get a momentary shudder or jerk followed by correct motion and it would only happen every now and again. How would you determine that this was a software fault and where the fault lay? It could be symptomatic of any number of issues including Mechanical Design and Electrical Design. How about this similar real world case. I won’t mention the company, but their elevators had an Integer Overflow problem in the motor controller that caused the elevator to go in the wrong direction, about once a month, for half a floor. Very disconcerting to the passengers if they pressed up, and promptly dropped half a floor before then going up. Fortunately they found it and fixed it before it happened to someone at the top or bottom floor. All the Software Industry Metrics show for that for Software Development; Design Review, Code Review, Unit Tests and System Simulation save money and time. And yet in many projects they don’t happen enough or are done after the event as a Quality Assurance box ticking activity where they add mostly cost and little in the way of value. Lean Coding argues that you can reduce your Software Development Budget in particular by doing Code Inspections during the project as part of the Risk Management and Quality Management process. By reducing the bugging, you can reduce the debugging.

 

Stick to the Timeline

Project Development Timeline

Project Development Timeline

An attitude that the schedule slipping is normal can be very costly. Some examples of how to avoid this are:

  • Develop and Simulate the Software before the Hardware is ready
  • Prototype early and thoroughly
  • buy in IP where it makes financial sense – this can also reduce risk
  • get expert assistance with areas outside your competence
  • review regularly and honestly

As someone who has done a lot of team leading and project management, I have learned to ask about progress in more than one way. I find the following to be very common: Manager: “This module is estimated as 10 days of work to complete. How complete is it”? Developer: “About 80%”. Manager: “How many more days of work are required to fully finish everything”? Developer: “To fully finish everything, I would think 6 more days would cover it all”. The discrepancy is easy to spot. People estimate high on progress because they want to please. They also like to finish well so they tend to estimate conservatively on required effort. In practice the real answer lies somewhere between the 2 extremes. If the task had already consumed 6 days of effort then it is likely to run late. If you have ever built a house you might have experienced the knock on effect it has when one trades person doesn’t turn up and everyone else misses their scheduled action time because they are now waiting on a predecessor task, the trades person who has to come back again, before they can start their task. The same thing happens on projects. So fight hard to hold to the schedule. It is better to over resource a task (according to the plan) and get it done than to let everything and everyone slip which usually costs a lot more. Additionally, it is quite common that the later you are in the market, the lower the overall profit. So it is worth holding the schedule for this reason as well.

 

Test and Check Everything

Test Everything

Test Everything

This is another Risk Management related principle. Don’t assume it will be OK. Even if you have done it 100 times before, test it again this time. Make sure it really is OK. This ensures it really is 100% complete. This also implies that you are going to design things so they can be tested. Another principle. Design For Testability or somestimes called Design For Test. Do it. It will save you time, effort, money and sleep. Test Driven Development is another example of a Modern Development Methodology where you set up the test first then develop the product so it passes the test. If the Product Requirements change, you change the tests first, show that the old Product Design fails the test, then update the Product Design until it now passes the test.

 

It is not finished until no-one has to do anything else to it

Many tasks are called complete but they aren’t. The documents might be checked into the Revision Control System, also known as a Version Control System or Version Management System, but it isn’t complete until it is 100% tested, 100% integrated, 100% reviewed and 100% signed off and no-one has to do another thing. This also means that when tasks are identified that weren’t thought of in the original Project Plan, you then add them and don’t try and fiddle them into existing tasks. This is different to working out the fine detail of a task and realising it is under resourced or over resourced on the Project Plan. You also want the extra tasks visible on the Project Management Plan so when you do the next project you have evidence that they were required last time and can make allowances for them.

 

Trip Assurance for Developers

Satisfaction Guaranteed

Satisfaction Guaranteed

In marketing, the term Trip Assurance refers to the client having a clear expectation of this transaction or experience being a good one, just like every other one has been. I think we can begin to develop some of the same as developers whereby projects can be routinely good experiences and likely to be so each time.

 

This post is also available as an eZine article with Expert Author classification.

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile. This post is Copyright © Successful Endeavours Pty Ltd.

Reducing Electronics Manufacturing Parts Cost

This one is both easy and straight forward to understand.

 

Do as much as possible in software.

 

But it doesn’t stop there. Do as much as possible in software at every stage of the development. Here is how that pans out:

  • replace hardware with software that does the same function
  • verify operation using unit tests and system tests within a soft environment
  • do production test using on board software so the ATE is very simple
  • do field diagnostics with on board software to make the diagnostics as cheap as possible
  • do service and scheduled maintenance with on board software to minimise time and cost in these areas
  • where suitable, use a bootloader to allow in field upgrade of the software

If you don’t already know, ATE = Automated Test Equipment.

 

The best thing about making software the core part of each of these areas, is that the manufacturing cost of software is effectively the jig and the time to program and test the parts. Automation can be expensive, but if the device contains it’s own automation, then the production process costs plummet. A simplified example:

 

You have a device with 8 inputs and 3 outputs. You want to test all the inputs and outputs to make sure they work. The traditional approach is to have a production ATE which applies known loads to test points and then measures against a series of scheduled tests which are controlled from one of the major production test equipment and systems suppliers. It is not unusual to spend $50K on such a system even for a relatively simple device. If you don’t believe; add up the software toolset costs, the man hours spent designing then building then coding then debugging then commissioning, the opportunity cost of those man hours and the materials costs. It really does all add up.

 

Electronics Manufacture – lets look at the alternative

The test jig merely connects the outputs to the inputs with the appropriate loads in place. The device is programmed with its own ATE code that then goes through the test process including requesting a serial number, and communicates the outcome back to the system which merely records the time, date, serial number, product version and test results. It doesn’t matter if the inputs are analog or digital, the same philosophy can apply. And if there is a big mismatch in the inputs and outputs, then put a simple multiplexer on the jig and let the unit manage it’s own test sequencing.

 

Another bonus: you update the system but the interconnections remain the same however the test sequence would have required altering the ATE software. No need! The on board ATE sequencer does it automatically and you don’t have to alter the production process at all. It even tells you it is the new product and you didn’t have to touch a thing.

 

Of course there are classes of products that do need more than this. Processes like burn in and quality metrics based acceptance testing. But these are the 5% cases. The alternative approach outlined above covers the other 95% and at a cost which can be orders of magnitude lower. And you can always add extra features to the test jig if required and still let them be controlled by the unit under test.

 

Yet another bonus: self calibration! The unit can calibrate itself based on the test results. No need to support multiple different calibration techniques at the ATE. It just says “I read X” and the unit under test looks at this value and what it reads and uses the one calibration process that applies to it.

 

And this features in one of our earlier posts on Strategies To Be More Profitable as it applies to Low Cost Electronics Manufacture in Australia.

 

Now I know this is simplifying it to its core essential elements, but that makes it easy to see the advantages and how much you can leverage them.

 

Less Electronics Hardware = Less Cost

The same applies to the other areas mentioned above. Removing hardware and doing the same work in software is pretty obvious. Less parts usually leads to less cost. Above we looked at production line ATE. And the same concept can obviously be applied to field and service diagnostics.

 

Field and service diagnostics

So here is another scenario. Imagine you have a customer with a pump that isn’t pumping. What to check first? Easy, the simplest thing to swap out is the pump controller. So you send them a replacement pump controller. They pull the plugs, remove the device, put in a new one, and send the old one back under warranty. You send it to the manufacturer. They test it and there is nothing wrong and send it back to you. But it’s pretty grubby and not suitable for resale as brand new. Well maybe their test process isn’t up to scratch and it really wasn’t working in the field. Anyway, it was still the thing to try first since anything else is a much bigger job to swap out. But now you’ve got all the hassle, a potential dispute with the manufacturer and the pump might still not pump with the new controller. The score is basically NIL all round for this. Everyone loses.

 

Now imaging this: the customer rings you and you ask them to go and press the orange button on the side of the pump controller. It says via it’s LCD “Check Valve Reversed”. Aha. Not a pump controller problem at all. The customer calls the plumber and gets him to fix the installation. Done. You look good, the customer got timely service and you sure are going to recommend this pump controller to the next customer ahead of the ones that don’t do this.

 

For each product category, the equivalent of the above 2 situation exists. So will your product look this good if the customer has an issue. It can if you think about it, and the cost might be trivial. It might even cost less at manufacture, but it will always cost less in the long run.

 

And of course, if the product can have its software updated in the field, that saves a lot compared to having to return it to the manufacturer. Orders of magnitude this time.

 

So that looks at parts cost, production process costs and support costs.

 

Reducing Development Cost

The second of the bullet points is looking at development cost. The up front cost to get a working product. We do a lot of work with small 8 bit and 16 bit microcontrollers and the development environments often don’t give you a lot of facilities to find faults. It’s the combinations that get you. Stop when input A is on, output B is off and the variable C is exactly 122 so I can look at what’s going wrong with my code. Or you might have to pay a lot for an emulator with all those features. And of course you have to put the hardware into the exact state you want as well. How do you do that again? That’s right, either sea of pots and switches or some clever and expensive hardware test equipment.

 

What we do a lot is build the project inside a software clone of the final system. In the software industry this is called a mock. Then we can use our standard PC coding and debugging tools to create scenarios and test against them. You can test your logic in an automated way and you can put every possible input combination in and make sure it responds correctly. Robert Bosch Australia Pty Ltd is one of our clients and we have worked on a number of projects for them. For those who don’t know, the volume of Australian Electronics Manufacture they do at their Clayton Facility in Melbourne is very impressive. They design, make and export millions of automotive electronic control units (ECUs) to Europe, Japan and the USA. And the body electronics supplied by Bosch to the rest of the world is designed and made there. Great stuff guys.

 

So a simple example of how we use this in our projects with them is a battery charging system we did which was all in software. You will find reference to it from one of our Linked In recommenders Dale O’Brien who saw the process in action. Basically, the full suite of tests took a week in real time, the primary test sequence required 54 hours, one test required sub-zero temperatures and none of this was 100% coverage. Using a software mock of the system we were able to do all the testing in 15 seconds including tests specifically to ensure 100% coverage. That’s roughly a million time faster. Debugging at light speed! So we were able to address the logic and algorithm issues quickly and efficiently and have a very high confidence in the system. Final verification in real time with final hardware and a normal test platform confirmed the operation but it was 6 months later. So maybe we were really 25 million times faster.

 

Don’t get me wrong, I firmly believe in testing on the final hardware. After all, assumptions are one of the greatest dangers we face. But at least prove you did correctly implement your solution within the assumptions you did make first. Then when you learn something new you are only fixing one problem and not arguing about whether it was the assumption or the test that is wrong.

 

I feel a bit like I got on my hobby horse over that lot. But I really do believe this can make a huge difference.

 

OK, time to go and design some more products for low cost electronics manufacture in Australia 🙂

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile.

 

This one is both easy and straight forward to understand.

 

Do as much as possible in software.

 

But it doesn’t stop there. Do as much as possible in software at every stage of the development. Here is how that pans out:

  • replace hardware with software that does the same function
  • verify operation using unit tests and system tests within a soft environment
  • do production test using on board software so the ATE is very simple
  • do field diagnostics with on board software to make the diagnostics as cheap as possible
  • do service and scheduled maintenance with on board software to minimise time and cost in these areas
  • where suitable, use a bootloader to allow in field upgrade of the software

If you don’t already know, ATE = Automated Test Equipment.

 

The best thing about making software the core part of each of these areas, is that the manufacturing cost of software is effectively the jig and the time to program and test the parts. Automation can be expensive, but if the device contains it’s own automation, then the production process costs plummet. A simplified example:

 

You have a device with 8 inputs and 3 outputs. You want to test all the inputs and outputs to make sure they work. The traditional approach is to have a production ATE which applies known loads to test points and then measures against a series of scheduled tests which are controlled from one of the major production test equipment and systems suppliers. It is not unusual to spend $50K on such a system even for a relatively simple device. If you don’t believe; add up the software toolset costs, the man hours spent designing then building then coding then debugging then commissioning, the opportunity cost of those man hours and the materials costs. It really does all add up.

 

Electronics Manufacture – lets look at the alternative

The test jig merely connects the outputs to the inputs with the appropriate loads in place. The device is programmed with its own ATE code that then goes through the test process including requesting a serial number, and communicates the outcome back to the system which merely records the time, date, serial number, product version and test results. It doesn’t matter if the inputs are analog or digital, the same philosophy can apply. And if there is a big mismatch in the inputs and outputs, then put a simple multiplexer on the jig and let the unit manage it’s own test sequencing.

 

Another bonus: you update the system but the interconnections remain the same however the test sequence would have required altering the ATE software. No need! The on board ATE sequencer does it automatically and you don’t have to alter the production process at all. It even tells you it is the new product and you didn’t have to touch a thing.

 

Of course there are classes of products that do need more than this. Processes like burn in and quality metrics based acceptance testing. But these are the 5% cases. The alternative approach outlined above covers the other 95% and at a cost which can be orders of magnitude lower. And you can always add extra features to the test jig if required and still let them be controlled by the unit under test.

 

Yet another bonus: self calibration! The unit can calibrate itself based on the test results. No need to support multiple different calibration techniques at the ATE. It just says “I read X” and the unit under test looks at this value and what it reads and uses the one calibration process that applies to it.

 

And this features in one of our earlier posts on Strategies To Be More Profitable as it applies to Low Cost Electronics Manufacture in Australia.

 

Now I know this is simplifying it to its core essential elements, but that makes it easy to see the advantages and how much you can leverage them.

 

Less Electronics Hardware = Less Cost

The same applies to the other areas mentioned above. Removing hardware and doing the same work in software is pretty obvious. Less parts usually leads to less cost. Above we looked at production line ATE. And the same concept can obviously be applied to field and service diagnostics.

 

Field and service diagnostics

So here is another scenario. Imagine you have a customer with a pump that isn’t pumping. What to check first? Easy, the simplest thing to swap out is the pump controller. So you send them a replacement pump controller. They pull the plugs, remove the device, put in a new one, and send the old one back under warranty. You send it to the manufacturer. They test it and there is nothing wrong and send it back to you. But it’s pretty grubby and not suitable for resale as brand new. Well maybe their test process isn’t up to scratch and it really wasn’t working in the field. Anyway, it was still the thing to try first since anything else is a much bigger job to swap out. But now you’ve got all the hassle, a potential dispute with the manufacturer and the pump might still not pump with the new controller. The score is basically NIL all round for this. Everyone loses.

 

Now imaging this: the customer rings you and you ask them to go and press the orange button on the side of the pump controller. It says via it’s LCD “Check Valve Reversed”. Aha. Not a pump controller problem at all. The customer calls the plumber and gets him to fix the installation. Done. You look good, the customer got timely service and you sure are going to recommend this pump controller to the next customer ahead of the ones that don’t do this.

 

For each product category, the equivalent of the above 2 situation exists. So will your product look this good if the customer has an issue. It can if you think about it, and the cost might be trivial. It might even cost less at manufacture, but it will always cost less in the long run.

 

And of course, if the product can have its software updated in the field, that saves a lot compared to having to return it to the manufacturer. Orders of magnitude this time.

 

So that looks at parts cost, production process costs and support costs.

 

Reducing Development Cost

The second of the bullet points is looking at development cost. The up front cost to get a working product. We do a lot of work with small 8 bit and 16 bit microcontrollers and the development environments often don’t give you a lot of facilities to find faults. It’s the combinations that get you. Stop when input A is on, output B is off and the variable C is exactly 122 so I can look at what’s going wrong with my code. Or you might have to pay a lot for an emulator with all those features. And of course you have to put the hardware into the exact state you want as well. How do you do that again? That’s right, either sea of pots and switches or some clever and expensive hardware test equipment.

 

What we do a lot is build the project inside a software clone of the final system. In the software industry this is called a mock. Then we can use our standard PC coding and debugging tools to create scenarios and test against them. You can test your logic in an automated way and you can put every possible input combination in and make sure it responds correctly. Robert Bosch Australia Pty Ltd is one of our clients and we have worked on a number of projects for them. For those who don’t know, the volume of Australian Electronics Manufacture they do at their Clayton Facility in Melbourne is very impressive. They design, make and export millions of automotive electronic control units (ECUs) to Europe, Japan and the USA. And the body electronics supplied by Bosch to the rest of the world is designed and made there. Great stuff guys.

 

So a simple example of how we use this in our projects with them is a battery charging system we did which was all in software. You will find reference to it from one of our Linked In recommenders Dale O’Brien who saw the process in action. Basically, the full suite of tests took a week in real time, the primary test sequence required 54 hours, one test required sub-zero temperatures and none of this was 100% coverage. Using a software mock of the system we were able to do all the testing in 15 seconds including tests specifically to ensure 100% coverage. That’s roughly a million time faster. Debugging at light speed! So we were able to address the logic and algorithm issues quickly and efficiently and have a very high confidence in the system. Final verification in real time with final hardware and a normal test platform confirmed the operation but it was 6 months later. So maybe we were really 25 million times faster.

 

Don’t get me wrong, I firmly believe in testing on the final hardware. After all, assumptions are one of the greatest dangers we face. But at least prove you did correctly implement your solution within the assumptions you did make first. Then when you learn something new you are only fixing one problem and not arguing about whether it was the assumption or the test that is wrong.

 

I feel a bit like I got on my hobby horse over that lot. But I really do believe this can make a huge difference.

 

OK, time to go and design some more products for low cost electronics manufacture in Australia 🙂

 

Ray Keefe has been developing high quality and market leading electronics products in Australia for nearly 30 years. For more information go to his LinkedIn profile.