Sdílet prostřednictvím

World Class Testing

  • Článek

I've highlighted my take-aways from the "World Class Testing" section of Managing the Design Factory by Donald G. Reinertsen.  It's an insightful book whether you're optimizing your product line or designing a new product.  It's packed with empirical punch, counter-intuition, and practical techniques for re-shaping your engineering results.

Viewing Testing as an Asset

  • View testing as an asset not a problem.  If you don't, you'll likey have an underresourced and undermanaged test department.  
  • Testing is typically 30 to 60 percent of the overall dev cycle - treat it as a major design activity.
  • The mismatch between theory and real-world is often unexpected - test results have inherently high information content

Ways to Optimize Testing

  • Distinguish between design testing and manufacturing testing.  Manufacturing testing is done to identify mistakes in the manufacturing process. 
  • Design testing is done to generate information about the design.
  • Test at the level in the product structure where you can find defects most efficiently.
  • Identify what you're optimizing in your testing - expense, unit cost of impact, performance, or time. 
  • Use economic analysis to help choose what to optimize.
  • To reduce cost, eliminate duplicate testing, test at the most efficient subsystem level, automate testing processes, and avoid overtesting the product.
  • Avoid overtesting, by branching test plans.  If the product fails certain tests, follow different paths.  Don't blindly run the tests where the rest of the tests no longer have significance.
  • To reduce the unit cost impact of testing, you can eliminate product features that exist only to make the product easier to test, and use testing as a tool to fine tune product costs.  Sometimes a design improved through testing, is cheapter than a do-it-right-the-first-time design.
  • To optimize product performance, you can either increase your test coverage or enhance the validity of your tests.  When you increase testing coverage, it's more practical to test probable applied use vs. all the possible permutations (which is usually statiscially impossible or inefficient).  To improve your test validity, generate the same types of failures in your labs, that you see in the field.
  • To decrease testing time, increase the amount of parallel testing, use relability prediction, decrease testing batch sizes, or monitor and manage queues.  To use reliability prediction, begin downstream activities when you can predict that you will likely achieve your targeted reliability for the product.

What I like about this particular book, is that it doesn't prescribe a one-size fits all.  Instead, you get a pick list of options and strategies, depending on what you're trying to optimize.  It's effectively a choose-your-own-adventure book for product developers.

Comments

  • Anonymous
    February 02, 2007
    This sounds like some details behind a lesson I learned as a young Army Officer years ago. A plan is meaningless without a mission. (i.e. from grid 12345678, at 0500 tomorrow, move NNW for 5 kilometers in a defensive posture, Squad A will flank to the right and move another 2 kilometers while Squad B takes up a defensive position, blah, blah, blah...) Not particularly useful, right? Now add a mission.  (i.e. Secure hilltop 42 NLT 0545 tomorrow) Ok, now I have a clue, but what if something goes wrong and I fail to secure the hilltop, or I'm late? That is where lesson two comes in.  Missions and Plans are only useful until something goes wrong... then you need the "Commander's Intent".  (i.e. Secure hilltop 42 NLT 0545 tomorrow to ensure safe passage of the supply convoy over the bridge found there.) Ah-Ha!  Now I know that even if I secure the hilltop, the broader mission could still fail if, say, the bridge has been destroyed.  If the bridge is gone, I now know that this is a problem I should report.  Without the "Commander's Intent" I'd have never known that there was supposed to be a bridge in the first place! Such a simple concept... So rarely applied... So sad. -- Scott Barber President & Chief Technologist, PerfTestPlus, Inc. Executive Director, Association for Software Testing www.perftestplus.com www.associationforsoftwaretesting.org sbarber@perftestplus.com "If you can see it in your mind...     you will find it in your life."

  • Anonymous
    February 02, 2007
    J.D. Meier of patterns & practices fame has an interesting post on testing based on a book he recently