The advent of agile methodologies and test-driven development has brought
software testing to the forefront of application development. Yet in today’s
harried rush-to-market development environment, organizations must find a
delicate balance between product release and product quality.
In Critical Testing Processes, the author distills knowledge
gained from 20 years of testing experience into twelve critical processes. These
include highly visible processes by which peers and management judge competence,
and mission-critical processes in which performance affects the company’s
profits and reputation.
After each process is introduced, the author demonstrates its use through an
engaging case study. Instead of cumbersome regulations, this book provides
checklists—lightweight, flexible tools for implementing process-oriented
testing, gathering metrics, and making incremental process changes. By
demonstrating critical processes in various organizational, operational, and
technological contexts, this book shows readers how to:Handle recurrent tests efficiently and consistently
Develop a cohesive, cooperative team that sidesteps redundancies
Build a reputation for reliability through the effective communication of
Define the focus of tests for maximum customer satisfaction and
Because testing is a collaborative process with the participation of staff
throughout the organization, the author discusses interpersonal and cultural
issues in depth. This book also devotes ample coverage to the often-overlooked
areas of planning and perfecting tests. Whatever your role in testing—from test
engineering to managing hundreds of test engineers—Critical Testing
Processes will offer valuable insights into what you do, why it’s
important, and how you can perform better.
Table of Contents
Who Should Read This Book?
Types and Relationships of Critical Testing
Critical Testing Processes in Context.
Sumatra, a Hypothetical Project.
On Using This Book.
Recognize Good Processes.
To Your Testing Success!
1. Start with the Big Picture: Put the Test
Role in the Broader Context.
A Testing Process.
Understanding Operational and Organizational
Introducing the Sumatra Project and the
SpeedyWriter Test Team.
Testing within the System Development
Organizing the Testers and the Testing.
Beyond Testing Process Context.
2. Focus the Effort: Analyze Quality
A Quality Risk Analysis Process.
Jamal Introduces the Team to Quality Risk
Recognize a Good Quality Risk Analysis
Involve the Right Participants.
Employ the Right Technique.
Separate the Vital Few from the Trivial
Find Potential Bugs before Testing.
Start at the Right Time.
Adapt to Change.
Getting People to Get Real about Risk.
Quality Risks from Various Sources.
Who Drives the Quality Risk Analysis Bus?
Eliminating the Appearance of Magic in the
Quantification of Risk.
Dealing with Variation in Ratings.
Convincing Stakeholders to Invest Time and
3. Gaze into the Crystal Ball: Estimate the
An Estimation Process.
Jamal Predicts the Future.
Using Rules of Thumb for Rough Estimation.
Effort, Duration, and Dependencies.
Test Environment Configuration.
Moving on to Dollars and Cents.
4. It's Not What It Costs, It's What It
Saves: Budget and Return on Investment.
Analyzing Return on Investment for
Bugs That Get Fixed-or Prevented.
Bugs That Don't Get Fixed-but Are Known.
Tests That Mitigate Risks.
Guiding the Project.
A Final Note on Return on Investment
Jamal Prepares the Bill-and the
Moving Forward, Then Looking Back.
5. From Estimate to Baseline: Obtain
Commitment to Realistic, Actionable, Truthful Estimates.
Selling the Estimate.
Jamal Makes His Case.
Recognize a Good Estimation Process.
Understand the Factors that Affect the
Paint a Realistic Picture of the Project's
Assign Responsibility for Action.
Predict with Honesty.
What Kind of Investment Is Testing,
It Might Look Like Magic-But It's Only an
The Nine-Month Rule.
Be Careful with Nonstakeholders.
When Selling Fails.
Safety- or Mission-Critical Systems.
6. Gain and Communicate Insights: Plan the
A Test Planning Process.
Jamal Has a Plan.
Beyond the Process: Key Considerations for the
Selecting Strategies for Testing.
A Special Case: Regression Risk Management
Understanding Test Environment Execution
Managing Test Systems and Environment
Test Subproject Risks.
Getting Everyone On Board.
7. From Suggestion to Commitment: Assemble
Stakeholder Support for Good Test Plans.
Jamal Makes His Pitch.
Recognize a Good Test Planning Process.
Establish Clear Criteria for Phases.
Establish Consensus, Common Expectations, and
Complete at a Reasonable Time.
Promote Reasonable Flexibility and
Produce Appropriate Documentation.
Provide Opportunities to Catch Errors.
Exploit Synergies with the Project, Development,
Build, and Integration Plans.
Transcending the Template.
Planning for Outsourcing.
When Key Players Don't Support the Test
Obeying the Law and Pleasing Regulators.
8. Bring on the Great Testers: The How and
the Who of Hiring.
A Team Building Process.
Variations on the Team Building Process.
The Sumatra Test Team Grows.
Identifying and Managing the Critical Test Team
Regarding Test Technicians.
Test Team Staffing Variations: Temporary
Assignment, Rotation, Farm Team, and Backwater.
Education, Training, Certification, and the
Professionalization of Testing.
Attitude Matters Too.
Beyond the Obvious Interview Questions.
From Adding People to Growing Skills.
9. Grow Excellent Test Teams: Skills,
Attitudes, and Career Paths.
Jamal and Lin-Tsu Discuss Skills Growth.
Using Skills Assessment as a Career Growth and
Team Management Tool.
Recognize a Good Team Building Process.
Operate on a Win-Win Philosophy.
Accurately Define the Job.
Provide the Employee with a Long-Term Career
Give Employees Appropriate and Fair
Retain Employees with Sufficient Education,
Experience, and Skill.
Allow Appropriate Specialization.
What Do Testers Need to Know-and What Can You
Using Contractors and Consultants.
10. Archimedes' Bathtub: Design and Implement
A Test System Design and Implementation
Emma Creates Stress on the Job.
Three Crucial Considerations.
Shifting to the Management Picture.
11. Fill the Bathtub: Test System Coverage
A Brief Survey of Test Coverage Analysis
Jamal Assesses Coverage.
Recognize a Good Test System Design and
Provides Capable Tools for Assessment of
Efficiently Implements, Uses, Reuses, and
Maintains the Test System.
Selects the Right Techniques.
How Many Conditions in a Test Case?
Retesting Third-Party Components.
Vaguely Defined Requirements, Usage Profiles,
Balancing Coverage, Schedule, and Budget.
12. An Essential Handoff: Manage Test
A Test Release Process.
The Big Build Hits the Test Lab.
Recognize a Good Test Release Process.
Provide Installable, Improved, and Stable Test
Produce Predictable, Timely, and Regular Test
Use Defined, Customer-Like Install and Uninstall
Give the Test Release a Name That It
Build from Carefully Selected, Coordinated, and
Who Owns the Test Release Processes?
Complexity of the System Under Test-and
Incomplete and Unrealistic System and Acceptance
13. Assess Quality: Execute Test Cases.
A Test Execution Process.
Test Tracking Worksheets.
The Attack on the Big Build.
Recognize a Good Test Execution Process.
Find the Scary Stuff First.
Produce, Gather, and Disseminate Valuable
Correctly Interpret Test Results.
Exhibit the Proper Attitudes.
Dealing with Extended Shifts and Outsourced
Accommodating Holidays, Cultures, and
Capturing the History on the Ground.
14. Where Quality Falls Short: Report
A Bug Reporting Process.
A Big Bug in the Big Build.
Beyond the Failure Description.
Recognize a Good Bug Reporting Process.
Effectively Communicate Useful
Describe One Symptom per Report and File One
Report per Symptom.
Draw a Clear Boundary Between Testing and
Separating Bugs, Features, and "So What".
Handling Bugs That Got Fixed by Accident and
Keeping Noise Out of the Signal.
Building Trust and Credibility with
Selecting the Right Bug Tracking Tool.
15. Illuminate the Fourth Element: Report the
A Test Results Reporting Process.
Jamal Defines a Dashboard-and Reports on the Big
Build's Test Results.
Recognize a Good Test Results Reporting
Deliver Useful, Credible, and Timely
Relate Test Status to Project Progress-and
Influence Future Progress.
Show Trends and Destinations in Quality over
Use Appropriate Mechanisms and Documents.
Tune the Communication to Each Listener.
Handling the Presentation.
The Bearer of Bad News.
An Ever-Changing Complex Story.
Maintaining a Passion for Quality While Giving a
16. Leverage Opportunities to Learn: Manage
Change and Its Effects on Testing.
A Change Management Process.
Jamal Makes a Case-and Accommodates.
An Interconnected Process.
Recognize a Good Change Management
Select the Right Changes in the Right
Balance Considerations of Features, Schedule,
Budget, and Quality.
The Complex Nature of Change Effects.
The Ripple Effects of Change.
Becoming a Roadblock-or Becoming Perceived as a
17. Return to the Big Picture: Perfect the
Recognize a Good Test Process.
Provide Valuable, Economical Services.
Pervade the Project.
Use a Mature Process.
Apply the Appropriate Level of Formality and
Employ Appropriate Strategies.
Successfully Introduce Tools.
Enjoy Continuous Learning and Inspiration.
How Can We Measure What We Can't Define?
Immature Development or Maintenance
"Addicted to Test".
Incremental Process Improvement.
Fine-Tune the Big Processes.
Identify and Remove the Big Process
Jamal Brown Looks Back-and Looks Forward.