When it comes to launching digital products, clients always ensure not to compromise on the QA testing of the applications. Currently, some of the software testing methods and engagement models have not been modified to meet the industry’s rapidly changing needs. Defective software can cost businesses dearly each year. Software companies often experience budget overruns well over 20% while producing market-ready software.

The testing services industry today has identified that ROI can be maximized while CAPEX and OPEX can be cut down significantly by focusing on improving the testing delivery value chain.

In this endeavor, the decoupling of the commercial aspects of testing from the headcount aspect and associating them directly with the quality of the testing outcome is important.

Striking a delicate balance while garnering a strong sourcing partnership that is a win-win for the key participants is pivotal to reaping the benefits of software testing services.

This blog will focus on the comparison between the two important testing models in the testing services industry, highlight their advantages individually, and then move on to the sharing economy in software testing, and an interesting hybrid model that has disrupted the software testing services industry recently.

What is an Output-Based Testing Model?

Put in simple words, an output-based testing model is the one in which pricing is directly linked to discrete units of output delivered by the test service provider (e.g., test coverage, requirement or function points, test cases, etc.) or units of consumption (resources) from the perspective of the customer (e.g., per ticket raised, per application used, per device used etc.)

Advantages of the Output-Based Testing Models

  • Knowing the pricing, QA effort-expected, and the associated SLAs at the point of work allocation
  • Improvement goals can be discussed on a continuous basis and vested interest is built to achieve those goals
  • Lesser time invested in value addition, future savings, and resources details because they have been translated into tangible pricing or SLAs
  • The service provider can increase output with the same team-size, or even reduce the team-size by using automation and improved processes. Customers benefit too, as SPs can offer yearly benefits, increasing their units of delivery per unit of cost

The shortcomings of the output-based testing model include efforts and risks of the initial calibration of tasks. In this phase, both the clients and the test service provider agree on a unit of testing deliverable (known as Quality Points), and on the rate for each unit to build efficiencies which may take many years to complete.

The requirement for reviews planning, performance tracking, and output-based pricing in line with the goals of outsourcing can sometimes hamper the quality and performance of testing.

In the output-based testing paradigm, the rate card only mentions prices for the test cases executed and delivered which may often neglect the quality of testing and the extent of test coverage. Test reports may not encompass a detailed account of the various levels of testing conducted on the product along with the KPIs.

Output-based testing has the dependability of repeated executions over multiple years with the same test service provider without regard to the quality maintained by the testers to deliver premium results. The onus of ensuring a quality outcome is still on the client in the output-based testing model and sometimes quality is not influenced by testing at all. The focus in output-based testing is always on increasing the output and the denominator is the cost of testers.

Why some Enterprises prefer this model?

  • Build more tests, automate more and execute more while spending Less
  • Use a professionally certified testing team and test center of excellence
  • Unit-based pricing which is directly proportionate to the number of test cases covered
  • Test more cases with reduced costs and produce more output
  • On-demand testing or minimum number of tests per month
  • More discounts with more testing volume

What is an Outcome-Based Testing Model?

An outcome-based testing model is a non-headcount-based model in which pricing is based on the measurable cost or revenue impact delivered by the test service provider to the customer which can help in developing a defect-free application (e.g., pay proportionate to performance, gain share model, etc.).

This model has three key characteristics,

  • Focus on business outcomes, instead of activities and tasks
  • The use of measurable performance indicators depend on the defined outcomes
  • A pricing model that includes rewards and risks to help incentivize the achievement of these outcomes, with delivery against those outcomes determining the amount of fees paid

This method gives control to Product Companies by helping them preserve Quality Engineering internally and Virtualize Trivial Testing.

The risks of efficiency may be covered by output-based testing models, but, the risks of testing quality are not. Outcome-based testing models maximize value in addition to testing predictability and accessing the right skills at the right time with the right pricing.

Clients are generally unable to map the benefits of procuring IT services from third-party providers to the realization of business objectives and pay for the quality of the results delivered. Test service providers look for limited ownership and accountability of test results relegating them to the role of just testing implementers and not trusted partners.

Outcome-based testing has introduced a new breed of “next-generation” buyers and the “best-in-class” test service providers with the “quality” of testing playing the pivotal role. Outcomes are measurable impacts delivered by the service providers that are measured by how useful, functional, and efficient the products are for the end-user along with the value-addition that they offer.

The outcome-based model is an emerging model in which the test service provider assumes the overall IT delivery responsibility and is accountable for the quality of the tests conducted on the applications.

Advantages of the Outcome-based Testing Model

  • Deliverables are evaluated by the agreed Metrics and KPIs
  • Delivering Business Value, Innovation, and Consistent Improvement are the key points of agreement between the clients and the test service providers
  • Higher customer satisfaction, improved product quality, predictable outcome, increased productivity, and cost efficiency are the perks of this model
  • The onus of test delivery and quality-related risks are moved from the client to the test service providers
  • Service provider performance is measured based on the business needs that testing services satisfy and the impact of the services delivered
  • Optimization and process improvements are proactively undertaken by the test service providers
  • Test service planning takes lesser time since the quality standards need to maintained by the service providers in line with already laid out guidelines
  • Costs of re-testing or expenses incurred due to application mishaps can be reduced to a great extent since the quality of testing and consumption-based pricing is given priority
  • Incentives to push the envelope beyond manpower-linked linear growth in outcome-based testing leads to better profitability over time with cost-savings
  • Up to 30% reduction in TCO for testing
  • Headcount is not considered in this model for pricing
  • Improved time-to-market is achieved through predictability and optimized testing cycle
  • Reduced Defect Rejection Ratio
  • Reduced defect leakage through multi-level testing is done using the right mix of manual and automation testing
  • Increased test coverage and reduced test cycle

Why most Enterprises today prefer this model?

  • QA and Testing services are delivered based on predefined quality and service levels, tailored to the clients’ needs
  • Concentrated efforts are achieved on each scope in parallel
  • Prevention of defects at reduced costs rather than reducing the defects for saving costs
  • Clients can take control over the QA spend-over commitments
  • Costs can be planned and predicted
  • For improving test quality, the applications can be tested on various devices by testers using multiple testing tools on demand
  • The true concept of crowdsourced testing can be materialized using the outcome-based model
  • DIY solutions can be built across various testing scopes
  • Quality control sans KPI/SLA can be realized using this model
  • Seamless testing can be done with non-linear costing
  • Reduced Turnaround Time, CAPEX, and OPEX

Output- and outcome-based service delivery and commercial models are likely to work well when the following conditions are satisfied:

  • The proposed work is standardized and can be offered by the service provider in a repeatable manner on a sustained basis
  • The scope of work can be decomposed into smaller units of work/consumption (e.g., tickets, applications, devices)
  • The proposed work is sufficiently large to allow economies of scale
  • The buyer is willing to transfer some control to the test service provider
  • The test service provider will have adequate rights to proactively optimize processes, technology, and testers

According to studies, significant secular shifts are reshaping the software testing sourcing landscape. The two trends that would influence the future of software testing are:

  • Managed services-based service delivery arrangements will gain prominence with value propositions of labor arbitrage, scale benefits and access to skills taking center stage
  • Non-headcount-based commercial models will gain momentum with preference given to providers that can take end-to-end ownership of testing and link price to the value offered

The Sharing Economy in Software Testing

As enterprises across industries face greater global economic uncertainty, changing market dynamics and disruptive technology (e.g., social media, mobile, analytics, and the cloud), clients of IT services are increasingly looking to enter into sourcing arrangements with service providers that are predicated on the ability to obtain predictable and measurable results. Businesses are no longer willing to let test service providers operate merely as suppliers of testing input.

The sharing economy in software testing is an economic model that developed as a result of this disruption and is often defined as a peer-to-peer (P2P) based activity of acquiring, providing, or sharing access to testing services that are facilitated by a community-based online platform.

Read more about Disruption in Software Testing on our blog.

Crowd testing implements outcome-based testing to deliver quality test results. But, the clients seldom have control over the testers or the mechanisms that they use to execute the tests.

Qualifly combines the two key concepts of software testing to provide curated crowdsourced testing solutions.

Curated Crowdsourced Testing (Outcome-based Software Testing Model + Selective Crowd Testing)

This method provides a Contractual guarantee for pre-defined quality and service levels. The client pays for actual testing deliverables and not for the testing resources. Seasoned test service providers are selected for the relevant tasks so that the clients can rest assured that testers’ skills, automation ROI, or testing schedule are taken care of. After the KPIs are set, the clients know exactly what they will receive for the cost.

Curated crowd testing services offered by Qualifly align interests and promote a collaborative culture of continuous quality improvement between the client and the test service provider while allowing them to individually tap into the software testing value chain.

The advantages of this hybrid model are:

  • It enables testing and exploits vulnerabilities of un-distorting demand and un-constraining supply
  • It enriches products with information, makes the testing efforts social, and transforms testing services while fundamentally improving the customer experience
  • Pay-as-you-use ensures that businesses do not have to pay for unutilized capacity
  • Test Service Providers can actively engage their unused testing resources based on the changing requirements of the client/application testing
  • Applications can be tested on-the-fly since testing resources are readily available
  • Value-based pricing for the outcome is incorporated in the pricing structure
  • Changes the supply side cost structure by automating, visualizing, disintermediating and becoming more predictive
  • Business teams can be leveraged in testing to generate additional income and increase utilization, un-lump capacity, and enlist new supply
  • Open Source tools can be leveraged using this framework
  • Avoids long term service contracts and uneven demand

To extract more from IT sourcing arrangements, clients and best-in-class test service providers must embrace value-based models like the one implemented by Qualifly that prioritize measurable and meaningful results over human resource-oriented inputs.

To know more about curated crowd testing that incorporates the outcome-based testing model, visit our website today.