Outcome metrics for product desirability

Outcome metrics for product desirability

A solution to merging business metrics with things humans care about

Measuring the users’ experience with a product is fraught with difficulties. There are two traps—business metrics or dubious quantitative measures like SUS, NPS, C-Sat, or CES. Jared Spool developed “Outcome Metrics” to combat this problem. These enable product design praxes ensure smooth product interactions.

Outcome Metrics answer three questions that any strategy must address

  1. Has a goal been achieved?
  2. How far do we need to go to achieve a goal?
  3. What obstacles exist that prevent the goal from being achieved?

Spool is careful to note that every strategy is different, thus the metrics that one business chooses will not make sense for every business. There are no universal metrics important to every organization. Instead, product teams discover UX outcomes in the course of ordinary qualitative research.

Great products combine Desirability, Feasibility, and Viability; the three lenses of human-centered design. Business metrics govern Viability and agile measures Feasibility. UX outcomes show how well individuals overcome obstacles and achieve their goals— Desirability.

Defining UX outcomes

The UX Outcome answers this major question.

If we do a fantastic job delivering this product (or service or feature), how will we improve someone’s life (Spool, 2020)?

There are two ways to look at UX outcomes. The first is the three types of goals outlined in About Face:

  1. Life Goals illuminate high-level aspirational ontology.
  2. End Goals concentrate on achievement of some task or level and
  3. Experience Goals that describe how someone wants to feel.

The second comes from philosopher James K. A. Smith who describes what he calls “habit forming practices.” These are another way to get at UX outcomes. He classifies habit forming practices as either thick or thin depending on the significance to our identity. Thin habits are routines that have little to do with our identity such as brushing our teeth. They align with the tasks and could be an “end goal,” though these are generally activities that a product enables rather than at which it aims. Thick habits are closer to who the person wants to be, like riding public transportation. Thick habits are ripe for the types of goals that designers consider in creating a product.

Designers can leverage these epistemological lenses to formulate UX outcomes.

Components of UX outcomes

Four components point towards the set of UX outcomes that a product sets out to achieve.

  1. Success metrics
  2. Progress metrics
  3. Problem-Value metrics
  4. Value Discovery metrics

Success metrics

By nature, UX outcomes are intangible. Success metrics further define how to tell when that outcome is achieved. It is important to be precise here. This is where measures such as SUS and NPS fall short. An NPS score of 50 is probably better than 40, but what do “40” and “50” signify in reality? They are relative numbers whose only connection to the real world are users, those imprecise appraisers of a singular experience. The context they are operating under is unclear. The past knowledge and experience they bring with them also is not shared with the designers. In short, their goals are unknown.

Success metrics overcome this by focusing on when the user achieves something significant. Jared Spool points to the example of a mobile ordering app a restaurant might use. Think of the outcome as fewer angry customers. Research reveals that many customers are angry due to cancelled orders. The restaurant has to cancel orders when particular ingredients run out. Some success metrics the design team might consider include

  • The number of angry calls the restaurant gets
  • The number of items out of stock not reflected as such in the mobile system, or
  • The number of orders that the restauranteur rejects due to out of stock ingredients.

Progress metrics

Progress metrics measure how far the product has come on the way to delivering the success metric. There are many ways to look at progress and measurement may take various forms. These are some of the most common.

Journey maps

Journey maps are indispensable to build the bridge between Personas and Scenarios. Another potential use is in seeing the level of support users have at which stages in the journey. It may also help you see the reach of your product across an ecosystem map if the product covers a wide region.

Progress Frameworks

Bob Moesta, a JTBD adherent, measures consumer progress through a series of steps from “first step to “satisfaction.” Tony Ulwick, on the ODI side of JTBD, lays out 8 steps that all customers move through.

  1. Define/Plan
  2. Locate/Gather
  3. Prepare/Organize
  4. Confirm/Verify
  5. Execute
  6. Monitor
  7. Modify
  8. Conclude

Amy Jo Kim has a different take called “Mastery Progress” that outlines 4 steps of adopting a new system. This can be advantageous for non-consumer applications since it is about adoption rather than getting the sale.

  1. Discovery
  2. Onboarding
  3. Habit-Building
  4. Mastery

Strive to make each nexus point in the journey delightful. Measure the degree to which the product achieves this to communicate progress.

Attitudinal Measurements

Sean Ellis’ Product/Market fit question is helpful here to draw a bead on where the product is within the marketplace. What sets this apart from NPS and similar metrics is it tells how desirable the product is. Though the survey asks additional questions, the main one is, “How would you feel if you could no longer use this product?” Sean Ellis and Rahul Vohra have found that if 40% of respondents answer “very disappointed,” the product has growth potential.

Problem-Value metrics

UX outcomes measure product desirability. Problem-Value metrics illuminate obstacles, assigning a dollar value to frustration with the product. Designers observe users to see the problem, then implement or utilize existing metrics to tell when and how often the problem occurs. An example is additional customer service calls due to a usability error in the system or time waste.

There are a couple ways to look for these Problem-Value metrics during the research phase. First, Lean Manufacturing has 8 lenses that may be helpful to detect costly waste denoted by the acronym WORMPIIT.

  • W – Waiting
  • O – Overproduction
  • R – Rework
  • M – Motion
  • P – Over-Processing
  • I – Inventory
  • I – Intellect unrealized
  • T – Transportation
    Though all may not apply to strictly digital properties, they suggest ideas to investigate.

The Interaction Design Foundation adds 5 ROI gains from effective design work that could help spot obstacles when they are not performing.

  • Boost overall revenue or conversion
  • Lower costs
  • Reduced development waste
  • Increased customer satisfaction
  • Reduced risk of building the wrong thing

Value Discovery Metrics

With Value Discovery Metrics, the switch flips from measuring performance to measuring opportunity. Value Discovery Metrics analyze existing data identifying opportunities for further growth. It looks across various data sources and attempts to tie them together to drive a greater opportunity for the user and the business. An example Spool uses is Amazon’s “Customers also bought” product suggestion engine or Spotify’s “Discovery” playlists. They realize a unique experience for the user through combining various data streams.

UX outcomes spotlight a product’s performance and suggest ways to improve

Implementing a UX outcomes strategy allows designers to measure a product’s Desirability. Each organization is different, therefore each organizations UX outcome metrics must be different. The customers of one company may not be the same customers as another store in the same way. UX outcomes offer a way to find the specific measures that make sense for your organization. These include Success metrics, Progress metrics, and Problem-Value metrics, and Value Discovery metrics. Together, they provide a realistic look at your users and their experience with your organization.


For more information

Sources

Cooper, A., Reimann, R., Cronin, D., & Noessel, C. (2014). About face: The essentials of interaction design (Fourth edition). John Wiley and Sons.

Smith, J. K. A. (2009). Desiring the kingdom: Worship, worldview, and cultural formation. Baker Academic.

Spool, J. (2020, May 19). Article: Why UX Outcomes Make Better Goals Than Business Outcomes. Leaders of Awesomeness. https://leaders.centercentre.com/posts/6453848

References & Related Notes

References to Getting to know Outcome Metrics

  • [[Personas]]
  • [[UI20 – Scenarios – Kim Goodwin]]
  • [[Scenario Creation]]
  • [[Journey Mapping]]
  • [[Mapping the Ecosystem]]
  • [[Hone Your Observation Skills]]
  • [[Ethnography]]
  • [[Lean’s 8 Forms of Waste]]
  • [[The Mindset Shifts Your Stakeholders Need for Implementing UX Metrics]]

Based on these Notes

[[UX Outcomes]]
[[Desirability, Feasibility, Viability Venn diagram]]
[[My Reflections on UX metrics]]
[[Persuasive UX Metrics – Day 1 – UX Outcomes]]
[[Persuasive UX Metrics – Day 2 – UX Success Metrics]]
[[Persuasive UX Metrics – Day 3 – UX Progress Metrics]]
[[Persuasive UX Metrics – Day 3 – UX Progress Metrics/Problem-Value Metrics]]
[[Persuasive UX Metrics – Day 4 – Value Discovery Metrics]]
[[Putting Your UX Metrics Game Plan to Work: Create an Action Plan to Start Measuring What’s Important]]
[[5 Proven ROI Gains from UX]]


Leave a Reply

Your email address will not be published.

This site uses Akismet to reduce spam. Learn how your comment data is processed.