Why NPS isn't enough: the right evidence at the right time

You've launched your product, users seem engaged, and your Net Promoter Score (NPS) is climbing. Mission accomplished, right? Not quite.

While NPS provides valuable insight into user satisfaction, it's a poor proxy for actual educational impact. NPS tells you if users like your product, but not whether students are learning more, teachers are saving time, or administrators are making better decisions.

In my last piece, I wrote about how evidence is a key value driver for companies. But to unlock this value, companies need more than NPS scores.

Evidence as a journey:

Securing meaningful evidence is a journey, not a destination. Each type of research serves different needs at different stages of growth:

For early-stage products:

• Logic models articulate how your product contributes to specific outcomes and helps you set the right metrics to track progress.

• Feasibility and usability studies evaluate whether your intervention can be implemented as intended and identify implementation challenges.

As you scale:

• Correlational studies establish relationships between product usage and outcomes.

• Quasi-experimental designs (QEDs) compare results between users and non-users, helping show your product likely caused improvements.

• Randomised control trials (RCTs) randomly assign participants to either use your product or not, providing the strongest evidence that your product likely caused improvements. 

Across stages: 

• Rapid iterative testing through A/B testing and automated evaluations to provide feedback on feature effectiveness and content quality. At an early stage this can help validate core assumptions, at a later stage this can support optimisation. 

Timing matters. A founder I spoke with recently jumped straight to an expensive RCT before product-market fit was established. Unsurprisingly the results from the RCT were mixed, and this hindered company growth. Had they started with a logic model and correlational data, they could have refined their product, generated data to know they were on track and then committed to an impact evaluation with confidence. 

Research as a Health Check

Think of research as a regular health check for your product. The right-sized evidence helps you make informed decisions about product development, go-to-market strategy, and fundraising narratives. Internally tracking key metrics that give you some insight into impact, combined with independent research projects with experts for validation and an unbiased view can be a powerful combination. This type of approach helps you to know that you’re on track to deliver the impact you promise. 

The AI Advantage

The good news? Building evidence is becoming more accessible. AI tools like Consensus, Elicit, and Julius AI can help with evidence synthesis and analysis. New infrastructure tools like knowledge graphs and evaluators can help developers evaluate and improve the performance of the AI outputs used in their products. Even non-specialised tools like ChatGPT can jump start your research and evidence gathering activities. 

That said, having an expert in the loop remains valuable to navigate nuance and ensure methodological rigour. Your investment partner can be an excellent resource, or you can connect with specialised organisations like LeanLab Education, International Centre for EdTech Impact, EdTech ReCharge, Learning Collider, WestEd, or Learn Platform by Instructure for tailored guidance. This guidance could include being matched with an expert mentor able to advise you on metric definition, data collection and reporting practices, as well as support with designing and running a study. 

Remember: The right evidence isn't necessarily the most complex or expensive. It's the evidence that answers your most pressing questions at your current stage of growth.

In the next piece, we'll explore how to finance this evidence-building journey without breaking the bank.