<img height="1" width="1" style="display:none" src="https://www.facebook.com/tr?id=1733895583540788&amp;ev=PageView&amp;noscript=1">
Cobloom logo

Pros and Cons of the Sean Ellis Product/Market Fit Test

By Ryan Law on Wed, Mar 23, 2016

Trying to scale too early can cripple your startup; too late, and a competitor will beat you to the punch.

So, to help you determine when it's time to hit the accelerator and scale-up your SaaS business, I'm taking a look at one of the most popular methods for calculating Product/Market Fit: the Sean Ellis test.

The Sean Ellis Product/Market Fit Test

With so many competing definitions, Product/Market Fit can be difficult to get to grips with; it's both essential to understand and extremely difficult to do so. 

It's for that reason Sean Ellis (DropBox, Lookout, Eventbrite, LogMeIn...) set out to create a straightforward test to help founders identify the crucial moment when their solution finds Product/Market Fit. 

Sean's simple PMF survey takes a step back from complex metrics, and instead focuses on the most revealing data source at any SaaS company's disposal: their users. 

How would you feel if you could no longer use [product]?

  1. Very disappointed
  2. Somewhat disappointed
  3. Not disappointed (it isn’t really that useful)
  4. N/A – I no longer use [product]

If over 40% of users respond to the survey saying they'd be "Very disappointed" to stop using your product, there's a great chance your solution has found Product/Market Fit. Why?

After comparing nearly 100 startups, Sean found that those that struggled to reach traction always scored under 40% on this particular test. In contrast, those that managed to gain strong traction always scored over 40%.

Applying this to your business, you have a quick and dirty way of gauging Product/Market Fit. With enough good-fit users (see below) to survey (Buffer determined that 40-50 responses should carry statistical significance), you can make an informed decision to either double-down on product development, or start to scale-up your sales, marketing and customer success.

"I generally recommend to survey the following:

  • People that have experienced the core of your product offering.
  • People that have used your product at least twice.
  • People that have used your product in the last two weeks."
- Sean Ellis

Download your free eguide: everything you need to know about Product/Market Fit

Pros

1) It's Super Simple

The simpler a metric is, the more likely it is that people will actually use it.

Even the best-intentioned of tech startups end-up with a huge dashboard of KPIs that look useful, without having any real understanding of how to actually use the metrics to help their business grow.

The Sean Ellis Product/Market Fit test sidesteps this problem - creating a metric that's easy to calculate, easy to understand, and easy to use.

2) It's User Focused

Despite the perceived complexity of trying to determine Product/Market Fit, at it's heart, it's pretty simple: we're trying to find out if people want our product.

"...most startups fail not because they don’t manage to develop and deliver a product to the market; they fail because they develop and deliver a product that no customers want or need." - Steve Blank

Sean's test hones in on the people that engage with your product. Instead of fixating on a ton of quantitative, context-less data, it helps founders touch base with reality, and find out how their users and customers actually feel about their product.

"Sometimes we lose track of our customers and instead focus on terms, such as “product/market fit” and if we were in closer touch with our customers we would know whether we have it or not." - Leo Widrich, Buffer

Cons

1) Intention =/= Action

This is a classic problem with any type of survey, from the Sean Ellis test to Net Promoter Scores (also used as a PMF indicator): there's almost always a disconnect between what people say they'll do, and what they actually do. 

There's no guarantee that survey responses, no matter how well intentioned, will actually foreshadow action. If a business bets their entire growth strategy on a single survey, they're exposing themselves to a whole lot of risk.

"While surveys are useful, I would much rather take bets on what people are doing, not what they say they would do or are doing.  

This is why I think about these surveys more of as a leading indicator of product market fit vs proof of product market fit."   - Brian Balfour

 2) It Indicates PMF, but Doesn't Guarantee

Sean's testing revealed that a score of 40% or higher correlates with Product/Market Fit - but as my Stats professor used to hammer into us every lecture, correlation does not equal causation.

A positive score on this test strongly indicates PMF, but it doesn't guarantee it. The market for your product is going to be nuanced and complicated, and it's entirely possible for a product to excel at this test, only to crash and burn during the attempt to scale.

Knowledge is Power

Any Product/Market Fit test needs to be viewed as an indicator of PMF, and not proof.

It's impossible for any test to generate a guaranteed Yes/No, because the question we're trying to ask is so complicated, and the variables we're trying to measure change so frequently.

As a result, the more information you have, the more you can de-risk the decision to scale.

Though imperfect, the Sean Ellis Product/Market Fit test plays a huge part in calculating PMF, allowing you to take the pulse of the people that will make or break your business: your users.

free eGuide: how to generate more saas leads

Wait! Get a PDF copy of this article so you can read or use it later

We'll email it to you straight away