One useful way I find of thinking about startups, and their pace of iteration, experimentation, morphing, and pivoting, is that they are an experiment to test the hypothesis of a product <-> market <-> business-model relationship. As such, we can treat them with the scientific method, and bring to bare the tools we’ve learnt from other scientific endeavours.
A hypothesis exists within a network of supporting hypotheses, many of which are just accepted as assumptions, but all of which are subject to scrutiny and testing.
In a physics experiment to test the resistance of a piece of metal, the supporting hypotheses would include: ohms law, electro magnetism, and that multimeters work and are accurate (enough). The first two are pretty safe assumptions given the long history of predictions and experiments that have failed to disprove them, including the fact that we have electricity powering the computing device you are reading this on. The last assumption is a bit weaker, but through some experiments, such as measuring known things, such as batteries, and resisters, we can satisfy ourselves that this holds, or that we need to replace it with working metre.
In a startup the supporting hypotheses are far messier, numerous, and overlapping, such as:
- How will the potential user express their problem
- What will the potential user expect a solution to look like
- How important is this problem to the user
- What are the key feature/integrations that will block a potential user from trying the product
- How the user values their time
- How the user values the problem
You should consider doing the following to avoid deceiving yourself with the data you collect:
- Preregister you hypothesis - what do you expect to see from the data
- Be clear about what you are measuring, and why you think it is a good proxy for the effect you want to understand. This includes being clear about what counts, and what doesn’t. e.g., what does user activation actually mean?
- Don’t reuse data collected for another purpose, as it may be more narrowly collected and bias the outcome
- Run the best type of experiment you can for where you are at, in terms of sample size, controls, etc
- Try to understand how confident you can be in the results, considering the method of data collection (online forms for instance may only attract extreme responses)
Expect that most experiments will not give a black and white answer, because of the limitations in the experiment, but also because the system is messier than you are expecting.
Remember, a successful experiment is one that has a result, not one the confirms your hypothesis/prejudices. A result of no impact is successful experiment that says “this is a dead end”, within the limits of the experiment.