When you're working on your product and launching new features you will often find yourself in a situation where you are not sure if launching a new feature is good or bad. This is where experimentation tools come in. Such tools allow us to run A/B tests in order to make decisions in a more rigorous scientific way. A/B tests as the name suggest allow us to compare two (or more) versions of a feature in a statistically significant way.
In this lab we are going to be using PostHog and two key tools to help us run our experiment:
You are going to take the Todo List App built with Next.js in the analytics lab that is already hooked up to send data to PostHog via the Javascript SDK. Remember that our app had this simple 4-step funnel where each step is a custom event that is explicitly logged:
In this lab we are going to be focusing on the first 2 steps of the funnel.
We have come up with a brilliant idea to improve the conversion rate from our landing page to the signup page.
Our experiment hypothesis is that by changing the Call To Action button color from blue to green more people will click on it and thus more people will use our product because green is more clickable than blue.
For an early-stage product you're probably going to want to be focusing more on types of experiments with high potential impact. This is because:
Ok, now let's implement this experiment! ๐
First, make sure you have a PostHog account configured. You will need the key and host values in order to send data to PostHog. If you haven't configured an account already, follow the configuration steps in the Analytics lab.
Then make sure to update your .env.local file at the root of the Next.js project:
NEXT_PUBLIC_POSTHOG_KEY=TODO NEXT_PUBLIC_POSTHOG_HOST=TODO
The Feature flags tool allows you to toggle different features in your product on or off, and also precisely control what groups of users get to see these features turned on or off.
This is a necessary capability we need for running A/B tests because we need to consistently show/hide a feature to a group of users in order to run our experiment. You can also use feature flags when rolling our new features in an incremental way but we will focus on the experiment use case for the purposes of this lab.
You can go to the Feature flags section to see all feature flags. Note that in PostHog, if we want to link the flag to an experiment we need to create it from the Experiments section instead. So don't create any flags from here, we will manage the experiment flag from the Experiments section instead.
For now, just have a brief look at the Feature flag documentation.
The Experimnts tool allows you to setup and run A/B tests.
First, briefly familiarize yourself with the Experiments documentation. You might need to come back here.
In order to create your first experiment use the New experiment button and configure your A/B test.
Configure the experiment with a name, feature flag, description, variants and then hit Save as draft.
Now configure the experiment goal and minimum detectable effect.
For the Experiment goal we want to use a Funnels type metric and look only at the LandingPage to Login step. Also, for our example purposes set the conversion window to 1 minute and uncheck Filter out internal and test users.
For Minimum detectable effect set 10%.
Now hit the Launch button at the top-right to start your experiment. The results section will show an empty state initially until experiment data is received.
Now let's add the necessary code to our app to conditionally display the blue/green button based on the feature flag:
Now open some incognito browsers repeatedly and simulate some user behaviors. You should eventually see experiment data coming into the results section.
Use this Todo List App built with Next.js as a starting point: se-experimentation-lab-starter.zip
It's essentially the same Todo List App you built in the Analytics lab with some comments where you need to add the feature flag code.
1. Configure the app so that it sends data to PostHog succesfully (key and host environment variables).
2. Get some events into PostHog if you haven't already, at least the LandingPage and Login events.
3. Create an experiment and linked feature flag for our blue/green button experiment.
4. Implement the feature flag in our app so that it conditionally renders a blue/green button based on the feature flag.
5. Launch the experiment and generate some test data.
6. Inspect the funnel data in the results section.
7. [Bonus] Deep dive into the methodology behind A/B testing: Traffic allocation, Sample size and running time, Experiment significance.
1. Run your app in incognito mode to avoid any interference from browser plugins like ad blockers that might prevent event data from being sent to PostHog.
2. If you don't see data in PostHog immediately please allow a few minutes for PostHog to do some post-processing. Also, the EU region seems to be faster than the US region. Consider selecting EU when you create your account.
3. Try doing a shift+refresh to force a hard page refresh of your app. In some cases, after you make some changes, this might be needed.
Please take a minute to fill in the feedback form for this lab.