Differences
This shows you the differences between two versions of the page.
|
se:labs:07 [2024/11/26 16:01] cosmin.dumitrache |
se:labs:07 [2024/11/28 16:35] (current) cosmin.dumitrache |
||
|---|---|---|---|
| Line 4: | Line 4: | ||
| ==== Introduction ==== | ==== Introduction ==== | ||
| - | When you're working on your product and launching new features you will often find yourself in a situation where you are not sure if launching a new feature is good or bad. This is where experimentation tools come in. Such tools allow us to run A/B tests in order to make decision in a more rigorous scientific way. A/B tests as the name suggest allow us to compare two (or more) versions of a feature in a statistically significant way. | + | When you're working on your product and launching new features you will often find yourself in a situation where you are not sure if launching a new feature is good or bad. This is where experimentation tools come in. Such tools allow us to run A/B tests in order to make decisions in a more rigorous scientific way. A/B tests as the name suggest allow us to compare two (or more) versions of a feature in a statistically significant way. |
| - | In this lab we are going to be using [[https://posthog.com/|PostHog]] and using two key tools to help us run our experiment: | + | In this lab we are going to be using [[https://posthog.com/|PostHog]] and two key tools to help us run our experiment: |
| - Feature flags - allows us to turn a feature on or off to specific groups of users | - Feature flags - allows us to turn a feature on or off to specific groups of users | ||
| - Experiments - allows us to run A/B tests and get statistically significant results | - Experiments - allows us to run A/B tests and get statistically significant results | ||
| Line 20: | Line 20: | ||
| We have come up with a brilliant idea to improve the conversion rate from our landing page to the signup page. | We have come up with a brilliant idea to improve the conversion rate from our landing page to the signup page. | ||
| - | Our experiment hypothesis is that by changing the Call To Action button color from **blue** to **green** more people will click on it and thus more people will use our product because blue is more clickable than green. | + | Our experiment hypothesis is that by changing the Call To Action button color from **blue** to **green** more people will click on it and thus more people will use our product because green is more clickable than blue. |
| {{:se:labs:posthog-ab-test.png?600|}} | {{:se:labs:posthog-ab-test.png?600|}} | ||
| Line 29: | Line 29: | ||
| For an early-stage product you're probably going to want to be focusing more on types of experiments with high potential impact. This is because: | For an early-stage product you're probably going to want to be focusing more on types of experiments with high potential impact. This is because: | ||
| - As an early-stage startup you probably will not have a big enough volume of users to get statistically significant data in a reasonable amount of time unless the experiment makes a big impact (in which case you will be able to make decisions with less data). | - As an early-stage startup you probably will not have a big enough volume of users to get statistically significant data in a reasonable amount of time unless the experiment makes a big impact (in which case you will be able to make decisions with less data). | ||
| - | - Your product is not mature so it makes more sense to work on things with high impact first. Making improvements with low impact is not the optimal path to finding product/market fit. So best to defer these for later. | + | - Your product is not mature so it makes more sense to work on things with high impact first. Making improvements with low impact is not the optimal path to finding product/market fit. So it's best to defer these for later. |
| </note> | </note> | ||
| Line 52: | Line 52: | ||
| {{:se:labs:posthog-feature-flags.png?600|}} | {{:se:labs:posthog-feature-flags.png?600|}} | ||
| - | You can go to the **Feature flags** section to see all feature flags. Note that in PostHog, if we want to link the flag to experiment we need to create it from the **Experiments** dashboard instead. So don't create any flags from here, we will manage the experiment flag from the experiment section instead. | + | You can go to the **Feature flags** section to see all feature flags. Note that in PostHog, if we want to link the flag to an experiment we need to create it from the **Experiments** section instead. So don't create any flags from here, we will manage the experiment flag from the Experiments section instead. |
| For now, just have a brief look at the [[https://posthog.com/docs/feature-flags|Feature flag documentation]]. | For now, just have a brief look at the [[https://posthog.com/docs/feature-flags|Feature flag documentation]]. | ||
| Line 64: | Line 64: | ||
| In order to create your first experiment use the **New experiment** button and configure your A/B test. | In order to create your first experiment use the **New experiment** button and configure your A/B test. | ||
| - | Configure the experiment with a name, feature flag, description, and variants and then hit **Save as draft**. | + | Configure the experiment with a name, feature flag, description, variants and then hit **Save as draft**. |
| {{:se:labs:posthog-experiment.png?600|}} | {{:se:labs:posthog-experiment.png?600|}} | ||
| Line 75: | Line 75: | ||
| <note tip> | <note tip> | ||
| - | Note that you will need have some **LandingPage** and **Login** events already recorded in order to be able to select these. Have a look at the [[https://ocw.cs.pub.ro/courses/se/labs/06|Analytics lab]] if you can't find any such events for your funnel. | + | Note that you will need to have some **LandingPage** and **Login** events already recorded in order to be able to select these. Have a look at the [[https://ocw.cs.pub.ro/courses/se/labs/06|Analytics lab]] if you can't find any such events for your funnel. |
| </note> | </note> | ||
