Growth Experiments: Lessons From a Failed Attempt

I had the pleasure to talk at the wonderful WordCamp Bucharest about setting up a growth experiments process. Here’s a short recap of the talk, together with the slides from my presentation. Read on and enjoy! Once the video is live, I’ll add it to the post for easy listening.

---

I hadn’t felt this nervous at a meeting recently. My heart was beating, my hands were shaking, as I stood in front of people I knew well, respected, and liked. I took a glance around the room, cleared my throat, and said: “Uuh, guys, thanks for joining our first growth experiments meeting!”

10 months have passed since that Tuesday afternoon. Our process of doing experiments at Enhancv has changed tremendously. I wouldn’t say we’re extremely successful at it – there’s still a lot to be desired. But we’ve learned a lot and I want to share these lessons with you today.

How and why we started?

We began our experiments journey to speed up execution and learn more about the customer acquisition process quicker. Experiments are a great way to test new ideas and get a regular schedule of shipping them. It also keeps the mind of the team set on the concept of testing, measuring and analyzing. If you’re not familiar with the process, I’d highly recommend the book Hacking Growth that I reviewed recently as a primer on the topic.

To do all this, we started conducting biweekly experiments meetings where people could share ideas on acquisition. Once we had all suggestions on the table, we’d prioritize the ideas by doing a group ICE scoring. We’d discuss and assign a score from 1 to 10 for the idea’s impact, our confidence in the result and the ease of execution. Ordering all ideas by the average total score gave us the ideas to execute. We’d assign a lead responsible for the execution and we’d expect results within 2 weeks – by the next experiments meeting.

We killed experiments 3 months later

Despite all the effort, the process simply wasn’t working. In 3 months we got to execute 6 experiments. Depressing! Additionally, there was a lot of lost man-hours, especially during meetings. No matter how many times we reiterated the goal, there were roughly shaped ideas that lacked detail or that could never possibly be executed in 2 weeks.

Eventually, we decided to kill the whole idea.

The rebirth: how to do growth experiments right

I’m not famous for giving up easily! I wanted to get back on track with experiments, as I knew there was value there. But first, we had to make sure we update the process.Here are the lessons learned and changes made to our growth experiments process.

Get the right team on board

We started off our experiments by inviting the whole company. The concept was the more people took part, the more ideas we’d gather. The unintended consequence was that, since different people came to every meeting, the rules of the game weren’t clear.

We kicked off the second installment with a clear-cut core team. We don’t have what’s generally considered an autonomous growth team. Despite that, since the team is fixed, team members take ownership and make sure to involve anyone else they’ll need for execution.

Here’s what a typical autonomous growth team would look like, but feel free to start smaller with the core capabilities you need:

Make the rules clear

It is key to build understanding within the team of what good experiments are. We currently use three basic rules of thumb:

• The experiment has to be clear and specific.
• The experiment has to affect the current objective directly.
• The experiment has to be executed within 2 weeks.

These rules stand at the top of our experiments work file so that people can go through them when they suggest ideas.

It’s important to explain these and have all team members agree with them. This makes it easier to enforce rules and “disqualify” ideas from the experiments process. It was hard for me to tell people that their ideas weren’t really experiment ideas – but once we all had agreed what an experiment idea looks like, you can go and say “Guys, I think this one doesn’t really comply with our checklist, do you think we should remove it?”

Set a specific problem

At first, we had only decided we want to focus on acquisition. Having a broader topic will help us get more ideas, right? Well, yes… and no. We got more ideas, but they were just as detailed as our objective.

Once we moved to a very specific problem we wanted to solve, ideas got more detailed, too. To set the objective, we rely on data and involve key decision makers. We look at the issue, what leading metric represents it best and how we want to move the needle there. We went from “give me acquisition ideas” to “we need to increase the blog traffic from new visitors for our key countries by 20% this month”.

Do the work before the meeting

Meetings were a specific point we had to work on. Due to lack of knowledge, we left a lot of points for the experiment prioritization meeting and, combined with the bloat of people attending, it resulted in 90+ minute meetings.

We cut everything unnecessary from the prioritization meeting with a couple of policies:

• Every team member writes down their ideas in our experiments backlog file before the meeting.
• Team members do the ICE scoring for their ideas on their own. 90% of the time we have the same views on scores and we only discuss the cases when someone feels a score is way off.
• We rely on ICE scores as an indicator for feasibility, but they are not the end all and be all. We prioritize taking into consideration other factors, too.
• Experiments’ leads write down the results and conclusions of their experiments as soon as those end. During the meeting, we just go shortly through those.
• Once assigned, all leads must decide on their own which people they will need to execute the experiment and what metrics to use to evaluate success.

This results in the following agenda:

1. Discussion on experiment results and status – 10 min.
2. Discussion of the new ideas – 20 min.
3. Prioritization – 10 min.
4. Distribution of roles – 10 min.

I’m happy to report we almost always end on time now!

Ensure there’s cadence

Major projects like this are like health habits – they are hard to follow unless you build consistency. That’s why you should aim to do regular prioritization meetings. Even if there are no results to discuss or you don’t have a lot of slack to get a new experiment going, do the meeting. Put it down on everyone’s calendar as a recurring event. In that way there’s a deadline that keeps everyone on their toes, there’s clarity and there’s habit.

Once you build this habit you will raise your team’s velocity and you’ll start executing more and more. I love this piece by Growth Tribe that goes into numbers to explain why biweekly experiment periods are the best.

Set up experiments the right way

There’s no way around it, you’ll have to do some reading on statistics to get how experiments work. You need to be familiar with sample size, significance, and more. There are tools like Google Optimize that will do the heavy lifting for you, but you need to be able to interpret results, form hypotheses, and design experiments.

As for experiment design, we always try to think about the lean approach. Having just 2 weeks to execute and get results puts on a healthy constraint that’s challenging but almost always possible. Here’s a list of experiment types you might like to consider, in addition to the classic A/B tests everyone talks about:

To recap

The moment we decided to end our first experiments run I felt very disappointed. I knew this initiative can bring lots of results and I knew we were doing something wrong. Doing the process analysis helped a lot to see what we were doing wrong and how we can make things right. I’d love to hear what’s your experience with growth experiments – go down to the comments section and let me know!

And here are the slides:

[slideshare id=80555748&doc=growth-presentation-171007114420]