What are some of your favorite tools for running great experiments

At Meta/Facebook we're lucky enough to have really good internal infrastructure for setting up experiments and evaluating results. So in recent years I've come to rely on these internal tools.

At other companies, I've used analytics and optimization services like:

1. Google Analytics/Firebase

2. Optimizely

3. Mixpanel

4. ApSalar (now Singular)

5. I would also expect great things from Statsig, but haven't used it yet myself.

A few other points about planning experiments and your iteration strategy:

1. Learn about statistical significance and educate your stakeholders on it.

2. On early stage products or those with smaller active user bases, you may need to drive more people through your funnels specifically to reach stat sig results.

3. You can run a/b tests without code. I've used survey tools like Survey Monkey, Qualtrics, and Mturk (now deprecated) to test variations of design mocks before deciding which one to build and test in production.

4. I've seen startups falter when people in positions of power required they approve every experiment. To get good results, define your goals and metrics, then allow team members the freedom to test their ideas.

5. A/B tests and analytics will tell you what people are doing, but not why they're doing it. You should also use qualitative research techniques to form hypotheses about pain points and identify opportunities to improve.

6. Evolutionary experiments can lead you to a "local maxima." They won't reveal whether a much higher ceiling for performance exists with a boldly different approach. So talk to customers, do contextual inquiry, listen in on support calls, and build deep customer empathy within your team.