Thumbtack currently runs about 30 A/B tests per month, ranging in duration from a week to six months. We experiment on virtually every area of our product — customer signup, pro signup, matching algorithm of customers with pros, messaging features, reviews features, SEO traffic, and many more. Our experiment analysis computes 600 different metric definitions defined by both analysts and engineers.
This blog post is in seven parts:
- Experiment Design
- Experiment Configuration
- Experiment Assignment Service
- A brief overview of our Data Platform
- Experiment Result Computation
- Metric Definitions
- Experiment Result Visualization Service
Here’s how these parts are connected:
About half of Thumbtack engineers will own at least one A/B test within their first 6 months.