I. Brief Summary
An overview of key metrics and benchmarks. By measuring and analyzing as you grow, you can validate whether a problem is real. This book shows how to validate an idea in every stage of your business, find the right customers, decide what to build, how to monetize your business. The authors have packed more than thirty case studies and insights from over a hundred business experts.
II. Big Ideas
- Ratios are good for comparing factors that are opposed or have some kind of tension. A good metric changes behavior.
- Types of metrics
- Qualitative/Quantitative
- Exploratory/Reporting
- Leading/Lagging
- Correlated/Causal
- We care about active users, because it probably lead-indicates our churn rate. This requires a lot of different queries to diff databases.
- Find commonalities. Analyze patterns of engagement and desirable behavior.
- Knowledge Matrix
- Known Knowns are facts that may be wrong and should be checked against data.
- Known Unknowns are questions we can answer with automated reporting.
- Unknown Knowns are intuitions we should quantify.
- Unknown Unknowns are located through exploratory reporting and where we can locate unfair advantages and new insights.
- Pivot hard or go home, and be prepared to burn bridges.
- Churn is a lagging indicator. Cohort analysis (comparing customer groups over time) is the road to leading indicators.
- Are the metrics we track helping us make better decisions faster?
- A segment is a group that shares a common characteristic. Segment visitors then compare segments to each other to understand differences in metrics. Look for disproportionate relationships.
- Cohort analysis allows patters to emerge across customer lifecycles.
- We can test anything, but focus on the critical steps and assumptions.
- Venn diagram—Expertise, Desire & Monetization. All 3 = Victory.
- E+D = Learn to Monetization
- E+M = Improve Desire
- D+M = Say No
- The One Metric That Matters (OMTM)
- Focusing on one metric above all others provides a uniquely powerful sense of focus across the entire company — a focus that is much needed in the early stages of a company’s life.
- Pick a metric from the riskiest portion of your business.
- OMTM gives you a single metric that answers the most important question you have.
- The OMTM forces you to draw a line in the sand.
- It focuses the entire company.
- It inspires a culture of experimentation.
- There’s a powerful example of an OMTM in this chapter: in the restaurant business, the one metric that matters is the ratio of staff costs to gross revenues for the previous day. When staffing costs exceed 30% of gross revenues, that’s bad because it means you’re either spending too much on staff, or not deriving enough revenue per customer. This ratio works because:
- It’s a simple number.
- It’s immediate—you can generate it every night.
- It’s actionable—you might not be able to renegotiate ingredient costs, change your menu, or redo your lease agreement immediately. But you can make changes to staffing or encourage upselling the very next day.
- It’s comparable—you can track it over time, and compare it to other restaurants in your neighborhood or category.
- It’s fundamental—it’s fairly well known, and reflects two basic facets of the restaurant business model.
- In Lean Analytics, they’ve decided to synthesize the findings from four frameworks in particular:
- Dave McClure’s AAAR Pirate Metrics
- Ash Marya’s Lean Canvas
- Sean Ellis’s Startup Growth Pyramid
- The Long Funnel
- The analytics framework presented in Lean Analytics takes many of the stages in the above four frameworks and combines them into a single framework with five stages:
- Empathy—you’re looking for a real, poorly-met need that can be found in a reachable market. Once you do so, you’re figuring out how to solve their problem in a way customers will accept and pay for.
- Stickiness—you’re looking for the right mix of products/features/functionality that will keep users around.
- Virality—you’re looking for ways to fuel growth organically and artificially.
- Revenue—you’re looking for a scalable and sustainable business with the right margins in a healthy ecosystem.
- Scale—you’re looking to scale up the business on all fronts.
- Find days where unsubscribe rate is high, then find out why. Need to tweak the unsubscribe process to get better resolution on this. Subs expire some time after cancellation, so a decrease in sub numbers on a given day is not necessarily indicative of anything. Find the action, not the result.
- Properly calculating churn—select a time period. Average the number of customers at the beginning and the end of the period. Divide the number of cancellations by this number. To increase data integrity, measure churn daily.
- User generated content—use the forums as a source of UGC and find ways to repackage it and syndicate it.
- Find out what's actually important to people. Get inside their head. Delve for this information aggressively.
- The best companies warehouse every possible data point about their site's interactions and use only the data they need. Rally (a software company) records everything from kernel-level performance to HTTP-based user gesture interactions between the browser and software. They can then correlate changes in site performance to user behavior and vice versa.
- Get a better understanding of what new visitors really do.
- Limit the company's vision to a 3-pronged strategy. Refresh the 3-year plan every 18 months. Align the entire company around the vision.
- For each C-Level strategic assumption, what are the 3 line-level tactics that can be used to survey, test, prototype, then fill/kill quickly?
- Hypothesis to test—most people unsubscribe because they don't need our service anymore, not because we're crappy. Limited data bears this out, but we need to conduct far better exit surveys.
- Product pricing has nothing to do with cost, and everything to do with what the customer pays and how they derive value from the product.
- Try an intentionally absurdly priced package to anchor high prices, as well as to see if anyone actually bites.
- Site load time matters a great deal. Spend a lot of time to get this down.
- Negative churn is the long-tail of brand awareness. We might convert some “Long time listeners, first time callers” after a year, or longer. Focus on customers that have been on our mailing list for a long time but are not subscribers.
- What kind of content do different traffic sources expect? Twitter Time on Site is disproportionately high. Why is this?
III. Quotes
- Don’t just ask questions. Know how the answers to the questions will change your behavior. In other words, draw a line in the sand before you run the survey.
- Don’t sell what you can make; make what you can sell.
- Your job isn’t to build a product; it’s to de-risk a business model.
- You need to know which aspects of your business are too risky and then work to improve the metric that represents that risk.
- Instincts are experiments. Data is proof.
- Customers are people. They lead lives. They have kids, they eat too much, they don’t sleep well, they phone in sick, they get bored, they watch too much reality TV.
- First, know your customer. There’s no substitute for engaging with customers and users directly. All the numbers in the world can’t explain why something is happening. Pick up the phone right now and call a customer, even one who’s disengaged.
- A good metric is a ratio or a rate. Accountants and financial analysts have several ratios they look at to understand, at a glance, the fundamental health of a company. You need some, too.
- If you’re going to survive as a founder, you have to find the intersection of demand (for your product), ability (for you to make it), and desire (for you to care about it).
- Quantitative data abhors emotion; qualitative data marinates in it.
- Correlation is good. Causality is great.
- If two metrics change together, they’re correlated, but if one metric causes another metric to change, they’re causal. If you find a causal relationship between something you want (like revenue) and something you can control (like which ad you show), then you can change the future.
- Markets that don’t exist don’t care how smart you are. — Marc Andreesen
- Sometimes what you need is a new market, not a new product.