Truth and Metrics
Scott Maxwell has posted a list of 10 ways to lie with metrics on his blog which Juice Analytics summarized:
1. Only present metrics that are positive. That’s why you collect all those metrics.
2. Only present metrics that are easy to manage.
3. Use many metrics.
4. Be extremely precise with your numbers.
5. Present quickly, drown ‘em with data.
6. Say “you don’t break down metrics” if they aren’t flattering to you.
7. Put lipstick on that pig–apply lots of gloss to your charts. Hello, Crystal XCelsius!
8. Show off your bona fides by sharing some metrics “off the cuff”.
9. Prep your team by feeding them lines.
10.Your job isn’t to educate your audience about your metrics. If people don’t know what you’re talking about, it’s because they’re stupid.
I work with data every day and I’m probably guilty of some of these points myself. However, I don’t necessarily think that all of them are “lies” — some are really a matter of client/personal preference. The biggest challenge is to get people to think about measurement and ROI — in fact, most recent surveys reveal that most managers and executives have no idea what their return on advertising is even though the tools are available to help do that.
I personally do tend to use many metrics (#2) to provide insight into performance and behavior from multiple angles. You don’t just look at sales, you look at average order value, margins, sales categories, sales conversion rates, shopping cart abandonment, etc. in order to get a complete picture of what’s happening. Multiple metrics are needed to go beyond the “what” and get to the “whys” and “hows”.
Fortunately, our agency culture is about accountability and we are very open about how we measure with our clients. We also make sure we take the time to educate and understand what are the measurements that they care about in order to tailor our reports accordingly. At the same time, we do pride ourselves in being precise (#4) and prepared (#9) but not to cover up any deficiencies but because our clients expect us to know the data intimately and to be ready to present it clearly.
My personal advice to those who are victims to #1, #5, and #7 are as follows:
- Don’t be afraid to share negative metrics. There’s a story there that needs to be told. In my experience, clients are much more interested in the learnings and insights than they are about the actual number. Never give a negative metric without having a plan to turn that around.
- Use data to support your ideas and insights. Don’t throw up charts into a presentation without preparing the audience for it. Never assume that everyone can read a chart the same way you can — even when it’s extremely obvious to you
- Please, no more driver’s dashboard themes and pressure gauges on reports from anyone. I stick to clean Excel charts when possible utilizing easy-to-read color themes. Yes, it’s sort of dressing the charts up, but it’s primariliy for visual clarity and to bring Excel’s fashion sensibility up-to-speed. (This doesn’t apply to Excel 2007 which has a modernized palette). We also use a data visualization tool called Tableau which helps us build more interesting data visual aids and allows us to add additional data dimensions through variable line widths and color gradiations based on data set values.
Rebuttals, Agreements, Discussions in the Comments Section Please.
