How We Use Data At AnyPerk

I don’t like the term “big” data. “Big” data is data collected in a pile and unused, shoved off in a corner collecting dust.

A wise man once said, “All data has an opinion. It’s almost worthless to analyze data which has an opinion that isn’t yours.” And big data is just that, a black morass of strange opinions.

I live in the Bay Area. I like my data small batch and handcrafted. Artisanal data, if you will (I’m sorry). It’s a discipline I learned at Yammer and one we’ve instituted at AnyPerk.

We use data for very specific business outcomes. I bucket the core capabilities of actionable data this way:

• Ad Hoc Queries —ask a question, get an answer, share the results
• Testing — validate hypotheses as rigorously as possible
• Funnel Visualization — success metrics for usage flows
• KPI/Dashboarding — visualize the success of the app over time

Within a couple months at AnyPerk, I had data I was comfortable with and set about to evaluating the tools to help us achieve these capabilities. Still early, but here are our results to date.

I should note, I’ve been able to devote about 15% of my time to date. We are hiring our very first analyst to take us to the next level.

Ad Hoc Queries

Any SQL client will let you noodle around to answer data questions, so the most important characteristic of a query tool is the ability to easily share the results. Yammer had internal tools purpose-built for sharing answers to ad hoc queries and I couldn’t imagine running a Product or company without something similar.

Evaluated: building internally, Mode

Selected: Mode

You might not have heard of Mode yet, but if you use data, you will. Mode has two sides:

1. A sort of Github for data folks, to build/share visualizations and hack on public data sets
2. An app for companies to build and share queries and visualizations of internal data

Sample churn metrics, based off of Andrew Chen’s churn spreadsheet

I can’t say enough about the transformative power of open data within an organization. The ability to ask any crazy question about your users, your product, your sales pipeline and have the answer within seconds is a super power so many companies still lack.

Their blog is pretty good too.

Testing

Call it A/B, multivariate, or functional testing. The correlations found from those ad hoc queries will form hypotheses that need to be tested in the wild for validation. This certainly goes for Product, but can be applied to Engineering, Marketing, or Sales.

Evaluating: Optimizely, split, building internally

I like Optimizely, it lets me make lots of little tweaks that might have a big impact without using dev time. But I have two biases that prevent me from wanting to lean on it too hard:

1. I believe client-side metrics are fundamentally error prone. Go ahead and argue with me, this is my opinion. The closer you get to the data being manipulated in the database, the more likely it is your numbers are accurate.
2. The most impactful changes tend to be large. Large UX changes with Optimizely require a lot of custom code. Enough of that work over time to make a number of changes would approach, and eventually exceed, the effort to build something specific to do this on our own.

There’s nothing fancy about split, it’s a ruby gem that gives you treatment groups and lets you keep counts. It’s been good enough for some of the larger functional projects to date.

I’m pretty well convinced we will need to build a custom testing solution. Likely just a denormalized table of all Rails API hits stored in Redshift with associated treatment groups.

Funnel Visualization

Your product, whether social or transactional is comprised of funnels. How many people, when confronted with a choice, do the thing you want them to? How many people fall out? It’s important not to obsess over funnel optimization, but a little hill climbing is healthy.

Evaluating: Mixpanel and Heap

Of the two, I did like how easy it was to get Heap up and running. With Segment, I was able to dump about a month’s worth of data into Heap and have funnels built within minutes.

They weren’t perfect, I quickly realized I needed a full-time analyst, and you can almost evaluate funnels by their outcomes, so I’m still dabbling.

Dashboarding

In order to have the whole company internalize the data that drives our business forward, I want it everywhere. I want our KPI on screens around the office and every member of every team to be able to tell me what they all mean.

You have to be careful with dashboards, though, a lot of them will try to sell you on visualizations that make you feel good about your data. So called “vanity metrics.” Never be satisfied that your numbers are good, that’s the road to failure.

There are only two honest KPI visualizations, a line graph and a bar chart. Anything else is a sales pitch.

Evaluating: none yet

Mode is almost this for me today. I have no foreknowledge of their roadmap, but what I have today has been enough to keep me from evaluating Geckoboard or more specific solutions like Baremetrics.

Those TVs are coming as soon as possible, though, so let me know if you have any suggestions in the comments.

Reporting

A cousin of KPIs/Dashboards, the ability to periodically report either to customers or individual team performance. Deal size from Salesforce isn’t accurate for our vendor team. Redemption counts aren’t detailed enough for our CSMs to provide customers.

Here, the ease of access and polish of the presentation is more important than the depth of the information. Still considering here as well.

Other

I’d be remiss in not mentioning Segment.

Even services that don’t fit the buckets above are data hungry these days. AnyPerk’s CSMs want to try out churn tracking tools, our Marketing team want better data into Marketo, our Sales team want customer contacts in Salesforce. Some of those services are worthless without at least a few days of data, so it’s hard to properly evaluate alternatives or switch.

We could spend weeks or months hacking APIs at the Rails level or adding an ungodly number of JS libraries onto pages themselves. Then, like the Google Analytics Universal switch, the libraries need updating.

Enter Segment. We integrated once and can now flip services on and off at will. It’s huge for evaluating new systems before you buy them and, while they are provider agnostic, their team has great knowledge of the solutions they integrate with.

We still have a long way to go. Even with all of these systems in place, the work will continue to use all these tools effectively.

As a Product, AnyPerk needs to do a great job targeting our perk offerings to individual users. We need to experiment with different ways of communicating with our users based on that classification.

We‘re building a team to take these efforts to the next level. If this sounds interesting to you, come join us!