Often times, product teams say that they are data driven or express an interest to become data-driven. But what does “data-driven” actually mean?
Data-driven means you rely on data (facts) than personal opinion or intuition to make informed decisions. One of my favorite quotes is the following:
“If we have data, let’s look at the data. If all we have are opinions, let’s go with mine.
Jim Barksdale, former CEO of Netscape
Data is a product manager’s friend. It allows you to trump strongly held opinions of internal stakeholders and in some cases also allay fears of your stakeholders. Data builds credibility of a Product Manager because you are stating facts and not your personal opinions.
However, a common pitfall I have seen is the way in which data is interpreted. It is not uncommon for two team members to analyze the same data and arrive at two different conclusions. The root cause for this often ends up to be the lack of a common data glossary. A data glossary nails down very specific definitions for terms you use in your analytics. For example, if you loosely use the term “visitors”, which of the following does it mean?
Number of visitors to your site for a given timeframe
or
Number of first time visitors on your site
or
Number of visitors who are not registered members on your site?
As you can see, a simple term called “visitors” could be interpreted in multiple ways causing confusion, lack of trust in all of the data and eventually loss of your credibility as a Product Manager.
So how do you go about creating a data-driven product culture in your organization? Here are the 5 steps I recommend product teams use to get to this goal.
Step 1: Define Success Metrics
For every new product or feature, the product development team led by the Product Manager should define the questions they would ask themselves to determine if the product or feature is a success. I highly recommend that as a group, you write these down so that the team can come up with questions with very detailed specificity to avoid issues described in the definition of “visitors” above.
Step 2: Define data that needs to be captured
Once you have agreed on the questions you should answer to determine success, you should now determine what data should be collected to answer these questions. I would recommend you write down very specific definitions for each of these data elements.
Step 3: Define Data Source
Once you have determined what the success metrics are and what data needs to be captured, determine where the data will be stored. Don’t assume this is obvious, because in my experience if this is not well planned upfront, some data could end up in tools such as Google Analytics and some in say internal databases. The disparate data sources will then make it difficult to get a unified view of the product or feature performance.
Step 4: Pull data
Define how, when and who will do the data pull once the product or feature is launched. Would this be done by merely querying the database directly or will you be creating visualizations in the BI tool such as Looker or Tableau that you may be using in your company? I would encourage you to keep it simple, especially if you do rapid experimentation via A/B testing or iterative development because usually a lot of the experiments fail and you do not want to build out visualizations to only discard them if the experiments did not move the needle in the direction you were hoping for.
Step 5: Generate insights
All the instrumentation and data capture does not mean anything, unless product teams religiously look at the data and generate insights from them to see if you accomplished the success you set out to achieve. So create a cadence where each of the product manager presents the insights learnt from the last set of experiments or released features. Such a cadence will help create a learning culture not just in the product management team but in the product development team as a whole.
It is imperative that the Product Leadership team is held responsible and accountable to provide the structure necessary to coach and mentor the product organization to build a data driven culture. The leadership team should be measured against this goal by the senior leadership. One way for Product Leadership to facilitate this is to reserve time in weekly or bi-weekly product team meetings to review data and insights generated from the last set of experiments or most recently released features.
Thoughts? How have you setup your product teams to be data driven?
Software Product Manager by Gopal Shenoy 