- Topic
2k Popularity
7k Popularity
6k Popularity
5k Popularity
69k Popularity
69k Popularity
140k Popularity
42k Popularity
1757k Popularity
53k Popularity
- Pin
2k Popularity
7k Popularity
6k Popularity
5k Popularity
69k Popularity
69k Popularity
140k Popularity
42k Popularity
1757k Popularity
53k Popularity
Mixpanel founder: How do you measure your product?
Author: Suhail Doshi, founder of Mixpanel Compilation: Deep Tide TechFlow
Users are people who use your product, regardless of whether they have paid or done something, it is important in your product.
About Mixpanel
In my understanding, users are people who use your product, regardless of whether they have paid or done something, it is important in your product.
I started a company called Mixpanel and technically wrote the first line of code in October 2008. I was about 20 years old, and the place where I started my business was in my parents' bedroom. 10 years later, we have about 300 employees, raised a lot of money, have about 7,000 paying customers, and generate about $100 million in annual recurring revenue.
Mixpanel mainly provides the following key indicators:
User dynamic analysis (Trends): How many times the user behavior you care about has occurred, and what is the proportion of the total.
Behavioral funnel model (Funnels): How much retention rate and churn rate are there in each step of certain key behaviors.
User activity (Cohorts): The activity of website users can be used to distinguish loyal users from ordinary users.
Single user behavior analysis (People): what operations a single user has done on the website, and what is the process.
In addition, Mixpanel can also help with the following decisions:
**What is important to you? **
I think taking a top-down approach to thinking about analysis and measurement is more effective than using charts, visualizations, numbers, etc. If you want to really solve the problem, you must carry out complex analysis and calculation, and ignore those superficial phenomena for the time being. When starting a company, I pick three things to focus on:
What I want to emphasize is that basic knowledge is very important, and even thousands of companies make basic mistakes. Overcomplicating the number of things that should actually be measured and tracked is a common mistake.
All data is contestable, and with just one or two key metrics chosen, team performance can be effectively monitored and evaluated without overcomplicating it.
official
With this formula, you can evaluate any problem in the company. For example, if your landing page is very confusing, you can use a formula to assess why. Likewise, if the registration process is too difficult, the formula can help you understand why it is. And if your product isn't popular with people, just use a formula to find out why.
The simplicity of the formula is the key to being able to assess why a company is not growing from the start. But over time, you may find that deeper analysis is needed, such as when landing page views suddenly drop, and you can break it down by country to find the root of the problem.
This formula basically solves three problems: how to make your product easy for people to understand, how to attract people to come back, and how to retain and spread the product.
Even at a company as large as Airbnb, the process of optimizing each step is never complete because your competitors are constantly improving.
By measuring these five metrics and assessing where the problem lies, you'll have your leverage and be able to take action to fix any issues. It would be helpful to form a dedicated team to address these issues.
**Is my product easy to understand? **
Some landing pages are poorly designed, which can cause users to lose interest in your product. Therefore, we must pay attention to avoid letting the user return to the original interface.
Our design and tagline are constantly improving. We try to be profit-oriented and demonstrate our product features and benefits to users through customer case studies. But the real question is how to measure whether people actually understand and use our products.
Registration does not mean that users will use our products. We need to make sure that users actually use our product and get real value from it. Therefore, we need to constantly adjust and improve our product designs to create valuable products.
**Is my product easy to use? **
Many people put too much effort into trying if the first steps are difficult. So instead of including all cool features, we should be thinking about those features that really bring value to customers. If the first step is not done well, then the whole process has no chance of running smoothly.
To use a product right away, just Google it, no need to sign up or fill out Captchas, much like Airbnb. However, Airbnb may require the user to complete about 10 steps or something similar. We've been working with Airbnb for a while, helping them optimize the first-time user experience. The key to improving the user experience is making it as quick and easy as possible for users to start using the product. Therefore, Airbnb had to solve two problems:
Getting started with Airbnb is as easy as filling in basic information and hitting search. This is only the first step of the product, but it is really the starting point of the whole process. Getting it in the right direction is the really hard part, and it takes a lot of work and trying. To measure the overall initial user experience, a funnel can be used to measure the conversion rate of a user from a landing page to a registration page to a valuable action (for example, performing at least one search on Airbnb). Speed is also an important factor that can tell you about the sophistication of the user experience. If you want to optimize the user experience, be careful not to make things worse. Verification via email and text confirmations, for example, can lead to churn. Iterating on the initial user experience is also critical and will be a never-ending process.
In the end, it is always better to get users into the product as much as possible, but not all companies or products do this. For example, Pinterest requires users to sign up. Although people may not be used to it in the past, it is now commonplace, and there are good reasons to support it.
**Are people coming back to use my product? **
In startups, don’t overlook the metric of user retention. I think this is one of the reasons many companies (even with millions of users) end up going out of business.
I've seen 15 different companies that grew sickly and then went bankrupt.
I want to demonstrate this with a graph, and I call it the shark fin effect. The name comes from the fact that the default line color for the first data point on a mixpanel is blue and takes on the shape of a shark's fin in water.
If you find a way to spread viral growth, say, on Facebook, Instagram, or whatever, you'll be very happy and think you've made a lot of money, a lot of users, and a fortune.
However, if you acquire a large number of users too early without adequate consideration for retention, this growth can become very dangerous and can be harmful to the company. Basically, the app goes viral and you lose users far faster than you gain new ones.
So, we have to be wary that when an app becomes popular, it's hard to contain the inner urge to feel, "I have to take advantage of this moment, or my competitors will get there first."
At this point, we would have been in a state of optimization, but in the end we found that our efforts did not produce much value, because if you lose all users, it will be very difficult to reactivate them. If you try to reactivate a user, you'll understand that it's usually not a good idea.
You need to keep users away for a while, until they are determined to come back and try your product again. If you take the wrong approach, such as contacting and rushing back through frequent emails, then things can get worse. The shark fin effect is really, really dangerous. In fact, LivingSocial is a great example.
While many people don't know about the app these days, it was once very popular on Facebook, competing with apps like Groupon, Deal Cutting, and others. But due to low retention rates for the app, it was eventually forced to compete with Groupon.
Methods of Measuring Customer Return
One is to track brand new, freshly registered users and calculate their return rate after a week or 30 days. Also, it is important to track for a longer period of time, as only loose data is available after a week. The most critical thing is to pay attention to the rate of losing users, so that new methods can be found and improved in time. When you find that you have more and more users, you must be alert to whether there is a shark fin effect.
The second measure is daily active users. While some may disagree with this, I think monthly active users have become a metric like the new BS metric. Like the number of registered users, it's a pretty dumb metric. Seven or eight years ago, LinkedIn announced that "we have 200 million registered users," but actually who cares how many people are using LinkedIn? Now, people like Mau are getting closer to that.
Daily active users are difficult to maintain. I mean how many apps you use per day. If you just look at your phone and browse the possible apps, how many apps do you actually use every day? I think that's a pretty strict metric, and we found that with MixPanel. So even if you're a B-to-B company, you might be thinking, "Well, I don't know, my sales business is like this, can I really keep it like daily active users? Especially in Saturday and Sunday?” In fact, there is a strong correlation between daily usage and churn, where people stop paying for the product.
The third is the revenue churn rate. It is necessary to pay attention to the speed of losing users, so that new methods can be found in time to improve. If you lose all your users, it will be very difficult to reactivate them. In this regard, there are some basic approaches that can be used. The most obvious of these is tracking brand new, freshly signed up users and calculating their return rate after a week or 30 days. Tracking for longer periods of time is also important, as only loose data is available after a week.
Issues related to day-to-day usage and churn really hurt us in the early days of MixPanel. Therefore, various ways to solve this problem need to be thought of, such as using products or pricing, etc. What would your growth rate be if churn remained constant? This is very important for all B to B companies.
**What if there are less than 50 users? **
I think the best way is to communicate with your users so you can understand them better. Let me tell you a little story about our mixing panels.
When we first started, we didn't have real users yet, but we had pages and metrics, which were fascinating. I use a technique where I put about 10 to 12 prospects all in an IM, like a chat. I'm not sure what tools kids use these days, maybe something else, but I just want to hear their thoughts, ask them what they're interested in, and investigate.
We tried to redesign the funnel UI, initially we had a vertical funnel while our competitors had a horizontal funnel, which seemed like a better idea as it was more intuitive. But we don't know how to get the data to judge whether the user likes one of these two forms, so we can only make a very rough UI version by ourselves, which is a horizontal funnel, similar to grayscale testing. I asked 11 people which funnel was better and 10 chose the horizontal funnel, so we decided to go with that, and it was a good decision.
By talking to customers, you can get more in-depth and detailed information than looking at data charts. No matter how you analyze data, you will never be able to gain the insight into their thoughts and needs that you can get from communicating directly with customers.
Overcomplicating things is a common mistake people make when it comes to analysis. They will think that they need to track various metrics, tasks and workloads, and so on. However, I think it's enough to just pick a few key metrics and monitor them in an easy way. For example, pick a North Star metric and ask yourself which number the company is willing to place its bets on and spend six months monitoring it. Feel free to change the metrics if you want, but keep it simple. At the same time, don't try to monitor too many indicators, just choose three to five key indicators.
**What is a good benchmark? **
Any one of these steps of accessing, registering, and using a product has the potential to translate into good metrics. For the part of the benchmark that relies on B to B business, it's similar to the conversion rate from visits to signups. I think a conversion rate of 4% to 5% is pretty good.
For a site like Airbnb, where it's like going to a landing page and looking for the first place you want to book, I think it's really dependent on the business itself.
Mixpanel publishes a lot of benchmarks on their blog, marketing teams like to post various benchmarks, such as gaming companies, e-commerce companies, B-to-B companies, social companies, video companies and so on. Various benchmark reports can be found there, but it really depends on the specific business situation.
login page
Finding which conversion works more like a social media button to quickly verify a user rather than the time-consuming effort of a standard sign-up requires a lot of specific considerations. However, these factors are not always clear-cut.
To find best practice, we can turn to friends who work in non-competitive industries to learn about their conversion rates and what methods they are trying. While such data may not be commercially confidential, it is still somewhat odd to ask a direct competitor, so it should be kept secret.
Sites like Airbnb, they can keep it simple because a lot of people just know what to use it for then that's why they can keep it simple instead of like having something more complicated.
And Craigslist's page design is too cumbersome. If you visit the home page of Craigslist, it can be quite confusing.
Even for a company with as much brand power as Amazon, its website should make it easy for you to log in. When you visit Amazon, you immediately see millions of items.
So how do these companies strike a balance and find ways to deal with complexity?
In fact, I think there is an important guiding principle.
First, it is necessary to determine what the goal of optimization is. Taking YouTube as an example, they cannot recommend videos based on your preferences like Google. They have to provide information related to the video and then conduct experiments to try to understand what the important factors are. It's about which experience users actually prefer.
If you provide a simple interface, but users are still confused because they don't know how to use it or you didn't ask them, then you need to capture new information.
I think the key is to find what minimizes the number of additions so that other people can start using it.
I believe a company like Amazon has tested countless homepage designs before finding the perfect one.
**Are the one to two minute video tutorials effective? **
I think this matter is complicated. As an example, we generated a video tutorial when we developed the process, but many people did not watch it. Companies that focus too much on how their users use their product tend to go too far and end up making mistakes, and they just optimize weird stuff. Even if you make a button more and more yellow and bigger, it will not satisfy the user in the end.
I find a lot of people just click randomly like all the side menu options. My assumption is that they haven't integrated Mixpanel yet, so there's nothing to look at. So what are we going to do? Keeping this data is like clicking a funnel, but why would you want to? My assumption is that they're just curious and want to know what the product does and what they've done themselves. So we did an AB test. In testing, we had an option that didn't have similar controls, just had people click around a lot and see some images.
Then we have another option with something like a video. We try to see what makes people like to use the integration, what is the conversion rate of their first integration. The result, like the video, was very successful.
In fact, we did a simpler test than this. We just sent people a video and said, "Watch this video, it's super basic," and I didn't even bother implementing it on the site, but we got amazing conversion rates. Next, we decided to add videos to all tabs, but they didn't make a difference anyway.
So I want to emphasize that it really depends on what. It is important that you run the experiment to develop a hypothesis. Art is not science, measurement is science. By measuring, you can find out if the result you want is correct.
User retention vs. dollar churn
OK, so there are two different situations here. One is similar to user retention and the other is similar to dollar churn. With the dollar plummeting, I think the reason is that companies continue to pay for products even when no users are using them. This means that companies have an incentive to cut back, and employees have an incentive to cut back because they lose money in the budget. As for dollar churn, there are many cases, for example, in B2B companies, even if only one user uses the product, the company can continue to spend money on the product, so you can see a 40% retention rate. While the company can keep spending, losing 40% of dollars in a year is really hard and not good for company retention. It is important to note that these situations are not directly linked.
For a product as heavily used as a chat app, retention rates typically range from 60% to 70%, while an app like Slack might have an 80% retention rate, which is the highest.
For a product like MixPanel, in the first year of launch, the retention rate is usually only 30% to 40%, which is not ideal.
Hardware startups need to follow four metrics like the one you are building. Most hardware companies have software components because they are often connected to the internet and need to track data. As a hardware company, tracking the data can be more challenging, but it's totally doable.
When it comes to goal setting, we need to set things up like companies of all sizes, adjusting for market characteristics and other information. In the early stages, we may not have a clear idea of who our customers are, but after a few years, we can gain more information by talking to users. For us, the initial customer base was called small businesses.
**How to ensure that the product can successfully enter the market and be recognized? **
It took us less than six months to realize that we couldn't sign a big company because our product didn't exist in the market and we didn't have enough time to develop it. Only those companies that are willing to take the risk of trying our product will consider partnering with us, and most of these small businesses are also vulnerable in other ways, so it is easier to try new products.
In this case, if the target company has nothing to lose and is willing to take the risk, I usually use this technique, which is to charge them a fee, and if they say no, I offer the service for free, because feedback is important to us. We wondered if anyone would pay, but it turned out only small businesses were willing to try.
We don't build products just to attract new users, we also focus on specific features with the primary goal of retaining existing users. We think some features actually make a difference and offer clear advantages over other possible options. For example, forgotten passwords can be a problem because it would be annoying for users to keep asking to reset their passwords, and without building this functionality, users can't log in, so we have to build it.
Even if MixPanel didn't initially build a forgotten password feature, after 12 to 18 months, we found that users would fail multiple login attempts and be presented with the login screen 5 times. This reminds us of banking services, as people tend to get used to multiple attempts to successfully log in. In fact, even if we didn't build a forgotten password feature, we ended up retaining users through other means, such as the trick Facebook uses to help users log in by simply sending them an email after three failed logins .
Ultimately, retaining existing users is our main goal, not just attracting new ones.
**How to measure product market fit? **
This is an important question and I think there can be both quantitative and qualitative measures. A qualitative measure is whether users are willing to recommend your product to their friends, which is usually a good signal.
Another quantitative approach is to measure user retention, such as the percentage of total users who return to use the product after a week or 30 days, similar to DAU (Daily Active Users). In addition, you can also measure product fit based on how users use it, such as setting strict indicators, such as users must watch 5 videos per day to be counted as active users.
Finding the right metric is key, and it can be benchmarked against the industry average, but ideally exceeds it. If your product's performance exceeds these standards, then it can prove that your product market fit is relatively good.
Of course, in order to measure more accurately, we can also measure by means of overall and frequent usage rate. The overall measurement can look at the situation of repeated use, and frequent use can filter out high-quality users more strictly.
In summary, it is important to assess product market fit through rigorous metrics. Now, the retention rate of MixPanel is only 30-40%, which shows that we have not fully adapted to the product market and need further optimization and improvement.
**How to avoid over-complicating product development? **
After I've been working on MixPanels for about a decade, I've realized that we often overcomplicate things. We tend to add more features with each release, then feel like we've solved the problem and ignore other important metrics like retention and profitability.
The same thing happens with numbers, we start with three indicators and gradually increase to 20 or more, hoping that this will give us more control, but in reality it just distracts us and leads to mistakes. I think this is one of the common mistakes founders make, they tend to be easily distracted and have trouble focusing on seemingly boring but vital things.
We must learn to focus, constantly challenge ourselves, and not give up easily. Sometimes it takes us longer to perfect a product than it is to keep launching new ones. We need to find a balance, avoid over-optimization and adding too many functions, but focus on solving the real needs of users.
In my career, I have also made some mistakes, such as developing unnecessary new products, over measuring and optimizing things that do not need to be optimized, etc. But I believe we can learn from it and become more focused and more focused on achieving our goals.