This is the third of a series of articles about how the design team operates at Envato. In the first part I discussed the tools we use like Sketch and InVision. In the second part I explored how we structure the UX and product design practice. In this part I will focus on research, how we get to know our users and put them at the heart of our day-to-day work.
Why do we user research and testing
To be truly user centred in our approach we must focus on meeting the needs of our users. To meet their needs we must first try to understand them. Then we must validate our solutions with real users to ensure we are actually meeting those needs. At Envato this is one of the main tasks of the design team.
We must find the most common jobs and behaviours and then try to understand them.
Envato’s user base is wide and varied. Our users come from all over the globe from many different cultures and backgrounds. As a marketplace we have 2 main types of user—those that buy and those that sell. But it’s not that simple. Those two groups cover a wide spectrum of different user types and behaviours.
It’s impossible to know everything about everyone. So we must find the most common jobs and behaviours and then try to understand them.
Two types of research (Gueriila and Planned)
To meet our needs when conducting research, we use 2 types of user research.
- We have an on-going research sprint. Which we use to investigate and learn from our users to help drive our product strategy. Then decide which initiatives we should focus on in the future.
- When the designers identify a gap in our knowledge or when they need to test out a solution they will conduct quick guerrilla research. Just enough to answer the questions we have.
1. On-going research sprint
We don’t currently have full-time user researchers at Envato. So all research is conducted and managed by all members of the design team. To enable this we have developed an on-going user research programme. This consists of running 3 month long research sprints continuously—alongside the delivery work of the team.
We have regular check-ins during the sprint, to monitor progress against the goal and course correct if necessary.
Each sprint starts with a kick-off to identify the goal of the sprint—summed up in the theme. We have regular check-ins during the sprint, to monitor progress against the goal and course correct if necessary. At the end of the sprint, we look back and iterate the process for the next sprint. This ensures our research practice improves as we get better at understanding our users. Building the research skills of the team.
Tracking progress against the sprint plan
The theme
We found that the best way to start each sprints to align all the stakeholders involved. The stakeholders include all the design team, the product managers, marketing and strategy. This is a really important step. There is nothing worse than getting to the end of the study only to find out that the stakeholders have a different outcome in mind.
Together we create a theme by agreeing on the part of our product or user base that we would like to understand better. This is largely driven by the roadmap and the initiatives which are scheduled in the near future. For instance our latest research sprint was based on the theme ‘People who buy video’.
There is nothing worse than getting to the end of the study only to find out that the stakeholders have a different outcome in mind.
We ask our stakeholders if they have any questions they would like answered or any hypotheses they need to validate. Taking all views into account, we formulate a goal which tries to answer as much of this as possible. We then replay the result to ensure everyone is aligned with the goal of the sprint.
The last thing we do is ask whether the stakeholders need a strategic or a tactical result from the research.
- Strategic – builds our knowledge and informs us of the users needs to give us direction to shape our product strategy and roadmap
- Tactical – tries to answer the question “what should we build next” or “how should we solve a problem we already know exists”
The survey
The first phase of our sprint is to identify the types of users that are most interesting. We use a survey to do this. This approach gives us the ability to validate any hypotheses we already have. As well as identify the groups of users that exist and work out the most interesting groups to study further. We are then looking to find the right people to actually interview, the ones with the best stories to tell.
The Lean Survey Canvas
We created the Lean Survey Canvas to help us create really powerful surveys. It avoids the pitfalls of typical surveys which focus on the questions to ask rather than the answers that are needed.
“You’re surveys are pretty good and not too cliché. They don’t bore me.”
We typically survey about 15,000+ users at this stage and expect to receive a 2–4% response rate. This gives a large enough amount of data to make sense. But small enough for us to identify the individuals we are interested in talking to.
We attach transactional data to the survey result to add real value to the insight gained. For example we apply the average life-time-value—how much our customers are worth—to the results of each question. This is very useful in getting the sceptics of user research interested.
We apply the average life-time-value—how much our customers are worth—to the results of each question. This is very useful in getting the sceptics of user research interested.
The interviews
For the second phase we use the Jobs-to-be-done framework for our user interviews. Check out the mattress interview if you want an example of how we do it.
Using a Jobs-to-be-done mindset, we focus on trying to uncover the job that our users hire Envato to do. We also discover the triggers that push and pull our users towards or away from our marketplace and the alternatives. This includes our competitors, or a user’s own skills and ability. Plus positive things like the quality of our items or the ease of our search function.
We based each interview on the users most recent purchase. During the interview we will try to build a timeline capturing all the events that lead up to the sale. This also includes everything else about that user at the time of the sale.
Using a Jobs-to-be-done mindset, we focus on trying to uncover the job that our users hire Envato to do.
Following each interview the team creates a lean persona. We converge on the job that the user hired Envato for, their needs, substitutions and triggers. We also call out any of the general themes that start to emerge after multiple interviews. We give them a fictional name based on their behaviour and try to find other interviews which follow a similar pattern.
From all the lean personas created we compile a list of problem statements which we use in the following phase.
Outcome Driven Innovation
Outcome Driven Innovation (ODI) is the really clever part and the final phase of our research sprint. The purpose of ODI is to gather enough quantitive data to identify the most important problems that our users have. And where the biggest opportunities are for our product. The result is the business focused outcome that is often missing from other forms of research.
The result is the business focused outcome that is often missing from other forms of research.
We run an ODI survey to around 15,000 users to reach a statistically relevant number of results of between 300 and 700 responses.
The survey proposes a number of problem statements in the following format (made easier for users):
[ Direction of change ] + [ Unit of measure ] + [ Object of control ] + [ Context or scenario ]
eg: Minimise the cost of producing high quality work for my clients
Survey candidates rate the importance of solving each problem. Plus their satisfaction with how our product addresses it. We then apply the ODI formula (below) which ranks the problem statement according to it’s value (both for the customer and for Envato).
Opportunity = Importance + max(Importance – Satisfaction, 0)
We can also test these problem statements on our competitors. This helps find the big opportunities for disruption in our market.
An example opportunity map produced from the results of ODI
The output from all this research informs our roadmap and helps decide on the next features to build. It also ensures that we build our empathy for our users and put them at the heart of everything we do.
2. Guerrilla research and hallway testing
The research process above gives the team a sound understanding of the customer and their needs. However, it is aimed at helping us with strategy, roadmaps and shaping future features. But what about the work that is in progress, how do we research and test that?
The UX designers will conduct just enough research and testing as and when needed.
Our UX designers are embedded within development teams. They each work on different areas of the product and different user types—the team is split between a buyer and seller focus.
The UX designers will conduct just enough research and testing as and when needed. For example, if they encounter anything unknown about the user, they will go and find out. Or, they’ll prototype solutions and test them with users to establish the right solution. Then craft it further with input from users.
The gamification of hallway testing
Before testing anything on our users we always test first with the wider Envato team. We’re lucky that most people at Envato use our sites regularly either as a buyer or seller. So their quick feedback is really valuable. We can test an idea—with quite a few internal users—within a few minutes using Slack and InVision.
We’re lucky to have a passionate user base. They are always happy to help us with user testing, and to have a part in shaping the future of our product. Testing or research interviews can be arranged very quickly. Plus, using Skype or Google Hangouts the technical setup required to conduct global research interviews is quite minimal.
Taking opportunities as they arise (Ethnographic studies etc)
When the opportunity arises, we conduct ethnographic studies by visiting our users in their place of work or environment.
We recently visited a Video Production company who use VideoHive. We found out how they work, how they integrate our product into their workflow and got a sense of the environment they work in.
Distributing the knowledge
Distributing the knowledge gained from research is one of the most important parts of the process. It’s also one of the most difficult and one that we are still trying to get right. It takes a significant culture shift in an organisation to put the customer at the heart of the process.
It takes a significant culture shift in an organisation to put the customer at the heart of the process.
Some of the things we are doing include:
- Splitting the design and product team into two groups (buyer and seller focused) to build extensive knowledge about that type of user
- The whole design team owns the research, we take it in turns to run the interviews, and as a group we synthesis the findings
- We have moved on from personas which have let us down in the past—we used to describe our buyers as ‘freelancers’ which led to confused outcomes from people’s different experiences with freelancers
- We openly talk about the jobs our users hire us to do, for example one of the recent discoveries is the need to keep Envato a secret as users realise we are their secret weapon
We’re starting to see the change with more and more teams and non-designers asking to be involved in the research. They are also looking for the insight from research to help them make decisions.
Closing thoughts
Research is the most important aspect of our work but perhaps on of the hardest. It makes the difference between the good ideas we have and the validated solutions we need. It provides the insight needed to shape the product around the customer.
Research makes the difference between the good ideas we have and the validated solutions we need.
We don’t have dedicated researchers. So, it’s all too easy to forget research when the team is busy delivering within the fast pace environment that we work. We must plan research carefully, make sure it’s realistic given the time constraints we have and ensure it gets done by holding ourselves accountable.
We focus on doing just enough research to provide the insight and knowledge that we need. But without letting the research and testing being seen as blockers to delivery. We also focus on a wider culture change to get everyone in the organisation to think in a user centred way.