User testing on a shoestring budget
How to get valuable insight without breaking the bank
I recently did a talk at the Product Tank Meetup here in Barcelona where I spoke about this very topic to a room of Product Owners, Developers, Designers, and Researchers. This is a follow-up article to accompany the presentation which you can find on SlideShare.
Why do user research?
First up, let’s consider why we do research in the first place. There could be many reasons and motivations for speaking with current and potential customers but I’d like to focus on the aspect of validation — getting feedback on existing or upcoming parts of your product or service. There are of course other areas of validation, for example, market fit but that’s a whole other topic.
A good introduction to the topic of validation would be to mention Eric Ries’ book — The Lean Startup. Many points of the book refer to the concept of the MVP (Minimum Viable Product).
The lesson of the MVP is that any additional work beyond what was required to start learning is waste, no matter how important it might have seemed at the time.
Eric Ries, The Lean Startup
An issue with the MVP is the baggage that it carries. The “minimum” effort could be very subjective and fails to ask to what end. Minimum should be focused on the question “is this in a state that will allow me to gather valuable feedback from customers?” but I feel this is sometimes overlooked.
“Viable” is also a grey area. Is it viable for the market? Your customers? The business? All of the above? Without truly understanding the meaning of the M and V parts of MVP it can be difficult to know if it has been achieved or not.
Then finally, the Product part implies that you will always end up with a product. But you could be testing is an idea, a service or a simple feature.
Getting feedback on the right thing
An interesting article I read recently from a former colleague at Skyscanner was this concept of RAT (Riskiest Assumption Test) where it too challenges the notion of the MVP and instead focuses on the idea of finding the area of your assumption that is most risky and validating your hypotheses against that.
As Rik puts it, this approach “puts a focus on learning” which is the ultimate goal of user research. Whether it is a product, a feature or a service there will inevitably be unknowns lurking within. These are the areas you want to focus on. Getting feedback on these unknowns could take many forms. It could be discussions with customers, testing prototypes or competitor analyses. In any of the methods above, you should now have a clear understanding of the greatest risks. Any discussion or testing can now be scripted in such a way to focus on these aspects with the ultimate goal being to reduce the risk of time and money being invested in wrongful assumptions further down the line.
Methods of testing
Now that you have identified the area of greatest risk in your hypothesis, there are many ways to gather feedback in order to validate or invalidate your assumptions — too many to cover in this article. Instead, I will focus on the time and cost effective techniques which can be used within startups or teams which have limited resources.
On-site guerilla testing
Depending on the type of product or service you offer this can be a very effective way of gathering feedback from potential customers. In short — Guerilla Testing is a quick, low-cost way of gathering feedback “in the wild”. By this, we mean that it is not something done in a highly structured way in a usability lab with candidates who have been sourced based on some predefined market segment. Instead, this is about looking at places where your typical users are likely to be and then approaching them for feedback in that setting. (eg. If you are working on a fitness product, go to your local gym. If your product is within the travel market, try heading to some tourist hotspots or transport hubs).
This has two benefits over the more structured approach: firstly, 2 seconds ago that person was unaware they were about to test a product meaning their feedback will be much more raw and honest; secondly, you have an opportunity to observe that person in a more natural context. Of course, this can only be true if you can find your customers in public spaces.
Some things to bear in mind if doing this sort of testing:
- Prepare screener questions. The last thing you want to do is waste your time and theirs when you realise this person is not the type of audience for your product. These questions should give you confidence that speaking with this person will bring you valuable, and relevant feedback. If their answers don’t offer that confidence, politely move on.
- Make it a 5-minute test. Get to the point quickly and construct your test in a way that can garner valuable feedback in less than 5 minutes—maintaining focus on that riskiest assumption. That persons time is a valuable resource, so unless they are willing to offer more, respect any amount which they offer.
- Capture the feedback. After going to the effort of preparing your test and gathering feedback, you want to make sure you don’t miss any of those valuable insights. Tools like Lookback will allow you to record those sessions for later playback, but even things like the sound recorder on your phone (Android and iOS phones should have this by default) will allow you to capture the conversation. Just make sure that the person you are speaking with is OK with being recorded.
Coffee shop testing
This practice was something I used to do quite a bit when working at Skyscanner and the great thing is you can find coffee shops almost anywhere. When testing in a coffee shop, you can, of course, use the Guerilla method I mentioned before, or alternatively, you can arrange a more organised event.
In the past, I have approached this in two ways. The first was to call on Skyscanner’s social media team to recruit followers of the brand via a small campaign. In short, we created some posts which offered up free coffee to anyone willing give some feedback in a nearby coffee shop. In less than 24 hours, I had over 20 participants contact a temporary mailbox I had set up for the campaign. After scheduling, I had 8 users on the day who were able to come along.
With the prototype ready, I set up camp in the corner of Brew Lab waiting for the first person to arrive. A few things to note here which I learned from this and other tests which I ran:
- Find a good spot in the coffee shop which has a working power supply (especially if you are likely to be there for any length of time).
- Ensure the ambient noise is not too loud to distract the participant or to disrupt your recordings.
- Test, test, test the WiFi. Test your prototypes as the user would using the WiFi, or even better, download it locally if you can.
- Make sure the seating arrangement is such that you will both have sufficient room for devices, drinks, notepads etc and when possible, I prefer to sit alongside participants, rather than opposite (feels less like an interview and places you in a better position to observe behaviour).
The beauty of this approach, even with the mini-campaign that we ran, is that it is a highly cost effective way of getting rapid feedback on a shoestring budget. All in, the bill came to around 25€ for the day and the entire process from sourcing participants to collating feedback was done in around 5 business days. There are really no excuses when it comes to getting some form of insight and feedback.
Another approach that I took was to build a good relationship with another nearby coffee shop where I positioned myself in the corner, set up and ready to test, and had a small poster at the checkout which offered people to give feedback on an upcoming part of our product in exchange for their coffee. However, this yielded a more random audience which was arguably less invested in the brand itself and just more invested in the free coffee. That being said, there were still many interesting conversations and usability issues raised. Always be learning!
The new recruit
Another method which can be useful if you need some quick feedback is to ask any new-starter in your company to sit with you and complete some tasks using your product. Ideally, this person would be a target customer, but so long as they have a foundational understanding of the industry (which they should if they are coming to work in your company) you can at least get some general usability insights like “does the copy make sense?”, “is the order of tasks clear and aligned with their expectations?”. Often we are so close to the project that we can overlook the simplest of details, and having a set of fresh eyes, regardless of who they belong to, can often be a great help.
Culturally this is also a great way for new starts to feel invested in the company and feel that they have an opportunity to make an impact on day one and will help you to build relationships with your “internal customers”.
Targeted, one-to-one moderated sessions
Sticking with the low-budget angle, this too can be a great way to get feedback from both existing and potential customers. Looking at your contacts list or doing a few targeted searches on LinkedIn or on your own customer database will soon reveal a potential network of people that you can reach out to. Once contacted you can either arrange to visit that person on-site, if possible, or alternatively run a session via Skype or Google Hangouts. This is an approach I have used repeatedly here at TravelPerk.
An on-site session will most likely yield greater feedback as you can observe that persons daily life beyond the screen. You see their interactions with their peers, see how they use their devices, which software they use plus much more that you would never be able to observe from the other side of a video call.
That being said, a moderated test can be just as useful if you have your questions prepared and manage to focus on the riskiest assumptions I mentioned earlier. A key thing to note here is that when you ask questions, whether in person or remote, always be sure to ask open questions. Things like “can you walk me through how you would normally do X?” vs “is this how you do X?”. Answers which are likely to yield one-word answers are the ones you want to avoid. Starting questions with “why” or “how” will tend to leverage more insight in return.
You may have also heard of the “5 whys” concept which suggests repeatedly asking “why?” to get to the root understanding of a problem or process. Used correctly, this will often uncover interesting insights that even the participant will be surprised by.
Wrapping up
These are just some ways in which I have been gathering feedback over the years and are great ways of getting quick insights without needing budget sign-off. In a follow-up article, I will share some specific tips on preparing and structuring prototypes to be able to get feedback on your designs. Until then, why don’t you go out and try getting some feedback on your latest work and see what you find! And remember…
“A desk is a dangerous place from which to view the world”
Happy testing.
