Category: product usability

I am a strong proponent of software usability testing. I have written in the past on best practices for software usability testing. But one of the time consuming tasks of doing such tests is recruiting users for a usability test, especially when it comes to consumer applications. It appears that this should be easy because everyone is a consumer. True, a pitfall you want to avoid is self referential design – you may unknowingly select those who are likely to agree with you, they may think like you, they already know what you do and they may likely not want to tell you that your baby (your product) is ugly – so in some ways they are tainted.

This is why I have fallen in love with a site called usertesting.com. I used it recently for the first time and I have to tell you I just love it. The results I got were just awesome and it involved very minimal setup.

Here is how it works – you fill out a simple form where you explain the scenario to be presented to the user, the site the users should use and the tasks that they should do. You select the type of users you want by specifying gender, age, country (US, UK or Canada), household income and web expertise level. Usertesting.com even allows you to select your own customers instead of using users from their database. The test costs mere $39/user and I was able to get results very quickly. You even get a high quality video of each user doing the tasks – how great is that! In one case, I had a poor quality video from one of the testers, I wrote to their customer support, they agreed and gave me a free credit for another test to make up for this. So five stars there as well!

One enhancement I would like to see is, being able to first select if I want to use one of their highly rated testers or not and then specify all of the other qualifying criteria. This way, I can specify that I want to use their highly rated testers who are male, aged 55 years or older with an annual income of less than $40,000 and who have advanced web skills. Everything can be done except the last step. I would also like to select a region in the United States. For example, I may want testers from the West Coast vs. say the midwest. If I get a selection of states available, it would be awesome. I would think that these are very simple UI fixes because they already have all of the other elements in their user database.

I wholeheartedly agree with the testimonial of Evan Williams (co-founder of Twitter) – “Use it and your site will become better.”

Now, should this be the sole way you do usability testing? – No! I strongly recommend that you still do one on one usability testing you do,and then use usertesting.com to get more datapoints so that you can get more data points very quickly.

Image: Courtesy of usertesting.com

If you enjoyed this post, please consider leaving a comment, share this blog with your friends and colleagues or subscribe to the feed to receive future articles delivered to your feed reader or via email (use the text box at the top of the right column)

The KISS principle – Keep it simple stupid – something all of us as software product managers have heard one time or another. But when it comes to products, what exactly is product simplicity? Product usability and simplicity typically falls into three different categories, in my perspective. This perspective is based on my 13 year experience as a software product manager building products for the mass market – small businesses in particular. Enterprise products may be a different ball game and hence your mileage may vary.

1) Less is more – Don’t overload your product with all the features your users may ever want. Instead focus on those features that 80% of your customers would need. Yes, this will mean telling the remaining 20% one of two things – this product is not for you (sales – yes, be honest with the customer) or get used to the feature set we have because here are the benefits. In the latter case, here are couple of examples that come to mind: Apple iPod comes to mind. I would love to delete songs directly from my iPod, but I cannot. I need to go to iTunes for that. I can choose to complain that Apple was wrong, but given the huge other benefits offered by the iTunes-iPod integration, I get used to it not being there. Apple drew the line in the sand and said No.

One of the common pitfalls companies fall into is full product configurability. Full configurability is not necessary. What it does is create a slew of options, intimidates users and on top of it creates a maintenance and testing nightmare. Majority of the users tend to use products in the default configuration. How often have you gone to the preferences or options dialog to configure something, especially when you are a brand new user? Provide basic configurations and then let the customers guide you with the other configuration options they would need. Again, stick to the 80% rule.

This is hard to do because the 20% of those customers who need all of the complicated features could be your largest customers, the most vociferous ones. But as a software product manager, you should own your product direction and learn to say No to those requests which will take your product in a direction you don’t want to go. I do not mean be arrogant, but reason with the customer as to why you don’t want to do it – in the past, I have been very honest with the customers as to how the general user base would not need the functionality being requested and they have backed down. The trick here is that your product should have such a high value proposition that these things end up to be gravy in the whole scheme of things. And if the features requests (which fail the 80% test) are the meat and potatoes for these customers. then your product after all may not be the right fit for them. Be honest with such customers and recommend the products they should look at – this may even be your competitor’s products. Don’t miss out on building the relationship, you would never know when it will come handy – for all you know they may find the competitor’s product complex/expensive and may decide to buy your product instead.

2) Your product behaves the way users expect it to – In this case, it is not about less features. Often, users expect the product to have the functionality to get their job done. To facilitate this, your product could end up being feature rich. But there is a difference – there are many products that are feature rich but not very user friendly. Then there are other products that are equally feature rich, but much easier to use. Personally, I was involved in one such product called SolidWorks which made its mark because of its production readiness and because of its ability to run on Windows with the same ease of use as Office applications at 1/5th the price of PTC’s product called Pro/ENGINEER. Pro/ENGINEER was powerful, super complex to use, ran on Unix workstations and costs tens of thousands of dollars at that time.

The trick here is to minimize the deviation between how the user expects to complete a particular task and how the software allows the user to do the task. Larger the deviation, lower the usability of the product. This is because users will have to learn new ways to do tasks that they think should be very easy to do. Usability testing is key to ensure that there is minimal deviation between the user expectation and actual difficulty in completing a particular task.

3) Your product is a game changer – Your product comes up with a very innovative way to do things the users could never imagine possible with the current systems he uses. Two products that are classic examples are 1) Bose products and 2) Shazam, the nice little iPhone app.

Whether it is Bose’s Wave radio or the acoustic noise reducing headphones or the iPod sounddock they sell, their products are game changers. They have hidden all the dials/controls that the user had to muddle with to listen to music and  made it dead simple – no dials, no controls, it just works. This to me is a game changer. Shazam is equally simple. It listens to the music that is being played, analyzes it and identifies the song – number of clicks needed – ONE to start the process. It even integrates with iTunes to download the song. To top it all, it is FREE. Can it get any better than that?

You can do this, it takes effort. When it comes to software, this is easier to do (or at least I think it is because all I have done is software in my career). One technique I have used successfully is to write down each and every step that is currently involved in completing a task and see how many of these can be automated or compressed into less number of tasks. You will be surprised in what you find. But when doing this, you will come across the naysayers who ask you – “What if the user wants to do this, what if they want to do that?”. The best way to counter is by making sure you know what the 80% of your users will need and then stick to the principle of Less is more.

Thoughts? What have you as a software product manager experienced when it comes to product simplicity?

If you enjoyed this post, please consider leaving a comment or subscribing to the feed to receive future articles delivered to your feed reader.

Images: Courtesy of gizmodo.com, bose.com, shazam.com

In the last post, I talked about best practices for doing usability testing. But how does a software product manager go about finding participants for usability testing. Here are some tips:

1) Define your target user who will benefit from the product/feature you have developed. If it is an IT management software, is it the IT admin? Or is it the CIO who will be using your product? If it is an eCommerce site that sells shoes, do you want women who will buy shoes? Do you want to target frequent buyers or the infrequent buyers?

2) Find them – In the B2B software market that I have been involved in, I have typically recruited users who I know have the pain points that are being solved by this new widget. This information has come from our enhancements database, customers I have interviewed and have this problem or users who I know have this problem based on my domain knowledge. Image the excitement such users have when you call them up and say -“Hey, remember the enhancement request you had send in 6 months back? We are planning to implement a widget to solve that problem and wanted to know if you would have 45 minutes to take a look at an early version of that widget. Your feedback will help us make sure that the solution meets your needs and since it very early, we can also make changes to it based on your feedback.” And to make them special, I also add “We are doing this only with few selected customers. Would you have the time?”. 99% of the time I have had heard a Yes. You kill two birds with one stone – 1) send the message that you are responsive to your customer needs and 2) you value their feedback on the new widget before it is set in stone.

3) Paid or Free? – 90% of the time I have never paid my usability testers especially in the scenario I have described above where the new widget is a result of an enhancement request that has come from a customer. The customer is more than happy to be given a chance to be heard and also very curious to see the early version of this new widget. They realize their time investment will pay off. In other cases where I have to find new users who are not customers, I have paid anywhere from $75-100 per test. I typically try to see if the user raises the question of $, before I offer to pay (OK, call me a cheapskate, but I like to bootstrap as much as I can). But whether it is a paid user or a free, I make sure that the user leaves with some company goodies such as hats, mugs, pens, t-shirts etc. If they are a new user, I want them to remember my brand and I want them to advertise my brand. This does not apply if you are a brand new company in a stealth mode.

4) NDA needed? – Depends on how much you can trust the user. If the user is not a customer, absolutely Yes. If the user is a current customer, in majority of the cases I get them to sign an NDA especially if it is a new product or a killer idea that has competitive benefits. But, one NDA per user, please. Typically NDAs cover two-three years. But I trust the majority of the customers I pick. In case I don’t get them to sign an NDA, I make it very clear that this information is confidential. If the user has signed an NDA in the past, I make it a point to mention that what they are about to see is governed by the NDA they have previously signed. The best part is to check with your boss – does he want an NDA to be signed – this will make sure that you are following company policies and also that your manager knows about you showing future functionality to customers/prospects.

Thoughts? Do you have any tips you would like to share?

If you enjoyed this post, please consider leaving a comment or subscribing to the feed to receive future articles delivered to your feed reader.

Image: Courtesy of NUS School of Computing

I have been doing software usability testing for the past 10 years or so and here are 10 rules I go by every time I do a usability test. I have taken the liberty of calling it “best practices” only because these work for me really well. Take that as one man’s opinion if you want, since your mileage may vary depending on your product or service.

Rule 1: Check your ego at the door before you sit down with your participants. The objective is to figure out if the software behaves the way “real users” expect it to. It is not a test of the users. And believe it or not, your software will NOT always behave your participants would expect it to behave. Do not get ticked off if the user slams your product or the new feature s/he is being asked to test – no one cares how much time you and/or your development team put in to build it, all that matters is if the user can do what s/he wants to do the way s/he expects to.

Rule 2: Make the user comfortable at the start of the test.  Why? Because it is difficult not to be nervous when you have someone watching over your shoulder while you are trying to figure out this half-baked, buggy new piece of software that you may have never used before. Chit chat, get the user some coffee or water. Do not dive right in.

Here is what I tell my participants before we start the software testing. “I want to make sure you understand that we are testing the software “usability” and not you the “user”. There is nothing and I mean NOTHING that you can do wrong in this test. If you don’t like something tell us and tell us why. If you like something tell us and tell us why. It is important for us to know both so that we don’t ruin what you like while trying to eliminate/fix what you don’t like. What we are trying to determine here is the deviation between how you expect to do certain tasks using our product and the way the product actually behaves. I would really appreciate if you could talk out loud what you are thinking while you are try doing the tasks you are about to do.”

Rule 3: Plan the test. Have a written down script for the tasks the user is supposed to do. Don’t change the script from one user to the other and importantly, stick to the script.

Rule 4: Forget about recording the audio/video of the test. Don’t listen to the usability pundits who tell you to do this. For the same reason, I have never spend a dime in renting professional usability testers or the facilities with the one way mirrors etc. Here are my reasons why I do not recommend any of this:

1. It is expensive – read – $$$$$ and time consuming. I just cannot justify the extra return on investment compared to doing it myself.
2. It is time consuming to set up and one more thing that can go wrong. I have tried using tools like Morae and even they introduce unnecessary complexities and modes of failure.
3. I like to keep it very simple. And to me, nothing comes close than having a simple setup with my users. After all, it is my job as a product manager to make sure my product behaves the way the user expects it to.
4. Getting the user not to be nervous without doing all of this is difficult in the first place. Their nervousness level goes up even higher if they find out that whatever they say or do is being recorded or they are being watched by folks behind a one-way mirror.
5. No one is going to watch or listen to the recording – do you really think your product team has the time? – putting together the test is hard enough. I would rather spend time figuring out how to resolve the issues found.

So what do I use – the good old pen and paper. I take so many notes while the user is trying to do. Talking, writing and observing is hard to do all at the same time. So this helps you reduce the “talking” part. The main point of the testing is to observe.

Rule 5: Never ever take control of the mouse. Usability testing is not a demo of the brand new widget you have slogged on to create. So if the user cannot figure out, it is not good. After all, when the product ships, you are not going to be there to help the user figure it out. The objective is simple – “Can the user figure it out on his own and does it behave like s/he expects to. If not why not?” On the other hand, this does not mean that you have to leave the user to be lost, frustrated and feeling helpless if he cannot figure it out. Give it some time and if the user still cannot figure out, provide some hints to see if it helps. If it does not, note it down as a serious issue and ONLY THEN show the user how it is supposed to behave. But again, make sure you understand that you may have a serious usability issue.

Rule 6: Never do the usability test alone without announcing it to your cross-functional product team. Announce it, inform them about the test schedule and make sure you get one person from your cross-functional product team to help you during the testing. But keep it to one additional person – so that you don’t make the user nervous. My choice for that one person – the developer who worked on this new thing that you are testing. This extra person can help you take notes, can observe it from his perspective etc. Plus, it is one more person who is witness to the test such that later when you present your findings to the team, you do not have to deal with the “data you collected” being dismissed as “your opinion” or because of the incorrect testing method used. Trust me, this happens.

Rule 7: Make the test no longer than 45 minutes. If it is anything longer, you will see a significant drop in feedback from the user. It is very difficult to work while the user is nervous for more than 45 minutes.  How do you keep it this short? Keep the number of tasks the user has to do no more than 2 or 3 that can be easily completed in 20 minutes or so. This will take 45 minutes to complete during the test because of the Q&A and discussions.

Rule 8: Never make any changes to your product unless you have convergence from multiple tests. Don’t run to your development team after one test and ask them to make changes. Wait till you have more data points from more tests before you see a pattern of the issues that are tripping users up. How many tests do you need to do? I recommend at least 5 data points.

Rule 9: Analyze the data After all the tests are completed, read through your extensive notes and process it along with the colleague who observed the test with you. What did the users really like? (make sure your preserve this). What are the things you need to fix? What should be the action plan to resolve these issues. Do it right after all the tests are completed while it is all fresh in your mind.

Rule 10: Act upon the data, execute your action items. Make sure your stakeholders know about the results and what you learnt. Make sure that they know your product team participated in the effort and learnt a lot from it. It is not “your” show, but make sure they understand that it was a “we” show and the product team is on-board to make the changes. Then make the changes and repeat the tests if needed.

Note: I have had a lot of success doing usability testing using web conference tools such as GotoMeeting and have successfully applied these same rules.

Your thoughts from your experiences – are there any other best practices you have learnt along the way? Do you agree or disagree with the above? Please share via comments.

If you enjoyed this post, please consider leaving a comment or subscribing to the feedto receive future articles delivered to your feed reader.

Image: Courtesy of Cadfanatic.com