Steve Krug’s Presentation- The Least You Can Do About Usability

April 3rd, 2009

From our quote in the title of the blog and this post, you can probably tell we are Steve Krug disciples. This is a little long (about an hour), but if you don’t have time to read Steve Krug’s book, this is great intro on how to get started with usability testing.

A few things we found interesting about this presentation:

  1. Loved the walk-through usability test at the end. Notice how Steve gives the user a “key task” – these are great tests to do with OpenHallway.
  2. Steve’s advice on keeping things simple is great.  Especially like the slide about NOT having exit questions or being focused on stats (the idea is to identify the big issues and prioritize them)
  3. “Don’t wait to test.”
  4. It’s impossible to get rid of all usability problems, even Amazon who has been doing this for a really long time has usability problems.
  5. Not sure why Steve is using IE6 in his test on the video (maybe he got stuck with the laptop that has IE6 on it?)  Steve – how do we get you to upgrade to IE7/8, or maybe even better, switch to Firefox on the Mac?  We promise your web browsing experience will be much improved.  Maybe you look for the lowest common denominator of bad web experiences by using IE6?
Bookmark and Share

To Moderate or not to Moderate

March 24th, 2009

Let me start by being open about our team… if I were to generalize us here at OpenHallway, we’re a fairly shy group (the rest of the team would probably argue that I’m the least shy).  Our team tends to prefer writing code over creating elaborate usability testing scenarios that require people to come into our office. Spending a lot of time putting a moderated plan together, and sitting next to people completing our usability tests while probing them for more data – sounds like a draining activity for the Myers-Briggs typed introvert. You can probably already tell that we lean towards the unmoderated approach for our usability testing. OpenHallway is based on the concept of hallway usability testing (grab the person out in the hall and watch them use what you’ve just developed) – it was born out of the model that usability testing should be simple and that taking the time to moderate doesn’t always allow for a fast, iterative approach. We wanted a way to IM/email a link to a colleague or friend, and have it record their screen and voice.  That would be some quick data that we could make good user decisions from.  While OpenHallway works just great if you want to bring someone into the office and sit them down and record them, it really shines in the remote part of things.  Having conducted both moderated and unmoderated approaches to usability testing, here’s what we think are the advantages and disadvantages of both methods:

Advantages of unmoderated (also known as “remote” or “asynchronous”) usability testing:

  1. Time savings. In a remote case (as with using the OpenHallway app) you don’t have to spend time scheduling people to come to you. Generally, prep time increases when you’re going to meet people in-person and guide them through a test.
  2. Cost savings. When testing remotely, you can typically avoid things like travel, etc.  Testers tend to have lower expectations for compensation when they don’t have to go somewhere. If you’re testing colleagues and friends, it’s usually free.
  3. User can experience (as much as possible) testing in their own environment from the their home or office. An “observer effect” can be minizimed. (although, because you’re going to record the test, an observer effect cannot be completely eliminated in either method)

When is unmoderated usability testing appropriate?

The unmoderated approach works well in scenarios where you have specific tasks in mind, e.g. try our registration process, or fill out this form, or try to accomplish this thing. It is also appropriate when you aren’t necessarily looking for a lot of qualitative feedback (although tools like OpenHallway allow you to get qualitative feedback  if you ask your tester to use the think aloud approach).  Steve Krug sometimes refers to “key task” testing which he defines as “when you are asking a user to do something, and waching how well they do.”

Advantages of moderated usability testing

  1. Contextual feedback. When testing with a moderator present, the moderator can observe the users natural reactions (those who favor the unmoderated approach may argue that having voice recording and screen recording gathers enough context)
  2. Ability to change your test on-the-fly. This approach allows you to adapt your test as it unfolds. You can make quick changes and ask questions you may not have thought about when a user gets to a specific situation.
  3. Quickly identify bad testing instructions. In the unmoderated approach, you typically provide written or recorded instructions. In the moderated approach, the tester has the opportunity to ask questions and clarify instructions they’ve been given.

When is moderated usability testing appropriate?

The moderated approach works well when you are looking to get more generalized feedback on your site, or if you want to do some exploratory testing with the user. It’s particularly appropriate in cases where you are unsure of the outcome and you’d like to adapt your test with the user present.

In Summary

If you’re using a moderated approach because you think you’re site is too complicated, you may want to rethink the usability test in the first place and start  fixing the glaring issues. Both moderated and unmoderated approaches can provide valuable feedback to improve your website or application. Do what works for your team, but most importantly, get users to be the center of your process. OpenHallway is a tool that is typically used for the unmoderated approach; however it does not limit you from picking up the phone or using the tool to conduct tests right in your own office.

Bookmark and Share

Usability Testing Should Be Simple – A Rehash

March 21st, 2009

I was recently doing some research on products out there in the market for usability testing.  I was amazed…mostly because of how complicated companies are making software for do-it-yourself usability testing. As I watched one demo in particular, my eyes glazed over with the complicated reporting features and survey tools dancing on the screen. While perusing these products I had one question in my mind, “In an agile development world, who has the time to figure out complicated software? with all those metrics, reports, and survey tools?”  Why can’t I just Skype a link to another developer, a friend, or a family member and record them using what I just built  (including screen and voice capture), and then have it seamlessly uploaded to me to view?

At OpenHallway, we don’t claim to be all-knowing usability experts, so maybe there is a time and place (maybe when you reach the Amazon or Youtube level?) to go for that million dollar usability study or to hire that team of 12 people into your marketing team that work on grand usability tests all day?  While watching these prodcut demos, Steve Krug’s voice kept ringing over and over in my head, “Keep testing simple – so you do enough of it.” It just seems a lot simpler to watch a video of 3-5 people using your site/application over lunch with your colleagues.  Isn’t it easier to have a process where you don’t have to bring people into your office (the unmoderated/remote/asynchronous approach) or buy any fancy equipment?  We don’t hide the fact that we’ve been big fans of what 37signals has done for the web world.  They brought an attitude of simplifying your application and “just saying no” to feature-bloat. We subscribe to the “getting real” philosophy, and feel like we took that same attitude, combined it with Steve Krug’s advice and applied it to usability testing.

Bookmark and Share

The Ceremonial First Post

March 5th, 2009

How it all Started:

So here we are, a couple of years after a sushi dinner between friends when we decided to pursue our own web app dreams. We had always been passionate about agile web development, MVC frameworks, and the Rails craze. We knew the “Getting Real” playbook by heart and we’d built many-a-client’s dream web application. We had always been big believers in usability testing, but had never quite found how we could make it work within our short timelines and agile process. Years ago one of our founding members had even talked about doing a Master’s degree project on remote usability testing. From years of experience working with clients and ad agencies, we were also very familiar with what Steve Krug describes as the “religious debate.” Those times where the discussion changes and everyone believes they are an expert in UI, and you catch yourself saying things like “I know the user, and a drop-down box just doesn’t work for this view.”  Over our sushi dinner, we had found our dream app idea – the one that would scratch our own itch, improve our development processes, and help other developers like us.

Early on in our experiences, we stumbled upon some research done by Jakob Nielson that said, “95% of usability issues could be discovered by testing 5 users.” We also discovered the concept of “hallway usability testing.”  Joel Spolsky explains hallway testing in one of his posts, “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.” As a small team that often works virtually, there aren’t many people in our hallway to grab for a usability test.  We realized that we wanted the advantages that hallway usability testing provides without the need for a physical hallway.  We didn’t want to go the traditional usability route by spending lots of money on expensive usability software or spending a lot of time  bringing testers into an office to experience our awkward home-grown usability lab with a video camera we purchased from Best Buy. We wanted that ability to test discrete parts of our application quickly with people by just Skyping or emailing a link that would allow the users screen and voice to be recorded, and have the video uploaded right to our screen.  The other big thing we knew we wanted – the ability for it to be cross-platform, that is to work in any testers browser on both the Mac and the PC.

Our Concept in a Nutshell:

OpenHallway allows you to conduct remote web usability testing. No need for installation of any software, it all takes place right in the browser. You start by creating a project (we have multiple clients, you probably do too, so you can create many projects). Projects can have many testing scenarios. (For example, you may want to test the registration process, a purchasing form, or maybe you just want to watch the user try to find the login button) Next, you create the scenario you want to test by writing a few instructions and providing the link to the site you want to test. OpenHallway then gives you a URL to send to anyone you want. (Send via IM/email or maybe even post on your Twitter or Facebook feed) When a tester gets the URL, they simply click on it, read the instructions, and begin the test right from the browser.  The tester’s screen and voice are recorded as he/she attempts to accomplish the user goal that you’ve created (It’s good to encourage your testers to “think aloud” as they test). After the test is complete, the video is uploaded to your OpenHallway account.  You are sent an email notification that the tester has completed the process, and you can then log in to your account anytime to watch the recorded screen and voice capture of your usability testers!

As big fans of Steve Krug’s book, “Don’t Make Me Think,” we like the methodology he proposes: “Ideally, I think every web development team should spend one morning a month doing usability testing. In a morning you can test three or four users, then debrief over lunch. When you leave lunch, the team will have decided what you’re going to fix, and you’ll be done with testing for the month. No reports, no endless meetings.” We recommend chapters 8-10 to learn more about web usability testing in general and what approaches you can take. There are many methodologies out there for usability testing such as the Hallway Test or the RITE method.

Launch Time

If all goes well, we will be launching OpenHallway by May 2009. If you would like to be notified of our launch, give us some feedback, or just get in touch with us via email: support at openhallway dot com.

Where We’re Going:

We don’t claim to perfect at usability ourselves – we just wanted a tool that could improve our development and help other developers/web marketers improve the web. We know our app will have usability issues, but we promise to eat our own dog food and test it and improve it!

Bookmark and Share