Archive for the ‘Uncategorized’ Category

What is Hallway Usability Testing?

Friday, May 29th, 2009

We’ve had a great response to OpenHallway since our launch three weeks ago!  Over the last few weeks, it’s been interesting to interact with with users and to find out that the concept of Hallway Usability Testing is not really well-known in the web development community.  So, I wanted to put together just a couple of references on Hallway testing and where we stumbled upon it.  We’re big fans of using Ruby on Rails and we stick to an agile development philosophy.  Being big believers in usability testing and fancying ourselves decent UX folks, we were surprised to see that there wasn’t much buzz at all in the Rails community for usability testing in general.  Just from a few discussions with some major players in the Rails world, we seriously wondered how many shops like ours were performing usability tests.  Before we built OpenHallway, we set out to do a bit of research on how we could fit usability testing into our own agile development process.  When we read about Hallway Testing, we loved it because it fit into our flow, and it came naturally to how we work.  Hallway testing is about testing the code you just wrote by “grabbing someone from the hall” and sitting them down to try out what you just built.

The Wikipedia entry for usability testing mentions Hallway Testing this way, “Hallway testing (or hallway usability testing) is a specific methodology of software usability testing. Rather than using an in-house, trained group of testers, just five to six random people, indicative of a cross-section of end users, are brought in to test the software (be it an application, web site, etc.); the name of the technique refers to the fact that the testers should be random people who pass by in the hallway. The theory, as adopted from Jakob Nielsen’s research, is that 95% of usability problems can be discovered using this technique.”  Since “bringing in” testers into our office was a hurdle for us, we decided to make a web application that would enable us to do usability testing remotely. We didn’t set out to be consultants or to claim we are usability experts, we just wanted a tool to give us quick usability feedback.

Joel Spolsky’s blog “Joel on Software” mentions it this way, “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.”

So there you go, a short primer on Hallway Usability Testing and how we got started.  It works for us, we hope it works for you too.

Last Night’s Update

Thursday, May 28th, 2009

Last night we posted an update to OpenHallway. We made some UI changes based on some usability testing feedback we received (more to come). We also posted a quick tour video on the home page.  We’ve received some great feedback from our users and we look forward to getting it implemented.  Thanks to our new customer base for all the input.

It’s Live!

Monday, May 4th, 2009

Reid Hoffman, the founder of LinkedIn, once said, “If you’re not embarassed by the first version of your product, you’ve launched too late.”  (A good blog post on the quote)  I think we’d be lying if we didn’t say we had some small anxieties last night at 3am as we made the deployment and switched to production.  The good news is that we felt really confident about the app itself and the recording technology.  We’re confident that our customers can use the app for what it’s intended – recording and watching remote usability tests!

Launch Preview

Saturday, May 2nd, 2009

We’re getting close.  We’re planning to launch within the next few days, and we wanted to post up a couple of screenshots.  We’re looking forward to doing more usability testing on our own usability testing application! Thanks to my brother-in-laws and friends on Skype for putting up with me in the last few days. Moral of the story, good idea to pay those who test for you, or at least take them out to dinner.

- The test creator can create a new scenario for a project.  After this step, they get a link to send to people.

New Scenario

New Scenario - Test Creator

- When the tester receives the link from the test creator, here’s what they see…three simple steps to get them started.  Here’s step 1.  (We’re going to make Email optional)

Step 1

Step 1 - Test Taker

- Some important info for testers to read in Step 2.

Step 2

Step 2 - Test Taker

- In Step 3, the tester reads your instructions and can launch the test.

Step 3

Step 3 - Test Taker

There you go…the sneak peak.  Time to court the mavens.  We’ll have some changes to this I’m sure, but hopefully this helps you see where we’re going with it.

Feed2Tweet WP Plugin

Thursday, April 16th, 2009

Just installed the Feed2Tweet WP plugin.  Hooked up the bit.ly shortening with it.  Posts right from blog to Twitter.  Nice.

Steve Krug’s Presentation- The Least You Can Do About Usability

Friday, April 3rd, 2009

From our quote in the title of the blog and this post, you can probably tell we are Steve Krug disciples. This is a little long (about an hour), but if you don’t have time to read Steve Krug’s book, this is great intro on how to get started with usability testing.

A few things we found interesting about this presentation:

  1. Loved the walk-through usability test at the end. Notice how Steve gives the user a “key task” – these are great tests to do with OpenHallway.
  2. Steve’s advice on keeping things simple is great.  Especially like the slide about NOT having exit questions or being focused on stats (the idea is to identify the big issues and prioritize them)
  3. “Don’t wait to test.”
  4. It’s impossible to get rid of all usability problems, even Amazon who has been doing this for a really long time has usability problems.
  5. Not sure why Steve is using IE6 in his test on the video (maybe he got stuck with the laptop that has IE6 on it?)  Steve – how do we get you to upgrade to IE7/8, or maybe even better, switch to Firefox on the Mac?  We promise your web browsing experience will be much improved.  Maybe you look for the lowest common denominator of bad web experiences by using IE6?

To Moderate or not to Moderate

Tuesday, March 24th, 2009

Let me start by being open about our team… if I were to generalize us here at OpenHallway, we’re a fairly shy group (the rest of the team would probably argue that I’m the least shy).  Our team tends to prefer writing code over creating elaborate usability testing scenarios that require people to come into our office. Spending a lot of time putting a moderated plan together, and sitting next to people completing our usability tests while probing them for more data – sounds like a draining activity for the Myers-Briggs typed introvert. You can probably already tell that we lean towards the unmoderated approach for our usability testing. OpenHallway is based on the concept of hallway usability testing (grab the person out in the hall and watch them use what you’ve just developed) – it was born out of the model that usability testing should be simple and that taking the time to moderate doesn’t always allow for a fast, iterative approach. We wanted a way to IM/email a link to a colleague or friend, and have it record their screen and voice.  That would be some quick data that we could make good user decisions from.  While OpenHallway works just great if you want to bring someone into the office and sit them down and record them, it really shines in the remote part of things.  Having conducted both moderated and unmoderated approaches to usability testing, here’s what we think are the advantages and disadvantages of both methods:

Advantages of unmoderated (also known as “remote” or “asynchronous”) usability testing:

  1. Time savings. In a remote case (as with using the OpenHallway app) you don’t have to spend time scheduling people to come to you. Generally, prep time increases when you’re going to meet people in-person and guide them through a test.
  2. Cost savings. When testing remotely, you can typically avoid things like travel, etc.  Testers tend to have lower expectations for compensation when they don’t have to go somewhere. If you’re testing colleagues and friends, it’s usually free.
  3. User can experience (as much as possible) testing in their own environment from the their home or office. An “observer effect” can be minizimed. (although, because you’re going to record the test, an observer effect cannot be completely eliminated in either method)

When is unmoderated usability testing appropriate?

The unmoderated approach works well in scenarios where you have specific tasks in mind, e.g. try our registration process, or fill out this form, or try to accomplish this thing. It is also appropriate when you aren’t necessarily looking for a lot of qualitative feedback (although tools like OpenHallway allow you to get qualitative feedback  if you ask your tester to use the think aloud approach).  Steve Krug sometimes refers to “key task” testing which he defines as “when you are asking a user to do something, and waching how well they do.”

Advantages of moderated usability testing

  1. Contextual feedback. When testing with a moderator present, the moderator can observe the users natural reactions (those who favor the unmoderated approach may argue that having voice recording and screen recording gathers enough context)
  2. Ability to change your test on-the-fly. This approach allows you to adapt your test as it unfolds. You can make quick changes and ask questions you may not have thought about when a user gets to a specific situation.
  3. Quickly identify bad testing instructions. In the unmoderated approach, you typically provide written or recorded instructions. In the moderated approach, the tester has the opportunity to ask questions and clarify instructions they’ve been given.

When is moderated usability testing appropriate?

The moderated approach works well when you are looking to get more generalized feedback on your site, or if you want to do some exploratory testing with the user. It’s particularly appropriate in cases where you are unsure of the outcome and you’d like to adapt your test with the user present.

In Summary

If you’re using a moderated approach because you think you’re site is too complicated, you may want to rethink the usability test in the first place and start  fixing the glaring issues. Both moderated and unmoderated approaches can provide valuable feedback to improve your website or application. Do what works for your team, but most importantly, get users to be the center of your process. OpenHallway is a tool that is typically used for the unmoderated approach; however it does not limit you from picking up the phone or using the tool to conduct tests right in your own office.

Usability Testing Should Be Simple – A Rehash

Saturday, March 21st, 2009

I was recently doing some research on products out there in the market for usability testing.  I was amazed…mostly because of how complicated companies are making software for do-it-yourself usability testing. As I watched one demo in particular, my eyes glazed over with the complicated reporting features and survey tools dancing on the screen. While perusing these products I had one question in my mind, “In an agile development world, who has the time to figure out complicated software? with all those metrics, reports, and survey tools?”  Why can’t I just Skype a link to another developer, a friend, or a family member and record them using what I just built  (including screen and voice capture), and then have it seamlessly uploaded to me to view?

At OpenHallway, we don’t claim to be all-knowing usability experts, so maybe there is a time and place (maybe when you reach the Amazon or Youtube level?) to go for that million dollar usability study or to hire that team of 12 people into your marketing team that work on grand usability tests all day?  While watching these prodcut demos, Steve Krug’s voice kept ringing over and over in my head, “Keep testing simple – so you do enough of it.” It just seems a lot simpler to watch a video of 3-5 people using your site/application over lunch with your colleagues.  Isn’t it easier to have a process where you don’t have to bring people into your office (the unmoderated/remote/asynchronous approach) or buy any fancy equipment?  We don’t hide the fact that we’ve been big fans of what 37signals has done for the web world.  They brought an attitude of simplifying your application and “just saying no” to feature-bloat. We subscribe to the “getting real” philosophy, and feel like we took that same attitude, combined it with Steve Krug’s advice and applied it to usability testing.

The Ceremonial First Post

Thursday, March 5th, 2009

How it all Started:

So here we are, a couple of years after a sushi dinner between friends when we decided to pursue our own web app dreams. We had always been passionate about agile web development, MVC frameworks, and the Rails craze. We knew the “Getting Real” playbook by heart and we’d built many-a-client’s dream web application. We had always been big believers in usability testing, but had never quite found how we could make it work within our short timelines and agile process. Years ago one of our founding members had even talked about doing a Master’s degree project on remote usability testing. From years of experience working with clients and ad agencies, we were also very familiar with what Steve Krug describes as the “religious debate.” Those times where the discussion changes and everyone believes they are an expert in UI, and you catch yourself saying things like “I know the user, and a drop-down box just doesn’t work for this view.”  Over our sushi dinner, we had found our dream app idea – the one that would scratch our own itch, improve our development processes, and help other developers like us.

Early on in our experiences, we stumbled upon some research done by Jakob Nielson that said, “95% of usability issues could be discovered by testing 5 users.” We also discovered the concept of “hallway usability testing.”  Joel Spolsky explains hallway testing in one of his posts, “A hallway usability test is where you grab the next person that passes by in the hallway and force them to try to use the code you just wrote. If you do this to five people, you will learn 95% of what there is to learn about usability problems in your code.” As a small team that often works virtually, there aren’t many people in our hallway to grab for a usability test.  We realized that we wanted the advantages that hallway usability testing provides without the need for a physical hallway.  We didn’t want to go the traditional usability route by spending lots of money on expensive usability software or spending a lot of time  bringing testers into an office to experience our awkward home-grown usability lab with a video camera we purchased from Best Buy. We wanted that ability to test discrete parts of our application quickly with people by just Skyping or emailing a link that would allow the users screen and voice to be recorded, and have the video uploaded right to our screen.  The other big thing we knew we wanted – the ability for it to be cross-platform, that is to work in any testers browser on both the Mac and the PC.

Our Concept in a Nutshell:

OpenHallway allows you to conduct remote web usability testing. No need for installation of any software, it all takes place right in the browser. You start by creating a project (we have multiple clients, you probably do too, so you can create many projects). Projects can have many testing scenarios. (For example, you may want to test the registration process, a purchasing form, or maybe you just want to watch the user try to find the login button) Next, you create the scenario you want to test by writing a few instructions and providing the link to the site you want to test. OpenHallway then gives you a URL to send to anyone you want. (Send via IM/email or maybe even post on your Twitter or Facebook feed) When a tester gets the URL, they simply click on it, read the instructions, and begin the test right from the browser.  The tester’s screen and voice are recorded as he/she attempts to accomplish the user goal that you’ve created (It’s good to encourage your testers to “think aloud” as they test). After the test is complete, the video is uploaded to your OpenHallway account.  You are sent an email notification that the tester has completed the process, and you can then log in to your account anytime to watch the recorded screen and voice capture of your usability testers!

As big fans of Steve Krug’s book, “Don’t Make Me Think,” we like the methodology he proposes: “Ideally, I think every web development team should spend one morning a month doing usability testing. In a morning you can test three or four users, then debrief over lunch. When you leave lunch, the team will have decided what you’re going to fix, and you’ll be done with testing for the month. No reports, no endless meetings.” We recommend chapters 8-10 to learn more about web usability testing in general and what approaches you can take. There are many methodologies out there for usability testing such as the Hallway Test or the RITE method.

Launch Time

If all goes well, we will be launching OpenHallway by May 2009. If you would like to be notified of our launch, give us some feedback, or just get in touch with us via email: support at openhallway dot com.

Where We’re Going:

We don’t claim to perfect at usability ourselves – we just wanted a tool that could improve our development and help other developers/web marketers improve the web. We know our app will have usability issues, but we promise to eat our own dog food and test it and improve it!