Usability studies – Small changes, large impact

Intro

As a team focused on developer experience, our goal is to build internal tools that make life easier for Axis developers. This means creating tools that are intuitive and easy to use, reduce distractions and errors, and help our users understand complex things.

To achieve this goal, time and again, usability testing has provided us with invaluable, hands-on insights, leaving us wondering why we didn’t do it sooner. However, like many other software developer teams, the main reason that we are not doing usability testing enough isn’t that we don’t think it’s important. Rather, it’s because of reasons like:

It gets pushed aside, or simply falls through the cracks.
It feels too overwhelming to schedule because we think that it has to be a big production.

Steve Krug, a usability/UX consultant, refers to the issue of not doing usability testing because it’s seeming like a big production as the Big Honkin’ Test. This summer, I read his book Rocket Surgery Made Easy: The Do-It-Yourself Guide to Finding and Fixing Usability Problems, where he also proposes his master plan.

And my team decided to give it a go.

Steve Krug’s master plan

Krug recommends scheduling a recurring usability testing session for one morning a month. During this morning, you invite three participants and ask them to perform a few tasks in your application. Then you simply watch them do it and write down all the issues.

In his book, Krug motivates his plan with:

“Making it a routine eliminates having to decide when to test, you just test whatever you’ll have ready on testing day. (If you think about when you’re going to test, you’re not going to end up testing as often).”

This sounded great to my team, as it meant that after just one morning, we would be done with our usability testing for the entire month. No more guilt about not doing enough, and no time wasted on unnecessary bookings and administration.

That’s why, on October 1st, I bought some smoothies and crackers, and my team met with three users.

So, how did it go?

Watching users use your application always makes you realize how stuck you have been in your own developer-ways. This morning was no exception. But the good news is that it’s usually small changes that make a big difference. Sometimes it’s as simple as adding helpful text or changing a button label to guide users forward. Other times, it’s about simplifying, like removing cluttered tabs so users can easily find what they need.

According to Krug’s plan, after lunch, we had a debriefing meeting where we decided on the following:

The most serious usability problems that participants encountered while using our tool.
A list of the problems we are going to fix before next month’s round of testing.

All of this with a clear focus: What can we do now to improve our site for our users?

Hackathon

One addition that my team made to Krug’s master plan was to hold a hackathon on the days following the usability testing. During this hackathon, the Developer Experience team dedicated our full time to fixing the most serious usability problems. The idea behind this was to ensure that the issues weren’t just added to our backlog but were actually fixed.

This hackathon itself was just as big a success as the usability testing:

Since the usability problems were fresh in our minds, we felt very motivated and productive to fix them.
Focusing on solving these specific usability problems minimized context switching from other tasks.
Fixing issues during the same week as the usability study created a quick feedback loop, both for us and for the users. By taking user concerns seriously, we can hopefully make users motivated to participate in future studies, making user recruitment easier.
Having the team focusing on the same tasks at the same time created great teamwork. At one point, the entire team even rushed to a whiteboard, looking like a meme of a developer team solving problems.

Lessons learned

In the end, the usability testing and the hackathon led us to fixing 18 of the 38 identified issues and improvement areas identified. Reflecting on the first round of testing, we’ve decided to make a few changes to our own master plan:

The two-day hackathon will be extended to four days. Implementing changes takes time – thinking them through, coding, and getting feedback from the team. When regular tasks start interrupting this flow, it feels very disruptive. So, we figured, let’s just keep going while we are doing so much effective work.
We did record all the usability testing session; however, in the end, no one on the team reviewed any of the recordings, so we could have skipped it and just relied on our notes. The recordings just added extra stress and overhead.
We have decided to add a demo-session to the end of the hackathon, where the Developer Experience team can showcase and discuss our results. This is also a great chance for knowledge sharing in the team, which feels like a nice ending to the week.

The next usability testing session will take place on December 3, and we will begin planning for it soon. We really hope that our monthly usability studies will become a seamless and appreciated routine.

Want to learn more about recurrent usability studies? Read the great book by Steve Krug that inspired us.

Thank you for reading!

Would you like to work with these kinds of things too? We are constantly looking for skilled engineers to join our team and the rest of the company. Please keep an eye open on our career page or check out any open positions here!

Author: Frida Englund

I am a web developer who specializes in improving the developer experience for AXIS OS developers. When I’m not coding, I enjoy hiking and camping. I’m also a hobby hamburger reviewer, always on the lookout for the perfect burger.