Seth Nickell, project lead for GNOME Usability, recently posted a realistic piece on the shortcomings of usability testing in the software industry [On Usability Testing].
It makes me feel a little treacherous to publicly slight the value of user-feedback testing; it's still almost always a battle to make sure it happens at all. But Nickell's right: while the technique is globally underapplied, it's often locally overapplied. For all it's value, usabilty testing is one technique among many, and it's not a replacement for good design.
He also mentions a guilty but pragmatic secret of interface designers who also do testing:
"Usability testing can be a very useful technique for trying to get improvements implemented in a "design hostile environment".
The "hard data" from a usability test can be just the right distraction (misdirection is too strong) for a little procedural sleight of hand; while addressing valid but unimportant usability issues drawn from the test results, it's often possible to perform a minimally-invasive design transplant.
[Two entries in one day! Cue the End Times.]
[Hmm. I'm still typing. One last note: Mike's description of the Kevin Smith talk yesterday evening is pretty apt.]
Posted by madhava at March 13, 2004 06:13 PMI finally read the article, and agree with pretty much everything there. I'd actually expand his argument beyond usability testing, and boldly assert that most formal HCI evaluation techniques are not suitable for complex software such as an operating system.
The problem (as I see it, anyway) is that formal HCI techniques usually depend on exhaustively exploring use cases, or even exploring a significant portion of use cases. While this is trivial for your average kiosk or web application, it's not so easy when considering a product that works as a platform (an OS, a RDBMS, an IDE, &c.) While formal HCI methods are possible on these types of products, they become prohibitively expensive. As Nickell points out, usability testing isn't useless in these cases, but paying more attention to designing the product right in the first place will end up yeilding better results.
Ultimately, he's repeating the mantra of the HCI practitioner: it's better to design than to test. He's just doing it very convincingly. I especially like his assertion that usability tests on software are often judging the learnability, not the usability, of an interface.
Posted by: Mike at March 17, 2004 01:05 PMIs there a way to weaken the combinatorial-explosion monster by testing the characteristics of the component subtasks from which the interface is composed? The relationship between the tasks is obviously important, but I wonder about the value of some half measures.
I hear you loud and clear about the learnability-vs-usability issue, too. For some interfaces, I suppose that learnability is perhaps more important (if there's a low level of re-use, for example, or if the problem domain makes errors by novices especially costly), but I think that the "but what about the CHILDREN^W NEW USERS?" pleading is often way out of hand.
(One such problem domain, I suppose, is try-before-you buy software, in which the impression of the novice-cum-prospect can seriously affect sales. I'm glad that my customers have bought the tool before they see it, because it takes a lot of heat off. =) )
Posted by: shaver at March 18, 2004 01:43 PMOh, and I think you mean "a little traitorous".
Posted by: shaver at March 19, 2004 07:04 AM