A one-time usability study is not enough.
Even if the recommendations of one study are accurate and we do what the study recommends, a followup study with the new version can reveal new surprises or an opposite outcome. To really do a great job, we have to resolve seemingly contradictory feedback by thinking creatively about how we present the UI, and doing it over and over, until a new usability test finally comes out 100% successful.
No matter how good our usability studies are (and I have never been disappointed by the teams doing these studies) it is impossible to predict how a particular change will affect users. After all, if it were possible to predict, we wouldn't need the studies. Making good user interfaces is extremely frustrating for this reason. It is this iterative, frustrating process that Apple follows religiously and others mostly don't.
My favourite story about this is a story from "The Trouble With
Computers" about usability studies at Apple when they were developing the original Macintosh:
Apple interface guru Bruce Tognazzini tells this story. The in-box tutorial for novices, "Apple Presents... Apple," needed to know whether the machine it was on had a color monitor. He and his colleagues rejected the original design solution, "Are you using a color TV on the Apple?" because computer store customers might not know that they were using a monitor with the color turned off. So he tried putting up a color graphic and asking, "Is the picture above in color?" Twenty-five percent of test users didn't know; they thought maybe their color was turned off.
Then he tried a graphic with color named in their color, GREEN, BLUE, ORANGE, MAGENTA, and asked, "Are the words above in color?" Users with black and white or color monitors got it right. But luckily the designers tried a green-screen monitor too. No user got it right; they all thought green was a fine color.
Next he tried the same graphic but asked, "Are the words above in more than one color?" Half the green-screen users flunked, by missing the little word "in". Finally, "Do the words above appear in several different colors?"
That was what Apple did for a single throwaway UX question in a
non-core product - before its first release. It took about 5 iterations of UX design followed up by UX research before they finally converged on the right answer.
We are google, we have excellent UX researchers, and we have the resources to push that hard for excellence. We just have to have the willpower.
Why would you follow me on twitter? Use RSS.