A couple of weeks back Jeffrey Zeldman posted about a gas leak in New York. This sentence jumped out at me:
We knew that the smell was not natural gas but mercaptan, a chemical that is injected into natural gas to let people know when there’s a leak.
I’d never heard of mercaptan (or Methanethiol) before, but of course it makes sense given that natural gas has no smell of its own.
What do we “inject” into software to let us know when we have a problem?
Logging errors is pretty common. At Trade Me we record the details of every unexpected exception in the application and database, including the page details and query string, stack trace, error details, member details, user agent, etc. We have SMS alerts setup so that when clusters of errors occur we are automatically notified. We monitor the error logs pretty closely whenever we deploy changes to the site to make sure that we haven’t introduced any new errors. This works pretty well.
But, a system like this only tells us when hard errors occur.
What about less obvious things, like design related problems which don’t result in hard errors?
How about validation errors? Most developers would argue that validation errors are the users problem – where required fields are not completed, or invalid data is entered etc. However, if the same validation error is occurring repeatedly then it is a pretty good indication of problems with the design of the form. And, getting visibility to that would provide useful feedback to the people responsible for designing the form.
How about the amount of time spent viewing each page? Suppose we could measure the time between page requests as a user works through a given process (e.g. the sell process on Trade Me). If users are spending a long amount of time on one particular page that could be an indication that the design is not clear. Likewise if users consistently abandon the process at the same point.
How about commonly used help links or help search terms? On Trade Me the #1 viewed help topic (by quite a margin) is the page which explains auto-bids. What are users stuck on when they click the help link? Can we use this information to better explain the parts of the application that users are actually struggling with.
How about client-side HTML validation errors and JavaScript errors?
I’m sure there are lots of others.
Has anybody tried to implement this sort of soft error logging? I’d be interested to hear if it worked.
I once had the misfortune of looking after a web application a few years ago that did a ridiculous amount of logging for client-side validation errors, page views and a whole host of other things. Other than poor database design (the logging table corrupted no less than twice), it seemed to work well for generating the reports the application’s owners needed and was a big help in helping users with cookie problems.
I had planned (when I get the time) to look into using Ajax as a way of logging and capturing page viewed/interaction times and events.
I’ve often wondered if the numbers can tell us more about what is really happening with users. That would be the holy grail to fully understand from a quant perspective what people are struggling with. And suppose a site is then smart enough to automatically adjust to the problem (which has been compared against a large library of known issues and recommended responses)?
While I don’t think we could ever get numbers to tell the full story about what’s happening for users, there seems to be very little being done to even make an attempt at this. Does Trademe do anything of the sort?
It’s definitely hard to tell the full ’soft error’ story with purely quant data, but using heatmaps and quant analysis of clicktrails can help tell some of the story. In the meantime, nothing does a better job of identifying ’soft errors’ than regular user testing in a lab environment. If you think you’ve spotted a possible soft error with one user, sit another 10 down and ask them to follow the same path, and see if they hit the same problem.
By the way, there’s no need to go whacking a gas main with a pickaxe to try the smell of mercaptan for yourself. Just serve a meal to a group of friends/family and make sure you include some asparagus in the ingredients.
One of the byproducts of digesting asparagus is mercaptan, which is rapidly expressed in urine in levels high enough to be clearly noticeable. So why test a group of people and not just yourself? Well, the expression of mercaptan is a common (but not universal) genetic trait which you may not have, and the ability to smell mercaptan may also be genetically linked – see this piece on http://www.webmd.com
…can’t believe the Wikipedia entry didn’t mention this! At last, a chance to contribute something substantial to Wikipedia! W00t!