Stanford CIS

Mistakes, and why it's ok that we make them

By Jim Youll on

People make mistakes: errors of judgement (say, choosing a vote counting technology that is unreliable or open to fraud); and errors of execution ("pushing the wrong button" causing votes to be lost). I suppose VTP is fundamentally about the design of voting processes that stand up reasonably in the face of inevitable execution errors, whatever the cause.

Voting presents a particularly unique challenge to mistake-handling - the anonymous ballot changes just about everything you'd otherwise intuit about how to tally, record, and audit votes.

Designers of complex systems - e-commerce web sites or vote counting machines or airplanes - may design with an expectation of frequent human execution errors, trying to protect us from inevitable mistakes - or may design with the sentiment that "user error" is solved by blame-placing and re-training, because the human "failed to perform correctly" in a pristine process and that the fault lies in the human rather than the system.

Is "human error" not the designer's concern? I believe it is, and that to blame the user reveals lack of effort (for whatever reason - lack of time or money; disinterest) on the part of the designer. Operators ("users"), even when consciously making a wrong choice, are only doing what humans ordinarily do.

If the process involves people - and they all do -  then human errors will occur. But anticipating, designing for, and managing human errors in complex processes is hard work. The designer of a system that tries to catch and stop errors may fear moral, and maybe legal, responsibility when the system fails to trap all human errors, so perhaps may have some reluctance to try to make failsafe systems, or to reveal that there's been an attempt to make a system failsafe.

The recent Mishuo Securities debacle on the Tokyo Stock Exchange is an example of a complex system allowing completely unreasonable data to be carried along until it causes damage - in this case, great financial damage. An inexperienced trader miskeyed an order, offering 610,000 shares of stock for sale a 1 Yen each, instead of 1 share at 610,000 Yen. This happened in part due to the data entry error, in part because Mizuho's systems didn't aggressively try to prevent the order from reaching the exchange, and in part because the TSE's own systems failed to allow timely cancellation of the trade once Mizuho noticed the error. Both Mizuho and the TSE have taken a share of the blame, people have resigned, money has been lost, and it's a real mess... all because a simple sanity check ("is this order way outside the market price?" was easily overridden with a mouse click on a popup warning...

The whole point of this post was to get you to an informative interview with psychologist Jim Reason, Emeritus Professor at the University of Manchester in the UK, on absentmindedness.

Full article

Here's an excerpt that really gets to the heart of the matter ...

Jim Reason: ... And this is when you start to think beyond slips and lapses, beyond absentmindedness, beyond mistakes, beyond the single person, it doesn’t take you very far, and if you’ve got a complex system like aviation, where you have many, many automated defences, many barriers, many safeguards, no one error, no one technical failure can usually achieve a disaster, it takes an often diabolical combination of factors coming together, often with tiny windows of opportunity for these defences to be penetrated, and for hazards to come in damaging contact with victims and environment and so on.

Norman Swan: Which is why people such as you say that it’s easy to blame the individual for the lapse, but the system has got to have gone wrong usually for a disaster to happen?

Jim Reason: Well if you blame the individual, a lot of bad things follow really. I must first of all say that blaming is something that’s a delicious emotion, we all enjoy it, and there’s an enormous number of psychological factors which lead you to blame, particularly if there’s been a bad outcome, you know, there’s a thing that children believe, the just world hypothesis, that bad things happen to bad people and so forth. So blaming itself is a very satisfying emotion, but there is no remedial benefit from it.
Published in: Blog