DO WE EXPECT TOO MUCH OF COMPUTERS?

By

Hugh Jackson

Contents
Introduction

       Anyone who has ever used a computer will know the feeling of intense annoyance you get when you want to do something but the machine decides to be stubborn. We immediately lay the blame squarely on the computer and start to have a good moan about why everything could be so much better.

       What if, for example, you had to depend on a computer to fly a plane? What if the systems of your plane, which have the lives of hundreds of passengers in its control, started giving strange, inconsistent readings in the middle of a night time approach? It's at this point you realise that relying on a computer system becomes a risk.

 Risk and the Human Factor

       When the average person uses a computer, they will look no further than the icons and text on the screen, not knowing what goes on in the 'real world' of transistors and resistors. We need to remember that behind every computer system are hundreds of people who design the underlying model of the system, the hardware and software, as well as the programmers and testers. Only one of these people need be slightly inadequate for the whole system to be, potentially, fatally flawed.

       Most of the risk we take is in blindly believing that the systems we use are perfect, whereas in reality everywhere we look we see examples of problems caused by the computers we use not acting the way we expect them to, or, as is often the case, the human operators misinterpreting or mishandling the data they are given. The ideal computer system would, of course, be both secure and predictable, but it also needs to eliminate the risk we get exposed to by becoming less sensitive to the mistakes of the humans involved with it, whether in its use or its manufacture.

       The problem with humans is that we do not seem to learn from our mistakes. We learn very little from our successes, but we have the opportunity to learn much from our failures. However, we trust systems too much, and catastrophes such as Chernobyl make little impact on us in the long run. With examples like Chernobyl the risk is so great that neither the people nor the systems involved with it should be trusted, or even, in Peter Neumann's opinion, "that the systems should be built at all".

 Examples of Error

       Peter Neumann, in his book 'Computer Related Risks', sets out a large number of areas where people 'fail' in responsible use of computer systems. He has collated these from real life mishaps reported to him through his 'Risks Forum', the latest edition of which I have 'html-ized' here. This can also be found at the newsgroup comp.risks.

Here are some of his 'problem' areas:
  • Blind faith - leave it to the autopilot
  • Trust in technology - with all the checks nothing can go wrong
  • Trust in technology trust in foolproofedness - the system did something it was not supposed to do
  • False sense of security - the system didn't do something it was supposed to do
  • Complacency - we know something is wrong, but it probably doesn't matter
  • Overconfidence - there are too many checks and controls, let's turn them off while we experiment
  • Confusion - the outputs don't make sense
  • Loss of human control - the computers are down - sorry we can't help

           With all of these you could argue that the computers are the problem, but as I have said before it is human error not to anticipate problems like these occurring, whether it be a basic flaw in the machine, the computer system controlling it, or in the interface that is used [otherwise known as the Human-Computer Interface (HCI)].

    HCI Design

           Many of the cases reported to Peter Neumann can be attributed to bad HCI design. Examples include lack of clarity in relaying back to the operator what has been inputted to poor organisation of data that has led to confusion in pressure situations. Terrible cases, for example where radiation treatment has gone severely wrong and patients have died due to an overdose of nearly 1000 times intended, can be blamed on a '.' being used in a readout instead of a ',' (i.e. 2.000 representing 2,000). The computer did not malfunction, nor did the operator err, the flaw was in the underlying design of the HCI. I will examine more examples of this kind further into the survey.

     Conclusion

           It can safely be said that computers leave much to be desired in terms of reliability. This is not because the systems themselves are flawed, although this is sometimes the case [the most notorious example of which is the Pentium Pro floating point error where approximately 140,739,635,839,000 floating-point numbers are affected], but instead the flaw is in the humans associated with them and their actions. It would be naive to trust a computer system's reliability, so the only thing we can do is to try to reduce the risk we put ourselves in enough to make their use acceptable in critical applications.

     Bibliography

  • Peter G. Neumann, 'Computer related risks', 1995
  • www.useit.com - Usable Information Technology (Jakob Nielson's site) - Jakob Nielson is a leading authority on HCI design
  • comp.risks - Forum on risks to the public in computers and related systems