The world needs a good tech disaster from time to time, to remind us what value and reliance we have on digital systems. Chatbot builders and businesses tinkering with AI and algorithms can learn plenty from the UK government’s disastrous dalliance with its recent exam-balancing plan that upset the nation.

Wherever you are in the world, it is likely that your government, utility providers, banks and services are relying increasingly on AI and algorithms to categorise you in a huge number of metrics. Judging if you are suitable for a loan, what property band your home falls into, how much your insurance premiums are, and so on.

Many of these algorithms have been built up over decades, carefully tested and resilient to outlier cases that need a special investigation. Then, there are examples where someone needs a way to cast a large-scale judgment quickly, and things go wrong.

British politicians demonstrate how not to do algorithms

Step forward the British Government. While it isn’t their fault that 2020’s exams were cancelled because of COVID, it is their fault that OfQual, the examinations supervisor decided to run an algorithm to overrule (or better standardize) the mock exam results and teacher’s suggested grades.

What happened is a classic politically-mandated digital disaster. As highlighted in tweets by UK political journalist Carole Cadwalladr, “Ofqual had no choice but to come up with a methodology because Gavin Williamson (Minister of Education) couldn’t stomach centre-assessed grades. Really not Ofqual’s fault, but they should have come up with a better appeals system. And they still need to, for the kids who need to appeal teacher grades. But given Williamson’s decision, they needed a statistical model. So they collaborated with the exam boards to come up with different models. And this was the one that came out best in tests. Done in collaboration with Cambridge Assessment.”

Note, this is the one that did best in the tests! Imagine how the worse examples fared. The problems start with the fact that no algorithm is “fair” and that rushing one into use creates all sorts of unintended consequences, in this case gathering worldwide media attention and condemnation.

The problems were:

Small, mostly private, schools got better grades because the algorithm deliberately ignored them. While more than a third of A-Level grades were lowered in larger (typical) schools to enforce the grade suppression. This affected poorly performing schools most, or those where mock exam grades had been kept low to encourage students to work and study harder.

The knock-on effect was pupils looking to go to university from state schools lost their places while, as Hugo Rifkind quoted, “Two universities told me they have their poshest cohort ever because privately educated kids got their grades. The universities are filled and have no places left.”

Ignoring the classist nature of British politicals and the “posh school tie” nature of the top colleges and universities, the errors in this algorithm have been visible for weeks but ignored because they were what the politicians wanted and expected to see.

The business approach to AI and bots

Businesses might think they are smarter and more understanding of technology, yet there are still plenty of examples of algorithm failures across recent history, from poor marketing choices to crashing self-drive cars and terrible image seclection.

So, when a business decides it needs to adopt a chatbot, AI or algorithm, there are many pitfalls to avoid.

The first is that these technologies are billed as easy and quick to adopt. Yes, your business can adopt a chatbot in a matter of hours, if all you expect it to do is provide some generic, but easily updated, information, as many businesses and health agencies used to address the early COVID panic.

Second, if you expect the bot to make decisions or judgments then it will need long-term training, with high-quality data as a basis. And beyond that, it needs to be able to deal with outlier cases ranging across a huge number of data points. How many bots or AI-powered forms still:

• Expect a startup company to have three years of financial data?
• Assume a post or zip code still denotes a good or bad area in regenerated areas?
• Provides very limited options to explain a family situation?
• Demands a home phone number that is different to the mobile, in an era when many people are giving up their landlines?

Thirdly, there is both business, personal and digital bias to consider. The highly paid quants and leaders working in a startup challenger bank might have a very different idea of what a “good customer” is to their marketing department or the people who see the glossy advert for an easy-to-use banking app.

Personal biases might envision everyone using your chatbot speaks great English, while the multicultural nature of most societies and the global nature of business requires more flexibility. On the plus side, it is easy to arm a chatbot with translation tools, but the business needs to be aware of them, and appreciate the value it can add.

Finally, the UK exam debacle example highlights the digital bias implicit in many algorithms. Ignoring small examples of data as not relevant had huge consequences with massive implications for students and the universities trying to fit them all in.

While a business might have a number of data scientists, validation experts and other roles working on a digital project. Someone needs to be looking at the wider impacts, the outlier or political implications, and that the needs of the business and user are fulfilled at the core.

Just because your early chatbot or AI experiments worked, it doesn’t mean your business can scale them up without extra checks and balances. Because a rival company has a great AI tool, it doesn’t mean there isn’t some intensive work behind the scenes to make sure it delivers every day.

Yes, chatbots and AI are plug-in and play services, but at every level each business implementation needs a lot of hard, clear thinking to ensure success.