When things go wrong, a business cannot blame artificial intelligence (AI) as a shield to get out of legal consequence. This issue, which many businesses continue to grapple with, is explained in a new book by researchers from the University of Johannesburg (UJ). Professors Husain Coovadia and Michael Adelowotan are the co-editors and co-writers, along with other UJ-affiliated authors, of the book, which is titled ‘AI and Accountability: Shaping Ethical Governance in the Age of Algorithms’, published by Springer.
When a business turns decision-making over to an autonomous machine, the organisation is still held accountable if things go wrong, says Prof Coovadia. He is a researcher in the School of Accounting, Department of Commercial Accounting, and Vice-Dean in the UJ College of Business and Economics.
“It is not possible for a computer to carry out legal or corporate responsibility in the way and manner that a company director or CEO would do. AI can only make or support a decision, whereas the organisation is the one that decides when they want to use a computer or an automated system to carry out a job,” he adds.
A good example is a hospital where an automated AI-based dispensing system gives a patient the wrong medicine or the wrong dosage.
“Who will be held responsible? Is it possible for the hospital to tell the patient or their family, ‘We are sorry, the computer made a mistake’?” he asks. “Definitely not.”
“When anything goes wrong, the real questions are: Who programmed the system, who loaded the medicines and who was responsible for stopping it when something appeared wrong. In all this, such persons will be held liable. In South Africa, healthcare policy places these responsibilities on healthcare providers,” he says.
In corporate governance, the AI buck should ultimately stop with the organisation’s leadership and management, he adds.
“Think of AI as a strong organisational resource, just like financial capital. The board would never say, ‘The accounting system made a mistake, so nobody is responsible.’ The same applies to AI,” says Prof Coovadia.
When the AI is biased
A related risk is automation bias, says Prof Michael Adelowotan, an Associate Professor in the UJ School of Accounting. “Humans can place too much reliance on the recommendations or decisions generated by automated systems, artificial intelligence or algorithms. In particular, people may not consider contradictory evidence. Automation bias can arise in a business, financial institution or healthcare facility.
Banks, for example, use AI and machine learning models in credit approval systems to assess loan applications, says Prof Adelowotan.
“An AI model may reject the application of a small business owner with strong cash flow and limited formal credit history. This is because it cannot fully account for these contextual factors. If the bank’s officer simply accepts the recommendation, a potentially profitable and deserving customer is denied credit,” he explains.
If a bank’s AI credit model is ungoverned, many creditworthy applicants may be rejected. This may bring shorter processing time and lower operational cost in the short run. But it may lead to a loss of customers and a reduction in market share.
“Responsible AI should not be seen as a compliance expense but as a trust-protection, risk-management strategy and value-preservation corporate strategy,” Prof Adelowotan adds.
Explaining an AI system’s decisions
Where an automated bank system says: “Loan rejected,” Explainable AI, or XAI, can give the customer detailed reasons why the application was rejected, says Prof Coovadia.
“Without proper explanation, organisations can end up with what we call ‘trust without understanding.’ XAI does not mean every algorithmic decision must be understandable to everyone. Rather, it means every person responsible for important decisions must understand, interrogate and challenge the system’s output,” he explains.
Bringing AI-based services to market
When smaller or mid-sized companies want to bring new AI-based services or products to market, the risk can be immense. But there is a way to develop these in a much safer way, says Prof Coovadia.
“Regulatory sandboxes are very valuable for companies that want to innovate yet do not have the resources that multinationals have. Rather than releasing a new AI system directly into the market and discovering its problems afterwards, a company can test an innovative solution in a controlled environment, with regulatory engagement and safeguards.
“The benefit is that regulators and innovators can identify problems earlier and clarify regulatory expectations before a technology is deployed at scale. For smaller businesses, this form of collaboration is important because responsible innovation should not be something only the large-sized firms can afford,” he concludes.
