Introduction
On September 26, 1983, an automated system in the Soviet Union sounded the alarm on five incoming nuclear missiles, almost leading to global destruction. Stanislav Petrov, a lieutenant colonel in the Soviet Air Defense Forces, was alerted intercontinental ballistic missiles. He had minutes to make the fateful decision about whether to inform his superiors about the surprise attack or not. Although one could argue this wasn’t officially an AI incident, it is the first incident that appears in Surfshark’s report on AI incidents.
In its Artificial Intelligence Risk Management Framework
For AI systems, “the contexts in which they are deployed are frequently complex, making it difficult to detect and respond to failures when they occur,” contends NIST . Human behavior and societal dynamics can influence AI systems, something that isn’t always easy to predict or even understand. This has both risks and benefits, which make AI a uniquely challenging technology to deploy and utilize both for organizations and within general society, says NIST. “Without proper controls, AI systems can amplify, perpetuate, or exacerbate inequitable or undesirable outcomes for individuals and communities. With proper controls, AI systems can mitigate and manage inequitable outcomes,” NIST warns.
What is an AI Incident?
According to the OECD’s Defining AI Incident and Related Terms
“An AI incident is an event, circumstance or series of events where the development, use or malfunction of one or more AI systems directly or indirectly leads to any of the following harms:
Whereas an AI Incident might cause injury or harm, the OECD claims a serious AI incident is an event, circumstance or series of events that could lead to the death or serious harm to a person or groups of people. A serious AI incident could also be “serious harm to property, communities or the environment” and/or “a serious and irreversible disruption of the management and operation of critical infrastructure,” says OECD.
Serious violations of human rights or serious breaches “of obligations under the applicable law intended to protect fundamental, labour and intellectual property rights,” are also included under serious AI incident definition, notes OECD.
Beyond a serious AI incident, OECD claims, is the AI disaster, which is “a serious AI incident that disrupts the functioning of a community or a society and that may test or exceed its capacity to cope, using its own resources. The effect of an AI disaster can be immediate and localised, or widespread and lasting for a long period of time.”
The Rising Tide of AI Incidents
In its 2023 G2 Software Buyer Behavior Report recent survey of 1,700 software buyers that found, “Even amidst wide skepticism surrounding AI use, 78% of respondents stated that they trust the accuracy and reliability of AI-powered solutions.” However, AI incidents, which are on the rise (see Figure 1), threaten to derail this good will.
Figure 1: AI Incidents 2012-2023
Today, to ensure this goodwill, it is crucial for companies to address an AI incident as quickly as possible. The deployment of AI systems require users to trust all outputs coming from that AI system. This isn’t always easy because of AI’s blackbox nature, which can be highly opaque. It is often hard to understand what an AI model is doing under the hood, especially unsupervised models. An effective AI incident response procedure can foster trust among users.
In addition, organizations trying to manage and mitigate risks associated with AI technologies show that compliance with regulatory standards and ethical practices is meaningful to them. This is no small thing as governments and regulatory bodies are increasingly focusing their attention on AI oversight. Having a robust incident response plan will ensure organizations can comply with evolving regulations. This includes establishing clear protocols for incident reporting and management, which can enhance accountability and transparency in AI usage.
AI Trust Statistics
78%
Trust AI-Powered Solutions
In its 2023 G2 Software Buyer Behavior Report, the software marketplace, G2, conducted a recent survey of 1,700 software buyers that found, “Even amidst wide skepticism surrounding AI use, 78% of respondents stated that they trust the accuracy and reliability of AI-powered solutions.”
AI Incidents
In his article, AI ‘Incidents’ Up 690%: Tesla, Facebook, OpenAI Account For 24.5% , John Koetsier claims, “the National Eating Disorders Association (NEDA) was forced to shut down Tessa, its chatbot, after Tessa gave dangerous advice to people seeking help for eating disorders.” Other incidents include a self-driving Tesla almost hitting a person in a crosswalk as well as the wrongful arrest of a Louisiana native after a facial recognition system allegedly mistook him for someone else. Koetsier states that, according to Surfshark, a multination cybersecurity company, these AI incidents are rapidly fast.
AI Incidents Report
690%
Increase in AI Incidents
Dr. Marco’s YouTube video on AI incidents and how to counteract them. Deepfakes
According to Business Insider , “The term ‘deepfake’ comes from the underlying technology — deep learning algorithms — which teach themselves to solve problems with large sets of data and can be used to create fake content of real people.” Using deep learning models like neural nets, deepfakes swap out one person for another in an image or a video. In many cases, the reproduction is so good, it easily fools the human eye.
Deepfakes are AI-generated media that alters images, videos, or audio recordings to misrepresent individuals as doing or saying things they did not actually do or say. The term “deepfake” is a portmanteau of “deep learning” and “fake,” reflecting the technology’s reliance on advanced machine learning techniques, particularly generative adversarial networks (GANs), which involve two neural networks that create realistic content.
Deepfakes technology poses significant risks because fraudsters exploit it by creating realistic audio and even real-time video to impersonate executives, employees, and customers to deceive organizations into transferring huge sums of money to the fraudster. This isn’t a simple case of people paying ransoms because of blackmail. With deepfakes, the defrauded willingly make financial transactions to people they think they personally know.
Deepfake Incidents
Deepfake video fraud — According to CNN , a Hong Kong multinational finance company paid $25 million to fraudsters who used deepfake technology to pose as the company’s chief financial officer in a video conference call.
Deepfake audio fraud — Trendmicro reports cybercriminals used a deepfake audio to defraud the CEO of a U.K.-based energy company out of US$243,000.
Public figure impersonations — Deepfakes have promoted fraudulent investment schemes by impersonating public figures. For example, deepfakes of prominent individuals have been created to advertise fake trading bots and investment platforms, misleading potential investors.
Historical figure impersonations — As per the AI Incident Database , AI-generated English-language Adolf Hitler speeches are proliferating on TikTok, racking up millions of views despite violating the platform’s hate speech policies.
WhatsApp video fraud — The AI Incident Database reports that deepfake scam artists targeted WiseTech Global’s CEO with AI videos that requested money from his staff members through WhatsApp. Luckily, the employees realized they were not speaking to the real CEO and refused the monetary request.
AI Chatbots
AI chatbots, also known as conversational chatbots or intelligent virtual assistants, are computer programs that use AI to engage in conversations with users. They can understand user intent, provide relevant responses, and learn from interactions over time, making them far superior to traditional rule-based chatbots.
Chatbots use machine learning (ML), natural language processing (NLP), large language models (LLMs), and retrieval-augmented generation (RAGs) to understand and interpret human language by analyzing the structure and meaning of the input. Through ML algorithms, chatbots learn from previous interactions, interpreting human language by analyzing the meaning of the input; LLMs allows chatbots to generate contextually relevant and coherent responses based on user queries; and RAGs, which enable LLMs to access external data sources at inference time, can enrich the prompts used for generating responses.
Hallucinations
Although AI technology can be quite impressive, it often gets things wrong, sometimes completely wrong. Often, it simply makes things up. This problem has become so common, there is even a term for it — a “hallucination.” “AI hallucination is a phenomenon wherein a large language model (LLM)—often a generative AI chatbot or computer vision tool—perceives patterns or objects that are nonexistent or imperceptible to human observers, creating outputs that are nonsensical or altogether inaccurate,” says IBM .
AI chatbot hallucinations
McDonald’s AI Drive-Thru Incident: McDonald’s ended an AI experiment after its voice-ordering system made significant errors in processing customer orders, leading to operational challenges.
Grok AI False Accusation: The Grok AI chatbot falsely accused NBA star Klay Thompson of vandalism, raising questions about liability and the accuracy of AI-generated information.
MyCity Chatbot Legal Misguidance: A Microsoft-powered chatbot named MyCity provided incorrect legal information to business owners, potentially leading them to break the law.
Tay Chatbot Offensive Tweets: Microsoft’s Tay chatbot engaged with users on Twitter and posted racist and misogynistic tweets after being exposed to harmful content.
AI-Generated Sports Articles: Reports emerged that Sports Illustrated published articles written by AI-generated writers without proper disclosure, leading to concerns about transparency and authorship.
These incidents highlight the various risks associated with the deployment of AI technologies across different sectors, emphasizing the need for robust governance and oversight mechanisms.
Algorithmic Bias
Within the context of AI incidents, an algorithmic bias refers to the systematic favoritism or discrimination arising out of the design, training, and application of AI systems. This bias can lead to unfair outcomes that reinforce existing societal inequalities, particularly those affecting marginalized groups.
The sources of bias could be the system’s training data. AI systems learn from historical data, but this can often reflect long held, built-in societal prejudices. A hiring algorithm trained on data from a company with a history of discriminatory practices will probably perpetuate those biases by favoring one chosen demographic over another.
Even the structure of an algorithm can trigger bias. Decisions made during the development phase, such as which features to include or how to interpret data, can lead to biased outcomes. Context is key as well. An algorithm designed for one demographic may not perform equitably across different populations.
Unfortunately, algorithmic bias can lead to systemic discrimination in areas such as hiring, law enforcement, and healthcare. These biases can perpetuate existing stereotypes, continuing the cycle of inequality that tends to harm society’s most vulnerable members. As AI systems increasingly influence critical decisions, biased outcomes can erode public trust in these technologies and the institutions that deploy them.
Algorithmic Bias Incidents
Amazon’s Recruitment Tool: Amazon developed an AI system to streamline its hiring process, but it favored male candidates over females because the algorithm was trained on resumes from predominantly men.
Facial Recognition Systems: Numerous studies reveal that facial recognition algorithms are less accurate when identifying individuals with darker skin tones as well as women.
Criminal Justice System Algorithms: The Correctional Offender Management Profiling for Alternative Sanctions (COMPAS) tool disproportionately labeled black defendants as at a higher risk for reoffending compared to white defendants.
Healthcare Algorithms: A healthcare algorithm used to predict which patients would require additional medical care was found to favor white patients over black patients because biased training data reflected typical historical healthcare disparities.
Job Advertising Algorithms: Research indicated that Google’s advertising system tended to display high-income job openings more frequently to male users than female users, highlighting a bias in targeting and representation in job advertisements.
AI Avatar Apps: The AI avatar app Lensa was scrutinized for producing sexualized images of women while providing diverse and professional avatars for men. This outcome reflects underlying biases in the training data used to develop the app.
Apple Card Controversy: Reports emerged that the Apple Card’s credit scoring algorithm offered significantly lower credit limits to women compared to men, even when financial backgrounds were similar, demonstrating a gender bias in the financial decision-making system.
These examples reveal how algorithmic bias can manifest itself across many different sectors, leading to unfair treatment and reinforcing existing societal inequalities. Addressing these biases is essential to develop equitable AI systems that serve all individuals fairly.
Data Breaches and Privacy Violations
Several AI incidents that would be considered data breaches and privacy violations highlight the risks associated with the technology. These incidents raise concerns about the security of the AI tools available as well as the lack of security at many of these platforms.
Data Breaches and Privacy Violations Incidents
Sensitive User Data Leaks: In early 2024, OpenAI reported that sensitive user data, including personal details, conversations, and login credentials, leaked due to a suspected hack.
TaskRabbit Data Breach: In April 2018, TaskRabbit, an online marketplace matching freelancers with local demand, suffered a data breach that affected over 3.75 million records of users.
Yum! Brands Data Breach: In January 2023, Yum! Brands, a multinational fast-food conglomeration, experienced a ransomware attack that compromised corporate and employee information, leading to significant disruptions to its operation.
T-Mobile Data Breach: In early 2024, T-Mobile revealed that hackers used an AI-equipped application programming interface (API) to gain unauthorized access to sensitive customer information, including full names, contact numbers, and PINs of its customers.
23andMe Data Breach: In December 2023, DNA testing company 23andMe suffered a breach affecting 6.9 million users.
Clearview AI Scraping Incident: Clearview AI faced scrutiny for scraping billions of images from social media without user consent for its facial recognition technology. This raised significant privacy concerns and potential violations of data protection laws like GDPR.
These examples show how AI technologies can be used for both good and bad. AI technology can exploit data breaches as well as violate the privacy of customers. They emphasize the need for robust security measures and ethical practices in AI development and deployment. IBM has now put a figure on the price of a data breach. In its Cost of a Data Breach Report 2024 , IBM claims, “In 2024, the global average cost of a data breach was $4.88 million.” These is not a small number and it is probably set to grow substantially over the next few years.
Data Breach Cost Statistics
$4.88M
Average Cost of a Data Breach
Risk Measurement
The NIST advises organizations recognize that not all AI risks are the same. They recommend that companies allocate resources purposefully. “Actionable risk management efforts lay out clear guidelines for assessing the trustworthiness of each AI system an organization develops or deploys. Policies and resources should be prioritized based on the assessed risk level and potential impact of an AI system. The extent to which an AI system may be customized or tailored to the specific context of use by the AI deployer can be a contributing factor,” states NIST.
For NIST, the following factors complicate risk measurement:
Risks related to third-party software, hardware, and data: AI system developers may not be transparent about their risk metrics or methodologies and customers might integrate an AI system without sufficient internal governance structure and technical safeguards.
A lack of consensus on robust and verifiable measurement methods for risk and trustworthiness.
Differing risks at different AI lifecycle stages because AI systems can adapt and evolve.
Risk in lab settings may differ substantially from risks that emerge in operational, real-world settings.
Inscrutable AI systems, which can result because of the opaque, blackbox nature of AI, can complicate risk measurement.
There is a lack of any baseline metrics for comparison between human and AI systems because AI systems perform tasks differently to humans.
Best Practices for Mitigating AI Incidents
To counter any AI incidents, organizations must adopt a comprehensive set of best practices that encompasses proactive measures, incident response strategies, and continuous improvement. These should include:
An AI Incident Response Plan: Create a detailed plan to outline procedures to detect, respond to, and restore systems from AI-related incidents that includes roles and responsibilities, communication protocols, and recovery strategies to ensure a swift and concrete response. Regularly review and update the incident response plan as AI systems regularly change.
Enforce Robust Data Governance: Strong data governance helps ensure accountability, protects sensitive information, promotes transparency, mitigates risks, enhances collaboration, improves operational efficiency, and prepares an organization for an AI incident.
Implement systems for real-time monitoring of AI performance: This detects anomalies or biases early, which can help identify unusual behavior in AI models or network traffic patterns.
Ensure rigorous validation and sanitization of input data: This prevents malicious inputs from compromising AI systems.
Conduct Regular Audits and Testing: Perform regular security audits to identify vulnerabilities. Regularly test AI models for weaknesses through penetration testing specifically designed for AI systems.
Implement Robust Security Measures: Use strong encryption methods and secure communication protocols to protect sensitive data from breaches.
Zero-Trust Architecture: Adopt a zero-trust security model that verifies every user and device accessing the AI systems.
Foster Human Oversight: Maintain human oversight in AI decision-making processes to catch potential biases or errors generated by AI systems.
Establish Clear Communication Channels: Proactively develop clear communication plans for notifying stakeholders about incidents, including internal teams, customers, and regulatory bodies as necessary.
Conduct thorough post-incident reviews: Analyze what occurred and what improvements can be made.
Preventing AI hallucinations
IBM believes the best way to reduce the impact of AI hallucinations is to stop them in their tracks by utilizing the following methods:
Use high-quality training data — the quality and relevance of training datasets dictate the model’s behavior and the quality of its outputs. Train AI models on diverse, balanced and well-structured data, which helps minimize output bias and yields more effective outputs.
Define the purpose the AI model will serve. Establish the chosen AI system’s responsibilities and limitations, which should help the system complete tasks more effectively while minimizing irrelevant, “hallucinatory” results.
Use data templates — Data templates provide teams with a predefined format, increasing the likelihood that an AI model will generate outputs that align with prescribed guidelines. Relying on data templates ensures output consistency and reduces the likelihood that the model will produce faulty results.
Limit responses — AI models often hallucinate because they lack guardrails that limit possible outcomes. Companies should use filtering tools and/or clear probabilistic thresholds to define the AI model boundaries.
Test and refine the system continually — Testing your AI model rigorously before use is vital. These processes improve the system’s overall performance and enable users to adjust and/or retrain the model as data ages and evolves.
Rely on human oversight — Making sure a human being is validating and reviewing AI outputs is a final backstop. A human reviewer can also offer subject matter expertise that enhances their ability to evaluate AI content for accuracy and relevance to the task.
Combatting Algorithmic Bias
To counter algorithmic biases, organizations should perform regular audits of their data and algorithms. This includes hiring external auditors to provide an impartial assessment of the algorithms and their outputs, which can help uncover hidden biases that internal teams may overlook. When viable, companies should make their algorithms publicly available so that external parties can test them with their own datasets, creating accountability which could help uncover any biases in the algorithmic behavior.
Companies should ensure any datasets used for training AI models are diverse and representative of all relevant demographics. This helps mitigate biases that could arise from underrepresented groups.
Adversarial machine learning is another methodology that can help combat algorithmic bias. In his What Is Adversarial Machine Learning? Attack Methods in 2024
Human oversight belongs in the decision-making process as well, especially in high-stakes situations where biases often have significant consequences. Human reviewers can quickly spot and correct any AI systems biased output. Feedback loops allow users and stakeholders to report instances of perceived bias or unfairness in AI systems. This feedback can be an input in AI models going foward to limit future biases in the data.
Addressing algorithmic bias is crucial for companies to develop fair and equitable AI systems that avoid perpetuating societal biases. Technology has proven to be a force for good, but vigilance is a constant challenge.
NIST’s AI Risk Management Framework
The NIST AI Risk Framework goals include the following:
Be risk-based, resource-efficient, pro-innovation, and voluntary.
Be consensus-driven and regularly update through an open, transparent process.
Use clear and plain language that is understandable to a broad audience yet includes enough technical depth to be useful to practitioners.
Provide common language to manage AI risks.
Be easily usable and fit well with other aspects of risk management.
Be useful to a wide range of perspectives, sectors, and technology domains.
Focus on the outcome while being non-prescriptive.
Exploit and foster awareness of existing standards, guidelines, best
Be law- and regulation-agnostic.
Be a living document. Update as necessary.
As AI technology evolves, new types of AI incidents will undouteedly arise. Current frameworks probably won’t be able to keep up with them. Generative AI models could introduce novel harms that require updated reporting and new management strategies.
Conclusion
Organizations must develop comprehensive governance frameworks that include incident response plans tailored to the unique challenges posed by AI technologies. This proactive approach helps ensure responsible development and deployment of AI systems.
AI incidents can encompass trust issues, regulatory challenges, compliance needs, societal impacts, and the necessity for robust governance structures to manage risks effectively. They can exacerbate inequalities and infringe upon on human rights. Incidents that disrupt critical infrastructure or violate personal privacy can lead to significant public backlash and societal division. These risks make AI deployments uniquely challenging for both for organizations in particular and societies in general.
Without proper controls, AI systems can amplify, perpetuate, and/or exacerbate undesirable outcomes for individuals and communities. With proper controls, AI systems can mitigate and manage inequitable outcomes. AI incidents can erode trust in technology, increasing regulatory needs, as well as raising compliance and accountability issues. They impact various stakeholders, including individuals, organizations, and society at large.
On that fateful — and almost fatal — day back in September, 1983, Stanislav Petrov faced a critical decision: treat the warning of ballistic missile attack as a false alarm or alert his superiors, who would probably launch a counterattack. Petrov went with the former, reasoning the United States was unlikely to start a nuclear war with such a small aresenal. Thankfully, his intuition was correct. As the Arms Control Association explains , “The satellites had mistaken the reflection of sun off clouds for attacking missiles.” Although today’s AI incidents are unlikely to have such consequential outcomes as ending life as we know it, companies need to prioritize responsible AI development and governance today, because bad actors out there and they will stop at nothing to get what they want.
