Closing the Gap Between AI Principles and AI Reality
Table of Contents
Introduction
Artificial Intelligence is in the midst of a gold rush. In the last two years alone, more than US$95 billion worth of investment has poured into AI startups, driving a frenzied pace of innovation and adoption across almost every industry.
And, as is often the case in gold rushes, it’s probable that the value of the core craze will ultimately be eclipsed by the economies that grow around it—if anyone makes more money than the gold miners, it’s usually the people selling spades. In the AI boom that we’re living through now, that means more and more consultancies offering services that ostensibly help businesses translate the potential value of AI into commercial return.
In its essence, this is a crucial role.
Few companies who were around when ChatGPT launched in November 2022 were prepared for the level of disruption that followed. Seemingly overnight, AI moved from the periphery to the dead center of corporate strategy. Of course, OpenAI’s flagship product was far from the sole catalyst for the unfolding AI revolution (see my A Very Brief History of AI), but it was the one that most profoundly captured the public imagination, and therefore the one that set boardrooms scrambling for answers. The number one question that has dominated those executive debates has been, “What does this mean for our business?” and, following on from that, “What can we do about it?”
Top consultants are able to help leaders find solutions, supporting them as they navigate an AI landscape that is transforming at an accelerating pace. In my own work, that means specialized guidance on securing AI development and integration in a way that ensures any investment in AI delivers the good it’s intended for, not unforeseen harm.
However, as in any gold rush, rampant market growth means the lines of competence and credibility inevitably become blurred. The global AI consulting market is expected to grow at a CAGR of 37.46% between 2022 and 2028, reaching a projected market size in excess of US$630 billion. That’s a lot of people selling spades, and many clients aren’t equipped to tell the real difference between one offering and another.
Whether they know it or not, every decent business wanting to leverage AI is looking to do so in a way that is trustworthy and responsible, delivering AI systems that are ethical, transparent, and secure, while aligning their development and deployment with human values, fairness, and accountability. Over the longer term, that’s what will ensure brand sustainability while maintaining public trust.
Yet, in pursuit of AI systems that are ethical and robust we are seeing the emergence of an, ironically, ethical challenge: firms rushing to position themselves as leaders in Responsible AI without the necessary depth in technical expertise. Though their motivations for doing so may not be malicious, the impact is the dilution of the AI security field, something which society cannot afford as AI becomes more and more integrated into our workplaces, homes and everyday life.
In this article, we will explore the difficulties of achieving of AI security in an era of rapid technological growth, examining the gap between the high-minded ideals of “Principled AI” and the technical realities of securing AI systems. We will also discuss why it is critical to develop a workforce with the skills and expertise to tackle AI’s unique security risks, ensuring that the promises of Responsible AI are matched by the practical competence needed to deliver on them.
The Meaning of Responsible AI: Ethics, Accessibility, and the Gap in Security
Humankind’s mistrust of machines started pretty much as soon as we began building them. In the early 19th-century, the Luddites, a group of English textile workers, broke into industrial era factories and destroyed the machines they believed were stealing their livelihoods. Today, the same fears persist, only now they are targeted at AI. During the interceding two hundred years, popular culture has been filled with post-apocalyptic visions of a world in which people are dominated by the machines they created, from Samuel Butler’s 1870 novel Erehwon, to the modern era’s Terminator and scores of similar films on the ‘killer robot’ theme. I am exploring some of the real killer robot related risks at my site Defence.AI.
As AI systems become more deeply embedded in the fabric of our lives, many of these concerns still linger. Luminaries close to the frontier of AI development see AI as an existential risk, as do I, though most people in the UK, for example, the birthplace of the industrial revolution, regard AI as positive, with particular value perceived in use cases like disease diagnosis. But they also want regulation, and have concerns around privacy and a lack of transparency and accountability in decision-making.
These desires for ethical conduct, transparency, and accountability are core to the concept of Responsible AI, the practice of designing, developing, and deploying AI systems that are aligned with societal values, that minimize harm and maximize benefit.
[I describe Responsible AI in detail here, along with the related concepts of Trustworthy AI, Safe AI and Secure AI]
Principled AI
At the root of Responsible AI is a set of principles that are gradually solidifying over time as more and more institutions are trying to codify exactly what Responsible AI looks like. Under the umbrella of “Principled AI”, these themes provide the ethical structure that shapes the governance of Responsible AI systems.
The area has seen significant investment of time and resource over the last few years, with more than 84 public-private initiatives publishing a set of guiding principles or tenets intended to inform the responsible design and deployment of AI. A Harvard review of 36 of the most prominent principles documents found that there are eight key themes that are common to most Principled AI frameworks:
- Privacy
AI systems must protect individual privacy, ensuring transparency, control, and consent over data use. - Accountability
AI systems need clear accountability for their actions, with mechanisms for oversight and recourse. - Safety and Security
AI must be designed to function safely and securely, protecting against misuse or vulnerabilities. - Transparency and Explainability
AI should operate transparently and be able to explain its decisions clearly. - Fairness and Non-Discrimination
AI must prevent bias and promote fairness and inclusivity in both design and outcomes. - Human Control of Technology
Humans should retain control over AI, especially in critical decisions. - Professional Responsibility
AI developers must act ethically, considering long-term impacts and consulting stakeholders. - Promotion of Human Values
AI should align with human values, promoting well-being and societal benefit.
These themes mirror fundamental societal concerns, and are understandable to most educated individuals. That is what makes them so useful. However, that accessibility also helps feed an illusion that conceptual understanding equals technical competence. Understanding the ‘whats’ of Principled AI will not naturally lead to executing the ‘hows’ of Responsible AI, or grant the expertise required to deliver Secure AI.
Low Barriers to Entry for “Responsible AI” Consultancies
The heightened demand for Responsible AI has stimulated a rise in the number of consultancies and professional services firms offering “Responsible AI” assessments and support. Perhaps this is not surprising–the concept of Responsible AI, with its focus on ethics, governance, and transparency, is accessible to non-technical audiences, so many firms see it as an easy bolt-on to their existing offering. And with a broad supply of readily available frameworks to employ, but no regulatory requirements to satisfy, “Responsible AI” services can be developed with relative ease and speed.
With such low barriers to entry, many consultancies—especially those traditionally focused on governance, risk management, or business strategy—can easily position themselves as experts in Responsible AI without needing to invest in the deep technical expertise that underpins AI development. After all, it’s relatively straightforward to talk about ethical principles, develop governance frameworks, and advise on data privacy policies without having to understand the complexities of machine learning algorithms or the inner workings of AI systems.
However, while these non-technical aspects of Responsible AI are important, they represent only part of the puzzle. True Responsible AI is not just about governance and ethical principles, it is about ensuring that AI systems function safely, securely, and in a manner that minimizes risk.
Moving beyond Responsible AI to Secure AI
As we increasingly engage and transact with artificial intelligence, we need to drill down below the broad ethical, legal, and social considerations of Responsible AI and focus on Safe and Secure AI: approaches that are concerned with the specific technical challenges involved in safeguarding AI systems against attacks, vulnerabilities, and misalignment. Principles alone cannot safeguard the confidentiality, integrity, and availability of AI systems. That requires specialized skills and knowledge often missing from the solutions offered by “Responsible AI” agencies.
However, the gap between knowing about Responsible AI and knowing how to secure AI systems is not the only troubling inconsistency in this area. Even within the security field itself there is a worrying tendency to underplay the specific needs of AI security.
AI systems are uniquely vulnerable to a range of security risks that differ from those in traditional software or IT infrastructure. These technical threats are highly specialized, requiring expertise in, among others, machine learning, cryptography, and AI threat modeling. Traditional cybersecurity practices, while essential, are not designed to address the nuances of AI vulnerabilities. Yet, many firms go to market with services dressed up as AI security that, on closer inspection, turn out to be classical cybersecurity frameworks simply applied to AI systems. This failure to recognize the need for AI-specific approaches is sometimes due to a simple lack of technical capability; sometimes disingenuous marketing; and sometimes outright deception. Regardless of the motive, it is always dangerous, because traditional cybersecurity methods simply are not sufficient to keep AI systems safe and secure.
AI-Specific Security Risks
AI-specific security threats are fundamentally different from those encountered in traditional IT environments, and neutralizing them requires specialized interventions. Conventional approaches focus on securing infrastructure, preventing unauthorized access, and mitigating known software vulnerabilities. However, AI security involves safeguarding the algorithms, data, and model outputs themselves—elements that are far less predictable and much harder to control.
Traditional cybersecurity frameworks often rely on well-defined threat models and defense mechanisms tailored for software and systems that are deterministic. AI systems, by contrast, are dynamic and continuously learning. They can exhibit emergent behaviours. They are are non-deterministic. And from the security perspective, that is a problem. I wrote more about it here “Magical” Emergent Behaviours in AI: A Security Perspective. AI systems are trained on large and sometimes unstructured datasets, and the security and privacy of these datasets is critical to the integrity and security of the AI systems. AI models are also exposed to unique threats that exploit the way AI models learn, process, and generate outputs, making them particularly dangerous:
- Adversarial Attacks: In these attacks, malicious actors subtly manipulate input data to deceive an AI model into making incorrect predictions or classifications. For instance, an adversarial image designed to fool a facial recognition system might appear unaltered to human eyes but lead the model to misidentify a person. The challenge with defending against these attacks is difficult because the manipulations are often imperceptible to humans but can drastically affect AI models. Traditional monitoring approaches would not be able to detect such attacks. These attacks highlight how easily AI models can be exploited and underscore the need for robust defense mechanisms against them. See more here Adversarial Attacks: The Hidden Risk in AI Security.
- Model Inversion: Model inversion attacks allow bad actors to reconstruct sensitive data from an AI model’s outputs. Attackers can reconstruct sensitive training data (model inversion) or replicate the AI model itself (model extraction) by exploiting access to the model’s outputs. For example, a machine learning model used in healthcare might be queried to reveal individual patient information, even if it was not directly accessible. This vulnerability raises serious concerns about privacy and confidentiality, especially when AI models are trained on sensitive datasets. See more here How Model Inversion Attacks Compromise AI Systems.
- Data Poisoning: Data poisoning involves injecting malicious data into an AI system’s training process, compromising its accuracy and reliability. Attackers may subtly corrupt the dataset used to train a machine learning model, causing the system to learn incorrect patterns or biases. This can have devastating consequences, as poisoned AI models may produce unreliable or harmful outcomes. See more here Understanding Data Poisoning: How It Compromises Machine Learning Models.
- Emergent Behaviors and Non-Determinism: AI systems can exhibit unexpected behaviors not anticipated during development, particularly when interacting with complex environments. Predicting and controlling these behaviors is difficult, complicating efforts to ensure consistent and secure operation. See more here “Magical” Emergent Behaviours in AI: A Security Perspective.
- Data Leakage: AI systems rely heavily on large datasets, often containing sensitive or proprietary information. Data leakage can occur when sensitive data is inadvertently exposed during training, sharing, or querying of AI models. Protecting datasets and ensuring secure data handling practices are essential to preventing leakage and safeguarding intellectual property.
The ability of AI systems to autonomously make decisions—whether in financial trading, healthcare, or transportation—means that a single security breach could result in large-scale cyber or cyber-physical consequences. Traditional cybersecurity measures are ill-equipped to handle these unique vulnerabilities.
As I explain in detail in my paper Addressing the Full Stack of AI Concerns, establishing and maintaining the requisite levels of AI security requires a layered defense. We have to apply all of the well-worn enterprise cybersecurity fundamentals like access control; data encryption; network security; physical security; incident response and recovery; and regular audits and compliance checks. Then, we need to consider a host of AI-specific factors like MLOps, LLMOps and data pipeline security, model security, and monitoring and capability control.
Ultimately, securing AI requires a combination of deep technical expertise, robust security protocols, and a commitment to continuous monitoring and adaptation. Unfortunately, the market is not offering these in any meaningful way. Techies with AI skills are being lured into the well-funded arena of AI development, where they stand to earn significant returns, leaving a growing gap on the side of AI risk. In short, there is a looming skills shortage in AI security, which it is crucial we remedy.
The Current Skills Gap in AI Security
AI systems, particularly those using machine learning models, are vulnerable to a variety of threats that require specialized knowledge in areas like algorithmic behavior, data science, and the intricacies of how models learn and function.
However, I, and more than a third of technology professionals, believe the industry is seeing a critical shortage of these types of skills, leaving organizations vulnerable to risks that are either poorly understood or inadequately mitigated. At the same time, 88% of cybersecurity experts report that AI is already impacting their existing roles. The rapid adoption of AI technologies has outpaced the development of security standards, best practices, and training programs that address AI’s specific risks. And, the influx of ill-equipped consultancies purporting to deliver Responsible and Secure AI is only helping to mask the problem.
To close the gap, it is critical to train AI practitioners in security principles, just as it is necessary to upskill cybersecurity professionals in AI-specific risk management. Effective AI security demands a blend of both domains: understanding the fundamentals of cybersecurity while mastering the unique challenges of AI/ML systems.
Key skills that need to be developed among AI practitioners include:
- Data Integrity and Protection: AI systems rely on vast amounts of data for training and operation. Ensuring that this data remains untampered and uncorrupted is crucial to maintaining the system’s reliability. AI practitioners need to be trained on techniques like secure data pipelines, encryption of datasets, and the use of differential privacy to protect sensitive information.
- Model Explainability and Transparency: One of the biggest challenges in AI security is the “black box” nature of many models. Training AI developers to prioritize explainability and transparency is essential not only for ethical AI but also for understanding security risks. Techniques that enhance model interpretability help identify anomalies and vulnerabilities, making it easier to audit and secure AI systems.
- Robustness Against Adversarial Attacks: AI practitioners must be trained in adversarial defense strategies, such as adversarial training, to harden models against these types of threats.
- AI-Specific Threat Models: Security professionals must learn to build threat models tailored to AI systems, considering risks like adversarial input manipulation, model theft, and data poisoning.
- Monitoring and Auditing AI Systems: Continuous monitoring of AI systems for unusual behavior or performance is essential for identifying attacks or security breaches. Security teams need to understand how to monitor AI systems and perform real-time audits on data flows and model outputs.
Just as with AI security itself, closing the AI security skills gap will require a layered approach, incorporating a range of initiatives from individual professional training programs to industry-wide standardization. Learning and development should be formal–foundational cybersecurity education and specialized AI security training–and ongoing, through collaborative sharing of best practices, constantly evolving industry standards and guidelines, and recruitment of professionals with hybrid skills in AI and cybersecurity who are able to advance the field.
Conclusion
In a rapidly shifting AI ecosystem, it is crucial that we maintain clarity and awareness of the bigger aim: a world in which AI is integrated into society with minimal risk to humanity’s wellbeing. In the rush to capitalize on the Responsible AI movement, firms must not lose sight of the deeper responsibility that comes with deploying AI systems in the real world. By marketing themselves as Responsible AI specialists while underplaying the mechanics of AI security, these groups may be contributing to a false sense of confidence among businesses and consumers. The consequences of such misconceptions could be severe: from financial loss and reputational damage to compromised privacy and safety. To truly live up to the promise of Responsible AI, organizations must embrace both the ethical principles that guide responsible development and the technical rigor required to secure AI systems. This means investing not only in governance and policy but also in technical expertise—ensuring that AI systems are designed, developed, and deployed with security at their core. In the short term that means hiring genuine experts to do the work, but in the long term it requires building an AI risk workforce.
For 30+ years, I've been committed to protecting people, businesses, and the environment from the physical harm caused by cyber-kinetic threats, blending cybersecurity strategies and resilience and safety measures. Lately, my worries have grown due to the rapid, complex advancements in Artificial Intelligence (AI). Having observed AI's progression for two decades and penned a book on its future, I see it as a unique and escalating threat, especially when applied to military systems, disinformation, or integrated into critical infrastructure like 5G networks or smart grids. More about me.