Navigating the AI Security Risks: Understanding the Top 10 Challenges in Large Language Models

In the rapidly evolving world of artificial intelligence, Large Language Models (LLMs) like GPT-4 have emerged as groundbreaking tools, offering unparalleled capabilities in natural language processing and generation. These capabilities have been leveraged by many in the engineering industry for everything from enhancing customer service to powering advanced research, to lower level code generation, QA, and many other engineering applications.  The ways LLMs have been applied are many, and equally impressive. However, as with any revolutionary technology, they come with their own set of challenges and risks.

In this blog, we dive into the Top 10 AI & ML security risks associated with managing and deploying LLMs. Understanding these risks is not just about mitigating potential drawbacks; it's about responsibly harnessing the power of LLMs to ensure they are safe, fair, and beneficial for all. Whether you're a developer, a business leader, or just an AI enthusiast, this overview will provide you with a comprehensive insight into the challenges and considerations of working with these advanced models.

ML and AI Security - Mitigating the Top 10 Risks

Let's delve into this list of security risks associated with managing and deploying Large Language Models (LLMs) like GPT-4:

1. Prompt Injection Attacks. 

Prompt injection attacks are from the same family as any malicious injection attacks we are familiar with in the security engineering domain.  AI is not exempt from this kind of malicious activity.  Below we will explain how it works.

Example: An attacker designs a sequence of inputs that cause the model to switch contexts or modes, effectively bypassing filters or controls. For instance, using a sequence of seemingly innocuous inputs that, when processed in order, cause the model to interpret subsequent inputs as commands rather than regular queries, potentially exposing sensitive data or system functionalities.

2. Model Hallucination

While many of us have come to rely on GPT models for many important and day to day tasks, like with other engineering areas where the models aren’t flawless, and LLMs are known to make errors - this can be even more problematic in the world of data science & research.  It’s incredibly important to be aware of situations where data can be fabricated, the AI thinks the data they are providing is fact-–where research indicates this happens with LLMs as often as 27% of the time.

Example: When asked to provide statistical data, the model might generate convincing but entirely fabricated dataset statistics. This could be particularly problematic in fields like finance or healthcare, where decisions are data-driven. For example, generating false medical statistics about a drug's effectiveness.

3. Denial of Service (DoS)

Another recurring risk that is commonly exploited no matter what your stack, are Denial of Service attacks.  LLMs, and particularly their chat tools are at risk of being abused by malicious actors for DoS attacks.  

Example: Automated scripts rapidly sending complex, resource-intensive prompts (like those requiring deep contextual understanding or generating lengthy outputs) which overwhelm the model's computational resources, causing service slowdowns or crashes.

4. Training Data Poisoning

In today’s data driven world, much of our machine learning and AI data is built upon refining and tuning ML models to get the most accurate data sets possible.  Therefore, when these models are hijacked (i.e. poi, this can be damaging to actual data and research initiatives.  Below we’ll describe a modern attack scenario that can be risky for ML researchers.

Example: An attacker gains access to the dataset used for fine-tuning the model and inserts subtly biased or incorrect information. For instance, injecting slanted political content into articles used for training, which might lead the model to generate politically biased content.

5. Insecure Output Handling

In the same way that humans are oftentimes a weak link with exposing risky data––anything from technology stacks to social engineering, AI despite being machines are not immune to such mistakes either.  Some risks that may arise from leveraging AI include exposing important data on public resources.

Example: A chatbot built on an LLM automatically posts its outputs to a public forum without adequate filtering, leading to the unintentional publication of offensive or radical content, potentially even violating laws or platform policies.

6. Adversarial Attacks

With every technology come those bad actors who look to exploit them.  There are malicious prompt engineers today that are targeting AI models’ and their core technologies, building prompts that specifically know how to exploit LLM systems and all for their own personal gain.  Building guardrails around these capabilities will help secure your systems from malicious prompt engineering.

Example: Crafting inputs that exploit specific vulnerabilities in the model's tokenization or prediction algorithms, causing it to generate nonsensical, harmful, or biased outputs. For instance, input sequences that are known to trigger unintended model behavior, like generating aggressive or derogatory responses.

7. Bias and Fairness Issues

One of the central topics that consistently trouble data scientists is the morality, ethics and fairness of AI models that are solely based on machine generated data.  MIT tried to combat this through a long-standing game called the “moral machine”, used to create machine learning models to help build moral self-driving vehicles, and to ensure they make the right judgment calls in real time.  There are many examples of “machine bias”, and we need to be careful when using AI generated data of this potential risk.

Example: The model, trained on historical hiring data, may inadvertently perpetuate gender bias, unfavorably scoring resumes for a technical position when they contain female indicators (like names or women’s colleges).

8. Data Privacy and Leakage

Similar to the Insecure Output Handling risk, there is always a risk of data privacy and leakage, even internally with machines that cannot differentiate between confidential information, and just typical data they output.  It is important to take the proper precautions with sanitizing sensitive data, including PII and other critical data that could be considered data breaches if publicly accessed or leaked broadly.

Example: When asked about sensitive topics, the model might generate responses that include specific details from training data, such as reciting a unique incident from a confidential corporate document or a patient's health record that was part of the training set.

9. Manipulation and Generated Content

Anything that can be gamified, can ultimately be manipulated by bad actors, and this goes for any public-facing ratings or opinion systems.  With the sophistication of AI and chat-based LLMs, it is now possible to create fake reviews and even impersonation of real people at scale.  Those engineering these systems need to be aware and attuned to these risks, likewise those consuming this data, and understand that rating-based systems may begin to be compromised and to be careful when making business-critical decisions based on these algorithm based systems.

Example: Creating highly realistic fake reviews for products or businesses. The model, trained on genuine user reviews, can be fine-tuned or prompted to generate authentic-sounding but entirely fabricated reviews, manipulating public opinion or business ratings.

10. Dependency Vulnerabilities

These days, with all of the OSINT available, it’s quite easy to know which open source packages depend on other open source packages, where somewhere in the chain of these dependencies––due to the volume and consistently growing number of CVEs, it is quite possible to find a significant enough exploit in your LLM systems.  Malicious entities may scour the public web for environments that leverage compromised and vulnerable libraries and packages, and attempt to infiltrate and harm your systems, models and data, as a result.

Example: The LLM deployment environment might rely on third-party libraries for data processing or network communication. If one of these libraries has an unpatched vulnerability, it could be exploited to inject malicious code into the system, compromising data integrity or availability.



While the risks associated with LLMs are significant, there are various strategies and tools available to mitigate these challenges. One such tool is 'garak', an open-source vulnerability scanner for LLMs. The way it works is by probing LLMs for vulnerabilities like hallucination, data leakage, prompt injection, and more. This proactive approach allows developers and researchers to identify and address weaknesses, ensuring robust and secure AI model deployment.

Beyond specific tools, there are broader strategies that can be employed, such as rigorous testing and monitoring that are crucial to detect such threats. Regularly evaluating the model's outputs and updating its training data can help in minimizing biases and inaccuracies. Like with all engineering disciplines, it’s important to implement strict access controls and ethical guidelines that can prevent misuse and manage the impact of any generated content.

Collaboration and open dialogue within the AI community is also playing a vital role to mitigate these risks. Sharing knowledge about vulnerabilities, attack vectors, and defensive techniques can significantly enhance the overall security posture of LLMs. This collaborative approach fosters a culture of continuous improvement and collective responsibility.

Staying Ahead of AI Security 

As we've explored, the path of innovation with Large Language Models is not without its obstacles. The risks ranging from prompt injection attacks to dependency vulnerabilities highlight the complexity and responsibility that come with deploying these powerful tools.

However, recognizing these risks is the first step towards mitigating them. By being aware of and actively addressing these challenges, we can steer the development of LLMs in a direction that maximizes their potential while safeguarding against their misuse and the harm this brings with it. As we continue to advance in this field, it is crucial that developers, users, and policymakers collaborate to establish guidelines and best practices that ensure the ethical and secure utilization of LLM technology.

In the end, the goal is clear: to create an AI-driven future that is not only innovative and efficient but also safe, transparent, and equitable for all. The journey of LLMs is just beginning, and with mindful stewardship, the possibilities are limitless.