Artificial Intelligence Security Posture Management

Table of Contents

• Introduction to Artificial Intelligence
• AI Application Ecosystem Components
• AI-SPM Overview
• Onboarding AI-SPM
• Supported Assets

Introduction to Artificial Intelligence

Artificial intelligence (AI) enables computers and machines to simulate human intelligence and problem-solving. Common AI examples include digital assistants, self-driving cars, and tools like ChatGPT.
In computer science, AI includes machine learning and deep learning, which create algorithms inspired by the human brain's decision-making. These systems 'learn', in a manner similar to humans, from data and improve their accuracy in making predictions over time.
Recent AI advancements, like ChatGPT, mark significant progress, especially in natural language processing (NLP). Today, generative AI can understand and create human language, images, videos, software code, and even molecular structures.
AI applications are growing rapidly, providing new opportunities to streamline business processes, and enabling new use cases that were not possible before.

What is a Large Language Model?

A large language model (LLM) is a computational model recognized for its ability to perform general-purpose language generation and various natural language processing tasks like classification. LLMs develop these capabilities by learning statistical patterns from vast amounts of data during an intensive training process. They can generate text - a form of generative AI, by taking an input and predicting the next token or word repeatedly.
Some prominent LLMs include OpenAI's GPT series (such as GPT-3.5 and GPT-4, used in ChatGPT and Microsoft Copilot), Google's Gemini, Meta's LLaMA models, Anthropic's Claude models, and Mistral AI's models.

Security Risks and Attack Vectors in Large Language Model Deployments

When insecure models and deployments are used, several types of attacks can occur, many of which align with the OWASP (Open Web Application Security Project ) Top 10 for LLMs:

• Data Poisoning Attacks: In "Training Data Poisoning", malicious actors manipulate the training data to introduce biases or vulnerabilities into the model, causing it to make incorrect or harmful predictions.
• Model Inversion Attacks: Attackers can infer sensitive information about the training data by querying the model, potentially leading to data breaches and loss of intellectual property.
• Adversarial Attacks: Reflecting OWASP's "Adversarial Input," crafted inputs can deceive the model into making incorrect predictions, which is particularly dangerous in critical applications like autonomous driving or medical diagnosis.
• Evasion Attacks: Evasion attacks are a prevalent threat to machine learning models during inference. This type of attack involves crafting inputs that appear normal to humans but are misclassified by machine learning systems. For instance, an adversary might alter a few pixels in an image prior to submission, causing an image recognition system to misidentify it.
• Model Extraction Attacks: Attackers can approximate a model's functionality by repeatedly prompting it, effectively stealing the intellectual property and potentially using it for malicious purposes.
• Data Leakage: If a model unintentionally reveals sensitive information it was trained on or data that is used in inference, it can lead to breaches of confidential or personal data.
• Model Manipulation: Unauthorized access to the model can allow attackers to alter its parameters or behavior, leading to compromised functionality and trustworthiness.
• Inference Attacks: Attackers exploit the model to deduce whether specific data was part of the training set, potentially exposing sensitive information.

These attacks highlight the importance of implementing robust security measures, as outlined by the OWASP Top 10 for LLMs, in the deployment and maintenance of large language models and other AI systems.

AI Application Ecosystem Components

The AI application ecosystem comprises several critical components that work together to enable the functionality of AI-driven applications. Here are the main parts along with examples:

Model Endpoint

The model endpoint is the interface through which applications interact with the AI model. It acts as an access point for sending inputs to the model and receiving outputs. The endpoint is responsible for managing requests, routing them to the appropriate model instance, and returning the results to the application.
Example: An Azure OpenAII deployment using OpenAI's GPT-4, which allows developers to integrate natural language processing capabilities into their applications by sending text prompts and receiving generated text in response.
Example: Amazon Web Services (AWS) EC2 instances with GPU acceleration running Llama2 by Meta, which supports an application that communicates with the EC2 instance.

Model

The model is the core component of the AI ecosystem. It is the trained machine learning model that takes input data, processes it, and produces output. In the context of large language models (LLMs), this involves understanding and generating human-like text based on the given input.
Example: OpenAI's GPT-4 model, which can generate coherent and contextually relevant text, answer questions, and perform various other natural language processing tasks.

Plugin

A plugin is an auxiliary but highly capable model or tool that acts as a helper to the primary AI model. Plugins extend the functionality of the main model by providing specialized capabilities, such as accessing inference datasets, performing specific computations, or interfacing with other services. This approach, known as retrieval-augmented generation (RAG), enhances the primary model's ability to generate more accurate and contextually relevant outputs.
Example: A weather plugin integrated with an AI chatbot that allows the chatbot to fetch and provide real-time weather updates based on user queries. Another example is a language translation plugin that helps the main model translate text between different languages.

Training Datasets

Training is a fundamental stage in the AI development process where the model learns to perform its tasks by processing large amounts of data. During this phase, the model is exposed to various examples and adjusts its internal parameters to minimize errors in predictions or classifications. The dataset is the integral part of the process, with the insights learned by the model influenced by the training data.

Example: Training a model like GPT-4 involves using vast text corpora from various sources to help the model understand language patterns, context, and nuances, enabling it to generate coherent and contextually relevant text.

Inference Datasets

Inference datasets are specialized collections of data used during the inference phase of AI models, which is the stage where the model makes predictions or generates outputs based on new input data. Unlike training datasets, which are used to teach the model how to understand and process information, inference datasets help improve the model's performance by providing realistic, real-world data inputs for better contextual answering.

Example: when building a chatbot for customers to learn more about their spending habits, the financial institutions will use the transactions of the customer as inference data to provide contextually accurate answers

Example Scenario

AI-Powered Customer Support Chatbot

To illustrate how these components work together, consider an AI-powered customer support chatbot:

• Model Endpoint: The chatbot application interacts with the GPT-4 model through the Azure OpenAI Deployment, which serves as the model endpoint. This endpoint handles user queries, processes them, and directs them to the GPT-4 model to generate responses.
• Model: ​​The GPT-4 model receives the user's query, processes i, and generates a relevant and contextually appropriate response based on the information and nuances provided in the query.
• Plugin: The chatbot integrates a customer database plugin that allows it to fetch user-specific inference data, such as order status or account details, to provide more personalized and accurate support. The customer database used by the plugin is the Inference Dataset.
• Training Dataset: The chatbot undergoes fine-tuning* using a dataset of previous customer interactions and support tickets, making it adept at handling common inquiries and issues in the specific industry.
  *Refer to the section below, What is Fine-Tuning for more details.
• Application: The customer support platform integrates the chatbot with a user-friendly interface.

What is Fine-Tuning?

Fine-tuning in machine learning refers to the process of adapting a pre-trained model to perform specific tasks or cater to particular use cases. This technique has become essential in deep learning, especially for training foundation models used in generative AI.
Fine-tuning leverages data (similarly to training) in order to adjust the responses of the model to certain inputs, making it more suitable for the intended business case.

What is Retrieval-Augmented Generation?

Retrieval-Augmented Generation (RAG) enhances large language model (LLM) responses by incorporating information from knowledge bases and other sources. This allows the model to reference up-to-date inference data before generating a response, improving contextual accuracy. This approach is cost-effective and ensures the output remains relevant, accurate, and useful across different contexts.

AI-SPM Overview

AI-SPM provides a comprehensive overview of the AI assets within an organization. It is designed to ensure AI security by offering tools to review and prioritize AI risks effectively.

AI-SPM Overview Page
The AI-SPM Overview page serves as the central hub for information on the AI ecosystem within the organization. It provides a comprehensive overview of AI security posture and is designed to help users quickly access relevant information. The page’s layout and organization are tailored to guide users in understanding the AI environment and determining the next steps to take for effective AI management.

The icons on the left (Model Endpoints, Inference Datasets, Training Datasets, and Plugins), allow you to filter and view specific categories of assets and findings relating to different components of the AI ecosystem.
Clicking "View Training Datasets" reveals all training datasets and their data types, while "View Inference Datasets" displays all inference datasets and their data types. Additionally, selecting "View All" in the "Top AI Assets at Risk" section takes you to the AI Assets page sorted by risks.
The AI risk findings by severity section shows the number of findings categorized by their severity, giving you a clear understanding of the risk landscape in your AI environment.

AI Assets Inventory

The AI Asset Inventory page allows for viewing all AI assets in your environment, regardless of deployment mode or cloud provider. Connected assets are discovered, contextualized, and presented with detailed information. To dive deeper into the asset context, click on the asset name to navigate to the AI Asset page.

AI Risks

AI-SPM provides risk assessment for the supported AI assets, with risk rules crafted by the Prisma Cloud research team. These risk rules are designed to detect misconfigurations and security flaws in AI assets and alert on them. In addition to the default risk rules, AI-SPM also supports custom risk rule creation - allowing organizations to codify and integrate internal policies into the AI-SPM risk engine, streamlining remediation efforts. In order to create a custom risk rule, enter the rule definition as a query in the AI-SPM inventory, and click ‘Create custom risk’.

Onboarding AI-SPM

Prerequisite to Onboarding AI-SPM

Onboarding DSPM is a prerequisite to onboarding AI-SPM. This is because understanding and managing data exposure is fundamental to ensuring AI protection. By deploying DSPM first, you gain visibility into your data security posture, which is essential for the effective deployment and functioning of AI-SPM.

Integrated Security Approach

By integrating DSPM and AI-SPM, Prisma Cloud offers a unified approach to security. This integration helps in identifying potential threats and vulnerabilities across both data and AI systems, enabling a more robust and holistic security posture.

To ensure a smooth onboarding process, please refer to the detailed steps for onboarding DSPM.

When you onboard DSPM, AI-SPM is also automatically onboarded and starts running by default.

If you want to disable AI-SPM, disable scanning for your AI services in the ‘Scanning Settings’ section of the projects page.

Supported Assets

• AWS: Bedrock Provisioned Throughput, Bedrock Agent
• Azure: Azure OpenAI Deployment
• GCP: Vertex Endpoint