Detecting Malicious AI Models

Learn how Xray detects malicious AI models and which model types it scans.

JFrog Xray detects malicious machine learning models based on artifact scanning. This feature uses a model scanning mechanism that automatically scans machine learning models uploaded to public model repositories. It detects potential security risks and malicious code in various machine learning models. For more information about supported model scanning, see ML Models.

For example, Xray found the malicious package MustEr/gpt2-elite.

Supported Repositories

Xray currently supports scanning of AI models from the following repository:

Hugging Face: A popular platform for sharing and collaborating on machine learning models, particularly in natural language processing.

This repository is chosen because it is widely used in the AI and machine learning community, so scanning covers a significant portion of publicly available models.

Types of Models Scanned

The scanners detect malicious code in the following model types. Xray identifies the model types by their binary header values and by their file extensions.

H5: .h5 and .hdf5
Paddle: .pdparams
PyTorch: .bin, .pt, .pth, and .ckpt
Pickle: .pkl and .dat
NumPy: .npy
Joblib: .joblib
Dill: .dill
SavedModel: .pb
GGUF: .gguf

Each model type is associated with specific file extensions, and the processing functions are tailored to analyze the model's structure and content.

Scanning Frequency

The model scanning process is automated and occurs at regular intervals.

Xray scans automatically every few hours.
This frequent scanning ensures that newly uploaded or updated models are quickly analyzed for potential security risks.

Malicious Model Protection in Xray

To maintain an up-to-date defense against potential threats:

Xray continuously updates the database of known malicious models.
This update adds the latest findings from the scanning process and other security intelligence sources to the Xray database, so you're protected against the latest malicious models.

Scanner Logic

The scanning process uses sophisticated techniques to detect potential security risks.

The scanners identify models that execute code during the model loading process, which is critical for potential security breaches.
An advanced verification system developed by the JFrog Security Research team further analyzes the auto-executed code to determine whether it is malicious.
This secondary review process helps determine whether the flagged model is a true positive, meaning it is actually malicious, or a false positive.
The multi-layered verification ensures accuracy in the malicious model identification process and continuously improves the scanning algorithms.

Hugging Face Dangerous Classification

Not all Hugging Face models marked Dangerous are malicious, and not all malicious models are marked Dangerous.

The advanced security system checks each flagged model.
This process distinguishes true positives, which are actual security risks, from false positives, which are benign models that triggered a warning.
The automated system is designed to maintain a high level of security while avoiding unnecessary restrictions on legitimate models.
The automatic verification process is particularly adept at recognizing and clearing false positives that may arise from high-quality research, thanks to the expertise built into the system by the JFrog Security team.
Hugging Face can flag a model as Dangerous when Xray doesn't, and Xray can flag a model when Hugging Face doesn't.

For example, a model can have malicious content even when Hugging Face doesn't mark it as Dangerous.

For example, Hugging Face can mark a model as Dangerous even when the model has no malicious content.

Beyond security detection, you can manage models that Xray identifies in the JFrog AI Catalog. The catalog is a centralized registry for Detect Shadow AI, JFrog Curation, and policy-based approval of machine learning assets.

Frequently Asked Questions

This section provides answers to frequently asked questions.

plusFAQs

Q: How does Xray detect a malicious AI model?

A: The scanners flag models that execute code while the model loads. A verification system then checks whether that code is actually malicious. See Scanner Logic.

Q: What is the difference between a Hugging Face Dangerous model and a model Xray marks as malicious?

A: Hugging Face can mark a model Dangerous when Xray doesn't, and Xray can mark a model malicious when Hugging Face doesn't. A Dangerous label is not the same as an Xray malicious finding. See Hugging Face Dangerous Classification.

Q: Which model types can Xray scan for malicious code?

A: Xray scans H5, Paddle, PyTorch, Pickle, NumPy, Joblib, Dill, SavedModel, and GGUF files. It identifies each type by the binary header and the file extension. See Types of Models Scanned.

Q: How often does Xray scan public models?

A: Xray scans automatically every few hours. The database of known malicious models is updated continuously with those findings. See Scanning Frequency.

Q: How can I manage a model that Xray flags?

A: You can manage it in the JFrog AI Catalog, which provides detection, JFrog Curation, and policy-based approval. For more information, see Detect Shadow AI.