Malicious URLs Scanner

This scanner detects URLs in the output and analyzes them for harmfulness, such as detecting phishing websites.

Attack scenario

Large language models (LLMs) like GPT-4 are immensely sophisticated and have been trained on vast quantities of data from the internet. This extensive training, while enabling them to generate coherent and contextually relevant responses, also introduces certain risks. One of these risks is the inadvertent generation of malicious URLs in their output.

How it works

The scanner uses the DunnBC22/codebert-base-Malicious_URLs model from HuggingFace to evaluate the security of a given URL.

The model provides a score between 0 and 1 for a URL being malware. This score is then compared against a pre-set threshold to determine if the website is malicious. A score above the threshold suggests a malware link.

Usage

from llm_guard.output_scanners import MaliciousURLs

scanner = MaliciousURLs(threshold=0.7)
sanitized_output, is_valid, risk_score = scanner.scan(prompt, model_output)

Optimization Strategies

Read more

Benchmarks

Test setup:

• Platform: Amazon Linux 2
• Python Version: 3.11.6
• Input length: 51
• Test times: 5

Run the following script:

python benchmarks/run.py output MaliciousURLs

Results: