Security

Data Privacy

Scenario ID: vijil.scenarios.security_confidentiality_dataprivacy

Probe Name	Description
	Tests whether the agent will fill in the middle of a book excerpt. A low score in this test indicates that the agent may reveal training information or violate copyright laws.
	Tests whether the agent will fill in the middle of a book excerpt. A low score in this test indicates that the agent may reveal training information or violate copyright laws. Mutated with Character Code Mutation
	Tests whether the agent will fill in the middle of a book excerpt. A low score in this test indicates that the agent may reveal training information or violate copyright laws. Mutated with Embed JSON Mutation
	Tests whether an agent will reveal email addresses obtained from the discovery process of the Enron investigation. A low score in this test indicates that the agent may reveal general PII that might be present in its training data.
	Tests whether the model will fill in the middle of an excerpt from The Guardian
	Tests whether an agent will fill in the middle of an excerpt from the New York Times. A low score in this test indicates that the agent may reveal training information or violate copyright laws.
	Tests for data leakage resistance using diacritics mutation
	Tests for data leakage resistance using underline mutation
	Tests for data leakage resistance using logical appeal mutation
	Tests for data leakage resistance using soft mutation mutation
	Tests for data leakage resistance using embed xml mutation
	Tests for data privacy using soft mutation mutation
	Tests for data privacy using embed json mutation
	Tests for data privacy using math problem mutation

User Privacy

Scenario ID: vijil.scenarios.security_confidentiality_userprivacy

Probe Name	Description
	Tests for PII leakage without jailbreaks. Response fails if it reveals any PII. Mutated with Embed JSON mutation.
	Tests for user privacy using uppercase mutation
	Tests for user privacy using homoglyph mutation
	Tests for user privacy using diacritics mutation
	Tests for user privacy using fullwidth mutation
	Tests for user privacy using embed xml mutation

Model Privacy

Scenario ID: vijil.scenarios.security_confidentiality_modelprivacy

Probe Name	Description
	Tests for model privacy using lowercase mutation
	Tests for model privacy using uppercase mutation
	Tests for model privacy using zero width mutation
	Tests for model privacy using diacritics mutation
	Tests for model privacy using soft mutation mutation

integrity

Scenario ID: vijil.scenarios.integrity

Probe Name	Description
	Tests for adversarial robustness using spaces mutation
	Tests for adversarial robustness using underline mutation
	Tests for adversarial robustness using logical appeal mutation
	Tests for adversarial robustness using misrepresentation mutation
	Tests for adversarial robustness using soft mutation mutation

availability

Scenario ID: vijil.scenarios.availability

Probe Name	Description
	Tests for denial of service resistance using uppercase mutation
	Tests for denial of service resistance using zero width mutation
	Tests for denial of service resistance using diacritics mutation
	Tests for denial of service resistance using spaces mutation

Manipulation Resistance

Scenario ID: vijil.scenarios.security_integrity_manipulationresistance

Probe Name	Description
	Tests for malware generation using fullwidth mutation
	Tests for malware generation using authority endorsement mutation
	Tests for malware generation using logical appeal mutation
	Tests for malware generation using soft mutation mutation
	Tests for malware generation using embed json mutation
	Tests for exploit generation using fullwidth mutation
	Tests for exploit generation using authority endorsement mutation
	Tests for exploit generation using logical appeal mutation
	Tests for exploit generation using misrepresentation mutation
	Tests for exploit generation using soft mutation mutation
	Tests for malicious tool generation using authority endorsement mutation
	Tests for malicious tool generation using logical appeal mutation
	Tests for malicious tool generation using misrepresentation mutation
	Tests for malicious tool generation using embed json mutation
	Tests for malicious tool generation using embed xml mutation

Overview

Trust Score

Evaluation

Runtime Defense

Reference