Your agent works in demos. Your unit tests pass. But how do you know it will not hallucinate facts, leak customer data, or comply with malicious instructions when it encounters inputs you did not anticipate?
Diamond is Vijil’s evaluation platform. It sends hundreds of adversarial Probes to your agent (prompt injections, jailbreak attempts, data exfiltration payload, etc.) and measures how your agent responds. You get a quantified Trust Score and specific findings you can fix before deployment.
How Evaluation Works
Diamond sends test Probes to your agent and analyzes the responses:Trust Score
The Trust Score is a composite metric (0 to 100) based on three pillars:
Higher scores indicate more trustworthy behavior. Use scores to:
- Set deployment gates (e.g., require Trust Score ≥ 70)
- Compare agent versions
- Track improvements over time
- Identify specific vulnerabilities
Evaluation Options
Cloud-Hosted Agents
Evaluate agents deployed on supported cloud platforms such as OpenAI, Anthropic, AWS Bedrock, Google Vertex AI, DigitalOcean, and any OpenAI-compatible endpoint.Local Agents
Evaluate agents running locally without deployment. This creates a temporary authenticated tunnel (via ngrok) for Vijil to communicate with your local agent.Available Harnesses
Add
_Small suffix (e.g., security_Small) for faster iterations during development.
Evaluation Workflow
1
Choose your integration method
Use cloud provider integration for deployed agents, or LocalAgentExecutor for local development
2
Select harnesses
Start with
trust_score for comprehensive coverage, or specific Harnesses for targeted testing3
Run evaluation
Execute via Python client, REST API, or console
4
Analyze results
Review Trust Score, dimension scores, and individual failures
5
Remediate issues
Fix vulnerabilities, improve prompts, add Guardrails
6
Re-evaluate
Confirm fixes and track improvement
Rate Limiting
Control the evaluation pace to avoid overwhelming your agent or hitting API limits during evaluations.
Use the
_Small Harness variants (e.g., security_Small) for faster iteration runs that sample a representative subset of Probes.
Work in Progress
The programmatic evaluation capabilities are currently in private preview and subject to change.
Next Steps
Run Evaluations
Execute evaluations and monitor progress
Understand Results
Interpret scores and failures
Cloud Providers
Configure cloud platform integrations
Custom Harnesses
Create targeted evaluation Scenarios