June 30, 2025
New Features
Organization Role-Based Access Control (RBAC)
- Granular Permissions: Comprehensive role-based access control system for different platform areas
- Team Management: Assign specific roles to control what team members can access and modify
Customer Usage Tracking
- Trace Monitoring: Track customer usage by number of traces (agent invocations)
- Cost Analytics: Monitor LLM costs per customer for better resource management
- Usage Insights: Gain visibility into customer behavior
Enhanced Metadata and Reward Modeling Support
- Integrated Trace Exports: Enhanced metadata updating interface with seamless trace export functionality
- RL/SFT Ready: Built-in support for reward modeling workflows for reinforcement learning and supervised fine-tuning
- Flexible Metadata: Easy reward score integration via
judgment.set_reward_score()
Evaluation Batching System
- Parallel Processing: Split large evaluation runs into configurable batches (default: 20 examples)
- 5x Performance Improvement: Benchmarks show dramatic runtime reduction on 200-example evaluations
Improvements
Platform Security & Reliability
- Span Size Limits: Enforced 5 MB cap on span fields prevents memory/CPU issues and catches logging misuse before production
- Email Verification: Added account verification for enhanced authentication security
User Experience
- Improved Onboarding: Streamlined flow guides new users through organization setup and Judgeval integration
- Invite Management: Users can now view and revoke pending organization invites from Members settings
Documentation Updates
- New RBAC roles and permissions documentation
- Customer usage tracking guides
- Enhanced metadata and reward modeling tutorials
- Evaluation batching performance guides