Judgment Labs Logo
DocumentationSDK ReferenceInteractive DemoChangelog

Getting Started with Self-Hosting

Self-hosting Judgment Labs' platform is a great way to have full control over your LLM evaluation infrastructure. Instead of using our hosted platform, you can deploy your own instance of Judgment Labs' platform.

Part 1: Infrastructure Skeleton Setup

Please have the following infrastructure set up:

  1. A new/empty AWS account that you have admin access to: this will be used to host the self-hosted Judgment instance. Please write down the account ID.
  2. A Supabase organization that you have admin access to: this will be used to store and retrieve data for the self-hosted Judgment instance.
  3. An available email address and the corresponding app password (see Tip below) for the email address (e.g. no-reply@organization.com). This email address will be used to send email invitations to users on the self-hosted instance.
    Your app password is not your normal email password; learn about app passwords for Gmail, Outlook, Yahoo, Zoho, or Fastmail
Make sure to keep your AWS account ID and Supabase organization details secure and easily accessible, as you'll need them for the setup process.

Part 2: Request Self-Hosting Access from Judgment Labs

Please contact us at support@judgmentlabs.ai with the following information:

  • The name of your organization
  • An image of your organization's logo
  • [Optional] A subtitle for your organization
  • Domain name for your self-hosted instance (e.g. api.organization.com) (can be any domain/subdomain name you own; this domain will be linked to your self-hosted instance as part of the setup process)
  • The AWS account ID from Part 1
  • Purpose of self-hosting
The domain name you provide must be one that you own and have control over, as you'll need to add DNS records during the setup process.

We will review your email request ASAP. Once approved, we will do the following:

  1. Whitelist your AWS account ID to allow access to our Judgment ECR images.
  2. Email you back with a backend Osiris API key that will be input as part of the setup process using the Judgment CLI (Part 3).

Part 3: Install Judgment CLI

Make sure you have Python installed on your system before proceeding with the installation.

To install the Judgment CLI, follow these steps:

Clone the repository

git clone https://github.com/JudgmentLabs/judgment-cli.git
cd judgment-cli

Set up a fresh Python virtual environment

Choose one of the following methods to set up your virtual environment:

python -m venv venv
source venv/bin/activate  # On Windows, use: venv\Scripts\activate
pipenv shell
uv venv
source .venv/bin/activate  # On Windows, use: .venv\Scripts\activate

Install the package

pip install -e .
pipenv install -e .
uv pip install -e .

Verifying the Installation

To verify that the CLI was installed correctly, run:

judgment --help

You should see a list of available commands and their descriptions.

Available Commands

The Judgment CLI provides the following commands:

Self-Hosting Commands

CommandDescription
judgment self-host mainDeploy a self-hosted instance of Judgment (and optionally set up the HTTPS listener)
judgment self-host https-listenerSet up the HTTPS listener for a self-hosted Judgment instance

Part 4: Set Up Prerequisites

AWS CLI Setup

You'll need to install and configure AWS CLI with the AWS account from Part 1.

brew install awscli
Download and run the installer from https://awscli.amazonaws.com/AWSCLIV2.msi
sudo apt install awscli

After installation, configure your local environment with the relevant AWS credentials:

aws configure

Terraform CLI Setup

Terraform CLI is required for deploying the AWS infrastructure.

brew tap hashicorp/tap
brew install hashicorp/tap/terraform
choco install terraform
Follow instructions https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli

Part 5: Deploy Your Self-Hosted Environment

During the setup process, .tfstate files will be generated by Terraform. These files keep track of the state of the infrastructure deployed by Terraform. DO NOT DELETE THESE FILES.

Create a credentials file (e.g., creds.json) with the following format:

creds.json
{
    "supabase_token": "your_supabase_personal_access_token_here",
    "org_id": "your_supabase_organization_id_here",
    "db_password": "your_desired_supabase_database_password_here",
    "invitation_sender_email": "email_address_to_send_org_invitations_from",
    "invitation_sender_app_password": "app_password_for_invitation_sender_email",
    "osiris_api_key": "your_osiris_api_key_here (optional)",
    "openai_api_key": "your_openai_api_key_here (optional)",
    "togetherai_api_key": "your_togetherai_api_key_here (optional)",
    "anthropic_api_key": "your_anthropic_api_key_here (optional)"
}

Run the main self-host command. The command syntax is:

judgment self-host main [OPTIONS]

Required options:

  • --root-judgment-email or -e: Email address for the root Judgment user
  • --root-judgment-password or -p: Password for the root Judgment user
  • --domain-name or -d: Domain name to request SSL certificate for (make sure you own this domain)

Optional options:

For --supabase-compute-size, only "nano" is available on the free tier of Supabase. If you want to use a larger size, you will need to upgrade your organization to a paid plan.

  • --creds-file or -c: Path to credentials file (default: creds.json)
  • --supabase-compute-size or -s: Size of the Supabase compute instance (default: small)
    • Available sizes: nano, micro, small, medium, large, xlarge, 2xlarge, 4xlarge, 8xlarge, 12xlarge, 16xlarge
  • --invitation-email-service or -i: Email service for sending organization invitations (default: gmail)
    • Available services: gmail, outlook, yahoo, zoho, fastmail

Example usage:

judgment self-host main \
--root-judgment-email root@example.com \
--root-judgment-password password \
--domain-name api.example.com \
--creds-file creds.json \
--supabase-compute-size nano \
--invitation-email-service gmail

This command will:

  1. Create a new Supabase project
  2. Create a root Judgment user in the self-hosted environment with the email and password provided
  3. Deploy the Judgment AWS infrastructure using Terraform
  4. Configure the AWS infrastructure to communicate with the new Supabase database
  5. * Request an SSL certificate from AWS Certificate Manager for the domain name provided
  6. ** Optionally wait for the certificate to be issued and set up the HTTPS listener
*For the certificate to be issued, this command will return two DNS records that must be manually added to your DNS registrar/service.
**You will be prompted to either continue with the HTTPS listener setup now or to come back later. If you choose to proceed with the setup now, the program will wait for the certificate to be issued before continuing.

Setting up the HTTPS listener

This step is optional; you can choose to have the HTTPS listener setup done as part of the main self-host command.

This command will only work after judgment self-host main has already been run AND the ACM certificate has been issued. To accomplish this:

  1. Add the two DNS records returned by the main self-host command to your DNS registrar/service
  2. Monitor the ACM console here until the certificate has status 'Issued'

To set up the HTTPS listener, run:

judgment self-host https-listener

This command will:

  1. Set up the HTTPS listener with the certificate issued by AWS Certificate Manager
  2. Return the url to the HTTPS-enabled domain which now points to your self-hosted Judgment server

Part 6: Accessing Your Self-Hosted Environment

Your self-hosted Judgment API URL (referenced as self_hosted_judgment_api_url in this section) should be in the format https://{self_hosted_judgment_domain} (e.g. https://api.organization.com).

From the Judgeval SDK

You can access your self-hosted instance by setting the following environment variables:

JUDGMENT_API_URL = "self_hosted_judgment_api_url"
JUDGMENT_API_KEY = "your_api_key"
JUDGMENT_ORG_ID = "your_org_id"

Afterwards, Judgeval can be used as you normally would.

From the Judgment platform website

Visit the url https://app.judgmentlabs.ai/login?api_url={self_hosted_judgment_api_url} to login to your self-hosted instance. Your self-hosted Judgment API URL will be whitelisted when we review your request from Part 2.

You should be able to log in with the root user you configured during the setup process (--root-judgment-email and --root-judgment-password from the self-host main command).

Adding more users to the self-hosted instance

For security reasons, users cannot register themselves on the self-hosted instance. Instead, you can add new users via email invitations to organizations.

To add a new user, make sure you're currently in the workspace/organization you want to add the new user to. Then, visit the workspace member settings and click the "Invite User" button. This process will send an email invitation to the new user to join the organization.

Alerts

Set up automated alerts and actions based on evaluation metrics.

Security & Compliance

Next Page

On this page

Part 1: Infrastructure Skeleton SetupPart 2: Request Self-Hosting Access from Judgment LabsPart 3: Install Judgment CLIClone the repositoryNavigate to the project directorySet up a fresh Python virtual environmentInstall the packageVerifying the InstallationAvailable CommandsSelf-Hosting CommandsPart 4: Set Up PrerequisitesAWS CLI SetupTerraform CLI SetupPart 5: Deploy Your Self-Hosted EnvironmentSetting up the HTTPS listenerPart 6: Accessing Your Self-Hosted EnvironmentFrom the Judgeval SDKFrom the Judgment platform websiteAdding more users to the self-hosted instance