Self-Hosting

Make sure the Judgment CLI is installed before proceeding. Please refer to Installation for more information.

During the setup process, .tfstate files will be generated by Terraform. These files keep track of the state of the infrastructure deployed by Terraform. DO NOT DELETE THESE FILES.

Introduction

The self-host command is used to deploy and manage your own self-hosted instance of Judgment.

self-host main is used to deploy a Supabase project along with the Judgment AWS infrastructure.

self-host https-listener is used to configure HTTPS for the Judgment instance, and is only required to run if you don't proceed with the listener setup as part of the self-host main command.

Usage

To see usage information, run any of the following:

judgment self-host --help
judgment self-host main --help
judgment self-host https-listener --help

1. Prerequisites

Before proceeding, ensure you have:

An empty AWS account that has been registered with us for self-hosting (for hosting the self-hosted Judgment instance)
An Osiris API key from Judgment Labs (if you plan to use Osiris for evaluations)
An available email address and the corresponding app password (learn about app passwords for Gmail, Outlook, Yahoo, Zoho, or Fastmail) for the email address (for sending email invitations to users on the self-hosted instance)
A Supabase account and organization that you have admin access to (for data storage)

Refer to the Getting Started with Self-Hosting guide for more information about the prerequisites. Make sure you have followed all steps in Part 1 and 2 of the guide before proceeding.

AWS CLI Setup

You'll need to install and configure AWS CLI with the AWS account mentioned in the above warning.

brew install awscli

Download and run the installer from https://awscli.amazonaws.com/AWSCLIV2.msi

sudo apt install awscli

After installation, configure your local environment with the relevant AWS credentials:

aws configure

Terraform CLI Setup

Terraform CLI is required for deploying the AWS infrastructure.

brew tap hashicorp/tap
brew install hashicorp/tap/terraform

choco install terraform

Follow instructions https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli

2. Deploying

To deploy a self-hosted instance of Judgment:

Create a credentials file (e.g., creds.json) with the following format:

creds.json

{
    "supabase_token": "your_supabase_personal_access_token_here",
    "org_id": "your_supabase_organization_id_here",
    "db_password": "your_desired_supabase_database_password_here",
    "invitation_sender_email": "email_address_to_send_org_invitations_from",
    "invitation_sender_app_password": "app_password_for_invitation_sender_email",
    "osiris_api_key": "your_osiris_api_key_here (optional)",
    "openai_api_key": "your_openai_api_key_here (optional)",
    "togetherai_api_key": "your_togetherai_api_key_here (optional)",
    "anthropic_api_key": "your_anthropic_api_key_here (optional)"
}

Run the main self-host command. The command syntax is:

judgment self-host main [OPTIONS]

Required options:

--root-judgment-email or -e: Email address for the root Judgment user
--root-judgment-password or -p: Password for the root Judgment user
--domain-name or -d: Domain name to request SSL certificate for (make sure you own this domain)

Optional options:

For --supabase-compute-size, only "nano" is available on the free tier of Supabase. If you want to use a larger size, you will need to upgrade your organization to a paid plan.

--creds-file or -c: Path to credentials file (default: creds.json)
--supabase-compute-size or -s: Size of the Supabase compute instance (default: small)
- Available sizes: nano, micro, small, medium, large, xlarge, 2xlarge, 4xlarge, 8xlarge, 12xlarge, 16xlarge
--invitation-email-service or -i: Email service for sending organization invitations (default: gmail)
- Available services: gmail, outlook, yahoo, zoho, fastmail

Example usage:

judgment self-host main \
--root-judgment-email root@example.com \
--root-judgment-password password \
--domain-name api.example.com \
--creds-file creds.json \
--supabase-compute-size nano \
--invitation-email-service gmail

This command will:

Create a new Supabase project
Create a root Judgment user in the self-hosted environment with the email and password provided
Deploy the Judgment AWS infrastructure using Terraform
Configure the AWS infrastructure to communicate with the new Supabase database
* Request an SSL certificate from AWS Certificate Manager for the domain name provided
** Optionally wait for the certificate to be issued and set up the HTTPS listener

*For the certificate to be issued, this command will return two DNS records that must be manually added to your DNS registrar/service.

**You will be prompted to either continue with the HTTPS listener setup now or to come back later. If you choose to proceed with the setup now, the program will wait for the certificate to be issued before continuing.

Setting up the HTTPS listener

This step is optional; you can choose to have the HTTPS listener setup done as part of the main self-host command.

This command will only work after judgment self-host main has already been run AND the ACM certificate has been issued. To accomplish this:

Add the two DNS records returned by the main self-host command to your DNS registrar/service
Monitor the ACM console here until the certificate has status 'Issued'

To set up the HTTPS listener, run:

judgment self-host https-listener

This command will:

Set up the HTTPS listener with the certificate issued by AWS Certificate Manager
Return the url to the HTTPS-enabled domain which now points to your self-hosted Judgment server