CAMEOContinuous Automated Model EvaluatiOn

Runs N' Poses

runs-n-poses, version 2025.01

The original Runs N' Poses release, Zenodo v6 (https://zenodo.org/records/18366081): the 2579 systems its annotations list, proper ligands only.

Download the data for this version.

Scores and metrics

Each uploaded model gets scores: one value per instance of its target, for example per ligand. The leaderboard ranks runs by metrics, computed from the scores of all their models in a subset. See Help for how each one is defined.

Scores

Metrics

Subsets and leaderboards

A subset is a selection of the targets with a training cutoff date. Each subset has its own leaderboard, which shows the runs whose own training cutoff is the same or earlier.

SubsetTraining cutoffTargets
2021-09-302021-09-302579

Differences from the published Runs N' Poses numbers

The reference runs of this version (AF3, AF3 (no templates), Protenix, Chai-1, Boltz-1 and Boltz-1x) are the co-folding predictions published with the paper on Zenodo (version 6), uploaded and scored like any other run. They were made from Zenodo's inputs, without the modification lists of CAMEO's inputs. Their numbers here are not the paper's (Škrinjar et al., Nat Struct Mol Biol 2026; preprint v3), because CAMEO computes them differently:

Uploading predictions

Predictions are uploaded through the API, one model per request. The examples below create a run, upload one model for the target 5s9l__1__1.A__1.H_1.I, and wait until it is scored. Replace YOUR_API_TOKEN with the token on your Account page (sign in first), MODEL_FILES with the files of your model (one CIF or legacy PDB file with the protein chains plus one SDF file per ligand (optional .gz). Ligands are read from the SDF files only; a CIF with non-polymers (ions included), such as a raw co-folding output, is not accepted; at most 5 models per target, and at most 200000 uploads per day or 64 GiB per day for each user),my-method with a label for your run, and 2021-09-30 with your method's training cutoff: the run appears on every leaderboard whose cutoff is the same or later.

import time
from pathlib import Path

import requests

API = "https://beta.cameo3d.org/api/v1"
HEADERS = {"Authorization": "Token YOUR_API_TOKEN"}  # from your Account page


def ok(response):
    """The JSON body, or stop with the API's error message."""
    if not response.ok:
        raise SystemExit(f"{response.status_code} {response.url}: {response.text}")
    return response.json()


# 1. The benchmark, and the numeric id of the target to predict
benchmark = ok(requests.get(f"{API}/benchmarks/runs-n-poses/2025.01/"))
target_id = next(t["id"] for t in benchmark["targets"]
                 if t["local_id"] == "5s9l__1__1.A__1.H_1.I")

# 2. Create a run (private until CAMEO publishes it); labels are unique
#    among your runs of a benchmark. training_cutoff is the latest release
#    date of your method's training data; it cannot be changed later.
run = ok(requests.post(f"{API}/runs/", headers=HEADERS,
                       json={"label": "my-method",
                             "benchmark_id": benchmark["id"],
                             "training_cutoff": "2021-09-30"}))

# 3. Upload one model for that target: its structure file, plus one SDF
#    file per ligand if the ligands are not in the structure file
MODEL_FILES = ["model.cif"]
prediction = ok(requests.post(
    f"{API}/runs/{run['id']}/predictions/", headers=HEADERS,
    data={"target": target_id, "model_number": 1},
    files=[("files", (Path(f).name, Path(f).read_bytes())) for f in MODEL_FILES]))

# 4. Poll the prediction until it reaches a final status
FINAL = {"invalid", "succeeded", "partial", "failed"}
while prediction["status"] not in FINAL:
    time.sleep(30)
    prediction = ok(requests.get(f"{API}/predictions/{prediction['id']}/",
                                 headers=HEADERS))
    print(prediction["status"])
print(prediction["error"] or prediction["scores"])