CAMEO hosts curated benchmarks for structure prediction methods. A benchmark is a fixed set of targets with known reference structures, released as a version (for example 2026.01). Its targets are grouped into one or more subsets, each with a training cutoff date. Each subset has its own leaderboard, which shows the runs whose own training cutoff is the same or earlier: a run trained up to 2021-09-30 appears on the leaderboards with cutoff 2021-09-30 and 2025-06-30, a run trained up to 2025-06-30 only on the latter.
Participants download the modeling inputs of a benchmark version, predict the structures with their method on their own hardware, and upload the models to CAMEO through the API. CAMEO validates and scores every uploaded model against the reference structures and shows the results on the leaderboards.
Uploads are grouped in runs. A run is one set of predictions for one benchmark version, typically one method or one configuration of a method, with a label that you choose and thetraining cutoff of the method: the latest release date of its training data. The cutoff is set when the run is created and cannot be changed; to correct it, create a new run. You can have several runs per benchmark. A new run is private: on the leaderboard only you see it, and through the API it is visible to you and to CAMEO administrators. CAMEO makes a run public on request; it then appears on the leaderboard for everyone. The leaderboard never shows the owner of a run, only its label.
Authorization: Token <token>. Keep it secret; if it leaks, regenerate it on the Account page, which stops the old token from working.curl.The API is a REST API with JSON answers. Benchmarks, subsets and leaderboards are public; runs and predictions need your token. The main calls:
GET /benchmarks/ and GET /benchmarks/{name}/{version}/: the benchmarks, and one version with its scores, metrics, subsets, targets and upload rules. Uploads refer to a target by its numeric id; its local_id is the identifier used in the benchmark's own data.GET /benchmarks/{name}/{version}/subsets/{subset}/leaderboard/: a leaderboard, ranked by the metric given in ?sort=, or by coverage with ?sort=coverage. With your token, it also lists your private runs. Every bin of the subset is in the answer; ?bin= ranks the runs by one bin instead of the whole subset.POST /runs/ with label, benchmark_id and training_cutoff(YYYY-MM-DD): create a run. Labels are unique among your runs of one benchmark version.POST /runs/{id}/predictions/, a multipart upload with target,model_number (1 if omitted) and the model file(s) as files: upload one model.GET /runs/, GET /runs/{id}/predictions/ and GET /predictions/{id}/: your runs and predictions, with their status, errors and scores.POST /predictions/{id}/resubmit/: replace the files of a prediction that has reached a final status. Uploading the same target and model number again is refused; use resubmit instead. Once CAMEO has published a run, its predictions no longer change: uploads and resubmits answer 409. To submit a changed method, create a new run.The complete reference, generated from the API itself, is at /api/docs/ on the API's host (for https://cameo3d.org/api/v1, that is https://cameo3d.org/api/docs/).
One request uploads one model for one target. A benchmark accepts a limited number of models per target (shown on its page); the model number tells them apart. Only model 1 counts on the leaderboard. Each user can also make only a limited number of uploads and resubmits per day, enough for a full submission (also shown on the benchmark page); beyond it the API answers 429 until the count resets.
Each benchmark states its accepted files on its page. For Runs N' Poses, a model is one structure file with the protein chains (mmCIF or legacy PDB), plus one SDF file per ligand. Ligands are read from the SDF files only, never from the structure file: an mmCIF file that contains non-polymer entities, ions included, is rejected.
Raw mmCIF outputs of co-folding tools are therefore not accepted as they are. Put each ligand in an SDF file and remove every non-polymer from the mmCIF, ions included; Converting co-folding outputs below shows how.
Every file may be gzipped (.gz). Further rules:
.pdb or .cif); nothing but the structure file and the.sdf files.inf, no nan).N, CA, C and CB, glycine without CB; nucleotides O5', C5', C4', C3',O3'), as scoring ignores them. In each chain their numbers increase strictly, with no insertion codes, and no chain mixes peptides and nucleotides.U), the model may have U, C or X; where it has pyrrolysine (O), O, K or X; where it hasX, anything. Any other input letter needs the same letter, so an unknown residue (UNK, X) is accepted only at an input X. Chain names are free, and input chains or residues may be left out.The raw mmCIF outputs of co-folding tools (AlphaFold 3, Boltz, Chai-1, ESMFold2, Protenix, …) hold their ligands in the structure file and carry no bonds for them, so CAMEO does not accept them as they are. The Python example below shows how CAMEO converted the Runs N' Poses reference predictions with OpenStructure. It reads the model with OpenStructure's heuristic processor, which infers the bonds from interatomic distances, writes each non-polymer chain to its own SDF file, and writes the polymer chains to receptor.cif. Run it as ost split_model.py model.cif out_dir, then upload the files in out_dir.
import sys
from pathlib import Path
from ost import conop, io, mol
def load_with_heuristic_bonds(model_path):
"""Read the model with OpenStructure's heuristic processor.
The models carry no connectivity, so the bonds are guessed from
interatomic distances; the ligand SDF files get these bonds.
"""
profile = io.profiles.Get("DEFAULT").Copy()
profile.processor = conop.HeuristicProcessor()
entity, _info = io.LoadMMCIF(str(model_path), info=True, profile=profile)
return entity
model_path, out_dir = Path(sys.argv[1]), Path(sys.argv[2])
entity = load_with_heuristic_bonds(model_path)
# AF3, Boltz, Chai-1, ESMFold2: the _entity loop marks the ligands non-polymer.
ligands = [c.name for c in entity.chains if c.type == mol.CHAINTYPE_NON_POLY]
receptor = entity.Select("cname=" + ",".join(
mol.QueryQuoteName(c.name) for c in entity.chains if c.name not in ligands))
out_dir.mkdir(parents=True)
io.SaveMMCIF(receptor, str(out_dir / "receptor.cif"))
for name in ligands:
ligand = entity.Select("cname=" + mol.QueryQuoteName(name))
assert ligand.residue_count == 1, f"chain {name}"
io.SaveSDF(ligand, str(out_dir / f"{name}.sdf"))The example finds the ligands by their chain type, which OpenStructure reads from the _entity.type field of the mmCIF file. A model that gives no _entity.type for its ligands, such as a Protenix model, loads them as mol.CHAINTYPE_UNKNOWN instead; for such a model, select the chains of that type.
Bond orders are not used for scoring: the scorer matches the model ligands to the target ligands on a graph of elements and bonds, without bond orders. Connectivity inferred from distances can be wrong where the tool placed atoms too close together or too far apart. Such a ligand cannot be matched to the target ligand and counts as not predicted. On the Runs N' Poses reference predictions, this happened to between 0.7 % and 4 % of the ligands, depending on the tool. If your tool can write SDF files with its own bonds, prefer those.
An upload is answered right away with status received. CAMEO then validates and scores it in the background, which can take a few minutes. The status moves through queued, validating and running, and ends in one of:
succeeded: every instance of the target (for example every ligand) was scored;partial: some instances were scored, others could not be matched to the model;invalid: the upload did not follow the rules above; the error field says why. Fix the files and resubmit;failed: CAMEO could not score the model. The error field gives the reason as far as CAMEO can tell. Some causes lie in the model, for example when none of its ligands matches a reference ligand; fix the model and resubmit. Others are problems on CAMEO's side. If the error does not clearly point to your model, or resubmitting does not help, contact us.A prediction can also stop in limbo: CAMEO lost track of its job, and an administrator has been notified. You can resubmit it.
CAMEO computes scores for every uploaded model: one value per instance of the target, for example one per ligand. The leaderboard ranks runs by metrics, which pool the scores of the runs' models in a subset. Each benchmark page lists the scores and metrics that benchmark uses. All scores are computed with OpenStructure, as described in Studer et al. (2025), with alignments based on residue numbers.
Model ligands are assigned to the reference ligands automatically, by LDDT-PLI; the ligand RMSD is computed for the same pair. Hydrogens are ignored. A model ligand matches a reference ligand only if they have the same heavy atoms, bonded the same way. A reference ligand that no model ligand matches gets no score.
With the ground truth from the benchmark's downloads and OpenStructure installed, this command scores a model with the settings CAMEO uses:
ost compare-ligand-structures \
-m model/receptor.cif -ml model/*.sdf \
-r ground_truth/<system>/receptor.cif -rl ground_truth/<system>/ligand_files/*.sdf \
--lddt-pli --rmsd -rnaMetrics use the scores of model 1 of each target in the subset.
A subset can be split into bins, shown as tabs above its leaderboard. In Runs N' Poses a bin is a range of a ligand's similarity to the training data before the subset's cutoff, binned as the authors of each version did. A bin's tab computes the same metrics over the bin's ligands only, and its coverage over the targets with a ligand in the bin. Ligands without a similarity value are in no bin, but count in the whole subset. Above the table, a bar plot shows the ranked column for every bin, one bar per ticked run; reference runs, the top public run and your three newest runs are ticked by default.
CAMEO reads mmCIF files with OpenStructure. In addition to the atom_site category, the reader uses entity information from the entity, entity_poly, and entity_poly_seq categories to identify polymer and non-polymer chains and their sequences.
We strongly recommend validating your mmCIF files before uploading them. Below are several ways to validate your files.
If you have access to Docker, we provide a convenient validation script as a container. The mmCIF file to validate must be in the current working directory or one of its subdirectories.
docker run --rm -v "$(pwd)":/tmp --user $(id -u):$(id -g) registry.scicore.unibas.ch/schwede/modelcif-converters/mmcif-dict-suite:latest validate-mmcif-file -a . -r --dict-sdb /usr/local/share/mmcif-dict-suite/mmcif_pdbx_v50.sdb [FILE.cif]
To update to the latest version of the container image, run:
docker pull registry.scicore.unibas.ch/schwede/modelcif-converters/mmcif-dict-suite:latest
The script is based on the RCSB mmCIF Dictionary Suite (see below) and includes a report mode which is easier to read.
PDBe provides an easy-to-install mmCIF Validator extension for VS Code.
Once installed, the validator can also be used from the command line.
The RCSB provides an mmCIF Dictionary Suite that can be run from the command line and requires minimal dependencies to install. It is the reference implementation for validating mmCIF files as it comes from the same source as the mmCIF format.