CLI Reference

🔗Reference — the exp CLI

Hand-written from src/cli/mod.rs, updated 2026-07-28.

Binary name is exp (Cargo.toml default-run), though clap's name is set to "expman" (cli/mod.rs:71) so --help output says expman. Requires the cli feature. Entry: src/main.rs:6-8.

Logging: init_tracing (cli/mod.rs:15-23) reads RUST_LOG via EnvFilter, defaults to info, compact format without targets.

Errors surface as Error: {e} + exit(1). The CLI uses anyhow, not ExpmanError.

🔗exp serve [DIR]

Gated on the server feature. Starts the dashboard.

--read-only refuses every non-GET/HEAD/OPTIONS request with 403 {"error": "read_only"}. It is enforced as middleware over the whole router (api/mod.rs, enforce_read_only) rather than per handler, so routes added later are covered by default — including the Jupyter and TensorBoard start/stop POSTs, which spawn processes on the host. Use it to share a store with someone who should not be able to change it.

live_mode and read_only are stored in AppState and reported truthfully by /api/config. The two Jupyter flags are stored on the JupyterManager and are not reported by /api/config — nothing in the frontend needs them.

🔗--notebook-template — the Interactive tab's cells

Added 1.3.0. The notebook a run's Interactive tab opens comes from a project-supplied .ipynb, so the cells can do project-specific work — load this run's checkpoint, render a rollout the training host could not — instead of the built-in "read vectors.parquet and plot it".

Discovery order (jupyter_service.rs, NotebookTemplateConfig::resolve):

1. --notebook-template <PATH>
2. <DIR>/.expman/notebook.ipynb — the convention, so a store can carry its own notebook and the flag can stay unset
3. the built-in default

A --notebook-template that does not exist is warned about and skipped, falling through to 2 and 3; a typo in the flag does not stop the server.

The template is itself a .ipynb — valid JSON — and expman substitutes these placeholders anywhere in the file, cells or metadata alike:

Paths are absolute because a relative one would only resolve against the server's cwd, which the notebook has no reason to share.

Values are JSON-escaped before substitution. Placeholders sit inside JSON string literals, so a run directory containing " or \ would otherwise close the literal and corrupt the notebook. This is tested, not assumed (test_placeholder_values_are_json_escaped).

If the substituted template does not parse as JSON, expman logs an error naming the template path and writes the built-in default instead — a broken .ipynb would break the tab the template exists to improve.

The multi-run notebook is deliberately not templatable. POST /experiments/{exp}/jupyter/start writes an interactive.ipynb into the experiment directory, where {{run_dir}}, {{run_name}} and {{project}} have no value to take; a useful shared template would need its own placeholder set (a {{runs}} list plus a rule for rendering it), which is a separate design rather than a wider version of this one. It always gets the built-in default, and always will until that design exists.

🔗Notebook staleness — regenerated unless you edited it

Added 1.3.0. Before then, an existing interactive.ipynb was kept forever, so a run generated under an old template never picked up a fix.

Every notebook expman writes carries a namespaced metadata.expman block:

"metadata": {
  "expman": {
    "generated_by": "expman 1.3.0",
    "template": "/abs/path/.expman/notebook.ipynb",
    "template_hash": "sha256:…",
    "content_hash": "sha256:…"
  }
}

On each launch (POST …/jupyter/start, or the notebook POST):

content_hash covers the parsed JSON with content_hash itself removed (a field cannot hash itself), so a whitespace reformat still counts as unedited. Executing cells does not — outputs and execution_count are your work, and expman will not overwrite them. Delete the file to opt back into regeneration.

The rule applies to the multi-run notebook too, whose "template" is the built-in content — which names the runs, so changing the selected run set refreshes an unedited notebook.

🔗--jupyter-command — which interpreter the kernel is

Added 1.3.0. A command line, not a program name; split POSIX-shell-style (via shlex), so a quoted path with a space survives. No shell runs, so pipes, redirection and $VAR are not available. expman appends notebook --no-browser --port=… … after it:

exp serve ./experiments --jupyter-command 'uv run --extra nb jupyter'
# actually executed:  uv run --extra nb jupyter notebook --no-browser --port=8888 …

Why this rather than a kernelspec. The kernel a notebook gets is the interpreter Jupyter itself runs under. Launch Jupyter from inside the project's virtualenv and its built-in python3 kernel is that venv's interpreter, so import <your package> works with no ipykernel install step and no kernelspec name for expman and the project to keep in sync. expman therefore does not register a global kernel and does not write a named kernelspec into the generated notebook — either would mutate your ~/.local/share/jupyter and then have to stay correct forever.

Two failure modes, both with messages that name the configured value:

• unparsable (unbalanced quote) or empty → exp serve refuses to start, rather than failing at the first click on the Interactive tab
• the program does not exist → the start request fails with Failed to start Jupyter with the configured command `…`: No such file or directory, not a bare ENOENT

The trap. expman runs the command with the run directory as cwd, because that is where Jupyter must serve interactive.ipynb from. A command that discovers its environment from cwd — uv run walking up for a pyproject.toml — therefore only works when the store lives inside the project. When it does not, name the project explicitly instead of changing directory:

--jupyter-command 'uv run --project /path/to/project --extra nb jupyter'

🔗exp list [DIR]

Two views. Experiment view (the default) is a table of Experiment | Project | Runs | Display Name, narrowed by --project. Run view is a flat cross-experiment table of Run | Experiment | Project | Status | Started | Duration | Tags; a None duration prints as running.

Run view is selected by --runs, by --experiment, or by any run-level filter (--tag, --status, --group) — asking a question about runs gives you runs. A Group column appears only when something is actually grouped, shown as group[rank].

--tag takes an expression: clauses are ANDed, alternatives within a clause are ORed.

exp list experiments --tag "arm:tiered AND (study:1 OR study:2)"
exp list experiments --tag "arm:tiered,study:1"   # , is AND, | is OR

Operators are matched case-insensitively but only as bare whitespace-delimited words, so a tag like brand:ORACLE is not torn in half.

🔗exp project <SUBCOMMAND>

The projects layer without a server — a cluster node or tmux session has no dashboard, which is what made this layer unreachable from code before.

All take --dir (default ./experiments), except ls which takes DIR positionally.

🔗exp project sync

One-way. The manifest is authoritative; each sync overwrites the project's metadata, README, and experiment membership, and marks it generated: true. The dashboard then refuses edits to it with 409 generated_project and hides the edit affordances — accepting a write it is about to destroy would be worse than refusing it.

projects:
  - name: study-1
    display_name: "Study 1 — Drift regret"
    description: "Does tiered allocation reduce drift regret?"
    tags: [thesis, study1]
    generated_from: "studies.yaml (thesis repo)"
    experiments: [e1-drift-regret-slope, x2-shift-sweep]
    frontpage:
      question: "Does a tiered arm dominate a flat arm under covariate shift?"
      scope: "3 domains, 5 seeds, 4 shift levels"
      domains: [vision, tabular, text]
      target_venue: "NeurIPS"
      target_date: "2026-05-15"
      status: "running"
      sections:
        - heading: "Notes"
          body: "Free-form markdown."

A bare top-level list of specs is accepted too. readme: supplies a raw body instead of rendering frontpage:. Membership is reconciled: experiments listed are assigned, and experiments previously in the project but absent from the manifest are unassigned. Naming an experiment that does not exist yet is reported, not an error — a project can be declared before its first run.

🔗exp sweep <SUBCOMMAND>

Hyperparameter search. A sweep is a group of runs, one per trial — it needs no new storage concept and no server.

name: lr-sweep            # becomes the group every trial belongs to
experiment: e1-drift      # trials are logged under this experiment
project: study-1          # optional; created if it does not exist
method: grid              # or: random
trials: 40                # random only
seed: 0                   # random only; sweeps are reproducible from this
command: "python train.py --lr {lr} --bs {bs}"
params:
  lr:    {values: [0.1, 0.01, 0.001]}
  bs:    {values: [16, 32]}
  wd:    {min: 1.0e-5, max: 1.0e-2, log: true}   # random only
metric:
  name: val_loss
  goal: minimize

Params reach the trial two ways, deliberately: substituted into command (so an existing argparse script needs no edits) and exported as EXPMAN_PARAM_LR etc. (so a shell wrapper or sbatch script can read them without seeing the command string). expman.sweep_params() returns them as a typed dict.

The sweep also exports EXPMAN_GROUP, EXPMAN_RANK, EXPMAN_RUN_NAME, EXPMAN_EXPERIMENT, EXPMAN_PROJECT, EXPMAN_BASE_DIR and EXPMAN_TAGS, all of which the Python Experiment picks up automatically. A training script needs no sweep-specific code at all.

Grid needs values. min/max is a continuous domain and only works with method: random; grid says so rather than silently sampling. log: true samples the exponent, which is what you want for learning rates — a uniform sampler puts ~90% of its mass in the top decade.

Random search is reproducible: the same seed yields the same trials, using an internal SplitMix64 rather than the rand crate so a dependency bump cannot change results.

🔗exp probes

Shows which system-metric probes exist on this machine and takes one live sample. --all lists every probe considered, including unavailable ones.

This is the answer to "why am I not seeing GPU metrics?" — during a run an absent probe is skipped silently, which is right for logging and useless for debugging.

tpu-info has no JSON output. It prints Rich pipe tables, one per --metric. expman parses those directly. hbm_usage and duty_cycle_percent read N/A until a framework has the TPU open — libtpu only publishes them then — and are skipped rather than recorded as zero.

Custom probes need no rebuild; add them to config with format: nvidia_csv, json, pipe_table or key_value.

🔗exp reap [DIR]

Marks stale RUNNING runs as CRASHED. A hard kill leaves a run RUNNING forever, which silently inflates /stats.active_runs.

Staleness comes from heartbeat_at, which the engine refreshes every heartbeat_interval_secs (default 30s). Runs written before heartbeats existed have no heartbeat_at and fall back to started_at — the conservative direction, since such a run is only reaped once it is genuinely old.

finished_at is set to the last heartbeat, not to when you noticed; otherwise a run reaped a week late would report a week-long duration.

Reaping also compacts the run (storage::compact_run), folding its live .arrow segments into Parquet. A hard-killed run never reached the engine's close path, so without this it stayed terminal and slow — every later read re-parses the whole segment heap. Compaction runs before the status is written: this command only selects RUNNING runs, so an interrupted reap leaves the run RUNNING and the next one finishes the job, whereas marking first would strand it terminal and uncompacted forever. Compaction failures are logged, not fatal — the status still gets recorded.

🔗exp inspect <RUN_DIR>

Prints run/experiment/status/started/duration, then config.yaml verbatim, then a Vector | Value table of the last Parquet row plus a total row count, then a Scalar | Value table from run.yaml.scalars, then every artifact as path (N bytes) (cli/mod.rs:245-318).

The help text example (cli/mod.rs:111) shows experiments/my_exp/runs/20240101_120000 — the runs/ segment is spurious and does not match the real layout. See storage-layout.

🔗exp clean [EXPERIMENT]

Keeps the --keep most recent runs per experiment. At least one of EXPERIMENT, --project, --group or --tag is required — with no scope at all it refuses rather than cleaning the whole store.

With --tag, only matching runs are candidates and only they count toward --keep, so --tag arm:tiered --keep 5 keeps five tiered runs per experiment and leaves the others untouched.

Dry-run by default — prints what it would delete and "Dry run. Use --force to actually delete.". With --force, remove_dir_alls each victim. Relies on list_runs being newest-first plus split_off(keep) (cli/mod.rs:320-360).

🔗exp export <RUN_DIR>

Bails if vectors.parquet is absent (cli/mod.rs:380-443).

• json — serde_json::to_string_pretty of the row maps.

• csv — RFC 4180. The header is the union of every row's keys (step and timestamp first, the rest alphabetical), and fields containing a comma, quote, CR or LF are quoted with inner quotes doubled. A missing metric and an explicit null both render as an empty cell, since any placeholder would be indistinguishable from data.

  Both of those were previously wrong, and silently: the header came from rows[0], so a metric first logged mid-run was dropped from the file entirely, and unescaped values let a string containing a comma split into an extra column. Regression tests: test_cli_export_csv_includes_metrics_first_logged_mid_run.

• tensorboard — writes via tensorboard_rs::summary_writer::SummaryWriter. Only as_f64()-able columns are emitted, as add_scalar(k, v as f32, step); step and timestamp are skipped.

🔗exp import <INPUT>

If INPUT is a directory, takes the first entry whose filename contains tfevents. Experiment name is INPUT's file name (fallback imported_tb); run name is a UTC timestamp. Parses with tboard::SummaryReader, folding SimpleValue summaries into a BTreeMap<step, ...> so rows come out step-ordered, then writes via storage::append_vectors.

It also writes a run.yaml with status FINISHED, the tags imported and tensorboard, and the last row's values, plus an experiment.yaml if absent. Without those an imported run had no metadata at all and every reader fell back to minimal_run_metadata, so the import showed up in the dashboard as CRASHED.

🔗Via Python

pip install expman-rs installs an exp console script (pyproject.toml:24-25) pointing at expman.cli:main — a 31-line shim that execs the binary bundled at <site-packages>/expman/bin/exp and forwards the exit code (wrappers/python/expman/cli.py).

Nix caveat: nix build .#python3Packages.expman-rs has no step that populates expman/bin/, so that console script will report "Bundled binary not found" and return 1. Nix users should get the CLI from .#expman.

HTTP API Reference

🔗Reference — HTTP API

Hand-written from src/api/*.rs, 2026-07-27. Authoritative over src/api/README.md, which is stale (it documents a nonexistent /api/events endpoint and mentions utoipa).

Router: src/api/mod.rs:42-140. All routes below are prefixed with /api (mod.rs:153). Anything else falls through to the embedded frontend (mod.rs:155). CORS is fully permissive — Any origin/method/header (mod.rs:146-149).

Rows marked NEW are from the uncommitted TensorBoard work.

🔗Projects

🔗Experiments

🔗Runs

🔗Metrics and streams

Both streams terminate via .take_until(shutdown.cancelled()).

Note the vectors-SSE route is the one route that breaks the naming convention — /run/{exp}/{run}/... rather than /experiments/... (mod.rs:56). It is also currently unused by the frontend.

🔗Artifacts

🔗Global

🔗Jupyter

Per-run, plus a __multi__ variant scoped to the experiment.

Both start routes write the notebook first, and since 1.3.0 that write may replace an existing interactive.ipynb — when expman wrote it, you have not edited it, and the template has changed since. An edited notebook is never overwritten. Full rules, and the --notebook-template placeholder list: CLI reference.

GET /jupyter/available reports on the configured Jupyter command (exp serve --jupyter-command, default jupyter), not a hardcoded jupyter — so a project that reaches Jupyter through uv run is detected rather than falling back to the ipython/python copy-paste view.

409 from the notebook POSTs means "the file on disk stands" — whether because it is already current or because it was edited and left alone. The two are the same answer to the caller; the distinction is in the server log (warn for an edited file, info for a regeneration).

🔗TensorBoard — NEW / uncommitted

No multi-run variant exists, unlike Jupyter — see STATUS.md.

🔗Frontend fallback

GET /* → frontend::serve_frontend (frontend.rs:57). Serves a rust-embed asset if the path matches, else falls back to index.html for anything that looks like an SPA route, else 404.

The route/asset split is is_asset_request (frontend.rs:45): only the last path segment is examined, and only a suffix in ASSET_EXTENSIONS (:36) counts as an asset. A dot earlier in the path is part of a name, not an extension — so /experiments/dmc-cartpole.swingup and /experiments/dmc-cheetah.run/runs/latest are routes. html is excluded from the list on purpose. See architecture for why the list is wider than what the bundle embeds.

🔗Notes

• No WebSocket endpoints exist, despite axum's ws feature being enabled.
• No caching. Every request re-reads the filesystem and re-parses Parquet.
• Routes the frontend never calls: /run/{exp}/{run}/stream/vectors, /experiments/{exp}/runs/{run}/config, /experiments/{exp}/stats, /config.
• Test coverage: tests/api_test.rs covers 7 routes in-process via tower::ServiceExt::oneshot. The Jupyter and TensorBoard routes are untested.

🔗GET /projects/{project}/runs

GET /projects/{project} returns an experiment list only, which is not a view you can work in: with one project per study, the comparison worth living in cuts across experiments. This returns the flat runs table plus what a filter UI needs.

Query parameters (all optional):

{
  "project": "study-1",
  "total": 2,
  "runs": [{ "run": "tiered-s2", "experiment": "...", "project": "study-1",
             "status": "FINISHED", "started_at": "...", "heartbeat_at": null,
             "tags": ["arm:tiered"], "scalars": {}, "vectors": {},
             "path": "experiments/..." }],
  "facets": {
    "tags":        { "arm:tiered": 2, "seed:1": 1, "study:1": 2 },
    "status":      { "FINISHED": 2 },
    "experiments": { "e1-drift-regret-slope": 2 },
    "groups":      { "lr-sweep": 6, "job-77001": 4 }
  },
  "metrics": ["regret"]
}

This is core::dto::ProjectRuns, so the frontend deserializes the same type the handler serializes. It backs the dashboard's Compare tab.

Facets are counted over the returned runs, so they narrow as filters are applied — the behaviour a filter UI needs to avoid offering options that lead to empty results. metrics is the union of metric names across those runs, so a caller knows what is comparable before fetching any series.

🔗Read-only mode

exp serve --read-only rejects every request that is not GET/HEAD/OPTIONS with 403 {"error": "read_only", "message": ...}. It is middleware over the whole router (enforce_read_only in api/mod.rs), not a per-handler check, so new routes are covered by default. /config reports read_only so the frontend can hide controls rather than offering writes that will fail.

🔗Generated projects

A project created by exp project sync carries generated: true. Writes to it — PATCH /projects/{p} and PUT /projects/{p}/readme — are refused with 409:

{
  "error": "generated_project",
  "message": "Project 'study-1' is generated from studies.yaml (thesis repo) and is regenerated on each sync. Edit the source and re-run `exp project sync` instead.",
  "generated_from": "studies.yaml (thesis repo)"
}

Accepting the write would be worse than refusing it: the dashboard would report success and the edit would vanish at the next sync with no trace. DELETE is refused on the same grounds.

🔗Downsampling

GET /experiments/{exp}/runs/{run}/metrics returns at most 2000 points.

The reduction is Largest-Triangle-Three-Buckets, not a stride. A stride drops the single-row loss spike or divergence, which is exactly what someone opens a chart to find; LTTB keeps whichever point in each bucket contributes the most visible area, so extremes survive. First and last rows are always kept, so endpoints are exact.

Without this, a million-step run serialises to hundreds of MB of JSON and hangs the tab before it draws anything.

Python API Reference

🔗Reference — Python API

Hand-written from wrappers/python/expman/*.py and src/wrappers/python/mod.rs, updated 2026-07-28.

pip install expman-rs → import expman. expman.__version__ comes from CARGO_PKG_VERSION (mod.rs:295), i.e. it tracks Cargo.toml, not pyproject.toml.

The write path, the projects layer, and the read API cross into Rust. Console redirection, tensorboard_dir, the module-level singleton API, the to_pandas convenience, cli.py, and all of tensorboard.py are pure Python.

🔗class Experiment

Experiment(name, run_name=None, base_dir="experiments",
           flush_interval_rows=50, flush_interval_ms=500,
           redirect_console=True,
           project=None, tags=None, description=None,
           heartbeat_interval_secs=30,
           group=None, rank=None,
           system_metrics_interval_secs=15,
           capture_provenance=True, capture_diff=False)

project, tags and description are written at creation, offline — no server required, so this works on a compute node running a SLURM batch job. project lands in experiment.yaml; tags and description land in run.yaml. Before this, consumers had to hand-patch run.yaml after close().

heartbeat_interval_secs controls how often the engine refreshes run.yaml.heartbeat_at, which is what lets exp reap distinguish a hard-killed run from a legitimately long one. Pass 0 to disable.

experiment.yaml is only created when absent, but an explicit project= on a later run still takes effect: the engine updates that one field rather than ignoring it.

group and rank are auto-detected and rarely passed by hand. rank comes from EXPMAN_RANK / RANK / SLURM_PROCID / OMPI_COMM_WORLD_RANK / LOCAL_RANK; group from EXPMAN_GROUP or the scheduler's job id. A DDP script therefore needs no expman-specific changes: each rank becomes its own run (named <group>-rank<N>, so ranks cannot collide), and the dashboard rolls the group into a single expandable row.

system_metrics_interval_secs samples GPU/CPU/RAM into system.parquet via subprocess probes — nvidia-smi, rocm-smi, tpu-info, plus /proc. Probes whose binary is absent are skipped silently, so a laptop logs CPU and memory and says nothing about GPUs. 0 disables sampling.

capture_provenance records git commit/branch/dirty, the command line, hostname and scheduler job ids into provenance.yaml. capture_diff additionally records the working-tree diff; it is off by default because a dirty tree routinely contains an edited .env or a pasted key, and this store may be shared.

redirect_console is Python-only (not a PyO3 parameter). When true, it opens <run_dir>/console.log line-buffered and swaps sys.stdout/sys.stderr for a Tee (__init__.py:34-48) that writes to both. Redirect failure is caught and printed as a warning. atexit.register(self.close) is registered at __init__.py:112.

Status strings are not validated. close() maps anything it does not recognize silently to FINISHED (mod.rs:194). A typo'd status is not an error.

with expman.Experiment("my_exp", base_dir="./experiments") as exp:
    exp.log_params({"lr": 3e-4, "batch_size": 32, "amp": True})
    for step in range(1000):
        exp.log_vector({"train/loss": loss, "train/acc": acc}, step=step)
    exp.log_scalar("best_acc", best)
    exp.save_artifact("model.pt")

# Everything the projects layer needs, from a batch job with no dashboard.
with expman.Experiment(
    "e1-drift-regret-slope",
    base_dir="./experiments",
    project="study-1",
    tags=["arm:tiered", "seed:1", "study:1"],
    description="tiered allocation, seed 1",
) as exp:
    ...

🔗Module-level singleton

A global _current_exp (__init__.py:31). init(...) takes the same arguments as Experiment(...) and closes any existing global experiment first.

expman.init("my_exp")
expman.log_params({...}); expman.log_vector({...}, step=i)
expman.log_scalar(k, v); expman.save_artifact(path)
expman.tensorboard_dir()      # -> str | None   (uncommitted)
expman.info(msg); expman.warn(msg)
expman.close()

With no active experiment, log_*/save_artifact/tensorboard_dir print "Warning: No active experiment. Call expman.init() first."; info/warn silently no-op — an inconsistency, not a design.

🔗Read API

Everything here reads an existing store, and returns plain dicts and lists so an analysis script works in a bare environment. to_pandas=True imports pandas lazily and only then — pandas stays an optional extra, never a runtime dependency of the wheel.

This is what previously did not exist, and its absence is why analysis scripts read Parquet and a side-car JSON manifest directly, leaving the manifest rather than expman as the durable record.

Each load_runs() row carries run, experiment, project, status, started_at, finished_at, heartbeat_at, duration_secs, description, tags, scalars, vectors, and path. Feed that path straight to read_metrics() / load_config() — the two compose, so a caller never does path arithmetic.

load_runs() also takes group= to select one DDP job or sweep cohort.

tags accepts either a list (all must match) or an expression string with AND/OR and parentheses:

import expman

runs = expman.load_runs(tags="arm:tiered AND (study:1 OR study:2)")
for run in runs:
    metrics = expman.read_metrics(run["path"])
    params = expman.load_config(run["path"])
    print(run["run"], params["lr"], metrics[-1]["regret"])

df = expman.read_metrics(runs[0]["path"], to_pandas=True)   # needs pandas

assign_project() is Experiment.set_project() without an open run — for a sync script projecting an external manifest into the store.

read_metrics() reads the Parquet directly and is never downsampled — the cap applies to the HTTP endpoint the browser uses, not to analysis code.

🔗Media and distributions

exp.log_image("samples/epoch", pil_image, step=epoch)   # or bytes, path, ndarray
exp.log_figure("confusion", fig, step=epoch)            # matplotlib
exp.log_histogram("weights/fc1", model.fc1.weight, step=epoch)

Images are written to <run>/media/ with a media.jsonl manifest, not into Parquet — a column of image blobs would make the metrics file useless for its actual purpose. Histograms are stored as one row per (tag, step) with JSON edges/counts, because bin counts vary and a column-per-bin schema would churn on every call.

🔗Sweeps from a training script

import expman

params = expman.sweep_params()      # {"lr": 0.001, "bs": 32} — typed, or {}
with expman.Experiment("e1") as exp:  # group/project/run name come from the env
    exp.log_params(params)
    ...

🔗class SummaryWriter — TensorBoard drop-in

wrappers/python/expman/tensorboard.py, pure Python. Signature-compatible with torch.utils.tensorboard.SummaryWriter; purge_step, max_queue, flush_secs, and filename_suffix are accepted and ignored.

log_dir mapping (tensorboard.py:81-86):

Implemented: add_scalar → one log_vector; add_scalars → prefixes keys with f"{main_tag}/{k}" into one log_vector; add_text → info(...); add_hparams → log_params + conditional log_vector; flush() is a documented no-op (expman auto-flushes); close(), __enter__/__exit__.

Also implemented, since 2026-07-29: add_histogram, add_image, add_images and add_figure store natively; add_audio/add_video store encoded bytes.

Warn-once, no longer silent. add_graph, add_embedding, add_pr_curve and add_mesh cannot be stored, and now emit a UserWarning the first time each is called. They were previously silent no-ops, so swapping torch.utils.tensorboard.SummaryWriter for this class threw away every image and histogram a user logged with nothing said at runtime — code that looked like it worked. For those four, point a real TensorBoard writer at exp.tensorboard_dir; the dashboard renders it in the TensorBoard tab. add_custom_scalars stays quiet deliberately: it is a layout hint carrying no data.

The compat layer warns rather than raises on input it cannot encode, unlike the native log_image, which raises. It wraps code written for TensorBoard, often mid-way through a multi-day run; killing that run over one unencodable image would be a worse failure than the one being reported.

The constructor eagerly calls log_params({}) and info("SummaryWriter initialized") (:97-98) so the run's files appear on disk immediately.

🔗expman.cli:main

A 31-line shim (cli.py) exposed as the exp console script. Resolves <package_dir>/bin/exp (exp.exe on win32), best-effort chmod 0o755, then subprocess.call([bin_path] + sys.argv[1:]) and forwards the exit code. Missing binary → stderr message and return 1.

🔗Behaviors worth knowing

• All write-path errors are swallowed in the PyO3 layer. Lock poisoning and a closed engine both produce Ok(()). Only the getters raise.
• The GIL is never released. See architecture.
• panic = "abort" means a Rust panic aborts the whole Python process rather than raising a Python exception.
• abi3-py39 + requires-python = ">=3.9" means one wheel per platform covers 3.9+. Note ruff is configured with target-version = "py310" (pyproject.toml:36) — a mismatch that could let 3.10-only syntax through and break 3.9 users.

🔗Tests

wrappers/python/tests/test_experiment.py (6 tests) and test_tensorboard.py (14). The important one is test_vectors_vs_scalar (test_experiment.py:90), which pins the scalars-vs-vectors split in both directions.

Gaps: no test for tensorboard_dir; no test exercises cli.py; test_default_log_dir (test_tensorboard.py:45) takes no tmp_path and may write a runs/ dir into the CWD — which is why wrappers/python/.gitignore contains exactly runs/.

Storage Layout Reference

🔗Reference — data model and on-disk layout

Hand-written from src/core/models.rs and src/core/storage.rs, 2026-07-27.

🔗Directory layout

<base_dir>/                              default "./experiments"
├── .expman/
│   └── notebook.ipynb                   optional Jupyter template (read-only to expman)
├── .projects/                           Project storage directory
│   └── <project_name>/                  Project directory
│       ├── project.yaml                 ProjectMetadata
│       └── README.md                    Markdown frontpage
└── <experiment_name>/                   ExperimentConfig::experiment_dir()
    ├── experiment.yaml                  ExperimentMetadata (created if absent)
    ├── interactive.ipynb                multi-run notebook, if the tab was opened
    └── <run_name>/                      ExperimentConfig::run_dir()
        ├── run.yaml                     RunMetadata
        ├── config.yaml                  merged hyperparameters
        ├── vectors.parquet              metric rows
        ├── run.log                      engine log lines
        ├── console.log                  stdout/stderr tee (Python only)
        ├── interactive.ipynb            generated notebook, if the tab was opened
        ├── artifacts/                   user files, recursive
        └── tensorboard/                 only if Experiment.tensorboard_dir used

.expman/notebook.ipynb is the conventional notebook template (1.3.0): a .ipynb the store carries so the Interactive tab runs project-specific cells without anyone passing exp serve --notebook-template. expman only ever reads it. .expman/ is a dot-directory, so list_experiments already ignores it.

<run_name> defaults to chrono::Local::now().format("%Y%m%d_%H%M%S") (models.rs:32).

Run discovery is directory-name based, not index-driven. list_runs (storage.rs:59) returns every subdirectory except literal artifacts and .ipynb_checkpoints, sorted descending — which is newest-first only because of the timestamp naming convention. list_experiments (:27) ignores all dot-directories (e.g., .projects, .ipynb_checkpoints) and sorts ascending.

🔗Types

All derive Debug, Clone, Serialize, Deserialize. Note there is no Experiment, Run, Metric, or Artifact struct — those are directories on disk plus the metadata structs below.

🔗ExperimentConfig (models.rs:10-25)

Passed to the engine; not itself persisted.

Builder: new(name, base_dir), .with_run_name(run_name).

🔗MetricValue (models.rs:58-63)

#[serde(untagged)] — serializes as the bare scalar.

enum MetricValue { Float(f64), Int(i64), Bool(bool), Text(String) }

From impls for f64, f32, i64, i32, usize, bool, String, &str. No From<u64>. From Python, Int/Bool are effectively unreachable — see architecture.

🔗VectorRow (models.rs:119-123)

step: Option<u64>, timestamp: DateTime<Utc>, values: HashMap<String, MetricValue>.

🔗RunStatus (models.rs:138-143)

#[serde(rename_all = "UPPERCASE")] → RUNNING | FINISHED | FAILED | CRASHED.

🔗Files in a run directory

interactive.ipynb is the one file in a run directory expman writes for the user rather than about the run, so it is the one file the user may own. Since 1.3.0 each generated copy carries a metadata.expman block — generated_by, template, template_hash, content_hash — and that is what decides whether a later launch may refresh it. A notebook whose content no longer matches content_hash, or that has no such block at all, is never overwritten. See the CLI reference.

Metrics are append-only while a run is live. Each flush appends an Arrow IPC batch rather than rewriting the Parquet, so cost is proportional to the rows flushed rather than to the run's history. Rewriting made total write volume grow with the square of the step count; measured on 10k steps, the old path took 48s and the new one 0.4s.

A metric first logged mid-run rolls a new segment, because one IPC stream carries one schema. Readers union the segments with the Parquet and merge rows sharing a step, so read_run_vectors sees a live run's data — a reader that opened only the Parquet would see nothing until the run closed.

A segment truncated by a hard kill yields every batch completely written and stops. Compaction writes the Parquet before deleting segments, so an interruption leaves both and readers union them to the same result.

🔗RunMetadata → run.yaml

name, experiment, status, started_at, finished_at, duration_secs, description, tags, plus #[serde(default)] fields: heartbeat_at, scalars (latest scalar value per key), vectors (latest value per vector key — a summary, not the series), language, env_path.

heartbeat_at is refreshed by the engine every heartbeat_interval_secs (default 30s) for as long as the run is RUNNING. It exists so a hard-killed run — which stays RUNNING forever — is distinguishable from a legitimately long one. None on runs written before heartbeats existed; readers fall back to started_at, which is the conservative direction. See exp reap, storage::is_run_stale, and storage::looks_alive.

description and tags are now written at creation when supplied to Experiment(...), rather than requiring a hand-patch after close().

group and rank place a run in a cohort — the N ranks of a DDP job, or the trials of a sweep. Both are auto-detected from the launcher's environment.

All mutations go through storage::update_run_metadata, which takes an exclusive advisory lock on .run.lock. Under DDP every rank ticks its own metadata update; a bare load-mutate-save races and silently drops the loser's fields. save_yaml also writes atomically (temp file + rename), so a reader never sees a half-written run.yaml and misreports the run as CRASHED.

Default sets status: Crashed (models.rs:181). That is deliberate: a missing or unparseable run.yaml degrades to minimal_run_metadata (storage.rs:193-212), which infers name/experiment from the path and reports CRASHED.

🔗ExperimentMetadata → experiment.yaml

display_name: Option<String>, description: Option<String>, tags: Vec<String>, project: Option<String>.

project is the whole projects hierarchy: a run's project is resolved through its experiment, so no run data ever moves when a project is created or reassigned. It can be written offline — storage::set_experiment_project rewrites only that field — via Experiment(project=...), Experiment.set_project(), expman.assign_project(), exp project assign, or exp project sync.

🔗ProjectMetadata → project.yaml

display_name: Option<String>, description: Option<String>, tags: Vec<String>, created_at: Option<DateTime<Utc>>, plus the generated-projection marker: generated: bool, generated_from: Option<String>, generated_at: Option<DateTime<Utc>>.

A project with generated: true is a one-way projection of a source outside expman (see core/projects.rs). It is overwritten wholesale by the next exp project sync, so the HTTP API refuses writes to it with 409 and the dashboard hides its edit affordances. Its README.md carries a matching <!-- expman:generated ... --> marker on the first line.

🔗ArtifactInfo (storage.rs:144-150)

Serialize only; lives in storage.rs, not models.rs. Fields: path (relative), name, size, ext (lowercased), is_default.

list_artifacts (:69-111) returns two classes: whitelisted run-root files (vectors.parquet, config.yaml, run.yaml, run.log, console.log) with is_default: true, plus a recursive walk of artifacts/ with is_default: false.

🔗Parquet schema

Inferred per batch (rows_to_record_batch, storage.rs:448-535), not declared:

Metric column order is first-seen across the batch. Type comes from the first non-null occurrence: Float/Int → Float64 (ints widen); Text, Bool, or a leading null → Utf8 (Bool/Float/Int stringified as fallback).

Compression is SNAPPY (:379-388).

Schema evolution across batches goes through merge_schemas (:404-417) — existing fields first, then new-only fields appended — and align_batch (:419-446), which back-fills missing columns with typed nulls.

Known defect (verified 2026-07-27). storage.rs:272 downcasts the existing step column to UInt64Array while :475/:486 write it as Int64. The downcast always returns None, the else at :286 keeps the existing batch unfiltered, and the cross-batch step-dedup at :270-291 is dead code. Re-logging the same step in two different flush batches yields duplicate rows. In-batch dedup (:234-252) works fine.

🔗Read functions

record_batch_to_rows (:537-588) converts to JSON: nulls → Value::Null; Float64 NaN/Inf → Value::Null (keeps the JSON valid); timestamps → RFC3339 strings; unhandled Arrow types → Null.

All of these load the full Parquet file into memory — there is no predicate or row-group pushdown.

🔗Error type

ExpmanError (src/core/error.rs:6-35), thiserror:

Io · Arrow (not wasm32) · Parquet (not wasm32) · Yaml · Json · ChannelClosed · RunNotFound(String) · ExperimentNotFound(String) · Other(String)

The two NotFound variants are declared but never constructed — the code returns empty defaults instead.

Module Map Reference

🔗Reference — module map

Hand-written 2026-07-27. ~8,100 lines of Rust in src/.

Two orthogonal gating axes: cargo features (cli, server, python; all opt-in, default = []) and target arch (wasm32 vs not).

🔗Crate roots

// src/lib.rs
pub mod core;                                             // always
#[cfg(all(feature = "server", not(wasm32)))] pub mod api;
#[cfg(wasm32)]                               pub mod app;
#[cfg(feature = "cli")]                      pub mod cli;
#[cfg(feature = "python")]                   pub mod wrappers;

🔗src/core/ — always compiled

The wasm boundary is here: models and error are the only core modules the frontend sees. It reuses the data types but gets no engine, no storage, and no Arrow/Parquet/tokio.

Re-exported at core:: root: LoggingEngine, LogLevel, ExpmanError, ExperimentConfig, MetricValue, RunMetadata, RunStatus, VectorRow. Not re-exported (reach via the module path): models::ExperimentMetadata, storage::ArtifactInfo, error::Result, all storage free functions.

core/mod.rs:21-23 also has a doc-only pub mod jupyter_integration {} that exists purely to inline src/app/README.md into rustdoc.

🔗src/api/ — server feature, non-wasm

🔗src/app/ — wasm32 only (Leptos 0.8 CSR)

Frontend styling is Tailwind from a CDN hardcoded in src/app/index.html:8. stylist is a declared dependency that is used nowhere.

🔗src/cli/ — cli feature

Single mod.rs: clap definitions plus every subcommand implementation. See cli.

🔗src/wrappers/ — python feature

mod.rs re-gates pub mod python; python/mod.rs (298 lines) is the #[pymodule]. See python-api.

🔗Outside src/

🔗Stale in-tree docs

src/api/README.md, src/app/README.md, and src/app/components/README.md are rendered into rustdoc via #![doc = include_str!]. src/api/README.md is stale — it documents a nonexistent /api/events endpoint and mentions utoipa. Prefer docs/reference/.

Development Setup

🔗How-to — set up a dev environment

Verified 2026-07-27.

🔗Nix (the supported path)

nix develop

Gives you (flake.nix:57-80): the fenix stable Rust toolchain with the wasm32-unknown-unknown target, pkg-config, openssl, just, trunk, wasm-bindgen-cli, uv, maturin, protobuf, and cargo-nextest. Sets RUST_LOG=debug, RUST_BACKTRACE=1, and PYO3_PYTHON to nixpkgs python312.

A Cachix cache is declared in the flake's nixConfig, so it applies to you automatically. To use it outside the flake:

cachix use lokeshmohanty

🔗Without Nix

You need: a Rust toolchain with the wasm32-unknown-unknown target, just, trunk, wasm-bindgen-cli, uv, maturin, protobuf (the tboard crate needs protoc), and cargo-nextest.

rustup target add wasm32-unknown-unknown
cargo install trunk wasm-bindgen-cli cargo-nextest
# just, uv, maturin, protobuf via your package manager

🔗One-time, per clone — required

The exp binary bundled into the Python wheel lives at wrappers/python/expman/bin/. It must not be committed (it is 13 MB and platform-specific — on this machine it is dynamically linked against a /nix/store glibc) and it must not be added to .gitignore (maturin honours .gitignore and would drop it from the wheel).

The only correct place to exclude it is your local git exclude file:

echo "wrappers/python/expman/bin/" >> .git/info/exclude

This is documented in CONTRIBUTING.md:32-44. Until you run it, git add -A will stage a 13 MB binary.

🔗Python dev environment

just dev-py

Builds the CLI binary, copies it into the package, creates a uv venv --seed --python 3.12 if absent, runs maturin develop, and installs the package editable with its dev extra.

🔗Optional — pre-commit hooks

.pre-commit-config.yaml exists but nothing installs it, and CONTRIBUTING.md does not mention it. If you want it:

uv run --extra dev pre-commit install

It runs whitespace/EOF/yaml/toml checks, cargo fmt --check, cargo clippy --all-targets --all-features -D warnings, and ruff (check --fix plus format).

Two caveats. (1) The clippy hook uses --all-features, which enables server, which makes build.rs invoke trunk on every Rust commit unless dist/index.html already exists — so hooks can be slow or can fail on a clean checkout. Export EXPMAN_SKIP_FRONTEND_BUILD=1 to avoid it. (2) ruff-format runs here but not in CI, so formatting drift from contributors without hooks installed is never caught.

🔗Next

• build-and-run — the build graph and how to run things
• test-and-lint — what each suite covers

Building and Running

🔗How-to — build and run

Verified 2026-07-27. Prefer just recipes over raw cargo — several of them exist specifically to route around build.rs.

🔗The one thing to understand first: build.rs

When the server feature is enabled and the target is not wasm32, build.rs shells out to trunk build --release to produce dist/. On failure it falls back to an existing dist/index.html, writes placeholder assets during CARGO_DOC, and otherwise hard-exits(1) (build.rs:44).

So cargo build --all-features on a clean checkout without trunk on PATH fails at build-script time with an error that does not obviously say "install trunk".

Two escape hatches:

export EXPMAN_SKIP_FRONTEND_BUILD=1   # skip trunk; asserts dist/index.html exists
just prep-dist                         # mkdir -p dist && touch dist/index.html

CI uses both: every Rust job downloads a prebuilt frontend-dist artifact into dist/ and sets EXPMAN_SKIP_FRONTEND_BUILD=1.

build.rs reruns on changes to src/app, Trunk.toml, and Cargo.toml, and builds with CARGO_TARGET_DIR=target/wasm_build with MAKEFLAGS stripped (a recursive-cargo guard).

🔗Common commands

🔗Running

just serve [DIR]           # dashboard on 127.0.0.1:8000, DIR defaults to ./experiments
just list [DIR]            # list experiments
just run <ARGS>            # cargo run --features cli,server -- <ARGS>
just example-rust          # examples/rust/logging.rs
just example-py            # examples/python/basic_training.py (runs dev-py first)
just bench                 # test_log_vector_is_fast in release, with output
just stats                 # tokei over src/ and wrappers/python/

The dashboard needs both features: cargo run --features cli,server -- serve. serve is gated on server; without it the subcommand does not exist.

🔗Feature combinations that matter

Wheel builds set features = ["python"] only (pyproject.toml:31), which is why maturin never needs trunk.

🔗Frontend-only iteration

Trunk.toml targets src/app/index.html with dist = "dist". trunk serve will hot-reload the WASM app, but the API calls need a backend — run just serve in another shell and point the frontend at it (CORS is fully permissive, so cross-origin from the trunk dev server works).

Note the WASM bundle is built with data-wasm-opt="0" — the current expman-app_bg.wasm is ~2.6 MB.

🔗Styling

Tailwind is built, not fetched. src/app/index.html declares data-trunk rel="tailwind-css" href="../../assets/tailwind.css"; trunk runs the standalone Tailwind CLI, pinned in Trunk.toml:

[tools]
tailwindcss = "3.4.17"

Trunk downloads that version on a normal build. The Nix build runs with TRUNK_OFFLINE=true, so it takes pkgs.tailwindcss_3 from nativeBuildInputs instead — the two must stay the same version. They currently emit byte-identical CSS; if you bump one, bump the other.

Config is tailwind.config.js. Two things to know:

• Classes are scanned out of the Rust source (content: ["./src/**/*.rs"]). A class assembled at runtime — format!("text-{}", colour) — will not be generated and the element renders unstyled. Build such classes from a match that returns whole literals.
• @tailwindcss/typography needs no npm. The standalone CLI bundles it, so the prose classes used for project READMEs work. They did not work under the old cdn.tailwindcss.com script, which ships no plugins — that markdown had been silently unstyled.

🔗Fonts

Three type roles, self-hosted:

The woff2 files are vendored into assets/fonts/ (latin + latin-ext only, ~273 KB) with @font-face rules in assets/fonts.css. index.html copies them via data-trunk rel="copy-dir" and links the CSS; api/frontend.rs embeds *.woff2 into the binary.

Two ways this breaks silently. If frontend.rs loses its #[include = "*.woff2"], the faces 404 and the page falls back to system fonts — which looks like a CSS bug. And code/pre are styled by Tailwind preflight from its own variable, not from the font config, so the mono role is set directly in the <style> block; drop that rule and every code block silently reverts.

To regenerate after a Fontsource release, unpack the three @fontsource-variable/* tarballs and copy the latin subsets, taking the unicode-range values from each package's own unicode.json rather than transcribing them:

for pkg in space-grotesk nunito cascadia-code; do
  curl -sL "https://registry.npmjs.org/@fontsource-variable/$pkg/-/$pkg-<VERSION>.tgz" \
    | tar xz -C "$pkg" --strip-components=1
done
# then copy files/*-latin{,-ext}-wght-{normal,italic}.woff2 into assets/fonts/
# and rebuild assets/fonts.css from each package's unicode.json

Verify from a loaded page, not from the stylesheet:

getComputedStyle(document.querySelector("h1")).fontFamily    // Space Grotesk Variable
getComputedStyle(document.querySelector("code")).fontFamily  // Cascadia Code Variable
[...document.fonts].filter(f => f.status === "loaded").map(f => f.family)
// Should be empty — the dashboard makes no third-party requests:
performance.getEntriesByType("resource").map(r => r.name).filter(n => !n.startsWith(location.origin))

🔗New asset? git add it before nix build

A flake's src = ./. sees only git-tracked files. An untracked assets/whatever.css builds fine with trunk and then fails under nix build .#expman with a confusing "No such file or directory" for a path that plainly exists. Stage new assets before testing the Nix build.

🔗Nix builds

nix build .#expman                          # the exp CLI (features cli,server)
nix build .#python3Packages.expman-rs       # the Python package
nix run github:lokeshmohanty/expman-rs      # run the CLI without cloning

packages.expman builds the frontend in preBuild with TRUNK_OFFLINE=true (forcing the Nix-provided wasm-bindgen-cli) and doCheck = false — tests are CI's job.

Caveat: packages.python3Packages.expman-rs has no step that populates expman/bin/exp, so its exp console script reports "Bundled binary not found". Get the CLI from .#expman instead.

🔗Troubleshooting

Testing and Linting

🔗How-to — test and lint

Verified 2026-07-27.

🔗Commands

All Rust test commands use --all-features, which enables server and so triggers the build.rs trunk build. Set EXPMAN_SKIP_FRONTEND_BUILD=1 if you already have a dist/. See build-and-run.

cli_test.rs uses assert_cmd::cargo::cargo_bin_cmd!("exp"), which needs the cli feature — another reason the suite is always run with --all-features.

🔗What each suite covers

🔗tests/integration_test.rs — 12 tests, the core engine

Run dir creation · vectors → Parquet · params → YAML · test_log_vector_is_fast (the throughput benchmark, also just bench) · status persisted on close · artifact save (relative and absolute paths) · Parquet schema merge · latest-scalar readback · corrupt YAML metadata · concurrent logging from multiple threads · same-step replacement.

🔗tests/api_test.rs — 7 tests, the HTTP layer

In-process via tower::ServiceExt::oneshot against build_router + AppState — no network. A TempDir fixture builds one experiment with one run.

Covers: list experiments, get/update experiment metadata, list runs, get run metadata, server config, get metrics.

Not covered: every Jupyter route, every TensorBoard route, both SSE streams, artifact content.

🔗tests/cli_test.rs — 10 tests, the binary

list (with and without --experiment) · inspect · clean --keep 5 dry-run (asserts nothing is deleted without --force) · export json/csv, with and without data · import error paths · export --format tensorboard with no data.

🔗wrappers/python/tests/ — 20 tests

test_experiment.py (6): basic file creation, singleton init, artifact round-trip, complex types in config.yaml, crash → status: FAILED, and test_vectors_vs_scalar — the important one, pinning that scalars land in run.yaml and vectors in vectors.parquet, with cross-assertions both ways.

test_tensorboard.py (14): directory structure, context manager, default log_dir generation, scalars with and without step, add_scalars prefixing, add_text, add_hparams (including empty and None metrics), a parametrized sweep asserting all 11 stub methods do not raise, flush() no-op, sequential writers not interfering, and both log_dir mapping branches.

Not covered: tensorboard_dir; cli.py (the exp console script is never exercised from Python).

test_default_log_dir (test_tensorboard.py:45) takes no tmp_path and can write a runs/ directory into the CWD — which is why wrappers/python/.gitignore contains exactly runs/.

🔗What CI runs

ci.yml (push + PR on main, ignoring **.md, examples/**, docs/**, .gitignore, LICENSE) calls build-assets, then rust.yml and python.yml in parallel.

• rust.yml — just fmt-check + just lint-rust, then just test-release. Both jobs download frontend-dist and set EXPMAN_SKIP_FRONTEND_BUILD=1.
• python.yml — just lint-py (ruff over wrappers/python and examples/), then downloads the CLI artifact, just bundle-cli-bin, uv sync --extra dev, builds and installs a wheel, and runs pytest.

Two gaps worth knowing:

• ruff format never runs in CI — only ruff check. Formatting drift is caught only by pre-commit hooks, which nothing installs automatically.
• Python tests only ever run on Linux. python.yml:29 computes the CLI artifact name from runner.os with a nested ternary, implying an intended OS matrix that does not exist; the job is always ubuntu-latest.

And the big one: a release does not wait for tests. See release.

Adding an API Endpoint

🔗How-to — add an API endpoint end to end

Written 2026-07-27 by tracing how the TensorBoard integration was added.

Adding a route touches five or six files. The TensorBoard work in the current working tree is the best worked example — follow it file by file.

🔗1. Handler — src/api/<area>.rs

Handlers take State<AppState> and Path/Query extractors and return Json<serde_json::Value> (most handlers build responses with the json! macro rather than typed structs — follow the local style).

pub async fn status_tensorboard(
    State(state): State<AppState>,
    Path((exp, run)): Path<(String, String)>,
) -> Json<serde_json::Value> { ... }

Resolve paths from state.base_dir — never trust a client path. For anything that reads a user-supplied path, copy the canonicalize + starts_with guard in src/api/artifacts.rs:47-58.

🔗2. Register the route — src/api/mod.rs

Add pub mod <name>; and a .route(...) line in the router (mod.rs:42-140). Keep the /experiments/{exp}/runs/{run}/... shape — only one existing route deviates and it is regretted.

🔗3. State, if the endpoint owns something — src/api/state.rs

AppState is Clone and cloned per request. Anything shared goes behind Arc<Mutex<...>> in a Clone newtype, following JupyterManager (jupyter_service.rs:262-265).

Use std::sync::Mutex, and never hold the guard across an .await. Every existing critical section is a short map operation. If you need to hold a lock across await points, that is a design change, not a local one.

If the endpoint spawns a process, register cleanup in serve()'s shutdown path (mod.rs:186-187) and honour state.shutdown_token.

🔗4. Response type — src/core/dto.rs

Define the response shape here, once. core/dto.rs is compiled for native and wasm32, so the handler serializes exactly the type the frontend deserializes, and the compiler ties them together.

Do not return serde_json::json!({...}) from a handler and add a matching struct in src/app/models.rs. That is how these drifted for a long time: the two were hand-mirrored with nothing checking them, which silently shipped a status colour keyed on a "COMPLETED" value the API never emitted.

src/app/models.rs is a re-export list — add your type to the pub use there and nothing else.

Wire types stay distinct from the storage models in core/models.rs: a DTO may expose less than RunMetadata, but it does so through a constructor (dto::Run::new), so changing a storage field is a build error rather than a quietly changed endpoint.

🔗5. Frontend fetch — src/app/fetch.rs

Plain gloo_net::http::Request, returning Result<T, String>. The house style is resp.text() then serde_json::from_str rather than resp.json().

Check the HTTP status. Several existing functions (stop_jupyter, stop_tensorboard) ignore it — do not copy that.

🔗6. Component — src/app/components/

Wrap the fetch in LocalResource::new(...) and render inside <Suspense>. Declare it in components/mod.rs and mount it where it belongs (a tab in pages/experiment_detail.rs:375,405, for example).

If you open an EventSource or spawn anything, close it in on_cleanup — components/console.rs:105 is the correct pattern. components/tensorboard.rs omits this and leaks processes on navigation.

🔗7. Test — tests/api_test.rs

In-process via tower::ServiceExt::oneshot against build_router + AppState, with a TempDir fixture (api_test.rs:12-49). No network, fast. Add a case per route; the existing 7 tests are the template.

🔗Checklist

• handler with path-traversal guard where relevant
• route registered in mod.rs
• AppState field + shutdown cleanup if it owns a resource
• frontend model matching the wire format
• fetch function that checks the status
• component with on_cleanup for anything long-lived
• test in tests/api_test.rs
• docs/reference/http-api.md updated
• just ci green

🔗Things to know before you start

• There is no caching. Every request re-reads the filesystem and re-parses Parquet. If your endpoint is hot, that is the constraint to design against.
• Realtime is SSE only. Both existing streams are 500 ms poll loops with 15 s keep-alives, terminated by .take_until(shutdown.cancelled()) (metrics.rs:41, :78). Axum's ws feature is enabled but unused.
• CORS is fully permissive (mod.rs:146-149). Fine for a localhost tool; do not add anything that assumes an origin check.
• Spawned services are not proxied. Jupyter and TensorBoard are reached directly at http://localhost:{port}, which is why the dashboard only works from the machine running the server. If you are adding a third such service, consider building the proxy route instead — see decisions.

Cutting a Release

🔗How-to — cut a release

Verified 2026-07-27 against the 11 workflows in .github/workflows/.

🔗The procedure

just bump patch      # or minor, or major
git push

That is the whole thing. The commit message is the release trigger.

just bump:

1. reads the current version from Cargo.toml
2. rewrites it there and nowhere else — pyproject.toml takes it via maturin's dynamic = ["version"], and flake.nix reads it with builtins.fromTOML
3. cargo update -p expman to refresh Cargo.lock
4. runs just check-versions, which fails if a literal has been reintroduced
5. commits Cargo.toml + Cargo.lock as release: bump version to X.Y.Z

It does not tag and does not push. The tag v<version> is created later by the GitHub Release action.

🔗What fires on push

check-release.yml:26 regex-matches github.event.head_commit.message against:

^release: bump version to (\d+\.\d+\.\d+)$

Because a bump commit touches Cargo.toml and Cargo.lock, four workflows fire simultaneously: publish.yml (no path filter), nix.yml (matches the changed paths), docs.yml (no path filter), and ci.yml.

publish.yml orchestrates: check-release → build-assets → rust + python → publish-cargo + publish-pypi in parallel → github-release. The rust and python jobs are the release gate: nothing publishes unless both pass.

🔗The traps

🔗1. The regex is exact and failure is silent

Anchored, single spaces, no v prefix, no trailing period, exactly three numeric components. A squash-merge that rewrites the subject, a merge commit, or any suffix yields should_release=false — the push succeeds, nothing publishes, and no job fails. There is no negative signal. If a release did not happen, check the commit subject first.

Related: head_commit.message is only the last commit of the push. Pushing several commits where the bump is not last releases nothing.

🔗2. Releases wait for tests — but not the ones in ci.yml

Fixed 2026-07-28. publish.yml calls rust.yml and python.yml itself, and both publish jobs need them, so a failing test blocks publication.

The subtlety worth keeping in mind: the ci.yml run you see on the same commit is still a separate workflow and still gates nothing. The gate is the rust and python jobs inside the publish run. A release commit therefore runs the suite twice, in parallel. Run just ci locally anyway — it is faster than discovering it in CI.

🔗3. Two workflows race to create the same GitHub Release

Both publish.yml:33-42 (with artifact-pattern: wheels-*) and nix.yml:55-63 (with artifact-pattern: nix-build-results) invoke github-release.yml for the same tag. softprops/action-gh-release creates-or-updates, so whichever finishes second updates the release — the final asset set depends on race ordering, and release notes may be regenerated twice. This is the likely cause of any "the release is missing the wheels" or "missing the nix binary" flakiness.

🔗4. The dist/ + --allow-dirty knot

Read commits 49c24ca and d423ff8 together:

• dist/ is gitignored, so cargo package was omitting the built frontend, and consumers building with --features server hit build.rs's exit(1). Fix: Cargo.toml gained include = [... "dist/**/*"].
• That made things worse for CI: it downloads frontend-dist into dist/, which is now both gitignored and in Cargo's include, and cargo refuses to package files it will ship but that are not committed. Fix: publish-cargo.yml:29 gained --allow-dirty.
• The same commit added a CARGO_DOC branch to build.rs (:12, :30-40) writing placeholder dist/ assets instead of exit(1), which is what unblocked docs.yml.

If you touch Cargo.toml's include, build.rs, or publish-cargo.yml, re-read all three together.

🔗5. The version lives in exactly one place

Fixed 2026-07-28. Cargo.toml is the source; pyproject.toml is dynamic = ["version"] and flake.nix reads builtins.fromTOML ./Cargo.toml. just check-versions asserts no literal has crept back and runs in CI's lint job.

Why it mattered: Python's __version__ comes from CARGO_PKG_VERSION, not pyproject.toml, so a desync shipped a wheel whose metadata disagreed with expman.__version__ — a discrepancy nothing would have caught.

Still never hand-edit a version. Always just bump.

🔗6. Never commit wrappers/python/expman/bin/

13 MB, platform-specific, and it must stay untracked-but-not-gitignored so maturin still ships it. See setup.

🔗7. docs.yml uses force_orphan: true

Every push to main rewrites gh-pages history from scratch (docs.yml:33).

🔗Pre-flight checklist

just ci                      # nothing else gates this
git status                   # bin/ untracked? debris staged?
git log --oneline -1         # will the bump commit be LAST on the push?
just bump patch
git push

Then watch: publish.yml, nix.yml, docs.yml, ci.yml all running; the release at v<version> should end up with both the wheels and the nix results attached.