Operations and safety¶

Upload and model selection¶

Upload DICOM files, a folder, or ZIP archives. The portal groups images by Study, Series and modality. Select one assessed series and compatible models. The recommended starting models are total for CT and total_mr for MR. In “What do you want?”, select Segment structures, AI draft report (NV-Reason-CT), or both. The segmentation action adds the displayed default; “Choose other models…” opens the grouped catalog. No action is selected automatically. AI reports are restricted to diagnostic CT; MR, PET and detected CBCT remain disabled with reasons. PET1 ISRT tasks require CT in the same Study and Frame of Reference. Do not treat this check as image registration.

The browser uses resumable sessions with requests up to 32 MiB. A resumed chunk must use the server's expected byte offset. The receiving request times out after 60 seconds without new data. Session assessment runs in the background. Restart recovery marks interrupted jobs and sessions for review or retry; retention cleanup removes expired data.

Shared input orientation¶

Previously, DICOM-to-NIfTI conversion preserved stored pixel order. An array-based model could therefore see a left/right or anterior/posterior reversal on prone or feet-first storage, even when its prediction had a valid affine and its RTSTRUCT passed the geometry gate. The public head-and-neck fixture exposed this in LNCTVSeg and the production-enabled HNLNL task. Regenerate earlier outputs from non-HFS storage for exposed array-based tasks; a valid RTSTRUCT geometry check alone cannot exclude a model-side laterality error.

orientation_contract.py now applies one lossless contract at dicom_series_to_nifti.py. Standard HFS storage is axial L,P,S, coronal L,I,P, and sagittal P,I,R (NIfTI axcodes). These directions are checked against SimpleITK reading synthetic DICOM with the usual IOPs. The acquisition plane is the closest anatomical axis assigned to the source slice axis by nibabel's polar orientation decomposition. Its deterministic axis-order tie break applies at equal obliquity. Oblique direction cosines remain unchanged; there is no interpolation or deobliquing. PatientPosition text does not select the transform: the affine derived from IOP/IPP does. An already standard HFS file is not rewritten, preserving its array, affine and compressed NIfTI bytes.

Each input has a .orientation.json sidecar containing the original and model grids, axis transform and both index mappings. Both RTSTRUCT converters restore each prediction to acquired DICOM indices before constructing contours. A shifted, cropped, scaled or sheared grid is refused; a runner that crops or resamples must first restore its output to the complete input grid. The affine tolerance remains 0.001 mm with zero relative tolerance. Contour round-trip Dice ≥0.95, plane tolerance 0.05 voxel, SOP-reference checks and physical bounding-box checks are unchanged. Export sidecars retain the transform after inference scratch is removed and are included in SSH/Colab retrieval and result ZIPs.

Viewer points and box corners are transformed into model indices in a task-local prompt copy. The viewer continues to display original DICOM pixels and patient-space contours. The DICOM crop tool retains original pixels and contour coordinates. PET SUV conversion preserves the normalized affine; PET/CT fusion still registers PET onto the CT reference and exports against that CT series. These consumers do not need a second orientation operation.

The source catalogue contains 254 names, including 41 informational or blocked entries. The following is the route inventory in brief; the implementation report contains one row per task. “Changes” refers to portal storage on nonstandard input. It does not imply that an upstream canonical model tensor changes.

Task family	Input and downstream handling	Contract effect
TotalSegmentator (41 fallback names; runtime may advertise more)	Portal NIfTI, then TotalSegmentator canonicalisation and inverse	Portal storage changes; canonical tensor convention retained
nnU-Net v1/v2, LNCTVSeg, HNLNL, LN-Seg-FM, Choi, PyCAD, MultiTalent	Portal NIfTI; stored-array inference; MultiTalent exports overlapping masks	Shared contract replaces the proposed LNCTVSeg/HNLNL special cases
CADS, MOOSE/Dental, MRI/VIBE/MRSegmentator, lungmask, LungTumorMask, MedLSAM, Raidionics, UWLAIR, LNQ/CompAI/DBDMP, ISRT	Portal NIfTI; model-specific crops/preprocessing retained	Portal storage changes; treat unverified internal orientation as exposed
DOTATATE, LION and ISRT PET/CT	Portal PET NIfTI; existing intensity conversion and physical registration retained	PET and CT normalize independently; exact inverse uses the export reference
MONAI bundles, UniversalModel, CERR SMIT/legacy, NV-Segment	Portal NIfTI; bundle-specific transforms, including internal orientation where supplied	Portal storage changes; upstream transforms retained, not assumed identical across bundles
ANTsPyNet, HippUnfold, SynthSeg, FastSurfer, GOUHFI, SCT, SIAM, brain-extraction tools, TotalSpineSeg	Portal NIfTI; physical-space, conforming or model-specific pipelines	Portal storage changes; existing model preprocessing and restoration retained
MRAnnotator (4 tasks)	Portal NIfTI followed by release-required LAS (`models/manifests/mrannotator.json`)	LAS convention retained
LNQ atlas	Portal NIfTI followed by LPS and physical atlas registration	LPS requirement and oblique refusal retained
Prompted tools (12 tasks)	Portal NIfTI plus mapped viewer points/box; SAM-Med3D keeps its internal canonicalisation	Image and prompts change together
NV-Reason-CT report	Portal NIfTI, TotalSegmentator coverage and checked MONAI `Orientationd` processor	Portal storage changes; report pipeline remains separately gated
Informational/blocked entries (41)	No accepted inference route, including the other three LNCTVSeg architectures	Unchanged; no new availability or acceptance

No active catalogue model bypasses the portal NIfTI boundary to read DICOM pixels itself. DICOM reads in report validation inspect metadata; viewer, staging and crop tools consume DICOM for their existing functions. Upstream runtime source and installed bundle configurations were outside this worker's authorized inputs, so their internal preprocessing is not newly verified by this inventory. Existing release requirements are retained.

The manager can run the public storage-variant check with:

"$RTSEG_WORKER_PYTHON" "$RTSEG_APP_ROOT/scripts/run-orientation-contract-gpu.py" \
  --run-root "$RTSEG_ORIENTATION_RUN_ROOT"

Set RTSEG_MODEL_ROOT and RTSEG_RUNTIME_ROOT to the existing installation and use a fresh run directory on project storage outside the model bundle. The script creates a short /tmp directory for sockets, keeps images and inference outputs under the run directory, and closes its temporary directory on exit. It verifies identical CT voxels after HFS/HFP/FFS/FFP re-export, then runs HNLNL, LNCTVSeg and CADS head-and-neck OARs on the public head-and-neck fixture, and lungmask LTRC lobes on MSD lung. Each assigned task runs on all four variants and the original series. It reports per-label physical Dice from independently rasterized RTSTRUCTs, laterality, LNCTVSeg raw-label Dice against the author reference, and wall time. --prepare-only builds and checks fixtures without inference. Empty labels are reported explicitly, not counted as perfect Dice. The script does not publish or remove acceptance markers.

The synthetic checks verify software geometry. GPU inference on these variants remains a manager check; neither these tests nor local drafting establish clinical or manuscript readiness.

Prompts¶

Prompt coordinates are [column, row, slice] on the selected series, sorted along the DICOM slice normal. The upload preview and viewer APIs share this order and the stored pixel grid. Standard orientation changes the display only; prompt clicks map back to stored coordinates. Recreate prompts saved with the older raw-z preview for flipped or non-axial series. Keep prompt data with its source series. The catalog describes the required prompt type and supported modality for each model.

PAM propagates a positive point or axial box through a volume.
SAT3D needs a positive 3D point and supports negative points within its prompt-centred ROI.
SAM-Med2D is slice-wise; prompt every intended slice.
MedSAM, SAM-Med3D, MedSAM2/Medical-SAM2, Medical-SAM3, SegVol, nnInteractive, VISTA3D and VoxTell have separate prompt, license and field-of-view restrictions. Read their catalog entries and the model ledger before selecting them.
BiomedParse v2 weights are available. biomedparse_prompted remains gated by final runtime installation and per-device GPU reference, repeatability and DICOM acceptance.

The assessed-upload endpoint accepts prompt_json and validates the required prompt type and coordinate bounds. Direct multipart and direct session-job submission reject paired or prompted models with HTTP 400. Paired series must have present, matching Study and Frame of Reference identifiers. See API for route differences.

RTSTRUCT geometry gate¶

The converters require the prediction, model input and DICOM source grid to differ only by a full-grid axis permutation or flip, require an explicit name for every nonzero label, and check DICOM SOP class and source image references. Both writers use the same SimpleITK/GDCM file order as dicom_series_to_nifti.py. Each polygon uses its own slice's position, orientation and pixel spacing: DICOM row spacing applies to rows and column spacing to columns. Each contour references that slice's SOP Instance UID. Overlapping-mask conversion preserves separate ROIs.

Validation compares physical LPS bounds and independently rasterizes contours on the referenced acquired planes. A NIfTI affine cannot represent irregular slice spacing, so it is checked for source-grid correspondence but does not position contours. The thresholds are unchanged: minimum Dice 0.95, plane error at most 0.05 voxel, and bounding-box tolerance equal to the larger of 2.1 mm and 2.1 times the largest voxel spacing. Large gaps do not enlarge the plane tolerance.

External model availability also requires orientation acceptance markers. These technical checks detect conversion errors; they do not measure segmentation accuracy in a new patient. Empty candidate predictions may complete with a no-candidates note; contours are not fabricated.

The earlier rt-utils==1.2.7 volume transform swapped unequal row/column spacings and assumed uniform slice spacing. The physical gate rejected the tested non-square results before delivery. A supplemental square-pixel gapped fixture exposed a limit of the old affine-based gate: it passed despite a maximum 1.600000 mm displacement from the referenced slice. The corrected gate checks acquired planes. rtstruct_geometry.py now constructs per-slice contours while retaining rt-utils for the DICOM container and ROI metadata. The synthetic orientation matrix exercises the production converters and independent viewer mapping, including non-square pixels, gapped stacks and IOP 1,0,0,0,-1,0 and 0,1,0,0,0,-1. TotalSegmentator also uses this converter; its native RTSTRUCT writer is not used. The crop tool retains existing contour coordinates and only removes contours outside the crop and rewrites references.

The writer retains rt-utils' complete OpenCV contour set, including one-point, two-point and zero-area pieces at scan edges. The validator rasterizes their source pixel centres, and the independent viewer mapping preserves the same points. Every piece receives the same coordinate, plane and reference checks; the complete mask remains subject to the unchanged geometry thresholds. An ROI omission note applies only if a nonempty mask produces no contour at all. The converter reports its name and voxel count in the worker log; the portal includes these omissions in job details, conversion_notes and CONVERSION_NOTES.txt in the result ZIP. If no ROI remains, the job uses the no-structures outcome with explicit omission notes. SCT's valid empty masks use the same empty-result path; missing SCT files still fail. HD-CTBET mirrors model directories into job scratch so training logs do not target the read-only bundle. SegVol's catalog prompt_contract.allow_box_with_points=false is enforced by shared server validation with HTTP 400; text may accompany either points or a box.

Prostate SBRT crop¶

The CPU tool crops single-frame axial CT and matching RTSTRUCTs along patient z only. It accepts slice normals within 1 degree of patient z. The range spans every ROI with contour points whose name starts with PTV or CTV, such as PTV, PTV1, PTV 2 and CTV, across all supplied RTSTRUCTs, with a default 150 mm margin. Subtraction helpers such as Bowel - PTV are excluded. Matching ROIs without contours are listed as ignored. The report gives each target's z extent, and validation requires every target to keep all of its contours. The optional local total postprocess selects the ROI named prostate and retains the original full RTSTRUCT.

Pixel data are unchanged. Whole contours outside the retained slice range are removed. New Study, Frame, Series and SOP UIDs are the default, with RTSTRUCT references rewritten to match, including each ROI's ReferencedFrameOfReferenceUID. keep_uids is a debugging option. Downloads include a crop report. Review the cropped image range and contours before any clinical use.

Catalog and job lifecycle¶

A cold model catalog waits for its first preflight. Subsequent requests use the cache while background refresh runs. The default cache lifetime is 600 seconds; editing the enabled-model file also requests refresh. Enabled flags depend on installation, orientation markers and local policy. A local task that passes its technical gates but is absent from the enabled list reports not_enabled_on_this_server. Missing dependencies, pending markers and declared blockers keep their specific status.

Refresh exceptions are logged with a traceback. The portal continues serving the last good catalog, and health exposes catalog_built_at, catalog_refreshing and catalog_error. A successful refresh clears the error. An initial build failure returns HTTP 503 from the models endpoint. Inspect service logs for the full exception; the health response omits exception messages and paths.

Jobs run under one worker lock. Local and SSH inference use a default 12-hour per-task limit. The portal snapshots the local process tree, sends SIGTERM, waits up to 45 seconds, then kills survivors with PID start-time checks. The SSH path also requests termination of the matching remote runner. Colab compatibility paths retain their separate timeout behavior.

Public audit fixtures and private evidence¶

catalog/orientation-audit-fixtures.json contains public fixtures from MSD, KiTS19, TotalSegmentator MRI, TemplateFlow and other named releases. Missing private-case entries are intentional. Supply a complete external JSON task-to-fixture map with RTSEG_ORIENTATION_FIXTURES; values resolve under the model bundle's testdata. RTSEG_ORIENTATION_PROMPTS can select an external prompt map. Absolute prompt paths are supported by the existing path resolution.

Historical acceptance metadata retains measured counts and model limitations. Private fixture names, job identifiers and private artifact hashes are withheld; a public fixture has not been substituted into those historical results. Revalidation aliases can be supplied with RTSEG_LEGACY_ARTIFACT_ALIASES, and RTSEG_RAIDIONICS_RESULT_DIR selects the legacy result directory. These configurations belong outside Git.

scripts/audit-worker-storage.sh is a specialized NFS/loop-image audit, not a prerequisite for local-container deployment. Configure its RTSEG_STORAGE_ROOT, RTSEG_STORAGE_FSTYPE_PREFIX, RTSEG_AUDIT_HOME, RTSEG_WORKER_STATE_ROOT, application/runtime/venv roots and job/build/archive paths for that layout. It retains the existing 1 GiB Docker-storage threshold.

Privacy and checks¶

Restrict the portal to authenticated users over TLS. Job IDs are not credentials. Do not commit uploads, patient prompts, model weights, license files or logs. Retention defaults to 24 hours; set RETENTION_HOURS and review storage permissions and backup policies. /api/v1/config exposes the active retention and upload limits.

Completed segmentation and draft-report jobs retain the selected input images on the portal host in job/input for research viewing until retention expiry or DELETE /api/v1/jobs/{jid}. Retention is measured from job creation; the existing hourly cleanup removes the whole job. Viewer access returns 404 after expiry even before cleanup runs. Plan storage for these retained images as well as outputs. Companion CT input and remote worker copies keep their existing cleanup behaviour. Older completed jobs whose inputs were deleted cannot be viewed without a new upload. The separate prostate crop tool retains its existing output package and input cleanup.

Deletion is available for terminal jobs and removes local images, contours, reports and logs. Queued or running jobs return 409. SSH cleanup must succeed before local deletion; a 502 response means the local job remains for a retry. Contours are displayed on acquired slice planes without interpolation or image registration. These synthetic geometry checks and local API changes do not establish clinical or manuscript readiness.

curl -fsS http://localhost:8080/api/v1/health
curl -fsS http://localhost:8080/api/v1/models

See EXTERNAL_MODELS.md for the retained numerical results and model-specific restrictions.

NV-Reason-CT draft reports¶

Use diagnostic CT only. Known CBCT device names, explicit CBCT/cone-beam text, and a Varian/Elekta acquisition with a Ram-Lak kernel plus either 100–150 kV or a 450–480 mm reconstruction diameter are refused. The HU rule requires the 0.1 percentile between −1100 and −900 HU and a maximum above 300 HU. These rules can refuse tightly cropped diagnostic images and do not establish image quality.

Coverage uses total --fast (3 mm) by default, with no ROI subset. Set RTSEG_NV_REASON_CT_COVERAGE_MODE=full to use full resolution. Chest requires lung volume ≥1000 mL and heart ≥100 mL; abdomen requires liver ≥500 mL and at least one kidney ≥50 mL. If neither region qualifies, the report records not_generated. Compare fast and full decisions on the acceptance series before accepting the faster configuration.

Mapped statements are withheld if the organ group is absent. Truncated masks, insufficient volumes or missing required components receive “[partly in the scanned volume]”. Regional bones are checked for presence, without a truncation warning at expected scan boundaries. Pleura maps to lungs; mediastinum/hila to heart, aorta, esophagus and trachea; great vessels to aorta and the total task's pulmonary-vein mask; chest wall/breast to ribs, sternum and costal cartilages; bowel/appendix to small bowel, colon and duodenum. These group checks do not directly segment every structure named in a heading. Medical devices, CHEST/ABDOMEN labels and other unmapped headings are listed once under “Not checked against the segmentation”. Peritoneum/omentum, lymph nodes and abdominal wall/soft tissues stay unmapped. Impression/label mentions of absent anatomy remain flagged, with raw output available in a collapsed appendix.

The publisher crop is checked for fallback and ≥90% containment of regional organ masks. Failure triggers a mask-centred 384 mm cube, sampled on a 192³ LPS grid at 2 mm with air padding. Actual model tensors must match the expected normalized crop within absolute tolerance 1e-5. Larger anatomical extents can still exceed this crop and are flagged. Midplane thumbnails do not show the entire volume.

Contour volumes are approximate polygon area × source slice spacing / 1000; holes and overlaps are not subtracted. Non-CLOSED_PLANAR contours are omitted. Slice ranges are zero based. These are not dosimetric measurements.

Install the isolated runtime with scripts/bootstrap-nv-reason-ct-runtime.sh /runtimes/envs/nv-reason-ct. Runtime resolution checks RTSEG_NV_REASON_CT_PYTHON, then RTSEG_RUNTIME_ROOT/envs/nv-reason-ct/bin/python, then RTSEG_MODEL_ROOT/runtime-envs/nv-reason-ct/bin/python. Store the pinned snapshot under RTSEG_MODEL_ROOT/weights/nv-reason-ct/<revision>/. Set TOTALSEG_WEIGHTS_PATH to installed total weights: Dataset297 for fast coverage, Dataset291–295 for full. Missing coverage weights fail before a download attempt. Usage reporting is disabled in a job-local TotalSegmentator configuration. Model loading uses local files, offline Hub settings and pinned custom model code.

Use sm_80-class or newer CUDA hardware. The worker waits up to 120 seconds for 14 GiB free before coverage/model loading and checks for 4 GiB free before generation. Resource refusal returns a reason. Model caches use RTSEG_NV_REASON_CT_CACHE, defaulting to RTSEG_SCRATCH_ROOT/nv-reason-ct-cache; use writable persistent storage for warm Triton/Inductor caches. This supports read-only model/runtime mounts. When RTSEG_SCRATCH_ROOT is unset, the dispatcher creates a short per-job rtseg-* directory under system temp and removes it on exit. Configured scratch longer than 60 bytes uses a separate short directory for child TMPDIR, TEMP and TMP; an unwritable setting also redirects scratch there. The runner logs each configured-path fallback. This leaves room for nnU-Net's multiprocessing Unix sockets. Output locations are unchanged; configured scratch and persistent caches are not deleted. Forced termination with SIGKILL cannot run exit cleanup, so inspect abandoned rtseg-* directories after a forced stop.

Enable the task through ENABLED_EXTERNAL_MODELS or the enabled-model file in local mode. The marker at validation/nv_reason_ct_report.json requires status: "passed", the pinned revision, checks chest, abdomen, pelvis, cbct, and coverage_mode matching the configured mode. Older markers without a mode attest only full. Remote tasks use the remote worker's marker. Reports and input share job retention; retained remote results after failed retrieval also need operator retention. See the ledger for observed diagnostic failures and historical GPU measurements.

Automatic NV-Segment tasks¶

nv_segment_ct_auto and nv_segment_ctmr_ct_body accept a single CT series. nv_segment_ctmr_mri_body accepts one body MR series. nv_segment_ctmr_mri_brain uses T1 brain MR; skull stripping, normalization, registration and inverse mapping run internally. No prompt is needed. The dispatcher checks DICOM Modality before conversion; that tag does not establish the MR sequence. Review the sequence and field of view before selecting a task. CT modes exclude the publisher's tumour heads. Tiny spurious labels may occur. CTMR weights are restricted to noncommercial research or evaluation.

Each task requires installed assets, a matching validation/<task>.json acceptance marker and an orientation-validation/<task>.json marker. Acceptance must contain task, result: "passed", source_commit, checkpoint_revision and checkpoint_sha256 matching nv_segment_logic.py. Missing, malformed or mismatched markers block selection and ordinary dispatch. The runner also verifies the actual checkpoint, source files and ROI map. The marker is a technical execution gate, not clinical approval.

In local mode, add accepted tasks to ENABLED_MODELS_FILE or ENABLED_EXTERNAL_MODELS. No enabled-models file is changed by this integration. To route any of them to an SSH worker, use the existing comma-separated REMOTE_GPU_TASKS setting. Install this same code and the pinned assets on that worker; its preflight and markers are authoritative. Keep the existing task timeout and process-tree cancellation enabled. Pascal sm_61 compatibility remains untested.

Run the orientation audit only on the chosen GPU worker, with writable marker directories. Set RTSEG_APP_ROOT, RTSEG_MODEL_ROOT, RTSEG_RUNTIME_ROOT, and RTSEG_WORKER_PYTHON (the worker interpreter with DICOM/conversion dependencies). Set RTSEG_ORIENTATION_RUN_ROOT to a fresh directory on project storage outside the model bundle, which is read-only during ordinary jobs. Use a short task-owned RTSEG_SCRATCH_ROOT/TMPDIR such as a mktemp -d /tmp/nvs.XXXXXX directory for multiprocessing sockets, and remove it after the audit. Keep images, copied bundles and logs on project storage.

WORKER_MODE=local REMOTE_GPU_TASKS= "$RTSEG_WORKER_PYTHON" \
  "$RTSEG_APP_ROOT/scripts/run-orientation-audit.py" --nv-segment-acceptance \
  nv_segment_ct_auto nv_segment_ctmr_ct_body \
  nv_segment_ctmr_mri_body nv_segment_ctmr_mri_brain

This operator-only audit option runs fresh checks even if a marker already exists. It passes an explicit bootstrap argument to the local dispatcher; normal portal and SSH jobs do not pass it. Old markers for the audited task are removed before the run. New markers are written only after GPU inference, exact-grid uint16 output, RTSTRUCT conversion/round-trip validation and orientation auditing succeed. Failure returns a nonzero exit status and leaves that task unaccepted. TASK_TIME_LIMIT_SECONDS applies to each inference task; the default is 12 hours. Logs and retained predictions are under RTSEG_ORIENTATION_RUN_ROOT. The audit does not edit the enabled-models list.

The four fixture assignments are in catalog/orientation-audit-fixtures.json. Brain uses the existing public OpenNeuro-derived T1 DICOM fixture; the publisher's NIfTI-only brain example cannot check DICOM contour orientation. No entries are needed in the prompt map for these automatic tasks. Model-agreement, laterality and GPU-compatibility results are recorded in EXTERNAL_MODELS.md.

Viewing job results in the portal¶

Open a job from Jobs to follow its progress. A completed job shows the selected image series and RTSTRUCT overlays alongside Report, Structures and Downloads tabs. Scroll or use arrow keys on the focused image to change slices; drag or use Shift + arrow keys to pan, use +/− to zoom and 0 to reset. Keys 1–4 select soft tissue, lung, bone and brain windows when supplied by the API. Slice indices start at zero. Standard orientation is on by default: axial shows anterior at the top and patient right on the left, coronal shows head at the top and patient right on the left, and sagittal shows head at the top and anterior on the left. The option flips/transposes axis-aligned displays only; oblique edges stay as stored. Four patient-direction markers follow the displayed axes. Switching it off restores stored order. Images and contours share the same transform, including zoom and pan. API pixels and coordinates retain the stored grid.

Overlay controls group ROIs by model. Search, show/hide all, opacity and outline/filled display affect the preview only. “Jump to” selects an acquired slice in the middle of that ROI's available slices. Structures shows approximate volumes only when the report supplies them. The report retains coverage, withheld/partial findings and flags; Open full report includes the original provenance and raw output. The Markdown download is generated from report JSON; the original Markdown is in the result ZIP.

Delete job asks for confirmation and removes the job's images, results and logs. It is disabled while a job is active; it cannot cancel inference. Missing retained images or unsupported geometry can prevent viewing while downloads remain available. The standalone prostate SBRT crop is under More tools. Local UI checks and synthetic previews do not establish clinical readiness.

NPC nodal CTV technical audit¶

lnctvseg_nnunet_v1 has a pinned, strictly loadable nnU-Net v1 checkpoint and job-local model staging. It exports IDs 1–6 unchanged with their published names. The checkpoint also predicts an undocumented class 7, which is omitted without merging or relabelling. Every job records its voxel count and volume in conversion notes, job details and lnctvseg_nnunet_v1/OMITTED_LABELS.txt in the result ZIP, including zero volume. Preflight and execution require GPU acceptance tied to the current manifest digest. SwinUNETR loads strictly but lacks a verified LNCTVSeg preprocessing contract; it and the two UX-Net variants remain informational. The authors distribute weights without a separate weight licence. See the model ledger.

The public lnctvseg-example-hn-ct/dicom fixture covers the head and neck. It is assigned to LNCTVSeg, HNLNL, LN-Seg-FM and compatible H&N OAR tasks in catalog/orientation-audit-fixtures.json. A fixture assignment is not a successful inference result. The case belongs to the authors' development cohort.

For the manager's GPU audit, set RTSEG_MODEL_ROOT, RTSEG_RUNTIME_ROOT and RTSEG_WORKER_PYTHON. The worker interpreter needs pydicom, nibabel, SimpleITK, SciPy, OpenCV and rt-utils. Use a fresh run directory on project storage, WORKER_MODE=local, an empty REMOTE_GPU_TASKS, and a short task-owned TMPDIR such as mktemp -d /tmp/lnctv.XXXXXX. Keep RTSEG_SCRATCH_ROOT and caches on project storage; remove the short temporary directory after execution.

"$RTSEG_WORKER_PYTHON" scripts/run-nodal-gpu-checks.py --run-root "$RTSEG_NODAL_RUN_ROOT"

This operator-only script runs every task in lnctvseg_bundle.INTEGRATED_TASKS (currently lnctvseg_nnunet_v1) and hnlnl_2d3d sequentially through run_task.py --nodal-acceptance-audit. For LNCTVSeg, dice.json compares numeric IDs 1–6 with the checksum-pinned author reference and reports unmapped class 7 separately. Storage-axis differences are aligned; a different physical voxel grid fails. Dice uses raw predictions and is retained if conversion fails. Laterality uses the export copy containing only IDs 1–6. The resolved omission contract permits acceptance markers after all GPU checks pass; stale manifest digests are rejected. Development-cohort Dice is descriptive. The script requires the converter's contour-to-mask round trip, checks each labelled component ≥1 mL against the slice-wise CT body midline in patient LPS coordinates, and checks RTSTRUCT identity and references. Left has positive LPS X. Missing bilateral evidence fails the check; LNCTVSeg must also produce all six grouped labels at ≥1 mL on this fixture. Components below 1 mL are counted but excluded from the laterality decision. This centroid check detects gross side reversal; it does not measure contour accuracy.

The audit records elapsed time and peak task-process GPU allocation sampled every 0.5 seconds. It excludes unrelated processes, but transient peaks between samples may be missed. A failed run removes that task's acceptance and orientation markers. Successful LNCTVSeg markers are bound to the pinned manifest and include laterality evidence. The script does not change the enabled-model list or resolve a weight licence. GPU checks have not been executed by the implementation worker. A cropped CPU smoke and strict loading are separate from this full-fixture GPU audit.

Image libraries and MONAI inversion¶

The Ubuntu 22.04 image installs libxrender1 for libXrender.so.1, needed by OpenCV in the CERR prostate runtime. The Jammy package file list confirms the mapping. The Dockerfile keeps each image-library package on its own line; its regression test maintains the required package set. Rebuild the image to apply this change.

The catalog flag postprocess_on_cpu moves pred to CPU immediately before the first postprocessing inversion/resampling operation. Invertd also returns its tensor on CPU. The network, sliding-window inference and preceding softmax retain their official devices. JSON and YAML bundles receive a job-local JSON compatibility copy; installed bundle files remain unchanged. Host RAM replaces VRAM for multi-channel inversion, with possible numerical differences and longer postprocessing. This does not address GPU memory use in inference or softmax. See the per-bundle decisions in EXTERNAL_MODELS.md.

A manager can compare the production invocation with the flag off and on on an authorized GPU host:

python scripts/compare_monai_postprocessing.py \
  --dicom /fixtures/ct-series --model-root /models \
  --runtime-python /runtimes/envs/lnq/bin/python \
  --output /validation/monai-cpu-comparison \
  --task monai_swin_unetr_btcv_ct

Use a new output directory on project storage and a fixture appropriate to each selected model. Repeat --task for compatible bundles; omitting it selects all six flagged CT bundles; a single fixture may not match every model's anatomical domain. The caller needs nibabel, numpy and PyYAML; --conversion-python can select a Python with SimpleITK. Each run uses a fresh process and the same seed. Official bundle initialization can still enable nondeterministic CUDA kernels.

The script retains predictions, generated configs, commands, per-run logs and summary.json. It prints whole-volume exact voxel agreement and per-label voxel counts, binary voxel agreement and Dice; labels absent in both outputs have Dice 1. Shape or affine disagreement refuses comparison. Peak allocated/reserved MiB are PyTorch CUDA allocator peaks on cuda:0, excluding non-PyTorch allocations. Wall time includes a per-process measurement and a bundle measurement. Per-run wall time includes config preparation and process startup; shared DICOM conversion is excluded. An OOM or other failed baseline is reported as unavailable agreement; flag-on still runs, and the script exits nonzero. Use a smaller suitable fixture if both runs are needed to measure agreement. These results do not establish clinical accuracy.

Task progress and stall recovery¶

The job status panel shows the current model and stage, a structure count when available, job and stage elapsed time, the last activity age, and a model checklist with durations. Expand Live log to read the latest 40 lines. The Jobs list also shows the current stage. Counts describe work completed in that stage; they do not estimate inference time.

Runners emit flushed RTSEG_PROGRESS JSON lines for input staging, DICOM conversion, inference start/end, label post-processing, contour creation, RTSTRUCT writing, geometry validation and export. Per-structure updates are limited to about one per second. The monitor checks every five seconds and also recognizes Predicting, Resampling and Saving messages from existing runners. Local process-tree CPU ticks count as activity even when a task writes no log. SSH tasks currently use log activity only; remote CPU sampling is not implemented.

RTSEG_STALL_SECONDS defaults to 1200 (20 minutes). A task is stalled only when neither its log nor its local process tree shows activity for this interval. The warning remains visible for one monitor interval before recovery. RTSEG_STALL_RETRIES defaults to 1; zero disables the automatic retry. On a stall the portal stops the process tree and, for SSH, the remote task. It records an event, removes only that task's partial output and work directory, and reruns the task from retained input. Its log retains both attempts. Completed task outputs are preserved. A second stall with the default limit fails the job and reports the task and stage. GPU OOM fallback and the absolute TASK_TIME_LIMIT_SECONDS limit retain their existing behavior.

TASK_TIME_LIMIT_SECONDS is one absolute budget across all stall attempts of a task, including time spent stopping and preparing a retry. Attempt start times remain separate for progress reporting.

Before startup dispatches waiting jobs, recovery stops each unmonitored local task tree whose PID and recorded start time both match and confirms termination. A surviving process alone does not count as a live monitor. If termination cannot be confirmed, dispatch remains blocked and startup fails. Remote tasks retain their existing stop and result-recovery path.

Jobs from this version carry an input_manifest key from creation (null until staging completes); if staging did not finish, recovery fails them with the preparation error and manual retry requires a new complete upload. A manifest is never reconstructed from a possibly partial input directory. Jobs created before manifests existed (no input_manifest key, no staging marker) were recorded only after their series was in place and keep the earlier rule: one nonempty image series, matching the recorded slice count when present. Retention removes only finished (complete or failed) jobs older than RETENTION_HOURS; active jobs are left to the watchdog and recovery, which fail or requeue a stuck job first. A job is detached for deletion under the same locks as recovery and retry, so retention never races them.

The watchdog checks at most every 30 seconds. FIFO reservations, including staged jobs awaiting readiness, are live. The running dispatcher job is live while its dispatcher thread is alive. An active job with metadata older than the stall threshold, no FIFO reservation and no live worker uses the interrupted-job recovery path, including the same verified process-tree stop. Recovered jobs enter in original creation order among waiting jobs without reordering existing reservations. Submissions and manual retries receive a new acceptance sequence; a manual retry joins the end of the waiting queue. A running task keeps its queue position throughout its stall retry. Recovery requires complete retained input and allows at most three recoveries (recovery_count). New jobs verify the input manifest, including PET CT companions, before dispatch. Interrupted staging batches are rolled back to the assessed upload when safe; otherwise the job fails with "Interrupted while preparing input; start the job again". This is process-local thread tracking, consistent with the existing single-process queue; it does not add distributed coordination. updated remains a metadata timestamp and must not be interpreted as task activity.

RTSTRUCT comparison retains the Dice threshold (0.95), plane tolerance (0.05 voxel), physical bbox checks and report schema. Expected masks are recreated one ROI at a time from label data or the independent mask file, retaining the voxel box and its source-grid offset. Bounding boxes are found in contiguous 2-D slabs without a full-volume boolean comparison; the contour writer accepts the cropped mask and offset. Rasterization uses bounded per-slice contour boxes and counts pixels outside the expected box as well. No full-volume mask list or reconstructed full volume is retained. For synthetic CPU memory/time evidence run python scripts/benchmark-rtstruct-memory.py --compare; each implementation runs in a fresh subprocess. These checks do not establish clinical accuracy.

BiomedParse v2¶

Install the final layout described in BIOMEDPARSE.md, then run its acceptance command separately on each intended GPU class. The code accepts sm61 (24 GB class) and sm90 (80 GB class); neither has been tested by this CPU-only integration worker. Acceptance markers are device-specific and also bind the exact GPU name, memory, torch version, compiled architectures, source files, checkpoint digest, adapter files, tokenizer and environment lock. No marker is supplied by this integration.

For H100 routing, add biomedparse_prompted to the existing REMOTE_GPU_TASKS setting on the portal and enable it in the remote worker task list. This is a proposed configuration change for the manager; no production configuration was changed. A local P6000 marker never enables the remote H100 route, or conversely. The existing remote preflight timeout stays 900 seconds; checkpoint checks use the persistent held-MRI digest cache.