Julenmendieta/milab 6793 last tweaks

.changeset/fitted-background-equal-width-bins.md

@@ -0,0 +1,43 @@
+---
+'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor
+'@platforma-open/milaboratories.feature-integration.ui': minor
+'@platforma-open/milaboratories.feature-integration.model': patch
+---
+
+The fitted background draws the distribution the run holds
+
+The fitted-background grid is the only way to see whether a tag's counts separated into two
+populations, so a hump it invents, or one it hides, is the error this surface cannot carry. Three
+things were wrong with it.
+
+**The zeros were missing.** The plot was binned from the sparse counts frame, which has no rows for
+cells that read nothing. Those zeros are most of the background, so the plot showed one decaying hump
+whatever the fit had found — the left half of the distribution was simply absent. It is now binned over
+the cells the fit actually ran on, one entry per cell in the sample, and a cell that read nothing counts
+as a zero.
+
+**Every bar is now the same width.** Bins sit at `expm1(k * 0.2)`, uniform in `log1p`, which is what
+the plot's axis already is. The source paper histograms `log1p` counts at a fixed width for the same
+reason.
+
+Before, bins were whole numbers stepping geometrically, so their widths ran from 0.301 of a decade at
+`[0, 1)` down to 0.079 at `[4, 5)`. A raw count then made a wide bar stand above a narrow one holding
+the same density, so each bar had to be divided by its own width — and dividing by the wrong width hid
+a real signal component completely: on a mixture whose upper mode was cleanly separated at a mean of
+60, that hump drew at 0.9% of the background peak. Equal widths remove the division and the error with
+it. Bar height is a plain cell count again, so the hover readout reports a cell count.
+
+The cost of equal widths is that the edges are no longer whole numbers, and counts are — consecutive
+integers sit further apart than one bin until about count 13, so the low end is a comb of separated
+bars. The paper's own figures show the same gaps. `LOG1P_BIN_WIDTH` is coarser than the paper's 0.075
+for that reason: at 0.075 a real tag came back with 75 of 97 bins empty.
+
+**The bound line is drawn.** Under a fitted baseline the threshold is a probability, so a plot in counts
+had nothing to mark it against. Each fit now resolves the count at which the run's bound probability
+starts calling a cell bound, and the panel draws it. A fit that reaches no such count draws no line and
+says so, rather than marking one at the bottom.
+
+Each (sample, tag) is binned against its own range, so the emitted weight lists have different lengths
+and are shorter than the shared edge set. The plot pads them, which draws the same picture — past a
+pair's own maximum every bar is empty either way. Binning inside the fit rather than returning per-cell
+arrays takes the fitting step from 2.1 million numbers held to 7,764 on a 27-sample run.

.changeset/fitted-baseline-settings.md

@@ -0,0 +1,44 @@
+---
+'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor
+'@platforma-open/milaboratories.feature-integration.workflow': minor
+'@platforma-open/milaboratories.feature-integration.model': minor
+'@platforma-open/milaboratories.feature-integration.ui': minor
+'@platforma-open/milaboratories.feature-integration.test': patch
+'@platforma-open/milaboratories.feature-integration': minor
+---
+
+The fitted baseline states where it starts, and the scientist can move it
+
+Two settings appear under the fitted baseline, and nowhere else — neither reaches the declared or
+panel rung, so neither is offered there.
+
+**Expected binder %.** Roughly what share of cells are expected to bind an antigen. The fit splits the
+counts at the matching quantile and seeds one component from each side. It is not a threshold: the EM
+re-estimates both components from there, so the split the run ends up with is an output of the fit.
+
+It changes answers anyway, because the EM is not globally convergent on these distributions and the
+start decides which optimum it reaches. On a panel where 27% of cells really did bind, the shipped
+value put the split at 953 counts; told 30%, the same fit put it at 13 — which is where the gap in that
+tag's histogram actually is. The published value comes from a rare-binder regime, and the study behind
+this rung never tested a positive fraction above 25%.
+
+The trade runs one way, so no single value is right: raising it also makes the fit readier to carve a
+signal component out of a single population, so a tag that bound nothing invents more binders. Only
+the scientist knows which side of that to be on, which is why it is a setting.
+
+**Bound probability.** How sure the fit must be before a cell counts as bound. Previously fixed at
+0.9 with no way to see or move it. Now shown, with 0.9 as both the default and the lowest accepted
+value — below it a cell holding none of a tag could cross the line, and the run counts those cells by
+arithmetic rather than reading each one, so the two halves would disagree with nothing raised.
+
+**The fit now starts where the method says.** The split was taken at the median, which
+`what-plays-the-baseline` never specified. A median start begins from two halves of equal size, which
+is far from the truth on a mostly-background population — every tag here — and pulls the fit toward
+calling much of that background signal. On a control reagent, whose counts hold one population, a
+median start gives a background weight near 0.8 against 0.95 from the published split.
+
+That trade is not free, and the direction is recorded in the suite: on a background whose long tail
+puts its mean above the binders', the published start decomposes the counts into the bulk and the
+tail rather than into background and binders, and calls the tail the signal. The median start got that
+shape right and the mostly-background case wrong instead. Neither wins both. The run gives no warning
+in either case, which is why the fitted grid puts both means in front of the reader.

.changeset/per-cell-tag-counts-export.md

@@ -0,0 +1,24 @@
+---
+'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor
+'@platforma-open/milaboratories.feature-integration.workflow': minor
+'@platforma-open/milaboratories.feature-integration': minor
+---
+
+Per-cell antigen counts are exported, before the minimum is applied
+
+A new export, keyed `[sampleId, cellId, tagId]`, carrying the UMI count each cell held for each
+barcode. It is what a downstream per-cell composition plot needs: one bar per cell, split by antigen.
+
+**Before the count minimum, and that is the point.** The counts the verdicts are computed on have
+already had every value below the minimum set to zero, and the comparator tag is exempt from that —
+so on a declared-baseline run the control keeps its small counts while the antigens lose theirs. Those
+numbers answer "what counted as evidence of binding". This export answers "what did the cell capture",
+so a cell's tags add up to what that cell actually held. The column's own description says so, because
+the two do not reconcile and a reader who mixes them draws the wrong conclusion.
+
+It cannot be derived from the floored counts afterwards: once the minimum has run, a count of 3 and a
+count that was never there are both 0.
+
+Partitioned by sample, the only column in the block that is. It is the largest table the run produces,
+at one row per (cell, tag), and a composition plot reads one sample per view — so a reader after one
+sample touches one partition instead of the whole run.

.changeset/run-quality-grid-per-sample.md

@@ -0,0 +1,38 @@
+---
+'@platforma-open/milaboratories.feature-integration.ui': minor
+'@platforma-open/milaboratories.feature-integration.workflow': patch
+'@platforma-open/milaboratories.feature-integration.model': patch
+---
+
+The fitted-background grid reads one sample at a time
+
+The fit runs per (sample, tag), so the grid drew one panel per pair — 27 samples and 9 barcodes is 243
+panels on one page, and every title had to repeat the sample to tell them apart. It now shows one
+sample, chosen from a selector above it, so a panel is titled by its reagent alone and the sample is
+named once. Barcodes read in the order the panel file declares them, so a barcode holds the same slot
+whichever sample is shown.
+
+What that gives up is reading down one reagent across samples. The grid still supports that shape;
+nothing asks for it today.
+
+Each panel's caption now carries the bound count alone, and the fit's own numbers moved to the enlarged
+panel, where a value is read rather than scanned. The cell count that used to lead the caption was the
+sample's analysed population — the same number on every panel of the sample — and now sits once above
+the grid.
+
+**The Panel column is out of the tables.** It is a hash of the sorted barcode list, because no panel
+file names its panel, and a run declaring one panel for every sample repeated that hash identically on
+every row. It stays available in the column picker, and a multi-panel run should switch it on:
+`Seen in 2/3` cannot be read without knowing which three samples.
+
+**Fixes**
+
+- A fitted background mean of 0.000488 printed as `0`, a value the fit cannot produce — three
+  significant figures were computed and then discarded by a formatter keeping three decimal places.
+- The Run quality page failed to render at all: a watch read a value declared further down the file.
+- A run computed before the bound count existed reported "no count reaches the bound probability",
+  stating a finding no run had produced. Absent and null now read differently.
+- Resizing the window redrew every panel on every frame, and each redraw leaks a tooltip node in the
+  uikit. A few pixels of tolerance takes a drag from hundreds of redraws to a handful. The leak itself
+  is the uikit's.
+- The quality-report JSON was pretty-printed, which roughly doubled it for a file only the UI reads.

.changeset/run-quality-tabs-hidden-while-running.md

@@ -0,0 +1,14 @@
+---
+'@platforma-open/milaboratories.feature-integration.ui': patch
+---
+
+Run quality hides its tab strip while the run computes
+
+The tab set is derived from the baseline rung the run reports. Until the run
+settles that rung is unknown, so the strip offered every plot and then dropped
+the ones the served rung cannot draw — a reader could open a tab that then
+stopped existing.
+
+The strip is now hidden while the block computes. The open view's body keeps
+rendering and draws its own processing placeholder, so the page still shows the
+run's progress.

.changeset/undeclared-status-per-barcode.md

@@ -0,0 +1,29 @@
+---
+'@platforma-open/milaboratories.feature-integration.per-cell-metrics': minor
+'@platforma-open/milaboratories.feature-integration.workflow': minor
+'@platforma-open/milaboratories.feature-integration.model': minor
+'@platforma-open/milaboratories.feature-integration': minor
+---
+
+The undeclared-barcode Status judges each barcode, not the whole sample
+
+The Status column read the sample's aggregate undeclared share, so it was one
+word repeated down every row of a sample. A sample carrying one heavy
+undeclared sequence among many light ones said nothing about which sequence to
+look at.
+
+Status now reads each row's own share of its sample's pre-refine reads. It warns
+above 1% and alerts above 5%. Those two numbers are operator-set and
+overridable, not inherited: the field publishes 0.50/1.0 for a sample's
+AGGREGATE undeclared share, and that line does not transfer to one sequence,
+because an aggregate reaches 0.50 while no single sequence comes near it.
+
+The alert end changed direction with it. It compared for equality, which fired
+only at exactly the error threshold and let every larger share read *warn* — the
+worse finding being the one that never showed. Both ends now face the same way.
+
+The sample-level share keeps its column, renamed to "Sample Undeclared (%)", and
+carries no status.
+
+Every column description in that table was rewritten to one instruction per
+sentence, active voice, and short sentences.

block/package.json

@@ -18,7 +18,7 @@
   },
   "scripts": {
     "build": "ts-builder build --target block-facade && block-tools pack",
-    "prepublishOnly": "block-tools publish -r s3://milab-euce1-prod-pkgs-s3-block-registry/pub/releases/?region=eu-central-1 --registry-serve-url https://blocks.pl-open.science",
+    "prepublishOnly": "block-tools publish --unstable -r s3://milab-euce1-prod-pkgs-s3-block-registry/pub/releases/?region=eu-central-1 --registry-serve-url https://blocks.pl-open.science",
     "do-pack": "shx rm -f package.tgz && pnpm pack && shx mv *.tgz package.tgz",
     "check": "ts-builder type-check --target block-facade"
   },

model/src/index.ts

@@ -6,15 +6,15 @@ import type {
 } from "@platforma-sdk/model";
 import {
   BlockModelV3,
-  createPlDataTableStateV2,
   createPFrameForGraphs,
+  createPlDataTableStateV2,
   createPlDataTableV2,
   createPlDataTableV3,
   DataColumn,
   DataModelBuilder,
+  getAxisId,
   isPColumnSpec,
   parseResourceMap,
-  getAxisId,
 } from "@platforma-sdk/model";
 import { assemblePattern, CELL_TAG, FEATURE_TAG, UMI_TAG, validatePattern } from "./pattern";
 import { getPreset } from "./presets";
@@ -32,15 +32,17 @@ export type { PTableKey } from "@platforma-sdk/model";
 
 // The reading's own defaults, in one exported map. Exported so a test can compare it against the other
 // two copies: this map is what a workflow-driven run is actually answered under, because
-// verdict-args.lib.tengo emits every one of these flags UNCONDITIONALLY, substituting its own copy
-// wherever the stored value is undefined. The argparse defaults in the Python never govern such a run.
+// verdict-args.lib.tengo emits these flags UNCONDITIONALLY, substituting its own copy wherever the
+// stored value is undefined. The argparse defaults in the Python never govern such a run.
 // `test/src/qcDefaults.test.ts` asserts each value against verdict-args.lib.tengo and the Python module
 // that owns it, the same way it asserts the QC lines below.
 export const VERDICT_DEFAULTS = {
   // verdict.py DEFAULT_FLOOR
   countFloor: 4,
   // verdict.py BOUND_CUTOFF
   boundCutoff: 75,
+  // verdict.py DISTRIBUTION_BOUND_PROBABILITY. Both the default and the FLOOR: `args()` refuses below it.
+  boundProbability: 0.9,
   // combine.py DEFAULT_MIN_VOTERS
   minVotingCells: 1,
   // verdict.py DEFAULT_PANEL_MIN_MEMBERS. Gates rather than tunes: keep above the fifteen-tag cap of an
@@ -64,8 +66,8 @@ export const QC_LINE_DEFAULTS = {
   readsPerCellWarn: 5000,
   aggregateBarcodeWarn: 0.05,
   aggregateBarcodeError: 1.0,
-  undeclaredBarcodeWarn: 0.5,
-  undeclaredBarcodeError: 1.0,
+  undeclaredBarcodeWarn: 0.01,
+  undeclaredBarcodeError: 0.05,
   usableReadWarn: 0.2,
   usableReadError: 0.0,
 } as const;
@@ -215,6 +217,11 @@ export type TagCountBins = {
    * label column reaches p-frame surfaces only. Absent in full for a run that finished before this key.
    */
   tagLabels?: Record<string, string>;
+  /**
+   * The barcodes in the order the PANEL declares them, deduplicated on first appearance. A per-sample view
+   * reads in it, so that a barcode holds the same slot in every sample.
+   */
+  tagOrder?: string[];
   /**
    * The fit's two means and the background's share of cells, at the same (sample, tag) grain as the bins.
    * Here rather than in the p-frame beside them, so a grid of panels costs no driver query per panel.
@@ -224,7 +231,15 @@ export type TagCountBins = {
    */
   fitsBySample: Record<
     string,
-    Record<string, { backgroundMean: number; signalMean: number; backgroundWeight: number }>
+    Record<
+      string,
+      {
+        backgroundMean: number;
+        signalMean: number;
+        backgroundWeight: number;
+        boundAtCount?: number | null;
+      }
+    >
   >;
   /**
    * The run's own two spreads, each on its own LINEAR edges: `score` and `referenceReading`. Linear, unlike
@@ -505,8 +520,10 @@ type BlockDataV2 = Omit<
   | "distributionMinCells"
   | "countFloor"
   | "boundCutoff"
+  | "boundProbability"
   | "minVotingCells"
   | "minAgreement"
+  | "expectedBinderFraction"
   | "gateThreshold"
   | "grouping"
   | "contendingGroups"
@@ -710,6 +727,13 @@ export const platforma = BlockModelV3.create(dataModel)
       );
 
     if (data.countFloor < 0) throw new Error("The count floor cannot be negative");
+    if (
+      typeof data.boundProbability === "number" &&
+      (data.boundProbability < VERDICT_DEFAULTS.boundProbability || data.boundProbability > 1)
+    )
+      throw new Error(
+        `The fitted baseline's probability is at least ${VERDICT_DEFAULTS.boundProbability} and at most 1`,
+      );
     if (data.boundCutoff < 0 || data.boundCutoff > 100)
       throw new Error("The bound cutoff is a score between 0 and 100");
     if (data.minVotingCells < 1) throw new Error("At least one cell must vote");
@@ -722,6 +746,13 @@ export const platforma = BlockModelV3.create(dataModel)
       (data.minAgreement <= 0.5 || data.minAgreement > 1)
     )
       throw new Error("The agreement floor is a share above 50% and at most 100%");
+    // Strictly inside (0, 1). At either end the split hands every cell to one side and the fit silently
+    // falls back, so the run would record a fraction it never used.
+    if (
+      typeof data.expectedBinderFraction === "number" &&
+      (data.expectedBinderFraction <= 0 || data.expectedBinderFraction >= 1)
+    )
+      throw new Error("The expected binder fraction is a share above 0% and below 100%");
     // The cell condition GATES the fitted rung rather than tuning it, so it is a real population size.
     if (data.distributionMinCells < 1)
       throw new Error("A fitted baseline needs at least one cell to be fitted over");
@@ -855,6 +886,8 @@ export const platforma = BlockModelV3.create(dataModel)
       distributionMinCells: Math.round(data.distributionMinCells),
       countFloor: Math.round(data.countFloor),
       boundCutoff: data.boundCutoff,
+      boundProbability: data.boundProbability,
+      expectedBinderFraction: data.expectedBinderFraction,
       minVotingCells: Math.round(data.minVotingCells),
       // Off by default, and off means ABSENT. A minimum agreement of 0 passes every majority instead of skipping
       // the check. A gate of 0 sets aside every cell instead of gating none.

model/src/types.ts

@@ -90,6 +90,8 @@ export type BlockArgs = {
   distributionMinCells: number; // cells a sample needs before the rung may serve
   countFloor: number; // counts below this are not evidence of binding
   boundCutoff: number; // specificity score (0-100) at or above which a cell binds
+  boundProbability?: number; // the probability a count belongs to the signal component at or above which a cell binds
+  expectedBinderFraction?: number; // the share of cells expected to bind, seeding the fitted rung's split
   minVotingCells: number; // a verdict may rest on one cell and say so
   // Share (0-1) of answering cells the majority must reach. Off by default, and off means ABSENT rather
   // than zero: a floor of 0 passes every majority instead of skipping the check.
@@ -188,6 +190,8 @@ export type BlockData = {
   distributionMinCells: number;
   countFloor: number;
   boundCutoff: number;
+  boundProbability?: number;
+  expectedBinderFraction?: number;
   minVotingCells: number;
   minAgreement?: number;
   gateThreshold?: number;