Technical report · Evaluation
What the cohort packs can and cannot do
Two measured results on the same evidence packs — one positive, one negative — and the control arms that make either of them interpretable.
Abstract
We evaluate whether compiled cohort evidence packs carry real information about real audiences, on two tasks with different structure. On forecasting — will a member receive reactions after a temporal cutoff — the pack scores 0.5600 balanced accuracy against 0.6440 for a trivial popularity rule. The pack loses. On attribution — which of two matched comments was written by a member of a named cohort — it scores 0.7500 [0.7169, 0.7804] against 0.4930 for a model told nothing about the cohort. We argue the difference is structural rather than contradictory: the forecasting outcome sits almost on top of a feature every arm can already see, so it rewards counting prior activity, whereas attribution is built so prior activity cannot help. Attribution is recognition, and recognition does not imply prediction. No cross-model replication has been validated, so nothing here should be quoted as a product accuracy figure.
01Why four arms, and why the trivial one runs first
An accuracy figure alone is uninterpretable. The question is never “how often is it right” but “how often is it right compared with the cheapest thing that could have produced the same answer”. So every evaluation runs four arms over identical items, and the trivial arm runs first, before any spend — because if a rule with no model in it clears the task, nothing that follows means anything.
| Arm | What it sees | Why it exists |
|---|---|---|
| A | Full evidence pack — 8,025 chars median | The thing under test |
| B | Cohort title and build provenance — 329 chars | The industry default: a name, not evidence |
| F | Nothing about the cohort | Catches a regularity nobody thought to test for |
| T | No model at all — popularity, length, position, language, cohort size | Proves the framing does not leak the answer |
Arm F matters most. Popularity and length are channels we thought of; F is the one that catches the channels we did not.
02Result 1 — attribution, and it is positive
Two real comments side by side. One was written by a member of cohort C. Which? The distractor is matched to the true comment on platform, language, like-count bucket, length band, and answer position is balanced exactly 50/50.
Three things follow, in order of importance. The framing leaks nothing — every null, including a model given no cohort information at all, is at chance. Naming the cohort is worth something: 329 characters of title and provenance reach 0.6417. And the compiled evidence is worth substantially more than the name, with the intervals separated. The advantage is broad rather than driven by a few cohorts: mean per-cohort delta +0.108, median +0.083, and arm A falls below chance in 3 of 120 cohorts against 20 for B and 46 for F.
Two controls that cannot fail, and so are not evidence
The reported same-language rate of 1.0 is tautological — language is part of the match key, so it could not come out otherwise. T_language is likewise circular: with language matched its tie-branch always fires, making it arithmetically identical to T_first. The real evidence that the language channel closed is external — a modal-language oracle scored 0.7236 on the unmatched bank and exactly 0.5000 on this one.
03Result 2 — forecasting, and it is negative
The same packs, on the task the product actually sells: prediction. A temporal split at 1 May 2026. Packs are recompiled strictly from content published before the cutoff; membership comes from before it; the label comes from after it.
Why the trivial rule is so strong here: the outcome sits close to the feature. “Did this member receive reactions after May” tracks “how many did they receive before”. That is persistence of activity, not audience psychology. The task is a weak test of what a pack is for and a strong test of whether the harness is honest. It passed the second one. At n = 500 against roughly 1,570 needed to detect a five-point effect at 80% power, this null is underpowered rather than absent.
04Result 3 — the abstention channel is anti-informative
Arm A was offered “insufficient evidence” as a third outcome, then the forced-choice arm was scored on exactly the items it had declined.
| Measure | Value |
|---|---|
| Answer rate | 0.967513 of 400 declined |
| Forced-choice accuracy on answered items | 0.5375 |
| Forced-choice accuracy on declined items | 0.6154higher |
| Abstention informativeness | −0.0779wrong sign |
It declines on the questions it would have got right. Acting on that channel would remove the model’s better predictions, so it must not ship. Only 13 declines, so the direction is clear and the magnitude is not. Separately, confidence is monotonic against accuracy across all four bins — 0.500, 0.521, 0.571, 1.000 — and is worth surfacing even though the binary decline is not.
Read the method before you trust the number.
If you want the full report — including the run that was thrown away after a leakage bug invalidated it — ask.