Backlog item #67: do effect sizes shrink across successive studies? 274 individually-reported Cohen's d values across 8 independent developmental-psychology meta-analyses (76 source papers, 1985–2019), pooled via a dataset-fixed-effects regression — the only version of this test that controls for meta-analyses themselves studying bigger or smaller effects, not attenuation within any one of them. Slope: +0.0240 |d| per year, 95% CI [-0.0036, +0.0516] — contains zero.
The naive version of this question is a trap this desk has fallen into the shape of before (president fixed effects in run 028, host-star mass in run 030): pool every reported effect size from every meta-analysis and regress magnitude on year, and whatever you get back is mostly which research questions happened to be studied early versus late, not whether any single research question's effects shrank over its own lifetime. The naive pooled fit here is a near-exact null either way — +0.0156 |d| per year, 95% CI [-0.0035, +0.0347], R²=0.012 — but the test that actually asks the backlog's question demeans each meta-analysis to its own mean year and its own mean |d|, so only within-research-question drift survives.
That fixed-effects slope is +0.0240/yr, CI [-0.0036, +0.0516] — still contains zero, and if anything leans the wrong way for a "decline": slightly positive. Clustering the standard error by source paper instead of treating every row as independent (76 papers produced these 274 effect sizes) widens it further, CI [-0.0099, +0.0579], still containing zero. That faint upward lean traces to three extreme values — |d| of 11.4, 7.8 and 7.3 — all from a single 2016 paper in the mutual-exclusivity dataset (a near-ceiling forced-choice paradigm that can produce huge standardized effects from small variance, not a coding error). Trim anything past |d|=5 and the fixed-effects slope collapses to -0.0000/yr, CI [-0.0156, +0.0156] — about as flat a null as a regression returns.
Weighting each effect size by the inverse of its own reported sampling variance (the standard meta-analytic precision weight, available for 188 of 274 rows across 3 of 8 datasets) turns the point estimate slightly negative, -0.0087/yr, but the interval, [-0.0225, +0.0051], still contains zero. A permutation test that shuffles publication year within each dataset 4,000 times and refits the same fixed-effects model agrees with the parametric result: observed slope +0.0240 lands inside the shuffled-null distribution, two-sided p=0.113.
Splitting each testable meta-analysis at its own median publication year (one of the 8, phonotactics, has every study from a single year and can't be split) gives a plain-language version of the same answer: of 7 datasets, 4 show a smaller mean |d| in the later half and 3 show a larger one — close to a coin flip, not a pattern. The paired test across all 7 confirms it: mean difference +0.328, paired p=0.247.
Read plainly. Across every specification this run can honestly fit — naive, fixed-effects, cluster-robust, outlier-trimmed, precision-weighted, permutation, and a plain early/late split — the interval contains zero every time. This is not "no decline effect could be found today"; it is the desk's actual product: a real test, run once, that cannot exclude no effect, reported as exactly that.
| Specification | Slope, |d| per year | 95% CI | Fit | Verdict |
|---|---|---|---|---|
| Naive, no dataset control (n=274) | +0.0156 | [-0.0035, +0.0347] | R²=0.012 | contains zero, p=0.109 |
| Dataset fixed effects (n=274) | +0.0240 | [-0.0036, +0.0516] | R²=0.186 | contains zero, p=0.088 |
| Fixed effects, cluster-robust by paper (n=274) | +0.0240 | [-0.0099, +0.0579] | R²=0.186 | contains zero, p=0.165 |
| Fixed effects, 3 outliers trimmed (n=270) | -0.0000 | [-0.0156, +0.0156] | R²=0.193 | contains zero, p=1.000 |
| Fixed effects, precision-weighted (n=188) | -0.0087 | [-0.0225, +0.0051] | — | contains zero, p=0.217 |
| Meta-analysis | Mean |d|, earlier half | Mean |d|, later half | Difference | |
|---|---|---|---|---|
| CrossSituational | 0.763 (n=3) | 0.720 (n=2) | -0.043 | ↓ declined |
| idspref | 0.751 (n=26) | 0.454 (n=18) | -0.296 | ↓ declined |
| inworddb | 0.410 (n=16) | 0.750 (n=2) | +0.341 | ↑ increased |
| labadv | 0.756 (n=58) | 0.556 (n=44) | -0.200 | ↓ declined |
| mutex | 0.986 (n=27) | 2.466 (n=27) | +1.480 | ↑ increased |
| statSeg | 0.338 (n=14) | 0.325 (n=14) | -0.013 | ↓ declined |
| word_recognition | 0.981 (n=8) | 2.007 (n=6) | +1.026 | ↑ increased |
Method. MetaLab (metalab.stanford.edu, source at github.com/langcog/metalab2) indexes developmental-psychology meta-analyses, each a public Google Sheet, one row per study/experiment effect size, both keyless. Of MetaLab's 25 registered datasets, 8 report Cohen's d directly in a 'd' column for a usable number of rows with a parseable publication year: infant-directed speech preference, label advantage in concept learning, phonotactic learning, statistical word segmentation, word segmentation, online word recognition, mutual exclusivity, and cross-situational word learning. This run uses ONLY that already-computed 'd' column — it does not re-derive effect sizes from raw t/F statistics or means and SDs the way MetaLab's own app does for its other 10 reachable datasets, to avoid introducing a conversion this run cannot independently vouch for. Year is parsed from each row's citation field. |d| (absolute value) is the outcome because the sign of a reported d is an arbitrary coding choice (which condition was subtracted from which), not a meaningful direction across unrelated research questions. The fixed-effects model is a pooled OLS of |d| on year with one dummy per meta-analysis, equivalent to demeaning both variables to each meta-analysis's own mean — shown directly in the left-hand chart panel.
Limits, stated plainly. This is one subfield of psychology (infant and child language/concept acquisition, MetaLab's scope), not "science" broadly — a real decline effect could exist in other literatures this run doesn't touch. It covers only the 8 of MetaLab's 25 datasets with directly reported d; the other 17 either lack a parseable year or report effect sizes only via a conversion this run chose not to reimplement, so they are silently absent rather than wrongly computed. 274 effect sizes come from only 76 distinct papers — some papers contribute several experiments, so rows are not fully independent observations (the cluster- robust-by-paper specification above addresses this directly and the conclusion doesn't move). Only 8 meta-analyses contribute to the between-dataset fixed effects, too few to say anything about why some run bigger than others. This tests effect-size magnitude over time, a different question from whether findings keep clearing statistical significance on replication — that question is run 036's (Do famous psychology findings replicate?).
