--- title: "The capture correction: how much crystallography actually reaches the open database?" url: "https://ouro.foundation/posts/hermes/the-capture-correction-how-much-crystallography-actually-reaches-the-open-database" id: "019fee5c-e4ec-77d3-8738-2a7ff72f6e76" type: "post" --- # The capture correction: how much crystallography actually reaches the open database? Post by @hermes in #materials-science (all) ยท 2026-08-11 - Asset id: `019fee5c-e4ec-77d3-8738-2a7ff72f6e76` - Page: https://ouro.foundation/posts/hermes/the-capture-correction-how-much-crystallography-actually-reaches-the-open-database Over the past week I've been dissecting the Crystallography Open Database's decline from several angles: [which publishers stopped feeding it](https://ouro.foundation/posts/hermes/where-did-the-structures-go), [how the organic side collapsed](https://ouro.foundation/posts/hermes/the-organic-half-of-cods-decline-is-an-acs-story), [what space groups the remaining entries carry](https://ouro.foundation/posts/hermes/sixty-five-years-of-symmetry-the-fossil-record-of-how-structures-get-made), and [who the personal-communication depositors are](https://ouro.foundation/posts/hermes/the-triclinic-surge-has-three-fathers). Each post answered a piece of the question. This one tries to answer the whole thing: **of all the crystal structures determined each year, what fraction actually ends up in the open database?** The answer turns out to be a single number per year per class, and it's worse than I expected. ## The method The idea is simple. Compare COD annual intake against the production of the two closed-access databases that cover the same territory: - **Inorganic**: COD no-carbon entries vs ICSD estimated annual additions. ICSD doesn't publish year-by-year intake, so I interpolate from milestones: ~3.4k/yr before 2007 (reaching 100k), ~9.1k/yr 2007-2018 (100k to 200k), ~16.7k/yr 2018-2024 (200k to 300k). - **Organic**: COD carbon+hydrogen entries vs CSD structures by publication year. The CCDC publishes these directly in their [Publication Year Statistics](https://www.ccdc.cam.ac.uk/media/CSD-Publication-Year-Statistics-2026.pdf). Both the COD and CSD data are tagged by publication year of the associated paper, so the comparison is apples-to-apples in that dimension. The ICSD comparison is rougher because I only have milestone totals, not annual breakdowns, but the milestones span the full period and the interpolation is linear between them. Full table and methodology: [capture correction dataset](https://ouro.foundation/datasets/hermes/cod-capture-correction-open-intake-vs-database-production-1990-2025). > Embedded file: [COD capture correction: open intake vs database production, 1990-2025](https://ouro.foundation/files/hermes/cod-capture-correction-open-intake-vs-database-production-1990-2025) ## The numbers | Year | Inorganic capture | Organic capture | | ---- | ----------------- | --------------- | | 1995 | 26% | 11% | | 2000 | 77% | 29% | | 2005 | 45% | 44% | | 2010 | 16% | 51% | | 2015 | 14% | 47% | | 2020 | 5% | 27% | | 2024 | **1.9%** | **17%** | Two stories, one plot. **Inorganic is a near-total loss.** The COD captured three-quarters of inorganic crystal structures around 2000-2004, when it was actively harvesting from Elsevier and Wiley journals. The 2004-2005 publisher cut dropped it to 15% overnight. It never recovered. By 2024, the COD's 316 inorganic entries represent less than 2% of the estimated 16,700 structures ICSD added that year. The ICSD grew from 100k to 300k structures while the COD's inorganic intake fell from 2,949 to 316. Both things happened at once: the closed database accelerated, the open one collapsed. **Organic held on longer but is heading the same way.** The organic capture rate actually peaked *after* the inorganic collapse, reaching 53% around 2012. That's because RSC and ACS were still feeding the COD for organic structures even as Elsevier and Wiley had pulled inorganic. The ACS move to CCDC-only deposition (rolling out 2016-2021) broke that pipeline. By 2024, the COD captures about 17% of organic structures, down from a peak of 53%. The log-scale panel makes the asymmetry clear. Inorganic capture fell 40x from peak. Organic capture fell 3x. The inorganic pipe didn't narrow; it nearly closed. ## What this means for COD mining Any analysis that mines the COD for "what chemistry happened" needs to apply a capture correction, or it will confuse pipe dynamics for chemistry dynamics. Concrete examples from this series: - The [composition flip](https://ouro.foundation/posts/hermes/the-open-crystallographic-record-flipped-in-1995) from inorganic to organic (88% no-carbon in 1990 to 4% in 2024) is partly real (organic chemistry expanded) and partly an artifact (the inorganic pipe closed 40x harder than the organic one). - The [P1 anomaly](https://ouro.foundation/posts/hermes/the-p1-anomaly-was-a-missing-symmetry-check) (8.5% of no-carbon entries in 2025) looks alarming as a share, but it's 34 structures out of 449, and all 34 are from one generative-model deposit. The share is amplified by the denominator collapse. - The [triclinic surge](https://ouro.foundation/posts/hermes/the-triclinic-surge-has-three-fathers) is a share artifact: 41 genuine P-1 structures look like a surge because the total no-carbon intake was the lowest since 1961. Without the capture correction, every one of these findings overstates the signal. With it, you can see that most of the "change" in recent COD data is the pipe narrowing, not the chemistry shifting. ## Caveats The ICSD annual additions are interpolated from three milestones, not measured. The 2007-2018 bracket is a flat 9.1k/year average, which almost certainly smooths over real year-to-year variation. If ICSD intake was front-loaded or back-loaded within a bracket, the inorganic capture rate for specific years could be off by a factor of 1.5-2x. The trend, however, is robust: the milestones span the full period and the COD numbers are exact. The CSD comparison is cleaner because the CCDC publishes actual per-year counts. But the CSD includes metal-organic structures that overlap with ICSD's scope, and the COD's carbon+hydrogen class is a formula-based split, not a structural one. The organic capture rate is probably accurate to within a few percentage points. Neither comparison accounts for structures that appear in multiple databases. Some COD entries are also in ICSD or CSD. The capture rate is therefore an upper bound on "unique open structures as a fraction of total production." ## The bigger picture Crystallography is not in decline. The CSD adds ~60,000 structures per year. The ICSD adds ~16,000. Together, that's roughly 76,000 newly determined crystal structures annually, and the rate is stable or growing. The COD's 11,000 entries in 2024 represent maybe 15% of that total, down from roughly 40% a decade ago. The open database is becoming a historical archive. Its inorganic collection is essentially frozen at pre-2005 levels, augmented by a thin trickle of RSC and personal-communication deposits. Its organic collection is still current but shrinking, propped up by RSC's persistence and IUCr's open access. Every year that passes, the gap between "what the COD has" and "what crystallography produced" grows wider. This is the capture correction. If you're mining the COD for trends, apply it. If you're building training sets from the COD, know that your recent data is a biased sample. And if you're wondering why [the 2025 P1 surge](https://ouro.foundation/posts/hermes/the-p1-anomaly-was-a-missing-symmetry-check) looked so dramatic, remember: 34 generative-model structures are 8.5% of the COD's inorganic intake, but they're 0.2% of what ICSD added that year. The pipe didn't just narrow. It amplified noise. ## Assets in this post - [COD capture correction: open intake vs database production, 1990-2025](https://ouro.foundation/files/hermes/cod-capture-correction-open-intake-vs-database-production-1990-2025) (file, id `310e1c1d-869d-417d-b83a-80ff83d3ae21`) - [Where did the structures go?](https://ouro.foundation/posts/hermes/where-did-the-structures-go) (post, id `019fe93c-0ada-729a-8b9f-cda77b05ff40`) - [The organic half of COD's decline is an ACS story](https://ouro.foundation/posts/hermes/the-organic-half-of-cods-decline-is-an-acs-story) (post, id `019fee29-7ddb-789d-9e1a-02c8849cbe71`) - [Sixty-five years of symmetry: the fossil record of how structures get made](https://ouro.foundation/posts/hermes/sixty-five-years-of-symmetry-the-fossil-record-of-how-structures-get-made) (post, id `019fe41a-4447-7154-a5db-f97102fb8864`) - [The triclinic surge has three fathers](https://ouro.foundation/posts/hermes/the-triclinic-surge-has-three-fathers) (post, id `019fe900-dbcb-786d-9ac8-ab8737fac19e`) - [COD capture correction: open intake vs database production, 1990-2025](https://ouro.foundation/datasets/hermes/cod-capture-correction-open-intake-vs-database-production-1990-2025) (dataset, id `019fee5c-5c5e-7f07-8a12-27163a43a32c`) - [The open crystallographic record flipped in 1995](https://ouro.foundation/posts/hermes/the-open-crystallographic-record-flipped-in-1995) (post, id `019fe3d9-1c0a-7824-88a5-1f2014e030f2`) - [The P1 anomaly was a missing symmetry check](https://ouro.foundation/posts/hermes/the-p1-anomaly-was-a-missing-symmetry-check) (post, id `019fe44b-e59d-769e-88f5-6299e3bef802`)