Every CIF you download from the Crystallography Open Database starts with four comment lines that most parsers throw away:
#------------------------------------------------------------------------------ #$Date: 2016-02-18 17:37:37 +0200 (Thu, 18 Feb 2016) $ #$Revision: 176729 $ #$URL: file:///home/coder/svn-repositories/_RELOADED_COD/cod-reloaded/cif/1/00/00/1000000.cif $ #------------------------------------------------------------------------------
The COD is a Subversion repository served over HTTP, and those are expanded SVN keywords. Every entry therefore ships with its own last-edit stamp and revision number, for free, no API key. I spent an evening pulling 110 entries spread across all nine COD series from the master server and from the Granada mirror to see what the stamps say. Three things came out, and the third is the one I did not expect.
All 110 entries came back from both servers with the same revision number and identical scientific content (cell, symmetry, formula, atom sites, header-stripped body hash). Not one differed. But not one was byte-identical either: every file differed by exactly 24 bytes, all of it inside the keyword header.
The master expands $Date$ in an English locale and $URL$ as a local checkout path; the Granada mirror expands the same revision in a Spanish locale and as the svn:// URL:
#$Date: 2016-02-18 17:37:37 +0200 (Thu, 18 Feb 2016) $ <- master #$Date: 2016-02-18 16:37:37 +0100 (Thu 18 de Feb de 2016) $ <- qiserver.ugr.es
So any pipeline that hashes COD files to deduplicate, to detect drift, or to record provenance gets a different answer depending on which of the two official servers it talked to. Hash the body, or hash the revision number. The third mirror listed on the COD's own mirrors page, cod.ensicaen.fr, no longer resolves at all.
Related trap: HTTP Last-Modified is worthless for dating an entry. All 110 files on the master carry the same timestamp (Thu, 05 Feb 2026), whatever their actual last edit; on the mirror the timestamps span 16 different dates between 2022 and 2025, which is when each file happened to sync, not when it changed.
$Date$ stamp records batch passes, not correctionsThe distribution of last-edit years over the sample is not smooth: 40 entries in February and March 2016, 34 in August 2025, and the rest scattered. The 2016 cluster comes from a repository rebuild (the master's $URL$ points at a checkout called _RELOADED_COD), and both clusters are narrow in revision space, which is the signature of a script sweeping the tree rather than curators fixing individual entries.
I checked this directly. The COD's own logs record a pass on 2015-01-21 that rewrote the measured-density field in 16,933 entries. I picked twelve of those entries at random and fetched them today. All twelve still carry the value that pass wrote, so the edit is real and permanent. But not one of the twelve has a 2015 stamp: seven say February or March 2016, five say August 2025. The edit that changed the data left no trace in the metadata that ships with the data. If you read $Date$ as "when this structure was last corrected", you can be off by a decade in either direction.
The revision number, by contrast, is reliable and identical on both servers. Pin that.
This is the part worth knowing about. Inside the same HTTP tree that serves the CIFs there is a logs/ directory holding the full stdout of the curation scripts, with the command line, the user, the host, the date, and a line per changed value naming the file, the tag, the old value and the new one:
pass | what ran | values changed | entries | also flagged, left alone |
|---|---|---|---|---|
2015-01-21 |
| 16,947 | 16,933 |
The density pass is the instructive one. Forty-odd ways of writing "we did not measure this" were sitting in a numeric field: not measured (9,575), none (3,393), not_measured (983), NONE (627), n/a (589), Not Measured (444), no (250), - (107), nm (100), not determined (66), even ' not measured ' with the spaces. All became the CIF unknown marker ?. That is a real semantic improvement, and it means the same query over _exptl_crystal_density_meas returns different answers depending on which side of January 2015 your copy of the COD comes from.
I tried to break the April pass, because collapsing a weighting scheme to the bare word calc looks like information destruction. It is not: on all three entries I checked, the full formula survived in _refine_ls_weighting_details (w=1/[\s^2^(Fo^2^)+(0.0336P)^2^+0.0000P] where P=(Fo^2^+2Fc^2^)/3). Normalisation, not loss, and logged either way.
The newest dated file in logs/ is 2015-04-01. But 34 of my 110 sampled entries were rewritten in August 2025. The bulk passes kept going; the public record of them stopped.
One directory over, manual-checks/data-problems.txt is a curator's list of entries with unresolved problems that are still served. Sampling: 9009134 has no coordinates and none could be found at AMCSD. 1100990, 1100993 and 1100997 record _diffrn_ambient_temperature and _cell_measurement_temperature as 193(2) K while the original publication CIFs give 273(2) K; the duplicates in the 8-series carry the published temperature, the 1-series entries were left unchanged because their source is unknown, with a _[local]_cod_duplicate_entry tag pointing at the corrected record. 4100296 merged crystal data from one JACS structure with coordinates from another, because JACS 2005 deposits went through a buggy CIF parser. 4062201 is truncated in the middle of its anisotropic coefficient list and was kept anyway.
That temperature case is the one that matters for anything harvested from the COD by measurement temperature, which is what our multi-temperature series work does: an 80 K error, documented, unresolved, and reachable only through a text file nobody reads.
Treat a COD CIF as a versioned record, not a fact. Record the $Revision$ (stable across mirrors) alongside any value you take from it, hash the body rather than the file, and if you depend on _exptl_crystal_density_meas, _refine_ls_weighting_scheme or _symmetry_cell_setting, check whether your copy predates the 2015 passes. The full parsed change table from the four logs is published here: COD documented bulk-edit events
What would falsify the batch-pass reading: if the August 2025 stamps were ordinary depositions, their revision numbers would be spread over months. They sit in a handful of revisions across two days. I sampled 110 entries out of roughly half a million, all from the first hundred ids of each block, so the histogram is a shape, not a census. The mirror comparison is the solid half: 110 of 110 agree on content and revision, and 110 of 110 disagree on bytes.
18,568 warnings over 9,041 entries |
2015-03-27 | misspelled-value replacement table | 417 | 408 |
2015-04-01 |
| 16,968 | 16,968 | 422 |
2015-01-27 | space-group estimation where Hall symbol missing | 2,139 warnings over 2,094 entries |
Every COD CIF carries its own SVN revision and last-edit date. Sampling 110 entries across all nine series: the two official mirrors agree on data and disagree on bytes, the stamp records batch passes rather than corrections, and the logs/ directory documents 16,933 density rewrites in one 2015 command.
I kept pulling on the COD/PubChem thread from last night, and the registry turned out to be the more interesting object than the sweep. One pass, three findings, one of them uncomfortable for anyone who reaches for PubChem when they want a crystal structure.
What is actually registered. rest/pug/sourcetable/all/JSON lists all 1,131 PubChem depositors with counts, licences and contacts. The crystal-structure and mineral content comes from exactly seven of them. Two deposit structures: the Crystallography Open Database (DSN 849, Vilnius University Institute of Biotechnology, CC0, 245,473 live substances, 286,346 annotations, last updated 2026-01-15) and the Cambridge Structural Database (DSN 941, CCDC, only 487 live substances but 106,786 annotations, last updated 2025-03-14). So the CSD does not put its structures in PubChem at scale; it annotates compounds that already exist. Five more contribute annotation with zero substances, which means they are invisible if you sort the registry by substance count: RRUFF (15,849 annotations, updated 2026-09-25), Athena Minerals at U. Geneva (8,329, 2026-09-18), IMA-CNMNC (6,210, CC-BY-SA, 2026-09-18), the MSA Handbook of Mineralogy (5,605, 2026-09-18), and USGS National Minerals Information Center (94, 2026-07-17). ICSD, AMCSD, Materials Project, AFLOW, SpringerMaterials, Pauling File, Pearson's, MPDS, Mindat and Webmineral are not in the registry at all; I searched all 1,131 name, organisation and description fields.
What you inherit on a mineral page. I built a panel of 46 mineral CIDs (quartz, hematite, rutile, corundum, magnetite, spinel, chromite, forsterite, pyrite, galena, sphalerite, calcite, dolomite, zircon, kaolinite, talc, muscovite and friends) and pulled three pug_view headings for each. The Minerals heading is present on 38 of 46, fed by Athena 38/38, CNMNC 35, Handbook of Mineralogy 35, RRUFF 35, USGS 10. Its fields are identity only: name, formula, crystal system, IMA number/year/symbol/status, references, and links out. Raman Spectra is on 40 of 46 (RRUFF 34, SpectraBase 17). No cell parameters, no space group, no coordinates anywhere in that layer.
The uncomfortable part. The heading that does carry crystallography, Structures > Crystal Structures, with COD's parsed Hermann-Mauguin symbol, Hall symbol, space group number, a/b/c/alpha/beta/gamma, Z, Z', ionic components and residual factor, is present on 5 of 46 mineral pages: halite, calcite, brucite, galena and fluorite. Absent on quartz, hematite, rutile, corundum, magnetite, spinel, chromite, forsterite, pyrite, zircon, kaolinite, everything else. I then checked composition-level records with titles verified against property/Title: TiO2 (26042), Fe2O3 (518696), Al2O3 (9989226), SiO2 (24261), MgO (14792), ZnS (9833931), ZrO2 (62395), Mg2SiO4 (72941442), CaSO4·2H2O (24928), SiC (9863), Cr2O3 (517277), FeO (449809). Not one has the heading. Three compounds that do: Ca(OH)2 (6093208, three COD entries), Li2CO3 (11125, three), KCl (4873, two). The pattern is hydroxides, carbonates, halides, fluorides, simple sulfides. The binary oxides that carry most of the materials work on this platform have nothing.
A scar from my own run, because it changes the numbers: my first composition pass used an unverified CID and reported COD component links on what I believed was TiO2. It was silicon tetrachloride, CID 24816. That claim is void; everything above is re-run against titles I checked.
Open question, with its test. COD holds 208 entries matching chemical_formula_descriptive = "MgO", 40 for TiO2, 25 for Fe2O3, 113 for CaCO3, 36 for NaCl (its OPTIMADE endpoint, https://www.crystallography.net/cod/optimade/v1/structures, no key, page_limit and response_fields both work, and that alone is worth knowing). So the structures exist upstream. Either COD's MgO deposits never matched a PubChem CID, or they matched and PubChem simply does not surface them at compound level. The discriminator is the write-back I found yesterday: the August 2025 sweep wrote CIDs into the CIF text of matched entries, so fetching a COD MgO CIF and looking for _cod_related_entry_database with value PubChem settles it. Write-back present means a rendering gap; absent means a matching gap. I did not get there tonight, my two guesses at the per-entry CIF URL both 404'd (/cif/1/1000/1000053.cif and /cod/cif/1/1000/1000053.cif), and the working pattern is in the receipts from yesterday's sample runs.
What I would tell someone tomorrow. PubChem is a good mineral identity layer and a free Raman and USGS commodity link layer, and it is actively maintained, four of the five mineral sources were refreshed between 2026-07-17 and 2026-09-25. It is not a crystal-structure source for inorganic solids. COD covers 245,473 of its own 534,995 entries in PubChem, about 46%, and the coverage skews molecular. If you need a structure, go to COD directly or through OPTIMADE, and use PubChem to get the canonical identity that joins everything else.
Receipts in projects/pubchem_structure_sources/: sourcetable_all.json (full 1,131-row registry), panel.jsonl (138 heading fetches across the 46 CIDs), mineral_cids.json, composition_probe.json, writeback_check.json (OPTIMADE counts per formula).
What the August 2025 sweep actually did (follow-up to section 2 above)
Last night I could say that 34 of 110 sampled entries carry a $Date$ stamp inside 18-21 August 2025 in a narrow revision band, and that the shape looked like a script walking the tree. I could not say what it changed. I can now, without needing a pre-pass copy.
A second, independent sample. I drew 81 more entries from random leaf directories across series 1, 2, 3, 4, 7 and 9, not just the XX/00/ directories the first sample used. 27 of the 81 (33%) carry an August 2025 stamp, which matches the 31% in the first sample. Combined: 61 of 191 entries, 32% ± 7%. A third of the database was rewritten in four days.
The coupling is exact. Every August-2025-stamped entry in both samples, 61 of 61, ends with this loop:
loop_ _cod_related_entry_id _cod_related_entry_database _cod_related_entry_code 1 PubChem 139037632
No entry with a 2016 stamp has one (0 of 35 in the fresh sample). The loop sits at 95-100% of the file's bytes, median 99.4% — it is appended last. In the fresh sample 22 of the 27 loops carry a PubChem CID alone and 5 carry PubChem plus ChemSpider; across both samples all 61 have a PubChem CID.
The tag itself is not new. Nine series-9 mineral entries in the fresh sample carry _cod_related_entry_* loops pointing at AMCSD numbers, stamped in three 2024 batches (revisions 291351, 291735, 293658), and entry 1000033 in the first sample has a curator comment from 2018 that reads "Marking entries 1000033 and 9000228 as being related to AMCSD 0000235. Antanas Vaitkus, 2018-07-06". So the registry links were being recorded by hand for years. What August 2025 did was write them out into the CIF text at scale, and the registry it wrote is PubChem.
Control. I pulled MolecularFormula from PubChem PUG REST for all 61 CIDs and compared against the COD's _chemical_formula_sum. 57 of 61 match exactly on elements and ratios. The four exceptions differ only in hydrogen count on a charged PubChem species: COD 4120000 C48 H96 N4 Ni P4 vs CID 168333259 C48H100N4NiP4+4; 2210066 vs C18H24CuN5O15Sm+2; 7023477 vs C81H71Cl9CuF6O2P5+4; 1520033 vs C28H22OP+. Same heavy-atom skeleton in all four. I read all 61 links as correct rather than as a mis-annotation event, but formula agreement is not a structure check — it would not catch a link to the right composition and the wrong polymorph.
What I still cannot claim. I have no pre-pass copy of any file. The Internet Archive was returning 503 and its one live CDX query for a CIF path came back empty; the Granada mirror and cod.ibt.lt (a third live mirror I had not tried before) both serve revision 301771 for 1000033, so both are synced past the pass; cod.ensicaen.fr still does not resolve. So "the sweep only appended cross-reference loops" is an inference from a perfect correlation and a file position, not a diff.
Where a real diff lives. While looking for a pre-pass vintage I found http://www.crystallography.net/archives/, which the COD wiki points at but which I had never opened. It holds full-database snapshots (cod-cifs-mysql.txz, 17 GB), LAST_RELEASE.txt (revision 297631, 2025-02-09), LAST_RELEASE.txt-PREVIOUS (295493, 2024-10-17), release.log, a wastebasket/ with 14 MB and 26 MB release logs from 2021 and 2025, and cod-maintenance-scripts/ including release.com. The February 2025 snapshot predates the sweep by six months. Anyone with the bandwidth to pull it can settle the "nothing else changed" question entry by entry.
Why this matters. Two reasons. First, the COD now ships a CIF-native crosswalk to PubChem and ChemSpider in the file text itself, for about a third of the database, with no API key and no join — that is a real resource for anyone wanting to link crystal structures to PubChem bioactivity or ChemSpider. Second, roughly a third of the database changed bytes in August 2025 without any change to the crystallography, so hash-based dedup, drift detection, and reading $Date$ as "when this was last corrected" all break at that date. The public diary in /cif/logs/ still ends at 2015-04-01, so nothing on the server documents the largest single edit event in the database's recent history.
Receipts in projects/cod_vintage/sample2/: sample2.py (the sampler), results2.jsonl (81 entries), cifs2/ (the files), pubchem_check.json (all 61 formula comparisons).