Results
These are the numbers of 5 and 6 November 2026, the first two days of reading tests on Leonardo, with the second readings of 11 November and the trainings of 14 and 15 November. The samples are small, and we say which ones. Every error rate is measured against a transcription made by people.
How we measure
Eight test pages that are hard on purpose: Greek minuscule and cursive from the 10th to the 16th century, printed polytonic Greek, Latin Gothic and documentary cursive, Arabic naskh and Maghribi script. Each comes from a public dataset with a transcription made by people. None of the eight pages is ever used to train our models.
The figure is the character error rate: the share of characters that would need to be changed to turn the machine's reading into the human one, after Unicode normalisation. 0% is a perfect reading.
One caution. The public base model we start from for fine-tuning had some of these pages in its own training data. Where that is the case the figure is marked with an asterisk and should be read as a ceiling, not a measurement. The models we train from scratch never saw them, so their figures are clean.
Inside a collection
This is the situation the project is designed for: a curator corrects some pages of a manuscript or a fond, a model learns from them, and it reads the rest. Here everything in the manuscript except the test page was available for training.
| Test page | Best general-purpose model | Trained here, from scratch | Trained here, fine-tuned |
|---|---|---|---|
| EPARCHOS, Greek cursive, about 1500 | 46.9% | 7.6% | 4.3%* |
| Patrologia Graeca vol. 145, printed polytonic Greek | 4.9% | 1.2% | 0.8% |
| Stavronikita 114, Greek minuscule, 15th century | 50.5% | 19.8% | 7.4%* |
| Palatinus graecus 23, Greek minuscule, 10th century | 59.8% | 26.7% | 14.2%* |
| Sinai Arabic 418, Christian Arabic naskh, 13th century | 18.2% | 17.5% | 16.0%* |
| BULAC MS.ARA.609, Maghribi script, 1734 | 34.8% | 20.5% | 17.8%* |
| e-NDP, Latin documentary cursive, Notre-Dame of Paris | 50.5% | 21.1% | 17.1% |
| Rps 12533, Latin Gothic book hand | 37.3% | 36.0% | 25.2% |
Best general-purpose model: the lower of olmOCR-2 (7B) and LightOnOCR-3-1B over every setting we tried, reading the whole page or the human-drawn lines. Whether those models saw these pages in their training is not documented. Trained here: Kraken line recognisers, PP-OCRv6 architecture, one A100 GPU each, between one and three hours. * The base model of the fine-tuned column had this page in its training data.
The Greek is where the gain is clearest: a manuscript of around 1500 drops from 47% to under 8% of characters wrong, with a model that has never seen the page. On printed Greek the error is about one character in a hundred. On Arabic naskh we are level with the best general model; on Maghribi script we are well ahead. The Gothic Latin hand barely moves, because only fourteen pages of that kind were available to learn from.
On 11 November we read the same eight pages with a third general model, Mistral-Small-3.2 (24 billion parameters), to check whether a larger vision-language model does better. It does not. Its error ranges from 61% on the Greek cursive to more than 400% on the printed Greek and the Gothic Latin, because it writes text that is not on the page. The column of the best general-purpose model does not change.
On 14 and 15 November we also tried two small OCR models released this autumn, MiniCPM-OCR-1.0 and Xiaomi-OCR-0. They read a page in a few seconds, but neither beats the best general model on any of the eight pages: on the printed Greek they make 16% and 21% of errors, against 4.9%.
A hand the model has never seen
Here the whole manuscript of the test page was kept out of training. This is the harder question: does a model learn a script, or only the hands it was shown?
| Test page | Trained here, from scratch | Best public specialised model |
|---|---|---|
| Patrologia Graeca vol. 145, print | 2.0% | 1.5% |
| Stavronikita 114, minuscule | 31.5% | 24.5% |
| EPARCHOS, cursive | 53.0% | 37.6% |
| Palatinus graecus 23, minuscule | 73.3% | 31.1% |
| Sinai Arabic 418, naskh | 38.9% | 36.5% |
| BULAC MS.ARA.609, Maghribi | 25.9% | 22.5% |
Public models: CLLG (printed Greek; it may have seen the Patrologia), greekmix by Annette von Stockhausen (Greek minuscule), TariMa by BULAC and Calfa (Arabic), as published on Zenodo.
On a hand it has never seen, a small model trained on nine to eighteen thousand lines is not better than the best published ones. We did not expect it to be. It is the honest limit of today's work, and the reason the project trains on each collection with pages its own curators have corrected.
How many corrected pages are needed
A curator wants to know how many pages to correct before the machine helps. On 14 November we fine-tuned, on Leonardo, models that had never seen four of the test collections, with 5, 10, 25, 50 and all of their pages, and read each test page again.
| Collection | 0 pages | 5 | 10 | 25 | 50 | All |
|---|---|---|---|---|---|---|
| EPARCHOS, Greek cursive | 53.0% | 23.8% | 18.5% | 11.4% | 7.8% | 7.1% (114) |
| Stavronikita 114, Greek minuscule | 31.5% | 28.1% | 19.9% | 18.3% | – | 15.5% (38) |
| e-NDP, Latin documentary cursive | 32.1% | 28.6% | 28.0% | 24.6% | 21.3% | 18.5% (505) |
| Sinai Arabic 418, naskh | 38.9% | 26.6% | 25.9% | 28.5% | 20.9% | 19.9% (336) |
One test page per collection, never used for training. Stavronikita has 43 pages, so there is no 50-page point. The number of epochs was fixed, not tuned.
On the Greek cursive five corrected pages halve the error, and fifty get almost as far as the whole collection. On the other three the gain is slower, and most of it comes only with the whole collection. One page per collection gives a curve, not a promise: a single point can move by a few percentage points, as the Arabic does between 10 and 25 pages.
Handwriting in English and Italian, from open data
Two of our pilot collections are letters in English and twentieth-century Italian, and none of their pages is corrected yet. So we trained on openly licensed transcriptions and measured on pages the models had never seen.
| Test set | General model (PP-OCRv6) | Trained here |
|---|---|---|
| Bentham papers, official test set of the ICDAR 2015 competition, 50 pages | 17.5% | 12.8% |
| Revolutionary City, Library of Congress letters, 146 pages of hands never seen in training | 13.5% | 11.9% |
| Model | Error |
|---|---|
| General model (PP-OCRv6) | 8.7% |
| Trained on 72,723 modern handwritten lines, mostly French | 13.7% |
| General model, then 45 pages of the diary | 6.1% |
| Modern handwriting, then 45 pages of the diary | 5.1% |
In English the gain is real but moderate, because the general model already reads English hands fairly well. In Italian the lesson is sharper: a model trained on French hands reads Italian worse than the general one, and the error drops only once it has seen pages of the collection itself. That is how the project is designed to work. The Italian figures rest on two pages, about two thousand characters, so a point of difference is within the noise.
Open data used, with thanks: Transcribe Bentham, ICDAR 2015 HTRtS dataset (CC BY 4.0); The Revolutionary City Corpus 1758-1805 (CC BY 4.0); CATMuS Modern (CC BY 4.0); the diary of Bruno Celestino from Europeana 1914-1918, transcriptions CC0 and images CC BY-SA 3.0.
Sheet music
Four pages with a symbolic transcription made by people (OpenScore, public domain), read by copisteria into MusicXML and compared symbol by symbol with musicdiff (OMR-NED).
| Page | Error |
|---|---|
| Mozart, String Quartet K.155, Breitkopf edition of 1881, page 1 | 4.5% |
| Mozart, String Quartet K.465, Breitkopf edition of 1882, page 1 | 6.5% |
| Beethoven, String Quartet op. 18 no. 1, Breitkopf edition, page 1 | 7.8% |
| Mozart, String Quartet K.155, autograph manuscript, page 1 | 96.9% |
Printed music is read well: bars and parts come out right, and the errors are single notes and marks. Mozart's own hand is not read at all: the model sees eight parts of five bars instead of four parts of fourteen. The printed figures may be flattered, because the model was trained on a large collection of scores that may include these pieces.
On 11 November we read all 53 pages of the three quartets and the two autographs on Leonardo, in twelve minutes on one GPU. Only the first pages have a transcription cut to the page, so only they are measured: the figures came out within 0.2 points of the ones above.
The newspaper
On «Giustizia e Libertà» there is no human transcription yet, so there is no error rate. What we can report is behaviour. On fourteen pages, about 1,300 blocks, olmOCR-2 stopped early once. LightOnOCR-3-1B, the best general model on the manuscripts above, looped or stopped early 355 times and on several blocks described pictures instead of reading text. The two readings are in the editor, and a curator's corrections on a few pages will give the first measured figure.
On 11 November all 70 pages of the newspaper sample were read a second time, by Mistral-Small-3.2. Without a human transcription this does not say who is right, but it says where to look. The two readings share 89% of their words on the median page; the pages where they differ most are the ones a curator should open first. Mistral looped or stopped early on 14 blocks, olmOCR-2 on 5.
Compute: about 24 GPU hours on 6 November 2026 for ten training runs, two calibrations and the newspaper readings; about 3 GPU hours on 11 November for the second readings and the sheet music; about 22 GPU hours on 14 and 15 November for the page curve and the English and Italian models.
We acknowledge EuroHPC JU for awarding the project ID EHPC-AIF-2026PG01-1068 access to 5,000 GPU hours on Leonardo Booster hosted by CINECA, Italy.
