← try the model

Cicero

Notes from training a small latin-only language model from scratch · July 2026

I've built Cicero, a new small latin only language model.

Why did I do this?

Last year I bought a new computer, before the prices really went wild. And I knew I wanted to make it more future proof than I would otherwise, so I sprung for the 5090. My idea to justify it at the time was, maybe I'll learn how to train a language model. I'd never trained one before. Instead of doing that I just played a bunch of games on it.

A few weeks ago, while working on a small other side project, I thought: yea okay lets actually do this. And a question struck me: could you train a language model on only latin? The question is interesting. There's not a ton of latin language out there. What's survived from antiquity is not actually enough to train on, and most of what's survived isn't simple latin, it's very good literary latin. It would be a challenge to do, especially since I was only gonna train on my 5090. I wasn't going to rent out compute for a silly little side project.

I started out gathering that data programmatically (most of it is in fairly online accessible libraries, including some here on github that you can just clone). I tried to train a model on just that. I did a test run of it on a small segment and while it wasn't really producing coherent latin, it was producing something that looked like latin.

So I asked: how can we get it to produce real latin?

How I went about it

I tried pulling a few different levers. First I just tried to get as much latin as I could. Mixing in a ton of different latin libraries and spreading out to medieval and later latin writing. I tried a run dumping all of that in there. It didn't really help. The model became pretty confused and started largely spitting out bible verse names. You can tell where it might have gotten some of that from! So, instead, I tried a run of the cleanest prose I could find. This was actually pretty successful, it produced a model that could spit out vaguely coherent latin. Again, it was not really able to consistently spit out coherent latin.

I'm not going to go through all of what I did here.

Benchmarking

Before we get into my next steps, I'll take a little time to discuss how I benchmarked my model. I used a cloze score. A cloze score is when you remove words from a passage and have it fill them in. I asked the question: how much can we push the cloze score without killing myself or actually having to rent much compute? I didn't think I could get it to a 100%. I aimed for a 75%.

OCR Pipeline

The Internet Archive has a vast array of latin language on it. However, most of it is not in a text format. A lot of it was OCRd at some point, but it was over 10 years ago, and the OCR is really bad. I tried with the old OCR and it just made the model more psychotic. It's full of nonsense text and unicode errors. So I tried to re-ocr it with tesseract. (I tried a few modern OCR tools but tesseract did best). This helped a little bit but it wasn't enough, and this data was still very dirty. I would likely have to go through and manually label every data source, which wasn't really scalable for a fun little side project.

Synthetic Data

The final thing I tried was creating synthetic data. I bought some Kimi K2.5 credits (I benchmarked a few providers and kimi was both the clear cheapest and the best quality). And I created some synthetic latin, I also translated some English and Greek classics into Latin. This finally moved the needle and began to get the cloze scores up.

Everything else that didn't work

This project produced a genuinely impressive pile of negative results, and I think they're the most interesting part, so here's a quick tour:

• Adding any single extra source to the good data. After the synthetic win, I took my best corpus and added exactly one broad source at a time: latin wikipedia, Project Gutenberg's latin, the Patrologia (a giant 19th-century collection of church latin), latin wikisource, a few medieval and renaissance collections. Ten runs. Every single one made the model worse. Latin wikisource was the most spectacular failure: the model scored 25% on the cloze test, which is worse than random guessing. It fit the data just fine, it just confidently learned grammar that contradicts actual latin, because wikisource is full of uncorrected OCR uploads and weird 19th-century editorial latin.
• "Cleaning" the data. I had a cleanup pass that stripped macrons (ā → a) and normalized ligatures, on the theory that more uniform text is easier to learn. In two out of three controlled tests, the cleaned version did worse than the raw version. Macrons mark vowel length, which is real grammatical information in latin — I was throwing away signal the model was actually using. The one place cleanup helped was wikisource, where stripping the {{wiki markup}} outweighed the macron damage. Lesson: don't "normalize" things you don't understand.
• More synthetic data. Roughly 2.5x-ing the synthetic corpus made the model slightly worse, not better. Synthetic data has a sweet spot, not a more-is-better curve.
• A fancier tokenizer. I rebuilt the tokenizer (unigram model plus BPE-dropout, which is supposed to help with morphologically rich languages, and latin is nothing if not morphologically rich). It regressed, and it regressed worst on the held-out items, which means it broke generalization specifically.
• A modern architecture. I swapped the GPT-2-style stack for a llama-style one (RoPE, RMSNorm, SwiGLU, the works). It trained faster and got slightly better loss, but it generalized worse and memorized more. The architecture was not the wall.

One diagnostic in the middle of all this pointed somewhere useful though. I trained mini models on 10%, 30%, and 100% of my good data, same recipe, and the loss just kept getting better with more data, monotonically. So the model was still data-hungry — even though every attempt to actually feed it more data had made it worse. Those two facts together really only have one resolution: the extra data wasn't poison exactly, it was crowding out the good data at the short training lengths I was using.

Breakthrough

I tried a few more different things, but most of them ended up being middling to no help. The real issue was that I simply didn't have enough data and so much of the data was dirty. However, it's possible that I was thinking about this wrong — maybe I was just not training long enough. So, I used (almost) all the data I had collected, I cleaned it up a bit, and then I ran a much longer and larger training run. The loss vals looked a lot better and at the end the cloze score got above 80%. We got to a B!

However, it still wasn't quite perfect. I tried a few more things:

1. LAWA averaging, where you average together the final few checkpoints. This simply didn't work, the scores were worse.
2. Retrieval Augmentation, where I created a library of some of the best latin. This also didn't work.
3. Post Training

Post Training

The last big lever was post training. I ran two stages on top of the big model. First, continued pretraining on only the cleanest classical prose, to sharpen the register. Second, a targeted grammar curriculum: synthetic latin passages written specifically to exercise the grammar the model was worst at (preposition cases, subjunctive subordination, participle agreement, that kind of thing — I knew exactly which phenomena were weak because the cloze test is broken down by phenomenon).

The curriculum had my favorite failure of the whole project. I generated the first version with Kimi and asked it to grade its own output: it said 95% of it was clean latin. Then I had Claude Opus judge the same passages independently, and it flagged about 40% of them for real grammar errors. Models grade their own homework very generously. I threw the whole batch out, regenerated the curriculum with Opus as the writer, and then kept only the passages that two independent judges both approved. For synthetic grammar data, it turns out the capability of the generator is the whole ballgame — no amount of prompt engineering was going to rescue the first batch.

The results were honestly mixed. I had pre-registered statistical gates for these runs and neither one cleared them (the held-out improvements were real but not significant at the sample sizes I had). But the curriculum model improved on every held-out eval I cared about: literary cloze went from 0.75 to 0.82, the blind pack from 0.69 to 0.72, and on a sealed pack targeting the model's known weaknesses it went from 0.75 to 0.82. The textbook cloze actually went down a bit, from 0.80 to 0.77 — because the curriculum shed a bunch of medieval and neo-latin patterns that the original training mix had rewarded.

Deciding which model to ship

And that set up the most embarrassing part of the project: I flip-flopped on the release three times in one day.

On paper the 0.80 model is the best model. It has the highest score on the headline benchmark, and the post-trained model failed its statistical gates, so the defensible, by-the-book decision was to ship the 0.80 model. That's what I did. Then I actually played with the live demo, and the 0.80 model did things like this, unprompted:

]] Italia Aevo Aevo est urbs maxima Hispaniae

Literal wiki markup, leaking out of the model, followed by a degenerate loop, followed by a sentence claiming Italy is the largest city in Spain. On other prompts it would drift into lists of medieval kings. All of that traces straight back to the corpus — the big training mix is mostly wiki, church, and 19th-century latin, and it shows.

The curriculum model scores lower on the textbook benchmark, but ask it to continue a classical prompt and it writes things like "copias per montes asperas duxit, ne sine commeatu in hostium fines descenderet" — he led his troops through rough mountains, so that he wouldn't descend into enemy territory without supplies. That's real latin. Grammatical, coherent, and in the right register.

So the curriculum model is what shipped. The lesson here is one I thought I already knew, but apparently needed to learn personally: the benchmark picks the best test-taker, not the best writer. My cloze test was partly rewarding in-distribution memorization, and the whole point of the public model is to generate latin. For a generative model, live generation gets a vote.

Getting it into your browser

The model runs entirely in your browser — an int8-quantized ONNX export, about 100MB, running on ONNX Runtime Web. No server, no API. Getting there had two good bugs:

1. ONNX Runtime's WebGPU backend silently produced garbage logits for this model. Same file, same weights: coherent latin on the CPU/WASM backend, token salad on WebGPU. The site defaults to WASM now.
2. Generation never stopped, ever, because the tokenizer conversion left the end-of-sequence token null, so the model would happily fill the entire context window. Pinning the SentencePiece token ids fixed it. I also added a repetition penalty, which helps a small model like this a lot.

And a confession for the methodologically pure: every benchmark number in this post was measured on the full-precision PyTorch checkpoints. The quantized browser model was only ever checked by eye, after it was deployed. It's fine — but if I did this again, the artifact that ships would be the artifact that gets measured.

What I actually learned

1. At this scale, data quality dominates — but the full story is subtler. At a fixed short training budget, cleaner-but-smaller beat bigger-but-noisier every single time I measured it. But the scaling diagnostic showed the model was genuinely data-starved, and the breakthrough came from using (almost) all the data and training much longer. Both things were true at once: "quality beats quantity" and "the model needs more data."
2. Never let a model grade its own output. 95% self-reported vs 59% independently judged is not a rounding error.
3. Benchmarks are a proxy. The highest-scoring model leaked wiki markup at users. Ship on live behavior, not just the scorecard.
4. Negative results are cheap if you set up for them. Nearly everything I tried didn't work, and that was fine, because every experiment was a controlled A/B with a decision rule written down before training started. The expensive failure mode isn't a bad run, it's fooling yourself about a bad run.
5. Don't clean what you don't understand. Stripping macrons felt like normalization and was actually deleting grammar.

Try it

The model is live right here — it runs locally in your browser, so be patient with it, it's a small model on your hardware. The weights are on Hugging Face at gmmeyer/cicero. It is not going to write you a lost book of Livy. But it writes recognizable, often genuinely coherent latin, it was trained from scratch on one consumer GPU in my house, and the whole thing cost less than $200 in API credits on top of the electricity.

Not bad for a silly little side project.