Why AI images get the text wrong

2026-09-30 · original measurements

The figures in this article are listed in our data index, with what each one was measured from and when.

Some links on this site are affiliate links. If you sign up through them we may earn a commission at no extra cost to you.

We ran two production pipelines against the same sixteen reference images, both on fal, both gpt-image-2 at low quality and 1024×1024, and checked all 540 output cells by hand instead of sampling. One pipeline let the model draw the English and Chinese labels into the picture. The other forbade text in the image entirely and overlaid it afterwards with a script.

The model-written labels produced two hard content errors in 270 cells: a wrong Chinese character, and a mislabelled animal. The program-written labels produced zero, because they cannot produce any.

That contrast is the whole result, and it is not a measurement of model quality. It is arithmetic, and the difference matters for how you read everything below.

What was compared

Both pipelines were given the same sixteen reference images and asked for thirty 3×3 vocabulary sheets each, which is 270 cells per side and 540 in total. Both used fal, gpt-image-2, low quality, 1024×1024.

The comparison changed six things at once, including the API endpoint. That is the main reason this article is careful rather than triumphant.

Two of those rows need spelling out, because they limit what can be concluded.

The endpoints differ. Both records say gpt-image-2 with identical parameters, but the endpoint IDs are not the same string, and we have not verified whether they are aliases for one backend. If they are not, then model and workflow both changed, and no difference between the two columns belongs to either one.

The axes differ. Pipeline A varied the scene and held the renderer fixed. Pipeline B varied the style and reused templates. The images are therefore not paired. There is no “same cell, two ways” to point at.

What the cells showed

Full enumeration, no sampling: every one of the 540 cells was checked at full resolution.

The two failures on the model-written side are worth looking at, because they fail differently.

A wrong character. One cell should read 「骆驼」 (camel). It reads 「骆鼠」. The English CAMEL is correct and the animal is correct. Only the Chinese is wrong, and both characters are real ones, so no amount of glyph checking would catch it.

A label that drifted. One cell is labelled SWAN. The drawing is a short-necked, flat-billed duck. The same batch and the same pipeline produced a correct swan elsewhere, with a long neck and an orange bill with a black knob, so this is per-cell drift rather than a systematic misunderstanding.

Why this cannot attribute most of what it found

Six variables moved together. That is not an experiment but two teams’ worth of decisions compared after the fact, and the design separates none of them.

By the standard we applied to published research in an earlier piece, the honest reading is that this comparison cannot tell you whether the workflow, the endpoint, the prompt language or the axis design caused any of the differences above.

That includes the differences that flatter one side. Pipeline B covered more of the reference images’ visual variety, five styles against one. Pipeline A delivered 270 split cells in 48 MB where B needed 154 MB, because A wrote JPG and B wrote PNG. Both observations are real. Neither is attributable, and we are not going to dress them up as findings.

The one thing it does settle

There is a difference between a result and an identity.

Pipeline A’s zero is not a measurement of how well its model handles text. Pipeline A never asked the model to produce a character. Its prompts lock the image against any lettering, and it draws the labels afterwards from a word list it already holds. A pipeline that does not ask the question cannot get the answer wrong.

So this holds regardless of which endpoints were in play, and regardless of the other five variables:

The character error rate of a model that is never asked to render text is zero, by construction.

The consequence is the part worth keeping. The class of error that actually kills a delivered asset, a wrong character or a mismatched label, is removable from the generation step entirely, not merely reduced.

The two designs, in the same scene, show what that means in practice:

The pipelines’ own self-checks were wrong

Both sides reported on their own work, and both reports were unreliable in opposite directions.

Pipeline A reported its structure as 30 of 30 clean. It was not. Three sheets had genuine defects: one missing its horizontal divider, one missing its top frame, one with a column 11 pixels narrow. The report missed all three.

Pipeline B reported that all labels were correct, based on a full-resolution check of 5 sheets out of 30. Both failures happened to sit outside those five. At a 0.74% error rate, sampling 5 sheets and 45 cells is very likely to come back clean whether or not the batch is.

This is not a criticism of either report. It is a property of sampling an error rate near 1%. Five sheets looks like diligence and carries almost no information. Either check everything, or remove the error from the mechanism so there is nothing to check.

What we changed

We rebuilt the pipeline around those two findings. The design is one sentence: the model draws the picture, the code writes the text, and three deterministic gates sit in between.

Base images are now generated with lettering forbidden, and labels are overlaid afterwards. One addition matters more than it sounds. Before overlaying, the word list is compared against the font’s character map, and a missing glyph raises an error. That converts “we checked and found no typos” into an assertion that cannot be false.

The three gates cost nothing to run and were regressed against Pipeline A’s thirty historical sheets, where they did something the human review had not:

• They caught two real defects the first review missed. Two sheets had a second vertical divider that was visibly thin and broken in its middle section. A near-white pixel measurement put it at 0.68 and 0.49, against 0.95 to 1.00 for healthy sheets.
• They also overturned three false alarms. The first review had flagged three cells as having text covering the subject. Enlarging them showed the text sitting entirely inside the clean band, with the animal’s leg and a branch stopping above it.

Human judgement is unreliable at exactly the boundary “close to the text but not over it”, in both directions. A measurement is not.

Five sheets ran clean on the first attempt, 5 of 5 with no retries, at a mean of 19.0 seconds each and zero text errors of any kind.

What this does not say

It does not say one pipeline is better than the other. The comparison moved six variables and we have attributed one. Pipeline B’s wider style coverage is real and unmeasured. So is Pipeline A’s lighter delivery. This is a post-mortem of two confounded stacks, and we are reporting it that way on purpose.

It does not generalise beyond this task. The claim that survives is narrow and structural: if text accuracy matters more than stylistic range, take the text away from the model. Whether that trade is right for you depends on which failure you can afford. For a hundred sheets reviewed by eye, model-drawn text is fine. For ten thousand, a 0.74% error rate is roughly seventy-four cells to rework.

And the model is not the finding. We cannot even say the two sides used the same backend. What we can say is that the difference that mattered most was a decision about what to ask the model for, which is a workflow decision rather than a model choice. Our production data reached the same conclusion about cost, from the other side.

Method

• Sample: 30 sheets per pipeline, 270 cells each, 540 total. Full enumeration at full resolution, plus 2–8× enlargement of every suspect cell.
• Models: both gpt-image-2 on fal, low quality, 1024×1024. Endpoints: openai/gpt-image-2 and fal-ai/gpt-image-2. Whether these are aliases for one backend is unverified and is the comparison’s main confound.
• Attribution: only the text result is attributed, and only because it is structural rather than empirical.
• Gate regression: the three gates were calibrated against Pipeline A’s thirty historical sheets, not against new generations.
• Rebuilt pipeline: five sheets run end to end. Fifteen of its twenty scene definitions, and all non-animal categories, remain unvalidated.
• Client work: this is a client project. No client name, brand or commercial figure appears here, and the images carry no branding.
• Cost: no billed figure is published. At our measured baseline for this endpoint and quality, five sheets is about $0.03.

What we are working on next

Remove the endpoint confound. The one variable we can still resolve is whether those two endpoint IDs are the same backend. Until that is settled, the style-coverage difference stays unclaimed.

Validate the rebuilt pipeline at scale. Five sheets is a smoke test. The honest next step is all twenty scene definitions across all three categories, with the gates reporting rather than being tuned.

Widen the text assertion. The character-map check proves no glyph is missing. It cannot prove the right word was chosen, because a wrong word made of valid characters is still a wrong word. Catching that means checking the label against the intended vocabulary list, which is a data problem rather than a rendering one.