Measured against M2M100 418M on real content: the OpenNMT ｟0｠ convention was
dropped on every single occurrence of a protected term. SentencePiece
fragments punctuation runs, and the model then has nothing it recognises as a
unit to copy across.
The brand paid for it. "Viena Latina" came back as "Wien Latin" in one file
and "Vienna Latina" in another — the same name rendered two different wrong
ways across two pages, which is worse than being consistently wrong. "Grätzl"
survived untouched throughout, because it is genuinely unknown to the model,
whereas "Viena" reads as a city name and gets translated.
Masks are now word-shaped (Zq0Xv), which a model treats like an unknown proper
noun and carries through rather than translating. Matching is case- and
space-insensitive, since models re-case and pad these.
The tiered fallback stays as the safety net for when it is still dropped.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NizVpJ2dwzCbjCrTLCjeHn
8d07a6037a