Compare commits

...

3 Commits

Author SHA1 Message Date
ec1e790713 Add generated siblings for pages and test post
All checks were successful
ci/woodpecker/push/woodpecker Pipeline was successful
2026-09-16 15:03:02 +00:00
d47cd326bc Merge branch 'claude/relaxed-faraday-h4zd09' of https://github.com/pablovolenski/vienalatina 2026-09-16 14:01:34 +00:00
Claude
8d07a6037a
Use a word-shaped mask token so the model stops eating it
Measured against M2M100 418M on real content: the OpenNMT ⦅0⦆ convention was
dropped on every single occurrence of a protected term. SentencePiece
fragments punctuation runs, and the model then has nothing it recognises as a
unit to copy across.

The brand paid for it. "Viena Latina" came back as "Wien Latin" in one file
and "Vienna Latina" in another — the same name rendered two different wrong
ways across two pages, which is worse than being consistently wrong. "Grätzl"
survived untouched throughout, because it is genuinely unknown to the model,
whereas "Viena" reads as a city name and gets translated.

Masks are now word-shaped (Zq0Xv), which a model treats like an unknown proper
noun and carries through rather than translating. Matching is case- and
space-insensitive, since models re-case and pad these.

The tiered fallback stays as the safety net for when it is still dropped.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NizVpJ2dwzCbjCrTLCjeHn
2026-09-16 13:34:26 +00:00
7 changed files with 65 additions and 11 deletions

View File

@ -0,0 +1,8 @@
---
title: über
lang: de
manual_translation: false
translated_from: es
---
**Viena Latina** ist die lateinamerikanische Gemeinschaft in Wien: Tourismusführer, kulturelle Agenda, Gastronomie, Gemeinschaftsleben und lateinamerikanische Geschäfte in der Stadt.

View File

@ -0,0 +1,8 @@
---
title: Sobre
lang: pt-br
manual_translation: false
translated_from: es
---
**Viena Latina** é a comunidade latino-americana em Viena: guias de turismo, agenda cultural, gastronomia, vida comunitária e lojas latinas na cidade.

View File

@ -0,0 +1,8 @@
---
title: Kontakt
lang: de
manual_translation: false
translated_from: es
---
Schreiben Sie uns an **hola@vienalatina.com**.

View File

@ -0,0 +1,8 @@
---
title: Contacto
lang: pt-br
manual_translation: false
translated_from: es
---
Escreva-nos a **hola@vienalatina.com**.

View File

@ -0,0 +1,11 @@
---
title: Testartikel
date: 2026-09-16
lang: de
manual_translation: false
categories:
- Comunidad
translated_from: es
---
Dies ist ein Test des automatischen Übersetzungsflusses im Grätzl. Viena Latina veröffentlicht in Spanisch, Deutsch und Portugiesisch.

View File

@ -0,0 +1,11 @@
---
title: Artigo de prova
date: 2026-09-16
lang: pt-br
manual_translation: false
categories:
- Comunidad
translated_from: es
---
Este é um teste do fluxo de tradução automática no Grätzl. Viena Latina publica em espanhol, alemão e português.

View File

@ -5,9 +5,11 @@ model has no equivalent, so structure is protected here instead: non-prose
blocks pass through untouched, and inline constructs are masked with opaque blocks pass through untouched, and inline constructs are masked with opaque
placeholders whose survival is verified after the round trip. placeholders whose survival is verified after the round trip.
Placeholders use OpenNMT's protected-sequence convention (U+FF5F/U+FF60). Placeholders are word-shaped (``Zq0Xv``) rather than punctuation. Measured on
SentencePiece keeps these atomic; ``{{x}}``, ``<x>`` and ``%s`` get fragmented M2M100 418M: the OpenNMT ``⦅0⦆`` convention was dropped on every single
by BPE and dropped by the model. occurrence, because SentencePiece fragments punctuation runs and the model then
fails to copy them. A token that looks like an unknown proper noun gets carried
through, since the model has nothing to translate it to.
""" """
from __future__ import annotations from __future__ import annotations
@ -16,7 +18,8 @@ import re
from .provider import SITE_TO_MODEL, Provider from .provider import SITE_TO_MODEL, Provider
OPEN, CLOSE = "⦅", "⦆" _MASK = "Zq{}Xv"
_MASK_RE = re.compile(r"Zq\s*(\d+)\s*Xv", re.IGNORECASE)
# Community vocabulary that must reach readers unchanged. Not inherited from # Community vocabulary that must reach readers unchanged. Not inherited from
# the WordPress plugin, which had no glossary at all — edit freely. # the WordPress plugin, which had no glossary at all — edit freely.
@ -43,9 +46,6 @@ _INLINE = (
re.compile(r"https?://\S+"), # bare URLs re.compile(r"https?://\S+"), # bare URLs
) )
_PLACEHOLDER = re.compile(re.escape(OPEN) + r"\s*(\d+)\s*" + re.escape(CLOSE))
class PlaceholderError(RuntimeError): class PlaceholderError(RuntimeError):
"""A masked span did not survive translation intact.""" """A masked span did not survive translation intact."""
@ -57,7 +57,7 @@ class _Masker:
def _take(self, match: re.Match) -> str: def _take(self, match: re.Match) -> str:
self.spans.append(match.group(0)) self.spans.append(match.group(0))
return f"{OPEN}{len(self.spans) - 1}{CLOSE}" return _MASK.format(len(self.spans) - 1)
def mask(self, text: str) -> str: def mask(self, text: str) -> str:
for pattern in _INLINE: for pattern in _INLINE:
@ -68,10 +68,10 @@ class _Masker:
return text return text
def restore(self, text: str) -> str: def restore(self, text: str) -> str:
# Models pad and reorder placeholders; normalise spacing before matching. # Models pad, re-case and reorder placeholders; normalise before matching.
text = _PLACEHOLDER.sub(lambda m: f"{OPEN}{m.group(1)}{CLOSE}", text) text = _MASK_RE.sub(lambda m: _MASK.format(m.group(1)), text)
for index, span in enumerate(self.spans): for index, span in enumerate(self.spans):
token = f"{OPEN}{index}{CLOSE}" token = _MASK.format(index)
seen = text.count(token) seen = text.count(token)
if seen != 1: if seen != 1:
raise PlaceholderError( raise PlaceholderError(