Generative AI Google Brain

translation-translation-training-data-tsv-generator

3 min read

Generate training dataset for Cloud Translation API NMT (Neural Machine Translation) model training

Source notebook

Repo path: translation/translation_training_data_tsv_generator.ipynb · Open on GitHub · intermediate

Generates DOCX translation TSV data and starts Cloud Translation custom NMT training.

Summary

This notebook teaches how to turn paired source and reference DOCX files into a two-column TSV for Cloud Translation API NMT model training. It downloads source/reference documents from Cloud Storage, removes blank paragraphs, aligns paragraphs and tables in document order, records mismatches and over-200-word lines, and writes the TSV locally. It then shows REST calls to create a Translation dataset, import TSV data from Cloud Storage, and trigger custom model training.

Key code patterns

Colab authentication guard

import sys
 
if "google.colab" in sys.modules:
    from google.colab import auth
    auth.authenticate_user()

Authenticates only in Colab; the notebook states this is not required in Vertex AI Workbench.

Download DOCX pairs from GCS

client = storage.Client()
bucket = client.get_bucket(source_bucket_name)
src_blob = bucket.get_blob(file_name)
ref_blob = bucket.get_blob(ref_file_name)
 
src_blob.download_to_file(src_f)
ref_blob.download_to_file(ref_f)
source = docx.Document(src_filepath)
reference = docx.Document(ref_filepath)

Loads paired source and reference DOCX files before TSV generation.

Iterate document blocks in order

for child in parent_elm.iterchildren():
    if isinstance(child, CT_P):
        yield Paragraph(child, parent)
    elif isinstance(child, CT_Tbl):
        yield Table(child, parent)

Keeps paragraphs and tables in document order so source and reference blocks can be paired.

Remove blank paragraphs before pairing

for para in source.paragraphs:
    if len(para.text.strip()) == 0:
        p = para._element
        p.getparent().remove(p)
        p._p = p._element = None

Prevents empty lines from causing source/reference line-pair mismatches.

Call Translation REST endpoints

headers = {
    "Authorization": f"Bearer {ACCESS_TOKEN}",
    "Content-Type": "application/json; charset=UTF-8",
}
requests.post(f"{url}/datasets", data=json.dumps(data), headers=headers)
requests.post(importDataset_url, data=json.dumps(data), headers=headers)
requests.post(models_url, data=json.dumps(data), headers=headers)

Uses refreshed Google auth credentials with Translation v3 dataset, import, and training requests.

Models & APIs used

  • APIs / services: Cloud Translation API, Cloud Storage
  • SDKs / libraries: docx, python-docx, google-auth, google-cloud-storage, requests

When to use this

Use this pattern when preparing paired DOCX translations for Cloud Translation custom NMT model training.

Gotchas & caveats

  • Only DOCX is supported; PDFs are explicitly rejected and source/reference formats must match.
  • Colab requires auth.authenticate_user(); Vertex AI Workbench does not require that step.
  • The TSV is created in the local path, but import_data expects a Cloud Storage URI for the TSV.
  • Lines over 200 words are not added to the training dataset and are returned in more_than_200_words.
  • Source/reference block mismatches are captured in mismatched_block and stop TSV generation at that point.

Best practices

  • Remove empty lines from both source and reference documents before making line pairs.
  • Preserve document order across paragraphs and tables when generating training pairs.
  • Validate source and reference paths and formats before processing.
  • Capture mismatched blocks and over-200-word lines instead of silently adding bad TSV rows.
  • Use dataset display names built from prefix, source language, target language, and optional suffix.

Graph View

Backlinks