TransformerTranslation

Two end-to-end Transformer machine translation experiments: an English→French model in PyTorch and a German→English model in TensorFlow. This started as a class project and grew into a compact, from-scratch implementation and training workflow that I can point to when discussing model design, data processing, and training tradeoffs.

What’s In This Repo

• pytorch_french_translation.ipynb: Custom Transformer (encoder/decoder) built in PyTorch and trained on English→French.
• tensorflow_german_translation.ipynb: Custom Transformer built in TensorFlow and trained on German→English.

Datasets

• English→French: Hugging Face opus_books (en-fr) loaded via datasets.load_dataset. This is a parallel corpus of book sentences.
• German→English: TensorFlow Datasets wmt14_translate/de-en loaded via tfds.load.

Models And Training Details

PyTorch: English→French

• Architecture: Custom Transformer encoder/decoder with learned token and positional embeddings.
• Key hyperparameters:
  • hidden_size=512, num_attention_heads=8, num_hidden_layers=6
  • intermediate_size=2048, dropout=0.1
  • max_position_embeddings=128
  • batch_size=32, lr=5e-5
• Tokenization: bert-base-uncased tokenizer with <bos>, <eos>, <pad> added.
• Training note: Ran long overnight training (~28 hours) until translations looked reasonable.

TensorFlow: German→English

• Architecture: Custom Transformer modeled after the TensorFlow example notebook (with adjustments).
• Key hyperparameters:
  • num_layers=4, d_model=128, num_heads=8, dff via FFN block
  • dropout=0.1
  • MAX_TOKENS=128, BATCH_SIZE=64
  • Adam with custom schedule (defined; learning-rate schedule used in optimizer)
• Tokenization: TextVectorization with max_tokens=20000, <start>/<end> tokens, tokenizers cached to disk (tokenizer_en2.pkl, tokenizer_de2.pkl).
• Training note: Trained over multiple days (logs in notebook). Used a subset or capped sequence length to keep training time reasonable.

Hardware

Trained primarily on an NVIDIA T4 GPU.

Results (Qualitative)

Both notebooks include example translations on train/validation samples. The English→French model shows strong word-level alignment and captures sentence meaning; the German→English model achieves decent translation quality but is more sensitive to dataset noise and vocabulary size.

Demo Translations

English→French:

German→English:

How To Run

1. Open either notebook in Jupyter/Colab.
2. Run cells top-to-bottom. Training is expensive; for quick verification, reduce dataset size or epochs.

Notes And Future Work

• Better evaluation with BLEU/COMET and a proper train/val/test split.
• Train longer with larger context length (increase MAX_TOKENS) and more data.
• Swap tokenizer for a subword model (BPE/SentencePiece) and compare.

Training Time

Roughly one day of training per model (on an NVIDIA T4), depending on dataset subset size and max sequence length.