Date Normalization with Seq2Seq & Attention

This repository contains a from-scratch implementation of a Sequence-to-Sequence (Seq2Seq) neural network with Bahdanau Attention, operating strictly at the character level.

The main objective of this project is to map human-written dates, expressed in free-form, diverse, and noisy formats, into the international machine-readable standard ISO 8601.

---

Project Objective

The model receives variable-length input sequences ($T_x = 27$ maximum length) and learns character-level temporal dependencies in order to generate a fixed-length output sequence ($T_y = 10$), corresponding to the format:

YYYY-MM-DD

Input Category	Human-readable Input Example	Machine-readable Target
Written Date	july 21 2026	2026-07-21
Digits and Slashes	9/12/98	1998-12-09
American Format	Jan 5, 2010	2010-01-05

---

Computational Architecture & Optimizations

The network was implemented using TensorFlow, with a fully customized architecture that manually manages tensor operations and hidden states through the following components.

Architecture Overview

flowchart LR
    A[Human Date String] --> B[Character One-Hot Encoding]
    B --> C[Custom LSTM Encoder]
    C --> D[Bahdanau Attention]
    D --> E[Custom LSTM Decoder]
    E --> F[ISO 8601 Date YYYY-MM-DD]

Loading

---

Encoder (Custom LSTM)

The encoder processes the human-readable date string character by character.

When a padding token (<pad>, represented by an all-zero One-Hot vector) is detected, the model prevents unnecessary state updates using tf.where, preserving the previous hidden and cell states.

This allows the model to correctly handle variable-length sequences while maintaining fixed tensor dimensions during training.

---

Bahdanau Attention Mechanism

At each decoding step, the attention mechanism computes a mathematical alignment vector between the current decoder state and the encoder hidden states.

The attention scores corresponding to <pad> tokens receive a large negative masking value (-1e9) before applying Softmax, ensuring that padding positions receive zero attention weight.

This allows the decoder to focus only on meaningful characters from the input sequence.

---

Decoder (LSTM + Dense)

The decoder consumes the dynamically generated context vector from the attention mechanism and sequentially produces the final normalized date representation.

The output sequence is generated character by character until the complete ISO 8601 format is produced.

---

Memory Optimization

⚠️ Critical Memory Optimization

To reduce GPU memory consumption and prevent Out-Of-Memory (OOM) errors, dataset loading and native One-Hot tensor generation are performed using low precision:

dtype=tf.int8

This reduces the tensor memory footprint by approximately 75% compared with:

dtype=tf.int32

The optimization allows larger batch sizes during training while maintaining the required information for character-level encoding.

---

Dataset

The dataset consists of human-readable date expressions mapped to their normalized ISO 8601 representation.

Examples:

Input	Output
July 21 2026	2026-07-21
august 20 2010	2010-08-20
09/12/98	1998-12-09

The input vocabulary contains:

• Alphabetic characters;
• Numeric digits;
• Date separators (/, -, ,);
• Spaces;
• Padding token (<pad>).

The output vocabulary contains:

• Numeric digits;
• Date separator (-).

---

Training & Execution

The following workflow demonstrates how to:

• Load optimized tensors;
• Initialize the custom Seq2Seq model;
• Train the neural network;
• Perform inference;
• Generate attention visualizations.

---

Initializing and Training the Model

import tensorflow as tf
from keras.optimizers.schedules import ExponentialDecay

from model import MyModel
from utils import load_data, inferir_data, save_attention_map


# Load optimized training and validation tensors

X, Y = load_data()

train_and_val_size = int(len(X) * 0.9)


# Initialize the custom model
# 144 LSTM units and 144 attention units

lr_schedule = ExponentialDecay(
    initial_learning_rate=1e-2,
    decay_steps=10000,
    decay_rate=0.9
)


# ExponentialDecay is recommended,
# but other float learning rates can also be tested.
#
# Different hidden layer sizes can also be evaluated.

model = MyModel(
    lstm_units=144,
    atten_units=144,
    learning_rate=lr_schedule
)


# Restore previously saved weights if available

model.compile_and_restore(X, Y)


# Train the model

model.fit(
    X[:train_and_val_size],
    Y[:train_and_val_size],
    batch_size=1024,
    n_epoch=50
)


# Perform inference and generate attention visualization

inferir_data(
    model,
    "june 7 2000"
)

save_attention_map(
    model,
    "june 7 2000"
)


# Generated images are saved inside:
# images/

---

Attention Heatmap Example

---

Monitoring Training

The model is integrated with TensorBoard to monitor training progress.

During training, TensorBoard logs are generated containing:

• Accuracy metrics;
• Average entropy per batch;
• Training performance;
• Validation performance.

To start TensorBoard:

tensorboard --logdir logs

Then access:

http://localhost:6006

---

Model Checkpoints

During training, the model automatically saves checkpoints after each epoch.

The checkpoint system:

• Stores the best performing weights;
• Keeps a maximum of three checkpoints;
• Uses the configured evaluation metrics to determine the best models.

This allows training to be resumed without losing previous progress.

---

Results

Example predictions:

Input	Prediction	Expected
june 7 2000	2000-06-07	2000-06-07
21 de Julho de 2026	2026-07-21	2026-07-21
9/12/98	1998-12-09	1998-12-09

---

Technologies

• Python
• TensorFlow
• Keras
• LSTM Networks
• Sequence-to-Sequence Architecture
• Bahdanau Attention
• TensorBoard
• Character-level Neural Networks

---

Future Improvements

Possible improvements for future versions:

• Replace One-Hot Encoding with Character Embeddings;
• Add Beam Search decoding;
• Expand support for multilingual date formats;
• Compare results against Transformer-based architectures;
• Add automated dataset generation;
• Export the model using TensorFlow SavedModel.

---

License

This project is licensed under the MIT License.