Modified SparK for Handwritten Text Recognition

This repository is a modified version of SparK: Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling.

The modifications were developed as part of a Master's thesis investigating self-supervised pretraining for Handwritten Text Recognition (HTR) using the Document Attention Network (DAN).

We preserved the original SparK implementation as much as possible. The main extension is support for the convolutional encoder used by DAN, allowing the encoder to be pretrained using Masked Image Modelling (MIM) before supervised training for handwritten text recognition.

Main modifications

The main changes compared with the original SparK implementation are:

• Added support for the DAN FCN_Encoder.
• Adapted the SparK encoder/decoder interface to work with the hierarchical feature maps produced by the DAN encoder.
• Added support for pretraining on handwriting image datasets rather than ImageNet.
• Added several masking strategies for experiments with handwritten text.
• Added functionality used to transfer pretrained encoder weights to the downstream DAN handwritten text recognition pipeline.

The DAN encoder consists of convolutional and depthwise-separable convolutional blocks and was originally designed for line- and page-level handwritten text recognition.

Purpose

The purpose of this modification is to investigate whether representations learned from unannotated handwriting images using Masked Image Modeling can improve subsequent supervised handwritten text recognition.

The training pipeline used in the research is conceptually:

Unannotated handwriting images
            |
            v
    SparK / MIM pretraining
            |
            v
      DAN FCN encoder
            |
     pretrained weights
            |
            v
   supervised DAN training
            |
            v
    handwriting recognition

Only the encoder weights learned during MIM pretraining are required for the downstream DAN training.

Experimental use

The modified implementation was used to investigate:

• different masking percentages;
• different masking patterns;
• the influence of pretraining dataset size;
• the influence of supervised dataset size;
• cross-dataset transfer of pretrained representations;
• pretraining on unlabeled historical handwriting.

Experiments were performed using handwriting datasets including IAM, READ-2016 and RIMES, as well as unlabeled historical document images.

Relationship to the original SparK repository

This repository is based on the original SparK implementation by Keyu Tian et al. SparK enables BERT/MAE-style masked image modeling with convolutional neural networks by using sparse convolutions during encoding and a hierarchical decoder for reconstruction.

The remainder of this README is the documentation from the original SparK repository and describes the original implementation, installation and pretraining procedure.

---

SparK: the first successful BERT/MAE-style pretraining on any convolutional networks  

This is the official implementation of ICLR paper Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling, which can pretrain any CNN (e.g., ResNet) in a BERT-style self-supervised manner. We've tried our best to make the codebase clean, short, easy to read, state-of-the-art, and only rely on minimal dependencies.

SparK_demo_22s_4k_wo_bages.1.mp4

   

🔥 News

• A brief introduction (in English) is available on our ICLR poster page! [📹Recorded Video, Poster, and Slides].
• On May. 11th another livestream on OpenMMLab & ReadPaper (bilibili)! [📹Recorded Video]
• On Apr. 27th (UTC+8 8pm) another livestream would be held at OpenMMLab (bilibili)!
• On Mar. 22nd (UTC+8 8pm) another livestream would be held at 极市平台 (bilibili)! [📹Recorded Video]
• The share on TechBeat (将门创投) is scheduled on Mar. 16th (UTC+8 8pm) too! [📹Recorded Video]
• We are honored to be invited by Synced ("机器之心机动组 视频号" on WeChat) to give a talk about SparK on Feb. 27th (UTC+0 11am, UTC+8 7pm), welcome! [📹Recorded Video]
• This work got accepted to ICLR 2023 as a Spotlight (notable-top-25%).
• Other articles: [Synced] [DeepAI] [TheGradient] [Bytedance] [CVers [QbitAI(量子位)] [BAAI(智源)] [机器之心机动组] [极市平台] [ReadPaper笔记]

🕹️ Colab Visualization Demo

Check pretrain/viz_reconstruction.ipynb for visualizing the reconstruction of SparK pretrained models, like:

We also provide pretrain/viz_spconv.ipynb that shows the "mask pattern vanishing" issue of dense conv layers.

What's new here?

🔥 Pretrained CNN beats pretrained Swin-Transformer:

🔥 After SparK pretraining, smaller models can beat un-pretrained larger models:

🔥 All models can benefit, showing a scaling behavior:

🔥 Generative self-supervised pretraining surpasses contrastive learning:

See our paper for more analysis, discussions, and evaluations.

Todo list

catalog

• Pretraining code
• Pretraining toturial for customized CNN model (Tutorial for pretraining your own CNN model)
• Pretraining toturial for customized dataset (Tutorial for pretraining your own dataset)
• Pretraining Colab visualization playground (reconstruction, sparse conv)
• Finetuning code
• Weights & visualization playground in huggingface
• Weights in timm

Pretrained weights (self-supervised; w/o decoder; can be directly finetuned)

Note: for network definitions, we directly use timm.models.ResNet and official ConvNeXt.

reso.: the image resolution; acc@1: ImageNet-1K finetuned acc (top-1)

arch.	reso.	acc@1	#params	flops	weights (self-supervised, without SparK's decoder)
ResNet50	224	80.6	26M	4.1G	resnet50_1kpretrained_timm_style.pth
ResNet101	224	82.2	45M	7.9G	resnet101_1kpretrained_timm_style.pth
ResNet152	224	82.7	60M	11.6G	resnet152_1kpretrained_timm_style.pth
ResNet200	224	83.1	65M	15.1G	resnet200_1kpretrained_timm_style.pth
ConvNeXt-S	224	84.1	50M	8.7G	convnextS_1kpretrained_official_style.pth
ConvNeXt-B	224	84.8	89M	15.4G	convnextB_1kpretrained_official_style.pth
ConvNeXt-L	224	85.4	198M	34.4G	convnextL_1kpretrained_official_style.pth
ConvNeXt-L	384	86.0	198M	101.0G	convnextL_384_1kpretrained_official_style.pth

Pretrained weights (with SparK's UNet-style decoder; can be used to reconstruct images)

arch.	reso.	acc@1	#params	flops	weights (self-supervised, with SparK's decoder)
ResNet50	224	80.6	26M	4.1G	res50_withdecoder_1kpretrained_spark_style.pth
ResNet101	224	82.2	45M	7.9G	res101_withdecoder_1kpretrained_spark_style.pth
ResNet152	224	82.7	60M	11.6G	res152_withdecoder_1kpretrained_spark_style.pth
ResNet200	224	83.1	65M	15.1G	res200_withdecoder_1kpretrained_spark_style.pth
ConvNeXt-S	224	84.1	50M	8.7G	cnxS224_withdecoder_1kpretrained_spark_style.pth
ConvNeXt-L	384	86.0	198M	101.0G	cnxL384_withdecoder_1kpretrained_spark_style.pth

Installation & Running

We highly recommended you to use torch==1.10.0, torchvision==0.11.1, and timm==0.5.4 for reproduction. Check INSTALL.md to install all pip dependencies.

• Loading pretrained model weights in 3 lines

# download our weights `resnet50_1kpretrained_timm_style.pth` first
import torch, timm
res50, state = timm.create_model('resnet50'), torch.load('resnet50_1kpretrained_timm_style.pth', 'cpu')
res50.load_state_dict(state.get('module', state), strict=False)     # just in case the model weights are actually saved in state['module']

• Pretraining

  • any ResNet or ConvNeXt on ImageNet-1k:  see pretrain/
  • your own CNN model:  see pretrain/, especially pretrain/models/custom.py

• Finetuning

  • any ResNet or ConvNeXt on ImageNet-1k:  check downstream_imagenet/ for subsequent instructions.
  • ResNets on COCO:  see downstream_d2/
  • ConvNeXts on COCO:  see downstream_mmdet/

Acknowledgement

We referred to these useful codebases:

• BEiT, MAE, ConvNeXt
• timm, MoCoV2, Detectron2, MMDetection

License

This project is under the MIT license. See LICENSE for more details.

Citation

If you found this project useful, you can kindly give us a star ⭐, or cite us in your work 📖:

@Article{tian2023designing,
  author  = {Keyu Tian and Yi Jiang and Qishuai Diao and Chen Lin and Liwei Wang and Zehuan Yuan},
  title   = {Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling},
  journal = {arXiv:2301.03580},
  year    = {2023},
}