which tasks utilises the entity-relations?

[mllife]

I am curious on how entity-relations can be helpful and are you planning to release more task specific models like structeqtable; or any future for a unified model?

[MaoSong2022]

Hi, currently we are exploring how to make use of entity-relations to improve the OCR and understanding capability of MLLMs. For example, we find that MinerU, an excellent open-sourced OCR tool, fails to recognize the entity-relation in some cases. However, it is still a question how can we encode the entity-relationship to something like position information. So, ideas are welcomed to utilizes such information.

FYI, there are some references:

1. https://arxiv.org/pdf/2305.14590, how to extract relationships from document images
2. https://arxiv.org/html/2412.05983v1, our latest work on how to combine multiple domain-experts to form a stronger MLLM