A global-buffer-overflow in DCTStream::transformDataUnit

[WhereisDoujo]

Hello, I found a global-buffer-overflow in DCTStream::transformDataUnit in Stream.cc

Reporter:
WhereisDoujo from Ocean University of China

test platform:
pdf2json Version ：current
OS :Kali 6.6.9-1kali1 (2024-01-08)
kernel: 6.6.9-amd64

reproduced:
poc :poc3.zip

(pdf2json with asan build option)
./pdf2json -xml ./poc

==2128953==ERROR: AddressSanitizer: global-buffer-overflow on address 0x55cb60dec902 at pc 0x55cb6013a630 bp 0x7ffed79c18a0 sp 0x7ffed79c1898
READ of size 1 at 0x55cb60dec902 thread T0                                                                                                                                                                                                                            
    #0 0x55cb6013a62f in DCTStream::transformDataUnit(unsigned short*, int*, unsigned char*) /AFLplusplus/pdf2json/pdf2json/xpdf/Stream.cc:2787:18
    #1 0x55cb60133a21 in DCTStream::readMCURow() /AFLplusplus/pdf2json/pdf2json/xpdf/Stream.cc:2133:4
    #2 0x55cb60132516 in DCTStream::getChar() /AFLplusplus/pdf2json/pdf2json/xpdf/Stream.cc:2038:12
    #3 0x55cb601ed7fd in Object::streamGetChar() /AFLplusplus/pdf2json/pdf2json/xpdf/./Object.h:286:20
    #4 0x55cb601ed7fd in GfxFont::readToUnicodeCMap(Dict*, int, CharCodeToUnicode*) /AFLplusplus/pdf2json/pdf2json/xpdf/GfxFont.cc:319:20
    #5 0x55cb601f3af8 in Gfx8BitFont::Gfx8BitFont(XRef*, char*, Ref, GString*, GfxFontType, Dict*) /AFLplusplus/pdf2json/pdf2json/xpdf/GfxFont.cc:785:3
    #6 0x55cb601eaa1f in GfxFont::makeFont(XRef*, char*, Ref, Dict*) /AFLplusplus/pdf2json/pdf2json/xpdf/GfxFont.cc:132:16
    #7 0x55cb601ff043 in GfxFontDict::GfxFontDict(XRef*, Ref*, Dict*) /AFLplusplus/pdf2json/pdf2json/xpdf/GfxFont.cc:1533:18
    #8 0x55cb601c4532 in GfxResources::GfxResources(XRef*, Dict*, GfxResources*) /AFLplusplus/pdf2json/pdf2json/xpdf/Gfx.cc:282:19
    #9 0x55cb601c64ad in Gfx::Gfx(XRef*, OutputDev*, int, Dict*, double, double, PDFRectangle*, PDFRectangle*, int, int (*)(void*), void*) /AFLplusplus/pdf2json/pdf2json/xpdf/Gfx.cc:449:13
    #10 0x55cb60101eb1 in Page::displaySlice(OutputDev*, double, double, int, int, int, int, int, int, int, Links*, int, Catalog*, int (*)(void*), void*) /AFLplusplus/pdf2json/pdf2json/xpdf/Page.cc:313:13
    #11 0x55cb6010180a in Page::display(OutputDev*, double, double, int, int, int, Links*, int, Catalog*, int (*)(void*), void*) /AFLplusplus/pdf2json/pdf2json/xpdf/Page.cc:265:3
    #12 0x55cb60105d1b in PDFDoc::displayPage(OutputDev*, int, double, double, int, int, int, int, int (*)(void*), void*) /AFLplusplus/pdf2json/pdf2json/xpdf/PDFDoc.cc:319:27
    #13 0x55cb60105d1b in PDFDoc::displayPages(OutputDev*, int, int, double, double, int, int, int, int, int (*)(void*), void*) /AFLplusplus/pdf2json/pdf2json/xpdf/PDFDoc.cc:332:5
    #14 0x55cb600975b3 in main /AFLplusplus/pdf2json/pdf2json/src/pdf2json.cc:275:10
    #15 0x7f21bd76ad8f  (/lib/x86_64-linux-gnu/libc.so.6+0x29d8f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #16 0x7f21bd76ae3f in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x29e3f) (BuildId: 490fef8403240c91833978d494d39e537409b92e)
    #17 0x55cb5ffaa694 in _start (/usr/local/bin/pdf2json+0xb1694) (BuildId: fa31b815da9ddeb6f70aef0d9d9b0aa87af00dab)

0x55cb60dec902 is located 2 bytes after global variable 'dctClip' defined in '/AFLplusplus/pdf2json/pdf2json/xpdf/Stream.cc:1852' (0x55cb60dec600) of size 768
SUMMARY: AddressSanitizer: global-buffer-overflow /AFLplusplus/pdf2json/pdf2json/xpdf/Stream.cc:2787:18 in DCTStream::transformDataUnit(unsigned short*, int*, unsigned char*)                                                                                        
Shadow bytes around the buggy address:
  0x55cb60dec680: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60dec700: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60dec780: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60dec800: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60dec880: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
=>0x55cb60dec900:[f9]f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9 f9
  0x55cb60dec980: f9 f9 f9 f9 f9 f9 f9 f9 00 00 00 00 00 00 00 00
  0x55cb60deca00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60deca80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60decb00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x55cb60decb80: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==2128953==ABORTING