Parsing JSON is a minefield

[jorektheglitch]

Recently I read article Parsing JSON is a minefield and got interested in testing json_stream with very detailed dataset from article.

There is two news!
Good one: json_stream passed all positive cases (at least with json_stream.load)
Not a good one: apparently there is some problems with json_stream.visit and with json_stream.load in negative cases, according to tests. Some of them don't seems like a big problem (like threatning [ ,""] as valid JSON), but others seems really problematic (like treatning [3[4]] as valid JSON).

There is repository with tests I wrote from article's materials - json-stream-tests

And test log attached as well.
2024-01-17 00:53:05.284817606.log

[daggaz]

A full year later, and I've spent some time looking at this.

The positive cases for visit are all to do with it not handling the root element being a primitive. PR coming for that.

The negative cases fall into 3 categories:

• valid JSON with invalid trailing stuff (I'm happy with this, as json-stream has support for reading up to an error, I think the error would occur if you constructed a new reader with the stream at this position)
• allowing unescaped things (I'm okay with this)
• missing commas in lists (this one I feel should error out)

@smheidrich any fixes would need to be implemented in the rust tokenizer I think?

[daggaz]

that PR should address the positive visit cases

[daggaz]

I PR'd your repo:
jorektheglitch/json_stream_tests#1

• added some additional checks to the positive cases to verify the result from json_stream is the same as from json.
• added a print to the negative cases so we can see what the result was
• added a github action to run the test suite against latest json-stream

[jorektheglitch]

@daggaz sorry for huge response time and thank you for PR, will watch it today 👌

[daggaz]

@jorektheglitch bump

[neurotensin]

I have an example of
stock data that doesn't parse,:

File "lib/python3.10/site-packages/json_stream/loader.py", line 8, in load
token_stream = tokenizer(fp)
File "lib/python3.10/site-packages/json_stream/iterators.py", line 13, in readinto
buffer[:length], self.remainder = chunk[:length], chunk[length:]
TypeError: can assign only bytes, buffers, or iterables of ints in range(0, 256)

It's a stream of json objects 22M uncompressed, so uploaded if someone wants to take a crack.

test_datacludge_29oct25.gz

[daggaz]

This file doesn't break for me when opened and parsed like this:

with open('test_datacludge_29oct25') as f:
    obj = load(f)
    print(obj)
    to_standard_types(obj)
    print(obj)

But, @neurotensin, you seem to be hitting the iterable code path (vs the file-like object path). How are you opening the file and invoking json_stream?

[daggaz]

I think that you are feeding an iterable of strings? As mentioned in the docs for iterators, the iterator must yield bytes.

[neurotensin]

I found a solution but it used pandas that treats this blob as a dict.

This file doesn't break for me when opened and parsed like this:

with open('test_datacludge_29oct25') as f:
obj = load(f)
print(obj)
to_standard_types(obj)
print(obj)
But, @neurotensin, you seem to be hitting the iterable code path (vs the file-like object path). How are you opening the file and invoking json_stream?

when I run like that I get:
obj = json.load(f)
gives....
Traceback (most recent call last):
File "", line 2, in
File python3.10/json/init.py", line 293, in load
return loads(fp.read(),
File python3.10/json/init.py", line 346, in loads
return _default_decoder.decode(s)
File python3.10/json/decoder.py", line 340, in decode
raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 1 column 1806 (char 1805)

The solution I found used pandas but treating it as a dict NOT json. The top level gives keys and passed to an index it works:

def quote2stream(quote):
odfa=[]
topkeys=quote.keys()
for x in topkeys:
odfa.append(pd.DataFrame.from_dict(odict,orient='index')

odict is an artificial creation for this specific data source(empty dict with keys), which we populate with current values, prevents having to parse the whole thing. I appreciate your observation though

[daggaz]

I think that you are feeding an iterable of strings? As mentioned in the docs for iterators, the iterator must yield bytes.

This is now supported in json-stream 2.4.1 (see #71).