Why is this unknown_field_buffer necessary? For skipping over fields in properly nested json you only need a single integer for tracking the current nesting level. And the StreamingParser already checks that the json is properly nested (via the stack field).

Why is this unknown_field_buffer necessary? For skipping over fields in properly nested json you only need a single integer for tracking the current nesting level.

You need to verify that the json is well formed as you ignore things: you need to track if you are expecting a brace vs square brackets, and in which order e.g. ]]]]}]]}}}}}]]] vs }]]}]}}]]]}]]]]

And the StreamingParser already checks that the json is properly nested (via the stack field).

ah ha! I didn't know about this field.

Bit-stack for nested object/map literals (max 255 nestings).

That's a limitation I didn't know about either; seems to be undocumented outside of the code.

And the StreamingParser already checks that the json is properly nested (via the stack field).

ah ha! I didn't know about this field.

Trying to rewrite this PR, the detection for this in StreamingParser doesn't seem to work correctly?

test "mismatched close" {
    var p = TokenStream.init("[102, 111, 111}");
    checkNext(&p, .ArrayBegin);
    checkNext(&p, .Number);
    checkNext(&p, .Number);
    checkNext(&p, .Number);
    testing.expectError(error.UnexpectedClosingBracket, p.next());
}

I get:

.expected error.UnexpectedClosingBracket, found Token{ .ObjectEnd = void }

The detection for this is implemented here:

Closing brackets are handled in both the ValueBegin and ValueEnd states, but this check is missing from the latter. Copying those two checks to ValueEnd state should fix it.

converted to draft while I refactor to try to do #5959 at the same time.