docs: fix and expand README

[bzick]

Summary

Overhaul of readme.md after verifying every example against the actual code (token.go, stream.go, tokenizer.go) and running the doubtful ones through go test.

Fixes (things that were wrong)

• Non-existent methods: stream.GetInt64() / token.GetFloat64() → ValueInt64() / ValueFloat64(); package-level tokenizer.ParseString(...) → parser.ParseString(...).
• Invalid Go: var parser := tokenizer.New() → parser := tokenizer.New().
• Wrong output in the SQL table: token 7 (or) claimed == "and"; fixed alignment and switched string comparisons to ValueUnescapedString().
• fmt.Print with %d → fmt.Printf (%d / %g).
• Unknown-token example showed "One" for input one! → "one".
• Injection example now matches real output (fragments keep the frame quote and surrounding whitespace).
• Framed-string example registers the special string so \" actually unescapes to ".

Improvements

• Added Installation, Table of contents and a Stream API section (many stream/token methods were undocumented).
• Documented AllowNumberUnderscore(), the longest-match rule, Close(), and the "don't store *Token" caveat.
• Dropped the misleading "autodetect whitespace" claim (whitespace is configurable, not autodetected).

AGENTS.md

• Recorded the prose line-wrapping convention: one sentence per line, no mid-sentence hard wraps.

🤖 Generated with Claude Code