perf: second optimization round (render -78%, JSON -89%, geomean -57%) + low NodeMap API break

AGENTS.md

@@ -22,6 +22,7 @@ This repo is a library, not an app. The root package exposes the public entry po
 | `renderer/` | Schema/mock sample generation |
 | `orderedmap/` | Stable insertion-ordered map wrapper used throughout models/rendering |
 | `json/` | YAML-node to ordered JSON conversion |
+| `internal/jsonnode/` | Direct JSON parser building the exact `yaml.Node` tree yaml v4 builds; used for JSON specs, declines to `yaml.Unmarshal` otherwise |
 | `tests/` | Cross-package integration and benchmark coverage, especially sibling-ref behavior |
 | `test_specs/` | Realistic fixtures and regression specs used across packages |
 
@@ -61,6 +62,7 @@ This repo is a library, not an app. The root package exposes the public entry po
 | `go test ./bundler -run TestBundle` | Target bundler regressions |
 | `go test ./what-changed/... -run Test` | Target diff/breaking-rule regressions |
 | `go test -bench . ./index ./datamodel/low/... ./what-changed/...` | Run benchmarks in hot paths |
+| `go test -run xxx -bench BenchmarkPipeline -count 6 .` | End-to-end build/render/compare/bundle benchmarks on large specs (compare runs with `benchstat`) |
 | `GOCACHE=/tmp/go-build go test ./...` | Useful in restricted sandboxes where default Go build cache is not writable |
 
 ## Testing Caveats

MIGRATING.md

@@ -0,0 +1,177 @@
+# Migrating
+
+This file lists breaking changes to the libopenapi Go API and how to update code for each one.
+
+## Upgrading past v0.40
+
+This release makes two changes to the low-level model API. They don't change parsing, rendering, bundling or
+what-changed output.
+
+You don't need to change anything if your code only uses:
+- the high-level model
+- `GetNodes()`
+- `AddNode()`
+
+### `NodeMap.Nodes` is now a `*low.NodeLines`
+
+Every low-level model embeds `low.NodeMap`, which records the YAML nodes the model was built from, keyed by line
+number. `Nodes` was a `*sync.Map` keyed by `int`. It is now a `*low.NodeLines`. A `NodeLines` records writes as they
+happen and builds its line index the first time it is read. Most models are built and never read, so building a
+document and comparing documents with what-changed allocate less.
+
+| v0.40 | now |
+|---|---|
+| `Nodes *sync.Map` | `Nodes *low.NodeLines` |
+| `Nodes.Range(func(key, value any) bool)` | `Nodes.Range(func(line int, value any) bool)`, visiting lines in ascending order |
+| `Nodes.Load(key any) (any, bool)` | `Nodes.Load(line int) (any, bool)` |
+| `Nodes.Store(key, value any)` | `Nodes.Store(line int, value any)` |
+| `low.NodeMap{Nodes: &sync.Map{}}` | `low.NodeMap{Nodes: &low.NodeLines{}}` (the zero value is ready to use) |
+| `low.ExtractNodes(ctx, root) *sync.Map` | returns `*low.NodeLines` |
+| `low.ExtractNodesRecursive(ctx, root) *sync.Map` | returns `*low.NodeLines` |
+| `low.ExtractExtensionNodes(ctx, extensions, nodes *sync.Map)` | takes `*low.NodeLines` |
+| `low.MergeRecursiveNodesIfLineAbsent(dst *sync.Map, node)` | takes `*low.NodeLines` |
+
+`NodeLines` does not have `sync.Map`'s other methods:
+- `Delete` and `Clear`
+- `LoadOrStore` and `LoadAndDelete`
+- `Swap`, `CompareAndSwap` and `CompareAndDelete`
+
+A line's value is a `*yaml.Node`, or a `[]*yaml.Node` when several nodes share the line, as before.
+
+This program reads the nodes of an `info` object:
+
+```go
+package main
+
+import (
+	"fmt"
+
+	"github.com/pb33f/libopenapi"
+	"go.yaml.in/yaml/v4"
+)
+
+const spec = `openapi: 3.1.0
+info:
+  title: Burger Shop
+  version: 1.0.0
+paths: {}
+`
+
+func main() {
+	doc, err := libopenapi.NewDocument([]byte(spec))
+	if err != nil {
+		panic(err)
+	}
+	model, err := doc.BuildV3Model()
+	if err != nil {
+		panic(err)
+	}
+	info := model.Model.GoLow().Info.Value
+
+	// Range passes each line as an int, in ascending order. A line holds a *yaml.Node, or a
+	// []*yaml.Node when several nodes share it.
+	info.Nodes.Range(func(line int, value any) bool {
+		switch v := value.(type) {
+		case *yaml.Node:
+			fmt.Printf("line %d: %s\n", line, v.Value)
+		case []*yaml.Node:
+			for _, n := range v {
+				fmt.Printf("line %d: %s\n", line, n.Value)
+			}
+		}
+		return true
+	})
+
+	// Load takes the line as an int.
+	if value, ok := info.Nodes.Load(4); ok {
+		fmt.Printf("line 4 holds %d nodes\n", len(value.([]*yaml.Node)))
+	}
+
+	// GetNodes is unchanged.
+	fmt.Printf("GetNodes: %d lines\n", len(info.GetNodes()))
+}
+```
+
+Output:
+
+```
+line 3: title
+line 3: Burger Shop
+line 4: version
+line 4: 1.0.0
+line 4 holds 2 nodes
+GetNodes: 2 lines
+```
+
+### `NodeReference.Context` is removed
+
+`low.NodeReference[T]` had a `Context` field. libopenapi only ever set it on the operations of a `PathItem`:
+- `Get`, `Put`, `Post`, `Delete`, `Options`, `Head`, `Patch`, `Trace` and `Query`
+- the operations in `AdditionalOperations`
+
+On those references it held the context the operation was resolved and built with. On every other reference it
+was nil.
+
+Read that context from the operation instead:
+
+| v0.40 | now |
+|---|---|
+| `pathItem.Get.Context` | `pathItem.Get.Value.GetContext()` |
+
+`GetContext()` returns the context passed to the operation's `Build`, which is the context `Context` held. If an
+operation is a `$ref` to another file, that context carries the other file's index.
+
+If your code sets `Context` on references it creates, keep the context next to the reference in a type of your
+own, for example `struct { Ref low.NodeReference[T]; Ctx context.Context }`.
+
+This program reads the context the `get` operation of `/burgers` was built with:
+
+```go
+package main
+
+import (
+	"fmt"
+
+	"github.com/pb33f/libopenapi"
+	"github.com/pb33f/libopenapi/index"
+)
+
+const spec = `openapi: 3.1.0
+info:
+  title: Burger Shop
+  version: 1.0.0
+paths:
+  /burgers:
+    get:
+      operationId: listBurgers
+      responses:
+        '200':
+          description: OK
+`
+
+func main() {
+	doc, err := libopenapi.NewDocument([]byte(spec))
+	if err != nil {
+		panic(err)
+	}
+	model, err := doc.BuildV3Model()
+	if err != nil {
+		panic(err)
+	}
+	pathItem := model.Model.GoLow().Paths.Value.FindPath("/burgers").Value
+
+	// was: ctx := pathItem.Get.Context
+	ctx := pathItem.Get.Value.GetContext()
+
+	idx := ctx.Value(index.FoundIndexKey).(*index.SpecIndex)
+	fmt.Println("operation:", pathItem.Get.Value.OperationId.Value)
+	fmt.Println("built with the operation's index:", idx == pathItem.Get.Value.GetIndex())
+}
+```
+
+Output:
+
+```
+operation: listBurgers
+built with the operation's index: true
+```

README.md

@@ -76,6 +76,7 @@ See all the documentation at https://pb33f.io/libopenapi/
 - [Parsing Code](https://pb33f.io/libopenapi/parsing-code/)
 - [FAQ](https://pb33f.io/libopenapi/faq/)
 - [About libopenapi](https://pb33f.io/libopenapi/about/)
+- [Migrating: breaking API changes by version](MIGRATING.md)
 
 ### Generating TypeScript models
 

cache.go

@@ -4,6 +4,7 @@
 package libopenapi
 
 import (
+	"github.com/pb33f/libopenapi/datamodel/high"
 	highbase "github.com/pb33f/libopenapi/datamodel/high/base"
 	"github.com/pb33f/libopenapi/datamodel/low"
 	lowbase "github.com/pb33f/libopenapi/datamodel/low/base"
@@ -15,14 +16,15 @@ import (
 //
 // Calling it is not required to release memory: caches keyed by YAML nodes or model objects hold them weakly,
 // so a document is reclaimed as soon as the caller drops it. Use it to force hashes to be recalculated after
-// YAML nodes or low-level models were modified in place, or to empty the string-keyed caches (compiled JSONPath
-// expressions, schema quick hashes and remote content types). It is safe to call while other goroutines parse,
-// build or compare documents.
+// YAML nodes or low-level models were modified in place, or to empty the content-keyed caches (compiled
+// JSONPath expressions, schema quick hashes, encoded values and remote content types). It is safe to call
+// while other goroutines parse, build or compare documents.
 func ClearAllCaches() {
 	low.ClearHashCache()              // model and YAML node hashes
 	lowbase.ClearSchemaQuickHashMap() // SchemaQuickHashMap
 	index.ClearHashCache()            // nodeHashCache
 	index.ClearContentDetectionCache()
 	highbase.ClearInlineRenderingTracker()
+	high.ClearEncodeCache()
 	utils.ClearJSONPathCache()
 }