db: introduce support for commit hooks

[giorio94]

Introduce a new [statedb.New] option that allows to register commit hooks that get invoked every time that a transaction is committed, before releasing the associated locks. The hooks are strictly read only, and cannot abort the transaction. Additionally provide the possibility of registering the hooks through [statedb.Cell].

One possible use-case for the hooks is to validate properties of the entries of specific tables, for testing purposes. However, they may also unlock other use-cases, including the possibility of synchronously dumping the state to disk, and so on.

[github-actions]

$ make
go build ./...
go: downloading github.com/cilium/hive v1.0.4
go: downloading go.yaml.in/yaml/v3 v3.0.4
go: downloading golang.org/x/time v0.15.0
go: downloading github.com/spf13/cobra v1.10.2
go: downloading github.com/spf13/pflag v1.0.10
go: downloading github.com/cilium/stream v0.0.1
go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de
go: downloading github.com/spf13/viper v1.18.2
go: downloading go.uber.org/dig v1.17.1
go: downloading golang.org/x/term v0.16.0
go: downloading github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc
go: downloading github.com/mitchellh/mapstructure v1.5.0
go: downloading golang.org/x/sys v0.17.0
go: downloading golang.org/x/tools v0.17.0
go: downloading github.com/spf13/cast v1.6.0
go: downloading github.com/fsnotify/fsnotify v1.7.0
go: downloading github.com/sagikazarmark/slog-shim v0.1.0
go: downloading github.com/spf13/afero v1.11.0
go: downloading github.com/subosito/gotenv v1.6.0
go: downloading github.com/hashicorp/hcl v1.0.0
go: downloading gopkg.in/ini.v1 v1.67.0
go: downloading github.com/magiconair/properties v1.8.7
go: downloading github.com/pelletier/go-toml/v2 v2.1.0
go: downloading gopkg.in/yaml.v3 v3.0.1
go: downloading golang.org/x/text v0.14.0
STATEDB_VALIDATE=1 go test ./... -cover -vet=all -test.count 1
go: downloading github.com/stretchr/testify v1.11.1
go: downloading go.uber.org/goleak v1.3.0
go: downloading golang.org/x/exp v0.0.0-20240119083558-1b970713d09a
go: downloading github.com/pmezard/go-difflib v1.0.1-0.20181226105442-5d4384ee4fb2
ok  	github.com/cilium/statedb	298.873s	coverage: 79.7% of statements
ok  	github.com/cilium/statedb/index	0.006s	coverage: 48.1% of statements
ok  	github.com/cilium/statedb/internal	0.010s	coverage: 42.9% of statements
ok  	github.com/cilium/statedb/lpm	4.667s	coverage: 76.4% of statements
ok  	github.com/cilium/statedb/part	76.543s	coverage: 86.3% of statements
ok  	github.com/cilium/statedb/reconciler	0.273s	coverage: 93.3% of statements
	github.com/cilium/statedb/reconciler/benchmark		coverage: 0.0% of statements
	github.com/cilium/statedb/reconciler/example		coverage: 0.0% of statements
go test -race ./... -test.count 1
ok  	github.com/cilium/statedb	41.910s
ok  	github.com/cilium/statedb/index	1.015s
ok  	github.com/cilium/statedb/internal	1.027s
ok  	github.com/cilium/statedb/lpm	2.826s
ok  	github.com/cilium/statedb/part	37.510s
ok  	github.com/cilium/statedb/reconciler	1.344s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go test ./... -bench . -benchmem -test.run xxx
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb
cpu: AMD EPYC 7763 64-Core Processor                
BenchmarkDB_WriteTxn_1-4                      	  738175	      1585 ns/op	    631039 objects/sec	     704 B/op	      17 allocs/op
BenchmarkDB_WriteTxn_10-4                     	 1893451	       634.4 ns/op	   1576201 objects/sec	     386 B/op	       8 allocs/op
BenchmarkDB_WriteTxn_100-4                    	 2151789	       546.8 ns/op	   1828740 objects/sec	     349 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_1000-4                   	 2157883	       558.8 ns/op	   1789546 objects/sec	     350 B/op	       7 allocs/op
BenchmarkDB_WriteTxn_100_SecondaryIndex-4     	 1300070	       930.4 ns/op	   1074860 objects/sec	     507 B/op	      12 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_100Tables-4   	  900174	      1265 ns/op	    1128 B/op	       6 allocs/op
BenchmarkDB_WriteTxn_CommitOnly_1Table-4      	 1479670	       811.0 ns/op	     240 B/op	       6 allocs/op
BenchmarkDB_NewWriteTxn-4                     	 1493188	       802.3 ns/op	     216 B/op	       5 allocs/op
BenchmarkDB_WriteTxnCommit100-4               	  922058	      1258 ns/op	    1112 B/op	       6 allocs/op
BenchmarkDB_NewReadTxn-4                      	640438803	         1.873 ns/op	       0 B/op	       0 allocs/op
BenchmarkDB_Modify-4                          	    1928	    604020 ns/op	   1655575 objects/sec	  374571 B/op	    8080 allocs/op
BenchmarkDB_GetInsert-4                       	    1717	    705000 ns/op	   1418439 objects/sec	  358570 B/op	    8080 allocs/op
BenchmarkDB_RandomInsert-4                    	    2140	    566964 ns/op	   1763780 objects/sec	  350564 B/op	    7080 allocs/op
BenchmarkDB_RandomReplace-4                   	    1044	   1155617 ns/op	    865338 objects/sec	  520507 B/op	   13110 allocs/op
BenchmarkDB_SequentialInsert-4                	    2190	    552062 ns/op	   1811389 objects/sec	  350568 B/op	    7080 allocs/op
BenchmarkDB_SequentialInsert_Prefix-4         	     495	   2284135 ns/op	    437803 objects/sec	 2840378 B/op	   45536 allocs/op
BenchmarkDB_Changes_Baseline-4                	    1659	    719542 ns/op	   1389772 objects/sec	  444574 B/op	    9201 allocs/op
BenchmarkDB_Changes-4                         	     934	   1279992 ns/op	    781255 objects/sec	  661168 B/op	   12372 allocs/op
BenchmarkDB_RandomLookup-4                    	   23731	     51174 ns/op	  19541019 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_SequentialLookup-4                	   29313	     40967 ns/op	  24410135 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_Prefix_SecondaryIndex-4           	    6933	    165495 ns/op	   6042478 objects/sec	  124952 B/op	    1026 allocs/op
BenchmarkDB_FullIteration_All-4               	     966	   1377486 ns/op	  72596006 objects/sec	     104 B/op	       4 allocs/op
BenchmarkDB_FullIteration_Prefix-4            	     862	   1534009 ns/op	  65188655 objects/sec	     136 B/op	       5 allocs/op
BenchmarkDB_FullIteration_Get-4               	     230	   5161788 ns/op	  19373133 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_Get_Secondary-4     	      96	  12122167 ns/op	   8249350 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_FullIteration_ReadTxnGet-4        	     223	   5484947 ns/op	  18231716 objects/sec	       0 B/op	       0 allocs/op
BenchmarkDB_PropagationDelay-4                	  699765	      1660 ns/op	        14.00 50th_µs	        17.00 90th_µs	        58.00 99th_µs	     878 B/op	      20 allocs/op
BenchmarkDB_WriteTxn_100_LPMIndex-4           	  449026	      2379 ns/op	    420307 objects/sec	    1606 B/op	      37 allocs/op
BenchmarkDB_WriteTxn_1_LPMIndex-4             	  132483	     15272 ns/op	     65478 objects/sec	   13638 B/op	      82 allocs/op
BenchmarkDB_LPMIndex_Get-4                    	     355	   3710294 ns/op	   2695204 objects/sec	       0 B/op	       0 allocs/op
BenchmarkWatchSet_4-4                         	 2183486	       531.7 ns/op	     296 B/op	       4 allocs/op
BenchmarkWatchSet_16-4                        	  756760	      1582 ns/op	    1096 B/op	       5 allocs/op
BenchmarkWatchSet_128-4                       	   88540	     13582 ns/op	    8904 B/op	       5 allocs/op
BenchmarkWatchSet_1024-4                      	    8826	    134295 ns/op	   73743 B/op	       5 allocs/op
PASS
ok  	github.com/cilium/statedb	45.161s
PASS
ok  	github.com/cilium/statedb/index	0.004s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/internal
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_SortableMutex-4   	 6132109	       195.9 ns/op	       0 B/op	       0 allocs/op
PASS
ok  	github.com/cilium/statedb/internal	1.206s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/lpm
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_txn_insert/batchSize=1-4         	    1848	    645188 ns/op	   1549936 objects/sec	  822407 B/op	   13975 allocs/op
Benchmark_txn_insert/batchSize=10-4        	    3166	    372946 ns/op	   2681353 objects/sec	  369194 B/op	    6668 allocs/op
Benchmark_txn_insert/batchSize=100-4       	    3400	    349457 ns/op	   2861579 objects/sec	  329613 B/op	    6027 allocs/op
Benchmark_txn_delete/batchSize=1-4         	    1527	    764326 ns/op	   1308343 objects/sec	 1270473 B/op	   13976 allocs/op
Benchmark_txn_delete/batchSize=10-4        	    3343	    366572 ns/op	   2727979 objects/sec	  356419 B/op	    5769 allocs/op
Benchmark_txn_delete/batchSize=100-4       	    3679	    326787 ns/op	   3060096 objects/sec	  270754 B/op	    5038 allocs/op
Benchmark_LPM_Lookup-4                     	    9148	    130988 ns/op	   7634259 objects/sec	       0 B/op	       0 allocs/op
Benchmark_LPM_All-4                        	  139119	      9025 ns/op	 110800558 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_Prefix-4                     	  131221	      9138 ns/op	 109428779 objects/sec	      32 B/op	       1 allocs/op
Benchmark_LPM_LowerBound-4                 	  243897	      4739 ns/op	 105509389 objects/sec	     288 B/op	       2 allocs/op
PASS
ok  	github.com/cilium/statedb/lpm	11.978s
goos: linux
goarch: amd64
pkg: github.com/cilium/statedb/part
cpu: AMD EPYC 7763 64-Core Processor                
Benchmark_Set_Singleton_Create-4              	20930085	        55.92 ns/op	      24 B/op	       1 allocs/op
Benchmark_Set_Singleton_Has-4                 	100000000	        11.23 ns/op	       0 B/op	       0 allocs/op
Benchmark_StringMap_Txn_Insert-4              	    8409	    138914 ns/op	   7198715 items/sec	   98250 B/op	    1306 allocs/op
Benchmark_Uint64Map_Random-4                  	    1860	    637197 ns/op	   1569372 items/sec	 1321662 B/op	    6041 allocs/op
Benchmark_Uint64Map_Sequential-4              	    1916	    701141 ns/op	   1426248 items/sec	 1703089 B/op	    5753 allocs/op
Benchmark_Uint64Map_Sequential_Insert-4       	    2196	    562993 ns/op	   1776219 items/sec	 1695085 B/op	    4752 allocs/op
Benchmark_Uint64Map_Sequential_Txn_Insert-4   	    9878	    120910 ns/op	   8270622 items/sec	   90464 B/op	    2031 allocs/op
Benchmark_Uint64Map_Random_Insert-4           	    2132	    559654 ns/op	   1786820 items/sec	 1313225 B/op	    5037 allocs/op
Benchmark_Uint64Map_Random_Txn_Insert-4       	    6312	    185944 ns/op	   5377960 items/sec	  117933 B/op	    2411 allocs/op
Benchmark_Insert_RootOnlyWatch-4              	   10000	    114250 ns/op	   8752772 objects/sec	   75552 B/op	    2036 allocs/op
Benchmark_Insert-4                            	    8346	    139032 ns/op	   7192570 objects/sec	   84256 B/op	    3067 allocs/op
Benchmark_WatchReplace-4                      	   10000	    116182 ns/op	   8607149 objects/sec	   65346 B/op	    2011 allocs/op
Benchmark_Modify-4                            	   12902	     94421 ns/op	  10590847 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_GetInsert-4                         	   10000	    119433 ns/op	   8372893 objects/sec	   58056 B/op	    1007 allocs/op
Benchmark_Replace-4                           	33501170	        36.35 ns/op	  27510313 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Replace_RootOnlyWatch-4             	33692512	        35.28 ns/op	  28347374 objects/sec	       0 B/op	       0 allocs/op
Benchmark_txn_1-4                             	 7996836	       149.7 ns/op	   6680629 objects/sec	      64 B/op	       3 allocs/op
Benchmark_txn_10-4                            	10732447	       109.1 ns/op	   9161752 objects/sec	      76 B/op	       2 allocs/op
Benchmark_txn_100-4                           	12149876	        98.21 ns/op	  10182335 objects/sec	      67 B/op	       2 allocs/op
Benchmark_txn_1000-4                          	10683165	       111.0 ns/op	   9011020 objects/sec	      65 B/op	       2 allocs/op
Benchmark_txn_delete_1-4                      	 5249306	       227.0 ns/op	   4405445 objects/sec	     632 B/op	       3 allocs/op
Benchmark_txn_delete_10-4                     	11108098	       107.4 ns/op	   9314074 objects/sec	     103 B/op	       1 allocs/op
Benchmark_txn_delete_100-4                    	13348309	        88.46 ns/op	  11304911 objects/sec	      35 B/op	       1 allocs/op
Benchmark_txn_delete_1000-4                   	13578986	        88.63 ns/op	  11282337 objects/sec	      28 B/op	       1 allocs/op
Benchmark_Get-4                               	   42488	     27451 ns/op	  36428941 objects/sec	       0 B/op	       0 allocs/op
Benchmark_GetWatch-4                          	   41368	     29001 ns/op	  34481930 objects/sec	       0 B/op	       0 allocs/op
Benchmark_All-4                               	  141669	      8473 ns/op	 118016058 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_All-4                      	  156552	      7717 ns/op	 129577167 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Iterator_Next-4                     	  150525	      7884 ns/op	 126846986 objects/sec	     896 B/op	       1 allocs/op
Benchmark_Hashmap_Insert-4                    	   14596	     81879 ns/op	  12213158 objects/sec	   74264 B/op	      20 allocs/op
Benchmark_Hashmap_Get_Uint64-4                	  132968	      9016 ns/op	 110909609 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Hashmap_Get_Bytes-4                 	  106318	     11116 ns/op	  89957817 objects/sec	       0 B/op	       0 allocs/op
Benchmark_Delete_Random-4                     	      79	  16065753 ns/op	   6224420 objects/sec	 2539429 B/op	  102756 allocs/op
Benchmark_find16-4                            	213635923	         5.620 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex16-4                       	87083972	        13.58 ns/op	       0 B/op	       0 allocs/op
Benchmark_find64-4                            	295863688	         4.055 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_hit-4                   	295480542	         4.061 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex64_miss-4                  	295797870	         4.126 ns/op	       0 B/op	       0 allocs/op
Benchmark_find4-4                             	421111351	         2.886 ns/op	       0 B/op	       0 allocs/op
Benchmark_findIndex4-4                        	319373264	         3.747 ns/op	       0 B/op	       0 allocs/op
BenchmarkSmallWriteTxn/updates_1-4            	  464031	      2534 ns/op	    3529 B/op	       4 allocs/op
BenchmarkSmallWriteTxn/updates_2-4            	  308668	      3555 ns/op	    4742 B/op	       6 allocs/op
BenchmarkSmallWriteTxn/updates_4-4            	  208134	      5616 ns/op	    7155 B/op	      11 allocs/op
BenchmarkSmallWriteTxn/updates_8-4            	  125180	      9647 ns/op	   11927 B/op	      20 allocs/op
BenchmarkSmallWriteTxn/updates_16-4           	   67413	     17583 ns/op	   21290 B/op	      38 allocs/op
PASS
ok  	github.com/cilium/statedb/part	55.379s
PASS
ok  	github.com/cilium/statedb/reconciler	0.004s
?   	github.com/cilium/statedb/reconciler/benchmark	[no test files]
?   	github.com/cilium/statedb/reconciler/example	[no test files]
go run ./reconciler/benchmark -quiet
1000000 objects reconciled in 1.95 seconds (batch size 1000)
Throughput 514070.63 objects per second
568MB total allocated, 6015217 in-use objects, 239MB bytes in use