Add tests (and fix bugs) for commands WRONGTYPE'ing appropriately against Vector Sets#1935
Open
kevin-montrose wants to merge 59 commits into
Open
Add tests (and fix bugs) for commands WRONGTYPE'ing appropriately against Vector Sets#1935kevin-montrose wants to merge 59 commits into
kevin-montrose wants to merge 59 commits into
Conversation
… (as expected), but saved me some typing
…oing to require more serious work, so dropping a commit
19 tasks
…d to make sure we don't accidentally nest transactions
…empted) concurrent modification of the underlying key
…existing key (because that's how RI.CREATE works today, even if its a little strange for a Redis command)
… are disabled as BITOP appears to behave poorly in transactions, will open a separate issue for that
Contributor
There was a problem hiding this comment.
Pull request overview
This PR tightens WRONGTYPE behavior across Garnet when operating on Vector Set keys (and vice versa), adds broad test coverage for these expectations, and introduces overwrite/cleanup mechanics to ensure Vector Sets don’t leak resources when overwritten by string commands.
Changes:
- Added comprehensive tests to ensure (1) Vector Set commands WRONGTYPE on non-Vector Set keys, (2) most non-Vector Set commands WRONGTYPE on Vector Set keys, and (3) overwrite commands free Vector Set resources.
- Implemented
VCARDandVISMEMBERend-to-end; sketchedVLINKS,VRANDMEMBER, andVSETATTRenough to reach WRONGTYPE paths. - Propagated WRONGTYPE status consistently through storage sessions, RESP handlers, and (where needed) Tsavorite RMW plumbing; added overwrite retry logic (DELETE + SET) for commands that intentionally overwrite Vector Sets.
Reviewed changes
Copilot reviewed 46 out of 46 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| test/standalone/Garnet.test/TestUtils.cs | Adds server creation knobs (compaction, segment sizing) used by overwrite/eviction tests. |
| test/standalone/Garnet.test/RespTests.cs | Minor assertion cleanup (IsNull). |
| test/standalone/Garnet.test.vectorset/VectorSetWrongTypeTests.cs | New broad meta-test suite ensuring WRONGTYPE correctness across key-taking commands. |
| test/standalone/Garnet.test.vectorset/VectorSetOverwriteTests.cs | New suite validating overwriting Vector Sets triggers cleanup (in-memory and evicted-to-disk scenarios). |
| test/standalone/Garnet.test.vectorset/RespVectorSetTests.cs | Adds TYPE coverage for Vector Sets (RedisType.VectorSet). |
| test/standalone/Garnet.test.rangeindex/RespRangeIndexTests.cs | Removes/defers a prior “SET over RI must WRONGTYPE” assumption (now allows overwrite). |
| test/standalone/Garnet.test.acl/Resp/ACL/RespCommandTests.cs | Updates ACL command checks to match new Vector Set command behaviors/return shapes. |
| libs/storage/Tsavorite/cs/src/core/Index/Tsavorite/Implementation/InternalRMW.cs | Fixes propagation so WrongType maps to OperationStatus.WRONG_TYPE. |
| libs/storage/Tsavorite/cs/src/core/Index/StoreFunctions/StoreFunctions.cs | Exposes recordTriggers publicly so it can be passed to compaction. |
| libs/server/Storage/Session/ObjectStore/SortedSetOps.cs | Preserves WRONGTYPE across multi-key collect operations. |
| libs/server/Storage/Session/ObjectStore/SortedSetGeoOps.cs | Returns actual status (incl. WRONGTYPE) instead of forcing NOTFOUND. |
| libs/server/Storage/Session/ObjectStore/SetOps.cs | Early-exits union on WRONGTYPE. |
| libs/server/Storage/Session/ObjectStore/ListOps.cs | Propagates WRONGTYPE in list move paths. |
| libs/server/Storage/Session/ObjectStore/HashOps.cs | Preserves WRONGTYPE across multi-key hash collect operations. |
| libs/server/Storage/Session/ObjectStore/Common.cs | Maps Status.IsWrongType to GarnetStatus.WRONGTYPE. |
| libs/server/Storage/Session/ObjectStore/AdvancedOps.cs | Ensures RMW wrapper returns WRONGTYPE when Tsavorite reports it. |
| libs/server/Storage/Session/MainStore/VectorStoreOps.cs | Adds VCARD, VISMEMBER, and stubs for links/randmember/attr; integrates with vector index reads. |
| libs/server/Storage/Session/MainStore/RangeIndexOps.cs | Returns WRONGTYPE on create failures due to mismatched existing type. |
| libs/server/Storage/Session/MainStore/MainStoreOps.cs | Propagates WRONGTYPE for string operations (GET/GETEX/GETDEL/etc.) and some multi-key flows (LCS). |
| libs/server/Storage/Session/MainStore/HyperLogLogOps.cs | Propagates WRONGTYPE for PFCOUNT/PFMERGE input keys. |
| libs/server/Storage/Session/MainStore/BitmapOps.cs | Returns WRONGTYPE for BITOP inputs; adds delete+retry for BITOP destination overwrite. |
| libs/server/Storage/Session/MainStore/AdvancedOps.cs | Returns WRONGTYPE for main-store RMW wrapper. |
| libs/server/Storage/Functions/UnifiedStore/ReadMethods.cs | Extends TYPE to return vectorset for Vector Set record type. |
| libs/server/Storage/Functions/ObjectStore/RMWMethods.cs | Adjusts WrongType paths in IPU/CU to ensure WrongType propagates correctly. |
| libs/server/Storage/Functions/MainStore/RMWMethods.Etags.cs | Ensures etag copy-update returns WRONGTYPE for non-string record types. |
| libs/server/Storage/Functions/MainStore/RMWMethods.cs | WRONGTYPEs non-legal updates on Vector Set records (command legality gate). |
| libs/server/Storage/Functions/MainStore/ReadMethods.cs | Ensures MGET reader path copies raw bytes (enables “return null for non-string key” behavior). |
| libs/server/Storage/Functions/GarnetRecordTriggers.cs | Adds ICompactionFunctions implementation intended to drive Vector Set cleanup during compaction. |
| libs/server/Resp/Vector/VectorManager.cs | Adds IsMember helper for VISMEMBER implementation. |
| libs/server/Resp/Vector/VectorManager.ContextMetadata.cs | Adds test helper GetContextState for cleanup assertions. |
| libs/server/Resp/Vector/RespServerSessionVectors.cs | Implements network handlers for VCARD, VISMEMBER, and stubs for VLINKS, VRANDMEMBER, VSETATTR. |
| libs/server/Resp/RangeIndex/RespServerSessionRangeIndex.cs | Correctly emits WRONGTYPE error instead of flushing with no response. |
| libs/server/Resp/MGetReadArgBatch.cs | Simplifies/clarifies MGET input initialization. |
| libs/server/Resp/KeyAdminCommands.cs | Treats WRONGTYPE in DUMP as “null” (non-string) and adds WRONGTYPE error for GETDEL. |
| libs/server/Resp/HyperLogLog/HyperLogLogCommands.cs | Returns WRONGTYPE errors when storage reports WRONGTYPE (not just 0xFF sentinel). |
| libs/server/Resp/CmdStrings.cs | Adds vectorsett for TYPE output. |
| libs/server/Resp/Bitmap/BitmapCommands.cs | Adds WRONGTYPE handling for bitmap ops; refactors BITFIELD* to lock key in txn for multi-subcommands and handle WRONGTYPE correctly. |
| libs/server/Resp/BasicEtagCommands.cs | Adds WRONGTYPE responses for etag GETs; adds overwrite retry (DELETE + SET) on WRONGTYPE for SET-with-etag paths. |
| libs/server/Resp/BasicCommands.cs | Adds WRONGTYPE responses for more string commands; adds overwrite retry logic for SET/SETEX/conditional SET variants. |
| libs/server/Resp/ArrayCommands.cs | Adds overwrite retry logic for MSET; adds WRONGTYPE handling for LCS. |
| libs/server/Resp/AdminCommands.cs | Adds WRONGTYPE responses for HCOLLECT/ZCOLLECT. |
| libs/server/Objects/ItemBroker/CollectionItemBroker.cs | Treats WRONGTYPE as type mismatch for item broker flows. |
| libs/server/Databases/DatabaseManagerBase.cs | Passes compaction functions (recordTriggers) into Tsavorite Log compaction. |
| libs/server/API/IGarnetApi.cs | Adds Vector Set APIs: VCARD, VISMEMBER, VLINKS, VRANDMEMBER, VSETATTR. |
| libs/server/API/GarnetWatchApi.cs | Wires WATCH instrumentation for new Vector Set APIs. |
| libs/server/API/GarnetApi.cs | Wires new Vector Set APIs to StorageSession; fixes IncrementByFloat return status propagation. |
…hub.com/microsoft/garnet into users/kmontrose/vectorSetWrongTypeTests
…arge to cause compaction
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Most commands, when used on Vector Sets, should
WRONGTYPE. Similarly, Vector Set commands when used on non-Vector Set keys shouldWRONGTYPEWe have a bunch of commands, and we've been pretty laxidasical about checking
GarnetStatusso this is a pretty big PR, though mostly mechanical.Changes:
VCARDVISMEMBERVLINKS,VRANDMEMBER, andVSETATTR- far enough toWRONGTYPE, but not fully implementedSET,MSET, etc.) correctly free up their resourcesOverwriting Vector Sets is a more complicated operation - if the value is still in memory, a IPU or CU needs a preceeding delete (because the
RecordTypeis changing); while if the value is on disk the blind upsert is fine, but we need to cleanup the Vector Set at some later point.My solution is to fail IPU/CU's with a
WRONG_TYPEresult and then retry in a transaction, and to implementICompactionFunctions.IsDeletedonGarnetRecordTypeto cleanup Vector Sets which are unreachable but not explicitly tombstone'd.This is approach is taken for:
MSETPSETEXSETSETEXSETIFGREATERSETIFMATCHSETWITHETAGThis impacted one RI test ; I left this commented out with a TODO for review by someone with more RI knowledge.