Fix object store base64 encoding to keep url-safe padding - #58
Open
lifewcody wants to merge 1 commit into
Open
Conversation
nats.go encodes object names in meta subjects and object digests with padded url-safe base64 (base64.URLEncoding) and its digest decoder rejects unpadded input. We stripped the padding, so objects written by this client were unreadable by nats.go, and objects written by nats.go could not be looked up here whenever the encoded name required padding. Objects written by previous versions live under unpadded meta subjects and have to be re-put after this fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While cross-testing an object store bucket between this client and nats.go I found that the two cannot see each other's objects.
Store::base64encode()strips the base64url padding when encoding an object name into the meta subject ($O.<store>.M.<name>) and when encoding the SHA-256 digest. The reference implementation keeps the padding in both places:encodeName()andGetObjectDigestValue()usebase64.URLEncoding(the padded variant), andDecodeObjectDigest()decodes with the same strict codec, which errors on unpadded input. Seejetstream/object.go(encodeName,GetObjectDigestValue,DecodeObjectDigest); the legacyobject.gobehaves the same.The consequences, in both directions:
=, so nats.go fails digest validation on everyGet.info()queries the unpadded subject. Wildcard scans over$O.<store>.M.>still surface them, so they appear in listings butget()returns null.The fix is to keep the padding, which matches nats.go byte for byte. I also added an assertion to
testPutObjectpinning the digest to the padded reference form (the existingxfilefixture already exercises a padding-requiring name).One caveat, noted in the changelog: objects written by previous versions of this client live under unpadded meta subjects, so after this change they have to be re-put to be visible again. Since the store was not interoperable with other clients anyway, a clean cut seemed preferable to carrying a legacy fallback lookup, but happy to add one if you'd rather keep old buckets readable.