Token2022 suppport for buffers instructions - #120
Conversation
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
`CreateBuffer` and `ReclaimBuffer` each took a `token_program` account and rejected anything that wasn't the legacy SPL Token program. They now accept Token-2022 as well and issue all of their CPIs against whichever of the two they were handed, so a buffer can be allocated, initialized and closed under either. Token-2022 encodes the instructions this program issues exactly as the legacy program does, so only the CPI target changes. Two things do differ: - Account data. A Token-2022 account carrying extensions is longer than the base layout, so the legacy reader (exact length, legacy owner) rejects it. `token::read_token_account` dispatches on the validated program and reads by value, which also drops the borrow before `ReclaimBuffer` closes the same account. - Buffer sizing. A buffer now gets the length its mint actually needs: a mint with no extension data keeps the base layout, and anything longer is priced by asking the token program via `GetAccountDataSize`, the way the associated-token-account program does. That keeps the answer authoritative at run time rather than freezing a mint-extension-to-account-extension table into the program. The program account is shared by the whole instruction, so the mints one instruction touches must all live under the same token program; splitting a mixed batch across two instructions is the caller's job. Adds `SettlementError::BufferSizeUnavailable` (35), reachable only defensively: a token program that fails the size query aborts the instruction on its own. `BeginSettle` and `FinalizeSettle` keep rejecting everything but the legacy program; Token-2022 for the settlement pair follows separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3ef114e to
aa75ae4
Compare
CreateBuffer and ReclaimBuffer
`spl-token-interface` was a dependency of the interface crate for one thing: a 32-byte constant naming a program this code never calls directly. `spl-token-2022-interface` already carries that address in `inline_spl_token`, which exists precisely so a program that has to recognize both doesn't grow a second dependency for the one it only compares against. Both ids now come from there, and the legacy crate leaves the interface crate's dependency graph -- and with it the settlement program's. `instruction::settle` re-exported the same id straight from the legacy crate, so it moves to `token_program` too. That was the second import keeping the dependency alive, not a cosmetic change. The `.so` is byte-identical at 51,056 bytes: the constant was already inlined, so this narrows the dependency graph rather than the program. The program crate keeps `pinocchio-token`. Its state readers are not parameterized over the token program the way 0.7's instruction builders are -- `pinocchio_token_2022::state::Account` hardcodes an owner check against Token-2022 in all of its safe constructors -- so reading a legacy account without it means `from_bytes_unchecked` plus hand-rolled owner and length checks, in exchange for a crate `pinocchio-token-2022` depends on anyway. `test-cli` keeps the legacy crate as well. It talks to no other program, and the two crates' `native_mint` are different addresses, so swapping them there is a change to make deliberately rather than in passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
should work on interface as well as the program itself. that was almost bad!
The CLI's token resolution now has to know which token program owns a mint, which is a self-contained change with its own tests; it lands separately. All that stays here is the one field `CreateBuffers` gained, pinned to the legacy program, and the workspace entry for the legacy interface the CLI still builds against. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CreateBuffer and ReclaimBuffer
fedgiac
left a comment
There was a problem hiding this comment.
Design review. Overall, I agree it makes sense to use the token slot as a discriminant and only allow to bundle buffer creations for the same token program in a single instruction.
Nit: when evaluating an Anza dependency in the description, don't point at crates.io, whose content can easily be faked. Point to the GitHub repo by Anza with the Cargo.toml showing the right package name.
https://github.com/anza-xyz/pinocchio/blob/main/programs/token-2022/Cargo.toml#L2
There was a problem hiding this comment.
Literally no integration test covers the token 2022 path, which is imho a very bad idea.
I see this is annoying to test properly. We should also try doing something interesting, like adding some random extension. I suppose this could be new files, just to separate the two? Not sure at this point.
There was a problem hiding this comment.
Literally no integration test covers the token 2022 path, which is imho a very bad idea.
I see this is annoying to test properly. We should also try doing something interesting, like adding some random extension. I suppose this could be new files, just to separate the two? Not sure at this point.
this was intentional! see Out of scope and #121
There was a problem hiding this comment.
Sorry I missed that when reading the description. The approach makes sense. Would it be possible to create a new PR with only the tests for this PR so we can confirm everything works before building on top? It's also easier to think about the macro then.
fedgiac
left a comment
There was a problem hiding this comment.
It makes sense overall. While reviewing this PR I vibed-researched how token2022 and extensions are used, maybe you want to take a look but it doesn't change this PR I'd say.
There should be tests for a few critical cases: mint close and reopen with different extension; mint close and reopen as legacy token; trying to recover a buffer in the two cases before (should be possible according to my vibed tests). Also see comment on tests. The only thing I don't understand at this point in the current code is token_account_len, will look again after the comments are answered; I think there are a lot of tricky details hiding in there and I want to ask before going down the path of reading the token2022 program code.
There was a problem hiding this comment.
Sorry I missed that when reading the description. The approach makes sense. Would it be possible to create a new PR with only the tests for this PR so we can confirm everything works before building on top? It's also easier to think about the macro then.
Co-authored-by: Federico Giacon <58218759+fedgiac@users.noreply.github.com>
Co-authored-by: Federico Giacon <58218759+fedgiac@users.noreply.github.com>
Co-authored-by: Federico Giacon <58218759+fedgiac@users.noreply.github.com>
Co-authored-by: Federico Giacon <58218759+fedgiac@users.noreply.github.com>
Co-authored-by: Federico Giacon <58218759+fedgiac@users.noreply.github.com>
…ol/solana-programs into kaze/sc-153-token-2022-program
5db26f2 to
62776ff
Compare
| /// The base layout is the same under both programs, so one reader's idea of | ||
| /// its length is the other's too. | ||
| #[test] | ||
| fn both_programs_share_the_base_layout_length() { |
There was a problem hiding this comment.
Nit: this test is testing the test functions themselves and not the program. I like to use the term "sanity checks" in these cases to make sure that it's intentional that the test doesn't need to touch the program.
| fn both_programs_share_the_base_layout_length() { | |
| fn sanity_check_both_programs_share_the_base_layout_length() { |
| } | ||
|
|
||
| /// Seed the wrapped-SOL mint account, which `LiteSVM` does not create. | ||
| pub fn create_native_mint(svm: &mut LiteSVM) { |
| // Extensions are preceded by the account-type marker, which is what | ||
| // distinguishes a longer account from a mint of the same size. | ||
| data.push(AccountType::Account as u8); | ||
| data.extend_from_slice(&[0xab; 16]); |
There was a problem hiding this comment.
As commented before: let's avoid hand-rolling our own encoding for things that we don't control. What I'm expected to do here as a reviewer is determining that this block of code is equivalent to enabling a token extension and not just adding some arbitrary data bytes to the end of an account (and to certify that the risk of drift is minor). The imho better alternative is actually building a token with an extension using a library.
In any case, it would be a great help to have a link that shows this is the standard way to define extensions so I don't need to redo the independent research you already had to do.
| //! Token-program validation and token-account reads | ||
|
|
||
| use cow_settlement_interface::{ | ||
| token_program::TokenProgram::{self, SplToken}, |
There was a problem hiding this comment.
Nit: I found this a bit confusing because AI likes to generate matches like:
match token_program {
SplToken => ...
TokenProgram::Token2022 => ...
I think it looks better to always use the full form for both throughout the code.
| token_program::TokenProgram::{self, SplToken}, | |
| token_program::TokenProgram, |
| // Early return for SPL token (saves the GetAccountDataSize CPI call) | ||
| if token_program == SplToken { | ||
| return Ok(BASE_TOKEN_ACCOUNT_LEN); | ||
| } | ||
|
|
||
| let token_program = token_program.address(); | ||
| // SPL token provides a function to get the actual required account data size | ||
| GetAccountDataSize::new(mint).invoke_with_unverified_program(&token_program)?; | ||
|
|
||
| let reported = get_return_data().ok_or(SettlementError::BufferSizeUnavailable)?; | ||
| if reported.program_id() != &token_program { | ||
| return Err(SettlementError::BufferSizeUnavailable.into()); | ||
| } | ||
| let length: [u8; 8] = reported | ||
| .as_slice() | ||
| .try_into() | ||
| .map_err(|_| SettlementError::BufferSizeUnavailable)?; | ||
| let length = u64::from_le_bytes(length); | ||
|
|
||
| Ok(length) |
There was a problem hiding this comment.
(A bit of a nit.) The code makes sense as it's written but I struggled a lot to understand it so I'm suggesting a rewriting of it that should make it easier in the future to verify the logic. I searched a lot for ways not to use the length and, surprisingly to me, I couldn't find any
| // Early return for SPL token (saves the GetAccountDataSize CPI call) | |
| if token_program == SplToken { | |
| return Ok(BASE_TOKEN_ACCOUNT_LEN); | |
| } | |
| let token_program = token_program.address(); | |
| // SPL token provides a function to get the actual required account data size | |
| GetAccountDataSize::new(mint).invoke_with_unverified_program(&token_program)?; | |
| let reported = get_return_data().ok_or(SettlementError::BufferSizeUnavailable)?; | |
| if reported.program_id() != &token_program { | |
| return Err(SettlementError::BufferSizeUnavailable.into()); | |
| } | |
| let length: [u8; 8] = reported | |
| .as_slice() | |
| .try_into() | |
| .map_err(|_| SettlementError::BufferSizeUnavailable)?; | |
| let length = u64::from_le_bytes(length); | |
| Ok(length) | |
| match token_program { | |
| // SPL token accounts are always the base length, so skip the CPI. | |
| TokenProgram::SplToken => Ok(BASE_TOKEN_ACCOUNT_LEN), | |
| // Token-2022 accounts vary with the mint's extensions. This mirrors the | |
| // SPL Associated Token Account program's `get_account_len`: | |
| // https://github.com/solana-program/associated-token-account/blob/2dc55ee1009d787eea7e1c401b8f27e6892bff4b/program/src/tools/account.rs#L72-L97 | |
| TokenProgram::Token2022 => { | |
| GetAccountDataSize::new(mint) | |
| .invoke_with_unverified_program(&TokenProgram::Token2022.address())?; | |
| get_return_data() | |
| .ok_or(SettlementError::BufferSizeUnavailable.into()) | |
| .and_then(|reported| { | |
| if reported.program_id() != &TokenProgram::Token2022.address() { | |
| return Err(SettlementError::BufferSizeUnavailable.into()); | |
| } | |
| reported | |
| .as_slice() | |
| .try_into() | |
| .map(u64::from_le_bytes) | |
| .map_err(|_| SettlementError::BufferSizeUnavailable.into()) | |
| }) | |
| } | |
| } |
| Self::CloseAuthorityOnly => &[], | ||
| Self::WithNonTransferable => &[ | ||
| ExtensionType::NonTransferableAccount, | ||
| ExtensionType::ImmutableOwner, |
| /// The extension set a test mint is created with. | ||
| #[derive(Clone, Copy, Debug)] | ||
| pub enum Extensions { | ||
| CloseAuthorityOnly, | ||
| /// Non-transferable + CloseAuthority | ||
| WithNonTransferable, | ||
| /// Transfer Fee + CloseAuthority | ||
| WithTransferFee, | ||
| } | ||
|
|
||
| impl Extensions { | ||
| /// The extensions the mint itself is initialized with. | ||
| fn mint(self) -> &'static [ExtensionType] { | ||
| match self { | ||
| Self::CloseAuthorityOnly => &[ExtensionType::MintCloseAuthority], | ||
| Self::WithNonTransferable => &[ | ||
| ExtensionType::MintCloseAuthority, | ||
| ExtensionType::NonTransferable, | ||
| ], | ||
| Self::WithTransferFee => &[ | ||
| ExtensionType::MintCloseAuthority, | ||
| ExtensionType::TransferFeeConfig, | ||
| ], | ||
| } | ||
| } | ||
|
|
||
| /// The extensions Token-2022 requires of a token account holding the mint. | ||
| /// Spelled out rather than derived from [`Self::mint`], so the length a test | ||
| /// expects is stated independently of the program's own bookkeeping. | ||
| fn token_account(self) -> &'static [ExtensionType] { | ||
| match self { | ||
| Self::CloseAuthorityOnly => &[], | ||
| Self::WithNonTransferable => &[ | ||
| ExtensionType::NonTransferableAccount, | ||
| ExtensionType::ImmutableOwner, | ||
| ], | ||
| Self::WithTransferFee => &[ExtensionType::TransferFeeAmount], | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Nit: it feels that it would be a better design if Extensions was a vec of ExtensionType, the vec was created in-place in the relevant tests, and mint just returns the elements of the vec.
Then it becomes clear that token_account is a limited reimplementation of ExtensionType::get_required_init_account_extensions with hardcoded values. If you hardcode values, please provide a link for me to verify them.
The caveat: ExtensionType::get_required_init_account_extensions needs #[allow(deprecated)] (and imho deprecating that was bad design; I'm fine with using the deprecated one here). Alternatively, we inline it and vendor required_init_account_extensions (which is private unfortunately), with a link that allows me to verify that it's a 1-to-1 copy for that version.
| /// Create a Token-2022 mint at `mint`'s address carrying `extensions`, with | ||
| /// `payer` as both its mint authority and its close authority, and return the | ||
| /// address. Taking the keypair rather than generating one lets a test close the | ||
| /// mint and put something else back at the same address. | ||
| pub fn create_mint( | ||
| svm: &mut LiteSVM, | ||
| payer: &Keypair, | ||
| mint: &Keypair, | ||
| extensions: Extensions, | ||
| ) -> Pubkey { |
| extensions: Extensions, | ||
| ) -> Pubkey { | ||
| let space = ExtensionType::try_calculate_account_len::<Mint>(extensions.mint()) | ||
| .expect("every mint extension used here has a fixed length"); |
There was a problem hiding this comment.
I was super surpised by this. It implies that there are mint extensions with variable length.
This isn't true: this can only fail if the input has repeating extensions.
fedgiac
left a comment
There was a problem hiding this comment.
Nice, end of comments! Looks solid.
| #[test] | ||
| fn sizes_a_token_2022_buffer_to_the_extensions_its_mint_forces() { | ||
| let (mut svm, program_id, payer) = common::setup(); | ||
|
|
||
| for extensions in [ | ||
| Extensions::CloseAuthorityOnly, | ||
| Extensions::WithNonTransferable, | ||
| Extensions::WithTransferFee, | ||
| ] { |
There was a problem hiding this comment.
Looks good! Linking to a previous comment: here Vecs would be fine imho.
|
|
||
| /// Set up a Token-2022 mint with a close authority and its buffer, then close | ||
| /// the mint so `reopen` can claim the address. Returns the mint and its buffer. | ||
| fn buffer_whose_mint_was_reopened( |
Co-authored-by: Federico Giacon <58218759+fedgiac@users.noreply.github.com>
Adds the most basic level of Token-2022 support which widens the accepted token programs and focuses on dealing with the edge cases of
CreateBuffersandReclaimBuffers.What changes
CreateBufferandReclaimBuffereach take atoken_programaccount and, until now, rejected anything that wasn't the legacy SPL Token program. They now accept Token-2022 too and issue all of their CPIs —InitializeAccount3,CloseAccount— against whichever of the two they were handed.Token2022 accounts may have dynamic length. Previously, a buffer could only hold 165 bytes (the length of SPL token account), but now it is now allocated at the length its mint actually needs.
GetAccountDataSizeis used to verify the length of the token required before creating it.For
CreateBufferscreation of token accounts, We looked into the possibility of making the account longer than it needs to be and then shrinking it, but this doesn't work well because a extra CPI call is necessary to shrink the account back down to the correct size. But this requires a CPI call toReallocateon the token program, and that is more expensive than just getting hte length.There is an early return on
token_account_lenwhich allows for skipping theGetTokenAccountLengthCPI call. We originally did the early return if the Mint size was th ebase size (the logic being that a base size mint is either a canonical SPL token OR a token2022 mint with no required extensions), but since the vast majority of token2022 mints DO actually have extensions, there is very little benefit in having this broader check. So we decided to only do the early return for SPL tokens.Library Handling Changes
spl-token-2022-interface = "3", a library published by anza-team, andpinocchio-token-2022 = "0.4", another library published by anza-team, are added for hopefully obvious reasons.ended up being re-added after adding new tests because apparently a lot of mint creation utilities need to be recreated without it.spl-token-interfacewas removed from the settlement program because it is no longer needed (later it will also be removed from test-cli, eliminating it as a direct dependency from the repo).pinocchio-tokenis bumped to0.7in order to gain access to theinvoke_with_unverified_program(token_program)instruction builder function (prior to this release, there was no way to specify an alternative program). Its also the version thatpinnochio-token-2022transitively depends on.Most of the functions in both
pinnochio-tokenandpinnochio-token-2022are close to identical. For now most of the interfaces continue to usepinnochio-tokenbecause we never actually work with token 2022 tokens directly and the interfaces usually have slightly less dependencies (ex. not specifying the extension information).Out of Scope
BeginSettle/FinalizeSettlerequire a different methodology to support simultaneous settlement from both token programs, so those follow in #128.test-cliis covered separately in #134 .The integration tests for both this PR and #128 are in #121, as we want to expand coverage with 2022 across as many tests as possible.
At this time, only one token program can be supplied to the buffer functions. Two separate calls to
CreateBuffersis required if it is necessary to create buffers for tokens on two separate prgorams.Compute cost
bench-report.jsonis regenerated. The buffer instructions shift by roughly +0.2% to +0.4% from the added dispatch (reclaim_buffer/max_buffers_in_one_instruction136,501 → 137,046 is the largest); the one- and two-unit drift on the unrelated settle and transfer-authority lines is codegen, not behaviour.Test Plan
Verify the methodology. In particular, it would be good to verify the library dependency status as described above, because it is a bit awckward.
🤖 Generated with Claude Code