Ibhatt/accdb6 bundles [wip, draft]

[ibhatt-jumptrading]

No description provided.

[greptile-jt]

Greptile Summary

Restructures account handling across the runtime to support bundle transaction execution with cross-transaction account deduplication. fd_txn_out_t::accounts now holds pointers (fd_acc_t *) into a shared fd_runtime_t::accounts.account[] storage so that bundle transactions referencing the same pubkey share a single fd_acc_t instance, ensuring writes from earlier txns are visible to later txns in the same bundle. Adds new account_acquired[] / executable_acquired[] flags so commit/cancel can selectively release only the entries this transaction actually acquired.

• fd_executor_setup_accounts_for_txn now returns an int, internally invokes fd_runtime_load_txn_address_lookup_tables, and performs bundle-wide dedup against runtime->accounts.account[0..account_cnt] before calling fd_accdb_acquire_a.
• fd_runtime_load_txn_address_lookup_tables signature changed: txn_in removed and trailing fd_runtime_t const * runtime_opt added. The previous bundle-internal ALUT dedup, which required the prior referencing txn to mark the ALUT as writable, is replaced by an unconditional runtime->accounts.account[] scan — a behavior change worth confirming against Agave.
• Commit/cancel paths now iterate per-element and release only when account_acquired[i] / executable_acquired[i] is set, matching the new shared-pointer ownership model.
• Test fixtures across tests/, program/, vm/syscall/, and discof/ are updated mechanically to dereference fd_acc_t * and to initialize runtime->accounts.account_cnt = 0. test_bundle_exec.c adds a useful pointer-identity assertion verifying dedup actually shares the underlying fd_acc_t.
• test_runtime_alut.c swaps the huge-page workspace for a 16 GiB aligned_alloc that is never freed; acceptable for a test binary but worth noting.

Confidence Score: 3/5

• Marked WIP/draft; correctness hinges on the new bundle dedup ownership model and an ALUT lookup semantics change that should be confirmed before merge.
• Most of the diff is mechanical pointer-dereference plumbing that is straightforward and well-covered by existing tests (and a new pointer-identity assertion in test_bundle_exec.c). The score is held at 3 because (a) fd_runtime_load_txn_address_lookup_tables no longer requires the prior bundle txn to mark an ALUT writable when reusing it, which is a behavior change relative to the prior bundle path, and (b) the new acquire/release flag scheme in fd_runtime_commit_txn/fd_runtime_cancel_txn is subtle and any miswire would leak or double-free accdb references. The PR is explicitly marked WIP/draft.
• Closest scrutiny on src/flamenco/runtime/fd_executor.c (bundle dedup loop and the new setup_txn_account_keys reset semantics), src/flamenco/runtime/fd_runtime.c (ALUT lookup behavior change and per-element release), and src/flamenco/runtime/fd_runtime.h (struct layout / sizing of refcnt[] for bundles vs. per-txn indexing).

Important Files Changed

Filename	Overview
src/flamenco/runtime/fd_runtime.h	Restructures fd_runtime_t::accounts and fd_txn_out_t::accounts to support bundle-wide deduplication. The runtime now owns flat arrays sized FD_PACK_MAX_TXN_PER_BUNDLE * MAX_TX_ACCOUNT_LOCKS for account[], executable[], and refcnt[]. Each fd_txn_out_t holds pointers into the runtime arrays plus per-txn is_writable/account_acquired/executable_acquired flags. Sensible direction but the refcnt[] array is still indexed by per-txn index even though it is sized for bundles — most entries are unused.
src/flamenco/runtime/fd_executor.c	Major restructuring of fd_executor_setup_accounts_for_txn. Adds bundle deduplication via a search of runtime->accounts.account[] before calling fd_accdb_acquire_a. Changes return type to int and inlines the previous fd_executor_setup_txn_alut_account_keys call. The fallback bundle fee-payer lookup logic in fd_executor_create_rollback_fee_payer_account was removed because shared fd_acc_t pointers make it unnecessary. The bundle dedup uses fd_pubkey_eq with FD_LIKELY(!equal), which is the correct hot path. Removed the old fallback logic that walked previous txn outs in fd_executor_create_rollback_fee_payer_account — this is now correct because the underlying fd_acc_t is shared across bundle txns.
src/flamenco/runtime/fd_runtime.c	Updates fd_runtime_load_txn_address_lookup_tables signature: drops txn_in, takes fd_runtime_t const * runtime_opt instead. Behavior change: the previous code only re-used a bundle ALUT if the prior txn marked it writable; the new code reuses any deduplicated account regardless of the prior txn's writability view. Also adds fd_accdb_unread_one on the error path for !acc.lamports (good fix to avoid leaking the read). fd_runtime_commit_txn/fd_runtime_cancel_txn switched to per-element release based on account_acquired/executable_acquired flags. fd_runtime_new_txn_out now zeroes additional fields. fd_runtime_pre_execute_check now calls fd_executor_setup_accounts_for_txn instead of fd_executor_setup_txn_alut_account_keys (ALUT is invoked inside).
src/flamenco/runtime/fd_runtime_helpers.h	Signature change to fd_runtime_load_txn_address_lookup_tables: removed fd_txn_in_t const *, added trailing fd_runtime_t const * runtime_opt. Call sites updated accordingly.
src/flamenco/runtime/test_bundle_exec.c	Mechanical test updates for the new pointer-based txn_out->accounts.account[] and the move of writable flags into txn_out->accounts.is_writable[]. New assertion validates pointer sharing across bundle txns (txn_out[1].accounts.account[1] == txn_out[0].accounts.account[1]), which is a useful invariant for the dedup design. Also updates the loaded-programdata search to iterate txn_out->accounts.executable_cnt instead of runtime->accounts.executable_cnt.
src/flamenco/runtime/test_runtime_alut.c	Updates all fd_runtime_load_txn_address_lookup_tables calls to the new signature (drops txn_in, passes trailing runtime_opt=NULL). Also rewrites main() to back the workspace with aligned_alloc over normal pages instead of gigantic huge pages — useful for CI environments without huge pages, but aligned_alloc(FD_SHMEM_NORMAL_PAGE_SZ, ...) allocates wksp_footprint = 16GiB of process-private memory that is never freed.
src/flamenco/runtime/tests/fd_dump_pb.c	Updates pointer accesses to txn_out->accounts.account[] and txn_out->accounts.executable[]. Switches create_txn_context_protobuf_from_txn from iterating txn_out->accounts.cnt to iterating raw txn_descriptor->acct_addr_cnt, dumping account keys read directly from the txn payload — this matches the pre-execution state before ALUT expansion is applied to txn_out->accounts.keys[].
src/flamenco/runtime/tests/fd_instr_harness.c	Test harness updated to wire txn_out->accounts.account[i] and txn_out->accounts.executable[i] pointers to per-slot storage in runtime->accounts.account[] and runtime->accounts.executable[]. Also sets txn_out->accounts.is_writable[j]=1U parallel to acc->_writable=1.
src/flamenco/runtime/test_define_ltds_fee_only_semantics.c	Adds fd_memset( txn_in, 0, sizeof(fd_txn_in_t) ) to all builder helpers. This is now required because the runtime code branches on txn_in->bundle.is_bundle and uninitialized stack memory would be UB. Good defensive change for tests.

_{Last reviewed commit: c0fd096}

[Copilot]

Pull request overview

This PR refactors how transaction accounts are represented and managed during execution—especially for bundles—by making fd_txn_out_t store pointers into runtime-owned, deduplicated account storage. It updates bundle semantics (ownership transfer, single lthash commit, ordered vote ops), adjusts ALUT loading to optionally reuse bundle-opened accounts, and updates a large set of tests/harnesses to match the new pointer-based account model. It also includes a few bundle client/tile robustness tweaks and re-enables the bundle execution unit test.

Changes:

• Convert txn_out->accounts.account[] (and executable accounts) from inline structs to pointers into runtime->accounts.* pools, adding per-txn is_writable, acquisition/ownership, and starting-state tracking to support bundle forwarding and correct commit behavior.
• Rework bundle commit/cancel/release semantics (single-owner commits; ordered new_vote/rm_vote per writable txn; ownership transfer on permission upgrade) and extend test_bundle_exec with new regressions.
• Improve operational behavior in bundle client/tile (connection gating, backoff tweaks, stream end handling), and re-enable the bundle exec unit test in Local.mk.

Reviewed changes

Copilot reviewed 31 out of 32 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
src/flamenco/vm/syscall/test_vm_increase_cpi_account_info_limit.c	Update syscall test setup for pointer-based txn_out->accounts.account[].
src/flamenco/vm/syscall/test_cpi_shared_data_addr.c	Update CPI shared-data test to use runtime-owned account storage via pointers.
src/flamenco/vm/syscall/test_cpi_semantics.c	Adapt CPI semantics test to pointer-based accounts and new account storage wiring.
src/flamenco/runtime/tests/fd_vm_harness.c	Adjust fuzz/VM harness to treat transaction accounts as pointers.
src/flamenco/runtime/tests/fd_svm_mini.c	Initialize newly added runtime account pool counters.
src/flamenco/runtime/tests/fd_solfuzz.c	Initialize newly added runtime account pool counters in solfuzz runner.
src/flamenco/runtime/tests/fd_instr_harness.c	Update instruction harness account/executable wiring and writable tracking.
src/flamenco/runtime/tests/fd_dump_pb.c	Update protobuf dumping paths for pointer-based accounts and per-txn executable counts.
src/flamenco/runtime/tests/fd_bundle_harness.c	Update bundle harness to dereference pointer-based accounts.
src/flamenco/runtime/test_runtime_alut.c	Update ALUT tests for new loader signature and change workspace allocation approach.
src/flamenco/runtime/test_define_ltds_fee_only_semantics.c	Ensure txn input/output structs are fully reset between test cases.
src/flamenco/runtime/test_bundle_exec.c	Major bundle semantics regression expansion; update to pointer-based accounts and new flags.
src/flamenco/runtime/sysvar/fd_sysvar_instructions.c	Update sysvar instructions serialization to pointer-based account storage.
src/flamenco/runtime/program/test_vote_program.c	Update vote program test to use pointer-based accounts.
src/flamenco/runtime/program/test_bpf_loader_serialization.c	Update BPF loader serialization tests for pointer-based accounts and writable flags.
src/flamenco/runtime/program/fd_system_program_nonce.c	Update nonce handling to pointer-based transaction accounts.
src/flamenco/runtime/program/fd_bpf_loader_program.c	Update executable account lookup API usage.
src/flamenco/runtime/Local.mk	Re-enable test_bundle_exec unit test build/run.
src/flamenco/runtime/info/fd_instr_info.c	Update lamport summation helper to use pointer-based accounts.
src/flamenco/runtime/fd_runtime.h	Redefine txn_out account storage as pointers; add bundle-related per-txn tracking fields; expand runtime account pools for bundles.
src/flamenco/runtime/fd_runtime.c	Update ALUT loader signature/behavior; shift account setup ordering; update commit/cancel to ownership-based release and per-txn flags.
src/flamenco/runtime/fd_runtime_helpers.h	Update helper prototypes for ALUT loading and executable account lookup.
src/flamenco/runtime/fd_executor.h	Change account setup API to return status; update load-accounts signature.
src/flamenco/runtime/fd_executor.c	Implement bundle reuse/ownership transfer, starting-state tracking, executable reuse, and updated account acquisition/refund logic.
src/flamenco/runtime/fd_cost_tracker.c	Switch writable-account detection to txn_out->accounts.is_writable[].
src/flamenco/runtime/fd_alut.h	Minor formatting change in ALUT interpreter discriminant check.
src/flamenco/log_collector/fd_log_collector.h	Update log collector access for pointer-based program account.
src/discof/replay/fd_sched.c	Update ALUT loader callsite to new signature.
src/discof/execrp/fd_execrp_tile.c	Initialize newly added runtime account pool counter.
src/discof/execle/fd_execle_tile.c	Initialize newly added runtime account pool counter.
src/disco/bundle/fd_bundle_tile.c	Fix input-kind indexing for polled links and use fragment size in replay reset validation.
src/disco/bundle/fd_bundle_client.c	Improve connection gating, backoff behavior, and gRPC stream end/error handling.