wallet: audit trail for external transactions

[sir-opti]

Description

Implements an append-only ledger that makes every decision made by SN's external wallet mechanics visible and auditable, including the impact on monetary amounts. The ledger hooks directly into database mutations made by the ExternalTransaction logic atomically, reusing the same database transaction to ensure that there are no unresolvable states in the database at any time.

The ledger's accounting model exists of append-only rows that track:

• signed amounts in a "side" column that records CREDIT (+) and DEBIT (-) and an "amount" column containing the magnifier in mSats (>= 0)
• monetary movement across:
  • sources: ACCRUAL and SETTLEMENT, where ACCRUAL represents the invoice side and SETTLEMENT the payment side of each transaction.
  • direction: RECEIVE and SEND, representing the user requesting a payment or paying an invoice.
  • type: the nature of the action taken
    • OBLIGATION: the recording of an invoice
    • FULFILLMENT: the recording of a payment
    • CORRECTION: a correction to an invoice that was under- or overpaid and in a final state.
    • UNKNOWN: a state we cannot measure (often because of missing functionality for a protocol) and thus have to make assumptions about the state
    • ERROR: reversals due to technical errors
    • TIMEOUT: reversals because no confirmation was given in time
• de-normalized relations over userId, protocolId, walletId and externalTransactionId

The data is exposed through gql nested under the ExternalTransaction type (i.e. on querying transaction details) and on the /wallets/transactions/[id] page. The data is collapsed by default (modeled after "similar items") because it is only interesting when things go wrong.

See docs/dev/external-transaction-auditing.md for a little table that shows how types are mapped across source and debit/credit sides.

NOTE: proposed against the feature branch!

Screenshots

Pending, unexpired receive request

A settled receive request

A timed out request

Underpayments

Overpayments

When the protocol does not support tracking of settlement

Additional Context

The primary cause of this was me messing with hostile NWC responses to see what happens when something errors out or worsie - spreads lies on the 3rd party wallet-side. Since there is often no other way than optimistic interpretation with any of the current connection protocols - except for the preimage check - the only defense SN has against poor or hostile implementations is tracking. Therefore the main goal of this is to make monetary choices made by SN visible and auditable to a user, preventing drama.

PS: I will propose a separate PR outside of this feature topic to add a modified implementation of nostr-wallet-connect-lnd to the dev env.

Checklist

Are your changes backward compatible? Please answer below:

Yes against master, but not within a currently active dev env database on the wallet-external-tx-check feature branch (cannot backfill things that weren't captured.)

On a scale of 1-10 how well and how have you QA'd this change and any features it might affect? Please answer below:

7 - I have tested with CLINK, NWC and CLN. Most notably I haven't tested lnbits because my env isn't set up properly for that.

For frontend changes: Tested on mobile, light and dark mode? Please answer below:

Yes, yes (and light mode hurts eyes), simulated with viewport only.

Did you introduce any new environment variables? If so, call them out explicitly here:

no.

Did you use AI for this? If so, how much did it assist you?

Some. It helped me understand the base branch at the start and plan integration and with the front-end work. The only thing I have kept from the latter is the css.

[sir-opti]

Apologies for the force pushes - I messed up the source branch pick twice in a row.

[sir-opti]

Pushed a bugfix because ledgerType was being assigned too often (on non-terminal states). In some unknown - and maybe in the future, error - edge cases, the reason enum has to be checked against STOP_CHECK_REASONS to make sure that we're not tagging a transient state to be written to the ledger.

This didn't show up in practice, because the ledgerType was passed to functions that stripped it off again, so I fixed that too for the edge cases.

[huumn]

Have at most one terminal state of UNKNOWN, ERROR or TIMEOUT, as no
no further state changes are allowed after this. Note that FULFILLMENT is
explicitly not included to support any future protocol support of multiple
partial payments.

Do you have more context for this? Are there partial payment proposals circulating in ln dev circles or is this in anticipation of a feature on SN?

---

ExternalTransactionLedger denormalizes walletId, protocolId, and userId. Is that an optimization or do you anticipate a future use for those?

---

As discussed in chat, I'd like to derive transaction status using this audit trail rather than doing all the check-and-setting we currently have. This approach is far more architecturally sound. My plan is to rewrite #3121 to fit with this audit trail. To make it fit, I expect #3121 to implement:

• checkInvoice for wallet adapters
• ExternalTransaction for recording intent, check-with-lease fields and unknown diagnostics, terminal statuses (just to make the PR work)
• polling check-with-lease on ExternalTransaction rows
• "skeletons" for where ExternalTransactionLedger rows will be written

---

The main high-level concern I have with this PR is minor. It is UI/UX of the exposed audit trail. I suspect we'll want to do something like humanReadableLedgerEntry that turns each entry into a sentence that non-programmers and non-double-enterers can read.

[sir-opti]

at most one terminal state, except FULFILLMENT
Do you have more context for this? Are there partial payment proposals circulating in ln dev circles or is this in anticipation of a feature on SN?

It's an anticipation of not being able to guarantee that all encapsulating communications protocol implementations (like NWC) honor that constraint in the future. I should have made that more clear in the comment, can amend if this is going to be used. The other states are pure SN decisions, this one is (currently) triggered externally. Database constraints for external data are a pain.

---

denormalizes walletId, protocolId, and userId
Is that an optimization or do you anticipate a future use for those?

2 reasons:

1. There are multiple entry points to cascading delete, and this ensures that the history gets deleted properly - as a clear privacy enabler. PS: The thing that doesn't trigger, that I have in a list to open another discussion on at some time - was that "clear logs" does not remove the external transaction (and thus not the ledger entries) but I figured we could address that separately.
   • delete wallet
   • remove protocol config (in the form of remove receive, remove send)
   • delete user
2. I figured that if you ever want to expose the ledger to a user, for example in an "export all my transactions to CSV" feature (I'd honestly want that), having a star join schema instead of a snowflake is desirable anyway.

---

The main high-level concern I have with this PR is minor. It is UI/UX of the exposed audit trail.

Understood. The main function of doing accounting on this is because SN making decisions about rainy case scenarios, and have auditable financial integrity on those decisions.

I suspect we'll want to do something like humanReadableLedgerEntry that turns each entry into a sentence that non-programmers and non-double-enterers can read.

As we discussed initially, I was looking first to just add the accounting to walletLogs, but I let go of that. The problem, besides this table touching payIns, is that in production, the bulk of the information is (hopefully) transient and thus has no financial meaning, i.e. "Failed to get NWC response within 10s". So that means that we're combining financial accounting of decisions made with technical, transient, information. So I decided to:

1. Decouple log information from accounting logic, which means nothing prevents adding a line explaining to the user what happens in walletLogs. I also didn't want to link logging and accounting through an fk because of the log clear functionality.
2. Hide the accounting section in the UI so that it doesn't bother people that don't want to look at accounting.

The second step I envisioned would be to simply add walletLogs messages describing what decision was made so that users can see whats up without having to click the accounting detail section; like done for the timeout scenario. I think that it also needs to be done for other final decisions, and under/overpay situations.

---

While I do think that there is something to say for separate accounting functionality, a final solution can definitely be a pure state transition table, i.e. something that simply tracks state transitions. Not as free-form as walletLogs, not as rigid as the proposed accounting table here, which may be closer to what you're envisioning; simplified eventual happy path:

invoiceId	state	isTerminal	reason	hasError	error	accrual	settlement
1337	CREATED	f	null	f	null	1,000,000	0
1337	UNKNOWN	f	TRANSIENT_CHECK_FAILED	t	Connection timed out after 10 s	0	0
1337	SETTLED	t	null	f	null	0	1,000,000

This may feel more efficient than 2 separate tables, and if there would ever be an export function, WHERE accrual != settlement would be a proper hack to not display non-financial rows. I normally dislike hacking/mixing up financial functionality though; rigidity is generally considered a feature in money - but consider that a matter of taste for this case, because none of this data can ever be financially authoritative and is only informational: it fully depends on "gossip" from third party APIs that do not have to be true; the "truth" is always in the third party source (or if it is defunct, gone), but it can never be on the SN side for ExternalTransaction.

Note that for PayIns, this is a different story and there it does matter as there the only source are SN decisions, and thus I decided that the rigidity is not a bad thing to have, if ever extended to that side of the wallet interactions.

[huumn]

Thank you for the explanations! I understand your intent better now.

Hide the accounting section in the UI so that it doesn't bother people that don't want to look at accounting.

You're right I think that's sufficient.

export all my transactions to CSV

I've wanted to do this since forever. It just needs to be async (request, queue, do, store, then notify) and is quite involved.

---

I'm reworking the base for this branch to avoid doing anything this ledger does better (and rehumanizing it ... ugh the siren song of llms). I have a day worth of rounds to make on that but hope to update that PR tonight.

[sir-opti]

ugh the siren song

Word!

update that PR tonight.

Cool. I'll expand doc and comments a bit to make the intent clearer when I rebase. Do you want to go the merged table route or the strict separation of concerns route? I can pretty quickly transform this into either - takes a day on top of yours at most.

[huumn]

Do you want to go the merged table route or the strict separation of concerns route?

Keeping them separate makes sense. The boundary here is already helpful. It allowed me to simplify the receive stuff a lot.

[sir-opti]

Closing this in favor of a less complicated approach