feat: ExampleSponsoredDataSet

[wjmelements]

Closes #531
Design Doc
Reviewer @ZenGround0
CC @essamhassan @SgtPooki @rvagg @Kubuxu

Context

We want to start iterating on this use case before replication+repair and before FPv2 since it seems several customers want this.

Overview

This implements a smart contract payer for a singular data set.
The payer is funded by deposits to its FilecoinPay account or by FilecoinPay Rails.

Motivation

One goal is to minimize custody: donated funds can only pay for the data set.
Another goal is to allow finalization: funds paying for an immutable data set can only pay for that data set.

Participants

Anyone can donate to sponsor the data set.
There is a curator, who can add or remove pieces from the data set.
The curator can abdicate, which finalizes the data set.
There is also a fallback beneficary, who receives the funds in the event the data set is lost.

Migration

Migration is only allowed to sibling SponsoredDataSets and requires proof of data possession in the recipient.
For finalized data sets we require total data set equality.

Changes

• add SponsoredDataSet factory and children
• add tests

[wjmelements]

This function will also let us simplify some of our SignatureVerificationLib

[JAG-UK]

Thanks @wjmelements I like this a lot: the principle that a smart contract acts as payer for a dataset is certainly cleaner than other ideas like handing over active rails and things like that.

The one big missing piece for most of my use cases is that we are concerned with very large data sets - like 10s of TiBs at least - which means P2PP DDO PoRep deals for the foreseeable future. This PR is very tightly coupled with FWSS and therefore not a good fit for this use case.

Conceptually it's very close though. In P2PP deals are funded with Filecoin Pay rails and the NFRs of not wanting to deal with custody, maintaining tight control of migration etc all apply pretty much exactly as implemented here. We are planning to integrate P2PP PoRep with FoC very soon as an archival offering ("Filecoin Cold Storage Service"). Do you think it's reasonable to try to make this PR a little more flexible to talk about deals and CIDs in multiple dialects? Or would you rather deploy this just for FWSS PDP deals and we take it as inspiration to deploy a separate parallel system for FCSS?

[JAG-UK]

One other system-wide thought: will there be a problem making a smart contract the payer in terms of identifying who 'owns' a dataset for the purposes of signing vouchers for retrieval or applying UCAN access control? Are we comfortable that the designs for those parts of the puzzle will play nice with this model?

My immediate use cases for sponsored large datasets are public goods so none of this applies, but things can grow and I don't want us to miss an obvious face rake if we can avoid it.

[wjmelements]

One other system-wide thought: will there be a problem making a smart contract the payer in terms of identifying who 'owns' a dataset for the purposes of signing vouchers for retrieval or applying UCAN access control? Are we comfortable that the designs for those parts of the puzzle will play nice with this model?

My immediate use cases for sponsored large datasets are public goods so none of this applies, but things can grow and I don't want us to miss an obvious face rake if we can avoid it.

Our current plan for UCAN support is #536 which also allows secp256r1 and other custom auth.

[jennijuju]

Want to set expectations on this PR from the product side, because the scope here is narrower than the request that started it — and that narrowing happened without product in the loop.

The customer ask was "can our users sponsor a rail," meaning fund both storage and egress on a given dataset of FWSS. The general idea we discussed in NYC was: make the payer of a rail a contract and any service on that rail can be sponsored. This PR doesn't deliver that. It's a stopgap scoped to FWSS PDP rails only. That's fine as a first step, but let's be explicit about what it is and isn't so nobody downstream over-reads it (correct me if im wrong):

• It is not a reusable sponsorship primitive. : It's FWSS/PDP-coupled and immutable; other services (egress, FCSS/P2PP — see JAG-UK's comment) each need their own contract.
• It does not mean "sponsored service" is now a FOC capability.
  • : delivering the actual "sponsor a rail" request — storage and egress, reusable across services — needs FilecoinPay v2 with depositToRail (rail-scoped balances). Is that the next step, and is the egress rail dependency in its scope? I don't want us to ship this, call "sponsorship" done, and quietly leave the real ask to a v2 that isn't sequenced.
• It's plumbing for integrators who onboard datasets and want easier donations toward storage — not a self-serve feature for end storage users. Someone still deploys, binds, and operates a vault per dataset.
• Most important, it does not serve the egress half of the customer ask.

The design doc suggests the CDN rail is out of scope, however, i read the PR correctly, a CDN-enabled dataset would have egress bleed the same deposits with no safe guard still. How is that managed atm?

Net: I'm fine shipping this as an explicitly-scoped storage-only sponsorship for FOC. I'm less fine if it's positioned as "Sponsor/Sponsor service is supported on FOC " — because the request was sponsored service, and that's still ahead of us.

[wjmelements]

a CDN-enabled dataset would have egress bleed the same deposits with no safe guard still. How is that managed atm?

That's true. Data sets in FWSS are tightly bound to their FilBeam add-on. The same client must pay for both; there isn't a way to pay for one but not the other. The way it is handled is that you can turn it off in FWSS and pay for it out of band. Whether it is turned on or off, this setting is persisted to successor SponsoredDataSets, so if you donated to pay for FilBeam, you will keep paying for FilBeam, and if you did not you will not.

it does not serve the egress half of the customer ask.

Does this mean:

The customer ask was "can our users sponsor a rail," meaning fund both storage and egress on a given dataset of FWSS

This feature delivers funding both storage and egress on a FWSS dataset. What it does not do is allow "deposit to rail" since rails don't have accounts in FilecoinPay (even though that would have been a better way to implement lockups, particularly with ERC-6909).

[wjmelements]

Per discussion with @rvagg today we're renaming this to ExampleSponsoredDataSet to signal that this is a template for developers to use as a starting point.

[rvagg]

This is good by me, it just needs a doc somewhere - either in the readme or alternatively an "examples" directory with its own readme and this file in it.

Unfortunate we have a new dependency but it's only for tests at least.

[parkan]

sorry for all the delays in looking at this, was AFK at dwebcamp (still at camp for breakdown but feeling bad about missing the review request so sneaking away to review)

EDIT: hmm I see some issues here, will post analysis shortly

[parkan]

ok there's a few issues here that combine into a bad path:

• pick a set
• create a "matching" successor set with matching leaf count but without real data
• propose migration and delete successor in a single block/bundle, no one can challenge because this is basically atomic
• challenges revert because there's no dataset to compare with
• completeMIgration happily runs
• we take the unlocked funds from the source and release

as far as I can tell this is only guarded by registered status

other issues/more details in comments

[parkan]

I think this might break migrations if pieces were ever deleted -- repacking the surviving pieces after a deletion results in non-matching NextPieceId (deleted ids are not reused) and this reverts

[parkan]

re:

For finalized data sets we require total data set equality.

this is only actually checked here:
https://github.com/FilOzone/filecoin-services/pull/540/changes#diff-e4c4ab4e9a41f0966f2475cad1096f7894ace1f3d02566fbec4691fdbe1ffb10R359

the propose flow does not -- this is the "optimistic" path that avoids enumerating pieces but as it stands now is exploitable

(everything below is speculative/needs more work, fixing the above closes the immediate gaps for now)

the real solution would be for PDPVerifier to expose a merkle root per set with tombstones filtered (getActivePieces already does this so just need an amortized set-root update, this might even exist somewhere but I can't find it) -- this way all the migration dances would reduce to O(1) fully deterministic and provable comparisons

EDIT: actually the real/simple fix is even simpler: since only finalized datasets may be migrated then we only need to compute the root once, at finalize time

my guess is that this was considered and rejected for some concrete reason, but the merkle root computation should already be optimized and doing so once upon finalize should not be expensive relative to running all the fraud proof machinery -- let me know if I'm overlooking something here

EDIT2: ok the other other observation is that beyond tombstone count making sets unequal even if alive pieces are exactly the same and in the same order (by the nextId metric) the thing that we actually want is an unordered set equality -- the order of addition is only meaningful incidentally, it does not reflect contributor intent in any case I can think of, so the construction that we want is not a strict merkle root but an order-independent multiset like MuHash in BTC

[wjmelements]

PDPVerifier to expose a merkle root

I think it's too late to add that kind of accounting. I also considered a trie for the proof system. Data "sets" can have duplicates though, and there's also piece metadata which would ideally also match.

[parkan]

PDPVerifier to expose a merkle root

I think it's too late to add that kind of accounting. I also considered a trie for the proof system. Data "sets" can have duplicates though, and there's also piece metadata which would ideally also match.

yeah, understandable -- thus it's under the speculative section (and yes, merkle is not really the right aggregation here, just the first tool on the shelf in PL-land)