Propagate storage_options to all backends resolved by GenericFileSystem (#1856)

[balramthewarrior]

Fixes #1856.

Root cause

GenericFileSystem stores storage_options as self.st_opts and resolves a backend per call via _resolve_fs. With default_method="options", _resolve_fs looks up storage_options[protocol] to construct the backend — so omitting the argument silently builds an unconfigured filesystem.

Of the 14 _resolve_fs call sites, only 5 passed storage_options=self.st_opts. The other 9 dropped it: _info, _ls, _cat_file, _rm, _cp_file (both sides), _make_many_dirs, and _copy (both sides).

Any credentialed backend reached through those paths was constructed bare, which is what breaks rsync across filesystems with different protocols or credentials.

On the reported isdir failure

The issue proposes adding an explicit GenericFileSystem._isdir. I don't think that's needed, and it matches your reply on the thread:

This is surprising, since it ought to fall back to info() which in turn falls back to ls()[0].

That fallback works fine. The reason isdir failed is that it reaches the backend through _info, which was one of the 9 sites dropping storage_options. Fixing the propagation makes isdir work through the existing fallback — no new method required. There's a regression test covering exactly this.

Change

Pass storage_options=self.st_opts at the 9 remaining _resolve_fs call sites. No behaviour change when storage_options is unset (_resolve_fs already coerces None to {}), and no change for the default/generic/current methods, which ignore the argument.

Tests

Added to fsspec/tests/test_generic.py:

• _OptionsRequiredFS, a MemoryFileSystem subclass that raises unless its token storage option arrives — standing in for a filesystem needing real credentials, so dropped options fail loudly instead of silently falling back to an unauthenticated instance.
• test_storage_options_propagated_to_backend — parametrized over info, ls, cat_file, isdir, find.
• test_storage_options_propagated_cross_protocol — the reporter's original call shape: rsync from a credentialed remote to local, configured via inst_kwargs.

Against unpatched master these give 5 failed / 8 passed. find passes on both, since _find was already among the 5 correct call sites, so the tests track this specific defect rather than failing broadly. Full suite: 1558 passed, 0 failed. ruff check, ruff format and codespell clean.

[martindurant]

Same as if token is not None? Then you would not need the class attribute _token.

[martindurant]

This is unnecessarily verbose: please just move those operations inline rather than parametrising, and remove the comment here and in the other tests.

[balramthewarrior]

Thanks for the review. Done both:

• Dropped the _token attribute; the backend now just checks token is None.
• De-parametrised test_storage_options_propagated_to_backend into inline calls and removed the explanatory comments.

Pushed. Confirmed the two tests still fail against master without the generic.py change and pass with it.