Feat: support HTTPS URLs in the -c/--config flag

[Omar1H1]

Summary

Implements #939, allows -c/--config to accept an https:// URL
in addition to a local file path.

Changes

• internal/config/reader.go — URL detection, fetch logic, watcher guard
• internal/config/lint.go — URL support for ReadYAMLFileLinted
• internal/cli/run.go — --config-header flag definition
• internal/cli/common/reader.go — watcher fatal error, headers wired into NewReader
• internal/config/reader_test.go — added unit tests for fetchRemoteConfig and remote reader behaviour
• internal/config/lint_test.go — new file, unit tests for ReadYAMLFileLinted with remote URLs

Behaviour

• https:// prefix → remote fetch before config parsing
• Everything else → unchanged file path behaviour
• Non-2xx response → exits with status code in error message
• --watcher + URL → exits with fatal error immediately
• env var interpolation (${FOO:default}) still works after fetch
• --config-header "Name: Value" (repeatable) for auth headers

Testing

• bento -c url → stream starts correctly
• bento -c url echo → prints normalised config
• bento -c url streams → streams mode works
• bento lint url → lints remote config
• bento -c url --config-header "K: V" → auth header forwarded
• bento -c url --config-header "bad" → clear format error
• bento -c url-returning-404 → clear 404 error
• bento -c url-returning-401 → clear 401 error
• bento -c url --watcher → fatal error on startup
• bento -c path/to/file.yaml → unchanged behaviour

Known limitations

• Only https:// URLs are supported. plain http:// falls back to
  file path behaviour.

• bento lint <url> works for public endpoints but does not support
  --config-header. ReadYAMLFileLinted has no access to CLI flags
  at that call site. Can be addressed in a follow-up.

  Open question

As noted in #939. happy to move to a dedicated --config-url flag if that is preferred over extending -c.

[jem-davies]

Would need to document the new feature by changing the documentation strings for the -c / --config flag as well

[jem-davies]

Suggested change

	if strings.HasPrefix(path, "https://") {
	parsedPath, err := url.Parse(path)
	if err != nil {
	...
	}
	if parsedPath.Scheme == "https" || parsedPath.Scheme == "http" {
	...
	}

Could use url.Parse here - if the provided path is a relative filepath ./config.yaml scheme should be empty string, you could also have a scheme file too - file:///Users/.../config.yaml

[jem-davies]

could use a http.NewRequestWithContext() and pass a context with a short deadline.

[jem-davies]

I think we might want to expose some more options for the client - such it could work with TLS certificates that are signed with private CA's etc.

[Omar1H1]

Hello @jem-davies
Thanks for the feedback, i'm just wondering should this reuse existing CLI flags or
env vars, or would you prefer dedicated flags like --config-tls-ca-file ?

[jem-davies]

Hi @Omar1H1

Me and @gregfurman have just been discussing this PR.

I am in the position that the config options for this new http client could be handled similarly to the http root level config - perhaps with a field http_client.

That it would be possible to provide this to Bento in a ./resources.yaml - such it could then subsequently fetch a stream config.

Added multiple CLI flags I think wouldn't be the way to go, there is precedent for the majority of config be provided via YAML files.

@gregfurman - perhaps was more critical that there is other ways to handle this situation i.e. sidecar / init containers, posting into bento running streams_mode, that we discussed on discord before. That adding this feature is moving the complexity from already existing solutions into Bento.

Also Greg mentioned that we could perhaps have the concept of 'hooks', such there would be an 'on_init' hook that would evoke some user-defined logic at start-up and there we could fetch the config.

[Omar1H1]

Hi @jem-davies thanks for the detailed direction.

I think what you have suggested is the right way to go. If I may add, maybe naming it remote_config or config_source instead of http_client to make the intent clearer.

I totally understand @gregfurman concerns, the alternatives he mentioned are valid. That said, I do think there's value in making this a first-class feature rather than pushing it to the infrastructure layer. Happy to implement this direction if there's alignment. If not, that's completely fine and I can close the PR for now. maybe a clearer need will come up in the future.