[SparkConnect] Support url path in SparkConnect connection string

[hiboyang]

Current Spark Connect uses connection string like sc://host1:15002, which does not support a path in the url, e.g. not support value like sc://host1/path1:15002.

In Kubernetes, people can set up an Ingress to expose Spark Connect driver endpoint by a matching url, e.g. http://host1/path1 can be routed to the driver endpoint. Need to update Spark Connect client to support this scenario.

This could be a Spark Client side change, e.g. in Python Spark Connect client, connect to server based on the host name, and intercept GPRC call to avoid path interfering GRPC protocol.

I am working on a prototype, also open to hear feedback / suggestions.

[viirya]

Thanks @hiboyang for creating this issue.

So per our discussion:

For an Ingress-routing-hint prototype, a gRPC metadata header (e.g. an x- header the client sets from the URL path) is probably less invasive than changing :path, and Ingress controllers can route on headers.

[hiboyang]

Thanks @viirya for the comment! Yes, we can explore both the header and path, and make them work if possible.