Add Spark 4.0 scalar Arrow UDF and grouped-map support

[SparkSnail]

Summary

Add Spark 4.0.x support for scalar Arrow UDFs (eval type 200) and grouped-map UDFs (eval type 201), using both Apache Arrow and Microsoft.Data.Analysis representations.

Changes

• Add Spark 4-specific command parsing and execution for the two supported eval types.
• Pass the declared grouped-map schema consistently to the JVM and .NET Worker.
• Validate output shape, schema, types, and nullability before returning results.
• Handle Arrow stream completion and failures without appending protocol frames to partially written output.
• Preserve existing Spark 2.x/3.x routing and command formats.
• Add unit and E2E coverage, and include the ArrowUdf category in Spark 4.0.0–4.0.4 CI lanes on Windows and Linux.

Validation

• Managed unit tests: 225 passed.
• Worker unit tests: 243 passed; one Linux-only test skipped on Windows.
• Worker builds: net472, net48, and net8.0 succeeded with no warnings or errors.
• Local Spark 4.0.4 E2E on Windows and Linux: 23 passed per platform, covering Arrow UDFs and existing compatibility cases.
• Local official Apache Spark 3.5.3 Arrow regression tests on Windows and Linux: 6 passed per platform; 8 Spark 4-only tests skipped.

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

[SparkSnail]

/AzurePipelines run

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).

[azure-pipelines]

Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.