[SPARK-57681][SQL] Support dynamic table options for DELETE and UPDATE

[anuragmantri]

What changes were proposed in this pull request?

This PR add the dynamic table options syntax to DELETE and UPDATE statements.

Why are the changes needed?

This is a follow up to SPARK-49098 which itself was a follow up to SPARK-36680

The DSv2 API already provides RowLevelOperationInfo.options() and LogicalWriteInfo.options() for this purpose, but they were always empty for DELETE and UPDATE because the rewrite rules hardcoded CaseInsensitiveStringMap.empty().

This issue came up again in the Iceberg repo.

Does this PR introduce any user-facing change?

Yes, it adds new SQL syntax

  DELETE FROM catalog.db.table WITH (`k` = 'v') WHERE id > 10

  UPDATE catalog.db.table WITH (`k` = 'v' )
  SET status = 'archived' WHERE created_at < '2024-01-01'.

How was this patch tested?

• Parser (DDLParserSuite)
• End-to-end (DeleteFromTableSuiteBase, UpdateTableSuiteBase)DataSourceV2Relation
• LogicalWriteInfo (InMemoryRowLevelOperationTable)

Was this patch authored or co-authored using generative AI tooling?

I used Claude Code (Claude Opus 4.8) to generate the code and tests and verified manually.

[anuragmantri]

@szehon-ho, tagging you since you made the INSERT counterpart :)

[uros-b]

Please consider this concern: silent option-drop on non-rewrite delete paths: for a table implementing both SupportsRowLevelOperations and SupportsDeleteV2 (e.g. Iceberg), a DELETE … WITH(...) WHERE can take the metadata-only/deleteWhere path (OptimizeMetadataOnlyDeleteFromTable / DataSourceV2Strategy), which has no options parameter; so the user's options are silently ignored. Same for the TruncatableTable truncate path. This would be a new user-visible "silently ignored" surprise, and no test currently covers this path.

[anuragmantri]

Great catch. I looked at those paths. The options are indeed not being passed all the way. Fixing this requires DSv2 API changes something like this with default methods.

  // SupportsDeleteV2
  default void deleteWhere(Predicate[] predicates, CaseInsensitiveStringMap options) {
    deleteWhere(predicates);   // default ignores options → fully back-compatible
  }
  // TruncatableTable
  default boolean truncateTable(CaseInsensitiveStringMap options) {
    return truncateTable();
  }

I can make this change in this same PR or can do a follow up if DSv2 changes need separate PRs. Let me know what you think is the best way forward.

[uros-b]

So this PR makes INSERT/DELETE/UPDATE accept WITH(...), but how about MERGE?

[anuragmantri]

I intentionally did not add MERGE as the semantics are a bit different since MERGE statements have source and target relations, I'm still thinking about how options would look like there. I can file a separate JIRA and PR for MERGE later.

[uros-b]

Please make sure to either update the SQL reference docs, or file a followup Jira ticket to do so later.

[anuragmantri]

For sure. I took a look and looks like INSERT docs are also not updated. I will create a follow up JIRA to update all options related DMLS at once.