Cannot render XY chart in Databricks

[Alek99]

Summary

XY charts do not render when displayed directly in a Databricks notebook.

Source: Linear ENG-10647

Reproduction

Install XY and restart Python:

%pip install xy
%restart_python

Then run:

import xy

sales_df = spark.table("samples.bakehouse.sales_transactions")
top_items_df = (
    sales_df.groupBy("product")
    .sum("quantity")
    .orderBy("sum(quantity)", ascending=False)
)

display(
    xy.bar_chart(
        xy.bar(
            top_items_df.select("product").toPandas()["product"].tolist(),
            top_items_df.select("sum(quantity)").toPandas()["sum(quantity)"].tolist(),
        ),
        title="Top Items Sold",
    )
)

Observed behavior

The chart does not render.

The notebook reports this anywidget warning:

[anywidget] Deprecation Warning for xy.widget.FigureWidget: Direct export of a 'render' will likely be deprecated in the future. To migrate ...

Remove the 'export' keyword from 'render'
-----------------------------------------

export function render({ model, el }) { ... }
^^^^^^

Create a default export that returns an object with 'render'
------------------------------------------------------------

function render({ model, el }) { ... }
         ^^^^^^
export default { render }
                 ^^^^^^

Pin to anywidget>=0.9.0 in your pyproject.toml
----------------------------------------------

dependencies = ["anywidget>=0.9.0"]

To learn more, please see: https://github.com/manzt/anywidget/pull/395.

Chrome DevTools shows:

index.js:37 TypeError: chart payload must be an ArrayBuffer or ArrayBuffer view
    at a (f2e80c57-6672-4aa1-9d7e-64205b957ea9:1:322)
    at o (f2e80c57-6672-4aa1-9d7e-64205b957ea9:1:403)
    at Array.map (<anonymous>)
    at c (f2e80c57-6672-4aa1-9d7e-64205b957ea9:1:954)
    at Object.bn [as render] (f2e80c57-6672-4aa1-9d7e-64205b957ea9:894:39145)
    at WidgetBinding.createView (index.js:107:41)

Workaround

Rendering the chart as HTML works:

def displayXY(chart):
    displayHTML(chart.to_html())