FIREFLY-2066: Consolidate the loading UI of Spectral Lines feature

[jaladh-singhal]

Fixes FIREFLY-2066

• Moved the spectral lines panel out of Chart options into its own dialog (toolbar button, left side) rendering SpectralLinesPanel - loading is a separate concern from per-trace display options, and the panel is shared app-wide, not per-chart
• Recommended line lists are now driven by a charts.spectrum.linelists app-config property ({label, src?} array, src can be a bundled resource or remote URL) instead of one hardcoded combined CSV - retired linelist_combined.csv for separate hspot_lines.csv/pahfit_lines.csv/spherex_lines.tbl resources plus a remote JWST list as a src-as-URL example
• Checked lists and the uploaded list merge into one client-side table (spectral-lines), rebuilt only on explicit "Load Lines"/"Clear All", never automatically on checkbox/mapping change
• Wavelength units normalize to microns at merge time for both sources; SpectralLinesProcessor warns server-side when an available line list's wavelength column has no units metadata, and the upload column mapper shows similar feedback for the uploaded file's wavelength column

Additional changes/cleanup:

• UploadTableSelector enhanced for use in SpectralLinesPanel: added allowClear (adds a chip to clear the uploaded file), uploadTblOptions (keeps the upload out of the main Results view), and a per-field getFeedback hook (surfaces the uploaded wavelength column's detected/assumed unit)
• SpectrumOptions: Standardized default resolution logic for redshift options (getEffectiveSpectralFrameOption) and de-duplicated a few hardcoded spectral-frame/redshift constants
• Added µJy as a supported flux unit in SpectrumUnitConversion (for SPHEREx spectrum results)
• Fixed an off-center "no data" message in BasicTableView

Testing

All the test files linked in the ticket.

https://firefly-2066-spec-lines-loading.irsakubedev.ipac.caltech.edu/firefly

Core functionality

1. Upload "60..." (SPHEREx spectrum) -> open the spectral lines dialog from the top-left of the "Active chart" tab -> select the SPHEREx line list -> "Load Lines". They should populate the Lines table in the dialog and get plotted in the chart
2. Deselect a couple of lines from the table - they should disappear from the chart in real time. Filter the lines table by label "Paschen" -> only Paschen lines should show up. Close the dialog
3. Now upload the "Spitzer_IRS..." spectrum. The spectrum chart should plot with the same line selection as the previous spectrum
4. Open the spectral lines dialog for this (2nd) spectrum, make changes to the line selection here -> switch back to the 1st spectrum's table tab -> the spectral lines in the 1st spectrum's active chart should have updated compared to before
5. "Clear All" in the dialog should remove all lines from all charts - a one-click revert to the app's initial state

Upload line list

• Upload "jwst_linelist.tbl" in the "Select line lists to load" collapsible -> map columns, confirm feedback text and loading behavior
• The "Clear" chip should remove the uploaded list - the same behavior as unchecking an available-list checkbox

Redshift correction

• Open "Chart options" for an active spectrum with "Observed Frame" selected -> under "Redshift correction," set "Enter Redshift" to 0.5 and "Apply" -> spectral lines should redshift right of their lines-table wavelength. Switch to "Rest Frame" with the same redshift and "Apply" -> lines return to their original wavelength and the spectrum blueshifts left instead. Net effect: lines and spectrum end up in the same relative position, just with the X-axis shifted - confirming the correction applies correctly in both frames
• Most of our spectrum files have no "target"/"derived" redshift in their metadata, so they start at zero redshift correction - "Spitzer_IRS..." is the exception. Refresh the app, load it, and in the spectral lines dialog load the SPHEREx line list -> lines should plot correctly shifted (the last line's wavelength in the Lines table should be less than its hover-tip wavelength in the chart). Open "Chart options" -> "Observed Frame" and the target redshift should already be selected, confirming this is the factor lines were shifted by. This verifies the redshift is picked up correctly even when Chart options was never opened (so it's not yet in the chart store)

Regression

• Pinned charts should propagate spectral lines and otherwise work as before
• Upload two "table_OBJECT.." (Euclid) spectrum files -> pin their charts -> in "Pinned Charts," combine them using the toolbar's combine option (two-rings icon) -> spectral lines should propagate without affecting the combine functionality
• Grouped traces should work as before

[jaladh-singhal]

NOTE: Last option "JWST line list (remote)" is just a test for recommended/pre-installed line list coming from a remote server rather than bundling with firefly static resources.

I plan to remove it when merging this PR.

[lrebull]

This is so cool!

See here for two items, at least on my screen:

Item 1: The labels are a tiny bit too far away from their line. I've weeded down this list, but even so, on the edges you can see that the label for "OCS" is actually closer to the line for "13 CO" than it is its own line, and same for Paschen gamma -- it's closer to Paschen delta than its own line. These need to be a bit closer to the corresponding line, IMHO.

Item 2: you can see here how a lot of the labels run off the top. Less than ideal. Is there a way for users to tweak this? move them down?

I love the dynamic response of the pop-up. At the end of the process, we have this at the bottom of the pop-up:

We have trained people elsewhere that clicking on the green version of that little plot icon does things. Now it's blue, and clicking on it doesn't do anything. Is that good UI/UX? The arrow at the end also doesn't do anything. What is it meant to indicate? I mean, the first thing I did with the pop-up was grab and move it to cover the image in the tri-view so that i could see the plot, and maybe that makes me different than you expected?

Ooh, this should not be default behavior:

even if the line list goes out past 50 um, if the data only go to 8.6 um, the plot, even after adding the line list, should only go to 8.6 um. The priority should be to show me the data, not the line list.

Actually, this is a problem. Look, I've tried to weed the line list down to only be <8.6 um, and the table is filtered appropriately (see scrollbar on the line list), but the plot is showing me lines >8 um.

OH WAIT you applied a redshift to FV Tau/c ?!? lolololol ok you threw me for a loop with that one. This thing is 140 pc away; it definitely doesn't have a redshift of 0.546. But the plot says "observed frame" not "rest frame" ... hm. I'm still confused; we need an xgal person to look at this.

In any case, I think what we need to do is echo the redshift to the top of the spectral line list. In between the "select line lists to load" pulldown thing, and before "select lines to plot", I propose something in italics: "redshift has been applied to lines in plot; see plot options" or something better than that. (i'm sure others will Have Opinions.) The idea is to get a pointer or hint or something in here, because I, for one, didn't understand what could possibly be going on here.

Let me see if I can get some other scientists in on this.

[jaladh-singhal]

@lrebull thanks for the feedback.

Item 1: The labels are a tiny bit too far away from their line. I've weeded down this list, but even so, on the edges you can see that the label for "OCS" is actually closer to the line for "13 CO" than it is its own line, and same for Paschen gamma -- it's closer to Paschen delta than its own line. These need to be a bit closer to the corresponding line, IMHO.

I'm not seeing this issue in Chrome - the labels are sticking to the lines. But I can see this issue in Firefox assuming that's what you used for testing? Will debug why and try to fix.

Item 2: you can see here how a lot of the labels run off the top. Less than ideal. Is there a way for users to tweak this? move them down?

Yes, I meant to fix this but forgot. And now it's more pronounced with looong labels in the lists you all gave me. Will fix.

I love the dynamic response of the pop-up. At the end of the process, we have this at the bottom of the pop-up ...We have trained people elsewhere that clicking on the green version of that little plot icon does things. Now it's blue, and clicking on it doesn't do anything. Is that good UI/UX? The arrow at the end also doesn't do anything. What is it meant to indicate?

This is just a decorator to indicate that "hey, look beneath this popup where results screen is". I can see the click-behavior confusion due to blue color. How about I revert it to the same color of text (grey-ish)? OR do you want me to remove it altogether? I'm also open to changing the wording of response text to incorporate some version of "look beneath in results" if you have any better suggestions.

even if the line list goes out past 50 um, if the data only go to 8.6 um, the plot, even after adding the line list, should only go to 8.6 um. The priority should be to show me the data, not the line list.

Hmm, I think this happens because this what plotly is doing by default to show all elements you plotted on chart. I'll have to override the chart layout to always only show the range of data traces. Will try to fix.

OH WAIT you applied a redshift to FV Tau/c ?!? lolololol ok you threw me for a loop with that one. This thing is 140 pc away; it definitely doesn't have a redshift of 0.546. But the plot says "observed frame" not "rest frame" ... hm. I'm still confused; we need an xgal person to look at this.

I changed the original redshift value in the file (from 1e-3 something) to 0.5 so that redshift correction is easily noticeable. Please disregard the scientific accuracy for just this spectrum file that has hand-crafted metadata 😄

In any case, I think what we need to do is echo the redshift to the top of the spectral line list. In between the "select line lists to load" pulldown thing, and before "select lines to plot", I propose something in italics: "redshift has been applied to lines in plot; see plot options" or something better than that. (i'm sure others will Have Opinions.) The idea is to get a pointer or hint or something in here, because I, for one, didn't understand what could possibly be going on here.

This becomes clear if you open the chart options dialog because it tells that redshift correction is being applied to spectral lines in observed frame whereas to spectrum in rest frame. Because we settled on this one spectral line dialog to rule them all, I cannot show any wording specific to one spectral chart since it may be irrelevant for other spectral charts open. How about right after the table we add a note: "If spectral chart is in observed frame, redshift gets applied to the selected lines in the plot; see Chart Options dialog"? I plan to keep it after the table because the lines in table are always in rest frame, it's only when plotting (after table selection before "n lines plotted" response) it gets determined where they land on wavelength axis.

[robyww]

Good job. I found it go be working as planned and I don't see any bugs.

Code

A couple of small suggestions and I want to have a conversation about how we generate ids and how the person doing firefly config uses them. I don't think we have it right yet.

UI

• The is a UI issue with the Upload table tables dialog. The dialog is coming up too narrow and the Table Meta section is not fully displayed.

• in Spectral Lines Option the table sometime looks like this:

[robyww]

is it necessary to the a custom hook if this is the only place it is use? Or is there a plan to use it elsewhere?

[jaladh-singhal]

This is just to encapsulate useEffect logic which is specific to spectral lines. Otherwise there would be so many other imports from SpectralLines.jsx.

And yes in future if we initialize chart differently or there is some other call site that needs it, this can be reused. It was super helpful in testing too.

[robyww]

useSpectrumInputs is written as a hook but it looks like it is only a utility function. Maybe it should be useSpectrumInputs

[jaladh-singhal]

I don't understand your comment completely. This wasn't a great naming choice but this use* pattern is what we have all over Spectrum|Scatter|BasicOptions.jsx UI components. My changes don't effect this, I'm just taking out a component it returns.

[robyww]

After looking at this. I have two concerns.

1. because there is nota separate id only an id build from the title, If the title get changed (at it will) then another app the is specifying just one will break.
2. This is really only only place our predefined lines list is documented. I don't have a solution it is just a concern.

I think you, I and @loitly should have a discussion about this before you merge.

I think the ID should be something like.

• SPHEREx-v1
• Spitzer-PAHFIT
• HSPOT

[loitly]

Yes, let's talk about it. I don't quite understand the concerns.

[jaladh-singhal]

I'm generating IDs on the fly as we discussed but it happens one time any app is intialised server side and then client queries that cached {ID, label, src} so I don't see how it will break other apps.

[lrebull]

Yes, i'm using firefox.

Yeah, i really don't want to use the 'results' icon in a way differently here from where it gets used elsewhere; that seems messy. let me see if can rally the other scientists for input on this one ...

A note after the table would work. again, need input from other scientists ...

[loitly]

UI worked great. Just a few suggested changes.

[jonesmg]

I tried this out for a SPHEREx spectrum at non-zero redshift (3C273 at z = 0.15823). The line list was super helpful for figuring out what various features in the spectrum were and the redshift shift seems to have worked as intended.

There are some places where the lines are so close together that I couldn't read which was which.

This is probably an unavoidable problem because the lines are where they are. It was also not a big deal because I was able to zoom in. However, it might be helpful to have an easy way to zoom only in the x-direction. I didn't really want to change the y-scale, just zoom in in the x-direction to separate the lines. This might be too much of a change though.

I did find that a lot of the line name overlapped with the top of the panel, which made them a little difficult to read, even after zooming in.

[jaladh-singhal]

Thanks for testing it @jonesmg.

I did find that a lot of the line name overlapped with the top of the panel, which made them a little difficult to read, even after zooming in.

I'm fixing it. Luisa also mentioned it.

However, it might be helpful to have an easy way to zoom only in the x-direction. I didn't really want to change the y-scale, just zoom in in the x-direction to separate the lines. This might be too much of a change though.

It's already possible with plotly's interactive chart features. Take your mouse near the X-axis until it changes to the following cursor (< >).

Then scrolling/pinching your mouse here will zoom only the X-axis, which is similar to if your cursor was inside the chart area, scrolling/pinching would have zoomed both axes. @lrebull this might be worth documenting when you make docs for spectral lines feature.

[lrebull]

wow, i did not know that! making a note of it ..