Skip to content

Data quality: measurements that don't add up, and automatic unit and date fixes to confirm #133

Open

Description

@Luen

What. The importer flags the items below on the Data quality page, with workbook, sheet and row. It keeps the values in the analyses.

Measurements that don't add up (32 in all, category "morphometrics")

Workbook TL shorter than FL FL shorter than PCL Mass far from expected for TL
2016-17 1
2018-19 2 2
2019-20 1
2020-21 1 1
2021-22 4 1
2022-23 1 2 3
2023-24 1 1
2024-25 5 1 2
2025-26 1 1 1

Automatic fixes (warnings under "units" and "dates")

  • Masses in grams: 15 masses look like grams and are divided by 1000 (14 in 2021-22, 1 in 2022-23). After conversion, the 2022-23 one is far too light for its length, so that fix is probably wrong; it may be a missing decimal point.
  • Lengths in cm: 5 lengths in mm sheets look like cm and are multiplied by 10 (2017-18, 2018-19, 2022-23).
  • Dates a year off: 13 dates fall a year outside their season and are moved into it (2013-14, 2014-15, 2015-16, 2017-18, 2020-21).

Other

  • One 2023-24 set has no readable date.
  • 23 field captures have no set-log row for their date and nursery, so they don't count towards catch rates.

Questions for @JodieRummer

  1. Could someone check the 32 measurements against the field sheets and fix them at the source?
  2. Until then, should the size and condition analyses leave out measurements that don't add up? For now they're kept.
  3. Are the automatic fixes right (grams, cm, and dates a year off)?
  4. Are the sets for those 23 captures missing from the set logs?

Acceptance

  • The sheets are fixed at the source, or the lab's rule is recorded in src/lib/server/import/logs.ts and AGENTS.md.

Part of #75.

Activity

  1. added
    questionFurther information is requested
    data-qualityIssues in the source spreadsheets for the lab to review
    on Nov 3, 2026
  2. JodieRummer commented on Nov 5, 2026

    @JodieRummer
    Collaborator

    Some times the dates is off because the shark is caught in a previous year initially and then recaptured in that current season.

  3. Luen commented on Nov 8, 2026

    @Luen
    CollaboratorAuthor

    Thanks. I checked all 13 automatic date fixes against the set logs and each shark's capture history:

    • All 13 look like year typos. The corrected dates fall into the set-log sequence or match the row's set number, and none matches an earlier capture of the same shark.
    • One row fits what you describe: a 2015-16 sicklefin lemon noted "one from last year?". Its set number puts the capture on the date the importer already uses.

    #164 handles your case for future sheets. If a recapture row carries the shark's first capture date from an earlier season, moving the year would land on that capture's anniversary. So the importer now takes the date of the row's set, when it has a set number. It changes no capture in the current sheets.

    @JodieRummer, these are still open:

    1. The 32 measurements that don't add up: could someone check them against the field sheets and fix them at the source? The Data quality page lists each by workbook, sheet and row. Once fixed, they drop off that page.

    2. Until then: should the size and condition analyses leave those 32 out? If yes, /morphometrics, /neonates and /growth stop using them; for now they're kept.

    3. Units: are the automatic conversions right?

      • 15 masses read as grams and divided by 1000 (14 in 2021-22, 1 in 2022-23). The 2022-23 one is far too light for its length after conversion; is it a missing decimal point?
      • 5 lengths read as cm and multiplied by 10 (2017-18, 2018-19, 2022-23).

      Each conversion you confirm stays; any you reject is undone.

    4. 23 field captures have no set-log row for their date and nursery. Are those sets missing from the set logs? If they're added at the source, those captures count towards catch rates.

  4. added a commit that references this issue on Nov 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    data-qualityIssues in the source spreadsheets for the lab to reviewquestionFurther information is requested

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions