Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,8 @@ samples/fixture/ synthetic workbooks for tests/CI (committed)

- Sheet layouts differ every season. Columns are matched by **normalised header aliases** (`headerKey`), never position. Add aliases rather than special-casing a season.
- **Units:** lengths are stored in mm. 2013-15 logs are cm despite "mm" headers; 2021-22 `Cm Log`/`Na Log` are cm; adults are cm. `harmoniseUnits()` decides per sheet from medians and logs every conversion.
- **Dates** a year outside their season (e.g. `2014-01-11` in 2014-15) are corrected by `fixSeasonYear` and logged. Seasons run Aug–Jul.
- **Dates** a year outside their season (e.g. `2014-01-11` in 2014-15) are corrected by `fixSeasonYear` and logged. Seasons run Aug–Jul. Most are year typos (all 13 in the current sheets fit their set log), but a recapture row can carry the shark's first capture date from an earlier season (Jodie Rummer, #133): a year-off capture with a set number takes that set's date (`yearOffSetDate`) instead of the anniversary.
- **Human pressure** (#131): the per-nursery scores are the paper's own, from the Bouyoucos et al. (2024) dataset (doi:10.25903/2SFB-QX79, CC BY-NC 4.0) committed as `external/pressure-bouyoucos2024.csv` (`fish`, `tramp` = direct human impact, `art`). They override the private ranking workbook, whose values were digitised from Fig. 4. The composite and tier follow the workbook: min–max normalise, average, thirds of 0–1. The JCU record lists the 2020 Coral Reefs paper as its related publication, but the 2024 paper's data statement cites it and its CVs match the paper's.
- 2013-14 `Shark log physioshark` / `Set Log physioshark` duplicate `Catch data` / `Set Log` and are skipped (#9). 2013-14 recaptures are grouped in its `Recaptures` sheet; 2014-15 `Recaptures` holds extra catches; 2015-16 stores recaptures as horizontal column blocks (`1st Recap Site`…).
- 2016-17 `Shark Log` has a date _value_ as its date header; 2021-22 `Set Log` has a garbled date header (`16(`). Both are handled.
- Species codes: `Cm` blacktip, `Na` sicklefin lemon, `Nf` tawny nurse shark. 2024-25 writes the nurse shark as `GF` / `Gf1` (`GF Log Juv`), probably from _Ginglymostoma_, the Atlantic nurse shark genus; it is read as `Nf` (confirmed by Jodie Rummer, #12).
Expand Down
11 changes: 11 additions & 0 deletions external/pressure-bouyoucos2024.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
site,fish,art,tramp,cpue_cm,cpue_na,sem_cm,sem_na
Apaura,42.69236451,0.123961573,0.091762846,0.08,0.62,0.055902529,0.148493435
Haapiti,29.47250429,0,0.039056976,1.43,0,0.250021024,0
Maharepa,56.29875083,0.206968724,0.072982856,0.12,0.07,0.065778905,0.048951049
Papetoai,47.14283681,0.265649336,0.128103445,1.03,0.05,0.19634497,0.048387097
Pihaena,67.86138812,0.340907026,0.040636815,0.04,0.75,0.291767313,0.302063044
Pointe de Paorea,43.13851462,0.175035251,0.061698585,0.66,0,0.036832413,0
Tiki,16.51820484,0.06899137,0.020396012,0.16,1.11,0.118092562,0.304299991
Vaiane,55.77345143,0.031963967,0.064490557,0.21,4.78,0.156497555,1.330220239
Vaiare,38.54370972,0.008827053,0.035196653,0.56,0,0.171137721,0
Valorie,50.77844287,0.169001643,0.142854799,0.96,0.26,0.300298717,0.155061561
3 changes: 2 additions & 1 deletion src/lib/server/import/external.ts
Original file line number Diff line number Diff line change
Expand Up @@ -36,5 +36,6 @@ export const EXTERNAL_FILES = [
'tide-papeete-hourly.csv.gz',
'climate-monthly.csv',
'cyclones.csv',
'school-holidays.csv'
'school-holidays.csv',
'pressure-bouyoucos2024.csv'
];
24 changes: 22 additions & 2 deletions src/lib/server/import/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,7 @@ import {
parseRecaptures2014,
parseSetSheet,
recaptureGroups2013,
yearOffSetDate,
type RawCapture,
type RawSet
} from './logs';
Expand All @@ -54,6 +55,7 @@ import {
parseAdults,
parseNurseryGps,
parsePressure,
pressureFromDataset,
type AdultCapture,
type AdultEffort,
type SitePressure,
Expand All @@ -73,7 +75,7 @@ import { SITE_DEFS } from './sites';
import { readWorkbook } from './xlsx';

/** Bump when import logic changes so existing caches rebuild on the next start. */
export const IMPORTER_VERSION = '9';
export const IMPORTER_VERSION = '10';

const MIGRATIONS = import.meta.glob('/drizzle/*.sql', {
query: '?raw',
Expand Down Expand Up @@ -432,6 +434,18 @@ export function runImport(
for (const c of caps) {
if (c.kind === 'lab_release') continue;
let s: (typeof sets)[number] | undefined;
const setDate = yearOffSetDate(c, bySeasonNo.get(`${c.season}|${c.setNo}`) ?? []);
if (setDate && setDate !== c.date) {
log.add(
'warning',
'dates',
`Capture dated ${c.writtenDate} (outside season ${c.season}) is in set ${c.setNo}, dated ${setDate}; read as ${setDate}`,
c.ref,
'date',
c.writtenDate
);
c.date = setDate;
}
if (c.setNo != null) {
const cands = (bySeasonNo.get(`${c.season}|${c.setNo}`) ?? []).filter(
(x) => !c.date || !x.date || Math.abs(daysBetween(x.date, c.date)) <= 1
Expand Down Expand Up @@ -775,7 +789,13 @@ export function runImport(
aliases.set(x.siteId, s);
}
const gps = new Map(p.gps.map((g) => [g.siteId, g]));
const pressure = new Map(p.pressure.map((g) => [g.siteId, g]));
// The paper's dataset (#131) overrides the ranking workbook's values digitised from its Fig. 4.
const dataset = pressureFromDataset(
readCsv(opts.externalDir, 'pressure-bouyoucos2024.csv'),
log,
'external/pressure-bouyoucos2024.csv'
);
const pressure = new Map([...p.pressure, ...dataset].map((g) => [g.siteId, g]));
const desc = new Map(p.descriptions.map((g) => [g.siteId, g]));
const siteRows: (typeof schema.sites.$inferInsert)[] = SITE_DEFS.map((d) => {
const g = gps.get(d.id);
Expand Down
24 changes: 22 additions & 2 deletions src/lib/server/import/logs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -75,9 +75,26 @@ export interface RawCapture {
stomachContents: string | null;
/** Grouping key from a "Recaptures" sheet (2013-15) used to link untagged captures. */
group?: string;
/** The date as written, when it was a year outside the season and got moved into it (#133). */
writtenDate?: string;
ref: SourceRef;
}

/**
* The date of the set a year-off capture belongs to (#133). A capture date a year outside its
* season is usually a typo, but it can be the shark's first capture in an earlier season written on
* its recapture row; then moving the year lands on that capture's anniversary, not the recapture.
* When the set number finds the set (same nursery, or the only set with that number), its date wins.
*/
export function yearOffSetDate(
c: Pick<RawCapture, 'writtenDate' | 'setNo' | 'siteId'>,
sets: { siteId: string | null; date: string | null }[]
): string | null {
if (!c.writtenDate || c.setNo == null) return null;
const s = sets.find((x) => x.siteId === c.siteId) ?? (sets.length === 1 ? sets[0] : undefined);
return s?.date ?? null;
}

export interface RawSet {
season: string;
setNo: number | null;
Expand Down Expand Up @@ -317,17 +334,19 @@ export function parseCaptureSheet(
): RawCapture | null => {
const ref = { file, sheet: sheet.name, row: r };
let date = dateOf(cell(row, c.date));
let writtenDate: string | undefined;
if (date) {
const fixed = fixSeasonYear(date, season);
if (fixed.fixed) {
log.add(
'warning',
'dates',
`Date ${date} outside season ${season}; read as ${fixed.date}`,
`Date ${date} outside season ${season}; read as ${fixed.date} (a year typo, or the shark's first capture in an earlier season)`,
ref,
'date',
date
);
writtenDate = date;
date = fixed.date;
}
}
Expand Down Expand Up @@ -378,6 +397,7 @@ export function parseCaptureSheet(
siteId,
setNo: int(cell(row, c.setNo)) ?? inherit?.setNo ?? null,
date,
writtenDate,
sharkNo: str(cell(row, c.sharkNo)) ?? inherit?.sharkNo ?? null,
captureNo: captureNo ?? int(capNoRaw),
kind,
Expand Down Expand Up @@ -537,7 +557,7 @@ export function parseRecaptures2014(
log.add(
'warning',
'dates',
`Date ${date} outside season ${season}; read as ${fixed.date}`,
`Date ${date} outside season ${season}; read as ${fixed.date} (a year typo, or the shark's first capture in an earlier season)`,
ref,
'date',
date
Expand Down
62 changes: 62 additions & 0 deletions src/lib/server/import/reference.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,62 @@
import { describe, expect, it } from 'vitest';
import { IssueLog } from './issues';
import { yearOffSetDate } from './logs';
import { pressureFromDataset } from './reference';

describe('pressureFromDataset (#131)', () => {
// Synthetic scores, laid out like the Bouyoucos et al. (2024) dataset.
const rows = [
{ site: 'Apaura', fish: '10', tramp: '0.1', art: '0' },
{ site: 'Haapiti', fish: '30', tramp: '0.3', art: '0.2' },
{ site: 'Pointe de Paorea', fish: '20', tramp: '0.2', art: '0.1' },
{ site: 'Nowhere', fish: '5', tramp: '0.5', art: '0.5' }
];

it('normalises each metric, averages them and tiers the composite in thirds of 0–1', () => {
const log = new IssueLog();
const out = pressureFromDataset(rows, log, 'external/pressure.csv');
expect(out.map((s) => s.siteId)).toEqual(['apaura', 'haapiti', 'paorea']);
expect(out[0]).toMatchObject({ fishingEffort: 10, humanImpact: 0.1, artificialisation: 0 });
expect(out.map((s) => s.composite)).toEqual([0, 1, 0.5]);
expect(out.map((s) => s.tier)).toEqual(['Low', 'High', 'Medium']);
expect(log.items).toHaveLength(1);
expect(log.items[0].message).toMatch(/unknown site "Nowhere"/);
});

it('leaves the composite blank when a metric is missing', () => {
const out = pressureFromDataset(
[rows[0], { ...rows[1], art: '' }, rows[2]],
new IssueLog(),
'external/pressure.csv'
);
expect(out[1].composite).toBeNull();
expect(out[1].tier).toBeNull();
});
});

describe('yearOffSetDate (#133)', () => {
const sets = [
{ siteId: 'tiki', date: '2016-03-10' },
{ siteId: 'apaura', date: '2016-03-03' }
];

it('dates a year-off capture by its set at the same nursery', () => {
expect(yearOffSetDate({ writtenDate: '2015-01-20', setNo: 94, siteId: 'apaura' }, sets)).toBe(
'2016-03-03'
);
});

it('uses the only set with that number when the nursery differs or is missing', () => {
expect(yearOffSetDate({ writtenDate: '2015-01-20', setNo: 94, siteId: null }, [sets[0]])).toBe(
'2016-03-10'
);
expect(yearOffSetDate({ writtenDate: '2015-01-20', setNo: 94, siteId: null }, sets)).toBeNull();
});

it('ignores captures whose date was in season or that have no set number', () => {
expect(yearOffSetDate({ setNo: 94, siteId: 'apaura' }, sets)).toBeNull();
expect(
yearOffSetDate({ writtenDate: '2015-01-20', setNo: null, siteId: 'apaura' }, sets)
).toBeNull();
});
});
48 changes: 47 additions & 1 deletion src/lib/server/import/reference.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,10 @@
/**
* Reference workbooks (#5, #6): nursery logger GPS, the anthropogenic pressure ranking digitised
* from Bouyoucos et al. (2024) and the 2025-26 adult blacktip capture datasheet.
* from Bouyoucos et al. (2024) and the 2025-26 adult blacktip capture datasheet. The exact
* pressure scores come from the paper's dataset in external/ (#131) and replace the digitised ones.
*/
import { cell, mapColumns, type Matcher } from './columns';
import { val, type Row } from './external';
import type { IssueLog } from './issues';
import { dateOf, dmsToDecimal, durationOf, int, num, sexOf, str, text, timeOf } from './normalize';
import { resolveSite } from './sites';
Expand Down Expand Up @@ -53,6 +55,50 @@ export interface SitePressure {
tier: string | null;
}

/**
* Per-nursery scores from the Bouyoucos et al. (2024) dataset (JCU Research Data,
* doi:10.25903/2SFB-QX79, CC BY-NC 4.0; external/pressure-bouyoucos2024.csv): `fish` is relative
* fishing effort, `tramp` direct human impact (trampling) and `art` coastal artificialisation.
* The composite follows the ranking workbook: each metric min–max normalised to 0–1 across the
* nurseries, averaged, then Low / Medium / High in thirds of 0–1.
*/
export function pressureFromDataset(rows: Row[], log: IssueLog, file: string): SitePressure[] {
const sites = rows.flatMap((r, i) => {
const siteId = resolveSite(r.site);
if (!siteId) {
log.add('warning', 'sites', `Pressure dataset: unknown site "${r.site}"`, {
file,
sheet: '',
row: i + 2
});
return [];
}
return [{ siteId, fish: val(r.fish), tramp: val(r.tramp), art: val(r.art) }];
});
const norm = (key: 'fish' | 'tramp' | 'art') => {
const xs = sites.map((s) => s[key]).filter((v): v is number => v != null);
const lo = Math.min(...xs);
const span = Math.max(...xs) - lo;
return (v: number | null) => (v == null || !(span > 0) ? null : (v - lo) / span);
};
const [nf, nt, na] = [norm('fish'), norm('tramp'), norm('art')];
return sites.map((s) => {
const parts = [nf(s.fish), nt(s.tramp), na(s.art)];
const composite = parts.every((v) => v != null)
? parts.reduce((a, b) => a! + b!, 0)! / parts.length
: null;
return {
siteId: s.siteId,
fishingEffort: s.fish,
humanImpact: s.tramp,
artificialisation: s.art,
composite,
tier:
composite == null ? null : composite < 1 / 3 ? 'Low' : composite < 2 / 3 ? 'Medium' : 'High'
};
});
}

export function parsePressure(wb: Workbook, log: IssueLog, file: string): SitePressure[] {
const sheet = wb.sheets.find((s) => /composite/i.test(s.name));
if (!sheet) return [];
Expand Down
3 changes: 2 additions & 1 deletion src/routes/+layout.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -213,7 +213,8 @@
Weather &amp; waves: Open-Meteo (ERA5, CC BY 4.0). SST: {SST_SOURCE}. Heat stress: NOAA
Coral Reef Watch via PacIOOS. Chlorophyll: NOAA STAR and CoastWatch (VIIRS). Tides: UHSLC
Papeete gauge. ENSO: NOAA CPC. Cyclones: NOAA IBTrACS. School holidays: Ministère de
l'Éducation nationale (Licence Ouverte 2.0). Coastline © OpenStreetMap contributors.
l'Éducation nationale (Licence Ouverte 2.0). Human pressure: Bouyoucos et al. (2024), JCU
(CC BY-NC 4.0). Coastline © OpenStreetMap contributors.
</p>
</div>
</div>
Expand Down
6 changes: 6 additions & 0 deletions src/routes/about/+page.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -81,6 +81,12 @@
<b>Sun and moon:</b> computed for Mo'orea (17.53° S, 149.83° W, UTC−10) from standard astronomical
formulas.
</li>
<li>
<b>Human pressure:</b> per-nursery fishing effort, direct human impact and coastal
artificialisation from Bouyoucos et al. (2024), Drivers of reef shark neonate abundance in
French Polynesia,
<a href="https://doi.org/10.25903/2SFB-QX79">James Cook University</a> (CC BY-NC 4.0).
</li>
<li><b>Coastline:</b> © OpenStreetMap contributors (ODbL), simplified.</li>
<li>
<b>Interactive maps:</b> street tiles © OpenStreetMap contributors; satellite imagery © Esri,
Expand Down
2 changes: 1 addition & 1 deletion src/routes/nurseries/[site]/+page.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -77,7 +77,7 @@
</ChartCard>
<ChartCard
title="Human pressure"
subtitle="Bouyoucos et al. (2024), values digitised from Fig. 4"
subtitle="Bouyoucos et al. (2024), scores from the paper's dataset"
>
{#if s.fishingEffort != null}
<table class="data">
Expand Down
8 changes: 5 additions & 3 deletions src/routes/pressure/+page.svelte
Original file line number Diff line number Diff line change
Expand Up @@ -41,9 +41,11 @@
<p class="lede">
Ten nurseries were scored for three human pressures by Bouyoucos et al. (2024, <i
>Marine and Freshwater Research</i
> 75, MF24080): relative fin-fish fishing effort (a proxy for bycatch, not shark fishing, banned
since 2012), direct human impact (people moving around homes and tourism sites) and coastal artificialisation
(walls and embankments). Values here were digitised from the paper's Figure 4.
>
75, MF24080): relative fin-fish fishing effort (a proxy for bycatch, not shark fishing, banned since
2012), direct human impact (people moving around homes and tourism sites) and coastal artificialisation
(walls and embankments). The scores are the paper's own, from its
<a href="https://doi.org/10.25903/2SFB-QX79">dataset</a> at James Cook University (CC BY-NC 4.0).
</p>
<p class="callout small">
The paper found <b>no significant effect</b> of any of these pressures on newborn abundance. With
Expand Down
Loading