Derive Clinical Trials source links from each trial's own registration id - #19
Merged
Conversation
`transform_clinical_trials` now computes a `source_link` column immediately after `df = df.copy()`, while `vin_name` and `vin_source` are still present under their raw names. The column is built by calling `build_source_link(name, source)` row-wise: the registration id (`vin_name`) is used to construct an authoritative registry URL, with `vin_source` consulted only as a fallback for unrecognized ids. The gold schema map for `fact_clinical_trial_event` is updated to read `source_text` from `source_link` instead of the raw `vin_source` column, which was unreliable (bulk imports share a single constant URL and there are blank/literal "CT.gov"/"N/A" values).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
On the dashboard's Clinical Trials table, the Source column (
source_text)did not reflect each trial's own registration. It was a raw pass-through of
the Dataverse column
vin_source, which is unreliable: bulk-import batchesshare a single constant URL (e.g. a run of consecutive trials all pointing at
NCT04882514), plus hundreds of blanks and literalCT.gov/N/A/Embasevalues. Across a 1,100-row sample, ~37% of source NCTs did not matchthe trial.
Fix
Derive each trial's source link from its own authoritative registration id,
vin_name, via a registry-aware URL builder, and source the goldfact_clinical_trial_event.source_textfrom it instead ofvin_source.The builder applies an ordered fallback chain:
NCT, ISRCTN, ACTRN, TCTR, DRKS, jRCT, SLCTR (slug), and EudraCT vs. the
newer EU CT number (distinguished by the trailing group).
(
https://trialsearch.who.int/?TrialID=…) for registries whose publicURL uses an internal database id that cannot be derived from the
registration number — ChiCTR, CTRI, IRCT, PACTR, NTR/NL.
vin_sourceURL whenvin_nameis unrecognized (junk suchas
N/A,Unknown, or grant numbers).The URL templates were grounded in the hosts the data already uses (each
registry id checked against the host of its existing
vin_source).Notes
steps, while the raw
vin_name/vin_sourcecolumns still exist; the goldschema map then points
source_textat it.registry id, ~470 keep a
vin_sourceURL fallback, ~140 are blank.WHO-resolvable registries (EUCTR, JPRN/UMIN, KCT, RBR) through ICTRP for
even broader coverage.
The user-visible fix also requires the companion dashboard change
(
akvo/igh-dashboardbranchct-source-link-fix), which corrects thestar-schema fixture and renders the Source cell as a clickable link.