You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
nixpkgs-clarity: Create dataset of curated and validated license metadata for Nix packages #754
Hi @pombredanne ,
I'm currently finalizing my GSoC proposal for PurlDB and I would love to tackle this dataset curation to get more familiar with the ingestion pipelines.
My plan is to write a Python pipeline that:
1).Extracts a slice of the top 500 Nix packages.
2).Uses nix eval --json to extract the raw meta.license attributes.
3).Normalizes the output into strict SPDX identifiers.
4).Outputs a curated JSON dataset keyed by purl (Package URL).
Let me know if you have a preferred source for determining the 'most popular' slice of Nixpkgs, otherwise I will compile a standard top-500 list to start the PoC.
Extraction: I used subprocess to call nix eval --json with the new experimental flags to dump the raw meta.license attribute.
Normalization: Nix license formats are highly inconsistent (sometimes dictionaries, sometimes strings, sometimes lists of duplicates like nginx). I wrote a normalization function to parse these variations, extract the exact spdxId, and deduplicate the results.
PURL Standard: The output dataset strictly keys every package using the Package URL standard (e.g., "purl": "pkg:nix/ripgrep").
If this PoC JSON structure looks good to the team, I can scale the POPULAR_PACKAGES list to the top 500/1000 packages and submit the final dataset as a PR. Let me know if you'd like me to format the output differently (e.g., CSV instead of JSON) before I scale it up!
A quick question regarding scale: For this PoC, I hardcoded a static list of 10 packages to isolate and test the normalization logic. When we scale this dataset to the top 500/1000 packages, does AboutCode have a preferred data source for defining Nix "popularity" (e.g., querying Repology's API, or cross-referencing with PyPI/npm download stats?), or should I compile a static seed list for the first major PR?
(Note: I am also referencing this pipeline in my GSoC proposal for the PurlDB PopularCode project as an example of data extraction & normalization).
I created a list of packages that include the PURL field, which can be used as a dataset. Unfortunately, I couldn’t find a way to collect packages based on popularity, as there is no popularity tag or similar indicator available. Therefore, I compiled a list of 3,000 packages, which I believe can still be useful for addressing the issue.
Activity
SahilJat commentedon Mar 29, 2026
Hi @pombredanne ,
I'm currently finalizing my GSoC proposal for PurlDB and I would love to tackle this dataset curation to get more familiar with the ingestion pipelines.
My plan is to write a Python pipeline that:
1).Extracts a slice of the top 500 Nix packages.
2).Uses nix eval --json to extract the raw meta.license attributes.
3).Normalizes the output into strict SPDX identifiers.
4).Outputs a curated JSON dataset keyed by purl (Package URL).
Let me know if you have a preferred source for determining the 'most popular' slice of Nixpkgs, otherwise I will compile a standard top-500 list to start the PoC.
SahilJat commentedon Mar 30, 2026
hii @pombredanne
Update: I have built the PoC pipeline and generated the initial curated dataset for a slice of the top 10 most popular packages.
Link to the Extractor Script & JSON Output: https://gist.github.com/SahilJat/587b626eda9068d9c67876ccdc1d7ec1
Pipeline Architecture:
Extraction: I used
subprocessto callnix eval --jsonwith the new experimental flags to dump the raw meta.license attribute.Normalization: Nix license formats are highly inconsistent (sometimes dictionaries, sometimes strings, sometimes lists of duplicates like nginx). I wrote a normalization function to parse these variations, extract the exact spdxId, and deduplicate the results.
PURL Standard: The output dataset strictly keys every package using the Package URL standard (e.g., "purl": "pkg:nix/ripgrep").
If this PoC JSON structure looks good to the team, I can scale the POPULAR_PACKAGES list to the top 500/1000 packages and submit the final dataset as a PR. Let me know if you'd like me to format the output differently (e.g., CSV instead of JSON) before I scale it up!
A quick question regarding scale: For this PoC, I hardcoded a static list of 10 packages to isolate and test the normalization logic. When we scale this dataset to the top 500/1000 packages, does AboutCode have a preferred data source for defining Nix "popularity" (e.g., querying Repology's API, or cross-referencing with PyPI/npm download stats?), or should I compile a static seed list for the first major PR?
(Note: I am also referencing this pipeline in my GSoC proposal for the PurlDB PopularCode project as an example of data extraction & normalization).
chinyeungli commentedon Sep 1, 2026
I created a list of packages that include the PURL field, which can be used as a dataset. Unfortunately, I couldn’t find a way to collect packages based on popularity, as there is no popularity tag or similar indicator available. Therefore, I compiled a list of 3,000 packages, which I believe can still be useful for addressing the issue.
nix_packages.csv