From 699855c4cf30f4bb6c07dbfae9a0ace6e0292590 Mon Sep 17 00:00:00 2001 From: Jon Niehof Date: Tue, 12 Mar 2024 09:42:24 -0400 Subject: [PATCH 1/8] Instructions for making a release of an approved PHEP --- RELEASE.md | 94 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 RELEASE.md diff --git a/RELEASE.md b/RELEASE.md new file mode 100644 index 0000000..b6f1a90 --- /dev/null +++ b/RELEASE.md @@ -0,0 +1,94 @@ +# Releasing PyHC Enhancement Proposals + +This document describes the technical, bits-and-bytes steps necessary +to upload PHEPs once they have been accepted by a vote. It does not +set a standard for PyHC, but documents implementation of a part of +that policy. As such it can be updated in light of new technical +realities without requiring an updated standard. + +Unless otherwise noted, these steps can be carried out by the PHEP author, PHEP editors, or PyHC leadership. + +## Reserve a DOI on Zenodo + * Click the "+" sign on Zenodo or go to the [new upload page](https://zenodo.org/uploads/new). + * At the top of the page, click the "Select a community" button. Search for "heliophysics" and find the "Python in Heliophysics Community" (with our nifty logo); click "Select". + * Under "Basic information", "Do you already have a DOI for this upload?", click "No". Click the "Get a DOI now!" button that appears after clicking No. + * Click the "Save draft" button on the right towards the top, we'll come back to this. + +Note: Zenodo assigns [two DOIs](https://zenodo.org/help/versioning) on first upload, what we want to have on our citations is the version-specific one. When you reserve a DOI, the version-specific one is shown, but this can be verified by saving the draft record and previewing it. It looks like, in order to keep them connected, future revisions of a PHEP should not be uploaded as new records, but as updated versions of files on the existing record. This process is not entirely clear without going through it, and this document should be updated as we figure it out. + +## Final git commit + * Update the contents of the PHEP as documented in PHEP 1 (Status, current date in Post-History, Revision with date accepted/rejected, updated copyright date, and updated DOI with the DOI reserved above). + * Add the Resolution header to the PHEP with a link to the minutes of the two votes on the PHEP. + * Commit and push to the PR. + * PHEP editors or PyHC leadership merge final commit. History should be maintained, so merge or rebase is fine, squash is not. (Which is preferred can be selected before merging this PR). + +## Creating files and release + * Create a PDF of the PHEP from the markdown of the latest pushed commit; use pandoc: `pandoc -f commonmark phep-0001.md -V geometry:letterpaper,margin=1in -o phep-0001.pdf`. (This is in the `pandoc` package in Ubuntu and uses `rsvg-convert` from package `librsvg2-bin`.) Pandoc converts via LaTeX, which means if code blocks overflow the line length they will not be wrapped (a problem with a very long `Post-History` header); use RFC-2822 header folding as necessary. If the Markdown source references other files with relative reference, `pandoc` needs to be run in the same directory. Unfortunately pandoc's LaTeX renderer does not handle internal links well, but other renderers have other problems. + * PHEP editors or PyHC leadership [create a release](https://github.com/heliophysicsPy/standards/releases/new) on GitHub. The release tag is the phep number and revision number, e.g. phep-2-1 for revision 1 of PHEP 2. (Potentially this can be zero-padded). The rendered PDF is the release file. + +## Sending to Zenodo + +Resume the draft record on Zenodo + +### Files + * Upload the rendered PDF. + +### Basic Information + * Resource Type "Publication / Standard". + * Title: PHEP number plus its title, e.g. "PHEP 1: PHEP Purpose and Guidelines". + * Publication Date: Date when the release was tagged on GitHub. + * Creators: The author of the PHEP, ideally including their ORCID. + * Description: Leave blank? + * Licenses: CC0 (there is no "public domain" option explicitly). + +### Recommended Information + * Contributors: the PHEP editor(s) who managed the process for this PHEP; consider including other major contributors to the discussion. + * Keywords and subjects: None? + * Languages: English. + * Dates + * Created: same as Created in the PHEP header, i.e. the date the PHEP number was assigned. This will always be the created date of revision 1. + * Accepted: date of the final vote that accepted the PHEP as Final. This will always be the accepted date of revision 1. + * Withdrawn: date of a PHEP's withdrawal, a final vote where it was Rejected, or the date of the vote that accepted its replacement. + * Updated: only for revised PHEPs, date that a revision was merged to `main`. + * Version: Revision number. + * Publisher: Leave as Zenodo? + +### Funding +Leave blank? + +### Alternate Identifiers +Leave blank? + +### Related Works + * Is derived from (or Is variant form of?), Identifier is link to GitHub release e.g. https://github.com/heliophysicsPy/standards/releases/tag/phep-2-12, Scheme URL. + * Is derived from (or Is variant form of?), Identifier is link to the Markdown source in the tagged commit, e.g. https://github.com/heliophysicsPy/standards/blob/a4b3f558b9ffa712324e63ff6a83325cc69e367f/pheps/phep-0001.md, Scheme URL. + * For revisions of a single PHEP: Use "Is new version of", scheme DOI, and the Zenodo DOI of the previous revision. On the previous revision, add a new related work of "Is previous version of" and put in the new Zenodo DOI (this field should be updatable after the DOI is minted). Type is Publication/Standard for both. + * Related works for PHEPs that replace: Use "Obsoletes", scheme DOI, and the Zenodo DOI of any PHEPs that are replaced; similarly update their records to "Is obsoleted by" and the new PHEP's Zenodo DOI. Type is again Publication/Standard. + +### References +Leave blank? + +### Software + * Repository URL: https://github.com/heliophysicsPy/standards/ . + * Programming language: Markdown. + * Development Status: Active. + +### Publishing Information +Leave blank? + +### Conference +Leave blank? Or put in information on the vote? + +### Domain Specific Fields +Leave blank? + +### Visibility +Public + +### Draft +Save the draft, preview, and publish when ready + +## Updating website + +## Related PHEPs +The process above should be repeated as necessary to update affected PHEPs. For example, if a PHEP is replaced, a new revision needs to be uploaded, including which PHEP replaces it (and all the relevant Zenodo records, as noted). If a new revision of a PHEP is uploaded, the metadata on the Zenodo record of the previous revision need to be updated. \ No newline at end of file From 82d6fc3d16953d38e87ae6e291f76317adadb30d Mon Sep 17 00:00:00 2001 From: Jon Niehof Date: Tue, 12 Mar 2024 11:57:46 -0400 Subject: [PATCH 2/8] phep_to_yaml.py: parse all phep headers to yaml --- phep_to_yaml.py | 50 +++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 phep_to_yaml.py diff --git a/phep_to_yaml.py b/phep_to_yaml.py new file mode 100644 index 0000000..ec86f24 --- /dev/null +++ b/phep_to_yaml.py @@ -0,0 +1,50 @@ +#!/usr/bin/env python3 + +import email +import glob +import os.path +import re +import warnings + +import yaml + + +phep_src = glob.glob(os.path.join( + os.path.dirname(os.path.abspath(__file__)), 'pheps', 'phep-*.md')) +pheps = [] # fully parsed +for ph in phep_src: + if not re.match(r"^phep-\d+\.md$", os.path.basename(ph)): + continue + with open(ph, "rt") as f: + if f.readline().rstrip() != "```": + warnings.warn(f"{ph} preamble has no opening code fence, skipping.") + continue + headers = [] + for l in f: + if l.rstrip() == "```": + break + headers.append(l) + else: + warnings.warn(f"{ph} preamble has no closing code fence, skipping.") + continue + phep = dict(email.message_from_string("".join(headers))) + required = ( + "PHEP", "Title", "Author", "Discussions-To", "Revision", "Status", + "Type", "Content-Type", "Created", "Post-History", + ) + missing = [k for k in required if k not in phep] + if missing: + warnings.warn(f"{ph} missing headers {', '.join(missing)}, skipping.") + continue + optional = ("Requires", "Replaces", "Replaced-By", "Resolution", ) + bad = [k for k in phep if k not in required + optional] + if bad: + warnings.warn( + f"{ph} has unexpected headers {', '.join(bad)}, skipping.") + continue + for k in ('PHEP', 'Revision'): + phep[k] = int(phep[k]) + phep["Filename"] = os.path.basename(ph) + pheps.append(phep) +with open("pheps.yml", "wt") as f: + f.write(yaml.dump(pheps)) From 8af588a95038bad3ee0b59d4b8fce6fc16da44b5 Mon Sep 17 00:00:00 2001 From: Jon Niehof Date: Wed, 17 Apr 2024 11:24:43 -0400 Subject: [PATCH 3/8] phep_to_yaml.py: try to extract DOI --- phep_to_yaml.py | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/phep_to_yaml.py b/phep_to_yaml.py index ec86f24..6bd894d 100644 --- a/phep_to_yaml.py +++ b/phep_to_yaml.py @@ -27,6 +27,7 @@ else: warnings.warn(f"{ph} preamble has no closing code fence, skipping.") continue + body = f.readlines() phep = dict(email.message_from_string("".join(headers))) required = ( "PHEP", "Title", "Author", "Discussions-To", "Revision", "Status", @@ -45,6 +46,20 @@ for k in ('PHEP', 'Revision'): phep[k] = int(phep[k]) phep["Filename"] = os.path.basename(ph) + idx = next((i for i, l in enumerate(body) + if l.startswith("# Copyright")), None) + if idx is not None: + doi_lines = [l for l in body[idx:] if " doi " in l] + for doi_line in doi_lines: + m = re.search(r"10\.\d{4}(\.\d+)*/[-._;()/:A-Z0-9]+", + doi_line, flags=re.IGNORECASE) + if m: + phep["DOI"] = m.group(0) + break + if "DOI" not in phep: + warnings.warn( + f"Found no DOI in {ph}, must fix output by hand.") + phep["DOI"] = "Unknown" pheps.append(phep) with open("pheps.yml", "wt") as f: f.write(yaml.dump(pheps)) From ab2f4e5f6e4716768a199d2aa4d4a7f062c64c0b Mon Sep 17 00:00:00 2001 From: Jon Niehof Date: Wed, 17 Apr 2024 12:29:06 -0400 Subject: [PATCH 4/8] phep_to_yaml.py: unfold headers --- phep_to_yaml.py | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/phep_to_yaml.py b/phep_to_yaml.py index 6bd894d..232c977 100644 --- a/phep_to_yaml.py +++ b/phep_to_yaml.py @@ -1,6 +1,7 @@ #!/usr/bin/env python3 import email +import email.policy import glob import os.path import re @@ -28,7 +29,8 @@ warnings.warn(f"{ph} preamble has no closing code fence, skipping.") continue body = f.readlines() - phep = dict(email.message_from_string("".join(headers))) + phep = dict(email.message_from_string("".join(headers), + policy=email.policy.EmailPolicy())) required = ( "PHEP", "Title", "Author", "Discussions-To", "Revision", "Status", "Type", "Content-Type", "Created", "Post-History", From 8ac8024fa6b2dd8f96bb5fb9dddb49b9330ee89a Mon Sep 17 00:00:00 2001 From: Shawn Polson Date: Wed, 17 Apr 2024 13:54:50 -0600 Subject: [PATCH 5/8] Add updating website instructions --- RELEASE.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index b6f1a90..d8c6d73 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -89,6 +89,14 @@ Public Save the draft, preview, and publish when ready ## Updating website +Follow these steps to update PHEPs on the PyHC website: +- Clone the website repo: https://github.com/heliophysicsPy/heliophysicsPy.github.io +- Create a new feature branch for your changes +- Clone this [`standards`](https://github.com/heliophysicsPy/heliophysicsPy.github.io) repo where PHEPs are kept +- Run the `phep_to_yaml.py` script to get a new `pheps.yml` file (which will contain entries for every PHEP) + - If a PHEP has a DOI but no corresponding `DOI` yaml field, manually add a `DOI` field with its DOI in the format `10.XXXX/YYYYYYYY` (`phep_to_yaml.py` prints a warning when this is needed) +- In the website repo, place your new `pheps.yml` file inside the `_data/` folder (replacing the old one) +- Open a PR to merge your `pheps.yml` changes ## Related PHEPs -The process above should be repeated as necessary to update affected PHEPs. For example, if a PHEP is replaced, a new revision needs to be uploaded, including which PHEP replaces it (and all the relevant Zenodo records, as noted). If a new revision of a PHEP is uploaded, the metadata on the Zenodo record of the previous revision need to be updated. \ No newline at end of file +The process above should be repeated as necessary to update affected PHEPs. For example, if a PHEP is replaced, a new revision needs to be uploaded, including which PHEP replaces it (and all the relevant Zenodo records, as noted). If a new revision of a PHEP is uploaded, the metadata on the Zenodo record of the previous revision need to be updated. From 66182e81c92ec9a7b51504f860db4b9d87970704 Mon Sep 17 00:00:00 2001 From: Shawn Polson Date: Wed, 17 Apr 2024 13:58:42 -0600 Subject: [PATCH 6/8] Update website instructions --- RELEASE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/RELEASE.md b/RELEASE.md index d8c6d73..881b13c 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -94,7 +94,7 @@ Follow these steps to update PHEPs on the PyHC website: - Create a new feature branch for your changes - Clone this [`standards`](https://github.com/heliophysicsPy/heliophysicsPy.github.io) repo where PHEPs are kept - Run the `phep_to_yaml.py` script to get a new `pheps.yml` file (which will contain entries for every PHEP) - - If a PHEP has a DOI but no corresponding `DOI` yaml field, manually add a `DOI` field with its DOI in the format `10.XXXX/YYYYYYYY` (`phep_to_yaml.py` prints a warning when this is needed) + - If a PHEP has a DOI but no corresponding `DOI` yaml field, manually add it in the format `10.XXXX/YYYYYYYY` (`phep_to_yaml.py` prints a warning when this is needed) - In the website repo, place your new `pheps.yml` file inside the `_data/` folder (replacing the old one) - Open a PR to merge your `pheps.yml` changes From 7ea424dc965e41739ce1e1fec75487b7329d7eed Mon Sep 17 00:00:00 2001 From: Jon Niehof Date: Mon, 20 May 2024 16:30:39 -0400 Subject: [PATCH 7/8] phep_to_yaml.py: fix for apparent changes in email parser * Explicitly cast all headers to string --- phep_to_yaml.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/phep_to_yaml.py b/phep_to_yaml.py index 232c977..0dda2cb 100644 --- a/phep_to_yaml.py +++ b/phep_to_yaml.py @@ -29,8 +29,9 @@ warnings.warn(f"{ph} preamble has no closing code fence, skipping.") continue body = f.readlines() - phep = dict(email.message_from_string("".join(headers), - policy=email.policy.EmailPolicy())) + m = email.message_from_string("".join(headers), + policy=email.policy.EmailPolicy()) + phep = {k: str(v) for k, v in m.items()} required = ( "PHEP", "Title", "Author", "Discussions-To", "Revision", "Status", "Type", "Content-Type", "Created", "Post-History", From e489c98c623a67b8d86ebedf0814ea30e33119ec Mon Sep 17 00:00:00 2001 From: Jon Niehof Date: Mon, 20 May 2024 16:41:14 -0400 Subject: [PATCH 8/8] RELEASE: Resolve some open ?s, include source in Zenodo upload. --- RELEASE.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/RELEASE.md b/RELEASE.md index 881b13c..f211246 100644 --- a/RELEASE.md +++ b/RELEASE.md @@ -20,7 +20,7 @@ Note: Zenodo assigns [two DOIs](https://zenodo.org/help/versioning) on first upl * Update the contents of the PHEP as documented in PHEP 1 (Status, current date in Post-History, Revision with date accepted/rejected, updated copyright date, and updated DOI with the DOI reserved above). * Add the Resolution header to the PHEP with a link to the minutes of the two votes on the PHEP. * Commit and push to the PR. - * PHEP editors or PyHC leadership merge final commit. History should be maintained, so merge or rebase is fine, squash is not. (Which is preferred can be selected before merging this PR). + * PHEP editors or PyHC leadership merge final commit. History should be maintained, so merge or rebase is fine, squash is not. We have been using merge. ## Creating files and release * Create a PDF of the PHEP from the markdown of the latest pushed commit; use pandoc: `pandoc -f commonmark phep-0001.md -V geometry:letterpaper,margin=1in -o phep-0001.pdf`. (This is in the `pandoc` package in Ubuntu and uses `rsvg-convert` from package `librsvg2-bin`.) Pandoc converts via LaTeX, which means if code blocks overflow the line length they will not be wrapped (a problem with a very long `Post-History` header); use RFC-2822 header folding as necessary. If the Markdown source references other files with relative reference, `pandoc` needs to be run in the same directory. Unfortunately pandoc's LaTeX renderer does not handle internal links well, but other renderers have other problems. @@ -31,19 +31,19 @@ Note: Zenodo assigns [two DOIs](https://zenodo.org/help/versioning) on first upl Resume the draft record on Zenodo ### Files - * Upload the rendered PDF. + * Upload the rendered PDF, the Markdown source, and any other sources (e.g. images). ### Basic Information * Resource Type "Publication / Standard". * Title: PHEP number plus its title, e.g. "PHEP 1: PHEP Purpose and Guidelines". * Publication Date: Date when the release was tagged on GitHub. * Creators: The author of the PHEP, ideally including their ORCID. - * Description: Leave blank? + * Description: Leave blank. * Licenses: CC0 (there is no "public domain" option explicitly). ### Recommended Information * Contributors: the PHEP editor(s) who managed the process for this PHEP; consider including other major contributors to the discussion. - * Keywords and subjects: None? + * Keywords and subjects: None. * Languages: English. * Dates * Created: same as Created in the PHEP header, i.e. the date the PHEP number was assigned. This will always be the created date of revision 1. @@ -51,22 +51,22 @@ Resume the draft record on Zenodo * Withdrawn: date of a PHEP's withdrawal, a final vote where it was Rejected, or the date of the vote that accepted its replacement. * Updated: only for revised PHEPs, date that a revision was merged to `main`. * Version: Revision number. - * Publisher: Leave as Zenodo? + * Publisher: Leave as Zenodo. ### Funding -Leave blank? +Leave blank. ### Alternate Identifiers -Leave blank? +Leave blank. ### Related Works - * Is derived from (or Is variant form of?), Identifier is link to GitHub release e.g. https://github.com/heliophysicsPy/standards/releases/tag/phep-2-12, Scheme URL. - * Is derived from (or Is variant form of?), Identifier is link to the Markdown source in the tagged commit, e.g. https://github.com/heliophysicsPy/standards/blob/a4b3f558b9ffa712324e63ff6a83325cc69e367f/pheps/phep-0001.md, Scheme URL. + * Is derived from: Identifier is link to GitHub release e.g. https://github.com/heliophysicsPy/standards/releases/tag/phep-2-12, Scheme URL. + * Is derived from: Identifier is link to the Markdown source in the tagged commit, e.g. https://github.com/heliophysicsPy/standards/blob/a4b3f558b9ffa712324e63ff6a83325cc69e367f/pheps/phep-0001.md, Scheme URL. * For revisions of a single PHEP: Use "Is new version of", scheme DOI, and the Zenodo DOI of the previous revision. On the previous revision, add a new related work of "Is previous version of" and put in the new Zenodo DOI (this field should be updatable after the DOI is minted). Type is Publication/Standard for both. * Related works for PHEPs that replace: Use "Obsoletes", scheme DOI, and the Zenodo DOI of any PHEPs that are replaced; similarly update their records to "Is obsoleted by" and the new PHEP's Zenodo DOI. Type is again Publication/Standard. ### References -Leave blank? +Leave blank. ### Software * Repository URL: https://github.com/heliophysicsPy/standards/ . @@ -74,13 +74,13 @@ Leave blank? * Development Status: Active. ### Publishing Information -Leave blank? +Leave blank. ### Conference -Leave blank? Or put in information on the vote? +Leave blank. ### Domain Specific Fields -Leave blank? +Leave blank. ### Visibility Public