Published 2023-07-21 by Seth Larson
Reading time: 5 minutes
This critical role would not be possible without funding from the OpenSSF Alpha-Omega Project. Massive thank-you to Alpha-Omega for investing in the security of the Python ecosystem!
Ever since the 3.11.0 release of Python, all Python release tarballs are signed using Sigstore. You can read this page for more information about Sigstore signatures and how to verify the signatures using the Python Sigstore client.
The gist is each Release Manager for a given Python release (ie 3.7, 3.8, etc) has an identity (email address with @python.org)
defined in the Sigstore information page. This is the identity used to verify who signed the release. Sigstore uses OpenID Connect
which means we also need to designate an identity provider (IdP) to verify signatures.
# Currently documented example of how to verify Python signatures
$ python -m sigstore verify identity \
--certificate Python-3.11.0.tgz.crt \
--signature Python-3.11.0.tgz.sig \
--cert-identity pablogsal@python.org \
--cert-oidc-issuer https://accounts.google.com \
Python-3.11.0.tgz
It was reported that the instructions for verifying signatures weren't consistent with the reality of how the artifacts had been signed. I put together a few simple scripts which downloaded and attempted to verify every Python release artifact against its signatures and published the results. These scripts found a few issues:
lukasz@langa.pl, instead of the documented lukasz@python.org.python.org/download due to permission issues.With these findings the release managers took the following steps to make the signature verifications consistent for consumers:
You can see the change in state in this commit to the above dataset. From here all that's needed is fixing Łukasz's identity on signatures which may prove difficult as using GitHub as an IdP only allows the primary email address as an identity rather than all configured email addresses. William Woodruff opened an issue after discussion of this any other GitHub IdP use-cases on Sigstore's Slack channel. I've also opened a PR in Python's release-tool to keep Sigstore signing consistent with the documentation for future Python releases.
All of this highlights how even though Sigstore is much more ergonomic signing experience than GPG, there are still little things that can create an inconsistent experience for consumers. I'll be updating the Sigstore documentation once we've completed this task. Thanks to all the release managers for working on this issue!
I stumbled on the discuss thread for PEP 710 written by Fridolín Pokorný which proposes a new metadata file provenance_url.json
to be placed in an installed Python distribution's .dist-info directory if the package was
installed via an index with resolution (ie python -m pip install urllib3). You can read the PEP for
the exact structure of the document which includes the original download URL and a list of available hashes
for the distribution like SHA-256, etc.
This PEP would allow tools to inspect an already installed Python environment and be able to recreate exact download URLs for installed distributions along with hashes in order to ensure integrity of the distributions. This makes the installed environment more reproducible if the download URLs are still intact.
Having this information available also enables tools to create a Software Bill of Materials (SBOM) from a pre-installed environment. I created an experimental tool that uses the reference implementation of PEP 710 for pip to create SPDX and CycloneDX SBOM documents.
From sharing this thread on Twitter I was also pointed to pip installation reports, a feature I wasn't aware of that's new in pip v22.2. (thanks Stéphane Bidoul!)
This feature lets you hook into the results of pip's dependency resolution for a given CLI input to pip (ie pip install -r requirements.txt). I'll be experimenting with this feature in the future.
#Python environments to #SBOM using draft PEP 710 👀 pic.twitter.com/htt1QEwJLe
— Seth Michael Larson (@sethmlarson) July 19, 2023
After reviewing the PEP draft I made a few suggestions
including upgrading the "SHOULD" to a "MUST" for installing a resolved distribution from the installers' cache and clarifying if there was any
case where either provenance_url.json or PEP 610's direct_url.json wouldn't be installed for a given distribution. If provenance_url.json can
be made required for the inverse case of PEP 610 it would mean that all installed distributions would
have a metadata file describing the distribution's origin.
I also suggested adding another key index_url which would contain the URL of the index that the distribution
was resolved from. This is useful for auditing which indices were used for which packages to arrive at the final download URL.
I am hopeful that the amount of positive interest in this PEP will result in its acceptance. If you're interested in this PEP please participate in the discussion thread.
That's all for this week! 👋 If you're interested in more you can read next week's report or last week's report.
Wow, you made it to the end!
If you're like me, you don't believe social media should be the way to get updates on the cool stuff your friends are up to. Instead, you should either follow my blog with the RSS reader of your choice or via my email newsletter for guaranteed article publication notifications.
If you really enjoyed a piece I would be grateful if you shared with a friend. If you have follow-up thoughts you can send them via email.
Thanks for reading!
— Seth