Python Dependency Provenance - Guide to Attestations, Rebuilds, and Verifying Python Artifacts

Adding a dependency is a trust decision. By using any dependency, enterprise teams put up their own systems as collateral. In case of Python, this trust decision has a wider range of things. For instance, the artifacts span from a compiled package on PyPI upto model weights fetched from a hosting platform at runtime.

Python has made notably real progress to help the decision. Attestations were finalised on PyPI in late 2024 and it significantly helped organizations. It cryptographically allows any consumer verify who published a release file and from which repository - without trusting the index itself.

Knowing who published an artifact, however, is not the same as knowing its contents match their source, and neither is the same as verifying an artifact that has no source to rebuild from. Since AI-assisted development accelerates the pace of shipping and adoption, these distinctions help differentiate between a valid signature and a trustworthy dependency. 

This guide clarifies and works through the following questions-

1. Who published it?  (provenance)
2. Whether its contents match the source?  (verification by rebuilding)
3. Whether it can be verified at all?  (especially when AI agents are part of the development lifecycle)

Key takeaways

A PyPI attestation provides strong provenance for a package’s publisher. However, provenance is just one layer of trust, not the whole point of trust. 

PEP 740 is the 2024 standard that added attestations to PyPI. Hence, this standard answers “who published this file, and from which repository?” It must not be mistaken to convey file’s contents matches repository’s source. PEP 740 says nothing at all about the AI models a modern Python service loads at runtime.

A secure Python supply chain means moving past “the signature is valid, so this is handled” toward evidence that explains, at the right depth for each kind of artifact, why the software should be trusted.

1. What PyPI Verifies - The Publisher, Not the Package

Let’s start with the problem statement - Before attestations, an attacker who stole a maintainer’s PyPI upload token could publish any malicious release, indistinguishable from a legitimate one. It used the same project name, the same version scheme, and the same packaging as the real thing. There was no way to tell the fake from the real, because nothing in the file’s metadata said which token uploaded it. 

An obvious solution to this problem could be to sign each release. PyPI tried doing exactly that with the PGP Signatures historically. The drawback in PGP signing was that it binds an artifact to a key pair – and the key pair often gets lost, stolen, expire, and go unrotated. Adoption was poor, verification rarer still, and the signatures that did exist frequently couldn’t be checked against anything meaningful.

PEP 740 takes a different approach- mainly planned & built around the PGP failure itself. Attestations are signed by an identity, not a key pair, using Sigstore and OIDC. The signing key is short-lived and bound to a verifiable identity — typically a specific GitHub Actions workflow in a specific repository. There is no long-lived secret for anyone to steal, and the identity is the thing being attested to.

The mechanism fairly exists in two forms. A publish attestation proves a distribution was uploaded through a specific Trusted Publisher rather than by some other mechanism, such as a locally held API token. SLSA provenance attests to the artifact’s original source location. The binding is done properly: the attestation uses in-toto Statements to tie the distribution’s contents — by SHA-256 digest — to its filename, closing the gap where a signature over a name could be replayed against different content.

The Bottom Line Note- A valid attestation tells you one thing clearly: this exact file — identified by its cryptographic fingerprint, not just its name — was built and uploaded by a specific workflow in a specific repository. Not "someone with the right token," but a named, verifiable source. Before 2024, you simply could not ask who uploaded a file; there was no answer to give. This is also where Python quietly leads. Go, often treated as the more careful ecosystem, has no equivalent — it cannot tell you who published a module at all.

The Limitation of Publisher Provenance 

Provenance proves who published the file. It says nothing about what's inside it - and the reason is that a Python wheel (the pre-built .whl file pip installs) is a built artifact. It’s built from the source tree by a separate build step. That step runs setup.py or a PEP 517 backend, and can compile extensions, bundle data files, and even execute arbitrary code.

The attestation ties the finished wheel's fingerprint to a publisher and a repository. It never analyses deeper into repository’s source or whether the wheel was faithfully built from it. Hence, attestation would still pass if the source and the wheel differs each other. This is because it verifies the publisher, not it’s package.

Three questions matter for any dependency, and it's worth being exact about which one an attestation answers:

2. Attestation can still sign Malicious Code 

The gap between provenance and verification is easiest to see in a real failure. The most common assumption about provenance is that a package from the legitimate repo carrying a valid attestation would have been verified. 

It certainly is not the case. Provenance is built to prove who published an artifact — not to judge whether what they published was safe. So a package can pass every check and still be hostile.

Two consequences follow from that boundary. Here’s a simple explanation how it’s possible?

Consequence 1 - The Pipeline Signs Whatever It Builds

A compromised build pipeline produces perfectly valid attestations. Since, the python wheel is built by running arbitrary code, the attacker can get into that step, they would also get into the wheel. Attackers might intrude via-

1. A compromised GitHub Action, 
2. a poisoned build dependency, or 
3. a malicious pull request that reaches a privileged part of the workflow. 

After the attacker somehow gets in, the packaged build output (i.e. wheel) is signed by the real Trusted Publisher, from the real repository. Therefore, attestation will be correct in every detail however, the contents will be hostile. 

Consequence 2 - The Wheel Needn't Match the Source

The source and the built file can disagree, and nothing would catch it. Most projects publish two things: 

1. the sdist - which is the source distribution
2. the wheel - the standard built binary package format which is the built output. 

Both the sdist and the wheel get their own separate attestations.But nothing in PEP 740 checks that the wheel was actually built from that sdist, or that either one matches the actual code on GitHub. Therefore, you can review the source, trust it, and still install a wheel that carries something the source never had.

Proof of incidents from real-world 

The clearest proof that consequence one is real came from a package with roughly 60 million downloads. In late 2024, an attacker compromised the build pipeline of ultralytics, a widely used computer-vision library. The entry vector was a template injection in the project’s GitHub Actions: a pull_request_target workflow ran code from a fork PR and let it poison the GitHub Actions cache. The legitimate publish workflow later restored that poisoned cache, built a wheel containing a cryptominer, and uploaded it to PyPI.

Here is the part that matters. As the security researchers who reconstructed the timeline documented, those malicious releases were uploaded with a Trusted Publisher and valid attestations for each distribution, matching the legitimate ultralytics/ultralytics identity. The attestations were correct. The publisher was genuine. The repository was genuine. The wheel was mining cryptocurrency on every machine that installed it.

The same shape recurred through 2026. In one incident, a hijacked maintainer account published a poisoned version of a scientific Python library where the malicious code lived only in the PyPI artifact — the matching version tag never existed in the source repository, so a source-code review would have seen nothing wrong. In another, an attacker used a stolen API token to publish three malicious versions of a Microsoft-associated library directly to PyPI in 35 minutes, bypassing CI entirely; the package page still pointed at the legitimate GitHub repo, lending the compromised versions unearned credibility. Researchers noted that no provenance existed on any release, so there was no cryptographic way to tie the published artifact back to a specific commit.

The pattern and it’s analysis

Attestations raise the cost of the stolen-token attack, which is real progress. But they don't stop the attacker — they push him earlier, into the build pipeline and the maintainer's account, where the output still validates. A green checkmark tells you who published a file, not whether it's safe. Those are different questions, and the difference is not academic.

3. Verifying the Publisher in Practice

So far this has been about what an attestation can and can't prove. The good news is that checking it yourself is straightforward today, and the tooling is decent. Every release file on pypi.org shows a Provenance badge that links back to the workflow that produced it. And if you need to do it programmatically, the Integrity API exposes the signing identity and source workflow for each artifact.

For command-line verification, pypi-attestations does the work:

# install the verifier  python -m pip install pypi-attestations     # verify an artifact against its provenance AND a named repo  pypi-attestations verify pypi \    --repository https://github.com/sigstore/sigstore-python \    pypi:sigstore-3.6.1-py3-none-any.whl 

Explanation- 

Check 1: That downloads the artifact and its provenance from PyPI, 

Check 2: verifies the artifact against the provenance, 

Check 3: checks that the provenance’s signing identity matches the repository you specified. 

That last check is what defeats an attacker who publishes from a repository they control: they cannot produce an attestation naming a repository they don’t own.

IMPORTANT NOTE- 

Don't build the verification yourself. Doing it by hand means working directly with X.509 certificates and transparency logs, which is easy to get wrong — and the PyPI documentation is unusually blunt about this, pointing you instead toward pypi-attestationsor .sigstore-python . Use them.

And if you're on the publishing side, turning attestations on is close to free. Use Trusted Publishing with the PyPA publish action (v1.11.0 or later), and attestations are generated and uploaded for you by default.

Verification is Opt-in

Here's the catch that surprises most teams. Today, pip and uv do not reject unsigned or unattested packages by default — and PEP 740 doesn't ask them to. The standard deliberately stops at providing the evidence; it makes no rule about installers having to check it. So an attestation is something you can record and audit after the fact, not a gate that blocks a bad package at install time.

That ordering is actually correct — a standard has to exist before anyone can enforce it. But it has a consequence worth being blunt about: a team that read about attestations and assumed the problem was "handled" is mistaken. Unless you are explicitly verifying, nothing in your install path is checking anything at all.

Provenance in practice – the checklist

✓  Use --require-hashes with fully pinned requirements, or a lock file that pins hashes.

✓  If you publish, enable Trusted Publishing and let attestations generate by default.

✓  Record attestation status for your dependencies as audit evidence — nothing enforces it at install time yet, so recording it is the point.

✓  Treat pip-install as arbitrary code execution, because sdists can execute during build.

✓  Do not assume the Provenance badge says anything about the contents.

The checkmark tells you who published the wheel — and as Ultralytics showed, that isn't enough. So the obvious next move is to stop trusting the checkmark and check the contents yourself: take the source, build it, and compare the result. That instinct is exactly right. It's also exactly where Python turns out to be harder than almost any other ecosystem.

4. Verifying the Contents from wheel to source

Closing the contents gap means rebuilding. Includes taking the source, building it yourself, and comparing the result against the published wheel. This is a meaningful control in Python specifically, for a reason.

For instance, for Golang, a module is source . There is no compile step in distribution, hence “rebuild from source and compare” is close to a category error. So, the trust gap sits elsewhere. 

Whereas, in Python the wheel is genuinely built, so the gap between source and artifact is real. There comes a need to actually rebuild something in this ecosystem. 

These both languages are near mirror images Python has the publisher answer which Go lacks, but Go sidesteps the build problem Python is stuck with.

Main Problem lying in Python’s Build Environment

Most discussions of reproducible Python builds start with timestamps. Timestamps are the easy part. The hard part is upstream of that, and it is baked into the packaging standards. A pyproject.tomldeclares build requirements as abstract, unpinned dependencies — for example, setuptools>==40.8.0and a Cython lower bound. When pip or build constructs an isolated environment to build a wheel from that sdist, it resolves those fresh, every time, against whatever is currently on PyPI.

The result: you will get different Cython, and hence also different wheel when building the same sdist today and in three months, and you can get a different setuptools, possibly a different Cython, and therefore a different wheel — not because anyone tampered with anything, but because the build environment was never pinned down in the first place. Projects without a pyproject.toml are no better off; the default is the same unpinned problem, just with fewer words describing it.

Build Failure Case Example of gevent 21.1.2

gevent 21.1.2 listed a specific Cython alpha in its build requirements. Built later, the resolver pulled a newer Cython, which rejected gevent's code and failed the build. The fix was to pin Cython to that alpha in a lock file for the build environment. A dependency that can turn a working build into a failing one can equally turn one wheel into a different wheel, with no error to signal it — so reproducibility is impossible until the build environment is a pinned input.

The Smaller Things That Break a Rebuild

Pinning the environment fixes the biggest problem, but it isn't the end of it. Several smaller sources of nondeterminism remain, and while each one is mundane on its own, together they're why a naive rebuild almost never matches. Timestamps are the familiar one: a wheel is really a zip archive, zip entries record a modification time, and SOURCE_DATE_EPOCH is the standard fix for it.

The one that catches people off guard, though, is file permissions. A wheel stores the file mode of everything inside it, and those modes depend on the umask of whatever process built the wheel:

1umask 022 && python -m build --wheel   # one hash 2umask 000 && python -m build --wheel   # a different hash 

Nothing about the software changed. An environment variable nobody thinks of as a build input changed the artifact. there are more differences of the same kind underneath. Who owns each file in the archive, the owner's name recorded alongside it, header fields in the compressed data — all of it can vary between builds. The deepest layer is compiled extensions: anything with C, C++, Rust, or Fortran in it is only as reproducible as that entire toolchain allows. That's the point where a claim like "we verify everything the same way" quietly stops being true.

What It Takes to Rebuild and Match

A qualified but real answer: byte-identical rebuilds are achievable within a pinned environment, for a substantial fraction of packages. The recipe isn't mysterious, and it comes down to four things:

1. Pin the build environment - Use a lock file for your build dependencies, not just your runtime ones.
2. Pin the platform- Reference a container image by its digest, not by a tag.
3. Normalise the archive -Control SOURCE_DATE_EPOCH , umask , file modes, and ownership — either handle it yourself or use a build backend that does it for you by default.
4. Build from the sdist, not the repository- The two differ more often than people expect, and the sdist is what PyPI actually serves as source.

Try it on something in your own dependency tree — the exercise is worth an hour:

1pip download --no-binary :all: --no-deps requests==2.32.3 2SOURCE_DATE_EPOCH=1700000000 umask 022 python -m build --wheel requests-2.32.3/ 3sha256sum dist/*.whl 

This is followed by comparing against the wheel PyPI serves. For a pure-Python package with a well-behaved build, these can match. For anything with compiled components, expect them not to — and the reason will usually be the toolchain rather than anything sinister. What that hour buys you is not a pass/fail result; it is a concrete sense of which of your dependencies are verifiable this way and which are not.

What to Do When Rebuilds Don't Match in Python

For many real dependencies, a byte-for-byte match is off the table. This is true for anything with native extensions, or anything built on a platform you can't reproduce exactly.

When that happens, the easy move is to call the problem unsolvable and fall back to scanning. The better move is to stop trying to prove the bytes are identical, and prove something else instead.

Two more controls round it out. Capability diffing compares what the rebuilt package does against the previous version — network access, filesystem operations, subprocess execution, imports. A version that suddenly opens a socket is interesting regardless of whether its hash is stable. And install-time behaviour deserves stating explicitly, because Python is unusual here: pip install from an sdist executes code. setup.py runs. PEP 517 backends run. Hash pinning protects the integrity of what you downloaded and does nothing about what it does when it builds. Sandboxed observation at install and import is not optional garnish in Python — it covers a live execution path.

What None of This Solves

Upstream compromise. A source-repository backdoor survives every check in this section. The rebuild matches, the tests pass — because the attacker made sure they would — and the attestation validates. Behavioural analysis is the partial answer, and it is partial. Reachability is untouched too: “verified” says nothing about whether your code reaches a vulnerable path. And the publisher gap from Section 1 remains — rebuilding tells you contents match source; it does not tell you the maintainer’s account was not taken over by someone who then legitimately published from the legitimate repository.

The Package-Level Answer, Stated Plainly

The practical answer is not one control but two. Verify the publisher with attestations, and verify the contents by rebuilding and running the tests. Neither is sufficient alone. The first tells you who uploaded the file; the second tells you the file matches its source. Together they cover most of what matters for a Python package — and the load-bearing word in that sentence is “package.

Both controls so far assumed the thing you are consuming is a Python package — something with a source tree, a build, and a wheel you can rebuild and compare. Increasingly, the most important third-party content in a Python system is none of those things. And when the artifact stops being a package, both controls stop applying.

5. The Model Is Not a Package

A modern Python service pulls model weights from Hugging Face, datasets from somewhere else, and — new in the last eighteen months — agent skill files and MCP server definitions that instruct an AI agent how to behave. None of these are packages. None go through PyPI. None carry attestations, and most of the reasoning in the previous sections does not apply to them. This is the least-covered area in supply-chain security, and Python is where it concentrates, because Python is the language AI and ML are written in.

Why this is a Python problem specifically

Python became the top base image in production container estates — around 71.7% adoption in one large survey of container-image projects — substantially because of AI and ML workloads. And the dependency graph of an ML service is not just numpy and pandas. It is a pip install list that pulls a framework, plus a set of runtime artifact fetches that are not dependencies in any sense your tooling recognises:

from transformers import AutoModel  model = AutoModel.from_pretrained("some-org/some-model") 

That second line downloads gigabytes of third-party binary content from the internet at runtime and loads it into your process. It appears in no lock file, no SBOM, and no dependency scan. Your requirements.txt records that you depend on transformers . It does not record that you depend on some-org/some-model — which is the part that will actually determine what your service does.

Rebuilding does not translate at all

Section 4’s entire method rested on one thing: that there is a coherent notion of “build this from source and compare.” For Python wheels, .NET assemblies, and Java jars, that holds — they are built artifacts. For a trained model weight, there is no such notion. You cannot rebuild a model from source in any practical sense. Training is non-deterministic, ruinously expensive, and dependent on data you almost certainly do not have. A 70-billion-parameter checkpoint is not the compiled output of a repository — it is the result of a process nobody is going to repeat to check your work.

The format problem: execution moves to load time

Model artifacts carry a second issue that packages mostly don’t. The traditional PyTorch checkpoint format is Python pickle, and deserialising a pickle is arbitrary code execution — it is not a data format that can be parsed safely, it is a serialised object graph that can invoke anything on load. tourch.load on an untrusted checkpoint is equivalent to executing an untrusted script.

The remediation is real: safetensors exists specifically to provide a format that is just tensors, with no execution semantics, and Hugging Face has pushed adoption hard and scans uploads for unsafe pickles. But the ecosystem is a decade deep in pickle-based checkpoints. Older models, research artifacts, and anything published before the shift are still out there in the original format, and tourch.load will still load them.

Read that against the rest of the guide, and watch a single thread pay off. Python has an install-time execution problem in setup.py, flagged back in Section 3. It has a load-time execution problem in pickle that is arguably worse — because it fires at runtime, on content fetched from the internet, in production, after every control you have already run. The point where untrusted code executes has migrated from build, to install, to the moment your production service loads a model.

What Hugging Face gives you, and what is still missing

To be fair to the platform: Hugging Face has moved faster on this than most package registries did at a comparable stage — malware scanning on uploads, pickle inspection, organisation verification, and gated access for models that require it. What is not there yet is the thing PyPI and npm now have:

• No standard attestation format binding a model file to the training run, the dataset, or an identity — nothing comparable to PEP 740.
• No lock file. A model reference in code is typically a repository name and, optionally, a revision. Most code pins neither.
• No transitive graph. A model fine-tuned from a model derived from a base model has a lineage that exists in prose on a model card, if the author bothered — and nowhere machine-readable.
• No advisory database. There is no GitHub Advisory equivalent for “this checkpoint has been found to be backdoored.”

That last one deserves a pause. Model backdooring is an established research area — a model can be trained to behave normally except on a specific trigger input. No scanner detects that, because there is nothing structurally wrong with the file. The weights are just weights. This is the model-world equivalent of the upstream-compromise problem from Section 4, except there is not even a rebuild to fall back on.

The new artifact classes nobody is governing

Two categories have appeared very recently and are under-governed even by the standards above. Agent skill files are machine-readable instructions that tell an AI agent how to perform a task, including which packages to install — an executable artifact in every sense that matters, distributed with none of the controls a package gets. The concrete failure mode is already visible: a hallucinated package name propagating through machine-generated skill files until it is referenced across hundreds of repositories, ready for an attacker to register. MCP server definitions are the other: a connected MCP server can read data and take actions on a user’s behalf, so its trust question is identical to a dependency’s, and its distribution mechanism is currently far less governed. Both have started appearing on registry roadmaps, but as of today nobody has shipped provenance for them.

What you can actually do about models

✓  Pin model revisions. from_pretrained("org/model", revision="<commit sha>")— a bare name resolves to whatever is current, so production behaviour can change without a deploy.

✓  Prefer safetensors, refuse pickle where you can. use_sagetensors=True where the framework supports it. Treat a pickle-only model as an exception requiring justification, not a default.

✓  Mirror models you depend on. Fetching gigabytes from a third party at container start is an availability problem before it is a security one. Pull once, store internally, reference internally.

✓  Record model artifacts in your SBOM — manually if necessary. Your tooling will not do it, and a regulator asking what is in your product will not accept “the model is not a dependency.”

✓  Isolate model loading. If you must load pickle-format checkpoints, do it in a process with no network access and minimal filesystem reach.

✓  Treat skill files and MCP configs as dependencies. Review them, pin them, put them under the same change control as code. Most teams don’t yet know how many they have.

6. Connecting all the Analysis

Python’s provenance position is better than the ecosystem’s reputation suggests and narrower than the phrase “signed packages” implies. Run the descent back the other way and the whole picture fits on one card.

The through-line is the honest one. At each level the guarantee shrinks, the execution point moves later, and the responsible move is to name the shrinkage out loud rather than average it into a single reassuring score. For a package, the two controls together — verify the publisher, verify the contents — cover most of what matters. For a model, you are left with a weaker toolkit and an obligation to say so.

None of this is a reason to distrust Python’s ecosystem. It is a reason to know exactly what each green checkmark means, and to build the layer your tooling does not give you for free. That is the whole job: not finding problems faster, but establishing — with evidence, at the right depth for each kind of artifact — why the software you ship should be trusted.

7. How CleanStart Approaches Python Verification

CleanStart builds verified dependencies and hardened container images under a single attestation chain. For Python packages, the honest claim is narrower than “we rebuild everything from source,” and more useful because it is reported at the depth each artifact actually supports:

• Pinned-environment rebuilds with the build dependency graph locked, so results are comparable rather than incidental.
• Byte-identity where it is achievable, reported as such.
• Functional equivalence via upstream test execution where byte-identity is not — with the distinction reported, not averaged into a single score.
• Behavioural analysis at install and import, covering the execution path that hash pinning does not.
• Signed attestations in standard formats, verifiable without CleanStart’s tooling.

For AI/ML artifacts, the deliverable is deliberately narrower: provenance recording, format policy, scanning, mirroring, and inventory. That is useful, and it is not the same thing as verification — a distinction that should survive contact with anyone’s marketing department, including CleanStart’s. Coverage is reported as a breakdown, not a single number, because a vendor claiming uniform depth across pure-Python and C-extension packages is glossing over the hardest third of the problem.

Final takeaway

Python’s dependency model demonstrates a principle that generalises past Python: strong provenance is essential, but provenance alone does not establish trust, and no single control covers every kind of artifact.

A secure Python supply chain combines:

• Publisher provenance — attestations proving who published a file and from where.
• Contents verification — rebuilding from pinned source, byte-identical where possible and functionally equivalent where not.
• Behavioural analysis — covering the code that runs at install, and the code that runs at model load.
• Inventory and governance — including the model weights, skill files, and MCP configs your tooling will not record on its own.

The future of software security is not only about finding problems faster. It is about establishing, with evidence at the right depth for each artifact, why the software an organisation ships should be trusted.

Verification builds trust.