A (Security) Primer for Open Source AI
Over the last few months, generative AI has gone through both technological and commercial shifts. The cost effects of the large frontier labs, the control of high quality intelligence driven by a few players, the increasing value of agent harnesses and the leaps being made in open source flavors of AI have moved the open source conversation from “Should we?” to “How do we?”.
Open-source AI is not a single artifact that a security team can scan once. It is a chain of data, code, weights, configuration, documentation, infrastructure, and human decisions that may be produced and operated by different parties.
Most conversations revolve around open-weight models (Gemma, Kimi, GLM, Deepseek). However, teams need to assess both the integrity of all the components and the authority granted to them when they are assembled and executed.
Open-source AI collapses supply-chain security and delegated-authority security into a single problem, and no existing control stack is organized that way.
Terminology
Open source is a set of freedoms, not a package pulled from a repository
The Open Source AI Definition 1.0 describes an open-source AI system through the freedoms to use, study, modify, and share it, together with access to the preferred form for modification. This is a system-level idea. A downloadable checkpoint alone may not provide the data information, code, or modification inputs needed to study and change the complete system.
For security work, keep these labels separate:
These categories can overlap: a release such as OLMo or Nemotron may provide weights, recipes, and training data. Model-family names are only shorthand because artifacts and licensing can differ between versions. Verify the exact release rather than inheriting the label from another model in the family.
Marketing frequently uses “open source” for systems that are better described as open weights. The threat model should record what is actually available and modifiable instead of relying on the label alone.
The Open Source AI Component Model
A model repository is a trust bundle. Platforms like HuggingFace host models, datasets, and Spaces as versioned repositories, but the files inside them do not all have the same security semantics. Some are descriptive metadata; some shape behavior; some are executable code; some are large opaque numerical artifacts.
The Most Common Ones
The Less Common Ones
A model repository can cross security categories
A single repository may contain:
Markdown documentation and metadata.
JSON configuration and tokenizer files.
Safetensors or other weight files.
Pickle-based files with deserialization risk.
Python code enabled through custom-model loading.
Templates that shape model and tool behavior.
The correct unit of trust is not “the model.” It is the resolved revision of every component that participates in the deployed system, plus the loader and runtime that interpret those components.
Hugging Face and similar platforms document malware, pickle, and secrets scanning for Hub repositories. These controls are valuable distribution-layer signals. They do not establish that a model behaves safely, that custom code has been reviewed, that metadata is truthful, or that a clean file will remain clean at a mutable branch name.
Platform scanning is a signal, not a trust decision
Security Mental Model
Four security disciplines converge
Open-source AI security combines:
Software supply-chain security: contributors, dependencies, builds, packages, signing, distribution, patching, and provenance.
ML integrity: data poisoning, backdoors, model behavior, evaluation quality, drift, and derivative effects.
Content provenance: whether cards, datasets, templates, instructions, and retrieved content are authentic and authorized for their use.
Delegated-authority security: what a model or agent can read, write, execute, spend, disclose, and change.
A control stack that covers only one discipline will miss realistic gaps.
A signed artifact can contain an intentionally backdoored model.
A behaviorally safe model can be loaded by vulnerable code.
A safely loaded model can be given excessive tool authority.
Five properties that shape differentiated risk for open source AI
Decentralized trust
Trust decisions move from one provider to a network of maintainers, contributors, hub operators, packagers, application builders, and runtime owners. This can weaken accountability, but it also enables independent verification.
Composite repositories
One repository identity can represent documentation, metadata, code, configuration, weights, and behavior. Each component requires a different analysis method.
Local execution
Downloadable artifacts can run beside local files, credentials, networks, and accelerators. Provider guardrails and telemetry may no longer exist; host isolation and adopter-owned policy become more important.
Derivative lineage
Adapters, merges, quantizations, fine-tunes, and forks inherit some properties and change others. The base model’s reputation does not automatically transfer to the derivative, and the derivative’s patch does not automatically reach every downstream copy.
Reduced central monitoring
Private inference can improve confidentiality and resilience, but it can also reduce abuse detection, coordinated response, and visibility into vulnerable versions. Monitoring must be deliberately rebuilt at the deployment and organizational layers.
What questions can you ask today?
If you are building an AI governance model and actively using or considering open source AI, start collecting the following answers today:
What exactly is it?
Code, data, weights, metadata, configuration, hosted execution, or a combination. This becomes the most important set of primitives to capture.
Who produced this revision?
What identity, review, and release evidence exists? Provenance becomes key when intelligence enters your workflow
What changed?
How does this revision or derivative differ from the trusted baseline? Most supply chains attacks start here
What will interpret it?
Which loader, library, runtime, or application execute the various parts of the model?
What authority will it receive?
Files, secrets, network, tools, users, compute, and production effects. This becomes the second-most set of important primitives to capture
What evidence will remain?
Hashes, commits, scanner results, logs, policy decisions, evaluations, and cleanup records
Who responds?
How will owners, downstream users, and operators coordinate a fix or revocation?