Conda
Conda channel: per-platform repodata.json, package download, upload and delete; also as a proxy of an upstream channel (conda-forge) and as a group.
| Repo URL | HOST/repos/REPO/<platform>/repodata.json (channel HOST/repos/REPO) |
| Repository types | hosted, proxy, group. See hosted, proxy, group |
Client setup
internal/clientsetup produces:
conda config --add channels HOST/repos/REPO
curl -u <USERNAME>:<TOKEN> -H "X-Filename: my-package-1.0-0.conda" --data-binary @my-package-1.0-0.conda HOST/repos/REPO/linux-64Credentials are Basic with the token as password; see authentication.
Endpoints
Relative to HOST/repos/REPO; <platform> is any path segment such as linux-64, osx-arm64, noarch.
| Method and path | Purpose |
|---|---|
GET /<platform>/repodata.json | index for that platform |
GET /<platform>/<filename> | package download |
POST /<platform> | upload (body = package) |
DELETE /<platform>/<filename> | delete |
Upload
- Filename comes from the
filenamequery parameter or theX-Filenameheader (400 filename requiredif neither). - The file name must be
<name>-<version>-<build>.tar.bz2or<name>-<version>-<build>.conda. It is split from the right: the build is the part after the last-, the version the part before that, and the name everything before, so names containing-(scikit-learn-1.0-py39_0.tar.bz2) work. Earlier releases split at the first two hyphens and stored such a package as namescikit, versionlearn; migration019re-splits those rows. Any other extension, fewer than three parts, an empty part, a path separator or a part longer than its column (name 255, version 100, build 255) is400 invalid filename format. - The
.tar.bz2and the.condaof one name, version, build and platform are separate files and can both be uploaded. Uploading a file that already exists is409 package file already existsand changes nothing; delete it first to replace it. - Body limit 500 MiB. sha256 and md5 are computed; success is
201with{id, name, version, sha256}. - The storage quota is enforced on upload (
507over quota) and released when the file is deleted for good. info/index.jsonis read from the package (frominfo-*.tar.zstinside a.conda) for the license policy and for the repodata fields below. Reading is bounded: at most 256 MiB of the decompressed archive and 100000 archive entries are scanned, and a package whoseinfo/index.jsoncannot be found or parsed within that is accepted with emptydepends, build number0and no timestamp.
repodata.json
info.subdir is the platform. .tar.bz2 files are listed under packages and .conda files under packages.conda, keyed by file name (<name>-<version>-<build>.tar.bz2 or .conda). Each entry has:
| Field | Source |
|---|---|
name, version, build | the file name |
build_number | info/index.json (0 when absent) |
depends | info/index.json ([] when absent) |
subdir | the platform the file was uploaded to |
sha256, md5, size | computed on upload |
timestamp | info/index.json, in milliseconds; omitted when the package declares none |
constrains, license, license_family, noarch, arch, platform, features, track_features | info/index.json, copied as they are when present |
Packages stored before these fields were kept have no info/index.json data (empty depends, build number 0, no md5 or timestamp). They are all listed as .tar.bz2, including files that were uploaded with a .conda name (the extension was not recorded then), until they are deleted and uploaded again.
Proxy and group repositories
A conda proxy takes a channel URL as its upstream (https://conda.anaconda.org/conda-forge; the URL is required, there is no default). Clients use the proxy or group exactly like a hosted channel, with conda, mamba, micromamba or pixi:
conda create -n demo --override-channels -c HOST/repos/REPO tzdata
micromamba create -n demo --override-channels -c HOST/repos/REPO tzdata
pixi init --channel HOST/repos/REPO && pixi add tzdataA repository that is not public needs credentials: put them in the channel URL as http://USERNAME:TOKEN@HOST/repos/REPO (Basic authentication with the token as password, see authentication); conda, mamba, micromamba and pixi all accept it, so mind where the URL is stored (.condarc, pixi.toml). Proxy and group repositories are read-only: uploads and deletes are refused with 405.
What a proxy serves
Only these paths (relative to HOST/repos/REPO) are answered; everything else is 404 and clients fall back to what is served:
| Path | Notes |
|---|---|
<subdir>/repodata.json, .json.zst, .json.bz2 | the upstream file, byte for byte |
<subdir>/current_repodata.json, .json.zst, .json.bz2 | the upstream file, byte for byte |
<subdir>/<name>-<version>-<build>.conda / .tar.bz2 | package, verified (below) |
channeldata.json | the upstream file, byte for byte |
repodata.jlap (incremental updates) and sharded repodata (repodata_shards.msgpack.zst) are not served: conda, mamba and pixi fall back to the whole repodata.json.zst, which is also the cheapest file to move (conda-forge linux-64 is 59 MB compressed against 460 MB plain). Repodata is not rewritten: package URLs in it are relative to the channel, so downloads resolve through the proxy.
- Caching. A repodata or channeldata file is cached for the repository's
cache_ttl_secondsand served from the cache until it expires, then downloaded again whole (there is no conditional revalidation, so set a TTL that fits how fresh the channel has to be; conda clients revalidate against the proxy'sETagon their own schedule). When the upstream is down a cached copy is served for up tomax_stale_seconds. Unlisted names are remembered fornegative_cache_ttl_seconds. - Large indexes. Index files are streamed to a temporary file while they are downloaded, validated with a streaming JSON scan (no document is ever held in memory), and then served from storage. The limit is 2 GiB, as downloaded and as decompressed (a compression bomb is refused). An index that is not a JSON object with
packagesorpackages.conda, or that carriesinfo.base_url(which would send clients to download from another host), is refused with502and not cached. - Package verification. A package is only served if the upstream's
repodata.json.zst(elserepodata.json) lists it, and only if the bytes downloaded match the listedsha256. An entry withoutsha256is checked againstmd5andsize. A mismatch is502and nothing is cached; a file the repodata does not list is404. A package whose entry has no digest at all is cached for the repository TTL only, not as an immutable file. Verified packages are cached immutably. - Other behavior that every proxy shares (credentials sent only to the upstream origin, quarantine and scanning, RBAC, size limits, never exposing upstream error text) is described under proxy repositories.
What a group serves
A group of conda repositories (hosted and proxy members) serves:
| Path | Behavior |
|---|---|
<subdir>/repodata.json, .json.zst, current_repodata.json, .json.zst | merged across the readable members |
<subdir>/<file> | the first member that has the package, after hosted wins |
repodata.json.bz2, repodata.jlap, sharded repodata and channeldata.json are 404 for a group.
- Merge. Each member's
packagesandpackages.condaare united by file name, the earliest member winning a name two members list; the result hasinfo.subdirandrepodata_version: 1(removedis dropped). Proxy members are read through their.zst, falling back to the plain file; a hosted member renders itsrepodata.json.current_repodata.jsonof a group is the merge of the proxies'current_repodata.jsonand the hosted members' wholerepodata.json. - Hosted wins. Unless the group has
merge_proxy_versions, a proxy member's entries are left out of the merged repodata when a hosted member holds a package of the same name (compared case-insensitively, in any platform: an internalnoarchpackage hides a publiclinux-64package of that name). The same rule decides package downloads: a proxy's file for a name a hosted member holds is404. If a hosted member's names cannot be read, the request fails with502rather than serve what should be hidden. - Validators. The group's
ETagis derived from the validators the members answer for their repodata (a conditional request to each, which a hosted member and a proxy answer without sending the file) and from the set of hosted names, so a revalidation that nothing changed is answered304in milliseconds without reading or merging anything. A member that failed, or that gives no validator, makes the response partial: it is served withCache-Control: no-storeand noETag(the same holds when a member's repodata turns out unreadable while merging; that member's packages are missing and the others are served). With no member contributing the answer is404, or502when a member failed. - Cost. The merged document is built on every full request, streamed through temporary files (
TMPDIR), so disk space of about the size of the plain merged document (about 460 MB for conda-forgelinux-64) is used for the duration of a request, at most two merges run at a time, and memory stays flat. The.zstresponse is compressed on the fly at the default level, so it is about a quarter larger than the upstream's file. On a laptop with conda-forge as the upstream, a coldlinux-64merge takes about 5 s and the process peaks at about 175 MB.