Crystal shards and wol shard

What it is

A shard is a Crystal library. WolOS keeps one signed list of the shards it has read, each pinned to a full commit, and wol shard adds, updates and checks them in a project.

Why

A version range or a tag can move, so the same project could build different code on two days. A pinned list and a lock file mean a build reads only what is already on the computer, and gets the same code every time.

How

1. Find one: wol shard list [words].
2. Add it: wol shard add <shard> in the project folder. You see shard.yml gain the shard with its git: and commit:, and a shard.lock next to it.
3. Check: wol shard check before you commit; it exits 1 when something is wrong.
4. Move pins: wol shard update when the list vouches for newer commits.

If something goes wrong

Related

Cryswol · Crystal rules · wopkg

Reference

wopkg_shard_index, wopkg_shard_pin, wopkg_shard_check in crates/wopkg/src/ffi.rs (include/wopkg.h, node WOPKG_1.1), wopkg/src/wopkg/shard.cr (wopkg shard), crates/controller/src/shard.rs (wol shard). Rules: RULES-crystal.md RCR-040 to RCR-044.

Why

A Crystal project names its dependencies in shard.yml, and shards install fetches them from git. A version range or a tag can move, so the same project can build different code on two days (RCR-040). WolOS answers with one signed list of shards it has read, each pinned to one full commit, and with tools that write the pins and the lock for you. A build then runs shards install --frozen --local and reads only what is already on the machine (RCR-044).

Commands

dir is the project folder (default: the current one). --local on add and update skips the cache fetch (the files are still written); on check it skips the list and checks the files only. --json prints libwopkg's documents unchanged. Off a terminal (a pipe, a script, wol mcp) wol shard asks for that JSON itself, captures it and prints it, and the exit status still comes back (1 when check found something), so a helper gets the findings and the failure together. wol shard runs wopkg shard with the same words: nothing needs the administrator, and only the project's two files and the person's shards cache change.

After add:

dependencies:
  wolkit:
    git: https://git.wolos.org/shards/wolkit.git
    commit: 0123456789abcdef0123456789abcdef01234567

and shard.lock (the format shards 0.19 reads):

version: 2.0
shards:
  wolkit:
    git: https://git.wolos.org/shards/wolkit.git
    version: 0.3.0+git.commit.0123456789abcdef0123456789abcdef01234567

Other keys of the dependency (version:, branch:, tag:, github:) are dropped, and the rest of shard.yml stays as it was. A dependency the list does not carry makes add and update refuse: the lock has to name the whole tree, and the tree is curated.

The offline install

shards install --frozen --local installs exactly the lock and reads only its cache: one git mirror per shard under $SHARDS_CACHE_PATH, else $XDG_CACHE_HOME/shards, else ~/.cache/shards, at <host>/<path>.git. add and update make that mirror (git clone --mirror, then git fetch) and check that it holds the pinned commit; a commit no branch carries any more is fetched by its hash and kept under refs/wolos/pinned/<commit> so git does not prune it. When a commit cannot be had, the files are written and the command fails with the shard's name, so the next online wol shard update finishes the job.

A package build (PKGBUILD) does not fetch at all: the shard arrives in source=(...) with a real checksum and is built with shards install --frozen --local (RCR-044).

The list

https://shards.wolos.org/index.json and index.json.sig, published from the wolos-shards repository (layout and generator: wolos-shards/README.md).

{"format":1,"serial":7,"expires":1767225600,"shards":[
  {"name":"wolkit","git":"https://git.wolos.org/shards/wolkit.git","version":"0.3.0",
   "commit":"<40 hex>","license":"BSD-2-Clause","summary":"Reactive UI on libWolkit",
   "crystal":">= 1.22.0","dependencies":["cryswol"]}]}

An entry that breaks a rule is dropped on its own, as is a second entry with the same name.

Trust, in order (the same code paths as the plugin list, index/plugins.rs):

1. The Ed25519 signature over the exact bytes is checked before anything is parsed. The keys are *.pub files in /usr/share/wolos/keys/shards/ (with its own revoked.list). They are kept apart from the package and release keys in /usr/share/wolos/keys/: a key read for one purpose never vouches for another.
2. A list with a lower serial than the cached one is refused (replay), and so is a list past its expires.
3. The cache (~/.cache/wopkg/shard-index/) is verified again on every read, and replaced only by a list that passed 1 and 2 (temp file, then rename).

How the key is made and rotated

The shard list has its own Ed25519 key, a trust root apart from the package and release keys.

• Made in the publish pipeline, on the release owner's offline machine, never on a CI runner: openssl genpkey -algorithm ed25519 -out shards.key.pem. The private key stays offline; it signs index.json with crystal run tools/gen_index.cr -- sign shards.key.pem in wolos-shards (openssl pkeyutl -sign -rawin, 64 raw bytes in index.json.sig).
• Shipped as openssl pkey -in shards.key.pem -pubout > shards-<year>.pub (PEM, SubjectPublicKeyInfo) in /usr/share/wolos/keys/shards/ of the system image. Every *.pub there is a trusted shard key; nothing else is.
• Rotated in two releases: the first ships the new .pub beside the old one and the list is signed with the new key from then on; a later release removes the old file.
• Withdrawn at once by a line in /usr/share/wolos/keys/shards/revoked.list (the same format as the package keys' list: 64 hex characters of the key, a space, the reason); a revoked key verifies nothing, even while its file is still there.

A list signed by a package or release key is refused, and the shard key never verifies a package, a release or the plugin list.

list, add and check use the cached list while it is valid and download it when there is none; update always downloads, and falls back to a valid cache when the download fails.

C entry points (libwopkg 1.1)

All three take a JSON request and return a JSON document (wopkg_string), like wopkg_verify.

url is another signed list for the same keys (empty: the default). Errors: NOT_FOUND for a shard the list does not carry or a missing shard.yml, UNTRUSTED for a list that does not verify, is older or expired, or a curated shard that needs one the list lacks, UNAVAILABLE when no list can be had, INVALID_ARGUMENT for a bad request or name.

Tests

• cargo test -p wopkg --lib index::shards: signature, entry rules, bounds, closure, the shard.yml edits, lock text, check findings, keys folder, cache and replay; and the signed fixture the Crystal specs use (wopkg/spec/fixtures/shards, test-only key; WOL_BLESS=1 writes it again).
• cargo test -p wopkg --test abi --test c_smoke: the 1.1 node, header and snapshot.
• cargo test -p controller --test shard: argv mapping, exit status, catalogue access, and the app template after wopkg shard add passing the RCR-040/041 scanners (needs the built CLI).
• make spec in wopkg/: spec/shard_spec.cr, including a real shards install --frozen --local from a primed cache whose remote is gone (when git and shards are installed).