Zig 0.17.0

Writing a module

A module is a package whose root file declares what it is, in a manifest, and may declare a check, a boot, a shutdown, jobs, routes and migrations. Spider holds each of those to an exact shape: a module that gets one wrong does not compile, in any app.

The code on this page is from examples/modules (source). Its module, heartbeat, tells an uptime monitor that the app's background work is alive: a job beats once a second, and a route answers how many beats there have been, to whoever sends the right token. Modules is about using one.

The manifest

zig
/// What the module is, for Spider: checked when the app compiles.
pub const spider_module = spider.module(.{
    .name = "heartbeat",
    .version = "0.1.0",
    // The Spider versions this module was tested with.
    .spider = ">=0.11.0 <0.12.0",
    // Spider refuses to start the server while one of these is not set.
    .env = .{ .required = .{"HEARTBEAT_TOKEN"} },
});
  • .name: lowercase letters, digits and _, starting with a letter, at most 64 bytes. Unique in the app.
  • .version: the module's own version, major.minor.patch.
  • .spider: the versions of Spider the module was tested with. One or more comparisons separated by spaces, all of which must hold: >=0.11.0 <0.12.0. ^ and ~ are not understood. Spider compares it with its own version, spider.version.
  • .requires: other modules, by name. They start before this one, and must be mounted in the same call.
  • .migrations: its tables, below.
  • .env: the variables it reads. .required ones must be set, and not blank, for the server to start; .optional ones are there for the reader.

An unknown field does not compile.

What does not compile

Every rule is checked when the app compiles, and the message names the module. The example keeps four modules that must fail, and its build checks each message: one tested with another Spider, a boot that Spider cannot call, a requirement that is not mounted, and migrations out of order:

text
spider.module("old_widget"): supports Spider >=0.12.0 <0.13.0, but this is Spider 0.11.0
mountModules: module "late": boot must be `pub fn boot(b: spider.Boot) !void`
mountModules: module "billing" requires "audit", which is not mounted: add it to the same mountModules call
spider.module("audit"): migration "0001_init" comes after "0002_actor": list migrations in the order they run, without repeats

The second one:

zig
//! Does not compile, on purpose: a boot() Spider could not call.
const spider = @import("spider");

const late = struct {
    pub const spider_module = spider.module(.{
        .name = "late",
        .version = "0.1.0",
        .spider = ">=0.11.0 <0.12.0",
    });

    pub fn boot() void {}
};

test {
    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{late});
}

The rest of the rules: a version that is not semver, a range that does not parse, a module that requires itself or names one twice, requirements in a cycle, the same module mounted twice, a variable name that is not [A-Z][A-Z0-9_]*, a check or shutdown with another signature, jobs that are not spider.every, and a function in routes that returns a spider.Group but takes parameters (skipping it would hide routes, so it is an error).

check

zig
/// Runs before the server listens. Spider has already checked that
/// HEARTBEAT_TOKEN is set; this checks what only the module knows.
pub fn check(c: *spider.Check) !void {
    const token = c.env("HEARTBEAT_TOKEN") orelse return;
    if (token.len < min_token_len)
        c.fail("HEARTBEAT_TOKEN must be at least {d} characters (it has {d})", .{ min_token_len, token.len });
}

check runs before the server listens, for every module, before any of them boots. Spider has already checked the variables in .env.required and the migrations; check is for what only the module knows: a setting that is set but wrong, two settings that disagree, a table it cannot read.

  • c.fail(format, args) records a problem and the check goes on, so the report lists every one. Returning an error counts as one more.
  • c.env(name) reads a variable, c.arena is memory for the check, c.io is for a query or a file.
  • It must not change anything. spider check --modules runs it on a server that never starts.
  • It does not run when Spider's own checks already failed for the module: it would only repeat the cause.

With a token of five characters:

zig
test "a token too short: the module's own check keeps the server from starting" {
    var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena.deinit();
    var threaded: std.Io.Threaded = .init(std.testing.allocator, .{});
    defer threaded.deinit();

    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{heartbeat});

    const report = server.check(arena.allocator(), threaded.io());
    try std.testing.expect(!report.ok());
    try std.testing.expectEqualStrings("heartbeat", report.problems.items[0].module);
    try std.testing.expectEqualStrings(
        "HEARTBEAT_TOKEN must be at least 16 characters (it has 5)",
        report.problems.items[0].message,
    );
    try std.testing.expectError(error.ModulesNotReady, server.listen(.{}));
}

boot, shutdown, jobs and routes

text
compile   manifests, signatures, the order the modules start in
listen()  every module's checks: a problem? refuse to start, listing them all
          boot(module 1) → boot(module 2) → … → the features' boot → serve
deinit()  the jobs stop → shutdown(module N) → … → shutdown(module 1)
zig
/// Runs once, after every check passed and before the first request.
pub fn boot(b: spider.Boot) !void {
    started_s = std.Io.Clock.real.now(b.io).toSeconds();
    beats.store(0, .monotonic);
}

/// Runs from `server.deinit()`, after the jobs stopped: a server started
/// again in the same process begins from nothing.
pub fn shutdown(_: spider.Boot) void {
    started_s = null;
    beats.store(0, .monotonic);
}
  • boot runs once, after every check passed, in the order the modules start, before the features' own boot. An error stops listen(), which returns it. It gets the same spider.Boot as a feature's: an allocator that lives as long as the process, and the server's io.
  • shutdown runs from server.deinit(), after the jobs stopped, in the reverse order. It also runs for a module that never booted (a test that mounts it and stops), so it must not count on boot having run.

Jobs are a feature's jobs (Scheduled jobs): listen() starts them, and the first run comes one interval later.

zig
/// How often the job beats, in milliseconds.
pub const every_ms = 1000;

/// Started by `listen()`; the first beat comes one interval after it.
pub const jobs = .{spider.every(every_ms, beat)};

fn beat(_: *spider.Hub) void {
    _ = beats.fetchAdd(1, .monotonic);
}

Routes are a struct of functions that take nothing and return a spider.Group, as in a feature's routes.zig. Each route declares who may call it, as any route does (Routing). heartbeat's is public to the app's login and checks its own token instead:

zig
/// Mounted by `mountModules`, like a feature's routes.
pub const routes = struct {
    pub fn build() spider.Group {
        var g = spider.Group.init("/heartbeat");
        // Public to the app's login: the token is the module's own.
        _ = g.get("", status, .{ .public = true, .quiet_log = true });
        return g;
    }
};

fn status(c: *spider.Ctx) !spider.Response {
    const token = spider.env.get("HEARTBEAT_TOKEN") orelse "";
    if (!authorized(c.header("Authorization"), token)) return error.Unauthorized;
    const now_s = std.Io.Clock.real.now(c.io()).toSeconds();
    return c.json(.{
        .beats = beats.load(.monotonic),
        .up_seconds = now_s - (started_s orelse now_s),
    }, .{});
}

Migrations

A module with tables lists its migrations in the manifest, in the order they run. The SQL lives in files of the package, one per migration, embedded when the app compiles:

zig
pub const spider_module = spider.module(.{
    .name = "audit",
    .version = "0.1.0",
    .spider = ">=0.11.0 <0.12.0",
    // In the order they run. Recorded as audit/0001_init, audit/0002_actor.
    .migrations = .{
        .{ .name = "0001_init", .sql = @embedFile("sql/0001_init.sql") },
        .{ .name = "0002_actor", .sql = @embedFile("sql/0002_actor.sql") },
    },
});
sql
-- The module's table. No BEGIN/COMMIT: Spider runs each migration in a
-- transaction of its own. IF NOT EXISTS: an app that made this table by
-- hand before only records the migration.
CREATE TABLE IF NOT EXISTS audit_events (
    id BIGINT GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
    action TEXT NOT NULL,
    at TIMESTAMPTZ NOT NULL DEFAULT now()
);
  • Names are at least four digits, _ and a description in lowercase: 0001_init. The row in schema_migrations is audit/0001_init, with a SHA-256 checksum of the SQL.
  • PostgreSQL, forward only. The whole file runs; there is no down part.
  • No BEGIN or COMMIT. Spider runs each migration in a transaction of its own, with its row. One that controls transactions is refused before anything runs (error.MigrationHasTransactionControl).
  • A statement that cannot run in a transaction (CREATE INDEX CONCURRENTLY, VACUUM) must be alone in its migration, and then runs outside one.
  • Never edit a migration you released. An applied migration whose SQL changed stops the migration and the start (error.MigrationChanged). Add the next one, as 0002_actor adds a column:
sql
-- A released migration is never edited: what changed comes as a new one.
ALTER TABLE audit_events ADD COLUMN actor TEXT;
  • A first migration that may meet a table an app made by hand uses CREATE ... IF NOT EXISTS: applying it then only records it.

Packaging

A module is a Zig package with its own build.zig.zon, and a real fingerprint (the first zig build suggests one). The package has no dependency on Spider: the app gives the module its own spider, so there is one Spider in the build.

A repository that also tests its module needs a Spider for those tests. It pins one as a lazy dependency, which only that repository's own build fetches:

zig
.{
    .name = .spider_heartbeat,
    .version = "0.1.0",
    .fingerprint = 0x425c822b666c8cf5,
    .minimum_zig_version = "0.17.0",
    .dependencies = .{
        // The Spider the module's tests run on. Lazy: only this package's
        // own build fetches it. An app that depends on the package gives
        // the module its own spider instead.
        .spider = .{
            .url = "git+https://github.com/llllOllOOll/spider?ref=v0.11.0#4deaac0c713a78d801763cc072c9b48182447625",
            .hash = "spider-0.11.0-P7cGrgVS0wDwj8Kifdr2RaEiXPkVaFbNaDNbzGeKiI3f",
            .lazy = true,
        },
    },
    .paths = .{
        "build.zig",
        "build.zig.zon",
        "src",
    },
}

and its build.zig stops at once when it is not the build being run. An app that depends on the package then takes only the module's file; a second Spider in the app's build would clash with the app's (file exists in modules 'spider' and 'spider0'):

zig
pub fn build(b: *std.Build) void {
    // An app that depends on this package only takes src/heartbeat.zig and
    // gives it the app's own spider. A second Spider here would meet the
    // app's in one build ("file exists in modules 'spider' and
    // 'spider0'"): what follows is this package's own build, its tests.
    if (!b.isRoot()) return;

    const target = b.standardTargetOptions(.{});
    const optimize = b.standardOptimizeOption(.{});
    // Lazy: only fetched here, never by an app that depends on the package.
    const spider_dep = b.lazyDependency("spider", .{ .target = target, .optimize = optimize }) orelse return;

    const tests = b.addTest(.{ .root_module = b.createModule(.{
        .root_source_file = b.path("src/heartbeat.zig"),
        .target = target,
        .optimize = optimize,
        .imports = &.{.{ .name = "spider", .module = spider_dep.module("spider") }},
    }) });
    const run = b.addRunArtifact(tests);
    run.setEnvironmentVariable("HEARTBEAT_TOKEN", "a-token-of-the-module-tests");
    b.step("test", "The module's own tests").dependOn(&run.step);
}

The app, for its part, builds the module from that file with its own spider:

zig
/// The heartbeat module, given the app's own spider. Its package has no
/// Spider of its own: there is one Spider in the build, the app's.
fn heartbeatModule(b: *std.Build, spider_mod: *std.Build.Module, target: std.Build.ResolvedTarget) *std.Build.Module {
    const dep = b.dependency("heartbeat", .{});
    return b.createModule(.{
        .root_source_file = dep.path("src/heartbeat.zig"),
        .target = target,
        .imports = &.{.{ .name = "spider", .module = spider_mod }},
    });
}

The conventions of the official modules hold for anyone's: the package is called spider_<name>, and its README says how to install it, what to configure and what check prints when a variable is missing.

Testing a module

Unit tests go next to the code. A test can also mount the module on a server that never listens and ask for the report, then play the lifecycle:

zig
test "through Spider: ready with a token, and deinit runs shutdown" {
    var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena.deinit();
    var threaded: std.Io.Threaded = .init(std.testing.allocator, .{});
    defer threaded.deinit();

    // build.zig gives this test run a HEARTBEAT_TOKEN long enough.
    var server = spider.app(.{});
    _ = server.mountModules(.{@This()});
    try std.testing.expect(server.check(arena.allocator(), threaded.io()).ok());

    // listen() would boot it; here the test does.
    try boot(.{ .allocator = std.testing.allocator, .io = threaded.io() });
    try std.testing.expect(running());
    server.deinit();
    try std.testing.expect(!running());
}

zig build test in the package runs them on the Spider of its build.zig.zon. To run them on a Spider checkout instead, a release candidate or main:

bash
zig build test --fork=../spider

Give --fork a path relative to the repository: Zig 0.17 refuses an absolute one for a package that has path dependencies of its own, as Spider does.

A module with tables also needs tests on a database, through Spider: not ready, migrated, ready. Run them on a PostgreSQL that exists for one run, never on a database you care about. examples/modules/test-db.sh does that for the test in Modules.

Publishing

Publish the package in any repository: apps fetch it by its address and pin it by its hash, as they do Spider. There is no registry.

  • Version. Follow semver. Before 1.0 a breaking change raises the minor version, and goes in the changelog.
  • The Spider range says what your tests passed on, not what you hope works. Widen it after the tests pass on the new Spider, never before.
  • The release gate. The official modules in spider-modules come with scripts/ecosystem.sh, which builds and tests every one of them, database tests included, against a Spider checkout. Spider's CI runs it on every push and pull request to its develop and main branches, and a release runs it on the tag first: a release does not ship while an official module fails. When it passes on a new Spider, the modules' ranges are widened to include it.
  • A module of your own is not part of that gate today. Spider's plan for modules (RFC 0001) has a list of third-party modules in the docs, and an opt-in to the same check, whose failures are reported to the module's authors and never block a release. Until then, run your tests against Spider's main in your own CI with --fork.