Modules

A module is a package that plugs into an app through one contract. It says what it is, which versions of Spider it was tested with, which other modules it needs, which tables it brings and which variables it reads. Spider checks all of that when the app compiles, and again before the server listens: a module that does not fit does not compile, and one that is not ready keeps the server from starting, saying why.

The official modules live in the spider-modules repository. The first one is notify: notifications through an inbox, live pages, Web Push, FCM and email, with retries. Anyone can publish a module of their own, in any repository: Writing a module.

The code on this page is from examples/modules (source): an app with two modules, heartbeat, written for the example, and notify.

Adding one

A module comes as a Zig package, and goes in build.zig.zon like any other dependency:

zig

.{
    .name = .modules_example,
    .version = "0.1.0",
    .fingerprint = 0x1aa1d8ab4e9bf911,
    .minimum_zig_version = "0.17.0",
    .dependencies = .{
        .spider = .{
            .url = "git+https://github.com/llllOllOOll/spider?ref=v0.11.0#4deaac0c713a78d801763cc072c9b48182447625",
            .hash = "spider-0.11.0-P7cGrgVS0wDwj8Kifdr2RaEiXPkVaFbNaDNbzGeKiI3f",
        },
        // The official modules: one package, pinned to a commit.
        .spider_modules = .{
            .url = "git+https://github.com/llllOllOOll/spider-modules#d99f79f78e29fac413170c92c8f879d9441d60d5",
            .hash = "spider_modules-0.1.0-9C7p2CmAAgCFAVi-gFFRvcerLcu2TO8qm-GZOtmg3ax0",
        },
        // A module of this example, in a folder of its own.
        .heartbeat = .{
            .path = "heartbeat",
        },
    },
    .paths = .{
        "build.zig",
        "build.zig.zon",
        "spider.config.zig",
        "src",
        "heartbeat",
    },
}

spider-modules has no release tags yet, so the app pins a commit. zig fetch writes the two lines:

bash

zig fetch --save=spider_modules "git+https://github.com/llllOllOOll/spider-modules#d99f79f78e29fac413170c92c8f879d9441d60d5"

Then build.zig gives the module the app's own spider:

zig

/// notify, from the spider-modules package, given the app's own spider.
fn notifyModule(b: *std.Build, spider_mod: *std.Build.Module, target: std.Build.ResolvedTarget) *std.Build.Module {
    const modules_dep = b.dependency("spider_modules", .{});
    return b.createModule(.{
        .root_source_file = modules_dep.path("notify/src/notify.zig"),
        .target = target,
        .imports = &.{.{ .name = "spider", .module = spider_mod }},
    });
}

and the app imports it under a name, in the .imports of the executable: .{ .name = "spider_notify", .module = notify_mod }.

A module never brings a Spider of its own. There is one Spider in the build, the app's, and every module gets it. That is why the app builds the module from its source file instead of taking something the package compiled.

Mounting

Every module goes in one call to mountModules, before the features:

zig

/// Every module of the app, in one list: mountModules checks them together
/// and starts them in the order their `requires` give.
const modules = .{heartbeat};

/// Builds the server and listens. main() calls it, and so do the tests in
/// src/app_test.zig.
pub fn serve() !void {
    var server = spider.app(.{});
    defer server.deinit();

    try server
        .use(spider.logger)
        // The modules first: their routes and jobs, their checks before
        // listen() accepts a connection, their boot() before the features'.
        .mountModules(modules)
        .mountFeatures(features)
        .listen(.{});
}

When the app compiles, mountModules checks each module's manifest, the exact signatures of what it declares, that no name is mounted twice, and that every module a module requires is in the same call, with no cycle between them. Then it works out the order they start in: a module after the ones it requires, and otherwise in the order given.

zig

const audit = struct {
    pub const spider_module = spider.module(.{
        .name = "audit",
        .version = "1.0.0",
        .spider = ">=0.11.0 <0.12.0",
    });

    pub fn check(_: *spider.Check) !void {
        record("audit");
    }
};

const billing = struct {
    pub const spider_module = spider.module(.{
        .name = "billing",
        .version = "1.0.0",
        .spider = ">=0.11.0 <0.12.0",
        // audit starts before billing, whatever the order they are mounted in.
        .requires = .{"audit"},
    });

    pub fn check(_: *spider.Check) !void {
        record("billing");
    }
};

test "modules start after the modules they require" {
    var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena.deinit();
    var threaded: std.Io.Threaded = .init(std.testing.allocator, .{});
    defer threaded.deinit();

    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{ billing, audit });

    checked_len = 0;
    const report = server.check(arena.allocator(), threaded.io());
    try std.testing.expect(report.ok());
    try std.testing.expectEqual(@as(usize, 2), report.checked);
    try std.testing.expectEqualStrings("audit", checked[0]);
    try std.testing.expectEqualStrings("billing", checked[1]);
}

Mounting them in two calls would hide their dependencies from each other, so listen() refuses it (mountModules ran 2 times: mount every module in one call).

What a module declares is mounted with it: its routes beside the app's, its jobs started by listen(). spider routes lists them with the app's own:

text

GET     /                                                public
GET     /_spider/health                                  public  quiet_log
GET     /heartbeat                                       public  quiet_log
GET     /up                                              public  quiet_log
4 routes
1 background jobs, every: 1000ms

Before the server listens

listen() runs the checks of every module before it opens the port. Spider's own come first: every variable the module requires is set and not blank, and its migrations are applied and unchanged. Then the module's own check, for what only the module knows. One problem is enough to stop the start, and every problem is listed:

text

spider: 1 module(s) not ready, refusing to start
  heartbeat: HEARTBEAT_TOKEN is required but not set

listen() then returns error.ModulesNotReady. The test that runs heartbeat without its token:

zig

test "without HEARTBEAT_TOKEN the server refuses to start, and says why" {
    var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena.deinit();
    var threaded: std.Io.Threaded = .init(std.testing.allocator, .{});
    defer threaded.deinit();

    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{heartbeat});

    const report = server.check(arena.allocator(), threaded.io());
    var out: std.Io.Writer.Allocating = .init(arena.allocator());
    try report.write(&out.writer);
    try std.testing.expectEqualStrings(
        \\spider: 1 module(s) not ready, refusing to start
        \\  heartbeat: HEARTBEAT_TOKEN is required but not set
        \\
    , out.written());

    // listen() runs the same checks, before it binds the port.
    try std.testing.expectError(error.ModulesNotReady, server.listen(.{}));
}

When every module is ready, the modules boot in their order, then the features. A deploy with a missing setting fails at once, on the server, instead of on the first request that needed it.

Checking without starting

bash

spider check --modules

It builds the app and runs it with SPIDER_CHECK=1: listen() prints the report and returns without serving. The exit status is 1 when a module is not ready, which makes it a step for CI or for a deploy script, run with the same environment as the app. (spider check alone checks the project's conventions; --modules does both.)

A module with tables

notify keeps its data in PostgreSQL, so it brings migrations. The app needs .pg = true on the Spider dependency, and spider.pg.init before listen(). Without a database, the report says what is missing:

zig

test "notify without a database: the report says what to do" {
    var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena.deinit();
    var threaded: std.Io.Threaded = .init(std.testing.allocator, .{});
    defer threaded.deinit();

    // No spider.pg.init(...) in this test.
    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{ notify, heartbeat });

    const report = server.check(arena.allocator(), threaded.io());
    var out: std.Io.Writer.Allocating = .init(arena.allocator());
    try report.write(&out.writer);
    try std.testing.expectEqualStrings(
        \\spider: 1 module(s) not ready, refusing to start
        \\  notify: ships database migrations, but the database is not open: call spider.pg.init(...) before listen()
        \\
    , out.written());
}

On a database where they have not run yet, each migration is a problem, by name: notify: migration notify/0001_init is not applied. spider migrate applies them: the modules' first, in the order the modules start, then the app's own. Each runs in its own transaction with its row in schema_migrations, recorded as module/name with a checksum, so an edited migration is noticed. Migrations in production has the rest, and this test runs it on a disposable PostgreSQL (examples/modules/test-db.sh):

zig

test "pending migrations keep the server from starting; migrate applies them, modules in start order" {
    const gpa = std.testing.allocator;
    var threaded: std.Io.Threaded = .init(gpa, .{});
    defer threaded.deinit();
    const io = threaded.io();
    var arena: std.heap.ArenaAllocator = .init(gpa);
    defer arena.deinit();
    const a = arena.allocator();

    try spider.pg.init(gpa, io, .{});
    defer spider.pg.deinit();

    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{ notify, audit });

    // A new database: nothing applied yet.
    var before: std.Io.Writer.Allocating = .init(a);
    try server.check(a, io).write(&before.writer);
    try std.testing.expectEqualStrings(
        \\spider: 2 module(s) not ready, refusing to start
        \\  notify: migration notify/0001_init is not applied (run `spider migrate`)
        \\  audit: migration audit/0001_init is not applied (run `spider migrate`)
        \\  audit: migration audit/0002_actor is not applied (run `spider migrate`)
        \\
    , before.written());

    // What `spider migrate` (SPIDER_MIGRATE=1) runs.
    try std.testing.expectEqual(@as(usize, 3), try server.migrate(a));
    try std.testing.expect(server.check(a, io).ok());
    // Nothing left: a second run applies none.
    try std.testing.expectEqual(@as(usize, 0), try server.migrate(a));

    const Row = struct { version: []const u8, checksum: ?[]const u8 };
    const rows = try spider.pg.query(Row, a, "SELECT version, checksum FROM schema_migrations ORDER BY version", .{});
    try std.testing.expectEqual(@as(usize, 3), rows.len);
    try std.testing.expectEqualStrings("audit/0001_init", rows[0].version);
    try std.testing.expectEqualStrings("audit/0002_actor", rows[1].version);
    try std.testing.expectEqualStrings("notify/0001_init", rows[2].version);
    // The SHA-256 of the SQL that ran, in hexadecimal.
    try std.testing.expectEqual(@as(usize, 64), rows[0].checksum.?.len);
}

What notify needs at boot (its channels, an address book) is in its README.

In tests

server.check(arena, io) runs the same checks and returns the report: report.ok(), report.problems (each with its .module and .message), report.warnings, and report.write(w) for the text listen() would log. A test can mount the modules on a server that never listens:

zig

const mailer = struct {
    pub const spider_module = spider.module(.{
        .name = "mailer",
        .version = "1.0.0",
        .spider = ">=0.11.0 <0.12.0",
        .env = .{ .required = .{"EXAMPLE_SMTP_URL"} },
    });
};

test "a required variable that is not set keeps the server from starting" {
    var arena: std.heap.ArenaAllocator = .init(std.testing.allocator);
    defer arena.deinit();
    var threaded: std.Io.Threaded = .init(std.testing.allocator, .{});
    defer threaded.deinit();

    var server = spider.app(.{});
    defer server.deinit();
    _ = server.mountModules(.{ heartbeat, mailer });

    const report = server.check(arena.allocator(), threaded.io());
    var out: std.Io.Writer.Allocating = .init(arena.allocator());
    try report.write(&out.writer);
    try std.testing.expectEqualStrings(
        \\spider: 1 module(s) not ready, refusing to start
        \\  mailer: EXAMPLE_SMTP_URL is required but not set
        \\
    , out.written());
}

The module's routes and jobs are tested like the app's own, through spider.testing.start: Testing.

Versions

A module says which versions of Spider it was tested with, as a range: .spider = ">=0.11.0 <0.12.0". An app built with a Spider outside it does not compile:

text

error: spider.module("old_widget"): supports Spider >=0.12.0 <0.13.0, but this is Spider 0.11.0

So after spider update, a module that was not tested with the new Spider stops the build until a version of it that was is out. The official modules are tested before Spider is: a release does not ship until every one of them passes on it, and their ranges are then widened to include it (Writing a module says how).