Zig Msgpack

MessagePack is a binary serialization format. It solves the same problem JSON does, exchanging structured data between programs written in different languages, but it produces smaller output and parses faster.

The efficiency comes from not spending bytes on syntax. Small integers encode into a single byte. A short string costs one byte of overhead plus the string itself. There are no quotes, no commas, no whitespace to skip.

You have probably used it without noticing. Neovim speaks MessagePack for its RPC protocol, which is how every remote plugin and GUI talks to the editor. Redis uses it too.

This is a walkthrough of zig-msgpack, a MessagePack implementation for Zig. The format specification is here if you want the authoritative version.

How the format works

Every value starts with a one-byte marker describing what follows. For fixed-size types the marker is enough on its own. For variable-length types, a few bytes of length follow the marker, and then the data itself.

Here is the read path for a simple value:

The types are Nil, Bool, Int, Float, Str, Bin, Array, Map, and Ext. Timestamps are a predefined extension type, and zig-msgpack implements all three of their encodings.

Adding it to your project

Fetch the package:

zig fetch --save https://github.com/zigcc/zig-msgpack/archive/{COMMIT_OR_BRANCH}.tar.gz

Then wire the module into your executable in build.zig:

const msgpack_dep = b.dependency("zig_msgpack", .{
    .target = target,
    .optimize = optimize,
});

exe.root_module.addImport("msgpack", msgpack_dep.module("msgpack"));

The short path: PackerIO

On Zig 0.15 and later, PackerIO wraps the standard std.Io.Reader and std.Io.Writer. This is the API you want unless you have a reason not to.

const std = @import("std");
const msgpack = @import("msgpack");

pub fn main() !void {
    var gpa = std.heap.GeneralPurposeAllocator(.{}){};
    defer _ = gpa.deinit();
    const allocator = gpa.allocator();

    var buffer: [1024]u8 = undefined;
    var writer = std.Io.Writer.fixed(&buffer);
    var reader = std.Io.Reader.fixed(&buffer);

    var packer = msgpack.PackerIO.init(&reader, &writer);

    var map = msgpack.Payload.mapPayload(allocator);
    defer map.free(allocator);
    try map.mapPut("name", try msgpack.Payload.strToPayload("Alice", allocator));
    try map.mapPut("age", msgpack.Payload.uintToPayload(30));
    try packer.write(map);

    reader.seek = 0;
    const decoded = try packer.read(allocator);
    defer decoded.free(allocator);

    const name = (try decoded.mapGet("name")).?.str.value();
    const age = (try decoded.mapGet("age")).?.uint;
    std.debug.print("Name: {s}, Age: {d}\n", .{ name, age });
}

msgpack.packIO(&reader, &writer) is a shorthand for the same thing.

Reading and writing a file works the same way, since a file's reader and writer implement the same interfaces:

var file = try std.fs.cwd().createFile("data.msgpack", .{ .read = true });
defer file.close();

var reader_buf: [4096]u8 = undefined;
var writer_buf: [4096]u8 = undefined;
var reader = file.reader(&reader_buf);
var writer = file.writer(&writer_buf);

var packer = msgpack.PackerIO.init(&reader, &writer);

Remember to flush() the writer and rewind before reading back.

Building payloads

Everything goes through Payload, a tagged union covering every MessagePack type. Constructors exist for each:

const nil_val = msgpack.Payload.nilToPayload();
const bool_val = msgpack.Payload.boolToPayload(true);
const int_val = msgpack.Payload.intToPayload(-42);
const uint_val = msgpack.Payload.uintToPayload(42);
const float_val = msgpack.Payload.floatToPayload(3.14);

const str_val = try msgpack.Payload.strToPayload("hello", allocator);
const bin_val = try msgpack.Payload.binToPayload(&[_]u8{ 1, 2, 3 }, allocator);

var arr = try msgpack.Payload.arrPayload(2, allocator);
try arr.setArrElement(0, msgpack.Payload.intToPayload(1));
try arr.setArrElement(1, msgpack.Payload.intToPayload(2));

var map = msgpack.Payload.mapPayload(allocator);
try map.mapPut("key", msgpack.Payload.intToPayload(42));

Anything allocating takes an allocator and needs a matching free(allocator).

Map keys are not restricted to strings. mapPut and mapGet take []const u8 for the common case, but mapPutGeneric and mapGetGeneric accept any Payload as a key:

var m = msgpack.Payload.mapPayload(allocator);
try m.mapPutGeneric(
    msgpack.Payload.intToPayload(1),
    try msgpack.Payload.strToPayload("one", allocator),
);

Reading values back

Decoding gives you a Payload, which means you can parse data whose shape you don't know ahead of time. Getting a concrete Zig value out comes in two flavours.

Lenient accessors convert where it is safe to do so. getInt() accepts a uint that fits in i64; getUint() accepts a non-negative int:

const n = try payload.getInt();

Strict accessors refuse to convert. asInt() only accepts .int, even when a .uint would fit:

const n = try payload.asInt();

The same pairing applies to asUint, asFloat, asBool, asStr, and asBin. There are also isNil(), isNumber(), and isInteger() for checking before you commit.

The generic Pack API

PackerIO is a thin layer over Pack, which takes read and write behaviour as comptime parameters. Use it directly when you need control over the underlying transport, or when you need to support Zig versions before 0.15.

fn Pack(
    comptime WriteContext: type,
    comptime ReadContext: type,
    comptime WriteError: type,
    comptime ReadError: type,
    comptime writeFn: fn (context: WriteContext, bytes: []const u8) WriteError!usize,
    comptime readFn: fn (context: ReadContext, arr: []u8) ReadError!usize,
) type

If that shape looks familiar, it is deliberate. It mirrors the old std.io.GenericWriter and std.io.GenericReader.

One wrinkle: std.io.FixedBufferStream was removed in Zig 0.16. zig-msgpack ships a compatibility layer that papers over the difference, so the same code compiles on 0.14 through 0.16:

const compat = msgpack.compat;

var buffer: [1024]u8 = undefined;
var write_buffer = compat.fixedBufferStream(&buffer);
var read_buffer = compat.fixedBufferStream(&buffer);

const BufferType = compat.BufferStream;
var packer = msgpack.Pack(
    *BufferType,
    *BufferType,
    BufferType.WriteError,
    BufferType.ReadError,
    BufferType.write,
    BufferType.read,
).init(&write_buffer, &read_buffer);

try packer.write(msgpack.Payload.boolToPayload(true));

read_buffer.pos = 0;
const decoded = try packer.read(allocator);
defer decoded.free(allocator);

Strings, binary, and extensions

Zig has no dedicated string type, so the library defines one to keep Str and Bin distinguishable at the type level. Both are thin wrappers with a value() accessor:

pub const Str = struct {
    str: []const u8,
    pub fn value(self: Str) []const u8 {
        return self.str;
    }
};

pub inline fn wrapStr(str: []const u8) Str {
    return Str{ .str = str };
}

Bin is the same shape over []u8, with wrapBin. Extension types carry an 8-bit type tag alongside their payload:

pub const EXT = struct {
    type: i8,
    data: []u8,
};

pub inline fn wrapEXT(t: i8, data: []u8) EXT {
    return EXT{ .type = t, .data = data };
}

Timestamps

Timestamp is extension type -1 in the specification, and the library treats it as a first-class Payload variant rather than something you decode by hand:

const ts1 = msgpack.Payload.timestampFromSeconds(1234567890);
const ts2 = msgpack.Payload.timestampToPayload(1234567890, 123456789);

try packer.write(ts2);

const decoded = try packer.read(allocator);
defer decoded.free(allocator);

std.debug.print("{d}s + {d}ns\n", .{
    decoded.timestamp.seconds,
    decoded.timestamp.nanoseconds,
});

All three encodings (32-bit, 64-bit, and 96-bit) are handled. toFloat() collapses one into a single f64 when you don't need nanosecond precision.

Parsing untrusted input

Anything that decodes data off a network deserves suspicion, and MessagePack has a specific hazard: a length prefix is a claim, not a fact. A handful of bytes can claim a billion-element array.

Two design decisions address this. The parser is iterative rather than recursive, keeping its stack on the heap, so deeply nested input cannot blow the call stack. And every limit is checked *before* allocation, so a hostile length prefix is rejected instead of honoured.

The defaults are reasonable for most uses: 1000 levels of nesting, 1M elements per container, 100MB strings. When you need something tighter, PackWithLimits takes the same parameters as Pack plus a limits struct:

const StrictPacker = msgpack.PackWithLimits(
    *Writer,
    *Reader,
    Writer.Error,
    Reader.Error,
    Writer.write,
    Reader.read,
    .{
        .max_depth = 50,
        .max_array_length = 10_000,
        .max_map_size = 10_000,
        .max_string_length = 1024 * 1024,
        .max_bin_length = 1024 * 1024,
        .max_ext_length = 512 * 1024,
    },
);

Violations surface as ordinary Zig errors, so you handle them like anything else:

msgpack.MsgPackError.MaxDepthExceeded
msgpack.MsgPackError.ArrayTooLarge
msgpack.MsgPackError.MapTooLarge
msgpack.MsgPackError.StringTooLong
msgpack.MsgPackError.BinDataLengthTooLong
msgpack.MsgPackError.ExtDataTooLarge

Zig version support

The library's minimum is Zig 0.14.0.

Where to look next

The unit tests are the most complete set of examples, covering every type and both APIs. zig build test runs them, zig build bench measures throughput, and zig build docs generates the API reference.