The Standard Library

Cone's standard library is a set of packages that come with the language, in the packages folder of its repository. Each is written in Cone (a few lean on the C library or the operating system), each is built on its own, and a program uses one by importing it by name. This page lists them, says what each is for, how to import it, and where it is documented. Seven of them have pages of their own: text: str, String and cstr; collections; files, folders, paths, the environment and processes; time; HTTP, TLS and URLs; game networking; and testing.

The graphics, windowing and 3-D packages in the same folder are not described here.

Importing a package#

A package is imported by its name, with the import statement of Namespaces (and Talking to C for extern). There is no list of dependencies to write: Congo reads the import lines, finds each package by name (Packages and Congo), builds it, and links it. The three usual forms:

mod tour;

import stdio use *;
import collections use *;
import time use Duration;
import textdiff;

fn main() {
  // 'use *' brings in every public name: List, String, Dict...
  var names = List[String].empty();
  names.push(String.from("collections"));
  names.push(String.from("time"));
  printUInt(u64.from(names.len()));
  printStr(" packages used\n");

  // 'use Duration' brings in just that one
  let pause = Duration.fromMillis(1500);
  printInt(pause.millis());
  printStr(" ms\n");

  // with no 'use', the package's name is the way in: textdiff.unifiedText
  let shown = textdiff.unifiedText("a\n", "b\n", "old", "new");
  printStr(shown.view());
}

which prints:

2 packages used
1500 ms
--- old
+++ new
@@ -1 +1 @@
-a
+b

One package needs no import at all: core, the prelude, whose names ( Option, Result, str, the atomic values, the panic functions) every module has already.

The packages#

Foundations#

Package	What it is for
core (no import)	The prelude: Option, Result, str and the other types and functions of the language itself, and its intrinsics (Intrinsics).
stdio import stdio use *;	Printing text and numbers to standard output and standard error: printStr, printInt, print <- x, and the eprint forms for standard error. The <- prints text, integers, floats, a bool (as true or false) and a char (as its UTF-8 bytes), and a borrow of a number, bool or char as the value it lends. Standard output is buffered; print.flush() writes it out before another program writes to the same place.
libc import libc;	The ISO C library as C spells it: C names, C types, errors as C reports them. A raw layer for programs that call C; the packages below wrap it.
posix import posix;	The POSIX functions beyond ISO C (files, directories, processes, the environment), as raw bindings over the Windows C runtime. fs and env are the Cone layer over it.
conestd (no import)	The runtime every program links beside its own code: the end of a panic, printing, and the raw blocks the heap's memory comes from. A program never imports it.

Collections and memory#

Package	What it is for
collections import collections use *;	List, String, Dict, OrderedDict, Set and Deque (Collection Types).
arena import arena use *;	Arena: values allocated one after another and freed together (Region Strategies).
pool import pool use *;	Pool[T]: values added and removed one at a time, reached by a generational Ref that stops reaching anything once its value is removed (Region Strategies).
rcweak import rcweak use *;	Rcw and Weak[T]: reference counting with weak references that do not keep a value alive (Weak References).
collector import collector;	Gc, a tracing garbage-collected region. Single-threaded, for code outside actors only, and discouraged: inside an actor, use the actor's own collected region (Region Strategies).

Threads, locks and actors#

Package	What it is for
thread import thread;	OS threads: start, join, detach, sleep, park (Threads, atomics and locks).
sync import sync use *;	What threads share safely: Arc, the futex, Mutex, Rwlock, Condvar, Once, channels and the one-shot Oneshot (Threads, atomics and locks, Thread Communications).
actors import actors;	The runtime an actor declaration runs on: mailboxes, the scheduler of worker threads, futures, parallel loops. A module that declares an actor or uses parallel each imports it (Actors, Parallel each).

Files, paths, environment and processes#

Package	What it is for
fs import fs;	Files and folders: read and write whole files, list, walk, rename, remove, stat, the current folder.
path import path;	File paths taken apart and put together as text; never asks the file system anything.
env import env;	The program's environment variables.
process import process;	Running another program and waiting for it; the program's own arguments.

All four are on their own page.

Time#

Package	What it is for
time import time use *;	Duration, Instant, Timestamp and Clock.
calendar import calendar;	Civil dates and times over a Timestamp, and RFC 3339 and HTTP date text.
timewheel import timewheel use *;	Timers that cost almost nothing to arm and cancel, for the I/O loop.

All three are on their own page.

Asynchronous I/O#

Package	What it is for
iobuf import iobuf use *;	Owned buffers for I/O, lent to an operation by moving them: IoBuf, Pool, Bytes, Chain.
iocore import iocore use *;	The completion loop: hand it a job (receive, send, accept, read a file, send or receive a datagram, wake at a time) and later get exactly one answer. Jobs may carry a deadline and join a cancel group.
aio import aio;	I/O an actor awaits: a behaviour awaits a socket's (TCP or UDP) or a file's operation, a timer, a name lookup, a file's metadata or a child process, and pauses until the answer comes. An operation may join a cancel group and carry a deadline.
actorhttp import actorhttp;	HTTP and HTTPS an actor awaits: a behaviour awaits client.get(url), or a request on a connection, and pauses until the whole response comes, or reads a response as it arrives, in chunks or as server-sent events. A fetch may join a cancel group and carry a deadline. The other way round, an actor serves: it awaits each request of a listening server, plain or through TLS, and answers it, whole or streamed (server-sent events too). A working name.
diag import diag;	Records of where an I/O operation's time went: one record per operation, a loop's counters, latency histograms.

I/O from an actor and The I/O loop underneath describe them.

HTTP, TLS and URLs#

Package	What it is for
url import url;	URLs read, resolved and written as text.
http import http;	HTTP's message types, shared by every version.
http1 import http1;	HTTP/1.1's connection core, with no I/O of its own.
httpconn import httpconn;	HTTP over sockets on an I/O loop, client and server.
httpclient import httpclient;	Fetching an http:// or https:// URL in one call.
tls import tls;	TLS 1.2 and 1.3 over OpenSSL, with no I/O of its own.
sse import sse;	Server-sent events (text/event-stream) read from bytes as they arrive, and written, in Cone, with no I/O of its own.

All seven are on their own page.

Game networking#

Package	What it is for
netmsg import netmsg use *;	A game's protocol over UDP, after Gaffer On Games: reliable ordered messages and latest-wins values on one connection, acknowledgements that survive heavy loss, a challenge handshake; a connection is an actor, the same over a socket, in one process, or across a simulated lossy link. A working name.

Game Networking describes it.

Chat APIs: chatapi#

Package	What it is for
chatapi import chatapi use *;	A client for the OpenAI-compatible Chat Completions API, pointed at any base URL (OpenAI's, or a llama.cpp, vLLM or Ollama server on this machine): messages and settings in, a whole reply or a stream of chunks out, over json, sse and actorhttp. The key is read from an environment variable the caller names and is sent only as the Authorization field. Written in Cone, with no C library.

chatapi#

A Chat is an actor made from an actorhttp.Client and an Endpoint (a base URL and the name of the environment variable that holds the key, or none). A behaviour awaits chat.complete(request) for a whole reply, or chat.stream(request) for a ChatStream whose next() awaits each chunk's Delta (the text it adds, then None at the end) and whose collect() gathers the rest into a reply. A ChatRequest holds the model, the messages (system, user, assistant, tool) and the settings temperature, topP, maxTokens and stop; a setting left None is not sent. Every call answers a Result: the error says whether the key was missing, the network failed, the API refused it (its status and error object) or the answer was not what the API writes, and isRetryable() whether trying again may help. The package's header lists what it chose and what it left out (tool calls, images, several choices, retries).

mod ask;

import stdio use *;
import sync use Oneshot, OneshotSender, OneshotReceiver;
import actors;
import actorhttp use Client;
import chatapi use *;

actor Asker {
  chat Chat;

  pub async do ask(self, done OneshotSender[i32]) {
    var req = ChatRequest.make("my-model");
    req.addUser("Why is the sky blue?");
    let opened = await chat.stream(req);
    match opened {
      case let cs Ok {
        var going = true;
        while going {
          let n = await cs.next();
          match n {
            case let ok Ok {
              match ok {
                case let d Some {printStr(d.content.view());}
                case is None[Delta] {going = false;}
              }
            }
            case let e Error {printStr(e.message().view()); going = false;}
          }
        }
      }
      case let e Error {printStr(e.message().view());}
    }
    var d = done;
    d.send(0i32);
  }
}

fn main() i32 {
  initAll();
  var tx OneshotSender[i32];
  var rx OneshotReceiver[i32];
  tx, rx = Oneshot[i32].make();
  {
    // a server on this machine needs no key; OpenAI's is Endpoint.openai(), the key in $OPENAI_API_KEY
    let a = new Asker(new Chat(Client.standard(), Endpoint.loopback(8080u16)));
    a.ask(tx);
  }
  rx.recv();
  finalAll();
  0i32;
}

chatapi's tests run against a mock server on loopback, and examples/ask.cone takes a base URL, a model, the key's variable and a prompt.

Text: textdiff, toml and json#

Package	What it is for
textdiff import textdiff;	The differences between two texts, line by line, and the unified diff that shows them, as diff -u prints it.
toml import toml;	A reader for a subset of TOML: tables, and string, integer, boolean and list-of-string values. It reads Congo's manifests, and suits any small configuration file that keeps to the subset.
json import json;	JSON (RFC 8259) read and written: a value tree, a streaming reader of events, a writer, numbers kept as written and read as i64 or f64 on request, UTF-8 checked. Written in Cone, with no C library.

textdiff#

textdiff.unifiedText(a, b, aName, bName) compares two texts, split into lines, and answers the unified diff as a String, empty when the texts are the same. A fifth argument, around, sets how many unchanged lines show around each change (3 unless given). unified does the same for two lists of lines, and spans answers the edit itself as a list of stretches, each equal, deleted, inserted or replaced.

mod showdiff;

import stdio use *;
import textdiff;

fn main() {
  let shown = textdiff.unifiedText("one\ntwo\nthree\n", "one\n2\nthree\n", "want", "got");
  printStr(shown.view());
}

prints:

--- want
+++ got
@@ -1,3 +1,3 @@
 one
-two
+2
 three

toml#

toml.parse(text) answers a Result[Document, TomlError]. A Document holds tables, and doc["package"] is a table (a panic if there is none; ask doc.has("package") first). A table answers getString, getInt, getBool and getStrings, each an Option that is None when the key is missing or holds another kind of value. A table name may be dotted: [link.windows] is the table doc["link.windows"], found by its whole name, whether or not [link] is written. A TOML feature outside the subset (floats, dates, inline tables, dotted keys) is an error whose message names the line.

mod readname;

import stdio use *;
import toml;

fn main() i32 {
  var doc = new toml.Document();
  match toml.parse("[package]\nname = \"hello\"\nversion = \"0.1.0\"\n") {
    case var ok Ok { doc <=> ok; }
    case let e Error {
      let m = e.message();
      printStr(m.view());
      return 1i32;
    }
  }
  match doc["package"].getString("name") {
    case let n Some { printStr(n.view()); printStr("\n"); }
    case is None { printStr("no name\n"); }
  }
  0i32;
}

prints hello.

json#

json.parse(text) answers a Result[Json, JsonError]. A Json is an enum (Null, Bool, Number, Text, Array, Object) that can be matched, and answers the common questions as methods: get and at reach a member or an element, and getText, getI64, getF64 and getBool answer an Option that is None when the member is missing or of another kind (i64Or and its kin take a fallback; requireText and its kin answer an error that names the member). A number is kept as the text it was written in, and is read as an integer or an f64 only when asked: asI64 is exact and refuses a fraction or a value out of range, asF64 is the nearest f64. toText() and toPretty(2usize) write a tree back. A Reader answers a text's events one at a time with no tree built, and a Writer writes without one. Nesting is limited to 512 levels by default, so a hostile text is refused early. There is no reflection yet: mapping a struct to JSON and back is written by hand (the package's examples/mapping.cone is one). The package's header lists what it chose and what it left out.

mod tour;

import stdio use *;
import json;

fn main() i32 {
  var doc json.Json = json.Json.Null;
  match json.parse("{\"name\": \"hello\", \"size\": 12, \"tags\": [\"a\", \"b\"]}") {
    case var ok Ok { doc <=> ok; }
    case let e Error {
      let m = e.message();
      printStr(m.view());
      return 1i32;
    }
  }
  match doc.getText("name") {
    case let n Some { printStr(n); printStr("\n"); }
    case is None { printStr("no name\n"); }
  }
  printInt(doc.i64Or("size", -1i64));
  printStr("\n");
  match doc.toPretty(2usize) {
    case let t Ok { printStr(t.view()); printStr("\n"); }
    case let e Error {
      let m = e.message();
      printStr(m.view());
      return 1i32;
    }
  }
  0i32;
}

prints:

hello
12
{
  "name": "hello",
  "size": 12,
  "tags": [
    "a",
    "b"
  ]
}

Random numbers#

Package	What it is for
random import random;	A seeded, repeatable generator (xoshiro256**) whose state is a value, not a global: integers in a range with no modulo bias, floats in [0, 1), a shuffle, a choice, and normal and exponential numbers. The same seed gives the same numbers on Windows and Linux. Written in Cone; not for keys or tokens.

random#

An Rng is made from a seed, Rng.fromSeed(42u64), or from the operating system's entropy, Rng.fromEntropy() (an Option: None if the OS refuses; fromEntropyOr(seed) falls back to a seed). It holds its own state, so a program makes as many as it needs and hands one to a function as &mut Rng; jumped() gives a generator that cannot overlap this one's numbers, one per thread. below(n) is an integer in [0, n) with every value equally likely; rangeI64(lo, hi) and closedI64(lo, hi) take a half-open and a closed range (and the U64 forms). unit() is an f64 in [0, 1), shuffle(&mut list) a Fisher-Yates shuffle of a List, choice(&list) a borrowed random element (None if the list is empty), normal() and exponential() the usual distributions. The package's header lists each method chosen and what was left out; the numbers from a seed are pinned in its tests against the reference implementations.

mod dice;

import stdio use *;
import random;

fn main() i32 {
  var rng = random.Rng.fromSeed(42u64);
  printInt(i64.from(rng.below(6u64)) + 1i64);
  printStr(" ");
  printInt(rng.rangeI64(-10i64, 10i64));
  printStr("\n");
  0i32;
}

prints 1 -3 on every machine.

Bytes as text: base64 and hex#

Package	What it is for
binarytext import binarytext;	Base64 (RFC 4648's standard and URL-safe alphabets, with or without padding) and hex, encoded and decoded, one-shot or streamed in chunks split anywhere. Decoding is strict by default. Written in Cone, with no C library and no call to the OS.

binarytext#

encodeBase64(bytes) and encodeBase64Url(bytes) answer a String (padded); encodeBase64With(bytes, &format) takes a Base64Format: standard(), urlSafe(), standardUnpadded() or urlSafeUnpadded(). encodeHex(bytes) writes lower case, encodeHexUpper upper. decodeBase64(text), decodeBase64Url, decodeBase64With(text, &format) and decodeHex answer a Result[List[u8], DecodeError]. Decoding is strict: a character outside the alphabet (the two alphabets are not mixed), whitespace, padding that does not match the format, a lone last digit, and unused trailing bits that are not zero (so that one value has one text) are each refused, with the kind of fault and its byte offset. A format opts in to what is looser: acceptingWhitespace(), acceptingAnyPadding(), acceptingTrailingBits(); hex decoding takes either case and has decodeHexSkippingWhitespace. Base64Encoder, Base64Decoder, HexEncoder and HexDecoder are values that hold their own state: feed them chunks with update, then finish. The package's header lists each choice and what was left out; the tests carry RFC 4648's section 10 vectors.

mod tokens;

import stdio use *;
import binarytext;

fn main() i32 {
  let text = binarytext.encodeBase64("foobar");
  printStr(text.view());
  printStr(" ");
  let hex = binarytext.encodeHex("foobar");
  printStr(hex.view());
  printStr("\n");
  0i32;
}

prints Zm9vYmFy 666f6f626172.

Hashes and message authentication#

Package	What it is for
cryptohash import cryptohash;	SHA-256, SHA-512 and SHA-1 (legacy, not for new security uses), HMAC over each, and a constant-time comparison for MACs. A hasher is a value fed with update; one call hashes a whole message. Written in Cone with no C library; the same bytes on Windows and Linux; checked against NIST's and the RFCs' published vectors. Not a password hash.

cryptohash#

sha256(data), sha512(data) and sha1(data) hash a byte slice (a string literal, a String's or List[u8]'s view()) and answer the digest as a fixed array, Array[u8, 32], Array[u8, 64] and Array[u8, 20]. To hash a message in pieces, make a hasher, Sha256.make() (or Sha512, Sha1), feed it with update(data) as many times as needed and ask finish(). The hasher holds its own state, with no global behind it; finish leaves it as it was, so it can take more and be asked again, copyOf() forks one in progress, and reset() empties it. HmacSha256.make(key) (and HmacSha512, HmacSha1) is the same for a keyed MAC, with hmacSha256(key, data) for one call; verify(tag) compares a received tag with the right one in constant time (constantTimeEq(a, b) does it for any two byte strings). SHA-1 collisions can be made: use it only for a format that names it. The package's header lists each method chosen and what was left out, and what the compiler could still do to the constant-time loop.

mod digest;

import stdio use *;
import cryptohash;

fn main() i32 {
  let d = cryptohash.sha256("abc");
  printInt(i64.from(d[0]));
  let tag = cryptohash.hmacSha256("a key", "a message");
  var mac = cryptohash.HmacSha256.make("a key");
  mac.update("a message");
  printStr(" ");
  printInt(if mac.verify(&tag) { 1i64; } else { 0i64; });
  printStr("\n");
  0i32;
}

prints 186 1 on every machine (186 is 0xba, the first byte of SHA-256 of "abc", ba7816bf...).

LLVM#

Package	What it is for
llvmc import llvmc;	LLVM's C API, declared by hand: make a context, a module, types, constants, globals and functions with the instruction builder, verify and print the IR, run the new pass manager's pipelines over it, attach debug information, and have a target machine write an object file. It covers every LLVM-C function the C compiler calls. A C binding to LLVM 23.1's library, which must be installed.

A function is called by its C name without the LLVM prefix (llvmc.BuildAdd), a handle is a pointer to a type of its own (*llvmc.Value is not a *llvmc.Type), and the C enums are Cone enums with LLVM's values (llvmc.IntPredicate.IntEQ). A few functions beginning with a lower-case letter copy the text LLVM hands out into a String (moduleToString, checkModule). The package's header lists each choice and what was left out; the tests build functions, verify them, break a module on purpose, write an object file and run its code.

Compression#

Package	What it is for
deflate import deflate;	DEFLATE (RFC 1951) compressed and decompressed, with zlib (RFC 1950) and gzip (RFC 1952) framing, the Adler-32 and CRC-32, and streaming both ways: an inflater and a deflater whose state is a value, fed input split anywhere and answering output in bounded pieces. Written in Cone, with no C library; the PNG reader and writer of imagedecode stand on it.

deflate#

For data in memory, one call each way: zlibCompress(data, level), gzipCompress(data, level) and rawDeflate(data, level) answer a List[u8]; zlibDecompress(data, expected) and gzipDecompress(data, expected) answer a Result[List[u8], DeflateError], the error holding a kind (a bad header, a bad code, a match reaching before the start, a cut stream, a checksum that does not match, more output than allowed, bytes after a gzip file) and a reason. The level is 0 (stored), 1 (LZ77 with the fixed Huffman code) or 2 (the same matcher with a dynamic Huffman code per block); level 2 runs about as fast as zlib's level 6 and makes streams a few per cent bigger. The checksums check themselves: a stream that fails its Adler-32, its CRC-32 or its length is refused.

For a stream that arrives in pieces, or output that must not be held whole, an Inflater (and a Deflater) is made for a Format (Raw, Zlib or Gzip), is given input with feed in pieces of any size cut anywhere, is told with finish that none is left, and answers pull(&mut out, max): at most max bytes appended to out, and a Status that says what it waits for (NeedInput, MoreOutput, Done, or for an inflater Failed). The state is all in the value, so there is no global; an inflater caps its output at 1 GiB unless setLimit says otherwise, since a small stream can expand a thousandfold. A gzip file with several members is inflated member after member and the data joined, and the header fields of the member (time, name, comment, extra field) are kept in a GzipHeader. The package's header lists each method chosen and what was left out, and its speed and size next to zlib's on six files.

mod squeeze;

import stdio use *;
import collections use *;
import deflate;

fn main() i32 {
  let text = "the quick brown fox jumps over the lazy dog, the quick brown fox jumps over the lazy dog";
  let packed = deflate.gzipCompress(text, 2);
  printInt(i64.from(text.len));
  printStr(" bytes became ");
  printInt(i64.from(packed.len()));
  printStr(" as a gzip file\n");
  match deflate.gzipDecompress(packed.view(), 0usize) {
    case let ok Ok { printStr(String.fromBytes(ok.view()).view()); printStr("\n"); }
    case let e Error { printStr(e.reason.view()); return 1i32; }
  }
  0i32;
}

prints:

88 bytes became 72 as a gzip file
the quick brown fox jumps over the lazy dog, the quick brown fox jumps over the lazy dog

Language models: weights and layers#

Package	What it is for
llmreference import llmreference use *;	The CPU side of running a Llama-family model: the weight block layouts of a GGUF file (F32, F16, Q8_0, Q4_K, Q6_K) and their decoders, half-precision conversion, seeded synthetic weights, and a CPU twin of each GPU kernel (matrix-vector and matrix-matrix products over the quantised weights, RMSNorm, rotary position embedding, softmax, attention over a KV cache, SiLU, GELU, add, multiply). Written in Cone with no GPU.
llmkernels import llmkernels use *;	The same layers as GPU compute kernels (Slang, compiled ahead of time and run through gpuwork; see The GPU), each checked against its twin. Needs a Vulkan 1.3 device.

llmreference and llmkernels#

A weight matrix is a QMatrix: rows of cols elements of one type, each row a whole number of the type's blocks, the bytes in the tensor's own layout. syntheticMatrix(&mut rng, QTYPE_Q4_K, rows, cols) makes one of seeded random blocks (so a test needs no model file), m.row(r) decodes a row to f32, and m.words() packs the bytes into the 32-bit words a GPU buffer holds. The kernels read those words by byte offset and unpack the half-precision scales with integers, so the device needs no 8-bit or 16-bit storage feature. In llmkernels, Kernels.make(&device) builds the eleven pipelines once, and each method (matvec, matmul, rmsnorm, rope, softmax, attention, kvAppend, silu, gelu, add, mul) records one dispatch into a gpuwork Job over buffers already given to it, answering whether it was recorded.

The twins work in f64 and round once, so a GPU result's difference from its twin is the GPU's own error. The tests bound each kernel by a stated tolerance (a sum of products by (n + 4) 2^-24 times the sum of its terms' absolute values; the others by units of 2^-24 that the packages' headers derive) and, run with LLMKERNELS_REPORT set, print the worst error seen: on the RTX 4060 Laptop GPU the sums are within 4 units of a bound of up to 2300. Both packages' headers list the method choices and what was left out (flash attention, several sequences at once, f16 activations, subgroup and 16-bit fast paths, the other quantisation types), the block layouts, and the one check still to do: the layouts are written from knowledge of ggml and have not been compared with a real model file.

Model files: GGUF#

Package

What it is for

gguf import gguf;

GGUF files, the container that llama.cpp's language models come in, read and written over a byte slice: the header, the metadata (every value type, arrays and arrays of arrays, so a tokenizer's vocabulary and merges), the tensor table, and a tensor's elements read as f32 for F32, F16, BF16, Q4_0, Q8_0, Q4_K and Q6_K. A Writer makes synthetic models for tests. Written in Cone, with no C library; groundwork for running small open models locally, it runs none.

gguf#

parse(bytes) answers a Result[Model, GgufError]. A Model holds the metadata in file order (text(key), uint(key), int(key), float(key), flag(key) and list(key) answer an Option of that kind of value; get(key) the whole Value) and a table of Tensors (name, kind, dims, offset, elements, bytes), found by find(name). It does not hold the tensors' bytes: they stay in the bytes that were parsed (a buffer read from a file, or later a mapped file), and tensorBytes(file, t) is a view of one. tensorToF32, rowToF32 and dequantize read tensors as f32. Every field of the file is checked, and a fault is a GgufError with a kind, the offset of the byte at fault and a reason: a wrong magic, a big-endian file, a version other than 2 and 3, a count the file cannot hold, a string or array past the end, a tensor whose dimensions overflow or are not whole blocks, a tensor outside the file or off its alignment. The package's header lists the block layouts, each method chosen and what was left out (other quantised types are refused as a whole file), and says which parts of the Q4_K and Q6_K layouts were checked only against values worked out by hand.

mod toy;

import stdio use *;
import collections use *;
import gguf;

fn main() i32 {
  var w = gguf.Writer.make();
  w.putText("general.architecture", "toy");
  w.addValues("eye", gguf.TensorType.F32, &[2u64, 2u64], &[1.0, 0.0, 0.0, 1.0]);
  var file = List[u8].empty();
  match w.finish() {
    case var ok Ok { file <=> ok; }
    case is Error { return 1i32; }
  }
  match gguf.parse(file.view()) {
    case let model Ok {
      each t in model.tensors {
        printStr(t.name.view());
        printStr(" ");
        printStr(t.kind.name());
        printStr(" ");
        printInt(i64.from(t.bytes));
        printStr("\n");
      }
    }
    case is Error { return 1i32; }
  }
  0i32;
}

prints eye F32 16.

Language-model tokens#

Package

What it is for

bpetokenize import bpetokenize;

Byte-level byte-pair-encoding tokenization, the tokenizer of the GPT-2 and Llama 3 families: a text is cut into words, each word is merged by rank into a vocabulary's tokens, and the tokens are numbered; decoding numbers back to the bytes. The vocabulary and the merge list come in as plain arrays of strings, so it depends on no file format. Written in Cone, with no C library, and any bytes (UTF-8 or not) encode and decode as they are.

bpetokenize#

Tokenizer.build(tokens, merges, special, preTokenizer) answers a Result[Tokenizer, BuildError]. tokens is the vocabulary in id order, each token spelt as a byte-level vocabulary spells it (toByteLevel(bytes) makes that spelling: GPT-2's table, in which a space is Ġ); merges are the rules, best first, each the text left right (or MergePair values with buildPairs); special are the texts of the tokens matched whole in an input. The 256 single-byte tokens must be in the vocabulary, and every merge must name tokens that are, or the build is refused with a BuildError that says which. encode(bytes) answers a List[u32] of ids, encodePlain the same with no special token recognised, and encodeWith(bytes, parseSpecial, &mut out) appends to a list. decode(ids) answers the bytes (an id outside the vocabulary is an error, with its place); decodeInto can leave the special tokens out. The cut made before merging is a PreTokenizer: Gpt2, Llama3 (cl100k's expression, which GPT-4 and Llama 3, 3.1 and 3.2 use), Qwen2 or Whole (no cut). It is hand-written, with no regular-expression engine, and matches the published expressions on the scripts it lists (Latin, Greek, Cyrillic, Armenian, Hebrew, Arabic, Devanagari, Thai, Georgian, Hangul, Hiragana, Katakana and the CJK ideographs); another script is cut as symbols. Merging is rank-priority merging with a heap, O(n log n) in the length of a word. Not covered: SentencePiece and Unigram tokenizers, byte-fallback vocabularies, normalisers, other families' cuts (GPT-4o, DeepSeek, Falcon, StarCoder), and a trainer. The package's header lists each method chosen and what was left out.

mod tokens;

import stdio use *;
import collections use *;
import bpetokenize;

fn main() i32 {
  var tokens = List[String].empty();       // the 256 byte tokens, then the merged ones
  var b = 0u32;
  while b < 256u32 {
    var one = List[u8].empty();
    one.push(u8.from(b));
    tokens.push(bpetokenize.toByteLevel(one.view()));
    b = b + 1u32;
  }
  tokens.push("he");                       // 256
  tokens.push("ll");                       // 257
  tokens.push("hell");                     // 258
  tokens.push("<|end|>");                  // 259, a special token
  var merges = List[String].empty();
  merges.push("h e");
  merges.push("l l");
  merges.push("he ll");
  var special = List[String].empty();
  special.push("<|end|>");
  match bpetokenize.Tokenizer.build(tokens.view(), merges.view(), special.view(),
                                    bpetokenize.PreTokenizer.Gpt2) {
    case let tok Ok {
      let ids = tok.encode("hello<|end|>");
      var i = 0usize;
      while i < ids.len() {
        printInt(i64.from(ids.view()[i]));
        printStr(" ");
        i = i + 1usize;
      }
      match tok.decode(ids.view()) {
        case let bytes Ok { printStr(bytes.view() as &str); }
        case let e Error { return 1i32; }
      }
      printStr("\n");
    }
    case let e Error { return 1i32; }
  }
  0i32;
}

prints 258 111 259 hello<|end|>: hell is one token, the o is the byte 111, and the special token is matched whole.

Testing#

Package	What it is for
testing import testing use *;	Checks for a package's tests (expectInt, require, done), run by congo test; see Testing.

Where they run#

These run on Windows and on Linux, each over the system's own calls (chosen by isWindows(), so only the target's are generated): posix, fs, env (over posix) and process; iobuf (its blocks come from VirtualAlloc on Windows and mmap on Linux), iocore (over an I/O completion port on Windows, epoll on Linux), aio, http1, httpconn, httpclient and tls (over OpenSSL on both, which judges a server's certificate with the system's own verifier on Windows and with its own check against the system's roots on Linux, where libssl-dev must be installed); and path reads the paths of the system it is built for: on Windows drive letters and shares, and both / and \ as separators; on Linux only /, with a root and no volume.

What is Windows-only beneath fs, path and env is posix's: on Windows its POSIX names map onto the Windows C runtime, and its folder listing (opendir, readdir) and lstat are written over the runtime's _findfirst64; on Linux they are the C library's. process has an interface that belongs to no one system, but its insides are written over kernel32 (CreateProcessW, pipes, a job object) on Windows and over posix_spawn on Linux, since neither ISO C nor POSIX alone can do it on both.

random asks the OS for one thing, entropy to seed from (rand_s on Windows, getentropy elsewhere, as the hash seed is drawn). Its numbers from a seed are the same on both, and have been run on Windows only.

cryptohash asks the OS for nothing; its tests pass on Windows and on Linux (Ubuntu 24.04), with the same digests and tags.

llvmc links LLVM's C library on both (LLVM-C.dll from C:\LLVM\23 on Windows, libLLVM from /usr/lib/llvm-23 on Linux); its tests pass on both, with the same IR text and answers.

deflate calls nothing of the system's; its tests, and those of imagedecode over it, have been run on both, and write the same bytes.

llmreference is Cone alone and has no call of the system's; its tests pass on Windows and on Linux (Ubuntu 24.04), and its example prints the same numbers on both. llmkernels needs a GPU and the GPU packages, which run on Windows only for now: it compiles on Linux and its tests (which run on a real GPU) do not.

gguf calls nothing of the system's; its tests pass on Windows and on Linux (Ubuntu 24.04), and write and read the same bytes.

bpetokenize calls nothing of the system's; its tests pass on Windows and on Linux (Ubuntu 24.04), with the same token ids.

The others use nothing that belongs to Windows alone. Of these, sync, thread, actors and time are the ones that have been run on Linux so far, with core and a program that only prints.

Testing is on its own page, awaiting I/O from an actor is in I/O from an actor, and the platform notes are gathered in Platforms and targets.