HTTP, TLS and URLs
On this page
- URLs: the url package
- Reading and writing a URL
- Resolving a reference
- Refusals
- Encoding a component
- Messages: the http package
- Request and response heads
- Header fields
- Methods, versions and statuses
- Bodies and errors
- http1: HTTP/1.1 with no I/O of its own
- Serving and fetching over sockets: the httpconn package
- The driver
- A server
- Streaming a body
- Requests on a client connection
- Where events go: sinks
- Events
- Cancelling and deadlines
- Failures
- Configuration
- Layers
- Fetching a URL: the httpclient package
- One call: fetchWhole and getWhole
- A fetch's events
- The pool
- Order: intents and scores
- Retries and redirects
- Failures and configuration
- TLS: the tls package
- HTTPS: a server, and a client that trusts it
- Who verifies the server
- Without a socket: sessions
- Streams
- Errors and diagnostics
Note: Not built: HTTP/2 and HTTP/3; proxies, cookies, caches, authentication and compression (a client sends no Accept-Encoding and leaves a Content-Encoding to the program); Upgrade and CONNECT; a request body streamed by httpclient (it sends one whole); internationalized host names, IPv6 zones and the WHATWG URL standard's departures from RFC 3986; an actor as the place a driver's or a client's events go; and a TLS back end other than OpenSSL.
Six library packages speak HTTP and its parts:
- url: a URL read from text, resolved against another, and written back.
- http: HTTP's messages as data — methods, statuses, header fields, request and response heads, what is known of a body's length, and errors — the same for every version of the protocol.
- http1: HTTP/1.1 as a state machine with no I/O of its own. It is described briefly below; most programs never call it.
- httpconn: HTTP/1.1 over real sockets on an I/O loop, as a client and as a server. This is where a program that serves HTTP lives.
- httpclient: fetch a URL, http:// or https://, in one call; or many at once, with a pool of connections per server, retries, redirects, and a queue that orders what is fetched.
- tls: TLS 1.2 and 1.3 over OpenSSL, client and server. It does no I/O of its own, and the operating system decides whether a server's certificate can be trusted.
A program reaches them with import url use *;, import http use *;, import httpconn use *; and so on. httpconn, httpclient and tls run on Windows only, for now: httpconn is written over iocore's completion port and Winsock (I/O and Concurrency and Timers, deadlines and cancelling in I/O describe the loop and its deadlines), and tls asks Windows' own certificate verifier (crypt32) whether a server is who it says it is. httpclient uses both. url, http and http1 call nothing of the operating system's. Congo copies OpenSSL's two libraries beside any program that imports tls or httpclient. Where a program imports both httpconn and httpclient, each has a Config, a Sink, an Event and so on, so it writes import httpconn; and says httpconn.Driver and httpconn.Event.
Every example below is a complete program that was compiled and run, and its output is shown. The ones with a client and a server run both in the one program, over 127.0.0.1, so they touch no other machine.
URLs: the url package#
A Url is a URL as its parts: a scheme, an authority (user information, a host and a port), a path, a query and a fragment, each kept as the text it is on the wire (percent-encoded). It follows RFC 3986 and knows nothing of HTTP.
Reading and writing a URL#
Url.parse(text) reads an absolute URL and gives a Result[Url, UrlError]. What is merely untidy is made tidy as a browser would: the scheme and the host name are lower-cased; spaces at either end are dropped; in the path, query and fragment a space, a quote, < > \ ^ ` { | } and every byte from 0x80 up are percent-encoded (a % already there is left as it is); the path's . and .. segments are removed; and an http or https URL with no path gets /. The parts are public fields: scheme, userinfo, host (an IPv6 address without its brackets), hostKind (HostKind.Empty, Name, Ipv4 or Ipv6), port and hasPort, path, query and hasQuery, fragment and hasFragment, and hasAuthority. The methods write it, or ask about it:
- text(): the URL as text, the port only when it is not the scheme's default (80 for http, 443 for https).
- origin(): scheme://host:port, the port always written. It is what a client pools connections under.
- authority(): what HTTP writes as Host.
- target(): the request target, the path (at least /) and the query, never the fragment.
- effectivePort(), isHttp(), isSecure(), sameOrigin(&other) and copy(); defaultPort(scheme) is the free function behind the first.
Resolving a reference#
u.resolve(ref) reads ref as RFC 3986 does, against u as its base: it may be absolute, //host/p, /p, p, ../p, ?q, #f or empty. This is what a Location header needs. resolveRef(base, ref) is the same as a function.
Refusals#
A URL that cannot be read gives a UrlError: at, the byte where it was found, why, and message(), which joins them. A scheme that is not a letter followed by letters, digits, + - ., a port that is not decimal or is past 65535, a host with a character no host may hold, a control character, an http or https URL with no host, and an IPv6 address that is not one are refused. Refused by name, because they are not read yet: a host name with bytes past ASCII, a percent-encoded host name, an IPv6 zone (%25eth0) and [v1.x].
Encoding a component#
encodeComponent(text) percent-encodes every byte that is not a letter, a digit or one of - . _ ~: it is for a value put into a query or a path segment. decode(text) undoes percent-encoding and gives an Option[String], None when a % is not followed by two hex digits. removeDotSegments(path) is also public.
mod urls;
import stdio use *;
import collections use String;
import url use *;
fn show(label &str, s &str) {
printStr(label);
printStr(" ");
printStr(s);
printStr("\n");
}
fn main() {
// Read a URL: it is tidied, and every way of writing it back is a method
match Url.parse("HTTPS://Example.com:443/a/./b/../c?x=1#top") {
case imm u Ok {
show("text ", u.text().view());
show("origin ", u.origin().view());
show("authority", u.authority().view());
show("target ", u.target().view());
show("host ", u.host.view());
show("path ", u.path.view());
show("query ", u.query.view());
show("fragment ", u.fragment.view());
printStr("port ");
printUInt(u64.from(u.effectivePort()));
printStr(if u.hasPort {" written\n";} else {" default\n";});
// Resolve a reference against it
match u.resolve("../d?y") {
case imm t Ok {show("resolved ", t.text().view());}
case imm e Error {show("error ", e.message().view());}
}
match u.resolve("//other.example/x") {
case imm t Ok {show("resolved ", t.text().view());}
case imm e Error {show("error ", e.message().view());}
}
}
case imm e Error {show("error ", e.message().view());}
}
// A refusal says where and why
match Url.parse("http://example.com:99999/") {
case imm u Ok {show("text ", u.text().view());}
case imm e Error {
show("error ", e.message().view());
printStr("at byte ");
printUInt(u64.from(e.at));
printStr("\n");
}
}
// The same origin, however the default port is written
match Url.parse("http://a:80/x") {
case imm a Ok {
match Url.parse("http://A/y") {
case imm b Ok {printStr(if a.sameOrigin(&b) {"same origin\n";} else {"other origin\n";});}
case imm e Error {}
}
}
case imm e Error {}
}
// A value for a query, encoded; and decoded again
imm enc = encodeComponent("a b&c=d/é");
show("encoded ", enc.view());
match decode(enc.view()) {
case imm d Some {show("decoded ", d.view());}
case is None[String] {printStr("not decodable\n");}
}
match decode("%zz") {
case imm d Some {show("decoded ", d.view());}
case is None[String] {printStr("%zz is not decodable\n");}
}
}
Output:
text https://example.com/a/c?x=1#top origin https://example.com:443 authority example.com target /a/c?x=1 host example.com path /a/c query x=1 fragment top port 443 written resolved https://example.com/d?y resolved https://other.example/x error url: a port is 0 to 65535 (at 19) at byte 19 same origin encoded a%20b%26c%3Dd%2F%C3%A9 decoded a b&c=d/é %zz is not decodable
Messages: the http package#
The http package holds what an HTTP exchange is made of, as plain data. Nothing in it touches a socket, and nothing in it is about one version of the protocol: HTTP/1.1 writes a request's head as lines of text, and a later version would write the same RequestHead as binary frames, so the code above a core never asks which carried a message.
Request and response heads#
RequestHead.make(method, target, authority) makes a request's head: the method (any token, "GET" or an extension's), the target (the path and query, /a/b?c) and the authority (example.com:8080, which HTTP/1.1 writes as Host). RequestHead.get(target, authority) is the common case. The fields are public: method (a Method), target, authority, scheme (only HTTP/2 and HTTP/3 send it), version and headers; methodName() is the method's text, an extension's spelling kept. ResponseHead.make(status) makes a response's head, with a status (a u16), a version and headers. Host is never one of a request's fields: a core moves it into authority as it reads, and writes it from there. Neither are the framing fields, Content-Length and Transfer-Encoding: a core reads and writes those, from a BodyLength.
Header fields#
A Headers is a message's header (or trailer) fields in the order they came or were added, a name repeated as often as it was. Names compare ignoring ASCII case and are kept as written. Nothing is joined unless asked, because Set-Cookie cannot be.
- h.add(name, value) adds a field after the others, and says false, adding nothing, when the name is not a token or the value, its spaces and tabs at either end dropped, holds a CR, an LF, a NUL or a control character. So a Headers holds only fields that any version may send. addUInt(name, n) adds a number; set replaces every field of the name; remove removes them and says how many.
- get(name) is an Option[&Array[u8]], the first value; count, has, joined(name) (the values joined by a comma and a space) and hasToken(name, token), which asks whether a comma-separated list field such as Connection has a member, ignoring case.
- len(), nameAt(i) and valueAt(i) walk the fields; copy(), clear() and isEmpty().
A name is text, a &str; a value is bytes, a &Array[u8], since it may hold bytes that are not UTF-8. checkFieldName and checkFieldValue say why one is refused ("" when it is not), and isToken and trimOws are there too.
Methods, versions and statuses#
Method is Get, Head, Post, Put, Delete, Connect, Options, Trace, Patch or Other. methodOf(token) reads one, case-sensitively (get is Other), and methodToken(m) writes it. isSafe(m) and isIdempotent(m) are RFC 9110's: a request may be repeated automatically only when its method is idempotent. Version is Http10, Http11, Http2 or Http3.
A status is its three-digit code. reasonPhrase(code) is the registered phrase ("" for any other code); isInformational, isSuccess, isRedirect, isClientError and isServerError sort codes by their hundreds; and responseHasBody(method, code) is the rule every version shares: a response to HEAD, and a 1xx, 204 or 304, has no body, whatever its fields say.
Bodies and errors#
A BodyLength is what is known of a body: BodyLength.empty() (none), BodyLength.sized(n) (exactly n bytes) or BodyLength.unsized() (until the stream ends: HTTP/1.1's chunked coding or the connection's close). Its kind and length are public, and isEmpty(), isSized() and isUnsized() ask. The bytes themselves travel separately, in buffers.
An HttpError says why an exchange or a connection failed: a kind — Malformed (the peer broke the protocol), TooLarge (a configured bound was passed), Unsupported, Closed (the connection ended mid-message) or TimedOut — the status a server would answer it with (0 for none), noResponse (no byte of a response had arrived: with an idempotent method, the time a client may retry), and detail, one line. message() joins the kind and the detail.
mod msgs;
import stdio use *;
import collections use String;
import http use *;
fn main() {
// A request: method, target, authority; fields beside them
mut req = RequestHead.make("GET", "/tiles/7/12.bin", "example.com");
req.headers.add("Accept", "application/octet-stream");
req.headers.add("Range", "bytes=0-65535");
printStr(req.methodName());
printStr(" ");
printStr(req.target.view());
printStr(" to ");
printStr(req.authority.view());
printStr(", ");
printUInt(u64.from(req.headers.len()));
printStr(" fields\n");
// Names compare ignoring case; the first value comes back
match req.headers.get("RANGE") {
case imm v Some {
printStr("Range is ");
printStr(v as &str);
printStr("\n");
}
case is None[&Array[u8]] {printStr("no Range\n");}
}
// A field that is not allowed is refused, and nothing is added
imm ok = req.headers.add("Bad Name", "x");
printStr(if ok {"added\n";} else {"refused: ";});
printStr(checkFieldName("Bad Name"));
printStr("\n");
printStr(if req.headers.add("X-Note", "two\nlines") {"added\n";} else {"refused a value with a newline\n";});
// A repeated field is kept as often as it came; joined only when asked
mut h = Headers.empty();
h.add("Set-Cookie", "a=1");
h.add("Set-Cookie", "b=2");
h.add("Connection", "keep-alive, Upgrade");
printUInt(u64.from(h.count("set-cookie")));
printStr(" cookies; joined: ");
printStr(h.joined("Set-Cookie").view());
printStr("\n");
printStr(if h.hasToken("connection", "upgrade") {"Connection lists upgrade\n";} else {"no upgrade\n";});
h.set("Set-Cookie", "c=3");
printUInt(u64.from(h.count("Set-Cookie")));
printStr(" cookie after set\n");
// Methods, statuses, bodies
imm m = methodOf("PUT");
printStr(if isIdempotent(m) and not isSafe(m) {"PUT is idempotent, not safe\n";} else {"?\n";});
printStr(if methodOf("get") == Method.Other {"'get' is not GET\n";} else {"?\n";});
printStr(reasonPhrase(404u16));
printStr("; ");
printStr(if isRedirect(302u16) {"302 is a redirect; ";} else {"?; ";});
printStr(if responseHasBody(Method.Head, 200u16) {"HEAD has a body\n";} else {"a response to HEAD has no body\n";});
mut resp = ResponseHead.make(200u16);
resp.headers.addUInt("X-Count", 42u64);
match resp.headers.get("x-count") {
case imm v Some {printStr(v as &str);}
case is None[&Array[u8]] {}
}
printStr("\n");
imm body = BodyLength.sized(1024u64);
printStr(if body.isSized() and not body.isUnsized() {"sized\n";} else {"?\n";});
imm err = HttpError.make(ErrorKind.Malformed, 400u16, "a header field's name has a space before its colon");
printStr(err.message().view());
printStr("\n");
}
Output:
GET /tiles/7/12.bin to example.com, 2 fields Range is bytes=0-65535 refused: a field's name has a space or a tab in it, or before its colon refused a value with a newline 2 cookies; joined: a=1, b=2 Connection lists upgrade 1 cookie after set PUT is idempotent, not safe 'get' is not GET Not Found; 302 is a redirect; a response to HEAD has no body 42 sized malformed: a header field's name has a space before its colon
http1: HTTP/1.1 with no I/O of its own#
A http1.Connection is the protocol state of one TCP connection, as plain data, in either role: Connection.client(config, pool, now) sends requests and reads responses, Connection.server(config, pool, now) reads requests and sends responses. It owns no socket, reads no clock and starts no thread. The program that does own the socket (for a socket on a loop, httpconn is that program) calls it four ways:
- handleInput(buf, now): the bytes that arrived, in an owned buffer; an empty buffer is the end of the peer's stream.
- pollTransmit(now): the next buffer to send, empty when there is none.
- pollEvent(): what happened. A server's request arrives as Request (the head and what is known of the body), any number of Data (each an owned buffer), then End (with the trailer fields); a client's response as Response, Data, End, with a 1xx first as Interim. The others are NeedData, Paused (waiting for the program), Failed (an HttpError; every later event is Closed) and Closed.
- pollTimeout() and handleTimeout(now): the one deadline the program must wake for, and the wake.
The program sends with sendRequest(head, body) or sendResponse(head, body) (and sendInterim, sendContinue), then sendData(buf) and finish() or finishWith(trailers) for a body. The memory it holds is bounded: readCredit() says how many bytes it will take now, body bytes handed out are on loan until the program calls release(n), and sendData hands a buffer back when too much waits to be sent. A Config sets the bounds (head, line, field count, chunk line, buffering, window) and the timeouts (head, body, idle, Expect: 100-continue); each limit passed is a visible Failed. A server's failure sends the matching status first (400, 408, 414, 431, 501, 503 or 505) with Connection: close. A message that two readers could frame differently (both Content-Length and Transfer-Encoding, a space before a colon, a second Host) is refused, never repaired.
The listing feeds a server connection a request, and shows what it says and what it wants sent. Each line of the wire text ends in a carriage return and a line feed.
mod http1demo;
import stdio use *;
import collections use String;
import time use Instant;
import iobuf use *;
import http use *;
import http1 use *;
// Everything the connection has to say, until it needs something from the program
fn events(c &mut Connection) {
while true {
match c.pollEvent() {
case mut e Request {
printStr("event Request ");
printStr(e.head.methodName());
printStr(" ");
printStr(e.head.target.view());
printStr(" for ");
printStr(e.head.authority.view());
printStr("\n");
}
case mut d Data {
printStr("event Data ");
printUInt(u64.from(d.buf.len()));
printStr(" bytes\n");
c.release(d.buf.len());
}
case is End {printStr("event End\n");}
case mut f Failed {
printStr("event Failed ");
printStr(f.error.message().view());
printStr("\n");
}
case is Closed {
printStr("event Closed\n");
return;
}
case is NeedData {
printStr("event NeedData\n");
return;
}
else {return;}
}
}
}
// The bytes the connection wants sent, written out
fn wire(c &mut Connection, at Instant) {
while true {
imm b = c.pollTransmit(at);
if b.len() == 0usize {break;}
printStr(b.view() as &str);
}
}
fn main() {
imm pool = Pool.make();
imm now = Instant.now();
mut server = Connection.server(Config.make(), pool, now);
// Bytes arrive, as a socket would bring them
mut arrived = pool.mustTake(256usize);
arrived.append("POST /notes HTTP/1.1\r\nHost: example.com\r\nContent-Length: 5\r\n\r\nhello");
server.handleInput(arrived, now);
events(&mut server);
// The program answers; the connection says what to put on the wire
mut head = ResponseHead.make(201u16);
head.headers.add("Content-Type", "text/plain");
server.sendResponse(head, BodyLength.sized(3u64));
mut body = pool.mustTake(16usize);
body.append("ok\n");
server.sendData(body);
server.finish();
printStr("--- to send:\n");
wire(&mut server, now);
printStr("---\n");
// A request it will not accept is refused, and the connection ends
mut bad = Connection.server(Config.make(), pool, now);
mut junk = pool.mustTake(256usize);
junk.append("GET / HTTP/1.1\r\nHost: a\r\nHost: b\r\n\r\n");
bad.handleInput(junk, now);
events(&mut bad);
printStr("--- to send:\n");
wire(&mut bad, now);
printStr("---\n");
}
Output:
event Request POST /notes for example.com event Data 5 bytes event End --- to send: HTTP/1.1 201 Created Content-Type: text/plain Content-Length: 3 ok --- event Failed malformed: the request has more than one Host event Closed --- to send: HTTP/1.1 400 Bad Request Connection: close Content-Length: 0 ---
Serving and fetching over sockets: the httpconn package#
httpconn joins http1 to sockets on an iocore Loop, in both roles. A client connection looks its host up, connects by a deadline, sends requests and streams their responses back; a server listens, accepts, and runs one connection for each socket it accepts, streaming each request to the program and each response back out. One Driver is all of it for one loop.
The driver#
Driver.make(config, pool) makes a driver. The pool is an iobuf Pool: every body travels in an IoBuf taken from it (pool.mustTake(n), then buf.append(bytes)), and a buffer a Data event gives the program is the program's. A driver is driven by the thread that owns its loop. The simplest way is drv.poll(&mut lp, until, &mut events): it waits in the loop for at most until (iocore.forever() for no limit; not at all while events are ready) and appends to events, a List[Event], what happened; the program handles them and clears the list. It never waits anywhere else. A program that also submits operations of its own to the loop polls the loop itself, gives each completion to drv.complete(lp, &mut c) (which says whether it was the driver's, by drv.owns(token)), then calls drv.pump(lp, &mut events); the loop's poll may wait no later than drv.wakeAt().
A server#
A server is a driver that listens. drv.listen(&mut lp, addr, sink) gives a Result[ListenerId, IoError]; SocketAddr.localhost(port) is 127.0.0.1 only (port 0 asks for any free port, and drv.localAddr(&lp, listener) says which), and SocketAddr.v4(a, b, c, d, port) makes any other. Every connection it accepts, and every request on them, is told to the sink as events: Accepted for each connection, then for each request Request (the head: e.request, a RequestHead; and e.body, what is known of the body), Data for each piece of its body, and End when it is whole (e.trailers). Each carries e.req, the request's ReqId, and e.conn. The program answers a request with drv.respond(req, head, bodyLength), then drv.sendData(req, buf) for each piece and drv.finish(req) (or drv.finishWith(req, trailers)). The request's last event is Done once the response has gone to the kernel whole, or Failed if the connection ended first. A server answers one request at a time on a connection; a pipelined request is read once the answer before it is whole. A server wanting a body only on its say-so sees Expect: 100-continue in the request's head and calls drv.sendContinue(req); drv.sendInterim(req, head) sends another 1xx. The client that sent the field may queue its body at once, but the bytes go only when the 100 comes (it sees an Interim event), or when the expect timeout, one second unless Config says otherwise, passes without one.
The program below is a whole server and a client to try it with, on one loop. The server keeps a Pending for each request it is reading, since a request's events can interleave with another connection's; at End it takes the Pending out and routes it; at a final event it forgets one it never answered. (The body is copied out, or the Pending moved in whole, before the answer is started, since answering changes the driver and the compiler refuses to lend a string across that.) The client asks three things, one after another on one kept-alive connection, and prints what comes back.
mod server;
import stdio use *;
import collections use List, String;
import time use Instant, Duration;
import iobuf use *;
import iocore use Loop, SocketAddr, forever;
import http use *;
import httpconn use *;
const tokServer = 100u64;
const tokConn = 1u64;
const tokReq = 2u64;
// A request being read: what its head said, and its body so far
struct Pending {
pub req ReqId;
pub method Method;
pub target String;
pub body String;
}
fn find(pending &List[Pending], req ReqId) usize {
mut k = 0usize;
while k < pending.len() {
if pending[k].req == req {return k;}
k = k + 1usize;
}
pending.len();
}
// Answer with a body that is a whole text
fn reply(drv &mut Driver, pool &Pool, req ReqId, status u16, text &Array[u8]) {
mut head = ResponseHead.make(status);
head.headers.add("Content-Type", "text/plain");
drv.respond(req, head, BodyLength.sized(u64.from(text.len)));
mut buf = pool.mustTake(text.len + 1usize);
buf.append(text);
drv.sendData(req, buf);
drv.finish(req);
}
// The routes: a request read whole
fn route(drv &mut Driver, pool &Pool, p Pending) {
if p.method == Method.Get and p.target.view() == "/" {
reply(drv, pool, p.req, 200u16, "Hello from Cone.\n");
} elif p.method == Method.Post and p.target.view() == "/echo" {
reply(drv, pool, p.req, 200u16, p.body.view());
} else {
reply(drv, pool, p.req, 404u16, "No such page.\n");
}
}
// What the server does with one event of its own
fn serve(drv &mut Driver, pool &Pool, pending &mut List[Pending], e &mut Event) {
if e.kind == EventKind.Request {
pending.push(new Pending(e.req, e.request.method, e.request.target.copy(), String.empty()));
} elif e.kind == EventKind.Data {
imm k = find(pending, e.req);
if k < pending.len() {(&mut pending[k]).body.appendBytes(e.buf.view());}
drv.release(e.req, e.buf.len());
} elif e.kind == EventKind.End {
imm k = find(pending, e.req);
if k < pending.len() {
pending.swap(k, pending.len() - 1usize);
match pending.pop() {
case imm p Some {route(drv, pool, p);}
case is None[Pending] {}
}
}
} elif e.isTerminal() {
// Done, or the connection failed first: either way, the request is over
imm k = find(pending, e.req);
if k < pending.len() {
pending.swap(k, pending.len() - 1usize);
pending.pop();
}
}
}
// The client's requests, one after another on one kept-alive connection
fn ask(drv &mut Driver, lp &mut Loop, conn ConnId, step u32) {
if step == 0u32 {
drv.request(lp, conn, RequestHead.get("/", "localhost"), BodyLength.empty(), Sink.Here(tokReq));
} elif step == 1u32 {
drv.request(lp, conn, RequestHead.get("/nothing", "localhost"), BodyLength.empty(), Sink.Here(tokReq));
} else {
drv.request(lp, conn, RequestHead.make("POST", "/echo", "localhost"), BodyLength.sized(12u64),
Sink.Here(tokReq));
}
}
fn main() {
imm pool = Pool.make();
mut lp = Loop.mustMake();
mut drv = Driver.make(Config.make(), pool);
// Listen on any free port of this machine
mut port = 0u16;
match drv.listen(&mut lp, SocketAddr.localhost(0u16), Sink.Here(tokServer)) {
case imm l Ok {
match drv.localAddr(&lp, l) {
case imm a Ok {port = a.port();}
case imm e Error {}
}
}
case imm e Error {printStr(e.message().view());}
}
// A client on the same loop asks it three things
imm conn = drv.connect(&mut lp, "127.0.0.1", port, Sink.Here(tokConn));
mut step = 0u32;
ask(&mut drv, &mut lp, conn, step);
mut pending = List[Pending].empty();
mut events = List[Event].empty();
imm giveUp = Instant.now() + Duration.fromSeconds(10i64);
while step < 3u32 and Instant.now() < giveUp {
drv.poll(&mut lp, giveUp, &mut events);
mut k = 0usize;
while k < events.len() {
imm e = &mut events[k];
k = k + 1usize;
if e.token == tokServer {
serve(&mut drv, &pool, &mut pending, e);
} elif e.token == tokReq {
if e.kind == EventKind.Response {
printUInt(u64.from(e.response.status));
printStr(" ");
printStr(reasonPhrase(e.response.status));
printStr(": ");
} elif e.kind == EventKind.Data {
printStr(e.buf.view() as &str);
drv.release(e.req, e.buf.len());
} elif e.kind == EventKind.Writable {
mut buf = pool.mustTake(16usize);
buf.append("hello, Cone\n");
drv.sendData(e.req, buf);
drv.finish(e.req);
} elif e.isTerminal() {
step = step + 1u32;
if step < 3u32 {ask(&mut drv, &mut lp, conn, step);}
}
}
}
events.clear();
}
drv.abortAll(&mut lp);
}
Output:
200 OK: Hello from Cone. 404 Not Found: No such page. 200 OK: hello, Cone
Streaming a body#
A body is never gathered whole, in either direction. A server sends one a piece at a time with sendData, which takes the buffer (None comes back) or hands it back, Some(buf), when the connection already has a watermark's worth waiting to go out. A Writable event then says there is room again. The same holds for a client sending a request body. In the other direction, a body the program receives is on loan: drv.release(req, n) says it is done with n bytes, and once bodyWindow bytes (256 KiB) are out unreleased the driver stops reading and TCP's own flow control holds the peer. The server below sends 1 MiB in 64 KiB pieces, stopping when a piece is refused, and the client checks every byte. Its main loop is the previous one's, with these two parts changed:
// A response going out: how much of it has been sent
struct Sending {
pub req ReqId;
pub sent usize;
pub active bool;
}
// Send as many pieces as the connection takes now; finish when all are gone
fn sendPieces(drv &mut Driver, pool &Pool, s &mut Sending) {
while s.sent < total {
mut buf = pool.mustTake(piece);
imm room = buf.spare();
mut i = 0usize;
while i < piece {
room[i] = u8.from((s.sent + i) % 251usize);
i = i + 1usize;
}
buf.setLen(piece);
match drv.sendData(s.req, buf) {
case is None[IoBuf] {s.sent = s.sent + piece;} // taken
else {return;} // refused: wait for Writable
}
}
drv.finish(s.req);
s.active = false;
}
// ... in the loop over the events, for the server's:
if e.kind == EventKind.End {
// The request is read; answer with a body of known length, sent as the connection takes it
drv.respond(e.req, ResponseHead.make(200u16), BodyLength.sized(u64.from(total)));
out.req = e.req;
out.active = true;
sendPieces(&mut drv, &pool, &mut out);
} elif e.kind == EventKind.Writable and out.active {
writable = writable + 1u32;
sendPieces(&mut drv, &pool, &mut out);
}
The program prints (total is 1048576 and piece 65536):
1048576 bytes received, 0 wrong; the server was told to wait at least once
An unknown length is sent with BodyLength.unsized(), which HTTP/1.1 writes in the chunked coding, and finishWith(req, trailers) ends it with trailer fields.
Requests on a client connection#
drv.connect(&mut lp, host, port, sink) makes a client connection and gives its ConnId at once. The host is looked up on the loop's name-lookup lane unless it is an IPv4 address written as one, each address it gives is tried in turn, and the whole is bounded by connectTimeout (10 s). drv.connectAddr(&mut lp, addr, sink) skips the lookup. The sink is told Connected first (e.peer is the other end) and, last, exactly one Closed, e.clean when it ended as HTTP meant it to (a kept-alive connection's end, an idle timeout, Connection: close) and e.failure saying why when it did not.
drv.request(&mut lp, conn, head, bodyLength, sink) asks for an exchange and gives a ReqId. Its events go to its own sink: Response (e.response, a ResponseHead, preceded by an Interim for each 1xx), Data, End (with trailer fields), and then exactly one of the four final answers, Done (e.timing: when it was asked for, put on the wire, first answered and ended), Failed, Cancelled or TimedOut, and nothing after it. A request with a body sends it with sendData and finish once Writable says it is under way. A request that cannot be made — the connection closed or a server's, its deadline past, its group cancelled — is answered at once and never sent. A client connection carries one request at a time and is kept for the next; drv.isIdle(conn) says it is free. At most maxWaiting (1) requests may wait on a connection that is busy or still connecting; a request past that is Failed at Stage.Limit. (httpclient, below, does the pooling a program that makes many requests wants.)
The same driver can be a server and a client at once, as the programs above are. drv.close(lp, conn) lets what was asked finish and closes politely; drv.abort(lp, conn) resets it now, every request on it answered Cancelled; drv.abortAll(lp) aborts every connection and stops every listener; drv.stopListening(lp, l) stops one listener and leaves its connections. isOpen, peer and layerInfo ask about a connection.
Where events go: sinks#
Every connection, listener and request is given a Sink and the token it carries back, a u64 the driver never reads. The program tells its events apart by token, as the programs above do. Sink.Here(token) delivers to the thread that polls, appended to events. Sink.Chan(token, tx) sends on a channel's Sender[Event] to another thread, never waiting: when the channel is full the event is held, the connection's reading stops behind it, and it is tried again at every pump, so the channel's bound bounds what the connection holds. Events keep their order within one sink, not across sinks; a program drives the driver until drv.held() is 0 before it stops. A drv.stats() counts connections, requests by final answer, events and bytes.
Events#
An Event is a struct a program reads in place: its kind (an EventKind: Connected, Accepted, Closed, Request, Response, Interim, Data, End, Writable, Done, Failed, Cancelled, TimedOut), its token, conn and req, and the fields its kind fills: listener, peer, clean, failure, request, response, body, buf, trailers and timing. e.isTerminal() is true of the four final answers; eventName(kind) is a kind's name.
Cancelling and deadlines#
drv.cancel(lp, req) asks a request to end. Still waiting, it is taken out and answered Cancelled; under way, HTTP/1.1's only way to stop it is to end the connection, so the connection is reset, the request answered Cancelled and any waiting behind it Failed, unsent. drv.requestBy(..., deadline) takes an absolute Instant (never a duration) and answers the request TimedOut then, the same way. drv.requestIn(..., group, deadline) also puts it in a CancelGroup (Cancel groups), whose cancel, from any thread, cancels it and whose deadline it inherits when that is tighter. A connection's own deadlines are the core's: a head, a body, an idle connection, an Expect.
Failures#
A Failed answer, and a Closed that is not clean, carry a Failure: the stage it failed at (Resolve, Connect, Read, Write, Http, Tls, Closed or Limit); the Windows error number in code when the system gave one (a reset reads as 64, and the system's certificate verifier refusing as CERT_E_UNTRUSTEDROOT and its kin); an HttpError (its kind, detail and noResponse); and unsent, no byte of the request reached the socket. A request may be tried again on another connection when it is unsent, or when it is idempotent and has noResponse; the client above decides, and these two say exactly. message() joins them as connect: closed: ... (Windows error 10061: ...).
Configuration#
Config.make() gives the defaults, each a public field to change: protocol (http1.Config: the bounds and timeouts above), connectTimeout (10 s), recvSize (16 KiB, each receive's buffer), sendDepth (4 sends in flight per connection), maxConnections (10,000; an accept past it is not made), maxWaiting (1), lingerTimeout (2 s: a connection closing sends its FIN and waits that long for the peer's, so a reset never cuts off a response the peer has not finished reading), acceptDepth (4 accepts outstanding per listener) and idleRecvReady (an idle connection waits with a zero-byte receive and pins no buffer).
Layers#
Between the socket and the protocol is a byte-stream layer, a Layer: bytes off the wire go in, the protocol's come out, and the other way round. A plain connection's is Plain (plainLayer()), which passes buffers through whole; a TLS session is the same shape, and the tls package makes one. drv.connectWith(lp, host, port, layer, sink) and drv.listenWith(lp, addr, maker, sink) take a layer, or a LayerMaker whose make(self &mut) gives one for each connection accepted. A type complies with Layer by having its methods, so httpconn imports no TLS. A layer that has work that may block (a certificate check the system makes may fetch over the network) hands it to the driver, which runs it on the blocking pool; a layer that fails closes its connection, every request on it Failed at Stage.Tls.
Fetching a URL: the httpclient package#
A Client takes a request for a URL and answers it with the response's events and then exactly one final answer, as httpconn does for one connection. Between the two it does what a program should not have to: it keeps a pool of connections per origin and reuses them, opens new ones within its limits, orders the requests that wait, tries again a request that is safe to repeat, follows redirects, and carries each request's cancel, group and deadline down to its connection. Like a driver, a client belongs to one loop and the thread that polls it.
One call: fetchWhole and getWhole#
Client.make(config, pool) makes a client. Request.get(url) is a request for a URL written as text and Request.make(method, url) any method's; Request.to(method, Url) takes a parsed Url. A Request has public fields to set: headers, body (sent whole, with its Content-Length, so that a retry or a 307 can send it again), intent and score, deadline (an absolute Instant), group (an Option[CancelGroup]) and maxRedirects. The client writes Host from the URL and the framing fields from the body, and adds User-Agent and Accept: */* when the request has none. A URL's user information is never sent.
client.fetchWhole(&mut lp, request, limit) does it all in one call: it drives the loop until the fetch has its final answer and gives a Response: outcome (an EventKind: Done, Failed, Cancelled or TimedOut; isOk() is true of Done), status, headers, the whole body as a String, url (the URL that answered, after any redirects), failure and timing. A body longer than limit bytes fails the fetch at Stage.Limit. A 404 is an answer, not a failure. Events of other fetches that arrive meanwhile are kept for the next poll. Since fetchWhole polls the loop itself, a loop that also runs a server's driver cannot use it, because what completes for the server would be dropped: the examples below put the server on a thread of its own. getWhole(url, limit) is the same with a loop and client of its own, for a program that wants one URL and nothing more.
The server is the previous page section's, with three more routes, run by a thread of its own whose listening port it sends back on a channel: / answers 200 Hello from Cone., /old answers 302 with Location: /, /echo answers 200 with the method, a space and the body it was sent, and /reset resets the connection without an answer. (The whole program is the file named below; the client is:)
fn urlOf(port u16, path &str) String {
mut u = String.from("http://127.0.0.1:");
u.appendUInt(u64.from(port));
u.append(path);
u;
}
fn show(label &str, r &Response) {
printStr(label);
printStr(": ");
if r.isOk() {
printUInt(u64.from(r.status));
printStr(" ");
printStr(r.body.view());
printStr(" (");
printUInt(u64.from(r.timing.redirects));
printStr(" redirects, ");
printUInt(u64.from(r.timing.attempts));
printStr(" requests, ");
printStr(if r.timing.reused {"kept connection";} else {"new connection";});
printStr(")\n");
} else {
printStr(eventName(r.outcome));
printStr(", ");
printStr(stageName(r.failure.stage));
printStr(" stage\n");
}
}
fn main() {
// The server, on its own thread, and the port it listens on
mut portTx Sender[u16];
mut portRx Receiver[u16];
portTx, portRx = Channel[u16].make();
imm stop = new Arc[imm, Atomic[bool]](Atomic[bool].of(false));
imm server = thread.start(new ServerArgs(portTx, stop), &serverThread);
mut port = 0u16;
match portRx.recv() {
case imm p Some {port = p;}
case is None[u16] {}
}
mut lp = Loop.mustMake();
mut client = Client.make(Config.make(), Pool.make());
// One call, the whole body: up to 64 KiB of it
imm hello = client.fetchWhole(&mut lp, Request.get(urlOf(port, "/").view()), 65536usize);
show("GET /", &hello);
// A redirect is followed; the URL that answered is on the response
imm moved = client.fetchWhole(&mut lp, Request.get(urlOf(port, "/old").view()), 65536usize);
show("GET /old", &moved);
printStr(if moved.url.view() == urlOf(port, "/").view() {" answered by the URL of /\n";} else {" answered by another URL\n";});
// Redirects are the caller's to follow when it says none
mut stay = Request.get(urlOf(port, "/old").view());
stay.maxRedirects = 0u32;
imm still = client.fetchWhole(&mut lp, stay, 65536usize);
show("GET /old, no redirects", &still);
match still.headers.get("location") {
case imm v Some {
printStr(" Location: ");
printStr(v as &str);
printStr("\n");
}
case is None[&Array[u8]] {}
}
// A POST with a body and a header field
mut post = Request.make("POST", urlOf(port, "/echo").view());
post.headers.add("Content-Type", "text/plain");
post.body = String.from("sent whole");
imm echoed = client.fetchWhole(&mut lp, post, 65536usize);
show("POST /echo", &echoed);
// A page that is not there is an answer, not a failure
imm missing = client.fetchWhole(&mut lp, Request.get(urlOf(port, "/missing").view()), 65536usize);
show("GET /missing", &missing);
// A failure says where it failed
imm bad = client.fetchWhole(&mut lp, Request.get("ftp://127.0.0.1/"), 65536usize);
show("GET ftp://", &bad);
printStr(" ");
printStr(bad.failure.message().view());
printStr("\n");
// A GET is repeated on a fresh connection when the first is reset under it; two repeats, then it fails
imm reset = client.fetchWhole(&mut lp, Request.get(urlOf(port, "/reset").view()), 65536usize);
show("GET /reset", &reset);
printUInt(u64.from(reset.failure.attempts));
printStr(" attempts\n");
imm s = client.stats();
printUInt(s.fetches);
printStr(" fetches, ");
printUInt(s.reused);
printStr(" on a kept connection, ");
printUInt(s.retries);
printStr(" retries, ");
printUInt(s.redirects);
printStr(" redirects\n");
client.abortAll(&mut lp);
stop.store(true);
server.join();
}
Output:
GET /: 200 Hello from Cone. (0 redirects, 1 requests, new connection) GET /old: 200 Hello from Cone. (1 redirects, 2 requests, kept connection) answered by the URL of / GET /old, no redirects: 302 moved (0 redirects, 1 requests, kept connection) Location: / POST /echo: 200 POST sent whole (0 redirects, 1 requests, kept connection) GET /missing: 404 No such page. (0 redirects, 1 requests, kept connection) GET ftp://: Failed, url stage url: malformed: only http and https URLs are fetched GET /reset: Failed, read stage 3 attempts 7 fetches, 6 on a kept connection, 2 retries, 1 redirects
A fetch's events#
A program that wants a response as it arrives, or many fetches at once, asks and polls. client.get(&mut lp, url, sink) and client.send(&mut lp, request, sink) give a FetchId and later exactly one final answer, Done, Failed, Cancelled or TimedOut, and nothing after it. Before it: Response (e.response, the head of the response that will be the answer, after any redirects; e.url, the URL that gave it), Data (the body's bytes, in order, each an IoBuf the program owns, handed back with client.release(id, n)) and End (e.trailers). Sinks are Sink.Here(token) and Sink.Chan(token, tx) as in httpconn, and the polling is client.poll(&mut lp, until, &mut events), or complete and pump beside a program's own operations, with wakeAt(). A fetch that cannot be made — a URL that does not parse or is not http or https, the queue full, its group cancelled, its deadline past — is answered at once. client.cancel(&mut lp, id) asks a fetch to end; waiting, it is answered Cancelled at once, and under way its connection's exchange is ended and the answer says how. A Request's deadline answers it TimedOut then, across its retries and redirects, and its group cancels it with the group's own cancel, from any thread. A final answer's e.timing says when it was queued, last given a connection, first answered and ended, how many requests it took, how many redirects, and whether its connection had been used before. client.stats() and client.driverStats() count what happened; live(), queued(), held() and isLive(id) say what is outstanding; abortAll(lp) ends everything.
The pool#
Connections are pooled per origin, the URL's scheme, host and port. A connection that ends an exchange with both sides willing to keep it waits idle in its origin's pool, and the next request for that origin takes it. At most maxPerOrigin (6) connections are open to an origin and maxTotal (64) in all; an idle one is closed after idleTimeout (90 s), or when the total is reached and another origin needs a connection, or by client.closeIdle(lp). HTTP/1.1 carries one request at a time on a connection, so these are also the most requests under way at once. All of an https client's connections take their TLS sessions from one tls.Client, so a new connection to a server resumes an earlier one's session. A connection that cannot be made fails the requests waiting for its origin.
Order: intents and scores#
A request that waits for a connection is queued by its intent: Intent.Urgent (the frame waits on it), Visible (in view now; the default), Prefetch or Background; then by its score within the intent, higher first; then by the order asked. An intent is strict: a Background request waits while any Urgent one does for the same connections. A request is given a connection only once it is free, never queued behind another's exchange, so one asked for later at a higher intent overtakes every one still waiting. client.reprioritize(id, intent, score) changes a waiting fetch's place and says whether it still could. At most maxQueued (10,000) requests wait; past that a fetch is Failed at once, at Stage.Queue. The program below limits the client to one connection, so the order shows, and sends five fetches before it first polls, one of them cancelled while it waits. (The server answers every GET with 200 and its own target as the body; the first number on each line is the fetch's token.)
fn getAt(port u16, path &str, intent Intent) Request {
mut r = Request.get(urlOf(port, path).view());
r.intent = intent;
r;
}
fn main() {
// ... the server thread and its port, as before ...
// One connection to an origin at a time, so the order requests are
// given it shows
mut config = Config.make();
config.maxPerOrigin = 1u32;
mut lp = Loop.mustMake();
mut client = Client.make(config, Pool.make());
// Five requests, asked for in this order; the fourth is changed its mind about
client.send(&mut lp, getAt(port, "/background", Intent.Background), Sink.Here(1u64));
client.send(&mut lp, getAt(port, "/prefetch", Intent.Prefetch), Sink.Here(2u64));
client.send(&mut lp, getAt(port, "/visible", Intent.Visible), Sink.Here(3u64));
imm dropped = client.send(&mut lp, getAt(port, "/cancelled", Intent.Visible), Sink.Here(4u64));
client.send(&mut lp, getAt(port, "/urgent", Intent.Urgent), Sink.Here(5u64));
client.cancel(&mut lp, dropped);
// Each fetch's events, as they come
mut events = List[Event].empty();
mut finals = 0u32;
imm giveUp = Instant.now() + Duration.fromSeconds(10i64);
while finals < 5u32 and Instant.now() < giveUp {
client.poll(&mut lp, giveUp, &mut events);
mut k = 0usize;
while k < events.len() {
imm e = &mut events[k];
k = k + 1usize;
printUInt(e.token);
printStr(" ");
printStr(eventName(e.kind));
if e.kind == EventKind.Response {
printStr(" ");
printUInt(u64.from(e.response.status));
} elif e.kind == EventKind.Data {
printStr(" ");
printStr(e.buf.view() as &str);
client.release(e.id, e.buf.len());
} elif e.isTerminal() {
finals = finals + 1u32;
}
printStr("\n");
}
events.clear();
}
client.abortAll(&mut lp);
stop.store(true);
server.join();
}
Output:
4 Cancelled 5 Response 200 5 Data /urgent 5 End 5 Done 3 Response 200 3 Data /visible 3 End 3 Done 2 Response 200 2 Data /prefetch 2 End 2 Done 1 Response 200 1 Data /background 1 End 1 Done
Retries and redirects#
A request is tried again, on another connection, when its connection failed under it (the stages Read, Write and Closed) and it is known safe to repeat: no byte of it reached the socket, or its method is idempotent and no byte of a response arrived. At most maxRetries (2) times, and never once its Response has gone to its sink. A connection that could not be made, a TLS failure, a protocol refusal and a timeout are not retried. A retried request keeps its place in the order.
A response of 301, 302, 303, 307 or 308 with a Location is followed, up to maxRedirects (10) or the request's own; past it the fetch fails at Stage.Redirect, and with 0 none is followed and the 3xx is the answer. The Location is resolved against the URL that answered. A 303 makes the next request a GET (a HEAD stays a HEAD), as 301 and 302 do of a POST; the body and the Content- fields go. A 307 or 308 sends the same method and body again. A redirect to another origin drops the request's Authorization, Proxy-Authorization and Cookie fields.
Failures and configuration#
A Failed answer carries a Failure: its stage (Url, Queue, Resolve, Connect, Read, Write, Http, Tls, Closed, Limit or Redirect), the Windows code when there is one, an HttpError, unsent, and attempts, how many requests were made. message() joins them, and stageName, intentName and eventName name the enums. Config.make() gives conn (an httpconn.Config, its driver's), tlsConfig (a tls.ClientConfig), maxPerOrigin, maxTotal, idleTimeout, maxQueued, maxRetries, maxRedirects and userAgent (cone-httpclient/0.1), each a public field.
TLS: the tls package#
A TLS Session is one connection's state and never touches a socket: its owner hands it the bytes that came off the wire, and takes back the bytes to send and the plaintext that came out. TLS 1.2 and 1.3 are done by OpenSSL 3.5 (tls.backendVersion() names it), and a server's certificate is judged by the operating system. A program rarely calls a Session directly: httpclient uses tls for every https:// URL, and a server uses it through httpconn's layers.
HTTPS: a server, and a client that trusts it#
A TLS server is made from a certificate chain and a private key in PEM files: tls.Server.make(tls.ServerConfig.make(&certFile, &keyFile)) gives a Result[Server, TlsError]. A Server is a handle, copied to share between threads. For an HTTP server, server.stream(pool) makes a Stream, a TLS session in the shape of httpconn's Layer; the listener makes one for each connection it accepts through a LayerMaker the program writes:
struct TlsMaker {
server tls.Server;
pool Pool;
pub fn make(self &mut) So[httpconn.Layer] {new So[tls.Stream](server.stream(pool));}
}
// ... and where the plain server called drv.listen:
match tls.Server.make(tls.ServerConfig.make(&certFile, &keyFile)) {
case imm tlsServer Ok {
imm maker = new So[TlsMaker](new TlsMaker(tlsServer, pool));
match drv.listenWith(&mut lp, SocketAddr.localhost(0u16), maker, httpconn.Sink.Here(100u64)) {
...
}
}
case imm e Error {printStr(e.message().view());}
}
Nothing else about the server changes: its requests are the same events, answered the same way. A client is httpclient with a tls.ClientConfig in its tlsConfig. By default the operating system verifies the server. The test certificates the tls package keeps (server.pem and server.key, for localhost and 127.0.0.1, under a test root the machine does not trust) show both answers: a client given the test root as its only trust anchor, and a client left to the system. The program is in two parts, a server on a thread (the file named below) and:
fn main() {
// ... the server thread and its port, as before ...
mut address = String.from("https://127.0.0.1:");
address.appendUInt(u64.from(port));
address.append("/");
// A client that trusts the test root, and only it
mut trusting = httpclient.Config.make();
trusting.tlsConfig.verify = tls.Verifier.roots(readText("ca.pem").view());
trusting.tlsConfig.offline = true;
mut lp = Loop.mustMake();
mut client = httpclient.Client.make(trusting, Pool.make());
imm ok = client.fetchWhole(&mut lp, httpclient.Request.get(address.view()), 4096usize);
printStr(if ok.isOk() {"trusting client: ";} else {"trusting client failed: ";});
printUInt(u64.from(ok.status));
printStr(" ");
printStr(ok.body.view());
printStr("\n");
// A client that asks the operating system, which has never heard of that root
mut asking = httpclient.Config.make();
asking.tlsConfig.offline = true;
mut lp2 = Loop.mustMake();
mut client2 = httpclient.Client.make(asking, Pool.make());
imm refused = client2.fetchWhole(&mut lp2, httpclient.Request.get(address.view()), 4096usize);
printStr("asking the OS: ");
printStr(httpclient.eventName(refused.outcome));
printStr(", ");
printStr(refused.failure.message().view());
printStr("\n");
client.abortAll(&mut lp);
client2.abortAll(&mut lp2);
stop.store(true);
server.join();
}
Output:
trusting client: 200 secure asking the OS: Failed, tls: closed: certificate untrusted (Windows error 2148204809: A certificate chain processed, but terminated in a root certificate which is not trusted by the trust provider)
Who verifies the server#
A tls.ClientConfig.make() gives the defaults, each a public field: verify (a Verifier), alpn (the application protocols offered, in order: "http/1.1"), minVersion and maxVersion (TlsVersion.Tls12 to Tls13), resume (sessions are resumed when the server allows), backend, verifyLane (the blocking pool's lane the system's verifier runs on) and offline (the system's verifier fetches nothing over the network: no intermediates, no revocation data). The Verifier decides whether a server is who it claims to be:
- Verifier.platform(), the default: the operating system. On Windows the chain the server sent goes to the certificate chain engine and is judged by the SSL policy for the name asked for, as the browser on the machine would judge it, with the machine's own roots and policy. It may fetch over the network, so it never runs on the loop's thread. Revocation that cannot be checked does not refuse a server.
- Verifier.roots(pem): for development. The chain is checked against these roots alone, and the name; the machine's roots are not used.
- Verifier.pinnedSha256(hex): for development. The server is accepted when the SHA-256 of its public key is one of the 64-hex-digit values given, whatever signed it. A malformed value is a panic.
- Verifier.insecureDevelopmentOnly(): nothing is checked. Anyone in the path can read and change the connection. Never use it against a real server.
A ServerConfig.make(certFile, keyFile) has alpn (the protocols it accepts, in its order of preference: a client offering none of them is refused), minVersion, maxVersion and backend. tls.Client.make(config) and tls.Server.make(config) fail with a TlsError when the configuration cannot be loaded (a root bundle with no certificate, a file that cannot be read, a key that is not the certificate's). A client that resumes keeps, for each server name, what it was sent to resume with, so that a new connection to that server takes the newest; a TLS 1.3 ticket is used once, up to 8 are kept for a server, and client.cachedSessions() says how many.
Without a socket: sessions#
client.session(serverName) and server.session() make a Session directly; the server name is sent by SNI and checked against the certificate (an IP address is checked only). The owner moves the bytes:
- s.feed(cipher) takes ciphertext that arrived (all of it; a partial record is buffered), and s.feedEnd() says the transport's stream ended.
- s.pending() says how many bytes are ready to go out, and s.transmit(out) copies them into a buffer: handshake messages, records, alerts.
- s.write(plain) protects plaintext (none is taken before the handshake is done, isOpen()) and s.read(out) gives plaintext while canRead() says some may be there.
- s.close() queues a close_notify; s.peerEnded() says the peer's stream ended and was read. A stream that ends with no close_notify still ends, as a TCP FIN does, and info().truncated says so.
- s.pollEvent() says once each what happened: HandshakeDone, VerifyNeeded, PeerClosed or Failed (Nothing when none). isFailed() and error() say why a session failed.
- When the system's verifier is to decide, the handshake stops at VerifyNeeded (s.wantsVerify()): run s.verifyNow() where blocking is fine, or give s.verifyOp(&pool) to a loop's blocking pool and hand its completion to s.verifyDone(&mut c). A Stream does this through the driver.
The program moves bytes between a client's session and a server's by hand:
// Move what each side has to send to the other until neither has any;
// what each reads is appended to its string
fn pump(c &mut tls.Session, s &mut tls.Session, cin &mut String, sin &mut String) {
mut buf = new Array[u8, 20000] <- fill 0u8;
mut moved = true;
while moved {
moved = false;
while c.pending() > 0usize {
imm n = c.transmit(&mut buf[0usize ..< 20000usize]);
s.feed(&buf[0usize ..< n]);
moved = true;
}
while s.pending() > 0usize {
imm n = s.transmit(&mut buf[0usize ..< 20000usize]);
c.feed(&buf[0usize ..< n]);
moved = true;
}
while c.canRead() {
imm n = c.read(&mut buf[0usize ..< 20000usize]);
if n == 0usize {break;}
cin.appendBytes(&buf[0usize ..< n]);
}
while s.canRead() {
imm n = s.read(&mut buf[0usize ..< 20000usize]);
if n == 0usize {break;}
sin.appendBytes(&buf[0usize ..< n]);
}
}
}
fn main() {
// The server's side: its certificate chain and key
imm certFile = certs("server.pem");
imm keyFile = certs("server.key");
imm server = tls.Server.make(tls.ServerConfig.make(&certFile, &keyFile));
// The client's side: trusts the test root alone
mut rootPem = String.empty();
imm caFile = certs("ca.pem");
match fs.readFile(&caFile) {
case mut r Ok {rootPem <=> r;}
case imm e Error {}
}
mut config = tls.ClientConfig.make();
config.verify = tls.Verifier.roots(rootPem.view());
match server {
case imm srv Ok {
match tls.Client.make(config) {
case imm cl Ok {
// One session each; no socket anywhere
mut c = cl.session("localhost");
mut s = srv.session();
mut cin = String.empty();
mut sin = String.empty();
pump(&mut c, &mut s, &mut cin, &mut sin);
printStr(if c.isOpen() and s.isOpen() {"handshake done\n";} else {"no handshake\n";});
imm info = c.info();
printStr(info.versionName.view());
printStr(" ");
printStr(info.cipher.view());
printStr(", ALPN ");
printStr(info.alpn.view());
printStr(", verified by ");
printStr(tls.verifyKindName(info.verifiedBy));
printStr("\n");
// Plaintext in, ciphertext out, and the other way
c.write("ping");
pump(&mut c, &mut s, &mut cin, &mut sin);
printStr("server read: ");
printStr(sin.view());
printStr("\n");
s.write("pong");
pump(&mut c, &mut s, &mut cin, &mut sin);
printStr("client read: ");
printStr(cin.view());
printStr("\n");
// Closing is a message too
c.close();
pump(&mut c, &mut s, &mut cin, &mut sin);
printStr(if s.peerEnded() {"the server saw the client close\n";} else {"still open\n";});
printStr(if s.info().closeNotify {"...with a close_notify\n";} else {"...without one\n";});
// The same client, the wrong name
mut c2 = cl.session("example.org");
mut s2 = srv.session();
pump(&mut c2, &mut s2, &mut cin, &mut sin);
if c2.isFailed() {
imm why = c2.error().message();
printStr(why.view());
} else {
printStr("no failure");
}
printStr("\n");
}
case imm e Error {printStr(e.message().view());}
}
}
case imm e Error {printStr(e.message().view());}
}
}
(certs(name) makes the Path of one of the test certificates.) Output:
handshake done TLSv1.3 TLS_AES_256_GCM_SHA384, ALPN http/1.1, verified by roots server read: ping client read: pong the server saw the client close ...with a close_notify tls: certificate name mismatch: hostname mismatch (OpenSSL's verify result 62)
Streams#
A Stream is a Session with buffers, in the shape of httpconn's Layer: client.stream(serverName, pool) and server.stream(pool) make one, and new So[tls.Stream](...) slots it into drv.connectWith or a LayerMaker. The protocol's bytes handed to it before the handshake is done wait and go out after it. A stream that fails reports it once its last bytes (an alert, usually) have gone out. st.session() reaches the session.
Errors and diagnostics#
A TlsError has a kind (a TlsErrorKind: CertExpired, CertNameMismatch, CertUntrusted, CertRevoked, CertOther, ProtocolVersion, NoAlpnMatch, Alert, Protocol, Config or Backend), code (the back end's or the system's own number), osError (Windows' error number when the system gave the reason), and detail. message() joins them as tls: certificate untrusted: ..., and errorName(kind) names a kind.
s.info() gives a TlsInfo of what a session negotiated: version and versionName, cipher, alpn, resumed, verifiedBy, handshakeTime and verifyTime (the system's verifier's share of it), closeNotify and truncated; text() says it in one line, which is what a connection's layerInfo gives. tls.stats() counts, for the process, handshakes (and those resumed), failures, verifications by the system's verifier, close_notify sent and received, and streams that ended without one.