Cone Language Reference

How the runtime works

On this page

Note: The actors package runs on Windows and Linux (x86-64), not yet on macOS.

Every actor, every parallel each and every awaited answer so far has run on a runtime that started and stopped by itself. Most programs never need to look at it. This page is for the program that does: one that tunes how many workers there are, wants to know when an actor's memory is collected, or needs to call the parallel loop underneath directly. It is ordinary library code, the actors package, and afterwards you will know what it does on your behalf and which knobs it has. Here the workers are set before the first actor, and asked afterwards what they did:

mod workers;

import stdio use *;
import actors;

actor Counter {
  n u64;
  pub async do add(self, k u64) { n = n + k; }
  pub async do total(self) u64 { n; }
}

actor Boss {
  c Counter;
  pub async do run(self) {
    each i in 1u64 .. 1000u64 {
      c.add(i);
    }
    let t = await c.total();
    let s = actors.stats();
    print <- "total ", t, " from ", s.workers, " workers\n";
    print <- "at least ", s.messages, " messages handled so far\n";
  }
}

fn main() {
  // Before the first actor: four workers, each running at most
  // 16 of an actor's messages at a turn
  actors.configure(new actors.Options(workers: 4u32, batch: 16u32));
  let b = new Boss(new Counter(0u64));
  b.run();
}

It prints:

total 500500 from 4 workers
at least 1002 messages handled so far

The runtime starts and stops by itself#

The workers start with the first actor, on whichever thread starts it; two threads starting a first actor at once start them once. A program that starts no actor starts no thread. They start then, not in the module's initializer, which is initpure. Before the first actor, actors.configure(new actors.Options(workers: n, batch: b)) sets n workers (by default one per logical processor), each running at most b of an actor's messages at a turn (100 by default) before the actor goes to the back of the queue; once the workers run, it is a panic.

The actors module's finalizer, which the program's entry runs after main returns, waits until the runtime is quiescent (no actor queued or running, so no message in any mailbox), then stops the workers and joins them. The final finalizes the program's own module first, so the handles its globals hold are let go before the wait; handles in main's locals go when it returns, before the final begins. An actor paused on I/O is neither queued nor running, so it does not hold the program open (The I/O thread, and the program's end), unless main ends by calling actors.waitUntilIdle(), which waits for that I/O too (interim; the same section). A program that calls finalAll() itself, as programs did before the entry was built, waits at that call, and handles still held in its locals are not let go by then: it starts its actors in a block that ends before the call. The runtime is not started again: starting an actor or sending a message once the workers have stopped is a panic, since nothing would run the message. Quiescence assumes that once the finalizer has begun every message comes from an actor: another thread that goes on sending is joined first.

actors.stats() answers what the workers have done so far (Stats: workers, messages handled, activations or turns, steals, and taskSteals, loop pieces stolen), without stopping them, and the run's totals once they have stopped. actors.workerIndex() is the index of the worker running the caller, or actors.noWorker.

The final runs when main returns (Running every module's initializer and finalizer). A program that panics or calls the C library's exit ends whatever its actors are doing: messages still in flight are silently dropped, and no actor still alive is finalized.

How the scheduler works#

A mailbox is Vyukov's intrusive queue of many senders and one receiver: a send is one atomic exchange and one store; the worker running the actor takes messages with no read-modify-write; and a mark on the mailbox says the actor is idle, so the send that finds it so queues the actor. Each worker has a run queue of its own, and there is one shared inject queue: an actor made runnable by a send from a worker is queued on that worker, one made runnable by any other thread on the inject queue. A worker with nothing of its own takes a share of the inject queue, then steals half of another worker's queue. An idle worker parks on the futex, and is woken, one at a time, as work arrives. Quiescence is found by counting the activations each worker begins and ends, and reading the counts twice.

The actor's own garbage-collected region#

Some data is a graph: nodes that point at each other, shared and cyclic, which a single owner cannot hold and reference counting cannot free. An actor may keep such a graph in a garbage-collected region of its own, part of the actors package: nothing is imported beyond import actors;, which also binds ActorGc, the region's reference. Only one worker at a time runs an actor, and between its messages nothing holds a reference into the region but the actor's state and its paused behaviours, so the region is collected there, between that actor's own messages (below).

import actors;

struct Node { pub n i64; pub next Option[ActorGc[mut, Node]]; }

actor Keeper {
  head Option[ActorGc[mut, Node]];

  fn init(self &new) {*self = new Self(None[ActorGc[mut, Node]]);}

  pub async do push(self, n i64) {
    head = Some[ActorGc[mut, Node]](new self[Node](n, head));
  }
}

Each actor's state has a method allocRegion that answers the region, and new self[T](args) inside the actor's methods (behaviours and synchronous methods alike, init after *self is filled) allocates there, as new a[T](args) allocates in an arena (Region strategies): it is self.allocRegion().alloc(new T(args)), the general rule that new x[T](...) allocates in the region x is or leads to (provisional: the allocRegion method behind it is not settled). A reference is an ActorGc[mut, T], or an ActorGc[imm, T] from self.allocRegion().allocImm(v), a provisional spelling that stands in for new self[imm, T], which is not decided. A bare new ActorGc[mut, T](...) names no actor, so it has no region to be in and panics.

The region's record is made when the actor first asks for it, and holds no memory until it first allocates, so an actor that never does pays only one pointer in its header (8 bytes). Each object in the region carries a header of its own, 24 bytes before the value, and every object is rounded up to a multiple of 16 bytes. Only the actor's own code holds self, so only it allocates in the region; other code is handed the actor's state to allocate there. A collection made so, new self[List[T]](16), is handed the region as any collection made in a region is (a collection in a region): its words are an object of the region, and its block comes from the region's own allocate, resize and release, which count it with what the actor has allocated. The block is a block of its own, never moved by a collection (a collection's block pointer is a raw pointer, which the collector has no way to find and rewrite), and the list's own final gives it back, which the collector runs when it finds the list unreached, or at the actor's death.

A reference into the region is made, held and copied in the actor's state, its locals, a List, a Dict's values (a traced key is refused) or an Rc it owns, and a record parked by an await. It is never Sendable: it stays in its actor, so it is refused in a message, an answer or a thread's start (Sendable). It has no write barrier and no count, so a reference behind an Rc is allowed (the Rc is traced once for each collection, so a cycle of them ends).

Collected between messages, by a copying collector#

After each message (and after the rest of a behaviour that began a parallel each, once the loop has finished) the worker may collect, when the bytes allocated since the last collection exceed the bytes that survived it, and not before 64 KiB have been allocated (a floor, so a tiny actor does not collect all the time). The roots are the actor's state and the records its awaits have parked; nothing else holds a reference, since no borrow and no stack of the actor's exists between messages. Everything reached is copied into new chunks and the references are rewritten, so an object moves: nothing holds its address across a message. An object over 4096 bytes (its header counted) is a large object, a block of its own that is marked where it sits and never moves. A value with a final, or a field that needs finalizing, is finalized when a collection finds it unreached (it and what it reaches are kept for that one cycle, its death run once, newest first, and the collection after frees it), and when the actor dies, after its state's own death, every one still alive is finalized, newest first, and the region is freed whole.

actors.setFloor(bytes) and actors.setStress(on) (collect after every message; the environment variable CONE_ACTORGC_STRESS does the same for a whole run) tune it, for the process, before the first actor allocates; self.allocRegion().stats() says what the region holds and has done, for tests. Inside a parallel each a pass reads a read-only reference (In an actor's behaviour), and allocates nothing.

The parallel loop underneath#

A parallel loop runs on the same workers as the actors. The language writes one as parallel each, with its checked body. Below it is the low-level form it is written over, a library function, for code that writes through raw pointers or needs control of the grain:

import libc;
import actors;

let out = libc.calloc(100000usize, 8usize) as *u64;
// the body is a closure taking a sub-range [lo, hi); it lists the pointer it writes through
actors.parallelRange(0usize, 100000usize, 256usize, fn(lo usize, hi usize) [out] {
  var i = lo;
  while i < hi {*(out + i) = u64.from(i) * u64.from(i); i = i + 1usize;}
});
// *(out + 99999usize) is 9999800001

actors.parallelRange(lo, hi, grain, body) calls body(a, b) over pieces of [lo, hi) of at most grain items, on any worker and at once, and returns when every piece has finished. The body is any callable taking two usize: a closure written at the call (Closures), a function reference, or a struct with a (). It is a [F fn(lo usize, hi usize) + Shareable] (Generics), lent for the call and compiled for each body, so the call of it from a piece is the body itself. Its pieces run at once, so the compiler refuses a body that is not Shareable: one that changes a variable it borrows, borrows a &mut, or holds an Rc.

The loop starts as one task for the whole range; whoever runs a task splits it in half while it is longer than the grain, puts one half on top of its worker's queue of tasks, and works the other, and an idle worker steals the oldest, largest half, so uneven work spreads itself and nothing is divided ahead of time. A worker takes its own tasks before its actors, and a worker that steals prefers tasks to actors. The caller works on its own loop while it waits, so the loop may be called from main (it starts the workers if no actor has), from any thread, from inside the body of another loop, and from inside an actor's behaviour: there the actor is not run again until the loop has finished, while its mailbox goes on accepting messages, which are handled afterwards in order.

It trusts its caller beyond the Shareable check: the body must write only its own sub-range's items (through a raw pointer or an atomic), must not await, and must not wait on another piece; nothing checks it. That is the price of the low-level form, and the reason parallel each exists.

  • actors.parallelRangeThen(lo, hi, grain, body, onDone) starts the same loop and returns at once, and the piece that finishes last calls onDone() on whichever thread it runs. Both callables outlive the call, so they are moved in and kept in owners until the loop has finished: closures that list what they use (a closure that borrows a local is refused), the body Sendable and Shareable, onDone Sendable.
  • actors.parallelEach(lo, hi, piece, data) is the loop a parallel each becomes, called by the compiler's code with the loop it outlined into a function reference and a pointer to the record of the variables it uses (the one place that form stays). It is the same loop with the grain chosen for the workers (pieces of about an eighth of a worker's even share, but never split below 128 items, so a loop of 128 items or fewer is one piece, run at once on the caller's thread), and with the messages the pieces send to actors gathered: a piece's sends are kept, not made, and made after the loop by the code that began it, in the order of the passes, so that an actor sees the same order every run (parallelRange sends at once, in no defined order).
  • actors.parSum, parFold and parFindFirst are the functions the parallel reductions (xs.parallel().sum()) become. They cut a view into runs by its item count alone, never by the workers, run the runs on the same workers, and join their results pairwise up a tree fixed by the count, so the result is the same on every run with any number of workers.

You can now tune the workers, keep a graph in an actor's own collected region, and call the parallel loop beneath parallel each. All of that runs on the CPU; the next page runs the same kind of loop on the GPU.