Actors

Note: Not implemented: actor trait, a generic actor, and qfn. The actors package runs on Windows and Linux (x86-64), not yet on macOS.

An actor is state that lives on, reached by nothing but messages. Each message is a call of one of the actor's behaviours, its methods declared async do, queued in its mailbox and run later, one at a time, on whichever of a pool of worker threads takes the actor up. A behaviour runs to completion, on that worker's own stack, with the actor's state mut, or to an await, where it is cut (Awaiting a reply); no other behaviour of the same actor runs until it returns there. An actor is declared with actor, and it runs on the actors package, which a module declaring one imports: the mailbox, the work-stealing scheduler, the counting that decides when an actor dies, and the quiescence a program ends at. The package starts and stops by itself: the first actor made starts the workers, and the module's finalizer waits until all is quiet and stops them.

import actors;

actor Counter {
  count u64;                       // its state: private to it

  fn init(self &new, start u64) {   // run by 'new Counter(start)', in the actor
    *self = new Self(count: start);
  }

  pub async do add(self, n u64) {  // a behaviour: sent as a message
    count = count + n;
  }

  pub async do report(self &, to Printer) {   // one carrying another actor's handle
    to.show(self.doubled());
  }

  fn doubled(self &) u64 {         // synchronous: called, inside the actor
    count * 2u64;
  }

  fn final(self &uni) {}           // when it dies
}

fn main() {
  imm c = new Counter(0u64);       // the first actor starts the workers
  c.add(5u64);                     // a message: queued, and run later
  c.add(7u64);                     // run after the first
}                                  // then the entry's final waits until all is quiet, stops the workers

Behaviours and methods#

A method declared async do is a behaviour: it runs asynchronously, when the actor is scheduled, and an await on one waits for its reply. async do is one keyword of two words, and neither means anything alone: written without the other, either is refused. async says when the behaviour runs, and pairs with await, as in C# and JavaScript, where a call of an asynchronous function need not be awaited; do is the declaration's verb, as fn is a function's. Unlike an asynchronous function there, a behaviour never starts on the sender's thread: a send only queues the message, so no thread but the actor's touches its state.

Synchronous methods#

A method declared fn is synchronous: the actor's init and final, and any helper, which sees the actor's fields directly and runs inside the method that calls it, as a struct's method does (doubled above). It is the actor's alone: never pub, and not carried by the handle. It holds no await: an await belongs at the top of a behaviour, and the synchronous work between awaits is pushed down, onto a helper or onto a type the actor owns (Await in a behaviour).

The four behaviour shapes#

Whether a behaviour declares a return type, and whether it holds an await, are independent, and each decides one thing: the return type, whether a reply is sent; the await, whether the behaviour is cut and waits. So a behaviour has one of four shapes, each known from its declaration alone:

• Fire and forget: no return type, no await. It runs once and ends; nothing is sent back, and nothing waits. The cheapest shape.
• A cheap query: a return type, no await. It runs once and replies with its value as it ends, to a caller that awaited it.
• Awaits, owes nobody: an await, no return type. It waits for other actors' answers, and answers no one.
• The full handshake: both. It waits for answers, then replies with its own value.

actor Store {
  total u64;

  pub async do add(self, k u64) {           // fire and forget
    total = self.plus(k);
  }

  pub async do count(self) u64 {total;}     // a cheap query

  fn plus(self &, k u64) u64 {total + k;}   // synchronous: an ordinary call
}

actor Client {
  store Store;

  pub async do run(self) {                  // awaits, owes nobody
    store.add(5u64);
    store.add(7u64);
    imm c = await store.count();
    say("count", c);
    imm r = await self.report();
    say("report", r);
  }

  pub async do report(self) u64 {           // the full handshake
    imm c = await store.count();
    c * 10u64;
  }
}

(say prints a word and a number on a line.) Sent client.run(), with a Store whose total starts at 0, this prints count 12, then report 120: both adds reach the Store before the count does, since one sender's messages are handled in the order sent.

The declaration#

An actor's body is written as a struct's: fields, an init, behaviours, synchronous methods, a final. Its fields are its state, private to it: a field is never pub, and nothing outside the actor reaches one. A method's self is the state, lent to a behaviour for the message it handles, and to a synchronous method by its caller: written bare, it is self &mut, which the method has alone while it runs; self & only reads it. A behaviour that is not pub is a message all the same, which only its own module sends; a function without self is the actor's own, called from its methods as a struct's is. A behaviour is not generic, and every parameter it takes is Sendable, since the message carries its arguments to the thread the actor runs on; a borrow is refused unless it lasts the whole program, is written so (&'static imm T, &'static opaq T or &'static uni T) and reaches what may cross (Race condition safety). A behaviour may return a value, which must be Sendable too, by the same rule, so a borrow in it is written 'static as a parameter's is: a send returns at once, before the behaviour runs, and the value is dropped, unless the behaviour was awaited (Awaiting a reply). An actor declares no static, macro, extern method or use, no type parameters, and names no abstraction: actor trait, its abstraction, is not built yet.

Private state#

The state exists only inside its actor. It is constructed in one place, as the actor is made, where its init fills it, *self = new Self(count: 0u64). Any other construction of it, new Self(...) in one of the actor's methods or as an allocation's value, by its fields' names or by its init, is refused, since the state it made would be in no actor, and the sends it or a helper made to its actor would reach nothing. Nor is the state copied out of its actor: it is a move type whatever its fields are, so imm s = *self is refused, as taking any value out through a borrow is. For a value of the state's shape, declare a plain struct.

Making an actor#

new Counter(args) makes an actor and answers its handle. The actor is real before its init runs: the runtime's part of it is made first, and the init then runs, on the calling thread, on the state where it will stay. So the init, and any synchronous method it calls, may send to the actor itself, with self.m() or selfactor, once *self = ... has filled the state (before that, self holds nothing, and every use through it is refused, as in any initializer), and may hand selfactor to another actor. The actor runs no message until its init has returned: a message sent meanwhile, by the init or by another actor already given the handle, waits in the mailbox and runs afterwards, in the order sent. A panic in the init is a panic, which ends the program.

actor Starter {
  count u64;

  fn init(self &new) {
    *self = new Self(count: 0u64);
    self.note(1u64);              // a send to itself: queued, run after the init
    self.helper();                // a helper the init calls may send too
    say("init ends", count);
  }

  fn helper(self) {
    selfactor.note(2u64);
  }

  pub async do note(self, k u64) {
    count = count + k;
    say("note", count);
  }
}

new Starter() prints init ends 0, then note 1 and note 3: both notes wait until the init has returned.

The initializer's parameters must be Sendable too, since the state made from them runs on other threads. An actor may declare several initializers, as a struct may, each joining the overload name init. One that declares none is made by its state's implicit initializer: new Counter(5u64) takes the fields whose types are written, in order, a field with a default taking it where the argument is left out. That initializer is the module's own, as a struct's implicit one is where its fields are private, except for a pub actor with no fields at all. There is no other way to make an actor, and no creation function: spawn stays reserved.

The handle#

The actor's name names its handle, a value holding one counted owner of the actor's mailbox, the runtime's part of it: copying a handle (passing it, storing it, putting it in a message) makes another owner, and each going drops one. A handle carries the actor's behaviours and nothing else: calling one on it, c.add(5u64), sends that message, and reaching a field of the state or a synchronous method through it is refused as private. A handle is Sendable whatever the state holds: it may be sent to a thread, over a channel, in another actor's message, or kept in another actor's state. So the state may hold what may not cross threads (Rc, a mut owner, borrows inside a method), and never crosses as a value: only the runtime reaches it.

What the declaration stands for#

The compiler turns an actor into four ordinary declarations the actors package runs: the state, a struct of the body as written, named Counter.State; the messages, an enum with one variant per behaviour, named for it, its fields the behaviour's parameters, named Counter.Msg; the handle, Counter, whose one field is an Arc[opaq, actors.Mailbox[Counter.Msg]], an initializer for each of the state's, which makes the actor's block with actors.makeActor, constructs the state in it in place, and starts the actor with actors.startActor, and a method per behaviour, as public as the behaviour, which calls actors.send; and the dispatch function, Counter.dispatch, which the runtime hands each message where it lies, and which moves each argument out of it into the behaviour's call. Those names are what a diagnostic about them says; no source can write them, so nothing but the actor reaches its state, its messages or its mailbox. The mailbox and the state are one allocation, so a send touches one block. Actors and the declarations that use them may be written in any order: a message enum holds handles by value, and a handle reaches the enum only through its owning reference to the mailbox, and a reference never needs what it points at to be laid out first.

An actor's life#

While an actor is queued to run, or running, the scheduler holds one owner more, taken by the send that found it idle and dropped when its mailbox is found empty. So an actor dies exactly once, when its last handle has gone and its mailbox is empty: a handle dropped with messages still queued leaves the actor alive until they are handled. At its death its state is finalized (its final, then its fields, handles to other actors among them), on whichever thread let go last, then the block is freed. Actors that hold handles to each other in a cycle never die by counting, as a cycle of Arc owners never does, until one of them lets go.

Messages#

A send is one-way: it queues the message and returns. An answer comes back only to an await (Awaiting a reply), or as a message sent back to a handle passed along. Each sender's messages to one actor are handled in the order sent, and a message is never overtaken by one it caused. Each argument arrives moved into the behaviour's call, once: a value that may only move (a uni or So owner) arrives whole, and a counted one (a handle, an Arc, a lock-managed Arc[Mutex, T]) is counted once.

self and selfactor#

An actor has two selves. self is its state, lent to the method while it runs; it never leaves the actor. selfactor is its handle: another owner of the actor, a value that may go anywhere a handle may — into a message, into another actor's state, or to an await on one of its own behaviours:

actor Doubler {
  pub async do double(self, n u64) u64 {n * 2u64;}

  pub async do twice(self, n u64) u64 {
    imm once = await selfactor.double(n);  // a message to itself
    imm again = await selfactor.double(once);
    again;
  }
}

await doubler.twice(5u64) is 20. Each double is queued in Doubler's own mailbox, and runs because twice has paused and the actor is free. await self.double(n) is the same: a behaviour named through self is sent through the actor's handle (Sending to itself), never run at once. selfactor may be written in any of the actor's methods, its init among them once the state is filled (Making one), but not in its final, nor anywhere outside the actor.

Sending to itself#

In the actor's own methods, self.m(args), where m is one of its behaviours, sends m to the actor itself, as selfactor.m(args) does: the message joins the end of the actor's own mailbox, the method goes on at once, and m runs later. A behaviour is never called, not even by its own actor: it runs only when its dispatcher takes up a message for it. A synchronous method is called, as a struct's is.

actor Echo {
  pub async do start(self) {
    say("start", 1u64);
    self.later(1u64);             // a send to itself: start goes on at once
    self.relay();                 // a synchronous helper, an ordinary call
    say("start ends", 1u64);
  }

  async do later(self, k u64) {
    say("later", k);
    if k == 2u64 {
      self.finish();
    }
  }

  fn relay(self) {
    later(2u64);                  // 'self.later(2)': a send, from a helper too
  }

  async do finish(self) {
    imm d = await self.double(21u64);
    say("finish", d);
    self.show(d / 2u64 - 11u64);
  }

  async do double(self, k u64) u64 {k * 2u64;}

  async do show(self, k u64) {
    say("show", k);
  }
}

Sent echo.start(), this prints start 1 and start ends 1 before either later runs, then later 1, later 2, finish 42 and show 10. A behaviour reached through anything but the method's own self — such as another borrow of a state, a parameter's — is refused, and so is one sent through self in the actor's final, as it dies. In its init it is a send like any other, run once the init has returned (Making one).

Awaiting. In a behaviour, await on a behaviour of an actor that returns a value sends it and waits for that value: the behaviour pauses there, the actor handles other messages meanwhile, and the reply resumes it (Awaiting a reply). In a method, selfactor is the actor's own handle, another owner of it, which may be sent anywhere a handle may; self is its state, which never leaves it.

Inside an actor#

A method of an actor may use what a single thread may: borrows, and So and Rc owners, whose count need not be atomic, since one worker at a time runs an actor and every hand-off of an actor between workers orders what the first did before anything the next does. Three things differ from a plain thread:

• The collector package's Gc is not supported inside an actor (it is discouraged everywhere, and likely to be replaced). That collector is single-threaded, and an actor runs on every worker in turn; a traced reference made, held, reached or dropped by an actor corrupts it. So the compiler refuses one made or held in an actor — a field of its state, a synchronous method's parameter, an allocation, a call that returns one or a literal holding one in any method's body, init and final included, a struct, tuple, array or enum holding one as well — and says to hold an ActorGc instead. An actor has a garbage-collected region of its own (the actor's own region, below), whose references it makes, holds and copies freely.
• A thread-local global seen from an actor's method is the current worker's copy, not the actor's: its next message may run on another worker.
• A method that blocks holds its worker while it does. Actors wait by being sent a message.

The actor's own garbage-collected region#

Every actor owns a garbage-collected region, part of the actors package: nothing is imported beyond import actors;, which also binds ActorGc, the region's reference. Each actor's state has a method allocRegion that answers the region, and new self[T](args) inside the actor's methods — behaviours and synchronous methods alike, init after *self is filled — allocates there, as new a[T](args) allocates in an arena (Region strategies): it is self.allocRegion().alloc(new T(args)), the general rule that new x[T](...) allocates in the region x is or leads to (provisional: the allocRegion method behind it is not settled). A reference is an ActorGc[mut, T] (or ActorGc[imm, T], from self.allocRegion().allocImm(v), a provisional spelling that stands in for new self[imm, T], which is not decided). A bare new ActorGc[mut, T](...) names no actor, so it has no region to be in and panics. The region's record is made when the actor first asks for it, and holds no memory until it first allocates, so an actor that never does pays only one pointer in its header (8 bytes). Each object in the region carries a header of its own, 24 bytes before the value, and every object is rounded up to a multiple of 16 bytes. Only the actor's own code holds self, so only it allocates in the region; other code is handed the actor's state to allocate there. A collection made so, new self[List[T]](16), is handed the region as any collection made in a region is (a collection): its words are an object of the region, and its block comes from the region's own allocate, resize and release, which count it with what the actor has allocated. The block is a block of its own, never moved by a collection (a collection's block pointer is a raw pointer, which the collector has no way to find and rewrite), and the list's own final gives it back, which the collector runs when it finds the list unreached, or at the actor's death.

import actors;

struct Node { pub n i64; pub next Option[ActorGc[mut, Node]]; }

actor Keeper {
  head Option[ActorGc[mut, Node]];

  fn init(self &new) {*self = new Self(None[ActorGc[mut, Node]]);}

  pub async do push(self, n i64) {
    head = Some[ActorGc[mut, Node]](new self[Node](n, head));
  }
}

A reference into the region is made, held and copied in the actor's state, its locals, a List, a Dict's values (a traced key is refused) or an Rc it owns, and a record parked by an await. It is never Sendable: it stays in its actor, so it is refused in a message, a reply or a thread's start (ErrorNotSendable). It has no write barrier and no count, so a reference behind an Rc is allowed (the Rc is traced once for each collection, so a cycle of them ends).

Collected between messages, by a copying collector#

After each message (and after the rest of a behaviour that began a parallel each, once the loop has finished) the worker may collect, when the bytes allocated since the last collection exceed the bytes that survived it, and not before 64 KiB have been allocated (a floor, so a tiny actor does not collect all the time). The roots are the actor's state and the records its awaits have parked; nothing else holds a reference, since no borrow and no stack of the actor's exists between messages. Everything reached is copied into new chunks and the references are rewritten, so an object moves: nothing holds its address across a message. An object over 4096 bytes (its header counted) is a large object, a block of its own that is marked where it sits and never moves. A value with a final, or a field that needs finalizing, is finalized when a collection finds it unreached (it and what it reaches are kept for that one cycle, its death run once, newest first, and the collection after frees it), and when the actor dies, after its state's own death, every one still alive is finalized, newest first, and the region is freed whole.

actors.setFloor(bytes) and actors.setStress(on) (collect after every message; the environment variable CONE_ACTORGC_STRESS does the same for a whole run) tune it, for the process, before the first actor allocates; self.allocRegion().stats() says what the region holds and has done, for tests. Inside a parallel each a pass reads a read-only reference (Parallel each in a behaviour), and allocates nothing.

The runtime starts and stops by itself#

The workers start with the first actor, on whichever thread starts it; two threads starting a first actor at once start them once. A program that starts no actor starts no thread. They start then, not in the module's initializer, which is initpure. Before the first actor, actors.configure(new actors.Options(workers: n, batch: b)) sets n workers (by default one per logical processor), each running at most b of an actor's messages at a turn (100 by default) before the actor goes to the back of the queue; once the workers run, it is a panic. The actors module's finalizer, run by the program's entry runs after main returns, waits until the runtime is quiescent — no actor queued or running, so no message in any mailbox — then stops the workers and joins them. The final finalizes the program's own module first, so the handles its globals hold are let go before the wait; handles in main's locals go when it returns, before the final begins. A program that calls finalAll() itself, as programs did before the entry was built, waits at that call, and handles still held in its locals are not let go by then: it starts its actors in a block that ends before the call. The runtime is not started again: starting an actor or sending a message once the workers have stopped is a panic, since nothing would run the message. Quiescence assumes that once the finalizer has begun every message comes from an actor: another thread that goes on sending is joined first. actors.stats() answers what the workers have done so far (Stats: workers, messages handled, turns, steals, loop pieces stolen), without stopping them, and the run's totals once they have stopped. actors.workerIndex() is the index of the worker running the caller, or actors.noWorker.

The final runs when main returns (Running every module's initializer and finalizer). A program that panics or calls the C library's exit ends whatever its actors are doing: messages still in flight are silently dropped, and no actor still alive is finalized.

How it works#

A mailbox is Vyukov's intrusive queue of many senders and one receiver, as Pony's runtime has it: a send is one atomic exchange and one store; the worker running the actor takes messages with no read-modify-write; a mark on the mailbox says the actor is idle, and the send that finds it queues the actor. Each worker has a run queue of its own, and there is one shared inject queue: an actor made runnable by a send from a worker is queued on that worker, one made runnable by any other thread on the inject queue. A worker with nothing of its own takes a share of the inject queue, then steals half of another worker's queue. An idle worker parks on the futex, and is woken, one at a time, as work arrives. Quiescence is found by counting the activations each worker begins and ends, and reading the counts twice.

The parallel loop underneath#

A parallel loop runs on the same workers. The language writes one as parallel each, which in an actor's behaviour cuts the behaviour at the loop's end; below it is the low-level form it is written over, called as a library function:

import actors;

// the body is a closure taking a sub-range [lo, hi); it lists the pointer it writes through
actors.parallelRange(0usize, 100000usize, 256usize, fn(lo usize, hi usize) [out] {
  mut i = lo;
  while i < hi {*(out + i) = u64.from(i) * u64.from(i); i = i + 1usize;}
});

actors.parallelRange(lo, hi, grain, body) calls body(a, b) over pieces of [lo, hi) of at most grain items, on any worker and at once, and returns when every piece has finished. The body is any callable taking two usize: a closure written at the call (Closures), a function reference, or a struct with a (); it is a [F fn(lo usize, hi usize) + Shareable] (Generics), lent for the call and compiled for each body, so the call of it from a piece is the body itself. Its pieces run at once, so the compiler refuses a body that is not Shareable: one that changes a variable it borrows, borrows a &mut, or holds an Rc. The loop starts as one task for the whole range; whoever runs a task splits it in half while it is longer than the grain, puts one half on top of its worker's queue of tasks, and works the other, and an idle worker steals the oldest, largest half, so uneven work spreads itself and nothing is divided ahead of time. A worker takes its own tasks before its actors, and a worker that steals prefers tasks to actors. The caller works on its own loop while it waits, so the loop may be called from main (it starts the workers if no actor has), from any thread, from inside the body of another loop, and from inside an actor's behaviour: there the actor is not run again until the loop has finished, while its mailbox goes on accepting messages, which are handled afterwards in order. It trusts its caller beyond that check: the body must write only its own sub-range's items (through a raw pointer or an atomic), must not await, and must not wait on another piece; nothing checks it. actors.parallelRangeThen(lo, hi, grain, body, onDone) starts the same loop and returns at once, and the piece that finishes last calls onDone() on whichever thread it runs. Both callables outlive the call, so they are moved in and kept in owners until the loop has finished: closures that list what they use (a closure that borrows a local is refused), the body Sendable and Shareable, onDone Sendable. actors.parallelEach(lo, hi, piece, data) is the loop a parallel each becomes, called by the compiler's code with the loop it outlined into a function reference and a pointer to the record of the variables it uses (the one place that form stays): the same loop with the grain chosen for the workers (pieces of about an eighth of a worker's even share, but never split below 128 items, so a loop of 128 items or fewer is one piece, run at once on the caller's thread), and with the messages the pieces send to actors gathered: a piece's sends are kept, not made, and made after the loop by the code that began it, in the order of the passes, so that an actor sees the same order every run (parallelRange sends at once, in no defined order). actors.parSum, parFold and parFindFirst are the functions the parallel reductions (xs.parallel().sum()) become: they cut a view into runs by its item count alone, never by the workers, run the runs on the same workers and join their results pairwise up a tree fixed by the count, so the result is the same on every run with any number of workers.

Queued methods: qfn (not built)#

Any particular channel's messages are all the same type, but might very well be some sort of variant type, allowing different kinds of messages to be sent. Cone makes it easier to support this kind of message-passing by supporting the definition of queued-methods for the message type. For example:

struct CommandMessage {
  pub counter u32;
  pub qfn increment() {
    ++counter;
  }
  pub qfn decrement() {
    --counter;
  }
}

A message can be sent to the actor in what looks like a normal method call:

actor.increment();

However, instead of executing the function right away, the message is appended to the actor's queue. Later, the actor can pop this message off and dispatch it, which automatically then dispatches to the appropriate method's logic, updating the counter that is part of the actor's state.

A "qfn" will never specify a return type, because it is always being performed asynchronously with no guarantee of any value ever being calculated or returned. If a return value is desired, then one can pass the along a reference to the actor to which a message back can be sent.