using, because the JSON package does not; (6) the JSON the
output example writes does not escape quotes, and the page says so. The page has no numbers.
Unresolved links: the playground ("Try it", all #play-TBD), the manual pages
(../coneref/…, copied from the memory page's convention and unverified), and the sibling pages
concurrency.html and modularity.html (being drafted tonight). Code that appears here is real and compiles; where the
page shows part of a file, the caption says so.
Every program I have written does three things. It takes data in, it works on the data, and it sends data out. IBM taught this long ago as HIPO: hierarchy, input, process, output. We talk about the middle as if it were the program. But nearly always the data arrives in one format, I have to turn it into another before I can work on it, and I have to turn it into a third to send it back out. A tremendous amount of a program's time goes into that phase shift, and into the code that does it.
Web 3D shows it plainly. A model arrives as JSON and binary buffers, the engine wants arrays of vertices, and the network wants packets. This page is about the two shifts at either end of the middle, and how Cone helps with each. Input is built, with the tools I already had. Output is built in part, and I will mark what is still only a design. After it you can read and write a small parser in Cone, and you will know which half of the output story is finished.
I will use one small example throughout: a script that places things in a scene.
put cube 3,4 put sphere 10,-2 move cube 1,1
Data in: a loop and a table of cases
Parsing is where the first shift happens, and I wanted it to read like the grammar it implements. In most
languages a hand-written parser is a loop full of ifs, and the grammar disappears into them. A table
keeps the grammar visible. Cone has two things that make a table: loops, and match.
The key is that a match can use an ordinary function as a pattern. Nothing is declared to make a
function a pattern. A case names it with using, and the matched value becomes the function's first
argument. If the function gives back a bool, the case only tests. If it gives back an
Option, the case tests and binds: it is taken when the function gives Some, and the name
before using gets what is inside. So a pattern that recognizes a prefix returns the text left after
it, and the case carries on with that text:
// The text after p, if the text starts with it
fn lit(text: &str, p: &str) Option[&str] {
if text.startsWith(p) { Some(&text[p.len() ..]) } else { None[&str] }
}
// The text after a run of letters, if it starts with one
fn letters(text: &str) Option[&str] {
var i = 0usize;
while i < text.len() && text[i] >= 'a' && text[i] <= 'z' { i += 1; }
if i > 0 { Some(&text[i ..]) } else { None[&str] }
}
data_parse.cone (the number pattern, two small helpers and the imports are left out)
The loops are inside the patterns, and the grammar is outside them. A line of my script starts with
put or move , and each is followed by a name and two numbers. That is three lines:
// The grammar of a line, as a table of what it may start with
fn parse(line: &str) Option[Cmd] {
match line {
case let rest using lit("put ") { args(Verb.Put, rest) }
case let rest using lit("move ") { args(Verb.Move, rest) }
else { None[Cmd] }
}
}
data_parse.cone
Cases are tried from the top, a pattern's function runs only when its case is reached, and a pattern never consumes anything: nothing moves until a case has matched. For a sequence of parts, such as a name, a space, a number, a comma and a number, there is a flat form that leaves when a part is missing:
// "cube 3,4": a name, a space, a number, a comma, a number, and nothing more
fn args(verb: Verb, text: &str) Option[Cmd] {
let a using letters() = text else { return None[Cmd]; };
let b using lit(" ") = a else { return None[Cmd]; };
let c using number() = b else { return None[Cmd]; };
let d using lit(",") = c else { return None[Cmd]; };
let e using number() = d else { return None[Cmd]; };
if e.len() != 0 { return None[Cmd]; }
Some(Cmd(verb, String.from(before(text, a)), valueOf(before(b, c)), valueOf(before(d, e))))
}
data_parse.cone
Each line of args is one step of the grammar, and the first step that fails ends the parse. The main
loop walks the script with each and matches what parse gave back:
1: put cube at 3, 4 2: put sphere at 10, -2 3: move cube by 1, 1 4: not understood: put cube three,four
Try it: a scene script, parsed
The last line is the point of writing the parser this way. A line that does not fit the grammar is not a crash
or a silent wrong answer. It is a value, None, and the program says what it will do about it.
I do not think this is the whole toolkit yet. It is the part I have, and it has carried more than this small example. Cone's packages read and write JSON (as a tree, as a stream of events, and indented for people), a subset of TOML, URLs, glTF models, PNG and JPEG images, and ZIP archives, all in Cone. I want to come back to parsing and see what else a language should give the person who writes one.
The middle: the compiler checks you handled every kind
Once the data is in a shape I chose, the middle is ordinary work, with one help worth naming. My
Verb is an enum, and a match over an enum must say what happens to every variant. Add a
third verb to the format, and the compiler walks me to each place that has to learn it:
enum Verb { Put, Move, Delete }
fn describe(v: Verb) {
match v {
case Put { print <- "put\n"; }
case Move { print <- "move\n"; }
}
}
data_refused_verb.cone
Error 1417: This match does not cover Delete: add a case for each, or end the match with 'else {}'.
--> match v {
^--- data_refused_verb.cone:8:3
That is the change staying small: one new line in the grammar table, and one new case wherever the verb is used. The modularity page has more of it.
Data out: one operator, and the receiver decides
The second shift starts with the same question turned around: how do I turn the data I have into the format the
other side needs, and send it? In Cone, the answer starts with <-. It came out of an early builder
pattern, which is why it builds collections as well as text, and why a loop can sit inside what would ordinarily
be string interpolation. It adds what is on its right to whatever is on its left, and the thing on its left decides
what that means. Here the same scene leaves twice, once as a report for a person and once as a line of JSON for a
program:
fn main() {
var scene = List[Thing]();
scene <- Thing(String.from("cube"), 4, 5), Thing(String.from("sphere"), 10, -2);
// For a person: the loop sits among the words
print <- "The scene has ", scene.len(), " things:\n";
each t in scene.iter() {
print <- " ", t.name.view(), " at ", t.x, ", ", t.y, "\n";
}
// For a program: the same things, as one line of JSON built into a String
var json = String.empty();
json <- "[", each i, t in scene.indexed() yield entry(i, t).view(), "]";
print <- json.view(), "\n";
}
data_out.cone (Thing and entry, which writes one object, are left out)
The scene has 2 things:
cube at 4, 5
sphere at 10, -2
[{"name":"cube","at":[4,5]},{"name":"sphere","at":[10,-2]}]
Try it: one scene, two formats
Look at the JSON line. An each sits in the middle of the list that <- sends, between
the opening bracket and the closing one. That is the builder showing through.
The same operator builds other things, because each receiver says what it does with what it is sent. Here it
builds a list from a loop, a dictionary from pairs, a list one line at a time inside a with block, and
then it sends values into a channel:
let xs = List[i64]() <- each t in scene if t.x > 5 yield t.x * 2;
let rows = Dict[String, i64]() <- each t in scene yield String.from(t.name): t.y;
var heights = List[i64]();
with &mut heights {
<- 3;
<- 9;
}
tx <- each t in scene yield t.y;
data_builder.cone (declarations, the channel's setup and the printing are left out; the channel is the one in the concurrency page)
xs has 2 items, the first 20 sphere is at y = -2 heights has 2 items the channel holds 5 first
Try it: one operator, four receivers
What output still lacks
Writing those two examples showed me where the design runs out. Three limits, all real today.
First, my own types cannot say how they are written. Printing a struct I defined is refused, because the only
thing that can say how a value is written is the receiver, and print has no way to write a Thing:
struct Thing { pub name: &str; pub x: i64; pub y: i64; }
fn main() {
let t = Thing("cube", 4, 5);
print <- t, "\n";
}
data_refused_display.cone (its module and import lines left out)
Error 1056: No method declared by `<-` accepts the call's arguments.
--> print <- t, "\n";
^--- data_refused_display.cone:10:12
Second, generation and sending are one act. The loop that walks the data and the receiver that takes the text
are fused in a single statement, so I cannot keep what I generated and send it somewhere else later. And the
small things are mine to do by hand: a comma between items needed the position passed in (that is
entry(i, t)), and nothing escapes a quote in a name, so the JSON above is wrong for a name that has
one.
Third, how a value looks is not separate from what it says. There is no place to say that a number is wide, or in hex, or that a word is an error, apart from the content. A String can be padded to a width after it is built, and hex digits come from a method of String, but that is the writer fussing with text, not a description of form.
Where I am taking it: output as the dual of input
Parsing takes a shape apart: a match looks at text and says what it is made of. I think output
should do the opposite with the same ease: a grammar that builds a shape. The input side is a table of what the
text may start with. The output side should be a way to say what you are generating and where it goes, in the
pattern that suits you, and simply.
When I looked for others who had done pieces of this, the best ones all had one thing in common. They put a
model in the middle, between producing the data and writing it out: a document tree in Pandoc, the data model in
serde, the DOM in HTML and CSS. Then producers and writers vary independently, and the look of the output is
declared apart from its content, with CSS the clearest example. The weak ones fuse everything into one string
at once, as printf does. Cone's <- today is on the weak side of that line, and the
limits above are why.
print <- t works and the
receiver can still choose the format. Generation separated from sending, so the same walk can feed a screen, a
file or a socket. Form kept apart from content, so a stylesheet-like default can say how a type looks and a
single call can override it. Escaping owned by the writer, not by whoever remembered it. I have not settled the
design, and I do not know that anyone has put the whole picture together. I would rather get it right than add
another spelling for every case.
Data in and data out is most of what programs do, and the middle is better when the ends are easy. Input is something you can use today. Output works today, and is the half I am still designing. If you want to follow it, the entry page has the other pages.
Further reading, in the manual: Patterns of your own: using,
Blocks and conditionals: if and match,
String, Collections (the
send operator and each inside it) and with blocks.