Embedding candela in Rust¶

candela runs inside a Rust program as a library. You register Rust functions the script can call, load a script once, and then call its functions repeatedly while the interpreter keeps its state between calls.

There are two ways to do it, and they differ in one thing: whether the compiler is in your process.

• Link candela and use Engine/Program to compile source at run time. Scripts can be edited and reloaded while the program runs.
• Link candela-vm alone and use HostRegistry/RuntimeProgram to load a .cdlb artifact built beforehand. The compiler is absent, the binary is smaller, and no source is shipped.

Everything else is shared: the same host functions, the same Value type, the same call-by-name. Start with the first if you are unsure; jump to running a precompiled artifact for the second.

Add the crate as a dependency and enable the embed feature:

[dependencies]
candela = { package = "candela-lang", version = "0.0.4", features = ["embed"] }

The crate publishes as candela-lang, because the name candela on crates.io belongs to an unrelated project. The package key renames it back, so your code keeps writing use candela::... as everything below does.

The feature changes what a fatal error does: instead of ending the process, it unwinds so the host survives and receives the error as a value. It needs a profile that unwinds, which the crate provides as embed:

cargo build --profile embed --features embed

A whole program¶

use candela::{Engine, Value};

fn main() -> Result<(), candela::Diagnostic> {
    let mut engine = Engine::new();
    engine.register_host_fn("app", "width", |name: &str| name.len() as i64);

    let mut program = engine.compile(
        r#"
        host "app" {
            int width(string);
        }

        fn banner(label: string) -> int {
            return app::width(label) + 2;
        }

        fn main() {}
        "#,
        "banner.cdl",
    )?;

    let cells = program.call("banner", &["title".into()])?;
    assert_eq!(cells, Value::Int(7));
    Ok(())
}

Engine¶

Engine holds the table of registered host functions and compiles scripts.

register_host_fn¶

engine.register_host_fn(namespace, name, closure);

Binds a Rust closure to the name a script reaches through a host block. The closure may take up to five arguments of any of these types, and return one of them or ():

The closure's signature is derived from its Rust types and checked against the script's host declaration when you compile. A disagreement in arity, argument type or return type is returned as a Diagnostic, never a panic.

A namespace is a namespace, so a host function may take a name a built-in already has: gpio::read is the int its block declares, not the read that returns a string.

A script calls a host function as app::rows(id) or as app.rows(id); both spellings name the same function. A variable takes the name back: where app holds a value, app.rows(id) is that value's own method and the block is reachable only through app::.

A closure that can fail returns Result<T, HostError> in place of T:

use candela::HostError;

engine.register_host_fn("fs", "read", |path: &str| {
    std::fs::read_to_string(path).map_err(HostError::new)
});

HostError::new takes anything that renders, so an error from the work the closure was doing carries through with map_err. What the script sees is a runtime error at the call, naming the function and repeating the message; it can be caught with catch "host_fn_error", and it reaches the host as a Diagnostic with that code when it is not. The type checked against the declaration is the T inside the Result, so the two spellings bind to the same host signature.

register_host_fn_variadic¶

engine.register_host_fn_variadic("app", "log", |args: &[Value]| {
    for arg in args {
        println!("{arg:?}");
    }
    Ok(Value::Null)
});

Binds a closure that receives every argument as a slice and returns one Value, or a HostError to raise in the script, so mixed and dynamically shaped arguments cross the boundary without a fixed Rust signature. The script must declare the function with ...:

host "app" {
    log(...);
}

No arity or type checking happens at the call site; the closure interprets what it is handed. A variadic declaration bound to a fixed closure, or the reverse, is a Diagnostic at compile time.

register_host_fn_typed¶

use candela::{HostType, Value};

engine.register_host_fn_typed(
    "gpio",
    "read",
    vec![HostType::Int],
    HostType::Int,
    |args: &[Value]| Ok(Value::Int(args[0].as_i64().unwrap_or(0))),
);

Binds a closure that takes a slice, like the variadic form, but with the signature handed over as data instead of read off a Rust closure. The declaration is checked against those types the same way, so it must not use ..., and a script calling gpio::read gets an int.

Use it when the signature is only known at run time: a plugin table, a generated binding, anything a fixed Rust closure cannot spell.

pub enum HostType {
    Int,
    Float,
    Bool,
    String,
    Unit,
    Array(Box<HostType>),
    Map(Box<HostType>),
    Struct(Vec<(String, HostType)>),
}

These are the same types the table above lists, in the order arguments are passed. Unit is candela's null, which is what a function that returns nothing declares. Map is always string-keyed, so only the value type is carried. Struct is a struct the host block declares, as its fields in declaration order; see structs in a host block.

Structs in a host block¶

A function that takes a set of options takes them as a struct the host block declares next to it. Each field gives its type and, if it has one, the value it starts with:

host "process" {
    struct StartOptions {
        cwd: string = ".",
        env: {string: string},
        end_at_exit: bool = true,
    }
    bool start(string, string[], string, StartOptions);
}

Inside the block the struct is named bare. A script names it behind the namespace, the way it calls the block's functions, and builds it like any struct, defaults included:

fn launch() -> bool {
    return process::start("java", ["-jar", "game.jar"], "game",
        process::StartOptions { cwd: "instances/a", ..Default::default() });
}

process::StartOptions::default() and a bare Default::default() in the struct's parameter work as well; see default values.

The closure is registered with register_host_fn_typed, naming the struct's fields in the order the block declares them, and receives the value as a Value::Map keyed by field name:

use candela::{HostType, Value};

engine.register_host_fn_typed(
    "process",
    "start",
    vec![
        HostType::String,
        HostType::Array(Box::new(HostType::String)),
        HostType::String,
        HostType::Struct(vec![
            ("cwd".into(), HostType::String),
            ("env".into(), HostType::Map(Box::new(HostType::String))),
            ("end_at_exit".into(), HostType::Bool),
        ]),
    ],
    HostType::Bool,
    |args: &[Value]| {
        let opts = args[3].as_map().expect("a record");
        let cwd = opts["cwd"].as_str().unwrap_or(".");
        Ok(Value::Bool(!cwd.is_empty()))
    },
);

The binding checks every field name and type against the declaration, so a registration written for another version of the struct is a host_fn_signature_mismatch, not a missing key at run time. The defaults live in the declaration: the script builds the whole value before the call, and the closure always receives every field.

A struct crosses into the host only. A host function cannot return one, since the value would come back as a map; a declaration that does is refused when it binds. A .cdlb built from the script records the struct with the program's own, so RuntimeProgram binds and passes it the same way.

register_macro¶

use candela::macros::MacroError;

engine.register_macro("lmn", |body: &str| {
    Ok::<String, MacroError>(format!("\"{}\"", body.trim()))
});

Gives lmn!( ... ) a meaning in the scripts this engine compiles. The closure receives the raw text between the parentheses, which candela does not interpret, and returns candela source for one expression, which is parsed where the macro stands. This is how a host puts its own syntax into a script; see macros for what a script author sees. An expansion may use a macro itself, up to 32 levels deep, so an expander that emits its own macro fails the compile instead of running out of stack.

Returning MacroError instead fails the compile at the macro:

pub struct MacroError {
    pub message: String,
    pub offset: Option<usize>,
}

offset is a byte offset into the region body. Set it (MacroError::at) and the diagnostic points at that position in the file the macro was written in; leave it out (MacroError::new) and it covers the whole invocation. The region ends at the parenthesis balancing the one that opened it, ignoring parentheses inside candela string literals and after //.

allow_unknown_macros¶

engine.allow_unknown_macros(true);

A macro with no registered expander fails the compile by default, naming it. Turning this on compiles it as null instead. It is for tools that read scripts written for a host they are not part of, and would otherwise report every one of that host's macros as an error; candela's own language server does this.

with_cfg¶

use candela::Cfg;

let mut cfg = Cfg::new();
cfg.enable("web");
cfg.enable_value("target", "wasm32");
let engine = candela::Engine::new().with_cfg(cfg);

Sets the flags @cfg(...) tests in the scripts this engine compiles. enable turns on a flag, so @cfg(web) holds; enable_value turns on a pair, so @cfg(target = "wasm32") holds. A flag the configuration does not enable is false. Cfg also collects from an iterator of flag names: Cfg::from_iter(["web", "debug"]).

candela gives no flag a meaning. Pick the names your host sets, document them for the people writing scripts against it, and pass the same set to every compile of one target.

A compile that does not go through an Engine, such as build_bytecode or compile_checked, reads the configuration of the Cfg::scope it runs in:

let bytes = cfg.scope(|| candela::build_bytecode(source, "main.cdl", &resolver));

The configuration covers every file the compile imports. Outside a scope, nothing is enabled.

compile¶

let mut program = engine.compile(source, filename)?;

Parses and type-checks the source, binds every host function it declares to a registered closure, and runs main once so top-level setup is done before the host makes its first call. The filename is what error reports name.

It also compiles the body of every function in the file you pass whose parameters are all annotated, at those declared types, whether or not anything calls it. An error in a function main never calls comes back from compile rather than from the first call. A function an import brought in is left for the call. The check runs before main, so a broken body is reported before any top-level setup has run.

A function with a bare parameter that nothing in the script calls is one the host calls, so compile warns about each bare parameter and checks the body with those parameters typed any. A body that does not compile that way is another warning, not an error; the function is still compiled by the first call that reaches it. The warnings come back on the program:

for warning in program.warnings() {
    eprintln!("{}: {}", warning.code, warning.message);
}

Each is a Diagnostic like an error, with a code, a plain-text message and a span. See errors for the codes.

candela::compile_checked is that check on its own, the check candela build runs before its release passes. It prints its warnings to stderr; run it inside candela::collect_warnings to get them back as Diagnostics instead. To check without opening libraries, use check.

A main is required: an embedded program runs one the way a run from the CLI does. candela check is the step that compiles a file without one.

Returns a Diagnostic when the script does not compile, when the body of an annotated function does not compile at its declared parameter types, when the script declares no main, when a declared host function has no registered closure, when a registered closure disagrees with its declaration, or when running main raises a runtime error.

check¶

let warnings = engine.check(source, filename)?;

Type-checks the source the way compile does and stops there: no Program comes back and nothing runs. It returns the warnings the compile raised, or the first error as a Diagnostic.

A check needs less than a compile. A dylib block binds the signatures it declares without opening its library, so a script whose C library a build step has not produced yet still checks; the call sites are type-checked against the declared signatures, and a signature naming a type C cannot represent is still an error. host blocks are not matched against registered closures, and main is not required. A source without main is checked as a library, so it raises no warning about bare parameters. candela check and the language server check this way, and candela::check_only is the same check for a frontend that wants the compile tables it produces.

Program¶

Program is a compiled script with live interpreter state. Registers and heap stay resident, so anything one call establishes is visible to the next. It is single-threaded and neither Send nor Sync, matching the VM.

call¶

let value = program.call("banner", &["title".into()])?;

Invokes a script function by name and returns its value, or Value::Null for a function that returns nothing. Arguments are Values; .into() covers the scalars.

The first call with a given set of argument types compiles an entry for it, and later calls with the same types reuse that entry, so a function a host calls every frame is compiled once.

The call is type-checked against the function's signature, so a wrong argument type comes back as a Diagnostic rather than corrupting the run. Annotate the parameters of a function a host calls. A function called only from the script takes its parameter types from the call site, but a host-called function has no such call site, and the annotation is what the arguments are checked against:

fn banner(label: string) -> int {
    return app::width(label) + 2;
}

Leaving the parameters bare still works, and the types are then taken from each host call. compile warns about such a parameter and checks the body with it typed any, which catches an error that does not depend on the argument. Annotate when you want a mismatch reported against the declaration rather than accepted as a new specialisation, and when you want the body checked at the types the host passes.

warnings¶

let warnings: &[Diagnostic] = program.warnings();

The warnings compile raised, in the order it raised them. A warning does not stop the compile.

Returns a Diagnostic when the function is unknown, when the arguments do not type-check, or when the call raises a runtime error.

A Program that returns a Diagnostic stays usable. Some of that error handling happens while call compiles the specialization it needs (an undeclared name at the call site, an argument that fails to type-check, a nested call that needed compiling first); whatever that compile step wrote to the program's resident tables before hitting the error is undone, so the next call runs as if the failed one had never been made. A runtime error, once the call has started running, leaves the program's globals and any host-side state exactly as the call left them; nothing after the failing point ran.

Finding macro regions¶

A build tool often needs to know where a macro is used before anything is compiled: to collect the markup in a project, to hash it, to generate assets from it. scan_regions runs the scanner the lexer runs, over plain source:

use candela::macros::scan_regions;

for region in scan_regions(&source, "lmn") {
    println!("{} at {:?}", region.body, region.span);
}

pub struct Region<'a> {
    pub body: &'a str,
    pub body_start: usize,
    pub span: Range<usize>,
}

body is the raw text between the parentheses, body_start its byte offset in the source, and span the byte range of the whole invocation. The results match what compiling the same source expands, including which name!( occurrences do not count: one written inside a string literal or after // is not an invocation. Scanning stops at a region the source never closes, so every region returned is complete.

Running a precompiled artifact¶

Build the script to a .cdlb first, with candela build or build_bytecode, then link only candela-vm in the program that runs it. Both compile in the release profile, which runs the passes that make the program faster; see the CLI reference. build_artifact takes a BuildOptions to choose the profile and the machine code to include; machine code needs the native feature of candela-lang, and without it an artifact carries none. candela-vm runs an artifact's machine code where it fits the machine, and RuntimeProgram::native_functions says how many functions run that way. The program that runs the artifact depends on the runtime alone:

[dependencies]
candela-vm = "0.0.4"

use candela_vm::{HostRegistry, Value, load_program};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let mut hosts = HostRegistry::new();
    hosts.register_host_fn("app", "width", |name: &str| name.len() as i64);

    let bytes = std::fs::read("banner.cdlb")?;
    let mut program = load_program(&bytes, &hosts)?;
    program.run();

    let cells = program.call("banner", &["title".into()])?;
    assert_eq!(cells, Value::Int(7));
    Ok(())
}

The library leaves your process's allocator as you set it. The candela-vm binary sets glibc's for itself, which makes programs that build and drop big lists much faster (see running an artifact). A host on Linux with glibc gets the same by starting with GLIBC_TUNABLES=glibc.malloc.top_pad=67108864:glibc.malloc.mmap_threshold=33554432, or by calling mallopt with those two values before it loads a program.

HostRegistry¶

The table of closures a script's host blocks bind to. register_host_fn, register_host_fn_typed and register_host_fn_variadic take the same arguments and derive the same signatures as their Engine counterparts above.

Binding happens in load_program, and it checks what compiling a script checks: every declared function must be registered, and each closure's arity, argument types and return type must match the declaration. A LoadError::HostBinding comes back naming what is missing or what disagrees, before any instruction runs. Engine holds one of these registries internally, which is why the two paths accept the same closures.

run¶

Runs main, the same way candela-vm does: a runtime error prints its report and ends the process. Call it once, before the first call, so top-level setup is done. To keep a failing script from taking the process with it, run it inside collect_diagnostic, which turns the error into a Diagnostic you can handle.

call¶

let value = program.call("banner", &["title".into()])?;

Invokes a function by name against the resident state, returning its value or Value::Null for a function that returns nothing. Arguments are checked against the declared parameter types first.

Only functions the artifact exports are callable, and program.exports() lists them. A function is exported when it is defined in the file that was built, is reachable by its bare name, is not main, and has a type a host value can be on every parameter. A bare parameter on a function nothing in the script calls is exported as any, with a build warning; see artifacts for the full rule. This is the difference that matters when moving a script from Engine to an artifact: bare parameters take their types from each host call there, but an artifact has no compiler to specialise them later, so the body runs with them typed any. Annotate them to have the body compiled at the types the host passes.

Errors come back as a CallError: the name is not exported, the argument count or an argument type disagrees with the declaration, or the call raised a runtime error, which arrives as the Diagnostic it produced.

Collecting garbage between frames¶

// After the frame's calls, where the event loop would otherwise wait.
program.collect(2000);

Program and RuntimeProgram both have collect. The collector frees heap values the program can no longer reach. Left to itself, it works while the program allocates: once a collection is under way, each allocation does a small share of it, so no allocation stops the program for long, but the frame that allocates pays for the work. A collection begins when the heap holds enough more objects than the last one left, or when the program has allocated enough more bytes than the last one found in use, so a program that builds a few large lists is collected as well as one that builds many small values. collect lets the host do that work when nothing is waiting on the program, such as between frames.

collect(budget) carries on a collection already under way, or begins one when the heap has grown enough since the last, and stops after about budget units of work. It returns true when no collection work is left, and false when a collection is still under way; the next call carries on from where this one stopped, and so do the allocations of any call in between. On a heap that has not grown, it does nothing and returns true, so calling it every frame costs next to nothing when there is nothing to collect.

The budget counts work, not time: a unit is about one object traced or one slot freed. The same calls do the same work on every machine, in a browser as well, where the runtime has no clock to go by. collect(u32::MAX) runs a whole collection in one call. The call that begins a collection reads every register at once whatever the budget, and a call can run past its budget by up to one batch of slots, so treat the budget as a target rather than a hard limit.

To pick a budget, time collect on the slowest machine you support and raise the budget while the call still fits the idle time a frame leaves. A budget too small to finish a collection between frames is safe: the allocations of the next frame pick up what it leaves.

Once a host has called collect, allocations wait for the heap to grow further before they begin a collection of their own, which leaves the work to the host. A program no host collects for keeps the default thresholds.

gc_stats¶

let stats: GcStats = program.gc_stats();

What the collector has done over the program's heap, and how large the heap is:

The arrays pool holds lists, structs, enum values and functions; strings holds strings too long to fit inside a value. There are no time fields. To see how long collecting takes, time your own collect calls.

Where an import looks¶

A script's imports resolve against two things the host can name: the standard library, and the packages it should be able to reach.

let engine = candela::Engine::new()
    .with_lib_dir(app_root.join("libs"))
    .with_import_root("shapes", cache.join("shapes/1.2.3"));

with_lib_dir names the directory holding std/, so import "std/string"; reads libs/std/string.cdl. It is what an application that ships the standard library inside its own installation uses instead of setting CANDELA_LIB_PATH. Without it, the default applies: CANDELA_LIB_PATH when set, and libs/ beside the running executable otherwise.

with_import_root makes a directory the root of a package. The package is entered through the file its candela.toml names as entry, src/main.cdl when it names none or there is no manifest, so import "shapes"; reads that file and import "shapes/circle"; reads circle.cdl beside it. This is how a host offers a script code it did not ship with, and it is the same mechanism the candela command uses for the packages a project depends on.

Both are builders, so they chain, and both take effect on every script the engine compiles. Engine::compile is unchanged.

Where a dylib import looks¶

candela::set_dylib_dirs(vec![app_root.join("lib")]);

A script's dylib import is looked for beside the script itself, which is the wrong place for an application that keeps its sources in one directory and its native libraries in another. Name those directories and they are searched first, in order, by a compile and by an artifact load alike, so dylib "md" in src/main.cdl finds lib/libmd.so.

A library the standard library owns is looked for under the toolchain's libs directory as well, after the directories named here, so an artifact that imports std/math loads wherever it runs.

The call returns the list that was in effect, so a host that changes it for one script can put the previous one back, and an empty list goes back to looking beside the script alone. It is a per-thread setting, read while a script compiles and while an artifact loads, so set it before either. candela_vm::set_dylib_dirs is the same function, for a program that links only the runtime.

Every directory named with with_import_root is searched for native libraries too, after the ones set here, so a package that ships a shared library at its root binds without further setup.

See C libraries for what a dylib import resolves to before any directory is searched.

Values¶

Value is the type that crosses the boundary in both directions:

pub enum Value {
    Null,
    Int(i64),
    Float(f64),
    Bool(bool),
    String(String),
    Array(Vec<Value>),
    Map(BTreeMap<String, Value>),
    Enum { variant: String, payload: Vec<Value> },
}

From is implemented for i64, i32, f64, bool, String, &str and (), so 5i64.into() and "title".into() build one. Going the other way, as_i64, as_f64, as_str, into_string, into_array and into_map unwrap one when it holds what you ask for.

Value::Int is an i64, the width a candela int holds, so an integer crosses either way unchanged. Arrays are homogeneous and maps are string-keyed, matching how candela types them. A struct read back from a script arrives as a Map of its fields.

A list or map argument is built in the program's heap the way one the script builds is, reusing the space of values the collector has freed. Once the call returns, it is garbage unless the script kept it, so a host that passes a fresh list every frame does not grow the heap.

An enum arrives as Value::Enum, naming the variant it holds and carrying that variant's payload in declaration order; a nullary variant has an empty payload. The enum's own type name is not carried, because the variant is what you match on:

match program.call("pick", &[Value::Int(1)])? {
    Value::Enum { variant, payload } if variant == "Line" => {
        println!("line {:?} to {:?}", payload[0], payload[1]);
    }
    other => println!("{other:?}"),
}

Enums travel outward only. A Value::Enum is refused as an argument in any position, an any parameter and a list element included, and the call comes back as an argument_type_mismatch naming the variant it was handed. A parameter typed as an enum is not host-callable at all: an artifact gives such a function no export. Build the value inside the script and return it.

A generic type is instantiated at compile time, so nothing generic reaches the host: Cell<int> crosses the boundary as the ordinary struct it compiles to.

Errors¶

Every fallible Engine/Program call returns Diagnostic:

pub struct Diagnostic {
    pub filename: String,
    pub span: Range<usize>,
    pub message: String,
    pub code: String,
}

code is a stable snake_case identifier you can match on, such as unknown_variable, argument_type_mismatch, index_out_of_bounds, unregistered_host_fn, host_fn_signature_mismatch or host_fn_error (one of your own closures returned a HostError). message is plain text with the terminal colouring removed, and span is a byte range into the file filename names, which is enough to underline the offending source yourself. See errors.

Only one diagnostic comes back per call, because compilation stops at the first error.

On the artifact path the errors are LoadError and CallError instead, both of which print themselves. A runtime error inside a call still arrives as the Diagnostic above, wrapped in CallError::Runtime.

The VM hands a runtime error back to call as a value, on both paths, so a call does not unwind and leaves the process panic hook alone. Compiling is different: the compiler recovers from an error by unwinding, and while it compiles, a hook of candela's silences that unwind and passes every other panic on to yours. That is why Engine::compile needs a profile that unwinds.

Evaluating source at run time¶

Compiling is opt-in and explicit: a host that wants to evaluate new source calls Engine::compile again. There is no eval inside the language, so a script cannot compile new code by itself, and a host that never calls compile after start-up cannot be made to.

This is also the difference between the two ways of shipping candela in a host. Linking the candela crate brings the compiler, so scripts can be compiled at run time and reloaded. Shipping precompiled .cdlb artifacts and linking only candela-vm leaves the compiler out of the process entirely, and with it any way to evaluate source at all; see artifacts.

One-shot execution¶

For running a script and collecting what it printed, without keeping any state, the crate exports a C entry point: candela_run takes source and returns the captured output, and candela_free_output releases it. Use Engine and Program for anything that calls back and forth.