Cap'n Proto in WASM

One of the main features of Wetware is the execution of safe and distributed processes. Distribution is provided and managed natively by Wetware, while safety is achieved through isolation. Isolation is achieved by running processes as Wasm guests on a restricted Wasm runtime.

Of course, distributed applications require communication; therefore processes cannot be fully isolated. Our approach is to fully isolate the process and only allow it to access the outside world through object capabilities (ocaps); realized by Cap’n Proto. The capability security model is a natural fit for distributed executions in which…

This article provides some context into how Wetware handles capabilities in isolated processes, the main challenge it presents, and the solutions we’ve developed until arriving at the latest one. May it serve as a study on how Cap’n Proto can be run in WebAssembly programs.

Cap’n Proto

Cap’n Proto (capnp), on which Wetware is based, takes it a step further by building an RPC and data serialization system based on ocaps. It has similarities with traditional structs and interfaces: structs define data types while interfaces describe behaviors. The example below shows a Cap’n Proto schema defining a Person struct, as well as a Dialer interface which can call a Person. The specific example is a function that receives a Person and returns nothing.

@0xfd66a7ca4e168218;  # unique file ID, generated by `capnp id`

struct Person {
name      @0 :Text;
birthdate @1 :Date;
emails    @2 :List(Text);
phone     @3 :Text;
}

interface Dialer {
call @0 (person :Person) -> ();
}

Interfaces defined in capnp schemas need to be implemented and provided to clients, which can be done in a variety of programming languages. Servers providing a capability and clients utilizing them are connected through a transport, which can be a TCP port, a Unix socket, a local file, an in-process pipe, or anything you can do async IO over. Calls over the network will necessarily use some networking component as transport, while local ones can be carried out through lighter means.

In a cool twist, capnp can then be used as a transport for other applications. I used it to communicate Raft nodes in my Master’s thesis a few years back.

WebAssembly

Wetware processes are run as WebAssembly (Wasm) guests: bytecode executed by a Wasm runtime such as Wasmtime, Wasmer, or Wazero. Wasm guest processes are portable and fully isolated by default: no filesystem, networking, or other IO access. They have no view into the program and OS they are running on.

Wasm is still in the works, and things are changing quickly. At the moment, many of the Wasm runtimes allow deliberately breaking the process isolation by explicitly granting it access to network ports, file descriptors, etc. This is usually done through WASI, an spec on how to grant Wasm processes access to resources by implementing system calls. This includes access to files, randomness, networking…

It’s worth mentioning all Wasm guests are single-threaded, at least for now, which will become relevant later in the article.

Wasm + Capnp

As stated earlier, Wetware isolates processes through Wasm and grants them outside access only through object capabilities. Each process is passed a capability at spawn, which is provided by the caller. A process may be passed capabilities for file system access (local or remote), networking, process execution, or anything the caller implements or has access to.

The key points are:

A process can only do what its capabilities allow it to do, and nothing more.
A process can share its capabilities, or implement and share new ones. The end result is a clean, safe, and intuitive way of building and running distributed applications.

The initial Wetware implementation implemented a process executor capability. Processes spawned by the executor were given capabilities to spawn new processes and communicate with other processes. Executors were also given capabilities to other executors, which were running in separate Wetware nodes. The result was a fully distributed process executor that somewhat resembled the erlang model, with trees, linked processes, and a few more cool features. Inter-process communication was made through capabilities, allowing for direct message passing, or even pub/sub broadcasting. Because sending a message to a process involved invoking its capability, it does not matter if the process is running on the same executor or in a remote node. We included a distributed, p2p web-crawler running entirely on Wetware as a demo.

The Problem

Capnp requires a non-blocking transport, but Wasm guests are single-threaded and don’t support async IO. We solved this issue in the initial version of Wetware by using a TCP port as transport and granting the guest access to it through WASI. The guest would then bootstrap a capability from the socket, and have a working object capability. This is a patchy solution that allocates a TCP port (with the small-ish performance penalty that implies) and, worse, gives the guest unrestricted access to the TCP port. Untrusted processes might choose to do something other than bootstrap a capability over the port, and the runtime would be none the wiser. You can find a more detailed explanation here.

The solution we moved into was implementing async IO calls as exported Wasm functions that were mapped to an async transport by the host, then importing them from the client and mapping them to system calls that allow for async IO. This turned out to be quite an ordeal, which Louis got working in a later version of Wetware.

The Rewrite

At some point we decided to rewrite Wetware. We opted for switching Go for Rust for the following reasons:

Wasm performance: Wasm programs built from Rust are significantly faster, lighter, and produce smaller binaries than those built from Go, which need to include the Go runtime in the Wasm transpilation.
Runtime performance: we could still write Wasm in Rust, but build the runtime in Go. Instead, switching the host code to Rust allows us to keep the ecosystem uniform and benefit from faster Wasm runtimes.
Keep gaining experience with Rust. With the rewrite came the opportunity to solve the single-threaded capnp transport problem in a cleaner way. WASI Preview 2 recently introduced non-blocking options for guest stdin and stdout, which can then be used to implement the AsyncRead and AsyncWrite traits that capnp-rust requires for its transports.

After some deadlock debugging, we’ve created a proof of concept that successfully runs many concurrent capnp calls, forcing the transport to handle thousands of concurrent read/write operations in a single-threaded environment. The end solution is simpler, more maintainable, and more performant than our previous approaches. Excuse the error type gymnastics below.

The AsyncRead implementation takes advantage of the wasip2::io::streams crate, which immediately returns 0 when there is nothing to read.

impl futures::io::AsyncRead for Stdin {
fn poll_read(
self: std::pin::Pin<&mut Self>,
cx: &mut Context<'_>,
buf: &mut [u8],
) -> Poll<io::Result<usize>> {
// Non-blocking read: try to read available bytes; if none, yield Pending and self-wake.
let len = buf.len() as u64;
match self.stream.read(len) {
Ok(bytes) => {
let n = bytes.len();
if n == 0 {
// No data ready yet; yield and try again later.
cx.waker().wake_by_ref();
return Poll::Pending;
}
buf[..n].copy_from_slice(&bytes);
Poll::Ready(Ok(n))
}
Err(e) => Poll::Ready(Err(io::Error::new(io::ErrorKind::Other, format!("{e:?}")))),
}
}
}

The write implementation, however, relies on the write immediately completing; either because the other end read it or the data was buffered. Our specific case has buffered pipes backed by a multi-threaded host on the other side, and the capability calls will be quickly attended by the provider unless the CPU is under extreme load.

impl futures::io::AsyncWrite for Stdout {
fn poll_write(
self: std::pin::Pin<&mut Self>,
_cx: &mut Context<'_>,
buf: &[u8],
) -> Poll<io::Result<usize>> {
// Ensure we don't misreport partial writes: use blocking_write_and_flush so the
// entire buffer is committed before returning. This avoids frame truncation that can
// deadlock capnp on subsequent reads.
if buf.is_empty() {
return Poll::Ready(Ok(0));
}
match self.stream.blocking_write_and_flush(buf) {
Ok(()) => Poll::Ready(Ok(buf.len())),
Err(e) => Poll::Ready(Err(io::Error::new(io::ErrorKind::Other, format!("{e:?}")))),
}
}

fn poll_flush(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
match self.stream.blocking_flush() {
Ok(()) => Poll::Ready(Ok(())),
Err(e) => Poll::Ready(Err(io::Error::new(io::ErrorKind::Other, format!("{e:?}")))),
}
}

fn poll_close(self: std::pin::Pin<&mut Self>, _cx: &mut Context<'_>) -> Poll<io::Result<()>> {
match self.stream.blocking_flush() {
Ok(()) => Poll::Ready(Ok(())),
Err(e) => Poll::Ready(Err(io::Error::new(io::ErrorKind::Other, format!("{e:?}")))),
}
}
}

Next Steps

The PoC repository demonstrates client calls from Wasm guests work without issues. Our next steps will be to provide capabilities from within Wasm guests, trace any issues we encounter, and incorporate the solution into the repository.

Keep an eye on the wetware/rs for updates!