Closing the year before heading to 39th Chaos Communication Congress we decided to tag a release for the main project for those too conservative to run from the main branch.

For those on site that wants to have a chat, our DECT extension is 6681. Incidentally the same number as the control port for when Arcan was used to provide overlay, texting, memes, scheduling etc. for a pirate university TV station back in the early 2000s.

Before going into the project updates, there is tragic news to relay that put a massive damper on spirits and overall productivity for the last few months: Our most beloved of project members, Elijah “moon-child” Stone died on the 9th of September at the young age of 22. Elijah had been with the project since his early teens and was consistently kind, caring, courteous and clever well beyond his years. The 0.8 topic branch on one of his favourite subjects, performance engineering, will be dedicated to his memory. Our thoughts are with his family and partner.

Community Updates

It has been quite a while since we switched from GitHub to Fossil for development. Since we don’t expect people to tool around an uncommon tool we also mirror to git hosted on Codeberg.

A final friendly warning to packagers is to use those repositories. The ones on GitHub will no longer receive any mirror updates and any changes over there are likely to be of a more incendiary nature.

As covered in more detail, (part 1, part 2) we have had a longer in-person hackathon. The main outputs from those is that Alexander has a port of Gamescope to save us from keeping up to date with all the special quirks that running Steam over Xwayland takes. The clip below shows that running Baldur’s Gate 3.

Magnus keeps chipping away at his platform plugin for Qt5/Qt6 that largely works as intended for the likes of Qbittorrent and Binary Ninja but struggles still with hybrid 2D/3D complex window managed applications like FreeCad.

Valts is busy with his portable viewer for the A12 protocol that should soon be usable with some of the neat bits we cover further below.

In the bin of experimental applications, we have Atro with “Lasso” that is a hybrid ‘interactive canvas’ form of window manager.

On top of providing a number of bug fixes across the board, Bohdan created Xkbd2Lua for statically translating X Keyboard Layouts to our own format, removing the need to lug libxkbcommon around for those that want even fewer traces of X11 around in their lives.

Ariel has been toiling away at one approach for a bootable complete Arcan+Durden+Cat9 setup from as static a build as possible matching Arcan as OS design (there will be more). As part of that there is a nix oneliner that should hopefully work on a few setups:

nix run --impure 'git+https://codeberg.org/ingenieroariel/arcan?ref=nix-flake-build&dir=nix'

Status Update: Arcan

As normal, check the changelog for the fine grained changes.

A lot has happened on the network side, thanks in large part to the continued support from NLnet and a longer write up for that is coming up in a little while.

To start with, there is now support for ML-KEM as Post-Quantum cryptography to protect against ‘collect now, decrypt later’ (should the fantasy computers ever materialise). This is implemented as part of the forward secrecy ratcheting rekeying process. The same has been extended to set a signature verification key for file transfers and a proof of work scheme for load balancing search requests.

Connection resumption and Casting

Clients that act as sources now has connection resumption support. This means that if the network connection is lost, the source application is kept alive and re-paired when you connect back.

In the following clip I first host arcan running pipeworld on my networked machine and connect. The window pops up and I create a few cells to show that there is data/state remotely. I close the window and reconnect. The window reappears just as I left it.

Showing connection resumption on a arcan-net hosted client

There is also a –cast option added. This lets the first user that connects become the “driver” over the hosted application. Any subsequent connections gets a read-only copy of the stream.

Unified and Referential Links

The other major network changes has to do with the directory server part. To recap, any endpoint can have one out of three roles: Source, Sink or Directory. Normally a source hosts some kind of application and provides, either inbound or outbound, access to a Sink. The Directory works as a self-hosted rendezvous for discovery, but also state, file-store and coordination between multiple clients using the same appl.

For instance, if I have a directory server hosting the Durden desktop appl and multiple clients download and run it from the directory, they can use the messaging domain that the directory provides to synch clipboard state or share input devices like one would do with Synergy/Barrier in the past.

The admin API for configuring the directory server has received two new functions, reference_directory for referential links and link_directory for unified links.

A referential link lets users with access to directory server A access a referenced directory B, forming larger networks. Using the command-line tool, the following example:

arcan-net --path myfriend myserver@ myappl

Would have the client connect to *myserver*, then download and run *myappl* from the referenced directory *myfriend*. The path can/be/arbitrary/long. Connection primitives are negotiated by each step in the chain, which takes us into the very interesting space of transitive trust-discovery models.

In the example above, myserver gets a DIROPEN request for myfriend from the client. It sees that this is a referential link, and asks myserver for connection primitives on behalf of the client – forwarding the public key used by the client to authenticate to myserver. Myfriend returns with either direct connection information (ip, port etc.) or a request that myserver tunnels the traffic if it is not directly reachable.

The unified link is invisible to the user and is a more privileged connection. It lets multiple directories form a shared namespace such that they can access/host/mirror the same resources as if they were one logical server.

Say that you have a server in your home and another hosted on a VPS somewhere. With a unified link between the two, your devices can access the one when you are at home, and hand over to the other when you go outside.

Dynamically Hosted Directed Sources

For every directory hosted appl it is possible to slot in a controller. A controller is a set of server side scripts that regulate messaging and resource access for more advanced networked applications. The scripting API has received some new functions that are worth looking into.

The first is launch_target. This is best explained through an example. Say my directory server configuration database has a launch target defined, like:

arcan_db add_target "chromium" /usr/bin/Xarcan -redirect -exec chromium

I then slot in a controller for the ‘durden’ appl by doing this:

# mkdir durden; echo "
function durden_join(cl)
launch_target("chromium", {}, cl)
end " > durden/durden.lua

# arcan-net --sign-tag mytag --push-ctrl durden myserver@

This would package the controller for durden, sign it with whatever is assigned to mytag in the local keystore, and upload it to the directory server marked as myserver in local the keystore.

The server verifies permissions and that signing key match previous signatures, and assigns a runner VM. Now the next time any client would run the durden appl:

# arcan-net myserver@ durden

It would spawn an instance of chromium that connects as a source only visible to the specific client, with a polite suggestion to source it immediately. The client does so automatically and the window pops up.

There are more controls to add here for state management, sandboxing details, letting the controller script inject events and so on. The point of the feature is that the beefier server now has a mechanism for fine grained application hosting.

Speaking of, if the client end doesn’t have the full Arcan stack but something simpler like the Smash viewer, it is not possible (opt-in) to let the directory server host the arcan side of the equation turning the client side into a very thin one:

# arcan-net --host-appl myserver@ durden

External Resource Resolver

Other changes to the controller development side is that the event handlers used to to list/download/upload files can now be hooked up to an external resolver.

The way it normally works is that you add an event handler, like this:

function durden_load(cl, name)    return nameend

This does nothing fancy, it just forwards whatever the client requested to the server-side file-store. The function in this form is just to add any client specific block/name translation. Another options would be to return the result of open_nonblock to dynamically generate the data to be transferred.

If I modify the server’s config.lua to this:

function ready()    launch_resolver("durden", "/usr/bin/myresolver")end

Any resource requests forwarded from the runner scripts to the ‘durden’ appl will be routed to an external process. This is any shmif client that implements a handler for the BCHUNK_IN / BCHUNK_OUT events. A simple test client can look like this: https://codeberg.org/letoram/arcan/src/branch/master/tests/core/a12resolve/a12resolve.c

Had this been the assigned resolver, storage requests would be rejected outright and load requests would get the ‘test.mp4’ file contents regardless of what was asked for. The point of this feature is to provide caching and translation to other file providers. Ones currently being looked into covers regular URLs, Magnet-to-torrent and IPFS.

Custom Debugger Integration

The remainder of changes are mostly on the developer side of things. Recall that we once wrote a debugger frontend to Cat9. This implemented the Debug Adapter Protocol, but was designed in such a way that we could support other protocols. DAP is too heavy for our own needs, so a lighter one was thrown together to match how we use the Lua VM in both the engine and the directory server. This lets us debug locally as well as with a corresponding directory controller in lockstep mapped as remote threads.

The following clip shows the current state of that in Cat9:

Using cat9 debug frontend to attach, break and step arcan running pipeworld

It can also be attached on-demand on a scripting error. There is some heavy lifting left on the frontend side before this is completely seemless, but in a not too distant future we will be at a point where you can pause-edit-update-continue a fleet of devices running a single Arcan application at once.

With that we have all the building blocks in place for more interesting networked Arcan applications. The first target for that will be a community chat application to move us away from Discord.