In this installment we will see how to show something useful that integrates Rust + Lazarus/Free Pascal.

All the code and more can be found in the linked repo.

A simple JSON <> MessagePack converter

The Rust Library

We want to build a Rust library that handles the conversion between JSON and MessagePack (and viceversa). If you are new to MessagePack, think of it as a "compiled" binary JSON, a way to reduce JSON size bot in transmission and at rest. The problem with MessagePack is that, contrary to JSON, it is not human readable. That is where a JSON <> MessagePack converter could find its use case. So, let’s build one!

As for the Rust library, let’s create a new lib such as:

cargo new jsonmplib --lib

Let’s add what we need:

cargo add serde serde_json rmp-serde base64

Let’s have a thought on what we need:

• A function to convert JSON to MP (MessagePack)
• A function to convert MP to JSON
• Data structures to pass around between Rust and Pascal

As MessagePack is a binary format, we will represent it with a struct that can hold a pointer to the binary data (bytes = u8) as a C-style vector, and a dimension of the vector itself (least we read data that is not present in the vector).

#[repr(C)]
pub struct MpBuffer {
pub data: *mut u8,
pub len: c_int,
}

In this way we represent a nice old C-Style memory buffer.

To free the buffer’s memory we need a function that ca be called also over the fence, on Pascal’s side:

#[unsafe(no_mangle)]
pub extern "C" fn free_mpbuffer(buf: MpBuffer) {
if !buf.data.is_null() {
unsafe {
drop(Vec::from_raw_parts(
buf.data,
buf.len as usize,
buf.len as usize,
));
}
}
}

In the above, if the data pointer is not null we build a vector from "raw parts, i.e., pointer, length and capacity, and then drop it (remember it’s an unsafe operation.)

Yes capacity can be different from the length as the length is the dimension of the data, capacity is the reserved space fro the buffer, so that the actual dimension can grow up to take the whole vector capacity; for convenience we will set non-growing buffers (i.e., fixed space), so length and capacity are the same.

Now, onto the first function: convert JSON to MessagePack, saving it in the memory buffer.

First we’ll create an empty() constructor for our MpBuffer, so that we can create empty buffers to return when there’s no JSON and on error:

impl MpBuffer {
pub fn empty() -> Self {
MpBuffer {
data: ptr::null_mut(),
len: 0,
}
}
}

Then, the actual conversion function:

#[unsafe(no_mangle)]
pub extern "C" fn json_to_msgpack(json_ptr: *const c_char) -> MpBuffer {
if json_ptr.is_null() {
return MpBuffer::empty();
}

unsafe {
match CStr::from_ptr(json_ptr).to_str() {
Ok(json_str) => {
// Parse JSON
match serde_json::from_str::<Value>(json_str) {
Ok(value) => {
// Encode to MessagePack
match to_vec(&value) {
Ok(vec) => {
let len = vec.len();
let mut vec = vec;
let data = vec.as_mut_ptr();

// Avoid dropping the Vec
std::mem::forget(vec);
let c_len = len as c_int;

MpBuffer { data, len: c_len }
}
Err(_) => MpBuffer::empty(),
}
}
Err(_) => MpBuffer::empty(),
}
}
Err(_) => MpBuffer::empty(),
}
}
}

The function is easy enough: we accept a C-style string (pointer) with the JSON, and we convert this to MessagePack, saving it in the buffer.

The first thing we do is to return an empty buffer if the string is empty (pointer to null); then we try to construct the C-string form the pointer, matching the result: on error we return an empty MpBuffer throughout the function.

When we have our Cstring as Ok value, we parse it as JSON, and once this is Ok we encode it to a MessagePack Vec and we get a pointer out of it, and its length (converted to c-Syle int, c_int) to wrap it into the MpBuffer. Finally, when we no longer need it, we do not drop the MessagePack Vec; instead, we use std::mem::forget to "deconstruct" it cleanly.

As for the conversion back from MessagePack to JSON, the process is much easier, as we just return a C-style string (a pointer c_char), so we do not need a memory buffer for it:

#[unsafe(no_mangle)]
pub extern "C" fn msgpack_to_json(data: *const u8, len: c_int) -> *mut c_char {
if data.is_null() || len == 0 {
return ptr::null_mut();
}

unsafe {
let slice = std::slice::from_raw_parts(data, len as usize);

match from_slice::<Value>(slice) {
Ok(value) => match serde_json::to_string_pretty(&value) {
Ok(json_str) => CString::new(json_str).unwrap().into_raw(),
Err(_) => ptr::null_mut(),
},
Err(_) => ptr::null_mut(),
}
}
}

In the above function we get the MessagePack as data and length, unpacking as to say, the MpBuffer struct, so we can call it by literally unpacking the corresponding Pascal’s record.

We get a null pointer on an empty (or null) MessagePack, and throughout the function on error.

For the actual conversion we get a slice from the raw parts of data and length of the memory buffer, and we use the from_slice on it, to convert MessagePack to serde_json’s Value. If everything is Ok, we prettify it and return it as a C-string, nice and easy.

Once we have done with the basic functions, the final touch is to have a function that we’ll call from Pascal in order to free the memory occupied by the c-strings in Rust:

#[unsafe(no_mangle)]
pub extern "C" fn free_cstring(s: *mut c_char) {
if !s.is_null() {
unsafe {
drop(CString::from_raw(s));
}
}
}

If you got the foregoing, this is almost trivial: we just wrap a drop() on a Cstring constructed "from raw" pointer to c_char.

With these data structures and function,we have finished on Rust’s side. Now, after building we can copy the resulting library to the usual places (here I’m on Linux, it must be copied to /usr/lib/, on Windows it’s sufficient to copy it at the root of the Lazarus project, near the .exe that will be produced):

cargo build --release
sudo cp target/release/libjsonmplib.so /usr/lib/libjsonmplib.so

We can now pass the ball on Pascal’s court to see how to handle this new library.

The Pascal’s side of things

We will create a simpledemo folder and a new Lazarus project select a New.. Project/Simple Program called jsonmpdemo. As usual, let’s create first the .pas unit jsonmplib.pas to interface with the Rust library:

unit jsonmplib;

{$mode ObjFPC}{$H+}

interface

uses
Classes, SysUtils;

type
PMpBuffer = ^TMpBuffer;

TMpBuffer = record
Data: pbyte;
len: longint;
end;

function json_to_msgpack(jsonStr: pchar): TMpBuffer; cdecl; external 'libjsonmplib';

function msgpack_to_json(Data: pbyte; len: longint): pchar; cdecl; external 'libjsonmplib';

procedure free_mpbuffer(buf: TMpBuffer); cdecl; external 'libjsonmplib';

procedure free_cstring(str: pchar); cdecl; external 'libjsonmplib';

implementation

end.

If you have followed the style we use in the first tutorial, this code shouldn’t come with lots of surprises, but let’s review it briefly:

• we create a TMpBuffer record to map our Rust MpBuffer struct, and a pointer for it (PMpBuffer = ^TMpBuffer)
• after this, we map all of Rust’s extern C functions to Pascal ones with an external 'libjsonmplib' for each. No implementation needed as the Rust library is the implementation.

Now as for some code to use the unit, inside the jsonmpdemo.lpr:

program jsonmpdemo;

{$mode ObjFPC}{$H+}

uses
jsonmplib, Classes, SysUtils;

var
buf: TMpBuffer;
jsonStr: PChar;
msgpackStream: TMemoryStream;

begin
// Convert JSON to MessagePack
buf := json_to_msgpack(PChar('{"x":1,"y":2}'));

if (buf.data <> nil) and (buf.len > 0) then
begin
msgpackStream := TMemoryStream.Create;
try
// Write MessagePack bytes to stream/file
msgpackStream.WriteBuffer(buf.data^, buf.len);
msgpackStream.SaveToFile('out.msgpack');
finally
msgpackStream.Free;
end;
free_mpbuffer(buf);   // Rust-allocated memory must be freed
end
else
Writeln('ERROR: json_to_msgpack returned empty buffer.');

// Now read from file back into a NEW stream
msgpackStream := TMemoryStream.Create;
try
msgpackStream.LoadFromFile('out.msgpack');
msgpackStream.Position := 0;

// Convert MessagePack -> JSON
jsonStr := msgpack_to_json(msgpackStream.Memory, msgpackStream.Size);

if jsonStr <> nil then
begin
Writeln('Decoded JSON: ', jsonStr);
free_cstring(jsonStr); // Rust-allocated memory must be freed
end
else
Writeln('ERROR: msgpack_to_json returned NULL.');
finally
msgpackStream.Free;
end;
end.

The only "complication" here is that we’re handling FPC’s memory streams, as we need to handle MessagePack which of course is a binary stream.

In the code above, we first create a TMpBuffer record using the function json_to_msgpack with a JSON object {"x":1,"y":2} then we save the data (binary part) to a file, using a TMemoryStream to handle the binary format, as said. We then free the streaming, so that the data is lost to the program, and so to prove that we can indeed write and read MessagePack to and from a file. For this reason we free also the record (which is kept in the Rust library’s memory, so we use one of the function we prepared in Rust: free_mpbuffer).

Reading back the lost data from the saved file is precisely what we do next: we allocate again the TMemoryStream and we load it with bytes taken from the file we saved earlier. Then we use again the library to convert that byte stream into a JSON object with the function msgpack_to_json, and show it to the user. After this we free again all memory, both in Rust and Lazarus.

Once the program is compiled, the output is as expected:

./jsonmpdemo

Decoded JSON: {
"x": 1,
"y": 2
}

It is even pretty printed (the original JSON object was not, remember?).

Advanced GUI

Let’s see if we can create a more interesting application out of this library. This time we create a New.. Project/Application in Lazarus.

We need to place graphical components on the main form here called Form1. Take a look at this hierarchy:

Form1: TForm1
Panel1: TPanel (alTop)
btnConvert: TButton (alLeft)
btnReverse: TButton (alLeft)
CheckBase64: TCheckBox (alLeft)
Panel2: TPanel (alClient)
Panel3: TPanel (alLeft)
Label1: TLabel (alTop)
Memo1: TMemo (alClient)
Splitter1: TSplitter
Panel4: TPanel (alClient)
Label2: TLabel (alTop)
Memo2: TMemo (alClient)

You need to replicate the parent/children structure with the same component name: component type I used. In parentheses I placed also the Align property to select. Maybe it’s a naive way of creating layouts, but in Lazarus we can go far with nesting panels, splitters and the Align property.

We can copy the jsonmplib.pas we created in the simple app to interface with the Rust library, we don’t need to reinvent the wheel here. and then connect the dots.

Here’s some code to get you going.

unit main;

{$mode objfpc}{$H+}

interface

uses
Classes, SysUtils, Forms, Controls, Graphics, Dialogs, ExtCtrls, StdCtrls,
jsonmplib;

type

TDataFormat = (dfJSON, dfMsgPack);

{ TForm1 }

TForm1 = class(TForm)
btnConvert: TButton;
btnReverse: TButton;
CheckBase64: TCheckBox;
Label1: TLabel;
Label2: TLabel;
Memo1: TMemo;
Memo2: TMemo;
Panel1: TPanel;
Panel2: TPanel;
Panel3: TPanel;
Panel4: TPanel;
Splitter1: TSplitter;
procedure btnConvertClick(Sender: TObject);
procedure btnReverseClick(Sender: TObject);
procedure FormCreate(Sender: TObject);
private
FLeftFormat: TDataFormat;
public

end;

var
Form1: TForm1;

implementation

{$R *.lfm}


{ TForm1 }

procedure TForm1.FormCreate(Sender: TObject);
begin
FLeftFormat := dfJSON;  // left starts as JSON
Label1.Caption := 'JSON';
Label2.Caption := 'MessagePack';
end;

procedure TForm1.btnReverseClick(Sender: TObject);
var
tmp: string;
begin
// swap captions
tmp := Label1.Caption;
Label1.Caption := Label2.Caption;
Label2.Caption := tmp;

// swap memos visually
tmp := Memo1.Text;
Memo1.Text := Memo2.Text;
Memo2.Text := tmp;

// toggle the state
if FLeftFormat = dfJSON then
FLeftFormat := dfMsgPack
else
FLeftFormat := dfJSON;
end;

procedure TForm1.btnConvertClick(Sender: TObject);
var
inp: string;
outp: PChar;
buf: TMpBuffer;
ms: TMemoryStream;
begin
inp := Memo1.Text;

if FLeftFormat = dfJSON then
begin
// JSON -> MessagePack
buf := json_to_msgpack(PChar(inp));
try
if (buf.Data <> nil) and (buf.len > 0) then
begin
ms := TMemoryStream.Create;
try
ms.WriteBuffer(buf.Data^, buf.len);
ms.Position := 0;
Memo2.Lines.Clear;
Memo2.Lines.Add('(binary msgpack, ' + IntToStr(buf.len) + ' bytes)');
finally
ms.Free;
end;
end;
finally
free_mpbuffer(buf);
end;
end
else
begin
// MessagePack -> JSON
// Memo1 contains *text* describing MessagePack; in practice, this
// will be raw binary loaded as hex or pasted, but for now assume raw bytes.
{
For now let's assume the user manually loads raw msgpack into Memo1.
We need to fix this later on.
}

// Convert Memo1.Text to a byte buffer
ms := TMemoryStream.Create;
try
ms.Write(PChar(inp)^, Length(inp));   // naive, for now
outp := msgpack_to_json(ms.Memory, ms.Size);
try
if outp <> nil then
Memo2.Text := outp;
finally
free_cstring(outp);
end;
finally
ms.Free;
end;
end;
end;


end.

The above code works, but it doesn’t show anything on the MessagePack part, as we cannot really dump binary into the text memos as-is, nor can we switch and convert from MessagePack to JSON: the program in this state is not yet functional.

Going the Extra Mile: Handling Base64

In the companion repository (and in the interface we used as layout) you can see an option to handle base64: we do not just signal that we can handle the binary data, we can actually use base64 to show this and even exchange MessagPack data with web APIs (which are incapable of dealing with binary as is, but love the base64 encoding of it). We added also a hex dump, so there’s something to show for all the effort done when we are not using the base64.

Refer to the JSONMPConv/JMPC/ folder in the companion repository.

As usual, I hope this tutorial was useful for you, so if you want, leave a comment below.