Table of Content

• Some contextal informations
• /etc/pf.conf and sysctls
• Testing configuration…
• Emerged problem from the PF+sysctl configuration

Some contextual informations

As everytime I’m trying myself on posting on this blog, I’ll take some time to try to put some context around my writings…

Some of you may known, there has been some works for now over a few years on FreeBSD to try to give “Containers” (Docker, Podman, …) to its users. I’ve been watching its evolution from my point of view, sometimes interesting and sometimes I was wondering if this subject was going anywhere.

Containers on themselves already existed for a while on FreeBSD with Jails anyway.

Until a year ago or so, the OCI proposal gave us some tools to try and use on FreeBSD with the Podman Suite adapted to my favorite OS of use.

Thanks to all contributors who gave birth to this.

And recently, a few weeks ago and also this week, some more informations came online in the form of articles and videos from the FreeBSD Foundation, but not only :

• An introduction to OCI Containers on FreeBSD
• Podman OCI Containers on FreeBSD

On themselves, these 2 blog posts are already good data on OCI Containers on FreeBSD.

As for myself, I’ve been using these data to test OCI containers in my daily environment:

• a PC with FreeBSD 14.3-RELEASE installed
• an existing home network
• the need to be able to start some Bhyve VMs to test things out
• the need to be able to build jails to also test things out. And now, to integrate OCI Containers with Podman Suite on this machine and not to break the existing configuration so that most of the things I use everyday still works even if I’m using OCI Containers and the required configuration for them to work.

It most likely start with installing the tools, of course, but since I’m having an already configured PF firewall… I needed:

• merge a sample pf.conf configuration to my /etc/pf.conf
• configure sysctls to be able to use podman run -p {localport}:{containerport} ...
• to be able to contact the opened tcp port from the computer itself, and from others on my network
• to test regularly that everything is still working the way it should
• and another thing that I can’t fix right now, SCP’ing big file to this FreeBSD computer.

/etc/pf.conf and sysctls

The sample configuration from the containers packages installed with Podman Suite is the following:

# Change these to the interface(s) with the default route
v4egress_if = "ix0"
v6egress_if = "ix0"

nat on $v4egress_if inet from <cni-nat> to any -> ($v4egress_if)
nat on $v6egress_if inet6 from <cni-nat> to !ff00::/8 -> ($v6egress_if)

rdr-anchor "cni-rdr/*"
nat-anchor "cni-rdr/*"
table <cni-nat>

It’s a simple configuration that works on its own if you do not have one already. It’s working, and you can start container, and access exposed ports from outsite. But locally, on the host, you also need to modify a sysctl value to be able to access exposed ports:

sysctl net.pf.filter_local=1
# and restart PF
service pf restart

I think this sysctl has been added because PF configurations were not able to give that existing aspect from containers on Linux, used with Docker or Podman. It may not be essential in some way, and there are probably some network architecturing/use not to have to use this sysctl, but I do not known right now which they could be.

And so, my own local pf.conf merged with these value turned to:

ext_if = "em0"
int_if = "vm_public"

icmp_types = "{ echoreq, unreach }"
icmp6_types = "{ 128, 133, 134, 135, 136, 137 }"

#set skip on lo
scrub in on $ext_if all fragment reassemble

nat on $ext_if from !($ext_if) -> ($ext_if:0)
nat on $ext_if inet from <jails> to any -> ($ext_if)
nat on $ext_if inet from <cni-nat> to any -> ($ext_if)

nat-anchor "ftp-proxy/*"
rdr-anchor "ftp-proxy/*"

rdr-anchor "cni-rdr/*"
nat-anchor "cni-rdr/*"
table <cni-nat>

block log on $ext_if all

pass out

anchor "ftp-proxy/*"
antispoof quick for { lo }

pass in on $ext_if from $ext_if:network to any
pass in on $ext_if from $int_if:network to any
pass inet proto icmp icmp-type $icmp_types

pass in on $ext_if inet6 proto ipv6-icmp all icmp6-type { 2, 128 }
pass in on $ext_if inet6 proto ipv6-icmp from any to any icmp6-type $icmp6_types

Some may say that I do not need to protect this computer since it’s on a local network, but my goal is to be able to test things that I might be using on production servers configured with PF.

If you look closely, you can see that I commented out the set skip on lo rule, mostly because if I let it configured, I still can’t connect to container exposed ports. I don’t really get why since the rule itself tells to skip rule evaluation on on lo interfaces. And probably that’s because I may not understand what this rule does really mean.

At least, this configuration enables almost everything: VM, jails, and now containers.

Testing configuration…

It’s very simple! Or at least I think it is … I’m just creating a FreeBSD container, I’m installing the necessary simple and single tool: nc (netcat, I guess the closest one from OpenBSD netcat). And then, I start nc in listen mode, on a port exposed to the host. To summarize the commands I used to start, as root:

podman run -p 7676:7676 -it --rm ghcr.io/freebsd/freebsd-runtime:14.3 sh
# Now inside the container
pkg install -fy -r FreeBSD-base FreeBSD-utilities
# Installing a bunch of things I mostly do not use in that case...
# and starting `nc` in listen mode
nc -l 0.0.0.0 7676

And in another terminal, I also start:

nc 192.168.12.34 7676

then type a few characters and check on return that what I typed appears in the listening nc inside the container.

Simple! Isn’t it? Nah!?

Ok, I admit it: it’s quite cumbersome. Typing all this take some time, and forgetting some parts might be an action of time on my aging brain… So what?

So I decided I just need the nc command inside that container. I’m not a “Container” pro, nor do I think I could be that, and I started to think I need to build my own FreeBSD container image with nc inside it, so that I could start that container this way:

podman run -p 7676:7676 -it --rm myimage nc -l 0.0.0.0 7676

Easy! Heh? Not when your brain thinks over every possible aspect… And to summarize it … it went bad, of course, because I thought to build an freebsd image with the net/netcat package installed, which I thought was the same command as the FreeBSD base command nc, but it isn’t. Then, managed to podman build from the ghcr.io/freebsd/freebsd-runtime:14.3 image and to include that image and only the nc command and the /lib/libstats.so.0 it’s depending on… not the only one, but the only one missing without everything from the FreeBSD-utilities package from the FreeBSD-base packages.

• The ghcr.io/freebsd/freebsd-runtime:14.3 container image size is 33 MB.
• My container image based on the latest, containing nc and a .so: 50 MB or so

But wait… that seems big for just a container image in which I only use nc, doesn’t it?

That’s when I remember that FreeBSD comes with “rescue” tools, and luckily nc is one of them:

• /rescue/nc exists! Statically compiled!
• FreeBSD-base packages contains a FreeBSD-rescue package containing nc!
• scratch containers exists! So in the end, I can just create a scratch container only with nc, can’t I?

Here is my Dockerfile for that, simple and readable:

FROM ghcr.io/freebsd/freebsd-runtime:14.3 as runtime
RUN pkg inst -fy -r FreeBSD-base FreeBSD-rescue

FROM scratch
COPY --from=runtime /rescue/nc /nc
ENV PATH=/

The result is a “FreeBSD” container image, because the tool is from FreeBSD, which contains nc statically compiled, and only that, and the image itself is 17.1 MB. It’s working nicely as expected and I can run it with that single simple command:

podman run -p 7676:7676 -it --rm nc-static:freebsd-14.3 nc -l 0.0.0.0 7676

Emerged problem from the PF+sysctl configuration

For some reason, when I’m trying to SCP big files from another computer to this podman containers hosts, the SCP commands stalles. For now I could not find how I can configure my PF not to make the SCP to stall. Small files SCP works fine. I’ve tried some MTU shenanigans, but not in great success.