This week, some of the DebConf Video Team met in Herefordshire (UK) for a sprint. We didn’t have a sprint in 2024, and it was sorely needed, now.

At the sprint we made good progress towards using Voctomix 2 more reliably, and made plans for our future hardware needs.

Attendees

• Chris Boot (host)
• Stefano Rivera
• Kyle Robbertze
• Carl Karsten
• Nicolas Dandrimont

Voctomix 2

DebConf 25 was the first event that the team used Voctomix version 2. Testing it during DebCamp 25 (the week before DebConf), it seemed to work reliably. But during the event, we hit repeated audio dropout issues, that affected about 18 of our recordings (and live streams).

We had attempted to use Voctomix 2 at DebConf 24, and quickly rolled back to version 1, on day 1 of the conference, when we hit similar issues. We thought these issues would be resolved for DebConf 25, by using more powerful (newer) mixing machines.

Trying to get to the bottom of these issues was the main focus of the sprint. Nicolas brought 2 of Debian’s cameras and the Framework laptop that we’d used at the conference, so we could reproduce the problem. It didn’t take long to reproduce, in fact, we spent most of the week trying any configuration changes we could think of to avoid it. The issue we’ve been seeing feels like a gstreamer bug, rather than something voctomix is doing incorrectly. If anything, configuration changes are avoiding hitting it.

Finally, on the last night of the sprint, we managed to run voctomix all night without the problem appearing. But... that isn’t enough to feel confident that the issue is avoided. More testing will be required.

Detecting audio breakage

Kyle worked on a way to report the audio quality in our Prometheus exporter, so we can automatically detect this kind of audio breakage. This was implemented in helios our audio level monitor, and lead to some related code refactoring.

Framework Laptops

Historically, the video team has relied on borrowed and rented computer hardware at conferences for our (software) video mixing, streaming, storage and encoding. Many years ago, we’d even typically have a local Debian mirror and upload queue on site.

Our video mixing machines had to be desktop size computers with 2 Blackmagic Declink Mini Recorder PCI-e cards installed in them, to capture video from our cameras.

Now that we reliably have more Internet bandwidth than we really need, at our conference venues, we can rely on offsite cloud servers. We only need the video capture and mixing machines on site.

Blackmagic also has UltraStudio Recorder thunderbolt capture boxes that we can use with a laptop. The project bought a couple of these and a Framework 13 AMD laptop to test at DebConf 25.

We used it in production at DebConf, in the “Petit Amphi” room, where it seemed to work fairly well. It was very picky about thunderbolt cable and port combinations, refusing to even boot when they were connected.

Since then, Framework firmware has fixed these issues, and in our testing at the sprint, it worked almost perfectly. (One of the capture boxes got into a broken state, and had to be unplugged and re-connected to fix it.)

We think these are the best option for the future, and plan to ask the project to buy some more of them.

HDCP

Apple Silicon devices seem to like to HDCP-encrypt their HDMI output whenever possible. This causes our HDMI capture hardware to display an “Encrypted” error, rather than any useful image.

Chris experimented with a few different devices to strip HDCP from HDMI video, at least 2 of them worked.

Spring Cleaning

Kyle dug through the open issues in our Salsa repositories and cleaned up some issues.

DebConf 25 Video Encoding

The core video team at DebConf 25 was a little under-staffed, significantly overlapping with core conference organization, which took priority.

That, combined with the Voctomix 2 audio dropout issues we’d hit, meant that there was quite a bit of work left to be done to get the conference videos properly encoded and released.

We found that the encodings had been done at the wrong resolution, which forced a re-encode of all videos. In the process, we reviewed videos for audio issues and made a list of the ones that need more work. We ran out of time and this work isn’t done, yet.

DebConf 26 Preparation

Kyle reviewed floorplans and photographs of the proposed DebConf 26 talk venues, and build up a list of A/V kit that we’ll need to hire.

Carl’s Video Box

Carl uses much of the same stack as the video team, for many other events in the US. He has experimenting with using a Dell 7212 tablet in an all-in-one laser-cut box.

Carl demonstrated this box, which could perfect for small miniDebConfs, at the sprint. Using voctomix 2 on the box requires some work, because it doesn’t use Blackmagic cards for video capture.

gst-fallbacksrc

Carl’s box’s needs lead to looking at gstfallbacksrc. This should let Voctomix 2 survive cameras (or network sources) going away for a moment.

Matthias Geiger packaged it for us, and it’s now in Debian NEW. Thanks!

voctomix-outcasts

Carl cut a release of voctomix-outcasts and Stefano uploaded it to unstable.

Ansible Configuration

The videoteam’s stack is deployed with Ansible, and almost everything we do involves work on this stack. Carl upstreamed some of his features to us, and we updated our voctomix2 configuration to take advantage of our experiments at the sprint.

Miscellaneous Voctomix contributions

We fixed a couple of minor bugs in voctomix.

More Nageru experimentation

In 2023, we tried to configure Nageru (another live video mixer) for the video team’s needs. Like voctomix it needs some configuration and scaffolding to adapt it to your needs. Practically, this means writing a “theme” in Lua that controls the mixer.

The team still has a preference for Voctomix (as we’re all very familiar with it), but would like to have Nageru available as an option when we need it. We fixed some minor issues in our theme, enough to get it running again, on the Framework laptop. Much more work is needed to really make it a useable option.

Thank you

Thanks to the Debian project for funding the costs of the sprint, and Chris Boot’s extended family for providing us with a no-cost sprint venue.

Thanks to c3voc for developing and maintaining voctomix, and helping us to debug issues in it.

Thank you to everyone in the videoteam who attended or helped out remotely! And to employers who let us work on Debian on company time.

We’ll likely need to keep working on our stack remotely, in the leadup to DebConf 26, and/or have another sprint before then.