With redpesk, we provide customers with the ability to cross-build an embedded, CentOS Stream-based Linux distribution in the cloud. This requires a significant infrastructure: Koji/RPM builders, Angular-based WebUI, Gitlab forge, network and RPM package dependency management, Qemu test lab management, all need to come together and be connected, in a mix of Qemu virtual machines and LXC containers. Fortunately, Ansible and Proxmox comes to the rescue to manage this complexity.

In this talk, we'll present our architecture of a self-contained CI/CD environment in the cloud, to cross-build RPM packages and Linux images. We will then dive into the specifics of using Ansible to drive Proxmox and deploy a mix of Packer-built Qemu virtual machines and LXC containers. These provide a full Koji build system (hub and builders), an Angular frontend, Go backend, a Gitlab forge as well as network isolation/firewalling and a Qemu virtual target lab. We'll continue with lessons learned from doing these deployments for multiple customers. We will finish by describing solutions we are currently working on, like Ansible AWX, to address the challenges of doing it at scale and increase automation.

This talk was presented at FOSDEM 2022 in the Infra Management Devroom

Slides: [click here]

Videos: [click here for MP4] [click here for WEBM]