1. Overview
The Recipe Bottle Workbench enables developers to safely run untrusted containers - a significantly distinct use case than typical container deployments of carefully crafted code.
While containers excel at packaging known applications, running third-party or experimental code poses security risks. The Recipe Bottle Workbench addresses this by interposing a security layer (Sentry Container) between untrusted containers (Bottle Containers) and system resources, without requiring modifications to existing container images.
This approach allows developers to leverage the vast ecosystem of containerized tools while maintaining strict security boundaries - essential for small teams experimenting with diverse, potentially risky codebases.
2. Remaining Work Before First Release
The discovery that GHCR does not implement OCI-compliant reference counting creates a critical operational risk for small development teams. When deleting older container images, GHCR may also remove shared layers that newer images depend upon, potentially breaking the entire image repository. This violates the OCI specification’s requirement for proper layer management and makes GHCR unsuitable for projects requiring predictable image lifecycle management.
For teams maintaining multiple versions of containers or performing regular cleanup of old builds, this deficiency transforms routine maintenance into a dangerous operation. The lack of reference counting means there’s no safe way to know which images share layers, forcing teams to either keep all historical images (consuming storage and increasing costs) or risk breaking their container infrastructure.
The Recipe Bottle Workbench will disconnect from exclusive GHCR integration and instead allow Users to select any fully OCI-compliant container registry such as AWS ECR, Azure ACR, or Quay. These registries properly implement layer reference counting, ensuring that shared layers remain available as long as any image requires them.
Additionally, the project’s build infrastructure currently relies on Makefiles, which limits portability. Converting to pure bash 3.2 implementation will ensure broader compatibility across development environments.
3. The Recipe Bottle Workbench Vision
Maintaining build and service environments is a headache for the small development organization. Containers are an amazing tool for controlling package version constellations. Necessary, but insufficient; I find that I need more redundancy, security, and control for my build setups.
The Recipe Bottle Workbench is my answer for filling in the gaps. The vision is simple: how can a User run only a few apps natively on a Workstation to coordinate and arrange containers in sophisticated yet safe ways for me and my customer’s work?
Recipe Bottle Workbench uses only bash, git, curl, jq and a
Container Runtime
(currently only podman) natively to produce a safe and controlled space for development involving
containers
heavily.
Recipe Bottle Workbench
itself is largely a set of bash scripts designed to be easily incorporated into arbitrary projects via
git subtree, git subrepo, or git submodule
graft.
This is purely an open-source undertaking.
3.1. Part One: Image Management
The Recipe Bottle Workbench streamlines container Image creation and maintenance through GitHub Actions and OCI-compliant registries. With a configured access token, the Build Service:
3.2. Part Two: Bottle Service Orchestration
For development services requiring internet and/or IP connectivity at the Workstation, Recipe Bottle Workbench then orchestrates startup and configuration of Bottle Services, which are comprised of a Sentry Container, a Censer Container, and a Bottle Container operating together.
The Recipe Bottle Workbench allows any Image providing or using network services to function as a Bottle Container. The Censer Container establishes a privileged network namespace, configures it to route all traffic through the Sentry Container, then shares this pre-configured namespace with the Bottle Container. This ensures security policies are enforced from the first packet, and the Bottle Container experiences only a functional path to its Sentry Container gateway.
The
Sentry Container
thus sets up a potentially sophisticated set of network security safeguards that prevent malicious or compromised
Bottle Containers
from exfiltrating the
User’s
assets.
Through configuration of deeply mature tools iptables and dnsmasq, the
Sentry Container
prevents such illegal accesses.
By providing these controlled yet accessible tools, the Recipe Bottle Workbench enables small development teams to maintain proper container hygiene throughout their workflow - from initial building through deployment and eventual cleanup. This empowers organizations to leverage the expanding ecosystem of containerized development tools without requiring specialized DevOps expertise.
4. Definitions
- Bottle Service
-
A complete service instance consisting of one Bottle Container, one Censer Container, and one Sentry Container working together to provide secure functionality.
- User
-
An individual operating the Recipe Bottle Workbench to run containerized services securely on their Workstation.
- Workstation
-
The local computing environment where the User runs the Recipe Bottle Workbench. This environment hosts all Bottle Services and their associated networks while providing isolation from other system resources.
- Enclave Network
-
An isolated network connecting a Bottle Container to its Sentry Container, using the {rbn_enclave_base_ip}/{rbn_enclave_netmask} for addressing. This network enforces strict security policies and provides the only communication path for the Bottle Container.
- Transit Network
-
The network interface that connects a Sentry Container to the Workstation’s network resources. This network enables controlled external communication while maintaining security boundaries.
- Sentry Container
-
A privileged container that enforces network security policies for a Bottle Service. It connects to both Transit Network and Enclave Network, controlling all network traffic and providing DNS services.
- Censer Container
-
A privileged container using a minimal image that establishes and configures the network namespace before the Bottle Container starts. It sets up routing to ensure all traffic flows through the Sentry Container, then shares this pre-configured namespace with the Bottle Container.
- Bottle Container
-
A container that runs the actual service functionality, inheriting its network namespace from the Censer Container via
--net=container:censer. Connected only to the Enclave Network and configured with specific Volume Mounts, all external communication flows through its associated Sentry Container.
- Recipe Bottle Workbench
-
The complete Recipe Bottle system, enabling secure deployment and management of containerized services through network isolation and controlled resource access.
- Volume Mount
-
A configuration that allows containers to access specified portions of the host filesystem.
- Build Service
-
A system managing container creation through Build Workflows and OCI-compliant registries, enabling Users to build containers with complete traceability.
- Build History
-
A permanent record maintained by the Build Service for each container build, including build transcripts and commit references that verify container origins.
- Remaining Work Before First Release
-
The set of technical challenges that must be resolved before the Recipe Bottle Workbench can be released, including registry selection and build system implementation.
- Build Workflow
-
The automated process within the Build Service that standardizes how Users build and publish containers while maintaining security controls.
- Container Runtime
-
Application capable of running containers on the Workstation. Early experiments with
dockerled to disappointing results concerning interlacing host networking and internal networking. At the time of this writing,podmanprovides the necessary features for full Bottle Service provisions.
5. Significant Events
| Date | Event |
|---|---|
Dec 10, 2023 |
First try at wrangling Docker in local cygwin environment. |
Mar 18, 2024 |
Embrace multiple ephemeral docker container images for single build. |
Aug 7, 2024 |
Started Jupyter Notebook in docker container. |
Aug 11, 2024 |
Learned docker can’t connect to host and internal network simultaneously. |
Aug 17, 2024 |
Switched to podman for better networking. |
Sep 2, 2024 |
First attempt to use asciidoc for concept models. |
Oct 7, 2024 |
GitHub Action performed docker Image build. |
Nov 2024 |
Bespoke Bottle Container images with custom networking use Sentry Container networking via Claude curated requirements and script generation. |
Dec 2024 |
Experiments to start unmodified Bottle Container images using podman networking configurations fail. |
Dec 31, 2024 |
Submitted Podman feature request issue #24920 for gateway feature. |
Jan 6, 2025 |
Feature request issue #24920 rejected. |
Jan 2025 |
Podman 5.2 VMs running unmodified Bottle Container images using VM network namespace machinations. |
Feb 2025 |
Podman 5.3 VM update breaks previous, and reversion to 5.2 fails due to reissued VM. Podman VM tags determined not immutable. |
Mar 2025 |
Developed full reproducibility scenarios and determined that VM network namespace approach no longer valid. |
June 25, 2025 |
Attempting to use eBPF frame rewriting to bypass Netavark gateway behaviors lead to third container concept. |
July 3, 2025 |
Censer Container as privileged network namespace configurator hosting unprivileged Bottle Container functions; all Sentry Container/Bottle Container test cases pass again. |
July 27, 2025 |
GHCR Image deletions determined unsafe due to lack of OCI-specified reference counting on layers; deleting old images can destroy new ones. |