Installing an independent webapp inside an LXC

I asked about this topic at the IRC (June 7-8, 2020) and @fred1m proposed me to move it to the forum. So here it is! :eyes:

Context
I’m running freedombox in an A20 olimex board, and I’m happy with it. However, I would like to install an extra webapp out of control of plinth, to take more profit of the board and the hardened setup that FB provides.

My way to go is to create an unprivileged linux container inside the freedombox OS, and then, inside that lxc, install the webapp with its database and everything. I will manage the TLS certificate at an external reverse proxy, out of FB control, and will connect internally without encryption.

Steps I tried to follow

  1. Create new user for everything related to this app, probably only its lxc
  2. Create an unprivileged lxc owned by this user
  3. Create a network with libvirt tools, that includes a dhcp server, a virtual bridge, and a veth that connects the bridge to the container’s namespace.
  4. Configure the firewall to allow incoming connections to certain ports
  5. Upgrade guest system, install app, etc.

Getting into details of the firewall setup:

  1. Create a new zone some-lxc-app
  2. Add services to the zone (open some tcp and udp ports)
  3. Associate the virbr0 interface to it

Problem Description
When trying again today, I’ve had some issues with 3, setting up the network. So if you have some hints on libvirt’s nets or how to do it with lxc-net, I’m listening :slight_smile:

My biggest problem though, is that when I’ve had the network working, that is, I can ping external servers from within the lxc, I don’t happen to be able to make or receive TCP or UDP connections.

Added firewall setup

$ sudo firewall-cmd --info-zone some-lxc-app
some-lxc-app (active)
  target: default
  icmp-block-inversion: no
  interfaces: virbr0
  sources: 
  services: dns http ssh whois
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

Steps to Reproduce
With the setup described above, when trying to upgrade the guest system, I would be able to do so. Right now, the guest isn’t able to establish a new TCP connection even outwards.

Expected Results

I want to be able to establish outwards HTTP, HTTPS, SSH and Whois connections, and perform successful DNS queries. I want also to be able to receive inwards HTTP connections.

Information

  • FreedomBox version: 20.11 (up to date)
  • Hardware: A20-OLinuXino-LIME2
  • How did you install FreedomBox?: bought pre-installed

What I ask for

  • Advices about long-term lxc and virt networks
  • Guiding or ideas to troubleshoot the firewall problem.
4 Likes

Correcting as a reply, as it appears that I can’t edit anymore my post

SSH connections would be inwards to the LXC, not outwards.

Whether you will be able to access the outside networks from within LXC container will depend on the what interface the LXC interface is bridged with.

Here is what happens when two network interfaces are bridged: let us say there is an outside machine O1 connected to our machine with interface A, then there is another machine O2 which is connected to interface B in our machine. Let us also assume that O1 and O2 can’t talk to each other. Then, when we bridge interfaces A and B, we will create a new interface BR (and A and B are no longer be usable on their own). Now, O1 talks to BR, O2 talks to BR and O1 and O2 can talk to each other as if they are directly connected to each other at physical layer. O1 and O2 can be containers or real machines.

The solution in your case will be to check what interfaces are part of the virb0 bridge: brctl show virbr0. If you only have the virtual interface created for LXC, then the bridge is no better than a single interface. You also need to have another interface eth0 participating in that bridge. brctl addif virbr0 eth0 (something like that. Any you need a permanent way to do this). That would remove eth0 from normal usage and add it into virbr0. On the host, now only virbr0 remains. That would make your container reach outside machines (assign DHCP address normally and receive traffic from outside normally) and fix your problem.

However, I have a better suggestion for how to manage firewall. Make ‘FreedomBox WAN’ connection use the ‘virbr0’ interface instead of the ‘eth0’. Let FreedomBox manage the ‘virbr0’ interface. Let it get added to firewalld with the usual zone (external/internal). Let FreedomBox open and close ports on it as needed. This should not at all effect the container and the container gets all traffic and may setup it’s own firewall.

BTW, except for the firewall part which does not effect the container, everything here is regular setup on any server/container. So, you can also follow generic documentation about LXC.

1 Like

Hey @sunil, thank you for your answer!

This is all bridges I see:

$ sudo brctl show                                                                                                                                   
bridge name     bridge id               STP enabled     interfaces                                                                                                       
virbr0          8000.525400811089       yes             veth4O0P9C
                                                        virbr0-nic

It has not only the veth connecting the host with the guest, but also virbr0-nic . I read that this is a dummy interface to make sure that the bridge keeps the same MAC address, with no other function.

I see… but I don’t like to disturb that much the rest of the system. I will read further the libvirt docs and forums if this is the way they expect all admins to follow.

This sounds like a bridged connection, but I wanted libvirt to make me a NAT’d one.

Ok, so this is the way to go if I add eth0 to the virtbr0, I see the logic!

So, building on @sunil highlights, and checking the rough edges, I found and read

libvirt’s strategy is not to add the main (eth0) interface to the virt bridge (virbr0), but to add table rules that do the natting between host and guest and forwarding between the bridge and the system.

I think I have enough info for today, thank you! I will check with my co-admin and will come back with more answers and maybe more questions.

Thank you!
enoki

The link I couldn’t share

Not messing with the default setup in the host is quite prudent. Here is what I would try then:

  • Create the container with private networking (one interface on the host is connected to one interface inside the container). I am not sure what this type is called in LXC. In case of systemd-nspawn the interface will be named ve-<container> on host and host0 inside container.
  • On the host side, create a network-manager connection for this interface and set it in ‘shared’ mode. See the FreedomBox manual page for details. This will give the host side IP address like 10.42.0.1 and on client slide an IP in the range 10.42.0.10-255. Host will run a special DHCP and DNS server just for this connection. Client will have Internet access.
  • Then let FreedomBox manage the domain for container too. It will obtain the LE certificate and do TLS. Then drop a small Apache snippet that will forward all the connections to the domain or path to the container.
  • If non-web ports are needed, open them in firewall and then write an nftables rule to forward traffic from them to container IP.