Installing an independent webapp inside an LXC

Support
1

I asked about this topic at the IRC (June 7-8, 2020) and @fred1m proposed me to move it to the forum. So here it is! :eyes:

Context
I’m running freedombox in an A20 olimex board, and I’m happy with it. However, I would like to install an extra webapp out of control of plinth, to take more profit of the board and the hardened setup that FB provides.

My way to go is to create an unprivileged linux container inside the freedombox OS, and then, inside that lxc, install the webapp with its database and everything. I will manage the TLS certificate at an external reverse proxy, out of FB control, and will connect internally without encryption.

Steps I tried to follow

  1. Create new user for everything related to this app, probably only its lxc
  2. Create an unprivileged lxc owned by this user
  3. Create a network with libvirt tools, that includes a dhcp server, a virtual bridge, and a veth that connects the bridge to the container’s namespace.
  4. Configure the firewall to allow incoming connections to certain ports
  5. Upgrade guest system, install app, etc.

Getting into details of the firewall setup:

  1. Create a new zone some-lxc-app
  2. Add services to the zone (open some tcp and udp ports)
  3. Associate the virbr0 interface to it

Problem Description
When trying again today, I’ve had some issues with 3, setting up the network. So if you have some hints on libvirt’s nets or how to do it with lxc-net, I’m listening :slight_smile:

My biggest problem though, is that when I’ve had the network working, that is, I can ping external servers from within the lxc, I don’t happen to be able to make or receive TCP or UDP connections.

Added firewall setup

$ sudo firewall-cmd --info-zone some-lxc-app
some-lxc-app (active)
  target: default
  icmp-block-inversion: no
  interfaces: virbr0
  sources: 
  services: dns http ssh whois
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:

Steps to Reproduce
With the setup described above, when trying to upgrade the guest system, I would be able to do so. Right now, the guest isn’t able to establish a new TCP connection even outwards.

Expected Results

I want to be able to establish outwards HTTP, HTTPS, SSH and Whois connections, and perform successful DNS queries. I want also to be able to receive inwards HTTP connections.

Information

  • FreedomBox version: 20.11 (up to date)
  • Hardware: A20-OLinuXino-LIME2
  • How did you install FreedomBox?: bought pre-installed

What I ask for

  • Advices about long-term lxc and virt networks
  • Guiding or ideas to troubleshoot the firewall problem.
4 Likes
2

Correcting as a reply, as it appears that I can’t edit anymore my post

SSH connections would be inwards to the LXC, not outwards.

3

Whether you will be able to access the outside networks from within LXC container will depend on the what interface the LXC interface is bridged with.

Here is what happens when two network interfaces are bridged: let us say there is an outside machine O1 connected to our machine with interface A, then there is another machine O2 which is connected to interface B in our machine. Let us also assume that O1 and O2 can’t talk to each other. Then, when we bridge interfaces A and B, we will create a new interface BR (and A and B are no longer be usable on their own). Now, O1 talks to BR, O2 talks to BR and O1 and O2 can talk to each other as if they are directly connected to each other at physical layer. O1 and O2 can be containers or real machines.

The solution in your case will be to check what interfaces are part of the virb0 bridge: brctl show virbr0. If you only have the virtual interface created for LXC, then the bridge is no better than a single interface. You also need to have another interface eth0 participating in that bridge. brctl addif virbr0 eth0 (something like that. Any you need a permanent way to do this). That would remove eth0 from normal usage and add it into virbr0. On the host, now only virbr0 remains. That would make your container reach outside machines (assign DHCP address normally and receive traffic from outside normally) and fix your problem.

However, I have a better suggestion for how to manage firewall. Make ‘FreedomBox WAN’ connection use the ‘virbr0’ interface instead of the ‘eth0’. Let FreedomBox manage the ‘virbr0’ interface. Let it get added to firewalld with the usual zone (external/internal). Let FreedomBox open and close ports on it as needed. This should not at all effect the container and the container gets all traffic and may setup it’s own firewall.

BTW, except for the firewall part which does not effect the container, everything here is regular setup on any server/container. So, you can also follow generic documentation about LXC.

2 Likes
4

Hey @sunil, thank you for your answer!

This is all bridges I see:

$ sudo brctl show                                                                                                                                   
bridge name     bridge id               STP enabled     interfaces                                                                                                       
virbr0          8000.525400811089       yes             veth4O0P9C
                                                        virbr0-nic

It has not only the veth connecting the host with the guest, but also virbr0-nic . I read that this is a dummy interface to make sure that the bridge keeps the same MAC address, with no other function.

I see… but I don’t like to disturb that much the rest of the system. I will read further the libvirt docs and forums if this is the way they expect all admins to follow.

This sounds like a bridged connection, but I wanted libvirt to make me a NAT’d one.

Ok, so this is the way to go if I add eth0 to the virtbr0, I see the logic!

5

So, building on @sunil highlights, and checking the rough edges, I found and read

libvirt’s strategy is not to add the main (eth0) interface to the virt bridge (virbr0), but to add table rules that do the natting between host and guest and forwarding between the bridge and the system.

I think I have enough info for today, thank you! I will check with my co-admin and will come back with more answers and maybe more questions.

Thank you!
enoki

6

The link I couldn’t share

7

Not messing with the default setup in the host is quite prudent. Here is what I would try then:

  • Create the container with private networking (one interface on the host is connected to one interface inside the container). I am not sure what this type is called in LXC. In case of systemd-nspawn the interface will be named ve-<container> on host and host0 inside container.
  • On the host side, create a network-manager connection for this interface and set it in ‘shared’ mode. See the FreedomBox manual page for details. This will give the host side IP address like 10.42.0.1 and on client slide an IP in the range 10.42.0.10-255. Host will run a special DHCP and DNS server just for this connection. Client will have Internet access.
  • Then let FreedomBox manage the domain for container too. It will obtain the LE certificate and do TLS. Then drop a small Apache snippet that will forward all the connections to the domain or path to the container.
  • If non-web ports are needed, open them in firewall and then write an nftables rule to forward traffic from them to container IP.
2 Likes
8

Hi, I managed to make it work some months ago! For internet’s history sake and better for this community, if anyone is interested I can detail it more, but the birdseye setting uses:

  • standard freedombox with nftables and firewalld
  • packaged debian buster lxc. User containers settings.
  • a user to run this user container with its home dir to hold all lxc data.
  • libvirt networking, default nat setting.
  • new firewalld zone named after the app, allowing the needed ports and also not-so obviously, the dns port, and also, binding it to the lxc virtual interface.
  • freedombox’s apache changed to listen to secondary ports. tweak apache configs to allow for good redirection of / → plinth; /something_random → plinth; http → https, and so on.
  • sniproxy listening to port 80 and 443
  • sniproxy to forward freedombox domains to apache/freedombox, and to redirect app’s domain to its lxc container ip address with the firewall-allowed ports (can be the standard ones)

The more complicated parts for me where to figure out which ports allow where in firewalld, and to tweak apache configs; it was so slow to check that each change did not break previous fixes (and actually did many times)

Justification for some pieces:

  • Freedombox does not know how to manage more than 1 dns domain name of each category: onion, “static” dns, and dynamic dns.
  • However, I wanted another name for my other app. So I wanted it to manage certs by itself (which is not hard anyway with certbot + nginx)
  • Here comes sniproxy, as lets encrypt tries to contact your host via port 80, doesn’t matter if your app listens to other ports.
  • changing freedombox’s apache port, made the redirects stop working. For instance, some redirects pointed to the internal, secondary http port apache was listening to, instead of the public ones (443 and 80)

Also:

  • sniproxy needed compiling for olimex a20 (armhf), but it was pretty straighforward (I’m no C master) and it compiles quite quickly 2~10 minutes iirc.

There are some details that I don’t remember exactly, but take me too much effort rigth now to doublecheck. I prefer to just let it written by now as is.

Hope that this can be helpful anyway to somebody!
enoki

1 Like