Email server progress?

Dear All,

Google Summer of Code is around the corner. I will be joining Debian next week to implement an integrated email server for FreedomBox. Issue !56 has been assigned to me.

Overview of Deliverables

The project is divided into 3 working alpha releases.

After Alpha Release 1: Freedombox accepts emails from the same LAN or virtual private network. (I highly recommend a private name server for testing purposes.)

After Alpha Release 2: Authentication works. Emails are retrievable through IMAP and RoundCube. Anti-virus software and spam filters can be used.

After Alpha Release 3: The email stack can be configured from Plinth web interface. Command-line utilities will support TLS certificate installation, Maildir backup and data import from some third-party providers.

Bonus

EC 1: Option to use DNS over HTTPS. Compartmentalization. Dedicated network namespace.

EC 2: Onion service integration. Note that widely-used email authentication standards (SPF, DKIM, DMARC) rely on domain name infrastructure, which Tor lacks. We may resort to using unstandardized stuff at the cost of interoperability.

“Future Ideas”

These are good ideas, but I may not have the time to implement them during GSoC:

1 Like

Hi, I’m sure that’s very welcome by a lot of people.

One thing I noticed is that a self-hosted email server seems to require some particular hoops to actually allow it to work well.

Possibly you already know and planed for testing these things, but here are some things I remembered:

These options possibly undermine the self-hosting, depending on the external server’s owner, but they may be required to make an email server work on a dynamic IP:

  • Option to use and specify an external, outgoing smtp relayhost (sometimes misleadingly called “smarthost”), e.g. the ISP provider’s smtp-out server (needed if own, possibly dynamic IP is not usable, because of a blocked port 25, or because emails with this source IP address are rejected by the recipient’s email providers). Debian’s debconf has an option for that, but I’m not sure if safely encrypts and stores the password into a separate postix config file. (Example: Configure SMTP relay | Yunohost Documentation)

  • Option to use and specify an external smtp-submission server, e.g. as fallback for users to still be able to communicate (send emails) during offline- or down-times of the self-hosted email server.
    The self-hosted smtp server would need to accept emails from the smtp-submission server even if their FROM field contains the own email domain (to allow internal communication to function once the self-hosed smtp server comes up again). Ideally sort such emails into a “from external smtp-submission server” imap folder to make users aware to switch back to using the self-hosted smtp server.
    Fallback operation:
    Note 1: If the external smtp-submission server is not configured as default in the email clients, i.e. to prefer the self-hosted one during normal operation, users may have to manually select or switch to a fallback configuration (sending identity or account) in some email clients.
    Note 2: Continuous receiving of email during self-hosting down-time would require a second IMAP mailbox, possibly on the external backup MX server (backup freedombox?), or using a provider’s IMAP server, and configuring the email clients to always check the external backup inboxes as well. (While the external MX or inboxes are configured to forwarding to the internal MX or domain to enable the normal operation.)

  • Option to use and specify an an external MX (main, fallback or backup) (incoming smtp server, possibly but not necessarily same as smtp-out relay). The freedombox smtp would then have to allow (accept) all incoming emails from it: Whitelist the domain or IP to allow receiving email from it after offline- or downtimes of the selfhosted server. The self-hosted smtp server would explicitly need to also accept emails from this external MX server (after an outage) if the FROM field contains the own email domain (smtp-submission server may deliver to backup MX).

EDIT added later:

  • Option to use and specify an external forwarding domain (and an optional server name/IP). The emails could be received by the standard MX email server from the domain registrar (a not-preferable workaround undermining the self-hosting) or a freedombox install on a public server, and configuring it to forward the emails. For example, example.org may be the external forwarding domain, and configured with a wildcard forwarding rule, or having individual (backup) mailboxes that are configured e.g. with sieve rules to forward emails to the internal freedombox’es (sub)domain, e.g. mail.example.org.
    The local freedombox smtp sever would be configured mail.example.org and have to only accept emails coming from the specified IMAP/SMTP server name or IP (and from the external forwarding domain’s MX, determined by dns lookup?), but not from anywhere else. And use that external email domain as sender’s domain in all outgoing emails. (An outgoing relay would only be used if configured with the corresponding option above.)

  • Another and possibly most elegant configuration for a self-hosted smtp server on a dynamic IP may be a VPN or SSH connection to a server with a (static) public IP that has proper (forward and reverse) DNS entries and an open port 25. (Allowing to reach and receive the world without having to use or maintain any external email relaying setup while self-hosting on a dynamic IP.)

The required MX and SPF dns records, and even more external configs like email forwarding or catch-alls, will depend on the chosen options. And consequently the domain (zone) configuration information (i.e. configuration request shown to the freedombox admin) would heavily depend on the chosen options.

Edit 2: A very helpful feature could be if the freedombox smtp setup could detect any preexisting configuration from the published dns zone, and use the current dns entries to populate the presented default values of the freedombox email configuration options (i.e. outgoing smtp/submission/MX), but still allow for adjustments (e.g. to switch over from the current email provider to self-hosting on a static public IP or an internal mail.(sub)domain, and present the required updated dns records to the admin).

BTW: For quick testing, you may also enjoy using this mobile email client:

Not sure if it’s worth the effort to integrate this. Maybe just mention some imap migration (copy) tools the user or admin could use. (Example: Migrating email from an email provider to a YunoHost instance | Yunohost Documentation)

I guess testing and solving the typical smtp self-hosting sticking points (external relay, backup MX and -submission, DNS and SPF (~all) config, … mentioned above) with the plinth app would be more important.

Is there a list of planned configuration fields and widgets for the plinth email server configuration page somewhere?

I mean when (re)using the already configured freedombox domains and users, I guess email could pretty much work without config options, if there weren’t a need to have an external relay, backup MX, and smtp-submission.

EDIT: Maybe freedombox could configure the external email server peculiarities based on querying the DNS servers? That way, entering these servers in the plinth interface would only have to render custom DNS configuration records for the admin, as guide for configuring the dns zones. But the functionality itself could work, and be tested, independently from the plinth administration helper interface.

(an additional point that was originally posted here has been moved into the post above)

Dear Email Fans,

We are looking for testers! See this thread for what works and what does not work in “Alpha Release 2”. Testing instructions are included.

Do not run production data on it.

Dev environment setup

git clone https://salsa.debian.org/freedombox-team/freedombox.git
cd freedombox
git fetch origin merge-requests/2064/head:gsoc-email
git checkout gsoc-email
./container up
./container ip
./container ssh
freedombox-develop  # run in container

Visit the Plinth web interface at https://10.42.0.nnn

What to look for

See this thread for what works and what does not work.

Screenshots

4 Likes

Failed at ```
./container ip

Which failure message? Which FreedombBox commit did you test?

Most likely a configuration problem, but I haven’t yet been able to send or receive any emails, internal or external (they bounce, for me). I need some help and instructions need some love.

Has anybody been able to successfully send or receive any email? Witness of email working is welcome.

Sorry, I can’t test or help much. But it seems to me that bouncing and “Relay access denied” errors are pointing to the typical problems of properly specifying own domain, own network, etc. that the server accepts and feels responsible for etc. In short, the set of configuration parameters that would ideally depend on simpler, possibly already existing, local (meta) configuration options in the admin interface, as suggested at: Email server progress? - #22 by NickA

Sorry, I can’t test or help much.

You did help me, at least… with your comment on the salsa issue. Thank you.

Downloading .container/freedombox-testing-free_dev_all-amd64.img.xz
.container/freedomb 100%[===================>] 909.75M 667KB/s in 23m 34s
Downloading .container/freedombox-testing-free_dev_all-amd64.img.xz.sig
.container/freedomb 100%[===================>] 566 --.-KB/s in 0.001s
Receiving GPG keys
gpg: keyserver receive failed: No name
Traceback (most recent call last):
File “./container”, line 1026, in
main()
File “./container”, line 1022, in main
subcommand_method(arguments)
File “./container”, line 952, in subcommand_up
image_file = _download_disk_image(arguments.distribution)
File “./container”, line 504, in _download_disk_image
_verify_signature(target_file, signature_file)
File “./container”, line 429, in _verify_signature
] + TRUSTED_KEYS, check=True)
File “/usr/lib/python3.7/subprocess.py”, line 487, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command ‘[‘gpg’, ‘–quiet’, ‘–homedir’, ‘.container/gpg’, ‘–keyserver’, ‘keys.gnupg.net’, ‘–recv-keys’, ‘013D86D8BA32EAB4A6691BF85D4153D6FE188FC8’]’ returned non-zero exit status 2.

I recently ran into this issue due issue with keys.gnupg.net. I had to edit the container script and change the KEY_SERVER value to another keyserver (I think keyserver.ubuntu.com worked).

Issue for key server update in container script: container: Script fails due to gpg keyserver error (#2091) · Issues · FreedomBox / FreedomBox · GitLab