Freedombox becomes (fully or partially) non-functional after the nightly backup

Again, I noticed that the last battery status message from my phone (via ejabberd) was at 2:33 UTC (it comes every few minutes normally), so this is approximately the time of the problem. Looking at the journal, this is at the nightly backup.

After hours of playing with journalctl (with sudo), I found several things:

  • on Oct 11 (and previous days as well), after the nightly backup, all services were started again and Freedombox was functional

  • on Oct 12, in the morning, I had lost all contact with the freedombox, so I removed the power and put it back on. The journal shows that after backup, bind and coturn were restarted, then the command to restart ejabberd is show, and then nothing until boot.

  • on Oct 13, in the morning, the web interface works ok but no service works. After some time, I asked to reboot, then all services work. The journal shows a failure in the backup, no trace of the backup post hooks in the journal, no service is started. In the web interface, services are not running, there is an entry for the backup as if it were successful, no notification or warning anywhere besides inside each service page and the diagnostics.

  • on Oct 14, same situation, the web interface works ok but no services works.

Here are extracts of journal that show all this:

Oct 11 (no problem that day)
Extract 1: the nightly backup and after

Oct 11 04:04:39 fbox /usr/bin/plinth[463]: Running backup_pre hooks
Oct 11 04:04:39 fbox /usr/bin/plinth[463]: # backups create-archive --path '/media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::scheduled: daily: 2021-10-11:04:03' --comment '{{"type": "scheduled", "periods": ["daily"]}}' --paths /e>
Oct 11 04:04:39 fbox sudo[2691]:   plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/backups create-archive --path /media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::scheduled: daily: 2021-10-11:04:03 --comment {{"type>
Oct 11 04:04:39 fbox sudo[2691]:   plinth : (command continued) /etc/ejabberd/ejabberd.pem /etc/matrix-synapse/homeserver.yaml /etc/matrix-synapse/log.yaml /var/lib/matrix-synapse/homeserver.db /etc/matrix-synapse/homeserver.>
Oct 11 04:04:39 fbox sudo[2691]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
Oct 11 04:05:38 fbox sudo[2691]: pam_unix(sudo:session): session closed for user root
Oct 11 04:05:38 fbox /usr/bin/plinth[463]: Running backup_post hooks
Oct 11 04:05:38 fbox /usr/bin/plinth[463]: # service start bind9
Oct 11 04:05:38 fbox slapd[946]: slap_global_control: unrecognized control: 1.3.6.1.4.1.4203.666.5.16
Oct 11 04:05:38 fbox sudo[2698]:   plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/service start bind9
Oct 11 04:05:38 fbox sudo[2698]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
Oct 11 04:05:46 fbox systemd[1]: Started BIND Domain Name Server.
Oct 11 04:05:46 fbox named[2702]: starting BIND 9.16.15-Debian (Stable Release) 
Oct 11 04:05:46 fbox named[2702]: running on Linux armv7l 5.10.0-9-armmp-lpae #1 SMP Debian 5.10.70-1 (2021-09-30)
Oct 11 04:05:46 fbox named[2702]: built with '--build=arm-linux-gnueabihf' '--prefix=/usr' '--includedir=/usr/include' '--mandir=/usr/share/man' '--infodir=/usr/share/info' '--sysconfdir=/etc' '--localstatedir=/var' '--disabl>
Oct 11 04:05:46 fbox named[2702]: running as: named -f -u bind
Oct 11 04:05:46 fbox named[2702]: compiled by GCC 10.2.1 20210110
Oct 11 04:05:46 fbox named[2702]: compiled with OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021
Oct 11 04:05:46 fbox named[2702]: linked to OpenSSL version: OpenSSL 1.1.1k  25 Mar 2021
Oct 11 04:05:46 fbox named[2702]: compiled with libxml2 version: 2.9.10
Oct 11 04:05:46 fbox named[2702]: linked to libxml2 version: 20910
Oct 11 04:05:46 fbox named[2702]: compiled with json-c version: 0.15
Oct 11 04:05:46 fbox named[2702]: linked to json-c version: 0.15
Oct 11 04:05:46 fbox named[2702]: compiled with zlib version: 1.2.11
Oct 11 04:05:46 fbox named[2702]: linked to zlib version: 1.2.11
Oct 11 04:05:46 fbox named[2702]: ----------------------------------------------------
Oct 11 04:05:46 fbox named[2702]: BIND 9 is maintained by Internet Systems Consortium,
Oct 11 04:05:46 fbox named[2702]: Inc. (ISC), a non-profit 501(c)(3) public-benefit
Oct 11 04:05:46 fbox named[2702]: corporation.  Support and training for BIND 9 are
Oct 11 04:05:46 fbox named[2702]: available at https://www.isc.org/support
Oct 11 04:05:46 fbox named[2702]: ----------------------------------------------------
Oct 11 04:05:46 fbox named[2702]: adjusted limit on open files from 524288 to 1048576
Oct 11 04:05:46 fbox named[2702]: found 2 CPUs, using 2 worker threads
Oct 11 04:05:46 fbox named[2702]: using 2 UDP listeners per interface
Oct 11 04:05:46 fbox named[2702]: using up to 21000 sockets
Oct 11 04:05:46 fbox named[2702]: loading configuration from '/etc/bind/named.conf'
Oct 11 04:05:46 fbox named[2702]: /etc/bind/named.conf.options:16: option 'dnssec-enable' is obsolete and should be removed
Oct 11 04:05:46 fbox named[2702]: reading built-in trust anchors from file '/etc/bind/bind.keys'
Oct 11 04:05:46 fbox named[2702]: looking for GeoIP2 databases in '/usr/share/GeoIP'
Oct 11 04:05:46 fbox named[2702]: using default UDP/IPv4 port range: [32768, 60999]
Oct 11 04:05:46 fbox named[2702]: using default UDP/IPv6 port range: [32768, 60999]
Oct 11 04:05:46 fbox named[2702]: listening on IPv4 interface lo, 127.0.0.1#53
Oct 11 04:05:46 fbox named[2702]: listening on IPv4 interface eth0, 192.168.1.66#53
Oct 11 04:05:46 fbox named[2702]: listening on IPv4 interface tun0, 10.91.0.1#53
Oct 11 04:05:46 fbox named[2702]: IPv6 socket API is incomplete; explicitly binding to each IPv6 address separately
Oct 11 04:05:46 fbox named[2702]: listening on IPv6 interface lo, ::1#53
Oct 11 04:05:46 fbox named[2702]: listening on IPv6 interface eth0, fe80::2272:e4ce:2d48:7bc2%2#53
Oct 11 04:05:46 fbox named[2702]: listening on IPv6 interface tun0, fe80::2283:af87:9fe2:53c9%3#53
Oct 11 04:05:46 fbox named[2702]: generating session key for dynamic DNS
Oct 11 04:05:46 fbox named[2702]: sizing zone task pool based on 5 zones

Extract 2: after the nightly backup, and bind9 and coturn were started:

Oct 11 04:05:55 fbox /usr/bin/plinth[463]: # service start ejabberd
Oct 11 04:05:55 fbox sudo[2721]:   plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/service start ejabberd
Oct 11 04:05:55 fbox sudo[2721]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
Oct 11 04:06:02 fbox systemd[1]: Started Erlang Port Mapper Daemon.
Oct 11 04:06:02 fbox systemd[1]: Starting robust, scalable and extensible realtime platform (XMPP server + MQTT broker + SIP service)...
Oct 11 04:06:02 fbox audit[2744]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2744 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:02 fbox kernel: audit: type=1400 audit(1633925162.613:53): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2744 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:02 fbox audit[2744]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2744 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:02 fbox kernel: audit: type=1400 audit(1633925162.625:54): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2744 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:02 fbox audit[2754]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2754/fd/" pid=2754 comm="erl_child_setup" requested_mask="r" denied_mask="r" fsuid=116 ouid=116
Oct 11 04:06:02 fbox kernel: audit: type=1400 audit(1633925162.681:55): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2754/fd/" pid=2754 comm="erl_child_setup" requested_mask="r" denied_mask=>
Oct 11 04:06:04 fbox audit[2780]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2780 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:04 fbox kernel: audit: type=1400 audit(1633925164.841:56): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2780 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:04 fbox audit[2780]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2780 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:04 fbox kernel: audit: type=1400 audit(1633925164.849:57): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2780 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:04 fbox audit[2791]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2791/fd/" pid=2791 comm="erl_child_setup" requested_mask="r" denied_mask="r" fsuid=116 ouid=116
Oct 11 04:06:04 fbox kernel: audit: type=1400 audit(1633925164.925:58): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2791/fd/" pid=2791 comm="erl_child_setup" requested_mask="r" denied_mask=>
Oct 11 04:06:10 fbox audit[2819]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2819 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:10 fbox kernel: audit: type=1400 audit(1633925170.517:59): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2819 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:10 fbox audit[2819]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2819 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:10 fbox kernel: audit: type=1400 audit(1633925170.533:60): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2819 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:10 fbox audit[2830]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2830/fd/" pid=2830 comm="erl_child_setup" requested_mask="r" denied_mask="r" fsuid=116 ouid=116
Oct 11 04:06:10 fbox kernel: audit: type=1400 audit(1633925170.581:61): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2830/fd/" pid=2830 comm="erl_child_setup" requested_mask="r" denied_mask=>
Oct 11 04:06:16 fbox audit[2856]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2856 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:16 fbox kernel: audit: type=1400 audit(1633925176.513:62): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2856 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:16 fbox audit[2856]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2856 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:16 fbox kernel: audit: type=1400 audit(1633925176.525:63): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2856 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:16 fbox audit[2867]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2867/fd/" pid=2867 comm="erl_child_setup" requested_mask="r" denied_mask="r" fsuid=116 ouid=116
Oct 11 04:06:16 fbox kernel: audit: type=1400 audit(1633925176.585:64): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2867/fd/" pid=2867 comm="erl_child_setup" requested_mask="r" denied_mask=>
Oct 11 04:06:20 fbox audit[2744]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/etc/ssl/openssl.cnf" pid=2744 comm="1_scheduler" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:20 fbox kernel: audit: type=1400 audit(1633925180.649:65): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/etc/ssl/openssl.cnf" pid=2744 comm="1_scheduler" requested_mask="r" denied_mas>
Oct 11 04:06:22 fbox audit[2894]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2894 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:22 fbox kernel: audit: type=1400 audit(1633925182.229:66): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2894 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:22 fbox audit[2894]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2894 comm="beam.smp" requested_mask="r" denied_mask="r" fsuid=116 ouid=0
Oct 11 04:06:22 fbox kernel: audit: type=1400 audit(1633925182.241:67): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/sys/fs/cgroup/cgroup.controllers" pid=2894 comm="beam.smp" requested_mask="r" >
Oct 11 04:06:22 fbox audit[2905]: AVC apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2905/fd/" pid=2905 comm="erl_child_setup" requested_mask="r" denied_mask="r" fsuid=116 ouid=116
Oct 11 04:06:22 fbox kernel: audit: type=1400 audit(1633925182.309:68): apparmor="ALLOWED" operation="open" profile="/usr/sbin/ejabberdctl" name="/proc/2905/fd/" pid=2905 comm="erl_child_setup" requested_mask="r" denied_mask=>
Oct 11 04:06:25 fbox systemd[1]: Started robust, scalable and extensible realtime platform (XMPP server + MQTT broker + SIP service).
Oct 11 04:06:26 fbox sudo[2721]: pam_unix(sudo:session): session closed for user root
Oct 11 04:06:26 fbox /usr/bin/plinth[463]: # service start mumble-server

and then other services started as well.

Oct 12: after the backup, restart of bind and restart of coturn

Oct 12 02:05:56 fbox /usr/bin/plinth[463]: # service start ejabberd
Oct 12 02:05:56 fbox sudo[8359]:   plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/service start ejabberd
Oct 12 02:05:56 fbox sudo[8359]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
-- Boot 463ac2e38da5462abe6ad92effaef42a --
Oct 12 07:11:05 fbox kernel: Booting Linux on physical CPU 0x0
Oct 12 07:11:05 fbox kernel: Linux version 5.10.0-9-armmp-lpae (debian-kernel@lists.debian.org) (gcc-10 (Debian 10.2.1-6) 10.2.1 20210110, GNU ld (GNU Binutils for Debian) 2.35.2) #1 SMP Debian 5.10.70-1 (2021-09-30)

On Oct 13: no service is running

Oct 13 02:12:23 fbox /usr/bin/plinth[500]: Running backup_pre hooks
Oct 13 02:12:23 fbox /usr/bin/plinth[500]: # backups create-archive --path '/media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::scheduled: daily: 2021-10-13:02:11' --comment '{{"type": "scheduled", "periods": ["daily"]}}' --paths /e>
Oct 13 02:12:23 fbox sudo[7230]:   plinth : PWD=/ ; USER=root ; COMMAND=/usr/share/plinth/actions/backups create-archive --path /media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::scheduled: daily: 2021-10-13:02:11 --comment {{"type>
Oct 13 02:12:23 fbox sudo[7230]:   plinth : (command continued) /etc/ejabberd/ejabberd.pem /etc/matrix-synapse/homeserver.yaml /etc/matrix-synapse/log.yaml /var/lib/matrix-synapse/homeserver.db /etc/matrix-synapse/homeserver.>
Oct 13 02:12:23 fbox sudo[7230]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=110)
Oct 13 02:13:28 fbox sudo[7230]: pam_unix(sudo:session): session closed for user root
Oct 13 02:13:28 fbox /usr/bin/plinth[500]: Error executing command - ['sudo', '-n', '/usr/share/plinth/actions/backups', 'create-archive', '--path', '/media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::scheduled: daily: 2021-10-13:0>
                                               "archive": {
                                                   "command_line": [
                                                       "/usr/bin/borg",
                                                       "create",
                                                       "--json",
                                                       "--comment",
                                                       "{{\"type\": \"scheduled\", \"periods\": [\"daily\"]}}",
                                                       "/media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::scheduled: daily: 2021-10-13:02:11",
                                                       "/etc/ez-ipupdate/",
                                                       "/etc/letsencrypt/",
                                                       "/etc/coturn",
                                                       "/var/lib/ejabberd/",
                                                       "/etc/ejabberd/letsencrypt/",
                                                       "/etc/matrix-synapse/conf.d/",
                                                       "/var/lib/matrix-synapse/media/",
                                                       "/var/lib/mumble-server",
                                                       "/etc/openvpn/",
                                                       "/var/lib/quassel/",
                                                       "/etc/radicale/",
                                                       "/var/lib/radicale/",
                                                       "/etc/tor/",
                                                       "/var/lib/tor/",
                                                       "/var/lib/tor-instances/",
                                                       "/etc/timezone",
                                                       "/etc/security/access.d/50freedombox.conf",
                                                       "/etc/snapper/configs/root",
                                                       "/etc/default/snapper",
                                                       "/etc/ssh/ssh_host_ecdsa_key",
                                                       "/etc/ssh/ssh_host_ecdsa_key.pub",
                                                       "/etc/ssh/ssh_host_ed25519_key",
                                                       "/etc/ssh/ssh_host_ed25519_key.pub",
                                                       "/etc/ssh/ssh_host_rsa_key",
                                                       "/etc/ssh/ssh_host_rsa_key.pub",
                                                       "/etc/apt/apt.conf.d/20auto-upgrades",
                                                       "/etc/bind/named.conf.options",
                                                       "/etc/ejabberd/ejabberd.yml",
                                                       "/etc/ejabberd/ejabberd.pem",
                                                       "/etc/matrix-synapse/homeserver.yaml",
                                                       "/etc/matrix-synapse/log.yaml",
                                                       "/var/lib/matrix-synapse/homeserver.db",
                                                       "/etc/matrix-synapse/homeserver.signing.key",
                                                      "/etc/matrix-synapse/homeserver.tls.crt",
                                                       "/etc/matrix-synapse/homeserver.tls.key",
                                                       "/var/lib/plinth/backups-manifests/media8fa5a314-cb9b-11eb-96c2-020a0480e0fcscheduled_daily_2021-10-130211.json"
                                                   ],
                                                   "duration": 33.10464,
                                                   "end": "2021-10-13T02:13:06.000000",
                                                   "id": "c67f9dcb85bcb480a70dbcc169bbcee81558aaabf5a072a4da56c40283e2423e",
                                                   "limits": {
                                                       "max_archive_size": 8.602159151483784e-05
                                                   },
                                                   "name": "scheduled: daily: 2021-10-13:02:11",
                                                   "start": "2021-10-13T02:12:33.000000",
                                                   "stats": {
                                                       "compressed_size": 468886686,
                                                       "deduplicated_size": 6103281,
                                                       "nfiles": 5423,
                                                       "original_size": 636725782
                                                   }
                                               },
                                               "cache": {
                                                   "path": "/root/.cache/borg/fb645b6f9e1cefa82bc08cc3ebe0de387307915777167b99f2e7a12af0d95c0d",
                                                   "stats": {
                                                       "total_chunks": 66436,
                                                       "total_csize": 5584127171,
                                                       "total_size": 7600532859,
                                                       "total_unique_chunks": 8235,
                                                       "unique_csize": 910532717,
                                                       "unique_size": 1342580732
                                                   }
                                               },
                                               "encryption": {
                                                   "mode": "none"
                                               },
                                               "repository": {
                                                   "id": "fb645b6f9e1cefa82bc08cc3ebe0de387307915777167b99f2e7a12af0d95c0d",
                                                   "last_modified": "2021-10-13T02:13:06.000000",
                                                   "location": "/media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc"
                                               }
                                           }
                                           , /var/lib/ejabberd/PREVIOUS.LOG: stat: [Errno 2] No such file or directory: '/var/lib/ejabberd/PREVIOUS.LOG'
                                           Traceback (most recent call last):
                                             File "/usr/share/plinth/actions/backups", line 310, in 
                                               main()
                                             File "/usr/share/plinth/actions/backups", line 306, in main
                                               subcommand_method(arguments)
                                             File "/usr/share/plinth/actions/backups", line 144, in subcommand_create_archive
                                               run(command, arguments)
                                             File "/usr/share/plinth/actions/backups", line 296, in run
                                               return subprocess.run(cmd, check=check, env=env, **kwargs)
                                             File "/usr/lib/python3.9/subprocess.py", line 528, in run
                                               raise CalledProcessError(retcode, process.args,
                                           subprocess.CalledProcessError: Command '['borg', 'create', '--json', '--comment', '{{"type": "scheduled", "periods": ["daily"]}}', '/media/8fa5a314-cb9b-11eb-96c2-020a0480e0fc::sched>
Oct 13 02:13:28 fbox /usr/bin/plinth[500]: Error running scheduled backup: ('backups', '{\n    "archive": {\n        "command_line": [\n            "/usr/bin/borg",\n            "create",\n            "--json",\n            ">
                                           Traceback (most recent call last):
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/__init__.py", line 119, in backup_by_schedule
                                               repository.schedule.run_schedule()
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/schedule.py", line 205, in run_schedule
                                               self._run_backup(periods)
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/schedule.py", line 297, in _run_backup
                                               repository.create_archive(name, app_ids, archive_comment=comment)
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/repository.py", line 180, in create_archive
                                               api.backup_apps(_backup_handler, path=archive_path, app_ids=app_ids,
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/api.py", line 130, in backup_apps
                                              _run_operation(backup_handler, packet,
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/api.py", line 427, in _run_operation
                                               handler(packet, encryption_passphrase=encryption_passphrase)
                                             File "/usr/lib/python3/dist-packages/plinth/modules/backups/__init__.py", line 111, in _backup_handler
                                               actions.superuser_run('backups', arguments, input=input_data.encode())
                                             File "/usr/lib/python3/dist-packages/plinth/actions.py", line 104, in superuser_run
                                               return _run(action, options, input, run_in_background, True,
                                             File "/usr/lib/python3/dist-packages/plinth/actions.py", line 200, in _run
                                               raise ActionError(action, output, error)
                                           plinth.errors.ActionError: ('backups', '{\n    "archive": {\n        "command_line": [\n            "/usr/bin/borg",\n            "create",\n            "--json",\n            "--com>
Oct 13 02:17:01 fbox slapd[935]: slap_global_control: unrecognized control: 1.3.6.1.4.1.4203.666.5.16
Oct 13 02:17:01 fbox CRON[7240]: pam_env(cron:session): Unable to open env file: /etc/default/locale: No such file or directory
Oct 13 02:17:01 fbox CRON[7240]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Oct 13 02:17:01 fbox CRON[7241]: (root) CMD (   cd / && run-parts --report /etc/cron.hourly)
Oct 13 02:17:01 fbox CRON[7240]: pam_unix(cron:session): session closed for user root
Oct 13 02:20:07 fbox systemd[1]: Started WordPress Scheduled Events Trigger (Cron).
Oct 13 02:20:08 fbox php[7245]: Could not open input file: /usr/share/wordpress/wp-cron.php
Oct 13 02:20:08 fbox systemd[1]: wordpress-freedombox.service: Main process exited, code=exited, status=1/FAILURE
Oct 13 02:20:08 fbox systemd[1]: wordpress-freedombox.service: Failed with result 'exit-code'.
Oct 13 02:30:07 fbox systemd[1]: Started WordPress Scheduled Events Trigger (Cron).
Oct 13 02:30:08 fbox php[7256]: Could not open input file: /usr/share/wordpress/wp-cron.php
Oct 13 02:30:08 fbox systemd[1]: wordpress-freedombox.service: Main process exited, code=exited, status=1/FAILURE

In summary: I have no clue what causes the problem after backup (on Oct 12, no more journal after the post backup hooks initiated restart of ejabberd) and/or of backup (on Oct 13 and 14, journal shows a backup failure and no post backup hooks, all services remain down).

Since the problem repeated twice the same, I expect it may continue (i.e. no service is working after backup), some help is welcome.

1 Like

Thank you for a thorough report. There seem to be three separate issues here!

  1. A file /var/lib/ejabberd/PREVIOUS.LOG is inaccessible and caused the backup to fail. This could be because the file has I/O error, dangling symlink or so. Created an issue to consider if such failures should be ignored. Addressing the failed file should handle this issue for you. See what state that file is in.
  2. A failed backup should result in leaving services in stopped state. We specifically handled this and should not have happened. Created an issue. Fixing the backup failure to help your case.
  3. Bind should have been started after the backup is completed. This could be because of a special treatment we gave it for transitioning it’s name change. Created an issue for that too. Masking the bind service to work around the issue for you.

Trigger manual backups to check if the issues are resolved.

Thanks for the quick answer.

I am not sure how to check a file state, here is what I see:

root@fbox:/var/lib/ejabberd# ls -la 
ls: cannot access 'PREVIOUS.LOG': No such file or directory
total 1760
drwx------ 1 ejabberd ejabberd    1026 Oct 14 18:46 .
drwxr-xr-x 1 root     root         712 Sep  4 06:30 ..
-r-------- 1 ejabberd ejabberd      20 Feb 14  2021 .erlang.cookie
-rw-r--r-- 1 ejabberd ejabberd     161 Oct 14 18:46 DECISION_TAB.LOG
-rw-r--r-- 1 ejabberd ejabberd     161 Oct 14 18:46 DECISION_TAB.LOG
-rw-r--r-- 1 ejabberd ejabberd      97 Oct 14 18:46 LATEST.LOG
-????????? ? ?        ?              ?            ? PREVIOUS.LOG
-rw-r--r-- 1 ejabberd ejabberd  107337 Jun  2 09:36 archive_msg.DAT
-rw-r--r-- 1 ejabberd ejabberd    5905 Apr  3  2021 archive_prefs.DAT

I have never seen something like that before.

Should just try to delete the file? Or put the SD card in another computer and run fsck?

Ewww! This is very likely a disk I/O error that corrupted the file. First step run fsck. If that does not fix it, try removing a file (take an image backup of the entire SD card, just in case).

I used ddrescue to make a copy, it said there was no read error.
I ran “sudo btrfs check --repair”, it did a number of things.
(for both, it was a first time for me to do that, I never did before, I think I don’t fully understand this).

After putting back the SD card into the Pioneer, booting was ok. In the backup, none of the previous backup appeared. Then, I did a manual backup and after that, they all appeared.

I’ll see what the situation is tomorrow morning after the next nightly backup.

1 Like