Ideas to mitigate distribution upgrade problems

The distribution upgrade from Bullseye to Bookworm caused issues for several FreedomBox users. See here, here, here, here, among others.

Beginning at this post several users proposed ways to mitigate future upgrade problems. This is a short summary of the ideas there.

Please use this thread to propose further ideas.

Idea 1: Delay distribution upgrades except for advanced users:

  • For all users: Turn off default automatic distribution upgrades or have a toggle option for ‘Advanced Users’.
  • The advanced users who enable automatic distribution upgrades can help identify and resolve issues during the distribution upgrade process.
  • For all other users the upgrade process will be delayed until all known issues have been identified and resolved.

Idea 2: Adapt an existing upgrade tool to ease the process:

  • Ubuntu tool/package ubuntu-release-upgrader-core can be rewritten for Debian and used it to perform the release upgrades.

Another idea:

  • In System > Backups add an option to set up automatic backups before a distro-upgrade is made.
  • Users who select this will have the distro-upgrade delayed if the backup fails. The user will receive a warning in the WebGUI that the backup failed. In this case, oldstable upgrades will continue.
  • If the distro-upgrade is delayed beyond oldstable end of support, a distro-upgrade will automatically happen even without the backup. The user will receive a warning in the WebGUI before this happens.

Idea 4: Check and automatically fix invariant states:

  • Make a list of “invariants”, things that must always be the case on a FreedomBox system. For example, the firewalld default zone must always be “internal” (and this implies, we cannot support any use-case where it is not “internal”).
  • Create one or more components to represent invariants of configuration, file system, databases, etc., and add these components to the appropriate apps.
  • The component will add a diagnostic check to the app. If it fails, then show a “quick fix” option to try to return to expected state.
  • Once a day, run all of the diagnostic checks for enabled apps, and automatically correct any problems. (Perhaps with an option to notify the admin through the web interface or email.)

Not speaking from bookworm experience here - I did it manually. The bullseye upgrade was done by Freedombox but it was not very obvious that anything was happening (apart from high CPU usage). I wonder how many upgrades get broken because the user is not aware that a release upgrade is in progress? I think there could be improvement in alerting the user that the upgrade is happening and coach them to wait for completion.

  • Notification - I saw this in plinth with the bullseye upgrade and was pleased to know it was happening. I usually ignore the notifications.
  • Can the plinth app get a release upgrade skin? Until the completion of the upgrade the plinth screen has a “RELEASE UPGRADE IN PROGRESS. PLEASE DO NOT REBOOT.” watermark or something?
  • Can plinth open a modal dialog box for administrative users at login instructing them that the upgrade is ongoing and to please allow it to finish? That’s kind of harsh, but may be what’s needed.
  • Can we add an extra layer of dialog if you are trying to reboot during the upgrade?
    Restart Dialog: Are you sure you want to restart? You will not be able to access this web interface for a few minutes until the system is restarted. (user says okay)
    Upgrade Dialog (only after reboot dialog while upgrade is in progress): Freedombox is currently upgrading and if this process is interrupted your system may become unusable. Would you like to wait for Freedombox to finish upgrading before you reboot? (user says OK)
    Upgrade Allow Dialog (after user says OK to Upgrade Dialog): Freedombox will create a notification in Plinth when the upgrade is complete. Check back soon!
  • It probably needs to be made clear to admin users through plinth that services will be curtailed during upgrade until all the service dependencies are met which may take hours.
  • There should be a warning in plinth of an upcoming release upgrade. Any communication we provide through plinth would not be received by the user when plinth is down during the upgrade.
  • Create periodic progress notifications in plinth for release upgrades
    – Freedombox release upgrade started at 6:00
    – Beginning upgrade to point release.
    – Beginning upgrade to new release.
    – Beginning full-upgrade.
    – Release upgrade completed. Your Freedombox is up to date.
    – Pick any milestones that are easy to work with. Progress, time, anything to create some visibility.