Fedora 28->29 Atomic Host Upgrade Guide

Introduction

This week we put out the first release of Fedora 29 Atomic Host. This will be the last major release of Fedora Atomic Host as we prepare for Fedora CoreOS which will be released in Fedora 30.

In this post we’ll quickly list some known issues and then talk about updating an existing Fedora 28 Atomic Host system to Fedora 29. We’ll cover preparing the system for upgrade and performing the upgrade.

NOTE: If you really don’t want to upgrade to Fedora 29 see the later section: Appendix B: Fedora 28 Atomic Host Life Support.

Known Issues

Device timeout when installing from ISO with swap space.

We found an issue where instances installed via the ISO image via anaconda were having long boot times because of device timeouts. We traced this down to a bug in dracut and this will be fixed in the next release.

Incompatibilities with OpenShift Origin >= 3.11 and Kubernetes >= 1.10.4

In newer versions of systemd an API was removed that was needed by older versions of OpenShift/Kubernetes. It is recommended that you upgrade to Origin 3.11 or Kubernetes > 1.10.4 before moving to Fedora 29 Atomic Host. If you must use an older version of OpenShift/Kubernetes with Fedora 29 Atomic Host please see the later section: Appendix A: Using older versions of Kubernetes/OpenShift on Fedora 29 Atomic Host. Discussion of this issue can be found in the pagure ticket.

Preparing for Upgrade

Before we update to Fedora 29 Atomic Host we should check to see that we have at least a few GiB of free space in our root filesystem. The update to Fedora 29 can cause your system to retrieve more than 1 GiB of new content (not shared with Fedora 28) and thus we’ll need to make sure we have plenty of free space.

NOTE: Luckily if you skip this step OSTree will perform filesystem checks to make sure that upgrade will error gracefully before filling up the filesystem.

Upgrading

Now we should be ready for the upgrade. If you are hosting any services on your instance you may want to prepare for them to have some downtime.

Currently we are on Fedora 28 Atomic Host using the fedora/28/x86_64/atomic-host ref.

[vagrant@host ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-atomic:fedora/28/x86_64/atomic-host
                   Version: 28.20181007.0 (2018-10-07 20:43:44)
                    Commit: 8df48fa2e70ad1952153ae00edbba08ed18b53c3d4095a22985d1085f5203ac6
              GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1

In order to do the upgrade we need to add the location of the Fedora 29 repository as a new remote (similar to a git remote) for ostree to know about:

[vagrant@host ~]$ sudo ostree remote add --set=gpgkeypath=/etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-29-primary fedora-atomic-29 https://dl.fedoraproject.org/atomic/repo/

You can see from the command that we are adding a new remote known as fedora-atomic-29 with a remote url of https://dl.fedoraproject.org/atomic/repo/ We are also setting the gpgkeypath variable in the configuration for the remote. This tells OSTree that we want commit signatures to be verified when we download from a remote.

Now that we have our fedora-atomic-29 remote we can do the upgrade!

[vagrant@host ~]$ sudo rpm-ostree rebase fedora-atomic-29:fedora/29/x86_64/atomic-host
26 delta parts, 9 loose fetched; 264197 KiB transferred in 44 seconds        
Upgraded:                                      
  NetworkManager 1:1.10.12-1.fc28 -> 1:1.12.4-1.fc29        
  NetworkManager-libnm 1:1.10.12-1.fc28 -> 1:1.12.4-1.fc29
  NetworkManager-team 1:1.10.12-1.fc28 -> 1:1.12.4-1.fc29
  acl 2.2.53-1.fc28 -> 2.2.53-2.fc29                
  atomic 1.22.1-2.fc28 -> 1.22.1-27.gitb507039.fc29       
  atomic-devmode 0.3.7-4.fc28 -> 0.3.7-6.fc29        
  atomic-registries 1.22.1-2.fc28 -> 1.22.1-27.gitb507039.fc29
  attr 2.4.48-3.fc28 -> 2.4.48-3.fc29              
...
  python-unversioned-command-2.7.15-10.fc29.noarch
  python3-pyyaml-4.2-0.1.b4.fc29.x86_64
Run "systemctl reboot" to start a reboot
[vagrant@host ~]$ sudo systemctl reboot
[vagrant@host ~]$ Connection to 192.168.121.95 closed by remote host.
Connection to 192.168.121.95 closed.

After reboot we can log in and see the status:

$ vagrant ssh 
Last login: Wed Oct 31 21:14:44 2018 from 192.168.121.1
[vagrant@host ~]$ rpm-ostree status
State: idle
AutomaticUpdates: disabled
Deployments:
● ostree://fedora-atomic-29:fedora/29/x86_64/atomic-host
                   Version: 29.20181025.1 (2018-10-25 14:46:54)
                    Commit: 4a999b4b303b47468ff1464051a14fd075d2e7b8bb647584b7cc80fed48cf27b
              GPGSignature: Valid signature by 5A03B4DD8254ECA02FDA1637A20AA56B429476B4

  ostree://fedora-atomic:fedora/28/x86_64/atomic-host
                   Version: 28.20181007.0 (2018-10-07 20:43:44)
                    Commit: 8df48fa2e70ad1952153ae00edbba08ed18b53c3d4095a22985d1085f5203ac6
              GPGSignature: Valid signature by 128CF232A9371991C8A65695E08E7E629DB62FB1
[vagrant@host ~]$
[vagrant@host ~]$ cat /etc/os-release | head -n 2
NAME=Fedora
VERSION="29.20181025.1 (Atomic Host)"

We are now on Fedora 29 Atomic Host. Now is a good time to check your services (most likely running in containers) to see if they are still working. If not, then you always have the rollback command: sudo rpm-ostree rollback.

NOTE: Over time you can see updated commands for upgrading Atomic Host between releases by visiting this wiki page.

Appendix A: Using older versions of Kubernetes/OpenShift on Fedora 29 Atomic Host

Due to an issue that arises from the combination of the version of systemd that ships with Fedora 29 Atomic Host, and the version of runc that’s bundled with Kubernetes and with OpenShift Origin, users of certain Kubernetes and OpenShift Origin releases must take steps to keep their Kubernetes or Origin clusters running following an upgrade to 29.

Users running Kubernetes releases older than v1.10.4 or OpenShift Origin releases older than v3.11 would do best to upgrade their Kubernetes or Origin clusters to a more recent version before undertaking an upgrade to Fedora 29 Atomic Host.

For users who prefer to upgrade to 29 without changing their cluster version, it’s possible to work around the issue by changing the cgroupdriver for both docker and for either the kubelet or the orgin-node component from its default value of systemd to cgroupfs.

For docker (applies to both Kubernetes and OpenShift Origin clusters):

# cp /usr/lib/systemd/system/docker.service /etc/systemd/system/
# sed -i 's/cgroupdriver=systemd/cgroupdriver=cgroupfs/' /etc/systemd/system/docker.service

For kubelet (applies to Kubeadm-based Kubernetes clusters):

# sed -i 's/cgroup-driver=systemd/cgroup-driver=cgroupfs/' /etc/systemd/system/kubelet.service.d/kubeadm.conf

For origin-node (applies to OpenShift Origin clusters):

# vi /etc/origin/node/node-config.yaml

Add the following element to the kubeletArguments: section:

cgroup-driver:
  - "cgroupfs"

After making these changes, rebase to 29 as directed above. After reboot the nodes should return in Ready state as expected.

Appendix B: Fedora 28 Atomic Host Life Support

As with Fedora 27 we will keep updating the Fedora 28 Atomic Host tree but we won’t be performing regular testing on that tree and there will be no more formal releases for Fedora 28. Please move to Fedora 29 to take advantage of regular testing and formal releases every two weeks.

Conclusion

As in the past, the transition to Fedora 29 Atomic Host should be an easy process. If you have issues please join us in IRC (#atomic on freenode) or on the atomic-devel mailing list.

Project Atomic is now sunset