Raspberry Pi migration without a separate hardware

In this article I will show how you can migrate a Raspberry Pi if you need to migrate software that cannot have long down times and you also don't have separate hardware available.
To do this you need to have some experience with the Raspberry Pi and Linux systems. You also need experience in dealing with SystemD and services operated under it. When it comes to network services (including web servers, Samba shares, etc.) it is necessary to have a plan to switch the services and to switch off the old one beforehand. It's not easy to distinguish between the new and the old world on the system.

Why this migration was necessary

The background to the Raspberry Pi migration is that I had an old installation of the Raspberry Pi system. This had been upgraded over the years from Debian 7 until most recently Debian 12. In addition to the fact that it had become slow, I also noticed that the versions of various software components no longer matched the versions delivered via image with the same Debian 12 original. And installed packages such as kernel, bootloader and firmware no longer received any updates at all. In the meantime, the update to Debian 13 was also due. But I didn't see the possibility to upgrade for the reasons mentioned.
So this time I wanted to set up the system again with all software components. It makes sense to use Debian 13 as well. But there were things running on the old system that I could not switch off for a long time. Everything was installed natively in the system and the backup ran via the Pi, among other things. It was important that this components can run, no matter how long it would take me to migrate all the services. I didn't have a second Pi available to set up the new system in parallel. And everything should stay on the Pi, among other things because of the backup hard drive project. So I was looking for a way to continue to operate the old system for a while and being able to build the new system without separate hardware.

A virtual Pi

A popular variant to be able to operate the old system parallel to the new one for a short time is virtualization. But a complete virtual machine would not have been usable on the Raspberry Pi. It wasn't that easy to invent a virtual Raspberry Pi on another system. In addition, I would not have been able to toggle the power of the hard drive for the backup project. I didn't want to move everything to other systems and the backup hard drive project wouldn't be adaptable. Operating the system as a container would have been another solution. Docker wasn't my area of ​​expertise and furthermore I would want to freeze the old OS with it's kernel into it. I couldn't have made it into an LXC for Proxmox because of the architecture difference (Arm vs. x86). A Linux technology that is basically the basis for all container systems is called Chroot (Change Root for Linux processes). I knew that you can run very different Linux systems, e.g. Almalinux, in Debian. But the technology is not necessarily intended for operating a system. But with a little effort I could have started the old Debian 12 in the new Debian 13. Only supporting hardware access such as GPIO for the hard drive would have been difficult. But while researching Chroot and Raspberry Pi, I came up with a fourth and more suitable solution.

Systemd-nspawn

SystemD has many components that are intended to manage the Linux system, its services, users and more. After Docker and LXC containers already existed, the developers of SystemD also developed a container system. As part of Systemd-container, the systemd-nspawn command is included and, similar to a chroot, can start a system like a container from a directory tree. The condition for this is that the guest system must run with SystemD as system manager. Systemd-nspawn does the annoying thing for the user that you have to tie together on a chroot all kernel near things so that processes can be executed in it (loosely). Services that were installed in the guest are also started as regular systemd services. By default, the container starts without its own IP network, which is different than LXC or Docker. However, by default the processes are accessible on the same network level as host processes, similar to a process in the chroot. This means a web server uses the IP of the host and tries to bind it to the host IP with standard port 80! It should be mentioned that you can also change the network of such a container to a private virtual network. Of course, as is usual with containers, a separate kernel is not started. All processes in it use the host's (new) kernel.

Change of the architecutre

Another point when migrating is the change in architecture. The old system was still a 32 bit Arm Linux. The new system should also be a 64 bit Arm Linux. This change can sometimes be a problem. On the one hand, the software cannot be migrated 1:1 to a system like Proxmox, because that is usually an x86_64 system. This is not compatible. In this case you can only recreate the configuration and install the x86 equivalent software on the target system. With chroot processes, a compatible mode like in x86_64 cannot be used and environments cannot be started in this way. But the problem is no longer relevant for Arm either. Because Arm64 allows you to run old 32-bit programs without any problems.

Preconditions

Usage of systemd-nspawn

In order to start the old system as a container in the new one, you need it on two disks. In my case two SD cards. The card with the new system is already in the Pi and is booting. It is specified as the device file /dev/mmcblk0 and the partitions below it. For the old system, I connected an external card reader and inserted the card from the old system there. This is then specified in the system, for example as a device file /dev/sda, here too expanded to the included partitions. The partitions are often No.1 boot partition and No.2 root partition. The container does not need to boot a kernel and therefore does not need to run through a boot loader. Therefore you don't need the boot partition from the old system. You just need to mount the root partition. But be careful, if /mnt is to be used for additional host and guest mount points, you should create a separate new directory underneath. So I mount /dev/sda1 to /mnt/oldpi. I created other mount point directories that were under /mnt in the old system as I wanted to use them again as before. Now almost everything is ready for the container.

Bind Mounts

With chroot you would now have to laboriously mount special file systems such as /dev, /run, /proc and /sys. This is completely eliminated when using systemd-nspawn for the container runtime. However, the container has private virtual file systems in their place. But if you want to give the guest access to the host's devices, you have to create a bind mount for /dev. But this can be achieved using a parameter to systemd-nspawn. For syntax, see below.
I used multiple mount points under /mnt. Among other things, there is the important mount point for the backup hard drive or the cloud storage for external backup. Therefore, as said above, the root partition in the new system is only mounted under /mnt/oldpi and the directories of the mount points used differently are created. So that the guest can now use this themselves (mount devices there), I also had to pass /mnt as a bind mount using arguments.

Syntax of the command:

Container starting script

The arguments that must be passed to systemd-nspawn to start the container are complex. You don't want to have to enter this every time. You also don't want to rely on it being somewhere in history. So it made sense to briefly write the entire command into a script. So that I could call this with all the arguments inside and only use Auto-Complete on the script name in the terminal. This is very helpful for starting the container.
The systemd-nspawn command is now called in the script. There are also the -b and -D <path> [summarized in the script as -bD <path>] arguments also passed to the command. -b means that systemd should "boot" in the container, i.e. systemd is started there as an init system and after that all other system stuff lime services. Without the parameter, only one command would be executed (cf. chroot /mnt whoami). The -D parameter tells systemd-nspawn to take the specified path as a root file system to start. It could also use disks, image files or OCI images (Docker). Then another parameter is needed.

#!/bin/bash
systemd-nspawn -bD /mnt/oldpi --bind=/mnt

Using with an interactive terminal

When starting the container you have to make sure that it happens in the foreground. If you start the systemd-nspawn command, the input and the display are immediately in the container. So you can't just go back to a regular shell on the host system. On the other hand, it must also remain open as a foreground process so that the container is running. A note: To be honest, I haven't checked whether the problem can be circumvented with a screen session. I operated the system headless and only accessed the Raspberry via SSH. This means that the start would have worked in one SSH session, but then I would have been "trapped" in the guest system. And I couldn't have closed the SSH session.
To get around the problem (without the screen command), I simply used the Raspberry Pi terminal, which is offered with a login screen after the system boots. A display was connected and I also connected the mouse and keyboard. This is the login screen of the host system, where I first have to log in with user + password. ! I also have to become the root user via sudo, since systemd-nspawn only works as a root user. And there the script mentioned above was called and the container started. I was then able to leave the terminal open and connect to the host via SSH. When you start systemd-nspawn, you first find yourself on the host system's login screen with display and input and log in there. Then you start the container there from the shell. The shell input waits for the systemd-nspawn process until it has ended. This only happens when the container has "shut down". You can then see the startup process of the old system on the terminal, just as you could see with the old SD card. You then end up on a login screen again, but now on that of the old system. Now you could log in there again with the login data from the old system.

Login into the container for a second session

If, like me, you started the container in the Raspberry Pi terminal, then you can't use the host's shell there. I use the Raspberry headless, which means I usually only log in via SSH. If you then need access to the container, you are in a separate session on the host system using SSH. But you can open another terminal for the container, which then has very similar requirements regarding display and input and foreground process as the actual systemd-nspawn command. But you can then open another SSH session to have access to the host and guest. The command to get into the container with a second session is sudo machinectl login oldpi . Here again you have to work with root, hence the sudo command. The machinectl command manages systemd containers and the login argument opens the login session in the guest. The name oldpi comes from the fact that my old system was mounted under /mnt/oldpi and started from there as a directory. Once the login screen has appeared (it is not equivalent to an SSH session or the chroot), you have to log in there with your user and password.

Services, IP Adresses and Ports

There are services in every Raspberry Pi system. Some are also servers and bind to an IP address and a port. You may also want to migrate services such as web servers from the old system to the new system. There are a few things to consider.
The container starts with the identical network configuration as the host, as shown here. This means that host and guest processes share the IP addresses and thus also the available ports. For example, an SSH server is installed on the new system. This one starts first - with the host system start. The guest is then started, in which an SSH server is also installed. This is also attempted to start at container start. But that fails: “Address already in use”. To solve this you can change the network (see above) or change the SSH server in the guest to a different port. But that doesn't fit with the short guest appearance during the migration. The error in the guest is just normal. This means that when you try to establish an SSH connection you always end up in the host.
I also wanted to migrate services such as web servers. These were not yet installed in the host system and started in the guest. So I could continue to use the old web server that existed there. So I first prepared the new installation for the new system while the server in the new system was not yet started. Then I logged into the guest, stopped the service there and started the service there on the host. So I was able to have the services moved one by one.

The real migration done after

The start of the old system was only the preparation for the actual migration. It's about being able to keep the old system running while you try to make gradual progress with the migration. I then needed another step. The old backup project wanted to be able to turn on the hard drive. This is still using a command that hasn't been integrated for a very long time and also the project was dead. Despite the container's hardware support, I could no longer use the old gpio command because, due to the architectural difference, it couldn't find any information in the proc-FS to recognize the Raspberry Pi type. I had to prematurely migrate the GPIO control script to the host and to the new equivalent pinctrl command. That happened very quickly. But the backup system used a web interface for access from other servers. Because of the container this worked unchanged to the web server. But this tried to connect to the user pi via SSH and only then start the shell script. So the web request ended up in the host system. But the backup system wanted to work in the guest for the time being. Here too I had to "overwrite" the shell script from the guest using an extra bind mount. In theory, the old system remained unchanged, but in the container it used different instructions.
I then also changed the web interface of the backup system. More about this here.
I then gradually installed the services on the new system and deactivated the services on the old system. When the last services were migrated, I logged into the terminal one last time within the container. There I just shut down the guest system with a "poweroff" command. So it returned back on the shell in the host system from the terminal there. I then just left the shell and unmounted the old system. And then the migration was completed.