iSCSI/mdadm shenanigans

I’ve spent the better part of the last six months wrestling with a problem with Open-iSCSI on CentOS 7 and 8. Here’s the scenario:

I set up four server virtual machines (A, B, C, D), and have two extra disks/block devices per server. I add these disks as backstores/LUNs for the iSCSI target configuration (in targetcli). I then set up a fifth server (which I call the client), which acts as an iSCSI initiator, and loaded the iSCSI disks from all of the targets using iscsiadm. I then created an mdadm RAID10 array with these iSCSI disks (six disks in the array, with two spares). I then formatted and mounted the resulting /dev/md0 array. This all works with no problems.

Part of the test is to shut down one of the servers, and see if mdadm begins rebuilding the array with the configured spares. When I shutdown the target server, the client did notice the disconnect. I hadn’t waited long enough to see if mdadm started rebuilding the array with the spare. Another, larger problem surfaced: the server lost the backstores when it rebooted.

I went through several iterations reproducing the problem. I had initially found this problem on the Linux Academy’s Playground servers. I then set up my own local VirtualBox VMs and was able to replicate the problem. I even set up these VMs with Arch Linux. At first Arch didn’t reproduce the problem, but then I remembered that Arch doesn’t install mdadm by default. Once I installed mdadm on the Arch servers, the problem came back.

So, the problem was on reboot the mdadm subsystem would see that the attached extra disks were Linux mdadm RAID members, which would lock out the target configuration. When the LIO subsystem (handles iSCSI) tried to restore the storage objects/backstores, mdadm already had them loaded so LIO/iSCSI said they were already in use. targetcli confirmed this with zero storage objects (no LUNs anymore, either).

The fix was deceptively simple. I had to create or modify /etc/mdadm.conf on the servers, and ensure it had only the following contents:

AUTO -all
ARRAY <ignore> uuid=UUID_of_RAID_members

The UUID of the RAID members was visible in lsblk -f on the server in the failure state, or was available in lsblk -f on the client. I needed the UUID of the RAID members (all the same for each iSCSI block device), not the UUID of the md0 filesystem. I rebooted all servers after making this change, and then rebooted the client. The RAID array on the client came back OK, so I finally figured out the problem!

Leave a Reply