iSCSI/mdadm shenanigans

I’ve spent the better part of the last six months wrestling with a problem with Open-iSCSI on CentOS 7 and 8. Here’s the scenario:

I set up four server virtual machines (A, B, C, D), and have two extra disks/block devices per server. I add these disks as backstores/LUNs for the iSCSI target configuration (in targetcli). I then set up a fifth server (which I call the client), which acts as an iSCSI initiator, and loaded the iSCSI disks from all of the targets using iscsiadm. I then created an mdadm RAID10 array with these iSCSI disks (six disks in the array, with two spares). I then formatted and mounted the resulting /dev/md0 array. This all works with no problems.

Part of the test is to shut down one of the servers, and see if mdadm begins rebuilding the array with the configured spares. When I shutdown the target server, the client did notice the disconnect. I hadn’t waited long enough to see if mdadm started rebuilding the array with the spare. Another, larger problem surfaced: the server lost the backstores when it rebooted.

I went through several iterations reproducing the problem. I had initially found this problem on the Linux Academy’s Playground servers. I then set up my own local VirtualBox VMs and was able to replicate the problem. I even set up these VMs with Arch Linux. At first Arch didn’t reproduce the problem, but then I remembered that Arch doesn’t install mdadm by default. Once I installed mdadm on the Arch servers, the problem came back.

So, the problem was on reboot the mdadm subsystem would see that the attached extra disks were Linux mdadm RAID members, which would lock out the target configuration. When the LIO subsystem (handles iSCSI) tried to restore the storage objects/backstores, mdadm already had them loaded so LIO/iSCSI said they were already in use. targetcli confirmed this with zero storage objects (no LUNs anymore, either).

The fix was deceptively simple. I had to create or modify /etc/mdadm.conf on the servers, and ensure it had only the following contents:

AUTO -all
ARRAY <ignore> uuid=UUID_of_RAID_members

The UUID of the RAID members was visible in lsblk -f on the server in the failure state, or was available in lsblk -f on the client. I needed the UUID of the RAID members (all the same for each iSCSI block device), not the UUID of the md0 filesystem. I rebooted all servers after making this change, and then rebooted the client. The RAID array on the client came back OK, so I finally figured out the problem!

Private git server with gitea


With Microsoft acquiring GitHub (for $7.5 BILLION), I now have incentive to host my own Git repositories. For the longest time I had thought that GitHub was an Open Source project, but then I was stymied when I tried to find a way to host my own GitHub server (it seems you need to be an Enterprise to host a proper private GitHub).

Fast forward a couple of years, and Vivek Gite’s *nixcraft blog post on the subject was linked to me via email. The purpose of this article is to document my efforts to install Gitea. I had tried to install it at, but that would require retooling the other web application at (WordPress). The folks on IRC (#nginx@freenode) said to make a subdomain, rather than a subdirectory. After being reminded that it’s easy to add CNAMEs to my DNS records, I now have

Why do this? I’d like to start using git more for my personal projects. I’d rather not store sensitive materials anywhere but something I fully control. Also, being able to link folks to my own repository rather Debian’s (or whoever’s) paste bin when I’m having issues is quite attractive to me.


  • A Linux server (mine is a ChunkHost chunk running Debian 9.4 [stretch])
  • nginx installed (with optional SSL/TLS certificates [HIGHLY RECOMMENDED])
  • A database engine (I already have MariaDB [10.1.26-MariaDB] installed). If in doubt, go with Gitea’s built-in SQLite3 database
  • SSH service enabled on the target host (at an optional nonstandard port)


The following instructions use non-root user and host in the commands. Change accordingly.

  1. Log into the target host via SSH (user will be assumed to *NOT* be root):
    ssh host -l user
  2. Make a staging directory and change to it:
    mkdir -p ~/src/gitea && cd ~/src/gitea
  3. Install prerequisite packages git, golang (from stretch-backports), wgetand zip:
    sudo apt install git wget zip
    sudo apt -t stretch-backports install golang
  4. Add a new user for Gitea
    sudo adduser --disabled-login --gecos 'Gitea' git
  5. Change this new user git:
    sudo -u git -i
  6. Get the latest version of gitea (currently v1.4.2)
    mkdir -p bin
    wget -O bin/gitea
    chmod +x bin/gitea
  7. Exit the git user shell
  8. Create systemd service file /etc/systemd/system/gitea.service for Gitea:

          After=mariadb.service mysqld.service postgresql.service memcached.service redis.service
          # Modify these two values and uncomment them if you have
          # repos with lots of files and get an HTTP error 500 because
          # of that
          ExecStart=/home/git/bin/gitea web
          Environment=USER=git HOME=/home/git

  9. Start Gitea:
          sudo systemctl enable gitea
          sudo systemctl start gitea
  10. Create an nginx site configuration file /etc/nginx/sites-available/
    server {                                                                                                                                                                                      
        listen 80;                                                                                                                                                                            
        listen [::]:80;                                                                                                                                                                       
        return 301;                                                                                                                                          
        # Redirect non-https traffic to https                                                                                                                                                     
        # if ( != https) {                                                                                                                                                               
        #     return 301 https://;                                                                                                                                               
        # } # managed by Certbot                                                                                                                                                                  
    server {                                                                                                                                                                                      
        listen 443 ssl;                                                                                                                                                                           
        ssl_certificate /etc/letsencrypt/live/host/fullchain.pem; # managed by Certbot                                                                                                     
        ssl_certificate_key /etc/letsencrypt/live/host/privkey.pem; # managed by Certbot                                                                                                   
        root /var/www/;                                                                                                                                                              
        location / {                                                                                                                                                                              
                client_max_body_size 364M;                                                                                                                                                        
                proxy_set_header Host ;                                                                                                                                                      
                proxy_set_header X-Real-IP ;                                                                                                                                          
                proxy_pass http://localhost:3000;                                                                                                                                                 
                proxy_connect_timeout 600;                                                                                                                                                        
                proxy_send_timeout 600
  11. Enable the new
    ln -s /etc/nginx/sites-{available,enabled}/
  12. Restart nginx
    sudo systemctl restart nginx
  13. Enter MariaDB client shell (admin account)
    mysql -u root -p'password'
  14. Add gitea database
  15. Add gitea user
    CREATE USER 'gitea' IDENTIFIED BY 'new_password';
  16. Grant privileges to gitea user
    GRANT ALL PRIVILEGES ON gitea.* TO 'gitea'@localhost IDENTIFIED BY 'new_password';
  17. Exit the MariaDB mysql client shell
  18. Now, you’re ready to configure Gitea at!

ip command nuggets

View IP address related information (all interfaces):

ip address


ip a

View the ARP table

ip neighbors

Cross-reference the IP addresses in the ARP cache with their local hostnames (if known). Replace “@localhost” with the hostname/IP address of the DNS server you want to query, or remove it altogether to use the system’s default DNS server:

for ip in $(ip nei | awk '{print $1}'); do dig -x $ip @localhost | grep ""; done

Or, wrapped in a shell function:

arp () {
  for ip in $(ip nei | cut -d' ' -f1); do 
    dig -x $ip @localhost | grep "";