trey – discursions

iSCSI/mdadm shenanigans

I’ve spent the better part of the last six months wrestling with a problem with Open-iSCSI on CentOS 7 and 8. Here’s the scenario:

I set up four server virtual machines (A, B, C, D), and have two extra disks/block devices per server. I add these disks as backstores/LUNs for the iSCSI target configuration (in targetcli). I then set up a fifth server (which I call the client), which acts as an iSCSI initiator, and loaded the iSCSI disks from all of the targets using iscsiadm. I then created an mdadm RAID10 array with these iSCSI disks (six disks in the array, with two spares). I then formatted and mounted the resulting /dev/md0 array. This all works with no problems.

Part of the test is to shut down one of the servers, and see if mdadm begins rebuilding the array with the configured spares. When I shutdown the target server, the client did notice the disconnect. I hadn’t waited long enough to see if mdadm started rebuilding the array with the spare. Another, larger problem surfaced: the server lost the backstores when it rebooted.

I went through several iterations reproducing the problem. I had initially found this problem on the Linux Academy’s Playground servers. I then set up my own local VirtualBox VMs and was able to replicate the problem. I even set up these VMs with Arch Linux. At first Arch didn’t reproduce the problem, but then I remembered that Arch doesn’t install mdadm by default. Once I installed mdadm on the Arch servers, the problem came back.

So, the problem was on reboot the mdadm subsystem would see that the attached extra disks were Linux mdadm RAID members, which would lock out the target configuration. When the LIO subsystem (handles iSCSI) tried to restore the storage objects/backstores, mdadm already had them loaded so LIO/iSCSI said they were already in use. targetcli confirmed this with zero storage objects (no LUNs anymore, either).

The fix was deceptively simple. I had to create or modify /etc/mdadm.conf on the servers, and ensure it had only the following contents:

AUTO -all
ARRAY <ignore> uuid=UUID_of_RAID_members

The UUID of the RAID members was visible in lsblk -f on the server in the failure state, or was available in lsblk -f on the client. I needed the UUID of the RAID members (all the same for each iSCSI block device), not the UUID of the md0 filesystem. I rebooted all servers after making this change, and then rebooted the client. The RAID array on the client came back OK, so I finally figured out the problem!

Private git server with gitea

PURPOSE

With Microsoft acquiring GitHub (for $7.5 BILLION), I now have incentive to host my own Git repositories. For the longest time I had thought that GitHub was an Open Source project, but then I was stymied when I tried to find a way to host my own GitHub server (it seems you need to be an Enterprise to host a proper private GitHub).

Fast forward a couple of years, and Vivek Gite’s *nixcraft blog post on the subject was linked to me via email. The purpose of this article is to document my efforts to install Gitea. I had tried to install it at https://eldon.me/git/, but that would require retooling the other web application at https://eldon.me/ (WordPress). The folks on IRC (#nginx@freenode) said to make a subdomain, rather than a subdirectory. After being reminded that it’s easy to add CNAMEs to my DNS records, I now have https://git.eldon.me.

Why do this? I’d like to start using git more for my personal projects. I’d rather not store sensitive materials anywhere but something I fully control. Also, being able to link folks to my own repository rather Debian’s (or whoever’s) paste bin when I’m having issues is quite attractive to me.

PREREQUISITES

A Linux server (mine is a ChunkHost chunk running Debian 9.4 [stretch])
nginx installed (with optional SSL/TLS certificates [HIGHLY RECOMMENDED])
A database engine (I already have MariaDB [10.1.26-MariaDB] installed). If in doubt, go with Gitea’s built-in SQLite3 database
SSH service enabled on the target host (at an optional nonstandard port)

PROCEDURE

The following instructions use non-root user and host in the commands. Change accordingly.

Log into the target host via SSH (user will be assumed to *NOT* be root):
```
ssh host -l user
```
Make a staging directory and change to it:
```
mkdir -p ~/src/gitea && cd ~/src/gitea
```
Install prerequisite packages git, golang (from stretch-backports), wgetand zip:
```
sudo apt install git wget zip
sudo apt -t stretch-backports install golang
```

Add a new user for Gitea

sudo adduser --disabled-login --gecos 'Gitea' git

Change this new user git:
```
sudo -u git -i
```

Get the latest version of gitea (currently v1.4.2)

mkdir -p bin
wget -O bin/gitea https://dl.gitea.io/gitea/1.4.2/gitea-1.4.2-linux-amd64
chmod +x bin/gitea

Exit the git user shell
```
exit
```

Create systemd service file /etc/systemd/system/gitea.service for Gitea:

      [Unit]
      Description=Gitea
      After=syslog.target
      After=network.target
      After=mariadb.service mysqld.service postgresql.service memcached.service redis.service
    
      [Service]
      # Modify these two values and uncomment them if you have
      # repos with lots of files and get an HTTP error 500 because
      # of that
      ###
      #LimitMEMLOCK=infinity
      #LimitNOFILE=65535
      Type=simple
      User=git
      Group=git
      WorkingDirectory=/home/git
      ExecStart=/home/git/bin/gitea web
      Restart=always
      Environment=USER=git HOME=/home/git
    
      [Install]
      WantedBy=multi-user.target

Start Gitea:

      sudo systemctl enable gitea
      sudo systemctl start gitea

Create an nginx site configuration file /etc/nginx/sites-available/git.host:

server {                                                                                                                                                                                      
    listen 80;                                                                                                                                                                            
    listen [::]:80;                                                                                                                                                                       
    server_name git.host;                                                                                                                                                             
    return 301 https://git.host;                                                                                                                                          
                                                                                                                                                                                              
    # Redirect non-https traffic to https                                                                                                                                                     
    # if ( != https) {                                                                                                                                                               
    #     return 301 https://;                                                                                                                                               
    # } # managed by Certbot                                                                                                                                                                  
                                                                                                                                                                                              
}                                                                                                                                                                                             
                                                                                                                                                                                              
server {                                                                                                                                                                                      
    listen 443 ssl;                                                                                                                                                                           
    server_name git.host;
    ssl_certificate /etc/letsencrypt/live/host/fullchain.pem; # managed by Certbot                                                                                                     
    ssl_certificate_key /etc/letsencrypt/live/host/privkey.pem; # managed by Certbot                                                                                                   
                                                                                                                                                                                              
    root /var/www/git.host/;                                                                                                                                                              
    location / {                                                                                                                                                                              
            client_max_body_size 364M;                                                                                                                                                        
            proxy_set_header Host ;                                                                                                                                                      
            proxy_set_header X-Real-IP ;                                                                                                                                          
            proxy_pass http://localhost:3000;                                                                                                                                                 
            proxy_connect_timeout 600;                                                                                                                                                        
            proxy_send_timeout 600
    }
}

Enable the new git.host:

ln -s /etc/nginx/sites-{available,enabled}/git.host

Restart nginx
```
sudo systemctl restart nginx
```
Enter MariaDB client shell (admin account)
```
mysql -u root -p'password'
```
Add gitea database
```
CREATE DATABASE gitea;
```

Add gitea user

CREATE USER 'gitea' IDENTIFIED BY 'new_password';

Grant privileges to gitea user

GRANT ALL PRIVILEGES ON gitea.* TO 'gitea'@localhost IDENTIFIED BY 'new_password';

Exit the MariaDB mysql client shell
```
exit
```
Now, you’re ready to configure Gitea at https://git.host/!

Renewing my Free SSL certificate with StartCom

Since my Secure Sockets Layer (SSL) certificate needs are quite modest, I use the free SSL certificates from StartCom. Here are my instructions for renewing it, which is almost identical to the process for creating an SSL certificate with StartCom.

Go to the main site, and log in with your StartSSL Open Identity certificate. Create one if need be.
Validate your email address in the resulting dashboard. The validation will be in force for 30 days.
Validate every domain you wish to renew.

Generate the certificate for the domain you are renewing.

Select the StartCom SSL Certificates Wizard.
Choose Web Server SSL/TLS Certificate, and click Continue.
Your validated domain should show in the list. Enter the domain you’re generating the keys for into the “Please enter your full hostname here” input field.
Under “Please submit your Certificate Signing Request (CSR):” select “Generated by Myself”. You can use the form, but generally I remember reading that it’s more secure for you to generate your Certificate Request (CSR) and private key from it yourself, using an offline OpenSSL script invocation. The present a suitable openssl command invocation, to which I added the -subj option to avoid having it prompt me for that information:
```
openssl req -new -newkey rsa:2048 -nodes -out eldon.me.csr -keyout eldon.me.key -subj "/C=US/ST=Georgia/L=Mableton/O=Eldon Carl Blancher III/CN=eldon.me"
```

Copy the content of the CSR into the resulting form. It will look something like this:

The actual code has been obfuscated to protect it, I highly doubt that’s a valid CSR now.

Click “Submit.” If the CSR is accepted, the resulting page should provide a link to the new certificate files (the “here” link), as a ZIP archive.

Download and unpack the archive. There are several ZIP archives with in it, one for some possible web servers. The Apache.zip file contained two files:

1_root_bundle_crt, renamed to startcom.crt
2_eldon.me.crt, renamed to eldon.me.crt

Now that the certificates are generated, I need to add them to my apache2 web server.

My current sites-available/eldon.me apache2 configuration looks like this:

<VirtualHost *:80>
DocumentRoot /var/www/eldon.me
ServerName www.eldon.me
ServerAlias eldon.me
ServerAdmin trey@blancher.net
ErrorLog /var/log/apache2/eldon.me-error.log
TransferLog /var/log/apache2/eldon.me-access.log
RedirectPermanent / https://eldon.me/
</VirtualHost>

<VirtualHost *:443>
ServerName www.eldon.me
ServerAlias eldon.me
ServerAdmin trey@blancher.net
SSLEngine on
SSLProtocol all -SSLv2 -SSLv3
SSLCipherSuite ALL:!ADH:!EXPORT:!SSLv2:RC4+RSA:+HIGH:+MEDIUM
SSLCertificateFile /etc/ssl/certs/eldon.me.pem
SSLCertificateKeyFile /etc/ssl/private/eldon.me.key
SSLCertificateChainFile /etc/ssl/certs/startcom_sub.class1.server.ca.pem
SSLCACertificateFile /etc/ssl/certs/startcom_ca.pem
DocumentRoot /var/www/eldon.me
Customlog /var/log/apache2/eldon.me-access.log combined
ErrorLog /var/log/apache2/eldon.me-error.log
HostnameLookups On
</VirtualHost>

<Directory /var/www/eldon.me>
        Options FollowSymLinks            
#       AllowOverride Limit Options FileInfo
  AllowOverride All
        DirectoryIndex index.php
</Directory>

I changed these three lines:

SSLCertificateFile /etc/ssl/certs/eldon.me.crt
SSLCertificateKeyFile /usr/local/apache/conf/eldon.me.key
SSLCertificateChainFile /usr/local/apache/conf/startcom.crt

Reload apache2, and the new SSL certificate is loaded!

ip command nuggets

View IP address related information (all interfaces):

ip address

ip a

View the ARP table

ip neighbors

Cross-reference the IP addresses in the ARP cache with their local hostnames (if known). Replace “@localhost” with the hostname/IP address of the DNS server you want to query, or remove it altogether to use the system’s default DNS server:

for ip in $(ip nei | awk '{print $1}'); do dig -x $ip @localhost | grep "in-addr.arpa"; done

Or, wrapped in a shell function:

arp () {
  for ip in $(ip nei | cut -d' ' -f1); do 
    dig -x $ip @localhost | grep "in-addr.arpa"; 
  done
}