江心一庐

Build a storage cave for our story

Posted at 2022-10-13 making  linux 

When I organized lab stuff, I found something old: a HP P410 raid card and a HP 4x hard drive cage. Their well-designed and endured quality, inspires me to build a storage for backup and archive. I hope my data to be health and endured as they are. I wish it could be a cave to store our story.

After some attempts and failure, now I use Khadas VIM3 to drive JMB585 connected to hard drive cage. The files are transferred, synchronized to this cave by rsync or syncthing. This cave based on openZFS, can be shared by Samba, sshfs, NFS.

Another unrealized idea is that Thunderbolt as transfer interface, bypassing Khadas VIM3. I tracked both of my attempts and failure, for possible update.

JMB585 as SATA controller

Firstly, although I plan to use HP P410 to drive the hard drive cage, hp raid driver fail to load in ARM architecture. Then I found JMicron JMB585, which creates 5 SATA ports from PCIe, even shipped with M.2 interface.

JMB585 is a bridge controller between PCIe host and the storage devices with SATA/AHCI interface. JMB585 SATA Host provides five ports and supports Port Multiplier. It supports command-based switching (CBS) and FIS (Frame Information Structure)-based switching (FBS). JMB585 also support TRIM to the SSD and can transmit and receive data by both of AHCI mode and legacy IDE mode to and from the host respectively. ¹

The M.2 interface JMB585 can be installed in VIM3 M.2 socket. The five SATA cables are connected to HDD cage, powered from external 5V and 12V. At first, system only boot with no more than 3 plugged disks, but stuck in booting with 4 plugged disks. But if all 4 disks are plugged after system boot, all disks will be recognized. So I try to only load disks module(ahci, libahci) until system fully booted, and it does work. I guess that the module initialization may cost too much memory or other hardware resource while other modules are initialing at the same time.

Khadas didn't compiled ahci and libata module on kernel. I followed the Khadas guide to compile kernel in docker. Here is a note to avoid permission by using root account.

1

docker exec -u 0 -ti fenix /bin/bash

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

# Firstly, black ahci and libahci modules in modprobe setting
khadas@Khadas:~$ cat /etc/modprobe.d/blacklist.conf 
# black ahci for JMB585 
blacklist ahci
blacklist libahci
khadas@Khadas:~$sudo update-initramfs -u
# Then, load ahci and libahci util system booted
khadas@Khadas:~/bin$ cat /lib/systemd/system/data_cave.service

[Unit]
Description=Load and mount Cave
After=multi-user.target

[Service]
Type=idle
ExecStart=/home/khadas/bin/data_cave.sh enter
ExecStop=/home/khadas/bin/data_cave.sh out

[Install]
WantedBy=multi-user.target
khadas@Khadas:~/bin$ cat data_cave.sh
#!/bin/bash
# wait and load ahci module for JMB568
#sleep 10s
modprobe ahci
# wait and import cave zpool
sleep 10s
zpool import -a

ZFS on VIM3

Khadas VIM3 is running Ubuntu 20.04.4 LTS with Linux 6.0.0. Latest openZFS is compiled on VIM3 for better support².

1
2
3
4
5
6
7
8
9
10
11
12

sudo apt install build-essential autoconf automake libtool gawk alien fakeroot dkms libblkid-dev uuid-dev libudev-dev libssl-dev zlib1g-dev libaio-dev libattr1-dev libelf-dev linux-headers-generic python3 python3-dev python3-setuptools python3-cffi libffi-dev python3-packaging git libcurl4-openssl-dev
git clone https://github.com/openzfs/zfs
cd ./zfs
git checkout master
sh autogen.sh
./configure
make -s -j$(nproc)
make deb
sudo dpkg -i *.deb
modprobe zfs
sudo fdisk /dev/sdc
sudo zpool create cave mirror /dev/sda /dev/sdc

Because VIM3 only equip 2 GiB memory, zfs memory is limit to 128MB, to avoid system freeze after running out of memory.³

1
2
3
4
5

root@Khadas:~# cat /etc/modprobe.d/zfs.conf
options zfs zfs_arc_max=134217728
# enable smb share
sudo zfs set sharesmb=on cave
# add smb user

The HDD Cage

HP sell this cage as: Hot-plug Serial (SATA/SAS) hard drive cage, 402062-501.⁴ I enjoy its quality and appearance. It does work smoothly, except the activity indicator LED on front panel. They doesn't light up during reading/writing droved by JMB585. The JMB585 may not carry those signal to HDD cage.

Power Supply

The power meter reports that the typical consumption is 22W and no more than 25W. But more power is required to spin up 4 hard drives from parked state.

I'm using a modified HP 1U server power supply( HP 499249-001). This compact one still provides 460W output wattage in 12V. But its dimension is little bigger to fit existed frame. So it protrudes from frame.

I'm considering to deliver power in PD protocol for compact size, but find that few PD power supply generate more than 100W output and the cable are expensive.

HP P410 HBA card on ARM

(P410 is not used now, but I record this attempt for possible usage)

First, I check the datasheet of P410 or LSI2008, which say that LSI2008 works under PCIe 2.0 X1 in 5Gb/s. It just fits the PCIe bandwidth on Khadas VIM3. Using Khadas means that we don't expect high performance.

I rebuild the Khadas⁵ mainline linux kernel(5.18) with hpsa⁶ module, sg (Linux SCSI Generic) module and other maybe related. The PCIe slot delivery 3.3v and 12v. During booting, the hpsa module maybe crash then enter system directly after multiple reboot attempt (maybe cool boot is required). Or system booting hung all the time.

I also apply hpsahpa patch to kernel⁷:

1

patch -p1 < ../../5.18-patchset-v2/0001-scsi-hpsa-Add-function-to-check-if-device-is-a-disk-.patch

And even try to pass kernel parameters to u-boot^89101112.

1

kvim3#setenv bootargs 'hpsa.hpsa_use_nvram_hba_flag=true'

P410 fail in scsi generic initiation. Try to add BAR space.

This document outlines the Linux SCSI Generic (sg) driver interface as found in the 2.4 series kernels. The driver's purpose is to allow SCSI commands to be sent directly to SCSI devices. The responses of those commands can then be obtained. This type of driver is sometimes termed as a "pass through". In the case of SCSI disks, the block subsystem which is normally used to mount and access a disk, is bypassed permitting low level operations such as formatting to be performed. Various specialized applications for writing CD-Rs and document scanning use the sg driver.¹³

Some reviews which attempt to use hardware raid on RPI¹⁴¹⁵, suggests that P410 doesn't work on ARM easily now.

SSH over bluetooth

To access khadas without connectting to internet or serial port, I use bthelper¹⁶ from ThomasHabets¹⁷ to ssh khadas over bluetooth.

Install bthelper in both of khdas server and laptop client.

1
2
3

git clone https://github.com/ThomasHabets/bthelper
cd bthelper
./bootstrap.sh && ./configure && make && make install

After bluetooth is paired, start and enable bt-listener as system server at khadas server.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

khadas@Khadas:~$ cat /etc/systemd/system/bluetooth-ssh.service
[Unit]
Description=Bluetooth ssh
After=bluetooth.service
Requires=bluetooth.service

[Service]
ExecStart=/usr/local/bin/bt-listener -c 2 -t 127.0.0.1:22
Restart=always
RestartSec=10
StartLimitIntervalSec=0

[Install]
WantedBy=multi-user.target
khadas@Khadas:~$ sudo systemctl enable bluetooth-ssh.service
khadas@Khadas:~$ sudo systemctl start bluetooth-ssh.service

On the client,

1

ssh -oProxyCommand="bt-connecter AA:BB:CC:XX:YY:ZZ 2" user@cave-bt

Or in ~/.ssh/config:

1
2

Host cave-bt
    ProxyCommand bt-connecter AA:BB:CC:XX:YY:ZZ 2

And then just ssh user@cave-bt.

Misc

In the case of that I need to this disks directly for video editing. I want to expose the PCIe channel from Thunderbolt port. There are two types of adaptor: Thunderbolt to PCIe slot¹⁸, or Thunderbolt to M.2 slot. But they is almost use the same Thunderbolt chip. Thunderbolt 3 doesn't support USB3, so JMicron design a JMS58 to create PCIe channel beyond USB3. Some vendor combine JHL744L and JMS58 together, then it works on both of USB3 host and thunderbolt host¹⁹.

If VIM3 is used as a router, and the only one gigabyte Ethernet is used to as down link for quick access storage, the USB or M.2 B will be a place to introduce ISP up link. I can design a PCB like that: M.2 B -> USB -> RJ45. The AX88772 can be a reference chip.

Reference