Having tested various options, I’ll be using **** as the permanent backing store for my Raspberry Pi Kubernetes cluster workloads and **** for working storage. Here’s why…
Test setup
%%% INSERT drawing.net picture here %%%
I use:
- a “server” (2GB) and a “client” (8GB) Raspberry Pi 4 Model B
- the Pis are running off a microSD Card on either a
- … and using one of
- Raspberry Pi OS Lite (32-bit)
- Raspberry Pi OS (64-bit) beta (no lite version available)
- Ubuntu Server 20.04.1 LTS (32-bit)
- Ubuntu Server 20.04.1 LTS (64-bit)
- … and both without and with (Ubuntu only) iscsi iSER
- The server Pi has a Kingston 120GB M.2 SSD attached to it via either a:
- the pis are linked together via a Netgear GS305 gigabit switch
- the pi network is then linked to a NAS via a Netgear GS308 gigabit switch
- the NAS is a Synology DS415+ with a 3x Seagate Ironwolf 7200rpm 6TB raid backing store
Trying it out yourself
Familiarity with ansible itself is required to run this, it won’t be detailed here.
Download and install Raspberry Pi Imager.
Install two sd cards with either of:
- Raspberry Pi OS (other) -> Raspberry Pi OS Lite (32-bit)
- Ubuntu -> Ubuntu Server 20.04.2 LTS (RPi 3/4/400)
Once clean cards are installed, OS-specific setup steps are needed:
Raspberry Pi OS Setup
Login to each pi using the default username pi, password raspberry, then allow ssh login by doing:
sudo touch /boot/ssh
sudo reboot
Ubuntu Setup
Login to each pi using the default username ubuntu, password ubuntu.
Change the password when prompted (suggest raspberry for ease of interop with Raspberry Pi OS test setups)
SSH setup
Ensure that the ansible hosts file has the correct username and password for each pi.
Clear out ssh host key cache on ansible host machine by doing:
ssh-keygen -R {pi-ip-address}
Update the ECSDA key fingerprint by doing:
ssh {pi-ip-address}
...
Are you sure you want to continue connecting (yes/no/[fingerprint])? yes
There’s no need to actually do the ssh login after this, ansible is now good to go.
Ansible Setup
Ansible config
To test on Rasperry Pi OS x64 (as of Jan 2021), ensure the following section is in your ~/.ansible.cfg:
[ssh_connection]
scp_if_ssh=True
Ansible role
The ansible role is on github, and an example playbook is provided.
Be sure to change the playbook vars header:
- vars:
ssd_pi_hostname: "CHANGE_ME: server pi (with ssd) hostname known to ansible :CHANGE_ME"
client_pi_hostname: "CHANGE_ME: client pi hostname known to ansible hosts file :CHANGE_ME"
nas_nfs_target: "CHANGE_ME: nas nfs target ({ip address}:{path}) :CHANGE_ME"
nas_iscsi_target_iqn: "CHANGE_ME: nas iscsi target iqn :CHANGE_ME"
nas_iscsi_target_ip: "CHANGE_ME: nas iscsi target ip address :CHANGE_ME"
pi_nfs_serve_network: "CHANGE_ME: an ip network spec that includes server & client pis :CHANGE_ME"
test_dir: "CHANGE_ME: dir on pi to use for workings (e.g. /media/test) :CHANGE_ME"
ssd_device: "CHANGE_ME: pi ssd device (e.g. /dev/sda) :CHANGE_ME"
pi_ntp_servers: "CHANGE_ME: space-separated ntp servers :CHANGE_ME"
Then run with
ansible-playbook ./pi-throughput-test.yml
… and then wait … 
… about an hour, possibly more!
The ansible output will end with a message giving the local temp filename that the results are stored in.
Results
Underlying the ansible role it uses Pi Benchmark script by James A Chambers.
This script gives an overall score for each storage target. In the score results below, the score is divided by one thousand and rounded to give an x.x figure in each case, which is a little easier to “see”.
The score is the most comprehensive script output given for a target, but it isn’t available for a /tmp target addressing ram, so for comparison, the simpler DDwrites result is given to be able to compare to ram. TL;DR : the ram is twice as fast as the ssd.