4 x 2.5" HDD RAID5 RASPBERRY PI
​STORJ FARMING
very simple software setup
low-intermediate hardware build

For more info of what the STORJ network is, take a look at https://storj.io/
​
The guide is in two parts, a brief hardware overview, and a full software installation guide.

This guide last updated 22nd December '17. (Added new Raspbian OS "Stretch" and 2 GB Swapfile)​

HARDWARE:




​
 So, there are many projects in the crypto-currency scene that either require a large amount of storage for a blockchain or alternatively can rent out storage for income (Sia/Storj). And others like Burst actually mine on HDD. 


My thinking was that 2nd hand laptop HDD's would be the most compact but most importantly the cheapest method of storage. 


Also 2.5" SATA only requires a 5v input. (3.5" HDDs require 5v and 12v inputs) making laptop drives readily compatible with USB. If only there were adaptors for this.....There are! and they're very cheap also.
This prototype is constructed using 4 x 40GB drives. As this is an experiment I wasn't going to go nuts with my overheads. They were £3.50 each from CEX which conveniently gives a 24 month warranty which may come in useful if this is a success and would take most of the risk away from farming data on 2nd hand drives.

In theory, if it works with the cheap drives, it's just a case of scaling them to 2TB or 4TB drives with no additional software or hardware requirements. Just plug and play.


I've gone back to my trusted rack method of acrylic and motherboard risers, giving good ventilation and aesthetics at low cost. It's also very easy to work with and modular for future expansion, just add another layer.

The 6A power supply provides the 5v through the white wires, and are soldered directly to a USB hub (soldered to the barrel connector pins, with the hub sandwiched between the 4 drives). I show the construction below, layer by layer.

All drives are using the pig-tail type SATA adaptor to get their power from the hub only. The +5v pin on the USB to the Pi has been disconnected to prevent back-feeding. The USB hub is 7 port and so is also powering the Pi through it's micro USB connection.

The power supply is a 5v 6A DC out, 110/220v AC in and should be sufficient to power all drives and the pi. Each drive has a power rating on it's label of approx 0.7A (varied between the brands of drive I have, .7 is an average)

All images are thumbnails. Click to enlarge for detail.

Summary of parts (UK sellers):

4 x 40GB 2.5" HDDs                             £14 CEX link here

Raspberry Pi 3b                                     £28.79 ebay link here
​
16Gb SD card (class 10)                      £5.20 ebay link here

Naked 7 port USB Hub                        £7 MOD MY PI link here

2 x A3 size, 3mm thick Acrylic          £6 ebay link here

15cm USB to micro USB (pi power)£1 ebay link here

4 x Sata -> USB adapters                  £6.80 ebay link here

5v 6A 30W DC Power Supply           £8.50 ebay link here     Suitable plug required extra

M3 Nylon spacers/risers sets with screws/nuts. Many used and assembled to get desired height. Multiple kits approx £8 worth. £3.86 Free delivery ebay.

Total cost for this prototype approx £90
Estimated cost for 8TB version £325
Estimated cost for 16TB version £420


**Note all second hand drives from CEX come with 24 month warranty! (I don't work for them or get commission. It's just hands down the cheapest and most reliable place for components for crypto builds)

SOFTWARE:

Step (1) Preparing the Pi

Once again with this project the preparation steps are the same as for other nodes, I have made a beginner friendly guide with images, step-by-step that shows the basics of formatting the SD card and installing the disk-image. "Getting started" Link opens a new window.

​

                                                                                                                     Getting Started


Then continue below...
Step (2) WiFi - Optional
Wifi is available once the Pi is rebooted. Either reboot now using
sudo reboot
and remove the ethernet cable. Or continue, and reboot later as it is required after the next step.
With the Pi turned on, booted and you logged in with SSH we can start the setup.

First WiFi network connections are stored:
​
                         sudo nano /etc/wpa_supplicant/wpa_supplicant.conf

Go to the bottom of the file and add the following text:
    
                             network={
                                 ssid="YOUR WIFI ROUTER NAME"
                                 psk="YOUR WIFI ROUTER PASSWORD"
                             }

**Note: Enter details between the quotes, leave the  "" in
save and exit using

                             ctrl+O
Then 'enter'
                            ctrl+X
Step (3) Expand filesystem
So that the Pi makes full use of the SD card you've inserted, enter:​

                             sudo raspi-config

The default password for obvious security reasons should be changed. That is done in this menu.

select "1 change user password", and follow the on-screen instructions. This will be the new password when using PUTTY to SSH into the Pi.

Because we installed a 2GB image onto the card, the Pi may think that the card is only 2GB in size. So we tell it to expand the file-system (returns it to it's full size allowed):

select "7 advanced options"

select "A1 expand filesystem"

select 'finish' system will reboot, (remove the ethernet cable if you didn't at the previous step). And log back in with SSH and PUTTY once reboot is complete.
Step (4) Updates
The image we downloaded may have an update or two since they released it:

                             sudo apt-get update
                             sudo apt-get upgrade
                             

select 'y' to accept the storage requirement

Step (5) Configuring the HDDs to RAID5 (or any HDD configuration you prefer)

As this project is entirely storage centered there a quite a few steps, simple steps, to configure it.

​First we can install a program called MDADM (Multiple Disk or Device Administration, so the wiki says?).
This will be what manages the 4 drives used here, you can use as many drives as you like if you can find a way to power and connect them!

The program is installed simply with:

                                                            sudo apt install mdadm
In install select <ok>
and "all" as the default is fine.





Now we have mdadm installed we can make the RAID.

There are 3 commands shown to the left, first

                                                   lsblk

shows what USB devices are connected and recognized. Here listed as sda, sdb, sdc and sdd. (The mmcblk0 and its partitions are the SD card)
Your Pi will 99% of the time list USB devices as shown to the left. Make a note of the letters a,b,c,d etc it has assigned.

The next command builds the RAID drive:​
                                                       sudo mdadm --create /dev/md0 --level=5 --raid-devices=4 /dev/sda /dev/sdb /dev/sdc /dev/sdd

Here, customize as you see fit. I'll explain:
"/dev/md0" is what the cluster of drives will be labeled as once the RAID is built. You can change this if you have another preference.
"--level=5" is the level of RAID that will be built, look at Standard RAID levels, WIKI for other options, simply change the number.
"--raid-devices=4" is the number of HDDs you want mdadm to use to create your RAID. (usually all of them)
and after that, separated by spaces, is the current label of every drive you want mdadm to use. If your Pi has labelled them something different this is where you would your specific drive labels.

Once you enter your command the HDD activity lights should flicker manically as it builds the drive. It does this in the background.

To check on it's build progress before progressing to the next step we use :
​
                                                   ​​cat /proc/mdstat

As shown at the bottom of that last screenshot, devices used, a % progress of the build and an estimated time of completion. 

**Note this prototype is 4x40GB drives, making /dev/md0  120GB usable storage and 40GB as unusable parity. It estimates 1 Hr for the build. This setup in theory will work exactly the same if four 500GB or 2TB drives were used, but expect the build time to rise accordingly.

Next, now that we have our RAID, we should create a partition and format it.
For this we use a program called fdisk: (amend the below command if your drive label is different)

                                         sudo fdisk /dev/md127


Press:                               n                                 (for new partition)
Press:                               p                                 (for primary)

Then press enter a few times for the default size.

and at the end press                w                   to write the changes.
This then adds the partition layer (p1) to each drive when         lsblk       is used again.

Use:

                                       sudo mkfs.ext4 /dev/md127p1
​
​To format the new partition to ext4. (Again change md127p1 to what ever your partition is called) Use command           lsblk         for your label)



​
Because mount commands must be run as "sudo" superuser it has admin permissions. To be safe and avoid any problems we change the owner to "pi" or whatever your username is, with:

​                                sudo chown ${USER} -R /home/${USER}/storj
You'll notce the MOUNTPOINT column is empty.
First make a folder to moun the partition to:

                                mkdir storj

Mount the partition for our use by entering

​                   sudo mount -o rw,auto /dev/md127p1 /home/${USER}/storj
An explanation:
-o rw,auto (-o,gives options, rw means read/write permissions, auto should automount)
/dev/md127p1 (what to mount, enter the md<num><p1> you get from lsblk command)
/home/${USER}/storj (where to mount it)

​Our drive is completly good to go! MOUNTPOINT lists the partition as mounted to our desired destination!
Step(5) Installing "Screen"

Screen is a very useful program. It allows for multiple instances of terminal to be run, disconnected from the Pc they were started on and re-joined later. This will allow you to start the synchronization process of the blockchain for example, turn off or close the terminal window, and the sync process will continue on the Pi. Without it when you close the terminal window the process stops.

To install screen:
                                               sudo apt-get install screen

Using screen is very easy.

The program is started using:
                                           screen bash

This opens a new terminal window. It gives this window an ID number so we can find it later.

** For info, don't do this yet, but, to issue commands to "screen" press ctrl+a together. This enters command mode. Then pressing "d" detaches the screen to come back to later. I'll tell you when to do this further down the guide.
Step(6) Install Storj
Most of this step from https://docs.storj.io/v1.1/docs/setting-up-storj-share-on-a-raspberry-pi

First some dependencies are needed:

                                                    wget -qO- https://raw.githubusercontent.com/creationix/nvm/v0.33.2/install.sh | bash

Close your PUTTY window with the X in the top-right corner. Open a new PUTTY session to allow the next step:

                                                    nvm install --lts

                                                   sudo apt-get install git python build-essential -y

                                                   npm install --global storjshare-daemon

Create a large swapfile:
The Raspberry Pi comes with 1GB of RAM. Whilst this is enough to run one farming node, it struggles with multiple requests. To alleviate this as best we can we can use some of the spare SD card space as an extra RAM when it's under high demand. Known as a swapfile. To set-up:

                                                        sudo nano /etc/dphys-swapfile

Brings up the screen below.
​Just change the default size of 100(MB), to 2000 as shown to the left

This is the largest the Pi will support. Effectively now having 3GB of RAM (but it's a band-aid on the problem).

save, exit, ctrl+o ctrl+x


Then, to build the new swap file for use:

                             sudo dphys-swapfile setup
​and
                             sudo dphys-swapfile swapon

This is where we differ from the main guide. I found that some folders were missing making storj unable to start due to an incorrrect path. So we make some folders:

                                                   mkdir .config
                                                   cd .config
                                                   mkdir storjshare
                                                   cd storjshare
                                                   mkdir configs
                                                   cd

                                                   storjshare create --storj=ethereum-storj-token-address --storage=/home/pi/storj
So here I have entered the command above, with my ethereum address I've created for this. 

In reply it generates a config file with some generic settings. This config file has a long randomly generated file name. Shown to the left. ending in .json

It also opens this file in a default text editior (for raspbian this is "Vim"), I find vim not very intuitive so I usually quit using           :qw          and re-open the file to edit in this next step.

On quiting Vim copy and paste this file name and path and hold it in a notepad file, we need it for 2 things later.


​As we're going to adjust this configuration I use
                                    df -h

to display how much HDD space I have available for Storj. This prototype has 104Gb available. I'll remeber that for the next step.
I then use nano to edit the configuration file we just saw. I find nano editor much more intuitive:

nano .config/storjshare/configs/37d606d4243a0d6bb876af201ef4d810d4f1637b.json​  

(of course change the file name to yours!)

​And you'll see something like this:
And contains all your adjustable settings.
​For now you are probably only concerned with.



In green is the default share size of 2Gb for the device. In my case I will change it to 104Gb. Adjust yours as you wish for what you are willing to provide the network.

Once happy with your changes use
Ctrl+O, <enter> to save
​Ctrl+X to quit the editor.
​
Next we'll start the daemon up, (nearly there!)

                                                     storjshare daemon

After a few seconds it reads back "  * starting daemon in background  "
All good.

My file permissions changes again, so to be sure:
                                               sudo chown ${USER} -R /home/${USER}/storj

And finally run the Farm! (With your config name)

                         storjshare start --config /home/pi/.config/storjshare/configs/cc63ffa1267a5d66cde5bdaddbfb5893ffb05559.json
​
To check on your farm's status use:

​                                                                  storjshare status
Seeing connections to peers is good.  A low Delta figure is also good.
It then just takes time to acquire contracts.

 **edit - This screenshot shows the Raspberry Pi Farm performing MUCH better than previous versions of StorjShare. Their latest releases and methods of data distribution are now making this Raspberry Pi solution far more feasible. I will continue to update this screenshot as I get better data with time to as accurately as I can reflect the success/failures of this project.

I hope this guide has been useful.
​Dan