C-Fu's LYN guide for NAS & Homelab solutions

23/4/2018: First release
27/4/2018: Added Download and installation steps
3/5/2018: Added hardware considerations, and sixth step (almost final step)
28/11/2018: Updated to use simpler, easier tools, and newer DSM 6.2 version. Change name from NAS4Free > XigmaNAS (same thing)
29/10/2019: Added Table of Contents
21/07/2020: Added google drive integration via rclone


Table of Contents
Intro & Guide: this post
Docker & VM
LSI card for extra sata ports

I've been building and selling NAS solutions for many years now, exploring most of the options available in the market and DIY. Since my 60TB RAID is slooooowly building, verifying, parity checking (whatever they call them), I decide to make this guide. I've now come to the conclusion that DIY is the best.

But seems to me that not many people here have a clue, especially when it comes to deciding which DIY NAS solutions that you like. So here's what I know. A different kinda guide than what you can google, I think.

Guide level: Beginner and intermediate. This guide isn't for lazy people. You still need to google and read up. And if what I write here is not true because you're a pro, you're absolutely right. I wrote it in such a way that's easy for noobs to understand, that's all. Feel free to argue, I don't mind

Why go with NAS instead of traditional, typical shared storage (external HDD, windows share)?
- Centralised storage access, with/without cloud (anywhere) access
- Reliability of your data, because honestly, my data is much more expensive than the price of harddisks
- Low power, low cost
- More features than just "share files" (more on this in a bit, and the best thing about NAS in 2018)
- Flexibility in choosing the method (old servers, old pcs, old harddisks)
- Fun to explore and try out. You might just realize that NAS features are the exact thing that you need for a long, long, time but never knew existed for cheap (or free).

DIY, or ready-made solutions?
- ready-made for peace of mind, access via USB (DIY method uses ethernet/wlan only) but really lacking in features or hardware -> buy Synology, QNAP, other RAID-in-a-box devices like Seagate NAS (please, please don't)
- DIY for learning, experimenting, and because you're kedekut and wanna use existing stuff lying around, or even because you got cash to spare and wanna get the best bang for your buck and peace of mind.
- Various ways of RAID-ing your drives - RAIDz2,RAID 5/6/10,SHR,DriveBender,DrivePool,Linux MDADM,etc
- Can do more than just share folders - host servers, become media centers, download stations, automatic torrent downloaders, run multiple OSes at the same time, etc.

DIY NAS features that you don't know you want:
- Personal Google Drive
- Virtualization - also run a bunch of OS and appliance on the same PC. Think a PC with 8 Core CPU & 16GB RAM can do
2C+4GB: NAS OS
2C+4GB: Windows 10
1C+1GB: Webserver/Plex media center manager/Firewall OS/Point-of-sale via browser/Automatic Torrent downloader/Odoo ERP and CRM/Owncloud/Another NAS/other appliances
2C+4GB: Ubuntu/Raspbian/Linux Mint
1C+2GB: Android

All at the same time, accessed via the browser (think Windows 10 from browser), via VNC (TeamViewer/RealVNC) or via GPU's HDMI/VGA out (1 GPU outputs Windows 10, another GPU outputs Linux Mint).
- Containers - Run prebuilt,preinstalled apps ready to go without having to setup and configure shit, especially if you know next to nothing about Linux. Want your own Visio via browser? download draw.io container. Want a collaborative-based Office via browser? download Feng Office container (think 2 people opening the same file on the NAS, and both edits are updated in real time with each other). Want Odoo but dowan to reserve 1C+1GB of your resource? Can also.
- Need to do automatic backup to your preferred choice without nas-in-a-box company limitation, like Thecus with their Microsoft-centric solution.
- Want NAS and other OS in a virtualized (esxi) environment - think esxi as the host OS, and from there you deploy 2C+4GB+HDD abcd for NAS, 2C+4GB+1TB of HDD e for Windows 10 #1, 2C+4GB+1TB of HDD e for Windows 10 #2, etc.
- TONS of things that you can do!

----------Steps!-------------

FIRST STEP - storage
See the storage that you have - a bunch of old harddisks, or buying new mechanical hdds, or if you're rich, got yourselves a bunch of 500GB SSDs for storage :x

Old hdds will die. My experience, if you have old seagates, don't bother and get new hdds. YMMV.

Some solutions and methods demand same hdds only. some can mix. some have compatibility issues with certain hdd models and serials and date. So decide on what you have and want to use for this project. Benefit with Xpenology is, you can (sort of) mix and match harddisks with various capacities into one RAID, so if you have a bunch of harddisks lying around, only go with Xpenology (note that you need to add the smallest harddisks first into your SHR Raid, then add bigger and bigger capacity HDDs after that. Limitation is once older smaller hdd dah kong, can only replace with the biggest hdd in that RAID).

SECOND STEP - hardware
The hardware that you have/want to buy. In all intents and purposes of this guide, minimum requirements for a NAS (but not exactly bad performance!) is LGA775-grade, 4GB DDR2 RAM, Core2Duo PCs. Perfectly fine to run Windows 10 for basic usage on this ancient junk. Can get one whole PC for less than RM500. Most ready-made NAS boxes are using cheap old pc-grade stuff in them.

But if you want to get more benefit out of your NAS, then you need a PC/processor/motherboard with these features built-in: VT-d, HyperThreading, AES-NI (to run a bunch of windows 7/10), AVX (for video transcoding on the fly) 4 RAM slots, 4 or more SATA ports, PCIe 1x (for extra gigabit lan card, or USB3 card) and 16x slot (for GPU and/or additional SATA cards for more storage option), PCI slots (for SATA cards) and so on.

Also don't forget! Since your NAS uses gigabit ethernet, you would also need Cat5E/Cat6 cable AND a gigabit switch/router for best file transfer performance (around 60MB/s to 120MB/s file transfer speed).

Expansions for your old board:
PCIe slots can give you
- [PCI-e x1, x4] Extra gigabit ethernet cards (around RM20-100 each). Most NAS OS allows you to group a few ethernet ports into one, either for load-balancing (if one port transfer speed is full, the other card can take over), bonding (combine all cards' transfer speeds for double/triple the transfer rate) or failover (if one card dies, the other takes over). Personally I just load-balance the ports, better than failover. Bonding requires all of the cards to have bonding feature. Can also add USB 3.0 card just for some fast USB (RM30 at lowyat).
- [PCI-e x4] RAID/sata cards, 4/8 ports are pretty cheap, from RM20-RM250. LSI SAS3081E-R is a very famous name for you to find. better get the splitter cables as well. I got 2 of these in one NAS, giving me space for 16 hdd in addition to 4 hdd from the mobo (and 1 IDE hdd).
- [PCI-e x8] Some RAID/sata cards need x8. Some 10 gigabit cards need x4 or x8.
- [PCI-e x16] if your mobo has 1 x1 & 1 x16 PCI-e slots, use x16 la. Can also be used for (old) GPU, which you can use/reserve for VM use. For example: I use an old nvidia Quadro 512MB for 2 Win10 VMs, 256MB each in NAS4Free, so 1 machine runs a NAS & 2 OS at the same time (the NAS has no use for GPU). I got one very old Win98-grade PC at the office that on startup I open chrome fullscreen to one Win10 VM url.
[PCI] Pretty old, but I bought a PCI 4 port sata raid card that works well for this. But some older mobos have compatibility issues with this card. It's cheap though (RM45).

THIRD STEP - NAS OS/Filesystem
Choosing the filesystem will determine the NAS solution that you can choose. But one thing for sure, you HAVE to use RAID to take full functionality.
hardwareRAID - must have hardware/chip for the RAID to function. Hardware rosak, then hopefully all you have to do is to replace the RAID SATA card, or worse, the motherboard.
softwareRAID - the NAS OS (and your processor, and maybe RAM) will handle the RAID. Hardware rosak, just reinstall the NAS OS and import your RAID array.

Windows 10|NTFS is "free" in a sense that you don't need a serial, and don't need to activate. Just will have a watermark at the bottom right of your screen. Since NAS boxes are most of the time have no monitor attached to it, it doesn't matter la.
Pros: Familiar, easy to setup (initially). You're probably are running Win10 already. Can install Hyper-V for free (see Virtualization), easiest way to set up compared to esxi, proxmox, etc
Cons: Mafan, difficult to extend features other than having Windows fileshare, and have to pay for good things
Things to have/buy/search: DriveBender/DrivePool for modern-style softwareRAID

NAS4FreeXigmaNAS/FreeNAS/FreeBSD|ZFS is the enterprise-grade filesystem. Rock solid like crazy. FreeNAS needs more resource than XigmaNAS, but both are similar in terms of featureset.
Pros: https://www.youtube.com/watch?v=naKd9nARAes. Got VirtualBox built-in via plugin (see Virtualization). Easy to install
Cons: A bit of a learning curve, but not too difficult. Less flexible, and need to plan way ahead. The HDDs must be the same. For every few TB of storage you have, need to have 1GB of ECC RAM (if possible). So if you have 100TB of storage, you'd need about 128GB of ECC RAM. Although, for small-medium companies I've deployed NAS4Free with a few TB of storage with less-than-recommended RAM with no issue.

Xpenology/OpenMediaVault/UnRAID|btrfs/ext4 - Best of the best in terms of overall ease and functionality. I personally choose Xpenology over any other for home/sme use. Basically Synology uses open source codes for their Synology DSM, so people took the opensos code and made it run on consumer PCs. UnRAID is not free, but is a very close second. If you want to use it for playing games (GPU Passthrough), then UnRAID is your only sensible choice - think 2Core+2GB RAM for UnRAID, 6C+14GB+1GPU for Win10.

Xpenology
Pros: Most functionality built-in via packages - Plex for media, Office for collaboration, a bunch of ways to backup, google drive integration, super easy to setup (just use the DS3615xs package if you're unsure)
Cons: Very difficult to setup, need hours of reading, understanding and doing things that are not normally what you want to know in computing, like your USB PID & VID. But once you understood did it, it's actually not difficult at all. No GPU passthrough, so if you want to setup a virtual Windows 10 machine with Steam, you cannot make use of your GPU.

OpenMediaVault
Pros: Similar to xpenology, but less user-friendly, and less package. Much easier to install.
Cons: Not as feature-packed as Xpenology

UnRaid
Pros: Gaming-friendly, GPU passthrough, quite easy to setup, runs off USB (like most of the NAS OS above)
Cons: Not free

GPU passthrough example:
8 Core CPU, 16 GB RAM, 2 GPU, 2 monitor, 2 keyboard, 2 mouse can be splitted like this:
NAS OS: 2 Cores, 2GB RAM, access via browser
Windows 10 #1: 3 Cores, 7GB RAM, connect to 1 LCD with GPU #1, use keyboard #1 & mouse #1
Windows 10 #2: 3 Cores, 7GB RAM, connect to 1 LCD with GPU (or browser, kinda like TeamViewer or VNC) #2, use keyboard #2 & mouse #2

FOURTH STEP - Downloading necessary software & extra hardwares
Below are the common generic (free) softwares that you will need for any/all of the NAS OS above. Most NAS OS rely on installing the OS onto your USB, so you can free up sata ports for your data. You don't need a big USB drive, most can do with 8GB or even 4GB USB if you have that lying around as most OSes will be in your RAM.
ALL IN ONE TOOL FOR XPENOLOGY: https://mega.nz/#F!BtViHIJA!uNXJtEtXIWR0LNYUEpBuiA
1. Win32DiskImager - this is to flash (copy) the .img files directly into your USB drive
2. 7-Zip, download the 64bit x64 version
3. Rufus or Etcher.io, this is to flash .iso images into your USB drive
4. OSFMount, if there's a need to edit the .img file, usually to set boot options for certain NAS OSes
5. A browser, to view the dashboard/desktop of the NAS. Everything is headless, so no need for a monitor, just a spare pc connected to your home network via ethernet (no installation via wifi for now)
6. Ethernet (mobo built-in or ethernet card) & a monitor just for the initial view of the installation, and to find out the IP of the OS so we can login.
7. Putty & WinSCP - this is for post-installation, so not important. Putty is to enter the NAS OS punya terminal/command line, and WinSCP to view the files on your drive for some special purpose (normal use, can just browse via My Computer>Network)
That's it!

Download links

Name	Xpenology	NAS4Free	OpenMediaVault
Requirements	LGA775	ECC RAM/Servers preferred (not necessary), DDR2 min.	Any common PC minimum
Current Version	ALL IN ONE TOOL FOR XPENOLOGY	11.2.0.4, get liveUSB .img.gz file or liveCD .iso file	4.1.3, get the amd64/x64 file and flash to USB via Etcher
Notes	See FIFTH STEP	Extract, flash .img using Win32DiskImager	Flash .iso using etcher/rufus

FIFTH STEP - Installing and booting your NAS OS
When you boot your pc, you need to set your BIOS to boot from the USB (Changer the boot order).

XigmaNAS & OMV : This is a live image, meaning everything is ready to run WITHOUT INSTALLING. So you can check out the web interface and all, and as long as you don't change any setting related to the hdd in your machine, nothing will be saved and everything will be wiped out after reboot.

Xpenology setup:

Download the all-in-one tool and run it. if Windows SmartScreen says can't run, click on info, and Run Anyway (it's safe). Refer this image for each steps.



STEP 1: Click DOWNLOADS. Choose DS3617XS Loader (.zip file/synoboot.img file) and DSM 6.2 (.pat file) for the same version.
Note: If you are using 4th gen processor, like i7-4790 then it's best to use DS916+ as it supports hardware transcoding for video... if you need it. works with apps like Emby and Plex (Plex needs Plex Pass, about 110USD)


STEP 2. While downloading, use USB IMAGE TOOL to find your USB Drive's VID and PID. Xpenology's bootloader pelik sikit, it checks your USB info to successfully boot from it.


STEP 3: Unzip the loader zip file that has synoboot.img. Open OSFMount, find the synoboot.img and open partition 0 only (not entire image). We need to open usbdriveletter:\grub\grub.cfg file using notepad++.


STEP 4: Generate a new serial for your DS3617XS (or whatever model you choose to use). Not really needed, but since it's there why not.


STEP 5: Open usbdriveletter:\grub\grub.cfg file using notepad++ and change these three values. OPTIONAL you can change set timeout='1' to set timeout='5' if it's too fast for you during booting. Save in notepad++.

STEP 6: In the main OSFMOUNT window, click DISMOUNT ALL & EXIT.

STEP 7: Use USB IMAGE TOOL > Restore button and select your synoboot.img file. Your USB Drive needs to be formatted as FAT32, and there's no point in using a USB Drive more than 4GB so get the cheapest possible.

STEP 8: Boot your PC into the USB drive. Wait few minutes while your system boots (no need a monitor after the first time). Use SYNOLOGY ASSISTANT tool or http://find.synology.com to scan and find the IP of your machine. Follow instructions, pick the .pat file you've downloaded in STEP 1. DO NOT CHOOSE TO UPDATE AUTOMATICALLY. DO NOT USE QUICKCONNECT. Remember the IP, and you're done!
-----------------

Hopefully with these infos you can get started. This is not meant as a thorough guide, but merely an introduction to get you up and runningresearching. Feedbacks are welcome, and I might expand this later. Donations of old pc hardwares like CPU, RAM, are very much welcome

This post has been edited by C-Fu: Jul 21 2020, 12:15 AM

SIXTH STEP: The fun begins

You can find these in Xpenology's app store and various docker images.
Prerequisites for you to know:
Port forward from your router



Media Centers & Managers - catalog, stream your media locally or remotely, automatic transcoding (convert to whatever video format your device/phone/tv can accept automagically while playing, very CPU intensive)
Plex - Most SMART TVs can download Plex from the TV's app store
This is mine

Emby - Similar to Plex, but can use GPU for transcoding
This is mine


Documents & Office
Office - Very similar to MS Office, runs in your browser. If you convert into it's own format, all opened browsers of the same document will get live editing, best for collaboration
Draw.io - Visio-like diagramming and technical drawing software


Book/Comic/Manga reader
Ubooquity - mostly for comics and manga

Calibre/Calibre Web - for your ebooks, magazines, etc

OS
Ubuntu - since Xpenology is linux based, we can also use docker version of ubuntu. Access the OS via browser.



DOCKER vs VM
A very basic explanation is this:
with Docker, you share your PC's resources (like CPU, RAM, etc) with both the NAS and the appliances, mostly web-based. NAS and appliances share the same underlying OS (Xpenology uses Linux, so your applications must be Linux jugak).
with VM, you partition a subset of your PC's resources dedicated for that machine, kinda like a (few) smaller PCs inside a big PC (your NAS). So your NAS can be Xpenology or XigmaNAS, then use 2 Cores + 8GB RAM for a Windows 10 VM, usually accessible using TightVNC or web browser as the "monitor/display". The choice is yours! My reasoning for using Docker is simply because (one of) my Xpenology PC only has 12GB of RAM, and only 4 Cores so not enough to dedicate for a few VMs. Most of these specialized/dedicated OS (like webserver/wordpress OS) are also available as Docker, and in my use-case, most of my web-based apps are used by less than 50/100 people so no need to dedicate resources for multiple VMs.

A curated list of famous, good applications:

https://github.com/Kickball/awesome-selfhosted
Selfhosting is the process of locally hosting and managing applications instead of renting from SaaS providers.
This is a list of Free Software network services and web applications which can be hosted locally. Non-Free software is listed on the Non-Free page.

https://bitnami.com/stacks
Bitnami is the leader in application packaging providing the largest catalog of click to deploy applications and development stacks. Quickly and easily launch your favorites on your own servers or deploy to every major cloud environment. Choose from local installers, single VMs, multi-tier VMs, container images or Kubernetes Helm charts.

EXAMPLE: WORDPRESS APPLIANCE
VM Version: https://bitnami.com/stack/wordpress/virtual-machine
Docker Version: https://hub.docker.com/r/bitnami/wordpress/


DOCKER EXAMPLE
Basically, Docker is pretty cool, services/softwares are prebuilt, preinstalled with all the necessary configs and tools in an image. You download the image, create a container with that image, and access the image via ports (like http://yournasIP:portnumber). No need to install, configure this that and what you traditionally do before running the software. For example, if you wanna run a wordpress webserver, first you need to install the OS, then install the webserver, configure whatever, then install wordpress. With docker, download the wordpress image, and launch. That's it!

With xpenology, the GUI is pretty powerful, can do 99% of the time from the GUI itself. I'll show an example by installing draw.io.
1. From the package center, install Docker
2. Run docker, go to Registry, and search for draw.io. get the one with the most stars (the top one). If you want more info, click on the link next to the image name. Choose the latest version. We will need to consult the link for additional installation steps (if needed).

3. If you go to Image, you can see the download progress. Once it finishes, click Launch. Give it a name. Click Advanced Settings.
4. Go to Port Settings. From the image's docker hub page, we see that it uses port 8080 and 8443 (80 for http, 443 for https). You can change the Local Port to whatever you want/need. In my case, I already use 8080 for some other image, so I change it to 11080 and 11443.

5. Go to Advanced Settings and create a shortcut on desktop. point it to your NAS IP with the http port.

6. You're done! Turn on via Docker>Container or Xpenology Desktop icon.

This post has been edited by C-Fu: Nov 29 2018, 12:41 AM

LIKE

continue broo..i'm always interested to build my own NAS. Got some LGA775 board with Xeon proc lying around

nice post...
Reading and try to understanding...
sorry, uncle old liao.. a bit slow.
but want to build my own nas.

park for references. nice info.

added installation steps. Most NAS OS are quite straightforward - download the img/iso file, flash/burn to USB, boot from the USB, choose install, follow instruction. With the exception of Xpenology la, but a more straightforward way will be released soon.

Note that while we're using old pc, I would personally recommend to max out the RAM and the proc if possible. one LGA775 Core2Duo Acer machine that I got, I use 8GB DDR2 RAM (RM60 for 4x2GB RAM sticks).

ako pon uncle took me 1+ hour to do & understand the whole xpenology setup from the tutorial link hahaha the first time. 2nd time, 5 mins. Good luck

This post has been edited by C-Fu: Apr 27 2018, 02:30 PM

So it that meant, the PC running 24hour?

it's up to you really.

it's like any/most cloud-based service like google, with the exception that you control everything - from your data that you put there, to the services that you have.

you want it to auto turn on at 8am and shutdown at 12am, up to you.

you want it to turn on whenever you want (via WOL), up to you.

you wanna store movies, download movies, up to you.

you wanna have an online microsoft office, up to you.

wanna have visio-like software but web-based, up to you.

wanna have a comic book reader, up to you.

wanna have a windows 10/osx machine on the nas, but access via web browser, up to you.

minute 3:20

as I'm going to focus on xpenology more, in the next update i'll walk through some of the features I think most of you will really like.

This post has been edited by C-Fu: May 1 2018, 12:47 AM

excellent write up and knowledge.
kudos for sharing!
thanks

So the power consumption more or less like desktop pc?

I want everything! Currently got IIS, PHP, MySQL, MSSQL, Python, Activeperl, Windows File Sharing.


If you use a low power CPU like Pentium G4560, power consumption won't be high. It depends on amount of disks you have. Power supply also plays a role too.

This post has been edited by TristanX: May 3 2018, 10:19 AM

correct.

most of the time, NAS feature sets depend more on RAM than CPU speed. CPU matters during times like indexing your files and running VM OS, like you assign few CPU cores dedicated to that VM.

your power consumption consists more on the drives IMO.... which is very little. and disable spindown of your hdds, as the power saving that you get is very very little compared to the wear & tear of the drives from spinning up and down repeatedly.

can (with the exception of IIS since it's microsoft's)! Few ways to do it from Xpenology:

1. Install from the app store
2. Docker (easiest, fastest, most flexible)

I'll add some docker intros. edit: DONE!



This post has been edited by C-Fu: May 3 2018, 02:47 PM

I have tried both XPenology and OpenMediaVault.
I like XPenology very much.
But in the end I go for OMV.
Something about XPenology, I can't see how long it will last.
There are also cases where synology update may also stall your server.
I recalled the dev (jun loader) has gone missing, some hardwares do not work with it especially AMD.
With docker, OMV has so much to offer.
But I agree that OMV UI is less user friendly especially for first time user.

You can use anything you want, that's the beauty. Most of the additional features and services are the same across most NAS OSes, like Windows Share, Plex, Docker, etc.
I haven't tried any AMD machines before, but from what I've read AMD and xpenology works quite flawlessly.

The only main major advantage of Xpenology is SHR, which you can just dump any hdd of mixed sizes and it'll just work. Other than that, everything is pretty much the same.

Add anything you know to the guide, I'll add it to the post.

This post has been edited by C-Fu: May 3 2018, 05:38 PM

hi nice guide
i am finding casing on it
any recommend casing?

Depends on type of casing you want.
I cannot find nas case locally.
I bought mine at taobao.
Example:
https://item.taobao.com/item.htm?spm=a230r....bucket=9#detail
https://item.taobao.com/item.htm?spm=2013.1...14-e0f0342c09b6
https://item.taobao.com/item.htm?spm=2013.1...14-e0f0342c09b6
https://item.taobao.com/item.htm?spm=2013.1...14-e0f0342c09b6

Nice write up

I see you using both Plex and Emby. Is it purely for example purpose or you are using both of them?

This post has been edited by lol4291: Jun 2 2018, 12:38 AM

i do use both. I'm still evaluating the best ones for my needs, and here's what I've found:

plex
- very fast, responsive UI
- most smart tvs got plex app in their own app store, for free
- android app have to buy, but I bought one years ago for RM1 when got sale
- plex won't correctly find some movies and tv shows, and difficult to make sure plex correctly catalogues the media
- some movies cannot be converted, but probably because of my old processor lacking in certain instructions

emby
- not so fast UI
- a lot more plugins
- only some tvs have emby app
- easier to match media, like i could go to anidb, search for the specific anime, copy the ID in url, and point the media to that url
- got CPU AND GPU transcoding, and more transcoding options available

I read somewhere the tech guy say make sure have hardware raid controller, if just software raid it is too unreliable. How true is that?