I was so tired after all that maintainencing I didn't even get to explain everything that happened... went right to bed lol
So I got a notification that a disk was down. When I got to it, I found that 3 disks were dying but the other 2 disks weren't saying anything about it, just doing transfers at 2-4MB/s
Needless to say on a raidz2 this is bad. I backed everything up then began maintenance.
For reference all of these arrays are ZFS:
Original: 8x1tb drives
Backup: 10x3tb drives
Current: 10x mix of 1.5/2tb drives
Since I had a mix of 1.5/2/3TB drives and 3x 1tb drives was going to cost $150, I decided to upgrade the whole array while I was at it.
My first plan was to make my backup (which was backed up to 10x 3TB drives) the new array. I gave it all the same structure as the old array and put it in.
No dice, the onboard LSI controller doesn't like the 3TB drives. I should have tested that first...
So for 1.5-2tb drives, well I had 6x 1.5tb drives and 2x 2tb drives. I decided to make the array with this and 2 new 2tb drives (so I could upgrade the array over time), well this array worked. So I started copying back.
Well I found out that of all the things Kitsunet runs, mastodon is the most expansive file-wise. I think that about 2.2 million files on the server are just mastodon. Needless to say, syncing this over the network was taking ages...
So I switched to doing it over USB 3.0, brought the whole array (8 disks) up, due to the bad drives in the array it was still going slow but at least it was gonna finish this week lol.
One of the drives in my backup seemed to fail but it turned out that it was really a broken SATA controller card which thankfully are cheap as dirt to replace.
When the copy finished, it ended up in /z/opt and /z/home, which weren't actually the right filesystems since those were mounted at /opt and /home, and I ended up having to move those files into place too...
An instance for furries, therians and otherkin. News and info site https://soc.kitsunet.net/