Let’s have a peaceful evening
Note: Mainly for brevity, some commands / output were modified, but the essence was kept.
After noticing that a service from a server was no longer available, the journey started with a ls … which resulted in I/O errors.
$ ls test
ls: reading directory '/raid/': Input/output error
Investigating the root cause
$ cat /proc/mdstat
md0 : broken raid1 sdb[1](F) sda[0]
Yes, broken is the word you don’t want to read.
What does dmesg think about that?
...
[ 4.113971] ata1.00: 2000409264 sectors, multi 1: LBA48 NCQ (depth 32), AA
[ 4.112887] ata1.00: ATA-9: Verbatim Vi550 S3, W0704A0, max UDMA/133
...
[ 4.607618] ata2.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80)
...
Ok, so on the SATA but, only one disk says Hi.
It would be nice to fix the issue
In order to investigate further, let’s unmount the RAID:
$ sudo umount /dev/mapper/vg_raid-lv_data
umount: /raid: target is busy.
Let’s try to at least umount everything properly, stoping some services known to attempt to access the raid.
Ok still not umount-able, so lsof please help:
$ sudo lsof | grep /raid
lsof: no pwd entry for UID 999
lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing
Plenty of errors (but only 2 categories).
On this server, the Root File System is not in the RAID array (read: it can properly boot without RAID), so in despair expert mode, I rebooted the server.
$> shutdown now
LUKS on top of RAID
$ > sudo cryptsetup luksOpen /dev/md0 crypt_raid
Device /dev/md0 does not exist or access denied.
… oh, ok let’s mount manually
There are (normally 2 disks; but here was only /dev/sda listed by lsblk so let’s use this one)
$ sudo mdadm --assemble --run /dev/md0 /dev/sda
mdadm: no recogniseable superblock on /dev/sda
mdadm: /dev/sda has no superblock - assembly aborted
ok, if the mirror RAID’s broken because of 1 disk failure, it’s quite interesting right?
Hey, come on, I already saw some I/O errors in the past, this is fine, let’s do a proper power-cycle, it will work. $> shutdown now Unplugged the power supply, waited ~2 minutes. … the issue persists, it’s not a Microslop product (not that they work after a power cycle either…)
Spoiler alert: this power-cycle is a big mistake for the post-mortem investigation.
$ lsblk -o NAME,VENDOR,MODEL
sda ATA Verbatim Vi550 S3
sdb ATA SD Ultra 3D 1TB
└─md0
$ sudo mdadm --examine /dev/sda
mdadm: No md superblock detected on /dev/sda.
ok, no /dev/sda
$ sudo mdadm --examine /dev/sdb
/dev/sdb:
...
Raid level : raid1
...
Data Offset : 264192 sectors
...
ok, so I started to be lost, but thought “hey, who care about RAID, we just need data, so let’s skip the header”. After getting mad at the offset needed by losetup, browsing the internet brought the guess: 512 bytes per sectors (of course). (I genuinely do not know where to get this information from actually…). So 265_192 * 512 = 135_266_304
$ > sudo losetup --find --show --read-only --offset 135266304 /dev/sda
/dev/loop0
$ sudo cryptsetup luksOpen /dev/loop0 crypt_raid
Oh, it works! And took only ~1.5 hours so far…
$ sudo mount /dev/mapper/crypt_raid /raid
mount: /raid: unknown filesystem type 'LVM2_member'.
… but I forgot about LVM
LVM on top of LUKS, on top of RAID.
$ sudo fsck /dev/mapper/vg_raid-lv_data
fsck.ext4: Input/output error while recovering journal of /dev/mapper/vg_raid-lv_data
fsck.ext4: unable to set superblock flags on /dev/mapper/vg_raid-lv_data
Noooo… oh wait, the losetup mount was done in read only, and as a bonus for the good news, I’ll spare you the vgchange -an / luksClose / re-losetup / re-luksOpen
and the mighty and dreaded fsck said: “clean”. Mhhhnn’k
# mount /dev/mapper/vg_raid-lv_data /raid
$ ls /raid
-- Works, just works :D
So now, as you can guess after the hassle it’s bed BACKUP time! I do make backups, but… they stay on this very RAID, I just scp / test them from time to time, … manually (I know…).
Just hotswap the disk, trivial
So, no data loss, everything’s cool, was an adventure through the evening, nice, everything’s ends well! Right? … Just a simple question: which disk is the culprit? I mean /dev/sda or sata1 are not that explicit to me. Why does a disk not report to the SATA bus… and the other one has a broken header?
I left some hints here and there which you may have noticed: The RAID is composed of 2 disks.
- One from Verbatim
- One from SanDisk (“SD” as VENDOR in
lsblkoutput)
Please note that I’m not sponsored nor want to bash a manufacturer, it’s just easier to name a disk by its manufacturer than UUID for instance.
Confusing output
Initial boot: md0 : broken raid1 sdb[1](F) sda[0]
$ lsblk -o NAME,VENDOR,MODEL /dev/sdb
NAME VENDOR MODEL
sdb ATA SD Ultra 3D 1TB
└─md0
└─crypt_raid
└─vg_raid-lv_data
The SanDisk is the culprit, easy! And, after the first power cycle (shutdown -r now), only the Verbatim is on the SATA bus (mapped as /dev/sda)
$ > dmesg
[ 4.112887] ata1.00: ATA-9: Verbatim Vi550 S3, W0704A0, max UDMA/133
...
[ 4.183576] sd 0:0:0:0: [sda] Attached SCSI disk
[ 4.602340] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[ 4.607618] ata2.00: failed to IDENTIFY (INIT_DEV_PARAMS failed, err_mask=0x80)
And now comes the second power cycle (shutdown + no power supply for about two minutes) into play.
$ > cat /proc/mdstat
Personalities : [raid0] [raid1] [raid4] [raid5] [raid6] [raid10]
md0 : inactive sdb[1](S)
976630488 blocks super 1.2
$ > sudo mdadm --examine /dev/sda
mdadm: No md superblock detected on /dev/sda.
$ > lsblk -o NAME,VENDOR,MODEL
sda ATA Verbatim Vi550 S3
sdb ATA SD Ultra 3D 1TB
└─md0
It confirms everything: the SanDisk is dead as /de… oh no wait this time, the Verbatim is not in the RAID array, but the SanDisk is.
A day after the party, re-reading the logs confirmed what I first thought: I don’t understand what’s going on.
- Why is one disk not reporting on the bus?
- Why does one disk have sudden Software issues (header broken) at the same time?
- Why does this seem random?
- Could it be that the issue lays one level deeper (motherboard)?
Could it be thatI understand nothing
And some side quests pops up:
- Should I switch to HDDs? (these two SSDs are about 3 years old, nearly never under heavy load)
- Is ext4 a good idea?
- Should a disk be “mostly write”?
As you can see, this is still under investigation, but I wanted to re-read logs carefully, avoid guessing on what I did and now, I have a documentation of what happened / was done.
So, see you hopefully in a later blog post with answers.