Snap 4100 FSCK fatal error = 27

[rpmurray]

Well, just got a new RAID5 to replace the 4100 and brought it in today so I could start the backup. The 4100 looked OK (all the correct LEDs flashing on the front panel) but noticed that there was apparently a power failure over the holidays that was long enough that the UPS wasn't able to keep things running. Checked the 4100 using the browser and the Disk Status tells me there was a fatal error during disk check after it restarted.

E File System Check : FSCK fatal error = 27 Disk 60000 12/24/2009 6:37:16 PM
E File System Check : Partially allocated inode I=28329230 Disk 60000 12/24/2009 6:37:16 PM

Just wonderful, and how ironic. It occurred just a couple of hours after I left work for the holidays.

Checked the forum but didn't see anything I didn't already know.

The hard drives are all fine, no amber light for any of them on the front panel, and the 4100 still sees them as a RAID5 (no orphans). Not exactly sure what to do here. Most of the device was backed up, but there is some work on it that was not.

So I've restarted it (from the browser) in the hopes it does better this time. It took about 40 minutes to get to the error last time so I'll wait and report back. And I just realized that I didn't notice if the System LED was double-flashing or not before the restart, but it is now. I can also see all the disks getting accessed (rapidly flashing green LEDs) with none of them changing to amber.

My guess is I'll get the same error. I figure I'll try running the Disk Check manually next if it still reports the error.

Any ideas on how to bypass the error long enough to recover some files?

[rpmurray]

Well, got the same error, so it looks like I might need to run Disk Check with Repair all errors.

But, before I do that, I noticed in the forum that someone was able to get their snap back up by increasing the memory, so I'll give that a shot because it looks like this snap has only 64MB. I have a couple of other snaps that are no longer in use that have 256MB, so I'll borrow from them. They're off site, because I've been replacing them one at a time and this was my last one, so I'll give it a shot again tomorrow once I have the memory.

If anyone has any advice in the meantime, I'd appreciate it.

[blue68f100]

If your using APC you can auto shut down the 4100 during power outages. Not all APC units support the network cards. I have APC 1000 and 1500 SmartUPS with network card in the 1500. Mine are set to shut down on low battery power.

[rpmurray]

My 4100 is plugged into an APC 2200XL, but with all the other stuff plugged into it, only gets about 40 minutes before the UPS gives out. I haven't been able to figure out how to get the UPS to send the shutdown to the 4100. Most cases I wouldn't even need to worry, since we have a great big generator hooked up to the server room that should kick in within 20 seconds and keep things running for days, but this time it failed to kick in.

Anyway, I've swapped out the memory with a 256MB Kingston stick that I had in one of the other decommissioned 4100s and have started it up again just a moment ago. My guess is that it will fail in the same spot. I've checked the settings and it's set to automatically repair errors on startup if there are any suspected problems. I don't know if that means it does the Repair Most Errors on the Check or Repair Disk page, or if it tries the Repair All Errors (the one with the warning about data loss).

Assuming I get another failure, I'll try the Repair Most Errors first and then the Repair All Errors if that doesn't work.

[rpmurray]

Failed in the same spot. I'm now running it with Repair Most Errors, but looking at the log after I started it, it looks like this is what it was set for after a startup. So it will presumably fail in the same spot.

[rpmurray]

Repair Most Errors dies in the same spot. Trying to decide if I should run Repair All Errors and risk data loss, or see if there's another way around the problem.

[rpmurray]

Couldn't find anything that might let me get around the inode error, so decided to run the Repair All Errors.

While checking on the web, I came across some articles dealing with this kind of problem on other platforms, and they basically indicate that the only corrective action fsck can take is to clear the inode. I hope that fsck can get past the error by doing that, and that it doesn't wipe out the data I'm trying to recover (one folder with about 2 gigs of data, most of it backed up, but a few subfolders that have some stuff I'd prefer not to have to recreate). The snap wasn't doing anything at the time of the power failure, and there were only a few documents still open when the power went off.

Maybe I'll get lucky.

[rpmurray]

Repair All Errors found 21 partially allocated inodes (they were all grouped in the same area from 28329230 to 28329252) in Phase 1 - Check Blocks and Sizes, and 5 files with various inode related errors in Phase 2 - Check Pathnames (if the paths it displayed are correct, nothing I don't already have backed up). Phase 4 - Check Reference Counts had only 2 errors. It fixed all the errors and passed through the rest without problems and is now in the middle of re-syncing the raid.

At this point I'm able to access the snap and am copying off the one folder that I didn't have a backup of.

As soon as my new raid gets through with it's disk scans (just to make sure there are no problems on the new disks, only two and a half days to go) I'll copy everything over and then decommission the 4100.

I really wish the folks at GRC would bring out a new version of Spinrite that works natively with SATA drives. I couldn't use it with the new 2TB drives in the new raid because it was going to take over 180 hours each for the four drives and I didn't want to wait a month for it to finish. So I'm using the disk scan built into my drive formatting software, which will only take two and a half days. It's better than nothing.

[rpmurray]

Same unit apparently had a panic early in the AM. I needed to restart it just to be able to sign onto it. Looks like it paniced while backing up the extended rights with a inode error and the same "FSCK fatal error = 27" error message. I had luck the last time with having it do a Repair All Errors, so I tried it again. It just runs for a while and panics again, and won't let me sign on. If I restart and try to sign on it says "No Network Shares" and keeps asking for the UserID/Password when I try to go to the Administration pages, so I'm stuck at this point.

I figure it's finally gone this time, but if anyone has any ideas on what to try next, I'm all ears. I think I'll give Spinrite a try on the drives if nobody has any suggestions, but am not confident.

Edit: Yes, I realize that last time I said I'd be decommissioning this 4100, but I changed my mind, figuring I'd try to keep it going until it finally failed just because it was kind of handy to have around.

Edit2: Restarted it a couple of times. It always get to a point where Disk 1 lights solid and then it goes into panic with the System LED blinking like mad. Another thought I had was maybe pulling Disk 1 and trying to get it to come up in degraded mode. I'll probably try this last because if it hoses up there, then it really is done.

[Phoenix32]

Originally Posted by rpmurray

I figure it's finally gone this time, but if anyone has any ideas on what to try next, I'm all ears.

Originally Posted by rpmurray

I came. I saw. I didn't see anything of import. I left.

REF: Lack of LBA 48bit Support

Originally Posted by rpmurray

Sorry, I moved on from anything smaller than 500GB years ago.

Additional:

I came. I saw. I didn't see anything of import. I quoted. I left.

Sorry, I moved on from anything less than Gbit ethernet years ago.

Sorry, I moved on from anything less than Snap 4400 years ago.

Sorry, I moved on from anything less than zero value years ago.

[blue68f100]

Becareful removing a HD. If you remove a good one with one already failed your raid/data is lost.

[rpmurray]

Originally Posted by Phoenix32

I came. I saw. I didn't see anything of import. I quoted. I left.

I'm being haunted by my own quotes.

But...But...But with four 160GB drives it counts as a 640GB unit.

[rpmurray]

Originally Posted by blue68f100

Becareful removing a HD. If you remove a good one with one already failed your raid/data is lost.

None of the drives is actually showing the amber failure light. Unfortunately I'm unable to get it to the point where I can check the logs, although before I tried the Repair all Errors the log was just showing a problem with an inode.

I was just wondering if the panic I get when the Disk 1 LED goes solid might be indicating bad sectors on Drive 1. If that's the case, then it might come up in degraded mode if I pull the drive. But that's a longshot and will be the last thing I try. First I need to pry the drives from the unit and have SpinRite check them over.

[blue68f100]

Run SpinRite and see what it finds, it should repair inode errors. It will also tell you if you have a bad HD so it will eliminate the guess work.

[rpmurray]

The latest news.

After labeling all four drives I've run SpinRite on all of them. Each was given a clean bill of health with all sectors OK. The SMART display gets kind of scary because it shows these large numbers for ecc correctable and seek errors, but checking on the SpinRite discussion forum it looks like these are expected on Seagate drives.

The drives are Barracuda 7200.9 ST3160812A (160GB) and Barracuda 7200.7 ST3160023A (160GB), 2 of each. They've been running fine for the last couple of years so I don't think my problem is related to having the two different mechanisms.

I have replacements on the way so I can make sector clones of these and then I'll see what else I can try to resurrect the 4100 from the dead. The odd thing, as I mentioned before, is that after starting the Repair All Errors and getting the panic about ten minutes into the repair, every time I've tried to start up the 4100 since then and access the web config pages has been meet with a "No Network Shares" and asking for the username/password over and over again. I can't get to the debug page so I can do an "info dev" or check the logs.

So, these are the ideas I have.

1) Maybe the battery is dead, and I need to replace it (this I'll be doing anyway). I don't know if that's enough to keep me from being able to access the unit.
2) Maybe the settings have been corrupted and I need to reset the unit before I can log on again. I'll hold off on this until I've made the copies of the drives.
3) Maybe the Repair All Errors is running into some kind of problem doing the repair. Based on the Disk LEDs it seems to be running this every time I start it up, and roughly ten minutes into it the Disk 1 LED lights solid and the System LED starts blinking rapidly indicating a panic. If this is the case, would removing one of the drives put it into a situation where it would be able to come up in degraded mode, or would it still be giving the error? This is something I'll test once I've made the copies of the drives.

While SpinRite is giving all the drives a clean bill of health, Drive 1 on the SMART screen seems to show fewer seek errors than the others but more ecc corrections. So I'm wondering if this is a situation where the drive is reaching a timeout in how long the RAID will wait for the drive to recover from an ecc correction. I believe that Western Digital drives in their non-enterprise versions sometimes suffer from this in RAIDs, so this should be true of Seagates also.

If anyone wants to chime in with any ideas I'll give those a try also. This is my last Snap 4100 and it was kind of handy to keep because I could toss Mac or PC files on it and access it from anywhere. I'm going to the trouble of trying to fix this one because I really hate to give up on it until I've tried everything. I'm not in need of data recovery because it was backed up.

[rpmurray]

Just in case anyone is on the edge of their seats waiting for an update.

I have a nice matched set of four Barracuda 7200.10 ST3160215A drives that I've run through SpinRite and it says they're all fine. According to the information it's giving me, they have the same number of sectors and bytes as the original drives. I have a StarTech drive duplicator that I'm going to use to make clones of the original drives and then I'll start testing different scenarios to see if I can get the RAID to start working again.

[rpmurray]

And splat, he hits the wall.

Replaced the battery. This also had the effect of doing a reset on the unit, so I didn't have to do that.
Started it up with the four cloned drives and got the same problem, it goes into panic mode after about 10 minutes of disk checking. Shut it down.
Unplugged drive 1 and started it back up again. Same problem, about 10 minutes into the disk check it goes into panic mode.

I'm not sure what to do now. When it starts, it won't let me go to the Admin page to see the status of the drive, and I can't get into debug mode so I can check the logs and look for clues.

I also double-checked and this unit has the motherboard modification to allow the larger drives.

[bitor]

rpmurray, I'm wondering if your power supply is weak. Take a PC ATX power supply to temporarily supply the drives with power. Power on the drive then power on the unit right after.
What SnapOS version do you have?

bitor

Originally Posted by rpmurray

And splat, he hits the wall.

Replaced the battery. This also had the effect of doing a reset on the unit, so I didn't have to do that.
Started it up with the four cloned drives and got the same problem, it goes into panic mode after about 10 minutes of disk checking. Shut it down.
Unplugged drive 1 and started it back up again. Same problem, about 10 minutes into the disk check it goes into panic mode.

I'm not sure what to do now. When it starts, it won't let me go to the Admin page to see the status of the drive, and I can't get into debug mode so I can check the logs and look for clues.

I also double-checked and this unit has the motherboard modification to allow the larger drives.

[rpmurray]

bitor, the power supply doesn't seem to have any trouble spinning up the four hard drives, but I should test for this, just in case. I have a few other 4100s that aren't being used anymore so I can try switching over the drives to one of them to see if the problem persists. Might take awhile, I'll need to drag one out of storage.

I can't get into the admin pages to see what version it is, but I believe it was 3.4.790

[bitor]

rpmurray, In the debug menu you can re-size the drives. This is another thing to try. May or may not help, but at this point it's worth a try.
What is keeping you from upgrading to the latest SnapOS? It's free now from Overland and might fix the problem. When I had a similar problem with my unit and upgrade the OS it fixed the proglem. Again, may or may not help , but worth a try.

I would try the SnapOS upgrade first then power supply then re-size the drives via debug if you can get there.

Brain fart .. strike the info on the larger support for hard drives concerning v3.4.805. This unit is limited to that by default due to hardware issues.

bitor

Originally Posted by rpmurray

bitor, the power supply doesn't seem to have any trouble spinning up the four hard drives, but I should test for this, just in case. I have a few other 4100s that aren't being used anymore so I can try switching over the drives to one of them to see if the problem persists. Might take awhile, I'll need to drag one out of storage.

I can't get into the admin pages to see what version it is, but I believe it was 3.4.790

[rpmurray]

I've verified that the SnapOS is 3.4.790.

I would love to be able to try things in debug, but for some reason, even when I reset the unit and give it a new password, it won't let me sign on.

As an experiment, I hooked up Disk 1 and disconnected the other drives just to see if it would let me access the admin pages, but that didn't work. Even with just one drive, it goes immediately into panic mode. So now I'm suspecting hardware, because with one drive it should at least come up with a broken raid.

I'm not too keen on trying an update of the OS when I can't even sign onto the snap so I can go to the debug pages. But, if all else fails, I'll give that a shot too.

[bitor]

Brain storming...did not know if you knew this, but you don't have to setup a password when you reset it. You can leave it blank and enter "root" as user name and hit enter.
Little things like this could make a difference when things go wrong. It's another think to cross off your list and doesn't hurt anything. You are going to have to start thinking out of the box. Do you have a com1 port on the motherboard?
If you can update the SnapOS to the newest then I would do it. Once again, another thing to cross off the list that will not harm anything. Use the reset button to reset it. Some flags in the bios might not have been reset when the cmos battery was removed. Double check and redo everything to make sure.

Originally Posted by rpmurray

I've verified that the SnapOS is 3.4.790.

I would love to be able to try things in debug, but for some reason, even when I reset the unit and give it a new password, it won't let me sign on.

As an experiment, I hooked up Disk 1 and disconnected the other drives just to see if it would let me access the admin pages, but that didn't work. Even with just one drive, it goes immediately into panic mode. So now I'm suspecting hardware, because with one drive it should at least come up with a broken raid.

I'm not too keen on trying an update of the OS when I can't even sign onto the snap so I can go to the debug pages. But, if all else fails, I'll give that a shot too.

[blue68f100]

Since you can not access the admin with NO HDs installed, I would guess your MB is toast.

[bitor]

I was thinking the same thing , but wanted him to double check everything.