SNAP 1100 Never Finishes fsck

EnduroRacer · 04-18-2007, 06:45 AM

Hi all,

Last night my UPS in the server room failed. After switching it to bypass mode to get everyone back up and running I brought all my servers back up but was unable to connect to my SNAP server via NFS. I tried from Windows and was able to connect to it but neither of my shares were available. Explains why trying to mount the share failed.

I logged into the SNAP via the web configuration and found both of my shares had a red X over them saying they were unavailable due to disk errors. I ran the Disk repair utility and it got to 5% and since it was late I left with it running. I logged in and checked it again this morning and it is still at 5%. I checked the disk logs and it appears it is hitting a bad block and just trying the same block over and over and over. It's been trying the same block all night! Here is an excerpt from the log:

File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:01:13 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:01:09 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:01:04 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:59 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:54 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:49 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:44 AM
E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:39 AM

Right now I can't really do much with the SNAP. It won't allow me to restart it due to the disk check in progress. It won't let me start a new check either.

Any one else run into this problem? Are there any debug commands which will allow you to abort a fsck?

There is some data on here I really need to get since it was down during the backup last night. I think after this I am done with SNAP servers. This is our 3rd one that has failed in 6 yrs. They only seem to last about 2yrs. I love the fact you can turn them on and talk to anything but the speed and reliability leaves too much to be desired.

blue68f100 · 04-18-2007, 07:35 AM

I'm afraid the the HD has failed. Your only chance of recovery will be through a recovery service. Just pull the power connection to shut it down, it will not go any further. On all HD that I have, if they fail no matter what OS it's seams to be in the 4-5% range. I have noticed that Adaptec has started installing Maxtor HD in all of their systems. If yours have the junk drives I'm surprised it made it 2 yrs. Maxtor has the Highest Failure rate of all MFG. I bought a referb 2200 unit, it did not even make it through the raid 1 build without failing. I would recommend SpinRite but my experience tells me it will not help you in this case. But if you have a copy it may be worth a shot, but if your going to send it out for recovey, I recommend not runing it.

Did you have the snap setup to send you email or snmp traps if there were any problems? All have this feature, but most users fail to set it up.

You can still recovery the boot tracks if you want to install a new HD.

I recommend contacting Douglas at FrontLineDataRecovery.com . He has years of experience with Snap HD's, since he use to work in their drive division. I do not have his phone number handy at the moment, will add it this afternoon if needed.

There has been a study On HD saying the mean failure time is 3 yrs for all if they make it past the first 60-90 days. HD fail in all systems, the reason most users prefer redundt systems.

Did your UPS fail to startup due to inrush load, batteries or other means of faliure?

EnduroRacer · 04-18-2007, 08:10 AM

Thanks for the reply blue. I never setup the email function. Most of the data which isn't backed up will be needed by tonight so data recovery isn't an option. I instructed all the users to not use this thing for storage it was intended merely as a transfer point since we have a mixture of Unix, Mac OS 9 + X, and Windows. Of course they didn't listen so now they're going to be rebuilding ads for tomorrow's edition.

I'm not sure yet what caused the UPS to fail. It is an old PowerWare Prestige 6000 unit. The unit itself is probably 11 yrs old but the batteries are about 2 yrs old. It was late when it occurred so I just switched it to Bypass so my users could finish their work before deadline. I'll investigate further today.

I would hope to get 5 yrs at least out of a disk drive. I have some Seagate drives in Sun boxes which have been getting hammered every day for over 12 yrs and are still alive and kicking. Of course they weren't cheap.

I'll give tech support a try see if they can suggest anything. I would think there would be a way to mark the block as bad and continue on but I guess the Snap OS won't do that.

blue68f100 · 04-18-2007, 11:49 AM

SpinRite can mark bad sectors out, what it does on Maintance Mode. But if it's too bad it may not be recoverable.

If the Snap has Maxtors drives, it's your problem. I have yet to have one that was worth anything. If that's the case just install a god drive.

Before the drive fails completely you need to copy the 1st 25-40megs of the HD. This is where the OS resides. Without it you will not be able to boot a new HD. Instructions in the FAQ, link in my signature.

EnduroRacer · 04-19-2007, 06:54 AM

I called SNAP tech support but they wanted $120 just to talk to me. That to me is ridiculous since the damn thing isn't even 2 yrs old! I figure they're just going to tell me the drive is bad anyway.

I yanked the drive out of the SNAP and yup it was a %$&%$# Maxtor drive. I personally blacklisted Maxtor drives way back in 1994 when I had my first experience with them.

I had an old old old version of SpinRite but I went ahead and bought the latest version of it. I popped the drive in an unused PC and started SpinRite on it in recovery mode. It got up to 18% rather quickly but then it started hitting bad sectors. It took the rest of the day to get through the next GB. I left it running all night but I doubt it will be successful.

Yesterday was pretty chaotic. I have an AS/400, Solaris, AIX, RedHat, several macs, and numerous PCs all configured to mount the SNAP. Had to configure them to go elsewhere temporarily and transfer files for people since I didn't have time to show them what to do.

I'm debating whether to just toss the SNAP and configure one of my Linux boxes in its place. It should be more reliable but all of our other sites use SNAPs the same way we do and it would probably be better to conform. I really don't want to go through this again though! I guess I'll probably end up picking up a good drive today and setting it back up.

re3dyb0y · 04-19-2007, 08:37 AM

Quote:

Originally Posted by EnduroRacer

I yanked the drive out of the SNAP and yup it was a %$&%$# Maxtor drive. I personally blacklisted Maxtor drives way back in 1994 when I had my first experience with them.

Maxtors are evil

You just cant believe for the price, they cant use Western Digital, or even Seagate with their 5 year warrantys...

Sure, its a bit more expensive, but in the scheme of things, it'd be a pittance

04-18-2007, 06:45 AM	#1
EnduroRacer Cooling Neophyte Join Date: Apr 2007 Location: Lansdale Posts: 3	SNAP 1100 Never Finishes fsck Hi all, Last night my UPS in the server room failed. After switching it to bypass mode to get everyone back up and running I brought all my servers back up but was unable to connect to my SNAP server via NFS. I tried from Windows and was able to connect to it but neither of my shares were available. Explains why trying to mount the share failed. I logged into the SNAP via the web configuration and found both of my shares had a red X over them saying they were unavailable due to disk errors. I ran the Disk repair utility and it got to 5% and since it was late I left with it running. I logged in and checked it again this morning and it is still at 5%. I checked the disk logs and it appears it is hitting a bad block and just trying the same block over and over and over. It's been trying the same block all night! Here is an excerpt from the log: File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:01:13 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:01:09 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:01:04 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:59 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:54 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:49 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:44 AM E File System Check : Cannot Read: Blk 29950272 Disk 10000 4/18/2007 10:00:39 AM Right now I can't really do much with the SNAP. It won't allow me to restart it due to the disk check in progress. It won't let me start a new check either. Any one else run into this problem? Are there any debug commands which will allow you to abort a fsck? There is some data on here I really need to get since it was down during the backup last night. I think after this I am done with SNAP servers. This is our 3rd one that has failed in 6 yrs. They only seem to last about 2yrs. I love the fact you can turn them on and talk to anything but the speed and reliability leaves too much to be desired.

04-18-2007, 07:35 AM	#2
blue68f100 Thermophile Join Date: Jul 2005 Location: Plano, TX Posts: 3,135	Re: SNAP 1100 Never Finishes fsck I'm afraid the the HD has failed. Your only chance of recovery will be through a recovery service. Just pull the power connection to shut it down, it will not go any further. On all HD that I have, if they fail no matter what OS it's seams to be in the 4-5% range. I have noticed that Adaptec has started installing Maxtor HD in all of their systems. If yours have the junk drives I'm surprised it made it 2 yrs. Maxtor has the Highest Failure rate of all MFG. I bought a referb 2200 unit, it did not even make it through the raid 1 build without failing. I would recommend SpinRite but my experience tells me it will not help you in this case. But if you have a copy it may be worth a shot, but if your going to send it out for recovey, I recommend not runing it. Did you have the snap setup to send you email or snmp traps if there were any problems? All have this feature, but most users fail to set it up. You can still recovery the boot tracks if you want to install a new HD. I recommend contacting Douglas at FrontLineDataRecovery.com . He has years of experience with Snap HD's, since he use to work in their drive division. I do not have his phone number handy at the moment, will add it this afternoon if needed. There has been a study On HD saying the mean failure time is 3 yrs for all if they make it past the first 60-90 days. HD fail in all systems, the reason most users prefer redundt systems. Did your UPS fail to startup due to inrush load, batteries or other means of faliure? __________________ 1 Snap 4500 - 1.0T (4 x 250gig WD2500SB RE), Raid5, 1 Snap 4500 - 1.6T (4 x 400gig Seagates), Raid5, 1 Snap 4200 - 4.0T (4 x 2gig Seagates), Raid5, Using SATA converts from Andy Link to SnapOS FAQ's http://forums.procooling.com/vbb/showthread.php?t=13820

04-18-2007, 08:10 AM	#3
EnduroRacer Cooling Neophyte Join Date: Apr 2007 Location: Lansdale Posts: 3	Re: SNAP 1100 Never Finishes fsck Thanks for the reply blue. I never setup the email function. Most of the data which isn't backed up will be needed by tonight so data recovery isn't an option. I instructed all the users to not use this thing for storage it was intended merely as a transfer point since we have a mixture of Unix, Mac OS 9 + X, and Windows. Of course they didn't listen so now they're going to be rebuilding ads for tomorrow's edition. I'm not sure yet what caused the UPS to fail. It is an old PowerWare Prestige 6000 unit. The unit itself is probably 11 yrs old but the batteries are about 2 yrs old. It was late when it occurred so I just switched it to Bypass so my users could finish their work before deadline. I'll investigate further today. I would hope to get 5 yrs at least out of a disk drive. I have some Seagate drives in Sun boxes which have been getting hammered every day for over 12 yrs and are still alive and kicking. Of course they weren't cheap. I'll give tech support a try see if they can suggest anything. I would think there would be a way to mark the block as bad and continue on but I guess the Snap OS won't do that.

04-18-2007, 11:49 AM	#4
blue68f100 Thermophile Join Date: Jul 2005 Location: Plano, TX Posts: 3,135	Re: SNAP 1100 Never Finishes fsck SpinRite can mark bad sectors out, what it does on Maintance Mode. But if it's too bad it may not be recoverable. If the Snap has Maxtors drives, it's your problem. I have yet to have one that was worth anything. If that's the case just install a god drive. Before the drive fails completely you need to copy the 1st 25-40megs of the HD. This is where the OS resides. Without it you will not be able to boot a new HD. Instructions in the FAQ, link in my signature. __________________ 1 Snap 4500 - 1.0T (4 x 250gig WD2500SB RE), Raid5, 1 Snap 4500 - 1.6T (4 x 400gig Seagates), Raid5, 1 Snap 4200 - 4.0T (4 x 2gig Seagates), Raid5, Using SATA converts from Andy Link to SnapOS FAQ's http://forums.procooling.com/vbb/showthread.php?t=13820

04-19-2007, 06:54 AM	#5
EnduroRacer Cooling Neophyte Join Date: Apr 2007 Location: Lansdale Posts: 3	Re: SNAP 1100 Never Finishes fsck I called SNAP tech support but they wanted $120 just to talk to me. That to me is ridiculous since the damn thing isn't even 2 yrs old! I figure they're just going to tell me the drive is bad anyway. I yanked the drive out of the SNAP and yup it was a %$&%$# Maxtor drive. I personally blacklisted Maxtor drives way back in 1994 when I had my first experience with them. I had an old old old version of SpinRite but I went ahead and bought the latest version of it. I popped the drive in an unused PC and started SpinRite on it in recovery mode. It got up to 18% rather quickly but then it started hitting bad sectors. It took the rest of the day to get through the next GB. I left it running all night but I doubt it will be successful. Yesterday was pretty chaotic. I have an AS/400, Solaris, AIX, RedHat, several macs, and numerous PCs all configured to mount the SNAP. Had to configure them to go elsewhere temporarily and transfer files for people since I didn't have time to show them what to do. I'm debating whether to just toss the SNAP and configure one of my Linux boxes in its place. It should be more reliable but all of our other sites use SNAPs the same way we do and it would probably be better to conform. I really don't want to go through this again though! I guess I'll probably end up picking up a good drive today and setting it back up.

Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)