Snap 4100 orphaned drive question

[Reason_Man]

Hi, i'm new here and to Snap server maint so I could be asking something easily fixed. I searched around and couldn't really find anything that answered my question.

The other day we had a storm and the power went out. After that the server powered back up and seemed to work fine. Later that day someone tried to access files on it(from a Mac) and it threw some error about not being able to mount the volume. I checked around the office and found no one could connect but the web panel was still up and functional, except that the disks were all labeled "offline". I powered it down and back up and still nothing. I checked the logs, which only went back as far as May 10, that after the power up after the storm, the entry "File System Check : FSCK fatal error = 32" is in after every boot. Also, the command "info device" spits out:

Logical Device: 10006      Position: 0  JBOD     Size (KB):    32296  Free (KB):    22368  Private  Mounted
  Label:Private  Contains system files only
     Unique Id: 0x5C9268DD22EDF7B3    Mount: /priv      Index: 12  Order: 0
     Partition: 10006  Physical: 10007  FS       Size (KB):    32768  Starting Blk:   515  Private
      Physical: 10007    Drive Slot: 0  IDE      Size (KB): 60051456  Fixed    

Logical Device: 1000E      Position: 0  JBOD     Size (KB):    32296  Free (KB):    22600  Private  Mounted
  Label:Private  Contains system files only
     Unique Id: 0x0251F4EB11E9CD26    Mount: /pri2      Index: 13  Order: 1
     Partition: 1000E  Physical: 1000F  FS       Size (KB):    32768  Starting Blk:   515  Private
      Physical: 1000F    Drive Slot: 1  IDE      Size (KB): 60051456  Fixed    

Logical Device: 10008      Position: 1  ORPHAN   Size (KB): 59583056  Free (KB):        0  Public   Unmounted
  Label:Drive2  Orphan from SNAP - RAID5
     Unique Id: 0x51688E8607EA3027
     Partition: 10008  Physical: 1000F  ORPHAN   Size (KB): 59583056  Starting Blk: 58422  Public 
      Physical: 1000F    Drive Slot: 1  IDE      Size (KB): 60051456  Fixed    

Logical Device: 60000      Position: 2  RAID_CRACKED Size (KB): 178749168  Free (KB):        0  Public   Unmounted
  Label:RAID5  Large data protection disk
     Unique Id: 0x51688E8607EA3027    Mount: /0         Index: 0  Order: 255
     Partition: 10000  Physical: 10007  R 60000  Size (KB): 59583056  Starting Blk: 58422  Public 
      Physical: 10007    Drive Slot: 0  IDE      Size (KB): 60051456  Fixed    
     Partition: 10010  Physical: 10017  R 60000  Size (KB): 59583056  Starting Blk: 58422  Public 
      Physical: 10017    Drive Slot: 2  IDE      Size (KB): 60051456  Fixed    
     Partition: 10018  Physical: 1001F  R 60000  Size (KB): 59583056  Starting Blk: 58422  Public 
      Physical: 1001F    Drive Slot: 3  IDE      Size (KB): 60051456  Fixed

Now, i've never messed with RAID before so I don't really know how to go about solving this problem. Everything I have managed to find and read says to format the problem disk(assuming disk 2, or 10008?) and rebuild the RAID IF the drive is not dead. The documentation for the server said that if it shows up in the "disk status" panel as orphaned than it might be usable. This is the output of that particular page:

View log for disk Drive2.   	    	  Drive2   - Orphan from SNAP - RAID5
Error.  Unknown disk operation error.
View the disk log for details. 	  	58,186 	Unknown 	Unknown

View log for disk RAID5. 	  	RAID5  - Large data protection disk
Checking.  75% complete

After checking RAID5, the server seems to shutdown. There is no disk activity. Only the system and single "disk" LED flash. I checked the flash patterns and they don't seem to match anything. Also, NOTHING is accessible, not through the network and not through the web admin page, it just times out.

So, are there any ideas as to what went on and how to fix it? If you need any other information let me know. Thanks.

[blue68f100]

If you want to manually format the the drive. Login as admin and get to the debug page. Enter "co de format 10008 /reinit" If it fails on this you will need to repace the drive. If it formats, Set it up as a active spare. The Snap will grab it and start the rebuild of the raid array.

[Reason_Man]

And by formatting that single drive I shouldn't lose any data? I feel pretty confident that if anything happens to the data, the company may be in serious trouble. Thinking back I should have really pushed for some sort of secondary backup solution.

[jontz]

I'm a little curious about your situation. If only one of the drives is dead, I am not sure why all of the drives appear to be offline. When I had a disk fail in my 4100, the RAID volume showed as running in degraded mode, but I could still access data from it. Only drive 2 was offline, as it seized up and wouldn't spin anymore. Your problem sounds different.

I wouldn't try formatting anything just yet, even though you shouldn't lose any data if you try and format the drive that is orphaned. I would just replace the drive and set it up as a hot spare and see what happens. Hopefully it will just rebuild the array and go on it's merry way.

I'm just wondering if you don't have a more serious problem with the motherboard or something...I hope not, it just sounds odd from the way you described it.

[blue68f100]

Jontz is correct you should be in degrade mode. Depending on which version of the OS, Degrade may not be an option. I'm pretty sure I would go ahead and get a replacement drive. Once they have a major problem its time to replace them ALL.

I though there was another user that could not access there data till the drive was replaced. I had one uses report that in degrade mode the speed was about 10% of normal, real slooow.

[Reason_Man]

There was one time that the disk status page didn't freeze up and it displayed RAID5 running in degrade mode but it was only once. I havn't been able to get it back since.

I could replace the drive but the only spare I have lying around is an 80 gig Seagate(ST380021A) and the drives in the server are 61 gig Maxtors(5T060H6). Would mixing that Seagate cause problems? I've read around that mixing and matching is not recomended.

EDIT:

After checking through the log again, the "File System Check : FSCK fatal error = 32" isn't showing up after my last two boots. Ideas?

[blue68f100]

You are right that its not good to mix. Normally snap says the drive needs to be within 2% of capacity. But if the dirive geometry is too far off, it causes problems. What do you have to loose. Install it, the unit should format it, then set it as a hot spare. If every thing goes right it will take the drive and start the resync/rebuild.

[jontz]

Originally Posted by Reason_Man

There was one time that the disk status page didn't freeze up and it displayed RAID5 running in degrade mode but it was only once. I havn't been able to get it back since.

I could replace the drive but the only spare I have lying around is an 80 gig Seagate(ST380021A) and the drives in the server are 61 gig Maxtors(5T060H6). Would mixing that Seagate cause problems? I've read around that mixing and matching is not recomended.

Mixing is not a good idea for the long haul, but I would throw it in just to see what happens. If it gets the unit back on its feet, I would then back up all the data and THROW AWAY all of the maxtor drives in the unit. Maxtor drives have a very high rate of failure, as you are finding out. Replace them with some 120GB seagates and you have a bigger snap server with more reliable dives in it.

Originally Posted by Reason_Man

EDIT:

After checking through the log again, the "File System Check : FSCK fatal error = 32" isn't showing up after my last two boots. Ideas?

No idea. Again, as I said above, this doesn't sound like a simple drive failure issue. Have you powered down and checked to make sure that all of the connections inside the snap are tight? It might be worth checking.

[Reason_Man]

Checking the connections on the drives and board were the first thing I did. I even replaced the ribbon cables some extras I had lying around but the same thing happens.

I was thinking I would just throw the 80 in there as a temp until I could get some better drives. My first thought was to hook everything up to my machine and use that RAID reconstruction tool I saw mentioned on here but I don't know that that would work to well.

EDIT:

Now i've put the new drive in, it formatted on boot, I set it as a spare but the server still seems to freeze like I mentioned before after checking RAID5. I don't know how long i've let it run like that in the past so I think i'll just leave it alone and see what happens. Is there any possibility it's trying to rebuild the array during this time? Because it gives me no idication it's doing anything. No network access, no web access and the only light on the front panel is the system light that blinks ridiculously fast, a constant link light and a net light that flashes from time to time. If it is rebuilding, about how long will it take?

[Phoenix32]

Originally Posted by Reason_Man

I even replaced the ribbon cables some extras I had lying around but the same thing happens.

Well, I guess after this comment I need to throw in my 2 cents worth, Dave can correct me if I am out to lunch here.

Changed cables?

Many SNAP servers (with multiple drives), but not all, use cable select cables not standard IDE cable. While this may not be a problem in your case, and most certainly is not the orignal problem, changing the cables in the middle of this "MAY HAVE" created another problem in the process. Meaning, the new drive might have corrected your problem but changing the cables kept it from doing so. In fact this may be why your error messages changed and no longer showing a fatal disk error.

Hey, just my 2 cents worth. I do not have a 4100, but my 4000 does in fact use cable select cables and changing them to standard IDE cables would be a problem. But as I said, Dave can correct me if I am wrong here and I will just shut up.



Side Note: Different RAID controllers are more or less sensitive to the drives sizes in any type RAID that uses stripping, some will work, some will not, and the ones that will vary in how far they will go. I cannot speak for the controllers built into the SNAP servers. HOWEVER, I can tell you that from my experience, most RAID controllers in RAID 5 arrays using HOT SPARE will usualy NOT pick up a non identical drive to be used in a failed array. When non identical drives can be used, you usualy have to do it manualy as a general rule. Just a thought.

[Reason_Man]

Originally Posted by Phoenix32

Well, I guess after this comment I need to throw in my 2 cents worth, Dave cona correct me if I am out to lunch here.

Changed cables?

Many SNAP servers (with multiple drives), but not all, use cable select cables not standard IDE cable. While this may not be a problem in your case, and most certainly is not the orignal problem, changing the cable in the middle of this process "MAY HAVE" created another problem in the process. Meaning, the new drive might have corrected your problem but changing the cables kept it from doing so. In fact this may be why your error messages changed and no longer showing a fatal disk error.

Hey, just my 2 cents worth. I do not have a 4100, but my 4000 does in fact use cable select cables and changing them to standard IDE cables would be a problem. But as I said, Dave can correct me if I am wrong here and I will just shut up.

Well after I found the other cable didn't change anything I put the original back on so unless it stored the information away somewhere I would think it'd change it back?

[Phoenix32]

Originally Posted by Reason_Man

Well after I found the other cable didn't change anything I put the original back on so unless it stored the information away somewhere I would think it'd change it back?

If that original cable is put back in and in the same way it was originaly connected (same connectors to same original locations), then it should not be a problem.

[blue68f100]

I'm just the back stop for you 4x00 users.

system light that blinks ridiculously fast

This is refered to as PANIC MODE. Not Good. This is normal due to hardware failure/problem or OS. Havn't seen this on the 4x00 models, mainly on 1x00 & 2000 models. There is a way to recover sometimes with the 1k or 2k models. Have no idea how what will happen to the data or has happen to the data.

Was the 80 gig clean or did it have a file structure on it? If so I hope it wasn't unix.

You by chance didn't get the drives swaped around did you?

I would remove the failed drive (or 80gig) all together and see if the unit comes up.

If you have another 4100, I would move the drives over to it and test.

If the data is criticle I would stop right now and see if I could locate the software for recovering the raid set.

djontz is the 4100 user on the forum.

[Reason_Man]

Originally Posted by blue68f100

I'm just the back stop for you 4x00 users.

This is refered to as PANIC MODE. Not Good. This is normal due to hardware failure/problem or OS. Havn't seen this on the 4x00 models, mainly on 1x00 & 2000 models. There is a way to recover sometimes with the 1k or 2k models. Have no idea how what will happen to the data or has happen to the data.

Was the 80 gig clean or did it have a file structure on it? If so I hope it wasn't unix.

You by chance didn't get the drives swaped around did you?

I would remove the failed drive (or 80gig) all together and see if the unit comes up.

If you have another 4100, I would move the drives over to it and test.

If the data is criticle I would stop right now and see if I could locate the software for recovering the raid set.

djontz is the 4100 user on the forum.

The drive previously had a busted copy of Vista on it.

No, the 80 gig replaced drive 2(or 1 if we start at 0).

I'll try taking the drive out all together. I do not have another server but I could look around for a cheap used one to stick my drives in.

I got the server second hand so I don't have any extra software if any came with it. Is there a way I can pull the data from the drives without actually having to physically remove the drives?

EDIT:

Removing the drive does nothing but flash the drive 2 LED amber.

[blue68f100]

There was one user that installed the drives into a pc with a raid 5 controller. Then used a recovery/resync software external of the snap. The snaps use a XFS unix file system.

The OS is stored in flashram. There are several 4000 and 4100 units on ebay. I think the 4100 run a little cheaper because of there lack of upgrade ability. But they are a little faster than the 4000.

[Reason_Man]

I just found an empty 4100 on ebay, enclosure only(+ board and PS). One dollar so I think i'll try there.

If I were to take the drives out of the server and into my machine, is there a particular setup I could have to use? Like, my board supports RAID but I only ave 2 IDE ports. Would I be able to set two drives master and two slave or what?

[blue68f100]

Most MB do not support raid 5 only 0 or 1. You will have to find a controller that support raid 5. Search the threads for raid 5, xfs, recovery ... till you find the one with the user that did his.

[jontz]

WOW....lot of activity on this thread since this morning.

The fast flashing system disk is indeed NOT GOOD. I am convinced now that your problem is not a hard drive problem, or at least a single hard drive problem. Just out of curiosity, did the disk 2 light ever go amber when you had the original orphaned drive in there? Normally it would if there was a problem...

If you set the new drive as a hot spare, it should immediately pick up the drive and put it back into the array. During this time, the unit is accessilbe via the web interface, and you can see the status of the raid rebuild. This proccess should take about 6 hours or so with 120 GB drives, so maybe half that with 60 GB drives. The fact that you are saying that it is frozen and seems to be in panic mode doesn't bode well. The problem with the 4100's is that the OS is in flashram. If there is something wrong with that flash, or any part of the motherboard, there is very little you can do to recover the data from the drives.

You might try as blue suggested finding a PC raid 5 controller and trying to rebuild the array that way. I'd be interested to see your results.

If you are going to pull the drives and put them in a PC to try and rebuild them, try putting a single blank drive in your 4100 and see what happens. I am guessing that it will still be in panic mode and won't install the OS like it should.

If you are seriously interested in that snap on ebay, let me know. I am thinking about picking it up myself and I don't want to get into a bidding war with you if you need it.

Phoenix, just FYI the IDE cables in a 4100 are not CS cables, they are straight IDE cables with only one drive connector on them. All four drives in a 4100 are set as master.

[blue68f100]

Sometimes you can recover with a image file to boot the snap, then reload the OS. Did that several times on my 2000. There is a special reset sequence to clear panic modes. I would check to see if your cooling fans are still working. Most hardware failures are due to heat.

I think your best bet on recovery is with another 4100. Before you install any drives, connect and verify the unit is in raid 5 and not JBOD. You may have to come up with some drives to do the initial setup if a debug cmd want force it into raid 5. If it was reset back to factory settings, it does not change the drive configuration.

[Phoenix32]

Originally Posted by jontz

Phoenix, just FYI the IDE cables in a 4100 are not CS cables, they are straight IDE cables with only one drive connector on them. All four drives in a 4100 are set as master.

ok, which is why I hinted I was not sure on a 4100. Is this true for all 4100s? Asking because some/most 4000 use CS (like mine), but some do not.


This is sounding more and more like his 4100 itself has issues. Start checking CPU/RAM etc (as in seated good etc)?

[blue68f100]

For your information on 4000:
SN < 70700042-003 use M/S (40 ribbon connector)
Version 3 and > CS (80 ribbon connector)

[Reason_Man]

Yeah, I took everything out and reseated it as I know that can sometimes be the problem(i've run across more than one machine that wouldn't do anything just because certain things weren't seated properly).

I probably won't end up bidding on that server since there are 3 already at about 60$ and I can't wait 3 days for something that might end up costing me a couple hundred. If you know of any other place to get a relativly cheap, used 4100 let me know.

So currently my only option is to either get a new server(doubtful) or find a machine with a RAID 5 controller? Can I get one as an expansion card? If I find one with RAID 5, how would I go about setting them up? Currently I have my Windows boot drive on a SATA connection with 2 IDE ports. Could I get a double ribbon cable and hook up the 4 drives to each? If so, how would I configure them as far as master and slave go? Then, how would Windows react to it?

[blue68f100]

Your pc will destroy what's left on the drives. MS doesn't no squat when it comes to drives formated in something not MS.

Yes to a pci raid 5 controller. You need to find the post of the user that did it. You will need to match the raid controller chip on the snap.

Does the unit come up with no drives installed?
If not do a factory reset.
Normal (NOT PANIC)
if YES -> Use assist to locate and assign the IP address.
iF NOT -> pull the power while the unit on. Reboot
Did it come up Normal?
IF NOT Pull power again and DO the factory reset again
Did it come up normal this time?

Once it comes up normal with no drives, login as admin and check you platform byte in debug. If it still indicates 4 drives. Power down and install/connect the drives Powerup. If it comes up check to see if you have lost more than one drive. If So you are having hardware problems.

Their was 1 user that had a second drive fail 2-3 days after the first one. When this happens you are SOL.

This is a lesson learned. All though we use raids that have protection/redundency, Backups are still required.

There are compaines that specialize in raid recovery, and can provide quick turn around for $$$$$$$

[blue68f100]

I found the post on using the external raid controler. It was on a 4000. Need to verify same controller chip. But it's a start.

http://forums.procooling.com/vbb/sho...highlight=raid

[Reason_Man]

I just looked at the system log and saw this:

PANIC : General Protection Fault (#13) at $00244173
EAX=FB505AF0 EBX=08000000 ECX=8E479F9A EDX=00000000
ESP=004023EC EBP=00402C24 ESI=07F716F4 EDI=00404278

What's this mean? Nothing good I assume.