Recover broken 4100 raid (4 drives).

[bigcitymike2]

Ok first of all I have a old backup that can restore most of the data from this set. But i really, really, really, like to recover what i can. Being I work in the IT field, this opportunity recovering a toasted Raid set could lead to a good learning experience.

Using certain forms of software, I've recovered data from normal non raid drives in the past both mac and pc. I know this is raid and it's a new ball game, but I’d like to at least make a good stab at going at it.

ok here's the specs.
Dell pv705 converted to a snap 4100 with V3 software. I'll have to get it and update this thread. 4 200GB drives, but only being recognized as 128 gb useable after formatting.

So i get an email saying something is wrong...

================================================== ==================
05/05/2006 10:34:49 ERROR File System : RAID-5 device 60000 operating in degraded mode
================================================== ==================

The previous message occurred in the following context within the system log:

05/05/2006 10:34:49 ERROR File System : RAID-5 device 60000 operating in degraded mode
05/05/2006 10:34:49 WARNING File System : Logical device 80060000: no spares found to perform hot replacement
05/05/2006 10:34:49 WARNING File System : Disk I/O error on RAID-5 device 60000
05/05/2006 10:34:49 ERROR File System : Unrecoverable error on logical device 60000. Member 10018 failing

So drive 10018 seems bad. I shut the 4100 down for a week or two and proceeded to get another drive. Upon reboot before i installed the new drive I now get a new error.



New error:
================================================== ==================
06/16/2006 23:46:43 ERROR File System : Logical set member 0 not found. Original device ID: 60000
================================================== ==================

The previous message occurred in the following context within the system log:

06/16/2006 23:46:44 INFORMATION System Initialization: Initialization Complete! Memory to be released: 47067072 bytes.
06/16/2006 23:46:44 INFORMATION File System : Opened FDB for device 0x1000E
06/16/2006 23:46:44 INFORMATION File System Check : partition is clean.
06/16/2006 23:46:44 INFORMATION File System Check : Executing fsck /dev/ride1g /fix /fixfatal
06/16/2006 23:46:44 INFORMATION File System : Opened FDB for device 0x10006
06/16/2006 23:46:43 INFORMATION File System Check : partition is clean.
06/16/2006 23:46:43 INFORMATION File System Check : Executing fsck /dev/ride0g /fix /fixfatal
06/16/2006 23:46:43 WARNING File System : Set-up of logical disk failed. Device ID: 60000 (hex)
06/16/2006 23:46:43 WARNING File System : 2 member(s) missing in logical device (original ID: 60000)
06/16/2006 23:46:43 ERROR File System : Logical set member 3 not found. Original device ID: 60000
06/16/2006 23:46:43 ERROR File System : Logical set member 0 not found. Original device ID: 60000
06/16/2006 23:46:38 INFORMATION INIT: Setting IP address to 192.168.1.105
06/16/2006 23:46:37 System Initialization: Server v3.4.805
Build Date: Apr 21 2003 14:00:44
Boot Count: 85

So now it seems it's dropped two disks....or has it? Is 10018 bad and another drive orphaned?

I don't get any bad disk lights on the front panel. I'll have to connect to it again and I'll copy the recent log entries.

Laguna computer from this post:
http://forums.procooling.com/vbb/sho...ghlight=orphan


Seemed to have a similar task and he thought there might be a way to resync the drives. I know after looking for info about the snap 4100 is that there is more then a few data recovery companies out there that specialize in this. And the explcitly point out the Snap raid family of serverw.... So how do they "lift" that data..

-Bigcitymike

[re3dyb0y]

Dell pv705 converted to a snap 4100 with V3 software. I'll have to get it and update this thread. 4 200GB drives, but only being recognized as 128 gb useable after formatting.

Snap 4100's don't support LBA48bit

So you cant have hard drives above 137GB, which works out about 128/130Gb for you

[blue68f100]

Bigcitymike2, it apears you may have removed the wrong drive. 10018 is drive 4 (member 3) , which one did you replace? Member 0 is drive one. Did you remove a power connection???

Once you have 2 drive failures in a raid5 with no hot spares to aut replace one I'm afraid you will SOL.

Since the RAID 5 keeps parity data for ajacent drives, it is able to repair a single drive failure. Loosing 2 drives breaks the parity needed for all drives.

As re3dyb0y said the 4100 does not support LBA48 bit addessing. V3.4.805 does support it but not your hardware.

There is an article n the wiki section refering to repairing ORPHIN Drives. the procedure is basicly the same for a RAID 5 you just need to list the drives and 60000 for the set.

[bigcitymike2]

Thanks for the reply.

I was up late posting and I did not make it too clear. I'm aware of the hardware limits. Really as should anybody that spent any time reading here. You guys answer that question way too often. All I had were 200's so that's what went in.

All drives are still in their orig location. I have not swapped any of them with new drives or otherwise. According to a snap PDF I d/l 10018 is drive 4. The drive behind the led's and the power button. I did remove the connections and re connect but thats all. I did that per the service manual. It specifically states to if there is no amber disk light lit. Which there is not.

Here is a LINK to the image.

So does anyone know how the data recovery people do it?

Here is just a few links. They all mention SNAP servers or even go as far to say it's a specialty of theirs:


http://www.ontrack.com/special/0903drnews.aspx

(Snap server drives must be moved to Intel machine) this is interesting..


http://www.datarecoveryclinic.com/de...d_recovery.htm

(Our success rate on Snap and PowerVault is nearly 100% as long as the proper steps are taken. Dell support is often too quick to initialize arrays.)


http://www.dtidata.com/dell_snap_server-recovery.asp

(The RAID Engineers at dtidata.com have the expertise to recovery data and write custom software on the fly to fit your recovery situation. Yielding the best results and highest recovery rates for loss of multiple drives or botched rebuild failures using in-house custom software configured and written on the fly to fit your data loss situation.)

And then there these guys that seems to be Hollywood's answer to a crash..
http://www.drivesavers.com/why_drive...commended.html


They all make some rather bold claim. And sure anyone can, but allot of them refer to snap server specifically.

[re3dyb0y]

Im sure if you paid the lovely company enough they would be able to provide you with some software/customised OS for getting the data off....


The reference to intel machines.

They all use Intel CPU's and chipsets. I compared my Snap 2200 and it has the same spec CPU and exactly the same northbridge as my Slot 233Mhz PII pc.

That is probably why it needs to be on an intel machine



We just need a contact inside these companys to find out what they use...

[blue68f100]

Sorry to here it failed before the drive could be replaced. We had another user had 2 drives fail with in 24 hrs of each other. Had to ask. Some time drives do not spinup after shut down after running for long periods of time. You may try tapping on the drive while tring a startup.

The recovery services have several techniques thay use. One is to make sure it was not a controller board on the drives. Then they have special software to read the drives. Then there are some cases where they remove the platters then do a read, via idientical drive or slow speed specility hardware. Recovery for RAID are extermely expensive. But the cost is based on how quick you need the data.

The new version 6.0 of GRC SpinWrite indicates it can now repair some raid sets and drives. It does mention the XFS & unix file systems. I would only try this IF I COULD MAKE A CLOAN OF THE DRIVE. If able to do that, it means just coruption is present or damage table. AND SpinWrite may do the trick.

The 48bit thing. Alot of the users have gone with high capacity drives, mainly because the are cheaper. Fry's Electronics has 250gig seagates (5 yr) on sale for $79.99 USD right now.

[bigcitymike2]

CLOAN OF THE DRIVE.. I see. I can take a stab at it. Did i see that in the wiki somewhere? Maybe it was while I was searching here. The app was called DD or something like that.

I'll lookin to GRC SpinWrite 6.0 and see what it can do...

It's turning into more of a mission now.

Thanks for the heads up.


PS

blue68f100 i think i answered your question posed over here.

[blue68f100]

Since you are cloaning a whole drive download the CD version of it. I think the site was digitaldolly.??? It is menu drive and will work fine. The main thing is that it must be a RAW copy.

And yes their are instructions in the wiki section for use dd from Knoppix CD.

[bigcitymike2]

Well I'm going to try another approach i think first. I found this piece of sofware called RAID Reconstructor.

It appers you use this to "rebuild" the raid to an image. And them use GetDataBack to do just that.

It states
" Runtime's RAID Reconstructor will help you recover data from broken

* RAID Level 5 Array consisting of 3 to 14 drives
* RAID Level 0 Array (Striping) consisting of 2 drives"

and GetDataBack is more than an undelete or file recovery program or a system restore.

So I figure since it's read only it's worth a shot. I connected the 3 remaining drives of the raid to a test box. RAID Reconstructor seemed to identify the raid set and attempt to creat an image. I excluded the drive from the 4100 that was logged as being bad. I ran RAID Reconstructor and it started to building an image, at this point i'm not sure if its any good because I don't have enough disk space for the image. (i'm regretting upgrading those drives now i tell ya)

So i'm going to order a 500BG drive to restore the image to and see if i can really "getdataback"

[jontz]

I have used Get Data Back several times. Very good at recovering stuff. Let me know how it works in this instance though, as I am not sure how it will handle the modified XFS file system the snap uses. It might just say "yeah, there's something there. So what?" Or, it might work just flawlessly. Looking forward to hearing how it goes.

[re3dyb0y]

Stupid Post Posted

[blue68f100]

Raid Resconstructor is a ineresting piece of sotware. Would be interested if it worked.

Now a question for the XFS file system that Snap Uses. If I recall when I upgraded my 2000 snap from a v1. There was a note saying the bios had to be 1.1 or greater. The program from snap upped it to 1.2.180. At one point a message came up saying that the hd's had to be re-formated. Looking at some history on the XFS file system, it too went through a file structure change, before Snap did. I'm wondering how much changes snap actuall did, if any ...... hummmmmm

and are they actually using a modified or just a newer version of the XFS.

[re3dyb0y]

Well, they must have done some major changes, as you had to backup all your files ands stuff...

[jontz]

Originally Posted by re3dyb0y

Stupid Post Posted

eh?

[re3dyb0y]

Originally Posted by jontz

eh?

I'd posted and then re-read the post realising i hadn't read the post properly

So i edited it rahter than saying something stupid

[ticklemeozmo]

Arise from the dead, fair thread...

I used Raid Reconstructor to remake an image of my RAID 5 dead array and it seems to have worked. The Get Data Back for FAT version seemed to FIND that there were files and found types of files (I only stored three different common file types on the snap) and it found what I perceived to be the correct ratio of each file. However, obviously would not recover the file.

Any luck using XFS recovery tools from the RAID Reconstructed image before I attempt to do it?

[blue68f100]

If you only lost one drive, generally you can replace it with one = or >. But there is a bug if the unit was upgraded from a v2 to 3 and the unit was not reinit. If that is the case you will need to contact Snap-tech here at the forum. He has all of the recovery tools and hardware and is resonable.

[ticklemeozmo]

If you are talking about me, my story is different (aren't they all?).

Snap 4000, v4.0.860 running 4x300 (UltraATA 133). So you can see exactly where the problem is. Running for over 2 years with no problem. Then one dark and stormy night, all the directories have no files. It seems that the second half of the drives "failed" as they were red in the view.

The drives are fine, they are recognized properly in other boxen, just that the raid is broken and it fails to put them together. I've tried three Snap 4000s with the same config and it still fails to put it together. (It won't check, and it fails to put a non-checked drive into the array).

So, I threw them in my Intel box, and tried Raid Reconstructor, and got a nice 800+GB image file. And I wanted to know people's experiences from trying to recovery from that image, if any.

I've upgraded to Infrants in the meantime, and as soon as I recover this data off, I'm tossing them off the side of my building. I can confirm that UltraATA 133 drives WILL fail in a Snap with NO warning. 3 Snap 4000s with 4x300GBs. EVERY Snap presented the same problem. This one just crashed before I could replace it.

[blue68f100]

The problem is that the array was larger that 1T, 4 x 250gig is max for a 4000. User have resently done same testing and confirmed there is a 1T max linit. But users keep tring to push the limit.

If you haven't damaged the 300 gig drives by mounting them in PC.

CONTACT Sanp-tech. HE CAN DO THE RECOVERY FOR YOU.

If your going to throw them out I know of several users that would be happy to take them off you hands.

[Phoenix32]

Originally Posted by blue68f100

If your going to throw them out I know of several users that would be happy to take them off you hands.

LIKE ME!

[jontz]

Originally Posted by Phoenix32

LIKE ME!

Down boy! DOWN!

[Phoenix32]

Originally Posted by jontz

Down boy! DOWN!

LOL, yeah, but I been trying to get another 4000 for some time now...

[ticklemeozmo]

Originally Posted by blue68f100

The problem is that the array was larger that 1T, 4 x 250gig is max for a 4000. User have resently done same testing and confirmed there is a 1T max linit. But users keep tring to push the limit.

If someone has told me 2 years ago that 1TB was the safe limit I would've listened! It should be in the wiki!!

Originally Posted by blue68f100

CONTACT Sanp-tech. HE CAN DO THE RECOVERY FOR YOU.

Already spent 2K on the Infrant NAS that replaced the Snap Server. So the data really isn't worth another 2 grand for me to recover at this point. 500 maybe, 2K definitely not.

Originally Posted by blue68f100

If your going to throw them out I know of several users that would be happy to take them off you hands.

Offering to my friends first, but if you are interested, I'll take a silent auction. PM me your bid.

2x - Snap 4000, 2U, v4.0.860, 128MB, no drives. Factory reset. No warranty, expressed or implied. Controllers were fine, drives were just too much (over 250GB, faster than ATA100). Will ship however you want it.

[rpmurray]

Originally Posted by ticklemeozmo

Already spent 2K on the Infrant NAS that replaced the Snap Server.

Anything good, bad or otherwise to say about the Infrant?

[blue68f100]

ticklemeozmo,

The 1T showed up when we started questioning the extremely log time to build the array. You being the 1st to install 400gig, we did not reconize the problem. There were several users that could not build with 400 gig drives. This is when we know there was a problem. The new 7200.10 added more capacity (320gig vs 300) with less platers. Some users could not boot with 4 400's, weak PS. Another problem that has showed up with in the last few months.

We now only recommend 4 x 250gig. Yet a new user decided to install 4 x 750gig and run 2 raid 1. This should produce usable space of 2 x 700 gig = 1.4T. Of couse the speed will be slower than a raid 5. Do not know if the raid 1 can be split between contollers.