HELP! rebuild/sync 2 orphans in a 4 drive raid

[lagunacomputer]

Hi guys.......i used to work at quantum and have two 4100's.

I need help rejoining 2 orphaned disks to my 4 drive raid.

It was configured for a RAID 5 using 4 drives and has cracked into 2 orphans and a single cracked set of the other two drives.

What is the procedure for getting BOTH these orphans back into the raid?

I have tried ripping one out and putting in a new drive, formatting and setting to spare. When it reboots, it starts the re-build but it always leaves one of the orphans out no matter what.

I have been able to get it into 'degraded mode' with 3 drives and 1 orphan but i still cant get my shares online. I am too scared to format the second orphan for fear of data loss.
I have considered breaking the raid to get 4 individual disks, then forcing a 'config devices config raid 10000, 10008, 10010, 10018' but am scared im going to lose everything.

If i break this raid will i be left with 4 orphans that are rebuildable? or will i be left with 2 orphans and 2 blank drives? Also, if i get 3 drives together in a degraded raid with no share access, will 'spare' ing the 4th drive (2nd orphan) allow a total rebuild? Please help in any way before i delete my damn data for good, i still have hope.

here's what i have:
Logical Device: 10006 Position: 0 JBOD Size (KB): 32296 Free (KB): 23952 Private Mounted
Label:Private Contains system files only
Unique Id: 0x2D6964745FEB06DF Mount: /priv Index: 12 Order: 0
Partition: 10006 Physical: 10007 FS Size (KB): 32768 Starting Blk: 515 Private
Physical: 10007 Drive Slot: 0 IDE Size (KB): 134217216 Fixed

Logical Device: 1000E Position: 0 JBOD Size (KB): 32296 Free (KB): 23936 Private Mounted
Label:Private Contains system files only
Unique Id: 0x1FEDC58F0B7F8AD6 Mount: /pri2 Index: 13 Order: 1
Partition: 1000E Physical: 1000F FS Size (KB): 32768 Starting Blk: 515 Private
Physical: 1000F Drive Slot: 1 IDE Size (KB): 60238848 Fixed

Logical Device: 10010 Position: 1 ORPHAN Size (KB): 59834720 Free (KB): 0 Public Unmounted
Labelrive3 Orphan from DIRTYWHORE - RAID5
Unique Id: 0x2E85267F1A5A8467
Partition: 10010 Physical: 10017 ORPHAN Size (KB): 59834720 Starting Blk: 59656 Public
Physical: 10017 Drive Slot: 2 IDE Size (KB): 75067904 Fixed

Logical Device: 10018 Position: 2 ORPHAN Size (KB): 59834720 Free (KB): 0 Public Unmounted
Labelrive4 Orphan from DIRTYWHORE - RAID5
Unique Id: 0x2E85267F1A5A8467
Partition: 10018 Physical: 1001F ORPHAN Size (KB): 59834720 Starting Blk: 61583 Public
Physical: 1001F Drive Slot: 3 IDE Size (KB): 78150656 Fixed

Logical Device: 60000 Position: 3 RAID_CRACKED Size (KB): 119669440 Free (KB): 0 Public Unmounted
Label:RAID5 Large data protection disk
Unique Id: 0x2E85267F1A5A8467 Mount: /0 Index: 0 Order: 255
Partition: 10000 Physical: 10007 R 60000 Size (KB): 59834720 Starting Blk: 96624 Public
Physical: 10007 Drive Slot: 0 IDE Size (KB): 134217216 Fixed
Partition: 10008 Physical: 1000F R 60000 Size (KB): 59834720 Starting Blk: 50388 Public
Physical: 1000F Drive Slot: 1 IDE Size (KB): 60238848 Fixed



Please ANY IDEAS or DO NOT DO's or anything will help.....
lagunacomputer@gmail.com

[blue68f100]

Will send you instruction on rebuilid an orphan drive. But it does not apply in your case. Only for Raid 1.

The method you used was proper for raid 5.
Add drive, let it format. Through the disk utility MAKE it a Active/HOT Spare, Should grab the drive and start the rebuild.

BUT

Since you lost 2 drives i'm afraid you are SOL. It requires parity info from all good drives to do the rebuild. Since you lost 1/2 your parity data, it does not have enough info to do the rebuild. The only execption is a raid 5 with 7 drives.

To get the unit back to a working unit after the drives have been replaced, You may need to break the array to JBOD. After the process is completed, then back to RAID5.

[lagunacomputer]

i hear ya!

ok here's my 2 options as i see it, tell me what you think:

1. Break the raid set. then manually do a 'config devices config raid 10000, 10008, 10010, 10018' , and my raid will magically rebuild itself cause it has all the original parts/drives/parity.

PROBLEM? When i break the raid, i end up with 4 individual worthless disks or 2 orphans, 2 individuals and no chance of recovery. Does anyone know what this will do?

2. Format/Spare orphan #3 leaving a raid set of (1,2) and orphan #4. Reboot.
Raid rebuilds leaving Raid (1,2,3) & orphan 4. This *may* leave me in degraded mode, no shares. Reboot.
Format/Spare #4. Reboot. Raid now rebuilds magically with spare #4.

PROBLEM? I dont get a degraded mode with R1,2,3 - O4. or i do, but by formatting/spare the last orphan #4 i lose my parity, leaving option one above undoable as a last chance. OR lose my parity info by scrapping orphan 3 when i should have done orphan 4 first? Is there an order they need to keep parity for a rebuild?

Has anyone broken a raid set and ended up with 4 rebuildable orphans?
Does 'config devices config raid 10000, 10008, 10010, 10018' format everything?

Your thoughts, experiences or advice please. Thank You

[blue68f100]

LABEL THE ORIGINAL DRIVES 1-4, You need to keep the drive order. Read on .....

Did you actually loose both drives at the same time? If so, you may have a disk controller problem. Drive 1 and 2 are on the same controller. Do you have another unit you can transfer these drives into? This needs to be another 4100, a 4000 will not work.

Option 1: Loose every thing, no chance of recovery.

Option 2: sol
You have lost 1/2 of your data, IT's GONE.

There are recovery services that can recover data from raid 5 systems. The price starts at ??? > $500. All drives will have to be sent in, with the order of the drives help.

If you want to start over and actual use the snap. You will need to break the raid5 to JBOD, You should be able to do that from the disk utility section (html). then make sure you don't have any straglers from the split. If so you will need to use the "co de format xxxxx /reinit" to correct. Then set up for raid 5.

Leasons Learned:

setup smtp mail service to receive notification of all problems.

3 drive array with 1 spare: A hot spare, buys you time in the event of a failure.

ONLY use 4 disk array if you have a good backup program.

[lagunacomputer]

i know im fighting it but i just dont understand why my 2 orphans wont merge back into the raid....i swear ive done it before one time...... the never failed they just became separated from the set. all the data and parity is still on them, they just need to all resync back up.....they are still the same drives.....

well im going for broke but i thank you for your input! ill let you know how it works out.

[blue68f100]

NOT failing, but dropped form the array at the sametime, would indicate a controller/system problem.

What did the actual log show? Can you send this to me.

It's very odd to loose 2 at the same time. Looking at the info closely both drives 3 & 4 seperated from the array, and HAVE THE SAME ID ?...... These are suppose to be unique.

Do you have 4 new drives? If so make a backup set using dd. Then work off of your backups. Then if something goes wrong nothing is lost.

You said it rebuilt after installing 1 NEW drive or 2 at the same time?
A drive will/should only join the array after being made a HOT SPARE. The snap should have auto fromat the new drive. Then you have to make it a hot spare for it to join the array. This is all done through the web interface.

"co de config raid 10000 10008 10010 10018" would build an 4 disk array, but like you don't know if it would wipe your data out.
or
You might try a "co de resync 60000", but again have NO IDEA what will happen. If the cmd is not valid it will tell you.

Contact Mister_Bill, he has several of these units and may have run into this odd problem before.

[lagunacomputer]

ok, i made a misstatement about the orphans being the original drives now that i think about it. orphans 3 & 4 are not the original drives. ths is how the set got broke in the first place. also, this is happening on my 4000 which is actually a pre-production beta that was given to me when i worked at quantum. but it has always worked flawlessly, and even takes drives up to 120gb.
Model Software Hardware Server # BIOS
4000 series 4.0.860 (US) 2.2.1 555787 2.4.437


Here's how this all happened to the best of my memory. Just remember that i was very carefull not to ever lose 2 drives at once.

A drive failed, and i put in a new spare and it and the raid rebuilt. PERFECT.

Somewhere durring or shortly after that, another drive failed.
But i was able to do another spare/rebuild/replacement and bring the whole thing back up. worked fine for a week. NO PROBLEM.
except i used an older used drive for the replacement. BAD IDEA.

So 3 and 4 aren't the originals, but since they rebuilt the raid and worked, i consider them to have the same data as the originals. UNLESS.....

...But then one of these replacements started reporting bad sectors, but was still functional in the raid. im poor and had used used drives. BIG MISTAKE. here is where my memory gets fuzzy so bear with me.

This bad replacement drive finally failed enough to break the raid set, leaving me with 3raid/good+1orphan/failure/used. NO PROBLEM, just another damn replacement to do. All the data is still on the 3raid/good,

I opened it up to replace the 1oprhan/failed with a NEW clean drive.
BUT ACCIDENTLY PULLED A GOOD MEMBER from the working 3raid intact set. :<

Leaving me with 2raid/good+1orphan/failed+1spare/new (& 1raid/good in hand, thinking it was the 1orphan/failed). rebooted.....CRAP.

.....but I realized I had pulled the wrong drive and shut it down......but i think it started a rebuild and thats how i got 2 identical IDs
So I put back the1raid/good keeping the 1orphan/failed in the snap.

This where i know that u can re-join an orphan with a broken raid set with the proper command, im 99% that i was able to do this. this gave me back my original 3raid/good+1orphan/failure/used before the incorrect drive pull/mixup.

then I replaced the 1orphan/failed with the 1spare/new like it should have been to finally rebuild a solid good 4 drive raid. Then rebooted.

But it didnt rebuild, probably because of the duplicate id's on the two drives. Now I have 2raid/good+2r/good/orphan, . thats why i have a duplicate ID in there somehow. I mixed em up on accident. Lesson here is, don't drink & 'drive'.

ok the more i think about this i remember this happening to my 4100 about a year ago and i was able to put it all back using config/debug. but i dont remember exactly how i did it. i either broke or unmounted the whole thing and resynced.

well im almost done trying everything and pretty much would have given up but im so damn determined to figure it all out. (Not to mention i need the data bad) I believe in this because Quantum says to call them if you get 2 orphans and that sorta backs up my fuzzy memory that it is possible and im sure i figured it out once before on the 4100. I hope this can help someone out someday.

[blue68f100]

poogles_uk knows someone that worked for snapappliance, he maybe able to find out.

I've only used the orphan cmd on raid 1 configurations. Raid 5 structure is quite different.

I label and date all of my drives when I install them. At times I may have several of the same model and size setting on my desk. It's nice to know which ones are good, bad and used.

[jt99]

any follow-up here?