Snap Server 4100 Meltdown

[blue68f100]

It's on my list of things to do, trust me I will get to it.

This time of year I get busy with my leather craft, you have one of my big sellers. Joe has 3 (of ?) for allowing us to use his servers. All are great little gifts. It's how I fund my toys.

[Phoenix32]

Originally Posted by blue68f100

It's on my list of things to do, trust me I will get to it.

This time of year I get busy with my leather craft, you have one of my big sellers. Joe has 3 (of ?) for allowing us to use his servers. All are great little gifts. It's how I fund my toys.

I know, lol...

[snap-tech]

It took me a little while to find the info I was looking for but here it is.

The issue that caused this unit to have fail is the following.

If you look at the in dev log you will see that the last drives starting block is not the same as the other 3 drives. We discovered that this would ultimately cause the superblock error and but not until after a reboot once the raid was rebuilt with a replacement drive.

It had to do with what version the snap was running when the drives were first formated. The starting block was changed between versions 2x and 3x, and was increased by a factor of 1. to find the starting is multiplied by 16 and thats the actual sector were it is located on the disk.

This turned out to be a big issue at snap and from that point on all single drives that were shipped out to clients had to be pre-formted at snap under a certain os version depending on were the starting block was on the current snap the client had.

Again, the raid would build ok and issue would not show up until after the snap was rebooted the first time, then this error would show up and if you tried to force a mount it would cause snap to panic.

We never found a way to resolve it once it was in this condition and therefore data recovery was the clients only option.

I hope this makes sense. I found this in my notes. I am still looking for the actuall official document which explains this in full detail which was given to the techs.

Douglas

[Phoenix32]

Good info...

[blue68f100]

Great Info.

So the fix is when you replace a drive. Do a "co de fromat xxxxx /reinit" to all drives.

Or does all drives need to be totally clean and let the snap do it from scratch?

[snap-tech]

NO -- ONLY IF YOU DON'T CARE ABOUT WHAT IS ON THE SERVER AND JUST WANT TO REDO THE RAID.

only do a /reinit if you don't care about the currant data.

we had to use the /nocore and format the drive under a specific os so that the starting blocks were correct before shipping the drive out to clients.

Douglas

[blue68f100]

Let's say we just picked up a 4100 off ebay. The Info log shows the miss match in the drives. And the unit shows a drive failure. Would replacing the failed drive cause the system not to build?

Would you recommend reformating all drives to correct the inherit problem while the drives are clean?

[jontz]

Ummm...that's kind of a big ass bug for a "falut tolerant" RAID config, but good to know none the less.

I guess I don't totally get it though...why wouldn't replacing the drive with the different start block cure the problem? If a drive is mis-matched, pulled, and replaced with a new one, wouldn't the snap just format the drive with the "current" OS start block and go on it's merry way? Or are you saying that if a snap is set up with a given os, upgraded to a new OS, and the a drive fails, that when the new drive is formatted it won't match because it will be formatted with the "new" OS paramaters, which wouldn't match up with the other 3 drives formatted with the "old" paramaters....

[snap-tech]

Ok, found some more info from my days at snap.

If the snap was originally formated under a 2x version then upgraded to an 3x, and a drive fails -- the replacement drive must be formated under a 2x version.

If snap was original formated under 2x, but after upgrading to 3x the drives were re-formted with the /reinit option then a replacement drive would not need to be formated under 2x first.

The changes between 2x and 3x when it comes to starting block is 16 sectors only.


If the drives were inserted or formated while server was running a 3x version then there is no concern because they all have a 3x starting block location.


If you snap was originally formated under 2x then upgraded to 3x and you have a drive that failed you can remove all drives -- downgrade the server using a 2x version of the .sup file. insert the new drive and format it. Then remove it and re-upgrade server back to a 3x version. reconnect the drives, power up unit and make sure all looks as it did. Then shut down and add the new drive and insert it into raid and all should be ok.

It is ok to have a new drive that is larger than the rest, the starting block will not match the other drives(that can look confusing but is ok - as long as it was formated under the os version as the other drives were originally formated under)

I hope this makes sense. If not--post specific questions and I will try and re-explain.

Douglas snap-tech

[Phoenix32]

What about 3x to 4x? Is this required between them as well?

[snap-tech]

NO. There was no change between 3x and 4x for the starting block.

Douglas Snap-Tech

[snap-tech]

Just wanted to update everyone on this snap.

Mike shipped the drive that contained all 4 images of snap drives to me as well as a drive to put data on. I am in process of destriping raid and have 15GB so far. So it will be another successfull recovery from a snap.

Mike will be also posting something later after I give him the list of files recovered.

Douglas Snap-Tech

[snap-tech]

I was able to do a full recovery on this 4100 and will be shipping out the recovered data on Monday.

mvastola12 should be posting something this weekend to confirm recovery.

Douglas Snap-Tech

[jontz]

Bravo!