All shares show unavailable on 4000

[Sifter]

private - Unavailable: disk is offline.
public - Unavailable: disk is offline.

When I navigate to disk utilities and view the disk status I see:

RAID5 - Large data protection disk
OK 710,973 Unknown 58,129

When I look at the log, I see the following:

S System Initialization : Server v4.0.860
Build Date: Mar 2 2005 17:50:35
Boot Count: 16 System 4/22/2008 2:50:01 PM
F Call Stack :
$00105131
$002D1454
$002D8496
$002DF71A
$00101700
$0011807A
$0012D79F
$0012D713
$00107148
$00107C5A
$00108CFF
$00238322
$00238514
$002393BA
$0010413C
$001041C3
================================
System 4/22/2008 2:45:46 PM
F PANIC : Maximum yield time exceeded at $00101554 System 4/22/2008 2:45:46 PM

Any ideas?

[Sifter]

And here is what shows after a reboot:

I File System Check : ** Phase 1 - Check blocks and sizes Disk 60000
RAID 5 4/22/2008 3:14:09 PM
W File System Check : partition is NOT clean. Disk 60000
RAID 5 4/22/2008 3:14:06 PM
I File System Check : Executing fsck /dev/rraid0 /fix Disk 60000
RAID 5 4/22/2008 3:14:06 PM
I File System : Opened FDB for device 0x1000E Disk (Priv) 4/22/2008 3:14:06 PM
I File System Check : Cleanup completed... Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : 743 files, 1248 used, 2789 free (0 frags, 2789 blocks, 0.0%% fragmentation) Disk (Priv) 4/22/2008 3:14:03 PM
W File System Check : Clean flag not set in superblock (Fixed) Disk (Priv) 4/22/2008 3:14:03 PM
W File System Check : Modified flag set in superblock (Fixed) Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 5 - Check cylinder groups Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 4b - Check backlinks Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 4 - Check reference counts Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 3 - Check connectivity Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 2 - Check pathnames Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 1b - Rescan for more duplicate blocks Disk (Priv) 4/22/2008 3:14:03 PM
I File System Check : ** Phase 1 - Check blocks and sizes Disk (Priv) 4/22/2008 3:14:03 PM
W File System Check : partition is NOT clean. Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : Executing fsck /dev/ride1g /fix /fixfatal Disk (Priv) 4/22/2008 3:14:02 PM
I File System : Opened FDB for device 0x10006 Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : Cleanup completed... Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ***** File system was modified ***** Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : 721 files, 1226 used, 2811 free (0 frags, 2811 blocks, 0.0%% fragmentation) Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Clean flag not set in superblock (Fixed) Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Modified flag set in superblock (Fixed) Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Summary information bad (Salvaged) Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Blk(s) missing in bit maps (Salvaged) Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 5 - Check cylinder groups Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 4b - Check backlinks Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Link count Dir I=2480 Owner= Mode=41200
/dev/ride0g: Size=1024 Count 4 - should be 3 (Adjusted) Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Zero Length Dir I=14 Owner= Mode=41200
/dev/ride0g: Size=0 (Cleared) Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 4 - Check reference counts Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 3 - Check connectivity Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 2 - Check pathnames Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 1b - Rescan for more duplicate blocks Disk (Priv) 4/22/2008 3:14:02 PM
W File System Check : Incorrect block count I=14 (16 Should be 0) (Corrected) Disk (Priv) 4/22/2008 3:14:02 PM
I File System Check : ** Phase 1 - Check blocks and sizes Disk (Priv) 4/22/2008 3:14:01 PM
W File System Check : partition is NOT clean. Disk (Priv) 4/22/2008 3:14:01 PM
I File System Check : Executing fsck /dev/ride0g /fix /fixfatal Disk (Priv) 4/22/2008 3:14:01 PM
I INIT: Setting IP address to 10.0.0.3 Network 4/22/2008 3:13:54 PM

[Sifter]

Ok it looks like its checking itself now:

RAID5 - Large data protection disk
Checking. 5% complete 710,973 Unknown Unknown

[blue68f100]

Try cold booting the snap. You may have a HD or hardware failure. The reboot may correct the problem and/or bring the problem to the fore front.

[Sifter]

The green LED is Double flashing, which I guess is good since it means the System is running a performing a check operation. Still only shows the disk check at 5%. How long does this normally take with 4 drives, raid 5, 250gig each?

[Sifter]

Ok now its making progress...85% complete.

W File System Check : Free blk count(s) wrong in superblk (Salvaged) Disk 60000
RAID 5 4/22/2008 4:07:56 PM
I File System Check : ** Phase 5 - Check cylinder groups Disk 60000
RAID 5 4/22/2008 4:07:56 PM
I File System Check : ** Phase 4b - Check backlinks Disk 60000
RAID 5 4/22/2008 4:07:39 PM
I File System Check : ** Phase 4 - Check reference counts Disk 60000
RAID 5 4/22/2008 4:07:26 PM
I File System Check : ** Phase 3 - Check connectivity Disk 60000
RAID 5 4/22/2008 4:07:26 PM
I File System Check : ** Phase 2 - Check pathnames Disk 60000
RAID 5 4/22/2008 4:06:56 PM
I File System Check : ** Phase 1b - Rescan for more duplicate blocks Disk 60000
RAID 5 4/22/2008 4:06:56 PM
I System Database : SDB has been written to flash at 2008/04/22 16:03:54. System 4/22/2008 4:03:54 PM
I File System Check : ** Phase 1 - Check blocks and sizes Disk 60000
RAID 5 4/22/2008 3:14:09 PM
W File System Check : partition is NOT clean. Disk 60000
RAID 5 4/22/2008 3:14:06 PM

[Sifter]

RAID5 - Large data protection disk
OK Disk successfully repaired, see log for details. 710,973 Unknown 58,130

Should I just power it down now? And bring it back up after a cold boot?

[Sifter]

Ok it finally powered down. When starting back up I get the following in the logs... it looks like its just scanning the drives again. Back to 5% complete.



I File System Check : ** Phase 1 - Check blocks and sizes Disk 60000
RAID 5 4/22/2008 4:49:47 PM
W File System Check : partition is NOT clean. Disk 60000
RAID 5 4/22/2008 4:49:43 PM
I File System Check : Executing fsck /dev/rraid0 /fix Disk 60000
RAID 5 4/22/2008 4:49:43 PM
I File System : Opened FDB for device 0x1000E Disk (Priv) 4/22/2008 4:49:43 PM
I File System Check : Cleanup completed... Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : 767 files, 1572 used, 2465 free (0 frags, 2465 blocks, 0.0%% fragmentation) Disk (Priv) 4/22/2008 4:49:41 PM
W File System Check : Clean flag not set in superblock (Fixed) Disk (Priv) 4/22/2008 4:49:41 PM
W File System Check : Modified flag set in superblock (Fixed) Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 5 - Check cylinder groups Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 4b - Check backlinks Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 4 - Check reference counts Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 3 - Check connectivity Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 2 - Check pathnames Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 1b - Rescan for more duplicate blocks Disk (Priv) 4/22/2008 4:49:41 PM
I File System Check : ** Phase 1 - Check blocks and sizes Disk (Priv) 4/22/2008 4:49:41 PM
W File System Check : partition is NOT clean. Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : Executing fsck /dev/ride1g /fix /fixfatal Disk (Priv) 4/22/2008 4:49:40 PM
I File System : Opened FDB for device 0x10006 Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : Cleanup completed... Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : 724 files, 1256 used, 2781 free (0 frags, 2781 blocks, 0.0%% fragmentation) Disk (Priv) 4/22/2008 4:49:40 PM
W File System Check : Clean flag not set in superblock (Fixed) Disk (Priv) 4/22/2008 4:49:40 PM
W File System Check : Modified flag set in superblock (Fixed) Disk (Priv) 4/22/2008 4:49:40 PM
W File System Check : Free blk count(s) wrong in superblk (Salvaged) Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 5 - Check cylinder groups Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 4b - Check backlinks Disk (Priv) 4/22/2008 4:49:40 PM
W File System Check : Unref File I=15 Owner= Mode=101204
/dev/ride0g: Size=106496 (Reconnected) Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 4 - Check reference counts Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 3 - Check connectivity Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 2 - Check pathnames Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 1b - Rescan for more duplicate blocks Disk (Priv) 4/22/2008 4:49:40 PM
I File System Check : ** Phase 1 - Check blocks and sizes Disk (Priv) 4/22/2008 4:49:39 PM
W File System Check : partition is NOT clean. Disk (Priv) 4/22/2008 4:49:39 PM
I File System Check : Executing fsck /dev/ride0g /fix /fixfatal Disk (Priv) 4/22/2008 4:49:39 PM
I INIT: Setting IP address to 10.0.0.3 Network 4/22/2008 4:49:32 PM
S System Initialization : Server v4.0.860
Build Date: Mar 2 2005 17:50:35
Boot Count: 18 System 4/22/2008 4:49:32 PM

[Phoenix32]

Originally Posted by Sifter

How long does this normally take with 4 drives, raid 5, 250gig each?

20 Hours for a RAID 5 rebuild on a 4000 w/4x250...

[Sifter]

ouch, ok see you fella's in the mornin.

[blue68f100]

Andy, I think he may have a PS problem and maybe HD issues. Or is this a cable thing compounding the problem.

Are your IDE cables the colored ones with a lead cut?

1000E (drive 2) & 10006 (drive 1) are the drives on the same controller.

The 5% are is where all of the tables are kept on the HD. Problems here generated huge problems. I would run Spinrite on all of the drives. Start in maintance mode, it will switch to recovery if it detects a problem. You may have 1 drive that is causing all of the problems.

Does this bring back old memorides.

[Phoenix32]

Yeah, I as thinking the same thing. I am wondering about a drive going bad...

Let's see what happens with the rebuild...

[Sifter]

Well everything seems to be going ok...

Rebuilding backup disk. 86% complete 701,251 391,742 58,176

[Sifter]

Ok guys:

RAID5 - Large data protection disk
OK 701,251 410,831 58,062

Last entry in the log:

I File System : Logical set synchronization done on device 60000 Disk 60000
RAID 5 4/23/2008 1:50:28 PM

[Phoenix32]

This is part of the reason I went to Guardian units and RAID edition drives.

[blue68f100]

Now shutdown and cold boot again and see if drops into check disk & resync. If it does this every time it's useless.

Andy is this the ribbon thing?

[Phoenix32]

Originally Posted by blue68f100

Andy is this the ribbon thing?

No, I don't think so...

I more think this is a random failure, as is common with the SNAP OS units. After what I learned from using the 8 or so unts I had, I would never using anything except RAID Edition drives in them. They are too prone to drop outs and timing overruns otherwise.

In fact, this is what I meant by my previous comment. The SNAP OS units are too sensitive when it comes to larger drives. Timing is too tight and too prone to random events causing RAID 5 fallout.

IMO, in a 4000 with drives larger than 80 GB, it is WD RAID Edition drives or nothing... And what you are seeing here is exactly the why.

[Sifter]

Well here is the cold boot, everything seems to have come up ok:

I SSSirect Connect Started Unknown 4/24/2008 7:51:30 AM
I Jrc: exiting Unknown 4/24/2008 7:51:30 AM
I Jrc.execute() starting Unknown 4/24/2008 7:51:30 AM
I Jrc created Unknown 4/24/2008 7:51:30 AM
I SnapOS.init() executed Unknown 4/24/2008 7:51:30 AM
I SnapExtension Framework 1.2 initialized Unknown 4/24/2008 7:51:29 AM
I SSS: service started Unknown 4/24/2008 7:51:29 AM
I /Java$/Jeode/etc/hosts: 10.0.0.3 snap4000 Unknown 4/24/2008 7:51:27 AM
I /Java$/Jeode/etc/hosts: 127.0.0.1 localhost Unknown 4/24/2008 7:51:27 AM
I done Unknown 4/24/2008 7:51:26 AM
I building info for serversync.jar Unknown 4/24/2008 7:51:24 AM
I done Unknown 4/24/2008 7:51:24 AM
I building info for hello.jar Unknown 4/24/2008 7:51:19 AM
I installed security manager Unknown 4/24/2008 7:51:19 AM
I Java: Native methods linked OK Unknown 4/24/2008 7:51:18 AM
I Jrc.main() starting Unknown 4/24/2008 7:51:17 AM
I System Initialization : Initialization Complete! Memory to be released: 129152112 bytes. System 4/24/2008 7:51:17 AM
I File System : Opened FDB for device 0x60000 Disk 60000
RAID 5 4/24/2008 7:51:11 AM
I File System : Quota management active on device 60000 Disk 60000
RAID 5 4/24/2008 7:51:11 AM
I File System Check : partition is clean. Disk 60000
RAID 5 4/24/2008 7:51:11 AM
I File System Check : Executing fsck /dev/rraid0 /fix Disk 60000
RAID 5 4/24/2008 7:51:11 AM
I File System : Opened FDB for device 0x1000E Disk (Priv) 4/24/2008 7:51:11 AM
I File System Check : partition is clean. Disk (Priv) 4/24/2008 7:51:11 AM
I File System Check : Executing fsck /dev/ride1g /fix /fixfatal Disk (Priv) 4/24/2008 7:51:11 AM
I File System : Opened FDB for device 0x10006 Disk (Priv) 4/24/2008 7:51:11 AM
I File System Check : partition is clean. Disk (Priv) 4/24/2008 7:51:11 AM
I File System Check : Executing fsck /dev/ride0g /fix /fixfatal Disk (Priv) 4/24/2008 7:51:11 AM
I INIT: Setting IP address to 10.0.0.3 Network 4/24/2008 7:51:04 AM
S System Initialization : Server v4.0.860
Build Date: Mar 2 2005 17:50:35
Boot Count: 20 System 4/24/2008 7:51:04 AM

[Phoenix32]

Good deal...

Keep backups...