4100/705N System light blinking rapidly

[rpmurray]

I was just knocked off my Dell 705N (SNAP 4100). When I checked the front panel, the system LED was blinking rapidly (not the normal heartbeat blinks) but the other LEDS seemed to be OK. None of the drive LEDs was blinking, amber or green, so it doesn't look like a hard drive failure. I powered it down.

Anyone with any ideas as to what might have happened? Is it safe to turn it back on?

[rpmurray]

Well, according to some info I found, the system LED flashing like this indicates it's panicked or crashed. Assuming the drives are OK, it looks like a panic, and according to the docs, that means either hardware or software.

Anyone had any experience with this situation? I've pulled it from the rack and am thinking of turning it back on. I checked the case and it doesn't look like it overheated. The only thing is that I don't know if it's going to do a disk check. Nothing was in use at the time, this one is used strictly for nightly backups. It's just that it would take five hours to run and I don't want to leave it running overnight until I can see if it comes back up OK.

[blue68f100]

Go ahead and power it back up. A drive probably failed and put it in panic mode. Give it extra time to come up, it should indicate on the front panel.

Now here the catcher. If you have upgraded the OS (v3 to 4) since the raid 5 was built you will have problems in replacing the HD. Snap changed the way they calculated the starting point. So you have to use the original OS version that was used to built the array. You should be able to access the data in degraded mode.

While you have it open make sure the mod had been done. Info on top of forum Sticky Attn 4100 users.

[rpmurray]

Just to keep everyone up to date, and because this might be helpful to someone in the future, here's whats been happening.

I turned on the unit this morning at 7:48 (damn thing is showing 8:48 so DST is not set correctly). It came up like normal, the System LED is doing it's normal heartbeat indication. It is running a disk check, but it seems to be running it a lot faster, it's 8:32 and it's already 80 percent complete. So far I don't see any indications of problems. None of the drive LEDs is amber, which is probably explained by what the log is showing:

12/17/2007 23:06:58 30 D SYS | Begin SDB Backup.
12/17/2007 23:06:58 30 I SYS | System Database : System Configuration Backup has begun
12/17/2007 23:06:58 30 D SYS | Launched SDB Backup.
12/17/2007 23:06:58 30 I SYS | System Database : System Configuration Backup has completed successfully
12/18/2007 15:10:57 30 D SYS | Stack reference = 0x007A65F0
12/18/2007 15:10:57 30 D SYS | Detected a unit with 256M of memory.
12/18/2007 15:10:57 30 D SYS | Dumping 0x10000000 bytes of memory from address 0x0, file offset 0x200 ...
12/18/2007 15:10:57 30 D SYS | Done.
12/18/2007 15:10:57 30 F SYS | PANIC : m_free: freeing free mbuf
12/18/2007 15:10:57 30 F SYS | Call Stack :
$00150969
$00150A40
$0015B045
$0015C148
$001585DE
$0014FC72
================================

It looks like it had some kind of panic dealing with memory yesterday.

Right now I'm keeping my fingers crossed and hoping it doesn't happen again.

The unit has the mod mentioned in the sticky, it's also a 70715048-006A version.

Details: This is actually a Dell 705N that's been modified to run the Snap server software version 3.4.805. It currently has 4 160GB Seagate drives in it, and 256MB of memory.

[blue68f100]

Panics can be hardware. I would check to see if the cooling fan is running on the CPU. Over heating a cpu, they don't like that.

[rpmurray]

I have the cover off while it's resyncing and the cooling fan comes on every so often. It's 70% complete in the resyncing and I'm copying off some non-critical but still nice to have stuff as it does it. Seems to be taking about 3 minutes for each percent of the resync, which means it will take 5 hours total, which is what it's usually taken when it needs to resync.

The log entries still confuse me.

[rpmurray]

So far, since it got finished resyncing yesterday, it's been running fine. No panics or other errors in the log.

I'm kind of hoping this was just a one-time glitch.

[Phoenix32]

Memory error...

Note: ALL MEMORY get random errors due to naturally occurring radioactive isotopes and high energy cosmic rays...

[rpmurray]

Well, just had the same thing happen with another 705N(4100). Except this time for another reason. The log shows:

12/30/2007 0:07:37 25 I L01 | File System : Extended Rights Backup for device 0x60000 has begun
12/30/2007 1:56:33 25 I L01 | File System : Extended Rights Backup for device 0x60000 has completed successfully
12/30/2007 15:58:19 25 E AFP | AppleTalk : Reinitializing Corrupted Desktop DB: UNDCHK. Bad offset <[1]>. val 33279
12/30/2007 15:58:19 25 F SYS | PANIC : AFP: Bad DBRefNum in DTRebuildDP
12/30/2007 15:58:19 25 F SYS | Call Stack :
$001E1362
$001F0606
$001E0BC2
$001ED98A
$001DB926
$001DCD87
$001DCB13
$001DC68A
$0010413C
$001041C3
================================

I had to turn it off yesterday, and now that it's back on it's in the middle of doing the disk check.

[rpmurray]

Update. It took about three and a half hours to do the disk check and is now starting on the resync. This one took a lot longer than the other one because it has a lot more data on it.

I'm copying off the one folder that I don't have a current backup of, about a gig and a half. Then I'll just let it run and see if it happens again.

Anyone seen this happen before? I got a panic in both cases, but for different reasons. It's kind of odd. I had one 705n that ran non-stop for five years (except when I had to shut it down when we had extended power outages) before it croaked, but it never had a panic. Of these last two, the first ran for a little over a year before the panic, and the second ran for a bit over two years.

I'm wondering if age is starting to have it's effect on the components.

Edit: At the time I first installed these units, both had brand new hard drives, and the correct 256MB memory (Kingston). The first one is running SNAP OS 3.4.805 and the other is running the Dell version 3.4.790. Same hardware (2.2.1) and BIOS (2.4.437). Server IDs are both in the 51xxxx.

[blue68f100]

Age is a problem with most all server hardware. I know of several companies that buy new equipment every 3-5 yrs, depending on support. Their stance is that after that period, hardware can start failing.

I have seen panics, plenty times before. Some has been caused by drive failure, bad ram, bad cooling fan overheating, bad MB. So you can take your pick. Heat can cause all kinds of problems. Power Supplies can wrech havic too.

[netwraith]

Originally Posted by rpmurray

I was just knocked off my Dell 705N (SNAP 4100). When I checked the front panel, the system LED was blinking rapidly (not the normal heartbeat blinks) but the other LEDS seemed to be OK. None of the drive LEDs was blinking, amber or green, so it doesn't look like a hard drive failure. I powered it down.

Anyone with any ideas as to what might have happened? Is it safe to turn it back on?

This is just one of the problems with SnapOS... The filesystem code is rudimentary and is prone to any number of foibles. I have stopped trying to use these units for anything other than scratch space or online backups. They are just too prone to problems when you try to save and then delete a lot of files in a short period. They, also, have maddening periods of a few seconds when they seem to stop working. I think that this is a buffering issue with the filesystem as well. All my online systems are now Guardian units or FreeNAS.

It should be safe to turn back on. If there is something wrong, it will try to recover. If it fails, there was likely little enough to be done anyway. (Aside from getting another unit and migrating drives).