Snap 410 error "XFS:xfs_force_shutdown"

[KOBadmin]

Hi everyone,
I am new here and unsure if anyone can help. I know that there are not a lot of 410s here but every night after about an hour in no disk activity this error message comes up

"XFS:xfs_force_shutdown(lvm(58,0),0x8) called from line 4046 of file xfs_bmap.c. Return address = 0x401adbb2<1>Corruption of in-memory data detected. Shutting down filesystem: lvm(58,0)<1>Please umount the filesystem, and rectify the problem(s)"

and no one can access the snap server until it is rebooted. Can anyone Help?

[blue68f100]

I would try a couple of things. First and foremost BACKUP YOUR DATA.

I would take the time and run SpinRite on all of the HD's. To run Spinrite you will need to remove the HD ( number them, so you can put them back in the correct order) to test to a pc. Spinrite runs at the controller level so it will not cause any problem related to file structure, except repair damage. Runing at the controller level it does not inturpet data like most HD utilities do not cause problems. This will make sure you do not have a HD generating the error.

I would power down and re-seat the ram.

If that does not do it I would re load the OS using the recovery console.

The error to me indicates a hardware problem the reason for testing the HD and re-seating the ram. I think you may have a bad ram chip, but these steps need to be done to eliminate the other hardware.

[KOBadmin]

Thanks for the advice, already backed up the data. Will reset the RAM tonight and see what happens, will also get and run Spinrite like you suggested (most likely will have to wait until the weekend) any idea on how long Spinrite will take to run? (4x250GB drives).
Oh forget to mention errors started after we had a power outage here, was plugged into a ups but the ups has a dead battery (less then a year old).

[KOBadmin]

The server throws the error at the same time every night, 11/01 17:59:55, 10/31 17:59:45, 10/30 18:56:22, 10/30 17:59:53, 10/29 17:59:12. My boss here on the 30th and restarted the sever and was fine for about and hour. Most users leave between 4:30 and 5:00, so it happens when the server is idle. Any ideas as to what would be running then?

[blue68f100]

Did you have/purchase the AV package, SnapShots for backup?

Are you syncing time with a NTP server?

What OS version are you running?

Do you have any Vista, MAC OSX or unix/linux users? There are known bugs with older version (and current) when it comes to these os's.

Is access via SMB or FTP? There are problems with the GOS when it comes to releasing users after they shutdown without logging out.

Are you use AD Support? if so check to see if doing something. I'm not up on AD support at all, someone else will have to help you there.

If you want me to look at the log send a complete 24 hr span starting with the startup sequence. Save as html code so I can view it with a browser. Sometime a different pair of eyes help.

[blue68f100]

Originally Posted by KOBadmin

Thanks for the advice, already backed up the data. Will reset the RAM tonight and see what happens, will also get and run Spinrite like you suggested (most likely will have to wait until the weekend) any idea on how long Spinrite will take to run? (4x250GB drives).
Oh forget to mention errors started after we had a power outage here, was plugged into a ups but the ups has a dead battery (less then a year old).

Dead battery in less than a year, ouch, or did someone plug other hardware into it and over loaded the unit. They should go through a load test ever couple of weeks, min 1/mo. Need to add it to your already thin duties.

The power outage will definately cause file corruption and problems, SpinRite can help there. Doing a re-install of the OS may correct the problem. But in most cases I find it better to do a clean install so all files are replaced. Most support auto test of batteries that can send results and warnings to a SNMP server. I use OPManager by AdventNet, free if you want to monitor 10 devices or less. Supports email and pager alerts.

Spinrite runs different on all pc's so it's hard to say how long it will take. I've seen it take 8+hrs and as little as 2 hrs/hd. It also depends on how full the disk are.

[KOBadmin]

Not sure of the AV package, not using snapshots.

Not syncing time with a NTP server.

OS version: GuardianOS 4.4.049 SP3

All clients are XP machines (mostly SP2 some SP3 did sneak in)

Being access with SMB in P2P, no AD.

[sitin]

I have a similar issue with SNAP Server 4400, Guardian OS 3.2.026. If I reboot or recover the server, I'm able to browse the shares for only 5-120 minutes. Then all the contents of the shares disappear and I get the same error on the log file.

I've tried disaster recovery to no avail. I'm ready to reinstall the OS. Does anyone know the steps to reinstall the Guardian OD? I have the gsu file.

Thanks,
Sitin

[blue68f100]

Your problem also is hardware related, maybe thermal related. Over time thermal cycles loosen componets and cables. Opening one up and removing and reinstalling cleans the connections.

Since the 4400 is an older unit, I would do a through cleaning, re-seat the ram, re-apply thermal paste on the cpu/heatsink. Also check and make sure the fans are in good working order. If you can get access to the recovery console there is an option to reinstall and clean install. This will auto load if there is a problem booting. If you do not have access to it inticates toward hardware. The watchdog bios will try to boot from another HD if it fails to get a good boot signal after a set period of time, normally is 5min. Failing power supplies can cause havic too. If you have another unit you can use for parts, would be nice to eliminate the ram and PS.

The procedure to involk the recovery console is posted in one of the Guardian threads. Use the search feature on this forum to locate it. I would only do it after eliminating the hardware.

[KOBadmin]

Believe that the problem is finally resolved. What was happening was that there were too many users backing up to the snap at one time causing it run out of memory. Change the times on a couple of the backup and upgraded the memory too 2GBs (runs like a champ now).

Also did run Spinrite on all the disk and they came up clean.

Thanks for all your help.