DisasterRecovery - In Progress ( For 3 Days? )

[DarkSideOfFun]

I have a SnapServer 520 ( GuardianOS 5.0.133 ) I tried to run the disaster recovery on 3/7. The 'System' Image ran that day but the 'Main_Volume' is still showing - In Progress this morning (3/10). Is this normal?? The snap server has actaully need restarted due to a power outage in my area yesterday. My Main_Volume is a RAID 5 array (2.04 TB, 1.35 TB free). Thank you for any help.

-Ed

[blue68f100]

Please do not double post.... these errors you are getting are related.

Disaster recovery does not take long to run, so I would guess an error has accoured. Normally just a few minutes. Did you loose power while disk activity was going on? If your not using a UPS you should be. Servers do not like to be powered cycled during rw. I would interrupt the process. Re boot the server, if it drops in to the recovery console, reinstall the OS. See if that corrects the problem.

[DarkSideOfFun]

Sorry about the double post.

No I did not loose power while I was writing. I have 3 BBUs to cover all of my rack equipment. This issue happened a couple of days before the power outage.

How do I interrupt the process? I have tried to do a gracefull shutdown & restart and it still shows that it is in progress.

BTW.. the memory is upgraded to 2 GB.

The SnapServer is up and running. Able to get into it, access files, etc.

Sorry about the "Rookie" level of questions. I am pretty good with servers & networking, but i'm somewhat new with SnapServer, as you can already tell.

[blue68f100]

Snaps are a little different from most servers since these are specility units. They do one thing and do it good. What makes these nice is the OS is so advanced.

Will the server shut down or just hang due to the recovery process? hang by your response.

You will need to know some basic linux to determine what process it is. Using the ps to list what process are running. Once you indentify the process you just use the kill # to stop it.

If not your only options would be to unplug it. It then should boot into the recovery console if it detects a problem. At that point It may be easier to reload the OS. But I would not the first time around.

A lot of time programs hang when a process can not access a directory or it's busy and can not connect.

[DarkSideOfFun]

That is one of the reasons why I got a snapserver. I normally just used disk arrays in my servers to back things up, but wanted to learn these and heard nothing but great things about them.

If I hit the power button on the front the snapserver the snapserver will go thru the shutdown and shutdown normally. I can shutdown/restart from the web with no problems either.

No matter how many times it is shutdown or rebooted, it still shows in progress.. ( and also still not enough space to upgrade )

Image Last Creation Date Last Recovery Date
System 03/10/2009 10:03:05 AM Not Recovered
Main_Volume In progress Not Recovered

## Backing up File system ACLs on /hd/vol_mnt0 (Sat Mar 7 11:46:44 EST 2009) ##

sh-3.1$ ps
PID TTY TIME CMD
26073 ttyp0 00:00:00 sh
26251 ttyp0 00:00:00 ps

I also provided the error log showing the complete and lastest boot cycle.

[blue68f100]

Indication of a fan that may be going out....
Qhwmon: CPU Fan3b is slow (5400), try to kick start it.

Eithernet port not configured correctly.....
bonding: bond0: link status definitely up for interface eth0. kernel 10-Mar 4:30:08 PM
tg3: eth0: Flow control is on for TX and on for RX. kernel 10-Mar 4:30:08 PM
tg3: eth0: Link is up at 1000 Mbps, full duplex. kernel 10-Mar 4:30:08 PM
bonding: bond0: making interface eth1 the new active one. kernel 10-Mar 4:30:08 PM
bonding: bond0: link status definitely up for interface eth1. kernel 10-Mar 4:30:08 PM
tg3: eth1: Flow control is on for TX and on for RX. kernel 10-Mar 4:30:08 PM
tg3: eth1: Link is up at 1000 Mbps, full duplex. kernel 10-Mar 4:30:08 PM
W
bonding: bond0: Error: found a client with no channel in the client's hash
table kernel 10-Mar 4:30:08 PM
W
bonding: bond0: Error: found a client with no channel in the client's hash
table kernel 10-Mar 4:30:08 PM
bonding: bond0: now running without any active interface ! kernel 10-Mar 4:30:08 PM
device eth1 left promiscuous mode kernel 10-Mar 4:30:08 PM
bonding: bond0: link status definitely down for interface eth1, disabling it kernel 10-Mar 4:30:08 PM
tg3: eth1: Link is down. kernel 10-Mar 4:30:08 PM
device eth1 entered promiscuous mode kernel 10-Mar 4:30:08 PM
bonding: bond0: making interface eth1 the new active one. kernel 10-Mar 4:30:08 PM
bonding: bond0: link status definitely down for interface eth0, disabling it kernel 10-Mar 4:30:08 PM
tg3: eth0: Link is down. kernel 10-Mar 4:30:08 PM
bonding: bond0: link status definitely up for interface eth1. kernel 10-Mar 4:30:08 PM
tg3: eth1: Flow control is on for TX and on for RX. kernel 10-Mar 4:30:07 PM
tg3: eth1: Link is up at 1000 Mbps, full duplex. kernel 10-Mar 4:30:07 PM
bonding: bond0: making interface eth0 the new active one. kernel 10-Mar 4:30:07 PM
bonding: bond0: link status definitely up for interface eth0. kernel 10-Mar 4:30:07 PM
tg3: eth0: Flow control is on for TX and on for RX. kernel 10-Mar 4:30:07 PM
tg3: eth0: Link is up at 1000 Mbps, full duplex. kernel 10-Mar 4:30:07 PM
bonding: bond0: enslaving eth1 as an active interface with a down link. kernel 10-Mar 4:30:07 PM
bonding: bond0: enslaving eth0 as an active interface with a down link. kernel 10-Mar 4:30:07 PM
bonding: MII link monitoring set to 100 ms kernel 10-Mar 4:30:07 PM
I
bonding: In ALB mode you might experience client disconnections upon
reconnection of a link if the bonding module updelay parameter (0 msec)
is incompatible with the forwarding delay time of the switch kernel 10-Mar 4:30:07 PM
Ethernet Channel Bonding Driver: v3.0.1 (January 9, 2006)

Only real error that may indicate whats happening is this one...
I SnapExtension Framework shutdown failed snap_extension 10-Mar 4:26:39 PM
I snap_extension 10-Mar 4:26:39 PM
I Shutting down SnapExFramework: snap_extension 10-Mar 4:26:39 PM

The last one is probably were the problem arises from... This should beable to be stopped/disabled from the snap extensions...

Go to disaster recovery and terminate the scheduling.... should stop the process... delete previous files from server. If space has been filled and needs to overwrite may contribute to your original problem.

Now if the error is being generated due to ethernet connection problem , correct that.

The only other thing I can suggest is revolk the recovery console (touch /nopivot from debug) and reinstall or upgrade the OS... I know a clean install will work but you will loose everything on the server.

If you have Support with Overland, may ring their bell. There is a newer GOS out 5.1.. The reports are that areas of problem (ram management has been re-written and the OS now runs faster than previous model.

Are you have a Snap 10, 30 or 50 connected, saw indications that you may..

I've run out of ideas. I do believe the problems lies with the scheduling or does not have space to write the file, but it say it's looking for the file. Use the built in help to locate where the files are saved to and transfer off of server or delete them all may fix the problem.

[DarkSideOfFun]

First.. I do appreciate all of your help and your time.

The Server is on the network, and never had an issue with either network connection.

http://scottnas/cadmin/debug.cgi
Then I get a box that says "Command" & Go

It does say "SnapServer Debug Console" on top, this is what I get when I type help:

GNU bash, version 3.1.17(1)-release (i686-pc-linux-gnu)
These shell commands are defined internally. Type `help' to see this list.
Type `help name' to find out more about the function `name'.
Use `info bash' to find out more about the shell in general.
Use `man -k' or `info' to find out more about commands not in this list.

I do have - GuardianOS_5_1_041_full_OSImage. This is the OS that I wanted to Reinstall, upgrade to.

I'm beginning to think an OS reinstall might solve a lot of my little issues.

I do not see where or now to launch the recovery console like you mentioned.

I do not have any of the external SanBlocks connected. Its just the SnapServer 520.

This is what I have under the SnapExtension Tab.. I do not see anything for scheduling... and most are already disabled..
SnapExtension Status License
BakBone NetVault Disabled Licensed.
CA Antivirus Disabled Licensed.
iSCSI Enabled Licensed.
Snap Server Manager - License required.
Snapshots - Licensed.

[blue68f100]

from the debug console, then enter "touch /nopoviot" the server should reboot and the recovery console will launch. Then you can do your OS install. From this point you alnoy have the OS shell loaded and no services or programs.

in the past the debug window some time would not give you any indications that the cmd was executed. But it was in most cases it would give an error if it failed.

If it does not launch/reboot. you may needt to be loged in as admin and root access. To get to that, type osshell, then su root, then enter root password. then try the touch cmd.