Snap Server 12000

[achapman]

I've just put some notes about this on the wiki.

There doesn't seem to be too much info on these specific units (not too surprisingly) in the forum, so I thought it might help someone out. I just got one to play with for 100GBP (Apparently they were over 12000GBP when new!). They are definitely a step up from a 4100.

[bitor]

achapman, you can use 168-pin 1GB of memory ECC PC-133 in this unit. My name brand is SMART. FSB133, CPU P3 Socket 370. I strongly suggest you check the hot swappable power supply pot fans and all others to make sure they are working. Intel Pro 1000 T eithernet card. What else do you need to know about it?

FYI, Production for the Snap12000 was ONLY SnapOS. There is none sold to the general public with GOS(GuardianOS).

bitor

Originally Posted by achapman

I've just put some notes about this on the wiki.

There doesn't seem to be too much info on these specific units (not too surprisingly) in the forum, so I thought it might help someone out. I just got one to play with for 100GBP (Apparently they were over 12000GBP when new!). They are definitely a step up from a 4100.

[achapman]

The thing I'm not clear about is the storage limit. I understand that on a 4000 you can't go above 250Gb drives with a RAID5 array as SnapOS can't handle RAID arrays above 1Tb/1.2Tb. My reading of the articles in the forum seems to indicate that the limit relates to the largest array SnapOS can handle rather than the total usable storage placed in the box. The 12000 could have three 4drive RAID5 arrays or two six drive RAID5 arrays. If it had the latter the combined usable storage with 160Gb drives would be around 1.3Tb, but the size of each RAID array would be well below the 1Tb limit. My understanding is that as long as any volume/array/share stays below 1Tb there isn't a problem, the limit applying to the volume/array/share rather than the total storage inside the box. So could SnapOS manage this, and has 12x160Gb disks been tried? Obviously because of the LBA48 bit issue 130Gb is the largest disk you can have and I'm clear on this.

I also understood that the 12000 was also built with Guardian OS (with a different, larger firmware chip) as well as SnapOS. The earlier SnapOS version was not user-upgradeable to the later Guardian version as it would require this chip to be replaced. So these GOS models were only development machines and were not produced for sale?

Thanks for the info on the memory. 1Gb ECC SDRAM sells at a premium, so I didn't want to bother with it unless there was a reasonable chance it worked as I doubt it will make a great difference. You say PC133 works; but not what speed it works at - does it work at PC100 speed or PC133? Both PC100 and PC133 memory is available and if the higher speed is used I might as well buy PC133, but if it only operates at PC100 it doesn't matter which I get.

Finally I haven't upgraded the 12000 to version 4 as I've read that SnapOS 4.0.860 doesn't get on with the 12000. I'm running 3.805 as it's the latest version 3 I have. I also have 4.0.829, but I believe this too has issues on the 12000.

[bitor]

Yes you are correct as this forum states the SnapOS has a limit of 1-1.2TB, but if you would like to try what you have subjected as to building the Snap12000 to see if you could have three 4 drive RAID5 arrays or two six drive RAID arrays that would be great for the fourm's knowledge. I've wanted to do this , but the cost is what kept me from doing it. So, more power to you if you could do this and let us know if it worked or not.

As far as GOS and the Snap12000... what we heard was that it was an in house project and GOS 2.6 was the best that the Snap12000 could use and as far as I know it was not for sale to the general public. "with a different, larger firmware chip" I have looked for the larger chip size to install sometime ago, but had no luck in finding one. If you can find some for sale then let me know. It's something I wanted to do a while back.

As far as the memory is concerned, my cpu says it runs at 133Mhz so I would say that the FSB133 would be the speed throw the mobo and the PC133 would work in my case, but if you want to be extra safe, then go for the PC100. And If you really want to get the highest speed then pop off the CPU heat sink and see what it says. Remember to put some thermal paste back on IF it comes off. There is extra thermal paste on the bottom of the heat sink you can use if you don't have any. Just scrap some off and put it on the cpu if you need it.
Premium? You can pick up one for a very reasonable price on ebay here in the US, but maybe not where you are. The premium are the 2GB ECC SDRAM 133. If these were not so high I would try one of them in my Snap12000 to see if it would work. And I bet it would.


Let us/me know how the raid experiment goes. I'd really like to find out if what you said could be possible. Any other information that you need just ask and I'll do my best to answer it. Also, let me know how the hunt goes for the larger chip goes. 4 MB chip I is to small for GOS. So try to find something larger. Just looked at my notes and it is a TSOP 40 pin 4MB chip @5v. You would need for GOS a 12MB ,but might be able to do this with a 8MB. However, there is an open PLCC32 socket that they might(guess) have used in conjunction with the TSOP so they could get a larger memory or maybe not.


bitor

Originally Posted by achapman

The thing I'm not clear about is the storage limit. I understand that on a 4000 you can't go above 250Gb drives with a RAID5 array as SnapOS can't handle RAID arrays above 1Tb/1.2Tb. My reading of the articles in the forum seems to indicate that the limit relates to the largest array SnapOS can handle rather than the total usable storage placed in the box. The 12000 could have three 4drive RAID5 arrays or two six drive RAID5 arrays. If it had the latter the combined usable storage with 160Gb drives would be around 1.3Tb, but the size of each RAID array would be well below the 1Tb limit. My understanding is that as long as any volume/array/share stays below 1Tb there isn't a problem, the limit applying to the volume/array/share rather than the total storage inside the box. So could SnapOS manage this, and has 12x160Gb disks been tried? Obviously because of the LBA48 bit issue 130Gb is the largest disk you can have and I'm clear on this.

I also understood that the 12000 was also built with Guardian OS (with a different, larger firmware chip) as well as SnapOS. The earlier SnapOS version was not user-upgradeable to the later Guardian version as it would require this chip to be replaced. So these GOS models were only development machines and were not produced for sale?

Thanks for the info on the memory. 1Gb ECC SDRAM sells at a premium, so I didn't want to bother with it unless there was a reasonable chance it worked as I doubt it will make a great difference. You say PC133 works; but not what speed it works at - does it work at PC100 speed or PC133? Both PC100 and PC133 memory is available and if the higher speed is used I might as well buy PC133, but if it only operates at PC100 it doesn't matter which I get.

Finally I haven't upgraded the 12000 to version 4 as I've read that SnapOS 4.0.860 doesn't get on with the 12000. I'm running 3.805 as it's the latest version 3 I have. I also have 4.0.829, but I believe this too has issues on the 12000.

[achapman]

I just got hold of a batch of 1gb memory. I tried four different types all 1gb, ECC, 133Mhz, registered; they included Corsair, Infineon, and Compaq, and none gave me more than 512Mb (as indicated by 'debug memory'). So it looks as though my 12000 won't take more than that. They all seemed to work fine, but not at their rated capacity.

[blue68f100]

The older snaps OS did not required eec ram only the GOS units. Also the SnapOS units only worked with single sided ram, chips only on 1 side. I know there something else I'm missing on the chip density but it's been may years since I have messed with one.

[achapman]

Blue, You are right in that the 4100 and the other low/mid end SnapOS units use non-EEC memory. The 12000 is different though, it won't use non-EEC memory. (See my notes on the 12000). The standard fitted memory for the 12000 (Smart SM572324574E03R) does have chips on both sides, 9 chips per side and is EEC.

The 12000 appears to use a substantially different mainboard to the other SnapOS servers, as it also for example takes a PCI card, drives a small LCD display and has gigabit ethernet, not to mention supporting 12 drives. When new it cost substantially more than any of the other Snap models. It doesn't seem too surprising that such a enterprise level server would require EEC memory.

[bitor]

What version of SnapOS do you have running on your Snap12000 when you checked the 1GB memory? The one slot is NOT a "PCI" BUT a PCI-X(64bit) slot that can use a 3.5 or 5.0 Volt card. The new cost for this unit was around $12,000 US. I had problems using the newest SnapOS v4.0.860 and talked with others that had the same problem. I would go no higher than SnapOS v4.0.830 to be safe, but I've used v4.0.855 with not problems.
What about your RAID scenario that you subjected that you might try? Any luck?

bitor

Originally Posted by achapman

Blue, You are right in that the 4100 and the other low/mid end SnapOS units use non-EEC memory. The 12000 is different though, it won't use non-EEC memory. (See my notes on the 12000). The standard fitted memory for the 12000 (Smart SM572324574E03R) does have chips on both sides, 9 chips per side and is EEC.

The 12000 appears to use a substantially different mainboard to the other SnapOS servers, as it also for example takes a PCI card, drives a small LCD display and has gigabit ethernet, not to mention supporting 12 drives. When new it cost substantially more than any of the other Snap models. It doesn't seem too surprising that such a enterprise level server would require EEC memory.

[achapman]

Bitor, As I said earlier in the thread I'm running 3.805 as it's the latest version 3 I have. I also have 4.0.829, but I thought this too had issues on the 12000 so I haven't upgraded that far.

I'm still trying to source 6 identical drives without paying silly money for them. There are so many models of Seagate 160Gb, it takes some time to find one particular model. I've got three working 3160212ACE and four working 3160022ACE. I'm expecting a few more to arrive shortly, so assuming a) they have been correctly described and b) they work I might be able to test a single 6 drive RAID5 array shortly. Getting six of each type for a pair of 6 drive arrays may take a bit longer.

[bitor]

achapman

As you posted but I did not know if you had update/upgraded the unit since then. Maybe the SnapOS needs to be updated in order to achieve the 1GB memory? To me it doesn't sound right to have 12 hd and only having 512MB of RAM for it. I understand how reluctant you would be to upgrade the OS. But it might mean you could add a 1GB of memory to your 12000 and 4.0.829 should work perfectly fine.
Western Digital seems to have the correct drives to fit in the trays from 250GB and down, but always check first. Seagate 80GBs should work. I agree with finding identical drives will be hard. Keep us posted on the RAID configuration testing. This should be very interesting.

bitor

Originally Posted by achapman

Bitor, As I said earlier in the thread I'm running 3.805 as it's the latest version 3 I have. I also have 4.0.829, but I thought this too had issues on the 12000 so I haven't upgraded that far.

I'm still trying to source 6 identical drives without paying silly money for them. There are so many models of Seagate 160Gb, it takes some time to find one particular model. I've got three working 3160212ACE and four working 3160022ACE. I'm expecting a few more to arrive shortly, so assuming a) they have been correctly described and b) they work I might be able to test a single 6 drive RAID5 array shortly. Getting six of each type for a pair of 6 drive arrays may take a bit longer.

[achapman]

I've now got 6 identical Seagates (ST3160212ACE). These are model DB35.2, 160Gb, 7200rpm drives. Initial tests with a 6 disk array are not favourable. I've created a 6 drive RAID5 array successfully, but the disk check on the array fails on reboot with 'unknown drive operation'. The array seems to work OK as basic storage, but if the disk check is failing then there is some doubt about whether the RAID protection will work properly. All the disks test OK individually when formatted as JBOD. If this failure is consistent I will try a 5 disk RAID 5 array next.

Given that I won't be able to revert to V3 firmware, I'll do testing first with my present firmware level. Once I've tried the various configurations on V3, I'll then go to V4.0.829 and see if this allows me more memory and solves any of the issues I get with 160Gb disks on V3.

Any suggestions about particular tests to do on each drive configuration? I was thinking 1. reboot. 2. write to disk until full. 2.reboot. 3. fail one disk. 4. reboot. 5. replace failed disk by adding replacement and rebuild array. 6. final reboot.

I was going to try first 6 disk RAID5 array, then 5 disk RAID 5 array with hot spare.
If six disk RAID 5 doesn't work, then the next most efficient combination on a 12000 would be 3 x 4 disk RAID 5, so it would be important to know that 4 x 160Gb is a working combination. Since this works on other SnapOS platforms I'd be surprised if it doesn't on a 12000.

[bitor]

Hey achapman,

Thank you for updating me/us on your progress. It is greatly appreciated.

Unfortunately, it is not well documented on the fixes on each version of SnapOS. So, hopefully, the OS upgrade will fix any problems you might be having. Don't know if you know this, but you can create a 3 disk RAID in SnapOS also you can re size this disk too, but that might not pertain to you. This is all via the debug menu and I don't know if you would want to fool around in that area. Just thought I'd tell you.

Concerning you not being able to revert to v3, you can put in one hard drive(with v3 installed) in slot 1 with no HDD in the other slots. Make sure it's in JBOD format then shutdown remove the one drive with v3 and put it a side before you upgrade the OS. You can use this drive to revert back to your v3 simply by reinstalling this single drive with no other drives with the newer version OS them. You will have to wipe the other drives to remove the newer snapos you installed then place them in after slot 2-etc.. The os should transfer the v3 to the other drives that were wiped. Remember to break the RAID in JBOD with the newer Snapos before you revert back to v3

What settings do you have the drives on? CS OR MS? This could make a difference in the drives and the OS behavior. Try one CS and then try the other MS in your testing.

The fail testing method sounds good to me, but I don't know if I would fill the drive up to it's max. Say 1/2 or 3/4 would do just fine. The SnapOS needs 10% free disk space in order for it to work correctly. If you don't leave this amount you could run in to some problems.


Great work achapman! Also, the cpu can be upgraded to I think 933Mhz, but I'd have to check my notes to be sure. I think they came with a 733Mhz.

Originally Posted by achapman

I've now got 6 identical Seagates (ST3160212ACE). These are model DB35.2, 160Gb, 7200rpm drives. Initial tests with a 6 disk array are not favourable. I've created a 6 drive RAID5 array successfully, but the disk check on the array fails on reboot with 'unknown drive operation'. The array seems to work OK as basic storage, but if the disk check is failing then there is some doubt about whether the RAID protection will work properly. All the disks test OK individually when formatted as JBOD. If this failure is consistent I will try a 5 disk RAID 5 array next.

Given that I won't be able to revert to V3 firmware, I'll do testing first with my present firmware level. Once I've tried the various configurations on V3, I'll then go to V4.0.829 and see if this allows me more memory and solves any of the issues I get with 160Gb disks on V3.

Any suggestions about particular tests to do on each drive configuration? I was thinking 1. reboot. 2. write to disk until full. 2.reboot. 3. fail one disk. 4. reboot. 5. replace failed disk by adding replacement and rebuild array. 6. final reboot.

I was going to try first 6 disk RAID5 array, then 5 disk RAID 5 array with hot spare.
If six disk RAID 5 doesn't work, then the next most efficient combination on a 12000 would be 3 x 4 disk RAID 5, so it would be important to know that 4 x 160Gb is a working combination. Since this works on other SnapOS platforms I'd be surprised if it doesn't on a 12000.

[achapman]

Bitor,
An update on where I'm at, so you don't think I've forgotten.

I've got a bit stuck as I can create RAID5 volumes, but so far every combination has failed the disk check when I reboot. I get the "unknown disk operation" error. From reading around it seems that to run disks in RAID5 requires everything to be absolutely right, there seems no tolerance for the slightest error, so disks and power supplies that work OK for JBOD can fail for RAID. I haven't got Spinrite so I'm trying to identify the out-of-spec components by trial and error. I was only using 1 power supply, but I'm now using both to eliminate the power supply as a cause of failure. I have started using 3 disk RAID5 as it will be simpler to try various combinations, trying to find disks that work.

I have now also upgraded to 4.0.854 (by accident, as a snap_sys.sup I thought was 4.0.829 turned out to be mislabled!). Things seem to be working OK on this version, although the initial boot seems a lot longer. It hasn't made any other difference to the results I've been getting though.

I'm not sure on reverting the SnapOS version. I thought the SnapOS was burned in ROM, so that this overrode what was on disk1. When I've booted with 4.0.854 it comes up with the version number before it mounts the disks. I thought it was only certain other SnapOS servers that relied on a copy of OS on disk to boot from. So even if I use a disk1 I saved while using 3.4.805 It still boots using 4.0.854.

As regards MS or CS settings, the disks have to be set MS or they won't work.

I have discovered the detail logs available through the debug command "info log temp" and saw that Raid5Resync was failing after 54% with some lines that include:
DISK: req=0xC76370C dev=0xC0001 fn=1 blk=0x8AF9060 sts=20
DISK: req=0xC76370C dev=0x80001 fn=1 blk=0x8AF9060 sts=20

These device numbers (C0001 and 80001) aren't the usual ones (10000, 10008 etc) usually given, so I was unsure how they map to the actual drives - I wanted to know which one was failing... I switched drives 2 and 3 and this time Raid5Resync failed after 54% with slightly different results:
DISK: req=0xC796CF8 dev=0xC0002 fn=1 blk=0x8AF9060 sts=20
DISK: req=0xC796CF8 dev=0x80002 fn=1 blk=0x8AF9060 sts=20

My guess from this is that the faulty drive was in slot2 (C0001/80001) and is now in slot3 (C0002/80002).

(See http://forums.procooling.com/vbb/arc...p/t-13851.html for a similar discussion. Cayenne got errors on C0002/80002 and eventually found that a drive in slot 3 was at fault)

[achapman]

Latest on 12000 upgradeability is now on the wiki.

Summary:

Latest stable OS is 4.0.830. 4.0.860 and 4.0.854 not recommended.

You can use a six disk Raid5 set made up of 160Gb disks. The disk size limit is 128Gb so this gives you a 640gb volume.

You can upgrade the processor from 733MHz Pentium III to any other 133MHz fsb Pentium III. The fastest tested is a 1000MHz P3.

Most memory I can get recognised is 512Mb. (irrespective of SnapOS version).

[bitor]

Thank you achapman for your testing and valuable input on this subject matter. It is greatly appreciated. Please keep us posted.

[bitor]

achapman, as a test for the 2 sets of 6 disk RAID 5, have you tried to format the hard drive size(via debug command) to 80GB on the 160GB hard drives and then try to setup the two sets of 6 disk RAID ?

[achapman]

Now all working OK. I've now got the 12000 working with two Raid5 arrays of 6 disks. One is made up of 6 Seagate ST3160212ACE and ST3160212A (7200rpm) formatted to 128Gb per disk and the other set of 6 is a mixture of 4 IBM Deskstar 120Gb disks and two Seagate ST3160022ACE 160Gb disks (these are 5400rpm).

The total capacity is around 1.1Tb. The main problem was sourcing 12 reliable disks. I had to reject four ST3160022ACE's that appeared to work when used standalone, but which gave errors when configured in an array. The Deskstars ended up coming from a Snap 4400, so I knew that they would work in an array together.