Building an Overclocked Rack-Mount Server (1U or 2U)
Rothgar
Join Date: 2009-11-13 Member: 69372Members
Join Date: 2009-11-13 Member: 69372Members
Are there any people running 1U or 2U Servers that are Overclocked currently?
Can you please provide some advice on what hardware I should be looking at.
Mainly I expect cooling will be a problem and I have not seen/heard of any Rack-Mount Server Liquid Cooling for example and don't know what type of CPU fans etc would be available as my experience has been predominately with vendor servers such as HP and they have specific hot-plug fan sockets etc.
So yeah, does anyone have Rack-Mount Servers running overclocked? When I mean Overclocked, I am not talking about enabling Turbo-stepping, I am talking about actual Overclocking of the Hardware.
Ideally 1U would be good, but I understand that 2U would be more practical with regards to cooling. Also are you doing this with Dual CPU sockets, because that would also be great.
Thank you!
Can you please provide some advice on what hardware I should be looking at.
Mainly I expect cooling will be a problem and I have not seen/heard of any Rack-Mount Server Liquid Cooling for example and don't know what type of CPU fans etc would be available as my experience has been predominately with vendor servers such as HP and they have specific hot-plug fan sockets etc.
So yeah, does anyone have Rack-Mount Servers running overclocked? When I mean Overclocked, I am not talking about enabling Turbo-stepping, I am talking about actual Overclocking of the Hardware.
Ideally 1U would be good, but I understand that 2U would be more practical with regards to cooling. Also are you doing this with Dual CPU sockets, because that would also be great.
Thank you!
Comments
Case: Athena Power RM-2U2026S60 Black Steel 2U Rackmount Server Case w/ V2.91 EPS-12V 600W
MB: ASUS P8Z68-V PRO/GEN3 LGA 1155 Intel Z68
CPU: Intel Core i5-3570K
CPU Cooler: Dynatron K666 60mm 2 Ball CPU Cooler
RAM: Mushkin Enhanced Redline 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1866
HDD: 2xWestern Digital WD Blue WD5000AAKX 500GB 7200 RPM
That cooler isn't particularly great as we only pushed the CPU to 4.2GHz. I'm not sure if anyone's tried it, but I've thought some of the low profile desktop coolers would work, especially in a 2U case. For example:
Zalman's:
ZALMAN CNPS8900 60mm
ZALMAN CNPS8000B 66mm
Scythe's:
Scythe Kozuti 40mm
Scythe Big Shuriken 58mm
Phantek's:
Phanteks PH-TC90LS 43mm
Xigmatek's:
XIGMATEK Durin D982 65mm heatsink + 20mm fan = 85mm
Prolimatech's:
Prolimatech PRO-SAM17 Samuel 17 45mm (Heatsink only)
Noctua's:
Noctua NH-L12 66mm (without top fan)
Evercool's:
EverCool EC-HPL-815EP 45mm
I was thinking it would be good if I can get a cooling system such as:
<a href="http://www.asetek.com/datacenter/internal-loop-liquid-cooling.html" target="_blank">http://www.asetek.com/datacenter/internal-...id-cooling.html</a>
I am not sure how efficient it would be though but it looks clean.
However that said I would need to make sure that the case can handle the extra space required for the cooling blocks at the front of the server.
I expect your Server looks to be Single CPU?
Presuming you'd find a company who'd be willing to take a liquid cooled system, I've always toyed with the idea of hosting a setup like this:
<a href="http://www.newegg.com/Product/Product.aspx?Item=N82E16835181015" target="_blank">CORSAIR Hydro Series H60</a> (Radiator = 27mm)
<a href="http://www.coolerguys.com/840556094272.html" target="_blank">Coolerguys 120x25mm Blower Fan</a> = 25mm
Total = 52mm
Which would fit in 2U with roughly 37mm to spare if you mounted them horizontally. Sure, trying to draw air via a roughly 1.5in slit between the radiator and case isn't ideal, but pretty much all of the server CPU cooling solutions that are affordable and currently available suck. You could even get a few extra of those blowers and try to bombard the radiator slit from several angles to get as much airflow going as possible.
There are effective means to cool a 1U rack mount - However, it took us almost 4 months (and lots of $$) to find the right combo of case, fans, and CPU cooler to keep our overclocked CPUs cool enough in a 1U setup.
The info that Scardybob has given would be a good start and I wouldn't recommend going for a 1U unless you are willing to try multiple coolers.
Call us crazy, but we even go as far as lapping the bottom of our CPU heatsinks to ensure the best possible contact with the CPU IHS.
But not the CPU IHS? I guess I didn't think lapping would improve cooling unless you did it on both the cooler and CPU (i.e. having a flat vs bumpy surface setup is not much better than bumpy vs bumpy one).
Now, if you were delidding your Ivy Bridge CPUs and doing direct-to-die contact, then I'd call you crazy :)
Now, if you were delidding your Ivy Bridge CPUs and doing direct-to-die contact, then I'd call you crazy :)<!--QuoteEnd--></div><!--QuoteEEnd-->
I would love to be able to lap the CPU IHS, but unfortunately that voids the warranty.
Lapping just the heatsink does produce noticeable results (3-7 degree difference). While it may not sound like a lot, when you are overclocking in a 1U setup, you take anything that you can get =P.
I have done tests with lapping both the IHS and heatsink in a test build with a 3570k and saw a 14 degree drop in full load temps. If lapping didn't void the warranty I would do this on all my servers.
I did it on mine, and installed a Thermaltake Crystal Orb on the thing... My friend attempted the same thing on his, but lapped the ceramic on the side a bit and damaged it.
Lapping just the heatsink does produce noticeable results (3-7 degree difference). While it may not sound like a lot, when you are overclocking in a 1U setup, you take anything that you can get =P.
I have done tests with lapping both the IHS and heatsink in a test build with a 3570k and saw a 14 degree drop in full load temps. If lapping didn't void the warranty I would do this on all my servers.<!--QuoteEnd--></div><!--QuoteEEnd-->
Well, iirc OCing does void the warranty too, though I think quite a few people get away with RMAing dead OCd CPU's since you can't tell if its been overclocked from the CPU only (unless there's physical damage such as lapping).
3-7C is pretty good for just lapping (I would have expected only 1-2C), but I expect that's starting from a pretty high load temp (in the 70C+ range). The Ivy Bridge cooling issues are well known, so I'm not surprised the double lapping with the 3570k gave a solid drop in load temp. I've read about <a href="http://forums.anandtech.com/showthread.php?t=2261855" target="_blank">3770k OCd load temps being dropped by up to 20C</a> from delidding.
Now, I'm waiting for someone with the time and money to go balls-to-the-wall with a lapped water cooler + delidded 3570k using coollaboratory liquid ultra in a 1U/2U case :)
3-7C is pretty good for just lapping (I would have expected only 1-2C), but I expect that's starting from a pretty high load temp (in the 70C+ range). The Ivy Bridge cooling issues are well known, so I'm not surprised the double lapping with the 3570k gave a solid drop in load temp. I've read about <a href="http://forums.anandtech.com/showthread.php?t=2261855" target="_blank">3770k OCd load temps being dropped by up to 20C</a> from delidding.
Now, I'm waiting for someone with the time and money to go balls-to-the-wall with a lapped water cooler + delidded 3570k using coollaboratory liquid ultra in a 1U/2U case :)<!--QuoteEnd--></div><!--QuoteEEnd-->
Actually, using a third-party CPU cooler already voided the warranty, so the warranty was gone by the time he overclocked it. Yes, using a heatsink other than the one Intel provides voids your warranty, as silly as that sounds. There are probably exceptions for OEM use, but not with a retail boxed CPU.
Water cooling? Feh, kids stuff, just submerge the whole server in mineral oil :P Just make sure to use SSDs instead of HDDs (HDDs don't like being submerged, even in non-conductive liquid, but SSDs don't care)!
EDIT: The warranty thing isn't a rumour either, multiple official Intel sources confirm it, such as <a href="http://communities.intel.com/thread/25127" target="_blank">http://communities.intel.com/thread/25127</a>
Most people who get tripped up by this probably simply told Intel they used a third party cooler, not realizing it voided their warranty.
In terms of telling if you've overclocked or used an out-of-spec RAM setting, well, theoretically they could do something like an eFUSE, but I really doubt they bother.
Are you set on building an overclocked machine?
My recommendation would be to just buy an E3-1270. That processor is more then enough to run 24 slot servers. This will save you time and money. No need to worry about overheating, voiding your warranty or spending hours finding the best setup.
Unfortunately due to the Spark engine and current optimization CPU usage is excessive and so I am looking at the potential of building a box to co-locate, however at the end of the day this may not even happen. At least some people might get some beneficial information if they are trying to do the same thing.
Back on topic at hand, yes I would like to look at the option of overclocking if it's viable. i.e. personally I would not go into lapping of CPU's, I just want to get as much improvement as I can with decent cooling and not having to massively alter the system.
Since I have not built a custom made rack-mount server before, I am not sure of orientation and spacing in the chassis and therefore am asking for advice on people who have built similar systems before.
the Asetek option looks appealing and I may send them an e-mail to see if it's even released and where I can get them. Does anyone know of any other Inner Looped Liquid Cooling Options for servers?
I agree with the mention that data-centers would typically not really want Liquid Cooling to potentially disrupt the whole operation and that is why I am looking at Inner loop systems as that at least is somewhat contained, if I went for most of the other liquid cooling solutions I have seen, they cycle the liquid through the back of the server into an existing pump or rack cooling system.
Alternatively I could look at maybe some of the air cooling solutions mentioned, however not sure currently which would be more ideal in the confined server space.
Empty 2U Case:
<img src="http://i557.photobucket.com/albums/ss18/McGlaspie/2012-08-24_23-25-11_755.jpg" border="0" class="linked-image" />
Filled 2U Case:
<img src="http://i557.photobucket.com/albums/ss18/McGlaspie/Server2U_b3_zps0d84ce73.png" border="0" class="linked-image" />
Video of the system running:
<center><object width="450" height="356"><param name="movie" value="http://www.youtube.com/v/h_EkKMEqM_k"></param><embed src="http://www.youtube.com/v/h_EkKMEqM_k" type="application/x-shockwave-flash" width="450" height="356"></embed></object></center>
Personally, I think if you moved the HDD mounts to the front of the case, you could fit the H60 Radiator + Blower in the back next to the mobo (and exhausting out the back with a little case modding).
Edit: Just found this company that looks like they are offering overclocked servers for purchase. Though, as I expected, their cheapest system is a 4U i7-3960X @4.6GHz for $6150.
Yeah I have experience building custom Desktop PC's and I have been in the IT industry for over 10 years and so have dealt with a number of Rack-Mount Servers but as mentioned these have all been brand name such as HP, Dell, IBM etc.
The brand name servers have typically had a lot of design put into them and as such there is usually not much space left in the case and it has been designed with the best airflow possible. i.e. this video shows the insides of a HP DL380 Gen8 2U Server:
<a href="http://www.youtube.com/watch?v=UD0kBhOCL_4&feature=related" target="_blank">http://www.youtube.com/watch?v=UD0kBhOCL_4...feature=related</a>
You can see the fans up the front of the server to ensure that maximum airflow is sucked and distributed over all components. HP also has thermal sensors scattered across the motherboard and can dynamically alter the speed of the fans to get more airflow in areas that are having heat problems.
The box you showed above is a lot different to these and has quite a bit more space. It also does not look to have too many airflow considerations like HP for example has created.
Although it's interesting to see, I had not heard of Athena Rack-Mount cases before, the only brands I had known of were like Supermicro. i noticed that your server is using actual Desktop PSU, MB etc.
I saw a few Supermicro Servers with I5 support etc, are they the same or are they most likely going to have a more standard design?
I haven't had a chance to really look into them yet, this week is going to be a bit bust for me.
Thanks for all the information so far ScardyBob
i.e. there is no airflow really from what I can see in that video you provided when comparing against a proper HP Server which has a row of fans side by side which ensures that air will be cycled through the server (front to back) and not recycled if you know what I mean.