CPU and chassis fan replacements attempt to reduce Supermicro SYS-5028D-TN4T mini-tower's already low noise levels while increasing M.2 NVMe SSD cooling
I now have measurements of watt burn and dB levels for each of the SuperServer Xeon D form factors, seen near in TinkerTry's exclusive comparison table. The mini-tower is still by far the quietest and most versatile. The faster and cheaper Samsung 960 M.2 NVMe drives run cooler than the 950 PRO, so throttling is even less likely during my normal everyday use. See also various (warranty voiding?) ceramic heat sinks for M.2 NVMe that come bundled with various PCIe to M.2 adapters. I've given up worrying about replacement fans and heat sinks, at least for the two already-well-designed mini-towers that are currently the only Xeon Ds that I own, with even better cooling on the Xeon D-1567 12 core version, available here. Original article below.
I'm excited about how it's going so far, making this a system that can run not just in a home basement or closet, but perhaps right there in a home office.
This post is a work-in-progress, orignally drafted Feb. 22nd, see also Intel's Xeon D-1500 thermal documentation. Parts and process not yet finalized, for details, jump to the latest update below.
I demonstrated my fan adapter prototype with the M.2 cooling duct to a VMUG crowd this past Tuesday, along with a brief FLIR One thermal imaging demonstration. I was showing how the Xeon D-1541 motherboard components stayed cool even with the chassis cover off. Even under heavy load:
The configuration used for that brief demo was VMware ESXi 6.0 U1b running on a USB thumb drive seen in the front of the mini-tower, with a Windows Server 2016 Technical Preview 4 VM on my Samsung 950 PRO M.2 NVMe 512GB SSD. The VM was given 16 vCPUs, and 125GB of RAM. Good old Prime95 was used to abuse all 16 vCPUs, quickly and easily.
I haven't yet worked out my favorite combination that cools the CPU at least as well as the stock Delta fan, while keeping things quiet. It's looking like replacement 4 pin PWM fans may run a bit slow, so the Noctua FLX 3 ping models that run at full speed might be what I settle on, stay tuned.
Why just the CPU fan, what about the chassis fan?
I unplugged the chassis fan while the system was at idle and under load, and the perceived difference in noise emitted from the system was minimal. Can't go messing with the power supply fan, which appears to be pretty quiet anyway. That leaves the CPU fan. Yep, big noticeable volume difference noticed, when trying out a variety of 50mm and 60mm fans on it.
What about 1U?
Small fans mean higher RPMs, so it's unlikely to find a 1U system that's pleasant to live near.
Extra fan header on the new PBC Rev. 2.00 for Xeon D-1541
Note that the X10SDV-TLN4F now ships with the newer Rev. 2.00 PCB for the new Xeon D-1541, and it has an extra fan header, discussed here, and seen at bottom left of the above motherboard photo. Sliding the motherboard out the back of the chassis is rather easy, and takes only about 2-3 minutes, once it's become a familiar process to you. Howto videos on the way!
Why not cool the M.2 occupant just a bit, while I'm at it?
A nice bonus might be successfully keeping the Samsung 950 M.2 NVMe slightly cooler as well, with preliminary tests already showing the 3D printed duct can cause it to easily handle long benchmarks at full speeds, with no throttling, even with the BIOS fan speed settings at default normal settings. While the chassis top fan speed can easily keep the 950 PRO cool no matter how long you abuse it, I'm not interested in that increased noise 24x7. I'm not saying thermal throttling is really an issue for me at all during normal use, given a full cloning of a Windows 10 VM takes only 18 seconds. I'm just saying that it's possible that creative use of a $2 hunk of plastic (that can easily hold up to the mild heat of the CPU heat sink) may also be able settle that question once and for all, even for heavy benchmarking, without resorting to warranty-threatening NVMe sticker removal for heat sinks.
This approach of using an inexpensive 3D printed adapter was inspired partly by years of rave Noctua fan reviews, coupled with the recent Noctua fans featured in the cool Lego encased Xeon D-1540 system I wrote about back on January 19 2016:
- How cool is that Hypercube Lego Server, featuring the powerful Xeon D-1540 on tiny and familiar Supermicro X10SDV-F motherboard
where I also noticed that Noctua doesn't make a straight 50mm fan replacement, only 40mm and 60mm sizes.
I hope to be able to achieve good results with simple inexpensive parts, that can withstand the rigors of travel. I have such a journey coming up actually, to give a live demo in Las Vegas for my first Experts 2 Experts Virtualization Conference.
TinkerTry's Intel Xeon D-1500 approach is motherboard maker agnostic
If this part works out, it could make it possible for such a duct/fan combo to be useful in other vendor's Xeon D-1500 offerings, such as ASRock and Gigabyte. That's why the title of this article doesn't mention Supermicro.
I created my first 3D printout ever by first by modifying a thingiverse.com designs in tinkercad.com. Good name for a site, no? ;-) Special thanks to my older son for getting my exported .STL file printed at Tufts University's new Jumbo’s Maker Studio @ CLIC. My first prototype is what you see pictured in this article. He used a Creator Pro FDM 3D Printer with red ABS filament. I may also try out a local public library, using a Makerbot Replicator Mini, 5th generation. I'm not done with tweaking this prototype yet, as I'm not even sure what fan size I'll settle on. More temperature tests through IPMI and FLIR will be needed, especially since I may need to reposition that Supermicro black plastic internal air baffle a bit, that this CSE-721TQ-250B chassis comes with. Of course I'll be sharing whatever I come up with.
STH's approach to noise reduction
This is all a very different approach than what Patrick Kennedy coincidentally and independently published today, doing his usual great work here:
If you're reading this article and live anywhere near California, you may also be interested in his first STH meetup coming up in Mountain View, CA:
The way I see it, the more people interested, and tinkering with this motherboard, the better! Use whatever solution works for your use-case. Preserving your warranty, and ensuring things keep cool and safe, are your responsibility. Probably good to mention that Intel processors poweroff the system abruptly if your cooling modification efforts go horribly wrong and PCU temperatures are allowed to climb to the insanely high thermal limits. You may want to set your IPMI to alert you to issues via email. Just saying.
I suspect over time we'll see more and more folks getting creative with the Xeon D-1500 series of CPUs, much like we have for the Intel NUC. For many of my readers, it's especially exciting that the Supermicro SuperServer SYS-5028D-TN4T system is the very first home lab system on the VMware Hardware Compatibility Guide!
This beloved platform is going to be around a while. What's not to love about very few dollars and a few minutes spent to obtain a much quieter system that still runs cool, can run benchmarks at full speeds without throttling, and can be left running efficiently in your living space? Noise, heat, and cost were exactly the reasons the smaller Intel NUC and Mac Mini did so well for home lab virtualization enthusiasts. With Intel Xeon D-1500, we can take a huge step up from NUC with 4x the cores, 3x the cache, 4x the memory, and 4x the networking, all integrated nicely into a compact mini-tower with much faster storage, at about thrice the price. Nice!
All contents below will be modified and updated as I progress with my testing, eventually including a parts list.
What kind of improvement?
How does the following set of improvements sound, while the system's NVMe and CPU is heavily stressed?
- Increase CFM - from 13.57 CFM to TBD, something higher CFM
- Reduce noise levels - from TBD dB to TBD dB
- Reduce RPM - from 6,500 RPM to TBD, 3,000 to 4,000 range likely
- Decrease M.2 - from 140F to TBD
Challenges
There are some factors at play here that didn't make doing the research any easier, including:
- 50mm-50mm isn't a very common dimensional size for a CPU fan (with 40mm x 40mm hole spacing)
- there are no aftermarket integrated heat-sink cooling fans for the Xeon D-1500 series FCBGA1667 CPU design, but that's ok, I don't really want to recommend removing the thermal paste anyway, or delidding
- Supermicro doesn't seem to list the 50mm x 50mm CPU fan that's included with the X10SDV-TLN4F motherboard in their System Fan Matrix
- While I can find the listings for this Delta fan at Digi and AVNET, none of them are in the 4 wire PWM (Pulse Width Modulation) configuration that Supermicro uses
- unsure of dB test result measurement techniques and distance, and I don't have an anechoic chamber
My Tentative Test Plan
No problem. I don't need anything fancy. I can just configure my test environment the same way I had it running for my recent dB tests, seen in the video below
Once the environment is configured, I see if I get the same dB readings, at the same 30" distance. Once that's squared away, I'll retest once the fan has been replaced.
That's why it's all the more important to record a baseline, carefully measuring the CPU temperature at idle and under load, and the dB range, at a given distance.
Next, replace the CPU fan with a Noctua fan and have another identical look, measuring the CPU temperature at idle and under load, and the dB range, at a given distance.
I'm also in the midst of quieting down my original SuperServer Workstation design's low profile 65 watt Visiontek 7750 GPU, read about this vSphere 6 Datacenter/Workstation Bundle 1 combo here:
Reference Material
Stock Fan - Digi Part # (3 wire)
Digi-Key Part Number 603-1583-ND
Manufacturer Delta Electronics
digikey.com/product-detail/en/EFB0512HHA-R00/603-1583-ND/2560706
Manufacturer Part Number
EFB0512HHA-R00
Air Flow 13.6 CFM (0.385m³/min)
Static Pressure 0.232 in H2O (57.8 Pa)
Stock Fan - Delta Part # (3 wire) at AVNET
Manufacturer Part #: EFB0512HHA-R00
DELTA FMBG
50X50X10MM 12V DC FAN W/ SPEED SENSOR (LOCK)
Available at AVNET Express
EFB0512HHA-R00 Datasheet (March 05 2010)
Maximums: 6500 RPM / 13.57 CFM / 38.5 dB
data.datasheetlib.com/pdf1/12/69/126962/efb0512hha-r00_3962ffc302.pdf
Noctua NF-A6x25 PWM (4 wire)
Noctua NF-A6x25 PWM
550 RPM to 3000 RPM
noctua.at/en/nf-a6x25-pwm.html
Airflow 29,2 m³/h
Airflow with L.N.A. 22,6 m³/h
Acoustical Noise 19,3 dB(A)
Acoustical Noise with L.N.A. 13,7 dB(A)
The Noctua is 17.2 cfm max, see
convertunits.com/from/cubic+m/hr/to/cfm
I'm investigating using other size Noctua fans as well.
Frozen CPU 50 mm fans
frozencpu.com/cat/l3/g36/c15/s71/list/p1/Fans-12_Volt_Fans-50mm_x_10mm_Fans-Page1.html
Gelid Solutions Silent 5, 50mm Quiet Case Fan
One simple and affordable option I'm also investigating can be seen at quietpc.com/50mmfans:
- Amazon - $12.00 (Apr 12 2016)
- Newegg - $32.80 (Apr 12 2016)
- Coolerguys - $6.95 (Apr 12 2016)
- QuietPC - £5.98 (Apr 12 2016)
Mar 06 2016 Update
It's proving to be difficult to find a way to use a Noctua fan that will also keep the CPU as cool as the stock fan, but I'm getting close. Trying multiple sizes and adapters, that static pressure rating is important. If you're looking for a straight fan swap with just 8 screws to get the job done, the Gelid 50mm seems to work almost as well as the stock fan, with less noise, installed in the picture below.
I have more 3D adapter prototype tests coming up. These "prints" take a while, as do my tax return preparations. It could be a few more days before my next article update. Life, and work, currently intervening.
Mar 07 2016 Update
Keep in mind that ServeTheHome article is a 35 Watt CPU. What I'm finding for the Xeon D-1541 is that the 60 mm Noctua doesn't cool as well as the stock Delta fan, or as well as the Gelid 50mm Quiet Case Fan. That's one of the reasons that my search continues.
Have a look at that page 3-1 of the:
- SC721 CHASSIS SERIES SC721TQ-250B USER’S MANUAL
Fans
The chassis includes a 9cm rear fan with 25db whisper quiet operation, and an
optional 8cm front fan
Not sure which of the many 80mm Supermicro fans this is referring to, in the Supermicro:
Mar 11 2016 Update
More 3D printed prototypes (50mm to 80mm adapter), and some more brands in there. As for striking the right balance between noise reduction and effective cooling, the Gelid 50mm continues to be the champ, handling even the worst abuse Prime95 can dish out, cooling nearly as well as the stock fan.
You'll also see I'm messing around with VT-d/GPUs again, trying out different ways to reduce those tiny fan's noise as well. PNY NVIDIA Quadro K1200 is the newest, most powerful GPU that fits in a low profile PCI slot, with 4 mini DisplayPort 4K outputs. Looking good, with very little sound. VT-d passthru testing under ESXi 6.0U1b, TBD, still working on that...
As for testing and refining my final solution for my home, my testing is very much still underway. CyberTech RPM measuring device seen at top right, and toying with various combinations of adapters and fan settings which aren't always reported by the IPMI for pin configuration reasons. Remember, these aren't exactly OEM fans. PWM's 4 pins for IPMI controlled fan speeds matters less when full fan speed is barely audible.
Mar 24 2016 Update
I'm not sure yet whether I'll have time to get install and testing videos published soon, but I did want to get my findings published as soon as possible, for those of you who have been waiting. These are still preliminary findings, but it's looking likely that these recommendations will be my final ones, even before I complete all my thermal tests. These two fans will likely work out well for your need for increased quiet, even if my finalized solution winds up changing things a bit.
If you were to ever need to send your Supermicro system in for in-warranty repairs, I would suggest replacing these aftermarket fans with the original fans before shipping the unit in. If you are not comfortable with minor PC surgery, I would not recommend you replace any fans. If you are on the fence, I would recommend waiting until I'm able to publish how to videos on these straight forward fan swaps. No thermal paste needed, the CPU heat sink stays put. It's just philips head screwdriver work, and about 15 minutes of your time.
Turns out only one of the many CPU fans I tried kept the CPU very nearly as cool as the factory CPU fan, even when left running Prime95 for an extended duration. It's very important for the fan to keep its RPMs up even when there's backpressure, from either pumping a large area of air into smaller hole (using my 3D printed prototypes), or obstacles nearby (CPU heat vanes).
STEP 1 - Suitable CPU Fan Replacement - designed in UK
Gelid Solutions Silent 5, 50mm Quiet Case Fan
The only fan that handled cooling well while eliminating the stock fan's high pitch fan noise is the Gelid Solutions Silent 5 50mm Quiet Case Fan. It's currently priced somewhere between £4.98 (UK) and $32.80 (US), with Newegg currently offering the lowest prices in the US. Simple 4 screw drop-in replacement, no adapters required.
Specs and product information on all 50mm CPU fans at quietpc.com/50mmfans where you'll find only 2 models listed, with purchase links for the Gelid here:
- Amazon - $12.00 (Apr 12 2016)
- Newegg - $32.80 (Apr 12 2016)
- Coolerguys - $6.95 (Apr 12 2016)
- QuietPC UK - £5.98 (Apr 12 2016)
STEP 2 - Suitable Chassis Fan Replacement - designed in Germany
- SILENT WINGS 2 | 120mm
Keeps that M.2 SSD cooler too!
Specs and product information at bequiet.com/en/casefans/259, currently at Amazon for $34.95, and Newegg for roughly $10.95:
- Amazon - US
- Newegg - US
- Where to Buy reseller locator
If you're really into keeping things quiet and cool, you can also get your Samsung 950 PRO M.2 NVMe SSD temperatures to stay low enough to avoid thermal throttling. Unfortunately, the 3D printed adapters got this done, but at the expense of 10 to 15 degrees Celsius higher CPU temps, which isn't really acceptable. Even with the bypass duct (hole) blocked, all CPU fan reducing adapter scenarios meant a CPU running at noticeably higher temperatures when under load. Static pressure capability matters, and higher RPM/smaller fans did better.
The answer for me was not to just swap out the CPU fan, but to also swap out the chassis fan, using the be quiet! brand.
The exact model I chose to test first is NOT the PWM version. It's the 3 pin model that will keep its RPMs at the rated 1500rpm at all times, moving 50.5 cfm / 85.8 m3/h. Even at this fixed speed, it's still a little quieter than the stock fan it replaced at any speed. This be quiet! German manufacturer is also known for power supplies, and claim a life expectancy of 34 years of continuous operation, because of the Fluid Dynamic Bearing (FDB). This fan manages to improve airflow at all times, and preliminary results seem to indicate it can also prevents M.2 thermal throttling as well, at least for my 4-in-a-row tests of ATTO Disk Benchmark.
I will still be trying the SILENT WINGS 2 | 120mm PWM model as well, to determine if it keeps the system cool enough. It just hasn't arrived yet. It's currently available for $10.95 at Newegg here, and various Amazon sources here.
Product Video
Apr 12 2016 Update
I've discovered that PWM fans from be quiet! tend to hunt when used with this system. By hunt, I mean that they cycle between higher and lower RPMs roughly every 8 seconds, never settling on any one RPM. The 120 version for example would sometimes go as low as 500 rpm, which would signal a warning. So non-PWM 3 wire be quiet! variants seem to be the best choices.
I've also noticed that Patrick's scheme for using a front-mounted Noctua 60mm fan does allow you to run the Samsung 950 M.2 drive at full speeds without throttling, but at the cost of some extra noise.
I'm actually still experimenting with various combinations of fans that give the quietest experience under system stress while still providing excellent cooling and a 4+ year life expectency.
Jun 18 2016 Update
I've now published an unboxing video for the Supermicro SNK-C0057A4L Supermicro Heat Sink with retention back plane, which includes the Delta fan attached to the fan bracket that's attached to the CPU heat sink assembly, with thermal paste pre-applied.
These 4 tiny screws that hold the fan bracket onto the CPU heat sink are a little bit tricky to remove, if you wish to avoid removing the whole motherboard to get to the underside, where the CPU retention back plate lives. I spoke with Supermicro parts, and they unfortunately can't (or won't) sell the screws separately. So you'll want to take extra care to avoid stripping them.
Jul 13 2016
There are yet more options out there. There's a non-PWM fan I'll be trying soon by be quiet! called the
- Shadow Wings SW1 120mm 2200rpm
Fan speed @ 12V (rpm)2200
Air flow @ 12V (cfm, m3/h)76.2 / 129.1
Air pressure @ 12V (mm H2O)2.70
Noise level @ 12V (dB(A))29.7
And there's the 4 pin PWM fan that is likely to cycle RPM speeds like all the rest, that's this Corsair fan called the
- ML120 PRO 120mm Premium Magnetic Levitation Fan
Airflow 12 - 75 CFM
Static Pressure 0.2 - 4.2 mmH20
Sound Level 16 - 37 dBA
Speed 400 - 2400 RPM
Power Draw 0.225 A
Aug 11 2016
This has been a long and winding road, but now that I'm reasonably comfortable the PWM fan speed issue with hunting may be fixable (keep reading), it would seem that aftermarket fans are becoming a more viable choice.
It's complicated though, and with my recently Xeon D-1541 SYS-5028D-TN4T I've stuck with stock cooling, to give me a chance to measure watt burn and noise levels at factory defaults, as I compare the various tiny newcomers to the incumbents at:
The Mini Tower SYS-5028D-TN4T is still far and away the champ as far as noise levels. Replacing the CPU fan without removing the heat sink is a bit trickier surgery than I would have liked, with minimal noise reduction gains. Using the fan speed attenuator from the Noctua kits to quiet the too-high-RPM GPU fan in the Bundle 1 (SuperServer VMware Datacenter/Windows 10 Workstation combo) has been the most successful finding so far. Not much to show, from months of on-again/off-again tinkering.
I'm feeling more and more like the average user is rarely going to encounter throttling conditions in everyday use, thus, the stock fans and the stock IPMI RPM calibrations are going to be hard to beat, and supporting after market solutions is likely to be difficult.
All that aside, these little Xeon D-1500 CPUs run fine at high (65C and up) temps without throttling, so I regret if folks feel this is some sort of flaw or problem. This whole tiny chipset runs in an incredibly small space quite nicely, and the new SYS-E200-8D model will really put this notion to the test!
As I turn my attention to my VMworld 2016 demo, I'll be back-burner-ing these fan swaps for a while, out of necessity to get things done.
I think the technical aspects of this long convoluted story are summed up well in this recent dialogue from right here at TinkerTry, special thanks to Walter Franken and his notion of tweaking IPMI:
Oct 14 2016
- Copper cooling for the Supermicro X10SDV Xeon D motherboards
Oct 10 2016 by Patrick Kennedy at STHWarning
We wanted to start this post off with a warning: this will likely void your warranty if you damage your system. Furthermore, since the Xeon D package is a BGA package affixed to the motherboard, you will risk rendering your expensive motherboard and processor unusable, especially if you fail to tighten properly. This modification will be extremely easy to see for anyone checking motherboards for warranty returns so you will need to accept that as a risk before proceeding.
I'm not a fan of this approach, knowing that the decrease in noise levels for the SYS-5028D-TN4T are likely to be minimal, with the risk outweighing the benefit for most. But if you sit near your server, and you frequently use it under load and want to leave the factory default chassis fan speeds rather than max speeds, then an article like the one above may be of considerable value to you.
I did Google searches for what Patrick has come up with a few months back, but came up mostly empty, and I never had an extra system to work with that I was willing to risk violating the warranty on. Let's be thankful Patrick found some options, for those of you that really want an even quieter system.
My focus was more on what can be done to quiet the new mini 1U systems, such as the SYS-E300-8D and SYS-E200-8D systems. I couldn't find any simple solutions that ran as cool as Supermicro's stock fans, or were easy enough to install with only some light Dremel tool mods of the fan assembly itself. Here's one example that almost fits as-is, but left me with temps at least 10C higher than the 2 1U included fans in the stock servers:
Oct 24 2016 Update
Excellent update on folks efforts to quiet down the related SYS-5028D-TN4T system, over in the STH forums, you'll want to read the whole source article:
- Forums > Hardware > DIY Server and Workstation Builds > Supermicro SYS-E300-8D
RPMS:
Fan2: 4.4K (prev. 6.6K)
Fan3: 2.7K
FanB: 4.6K (prev. 6.8K)ETA: I also managed to squeeze in 5 flash based drives (2 SATADOM, 1 M.2, 1 MSATA, and one SATA)
Nov 06 2016 Update
I now have a video available, for a close view of how my experiment with trying to replace the Supermicro's two FFB0412SHN brushless 40mm fan assembly with a single Dyantron® CF-003 fan:
Here's another video that shows what the surgery is like, should you decide to replace your entire CPU fan/heatsink assembly, at your own risk:
Jun 25 2016 Update
I've been asked about the direction of the CPU fan that comes built in, wish I had been more clear on that in the article above. The stock fan sucks cooler air from above downward, into the heatsink assembly, causing warmer air to blow out both sides of the CPU heatsink. Hope this helps!
See also at TinkerTry
-
First Look at the FLIR ONE Thermal Imaging camera for smartphones and tablets
-
How to locate your triple monitor (up to 2K) PC 20 feet away for less noise and more joy
- Intel Xeon Processor D-1500 Product Family Thermal/Mechanical Specification and Design Guide
See also
Wikipedia - Computer fan control
Wikipedia - Pulse-width modulation
Socket FCBGA 1667 aftermarket cooling??
Huge thx #CTVMUG #USERCON 4 great day, & @RobKlobukowski for https://t.co/LXQGD5ZZ7e! @vBrownBag @vMattK @MyVMUG pic.twitter.com/D8QDGwDJ1s
— Paul Braren (@paulbraren) March 2, 2016
@TinkerTry @paulbraren pic.twitter.com/HywAhlW4od
— Matt Kozloski (@vMattK) March 1, 2016
Disclosure
TinkerTry.com, LLC is an independent site, has no sponsored posts, and all ads are run through 3rd party BuySellAds. All equipment and software is purchased for long-term productive use, and any rare exceptions are noted.
TinkerTry's relationship with Wiredzone is similar to the Amazon Associates program, where a very modest commission is earned from each referral sale from TinkerTry's SuperServer order page. I chose this trusted authorized reseller for its low cost and customer service, and a mutual desire to help folks worldwide, including a new way to reduce EU shipping costs. Why? Such commissions help reduce TinkerTry's reliance on advertisers, while building a community around the Xeon D-1500 chipset that strikes a great balance between efficiency and capability.
I personally traveled to Wiredzone near Miami FL to see the assembly room first-hand, and to Supermicro HQ in San Jose CA to share ideas and give direct product feedback.
I'm a full time IT Pro for the past 23 years. I've worked with IBM, HP, Dell, and Lenovo servers for hands-on implementation work across the US. Working from home lately, I'm quite enjoying finally owning a lower-cost Supermicro solution that I can recommend to IT Pro colleagues, knowing it will "just work." That's right, no tinkering required.
