The future of computer cooling? Energy use at scale in cloud computing

SpyderTracks

Bingo Bango Orchestrator
Moderator
I was lucky enough to be invited to a virtual tour around Microsoft's Datacenter in Ireland a couple of weeks ago and it really opened my eyes to several things.

When you're scaling up operations to the level of a Datacenter, things that may not seem so important to the desktop user such as power and thermals make a huge difference both in cost and environmental impact. 1 additional Watt on a server translates to 100,000 watts across the datacenter.

Microsoft, despite their downfalls, have done really well in adopting green initiatives. It's not just reducing emissions of course, it's also generally much cheaper operational costs. They currently invest about $1Billion a year to green initiatives. Their plan is to be carbon neutral across the corporation by 2030, and have repaid their historical carbon use for the last 50 years by 2050 by sequestering carbon back out of the atmostphere.

A few bulletpoints that I have no doubt will eventually filter down to the home market:

1/. Their latest servers are called "Mega Godzilla Beast", and their next server stack has been codenamed "Son Of A Beast". This instantly made me like them more!


2/. To lower energy use, where possible they use natural energy, so where most people think a datacenter is like a refrigeration unit with active air conditioning, in reality, they use a mostly green system that captures rainfall into a local reservoir, pumps that into a layer of fabric in the roof of the datacenter, they draw air through that and that naturally provides up to around 6 c lower air temperatures. This is all that's needed to optimise temps around the datacenter. And even then, in Ireland, they would only need that extra cooling capacity about 4 months of the year, the rest of the time, ambient temps are low enough by themselves.


3/. They've tried various methods to keep servers cooler with lower energy use, they tried custom watercooled loops, but obviously that's quite expensive to produce and leaks and levels have to be constantly monitored, as well as extra energy costs from the pumps. So they've come up with a solution that the entire rack can be submerged in to provide effective watercooling at all times:



4/. The above cooling solution has led to their long term plan that they're currently investigating which is to create "pods" of containers and submerge them beneath the sea! This is called Project Natick:


5/. To aid in their sequestering project to reduce carbon from the atmosphere, they are installing suquestering generators on the rooves of each datacenter, the idea is that these suck out carbon from the atmosphere and pump it into the bedrock where it quite literally crystallises and is stored again.


05_30_2017_Bobby_Magill_CC_Climeworks_FB_600_315_s_c1_c_c.jpg


shema_ccs.jpg


 
Last edited:

SpyderTracks

Bingo Bango Orchestrator
Moderator
The reason I went on this tour was that we're in the process of migrating all our services to Azure and Amazon Web Services, and we'll be selling off our physical datacenters. Cloud computing is now at a stage where it really is more and more appropriate to let go of hybrid solutions.

There are still the niche operations like high security government services where they will require a hybrid setup to keep data locally, that's probably always going to be the case for any high security environment. But for most operations, cloud computing is a really viable option.

This is one of the Data Centres we are working on, currently most of the work is in building the new centre in Leixslip in Kildare just outside Dublin.

Staggering scale these things
It's mind boggling! But really encouraging that the big operators are generally taking green initiatives seriously. Must be incredible to be involved at the building stage, I'm quite envious!
 

Steveyg

VALUED CONTRIBUTOR
It's mind boggling! But really encouraging that the big operators are generally taking green initiatives seriously. Must be incredible to be involved at the building stage, I'm quite envious!
The amount of engineers working in tandem on this is quite astounding and seeing everyone working in real time in the likes of Navis and Revit is incredible. Having machines rendering 20,000 items in a single room and having 300 engineers working with one another to piece it together is a herculean task, we are focused on the pipework throughout the centres making sure they are properly routed and supported for the loads going through them.

Some of the cooling loops are incredible. 72" Pipe in some areas and fun fact the largest sized diameter pipe ever worked on in Ireland was first done in our workshop
 

AgentCooper

An Absolute Savage
Moderator
Some of the cooling loops are incredible. 72" Pipe in some areas and fun fact the largest sized diameter pipe ever worked on in Ireland was first done in our workshop
But let’s get down to the real question that everyone wants to know the answer to, Stevey… have you shouted down one of those pipes yet? I bet the acoustics are ridiculous.
 

AgentCooper

An Absolute Savage
Moderator
72" mate you could nearly drive your car through it never mind shout
Avoiding the question, I see. You sang ‘I Will Always Love You’, didn’t you? 😜

Although, in all seriousness, this stuff is fascinating! Thanks to yourself and the Spyder for providing info. Is there any chance that you can post some pics of what you do, Stevey? Or is there a NDA hatchet man waiting to terminate with extreme prejudice if you do so?
 

Steveyg

VALUED CONTRIBUTOR
Avoiding the question, I see. You sang ‘I Will Always Love You’, didn’t you? 😜

Although, in all seriousness, this stuff is fascinating! Thanks to yourself and the Spyder for providing info. Is there any chance that you can post some pics of what you do, Stevey? Or is there a NDA hatchet man waiting to terminate with extreme prejudice if you do so?
I honestly don't know mate, I see a lot of drawings and can see engineers across from me in the office here working on it. But I'm the logistics and procurement manager I'm not personally working on the design just purchasing/getting the material and consumables required for fabrication of the parts we are physically working on

Design team is planning the work, fab team is putting it together and MTO (My team) is planning the priorities and getting the material here to work on

I've a collection of photos here of the site in question, in one photo I believe I counted 37 massive cranes. Not sure about sharing though but I'll ask a guy who should the know the answer fairly discreetly
 

Martinr36

MOST VALUED CONTRIBUTOR
But let’s get down to the real question that everyone wants to know the answer to, Stevey… have you shouted down one of those pipes yet? I bet the acoustics are ridiculous.
No the real question is what colour is the coolant................
 

RichLan564

Well-known member
You never really get the scale of these things until you have been in one, i have walked this site, including the data halls and its enormous, i swear you could see the curvature of the earth down the service corridor!

2021-11-09_12-30-10.jpg
 

SpyderTracks

Bingo Bango Orchestrator
Moderator
One other thing of note I forgot to add was long term storage mediums.

Back in the 60's they generally used Tapes as backup storage. Unfortunately tape has a shelf life which is fairly short really for long term storage, and we're getting to the stage now where a lot of these tapes have degraded to the point that the data is lost.

Also, with the sheer amount of data that's required to be stored these days with Big Data projects and AI, the volume that can be stored on current technologies means that you have to have a ridiculous amount of drives / tapes. It's getting to the point it's just not viable anymore.

I remember a Tomorrow's World episode, perhaps in the late 80's or early 90's where they showed the potential of a cube of silicon that could be used to store data in the future, and this is pretty much the tech that Microsoft are pioneering now to get around these issues.

I can't find the exact archive footage of the episode I'm remembering, but this is the closest I could find:


So now we're facing storage volumes of multiple petabytes of data on a daily basis, and that is only going to grow rapidly with further cloud and AI adoption.

So Microsoft partnered with Warner Brothers Studios with Project Silica which uses discs of glass to store data, in this case, the first Superman Movie, rather than in bits and bytes, it's in "Voxels" which are a 3 dimensional permeations of the glass at an atomic scale. In basic terms, the principal is very similar to a vinyl record.



The storage timeframe for these discs is around 10,000 years, and the storage amount can be scaled as you like.


There are other researchers looking into DNA storage which is equally mind boggling.
 
Last edited:

Steveyg

VALUED CONTRIBUTOR
Oh on storage mate and for a bit of context in how where the production is on that. We do some work for Seagate, they have a plant near us

The built an entire new wing that's about 50% of their current size to increase production. We worked on it and completed the project about 7/8 months ago.... power still hasn't been turned on to the new side of the building

They have to completely shut the plant down to enable power to the new facility but they estimate 24Hours downtime with no issues and if there is a problem powering back up then they could face longer down times

So instead they've decided to just leave it for the time being.... a completely new, spanking, shiny complete production facility they are afraid to turn on encase they lose a second of production it's insane

The buildings empty, doesn't even have the lights on.
 

ubuysa

The BSOD Doctor
Moderator
I'm having a deja vu moment.....

The old mainframe datacentres were centralised computing centres, both for data storage and for application execution. Then in the 1990's distributed computing was all the rage and centralised datacentres were old hat. Applications should run on cheap PCs where the users are located and data should be decentralised and shared via LANs and WANs.

Now, the pendulum is swinging back the other way, and running applications on PCs and storing (even shared) data locally is now old hat. Centralised datacentres and cloud computing is the future. Haven't we been here before?
 
Top