CCF Archive Site

I realize Gulag stores all our data, like searches, your map location, etc... maybe even a lot more than that, like emails. Supposedly yahoo never deletes any emails, even if you delete them, they store it.. not sure

But where do they get the space to hold all this stuff??

The requirements must be insane, Tera-Giga-Bazillion bytes. That's the *entire* world that uses Gulag and stuff. They are probably writing at the rate of 1 terabyte per second. Or something close to it.

(Gorn corrected Unix's completely unhelpful thread title to something meaningful )

Unix asked where Google stores all those hard disks it uses to store all the internet's data.

I did some research and some math.  If you want to skip all the calculations I did, here's the bottom line:

1.  I figure that Google spent a little over $9 Billion for new hard disks to store all the data that the internet generated in 2017.  Google can easily afford that.

2.  I figure that Google had to buy a building the size of one of the largest hotels in the world to house all those new hard disks (or a bunch of smaller buildings that add up to the same number of square feet).  But that building - even at about $2 Billion - costs a lot less than the hard disks, so Google can afford that too.


Now for the research (links to web sites) and details (calculations).  We need some numbers.

How many dollars per TB does hard disk storage cost?
The cheapest option [dollars per TB] on the web page linked below is an external 8TB external hard disk for $160.

Fry's Electronics Prices for External Seagate Drives

That works out to $20 per TB -- retail price.
Wholesale is probably about $10 per TB.  Google may buy storage cheaper than that, but probably not much cheaper.

So multiply the internet's daily data production by $10 for every TB to calculate Google's daily expense for new hard disks (assuming Google retains 100% of all new internet data -- a dubious assumption).


Now, how many Tera-Bytes are generated on the internet each day?
I did a little searching.  One web site says that, in 2017, 2.5 quintillion bytes per day were created.

https://www.iflscience.com/technology/how-much-data-does-the-world-generate-every-minute/

That means that the internet generated 2,500,000 Terabytes per day.  How do I know that?

Let's detour into the names of large numbers.
Note that I'm using powers of 10 instead of powers of 2 in order to keep the math easy.

   One Million is 10 to the sixth power = 10^6 = 1 Megabyte
   One Billion is 10 to the ninth power = 10^9 = 1 Gigabyte
   One Trillion is 10 to the 12th power = 10^12 = 1 Terabyte
   One Quintillion is 10 to the 18th power = 10^18 = 1 Exabyte = 1 Million Terabytes

Returning to the internet's data production, 2.5 quintillion bytes per day = 2,500,000 Terabytes per day.

Multiply by $10 per Terabyte to get $25,000,000 dollars per day to buy hard disks for all the new data in 2017.
Then multiply by 365 days per year to get $9,125,000,000 dollars per year.

Conclusion:
Google can easily afford to buy all the hard disks it needs to store all internet data production.


HOWEVER, we're not quite done yet.  Unix asked where all those hard disks are stored.

Well, my Seagate backup drive enclosure (sitting on my desk) is 8 inches X 5 inches X 1.5 inches.
Google mounts all its drives on racks, and space between the drives is needed for cooling air flow.
And all the wires take up a significant amount of space too.

I'll assume 10 X 7 X 3 inches of space per drive.

Now let's assume each rack has four stacks and each stack is 20 drives.
The height of the rack would be 20 drives X 3 inches per drive = 60 inches = 5 feet.
The width would be 7 inches X 2 stacks = 14 inches
The length would be 10 inches X 2 stacks = 20 inches.

Overall rack dimensions:  60 inches tall X 14 inches wide X 20 inches long.

Since people need to walk among the racks, we need 2 feet = 24 inches between rows of racks.

Let's assume that each row is 36 racks long = 36 X 14 = 504 inches = 42 feet

Let's assume 9 rows = (20 + 24) X 9 = 396 inches = 33 feet.

Total area = 42 feet X 33 feet = 1386 square feet for one room with 9 rows of racks.

Total number of drives in that room = 80 drives per rack X 36 racks per row X 9 rows = 25,920 drives.

Let's make the math easy and assume 10 TB per drive, giving us 259,200 TB.

10 rooms would give us 2,592,000 TB -- more than enough for one day's internet data.  And 3600 rooms would give us more than enough to store a full year's internet data.  Three hotels in Las Vegas have more rooms than that, according to this web page.

https://en.wikipedia.org/wiki/List_of_largest_hotels

Bottom Line:
Google has to buy a building the size of one of the largest hotels in the world to house all the new hard disks it buys every year (or a bunch of smaller buildings that add up to the same number of square feet).  But that building - even at about $2 Billion - costs a lot less than the hard disks, so Google can afford that too.

Somehow the estimates of Google storage requirements does not make sense.  For 10 years worth of data they would need the equivalent of a small city full or hard drives.   There must be a bit more than appears at first glance on their data capture scenarios.

pxsant wrote:

Somehow the estimates of Google storage requirements does not make sense.  For 10 years worth of data they would need the equivalent of a small city full or hard drives.   There must be a bit more than appears at first glance on their data capture scenarios.

90% of all the internet data in the world was created in only the last two years, according to the article at this link:

https://www.iflscience.com/technology/how-much-data-does-the-world-generate-every-minute/

Note that I linked to this article in my previous post.

If one huge hotel can contain enough hard disks to store one year's past data, then two is almost enough to store all the data that the internet has ever generated.

In my previous posts, I acknowledged that my estimate is based on some shaky assumptions.  Remember GIGO!

I don't know if Google really stores all the internet's data on their hard disks.  It might be only 50% or 25% or 10% for all I know.  Who - outside of Google - would know the difference?  And how?

Many web sites disappear from the internet every year.  Does Google keep all the data from deceased web sites?  If so, why?  I'm sure that Google at least removes deceased web sites from their indexes.  If expired sites are no longer indexed, why keep their data around either?

Many ongoing web sites delete old data from time to time.  Does Google keep expired data on their hard disks forever?  If so, why?  And even if Google did keep expired data, how would I access any of it if none of it's in Google's indexes?

Unix asked where Google stores all those hard disks it uses to store all the internet's data.

I did some research and some math.  If you want to skip all the calculations I did, here's the bottom line:

1.  I figure that Google spent a little over $9 Billion for new hard disks to store all the data that the internet generated in 2017.  Google can easily afford that.

2.  I figure that Google had to buy a building the size of one of the largest hotels in the world to house all those new hard disks (or a bunch of smaller buildings that add up to the same number of square feet).  But that building - even at about $2 Billion - costs a lot less than the hard disks, so Google can afford that too.


Now for the research (links to web sites) and details (calculations).  We need some numbers.

How many dollars per TB does hard disk storage cost?
The cheapest option [dollars per TB] on the web page linked below is an external 8TB external hard disk for $160.

Fry's Electronics Prices for External Seagate Drives

That works out to $20 per TB -- retail price.
Wholesale is probably about $10 per TB.  Google may buy storage cheaper than that, but probably not much cheaper.

So multiply the internet's daily data production by $10 for every TB to calculate Google's daily expense for new hard disks (assuming Google retains 100% of all new internet data -- a dubious assumption).


Now, how many Tera-Bytes are generated on the internet each day?
I did a little searching.  One web site says that, in 2017, 2.5 quintillion bytes per day were created.

https://www.iflscience.com/technology/how-much-data-does-the-world-generate-every-minute/

That means that the internet generated 2,500,000 Terabytes per day.  How do I know that?

Let's detour into the names of large numbers.
Note that I'm using powers of 10 instead of powers of 2 in order to keep the math easy.

   One Million is 10 to the sixth power = 10^6 = 1 Megabyte
   One Billion is 10 to the ninth power = 10^9 = 1 Gigabyte
   One Trillion is 10 to the 12th power = 10^12 = 1 Terabyte
   One Quintillion is 10 to the 18th power = 10^18 = 1 Exabyte = 1 Million Terabytes

Returning to the internet's data production, 2.5 quintillion bytes per day = 2,500,000 Terabytes per day.

Multiply by $10 per Terabyte to get $25,000,000 dollars per day to buy hard disks for all the new data in 2017.
Then multiply by 365 days per year to get $9,125,000,000 dollars per year.

Conclusion:
Google can easily afford to buy all the hard disks it needs to store all internet data production.


HOWEVER, we're not quite done yet.  Unix asked where all those hard disks are stored.

Well, my Seagate backup drive enclosure (sitting on my desk) is 8 inches X 5 inches X 1.5 inches.
Google mounts all its drives on racks, and space between the drives is needed for cooling air flow.
And all the wires take up a significant amount of space too.

I'll assume 10 X 7 X 3 inches of space per drive.

Now let's assume each rack has four stacks and each stack is 20 drives.
The height of the rack would be 20 drives X 3 inches per drive = 60 inches = 5 feet.
The width would be 7 inches X 2 stacks = 14 inches
The length would be 10 inches X 2 stacks = 20 inches.

Overall rack dimensions:  60 inches tall X 14 inches wide X 20 inches long.

Since people need to walk among the racks, we need 2 feet = 24 inches between rows of racks.

Let's assume that each row is 36 racks long = 36 X 14 = 504 inches = 42 feet

Let's assume 9 rows = (20 + 24) X 9 = 396 inches = 33 feet.

Total area = 42 feet X 33 feet = 1386 square feet for one room with 9 rows of racks.

Total number of drives in that room = 80 drives per rack X 36 racks per row X 9 rows = 25,920 drives.

Let's make the math easy and assume 10 TB per drive, giving us 259,200 TB.

10 rooms would give us 2,592,000 TB -- more than enough for one day's internet data.  And 3600 rooms would give us more than enough to store a full year's internet data.  Three hotels in Las Vegas have more rooms than that, according to this web page.

https://en.wikipedia.org/wiki/List_of_largest_hotels

Bottom Line:
Google has to buy a building the size of one of the largest hotels in the world to house all the new hard disks it buys every year (or a bunch of smaller buildings that add up to the same number of square feet).  But that building - even at about $2 Billion - costs a lot less than the hard disks, so Google can afford that too.

Thats great math and analysis. If it is as you say they can easily afford it.

P.S. to just make it perfectly clear, I kinda considered this a political topic and used the 'private' section

I didn't really approach it as a nuts and bolts technical issue per se, although I can see from the  analysis above that is a completely valid technical discussion, and thanks for steering this in a productive direction.
I think we should consolidate things into a single section without sub-forums. Just IMO.

Maybe. Maybe.

Maybe not.

I still think this discussion raises more questions than answers.

Where are they getting the funds to run this operation? It is expensive. who would pay them to store data on every gulag user in the world?