How big data came about!

All of us know that Big Data is the next big thing in the technology world. It is still a new field and a lot has been written on how Big Data can help the world in many ways. However, when I was reading about it, I became curious about what it really means and how it came about. Why is it happening now? Why are we suddenly hit by this craze on Big Data? Here’s what I discovered.

Big data defined

According to Wikipedia, Big Data is “a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications”. The reason why this data is hard to process using current technologies is because it is huge, is growing very rapidly and is of many kinds. Today, we are generating 2.5 x 10¹⁸ bytes of data on a daily basis. 90% of today’s data has been created in the last two years alone. In fact data is doubling almost every 18-24 months! This data is varied and comes from all kinds of sources –mobile, social media, sensors on devices, pictures, videos, transactions and many more. But how come we are suddenly moving towards such large amount of data creation?

How big data came about: Moore’s law + Parkinson’s law = Big data

Actually Big data is not really a sudden phenomenon. It is taking place because of advances made 60 years ago. Two laws help us understand why Big Data is taking place now. First, Moore’s Law, states that hardware power and capacity doubles every 24 months (it is now believed to be 18 months and this will continue till at least 2020). Second, Parkinson’s Law, states that Data expands to fill the space available for storage.

This means that as the ability to generate, store and process data improved, we generated as much more data and this process will continue. So if a computer had 1KB of memory 40 years ago, with doubling every 18-24 months, it became 1048576K or 1GB today! No wonder, data creation is doubling every two years! This is also known as Moore’s law of Big Data. If you notice, the data creation in your own organization would have doubled in the last two years and it will be twice more in the next 18-24 months! What are your plans to deal with Big Data?

Conclusion

Now that we understand why Big Data has become a reality today and at what rate is it growing, we can seriously start considering our next steps to be part of this revolution. A philosophical

 question also arises out of this realization  It seems nothing is ever enough for us. The more resources we get, the more we use them. Is there an end to our needs and desires? Are we going to become slaves of big data?