All
of us know that Big Data is the next big thing in the technology world. It is
still a new field and a lot has been written on how Big Data can help the world
in many ways. However, when I was reading about it, I became curious about what
it really means and how it came about. Why is it happening now? Why are we
suddenly hit by this craze on Big Data? Here’s what I discovered.
Big data defined
According
to Wikipedia, Big Data is “a collection of data sets so
large and complex that it becomes difficult to process using on-hand database
management tools or traditional data processing applications”. The reason why
this data is hard to process using current technologies is because it is huge, is
growing very rapidly and is of many kinds. Today, we are generating 2.5 x 1018
bytes of data on a daily basis. 90% of today’s data has been created in the
last two years alone. In fact data is doubling almost every 18-24 months! This
data is varied and comes from all kinds of sources –mobile, social media,
sensors on devices, pictures, videos, transactions and many more. But how come
we are suddenly moving towards such large amount of data creation?
How big data came about: Moore’s law +
Parkinson’s law = Big data
Actually
Big data is not really a sudden phenomenon. It is taking place because of
advances made 60 years ago. Two laws help us understand why Big Data is taking
place now. First, Moore’s
Law, states that hardware power and capacity doubles every 24 months (it is
now believed to be 18 months and this will continue till at least 2020).
Second, Parkinson’s
Law, states that Data expands to fill the space available for storage.
This
means that as the ability to generate, store and process data improved, we
generated as much more data and this process will continue. So if a computer
had 1KB of memory 40 years ago, with doubling every 18-24 months, it became 1048576K or
1GB today! No wonder, data creation is doubling every two years! This is also
known as Moore’s law of Big Data. If you notice, the data creation in your own
organization would have doubled in the last two years and it will be twice more
in the next 18-24 months! What are your plans to deal with Big Data?
Conclusion
Now
that we understand why Big Data has become a reality today and at what rate is it
growing, we can seriously start considering our next steps to be part of this
revolution. A philosophical
question also arises out of this realization It seems nothing is ever enough for us. The more resources we get, the more we use them. Is there an end to our needs and desires? Are we going to become slaves of big data?
