The Big Data Era

There’s a lot of buzz about Hadoop Architecture and Big Data  and how it can help organization do a real time analytics . Few weeks back I read an article somewhere in the internet which says the word “Hadoop“ is actually a name of the creator's son's toy elephant.

There is this 166 pages  book about Understanding Big Data that was shared across our organization. Having gone through the “Contents” and “About the book section” plus some few pages on this book, I  think I could understand at least what the book is trying to say and what they meant when they say “Big Data” or "Hadoop". Frankly speaking, I would myself have though about using an unstructured data if I was to start my own small scale business. But the mechanism I will be following would be the hard way. I would have tried to find a trend in most of the social networking site on what is the latest buzz on peoples mind and what are the things that concerns people the most ( I believe that’s what all these marketing and sales people do in present days). With this method I will just be focusing on a very small part of the unstructured data and make my decision based only on those small information which may be not that effective.


Well, the point I was trying to make here is “The World always finds some complex ways to do things which at times looks like an impossible thing but here it is. Big Data.. Its big.. they say....”. I couldn’t imagine what kind of algorithms are developed(or will need to be developed ) to do a real time analytics on unstructured data. The way I understand, in the next few paragraphs I will try to make the readers try to visualize how things could possibly work in the near future.



With the ever increasing improvement in communication and networking technology information moves real fast. Facebook (for example) has been a preferred choice of social networking and information sharing at least in my state. It has become a de-facto standard. I could actually see how facebook changed a very small Indian state Mizoram in the last 12 – 18 months. There was a Facebook group which is named “Special Report”. This group was created around April-May 2012 as a platform to share news and incidents about Mizoram the moments it happens. The members of these groups shared the local and Global news in as good as a real time. To know what is going on in which part of Mizoram, these group gives you the most updated information. In just 3-4 months these group has approximately 50000 members (~5% of the State population) and its growing. This can be attributed to the rise in the Internet Services provided by Mobile Network providers.

 For example, there was a minor accident in some part of the state in exactly 11.00am, This news about the accident is posted in another 2 minutes in Facebook with photos and the comments, suggestion to avoid accidents keeps flowing. In another 30 minutes I can expect to see the information and life history of the person who actually met this accident and these information shared are mainly by the first person view. This is the velocity of how information flows even in a small part of this huge country. Imagine there could be a lot of useful information which just flows but is not used for analytics even in  the rest of the world.

The news or any post written in facebook or any social networking site is an unstructured data. Lets us try to see a scenario in a "Big Data Era" on what we can expect in our daily life ( it can be weird...):

A person sitting at home in a remote village in Madurai updated his status :
                “Wish Samsung manufacture high end mobile devices which supports internet, 3G and wifi  without Camera....”

Another updates somewhere in Senapati Bapar Road, Pune says
                “Aarrrgghhhh… Bad Day.. My Company security guy confiscated my Samsung Champ Phone even though I have removed the embedded camera.. really need some high end phone without camera. Any suggestions..!! “

In the Big data Era, above unstructured data can be useful for mobile manufacturers. They will be able to dig in to the unstructured data and may see a  new market segment- High End Mobile phones without camera. They started manufacturing the devices and yes just as you must have imagined.. I was the first person who purchased it..

Another person was hospitalized for a minor medical reason but has to pay a huge medical expense. His facebook status says…
                “ What has happened with the Medicine Business.. Wish I had subscribed for a medical insurance”..

Few minutes after this update, Royal Chandaram - an Insurance Company contacted this guy and provide him all the medical plans. The guys decided to purchase one Insurance Plan and now both the Insurance Company and that guy are a happy.

There has been few instances where an incorrect information was posted in this Special Group which ignites a spark between some government department and the the individual who posted it resulting in an arrest of that individual. This can be a good example where big organization (who spent decades to build their brand) can monitor if any of these unstructured data or facebook status updates, blogs or anything in the interconnected online world are ruining the very thing the organization boast about-“The Brand”. If this organization could catch these unstructured data in almost real time, they could possible save themselves and their brand before it has a negative impact into their balance sheet.


If this is what is Big Data , then by now we could may have imagined what are its powers and what can it do. I am still getting into it. However, I couldn't stop to go ahead and write this article because I want you to be a part of this Big Data from the start and suggest you to spend time following it and learn about it.

The world moves fast because Information flows fast. Things are changing. The things that mattered 10 years back does not necessarily matter now. So, if they say Big Data Era has started, I am pulling up my socks to be a part of it and to grow with it.