We are constantly flooded with data today. In 1992, 100 gigabytes of data was generated per day and the number is going to reach around 50 terabytes per second by 2018. However, it's only 0.5 % of the data which is generated is actually used and analyzed. We are sitting on a goldmine of data where 90 percent of all the data produced is unstructured. So, how are we going to tap into this opportunity?

Earlier it was thought that finding value in big data was like finding a needle in ten thousand haystacks. But the real issue is not just looking for a needle in multiple haystacks, but that we are collecting and comparing every possible attribute of the data. Then comes the toughest part of analyzing, visualizing and finding value in it.

Big Data Analytics is a new ball game for a lot many industries but not for the oil and gas industry. This industry has, for the longest period of time, dealt with large volumes of data on which they are heavily reliant to make their business decisions.  For instance, let's take up an issue that drives 3.5 percent of the world's GDP – 'Corrosion'. It has a huge economic impact on all types of infrastructure. Corrosion also poses a threat to human safety, disrupts industrial operations and can cause major environmental damages. Fundamentally, corrosion is the deterioration or destruction of metals and alloys in the presence of an environment by chemical or electrochemical means. It is a major cost section with over US$2.2 trillion being spent on it globally every year.

Pipelines are the safest and the most efficient way to transport energy. Crude oil, natural gas, petroleum products are transported through thousands of kilometers of pipelines across the world. Most of the times, pipeline failures are prevented by evaluating, inspecting and maintaining the assets through integrity management. One such component of this management is in-line inspections. Thickness gauges and other tools commonly knowns as 'smart pigs' are used to conduct these inspections. These devices are inserted into the pipelines such that the fluid flow within is not disturbed. Corrosion or cracks that could lead to explosions, leakages or ruptures can be assessed using these devices. The devices and sensors generate vast volumes of data in varying formats and with varying degrees of accuracy. With analytics used on the data, a trail or pattern can be formed thereby easing out maintenance regimes and risk management.

Previously there was a certain level of conservatism in the oil and gas industry with respect to data management. Role of experts has always been of a paramount importance and their interpretation of what is happening in the field has been relied upon. Given that the age of easy oil is almost coming to an end, there has been a great emphasis on production automation.

The oil and gas industry has been around for a long period of time and a large amount of data has been collected as result. One of the key considerations in this industry is that an organization loses skills, experiences, and people out on a timely basis. So it has to be able to institutionalize the knowledge that has been there in the historical data. Say, how particular reservoirs react in given conditions with the geochemical composition in the area. These learnings can be a part of a statistical model and will help next time when someone has to function in the area with a certain chemical footprint. The experienced hand will not be in the field anymore. Hence, we can safely say that today's production data is tomorrow's legacy data.

An organization should run on a scientific basis rather than assumptions/predictions. A lot of business decisions in this industry are made solely on the basis of experience. Now oil and gas industry doesn't necessarily have the luxury to do that anymore. First, too much data is generated today. Secondly, mistakes cannot be afforded, implications will be too great for the organization at any level. Organization and management of data in real time & usage of predictive analytics to help decision making have become the way for an organization to run.

The three most important areas that analytics is impacting are:

•  Geological composition – Companies can come out with accurate characteristics of shale basins by comparing well performance data against surface geology.

•  Drilling/ Mining – The number of wells and lag time can be minimized with better analytics.

•  Field optimization – Collecting and analyzing massive data of different types can improve a company's ability to improve and optimize their drilling parameters.

The below infographics by IBM sums up the above mentioned points.

Since the oil and gas industry has a complex, legacy IT system, it has a problem with finding the right analytic talent which is now a big need to keep up with the competition. The only solution to this problem is the collaborative effort of senior executives along with leaders to find the right people who can balance industry knowledge with best analytical skills. It's probably high time that the oil industry maps out its ambitions against what it is actually capable of.