Just like any technology cycles, there is widespread interest and hype in Big Data. It is probably most talked about subject (although maybe it’s second to Cloud) in IT circles. I want to acknowledge the importance of a big data strategy. I would rank right up there with ERP –once upon time (be sure to read my blog post on ERP). Let’s put it this way: Big Data is in its first inning and is playing to win–it still maturing! This is also a wake up call for organizations to take control of their internal data. What is the relationship? Let’s start with understanding basic concept and this is directly from Wikipedia: Business Analytics (BA) refers to the skills, technologies, applications and practices for continuous iterative exploration and investigation of past business performance to gain insight and drive business planning. Business analytics focuses on developing new insights and understanding of business performance based on data and statistical methods. In contrast, business intelligence traditionally focuses on using a consistent set of metrics to both measure past performance and guide business planning, which is also based on data and statistical methods.
Analytics have been used in business since the time management exercises that were initiated by Frederick Winslow Taylor in the late 19th century. Henry Ford measured pacing of assembly line. But analytics began to command more attention in the late 1960s when computers were used in decision support systems. Since then, analytics have evolved with the development of enterprise resource planning (ERP) systems, data warehouses, and a wide variety of other hardware and software tools and applications.
There are plenty of examples available of Analytics enabled companies that save substantive amount of money by understanding the data and using analytical models to gain competitive advantage. Banks like Capital One for instance, use data analysis (or analytics, as it is also called in the business setting), to differentiate among customers based on credit risk, usage and other characteristics and then to match customer characteristics with appropriate product offerings. Harrah’s, the gaming firm, uses analytics in its customer loyalty programs. E & J Gallo Winery quantitatively analyzes and predicts the appeal of its wines. Between 2002 and 2005, Deere & Company saved more than $1 billion by employing a new analytical tool to better optimize inventory. There are many more examples, but these are still very limited if you consider entire corporate world. The above mentioned examples are also very specific and limited in nature. This goes to show that we have yet to exploit the potential of Analytics.
So what is stopping us? First and foremost, it’s our understanding of what we are looking for. We have been treating Business Analytics as a reporting mechanism which is reflective in the type of resources we hired to the training we provide in management schools. Do you remember how many statistics classes were mandatory during your MBA? This is exactly my point! Secondly, we have literally let data get out of our hand. It is too distributed and we don’t know how to bring it together. Lastly, the quality of data itself is a big issue for organizations. Enterprise Resource Planning was supposed to solve this problem by consolidating all front office and back office applications together in single global instance. But we all know that despite of the advancements in functionality and scope of ERPs, we still have several supporting applications (some of them are integrated and some not) and data that resides in them. Then the industry came up with service oriented architecture. A very sound concept and I would put it on equal footing to advent of personal computer (it was different way of thinking about integration and applications). However, we ran into several technical challenges, standards challenges, governance challenges etc. instead of solving problem it created new data type called metadata. We tried to use master data management and data warehouse techniques, but it received partial success.
So now what? As we are trying to get handle on our data, the consumers started getting more and more savvy and willing to share more data in exchange for a personalized service. This is from where concept of Big Data came along. Notice that I’m stating “concept” and not software. Concept is very simple. It is verity, velocity and volume of data has grown exponentially.
So what should we do? Does investment in technologies and integrated platforms like Big Data Appliance from Oracle will provide value? Technology by itself will not. The example you may be familiar with in your organization is Master Data Management initiative. if your organization has gone through it, you will know what type of coordination and collaboration it takes to really realize of MDM benefits. those lessons learned are applicable here. Organizations need to start taking control of Big Data NOW. Here are four good steps to follow:
- Understand the data in the context of application. This can be made more manageable if organizations first go through Application Rationalization (read my blog on application rationalization).
- Prioritize and sanitize internal data. Try to standardize the data definition across organization. It is difficult said than done.
- Bring in your assets that can give you insight into your customers, products, and what is competition is doing. This will require people from different part of organization to come together and really model out what do we want to learn.
- Find talent (the most difficult part) that understands statistical modeling. Make sure they understand your product set and have complete understanding of your customer and industry. I think the team approach is very effective including a team statistician, business expert, strategy expert and customer expert. Give the team time to fully understand and model internal data.
- Start integrating external data. This is where organizations need to provide tools that can support modeling of “Big Data.” In this context, Big Data is the combination of internal data (structured and unstructured data) and external data (social networks, internet, search trends etc.). This is when you will start to realize the real value of “Big Data Analytics”—when you will be able to contextualize someone’s comment on a Facebook page to your internal data and predict how you can gain customer or personalize your product to an individual. This is very powerful.
Depending on the size of the organization, the complexity of the customer base, depth and width of product set, proliferation of data etc., will determine how long it will take to implement and get ready for Big Data Analytics. It is critical to start now! You don’t have to wait to start realizing benefit until everything is done. You can strategize the phases and thoughtfully determine the slices of data that need to be exploited. Be prepared for some miss-steps and miss-interpretation. That is why it is important to test the hypothesis before trusting the result. Business analytics should be used in the business decision-making process, not just as reporting tool.
For IT, remember, Hadoop by itself is not enough. Big Data Analytics requires to contextualize unstructured data from Hadoop using traditional RDMS database, data warehouse and finally put analytical models using, analytical language (like Oracle R Enterprise) and visualization tools. You need to integrate all these technologies and make it available to the team in order for them to make most efficient and effective use of data. (I also recommended reading: http://assets1.csc.com/lef/downloads/LEF_2011Data_rEvolution.pdf.)Just like Service Oriented Architecture ( SOA) it will take some time for us to fully realize the benefit and fully understand the power of Big Data.