Are you ready for Big Data Analytics? Reply

Just like any technology cycles, there is widespread interest and hype in Big Data. It is probably most talked about subject (although maybe it’s second to Cloud) in IT circles. I want to acknowledge the importance of a big data strategy. I would rank right up there with ERP –once upon time (be sure to read my blog post on ERP).  Let’s put it this way: Big Data is in its first inning and is playing to win–it still maturing! This is also a wake up call for organizations to take control of their internal data. What is the relationship? Let’s start with understanding basic concept and this is directly from Wikipedia: Business Analytics (BA) refers to the skills, technologies, applications and practices for continuous iterative exploration and investigation of past business performance to gain insight and drive business planning.[1] Business analytics focuses on developing new insights and understanding of business performance based on data and statistical methods. In contrast, business intelligence traditionally focuses on using a consistent set of metrics to both measure past performance and guide business planning, which is also based on data and statistical methods.

Analytics have been used in business since the time management exercises that were initiated by Frederick Winslow Taylor in the late 19th century. Henry Ford measured pacing of assembly line. But analytics began to command more attention in the late 1960s when computers were used in decision support systems. Since then, analytics have evolved with the development of enterprise resource planning (ERP) systems, data warehouses, and a wide variety of other hardware and software tools and applications.

There are plenty of examples available of Analytics enabled companies that save substantive amount of money by understanding the data and using analytical models to gain competitive advantage. Banks like Capital One for instance, use data analysis (or analytics, as it is also called in the business setting), to differentiate among customers based on credit risk, usage and other characteristics and then to match customer characteristics with appropriate product offerings. Harrah’s, the gaming firm, uses analytics in its customer loyalty programs. E & J Gallo Winery quantitatively analyzes and predicts the appeal of its wines. Between 2002 and 2005, Deere & Company saved more than $1 billion by employing a new analytical tool to better optimize inventory. There are many more examples, but these are still very limited if you consider entire corporate world. The above mentioned examples are also very specific and limited in nature. This goes to show that we have yet to exploit the potential of Analytics.

So what is stopping us? First and foremost, it’s our understanding of what we are looking for. We have been treating Business Analytics as a reporting mechanism which is reflective in the type of resources we hired to the training we provide in management schools. Do you remember how many statistics classes were mandatory during your MBA? This is exactly my point! Secondly, we have literally let data get out of our hand. It is too distributed and we don’t know how to bring it together. Lastly, the quality of data itself is a big issue for organizations. Enterprise Resource Planning was supposed to solve this problem by consolidating all front office and back office applications together in single global instance. But we all know that despite of the advancements in functionality and scope of ERPs, we still have several supporting applications (some of them are integrated and some not) and data that resides in them. Then the industry came up with service oriented architecture. A very sound concept and I would put it on equal footing to advent of personal computer (it was different way of thinking about integration and applications). However, we ran into several technical challenges, standards challenges, governance challenges etc. instead of solving problem it created new data type called metadata. We tried to use master data management and data warehouse techniques, but it received partial success.

So now what? As we are trying to get handle on our data, the consumers started getting more and more savvy and willing to share more data in exchange for a personalized service. This is from where concept of Big Data came along. Notice that I’m stating “concept” and not software. Concept is very simple. It is verity, velocity and volume of data has grown exponentially.

So what should we do? Does investment in technologies and integrated platforms like Big Data Appliance from Oracle will provide value? Technology by itself will not. The example you may be familiar with in your organization is Master Data Management initiative. if your organization has gone through it, you will know what type of coordination and collaboration it takes to really realize of MDM benefits. those lessons learned are applicable here. Organizations need to start taking control of Big Data NOW. Here are four good steps to follow:


  1. Understand the data in the context of application. This can be made more manageable if organizations first go through Application Rationalization (read my blog on application rationalization).
  2. Prioritize and sanitize internal data. Try to standardize the data definition across organization. It is difficult said than done.
  3. Bring in your assets that can give you insight into your customers, products, and what is competition is doing. This will require people from different part of organization to come together and really model out what do we want to learn.
  4. Find talent (the most difficult part) that understands statistical modeling. Make sure they understand your product set and have complete understanding of your customer and industry. I think the team approach is very effective including a team statistician, business expert, strategy expert and customer expert. Give the team time to fully understand and model internal data.
  5. Start integrating external data. This is where organizations need to provide tools that can support modeling of “Big Data.” In this context, Big Data is the combination of internal data (structured and unstructured data) and external data (social networks, internet, search trends etc.). This is when you will start to realize the real value of “Big Data Analytics”—when you will be able to contextualize someone’s comment on a Facebook page to your internal data and predict how you can gain customer or personalize your product to an individual. This is very powerful.

Depending on the size of the organization, the complexity of the customer base, depth and width of product set, proliferation of data etc., will determine how long it will take to implement and get ready for Big Data Analytics. It is critical to start now! You don’t have to wait to start realizing benefit until everything is done. You can strategize the phases and thoughtfully determine the slices of data that need to be exploited. Be prepared for some miss-steps and miss-interpretation. That is why it is important to test the hypothesis before trusting the result. Business analytics should be used in the business decision-making process, not just as reporting tool.

For IT, remember, Hadoop by itself is not enough. Big Data Analytics requires to contextualize unstructured data from Hadoop using traditional RDMS database, data warehouse and finally put analytical models using, analytical language (like Oracle R Enterprise) and visualization tools. You need to integrate all these technologies and make it available to the team in order for them to make most efficient and effective use of data. (I also recommended reading: like Service Oriented Architecture ( SOA) it will take some time for us to fully realize the benefit and fully understand the power of Big Data.

Do you consider application rationalization part of your annual cycle? 3

Information Technology and its applications have become a critical part of today’s business operation. In many companies, CIOs have seat at the table to help determine business and operation strategy to gain a competitive advantage. Equally, CFOs and CEOs are participating in IT decision-making process and are no longer just signatures on the paper. Even the consumer has become IT savvy with the advent of mobile devices and tablets that eliminated the fear if computer for all generation. My father uses his iPad to perform most of his banking tasks, his review of MRI /CT Scan reports and to manage his portfolio. This ‘IT consumerization” has now forced C-Levels to pay attention to how IT in integrated in the business.

You can also see that result in today’s world overall. The uptake of technology has become instantaneous (and many times, not by choice)–as a necessity. Hence the problem. In the race to keep up with technology changes and increasing customer demands for more access, companies are deploying technologies very rapidly. And within that process of doing so, their application portfolio is growing rapidly as well. I have visited many government and commercial companies in this last year alone and I have personally witnessed this as a common problem.

Let me try to put this in context of everyday life.

I don’t know how often you look in your closet–I did it first time in 10 years just the other day. I had shirts and pants in there that I hadn’t worn in those 10 years. I have never really thought about getting rid of those clothes, but now the situation is that there is no more space (although my wife’s clothes do contribute to the problem which, of course, is beside the point and so there is nothing you can do about that). The only option I had was to discard some clothes or buy new house with bigger closet!

Now you may think that buying new house for clothes is an absurd thought, but think about what is happening in companies today. With advent of cloud services (especially infrastructure as a service), it is now very easy to add infrastructure and storage (no it’s not like buying a new house, but it’s certainly akin to renting a storage unit just to store the overflow of a closet full of clothes).  Cloud services have allowed me to continually add new technologies and applications. And just like I reached the threshold in my personal clothes closet, a great number of companies are going to soon realize (if they haven’t already) that the IT bill is only going to go up–not down. So the cleaning out of “the closet” will soon be a must in most cases.

CIOs are now struggling to keep control over proliferation of data and security of data. It is an increasing number of controls from an SOX perspective. Does this sound familiar? So what is the solution? Conceptually, the solution is very a simple one–provided in the “closet example:” I went through my closet, made three piles of clothes. One pile was the clothes I wear all the time. The second was a pile of clothes that I wear some of the time depending on occasion, and the third was a pile of clothes that I did not wear. I kept the first and second pile. I made another run through the third pile and took out couple of items that had sentimental value and donated the rest. I have also promised myself that “cleaning out my clothes closet” would now be an annual exercise. Naturally it is very easy in this case (of the closet), but in the case of application, the same concept also needs to be applied.

Application Rationalization is a key component for organizations and for every CIO so that he/she can ensure the organization’s IT is performing its role to stay in synch with its competitive positioning in the business world.  There are also two things that make IT application rationalization very difficult.  Organization Change and Data Proliferation. However, it’s similar to being able to ride a bicycle. Initially it’s difficult to implement, but once you have the processes in place and governance structure in place it will become much more natural (notice I did not say it was easy!). In my opinion, this needs to be part of every organization’s annual budget cycle. This is an area where the other C-level managers need come together with the CIO to perform the analysis.

In the age of Cloud Computing and Modern Application, be aware that you may have application that is mission critical and is still running on older platform. Application modernization methods are the needs of the moment and it is important to take a very systematic and detailed, but (at the same time) very agile approach to portfolio rationalization. Application modernization methods also combine application modernization as well as new technology selection/incorporation as part of a balanced approach to application rationalization. Gartner’s Pace Layered Application Strategy provides a good explanation of this and is well worth the read. This is really a very simple concept to understand, and fairly easy to learn how to segment an application portfolio and go through rationalization process. It is how I decided to make three piles of clothes from my closet.

I suggest that you keep your organization clutter free. Keep the focus on your consolidation and modernization approach while embracing advances in technology. This can be made possible by taking control of ALL APPLICATIONS and DATA (modern and legacy) and making those items part of an annual cycle to rationalize and consolidate. If done correctly (the right methodology, governance and discipline), this annual or regular task will provide your organization with a focused ability to be able to keep up with ever-changing IT landscape (if not full, at least partial). More importantly, IT will become your tool for growth and not burden on the organization.