Marc Andrews Observations On Demand

Why You Should Understand What's Going On With Your Data Warehouse

Data warehouse offerings are increasingly adding new tools to help manage and re-configure your warehouse with the goal of increasing performance.  In fact, IBM recently announced new "extreme" workload management capabilities as part of its recent DB2 Warehouse release to go beyond priority based workload management and enable companies to truly address the requirements for operational BI applications that must reliably deliver real-time responses.

However, to best take advantage of these tools, it is important to understand exactly what's going on with your data warehouse.  Many organizations are constantly re-configuring their data warehouse environments in a trial by error approach, blindly re-partitioning their data, setting up new nodes and defining various workload classes.  The best way to use these tools and make configuration changes is to first understand what data is being used, how it is being used, and by what users and applications.

IBM just introduced a new suite of tools to help organizations better understand how their warehouse is being used, and leverage that knowledge to improve performance, making it easier to grow their data warehouse environments.  This will not only provide better performance and greater scalability, but will help enable companies to use their enterprise data warehouses for operational BI applications.

A recent TDWI article provides some details on this.

The Convergence of BI and BPM

The convergence of BI and BPM is actually not new.  Forrester wrote about it 2 years ago and talked about how BPM products needed to integrate with BI "to support decision-intensive business processes like order management, credit risk management, and sales opportunity management," but that BPM vendors were not currently thinking about BI in this context.  Well, the market seems to be finally catching on.  Forrester is working on a new report on this topic, and there was just a great article posted on TDWI on The Myths of BPM, where they go as far as stating that "you can’t implement BPM or scorecards without a solid business intelligence and data warehousing infrastructure."

I recently participated in an ebizq podcast on BI in BPM and one of my colleagues brought up a great question about how IBM is delivering products to enable this convergence, and just as importantly, whether or not we had any customers that are really doing this.

This is actually a big focus area for us, and a major part of our new Dynamic Warehousing initiative.  Our belief is that the next generation of BI is about delivering information and business insights directly into operational applications as part of real-time business processes and activities.  This will enable a broader set of people across the organization to start leveraging BI, enable organizations to use BI for improving everyday business decisions, and drive up the value of their existing investments in related information infrastructure.

We are excited about this trend because we believe that our warehouse offering, DB2 Warehouse, is better suited to enable dynamic warehousing capabilities and address the demands of this next generation of business intelligence.  In addition, it drives requirements for a broader set of information management capabilities, which IBM is best positioned to deliver through our broader information management stack, including Information Server for information integration, WebSphere Customer Center for master data management and our industry data models to apply industry specific business perspectives.  Of course, with the need to deliver information and analytics directly into operational processes, and use that information as part of the decision making process, this will also drive greater demand for process management capabilities, which IBM can provide with WebSphere Process Server for transactional processing and IBM FileNet P8 BPM for content related processes, along with related BPM tools, such BAM and rules.

And in response to the question about real customer examples, I'm happy to say that we have several already.  One is Blue Cross Blue Shield of Tennessee, a large health care insurance company in the US.  They have moved from just reporting on customer issues to delivering insights on prior customer problems directly into their call center application so that a customer service representative can better understand and resolve the customer issues.  The Real Time Crime Center at the New York City Police Department (NYPD) is another great example, and there is a great video on this success story.  They are delivering relevant information to police detectives while they are being dispatched to the scene of a crime so that they already have a list of leads, potential suspects and their possible whereabouts before they even get there.  A complete case study is also available.  Kookmin Bank, in Korea, is an interesting example of a company that justified their initial investment based on the need to address Basel II requirements, but decided to analyze customer risk as part of its credit application process, as opposed to afterwards.  As a result, they were able to reduce credit processing errors by over 30% and improve the overall credit quality of their loan portfolio to better compete in their lending business.

So as you can see, the convergence of BI and BPM has finally arrived.  IBM is helping enable this convergence through Information On Demand initiatives such as Dynamic Warehousing.  And this is already proving to deliver real business value and allowing companies to achieve greater returns on their investments in information management and business process management technologies.

The Convergence of BI and [Insert Acronym Here]

You know a technology is hot when everything is talking about "converging" with it!  I think we all recognize that business intelligence is a hot topic right now.  According to a Merrill Lynch CIO survey, business intelligence and data warehousing was the top spending priority for IT.  And recently we've seen articles and blogs talking about the convergence of BI and Search, the convergence of BI and CRM, the convergence of BI and Performance Management, the convergence of BI and Web 2.0, the convergence of BI and SOA, the convergence of BI and enterprise applications, the convergence of BI and operational business applications, the convergence of BI and BPM, and so on and so on.

We get it!  Everything is trying to converge with BI!  I will try to comment on the reality of some of these intersections in future posts, starting with the convergence of BI and BPM.

What Exactly is the IBM Balanced Warehouse?

IBM recently announced the introduction of the IBM Balanced Warehouse, and there have been a few questions about this and how it is related to the BCU (Balanced Configuration Unit) and our recent Dynamic Warehousing announcement, so I thought I'd try to explain what it is...and what it isn't.  Let's start off with a quick recap on Dynamic Warehousing.

Dynamic Warehousing is about providing extended capabilities and infrastructure to enable third generation BI, which can be used to improve real time business processes and create competitive advantage. DB2 Warehouse is at the heart of IBM's solution for Dynamic Warehousing with its unique "best of both worlds" architecture that provides the benefits of an OLTP based, transactional data server, along with dedicated warehousing capabilities that enable it to better handle larger data volumes and more complex queries and analytics. This, along with the embedded analytics and support for extracting knowledge from unstructured information, make it the ideal platform for dynamic warehousing.

The IBM Balanced Warehouse is something completely orthogonal, and a new set of offerings that can be leveraged to simplify and optimize deployments of Dynamic Warehousing through an appliance like approach. It is the evolution of our BCU methodology to provide greater simplicity and supportability, additional entry points for more flexibility, and optimized performance by combining IBM hardware, storage and software as all encompassing, out-of-the-box building blocks that can be used to address an organization's enterprise wide or departmental warehousing requirements.

The IBM Balanced Warehouse comes pre-configured, pre-tested and pre-certified. It can be considered an appliance with benefits and there are several advantages to this solution. In addition to the drastic reduction in deployment time, we have found that a majority of customer problems or performance concerns with their warehouse are a result of configuration issues - most of which we have found to be resolved when customers deploy an IBM Balanced Warehouse. In addition to the resulting reduction in support issues, IBM Balanced Warehouse customers get a single number for complete solution support - hardware, storage and software. However, unlike most appliances, IBM provides multiple offerings that can address the entire spectrum of data warehousing needs - from large enterprise data warehouses, to high performance, departmental data marts, to packaged data warehouse solutions for small and medium sized businesses.

The IBM Balanced Warehouse actually came out of our initial efforts to provide customers with a methodology for deploying individual warehouse server nodes - balanced configuration units (BCUs). There have been several customers that have taken our BCU methodology and customized it to match their existing hardware and/or storage assets. This methodology helps them reduce the overall deployment time, but was just an initial step.  Customers purchasing a true IBM Balanced Warehouse experience even greater reduction of support issues, and benefit from having a pre-configured, pre-tested solution.

There are currently a lot of companies that have leveraged the BCU methodology, but are not necessarily "IBM Balanced Warehouse" customers - mainly because the BCU methodology has been around longer than the IBM Balanced Warehouse. However, we expect this trend to change.

Of course, we recognize that some customers will continue to have biases towards certain infrastructure components, and we will continue in true IBM fashion to maintain openness and support alternative hardware and storage devices. However, customers should clearly understand the additional value and benefits of a "true" IBM Balanced Warehouse.

Data Warehouse Appliance Vendors - Are you willing to risk your business?

Data warehouse appliances seem to be a growing trend, and a few small vendors, such as Netezza and DataAllegro, took in a bunch of venture capital and began shaking up the market with low cost, alternative solutions.  They have been competing on "good enough" functionality - I'll admit that they enable fast query and reporting, but, as they have admitted, they don't have all the bells and whistles of warehousing solutions that have been around in the market - but they have really differentiated on their low cost and simplicity.

Now, as Netezza announces their intentions to go public, we get our first glimpse into their actual financials.  And as told in the recent TDWI article, In Search of...DW Appliance Profitablity, and in Computerwire's Data Warehouse Appliances: a Profitable Venture?, the story is quite telling.  It is now clear that they have been overly aggressive in their attempts to "buy" customers, but have not truly figured out how to deliver a successful profitable warehousing solution.  As most of you know - and hopefully those of you that are buying  business intelligence platforms already know this - a company cannot survive if they continue to lose money and do not have any profitable products.

Netezza lost $14 million in fiscal 2007, $20 million in 2006, $7 million in 2005, almost $14 millin in 2004, and $17.5 million in 2003.  That's over $70 million lost in 5 years...and they only have around 70 customers! Thus, one of two things is going to happen.  Either these companies are going to have to start charging more money, and lose one of their key benefits - lower cost - OR, they will go out of business.  Either way, when you look at their financials, you should be extremely worried about investing in their products, unless you are just looking for a very short term solution that you are willing to throw away in a few years.

So the question is, are you willing to risk your business on this type of solution?

The State of Business Intelligence

David Stodder just did a great write up on The State of Business Intelligence in a feature story in Network Computing that gives a fairly complete vision of where the industry is heading, and more importantly, what organizations really need from the technology to gain business value.  The bottom line identified in this article is dead on - "Conventional data warehouses, disconnected BI and analysis tools need an overhaul. Actionable intelligence and speedier, automated decision-making are the objectives of next-generation BI."

These were the exact drivers behind IBM's dynamic warehousing strategy announced just two weeks ago.  As the Network Computing article (cleverly titled "Good BI Cruel World?") points out, "next-generation BI promises speedier, automated decision-making" and points out that most existing BI platforms are "mired in historical analysis across siloed back ends."  The key is that analytic capabilities must be injected directly into all of your different activities and business processes, and must be able to leverage ALL enterprise information across the organization, including all types of information (e.g. structured, unstructured and semi-structured).

I was talking to Mark Beyer, a Gartner analyst, about this at a recent Gartner BI Summit, and he gave us an interesting suggestion that we stop talking about our solution as a "warehouse" and start referring to it as an "Analytic Information Repository."  Given IBM's propensity to leverage acronyms, I'm not sure attempts to market AIR would go over so well, but this is in essence what we are trying to turn warehouses into - a platform that can be deployed to enable analytics on all relevant enterprise information.  Companies need technology that will enable all users across the enterprise to more easily access the information and generate the insights needed to make better decisions, faster.

I still haven't figured out what to really call this, although from initial feedback, such as Vincent McBurney's extremely entertaining post titled IBM puts the shizzel into the Data Warehouse, people seem to like Dynamic Warehousing and the new IBM Balanced Warehouse.  However, it's important to note that unlike one of the comments on that post, you can NOT achieve these types of benefits with standard databases and reporting tools.  Without dedicated warehousing capabilities, such as shared nothing parallelism and workload management, you will not be able to achieve this vision.  And even some of the new "appliance" solutions to hit the market, while providing affordable, optimized query speeds, will not be able to support the broader analytics requirements.  As highlighted by David Stodder, "appliances are customized for a specific purpose; they are not meant for mixed workloads, such as OLTP plus complex data querying, or even analytical workloads that require specific tuning to meet unusual objectives."

The next generation of BI will be what finally delivers the long promised benefits of BI to the masses.  And Dynamic Warehousing will be the AIR that enables companies to finally start realizing significant business value from their investements.

Dynamic Warehousing and the Next Generation of Business Intelligence

This past week, IBM announced a comprehensive strategy to enable dynamic warehousing, a new generation of business intelligence capabilities that enable organizations to gain real-time insight and value from their business information.  So what exactly is dynamic warehousing?  And what is this next generation of business intelligence?

Well, dynamic warehousing is about making information AND analysis available as needed to address ALL of the different demands organizations have for business intelligence.  This includes the traditional needs for historical reporting and analysis, but also extends to generate increased business insights from all information across the enterprise, and enables that knowledge to be delivered in real-time when needed - often referred to as operational intelligence.  Dan Everett, at Ventana Research has been talking about this concept and some of the challenges organizations face in achieving this lofty goal for over a year.  Let' s break it down by looking at what companies started out doing with BI tools to better understand what this next generation of cabilities is about.

Initial efforts at generating business insight were focused on query and reporting to understand what happened.  Police agencies reported on crime statistics to learn about frequent criminal activities, insurance companies reported on fraud to determine how much money to write off, and retailers reported on store sales to find out which locations were underperforming.  The second wave focused on technologies such as online analytical processing (OLAP) and data mining for historical analysis to understand why things happened and recommend future action – strategic and tactical planning.  Police agencies began analyzing criminal activity to optimize deployment of personnel, insurance companies analyzed characteristics of fraud for better customer segmentation, and retailers analyzed customer purchases to improve merchandising and better manage inventory.

Dynamic Warehousing provides a new approach to companies who want the ability to rapidly analyze and act upon the hidden benefits of their business information.  This third generation approach is about making available and analyzing information on demand to optimize each transaction – in the call center, in the field, when helping customers, or when taking orders.  Dynamic Warehousing is helping the New York City Police Department (NYPD) identify related incidents upon receiving an emergency call and deliver a list of potential suspects in real-time to detectives before they arrive at the crime scene (see the video).  Likewise, insurance companies can identify potentially fraudulent claims prior to approval and payment, while retailers can rapidly leverage buying patterns and changes in consumer behavior to identify the most effective cross sell and up sell opportunities at the point of sale.

There are four key things required to enable dynamic warehousing:

1. Real-time access, in context - being able to access and work with aggregated, cleansed information in the context of the business process or activities being performed
2. Analytics, as part of the business process - going beyond just providing access to information and providing embedded analytics capabilities that can be leveraged as part of a real-time business process
3. Incorporating knowledge from unstructured information - leveraging all types of information, including the ability to extract additional insights buried in unstructured forms of content, such as free form text fields, documents, web pages, audio recordings, etc.
4. Extended capabilities, tightly integrated - providing a complete set of integrated capabilities that extend beyond the warehouse to enable use of information on demand.  This includes search and text analysis, information integration, business process management, enterprise data modeling, master data management and industry specific models to make these technologies more applicable to a particular business area.

These are the key tenets of dynamic warehousing.  And these are the things needed by organizations to further innovate their businesses and drive the next level of competitive differentiation.

How Does MDM Relate to Data Warehousing

I just got this question from an analyst, so I figured I'd just post my initial thoughts here.  There are several nuances and areas where the technologies can compliment each other, but for this particular response, I'll keep the story pretty straight forward - when you're analyzing and reporting on aggregated information, if you want a real, single version of the truth, you have to have greater consistency in the customer and product data that is being captured from your different backend systems.

While information integration tools provide mechanisms for cleansing data, they don't necessarily provide tools for resolving differences in facts between different systems.  MDM solutions address this challenge by helping establish that single truth and ensuring that your different operational systems have a consistent view on customer information (such as an address, phone number, etc.), product information (SKUs, product names, etc.) and any other entities that require management of master data.  MDM capabilities can help ensure that the information in your warehouse, which is being used for analytics and reporting, is more accurate and can be trusted.  This can dramatically impact the ability to ensure that the appropriate decisions are being made, and becomes even more critical when organizations begin using this insight for operational decisions and activities.