Inside IT with Evan Levy

Free Ham for $6 Per Pound (and Other Oracle Marketing Ploys)

News from last week’s Oracle OpenWorld is still flying fast and furious. My fellow industry analysts and consultants look forward to Oracle OpenWorld because it gives them material to write about: new products, client case studies, new ideas, and so forth. With all the press releases emitted from OpenWorld, Larry Ellison’s $10 million IBM challenge stands out.

For Oracle product fans, this is exciting. I’ve been accused of getting into the weeds (and analyzing the specific details) when I should just appreciate the creativity and  impact of these types of visibility ploys, but I think Larry’s challenge requires some weeding.

I credit the creativity of the Oracle folks for establishing a performance challenge. Oracle has actually come up with a way of generating revenue from current customers AND prospects.  They’ve actually figured out a way to generate revenue from situations where Oracle isn’t necessarily the optimal solution.

When the worlds of marketing and the legal collide, the outcome is always interesting.   It’s been some time since I reviewed an Oracle software license, but as I recall, the company prohibits anyone from documenting and publicizing performance metrics about their product.

But what if someone is able to take on Ellison’s challenge and refute the “twice as fast” metric?  They’ve done their diligence, used the appropriate hardware settings, configured the DBMS environment, and loaded their real-world data in a pristine fashion, and run an exhaustive set of tests.  And, let’s say, just for argument’s sake, the Redwood Shores solution doesn’t meet expectations. Ostensibly Oracle comes and inspects them the results and—if the product has indeed underperformed, pays out the award money.

Unfortunately, there’s no way for someone to document and publicize their findings without breaking their obligations under Oracle’s license agreement.  So, Oracle not only doesn’t have to pay the $10 million prize – they may actually be able to go after the prospect for financial damages for breaking the licensing agreement.

This is marketing, legal, and sales brilliance.  If you’re Oracle, you are providing incentive for every programmer with time on his/her hands to download Oracle software, load data, and run some tests to win $10 million. The worst that happens is that someone challenges, wins, and Oracle encourages the challenger to pay for software instead of paying for breaking the license.   That’s clearly a win-win if you’re Oracle. But prospective customers may not see things quite the same way.

Maybe Oracle will modify its licensing to allow customers and prospects to document and publish performance results.  Or maybe this is just a minor oversight.   I’m sure whatever the outcome—everything is always in the best interests of their customers.  What other explanation could there be?

All Business Analysts are Not Created Equal

At a client meeting recently I was informed that the company had over 150 business analysts. Even the client’s business executives acknowledged this to be true. But that many business analysts suggests that these people are spending their time on efforts outside the realm of business analysis.

Most IT organizations model the business analyst (BA) role around transactional or operational systems. Whether organizations buy packages or build code from scratch, business analysis skills are usually focused on system and process analysis. The majority of data issues within an operational system are data entry-related. The challenge in business analysis is to establish standard business processes to automate.

Few operational systems are built with the goal of data share-ability. So issues such as standard business values and definitions don’t come up. When a new ERP system is being implemented, little attention is paid to the customer id number or the customer’s name. It’s fairly common for operational systems to be built in an isolated way to support a well-known business process with no attention to data standards.

But it’s different in the BI and analytics environments. To be successful in BI, it’s critical to have integrated data from individual systems to support often-complex analytics. The BA doesn’t need to focus on the business processes that created the data—rather he or she should focus on the business scenarios that mandate accurate and meaningful data. The expertise needed is around data content analysis and understanding data from different source systems and what it represents.

To be an effective BA for business intelligence or master data management it’s critical to understand how different systems see data, even data that’s ostensibly the same. In a telecommunications firm, System A was account-based and System B was customer-based. Customer details existed in both systems. A good BA understands which data is critical from each system and what the rules are for matching records across them.

The BA needs to:

• Be able to identify different business scenarios for how data will be used
• Interested and willing to go through often-tedious analysis of content, formatting, and definitional differences in data within and across systems
• Be comfortable with the tools necessary to dig into the data and profile it
• Excel at communicating data requirements and anomalies in business language

At the end of the day a good business analyst should understand that data should be independent of applications and reflective of good business terminology. Until then, we’ll have hundreds of them and they’re likely to be forklifting data from one system to another in an on-demand, non-repeatable, non-scalable, and inefficient way.

Good Data Warehouse DBAs are Hard to Find

As a consultant I’m often asked about how roles and responsibilities should be delegated or identified within the IT organization to support the data warehousing.  One role that seems to incite discussion is the role of the database administrator (DBA). Many of our clients pack a series of roles and responsibilities into a laundry list of heterogeneous tasks. However I usually recommend delineating DBA resources and assigning individuals to either transactional systems or data warehousing, an approach that tends to surprise my clients.  Let me explain why I see the roles across these different systems requiring entirely different skills.

DBAs are typically focused on the care and feeding of the DBMS system to ensure that processing is consistent and performance is maintained regardless of the circumstances.  The DBA is typically responsible for establishing table structures, configuring database systems, and designing queries and execution jobs to efficiently utilize the system.  Users don’t like surprises; they want daily activities to complete a specific time every day. Their jobs depend on timely information.  Many IT managers assume that if a DBA understands how a particular DBMS works, that DBA can address both transactional and analytical responsibilities.  It’s a risky assumption.

Assuming that because someone can design and manage a transactional database environment, they are qualified to design and manage an analytical system is flawed. The details associated with designing or troubleshooting a sophisticated transactional system have little in common with an enterprise, cross-functional data warehouse.  

Just because someone is a mechanic doesn’t mean they can fix or repair any type of car.  The mechanic who can fix  a diesel pickup is unlikely to be able to repair the engine of an 18-wheeler. While the basic skills are the same, the actual situations and experience required to solve specific problems are dramatically different.  Just because someone is a DBA doesn’t mean they can design or support any type of application/database system.

The design of a transactional application is typically preceded by detailed transaction and data specifications.  Because a transactional system supports specific business processes, the actual transactions, data details, and processing volumes are well understood prior to development.  Most queries are single statement queries accessing individual records within a single table.  It’s critical that the workload be well defined because of the enormous costs associated with these applications.  Consistent response time is critical.  Managing a system requires attention to transaction quantities, query plans, and data volumes to ensure that data and processing is distributed across the system’s resources.  Users are often grouped by specific application (or privileges) and while processing can vary across different applications, users are usually homogeneous.  System growth occurs with more users and the additional transaction volumes.   While ad-hoc processing or table joins are technically feasible, they are rarely supported. 

Contrast that with analytical systems. These DBAs have an entirely different set of challenges when undertaking development.  Database design is often undertaken with the knowledge that the content of the database will change.  It’s not uncommon for BI systems to start by supporting a single subject area only to grow exponentially in size due to the growth of additional subject areas and data volumes. The DBA designs data structures based on current and future data content needs and must also address the divergent processing needs of data loading and complex query processing.

Managing an analytical system also differs because of variety of user processing.  It’s not uncommon for a data warehouse to support numerous canned reports or queries along with a category of power users generating ad-hoc queries.  The challenge is preventing a single ad-hoc query from crippling the processing of the entire system.  Multi-statement queries, numerous table joins, and large volumes of historical content are commonplace in analytical environments. 

It becomes fairly clear that the role of a DBA is very different when comparing the work activities of analytical and operational systems.  I’m not suggesting that working in one environment is more complex or difficult than the other—they’re just different.  Thus the activities and their associated skills are very different.  Which is why we often recommend that a single individual may be hard-pressed to support both operational and analytical environments.

Can one person address both responsibilities? Maybe. But first, try contacting your diesel mechanic and see if he’s interested in becoming your operational system DBA.

photo by Kerry 2009 via Flickr.

Repurposing Your Data Warehouse Platform—Not!

I’ve noticed lately that data warehouse vendors are dusting off the arguments and pitches of days gone by. Don’t buy specialized hardware for your database needs! You’ll never be able to re-use the gear! One rep recently told a client, “With your data warehouse on our hardware, you can re-purpose the hardware at any time!”

The truth is, while data warehouse failures were rampant a few years ago, those failures are now the exception and not the rule. Data warehouses, once installed, tend to last a while. The good ones actually add more data over time and become more entrenched among user organizations. The great ones become strategic, and business people claim not to be able to do their jobs without them. A data warehouse platform is rarely for a single use, but for a multitude of needs. Data warehouses rarely just go away.

However don’t confuse an entrenched data warehouse with an entrenched data integration solution. I’ll teach a class at The Data Warehousing Institute conferences called “Architectural Options for Data Integration.” The class covers technologies like Enterprise Application Integration (EAI); Enterprise Information Integration (EII); Extract Transformation and Loading (ETL, and its sister, ELT); and Master Data Management (MDM). I present use cases for these different solutions as well as lists of the key vendors that offer them.

Attendees I talk to admit coming to the class with the intent of justifying the data warehouse as a multi-purpose integration system. They leave the class understanding the often-stark differences of these various solutions. And I hope they return to work with a different view of their future-state integration architectures, whether they re-purpose their hardware or not.

Note: Evan’s will be teaching Beyond the Data Warehouse: Architectural Options for Data Integration at the TDWI World Conference in San Diego on Thursday, August 6.

Sun and Oracle

The recent acquisition of Sun by Oracle has raised a lot of speculative discussion about the latter vendor’s strategic pursuits. The move may or may not result in a power triumvirate of HP-IBM-Oracle. But Oracle expanding its portfolio to include hardware could be a game-changer.

Oracle has a dubious record with hardware plays. The nCube investment (circa 1988) and network computer idea (circa 1996) both presented interesting vision, but didn’t deliver tactically. NCube video-on-demand (circa 1994) ceded to decommissioning the product (circa 2001).

While many are focused on the state of Sun’s numerous DBMS partnerships, I’m more interested in the fate of Storage Technologies, which was acquired by Sun (circa 2002). Do a little research and you’ll see that EMC stores the lion’s share of DBMS data across enterprise data centers. If Oracle keeps the Storage Tech products it might shave some revenue from EMC and gain an even larger wallet share with IT organizations. Oracle’s intentions are equally unclear around the Exadata product, which had previously relied on the HP partnership that’s certainly strained. With the acquisition of Sun, Oracle is more able to go head-to-head with the likes of HP’s Neoview and Teradata.

Clearly the company has the option of producing a database appliance on its own. Personally I’m waiting to see the level of fear, uncertainty, and doubt Oracle stir up into the data warehouse appliance market. Oracle hasn’t differentiated its DBMS in years. The differentiation has always been about the company’s size, the number of Fortune 500 customers, and its broad array of application offerings, and that they work on every conceivable hardware platform. Focus on non-database products has fanned the flames of the market’s perception that databases are a mere commodity.

I can only imagine what’s going on in Oracle’s slideware development organization right now. Here are some of the messaging scenarios that are likely to be on the table:

Scenario 1: “Through our acquisition of Sun, we can now deliver a more fully-functional database appliance.”

In reality, the whole point of an appliance is to reduce complexity and configuration effort. Prepackaging Oracle on a hardware platform already occurs with companies like Sun, HP, and Dell. This isn’t simpler or better.

Scenario 2: “Oracle can now be your de-facto desktop and development tool provider.”

This one could actually be true. Oracle can leverage Sun’s vast software capabilities in two significant ways. With Sun’s desktop office suite, StarOffice, Oracle could provide a captivating alternative to the Microsoft Office monopoly. Any executive would find it difficult to ignore an Oracle office option, particularly in cases where they’ve made significant investments in Oracle as the corporate database standard. Plus, Oracle can monetize open source software by dramatically improving support revenue from these customers. Microsoft does not deliver customer service and support the way Oracle does—and enterprise clients expect more sophisticated and consistent support than the channel usually delivers.

Scenario 3: “Our Java-based toolset covers the spectrum of development needs without forcing your reliance on a specific vendor. Whether it’s middleware, server development, or reporting, we have the tools to support a multi-tier network enabled environment.  You can now come to a single company for a single set of tools regardless of your platform type, desktop, server, or operating system.”

For IT organizations that still rely on custom development, this may dramatically reduce the number of suppliers they need. Over the past few years the number of middleware and application tool vendors has diminished—with Oracle being the buyer of many of them. Most IT organizations prefer fewer vendors. Whether open source or proprietary, the combined Oracle-Sun toolset offers Oracle a significant revenue stream in the support arena.

I’m fascinated that little or no attention has been paid to the software assets that Sun has. This combined with Oracle’s DBMS, middleware, and application toolsets offers an unexpected alternative to the ongoing IBM and Microsoft battles for enterprise development. Moreover, with Sun’s Java leadership and the popularity of Java in consumer electronics, Oracle can now enter into the world of consumer software, a la Apple. The opportunity for Oracle to support media companies that sell directly to the end consumer is wide open.

If it’s not careful, Oracle’s future may be in milking the legacy product cow instead of exploiting its newfound software assets.  The real question is, is Oracle a company of innovators or bean counters?

No Data Warehouse Required: BI Reporting Extends Its Reach

It’s rare these days to find clients who haven’t already decided on a standard BI platform. Most of the new BI tool discussions we get into with clients are with companies who’ve decided that it’s time to broaden their horizons beyond Microsoft.

The dirty little secret in most companies is that the BI reporting team has morphed into a de-facto enterprise reporting team. Why is this?

When it comes to reporting, there’s a difference between the BI team and the rest of IT. The fact is that BI teams are successful not because of the infrastructure technologies, but because of the technologies in front of the users: the actual BI tool. To the end user, data visualization and access are much more important than database management and storage infrastructure.  So when a new operational system is introduced, users expect the same functionality, look and feel as their other reports.

An insurance company we’re working with is replacing its operational systems. The company’s management has already decided not to use the vendor’s reports—they’re too limited and brittle. They expect these reports to dovetail into the company’s information portal and work alongside their BI reporting. Companies are refreshing their operational platforms every seven to ten years. It’s now 2009, and the last time they refreshed their operational systems was in reaction to Y2K. It’s once again time to revisit those operational systems.

If you look at the challenges BI tool vendors are facing, there is limited growth in data warehousing. Most companies have standardized their BI tool suite. Absent disruptive technology or new functionality, there’s limited growth opportunity for BI tools in the data warehousing space.

But for every data warehouse or data mart within a company, there are likely dozens of operational systems that users need access to. The opportunity for BI vendors now is delivering operational information to business users. This isn’t about complex analytics or advanced computation. This is the retrieval of operational information from where it lives.

Photo by jakeliefer (via Flickr)

The Rise of the Columnar Database

photo by eflon

I’m continually surprised that more vendors haven’t hurled themselves onto the columnar database bandwagon. The more this space matures, the more evident it becomes that analytics is a perfect match for column-based database architectures.

One of the most frustrating phenomena to IT is adherence to a theoretical view. In the 1970s the entire relational database industry implemented what was really an academic precept. For those pragmatists who haven’t dusted off their textbooks recently, I’ll recall the writings of Codd and Date. They introduced the concepts of organizing data in tuples, organizing primary values along with their descriptive details (aka: attributes). Vendors interpreted this to mean that data should be physically stored in this fashion, architecting their products to store data in tables, populated with rows consisting of columns. If you wanted to access a value, you had to retrieve the entire row.

With all due respect, this approach has been cumbersome since Day 1. The fact is, storing data the way the business looks doesn’t lend itself to the way people ask questions. When I create an outbound marketing list, I need a name, a phone number, and an address. I don’t need information on household, demographic segment, or the name of a customer’s dog.

While I do need to store all the customer data, I don’t want to be bogged down by processing all that data in order to answer my question. Herein lies the quandary: do I structure the data based on all the information we have, or based on the information I might access?

Vendors have tried to bridge the gap. We’ve seen partitioning, star indexes, query pre-processing, bitmap and index joins, and even hashing in an attempt to support more specific data retrieval. Such solutions still require examining the contents of the entire row.

Although my background is in engineering, I know enough about Occam’s razor to know that it applies here: the simplest solution is the best one. Vendors like Kickfire, Vertica, Paraccel, and Sybase—whose pioneering IQ product launched over a dozen years ago--went back to the drawing board and fixed the problem, architecting their products structure and store the data the way people ask questions—in columns.

For you SQL jockeys, most of the heavy-lifting in database processing is in the where clause. Columnar databases are faster because their processing isn’t inhibited by unnecessary row content. Because many database tables can have upwards of 100 columns, and because most business questions only request a handful of them, this just makes business sense. And In these days of multi-billion row tables and petabyte-sized systems, columnar databases make more sense than ever.

As the data warehouse market continues to consolidate through acquisitions, look for column-based startups—including several open-source solutions—to fill the void. If you ask me, there’s plenty of room.

Blurring the Line Between SOA and BI

photo by Siomuzzz

I recently read with interest an article in the Microsoft Architect Journal on so-called Service-Oriented Business Intelligence or, as the article’s authors call it, “SoBI.” The article was well-intentioned but confusing. What it confirmed to me is that plenty of experienced IT professionals are struggling to reconcile Service Oriented Architecture (SOA) concepts with business intelligence.

SOA is certainly a valuable tool in the architecture and development toolbox; however, I think it’s only fair to keep SOA in perspective.  It’s an evolutionary  technology  in IT that has numerous benefits to developer productivity and application connectivity.  I’m not sure that injecting SOA into a data warehouse environment or framework will do anything more than freshen a few low-level building blocks that have been neglected in some data warehouse environments.  I’m certainly not challenging the value of SOA; I’m just trying to put in perspective to those folks that are focused on data warehouse and business intelligence activities.

The idea around SOA is to create services (or functions, procedures, etc.) that can be used by other systems.  The idea is simple: build once, use many times.  This ensures that important (and possibly complicated) application processes can be used by numerous disparate applications. It’s like an application processing supply chain:  let the most efficient resource build a service and provide to everyone else for use.   SOA provides a framework for allowing multiple applications access to common, well-defined services.  These services can contain code and/or data.  

The question for most data warehouse environment’s isn’t whether SOA can improve (or benefit) the data warehouse; it’s understanding how SOA can benefit a data warehouse. 

We’ve got lots of clients leveraging SOA to support their data warehouse.  They’ve learned they can leverage SOA techniques and coding to deliver standardized data cleansing and data validation to a range of business applications.  They have also upgraded the operational system data extraction code to leverage SOA which allowed other application systems (or data marts) to reuse their code.

However, their use of the SOA hasn’t been focused on enhancing the data warehouse environment as much as has been focused on packaging their development efforts for others to use.  Most data warehouse developers invest heavily in navigating IT’s labyrinth of operational systems and application data in order to identify, cleanse, and load data into their warehouses.  What they’ve learned is that for every new ETL script, there are probably 20 other systems that have to custom developed their own data retrieval code and never documented it.  The value that many data warehouse developers find with SOA isn’t that they are improving their data warehouse;  they’re just addressing the limitations of the application systems.

The Flaw of the Hub-and-Spoke Architecture

By Evan Levy

I recently talked to a client who was fixated on a hub-and-spoke solution to support his company’s analytical applications. This guy had been around the block a few times and had some pretty set paradigms about how BI should work. In the world of software and data, the one thing I’ve learned is that there are no absolutes. And there’s no such thing as a universal architecture.

photo by John-Morgan

The premise of a hub-and-spoke architecture is to have a data warehouse function as the clearing house for all the data a company’s applications might need. This can be a reasonable approach if data requirements are well-defined, predictable, and homogenous across the applications—and if data latency isn’t an issue.

First-generation data warehouses were originally built as reporting systems. But people quickly recognized the need for data provisioning (e.g., moving data between systems), and data warehouses morphed into storehouses for analytic data. This was out of necessity: developers didn’t have the knowledge or skills to retrieve data from operational systems. The data warehouse was rendered a data provisioning platform not because of architectural elegance but due to resource and skills limitations.

(And let’s not forget that the data contained in all these operational systems was rarely documented, whereas data in the warehouse was often supported by robust metadata.)

If everyone’s needs are homogenous and well-defined, using the data warehouse for data provisioning is just fine. The flaw of hub-and-spoke is that it doesn’t address issues of timeliness and latency.  After all, if it could why are programmers still writing custom code for data provisioning?

When an airline wants to adjust the cost of seats, it can’t formulate new pricing based on old data—it needs up-to-the-minute pricing details. Large distribution networks, like retailing and shipping, have learned that hub-and-spoke systems are not the most efficient or cost-effective models.

Nowadays most cutting-edge analytic tools are focused on allowing the business to quickly respond to events and circumstances. And most companies have adopted packaged applications for their core financial and operations. Unlike the proprietary systems of the past, these applications are in fact well-documented, and many come with utilities and standard extracts as part of initial delivery. What’s changed in the last 15 years is that operational applications are now built to share data. And most differentiating business processes require direct source system access.

Many high-value business needs require fine-grained, non-enterprise data. To move this specialized, business function-centric content through a hub-and-spoke network designed to support large-volume, generalized data is not only inefficient but more costly. Analytic users don’t always need the same data. Moreover, these users now know where the data is, so time-sensitive information can be available on-demand.

The logistics and shipping industries learned that you can start with a hub-and-spoke design, but when volume reaches critical mass, direct source-to-destination links are more efficient, and more profitable. (If this wasn’t the case, there would be no such thing as the non-stop flight.) When business requirements are specialized and high-value (e.g., low-latency, limited content), provisioning data directly from the source system is not only justified, it’s probably the most efficient solution.

Why BI Development is Different

By Evan Levy

When companies initially embark on their BI development initiatives, they often underestimate its complexity. Some begin BI in the first place because their packaged applications don’t deliver the reporting functionality they need. Others embark on BI because the data they need to analyze is located in multiple, disparate application systems. While positioning a data warehouse to integrate and store historical data from packaged applications, like ERP or CRM, is a reasonable and proven approach, many companies try to repurpose the development methods associated with these packages to deliver BI.

But comparing development methods and skill sets for these two divergent types of systems is like comparing picking apples to making a fruit salad. The fact is the methodology for building a data warehouse is very similar to traditional code development using lower-level programming languages. To be successful building a data warehouse, a team should have skills in business requirements gathering, functional requirements definition, specification and design, data modeling, database design, as well as all the skills associated with loading the data and coding the application. This is clearly a complex mix of technical knowledge to deliver a business solution spanning everything from storage allocation to workload management to systems integration to application programming. The fact is you’re building something from scratch.

The packaged application world is complex in its own right, but it’s also very different, as are the skills and methodologies involved in building these environments. Most IT organizations accustomed to implementing packages use third-party firms to install and configure these systems. Their staff members don’t have the necessary skills to build these solutions, and often require training and multiple years of hands-on use to be proficient in supporting these systems. In addition, most organizations forget that implementing their business applications typically takes a year or longer.

When was the last time you were allowed a full year to implement your data warehouse? And was your team even half the size of the packaged app’s development team?