Lay a data pipeline: Exposing the gaps in today’s business intelligence market

The desire to analyze data that will drive insights and competitive differentiation is older than computing itself. Digitization just speeded things up. Richard Miller Devens used the term “business intelligence” as far back as 1865. The LEO computer was calculating optimal inventory deliveries based on shop performance and generating management reports for the Lyons’ tea rooms chain from 1951. And the very first edition of CIO magazine, published in 1987, included an editorial on “an increasing cadre of increasingly demanding customers seeking faster access to information”.

Fast-forward to today, and the dreary meme of “data is the new oil” that must be chanted at every tech conference by rule of law. The power of data is better understood than ever, but for many, harnessing data, checking its quality, and applying context to assist decision-making remains challenging. CIOs report fragmentation, slowness and silos, even as digital transformation has been accelerated by the pandemic. But there is cause for optimism, in the form of broad modern data pipelines that drive activity and allow what Qlik calls “Active Intelligence” – the ability to act on reliable data with a rich supporting fabric of context and collaboration to support the right decisions and take informed actions in the right moment. By assembling joined-up processes, companies are following the path from uncovering data to delivering it where it needs to go, governing it through data catalogs, understanding it, augmenting it, and putting it to use via context-sensitive alerts and actions taken in close to real time.

The 1990s rise of databases that cleaved to structured query language (SQL) led to a glut of developers and specialists and created a boom in analytics activities. But the dirty secret of SQL is that “it’s great for moving data, but not analytics”, says Mike Potter Chief Technology Officer at Qlik, so we’ve ended up using the wrong tool for the job.

“For change, you need to capture data and lay the foundations for an analytics supply chain and a pipeline that builds on this to enable Active Intelligence,” Potter explains. “You can’t create value in any business process unless you do something. If you believe analytics is all about driving change, increasing revenues and profits, and enabling digital transformation, none of that can occur unless you take action.”

Read more about Active Intelligence and how it can help you drive more business value>>

Today’s decision-makers have a lot of tools to work with – from cloud platforms’ massive elastic compute power and the Internet of Things generating sensor-data that supplements existing sources to networks that carry data instantly to the locations where decisions are made. But Potter is surely correct in highlighting the link between information overload and paralysis in decision-making.

So, we need systems that advise, working alongside smart human beings who understand business domain, context, and risk. Whether these are progressive (“it’s a great time to build a shop that sells finger spinners in New York”) or defensive (“this service level agreement is very close to breaking so we need to address it now”), decisions must be taken quickly before context has changed and the moment is gone. Seize that moment and the promise is enormous.

The importance of velocity and first steps can’t be overstated. We must be able to free data, find it and only then invest in data quality processes and add value via augmented data on the fly, to create a holistic, contextual basis for actions. At credit reporting giant Experian, for example, data integration has been critical to ensuring that data is dynamic and fresh to incorporate up-to-the-second verification.

Then, of course, we must be able to interrogate data and build insights, going beyond dashboards and adding the convenience and immediacy of natural language support, so that non-specialists can ask questions and receive sensible answers without drowning in jargon. As more data sources are added in, unforeseen connections are traced, leading to “a-ha” moments of serendipitous revelation. To that end, chief data officers are becoming popular appointments, and DataOps teams are becoming mainstream; but there must be buy-in across the company to build a culture for data success.

Assemble that supply chain of elements, and we begin to realize the promise of real-time analytics. In practice, it may not always be truly real-time, but if you can make a better decision, faster than your rival, you are fulfilling IT’s age-old pledge to provide an auditable end-to-end decision-support platform on which great decisions are made in a business moment.

For too long, we’ve struggled to connect the dots between what’s needed for a holistic approach to data and analytics, but today there’s no excuse as all the technology components are available. Now, it’s incumbent on leaders to lead. As Clayton Christensen wrote in The Innovator’s Dilemma, many companies have failed because they stuck to the road that had made them successful when they should have realized they were heading for a dead-end. Analysis paralysis is a silent killer for innovation and strategic change.

For dynamic companies, however, the rewards are large. For example, Schneider Electric’s finance department is able to predict some quarterly financial performance to within one per cent using analytics.

“Data is the thing that determines how bright the signal is in the fog of uncertainty,” says Clint Clark, the company’s Vice President, Finance Performance Systems and Data, Global Finance. “When you build a pipeline that’s robust, you can illuminate those signals clearer and with better timing, and people can make better decisions quicker.

“You need to build a culture of trust and show that data has value through repeated demonstrations,” he adds. “You have to find a way to put data at the center of your decision-making process and be honest about what you’re doing, including understanding your own hidden assumptions and biases.”

What can get neglected? Clark advises not to underplay the importance of data governance to avoid the “garbage in, garbage out” effect. Also, he says, watch out for the potential “tragedy of the commons”, where people act in individualistic, self-interested ways or use data to back up their prejudices.

Read the full Schneider Electric story>>

By synthesizing all the assets we have to hand, we can create a new wave of data-empowered companies that make the right decisions at the right time.

Elif Tutuk, Vice President, Innovation and Design at Qlik, believes we can advance enormously if we combine the best of tools, humans, and robots working side by side using natural language for interactions. “There’s a need to select a business moment to match the data. Active Intelligence enables the right action at the right moment … and gives users superpowers,” she says.

Now we just need men and women to apply that advice. Ready, set… action!

For more on this and for the latest trends, visit Qlik’s Executive Insights Center qlik.com/executiveinsights.