European enterprises are taking a range of steps to streamline data management and analytics development processes with DataOps
DataOps promises to drive business value by making data management more efficient. It helps teams make data available for use in value-adding analytics projects more quickly.
“DataOps is about extracting the data into value in real-time and seeing the data and analytics as an integral part of the whole business,” says Minna Kärhä, former Data and Analytics Lead at Finnair. “It's about really enabling access and usage of the data for adding value.”
What is DataOps?
It’s well known that data analysts spend the lion’s share of their time cleaning data and preparing it for use in analytics. Our survey of 100 European data and analytics leaders shows that, even today, these tasks eat up 45% of their time.
Given that the volume and variety of data sources enterprises collect is only going to increase, there’s an urgent need to increase efficiency. Our research shows that most enterprises already have thousands of systems or applications feeding into their data stores.
DataOps offers a solution to this challenge. But it’s not a ‘silver bullet’ for data and analytics teams. It’s a set of best practices that rests on the principles of process standardization and orchestration, agile or lean thinking and process improvement.
Three Guiding Principles for DataOps
Agile working practices and process standardization are two of the three guiding principles of DataOps. First, creating cross-functional teams that can partner efficiently with business stakeholders is key for driving value with analytics. Second, ensuring teams follow consistent processes ensures data-driven insights are consistent and repeatable.
The third principle of DataOps is process improvement. This depends on ensuring that staff across the enterprise understand their respective roles in making the data pipeline flow smoothly and on having the right tooling to support collaboration and process automation.
“Product-focused teams can become siloed unless you find some way of tying them together”
Harvinder Atwal, Chief Data Science Officer, MoneySuperMarket
Our research shows that enterprises in Europe are taking steps to facilitate this approach to delivering analytics value. A full 95% of survey respondents have adopted agile working methodologies. What’s more, 74% say agile has already improved the productivity of their teams.
“Although we have longstanding analytics teams, people don't fly around from project to project,” says Harvinder Atwal, Chief Data Science Officer at MoneySuperMarket. “They stay within a particular domain, so they get to understand the data.”
“They're building data products,” he adds. “So, they can't just build something and throw it over the fence for someone else to maintain. They know they're going to be looking after those products. So, it's in their interest to build in quality right from the start.”
Of course, different organizations are seeing different levels of success as they attempt to create environments that support DataOps practices. While 60% of respondents say they’ve been at least ‘quite successful’ at establishing cross-functional teams, just 26% would describe their efforts as ‘very successful’.
Similarly, 58% say they’ve adopted a continuous deployment methodology for analytics at least ‘quite successfully’, 80% have seen success investing in shared data workspaces or platforms and 85% report success rolling out self-service analytics tools to business stakeholders.
Greater Awareness of DataOps Principles is Needed
Our research also suggests there is still a lack of awareness around the steps that enterprises must take to achieve DataOps success.
Just 46% of respondents cite ‘agile development’ and ‘continuous integration’ as essential characteristics of a successful DataOps implementation, and 42% say the same about ‘data pipeline orchestration’.
Meanwhile, 39% view code repositories as essential for DataOps success, 36% say the same about continuous delivery of data-driven products, 32% view performance and application monitoring as important and just 26% think ‘collaboration and reuse’ is a core goal of DataOps.
Yet, experienced DataOps practitioners confirm that all these things play a role in driving data and analytics team productivity gains.
“The reason I say technology comes into this is, these days it's much easier to enforce governance and data security models and data classification models on data than it has been in the past,” Atwal says.
“Teams can become siloed unless you find some way of tying them together,” he adds. “For example, if we're creating features for a machine learning model, those features go into a common feature set or feature store so they're readable by other people.”
“Tooling creates the common visibility,” Kärhä confirms. “But you cannot get around the need for collaboration.”
“Data is usually created in systems that other parts of the organization are managing,” she continues. “So, staff need to understand that this is a collaboration that they need to be able to take responsibility for delivering that data.”
Ultimately, DataOps success depends on industrializing data team processes in much the same way DevOps has for software development. The first step on this path is building awareness of the tools and techniques involved in this transformation. In this respect, it looks like enterprises across Europe still have a way to go.
This is an excerpt from our The DataOps Agenda 2021 research report. To discover more about the maturity of DataOps practices in European enterprises today, click here now and claim your copy.
