The DataOps Trend is Real: 73% of Companies Plan to Invest in DataOps to Manage Data Teams

In order to grow successfully, companies need to leverage DataOps to better manage data teams. Nexla’s second annual Data Operations survey reveals vast majority of companies have plans to invest in DataOps to better manage data teams to fuel artificial intelligence and machine learning efforts.

We’re excited to announce the results of the industry’s only annual data operations survey. The comprehensive survey tracks the adoption and best practices of Data Operations (DataOps). It found that a staggering 73% of companies are investing in DataOps.

We asked our friends over at research firm Pulse Q&A to conduct a survey with 266 data professionals about how they use data, their team structure, and data challenges.

Machine Learning and AI, real-time streaming require dedicated data teams

• 85% of companies report that their company is working on machine learning and AI today, up from 70% in 2017. This work will continue, with 83% of companies saying they will do more in machine learning and AI next year
• To feed their models more data, 85% of companies are ingesting data from third parties. The Data Operations survey found 54% of companies are ingesting data from more than 10 partners
• Real-time streaming data is becoming critical to these efforts, with 58% of companies reporting they ingest data this way

Data teams understaffed to capitalize on the big data opportunity

• Fifty percent of respondents reported they do not have enough backend data engineers on their data teams to support their company’s data needs
• Seventy-three percent of companies report that they have plans to hire DataOps professionals in 2018
• The average company only has one data engineer for every 5 business users, processes 2.7 GB of new data a day, and manages 4,300 data sets

Overworked data teams crave automation

• Data engineers reported they spend an average of 18% of their time on troubleshooting. The average company loses 180 hours a week to troubleshooting
• Data pros see an opportunity for automation in all aspects of their work. 56% believe data clean-up could benefit from automation in the next two years, with 47% saying analytics could benefit and 46% saying integration could
• Data format consistency was the number-one challenge cited by data pros, with 39% saying this was a key challenge. 60% of companies ingest data in three or more formats, adding complexity
• Data integration, and reliability of data pipelines were the #2 and #3 most-cited challenges with 36% and 35% respectively. Data pros are spending 18% of their time on these activities

“It’s clear that backend data teams are strapped for resources, which is why 73% have plans to invest in DataOps,” said Saket Saurabh, Nexla Co-founder and CEO. “DataOps is as much about people as it is about tools and processes. To really drive value from machine learning, AI, and advanced analytics, data teams need to stop troubleshooting and start automating. We built Nexla to help data teams create automated, repeatable, and scalable data flows so they can focus on deriving value from data.”

Manage data teams efficiently with DataOps

Data Operations is an emerging organization-wide data management practice that controls the flow of data from source to value, with the goal of speeding up the process of deriving value from data. The outcome is scalable, repeatable, and predictable data flows for data engineers, data scientists, and business users. An important thing to note: DataOps is as much about people as it is about tools and processes.

DataOps takes care of the grunt work typically placed on IT or data engineers. This includes integrating with data sources, performing transformations, converting data formats, and writing or delivering data to its required destination. A DataOps practice can open data access to more stakeholders within an organization, further increasing capacity for scale.

With DataOps, tasks like data clean-up/prep, analysis, and data integration can be automated—effectively minimizing the engineering queue. When data engineers are no longer spending a majority of their time on these mundane tasks, they can switch focus towards higher value-add tasks for advanced machine learning, AI, and real-time streaming. In turn, business operations and production moves much faster when data teams are no longer strapped for resources.

About the Survey

The Data Operations survey was commissioned by Nexla and conducted by the executive research platform Pulse Q&A. The survey polled 266 IT and data professionals, including analysts, data scientists, data engineers, and executives. The survey was fielded between May 3 — May 21, 2018.

To get a copy of the full report, visit: https://nexla.com/definitive-data-operations-report-2018/

About Pulse Q&A

Pulse Q&A is a company dedicated to helping C-suite executives make better strategy, innovation and enterprise technology decisions – starting with CIOs. Today, many of these leaders still rely on one-size-fits all reports when setting important strategies. Pulse is challenging that practice with real-time, bite-sized data from thousands of IT executives discussing innovation and spend on their platform. Pulse’s goal is to shift focus away from analyst to the executives making the decisions every day. By giving users access to relevant conversations and crowdsourced data, Pulse Q&A aims to pull executives out of their echo chambers and, instead, make decisions based on current data and real world experiences. For more information, visit www.pulse.qa.