Three Predictions for DataOps in 2018

It has been a whirlwind of a year for DataOps! Companies like MapR, StreamSets and Qubole have started talking Data Operations. It seems that 2017 was the year of DataOps discovery—and for good reason. DataOps is all the tasks, processes, and work that moves and transforms the data that powers your business. Pretty important stuff. 

As more and more companies find themselves struggling to ingest, integrate, transform, and drive value from their data, DataOps has become a necessary and crucial aspect of any data-driven business. If 2017 was the year of discovering DataOps, then 2018 is the year DataOps becomes critical. Read on to see our top three predictions for DataOps in 2018.

1. DataOps Frustration Will Grow to a Critical Mass

Sometimes working with data can be like trying to get blood from a stone. In our 2017 Definitive Data Operations Report, we found that data professionals spend 47% of their time on DataOps—instead of on tasks like analytics or ML. The endless cycle of integration, ETL, and troubleshooting is enough to make anyone’s head spin. We know the frustration is mounting. Earlier this year, we learned a staggering 41% of data executives thought their workload is unsustainable.

We predict this unstainable workload will come to a head in 2018 as the volume, velocity, and variety of data growth accelerates. The McKinsey Global Institute estimates that the world’s data volume will double every three years. Forget about the storage or compute capacity to handle this—does your company have the processes and resources in place to actually make use of this data?

Let’s remember, not all of this data is coming from inside of your company’s four walls. There’s an infinite list of partners to integrate with: customers, suppliers, ad networks, marketing tech providers, your CRM, public data, market data… and the list goes on. The challenges related to integrating and maintaining pipelines for operational processes will become overwhelming. DataOps frustration will reach critical mass.

In 2018, data professionals will demand tools that can automate their DataOps workflows.

2. The Multi-Cloud Strategy will Bring on Multiple Headaches

I.T. has finally convinced the enterprise that the cloud is a requirement. But most large enterprises are adopting multi-cloud strategies, bringing on multiple headaches for data professionals. According to IDC, 87% of cloud users have adopted some capabilities for a hybrid cloud strategy. The same survey found that 56% of cloud companies run more than one type of cloud deployment.

As cloud providers continue to differentiate, it seems plausible that one team in a company will be using Google for Machine Learning, AWS for storage, and Azure for apps. The cost and effort of moving the associated data into and out of these various clouds is significant. Even the cost of moving data across regions can be prohibitive.

The complexity compounds when partners are on different clouds. Take the example of an analytics service, which runs on Google Cloud Platform. Their clients want the ability to receive large-volume raw data directly into AWS S3 or Redshift. Egress fees (the cost to move data out of the cloud) can start at a nickel and move up from there. Format optimization will become a cottage industry within companies, as customers and partners ask for data in the format that works best for their environment, not yours. The cost, time, and effort required to make business work in a multi-cloud reality is high.

The challenges of driving business value from data in multiple cloud environments will only intensify. We expect a lot of multi-cloud headaches in 2018, and a strong appetite for solutions that can dull the pain.

3. The Enterprise Will Struggle with the First IoT Integrations

The Internet of Things is real and it’s here. We’re already sending our health data to cell phone makers, fitness trackers, and auto manufacturers. Amazon and Google listen to everything we say via their smart speakers. Enterprises are placing sensors on factory floors, ships, and medical devices. And don’t get us started about drones— we have several just in the office. IDC predicts that the global spending on IoT will reach $1.4 trillion by 2021.

What will we get for this investment? A handful of intelligent toasters, refrigerators that can be hacked Silicon Valley-style, and perhaps a few meaningful industrial applications. But by and large we predict IoT will not yet live up to its promise in 2018. There are several reasons for this: security concerns, the time it takes to roll out large-scale industrial innovation, and data challenges. Let’s focus on the last reason, the data challenges.

How can companies ingest, analyze and operationalize IoT data? Today’s integration platforms can’t handle the variety, volume, or velocity produced by sensors and other IoT data. The inability operationalize IoT data could stall ambitious and laudable projects like Aetna’s partnership with Apple Watch. The promise of creating new products or processes from IoT data will likely require in-stream transformations and edge computing to be successful.

To accelerate the benefits of IoT to consumers and enterprises, companies need an increased focus on Data Operations for IoT. Platforms need to transform in flight because batch ETL will break down at these speeds and volumes. Orchestration needs to be nimble and handled carefully as the number of devices streaming data grows. Finally, pipeline monitoring will need to adapt so that when data errors occur—and we know they will—troubleshooting and triage can happen as quickly as possible.

2018 Will Be the Year DataOps Gets Its Due

The solution to many of the challenges companies are facing in 2018 is a more comprehensive approach to DataOps. This will likely include more robust teams, but more importantly, the right tools. Companies need tools that can tackle an ever-increasing number of external data integrations, multiple public and private clouds, and streaming data. With all this work, in 2018 we might see the Chief Data Officer turn into the Chief DataOps Officer!

Best wishes for a headache-free New Year from Team Nexla!