DataOps

In hundreds of conversations with customers, investors, and other data professionals, we’ve found that everyone believes they have heard the term before, but isn’t quite sure what it means, exactly. When asked to describe DataOps, most people intuitively understood it had something to do with moving data to the right place in the right format. To move the conversation forward, we need a clear definition we can all use. At Nexla, we believe:

DataOps is the function within an organization that controls the data journey from source to value.

The Internet Archive, also known as the Wayback Machine, has been capturing snapshots of the World Wide Web for over 20 years. But a new effort to save a specific type of Internet content- scientific data- has emerged among some scientists and professors. Groups like the Penn Program in Environmental Humanities and its offshoot Data Refuge are committed to preserving “…the facts we need at a time of ongoing climate change.”

These groups encourage others to download and store public scientific data. They’ve acted as a catalyst for groups around the country to host data meetups. At Nexla, we are committed to building tools that make it easier to collaborate with data. In that spirit, we’d like to share two of the methods we’ve found for archiving important data sets.

At Nexla, we think of APIs are belonging to one of two categories: service or data. A service API is a building block for a developer, a way to hook into another application’s functionality. It’s how developers can build apps in Slack, or add google maps to a web app. Data APIs on the other hand are a bit more limited. They allow developers to pull data from a source and then use that data however they see fit. They don’t offer any additional functionality or services. And that’s why they’re facing extinction.

Film buffs will remember the above scene from Stanley Kubrick’s iconic Dr. Strangelove, in which President Muffley says, “Gentlemen, you can’t fight in here! This is the War Room!” The absurd, satirical line is oddly applicable to what’s happening today in machine learning teams. The machine learning folks shouldn’t be data wrangling- they should be focused on machine learning. But because we don’t often receive our data in a usable format, valuable time is spent transforming, moving, and cleaning data.