3 Exciting Ways Data Engineers Are Changing the World

When we talk about exciting innovations like self driving cars, or ask Amazon’s Alexa to play a song for us, occasionally we’ll stop and marvel at just how much the world has changed in a couple of decades. Rarely, however, does the average consumer stop and think about the people enabling this awe-inspiring progress.

Innovation is happening at a more rapid pace than ever before, and that’s due in large part to the ways we’ve been able to aggregate and analyze large sets of data. Data engineers collect this data and process it either in batches or in real-time. Without this aggregation and organization, the data would be useless to the data scientists who are then able to query it. Data engineers are the people who develop the foundation for the innovations that inspire and wow us—but they seldom get the glory.

Here are three exciting ways these engineers are working with data to change the world.

The Internet of Things (IoT)

Does “Alexa” ever turn on the lights and play your favorite song for you after a long day at work? There’s a data engineer behind that. The ability for a machine to recognize your voice and perform the actions you command is due to the aggregation and analysis of humongous data sets. Speech recognition capabilities for a machine are only advanced and improved with the tracking built by data engineers.

The IoT will continue to evolve and there will only be more and more data to analyze. Some predict the future will entail smart dust, or sensors so small they can be embedded in things like paint to collect data on when people walk past a certain building. Data engineers will become even more critical as we rely on them to aggregate and process these larger and larger datasets.

Data engineers are the real backbone that keeps the innovation moving forward. Dan Woods, CTO of CITO Research, said, “the physical parts of the IoT will be easy. The data engineering will be hard and where differentiating value is created.” In other words, data engineers are doing the hard work that provides the foundation for how we interface with devices like Alexa, and how this once-crazy idea continues to build momentum.

Forensics

Advancements in technology have always been critical to solving violent crimes. Think about the impact of DNA testing and blood spatter analysis. Now, big data is enabling detectives to spot patterns and more easily identify serial killer activity. Recently, in Cleveland, Ohio, detectives used an algorithm to spot patterns in dozens of unsolved murders. Grouping the murders by factors like geography, type of death, and age and profession of the victim has allowed detectives to see these cold cases through a fresh lens.

The algorithm has already spotted patterns that the Murder Accountability Project (MAP) believes will save lives. Of the project, David Icove, a professor of electrical engineering and computer science and member of MAP, said, “The idea is to try to help solve cold cases through the use of big data and computer analytics. Data can reveal what the human component might not be able to distinguish.”

By aggregating the data from a vast number of sources (over many years), data engineers are organizing the information in a way that can be readily analyzed by law enforcement. While the detectives will inevitably get the recognition, the data engineers behind the infrastructure that allows this data set to be analyzed will have advanced forensics technology once again. In addition to solving violent crimes, police departments are starting to turn to big data for cases that may have previously gone unsolved due to disparate data across different departments and even state lines.

Transportation

The impact of self driving cars will be extreme. From the positive impacts to affordable, safe transportation to the secular shifts in the job market, autonomous vehicles will change the world. As CNN reports, a car’s data might become more valuable than the car itself. Estimates for how much data a connected car will produce range from 4,000 GB a day to 1 GB per second. Data engineers will be the ones responsible for figuring out how all of that data can be made useful.

In addition to the exciting progress engineers are making toward automated vehicles, big data is changing basic, long-felt transportation nuisances like traffic. Some towns and cities now have sensors which pick up millions of daily cell phone and GPS data points such as car speeds, sources of acceleration or deceleration, weather conditions, and even community events that might impact traffic irregularities. Synthesizing this data allows real-time traffic updates to digital maps so travelers can make more informed decisions about their commute.

Data Engineers Make it Happen

While these examples are very different outcomes, they’re all made possible by data engineers. They create transparency to take complicated, large systems and processes and create connectivity and cohesion. Data engineers are making the world both larger and smaller than it’s ever been before.

Are you a Data Engineer? We applaud you! Please add your opinions to our DataOps assessment to find out how your company compares to others in your industry.