4 Ways Generative AI is Transforming Data Engineering

Generative AI (GenAI) is a type of artificial intelligence that can generate new data, such as text, images, audio, and video, by learning patterns from existing data. In the race to capitalize on Generative AI and gain a competitive advantage, it is critical to have clean, reliable, & well-formatted data. As a result, the field of data engineering is undergoing a swift and transformative evolution. Data engineers can no longer afford to be caught in a relentless loop of maintenance and crisis management, which stifles their ability to engage in more forward-thinking projects. They will be called upon to not only manage their current projects around analytics, operations, & machine learning (ML) but also help with this new shiny realm of GenAI.

To delve deeper into this transformation, I had the privilege of moderating a panel with data leaders from Bloomreach, TripAdvisor, and Seattle Data Guy. They shared insightful perspectives on how Generative AI, particularly through large language models (LLMs), is reshaping the data engineering domain. 

Watch the expert panel discussion on the impact of GenAI on data engineering here.

Here are four key takeaways from our conversation:

1. Coping with the Surge in Data Demand

LLMs have revolutionized the way we interact with data, and data engineers are at the forefront of this change. “AI is basically powered by data,” said Xun Wang, CTO at Bloomreach. “And that’s why data engineering is just so important to, especially, this new generation of generative AI that we’re in right now. Now, part of the challenge is accessibility and availability.”

Bloomreach has observed a paradigm shift in consumer search behavior on their e-commerce platform, a change driven by the more intuitive interfaces enabled by LLMs. The way customers search for products has become more conversational, mirroring natural human interaction.

TripAdvisor is another example, catering to over 400 million monthly active users, with a repository of over a billion reviews. They’ve harnessed Generative AI to transform itinerary planning, offering personalized travel recommendations by analyzing vast amounts of data—a feat that would have been unattainable without sophisticated data models.

2. Boosting Productivity through Code Generation & Automation

Generative AI is a game-changer in automating the repetitive tasks that once consumed the daily routines of data engineers, such as constructing data pipelines and parsing API documentation. This automation allows engineers to redirect their focus to more complex, strategic challenges.

This doesn’t come without concerns, as Bloomreach noted, “I absolutely think that there’s much more technology frameworks and new companies that are springing up around generative AI that will help us protect our data and IP better.”

3. Evolving the Data Engineer’s Toolkit

While the core of data engineering remains problem-solving, the skill set required is expanding. Proficiency in machine learning operations and the management of AI models is becoming increasingly important. The modern data engineer needs to be as comfortable with ML models as they are with database management systems.

With GenAI, data engineers have a new ally in maintaining data quality. Generative AI can perform relevancy checks and other quality control measures, which are crucial in building trust and confidence in data-driven decisions. Ben Rojogan, the Seattle Data Guy, reminded us, “People will just stop using things [dashboards] if they don’t have trust in them,” and data teams can only afford so many mistakes before they are let go.

Rahul Todkar, head of data at Trip Advisor, said, “With my team, we have this concept of golden data sets, not by any means a new concept.,” but centers around thoroughly defining what good data quality looks like. “What data do you put in? With what metrics? business logic definition? Do you have a good source of a good quality measure around accuracy, delivery, consistency, availability of SLA?”. The complexity continues with questions around taxonomy, monitoring, and governance.

4. Redefining Roles and Responsibilities

Generative AI is democratizing data access, enabling those without technical backgrounds to analyze data more independently. This shift is altering the traditional division of labor, with routine tasks becoming increasingly automated. The future of data engineering will be increasingly focused on unsolved problems around data availability, accessibility, & quality.

Bloomreach pairs large language models and Nexla to power retrieval augmentation, which grounds GenAI outputs in the truth and real-time information. This method can address issues like hallucinations by combining the conversational prowess of LLMs with the factual accuracy provided by retrieval systems.

As we continue to embrace GenAI in data engineering, the role of the data engineer is becoming more crucial than ever, poised to shape the future of data-centric organizations.