How LLMs Are Transforming Data Team Roles

I’ve been building and managing data systems at Amazon for the last 8 years. Now that AI is everywhere, the way we work as data engineers is changing fast. Here are 5 real ways I (and many in the industry) use LLMs to work smarter every day as a Senior Data Engineer: 1. Code Review and Refactoring LLMs help break down complex pull requests into simple summaries, making it easier to review changes across big codebases. They can also identify anti-patterns in PySpark, SQL, and Airflow code, helping you catch bugs or risky logic before it lands in prod. If you’re refactoring old code, LLMs can point out where your abstractions are weak or naming is inconsistent, so your codebase stays cleaner as it grows. 2. Debugging Data Pipelines When Spark jobs fail or SQL breaks in production, LLMs help translate ugly error logs into plain English. They can suggest troubleshooting steps or highlight what part of the pipeline to inspect next, helping you zero in on root causes faster. If you’re stuck on a recurring error, LLMs can propose code-level changes or optimizations you might have missed. 3. Documentation and Knowledge Sharing Turning notebooks, scripts, or undocumented DAGs into clear internal docs is much easier with LLMs. They can help structure your explanations, highlight the “why” behind key design choices, and make onboarding or handover notes quick to produce. Keeping platform wikis and technical documentation up to date becomes much less of a chore. 4. Data Modeling and Architecture Decisions When you’re designing schemas, deciding on partitioning, or picking between technologies (like Delta, Iceberg, or Hudi), LLMs can offer quick pros/cons, highlight trade-offs, and provide code samples. If you need to visualize a pipeline or architecture, LLMs can help you draft Mermaid or PlantUML diagrams for clearer communication with stakeholders. 5. Cross-Team Communication When collaborating with PMs, analytics, or infra teams, LLMs help you draft clear, focused updates, whether it’s a Slack message, an email, or a JIRA comment. They’re useful for summarizing complex issues, outlining next steps, or translating technical decisions into language that business partners understand. LLMs won’t replace data engineers, but they’re rapidly raising the bar for what you can deliver each week. Start by picking one recurring pain point in your workflow, then see how an LLM can speed it up. This is the new table stakes for staying sharp as a data engineer.

What does the deep-integration of AI mean for data professionals and the 𝗵𝘂𝗺𝗮𝗻𝘀 𝗶𝗻 𝘁𝗵𝗲 𝗹𝗼𝗼𝗽 of big processes or systems? Is data engineering becoming more operations than development? Or the reverse? Alejandro Aboy, a data engineer with nearly a decade of experience across analytics & engineering, and a main data platform stack owner at present, does a great job projecting the implications of the AI overhaul and w𝗵𝗮𝘁 𝗶𝘁 𝗺𝗲𝗮𝗻𝘀 𝗳𝗼𝗿 𝗵𝘂𝗺𝗮𝗻𝘀 𝗯𝘂𝗶𝗹𝗱𝗶𝗻𝗴 𝗱𝗮𝘁𝗮. 🔖 Refer here: https://lnkd.in/d7Kzac-p While timelines are flooded with GPT prompts and AI workflows, the real shift is deeper: As Data Analysts, Engineers, and Platform developers, we're not meant to just use AI; we're building systems where agents collaborate with humans. 𝗧𝗵𝗶𝘀 𝗺𝗲𝗮𝗻𝘀 𝗼𝘂𝗿 𝗿𝗼𝗹𝗲𝘀 𝗶𝗻 𝗱𝗮𝘁𝗮 𝗮𝗿𝗲 𝗰𝗵𝗮𝗻𝗴𝗶𝗻𝗴. 🧠 𝗗𝗮𝘁𝗮 𝗮𝗻𝗮𝗹𝘆𝘀𝘁𝘀 must frame business questions for agents. 🛠️ 𝗗𝗮𝘁𝗮 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝘀 need to design secure backends for RAG pipelines and tool scopes. 📊 𝗗𝗮𝘁𝗮 𝘀𝗰𝗶𝗲𝗻𝘁𝗶𝘀𝘁𝘀 are evaluating LLMs and enabling fine-tuning. Data pros must now speak AI-native languages: structured prompts, context-rich inputs, and human-in-the-loop systems. Here’s where agentic design is already showing up: ✅ Pipeline Monitoring: agents detect bugs, retrieve context, and propose fixes 📁 Documentation: agents surface past failures to inform new builds. 📐 Data Models: agents draft models from business definitions + past context 📊 Analytics: agents guide stakeholders to the right dashboards. But with great power comes... head-spinning complexity: 🔍 Observability: Know what your agents are doing and why 🛑 Guardrails: Prevent bad or insecure decisions 🔐 Security: Minimal access, scoped tools, no mess We’re entering the Agentic Data Era. And it needs orchestration beyond just automation. 📌 𝗧𝗵𝗲 𝗱𝗮𝘁𝗮 𝘁𝗲𝗮𝗺𝘀 𝘄𝗵𝗼’𝗹𝗹 𝘁𝗵𝗿𝗶𝘃𝗲? 𝗧𝗵𝗼𝘀𝗲 𝘄𝗵𝗼 𝗹𝗲𝗮𝗿𝗻 𝗵𝗼𝘄 𝘁𝗼 𝗱𝗲𝘀𝗶𝗴𝗻 𝗳𝗼𝗿 𝗮𝗴𝗲𝗻𝘁𝘀 𝗯𝗲𝗳𝗼𝗿𝗲 𝗱𝗲𝗹𝗲𝗴𝗮𝘁𝗶𝗻𝗴 𝘁𝗼 𝘁𝗵𝗲𝗺. 💡 TL;DR: Agentic roles = new bridges between business, tech & AI Human learning is still essential for agents to learn well Guardrails, documentation & context are your new power tools And finally: knowing when not to use AI might be your biggest strategic edge Be agentic. Value still matters most. Appreciate the great insights from Alejandro! Find the full read here: https://lnkd.in/d7Kzac-p Image adopted from Debmalya Biswas and Alejandro Aboy's illustrations. #AIAgents #DataEngineering #AgenticAI

We spent a lot of time with enterprises evaluating AI for data and analytics last year. Our takeaway: Enterprises don't need text-to-SQL solutions. They need intelligent AI teammates to amplify the productivity of data and analytics teams. After ChatGPT was released in late 2022, almost everyone in data and analytics rushed to leverage GenAI models for text-to-SQL solutions. After two years, it became clear that text-to-SQL alone was simply not enough. While enterprises don't need a simple translation from natural language to SQL, they need agents and digital teammates across different roles and functions in the organization, including data and analytics. Enterprises need AI data engineers. Future AI data engineers will work alongside human data teams to carry out tasks such as building data assets, investigating ongoing issues, and optimizing costs. For an AI agent to be a helpful teammate, it is not enough to take a natural language query as input and output a SQL statement. AI agents for data and analytics need a comprehensive understanding of the existing underlying data assets, from the raw data through transformations and semantic modeling to reporting. More importantly, they need the ability to make changes to these assets as they reason and go through the chain-of-thought process. AI data engineers will increase productivity across the board. Data engineering teams will be able to tackle more complicated tasks more quickly, and less technical team members will be able to contribute to areas they couldn't contribute to before, e.g., dashboard designers contributing to data transformation pipelines. It is an exciting opportunity, but we still need to build many basic components to enable this future. Some of these components would require advancing fundamental enabling technologies, such as LLMs, while others would involve developing infrastructure for AI agents to retrieve, understand, and modify data assets.