Yesterday, I explained my data engineering job to a friend and he asked if I was worried that AI was going to replace me. And I thought about all the messy data sources with zero documentation I’ve worked with, the countless conversations I had with humans to understand what the heck I was looking at and how to interpret the data, the technical challenges of connecting to data sources, and the cross-functional teamwork required to actually make sense of data, and I can honestly say – at least for the time being: AI could never.
AI can certainly speed up writing code, reverse-engineering existing code and data, auto-generating documentation, and detecting errors, but unless it can find out who designed a production database from chatting with someone over lunch and send them a Slack message to say “hey, I’m seeing this in the data, is this an error or intentional?”, I believe it still takes humans to deal with human-generated data chaos.
Sam Bail 