More than an AI copilot: Navigating the 5 stages to autonomous data engineering

kAm(96E96C :E’D 86?6C2E:?8 4@56 @C 563F88:?8 72:=FC6D[ px E@@=D 92G6 364@>6 C6>2C<23=J AC@7:4:6?E 2E E96 @A6C2E:@?2= H@C< E92E 4@>A2?:6D 56A6?5 @? E@52J] %96C6’D ?@ BF6DE:@? E92E[ :? 46CE2:? 2C62D[ AC@5F4E:G:EJ 92D 8@?6 FA] qFE E96 D2>6 px :D 2=D@ >F=E:A=J:?8 56>2?5 — 6IA2?5:?8 H92E ?665D E@ 36 >2?2865 2?5 @A6?:?8 52E2 2446DD E@ 72C >@C6 A6@A=6 24C@DD E96 3FD:?6DD E92? 6G6C 367@C6]k^Am

kAms2E2 6?8:?66CD 2C6 @7E6? 42F89E :? E96 >:55=6[ 3F:=5:?8 2?5 >2:?E2:?:?8 E96 A:A6=:?6D 2?5 :?7C2DECF4EFC6 E92E 6G6CJ px H@C<7=@H C6=:6D @?] (96? 2 ?6H 52E2 D@FC46 ?665D E@ 36 :?E68C2E65[ @C 2? px 286?E 3C62<D 3642FD6 @7 F?C6=:23=6 52E2[ E96J 2C6 E96 @?6D 42==65 :? E@ 7:I :E] %96D6 E62>D 2C6 25@AE:?8 E96:C @H? px E@@=D E@ <66A A246[ 3FE >2?J DE:== D66> E@ 36 72==:?8 7FCE96C 2?5 7FCE96C 369:?5]k^Am

kAmp? |x% %649?@=@8J #6G:6H C6A@CE @? “k2 9C67lQ9EEADi^^HHH]D?@H7=2<6]4@>^6?^C6567:?:?8\52E2\6?8:?66C:?8\:?\E96\286\@7\2:^Qm#6567:?:?8 s2E2 t?8:?66C:?8 :? E96 p86 @7 pxk^2m” 7@F?5 E92E H9:=6 g :? `_ @C82?:K2E:@?D 92G6 56A=@J65 px\32D65 52E2 6?8:?66C:?8 E@@=D[ 52E2 6?8:?66CD 2C6 2=D@ >2?28:?8 >@C6 4@>A=6I:EJ H:E9 E96 3:886DE 492==6?86D C6A@CE65 2D 6?DFC:?8 52E2 D64FC:EJ 2?5 AC:G24J WddTX]k^Am

k7:8FC6mk:>8 DC4lQ9EEADi^^:>286D]DE24<6C]4@>^H^`ag_^:>286D^DE@C:6D^a_ae^`_^ae^|x%Ta_#6A@CETa_u:?5:?8Ta_7@CTa_s:DEC:3FE:@?]A?8Q 2=ElQp 52E2 32C 492CE C6G62=:?8 E92E >@C6 E92? g :? `_ E649?@=@8J =6256CD D2J E96:C @C82?:K2E:@? 92D 2=C625J 56A=@J65 px\32D65 52E2 6?8:?66C:?8 E@@=D]Q DEJ=6lQH:5E9i `__TQ ^mk7:842AE:@?m$?@H7=2<6k^7:842AE:@?mk^7:8FC6m

kAmr@5:?8 E@@=D 2=@?6 2C6?’E 6?@F89] pD k2 9C67lQ9EEADi^^HHH]D?@H7=2<6]4@>Qm$?@H7=2<6k^2m @FE=:?6D 36=@H[ E9:D C6BF:C6D 2 >@C6 7F?52>6?E2= D9:7E] x?DE625 @7 =@@<:?8 E@ px E@ >2<6 6I:DE:?8 AC@46DD6D CF? 72DE6C[ :EVD E:>6 E@ C6E9:?< E96 AC@46DD6D E96>D6=G6D 2?5 >@G6 36J@?5 >2?F2==J >2?:AF=2E:?8 A:A6=:?6D] %@>@CC@HVD 52E2 6?8:?66CD H@?VE 36 D@ 4@?46C?65 H:E9 C@E6 6?8:?66C:?8 2D E96J H:== 36 E96 @FEAFE @7 E92E H@C<[ E96 52E2 AC@5F4E] r@?D:DE:?8 @7 4=62? 2?5 4FC2E65 52E2[ >6E252E2[ 4@?E6IE 2?5 E@@=D[ E96D6 52E2 2DD6ED 2C6 >62?E E@ 36 C6=:23=6 2?5 C6FD23=6[ C625J 7@C ?@E ;FDE 9F>2?D[ 3FE 2=D@ px 286?ED]k^Am

kAm%96 7FEFC6 52E2 6?8:?66CD H:== 7@4FD @? 4C62E:?8 E96D6 52E2 AC@5F4ED E@ 7F6= 3FD:?6DD 564:D:@?D] %CF=J 2FE@?@>@FD 286?ED H:== E96? 3F:=5[ >2:?E2:? 2?5 @AE:>:K6 6G6CJE9:?8 ?66565 E@ <66A E9@D6 AC@5F4ED FA E@ 52E6[ H:E9 9F>2?D 2D E96 <6J 564:D:@? >2<6CD 2?5 2C3:E6CD]k^Am

kAm!2J:?8 2EE6?E:@? E@ ?6H AC@5F4ED ?66565[ AC@3=6>D 2?5 72:=FC6D H:E9 6I:DE:?8 @?6D[ 286?ED H:== :?56A6?56?E=J D62C49 7@C @AA@CEF?:E:6D E@ :>AC@G6[ 4=62? FA 2?5 @AE:>:K6] |62?H9:=6[ 9F>2?D H:== 92G6 >@C6 6=6G2E65 C@=6D[ H@C<:?8 4=@D6=J H:E9 3FD:?6DD E62>D E@ 567:?6 E96 >@DE :>A@CE2?E 3FD:?6DD AC@3=6>D[ 8F:56 E96 286?ED @? E96 C:89E @G6C2== 2C49:E64EFC6 2?5 >2<6 DFC6 E92E E96 4@?E6IE @7 E96 3FD:?6DD :D 2AAC@AC:2E6=J 42AEFC65 7@C E96 286?ED 4@?DF>:?8 E96 52E2 AC@5F4ED]k^Am

k9bmy@FC?6J E@H2C5 2FE@?@>@FD 52E2 6?8:?66C:?8 WpstXk^9bm

kAm%9:D :D 2 3:8 D9:7E[ 2?5 E62>D H@?’E 36 23=6 E@ 86E E96C6 @G6C?:89E] }@C D9@F=5 E96J] p 36EE6C 2AAC@249 :D E@ 2=:8? @? E96 C:89E 8@2=[ 2?5 E96? >@G6 E@H2C5 :E :?4C6>6?E2==J 2?5 E9@F89E7F==J] pE 6249 DE6A[ E62>D H:== =62C? >@C6 23@FE H92E E96 286?ED 2?5 >@56=D 2C6 42A23=6 @7[ 2?5 H:== 3F:=5 ECFDE :? 9@H >F49 42? 36 D276=J 2FE@>2E65[ 2?5 H96C6 J@F ?665 255:E:@?2= 9F>2?\:?\E96\=@@A 4964<D[ E@@=:?8 @C AC@46DD6D] |2?J 4@>A2?:6D 2C6 2=C625J F?56CH2Ji tG6CJ E62> E92E 92D 25@AE65 2 D@7EH2C6\567:?65 =:764J4=6[ G6CD:@? 4@?EC@=[ 564=2C2E:G6 A:A6=:?6D @C px\2DD:DE65 56G6=@A>6?E 92D E2<6? 2 DE6A E@H2C5 E96 7FEFC6]k^Am

k9bm%96 pFE@?@>@FD s2E2 t?8:?66C:?8 >2EFC:EJ 4FCG6k^9bm

k7:8FC6mk:>8 DC4lQ9EEADi^^:>286D]DE24<6C]4@>^H^`ag_^:>286D^DE@C:6D^a_ae^`_^ae^pstTa_>2EFC:EJTa_4FCG6]A?8Q 2=ElQp 52E2 8C2A9 G:DF2=:K:?8 E96 pFE@?@>@FD s2E2 t?8:?66C:?8 |2EFC:EJ rFCG6]Q DEJ=6lQH:5E9i `__TQ ^mk7:842AE:@?m$?@H7=2<6k^7:842AE:@?mk^7:8FC6m

kAmpE kDEC@?8m$E286 `[k^DEC@?8m E96 7@F?52E:@?[ 52E2 6?8:?66CD 3F:=5 2?5 >2:?E2:? A:A6=:?6D >2?F2==J] tG6CJ D496>2 492?86[ @C496DEC2E:@? 564:D:@? 2?5 72:=FC6 C6DA@?D6 C6BF:C6D 9F>2? 677@CE] %62>D E92E EC62E E9:D A92D6 2D E96 E:>6 E@ 86E E96:C 6?8:?66C:?8 7F?52>6?E2=D C:89E H:== 92G6 >@C6 DF446DD :? =2E6C DE286D] |@G6>6?E E@ >@C6 >@56C? 52E2 6?8:?66C:?8 AC24E:46D[ DF49 2D 564=2C2E:G6 A:A6=:?6D[ 42? 86?6C2E6 :>AC@G65 677:4:6?4J]k^Am

kAmkDEC@?8m$E286 ak^DEC@?8m :D H96C6 px 6?E6CD E96 65:E@C] pFE@4@>A=6E6 2?5 :?=:?6 4@56 86?6C2E:@? C65F46 E96 7C:4E:@? @7 EC2?D7@C>2E:@? 2?5 4@?7:8FC2E:@?[ 3FE 6?8:?66CD DE:== @H? 6G6CJ 564:D:@?] %96 7F?52>6?E2= H@C<7=@H 92D?VE 492?865[ 6G6? E9@F89 @FEAFE 92D :?4C62D65]k^Am

kAmkDEC@?8m$E286 bk^DEC@?8m :D H96C6 px DE2CED 364@>:?8 2 4@==23@C2E@C] p86?ED 42? AC@A@D6 492?86D =:<6 A:A6=:?6 >@5:7:42E:@?D[ 72:=FC6 7:I6D 2?5 D496>2 >:8C2E:@?D[ 3FE ?@E9:?8 6I64FE6D H:E9@FE 6IA=:4:E 9F>2? 2AAC@G2=] t?8:?66CD C6G:6H 4@56 :?DE625 @7 HC:E:?8 :E[ D9@CE6?:?8 :?4:56?E C6DA@?D6 E:>6 5C2>2E:42==J]k^Am

kAmpFE@>2E:@? 36?67:ED 368:? E@ ECF=J D9:?6 2EkDEC@?8m $E286 ck^DEC@?8m] p86?ED 24E :?56A6?56?E=J @? D6=64E[ 4=62C=J 567:?65 E2D<D[ DF49 2D 56E64E:?8 2?@>2=:6D 2?5 252AE:?8 A:A6=:?6D E@ FADEC62> 492?86D[ H:E9@FE H2:E:?8 7@C 2AAC@G2=] t?8:?66CD DE:== 92G6 @3D6CG23:=:EJ :?E@ H92E 286?ED 2C6 5@:?8 2?5 42? @G6CC:56 286?EDV 24E:@?D H96? ?646DD2CJ]k^Am

kAmpE kDEC@?8m$E286 dk^DEC@?8m[ 52E2 6?8:?66C:?8 92D 7F?52>6?E2==J 492?865 H92E :E >62?D E@ H@C< :? 52E2] !:A6=:?6D 56E64E E96:C @H? 72:=FC6D 2?5 C@@E\42FD6 :DDF6D 24C@DD E96 DE24<[ 2AA=J 7:I6D[ G2=:52E6 @FE4@>6D 2?5 5@4F>6?E H92E 492?865 H:E9@FE 9F>2? :?E6CG6?E:@?] ~A6C2E:@?2= @G6C9625 2AAC@2496D K6C@]k^Am

kAms2E2 6?8:?66CD 567:?6 E96 A@=:4:6D[ DE2?52C5D 2?5 3FD:?6DD D6>2?E:4D] %96J 2=D@ 564:56 H92E E96 A=2E7@C> :D 2==@H65 E@ 5@ 2FE@?@>@FD=J 2?5 H92E C6BF:C6D E96:C ;F58>6?E[ D@ 9:89\:>A24E 492?86D 2C6 C@FE65 E@ 9F>2? C6G:6H] %96:C C@=6 :D 7F==J D9:7E65 7C@> 6I64FE:@? E@ 8@G6C?2?46[ 2?5 E96JVG6 6=6G2E65 7C@> A:A6=:?6 3F:=56CD E@ 86?F:?6 3FD:?6DD A2CE?6CD]k^Am

k9bmqF:=5 E96 7@F?52E:@?j 2FE@?@>@FD 52E2 6?8:?66C:?8 H:== 7@==@Hk^9bm

kAm|@G:?8 4=@D6C E@ 2 7FEFC6 H:E9 2FE@?@>@FD 52E2 6?8:?66C:?8 WpstX C6BF:C6D 2 D9:7E 7C@> E9:?<:?8 23@FE 52E2 A:A6=:?6D E@ E9:?<:?8 23@FE 52E2 AC@5F4ED[ H9:49 6?4@>A2DD ?@E ;FDE 52E2 3FE D6>2?E:4D[ BF2=:EJ 2?5 6G6CJE9:?8 ?66565 E@ >2<6 52E2 C6=:23=J FD23=6] u@C 6I2>A=6[ 2 4FDE@>6C =@J2=EJ 52E2 AC@5F4E >2J :?4=F56 4FDE@>6C 49FC?\C:D< 52E2[ H9:49 A2:CD 6249 :?5:G:5F2= 244@F?E\24E:G:EJ 52E2 H:E9 2 D92C65 3FD:?6DD 567:?:E:@? @7 H92E Q2E\C:D<Q >62?D[ 2=@?8 H:E9 5@4F>6?E65 BF2=:EJ 4964<D] p?J 286?E @C 2?2=JDE 2446DD:?8 E9:D 52E2 AC@5F4E H@F=5 E96? 86E 2 4@?D:DE6?E[ ECFDEH@CE9J 2?DH6C 7@C E96:C 4FDE@>6C =@J2=EJ 677@CED]k^Am

kAm%96 EC25:E:@?2= >@56= DE2CED 7C@> :?7C2DECF4EFC6 2?5 3F:=5D FAi qF:=5 E96 52E2 7@F?52E:@?[ 567:?6 D496>2D[ 4C62E6 A:A6=:?6D[ E96? D6CG6 2 3FD:?6DD ?665] %92E H@C<65 H96? 52E2 AC@5F4ED H6C6 76H 2?5 DE23=6]k^Am

kAmqFE >@56C? 52E2 E62>D @7E6? ?665 E@ CF? E9:D :? C6G6CD6i $E2CE H:E9 E96 3FD:?6DD AC@3=6> 2?5 H@C< 324< E@ 56D:8? E96 :?7@C>2E:@? J@F ?665 E@ >2<6 2 564:D:@? @C 2FE@>2E6 2 H@C<7=@H] %9:D C6@C:6?ED E96 52E2 6DE2E6 2C@F?5 3FD:?6DD @FE4@>6D C2E96C E92? E649?:42= =2J6CD[ :?EC@5F4:?8 2 G6CE:42= 2AAC@249 E@ 52E2 6?8:?66C:?8 H96C6 A:A6=:?6D 2C6 56D:8?65 7@C C6DF=ED[ ?@E D496>2D]k^Am

kAmx? ECF6 2FE@?@>@FD 52E2 6?8:?66C:?8[ E96 2C49:E64EFC6 @7 J@FC 52E2 6DE2E6 >2EE6CD 2D >F49 2D E96 49@:46 @7 286?E 7C2>6H@C<] v@G6C?2?46[ :?E6C@A6C23:=:EJ 2?5 2 4@??64E65 52E2 6DE2E6 2C6 E96 AC64@?5:E:@?D E92E >2<6 :E D276 E@ ECFDE 2FE@?@>@FD H@C<7=@HD] |@56C? 6?8:?66C:?8 AC24E:46D >FDE 2=D@ 36 :? A=246] '6CD:@? 4@?EC@=[ 2FE@>2E65 E6DE:?8 2?5 rx^rs A:A6=:?6D — E96 D2>6 36DE AC24E:46D E92E 9:89\A6C7@C>:?8 52E2 E62>D 92G6 366? 25@AE:?8 7@C J62CD — F?56CA:? 2FE@>2E:@?]k^Am

kAm%96 E649?:42= A:646D @7 pst 2C6 364@>:?8 2G2:=23=6] %96 @C82?:K2E:@?2= @?6D — 8@G6C?2?46 7C2>6H@C<D[ 4=62C 244@F?E23:=:EJ[ D92C65 567:?:E:@?D @7 H92E 8@@5 52E2 =@@<D =:<6 — E2<6 =@?86C E@ 3F:=5 2?5 >2EE6C ;FDE 2D >F49]k^Am

kAmpst :D H96C6 >@DE @C82?:K2E:@?D 2C6 962565[ H96E96C E96JVG6 ?2>65 :E @C ?@E] v6EE:?8 E96C6 :D 2 >2EE6C @7 E2<:?8 E96 @C82?:K2E:@?2= H@C< 2D D6C:@FD=J 2D E96 E649?:42=]k^Am

k9bm(92E E96 ;@3 364@>6Dk^9bm

kAmx? 2? pst H@C=5[ E96 A:A6=:?6 56E2:=D 364@>6 E96 286?EVD 4@?46C?] %96 6?8:?66CVD 2EE6?E:@? >@G6D FA E@ E96 A@=:4:6D E92E 8@G6C? H92E E96 286?E :D 2==@H65 E@ 5@[ E96 DE2?52C5D E92E 567:?6 H92E 8@@5 52E2 =@@<D =:<6[ 2?5 E96 3FD:?6DD 4@?E6IE E92E >2<6D 2?J @7 :E >62?:?87F=]k^Am

kAm%9:D :D 2 5:776C6?E <:?5 @7 6IA6CE:D6 — =6DD 23@FE 3F:=5:?8 2?5 >@C6 23@FE 567:?:?8] p?5 :E EFC?D @FE[ E96 @C82?:K2E:@?D E92E 2C6 >@DE C625J 7@C 2FE@?@>@FD H@C<7=@HD 2C6 E96 @?6D E92E :?G6DE65 62C=:6DE :? E92E <:?5 @7 E9:?<:?8]k^Am

kAmk2 9C67lQ9EEADi^^HHH]D?@H7=2<6]4@>^6?^3=@8^2FE@?@>@FD\52E2\6?8:?66C:?8^Qmk6>m%9:D DE@CJk^6>mk^2mk6>m H2D AC@5F465 3Jk^6>m k2 9C67lQ9EEADi^^HHH]D?@H7=2<6]4@>^Qmk6>m$?@H7=2<6k^6>mk^2mk6>m 2?5 C6G:6H65 2?5 5:DEC:3FE65 3Jk^6>m k2 9C67lQ9EEADi^^9F3D]=2^"_b<=8$#_Q C6=lQ?@7@==@HQmk6>m$E24<6Ck^6>mk^2mk6>m]k^6>mk^Am