From Enriched Dimensions to Common Analytical Patterns

The evolution of data engineering and new patterns for common analytics management

The evolution of data engineering and new patterns for common analytics management

Poom Wettayakorn

Jun 6, 2024

Jun 6, 2024

data-modeling

👋 Hi there, welcome to Datascale: SQL+Data modeling. We help create database diagrams and map out data models from existing schema and SQL queries 🌱. Find us on LinkedIn 🎉

Evolution of data engineering

I just came across this cool article by Meta about the evolution of data engineering, and it got me thinking about what we're doing at Datascale.

Basically, data engineers used to focus mainly on bringing data together from different systems and automating the process through code.

They'd write SQL queries and draw complex diagrams to show how data moved around. This was essential for understanding the flow and relationships within the data, but it also created a lot of complexity.

As companies collect more and more data, it's becoming harder to ensure that everyone is using it correctly and how everything is organized.

Common analytical patterns

Creating common patterns

📌 By automating data relationship diagrams & creating shared definitions/rules for data, data engineers can make it easier for people and AI to understand what the data means (metadata).

This would save countless hours and reduce errors!

I think this shift from just bringing data together to creating common tools and patterns is a big deal for data engineering (and for GenAI). It's about making data accessible and useful for everyone, not just the technical folks.

"The purpose of data integration was to unify metrics & dimensions produced in siloed ETL systems and scale common analytical patterns"

Reverse engineering data models

A bit about our tool, at Datascale, we’re taking this a step further. We're helping data teams reverse engineer database models from DDLs, views, and SQL queries, turning them into easy-to-understand relationship diagrams.

// Feel free to reach out, I'll be happy to learn more how we can help model your database system.

Related blogs

Contents

Section

Model your database diagram

Datascale helps map out data models from existing schema and SQL queries

Try for free

Data Modeling for SQL

Data Modeling for SQL