Things To Know About The ELT Model For Cloud-Optimized Environments

Chetan Mathur is CEO of Next Pathway, the Automated Cloud Migration company. 

Cloud-based systems are enabling us to fundamentally change the way we work, from streamlining business operations to leveraging data analytics to driving innovative digital applications. Within the paradigm shift from legacy data centers to cloud-based systems lies another beneficial paradigm shift: a change in the model used to move data from one system to another.

When moving data from a legacy data center to a data warehouse, the traditional approach would be to extract, transform and then load the data (“ETL”). With this model, you’re extracting the raw data you need, moving it to a staging area and then converting and structuring the data for the desired target source before uploading the data to the data warehouse for use.

The more cloud-friendly approach uses a slightly different model — extract, load and transform ("ELT"). Using this approach, transformations occur after the data has been loaded to the data warehouse. Cloud-based systems allow for efficient and cost-effective extraction and loading of all your data — structured, unstructured or semi-structured — into block storage in the cloud target. You can then manipulate that data, provisioning and running transformations and analytics without restrictions.

The ELT model allows for a self-service approach. On-premise data warehouses have predefined data structures that require you to write the transformations to load that data into those data structures. Now, you can instead land that data on your cloud target as is, and then do whatever manipulations you want. ELT provides a flexible and agile approach to managing your data.

Keep in mind, though, that ELT is not replacing ETL. There are use cases in which the ETL model still makes sense, particularly for smaller jobs that only use structured data. But we’re finding that — given its compute power — the transformation logic tends to be better when done inside the cloud.

From The Old To The New

ETL works well in a pre-cloud environment, where 99.9% of data movement and transformation happens within the walls of a data center. When data is moving from application A to application B to application C, many CIOs and CISOs are more comfortable with it remaining in their walled garden — their data center. The data feels protected.

But in today’s world in which data might be moving from a data center into a data warehouse, then pulled out of that system and moved into another, facilitated by different tools within a SaaS platform, you don’t want to be monkeying around with the data as you move it from A to B to C. You want to ensure that data integrity is maintained so that anyone can refer back to that data and know where it came from.

Benefits Of An ELT Model When Using The Cloud

The ELT model offers several benefits when used in the modern, cloud-based environment. Here are a few:

Data integrity: Protecting data integrity is a key reason ELT is typically a good choice when moving data within cloud-based systems. The ELT model pushes the transformation logic down to the warehouse. Only the raw data is being moved and loaded to the cloud; transformations and manipulations are happening downstream, when the data is ready to be applied for an AI end-use case or a business operation. When that data lands on the target in its raw form from the source system, your analytics people or data team will know that they’re using the right or accurate data. The data remains pure during its journey.

Flexibility in data use: The ELT model lets analysts use the raw, untransformed data to run new and/or different provisioning or transformations to fulfill the end-use case. You can also easily test and augment different queries.

Scalability: One of the main advantages of a cloud-based system is its scalability — its nearly infinite storage capacity. The modern cloud data warehouse separates storage from compute resources, enhancing scalability. With an ELT model, if your data needs increase, you can easily and quickly increase your storage.

Low maintenance: As a general rule, ELT requires little maintenance, as ELT tools tend to automate the process. Also, with ELT, transforming is the final step in the journey, so if there’s a bug in the pipeline, it can be fixed beforehand. It’s also less costly to maintain jobs as raw data.

As companies look to further optimize their cloud-based systems, they’ll want to consider the ELT model, as its approach more closely aligns with many of the advantages offered by migrating to the cloud from on-premise legacy systems: low maintenance, flexibility, scalability and the power to drive analytics, business operations and innovation.

Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?