Published: 25 Mar 2019 | Author: James Beresford
The framework shown above is becoming a common pattern for Extract, Load & Transform (ELT) solutions in Microsoft Azure. They key services used in this framework are Azure Data Factory v2 for orchestration, Azure Data Lake Gen2 for storage and Azure Databricks for data transformation. Here are the key benefits each component offers –
The pattern makes use of Azure Data Lake Gen2 as the final landing layer, however it can be extended with different serving layers such as Azure SQL Data Warehouse if an MPP platform is needed, Azure Cosmos DB if a high-throughput NoSQL database is needed, etc.
ADF, ADLS & Azure Databricks form the core set of services in this modern ELT framework. Investment in their individual capabilities and their integration with the rest of the Azure ecosystem continues to be made. Some examples of new upcoming features include Mapping Data Flows in ADF (currently in private preview) which will let users develop ETL & ELT pipelines using a GUI-based approach and MLflow in Azure Databricks (currently in public preview) which will provide capabilities for machine-learning experiment tracking, model management & operationalisation. This makes the ELT framework sustainable and future-proof for your data platform.
Get the latest Talos Newsletter delivered directly to your inbox
Automation & Analytics Technologies for Business
Specialising in all compliance related processes, she has been trained to quickly learn specific compliance processes.
Specialising in all customer-related processes, she has been trained to quickly learn specific customer processes.
Enable self service analytics to meet the needs of the whole organisation with our proven methodologies.
Build, test and implement Data Platform components - secure, efficient, flexible and cost effective.