Published: 22 Nov 2023 | Author: James Beresford
One of the latest announcements for Fabric is mirroring. Combined with the existing Shortcuts feature, this is another step towards making Fabric an all encompassing cloud data virtualisation platform. Let's have a quick run through these topics.
One of the challenges with trying to build out a complete data estate is that data is rarely all in one location. The larger the organisation, the more complex the landscape will be, with multiple clouds, warehouses and lakes in play.
The Data Virtualisation paradigm is one where the analytical layer doesn't really care where the data is located. By using its own repository of metadata (data about the data) it can perform analytics on data regardless of where it is physically located.
This is of course a huge simplification of the concept, but it is clearly where Microsoft is trying to take the Fabric platform.
The Mirroring feature is conceptually similar to the replication capability that's been available in SQL server (and most other enterprise database platforms) for some time.
For supported systems - currently Azure Cosmos DB, Azure SQL DB and Snowflake, and coming are SQL Server, Azure PostgreSQL, Azure MySQL & MongoDB - you effectively get a live feed of data from these systems to OneLake. This is leveraging the Change Data Capture (CDC) capabilities of the source system, so we can assume some source side configuration is required.
Shortcuts are links to other cloud storage (Currently AWS S3 Buckets or ADLSGen2, more to come) that allow Fabric to extract data from these as if they were part of OneLake.
A key difference between this and Mirroring will be performance - depending on the location of the data it may have some network lag to accomodate.
Fabric is gearing up to provide a very flexible platform for performing analytics by simplifying the process of accessing source data. Some of these solutions will not be a substitute for a properly managed data lake, but they will facilitate the analysis required to build and model data at a much lower cost. The mirroring and shortcut features are great accelerators for performing analytics.
Don't forget to join us at the Official Microsoft Sydney Fabric Meetup to keep on top of all things Fabric!
Cheers, James
CEO, Talos
Get the latest Talos Newsletter delivered directly to your inbox
TECHNOLOGY PARTNERS
Our partners including Microsoft, UiPath, Databricks & Profisee enable us to deliver business outcomes using best of breed technologies and solutions.
Automation & Analytics Technologies for Business
Enable self service analytics to meet the needs of the whole organisation with our proven methodologies.
Build, test and implement Data Platform components - secure, efficient, flexible and cost effective.
Specialising in all invoice-related processes, he has been trained to quickly learn specific invoice-related processes.
Specialising in all compliance related processes, she has been trained to quickly learn specific compliance processes.
