Change Data Capture (CDC): How It Works and When to Use It
Change Data Capture (CDC) identifies inserts, updates and deletes in a database so those changes can be passed to another system. Instead of copying a whole table every time, a CDC pipeline moves the changes. A warehouse can use it to stay up to date with an operational database, a search index can reflect changes […]
Microsoft Fabric Best Practices for Data Teams
Microsoft Fabric brings data movement, storage, analysis and reporting into one platform. The day-to-day experience still depends on how you organise the data, manage changes and support the people using it. These Microsoft Fabric best practices focus on those choices, from selecting a data store to making sure a failed load does not quietly become […]
Microsoft Fabric: Lakehouse vs Warehouse
The Microsoft Fabric Lakehouse is generally suited to data engineering and data science teams working with diverse data, while a Microsoft Fabric Warehouse is designed primarily for structured data, SQL-based analytics and enterprise data warehousing. Lakehouse vs Warehouse Fabric Lakehouse Fabric Warehouse Best for Data engineering & flexible analytics Structured analytics & data warehousing Data […]
Microsoft Fabric Drawbacks: What to Know
Microsoft Fabric brings data integration, data engineering, data warehousing and Power BI into one platform. If your organisation already uses Microsoft tools, Fabric can make it easier to bring data together and build reports. But could it be that straightforward? The main drawbacks of Microsoft Fabric are its learning curve, capacity and cost management, SQL […]
Is Microsoft Fabric worth it?
Microsoft Fabric can be worth it when your data environment has become complex enough that managing separate systems, pipelines and reporting tools is creating more work than value. But if you have simple reporting needs, limited data sources and a setup that already works well, Fabric may be more than you need. This guide will […]
What Is Microsoft Fabric?
Microsoft Fabric is a unified data and analytics platform that brings different parts of the data lifecycle into one environment. It supports activities such as data ingestion, transformation and storage, analysis, and delivery of reports and insights. Instead of building a data environment from multiple disconnected services, Fabric provides integrated experiences of Data Engineering, Data […]
ClickHouse Primary Keys, Sorting Keys, and Partition Keys
These three concepts control how ClickHouse organises data on disk, how fast it can skip data during queries, and how cheaply you can drop old data. Getting them right is the difference between a schema that scales and one that does not. Most ClickHouse performance problems trace back to one of three schema decisions: the […]
ReplacingMergeTree, SummingMergeTree, and AggregatingMergeTree: When to Use Each
The MergeTree variants are specialised versions of the base MergeTree engine that add specific behaviour during background merges, each solving a different data problem. If you have used the base MergeTree engine and started hitting situations where you need deduplication, pre-aggregation, or update-like behaviour, you are ready for the ClickHouse MergeTree variants. This post explains […]
Getting Started with ClickHouse: Your First Queries and Schema Design
This post walks you through installing ClickHouse, creating your first table, inserting data, and writing queries that actually return useful results fast. Reading about ClickHouse is one thing. Running your first query and seeing it return in milliseconds is another. By the end of this post, you will have a working ClickHouse table, real data […]
ClickHouse Table Engines: Choosing The Right One For Your Workload
A table engine in ClickHouse controls how your data is stored, indexed, merged, and queried. Choosing the wrong one affects both performance and correctness. When you create a table in ClickHouse, you have to declare an engine. This is different from most databases, where the storage mechanism is invisible. In ClickHouse, it is an explicit […]