Press "Enter" to skip to content

Curated SQL Posts

Chain-Ladder Reserving Calculations in Python

Christian Lorentzen digs into loss reserving:

Ask a reserving actuary how they run a Chain-Ladder and you’ll usually hear “Excel” or the name of a pricey specialized tool. It turns out a modern dataframe library handles it just as well — in a few lines, for hundreds of companies at once.

We use the CAS loss reserving data, specifically the other liability line of business (LoB): 233 US insurers (“GRNAME”), 10 accident years (1998–2007), paid and incurred losses at every development lag (1-10). We treat 2007 as our reporting year, i.e. we simulate a year-end closing.

Click through for a demonstration and a comparison against R’s ChainLadder package.

Leave a Comment

Dealing with Bias in Data Stories

Cole Nussbaumer Knaflic provides some advice:

Yesterday, a participant in one of our workshops asked a question that I’ve been thinking about ever since. She works with researchers and statisticians, and the idea of telling a story with data can make some of them uncomfortable. Doesn’t telling a story mean choosing a perspective? And doesn’t choosing a perspective introduce bias? Wouldn’t it be more objective to simply show the data and let people draw their own conclusions?

Read on for Cole’s answer. Cole differentiates between “bias” and “perspective,” though I wouldn’t even go that far. Instead, I’d simply make it clear what your biases are. At its core, a bias is a preference for or against something. We normally think of bias as a negative, but that’s not strictly true. I do think it’s important to state where your biases come in, as they can undermine a point or attempt to drive people to a specific conclusion. But bias itself can extend to “I am showing you this information because I believe it will help you in making your decision,” even if that decision isn’t one you necessarily prefer.

Leave a Comment

Shortcuts to Iceberg Tables in Microsoft Fabric

Teo Lachev makes a connection:

Snowflake is an increasingly popular option for data warehousing. One nice Snowflake feature is the ability to configure a table to save its data in Iceberg file format in the Snowflake own managed storage or external storage, including S3, GCS, ADLS Gen 2, and OneLake. As I wrote in the post “Give Me Your Data!”, Fabric supports shortcuts to Iceberg tables, in which case the data is not copied but exposed as a Parquet Delta table in a Fabric lakehouse. This could be useful when building Fabric-centric solutions on top of Snowflake without moving the data.

Iceberg isn’t just a Snowflake thing, though this is certainly helpful if you do have both Snowflake and Fabric.

Leave a Comment

Migrating a Power BI Report between Tenants

Reitse Eskens makes a move:

Before I get started, yes, I’m going to write about Power BI. If you’re not familiar with my work, the long and short of it is that I tend to keep a healthy distance from anything involving semantic models and reports. Why? In one word, pie charts. Sorry if that triggered you.

But every now and again, I need to do work adjacent to this; specifically, when onboarding new clients or helping them migrate things, I have to touch the Power BI stuff too. But rest assured, a lot needs to happen before I’ll actually start creating reports. Many people are more qualified than me to do that.

Click through to see how you can do this through the Fabric/Power BI portal. And if you want to see this made easier, vote up Reitse’s suggestion.

Leave a Comment

Setting Fabric Connection Properties with the ODBC Driver

Ryan Adams digs into configuration settings:

Some SQL Server ingestion scenarios depend on connection-string properties that the built-in SQL Server connector in Data Factory for Microsoft Fabric doesn’t expose. If you need to route reads to an Availability Group secondary, connect reliably across a multi-subnet listener, or read columns protected by Always Encrypted, you’ve probably hit that wall. This post shows a supported way around it: use the ODBC connector together with an ODBC data source name (DSN) configured on an on-premises data gateway (OPDG). The example in this post uses Always Encrypted in a Copy job.

Click through to see some of the things you can set. These are standard SQL Server properties, so they can also work with normal SQL Server installations that are making use of Availability Groups and Always Encrypted.

Leave a Comment

Configuring Row-Level Security with OneLake Security

Reza Rad secures some data:

If you’ve set up row level security in Power BI before, you know the usual drill: open the semantic model, add a role, write a DAX filter. But once your data lives inside a Microsoft Fabric OneLake structure, there’s a better way to do this. You can implement it directly in OneLake instead, so that every object built on top of that data, your lakehouse, your SQL analytics endpoint, your semantic model, and your report, all follow the same security setup automatically. Define it once, upstream, and everything downstream inherits it. This is called OneLake security, and in this post I’ll walk through exactly how to set up row level security (RLS) this way, from the lakehouse all the way to your Power BI report.

Read on for a video and summary with timestamps.

Leave a Comment

Broker Tasks on Query Store Cleanup

Erik Darling does some digging:

While looking at a client’s server, I noticed there was, at any given time, 2-3 background sessions with the command column saying BRKR TASK, which I found quite odd.

Only the Microsoft-shipped broker queues were present, and none were activated. Nothing uses Service Broker, Mirroring, or Log Shipping.

Read on for Erik’s findings. If you need a Misfits song to listen to, we are close enough to the end of October for Halloween to work.

Leave a Comment

Finding Unused Objects in Power BI Semantic Models

Chris Webb goes looking for unused objects:

A few weeks ago I wrote about how Semantic Link Labs now has tools with interactive UIs and showed how you could use this to view lineage and so Vertipaq-Analyzer-stuff in a notebook. I didn’t show what I think is the coolest new feature though: the ability to find the tables, columns and measures in a semantic model that aren’t used and which therefore could be deleted. There are tons of excellent third-party tools that do this available already of course but the advantage of using Semantic Link Labs for this is that you can automate the process of finding unused columns and run it from a notebook and, crucially, you have two ways of finding those unused columns: from analysing the structure of downstream reports and from analysing DAX queries captured in Workspace Monitoring.

Read on to see how it works.

Leave a Comment

Digging into Entity-Attribute-Value Tables

Greg Low has started a series on entity-attribute-value tables. The first post covers what they are:

If you’ve been working with databases for any length of time, you will have come across implementations of Entity-Attribute-Value (EAV) tables (or non-tables as some of my friends would call them).

Instead of storing details of an entity as a standard relational table, rows are stored for each attribute.

The second post covers pros and cons:

In an earlier post , I discussed the design of EAV (Entity Attribute Value) tables, and looked at why they get used. I’d like to spend a few moments now looking at the pros and cons of these designs.

Greg is very much against EAV, and I agree with this. I do like Greg’s alternative of using something like JSON, with the proviso that the database simply become a whole-record storage and retrieval engine rather than trying to strip out and splice in new JSON via T-SQL. Otherwise, spend the time on proper data modeling and take advantage of what the platform can do for you.

1 Comment