Skip to main content
CYBER SCALLYWAGS
Play · Sandbox

Data Integrity Lab

Most data work is not analysis - it is finding out what is wrong with the file before anyone builds on it. This sandbox is about profiling, validating and documenting data, using open datasets messy enough to be worth the trouble.

What you practise here

  • Profiling a dataset before trusting it
  • Schema validation and data contracts
  • Metadata, lineage and documentation
  • Semantic modelling: what the columns actually mean

Notebooks & hosted labs

Environments that run in a browser tab or in one command on your own machine. Nothing here needs a licence.

Starter code samples

Short enough to read in one go and to paste into a cell. Change them until they break — that is the exercise.

A schema that reports rather than crashes

python

Pydantic collects every failure in a row instead of stopping at the first, which is the difference between a validation report and an exception.

from pydantic import BaseModel, EmailStr, Field, ValidationError


class Attendee(BaseModel):
    email: EmailStr
    display_name: str = Field(min_length=2, max_length=80)
    tickets: int = Field(ge=1, le=4)


rows = [{"email": "not-an-email", "display_name": "A", "tickets": 9}]

for line_no, row in enumerate(rows, start=1):
    try:
        Attendee(**row)
    except ValidationError as exc:
        for error in exc.errors():
            print(line_no, error["loc"][0], error["msg"])
# 1 email  value is not a valid email address: ...
# 1 display_name  String should have at least 2 characters
# 1 tickets  Input should be less than or equal to 4

Profile a CSV before you trust it

sql

DuckDB reads the file in place - no load step, no schema declared up front. Run this on any CSV you are handed, before you write anything else.

SELECT
    count(*)                                AS row_count,
    count(DISTINCT email)                   AS distinct_emails,
    count(*) FILTER (WHERE email IS NULL)   AS missing_emails,
    min(signed_up_at)                       AS earliest,
    max(signed_up_at)                       AS latest
FROM read_csv_auto('members.csv');

-- The duplicates you are about to inherit:
SELECT email, count(*) AS copies
FROM read_csv_auto('members.csv')
GROUP BY email HAVING count(*) > 1
ORDER BY copies DESC;

A data contract is just an assertion with a name

python

Expectations written down are the cheapest governance there is. Run them in CI and a broken upstream feed fails loudly instead of quietly.

import pandas as pd

df = pd.read_csv("members.csv", parse_dates=["signed_up_at"])

CONTRACT = {
    "email is never null": df["email"].notna().all(),
    "email is unique": df["email"].is_unique,
    "signed_up_at is not in the future": (df["signed_up_at"] <= pd.Timestamp.now()).all(),
    "tickets between 1 and 4": df["tickets"].between(1, 4).all(),
}

broken = [name for name, holds in CONTRACT.items() if not holds]
assert not broken, f"Data contract violated: {broken}"

Free practice grounds

Places to keep going once the samples run. Every one is usable without paying; where an account is needed we have said so.

  • data.gov.uk (opens in a new tab)

    Free

    Tens of thousands of open government datasets, in every state of repair. Realistic practice precisely because much of it is untidy.

    UK Government

  • StatsWales (opens in a new tab)

    Free

    Welsh statistics down to local-authority level. If you want a project about the Valleys, the numbers start here.

    Welsh Government

  • Kaggle Datasets (opens in a new tab)

    Free tier

    Hundreds of thousands of datasets, each with notebooks other people have already written against it - read their cleaning steps before your own.

    Kaggle

  • Frictionless Data (opens in a new tab)

    Open source

    Open standards and tooling for describing a dataset so somebody else can use it. The practical end of metadata engineering.

    Open Knowledge Foundation

Reference & documentation

What to read when the sandbox stops answering the question.

Keep going with the community

The links above belong to the wider web. These ones belong to us — written, reviewed and taught by members.

Join the Datanista Social Club → Tutorials from this club → Community coding nights → Share what you built →

Other sandboxes

Upcoming events

All events → Suggest an event