Query Hugging Face datasets

Training data often lives on the Hugging Face Hub. The questions about it are analytical: which sources account for most of this release? Which labels dominate each stage? Did the latest version fix an imbalance, or introduce a new one?

Answering those questions should not require a separate ingestion pipeline before you can run your first query.

Hugging Face is now available in beta as an external catalog in Altertable. Connect public or private datasets, expose them as tables, and query their files directly on the Hub. You can inspect distributions, build stratified samples, and compare branches, tags, or commits with SQL.

In Catalogs, choose New catalog, then Hugging Face. Select a file format and map table names to dataset locations. Parquet, CSV, and JSON are supported; the file format applies to the catalog.

For example, a catalog named hf could contain this mapping:

A location follows owner/dataset[@revision][/path]: the first table reads files under data/ at the v3 tag; the second follows the repository's default branch.

Leave the access token empty for public datasets. For private datasets, supply a token with read access.

The catalog is read-only. Connecting it does not copy the files into Altertable or write changes back to the Hub. Its tables appear in the catalog browser and are available to the Altertable agent as well as your SQL queries.

Before building a training set, check how records are distributed across sources, processing stages, and labels:

Copy
SELECT source, stage, label, count(*) AS n
FROM hf.main.train_records
GROUP BY source, stage, label
ORDER BY source, stage, label;

A global label ratio can look reasonable while hiding a shortage in one source or stage. Grouping across all three dimensions makes those gaps visible before you build the next training set.

Once you have a targets table with one row per combination and an n_target column, a window function can select up to that many records per group:

Copy
1
2
3
4
5
6
7
8
9
10
11
WITH ranked AS (
  SELECT *, row_number() OVER (
    PARTITION BY source, stage, label
    ORDER BY hash(id, 42), id
  ) AS rn
  FROM hf.main.train_records
)
SELECT r.* EXCLUDE (rn)
FROM ranked r
JOIN targets t USING (source, stage, label)
WHERE r.rn <= t.n_target;

Here, targets is a table you supply, and id is a unique record identifier. Groups with fewer available rows than their target return fewer rows; this query does not redistribute unused quotas. DuckDB's hash() can change across engine versions, so retain the selected IDs if you need to reproduce a sample across upgrades.

On the Hub, a dataset version is a Git revision. Altertable exposes that directly: each mapped table gets a companion <table>_at(rev) table function that accepts a branch, tag, or commit hash.

To compare the label distributions of two training mixes:

Copy
1
2
3
4
5
6
7
8
SELECT 'v1' AS revision, block, label, count(*) AS n
FROM hf.main.train_mix_at('v1')
GROUP BY block, label
UNION ALL
SELECT 'v3' AS revision, block, label, count(*) AS n
FROM hf.main.train_mix_at('v3')
GROUP BY block, label
ORDER BY block, label, revision;

The function keeps the dataset and path from the mapping and changes the revision. You can leave the main table pinned to the release you normally analyze while querying another revision for comparison.

Use commit hashes when a result must refer to an exact dataset state: branches and tags can move. This is Git revision selection, not lakehouse snapshot time travel; AT (VERSION => n) does not apply to these external tables.

DuckDB already provides an hf:// filesystem through its httpfs extension. We used that support rather than building a new extension. Altertable supplies the connection configuration, scoped credentials, table views, and revision functions around it.

Connect the files where they are. Inspect the distribution, build a sample, and compare the revisions that matter, directly from the Hub.

Connect a Hugging Face dataset