Skip to content
DocumentationLakehouse
Lakehouse
External Catalogs
Hugging Face

Hugging Face

Hugging Face external catalogs expose datasets hosted on the Hugging Face Hub as tables. Files are read in place from the Hub: nothing is copied into Altertable.

Requirements

  • One or more dataset repositories on the Hugging Face Hub, stored as Parquet, CSV or JSON files
  • For private or gated datasets, a Hugging Face access token with read access to those datasets

Hugging Face catalogs are read-only in Altertable.

Connect a Hugging Face Catalog

  1. Open Catalogs in the target environment.
  2. Click New catalog and choose Hugging Face.
  3. Paste an access token, or leave it empty for public datasets.
  4. Select the file format of the datasets.
  5. Map each table name to a dataset location, written as owner/dataset[@revision][/path].
  6. Test and create the catalog.
LocationReads
acme/train-recordsEvery file of the default branch
acme/train-records/dataEvery file under data/ on the default branch
acme/train-records@v3/dataEvery file under data/ at the v3 tag
acme/train-records@~parquetThe Parquet conversion maintained by Hugging Face

The token is only sent to the dataset repositories mapped in the catalog.

Query Example

SELECT language, stage, label, count(*) AS n
FROM hf.main.train_records
GROUP BY ALL
ORDER BY ALL;

Query Other Revisions

Every mapped table also gets a <table>_at(rev) table function that reads the same dataset at any branch, tag or commit:

SELECT block, label, count(*) AS n
FROM hf.main.train_mix_at('v2')
GROUP BY ALL;

Pin to a commit hash for fully reproducible results, since branches and tags can move.

Performance

Each query reads the dataset files from the Hub, so large scans are bound by download speed and by Hugging Face rate limits, and queries you run again download the same files again.

For a dataset you query repeatedly, copy it into an Altertable catalog once and keep the copy in sync instead:

  1. Open the Altertable catalog that should hold the copy — not this Hugging Face catalog, which is read-only.
  2. On its Ingest tab, click New synchronizer in Synchronizers.
  3. Choose External catalog as the source type, select this Hugging Face catalog, and pick the schema and tables to copy.
  4. Choose the destination schema and a schedule.

Each run replaces the copied tables with the latest data from the Hub, so queries read the local copy instead of downloading the dataset again.