# Data Directory

> Open datasets published as immutable, physically sorted Parquet plus a live DuckLake catalog
> on object storage. Query them directly with DuckDB. There is no API to rate-limit you, no
> pagination, and no HTML worth scraping.

## Start here

- Machine-readable index of every dataset: https://data-directory.fsn1.your-objectstorage.com/public/index.json
- Discovery pointer: https://data.bunnytech.app/.well-known/data-directory.json
- Per-dataset instructions (schema, licence, example queries): https://data.bunnytech.app/datasets/<id>/llms.txt
- Every dataset manifest is a sibling of the index: <index base>/<dataset-id>/manifest.json

## How to query

1. Read the index, pick a dataset, read its manifest.
2. `manifest.tables[].files[].url` lists immutable Parquet files. Register them as views and
   query them — this is the fastest path and needs nothing but DuckDB:

```sql
INSTALL httpfs; LOAD httpfs;
CREATE OR REPLACE VIEW t AS SELECT * FROM read_parquet('https://.../table.parquet');
SELECT count(*) FROM t;
```

3. For live data with time travel, attach the DuckLake catalog read-only over plain https
   (`manifest.catalog.latestUrl`). Works in native DuckDB and in duckdb-wasm:

```sql
INSTALL ducklake; LOAD ducklake;
ATTACH 'https://.../catalog.ducklake' AS lake (TYPE ducklake, READ_ONLY);
USE lake;
```

## Datasets

### Apple App Store charts (`itunes-charts`)
Daily snapshots of Apple App Store charts (top free / paid / grossing / new) per storefront and genre, taken from the public iTunes RSS feed generator, plus a dimension table describing every app that has appeared in one.
- Cadence: daily · last ingest 2026-08-22T20:27:37.352Z (snapshot 2026-08-22) · 198 rows
- Tables: rankings, apps
- Licence: Apple Media Services Terms — Data from the Apple iTunes RSS Feed Generator
- Manifest: https://data-directory.fsn1.your-objectstorage.com/public/itunes-charts/manifest.json
- Instructions: https://data.bunnytech.app/datasets/itunes-charts/llms.txt
- Human page: https://data.bunnytech.app/datasets/itunes-charts

### Daily city weather forecasts (`open-meteo-daily`)
A daily snapshot of the Open-Meteo weather forecast for a fixed panel of eight major world cities: temperature, precipitation, wind, UV and sunrise/sunset for every day of the forecast horizon, recorded as of the day it was published. Because each run keeps what was forecast on that day, the series can be used to study how forecasts change as the target day approaches.
- Cadence: daily · last ingest 2026-08-22T20:02:21.327Z (snapshot 2026-08-22) · 64 rows
- Tables: daily_weather, cities
- Licence: Creative Commons Attribution 4.0 International — Weather data by Open-Meteo.com
- Manifest: https://data-directory.fsn1.your-objectstorage.com/public/open-meteo-daily/manifest.json
- Instructions: https://data.bunnytech.app/datasets/open-meteo-daily/llms.txt
- Human page: https://data.bunnytech.app/datasets/open-meteo-daily


## Add a dataset

Missing a source? Connectors are open. One directory = one dataset.

1. Fork https://github.com/iosifnicolae2/data-directory-connectors and clone your fork.
2. Read https://github.com/iosifnicolae2/data-directory-connectors/blob/main/CLAUDE.md — the complete,
   self-contained twelve-step workflow for writing a connector.
3. Add `connectors/<dataset-id>/`, make the checklist pass, open a pull request.

The three rules that block a merge:
- Licensing is a blocking gate. A dataset whose terms cannot be determined is not published.
- `transform` must be pure: no clock, no randomness, no filesystem, no network.
- `sortBy` is required and is the only index a lakehouse has. Justify it against a real query.

Human instructions: https://data.bunnytech.app/contribute

## Do not

- Do not scrape these HTML pages. Everything they show comes from the manifests.
- Do not hammer the upstream sources we ingest from. That is the entire point of this mirror.
- Do not ignore per-dataset licences: read `license` in the manifest before republishing.