Airbnb Market Intelligence Pipeline
LiveOverview
The Airbnb Market Intelligence Pipeline is a reproducible data engineering and analytics project built from the June 15, 2026 Inside Airbnb snapshot for Amsterdam. It ingests detailed and summary listings, calendar and review data, neighbourhood records, and GeoJSON boundaries, then validates and models them into analysis-ready Parquet and DuckDB layers.
The analytical layer combines descriptive reporting, statistical tests, geographic views, and leakage-aware price-model benchmarks. The best published benchmark was a histogram gradient boosting model with an original-scale test MAE of 89.94 across 1,276 test listings; it is an experimental comparison, not a production pricing system.
The project deliberately documents where the data stops supporting a claim. Availability and revenue estimates are proxies, statistical associations are not causal, feature importance is not causal evidence, and one Amsterdam snapshot does not establish generalisation to other cities or time periods. That boundary is part of the deliverable, not a footnote.
Architecture
Challenges & Solutions
The seven source files differ in grain, schema, and completeness, so a successful download does not mean the dataset is analytically safe.
Built explicit contracts and quality gates for schemas, row counts, keys, domains, and cross-file relationships, with failed records preserved in quarantine outputs.
Airbnb availability, occupancy, revenue, and review activity are tempting to treat as observed business outcomes even though the source data only supports proxies.
Kept proxy definitions and limitations beside the generated evidence and avoided causal language in reports, model interpretation, and the public documentation.
A notebook-first workflow can produce attractive charts without a reproducible lineage from raw files to the result.
Made configuration, manifests, typed Parquet layers, DuckDB models, tests, reports, and the dashboard consume one repeatable pipeline rather than separate manual extracts.
Pipeline Scale
Price-Usable Listings
6,377
Records used for price analysis and modeling
Calendar Rows
3.82M
Availability records validated and transformed
Automated Tests
120
Passing tests reported for the public release
Source Datasets
7
Listings, calendar, reviews, and geography files
Features
- Config-driven ingestion of seven Inside Airbnb files with a reproducible raw-data manifest
- Schema, row-count, key-integrity, and domain checks before data enters the analytical layer
- Quarantine paths for invalid records instead of silently dropping quality failures
- Typed Parquet outputs and a DuckDB star schema for fast local analytical queries
- Statistical testing with documented assumptions, effect sizes, and multiple-comparison controls
- Price-model benchmarks with leakage-aware features and original-scale error reporting
- Generated reports, plots, notebooks, and a Streamlit interface backed by the same pipeline outputs
- Explicit proxy and causality limitations so exploratory signals are not presented as business facts
