Overview

EuroDealers is a global bicycle dealer directory that aggregates, deduplicates, and quality-scores retail locations from 10 distinct data sources into a single unified record set.

Data is collected by scraping brand locator APIs, open community databases, and industry directories. Raw records flow through a multi-stage pipeline that normalises contact details, blocks candidates for comparison, applies fuzzy name matching, and merges field values by source authority — producing a single Dealer record per physical location.

Data Sources

Ten sources are registered in the system, falling into three broad categories: OpenStreetMap Brand Locators Industry Directories Enrichment Only

Listed in descending merge priority (higher = wins conflicts).

The Pipeline

The pipeline transforms raw scraped records (RawDealerRecord with status="unprocessed") into unified, deduplicated Dealer rows. Run it with manage.py run_pipeline or via the Celery pipeline queue.

Dealer Fields & Definitions

IBD Status

IBD stands for Independent Bike Dealer — a privately owned, single-location (or small-chain) bicycle shop, as distinct from a large retail chain or big-box store.

Classification is applied during the Classify pipeline step. The highest-confidence signal wins when multiple sources contribute.

Note: is_verified=True records are never reclassified by the pipeline. Staff can override IBD status through the CRM.

Data Quality Score

The data_quality_score (0.0–1.0) reflects how completely a dealer record is populated, boosted slightly for records confirmed by multiple independent sources.

Fields counted (8 total):

• name
• address
• city
• postal_code
• phone
• email
• website
• latitude and longitude (counts as 1)

Management Commands

All commands use the Django management framework: manage.py <command> [options]

Tip: Prefix commands with PYTHONIOENCODING=utf-8 on Windows to avoid encoding errors when output contains Unicode or emoji.

Glossary

Raw Record

A RawDealerRecord — the unprocessed output of a single scraper run for one physical store. May contain duplicates and inconsistencies. Has status "unprocessed" until the pipeline runs.

Blocking

A technique to avoid comparing every raw record against every other (O(n²)). Records are only compared if they share at least one blocking key (zip+name-prefix, phone, website domain, or name n-gram). Reduces comparison work by ~99%.

Fuzzy Matching

Name similarity is computed using RapidFuzz's token-sort ratio, which handles word-order variation ("Bike Shop City" vs "City Bike Shop"). Scores range 0–100; the pipeline threshold is 75 (equivalent to 0.75).

Source Priority

When two sources provide conflicting values for the same field, the higher-priority source wins. Priority order (high → low): OSM > NBDA > Trek > ENVE > Specialized > Giant > Yelp > BikeExchange > Nominatim > OpenCorporates.

Geocoding

Converting a text address into latitude/longitude coordinates. Done automatically by scrapers that return coordinates directly (OSM, brand locators) or retroactively via the Nominatim geocoder for address-only records.

Overpass API

A read-only API for querying OpenStreetMap data using the Overpass Query Language (QL). Queries can select nodes/ways by tags (e.g. shop=bicycle) within a bounding box or radius. Hosted at overpass-api.de.

Nominatim

OSM's official geocoding and reverse-geocoding service. Converts addresses to coordinates and vice-versa. Strict usage policy: max 1 req/s, no bulk geocoding without prior permission.

IBD

Independent Bike Dealer. A privately owned bicycle shop not affiliated with a national retail chain. The NBDA (National Bicycle Dealers Association) is the primary US trade body for IBDs.

Data Quality Score

A 0.0–1.0 metric reflecting how completely a Dealer record is populated, with a bonus for multi-source confirmation. Displayed as dots in the UI.

Pipeline

The multi-stage process (Normalize → Block → Match → Merge → Classify & Score) that converts raw scraped records into unified Dealer rows. Triggered by manage.py run_pipeline or the Celery pipeline queue.

E.164

International telephone number format standardised by the ITU. All phone numbers are stored as +[country code][area code][number] with no spaces or dashes (e.g. +14155551234). Used as a blocking key during deduplication.

WGS-84

World Geodetic System 1984 — the coordinate reference system used by GPS and all coordinates in this project. Latitude and longitude are stored as decimal degrees (e.g. 48.8566, 2.3522 for Paris).