Integrate Social Mapping across the platform

.dockerignore

@@ -1,5 +1,11 @@
 .git
+.venv
 **/.git
+**/.venv
+**/node_modules
+**/dist
+**/build
+**/__pycache__
 __pycache__
 *.pyc
 *.pyo
@@ -8,3 +14,4 @@ __pycache__
 .env
 .env.*
 .DS_Store
+*.log

.env.example

@@ -4,6 +4,13 @@
 # Create it in an X Developer Portal project/app. Keep it secret.
 TWEEPY_BEARER_TOKEN=your_tweepy_bearer_token
 
+# social_mapping: Nominatim resolves only investigator-entered search origins.
+# Keep the identifying User-Agent and a contact/project URL when using the public service.
+NOMINATIM_BASE_URL=https://nominatim.openstreetmap.org
+NOMINATIM_USER_AGENT=social-mapping/1.0 (+https://github.com/osint-services/social_mapping)
+GEOCODER_CACHE_DAYS=30
+GEOCODER_MIN_INTERVAL_SECONDS=1
+
 # phone_search: Twilio API credentials from the Twilio Console.
 # The Account SID normally starts with AC. Keep the auth token secret.
 TWILIO_ACCOUNT_SID=your_twilio_account_sid

.github/workflows/integration.yml

@@ -0,0 +1,50 @@
+name: Platform integration
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+          cache: pip
+      - run: >-
+          pip install pytest pytest-asyncio
+          -r dataset_service/requirements.txt
+          -r profile_checker/requirements.txt
+          -r profile_search/requirements.txt
+          -r phone_search/requirements.txt
+      - run: pytest -q tests
+      - run: docker compose config --quiet
+
+  compose-e2e:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          submodules: recursive
+      - run: docker compose up --build -d dataset_service social_mapping proxy
+      - name: Wait for map readiness
+        run: |
+          for attempt in {1..30}; do
+            if curl --fail --silent http://localhost/map/readyz >/dev/null; then
+              exit 0
+            fi
+            sleep 2
+          done
+          docker compose logs social_mapping dataset_service proxy
+          exit 1
+      - run: python3 scripts/test_social_mapping_e2e.py
+      - if: always()
+        run: docker compose down --volumes

.gitignore

@@ -1,4 +1,5 @@
 .env
+.venv/
 __pycache__/
 *.py[cod]
 finder.log

.gitmodules

@@ -1,12 +1,15 @@
 [submodule "profile_checker"]
 	path = profile_checker
-	url = git@github.com:osint-services/profile_checker
+	url = https://github.com/osint-services/profile_checker.git
 [submodule "profile_search"]
 	path = profile_search
-	url = git@github.com:osint-services/profile_search
+	url = https://github.com/osint-services/profile_search.git
 [submodule "phone_search"]
 	path = phone_search
-	url = git@github.com:osint-services/phone_search
+	url = https://github.com/osint-services/phone_search.git
 [submodule "whoisit"]
 	path = whoisit
-	url = git@github.com:osint-services/whoisit.git
+	url = https://github.com/osint-services/whoisit.git
+[submodule "social_mapping"]
+	path = social_mapping
+	url = https://github.com/osint-services/social_mapping.git

CHANGELOG.md

@@ -5,6 +5,10 @@ All notable changes to this project will be documented in this file.
 ## [Unreleased]
 
 ### Added
+- Integrated `social_mapping` as a pinned submodule and Compose service with `/map/*` nginx routing, readiness checks, and persistent geocoder cache storage.
+- Added explicit geospatial dataset fields, additive migration, radius search, GeoJSON provenance, and associated phone metadata.
+- Added synthetic London, Arlington, and Portland entities plus credential-free import and end-to-end helper scripts.
+- Added a platform CI path that validates tests, Compose configuration, and the complete synthetic London map flow.
 - Added a sparse entity model that associates optional profile and phone identifiers while preserving their type-specific metadata.
 - Added a persistent SQLite dataset service for mapped profile and phone imports.
 - Added CSV, JSON, JSONL, and NDJSON parsing, previews, automatic field suggestions, per-row validation, provenance, and raw-record retention.
@@ -15,6 +19,8 @@ All notable changes to this project will be documented in this file.
 - Expanded regression coverage for dataset import, fuzzy profile search, normalized phone search, deletion, health endpoints, and client-side parsers.
 
 ### Changed
+- Routed the native whoisit Map workspace and standalone Social Mapping client through the same canonical `/map/search` API.
+- Switched public submodule URLs to HTTPS so recursive clones do not require organization SSH keys.
 - Replaced separate profile and phone dataset ingestion with one entity mapping pipeline; legacy API imports remain compatible and existing SQLite databases migrate in place.
 - Unified username and phone lookup into one Search workspace with conservative auto-detection, explicit Profile and Phone modes, and source filters that control which APIs and datasets are queried.
 - Reworked the Electron interface into cohesive Search, Datasets, Integrations, and History workspaces with type-specific master/detail results.

Dockerfile

@@ -9,13 +9,18 @@ COPY ./profile_checker /app/profile_checker
 COPY ./profile_search /app/profile_search
 COPY ./phone_search /app/phone_search
 COPY ./dataset_service /app/dataset_service
+COPY ./social_mapping /app/social_mapping_repo
 
 # Install the selected service dependencies
 ARG APP_DIR=profile_checker
-RUN pip install --no-cache-dir -r /app/${APP_DIR}/requirements.txt
+RUN if [ "$APP_DIR" = "social_mapping" ]; then \
+      pip install --no-cache-dir -r /app/social_mapping_repo/requirements.txt; \
+    else \
+      pip install --no-cache-dir -r /app/${APP_DIR}/requirements.txt; \
+    fi
 
 ENV PYTHONUNBUFFERED=1
-ENV PYTHONPATH=/app
+ENV PYTHONPATH=/app:/app/social_mapping_repo
 
 EXPOSE 8000
 

README.md

@@ -1,97 +1,100 @@
 # OSINT Services Platform
 
-The platform combines live public-data APIs with user-supplied datasets behind one nginx endpoint and one Electron investigation workspace.
+The platform combines public-data services, user-supplied datasets, and evidence-oriented mapping behind one nginx endpoint and the [`whoisit`](https://github.com/osint-services/whoisit) desktop workspace.
 
 ## Architecture
 
 ```text
-whoisit desktop app
-        |
-        v
-nginx :80
+whoisit desktop app and standalone Social Mapping client
+                            |
+                            v
+                         nginx :80
   |-- /scan/*          -> profile_checker  (public username availability)
-  |-- /focus*          -> profile_search   (X profile metadata via Tweepy)
-  |-- /phone_search*   -> phone_search     (caller-name data via Twilio)
-  `-- /datasets*       -> dataset_service  (SQLite entity imports and local search)
+  |-- /focus*          -> profile_search   (X profile metadata)
+  |-- /phone_search*   -> phone_search     (caller-name data)
+  |-- /datasets*       -> dataset_service  (SQLite imports and geo search)
+  `-- /map/*           -> social_mapping   (GeoJSON aggregation, X posts, origin geocoding)
 ```
 
-Docker Compose runs the four FastAPI services, nginx, and a persistent `dataset_data` volume. The Electron UI provides one search workspace with Auto, Profile, and Phone modes plus All, Live APIs, and Imported datasets source filters. It queries enabled sources independently and combines successful results, so one unavailable provider does not prevent other sources from being searched.
+Docker Compose runs the five FastAPI services, nginx, a persistent `dataset_data` volume, and a separate durable `social_mapping_cache` volume. The map service queries datasets internally and can use the same X bearer-token configuration as profile search. `whoisit` calls `/map/search` natively; it does not embed the standalone client.
 
-## Requirements
+## Requirements and setup
 
-- Docker with `docker compose`
-- Node.js 18 or newer to build or run the Electron UI
-- An X API bearer token for profile inspection
-- Twilio Account SID and Auth Token for live phone lookup
-
-## Setup
+- Docker with Compose v2
+- Node.js 22 or newer to build the current desktop UI
+- Optional X bearer token for profile inspection and X map posts
+- Optional Twilio credentials for live phone lookup
 
 ```bash
 git submodule update --init --recursive
 cp .env.example .env
-```
-
-Add your provider credentials to `.env`, then start the API stack:
-
-```bash
 ./scripts/start.sh
 ```
 
-Run the desktop UI from a second terminal:
+Run the desktop UI from another terminal:
 
 ```bash
 cd whoisit
 npm install
 npm start
 ```
 
-The UI checks service readiness on launch and can attempt to start the Compose stack when Docker is available. Its Integrations workspace shows the configured and reachable state for X/Tweepy, Twilio Lookup, and local datasets. Users can replace provider credentials there without exposing existing values to the renderer. Packaged builds are created with `npm run package`.
+The imported-dataset map works without live credentials. Load the synthetic fixture and verify London end to end:
+
+```bash
+python3 scripts/import_geospatial_demo.py
+python3 scripts/test_social_mapping_e2e.py
+```
 
 ## Environment values
 
 | Variable | Required | Used by | Meaning |
 | --- | --- | --- | --- |
-| `TWEEPY_BEARER_TOKEN` | For live profile inspection | `profile_search` | X API v2 application bearer token. Despite the variable name, the credential is issued by X; Tweepy is the Python client. |
-| `TWILIO_ACCOUNT_SID` | For live phone lookup | `phone_search` | Twilio account identifier, normally beginning with `AC`. |
-| `TWILIO_AUTH_TOKEN` | For live phone lookup | `phone_search` | Secret used with the Account SID to authenticate Twilio API calls. |
-| `DATASET_DB_PATH` | No | `dataset_service` | SQLite file location. Compose sets this to `/data/datasets.db` inside its persistent volume. |
+| `TWEEPY_BEARER_TOKEN` | No | `profile_search`, `social_mapping` | X API v2 bearer token; map searches degrade to datasets when absent |
+| `TWILIO_ACCOUNT_SID` | No | `phone_search` | Twilio account identifier |
+| `TWILIO_AUTH_TOKEN` | No | `phone_search` | Twilio API secret |
+| `DATASET_DB_PATH` | No | `dataset_service` | Compose uses `/data/datasets.db` in persistent storage |
+| `NOMINATIM_BASE_URL` | No | `social_mapping` | Server-side search-origin geocoder |
+| `NOMINATIM_USER_AGENT` | Recommended | `social_mapping` | Identifying User-Agent and contact/project URL |
+| `GEOCODER_CACHE_DAYS` | No | `social_mapping` | Durable normalized place-cache refresh interval; default 30 |
+| `GEOCODER_MIN_INTERVAL_SECONDS` | No | `social_mapping` | Process-wide Nominatim request spacing; default 1 |
 
-Do not commit `.env`. Imported datasets may contain sensitive or licensed information; only import data you are authorized to retain and use.
+Do not commit `.env`. Imported datasets may contain sensitive or licensed information; import only data you are authorized to retain and use.
 
 ## API routes
 
 | Route | Purpose |
 | --- | --- |
 | `GET /scan/{username}` | Check supported sites for a public username |
 | `GET /focus?url=https://x.com/{username}` | Retrieve expanded X profile metadata |
-| `GET /phone_search?phone_number=+18135551212` | Retrieve live caller-name metadata |
+| `GET /phone_search?phone_number=...` | Retrieve live caller-name metadata |
 | `GET /datasets` | List imported datasets |
-| `GET /datasets/schema/entity` | Get the unified sparse entity schema and sample |
-| `POST /datasets/import` | Import mapped entities containing profile identifiers, phone identifiers, or both |
-| `GET /datasets/search/profiles?query=...&fuzzy=true` | Search imported profiles |
-| `GET /datasets/search/phones?phone_number=...` | Search imported phone records |
-| `DELETE /datasets/{dataset_id}` | Delete a dataset and its records |
+| `POST /datasets/import` | Import sparse entities, profiles, or phone records |
+| `GET /datasets/search/geo` | Radius-search only records with explicit coordinate pairs |
+| `GET /map/search` | Return combined GeoJSON for a place or coordinate origin |
+| `GET /map/healthz` | Map process liveness |
+| `GET /map/readyz` | Map cache and configuration readiness |
+
+Dataset imports accept nullable `latitude`, `longitude`, `location_accuracy`, and `location_source` fields. Latitude and longitude must be supplied as a valid pair. The migration is additive for existing SQLite databases. Free-text record locations are not geocoded into evidence coordinates. Associated phone records can appear in marker details but never produce or imply phone/device locations.
 
-The import API accepts sparse entity rows after the UI parses CSV, JSON, JSONL, or NDJSON. Each row needs at least one username, profile URL, or E.164 phone number; profile and phone fields can coexist and remain associated in search results. A single import is limited to 10,000 records and the UI limits files to 10 MB. Canonical records preserve their source row, dataset provenance, observation time, and optional confidence score. Legacy typed imports remain API-compatible and existing databases migrate in place.
+`/search/tweets?city=&distance=` remains a deprecated compatibility alias until the next major Social Mapping release.
 
 ## Operations and tests
 
 ```bash
-./scripts/build.sh      # rebuild images
-./scripts/logs.sh       # stream Compose logs
-./scripts/stop.sh       # stop the stack
-.venv/bin/pytest -q     # backend tests
-cd whoisit && npm test  # importer parser tests
+./scripts/build.sh
+./scripts/logs.sh
+./scripts/stop.sh
+.venv/bin/pytest -q tests
+docker compose config --quiet
 ```
 
-Liveness endpoints are available at `/scan/healthz`, `/phone_search/healthz`, and `/datasets/healthz`. Profile inspection also exposes `/focus/readyz`, which verifies that its X credential is configured.
+CI mocks external X and Nominatim behavior, runs platform tests, validates Compose, and starts the dataset/map/proxy path for a credential-free synthetic London test. Live X testing is optional and deliberately separate so routine CI never spends X credits.
 
 ## Documentation
 
+- [Social Mapping](social_mapping/README.md)
 - [Desktop UI guide](whoisit/README.md)
 - [Dataset service guide](dataset_service/README.md)
-- [Sample combined entity dataset](examples/datasets/sample_entities.csv)
-- [Sample profile dataset](examples/datasets/sample_profiles.csv)
-- [Sample phone dataset](examples/datasets/sample_phone_records.jsonl)
+- [Synthetic geospatial fixture](examples/datasets/geospatial_entities.csv)
 - [Changelog](CHANGELOG.md)
-- [2026-07-17 release notes](CHANGELOG-2026-07-17.md)

dataset_service/README.md

@@ -1,42 +1,41 @@
 # Dataset service
 
-The dataset service provides normalized, persistent identity sources for the platform's profile and phone searches. It uses SQLite and has no external provider dependency.
+The dataset service stores normalized identity evidence for profile, phone, and geospatial searches. It uses SQLite and has no external provider dependency.
 
-## Supported input
+## Imports and coordinates
 
-The Electron UI parses `.csv`, `.json`, `.jsonl`, and `.ndjson` files, then maps every import through the entity schema returned by:
+The desktop UI parses CSV, JSON, JSONL, and NDJSON and maps rows through `GET /datasets/schema/entity`. Each sparse entity needs a username, profile URL, or E.164 phone number. Optional explicit geospatial fields are:
 
-```http
-GET /datasets/schema/entity
-```
+- `latitude` and `longitude`, supplied together and within valid ranges;
+- `location_accuracy`, such as `exact`, `place`, or `approximate`; and
+- `location_source`, describing where the coordinate evidence came from.
 
-Each sparse entity row requires at least one searchable identifier: a username, profile URL, or E.164 phone number such as `+12025550101`. All other fields are optional. When both profile and phone fields exist, the service stores typed identifiers under one entity so a match for either identifier exposes the association. Invalid booleans, negative metrics, confidence outside `0..1`, and malformed identifiers are rejected per row without discarding valid rows.
+Free-text `location` remains descriptive text. The service never geocodes it. Existing databases receive these nullable columns through an additive startup migration.
 
-The legacy `profile` and `phone` schemas and import values remain accepted for API compatibility, but the desktop application uses only the entity pipeline. Existing SQLite databases are migrated in place when the service starts.
+Associated profile and phone identifiers remain attached to the entity. Phone information may be returned as marker metadata, but a phone row never creates or implies a device location.
 
-Each accepted entity includes:
+## Geospatial search
 
-- its dataset and source;
-- an observation timestamp;
-- optional confidence;
-- normalized searchable identifiers; and
-- the complete original source row for audit.
+```http
+GET /datasets/search/geo?latitude=51.5074&longitude=-0.1278&radius_miles=5&query=demo&limit=100
+```
 
-## Storage and limits
+The route returns a GeoJSON `FeatureCollection` containing radius-filtered entity and standalone profile records with source, accuracy, confidence, timestamp, distance, provenance, and associated identifiers.
 
-Compose stores `/data/datasets.db` in the named `dataset_data` volume. Deleting a dataset cascades to its entities and typed identifiers. Each import accepts 1–10,000 JSON object rows; the desktop UI additionally enforces a 10 MB file limit.
+## Storage and limits
 
-The API is intentionally local and currently has no authentication layer. Do not expose the nginx proxy to an untrusted network without adding access control.
+Compose stores `/data/datasets.db` in `dataset_data`. Deleting a dataset cascades to its entities and typed identifiers. Imports accept 1 to 10,000 JSON object rows; the desktop UI also limits files to 10 MB. The API is local and has no authentication layer, so do not expose nginx to an untrusted network without access control.
 
 ## Sample data
 
-The platform repository includes a combined fictional entity dataset plus separate fixtures for exercising compatibility and source-specific fields:
-
-- [`examples/datasets/sample_entities.csv`](../examples/datasets/sample_entities.csv)
-- [`examples/datasets/sample_profiles.csv`](../examples/datasets/sample_profiles.csv)
-- [`examples/datasets/sample_phone_records.jsonl`](../examples/datasets/sample_phone_records.jsonl)
+- [`geospatial_entities.csv`](../examples/datasets/geospatial_entities.csv): synthetic London, Arlington, and Portland entities with explicit coordinates and reserved fictional phone numbers.
+- [`sample_entities.csv`](../examples/datasets/sample_entities.csv): combined identities.
+- [`sample_profiles.csv`](../examples/datasets/sample_profiles.csv): profile compatibility import.
+- [`sample_phone_records.jsonl`](../examples/datasets/sample_phone_records.jsonl): phone compatibility import.
 
-Import any of them from the same desktop Datasets workflow; canonical headers are auto-mapped. The combined file demonstrates a username and phone on one entity, a profile-only entity, and a phone-only entity. Search for `demo_ada_1843` or `+12025550101` to see the association in both directions. The phone values use the reserved North American `555-01xx` fictional range and do not identify real subscribers.
+```bash
+python3 scripts/import_geospatial_demo.py
+```
 
 ## Development
 

dataset_service/models.py

@@ -15,7 +15,7 @@ class DatasetImportRequest(BaseModel):
     record_type: RecordType = RecordType.entity
     filename: str | None = Field(default=None, max_length=255)
     mapping: dict[str, str] = Field(default_factory=dict)
-    rows: list[dict[str, Any]] = Field(min_items=1, max_items=10_000)
+    rows: list[dict[str, Any]] = Field(min_length=1, max_length=10_000)
 
 
 class RejectedRow(BaseModel):