Add facilities proximity pipeline#1095
Open
amit-spatial wants to merge 51 commits into
Open
Conversation
amit-spatial
requested review from
amanodt and
shiv1122prakash
and removed request for
amanodt and
shiv1122prakash
amit-spatial
force-pushed
the
feat/facilities_v3
branch
3 times, most recently
from
7c1bb84 to
5fa0920
Compare
Adds the reviewable package overview and Mermaid flow for the shared local-compute pipeline core. This documents the non-pyogrio runtime direction before dataset-specific v3 pipelines build on it.
Introduces reusable GeoPackage inspection, index creation, filtered table reads, and direct geometry decoding so local pipelines can read only the requested scope through SQLite indexes.
Adds a fast selector for cs_admin_standard.gpkg with lowercase admin lookup indexes and state, district, tehsil, and village scopes for all keyed local pipelines.
Defines the shared request contract plus column, value-domain, and numeric-range validation helpers used by dataset pipelines and API wrappers.
Provides repeatable CSV-to-SQLite materialization with source freshness metadata and indexed key lookups for wide village-level resources such as Antyodaya and livestock.
Adds bundle writers for CSV, GeoPackage, EDA, metadata, and README outputs so pipeline results are consistent across API, batch, GeoServer, and Excel handoff use cases.
Adds JSON, YAML, CSV, and simple text request parsing so local pipelines can run many admin scopes without bespoke batch input formats.
Wraps the existing GeoServer publishing path and standardizes WFS/WMS link generation for local pipeline GeoPackage outputs.
Exports the shared admin source and standard request loader so dataset-specific pipelines can import the core API from one stable package path.
Normalize non-scalar GeoDataFrame fields before Fiona writes, preserve source FID values without colliding with GeoPackage feature ids, and create each named layer in replace mode so multi-layer outputs work reliably in local GDAL/Fiona environments. This keeps the shared local pipeline writer suitable for facilities, Antyodaya, livestock, and future local geospatial exports.
Adds reusable output-mode parsing, compact admin presentation columns with readable title-cased names, and selective CSV-to-SQLite sidecar materialization. These utilities let runtime pipelines return focused Excel/report outputs by default, expose all diagnostic artifacts on request, and avoid carrying unnecessary source CSV columns into indexed sidecars.
Adds pc11_state_id, pc11_district_id, and pc11_subdistrict_id to the default admin selection so focused report outputs can expose state_id, district_id, and tehsil_id without requiring each runtime pipeline to request extra columns manually.
Adds shared manifest helpers for local pipeline outputs: stable request hashing, fast input file signatures with optional .sha1 sidecar support, and cached result validation that checks request key, input signatures, and required output files before returning precomputed artifacts. This gives pipelines a defensible use_pregenerated path without hashing multi-GB assets on every request.
Keep Core Stack internal admin identifiers out of presentation outputs, add stable scope-aware output naming, make EDA robust to duplicate labels, and annotate cached responses with lookup timing while preserving original generation timing.
Provide a shared presentation helper that keeps only standard admin identifiers and configured dataset columns in user-facing outputs, while excluding internal Core Stack admin identifiers used by the base admin asset and pipeline joins.
Treat raw admin asset fields such as fid, pc11_* ids, TEHSIL, and NAME as internal source columns after they have been normalized into index, state_id, district_id, tehsil_id, and presentation names for pipeline outputs.
Remove cs_feature_id, cs_admin_uid, and core_admin_uid from the default admin selection used by local pipelines. Village scope now resolves only through village_id and pc11_village_id, while the cs_* names remain only in the exclusion list to prevent legacy frames from leaking them into outputs.
Extend the shared admin output helper so temporary pipeline columns such as _admin_key cannot appear in CSV, GeoPackage, EDA, or Excel-facing output frames.
Adds a package-based facilities runtime that reads cs_admin_standard.gpkg and cs_pan_india_facilities.gpkg directly, clips inventory facilities to the requested admin scope, computes nearest L3 services with KD-tree search, derives L2 village-service summaries, and writes GPKG, CSV, README, EDA, STAC, and optional GeoServer outputs. The pipeline keeps the legacy Celery task import surface inside the package while removing the older single-file facilities_proximity module and avoiding pyogrio in computing/misc.
Updates the existing facilities endpoint import to use computing.misc.facilities after replacing the old facilities_proximity.py module. This keeps the API and Celery task interface stable while the implementation moves to the indexed local GeoPackage pipeline.
Tracks the facilities L3/L2 classification and rollup decisions in package config, uses those decisions for min/max/direct L2 access summaries, adds focused and Excel-ready village service CSVs with compact admin columns and row-level status markers, supports structured API request bodies with output modes, and avoids a duplicate spatial join during nearest-service computation.
Move Excel and L2 distance output contracts into facilities_pipeline.yaml, switch runtime joins from cs_feature_id to fid-derived admin keys, keep focused/Excel/GPKG outputs on the standard admin presentation columns, and always emit a GeoServer links CSV even when publishing is disabled or fails.
Replace the rollup-specific column names (baseline_bundle_distance, nearest_gateway_distance, direct_access_distance) with a uniform <category>_cat_distance_km scheme across the L2 contract and the Excel-ready column order. Also fixes a double-underscore typo in cooperative_societies_cat_distance_km that produced an empty column in the Excel-ready CSV. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
During integration testing the pipeline YAML pins geoserver_workspace to test_workspace so runs cannot write into the production facilities_proximity workspace. Workspace resolution now falls back to FACILITIES_GEOSERVER_WORKSPACE from utilities/constants.py when the key is removed from the YAML at deploy time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Use the shared local pipeline output flags for the facilities runtime, write the default CSV/GPKG/README/metadata/STAC/GeoServer bundle, and keep UAT outputs under the test output root/workspace.
Allow structured request bodies and explicit output flags for the facilities local pipeline while preserving legacy flat state/district/block requests. This removes mode defaults from the API layer and lets the pipeline YAML resolve defaults.
Point the facilities local pipeline UAT config at the existing GeoServer test workspace testworkspace, which has a matching namespace and supports verified GeoPackage publication.
amit-spatial
force-pushed
the
feat/facilities_v3
branch
from
f97aa4e to
cc5f6d7
Compare
…iptions
- outputs.py: add friendly_datatype, column_dictionary, and frame_profile so
every pipeline can emit a per-output {column, description, datatype} list
alongside the EDA summary; drop the standalone write_eda artifact.
- schema.py: add STATUS_* constants and status_column_config so pipelines can
place a per-village data-availability column right after the admin columns
and configure which artifacts keep it; drop the legacy output-key aliases.
- admin.py: add ADMIN_COLUMN_DESCRIPTIONS for the standard admin columns.
- Track a local pipeline documentation pack under data/docs/local_pipeline
(architecture, data structures/schema, troubleshooting) by switching the
gitignore rule to data/* with !data/docs/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Derive the report CSV columns from facility_classifications.yaml: for each L2 group its category distance column, then the nearest-facility detail and distance pair for each member L3 class. Names are overridable through output_contract.l2_distance_columns (market/cooperative/dairy livestock). Machine columns (l2_*, l3_*) stay in the GeoPackage only. - Enrich facility_classifications.yaml from facilities_column_metadata.yaml: per-class descriptions, source datasets, cleaned files, id columns, and the output column description templates used for the runtime column dictionary. - Add the configurable facilities_status column right after the admin columns. - Emit the column dictionary and EDA per output in the run metadata, and the same column reference table in the runtime README. - Write the GeoPackage data table under the scoped layer name so GeoServer feature types do not collide across tehsils. - Rewrite the package README as a dataset knowledge article. - Require a structured 'scope' body on the API, drop _request_from_legacy_args and the legacy task kwargs, and delete the facilities_proximity shim. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The local pipeline documentation pack is for local consumption only. Restore the blanket data/ ignore rule and untrack data/docs; the files remain on disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add api_request_payload, one shared normalizer for both supported request shapes: the simple state/district/block body used by the other Core Stack layer APIs (implying a tehsil scope, with publish flags read from the top level), and the structured scope/outputs/publish body. Bodies naming no resolvable geography raise ValueError so views can answer 400. This keeps the API views to a single normalization call instead of per-view branching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route both request shapes through api_request_payload so the view stays a single normalization call, matching the structure of the neighbouring layer APIs. Bodies naming no geography answer 400 instead of 500. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
| from computing.mws.mws_connectivity import generate_mws_connectivity_data | ||
| from computing.misc.ndvi_time_series import ndvi_timeseries | ||
| from computing.zoi_layers.zoi import generate_zoi | ||
| from computing.mws.mws_centroid import generate_mws_centroid_data |
Collaborator
There was a problem hiding this comment.
Import tje facilities
shiv1122prakash
previously approved these changes
Jul 10, 2026
amit-spatial
dismissed
shiv1122prakash’s stale review
The merge-base changed after approval.
shiv1122prakash
previously approved these changes
Jul 10, 2026
amit-spatial
dismissed
shiv1122prakash’s stale review
The merge-base changed after approval.
Add register_layer, a scope-aware wrapper around the repo's standard save_layer_info_to_db/update_layer_sync_status path, so local pipeline layers appear in the Layer table alongside every GEE-backed layer. It creates the Dataset row if missing, records the GeoServer WFS URL as the asset id, marks is_gee_asset false and is_sync_to_geoserver true, and never raises: a database problem is reported in the run result rather than losing computed outputs. The Layer model is keyed on state/district/block, so only tehsil-scoped runs register; district, state, and village scopes report a skip reason. Default publish.register_layers to true, matching the other pipelines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The l2_*/l3_* columns come from pivot tables, which sort their columns alphabetically, so the GeoPackage and the GeoServer feature type built from it did not follow the classification order even though the report CSV did. Order them by L2 group and then by member L3 class, matching the CSV, and append any remaining columns so nothing is dropped. Also register the published layer in the Layer database after a successful GeoServer publish, through the shared register_layer helper. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Starts directly with Facility Point Files GPKG and uses Admin-Boundary GPKG with its embedded rtree to directly implement proximity finding algorithm within live pipeline, and then these distance properties, and nearest points are collected, along with tehsil points, and standard state-district-block request will simply provide two outputs, a collection of points within the geometry, and distances, with facilities specified. Our config-driven facilities local pipeline produces focused Excel-ready outputs, GPKG layers, basic data quality analysis, GeoServer link outputs, README artifacts in local
data/directory.