Dev#1073
Open
pawangramvaani wants to merge 78 commits into
Open
Conversation
Provides repeatable CSV-to-SQLite materialization with source freshness metadata and indexed key lookups for wide village-level resources such as Antyodaya and livestock.
Adds bundle writers for CSV, GeoPackage, EDA, metadata, and README outputs so pipeline results are consistent across API, batch, GeoServer, and Excel handoff use cases.
Adds JSON, YAML, CSV, and simple text request parsing so local pipelines can run many admin scopes without bespoke batch input formats.
Wraps the existing GeoServer publishing path and standardizes WFS/WMS link generation for local pipeline GeoPackage outputs.
Exports the shared admin source and standard request loader so dataset-specific pipelines can import the core API from one stable package path.
Normalize non-scalar GeoDataFrame fields before Fiona writes, preserve source FID values without colliding with GeoPackage feature ids, and create each named layer in replace mode so multi-layer outputs work reliably in local GDAL/Fiona environments. This keeps the shared local pipeline writer suitable for facilities, Antyodaya, livestock, and future local geospatial exports.
Adds reusable output-mode parsing, compact admin presentation columns with readable title-cased names, and selective CSV-to-SQLite sidecar materialization. These utilities let runtime pipelines return focused Excel/report outputs by default, expose all diagnostic artifacts on request, and avoid carrying unnecessary source CSV columns into indexed sidecars.
Adds pc11_state_id, pc11_district_id, and pc11_subdistrict_id to the default admin selection so focused report outputs can expose state_id, district_id, and tehsil_id without requiring each runtime pipeline to request extra columns manually.
Adds shared manifest helpers for local pipeline outputs: stable request hashing, fast input file signatures with optional .sha1 sidecar support, and cached result validation that checks request key, input signatures, and required output files before returning precomputed artifacts. This gives pipelines a defensible use_pregenerated path without hashing multi-GB assets on every request.
Keep Core Stack internal admin identifiers out of presentation outputs, add stable scope-aware output naming, make EDA robust to duplicate labels, and annotate cached responses with lookup timing while preserving original generation timing.
Provide a shared presentation helper that keeps only standard admin identifiers and configured dataset columns in user-facing outputs, while excluding internal Core Stack admin identifiers used by the base admin asset and pipeline joins.
Treat raw admin asset fields such as fid, pc11_* ids, TEHSIL, and NAME as internal source columns after they have been normalized into index, state_id, district_id, tehsil_id, and presentation names for pipeline outputs.
Remove cs_feature_id, cs_admin_uid, and core_admin_uid from the default admin selection used by local pipelines. Village scope now resolves only through village_id and pc11_village_id, while the cs_* names remain only in the exclusion list to prevent legacy frames from leaking them into outputs.
Extend the shared admin output helper so temporary pipeline columns such as _admin_key cannot appear in CSV, GeoPackage, EDA, or Excel-facing output frames.
Deploy-time GeoServer workspaces for the Antyodaya and livestock local pipelines live in utilities/constants.py, next to the existing facilities constant. Pipeline YAMLs can then drop their geoserver_workspace key at deploy time and fall back to these values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace mode-specific local pipeline output options with explicit artifact flags resolved from YAML defaults and request overrides. This keeps the shared core contract smaller for UAT while preserving legacy request aliases for focused/excel/metadata callers.
Replace the shapefile-oriented GeoServer helper with a local pipeline publisher that validates workspace namespaces, uploads GeoPackages directly, publishes the native GPKG table under the unique pipeline layer name, and verifies WFS row counts and properties before reporting success. Add a smoke script so GeoServer upload regressions are caught before pipeline UAT.
…iptions
- outputs.py: add friendly_datatype, column_dictionary, and frame_profile so
every pipeline can emit a per-output {column, description, datatype} list
alongside the EDA summary; drop the standalone write_eda artifact.
- schema.py: add STATUS_* constants and status_column_config so pipelines can
place a per-village data-availability column right after the admin columns
and configure which artifacts keep it; drop the legacy output-key aliases.
- admin.py: add ADMIN_COLUMN_DESCRIPTIONS for the standard admin columns.
- Track a local pipeline documentation pack under data/docs/local_pipeline
(architecture, data structures/schema, troubleshooting) by switching the
gitignore rule to data/* with !data/docs/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The local pipeline documentation pack is for local consumption only. Restore the blanket data/ ignore rule and untrack data/docs; the files remain on disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add api_request_payload, one shared normalizer for both supported request shapes: the simple state/district/block body used by the other Core Stack layer APIs (implying a tehsil scope, with publish flags read from the top level), and the structured scope/outputs/publish body. Bodies naming no resolvable geography raise ValueError so views can answer 400. This keeps the API views to a single normalization call instead of per-view branching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…re_v3 Add shared local GPKG pipeline utilities
Feature/et downscaling
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.