Feat/livestocks#1097
Open
amit-spatial wants to merge 50 commits into
Open
Conversation
amit-spatial
force-pushed
the
feat/livestocks
branch
3 times, most recently
from
6597eab to
b814e3f
Compare
Adds the reviewable package overview and Mermaid flow for the shared local-compute pipeline core. This documents the non-pyogrio runtime direction before dataset-specific v3 pipelines build on it.
Introduces reusable GeoPackage inspection, index creation, filtered table reads, and direct geometry decoding so local pipelines can read only the requested scope through SQLite indexes.
Adds a fast selector for cs_admin_standard.gpkg with lowercase admin lookup indexes and state, district, tehsil, and village scopes for all keyed local pipelines.
Defines the shared request contract plus column, value-domain, and numeric-range validation helpers used by dataset pipelines and API wrappers.
Provides repeatable CSV-to-SQLite materialization with source freshness metadata and indexed key lookups for wide village-level resources such as Antyodaya and livestock.
Adds bundle writers for CSV, GeoPackage, EDA, metadata, and README outputs so pipeline results are consistent across API, batch, GeoServer, and Excel handoff use cases.
Adds JSON, YAML, CSV, and simple text request parsing so local pipelines can run many admin scopes without bespoke batch input formats.
Wraps the existing GeoServer publishing path and standardizes WFS/WMS link generation for local pipeline GeoPackage outputs.
Exports the shared admin source and standard request loader so dataset-specific pipelines can import the core API from one stable package path.
Normalize non-scalar GeoDataFrame fields before Fiona writes, preserve source FID values without colliding with GeoPackage feature ids, and create each named layer in replace mode so multi-layer outputs work reliably in local GDAL/Fiona environments. This keeps the shared local pipeline writer suitable for facilities, Antyodaya, livestock, and future local geospatial exports.
Adds reusable output-mode parsing, compact admin presentation columns with readable title-cased names, and selective CSV-to-SQLite sidecar materialization. These utilities let runtime pipelines return focused Excel/report outputs by default, expose all diagnostic artifacts on request, and avoid carrying unnecessary source CSV columns into indexed sidecars.
Adds pc11_state_id, pc11_district_id, and pc11_subdistrict_id to the default admin selection so focused report outputs can expose state_id, district_id, and tehsil_id without requiring each runtime pipeline to request extra columns manually.
Adds shared manifest helpers for local pipeline outputs: stable request hashing, fast input file signatures with optional .sha1 sidecar support, and cached result validation that checks request key, input signatures, and required output files before returning precomputed artifacts. This gives pipelines a defensible use_pregenerated path without hashing multi-GB assets on every request.
Keep Core Stack internal admin identifiers out of presentation outputs, add stable scope-aware output naming, make EDA robust to duplicate labels, and annotate cached responses with lookup timing while preserving original generation timing.
Provide a shared presentation helper that keeps only standard admin identifiers and configured dataset columns in user-facing outputs, while excluding internal Core Stack admin identifiers used by the base admin asset and pipeline joins.
Treat raw admin asset fields such as fid, pc11_* ids, TEHSIL, and NAME as internal source columns after they have been normalized into index, state_id, district_id, tehsil_id, and presentation names for pipeline outputs.
Remove cs_feature_id, cs_admin_uid, and core_admin_uid from the default admin selection used by local pipelines. Village scope now resolves only through village_id and pc11_village_id, while the cs_* names remain only in the exclusion list to prevent legacy frames from leaking them into outputs.
Extend the shared admin output helper so temporary pipeline columns such as _admin_key cannot appear in CSV, GeoPackage, EDA, or Excel-facing output frames.
Deploy-time GeoServer workspaces for the Antyodaya and livestock local pipelines live in utilities/constants.py, next to the existing facilities constant. Pipeline YAMLs can then drop their geoserver_workspace key at deploy time and fall back to these values. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Replace mode-specific local pipeline output options with explicit artifact flags resolved from YAML defaults and request overrides. This keeps the shared core contract smaller for UAT while preserving legacy request aliases for focused/excel/metadata callers.
amit-spatial
force-pushed
the
feat/livestocks
branch
from
b814e3f to
4d64058
Compare
Replace the shapefile-oriented GeoServer helper with a local pipeline publisher that validates workspace namespaces, uploads GeoPackages directly, publishes the native GPKG table under the unique pipeline layer name, and verifies WFS row counts and properties before reporting success. Add a smoke script so GeoServer upload regressions are caught before pipeline UAT.
Adds the source paths, join key, livestock count metrics, output names, validation assumptions, and Mermaid documentation for the CSV plus standard-admin livestock runtime pipeline.
Adds the package entrypoints and runtime implementation that reads cs_admin_standard by scope, fetches livestock rows through the SQLite CSV sidecar, validates non-negative count metrics, writes standard output bundles, and preserves generate_livestocks_layer_task for existing API imports.
Adds a tracked livestock hierarchy schema, narrows the CSV SQLite sidecar to village keys plus male/female count columns, derives species and large/small animal totals at runtime, writes focused and Excel-ready report CSVs with row-level status markers, and lets the livestock API endpoint pass structured output-mode requests to the local pipeline.
During integration testing the pipeline YAML pins geoserver_workspace to test_workspace so runs cannot write into the production livestocks workspace. Workspace resolution now falls back to LIVESTOCK_GEOSERVER_WORKSPACE from utilities/constants.py when the key is removed from the YAML at deploy time. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Use the shared local pipeline output flags for livestock, write EDA into run metadata, keep UAT output paths in config, and expand the README with source context for the 20th Livestock Census.
Allow structured request bodies and explicit output flags for the livestock local pipeline while preserving legacy flat state/district/block requests. This removes mode defaults from the API layer and lets the pipeline YAML resolve defaults.
Point the livestock local pipeline UAT config at the existing GeoServer test workspace testworkspace, which has a matching namespace and supports verified GeoPackage publication.
amit-spatial
force-pushed
the
feat/livestocks
branch
from
4d64058 to
521ec7b
Compare
…iptions
- outputs.py: add friendly_datatype, column_dictionary, and frame_profile so
every pipeline can emit a per-output {column, description, datatype} list
alongside the EDA summary; drop the standalone write_eda artifact.
- schema.py: add STATUS_* constants and status_column_config so pipelines can
place a per-village data-availability column right after the admin columns
and configure which artifacts keep it; drop the legacy output-key aliases.
- admin.py: add ADMIN_COLUMN_DESCRIPTIONS for the standard admin columns.
- Track a local pipeline documentation pack under data/docs/local_pipeline
(architecture, data structures/schema, troubleshooting) by switching the
gitignore rule to data/* with !data/docs/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Report CSV keeps the finalized column order: admin columns, livestock_status, all_livestock_total, large_animals_total, cattle_total, buffalo_total, small_animals_total, sheep_total, goat_total, pig_total. - GeoPackage and GeoServer keep the female and male counts alongside each animal total in the same order (gpkg_columns in livestocks_schema.yaml). - Add the configurable livestock_status column right after the admin columns. - Describe output columns from the runtime schema and emit the column dictionary and EDA per output in the run metadata and README. - Write the GeoPackage data table under the scoped layer name so GeoServer feature types do not collide across tehsils. - Rewrite the package README as a dataset knowledge article. - Require a structured 'scope' body on the API and drop the legacy request helper and task kwargs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The local pipeline documentation pack is for local consumption only. Restore the blanket data/ ignore rule and untrack data/docs; the files remain on disk. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add api_request_payload, one shared normalizer for both supported request shapes: the simple state/district/block body used by the other Core Stack layer APIs (implying a tehsil scope, with publish flags read from the top level), and the structured scope/outputs/publish body. Bodies naming no resolvable geography raise ValueError so views can answer 400. This keeps the API views to a single normalization call instead of per-view branching. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Route both request shapes through api_request_payload so the view stays a single normalization call. Bodies naming no geography answer 400. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
shiv1122prakash
previously approved these changes
Jul 10, 2026
amit-spatial
dismissed
shiv1122prakash’s stale review
The merge-base changed after approval.
Add register_layer, a scope-aware wrapper around the repo's standard save_layer_info_to_db/update_layer_sync_status path, so local pipeline layers appear in the Layer table alongside every GEE-backed layer. It creates the Dataset row if missing, records the GeoServer WFS URL as the asset id, marks is_gee_asset false and is_sync_to_geoserver true, and never raises: a database problem is reported in the run result rather than losing computed outputs. The Layer model is keyed on state/district/block, so only tehsil-scoped runs register; district, state, and village scopes report a skip reason. Default publish.register_layers to true, matching the other pipelines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
After a successful GeoServer publish, register the layer through the shared register_layer helper (save_layer_info_to_db + update_layer_sync_status), with the dataset name configured in the pipeline yaml. Registration is on by default and never fails the run. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds Livestock local pipeline with schema-driven animal grouping, derived totals, requested Excel column order, GPKG output, cache tracking, validation, and GeoServer link artifacts.