Add Standard Core Stack GEE Ingestion Workflow#1086
Open
amit-spatial wants to merge 3 commits into
Open
Conversation
This script standardizes memory-bounded publication of large Core Stack GeoPackage and GeoJSON assets to Earth Engine through TSV, GCS, and manifest ingestion. It is kept separate from dataset-building code so upload/retry/verify workflows can be reviewed independently.
This YAML declares the local source files, Earth Engine asset ids, layer names, geometry policy, and column handling for the core CS assets. Tracking it makes GEE uploads reproducible without committing generated staging files.
This document records how to build, upload, monitor, verify, and publish the standard Core Stack GEE assets. It is included with the uploader so operators can rerun the flow without relying on local shell history.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Now, since we regularly need to upload these assets to GEE, I have also added the script which so brilliantly and easily, in a standard manner creates standard GEE assets using the standard approach used by gee awesome community's geemap approach. It simply now creates tsv (tab separated values), pushes to GCS temporarily, from where it creates a GEE task to ingest it to GEE as its asset. I haven't seen any issues with any newly GEE uploaded assets after using this approach, and its fast too, and can finish full task from local PC to GCS as fast as your intenet allows, and then finishes pushing a resource which would have been 10 GB as geojson locally (it works with both GPKG or geojson) to GEE (from GCS) in around half an hour.
utilities/scripts/gee/core_stack_gee_ingest.mddocuments the current GEE ingested assets.utilities/scripts/gee/core_stack_gee_assets.yamlkeeps track of currently used configs for current Facilities, Antyodaya, Livestocks GEE datasets, and new configs can be appended to it, and can be used for any new GEE asset uploads.