Make example scripts self-consistent in naming and distributed behavior#688
Draft
kmontemayor2-sc wants to merge 17 commits into
Draft
Make example scripts self-consistent in naming and distributed behavior#688kmontemayor2-sc wants to merge 17 commits into
kmontemayor2-sc wants to merge 17 commits into
Conversation
…_process in examples Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… in examples Standardizes field naming for model URIs across inference examples and KDD tutorial to match the training examples. Renames: - model_state_dict_uri → model_uri in 5 inference example files - saved_model_path/saved_model_uri → model_uri in 2 KDD tutorial files - model_state_dict_uri parameter → model_uri in api_test_inference.py Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipts Unify how the five inference examples resolve processes-per-machine: one config key (local_world_size), one default rule (GPU count if CUDA is available, else DEFAULT_CPU_BASED_LOCAL_WORLD_SIZE = 4), and one failure rule (raise ValueError on GPU oversubscription instead of only warning). The two heterogeneous examples also rename num_inference_processes_per_machine to local_world_size and gain the module constant they previously lacked. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unify local_world_size resolution in the five training example files:
prefer trainer_args, else default to GPU count if available, else fall
back to a per-file DEFAULT_CPU_BASED_LOCAL_WORLD_SIZE=1 constant. This
replaces divergent inline defaults ("1", "2", GPU-count/2) with a single
consistent policy.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… KDD tutorial Rename identifiers in both training and inference tutorial files: - process_number → local_rank - process_count → local_world_size - --process_count → --local_world_size Updates module docstrings, function signatures, log messages, argparse flags, and spawn block parameters. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ples Move device selection and torch.cuda.set_device before init_process_group (NCCL-recommended ordering), derive rank/world_size from torch.distributed.get_rank()/get_world_size() right after init instead of the manual machine_rank/local_world_size formula, and replace `machine_rank == 0 and local_rank == 0` lead-process guards with `rank == 0` across the six colocated link_prediction and node_classification example files. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Derive rank/world_size from torch.distributed after init_compute_process instead of the manual machine-formula, matching the sibling graph-store files. Drops the now-redundant machine_rank/machine_world_size fields. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add missing torch.distributed.destroy_process_group() to LP homogeneous and heterogeneous colocated inference, matching NC-HomI's cleanup tail. Add missing shutdown_compute_process() to GS-HomI's cleanup, matching GS-HetI. Move the BigQuery embedding load out of the spawned _inference_process into the orchestration function (after mp.spawn, guarded on the compute-cluster lead rank), also matching GS-HetI, and drop the now-unused gbml_config_pb_wrapper field from InferenceProcessArgs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add mp.set_start_method("spawn") to the top of _run_example_inference
in all five non-KDD inference examples to ensure consistent subprocess
initialization across different platforms, mirroring the pattern already
established in training examples.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…example GS hetero training uses query=dst (reverse of standard mode's query=src). This is deliberate: the storage-side DistNodeAnchorLinkSplitter is configured via splitter_kwargs.supervision_edge_types with the reversed edge, so with sampling direction "in" it keys the train/val/test splits by the taskMetadata dst node type. Added explanatory comments at both query=dst sites; no behavior change. Also fixed the spillter_kwargs -> splitter_kwargs docstring typo. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ated loader args Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
Author
|
/all_test |
Contributor
GiGL Automation@ 18:33:43UTC : 🔄 @ 18:42:42UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 18:33:43UTC : 🔄 @ 19:42:58UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 18:33:46UTC : 🔄 @ 20:25:50UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 18:33:47UTC : 🔄 @ 20:09:58UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 18:33:47UTC : 🔄 @ 18:41:46UTC : ✅ Workflow completed successfully. |
Contributor
GiGL Automation@ 18:33:48UTC : 🔄 @ 18:35:56UTC : ✅ Workflow completed successfully. |
Delete the redundant "If GPUs are available..." comments (the elif, assignment, and log line already say it), explain WHY model saving is gated to rank 0 instead of restating the guard, and fix the dblp.ipynb comment that still described building dicts the cell no longer builds. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
A consistency pass over the 11 example train/infer scripts (plus
tests/api_test), making every script say and do the same thing the same way, followed by adopting the newGbmlConfigPbWrappertyped feature maps (#643). Changes fall into six buckets:1. Renames — one canonical name per concept
local_world_sizenum_inference_processes_per_machine, KDD--process_countlocal_rankprocess_number(KDD)sampling_workers_per_processsampling_workers_per_inference_processmodel_urimodel_state_dict_uri,saved_model_uri/saved_model_pathmachine_{...}_local_process_{local_rank}..._local_process_number_{...}Also fixed the
spillter_kwargs→splitter_kwargsdocstring typo. The KDD_2025 tutorial got renames only — its deliberately simple logic is untouched.2.
local_world_sizeresolution — one rule everywheretorch.cuda.device_count()when CUDA → else a per-fileDEFAULT_CPU_BASED_LOCAL_WORLD_SIZEconstant (4 for inference, 1 for training).ValueErroron GPU oversubscription — the homogeneous inference scripts previously only warned.local_world_sizedirectly fromcluster_info.num_processes_per_compute(the launcher-setCOMPUTE_CLUSTER_LOCAL_WORLD_SIZE) instead of a config arg — the old CPU default of 2 silently disagreed with the launcher's 1, desyncing spawned processes from the computeprocess group.
3. Distributed init — one canonical sequence
Every colocated spawned process now does: get device →
cuda.set_device→init_process_group(rank/world_size computed inline via the machine formula, the only place it appears) →rank = torch.distributed.get_rank()/world_size = get_world_size()locals used everywhere downstream. Lead-processguards are uniformly
if rank == 0:. Graph-store heterogeneous inference now also reads rank from the process group afterinit_compute_processinstead of a manual formula (its redundantmachine_rank/machine_world_sizefields are gone).4. Cleanup — one canonical teardown
barrier → loader.shutdown() → gc.collect() → destroy_process_group()(LP files were missing the destroy).loader.shutdown() → shutdown_compute_process() → gc.collect()(GS homogeneous inference was missingshutdown_compute_process).cluster_info.compute_node_rank == 0, matching its hetero sibling.mp.set_start_method("spawn")like the training scripts always have.5. Dead/deprecated API usage removed
Dropped the deprecated
local_process_rank/local_process_world_sizeloader kwargs from every example call site — they are silently ignored unless a (also-deprecated)contextis passed, so they only taught users a dead parameter.6. Adopt
GbmlConfigPbWrappertyped feature maps (#643)All hand-built
{node,edge}_type_to_feature_dimdict comprehensions and condensed-key lookups (8 scripts, 2 api_test files,cora.ipynb,dblp.ipynb) replaced withgbml_config_pb_wrapper.node_type_to_feature_dim_map/.edge_type_to_feature_dim_map.Investigated, not a bug
graph_store/heterogeneous_training.pyderivesquery_node_type = supervision_edge_type.dst_node_type— reversed vs. standard mode. This is deliberate compensation for the reversed storage-sidesplitter_kwargs.supervision_edge_typesin the GS DBLP config (traced throughDistNodeAnchorLinkSplitter's direction handling), and is now documented with comments instead of "fixed".Verification
make type_checkandmake format_pyclean; repo-wide greps show zero stale occurrences of any old name.make unit_test_py PY_TEST_FILES="config_validator_test.py"(imports the example configs): 4/4 pass.Known follow-ups (out of scope)
mp.set_start_methodagainst pre-initialized contexts (equally applies to the pre-existing training pattern).local_process_rank/local_process_world_sizeparams.dblp.ipynbcell referencesnode_feature_dims[node_type]wherenode_typemay be undefined.