Skip to content

Make example scripts self-consistent in naming and distributed behavior#688

Draft
kmontemayor2-sc wants to merge 17 commits into
mainfrom
kmonte/example-consolidation
Draft

Make example scripts self-consistent in naming and distributed behavior#688
kmontemayor2-sc wants to merge 17 commits into
mainfrom
kmonte/example-consolidation

Conversation

@kmontemayor2-sc

@kmontemayor2-sc kmontemayor2-sc commented Jul 7, 2026

Copy link
Copy Markdown
Collaborator

Summary

A consistency pass over the 11 example train/infer scripts (plus tests/api_test), making every script say and do the same thing the same way, followed by adopting the new GbmlConfigPbWrapper typed feature maps (#643). Changes fall into six buckets:

1. Renames — one canonical name per concept

Concept Canonical Replaced
Processes per machine (config key / field / CLI) local_world_size num_inference_processes_per_machine, KDD --process_count
Spawned process index local_rank process_number (KDD)
Sampling workers (config key / field) sampling_workers_per_process sampling_workers_per_inference_process
Model artifact field model_uri model_state_dict_uri, saved_model_uri / saved_model_path
Output shard filename machine_{...}_local_process_{local_rank} ..._local_process_number_{...}

Also fixed the spillter_kwargssplitter_kwargs docstring typo. The KDD_2025 tutorial got renames only — its deliberately simple logic is untouched.

2. local_world_size resolution — one rule everywhere

  • Colocated scripts share one identical block: explicit config value wins → torch.cuda.device_count() when CUDA → else a per-file DEFAULT_CPU_BASED_LOCAL_WORLD_SIZE constant (4 for inference, 1 for training).
  • Always raise ValueError on GPU oversubscription — the homogeneous inference scripts previously only warned.
  • Graph-store scripts now take local_world_size directly from cluster_info.num_processes_per_compute (the launcher-set COMPUTE_CLUSTER_LOCAL_WORLD_SIZE) instead of a config arg — the old CPU default of 2 silently disagreed with the launcher's 1, desyncing spawned processes from the compute
    process group.
  • ⚠️ Behavior note: colocated LP training previously defaulted to 1 process even on GPU machines; it now defaults to GPU count, so e2e loss curves / step counts will shift on multi-GPU machines.

3. Distributed init — one canonical sequence

Every colocated spawned process now does: get device → cuda.set_deviceinit_process_group (rank/world_size computed inline via the machine formula, the only place it appears) → rank = torch.distributed.get_rank() / world_size = get_world_size() locals used everywhere downstream. Lead-process
guards are uniformly if rank == 0:. Graph-store heterogeneous inference now also reads rank from the process group after init_compute_process instead of a manual formula (its redundant machine_rank/machine_world_size fields are gone).

4. Cleanup — one canonical teardown

  • Colocated inference: barrier → loader.shutdown() → gc.collect() → destroy_process_group() (LP files were missing the destroy).
  • Graph-store: loader.shutdown() → shutdown_compute_process() → gc.collect() (GS homogeneous inference was missing shutdown_compute_process).
  • GS homogeneous inference's BigQuery load moved out of the spawned process into orchestration, guarded by cluster_info.compute_node_rank == 0, matching its hetero sibling.
  • LP heterogeneous inference orchestration gained the missing temporary-gloo-group destroy.
  • All five non-KDD inference orchestrations now call mp.set_start_method("spawn") like the training scripts always have.

5. Dead/deprecated API usage removed

Dropped the deprecated local_process_rank / local_process_world_size loader kwargs from every example call site — they are silently ignored unless a (also-deprecated) context is passed, so they only taught users a dead parameter.

6. Adopt GbmlConfigPbWrapper typed feature maps (#643)

All hand-built {node,edge}_type_to_feature_dim dict comprehensions and condensed-key lookups (8 scripts, 2 api_test files, cora.ipynb, dblp.ipynb) replaced with gbml_config_pb_wrapper.node_type_to_feature_dim_map / .edge_type_to_feature_dim_map.

Investigated, not a bug

graph_store/heterogeneous_training.py derives query_node_type = supervision_edge_type.dst_node_type — reversed vs. standard mode. This is deliberate compensation for the reversed storage-side splitter_kwargs.supervision_edge_types in the GS DBLP config (traced through
DistNodeAnchorLinkSplitter's direction handling), and is now documented with comments instead of "fixed".

Verification

  • make type_check and make format_py clean; repo-wide greps show zero stale occurrences of any old name.
  • make unit_test_py PY_TEST_FILES="config_validator_test.py" (imports the example configs): 4/4 pass.
  • Dual final review — an adversarial whole-branch review (MERGE-READY) and a Codex review; every confirmed finding was fixed on-branch.

Known follow-ups (out of scope)

  • Optional hardening: guard mp.set_start_method against pre-initialized contexts (equally applies to the pre-existing training pattern).
  • Library-level: finish deprecating/removing the loader local_process_rank/local_process_world_size params.
  • Pre-existing on main: a dblp.ipynb cell references node_feature_dims[node_type] where node_type may be undefined.

kmontemayor and others added 16 commits July 7, 2026 16:17
…_process in examples

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… in examples

Standardizes field naming for model URIs across inference examples and KDD tutorial
to match the training examples. Renames:
- model_state_dict_uri → model_uri in 5 inference example files
- saved_model_path/saved_model_uri → model_uri in 2 KDD tutorial files
- model_state_dict_uri parameter → model_uri in api_test_inference.py

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipts

Unify how the five inference examples resolve processes-per-machine: one
config key (local_world_size), one default rule (GPU count if CUDA is
available, else DEFAULT_CPU_BASED_LOCAL_WORLD_SIZE = 4), and one failure
rule (raise ValueError on GPU oversubscription instead of only warning).
The two heterogeneous examples also rename
num_inference_processes_per_machine to local_world_size and gain the
module constant they previously lacked.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Unify local_world_size resolution in the five training example files:
prefer trainer_args, else default to GPU count if available, else fall
back to a per-file DEFAULT_CPU_BASED_LOCAL_WORLD_SIZE=1 constant. This
replaces divergent inline defaults ("1", "2", GPU-count/2) with a single
consistent policy.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… KDD tutorial

Rename identifiers in both training and inference tutorial files:
- process_number → local_rank
- process_count → local_world_size
- --process_count → --local_world_size

Updates module docstrings, function signatures, log messages, argparse flags,
and spawn block parameters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ples

Move device selection and torch.cuda.set_device before
init_process_group (NCCL-recommended ordering), derive rank/world_size
from torch.distributed.get_rank()/get_world_size() right after init
instead of the manual machine_rank/local_world_size formula, and
replace `machine_rank == 0 and local_rank == 0` lead-process guards
with `rank == 0` across the six colocated link_prediction and
node_classification example files.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Derive rank/world_size from torch.distributed after init_compute_process
instead of the manual machine-formula, matching the sibling graph-store
files. Drops the now-redundant machine_rank/machine_world_size fields.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add missing torch.distributed.destroy_process_group() to LP homogeneous
and heterogeneous colocated inference, matching NC-HomI's cleanup tail.
Add missing shutdown_compute_process() to GS-HomI's cleanup, matching
GS-HetI. Move the BigQuery embedding load out of the spawned
_inference_process into the orchestration function (after mp.spawn,
guarded on the compute-cluster lead rank), also matching GS-HetI, and
drop the now-unused gbml_config_pb_wrapper field from
InferenceProcessArgs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add mp.set_start_method("spawn") to the top of _run_example_inference
in all five non-KDD inference examples to ensure consistent subprocess
initialization across different platforms, mirroring the pattern already
established in training examples.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…example

GS hetero training uses query=dst (reverse of standard mode's query=src). This is
deliberate: the storage-side DistNodeAnchorLinkSplitter is configured via
splitter_kwargs.supervision_edge_types with the reversed edge, so with sampling
direction "in" it keys the train/val/test splits by the taskMetadata dst node type.
Added explanatory comments at both query=dst sites; no behavior change. Also fixed
the spillter_kwargs -> splitter_kwargs docstring typo.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ated loader args

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@kmontemayor2-sc

Copy link
Copy Markdown
Collaborator Author

/all_test

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:33:43UTC : 🔄 Scala Unit Test started.

@ 18:42:42UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:33:43UTC : 🔄 Python Unit Test started.

@ 19:42:58UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:33:46UTC : 🔄 Integration Test started.

@ 20:25:50UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:33:47UTC : 🔄 E2E Test started.

@ 20:09:58UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:33:47UTC : 🔄 Lint Test started.

@ 18:41:46UTC : ✅ Workflow completed successfully.

@github-actions

github-actions Bot commented Jul 7, 2026

Copy link
Copy Markdown
Contributor

GiGL Automation

@ 18:33:48UTC : 🔄 C++ Unit Test started.

@ 18:35:56UTC : ✅ Workflow completed successfully.

@kmontemayor2-sc kmontemayor2-sc changed the title Kmonte/example consolidation Make example scripts self-consistent in naming and distributed behavior Jul 7, 2026
Delete the redundant "If GPUs are available..." comments (the elif,
assignment, and log line already say it), explain WHY model saving is
gated to rank 0 instead of restating the guard, and fix the dblp.ipynb
comment that still described building dicts the cell no longer builds.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant