Add Juniper ISSU and NSSU support

[mattmiller87]

New Pull Request

This PR adds Juniper install options for in-service software upgrade and nonstop software upgrade for SRX and EX devices.

This is in testing and may need some revisions.

• EX Virtual-Chassis
• SRX Chassis Cluster
• EX single device

SDK docs - https://junos-pyez.readthedocs.io/en/2.8.2/jnpr.junos.utils.html#jnpr.junos.utils.sw.SW.install

rpc.cli() - used to run commands not available in the sdk.

Change Notes

Add issu and nssu choices on install_os.
Add install_os_multi which performs an os change on multiple devices without using nssu.
Add request_system_snapshot to enforce best practices with dual partitions.

Justification

[Copilot]

Pull request overview

This PR extends the Junos device driver to support Juniper ISSU/NSSU upgrade flows and adds a system snapshot helper intended to enforce dual-partition best practices during upgrades.

Changes:

• Add issu / nssu options to JunosDevice.install_os() and adjust upgrade/reboot handling logic.
• Add JunosDevice.install_os_multi() to perform multi-member (virtual-chassis) upgrades without NSSU.
• Add request_system_snapshot() plus polling-based verification for snapshot completion.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 7 comments.

File	Description
pyntc/devices/jnpr_device.py	Adds ISSU/NSSU install options, a multi-device install flow, and snapshot request/verification helpers.
changes/mm.added	Adds a changelog entry for the new Juniper install options.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 5 comments.

[jtdub]

Lab test findings — EX3300 2-member Virtual Chassis (15.1R7-S2 ↔ 12.3R12-S10)

Ran the install_os flow end-to-end against a 2-member EX3300 VC (ex3300-48t-bf master + ex3300-48t). The multi-member detection, all-members install/reboot, reboot-wait, and post-install version verification all worked. Three issues surfaced on real hardware that should be addressed before merge (not fixing them in this comment thread — logging them for discussion):

1. A failed member install is masked by the "A reboot is required" heuristic, and the chassis reboots half-installed

install_os raises OSInstallError when install_ok is falsy unless the install message contains "A reboot is required". On a VC that message is aggregated across members, so one member's success text suppresses another member's failure:

• fpc1's install failed (config validation rejected chassis redundancy graceful-switchover on the target release → pkgadd package-result: 1, "There is already a reboot pending for software rollback").
• fpc0's install succeeded (package-result: 0) and its output contained WARNING: A reboot is required to install the software.
• sw.install() returned (False, msg) — correctly signalling failure — but the aggregate msg matched the heuristic, so pyntc logged install_ok result: False at INFO and issued request system reboot all-members anyway.

Result: mixed-version VC, fpc1 came back Inactive (Linecard role), and manual recovery was required (request system software add ... member 1 no-validate reboot). Recommendation: raise OSInstallError whenever install_ok is falsy and include install_msg in the log — don't infer success from message text.

Related inversion in the same block: with reboot=False, a successful staged install raises OSInstallError, because normal Junos success output always contains "A reboot is required" — the exact opposite of the intended "stage now, reboot later" contract.

2. Post-upgrade snapshot verification timeout is undersized for EX hardware (and hides its polling errors)

request system snapshot slice alternate all-members on this VC took ~15 minutes (members snapshot sequentially: fpc0 finished 16:33:18, fpc1 16:40:20 — 30 seconds before the 900 s timeout fired at 16:40:50). The snapshots actually completed; pyntc reported failure.

Compounding factors in _wait_for_system_snapshot:

• show system snapshot media internal blocks/fails while a snapshot is in progress, so polling can only succeed after full completion — the poll interval provides no early signal.
• Poll exceptions are swallowed at DEBUG (except Exception: log.debug(...)), so a normal run gives no clue whether polls are erroring or just not matching.
• On timeout it raises a bare built-in TimeoutError, while the request_system_snapshot docstring claims OSInstallError — callers writing except OSInstallError (or except NTCError) won't catch it.

Recommendations: raise the default timeout substantially (≥ 45 min for EX-class hardware) and/or expose it as a parameter; use an NTCError subclass; log poll failures visibly. Worth discussing: whether a snapshot timeout should fail install_os at all — at that point the OS is installed, rebooted, and version-verified, so raising makes callers believe a successful upgrade failed.

3. RPC timeout resets to the PyEZ default after the post-reboot reconnect

JunosDevice.__init__ sets native.timeout = 120, but the snapshot RPC issued right after _wait_for_device_reboot reconnected timed out at exactly 30 s (PyEZ's default) — the timeout does not survive the close/reopen cycle in the reboot wait. open() should re-apply DEFAULT_TIMEOUT.

---

Happy to share full console/log captures from the lab session for any of these.

[jtdub]

Follow-up lab finding — remote_file_copy staging-filesystem blind spot

Continued EX3300 VC testing surfaced a fourth issue, this one in the transfer path:

Junos file copy (the file-copy RPC behind remote_file_copy) stages the download through the login user's home directory (/var/home/<user>, on the /var filesystem) before copying to the destination. Observed on the console:

ntc@device> file copy "http://.../jinstall-ex-3300-15.1R7-S2-domestic-signed.tgz" /var/tmp/
/var/home/ntc/...transferring.file.........   <- staging location

Consequences observed on real hardware (EX3300, /var = 123 MiB total, /var/tmp = separate 225 MB filesystem):

• A 96 MB image pulls successfully; a 122 MiB image can never be pulled — the fetch dies when /var fills (~40 s in, protocol-independent: identical failure on HTTP and FTP), reported only as error: File-fetch failed.
• The pre-transfer free-space check added in this PR validates the destination filesystem (/var/tmp), which passes — so pyntc proceeds into a transfer that cannot succeed. The real constraint is the staging filesystem.
• Bonus operational gotcha: if the destination file already exists owned by another user (e.g., root left it there from a console session), the post-fetch cp fails with put-file failed / Permission denied after a fully successful download. remote_file_copy's checksum early-return avoids this when the existing file matches, but a stale/partial root-owned file will fail the copy.

Suggestions (not implemented here):

1. Extend the pre-transfer check to also probe the staging filesystem — the machinery exists: _get_free_space(file_system="/var/home/<user>") resolves to /var via the longest-prefix mount match. Failing fast with NotEnoughFreeSpaceError naming the staging filesystem beats a 40-second opaque File-fetch failed.
2. Document the staging behavior and the resulting platform constraint: on small-/var platforms (EX3300-class), remote_file_copy is limited to images that fit the staging filesystem, regardless of destination free space.