24.1. PRRTE v5.x series
This file contains all the NEWS updates for the PRRTE v5.x series, in reverse chronological order.
24.1.1. 5.0.0 – TBD
Important
This is the first release in the v5 family. Where the v4 series closed out support for the PMIx Group family of APIs, the work in this series has gone into making the DVM elastic, survivable, and testable: a running DVM can grow and shrink under a scheduler’s direction, daemon failures no longer take it down, and PRRTE now ships automated test harnesses from unit tests up to a multi-node suite that runs on real clusters. A DVM run as root can also be shared between users, each job running as the user it belongs to. A systematic review of the tree corrected several hundred defects along the way.
The release requires PMIx v7.0.0 or later for both build and execution.
24.1.1.1. Highlights
Elastic DVM. A running DVM can add and release nodes through allocation requests passed via RAS to a scheduler. A grow that fails is rolled back, jobs are held until their nodes have daemons, and the requester is told when the change completes.
Slurm elastic support. Expander jobs via
salloc --no-shell, release by node list or allocation id, and cancellation by request id. Gated byprte_elastic_mode; requires jansson and Slurm v24.05 or later (see--enable-slurm-extensions).scontrol show job --jsonoutput is streamed, so the HNP’s memory no longer grows with the size of the job. An expander inherits the parent job’s GRES, reservation, required features, and excluded nodes (ras_slurm_propagate_extranames further job record members), and a queued extend is named by its allocation id, so it can be cancelled from another tool. The requester is told when Slurm granted its extend (PMIX_ALLOC_GRANT_TIME), and a cancel that arrives after the grant is answered with the allocation id to release.srunreads its node list from a file, so a launch is no longer bounded by the 128 KiB argument limit, and grows work under Slurm step management.Node reservation and session targeting. A job may name the allocation it is to be mapped onto. Each allocation has one owner, access is decided by permissions, and its properties can be queried. An allocation’s id is the one its grantor gives it, a request id names an allocation only to the job that made it, and the time an allocation request asks for is applied.
Sharing a DVM between users. A DVM run as root with
prte_pmix_no_foreign_tools 0accepts other users’ tools at every daemon, starts each job’s processes as the user the job belongs to, and opens the files a job names as that user. Who may act on a job, session, reservation, process set, or allocation, and who may grow the DVM, is decided by user and group through PMIx’s access rule. See Sharing a DVM between users.Authenticated daemon connections. Each daemon proves it holds the DVM key when it connects, and every message on the connection carries a Poly1305 tag that is checked before the message is acted on.
Launcher-less bootstrap. Daemons can assemble a DVM themselves, with no ssh, srun, or other launcher.
Survivable daemon failures. A reliable messaging layer (RELM) beneath the RML, routing-tree repair when a daemon dies, and boot-epoch stamps so a stale incarnation cannot rejoin. A daemon that returns is re-inserted and caught up. Group constructs complete on the survivors. A lost daemon’s node is marked down and leaves the allocation, fences complete once every process on a daemon has departed, and a restarted process takes part in them again.
Mapping by device.
--map-by device=<class|name>places processes against GPUs, NICs, or fabric devices, withppr:N:device=,ndev, andinterleave. Each process is told its device, and the vendor’s visibility environment variable is set. Anndevgroup stays under one PCI root complex, and--map-by device=gpu:nicgives each process the RDMA NICs under its GPU’s root complex, ranked by PCIe link speed.Memory binding.
--membindbinds each process’s memory to the local NUMA node(s) that are best byCAPACITY,LOCALITY,BANDWIDTH, orLATENCY– for example, to on-package HBM – and is carried on PMIx’sPMIX_MEMBIND.Per-app placement for MPMD jobs. Each app context carries its own
--map-by,--rank-by,--bind-to,--membind, andpe-listdirectives.Collectives reworked.
grpcommis now plain code rather than an MCA framework. Large broadcasts are scattered and gathered, fences can gather laterally, and releases travel a separate low-radix tree. A group ID already in use is refused, contributions that name different members fail the operation, and a fence or group operation that times out names the members it was waiting for.Connected jobs. Departures from a connected set of jobs are announced as sets rather than with one broadcast per process. A spawned job inherits its parent’s error handling, inheritance passes on to the spawned job’s own children, and a recoverable job is no longer terminated because a job connected to it failed. PMIx is told which process spawned a job, so a parent and its children can reach each other through Omni-Path (OPX) shared memory.
Messaging reworked. Per-peer progress threads and shared payloads in the RML/OOB.
prte_if_includeandprte_if_excludenow decide what is listened on, so a loopback-only DVM works (and avoids the macOS firewall prompt). A silent peer can no longer stall a daemon.I/O forwarding. XON/XOFF actually stops the producer, large stdin writes are split rather than truncated, output file names can be composed by the user, and tools connected to a non-master daemon get working I/O.
Data server. Published data honors its requested lifetime and persistence, access is by permission and then range, per-user holdings are bounded, and an external data server is reached as a PMIx tool.
Command-line parsing and mapper options. Numerous fixes to how the values of
--map-by,--rank-by,--bind-to,--output,--display, and--rtosare parsed and applied, including refusing ambiguous abbreviations. See the DIRECTIVES AND QUALIFIERS section of prterun(1) and the Process placement documentation.Hostfile and rankfile parsing. Both are now read by one line reader, with numerous fixes and error messages that name the line at fault. Node lookups are indexed, so reading a large hostfile no longer takes time quadratic in its length. See Hostfiles and Rankfiles.
Fewer hangs. Races between kill and launch, late launch reports, daemons failing during DVM formation,
MPI_Abort, the elastic launch fence, DVM shutdown while a spawned child was still dying, and aptermwhose halt was refused have all been closed.Hardening. Several hundred leaks, use-after-free errors, unchecked unpacks, and crashes on malformed input were fixed; command lines that used to crash a tool are now refused with a message. Session directories, logs, and staged files resist planted links and foreign ownership.
PMIx server surface. Every PMIx callback and upcall is thread-shifted onto PRRTE’s progress thread. Queries a daemon cannot answer, including
PMIX_QUERY_PROC_TABLE, are relayed to the DVM master, so the answer no longer depends on which daemon was asked.prun,prterun, andptermreflect the job’s outcome in their exit status. With PMIx v7,--report-uritakestcp-andtcp+for the TCP URI that clients of earlier PMIx releases use; Troubleshooting says how to run those clients with legacy peers.prun matches prterun.
prunkeeps quotes in application arguments, forwards exportedPMIX_MCA_*/OMPI_MCA_*settings under a persistent DVM, and honors--dvmunder the ompi personality. Aprunin aprterunjob script has its output shown once and keeps its step’s exit status to itself, and aprunon any node of a DVM receives the master’s--displayreports.MCA parameter routing. A generic
--mcais routed by the same prefix list whatever else is on the command line, andif_*parameters reach PMIx (aspif_*) and Open MPI.Options that were ignored now work.
--tuneunder the native personality,--xterm,prte_info --param, and comments inprunappfiles. Diagnostics are printed once, and everyshow_helpmessage names its job so a later job’s is not suppressed. A message about a tool’s request goes only to that tool, and the build checks everyshow_helpcall against its topic and argument count.Also new: resource usage monitoring,
ras/flux,rmaps/lsf, PBS launch viapbs_tmrsh, and--activateto start a daemon on an allocated node.Documentation. Full man pages for
prte,prterun,prun,pterm, andprted, built from the same source as--help; a Testing chapter; launch guides for each supported resource manager; a “how things work” section; design records underdocs/plans; a Code of Conduct; and a security policy inSECURITY.md.Testing. Five harnesses, where v4.0.0 had none:
make checkunit tests;make -C test/offline check-offlinefor over a thousand mapping, ranking, and binding cases;contrib/dockerswarm, a ten-node multi-node suite that also runs on a real cluster over ssh;contrib/slurmswarm, the same suite under a real Slurm; andcontrib/scalingfor collective performance.
24.1.1.2. Compatibility notes
PMIx v7.0.0 or later is required to build and run, including
pmix_cli_match()(theCLI_MATCHcapability). Building from a Git clone needs Python v3.6 or later and no longer needs flex.Every process in a DVM must come from the same PRRTE build. There is no compatibility between versions of
prte,prted, andprun.Out-of-tree MCA components must be rebuilt; framework interface versions are now checked at load time.
PRRTE error codes have new values, now based at
PMIX_EXTERNAL_ERR_BASE. 46 of them previously collided with live PMIx status codes.Ambiguous abbreviations are refused. Spellings that were resolved silently before –
--bind-to n,:s,:i,--rank-by s,--output ta– now produce an error naming the candidates.Daemon authentication cannot be turned off.
prte_oob_authenticateandprte_oob_authenticate_messagesare gone. The DVM key is handed to each daemon through the launch agent, sorshandremshagents are refused and the default agent list isssh; under Grid Engine,qrshnow forwards the key.A host may not name an account to reach it as.
user@host, in any host list, hostfile, rankfile, or allocation request, and the hostfileusername=key are refused.prte.conf, aDVMNodes=file:list, and the data server’s URI file are read only when their ownership and permissions meet the rule in PRRTE DVM Configuration.On a DVM not run as root, a job belonging to another user is refused. A scheduler that instantiates a session must name its owner with
PMIX_ALLOC_UIDandPMIX_ALLOC_GID.The processes a node may hold, even when oversubscribed, are capped at
prte_rmaps_base_max_procs_per_slot(default 64) times its slots, but never belowprte_rmaps_base_max_procs_per_node_floor(default 1024). Set the first to 0 to remove the cap.configurenow refuses hwloc older than v2.1.0, which it had accepted.The release scripts moved from
contrib/disttocontrib, andmake_dist_tarballtakes the same options as PMIx’s.A DVM not started in elastic mode refuses
--add-host,--add-hostfile,--activate, and PMIx allocation requests.round_robinmapping is held to each node’smax_slots, which no oversubscribe directive lifts.With
--uniform-nodes, a node whose topology differs fails the launch rather than being mapped as something it is not.Each top-level session directory is now
<tmpdir>/<prefix>.XXXXXX, made bymkdtemp(), so a directory already at a fixed name is never reused.Group membership keeps the order the participants gave it;
PMIX_GROUP_FINAL_MEMBERSHIP_ORDERis gone.Removed: Java support, the
prtedlframework, thedistmapping policy, LIKWID references, the--tmpdir,--test-suicide, and--alloc-refidoptions (name an allocation with--alloc-id), and theprte_pmix_remote_connectionsandprte_pmix_allow_client_clonesparameters.
A full list of individual changes will not be provided here, but will commence with the v5.0.1 release.