torchrun (Elastic Launch) — PyTorch 2.3 documentation (2024)

Table of Contents
Transitioning from torch.distributed.launch to torchrun¶ Usage¶ Note on rendezvous backend¶ Definitions¶ Environment Variables¶ Deployment¶ Failure Modes¶ Membership Changes¶ Important Notices¶ References
  • Docs >
  • Torch Distributed Elastic >
  • torchrun (Elastic Launch)

Shortcuts

Superset of torch.distributed.launch.

torchrun provides a superset of the functionality as torch.distributed.launchwith the following additional functionalities:

  1. Worker failures are handled gracefully by restarting all workers.

  2. Worker RANK and WORLD_SIZE are assigned automatically.

  3. Number of nodes is allowed to change between minimum and maximum sizes (elasticity).

Note

torchrun is a pythonconsole scriptto the main moduletorch.distributed.rundeclared in the entry_points configuration insetup.py.It is equivalent to invoking python -m torch.distributed.run.

Transitioning from torch.distributed.launch to torchrun

torchrun supports the same arguments as torch.distributed.launch exceptfor --use-env which is now deprecated. To migrate from torch.distributed.launchto torchrun follow these steps:

  1. If your training script is already reading local_rank from the LOCAL_RANK environment variable.Then you need simply omit the --use-env flag, e.g.:

    torch.distributed.launch

    torchrun

    $ python -m torch.distributed.launch --use-env train_script.py
    $ torchrun train_script.py

  2. If your training script reads local rank from a --local-rank cmd argument.Change your training script to read from the LOCAL_RANK environment variable asdemonstrated by the following code snippet:

    torch.distributed.launch

    torchrun

    import argparseparser = argparse.ArgumentParser()parser.add_argument("--local-rank", type=int)args = parser.parse_args()local_rank = args.local_rank
    import oslocal_rank = int(os.environ["LOCAL_RANK"])

The aformentioned changes suffice to migrate from torch.distributed.launch to torchrun.To take advantage of new features such as elasticity, fault-tolerance, and error reporting of torchrunplease refer to:

  • Train script for more information on authoring training scripts that are torchrun compliant.

  • the rest of this page for more information on the features of torchrun.

Usage

Single-node multi-worker

torchrun --standalone --nnodes=1 --nproc-per-node=$NUM_TRAINERS YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)

Stacked single-node multi-worker

To run multiple instances (separate jobs) of single-node, multi-worker on thesame host, we need to make sure that each instance (job) issetup on different ports to avoid port conflicts (or worse, two jobs being mergedas a single job). To do this you have to run with --rdzv-backend=c10dand specify a different port by setting --rdzv-endpoint=localhost:$PORT_k.For --nodes=1, its often convenient to let torchrun pick a free randomport automatically instead of manually assigning different ports for each run.

See Also
Learn about Configs — MMDetection 2.21.0 documentationPerformance Tuning Guide — PyTorch Tutorials 2.3.0+cu121 documentationLearn about Configs — MMClassification 0.25.0 documentationLearn about Configs — MMDetection3D 1.4.0 documentation

torchrun --rdzv-backend=c10d --rdzv-endpoint=localhost:0 --nnodes=1 --nproc-per-node=$NUM_TRAINERS YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)

Fault tolerant (fixed sized number of workers, no elasticity, tolerates 3 failures)

torchrun --nnodes=$NUM_NODES --nproc-per-node=$NUM_TRAINERS --max-restarts=3 --rdzv-id=$JOB_ID --rdzv-backend=c10d --rdzv-endpoint=$HOST_NODE_ADDR YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)

HOST_NODE_ADDR, in form <host>[:<port>] (e.g. node1.example.com:29400), specifies the node andthe port on which the C10d rendezvous backend should be instantiated and hosted. It can be anynode in your training cluster, but ideally you should pick a node that has a high bandwidth.

Note

If no port number is specified HOST_NODE_ADDR defaults to 29400.

Elastic (min=1, max=4, tolerates up to 3 membership changes or failures)

torchrun --nnodes=1:4 --nproc-per-node=$NUM_TRAINERS --max-restarts=3 --rdzv-id=$JOB_ID --rdzv-backend=c10d --rdzv-endpoint=$HOST_NODE_ADDR YOUR_TRAINING_SCRIPT.py (--arg1 ... train script args...)

HOST_NODE_ADDR, in form <host>[:<port>] (e.g. node1.example.com:29400), specifies the node andthe port on which the C10d rendezvous backend should be instantiated and hosted. It can be anynode in your training cluster, but ideally you should pick a node that has a high bandwidth.

Note

If no port number is specified HOST_NODE_ADDR defaults to 29400.

Note on rendezvous backend

For multi-node training you need to specify:

  1. --rdzv-id: A unique job id (shared by all nodes participating in the job)

  2. --rdzv-backend: An implementation oftorch.distributed.elastic.rendezvous.RendezvousHandler

  3. --rdzv-endpoint: The endpoint where the rendezvous backend is running; usually in formhost:port.

Currently c10d (recommended), etcd-v2, and etcd (legacy) rendezvous backends aresupported out of the box. To use etcd-v2 or etcd, setup an etcd server with the v2 apienabled (e.g. --enable-v2).

Warning

etcd-v2 and etcd rendezvous use etcd API v2. You MUST enable the v2 API on the etcdserver. Our tests use etcd v3.4.3.

Warning

For etcd-based rendezvous we recommend using etcd-v2 over etcd which is functionallyequivalent, but uses a revised implementation. etcd is in maintenance mode and will beremoved in a future version.

Definitions

  1. Node - A physical instance or a container; maps to the unit that the job manager works with.

  2. Worker - A worker in the context of distributed training.

  3. WorkerGroup - The set of workers that execute the same function (e.g. trainers).

  4. LocalWorkerGroup - A subset of the workers in the worker group running on the same node.

  5. RANK - The rank of the worker within a worker group.

  6. WORLD_SIZE - The total number of workers in a worker group.

  7. LOCAL_RANK - The rank of the worker within a local worker group.

  8. LOCAL_WORLD_SIZE - The size of the local worker group.

  9. rdzv_id - A user-defined id that uniquely identifies the worker group for a job. This id isused by each node to join as a member of a particular worker group.

  1. rdzv_backend - The backend of the rendezvous (e.g. c10d). This is typically a stronglyconsistent key-value store.

  2. rdzv_endpoint - The rendezvous backend endpoint; usually in form <host>:<port>.

A Node runs LOCAL_WORLD_SIZE workers which comprise a LocalWorkerGroup. The union ofall LocalWorkerGroups in the nodes in the job comprise the WorkerGroup.

See Also
Adding New Dataset — MMDetection 2.2.1 documentation

Environment Variables

The following environment variables are made available to you in your script:

  1. LOCAL_RANK - The local rank.

  2. RANK - The global rank.

  3. GROUP_RANK - The rank of the worker group. A number between 0 and max_nnodes. Whenrunning a single worker group per node, this is the rank of the node.

  4. ROLE_RANK - The rank of the worker across all the workers that have the same role. The roleof the worker is specified in the WorkerSpec.

  5. LOCAL_WORLD_SIZE - The local world size (e.g. number of workers running locally); equals to--nproc-per-node specified on torchrun.

  6. WORLD_SIZE - The world size (total number of workers in the job).

  7. ROLE_WORLD_SIZE - The total number of workers that was launched with the same role specifiedin WorkerSpec.

  8. MASTER_ADDR - The FQDN of the host that is running worker with rank 0; used to initializethe Torch Distributed backend.

  9. MASTER_PORT - The port on the MASTER_ADDR that can be used to host the C10d TCP store.

  10. TORCHELASTIC_RESTART_COUNT - The number of worker group restarts so far.

  11. TORCHELASTIC_MAX_RESTARTS - The configured maximum number of restarts.

  12. TORCHELASTIC_RUN_ID - Equal to the rendezvous run_id (e.g. unique job id).

  13. PYTHON_EXEC - System executable override. If provided, the python user script willuse the value of PYTHON_EXEC as executable. The sys.executable is used by default.

Deployment

  1. (Not needed for the C10d backend) Start the rendezvous backend server and get the endpoint (to bepassed as --rdzv-endpoint to the launcher script)

  2. Single-node multi-worker: Start the launcher on the host to start the agent process whichcreates and monitors a local worker group.

  3. Multi-node multi-worker: Start the launcher with the same arguments on all the nodesparticipating in training.

When using a job/cluster manager the entry point command to the multi-node job should be thislauncher.

Failure Modes

  1. Worker failure: For a training job with n workers, if k<=n workers fail all workersare stopped and restarted up to max_restarts.

  2. Agent failure: An agent failure results in a local worker group failure. It is up to the jobmanager to fail the entire job (gang semantics) or attempt to replace the node. Both behaviorsare supported by the agent.

  3. Node failure: Same as agent failure.

Membership Changes

  1. Node departure (scale-down): The agent is notified of the departure, all existing workers arestopped, a new WorkerGroup is formed, and all workers are started with a new RANK andWORLD_SIZE.

  2. Node arrival (scale-up): The new node is admitted to the job, all existing workers are stopped,a new WorkerGroup is formed, and all workers are started with a new RANK andWORLD_SIZE.

Important Notices

  1. This utility and multi-process distributed (single-node ormulti-node) GPU training currently only achieves the best performance usingthe NCCL distributed backend. Thus NCCL backend is the recommended backend touse for GPU training.

  2. The environment variables necessary to initialize a Torch process group are provided to you bythis module, no need for you to pass RANK manually. To initialize a process group in yourtraining script, simply run:

>>> import torch.distributed as dist>>> dist.init_process_group(backend="gloo|nccl")

  1. In your training program, you can either use regular distributed functionsor use torch.nn.parallel.DistributedDataParallel() module. If yourtraining program uses GPUs for training and you would like to usetorch.nn.parallel.DistributedDataParallel() module,here is how to configure it.

local_rank = int(os.environ["LOCAL_RANK"])model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank], output_device=local_rank)

Please ensure that device_ids argument is set to be the only GPU device idthat your code will be operating on. This is generally the local rank of theprocess. In other words, the device_ids needs to be [int(os.environ("LOCAL_RANK"))],and output_device needs to be int(os.environ("LOCAL_RANK")) in order to use thisutility

  1. On failures or membership changes ALL surviving workers are killed immediately. Make sure tocheckpoint your progress. The frequency of checkpoints should depend on your job’s tolerancefor lost work.

  2. This module only supports hom*ogeneous LOCAL_WORLD_SIZE. That is, it is assumed that allnodes run the same number of local workers (per role).

  3. RANK is NOT stable. Between restarts, the local workers on a node can be assigned adifferent range of ranks than before. NEVER hard code any assumptions about the stable-ness ofranks or some correlation between RANK and LOCAL_RANK.

  4. When using elasticity (min_size!=max_size) DO NOT hard code assumptions aboutWORLD_SIZE as the world size can change as nodes are allowed to leave and join.

  5. It is recommended for your script to have the following structure:

def main(): load_checkpoint(checkpoint_path) initialize() train()def train(): for batch in iter(dataset): train_step(batch) if should_checkpoint: save_checkpoint(checkpoint_path)

  1. (Recommended) On worker errors, this tool will summarize the details of the error(e.g. time, rank, host, pid, traceback, etc). On each node, the first error (by timestamp)is heuristically reported as the “Root Cause” error. To get tracebacks as part of thiserror summary print out, you must decorate your main entrypoint function in yourtraining script as shown in the example below. If not decorated, then the summarywill not include the traceback of the exception and will only contain the exitcode.For details on torchelastic error handling see: https://pytorch.org/docs/stable/elastic/errors.html

from torch.distributed.elastic.multiprocessing.errors import record@recorddef main(): # do train passif __name__ == "__main__": main()
' document.getElementById("pytorch-article").insertAdjacentHTML('afterBegin', div) }
torchrun (Elastic Launch) — PyTorch 2.3 documentation (2024)

References

Top Articles
The Best Gluten Free Challah Bread Recipe | Easy and Delicious!
5 Great Dutch Ovens And 18 Recipes to Put Them to Work
Frigidaire Dehumidifier Fad504Dwd Manual
Motor City Airsoft Photos
2023 Ucsd Payroll Calendar
Maria Butina Bikini
Green Pets And Animals For Sale
100+ Birthday Wishes For Sister: Simple, Heart Touching, Short & Funny Wishes - LIC Merchant
WATCH— Reagan 2024 FullMovie Free Online ON 123MOVIES
Fifty Shades Freed | Rotten Tomatoes
Single Sign On Davita
Craigslist Atv For Sale By Owner
Latest Posts
Harry Potter Butterbeer Fudge (Copycat Recipe)
25 Delicious Keto Snack Recipes That Are Worth Making
Article information

Author: Edwin Metz

Last Updated:

Views: 6207

Rating: 4.8 / 5 (78 voted)

Reviews: 85% of readers found this page helpful

Author information

Name: Edwin Metz

Birthday: 1997-04-16

Address: 51593 Leanne Light, Kuphalmouth, DE 50012-5183

Phone: +639107620957

Job: Corporate Banking Technician

Hobby: Reading, scrapbook, role-playing games, Fishing, Fishing, Scuba diving, Beekeeping

Introduction: My name is Edwin Metz, I am a fair, energetic, helpful, brave, outstanding, nice, helpful person who loves writing and wants to share my knowledge and understanding with you.