shQSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r SSK J r Jr SSKJr SSKJrJrJrJr SSKJs Js Jr SSKJs Js Js Jr SSKJ r J!r!J"r" SSK#J$r$J%r% SS K&J'r'J(r( SS KJ)r) SS K*J+r+ /S Qr,S r-Sr.\+"\/5r0\ "SS55r1"SS5r2"SS\3\5r4"SS5r5"SS5r6\ "SS55r7S\34Sjr8"SS\Rr5r:"SS \:5r;g)!N) defaultdict)contextmanager) dataclassfield)Enum)AnyCallableOptionalUnion)Event EventSourcerecord)prof put_metric)ProcessFailureSignalException)RendezvousGracefulExitError) get_logger) WorkerSpecWorker WorkerState WorkerGroup RunResult ElasticAgentSimpleElasticAgentz!torchelastic/agent/terminal_statedefaultc\rSrSr%Sr\\S'\\S'\R\S'Sr \ \ \S'Sr \\ \S4\S'S r\\S 'S r\\S 'S r\\S'Sr\ \\S'Sr\ \\S'Sr\ \\S'SrSrSrg)r/a Blueprint information about a particular type of worker. For a given role, there must only exist a single worker spec. Worker spec is expected to be homogeneous across all nodes (machine), that is each node runs the same number of workers for a particular spec. Args: role: user-defined role for the workers with this spec local_world_size: number local workers to run fn: (deprecated use entrypoint instead) entrypoint: worker function or command args: arguments to pass to ``entrypoint`` rdzv_handler: handles rdzv for this set of workers max_restarts: number of max retries for the workers monitor_interval: monitor status of workers every ``n`` seconds master_port: fixed port to run the c10d store on rank 0 if not specified then will chose a random free port master_addr: fixed master_addr to run the c10d store on rank 0 if not specified then will chose hostname on agent rank 0 redirects: redirect std streams to a file, selectively redirect for a particular local rank by passing a map tee: tees the specified std stream(s) to console + file, selectively tee for a particular local rank by passing a map, takes precedence over ``redirects`` settings. rolelocal_world_size rdzv_handlerNfn entrypointargs max_restartsg?monitor_interval master_port master_addr local_addrcURS:deURS:deUR(a*[R"S[ S9 URUlUR (deg)NrzJWorkerSpec.fn will be deprecated, please use WorkerSpec.entrypoint instead)category)r r(r"warningswarnDeprecationWarningr#selfs ~/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/elastic/agent/server/api.py __post_init__WorkerSpec.__post_init__Zs^$$q((($$q((( 77 MM<+  #ggDOc[UR[5(a)[RR UR5$URceURR $)zGet the entry point name. If the entrypoint is a function (e.g. ``Callable``) returns its ``__qualname__`` else if the entrypoint is a binary (e.g. ``str``), returns the binary name. ) isinstancer#strospathbasename __qualname__r1s r3get_entrypoint_nameWorkerSpec.get_entrypoint_namegsN doos + +77##DOO4 4??. ..??// /r6)r#)__name__ __module__r=__firstlineno____doc__r9__annotations__intrdzvRendezvousHandlerr"r r r#r r%tupler'r(floatr)r*r+r4r>__static_attributes__r$r6r3rr/s8 I(((!B!-1JhT)*1D%L#!e!!%K#%!%K#% $J $  0r6rc V\rSrSrSr/SQrSS\S\S\S\S\4 S jjrS rS r S r g )rta,A worker instance. Contrast this with ``WorkerSpec`` that represents the specifications of a worker. A ``Worker`` is created from a ``WorkerSpec``. A ``Worker`` is to a ``WorkerSpec`` as an object is to a class. The ``id`` of the worker is interpreted by the specific implementation of ``ElasticAgent``. For a local agent, it could be the ``pid (int)`` of the worker, for a remote agent it could be encoded as ``host:port (string)``. Args: id (Any): uniquely identifies a worker (interpreted by the agent) local_rank (int): local rank of the worker global_rank (int): global rank of the worker role_rank (int): rank of the worker across all workers that have the same role world_size (int): number of workers (globally) role_world_size (int): number of workers that have the same role id local_rank global_rank role_rank world_sizerole_world_sizerOrPrQrRrScNSUlXlX lX0lX@lXPlgNrM)r2rOrPrQrRrSs r3__init__Worker.__init__s0 * !, ( *%4r6c SURSURSURSURSUR3 $)Nz local_rank=z ,global_rank=z ,role_rank=z ,world_size=z,role_world_size=rOrPrQrRrSr1s r3__str__Worker.__str__sJ$//*-8H8H7I$..)doo5F 4 45 7 r6c[U5$rU)r9r1s r3__repr__Worker.__repr__s 4yr6)rPrNrOrQrSrRN)r_r_r_) r@rAr=rBrC __slots__rErVrZr]rJr$r6r3rrts\(I! 444 4  4  4B r6rcP\rSrSrSrSrSrSrSrSr Sr S r \ S SS \ 4S j5rS rg)raA state of the ``WorkerGroup``. Workers in a worker group change state as a unit. If a single worker in a worker group fails the entire set is considered failed:: UNKNOWN - agent lost track of worker group state, unrecoverable INIT - worker group object created not yet started HEALTHY - workers running and healthy UNHEALTHY - workers running and unhealthy STOPPED - workers stopped (interrupted) by the agent SUCCEEDED - workers finished running (exit 0) FAILED - workers failed to successfully finish (exit !0) A worker group starts from an initial ``INIT`` state, then progresses to ``HEALTHY`` or ``UNHEALTHY`` states, and finally reaches a terminal ``SUCCEEDED`` or ``FAILED`` state. Worker groups can be interrupted and temporarily put into ``STOPPED`` state by the agent. Workers in ``STOPPED`` state are scheduled to be restarted in the near future by the agent. Some examples of workers being put into ``STOPPED`` state are: 1. Worker group failure|unhealthy observed 2. Membership change detected When actions (start, stop, rdzv, retry, etc) on worker group fails and results in the action being partially applied to the worker group the state will be ``UNKNOWN``. Typically this happens on uncaught/unhandled exceptions during state change events on the agent. The agent is not expected to recover worker groups in ``UNKNOWN`` state and is better off self terminating and allowing the job manager to retry the node. UNKNOWNINITHEALTHY UNHEALTHYSTOPPED SUCCEEDEDFAILEDstatereturncHU[R[R1;$)zReturn the state of the Worker. Returns: True if the worker state represents workers still running (e.g. that the process exists but not necessarily healthy). )rrerf)rjs r3 is_runningWorkerState.is_runnings ,,k.C.CDDDr6r$N)r@rAr=rBrCrcrdrerfrgrhri staticmethodboolrmrJr$r6r3rrsQ DG DGIGI FE-EDEEr6rc.\rSrSrSr/SQrS\4SjrSrg)rzA set of ``Worker`` instances. The class defines a set of ``Worker`` instances for the given ``WorkerSpec`` managed by ``ElasticAgent``. Whether the worker group contains cross instance workers or not depends on the implementation of the agent. )specworkersstore group_rankgroup_world_sizerjr*r)rscXl[URR5Vs/sH n[US9PM snUlSUlSUlSUlSUlSUl [RUl gs snf)N)rO) rsranger rrtrurvrwr*r)rrdrj)r2rsis r3rVWorkerGroup.__init__sn 6;DII5> %  s$$$c$eCHo$$r6r}cr\rSrSr%Sr\\S'\"\S9r \\ \ 4\S'\"\S9r \\ \ 4\S'S\4SjrS rg ) riMaReturn results of the worker executions. Run results follow an "all-or-nothing" policy where the run is successful if and only if ALL local workers managed by this agent complete successfully. If the result is successful (e.g. ``is_failed() = False``) then the ``return_values`` field contains the outputs (return values) of the workers managed by THIS agent mapped by their GLOBAL ranks. That is ``result.return_values[0]`` is the return value of global rank 0. .. note:: ``return_values`` are only meaningful for when the worker entrypoint is a function. Workers specified as a binary entrypoint do not canonically have a return value and the ``return_values`` field is meaningless and may be empty. If ``is_failed()`` returns ``True`` then the ``failures`` field contains the failure information, again, mapped by the GLOBAL rank of the worker that failed. The keys in ``return_values`` and ``failures`` are mutually exclusive, that is, a worker's final state can only be one of: succeeded, failed. Workers intentionally terminated by the agent according to the agent's restart policy, are not represented in either ``return_values`` nor ``failures``. rj)default_factory return_valuesfailuresrkc<UR[R:H$rU)rjrrir1s r3 is_failedRunResult.is_failedkszz[////r6r$N)r@rAr=rBrCrrDrdictrrErrrrprrJr$r6r3rrMsK0 $)$$?M4S>?*/*EHd3&'E040r6rrkcT[R"[R"55$rU)socketgetfqdn gethostnamer$r6r3_get_fq_hostnameros >>&,,. //r6c\rSrSrSr\R \4S\S\ 4Sjj5r \R \4S\S\ 4Sjj5r Sr g) risaAn agent process responsible for managing one or more worker processes. The worker processes are assumed to be regular distributed PyTorch scripts. When the worker process is created by the agent, the agent provides the necessary information for the worker processes to properly initialize a torch process group. The exact deployment topology and ratio of agent-to-worker is dependent on the specific implementation of the agent and the user's job placement preferences. For instance, to run a distributed training job on GPU with 8 trainers (one per GPU) one can: 1. Use 8 x single GPU instances, place an agent per instance, managing 1 worker per agent. 2. Use 4 x double GPU instances, place an agent per instance, managing 2 workers per agent. 3. Use 2 x quad GPU instances, place an agent per instance, managing 4 workers per agent. 4. Use 1 x 8 GPU instance, place an agent per instance, managing 8 workers per agent. Usage :: group_result = agent.run() if group_result.is_failed(): # workers failed failure = group_result.failures[0] logger.exception("worker 0 failed with exit code : %s", failure.exit_code) else: return group_result.return_values[0] # return rank 0's results rrkc[e)a4Run the agent. Supports retrying the worker group on failures up to ``max_restarts``. Returns: The result of the execution, containing the return values or failure details for each worker mapped by the worker's global rank. Raises: Exception - any other failures NOT related to worker process NotImplementedErrorr2rs r3runElasticAgent.runs "!r6c[e)zReturn the ``WorkerGroup`` for the given ``role``. Note that the worker group is a mutable object and hence in a multi-threaded/process environment it may change state. Implementors are encouraged (but not required) to return a defensive read-only copy. rrs r3get_worker_groupElasticAgent.get_worker_group "!r6r$N)r@rAr=rBrCabcabstractmethod DEFAULT_ROLEr9rrrrrJr$r6r3rrss^ D * " "y " " +7"S"K""r6rc \rSrSrSrS-S\S\4Sjjr\4S\ S\ 4Sjjr \ RS \ S\\\44S j5r\ RS.S \ S \SS 4Sjj5r\ RS \ S\4Sj5r\ R\R.S 4S\R0S \SS 4Sjj5r\S \ SS 4Sj5r\S\S\S\S\\4Sj5r\S \ SS 4Sj5r\S \ SS 4Sj5r \\4S\ S\4Sjj5r!S\"4Sjr#S\"4Sjr$S\SS 4Sjr%S\S\S\ 4Sjr&\'S\ 4S j5r(S/S\ S!\)S\*\S"\*\ S#\*\S\"4 S$jjr+S%\4S&jr,S'r-S.S(\4S)jjr.\4S\ S\4S*jjr/S+r0S,r1g )0rizAn ``ElasticAgent`` that manages one particular type of worker role. An ``ElasticAgent`` that manages workers (``WorkerGroup``) for a single ``WorkerSpec`` such as one particular type of worker role. rsexit_barrier_timeoutc[U5UlURRRUlSUlX lSUlgNr)r _worker_grouprsr'_remaining_restarts_store_exit_barrier_timeout_total_execution_time)r2rsrs r3rVSimpleElasticAgent.__init__s>(.#'#5#5#:#:#G#G  %9"%&"r6rrkcUR$rU)rrs r3r#SimpleElasticAgent.get_worker_groups!!!r6 worker_groupc[e)zStart ``worker_group.spec.local_world_size`` number of workers. This is according to worker spec for the worker group . Returns a map of ``local_rank`` to worker ``id``. rr2rs r3_start_workers!SimpleElasticAgent._start_workerss "!r6F is_restartNc[e)zStop all workers in the given worker group. Implementors must deal with workers in all states defined by ``WorkerState``. That is, it must gracefully handle stopping non-existent workers, unhealthy (stuck) workers, etc. r)r2rrs r3 _stop_workers SimpleElasticAgent._stop_workerss "!r6c[e)znCheck on the workers for the ``worker_group``. This function also returns the new state of the worker group. rrs r3_monitor_workers#SimpleElasticAgent._monitor_workerss "!r6 death_sigc[e)zClean up any resources that were allocated during the agent's work. Args: death_sig: Signal to send to the child process, SIGTERM is default r)r2rrs r3 _shutdownSimpleElasticAgent._shutdownrr6c&URnURS5 URR5nSSS5 WRnUR nUR nUR=(d URRnUR=(d URRnX@l URS5 URXEXb5n SSS5 W Ul XAlXQl XalXqlXl URUR - n ["R%SUR&U UUUUU V s/sHoR(PM sn U V s/sHoR*PM sn U V s/sHoR,PM sn U V s/sHoR.PM sn U V s/sHoR PM sn S. 5 g!,(df  GN=f!,(df  GN=fs sn fs sn fs sn fs sn fs sn f)zRun rendezvous for the workers specified by the worker spec. Assigns workers a new global rank and world size. Updates the rendezvous store for the worker group. RENDEZVOUSNASSIGN_WORKER_RANKSa[%(role)s] Rendezvous complete for workers. Result: restart_count=%(restart_count)s master_addr=%(master_addr)s master_port=%(master_port)s group_rank=%(group_rank)s group_world_size=%(group_world_size)s local_ranks=%(local_ranks)s role_ranks=%(role_ranks)s global_ranks=%(global_ranks)s role_world_sizes=%(role_world_sizes)s global_world_sizes=%(global_world_sizes)s ) r restart_countr*r)rvrw local_ranks role_ranks global_ranksrole_world_sizesglobal_world_sizes)rsrecord_durationr!next_rendezvousrurrRr*bootstrap_store_infor)r_assign_worker_ranksrtrvrwr'rloggerinforrOrQrPrS) r2rrs rdzv_inforurvrwr*r)rtrworkers r3 _rendezvousSimpleElasticAgent._rendezvouss    ! !, /))99;I0^^ $//&&T)*H*H*T*T &&T)*H*H*T*T   ! !"7 8//#3G9 ' "",(8%#. #. ))D,D,DD   < !.**($4@GHf 1 1H>EFgF//gFBI J'!3!3' JJQ$R'%;%;'$RGN&OwV'8'8w&O  50 /9 8@ IF J$R&Os5GG( G:&G?HH 7H G%( G7rvrwc [RRSS5S:Xa"X4R-nX$R-nUnUnGOSn Sn [ UR X$R5n UR U U3U R55 US:XGa[UR[U5V s/sHn SU 3PM sn 5n U Vs/sHn[RU5PM nn[S5nSnUH3nUUR ==UR- ss'UUR- nM5 Sn[S5n/n/n[U5Hun nURU U 35 UR[R"UUUUR UUR /55 UUR- nUUR ==UR- ss'M UR!UU5 [R""URU U355unnnn/n[UR5H'n[%UUU-UU-UUS 9nURU5 M) U$s sn fs snf) aDetermine proper ranks for worker processes. Fast Path: when all workers have the same role and world size. We calculate the global rank to be group_rank * group_world_size + local_rank. And the `role_world_size` is the same as `global_world_size`. No TCP store is used in this case. This is only enabled when users set the environment variable `TORCH_ELASTIC_WORKER_IDENTICAL` to 1. Time complexity: each worker O(1), overall O(1) Slow Path: when workers have different roles and world sizes. We use the the following algorithm: 1. Each agent writes its configuration(group_rank, group_world_size , num_workers) to the common store. 2. The rank 0 agent reads all the role_info from the store and determines each agents worker ranks. 3. Determine the global rank: the global rank of the workers is computed by cumulative sum of the local_world_size for all workers in front of it. For efficiency reasons each worker is assigned a base global rank such that it's workers are in the range [base_global_rank, base_global_rank + local_world_size). 4. Determine the role rank: The role rank is determined using the algorithms in the point 3 with the exception that the ranks are calculated with respect to the role name. 5. The rank 0 agent writes the assigned ranks to the store. 6. Each agent reads the assigned ranks from the store. Time complexity: each worker O(1), rank0 O(n), overall O(n) TORCH_ELASTIC_WORKER_IDENTICAL01ztorchelastic/role_info/ztorchelastic/assigned_ranks/rcgrr$r$r6r39SimpleElasticAgent._assign_worker_ranks..er6cgrr$r$r6r3rrlrr6rY)r:environgetr r}rsetr multi_getryrrrappendrr multi_setrr)r2rurvrwrsglobal_world_sizebase_global_rankbase_role_rankrSROLE_INFO_PREFIXASSIGNED_RANKS_PREFIXagent_role_inforzrole_infos_bytes info_bytes role_infos role_sizes global_sizerrkeysvaluesrtrOrs r3r'SimpleElasticAgent._assign_worker_ranks*sF ::>>:C @C G 03H3H H ),A,AA -N/O8 $B !/ :'<'<O II)*:,79R9R9T UQ#(??MM 0 + *9>> : *9>> :  % (B(BB$y~~.)2L2LL.%: f- 599(='>zl%KLM   ! 5 56J%,z9(:5, / F NN6 "7mUs .I#I(cRURRn[RSU5 UR U5 [RSU5 UR U5nUR 5HupEURUnXVlM [RUl g)axStart a fresh set of workers for the worker_group. Essentially, a rendezvous followed by a ``start_workers``. The caller should first call ``_stop_workers()`` to stop running workers prior to calling this method. Optimistically sets the state of the worker group that just started as ``HEALTHY`` and delegates the actual monitoring of state to ``_monitor_workers()`` method z [%s] Rendezvous'ing worker groupz[%s] Starting worker groupN) rsrrrrritemsrtrNrrerj)r2rr worker_idsrOw_idrs r3_initialize_workers&SimpleElasticAgent._initialize_workerss  %% 6= & 0$7((6 * 0 0 2 J!))*5FI!3)00 r6cURRn[RSU5 UR USS9 [ R UlURU5 g)zCRestart (stops, rendezvous, starts) all local workers in the group.z[%s] Stopping worker groupT)rN) rsrrrrrrgrjr)r2rrs r3_restart_workers#SimpleElasticAgent._restart_workerssS  %% 0$7  >  NNLahh W NN188 $"O    ?# ),T^^-= -J)KD &# ),T^^-= -J)KD &s6AB11 D&;CE( D&#>D!!D&&E((?F'cfURS[R[R"5S9$)Nri)rjsource raw_error)_construct_eventr AGENT traceback format_excr1s r3get_event_failed#SimpleElasticAgent.get_event_faileds2$$$$**,%  r6c>URS[RS9$)Nrh)rjr!)r#r r$r1s r3get_event_succeeded&SimpleElasticAgent.get_event_succeededs&$$$$%  r6rc PURRHnURRUR5nUR X!5nU(a [ R"UR5OSn[URU[RX%55 M grU) rrtrrrP_get_worker_staterrerror_file_datarr#r WORKER)r2rrfailurerjr"s r3r(SimpleElasticAgent._record_worker_eventssw((00Foo))&*<*<=G//?E?F 7#:#:;DI 4(( 0B0BFV W 1r6rcVURRUR5nUR[R [R 1;aU(dgU(dURUR;aURR$[SUR35e)N TERMINATEDzUnknown worker: ) rrrPrjrrfrirvalue ValueError)r2rrr0s r3r-$SimpleElasticAgent._get_worker_states~//%%f&8&89 <rOrQrSrPr9rNrrr! get_run_idrvrrr get_backendr'rr ) r2rjr!rr"r8wgrsmdrPr?md_strrCs r3r##SimpleElasticAgent._construct_events*  ww " 3 3335   & 1 13B| %//1B{O%+%;%;$=B ! ,,KFIIIKIB''224&--"II(*"88 --99;""//$2J2JJ& )% 1&  r6 group_resultscUR5nURU5 URRnURUR :gn[ SURS3S5 URSU(+=(a U5 URSU(+=(a U(+5 URSU=(a U5 URSU=(a U(+5 g)Nworkers.z .run_totalrrun_success_with_retriesrun_success_no_retriesrun_failed_with_retriesrun_failed_no_retries) r_record_flakiness_metricrrsrr'rr_record_metric_with_condition)r2rKrrsrestarts_happeneds r3r"SimpleElasticAgent._record_metrics0s!++-  %%i0!!&& 448I8IIXdii[ 3Q7 ** &I (K:K  ** $)m&Mrr(r!rjrrdrsleeprrrnamelowerrhr _exit_barrierrfrir'rrrenum_nodes_waitingrv Exception) r2rrsr(r! run_resultrjrdrvs r3rSimpleElasticAgent._invoke_runVs!!&&yy 6>V>V>X    !3!3400(( %%++{/?/?? ?? JJ' (..t/A/ABJ$$E',   $ $':;T=U=U V $q)9)9);(<=q A --- F..  ""$!!;00+2D2DEE++a/KK5 00)),,1,))$*<*<=&&t'9'9:/:/A/AD&&,%%+---$0$B$B$D!!//:: $q(KK4)" ))$*<*<=v/ |6Bgr6cL[RSURRUR5 [ R "5n[ R"URURR[URS9 [RS[ R "5U- 5 g![a&n[RSUR5 eSnAf[a/ [RS[ R "5U- 5 gf=f)a# Define a barrier that keeps the agent process alive until all workers finish. Wait for ``exit_barrier_timeout`` seconds for all agents to finish executing their local workers (either successfully or not). This acts as a safety guard against user scripts that terminate at different times. zOLocal worker group finished (%s). Waiting %s seconds for other agents to finish)rurR key_prefixbarrier_timeoutz2Done waiting for other agents. Elapsed: %s secondszGot termination signal: %sNz2Error waiting on exit barrier. Elapsed: %s seconds)rrrrjrr store_utilbarrierrrw_TERMINAL_STATE_SYNC_IDrrrre exception)r2startrs r3rc SimpleElasticAgent._exit_barriers   <    $ $  & &       kk-->>2 $ : :   KKD e#   NN7 B     D e#  s A/B<< D#!C''9D#"D#)rrrrr)i,)F)NNN)2r@rAr=rBrCrrIrVrr9rrrrrrErrrprrrsignalSIGTERMSignalsrrrrrrrrrr r'r*rr-rrr r r#rrSrRrrcrJr$r6r3rrs 'Z'u',8"S"K" ";"4S>"" rs #%(1133::GG>UL> > H  A0A0 A0HGGT3E#t3El&&@6$6$r 00 0B0#0;"377;"|LLr6