sh SSSKrSSKrSSKrSSKrSSKJrJrJr SSKJ r SSK J r J r SSK Jr SSKJrJrJrJrJrJrJr SSKrSSKJr SSKJs Js Jr SSKJ r SSK!J"r" SSK#J$r$ SS K%J&r&J'r'J(r(J)r)J*r*J+r+J,r, SS K-J.r. SS K/J0r0J1r1 SS K2J3r3J4r4 SS K5J6r6J7r7 SSK8J9r9J:r:J;r; SSKr> SSK?J@r@ \(aSSKAJBrB \R"\D5rE\ "SS55rFS\G\H\4S\\I\H\44SjrJ\ "SS55rK"SS\5rL"SS\5rMS\FS\G\H\4S\NS \NS!\NS\O\G\H\44 S"jrPS#\S\N4S$jrQS\FS\G\H\4S\K4S%jrRS\FS&\KS \NS\O\G\H\44S'jrSS(\'S)\G\H\4S*\\RS\G\H\44S+jrUS(\'S&\S*\\RS\4S,jrVS\FS-\HS)\G\H\4S\G\H\44S.jrWShS/\G\H\4S0\ RS1\NS2\\RRS3\NS*\\RS\G\H\44S4jjr[S\FS5\G\H\G\H\44S6\O\HS\G\H\44S7jr\S8\HS9\O\RS6\O\HS:\\RS;\1S\R4 S<jr_S8\HS=\O\RS6\O\HS\R4S>jr`S8\HS?\O\S6\O\HS\4S@jraSiSA\G\H\4S0\ RS2\RRSB\\\O\G\H\4\\ R4SC\NSD\NS\G\H\44SEjjrcSjS0\ RSB\\\O\G\H\4\\ R4S\G\d\ R44SFjjreS0\R@RS\G\0\H44SGjrfSkS2\RRS0\\ RSD\NSH\\G\ R\O\H4SI\\G\0\H4S\G\\d\H4\ R44 SJjjrgSkS2\RRS0\\ RSD\NSH\\G\ R\O\H4SI\\G\0\H4S\G\ R\\d\H444 SKjjrhSjS0\ RSB\\\O\G\H\4\\ R4S\G\ R\d44SLjjriSM\O\MSN\G\M\\H\d44SO\G\\H\d4\ R4S*\\RSS4 SPjrjSiS/\G\H\4S*\\RSO\G\\d\H4\ R4SH\G\ R\O\H4SQ\G\H\F4SR\NS\I\O\M\G\M\\d\H4444SSjjrkST\G\H\4SO\G\\d\H4\ R4SH\G\ R\O\H4S\O\G\H\44SUjrlS/\G\H\4S\N4SVjrm\ "SWSX55rnS(\'SY\G\H\4S\O\G\H\n44SZjroS\FS[\O\G\H\n4SY\G\H\4S\\G\H\G\H\44S\I\\R\G\H\O\\R444 S]jrqS\FS\\G\H\G\H\44S8\HS \NS\NS!\NSS4S^jrrS\FS[\O\G\H\n4SY\G\H\4S \NS\NS!\NS\G\H\G\H\444S_jrsS\FSY\G\H\4S \NS\NS!\NS\G\H\44 S`jrtSlSa\O\MSN\G\M\\d\H44SQ\G\H\F4S/\G\\H\d4\4S\NS \NS!\NS\G\H\44SbjjruSlSa\O\MSN\G\M\\d\H44SQ\G\H\F4S/\G\\H\d4\4S\NS \NS!\NS\G\H\44Scjjrv\R"5SmS0\ RS2\RRS/\G\H\4SB\\\O\G\H\4\\ R4S3\NS \NS*\\RSC\NS1\NS!\NS\G\H\44Sdjj5rxS0\ RS\G\H\F44Sejry\S(\'Sf\:SS4Sgj5rzg)nN)IterableIteratorSequence) ExitStack) dataclassfield)chain)Anycast NamedTuple no_type_checkOptional TYPE_CHECKINGUnion)_gather_state_dict)_get_pg_default_device)_apply_to_modules _FSDPState._get_module_fsdp_state_if_fully_sharded_module_get_param_to_fqns_module_handle!_named_parameters_with_duplicatesclean_tensor_name)SimpleProfiler) FlatParameterFlatParamHandle)_ext_chunk_dtensor_ext_chunk_tensor) _lazy_init%_reset_flat_param_grad_info_if_needed)ShardingStrategyStateDictSettings StateDictType)DTensor Replicate) tree_map_only) ShardedTensorcN\rSrSr%\\S'\\S'\\\ 4\S'\ \ \S'Sr g) FSDPParamInfo5statehandle param_indicesparam_requires_gradN) __name__ __module__ __qualname____firstlineno__r__annotations__rdictstrintlistbool__static_attributes__r/w/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/fsdp/_optim_utils.pyr)r)5s&  S>!d#r;r) dictionaryreturnc#`# [UR55nUH nX U4v M g7fN)sortedkeys)r=rBks r< sorted_itemsrD=s- *//# $D As,.c\rSrSr%Sr\"\S9r\\\ R4\ S'\"\S9r \\\ R4\ S'\"\S9r \\\4\ S'Srg) _ConsolidatedOptimStateCa4 This holds the consolidated optimizer state on the target rank. Positive- dimension tensor state is communicated across ranks, while zero-dimension tensor state and non-tensor state is taken directly from the target rank. PyTorch version 1.12 moved to using zero-dimension tensors for scalar values, but user implemented optimizers may still use float (i.e. a non-tensor). Thus, we support both and handle them identically. Attributes: tensor_state (Dict[str, torch.Tensor]): Mapping from positive-dimension tensor state name to the unsharded flat tensor representing the state. zero_dim_tensor_state (Dict[str, torch.Tensor]): Mapping from zero- dimension tensor state name to its value. non_tensor_state (Dict[str, Any]): Mapping from non-tensor state name to its value. )default_factory tensor_statezero_dim_tensor_statenon_tensor_stater/N)r0r1r2r3__doc__rr5rIr6torchTensorr4rJrKr r:r/r;r<rFrFCs\&-2$,GL$sELL()G5:45P4U\\ 12P',T'Bd38nBr;rFcV\rSrSr%Sr\R \S'\R\S'Sr g)_PosDimTensorInfo]ac Metadata for positive-dimension tensors used internally for :meth:`scatter_full_optim_state_dict`. Attributes: shape (torch.Size): Sharded tensor shape (which is equal to the unsharded tensor shape if the tensor is optimizer state for a non-FSDP parameter and is hence not sharded). dtype (torch.dtype): Data type of the tensor. shapedtyper/N) r0r1r2r3rLrMSizer4rSr:r/r;r<rPrP]s  :: ;;r;rPc8\rSrSr%Sr\\S4\S'\\S'Sr g)_OptimStateKeymz This represents an optimizer state key that may be used commonly across ranks. It is based on the unflattened parameter names rather than parameter IDs to make it independent of each rank's own optimizer construction. .unflat_param_namesis_fsdp_managedr/N) r0r1r2r3rLtupler6r4r9r:r/r;r<rVrVms c3h'r;rVfsdp_param_infoflat_param_stateto_save shard_state cpu_offloadcFU(aU(dS5e[UU5nU(ax[UUU5nUHcnU(dM [UR55H:nXxn [ U [ R 5(dM(U R5Xx'M< Me U$/$)a Unflattens the optimizer state, consisting of the "state" part and the "param_groups" part. Unflattening the "state" part involves consolidating the state on the target rank and remapping from flattened to unflattened parameter IDs, and the "param_groups" part only involves remapping from flattened to unflattened parameter IDs. Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): Entry for the flat parameter in the "state" part of the optimizer state dict. to_save (bool): Whether to save the state on this rank. Returns: List[Dict[str, Any]]: A :class:`list` holding the entries in the "state" part of the optimizer state dict corresponding to the unflattened parameters comprising the flat parameter if on the target rank or an empty :class:`list` otherwise. The final optimizer state dict will need to map these entries using the proper unflattened parameter IDs. z7If ``shard_state`` is True, ``to_save`` has to be True.)_communicate_optim_state#_unflatten_communicated_optim_stater8rB isinstancerMrNcpu) r[r\r]r^r_consolidated_stateunflat_param_state optim_statekeyr+s r<_unflatten_optim_staterixs:gA %2@     .K{ 0 0 23C',E%eU\\:: ',yy{K$ 4."! r;xcb[R"U5=(a UR5S:H$Nr)rM is_tensordim)rjs r<_is_zero_dim_tensorros ??1  .!%%'Q,.r;cURnURRn[5nURUR UR pvn[U5GHup[R"U 5(GaDU R5S:Ga/URS:XdUR[R:XaXU'MkURcS5eU R R"URR":waU R%UR5n UR&R)5n U R*"U 6n [,R."XUR0S9 UR2R55 [7[8R:UR<5R?5n U SU XX'GMf[AU 5(a#U RC5RE5Xh'GMXU'GM U$)a Communicates the optimizer state for a flat parameter across ranks. All ranks will hold the entire non-sharded optimizer state on GPU. If ``N`` is the number of tensor optimizer states in the optimizer state dict, then the communication complexity is 0 if ``N = 0`` and ``N + 1`` otherwise (where the plus 1 comes from all-gathering the padding per rank). Args: fsdp_param_info (FSDPParamInfo): The FSDP state, the handle, and a mapping from FQN to original parameter index. flat_param_state (Dict[str, Any]): The entry in the "state" part of the optimizer state dict corresponding to the flat parameter. Returns: ConsolidatedOptimState: Consolidated optimizer state for the target flat parameter. rNz'compute_device has not been initializedgroup)#r+r, flat_paramrFrIrJrKrDrMrmrn world_sizesharding_strategyr!NO_SHARDcompute_devicedevicetypeto_full_param_paddedsize new_zerosdistall_gather_into_tensor process_group_device_handle synchronizer nn Parameter_unpadded_unsharded_sizenumelrodetachclone) r[r\ fsdp_statertr+rIrJrK state_namevalue buffer_size tensor_bufferunpadded_numels r<raras,!&&J ''22J # %E  ## *:L **:; ??5 ! !eiikAo %%*//3C3L3LL+0Z(,,8 9 8||  J$=$=$B$BB!:!:;%77<<>K!OO[9M  ' 'J,D,D   % % 1 1 3! jAAeg (5_n'EL $#5))49LLN4H4H4J%1/4,G>>"4#""//"22 #K&33???"3#""--"11>>@"00"22 #K.9  *9(B>,88M+N 'J-<  *,O'33C&D "J -7  *'E!!"45O&P r;rrgrscS/n[R"U5S:Xa[[RSU5US'[R "USUS9 [R"U5S:XaU$US$)NrcUR5S:XaUR5$[URUR5$rl)rnrdrPrRrS)vs r<,_broadcast_processed_state..Os.AaeegV3DQWWagg3VVr;srcrs)rget_rankr&rMrNbroadcast_object_list)rrgrsobjectss r<_broadcast_processed_staterFsl G }}Uq " LL V    wAU; }}Uq qzr;c[R"U5S:XaQ[U[R5(aUR 5S:XaU$UR UR5nO[U[R5(aUR 5S:XdS5eU$[U[5(dU$[R"URURURS9n[R"USUS9 U$)NrzlFor non-zero ranks, a tensor state should have zero dimension, but got the state with shape {state.shape()}.rSryr) rrrcrMrNrnr{rxrPzerosrRrS broadcast)rr+rstensors r<_broadcast_staterYs }}Uq %..%))+2BL*334 eU\\ * *99;!# @ #LE#455L KKu{{:3L3L  NN6q. Mr;fqncHU(d0$URnURRnURUnURUn[ X#R URS9nUR(d0$0nURnURn UR5Hyup[R"U 5(aUU R5S:aAUR[ R":wa#U R%5XS-R'5n XU 'M{ U$)z Shard the optimizer state for the original parameter with the name ``fqn``. This API should only be used when ``use_orig_params`` is True. pgryrrq)r+r,rtr-_shard_param_infosrrrxin_shardintra_param_start_idxintra_param_end_idxitemsrMrmrnrvr!rwflattenr) r[rrgrrt param_idxshard_param_infonew_optim_staterrrrs r<_shard_orig_param_staterps   &&J ''22J--c2I!44Y?$009R9RK  $ $ &(O,BB*>>(..0 OOE " " a,,0@0I0IIMMO%)eg ', #1 r;optim_state_dictmodeluse_orig_paramsoptim rank0_onlyc F[R"5 UnSU;aU(d [S5e[U5n[ U5n[ [ UR555Rn U(a [XUS9n0n USn [U R55n UR5GH<upUSnX;aMU RU5 U(aGUH<nX(dMXR5Hn[XUUUS9XU'M M> USnX;GaXnU(aG[R"[R R"5 [%UUX5nSSS5 O ['UU U5n[)[+U5S5nW(aUU U'OU(a~[-U5S:Xd SUS 35eUb`URR/U S5nUb[0R2"U5U U'Ox[4R6"S US U R8S 35 OP[;S US 35e[-U5S:Xde[)[+U5S5n[0R0"X5U U'U(dGMUH_nX(dM[=XR55H2unnU R8S:aXU MUR?5XU'M4 Ma GM? U HYnU Un[AU[BRD5(aU(aU(a [U UUS9n[0R0"U5U U'M[ [RF"S5 SU;a[0R2"US5nU US.$SU 0$!,(df  GN=f)a~ Flattens the full optimizer state dict, still keying by unflattened parameter names. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP know how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- it is managed by other parallelism and FSDP does not know ho to handle/aggregate them. Note that ``_flatten_tensor_optim_state`` does not need ``optim`` to flatten/shard the state. However, NamedOptimizer and KeyedOptimizer require all the states even if the corresponding parameters are empty. To this end, ``optim`` will be used to to get the initial state of the empty parameters. ``optim`` should only be non-None if the ``optim` is KeyedOptimizer or NamedOptimizer. Returns: Dict[str, Any]: The flattened optimizer state dict. r+zO`optim_state_dict` must have the keys "state"to be a valid optimizer state dictrrrNTrqz5use_orig_params is True but there are multiple FQNs, .z optim_state[z] is not on rankz The state of z8 is empty. This should happen when use_orig_params=True.Fz,FSDP _flatten_optim_state_dict() profiling: param_groupsr+r)$rreset ValueErrorr_get_fqn_to_fsdp_param_inforitervaluesr+rsetrBrdifference_updaterprofileType RESHARDINGr_flatten_optim_staterVrZlengetcopydeepcopywarningswarnr RuntimeErrorr8rdrcrMrNdump_and_reset)rrrrrrs unflat_osd param_to_fqnsfqn_to_fsdp_param_inforflat_osd_stateunflat_osd_stateall_state_keysparamfqnsrrr[ flat_staterhr+ param_state user_stateflat_osd_param_groupss r<_flatten_optim_state_dictrs@!Jj  1  'u-M8?d188:;<BBJ/ eT =?N!'*)..01N$**, 1g  & ((. ',"2"7"<"<">J8H"S$9*$EU9$)*5#?q'C  (49O#++N,?,?,J,JK!8'(-"JLK2#$ !td3C&0s# 4yA~KD6QRS~$!KKOOE48E(.2mmE.Bs+ *3%/? ?PPQR ##C5),, t9> !> te4C"&)),<,A"BN3  :',/34D4I4O4O4Q/R+J !*,1*==HOO> "4 "4   0   /'',,   "4 K1  %   05578Kc"4"9"9";<< ##5"682  "" "57J! ': &9"/A.L  *RV V&9  -9 JLqAL J$%)J " GKKK2_&A 5??1#5#E!%%'A+ E  $7$: : 5??1#55 5O'#88"7Qa"78 u:? #7? @  f E_B  >c%jAoUV V  FLLE$9$ll^1UG5  Ce$J#&o"K #L K  "  knnZ01 ]] KK!   #L (();=R=RSK!::    0 0 {0011BCSBTU 0 UE 0. sF+F+F0:AF5zero_dim_tensorsc UVs/sH o3cMUPM nnUVs1sHo3bUR5OSiM nnUVs1sHo3b UROSiM nn[U5[U5:wd[U5S:wd[U5S:wa[SUSUSUSU35e[ [ U55n[ [ U55n[ R"Xx[ R"S5S9$s snfs snfs snf) a, Flattens the zero-dimension tensor optimizer state given by the values ``zero_dim_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all tensors are the same and using that common value. NOTE: The requirement that the tensors are the same across all unflattened parameters comprising the flat parameter is needed to maintain the invariant that FSDP performs the same computation as its non-sharded equivalent. This means that none of the unflattened parameters can be missing this state since imposing a value may differ from having no value. For example, for Adam's "step", no value means maximum bias correction, while having some positive value means less bias correction. Args: state_name (str): Optimizer state name. zero_dim_tensors (List[torch.Tensor]): Zero-dimension optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: torch.Tensor: A zero-dimensional tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. NrqAll unflattened parameters comprising a single flat parameter must have scalar state with the same value and dtype but got values z and dtypes rrrdr) itemrSrrrrrMrry) rrrXr r  values_setr rrSs r<rrsB$4E#3a#3E?OP?O!m!&&(5?OJP:J K:JQaggD0:JF K %5!66 z?a  v;!  (\fX[l;!"  $   j! "E f E <<5<<3F GG%FP KsC9C9C>D non_tensorscUVs/sH o3cMUPM nn[U5n[U5[U5:wd[U5S:wa[SUSUSU35e[[ U55nU$s snf)a@ Flattens the non-tensor optimizer state given by the values ``non_tensors`` for the state ``state_name`` for a single flat parameter corresponding to the unflattened parameter names ``unflat_param_names`` by enforcing that all values are the same and using that common value. See the note in :func:`_flatten_zero_dim_tensor_optim_state`. Args: state_name (str): Optimizer state name. non_tensors (List[Any]): Non-tensor optimizer state for the unflattened parameters corresponding to the single flat parameter. unflat_param_names (List[str]): A :class:`list` of unflattened parameter names corresponding to the single flat parameter. Returns: Any: A non-tensor giving the value of the state ``state_name`` for all unflattened parameters corresponding to the names ``unflat_param_names``. rqrrz" and unflattened parameter names )rrrrr)rrrXntnon_none_non_tensorsnon_tensor_setrs r<rr8s2*5G2BG%N  C $44N8Kq8P ,-[ E++=*> @  d>*+J Hs A3A3 sharded_osd optim_inputusing_optim_inputis_named_optimizerc 6[U5n[U5n[[[R [ [[44U(a [X5O [X!XVU55n[U5[U5::de0n 0n UR5H*upX;aM Xn X[U 5'U HnXU'M M, USn0nUR5HMunn[U[5(aUUU'M"U RUR UR 5n UUU 'MO SU;a`/nUSHPn["R$"U5n['USVs1sHnXiM sn5nUUS'UR)U5 MR UUS.$SU0$s snf)a Rekeys the optimizer state dict from unflattened parameter names to flat parameter IDs according to the calling rank's ``optim``, which may be different across ranks. In particular, the unflattened parameter names are represented as :class:`_OptimStateKey` s. r+rparamsr)r_get_flat_param_to_fqnr r5rrrr7r6'_get_param_to_param_id_from_optim_input_get_param_to_param_keyrrrZrcrrXrrrAr)rrrrrrrflat_param_to_fqnparam_to_param_key$unflat_param_names_to_flat_param_key#unflat_param_name_to_flat_param_keyrrXflat_param_keyrsharded_osd_staterekeyed_osd_staterhrrekeyed_osd_param_groupsunflat_param_groupflat_param_groupflat_param_keyss r<_rekey_sharded_optim_state_dictr0_s&'u-M.u5>B R\\5c? *+! 4E G(0AR ? ! "c-&8 88 8 ) (&3%8%8%:!  * +2JXU3E-FG!3 ES0A B"4 &;$G,46-335[ c3  %0 c " =AA  " "C$:$: -8.)6$9; "-n"= #}}-?@ $.@-I-I)8J-IO *9 X & $ + +,< =#>+??KIz%66 Zt,,  ySTTIf%&& 9,. ![0  & ~   X!67  +, --;  "m %  s*D++ E 5EE c xSnSn0n[UUU[U5VVs/sHupEUPM snnU5$s snnf)a Constructs a mapping from ``FlatParameter`` to a cleaned (devoid of prefixes from wrappers) fully qualified name (FQN). Note that this FQN is "non-canonical" because ``FlatParameter`` s do not come from the original module but are registered only after FSDP has been applied. This function returns the FSDP-given name for the ``FlatParameter`` (usually module._flat_param) as opposed to the canonical FQNs returned for ``FlatParameter`` s in ``_common_utils._get_param_to_fqns(...)``). Consequently, this function will only return a non-empty mapping if FSDP was applied with ``use_orig_params=False`` as, otherwise, the original parameters are used within the module and there would be no ``FlatParameter`` s in the module. cz[USS9H-upE[U[5(dM[X-5nXcU'M/ g)NF)recurse)rrcrr)moduleprefix tree_levelr% param_namerrs r< module_fn)_get_flat_param_to_fqn..module_fn s@!B E" Je]33#F$78C'*e $ " r;cU$r@r/)r%s r< return_fn)_get_flat_param_to_fqn..return_fn  r;rr)rrDrGflat_param_to_fqn_retrrs r<r"r"sN+!79  -J R J   7>FCi#6H2I&I = I2$E *F,,L 12%++fUKOOK-A"'(;(;E(B"CD"CQD"CD xUC /  $H-JD^,d3D4y1}dV#J:>?$3..$?@B .9%%D@s  E/E4r merge_keysc[R"U5n0n/nUR5HupXS;aMX:n [U [5n U (a,U SU;d#U S[ UR 5545eU SU;n [[U 5U S9n US:XdU(aURU 5 XU 'M U(as[[R"U55Vs/sHn/PM nn[R"XUS9 /[R"U5Qn[[!U55nX4$US:XaU/OS/n[R""USUS9 USceUSn[%UUUU5 X4$s snf)a, Construct the local mapping between the ``_OptimStateKey`` and parameter keys and all the ``_OptimStateKey`` across ranks. If ``merge_keys`` is False, rank0 must contain all the ``_OptimStateKey``, an exception will be raised otherwise. Note that ``merge_keys`` should equal to ``use_orig_params``. r+r)rXrYrrNr)rrrrcrr8rBrVrZrrr[r\r from_iterablerArrrc)rrsrPrrrdrrWall_optim_state_keysr_rrrYoptim_state_keyrall_keysmerge_all_optim_state_keys key_obj_lists r<_map_param_key_to_optim_keysrls == DJL 13.446  W5 5 #$UM: 744 Q+00237 4q'%;;($T{+  19 ' ' 88A_5'7*d11%890 91B9 0  xUK%Eu':':8'D%E"%c*D&EF =='+ai ! "dV  ""BcURSS5nU(dg[[UR555n[ U[ 5$![an[ U5UeSnAff=f)z Returns whether the state_dict is from a NamedOptimizer. This function checks that the keys in the state_dict['state'] are strings (which usually are FQNs) versus integers (which usually refer to param_ids from a vanilla torch.optim.Optimizer). r+NF)rrrrB Exceptionrcr6)rr+rhr6s r<_is_named_optimizerrtsg  $ /E 14 %& c3  1()q01s"A A) A$$A)cf\rSrSr%\\\4\S'\\\R4\S'\\\ 4\S'Sr g) StateInfoitensorsscalar_tensorsrr/N) r0r1r2r3r5r6rPr4rMrNr r:r/r;r<rvrvs7#(( ))ell*++c3hr;rv input_statesc00n[UR5Vs/sHn0PM nnUR5HupV[0005n[ U5Hup[ R "U 5(abU R5S:XaU R5URU'MS[U RU R5URU'MXRU'M XrU'M [R "UUUR"S9 U$s snf)z Given the ``input_states``, allgather StateInfo for each state. The function uses all_gather_object to gather StateInfo so no GPU tensors are sent. rrr)rrurrvrDrMrmrnrdrxrPrRrSrwrrr\r) rryprocessed_state_dictrgathered_state_inforrgprocessed_staterrs r<_allgather_state_infor~ s24*//070q07)..0#BB/!-k!: Ju%%99;!#AFO22:>:K U[[;O++J7;@++J7";%4S!1 && 17s Dr| output_statesc r0nUR5GHupVUVs/sHowUPM nn[UV V s1sH%oRR5HoiM M' sn n 5n [ 5n Sn U GH9n/n[ 5n[ U5HunnUR S5 URRUS5nUbCURR5US'U (d URn OU UR:XdeUSS:XdMURU5 M U (aU U:XdeUn X;aURVs/sHnSPM snXN'X%RUS5nUb%URURR5nUXNURU'GM< [ U5HunnUU ;aMUR R5H=unnURUS5nUbUU:XdSUSUSUS3SU3-5eUUU'M? UR"R5HSunnURUS5nUb3[$R&"UU5(dSUSUSUS3SU3-5eUUU'MU M GM W U4$s snfs sn n fs snf) a Given the ``gathered_state_info`` and ``input_states``, the API converted the StateInfo into the original state if the state is not a non-scalar tensor. For a multi-dimensional tensor, the local state will be stored in ``state_buffer`` in a correct order for later allgather purpose. NrzRank z has different values for z: rz Other ranks: )rrArwrBrr2rrrRrrSaddr-r{r+rxrrxrMequal)r[r|ryr state_buffersrgathered_states state_infor+nall_tensor_states empty_ranksrSrnumels _empty_ranksr object_stateinfor local_statenamenon_tensor_valuecurr_non_tensor_valuescalar_tensor_valuecurr_scalar_tensor_values r<_convert_all_state_infor1s>@M,224&9:&9f&9 :"& EJ5 0B0B0D1Q0DQJ E !$ '+,JF%(UL&/ &;"l a #++// DA#!%!1!1!3F2J $ $ 222":? $$T*'<#k\&A AA&K."1"?"?-"?QD"?- )'+// DAK&)nn_-B-B-Q-QR LWM %o&C&CC&H I7,@#,J"7 D,{"*6*B*B*H*H*J&&(6(:(:4(F%)1,0@@D6!;D6DTCUUVW&'<&=>? A (8t$+K.:-H-H-N-N-P))+9+=+=dD+I(/75;;')A<<D6!;D6DWCXXYZ&'?&@AB (;t$.Q#8S 5B - A; E0-sJ),J.  J4c U(dgURRnURnUR5GHhupXn URUn [ U [ 5(aU RSn U [5:waU Rn U R[54S9 [URU 5nX==U RRS5-ss'[R "U5nU R#U5n O=U R#URU 5n OU R#URU 5n U(aUR$n['USS5(a<UR(ce[+U UR,UR(UR.5n OUR0ce[3U UR,UR4UR6R95UR0UR.5n ODU(d=[:R<"S5 U R?5RA5n SSS5 U(aG[:R<"[:RBRD5 U RG5n SSS5 XU'GMk g!,(df  Nf=f!,(df  N)=f)a Given a output state dict, ``output_states``, which the keys are FQNs to the original parameters (not FlatParameters nor parmeter ID), and the values are gathered states, unflatten the states to the original dimensions. This function performs the unflattening process in-place. Nr) placementsrFr)$r,rtr+rr-rcr$rr%rn redistributer8r device_meshr}rMrTreshaperrrrrrrrrurrrrrrrD2Hrd)r[rrr^r]r_rtrrrrr placement placement_dim reshape_sizers r<_unflatten_orig_param_statesrsF  ''22J &&J,224*#11#6  eW % %((+IIK' ) ""y{n"=#J$6$6y$AB +u/@/@/E/Ea/HH+$zz,7  l3 j&8&8&CDMM*"4"4Y"?@E #<599!..:::*OO++.. "//;;;)OO))--::<,,.. ''0 ,,.1 ''(;(;(?(?@ A%*z"e 5X10A@s*J)J:) J7 : K c URnURS:Xa_[R"5[RR :Xa.[ RSURR55 UR5Vs0sHow0_M nn[XX(5up[U 5S:XaU$URR5VV s/sHup{Xx;aSOSPM n nn URR n ["R$"[&R(XR*S9nU"U R,5nURR/5 U R5GHunn/nURU R0R35-nUU R0R35-S- nSunn[5U R6U R85GHunnU(+=(a% UR:U(+=(a U U(+nU(dU(aUUU-S- nnUUs=::aU::aO OUU:aUU- S-OUU- S-nOCUUs=::aU::aO OUU::aUU- S-OUU- S-nOUUs=:a Us=::aU:aO OUnOSnU(aUR=U"U55 U(d7UUb,UR=[?[&R@UU55 US- nUU- nGM U R0R35[CS U55- nU RDU:XdFS U RDS US U R0R35S U R6SUSUS3 5eUS:aUR=U"U55 [&RF"U5nUR35URH-UR35:XdS5eURR/5 [JRL"[JRNRP5 [RR"UUURTS9 URR/5 SSS5 USU RVR35nURn U RYU5n![U!5[UR5:XdS5eURR5H*up{UR:U (dXx;dM U!U XU'M, [[UUUUUU5 GM AU$s snfs sn nf!,(df  N=f)z Given the ``gathered_state_info`` and ``input_states``, the API allgathers all tensor states and restore non-tensor states from ``gathered_state_info``. rz@Memory Summary before calling to _allgather_orig_param_states %sTFrrq)rrNc3@# UHoR5v M g7fr@)r).0r s r< /_allgather_orig_param_states..>s1=a =szLManually calculated _sharded_numel_padded is incorrect. _shard_numel_padded=z, shard_numel_padded=z, _sharded_size.numel=z, _numels_with_padding=z, begin=z, end=,zThe size of local shard times the world size should equal to the gathered tensor size. The inconsistency may be from a bug of FlatParameter's metadata or the reconstruction logic in optimizer state dict.rrzThe number of parameters from FlatParameter is not consistent to the number of states used by optimizer state dict reconstruction logic.).r+rrget_debug_level DebugLevelDETAILloggerrrmemory_summaryrBrrr-rr,rt functoolspartialrMemptyrx_padded_unsharded_sizer _sharded_sizerr_numels_with_padding_is_padding_maskr.rr rNsum_shard_numel_paddedcatrurrr ALLGATHERrrr_get_unflat_views_alignedr)"r[r|ryr^r]r_rrrrSridxhas_state_paramsrt empty_funcgathered_tensorrbuffers local_buffersbeginend mem_offsetrr is_paddingfrozen_and_no_state padding_begin padding_end padding_lenshard_numel_padded local_shardunpadded_tensorflat_param_handle orig_statess" r<_allgather_orig_param_statesrsG!&&J! 4 4 6$//:P:P P N  % % 4 4 6 DPCTCTCV/WCVCRCVM/W2lE =Q(55;;=$=HC$%/=$!''22J"" 5)B)BJ!!B!BCO))+,224 G,. *":":"@"@"BBj..4466: $ I!$  + +Z-H-H" E:'1.##77 BB4(33 0 .8e9Ka9O{  E8[8 +-$e+a/ 5[1_ #c8[8 !M1m+a/ 5[1_ ]?k?C?#(K"#K!((K)@A9%1!((ellGI!? @ii .   "Z%:%::o>S>S>UU   U !!--/  # #N$7$7$A$A B  ' 'J4L4L   % % 1 1 3 C**WJ,O,O,U,U,WX+22'AA/R ;3'D'D#EE   E (55;;=HC22373;O1 %        W 5h  e0X$XC Bs UU:;U  U c URnURS:XdUR[R:Xa U(aU$0$[ R "[ RR5 [ R "[ RR5 [XQ5nSSS5 [UWUUUU5nSSS5 U(aURR5HmupUW;aM URU (dM#[US[!URR#55S[!UR#55S35e W$0$!,(df  N=f!,(df  N=f)a" Given a optimizer state dict, ``input_states``, which the keys are FQNs to the original parameters (not FlatParameters nor parmeter ID), gather all the states and unflatten them to the original dimensions. Note that all the params referred by the ``input_states`` must be managed by FSDP. rqNzB is not in the output state. The FSDPParamInfo has the param keys z, while the output_states has the param keys r)r+rurvr!rwrrrr ALLGATHER_OBJr~rr-rr.rrArB) r[ryr^r]r_rr|rrhrs r<_gather_all_orig_param_statertsU!&&J"  ' '+;+D+D D&|.B.    3 3 > > ?  # #N$7$7$E$E F"7 "Q G4         @'55;;=HCm#"66s;%8/77<<>?@A8-,,./0 3  > 5G F @ ?s$4/E6# E%/E6% E3 /E66 Frgc 80n0nUGHtn URU S5n U cU R(dM,U R(a^U RSn URU S5n U cMcU c0OX:n [U 5U;a0U[U 5'X[U 5U 'MU(dM[ U R5S:XdeU RSn[ R "S5 [[[[4U 5n [R"X:5X~'U(aI[X~5H8unn[R"U5(dM#UR5X~U'M: SSS5 GMw UR!5Hn[#[%UR'555n X+n [ U R(5S:dS5eU R*R-5HnunnUU;aMU R(U(dM$[/US[1U R*R'55S[1UR'55S35e UR3[5U UUUU55 M U$!,(df  GM=f)Nrrqnone_fsdp_managed_copyzgWith use_orig_params, FSDPParamInfo should have requires_grad information. However, the length is zero.zE is not in the optimizer state. The FSDPParamInfo has the param keys z( while the optimizer has the param keys r)rrYrXidrrrr rr6r7rrDrMrmrdrrrrBr.r-rrrAupdater)rgrWrrr]r^r_fsdp_osd_state all_statesrhr_rr[r+rrr _all_statesrhrs r<_convert_state_with_orig_paramsrs&(N -/J0+G+K+K T,   _%D%D   * *!44Q7C488dCO&#+B1A1LE/"*424 2o./38r/* +C 0 W99:a? ?? / B B1 E ''(@A sCx)< 48II$/51-9&9.) E %u55$HM 9*E . BA70T"((* 4 ((*+,05?667!;  8 ;(55;;=HCk!"66s;%8/77<<>?@A4+**,-.a 1  >  (   '+: YBAs +BJ  J c0nUGHMnURUS5n U c SUSU 35eUR(avURSn X*n [U X9UUU5n U(aI[ U 5[ UR5:Xde[ URU 5H upXU 'M MMU(dM[ UR5S:XdeURSn [ R "X95X}'U(dGM[X}5H8unn[R"U5(dM#UR5X}U'M: GMP U$)NzQIf use_orig_params is False, we must be able to find the corresponding param id. rrrq) rrYrXrirrrrDrMrmrd)rgrWrrr]r^r_rrhr_rr[ unflat_staterrfrrs r<_convert_state_with_flat_paramsrs&(N0+G+K+K T, $  ''6&7q  E $  * *"44Q7C49O1 + L<(C0R0R,SSSS=@#66 >9%9K#45 >W99:a? ?? / B B1 E 04 :J:U0VN -{)5"5*%J!??511 DIIIKN5jA *G0T r;c P[R"5 [5n U R[R"[R R 55 [[R"U55 U(+=(d" [R"U5S:H=(d Un [R"S5 [U5n [U5n [U5n[[ ["[$[&4[(R*4U(a [-X5O [/XXU 55n[1U5nSSS5 [R"S5 [3UUWW WUS9unnSSS5 [R"S5 U(a[4O[6nU"WWWUSU UU 5nSSS5 U (d0$SW0n[9W R;55nUSR=5H=unnUU;aMUU;aMUW;aM[>R@"SUS 35 UUU'M? S U;a[CUWW 5US 'U RE5 [RF"S 5 U$!,(df  GN:=f!,(df  GN=f!,(df  N=f) av Consolidates the optimizer state and returns it as a :class:`dict` following the convention of :meth:`torch.optim.Optimizer.state_dict`, i.e. with keys ``"state"`` and ``"param_groups"``. The flat parameters in ``FSDP`` modules contained in ``model`` are mapped back to their unflattened parameters. Parameter keys are not well-defined. For a regular optimizer, the optimizer state_dict contains a mapping from parameter IDs to parameter states. Parameter IDs are the order of parameters in ``optim.param_groups()`` across all the groups. This API also allows user to pass ``optim_input`` for the mapping between parameters and parameter IDs. Using ``optim_input`` is being deprecated. If the optimizer is a ``NamedOptimizer``, the optimizer state_dict does not contain parameter IDs mapping but a mapping from parameter FQNs to parameter states. This API finds the mapping from FQNs to parameters if the optimizer is a ``NamedOptimizer``. If ``use_orig_params`` is True, each rank will have all FSDP-managed parameters but some of these parameters may be empty due to the sharding. For a regular optim.Optimizer, states for those empty parameters will not be initialized. So, when aggregating the FQNs across ranks, no assert will be raised on a rank even if it does not have all the states -- it is valid and FSDP knows how to aggregate them. However, FSDP has to ignore handling those parameters that are not managed by FSDP and do not exist on the local rank -- those are managed by other parallelisms and FSDP does not know how to handle/aggregate them. Args: model (nn.Module): Root module (which may or may not be a :class:`FullyShardedDataParallel` instance) whose parameters were passed into the optimizer ``optim``. optim (torch.optim.Optimizer): Optimizer for ``model`` 's parameters. rank0_only (bool): If ``True``, saves the populated :class:`dict` only on rank 0; if ``False``, saves it on all ranks. (Default: ``True``) shard_state (bool): If ``True``, shard and distribute all non-zero-dimension states. Returns: Dict[str, Any]: A :class:`dict` containing the optimizer state for ``model`` 's original unflattened parameters and including keys "state" and "param_groups" following the convention of :meth:`torch.optim.Optimizer.state_dict`. If ``rank0_only=False``, then nonzero ranks return an empty :class:`dict`. r preprocessingNpreprocessing_with_comm)rdstate_convertingr+zFound a optim state, aK, that FSDP cannot process. FSDP will directly copy everything to the returned state_dict. In most cases, this is a user-defined state that is not associated with any particular parameter. Another possible case is this state is managed by TorchRec. Otherwise, there may be a mismatched assumption of optim_state_dict of this mode.rz$FSDP _optim_state_dict() profiling: )$rrr enter_contextrrALLr traversal_utils_get_fsdp_handlesrrrr"rtr r5rr7r6rrr<rRrrlrrrrrrrrqcloser)rrrrrr^rsrrr_cmr]rr%rrPrrgrW convert_fnrfsdp_osdflat_param_fqnsrhrs r<_optim_state_dictr:soD B^++N,?,?,C,CDE)/*K*KE*RSnH e 4 9H[G    0*51 25901AB! sCx",,. /%8K,"4EV  "=U!C 1"    9 : )     "&   ( ;    2 3 ,0  $ ( " W %     4$  '8H+2245O&w/557 U .  / !  $ $   #C5)L L $s%8())#: 0-$  HHJ!!"HI Oe 1 0" ; : 4 3s%A>I3J%J3 J J J%c xSnSn0n[UUU[U5VVs/sHupEUPM snnU5$s snnf)a Construct the mapping from a param's fqn to its corresponding ``FSDPParamInfo`` if the param is managed by FSDP. Shared parameters, or original parameters that are shared across multiple nn.Modules, are required to belong to one and only one FSDP instance and thus correspond to one ``FlatParameter``. Within the one ``FlatParameter``, ``FlatParameter._fqns`` only stores the first FQN of a shared parameter. Thus, the keys in the mapping are guaranteed to map to unique parameters. c[U5nUcg[X@5 [X@5nU(dgURn[ XE0/5n[ UR 5Hup[X-5n X;a"X:RRULdU 5eXsU 'XRU 'URcMZURRURUR5 M gr@)rrrrtr)r2_fqnsrr,r-_paramsr.r requires_grad) r@rArBfqn_to_param_inforr,rtr[r local_fqnrs r<rD._get_fqn_to_fsdp_param_info..module_fnsCFK   :& 3 && ' BC( (8(89NC#F$67C'(-44??:MRsRM%4c "14 ) )# .!!-33::&&s+99:r;cU$r@r/)rs r<rG._get_fqn_to_fsdp_param_info..return_fnrIr;rJ)rrDrGrrrs r<rrsP.!35  400-== m<< <9E]  HL  7 7 D1r;)FNFN)Fr@)NFNN)T)FT){rrloggingrcollections.abcrrr contextlibr dataclassesrr itertoolsr typingr r r r rrrrMtorch.distributed distributedr'torch.distributed.fsdp._traversal_utilsfsdp_traversal_utilsrtorch.nnr#torch.distributed._state_dict_utilsr"torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr#torch.distributed.fsdp._debug_utilsr"torch.distributed.fsdp._flat_paramrr'torch.distributed.fsdp._fsdp_extensionsrr%torch.distributed.fsdp._runtime_utilsrr torch.distributed.fsdp.apir!r"r#torch.distributed.tensorr$r%torch.utils._pytreer&'torch.distributed._shard.sharded_tensorr' getLoggerr0rr)r5r6rZrDrFrPrVr9r8rirorarb ProcessGrouprrrModuler OptimizerrrrNrTrrrrr0r7r<r"rRr$r#rcrlrqrtrvr~rSrrrrrrno_gradrrrr/r;r<rs  88 (WWW AABE?M 8-E   8 $ $$ $T#s(^sCx0I  CC C2   Z4"438n44 4  4  $sCx. 4n/3/4/C"C38nCCLJ"J "JJ $sCx. JZc3h D%% & #s(^ &#&/78I8I/J.#"# #c3h# #s(^ #R"-1)- L)38nL) 99L)L) EKK)) * L)  L) D%% & L) #s(^L)^F"F3S#X./FS F #s(^ FRQQ%,,'QS Q"%**- Q  Q  \\ Qh3H3H5<<(3HS 3H \\ 3Hl$$c$S $ $d %M,c3hM, 99M, ;; M, c3h R\\ " $  M,M,M, #s(^M,n H. 99H. c3h R\\ " $ H. #r|| H.V"%((//"d=#;M6N"N"&$=A<@ 0 ;; 0 BII 00DtCy!89: 0  ]C%7 89 0  %S/2<< '( 0j"&$=A<@ N ;; N BII NNDtCy!89: N  ]C%7 89 N  ",,c3h '( N2  N 99 N c3h R\\ " $  N ",,  N'&n-'&"&~uS#X'F"G'&U38_bll:;'& D%% & '&  '&` <>38n<> D%% &<>U38_bll:;<> d3i/0 <> !m!34 <>  <> 4 neCHo&E!F FG<>~S#XU38_bll:; d3i/0 $sCx. , $sCx. T &     ""sCx." $sI~ "JP "P d3 >23P sCx.P T#s(^+, P  8EKK $sD%,,1G,H'H"I IJ P fE+"E+T#s(^+,E+E+ E+  E+  E+ E+Pe"ed3 >23esCx.e e  e  e #tCH~ eP/"/sCx.// /  /  #s(^ /rX~.X"&~uS#X'F"GX!m!34X5c?C/0 X  X  XX #s(^XD6~.6"&~uS#X'F"G6!m!3465c?C/0 6  6  66 #s(^6r"Y 99Y ;; Y38nY c3h R\\ " $  YYY D%% &YYYY  #s(^!YYx.ryy.T#}:L5M.bLL*L LLr;