shSSKrSSKrSSKrSSKrSSKJrJrJr SSKJ r J r J r J r J r Jr SSKrSSKJr SSKJs Js Jr SSKJs Js Jr SSKJs Js Jr SSKJr SSKJ r SSK!J"r"J#r# SSK$J%r% SSK&J'r'J(r(J)r)J*r*J+r+J,r,J-r- SSK.J/r/J0r0J1r1J2r2 SS K3J4r4 SS K5J6r6J7r7J8r8J9r9J:r:J;r;JJ?r? SS K@JArA SS KBJCrC SSKDJErE \ (aSSKFJGrG SrHSSKIJJrJJKrK \M"S5rNSrO\P\R\R4rR\ \\R\R4rS\;R\2R\;R\2R\;R\2R\;R\2R\;R\2R0rY\;R\;R/rZ\;R\;R4r[\ S_S\(S\SS\;S\ \?S\ \#S\(4 Sjj5r\\ S\(S\SS\#S\(4Sj5r]\ S\ S\^4Sj5r_\ S\#S\^4Sj5r`\ S\MS\R4S j5ra\ S!\RS\MS\R4S"j5rbS!\RS\MS\P\R\R44S#jrc\ S_S\(S$\RS%\ \\R<RS&\\ \\R<R\ \\R<R4S\(4 S'jj5rfS&\g\ S(\^SS4S)jrh\ S\(S$\RS*\i\RS+\ \\M\R4S\(4 S,j5rk\ S\(S$\RS\(4S-j5rl\ S\(S\ \;S.\ \:S/\ \7S0\^S1\^S2\MS3\MS\(4S4j5rm\ S\(S\(4S5j5rn\ S\(S6\6S7\^S\(4S8j5ro\ S_S\(S\#S\(4S9jj5rp\ S\(S\(4S:j5rqS$\RS;\g\RSS4S<jrr\ S\(S=\RS+\ \\M\R4S>\ \ \R/S4S?\^S\(4 S@j5rs\ S\(S;\g\RS=\R4SAj5rtSB\RSC\ \\R<RS\i\R4SDjruS_SB\R<RS%\i\R<RSE\ \\R<RS\i\R<R4SFjjrvSB\R<RS%\i\R<RS\i\w4SGjrxSB\RS\i\w4SHjryS$\RS*\i\RS+\ \\M\R4SS4SIjrzS+\ \\M\R4SJ\MSK\'S\ \R4SLjr{S$\RS*\i\RS%\i\RS\P\^\^44SMjr|SB\RS>\ \R/S4S%\i\RSS4SNjr}SB\RSO\ \RS%\i\RSK\'4SPjr~SB\RS%\i\RS\g\R4SQjrS$\RS*\i\RSR\i\GRSO\ \RSS4 SSjrS;\g\RST\g\GRSO\ \RSS4SUjrSVrS$\RS*\i\RSO\ \RSJ\MSK\'S\R4 SWjrS$\RS;\g\RS\RSS4SXjrSY\g\GRSS4SZjrS$\RS*\i\RS\\R4S[jrS*\i\RSS4S\jrS\;4S]jrS\RS\ GR4S^jrg!\La SrHGNf=f)`N) GeneratorIterableIterator)AnyCallable no_type_checkOptional TYPE_CHECKINGUnion) default_hooks)_mesh_resources DeviceMesh)_get_default_group)_FSDPDeviceHandle _FSDPState_get_module_fsdp_state_is_fsdp_flattened!_named_parameters_with_duplicatesclean_tensor_name TrainingState)_FSDP_USE_FULL_PREC_IN_EVAL FlatParameterFlatParamHandleHandleShardingStrategy)_FreeEventQueue)BackwardPrefetch CPUOffloadFullOptimStateDictConfigFullStateDictConfigMixedPrecisionShardingStrategyStateDictConfig StateDictType)_Policy)DTensorExtensions)_sync_params_and_buffers)is_traceable_wrapper_subclass)RemovableHandleT) deferred_initfakeFi _fsdp_syncedstate process_groupsharding_strategypolicy device_meshreturncFUbUb [S5eU[;nU(a%UcUcUc[SUS35e[XU5nO6U(aX@lUR SS9UlOUbUO [ 5UlUR R5UlUR R5Ul URnU(aX`RR5-n[RRU5UlX`R- UlU$)NzcCannot pass both process_group and device_mesh at the same time. Please just pass only one of them.zManual wrapping with zA requires explicit specification of process group or device_mesh.rmesh_dim) ValueErrorHYBRID_SHARDING_STRATEGIES*_init_process_group_state_for_hybrid_shard _device_mesh get_groupr-rranksize world_size_inter_node_pgr DefaultState_get_gradient_predivide_factor_gradient_predivide_factor_gradient_postdivide_factor)r,r-r.r/r0is_hybrid_strategydata_parallel_world_sizes v/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/fsdp/_init_utils.py_init_process_group_staterEYsA [%< <  +.HH  V^ 8K'(9':;SS  ?kE !, "-"7"7"7"CE "/!: @R@T  $$))+EJ**//1E$// $8$8$=$=$?? ""AA $  $ !#C#CC % LcU(aW[U5(a/X lURSS9UlURSS9UlO[ SUR 35eUc<[5n[X0RR55upEX@lXPlO7[U5(aUuUlUlO[ S[U535e[URS9UlU$)Nrr3z,Expected device_mesh to have ndim=2 but got zmExpected process_group to be passed in as either None or Tuple[dist.ProcessGroup, dist.ProcessGroup] but got r-)"_is_valid_hybrid_shard_device_meshr8r9r=r-r5ndimr!_init_intra_and_inter_node_groups_device_handle device_count_is_valid_hybrid_shard_pg_typetype_get_default_comm_hook_state_inter_node_state)r,r-r0 default_groupintra_node_groupinter_node_groups rDr7r7s  -k : :!, $/#8#8!#8#DE "-"7"7"7"CE >{?O?O>PQ   *, -N //<<>. *// *- 8 89F 5E !5GGKMGZF[]  ;**E LrFc~[U[5=(a' [U5S:H=(a [SU55$)Nc3V# UHn[U[R5v M! g7fN) isinstancedist ProcessGroup).0pgs rD 1_is_valid_hybrid_shard_pg_type..s JMb 2t0011M'))rZtuplelenallrIs rDrOrOs: =%( K  ! # K JMJ JrFcN[U[5=(a URS:H$)NrW)rZrrK)r0s rDrJrJs k: . H;3C3Cq3HHrFnum_devices_per_nodec6[R"U5upU$)a5 Return a process group across the current node. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return an intra-node subgroup across [0, 1, ..., 7] or [8, 9, ..., 15] depending on the process's rank. For example, rank 3 would get [0, 1, ..., 7]. )r[ new_subgroups)rfintra_node_subgroup_s rD_init_intra_node_process_grouprks"//0DE rFglobal_process_groupc\Sn[R"U5n[R"U5nXA-n[R"U5U-n[ U5H?n[ U5Vs/sH oX--PM n n[R "XS9n Xv:XdM=U nMA Uc US35eU$s snf)al Return an inter-node process group where each contained rank has the same local rank. For example, given each row is a distinct node: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 This API would return inter-node process group [0, 8], [1, 9], [2, 10], and so forth depending on the process's rank. For example, rank 1 would get [1, 9], rank 5 would get [5, 13]. N)ranksbackendz. expected to assign inter-node pg, but did not)r[ get_backendget_world_sizeget_rankrange new_group) rlrf inter_node_pgsharding_backendr< num_nodes my_local_rank local_rankiranks_for_inter_groupgrps rD_init_inter_node_process_groupr}s M''(<=$$%9:J2IMM"67:NNM01 =B9=M! =M!2 3=M ! nn#8S  &M2  $ /GH $ ! s'B)c.[U5[X54$)a Initialize intra and inter-node process groups and return the ones corresponding to this process's rank. This function can be used to initialize process groups for ``HYBRID_SHARD`` or ``_HYBRID_SHARD_ZERO2`` in FSDP. This function assumes each node has an equal number of CUDA-enabled devices. Returns: Tuple[dist.ProcessGroup, dist.ProcessGroup]: Intra and inter-node process group. )rkr})rlrfs rDrLrLs  '';<&';R rFmoduleignored_modulesignored_statescUbUb [S5eSnUSLnU(a[U5n[US5 O/n[Ub [U5O/S5 [U5S:a'[ US[ R 5(aUnOUn[X5Ul[UURU5Ul [UUR5Ul U$)NzfCannot pass both ignored_modules and ignored_states at the same time. Please just pass ignored_states.TFr) r5list_check_ignored_statesrcrZnn Parameter_get_ignored_modules_ignored_modules_get_ignored_params_ignored_params_get_ignored_buffer_names_ignored_buffer_names)r,rrrignored_parameterspassed_as_ignored_statesignored_states_lists rD_init_ignored_module_statesrs"~'A :  -T9">2148 %4%@D !b%  !# )!,bll ; ;!4 1O1&JE/ E #< #E LrFrc[U5S:XagU(aj[SU55n[SU55nU(d>U(d6[UVs1sHn[U5iM sn[S9n[ SU35egg[SU55(d6[UVs1sHn[U5iM sn[S9n[ SU35egs snfs snf) z Check that the ignored states are uniformly parameters or uniformly modules. We may remove this check in the future if we permit mixing. rNc3V# UHn[U[R5v M! g7frY)rZrrr]r,s rDr_(_check_ignored_states..IsUnUE2<<88nrac3V# UHn[U[R5v M! g7frYrZrModulers rDr_rJsSN5*UBII66Nra)keyzUignored_states expects all nn.Parameter or all nn.Module list elements but got types c3V# UHn[U[R5v M! g7frYrrs rDr_rSsL^E:eRYY//^raz>ignored_modules expects nn.Module list elements but got types )rcrdsortedrPreprr5)rr all_params all_modulesr, sorted_typess rDrr>s >aUnUU SNSS +!N"KN54;N"KQUVL**69 #.zL^LLL!N"KN54;N"KQUVL%( M #L#Ls C"Cignored_params device_idcbSnUb7[U[R5(aUO[R"U5nUc[X5HnURRS;aMUcURnM0URRUR:wdMV[ SURSURR35e U=(d [R R5nURS:Xa [ S5e[R"U5Ul U$)a Determine device handle used for initializing FSDP. If a device is specified by ``device_id``, then returns device handle corresponds to that device type. Otherwise, If the module is already on a non-CPU device, then the device type is that non-CPU device type. If the module is on CPU or meta, then the device type is the current accelerator device. See the :ref:`Accelerators` for details. This method will be called once ignored paramters was determined, as the device handle maybe needed for other initialization. N>cpumetazLFSDP does not support modules with different device types but got params on z and rzOFSDP needs a non-CPU accelerator device, but no accelerator device is detected.) rZtorchdevice_get_orig_paramsrP RuntimeError_C_get_acceleratorr from_devicerM)r,rrrdetermined_deviceparams rD_init_device_handler[s()U\\22 i(   %f=E||  O3 ($)LL!<<$$(9(>(>>&-->-C-C,DE%,,J[J[I\^>.L1J1J1L  ! !U *a -889JKE LrFc[U5Ul0nUR5Hup4[U5nURX#'M X lU$rY)_get_buffer_names _buffer_names named_buffersrdtype_buffer_name_to_orig_dtype)r,rr buffer_namebuffers rD_init_buffer_statersT ,F3E :<%335 ' 4 28,,"/ 6(B$ LrFmixed_precision cpu_offloadlimit_all_gathersuse_orig_paramsbackward_prefetch_limitforward_prefetch_limitcURS:XaVU[R:wa1[R"SU=(d [R S35 [RnO.U[R:Xa[R"S[ SS9 U=(d [R UlU=(d [5Ul Ub5[RRS[UR535 [RR!["S5S :HUlU=(d ['5UlX@lXPl[.R0UlSUl[75Ul[:R<"5Ul[@RB"UR>UU5Ul"SUl#0nXl$Sn Xl%/n Xl&U$) NrHz/FSDP is switching to use `NO_SHARD` instead of z since the world size is 1.zoThe `NO_SHARD` sharding strategy is deprecated. If having issues, please use `DistributedDataParallel` instead.) stacklevelz'torch.distributed.fsdp.mixed_precision.1)'r<r!NO_SHARDwarningswarn FULL_SHARD FutureWarningr.r rrr_log_api_usage_oncestrosenvirongetr_use_full_prec_in_evalrrr_use_orig_paramsrIDLEtraining_state_is_rootr_free_event_queuer[get_debug_level _debug_levelexec_order_utils_ExecOrderData_exec_order_data_unshard_event_fully_sharded_module_to_handle_handleparams) r,r.rrrrrrrrrs rD_init_core_staters 1  0 9 9 9 MMA$C(8(C(CDE''  -55 .77 7  <   0N3C3N3NE+?~/?E" $$5c%:O:O6P5Q R  2B73>  $3z|E/,(--EEN-/E--/E-<< E  EIK#,K)*.GM"$FL LrFc`/nXl/nX l/nX0lSUlSUlSUlU$)NT)_root_pre_forward_handles_pre_forward_handles_post_forward_handles_sync_gradients _comm_hook_comm_hook_state)r,rrrs rD_init_runtime_statersE8:&?#24!535"7 EE!E LrFbackward_prefetchforward_prefetchcXlX lU$rY)rr)r,rrs rD_init_prefetching_staters 0- LrFc[R"U5nU(a+X R:wa[UR5UlU$SUlU$rY)r get_root_meshr8r%rM_fsdp_extension)r,r0 root_meshs rD_init_extensionrsM --k:Iy$6$66 1%2F2F G L!% LrFc~[RUl[5n[ 5UlXl0nX lU$rY)r#FULL_STATE_DICT_state_dict_typerr_optim_state_dict_config_state_dict_config_unshard_params_ctx)r,state_dict_configunshard_params_ctxs rD_init_state_dict_staters;*::E)<)>%=%?E"057 2 LrFrcUHWn[UR5S:XdMSnUR5H upEX%LdM Un O U(de[SUS35e g)z Verify if the parameters are accepted by FSDP. The only restriction now is that the parameter cannot be a scalar tensor (param.shape == []). rrz/FSDP doesn't support scalar parameters. Change z& to a 1D tensor with numel equal to 1.N)rcshapenamed_parametersr5)rrr param_namenameparam_s rD_verify_managed_paramsrsp  u{{ q J & 7 7 9 ?!%J!: :$%KM rFfully_sharded_module param_init_fnsync_module_statesc^[UTRU5 [UTRTR5n[ UTRTR 5upgU(dU(aUb[XTR 5 OJU(a#[UUTR TR5 O U(a[R"UU4SjS9 TR VV s1sHnUR5Hn U iM M n nn [UTRU U5 [UTRUTRTR5Tl[[!UTR55n [#X5 U(a@[%XTR&5 TR([*;a[%XTR,5 [/TX5 T$s sn nf)zHInitialize a ``FlatParamHandle`` from a module ``fully_sharded_module``.cJ>[U5SL=(a UTR;$rY)rr) submoduler,s rD0_init_param_handle_from_module..Ls('=i'HD'P(8!7!77(8rF)check_fn)_check_single_device_moduler_get_device_from_device_idr:rM_need_to_materialize_moduler_materialize_with_param_init_fn_materialize_meta_moduler)materialize_modulebuffers_move_module_to_device_get_compute_devicecompute_devicerrr_sync_module_params_and_buffersr-r.r6r=_init_param_handle_from_params) r,rrrrdevice_from_device_idis_meta_moduleis_torchdistX_deferred_initignored_modulerignored_buffersmanaged_paramss ` rD_init_param_handle_from_moduler-s 4e6K6KYW65::u333Ne33U5K5K3/N 5=;T' 1G1G     " "   %(( 8 $444N$,,.F . 4    /    E*+?AVAVWXN/@' %2E2E   " "&@ @ +$e6J6J #5.O L?s$Gc[U5S:Xag[UUUR[URUR R URRURRURRURURURS9 nUR5 UR(aeUR R#UR$5 X0lX0R&UR('[*R,"S5nUR R (a-UR$R,U:waUR/U5 ggg)Nr)fsdp_extensionr)rcrrSHARDING_STRATEGY_MAPr.roffload_paramsr param_dtype reduce_dtypekeep_low_precision_gradsr-rrshardrrappend flat_paramr_fully_sharded_modulerr flat_param_to)r,rrhandle cpu_devices rDrrrs!  6{a  e556 (( )) ** 66  ,, F LLN}}  LL))*MJP))&*F*FGe$J ''F,=,=,D,D ,RZ(-S'rF root_modulerc zSnUb [U5O [5nUHan[U[R R 5(d[US[U53-5e[U5(dMX[S5e UR5H1n[R"U5(aM URU5 M3 UVVs1sH>nUR5H&n[U[R5(aM$UiM( M@ nnnX;a[R "SW35 UR5HAn[U5n U cM[#U S5(deUR%U R&5 MC U$![a n[US[U53-5UeSnAff=fs snnf)aP Check that ``_ignored_modules`` is an iterable of ``nn.Module`` s without any FSDP instances. Return the modules contained in their module subtrees as a :class:`set`. Nested FSDP instances are excluded, but their already-computed ignored modules are included. ``_ignored_modules`` represents the argument passed by the user to FSDP. z>`ignored_modules` should be an iterable of `torch.nn.Module`s Nzbut got zbut got an iterable with z1`ignored_modules` should not include FSDP moduleszTrying to ignore the top-level module passed into the FSDP constructor itself will result in all parameters being ignored and is not well-supported: r)set TypeErrorrPrZrrrrr5modulestraversal_utils _composableadd fsdp_fileFullyShardedDataParallelrrhasattrupdater) r'r msg_prefixignored_root_moduleserchildrroptional_fsdp_states rDrrsRJQ%5%AC !su  '&%((//22J+DT&\N)SST T !& ) )PQ Q '%%'**622 $ $V ,(+*F^^%E%!C!CD % * %  228 ; !((* 4Y?  *.0BCC CC  " "#6#G#G H + I Q x5E0F/G%HHIqPQ$s#F 7F7 F7 F4F//F4rc[5nUVVs1sH.oDR5Hn[U5(aMUiM M0 nnnURU5 Ub5UVs1sHn[U5(aMUiM nnURU5 UR 5HAn[ U5n U cM[ U S5(deURU R5 MC U$s snnfs snf)z Return the parameters of the modules in ``ignored_modules`` and the parameters in ``ignored_parameters``. :class:`FlatParameter` s are excluded from the result. r)r) parametersrr2r+rr1r) r'rrall_ignored_paramsmpparams_in_ignored_modulesparams_in_ignored_parametersrr7s rDrrs36%#!"aLLNqBTUVBWN?!78%)( )!1CA1FA) %(  !!">?!((* 4Y?  *.0ABB BB  % %&9&I&I J + '!( s'C( C( C.8C.c [5nUVVs1sHo3R5HoDiM M nnnURUR5VVs1sHupdXE;dM [ U5iM snn5 UR 5HAn[ U5nUcM[US5(deURUR5 MC U$s snnfs snnf)z6Return the cleaned buffer FQNs in ``ignored_modules``.r) r)r r2rrr+rr1r) r'rall_ignored_buffer_namesr;rbuffers_in_ignored_modulesrrr7s rDrrs *-("'1iikFk"##(3'@'@'B 'B# 3 + k *'B !((* 4Y?  *.0GHH HH $ + +,?,U,U V + $#'"  s"C C "C cjUR5VVs1sHup[U5iM snn$s snnf)zrReturn the fully prefixed names of all buffers in the module hierarchy rooted at ``root_module`` as a class:`set`.)rr)r'rrjs rDrr s8>I=V=V=X =X>;+&=X  s/c[X5Vs1sHo3RiM nn[U5S:Xa)[R"S5U;aUc [ S5eg[U5S:a[ SU35egs snf)z Raise an error if ``module`` has original parameters on multiple devices, ignoring the parameters in ``ignored_params``. Thus, after this method, the module must be either fully on the CPU or fully on a non-CPU device. rWrNzTTo support a module with both CPU and GPU params, please pass in device_id argument.rHz;FSDP only supports single device modules but got params on )rrrcrr)rrrrdevicess rDrrs*:&)QR)Q||)QGR 7|qU\\%0G;  5   W I' S   SsA;r: device_handlec tUcg[U[R5(aUO[R"U5nURS:walURc_[ R "SUSUSUR5SURS3 5 [R"UR55nU$)z Return a ``torch.device`` for the specified ``device_id``. Processes ``device_id`` and returns either the corresponding device or ``None`` if ``device_id`` is ``None``. Nrz"FSDP got the argument `device_id` z on rank zJ, which does not have an explicit index. FSDP will use the current device z6. If this is incorrect, please explicitly call `torch.zk.set_device()` before FSDP initialization or pass in the explicit device index as the `device_id` argument.)rZrrrPindexrrcurrent_device)rr:rErs rDrr.s 5<<88 ell9>U {{e 4 0 9f00=0L0L0N/OPCCI;;-P1 1 m::<= MrFc2[[X55n[SU55nUR5H.nXR;aM UR SS9HnXFR -nM M0 U(+=(a [ =(a [SU55nXG4$)z Return if ``module`` has parameters on meta device and if ``module`` is using torchdistX deferred initialization. At most of the returned bools can be ``True``. If either is ``True``, then ``module`` needs to be materialized. c38# UHoRv M g7frY)is_metar]rs rDr_._need_to_materialize_module..XsCN5NsFrecursec3N# UHn[R"U5v M g7frY)r*is_fakerLs rDr_rMds@ U##s#%)rranyr+r rK_TORCHDISTX_AVAIL)rrrrrrbufrs rDrrKs*6BCNCNCCN^^%  ' $$U$3C kk )N4&  A  A @@ @  66rFc[U5(d[SUS[U535e[X5nUH nU"U5 M g)Nz Expected z to be callable but got )callabler5rP_get_modules_to_materialize)r'rrmodules_to_materializers rDr r isR M " " &>tM?R>S T  9V(f)rFrc RU=(d$ [R"UR55n[X5nSn[R"5 UHtn[ R "URSS9URSS95n[[U55S:nU(dMTURUSS9 UR5 Mv SSS5 g!,(df  g=f![a6n [R"S[!U 5S[#U5S35 U eSn A ff=f)NFrNr)rrOzIUnable to call `reset_parameters()` for module on meta device with error z(. Please ensure that your module oftype z* implements a `reset_parameters()` method.)rrrHrWno_grad itertoolschainr9r rcrto_emptyreset_parameters BaseExceptionrrrrP) r'rrrEmaterialization_devicerXrmodule_state_iterhas_module_statesr5s rDr r ws3ell$$&79V F]]_0%.OO%%e%4fnnUn6S%!%(->(?$@1$D!$$OO+A5OQ++-1__   !!$Q)L>!K M  s<C&AC($C C& C#C&#C&& D&01D!!D&cJ/n[R"U/5nU1nU(aUR5nURU5 UR 5HAnXd;dM [ U5bMXa;dMUR U5 URU5 MC U(aMU$rY) collectionsdequepopleftr!childrenrr.)r'rrXqueuevisited_modulesr child_modules rDrWrWs /1   {m ,E'2mO %%f-"OO-L3*<8@ 7##L1 \*. % "!rFrc^ [R"S5m UGb"[R"5nUR U5 /n/nU(aUR 5nUR U 4SjURSS955 UR U 4SjURSS955 UR5H5n[U[R5(aM$UR U5 M7 U(aMUV s/sH oU;dM U PM n n UV s/sH oU;dM U PM n n [XU5 g[[X5S5n U bU RT :Xa [!5 gggs sn fs sn f)a Move ``module`` depending on ``device_from_device_id`` and its current device. This includes moving ignored modules' parameters. - If ``device_from_device_id`` is not ``None``, then this moves ``module`` to the device. - If ``device_from_device_id`` is ``None``, then this does not move ``module`` but warns the user if it is on CPU. Precondition: ``_check_single_device_module()``. rNc3L># UHnURT:XdMUv M g7frYr)r]rr&s rDr_)_move_module_to_device..s&BE<<:-B$ $FrNc3L># UHnURT:XdMUv M g7frYrm)r]rr&s rDr_rns&@F==J.@ro)rrrdrer!rfextendr9r rgrZr/r0_move_states_to_devicenextr_warn_cpu_init)rrrrrhrr  curr_modulerr<params_to_move bufs_to_moverr&s @rDr r sV$e$J(/:.?.?.A V%'&(--/K MM(33E3B  NN)11%1@  )113 !)Y-O-OPPLL+4!e&&,GV/F!VG#*G7a.F7 G~=RS !&94 @E U\\Z78 HGs E3E3 E8)E8r c\[U5S:Xa[U5S:Xag[U5S:aUSRnO[U5S:aUSRn[R"S5nUbUHnn[R"5 UR U5UlUR b*UR R U5UR lSSS5 Mp UHnUR U5UlM gWU:Xa [5 gg!,(df  M=f)z Move states to the specified device. Precondition: ``_check_single_device_module()`` and module's parameters and buffers have been materialized if needed. rNr)rcrrrZtodatagradrt)rr rrHr&rrs rDrrrrs 6{aCLA- 6{Q)) W  **e$J(E"XX&;< ::)&+jjmm4I&JEJJO! F ))$9:FK : % & !s AD D+ c0[R"S5 g)NamThe passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.)rrrFrDrtrts MM 1rFc[[X5S5nUb'URRS:wa URnO$[R"UR 55nUbXb:wa[ SUSUSU35eU$)a Determine and return this FSDP instance's compute device. If the module is already on a non-CPU device, then the compute device is that non-CPU device. If the module is on CPU, then the compute device is the current device. Since this method should be called after materializing the module, any non-CPU device should not be meta device. For now, the compute device is always a CUDA or CUDA-like device with its explicit index. Precondition: ``_check_single_device_module()`` and ``_move_module_to_device()``. Nrz4Inconsistent compute device and `device_id` on rank z: z vs )rsrrrPrrHr5)rrrr:rErrs rDrrs* !&94 @E U\\..%7m&B&B&DE(^-TB4&d#8"9 ;   rFc/nUR5Hn[U[S5(aM[U[S5 UR 5n[ U5(a@UR 5upgUVs/sHn[XX5PM n nURU 5 MURU5 M UHtn U R 5n [ U 5(a@U R 5upgUVs/sHn[X5PM n nURU 5 McURU 5 Mv [U5 [UU[SS9 gs snfs snf)z Synchronize module states (i.e. parameters ``params`` and all not-yet-synced buffers) by broadcasting from rank 0 to all ranks. Precondition: ``sync_module_states == True`` and ``self.process_group`` has been set. FTr)srcN) r getattr FSDP_SYNCEDsetattrdetachr'__tensor_flatten__rqr!+_check_module_states_for_sync_module_statesr&PARAM_BROADCAST_BUCKET_SIZE) rrr- module_statesrdetached_bufferattrsrjattr inner_buffersrdetached_param inner_paramss rDrr4s#)+M.."v{E22 FK .$mmoO,_==+==?LQ RED!?E R$$]3$$_5# ( 8 8%88:HEFKLedGN9eLL   .   00 ># !SMs 5E /ErcXU(a#[SU55(a [S5egg)Nc3f# UH'oR[R"S5:Hv M) g7f)rN)rr)r]tensors rDr_>_check_module_states_for_sync_module_states..cs#;H e,,=s/1zThe module has CPU parameters or buffers when `sync_module_states=True`, which requires them to be on GPU. Please specify the `device_id` argument or move the module to GPU before passing it to FSDP.)rRr5)rs rDrr`s<;H C  }rFc## UR5n[U5nX1;a[U5(dUv M&![a gf=f7f)a, Return an iterator over the original parameters in ``module``. The iterator does not return the parameters in ``ignored_params``, any ``FlatParameter`` s (which may be present due to nested FSDP wrapping), or any original parameters already flattened (only relevant when ``use_orig_params=True``). N)r9rsr StopIteration)rr param_genrs rDrrmsT!!#I OE*3Ee3L3L     s A '; AA AA c [U5HHup#X1;dM [U5(aM[SUSUR5SUR35e g)a Check that original parameters in ``fsdp_module`` have been flattened. The flattened parameters are made invisible to ``named_parameters()`` for the module hierarchy rooted at ``fsdp_module``. This should be called as a sanity check after flattening the wrapped module's parameters. z Found an unflattened parameter: z;  N)rrrr; __class__) fsdp_modulerrrs rD_check_orig_params_flattenedrsY?{K  &/A%/H/H2:,b::<.%//!24 LrFcjU[R:Xa[R$[R$rY)r!rr allreduce_hookreduce_scatter_hook)r.s rD_get_default_comm_hookrs3  0 9 9 9 $$ . .rFc*[R"US9$)NrI)r r>rIs rDrQrQs  % %M BBrFrY)rdr[rrcollections.abcrrrtypingrrrr r r rtorch.distributed distributedr[(torch.distributed.fsdp._exec_order_utilsfsdp_exec_order_utilsr'torch.distributed.fsdp._traversal_utils_traversal_utilsr,2torch.distributed.fsdp.fully_sharded_data_parallelfully_sharded_data_parallelr/torch.nnr(torch.distributed.algorithms._comm_hooksr torch.distributed.device_meshr r"torch.distributed.distributed_c10dr$torch.distributed.fsdp._common_utilsrrrrrrr"torch.distributed.fsdp._flat_paramrrrr%torch.distributed.fsdp._limiter_utilsrtorch.distributed.fsdp.apirrrrr r!r"r#torch.distributed.fsdp.wrapr$&torch.distributed.tensor.parallel.fsdpr%torch.distributed.utilsr&torch.utils._python_dispatchr'torch.utils.hooksr(rS torchdistxr)r* ImportErrorintrrrbr\HybridShardProcessGroupTypeProcessGroupTyperr SHARD_GRAD_OP HYBRID_SHARD_HYBRID_SHARD_ZERO2rr6#NO_RESHARD_AFTER_FORWARD_STRATEGIESrEr7boolrOrJrkr}rLrrrrrr)rrrrrrrrrrrrrrrrrrrr r rWTensorr rrrtrrrrrrr>rQr}rFrDrs  99OO CCAAFFBEA B   0D<F1.""34 #D$5$5t7H7H$HIE$"3"35P"PQR 5>>!7!B!B""$:$H$H!!#9#F#F((*@*T*T !!(( ""(('# )- 0 0#0(0 W  0 *% 0  00f( (#(( ((V#$IJI4II  ARAR  !++!! !!H++ 4  d// /0& + + II+huxx78+%((,,-.%((//9R0SS +++\I9= :- - II- %-c5<</01 -  --`  II"> > 01>n->*% >  >  >!> >>>B         '     : J *  *299d2<<6HT&A A))Ac5<</01AHbii[$%678 A  A  AAH) )  )))))<66x896 ^6xBF)!%((*<*    D$$)$ X$:299S II  % c5<</01   <c5<</01 %ell :7 II7 %7^7 4: 7<  RYYK-. ^    #ELL1 ^ % F""-0^" "))_",3 II3 %3&3$ELL1 3  3l   %,, $ELL1 @ II %$ELL1   %   \\ F) II)  )$$) )X   %      II  % bll , % (.>C$$CCO#s__ _