sh SSKrSSKrSSKrSSKrSSKrSSKrSSKJrJr SSKJ r SSK J r SSK J r Jr SSKJr SSKJrJrJrJrJrJr SSKJr SSKrSSKJr SSKJr SSK Js J!r" SS K#J$r$ SS K%J&r& SS K'J(r(J)r)J*r+ SS K,J-r- SS K.J/r/J0r0 SSK1J2r2 SSK3J4r4J5r5J6r6 SSK7J8r8 SSK9J:r:J;r;Jr>J?r?J@r@ SSKAJBrBJCrCJDrDJErE SSKJFrFJGrG SSKHJIrJ SSKKJLrLJMrMJNrNJOrO SSKPJQrQJRrRJSrSJTrTJUrU SSKVJWrW SrX\S(aSrYSrZ\RR5rXO8\T(aSrYSrZO,\U(aSrYSrZ\RR5rXOS rYS!rZS"rX"S#S$\5r^"S%S&\5r_"S'S(\R\5raS)\RS*\RS+\4S,jrcS-rdStS)\RS.\e4S/jjrfSuS0jrgS1rhS2riSvS)\RS3\e4S4jjrjS)\RS5\e4S6jrkS)\RS7\e4S8jrl"S9S:5rm"S;S<\a5rn"S=S>\a5ro"S?S@\o5rp"SASB\o5rq"SCSD\a5rr"SESF\r5rs"SGSH\R5rt"SISJ\o5ru"SKSL\R5rv"SMSN\R5rx"SOSP\R5ry\RSQ\4SRj5r{\RSS\4STj5r|\RSU\4SVj5r}\\RSW\4SXj55r~\\RSY\4SZj55r\\RS[\4S\j55r\\RS]\4S^j55rS_\S+\S`\Sa\4SbjrSwSc\RSd\RSe\\Sf44Sgjjr"ShSi\M5r"SjSk\L5rSxSl\\4Smjjr"SnSo\R5r"SpSq\R5r"SrSs\R5rg)yN)ABCabstractmethod) nullcontext)deepcopy)autoEnumwraps)AnyCallablecast no_type_checkOptionalUnion)mock) checkpoint) DeviceMesh) CPUOffload fully_shardFullyShardedDataParallel) TrainingState)FSDPParamGroupRegisterPostBackwardFunction)#NO_RESHARD_AFTER_FORWARD_STRATEGIES)BackwardPrefetchMixedPrecisionShardingStrategy)ShardedGradScaler)always_wrap_policyModuleWrapPolicywrap)distribute_tensorDTensorShard)ColwiseParallelparallelize_moduleRowwiseParallelSequenceParallel)TransformerDecoderLayerTransformerEncoderLayer)DistributedDataParallel)MultiProcessTestCaseMultiThreadedTestCase run_subtests TEST_SKIPS) FILE_SCHEMAget_cycles_per_ms TEST_CUDATEST_HPUTEST_XPU) has_tritoncudancclzhpu:0hcclxpuxcclcpuglooc0\rSrSr\"5r\"5rSrg) FSDPInitModeVN)__name__ __module__ __qualname____firstlineno__rNO_FSDP RECURSIVE__static_attributes__rBw/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/testing/_internal/common_fsdp.pyr@r@VsfGIrJr@c>\rSrSr\"5r\"5r\"5rSrg)DEVICEInitMode_rBN) rCrDrErFr DEVICE_BEFORE DEVICE_AFTER DEVICE_NEVERrIrBrJrKrMrM_sFM6L6LrJrMc\rSrSrSr\S\\RS44Sj5r \S\R4Sj5r \S Sj5r \ \S \ S \ S\R4S j55rS rg) FSDPTestModelhzVThis defines the interface expected from all models used commonly for FSDP unit tests.return.cg)z+Returns an input for the model as as tuple.NrBselfdevices rK get_inputFSDPTestModel.get_inputl rJcg)z,Returns the loss given the input and output.NrB)rXinputoutputs rKget_lossFSDPTestModel.get_lossqr\rJNcg)z) named_parametersdetachr< named_buffersdistget_world_sizerangeall_gather_objectzip)rtrurv param_nameparamnamed_module_states buffer_namebuffer world_size_olist rank0_statesstatep1p2s rK_assert_module_statesrs%"'!7!7!9!9 J \\^'')*!9#(#6#6#8#8 K mmo))+,#8$$]3J , -,aT,E -5]K8L  ## #qr   #L 8 GQWa b !9 .s+D +D* Dc6[R"[5$N)rorY DEVICE_TYPErBrJrK get_devtypers << $$rJ zero_bufferscU(a[R"U5O [5nU UR5H1n[R "5 UR 5 SSS5 M3 U(aEUR5H1n[R "5 UR 5 SSS5 M3 SSS5 g!,(df  M=f!,(df  M_=f!,(df  g=f)zBZeros the parameters and optionally buffers of ``model`` in place.N)FSDPsummon_full_paramsr parametersrono_gradzero_buffers)rtr summon_fullctxrrs rK _zero_modelrs -8$ ! !% ([]C %%'E !( --/]]_LLN%_*  %_ s;)C)C$;C)C0 C) C C) C& !C)) C7cU(dUR[5nU(aUR5 UR5$r)torhalf state_dict)rt cpu_offloadrs rK_get_state_dictrs. %     rJc pSRUVs/sHo"bU[U5OSPM sn5$s snf)Nrnone)joinstr)test_name_mappingrgss rK subtest_namers9 88IMNAm 3q6 " ?N Ns3cUR5H=up#UR[R"S5:wdM+UR5X'M? US:XaUOS/n[R "U5 [ [[[R4US5nUR5HnXR[5X'M U$)Nr<r) itemsrYror<r}broadcast_object_listr dictrrpkeysrr)rankrrrrs rK_broadcast_state_dictrs(--/ <<5<<. .%*YY[J "0 19Z$ /Eu%d3 ,-uQx8J oo' !+!7!:!:;!G ( rJrecursec[R"XS9 [[UR 555sSSS5 $!,(df  g=f)a? Returns the full unsharded parameters of ``model``. Any FSDP-managed parameters offloaded to CPU are moved to GPU in the returned list. Args: recurse (bool): If ``False``, only unshards the parameters immediate to ``model``; if ``True``, recurses through the module hierarchy rooted at ``model``. )rN)rrrlistr)rtrs rKget_full_paramsrs4   8U--/01 9 8 8s "A Amove_to_devicec>U(aUR[5$U$r)rr)rtrs rK_move_to_devicers$2588K ==rJ wrap_fsdpc2U(dU$[U/UQ70UD6$rr)rtrrgrhs rK_maybe_wrap_fsdprs!5CtE'CD'CF'CCrJcH\rSrSrS\S\4SjrS\4SjrS\4SjrSrS r g ) DummyProcessGrouprsizecXlX lgr_rank_size)rXrrs rK__init__DummyProcessGroup.__init__s   rJrUcUR$r)rrXs rKrDummyProcessGroup.rank zzrJcUR$r)rrs rKrDummyProcessGroup.sizerrJcB[R"5nSnXClU$)Ncd[RR5nURS5 U$)Nr>)rofuturesFuture set_result)futures rK get_future/DummyProcessGroup.allreduce..get_futures'+0==+?+?+AF   a MrJ)rMockr)rXrgrh dist_waitrs rK allreduceDummyProcessGroup.allreduces IIK   *rJrN) rCrDrErFintrrrrrIrBrJrKrrs2Scc rJrc^\rSrSrS\R S\S\S\4U4SjjrSr Sr S r S r \ SS\R S \S\S \\\\4S\S\S \\R*\44Sjj5rSrSrU=r$)TransformerWithSharedParamsirydevice_init_modeadd_bn deterministicc>[TU]5 UR5UlUR5UlU(a[ R "S5 SnSn[R"XV5Ul [R"USSSSS9Ul [R"Xe5Ul URRURlURSURRR!U455 URS [ R""UR$[ R&S 95 SUlU(a)[ RR+UR(5O[ RR-5UlU[0R2:XaUR5[65nU(aUR95 gg) Nrg?)d_modelnum_encoder_layersnum_decoder_layersdim_feedforwarddropout vocab_bias long_buffer)dtype)superrrrrro manual_seedrr Embedding embed_tokens Transformer transformerLinear output_projweightregister_buffernew_ones zeros_likerlongbs BatchNorm1dIdentitybnrMrOrreval)rXryrrrd_vocabr __class__s rKr$TransformerWithSharedParams.__init__s] JJL **,    a LL:>>    99W6#'"3"3":":  $++22;;WJG      T__EJJ ? 39%((&&tww/uxx?P?P?R ~;; ;77;'D  IIK rJc$[R"SUR-5 [R"SUS9R SUR 5n[R"UR S-US9R SUR 5nX#4$)Nr> rYr6)rorrarangeviewr)rXrYsrctgts rKrZ%TransformerWithSharedParams.get_input.sj !dii-(ll2f-221dgg>ll477Q;v6;;AtwwGzrJcURU5nX0R-URRU5-nURU5nUR U5nUR X45nUR U5$r)rrrtype_asrrr)rXsrc_idstgt_idsrrxs rKforward#TransformerWithSharedParams.forward4sr(OO#d&6&6&>&>s&CC(ggcl   S &""rJcUup4[RRURSUR S55URS5SS9$)Nsum) reduction)rr functional cross_entropyrr)rXr^r_rrs rKr`$TransformerWithSharedParams.get_loss<sG}}** KKFKKO ,chhrle+  rJc$UR5 grbackwardrcs rKre(TransformerWithSharedParams.run_backwardB  rJfsdp_init_mode fsdp_kwargsrUcxUc0nU[R:Xa)[U[5(aUSnOUn[ XbXT5$U[R :XaSU;a[ [[15nOURS5nSU;a?US[R[R1;a[U[5(dSnOUn[U[5(aUSn OUn [ XXT5n [U U4SU0UD6n U[R:XaU R!["5n U $[%SU35e)a Initializes a :class:`TransformerWithSharedParams` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps with top-level FSDP. By default, the top-level FSDP uses the ``ModuleWrapPolicy`` for encoder and decoder layers, but a different auto wrap policy may be specified via ``fsdp_kwargs``. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. add_bn (bool): Whether to include batch norm in the model. Nrauto_wrap_policysharding_strategyUnsupported FSDP init mode: )r@rG isinstancernrrHr r*r)popr HYBRID_SHARD_HYBRID_SHARD_ZERO2rrMrPrr ValueError) ryrrrrrpgr fsdp_pg tformer_pgm fsdp_models rKrj TransformerWithSharedParams.initEsW6  K \11 1%''1X.f |55 5!4#3//$ $/??3E#F ${2 34$113C3W3WXY"5%00%''"1X " +fA"2 J  >#>#>>']];7  77GHIIrJcUR/$r)rrs rKget_ignored_modules/TransformerWithSharedParams.get_ignored_moduless  !!rJ)rrrrrrr)NFT)rCrDrErFr} ProcessGrouprMboolrrZrr`rerqr@rrrr rrrrsrrjr/rI __classcell__rs@rKrrs(  ()( (  (T #  15# KJ  KJ$KJ)KJd38n- KJ  KJ  KJ ryy$ KJKJZ""rJrc^\rSrSrS\R S\S\S\4U4SjjrSr Sr S r S r \ SS\R S \S\S \\\\4S\S \R(4 Sjj5rSrU=r$)NestedWrappedModuleiryrrrc >^^^[TU]5 TR5UlTR5UlU[ R :HnUUU4SjnU(a[R"S5 [R"[[R"SS5U5U"[R"U"[[R"SS5U55[[R"SS5U555U"[[R"SS5U55[[R"SS5U55Ul g)Nc0>T(a [UT40TD6$U$rrlayerrryrs rK _maybe_wrap1NestedWrappedModule.__init__.._maybe_wrapE58K88LrJrrr6r)rrrrrrMrOrorrr Sequentialrrmodule rXryrrrrrr;rs `` ` rKrNestedWrappedModule.__init__s JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   "a(8.I J BIIaO^ <   rJcv[R"SUR-5 [R"SSUS94$)Nr>r6rr)rorrrandrWs rKrZNestedWrappedModule.get_inputs. !dii-( 1a/11rJc$URU5$rr?rXrs rKrNestedWrappedModule.forward{{1~rJc&UR5nU$r)rrXr^r_rds rKr`NestedWrappedModule.get_lossszz| rJc$UR5 grrrcs rKre NestedWrappedModule.run_backwardrrJrrrUcUc0nU[R:Xa [USUUS9$U[R:Xa;[U4SUUS.UD6nU[R :XaUR [5nU$[SU35e)a Initializes a :class:`NestedWrappedModule` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. FrrrTr") r@rGr6rHrMrPrrr')ryrrrrr,s rKrjNestedWrappedModule.inits.  K \11 1&!1+   |55 5,!1+   J >#>#>>']];7  77GHIIrJr?rrNF)rCrDrErFr}r1r2rMrrZrr`rerqr@rrrr rrrsrjrIr3r4s@rKr6r6s     )    @2 15# +J  +J$+J)+Jd38n- +J  +J  +J+JrJr6c v^\rSrSr\S S\R S\S\S\ \ \ \ 4S\ 4 U4Sjjj5rSrU=r$) AlwaysWrapNestedWrappedModuleiryrrrrc:>[[[] U[RUUUS9nU[R:XaU$U[R :XaHU=(d 0n[ U4S[0UD6nU[R:XaUR[5nU$g)z Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, for the ``RECURSIVE`` init mode, this wraps with top-level FSDP and the ``always_wrap_policy()`` auto wrap policy. )ryrrrrr N) rrUrjr@rGrHrrrMrPrr)ryrrrrrtr,rs rKrj"AlwaysWrapNestedWrappedModule.inits )+H '//-#'    \11 1L |55 5%+KeX6HXKXJ>#>#>>']];7   6rJrBrS)rCrDrErFrqr}r1r@rMrrrr r2rjrIr3r4s@rKrUrUs^ 15#   $)d38n-   rJrUc ^\rSrSrS\R S\S\S\4U4Sjjr\ S Sj5r \ S S\R S\ S\S \ \ \\4S\4 S jj5rS rU=r$)NonUniformReqGradNWMiryrrrc >^^^[[U] 5 TR5UlTR 5UlU[ R:HnUUU4SjnU(a[R"S5 [R"[[R"SS5U5U"[R"U"[[R"SS5U55[[R"SS5U555U"[R"[[R"SS5U5[[R"SS5U5555Ulg)Nc0>T(a [UT40TD6$U$rrr9s rKr;2NonUniformReqGradNWM.__init__.._maybe_wrap+r=rJrrr6r)rr6rrrrrMrOrorrrr>rrr?r@s `` ` rKrNonUniformReqGradNWM.__init__s !413JJL **,)^-I-II     a mm BIIaO^ <   !R0@. QR#BIIb"$5~F   #BIIb!$4nE#BIIaO^D   rJcUR5H3up#[R"X5(aM"URS5 M5 grS)rzrematchrequires_grad_)rt req_grad_masknps rK_set_nonuniform_req_grad-NonUniformReqGradNWM._set_nonuniform_req_gradBs4**,DA88M--  '-rJrrc[R"S5nU[R:Xa#[ USUUS9n[R Xe5 U$U[R :XaUUc0n[ U4SUUS.UD6nU[R:XaUR[5n[R Xu5 U$[SU35e)a Initializes a :class:`NestedWrappedModule` instance, but unlike :meth:`NestedWrappedModule.init`, it wraps a second :class:`torch.nn.Sequential` container to enable the desired non-uniform ``requires_grad`` ``use_orig_params=True`` tests. For both ``RECURSIVE`` and ``NO_FSDP`` init modes, freezes all parameters except the last two to validate ``ShardedGradScaler`` support for ranks with no (non-zero sized) local shards in FSDP ``use_orig_params=True`` mode. zmodule\.2.*\.1.*FrPTr") r_compiler@rGrYrerHrMrPrrr')ryrrrrreq_grad_pattern ddp_modelr,s rKrjNonUniformReqGradNWM.initHs(::&9: \11 1,!1+ I ! 9 9) V  |55 5" -!1+   J >#>#>>']];7 9 9* W 77GHIIrJrRrlrS)rCrDrErFr}r1r2rMrrqrer@rrrr rjrIr3r4s@rKrYrYs(   ( ( ) (  ( T((  15# +J  +J$+J)+Jd38n- +J  +J+JrJrYc ^\rSrSrSrS\R S\S\4U4SjjrSr Sr S r S r \ S \\S \S\S\S \4 Sj5rSrU=r$)ModuleWithDelayiwzThis class wraps a :class:`FSDPTestModel` to optionally add a delay after computing the loss and/or before the gradient reduction.r?delay_after_loss_msdelay_before_reduction_mscF>[TU]5 X lX0lXlgr)rrrnror?)rXr?rnrors rKrModuleWithDelay.__init__{s! #6 )B& rJc8URRU5$r)r?rZrWs rKrZModuleWithDelay.get_inputs{{$$V,,rJc$URU5$rrFrGs rKrModuleWithDelay.forwardrIrJcbURRX5nURS:a[(d [(a%[ R "URS- 5 U$[(a=[RR[UR[5-55 U$Nr) r?r`rnr3r4timesleepr2ror7_sleeprr1rKs rKr`ModuleWithDelay.get_losss}{{##E2  # #a 'x88 433d:;  !!#d&>&>ARAT&T"UV rJc^^[RRmUU4Sjn[R"SU5 TR R U5 SSS5 g!,(df  g=f)Nc8>TRS:a[(a>[RR [ TR[ 5-55 O9[(d [(a#[R"TRS- 5 T"U0UD6$rw) ror2ror7r{rr1r3r4ryrz)rgrhorig_reduce_scatterrXs rK_delayed_reduce_scatter=ModuleWithDelay.run_backward.._delayed_reduce_scattersq--19JJ%%D::=N=PPQXJJt==DE&77 7rJz'torch.distributed.reduce_scatter_tensor)ro distributedreduce_scatter_tensorrpatchr?re)rXrdrrs` @rKreModuleWithDelay.run_backwardsR#//EE 8ZZ 57N  KK $ $T *   s A  A. module_class model_args model_kwargsc<[UR"U0UD6UU5$)a Args: module_class (Type[FSDPTestModel]): Wrapped module class to which to add delays. model_args: Positional arguments forwarded to the ``module_class`` ``init()``. delay_after_loss_ms (int): Delay after computing the loss/before the optimizer step (in ms). delay_before_reduction_ms (int): Delay before reduce-scattering gradients (in ms). model_kwargs: Keyword arguments forwarded to the ``module_class`` ``init()``. )rmrj)rrnrorrs rKrjModuleWithDelay.inits**   z :\ :  %  rJ)rnror?)rCrDrErFrmrrrsrrrZrr`rerqtyperSr rjrIr3r4s@rKrmrmwsF   ! $' -+$ =)  ! $'     rJrmc\rSrSr\\R SSSS4S\RS\ S\S\ \ \ \ 4S \S \S \4S jj5rS rg)NestedWrappedModuleWithDelayiNFrryrrrrrnroc >[R[UUUUUUUS9$)Nryrrrrrnro)rmrjr6rs rKrj!NestedWrappedModuleWithDelay.inits4## )-#' 3&?$  rJrB)rCrDrErFrqrMrPr}r1r@rrrr r2rrjrIrBrJrKrrs,:+F+F04##$)*    $ ) d38n-    !  $'  rJrc.^\rSrSrU4SjrSrSrU=r$)DummyDDPic.>[TU]5 Xlgr)rrr?)rXr?rs rKrDummyDDP.__init__s  rJc&UR"U0UD6$rrFrXrgrhs rKrDummyDDP.forwards{{D+F++rJrFrCrDrErFrrrIr3r4s@rKrrs,,rJrc^\rSrSrS\R S\S\S\S\4 U4Sjjr Sr S r \ SS\R S \ S\S \\\\4S\S\4 S jj5rS rU=r$)MixtureOfExpertsiryrrdelay_before_free_msrc >[TU]UUUUS9 XlX@lX lU[ R :HUlU(a#[R"SUR-5 SnSnSn [[R"Xx5UR5n [SU R555UlU R5H n SU lM U(a[R"S5 [[R"X5UR5n U(aF[R$R'UR5/5n [)X40UD6n [)X40UD6n [R*"[[R"X5UR5U U [[R"X5UR55Ulg) N)ryrrr*rrrc3@# UHoR5v M g7fr)numel).0rds rK ,MixtureOfExperts.__init__..s$L8K1WWYY8KTr)rrryrrrMrOrrorrrrrrrrnum_expert_paramsexpertr new_grouprr>r?)rXryrrrrrd_expertd_sharedd_inputrrdshared expert_grouprs rKrMixtureOfExperts.__init__s{ -'   $8!"..2N2NN    b499n - 8!>@S@ST!$$L8I8I8K$L!L""$AAH%    a  8!>@S@ST  ,,66L&>+>F&7;7Fmm BIIg8$:M:M N   BIIh8$:M:M N   rJc^^TRS:aTRSn[U[5(ag[R R RRmUU4Sjn[R"SU5 TRU5sSSS5 $TRU5$!,(df  N=f)Nrrc>[(a>[RR[ TR [ 5-55 O9[(d [(a#[R"TR S- 5 T"U0UD6$)Nrx) r2ror7r{rrr1r3r4ryrz)rgrh orig_reshardrXs rK_delayed_reshard2MixtureOfExperts.forward.._delayed_reshardsc y )) 9 9 {{1~ s B// B=cUR5 UR(d[R"5 UR 5H|n[ US5(aMUR cM%UR RUR5 [RRUR URS9 M~ SSS5 gg!,(df  g=f)Nrrx) rrrorrhasattrgraddiv_rr all_reducery)rXrdrds rKreMixtureOfExperts.run_backward1s ~~*Aq(++ vv) DOO4))44QVV4::4N +! s2C-AC C rrc Uc0nU[R:Xa [USUUUS9$U[R:Xa<[U4SUUUS.UD6nU[R :XaUR [5nU$[SU35e)a Initializes a :class:`MixtureOfExperts` instance. Args: fsdp_init_mode (FSDPInitMode): If ``NO_FSDP``, then does not wrap any modules with FSDP. If ``RECURSIVE``, then wraps some nested modules with FSDP, including the expert and shared layers, but not the top-level module. The model may later be wrapped with a top-level FSDP external to this method if desired. device_init_mode (DEVICEInitMode): Determines model movement to DEVICE. fsdp_kwargs (Optional[Dict[str, Any]]): Optional keyword arguments forwarded to the FSDP constructor. deterministic (bool): Whether to make the model deterministic across constructions. delay_before_free_ms (int): Delay before resharding expert parameters in the forward pass (in ms). F)rrrrTr") r@rGrrHrMrPrrr')ryrrrrrr,s rKrjMixtureOfExperts.init=s4  K \11 1#!1%9+  |55 5)!1%9+   J >#>#>>']];7  77GHIIrJ)rryr?rrr)NFr)rCrDrErFr}r1r2rMrrrrerqr@rrrr rjrIr3r4s@rKrrs2   2 2 ) 2 " 2  2 h0 O 15#$% 0J  0J$0J)0Jd38n- 0J  0J " 0J0JrJrc ^\rSrSrSSSSS.S\S\\RS\S \S \4 U4S jjjjr S \RS \R4Sjr Sr Sr U=r$)MLPiqTFr6)bias with_bufferdim_multiplierdimrYrrrc>[TU]5 [R"XU-X#S9Ul[R"XQ-XUS9UlU(a'UR S[R"U4US95 gSUl g)N)rYrrr) rrrrrin_projout_projrrorandnr)rXrrYrrrrs rKr MLP.__init__rsf yys&:6U  ."6QUV   5;;vf+M NDKrJrrUcURU5n[R"U5nURU5n[R"U5nURbX R-nU$r)rFrelurr)rXrzs rKr MLP.forwardsQ LLO FF1I MM!  FF1I ;; "KKArJcURb4[RRR UR5 ggr)rrorrrjnormal_rs rKreset_parametersMLP.reset_parameterss+ ;; " HHMM ! !$++ . #rJ)rrrr)rCrDrErFrrrorYr2rrprrrIr3r4s@rKrrqs*. ! &    "%,,//rJrcV^\rSrSrSS.S\S\4U4SjjjrS\S\S \S S4S jrSr U=r $) MLPStackiF)with_seq_parallelmlp_dimrc>[USS9[U5[USS9/nU(a$UR[R"USS95 [TU]"U6 X lg)N)rFr)rappendrr LayerNormrrr)rXrrmodulesrs rKrMLPStack.__init__sV  * L  * $   NN2<<e< = '"!2rJtp_meshdp_meshuse_activation_checkpointingrUc [SS9[SS9[SS9[SS9[SS9UR(a[[S5S9O [5S.nUR(a [ SS9US'[ XUS9 UHDn[ U[R5(aM$U(a [U5 [U4S U0UD6 MF [U4S U0UD6 U$) NF)use_local_outputr>)output_layouts)z 0.in_projz 0.out_projz 1.in_projz 1.out_projz 2.in_projz 2.out_proj) sequence_dim3) device_meshparallelize_planmesh) r%r'rr$r(r&r#rrrrr)rXrrrrrr?s rK parallelizeMLPStack.parallelizes)%@)5A(%@)5A(%@%%*qB "    ! !$4!$D S !4GWXF&",,//+6"  [TU]5 [R"X5Ul[R"X5Ul[R "5UlX lgr) rrrrrlin1lin2ReLUrr)rXrrrs rKrDoubleLinear.__init__sA IIc' IIc' GGI !2rJrrUcUR(a@URURU55URURU554$URURU55$r)rrrrrGs rKrDoubleLinear.forwardsQ  ! !99TYYq\*DIIdiil,CC Cyy1&&rJ)rrrrT)rCrDrErFrmrr2rrorprrnrrIr3r4s@rKrrs^ 3C3D33'' uU\\5<</0%,,> ?''rJrnew_all_gather_into_tensorc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)r}all_gather_into_tensorbarrier)rorig_all_gathers rKpatch_all_gatherrsN11OLLN"A!A>"A;;A>new_reduce_scatter_tensorc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)r}rr)rrs rKpatch_reduce_scatterrsO44LLN!:D9  %8" %8"rnew_all_reducec## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)r}rr)rorig_all_reduces rKpatch_all_reducersIooOLLN$DO*  ) )r new_unshardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)runshardr}r)r orig_unshards rK patch_unshardrP"))LLLN(N.  !- !-r new_reshardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rreshardr}r)r rs rK patch_reshardr  rrnew_post_backwardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)r post_backwardr}r)r orig_post_backwards rKpatch_post_backwardrsQ(55LLN#4N :  '9$ '9$r new_backwardc## [Rn[R"5 U[lSv [R"5 U[lg![R"5 U[lf=f7fr)rrr}r)r orig_backwards rK*patch_register_post_backward_hook_backwardr$sS199MLLN,8 )>  0=$- 0=$-rrrgrhc[U5S:aUSnOSU;aUSnO[SUSU35eU"U5 U"U0UD6$)Nrr_z,Cannot get reduce-scatter output from args: z kwargs: )lenAssertionError)clsrrvrgrhr_s rKreduce_scatter_with_assertr1sa 4y1}a V !;D6F8 T  f  / //rJreplicated_modulesharded_moduleprefixes_to_ignore.c[UR5UR55GHuupEupgUnUHn URU S5nM URXH5 UR U[ 5 [ U[ 5(deURURp[U 5[S5[S54:Xa [S5e[XZU 5n URUR5U R55 URcURUR5 GMUR!UR5 [URX5n UR UR[ 5 [ UR[ 5(deURURR5U R55 GM g)NrzmFSDP's (Shard(0), Shard(0)) layout differs from distribute_tensor(), so we cannot check for equality using it)rrzreplace assertEqualassertIsInstancer#r#r placementsrnr$rr"to_localr assertIsNoneassertIsNotNone)rrrrreplicated_namereplicated_param sharded_name sharded_paramclean_sharded_nameprefixrr#sharded_ref_paramsharded_ref_grads rKcheck_sharded_parityr/Ds OR**,n.M.M.OOJ+-Jl*(F!3!;!;FB!G ) < ]G4-1111(44m6N6Nj  q58 4 4 ; ..>jQ  ..02C2L2L2NO  (   ]// 0  M../,-=-B-BDU ]//9-,,g6666  **3357G7P7P7RS1OrJcJ^\rSrSr\S5rU4SjrSrSrSr Sr U=r $)FSDPTestMultiThreadiec[$r DEVICE_COUNTrs rKrFSDPTestMultiThread.world_sizefrJcB>[TU]5 UR5 gr)rsetUp_spawn_threadsrXrs rKr8FSDPTestMultiThread.setUpjs   rJc [U/UQ70UD6$rr.rs rKr. FSDPTestMultiThread.run_subtestsnD242622rJc@[RR5 grro_dynamoresetrs rKperThreadSetUp"FSDPTestMultiThread.perThreadSetUpq rJc@[RR5 grrArs rKperThreadTearDown%FSDPTestMultiThread.perThreadTearDowntrFrJrB) rCrDrErFpropertyrr8r.rDrHrIr3r4s@rKr1r1es. 3rJr1c$^\rSrSrU4Sjr\S5r\S5r\S\4Sj5r \S5r Sr S r S r S r\S 5rS)S\R$S\S\S\S\\S\S\\S\S\S\\\\44SjjrS SS\"5S S S SSSSS S 4 S\\S\S\S\\ S \S\S!\S"\\!S#\\"S\\S$\S%\S\S\S&\\\\4S\\\\44 S'jjr#S(r$U=r%$)*FSDPTestixch>[TU]5 S[RS'UR 5 g)N0TORCH_NCCL_DESYNC_DEBUG)rr8osenviron_spawn_processesr:s rKr8FSDPTest.setUpys)  14 ,- rJc[$rr3rs rKrFSDPTest.world_sizer6rJc>[RR5$r)r}distributed_c10d_get_default_grouprs rKruFSDPTest.process_groups$$7799rJrUcgrSrBrs rKdestroy_pg_upon_exitFSDPTest.destroy_pg_upon_exitsrJc*[UR3$r)r0 file_namers rK init_methodFSDPTest.init_methodst~~.//rJc:URX!R5 gr)r!r)rXr,rs rK_check_cpu_offloadFSDPTest._check_cpu_offloads &<&<=rJc:URX!R5 gr)r!backward_prefetch)rXr,res rK_check_backward_prefetch!FSDPTest._check_backward_prefetchs *,H,HIrJc:URX!R5 gr)r!forward_prefetch)rXr,ris rK_check_forward_prefetch FSDPTest._check_forward_prefetchs )+F+FGrJc [U/UQ70UD6$rr=rs rKr.FSDPTest.run_subtestsr?rJc 4U"U5nXlX6lURSS5n[SURSUR35 U(a^[ R RRRR5n[R"SURUUS9 OC[R"UR[[UR5URS9 Sn UR[,-n [.(d [0(a[ R2R5U 5 U /n [R6"U S 9 [ R8R;5 UR=X$5 [ R8R;5 [R6"U S 9 [R>"5 g![ a@n SU R"S ;a'[$R&"[(S R*5 eSn A ff=f) Nfake_pgFz dist init r=z, world=fake)backendrrstore)r_rqrr recompilerbackend_unavailable) device_ids) rr^getprintrrotesting _internalrro FakeStorer}init_process_groupr_DISTRIBUTED_BACKENDr RuntimeErrorrgsysexitr/ exit_coder4r2r4 acceleratorset_device_indexrrBrCrun_testdestroy_process_group) rr test_namer^piperhrXrorreru device_ids rK_run FSDPTest._runs9~ "**Y. TYYKx/@AB  //;;CCMMO''"# '' $ 0 0/"4??3  II , 9    . .y 9[ +  i&  + ""$/ affQi'$9:DDE   s A$G 2AG H;HHNFrt num_stepsautocastlrfsdp_cpu_offload save_modelmixed_precisionenable_sharded_grad_scaler use_pure_fp16sharded_grad_scaler_kwargsc U=(a URn [UR55Rn U c0n [ SSU0U D6n [ R RUR5USS9n[U5GHgnUR5 [ RR[US9 URR[ R"[55nU (dU(aW[U[ 5(dB[U[ R"5(aUR%5nO['SU55nU"U6nU (ap[U[ 5(a[UR([*;aGUR5H3nUR-UR[ R"S55 M5 URR/UU5R1U 5nSSS5 U R3W5nU(d-U (d&UR4[ R6:XdS5eOU (a+UR-UR4[ R85 Ok[U[ 5(a,UceUR-UR4UR:5 O*UR-UR4[ R65 URR=U5 U (a\[U[ 5(aGUR5H3nUR-UR[ R"S55 M5 U R?U5 U RA5 U(dGM URC5RE5VVs0sHunnUURG5_M nnn[IU5 URKU5 GMj [U[ 5(aURM[NRP5 WRS5$!,(df  GNA=fs snnf) Nenabledg?)rmomentum)rc3@# UHoR5v M g7fr)r)rrs rKr4FSDPTest._train_for_several_steps..s%>1ffhhrr<zeloss data type should be float32, as the original parameter data type is float32.rB)*offload_paramsnextrrYrrooptimSGDr zero_gradamprrr?rZr#rrprrnr!rr!r`rscalerfloat32float16 param_dtyperestepupdaterrclonerload_state_dict _assert_staterIDLEr{)rXrtrrrrrrrrrcpu_offload_params model_devicesharded_grad_scalerrrr^r_rdrdkvrs rK_train_for_several_steps!FSDPTest._train_for_several_stepss/.Q2B2Q2QE,,./66 % -)+ &/ . 2L    0 0 2rCHy!A OO ##K#B ..u||K/HI _Zt=T=T!%66 %  %%>%> >'"5$////>?#--/((5<<3FG0||,,UF;>>|L-C.',,T2D"=JJ%--/555/!$$TZZ?t,,*666$$TZZ1L1LM$$TZZ? LL % %d +!j&=&=))+A$$QXXu||E/BC, $ $U +  & & (z7<7G7G7I7O7O7QR7Qtq!al7Q RE"%%j1s"v eT " "    2 2 3{{}wCBfSs9EP3-Q3 Q rT model_classrr ref_init_fn num_itersrrer!riuse_orig_params init_kwargsc U[R:wdS5eUc0nSnURR5nUR"UR[R[ R 4SS0UD6nUc-[(a[U[/[S9nO[UU/US9nOU"U5nU(aUR5nURUUU SLUUU U UUS9 n[UR55nURUUU U U U S.5 UR"URUUU4SS0UD6n[%U[&5(d['UUR40UD6nU(aUR5nU[ R(:XaUR+[5nUSL=(a UR,nU=(a U[ R(:HnU=(a U[ R(:gnU(aI[.R0"S 5nUR5HnUR3UR0U5 M! U(aUR5[6S [35O [95nU URUUS UUUU U UUS9 n SSS5 U(agU(a^[.R0"S 5nUR5HnUR3UR0U5 M! W R+[5n [;U5n![.R<R?UW S S9 U cU(dUR3UU!SSS9 ggg![a n[!S US [#U535UeSnAff=f!,(df  N=f)a Tests FSDP training against a reference, which defaults to DDP but may be customized with ``ref_init_fn``. Args: model_class (Type[FSDPTestModel]): A model class that inherits from ``FSDPTestModel``, which defines the expected interface. fsdp_init_mode (FSDPInitMode): The mode to initialize the FSDP-wrapped model. This should not be ``NO_FSDP``. ref_init_fn (Optional[Callable]): A callable to invoke that wraps a non-wrapped model to construct the reference model, where this wrapper should provide data parallel semantics. If ``None``, then the callable defaults to the DDP constructor. z.Expects an FSDP init mode that wraps with FSDPN{Gz?rT)ru output_device)rrrrrrr)rrer!rrirz Initializing z raised error r<zOAn FSDP-managed module with parameter CPU offloading enabled has parameters on F)rrrrrrrr) check_dtypezFSDP did not match DDP) exact_devicemsg) r@rGrurrjrMrOr3DDPrrrrrr Exceptionr'rr#rrPrrrorYr!assertRaisesRegexr}rrrx assert_close)"rXrrrrrrrrer!rrirrrrrrrrrt ref_modelref_loss ddp_paramsr,rrexpects_device_errorexpects_cpu_device cpu_devicercontext fsdp_lossfsdp_unsharded_paramss" rK_test_fsdp_parityFSDPTest._test_fsdp_parity)sF l22 2 < ; < 2  K !!&&(       ( (       x{m;  4&M #E*I !(I00  $D0(+'A''A1  )..01 *%6%6#2$4#2    Y$))""   #   J*d++j$*<*<L LJ #*J ~:: :#{3J$D0O[5O5O  N/>3N3NN   N/>3N3NN  e,J#..0  z:1$  " "%%0M3   55!,% /+E++E6 I  e,J#..0  z:1! [1I / ;  ""8YE"J  "=   %!,  ,9 "K Y}[MAxPQWX X YFWs$"M1M5 M2M--M25 N)r^r)rNFNFFN)&rCrDrErFr8rJrrur2r[r_rbrfrjr. classmethodrrrrsrfloatrrrrrr rrrSr@rMr rrrrIr3r4s@rKrLrLxsI ::d00>JH31%1%p15 48+0#?CUyyUU U  U #:. UU".1U%)UU%-T#s(^$<Ux+/",,8<8<48!& %+0#04?C#g-(g%g) g h' g  gg g$$45g$$45g".1ggg%)gg d38n-!g"%-T#s(^$<#ggrJrLcompile_compute_on_modulecD^^^U4Sjm"SS[5mUU4SjnU$)Nc>[RRR"U0UD6 Tb[ UST5(aUSR 5 gg)Nr)rorrrr#rh)rgrhrs rK!fully_shard_with_compiled_compute=compiled_fsdp_test..fully_shard_with_compiled_computesS **D;F; $ , G.1 1  GOO 1 rJc0\rSrSr\"5r\"5rSrg)*compiled_fsdp_test..FullyShardModeirBN)rCrDrErFrEAGERCOMPILED_COMPUTErIrBrJrKFullyShardModers6rJrc6>^[T5UUU4Sj5nU$)Nc>[RRRnTGHnUTR:wa'[ 5(d[ R"S5 M;[RRRn[RRRn[RR5 UTR:XaUnO_UTR:XaAS[RRl S[RRl TnO[SU35eUT R UR"'T "U0UD6 [RR5 UT R UR"'U[RRl U[RRl GM g)Nz0Inductor on GPU needs Triton and recent GPU archTr>z!Need to implement FullyShardMode=)rorrrrr5warningswarnrBconfigskip_fsdp_hooks _inductorcompile_threadsrrNotImplementedError __globals__rC) rgrhoriginal_fully_shardmodeoriginal_skip_fsdp_hooksoriginal_compile_threadsfully_shard_patchrrfuncs rKwrapper6compiled_fsdp_test..decorator..wrappers_(-(9(9(>(>(J(J &>/// MM"TU+0==+?+?+O+O(+0??+A+A+Q+Q(!!))+>///(<%^<<<;?EMM((8=>EOO**:(I%-;D6B CT  !5!>!>?d%f%!!))+BV  !5!>!>?7O $$49Q&&69'rJr )rrrrs` rK decorator%compiled_fsdp_test..decorators# t R  R@rJ)r)rrrrs` @@rKcompiled_fsdp_testrs""""H rJc2^\rSrSrSU4SjjrSrSrU=r$) SkipModuleicX>[TU]5 [R"SSSS9UlgN Fr)rrrrrlinr:s rKrSkipModule.__init__s" 99R%0rJc$URU5$rrrGs rKrSkipModule.forward sxx{rJrrlrr4s@rKrrs1rJrc.^\rSrSrU4SjrSrSrU=r$) NestedLinearic>[TU]5 U(a8[[R"SSSS9R [ 55Ulg[R"SSSS9R [ 5Ulgr)rrr!rrrrr nested_linear)rX fsdp_wraprs rKrNestedLinear.__init__sV  !%biiBU&C&F&F{&S!TD !#2r!>!A!A+!ND rJc$URU5$rrrGs rKrNestedLinear.forwards!!!$$rJrrr4s@rKrrsO%%rJrc.^\rSrSrU4SjrSrSrU=r$) SkipModelic>[TU]5 [R"SSSS9R [ 5Ul[5R [ 5Ul[[US9[ S9Ul g)NrFr)r)r) rrrrrrrlinearr linear_skipr!rr)rX double_nestrs rKrSkipModel.__init__sW iiBU366{C %rrcontextmanagerrrrrr rrrrnrr/r1rLrrrrrrBrJrKr#sa #"FF 44 ?S IRRFF FH +  K ::**,L K  K 99))+LK L4T BIIs 4 99$$>% # 99##" " 2299 2t 2>299>d>DBIID$D.Q"-Q"h[J-[J|$7D]J.]J@J mJ Z ? .,ryy,JJ*JJZ/"))/@*r}}*Z'299'6 666 9H99 *X** .x.. .x.. :8:: >X>>0!00  0  0.+- TyyTIITc3h TB/&X#Xv 0(4.0f %299 %  rJ