shySSKrSSKrSSKJr SSKJrJrJrJr SSK J r SSK r SSK J s Js Jr SSKJ s Js Jr SSKJr SSKJrJr SSKJrJr SSKJrJr SSKJ r J!r! SS K"J#r#J$r$J%r% SS K&J'r'J(r(J)r)J*r* /S Qr+\ RXRZr-"S S \ R\R^5r0"SS\ R\R^5r1"SS\ Rd5r3S/SS.S\ RdS\\S\\\(S\\4S\34 Sjjjr5\ "S5S0S\ RdS\\*S\\SS4Sjj5r6S1S\RnS\\S\\\8\Rn\/S4S\\\Rn\\/S4S\\\Rn\\/S4S\Rn4 S jjr9S/S!\ RtS\\S\\\(S\34S"jjr;S\ RxS#SSS$.S%\\ RzS&\ R|S'\?S\\S\\\(S\34 S(jjr@S\ RxS#SSS$.S%\\ RzS&\ R|S'\?S\\S\\\(S\34 S)jjrAS\ RxS#SSS$.S%\\ RzS&\ R|S'\?S\\S\\\(S\34 S*jjrBS#S\ RxSSS+.S'\?S%\\ RzS&\ R|S\\S\\\(S\34 S,jjrCS#S\ RxSSS+.S'\?S%\\ RzS&\ R|S\\S\\\(S\34 S-jjrDS#S\ RxSSS+.S'\?S%\\ RzS&\ R|S\\S\\\(S\34 S.jjrEg)2N)Sequence)AnyCallablecastOptional) deprecated)_mesh_resources DeviceMesh)check_tensor_metamesh_broadcast) DTensorSpec TensorMeta) Redistributeredistribute_local_tensor)compute_global_tensor_info%compute_local_shape_and_global_offsetnormalize_to_torch_size)Partial Placement ReplicateShard) DTensordistribute_tensordistribute_moduleonesemptyfullrandrandnzerosch\rSrSr\SSS\\\4Sj5r\S\ R4Sj5r Sr g ) _ToTorchTensorEinputrgrad_placementscjURUlX lURnUR U5$N)_spec dtensor_specr% _local_tensorview_as)ctxr$r% local_tensors q/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/tensor/_api.pyforward_ToTorchTensor.forwardFs4 !;;-** ##L11 grad_outputc PURnURnURnURn[ XUR 5upg[ U5nU=(d UR n[UU[URUURS9S9n[UUURS9S4$)Nshapestridedtype tensor_meta requires_grad) r)meshr%r9r placementstupler rr5r7rr;) r,r2r)r<r% dtensor_meta_ tensor_stride grad_specs r.backward_ToTorchTensor.backwardUs''   --#// 5 |66 m, )D\-D-D  ""(($"((  )77     r1N) __name__ __module__ __qualname____firstlineno__ staticmethodrrrr/torchTensorrC__static_attributes__rEr1r.r"r"EsO 2 2"(9"56 2 2 5<<  r1r"c\rSrSr\SS\R S\S\\ S4S\ S\ \RS \ \\ S4S S 4S jj5r\SS j5rSrg)_FromTorchTensoruNr$ device_meshr=. run_checkr5r6returnrc X0lX lU(a U(aXVpOPU(d6U(d/[XU5up[R"U 5[ U 5pO[ SUSUS3S5eUR5cURSURS9nOjU(acU(+=(a U(+n [XS9 [U5H6upU R5(dMUR5n[XU S9 M8 [UU[!UUUR"5S 9n[%UR'U5UURS9nU$) Nz Found shape:z , stride:.z3Please pass both shape and stride at the same time.rr:)check_shape_stride)mesh_dimr8)previous_placementprevious_device_meshrrKSizer> RuntimeErrorget_coordinate new_emptyr;r enumerate is_replicate contiguousr r rr7rr+)r,r$rQr=rRr5r6 tensor_shaperA global_shape global_striderVidx placement dist_spec dist_tensors r.r/_FromTorchTensor.forwardvsE",#. V*/-v+EJ+ 'L+0**\*BE-DX-ugYvha8E   % % ' /OOAU5H5HOIE &+!96z  e K#,J"7))++",,.E"5D #8  "   MM%  --  r1cURnURnURU:waJURn[ UUURR S9nUR n[XdUSS9nUSSSSS4$UR5SSSSS4$)Nr8T) is_backward) rXrYr=r(r r9r*rto_local)r,r2rXrY current_spec target_specr-outputs r.rC_FromTorchTensor.backwards 33"77  ! !%7 7&,,L%$"'--99K '44L.KTF 4tT47 7##%tT4tCCr1rENN)r2r)rFrGrHrIrJrKrLr r>rboolrrZintr/rCrMrEr1r.rOrOus'+,0@||@ @)S.) @  @  # @sCx)@ @@DDDr1rOc\rSrSr%Sr\R \S'\\S'SS/r \ R"5r \ R\S'\ \RS\R S\S\S S4S j55rS rS r\ S 5rSrS*Sjr\\RS+Sj55r\ S,SSSS.S\R S\\S\\\S\S\\R8S\\\S4S S4Sjjj5rSS.S\\\S \R 4Sjjr S,SS.S\\S\\\S \S S4S!jjjr!SS.S\\\S \R 4S"jjr"\#S \4S#j5r$\#S \\S44S$j5r%S%\&S&\'4S'jr(S(r)S)r*Sr+g)-ray ``DTensor`` (Distributed Tensor) is a subclass of ``torch.Tensor`` that provides single-device like abstraction to program with multi-device ``torch.Tensor``. It describes the distributed tensor sharding layout (DTensor Layout) through the :class:`DeviceMesh` and following types of :class:`Placement`: * :class:`Shard`: Tensor sharded on the tensor dimension ``dim`` on the devices of the ``DeviceMesh`` dimension * :class:`Replicate`: Tensor replicated on the devices of the ``DeviceMesh`` dimension * :class:`Partial`: Tensor is pending reduction on the devices of the ``DeviceMesh`` dimension When calling PyTorch operators, ``DTensor`` overrides the PyTorch operators to perform sharded computation and issue communications whenever necessary. Along with the operator computation, ``DTensor`` will transform or propagate the placements (DTensor Layout) properly (based on the operator semantic itself) and generate new ``DTensor`` outputs. To ensure numerical correctness of the ``DTensor`` sharded computation when calling PyTorch operators, ``DTensor`` requires every Tensor argument of the operator be DTensor. .. note:: Directly using the Tensor subclass constructor here is not the recommended way to create a ``DTensor`` (i.e. it does not handle autograd correctly hence is not the public API). Please refer to the `create_dtensor`_ section to see how to create a ``DTensor``. r*r(_op_dispatcherr-specr;rSc tUR(aU(d[R"S5 URcS5e[R R UURRURRURURURUS9nX$l Xl U$)a Construct a DTensor from a local tensor, device mesh, and placement and other tensor properties (i.e. shape, requires_grad, strides, etc). .. note:: This is not a public API and it's only supposed to be used by the operator implementations and internals. If you want to construct a DTensor from a local tensor, consider using ``DTensor.from_local``, if you want to construct a DTensor from a "global" tensor (where you already have tensor initialized and want to shard this tensor), consider using ``distribute_tensor``. zxTo construct DTensor from torch.Tensor, it's recommended to use local_tensor.detach() and make requires_grad consistent.zTensorMeta should not be None!)stridesr7devicelayoutr;)r;warningswarnr9rKrL_make_wrapper_subclassr5r6r7ryrzr(r*)clsr-rvr;rs r.__new__DTensor.__new__s(  % %m MMO +M-MM+ LL / /     " "$$++$$&&&&' 0 &r1c~SURSURRSURRS3$)NzDTensor(local_tensor=z, device_mesh=z , placements=))r*r(r<r=selfs r.__repr__DTensor.__repr__s=&t'9'9&:.HYYfgkgqgqg|g|f}}~r1c8S/URUR44$)zM protocol to inform how to flatten a DTensor to local tensor for PT2 tracing r*)r(r;rs r.__tensor_flatten__DTensor.__tensor_flatten__#s!   4::t/A/A"BBBr1cUcS5eUSnUupV[UUURRS9n[URUR US9n[ UUUS9$)NzEExpecting spec to be not None from `__tensor_flatten__` return value!r*r4r8r:)rr9r7r r<r=r) inner_tensors flatten_spec outer_size outer_strider-rvr;unflatten_tensor_metaunflatten_specs r.__tensor_unflatten__DTensor.__tensor_unflatten__*s' S '%_5 * *""((!  % II OO-    '  r1c[SUR55(dU$URVs/sH$n[U[5(a [ 5OUPM& nnUR UR US9$s snf)Nc3B# UHn[U[5v M g7fr') isinstancer).0ps r. 6DTensor.__coerce_tangent_metadata__..BsC?a:a))?srQr=)anyr=rrr redistributerQ)rrr=s r.__coerce_tangent_metadata__#DTensor.__coerce_tangent_metadata__AsrC4??CCCKBF// BQQ:a11IKq 8/    T-=-=* UU s+A9NcZUbgUup4URURURS9$)Nr)rrQr=)rr expected_typervr@s r.#__coerce_same_metadata_as_tangent__+DTensor.__coerce_same_metadata_as_tangent__Is:  $   ((!  r1rEcV[RRUUU=(d 05$r')rrudispatch)r~functypesargskwargss r.__torch_dispatch__DTensor.__torch_dispatch__Ss* %%..   Lb  r1F)rRr5r6rQr=rRr5r6.cvU=(d [R"5nURnX`RR:wa"UR (dUR U5nUc.[UR5Vs/sH n[5PM nnO|[U5n[U5HbupU R5(dM[[U 5n U RS:dM>[U RUR-5X('Md [ R#UU[%U5UUU5$s snf)a Create a :class:`DTensor` from a local torch.Tensor on each rank according to the ``device_mesh`` and ``placements`` specified. Args: local_tensor (torch.Tensor): local torch.Tensor on each rank. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the local torch.Tensor on DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. Keyword args: run_check (bool, optional): at a cost of extra communications, perform sanity check across ranks to check each local tensor's meta information to ensure correctness. If have :class:`Replicate` in ``placements``, the data on first rank of the device mesh dimension will be broadcasted to other ranks. default: False shape (torch.Size, optional): A List of int which specifies the size of DTensor which build on top of `local_tensor`. Note this needs to be provided if the shape of ``local_tensor`` are different across the ranks. If not provided, ``shape`` will be computed assuming the given distributed tensor is evenly sharded across ranks. default: None stride (tuple, optional): A List of int which specifies the stride of DTensor. If not provided, ``stride`` will be computed assuming the given distributed tensor is evenly sharded across ranks. default: None Returns: A :class:`DTensor` object .. note:: When ``run_check=False``, it is the user's responsibility to ensure the local tensor passed in is correct across ranks (i.e. the tensor is sharded for the ``Shard(dim)`` placement or replicated for the ``Replicate()`` placement). If not, the behavior of the created DTensor is undefined. .. note:: ``from_local`` is differentiable, the `requires_grad` of the created `DTensor` object will depend on if `local_tensor` requires_grad or not. r)r get_current_mesh device_typerytypeis_metatorangendimrlistr^is_shardrrdimrOapplyr>) r-rQr=rRr5r6rr@rdres r. from_localDTensor.from_local^s l"G_%E%E%G !--  --22 2<;O;O'??;7L  /4[5E5E/FG/F!)+/FJGJj)J"+J"7%%'' $UI 6I }}q(*/ @Q@Q0Q*R  #8 %%   *       HsD6)r%r%c[R"5(d UR$Ub [U[5(d [ U5n[ R X5$)aT Get the local tensor of this DTensor on its current rank. For sharding it returns a local shard of the logical tensor view, for replication it returns the replica on its current rank. Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the Tensor returned from this function. `to_local` converts DTensor to local tensor and the returned local tensor might not be used as the original DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original DTensor layout. If not specified, we will assume the gradient layout remains the same as the original DTensor and use that for gradient computation. Returns: A :class:`torch.Tensor` or ``AsyncCollectiveTensor`` object. it represents the local tensor on its current rank. When an ``AsyncCollectiveTensor`` object is returned, it means the local tensor is not ready yet (i.e. communication is not finished). In this case, user needs to call ``wait`` to wait the local tensor to be ready. .. note:: ``to_local`` is differentiable, the ``requires_grad`` of the local tensor returned will depend on if the `DTensor` requires_grad or not. )rKis_grad_enabledr*rr>r"r)rr%s r.rkDTensor.to_localsQ8$$&&%% %  &z/5/Q/Q#O4O##   r1)async_oprcU=(d URnUc [S5e[U5n[U5HrupEUR 5(a [S5e[ U[ 5(dM<URS:dMN[ URUR-5X$'Mt [U5n[R"XX#5$)a{ ``redistribute`` performs necessary collective operations that redistribute the current DTensor from its current placements to a new placements, or from is current DeviceMesh to a new DeviceMesh. i.e. we can turn a Sharded DTensor to a Replicated DTensor by specifying a Replicate placement for each dimension of the DeviceMesh. When redistributing from current to the new placements on one device mesh dimension, we will perform the following operations including communication collective or local operation: 1. ``Shard(dim)`` -> ``Replicate()``: ``all_gather`` 2. ``Shard(src_dim)`` -> ``Shard(dst_dim)``: ``all_to_all`` 3. ``Replicate()`` -> ``Shard(dim)``: local chunking (i.e. ``torch.chunk``) 4. ``Partial()`` -> ``Replicate()``: ``all_reduce`` 5. ``Partial()`` -> ``Shard(dim)``: ``reduce_scatter`` ``redistribute`` would correctly figure out the necessary redistribute steps for DTensors that are created either on 1-D or N-D DeviceMesh. Args: device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the DTensor. If not specified, it would use the current DTensor's DeviceMesh. default: None placements (List[:class:`Placement`], optional): the new placements that describes how to place the DTensor into the DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. default: replicate on all mesh dimensions Keyword args: async_op (bool, optional): whether to perform the DTensor redistribute operation asynchronously or not. Default: False Returns: A :class:`DTensor` object .. note:: ``redistribute`` is differentiable, which means user do not need to worry about the backward formula of the redistribute operation. .. note:: ``redistribute`` currently only supports redistributing DTensor on the same DeviceMesh, Please file an issue if you need to redistribute DTensor to different DeviceMesh. z&placements is needed for redistribute!zTCan not redistribute to Partial, redistributing to Partial is for internal use only!r) rQr[rr^ is_partialrrrrr>rr)rrQr=rires r.rDTensor.redistributesj"5T%5%5  GH H*% %j1LA##%%"jIu--)--!2C %immdii&? @ 2:& !!$ZJJr1cUR[5/URR-SS9n[R X!5$)aA Return the full tensor of this DTensor. It will perform necessary collectives to gather the local tensors from other ranks in its DeviceMesh and concatenate them together. It's a syntatic sugar of the following code: ``dtensor.redistribute(placements=[Replicate()] * mesh.ndim).to_local()`` Keyword args: grad_placements (List[:class:`Placement`], optional): the placements describes the future layout of any gradient layout of the full Tensor returned from this function. `full_tensor` converts DTensor to a full torch.Tensor and the returned torch.tensor might not be used as the original replicated DTensor layout later in the code. This argument is the hint that user can give to autograd in case the gradient layout of the returned tensor does not match the original replicated DTensor layout. If not specified, we will assume the gradient layout of the full tensor be replicated. Returns: A :class:`torch.Tensor` object that represents the full tensor of this DTensor. .. note:: ``full_tensor`` is differentiable. F)r=r)rrrQrr"r)rr% redist_ress r. full_tensorDTensor.full_tensor sF4&&! }t'7'7'<'< ? ?%%;;= = **ELL 9 9.t45 59: :r1c[URS5(aURRU5$[UR[R 5(aUR 5$[S5e)N__get_tensor_shard__r)rr*rrrKrLrkr[)rindexs r.rDTensor.__get_tensor_shard__ts\ 4%%'= > >%%::5A A **ELL 9 9==? "9: :r1r')rENrp),rFrGrHrI__doc__rKrL__annotations__r __slots__ op_dispatch OpDispatcherrurJ_disable_dynamorqrrrrrr classmethodrrr rrrZr>rrrrkrrpropertyrQr=strrrrrrMrEr1r.rrs*<<  '*I0;/G/G/INK,,I 'll''  '  ''V@C  ,V    -148R  &*,0R llR j)R Xi01R  R  # R sCx)R  R R jCG# "*8I+>"?# # N-148FK  FKj)FKXi01FK  FK  FKRCGA"*8I+>"?A A>Z%E)S.1%% ;# ;s ;;,;r1r src_data_ranktensorrQr=rrSc L[RRS5 U=(d [R"5nUR nUS:XaSSKJn U"XU5$UR(d [S5eX@RR:wa"UR(dURU5nUc-[UR 5Vs/sH n[#5PM nn[%U5UR :wa%['S[%U5S UR S 35e[)U[*5(acUR,U:wa['S UR,S US 35eUR.[1U5:wa['S UR.SUS35eU$UR35n [5U5n[7U5HupU R95(aZ[;[<U 5n U R>S:a&[=U R>UR -5n XU 'U RAXX5n MtU RC5(a$[;["U 5n U REXX5n M[SU SU S35e [1U5nU cS5e[GUU[IURK5URM5URNS9S9n [+U RQURR5U URRS9$![anSn[U5UeSnAff=fs snf)a, Distribute a leaf ``torch.Tensor`` (i.e. nn.Parameter/buffers) to the ``device_mesh`` according to the ``placements`` specified. The rank of ``device_mesh`` and ``placements`` must be the same. The ``tensor`` to distribute is the logical or "global" tensor, and the API would use the ``tensor`` from first rank of the DeviceMesh dimension as the source of truth to preserve the single-device semantic. If you want to construct a DTensor in the middle of the Autograd computation, please use :meth:`DTensor.from_local` instead. Args: tensor (torch.Tensor): torch.Tensor to be distributed. Note that if you want to shard a tensor on a dimension that is not evenly divisible by the number of devices in that mesh dimension, we use ``torch.chunk`` semantic to shard the tensor and scatter the shards. The uneven sharding behavior is experimental and subject to change. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to distribute the tensor, if not specified, must be called under a DeviceMesh context manager, default: None placements (List[:class:`Placement`], optional): the placements that describes how to place the tensor on DeviceMesh, must have the same number of elements as ``device_mesh.ndim``. If not specified, we will by default replicate the tensor across the ``device_mesh`` from the first rank of each dimension of the `device_mesh`. Keyword args: src_data_rank (int, optional): the rank of the source data for the logical/global tensor, it is used by :meth:`distribute_tensor` to scatter/broadcast the shards/replicas to other ranks. By default, we use ``group_rank=0`` on each DeviceMesh dimension as the source data to preserve the single-device semantic. If passing ``None`` explicitly, :meth:`distribute_tensor` simply uses its local data instead of trying to preserve the single-device semantic via scatter/broadcast. Default: 0 Returns: A :class:`DTensor` or ``XLAShardedTensor`` object. .. note:: When initialize the DeviceMesh with the ``xla`` device_type, ``distribute_tensor`` return `XLAShardedTensor` instead. see `this issue `__ for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_tensorxlar)xla_distribute_tensorDTo use DTensor API with xla, you must install the torch_xla package!NzY`distribute_tensor` should be used to distribute leaf tensors! but found non-leaf tensor!zW`placements` must have the same length as `device_mesh.ndim`! Found placements length: z, and device_mesh.ndim: rUz-Cannot distribute a DTensor with device mesh z to a different device mesh z,Cannot distribute a DTensor with placements z to a different placements z-. do you want to call `redistribute` instead?z8Trying to distribute tensor with unsupported placements z on device mesh dimension !z(distributing a tensor should not be Noner4)r<r=r9r:)*rK_C_log_api_usage_oncer rrtorch_xla.distributed.spmdr ImportErroris_leafr[ryrrrrrrlen ValueErrorrrrQr=r>detachrr^rrrr _shard_tensorr__replicate_tensorr rsizer6r7requires_grad_r;) rrQr=rrremsgr@r-rdrervs r.rr}s$^ HH  !BCC!A!A!CK))Ke * )jI I >> g  mm(((;'+01A1A+BC+Baik+B C :+***((+J'88PQ\QaQaPbbc e  &'""    ,?@R@R?ST..9]!=    j 1 1>v?P?P>QR--7L9*+   ==?Lj!J#J/     UI.I}}q !)--&++"=> "+3$223L # # % %Y 2I$663LJ9+Uopsottuv !0&z"J  #O%OO #  ++-==?,,  D ##F$8$89 ** ] *XCc" ) *DsLL! L LLzAPlease use `distribute_tensor` with `src_data_rank=None` instead.rc[XUSS9$)a4 Locally shards a full tensor based on indicated sharding arrangement, and returns a DTensor containing the local shard. .. warning:: This is a private API that is subject to change. It skips the communication otherwise required by `distribute_tensor`. It is only applicable to cases where all ranks have the same `full_tensor`. For example, in distributed inference all ranks load from the same checkpoint. This API will not check for data equality between ranks, it is thus user's responsibility to ensure the `full_tensor` is the same across ranks. Args: full_tensor (torch.Tensor): the full tensor to be sharded. placements (Sequence[:class:`Shard`]): the placements that describes how to place the local tensor on DeviceMesh. device_mesh (:class:`DeviceMesh`, optional): DeviceMesh to place the DTensor. Must have same dimension as the number of placements. If not specified, would be retrieve from current context. Returns: A :class:`DTensor` object with the shard as its local tensor. Examples: >>> # xdoctest: +SKIP("need world_size and rank") >>> device_mesh = dist.init_device_mesh("cuda", (world_size,)) >>> full_tensor = torch.arange(world_size, device=f"cuda:{rank}") >>> dtensor = _shard_tensor(full_tensor, [Shard(1)], device_mesh) Nr)r)rr=rQs r.rrsF [zQU VVr1module partition_fninput_fn output_fnct^^^[RRS5 [USS5nU(a [ S5eT=(d [ R "5mTRnUS:XaSSKJ n U"UTUTT5$S [RS [S S 4S jn Uc#UR5HupU "U T5 M O+UR5HupU"XT5 U "U T5 M Tb[[ R""T5R$5n U S:Xa1[&R("S[*SS9 UR-UU4Sj5 O,U S:XaUR-UU4Sj5 O[/SU S35eTb[[ R""T5R$5n U S:Xa1[&R("S[*SS9 UR1UU4Sj5 O,U S:XaUR1UU4Sj5 O[/SU S35eSUlU$![anSn [U 5UeS nAff=f)a This function expose three functions to control the parameters/inputs/outputs of the module: 1. To perform sharding on the module before runtime execution by specifying the ``partition_fn`` (i.e. allow user to convert Module parameters to :class:`DTensor` parameters according to the `partition_fn` specified). 2. To control the inputs or outputs of the module during runtime execution by specifying the ``input_fn`` and ``output_fn``. (i.e. convert the input to :class:`DTensor`, convert the output back to ``torch.Tensor``) Args: module (:class:`nn.Module`): user module to be partitioned. device_mesh (:class:`DeviceMesh`): the device mesh to place the module. partition_fn (Callable): the function to partition parameters (i.e. shard certain parameters across the ``device_mesh``). If ``partition_fn`` is not specified, by default we replicate all module parameters of ``module`` across the mesh. input_fn (Callable): specify the input distribution, i.e. could control how the input of the module is sharded. ``input_fn`` will be installed as a module ``forward_pre_hook`` (pre forward hook). output_fn (Callable): specify the output distribution, i.e. could control how the output is sharded, or convert it back to torch.Tensor. ``output_fn`` will be installed as a module ``forward_hook`` (post forward hook). Returns: A module that contains parameters/buffers that are all ``DTensor`` s. .. note:: When initialize the DeviceMesh with the ``xla`` device_type, ``distribute_module`` return nn.Module with PyTorch/XLA SPMD annotated parameters. See `this issue `__ for more details. The XLA integration is experimental and subject to change. ztorch.dtensor.distribute_module_distribute_module_appliedFzhdistribute_module should only be called once on a module, but it has already been called on this module!rr)xla_distribute_modulerNmr<rSc [5/UR-nURR5H[up4UcM [ U[ 5(aM!UR U[R"[URX555 M] URR5H:up5UcM [ U[ 5(aM![XQU5URU'M< gr') rr _parametersitemsrrregister_parameternn Parameterrdata_buffers)rr<full_replicatekeyparambuffers r.replicate_module_params_buffers:distribute_module..replicate_module_params_buffersys$+2----/JC E7)C)C$$LL!25::t!TU0 ::++-KC!*VW*E*E"3F."Q 3.r1zDeprecating input_fn that takes two arguments (inputs, device_mesh), please use input_fn that takes in (module, inputs, device_mesh) instead!) stacklevelc>T"UT5$r'rE)r@inputsrQrs r.#distribute_module..s (6;"?r1c>T"XT5$r'rE)modrrQrs r.rrs HS+$Fr1z-input_fn should take in 3 arguments, but got z arguments!zDeprecating output_fn that takes two arguments (inputs, device_mesh), please use output_fn that takes in (module, inputs, device_mesh) instead!c>T"UT5$r'rErroutputsrQrs r.rrs Yw -Lr1c>T"XT5$r'rErs r.rrs Ys[-Qr1z.output_fn should take in 3 arguments, but got T)rKrrgetattrr[r rrrrrrModuler named_modulesrinspect signature parametersr{r| FutureWarningregister_forward_pre_hookrregister_forward_hookr)rrQrrralready_distributedrrrrrnamesubmodnum_argss ` `` r.rr5sHR HH  !BC!&*FN =  C!A!A!CK))Ke * ) \8Y R299RJR4R&#002LD +FK @3#002LD { 3 +FK @3 w((2==> q= MM[    , ,? ]  , ,F ?zU w((3>>? q= MM\    ( (L ]  ( (Q @ +V )-F% Me *XCc" ) *s3H H7$H22H7rc U=(d [R"5nURUS'U=(d% [S[ UR 555nUR [ U5:XdS5eUS[R:XdS5e[RRU5n[XU5upgU[R:XaURSS5nU"Xh40UD6n GO U[R:XdU[R:XaUR!S[R""55n [%US U 5n ['U[U5U S 9n [(R*"U5(a4[(R,(d[(R."U5[(l[(R,ce[(R,R1U 5 U"U40UD6n SSS5 O U"U40UD6n ['U[U5[%UUW R25S 9n [5U U US S 9$!,(df  NE=f) Nryc36# UHn[5v M g7fr')r)rr@s r.r'_dtensor_init_helper..s$R:QQY[[:Qsz6mesh dimension does not match the length of placementsrzzlayout value not supported! fill_valuerr7)rr8r;r:)r rrr>rrrrKstrided _prims_commonmake_contiguous_strides_forrrpoprrgetget_default_dtyperr randomis_rng_supported_mesh _rng_trackerOffsetBasedRNGTracker_distribute_regionr7r) init_oprrQr=r torch_stride local_shaper@r.r-r7r9rvs r._dtensor_init_helperr=sC!A!A!CK"..F8Ru$R% @P@P:Q$RRJ   s: .@ . ( u}} ,K.KK ,&&BB4HL; :NK %**ZZ a0 {A&A EJJ 'U[["8 7E$;$;$=> tU3 ;j(9{S  ' ' 4 4V=P=P"(">">{"KF ""...  3 3D 9";9&9L: 9{5f5  j       D  _- : 9s ' H:: IFr7rzr;rQr=r7rzr;c R[U5n[[RUUUUUUS9$)a  Returns a :class:`DTensor` filled with the scalar value 1, with the shape defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r>)rr=rKrr7rzr;rQr=r torch_sizes r.rrs4<).J  # r1c R[U5n[[RUUUUUUS9$)a0 Returns a :class:`DTensor` filled with uninitialized data. The shape of the :class:`DTensor` is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: empty(1,2,3..) or empty([1,2,3..]) or empty((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r>)rr=rKrr@s r.rr=s4<).J  # r1c T[U5n[[RUUUUUUUS9$)aq Returns a :class:`DTensor` filled with ``fill_value`` according to ``device_mesh`` and ``placements``, with the shape defined by the argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) fill_value(Scalar): the value to fill the output tensor with. Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank )r.r7rzr;rQr=)rr=rKr)rr.r7rzr;rQr=rAs r.rrhs8B).J  #  r1)r;r7rzrQr=c R[U5n[[RUUUUUUS9$)aK Returns a :class:`DTensor` filled with random numbers from a uniform distribution on the interval ``[0, 1)``. The shape of the tensor is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r>)rr=rKrr;r7rzrQr=rrAs r.rrs4>).J  # r1c R[U5n[[RUUUUUUS9$)aJ Returns a :class:`DTensor` filled with random numbers from a normal distribution with mean 0 and variance 1. The shape of the tensor is defined by the variable argument ``size``. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: ones(1,2,3..) or ones([1,2,3..]) or ones((1,2,3..)) Keyword args: dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned DTensor. Default: ``torch.strided``. requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks. placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r>)rr=rKrrEs r.rrs4>).J  # r1c R[U5n[[RUUUUUUS9$)a Returns a :class:`DTensor` filled with the scalar value 0. Args: size (int...): a sequence of integers defining the shape of the output :class:`DTensor`. Can be a variable number of arguments or a collection like a list or tuple. E.g.: zeros(1,2,3..) or zeros([1,2,3..]) or zeros((1,2,3..)) Keyword args: requires_grad (bool, optional): If autograd should record operations on the returned :class:`DTensor`. Default: ``False``. dtype (:class:`torch.dtype`, optional): the desired data type of returned :class:`DTensor`. Default: if ``None``, uses a global default (see :func:`torch.set_default_dtype`). layout (:class:`torch.layout`, optional): the desired layout of returned :class:`DTensor`. Default: ``torch.strided``. device_mesh: :class:`DeviceMesh` type, contains the mesh info of ranks placements: a sequence of :class:`Placement` type: ``Shard``, ``Replicate`` Returns: A :class:`DTensor` object on each rank r>)rr=rKr rEs r.r r s48).J  # r1rpr')NNNN)Fr!r{collections.abcrtypingrrrrtyping_extensionsrrK"torch.distributed.tensor._dispatch distributedr _dispatchr torch.distributed.tensor._random_randomr5torch.nnrtorch.distributed.device_meshr r *torch.distributed.tensor._collective_utilsr r &torch.distributed.tensor._dtensor_specr r&torch.distributed.tensor._redistributerrtorch.distributed.tensor._utilsrrr(torch.distributed.tensor.placement_typesrrrr__all__opsatenautogradFunctionr"rOrLrrrrrrrrrZr=r/r7rzrqrrrrrr rEr1r.r\s$00( 8811EXJ   yy~~.- U^^,,- `]Du~~..]D@e;elle;T )-04O $% O LLO*%O),-O C= O  Od OP)-"W"W"W*%"W "WQ"WN)-KOGKHL R IIR*%R8S"))Z$@$$FGHRxC