sh2 SSKJr SSKJr SSKrSSKJs Js Jr SSK J r SSK J r SSKJr SSKJrJrJrJrJr S\ S \ S \\S \\\S 4\\S 444S jrS\R2S \ S \\S \\\\\44SjrS\R8R:S\\S \ 4SjrS\ S \ S \\S \\S 44Sjr S \RB4Sjr"g))Sequence)castN) ShapeType) DeviceMesh) DTensorSpec) _StridedShardPartial Placement ReplicateShard global_shapemesh placementsreturn.cUR5nUcg[U5nS/[U5-n[[U55Vs/sHnS/UR-PM nnS/[U5-n[ U5HupUR U 5n [U [5(dM-U Rn S/[U5-n U [U5:dSU S[U535eU RXLU X9SS9upXU 'XU 'X\X::aXX\'OX\==X- ss'X==U -ss'M [SU55nU(GaRS /[U5-nS /[U5-n[ U5HupUR U 5n [U [5(dM-U Rn UU (a[S U S U S US 35eUU (aSUU '[U [5(a SUU 'XU RU --X|U 'MX==U -ss'XX|U 'M [ U5V VVVs/sH4un n[[!UU5VVs/sH unnUU-PM snn5PM6 nnnn n[!UU5VVs/sH unnUU-PM nnn[#U5[#U54$s snfs snnfs snnnn fs snnf)a* Compute the local tensor shape and the global offsets into the original tensor of a DTensor on its current global rank. This is useful for checkpointing purpose. Example (2 host with 4GPUs each): # Below is a DeviceMesh with mesh_shape of (2, 4) mesh = DeviceMesh(device_type="cuda", mesh=[ [0, 1, 2, 3], [4, 5, 6, 7] ], ) Let's say we distribute a global_tensor of shape (8,4) over the above DeviceMesh with a placements of [Shard(0), Shard(0)]. The local shape and global offset will be as follows: rank0 -- local_shape:[1, 4], global_offset:[0, 0] rank1 -- local_shape:[1, 4], global_offset:[1, 0] rank2 -- local_shape:[1, 4], global_offset:[2, 0] rank5 -- local_shape:[1, 4], global_offset:[5, 0] rank3 -- local_shape:[1, 4], global_offset:[3, 0] rank4 -- local_shape:[1, 4], global_offset:[4, 0] rank6 -- local_shape:[1, 4], global_offset:[6, 0] rank7 -- local_shape:[1, 4], global_offset:[7, 0] Let's say we distribute a global_tensor of shape (2) over the above DeviceMesh with a placements of [Shard(0)]. We will not have non-empty local tensor for all the ranks. The local shape and global offset will be as follows: rank0 -- local_shape:[1,], global_offset:[0,] rank1 -- local_shape:[1,], global_offset:[1,] rank2 -- local_shape:[0,], global_offset:[2,] rank5 -- local_shape:[0,], global_offset:[2,] rank3 -- local_shape:[0,], global_offset:[2,] rank4 -- local_shape:[0,], global_offset:[2,] rank6 -- local_shape:[0,], global_offset:[2,] rank7 -- local_shape:[0,], global_offset:[2,] ))rr Sharding dim  greater than tensor ndim T) return_offsetc3B# UHn[U[5v M g7fN) isinstancer).0ps s/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/tensor/_utils.py 8compute_local_shape_and_global_offset..{sPZz!];;ZsFzTStrided sharding does not allow Shard() to appear after the strided part has ended. z at idx z in z violates this assumption.)get_coordinatelistlenrangendim enumeratesizerr dim_local_shard_size_on_dimanyNotImplementedErrorr split_factorsumziptuple)r rr my_coordinate local_shape global_offset_shard_idx_stride_by_mesh_dimnum_shards_by_tensor_dimidx placement mesh_dim_size shard_dim local_offset shard_size shard_offsetstrided_shardingstrided_part_seenstrided_part_endshard_idx_stridexy shard_idxs r%compute_local_shape_and_global_offsetrBs5P'')M<( c,// %*3|+<%=( %=QC$))O%= %( %&3\):#: ' 3NC IIcNM)U++%MM !sS%66  3{#33#I;.H[IYHZ[3,5+M+M*!!&"& ,N,( *4I&*6Y' !+|/FF/;/FM,!, 0GG,(3}D374hPZPP !&#l*; ;  %w\):: "+J"7 $ # i// ) I' 21;;D+XcURV)l*DF )36:(3!)];;7;))44? ) 6 6 FH5?D 1; M;4?5?D/#8:4=044/I/s+;]'KL'Ktq!QU'KLM4 03; /JK/Jtq!QU/JMK[!5#777C( rMLs$J9?K J>. K K >K tensorc [UR55n[UR55n[U5GHupVURU5nUR 5(a[ [ U5nURS:a[SU35eURn XR:dSU SURSUS35eX9n X-X9'[[U55Hn X:wdM XKXI:dMXKU-XK'M! M[U[[45(aM[S[!U5S35e X44$) a Compute the global size and stride of a DTensor from the given local tensor. The local size is multiplited by `world_size` per Sharding dim. The local stride is multiplited by `world_size` per Sharding dim, as long as the dimension is outside sharding dim. For example, if we have a local tensor with size (4, 8, 2) and stride (16, 1, 8). If the DTensor placements are [Shard(2)] and world_size is 2; then the global size is (4, 8, 4) and stride is (16 * 2, 1, 8). Args: tensor (:class:`torch.Tensor`): Local tensor which DTensor will be constructed from. mesh (:class:`DeviceMesh`): Object which describes the mesh topology of devices for the DTensor. placements (Sequence[:class:`Placement`]]): The attribute of the DTensor that describes its layout on the mesh topology. Return: tensor_shape: A List of int which specifies the size of DTensor which build on top of the local tensor. tensor_stride: A List of int which specifies the stride of DTensor. rzOShard placements should have negative dims normalized in the user-facing APIs: rrz for placement number .zplacement type z not supported!)r r%strider$is_shardrr r&AssertionErrorr#r"r!rr r RuntimeErrortype) rCrr tensor_shape tensor_strider4r5r6shard_placementr7local_dim_sizeis rcompute_global_tensor_inforPsR8 &L)M#J/ #     "5)4O""Q&$--<,=?(++I{{*  {*DV[[MQghkgllmn **4N&4&DL #3}-.>m&6-:R&R'4'7-'GM$/I 7';<<i0AQR R304  &&op_callargscfUHn[U[R[45(aURs $[U[ [ 45(dMS[U5S:dMd[US[R[45(dMUSRs $ [SUS35e)zy Find the device mesh object from args. It returns None if no mesh is found. NOTE: we can optimize this search if needed rz+Cannot find device mesh from args for op : rE) rdtensorDTensorr device_meshr r-r! ValueError)rRrSargs rtry_find_mesh_from_argsrZs cGOO[9 : :?? " sT5M * *C1 3q6GOO[#ABBq6%% % B7)1M NNrQ global_stridec^^S/[T5-m[U5H{up4UR5(dM[[U5R n[ [T55H-nTUTU:dMTU==URU5-ss'M/ M} [UU4Sj[ [T5555$)z Compute the stride of a local tensor shard, given the global stride of the DTensor. NOTE: Currently this function is assuming the DTensor is evenly shardable. rc3:># UHnTUTU-v M g7frr)rrOr[stride_divisorss rr'compute_local_stride..s$8Q1 aOA..8Qs) r!r$rGrr r&r"r%r-)r[rrmesh_idxrrOjr^s` @rcompute_local_striderbscC ..O ,  ::<<UA""A3}-. #mA&66#A&$))H*==&/ - 8=c->P8Q rQc$[U[R5(aU$[U[5(aU/nOA[ U5S:Xa'[US[ 5(a[ US5nO [ U5n[R"U5$)z Unify variable types of size argument to torch.Size Acceptable types include: int, Sequence[int], Tuple[int], Tuple[Sequence[int]], or torch.Size rr)rtorchSizeintr!rr )r% torch_sizes rnormalize_to_torch_sizerh sn$ ## $V TaJtAw99$q'] $Z ::j !!rQ)#collections.abcrtypingrrdtorch.distributed.tensor._api distributedrC_apirUtorch._prims_commonrtorch.distributed.device_meshr&torch.distributed.tensor._dtensor_specr(torch.distributed.tensor.placement_typesrr r r r r-rfrBTensorr rP_ops OpOverloadobjectrZrbrerhrrQrrvs$$ //)4>Q8Q8#-Q8;CI;NQ8 5c?E#s(O +,Q8h8' LL8' *8'8@8K8' 49d3i  8'vO ZZ " "O*26*:OO*$.