sh;?%SSKrSSKrSSKJrJr SSKrSSKJr SSKJ r SSK J r J r SSK Jr SSKJr /SQrSq\S\S 'S \ S \4S jrS \S \ S S4Sjr"SS5r"SS\5rS \ S \R44Sjrg)N)OptionalUnion)Tensor)_get_device_handle DeviceMesh) DTensorSpec)Shard)is_rng_supported_mesh manual_seedOffsetBasedRNGTracker_RNGStateTracker _rng_tracker device_meshreturnc[UR5nU(a[US5(ag[R"SURS35 g)aChecks if the current device of ``device_mesh`` supports DTensor's random APIs. Currently DTensor Random APIs only supports cuda/cuda-like devices. We suggest users call this API to test the availability before using our random APIs. Args: device_mesh (:class:`DeviceMesh`): The device mesh on which we check if the random ops APIs are supported. Returns: A bool value. True if ``device_mesh`` supports DTensor Random APIs; False otherwise. .. warning:: Currently we only support correct RNG on cuda/cuda-like devices. set_rng_stateTz:DTensor random operators may not have complete support on device meshF)r device_typehasattrwarningswarn)r device_handles t/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/tensor/_random.pyr r sM'{'>'>?M@@  HI`I`Haam n seedc[U5(d%[R"SURS35 g[(d [ USS9qUR 5b[RU5 g[S5e)a9Sets the seed for generating random numbers for the calling rank. Args: seed (int): The desired seed. device_mesh (:class:`DeviceMesh`): The device mesh to set the seed. It is required that the ``device_mesh`` include the calling rank. This is to ensure that the SPMD region maintains a synchronous RNG state, which means no ranks should be initialized with values other than ``seed``. Returns: None .. warning:: :func:`manual_seed` does not check the ``seed`` value correctness. Users must ensure on their own that the value passed in is the desired ``seed`` for ranks within ``device_mesh``. If ``device_mesh`` is a sub-mesh and the calling rank is not a part of it, ``manual_seed`` will throw an error. Current implementation only supports a GPU device mesh. z7DTensor manual_seed() may not have complete support on rNF)run_state_synczmanual_seed requires the current rank to be a part of the device mesh otherwise DTensor RNG state on the rank will not be initialized and the behavior of DTensor random ops is undefined.) r rrrrr get_coordinate _manual_seed RuntimeError)rrs rr r 2s|* ! - -  ))*, 8   <,[O !!#/!!$' ?  rc\rSrSrSrS\R 4Sjr\S\ \ \ 44Sj5r \S\ 4Sj5r\RSS j5rS\ 4S jrS \ S\4S jrS \ S \SS4SjrS\4SjrS\SS4SjrSrg)r _ay _RNGStateTracker stores Random Number Generator (RNG) state (a ByteTensor object) in a dict, mapping from a corresponding tag to each state tensor. It also provides a set of convenient utility methods to help access/modify the state tensors. The most important interface is _distribute_region which will be used when DTensor executes a random op (an operator that calls RNG). devicec2Xl[URR5UlUR(aURR 5(d/[ UR RSURS35e0UlSUl g)Nz( instantiation requires the presence of z device but couldn't find.T) _devicertype_device_handle is_availabler __class____name___states_use_distribute_region)selfr#s r__init___RNGStateTracker.__init__hs} 01B1BC##(;(;(H(H(J(J>>**++S;;-9;  +- &*#rrcUR$N)r+r-s r rng_states_RNGStateTracker.rng_statests ||rcUR$r1r,r2s rdistribute_region_enabled*_RNGStateTracker.distribute_region_enabledxs***rNcXlgr1r6)r-values rr7r8|s&+#rcXR;$r1)r3)r-names rrng_state_is_sync"_RNGStateTracker.rng_state_is_syncs&&rr<cXR;a$[URRSU35eURUSSR [ R S9n[UR55$)N does not have random state for rdtype r3r r)r*viewtorchint64intitem)r-r< seed_tensors rget_seed_RNGStateTracker.get_seedsp  &>>**++KD6R t,a277ekk7J ;##%&&rrcT[R"U/[RSS9R[R5n[R"S/[RSS9R[R5n[R "X4/5UR U'g)NcpurCr#r)rFtensoruint64rEuint8catr3)r-r<rrJ offset_tensors rset_seed_RNGStateTracker.set_seedstllD6eLQQ KK  aS UKPP KK !& ;*F Grspeccgr1)r-rWs r_distribute_region#_RNGStateTracker._distribute_region r parallel_seedcgr1rYr-r]s rr_RNGStateTracker._manual_seedr\r)r%r'r+r,)rN)r* __module__ __qualname____firstlineno____doc__rFr#r.propertydictstrrr3boolr7setterr=rHrKrUrrZr__static_attributes__rYrrr r _s +u|| +Df-+4++%%,&,'''S'S'HSHHH {  # $ rc^\rSrSrSrSS\S\4U4SjjjrS\SS4S jr \ RS \ 4S j5r S \S\4S jrS \S\SS4SjrS \ SS4SjrS \ S\SS4SjrS\\S\\S\4SjrSrU=r$)r z This subclass of ``_RNGStateTracker`` defines the default policy of how RNG states should be shared and synchronized among all ranks to respect the semantics of DTensor random operators. note: _RNGStateTracker only supports cuda/cuda-like device. rrc>[TU][US95 URceURR S:Xa9[ URRSURR S35eURR5RUR5nU(a[R"US5 URS5URS'g)N)rrNzG instantiation requires the presence of CUDA/CUDA-like/XPU device. Got z instead.r parallel-rng)superr._resolve_devicer'r%r&r r)r* get_rng_statetodist broadcastr3)r-rr rng_stater)s rr.OffsetBasedRNGTracker.__init__s [AB""... <<   %>>**+,226,,2C2C1DIO  ''557::4<<H  NN9a (*3,,u*='rr]rNc(URSU5 g)Nrn)rUr_s rr"OffsetBasedRNGTracker._manual_seeds nm4rrWc#4# URS5(d [S5eUR(aURS5nUR U5 [ R RUR/URRS9 URceURRURS5 Sv URX5 SSS5 gSv g!URX5 f=f!,(df  g=f7f)NrnzlOffsetBasedRNGTracker requires the random state to be synchronized before entering into a distribute region!)devicesr)r=r r7 get_offset_set_pre_op_offsetrFrandomfork_rngr%r&r'rr3_set_post_op_offset)r-rW old_offsets rrZ(OffsetBasedRNGTracker._distribute_regions%%n55<   ) )8J  # #D )&&DLL4E4E'**666##11$//.2QR?,,T> ,,T>s6BD8D C1D#D1DD DDr<cXR;a$[URRSU35eURUSSR [ R S9n[UR55$)Nr@rArBrD)r-r<rTs rr{ OffsetBasedRNGTracker.get_offsetsp  &>>**++KD6R .388u{{8K =%%'((roffsetcZXR;a$[URRSU35eURUSSn[R "U/[R SS9R[R5n[R"X4/5URU'g)Nr@rrArNrO) r3r r)r*rFrPrQrErRrS)r-r<rrJrTs r set_offset OffsetBasedRNGTracker.set_offsets  &>>**++KD6R t,a2  fXU\\%PUU KK !& ;*F GrcbURnURnS/UR-n[UR5HcupV[ U[ 5(dMURnXGS:XaU/XG'M7XGn[ U[5(deURU5 Me UR5n U ceURn /n /n UHn SnSn[ U [5(aMU Vs/sHnU UPM nnU Vs/sHnU UPM nn[UU5HunnUU-U-nUU-nM U RU5 U RU5 M URX5n[U5n[UR5HTunn[ U[ 5(dMURU5nURnURX'USSS9SUU'MV SSKJn U"U5nUR#S5nUU-S -S -S -nUR%SUU-5 gs snfs snf) aO Set the starting RNG offset for current device's local shard before actual op execution. The pre_op_offset value should start from the current RNG offset and increment by the size of local shard until it reaches the size of the whole DTensor. For different ranks that hold the same DTensor shard, their pre_op_offset will be the same. Args: spec (:class:`DTensorSpec`): the spec of the DTensor object on which we prepare the offset for running random ops. Returns: None .. warning:: Note that, current implementation does not consider DTensor's continguity. Example: take a DTensor of shape [8, 16] as an example. Assume that the DTensor is placed on a device mesh with placements ([Shard(1), Replicate(), Shard(0)]), and the mesh is: [[[0, 1], [2, 3]], [[4, 5], [6, 7]]] ``spec.mesh.get_coordinate()`` provides the coordinate of the current rank in the mesh. For example, the coordinate of rank 5 is (1, 0, 1). Another concept to introduce besides rank coordinate is shard coordinate. Each rank holds a local shard of the DTensor. In the example, the DTensor is partitioned into 4 [4, 8] shards. The first shard has 2 replicas and rank 0 (coord (0, 0, 0)) and rank 2 (coord (0, 1, 0)) have 1 replica each. That being said, the local shard on rank 0 and rank 2 correspond to the same shard of the DTensor. To denote each DTensor shard, we use a shard coordinate (in the example, it will be a tuple (i, j) where shard (i, j) has the slice DTensor[4 * i : 4 * (i + 1), 8 * j : 8 * (j + 1)], 0 <= i < 2, 0 <= j < 2). Once we have rank coordinate and shard coordinate, we can calculate on each rank what shard of the DTensor the rank holds, with the help of dim_map. The dim_map of the above DTensor is [2, 0] so the shard coordinate of a rank with rank coord (x, y, z) is simply (z, x) by taking(rank_coord[dim_map[0]],rank_coord[dim_map[1]]). Following this calculation, rank 0 and rank 2 holds the shard of coord (0, 0); rank 1 and rank 3 holds the shard of coord (0, 1); rank 4 and rank 6 holds the shard of coord (1, 0); rank 5 and rank 7 holds the shard of coord (1, 1); The last value to calculate before obtaining the starting offset is the shard linear index. The starting offset for each rank will be its shard_linear_index * local_tensor_numel. NrF) return_offsetprodrn)shapemeshndim enumerate placements isinstancer dimlistappendrzip_calc_shard_linear_idxsize_local_shard_size_on_dim#torch.distributed.tensor._ops.utilsrr{r)r-rW dtensor_shaperdim_mapi placement shard_dim mesh_dim_listmesh_coordinate mesh_sizeshard_idx_by_dimtotal_num_shards_by_dimmesh_dim shard_idxtotal_num_shardsd rank_coord num_shardsidxrshard_linear_idxlocal_size_on_rank_0 mesh_dim_sizer local_sizecurrent_offset offset_incrs rr|(OffsetBasedRNGTracker._set_pre_op_offsetsN^ yy13tdii/?%doo6LA)U++%MM %+*+G&$+$6M%mT::::!((+7--/***JJ "$HI (D)):BC(Qoa0( C4<=HqilH =!$Z!B  (DEMD=s ,H'H,rcvURnSSKJn U"U5nUS-S-S-nURSX%-5 g)aSets the RNG to a synchronized state after running the local random op. Every rank should set its RNG offset to `old_offset + DTensor.numel()` where old_offset is the offset before calling `set_pre_op_offset` i.e. the offset before running DTensor random ops. Args: spec (:class:`DTensorSpec`): the spec of the DTensor object on which we post-process the offset for running random ops. Returns: None rrrrrnN)rrrr)r-rWrrrnumels rr)OffsetBasedRNGTracker._set_post_op_offset_s@ <]#q 1$  (:;r shard_coord shard_sizecrSnSn[[U5[U55HupVX5U-- nXF-nM U$)Nrr)rreversed)r-rrrshard_coord_striderrs rr,OffsetBasedRNGTracker._calc_shard_linear_idxvsNXk2HZ4HIIC &8 8 8   & J rrY)T)r*rarbrcrdrrhr.rHr contextlibcontextmanagerrrZrgr{rr|rrrrj __classcell__)r)s@rr r s $>>>>*5#5$5{0)s)s) Hs HC HD HrF{rFtrFh< <<<.  9  26s)     rr cURn[U5nUceUR5UR5-n[R "USUS35$)N:r)rrget_rank device_countrFr#)rrr device_idxs rrprps[))K&{3M  $$ $%%'-*D*D*FFJ <<;-qA7 88r)rrtypingrrrFtorch.distributed distributedrsrtorch.distributed.device_meshrr&torch.distributed.tensor._dtensor_specr(torch.distributed.tensor.placement_typesr __all__r__annotations__rhr rHr r r r#rprYrrrs" H>:  .2 h)*1zd4* c* * t* Z: : zd ,d N99 9r