shjSSKrSSKrSSKJrJr SSKJr SSKJ r J r SSK J r J r JrJrJr SSKJr SSKJr SSKJr SS KJrJr /S Qr\"S S S 9r\"S5"SS\55r\"S5"SS\55r"SS\5rSr"SS\\5r "SS\5r!\"S5"SS\55r""SS\\5r#\"S 5"S!S"\55r$\"S#5"S$S%\\%\55r&g)&N)ABCabstractmethod)deque)IteratorSized)AnyCallableLiteralOptionalTypeVar)functional_datapipe)_SnapshotState) IterDataPipe)_check_unpickable_fn StreamWrapper)ConcaterIterDataPipeDemultiplexerIterDataPipeForkerIterDataPipeMultiplexerIterDataPipeZipperIterDataPipe_T_coT) covariantconcatcT\rSrSr%Sr\\\S'S\4SjrS\ 4Sjr S\ 4Sjr Sr g ) ra Concatenates multiple Iterable DataPipes (functional name: ``concat``). The resulting DataPipe will yield all the elements from the first input DataPipe, before yielding from the subsequent ones. Args: datapipes: Iterable DataPipes being concatenated Example: >>> # xdoctest: +REQUIRES(module:torchdata) >>> import random >>> from torchdata.datapipes.iter import IterableWrapper >>> dp1 = IterableWrapper(range(3)) >>> dp2 = IterableWrapper(range(5)) >>> list(dp1.concat(dp2)) [0, 1, 2, 0, 1, 2, 3, 4] datapipesc[U5S:Xa [S5e[SU55(d [S5eXlg)Nrz/Expected at least one DataPipe, but got nothingc3B# UHn[U[5v M g7fN isinstancer.0dps }/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/utils/data/datapipes/iter/combining.py 0ConcaterIterDataPipe.__init__..4D)B:b,//)z(Expected all inputs to be `IterDataPipe`)len ValueErrorall TypeErrorrselfrs r%__init__ConcaterIterDataPipe.__init__1s< y>Q NO OD)DDDFG G"returnc#J# URH nUShvN M gN 7frr)r/r$s r%__iter__ConcaterIterDataPipe.__iter__8s..BMM! s #! #c[SUR55(a[SUR55$[[ U5R S35e)Nc3B# UHn[U[5v M g7frr!rr"s r%r&/ConcaterIterDataPipe.__len__..=>~z"e$$~r)c38# UHn[U5v M g7frr*r"s r%r&r;>82s2ww# instance doesn't have valid length)r,rsumr-type__name__r/s r%__len__ConcaterIterDataPipe.__len__<J >t~~> > >888 8tDz2233VWX Xr2r5N)rD __module__ __qualname____firstlineno____doc__tupler__annotations__r0rr6intrF__static_attributes__r2r%rrs:$\""#<#(YYr2rforkc F\rSrSrSrS S\S\S\S\\S4S jjr S r g) rCa} Creates multiple instances of the same Iterable DataPipe (functional name: ``fork``). Args: datapipe: Iterable DataPipe being copied num_instances: number of instances of the datapipe to create buffer_size: this restricts how far ahead the leading child DataPipe can read relative to the slowest child DataPipe. Defaults to ``1000``. Use ``-1`` for the unlimited buffer. copy: copy strategy to use for items yielded by each branch. Supported options are ``None`` for no copying, ``"shallow"`` for shallow object copies, and ``"deep"`` for deep object copies. Defaults to ``None``. Note: All branches of the forked pipeline return the identical object unless the copy parameter is supplied. If the object is mutable or contains mutable objects, changing them in one branch will affect all others. Example: >>> # xdoctest: +REQUIRES(module:torchdata) >>> from torchdata.datapipes.iter import IterableWrapper >>> source_dp = IterableWrapper(range(5)) >>> dp1, dp2 = source_dp.fork(num_instances=2) >>> list(dp1) [0, 1, 2, 3, 4] >>> list(dp2) [0, 1, 2, 3, 4] Ndatapipe num_instances buffer_sizecopyshallowdeepcUS:a[SUS35eUS:XaU$[XX45n[U5Vs/sHn[XV5PM sn$s snfNz,Expected `num_instances` larger than 0, but z is found)r+_ForkerIterDataPiperange_ChildDataPipe)clsrUrVrWrX containeris r%__new__ForkerIterDataPipe.__new__bsd 1 >}oYW  A O'S 6;M6JK6Jy,6JKKKsArQN) rDrIrJrKrLrrOr r rerPrQr2r%rrCsQB 59 LLL L w012 LLr2rct\rSrSrSr\S\4Sj5r\S\4Sj5r \S Sj5r \S\4S j5r S r g) _ContainerTemplateszVAbstract class for container ``DataPipes``. The followings are three required methods. instance_idcgrrQr/rls r%get_next_element_by_instance/_ContainerTemplate.get_next_element_by_instancev r2r3cgrrQrEs r%is_every_instance_exhausted._ContainerTemplate.is_every_instance_exhaustedzrqr2NcgrrQrEs r%reset_ContainerTemplate.reset~rqr2cg)zSRaise TypeError if it's not supposed to be implemented to support `list(datapipe)`.NrQrns r%get_length_by_instance)_ContainerTemplate.get_length_by_instancesr2rQr3N) rDrIrJrKrLrrOroboolrsrvryrPrQr2r%rjrjsska    T    c#ccr2rjcU$rrQ)xs r%_no_oprs Hr2c \rSrSrSrSS\S\S\S\\S4S jjr S r S \4S jr S \ 4Sjr S \S \4SjrSSjrSrSrSrSrSrg)r_z Container to hold instance-specific information on behalf of ForkerIterDataPipe. It tracks the state of its child DataPipes, maintains the buffer, and yields the next value as requested by the child DataPipes. NrUrVrWrXrYcXlSUlX l[5UlX0lUR S:a[ R"S[5 Uc [Ul OGUS:Xa[RUl O+US:Xa[RUl O[SUS35eS/U-UlSUlSUlSUl['U5Vs/sHnSPM snUlgs snf)NrzPUnlimited buffer size is set for `fork`, please be aware of OOM at random placesrZr[zUnknown copy method `z5` requested, choose one of None, `shallow` or `deep`.T) main_datapipe_datapipe_iteratorrVrbufferrWwarningswarn UserWarningrcopy_fn copymodulerXdeepcopyr+child_pointers slowest_ptr leading_ptrend_ptrr` _child_stop)r/rUrVrWrX_s r%r0_ForkerIterDataPipe.__init__s&;?*"W &   a  MM:  <!DL Y %??DL V^%..DL'v-bc  * *&* 6;M6J'K6J6J'K'Ks C,c,[UR5$rr*rrEs r%rF_ForkerIterDataPipe.__len__4%%&&r2rlc#Z# URcnURU(aZ[UR5Ul[R Ul[UR5HnSURU'M URU(GdURU==S- ss'URb.URUUR:XaSURU'GOUR(aLURUUR::a/URUUR- S- nURUnOEURUUl [UR5nURRU5 URUURS-:XaE[#UR5nURU:a XPl URR%5 UR&S:aCURUR&UR-:a[)SSUR&S3-5eUR+U5v URU(dGMSURU'[-UR5(aSUlUR/5 gg![ a, SURU'SUlURUl GMbf=f!SURU'[-UR5(aSUlUR/5 ff=f7f)NFr^Trz#ForkerIterDataPipe buffer overflow,z buffer size  is insufficient.)rriterrr Iterating_snapshot_stater`rVrrrrrnextappend StopIterationminpopleftrW BufferErrorrr,_cleanup)r/rlrdidx return_valnew_mins r%ro0_ForkerIterDataPipe.get_next_element_by_instances  " " *t/?/? /L&*4+=+=&>D ##1#;#;D 4--.&+  #/- &&{33##K0A50LL,++K8DLLH48D$$[1;;4#6#6{#CtGWGW#W--k:T=M=MMPQQC!%S!1J'+':':;'GD$!%)$*A*A%B  **:6 &&{3t7G7G!7KK!++G'''1+2( ++-$$)((4+;+;d>N>N+NN%=()9)9(::KLM ll:..M&&{33P-1D  [ )4##$$*.' %3)!8<((526/'+'7'7  !.-1D  [ )4##$$*.' %sFA>> # xdoctest: +REQUIRES(module:torchdata) >>> # Singler Iterator per IteraDataPipe Invalidation >>> from torchdata.datapipes.iter import IterableWrapper >>> source_dp = IterableWrapper(range(10)) >>> cdp1, cdp2 = source_dp.fork(num_instances=2) >>> it1, it2 = iter(cdp1), iter(cdp2) >>> it3 = iter(cdp1) >>> # The line above invalidates `it1` and `it2`, and resets `ForkerIterDataPipe`. >>> it4 = iter(cdp2) >>> # The line above doesn't invalidate `it3`, because an iterator for `cdp2` hasn't been created since >>> # the last invalidation. Args: main_datapipe: Main DataPipe with a method 'get_next_element_by_instance(instance_id)' instance_id: integer identifier of this instance T_is_child_datapiperrlcJ[U[5(deXlX lgr)r!rjrrl)r/rrls r%r0_ChildDataPipe.__init__Fs"-);<<<<+8&r2cLURRUR5$r)rrorlrEs r%r6_ChildDataPipe.__iter__Ls!!!>>t?O?OPPr2cLURRUR5$r)rryrlrEs r%rF_ChildDataPipe.__len__Qs!!889I9IJJr2r3cURRcSURlOURRUR:XasUR=RS- slURR5(d[R"S[ 5 URR 5 URRUlUR$)z Update the valid iterator ID for both this DataPipe object and `main_datapipe`. `main_datapipe.reset()` is called when the ID is incremented to a new generation. rr^zSome child DataPipes are not exhausted when __iter__ is called. We are resetting the buffer and each child DataPipe will read from the start again.)rrrsrrrrvrEs r%$_set_main_datapipe_valid_iterator_id3_ChildDataPipe._set_main_datapipe_valid_iterator_idUs    0 0 845D   1   2 2d6M6M M    1 1Q 6 1%%AACC Y    $ $ &#'"4"4"G"G&&&r2c\XR:H=(a XRR:H$)zXCheck the valid iterator ID against that of DataPipe object and that of `main_datapipe`.)rr)r/ iterator_ids r%_check_valid_iterator_id'_ChildDataPipe._check_valid_iterator_idps- 22 2 E11DDD r2)rrlrN)rDrIrJrKrLrr|rNrrOr0r6rFrrrPrQr2r%rara&sK: $#'l'' Q K'c'6 t r2rademuxc P\rSrSrSrS S\S\S\\/\ \4S\ S\4 Sjjr S r g ) rixa Splits the input DataPipe into multiple child DataPipes, using the given classification function (functional name: ``demux``). A list of the child DataPipes is returned from this operation. Args: datapipe: Iterable DataPipe being filtered num_instances: number of instances of the DataPipe to create classifier_fn: a function that maps values to an integer within the range ``[0, num_instances - 1]`` or ``None`` drop_none: defaults to ``False``, if ``True``, the function will skip over elements classified as ``None`` buffer_size: this defines the maximum number of inputs that the buffer can hold across all child DataPipes while waiting for their values to be yielded. Defaults to ``1000``. Use ``-1`` for the unlimited buffer. Examples: >>> # xdoctest: +REQUIRES(module:torchdata) >>> from torchdata.datapipes.iter import IterableWrapper >>> def odd_or_even(n): ... return n % 2 >>> source_dp = IterableWrapper(range(5)) >>> dp1, dp2 = source_dp.demux(num_instances=2, classifier_fn=odd_or_even) >>> list(dp1) [0, 2, 4] >>> list(dp2) [1, 3] >>> # It can also filter out any element that gets `None` from the `classifier_fn` >>> def odd_or_even_no_zero(n): ... return n % 2 if n != 0 else None >>> dp1, dp2 = source_dp.demux(num_instances=2, classifier_fn=odd_or_even_no_zero, drop_none=True) >>> list(dp1) [2, 4] >>> list(dp2) [1, 3] rUrV classifier_fn drop_nonerWcUS:a[SUS35e[U5 [XX4U5n[U5Vs/sHn[ Xg5PM sn$s snfr])r+r_DemultiplexerIterDataPiper`ra)rbrUrVrrrWrcrds r%re!DemultiplexerIterDataPipe.__new__sd 1 >}oYW  ]+ /x bmn 6;M6JK6Jy,6JKKKsArQN)Frh) rDrIrJrKrLrrOr rr r|rerPrQr2r%rrxsa!P  LLL # 67 L  L  LLr2rc \rSrSrSrS\\S\S\\/\ \4S\ S\4 Sjr S \S \4S jr S \4S jr S \ 4S jrS \S \4SjrSSjrSrSrSS \ \4SjjrSrSrg)riz Container to hold instance-specific information on behalf of DemultiplexerIterDataPipe. It tracks the state of its child DataPipes, maintains the buffer, classifies and yields the next correct value as requested by the child DataPipes. rUrVrrrWchXlSUlX lXPlURS:a[R "S[ 5 SUl[U5Vs/sH n[5PM snUl X0l X@l SUl [U5Vs/sHnSPM snUlgs snfs snf)NrzQUnlimited buffer size is set for `demux`, please be aware of OOM at random placesFT)rrrVrWrrrcurrent_buffer_usager`r child_buffersrrmain_datapipe_exhaustedr)r/rUrVrrrWrs r%r0#_DemultiplexerIterDataPipe.__init__s&;?*&   a  MM:  %&!CHCW1XCWa%'CW1X*"',$6;M6J'K6J6J'K 2Y(Ls B* B/rlr3cUR(dURU(a[eURc [ S5e[ UR5nUR U5nUc)UR(a[R"U5 MUbX0R:dUS:a"[ SURS- S3US3-5eX1:XaU$URURU5 U=RS- sl URS:a3URUR:a[SURS35eGMO) Nz_datapipe_iterator has not been set, likely because this private method is called directly without invoking get_next_element_by_instance() first.rz8Output of the classification fn should be between 0 and r^z. z is returned.z7DemultiplexerIterDataPipe buffer overflow, buffer size r)rrrrr+rrrrrrVrrrrWr)r/rlvalueclassifications r% _find_next%_DemultiplexerIterDataPipe._find_nextsN++t/?/? /L##&&. M001E!//6N%$..++E2&!%7%77!A% NtOaOadeOeNffhi'( 67,   ~ . 5 5e <  % % * %1$)B)BTEUEU)U!MdN^N^M__pq7r2c## URcuURU(aa[UR5Ul[R UlSUl[UR5HnSURU'M URU(dsURU(a5U=RS-sl URUR5v OURU5v URU(dMsSURU'[UR5(aSUlURU(aUR!U5 gg![a SURU'SUlSUlNf=f!SURU'[UR5(aSUlURU(aUR!U5 ff=f7f)NFr^T)rrrrrrrrr`rVrrrrrr,r)r/rlrds r%ro7_DemultiplexerIterDataPipe.get_next_element_by_instances  " " *t/?/? /L&*4+=+=&>D #((  ,1D (4--.&+  #/ +&&{3%%k2--2-,,[9AACC7"ook:: &&{33-1D  [ )4##$$*.'!!+. k*/)78<((57;426/7 -1D  [ )4##$$*.'!!+. k*/sEBG,AF$E$7F AG,$'F F FFAG))G,cRUR=(a [UR5$r)rr,rrEs r%rs6_DemultiplexerIterDataPipe.is_every_instance_exhausteds++ED4D4D0EEr2c[er)r-rns r%ry1_DemultiplexerIterDataPipe.get_length_by_instancesr2NcSUlSUl[UR5Vs/sH n[ 5PM snUl[UR5Vs/sHnSPM snUlSUlgs snfs snfNrTF)rrr`rVrrrrrs r%rv _DemultiplexerIterDataPipe.resetsm"&$%!/4T5G5G/HI/H!eg/HI*/0B0B*CD*CQD*CD',$JDs A5 A:cURURURURURUR UR 4n[Rb[R"U5$U$r) rrVrWrrrrrrrs r%r'_DemultiplexerIterDataPipe.__getstate__sj             NN  # #  + +   % % 1--e4 4 r2cXUuUlUlUlUlUlUlUlSUlSUl[UR5Vs/sH n[5PM snUl [UR5Vs/sHnSPM snUl SUl gs snfs snfr)rrVrWrrrrrrr`rrrrrs r%r'_DemultiplexerIterDataPipe.__setstate__-s          N  #  +"&$%!/4T5G5G/HI/H!eg/HI*/0B0B*CD*CQD*CD',$JDs B" B'cUc[UR5OU/nUHJnURUnU(dMUR5n[R "U5 U(aM/ML gr)r`rVrrrr)r/rlidsrdqrs r%r#_DemultiplexerIterDataPipe._cleanup=sj" $$$ % A""1%A!IIK++A.!r2c$UR5 grrrEs r%r"_DemultiplexerIterDataPipe.__del__Krr2) rrrrrrWrrrrrrrVr{r)rDrIrJrKrLrrrOr r r|r0rrorsryrvrrrrrPrQr2r%rrsLu%LL # 67 L  L  L2ce@++<FTF##- - /HSM /r2rmuxcF\rSrSrSrSrSrSrS SjrSr S r S r S r g) riOa Yields one element at a time from each of the input Iterable DataPipes (functional name: ``mux``). As in, one element from the 1st input DataPipe, then one element from the 2nd DataPipe in the next iteration, and so on. It ends when the shortest input DataPipe is exhausted. Args: datapipes: Iterable DataPipes that will take turn to yield their elements, until the shortest DataPipe is exhausted Example: >>> # xdoctest: +REQUIRES(module:torchdata) >>> from torchdata.datapipes.iter import IterableWrapper >>> dp1, dp2, dp3 = IterableWrapper(range(3)), IterableWrapper(range(10, 15)), IterableWrapper(range(20, 25)) >>> list(dp1.mux(dp2, dp3)) [0, 10, 20, 1, 11, 21, 2, 12, 22] cXl/Ulgr)rrr.s r%r0 MultiplexerIterDataPipe.__init__bs"  r2c## URVs/sHn[U5PM nn[U5(aqUH*n[U5nURR U5 M, URShvN URR5 [U5(aMpggs snf![ a URR5  gf=fNb7fr)rrr*rrrrclear)r/r~ iteratorsitrs r%r6 MultiplexerIterDataPipe.__iter__hs&*nn5nT!Wn 5)nn HEKK&&u- {{ " " KK   )nn6 %KK%%' #sDCB*C&B/$C7C8.C(C/$CCCCc[SUR55(a2[SUR55[UR5-$[ [ U5R S35e)Nc3B# UHn[U[5v M g7frr:r"s r%r&2MultiplexerIterDataPipe.__len__..vr<r)c38# UHn[U5v M g7frr>r"s r%r&rwr?r@rA)r,rrr*r-rCrDrEs r%rFMultiplexerIterDataPipe.__len__usX >t~~> > >8883t~~;NN NtDz2233VWX Xr2Nc/Ulgr)rrEs r%rvMultiplexerIterDataPipe.reset{s  r2cURURUR4n[Rb[R"U5$U$r)rrrrrrs r%r$MultiplexerIterDataPipe.__getstate__~sF NN  # #  + +   % % 1--e4 4 r2c<UuUlUlUl/Ulgr)rrrrrs r%r$MultiplexerIterDataPipe.__setstate__s$   N  #  + r2c8URR5 gr)rrrEs r%rMultiplexerIterDataPipe.__del__s r2)rrrrr{) rDrIrJrKrLr0r6rFrvrrrrPrQr2r%rrOs+"  Y r2rzipcn^\rSrSr%Sr\\\S'S\4U4SjjrS\ \\ 4Sjr S\ 4Sjr SrU=r$) ria5 Aggregates elements into a tuple from each of the input DataPipes (functional name: ``zip``). The output is stopped as soon as the shortest input DataPipe is exhausted. Args: *datapipes: Iterable DataPipes being aggregated Example: >>> # xdoctest: +REQUIRES(module:torchdata) >>> from torchdata.datapipes.iter import IterableWrapper >>> dp1, dp2, dp3 = IterableWrapper(range(5)), IterableWrapper(range(10, 15)), IterableWrapper(range(20, 25)) >>> list(dp1.zip(dp2, dp3)) [(0, 10, 20), (1, 11, 21), (2, 12, 22), (3, 13, 23), (4, 14, 24)] rcr>[SU55(d [S5e[TU] 5 Xlg)Nc3B# UHn[U[5v M g7frr r"s r%r&.ZipperIterDataPipe.__init__..r(r)zCAll inputs are required to be `IterDataPipe` for `ZipIterDataPipe`.)r,r-superr0r)r/r __class__s r%r0ZipperIterDataPipe.__init__s6D)DDDU  "r2r3c## URVs/sHn[U5PM nn[U6ShvN gs snfN 7fr)rrr)r/rUrs r%r6ZipperIterDataPipe.__iter__s748NNCNT(^N C ?""D"sA9 A> Ac[SUR55(a[SUR55$[[ U5R S35e)Nc3B# UHn[U[5v M g7frr:r"s r%r&-ZipperIterDataPipe.__len__..r<r)c38# UHn[U5v M g7frr>r"s r%r&rr?r@rA)r,rrr-rCrDrEs r%rFZipperIterDataPipe.__len__rHr2r5)rDrIrJrKrLrMrrNr0rrr6rOrFrP __classcell__)rs@r%rrsJ \""#<##(5<0#YYYr2r)'rXrrabcrr collectionsrcollections.abcrrtypingrr r r r %torch.utils.data.datapipes._decoratorr )torch.utils.data.datapipes._hook_iteratorr#torch.utils.data.datapipes.datapiper'torch.utils.data.datapipes.utils.commonrr__all__rrrrjrr_rarrrrMrrQr2r%r's=#+<<ED<W  4(X$Y<$Y$YNV,L,L,L^cc( X,(:XvO \O dW7L 7L7LtY/AYxUAlAAHU#YeEl3#Y#Yr2