shB0$SSKrSSKJr SSKJrJr SSKJrJrJ r J r SSK r SSK Jr SSKJrJr SSKJr /SQr\ "S S S 9rS \4S jr\"S5"SS\\55r\"S5"SS\55r\"S5"SS\\55rg)N) defaultdict)IteratorSized)AnyCallableOptionalTypeVar)functional_datapipe) DataChunk IterDataPipe)_check_unpickable_fn)BatcherIterDataPipeGrouperIterDataPipeUnBatcherIterDataPipe_T_coT) covariantnamecUS;ac[R"SUSUS3[SS9 [[R R RRRU5$[S[SU35e) N)SHARDING_PRIORITIESShardingFilterIterDataPipe`zc` from `torch.utils.data.datapipes.iter.grouping` is going to be removed in PyTorch 2.1Please use `z5` from the `torch.utils.data.datapipes.iter.sharding`)category stacklevelzmodule z has no attribute ) warningswarn FutureWarninggetattrtorchutilsdata datapipesitershardingAttributeError__name__)rs |/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/utils/data/datapipes/iter/grouping.py __getattr__r(s{ DD v& U W"  u{{''1166??FF 78*,>tfE FFbatchc ^\rSrSr%Sr\\S'\\S'\\S'S\ 4S\S\S\S\ \ SS 4 U4S jjjr S\ \ 4S jr S\4S jrS rU=r$)r%a Creates mini-batches of data (functional name: ``batch``). An outer dimension will be added as ``batch_size`` if ``drop_last`` is set to ``True``, or ``length % batch_size`` for the last batch if ``drop_last`` is set to ``False``. Args: datapipe: Iterable DataPipe being batched batch_size: The size of each batch drop_last: Option to drop the last batch if it's not full wrapper_class: wrapper to apply onto each batch (type ``List``) before yielding, defaults to ``DataChunk`` Example: >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> dp = IterableWrapper(range(10)) >>> dp = dp.batch(batch_size=3, drop_last=True) >>> list(dp) [[0, 1, 2], [3, 4, 5], [6, 7, 8]] datapipe batch_size drop_lastF wrapper_classreturnNcl>US:dS5e[TU]5 XlX lX0lX@lg)Nrz+Batch size is required to be larger than 0!)super__init__r-r.r/r0)selfr-r.r/r0 __class__s r'r4BatcherIterDataPipe.__init__As7A~LLL~  $"*r)c#"# /nURHDnURU5 [U5UR:XdM/UR U5v /nMF [U5S:a&UR (dUR U5v ggg7fNr)r-appendlenr.r0r/)r5r*xs r'__iter__BatcherIterDataPipe.__iter__OsxA LLO5zT__,((//  u:>>>((//" s :BABcL[UR[5(aeUR(a"[ UR5UR -$[ UR5UR -S- UR -$[ [U5RS35e)Nz# instance doesn't have valid length) isinstancer-rr/r;r. TypeErrortyper&r5s r'__len__BatcherIterDataPipe.__len__Zsv dmmU + +~~4==)T__<<DMM*T__>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> source_dp = IterableWrapper([[[0, 1], [2]], [[3, 4], [5]], [[6]]]) >>> dp1 = source_dp.unbatch() >>> list(dp1) [[0, 1], [2], [3, 4], [5], [6]] >>> dp2 = source_dp.unbatch(unbatch_level=2) >>> list(dp2) [0, 1, 2, 3, 4, 5, 6] r- unbatch_levelcXlX lgNr-rS)r5r-rSs r'r4UnBatcherIterDataPipe.__init__|s  *r)c#x# URH$nURXRS9ShvN M& gN 7f)NrS)r-_diverS)r5elements r'r=UnBatcherIterDataPipe.__iter__s0}}Gzz'9K9KzL L L% Ls ,:8 :c## US:a [S5eUS:XaB[U[[45(a"UHnUR USS9ShvN M gUv gUS:XaUv g[U[[45(a$UHnUR X2S- S9ShvN M g[ SUR S35eNqN$7f)Nz unbatch_level must be -1 or >= 0rYrr@zunbatch_level z" exceeds the depth of the DataPipe) ValueErrorrAlistr rZ IndexErrorrS)r5r[rSitems r'rZUnBatcherIterDataPipe._dives 2 ?@ @ B 'D)#455#D#zz$bzAAA$ a M'D)#455#D#zz$a>OzPPP$!$T%7%7$88Z[BQs%A C B?ACC#CCrVN)r@) r&rGrHrIrJr rLr4r=rZrNr)r'rrds%,++c+Mr)rgroupbyc\rSrSrSrSSSSSS.S\\S\\/\4S \ S \ S \ \ S \ \ S \ 4Sjjr Sr SrSSjrSrSrSrSrg)ra Groups data from IterDataPipe by keys from ``group_key_fn``, yielding a ``DataChunk`` with batch size up to ``group_size``. (functional name: ``groupby``). The samples are read sequentially from the source ``datapipe``, and a batch of samples belonging to the same group will be yielded as soon as the size of the batch reaches ``group_size``. When the buffer is full, the DataPipe will yield the largest batch with the same key, provided that its size is larger than ``guaranteed_group_size``. If its size is smaller, it will be dropped if ``drop_remaining=True``. After iterating through the entirety of source ``datapipe``, everything not dropped due to the buffer capacity will be yielded from the buffer, even if the group sizes are smaller than ``guaranteed_group_size``. Args: datapipe: Iterable datapipe to be grouped group_key_fn: Function used to generate group key from the data of the source datapipe keep_key: Option to yield the matching key along with the items in a tuple, resulting in `(key, [items])` otherwise returning [items] buffer_size: The size of buffer for ungrouped data group_size: The max size of each group, a batch is yielded as soon as it reaches this size guaranteed_group_size: The guaranteed minimum group size to be yielded in case the buffer is full drop_remaining: Specifies if the group smaller than ``guaranteed_group_size`` will be dropped from buffer when the buffer is full Example: >>> import os >>> # xdoctest: +SKIP >>> from torchdata.datapipes.iter import IterableWrapper >>> def group_fn(file): ... return os.path.basename(file).split(".")[0] >>> source_dp = IterableWrapper(["a.png", "b.png", "a.json", "b.json", "a.jpg", "c.json"]) >>> dp0 = source_dp.groupby(group_key_fn=group_fn) >>> list(dp0) [['a.png', 'a.json', 'a.jpg'], ['b.png', 'b.json'], ['c.json']] >>> # A group is yielded as soon as its size equals to `group_size` >>> dp1 = source_dp.groupby(group_key_fn=group_fn, group_size=2) >>> list(dp1) [['a.png', 'a.json'], ['b.png', 'b.json'], ['a.jpg'], ['c.json']] >>> # Scenario where `buffer` is full, and group 'a' needs to be yielded since its size > `guaranteed_group_size` >>> dp2 = source_dp.groupby(group_key_fn=group_fn, buffer_size=3, group_size=3, guaranteed_group_size=2) >>> list(dp2) [['a.png', 'a.json'], ['b.png', 'b.json'], ['a.jpg'], ['c.json']] Fi'N)keep_key buffer_size group_sizeguaranteed_group_sizedrop_remainingr- group_key_fnrhrirjrkrlc4[U5 XlX lX0lX@l[ [ 5UlSUlXPl SUl UbUbSUs=:aU::de eXPl UbUbSUs=:aU::de eX`l Xpl [Ul gr9)r r-rmrhmax_buffer_sizerr`buffer_elementscurr_buffer_sizerjrkrlr r0)r5r-rmrhrirjrkrls r'r4GrouperIterDataPipe.__init__s \* ( *7B47H !$%)"  !k&=z0[0 00 00)3 & ,)a2G.U:.U UU.U UU)> &,&r)cSnSnSnURR5H;n[URU5U:dM![URU5nUnM= URbBX R:a3UR(d"[ S[ URU55eURbX R:aURUnU=RU-slURU U$)NrzFailed to group items)rpkeysr;rkrl RuntimeErrorstrrq)r5 biggest_key biggest_sizeresult_to_yieldfindkeys r'_remove_biggest_key'GrouperIterDataPipe._remove_biggest_keys  ++002G4''01L@"4#7#7#@A % 3  & & 2999'''T-A-A+-N)O   & & .999"22;?O -   -r)c## URGHEnURU5nURURU5 U=RS- slUR bUR [ URU5:XanURURU5nUR(aX#4OUv U=R[ URU5-slURU URUR:XdGMUR5nUcGMURU5nUR(aX#4OUv GMH [URR55HcnURURRU55nU=R[ U5-slUR(aX#4OUv Me g7f)Nr@)r-rmrpr:rqrjr;r0rhror{tuplertpop)r5r<keyresultrys r'r=GrouperIterDataPipe.__iter__sA##A&C   % , ,Q /  ! !Q & !*t#$$S)C0*.););Drs#+33/EGH  4( Gc GW;Y,y1;Y;Y|Y1L1 1hYh%,y1h% h%r)