sh' \SSKrSSKrSSKJr SSKJrJrJr SSKrSSK J r SSK J r SSKJrJr SSKJr SSKJrJr SSKJrJrJrJrJr SS KJrJrJr SS K J!r! SS K"J#r# SS K$J%r% SS K&J'r' SSK(J)r) /SQr*"SS\'5r+"SS\5r,S\\-\R\4S\\-\R\44Sjr/S\\-\R\4S\\-\R\44Sjr0\1S:XGa"SS\5r2\Rf"5r4\4RkS\-S\2Vs/sHoRlPM sn\2RnS9 \4RkS\-SS 9 \4RkS!\-S"S 9 \4Rq5r9\:"S#\9RvS$\9RxS%\9RzS&35 S'\9RvS(3r>\9Rz\2RnRl:XaQ\R~R\9Rv5(a\0"\9Rv\9Rx5 g\:"\>5 g\9Rz\2RRl:XaQ\R~R\9Rv5(a\/"\9Rv\9Rx5 g\:"\>5 g\C"S)\9Rz35egs snf)*N)Enum)castOptionalUnion)narrow_tensor_by_index)FileSystemReaderFileSystemWriter)flatten_state_dict)_EmptyStateDictLoadPlannerDefaultLoadPlanner)MetadataSTATE_DICT_TYPE STORAGE_TYPESTensorPropertiesTensorStorageMetadata) LoadItemTypeLoadPlan LoadPlanner)_create_chunk_list)_load_state_dict)_save_state_dict) StorageReader)Future)dcp_to_torch_savetorch_save_to_dcpBroadcastingTorchSaveReaderDynamicMetaLoadPlannerc:\rSrSrSrSS\\\\R4S\ SS4Sjjr S\ 4Sjr S \S \S\S4S jrS \ S \SS4SjrS \S\4SjrS\\S\\4SjrSS\\\RS4SS4Sjjr\S\\\R4S\4Sj5rSrg)r'a StorageReader for reading a Torch Save file. This reader will read the entire checkpoint on the coordinator rank, and then broadcast and shard each tensor to all ranks. . N.B. Intended to be used with DynamicMetaLoadPlanner .. warning:: Current implementation only supports loading Tensors. >>> # xdoctest: +SKIP("undefined vars") >>> sd = {"mode": model} >>> dcp.load( >>> sd, >>> storage_reader=BroadcastingTorchSaveReader(), >>> planner=DynamicMetaLoadPlanner(), >>> checkpoint_id="path_to_model.pt" >>> ) N checkpoint_idcoordinator_rankreturncXlX lgN)r r!)selfr r!s }/Users/tiagomarins/Projetos/claudeai/copy_bank/venv/lib/python3.13/site-packages/torch/distributed/checkpoint/format_utils.py__init__$BroadcastingTorchSaveReader.__init__;s + 0c[0S9$)zGExtends the default StorageReader to support building the metadata filestate_dict_metadata)r )r%s r& read_metadata)BroadcastingTorchSaveReader.read_metadataCsB//r)planplannerc [[U5nUR(aNURce[R "URSSS9nUR (a [ U5up4OSnURGHnUR[R:Xa9[SURRS[U5RS35eUR(aF[R R#5nX5RRR%U5nO7[R&"UR(URR5n[R*"XpR,SS9 [/XuR0UR25nUR5U5R75nUR95UR95:Xd6S URS UR95S UR9535eUR;U5 UR=XX5 GM [?5n U RAS5 U $) z Reads torch save data on the coordinator rank, and broadcast afterwards this incurrs a communication cost, but avoids having to load the entire checkpoint on each rank, hopefully preventing OOM issues NcpuF) map_location weights_onlyNon-tensor value identified at . At this time  only supports loading Tensors.)srcasync_opzreq z mismatch sizes, z vs )!rr is_coordinatorr torchloadr itemstyperBYTE_IO RuntimeError storage_indexfqn__name__distdistributed_c10d_get_pg_default_deviceto empty_like state_dict broadcastr!rstorage_offsetslengthsresolve_tensordetachsizecopy_ commit_tensorr set_result) r%r/r0torch_state_dict_req pg_devicetensor target_tensorfuts r& read_data%BroadcastingTorchSaveReader.read_dataIs )73   %%1 11$zz""U  ))&89I&J# # ::Cxx<///"5c6G6G6K6K5LM$$(J$7$7#88WY "" 11HHJ )*;*;*?*?@CCIN))'*<*>@M %%'6;;=8 s(()): %%'(V[[]O= 8    '  ! !# 5/2h t r)metadatar:cX lUR(a%[R"5UR:XdeURceg*Implementation of the StorageReader methodN)r:rDget_rankr!r )r%r\r:s r&set_up_storage_reader1BroadcastingTorchSaveReader.set_up_storage_reader|s=,   ==?d&;&;; ;;!!---r)cU$r_)r%r/s r&prepare_local_plan.BroadcastingTorchSaveReader.prepare_local_plans r) global_plancU$rdre)r%rhs r&prepare_global_plan/BroadcastingTorchSaveReader.prepare_global_plansr)cXlgr^)r )r%r s r&reset!BroadcastingTorchSaveReader.resets*r)c@[RRU5$rd)ospathisfile)clsr s r&validate_checkpoint_id2BroadcastingTorchSaveReader.validate_checkpoint_idsww~~m,,r))r r!r:)Nrr$)rC __module__ __qualname____firstlineno____doc__rrstrrpPathLikeintr'r r-rrrrZboolrarflistrjrm classmethodrt__static_attributes__rer)r&rr's *<@ !1c2;;&6 7811  10x0 1h111f.h..QU.xHtH~$x.+5bkk4)?#@+D+-5bkk9I3J-t--r)rc N^\rSrSrSrS S\S\\S\SS4U4Sjjjr S r U=r $) ra` Extension of DefaultLoadPlanner, which creates a new Metadata object based on the passed in state dict, avoiding the need to read metadata from disk. This is useful when reading formats which don't have a metadata file, like Torch Save files. . N.B. Intended to be used with BroadcastingTorchSaveReader .. warning:: Current implementation only supports loading Tensors. >>> # xdoctest: +SKIP("undefined vars") >>> sd = {"mode": model} >>> dcp.load( >>> sd, >>> storage_reader=BroadcastingTorchSaveReader(), >>> planner=DynamicMetaLoadPlanner(), >>> checkpoint_id="path_to_model.pt" >>> ) NrIr\r:r"c z>[TU]XU5 0nURR5H|upV[R "U5(d%[ SUS[U5RS35e[[URS9UR5[U55XE'M~ [US9Ulg)zdSetups of the planner, extnding default behavior by creating the Metadata object from the state dictr5r6r7)dtyper+N)superset_up_plannerrIr=r; is_tensorr@r>rCrrrrOrr r\)r%rIr\r:r,keyrW __class__s r&r%DynamicMetaLoadPlanner.set_up_planners z^D8:??002KC??6**"5cU;$$(J$7$7#88WY (= v||4 "6*(  $3!5HI r))r\)NF) rCrvrwrxryrrr r}rr __classcell__)rs@r&rrsK.(,$ J#J8$J J  JJr)rdcp_checkpoint_dirtorch_save_pathcn0n[U[U5[5SS9 [R"X!5 g)aQ Given a directory containing a DCP checkpoint, this function will convert it into a Torch save file. Args: dcp_checkpoint_dir: Directory containing the DCP checkpoint. torch_save_path: Filename to store the converted Torch save file. .. warning:: To avoid OOM, it's recommended to only run this function on a single rank. T)storage_readerr0no_distN)rrr r;save)rrsds r&rrs4B '(:;*,   JJr#r)cV[R"USS9n[U[U5SS9 g)a& Given the location of a torch save file, converts it into a DCP checkpoint. Args: torch_save_path: Filename of the Torch save file. dcp_checkpoint_dir: Directory to store the DCP checkpoint. .. warning:: To avoid OOM, it's recommended to only run this function on a single rank. F)r4T)storage_writerrN)r;r<rr )rrrIs r&rrs-O%@J#34F#GQUr)__main__c\rSrSrSrSrSrg) FormatMode torch_to_dcp dcp_to_torchreN)rCrvrwrx TORCH_TO_DCP DCP_TO_TORCHrrer)r&rrs % % r)rmodezConversion mode)r>helpchoicesdefaultr8zPath to the source model)r>rdstzPath to the destination modelzConverting checkpoint from z to z using method: ''zNo checkpoint found at z. Skipping conversion.zUnknown conversion mode: )Dargparserpenumrtypingrrrr;torch.distributed distributedrDtorch.distributed._shard._utilsrtorch.distributed.checkpointrr )torch.distributed.checkpoint._nested_dictr ,torch.distributed.checkpoint.default_plannerr r %torch.distributed.checkpoint.metadatar rrrr$torch.distributed.checkpoint.plannerrrr,torch.distributed.checkpoint.planner_helpersr.torch.distributed.checkpoint.state_dict_loaderr-torch.distributed.checkpoint.state_dict_saverr$torch.distributed.checkpoint.storager torch.futuresr__all__rrrzr{rrrCrArgumentParserparser add_argumentvaluer parse_argsargsprintr8rrcheckpoint_missing_warningrqrrrisdir ValueError)ms0r&rsk (( BKHUTKKJ>  l--l-^+J/+J\$c2;;./$3 +,$23 +,c2;;./. z&T&  $ $ &F   ",-*Q*-''  C.HI C.MN    D %dhhZtDHH:=Mdii[XYZ "$((+AB yyJ++111 77>>$(( # # dhh 1 , - j--33 3 77== " " dhh 1 , -4TYYK@AAI.sJ)