# Error with the tmp **URL:** \ **Category:** Beginners\ **Created:** [December 4, 2024, 5:35pm UTC](https://discuss.huggingface.co/t/error-with-the-tmp/129278 "2024-12-04T17:35:14Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![markikojr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/markikojr/32/36950_2.png) [@markikojr](https://discuss.huggingface.co/u/markikojr)\ **Post date:** [December 4, 2024, 5:35pm UTC](https://discuss.huggingface.co/t/error-with-the-tmp/129278/1 "2024-12-04T17:35:15Z")
## 1)When running #import os # Set a different temporary directory os.environ[‘TMPDIR’] = ‘./tmp’ torch.set\_num\_threads(1) # Limit the number of threads used by PyTorch #Tokenize the text field in our dataset without adding special tokens or padding since we will do this manually. def tokenize(element): return tokenizer( element[“text”], truncation=True, #max\_length=2048, max\_length=1024, #max\_length=512, #padding=True, add\_special\_tokens=False, #add\_special\_tokens=True, ) num\_cores = max(1, os.cpu\_count() // 2) # Use half of your available CPU cores ## dataset\_tokenized = dataset\_QA.map( tokenize, batched=True, #num\_proc=num\_cores, # Use fewer processes #num\_proc=4, # Disable multiprocessing num\_proc=os.cpu\_count(), # multithreaded remove\_columns=[“text”], # don’t need this anymore, we have tokens from here on keep\_in\_memory=True ) ## Got the error: ##################### Process SyncManager-196: Traceback (most recent call last): File “/usr/local/lib/python3.12/site-packages/multiprocess/process.py”, line 314, in \_bootstrap self.run() File “/usr/local/lib/python3.12/site-packages/multiprocess/process.py”, line 108, in run self.\_target(\*self.\_args, \*\*self.\_kwargs) File “/usr/local/lib/python3.12/site-packages/multiprocess/managers.py”, line 591, in \_run\_server server = cls.\_Server(registry, address, authkey, serializer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/site-packages/multiprocess/managers.py”, line 156, in **init** self.listener = Listener(address=address, backlog=16) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/site-packages/multiprocess/connection.py”, line 461, in **init** address = address or arbitrary\_address(family) ^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/site-packages/multiprocess/connection.py”, line 80, in arbitrary\_address return tempfile.mktemp(prefix=‘listener-’, dir=util.get\_temp\_dir()) ^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.12/site-packages/multiprocess/util.py”, line 146, in get\_temp\_dir tempdir = tempfile.mkdtemp(prefix=‘pymp-’) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/lib64/python3.12/tempfile.py”, line 384, in mkdtemp \_os.mkdir(file, 0o700) OSError: [Errno 5] Input/output error: ‘/tmp/pymp-7wlx2o08’ EOFError Traceback (most recent call last) Cell In[24], line 20 8 return tokenizer( 9 element[“text”], 10 truncation=True, (…) 16 #add\_special\_tokens=True, 17 ) 18 num\_cores = max(1, os.cpu\_count() // 2) # Use half of your available CPU cores —\> 20 dataset\_tokenized = dataset\_QA.map( 21 tokenize, 22 batched=True, 23 #num\_proc=num\_cores, # Use fewer processes 24 #num\_proc=4, # Disable multiprocessing 25 num\_proc=os.cpu\_count(), # multithreaded 26 remove\_columns=[“text”], # don’t need this anymore, we have tokens from here on 27 keep\_in\_memory=True 28 ) File /projetos/fl6f/lib/python3.12/site-packages/datasets/dataset\_dict.py:887, in DatasetDict.map(self, function, with\_indices, with\_rank, input\_columns, batched, batch\_size, drop\_last\_batch, remove\_columns, keep\_in\_memory, load\_from\_cache\_file, cache\_file\_names, writer\_batch\_size, features, disable\_nullable, fn\_kwargs, num\_proc, desc) 883 if cache\_file\_names is None: 884 cache\_file\_names = {k: None for k in self} 885 return DatasetDict( 886 { → 887 k: dataset.map( 888 function=function, 889 with\_indices=with\_indices, 890 with\_rank=with\_rank, 891 input\_columns=input\_columns, 892 batched=batched, 893 batch\_size=batch\_size, 894 drop\_last\_batch=drop\_last\_batch, 895 remove\_columns=remove\_columns, 896 keep\_in\_memory=keep\_in\_memory, 897 load\_from\_cache\_file=load\_from\_cache\_file, 898 cache\_file\_name=cache\_file\_names[k], 899 writer\_batch\_size=writer\_batch\_size, 900 features=features, 901 disable\_nullable=disable\_nullable, 902 fn\_kwargs=fn\_kwargs, 903 num\_proc=num\_proc, 904 desc=desc, 905 ) 906 for k, dataset in self.items() 907 } 908 ) File /projetos/fl6f/lib/python3.12/site-packages/datasets/arrow\_dataset.py:560, in transmit\_format..wrapper(\*args, \*\*kwargs) 553 self\_format = { 554 “type”: self.\_format\_type, 555 “format\_kwargs”: self.\_format\_kwargs, 556 “columns”: self.\_format\_columns, 557 “output\_all\_columns”: self.\_output\_all\_columns, 558 } 559 # apply actual function → 560 out: Union[“Dataset”, “DatasetDict”] = func(self, \*args, \*\*kwargs) 561 datasets: List[“Dataset”] = list(out.values()) if isinstance(out, dict) else [out] 562 # re-apply format to the output File /projetos/fl6f/lib/python3.12/site-packages/datasets/arrow\_dataset.py:3147, in Dataset.map(self, function, with\_indices, with\_rank, input\_columns, batched, batch\_size, drop\_last\_batch, remove\_columns, keep\_in\_memory, load\_from\_cache\_file, cache\_file\_name, writer\_batch\_size, features, disable\_nullable, fn\_kwargs, num\_proc, suffix\_template, new\_fingerprint, desc) 3141 logger.info(f"Spawning {num\_proc} processes") 3142 with hf\_tqdm( 3143 unit=" examples", 3144 total=pbar\_total, 3145 desc=(desc or “Map”) + f" (num\_proc={num\_proc})", 3146 ) as pbar: → 3147 for rank, done, content in iflatmap\_unordered( 3148 pool, Dataset.\_map\_single, kwargs\_iterable=kwargs\_per\_job 3149 ): 3150 if done: 3151 shards\_done += 1 File /projetos/fl6f/lib/python3.12/site-packages/datasets/utils/py\_utils.py:696, in iflatmap\_unordered(pool, func, kwargs\_iterable) 694 pool\_changed = False 695 manager\_cls = Manager if isinstance(pool, multiprocessing.pool.Pool) else multiprocess.Manager → 696 with manager\_cls() as manager: 697 queue = manager.Queue() 698 async\_results = [ 699 pool.apply\_async(\_write\_generator\_to\_queue, (queue, func, kwargs)) for kwargs in kwargs\_iterable 700 ] File /usr/local/lib/python3.12/site-packages/multiprocess/context.py:57, in BaseContext.Manager(self) 55 from .managers import SyncManager 56 m = SyncManager(ctx=self.get\_context()) —\> 57 m.start() 58 return m File /usr/local/lib/python3.12/site-packages/multiprocess/managers.py:566, in BaseManager.start(self, initializer, initargs) 564 # get address of server 565 writer.close() → 566 self.\_address = reader.recv() 567 reader.close() 569 # register a finalizer File /usr/local/lib/python3.12/site-packages/multiprocess/connection.py:253, in \_ConnectionBase.recv(self) 251 self.\_check\_closed() 252 self.\_check\_readable() → 253 buf = self.\_recv\_bytes() 254 return \_ForkingPickler.loads(buf.getbuffer()) File /usr/local/lib/python3.12/site-packages/multiprocess/connection.py:433, in Connection.\_recv\_bytes(self, maxsize) 432 def \_recv\_bytes(self, maxsize=None): → 433 buf = self.\_recv(4) 434 size, = struct.unpack(“!i”, buf.getvalue()) 435 if size == -1: File /usr/local/lib/python3.12/site-packages/multiprocess/connection.py:402, in Connection.\_recv(self, size, read) 400 if n == 0: 401 if remaining == size: → 402 raise EOFError 403 else: 404 raise OSError(“got end of file during message”) ## EOFError: ################################# I checked the disk space and it is not the problem. It was running before, and stopped runing, i don’t understand why… * * * 2)Same happens when i run the Trainer * * * #from dotenv import load\_dotenv #load\_dotenv() # Set a different temporary directory os.environ[‘TMPDIR’] = ‘/home/fl6f/work/tmp’ #Clear GPU memory before training clear\_gpu\_memory() #wait\_until\_enough\_gpu\_memory(min\_memory\_available) torch.cuda.empty\_cache() torch.cuda.reset\_max\_memory\_allocated() checkpoint\_path = “/projetos/fl6f/LAMMA\_QuestionAnswering\_Pira/out/checkpoint-156” #Train the model (FineTuning) trainer = Trainer( model=model, tokenizer=tokenizer, args=args, #data\_collator=collate, data\_collator=data\_collator, train\_dataset=dataset\_tokenized[“train”], eval\_dataset=dataset\_tokenized[“validation”], #train\_dataset=subset\_train, #eval\_dataset=subset\_test, #callbacks=[early\_stopping, PlotLossCallback()] callbacks=[early\_stopping]# Adiciona o callback aqui# Add the early stopping callback here ) model.config.use\_cache = False # silence the warnings. Please re-enable for inference! ## #Train trainer.train() ## I got this error: ####################### OSError Traceback (most recent call last) Cell In[23], line 32 29 model.config.use\_cache = False # silence the warnings. Please re-enable for inference! 31 #Train —\> 32 trainer.train() File /projetos/fl6f/lib/python3.12/site-packages/transformers/trainer.py:1938, in Trainer.train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs) 1936 hf\_hub\_utils.enable\_progress\_bars() 1937 else: → 1938 return inner\_training\_loop( 1939 args=args, 1940 resume\_from\_checkpoint=resume\_from\_checkpoint, 1941 trial=trial, 1942 ignore\_keys\_for\_eval=ignore\_keys\_for\_eval, 1943 ) File /projetos/fl6f/lib/python3.12/site-packages/transformers/trainer.py:2075, in Trainer.\_inner\_training\_loop(self, batch\_size, args, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval) 2071 gradient\_checkpointing\_kwargs = args.gradient\_checkpointing\_kwargs 2073 self.model.gradient\_checkpointing\_enable(gradient\_checkpointing\_kwargs=gradient\_checkpointing\_kwargs) → 2075 model = self.\_wrap\_model(self.model\_wrapped) 2077 # as the model is wrapped, don’t use `accelerator.prepare` 2078 # this is for unhandled cases such as 2079 # FSDP-XLA, SageMaker MP/DP, DataParallel, IPEX 2080 use\_accelerator\_prepare = True if model is self.model else False File /projetos/fl6f/lib/python3.12/site-packages/transformers/trainer.py:1693, in Trainer.\_wrap\_model(self, model, training, dataloader) 1690 return smp.DistributedModel(model, backward\_passes\_per\_step=self.args.gradient\_accumulation\_steps) 1692 # train/eval could be run multiple-times - if already wrapped, don’t re-wrap it again → 1693 if self.accelerator.unwrap\_model(model) is not model: 1694 return model 1696 # Mixed precision training with apex (torch \< 1.6) File /projetos/fl6f/lib/python3.12/site-packages/accelerate/accelerator.py:2583, in Accelerator.unwrap\_model(self, model, keep\_fp32\_wrapper) 2552 def unwrap\_model(self, model, keep\_fp32\_wrapper: bool = True): 2553 “”" 2554 Unwraps the `model` from the additional layer possible added by [`~Accelerator.prepare`]. Useful before saving 2555 the model. (…) 2581 ``` 2582 “”" → 2583 return extract\_model\_from\_parallel(model, keep\_fp32\_wrapper) File /projetos/fl6f/lib/python3.12/site-packages/accelerate/utils/other.py:85, in extract\_model\_from\_parallel(model, keep\_fp32\_wrapper, recursive) 82 options += (DeepSpeedEngine,) 84 if is\_torch\_version(“\>=”, FSDP\_PYTORCH\_VERSION) and is\_torch\_distributed\_available(): —\> 85 from torch.distributed.fsdp.fully\_sharded\_data\_parallel import FullyShardedDataParallel as FSDP 87 options += (FSDP,) 89 while isinstance(model, options): File /usr/local/lib64/python3.12/site-packages/torch/distributed/fsdp/ **init**.py:1 ----\> 1 from .\_flat\_param import FlatParameter as FlatParameter 2 from .fully\_sharded\_data\_parallel import ( 3 BackwardPrefetch, 4 CPUOffload, (…) 18 StateDictType, 19 ) 21 **all** = [ 22 “BackwardPrefetch”, 23 “CPUOffload”, (…) 37 “StateDictType”, 38 ] File /usr/local/lib64/python3.12/site-packages/torch/distributed/fsdp/\_flat\_param.py:30 28 import torch.nn.functional as F 29 from torch import Tensor —\> 30 from torch.distributed.fsdp.\_common\_utils import ( 31 \_FSDPDeviceHandle, 32 \_named\_parameters\_with\_duplicates, 33 \_no\_dispatch\_record\_stream, 34 \_set\_fsdp\_flattened, 35 HandleTrainingState, 36 ) 37 from torch.distributed.utils import ( 38 \_alloc\_storage, 39 \_data\_ptr\_allocated, 40 \_free\_storage, 41 \_p\_assert, 42 ) 43 from torch.nn.parameter import \_ParameterMeta # type: ignore[attr-defined] File /usr/local/lib64/python3.12/site-packages/torch/distributed/fsdp/\_common\_utils.py:35 31 from torch.distributed.algorithms.\_checkpoint.checkpoint\_wrapper import ( 32 \_CHECKPOINT\_PREFIX, 33 ) 34 from torch.distributed.device\_mesh import DeviceMesh —\> 35 from torch.distributed.fsdp.\_fsdp\_extensions import FSDPExtensions 36 from torch.distributed.utils import \_apply\_to\_tensors 37 from torch.utils.\_mode\_utils import no\_dispatch File /usr/local/lib64/python3.12/site-packages/torch/distributed/fsdp/\_fsdp\_extensions.py:6 4 import torch 5 import torch.distributed as dist ----\> 6 from torch.distributed.\_shard.sharded\_tensor.api import ShardedTensor 7 from torch.distributed.\_shard.sharded\_tensor.shard import Shard 8 from torch.distributed.\_tensor import DeviceMesh, DTensor File /usr/local/lib64/python3.12/site-packages/torch/distributed/\_shard/ **init**.py:1 ----\> 1 from .api import ( 2 \_shard\_tensor, 3 load\_with\_process\_group, 4 shard\_module, 5 shard\_parameter, 6 ) File /usr/local/lib64/python3.12/site-packages/torch/distributed/\_shard/api.py:7 5 import torch.nn as nn 6 from torch.distributed import distributed\_c10d ----\> 7 from torch.distributed.\_shard.sharded\_tensor import ( 8 ShardedTensor, 9 ) 10 from .sharding\_spec import ( 11 ShardingSpec, 12 ChunkShardingSpec 13 ) 14 from .sharding\_plan import ( 15 ShardingPlan 16 ) File /usr/local/lib64/python3.12/site-packages/torch/distributed/\_shard/sharded\_tensor/ **init**.py:11 8 else: 9 ShardingSpec = “ShardingSpec” —\> 11 from .api import ( 12 \_CUSTOM\_SHARDED\_OPS, 13 \_SHARDED\_OPS, 14 Shard, 15 ShardedTensorBase, 16 ShardedTensor, 17 ShardedTensorMetadata, 18 TensorProperties, 19 ) 20 from .metadata import ShardMetadata # noqa: F401 21 from torch.distributed.\_shard.op\_registry\_utils import \_decorator\_func File /usr/local/lib64/python3.12/site-packages/torch/distributed/\_shard/sharded\_tensor/api.py:38 36 from .metadata import TensorProperties, ShardedTensorMetadata 37 from .shard import Shard —\> 38 from .reshard import reshuffle\_local\_shard, reshard\_local\_shard 39 from .utils import ( 40 \_flatten\_tensor\_size, 41 \_parse\_and\_validate\_remote\_device, (…) 44 build\_global\_metadata 45 ) 46 from torch.distributed.remote\_device import \_remote\_device File /usr/local/lib64/python3.12/site-packages/torch/distributed/\_shard/sharded\_tensor/reshard.py:14 9 import torch.distributed.\_shard.sharding\_spec as shard\_spec 10 from torch.distributed.\_shard.sharding\_spec.\_internals import ( 11 get\_split\_size, 12 get\_chunked\_dim\_size, 13 ) —\> 14 from torch.distributed.nn.functional import ( 15 all\_to\_all, 16 all\_to\_all\_single, 17 ) 18 from torch.distributed.\_shard.metadata import ShardMetadata 20 from .shard import Shard File /usr/local/lib64/python3.12/site-packages/torch/distributed/nn/ **init**.py:3 1 import torch 2 if torch.distributed.rpc.is\_available(): ----\> 3 from .api.remote\_module import RemoteModule 4 from .functional import \* File /usr/local/lib64/python3.12/site-packages/torch/distributed/nn/api/remote\_module.py:24 22 import torch.distributed.rpc as rpc 23 from torch import Tensor, device, dtype, nn —\> 24 from torch.distributed.nn.jit import instantiator 25 from torch.distributed import \_remote\_device 26 from torch.distributed.rpc.internal import \_internal\_rpc\_pickler File /usr/local/lib64/python3.12/site-packages/torch/distributed/nn/jit/instantiator.py:19 15 logger = logging.getLogger( **name** ) 18 \_FILE\_PREFIX = “_remote\_module_” —\> 19 \_TEMP\_DIR = tempfile.TemporaryDirectory() 20 INSTANTIATED\_TEMPLATE\_DIR\_PATH = \_TEMP\_DIR.name 21 logger.info(“Created a temporary directory at %s”, INSTANTIATED\_TEMPLATE\_DIR\_PATH) File /usr/lib64/python3.12/tempfile.py:882, in TemporaryDirectory. **init** (self, suffix, prefix, dir, ignore\_cleanup\_errors, delete) 880 def **init** (self, suffix=None, prefix=None, dir=None, 881 ignore\_cleanup\_errors=False, \*, delete=True): → 882 self.name = mkdtemp(suffix, prefix, dir) 883 self.\_ignore\_cleanup\_errors = ignore\_cleanup\_errors 884 self.\_delete = delete File /usr/lib64/python3.12/tempfile.py:384, in mkdtemp(suffix, prefix, dir) 382 \_sys.audit(“tempfile.mkdtemp”, file) 383 try: → 384 \_os.mkdir(file, 0o700) 385 except FileExistsError: 386 continue # try again ## OSError: [Errno 5] Input/output error: ‘/tmp/tmpfe\_zflge’ It was running before as well, but it gives this error and i have no idea how to solve… ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 4, 2024, 11:23pm UTC](https://discuss.huggingface.co/t/error-with-the-tmp/129278/2 "2024-12-04T23:23:48Z")
The most suspicious of this information is Python 3.12. Python 3.12 frequently causes errors in the HF space. ---
**Author:** ![markikojr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/markikojr/32/36950_2.png) [@markikojr](https://discuss.huggingface.co/u/markikojr)\ **Post date:** [December 9, 2024, 1:34pm UTC](https://discuss.huggingface.co/t/error-with-the-tmp/129278/3 "2024-12-09T13:34:26Z")
Thank you to reply! Considering your comment… Which version of python should i use for LLM fine tuning? Which one is recommended? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [December 9, 2024, 2:14pm UTC](https://discuss.huggingface.co/t/error-with-the-tmp/129278/4 "2024-12-09T14:14:02Z")
The version used by HF is 3.10. 3.11 seems to be relative stable in the newer versions. I’m using 3.9 in my local environment, but since it’s close to the end of support and more and more things are not working, I recommend using 3.10 or later if you’re going to use it.😅 ---
**Author:** ![kasunw](https://avatars.discourse-cdn.com/v4/letter/k/e68b1a/32.png) [@kasunw](https://discuss.huggingface.co/u/kasunw)\ **Post date:** [December 24, 2024, 4:56pm UTC](https://discuss.huggingface.co/t/error-with-the-tmp/129278/5 "2024-12-24T16:56:29Z")
Could be access timeout error due to large number of files and/or subfolders as per the following explanation. [https://research.google.com/colaboratory/faq.html#drive-timeout](https://research.google.com/colaboratory/faq.html#drive-timeout)