# Pickling issue using map **URL:** \ **Category:** 🤗Datasets\ **Created:** [April 6, 2025, 5:44pm UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130 "2025-04-06T17:44:00Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![MagicLuke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/magicluke/32/44922_2.png) [@MagicLuke](https://discuss.huggingface.co/u/MagicLuke)\ **Post date:** [April 6, 2025, 5:44pm UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/1 "2025-04-06T17:44:00Z")
I am mapping my dataset with the following compute\_metrics method which give me a pickling issue. ```auto metric_cfg_list = config["metric_list"] metrics = [evaluate.load(metric_cfg["path"]) for metric_cfg in metric_cfg_list] # Placeholder for a tokenizer or normalizer class if needed. tokenizer = None def compute_metrics(sample): for metric in metrics: sample[metric.name] = metric.compute( predictions=[sample["clean_prediction"]], references=[sample["clean_label"]] ) return sample ``` the following is the error message ```sh Parameter 'function'=.compute_metrics at 0x7aa60a95f0a0> of the transform datasets.arrow_dataset.Dataset._map_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mec hanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed. Map (num_proc=16): 0%| | 0/2116 [00:00 ... StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 578, in save rv = reduce(self.proto) TypeError: cannot pickle 'ThreadLocalFileContext' object ``` I saw a relevant post about the nonpicklable issue with some tokenizer and ppl solved it by implementing the **getstate** method or so. In my case, it’s an object from the evaluate package. I wonder how I should modify them to avoid this error. ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 6, 2025, 6:31pm UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/2 "2025-04-06T18:31:47Z")
Hmm… unless it’s a problem with dill, multiprocessing, or the cache, it’s better to call lhonestq… > > > \### Describe the bug > > \_Parameter 'function'=\ of the transform datasets.arrow\_dataset.Dataset.\\\_map\_single couldn't be hashed properly, a random hash was used instead. Make sure your transforms and parameters are serializable with pickle or dill for the dataset fingerprinting and caching to work. If you reuse this transform, the caching mechanism will consider it to be different from the previous calls and recompute everything. This warning is only showed once. Subsequent hashing failures won't be showed.\_ > > This issue with \`.map()\` happens for me consistently, as also described in closed issue #4506 > > Dataset indices can be individually serialized using dill and pickle without any errors. I'm using tiktoken to encode in the function passed to map(). Similarly, indices can be individually encoded without error. > > \### Steps to reproduce the bug > > \`\`\`py > from datasets import load\_dataset > import tiktoken > > dataset = load\_dataset("stas/openwebtext-10k") > > enc = tiktoken.get\_encoding("gpt2") > > tokenized = dataset.map( > process, > remove\_columns=\['text'\], > desc="tokenizing the OWT splits", > ) > > def process(example): > ids = enc.encode(example\['text'\]) > ids.append(enc.eot\_token) > out = {'ids': ids, 'len': len(ids)} > return out > \`\`\` > > \### Expected behavior > > Should encode simple text objects. > > \### Environment info > > > Python versions tried: both 3.8 and 3.10.10 > \`PYTHONUTF8=1\` as env variable > > Datasets tried: > \- stas/openwebtext-10k > \- rotten\_tomatoes > \- local text file > > OS: Ubuntu Linux 20.04 > > Package versions: > \- torch 1.13.1 > \- dill 0.3.4 (if using 0.3.6 - same issue) > \- datasets 2.9.0 > \- tiktoken 0.2.0 > > > \## Describe the bug > When I \`map\` with multiple processes, this error occurs. Th…e \`.name\` of the \`logger\` that fails to pickle in the final line is \`datasets.fingerprint\`. > \`\`\` > File "~/project/dataset.py", line 204, in \ > split: dataset.map( > File ".../site-packages/datasets/arrow\_dataset.py", line 2489, in map > transformed\_shards\[index\] = async\_result.get() > File ".../site-packages/multiprocess/pool.py", line 771, in get > raise self.\_value > File ".../site-packages/multiprocess/pool.py", line 537, in \_handle\_tasks > put(task) > File ".../site-packages/multiprocess/connection.py", line 214, in send > self.\_send\_bytes(\_ForkingPickler.dumps(obj)) > File ".../site-packages/multiprocess/reduction.py", line 54, in dumps > cls(buf, protocol, \*args, \*\*kwds).dump(obj) > File ".../site-packages/dill/\_dill.py", line 620, in dump > StockPickler.dump(self, obj) > File ".../pickle.py", line 487, in dump > self.save(obj) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../pickle.py", line 902, in save\_tuple > save(element) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../site-packages/dill/\_dill.py", line 1963, in save\_function > \_save\_with\_postproc(pickler, (\_create\_function, ( > File ".../site-packages/dill/\_dill.py", line 1140, in \_save\_with\_postproc > pickler.save\_reduce(\*reduction, obj=obj) > File ".../pickle.py", line 717, in save\_reduce > save(state) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../pickle.py", line 887, in save\_tuple > save(element) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../site-packages/dill/\_dill.py", line 1251, in save\_module\_dict > StockPickler.save\_dict(pickler, obj) > File ".../pickle.py", line 972, in save\_dict > self.\_batch\_setitems(obj.items()) > File ".../pickle.py", line 998, in \_batch\_setitems > save(v) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../site-packages/dill/\_dill.py", line 1963, in save\_function > \_save\_with\_postproc(pickler, (\_create\_function, ( > File ".../site-packages/dill/\_dill.py", line 1140, in \_save\_with\_postproc > pickler.save\_reduce(\*reduction, obj=obj) > File ".../pickle.py", line 717, in save\_reduce > save(state) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../pickle.py", line 887, in save\_tuple > save(element) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../site-packages/dill/\_dill.py", line 1251, in save\_module\_dict > StockPickler.save\_dict(pickler, obj) > File ".../pickle.py", line 972, in save\_dict > self.\_batch\_setitems(obj.items()) > File ".../pickle.py", line 998, in \_batch\_setitems > save(v) > File ".../pickle.py", line 560, in save > f(self, obj) # Call unbound method with explicit self > File ".../site-packages/dill/\_dill.py", line 1963, in save\_function > \_save\_with\_postproc(pickler, (\_create\_function, ( > File ".../site-packages/dill/\_dill.py", line 1154, in \_save\_with\_postproc > pickler.\_batch\_setitems(iter(source.items())) > File ".../pickle.py", line 998, in \_batch\_setitems > save(v) > File ".../pickle.py", line 578, in save > rv = reduce(self.proto) > File ".../logging/\_\_init\_\_.py", line 1774, in \_\_reduce\_\_ > raise pickle.PicklingError('logger cannot be pickled') > \_pickle.PicklingError: logger cannot be pickled > \`\`\` > > \## Steps to reproduce the bug > Sorry I failed to have a minimal reproducible example, but the offending line on my end is > \`\`\`python > dataset.map( > lambda examples: self.tokenize(examples), # this doesn't matter, lambda e: \[1\] \* len(...) also breaks. In fact I'm pretty sure it breaks before executing this lambda > batched=True, > num\_proc=4, > ) > \`\`\` > This does work when \`num\_proc=1\`, so it's likely a multiprocessing thing. > > \## Expected results > \`map\` succeeds > > \## Actual results > The error trace above. > > \## Environment info > \- \`datasets\` version: 1.16.1 and 2.5.1 both failed > \- Platform: Ubuntu 20.04.4 LTS > \- Python version: 3.10.4 > \- PyArrow version: 9.0.0 > You can also provide your own unique hash in `map` if you want, with the `new_fingerprint` argument. > Or disable caching using ```py import datasets datasets.disable_caching() ``` ---
**Author:** ![MagicLuke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/magicluke/32/44922_2.png) [@MagicLuke](https://discuss.huggingface.co/u/MagicLuke)\ **Post date:** [April 7, 2025, 2:12am UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/3 "2025-04-07T02:12:40Z")
I tried both new\_fingerprint and disable\_cache(), but all still gave the same bug. the complete error is as follow: ```sh Map (num_proc=16): 0%| | 0/2116 [00:00 main() File "/ws/ifp-54_2/hasegawa/haolong2/AI4EE/CSR4RSR/evaluation.py", line 178, in main ds[split] = ds[split].map( File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 557, in wrapper out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/arrow_dataset.py", line 3166, in map for rank, done, content in iflatmap_unordered( File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 720, in iflatmap_unordered [async_result.get(timeout=0.05) for async_result in async_results] File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/datasets/utils/py_utils.py", line 720, in [async_result.get(timeout=0.05) for async_result in async_results] File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/pool.py", line 774, in get raise self._value File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/pool.py", line 540, in _handle_tasks put(task) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/connection.py", line 209, in send self._send_bytes(_ForkingPickler.dumps(obj)) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/multiprocess/reduction.py", line 54, in dumps cls(buf, protocol, *args, **kwds).dump(obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 420, in dump StockPickler.dump(self, obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 487, in dump self.save(obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 902, in save_tuple save(element) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 887, in save_tuple save(element) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict StockPickler.save_dict(pickler, obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict self._batch_setitems(obj.items()) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems save(v) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 902, in save_tuple save(element) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 887, in save_tuple save(element) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict StockPickler.save_dict(pickler, obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict self._batch_setitems(obj.items()) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems save(v) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1985, in save_function _save_with_postproc(pickler, (_create_function, ( File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1117, in _save_with_postproc pickler.save_reduce(*reduction) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 692, in save_reduce save(args) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 887, in save_tuple save(element) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 932, in save_list self._batch_appends(obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 956, in _batch_appends save(x) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 717, in save_reduce save(state) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict StockPickler.save_dict(pickler, obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict self._batch_setitems(obj.items()) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems save(v) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 932, in save_list self._batch_appends(obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 959, in _batch_appends save(tmp[0]) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 603, in save self.save_reduce(obj=obj, *rv) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 717, in save_reduce save(state) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 560, in save f(self, obj) # Call unbound method with explicit self File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 1217, in save_module_dict StockPickler.save_dict(pickler, obj) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 972, in save_dict self._batch_setitems(obj.items()) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 998, in _batch_setitems save(v) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/dill/_dill.py", line 414, in save StockPickler.save(self, obj, save_persistent_id) File "/ws/ifp-53_2/hasegawa/haolong2/miniconda3/envs/csr4rsr/lib/python3.10/pickle.py", line 578, in save rv = reduce(self.proto) TypeError: cannot pickle 'ThreadLocalFileContext' object ``` ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 7, 2025, 4:00am UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/4 "2025-04-07T04:00:08Z")
Hmm… @lhoestq map function or PyArrow issue…? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [April 7, 2025, 9:51am UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/5 "2025-04-07T09:51:47Z")
It looks like the `ThreadLocalFileContext` from `filelock` is not picklable, and therefore can’t be used with `.map()` with `num_proc=...` Apparently thid can be fixed using `thread_local=False`, see the docs at [filelock](https://py-filelock.readthedocs.io/en/latest/index.html#filelocks-and-threads) Can you modify `evaluate` to pass `thread_local=False` to all `FileLock` objects and try again to see if it works ? ---
**Author:** ![MagicLuke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/magicluke/32/44922_2.png) [@MagicLuke](https://discuss.huggingface.co/u/MagicLuke)\ **Post date:** [April 7, 2025, 9:05pm UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/6 "2025-04-07T21:05:59Z")
I am not sure if I do it right. I modify the function `get_from_cache` in the `file_utils` located …/miniconda3/envs/csr4rsr/lib/python3.10/site-packages/evaluate/utils/file\_utils.py from ```python with FileLock(lock_path): # Origin ``` to ```python with FileLock(lock_path, thread_local=False): # Modified ``` but the problem persist. ---
**Author:** ![MagicLuke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/magicluke/32/44922_2.png) [@MagicLuke](https://discuss.huggingface.co/u/MagicLuke)\ **Post date:** [April 7, 2025, 9:30pm UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/7 "2025-04-07T21:30:34Z")
By adding this code chunck before importing evaluating seems solved the problem. ```python from filelock import FileLock as OriginalFileLock class PatchedFileLock(OriginalFileLock): def __init__ (self, *args, **kwargs): kwargs["thread_local"] = False # Force it every time super(). __init__ (*args, **kwargs) import filelock filelock.FileLock = PatchedFileLock ``` Thanks for the insight @lhoestq. Would you mind telling where you find the clue for the error if it’s not too much trouble In this way, I might be able to fix it the same way in the future. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [April 8, 2025, 8:56am UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/8 "2025-04-08T08:56:07Z")
Great ! Let me know if you think we should make this the default in `datasets` and `evaluate`, apparently this logic appears with python \>= 3.11 > Would you mind telling where you find the clue for the error if it’s not too much trouble > In this way, I might be able to fix it the same way in the future. The `dill` error says “TypeError: cannot pickle ‘ThreadLocalFileContext’ object”, so it means that in the function you pass to `map()` there is an object that contains a ThreadLocalFileContext that is not supported by `dill` for multiprocessing. I searched on google for ThreadLocalFileContext on [github.com](http://github.com) to look for packages that have such objects and figured it came from `filelock` which is a dependency of `evaluate`. Finally the `filelock` changelog they mention ThreadLocalFileContext as a recent addition for FileLock ---
**Author:** ![MagicLuke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/magicluke/32/44922_2.png) [@MagicLuke](https://discuss.huggingface.co/u/MagicLuke)\ **Post date:** [April 8, 2025, 4:54pm UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/9 "2025-04-08T16:54:17Z")
Thanks for the explanation! I think it would be great to set it as the default in my case, which is several metrics that need to be computed for a dataset. For me, I just want to avoid using multiple rounds of map. Or maybe there is a better way to do it that I haven’t figured out. ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [July 6, 2025, 4:04am UTC](https://discuss.huggingface.co/t/pickling-issue-using-map/149130/10 "2025-07-06T04:04:52Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.