# .get\_nearest\_examples() throws ArrowInvalid: offset overflow while concatenating arrays **URL:** \ **Category:** 🤗Datasets\ **Created:** [September 29, 2020, 2:16pm UTC](https://discuss.huggingface.co/t/get-nearest-examples-throws-arrowinvalid-offset-overflow-while-concatenating-arrays/1297 "2020-09-29T14:16:58Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![AndreGodinho](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andregodinho/32/3005_2.png) [@AndreGodinho](https://discuss.huggingface.co/u/AndreGodinho)\ **Post date:** [September 29, 2020, 2:16pm UTC](https://discuss.huggingface.co/t/get-nearest-examples-throws-arrowinvalid-offset-overflow-while-concatenating-arrays/1297/1 "2020-09-29T14:16:58Z")
Hello everyone, I am [adding a FAISS index](https://huggingface.co/docs/datasets/faiss_and_ea.html) on the [MSMARCO passages dataset](https://github.com/microsoft/MSMARCO-Passage-Ranking) that has ~8.8M passages. I have already created the embeddings with DPRContextTokenizer and DPRContextEncoder. Dataset info: ![ms marco features](https://us1.discourse-cdn.com/hellohellohello/original/2X/8/83abf4a55d5703019b47ea4e92ab677f3d9e4bdf.png) After adding the FAISS index to it, I tried to retrieve some documents with a query. ``` q_encoder = DPRQuestionEncoder.from_pretrained("facebook/dpr-question_encoder-single-nq-base") q_tokenizer = DPRQuestionEncoderTokenizer.from_pretrained("facebook/dpr-question_encoder-single-nq-base") question = 'what is the difference between a c-corp and a s-corp?' question_embedding = q_encoder(**q_tokenizer(question, return_tensors="pt"))[0][0].detach().numpy() scores, retrieved_documents = dataset_embedded_passages['train'].get_nearest_examples('embeddings', question_embedding, k=10) ``` And then it threw, ``` ArrowInvalid Traceback (most recent call last) ``` in ``` ~/.conda/envs/andregodinho/lib/python3.6/site-packages/datasets/search.py in get_nearest_examples(self, index_name, query, k) 564 self._check_index_is_initialized(index_name) 565 scores, indices = self.search(index_name, query, k) --> 566 return NearestExamplesResults(scores, self[[i for i in indices if i >= 0]]) 567 568 def get_nearest_examples_batch( ~/.conda/envs/andregodinho/lib/python3.6/site-packages/datasets/arrow_dataset.py in __getitem__ (self, key) 1069 format_columns=self._format_columns, 1070 output_all_columns=self._output_all_columns, -> 1071 format_kwargs=self._format_kwargs, 1072 ) 1073 ~/.conda/envs/andregodinho/lib/python3.6/site-packages/datasets/arrow_dataset.py in _getitem(self, key, format_type, format_columns, output_all_columns, format_kwargs) 1037 ) 1038 else: -> 1039 data_subset = self._data.take(indices_array) 1040 1041 if format_type is not None: ~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/table.pxi in pyarrow.lib.Table.take() ~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/compute.py in take(data, indices, boundscheck) 266 """ 267 options = TakeOptions(boundscheck) --> 268 return call_function('take', [data, indices], options) 269 270 ~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/_compute.pyx in pyarrow._compute.call_function() ~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/_compute.pyx in pyarrow._compute.Function.call() ~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status() ~/.conda/envs/andregodinho/lib/python3.6/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status() ``` `It threw ArrowInvalid: offset overflow while concatenating arrays` The dataset is very large. Any workaround? ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 30, 2020, 11:52am UTC](https://discuss.huggingface.co/t/get-nearest-examples-throws-arrowinvalid-offset-overflow-while-concatenating-arrays/1297/2 "2020-09-30T11:52:18Z")
Thanks for reporting ! It will be fixed in the `datasets` release of this week ---
**Author:** ![AndreGodinho](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andregodinho/32/3005_2.png) [@AndreGodinho](https://discuss.huggingface.co/u/AndreGodinho)\ **Post date:** [September 30, 2020, 12:16pm UTC](https://discuss.huggingface.co/t/get-nearest-examples-throws-arrowinvalid-offset-overflow-while-concatenating-arrays/1297/3 "2020-09-30T12:16:49Z")
Thank you Quentin. Please let me know once you have fixed it. ---
**Author:** ![lhoestq](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lhoestq/32/52888_2.png) [@lhoestq](https://discuss.huggingface.co/u/lhoestq)\ **Post date:** [September 30, 2020, 1:12pm UTC](https://discuss.huggingface.co/t/get-nearest-examples-throws-arrowinvalid-offset-overflow-while-concatenating-arrays/1297/4 "2020-09-30T13:12:22Z")
Actually I think it’s already included in datasets==1.0.2 Could you update the lib and le me know if it fixes your issue ? ```auto pip install --upgrade datasets ``` ---
**Author:** ![AndreGodinho](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/andregodinho/32/3005_2.png) [@AndreGodinho](https://discuss.huggingface.co/u/AndreGodinho)\ **Post date:** [September 30, 2020, 2:47pm UTC](https://discuss.huggingface.co/t/get-nearest-examples-throws-arrowinvalid-offset-overflow-while-concatenating-arrays/1297/5 "2020-09-30T14:47:49Z")
Problem solved. Cheers!