# Add\_faiss\_index usage example **URL:** \ **Category:** Beginners\ **Created:** [December 22, 2020, 5:02pm UTC](https://discuss.huggingface.co/t/add-faiss-index-usage-example/2925 "2020-12-22T17:02:47Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![zuujhyt](https://avatars.discourse-cdn.com/v4/letter/z/f17d59/32.png) [@zuujhyt](https://discuss.huggingface.co/u/zuujhyt)\ **Post date:** [December 22, 2020, 5:02pm UTC](https://discuss.huggingface.co/t/add-faiss-index-usage-example/2925/1 "2020-12-22T17:02:47Z")
Hi, I am trying to know how to use Rag/DPR, but first I want to get familiar with faiss usage. I checked the official example in > **[Main classes — datasets 1.1.3 documentation](https://huggingface.co/docs/datasets/package_reference/main_classes.html?highlight=add_faiss_index#datasets.Dataset.add_faiss_index)** But it seems the snippet code is not self-executable. So I did some modification, aiming to retrieve similar examples in the sst2 dataset with query ‘I am happy’. ```auto import datasets from transformers import pipeline embed = pipeline('sentiment-analysis', model="nlptown/bert-base-multilingual-uncased-sentiment") ds = datasets.load_dataset('glue', 'sst2', split='test') ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['sentence'])}) ds_with_embeddings.add_faiss_index(column='embeddings') # query scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed('I am happy.'), k=10) # save index ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss') ds = datasets.load_dataset('glue', 'sst2', split='test') # load index ds.load_faiss_index('embeddings', 'my_index.faiss') # query scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed('I am happy.'), k=10) ``` My problem is at `ds_with_embeddings.add_faiss_index(column='embeddings')` I got error with there with " TypeError: float() argument must be a string or a number, not ‘dict’ " If I changed it to ```auto ds_with_embeddings_score = ds_with_embeddings.map(lambda example: {'embeddings_score': example['embeddings'][0]['score']}) ds_with_embeddings_score.add_faiss_index(column='embeddings_score') ``` Then I got error " TypeError: len() of unsized object " Any adivce?Thanks. ---
**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\ **Post date:** [December 22, 2020, 9:13pm UTC](https://discuss.huggingface.co/t/add-faiss-index-usage-example/2925/2 "2020-12-22T21:13:23Z")
I have little experience with pipelines, but I think the issue is that `embed(example['sentence'])` should return a vector representation for `example['sentence']`. However, calling a text classification pipeline returns a dict with labels and scores. Instead, you need to run a [feature extraction](https://huggingface.co/transformers/main_classes/pipelines.html#transformers.FeatureExtractionPipeline) pipeline which should return vectors. (You may need to unpack it though, as the return type is a nested list, presumably for batched processing.) So (untested) you can try something like: ```python embed = pipeline('feature-extraction', model="nlptown/bert-base-multilingual-uncased-sentiment") ... ds_with_embeddings = ds.map(lambda example: {'embeddings': embed(example['sentence'])[0]}) ``` You may need to test a bit with the `[0]`. Not sure whether it is necessary. ---
**Author:** ![zuujhyt](https://avatars.discourse-cdn.com/v4/letter/z/f17d59/32.png) [@zuujhyt](https://discuss.huggingface.co/u/zuujhyt)\ **Post date:** [December 24, 2020, 2:45pm UTC](https://discuss.huggingface.co/t/add-faiss-index-usage-example/2925/3 "2020-12-24T14:45:45Z")
It seems the `embed` needs to give a fixed size feature, if I use pipeline with ‘feature-extraction’, I need to conduct another pooing . I used a [third-party](https://github.com/UKPLab/sentence-transformers) sentence embedding extractor and it seems work My executable code is: ```auto import datasets from sentence_transformers import SentenceTransformer embed = SentenceTransformer("nli-bert-large-max-pooling") ds = datasets.load_dataset('glue', 'sst2', split='test') ds_with_embeddings = ds.map(lambda example: {'embeddings': embed.encode(example['sentence'])}) ds_with_embeddings.add_faiss_index(column='embeddings') # query scores, retrieved_examples = ds_with_embeddings.get_nearest_examples('embeddings', embed.encode('I am happy.'), k=10) # save index ds_with_embeddings.save_faiss_index('embeddings', 'my_index.faiss') ds = datasets.load_dataset('glue', 'sst2', split='test') # load index ds.load_faiss_index('embeddings', 'my_index.faiss') # query scores, retrieved_examples = ds.get_nearest_examples('embeddings', embed.encode('I am happy.'), k=10) print("\n scores: \n{}".format(scores)) print("\n retrieved_examples: \n{}\n".format(retrieved_examples)) ``` with the output: ```auto scores: [ 93.086624 115.375305 135.21281 137.25983 146.3393 152.40036 165.03342 165.48126 166.6388 181.6477 ] retrieved_examples: OrderedDict([('idx', [138, 874, 1251, 157, 144, 917, 21, 169, 1502, 1392]), ('label', [-1, -1, -1, -1, -1, -1, -1, -1, -1, -1]), ('sentence', ['i loved it !', 'the result is something quite fresh and delightful .', 'remarkably accessible and affecting .', 'i admired this work a lot .', 'witty , touching and well paced .', '... in this incarnation its fizz is infectious .', 'a feel-good picture in the best sense of the term .', "brilliant ! '", "is n't it great ?", 'go see it and enjoy .'])]) ``` If anyone could provide a HF-native code snippet, I would appreciate. ---
**Author:** ![foton263](https://avatars.discourse-cdn.com/v4/letter/f/ee7513/32.png) [@foton263](https://discuss.huggingface.co/u/foton263)\ **Post date:** [April 22, 2022, 8:06am UTC](https://discuss.huggingface.co/t/add-faiss-index-usage-example/2925/4 "2022-04-22T08:06:17Z")
You may use **add\_faiss\_index\_from\_external\_arrays** instead.