# Understanding DataCollation **URL:** \ **Category:** 🤗Transformers\ **Created:** [July 18, 2024, 4:05pm UTC](https://discuss.huggingface.co/t/understanding-datacollation/98188 "2024-07-18T16:05:32Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![rkbelew](https://avatars.discourse-cdn.com/v4/letter/r/43a26b/32.png) [@rkbelew](https://discuss.huggingface.co/u/rkbelew)\ **Post date:** [July 18, 2024, 4:05pm UTC](https://discuss.huggingface.co/t/understanding-datacollation/98188/1 "2024-07-18T16:05:32Z")
[Cross-posting from [Pytorch forum](https://discuss.pytorch.org/t/understanding-datacollation/206471) ] I’m interested in the [Caseholder dataset](https://huggingface.co/casehold), a [multiple choice task ala SWAG](https://huggingface.co/datasets/allenai/swag) where examples include five separate possible answers. I use a preprocess function like this: ``` def preprocess_function(examples,mapped=True): # Repeat first sentence NMultChoice times to go with the NMultChoice possibilities of second sentences. first_sentences = [[context] * NMultChoice for context in examples[Context_name]] # Grab all second sentences possible for each context. second_sentences = [[ f"{examples[str(i+1)][0]}" for i in range(NMultChoice) ] for context in examples[Context_name]] # Flatten out first_sentences = sum(first_sentences, []) second_sentences = sum(second_sentences, []) # Tokenizer will return a list of lists of lists for each key: # a list of all examples (here 5), then a list of all choices (4) and a list of input IDs # (length varying here since we did not apply any padding) tokenized_examples = tokenizer(first_sentences, second_sentences, truncation=True) # Un-flatten so that each example has NMultChoice input ids, attentions masks, etc. tokDict = {k: [v[i : i + NMultChoice] for i in range(0, len(v), NMultChoice)] for k, v in tokenized_examples.items()} tokDict['label'] = examples[Label_name] tokDict['idx'] = examples[Idx_name] # capture casehold scores tokDict['scores'] = {f'{i}': example[f'{i+6}'] for i in range(NMultChoice)} return tokDict ``` and generate a processed dataset like this: ``` ds = datasets.load_dataset(csvDir,data_files=dsFiles, download_mode="force_redownload") ds2 = ds.map(preprocessOne) ``` The tokenized dataset has features that looks like this: ``` Dataset({ features: ['Unnamed: 0', '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '10', '11', 'input_ids', 'token_type_ids', 'attention_mask', 'label', 'idx', 'scores'], num_rows: 5314 }) ``` Eg, focusing on the `input_ids` generated by the tokenizer, it is a list of lists of varying lengths: ``` iid = eval_dataset['input_ids'] iid00 = iid[0][0] for l in iid00: print (len(l)) > 224 > 225 > 229 > 228 > 224 ``` Also because of the multiple choice task, I’m using a DataCollator modeled after [this swag example](https://colab.research.google.com/github/huggingface/notebooks/blob/main/examples/multiple_choice.ipynb#scrollTo=5CjJtk-AuAtj&line=2): ``` class DataCollatorForMultipleChoice: tokenizer: PreTrainedTokenizerBase padding: Union[bool, str, PaddingStrategy] = True max_length: Optional[int] = None pad_to_multiple_of: Optional[int] = None def __call__ (self, features): label_name = "label" if "label" in features[0].keys() else Label_name # "labels" labels = [feature.pop(label_name) for feature in features] batch_size = len(features) num_choices = len(features[0]["input_ids"]) flattened_features = [] for feature in features: mclist = [] for i in range(NMultChoice): mcdict = {} mcdict['input_ids'] = feature['input_ids'] mcdict['token_type_ids'] = feature['token_type_ids'] mcdict['attention_mask'] = feature['attention_mask'] mclist.append(mcdict) flattened_features.append(mclist) flattened_features = list(chain(*flattened_features)) batch = self.tokenizer.pad( flattened_features, padding=self.padding, max_length=self.max_length, pad_to_multiple_of=self.pad_to_multiple_of, return_tensors="pt", ) # Un-flatten batch = {k: v.view(batch_size, num_choices, -1) for k, v in batch.items()} # Add back labels batch["labels"] = torch.tensor(labels, dtype=torch.int64) return batch ``` This DataCollator is then used in specification of the Trainer, specifically _specifying pad\_to\_multiple\_of=8_ and then using it to create batches: ``` data_collator = DataCollatorForMultipleChoice(tokenizer=tokenizer, pad_to_multiple_of=8), eval_loader = trainer.get_eval_dataloader(eval_dataset) for batch in eval_loader: ``` Now I am getting this exception: ``` File "/Users/rik/Code/eclipse/ai4law/src/casehold_demo-v2.py", line 444, in main for batch in eval_loader: File ".../lib/python3.11/site-packages/accelerate/data_loader.py", line 452, in __iter__ current_batch = next(dataloader_iter) ^^^^^^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 630, in __next__ data = self._next_data() ^^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/torch/utils/data/dataloader.py", line 674, in _next_data data = self._dataset_fetcher.fetch(index) # may raise StopIteration ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/torch/utils/data/_utils/fetch.py", line 57, in fetch return self.collate_fn(data) ^^^^^^^^^^^^^^^^^^^^^ File "/Users/rik/Code/eclipse/ai4law/src/casehold_demo-v2.py", line 157, in __call__ batch = self.tokenizer.pad( ^^^^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 3380, in pad return BatchEncoding(batch_outputs, tensor_type=return_tensors) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File ".../lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 224, in __init__ self.convert_to_tensors(tensor_type=tensor_type, prepend_batch_axis=prepend_batch_axis) File ".../lib/python3.11/site-packages/transformers/tokenization_utils_base.py", line 775, in convert_to_tensors raise ValueError( ValueError: Unable to create tensor, you should probably activate truncation and/or padding with 'padding=True' 'truncation=True' to have batched tensors with the same length. Perhaps your features (`input_ids` in this case) have excessive nesting (inputs type `list` where type `int` is expected). ``` I’m doing truncation as part of tokenization, and padding the batch examples with `pad_to_multiple_of=8`. Would you knopw why would I be getting this error?