# Chapter 5 questions **URL:** \ **Category:** Course\ **Created:** [November 15, 2021, 2:12pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744 "2021-11-15T14:12:53Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [November 15, 2021, 2:12pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/1 "2021-11-15T14:12:53Z")
Use this topic for any question about [Chapter 5](https://huggingface.co/course/chapter5) of the course. ---
**Author:** ![ducatyb](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@ducatyb](https://discuss.huggingface.co/u/ducatyb)\ **Post date:** [December 3, 2021, 5:52am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/2 "2021-12-03T05:52:04Z")
Please correct me if I’m wrong. A DataSet object can be thought of as some tabular data, whose rows and columns are examples and features respectively. The _length_ of a DataSet object is the number of examples, which equals to the number of its rows. Each column corresponds to one feature. Given this understanding of DataSet, I found the following descriptions in chapter 5 are confusing (in other words _incorrect_?). > … here those 1,000 examples gave 1,463 new features, resulting in a shape error. 1,463 is the number of rows (i.e. **examples** ) of the newly added columns (i.e. features) such as `attention_mask`, `input_ids`, etc. > We can check that our new dataset has many more features than the original dataset by comparing the lengths: > > len(tokenized\_dataset[“train”]), len(drug\_dataset[“train”]) > > (206772, 138514) Obviously the above two numbers are the numbers of rows, i.e. the numbers of **examples** , not the number of columns (features). The **number of features** in this case is 4. Specifically, these 4 features are `attention_mask` `input_ids` `overflow_to_sample_mapping` and `token_type_ids` ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [December 3, 2021, 8:19am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/3 "2021-12-03T08:19:43Z")
Hey @ducatyb, your understanding is indeed correct: rows are examples, and columns are features. We sometimes use “examples” and “features” interchangeably, but I agree this is confusing. I’ll improve the wording to make this clear - thank you for the feedback! ---
**Author:** ![ducatyb](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@ducatyb](https://discuss.huggingface.co/u/ducatyb)\ **Post date:** [December 4, 2021, 9:39am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/4 "2021-12-04T09:39:15Z")
Thanks for clarifying! ---
**Author:** ![ducatyb](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@ducatyb](https://discuss.huggingface.co/u/ducatyb)\ **Post date:** [December 4, 2021, 11:00am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/5 "2021-12-04T11:00:15Z")
In the section **Creating your own dataset** I got an import error when trying this code ```auto from huggingface_hub import list_datasets all_datasets = list_datasets() print(f"Number of datasets on Hub: {len(all_datasets)}") print(all_datasets[0]) ``` Perhaps the package name should’ve been `datasets` instead of `huggingface_hub` ? ---
**Author:** ![ducatyb](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@ducatyb](https://discuss.huggingface.co/u/ducatyb)\ **Post date:** [December 4, 2021, 12:20pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/6 "2021-12-04T12:20:39Z")
My bad. I was using an older version of `huggingface_hub`. The import error has gone after I installed the latest version, i.e. version 0.2.1 ---
**Author:** ![dansbecker](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/dansbecker/32/25019_2.png) [@dansbecker](https://discuss.huggingface.co/u/dansbecker)\ **Post date:** [December 5, 2021, 5:27am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/7 "2021-12-05T05:27:17Z")
The course is my first experience with HF, I’m totally wowed. Thanks for the great course and great tools! For question in the end of chapter quiz for chapter 5, the grading system shows the correct answer as **dataset.shuffle().select(range(len(50))**. I think you don’t mean to have the **len** in there, and it would instead be **dataset.shuffle().select(range(50))**. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [December 5, 2021, 10:09pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/8 "2021-12-05T22:09:52Z")
Hi @dansbecker thank you for the kind words! You’re totally right about the quiz - that’s a bug which I’ll fix right away 🙂 ---
**Author:** ![sgugger](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sgugger/32/2291_2.png) [@sgugger](https://discuss.huggingface.co/u/sgugger)\ **Post date:** [December 6, 2021, 3:25am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/9 "2021-12-06T03:25:29Z")
> [@ducatyb](#): > > A DataSet object can be thought of as some tabular data, whose rows and columns are examples and features respectively. The _length_ of a DataSet object is the number of examples, which equals to the number of its rows. Each column corresponds to one feature. I disagree with that analysis. Here we are in a situation where we create several training samples from one example by applying some preprocessing, which is generally called _feature extraction_. In tabular data feature extraction often means adding new columns to the dataset (although it sometimes means removing some), which leads to people often calling the columns features. Here in this example, it means adding more rows, wo calling the rows the features makes perfect sense to me. We could however add a remark to explain this 🙂 ---
**Author:** ![satpalsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/satpalsr/32/3553_2.png) [@satpalsr](https://discuss.huggingface.co/u/satpalsr)\ **Post date:** [December 14, 2021, 5:26am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/10 "2021-12-14T05:26:26Z")
In HF Chapter 5, “Time to slice and dice”: [Time to slice and dice - Hugging Face Course](https://huggingface.co/course/chapter5/3?fw=pt#the-%3Ccode%3Emap()%3C/code%3E-method%E2%80%99s-superpowers) I observe ```auto new_drug_dataset = drug_dataset.map( lambda x: {"review": html.unescape(x["review"])}, batched=True) ``` to work slightly better than the used ```auto new_drug_dataset = drug_dataset.map( lambda x: {"review": [html.unescape(o) for o in x["review"]]}, batched=True ) ``` Results: ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/5/544a232549ce75c8175cb19eb19462f6674f58af.png) ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [December 14, 2021, 10:15am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/11 "2021-12-14T10:15:32Z")
Hey @satpalsr that’s an interesting result, but I think you’ll find that `html.unescape()` won’t actually unescape the HTML characters in your first example because it expects a string, not a list of strings. You can check the outputs by inspecting the first element of the `train` split in both cases 🙂 ---
**Author:** ![DoyyingFace](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@DoyyingFace](https://discuss.huggingface.co/u/DoyyingFace)\ **Post date:** [January 18, 2022, 5:10pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/12 "2022-01-18T17:10:01Z")
Hi! I have an issue when saving the dataset: ```auto drug_dataset_clean.save_to_disk("drug-reviews") ``` A TypeError has been raised. I’ve tried another dataset with the type Dataset (freq\_dataset in the tutorial, for example), and it works. So maybe the problem is the data type of drug\_dataset\_clean. ---
**Author:** ![DoyyingFace](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@DoyyingFace](https://discuss.huggingface.co/u/DoyyingFace)\ **Post date:** [January 19, 2022, 10:40am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/13 "2022-01-19T10:40:57Z")
I think there is a mistake in the ‘Creating your own dataset’ section: ```auto issues_with_comments_dataset = issues_dataset.map( lambda x: {"comments": get_comments(x["number"])} ) ``` It will raise a TypeError because the data type of x[‘number’] is numpy.int64 instead of int, and the former could not be used as an index. So ```auto issues_with_comments_dataset = issues_dataset.map( lambda x: {"comments": get_comments(int(x["number"]))} ) ``` will work. ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [January 20, 2022, 11:39am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/14 "2022-01-20T11:39:56Z")
Hi @DoyyingFace, thank you for reporting this error! Unfortunately, I am not able to reproduce the error using the code in the chapter’s Colab. Perhaps you missed a cell that needed executing or are using an old version of `datasets`? If the problem remains, can you share a copy of the notebook that you’re running? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [January 21, 2022, 2:25pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/15 "2022-01-21T14:25:35Z")
Hi @DoyyingFace thank you for raising this error. Unfortunately I am not able to reproduce it using the Colab notebook provided with the chapter. Perhaps you are using an old version of `datasets`? If the problem persists, can you please share the notebook you are getting the error in? ---
**Author:** ![DoyyingFace](https://avatars.discourse-cdn.com/v4/letter/d/e8c25b/32.png) [@DoyyingFace](https://discuss.huggingface.co/u/DoyyingFace)\ **Post date:** [January 23, 2022, 2:10am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/16 "2022-01-23T02:10:42Z")
Hi! I ran the notebook attached in the tutorial and it worked as you said (for both replies I made here). So maybe there is something wrong with my notebook, and I might check my version. Thanks for your help! BTW, what is about the old version of `datasets`? ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [January 24, 2022, 3:54pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/17 "2022-01-24T15:54:51Z")
Glad to hear it is working! The comment I made about the `datasets` version is that there each release often contains various bug fixes, so upgrading to the latest version is often a quick way to ensure a bug is really coming from the code, not the library 🙂 ---
**Author:** ![Evan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/evan/32/2539_2.png) [@Evan](https://discuss.huggingface.co/u/Evan)\ **Post date:** [March 5, 2022, 1:40pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/18 "2022-03-05T13:40:01Z")
Hello Everyone! I’m trying to scrape spaCy’s github issues using the steps outlined in [Creating your own dataset]([Creating your own dataset](https://huggingface.co/course/chapter5/5?fw=pt)) as recommended in the [✏ **Try it out!**](https://huggingface.co/course/chapter5/5?fw=pt#:~:text=%E2%9C%8F%EF%B8%8F%20Try%20it%20out!%20Go%20through%20the%20steps%20we%20took%20in%20this%20section%20to%20create%20a%20dataset%20of%20GitHub%20issues%20for%20your%20favorite%20open%20source%20library%20(pick%20something%20other%20than%20%F0%9F%A4%97%20Datasets%2C%20of%20course!).) at the end of the tutorial, which, btw, are fantastic when I hit the following error: ```auto from datasets import load_dataset issues_spacy = load_dataset('json', data_files="spacy-issues.jsonl", split='train') issues_spacy --------------------------------------------------------------------------- TypeError Traceback (most recent call last) in () 3 issues_spacy = load_dataset('json', 4 data_files="spacy-issues.jsonl", ----> 5 split='train') 6 7 issues_spacy 14 frames /usr/local/lib/python3.7/dist-packages/datasets/table.py in array_cast(array, pa_type, allow_number_to_str) 1017 ) 1018 return array.cast(pa_type) -> 1019 raise TypeError(f"Couldn't cast array of type\n{array.type}\nto\n{pa_type}") 1020 1021 TypeError: Couldn't cast array of type struct, open_issues: int64, closed_issues: int64, state: string, created_at: timestamp[s], updated_at: timestamp[s], due_on: null, closed_at: timestamp[s]> to null ``` I’ve already searched for what to do but am at a loss at the moment. Any ideas? Cheers! ---
**Author:** ![lewtun](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/lewtun/32/30995_2.png) [@lewtun](https://discuss.huggingface.co/u/lewtun)\ **Post date:** [March 9, 2022, 11:13am UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/19 "2022-03-09T11:13:52Z")
Hey @Evan thanks for reporting this error! It looks like it might be a low-level problem with the way we parse JSON files in `datasets`. Would you mind uploading your dataset to the Hub and sharing it here so I can try to reproduce it on my side? Thanks! ---
**Author:** ![Evan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/evan/32/2539_2.png) [@Evan](https://discuss.huggingface.co/u/Evan)\ **Post date:** [March 13, 2022, 9:28pm UTC](https://discuss.huggingface.co/t/chapter-5-questions/11744/20 "2022-03-13T21:28:30Z")
Hey @lewtun! A thousand pardons for not responding sooner. I’m actually having a really hard time reproducing the error myself: when I set `num_issues` to 2\_500 or 5\_000, everything runs just fine. However, when I bump it up to 10\_000 in the example, I get the error above 😖 As for loading the data to the hub, I’ve uploaded the .json file here: [Evan/spaCy-github-issues · Datasets at Hugging Face](https://huggingface.co/datasets/Evan/spaCy-github-issues) Cheers! [Next page](https://discuss.huggingface.co/t/chapter-5-questions/11744.md?page=2)