# DALL-E - mini version **URL:** \ **Category:** Flax/JAX Projects\ **Created:** [June 25, 2021, 1:52am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324 "2021-06-25T01:52:11Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [June 25, 2021, 1:52am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/1 "2021-06-25T01:52:11Z")
**Project** Recreate a mini-version of DALL-E **Useful links:** - [Project Report](https://wandb.ai/dalle-mini/dalle-mini/reports/DALL-E-mini--Vmlldzo4NjIxODA) - [Demo](https://huggingface.co/spaces/flax-community/dalle-mini) - [Github Repo](https://github.com/borisdayma/dalle-mini) **Original post:** It would be cool to recreate something like [DALL-E](https://openai.com/blog/dall-e/). I can see some challenges though: - difficult paper though the main concepts could be simplified - availability/collection of dataset - the model used was GPT-3, not sure if interesting results can be obtained with smaller models - may require more time to implement Also I don’t believe a public version is available so maybe it will be easier to just do it in Pytorch (I have no experience yet with Flax/JAX). ---
**Author:** ![Vaibhavbrkn](https://avatars.discourse-cdn.com/v4/letter/v/74df32/32.png) [@Vaibhavbrkn](https://discuss.huggingface.co/u/Vaibhavbrkn)\ **Post date:** [June 25, 2021, 3:12am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/2 "2021-06-25T03:12:41Z")
We Could train NLP based models to generate embedding for sentences and then use this embeddings as input embedding for gan model to generate images. This is pretty simple idea just vice versa of image captioning, but I think it can work pretty well given the performance of transformers based models. ---
**Author:** ![raghav66](https://avatars.discourse-cdn.com/v4/letter/r/a9a28c/32.png) [@raghav66](https://discuss.huggingface.co/u/raghav66)\ **Post date:** [June 25, 2021, 6:08am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/3 "2021-06-25T06:08:31Z")
Thats a great idea. Can I join? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 25, 2021, 12:00pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/4 "2021-06-25T12:00:03Z")
Hey @boris That’s a great idea, I’m super excited about this! Regarding the dataset, here are a few that can be used for this - [Conceptual Caption](https://ai.google.com/research/ConceptualCaptions/) - ~3.3M image-text pairs - [Conceptual 12M](https://github.com/google-research-datasets/conceptual-12m) - 12M - [WIT](https://github.com/google-research-datasets/wit)- ~37M Also regarding architecture, seems using a simple scaled-up GPT2 as LM can also give good results (see [CogView](https://github.com/THUDM/CogView)). By simple, I mean no sparse attention or row-column attentions, etc which is used in the DALL-E I guess. And GPT2 is already available in JAX. And JAX is way faster on TPU than PT 😉 And if you look at this [discussion](https://github.com/lucidrains/DALLE-pytorch/discussions/131) it seems an even smaller model could give good enough results on the domain-specific dataset. Also as suggested in the discussion above using the VQGAN from [taming-transformers](https://github.com/CompVis/taming-transformers/) as the images tokenizer can further reduce the complexity of training such models as the max image token length for these VQGAN models is 256 (way less than DALL-E’s VQVAE, which uses 1024), so overall 256 text tokens and 256 image tokens = 512, should be manageable on a single v3-8. ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [June 25, 2021, 3:52pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/5 "2021-06-25T15:52:50Z")
Those are really great ideas @Vaibhavbrkn @valhalla (and thanks for the datasets)! @raghav66 Anybody can join! Also I noticed lucidrains has a pretty awesome repo that should help so we can pull some items from it as well: [DALLE-pytorch](https://github.com/lucidrains/DALLE-pytorch) ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 25, 2021, 3:55pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/6 "2021-06-25T15:55:25Z")
> Also I noticed lucidrains has a pretty awesome repo that should help so we can pull some items from it as well: [DALLE-pytorch](https://github.com/lucidrains/DALLE-pytorch) Yeah, the linked discussion and the VQGAN idea is from the DALLE-pytorch repo 😁 Also, this just dropped [GitHub - google-research/xmcgan\_image\_generation](https://github.com/google-research/xmcgan_image_generation) ---
**Author:** ![srmsoumya](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/srmsoumya/32/1827_2.png) [@srmsoumya](https://discuss.huggingface.co/u/srmsoumya)\ **Post date:** [June 25, 2021, 4:34pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/7 "2021-06-25T16:34:27Z")
I would love to try this as well. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 28, 2021, 4:38pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/8 "2021-06-28T16:38:08Z")
Let’s officially define this project 🙂 Putting everybody in the official sheet [here](https://docs.google.com/spreadsheets/d/1GpHebL7qrwJOc9olTpIPgjf8vOS0jNb6zR_B8x_Jtik/edit?usp=sharing). More people can still join! Leave a comment here or on the sheet if you want to change something. ---
**Author:** ![morgan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morgan/32/2293_2.png) [@morgan](https://discuss.huggingface.co/u/morgan)\ **Post date:** [June 29, 2021, 10:44am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/9 "2021-06-29T10:44:38Z")
Great idea, would love to join! ---
**Author:** ![morgan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morgan/32/2293_2.png) [@morgan](https://discuss.huggingface.co/u/morgan)\ **Post date:** [June 29, 2021, 11:11am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/10 "2021-06-29T11:11:06Z")
Also we could try use [GPT-J](https://github.com/kingoflolz/mesh-transformer-jax/#gpt-j-6b), a 6bn param version of GPT-3. There is a [PR for a HF](https://github.com/huggingface/transformers/pull/12243) port open atm too ---
**Author:** ![devv](https://avatars.discourse-cdn.com/v4/letter/d/5f8ce5/32.png) [@devv](https://discuss.huggingface.co/u/devv)\ **Post date:** [June 29, 2021, 11:53pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/11 "2021-06-29T23:53:20Z")
Thank you @Boris, for opening accepting more people to join. I am very interested in joining as well. Thanks. ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 30, 2021, 8:46am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/12 "2021-06-30T08:46:31Z")
Added you to the team 🙂 ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [June 30, 2021, 8:50am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/13 "2021-06-30T08:50:16Z")
I would love to be involved with this! Yeah, we could `GPT-J`, but we might need to choose smaller model (maybe 1B ?) due to the time and compute constraint. The HF PR is for PT, adding JAX version will take some time. But it’s somewhat similar to GPT2 which is already available in flax, which we could use. I wrote a [JAX version of VQGAN](https://github.com/patil-suraj/vqgan-jax) which can be used as the image tokenizer. ---
**Author:** ![morgan](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/morgan/32/2293_2.png) [@morgan](https://discuss.huggingface.co/u/morgan)\ **Post date:** [June 30, 2021, 10:32am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/14 "2021-06-30T10:32:46Z")
noice! Yeah I was unsure about the speed of GPT-J, at least its already in jax. Curious to see how much a TPUv3-8 can handle! ---
**Author:** ![boris](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/boris/32/19312_2.png) [@boris](https://discuss.huggingface.co/u/boris)\ **Post date:** [June 30, 2021, 3:18pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/15 "2021-06-30T15:18:48Z")
I created a [Discord channel here](https://discord.gg/wZ3Y8nWM). Let’s try to finalize the details. I tried to summarize all the ideas from the discussion: - task: image generation from text - model - dataset (thanks @valhalla): - training script: TODO - outcome: model that accepts a sentence and generate a realistic image from it Also feel free to pick a specific area you would want to focus on! On my side I can also help instrument the training with W&B so we can visualize metrics and some sample images generated live. ---
**Author:** ![ghosh-r](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ghosh-r/32/2183_2.png) [@ghosh-r](https://discuss.huggingface.co/u/ghosh-r)\ **Post date:** [June 30, 2021, 7:12pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/16 "2021-06-30T19:12:43Z")
I would really like to join this project. - I am working as a Computer Vision Research Consultant and can contribute to topics that require knowledge in Vision. - DALL-E is something that is quite new and I would like to be a part of this project that recreates it. - I am a fast learner and a team player. - I have experience working with Python (3 years) and PyTorch (1+ year) and version control (2 years) - I have experience in public-speaking, and good communication and writing skills. With this, I will be able to contribute significantly towards creating the demo. I am especially interested in this because **it democratizes Dall-E and lowers the barrier to use this model** , as I have special interest in such projects. Please consider my request to join in this effort. Thanks. 🙂 ---
**Author:** ![tmabraham](https://avatars.discourse-cdn.com/v4/letter/t/858c86/32.png) [@tmabraham](https://discuss.huggingface.co/u/tmabraham)\ **Post date:** [June 30, 2021, 7:24pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/17 "2021-06-30T19:24:13Z")
I would love to participate in this project as well. Please let me know what I can do to help 🙂 ---
**Author:** ![ghosh-r](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ghosh-r/32/2183_2.png) [@ghosh-r](https://discuss.huggingface.co/u/ghosh-r)\ **Post date:** [June 30, 2021, 9:03pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/18 "2021-06-30T21:03:00Z")
@patrickvonplaten will you please _officially_ add me? 🤗 (and also add to the Sheets document) Received confirmation from @boris on Discord. ---
**Author:** ![tmabraham](https://avatars.discourse-cdn.com/v4/letter/t/858c86/32.png) [@tmabraham](https://discuss.huggingface.co/u/tmabraham)\ **Post date:** [June 30, 2021, 9:05pm UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/19 "2021-06-30T21:05:53Z")
@patrickvonplaten I would also like to be also officially added, thanks! ---
**Author:** ![lkhphuc](https://avatars.discourse-cdn.com/v4/letter/l/2bfe46/32.png) [@lkhphuc](https://discuss.huggingface.co/u/lkhphuc)\ **Post date:** [July 1, 2021, 8:22am UTC](https://discuss.huggingface.co/t/dall-e-mini-version/7324/20 "2021-07-01T08:22:55Z")
I would like to join this project as well if there’s still open slot. @valhalla @boris [Next page](https://discuss.huggingface.co/t/dall-e-mini-version/7324.md?page=2)