The #RedPajama #LLM is so painfully close to being truly #OpenSource. Just a few tweaks needed:
- Dropping CommonCrawl/C4 entirely
- Fixing the Gutenberg crawler to stick to public domain books
- Filtering arXiv to return only CC-By(-SA) papers
huggingface.co/datasets/togethercomputer/RedPajama-Data-1T
- Dropping CommonCrawl/C4 entirely
- Fixing the Gutenberg crawler to stick to public domain books
- Filtering arXiv to return only CC-By(-SA) papers
huggingface.co/datasets/togethercomputer/RedPajama-Data-1T
togethercomputer/RedPajama-Data-1T · Datasets at Hugging Face
We’re on a journey to advance and democratize artificial intelligence through open source and open science.huggingface.co