Nonprofit Common Crawl Offers a Database of the Entire Web, For Free, and Could Open Up Google to New Competition | MIT Technology Review

An uncommon resource

"A nonprofit called Common Crawl is now using its own Web crawler and making a giant copy of the Web that it makes accessible to anyone. The organization offers up over five billion Web pages, available for free so that researchers and entrepreneurs can try things otherwise possible only for those with access to resources on the scale of Google’s.
[...]
Elbaz is the founder and CEO of big data company Factual, and before that founded a company bought by Google to be the basis of its ad business for Web pages. Common Crawl also has Google’s director of research, Peter Norvig, and MIT Media Lab director Joi Ito on its advisory board."

Nonprofit Common Crawl Offers a Database of the Entire Web, For Free, and Could Open Up Google to New Competition | MIT Technology Review