Public Terabyte Dataset Project
Public Terabyte Dataset Project
This page has more details on the Public Terabyte Dataset project, which was recently announced at the ACM data mining unconference.
- The data comes from a crawl of 50-200M pages from the
100K1 million top (by US-based traffic) domains. - The crawl is done by a custom Bixo workflow created by Bixo Labs, built on top of Cascading/Hadoop and running in EC2 using Amazon’s Elastic MapReduce service.
- We’ll be trying hard to avoid spam/adult content, though getting totally clean results is of course impossible.
- We honor the robots nofollow and various “no archive” HTML meta tags, and we’ll comply promptly with requests by web site hosters to remove any of their content from the datasets. Note that the rel=”nofollow” attribute on links themselves does not mean that the link shouldn’t be followed, but rather that the link shouldn’t be used when calculating a “PageRank” score.
- The resulting data will be stored as compressed
warcAvro files in S3. - Hosting for the dataset is being provided by Amazon.
- Access to the data is free, assuming you’re running code in EC2.
- The code used to run the crawl, as well as code to access the crawl data, will be available at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263.
There’s a form where you can request information and provide input on the crawl.
10 Comments
leave one →
How did you get the top 100K websites by traffic?
Is the datasets all pages from those websites?
How did you get Amazon to host the data?
What do you view as the value of this dataset?
Hi Dan,
1. Top sites by traffic come from Alexa web services API.
2. The dataset focuses on these sites, but expands to include others.
3. Amazon is helping because they are interested in useful public datasets and examples of effectively using EMR.
4. The dataset has highest value for people working on text processing algorithms, and for doing performance baselining/optimizations.
– Ken
Alexa and quantcast both have a free “top 1 million” list that is updated daily.
Hi Joel,
Excellent input, thanks! The Alexa web service API offers a bit more information (e.g. is it “adult”), but it feels out of date (e.g. no walmart.com data). By merging these together, I’ll have a much better see list.
– Ken
premiumdrops.com is also offering copies of .com/.net/.org zones if you want to constitue a wide seed list.
@elhoim – I hadn’t found the domain lists on premiumdrops.com …good stuff.
Sounds like you’ve been poking around this space a bit in the past
– Ken