Comments on: Public Terabyte Dataset Project http://bixolabs.com Wed, 03 Mar 2010 15:09:34 +0000 http://wordpress.com/ hourly 1 By: kkrugler http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-68 kkrugler Wed, 03 Mar 2010 15:09:34 +0000 http://bixolabs.com/?page_id=101#comment-68 @elhoim - I hadn't found the domain lists on premiumdrops.com ...good stuff. Sounds like you've been poking around this space a bit in the past :) -- Ken @elhoim – I hadn’t found the domain lists on premiumdrops.com …good stuff.

Sounds like you’ve been poking around this space a bit in the past :)

– Ken

]]> By: elhoim http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-67 elhoim Wed, 03 Mar 2010 14:40:54 +0000 http://bixolabs.com/?page_id=101#comment-67 premiumdrops.com is also offering copies of .com/.net/.org zones if you want to constitue a wide seed list. premiumdrops.com is also offering copies of .com/.net/.org zones if you want to constitue a wide seed list.

]]> By: kkrugler http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-66 kkrugler Wed, 03 Mar 2010 14:35:00 +0000 http://bixolabs.com/?page_id=101#comment-66 Hi Joel, Excellent input, thanks! The Alexa web service API offers a bit more information (e.g. is it "adult"), but it feels out of date (e.g. no walmart.com data). By merging these together, I'll have a much better see list. -- Ken Hi Joel,

Excellent input, thanks! The Alexa web service API offers a bit more information (e.g. is it “adult”), but it feels out of date (e.g. no walmart.com data). By merging these together, I’ll have a much better see list.

– Ken

]]> By: elhoim http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-65 elhoim Wed, 03 Mar 2010 09:27:47 +0000 http://bixolabs.com/?page_id=101#comment-65 Alexa and quantcast both have a free "top 1 million" list that is updated daily. Alexa and quantcast both have a free “top 1 million” list that is updated daily.

]]> By: kkrugler http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-61 kkrugler Mon, 25 Jan 2010 20:45:30 +0000 http://bixolabs.com/?page_id=101#comment-61 Hi Dan, 1. Top sites by traffic come from Alexa web services API. 2. The dataset focuses on these sites, but expands to include others. 3. Amazon is helping because they are interested in useful public datasets and examples of effectively using EMR. 4. The dataset has highest value for people working on text processing algorithms, and for doing performance baselining/optimizations. -- Ken Hi Dan,

1. Top sites by traffic come from Alexa web services API.
2. The dataset focuses on these sites, but expands to include others.
3. Amazon is helping because they are interested in useful public datasets and examples of effectively using EMR.
4. The dataset has highest value for people working on text processing algorithms, and for doing performance baselining/optimizations.

– Ken

]]> By: Dan http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-60 Dan Mon, 25 Jan 2010 20:20:28 +0000 http://bixolabs.com/?page_id=101#comment-60 How did you get the top 100K websites by traffic? Is the datasets all pages from those websites? How did you get Amazon to host the data? What do you view as the value of this dataset? How did you get the top 100K websites by traffic?
Is the datasets all pages from those websites?
How did you get Amazon to host the data?
What do you view as the value of this dataset?

]]> By: The web is an endless series of edge cases « Ken's Techno Tidbits http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-49 The web is an endless series of edge cases « Ken's Techno Tidbits Thu, 17 Dec 2009 20:31:52 +0000 http://bixolabs.com/?page_id=101#comment-49 [...] that same week, I was debugging a weird problem with my Elastic MapReduce web crawling job for the Public Terabyte Datset project. At some point during one of the steps, I was getting LeaseExpiredExceptions in the logs, and the [...] [...] that same week, I was debugging a weird problem with my Elastic MapReduce web crawling job for the Public Terabyte Datset project. At some point during one of the steps, I was getting LeaseExpiredExceptions in the logs, and the [...]

]]> By: Bixolabs goes public « Ken's Techno Tidbits http://bixolabs.com/datasets/public-terabyte-dataset-project/#comment-4 Bixolabs goes public « Ken's Techno Tidbits Tue, 03 Nov 2009 00:48:12 +0000 http://bixolabs.com/?page_id=101#comment-4 [...] gave a talk at the ACM Data Mining Unconference on Sunday, where I also announced the Public Terabyte Dataset project, so the timing was [...] [...] gave a talk at the ACM Data Mining Unconference on Sunday, where I also announced the Public Terabyte Dataset project, so the timing was [...]

]]>