Comments on: Announcing the Public Terabyte Dataset project http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/ Wed, 03 Mar 2010 15:09:34 +0000 http://wordpress.com/ hourly 1 By: BotchagalupeMarks for November 13th - 11:23 | IT Management and Cloud Blog http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-18 BotchagalupeMarks for November 13th - 11:23 | IT Management and Cloud Blog Sat, 14 Nov 2009 06:26:50 +0000 http://bixolabs.com/?p=88#comment-18 [...] Announcing the Public Terabyte Dataset project « Elastic Web Mining | Bixolabs - This is a high quality crawl of top web sites, using AWS’s Elastic Map Reduce, Concurrent’s Cascading workflow API, and Bixolab’s elastic web mining platform. [...] [...] Announcing the Public Terabyte Dataset project « Elastic Web Mining | Bixolabs – This is a high quality crawl of top web sites, using AWS’s Elastic Map Reduce, Concurrent’s Cascading workflow API, and Bixolab’s elastic web mining platform. [...]

]]> By: kkrugler http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-16 kkrugler Tue, 10 Nov 2009 20:39:35 +0000 http://bixolabs.com/?p=88#comment-16 Hi Brewster, Thanks for the offer to host the crawl at the Internet Archive. I'll take you up on that offer, and follow up with Alexis. -- Ken Hi Brewster,

Thanks for the offer to host the crawl at the Internet Archive.

I’ll take you up on that offer, and follow up with Alexis.

– Ken

]]> By: Brewster Kahle http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-13 Brewster Kahle Sun, 08 Nov 2009 04:43:20 +0000 http://bixolabs.com/?p=88#comment-13 Thank you for doing this crawl! The Internet Archive would be happy to host this collection for free public access. You can push it to the Internet Archive given our implementation of S3. If this is interesting to you, please contact alexis rossi (alexis at archive). -brewster Thank you for doing this crawl!

The Internet Archive would be happy to host this collection for free public access. You can push it to the Internet Archive given our implementation of S3.

If this is interesting to you, please contact alexis rossi (alexis at archive).

-brewster

]]> By: Michael Nielsen » Biweekly links for 11/06/2009 http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-12 Michael Nielsen » Biweekly links for 11/06/2009 Fri, 06 Nov 2009 10:53:17 +0000 http://bixolabs.com/?p=88#comment-12 [...] The Public Terabyte Dataset project « Elastic Web Mining | Bixolabs [...] [...] The Public Terabyte Dataset project « Elastic Web Mining | Bixolabs [...]

]]> By: kkrugler http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-8 kkrugler Tue, 03 Nov 2009 23:08:15 +0000 http://bixolabs.com/?p=88#comment-8 Hi Otis, The dataset will be pushed to S3 as a set of compressed warc (web archive) files. Still working out how much additional data to include, but parse would be too big...might include compressed term vectors for Mahout-ers out there. The link you reference is where the sample code will be posted, for both the generation of the data (the crawl code) and examples of how to process the data files. -- Ken Hi Otis,

The dataset will be pushed to S3 as a set of compressed warc (web archive) files. Still working out how much additional data to include, but parse would be too big…might include compressed term vectors for Mahout-ers out there.

The link you reference is where the sample code will be posted, for both the generation of the data (the crawl code) and examples of how to process the data files.

– Ken

]]> By: Otis Gospodnetic http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-7 Otis Gospodnetic Tue, 03 Nov 2009 15:19:19 +0000 http://bixolabs.com/?p=88#comment-7 Ken, I'm looking at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263 but I don't see PTD there. What exactly is in the dataset? A bunch of raw, unparsed HTML pages, right? Ken, I’m looking at http://developer.amazonwebservices.com/connect/kbcategory.jspa?categoryID=263 but I don’t see PTD there.
What exactly is in the dataset? A bunch of raw, unparsed HTML pages, right?

]]> By: Notional Slurry » links for 2009-11-02 http://bixolabs.com/2009/11/01/announcing-the-public-terabyte-dataset-project/#comment-5 Notional Slurry » links for 2009-11-02 Tue, 03 Nov 2009 06:03:15 +0000 http://bixolabs.com/?p=88#comment-5 [...] Announcing the Public Terabyte Dataset project « Elastic Web Mining | Bixolabs "We’re very excited to announce the Public Terabyte Dataset project." (tags: data datasets mapreduce S3) [...] [...] Announcing the Public Terabyte Dataset project « Elastic Web Mining | Bixolabs "We’re very excited to announce the Public Terabyte Dataset project." (tags: data datasets mapreduce S3) [...]

]]>