Inspiration

Bixolabs was built to solve two major problems that the founders experienced first-hand and previous startups and consulting projects.

First, companies that want to leverage web data have expertise in the data, and how it gets analyzed/monetized. Companies DO NOT have expertise in the arcane art of web crawling, which means they wind up spending a lot of time and money dealing with getting the data, not using the data.

At Krugle, for example, we used a modified version of an open source web crawler (Nutch), running on a dedicated in-house cluster, to do vertical crawls of the technical web. The up-front hardware costs and on-going IT requirements were staggering. Even with the help of the lead Nutch developer as a consultant, we were frequently stuck solving web crawler problems. And our Hadoop cluster was idle most of the time.

With Bixolabs, the focus is on the specific customization required to specify what to crawl, and how to process the results. You only pay for what you’re using. You only develop what you need, using unique knowledge you have about the problem space. You don’t configure servers, manage clusters, monitor crawls, block honeypots, calm down angry webmasters, or pay for idle hardware.

Second, processing web data requires a workflow system that is efficient, reliable and scalable. Even a small crawl will result in millions of pages and many gigabytes of data. A workflow based on ad hoc scripts isn’t reliable. Single server solutions don’t scale.

With Bixolabs, data flows into a Cascading-based workflow system. The size of the cluster varies dynamically, based on the size of your data. Re-running an analysis is easy, reliable, and leverages the web cache for optimum efficiency.

Founder

Ken Krugler – Veteran developer and entrepreneur, 25+ years experience. Founder and President of TransPac Software, a 20 year leader in internationalization, mobile devices, and search consulting. Founder and CTO of Krugle, a vertical search engine and enterprise appliance for code and technical information (funded by Emergence Capital). Co-founder of Bixo web mining project. Author and speaker on vertical search and web mining.

Technical Advisors

	Stefan Groschupf – Founder and President of 101tec, a multinational search/Hadoop consulting company. Co-founder of Scale Unlimited, a cloud computing training company. Creator of Katta search project. Co-founder of Bixo web mining project.
	Peter Voß – Technical lead on large scale Hadoop-based data processing system. Former lead on ultra high volume data processing system for Deutsche Post.