Nutch, an open source search engine crawler.

For those of us who watch search engines closely and what they do on a daily basis, this article is interesting. Nutch is an open source search engine crawler which will do many of the same things that all the big search engine crawlers do. The only difference is it is open source and being worked on by any number of people for free. There are some people that have been trying this out for some time and are getting some good results. Can this be an alternative to other search engines that are out there when it is licensed to a company? As the article states, there needs to be a large amount of resources to create a real search engine because the crawler itself is only a part of the big picture. Those resources would cost a lot of money, so this isn't a way to have your own search engine on the cheap. But what is interesting is that here will be a working core similar to the Linux concept where people are using the Linux kernel to create their own operating system.

I'm not sure if this is the kind of thing that would be jumped on like early Linux development. There are so many resources that would have to be put in place to have a real search engine actually going out and crawling web sites. It's not a cheap undertaking at all, so even if the main core of the system is available free, it won't be free. I think it's great that people are putting together open-source software that is search engine related. It gives people more options for the future and will introduce more competition into the search engine game. I think you'll see companies adopt this open source called Nutch and use it in their own search engine from a marketing standpoint. That's really the power and all this, to marry some good technology in with your offering making it more sticky. This is how A9.com from Amazon has been doing things although they don't use Nutch, they use Google results. But the concept would be the same, to create a search engine that would augment their sales efforts.

With Nutch, there could be opportunities to make this kind of entry into the market much easier. Think of all the time and investment that would be avoided by having a search kernel all ready to be used and modify. It would bring in competition quickly and I think that this might be the avenue that will bring the competition that we would need to have better search results and not be stuck with only the major search engines. I believe were still at the beginning of all this search engine technology, with much more to come. Open source software once again will make things difficult for the more standard offerings in search just like it's caused problems for Microsoft in the operating system. It might be a couple of decades before things change where open source is the majority of what people use, but I believe it's on the way and Nutch is one of those technologies to watch.

Real search marketing for the web, RealWebMarketing.com

  Subscribe to this RSS news feed.