New Search Engine - YaCy - first of a kind?

[Clinton]

YaCy (pron: Ya See) follows a rich tradition of stupid names, but it looks interesting because of the distributed structure and the individual's control of their personalised search experience.

I've been predicting for years that ultimately search will be taken out of the hands of the big bullies and become a desktop application. I see increased personalisation resulting in individuals hosting and controlling the software/"personal assistant" that they can fine tune to their search preferences. Is this a move towards that kind of future?

"For such an essential function (search), we cannot rely on a few large companies, and compromise our privacy in the process," he said.

YaCy (pronounced "Ya See") is supported by the Free Software Foundation Europe (FSFE) which campaigns on digital rights and tries to help people control their own digital destiny.

FSFE said YaCy helps privacy by encrypting all queries and by letting peer owners build up and manage their own search profile.

"We are moving away from the idea that services need to be centrally controlled," said Karsten Gerloff, president of the FSFE. "Instead, we are realising how important it is to be independent, and to create infrastructure that doesn't have a single point of failure."

YaCy software is available for Windows, Linux and MacOS and users are being encouraged to download and run it for themselves.

User-Agent: yacy

You can submit sites for crawling if you're running your own peer i.e. you've downloaded the software and are using YaCy.

[JJMcClure]

I've been predicting for years that ultimately search will be taken out of the hands of the big bullies and become a desktop application.

I can't get it to load, you must have crashed it with all the traffic you sent from EP

I don't understand how search can become desk top, there's still got to be something to search so who'll host the index? Even if it's decentralised, someone somewhere has got to be crawling and indexing pages. If we're allowed to create our own search profile, does that mean we'll have to re-invent a way to score and rank reources? It sounds like a total nightmare and the spammers must be rubbing their hands with glee..

Besides, Google and the other search engines are only crawling a fraction of the web. What would be really innovative is if someone could come up with a way to search the deepweb and even Google haven't figured that one out.

[benitez17]

I finally got a chance to look at this, and I like it quite a bit. I plan to find some time to contribute to its development and will begin using it.

There's no reason the search index can't be hosted on PCs, crawling can be done with spare cycles, and the results can be filtered for our personal preferences locally, rather than doing all of this within Google's server farm.

[Clinton]

Even if it's decentralised, someone somewhere has got to be crawling and indexing pages

Yes. That's the dumb part of the job and relatively easily done.

The sorting results is the tricky part and the part that Google has done well because of their huge computing power and their complex algorithms developed over many years. However, individual desktops are a lot more powerful than they were a few years ago and a few million of those desktops in a distributed environment have the ability to eclipse Google's computing might several times over. Algorithms could be integrated into a desktop application - just like email spam filters and anti-virus programs - which allows a high degree of tweaking and maybe even has heuristic/metaheuristic elements.

In fact, what Google has is not one massive supercomputer but a lot of commodity class x86 machines.

[JJMcClure]

individual desktops are a lot more powerful than they were a few years ago and a few million of those desktops in a distributed environment have the ability to eclipse Google's computing might several times over.

True but unlikely to happen? Also, there still has to be something to be searched since searching in real time just isn't feasible, who's going to host the index?

[eddiebooth]

Ive had a look at and, although I like the general idea behind it, I have a number of concerns.
If it relies on you and me to build up its database, how long will it take to become as comprehensive as Google?
Will it be vulnerable to manipulation by people spamming it with their own websites?
How secure is it? I think it unlikely that without the financial clout and staff numbers of the big search engines YaCy will be able to guarantee that if I click onto a site I will actually get there rather than allow some malicious spyware etc to access my pc.
What does this mean for SEO if everyone can in effect customise their own algorithms?
And finally, at some point the monetisation of YaCy will become an issue. There is just too much potential money to be made if it succeeds to be ignored.

[benitez17]

True but unlikely to happen? Also, there still has to be something to be searched since searching in real time just isn't feasible, who's going to host the index?

The index can be distributed among users, just like it is distributed among Google's thousands of servers. It's more complex because you don't have any control over when users are online, but still not too difficult.

[benitez17]

Ive had a look at and, although I like the general idea behind it, I have a number of concerns.
If it relies on you and me to build up its database, how long will it take to become as comprehensive as Google?
Will it be vulnerable to manipulation by people spamming it with their own websites?
How secure is it? I think it unlikely that without the financial clout and staff numbers of the big search engines YaCy will be able to guarantee that if I click onto a site I will actually get there rather than allow some malicious spyware etc to access my pc.
What does this mean for SEO if everyone can in effect customise their own algorithms?
And finally, at some point the monetisation of YaCy will become an issue. There is just too much potential money to be made if it succeeds to be ignored.

If they are smart, they will seed their index using one of the existing open source indexes, like the one mentioned here: http://experienced-people.net/forums...-billion-pages)

It's open source, which makes it more difficult to subvert as long as there is an active development community associated with it. There's no reason it couldn't be monetized at some point without harming the quality of the results.

Personalized search is already here, this could be the next generation of it if successful. That's not good for SEOs, but it could be good for users.

[WannaBeGeekster]

Just like BigCouchDB that stores multiple copies of the data in a number of different locations just in case one of the nodes goes down. With BigCouchDB you can configure the number of duplicate copies to save in your configuration.

I am sure that the desktop component you install on your computer is used to crawl the web. I am sure that it also stores various pieces of the index on your computer as well. That seems to be the only way it will work.

I am more worried about what would happen if net neutrality goes away. The demise of Net Neutrality could kill this search engine with a simple block rule.

[benitez17]

Just like BigCouchDB that stores multiple copies of the data in a number of different locations just in case one of the nodes goes down. With BigCouchDB you can configure the number of duplicate copies to save in your configuration.

I am sure that the desktop component you install on your computer is used to crawl the web. I am sure that it also stores various pieces of the index on your computer as well. That seems to be the only way it will work.

Exactly.

I am more worried about what would happen if net neutrality goes away. The demise of Net Neutrality could kill this search engine with a simple block rule.

Why do you think ISPs would block this SE?