Yacy
Distributed API Search Server for Twitter & Social Media.
Collaborators: Michael Peter Christen, Mario Behling and The FOSSASIA community.
Background
YaCy is a free search engine that anyone can use to build a search portal for their intranet or to help search the public internet. When contributing to the world-wide peer network, the scale of YaCy is limited only by the number of users in the world and can index billions of web pages. It is fully decentralized, all users of the search engine network are equal, the network does not store user search requests and it is not possible for anyone to censor the content of the shared index. We want to achieve freedom of information through a free, distributed web search which is powered by the world's users.
Imagine if, rather than relying on the proprietary software of a large professional search engine operator, your search engine was run by many private computers which aren't under the control of any one company or individual. Well, that's what YaCy does! The resulting decentralized web search currently has about 1.4 billion documents in its index (and growing - download and install YaCy to help out!) and more than 600 peer operators contribute each month. About 130,000 search queries are performed with this network each day.
How it works?
YaCy defines its own address space and tries to keep every node informed about all the other nodes that are currently online. Further, it defines protocols for joining and leaving the network.
Every YaCy peer separates all the other peers in two local sets. Firstly, the active set contains all the other peers in the YaCy network which are currently online. Secondly, the passive set contains all the other peers that are currently offline and were online in the past month. If a peer is offline for longer than one month, the peer is removed from the passive set.
The YaCy software enables everyone to set up their own distributed search engine, which is independent from the freeworld network. This serves people that for example need a search engine in their local intranet and do not want to use commercial solutions, such as Google Enterprise Search. However, YaCy further offers users to add clusters to the freeworld network in order to enable a more federated approach.
A YaCy peer’s position in the YaCy address space is determined by its peer hash. Peer hashes are persistent and thus, a peer only generates a peer the first time it joins the YaCy network. For any subsequent join, a peer reuses its peer hash.
