Namespace Lucene.Net.Replicator

Files replication framework

The Replicator allows replicating files between a server and client(s). Producers publish revisions and consumers update to the latest revision available. ReplicationClient is a helper utility for performing the update operation. It can be invoked either manually or periodically by [starting an update thread](ReplicationClient.html#startUpdateThread(long, java.lang.String)). HttpReplicator can be used to replicate revisions by consumers that reside on a different node than the producer.

The replication framework supports replicating any type of files, with built-in support for a single search index as well as an index and taxonomy pair. For a single index, the application should publish an IndexRevision and set IndexReplicationHandler on the client. For an index and taxonomy pair, the application should publish an IndexAndTaxonomyRevision and set IndexAndTaxonomyReplicationHandler on the client.

When the replication client detects that there is a newer revision available, it copies the files of the revision and then invokes the handler to complete the operation (e.g. copy the files to the index directory, fsync them, reopen an index reader etc.). By default, only files that do not exist in the handler's current revision files are copied, however this can be overridden by extending the client.

An example usage of the Replicator:

// ++++++++++++++ SERVER SIDE ++++++++++++++ // IndexWriter publishWriter; // the writer used for indexing Replicator replicator = new LocalReplicator(); replicator.publish(new IndexRevision(publishWriter));

// ++++++++++++++ CLIENT SIDE ++++++++++++++ // // either LocalReplictor, or HttpReplicator if client and server are on different nodes Replicator replicator;

// callback invoked after handler finished handling the revision and e.g. can reopen the reader. Callable callback = null; // can also be null if no callback is needed ReplicationHandler handler = new IndexReplicationHandler(indexDir, callback); SourceDirectoryFactory factory = new PerSessionDirectoryFactory(workDir); ReplicationClient client = new ReplicationClient(replicator, handler, factory);

// invoke client manually client.updateNow();

// or, periodically client.startUpdateThread(100); // check for update every 100 milliseconds

Classes

IndexAndTaxonomyReplicationHandler

A IReplicationHandler for replication of an index and taxonomy pair. See IReplicationHandler for more detail. This handler ensures that the search and taxonomy indexes are replicated in a consistent way.

IndexAndTaxonomyRevision

A IRevision of a single index and taxonomy index files which comprises the list of files from both indexes. This revision should be used whenever a pair of search and taxonomy indexes need to be replicated together to guarantee consistency of both on the replicating (client) side.

IndexAndTaxonomyRevision.SnapshotDirectoryTaxonomyWriter

A DirectoryTaxonomyWriter which sets the underlying IndexWriter's IndexDeletionPolicy to SnapshotDeletionPolicy.

IndexInputStream

A System.IO.Stream which wraps an IndexInput.

IndexReplicationHandler

A IReplicationHandler for replication of an index. Implements RevisionReady(String, IDictionary<String, IList<RevisionFile>>, IDictionary<String, IList<String>>, IDictionary<String, Directory>) by copying the files pointed by the client resolver to the index Directory and then touches the index with IndexWriter to make sure any unused files are deleted.

IndexRevision

A IRevision of a single index files which comprises the list of files that are part of the current IndexCommit. To ensure the files are not deleted by IndexWriter for as long as this revision stays alive (i.e. until Release(), the current commit point is snapshotted, using SnapshotDeletionPolicy (this means that the given writer's IndexDeletionPolicy should return SnapshotDeletionPolicy).

When this revision is Release()d, it releases the obtained snapshot as well as calls DeleteUnusedFiles() so that the snapshotted files are deleted (if they are no longer needed).

LocalReplicator

A IReplicator implementation for use by the side that publishes IRevisions, as well for clients to CheckForUpdate(String) check for updates}. When a client needs to be updated, it is returned a SessionToken through which it can ObtainFile(String, String, String) the files of that revision. As long as a revision is being replicated, this replicator guarantees that it will not be Release().

Replication sessions expire by default after , and the threshold can be configured through ExpirationThreshold.

PerSessionDirectoryFactory

A ISourceDirectoryFactory which returns FSDirectory under a dedicated session directory. When a session is over, the entire directory is deleted.

ReplicationClient

A client which monitors and obtains new revisions from a IReplicator. It can be used to either periodically check for updates by invoking StartUpdateThread(Int64, String), or manually by calling UpdateNow().

Whenever a new revision is available, the RequiredFiles(IDictionary<String, IList<RevisionFile>>) are copied to the Directory specified by PerSessionDirectoryFactory and a handler is notified.

RevisionFile

Describes a file in a IRevision. A file has a source, which allows a single revision to contain files from multiple sources (e.g. multiple indexes).

SessionExpiredException

Exception indicating that a revision update session was expired due to lack of activity.

SessionToken

Token for a replication session, for guaranteeing that source replicated files will be kept safe until the replication completes.

Interfaces

IReplicationHandler

Handler for revisions obtained by the client.

IReplicator

An interface for replicating files. Allows a producer to Publish(IRevision) IRevisions and consumers to CheckForUpdate(String). When a client needs to be updated, it is given a SessionToken through which it can ObtainFile(String, String, String) the files of that revision. After the client has finished obtaining all the files, it should Release(String) the given session, so that the files can be reclaimed if they are not needed anymore.

A client is always updated to the newest revision available. That is, if a client is on revision r1 and revisions r2 and r3 were published, then when the client will next check for update, it will receive r3.

IRevision

A revision comprises lists of files that come from different sources and need to be replicated together to e.g. guarantee that all resources are in sync. In most cases an application will replicate a single index, and so the revision will contain files from a single source. However, some applications may require to treat a collection of indexes as a single entity so that the files from all sources are replicated together, to guarantee consistency beween them. For example, an application which indexes facets will need to replicate both the search and taxonomy indexes together, to guarantee that they match at the client side.

ISourceDirectoryFactory

Resolves a session and source into a Directory to use for copying the session files to.