BigMemory WAN Replication Setup

Requirements

The following components are required:

• BigMemory Max 4.1 or higher
• A terracotta-license.key file
• JDK 1.6 or higher
• At least one Terracotta Server Array (TSA) in each region
• Reliable WAN links between all target regions

Get Started

The following steps provide an overview of setting up BigMemory WAN Replication:

1. Ensure that the Terracotta clusters that will use WAN replication can run as expected without WAN replication.

   See the installation instructions for BigMemory Max and Terracotta Server Array for more information on installing a cluster.

   After verifying the Terracotta clusters, shut down all processes in order to set up for WAN replication.

2. Enable WAN replication for all caches that will be replicated across your WAN.

   For each cache to be replicated, its ehcache.xml configuration file must include the wanEnabledTSA attribute set to "true" within the <terracottaConfig> element:

   <terracottaConfig wanEnabledTSA="true"/>
   

   Note: Caches that will share data must have the same name.

3. Configure at least one Orchestrator for each region.

   Each Orchestrator requires a dedicated wan-config.xml configuration file. The wan-config.xml specifies the caches that will be Masters and failover Masters by listing the locations of the Master caches. All of the Masters listed for a cache must be in the same region. (Replica caches do not need to be specified in the Orchestrator configuration file, as they will register with their Master caches upon startup.)

   Following is a sample wan-config.xml file:

   <?xml version="1.0" encoding="UTF-8"?>
   <wan-config xmlns="http://config.wan.terracottatech.com">
   
     <bind host="0.0.0.0" port="9003"/>
     <logs>/path/to/mylogs</logs>
   
     <maxConnectionRetryCount>8</maxConnectionRetryCount>
     <replicatorIntervalMillis>125</replicatorIntervalMillis>
     <replicationMode>unidirectional</replicationMode>
     <replicaDisconnectBehavior>reconnectResync</replicaDisconnectBehavior>
   
     <cacheManager ehcacheConfigURI="file:///path/to/ehcache-1.xml">
       <cache name="test-cache-1">
         <master host="masterhost-A" port="9001"/>
         <master host="masterhost-B" port="9002"/>
       </cache>
       <cache name="test-cache-2">
         <master host="masterhost-A" port="9001"/>
       </cache>
     </cacheManager>
   
     <cacheManager ehcacheConfigURI="file:///path/to/ehcache-2.xml">
       <cache name="test-cache-3">
         <master host="masterhost-B" port="9002"/>
       </cache>
     </cacheManager>       
   
     // Optional for version 4.1.3 and above--see step 5 below
     <userClassDirectory>/temp/user_lib</userClassDirectory>
   
   </wan-config>
   

   Note: The ehcache.xml referenced in an Orchestrator's wan-config.xml must include the location of the tc-config.xml for that Orchestrator's region. The location is specified in the terracottaConfig element of the ehcache.xml. For more information, refer to Distributed BigMemory Max Configuration.

   For more information about the wan-config.xml parameters, see the Orchestrator Configuration Parameters section below.

4. Start the Terracotta Server Array in each region.

5. (Optional) If you have any non-JDK value types in your caches, and/or if you are using custom attribute extractors for Search, you will need to add these to the classpath of the Orchestrator process.

   a. Place any jars containing custom attribute extractors or custom user classes in a directory.

   b. For versions 4.1.0 through 4.1.2 only: Define the environment variable WAN_USER_LIB to the absolute path of the directory. For example:

   export WAN_USER_LIB=/temp/user_lib
   

   The WAN shell script will then add all .JAR files under the /temp/user_lib directory to the Orchestrator's classpath.

   For version 4.1.3 and above only: Starting with 4.1.3, the WAN_USER_LIB has been deprecated. The functionality previously provided by WAN_USER_LIB is now handled by the userClassDirectory tag in each Orchestrator's wan-config.xml file. Following is a sample userClassDirectory tag in a wan-config.xml file:

   <userClassDirectory>/temp/user_lib</userClassDirectory>
   

   The WAN shell script will then add all .JAR files under the /temp/user_lib directory.

6. Use the start-wan script to start the Orchestrators.

   To specify the configuration file location, use -f command-line option, for example:

   <bigmemory_kit>/server/bin/start-wan.sh -f /path/to/wan-config.xml
   

7. Start your application. You must start the instances with the Master caches first, and then start those with Replica caches.

   Upon starting the app servers, your clusters should begin replicating data across the WAN.

Orchestrator Configuration Parameters

The following parameters of the wan-config.xml configure locations for the Orchestrator instance, its logs, and its Master cache locations.

• The bind settings should include the URL and port of the Orchestrator instance.

• The logs parameter takes the path to the location where you want the Orchestrator's logs to be collected.

• Each cache to be replicated must have at least one Master instance identified in the wan-config.xml.

  • The ehcacheConfigURI should provide the path to the ehcache.xml configuration file for the cache.

  • Each master parameter should provide the host name and port for the instance of the Master cache.

The defaults of the parameters in the wan-config.xml are optimized for most environments, but the following parameters can be adjusted to tune operation of your BigMemory WAN deployment.

• maxConnectionRetryCount — Specifies the maximum number of connection attempts a disconnected Replica cache should try for each Master cache, before attempting to connect to another Master cache in the wan-config.xml. The default is 5. Note that this applies only when the replicaDisconnect behavior is set for reconnectResync (see below).

• replicatorIntervalMillis — Specifies how frequently the Orchestrator should send replication updates to the regions, in milliseconds. Depending upon the amount of RAM available to your Orchestrator, you may be able to optimize WAN replication by increasing the interval.

• replicationMode — Selects the type of replication, unidirectional or bidirectional. If unspecified, the default is unidirectional. Bidirectional mode must be specified for active-active deployments. For more information, refer to Replication Modes and Bidirectional WAN Replication.

• replicaDisconnectBehavior — Specifies the behavior of the Replica caches if they become disconnected from their Master cache. The options are:

  • reconnectResync (default) — When a Replica is disconnected from its Master, it attempts to reconnect to the Master and to any failover Masters listed in the wan-config.xml. Upon reconnection to a Master cache, the Replica is deactivated, cleared, resynchronized to the Master cache, and then reactivated. All local changes on the Replica region will be dropped in favor of whatever is in the Master region. Note: When a WAN-enabled cache is deactivated, it follows the nonstop behavior specified in the ehcache.xml. For more information, refer to Nonstop Operation.

  • remainDisconnected — When a Replica is disconnected from its Master, it remains isolated from the Master from this point on. The Replica will not attempt to connect to the failover Masters listed in the wan-config.xml. The disconnected Replica cache does remain active locally, however, but no replication will take place.

• monitoringEnabled — Specifies that the monitoring capability for the WAN Replication Service is enabled for each Orchestrator. If you want to disable the monitoring capability, set the monitoringEnabled parameter to false in each Orchestrator's wan-config.xml file.