Release: 2.6.2 Previous Releases
Publish Date: July, 2008

rate-13500823-47099

Article Rating?

1

2

3

4

5

Gotchas

Introduction
Lock-Then-Share Gotcha
Uninstrumented Access Gotcha
Improper Locking Gotcha
Multiple Initialization Gotcha
Transients Gotcha
Mutable Enums Gotcha
java.lang.ref.Reference Gotcha
Iterator Gotcha
Collections Gotchas
Hidden references in anonymous and non-static inner classes Gotcha
Different Hashcode on Separate JVMs
IBM JDK
Debuggers
Appendix

Contacting Terracotta
Platform Support

Introduction

We strive to make Terracotta as transparent as possible and we are constantly improving our transparency and usability, but there are still some things which may seem unintuitive at first glance.

This document seeks to highlight some of these issues and solutions for them.

Lock-Then-Share Gotcha

Summary
Terracotta requires that locks are acquired on an object that is already shared. In some instances, your code may inadvertently lock a local object, then share it. The unlock will then throw an exception.

Example

public class LockThenShare
{
    private static Map myMap = new HashMap();  // tc root

    public void lockThenShare()
    {
        Object foo = new Object();
        synchronized (foo) { // object is not shared yet so only a local lock is acquired
            synchronized (myMap) {  // ok, myMap is a root
                myMap.put(foo);  // foo is now shared
            }
        } // oops, tried to unlock the foo object which is now shared, but a tc lock was never acquired
    }
}

Solution

Share then lock. For this example:

public class ShareThenLock
{
    private static Map myMap = new HashMap();  // tc root

    public void lockThenShare()
    {
        Object foo = new Object();
         synchronized (myMap) {  // ok, myMap is a root
                myMap.put(foo);  // foo is now shared
        }
        synchronized (foo) { // tc lock on foo is acquired
             ... foo operations
        } // ok, tc lock on foo is released
    }
}

Uninstrumented Access Gotcha

Reading and writing to objects shared in Terracotta DSO from uninstrumented methods can be problematic under certain conditions.

The first problem involves reading/writing accessible fields of a shared object from uninstrumented code. For example, consider two classes (A and B) in the same package. Class A has an accessible field (foo), and is instrumented. Class B is not instrumented.

public class A  // instrumented
{
   public boolean bool = false;
}

public class B // not instrumented
{
  public void foo(A a) 
  {
    a.bool = true;  //access to bool is not instrumented - trouble looming
  }
}

If class B is not instrumented, the reads and writes of the accessible "bool" field of class A will not function correctly for shared A instances from class B. The negative effects of an uninstrumented read usually provoke a NullPointerException, but could also manifest in stale values (and not just nulls).

A second, similar problem can occur with arrays when uninstrumented code reads/writes a shared array instance.

A third problem can occur when uninstrumented code interns an instrumented string. Terracotta DSO compresses instrumented strings whose size exceeds a certain configured value. Interning compressed strings from instrumented code is safe because that code is monitored and the strings are decompressed when called. However, if uninstrumented code interns a compressed string by calling java.lang.String.intern(), that string is not decompressed.

You can configure the limit for compressing strings, or disable compression altogether. However, disabling string compression may reduce the efficiency of your clustered application. See the tc.properties documentation for more information.

Solution 1
Take an OOP best-practices approach and encapsulate all access to external fields.

public class A  // instrumented
{
   private boolean bool = false;

   public void setBool(boolean bool) { this.bool = bool; }
}

public class B // not instrumented
{
  public void foo(A a) 
  {
    a.setBool(true); //  ok
  }
}

Solution 2

Instrument all involved classes.

public class A  // instrumented
{
   public boolean bool = false;
}

public class B // instrumented
{
  public void foo(A a) 
  {
    a.bool = true;  // ok
  }
}

Improper Locking Gotcha

Summary

Locking in Terracotta honors the Java 1.5 Memory Model. It is important to understand this model, because race conditions and/or improper locking in your application can affect its operation in a clustered context using Terracotta.

You must, however, keep in mind that these problems are not Terracotta specific. If your application has race conditions and/or improper locking, it is susceptible to data corruption or unexpected behavior at any time. Moving your application to a multi-cpu machine, or integrating it with Terracotta will likely expose these flaws more quickly than if left to execute in a single-cpu single-core box, however these environments only serve to expose these existing flaws more quickly.

An application that works well in a clustered environment is a better written application, even on a single-cpu single-core environment.

Example

A classic data-race can be exposed by omitting synchronization. Here is a classic data race condition:

public class RaceCondition
{
    private boolean on = false;

    public void doSomething()
    {
        if (on) {
            // do an on something
        } else {
           // do an off something
        }
    }

    public void setOn(boolean on)
    {
        this.on = on;
    }
}

If two threads execute doSomething() at the same time, while another thread is calling setOn(), then the state of the on field will be non-deterministic. Worse, in a multi-cpu machine, updates to on may never even be seen by another thread and could cause serious problems.

The common refrain that is repeated too often is that "primitive updates are atomic". While this is certainly true, the compiler (and Terracotta) do not have any "happens-before" semantic that they are required to honor, and therefore updates to the on field do not have to be propagated across threads (or JVMs in the case of Terracotta.

The second problem that is quite common is the solve only part of the problem, by locking one half of the execution, but leaving the other half unlocked. This may or may not be intentional, but has the same problem as the previous example. Without explicit locking that gives strict "happens-before" semantics, your program is at risk of having unintended side-effects.

public class WriteLockedButNotReadLocked
{
    private static Map myMap = new HashMap(); // tc root

    public void readAndUpdate(String key, String value)
    {
        MyObject myObject = myMap.get(key);  // read is not protected!!!
        myObject.update(value);
    }

    public void put(String key, MyObject object)
    {
        synchronized (myMap) {
            myMap.put(key, object);
        }
    }
}

As in the previous example, the compiler, there is no happens-before relationship established between the write to the map, and the read from the map. Other threads may never see the write, so the readAndUpdate method could very easily throw a NullPointerException.

Solution

Synchronization is cheap so do not pre-optimize. Omitting synchronization is a classic case of early optimization, and can lead to subtle and often difficult to resolve bugs. Write your program correctly, profile it, and optimize the parts that are actually running slow.

The correct way to write the "WriteLockedButNotReadLocked" class is the following:

public class WriteLocked
{
    private static Map myMap = new HashMap(); // tc root

    public void readAndUpdate(String key, String value)
    {
         synchronized (myMap) {
             MyObject myObject = myMap.get(key);  // read is now protected!!
             myObject.update(value);
         }
    }

    public void put(String key, MyObject object)
    {
        synchronized (myMap) {
            myMap.put(key, object);
        }
    }
}

If you find that the readAndUpdate method has contention then you can convert this example into a striped-lock:

public class WriteLockedAndReadLocked
{
    private static Map myMap = new HashMap(); // tc root

    public void readAndUpdate(String key, String value)
    {
        synchronized (myMap) {  // read lock alleviates contention on myMap
            MyObject myObject = myMap.get(key);  // read is now protected!!!
            update(myObject)
        }
    }
  
    public void update(MyObject myObject, String value)
    {
         // lock is striped, it would be rare for multiple updates to happen to the 
         // same myObject
          synchronized (myObject) {
               myObject.update(value);
          }
    }
    public void put(String key, MyObject object)
    {
        synchronized (myMap) {
            myMap.put(key, object);
        }
    }
}

Multiple Initialization Gotcha

Summary

The behavior of root initialization can be a bit unexpected, and can cause more than one root object to be constructed. Terracotta will ensure the root field is initialized once-and-only once across the cluster, however it cannot prevent a constructor from being called more than once.

Example

public class Root
{
    public static Root root = new Root();  // configured as tc root

    public Root()
    {
        System.out.println("Root constructor called.  I am:" + this);
    }

    public static void main(String[] args) 
    {
        System.out.println("Cluster root is: " + root);
    }
}

If you run this program once, you will see:

$ dso-java Root
Root constructor called.  I am @12345
Cluster root is: @12345

Run it again, you will now see:

$ dso-java Root
Root constructor called.  I am @56789
Cluster root is: @88123

Confusing? It's not that bad. Here's what happened.

The second time you ran this program, in a separate VM, the constructor was called per standard VM rules. And it's object id on the heap is @56789. However, this second time it runs, there is already a cluster wide instance for the root. When the assignment happened to the root field, Terracotta discarded the new instance and instead gave the field the instance that corresponds to the cluster instance.

So why is that instance id not the same as the first time through? Well even though Terracotta preserves Object Identity across the cluster, it does so by proxy. There is no way to make object ids the same, but it is possible to make the behavior identical. Terracotta does the latter, so the answer is that @88123 and @12345 are both the same logical instance.

Solution

This will always happen. You just should be aware that it will happen. Your constructors (for root objects) should not have side-effects.

Transients Gotcha

Summary

Transient mostly works exactly like it does in Serialization. When an object is transported to a second VM that does not have the object in memory, any transient fields will not be replicated. In the target VM, the transient field will be null.

However, in Terracotta, this process can happen at any time, due to Virtual Memory, and can be a bit unexpected.

Example

In a single VM, you wouldn't expect the following to happen:

public class TransientGotcha
{
    private transient Object object = new Object();

    public static TransientGotcha create()
    {
        TransientGotcha gotcha = new TransientGotcha();
         SharedMap.put("theObject", gotcha);  // assume SharedMap shares the object
         return gotcha;
    }
    
    public static TransientGotcha get()
    {
        TransientGotcha gotcha = SharedMap.get("theObject");
         gotcha.check();
         return gotcha;
    }

    public void check()
    {
        if (object == null) {
            System.out.println("Object ref is null!!");
        }
    }
    
    public static void main(String[] args)
    {
        create();
         .... // more code here
        get();  // this might in fact detect a null reference
    }
}

This can happen because Terracotta's Virtual Memory subsystem may have decided to evict the transientObject object. Then, after some time, the same VM requests the object again, and therefore it is faulted in. Since this happened at the discretion of Terracotta, and not the programmer, it can be expected.

Solution

This will always happen. Just be aware that it may happen even if you only have one VM. Make sure you test your shared objects being faulted in to a new VM. If the new VM throws a null pointer exception because a transient was not initialized, you can fix that using the "onLoad" configuration setting explained in the Concept and Architecture Guide:

Concept and Architecture Guide - Transients in Terracotta

Mutable Enums Gotcha

TODO: Check - this may not be a gotcha as of 2.5 or 2.6. Describe that changes to enumerations aren't clustered (DSO behavior mimics their out-of-the-box serialization/deserialization behavior)

java.lang.ref.Reference Gotcha

If user code has a java.lang.ref.Reference to a shared object Foo, Foo can be paged out of memory by the DSO reaper. The reference will be nulled out.

This behavior may be surprising to some user code. The expectation is that the reference value won't become null until the object has entered some phase of the garbage collection cycle.

See http://jira.terracotta.org/jira/browse/CDV-330 for more information.

Iterator Gotcha

Summary
Explicit sharing of iterator instances is not supported. Iterators created from shared collections are not problematic, but again, these iterator instances themselves cannot be shared objects (only the underlying collection)

Solution
Don't share iterators. Share the collection, then iterate it from within a local context.

TODO: Check. Iterators may be shareable in Terracotta 2.5 or 2.6.

Collections Gotchas

Standard Java provides a number of collections (HashTable, SynchronizedMap etc.) that are thread-safe by default. For performance reasons, these data structures are not auto-locked by default in Terracotta, meaning they will not work as expected if they are part of a clustered graph.

The following Collections are not auto-locked in Terracotta:

HashTable
SynchronizedCollection
SynchronizedMap
SynchronizedSet
SynchronizedSortedMap
SynchronizedSortedSet
Vector

java.util.concurrent.ConcurrentHashMap is auto-locked by default

Solution

The Terracotta Forge project tim-collections provides pre-configured modules to enable auto-locking for these Collections. More details can be found on the Terracotta Forge:

Forge TIM Collections

Hidden references in anonymous and non-static inner classes Gotcha

Summary

The java compiler generates a synthetic reference back to the enclosing instance for anonymous and non-static inner classes.

This can unexpectedly lead to sharing of instances that were not intended to be shared.

Example

Consider this code:

class A 
{
    public void bar()
    {
        // does something here
    }

    public Runnable foo() 
    {
          return new Runnable()  {
              public void run() {  }
          };
   }

   private class Inner 
   {
         //  ...

         public void doSomething()
         {
             bar();
         }
   }
}

In both the non-static inner class named "Inner" and the anonymous inner class in the "foo" method, the compiler introduced as synthetic field named "this$0" containing a reference to the enclosing A instance.

This hidden reference becomes significant in Terracotta when trying to share inner class instances since the enclosing type must be portable, and the enclosing instance will become shared as well.

Solution

This will always happen. Consider using static inner classes, to prevent unintended side-effects. Using an explicit pointer for Inner classes can give you the ability to mark the pointer transient, in effect preventing the enclosing instance to not be shared.

class A 
{
    public void bar()
    {
        // does something here
    }

   private static class Inner 
   {
         private transient final A a;

         public Inner(A a) 
         {
              this.a = a;
         }

         public void doSomething()
         {
             a.bar();
         }
   }
}

Different Hashcode on Separate JVMs

Summary

For primitive types, if an Object is present on two separate JVMs, then it should return the same hashcode on both the JVMs. But the hashcode of a custom object depends on the Object.hashCode() method and could have different hashcodes on separate JVMs.

The inconsistency occurs when two key-value pairs with the same key are added in a Map by two different JVMs. This leads to different hashcodes for the same key, and possibly results in a clustered Map with two entries for the same key-value pair.

Cause

Object.hashCode() is a native call and uses the physical memory location of the object to generate a hashcode. On two separate JVMs, it may return different values and cause inconsistencies.

Solution

Override the Object's hashCode() method in custom objects to ensure that the hashcode generation for the custom object is through the overridden hashcode method, not through Object.hashcode(). The overridden method should always return a consistent and unique hashcode for the same Object across different JVMs and between JVM restarts.

Terracotta DSO does not manipulate the Object.hashCode() method in any way.

In the following example, overriding the hashCode() method ensures that no inconsistency exists in the hashcode generated for the same Object of type A in two different JVMs. The Class A uses String.hashCode() to compute the hashcode for Objects of A and not the Object.hashCode() method. This ensures that the hashcode for objects of class A is dependent on the int arithmetic done by String.hashCode() method and not based on the physical memory location of the object.

Class A {
    String str = "HashCodeTest";
    public int hashCode() {
        return (str != null ? str.hashCode() : 0);
    }
}

IBM JDK

There are some known issues with the IBM JDK. See integrations:IBM JDK for details.

Debuggers

Using the debugger, one can observe field values of shared objects to have phantom null values (or null element values in a array). This is a side effect of the just-in-time lazy loading and dynamic memory management features in DSO. Although a field can appear null in the debugger view, your application will see the correct values when/if the fields are read. The main issue here is that the debugger uses non-bytecode means of viewing object state. Stepping over some code that reads the phantom null field will cause the value to be resolved. Although it might be obvious, using the debugger to mutate a field of shared object will bypass Terracotta leaving your locals in an inconsistent state with the cluster.

Appendix

Contacting Terracotta

Contact Terracotta at the following:

Web site: http://www.terracotta.org
Online forums: http://forums.terracottatech.com/forums/
Information: info@terracottatech.com

Platform Support

See Platform Support for information on which platforms are supported by Terracotta.
See the Integrations space to see the status of integrations with third-party technologies.

Copyright Information

This publication (the "Documentation") and the Terracotta software which it describes (the "Software") are protected to the maximum extent permitted under applicable law, including but not limited to, the regulations set forth in Title 17 of the United States Code, and California law. This Documentation, or any parts thereof, may not be reproduced in any form, by any method, for any purpose, without the express written consent of Terracotta. Terracotta makes no warranty, either express or implied, including but not limited to any implied warranties of merchantability or fitness for a particular purpose, with respect to the Software discussed in this Documentation, and the Documentation itself (collectively, "the Materials"). The Materials are made available solely on an "as-is" basis. In no event shall Terracotta be liable to anyone for special, collateral, incidental, indirect, punitive, exemplary, or consequential damages in connection with, or arising from the purchase or use of, the Materials. Under no circumstances and regardless of the cause of action alleged, shall Terracotta's liability exceed the purchase price of the Software described herein. Terracotta reserves the right to revise and improve its Software and Documentation as it deems fit. The Documentation describes the state of the Software at the time of publication.

Trademarks
"Terracotta," the stylized "T" logo, and "Open Terracotta" are trademarks of Terracotta. All other brand names, product names, or trademarks belong to their respective holders. Java and all Java-based trademarks and logos are trademarks or registered trademarks of Sun Microsystems, Inc. in the United States and other countries. All other brand names, product names, or trademarks belong to their respective holders.

Government Use
Use, duplication, or disclosure by the U.S. Government is subject to restrictions as set forth in FAR 12.212 (Commercial Computer Software-Restricted Rights) and DFAR 267.7202 (Rights in Technical Data and Computer Software), as applicable.