Testing Distributed Software
Terracotta does a lot of distributed software testing. This is a start at collecting our knowledge about testing together into one place.
Tools We Use
Tools Other People Use
JProfiler
Anders Bengtsson reports that JProfiler
now works with Terracotta
Here's the related JIRA issue: https://jira.terracotta.org/jira/browse/CDV-283
Some Advice from Iyer
nmon
We suggest running nmon (it will give you a window in memory, cpu, disk,
network i/o consumption). E.g. /usr/bin/nmon -ft -s 5 -c 280 )
You can read more about it at:
http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/
You can download it at:
http://www-941.haw.ibm.com/collaboration/wiki/display/WikiPtype/nmon
You can get the analyzer (which draws pretty reports in Excel) at:
http://www.ibm.com/developerworks/aix/library/au-nmon_analyser/
jstat
Also run jstat to measure GC activity on the Terracotta server JVM and any
of the Terracotta clients.
e.g. jstat -gcutil <java-pid> 5s
vmstat
Vmstat also useful at times (esp on Terracotta server).
Terracotta Admin Console
Admin Console to measure how many Terracotta Transactions to an
application transaction.
Some Advice from Kunal
The top things to look out for:
L1 := Clustered JVM (Terracotta Client)
L2 := Terracotta Server
1. Monitor the CPU and the Heap to make sure you are not CPU or memory bound
(on the L1 and the L2). It is also useful to keep an eye on the network.
Nmon and Jstat/vmstat will do all of this for you.
2. Look at lock contention in your code. Wherever possible, try to batch
things together. The level of granularity of locks (coarse grained or fine
grained) depends on the data access patterns.
3. Try to correlate the number of Terracotta transactions (in the admin
console) with the number of application transactions. This will give you
hints about how coarse grained or fine grained your transactions should be.
E.g. If the Admin Console is showing 3000 Terracotta TPS and your
application is doing 20 TPS, it roughly means that in each Application
transaction you are doing 150 Terracotta transactions and that your locks
are too fine grained.
4. Try to avoid cross-referencing objects as much as you can. In other
words, maintain locality of reference (e.g. sticky load balancing) as much
as possible.
80% of Terracotta performance tuning boils down to memory (and GC) tuning
and lock tuning.
As far as testing goes, you should be able to easily integrate Terracotta in
your existing scripts. It is always better to simulate the real production
environment as much as possible.
E.g. Having the L1 and the L2 on the same box can peg the CPU or the Memory
or both. So if you plan to do so, you provision the hardware accordingly.
