Monday, February 8, 2010

Concurrent Modification Exception

I ran into a ConcurrentModificationException (CME) during stress testing.
What does CME actually mean?
It means that you've modified (add, remove, update) your Collection while you've been iterating over it (usually in a multi-threaded fashion, but it can occur in a single thread that modifies while iterating).

A few more things to note about CME:
Best effort detection
- If you see a CME printout, first off, consider yourself lucky, CMEs are thrown only in best effort. In another universe, the concurrent modification would not have been detected, causing your collection to become corrupted, instead of fast-failing with a CME.

IDing the problem - Like deadlocks, CME's are easy to pinpoint once you inspected the exception's stack trace.

Avoiding CME:

  1. ListIterator
    To modify a collection by the same thread that is currently iterating on it, use a ListIterator that will allow you to perform both.
    Drawbacks - single thread solution only.

  2. Naive solution: Synchronizers
    Use locks to for mutually excluding traversal and modification operations.
    Advantages - easy to code.
    Drawbacks - very long lock periods while iterating.

  3. CopyOnWrite
    Take advantage of the Java.util.concurrent collections like: CopyOnWriteArrayList, CopyOnWriteArraySet. If you require a map then grab CopyOnWriteMap from Apache (this guys have been doing Sun's dirty work for years now).
    Advantages - very good reading performance (no locks are used, instead visibility is obtained via map member volatility).
    Drawbacks - very bad write performance on large maps.
    Conclusion - use for seldom mutating collections.

  4. toArray()
    toArray will create a new array holding a copy of your Set (Map.keySet() for a Map).
    You can then iterate over the array, freely modifying the original collection (the array doesn't change of course).
    Advantages - write operations are cheap.
    Disadvantages - copying the entire set could be expensive if it occurs too often, and/or the set is very large.

  5. Concurrent Collections
    If you want to go heavyweight, consider using: ConcurrentHashMap (or one of its package friends).
    Once you create an iterator over a ConcurrentHashMap (CHM), it does not freeze the collection for traversal, updates to the collection may or may not appear during the traversal (weakly consistent).

The approach I ended up taking
My use case was seldom modifying a ~ten items cache. A copyonwrite map was what I used.
In other cases I had, ConcurrentHashMap was the easiest solution (though make sure your code can live in peace with the CHM weak consistency property).

[caption id="attachment_227" align="alignleft" width="200" caption="Best pic idea I could think of to visualize Threads :)"][/caption]

Monday, February 1, 2010

NAT in VMWare vSphere/ESX – In a nut shell

This post is about NATing an ESX VM, but first, why do I need NAT:

The SIP protocol is not NAT oblivious. To traverse NAT our application has to replace the DNS in the SIP message contact header to the external FQDN that the message receiver will be sending responses to (A NAT with static routing configured).
Therefore I needed to test our software in a NAT topology.

In the past, when we used VMWare player/workstation, it had a build-in NAT network. But, unfortunately, the ESX hypervisor does not provide a NATed network option.
Seeking alternatives at VMWare's appliance marketplace, I found and downloaded the Vyatta's community edition (VC5) router appliance (also downladble from sourceforge), and comes under the GPL license.
After 3-4 hours - guided by the official quick start guide -  I had a working NAT configuration in the ESX. Hurray!
Overall, not a hard nut to crack ;), though I wish VMWare will wise up and just add an build-in NAT option to vSphere.

Left to do:
Obtain some static IPs, so the config won't break each time the vm reboots and the DHCP lease expires.
Tip #1:
If you want want to access your NATed VM by RDP/VNC, without setting up extra NAT routing rules, consider adding the VM an additional un-NATed NIC, but when doing so, make sure that the OS routing tables are set to route through the NIC that is NATed.
Tip #2:
This short vyatta user installation report also helped me a bit.

Here's the complete configuration script I ended up feeding to the appliance console (network topology is similar to the one presented in the Vyatta's getting stated guide):
Where: is your department's DNS server is the VMs NATed private IP address (provided by the DHCP).
The script contains a NAT forward rule for VNC (port 5900)

set system host-name vyatta-nat
set interfaces ethernet eth0 address dhcp
set service ssh
set service https
# restart the appliance to switch from console remote desktop to SSH:

#login with user and password
show interfaces

set interfaces ethernet eth1 address


delete service dhcp-server
set service dhcp-server shared-network-name ETH1_POOL subnet start stop
set service dhcp-server shared-network-name ETH1_POOL subnet default-router
set service dhcp-server shared-network-name ETH1_POOL subnet dns-server
show service dhcp-server

set service nat rule 1 source address
set service nat rule 1 outbound-interface eth0
set service nat rule 1 type masquerade
show service nat
show nat rules
set service nat rule 20 type destination
set service nat rule 20 inbound-interface eth0
# use a negative fake address to so that all incoming communication will be nated
#set service nat rule 20 destination address !
#Forward traffic to address
set service nat rule 20 inside-address address
set service nat rule 20 protocol tcp
set service nat rule 20 destination port 5900