/ Java EE Support Patterns


DZone's Guide to Building and Deploying Applications on the Cloud

This post is to inform you that DZone has just released a great guide regarding Building and Deploying Applications on the Cloud. I recommend that you download your copy today!

Here is a snippet:

Overhyped or not, the cloud has deeply changed how we build and run software—and not just because IaaSes make VMs trivial to spin up and PaaSes make environments easy to set up. As a user you know what’s changed, and you understand the concept “as a service” (well, ever since you started running *nix); and, thank goodness, you don’t really have to worry about the physical details that make those services run. 


Java 8 - CPU Flame Graph

Brendan Gregg and Martin Spier from Netflix recently shared a very interesting article titled Java in Flames, describing their latest experimentation with a new JDK option (-XX:+PreserveFramePointer ) that allowed them to create a full CPU consumers view as a "flame" graph. This article is an advanced read but extremely interesting for Java Performance enthusiasts.

This option is now included in the recently released JDK 8u60.

We will create our own experiment shortly and post a video exploring this CPU profiling capability real-time vs. existing CPU profiling tools & techniques. As mentioned in the article, a clear added-value would be to automate and visualize CPU utilization delta (deviation from an established baseline) between releases or code changes. 

This approach would allow fast detection of CPU bottleneck or improvements following software changes, improving the overall performance and scalability of the production environment over the long run, as well as keeping the cloud or on-premise hardware cost under control.

Here is a small snippet from the original article:

"Java mixed-mode flame graphs provide a complete visualization of CPU usage and have just been made possible by a new JDK option: -XX:+PreserveFramePointer. We've been developing these at Netflix for everyday Java performance analysis as they can identify all CPU consumers and issues, including those that are hidden from other profilers..."


JVM Buzzwords Java developers should understand

This article will share with you a few JVM "buzzwords" that are important for Java developers to understand and remember before performing any JVM performance and garbage collection tuning. A few tips are also provided including some high level performance tuning best practices at the end of the article. Further recommendations regarding the Oracle HotSpot concurrent GC collectors such as CMS and G1 will be explored in future articles.

Before reading any further, I recommend that you first get familiar with the JVM verbose GC logs. Acquiring this JVM data analysis skill is essential, especially when combined with more advanced APM technologies.

JVM Buzzwords

Allocation Rate
Java objects allocated to the YoungGen space,
a.k.a. “short-lived’ objects.
Promotion Rate
Java objects promoted from the YoungGen to the OldGen space.
Java objects sitting in the OldGen space, a.k.a. “long-lived’ objects.
Stop-the-world Collection
Garbage collections such as Full GC and causing a temporary suspension of your application threads until completed.

First Things First: JVM GC Logs
  • Provides out-of-the-box fine-grained details on the Java heap and GC activity.
  • Use tools such as GCMV (GC Memory Visualizer) in order to assess your JVM pause time and memory allocation rate vs. sizing the generations by hand.

Allocation & Promotion Rates
  • It is important to keep track of your application allocation and promotion rates for optimal GC performance.
  • Keep the GCAdaptiveSizePolicy active, as part of the JVM ergonomics. Tune by hand only if required.

LIVE Data Calculation
  • Your live application data corresponds to the OldGen occupancy after a Full GC.
  • It is essential that your OldGen capacity is big enough to hold your live data comfortably and to limit the frequency of major collections and impact on your application load throughput.
Recommendation: as a starting point, tune your Java Heap size in order to achieve an OldGen footprint or occupancy after Full GC of about 50%, allowing a sufficient buffer for certain higher load scenarios (fail-over, spikes, busy business periods...).

  • *Hot Spot*: watch for OldGen memory leaks!
  • What is a memory leak in Java? Constant increase of the LIVE data over time...

LIVE Data Deep Dive
  • JVM GC logs are great…but how you can inspect your live data?
  • Java Heap Histogram snapshots and JVM Heap Dump analysis are powerful and proven approaches to better understand your application live data.
  • Java profiler solutions and tools such as Oracle Java Mission Control , Java Visual VM provide advanced features for deep Java heap inspection and profiling, including tracking of your application memory allocations.

Stop-the-world Collections: GC Overhead
  • YoungGen collections are less expensive but be careful with excessive allocation rate.
  • It is recommended to initially size (JVM default) the YoungGen at 1/3 of the heap size.
  • Remember: both YoungGen and OldGen collections are stop-the-world events!
  • PermGen and Metaspace (JDK 1.8+) are collected during a Full GC, thus it is important to keep track of the Class meta data footprint and GC frequency.

Final Words & Recommendations

Best Practices
  • Optimal Java Performance is not just about Java…explore all angles.
  • Always rely on facts instead of guesswork.
  • Focus on global tuning items first vs. premature fine-grained optimizations.
  • Perform Performance & Load Testing when applicable.
  • Take advantage of proven tools and troubleshooting techniques available.
To Avoid
  • There are dozens of possible JVM parameters: don’t over-tune your JVM!
  • You always fear what you don’t understand: good application knowledge > no fear  > better tuning recommendations.
  • Never assume that your application performance is optimal.
  • Don’t try to fix all problems at once, implement tuning incrementally.
  • Don’t get confused and keep focus on the root cause of performance problems as opposed to the symptoms.
  • Excessive trial and error approach: symptom of guesswork.


SSL SHA-2 and Oracle WebLogic

This post is to inform you that I will be releasing an article shortly on the industry adoption of SHA-2 SSL certificates and potential impact to your Java EE production environments. It will be especially useful if your secured application is still using an older version of Oracle WebLogic, packaged with the deprecated Certicom-based SSL implementation which does not support SHA-2 (SHA-256 signature algorithm).

In the meantime, I recommend that you consult the high level SHA-2 migration guide from Entrust. It is a very good starting-point and will help increase your awareness level on this upcoming SHA-1 to SHA-2 upgrade.


Java Application Scalability

Eric Smith from AppDynamics recently released a great article on application scalability.

Essentially the main point is that the ability or effectiveness of scaling vertically/horizontally your application depend on various factors, more complex than just looking at the OS CPU and memory utilization.

Proper usage of the right tools and capture of application specific metrics are crucial in order to identify tuning opportunities. This approach will also help you determine the right initial and incremental infrastructure/middleware sizing for your on-premise or in the cloud production environment, reducing your client hardware/hosting long-term cost and improve the ROI.

For example, if you we look at your Java application LIVE data (OldGen footprint after a major collection). Some applications have LIVE data which depend mainly on the concurrent load and/or active users e.g. session footprint and other long-lives cached objects. These applications will benefit well from vertical or horizontal scaling as load is split across more JVM processes and/or physical VM's, reducing pressure point on the JVM fundamentals such as the garbage collection process.

On the contrary, Java applications dealing with large LIVE data footprint due to excessive caching, memory leaks etc. will poorly scale since this memory footprint is "cloned" entirely or partially over the new JVM processes or physical VM's. These applications will benefit significantly from an application and JVM optimization project which can both improve the performance and scalability, thus reducing the need to "over-scale" your environment in long-term.


DevOps and Continuous Delivery - Weekly articles

I would like to inform my fellow readers that I am currently preparing a cluster of fresh articles on Java Performance following intense troubleshooting and performance tuning work over the past 12 months.

In the meantime, I recommend the following list of fresh DevOps related articles from Electric Cloud which offer different perspectives on this practice.

Please stay tune for more updates...



DevOps - 2015 Guide to Continuous Delivery from DZone

I would like to share with my fellow readers that DZone has published a great 2015 guide about Continuous Delivery, which is a core principle and goal of the DevOps methodology.

If you are part of an organization about to implement DevOps principles, emerging tools or simply wish to improve your knowledge and awareness on Continuous Delivery, I highly recommend that you download your own copy today, it is FREE!

Thank you.



Java Performance Optimization - DZone Refcard

This post is to inform you that I published recently a recent refcard on Java Performance Optimization which is now available from DZone. Refcardz are basically “Cheat Sheets” which can be very useful for Developers and Application Support individuals.

I highly recommend that you download your own copy today, it is FREE!

For now, find below a small teaser:

It is possible to define “optimal performance” in different ways, but the basic elements are: the ability of a Java program to perform its computing tasks within the business response time requirements, and the ability of an application to fulfill its business functions under high volume, in a timely manner, with high reliability and low latency. Sometimes the numbers themselves become patternized: for some major websites, a page response time of 500ms maximum per user function is considered optimal. This Refcard will include target numbers when appropriate, but in most cases you will need to decide these on your own, based on business requirements and existing performance benchmarks…”

Please feel free to comment and ask me any question on this subject.

Thank you.


Java 8 is now released!

This post is to inform you that Java 8 has been officially released by Oracle. You can download the official release here.

In a nutshell:

  • Lambda expressions: a new language feature  that enables you to treat functionality as a method argument, or code as data.
  • Other significant enhancements and changes to the Java language and standard libraries including default methods, the new java.util.stream package, and the Date-Time API.
  • Compact Profiles contain predefined subsets of the Java SE platform and enable applications that do not require the entire Platform to be deployed and run on small devices.
  • Security enhancements include updates to the Java Cryptography Architecture; limited doPrivileged, a mechanism that enables code to assert a subset of its privileges; SSL/TLS Server Name Indication (SNI) Extension; and keystore enhancements.
  • JavaFX documentation has been updated for this release.
  • A new JavaScript engine, Nashorn, is included in JDK 8.
  • Java Mission Control 5.3 is included in JDK 8.
  • Decommission of the JVM Permanent Generation space and replaced by the Metaspace.

I will publish more articles in the upcoming weeks, including tutorials on Java Mission Control 5.3 which is now part of the JDK 8.

In the meantime, I suggest that you revisit my original article on the replacement of the HotSpot PermGen space by the Metaspace. I will publish the part 2 in the upcoming weeks.


ClassNotFoundException: is it slowing down your JVM?

Most Java developers are familiar with the infamous and very common java.lang.ClassNotFoundException. While the sources of this problem are generally well understood (missing classes/libraries in classpath, class loader delegation problems etc.), the impact to the overall JVM and performance is often unknown. This situation can lead to significant impacts to your application response time and scalability.

Large Java EE enterprise systems with multiple applications deployed are the most exposed to this type of problem due to the large number of different application classloaders active at runtime. This increases the risk of facing “undetected” ClassNotFoundException unless a clear business impact is identified and close log monitoring is implemented, results: on-going performance impact along with possible JVM class loading IO and Thread lock contention.

The following article and sample programs will demonstrate that any occurrence of ClassNotFoundException found from your client production systems should be taken very seriously and addressed promptly.

Java class loading: the missing link for optimal performance

Proper understanding of this performance problem starts with proper knowledge of the Java class loading model. ClassNotFoundException essentially means a failure of the JVM to locate and/or load a particular Java class such as:

  • The Class.forName() method
  • The ClassLoader.findSystemClass() method
  • The ClassLoader.loadClass() method
While class loading of your application Java classes should occur only once in the JVM life-cycle (or through dynamic re-deploy functions), some applications also rely on dynamic class loading operations.

Regardless, repetitive valid and “failing” class loading operations can be quite intrusive, especially when the loading process is attempted by the default JDK java.lang.ClassLoader itself. Indeed, the default JDK 1.7+ behaviour, due to backward compatibility, will allow only one class to be loaded at the time, unless the class loader is tagged as “parallel capable”. Please keep in mind the even if the synchronization is done at the Class level only, a repetitive class loading failure against the same class name is still going to trigger thread lock contention depending of the Java thread concurrency level you are dealing with. The situation was much worst back with JDK 1.6 with synchronization done systematically at the class loader instance level.

For this reason, Java EE containers such as JBoss WildFly 8  are using their own internal concurrent class loaders for the loading of your application classes. These class loaders implements locking at a finer level of granularity, thus allowing loading different classes from the same instance of the class loader concurrently. This is also aligned with the latest JDK 1.7+ improvement introducing the support for Multithreaded Custom Class Loaders which also help prevent certain class loader deadlock scenarios.

That being said, class loading of system level classes such as java.* and Java EE container modules still fall back on the default JDK ClassLoader. This means that repetitive class loading failure for the same class name e.g. ClassNotFoundException can still trigger severe thread lock contention. This is precisely what we will replicate and demonstrate for the rest of the article.

Thread lock contention - problem replication

In order to recreate and simulate this problem, we created a simple application as per below specifications:

  • A JAX-RS (REST) Web Service executing Class.forName() against a dummy Class name “located” from the system package level:
             String className = "java.lang.WrongClassName";
The simulation essentially executes concurrently with 20 threads the JAX-RS Web service. Each invocation generates a ClassNotFoundException. The logging was fully disabled in order to reduce impact on IO and focus on the class loading contention only.

 Now let’s look at the results from JVisualVM from our 30-60 seconds run. We can clearly see a lot of BLOCKED threads waiting to acquire a lock on an Object monitor.

Analysis of the JVM thread dump is clearly exposing the problem: Thread lock contention. We can see from the execution stack trace that JBoss delegates the loading of the class to the JDK ClassLoader…why? This is because our wrong Java class name is detected to be part of the system class path e.g. java.*. In this situation, JBoss will delegate the loading to the system class loader, triggering systematic synchronization for that particular class name and waiters from the other threads waiting to acquire a lock to load the same Class name.

Many threads waiting to acquire LOCK 0x00000000ab84c0c8…

"default task-15" prio=6 tid=0x0000000014849800 nid=0x2050 waiting for monitor entry [0x000000001009d000]              
   java.lang.Thread.State: BLOCKED (on object monitor)                                                                  
            at java.lang.ClassLoader.loadClass(ClassLoader.java:403)                                                             
               - waiting to lock <0x00000000ab84c0c8> (a java.lang.Object)
  // Waiting to acquire a LOCK held by Thread “default task-20”                                                     
               at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)                                                     
               at java.lang.ClassLoader.loadClass(ClassLoader.java:356)    // JBoss now delegates to system ClassLoader..                                                          
               at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:371)                          
               at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119)                                 
               at java.lang.Class.forName0(Native Method)                                                                           
               at java.lang.Class.forName(Class.java:186)                                                                           
               at org.jboss.tools.examples.rest.MemberResourceRESTService.SystemCLFailure(MemberResourceRESTService.java:176)       
               at org.jboss.tools.examples.rest.MemberResourceRESTService$Proxy$_$$_WeldClientProxy.SystemCLFailure(Unknown Source) 
               at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)                                                       
               at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)                             
               at java.lang.reflect.Method.invoke(Method.java:601)    

Culprit thread - default task-20

"default task-20" prio=6 tid=0x000000000e3a3000 nid=0x21d8 runnable [0x0000000010e7d000]                             
   java.lang.Thread.State: RUNNABLE                                                                                   
               at java.lang.Throwable.fillInStackTrace(Native Method)                                                             
               at java.lang.Throwable.fillInStackTrace(Throwable.java:782)                                                        
               - locked <0x00000000a09585c8> (a java.lang.ClassNotFoundException)                                                 
               at java.lang.Throwable.<init>(Throwable.java:287)                                                                  
               at java.lang.Exception.<init>(Exception.java:84)                                                                   
               at java.lang.ReflectiveOperationException.<init>(ReflectiveOperationException.java:75)                             
at java.lang.ClassNotFoundException.<init>(ClassNotFoundException.java:82) // ClassNotFoundException!                                      at java.net.URLClassLoader$1.run(URLClassLoader.java:366)                                                          
               at java.net.URLClassLoader$1.run(URLClassLoader.java:355)                                                          
               at java.security.AccessController.doPrivileged(Native Method)                                                      
               at java.net.URLClassLoader.findClass(URLClassLoader.java:354)                                                      
               at java.lang.ClassLoader.loadClass(ClassLoader.java:423)                                                           
               - locked <0x00000000ab84c0e0> (a java.lang.Object)                                                                  
               at java.lang.ClassLoader.loadClass(ClassLoader.java:410)                                                           
- locked <0x00000000ab84c0c8> (a java.lang.Object)   // java.lang.ClassLoader: LOCK acquired                                                             at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)                                                   
               at java.lang.ClassLoader.loadClass(ClassLoader.java:356)                                                           
               at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:371)                        
               at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119)                               
               at java.lang.Class.forName0(Native Method)                                                                         
               at java.lang.Class.forName(Class.java:186)                                                                         
               at org.jboss.tools.examples.rest.MemberResourceRESTService.SystemCLFailure(MemberResourceRESTService.java:176)     
               at org.jboss.tools.examples.rest.MemberResourceRESTService$Proxy$_$$_WeldClientProxy.SystemCLFailure(Unknown Source)

Now let’s replace our Class name by a Java class tagged as part of the “application” package and re-run the test under the same load conditions.

String className = "org.ph.WrongClassName";

As we can see, we are no longer dealing with BLOCKED threads…why is that? Let’s have a look at the JVM thread dump to better understand this change of behaviour.

"default task-51" prio=6 tid=0x000000000dd33000 nid=0x200c runnable [0x000000001d76d000]                               
   java.lang.Thread.State: RUNNABLE                                                                                    
               at java.io.WinNTFileSystem.getBooleanAttributes(Native Method)    // IO overhead due to JAR file search operation                                                   
               at java.io.File.exists(File.java:772)                                                                                
               at org.jboss.vfs.spi.RootFileSystem.exists(RootFileSystem.java:99)                                                   
               at org.jboss.vfs.VirtualFile.exists(VirtualFile.java:192)                                                            
               at org.jboss.as.server.deployment.module.VFSResourceLoader$2.run(VFSResourceLoader.java:127)                         
               at org.jboss.as.server.deployment.module.VFSResourceLoader$2.run(VFSResourceLoader.java:124)                         
               at java.security.AccessController.doPrivileged(Native Method)                                                        
               at org.jboss.as.server.deployment.module.VFSResourceLoader.getClassSpec(VFSResourceLoader.java:124)                  
               at org.jboss.modules.ModuleClassLoader.loadClassLocal(ModuleClassLoader.java:252)                                    
               at org.jboss.modules.ModuleClassLoader$1.loadClassLocal(ModuleClassLoader.java:76)                                   
               at org.jboss.modules.Module.loadModuleClass(Module.java:526)                                                         
               at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:189)   // JBoss now fully responsible to load the class                                      
               at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:444) // Unchecked since using JDK 1.7 e.g. tagged as “safe” JDK                
               at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:432)                   
               at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:374)                          
               at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:119)                                 
               at java.lang.Class.forName0(Native Method)                                                                            
               at java.lang.Class.forName(Class.java:186)                                                                           
               at org.jboss.tools.examples.rest.MemberResourceRESTService.AppCLFailure(MemberResourceRESTService.java:196)          
               at org.jboss.tools.examples.rest.MemberResourceRESTService$Proxy$_$$_WeldClientProxy.AppCLFailure(Unknown Source)    
               at sun.reflect.GeneratedMethodAccessor60.invoke(Unknown Source)    

The above execution stack trace is quite revealing:

  • Since the Java class name was not detected to be part of the Java system package, no ClassLoader delegation was performed, thus no synchronization.
  • Since JBoss considers JDK 1.7+ as a “safe” JDK, the ConcurrentClassLoader .performLoadClassUnchecked() method was used, not triggering any Object monitor lock.
  • No synchronization means no Thread lock contention triggered as a result of the non-stop ClassNotFoundException error.
It is still important to note that while JBoss is doing a great job at preventing Thread lock contention in this scenario, the repetitive class loading attempt will still degrade the performance to a certain degree due to IO overhead associated with excessive JAR file search operations, again re-enforcing the need to take immediate corrective actions.

Final words

I hope that you enjoyed this article and have now a better understanding of potential performance impacts due to excessive class loading operations. While JDK 1.7 and modern Java EE containers brought great improvements on class loader related issues such as deadlocks and thread lock contention, potential problematic scenarios still remain. For this reason, I highly recommend that you closely monitor your application behaviour, logs and ensure that class loader related errors such as java.lang.ClassNotFoundException and java.lang.NoClassDefFoundError are aggressively corrected.

I’m looking forward for your comments and please share your troubleshooting experience with Java class loaders.

JCG Member DZone MVB