Monitoring WSO2 products with logstash JMX input plugin

These days, I got the chance to play with ELK (Elasticsearch, Logstash & Kibana). These tools are a great way to analyze & visualize all logs.

You can easily analyze all wso2carbon.log files from ELK. However we also needed to use ELK for monitoring WSO2 products and this post explains the essential steps to use logstash JMX input plugin to monitor WSO2 servers.

Installing Logstash JMX input plugin


Logstash has many inputs and the JMX input plugin is available under "contrib"

We can use "plugin install contrib" command to install extra plugins.

cd /opt/logstash/bin
sudo ./plugin install contrib

Note: If you use logstash 1.4.0 and encounter issues in loading the jmx4r, please refer Troubleshooting below.

Logstash JMX input configuration


When using the JMX input plugin, we can use a similar configuration as follows. We are keeping the logstash configs in "/etc/logstash/conf.d/logstash.conf"


input {
jmx{
    path => "/etc/logstash/jmx"
    polling_frequency => 30
    type => "jmx"
    nb_thread => 4
  }
}

output {
  elasticsearch { host => localhost }
}

Note that the path points to a directory. We have the JMX configuration in "/etc/logstash/jmx/jmx.conf"

{
  //The WSO2 server hostname
  "host" : "localhost",
  //jmx listening port
  "port" : 9999,
  //username to connect to jmx
  "username" : "jmx_user",
  //password to connect to jmx
  "password": "jmx_user_pw",
  "alias" : "jmx.dssworker1.elasticsearch",
  //List of JMX metrics to retrieve
  "queries" : [
    {
      "object_name" : "java.lang:type=Memory",
      "attributes" : [ "HeapMemoryUsage", "NonHeapMemoryUsage" ],
      "object_alias" : "Memory"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=Code Cache",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolCodeCache"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Perm Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolPermGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Old Gen",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolOldGen"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Eden Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolEdenSpace"
    }, {
      "object_name" : "java.lang:type=MemoryPool,name=*Survivor Space",
      "attributes" : [ "Name", "PeakUsage", "Usage", "Type" ],
      "object_alias" : "MemoryPoolSurvivorSpace"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=*MarkSweep",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorMarkSweep"
    }, {
      "object_name" : "java.lang:type=GarbageCollector,name=ParNew",
      "attributes" : [ "Name", "CollectionCount", "CollectionTime" ],
      "object_alias" : "GarbageCollectorParNew"
    }, {
      "object_name" : "java.lang:type=ClassLoading",
      "attributes" : [ "LoadedClassCount", "TotalLoadedClassCount", "UnloadedClassCount" ],
      "object_alias" : "ClassLoading"
    }, {
      "object_name" : "java.lang:type=Runtime",
      "attributes" : [ "Uptime", "StartTime" ],
      "object_alias" : "Runtime"
    }, {
      "object_name" : "java.lang:type=Threading",
      "attributes" : [ "ThreadCount", "TotalStartedThreadCount", "DaemonThreadCount", "PeakThreadCount" ],
      "object_alias" : "Threading"
    }, {
      "object_name" : "java.lang:type=OperatingSystem",
      "attributes" : [ "OpenFileDescriptorCount", "FreePhysicalMemorySize", "CommittedVirtualMemorySize", "FreeSwapSpaceSize", "ProcessCpuLoad", "ProcessCpuTime", "SystemCpuLoad", "TotalPhysicalMemorySize", "TotalSwapSpaceSize", "SystemLoadAverage" ],
      "object_alias" : "OperatingSystem"
    } ]
}


This is all we need to configure logstash to get JMX details from WSO2 servers. Note that we have given a directory as the path for JMX configuration. This means that all the configs inside "/etc/logstash/jmx" will be loaded. So, we need to make sure that there are no other files.

I'm querying only the required attributes for now. It is possible to add as many queries as you need.

Securing JMX access of WSO2 servers.


WSO2 servers by default start the JMX service and you should be able to see the JMX Service URL in wso2carbon.log

For example:
TID: [-1234] [] [DSS] [2014-05-31 01:09:11,103]  INFO {org.wso2.carbon.core.init.JMXServerManager} -  JMX Service URL  : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi

You can see JMX configuration in <CARBON_HOME>/repository/conf/etc/jmx.xml and the JMX ports in <CARBON_HOME&gt/repository/conf/carbon.xml

       
        <!-- The JMX Ports -->
        <JMX>
            <!--The port RMI registry is exposed-->
            <RMIRegistryPort>9999</RMIRegistryPort>
            <!--The port RMI server should be exposed-->
            <RMIServerPort>11111</RMIServerPort>
        </JMX>

You may change ports from this configuration.

It is recommended to create a role with only "Server Admin" permission and assign to the "jmx_user". Then the "jmx_user" will have the required privileges to monitor WSO2 servers.

Also if we enable Java Security Manager, we need to have following permissions. Usually the WSO2 servers are configured to use the security policy file at  <CARBON_HOME>/repository/conf/sec.policy if the Security Manager is enabled.


grant {
    // JMX monitoring requires following permissions. Check Logstash JMX input configurations
    permission javax.management.MBeanPermission "-#-[-]", "queryNames";
    permission javax.management.MBeanPermission "sun.management.MemoryImpl#*[java.lang:type=Memory]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.MemoryPoolImpl#*[java.lang:type=MemoryPool,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.GarbageCollectorImpl#*[java.lang:type=GarbageCollector,name=*]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ClassLoadingImpl#*[java.lang:type=ClassLoading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.RuntimeImpl#*[java.lang:type=Runtime]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "sun.management.ThreadImpl#*[java.lang:type=Threading]", "queryNames,getMBeanInfo,getAttribute";
    permission javax.management.MBeanPermission "com.sun.management.UnixOperatingSystem#*[java.lang:type=OperatingSystem]", "queryNames,getMBeanInfo,getAttribute";
}

That's it. You should be able to push JMX stats via logstash now.


Troubleshooting

First of all you can check whether the configurations are correct by running following command.

logstash --configtest

This must tell that the configuration is OK.

However I encountered following issue in logstash 1.4.0 when running the logstash command.

LoadError: no such file to load -- jmx4r
     require at org/jruby/RubyKernel.java:1085
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:55
     require at file:/opt/logstash/vendor/jar/jruby-complete-1.7.11.jar!/META-INF/jruby.home/lib/ruby/shared/rubygems/core_ext/kernel_require.rb:53
     require at /opt/logstash/lib/logstash/JRUBY-6970.rb:27
     require at /opt/logstash/vendor/bundle/jruby/1.9/gems/polyglot-0.3.4/lib/polyglot.rb:65
  thread_jmx at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:132
         run at /opt/logstash/bin/lib//logstash/inputs/jmx.rb:251


For this issue, we need to extract the plugins to the same directory of logstash installation instead of contrib plugin installation. I got help from #logstash IRC to figure this out. Thanks terroNZ!

I did following steps

wget  --no-check-certificate -O logstash-contrib-1.4.0.tar.gz http://download.elasticsearch.org/logstash/logstash/logstash-contrib-1.4.0.tar.gz
tar -xvf logstash-contrib-1.4.0.tar.gz
sudo rsync -rv --ignore-existing logstash-contrib-1.4.0/* /opt/logstash/

Please note that JMX input plugin works fine in logstash-1.4.1 after installing contrib plugin and above steps are not required.

Then the next issue can occur is when connecting to WSO2 server. Check <CARBON_HOME>/repository/logs/audit.log and see whether the user can connect successfully. If it is not successful, you should check user permissions.

Another issue can be the failure of JMX queries. You can run logstash with "--debug" option and see debug logs.

I noticed following.
{:timestamp=>"2014-05-30T00:09:29.373000+0000", :message=>"Find all objects name java.lang:type=Memory", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.392000+0000", :message=>"No jmx object found for java.lang:type=Memory", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}
{:timestamp=>"2014-05-30T00:09:29.393000+0000", :message=>"Find all objects name java.lang:type=Runtime", :level=>:debug, :file=>"logstash/inputs/jmx.rb", :line=>"165"}
{:timestamp=>"2014-05-30T00:09:29.396000+0000", :message=>"No jmx object found for java.lang:type=Runtime", :level=>:warn, :file=>"logstash/inputs/jmx.rb", :line=>"221"}


This issue came as we have enabled the Java Security Manager and after adding permissions as mentioned above, the logstash JMX input plugin worked fine.

Next is to create dashboards in Kibana using these data. Hopefully I will be able to write a blog post on that as well.


Comments

Popular posts from this blog

Specifying a custom Event Settings file for Java Flight Recorder

Flame Graphs with Java Flight Recordings

Benchmarking Java Locks with Counters