Archive for the ‘Tomcat’ Category

Calculating E2E no-JVM test code coverage with JCov

  1. What on earth is dynamic instrumentation code coverage and why you may want it?

  2. Java code coverage tools, like these embedded in IDEs or provided as CI environments plugins are great, but they have one limitation – the tests you run have to also be written in Java or other JVM language. What if you have suites of tests in other, non JVM languages and would like to know what is covered and what is not?

    I’ve faced such an issue – we had a really big suite of e2e REST API tests written in Python, and executed them against big Java application running on Tomcat. We wanted to track, where do these tests go in the code. But how to check it?
    Possible solution is so called dynamic instrumentation of Java code using Java agents. Shortly speaking Java agent triggers in the moment of classloading and transforms class bytecode to save some statistics about executed lines. Magic applied in practice.

    According to my research there are two reasonable solutions providing (among other features) the dynamic instrumentation :

    • Jacoco – having an agent mode; recognizable by some people (4090 Stack Overflow matches as of February 2019 ) , not necessarily due to it’s capabilities for outside-java-triggered execution, but for its Maven plugins integrated with Jenkins plugins, Sonar plugins, etc. I used it in some of previous projects to observe code coverage from release to release, so it was my first shot. Disappointingly for this use case it didn’t work well. More on it in the Why not Jacoco section
    • JCov – having an agent mode; developed as a tool targeted at Java Java developers (not a typo, I mean people actually developing Java, yep). Probably originating in the depths of Oracle basements. Known by a small group of unicorns (9 Stack Overflow posts in three years and one reasonable presentation). You have to checkout and build it yourself. I went there, I’m alive, and I’m coming with a practical tutorial.

    I used it for generating code coverage reports while running e2e Python tests, but of course with such flexible agent you can track coverage of the code called anyhow you want. Postman suite? No problem. Clicking around the GUI wondering how the heck it works inside without staring at the debugger? Here you go.

  3. Dynamic vs static instrumentation

  4. As I mentioned before, the instrumentation (process of altering the bytecode of examined classes) may be dynamic, and this is the flavour I used, but it can also be static. With static instrumentation you mutate the class files before you run them to gather coverage statistics. In comparison to dynamic instrumentation it makes execution faster (as there is no overhead on classloading) and it consumes much less memory resources in general. JCov also supports static instrumentation, I haven’t tried it though, as dynamic mode was more suitable for me. You can find more information on static instrumentation mode in the linked JavaOne presentation from JCov developers themselves and also in the indispensable verbose help mode built in the jcov.jar

  5. Setup and operating instructions for JCov in dynamic mode for Tomcat application

    1. Building the tool itself

    2. It is not that easy to start with JCov. Actually you have to build it yourself using manually downloaded dependencies, as site with released versions doesn’t really work. I have built the final jcov.jar for you, and also I provided intermediate dependencies copied to my repository to save you from hassle looking up the dependencies over the net. But I still recommend to download the sources yourself, as with documentation being scarce going to the sources can show you some hidden functionalities. At least that was my case.

      If you want to prepare it yourself go as follows :

      These are official building instructions, although they seem to be a bit ‘vintage’, as JCov supports modern Java versions, and readme still refers to JDK5. I built it on JDK8.

      1. Clone jcov mercurial repo : hg clone http://hg.openjdk.java.net/code-tools/jcov
      2. You will need following Jar dependencies built/downloaded manually to build JCov (I post exact versions which I used, maybe other will work too) :
      3. Asm-7.0.jar
        Asm-tree-7.0.jar
        Asm-util-7.0.jar
        jtharness-4_5-dev-bin-b27-19_apr_2013/lib/javatest.jar

        I have found them in the various places on the web. You can also download them from my github repository.

      4. Edit /build/build.properties file, setting paths to aforementioned jars.
      5. # path to asm libraries
        asm.jar = /asm-7.0.jar
        asm.tree.jar = /asm-tree-7.0.jar
        asm.util.jar = /asm-util-7.0.jar

        # path to javatest library (empty value allowed if you do not need jtobserver.jar)
        javatestjar = /javatest.jar

      6. Go to /build directory and execute ‘ant’ command. If you got all paths well this should finally build jcov.jar file in /JCOV_BUILD/jcov_/ directory
    3. Using the JCov tool.

    4. jcov.jar embeds all functions needed to generate coverage report, from instrumenting classes to compiling it in .html (or other format) report itself.

      The JCov tool has many sub-tools and usage scenarios – what’s below is a concrete case where dynamic instrumentation is used to generate and dump coverage data on the fly from application server which was not pre-instrumented before running.

      The phases to generate such report are as follows.

      1. Attach jcov.jar to server startup configuration in Java agent mode, exposing commands server
      2. Start the application server
      3. Command the agent to dump data to file (to get rid of server startup Java coverage from statistics )
      4. Run cases where coverage interests us, by any means on the application server (anyhow, for instance python nosetests, postman suite, even manual clicking around the GUI or browser automation plugin)
      5. Dump the coverage data again to file
      6. Generate HTML or other report from coverage data

      Now let’s go through these phases in details:

      1. Attach jcov.jar to server startup configuration in java agent mode, exposing commands server :
      2. In the case of Tomcat server the configuration went as follows :

        -javaagent://jcov.jar=file=important-coverage-data.xml,merge=gensuff,log,log.level=WARNING,include=com\.mycompany*,agent.port=3336

        Let’s explain the params :

        • file=important-coverage-data.xml,merge=gensuff – the root of filename where the coverage data will be dumped to. The ‘gensuff’ merge option makes JCov create separate file for coverage data every time dumping is requested, with a random suffix. There are also other modes of operation, for instance overwriting existing data in xml disk file, or merging its content. For my case the separate files with suffixes seemed to be the most practical mode
        • Log, log.level – you know what it is
        • include=com\.mycompany* – makes java agent instrument only classes in selected packages. By default all classes are instrumented, but beware of that – it may cause excessive memory consumption, even OOM errors for bigger systems. Most probably you’re not interested in coverage data for the external libraries anyway. Note that you can use multiple include and exclude statements to have more sophisticated configuration (refer to jcov.jar built in help for more information)
        • agent.port=3336 – in this configuration simple command accepting server will be started inside agent, waiting for signal to dump coverage data for file. Other options are just dumping data to file on VM exit, or using so called ‘Grabber’ module (read further)
      3. Start the server
      4. Start the server however you do it in normal operation – just make sure it picks up configuration from point 1. In this concrete configuration you will see that it works if something listens on localhost/3336.

      5. Command the agent to dump data to file
      6. In the agent.port configuration, commanding to dump current coverage statistics to XML file turns out to be really crude.

        Connect to the server using : telnet localhost 3336

        Every time you send ‘save’ string to server it will save next part of coverage data to xml file and respond with ‘saved’ string to your telnet session.

        Of course you can automate it, if you need to dump the excessive amounts of data periodically, even in bash, for instance

        (sleep 1; echo "open localhost 3336"; sleep 1; while :; do echo "save"; sleep 60; done ) | telnet

        Or any other telnet client integrated with your testing environment.

        This will dump new portion of data every minute.

        Remember you can merge the results later with Merge command from jcov.jar

      7. Run cases where coverage interests you, by any means on the application server (anyhow, for instance python nosetests, postman suite, even manual clicking around the GUI or browser automation plugin)
      8. Dump the coverage data again to file
      9. As in step three. My practice was to dump data just before running the suite, and just after, and then using just the second file to calculate coverage report.

      10. Generate HTML or other report from coverage data
      11. To generate the report in HTML navigable format, along with the covered lines marked in class sources, execute command like

        java -jar jcov.jar RepGen -sourcepath /path/to/module/one/src:/path/to/module/two/src /path/to/coverage-data.xml

        This will generate, by default, fully navigable HTML report in /report directory below where jcov.jar resides.

  6. Troubles I’ve encountered and you may also

  7. Apart from a lot of time I’ve spent on trivial things, which you don’t have to, after I’ve figured it out – like having to built it with good old Ant or figuring how to signal Agent to dump contents to file, there are some problems which you still may stumble upon:

    • Out Of Memory Errors / HeapSize
      don’t be surprised if your application throws OOMs at you or starts to act strangely. This may be GC dying there. Instrumenting the classes and storing execution statistics may use quite a lot of memory. I had to increase my HeapSize a lot to startup a big set of Tomcat applications, even after including only my company classes in the filter. Also it happened to me that if I haven’t applied ANY filter (just took all the classes) the JVM crashed sometimes instantly at startup. But with filters everything was alright.
    • Multiple source trees

      it was covered in tutorial but I would like to mention it again. Help for jcov.jar mentions just source location – singular. But in fact you can add multiple directories separated with your operating system separator (you can easily find it in JCov sources, going from the parameter name).

    • Spring/CGLib bytecode altering

      sometimes Spring resorts to generate class proxies which apparently don’t just nicely delegate calls to old implementations but dynamically subclass them. Effectively this means that the original class is lost in favor of newly generated one. You will see its statistics in the report, but without relation to original .java source. Another reason to reconsider using Java class inheritance nowadays!

    • Grabber module

      In this article I show how to use server embedded in agent, but in theory, according to JavaOne presentation the standard way to do it is using so called Grabber server, to which the Java agent connects. I’ve successfully set up the Grabber server, but the client didn’t seem to work, so I resorted to server embedded in agent.

    • Untouched classes

      In this example, only loaded classes are instrumented, so remember that if the class is not covered by tests at all, you won’t see it in report. If you have such need, you have to generate so called ‘template’ beforehand.

  8. Ok, what next? -hv is your friend

  9. JCov is quite mysterious tool in terms of community and documentation, but it’s really worth to note, that it has quite reasonable help built in the jcov.jar itself.
    It has many features which go beyond this introduction. For instance merging existing coverage data files, filtering results, even generating diffs from build to build.

    Start from typing :
    java -jar jcov.jar

    You will get a description of the modules of jcov.jar. You can get verbose info about every part by -hv option which stands for ‘help-verbose’. For instance :

    Java -jar jcov.jar Agent -hv

    It is also worth to explore the source code itself near places where command line parameters are processed.

    Also take a detailed look at JavaOne presentation I mentioned.


  10. Why not Jacoco?

  11. At first Jacoco seemed to be better choice, with much higher adoption, bigger community, IDE and CI integrations, but at least in my case it just basically didn’t work. After importing Jacoco coverage data to IntelliJ it was clear, that the lines which for sure were executed were marked as not executed in the report, which beats the purpose. Maybe it was the faulty plugin. I tried to generate report outside the IDE, using Jacoco tools themselves but.. The raport crashes when there are two classes of the same name in different packages, which for as big codebase as in mine case made this tool useless. I have to stress though, that I have previously had good experiences with this product while using Maven/Jenkins plugins for tests executed in JVM so for you it may be worth giving a shot.

  12. Sources

  13. My repository with JCov.jar and prerequisites built
    JavaOne JCov talk slides
    https://wiki.openjdk.java.net/display/CodeTools/How+To+Build+JCov

How to configure Eclipse to debug Alfresco java webscript code

How to configure Eclipse to debug Alfresco java webscript code

NOTE: It is assumed, that Eclipse environment has been configured to work with Alfresco SDK. If it’s not true the configuration process has been described here: http://wiki.alfresco.com/wiki/Alfresco_SDK_3.4#Set_Eclipse_Compiler_Compliance_Level_to_6.0.

1. Select: Run menu -> Debug configurations…
2. Create new configuration for ‘Remote Java Application’
3. In ‘Project’ field select ‘home’ project (presumably your webscript project), which you want to debug (but it is also possible to debug code from Alfresco core outside selected project)
4. In ‘Connection properties’ set Tomcat server IP address and debug port you defined earlier.

NOTE: It’s advised against to use ‘localhost’ instead of IP address on Windows because of hosts file issues.

5. Click Apply, Debug
6. If everything went correct you shouldn’t get any message.
7. Switch to debug perspective
8. You should see something like this:

Proper screen from eclipse debug perspective

Proper screen from eclipse debug perspective


In case of debug session disconnection (eg. Tomcat restart) you may reconnect to it using Relaunch option:
Reconnecting to debugger

Reconnecting to debugger


9. Now you can normally set breakpoints and trace code execution of your webscript or other Alfresco SDK projects. A good test is setting a breakpoint in org.alfresco.web.bean.LoginBean at line
“FacesContext context = FacesContext.getCurrentInstance();” – breakpoint should be caught at Alfresco Explorer login attemp.

NOTE: You may get following error message: “Unable to install breakpoint in org.evolpe.webscripts.XMLwebs due to missing line number attributes. Modify compiler options to generate line number attributes.
Reason:
Absent Line Number Information”

Missing line info window

Missing line info window

CAUSE&SOLUTION: To debug your own webscripts or other code it’s essential to enable code line number information in the compiler. If you use ant, set debug=”true’ parameter in javac task.

How to configure tomcat to debug instance of Alfresco on Windows and Linux will be described soon in my next articles.;)

How to perform form field validation in Alfresco Share?

It’s possible to implement your own field validation function in Alfresco Share in JS by overwriting Share mandatory constraint which uses
Alfresco.forms.validation.mandatory located in /opt/alfresco-3.4.d/tomcat/webapps/share/js/forms-runtime.js
The default configuration can be suppressed the way described here:http://wiki.alfresco.com/wiki/Forms#constraint-handlers
validation-handler parameter is the name of your own JS function.
Here you can find description of validation function parameters http://wiki.alfresco.com/wiki/Forms_Developer_Guide#Validation_Handlers

In the current version of Alfresco (3.4d) the validation ‘fails silently’ which means you can’t dynamically display error message. When validation fails it’s not possible to submit the form. There are foundations for enhancement of this mechanism so it’s quite possible the ‘full’ validation will be available in near future.
You can

download full eclipse ant project for my example here

. I will provide
instructions for manual installation as well.
Ant installation
1. Set paths to tomcat directory in build.properties
TOMCAT_HOME=/opt/alfresco-3.4.d/tomcatAPP_TOMCAT_HOME=/opt/alfresco-3.4.d/tomcat
2. Just run ‘ant’ in project directory.

So, let’s start the core of this tutorial.
1. I implemented my own simple data model for this example. As it is not complicated, nor directly connected with validation its self I just attach needed files with my project.If you’re not familiar with defining data models you can read about it for instance, here
tutorial:http://www.tribloom.com/blogs/michael/2011/04/18/alfresco-repository-webscript/

detailed documentation: http://wiki.alfresco.com/wiki/Data_Dictionary_Guide#The_Data_Dictionary
validationModel.xml goes to /opt/alfresco-3.4.d/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/extension/model
validation-model-context.xml goes to /opt/alfresco-3.4.d/tomcat/webapps/alfresco/WEB-INF/classes/alfresco/extension/

2. share-config-validation-custom.xml goes to /opt/alfresco-3.4.d/tomcat/webapps/share/WEB-INF/classes/alfresco/web-extension
This file contains definition of create and edit forms for the new data type.
Piece of code concerning validation:


<field id="ex:gross" label-id="example.price.gross" mandatory="true"
	help-id="example.price.gross.help">
	<constraint-handlers>
		<!-- validation-handler param: js function name -->
		<constraint type="MANDATORY"
			validation-handler="Alfresco.forms.validation.exampleGrossValidation"
			event="keyup" />
	</constraint-handlers>
</field>

 

Alfresco.forms.validation.exampleGrossValidation is the name of js validation
function.
3. exampleFormValidation.js and exampleFormValidation-min.js go to/opt/alfresco-3.4.d/tomcat/webapps/share/components/form

Alfresco.forms.validation.exampleGrossValidation = function exampleGrossValidation(field, args,
  event, form, silent, message) {
 

  var valid = true;

  valid = YAHOO.lang.trim(field.value).length !== 0;  // check if field is not empty

// referencing to form fields is possible by following convention:
// field object passed to function representing validated field has an object form representing the whole form  
// field object names are created by concating 'prop_'+{namespace from data model}_+{property name from data model}  
  var net = field.form.prop_ex_net.value;
  var gross = field.form.prop_ex_gross.value;
  var rate = field.form.prop_ex_rate.value;

  var check = ((rate/100) + 1) * net;

  if((Math.abs(gross - check)) > 0.01)  // check if gross has a valid value with given precision
{
valid = false;
}

  return valid;
};

 
4. exampleValidation_pl_PL.properties (labels for forms) goes to/opt/alfresco-3.4.d/tomcat/webapps/share/WEB-INF/classes/alfresco/messages


You have to modify following files:


1. slingshot-application-context.xml at /opt/alfresco-3.4.d/tomcat/webapps/share/WEB-INF/classes/alfresco/
Paths to share-config-validation-custom.xml and exampleValidation_pl_PL.properties go there
At xPath ‘/beans/bean/constructor-arg/list’ add:

<value>classpath:alfresco/web-extension/share-config-validation-custom.xml</value>
At xPath ‘/beans/bean/property/list’ add:

<value>alfresco.messages.exampleValidation</value>

2. form.get.head.ftl – link to .js fileat /opt/alfresco-3.4.d/tomcat/webapps/share/WEB-INF/classes/alfresco/site-webscripts/org/alfresco/components/form
Add: <@script type=”text/javascript” src=”${page.url.context}/res/components/form/exampleFormValidation.js”></@script>

3. toolbar.get.config.xml – In this file we can add a new option in menu ‘Create content’ in Alfresco Share Site.
at /opt/alfresco-3.4.d/tomcat/webapps/share/WEB-INF/classes/alfresco/site-webscripts/org/alfresco/components/documentlibrary/
add :
<content mimetype=”text/xml” icon=”xml” label=”example.price.menu” itemid=”ex:price”/>

Tomcat uses wrong path although CATALINA_HOME and CATALINA_BASE have correct values.

The cause of the problem may be that your web application uses tomcat6.exe instead of catalina.bat to start up.  From my observations it seems tomcat6.exe as opposed to catalina.bat uses environment variables only for the 1st run to copy CATALINA_HOME and CATALINA_BASE values to Windows Registry. Thereafter these parameters start to live on their own;) I had one instance of Tomcat for my Openbravo developer stack and the other for Alfresco. Tomcat6.exe used old values regardless of completly separate instalation of Alfresco dev stack.
You can edit these and other variables directly via Windows Registry editor, the path is:

HKLM/SOFTWARE/Apache Software Foundation/…..

or more conveniently you can use tomcat6w.exe GUI:

tomcat6w //ES//<serviceName> (eg. tomcat6w //ES//alfrescoTomcat)

calls a neat window where you can edit variables you need to run your web app properly.

there’s also //MS// parameter which will call a simple application monitoring given service:

tomcat6w //MS//<serviceName>

If you want to read more about running Tomcat as Windows service with tomcat6.exe go to an article from Apache Tomcat documentation.

Hope this helps someone:)

I get NoClassDefFound for ContextLoaderListener class using Spring and Eclipse IAM Maven plugin

My most common headache during my last Spring project was:


java.lang.ClassNotFoundException: org.springframework.web.context.ContextLoaderListener

during start of Tomcat.

Every source pointed to problems with spring.jar, so I suspected having some version conflict between my Spring libraries (eg. I used also Spring-WS in an old version). But that wasn’t real cause and I’d wasted a lot of time. The real problem was faulty Eclipse IAM plugin.

Try out some tricks I’ve described in my another article : Problems using Eclipse IAM and Maven

They really may help you.

Problems using Eclipse IAM and Maven

During my last project I was involved, I used Eclipse IAM plugin for integration with Maven. It caused many problems, mainly Class not found exceptions and issues with Tomcat server.

Generally I perceived that Eclipse IAM tends to lose jars from pom.xml, what leads to unexpected, bizarre time wasting NoClassDefFoundError.

Unfortunately I hadn’t been sucessful in finding causes of my problems but I’d like to share with you my standard ‘IAM problems checklist’ 😉

If you have any problems try this (in specified order) :

  1. Use refresh, then fetch source jars option from maven menu.
  2. Try cleaning maven libraries, then do ‘1.’
  3. After doing ‘2.’, remove your app from Tomcat, clean server, then deploy again.
  4. Try removing whole server, then add it again.
  5. REMOVE AND REGENERATE WHOLE WORKSPACE

Any step mentioned above is not a joke. I had really to ‘reset’ workspace a few times to make things work again.. First three steps helped me in 80% cases.

I hope my tips will save your time.

How to capture content of eclipse Console window (web tomcat application)?

The console window in eclipse has rather short ‘history’ and surprisingly the tomcat log file doesn’t appear anywhere on the hardrive when you run your application directly from eclipse. (Or it is situated in some mysterious place) It took me some time to find an option allowing to capture the content of console window to a given file.

The solution :

Run menu – > Run configurations… – ><your tomcat server > – > common – > Standard input and output

Tick the option ‘file’ and type path to your log file.

Subscribe by mail