Client side testing

[Vincent Privat]

Hello Rory,
Thank you :)

Here is how we test various OpenJDK builds:
1.1: we use only EA builds from http://jdk.java.net/
2.1.1: no
2.1.2/2.1.3: we run all unit and integration tests of JOSM
2.1.4: yes, using jacoco
2.2: all tests are automated
2.3: it depends. During OpenJDK9 cycle we did not run a test series for
each weekly build, but rather once every two or three builds, unless we
were waiting for a specific correction from OpenJDK or another tool. At the
end of development, we were not waiting for any correction from OpenJDK but
were eagerly waiting for the updates of ASM and everything that as blocked
by the non-availability of ASM (in particular various Maven plugins)
2.4: Yes, they are visible here:
- JOSM core:
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM/
- JOSM community plugins:
https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM-Plugins/

I just have initiated JDK10 tests today, and we have 53 failures caused by
Byte Buddy not recognizing Java version 10:

java.lang.IllegalArgumentException: Unknown Java version: 10
	at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.ofJavaVersion(ClassFileVersion.java:135)


I'm sure Rafael is aware of it. I will see how to update our tools
(equalsverifier here).

Cheers,
Vincent

2017-11-03 14:05 GMT+01:00 Rory O'Donnell <rory.odonnell at oracle.com>:

>
> Hi Vincent,
> I noticed that JOSM have been commended in the last two Quality reports
> [1] [2] for the number and
> indeed the quality of bugs you have found and logged against various early
> access releases.
> Thanks again for your contribution to the quality of our releases!
>
> The big question is how do you do it ?
>
> We would love to hear more details about your test environment, what tests
> you run etc. Below is a
> series of questions that come to mind, but there may be other details that
> you would like to share.
>
>    1. Builds
>       1. do you build your own OpenJDK builds, use Oracle's or other EA
>       builds ?
>    2. Tests
>       1. What tests do you run
>       1. do you run the JCK ?
>          2. open regression tests ?
>          3. other tests ?
>          4. do you measure code coverage ?
>          2. Are the tests Automated or Manual ?
>       3. How frequently to do execute the tests
>       4. Are the results visible via an external link
>
> Rgds, Rory
> [1] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+September+2017
> [2] https://wiki.openjdk.java.net/display/quality/Quality+
> Outreach+report+March+2017
>
> --
> Rgds,Rory O'Donnell
> Quality Engineering Manager
> Oracle EMEA , Dublin, Ireland
>
>