Client side testing

[Rory O'Donnell]

Hi Vincent,

Thanks for the update, plenty of info for us to review. We may comeback 
with follow on questions.

Rgds,Rory
On 03/11/2017 15:33, Vincent Privat wrote:
> Hello Rory,
> Thank you :)
>
> Here is how we test various OpenJDK builds:
> 1.1: we use only EA builds from http://jdk.java.net/
> 2.1.1: no
> 2.1.2/2.1.3: we run all unit and integration tests of JOSM
> 2.1.4: yes, using jacoco
> 2.2: all tests are automated
> 2.3: it depends. During OpenJDK9 cycle we did not run a test series 
> for each weekly build, but rather once every two or three builds, 
> unless we were waiting for a specific correction from OpenJDK or 
> another tool. At the end of development, we were not waiting for any 
> correction from OpenJDK but were eagerly waiting for the updates of 
> ASM and everything that as blocked by the non-availability of ASM (in 
> particular various Maven plugins)
> 2.4: Yes, they are visible here:
> - JOSM core: 
> https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM/
> - JOSM community plugins: 
> https://josm.openstreetmap.de/jenkins/job/Java-EarlyAccess-JOSM-Plugins/
>
> I just have initiated JDK10 tests today, and we have 53 failures 
> caused by Byte Buddy not recognizing Java version 10:
>
> java.lang.IllegalArgumentException: Unknown Java version: 10
> 	at nl.jqno.equalsverifier.internal.lib.bytebuddy.ClassFileVersion.ofJavaVersion(ClassFileVersion.java:135)
>
> I'm sure Rafael is aware of it. I will see how to update our tools 
> (equalsverifier here).
>
> Cheers,
> Vincent
>
> 2017-11-03 14:05 GMT+01:00 Rory O'Donnell <rory.odonnell at oracle.com 
> <mailto:rory.odonnell at oracle.com>>:
>
>
>     Hi Vincent,
>
>     I noticed that JOSM have been commended in the last two Quality
>     reports [1] [2] for the number and
>     indeed the quality of bugs you have found and logged against
>     various early access releases.
>     Thanks again for your contribution to the quality of our releases!
>
>     The big question is how do you do it ?
>
>     We would love to hear more details about your test environment,
>     what tests you run etc. Below is a
>     series of questions that come to mind, but there may be other
>     details that you would like to share.
>
>      1. Builds
>          1. do you build your own OpenJDK builds, use Oracle's or
>             other EA builds ?
>      2. Tests
>          1. What tests do you run
>              1. do you run the JCK ?
>              2. open regression tests ?
>              3. other tests ?
>              4. do you measure code coverage ?
>          2. Are the tests Automated or Manual ?
>          3. How frequently to do execute the tests
>          4. Are the results visible via an external link
>
>     Rgds, Rory
>
>     [1]
>     https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017
>     <https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+September+2017>
>     [2]
>     https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017
>     <https://wiki.openjdk.java.net/display/quality/Quality+Outreach+report+March+2017>
>
>     -- 
>     Rgds,Rory O'Donnell
>     Quality Engineering Manager
>     Oracle EMEA , Dublin, Ireland
>
>

-- 
Rgds,Rory O'Donnell
Quality Engineering Manager
Oracle EMEA , Dublin, Ireland