Thursday, 7 December 2017

How to organise resource files for tests using Maven and Java

Resources are a very useful concept in Java. They are essentially files in the project which are compiled into the .jar

Java also has commands for finding them .getResource and reading them .getResourceAsStream

Very Handy.

But, they can be hard to wrap your head around. As evidenced by my own experiences trying to use them and the number of queries on stack overflow.

Friday, 13 October 2017

Simple ways to add and work with a `.jar` file in your local maven setup

TL;DR Hack - add as a library in IntelliJ project. Tactic - add as system scope in maven. Tactic/Strategic - install locally to .m2. Strategic - use a repository management tool, publish to maven central

Sometimes you want to work with a jar file that isn’t hosted in maven central.

It might be a 3rd party jar, it might be one that you have written.


You have a lot of options for this. The approaches that I have used:

  • add .jar files as an IntelliJ project dependency
  • install it locally to your .m2 repository
  • add it to your project as a system scoped file
  • use a repository management tool: like Nexus or Archiva
  • publish the dependency to maven central

Thursday, 12 October 2017

How to Diff Java Code in IntelliJ - 3 ways to use the Compare Tool

TL;DR IntelliJ has an inbuilt diff tool which you can use to compare files, classes or code with the clipboard. Just right click and choose Compare.

I was busy refactoring code in RestMud this morning because I want to try and open source the basic game engine, and then later the Web/REST API, but I want to tidy up the code a little first.

I’ve been working through the code:

  • moving classes into new packages
  • splitting classes to make code clearer and easier to manage
  • new classes make code easier to test

And the more I tidy it up in general, the easier it is to spot smaller problems of code duplication.

Thursday, 5 October 2017

Java 1.9 Reflection and Package Access Changes

TL;DR Java 1.9 does not like reflection as much as previous versions and access to proprietary packages is not possible out of the box.

Java 1.9 is out, and many companies now will be evaluating Java 1.9 and migrating their code base over to Java 1.9. I am now going through a similar process.

I have some fairly simple code that I was using for HTTP requests. This was originally written in Java 1.8 and to keep my codebase as simple as possible and I wanted to use no external libraries, therefore for HTTP requests I used an HttpURLConnection.

This started to fail on Java 1.9.

When I looked in my code more carefully I realised that it wasn’t HttpURLConnection that was failing in Java 1.9 it was my use of Reflection to bypass some constraints of the HttpUrlConnection that was failing.

What constraints?

With the HttpUrlConnection I could not find a way to get the actual request headers being sent.

I could see that the URLConnection class does have all the request headers:

private MessageHeader requests;

But it keeps them all to itself.

The class does support a mechanism to return these with the getRequestProperties method, which is public.

public Map<String, List<String>> getRequestProperties() {


But, wait a minute.

public Map<String, List<String>> getRequestProperties() {
return this.requests == null?Collections.emptyMap():this.requests.getHeaders((String[])null);

getRequestProperties does not actually want to return them if we are connected, the checkConnected method throws an exception if we try.

Fair enough.

I debug the code and find out when we can get them, and I’ll take them after they are set, but before we are connected.

But I could not actually find that point.

int statusCode = con.getResponseCode();

Prior to the above line I am not connected, and there are no request headers accessible.

After the above line, the request has been made, I am connected and the headers are not accessible.

I tried disconnecting, and a whole bunch of stuff, but since I had a problem to solve and the ‘class’ was getting in my way, I turned to Reflection to solve it.

What I did that failed

Since I new that the requests field on the URLConnection had the information that I needed, I simply used reflection to go and get the information that I needed.

Field field = con.getClass().getDeclaredField("requests");
MessageHeader requestHeaders = (MessageHeader) field.get((URLConnection) con);

lastRequest = new HttpRequestDetails();

int keyIndex = 0;
while(keyIndex < requestHeaders.getHeaders().size()){
lastRequest.addHeader(requestHeaders.getKey(keyIndex), requestHeaders.getValue(keyIndex));

And all of that was very jolly, until Java 1.9 came along.

Migrating to Java 1.9

After debugging the problem I faced, I found the following blog post from provided useful information:

The problem I faced was that when I migrated to Java 1.9 I faced the issue of:

Error:(7, 19) java: package does not exist
Error:(169, 13) java: cannot find symbol
  symbol:   class MessageHeader
  location: class com.javafortesters.course.casestudy.http.HttpRequestSender
Error:(169, 45) java: cannot find symbol
  symbol:   class MessageHeader
  location: class com.javafortesters.course.casestudy.http.HttpRequestSender

Now I found this odd.

The import of wasn’t throwing any errors. Until compilation.


And when I looked in the code for URLConnection the responses field was still there and was still of of type

So the package does exist, but just not for me.

Java 1.9 has finally made proprietary packages inaccessible. This was a long time warning in Java and has now come to fulfillment.

It is still in the JRE, but not accessible to me due to Java 1.9 changes.

Very well then.

Since the getRequestProperties method still exists, I decided to bypass the check for this.checkConnected();

I’m sure the library probably has very good reasons for not wanting me to access the List while we are connected, but in my code I view this as low risk since I have a single thread, I’ve made the request, I just want to document the headers that were sent.

So I used reflection to toggle the connectedfield value so that the checkConnected() method would not throw an exception.

Field connected = con.getClass().getSuperclass().getSuperclass().getDeclaredField("connected");
connected.setBoolean((URLConnection) con, false);

Map<String, List<String>> requestHeaders = con.getRequestProperties();
for(String header : requestHeaders.keySet()){
    StringBuilder headerValue = new StringBuilder();
    for(String headerText : requestHeaders.get(header)){
    lastRequest.addHeader(header, headerValue.toString());
connected.setBoolean((URLConnection) con, true);

And this worked in Java 1.9, and 1.8 and 1.7

But… Java 1.9 Doesn’t like reflection as much

But Java 1.9 tells me that this reflective access operation is ILLEGAL!

WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.javafortesters.course.casestudy.http.HttpRequestSender
 (file:/) to field
WARNING: Please consider reporting this to the maintainers of com.javafortesters.course.casestudy.http.HttpRequestSender
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release

I assume this is due to the new Module system in Java 1.9

So while my code works, it will probably fail again in the near future.

Reflection Comes with Risk

Reflection always comes with the Risk that the underlying code you are using changes.

And it has every right to do so.

I should not be accessing its private fields and methods.

But sometimes, when testing, I need access to features and functionality that, for seemingly arbitrary reasons, the class does not support.

Had this library been written to fully support a ‘testing’ process, where we don’t just want to make requests, we want to know exactly what details were sent in that request, then I wouldn’t have this problem.

The reason I learned to use reflection in Java in the first place was because I’ve worked with ‘frameworks’ that do stuff, that I want to use as part of my test process to validate results, or to trigger situations that are necessary for testing, but which the ‘framework’ decided that no-one would want to do that.

In theory, there is nothing wrong with the URLConnection telling me what the request headers are. It could have added them to a Synchronised or immutable collection, it is pure documentation.

Sometimes good solid technical reasons for functionality gets in the way of testing.


I could have added another HTTP library to my code to make the requests and give me access to the raw data.

I could have added a code based HTTP Proxy, sent all requests through the proxy and used the proxy to gain access to the headers.

All of that was possible, but went against my strategic aims of keeping the code base as free of external dependencies as possible, so I adopted a tactical maneuver to gain access to what I needed.

Final Results

In the public release of the above code I have removed all the reflection. But I have kept it in the code I use for internal testing because I can control the JVM and settings that the code runs against.

I will continue to use reflection to bypass constraints in libraries and frameworks that I use as a tactical measure to get work done.

Relying on this approach for strategic work is a risk that might bite you, at the time when you are least flexible to respond, so care needs to be taken when you do this.

Tuesday, 29 August 2017

Why does my code throw a null pointer exception? - common reason #1 Redeclaration

TLDR; check that you haven’t redeclared a field as a variable in a setup method

FAQ - why does my code throw a null pointer exception - common reason #1 Redeclaration

  • Using @BeforeClass or @Before can setup data for use in tests
  • Any ‘variables’ we instantiate need to be ‘fields’ rather than variables
  • We want to instantiate them in the setup method rather than redeclare them

Thursday, 6 July 2017

Implementing PATCH Verbs with Gson Jaxb and Spark Framework

TLDR; HttpURLConnection does not support PATCH, but we can sometimes use X-HTTP-Method-Override instead (this works with Spark Framework). I can Patch POJO using reflection, or deserialize to simple POJO and compare with null values instead

I had not planned to implement PATCH in my testing web service - at all, ever…

Wednesday, 5 July 2017

Architecting a Testable Web Service in Spark Framework

Architecting a Testable Web Service in Spark Framework

TLDR; Architecting a Web Service using Spark Framework to support more Unit testing and allow the inclusion of HTTP @Test methods in the build without deploying the application. Create API as a POJO. Start Spark in @BeforeClass, stop it in @AfterClass, make simple HTTP calls.

Background to the Spark and REST Web App Testing

I’m writing a REST Web App to help me manage my online training courses.

I’m building it iteratively and using TDD and trying to architect it to be easily testable at multiple layers in the architectural stack.

Wednesday, 14 June 2017

An introduction to Refactoring Java in IntelliJ using RestMud as an Example

TL;DR Never too late to refactor. Do it in small chunks. Protected by tests. Using IDE to refactor.

My RestMud game grew organically, I do have a fair bit of Unit testing, but I also perform more integration testing than unit testing.

The good thing about this approach is that the integration testing represents requirements i.e. in this game I should be able to do X, and my integration tests at a game engine level create test games and check conditions in there.

Thursday, 13 April 2017

JSoup Tip How to get raw element text with newlines in Java - Parsing HTML and XML with JSoup

TL;DR with JSoup either switch off document pretty printing or use textNodes to pull the raw text from an element.

A quick tip for JSoup.

I wanted to pull out the raw text from an HTML element and retain the \n newline characters. But HTML doesn’t care about those so JSOUP normally parses them away.

I found two ways to access them.
  • switching off pretty printing
  • using the textNodes

Friday, 17 March 2017

Mistakes using Java main and examples of coding without main

TL;DR A potentially contentious post where I describe how I've survived without writing a lot of Java main methods, and how learning from code that is often driven by a main method has not helped some people. I do not argue for not learning how to write main methods. I do not argue against main methods. I argue for learning them later, after you know how to code Java. I argue for learning how to use test runners and built in features of maven or other build tools to execute your @Test code.