[DRAFT] Forgotten production code and Example driven development

Automated testing should be treated as production code
Automated testing acts as documentation for all features
Automated testing must be human readable including non-technical
Automated testing can also act as user guide and be included in production
Automated testing must consider the testing pyramid by Martin Fowler
Automated testing is about quickly helping us, not about complete testing

It is quite hard to make changes to code that is used by another feature that is forgotten.

Maven version ranges inception

Have you ever encountered a situation where your changes to a configuration file (like owasp-suppressions.xml) aren't being reflected? If you're working on a Maven project that uses version ranges, you might be in for a surprise.

In a maven-based project that uses version ranges, there is an implicit expectation may lead to great frustration for developers. Frustration similar to an intermittent issue that is hard to figure out. Let's say you've defined the version range [1,2) for a dependency you also develop. You'd expect this to always point to the latest SNAPSHOT, as long as it is before the release version 2.0.0. However, this expectation may unexpectedly defy expectations, causing the latest SNAPSHOT to not be reflected during a local build.

When you define the version range [1,2)

for a dependency that you also develop for, normally you expect it to behave such that it will always point to the latest SNAPSHOT as long as it is before the release version 2.0.0.

In my case, the dependency was released a few days prior. But due to unrelated build issues, like Docker base Debian images suddenly failing to run apt-update, the release had to be done twice. So instead of release 1.5, the version was bumped to 1.6. Our build process uses the Maven release plugin to automatically update the version to the next development SNAPSHOT. However, due to the build issues, the release for 1.6 had to be done manually, and updating to the correct version of 1.7-SNAPSHOT was forgotten. The dependency project was left at version 1.6-SNAPSHOT.

During a Maven build, the version range resolves to the latest version. In my case, the latest was the release version 1.6, as release versions are considered greater than SNAPSHOTs. Additionally, Maven typically checks both local and remote repositories for a given dependency. So, when the remote repository has the released 1.6, but locally you're still on 1.6-SNAPSHOT, your build will use the release version instead of your local SNAPSHOT version. This causes a situation where your local code changes aren't reflected when you run your code.

I was able to diagnose this problem by looking at the metadata files that maven keep in:

cat ~/.m2/repository/path-to-your-dependency/maven-metadata-<remote-hostname>.xml

Reflection of thoughts #1 - Logs and Multithreaded applications

Ever since I have started working, applications have had to run an increasing number of threads. This is mostly due to the adoption of web applications in favor of desktop applications. Due to this change; One the more difficult work that has come up is looking at the logs and figuring out what happened.

For a significantly busy server, looking at the logs is not as straightforward as I would expect.

1. In the logs, each http request is represented as its own thread, so you can identify which logs belongs to which http request since the thread name is included in the log. However, it would still be hard to hard to follow because multiple threads be adding new lines to the log in a mixed fashion because these threads are running simultaneously.
2. Much more difficult is when each http request would also spawn new background threads where each has their own name which you cannot reliably associate with the parent http request. For example a spawn background thread called acme_Quartz_Worker-9 . With the logs being appended by many threads on-demand.. I think it is impossible to identify which worker thread belong to which http request thread.

There are basic I have learned over the years that are must-have things to keep the logs as helpful as possible:

1. In Java, always make sure to always include the exception when you call logger.error(). Failing to include it may hide valuable clue since most exceptions have a root cause exception that would usually be the most important clue.
2. If available, always add contextual information in the log message. Usually this would be the id of the object being processed (like the UUID of a particular entity). Human readable information is also welcome.
3. Have a well designed exception system where you ensure which layers do log an error and which layer does the logging of errors. For instance, logging an error in a class called StringUtils is a very bad idea since different callers of this utility class may want to handle errors in different ways. Also, it is hard to determine which process had an error with the StringUtils. In addition, you usually do not (and you should not) pass contextual information to a StringUtils method - which is recommended to include in #2.

Further thoughts: I found this great blog post reflecting on how our logs are not human readable yet it is also not computer-readable (since the software stack like ELK is popular).

Beyond TDD

 Today I was writing a class JUnit test code that of course runs very slow as it starts up by bringing the whole AWS infrastructure online (exaggeration) just to run several assertTrue statements..

As you might suspect, I am not fond of writing this test code. This is because I know in my heart that this test code will be more burden than useful:

1. The test will contribute to a test suite that will take 5 hours to complete
2. The test is very hard to read because it deals with CSV importing code.. and Java does not deal well with such string intensive inputs
3. The test has a lot of test specific code which adds maintenance burden
4. Dev only test code, modern test code strive towards making it readable for non-technical

This frustration brought me an idea. A vision of testing Beyond TTD. What if tests live inside the application and can be used run time? The main motivation for this is to really have a living test suite and documentation. This makes it so that non-technical users can add and maintain test cases themselves with normal user-interface.

It has pros and cons of course, but it feels to me that this is the right balance. You will of course needs to really design this and probably favor pure functions to aid in test simplicity..

PS: The title is inspired by Nico Rosberg's podcast called "Beyond Victory"

TIL: The JVM can ommit stacktraces in the name of optimization

While investigating an issue, I saw a NullPointerException without a stacktrace..

ERROR logs that omits the throwable object (and it is available) is quite devastating. I call such situation 'swallowing the exception'.

However, this time, it is very different. Apparently the HotSpot JVM may opt to not log the stacktrace but  only the exception:

source: https://stackoverflow.com/questions/2411487/nullpointerexception-in-java-with-no-stacktrace

Having a look at older instances of this ERROR log, I found that the occurrence the day before indeed had stacktrace. This is quite surprising and something I would not expect..

Software Methodologies and Religion

I have recently read an interesting 2006 post from Steve Yegge titled Good Agile, Bad Agile. The comments section has also informative criticisms.

It's good to read and understand various ideas. From my experience and knowledge so far, I think the biggest problem with methodologies is that company X can be so different and it is very easy to blindly 'adopt' a methodology without putting in the effort in understanding it. The same with religion and superstition. (Idea I got from this article).

The most interesting idea I have learned so far on this topic however comes from Edgar Schein. Paraphrasing the idea: "Professional Relationship hurts Communication". I highly recommend listening to him on YouTube. I look forward to read his books.

TIL (Today I Learned) - various important things for Software Development

1. When resetting a whole column on a relational database table, dropping/adding the column might be better than running an UPDATE. In my case, I had to update two columns to NULL from a table with 35 million rows.
2. OAuth 1. I had to support a third party to use our REST service that uses OAuth 1.0a. In only a couple of days, I had to learn how this spec works and write up the details needed by the 3rd party to write a client. I'm glad that today there's plenty of resources for this complicated spec; I can imagine that this would be very troublesome to understand in the past. At least a former colleague had a lot of issues when he worked with Facebook OAuth in the past. This blog post really helped me out in quickly understanding OAuth.
3. Monitor HTTP traffic of java programs via Fiddler. With networking setups, it is often easier said than done. By experience, I knew I would be in trouble with how to setup the SSL certs, force java to to use a proxy server, etc. Surprisingly, this is actually easy to do. I use this guide and was able to set it up in 5 mins!