Using the myAVR “mySmartUSB light” under Linux in the Arduino IDE

I recently bought a handful of Arduino Nano clones, just in order to learn that for a price of 4€ per CPU it is not granted that there is a bootloader on the ATmega328P. Hence I needed an “in-system-programmer” to be able to install the bootloader. I found the “mySmartUSB light” from myAVR at Conrad for a reasonable price.

The device works in principle on Linux with avrdude, given that you have a driver for the “CP210x USB to UART Bridge” already installed. To make it also work in the Arduino IDE, you just have to add the following lines to the file ~/.arduino15/packages/arduino/hardware/avr/<version>/programmers.txt:

mysmartusb.name=mySmartUSB light
mysmartusb.communication=serial
mysmartusb.protocol=stk500v2
mysmartusb.speed=115200
mysmartusb.program.extra_params=-b115200 -P{serial.port}

This adds a new entry to the “Programmer” submenu for the ISP device.

Help! Gradle is using a different version of a library to build my extension/plug-in!

Just a note for myself that might also help others.

Symptoms: When building your Gradle extension/plug-in, strange compile errors occur. E.g. the compiler can’t find methods, which are obviously there when you look into the library’s code.

Problem: Gradle carries a lot libraries. It is likely that you want to use one of those libraries in a more recent version. Gradle adds its libraries to the classpath when you add compile gradleApi() to your dependencies. Gradle also does this implicitely when you use the buildSrc mechanism to organize your extension/plug-in code.

Solution:

  • Do not put code directly into buildSrc, use subprojects within buildSrc instead.
  • Make sure to include the subprojects into buildSrc‘s runtime classpath to make them available in the build script of your project at build time:

// in buildSrc/build.gradle
dependencies {
runtime subprojects.collect { owner.project(it.path) }
}

  • Use compile files(gradleApi().resolve().findAll { !it.name.contains("<the library you want to use in a different version>") }) instead of compile gradleApi() in the subprojects of buildSrc.
  • Do not add apply plugin: 'java-gradle-plugin' in any of buildSrc‘s subprojects because it implicitely adds compile gradleApi().

The secret step to save your software from the creeping death of unmaintainability

Abstract

Software which is developed over several years usually tends to come into a state, where the effort and risk of change increases dramatically with each new feature. This often ends at the point, where either nobody dares to take the risk of any further change, or it is simply too expensive to implement a new feature.

What is necessary to guard against this destiny and keep a software fit for change even after years of development?

Constant Change

Change is inherent to software. We software developers have overcome the mentality that you write one specification, implement it and you are done for good. History tought us that it is impossible to get all things right the first time. With the success of the agile movement the insight has manifested that software is under constant change right from the beginning. The overwhelming majority of software developers has accepted this fact as a law of nature now.

There are several sources of change. Customers change their mind over time. When using the software, they want to improve and optimize the way how the software solves their original problem. They see possibilities to increase the value the software could provide to them by adding new features. They also start to find new problems, that could also be solved with the software, maybe with some “small” modifications. Since the customer and the customer-value drive the development of a software, we have to incorporate all this into the software – somehow.

We developers also change our mind. While working on a solution for a specific problem, we learn more about this problem. Assumptions that we made in the beginning turn out to be wrong or imprecise. Algorithms that we found do not scale and need to be optimized or replaced. We do housekeeping, restructure and refactor the codebase. Especially the agile movement developed several techniques that should keep the codebase in a shape that will allow us to constantly improve and optimize its internal structures. But practice shows that it is not that easy to follow the rules over time. The one or the other shortcut will be taken from time to time.

Finally the technology that we depend on, the libraries, the underlying platforms, and the frameworks, evolve over time. New features are availabe, better alternatives are available, or a complete paradigm shift takes place.

Ten years ago, the main computing device for consumers was the PC. Broadband internet was not wide spread yet, not to mention mobile internet. If you provided an application that makes intensive use of user interaction or processes a reasonable amount of data, you would have implemented it as a desktop application for the PC. Now, post-iPhone-web-2.0, you would provide the same solution as a cloud-based service which can be used on different devices, ranging from a smartphone over the webbrowser to a rich client application on the PC.

Based on the technology available in 2006, nobody would have anticipated this. If the above mentioned software is the main asset that your business is based on, you better have a plan to adapt. Otherwise the paradigm shift may kick you out of business.

Constant Growth

Software under development is constantly growing. Each new feature is adding to the codebase, as well as each refactoring (e.g. extract method, introduce parameter object, extract class etc.) increases its size. This is not bad in itself, don’t get me wrong. For example according to the Open-Closed-Principle, one of the SOLID principles, you should implement in a way that you do not need to change existing code in order to extend its functionality. Instead you should add separate new code for the new feature and dependencies to the existing code.

Also refactoring is a necessary practice in order to keep the codebase in shape. Otherwise the “freezing” effect which makes the software inmaintainable sets in much much earlier.

So working on a codebase naturally increases its size, growth is unavoidable. But growth also causes problems. Increased complexity leads to increased costs for maintenance: there is simply more code to read and understand in order to make changes, there are more dependencies to take into account, there are more concepts, more structures to learn about.

More code usually also means more people who work on the code. This itself is a source of new complexity: the structure of teams will reflect in the codebase (Conway’s Law) and due to the increase of communication, the whole development process will become more formal and hence slow down.

Decisions

Software development, especially software architecture, is all about decisions. You need to decide about technology, algorithms, features sets, structure etc. Each decision is based on the knowledge that you have at a certain point in time and assumptions about all the things you don’t have knowledge about.

The longer a software lives, the more of the decisions will have to be revised at some point:

  • you gain additional knowledge and assumptions turn out to be wrong,
  • the world changes and assumptions that have been true at some point are not valid anymore.

Then the question is, how big is the effort to replace all code that is related to a certain decision? Is this even possible? If not, you will have to live with your original decision.

For each decision, you either have to implement it in a way that it can be revised later, or you have to be aware that your decision now may constraint your possibilities in the future. Since you don’t know the future alternatives, it is difficult to anticipate the consequences of this.

Responsibilities

The concept of responsibilities helps you to find related code. A responsibility can be defined as one reason for change. This means all code that has to be changed for the same reason serves the same responsibility.

According to the “Single Responsibility Principle”, responsibilities should not be mixed. Each unit of code should have only one reason to change, i.e. one responsibility.

There also exist different levels of granularity for responsibilities.

An example: the class TCPConnection allows you to send and receive data over the network using TCP.

public class TCPConnection { 
    public void connect() { 
        // ...
    }

    public void disconnect() {
        // ...
    }

    public void write(char c) {
        // ...
    }

    public char read() {
        // ...
    }   
}

At the first glance, this class has the responsibility to handle network communication over TCP. At a closer look, it serves two separate responsibilities:

  1. connect() and disconnect() do connection management,
  2. write(char) and read() handle data transfer.

Over time, responsibilities tend to be mixed. For example, imagine you need to add a feature for measuring the round-trip time of the TCPConnection. It is very tempting to add just another method for this to the TCPConnection class, because everything you need is already there. This adds another responsibility to TCPConnection on the finer grained level.

Dependencies

Dependencies are obviously related to decisions. For example, you decide to use a certain graphics library. There will be code that depends on this library. You decide to use the MVC pattern, then there will be a certain structure of dependencies in your code. This structure looks different if you choose another pattern.

Dependencies are also related to responsibilites. In the TCPConnection example, there is code, that uses connect() and disconnect() to establish a connection, and there is code that uses read() and write(char) to transfer data. This code shares the dependency to the TCPConnection class, but might otherwise not be related at all.

Dependencies define the amount of impact that a certain change to the codebase has. When you modify a piece of code, then everything that depends on this piece might have to be adapted to the change. This is true for direct dependencies and might be true for indirect dependencies as well.

Dependencies tend to spread throughout the codebase over time. The more secure a decision for example for a specific technology seems to be, the higher the probability is that this technology is used directly allover the place. E.g. you decide to use SWT as your GUI library. Then all your GUI related code might directly access the SWT library. Even if you build a facade to hide SWT, the structures and constants of SWT will very likely slip through the facade and be used directly somewhere in your codebase. This makes it very difficult to revise the decision for SWT and for example replace it with JavaFX at some point.

Isolate Responsibilities and Minimize Dependencies

Let’s recap what we have until now: Software development is based on decisions that are valid only for a limited amount of time. They might have to be revised sooner or later. Responsibilities and dependencies define the amount of code that is related to a certain decision. Hence, the effort needed to revise a given decision depends on the way you care for responsibilities and dependencies in your codebase.

To keep the effort low, proper measures would be to have responsibilities isolated on a fine grained level and to minimize the number of dependencies of each unit in your codebase. There are several techniques to accomplish this. To examine those, it makes sense to first look at some code:

public void selectUserProfileImage() {
    FileDialog dialog = new FileDialog(shell, SWT.OPEN);
    dialog.setText("Select Image");
    dialog.setFilterPath("~/images/");
    dialog.setFilterExtensions({ "*.jpg", "*.png", "*.gif", "*.*" });
    String selectedImageFilename = dialog.open();

    if (selectedImageFilename == null) return;

    Properties userSettings = new Properties();
    try (BufferedReader reader = new BufferedReader(FileReader(USER_SETTINGS_FILE))) {
        userSettings.load(reader)
    } catch (IOException e) {
        return;
    }

    userSettings.setProperty("profile.image.filename", selectedImageFilename);

    try (FileWriter writer = new FileWriter(USER_SETTINGS_FILE)) {
        userSettings.write(writer, "user settings");
    } catch (IOException e) {
        return;
    }

    updateUserProfileImageInUI();
}

The method above lets the user select an image file as her profile image. If the user actually selects one, it stores this information in the user settings and updates the UI to make sure the selected image is shown.

What are the fine-grained responsibilities that this method implements?

  1. handle the selection of an image file using a file dialog,
  2. handle the case, that the user did not select anything (by simply returning without further action),
  3. load the user settings from a file in the filesystem,
  4. handle any error condition while reading the file (by simply returning, but anyway),
  5. store the filename of the user profile image in the user settings,
  6. store the user settings into a file in the filesystem,
  7. handle any error condition while writing the file (by simply returning, you get it),
  8. make sure, that the UI is updated.

Let’s also have a look at the outgoing dependencies of the method:

  1. the UI framework, especially the FileDialog class,
  2. the underlying filesystem (“~/images/”),
  3. the image file formats which are supported by the application,
  4. the Java Properties class,
  5. the Java IO framework.

If any decision changes which is related to one of the above responsibilities or dependencies, the method needs to be changed. And even worse, some of the responsibilities are very likely to appear also in other places in the codebase. You will have to find all places and change them all consistently – this is also called “shotgun surgery”.

Layering

A very common approach in programming is to split a problem into different layers with different levels of abstraction. The highest level provides a description on what we do, with as little technical knowledge as possible. The levels beneath take care about how we do it, with increasing technical detail as you descend the levels of abstraction. On the lowest level you find very detailed and generic implementations, that have no knowledge about the original business problem at hand.

In practice, different levels of abstraction should not be mixed in one method. Each method is on a certain level and uses methods one level below (this principle is valid for classes and packages as well).

Rewriting our top-level method using this principle would lead to something like this:

public void selectUserProfileImage() {
    Optional<String> selectedImageFilename = ui.selectUserProfileImageFile();
    if (selectedImageFilename.isPresent()) {
        userSettings.setProfileImage(selectedImageFilename);
        ui.updateUserProfileImage();
    }
}

The new version describes what we do and reads pretty like the prose description I gave further above, without any technical detail on how the things are done. Many of its responsibilities were moved to other places: * all details related to the user interface and user interaction are moved into a separate object (ui) * all details related to user settings are moved into another separate object (userSettings)

Let’s have a closer look on how the file selection is further implemented:

public class UI {
    // ...

    public Optional<String> selectUserProfileImageFile() {
        return uiFramework.selectFileToRead("Select Image", filesystem.getImagesFolder(), SUPPORTED_IMAGE_FORMATS);
    }

    // ...
}

public class UIFramework {
    private Shell rootShell;

    // ...

    public Optional<String> selectFileToRead(String title, String filterPath, String[] filterExtensions) {
        FileDialog dialog = new FileDialog(rootShell, SWT.OPEN);
        dialog.setText(title);
        dialog.setFilterPath(filterPath);
        dialog.setFilterExtensions(filterExtensions);
        String selectedImageFilename = dialog.open();
        return Optional.ofNullable(selectedImageFilename);
    }

    // ...
}

The method selectUserProfileImageFile contains all the details (title, file formats, etc.) that are specific to selecting a user profile image file. On the other hand, selectFileToRead knows how to show a file dialog in order to open a file for reading. Use case specific details are separated from technical details that depend on the underlying UI framework.

Doing layering well requires discipline, because it is not easily enforced by our languages and tools out of the box. It is the developer’s responsibility to adhere to the rules which define the layers, maintain the separation of the different layers and to introduce new layers as soon as they are needed. There are in fact tools, that can help to find violations of the rules (e.g. JDepend to check dependencies), but anyway, the rules have to be defined first by the developers.

The reward of this effort is a codebase with a good separation of responsibilities and a clearer dependency structure. If you look into a certain use case, you have the choice on the level of detail you want to see. Code related to technical aspects tends to move into the lower layers and stabilizes very soon (as long as you do not change the underlying technology) while the business logic resides in the upper layers.

Business logic tends to be ugly and complex, because it takes care of all the details, shortcuts and special cases coming “real life”. Business logic is also inherently instable, because “real life” changes. Layering helps to separate this instable ugly part and keep the rest of your codebase nice and clean. Nothing should depend on ugly code.

But layering also comes with a price: the code related to one single use case is distributed in small units over several layers. If you want to understand in the very detail what is going on, you need to look at all the layers. It is easy to loose track when everything is spread over several classes, packages and modules.

Dependency Injection

Layering separates code with different levels of abstraction and helps to clarify dependencies. The next step is to loosen the coupling between different parts in your codebase. This is where dependency injection (DI) comes onto the stage. But first a little bit of code to ease the discussion with some meat:

public class UserProfileController {
    private UI ui;
    private UserSettings userSettings;

    public UserProfileController() {
        ui = new UI();
        userSettings = new UserSettings();
    }

    public void selectUserProfileImage() {
        Optional<String> selectedImageFilename = ui.selectUserProfileImageFile();
        if (selectedImageFilename.isPresent()) {
            userSettings.setProfileImage(selectedImageFilename);
            ui.updateUserProfileImage();
        }
    }
}

The class UserProfileController directly depends on the classes UI and UserSettings and even worse, creates instances of those classes. Any change to one of those classes may also have impact on UserProfileController. If you want to replace one of the classes with a new implementation, this leads to a change in UserProfileController and all other places that instantiate the class.

Using dependency injection to solve this two issues, we will have to change two things: 1. UserProfileController should get the instances of UI and UserSettings from the outside (the actual injection), 2. UserProfileController should talk to interfaces instead of concrete classes.

Let’s discuss both measures in more detail:

Inject dependencies

The injection of dependencies can be implemented in three different ways: 1. injection through the constructor 2. injection through a setter 3. injection using whatever magic stemming from a decent DI framework

As usual, there are pros and cons for each variant, and you have to choose wisely in practice. Using (1) or (2), it is still easily comprehensible where the injected instances are coming from and at which point in the application life-cycle they are injected. On the other hand, this means that there is still tight coupling in the codebase between the parts where the injected classes are used and where they are instantiated. It also usually means the need to recompile the software if anything is changed in the dependency configuration.

A dependency injection framework allows much looser coupling in the whole codebase. The dependency configuration is described in a separate file or using a set of conventions. Changing the dependency structure can in the best case be done at runtime or means the need for a restart of the application in the worst case. The downside of this flexibility is, that it is less obvious what is going on in the background. There is a lot of glue provided by the framework between the units from your codebase, which makes it harder to reason about the behavior of your code. To be able to understand your application, it is mandatory to also understand the dependency injection framework to a certain degree.

Separate contract from implementation

The second step to apply dependency injection is to separate the contract from the implementation. UserProfileController should not talk directly to UI or UserSettings, but use an interface instead. This allows to easily replace either implementation, without any impact on UserProfileController. Using a setter or a DI framework, this can even happen at runtime.

Going one step further, UserProfileController should use interfaces, that contain only the part of UI and UserSettings which is needed by UserProfileController. This is also called the Interface Segregation Principle (ISP).

Dependency injections puts the wiring of objects into one dedicated place. This way it helps to reduce dependencies and coupling in your codebase. It comes with the price that sometimes it is less obvious, who the actual actors are in a certain use case.

Vertical slicing

Using a layered architecture, it is still possible to have strong coupling between the distinct features of your software. Layers just order the dependencies on the axis of abstraction. It is very easy to add dependencies within a layer or to a lower layer between parts that originally belonged to different features.

Layers with strongly coupled features

In the image above, only feature A is independent from the rest of the features. The features B-D are coupled together and may not be changed independently.

To make those dependencies explicit, you need to cut the software also into vertical slices. This makes parts in your codebase visible that, although they belong to different features, share a dependency to some common functionality. Those common parts can then be extracted, you define an interface and so on.

Another name for these vertical slices of functionality related to a certain feature with well-defined interfaces for interaction is “modules” – a very old concept in software engineering. A module has a certain responsibility and may have dependencies to other modules. With regard to layers, modules are orthogonal, i.e. a module contains all the code of a feature, regardless of the layer that the code belongs to.

While the concept of modules has, like each older concept in the software trade, undergone several re-definitions over time, there are currently several ways to use modules in practice.

In languages like Java, a very simple approach is to put everything that belongs to a certain module into one package. You can further use the concept of “package private” visibility, to distinguish the public interface of the module from its inner parts. This can all be done with bare language features, there is no need for a special module framework.

Since we are now in 2016 and some years have gone by since the discovery of modular programming, there are also more sophisticated incarnations of the module concept nowadays. Frameworks like OSGi provide much more possibilities to define and handle modules. You can have different modules that provide the same services, you can load and replace modules at runtime, modules are versioned and different versions of the same module can be present at the same time. You have control on a very fine-grained level over the public interface and the dependencies of each module. With a modern module system, you can even go so far that the actual actors of your software are determined at runtime, without the need to do any explicit wiring at development time.

All this allows you to keep different parts of your software independent of each other and keep this independence stable over time. This is key to be able to replace or remove a certain part of your codebase without too much impact on the rest. You need this ability in order to evolve features independently whenever it is needed.

The secret step to save your software from creeping death

All the things discussed so far can be called common knowledge in software engineering; I did not present any new insights until now. But all of the above techniques need to be applied scrupulously precise without any exception to be able to do the one vital step that keeps your software fit for all eventualities.

A knightmare on Wall Street

To illustrate the importance of this last step, let me tell you a story that I found on Doug Seven’s blog about a high-volume day-trading firm called Knight Capital Group.

In 2012 Knight was the largest trader in US equities with market share of around 17% on each the NYSE and NASDAQ. Knight’s Electronic Trading Group managed an average daily trading volume of more than 3.3 billion trades daily, trading over 21 billion dollars daily.

On July 31, 2012 Knight had approximately $365 million in cash and equivalents.

The NYSE decided to launch a new Retail Liquidity Program on August 1st, 2012. Hence Knight needed to adapt a part of their electronic trading software called SMARS. This update was intended to replace an 8 year old module called “Power Peg”, which was actually not used any more since years, but was still in the codebase. To activate the new SMARS module, they decided to reuse the switch witch formerly was used to activate the old “Power Peg” module.

The deployment of the software was done manually – accidentally only on 4 of 5 nodes. This meant, on one node, the old “Power Peg” code was now active. Over the years one small but vital part of the “Power Peg” module has been removed: the check which stopped the processing of an order as soon as the desired number of shares was bought or sold.

Long story short: it took the one node with the revived “Power Peg” module only 45 minutes to burn 460 million dollars (remember, they had only $365 million cash) and boost the Knight Capital Group out of business.

The Goal

Being aware of responsibilities and dependencies, separating features into modules and functionality into layers of different abstraction, applying the SOLID principles and making all this part of your daily routine; all this serves one single goal: being able to delete obsolete code when time has come.

Ok. Deleting code is hard. You have spent weeks, month, even years within certain parts of your codebase, often more time per day than with your family. These places feel comfortable, familiar, intimate. But at some point in time, reality changes, and you will come to the conclusion that the concepts you are using do not work anymore in the changed reality. Whatever you try, you are not able to bend the new reality around your old concepts.

This is the point where it is time to say goodbye. You need to replace the code with a new implementation. And replacing here means: delete the old code, add a new implementation. The good news are: when your codebase is in good shape and all the affected parts are already separated, odds are good that you can keep a much bigger amount of your beloved code.

Deleting obsolete code reduces the size of your codebase and lowers the impact of the constant growth. Less code always means less bugs. Deleting obsolete code also eliminates the possibility that old code is accidentally revived and bites you in the back when you least expect it.

So the secret to save your software from the creeping death of inmaintainability is easy:

  • Be aware of responsibilities and dependencies,
  • whenever you make a decision, also think about how to revert it,
  • keep separate things separate,
  • DELETE OBSOLETE CODE.

References

null

The Problem

References to objects may be empty, i.e. they may not reference a particular object at all. In this case, their value is null. Whenever you want to access an object using such a reference, you have to make sure, that the reference is not null. Otherwise Java punishes every access to null with a NullPointerException.

There are different ways to take care of the null case in Java. But we start a discussion, we first need a small example to talk about: The method getUserData(int userId): UserData fetches the data of the user with the given userId from a data store. In case there is no user data for the given userId, it returns null.

In principle, you have two alternatives: either handle the null value, or avoid it. Sometimes you can choose yourself, other times somebody else already decided for you (e.g. when you use 3rd party code that returns null values from methods).

Handling null

Check or Catch

The first step to handle a null value is to either guard it with a check for null, or to catch the NullPointerException that will be thrown when accessing the null value.

// without a guard, prone to throw NPEs
if ("Hans".equals(getUserData(currentUserId).getName())) {
    //...
}

// using a null check as guard
UserData userData = getUserData(currentUserId);
if (userData != null)
    if ("Hans".equals(userData.getName())) {
        //...
    } 
} else {
    // handle the case that there is no user data for currentUserId
}

// catching the NPE
try {
    if ("Hans".equals(getUserData(currentUserId).getName())) {
        //...
    }
} catch (NullPointerException e) {
    // it is not that obvious, but only getUserData(...) could lead to a NullPointerException here
    // handle the case that there is no user data for currentUserId
}

Which one you choose depends on the situation. Using the first approach makes it easier to make explicit what the null value means. Imagine you have to make several calls that can return null in the same expression. In this case it is not clear which one threw the exception that you are handling:

// catching the NPEs of multiple calls
try {
    if ("Hans".equals(getUserData(userId1).getName()) || "Hans".equals(getUserData(userId2).getName())) {
        //...
    }
} catch (NullPointerException e) {
    // which user ID could not be found?
}

Null Annotations + Static Analysis

Finding all locations where you need to handle a null value is tedious, and doing it by hand, you will never be sure if you found all locations. Fortunately there are tools that support you. All current tools for static code analysis provide a possibility to check for unhandled null values. But those tools need a little bit of help, since they cannot fully automatically infer the coder’s intention (may a value be null or not). Usually the tools use some kind of annotation mechanism that you can use to give a hint about your intentions.

Unfortunately there is no standard for this annotations. There even exist two JSRs that are related to this topic. JSR-305 originates in the FindBugs community. It tries to standardize the annotations that FindBugs provides since years. JSR-308 describes a more generic extension of the Java annotations itself. Eclipse provides annotations based on this JSR that allow a null analysis within the IDE, done by the Eclipse compiler. There is a good overview on Stackoverflow about the current state of affairs regarding null annotations in Java.

In the end, this all means that it depends on your tool chain, which annotations you can use.

Avoiding null

Throw a Meaningful Exception Instead of Returning Null

Instead of returning null, the method could also throw a more meaningful exception. This would at least make the purpose of the catch block more obvious.

// catching a meaningful exception
try {
    if ("Hans".equals(getUserData(currentUserId).getName())) {
        //...
    }
} catch (UserNotFoundException e) {
    // handle the case that there is no user data for currentUserId
}

It does not solve the problem of knowing which of multiple calls in the same expression threw the exception. Therefor the exception should provide additional information to identify the cause.

// catching a meaningful exception
try {
    if ("Hans".equals(getUserData(userId1).getName()) || "Hans".equals(getUserData(userId2).getName())) {
        //...
    }
} catch (UserNotFoundException e) {
    String missingUserId = e.getUserId(); // <-- this user id was not found
    // handle the case that there is no user data missingUserId
}

Null Object

A null object is an immutable instance of a class or interface that behaves in a natural and harmless way. This means, you can use this object like other instances, but your interaction with it will have no effect. In our example, the null object might return the empty string as name:

public static final UserData NULL = new UserData() {
    public String getName() {
        return "";
    }
};

Using this, our example would look like the following:

UserData userData = getUserData(currentUserId);
if ("Hans".equals(userData.getName())) {
    //...
} 

We can use the userData object safely here without a check for null. Only if we want to handle the case that there is no userData available for the currentUserId, we need to check if userData == UserData.NULL.

Usually you have only one static instance that is used as NULL object. This way you can easily identify the null object, if you need to. As a variation, you can have several different static instances, if there are different notions of ‘natural behavior’ (in our example it might make sense to have a SUPERUSER object, for example).

Option

The Option Type construct combines the idea behind Null Object with polymorphism and generics. It provides a container for a given type T, that may or may not contain an instance of T. You could ask the container if it contains an instance, and you can ask the container for the instance it contains. This construct is available in many languages in one or the other way. Many of those languages provide two different subclasses for the case that there is something and for the case that there is nothing.

Option Type

Since Java 1.8, the Java runtime library provides the type java.util.Optional, without the usual specialized subclasses. In our example this would look like the following:

Optional<UserData> userData = getUserData(userId);
if (userData.isPresent()) {
    if ("Hans".equals(userData.get().getName())) {
        //...
    }
} else {
    // handle the case that there is no user data for currentUserId
}

In comparison to all other variants, here the code itself makes it pretty obvious that user data could not be available and we are handling this case also. Using Optional as a return type also ensures that you do not miss a case accidentally. If you don’t handle the null case or do not even check with isPresent(), then you do this by intent.

Conclusion

There are so many ways to prevent a NullPointerException. Please use them!

Oomph vs. Groundhog Day

Ahh, summertime: sunny weather, lots of barbecue and new a Eclipse release. As every year, this also means finding out again, which useful plug-ins you did install during the course of the last year, which tweaks you did to your Eclipse installation. All this just in order to reproduce your development environment based on the latest and greatest features from Mars. Sure, there is this new installer based on Oomph, that does the downloading and all this crazy stuff to prepare your workspace. Sadly your beloved little utility plug-ins are not part of the predefined Eclipse products.

But wait a minute! When you switch the installer to “Advanced mode”, there is this <User Products> node.

User Products node

What if it is possible to define your own product, that contains all this useful plug-ins that help you every day?

Create your own product

As a starting point, you can use the product definition for the Oomph installer itself. Download the file and put it into a “General Project” in your workspace.

Oomph Installer product

If not already done, install the latest Oomph version into your IDE. This allows you to easily edit the Oomph related files with the Setup Editor. You should also get the setups project into your workspace. This contains a lot of examples and material that you can use for your own product definition.

To add your plug-ins, add the required P2 repositories and the feature IDs (as requirement) under the P2 Director (Eclipse Installer) node. Since the Oomph Installer product is not really a useful base for an IDE product, you should also add the P2 information from your favorite EPP package. When you open the file setups/org.eclipse.setup in the Setup Editor, you can easily copy and paste this information to your product defintion.

My Customized Eclipse Committer Package

Now you can drag your setup file onto the <User Products> node and voilá: install your product with all the handy helpers already prepared for you.

Tune to 11

Tune to 11

This is great so far. But it gets even better. Of course, you’re not restricted to installing plug-ins. You can do all the magic, that Oomph provides. You find a lot of information about what is possible in the Oomph Authoring Guide.

What absolutely made my day, and what I was missing the last 13 years: you can set the preference /instance/org.eclipse.ui/SHOW_TEXT_ON_PERSPECTIVE_BAR to false.

Thank you Eike and Ed for this one! I owe you a beer.

One more thing

If you got so far, there is one more thing you should try. Put your setup file online (e.g. like I did on github). Now remove the entry under the <User Products> node and drag the URL to your online version there instead. You have now one central definition of your personal development environment, that you can use on several computers (e.g. or for all people in your team). If you change this definition, you can apply the changes with just on click on “Perform Setup Tasks…” on all your machines.

Processing Data in Tree Structures

In Vex there exist several tree structures, the most prominent are the XML DOM (semantics) and the box model (representation). Both structures consist of a heterogeneous set of node types which serve different purposes. The one thing that all nodes in a tree have in common is the responsibility to form the tree structure itself. Most tasks that are not related to the structure only involve a subset of node types.

The simplest thing that might work

Let’s start with a simple example: you have an arbitrary node and want to get the text it contains. This obviously works only for nodes of type IText because all other nodes do not contain any text. A very simple approach would be the use of instanceof:

public String getNodeText(INode node) {
    if (node instanceof IText) {
        return ((IText) node).getText();
    } else {
        return "";
    }  
}

This approach is using reflection, which has kind of a bad reputation, but in the end, it does what it should do. For this simple use case it is an appropriate solution.

There is one thing you should keep in mind: when using instanceof you need discipline. Since it is a very generic mechanism, it is easy to mix up different concerns in one place. Imagine a class TableCellNode which implements several interfaces INode, ITableCell and ITextContainer. There is nothing that prevents you from adding just another little if node instanceof... to handle things concerning the node structure in the same breath with things related to table cells or text containers. Clearly a “crack in the pane”.

Polymorphism

Going to the other extreme, you can use polymorphism. This means, there is this generic getNodeText method with a contract, that defines its external behavior. Each and every node type then has to implement getNodeText according to the contract. In our example, most nodes will just return the empty String.

Polymorphism is a very strong mechanism that comes with object orientation. It provides a way to diversify behavior by setting up a custom type system. The key is, that all members of this type system share the same responsibilities, but fulfill them in a different way.

In our example this would mean that each node type has a useful textual representation. But this is not true. The responsibility shared among node types is to build the tree structure. Only one type, IText, has the special feature to provide textual content. Code related to the textual content will only be interested in nodes of type IText. Putting the getNodeText method into the base of the node types breaks the SRP and is not the best solution for our problem. But if the required functionality was related to the tree structure (the common responsibility of all nodes), polymorphism would have been a proper way to go.

The Visitor pattern

We have seen so far that you can either implement the functionality inside your types using polymorphism, or outside using some kind of type checking mechanism like instanceof. What you do highly depends if the functionality you want to implement affects all of your node types, or only some of them.

Another, more object oriented approach to do type checking is the well known Visitor pattern. It allows you, to implement type specific behavior in dedicated methods. Ideally you implement a visitor class for one single purpose. So instead of adding more and more methods with different responsibilities to each node types, you get more and more visitor classes. Each one contains only the type specific behavior related to one responsibility.

How would our example look like, using the Visitor pattern? As a first step, we will implement it as simple as possible, then refine this solution step by step:

public String getNodeText(INode node) {
    final String[] nodeText = new String[1];
    nodeText[0] = "";
    node.accept(new INodeVisitor() {
        public void visit(IText text) {
            nodeText[0] = text.getText();
        }
        // ... here follows one empty visit method for each other node type
    });
    return nodeText[0];
}

Wow. Not really simple, isn’t it? Because we are using an anonymous implementation of INodeVisitor, we have to use a common trick to get information out of the anonymous class into the getNodeText method. This adds clutter. And we have a lot of empty method implementations (which I left out), that add even more clutter and obfuscate the original purpose of the method.

We will attack the superfluous empty visit methods first. To get rid of the, we just create an implementation of INodeVisitor that contains only empty implementations of all visit methods:

public class BaseNodeVisitor implements INodeVisitor {
    public void visit(IText text) {}
    // ... here follows one empty visit method for each other node type
}

This class is then used to derive an anonymous subclass in place of the anonymous implementation of INodeVisitor:

public String getNodeText(INode node) {
    final String[] nodeText = new String[1];
    nodeText[0] = "";
    node.accept(new BaseNodeVisitor() {
        @Override
        public void visit(IText text) {
            nodeText[0] = text.getText();
        }
    });
    return nodeText[0];
}

That was easy. But how to get rid of this array? Therefor we have to extend the vanilla visitor implementation a little bit. We need an interface that provides methods which return a result:

public interface INodeVisitorWithResult<T> {
    T visit(IText text);
    // ... here follows one visit method for each other node type
}

Of course to be able to use this, INode needs an accept method which also returns the result of the visitor, exemplary implemented for IText:

public <T> T accept(INodeVisitorWithResult<T> visitor) {
    return visitor.visit(this);
}

Equipped with this and a base implementation we can now remove the tricky array from our solution:

public String getNodeText(INode node) {
    return node.accept(new BaseNodeVisitorWithResult<String>() {
        @Override
        public void visit(IText text) {
            return text.getText();
        }
    });
}

So that’s it. Surely more complicated than instanceof. You need to understand the Visitor pattern, you need to understand anonymous implementation in Java, you need to know about the base implementation of your visitor interface. In exchange you get all the amenities of Java’s static type system. For example, with support of your IDE, you can easily find all places where type specific behavior is implemented. Casting is not necessary, because Java’s dynamic dispatching is taking care of routing your request to the right visit method.

Traversal

Traversal is a very common set of functionality needed when working with tree structures. Depending on the task at hands there are various ways to traverse a tree: breadth-first, depth-first in pre-order, in-order, or post-order, and a lot more. The Visitor pattern allows you to implement a base class that provides the generic traversal functionality you need (e.g. depth-first pre-order). To put it in use, you then just need to subclass it and implement the type specific behaviour by overriding the visit methods.

For a detailed example, see DepthFirstTraversal and how it is used in various locations to collect information from Vex’s box model.

switch..case

The well known alternative to the Visitor pattern is using switch..case in combination with a set of constants representing the possible types. For example the W3C’s DOM supports this kind of type checking:

public String getNodeText(org.w3c.dom.Node node) {
    switch node.getNodeType() {
    case Node.TEXT_NODE:
        return ((Text) node).getWholeText();
    default:
        return "";
    }
}

Here you need casting again, because, except the use of reflection, we are technically very close to instanceof. If you use an enum instead of int to define the type constants, you can make sure that nothing gets mixed up with implementing type specific behaviour for your polymorphic node type.

switch..case vs. Visitor

switch..case is much easier to understand than the Visitor pattern. You don’t need the anonymous implementations and all this small Visitor classes. The type constants replicate the polymorphic type system. This is, on the one hand, duplicate information (in the sense of DRY). On the other hand you get more flexibility. The Visitor pattern works only for the leaf types of type hierarchies. You cannot have a visit method for an intermediate type T and at the same time visit methods for subclasses of T. With type constants, any combination is possible.

Visitor instances are objects while type specific behaviour using switch..case (or instanceof) is usually implemented in form of a method. This means a Visitor instance is easier to combine with other patterns (like Chain of Responsibility or Command) to build behaviour dynamically at runtime. Visitor objects are also able to carry state information, if your algorithm requires this.

Choose wisely

As usual in software development, you should not rely on a golden hammer. Use the right tool instead. Having some alternatives and knowing their pros and cons always makes your life easier.

How the Cursor Learned to Move

While implementing the new box model for Vex, I came to the point, where I needed a representation for the cursor. I started with a very simple class that knew how to draw the cursor at a given offset. That’s easy to understand, one simple responsibility in one class: SRP check!

But then there happend what always happens when you have a simple and elegant solution in place: there was a new requirement: I needed the ability to move the cursor. Just add that damn feature

The first idea could be to add new methods to Cursor in order to extend Cursor‘s behaviour. The cursor should move, then let’s add move methods for that.

I know this is too easy and you don’t want to go for this solution. But I also know that there was a time where this would have been the only solution I would have seen. So I will at least mention here why it is not the solution to go for, just for the case.

Moving is a whole new responsibility for the Cursor class, that has nothing in common with the already existing responsibility of painting. And, there are a lot of different kinds of movement: left/right, up/down, page up/down, one word forward/backward, just to name a few.

Adding just new methods for each distinct movement will clutter the small elegant Cursor class. It will make the class harder to understand. It will lower the inhibition to add further features, which will make the class even harder to understand and in the end we have a Big Ball of Mud, a God Class or however you would call that mess.

There has to be a better solution.

Separate responsibilities

The SRP and “Composition over Inheritance” lead to another approach: put all the movement implementations into a separate class that takes the responsibilty for moving the cursor. This will leave the Cursor class as it is, isolating it against changes that are related to movement, which also makes the OCP happy – at least in this regard. So let’s try it and call this class CursorMovement for the further discussion.

When starting with the implementation of CursorMovement, it is clear very soon, that this approach also has some flaws. The different kinds of movement need different information about the current cursor position:

For left/right it is sufficient to know the current offset, because moving left or right is just a matter of decreasing or increasing the offset.

Moving the cursor in vertical direction requires detailed information about the spatial relation of boxes and the current graphical representation of the cursor. Having the implementation of the movement outside of the Cursor class means that this information somehow has to be made available through Cursor‘s public interface. This leads to a very tight coupling between Cursor and the CursorMovement class.

The need for access to the graphical representation revealed another flaw that was already there in the first approach. Information provided by the graphic framework (e.g. font sizes, text length in pixels) is only available during painting (see Double Buffering in SWT for more details), but the cursor movement is triggered by a key stroke – totally independent of the painting. To solve this in a clean way, vertical movement has to be postponed until the next paint event.

Content-related movement (e.g. one word forward/backward) requires information about the textual content of the document, totally independent of the graphical representation. This adds another dependency from CursorMovement to a new, so far unrelated entity.

All in all there is no cohesion in CursorMovement and no reuse of code. The class simply piles up unrelated methods and dependencies. We had this already. Together with the unsolved temporal dependency between the graphical information and the paint event, we can come to the conclusion that this approach is also not suitable.

At this point there are a lot of indicators that something is wrong, but it is difficult to see the way out of this. Some impulse from outside might be helpful.

Solutions have to be simple in order to work

Rich Hickey’s talk “Simple Made Easy” gave me this impulse: queues are simple and a great way to do temporal decoupling. This is the missing part in the puzzle. Now the solution is really straight forward:

  • We put each single move into its own class, implementing a simple functional interface (ICursorMove).
  • When triggered by the key strokes, those move instances are added to a queue in Cursor.
  • After adding the move to the queue, also a redraw of the widget is triggered.
  • In its paint method, Cursor first applies the queued moves and then does the actual painting.

This use of (some variant of) the command pattern in conjunction with the queue solved most of the problems:

  • The access to internal data of the Cursor class is restricted to the call of the execution method of the move implementations. No need to provide data through the public interface of Cursor.
  • Each move implementation can have only the dependencies that are needed. Dependencies of a certain kind of move are more explicit.
  • There is no single class that piles up unrelated move implementations and dependencies.
  • Moves are executed within the paint handler, this resolves the temporal dependency between the move and the availability of information from the graphic framework.
  • Adding new moves is just a matter of adding new classes, no changes in existing classes are required. It is even possible to have an extension point which allows other bundles to provide special move implementations for certain document structures (e.g. semantic tables).

Conclusion

It really pays off to know about clean code development. If you listen to your code and you are able to understand the signs, it will always show you the way to a better solution.

If you interested in more details about the implementation, just have a look into the package org.eclipse.vex.core.internal.cursor in Vex’s git repository.

Double Buffering in SWT

To draw the contents of a custom widget in SWT, you have to implement a PaintListener. The paint method is the only place where you have access to the widget’s GC (aka “Graphic Context”), which provides the actual drawing methods.Paint events are handled in the Display thread. This means, the UI is not responding while you are drawing. If you do complex calculations while drawing, this results in flickering.

To prevent this flickering and to keep the UI responsive, the usual approach is to use double buffering: you use two buffers, one is used for drawing and one is visible. When you are finished with drawing a complete frame, you swap the two buffers: the buffer for drawing is now visible and vice versa.

In SWT, you don’t have an explicit buffer you are working on, but you have the GC which provides the methods to manipulate the contents of the buffer. You get the GC instance for your widget visible on the screen as part of the PaintEvent that is provided to the paint callback method. To obtain an independent GC for drawing, you have to do a little trick: create an Image with the size of your widget. For this Image, you can create a GC instance which is usable independently of the UI thread.

Putting all pieces together, you use two Image instances which are used alternating for drawing and showing. The paint callback method just draws the one image that is currently used for showing to the GC of the widget.

SWT snippet 48 shows a basic version of this approach. For Vex, I started to implement a more gerneral version, applicable to Scrollable instances, that uses a dedicated rendering thread for drawing: the DoubleBufferedRenderer.

DoubleBufferedRenderer

The DoubleBufferedRenderer handles the paint events for an instance of Scrollable using double buffering as described above. It allows you to schedule a list of IRenderStep instances, a simple functional interface to implement the distinct steps of your drawing logic, that depend on access to the GC.

Internally, the execution of the render steps happens in a dedicated rendering thread. When all steps are executed, a redraw of the widget is triggered. The finished Image is then used in the paint callback method until a new image is ready.

Usage

Example: When the control is resized, we have to layout the content and paint it. Therefor we schedule two IRenderStep instances:

renderer.schedule(layoutContent(), paintContent());

To make it more readible, I put the creation of the IRenderStep into factory methods, which return an anonymous implementation:

private IRenderStep layoutContent() {
   return new IRenderStep() {
     public void render(final Graphics graphics) {
       rootBox.layout(graphics);
       cursor.reconcile(graphics);
       updateVerticalBar();
    }
  };
}

The anonymous class looks cumbersome. In Java 8 this can be expressed in a much more elegant way, but unfortunately the Eclipse platform is still on Java 6.

For more examples of how DoubleBufferedRenderer can be used, you should have a look at the implementation of BoxWidget.

Outlook

This all is still work in progress. For example the schedule method does not queue the lists of IRenderStep instances, it just knows about the currently executed list of steps and the following list of steps. Each call to schedule overwrites the latter.

Also the usage of threads is still very ineffective. If you have multiple widgets, each using its own instance of DoubleBufferedRenderer, each one will run its own thread. Using a thread pool or more sophisticated techniques will help here.

If you still want to give it a try, there is one dependency to Vex that you have to be aware of: IRenderStep provides an instance of Graphics, an abstraction of the graphics framework in use. For use outside of Vex, you can just replace this with an instance of GC.

The Maintenance Analogy

In engineering, maintenance means to tear a machine apart, clean it, replace broken parts, oil it and put everything together as it was before. Maintenance is needed to ensure the longevity, the quality of service and also the safety of a machine.

Some people say, that this is not needed for software. Bits do not wear out or break and bits do not need recurrent lubrication. You know what: they are right. This is not what happens in software development. But, as with many other analogies in software development, it is not that easy. You have to think about the goal of maintenance instead: make sure that the software is useful for a long time. Useful means in terms of software, that you are able to change the behavior of the software, either to fix bugs, or to react to changes in the requirements of the software.

Some people say, if you practice clean code development with due diligence, the software is always in shape for changes. But concepts, structures and algorithms wear out over time. When requirements change, the existing implementation does not fit anymore. The many small changes that accumulate over time dillute the original architecture. While the code in detail might be in perfect clean shape, the bigger picture will show some flaws after a certain period.

Some day you will come into the situation where you have to take apart your application and replace some of its parts with better fitting implementations. This is for sure. The problem most of the time is to realize this situation and actually do what is necessary instead of taking a shortcut. And this is the secret of successful software maintenance. Its not about fixing bugs and adding new features until a single change is more expensive than writing everything from scratch. Its mainly about doing those changes that keep the big picture in shape. Its about being able to fix bugs and add new features forever.

By the way: having clean code makes maintenance much much easier, because you only have to take care of the big picture’s shape. If you don’t do clean code development, start now! No really, do it! Now!