All posts RSS Feed

Deep Work - A summary

Sun, 19 Feb 2023 00:00:00 GMT

Several months ago, I read the book Deep Work, by Cal Newport and wanted to write a summary of the main takeaways I found within it. Originally I planned to write this shortly after reading the book, but life happens… On a positive note, it’s given me a chance to see what parts have stuck with me after finishing the book many months ago.

There are many things in Deep Work that I agree with or have made improvements to my work; however, there are points that I disagree with or do not place as much importance upon.

Before I start, remember this is a summary, my summary, more precisely. I’m not going to replace the act of reading the book itself. I’ll highlight parts that I found interesting and beneficial. I’m not trying to advertise the book, I swear, but if you need more thorough explanations after reading this, then you’ll probably have to buy the book!

Deep Work Is Valuable

The book’s first chapter explains why the author believes deep work is valuable.

Below are some quotes from this chapter:

“The ability to quickly master hard things.” (page 29)

“High-Quality Work Produced = (Time Spent) x (Intensity of Focus) (page 40)

It all sounds appealing so far.

Deep Work Is Rare

Life is full of distractions: emails, instant messaging and social media, all negatively affecting our ability to focus. These points aren’t entirely novel; you’ve probably thought this yourself or overheard others discussing it; however, the thought that quick and easy access to information could reduce your capability to plan and prepare was new to me.

“The Principle of Least Resistance: In a business setting, without clear feedback on the impact of various behaviors to the bottom line, we will tend toward behaviors that are easiest in the moment.” (page 58)

“If you couldn’t count on this quick response time you’d instead have to do more advance planning for your work, be more organized, and be prepared to put things aside for a while and turn your attention elsewhere while waiting for what you requested.” (page 58)

Personally, I used to have a tendency to run head first into my work and then ask questions about the first issues I faced. This solved the immediate problem, but I would inevitably have another issue and another question and so on. So why did I do it? Because it was the easiest thing to do.

Thankfully I’ve moved away from this behaviour, and these quotes collected and labelled positive habits that I’ve adopted.

Deep Work Is Meaningful

This chapter gave the most significant explanations as to why we should work deeply. A description of “Flow state”, which I’ve heard thrown around and have experienced first hand as a Software Engineer, struck a chord with me.

“The best moments usually occur when a person’s body or mind is stretched to its limits in a voluntary effort to accomplish something difficult and worthwhile.” (page 84, Mihaly Csikszentmihalyi)

This quote got to the core of why I felt frustrated about my current working habits and why I sought out this book. I was rarely entering a flow state anymore, and as a consequence, I was rarely enjoying my job anymore.

So I read on with vigour in search of guidance.

Rule #1 Work Deeply

The previous chapters covered the why and now the book moves onto the how.

This chapter shares information on ways to increase your focus and lower the boundary into switching into a flow state.

The author details different philosophies on deep working:

Monastic:
- Purposely being hard to reach to reduce distractions.
- Assign significant periods of time to focus, which could be weeks or months.
Bimodal:
- Separating your time, where blocks are spent in deep work, and others are open to everything else.
- Assign blocks of time to focus, which could be days to weeks.
Rhythmic:
- Create a habit of having deep work sessions.
- Generate a rhythm for this work style, so it doesn’t take energy to continue doing it.
- Assign specific times within a day to focus.
Journalistic:
- Switch into a deep work state whenever you have the time to do so.
- Hard to follow because it requires flipping a switch into flow mode.
- Requires a lot of willpower.

For me, the “Rhythmic” philosophy appealed to me the most. Especially for someone who was finding it extremely hard to focus before reading this book, the systematic process of a schedule gave me a way to hold myself accountable.

At the time of writing, several months after reading Deep Work, I have scheduled reoccurring meetings in my calendar for focusing. I pick a singular task, evict distractions and focus solely on the task until I’ve reached the end of the “focus” time. I started off scheduling 30-minute blocks and progressed to 1-hour blocks throughout the day.

I don’t always stick to my schedule. If I don’t have a task on which I can focus all my time, then I might cancel my “focus” time or accept that I can still focus on a few tasks, but I won’t complete any deep work.

This chapter also says you should be lazy. Now that’s a bold statement. Before you take it literally and stop reading here because you decided to be lazy, let me give you some extra context.

You should be lazy, in that you don’t work all the time. You need time to recharge. This recovery time will help you perform during your working hours as your mind will be refreshed and ready to work deeply. Furthermore, deep work can be draining, and if you are successfully working in that way for long periods of the day, you will eventually run out of gas. Forcing yourself to keep going might get some extra work done, but you’ll be sacrificing the energy and willpower reserved for the next day. The author also points out that the work you’re doing so late in the day is probably work that isn’t all that important. Meaning you’re burning yourself out for tasks that don’t provide much value.

Rule #2 Embrace Boredom

“The ability to concentrate intensely is a skill that must be trained.” (page 157)

“Efforts to deepen your focus will struggle if you don’t simultaneously wean your mind from a dependence of distraction.” (page 157)

As described earlier in this post and the book, life is full of distractions, and if you always submit to them, you’ll never refine your ability to truly focus.

“Don’t Take Breaks from Distraction. Instead Take Breaks from Focus.” (page 159)

For me, this works alongside my scheduled focus time, giving me a “reward” for focusing for a period of time. I give myself time to relax without feeling guilty because I know I just completed quality work (most of the time), rather than the opposite, where I feel bad while being distracted, which almost creates a cycle of distraction that can be hard to break.

This chapter details much more extreme methods of blocking time without distractions, but I didn’t really gravitate towards them.

The author also writes about trying to think deeply while you’re physically but not mentally distracted. This could be walking, running, cooking, cleaning and many more. For me, I’ve stopped listening to podcasts/music so much when walking around; instead focusing on my thoughts about work, life or whatever I currently see. This has helped me eke out some valuable insights without setting aside specific time to think about something.

Rule #3 Quit Social Media

While I agree with much of what was written in this chapter, it did not give noteworthy points that I’d like to write in this post.

Onto the next one.

Rule #4 Drain the Shallows

Similar to life being full of distractions, there is also a good chance that significant amounts of your time are spent on tasks that don’t require much thought or focus. These are low-value tasks that need to be done, but if scattered throughout the day, they provide constant distractions preventing you from sitting down and focusing on a singular piece of work.

This chapter advises you to consider how much time you spend on these sort of tasks and when you do them.

“Quantify the Depth of Every Activity” (page 228)

Once you know what tasks deliver the most value, prioritising the order you complete them and where your energy is directed becomes clearer.

That being said, sometimes these “low value” tasks might be urgent or unblock others, so figuring out the ranking between tasks isn’t always cut and dry. This book is written from the perspective of a professor (since that is the author’s profession), which doesn’t always transfer over cleanly to working professionals. For example, as a Software Engineer, I have to review my team’s code. While some reviews might seem low value to me, ignoring them negatively impacts my team, so I can’t simply stop doing them.

“Schedule Every Minute of Your Day” (page 221)

I do believe there is a benefit in scheduling more of your time. Consciously deciding what you want to do beforehand gives you a target to aim for with the time you spend, more precisely focusing your energy.

You might find that in your “unstructured” time, you don’t get so much done because you’re constantly trying to determine what to do next. With your mind continually flicking from one thought to another, stopping you from giving any of the thoughts any real attention.

I solved my problem of reviewing code mentioned a moment ago by blocking off time to do them. Even if some of them are “low value”, grouping them together and giving direction to my time improves the overall quality of the reviews I give. Even if there is a context switch between each review, staying in the mindset of “I’m doing reviews” is extremely beneficial.

Now, I’m not saying to schedule every minute of your day. However, being more aware of how you spend your time can be enlightening.

“Do More Work When You Send or Reply to E-mails” (page 248)

I’ve been putting more effort into providing well-written emails by including enough information and the next steps or outcomes needed from the recipients increasing the likelihood they reply with precise emails themselves.

If I wrote emails missing information, they might have to contact me again to ask for more context. If I wrote emails not specifying the outcome I wanted, the feedback I’d receive might be undirected and not deep enough in the desired areas. Both of these lead to each side investing more energy into reaching the original purpose of the email.

“What is the project represented by this message, and what is the most efficient (in terms of messages generated) process for bringing this project too a successful conclusion?” (page 249)

The content in a “good” email will always vary, but the point above should give you a solid foundation to work from.

My Main takeaway

My main takeaway from reading Deep Work is that I must be deliberate in my decisions.

Hoping to move away from being constantly distracted, aimless and unmotivated and into someone that can focus deeply in a directed manner is pointless. Instead, I have to consciously work to address these flaws.

Scheduling more of my time as “focus” time has worked wonders for me.

I’m getting vast amounts of quality work done during these periods, I’m more easily entering a flow state, and most importantly, I’m enjoying my work again.

I can’t promise that the same changes will work for you, but it might be worth a try.

This has been my summary of Deep Work, by Cal Newport. I hope what I’ve written is useful, giving you some pointers; however, I recommend reading the book itself.

Ktor WebSockets

Fri, 19 Aug 2022 00:00:00 GMT

Ktor provides a WebSocket plugin to allow your applications to push real-time data between backend servers and clients over HTTP.

The code in this post was tested against Ktor 2.1.0 and 2.0.3.

Including WebSockets

To include WebSockets in your Ktor server:

Include the ktor-server-websockets dependency:

implementation "io.ktor:ktor-server-websockets:$ktor_version"

Install the WebSockets plugin:

fun Application.module() {
  install(WebSockets)
}

Add a webSocket route:

fun Application.module() {
  install(WebSockets)
  routing { people(personRepository) }
}

fun Routing.people(personRepository: PersonRepository) {
  webSocket {
    personRepository.updates.collect { (id, person) ->
      when (person) {
        null -> outgoing.send(Frame.Text("Deleted person [$id]"))
        else -> outgoing.send(Frame.Text("Saved person [$id] $person"))
      }
    }
  }
}

Each new WebSocket connection invokes the webSocket handler.

The outgoing property allows you to send data (as Frames) to the consuming client.

Those steps will cover the bare minimum needed to leverage WebSockets in your Ktor server; however, it doesn’t tell you how to use Ktor’s implementation, which I’ll go through next.

Writing a WebSocket handler

As you saw above, writing a WebSocket handler requires calling the webSocket extension function inside a Route (which Routing is).

The previous example did not include its own path; if you want to have one, then you can write:

webSocket("/updates") {
  ...
}

webSocket’s signature is:

public fun Route.webSocket(
  protocol: String? = null, 
  handler: suspend DefaultWebSocketServerSession.() -> Unit
)

A few things to point out:

protocol is the path I mentioned before (I find the naming odd myself).
The handler is executed when a WebSocket connection is initiated, giving you a DefaultWebSocketServerSession to handle the back and forth between the client and server.
handler is a suspendable function so that you can use coroutines or Flows within it.

Outgoing data

To send data from the WebSocket connection to the initiating client, you should call send on DefaultWebSocketServerSession’s outgoing property.

webSocket {
    personRepository.updates.collect { (id, person) ->
    when (person) {
      null -> outgoing.send(Frame.Text("Deleted person [$id]"))
      else -> outgoing.send(Frame.Text("Saved person [$id] $person"))
    }
  }
}

Sending and receiving data via WebSockets in Ktor is done via Frames. Text data is transferred over the WebSocket using Frame.Text in this example. The other Frame types are: Binary, Close, Ping and Pong.

Although it is not evident from the code, personRepository.updates is returning a hot Flow that does not complete, meaning that this socket will remain open until the client’s termination due to the call to collect.

Incoming data

To receive data from the client’s side of the WebSocket, you should interact with DefaultWebSocketServerSession.incoming.

There are a few ways you can receive data from the incoming property:

Using a for loop:

webSocket {
  for (frame in incoming) {

    (frame as? Frame.Text)?.let { text ->

      filter = when (text.readText()) {
        "saves" -> PeopleFilter.SAVES
        "deletes" -> PeopleFilter.DELETES
        else -> PeopleFilter.ALL
      }
    }
  }
}

incoming is a ReceiveChannel and therefore provides an operator fun for an Iterator, which is why you can loop over it directly using a for loop. If you wanted to write Iterator styled code, you could call incoming.receive directly.

Creating a Flow using receiveAsFlow:

webSocket {
  incoming.receiveAsFlow().collect { frame ->

    (frame as? Frame.Text)?.let { text ->

      filter = when (text.readText()) {
        "saves" -> PeopleFilter.SAVES
        "deletes" -> PeopleFilter.DELETES
        else -> PeopleFilter.ALL
      }
    }
  }
}

In both versions, ReceiveChannel.receive is called under the hood, receiving incoming data over the WebSocket connection if there is any and suspending the coroutine otherwise. The use of coroutines and suspending functions means you don’t need to worry about blocking your executing threads or managing threads yourself.

Similarly, to sending outgoing data, the WebSocket receives Frames. The examples above check that each Frame is a Text frame, then parses its data using readText.

Sending and receiving at the same time

The code below combines the incoming and outgoing data streams from above to ensure data flows in both directions at the same time:

webSocket {
  var filter = PeopleFilter.ALL

  // Calls [launchIn] to create a [Job] that receives from the [incoming] stream.
  incoming.receiveAsFlow().onEach { frame ->
    (frame as? Frame.Text)?.let { text ->
      filter = when(text.readText()) {
        "saves" -> PeopleFilter.SAVES
        "deletes" -> PeopleFilter.DELETES
        else -> PeopleFilter.ALL
      }
    }
  }.launchIn(this)

  // Collect is a terminal operation.
  personRepository.updates.collect { (id, person) ->
    when (person) {
      null -> {
        if (filter == PeopleFilter.ALL || filter == PeopleFilter.DELETES) {
          outgoing.send(Frame.Text("Deleted person [$id]"))
        }
      }
      else -> {
        if (filter == PeopleFilter.ALL || filter == PeopleFilter.SAVES) {
          outgoing.send(Frame.Text("Saved person [$id] $person"))
        }
      }
    }
  }
}

The crucial difference compared to the independent examples from before is that the incoming stream launches a Job which allows the current thread to carry on. Without this, the thread would never reach the outgoing code.

To achieve the same behaviour, you can wrap the for loop version of the incoming stream from before in a launch block.

It is important to state that something must block the webSocket handler from completing. If you fail to do this, your WebSocket will initiate correctly but will terminate instantly as your code completes. You can successfully keep your WebSocket open by calling collect (a terminal Flow operation) or using a for loop.

In the example, collect streams updates to the client, while a coroutine thread pool handles the incoming stream.

The blocking terminal call could be switched around, and the code would still work:

webSocket {
  var filter = PeopleFilter.ALL

  // The outgoing stream now launches a job.
  personRepository.updates.onEach { (id, person) ->
    when (person) {
      null -> {
        if (filter == PeopleFilter.ALL || filter == PeopleFilter.DELETES) {
          outgoing.send(Frame.Text("Deleted person [$id]"))
        }
      }
      else -> {
        if (filter == PeopleFilter.ALL || filter == PeopleFilter.SAVES) {
          outgoing.send(Frame.Text("Saved person [$id] $person"))
        }
      }
    }
  }.launchIn(this)
  
  // The incoming stream now calls [collect].
  // This terminal operation blocks [webSocket] from completing.
  incoming.receiveAsFlow().collect { frame ->
    (frame as? Frame.Text)?.let { text ->
      filter = when(text.readText()) {
        "saves" -> PeopleFilter.SAVES
        "deletes" -> PeopleFilter.DELETES
        else -> PeopleFilter.ALL
      }
    }
  }
}

Terminating a WebSocket

To cleanly terminate a WebSocket connection, we want the following to occur:

A close frame is sent to the client.
The webSocket handler stops executing.

In simpler code where for loops or Iterators are used, returning from the webSocket handler achieves clean termination:

webSocket {
  for (frame in incoming) {
    (frame as? Frame.Text)?.let { text ->
      filter = when (text.readText()) {
        "saves" -> PeopleFilter.SAVES
        "deletes" -> PeopleFilter.DELETES
        "stop" -> return@webSocket // Terminates the WebSocket.
        else -> PeopleFilter.ALL
      }
    }
  }
}

The webSocket is terminated by completing it; however, calling return can be tricky depending on the code as you cannot return to outer scopes within a lambda/function.

Placing this code into a launch block prevents the return@webSocket from compiling and requires a different solution.

In this case, you should call WebSocketSession.close to terminate the socket:

launch {
  for (frame in incoming) {
    (frame as? Frame.Text)?.let { text ->
      filter = when (text.readText()) {
        "saves" -> PeopleFilter.SAVES
        "deletes" -> PeopleFilter.DELETES
        "stop" -> {
          close() // Sends a [Frame.Close] to the client.
          return@let // Returning here lets the code compile.
        }
        else -> PeopleFilter.ALL
      }
    }
  }
}

close sends a Frame.Close over the connection causing it to terminate.

You can also trigger similar behaviour with the updates heading outbound over the WebSocket rather than inbound:

personRepository.updates.collect { (id, person) ->
  when (person) {
    null -> {
      if (filter == PeopleFilter.ALL || filter == PeopleFilter.DELETES) {
        outgoing.send(Frame.Text("Deleted person [$id]"))
        close() // Sends a [Frame.Close] to the client.
      }
    }
    else -> {
      if (filter == PeopleFilter.ALL || filter == PeopleFilter.SAVES) {
        outgoing.send(Frame.Text("Saved person [$id] $person"))
      }
    }
  }
}

Note that collect may still execute after close is called but will only do so for the next pass-through of collect, meaning that the webSocket handler only fully terminates after this point. Where the close occurs does not matter here. Without looking into it, I’m assuming it has something to do with the coroutine processing the Flow being suspended while the connection is closed.

Ktor - A Kotlin web framework

Sun, 24 Jul 2022 00:00:00 GMT

Ktor is an asynchronous web framework written in and designed for Kotlin, leveraging coroutines and allowing you to write asynchronous code without having to manage any threads yourself.

Here is a bit more background information on Ktor. It is backed by Jetbrains, who are also the creators of Kotlin itself. Who better to make a Kotlin web framework than the people that work on the language.

Implementation

Dependencies

buildscript {
  ext.kotlin_version = '1.7.10'
  ext.ktor_version = '2.0.3'

  repositories {
    mavenCentral()
  }
  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
  }
}

apply plugin: 'java'
apply plugin: 'kotlin'

group 'ktor-and-kodein-di'
version '1.0.0'

repositories {
  mavenLocal()
  mavenCentral()
}

dependencies {
  implementation "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version"
  testImplementation "org.jetbrains.kotlin:kotlin-test:$kotlin_version"
  testImplementation "junit:junit:4.12"

  implementation "io.ktor:ktor-server-netty:$ktor_version"
  // each plugin is its own dependency which matches the name in code (default-headers = DefaultHeaders)
  implementation "io.ktor:ktor-serialization-jackson:$ktor_version"
  implementation "io.ktor:ktor-server-default-headers:$ktor_version"
  implementation "io.ktor:ktor-server-call-logging:$ktor_version"
  implementation "io.ktor:ktor-server-content-negotiation:$ktor_version"

  implementation group: 'org.kodein.di', name: 'kodein-di-generic-jvm', version: '6.5.1'
  implementation group: 'ch.qos.logback', name: 'logback-classic', version: '1.2.3'
  implementation group: 'com.datastax.oss', name: 'java-driver-core', version: '4.14.1'
}

A few things are going on here.

Ktor 2.0.3 uses a minimum version of Kotlin 1.7 from what I’ve determined from their documentation.
Dependencies on ktor-server-netty and several ktor-server plugins are brought in. As ktor-server-netty suggests, Netty will be used for this post. Depending on which you choose to import, different underlying web servers can be used. The other available options are Netty, Jetty, Tomcat and CIO. More information can be found in the supported engines documentation.
Logback is brought in to handle logging. This is not included in the Ktor dependencies and is needed if you plan on doing any sort of logging.
Kodein is a dependency injection framework written in Kotlin. I have used it loosely in this post, and due to the size of the code examples, I could probably remove it altogether.

Starting the web server

With the boring stuff out of the way, I can now run you through implementing a web server. The code below is all you need:

import io.ktor.server.engine.embeddedServer
import io.ktor.server.netty.Netty

fun main() {
  embeddedServer(Netty, port = 8080, module = Application::module).start()
}

fun Application.module() {
  // code that does stuff which is covered later
}

Bam. There you have it. A web server running with Ktor and Netty. Ok, yes, it doesn’t really do anything, but we’ll expand on this in the following sections.

The code is pretty self-explanatory. The only piece worth highlighting is the Application.module function. The module parameter of embeddedServer requires an Application.() -> Unit function to be provided that configures the server and will be the main entry point into the server code.

In the following sections, we will expand the contents of Application.module so that your web server actually does something worthwhile.

Routing

All incoming requests will be rejected at the moment since there are no endpoints to handle them. By setting up the routing, you can specify valid paths that requests can travel along and the functions that will process the requests when they reach their destinations.

This is done inside of a Routing block (or multiple Routing blocks). Inside of a block, routes to different endpoints are set up:

import io.ktor.http.HttpStatusCode
import io.ktor.server.application.call
import io.ktor.server.request.receive
import io.ktor.server.response.respond
import io.ktor.server.response.respondText
import io.ktor.server.routing.Routing
import io.ktor.server.routing.get
import io.ktor.server.routing.post
import io.ktor.server.routing.route

routing {
  // All routes defined inside are prefixed with "/people"
  route("/people") {
    // Get a person
    get("/{id}") {
      val id = UUID.fromString(call.parameters["id"]!!)
      personRepository.find(id)?.let {
        call.respond(HttpStatusCode.OK, it)
      } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
    }
    // Create a person
    post {
      val person = call.receive<Person>()
      val result = personRepository.save(person.copy(id = UUID.randomUUID()))
      call.respond(result)
    }
  }
}

The imports have been included due to the reliance on extension functions, making discovering functions without an IDE difficult.

routing is a convenience function to make the code flow smoothly. The context (a.k.a this) inside of routing is of type Routing. Furthermore, the functions route, get, and post are all extension functions of Routing.

route sets a base path to all its following endpoints. In this scenario, /people. get and post do not specify a path themselves since the base path is suffice for their needs. If desired, a path could be added to each one, for example:

routing {
  // Get a person
  get("/people/{id}") {
    val id = UUID.fromString(call.parameters["id"]!!)
    personRepository.find(id)?.let {
      call.respond(HttpStatusCode.OK, it)
    } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
  }
  // Create a person
  post("/people) {
    val person = call.receive<Person>()
    val result = personRepository.save(person.copy(id = UUID.randomUUID()))
    call.respond(result)
  }
}

Before you move onto the next section, I want to show you how I actually implemented the routing:

fun Application.module() {
  val personRepository by kodein.instance<PersonRepository>()
  // Route requests to handler functions
  routing { people(personRepository) }
}

// Extracted to a separate extension function to tidy up the code
fun Routing.people(personRepository: PersonRepository) {
  route("/people") {
    // Get a person
    get("/{id}") {
      val id = UUID.fromString(call.parameters["id"]!!)
      personRepository.find(id)?.let {
        call.respond(HttpStatusCode.OK, it)
      } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
    }
    // Create a person
    post {
      val person = call.receive<Person>()
      val result = personRepository.save(person.copy(id = UUID.randomUUID()))
      call.respond(result)
    }
  }
}

I extracted the code to a separate function to decrease the contents of Application.module. When trying to write a more significant application, this will be a good idea. Whether the way I went about it is the Ktor way or not is another question. From having a quick look at the Ktor docs, it looks like this is a decent solution. I believe I saw another way to do this, but I would need to spend more time with it.

Contents of a request handler

The code that executes when a request is routed to a request handler is obviously pretty important. The function needs to do something after all…

Each handler function executes within the context of a coroutine. I did not really use this fact since each of the functions I have shown are fully synchronous.

For the remainder of this post, I will try not to mention coroutines too much since they are not particularly important for this simple REST API.

In this section, the get function will be examined a little closer:

get("/{id}") {
  val id = UUID.fromString(call.parameters["id"]!!)
  personRepository.find(id)?.let {
    call.respond(HttpStatusCode.OK, it)
  } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
}

{id} indicates that a path variable is expected in the request, and its value will be stored as id. Multiple path variables can be included, but only one is needed for this example 👍. The value of id is retrieved from call.parameters, which takes in the name of the variable you want to access.

call represents the context of the current request.
parameters is a list of the request’s parameters.

The database searches for the corresponding record using the id from the path variables. In this scenario, if it exists, the record is returned along with the appropriate 200 OK. If it doesn’t, an error response is returned. Both respond and respondText alter the underlying response of the current call. You could do this manually, for example, by using:

call.response.status(HttpStatusCode.OK)
call.response.pipeline.execute(call, it)

You could do that, but there isn’t any need to since that is actually just the implementation of respond. respondText has some extra logic but delegates down to response to finalise everything. The final call to execute in this function represents the function’s return value.

Installing plugins

In Ktor, plugins can be installed when needed. For example, Jackson JSON parsing can be added to handle and return JSON from your application.

Below are the plugins installed to the example application:

import io.ktor.http.HttpHeaders
import io.ktor.serialization.jackson.jackson
import io.ktor.server.application.install
import io.ktor.server.plugins.callloging.CallLogging
import io.ktor.server.plugins.contentnegotiation.ContentNegotiation
import io.ktor.server.plugins.defaultheaders.DefaultHeaders
import org.slf4j.event.Level

fun Application.module() {
  // Adds header to every response
  install(DefaultHeaders) { header(HttpHeaders.Server, "My ktor server") }
  // Controls what level the call logging is logged to
  install(CallLogging) { level = Level.INFO }
  // Setup jackson json serialisation
  install(ContentNegotiation) { jackson() }
}

DefaultHeaders adds a header to every response with the server’s name.
CallLogging logs information about outgoing responses and specifies what level to log them at. A logging library needs to be included for this to work. The output will look something like this:
```
INFO  ktor.application.log - 200 OK: GET - /people/302a1a73-173b-491c-b306-4d95387a8e36
```
ContentNegotiation tells the server to use Jackson for incoming and outbound requests. Remember this required including ktor-serialization-jackson as a dependency. You could also use GSON if you prefer.

For a list of the other plugins that Ktor includes, you can go to start.ktor.io, where you can view the existing plugins (by pretending to create a new application).

Installing plugins ties all the way back to the routing done earlier. routing delegates down to install inside its implementation. So you could write:

install(Routing) {
  route("/people") {
    get {
      // Implementation
    }
  }
}

Whatever floats your boat, but I’d just stick to using routing. Hopefully, that helped you understand what is going on under the hood, even if it was just a little bit.

Brief mention for Kodein

I want to have a very brief look at Kodein since I used it in this post. Kodein is a dependency injection framework written in Kotlin, for Kotlin. Below is the super small amount of DI that I used for the example application:

val kodein = Kodein {
  bind<CqlSession>() with singleton { cassandraSession() }
  bind<PersonRepository>() with singleton { PersonRepository(instance()) }
}
val personRepository by kodein.instance<PersonRepository>()

Inside the Kodein block, instances of the application’s classes are created. In this scenario, only one instance of each class is needed. Calling singleton denotes this. instance is a placeholder provided by Kodein to pass into a constructor instead of the actual object.

Outside of the Kodein block, an instance of PersonRespository is retrieved.

Yeah, I know; there isn’t much point to using Kodein here since I could have replaced it with a single line…

val personRepository = PersonRepository(cassandraSession())

Instead, let’s consider it a very concise example to understand 👍.

Summary

In this post, we looked at initialising a web server using Ktor, routing requests to lambdas/handlers that generate responses and installing plugins to the server. We mainly stayed at the surface level in this post and focused on the foundational knowledge to get you up and running with Ktor. For more information, it is worth going to ktor.io and viewing Ktor’s documentation and samples.

Reading and writing with a ConcurrentHashMap

Sat, 09 Jul 2022 00:00:00 GMT

ConcurrentHashMap provides a Map implementation with thread-safe read and write operations.

The Map and ConcurrentMap interfaces provide methods that ConcurrentHashMap takes advantage of to provide thread-safe interactions. Generally, I tend to solely really on the Map interface as it provides most of the same methods that ConcurrentMap has; however, depending on your use case, it might be beneficial to check out the ConcurrentMap methods yourself.

This post will look at reading from and writing to a ConcurrentHashMap.

Reading

ConcurrentHashMap allows concurrent read access to its contents where a key can still be read from even if the same key is being modified by another thread.

Reading itself does not affect the map’s contents and is therefore safe and does not break any of the guarantees that ConcurrentHashMap provides. However, this does mean that read values could be “old”, as the map doesn’t block reads even when updates are actively occurring on other threads. That being said, as soon as the update completes, reading threads will receive the updated value.

From my searching, there doesn’t seem to be a native Map implementation that prevents reads and updates from occurring simultaneously on the same key. There is Collections.synchronizedMap, but that synchronises access to the whole map rather than per key.

Below is an example showing how reads are not blocked while the map is being written to using compute (which we’ll look at in the writing section):

public static void main(String[] args) {

  // Put some values into the map.
  Map<Integer, String> map = new ConcurrentHashMap<>(Map.of(
    1, "INITIAL",
    2, "INITIAL",
    3, "INITIAL"
  ));

  // Slowly write to key [1] in the map using `compute`.
  new Thread(() -> {
    while (true) {
      map.compute(1, (key, value) -> {
        for (int i = 0; i < 5; i++) {
          System.out.println("[1] computing...");
          sleep(1000);
        }
        return "COMPUTE";
      });
      System.out.println("[1] updated from compute");
    }
  }).start();

  // Read each key in the map in a loop.
  for (int key : map.keySet()) {
    new Thread(() -> {
      while (true) {
        String value = map.get(key);
        System.out.println("[" + key + "] read from thread - " + value);
        sleep(1000);
      }
    }).start();
  }
}

// Just added to contain the try/catch when sleeping.
private static void sleep(int time) {
  try {
    Thread.sleep(time);
  } catch (InterruptedException e) {
    e.printStackTrace();
  }
}

When run, the code outputs something like:

[1] computing...
[2] read from thread - INITIAL
[3] read from thread - INITIAL
[1] read from thread - INITIAL
[1] computing...
[2] read from thread - INITIAL
[3] read from thread - INITIAL
[1] read from thread - INITIAL
[1] computing...
[1] read from thread - INITIAL
[3] read from thread - INITIAL
[2] read from thread - INITIAL
[1] computing...
[2] read from thread - INITIAL
[3] read from thread - INITIAL
[1] read from thread - INITIAL
[1] computing...
[2] read from thread - INITIAL
[3] read from thread - INITIAL
[1] read from thread - INITIAL
[1] updated from compute
[1] computing...
[3] read from thread - INITIAL
[2] read from thread - INITIAL
[1] read from thread - COMPUTE

This shows that reads occur on every key (1, 2, 3) even though key 1 is being written to throughout the example’s execution.

Writing

ConcurrentHashMap prevents multiple threads from writing to the same key simultaneously. From a superficial perspective, when a write operation begins, such as compute, it takes a lock for the specified key and blocks any other threads trying to write to the same key. Once the write finishes, the lock is released, and the process begins again with one of the blocked threads taking the lock.

The more complicated explanation is that ConcurrentHashMap keeps key-value pairs in buckets/bins/segments (I’ll refer to them as buckets from now on), which hold many pairs and can be locked. Writing to a key locks its bucket, preventing writes to any key within the bucket. This means that seemingly unrelated writes to separate keys can still block each other; however, this is unlikely to noticeably impact your application. Writes to unlocked buckets are not blocked, with these buckets then transitioning to a locked state while the write executes.

All the write methods available to the Map and ConcurrentMap interfaces work like this when using a ConcurrentHashMap.

The behaviour described above ensures that each write behaves deterministically. Otherwise, a call to compute could read the current value while another thread comes in, writes a new value, and then the compute call finishes and updates the value again. This is wrong because the compute’s code did not accommodate this change and is operating on out-of-date data. The same can be said about the key-value mapping being removed while compute is executing.

I’ve mentioned compute many times throughout this post. It is the primary method I use when writing to a ConcurrentHashMap where the current value should be read when determining the new one. Methods such as, computeIfAbsent, computeIfPresent and merge have similar behaviour to compute.

The example below shows that multiple threads cannot write to the same key at the same time:

public static void main(String[] args) {

  // Put some values into the map.
  Map<Integer, String> map = new ConcurrentHashMap<>(Map.of(
    1, "INITIAL",
    2, "INITIAL",
    3, "INITIAL"
  ));

  // Slowly write to key [1] in the map using `compute`.
  new Thread(() -> {
    while (true) {
      map.compute(1, (key, value) -> {
        for (int i = 0; i < 5; i++) {
          System.out.println("[1] computing...");
          sleep(1000);
        }
        return "COMPUTE";
      });
      System.out.println("[1] updated from compute");
    }
  }).start();

  // Write to each key in the map in a loop.
  for (int key : map.keySet()) {
    new Thread(() -> {
      while (true) {
        map.put(key, "THREAD");
        System.out.println("[" + key + "] updated from thread");
        sleep(1000);
      }
    }).start();
  }
}

// Just added to contain the try/catch when sleeping.
private static void sleep(int time) {
  try {
    Thread.sleep(time);
  } catch (InterruptedException e) {
    e.printStackTrace();
  }
}

When run, the code outputs something like:

[1] computing...
[2] updated from thread
[3] updated from thread
[1] computing...
[2] updated from thread
[3] updated from thread
[1] computing...
[2] updated from thread
[3] updated from thread
[1] computing...
[3] updated from thread
[2] updated from thread
[1] computing...
[3] updated from thread
[2] updated from thread
[1] updated from compute
[1] updated from thread

Here compute and put are called for key 1 on different threads. The compute began executing first and claimed the lock, preventing the put from executing until the compute call was completed. You can see this in the output as there is no update to key 1 until compute finishes, and then the write from put instantly executes afterwards. While these threads contest each other, the writes to keys 2 and 3 continue without issue.

A closer look at compute

I wanted to quickly go over compute specifically as I often use it myself, but I still have to check how to use it correctly each time…

Although the JavaDocs for compute are actually quite thorough, seeing an example tends to make it sink in better (for me at least):

// Setup for the example.
ScheduledExecutorService executorService = Executors.newSingleThreadScheduledExecutor();
Map<Integer, ScheduledFuture<?>> map = new ConcurrentHashMap<>();

// Always schedule a new future that prints "Hello" after a delay.
// If a future already exists for the passed in `key`, then `cancel` it.
map.compute(key, (k, future) -> {
  if (future != null && !future.isDone()) {
    future.cancel(true);
  }
  return executorService.schedule(() -> System.out.println("Hello"), 10, TimeUnit.SECONDS);
});

I have written code similar to this example many times. compute is handy here because it allows you to atomically interact with the existing keyed value while providing a new mapping. In this case, it allows the existing future to be cancelled before creating a new one and holding it within the map.

compute also lets you remove a mapping by returning null. However, this means that compute cannot be used to map a key to a null value. For ConcurrentHashMap though, this is a non-factor as the data structure explicitly disallows null values. From the JavaDocs:

/**
* ...
* <p>Like {@link Hashtable} but unlike {@link HashMap}, this class
* does <em>not</em> allow {@code null} to be used as a key or value.
* ...
*/

Below is an example that removes a mapping using compute:

// Setup for the example.
Map<Integer, Integer> map = new ConcurrentHashMap<>();

map.compute(key, (k, value) -> {
  int newValue = value + input;
  if (newValue > 100) {
    return null;
  }
  return newValue;
});

This example removes from the map if the input and existing value surpass a specific number or updates the mapping to the new total.

Do not use get and put for atomic updates of existing mappings

ConcurrentHashMap cannot ensure deterministic behaviour if you write code like the following:

// `Runnable` created because non-final variables can't be used in lambdas.
static class MyThread implements Runnable {

  private int number;
  private AtomicInteger sharedUpdateCounter;
  private int maxSharedCount;
  private Map<Integer, Integer> sharedMap;

  public MyThread(
    int number,
    AtomicInteger sharedUpdateCounter,
    int maxSharedCount,
    Map<Integer, Integer> sharedMap
  ) {
    this.number = number;
    this.sharedUpdateCounter = sharedUpdateCounter;
    this.maxSharedCount = maxSharedCount;
    this.sharedMap = sharedMap;
  }

  @Override
  public void run() {
    // Keep updating until all the shared count is reached by the threads.
    while (sharedUpdateCounter.getAndIncrement() < maxSharedCount) {
      // Read and icrement the existing value.
      int existingValue = sharedMap.get(1);
      int newValue = existingValue + 1;
      // Update with the incremented value.
      sharedMap.put(1, newValue);
      System.out.println("Thread [" + number + "] 1 -> " + newValue);
      sleep(1000);
    }
  }
}

public static void main(String[] args) {

  Map<Integer, Integer> map = new ConcurrentHashMap<>(Map.of(
    1, 0
  ));

  // Setup a shared count and max value to limit the number of writes to 12.
  AtomicInteger j = new AtomicInteger(0);
  int maxSharedCount = 12;

  // Start threads.
  List<Thread> threads = Stream.of(1, 2, 3).map(i -> {
    Thread thread = new Thread(new MyThread(i, j, maxSharedCount, map));
    thread.start();
    return thread;
  }).toList();

  // Wait for the threads to finish.
  threads.forEach(thread -> {
    try {
      thread.join();
    } catch (InterruptedException e) {
      e.printStackTrace();
    }
  });

  System.out.println("Final value = " + map.get(1));
}

This code which reads a key’s current value using get and then updates it using put, allows updates from separate threads to overwrite each other, leading to an indeterministic outcome.

Thread [2] 1 -> 2
Thread [3] 1 -> 3
Thread [1] 1 -> 1
Thread [2] 1 -> 4
Thread [3] 1 -> 4
Thread [1] 1 -> 4
Thread [1] 1 -> 5
Thread [3] 1 -> 5
Thread [2] 1 -> 5
Thread [2] 1 -> 7
Thread [3] 1 -> 7
Thread [1] 1 -> 6
Final value = 7

The output would be 12 if reads and writes occurred atomically.

Rewriting the example to use compute instead of paired calls to get and put gives us the deterministic result that we desire:

@Override
public void run() {
  while (sharedUpdateCounter.getAndIncrement() < maxSharedCount) {
    int newValue = sharedMap.compute(1, (key, existingValue) -> existingValue + 1);
    System.out.println("Thread [" + number + "] 1 -> " + newValue);
    sleep(1000);
  }
}

Outputs:

Thread [3] 1 -> 1
Thread [2] 1 -> 2
Thread [1] 1 -> 3
Thread [3] 1 -> 4
Thread [2] 1 -> 6
Thread [1] 1 -> 5
Thread [2] 1 -> 7
Thread [3] 1 -> 9
Thread [1] 1 -> 8
Thread [2] 1 -> 10
Thread [3] 1 -> 11
Thread [1] 1 -> 12
Final value = 12

The threads that perform each update will vary, but the final value is always consistent.

Summary

ConcurrentHashMap allows concurrent reads while preventing simultaneous writes, ensuring that the map behaves deterministically. However, you must correctly use the writing methods to ensure your code behaves as expected. Relying on methods such as compute and merge, which allow you to atomically read an existing mapping and then act on it, is crucial; otherwise, the benefits of using ConcurrentHashMap are void.

I rarely need Stack Overflow anymore

Sun, 12 Jun 2022 00:00:00 GMT

I noticed recently that my dependence on Stack Overflow and similar platforms has wained as I have progressed in my career. The challenges I face and the questions arising from them have changed, becoming more abstract, handy-wavy, or super-specific to my company’s business logic. The likelihood that someone has already answered these sorts of questions becomes unlikely. Maybe, I could try to ask these questions myself (I’m more of a reader of Stack Overflow rather than a contributing member), but where would I even begin. These questions aren’t asked because of the amount of context and background knowledge required to comprehend the question.

Now, don’t get me wrong. I’m not bashing Stack Overflow at all here. There are plenty of questions that can be asked or are already answered, primarily revolving around using language features, libraries and frameworks. This is where it shines, which is why “rarely” instead of “do not” is included in the title. When I’m working on my own code outside of work, it is invaluable to me. But, that’s because my questions shift into insight about language features, libraries and frameworks like I mentioned a second ago.

At this point, you are probably wondering what point I’m trying to make, and believe me, I am also trying to figure that out right now. So here I go…

If, like me, you are trying to find answers to challenges you face at work, varying from writing better code, to solving architectural issues, to even the most abstract questions of “are we solving the correct problems”; point yourself in the direction of your colleagues. They (most of the time) have the context and foundational knowledge to correctly answer questions in a way that no one else can. I can’t even begin to quantify the amount I’ve learnt from turning to my teammates for help. They may have solved similar problems in the past and can share that experience with you, all based on their ability to actually understand the nuance in what you’re asking.

For more senior engineers reading this, do not underestimate the impact sharing your knowledge has on your teammates. Every time I hear the phrase, “At my old place”, I know we’re about to for a trip through their experiences, and I also know that without these conversations, I would not have progressed at the rate I have in my own career. You cannot rely on Stack Overflow and similar platforms to level up your team, it may help, but it will not produce well-rounded engineers.

For everyone else, you cannot rely on Stack Overflow and similar platforms for everything. Read longer-form content, such as blogs, books or podcasts, focusing more on the why than the how. However, having a back and forth between other engineers will be one of the most efficient ways to improve. If you fail to do this, you will never progress to the point where you rarely need Stack Overflow anymore.

Getting started with RSocket Kotlin

Sat, 04 Jun 2022 00:00:00 GMT

RSocket is a transport protocol designed for reactive applications. More information on RSocket can be found on their website, leaving me to focus on writing about how RSocket and Kotlin can be combined.

RSocket has several libraries written in various languages that implement the RSocket protocol. For Kotlin, this comes as an extension for Ktor (a Kotlin client and web-server library) named rsocket-kotlin. We will look at this extension through this post.

The RSocket + Ktor libraries heavily use Kotlin’s coroutines and flows. I recommend looking at these first if you have not used them before. Writing the content for this post was the first experience I’ve had using flows, and although they resemble constructs I am used to, calling the right method at the right time still proved challenging.

In this post, we will construct two services; one handling inbound requests via HTTP and a backend service that only exposes RSocket endpoints. This allows me to demonstrate both a working RSocket client and server, as well as fiddling around with some Ktor functionality to make it more interesting. In later sections, the inbound service will be known as the “client” and the backend service as the “server”.

Dependencies

At the time of writing, you’ll want to use the following dependencies:

implementation("io.rsocket.kotlin:rsocket-core:0.15.4")

implementation("io.rsocket.kotlin:rsocket-ktor-client:0.15.4")

implementation("io.rsocket.kotlin:rsocket-ktor-server:0.15.4")

implementation("io.ktor:ktor-server-netty:2.0.1")

implementation("io.ktor:ktor-client-cio:2.0.1")

As you can see, the RSocket dependencies are all 0.x.x versions, and therefore the APIs fluctuate over time. Hopefully, they’ll remain slightly stable after I’ve written this so that I don’t have to update it…

Building blocks of the inbound and backend services

In the two sections below, representing the inbound and backend services, we will look at the foundational code that endpoints can be built onto.

Inbound service (RSocket client + HTTP server)

fun main() {
    val client = HttpClient {
        install(WebSockets)
        install(RSocketSupport)
    }
    embeddedServer(Netty, port = 8000) {
        install(io.ktor.server.websocket.WebSockets)
        routing {
          // Add HTTP endpoints
        }
    }.start(wait = true)
}

Here we have a HTTPClient, which will be used to make RSocket requests. I know it doesn’t look related to RSocket yet; right now, it’s not. It’s a plain old HTTPClient provided by Ktor; however, as you’ll see later, the RSocket client library provides extension functions that leverage a HTTPClient to make requests.

The client installs WebSockets and RSocketSupport to add functionality via plugins for WebSockets and RSockets respectively. It is essential to point out here that WebSockets must be installed before RSocketSupport; otherwise, you will receive the following error:

Exception in thread "main" java.lang.IllegalStateException: RSocket require WebSockets to work. You must install WebSockets plugin first.
	at io.rsocket.kotlin.ktor.client.RSocketSupport$Plugin.install(RSocketSupport.kt:49)
	at io.rsocket.kotlin.ktor.client.RSocketSupport$Plugin.install(RSocketSupport.kt:40)
	at io.ktor.client.HttpClientConfig$install$3.invoke(HttpClientConfig.kt:78)
	at io.ktor.client.HttpClientConfig$install$3.invoke(HttpClientConfig.kt:73)
	at io.ktor.client.HttpClientConfig.install(HttpClientConfig.kt:96)
	at io.ktor.client.HttpClient.<init>(HttpClient.kt:165)
	at io.ktor.client.HttpClient.<init>(HttpClient.kt:80)
	at io.ktor.client.HttpClientKt.HttpClient(HttpClient.kt:42)
	at io.ktor.client.HttpClientJvmKt.HttpClient(HttpClientJvm.kt:21)
	at dev.lankydan.inbound.InboundKt.main(Inbound.kt:38)
	at dev.lankydan.inbound.InboundKt.main(Inbound.kt)

Thankfully, the exception is nice and explicit and tells you how to rectify the issue.

We then have the non-RSocket part, where a HTTP server is created and represents the entry point to the system. WebSockets are also installed here, which has no relation to RSocket itself but is used in this post’s examples to make them more interesting.

Backend service (RSocket server)

fun main() {
    embeddedServer(Netty, port = 9000) {
        install(WebSockets)
        install(RSocketSupport)
        routing {
            // Add RSocket endpoints
        }
    }.start(wait = true)
}

Onto the backend service, which only provides RSocket endpoints, which the inbound service communicates with. This means there is no need for a HTTPClient to be created. This time around, the HTTP server provided by embeddedServer installs WebSockets and RSocketSupport.

WebSockets should be installed before RSocketSupport here as well.

RSocket request types

In the following sections, we will look at examples for each RSocket request type, which include:

Fire-and-forget.
Request response.
Request stream.
Request channel.

The examples in the following sections will detail both the client and server-side implementations of each RSocket request type.

Fire-and-forget

A Fire and forget request sends an endpoint some data and moves on, not expecting a response to be sent back. This allows for optimisations in both the client and the server implementations.

From the RSocket documentation:

Fire-and-forget is an optimization of request/response that is useful when a response is not needed. It allows for significant performance optimizations, not just in saved network usage by skipping the response but also in client and server processing time as no bookkeeping is needed to wait for and associate a response or cancellation request.

Client side

fun Routing.fireAndForget(client: HttpClient) {
    get("fireAndForget") {
        val rSocket: RSocket = client.rSocket(path = "fireAndForget", port = 9000)
        rSocket.fireAndForget(buildPayload { data("Hello") })

        log.info("Completed fire and forget request")

        call.respondText { "Completed" }
    }
}

To send a fire-and-forget request, create a RSocket by calling rSocket on the HTTPClient made earlier. This is an extension function that the RSocket library provides to leverage Ktor’s HTTPClient. When creating a RSocket, the endpoint’s path is defined (which defaults to localhost) and a port is specified. RSocket.fireAndForget is then called using buildPayload to construct the data to send. buildPayload uses a lambda and a PayloadBuilder to assist in building a payload.

After fireAndForget is called, there is nothing left to do as you’re meant to forget about the response.

Server side

fun Routing.fireAndForget() {
    rSocket("fireAndForget") {
        RSocketRequestHandler {
            fireAndForget { request: Payload ->
                val text = request.data.readText()
                log.info("Received request (fire and forget): '$text' ")
            }
        }
    }
}

On the server-side, rSocket defines the endpoint’s path, then RSocketRequestHandler along with fireAndForget specifies the type of request that is handled. This pattern repeats in the following sections, with only the call to fireAndForget being replaced with the different request types.

Each request handler is provided with arguments that allow it to serve requests. In this fireAndForget example, only the request’s Payload is provided since that is all it needs.

Request response

Building upon the previous section’s knowledge, there is not much of a difference between requestResponse and fireAndForget requests. So this section will be short and sweet.

Client side

fun Routing.requestResponse(client: HttpClient) {
    get("requestResponse") {
        val rSocket: RSocket = client.rSocket(path = "requestResponse", port = 9000)
        val response: Payload = rSocket.requestResponse(buildPayload { data("Hello") })
        val text = response.data.readText()

        log.info("Received response from backend: '$text'")

        call.respondText { text }
    }
}

This is virtually the same as the fireAndForget example, with the only difference being the existence of a response Payload that the server replies with.

Server side

fun Routing.requestResponse() {
    rSocket("requestResponse") {
        RSocketRequestHandler {
            requestResponse { request: Payload ->
                val text = request.data.readText()
                log.info("Received request (request/response): '$text' ")
                delay(200)
                buildPayload { data("Received: '$text' - Returning: 'some data'") }
            }
        }
    }
}

This request handler will also look familiar. Here we call requestResponse and then return a Payload from the function. This Payload is sent back to the client in the response.

Note that the function passed into requestResponse is a suspending function, allowing it to call methods like delay. Each request handler method also shares this behaviour.

Request stream

A Request stream is a stream of data that flows in one direction, towards the client, from the RSocket endpoint. This data stream can be finite or indefinite; however, either can be cancelled to terminate the stream on both sides of the connection.

Client side

fun Routing.requestStream(client: HttpClient) {
    webSocket("requestStream") {
        val rSocket: RSocket = client.rSocket(path = "requestStream", port = 9000)
        val stream: Flow<Payload> = rSocket.requestStream(buildPayload { data("Hello") })
        
        // Receives data via a WebSocket
        incoming.receiveAsFlow().onEach { frame ->
            log.info("Received frame: $frame")
            if (frame is Frame.Text && frame.readText() == "stop") {
                log.info("Stop requested, cancelling socket")
                this@webSocket.close(CloseReason(CloseReason.Codes.NORMAL, "Client called 'stop'"))
            }
        }.launchIn(this)

        // Handles data sent back over the RSocket connection
        stream.onCompletion {
            log.info("Connection terminated")
        }.collect { payload: Payload ->
            val data = payload.data.readText()
            log.info("Received payload: '$data'")
            delay(500)
            send("Received payload: '$data'")
        }
    }
}

The example above looks more complex. Yet, that is partly due to the additional WebSocket code that didn’t exist in the previous sections.

Let’s look at the code now.

requestStream is called and passed a Payload that represents the initial Payload that the RSocket endpoint receives, returning a Flow of Payloads for the client to process. In the example, it logs the received payloads and sends them over WebSockets to the original caller of the HTTP endpoint. It does so by calling collect, which is a terminal operation on the Flow API that triggers the execution of the Flow (more information can be found in the Flow documentation).

The delay called inside collect here is interesting. RSocket manages back-pressure for you, meaning that if payloads are being received faster than they can be processed, the RSocket endpoint will stop streaming payloads until the previous batch is dealt with. You can run the code these examples are taken from if you wish to see for yourself.

At this point, we do not know if the request stream is finite or not, but whether it runs to completion or is cancelled, the onCompletion function will execute.

The code that we haven’t gone through yet is the handling of incoming data via a WebSocket. incoming is a ReceiveChannel that does just this. By chaining it with receiveAsFlow to simplify the manipulation of received data and launching the Flow to run as a separate Job, it can be left to run on another thread that terminates once the WebSocket ends (thanks to Coroutines). Forgetting to call launchIn will result in incoming data being ignored because the Flow never started processing. You also don’t want to call collect here as it is blocking and prevents the RSocket code that is called later on in the method from running. Trust me, I lost a lot of time to this mistake.

The final interesting part of this example is the call to this@websocket.close, which terminates the WebSocket and the RSocket Flow as it exists within the CoroutineScope of the webSocket function. This termination of the Flow is sent to the RSocket endpoint, which ends the stream running there.

Server side

fun Routing.requestSteam() {
    rSocket("requestStream") {
        RSocketRequestHandler {
            requestStream { request: Payload ->

                val prefix = request.data.readText()

                log.info("Received request (stream): $prefix")

                flow {
                    emitDataContinuously(prefix)
                }.onCompletion { throwable ->
                    if (throwable is CancellationException) {
                        log.info("Connection terminated")
                    }
                }
            }
        }
    }
}

suspend fun FlowCollector<Payload>.emitDataContinuously(prefix: String) {
    var i = 0
    while (true) {
        val data = "data: ${if (prefix.isBlank()) "" else "($prefix) "}$i"
        log.info("Emitting $data")
        emitOrClose(buildPayload { data(data) })
        i += 1
        delay(200)
    }
}

The server-side code above emits data continuously into the stream connected to the client.

As with the other request handlers, an initial Payload is received to determine the overall behaviour of the stream.

A Flow is then created to represent the stream (or flow of) data to the caller. Note that requestStream’s call signature is:

public fun requestStream(block: suspend (RSocket.(payload: Payload) -> Flow<Payload>))

Meaning that creating a Flow is required, and interacting with the response stream in any other way is not possible.

Data is continuously emitted data by using a while loop that calls emitOrClose each time round to send data to the response stream. emitOrclose is a more graceful way of sending data when compared to emit, as it explicitly handles cancellation, so that it can close any Payloads being sent to prevent any memory leaks.

Request channel

A request channel is a bi-directional stream of data. Where request streams only send data from the server to the client, request channels allow both sides to send and receive data; more interesting behaviour can then be modelled using this mechanism as either side can continuously influence the other.

Client side

fun Routing.requestChannel(client: HttpClient) {
    webSocket("requestChannel") {
        val rSocket: RSocket = client.rSocket(path = "requestChannel", port = 9000)

        // Receives data via a WebSocket and transforms it
        val payloads: Flow<Payload> = incoming.receiveAsFlow().transform { frame ->
            if (frame is Frame.Text) {
                val text = frame.readText()
                log.info("Received text: $text")
                if (text == "stop") {
                    log.info("Stop requested, cancelling socket")
                    this@webSocket.close(CloseReason(CloseReason.Codes.NORMAL, "Client called 'stop'"))
                } else {
                    emitOrClose(buildPayload { data(text) })
                }
            }
        }

        val stream: Flow<Payload> = rSocket.requestChannel(buildPayload { data("Hello") }, payloads)

        // Handles data sent back over the RSocket connection
        stream.onCompletion {
            log.info("Connection terminated")
        }.collect { payload: Payload ->
            val data = payload.data.readText()
            log.info("Received payload: '$data'")
            delay(500)
            send("Received payload: '$data'")
        }
    }
}

Some of this code overlaps with the request stream implementation, with the difference being the Flow passed into the requestChannel method.

The input Flow represents the stream of data transferred to the request endpoint, while the returned Flow receives the data sent by the endpoint.

The example leverages the data received over a WebSocket via the incoming method (the same one in the request stream snippet). The incoming data is converted into a Flow and transforms the data into Payloads to send over the channel. Although in one way, this is “incoming” data, it is converted and treated as the outgoing stream from the perspective of created RSocket channel. This shows how you can write flexible code that melds various concepts using Kotlin’s Flows that the RSocket API leverages.

Server side

private fun Routing.requestChannel() {
    rSocket("requestChannel") {
        RSocketRequestHandler {
            requestChannel { request: Payload, payloads: Flow<Payload> ->

                var prefix = request.data.readText()

                log.info("Received request (channel): '$prefix'")

                payloads.onEach { payload ->
                    prefix = payload.data.readText()
                    log.info("Received extra payload, changed emitted values to include prefix: '$prefix'")
                }.launchIn(this)

                flow {
                    emitDataContinuously(prefix)
                }.onCompletion { throwable ->
                    if (throwable is CancellationException) {
                        log.info("Connection terminated")
                    }
                }
            }
        }
    }
}

suspend fun FlowCollector<Payload>.emitDataContinuously(prefix: String) {
    var i = 0
    while (true) {
        val data = "data: ${if (prefix.isBlank()) "" else "($prefix) "}$i"
        log.info("Emitting $data")
        emitOrClose(buildPayload { data(data) })
        i += 1
        delay(200)
    }
}

As we saw in the client-side section, a Flow is passed into the requestChannel method to represent the flow of data from client to server. Therefore, the endpoint needs a way to receive this data; the payloads parameter of the requestChannel’s function does just that.

onEach is called to process each received Payload and then is launched to begin execution. This is important, as without actually starting the Flow, all the received Payloads will go unprocessed. I’ll just copy and paste what I wrote earlier about this.

Forgetting to call launchIn will result in incoming data being ignored because the Flow never started processing. You also don’t want to call collect here as it is blocking and prevents the RSocket code that is called later on in the method from running. Trust me, I lost a lot of time to this mistake.

Please, please, please, don’t falter as I did.

Other than that, the implementation of request channels is the same as that of request streams.

Summary

This post covered how to use RSocket with Kotlin and Ktor. The focus has been on implementation rather than “why RSocket”, which I’ll let you dig into yourself (if that is what you want anyway). I admit that I struggled to leverage the APIs correctly at first. However, once I had working implementations, their similarities became apparent, which is why I kept repeating, “this is the same as the previous section”. The primary difficulty was the use of Flows, which if you have not used them before, like me, you’ll also likely make all the mistakes I did. If you base your own examples or real-life work on the code I wrote here, you should at least bypass some issues and progress to a working implementation more smoothly.

Should you write comments, revisited

Mon, 11 Apr 2022 00:00:00 GMT

5 years ago, I wrote a blog post about leaving comments in your code. I leant heavily towards the side of not writing comments, and I see other people writing their own posts agreeing with that assertion. However, during these 5 years, even though my logical foundations of reducing comment usage remain, I have been presented with more and more scenarios where comments are a must-have. As it turns out, even if you write clean, structured and well thought out code, there reaches a point where the complexity is too high to fully process what the code does. In these situations, you must not be scared of writing comments; these are the exact moments that you should be.

It’s hard to really define a straightforward rule that one can follow on “when to write a comment”; experience will start to give you that gut feeling and was probably why I had a hardline stance on not writing comments in the past. That, or the software I’m producing, has increased in complexity over time.

I do stand by the two main points I wrote in my original post, which were:

Extract code into well-named methods instead of having long continuous streams of code.
Give variables descriptive names, so they don’t need a comment to explain them.

I doubt many people would disagree with those points, but what about everything else? Below are another two general areas where providing some extra information to the readers of your code would be greatly appreciated:

Complex interactions between classes - Classes can have indirect iterations between each other. Without documenting how they interweave, you will inevitably forget how they impact each other, or that they even do in the first place.
Highlighting workarounds due to bugs in other areas of code or dependent libraries - You can’t guarantee a fix in code that is out of your control. Sometimes you’ll have to work around the issue, and it is desirable to include a comment on why it was needed.

Really though, these are amalgamations of the same idea.

When complexity cannot be addressed directly or in a time-effective manner, a well-written comment will save future contributors vast amounts of time and considerably increase their understanding of the code. You could try and refactor the codebase to reduce complexity; however, it is not always worth spending days, weeks or months to do so. You’ll have to make a judgement call and decide whether it’s worth it on a case by case basis.

Over time, you’ll better find the line where you can balance complexity with self-explaining code and comments to fill the gaps, all while maintaining the velocity that your project or company requires from you.

If you have your own identifiers for code that requires commenting, do let me know. I spent a while thinking about this but felt that every situation fit into the two categories I listed here (or the two from my previous post).

Using GraphQL Java with Spring

Sun, 09 Jan 2022 00:00:00 GMT

GraphQL Java is one of the most popular GraphQL server-side implementations for Java that I’ve found (with over 5k stars at the time of writing). If you’re planning to expose a GraphQL API from a Java or JVM application, then this is an excellent library to start using.

This blog post will cover how to use GraphQL Java within a Spring application that exposes an endpoint for clients to send queries to. GraphQL Java does have its own official documentation covering this subject; however, I found it a bit too simplistic, which made it harder for me to wrap my head around it. I hope you don’t think the same about the content of this post, although I guess you could write your own that is even more complicated than my examples!

I will be writing this from the assumption that you understand some of the basics of GraphQL. I won’t be covering anything extremely complicated, so the basics will give you a good base for the content of this blog post. Knowing how to create a schema type and a query that does nothing fancy will be all you need. You can find this information from the official graphql.org site.

Also, here’s a heads up that I have written my examples in Kotlin, although I tried to keep it friendly for Java readers.

Dependencies

Below are the Spring and GraphQL related dependencies used in this post:

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.6.1</version>
</parent>

<dependencies>
  <dependency>
    <groupId>com.graphql-java</groupId>
    <artifactId>graphql-java</artifactId>
    <version>16.2</version>
  </dependency>
  <dependency>
    <groupId>com.graphql-java</groupId>
    <artifactId>graphql-java-spring-boot-starter-webmvc</artifactId>
    <version>2.0</version>
  </dependency>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-web</artifactId>
    <version>2.6.1</version>
  </dependency>
</dependencies>

The schema we’ll use

Below is the GraphQL schema that we’ll use throughout this post:

type Query {
  people: [Person]
  peopleByFirstName(firstName: String): [Person]
  personById(id: ID): Person
}

type Person {
  id: ID,
  firstName: String,
  lastName: String
  relationships: [Relationship]
}

type Relationship {
  relation: Person,
  relationship: String
}

I’ll cover how we register and interact with the schema later on; knowing the shape of the types in the schema will set up the following sections.

Fetching data with DataFetchers

GraphQL Java uses DataFetchers to fetch data to include in the result of a query. More specifically, a DataFetcher retrieves the data for a single field when a query is executed.

Every field has an assigned DataFetcher. When an incoming GraphQL query is received, the library will call the registered DataFetcher for each field in the query.

The official DataFetcher documentation can be found here.

It is worth pointing out now, as it personally caused me a lot of confusion to begin with, that a “field” can mean two things:

The name of a query.
A property/field in a schema type.

This is important as it means that a DataFetcher can be linked to a query. In fact, every query must have an associated DataFetcher. Not doing this will cause GraphQL query requests to fail as there is no entry point to begin processing the query.

I keep mentioning DataFetchers, but what are they really in terms of code? Below is the DataFetcher interface:

public interface DataFetcher<T> {

  T get(DataFetchingEnvironment environment) throws Exception;
}

So when I say DataFetcher, I’m really talking about implementations of the DataFetcher interface.

Let’s look at an example DataFetcher that will respond to the people query:

type Query {
    people: [Person]
}

The DataFetcher for this query:

@Component
class PeopleDataFetcher(private val personRepository: PersonRepository) : DataFetcher<List<PersonDTO>> {

  override fun get(environment: DataFetchingEnvironment): List<PersonDTO> {
    return personRepository.findAll().map { person -> PersonDTO(
        person.id,
        person.firstName,
        person.lastName
      ) 
    }
  }
}

The PeopleDataFetcher returns a List<PersonDTO> to correspond to the [Person] specified as the turn type of the GraphQL query. The PersonDTO contains the same fields found in the Person GraphQL type:

data class PersonDTO(val id: UUID, val firstName: String, val lastName: String)

A field for relationships has been omitted to simplify the examples

When PeopleDataFetcher.get is executed, it queries the database, maps the results to PersonDTOs and returns them.

When an incoming GraphQL query, like the following is received:

query {
    people {
        firstName
        lastName
        id
    }
}

The following is returned after GraphQL Java has called the PeopleDataFetcher:

{
  "data": {
    "people": [
      {
        "firstName": "John",
        "lastName": "Doe",
        "id": "00a0d4f2-637f-469c-9ecf-ba8839307996"
      },
      {
        "firstName": "Dan",
        "lastName": "Newton",
        "id": "27a08c14-d0ad-476c-ba09-9edad3e4c8f9"
      }
    ]
  }
}

That covers a first look at what DataFetchers do. We’ll expand on them in the following sections to improve your understanding of GraphQL Java.

The default PropertyDataFatcher

As mentioned in the previous section, every field must have an assigned DataFetcher. That means for the Person type:

type Person {
  id: ID,
  firstName: String,
  lastName: String
  relationships: [Relationship]
}

You would need to associate DataFetchers to:

Person.id
Person.firstName
Person.lastName
Person.relationships

That seems a bit onerous, especially when you might be able to retrieve all of this data in one go. For example, retrieving data from a database could be the difference between 1 SQL query vs 4.

To resolve this, any field without an assigned DataFetcher uses the PropertyDataFetcher by default.

The PropertyDataFetcher uses various methods (e.g. getters or keys in a map) to extract field values from the parent field (could be a query or schema type).

To provide a concrete example, the PeopleDataFetcher we saw previously was used to respond to the people GraphQL query (the query, type and PeopleDataFetcher are included below):

type Query {
  people: [Person]
}

type Person {
  id: ID,
  firstName: String,
  lastName: String
  relationships: [Relationship]
}

@Component
class PeopleDataFetcher(private val personRepository: PersonRepository) : DataFetcher<List<PersonDTO>> {

  override fun get(environment: DataFetchingEnvironment): List<PersonDTO> {
    return personRepository.findAll().map { person -> PersonDTO(
        person.id,
        person.firstName,
        person.lastName
      ) 
    }
  }
}

The PeopleDataFetcher returns a PersonDTO for each Person it finds to respond to the top-level query. This can be thought of as the “parent field”.

The GraphQL library will then move down to fetch values for each queried field in Person, for example, firstName and lastName. Using the PropertyDataFetcher, it accesses each PersonDTO returned by the parent field’s DataFetcher (PeopleDataFetcher) and extracts the values using their getters.

In concrete terms, this means:

Person.id -> Value provided by PersonDTO.id.
Person.firstName -> Value provided by PersonDTO.firstName.
Person.lastName -> Value provided by PersonDTO.lastName.
Person.relationships -> Empty because no value was provided.

PersonDTO.id is how you write Person.getId in Kotlin (the applies to the other fields).

You might need to run that through your head a few times so it makes sense. I only properly understood this after debugging the library when my code wasn’t working correctly.

Writing a DataFetcher for a schema type’s field

The PeopleDataFetcher we’ve seen throughout this post responds to a query. Now let’s look at a custom DataFetcher that should be associated with a schema type’s field.

The PersonRelationshipsDataFetcher fetches data for the Person.relationships field:

@Component
class PersonRelationshipsDataFetcher(
  private val relationshipRepository: RelationshipRepository
) : DataFetcher<List<RelationshipDTO>> {

  override fun get(environment: DataFetchingEnvironment): List<RelationshipDTO> {
    // Gets the object wrapping the [relationships] field
    // In this case a [PersonDTO] object.
    val source = environment.getSource<PersonDTO>()
    return relationshipRepository.findAllByPersonId(source.id).map { relationship ->
      RelationshipDTO(
        relation = relationship.relatedPerson.toDTO(),
        relationship = relationship.relationship
      )
    }
  }
}

It looks similar to the PeopleDataFetcher we saw previously, except for the new call to DataFetchingEnvironment.getSource. This method allows a DataFetcher to access the object returned by the DataFetcher associated with the parent field. After accessing this object, information is extracted from it (PersonDTO.id) to be used in the SQL query executed by the PersonRelationshipsDataFetcher.

Writing a DataFetcher for a query containing an argument

Queries become far more worthwhile when you can pass arguments into them.

Take the query:

type Query {
  peopleByFirstName(firstName: String): [Person]
}

To handle this you’d want a DataFetcher like the one below:

@Component
class PeopleByFirstNameDataFetcher(private val personRepository: PersonRepository) : DataFetcher<List<PersonDTO>> {

  override fun get(environment: DataFetchingEnvironment): List<PersonDTO> {
    // The argument is extracted from the GraphQL query
    val firstName = environment.getArgument<String>("firstName")
    return personRepository.findAllByFirstName(firstName)
      .map { person -> PersonDTO(person.id, person.firstName, person.lastName) }
  }
}

The important method call here is to DataFetchingEnvironment.getArgument, which does as it says and extracts an argument from the incoming GraphQL query. Handily, getArgument allows you to specify the type the argument should be (so you don’t have to convert it yourself).

DataFetchingEnvironment also contains other methods regarding arguments, e.g. getArguments and containsArgument.

Setting up a GraphQL instance

You’ve seen how to write a few DataFetchers by this point; we now need to tie everything together by creating a GraphQL instance and registering an application’s DataFetchers.

The @Configuration code below does just that:

@Configuration
class GraphQLConfiguration(
  private val peopleByFirstNameDataFetcher: PeopleByFirstNameDataFetcher,
  private val peopleDataFetcher: PeopleDataFetcher,
  private val personByIdDataFetcher: PersonByIdDataFetcher,
  private val personRelationshipsDataFetcher: PersonRelationshipsDataFetcher
) {

  @Bean
  fun graphQL(): GraphQL {
    val typeRegistry: TypeDefinitionRegistry = SchemaParser().parse(readSchema())
    val runtimeWiring: RuntimeWiring = buildWiring()
    val graphQLSchema: GraphQLSchema =  SchemaGenerator().makeExecutableSchema(typeRegistry, runtimeWiring)
    return GraphQL.newGraphQL(graphQLSchema).build()
  }

  private fun schemaFile(): File {
    return this::class.java.classLoader.getResource("schema.graphqls")
      ?.let { url -> File(url.toURI()) }
      ?: throw IllegalStateException("The resource does not exist")
  }

  private fun buildWiring(): RuntimeWiring {
    return RuntimeWiring.newRuntimeWiring()
      .type(newTypeWiring("Query").dataFetcher("peopleByFirstName", peopleByFirstNameDataFetcher))
      .type(newTypeWiring("Query").dataFetcher("people", peopleDataFetcher))
      .type(newTypeWiring("Query").dataFetcher("personById", personByIdDataFetcher))
      .type(newTypeWiring("Person").dataFetcher("relationships", personRelationshipsDataFetcher))
      .build()
  }
}

The purpose of this @Configuration class is to create a GraphQL instance that GraphQL Java uses. Further setup is not required as it will be picked up by Spring Boot’s auto-configuration.

The first step of creating the GraphQL instance requires reading the application’s GraphQL schema. SchemaParser.parse can take in Files, InputStreams, Readers or Strings, which it parses (as the class name suggests) and returns a TypeDefinitionRegistry to be used later. In this application, the schema is defined in a resource file which gets fed into SchemaParser.parse. This is what allows the GraphQL library to understand incoming queries and what can or cannot be handled.

The DataFetchers are then registered with a RuntimeWiring instance (through a RuntimeWiring.Builder returned by RuntimeWiring.newRuntimeWiring). Every time I mentioned “gets the DataFetcher associated to the field”, this is where the association actually happens. I can stop hand waving all the time now since you’ve seen the code.

Each DataFetcher in this example application is injected into the configuration class and linked to the RuntimeWiring instance through its type method. Each TypeRuntimeWiring.Builder instance (returned by newTypeWiring) requires 3 fundamental inputs:

The name of the schema type ("Query" or a type name).
The name of the field (query name or schema type field).
The DataFetcher associated with the type and field.

After registering each DataFetcher, the RuntimeWiring instance is finalised using build.

Finally, the TypeDefinitionRegistry and RuntimeWiring created previously are passed through a SchemaGenerator and then into GraphQL.newGraphQL to retrieve a fully functional GraphQL instance.

Sending a GraphQL query to the application

With the setup complete, the application now exposes a /graphql endpoint provided by the auto-configured code in graphql-java-spring-boot-starter-webmvc. This endpoint is where clients will send GraphQL queries to.

In this section, we’ll look at how to send a query using cURL and Postman (which has GraphQL functionality) and view the returned data.

The query we are trying to send:

query {
    peopleByFirstName(firstName: "Dan") {
        firstName
        lastName
        id
        relationships {
            relation {
                firstName
                lastName
            }
            relationship
        }
    }
}

cURL:

curl 'localhost:8080/graphql/' \
-X POST \
-H 'content-type: application/json' \
--data '{ "query": "query { peopleByFirstName(firstName: \"Dan\") { firstName lastName id relationships { relation { firstName lastName } relationship }}}"}'

Postman:

Both of these methods return the same data (I would be worried if they didn’t):

{
  "data": {
    "peopleByFirstName": [
      {
        "firstName": "Dan",
        "lastName": "Newton",
        "id": "27a08c14-d0ad-476c-ba09-9edad3e4c8f9",
        "relationships": [
          {
            "relation": {
              "firstName": "Laura",
              "lastName": "So"
            },
            "relationship": "Wife"
          },
          {
            "relation": {
              "firstName": "Random",
              "lastName": "Person"
            },
            "relationship": "Friend"
          }
        ]
      },
      {
        "firstName": "Dan",
        "lastName": "Doe",
        "id": "3c07b717-8b9c-4d88-926f-c892be38ee85",
        "relationships": []
      },
    ]
  }
}

The most crucial factor here is that a POST request is used. It seems standard for GraphQL APIs to use POST requests for both fetching and mutating data. This threw me off for a while, as the error I received from the /graphql endpoint didn’t cause me to believe it was due to using the wrong HTTP verb.

For clarity, using the wrong HTTP verb (e.g. a GET) through Postman leads to the following response and logline:

{
    "timestamp": "2022-01-03T16:50:58.376+00:00",
    "status": 400,
    "error": "Bad Request",
    "path": "/graphql"
}

Resolved [org.springframework.web.bind.MissingServletRequestParameterException: Required request parameter 'query' for method parameter type String is not present]

Improving the registration of DataFetchers

The registration of DataFetchers could be improved as they are currently registered by injecting every DataFetcher by name into the GraphQLConfiguration class and manually associating each with a type and field name. This isn’t too bad right now since the application is small; if you argued that it already looks dodgy, then I’d agree with you.

To make the code more maintainable and extensible moving forward, we can introduce a new structure for defining DataFetchers and registering them.

We can achieve this by defining a new interface that specifies the type and field name a DataFetcher should be linked to:

/**
 * [TypedDataFetcher] is an instance of a [DataFetcher] that specifies the schema type 
 * and field it processes.
 *
 * Instances of [TypedDataFetcher] are registered into an instance of [RuntimeWiring] 
 * after being picked up by Spring (the instances must be annotated with @[Component]
 * or a similar annotated to be injected).
 */
interface TypedDataFetcher<T> : DataFetcher<T> {

  /**
   * The type that the [TypedDataFetcher] handles.
   *
   * Use `Query` if the [TypedDataFetcher] responds to incoming queries.
   *
   * Use a schema type name if the [TypedDataFetcher] fetches data for a single field
   * in the specified type.
   */
  val typeName: String

  /**
   * The field that the [TypedDataFetcher] should apply to.
   *
   * If the [typeName] is `Query`, then [fieldName] will be the name of the query the 
   * TypedDataFetcher] handles.
   *
   * If the [typeName] is a schema type, then [fieldName] should be the name of a single
   * field in [typeName].
   */
  val fieldName: String
}

By using TypedDataFetcher, you can retrieve all its implementations as Spring allows injecting of Lists containing all instances implementing an interface.

Combining this interface with modifications to the DataFetcher registration code in GraphQLConfiguration brings it all together:

@Configuration
class GraphQLConfiguration(private val dataFetchers: List<TypedDataFetcher<*>>) {

  // Create the GraphQL instance (no changes from the previous example).

  /**
   * Loops through all injected [TypedDataFetcher] instances and includes them in the output [RuntimeWiring] instance.
   */
  private fun buildWiring(): RuntimeWiring {
    val wiring = RuntimeWiring.newRuntimeWiring()
    for (dataFetcher in dataFetchers) {
      wiring.type(newTypeWiring(dataFetcher.typeName).dataFetcher(dataFetcher.fieldName, dataFetcher))
    }
    return wiring.build()
  }
}

The registration now uses the typeName and fieldName provided by each TypedDataFetcher, breaking the rigid link between DataFetcher implementations and the types, fields or queries they represent. Adding new TypedDataFetchers becomes straightforward after making this alteration; you create a new implementation, define the typeName and fieldName, annotate with it @Component, and the rest is handled for you.

Summary

GraphQL Java lets you do what the name suggests, support GraphQL queries within Java (or other JVM languages).

It does this through associating DataFetchers to types and fields that you write and pointing incoming queries their way. With the library handling the parsing of queries, it frees you to focus on implementing the DataFetchers that contain your application’s primary functionality.

I’d love to try and summarise how to use GraphQL Java further; however, I already struggled to write the post in the most coherent way as possible; there is no way I could condense it to a few sentences.

Anyway, the end!

Building a modular REST API with Javalin

Sat, 20 Nov 2021 00:00:00 GMT

Javalin is a simple and lightweight web framework for Java and Kotlin (and technically any JVM language). I recently had to dig into Javalin a bit for work, and I even made a few contributions to the codebase once I started to understand it. Now I want to write down what I’ve learnt while using it so future me can remember what I know now but will inevitably forget.

In this post, I will write about building a modular REST API using Javalin. I will also leverage a few other technologies to create an example, using Kodein for dependency injection and Exposed for database access (although I will omit all database code from this post).

Dependencies

For Javalin you’ll need:

<dependency>
  <groupId>io.javalin</groupId>
  <artifactId>javalin</artifactId>
  <version>4.1.0</version>
</dependency>

<!-- You must set up your own logging when using Javalin -->
<dependency>
  <groupId>ch.qos.logback</groupId>
  <artifactId>logback-classic</artifactId>
  <version>1.2.6</version>
</dependency>
<dependency>
  <groupId>org.apache.logging.log4j</groupId>
  <artifactId>log4j-to-slf4j</artifactId>
  <version>2.14.1</version>
</dependency>
<dependency>
  <groupId>org.slf4j</groupId>
  <artifactId>jul-to-slf4j</artifactId>
  <version>1.7.32</version>
</dependency>

<!-- You must set up your own JSON serialization/deserialization when using Javlin -->
<dependency>
  <groupId>com.fasterxml.jackson.module</groupId>
  <artifactId>jackson-module-kotlin</artifactId>
  <version>2.13.0</version>
</dependency>

For dependency injection:

<dependency>
  <groupId>org.kodein.di</groupId>
  <artifactId>kodein-di-jvm</artifactId>
  <version>7.8.0</version>
</dependency>

Starting the Javalin server

Starting the Javalin server is nice and easy:

Javalin.create { config ->
  // Apply your configuration or use the [create] overload that takes no arguments
}.routes {
  // Register endpoints within [route] block
}.start(8080)

The amount of code here is small as most of the logic revolves around the functionality of the HTTP endpoints. We’ll see how to register endpoints in the next section.

See the Javalin documentation for a different explanation.

Registering HTTP endpoints with Javalin

Below is how I’ve been registering endpoints within my code (which is slightly different from what you’d see in the Javalin documentation):

import io.javalin.apibuilder.ApiBuilder.delete
import io.javalin.apibuilder.ApiBuilder.get
import io.javalin.apibuilder.ApiBuilder.path
import io.javalin.apibuilder.ApiBuilder.post
import io.javalin.apibuilder.ApiBuilder.put
import io.javalin.http.Context

fun register() {
  // Routes the nested registration methods under the "/people" path
  path("/people") {
    // I use method references but you could put the callback/lambda/function here directly instead
    get("/", ::all)
    // {id} represents a Path Variable named "id" that can be accessed by the endpoint
    get("/{id}", ::get)
    post("/", ::post)
    put("/{id}", ::put)
    delete("/{id}", ::delete)
  }
}

// Every endpoint function must take a [Context] as an argument
private fun all(ctx: Context) {
  // implementation
}

private fun get(ctx: Context) {
  // implementation
}

// Other endpoints

The registration above uses Javalin’s ApiBuilder for static builder functions. It is important to note that these functions will only work within a routes block, as shown in the previous section.

Calling path("/people") first makes it convenient to register multiple endpoints with the sane path prefix. If this was not done, then the registration code would look like:

fun register() {
  get("/people", ::all)
  get("/people/{id}", ::get)
  post("/people", ::post)
  put("/people/{id}", ::put)
  delete("/people/{id}", ::delete)
}

It’s not exactly the biggest issue in the world, but I’d personally prefer to not have to write “people” so many times.

After the path call is done, you register the endpoints that will process incoming requests by providing a string path and a callback function. In the example, method references specify the callback function, but these could be switched out, and the function could be written directly inside the builder methods when registering the endpoint. For example, the following are equivalent:

// Method reference
get("/", ::all)
// Function
get("/") { ctx: Context ->
  // Implementation
}

At the end of the day, choose the method that suits you and your code’s styling.

Finally, a few of the routes contain {id}, which denotes that a path variable with the name id exists. This id can then be accessed within the endpoint’s callback using Context.pathParam and passing "id" as the input. You’ll see examples of this later on in the post.

Other examples of registering endpoints can be found in the Javalin documentation.

Writing an HTTP endpoint

We’ve covered how to register endpoints, now onto implementing them.

I’ll break down a few examples for implementing a REST API in this section.

Before we look at any code, I want to point out the importance of the Context class. You’ll use this class for any request or response related functionality. For example:

Retrieving a path variable using Context.pathParam.
Accessing a query parameter using Context.queryParam.
Returning JSON using Context.json.
Setting a response’s status with Context.status.

A complete list of Context’s methods can be found in Javalin’s documentation.

Now that you know the importance of the Context class let’s look at some examples.

Starting with the simplest endpoint:

private fun all(ctx: Context) {
  val people: List<Person> = personRepository.findAll().map { entity -> entity.toPerson() }
  ctx.json(people)
}

The only Javalin related part of this implementation is the call to Context.json. This function delegates to Jackson and converts the input object into JSON, taking the output and setting it as the response to the caller.

It’s worth pointing out that the call to Context.json doesn’t instantly return a response to the caller; the sending of the response only happens once the endpoint’s function has finished. However, I imagine a lot of the time it is probably the last thing you’d do.

The next snippet includes a few more of Javalin’s features:

private fun get(ctx: Context) {
  val id: UUID = UUID.fromString(ctx.pathParam("id"))
  val person: Person = personRepository.find(id)?.toPerson() ?: throw NotFoundResponse()
  ctx.json(person)
}

The implementation starts by retrieving the id path variable using Context.pathParam to find a record.

If the record doesn’t exist, it throws a NotFoundResponse exception to tell Javalin to return a 404 Not Found code. Technically, you could mimic this behaviour by manually setting the status, but using an exception allows you to exit the function straight away.

Finally, if the record exists, it returns it using Context.json (the same way as before).

This last snippet does 2 things differently from what’s above:

private fun post(ctx: Context) {
  val person: Person = ctx.bodyAsClass()
  val persisted: Person = personRepository.persist {
      firstName = person.firstName
      lastName = person.lastName
  }.toPerson()
  ctx.json(persisted)
  ctx.status(HttpCode.CREATED)
}

This endpoint extracts the request’s JSON body and converts it to a Person object (or whatever object you specify), persists the person and returns the persisted person (which has the Person’s id set in this case). To better represent the fact that this endpoint creates a new record, it calls Context.status to set the response code to 204 Created.

Modular registration of endpoints

Breaking up your code into modules allows you to separate concerns and make your code easier to manage and understand. This section will show how I set the groundwork in my example application to follow a modular structure.

I chose to achieve this using dependency injection with Kodein, allowing code to be split into modules that handle their own registration with a central module to initialise everything.

Kodein has a mechanism that allows Modules to be created that can be imported into the main DI block. In other words, modules can define Modules. I mean, the wording suggests that it’s a perfect fit for this scenario.

To define a Module, you need to call DI.Module and specify that module’s services:

object People {
  val module = DI.Module("People module") {
    bindSingleton { PersonRepository() }
    bindSingleton { PersonController(instance()) }
  }
}

In relation to this post, this Module is found in the :people module.

To import a Moudle into a DI block:

val di: DI = DI {
  import(People.module)
}

In relation to this post, this DI block is called within the :application module.

The snippets above will handle initialising instances of the classes referenced in the DI.Module function.

Note, there are ways we could achieve similar functionality without using dependency injection.

We’ve initialised some classes, but there’s still no mention of endpoints anywhere; let’s fix that.

The endpoints are contained within the PersonController class, which you’ve already seen above in the DI code. We need to access them and register them inside Javalin’s route block.

Below is the relevant code from PersonController:

class PersonController(private val personRepository: PersonRepository) : Controller {

  override val path = "/people"

  override val endpoints = EndpointGroup {
    get("/", ::all)
    get("/{id}", ::get)
    post("/", ::post)
    put("/{id}", ::put)
    delete("/{id}", ::delete)
  }

  // Endpoints (you've seen them throughout this post)
}

This is similar to the code in the Registering HTTP endpoints with Javalin section. However, I decided to change the code a bit because I found it slightly more descriptive.

An EndpointGroup is defined that contains the PersonController’s endpoints along with its base path.

The Controller interface is what requires these two vals:

interface Controller {

  val path: String

  val endpoints: EndpointGroup
}

In relation to this post, this interface is found in the :web module.

There’s nothing fancy about this interface; its job is to define some structure to classes containing endpoints and make them discoverable for registration.

The last piece of the puzzle is shown below:

val di: DI = DI {
  import(People.module)
}

Javalin.create().routes {
  val controllers: List<Controller> by di.allInstances()
  controllers.forEach { path(it.path, it.endpoints) }
}.start(8080)

This code creates a Javalin server instance and calls its routes method. Inside the block, the DI instance finds all classes implementing the Controller interface. This was the primary reason for this interface, as it provides a handy way to access classes when used in conjunction with the dependency injection mechanism. With the Controller instances retrieved, each instance’s base path and endpoints are registered using ApiBuilder.path.

After running this code, the Javalin server is up and running with fully functional endpoints without any business logic found within the module that started it.

Summary

You can use Javalin to build modular REST APIs, allowing you to split out your code by functionality while still bringing it all together when you start your webserver.

The code below condenses the snippets and examples used through this post:

// In the `web` module
interface Controller {

  val path: String

  val endpoints: EndpointGroup
}

// In the `people` module
object People {
  val module = DI.Module("People module") {
    bindSingleton { PersonRepository() }
    bindSingleton { PersonController(instance()) }
  }
}

class PersonController(private val personRepository: PersonRepository) : Controller {

  override val path = "/people"

  override val endpoints = EndpointGroup {
    get("/", ::all)
    get("/{id}", ::get)
    post("/", ::post)
    put("/{id}", ::put)
    delete("/{id}", ::delete)
  }

  private fun all(ctx: Context) {
    val people: List<Person> = personRepository.findAll().map { entity -> entity.toPerson() }
    ctx.json(people)
  }

  private fun get(ctx: Context) {
    val id: UUID = UUID.fromString(ctx.pathParam("id"))
    val person: Person = personRepository.find(id)?.toPerson() ?: throw NotFoundResponse()
    ctx.json(person)
  }

  // POST, PUT and DELETE endpoints
}

// In the `:application` module
fun main() {

  val di = DI {
    import(People.module)
  }

  Javalin.create().routes {
    val controllers: List<Controller> by di.allInstances()
    controllers.forEach { path(it.path, it.endpoints) }
  }.start(8080)
}

You can view all the code on Github.

Documenting a Spring REST API following the OpenAPI specification

Sun, 26 Sep 2021 00:00:00 GMT

The OpenAPI specification defines how to write HTTP APIs that can be consumed by any programming language and provide insight into the APIs’ functionality without access to source code or documentation. In other words, following the specification makes it easier for consumers to understand what it does and how to use it. Tools, such as Swagger, can then be used to display documentation without developers maintaining documentation separate from an API’s code.

All these described points translate into happier users while mitigating some of the burdens you’ll face while supporting your APIs.

This post will show how to produce an OpenAPI compliant and documented REST API using Spring and springdoc-openapi. We will be using standard Spring @RestControllers to achieve this.

You can read more on the OpenAPI specification itself here.

What is springdoc-openapi?

springdoc-openapi is a library that automatically generates documentation for OpenAPI compliant REST APIs built with Spring. It inspects an application’s code at runtime, looking at HTTP endpoints, their annotations and returned objects and uses this information to generate documentation.

The library provides support for standard (don’t know what to really call this) Spring web code (e.g. endpoints defined in @RestControllers/@Controllers) and Spring Webflux.

To top everything off, springdoc-openapi can start a Swagger UI based on the documentation it generates, making it even easier for users to understand and interact with your APIs.

springdoc-openapi dependency

To include springdoc-openapi in your project, use the following dependency:

<dependency>
    <groupId>org.springdoc</groupId>
    <artifactId>springdoc-openapi-ui</artifactId>
    <version>1.5.10</version>
</dependency>

This dependency should be used alongside Spring Boot.

For more information on supported functionality and their dependencies, refer to the official springdoc-openapi documentation.

Taking your first steps with springdoc-openapi

After including the library in your project, how do you start generating documentation?

Well, it just works. As with most Spring Boot related libraries, it runs without requiring you to do anything and scans your @RestControllers (and @Controllers) for HTTP endpoints, generating OpenAPI related documentation based on the scan results.

The generated documentation can then be accessed by a Swagger UI which is started by default when using springdoc-openapi and is accessible by going to <server:port>/swagger-ui.html, which will redirect you to a slightly different URL straight after (<server:port>/swagger-ui/index.html?configUrl=/v3/api-docs/swagger-config). Going to /swagger-ui/index.html will instead display Swagger’s pet store example, which probably isn’t what you wanted to see.

Now, let’s look at what documentation is generated for a standard @RestController. We will use Swagger to observe the output documentation (this will continue for all following examples).

@RestController
@RequestMapping("/people")
public class PersonController {

    private final PersonRepository personRepository;

    PersonController(PersonRepository personRepository) {
        this.personRepository = personRepository;
    }

    @GetMapping
    public ResponseEntity<Iterable<Person>> all() {
        return ResponseEntity.ok(personRepository.findAll());
    }

    @GetMapping("/{id}")
    public ResponseEntity<Person> get(@PathVariable("id") UUID id) {
        return personRepository.findById(id).map(ResponseEntity::ok)
            .orElseThrow(() -> new NoSuchElementException("Person with id: " + id + " does not exist"));
    }

    @PostMapping
    public ResponseEntity<Person> post(@RequestBody Person person) {
        Person saved = personRepository.save(
            new Person(
                UUID.randomUUID(),
                person.getFirstName(),
                person.getLastName()
            )
        );
        URI uri =
            MvcUriComponentsBuilder.fromController(getClass())
                .path("/{id}")
                .buildAndExpand(saved.getId())
                .toUri();
        return ResponseEntity.created(uri).body(saved);
    }

    @PutMapping("/{id}")
    public ResponseEntity<Person> put(@PathVariable("id") UUID id, @RequestBody Person person) {
        if (!personRepository.existsById(id)) {
            return ResponseEntity.notFound().build();
        } else {
            Person saved = personRepository.save(
                new Person(
                    id, 
                    person.getFirstName(),
                    person.getLastName()
                )
            );
            return ResponseEntity.ok(saved);
        }
    }

    @DeleteMapping("/{id}")
    public ResponseEntity<Void> delete(@PathVariable("id") UUID id) {
        if (!personRepository.existsById(id)) {
            return ResponseEntity.notFound().build();
        } else {
            personRepository.deleteById(id);
            return ResponseEntity.noContent().build();
        }
    }
}

The output for the controller above is shown below:

We can then expand any of the endpoints, for example, the first GET endpoint:

Great, so we have some documentation and a swanky Swagger UI to display it. However, several areas could be improved.

Let’s focus on the expanded GET endpoint shown above. Here’s what is missing (from my knowledge anyway):

A description of what the endpoint does.
A description of what the id parameter is or how it’s used.
Possible error codes are missing; it’s unlikely that 200 Ok would always be returned.
The media type is not defined/restricted.

There are a few others outside of the GET endpoint as well.

The title grouping the endpoints together is not helpful.
There is no description of what the group does or is for.

In the next section, we will address these shortcomings.

Using OAS annotations for improved documentation

As mentioned above, you will miss out on a lot of documentation if you rely on springdoc-openapi to do everything without you giving it a helping hand.

Luckily, another library provides ways to add this extra documentation that is compatible with springdoc-openapi. The swagger-annotations library gives you access to numerous additional annotations (as the name suggests) to document your APIs as the OpenAPI specification wants you to do.

You will have access to swagger-annotations without further changes as it is an API dependency of springdoc-openapi.

You can put the annotations from swagger-annotations onto your @RestControllers and their methods to specify additional information about your APIs. When springdoc-openapi starts scanning your code as before, it now has more details of what to include in the generated documentation.

Let’s have a look at an example. Below is the same PersonController as before, minus most of the code so we can focus on issues mentioned earlier:

@RestController
@RequestMapping("/people")
@Tag(name = "People", description = "Endpoints for managing people")
public class PersonController {

    @GetMapping("/{id}")
    @Operation(
        summary = "Finds a person",
        description = "Finds a person by their Id.",
        tags = { "People" },
        responses = {
            @ApiResponse(
                description = "Success",
                responseCode = "200",
                content = @Content(mediaType = "application/json", schema = @Schema(implementation = Person.class))
            ),
            @ApiResponse(description = "Not found", responseCode = "404", content = @Content),
            @ApiResponse(description = "Internal error", responseCode = "500", content = @Content)
        }
    )
    public ResponseEntity<Person> get(
      @PathVariable("id") 
      @Parameter(description = "The Id of the person to find.") 
      UUID id
    ) {
        return personRepository.findById(id).map(ResponseEntity::ok)
            .orElseThrow(() -> new NoSuchElementException("Person with id: " + id + " does not exist"));
    }

    // Other endpoints
}

The first thing to notice is that there is way more code now. In fact, the content in the annotations is far greater than the functional code of the snippet, highlighting the work it takes to properly document your APIs against the OpenAPI specification.

So what is going on in the example?

Let’s start with the annotation on the class itself:

@Tag - Groups endpoints together, by their name. If no tag information is provided by a class and its methods, then a tag named after the class is used. You can also define a description of the tag to describe the overall functionality of a group of endpoints.

Now onto the GET endpoint, below I have expanded on the two annotations with the most impact:

@Operation - Adds details about what the endpoint does, including some properties to achieve this.
- summary - The summary (or title) of an endpoint.
- description - The description of the endpoint.
- tags - What tags to group the endpoint under. You don’t have to specify an array here if there is a single tag; writing tags = "People" also works.
- responses - An array of @ApiResponses that details what the endpoint can return, allowing you to show what happens on successful requests as well as unsuccessful or erroneous requests.
@ApiResponse - Provides more information on what an endpoint returns. By filling in a number of its properties, users will better understand your API.
- description - The name or description of the endpoint.
- responseCode - The response code that is returned.
- content - Allows you to specify what media types can be returned along with an example of the response body by using @Schema. In the example above, it causes Swagger to display JSON representation of the Person class.
@Parameter - Include more information about a request parameter.
- description - The description of the parameter.

There are many more properties available to these annotations than what is shown above.

It’s great and all me telling you what it does, but seeing the impact on the generated Swagger UI should cement your understanding of the annotations.

This time around, the title/tag of the grouped endpoints is People and contains the description written in the @Tag annotation.

Now let’s expand the GET endpoint since it contains the most changes. I’ve highlighted the differences since there are quite a few.

To clarify, the differences are:

The endpoint has a summary and description.
There is a description of the id argument.
The description of the 200 Ok response has been changed and also specifies the media type of the returned data.
Extra response codes have been included to cover failure scenarios.

That should give you a decent overview of using the OAS/Swagger annotations to better document your APIs in conjunction with springdoc-openapi.

If you want to see the alterations I made to the other endpoints shown in the original code, you can find them on GitHub.

Extra tips after playing around with `springdoc-openapi` and the OAS annotations

I thought I’d add this section to cover a few things I ran into when playing with the springdoc-openapi library. Useful observations that I didn’t want to scatter throughout the happy path shown in the rest of this blog post.

Let’s get started.

Don’t use the @ApiResponses annotation or multiple @ApiResponse annotations if you want to show response links.

There seems to be an issue that the library has been suffering from, which prevents it from correctly generating response links in some situations.

To resolve this, stick to using @Operation’s responses property.

If this is not done, the Links on the bottom right will show empty even if you specified a value.
If you don’t want example content to be generated for some responses, you will need to use an empty @Content annotation. Not using it will cause the library to create the example, as this is its default behaviour.

This was shown previously, but I didn’t highlight it at that point. Here is an example of the code again:
```
@ApiResponse(description = "Internal error", responseCode = "500", content = @Content)
```

To construct an example response that consists of an array of objects, use @Content.array in conjunction with the @ArraySchema annotation. For example:

@ApiResponse(
    description = "Success",
    responseCode = "200",
    content = {
        @Content(
            mediaType = "application/json",
            // Use the `array` property instead of `schema`
            array = @ArraySchema(schema = @Schema(implementation = Person.class))
        )
    }
)

It is worth noting that the library will not generate anything for a @RestController method returning a List, so you will have to use code similar to the above if you want to document the endpoint.

Summary

You should now understand how to use springdoc-openapi to generate documentation for your RESTful APIs that adhere to the OpenAPI specification.

You’ve seen how springdoc-openapi will generate documentation and display it with a Swagger UI without you having to do anything. However, you’ve also seen how the output lacks useful information without manual configuration by using the OAS annotations (via swagger-annotations).

At the end of the day, you’ll need to go through the pain of documenting your REST APIs so that your users, customers, teammates, and even your future self will have more enjoyable experiences with your work.

Setting up a Polywork subdomain for a Netlify hosted site

Wed, 08 Sep 2021 00:00:00 GMT

Polywork allows you to point a subdomain of your own site to your Polywork profile/feed. They do have instructions on setting this up; however, I made quite a few mistakes when trying this myself. I actually managed to make my site inaccessible due to screwing up the DNS records. I didn’t really know what I was doing and thought I was doing the right things while following their instructions, but I was clearly wrong.

This post will cover how to set up a Polywork subdomain for sites hosted by Netlify. I’m being specific because my own site uses Netlify, and the instructions on Polywork don’t cover this exact scenario.

Navigate to your Polywork settings page and scroll down to the Setting up a custom domain for your profile section. I started the list from 0 to sync it with Polywork’s listed out steps.
Choose your provider. Since this post is about Netlify hosted sites, you’ll need to choose the My Provider isn't here, but I know what I'm doing option from the drop-down box. It is worth noting here that since I brought my domain name from Google Domains, I thought that I needed to select Google Domains from the drop-down box. This proved incorrect as it conflicted with how Netlify hosts my site.

At the time of writing, there wasn’t a Netlify specific option. This might be different by the time you’re reading this.
Choose the name of your custom domain. I’m focusing on using a subdomain here, so I chose timeline.lankydan.dev for mine where lankydan.dev is my site’s name.
Generate a custom DNS target. This will be used later to connect your subdomain to your Polywork feed.
This is the point where we deviate from the steps listed out by Polywork.

You’ll want to navigate to your Netlify DNS settings page.

If you want to get there directly, use the following URL customised to your account/team and site name - https://app.netlify.com/teams/<team_name>/dns/<site_name>.

To navigate there from your Netlify homepage, do the following:
1. Go to your Netlify homepage/overview page
2. Select Domains, not the site you want to edit.
3. From the Domains page, now select the site you want to modify.
You will then see a page like the following:

You can now modify your site’s DNS settings.
Press Add new record. This will open a dialogue box to enter your configuration into.

You’ll want to fill it in with the following:
- Record type - CNAME.
- Name - The name of your desired subdomain, the same as chosen in Polywork’s settings previously (setting mine to timeline creates the overall subdomain of timeline.lankydan.dev).
- Value - The custom DNS target generated in Polywork’s settings.
- TTL (Time To Live) - Provide a value you want or leave it empty to default to 1 hour (3600 seconds).
Once filled with information similar to the above, press save.
Your DNS record is now added and should look like the following when selected.

Everything should be working now, meaning when you access your newly setup subdomain, it will redirect you to your Polywork feed.

Depending on how fast you tried to do this, you may run into some HTTPS issues, as the certificate for your subdomain hasn’t been acknowledged yet. This happened to me; waiting 5 minutes or so and trying again resolved the problem.

I hope that works nicely for you, and enjoy your new custom Polywork URL!

Perfect is the enemy of good

Mon, 30 Aug 2021 00:00:00 GMT

Perfect is the enemy of good.

Better a diamond with a flaw than a pebble without.

These are the quotes handed to me when I googled what I personally thought the quote was (“Perfection is the enemy of done”). Honestly, I don’t know if they’re right; they sound good, and I pasted them from Wikipedia, so there must be something true about them, right?

Anyway, I wanted to write down my own thoughts on aiming for perfection in Software Engineering. This is a subject I’ve discussed with one of my teammates as a means to improve themself; however, I don’t think I covered it particularly well with them. Hopefully, in this post, I’ll clarify and condense my thoughts, making it easier to provide feedback on this topic in the future.

The way I see it, aiming for perfection isn’t wrong. It’s an excellent destination to aim for at the start of a change, bug or feature. You want to create the best solution possible. If you don’t think like that, then you’ll never be able to make significant strides in improving whatever you’re working on.

However, aiming for perfection can lead you down a dead-end. You become so consumed by how you must reach the destination you envisioned that you become blind to all the signs telling you otherwise. Overlooking all the different routes you can take. Paying no attention to the directions that others give you. Where you finally reach the dead-end that your choices lead you to, and instead of turning back around, you pull out a shovel and try to dig a tunnel past this blockade. Unfortunately, you’re standing on concrete, and there is nothing left to try. Leaving you frustrated, anxious and maybe even distraught as you were so sure you were right, and it still went so wrong.

Now, I’m going to end my short stint of creative writing and really get to the points I was trying to make in the paragraphs above and how they relate to Software Engineering.

When you are working on a feature, there is a good chance you’ll produce a design or have an idea of what you want to do before you start coding. At this point, you might have decided the structure of your solution, decided to introduce another tool, library or framework, or even thought of specific code you want to write.

This is good; thinking ahead and ironing out potential issues is hugely beneficial.

Unfortunately, the likelihood of you perfectly identifying all the potential holes in your design is pretty low. Something almost always goes wrong; you could introduce an unforeseen circular dependency, the library you brought in might not work how you thought it did, or the code you wanted to write doesn’t even compile. All of these could throw you off.

Despite all these pitfalls, you could still think you’re on the correct path and double-down until you reach a dead-end. At this point you can feel defeated, as you have invested a lot time and effort to reach this point.

I know I’ve done this before, and it doesn’t feel great. I know teammates of mine have done this before, and it left them feeling apprehensive about what others might think. Each situation required others to step in and pull us out of our holes, which wasn’t easy because we had dug so deep, but eventually, we climbed out.

Everything up to this point has detailed what and where things can go wrong, but I haven’t given you anything constructive to work on. To change that, I’ll close out this post with my thoughts on preventing yourself from reaching the point of no return as you strive to attain your perfect solution.

Personally, I think you should do the following:

Adjust your route to your desired solution as needed - The first step is to introduce flexibility into your solutions. You should still strive to reach your desired outcome, but you should be willing to adjust the route you take. Being flexible will help you tackle the minor setbacks that always arise when working on a feature or bug.
Seek guidance or assistance if you think you are losing your way - It is a good idea to seek help from others if you feel that you’re veering off course. Ideally, you would turn to a colleague that understands what you are working on, however as you likely know already, sometimes the mere act of explaining your thought process to someone else can help you unlock new ideas or clarify your existing thoughts. The main point here is to stop and think before moving forward. Leaning on your teammates for assistance, even if they are only acting as a rubber duck, can help decrease the time it takes to correct your path.
Alter your solution as more significant problems arise - As larger issues occur while you work towards your preferred outcome, you will need to consider if delivering it is possible, this includes whether it literally cannot be done, cannot be done within the required timeframe or requires knowledge that you do not possess. If you have run into a critical flaw in your solution that prevents it from ever working, you’ll definitely need to change course; however, this assumes you realise the severity of the problem, which isn’t always obvious…
Do not wait too long to repeat the previous steps - If you can incorporate the steps above into your work, then you should be on a good path to reliably complete any piece of work assigned to you. The last helpful point I could think of is ensuring that you find the right balance of perseverance and adjustability. You want to try and achieve your perfect solution, but also be aware when you’re going in the wrong direction. I’d say it’s best to start on the end of persevering and then dial it down as you get more of a feel for your personal working style. Starting on the opposite end of the scale, where you’re super adjustable, will likely prevent you from achieving anything because you won’t know when to commit to an idea, so I’d avoid this if I was you.

At the end of the day, it’s about finding out what best works for you. If you’re successfully completing your work to a high quality and on time, then you’re on the right track.

Everything I have written down here comes from my own thoughts and observations from interacting with my colleagues and reflecting on my own working style. The points listed out earlier are the results of my thinking.

What I cannot do, is guarantee that they fit for you, only you’ll know that. Ideally, you’ll reflect on how you work to figure this out yourself and use what I’ve written here as guidance. Without self-reflection, self-improvement is much harder to achieve.

Intro to Kafka - Ordering related records

Fri, 06 Aug 2021 00:00:00 GMT

Ordering records in Kafka requires you to make a conscious decision to construct your system in a way that supports ordering. You don’t have to do this, as it depends on the functionality of your system. You don’t need ordering for all situations, however, many problems require ordering between events (or records in Kafka) to safely and adequately implement a process.

So how can records be ordered?

We can answer the question by using our knowledge about topics, partitions and how consumers retrieve records. The important information we need is as follows:

Consumers read records from partitions inside topics.
Each new record is inserted with an incremented offset based on the previous maximum offset.
These records are read from the lowest to highest offset values.
Records with the same key are always stored in the same partition.

By combining this information, we can infer that ordering between records can be maintained when they have the same key. By ensuring that all records that have some relationship between them have the same key, you can guarantee that they will be placed into the same partition, where a consumer will now retrieve them in the order they were stored.

I have also put together a diagram summarising the information above:

If you think about this from a more application-level perspective and replace “related records” with something with more context, say, a customer identifier, it becomes easier to understand how you could leverage the ordering provided by Kafka.

All records for a customer would have their keys set to the customer’s identifier, which guarantees Kafka will order any events that this customer triggered.

Code-wise, this might look like:

fun pay(customerId: UUID, amount: BigDecimal) {
  producer.send(
    ProducerRecord(
      PAYMENTS, // topic
      customerId, // key
      amount // amount
    )
  )
}

The customerId is a value that represents a single customer. By using this value as the key for all records related to a single customer, Kafka will place the records into the same partition. When producers read from the PAYMENTS topic, they will receive records in the order they were created, making writing the downstream processing logic more straightforward.

Now, I’m going to chuck a bit more information at you, that you don’t need to know. However, it will fill in the gaps you might already be questioning on how it works.

More precisely, how does a topic decide which partition to store a record in?

A formula determines the a record’s partition:

hash($key) % number_of_partitions

Where:

hash A function that hashes the input $key with Kafka’s chosen digest algorithm.
$key The key of a record.
number_of_partitions - The number of partitions in the topic.

% (Mod/Modulo) returns the remainder when dividing the left side by the right (e.g. 20 % 7 = 6)

Below is a diagram that visualises the process of determining the partition of a record:

Note, this is assuming that the default formula for calculating a record’s partition is used. You can configure this if desired, more information can be found in the Kafka documentation and in their JavaDocs (although the amount of information is honestly a bit lacking).

Summary

Records within a single partition are ordered by insertion order.
Records with the same key are stored in the same partition (when using default settings).
This ensures ordering between records with the same key.
The formula for calculating a record’s partition is hash($key) % number_of_partitions.

Intro to Kafka - Consumer groups

Sun, 20 Jun 2021 00:00:00 GMT

Consumers can form groups, aptly named “consumer groups”. These consumer groups determine what records a consumer receives. I know that sounds quite vague, but honestly, I can’t think of any other short and snappy way to summarise this topic.

To be more specific, consumer groups do the following:

Groups consumers by function
Shares a topic’s partitions between consumers in the same group

The following sections expand on these points.

Groups consumers by function

Consumer groups allow you to relate a set of consumers working together to perform a single function, process or task. Every consumer group will process every record in a topic independently from other groups.

For example, suppose you have a topic representing sales data. In that case, one group could maintain an aggregate of the sale values while another is passing it to a downstream system for later processing. Both of these groups will run independently and manage their offsets, even when subscribing to the same topic.

The diagram below has two consumer groups who receive all the records stored in the topic:

Shares a topic’s partitions between consumers in the same group

Every consumer inside a consumer group is assigned a number of a topic’s partitions, ensuring that records are shared between the group while also keeping each partition’s records in order. This mechanism is made more efficient by keeping the assigned partitions the same until introducing a new consumer to the group. At this point, Kafka reassigns the partitions between the group’s consumers, and this process is called “rebalancing”. Consumers can also be assigned multiple partitions if there are more partitions than consumers within a group.

The diagram shows the sharing of partitions within a single consumer group:

Introducing another consumer group does not affect the assignment of existing groups, as shown below:

The system will rebalance when a new consumer is added to one of the groups, visualised below:

A new consumer was added to group 1, which caused the assigned partitions to rebalance between them.

Creating a consumer group

You don’t have to create a group explicitly. Instead, it’s done as part of creating a consumer.

When constructing a consumer, pass in the group.id property of the group. In fact, this field is required, so any Kafka example or tutorial you’ve followed will have already included this piece of code.

fun createConsumer(): Consumer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  // The important stuff is below!
  props["group.id"] = "test"
  // That's the end of the important stuff!
  props["enable.auto.commit"] = "true"
  props["auto.commit.interval.ms"] = "1000"
  props["key.deserializer"] = "org.apache.kafka.common.serialization.StringDeserializer"
  props["value.deserializer"] = "org.apache.kafka.common.serialization.StringDeserializer"
  return KafkaConsumer(props)
}

Initialising more consumers adds them to the existing group without you having to do anything else. Kafka makes this work exceptionally smoothly since the Broker knows about all the consumers in the group and can therefore handle consumers joining (or leaving) a group.

How partitions are assigned

The chosen “assignment strategy” determines the assignment of partitions within a group.

The default strategy is the RangeAssignor, which I would rather not explain in my own words, so I’ll quote Kafka’s JavaDocs instead.

The range assignor works on a per-topic basis. For each topic, we lay out the available partitions in numeric order and the consumers in lexicographic order. We then divide the number of partitions by the total number of consumers to determine the number of partitions to assign to each consumer. If it does not evenly divide, then the first few consumers will have one extra partition.

For example, suppose there are two consumers C0 and C1, two topics t0 and t1, and each topic has 3 partitions, resulting in partitions t0p0, t0p1, t0p2, t1p0, t1p1, and t1p2.

The assignment will be:

C0: [t0p0, t0p1, t1p0, t1p1] C1: [t0p2, t1p2]

You can configure a consumer group’s assignment strategy by using the partition.assignment.strategy property and defining the qualified class name of the assignor to use.

More information on the different assignors can be found in Kafka’s documentation.

Summary

Consumer groups in Kafka allow you to:

Group consumers by their function in a system.
Split the processing load of a topic by sharing its partitions between consumers in a group.

You’ve seen how to create a consumer and place it in a group or add one to an existing group by specifying its group.id during initialisation. You will hopefully also recall that Kafka does the rest of the work and that selecting the group.id is all you have to do.

By finishing this post, we have now covered the core parts of Kafka (at least in my opinion), and you should have enough knowledge to start building applications using it. Probably nothing fancy, but you’ll at least have the foundational understanding to get started.

I’ll look at particular topics more closely in future posts that I might have brushed over previously, or I’ll cover new material as I learn it myself.

Intro to Kafka - Consumers

Sun, 09 May 2021 00:00:00 GMT

Consumers make up the Kafka client code that reads records from topics, working at the opposite end of the chain as producers and benefiting from the same client features such as connection pooling and network buffering. Thus, providing you with a concise API that still allows you to write highly effective applications.

The diagram below visualises the interaction between a consumer and broker, where the consumer receives records stored in a topic.

In the following sections of this post, we will take a closer look at how consumers work and the client code needed to leverage them in your applications.

The code written in this post is in Kotlin, but similar code should be applicable for any JVM language. Furthermore, clients in other languages will also look virtually the same.

Creating a consumer

The first step is creating a KafkaConsumer:

fun createConsumer(): Consumer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  props["group.id"] = "test"
  props["enable.auto.commit"] = "true"
  props["auto.commit.interval.ms"] = "1000"
  props["key.deserializer"] = "org.apache.kafka.common.serialization.StringDeserializer"
  props["value.deserializer"] = "org.apache.kafka.common.serialization.StringDeserializer"
  return KafkaConsumer(props)
}

When creating a KafkaConsumer, you must provide its configuration via a Properties object or a Map<String, Object>. In addition, KafkaConsumer’s constructor also accepts values for the key.deserializer and value.deserializer properties.

You can find the available configuration options in Kafka’s documentation.

Now that you have an initialised consumer, you’ll need to subscribe to some topics to receive records.

Subscribing to topics

A consumer must subscribe to topics to retrieve the records stored in them. To be clear, a single consumer can subscribe to multiple topics at once. You do not have to make a separate consumer per topic.

A consumer subscribes to topics using its subscribe method (seems pretty obvious, I know…).

consumer.subscribe(listOf("string name of topic"))

Even though subscribe sounds straight forward, it does quite a lot of work behind the scenes, dynamically assigning a consumer the partitions it receives records from. The assignment of partitions becomes essential when you start writing more exciting applications that contain a more significant number of consumers.

Polling (retrieving) records

After a consumer subscribes to some topics, it needs to request records to pass them off to the rest of the application. This process is called polling and is done using the poll function.

When calling poll, you must pass in a timeout duration to allow your application to move along if no records are available for retrieval.

ConsumerRecords<K, V> poll(Duration timeout);

When no records are passed to a consumer within the timeout duration, it will progress with an empty collection of records.

The objects returned from poll are ConsumerRecords. These are similar but different from the ProducerRecords used to create records stored within a topic.

ConsumerRecords have several extra fields and a few that become required compared to ProducerRecords.

A ConsumerRecord contains the following fields:

Topic - The topic to the record was stored in.
Partition - The partition to the record was stored in.
Timestamp - The timestamp of the record.
Key (optional) - The key of the record.
Value (optional) - The value of the record.
Headers - Extra metadata to go alongside the record’s value.
Offset - The offset that the record holds within its partition.
Timestamp type - Indicates how the record’s timestamp value was generated.
Serialized key size - The size of the serialized, uncompressed key in bytes. If key is null, the returned size is -1.
Serialized value size - The size of the serialized, uncompressed value in bytes. If key is null, the returned size is -1.
Leader epoch - The current value of the leader epoch (this StackOverflow post seems to give some information on how this value is generated).

You don’t have to worry about generating these fields as you will only be receiving them after calling poll. The values are either defined by a ProducerRecord or generated by Kafka.

Code-wise, polling records is done as follows:

val records: ConsumerRecords<String, String> = consumer.poll(Duration.ofSeconds(5))
for (record: ConsumerRecord<String, String> in records) {
  println("Consumer reading record: ${record.value()}")
}

ConsumerRecords implements Iterable and can therefore be iterated over directly.

poll returns a ConsumerRecords object which provides some convenience functions and can also be iterated over directly. For example, it allows you to extract all the records for a particular topic, which is handy since you can subscribe to multiple topics in a single consumer.

You’ll then want to loop over the returned ConsumerRecords and process them in whatever way your application desires.

Offsets

Every record stored in a partition is assigned an offset which denotes how far they are from the start of the partition. Adding new records increases the offset, guaranteeing that more recent records will always have a higher offset than older records (based on time stored).

Offsets are not significant when persisting new records as this part is all handled by Kafka; however, the concept plays an important role when consuming records.

Consumers use partition offsets to keep track of what records have been processed. As it polls records and computes whatever needs computing, it needs to note down the offsets of these records, so that the consumer knows what point to start polling records from in case of failure.

Committing offsets

The process of keeping track of a consumer’s offset is known as “committing”. In that it commits the current offset for a partition to storage.

A consumer does not store its offsets locally. Instead, it sends them to the Kafka broker, who then persists them to disk.

Maintaining offsets in this way keeps the importance of consumer clients low within the context of the whole application. Starting, stopping and even destroying the machines running the consumers is possible without preventing the application from recovering. You can initialise a new consumer client and instantly continue processing from where the previous consumer was.

Furthermore, it allows you to focus your efforts on securing the broker’s machines rather than maintaining maximum security on many consumer clients. The broker contains the critical information, and keeping offsets in this way works to ensure this.

To commit a consumer’s offsets to the Kafka broker, you should use one of the following methods:

// Consumer.java

// Synchronous offset committing
void commitSync();
void commitSync(Duration timeout);
void commitSync(Map<TopicPartition, OffsetAndMetadata> offsets);
void commitSync(final Map<TopicPartition, OffsetAndMetadata> offsets, final Duration timeout);
// Asynchronous offset commtting
void commitAsync();
void commitAsync(OffsetCommitCallback callback);
void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback);

These methods all commit a consumer’s offsets and come with different degrees of control. For example, the overloads of commitSync and commitAsync that have no arguments are fully managed solutions that determine the offsets from the polled records — setting the offsets of each partition to the highest offset value contained in the received records.

The other options give you more control but require you to do more work and set the offsets yourself.

You will want to commit your offsets after processing records. This will form part of the standard process when working with Kafka. Poll records, process them and commit the updated offsets to confirm that they should not be received again.

The code for this process will look like:

val records: ConsumerRecords<String, String> = consumer.poll(Duration.ofSeconds(5))
for (record: ConsumerRecord<String, String> in records) {
  println("Consumer reading record: ${record.value()}")
}
consumer.commitAsync()

Committing can also be done periodically in the background if configured to do so, see the Kafka documentation - enable.auto.commit.

Polling loop

To finish the process of polling records, processing them and committing the updated offsets, you’ll want to group these steps and repeat them as your application requires. For most use cases, repeating these steps in a loop might suffice, which would provide a consumer that infinitely pulls records from a Kafka broker and feeds them to the rest of the application.

This loop resembles what you’ll see in the Consumer’s JavaDocs as well as many examples online.

The most straightforward way to do something infinitely is to use a while(true) loop run on a dedicated thread:

thread {
  while (true) {
    val records: ConsumerRecords<String, String> = consumer.poll(Duration.ofSeconds(5))
      for (record: ConsumerRecord<String, String> in records) {
        println("Consumer reading record: ${record.value()}")
      }
    consumer.commitAsync()
  }
}

I don’t have anything left to say here, so let’s close up this post.

Summary

The overall process of using a Kafka consumer consists of:

Creating a consumer.
Subscribing to some topics.
Polling records from the Kafka broker.
Processing these records with the rest of your application.
Committing the updated offsets back to the Kafka broker.
Repeat, by going back to step 3 (polling records).

I have also put together a nice diagram to showcase this (I think it’s nice anyway!).

Bringing all these steps together gives you the following code:

fun main() {
  val consumer = createConsumer()
  thread { consumeMessages(consumer, topic) }
}

fun createConsumer(): Consumer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  props["group.id"] = "test"
  props["enable.auto.commit"] = "true"
  props["auto.commit.interval.ms"] = "1000"
  props["key.deserializer"] = "org.apache.kafka.common.serialization.StringDeserializer"
  props["value.deserializer"] = "org.apache.kafka.common.serialization.StringDeserializer"
  return KafkaConsumer(props)
}

fun consumeRecords(consumer: Consumer<String, String>, topic: String) {
  consumer.subscribe(listOf(topic))
  while (true) {
    val records: ConsumerRecords<String, String> = consumer.poll(Duration.ofSeconds(5))
      for (record: ConsumerRecord<String, String> in records) {
        println("Consumer reading record: ${record.value()}")
      }
    consumer.commitAsync()
  }
}

We have now looked at how to write the consuming side of an application that leverages Kafka. Combining this along with the information covered in previous posts about topics, partitions and producers, you should now have enough knowledge to write applications that interact with Kafka.

In the following posts, we will cover topics (not a Kafka topic) that expand on what we covered so far to allow you to start writing more effective Kafka applications.

Intro to Kafka - Producers

Sat, 24 Apr 2021 00:00:00 GMT

Producers make up the Kafka client code that sends records to topics. These clients dramatically reduce the complexity of leveraging Kafka as it handles problems like connection pooling and network buffering. You can choose to interact with a small amount of API while still writing highly effective applications.

The diagram below demonstrates how a producer sends a record to a broker, who then records it in one of its topics’ partitions.

From a high level, this is how a producer interacts within a Kafka system. The following sections will explore how to use the producer API and the code required to do so.

The code written in this post is in Kotlin, but similar code should be applicable for any JVM language. Furthermore, clients in other languages will also look virtually the same.

Creating a producer

Create a KafkaProducer to start sending records to Kafka:

fun createProducer(): Producer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  props["acks"] = "all"
  props["retries"] = 0
  props["linger.ms"] = 1
  props["key.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"
  props["value.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"

  return KafkaProducer(props)
}

When creating a KafkaProducer, you must provide its configuration via a Properties object or a Map<String, Object>. KafkaProducer’s constructor also accepts values for the key.serializer and value.serializer properties.

You can find the available configuration options in Kafka’s documentation.

You’re now ready to start producing records, but first, you need to know what a record contains.

Creating records

From the simplest perspective, a record consists of a key, value, and the topic that stores the record.

However, there is much more than just this. In fact, neither the key nor value are actually required (in that they can be null).

This is what a record really looks like:

Ordered in the same order as required by ProducerRecord’s full constructor.

To reiterate, and because not everyone likes diagrams, a record consists of the following properties (along with some explanations):

Topic - The topic to send the record to.
Partition (optional) - The partition to send the record to. When not provided, the partition is calculated by hashing the record’s key and modding it against the number of partitions.
Timestamp (optional) - The timestamp of the record. When not explicitly provided, it will default to System.currentTimeMillis.
Key (optional) - The key of the record.
Value (optional) - The value of the record.
Headers (optional) - Extra metadata to go alongside the record’s value.

Yes, both the key and value can be null. However, when you think about it, it doesn’t seem too crazy. It’s perfectly valid to receive null objects from the rest of your application code and Kafka being ok with that likely saves you a headache further down the line. However, it is probably still worth thinking about null values to be a good little developer.

The code for creating a ProducerRecord is shown below:

val record: ProducerRecord<String, String> = ProducerRecord(
  topic, // Topic
  time.toString(), // Key
  "Record sent at $time" // Value
)

Nothing fancy to show here since I only include the topic, key and value. Several constructors help you create ProducerRecords; however, you might be forced to pass in some nulls to make the records your applications require.

Sending records

After creating a record to send, all that remains is actually sending it. The code required for this is straightforward:

producer.send(record)

send asynchronously sends a record to the Kafka broker, returning a Future that you can use to make the call blocking if you desire (by calling Future.get).

You can also register a callback that executes when the sending of the record is acknowledged:

producer.send(record) { metadata: RecordMetadata, exception: Exception ->
  log.info("Stored in topic: ${metadata.topic()}, partition: ${metadata.partition()} with offset: ${metadata.offset()}")
}

That’s it. The record provides the rest of the information, allowing the process of sending a record to remain simple.

Multiple producers

Kafka supports multiple producers without significantly affecting your applications.

You’ll need to create extra producers using the same process as starting a single producer. You don’t have to do anything special here.

Summary

The overall process for sending a record to a Kafka topic consists of:

Creating a producer.
Creating a record.
Sending the record via the producer.

The diagram below brings together all the pieces discussed in this post:

Bringing the 3 steps together gives you code similar to the following:

// Create a producer
fun createProducer(): Producer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  props["acks"] = "all"
  props["retries"] = 0
  props["linger.ms"] = 1
  props["key.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"
  props["value.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"

  return KafkaProducer(props)
}

fun produceRecords(producer: Producer<String, String>, topic: String) {
  // Execute every 2 seconds
  fixedRateTimer(daemon = true, period = Duration.ofSeconds(2).toMillis()) {
    val time = LocalDateTime.now()
    // Create a record
    val record: ProducerRecord<String, String> = ProducerRecord(
      topic, // Topic
      time.toString(), // Key
      "Record sent at $time" // Value
    )
    println("Producer sending record: $record")
    // Send the record
    producer.send(record)
  }
}

You now know how to formulate the producing side of an application that leverages Kafka. Next up is understanding how to use the created records on the other end of your application. I will cover consumers in my following post.

Intro to Kafka - Topics and partitions

Mon, 12 Apr 2021 00:00:00 GMT

Topics

Kafka, at a high level, stores records in topics. When handing a new record off to Kafka, you must decide the name of the topic you wish to send it to. On the flip side, you must also provide a topic’s name to receive and process records out of Kafka.

Records, events, messages or whatever you want to call them equate to the same thing. Kafka seems to call them records, so that is what I will stick with within this post.

Think of topics as the queue or table that you want your data to be stored in. I quite liked the comparison between topics and database tables made in this talk by Robin Moffatt. For example, you have a topic that contains orders, which is quite reasonable to compare to a database table containing order information. Your applications can then use this topic to submit information about orders as they arrive while downstream services process them.

In Kafka terminology, producers submit new records, while consumers read these same records later (albeit possibly almost instantly). Topics tie producers and consumers together while creating a clear cut boundary between them.

Topics are denoted by name, meaning that as records are produced, they will include the name of the topic they should be stored in, while consumers can subscribe to several topics, again decided by name.

For example, this might look like:

The two diagrams above show what topics are in Kafka, but only from a high level. The following section will take a closer look at what goes on inside a topic.

Partitions

A single topic can consist of multiple internal queues, named partitions, which hold the records published to Kafka.

Each partition within a topic maintains its own order. Leveraging this fact is what allows applications that include Kafka to scale well (if done correctly).

Producers push records to any partition within a topic. When submitting a record, a partition is chosen, determined by its key, or explicitly specifying a partition. Divvying out records in this manner, allows each partition to grow at similar rates.

When consumers subscribe to a topic, they are assigned some partitions that they receive records from. For each partition, a consumer will receive records that maintain their produced order. Thanks to this ordering, you can start using multiple consumers to process events from a single topic to improve performance and availability.

The following diagram below gives a visual representation of how producers and consumers interact with a topic’s partitions.

Further explanations of partitions are complicated without looking more into other concepts such as producers, consumers and consumer groups. I will cover all of these subjects, illuminating how every component of Kafka works together to build scalable, performant and resilient applications.

Kafka producer and consumer written with Kotlin

Sat, 03 Apr 2021 00:00:00 GMT

I wanted to write a short and sweet blog post on writing a Kafka producer and consumer in Kotlin before delving deeper into the topic of Kafka. Yes, that was a “topic” pun… I’ll see myself out.

This post will contain example producer and consumer code written in Kotlin.

Producer

The code below creates a Producer that continually pumps messages to its chosen topic.

fun createProducer(): Producer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  props["acks"] = "all"
  props["retries"] = 0
  props["linger.ms"] = 1
  props["key.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"
  props["value.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"

  return KafkaProducer(props)
}

fun Producer<String, String>.produceMessages(topic: String) {
  fixedRateTimer(daemon = true, period = Duration.ofSeconds(2).toMillis()) {
    val time = LocalDateTime.now()
    val message = ProducerRecord(
      topic, // topic
      time.toString(), // key
      "Message sent at ${LocalDateTime.now()}" // value
    )
    println("Producer sending message: $message")
    this@produceMessages.send(message)
  }
}

This is an example, so the code is not particularly useful.

In this example, a StringSerializer is used for both keys and values, as the producer is pushing messages comprised purely of Strings. At a minimum, you have to include a single serializer, or an error will be coming your way:

Exception in thread "main" org.apache.kafka.common.config.ConfigException: Missing required configuration "key.serializer" which has no default value.
	at org.apache.kafka.common.config.ConfigDef.parseValue(ConfigDef.java:478)
	at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:468)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:108)
	at org.apache.kafka.common.config.AbstractConfig.<init>(AbstractConfig.java:129)
	at org.apache.kafka.clients.producer.ProducerConfig.<init>(ProducerConfig.java:536)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:330)
	at org.apache.kafka.clients.producer.KafkaProducer.<init>(KafkaProducer.java:302)
	at dev.lankydan.kafka.producer.ProducerKt.createProducer(Producer.kt:30)
	at dev.lankydan.kafka.MainKt.main(Main.kt:13)
	at dev.lankydan.kafka.MainKt.main(Main.kt)

Using the wrong serializer will cause the following error:

Exception in thread "Timer-0" org.apache.kafka.common.errors.SerializationException: Can't convert key of class java.lang.String to class org.apache.kafka.common.serialization.UUIDSerializer specified in key.serializer
Caused by: java.lang.ClassCastException: class java.lang.String cannot be cast to class java.util.UUID (java.lang.String and java.util.UUID are in module java.base of loader 'bootstrap')
	at org.apache.kafka.common.serialization.UUIDSerializer.serialize(UUIDSerializer.java:29)
	at org.apache.kafka.common.serialization.Serializer.serialize(Serializer.java:62)
	at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:918)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:886)
	at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:774)
	at dev.lankydan.kafka.producer.ProducerKt$produceMessages$$inlined$fixedRateTimer$1.run(Timer.kt:156)
	at java.base/java.util.TimerThread.mainLoop(Timer.java:556)
	at java.base/java.util.TimerThread.run(Timer.java:506)

You can find the available configuration options in Kafka’s documentation.

After creating the producer, it is ready to start sending events/messages/records, or whatever you wish to call them. You can send a record by building a ProducerRecord and executing Producer.send.

Producer is the interface that KafkaProducer implements, which it also shares with MockProducer.

A ProducerRecord consists of 6 components:

Topic - The topic to send the record to.
Partition - The partition to send the record to. This property is optional. When it is not provided, the partition is calculated by hashing the record’s key and modding it against the number of partitions.
Timestamp - The timestamp of the record. When it is not explicitly provided, it will default to System.currentTimeMillis.
Key - The key of the record.
Value - The value of the record.
Headers - Extra metadata to go alongside the record’s value.

Producer.send completes the process as it hands off the record to the Kafka broker/cluster.

Consumer

The following consumer code compliments the previously shown producer:

fun createConsumer(): Consumer<String, String> {
  val props = Properties()
  props.setProperty("bootstrap.servers", "localhost:9092")
  props.setProperty("group.id", "test")
  props.setProperty("enable.auto.commit", "true")
  props.setProperty("auto.commit.interval.ms", "1000")
  props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
  props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
  return KafkaConsumer(props)
}

fun Consumer<String, String>.consumeMessages(topic: String) {
  subscribe(listOf(topic))
  while (true) {
    val messages: ConsumerRecords<String, String> = poll(Duration.ofMillis(5000))
    if (!messages.isEmpty) {
      for (message: ConsumerRecord<String, String> in messages) {
        println("Consumer reading message: ${message.value()}")
      }
      commitAsync { offsets, exception ->
        for ((partition, metadata) in offsets) {
          println("Committed offset for topic: ${partition.topic()}, partition: ${partition.partition()}, offset: ${metadata.offset()}")
        }
      }
    } else {
      println("No messages to read and poll timeout reached")
    }
  }
}

Similar to initialising a KafkaProducer, you must provide configuration when creating a KafkaConsumer.

You can find the available configuration options in Kafka’s documentation.

The consumer is in charge of requesting records from the Kafka broker. To do this, it subscribes (by calling subscribe) to the topic(s) it wants records from. Calling Consumer.poll requests records from the subscribed topics and blocking the executing thread until the consumer receives some records or reaching the timeout duration. Surpassing the timeout causes the code to move on with an empty collection of records. No exceptions are thrown, which makes sense; the lack of messages isn’t unusual behaviour; it is only an indication that no new records are available for processing.

When messages are successfully received, poll will return ConsumerRecords (a collection of ConsumerRecords). These records consist of the following information (most of them match the data contained in ProducerRecord):

Topic - The topic the record is from.
Partition - The partition the record was placed in.
Timestamp - The timestamp that the record was recorded with.
Key - The key of the record.
Value - The value of the record.
Headers - Extra metadata to stored alongside the record’s value.
Offset - The offset of the record (the position in the partition).

ConsumerRecords implements Iterable so it can be looped over directly, as shown in the example above.

After processing the batch of received records, you should update the offset. Calling commitAsync will set the offset, for each subscribed topic, to the highest offset of the received records. You can call the commitAsync overload below if you want to specify the offsets to update to yourself:

void commitAsync(Map<TopicPartition, OffsetAndMetadata> offsets, OffsetCommitCallback callback);

You can use the OffsetCommitCallback to trigger a callback when committing the offset, which can be beneficial as it is happening asynchronously.

commitSync is the synchronous version of commitAsync which will block the current thread until the consumer propagates the update to the Kafka broker.

How you decide to update your offset is up to you and the behaviour your application requires.

Summary

You have now seen an example of how to write a Kafka producer and consumer in Kotlin. There isn’t anything special about it being written in Kotlin, but now you know what it could like like if you were wondering.

In short, to create a producer, you should instantiate a KafkaProducer, make ProducerRecords and send them to the Kafka broker. To consume these messages, create a KafkaConsumer, poll new records, and remember to call commitAsync/commitSync to update your topics’ offsets.

Running Kafka locally with Docker

Sun, 28 Mar 2021 00:00:00 GMT

There are two popular Docker images for Kafka that I have come across:

Bitmami/kafka (Github)
wurstmeister/kafka (Github)

I chose these instead of via Confluent Platform because they’re more vanilla compared to the components Confluent Platform includes.

You can run both the Bitmami/kafka and wurstmeister/kafka images locally using the docker-compose config below, I’ll duplicate it with the name of each image inserted:

Bitmami/kafka:

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

The Bitnami image is well documented and is where I pulled this nice docker-compose config from.

wurstmeister/kafka:

version: "3"
services:
  zookeeper:
    image: 'wurstmeister/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'wurstmeister/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_LISTENERS=PLAINTEXT://:9092
      - KAFKA_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

It is important to have KAFKA_ADVERTISED_LISTENERS set or you won’t be able to connect to Kafka from an external application. More information on this topic can be found here, which I found extremely useful when I did this incorrectly.

Once you have started the Kafka and Zookeeper containers, you’re good to go. You can start connecting to Kafka either directly or from an application.

Directly, via Kafka’s consumer and producer scripts:

# Create topic
docker exec -it kafka_kafka_1 kafka-topics.sh --create --bootstrap-server kafka:9092 --topic my-topic
# Create events
docker exec -it kafka_kafka_1 kafka-console-producer.sh --bootstrap-server kafka:9092 --topic my-topic
# Read events
docker exec -it kafka_kafka_1 kafka-console-consumer.sh --bootstrap-server kafka:9092 --topic my-topic --from-beginning

From an application (in Kotlin):

fun createProducer(): Producer<String, String> {
  val props = Properties()
  props["bootstrap.servers"] = "localhost:9092"
  props["acks"] = "all"
  props["retries"] = 0
  props["linger.ms"] = 1
  props["key.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"
  props["value.serializer"] = "org.apache.kafka.common.serialization.StringSerializer"
  return KafkaProducer(props)
}

fun Producer<String, String>.produceMessages(topic: String) {
  fixedRateTimer(daemon = true, period = Duration.ofSeconds(2).toMillis()) {
    val time = LocalDateTime.now()
    val message = ProducerRecord(
      topic, // topic
      time.toString(), // key
      "Message sent at ${LocalDateTime.now()}" // value
    )
    send(message)
  }
}

fun createConsumer(): Consumer<String, String> {
  val props = Properties()
  props.setProperty("bootstrap.servers", "localhost:9092")
  props.setProperty("group.id", "test")
  props.setProperty("enable.auto.commit", "true")
  props.setProperty("auto.commit.interval.ms", "1000")
  props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
  props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer")
  return KafkaConsumer(props)
}

fun Consumer<String, String>.consumeMessages(topic: String) {
  subscribe(listOf(topic))
  while (true) {
    val messages: ConsumerRecords<String, String> = poll(Duration.ofMillis(5000))
    for (message in messages) {
      println("Consumer reading message: ${message.value()}")
    }
    commitAsync()
  }
}

Public val with a private backing field

Sun, 14 Mar 2021 00:00:00 GMT

As learnt through many years of writing Java code, we have come to the collective decision to use getters and setters instead of directly accessing a class’ properties. However, this seems to slightly butt heads with Kotlin providing language support for exposing class properties directly through val and var rather than getters and setters.

Ok, yes, in Kotlin, it is a bit more nuanced as it allows you to define a get and set method for vals and vars (where appropriate), although this can become awkward to use in some scenarios.

Take the following code as an example:

class BackingProperty {

  val list: List<String> = mutableListOf()
    // Return a copy to prevent modification outside the class
    get() {
      return ArrayList(field)
    }

  fun add(value: String) {
    // Requires cast to [MutableList] as property is defined as a [List]
    (list as MutableList).add(value)
  }
}

fun main() {
  val obj = BackingProperty()
  println(obj.list)
  obj.add("adding a value")
  println(obj.list)
}

Which prints out:

list = []
list = []

This is not what I wanted to output. Instead, I wanted the add method to add the value to the list property. However, due to list defining a get, it instead returns a copy of the original object and adds it to that, thus rendering the code completely useless since list can never be mutated. Furthermore, list must be cast as the publicly exposed type is unmodifiable.

The issue here is the duality of the property. It behaves as both the public representation and private implementation, going against Java’s best practices. That being said, Java cannot define get and set methods and restrict the modification of a property while leaving it public in the same way Kotlin can.

This flaw is the cost Kotlin pays to support property accessors. By publicly exposing a property, a class can only access its own properties in the same way that anything outside does. Altering a class’ get, like in the example above, compounds this issue further as it can prevent the class from behaving correctly.

C# also does this with their backing properties.

So what can you get around this you say. There are two options, as far as I know:

Write getters and setters in the same way as Java:

class BackingProperty {

  private val list = mutableListOf<String>()

  // Return a copy to prevent modification outside the class
  fun getList(): List<String> = ArrayList(list)

  fun add(value: String) {
    list.add(value)
  }
}

fun main() {
  val obj = BackingProperty()
  println(obj.getList())
  obj.add("adding a value")
  println(obj.getList())
}

The downside here is that you lose some of the “flavour” of Kotlin as you are writing code similar to Java.

Write a private “backing” field (another version of the property with a different name):

class BackingProperty {

  private val _list = mutableListOf<String>()
  val list: List<String>
  // Return a copy to prevent modification outside the class
  get() {
    return ArrayList(_list)
  }

  fun add(value: String) {
    _list.add(value)
  }
}

fun main() {
  val obj = BackingProperty()
  println(obj.list)
  obj.add("adding a value")
  println(obj.list)
}

Adding _list allows you to get around the fact that list is public. Note, that the naming doesn’t matter, just that you need to give it another name. Prepending an underscore works as it looks virtually the same as the public version.

Annoyingly neither solution sits well with me. The Java/getter and setter version makes the call-sites worse, and the backing property option slightly confuses the class’ code itself. Language support needs to be added to find a solution that makes it pleasant to write code both inside and outside the class.

I also could not find any official recommendation on what to do in this situation. I would be more willing to accept the backing field option if it was an agreed-upon solution. For now, I’ll have to assume this is the correct choice as it seems more Kotlin-ish than adding getters and setters. That being said, I think this could lead to bugs since you could now access the wrong version of the property and incorrectly modify the code’s behaviour.

All experience matters

Fri, 12 Mar 2021 00:00:00 GMT

I wanted to write down my thoughts about “experience” as it is something that I have been thinking about over the last couple of months.

Now that I’m working more closely with colleagues of varying degrees of seniority, I have noticed how the team’s dynamics fluctuate back and forth, with each person bringing something different to the table. For example, I have been working at our company for longer and have a better understanding of our software, allowing me to provide insight on why things are the way they are, bugs we’ve fixed and features we’ve added. On the other hand, my colleagues who have been Software Engineers for longer than I have likely faced and solved more problems than I have, providing them with a broader understanding over time.

We all have things to offer. I have my experience. They have their experience. Together as a team, we have our experience. Only by sharing between us can we move from mine and theirs to ours. I seek out my colleagues to learn from lessons they’ve learnt, weird things they’ve seen and anything in between, and they come to me for the same reasons. The only difference is that we have different experiences to share between us.

If we all had the same experiences, then the benefit of working in teams would be severely harmed. Teams work well when they are diverse in background and expertise. Whether this is software related or not, it is crucial.

Please go out and spread your experience to your team and let them do the same for you.

Now, I think some readers might think, “why should I listen to someone with less experience than me” which isn’t necessarily a completely wrong point of view. Information given to you could indeed be incorrect, but that’s a problem you have wherever you go. Reading from StackOverflow or even from documentation directly can all have mistakes in them. You have to trust something. Therefore you should trust that your teammates are probably correct if they have the confidence to share their knowledge with you. You can always go back and corroborate what they shared later, which is perhaps a good idea in general when hearing something for the first time.

The main point is not to discount what others have to say before they’ve even said anything.

You are also more likely to believe someone the more that you trust them. I think that goes without saying. The more you work alongside your teammates, the amount of trust between you will increase as assumed knowledge can become proven. A lack of opportunities to demonstrate expertise has possibly lead to people assuming that more senior workers are more skilled by correlating time with experience (they are related, but time doesn’t directly lead to understanding). It seems safer to assume that people who have been doing something for longer know more than others. However, experience can be gained at different rates and is altered by the complexity and demands of problems faced throughout one’s career.

I personally take the point of view that I trust what others say unless my own experiences contradict theirs. When we have a different perspective on how something works, we now have a unique opportunity to discuss it. One, both or neither of us might be wrong or right. We now have the chance to figure out which category we fit in, hopefully coming out with better collective knowledge than we had going in.

I’m not really sure where I’m trying to go with this, but hopefully, I’ve said something that resonates with you or will promote change in your behaviour. By listening and trusting our colleagues, we can build a more inclusive workplace where everyone feels comfortable sharing their experience and helping others. I’m not trying to preach to you, but this does seem to be an area where we are failing, and by raising awareness of the current shortcomings, we open up the opportunity to address them.

Backwards compatibility of Kotlin's default arguments

Sun, 07 Mar 2021 00:00:00 GMT

Kotlin’s default arguments can make APIs easier to understand as a user and more effortless to write as a developer. No more manually writing out function overloads with every possible useful combination of arguments. Furthermore, when reading these functions, it’s easy to see what arguments can be left out and what values they use if you do.

That all sounds great, right? Most uses of Kotlin’s default arguments will work perfectly, bar any lousy code you write, but that isn’t really Kotlin’s fault. However, in some use-cases, there is a flaw with how default arguments work. More precisely, evolving the API of a platform without requiring existing code to recompile.

This post will focus on how Kotlin’s default arguments work under the hood, why that affects backwards compatibility, and what you can do about it.

How default arguments work

Kotlin’s default arguments allow you to write a function like:

interface MyInterface {

  fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D(), e: E = E())
}

Where the caller of the function decides which arguments it provides values for and relies on defaults for the rest. You can find a little bit more information on this in the Kotlin docs.

That’s what happens at a high level. For most people writing and using Kotlin code, that is all you’ll need. However, to understand how this can affect backwards compatibility, we’ll need to take a closer look. To do this, we’ll take a look at the generated bytecode.

Below is the Kotlin bytecode generated from the function shown before (large code snippet incoming!):

// ================dev/lankydan/api/MyInterface$DefaultImpls.class =================
// class version 50.0 (50)
// access flags 0x31
public final class dev/lankydan/api/MyInterface$DefaultImpls {


  // access flags 0x1009
  public static synthetic function$default(Ldev/lankydan/api/MyInterface;Ldev/lankydan/api/MyInterface$A;Ldev/lankydan/api/MyInterface$B;
  Ldev/lankydan/api/MyInterface$C;Ldev/lankydan/api/MyInterface$D;Ldev/lankydan/api/MyInterface$E;ILjava/lang/Object;)V
    ALOAD 7
    IFNULL L0
    NEW java/lang/UnsupportedOperationException
    DUP
    LDC "Super calls with default arguments not supported in this target, function: function"
    INVOKESPECIAL java/lang/UnsupportedOperationException.<init> (Ljava/lang/String;)V
    ATHROW
   L0
    ILOAD 6
    ICONST_1
    IAND
    IFEQ L1
   L2
    LINENUMBER 5 L2
    NEW dev/lankydan/api/MyInterface$A
    DUP
    INVOKESPECIAL dev/lankydan/api/MyInterface$A.<init> ()V
    ASTORE 1
   L1
    ILOAD 6
    ICONST_2
    IAND
    IFEQ L3
    NEW dev/lankydan/api/MyInterface$B
    DUP
    INVOKESPECIAL dev/lankydan/api/MyInterface$B.<init> ()V
    ASTORE 2
   L3
    ILOAD 6
    ICONST_4
    IAND
    IFEQ L4
    NEW dev/lankydan/api/MyInterface$C
    DUP
    INVOKESPECIAL dev/lankydan/api/MyInterface$C.<init> ()V
    ASTORE 3
   L4
    ILOAD 6
    BIPUSH 8
    IAND
    IFEQ L5
    NEW dev/lankydan/api/MyInterface$D
    DUP
    INVOKESPECIAL dev/lankydan/api/MyInterface$D.<init> ()V
    ASTORE 4
   L5
    ILOAD 6
    BIPUSH 16
    IAND
    IFEQ L6
    NEW dev/lankydan/api/MyInterface$E
    DUP
    INVOKESPECIAL dev/lankydan/api/MyInterface$E.<init> ()V
    ASTORE 5
   L6
    ALOAD 0
    ALOAD 1
    ALOAD 2
    ALOAD 3
    ALOAD 4
    ALOAD 5
    INVOKEINTERFACE dev/lankydan/api/MyInterface.function (Ldev/lankydan/api/MyInterface$A;Ldev/lankydan/api/MyInterface$B;Ldev/lankydan/api/MyInterface$C;Ldev/lankydan/api/MyInterface$D;Ldev/lankydan/api/MyInterface$E;)V (itf)
    RETURN
    MAXSTACK = 6
    MAXLOCALS = 8

  @Lkotlin/Metadata;(mv={1, 4, 1}, bv={1, 0, 3}, k=3)
  // access flags 0x19
  public final static INNERCLASS dev/lankydan/api/MyInterface$DefaultImpls dev/lankydan/api/MyInterface DefaultImpls
  // compiled from: MyInterface.kt
}

It’s quite likely that most of this doesn’t make sense. I’m not trying to be patronising, as I also struggle to read Kotlin/Java/JVM bytecode, although we can still extract some information from it.

Firstly, an alternative version of MyInterface was generated, dev/lankydan/api/MyInterface$DefaultImpls. I haven’t shown all the bytecode, but if I did, you would see dev/lankydan/api/MyInterface exists and is separate from the default version.

Secondly, a synthetic function was placed inside the previously mentioned MyInterface$DefaultImpls with arguments that don’t precisely match those of the original version in MyInterface. If you look more carefully, you’ll notice that the first argument is an instance of MyInterface and is followed by the same arguments defined in the Kotlin function. Although that is not entirely true, there are two extra arguments, an int and an Object (you can see them in ILjava/lang/Object; where the I represents the int and Ljava/lang/Object speaks for itself).

The next question is, why are these last two arguments there?

If you can understand the bytecode above, then maybe you’ve been able to figure out why they’re there. For the rest of us mortal beings though, I have added the decompiled version that shows it as Java code below:

public static final class DefaultImpls {
  // $FF: synthetic method
  public static void function$default(MyInterface var0, MyInterface.A var1, MyInterface.B var2, 
  MyInterface.C var3, MyInterface.D var4, MyInterface.E var5, int var6, Object var7) {
    if (var7 != null) {
      throw new UnsupportedOperationException("Super calls with default arguments not supported in this target, function: function");
    } else {
      if ((var6 & 1) != 0) {
        var1 = new MyInterface.A();
      }

      if ((var6 & 2) != 0) {
        var2 = new MyInterface.B();
      }

      if ((var6 & 4) != 0) {
        var3 = new MyInterface.C();
      }

      if ((var6 & 8) != 0) {
        var4 = new MyInterface.D();
      }

      if ((var6 & 16) != 0) {
        var5 = new MyInterface.E();
      }

      var0.function(var1, var2, var3, var4, var5);
    }
  }
}

From this version, we can now see how the int and Object are being used.

The Object - From what I could find out, the Object is not actually used yet. Supposedly, this Object will be used for adding super calls with default values some time in the future. Here are a few links that indicate this - StackOverflow, StackOverflow and Jetbrains’ YouTrack (currently 4 years old).

The int - Determines which arguments use their default values. This is done by executing a bitwise AND (&) against the int and a number that is increasing in powers of 2 (to represent binary numbers with a leading 1, e.g. 2 = 10 and 16 = 10000). The int itself is determined by the call-site depending on which arguments are passed into the function. For example:

// Kotlin call-site
myInterface.function(b = MyInterface.B(), c = MyInterface.C())

// Decompiled code of the call-site
DefaultImpls.function$default(myInterface, (A)null, new B(), new C(), (D)null, (E)null, 25, (Object)null);

In this example, the compiler decides to pass in 25 (11001). Let’s quickly do the bitwise operations to see which default values are used. I’ve also put an example below and a link to Wikipedia in case you’re interested in how the bitwise AND works (I haven’t linked to Wikipedia for a while!):

11001 AND
00001
-----
00001 = 1

Below are the operations matching the decompiled code:

Operation	Binary operation	Output	Default value used
`25 & 1`	`11001 & 00001`	1	✅
`25 & 2`	`11001 & 00010`	0	❌
`25 & 4`	`11001 & 00100`	0	❌
`25 & 8`	`11001 & 01000`	1	✅
`25 & 16`	`11001 & 10000`	1	✅

These results match up with what we expect. The decompiled code shows that if the bitwise AND returns anything other than 0, then a default value will be supplied. For the call-site above, this means that A, D and E all receive default values, which correctly matches the Kotlin code.

Finally, as seen in the decompiled code, now that all variables are correctly set (var1 to var5), they are passed into the real function, reaching the end of the process.

Hopefully, that all made sense. If not, give it another read. If it still doesn’t, then you can blame me for failing to explain it well enough. You will need this knowledge in the next section to understand how Kotlin’s default arguments affect backwards compatibility.

Why this affects backwards compatibility

First, I’ll define what I mean as “backwards compatibility”. I quite liked how Wikipedia stated it, so I’ll put that below:

Backward compatibility (also known as backwards compatibility) is a property of a system, product, or technology that allows for interoperability with an older legacy system, or with input designed for such a system, especially in telecommunications and computing.

In terms of what I am writing about, it means that a library, framework or platform should evolve and release new versions without breaking existing code.

More precisely, for this post, a function with default arguments can evolve without breaking existing function usages.

We will use the code from before and explore what happens as we evolve the interface’s function declaration. I’ve added it again since it’s so far away now:

interface MyInterface {

  fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D(), e: E = E())
}

While keeping the call-site the same as it represents “existing code”:

myInterface.function(b = MyInterface.B(), c = MyInterface.C())

Before I continue, I want to point out that removing an argument or adding a new one that does not have a default value from an existing function will never be backwards compatible.

// Removing an argument
❌ fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D())
// Adding an argument without a default value
❌ fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D(), e: E = E(), f: F)

You are guaranteed to break any existing code if you do that. The only way to do this without breaking anything is to add an overload with the change in arguments. As this post is focusing on default arguments, I won’t explain these two scenarios any further.

By excluding basic removing and adding, we are left with adding a new argument with a default value. This scenario is different from adding a new one without a default value because existing callers of the function should work. These callers will still be using the previous version of the function, and that’s ok due to their reliance on default values for missing arguments. Therefore the function’s new argument is not enough to break these applications.

// Adding an argument with a default value
✅ fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D(), e: E = E(), f: F = F())

But, and a big but 😏, the existing code will only continue to work if it is recompiled.

Now, this is ok in a lot of situations. Suppose you are using a library or framework and you update them. In that case, you will most likely recompile your project as your application is what starts everything and therefore cannot benefit from the updates without a rebuild.

However, if you are building something that resembles a platform that runs application code written by other developers, then these applications should continue to work without recompilation. A platform should evolve without breaking everything using it, assuming it’s not a major version update. This is where the inner workings of Kotlin’s default arguments unravel.

An analogy for this is, when your operating system updates, you don’t want to also update every application on your machine. Instead, you want them to continue working as they were with the possibility of updating to benefit from any specific improvements.

To understand why it breaks without recompilation, we need to look at the decompiled bytecode of the function and call-site again:

// Decompiled code of the original function with default arguments
public static void function$default(MyInterface var0, MyInterface.A var1, MyInterface.B var2, 
  MyInterface.C var3, MyInterface.D var4, MyInterface.E var5, int var6, Object var7)

// Decompiled code of the call-site
DefaultImpls.function$default(myInterface, (A)null, new B(), new C(), (D)null, (E)null, 25, (Object)null);

Now we’ll introduce the updated version with a new argument with a default value and take a look at the decompiled code:

// Decompiled code of the updated function with default arguments
public static void function$default(MyInterface var0, MyInterface.A var1, MyInterface.B var2,
  MyInterface.C var3, MyInterface.D var4, MyInterface.E var5, MyInterface.F var6, int var7, Object var8)

Note, that this is now the only version of the function in the bytecode.

Let’s match up the old call-site with the function’s new version:

Call-site	Extra argument	Matches
`MyInterface`	`MyInterface`	✅
`MyInterface.A`	`MyInterface.A`	✅
`MyInterface.B`	`MyInterface.B`	✅
`MyInterface.C`	`MyInterface.C`	✅
`MyInterface.D`	`MyInterface.D`	✅
`MyInterface.E`	`MyInterface.E`	✅
`int`	`MyInterface.F`	❌
`Object`	`int`	❌
-	`Object`	❌

The call-site no longer passes in the correct inputs for the function. Trying to run the existing code against the updated function will cause the following error (nicely formatted to make it easier to read):

Caused by: java.lang.NoSuchMethodError: 
'void dev.lankydan.api.MyInterface$DefaultImpls.function$default(
  dev.lankydan.api.MyInterface, 
  dev.lankydan.api.MyInterface$A,
  dev.lankydan.api.MyInterface$B,
  dev.lankydan.api.MyInterface$C,
  dev.lankydan.api.MyInterface$D, 
  dev.lankydan.api.MyInterface$E,
  int,
  java.lang.Object
)'

Ok, so we’ve seen what goes wrong when a new argument with a default value is added to a function without recompiling the code using it. Does this mean there is no hope? Stop worrying, and read the next section!

What you can do about it

The previous section showed why platform writers need to be careful when adding new arguments with default values to existing functions. However, I didn’t detail any solutions which I’ll now rectify.

As mentioned previously, depending on the situation, recompiling the depending code might be the simplest solution. If that is a valid choice for you, then I would go with that. From now on, we’ll assume that it’s not a good idea and consider how to keep everything working without recompilation.

Evolving the function without breaking anything will require a new overload. You should treat it the same as removing or adding a non-default argument as it’s the only option you really have (I’ll touch on a completely different choice in a second).

// Original version
✅ fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D(), e: E = E())
// New overload with the extra argument
✅ fun function(a: A = A(), b: B = B(), c: C = C(), d: D = D(), e: E = E(), f: F = F())

As shown earlier, you unfortunately cannot rely on the code to work without handling it yourself.

From a bytecode perspective, this works because there will be a synthetic function for each overload, keeping the original version around so existing applications can use it.

A different choice to circumvent this problem is not to use default arguments at all. I know that sounds odd, but one way to not fall into this trap is to stay on the well-trodden path. In Java, to evolve an API without breaking anything, you would always add a new overload. It worked for Java, so why wouldn’t it work for Kotlin.

The obvious downside is that you lose the benefit of default values altogether, but you’ll have to decide what works best for you and stick to it. Maybe a rule of “no default arguments” is more straightforward to follow than “always add an overload” even though the outcome is ultimately the same (for that singular change).

You might have to include a lot of overloads to keep the same functionality of default arguments.

I’m currently sitting on the side of adding a new overload while keeping default arguments around, but you should choose whatever makes sense for your situation.

Conclusion

Kotlin’s default arguments are beneficial for writing accessible code without explicitly overload a function to express every possible useful combination of arguments. However, they are not perfect. They are not backwards compatible unless the code using an evolved function is also recompiled. For most scenarios, that is perfectly fine, and I do not want to advocate that you remove all reliance on default arguments. Instead, I want to make sure that you have all the information you need to make the correct decision for your application, library, framework or platform.

After looking into this feature, my advice, is to keep freely using default arguments for your application, library or framework code, but if you are writing a platform that runs other developers’ applications, then you must carefully consider your use of this feature.

Sending a HTTP request in Apache Flink

Sun, 07 Feb 2021 00:00:00 GMT

You can send a HTTP request in an Apache Flink application using code similar to the following:

public class HttpRequestFunction extends RichAsyncFunction<String, String> {

  private transient OkHttpClient client;

  @Override
  public void open(Configuration parameters) {
    client = new OkHttpClient();
  }

  @Override
  public void asyncInvoke(String input, ResultFuture<String> resultFuture) {
    Request request = new Request.Builder()
      .get()
      .url("https://some-random-url.com")
      .build();
    Call call = client.newCall(request);
    call.enqueue(new Callback() {
      @Override
      public void onFailure(@NotNull Call call, @NotNull IOException e) {
        resultFuture.complete(Collections.emptyList());
      }

      @Override
      public void onResponse(@NotNull Call call, @NotNull Response response) throws IOException {
        resultFuture.complete(Collections.singleton(response.body().string()));
      }
    });
  }
}

The HTTP client you use doesn’t have to be an OkHttpClient, you can use whatever client you want, but ideally one that can send asynchronous requests. By doing so, you can mix it with Flink’s ability to execute asynchronous functions; otherwise you’re application will slow down as it waits for each request to come back.

You can write an asynchronous function by extending RichAsyncFunction and returning a future completed by the async code. In the code above, the HTTP request’s callback completes the future. I chose to complete the resultFuture with an emptyList when an error occurred to exclude it from further processing while allowing the stream to continue. I could have used completeExceptionally instead but would have lead to the termination of the stream.

resultFuture.complete expects a List<T> even though ResultFuture’s generic type is only T.

Once you have the function written, you’ll need to call it in a different way to other map or filter functions:

AsyncDataStream.unorderedWait(
  // Original stream
  stream,
  // The function
  new HttpRequestFunction(),
  // Tiemout length
  5,
  // Timeout unit
  TimeUnit.SECONDS
)

I’m not sure why you have to call it in a completely different way 🤷.

This code takes in a stream and returns a mutated stream that the HttpRequestFunction has altered. It also requires you to specify a timeout using the last two arguments.

You can also use AsyncDataStream.orderedWait if you want the output stream’s events to remain the same as the input. That being said, it might introduce an overhead to the processing of events, you can find more information in Flink’s documentation.

That should be enough to get you going! Happy coding!

Processing tweets with Apache Flink and the Twitter API

Sun, 31 Jan 2021 00:00:00 GMT

Recently I have been playing around with Apache Flink and decided to use the Twitter API as a datasource for my endeavor.

This post is going to be an extremely short summary of what I did and leave you to look at the code if you’re interested.

The code mentioned in this post is linked above but I’ll put it here as well - apache-flink-twitter-streaming.

What I did

I wrote a Flink streaming application that does the following:

Takes new tweets from Twitter.
Gets recent tweets from the streamed tweet’s author.
Keep tweets that contain similar mentions to the streamed tweet.
Prints out the remaining tweets, including their metrics, such as the retweet count.

How I did it

I used the flink-twitter-connector to pull tweets from Twitter. This connector, at the time of writing, uses the v1 Twitter API rather than v2. Originally I was going to use Kafka Connect Twitter but didn’t want to set up another service when I was primarily interested in Flink itself.

The Twitter connector ingested tweets into my application for processing, where I then requested further information from Twitters v2 API such as, recent tweets, author/user information and tweet metrics.

To be honest, since I was focused on learning Flink, the functionality of my code is not particularly interesting but it provided a decent learning experience. Although, a significant portion of my time was spent trying to understand the Twitter APIs…

If you’re still interested by this point

Since you’ve managed to read this far you might as well look at the code, otherwise you’re not going to take away anything useful from this post.

Here’s the link to the repository again.

Jackson - Deserialize subtypes without annotations

Sat, 30 Jan 2021 00:00:00 GMT

Sometimes you can’t or don’t want to use Jackson annotations to control your JSON serialization and deserialization. Unfortunately, Jackson does not behave very nicely when you aren’t using annotations everywhere, probably causing you to write code that doesn’t work. Which will then lead you to googling solutions, but then you’ll find the only answers ever given… Are to use more annotations…

Fine, I get it, that is how Jackson is typically used, and it’s also how I also use it most of the time. But it seems that if you are trying to solve a problem where annotations are out of bounds, then you are really screwed.

Anyway, I got a bit screwed… So I have dedicated this post to show you how I unscrewed myself from this situation.

Jackson 2.11.1 was used for the content of this post.

I talk about deserializers in this post since they’re related to the problem I was trying to solve, but everything here should hold for serializers as well.

What I needed to do

I needed to deserialize a class that couldn’t have annotations added directly to it.
The deserializer needed to be constructed manually as it delegated some of its functionality to another class.
Subtypes of the original class should be deserialized using the same deserializer.
Allow subtypes to use a separate deserializer if desired.

Potential solutions

A Mixin.
Add a deserializer manually.

Why a Mixin didn’t work

Mixins allow you to specify serializers, deserializers and other Jackson annotations without annotating the original class. You can find further information in the Jackson docs.

A further benefit that comes from using Mixins is the fact that they support inheritance. From the documentation:

Mix-ins work as expected within inheritance hierarchy: it is feasible (and useful) to attach mix-in annotations to super-classes — if so, mix-in annotations can further be overridden by annotations sub-classes (of target) provide.

This all sounds good, but it only meets 3 out of the 4 goals.

✅ I needed to deserialize a class that couldn't have annotations added directly to it.
❌ The deserializer needed to be constructed manually as it delegated some of its functionality to another class.
✅ Subtypes of the original class should be deserialized using the same deserializer.
✅ Allow subtypes to use a separate deserializer if desired.

Since Mixins still require annotations, it is impossible to instantiate a deserializer that takes in any dependencies as the deserializer is referenced by its class name, therefore requiring the object to have a default constructor.

As shown by the code below:

@JsonDeserialize(using = JacksonSupport.OpaqueBytesDeserializer::class)
private interface ByteSequenceMixin

The OpaqueBytesDeserializer has no way to inject dependencies into it, if it requires any, without setting some global properties that it can access.

Unfortunately, it didn’t meet all my requirements and was thrown out.

Why adding a deserializer manually didn’t work

You can add a deserializer manually by using SimpleModule.addDeserializer, for example:

class CordaModule : SimpleModule("corda-core") {
    init {
        addDeserializer(ByteSequence::class.java, JacksonSupport.OpaqueBytesDeserializer())
    }
}

class OpaqueBytesDeserializer : JsonDeserializer<OpaqueBytes>() {
    override fun deserialize(parser: JsonParser, ctxt: DeserializationContext): OpaqueBytes {
        return OpaqueBytes(parser.text?.toByteArray(UTF_8) ?: parser.binaryValue)
    }
}

The deserializer above is created manually and can therefore have any dependencies passed to it.

So how does this stack up against the requirements? Again 3 out of 4.

✅ I needed to deserialize a class that couldn't have annotations added directly to it.
✅ The deserializer needed to be constructed manually as it delegated some of its functionality to another class.
❌ Subtypes of the original class should be deserialized using the same deserializer.
✅ Allow subtypes to use a separate deserializer if desired.

After a lot of head-scratching and a bit of debugging, it seems that registering a deserializer in this way does not allow subtypes to be processed by the same deserializer. Instead, another deserializer must be created and registered for each subtype which is suboptimal and a significant downgrade from what Mixins can do in this regards.

I am unsure why it has this drawback when Mixins can manage inheritance, and honestly, I don’t have the effort to figure out why. There’s also the possibility I’m wrong or that this drawback has been rectified in later versions of Jackson.

For this reason, manually adding a deserializer did not work for me.

What I got working

I only listed out 2 potential solutions previously, because they were the only 2 that I thought I would need. Instead, I had to do some playing around and came across a different method of handling deserializers.

addBeanDeserializerModifier allows a BeanDeserializerModifier to be registered that participates in the constructing of deserializers. Yes, that does sound confusing… I personally find this one hard to understand so I’ll show you some code and go through that instead:

class CordaModule : SimpleModule("corda-core") {
    override fun setupModule(context: SetupContext) {
        super.setupModule(context)
        context.addBeanDeserializerModifier(ByteSequenceBeanDeserializerModifier(serializationService))
    }
}

private class ByteSequenceBeanDeserializerModifier(
  serializationService: P2pSerializationService
) : BeanDeserializerModifier() {

    private val serializedBytesDeserializer = SerializedBytesDeserializer(serializationService)
    private val opaqueBytesDeserializer = OpaqueBytesDeserializer()

    override fun modifyDeserializer(
        config: DeserializationConfig,
        description: BeanDescription,
        deserializer: JsonDeserializer<*>
    ): JsonDeserializer<*> {
        val original = super.modifyDeserializer(config, description, deserializer)
        return when {
            SerializedBytes::class.java.isAssignableFrom(description.beanClass) -> serializedBytesDeserializer
            ByteSequence::class.java.isAssignableFrom(description.beanClass) -> {
                return when {
                    SignedTransaction::class.java.isAssignableFrom(description.beanClass) -> original
                    // other branches returning [original]
                    else -> opaqueBytesDeserializer
                }
            }
            else -> original
        }
    }
}

A BeanDeserializerModifier does not require a generic type allowing it to intercept the construction of deserializers, regardless of the types they handle. The modification of deserializers occurs in modifyDeserializer (I mean the name does suggest that), which is overloaded to provide my implementation. super.modifyDeserializer is also called to retrieve the original deserializer before making any alterations.

Finally, for my implementation, I check the type that the deserializer is to be registered for and either return a different deserializer or the original. You could opt to modify the original rather than returning a new one depending on your use-case.

There are also different versions of modifyDeserializer, such as modifyEnumDeserializer and modifyReferenceDeserializer.

To wrap up this section, I’ll have to add the checklist again.

✅ I needed to deserialize a class that couldn't have annotations added directly to it.
✅ The deserializer needed to be constructed manually as it delegated some of its functionality to another class.
✅ Subtypes of the original class should be deserialized using the same deserializer.
✅ Allow subtypes to use a separate deserializer if desired.

Perfect!

If you think you know better

If you’re a Jackson savant and know a better solution, then I’m all ears. You can message me on Twitter (@LankyDanDev), and I’ll be happy to update my post.

Providing your own Jackson ObjectMapper in Quarkus

Mon, 07 Dec 2020 00:00:00 GMT

I am writing this short post after me, and a colleague had a painful experience trying to get Jackson working in a Quarkus application. The Quarkus documentation does actually tell you how to do so, but both my colleague and I found that it was rather blasé with its explanation.

Why did we feel this way?

Unfortunately, we were both inexperienced (noobs?) with CDI, which Quarkus uses for dependency injection. Furthermore, we also had to do something that wasn’t directly explained in the documentation. Making everything that should have been easy, unclear, and required a whole lot of googling to get right.

For anyone else that ran into the same walls we did, this post is for you.

What were we trying to do that made us lose our way?

We were trying to use a Jackson ObjectMapper provided to us by another library. This instantly took us off the path in the Quarkus documentation, and although it does briefly mention what we should do, we didn’t actually know how to do it…

If you don’t need to stray from the path included in the documentation, then what it says is short, sweet and most importantly works. The code below is from their documentation:

@Singleton
public class RegisterCustomModuleCustomizer implements ObjectMapperCustomizer {

    public void customize(ObjectMapper mapper) {
        mapper.registerModule(new CustomModule());
    }
}

This takes the ObjectMapper that Quarkus creates and applies changes that your application defines.

If instead, you want to provide your own ObjectMapper or leverage one from an external library, then everything below will aid you.

As the code you need to achieve this is short, I’ll just cut to the chase and show you:

public class ObjectMapperConfiguration {

  @Singleton
  ObjectMapper objectMapper(Instance<ObjectMapperCustomizer> customizers) {
    // Your own `ObjectMapper` or one provided by another library
    ObjectMapper mapper = CustomObjectMapper.get();
    // Apply customizations (includes customizations from Quarkus)
    for (ObjectMapperCustomizer customizer : customizers) {
      customizer.customize(mapper);
    }
    return mapper;
  }
}

I’ll keep the explanation short and restrict it to bullet points:

@Singleton defines an ObjectMapper instance that can be used by the rest of the application. You want to use @Singleton here over something like @ApplicationScoped, or you will run into client proxy issues.
Instance<ObjectMapperCustomizer> allows all instances of ObjectMapperCustomizer to be injected into this function. This is similar to writing List<ObjectMapperCustomizer> if you come from the Spring world.
All the instances of ObjectMapperCustomizer are then looped over to apply their customizations. This is included in the Jackson Quarkus documentation:

Users can even provide their own ObjectMapper bean if they so choose. If this is done, it is very important to manually inject and apply all io.quarkus.jackson.ObjectMapperCustomizer beans in the CDI producer that produces ObjectMapper. Failure to do so will prevent Jackson specific customizations provided by various extensions from being applied.
Finally return the ObjectMapper which is ready and raring to go.

You should now know how to customize an ObjectMapper, whether you stick to applying customizations to Quarkus’ ObjectMapper, defining your own or leveraging one provided by another library. All the code should be in this post.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Missing DataSource in Quarkus

Thu, 15 Oct 2020 00:00:00 GMT

If you see an error like this while developing a Quarkus application that uses Agroal as its DataSource:

ERROR [io.qua.dep.dev.IsolatedDevModeMain] (main) Failed to start quarkus: java.lang.RuntimeException: io.quarkus.builder.BuildException: Build failure: Build failed due to errors
    [error]: Build step io.quarkus.arc.deployment.ArcProcessor#validate threw an exception: javax.enterprise.inject.spi.DeploymentException: javax.enterprise.inject.UnsatisfiedResolutionException: Unsatisfied dependency for type javax.sql.DataSource and qualifiers [@Default]
    - java member: dev.lankydan.people.db.PersonRepository#<init>()
    - declared on CLASS bean [types=[dev.lankydan.people.db.PersonRepository, java.lang.Object], qualifiers=[@Default, @Any], target=dev.lankydan.people.db.PersonRepository]
    at io.quarkus.arc.processor.BeanDeployment.processErrors(BeanDeployment.java:990)
    at io.quarkus.arc.processor.BeanDeployment.init(BeanDeployment.java:234)
    at io.quarkus.arc.processor.BeanProcessor.initialize(BeanProcessor.java:122)
    at io.quarkus.arc.deployment.ArcProcessor.validate(ArcProcessor.java:391)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at io.quarkus.deployment.ExtensionLoader$2.execute(ExtensionLoader.java:932)
    at io.quarkus.builder.BuildContext.run(BuildContext.java:277)
    at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
    at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2046)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1578)
    at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1452)
    at java.base/java.lang.Thread.run(Thread.java:834)
    at org.jboss.threads.JBossThread.run(JBossThread.java:479)

Please scroll up 1 line in your logs (or somewhere near this amount), and you’ll see:

WARN  [io.qua.agr.dep.AgroalProcessor] (build-20) The Agroal dependency is present but no JDBC datasources have been defined.

As the error suggests, no DataSource has been defined. In my case, I needed to add the following to my application.properties:

quarkus.datasource.db-kind=postgresql

You’ll need to add this even if you have the url specified. All together you will want your properties to look something like:

quarkus.datasource.db-kind=postgresql
# Whatever the url to your database is
quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/my_test_db
# Whatever your permissions are
quarkus.datasource.username=admin
quarkus.datasource.password=admin

The error I showed is not related to Postgres, since the code never gets that far. So it could still happen with other databases.

Hopefully, this will save you some time, as I personally failed to read that log line and spent way too long trying to resolve this error 🤦‍♂️…

Building a REST API with Quarkus

Mon, 21 Sep 2020 00:00:00 GMT

Quarkus is a Java framework designed to run within containers. Focusing on fast start-up times and low memory usage making it more suitable to run within container orchestration platforms like Kubernetes.

I wanted to write about Quarkus for a while now, and have finally found the time to do so.

The content of this post will focus on writing a simple REST API using Quarkus. I will not explain how Quarkus works, because, well, I haven’t put the time in yet to do so correctly. Hopefully, I can rectify this in future posts. Therefore, if you want more information about Quarkus, I suggest you browse their docs. So far, I have found them detailed, well written, and I will personally continue using them while I explore Quarkus further.

Creating a project

Quarkus provides a maven command to bootstrap your projects, for example (the below works for Linux and MacOS, see the Quarkus - Getting started guide for more information):

mvn io.quarkus:quarkus-maven-plugin:1.7.2.Final:create \
    -DprojectGroupId=dev.lankydan \
    -DprojectArtifactId=quarkus \
    -DclassName="dev.lankydan.web.people.PersonResource" \
    -Dpath="/hello"

This generates a basic project with a HTTP endpoint in PersonResource with the path hello, along with several resources to build and run your application. In fact, you should have enough at this point to run the generated code, both locally and inside a container.

A pom.xml is included that contains the required dependencies. You will need to add to this if you want to write anything past a hello world example.

By running (from inside your project):

./mvnw quarkus:list-extensions

You can see the Quarkus dependencies/extensions available to your project.

Then you can add an extension using a command like the one below:

./mvnw quarkus:add-extension -Dextensions="hibernate-validator"

The other option is to add it manually as I would typically do when sorting out my dependencies. Just thought I’d mention the add-extension command as I hadn’t actually come across this method of dependency management when using Maven.

I keep mentioning Maven, but there is no reason you can’t using something else, such as Gradle.

Quarkus uses JAX-RS for its REST capabilities

Quarkus follows the JAX-RS specification with Resteasy providing its implementation. If you have worked with Java for a reasonable time, you have probably seen or written code that follows the JAX-RS spec. And if you haven’t, you probably will at some point. I haven’t needed to use it before myself (I tend to use Spring when in Java land) so it is reasonably new to me. Anyway, in general, you can implement most of a simple REST API by relying on a few annotations.

Quarkus uses CDI beans for dependency injection

Quarkus’ dependency solution is based on the CDI (Context and Dependency injection) specification. It should be noted that Quarkus does not provide a full CDI implementation, meaning that there will be limitations compared to other frameworks that you might have used. More information can be found in Quarkus’ CDI reference and their Introduction to CDI.

Implementing a REST API

I have put together an example REST API that manages people. I mean, the purpose of the API doesn’t really matter, it’s just an example after all. Here comes a long chunk of code:

@Path("/people")
public class PersonResource {

  private final PersonRepository personRepository;

  public PersonResource(PersonRepository personRepository) {
    this.personRepository = personRepository;
  }

  @GET
  @Produces(MediaType.APPLICATION_JSON)
  public List<Person> all() {
    return personRepository.findAll();
  }

  @GET
  @Path("/{id}")
  @Produces(MediaType.APPLICATION_JSON)
  public Person get(@PathParam("id") UUID id) {
    return personRepository.findById(id);
  }

  @POST
  @Produces(MediaType.APPLICATION_JSON)
  public Person post(Person person) {
    return personRepository.insert(
        new Person(UUID.randomUUID(), person.getName(), person.getAge())
    );
  }

  @PUT
  @Path("/{id}")
  @Produces(MediaType.APPLICATION_JSON)
  public Person put(@PathParam("id") UUID id, Person person) {
    if (personRepository.findById(id) == null) {
      throw new PersonNotFoundException(id);
    }
    return personRepository.update(new Person(id, person.getName(), person.getAge()));
  }

  @DELETE
  @Path("/{id}")
  @Produces(MediaType.APPLICATION_JSON)
  public void delete(@PathParam("id") UUID id) {
    if (personRepository.findById(id) == null) {
      throw new PersonNotFoundException(id);
    }
    personRepository.deleteById(id);
  }
}

As I mentioned earlier, Quarkus uses Resteasy as its JAX-RS implementation. All of the annotations in this class are related to JAX-RS. Let’s have a quick look at one of the functions:

@GET
@Path("/{id}")
@Produces(MediaType.APPLICATION_JSON)
public Person get(@PathParam("id") UUID id) {
  return personRepository.findById(id);
}

@GET specifies that the method handles GET HTTP requests.
@Path("/{id}") and @PathParam work together to take the HTTP path variable id and use it as the endpoint’s input
@Produces(MediaType.APPLICATION_JSON) denotes the type of data that the endpoint returns as JSON

That’s enough about annotations. The rest of the method consists of finding the Person matching the input id and returning it to whatever called the endpoint. Instead of returning a Person directly, you could use a Response object, but I wanted to keep it simple and easier to read.

JSON serialization

You have a choice between using JSON-B or Jackson for your serialization. Personally, I tend to use Jackson so I went ahead and plugged it in. You will need to add the extension for whichever you choose:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-resteasy-jackson</artifactId>
</dependency>

Or via the command:

./mvnw quarkus:add-extension -Dextensions="resteasy-jackson"

Replace “jackson” with “jsonb” to switch over to JSON-B

Connecting to a database using JDBC

This API is not going to do anything useful if it isn’t persisting data somewhere.

Quarkus provides JDBC implementations for connecting to databases. I chose to use Postgres for this project’s database, but you can obviously use whatever suits your needs. More information can be found in Quarkus’ data source documentation.

The first thing you need to add to connect to a database, is a JDBC driver dependency:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-jdbc-postgresql</artifactId>
</dependency>

You could also run:

./mvnw quarkus:add-extension -Dextensions="jdbc-postgresql"

Then add properties specifying the connection type and the details (to your application.properties):

quarkus.datasource.db-kind=postgresql
# Change this part to match your connection details!
quarkus.datasource.jdbc.url=jdbc:postgresql://localhost:5432/mytestdb
quarkus.datasource.username=admin
quarkus.datasource.password=admin

Connection pooling

You don’t want to directly manage the database connections yourself (I assume so anyway).

Quarkus’ preferred JDBC data source and connection pooling library is Agroal. Below is a quote from the Quarkus data source documentation.

Agroal is a modern, light weight connection pool implementation designed for very high performance and scalability, and features first class integration with the other components in Quarkus, such as security, transaction management components, health metrics.

What this means for you:

Add the Agroal extension to your dependencies:

<dependency>
  <groupId>io.quarkus</groupId>
  <artifactId>quarkus-agroal</artifactId>
</dependency>

Or run:

./mvnw quarkus:add-extension -Dextensions="agroal"

Inject the DataSource that it initialises into your code (either directly via AgroalDataSource or using the DataSource interface).

Creating a repository

Once you have sorted out the data source connection and have enabled connection pooling, you can now start writing queries. Below is a slimmed-down version of the PersonRepository since its using raw JDBC meaning that the code is quite lengthy and duplicated between functions:

@ApplicationScoped
public class PersonRepository {

  private static final String FIND_BY_ID = "SELECT * FROM people WHERE id = ?";

  private final DataSource dataSource;

  public PersonRepository(DataSource dataSource) {
    this.dataSource = dataSource;
  }

  public Person findById(UUID id) {
    try (Connection connection = dataSource.getConnection();
        PreparedStatement statement = connection.prepareStatement(FIND_BY_ID)) {
      statement.setObject(1, id);
      try (ResultSet resultSet = statement.executeQuery()) {
        if (resultSet.next()) {
          return new Person(
              UUID.fromString(resultSet.getString("id")),
              resultSet.getString("name"),
              resultSet.getInt("age"));
        }
      }
    } catch (SQLException e) {
      throw new PersistenceException(e);
    }
    return null;
  }
}

The main take away of this snippet is the usage of the DataSource which Agroal provided the implementation for. You can inject it into your components after only needing to include the dependencies for a JDBC driver and Agroal itself. In regards to the JDBC code, more information on general JDBC usage can be found here.

Running the application

To run a Quarkus application, execute:

./mvnw compile quarkus:dev

This runs the application in dev mode, which enables hot-reloading. Making your development experience a bit smoother. Although, I did find it hard to break the habit of restarting my application after every code change…

To run your application for real, or non dev mode, you need to build the application’s jar and run it. You can do this using:

./mvnw package
java -jar target/quarkus-1.0-SNAPSHOT-runner.jar

./mvnw package places the generated jar in the target directory

If you used the command I showed at the beginning of this post to generate your project, you would have a README.md containing all this information. Including how to construct a native executable and running your Quarkus application within a container. I will cover some of this in future blog posts.

Summary

You should now have a good idea on how to write a REST API using Quarkus. Like most other frameworks, you can leverage dependencies to quickly and effectively build your applications. Quarkus allows you to do the same. That can be seen by the number of times I told you to add a new dependency!

Bringing the focus back onto what Quarkus provides you, remember it’s a framework specialising on fast start-up times and low memory usage. I purposely left further discussion on these factors out to allow the post to focus on building an application using Quarkus, which is pretty much the same as building any REST application that uses JAX-RS. So when developers/engineers, like you, search for how to build a REST API using Quarkus, they can see that it is nothing alien. I will cover the benefits of using Quarkus in future content

I have left parts out to not distract from the main content, but you can find all of the code I used to create the example in my GitHub repository.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Async/await in coroutines

Sun, 02 Aug 2020 00:00:00 GMT

Async/await is a common feature in many languages (naming might vary), that allows you to execute functions asynchronously while waiting for their results at a later point. Doing this can improve the performance of your applications by doing work on multiple threads instead of synchronously doing one operation after another. Executing long-running operations that have no dependencies on each other in this manner can significantly reduce the overall runtime.

For example, the runtime of the code below can be decreased by running both operations asynchronously:

val count1 = occurrenceOfWordInFile("some-build-file.log", "jenkins")
val count2 = occurrenceOfWordInFile("some-other-file.log", "build")
println("Both results retrieved:\n$count1\n$count2")

// Yes, there probably is a better way to do this!
private fun occurrenceOfWordInFile(filePath: String, word: String): Int {
  return File(filePath).useLines { lines ->
    lines.map { line ->
      line.splitToSequence(" ", "-", ".", ",", ":", "/", ignoreCase = true)
        .filter { it == word }
        .count()
    }.sum()
  }
}

Then by using async/await (using the same occurrenceOfWordInFile implementation):

runBlocking {
  val count1 = async(context = Dispatchers.IO) { occurrenceOfWordInFile("some-build-file.log", "jenkins") }
  val count2 = async(context = Dispatchers.IO) { occurrenceOfWordInFile("some-other-file.log", "build") }
  println("Both results retrieved:\n${count1.await()}\n${count2.await()}")
}

I could add the timing code to the example and show you the output, or you could choose to believe me when I say its faster

Each call to async executes the file read in a separate thread (the context needs specifying). The main thread then suspends when it reaches the println since it calls await. Remember that when a coroutine suspends, it frees the current thread to be used elsewhere. This also adds a limitation to await, as it must be executed within the scope of a coroutine. But this limitation makes sense. Otherwise, you could use a CompletableFuture instead and call get which would block the current thread while it waits. Note, I said block and not suspend. That thread can’t be used for anything else, unlike the async/await combo.

Async launches a new job

Calling async is like calling launch, in that it starts a new job/coroutine. The difference is that async eventually returns its result when await is called, whereas you can join to the completion of a job started by launch, but it won’t return a result. You can somewhat achieve the same behaviour by setting a var outside of the launched job and using that after calling join:

runBlocking {
  var count1: Int? = null
  val job1: Job = launch(context = Dispatchers.IO) {
    count1 = occurrenceOfWordInFile("some-build-file.log", "jenkins")
  }
  var count2: Int? = null
  val job2: Job = launch(context = Dispatchers.IO) {
    count2 = occurrenceOfWordInFile("some-other-file.log", "build")
  }
  joinAll(job1, job2)
  println("Both results retrieved:\n$count1\n$count2")
}

It works, but it would be annoying to have to follow this pattern throughout your code.

Async returns a special job

async can’t simply return a Job, as the Job interface is more general and doesn’t have access to await. If it did, then it could have been called in the previous example using launch.

Therefore, a Deferred (a deferred value) is returned instead of a plain old Job.

public interface Deferred<out T> : Job {

    public suspend fun await(): T

    public val onAwait: SelectClause1<T>

    @ExperimentalCoroutinesApi
    public fun getCompleted(): T

    @ExperimentalCoroutinesApi
    public fun getCompletionExceptionOrNull(): Throwable?
}

As you can see, Deferred is also a Job allowing you to cancel it if needed.

Let me rewrite the original async example to highlight the use of Deferred:

runBlocking {
  val count1: Deferred<Int> = async(context = Dispatchers.IO) { occurrenceOfWordInFile("some-build-file.log", "jenkins") }
  val count2: Deferred<Int> = async(context = Dispatchers.IO) { occurrenceOfWordInFile("some-other-file.log", "build") }
  println("Both results retrieved:\n${count1.await()}\n${count2.await()}")
}

Async can leverage suspending functions

I believe this is worth mentioning since my examples throughout this post haven’t called any suspending code. Remember, async launches a new coroutine, so you can do everything you expect a coroutine to do. This includes calling suspending functions to allow the current thread to free up and context switch to another coroutine.

The power of async/await in coroutines is not fully utilised until you use them in this manner. You can launch a long-running and potentially complex coroutine that suspends and resumes multiple times to make optimal use of your machine’s processing power. All while the parent coroutine that awaits the deferred value is also suspended, thus reducing wasted resources.

Summary

You can use async to launch a coroutine that returns a Deferred value which can be accessed by calling await. From a simplistic perspective, you can treat it like a CompletableFuture to execute an operation on a separate thread and retrieve the result when complete. This will let you improve the performance of your application in some situations. You can utilise asyncs full power when you treat it like a coroutine (because it is), to complete more complex code that can suspend at various points. It also reduces the boilerplate you need to write, removing the need to set the result in a local variable and joining before accessing it.

Cancelling child coroutines

Sun, 26 Jul 2020 00:00:00 GMT

Following on from my previous post, Cancelling coroutines, we will look at how cancelling a coroutine affects its children.

First let’s clarify what a child is in this context:

runBlocking {
  val parent = launch(context = Dispatchers.Default) {
    val child = launch(context = Dispatchers.Default) {
      repeat(100) {
        println("[child] $it.. ")
        delay(50)
      }
    }
  }
}

The child is the second coroutine created in the snippet above. The coroutine whose scope the child was created in is the parent.

Cancelling a job cancels its children

Cancelling a parent coroutine/job will also cancel all of its child coroutines. Allowing you to submit a single cancellation request to stop many executing coroutines.

Below is example:

runBlocking {
  val parent = launch(context = Dispatchers.Default) {
    // children
    (1..2).map {
      launch(context = Dispatchers.Default) {
        repeat(100) { i ->
          println("[child $it] $i.. ")
          delay(50)
        }
      }
    }
    repeat(100) { i ->
      println("[parent] $i.. ")
      delay(50)
    }
  }
  delay(200)
  println("Cancelling")
  parent.cancel()
  parent.join()
  println("Finished cancelling job")
}

Which outputs:

[child 1] 0.. 
[parent] 0.. 
[child 2] 0.. 
[child 1] 1.. 
[child 2] 1.. 
[parent] 1.. 
[parent] 2.. 
[child 2] 2.. 
[child 1] 2.. 
[parent] 3.. 
[child 1] 3.. 
[child 2] 3.. 
Cancelling
Finished cancelling job

Both the parent and child coroutines here are cancelled by the single Job.cancel executed on the parent. You don’t need to manually retrieve the children from the parent and cancel them yourself. The Kotlin coroutine library developers are kind enough to add this functionality for you. To be honest, you’d probably be a bit annoyed if this ability was not available. Oh, the highs and lows of library development!

Cancelling via a coroutine’s scope

In the previous section the children were cancelled by a call to Job.cancel. You can achieve the same behaviour by calling CoroutineScope.cancel. Let’s look at a quick example:

runBlocking {
  val parent = launch(context = Dispatchers.Default) {
    // children
    (1..2).map {
      launch(context = Dispatchers.Default) {
        repeat(500) { i ->
          println("[child $it] $i.. ")
          delay(50)
        }
      }
    }
    delay(200)
    println("Cancelling in scope")
    // Added [this@launch] to make things a bit clearer
    // You could call [cancel] by itself in this situation
    this@launch.cancel()
  }
  parent.join()
  println("Finished cancelling job")
}

CourtineScope.cancel is an alternate path to Job.cancel:

public fun CoroutineScope.cancel(cause: CancellationException? = null) {
  val job = coroutineContext[Job] ?: error("Scope cannot be cancelled because it does not have a job: $this")
  job.cancel(cause)
}

Depending on what you are doing, this might be advantageous, but my current opinion is that Job.cancel is likely to be more useful. For example, you can maintain a list of executing coroutines/jobs that you can cancel when needed. Whereas CoroutineScope.cancel needs to be triggered from inside the scope itself. Therefore restraining its usefulness. Note, this comes from someone who has not had tons of experience using coroutines out in the wild.

A benefit it does provide is a clear point where new coroutines will not start. After calling CoroutineScope.cancel, no children will be spawned. This allows you to make a decision, from within the scope, whether to continue of not. Achieving such clear cut behaviour like this from outside the scope would be difficult.

Summary

Coroutines can create parent-child relationships allowing a parent to cancel the child coroutines/jobs running within its context. This ability will enable you to easier control the execution of your coroutines. You can choose to cancel only a parent coroutine and allow the library to handle the rest. You can use either Job.cancel or CoroutineScope.cancel, even though under the hood they do the exact same thing.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Missing @EntityScan annotation in Spring Data Neo4j

Mon, 20 Jul 2020 00:00:00 GMT

Not including the @EntityScan annotation and specifying the correct packages in your Spring Data Neo4j application can cause the following error:

java.lang.ArrayIndexOutOfBoundsException: Index 0 out of bounds for length 0
	at org.neo4j.ogm.context.EntityRowModelMapper.extractColumnValue(EntityRowModelMapper.java:75) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.context.EntityRowModelMapper.map(EntityRowModelMapper.java:64) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.session.delegates.ExecuteQueriesDelegate.lambda$executeAndMap$1(ExecuteQueriesDelegate.java:151) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:590) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.session.Neo4jSession.doInTransaction(Neo4jSession.java:564) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.session.delegates.ExecuteQueriesDelegate.executeAndMap(ExecuteQueriesDelegate.java:138) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.session.delegates.ExecuteQueriesDelegate.query(ExecuteQueriesDelegate.java:111) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at org.neo4j.ogm.session.Neo4jSession.query(Neo4jSession.java:425) ~[neo4j-ogm-core-3.2.12.jar:3.2.12]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:na]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
	at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
	at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282) ~[spring-core-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.data.neo4j.transaction.SharedSessionCreator$SharedSessionInvocationHandler.lambda$invoke$1(SharedSessionCreator.java:121) ~[spring-data-neo4j-5.3.1.RELEASE.jar:5.3.1.RELEASE]
	at org.springframework.data.neo4j.transaction.SharedSessionCreator$SharedSessionInvocationHandler.invokeInTransaction(SharedSessionCreator.java:159) ~[spring-data-neo4j-5.3.1.RELEASE.jar:5.3.1.RELEASE]
	at org.springframework.data.neo4j.transaction.SharedSessionCreator$SharedSessionInvocationHandler.invoke(SharedSessionCreator.java:123) ~[spring-data-neo4j-5.3.1.RELEASE.jar:5.3.1.RELEASE]
	at com.sun.proxy.$Proxy62.query(Unknown Source) ~[na:na]
	at org.springframework.data.neo4j.repository.query.GraphQueryExecution$CollectionExecution.execute(GraphQueryExecution.java:97) ~[spring-data-neo4j-5.3.1.RELEASE.jar:5.3.1.RELEASE]
	at org.springframework.data.neo4j.repository.query.GraphRepositoryQuery.doExecute(GraphRepositoryQuery.java:76) ~[spring-data-neo4j-5.3.1.RELEASE.jar:5.3.1.RELEASE]
	at org.springframework.data.neo4j.repository.query.AbstractGraphRepositoryQuery.execute(AbstractGraphRepositoryQuery.java:57) ~[spring-data-neo4j-5.3.1.RELEASE.jar:5.3.1.RELEASE]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor$QueryMethodInvoker.invoke(QueryExecutorMethodInterceptor.java:195) ~[spring-data-commons-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:152) ~[spring-data-commons-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.invoke(QueryExecutorMethodInterceptor.java:130) ~[spring-data-commons-2.3.1.RELEASE.jar:2.3.1.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:367) ~[spring-tx-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:118) ~[spring-tx-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.dao.support.PersistenceExceptionTranslationInterceptor.invoke(PersistenceExceptionTranslationInterceptor.java:139) ~[spring-tx-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:95) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212) ~[spring-aop-5.2.7.RELEASE.jar:5.2.7.RELEASE]
	at com.sun.proxy.$Proxy76.findAll(Unknown Source) ~[na:na]

Yes, there is a lot there, and you will need to scroll across to see it. The primary information to extract is that there is an ArrayIndexOutOfBoundsException when trying to map a query’s result to an object, as denoted by EntityRowModelMapper.map and EntityRowModelMapper.extractColumnValue.

The error doesn’t really tell you much.

Thankfully, there is a high probability that you are missing the @EntityScan annotation. Adding this to one of your configuration files or your main class (if you’re using Spring Boot, its likely to be the class you annotated with @SpringBootApplication) and specifying the basePackages or basePackageClasses should tell Spring where to find your entity and relationship classes.

You are more likely to see the error above if you are using Spring Boot and do not have your @SpringBootApplication class in one of the root packages of your application’s code. If this is the case, and you are using Spring Data’s inferred queries, you will also need to include the @EnableNeo4jRepositories annotation. It will face the same style of problem as the missing @EntityScan.

For clarity below is a quick example:

package dev.lankydan.neo4j.application;

// Already needed to specify packages for other parts of the application
@SpringBootApplication(scanBasePackages = "dev.lankydan.neo4j")
@EnableNeo4jRepositories(basePackages = "dev.lankydan.neo4j")
public class Application implements CommandLineRunner {

  public static void main(String[] args) {
    SpringApplication.run(Application.class);
  }
}

This is the main class of the application, note that the Application class is located in dev.lankydan.neo4j.application.

Now for the entity itself:

package dev.lankydan.neo4j.entity;

@NodeEntity(value = "City")
public class City {
  // contents of the class
}

The entity has been placed in dev.lankydan.neo4j.entity, which is not a sub-package of dev.lankydan.neo4j.application. Therefore, Application needs to be adjusted, giving the final iteration:

package dev.lankydan.neo4j.application;

// Already needed to specify packages for other parts of the application
@SpringBootApplication(scanBasePackages = "dev.lankydan.neo4j")
@EnableNeo4jRepositories(basePackages = "dev.lankydan.neo4j")
// The [basePackages] could be specific like the one below or more general like the declarations above
@EntityScan(basePackages = "dev.lankydan.neo4j.entity")
public class Application implements CommandLineRunner {

  public static void main(String[] args) {
    SpringApplication.run(Application.class);
  }
}

Spring can now detect the entity classes and allow the application to run smoothly. A second option would be to move Application to a different package, closer to the root of the project. In this scenario, moving it to dev.lankydan.neo4j would do the trick.

Hopefully, this will help you remedy your code’s issue.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Mapping a path query in Spring Data Neo4j

Sun, 19 Jul 2020 00:00:00 GMT

Spring data does a lot to help you focus on writing your cypher queries while it handles mapping the results for you. However, when your queries become more complex, it starts to struggle. This is where you need to step in and explicitly map the query results into your domain objects.

In this post, we will look at how you can query a path and map the results using Spring Data Neo4j.

Simple queries are mapped for you

For simple queries, you can get away with only defining the relevant entity and relationship classes, Spring will do the rest.

Let’s use a findAll function (returns all nodes):

@Query("MATCH (n:City) RETURN n LIMIT 25")
Iterable<City> findAll();

The City entity needs to defined, as such:

@NodeEntity(value = "City")
public class City {

  @Id
  @GeneratedValue
  private Long id;
  private String name;
  private double longitude;
  private double latitude;

  @Relationship(type = "NEXT")
  private Set<Connection> connections = Collections.emptySet();

  public City(
      Long id,
      String name,
      double longitude,
      double latitude,
      Set<Connection> connections
  ) {
    this.id = id;
    this.name = name;
    this.longitude = longitude;
    this.latitude = latitude;
    this.connections = connections;
  }

  public City() { }

  // getters + setters
}

City also has a relationship, so that needs to be defined as well:

@RelationshipEntity("NEXT")
public class Connection {

  @Id
  @GeneratedValue
  private long id;
  @StartNode
  private City start;
  @EndNode
  private City end;

  public Connection(long id, City start, City end) {
    this.id = id;
    this.start = start;
    this.end = end;
  }

  public Connection() { }

  // getters + setters
}

Calling the findAll function will correctly collect the cities from the database and map them for you.

Everything is good at this point.

Another example of setting up a Neo4j entity class can be found in the Spring Data Neo4j - getting started documentation.

Extracting paths requires your assistance

The code above is not suitable for a cypher query that returns a path.

The following query will be used for this section (taken from the Apoc plugin’s documentation):

MATCH (a:City {name:'bruges'}), (b:City {name:'dresden'})
MATCH p=(a)-[*]->(b)
WITH collect(p) as paths
CALL apoc.spatial.sortByDistance(paths) YIELD path, distance
RETURN path, distance

This query returns the path between two cities and the total distance of that path.

Unfortunately, there is no way to represent this information using the entity and relationship classes defined earlier. Furthermore, there is no way to currently do this using Spring Data Neo4j full stop. It personally took me a long time to realise this fact, until I came across this excerpt in their documentation.

Custom queries do not support a custom depth. Additionally, @Query does not support mapping a path to domain entities, as such, a path should not be returned from a Cypher query. Instead, return nodes and relationships to have them mapped to domain entities.

Somehow I managed to keep brushing over this information when I went through the docs…

This means you cannot do something like this:

public interface PathRepository extends Neo4jRepository<City, Long> {

  @Query("MATCH (a:City {name:$departure}), (b:City {name:$arrival})\n"
      + "MATCH p=(a)-[*]->(b)\n"
      + "WITH collect(p) as paths\n"
      + "CALL apoc.spatial.sortByDistance(paths) YIELD path, distance\n"
      + "RETURN path, distance")
  List<Path> getAllPaths(String departure, String arrival);
}

Where Path contains the following:

@QueryResult
public class Path {
  public List<City> path;
  public double distance;

  public Path(List<City> path, double distance) {
    this.path = path;
    this.distance = distance;
  }
}

The @QueryResult annotation allows you to define a custom object to contain the results of your cypher query, more can be found in the Spring docs

If you do this, it will still execute, but there will be no information about the cities filled in.

How to manually map a path

To be able to map the path shown in the previous section, you need to manually assign the results from the cypher query into objects.

This can be done with the following code (large snippet incoming):

@Repository
// Class instead of interface
// Extend [SimpleNeo4jRepository] instead of [Neo4jRepository]
public class PathRepository extends SimpleNeo4jRepository<City, Long> {

  private static final String GET_ALL_PATHS_QUERY =
      "MATCH (a:City {name:$departure}), (b:City {name:$arrival})\n"
          + "MATCH p=(a)-[*]->(b)\n"
          + "WITH collect(p) as paths\n"
          + "CALL apoc.spatial.sortByDistance(paths) YIELD path, distance\n"
          + "RETURN path, distance";

  // Needed to be able to query the database
  private final Session session;

  // Inject in the session
  // No need to create the session yourself, Spring has already created it
  public PathRepository(Session session) {
    super(City.class, session);
    this.session = session;
  }

  public List<Path> getAllPaths(String departure, String arrival) {
    Map<String, String> parameters = Map.of(
        "departure", departure,
        "arrival", arrival
    );

    // Execute the query and retrieve the result
    Result rows = session.query(GET_ALL_PATHS_QUERY, parameters);

    List<Path> results = new ArrayList<>();
    for (Map<String, Object> row : rows) {
      results.add(convertRow(row));
    }

    return results;
  }

  private Path convertRow(Map<String, Object> row) {
    InternalPath.SelfContainedSegment[] connections =
        (InternalPath.SelfContainedSegment[]) row.get("path");

    List<City> cities = new ArrayList<>();
    // Iterate through the segments in the path
    for (InternalPath.SelfContainedSegment connection : connections) {
      cities.add(convert(connection));
    }

    double distance = (Double) row.get("distance");
    return new Path(cities, distance);
  }

  private City convert(InternalPath.SelfContainedSegment connection) {
    // Extract the information about the [City] from the path segment
    // Information about the start node and the relationship could also be accessed
    return new City(
        connection.end().id(),
        connection.end().get("name").asString(),
        connection.end().get("latitude").asDouble(),
        connection.end().get("longitude").asDouble()
    );
  }
}

This iteration of PathRepository extends SimpleNeo4jRepository to inherit some of the more common queries without requiring you to implement them yourself. You do not need to extend this class, but I suggest you do.

I don’t think there is any need to explain the rest of the example, I tidied up the code as best I could and added comments for clarity. Hopefully, they are enough!

Conclusion

As I eventually found out and have now mentioned to you, at the time of writing this post, Spring Data Neo4j does not support the automatic mapping of queries containing paths. Therefore you will need to take your fate into your own hands and extract the data yourself. The code in the penultimate section (the long code snippet) shows you how to do this. Using a similar structure, you should be able to adapt this for your own use to solve your own problems. I hope this saves you some time as it took me a while to realise this was the only way to retrieve a path, when using Spring Data Neoj4 anyway…

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Java friendly Kotlin - default arguments

Thu, 25 Jun 2020 00:00:00 GMT

Kotlin functions and constructors can define default arguments, allowing calls to them to skip any argument that has a default value.

This allows the following function to be called in a number of ways:

fun doStuff(
  a: String = "Default value",
  b: Int = 1,
  c: Boolean = false
)

doStuff()
doStuff("here's a value")
doStuff("here's a value", 2)
doStuff("here's a value", 2, true)
doStuff("here's a value", c = true)
doStuff(b = 2)
doStuff(c = true)
doStuff(b = 2, c = true)

More information can be found in the Kotlin documentation.

For the rest of this post, we will look at how you can include default arguments in your API while still providing excellent Java compatibility.

Trying to call an unfriendly Kotlin function

You are not going to have a good time calling a function or constructor with default arguments from Java. The only way to call the function shown previously from Java is to provide every argument that it asks for:

doStuff("here's a value", 2, true);

Without some help, there is no way for Java to understand the concept of default arguments.

Applying the @JvmOverloads annotation

The @JvmOverloads annotation can be applied to functions and constructors to tell the compiler to generate extra overloads depending on their default arguments. The overloads are placed in the compiled bytecode. You can then execute them from Java as you would with any other function or constructor.

Let’s have a look at the KDoc for @JvmOverloads which has a precise declaration of how it generates overloads:

/**
 * Instructs the Kotlin compiler to generate overloads for this function that 
 * substitute default parameter values.
 *
 * If a method has N parameters and M of which have default values, M overloads 
 * are generated: the first one takes N-1 parameters (all but the last one that 
 * takes a default value), the second takes N-2 parameters, and so on.
 */
@Target(AnnotationTarget.FUNCTION, AnnotationTarget.CONSTRUCTOR)
@Retention(AnnotationRetention.BINARY)
@MustBeDocumented
public actual annotation class JvmOverloads

In other words:

Starts at the last argument.
If the argument has a default value, an overload is generated without that argument, who uses the default value in its implementation.
If the penultimate argument has a default value, an overload without the last 2 arguments is generated.
Continues to apply this logic until it reaches the first argument or hits an argument that does not have a default value.

Some examples should help grasp this definition.

Below are the different ways to call the doStuff function from earlier in Java:

// All arguments have default values
@JvmOverloads
fun doStuff(
  a: String = "Default value",
  b: Int = 1,
  c: Boolean = false
)

// Ways to call [doStuff] from Java
doStuff();
doStuff("here's a value");
doStuff("here's a value", 2);
doStuff("here's a value", 2, true);

There are now 4 options for Java to trigger (instead of 1 without the annotation). All the arguments have been removed, one by one, leaving 3 overloads where 1 requires no inputs at all.

It we remove one of the default values, then the generated options will change:

// Remove default value from [b]
@JvmOverloads
fun doStuff(
  a: String = "Default value",
  b: Int,
  c: Boolean = false
)

// Ways to call [doStuff] from Java
doStuff("here's a value", 2);
doStuff("here's a value", 2, true);

Now, when creating overloads, the compiler hits b (second argument) and stops. It does not matter that a has a default value, the compiler will not progress from this point. Therefore, only a single extra overload is available to Java.

Applying the annotation to a class’ constructor requires a slight manipulation to its structure:

class MyJvmOverloadsClass @JvmOverloads constructor(
  private val a: String = "Default value",
  private val b: Int = 1,
  private val c: Boolean = false
) {

  // Rest of the class
}

The annotation must be applied directly to one of the class’ constructor. In this case, as there is only a single constructor, the constructor keyword must be added (this can typically be omitted in a Kotlin class). Applying the annotation generates overloads in the same way that it does to a function.

The annotation isn’t perfect

The issue with the @JvmOverloads annotation is that it does not generate every possible combination of arguments. Kotlin can bypass this as it has access to named arguments.

Java does not have this ability. Even with the @JvmOverloads annotation, this is what prevents Java from accessing all the same options as Kotlin.

It makes sense when you think about it.

A function or constructor can only be overloaded when it has different arguments from all other versions, for example (written in Kotlin but Java follows the same rules):

fun doStuff(a: String = "Default value"): String
fun doStuff(a: String = "Default value", b: Int = 1): String
fun doStuff(a: String = "Default value", b: Int = 1, c: Boolean = false): String
// Return type can be different as long as it has different arguments
fun doStuff(a: String = "Default value", b: Int = 1, c: Boolean = false, d: Long = 2): Int

The difference between each overload is clear because the argument types are different. But if they were all the same type it becomes harder to use:

fun doStuff(a: String = "Default value"): String
fun doStuff(a: String = "Default value", b: String = "Another value"): String
fun doStuff(a: String = "Default value", b: String = "Another value", c: String = "Another one"): String

Named arguments makes this bearable (assuming they are named better than a, b and c). Allowing you to omit some arguments due to their default values.

Without the use of named arguments, the is no way for the compiler to distinguish between the objects that you pass to this function. So all it can do is pass them in, one by one, in order, into the function. This is the problem that @JvmOverloads faces. It cannot generate all possible iterations because there is not enough information to differentiate the overloads from each other. If all the arguments have different types, then technically the compiler can do it, but having a rule that is loosely applied would become confusing.

This is the reason that @JvmOverloads only generates overloads starting from the last argument, who must also have a default value, and then moves onto the next (backwards towards the first argument).

Cooperating with the annotation

You can work around the limitations of @JmvOverloads. Doing so requires you to think about how your functions will be used and what combinations of parameters will be frequently used.

Below are some points for you to consider when writing a function annotated with @JvmOverloads:

Order the arguments in order of importance, the first as the most important and decreasing as they go on.

@JvmOverloads
fun doStuff(
  superImportant: String = "I need this argument!",
  reallyImportant: Int = 1,
  somewhatImportant: Boolean = false,
  dontReallyCareAboutThisOne: Long = 2
)

Do not mix arguments with default values with ones that do not (easier to understand with the example).

// Stops generating overloads when [doesNotHaveDefaultValue] is reached
@JvmOverloads
fun doStuff(
  hasDefaultValue: String = "I need this argument!",
  doesNotHaveDefaultValue: Int,
  hasDefaultValue2: Boolean = false,
  hasDefaultValue3: Long = 2
)


// No overloads can be generated
@JvmOverloads
fun doStuff(
  hasDefaultValue: String = "I need this argument!",
  doesNotHaveDefaultValue: Int,
  hasDefaultValue2: Boolean = false,
  doesNotHaveDefaultValue2: Long
)

Manually create overloads if there are still combinations missing that you deem useful.

// Original
@JvmOverloads
fun doStuff(
  a: String = "I need this argument!",
  b: Int = 1,
  c: Boolean = false,
  d: Long = 2
)

// Calling with just [a] and [c] is useful
fun doStuff(a: String, c: Boolean) {
  // Add in the missing parameters
  // Copy the default values or maybe they are taken from properties of a class
  // instead of being passed into the function
  return doStuff(a, 1, c, 2)
}

The main piece of advice I want to give you here is to really think about the functions you create. When creating a public API as part of a library, especially when you want it to play well with Java, spending time considering how your functions will be leveraged will make everyone happier.

Developers consuming a Kotlin API when using the language themselves can use its features to get around potential problems in your code. Java, on the other hand, requires you to apply some of your brainpower to write functions that are friendly to use.

Summary

Adding the @JvmOverloads annotation to your functions and constructors persuades them to play nicer with Java. It does so by generating additional overloads and placing them in the bytecode that Java interacts with. Adding the annotation does not always make your API easily accessible to Java, in those situations, it is on you to put the work in and craft a well thought out API.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Transaction savepoints in Spring JDBC

Sun, 14 Jun 2020 00:00:00 GMT

Savepoints allow you to create markers within a transaction which you can rollback to, without preventing the transaction from being committed at a later point. These can be treated like intermediate transactions within a single overarching transaction. At the end of the day, you either commit the transaction and persist all the changes to the database or rollback everything. Using savepoints, you can handle potential database errors and return to a safe point within the transaction and carry on.

This post will look at how you can use savepoints within Spring JDBC.

Spring provides a few options to manage savepoints within your transactions, we will look at the following choices:

Using JdbcTemplate
Using TransactionTemplate
The @Transactional annotation

Savepoints with JdbcTemplate

Let’s start with the JdbcTemplate:

@Component
class PersonManager(private val jdbcTemplate: JdbcTemplate) {


  fun process() {
    jdbcTemplate.execute { connection: Connection ->
      connection.autoCommit = false

      connection.save(1, dan)
      connection.save(2, laura)

      val savepoint = connection.setSavepoint()
      try {
        connection.save(1, george)
      } catch (e: SQLException) {
        log.info("There was an exception, rolling back to savepoint: ${e.message}")
        connection.rollback(savepoint)
      } finally {
        connection.releaseSavepoint(savepoint)
      }
    }
  }

  private fun Connection.save(id: Int, person: Person) {
    prepareStatement("INSERT INTO people(id, name, age) VALUES (?, ?, ?)").apply {
      setInt(1, id)
      setString(2, person.name)
      setInt(3, person.age)
    }.executeUpdate()
  }
}

Which outputs the following (the same is output in all following sections of this post):

There was an exception, rolling back to savepoint: ERROR: duplicate key value violates unique constraint "people_pkey"
People => [Person(id=1, name=Dan, age=26), Person(id=2, name=Laura, age=25)]

Enabling savepoints using JdbcTemplate requires a bit of effort. You need to make sure that you disable the Connection’s autoCommit mode, otherwise you will see an error like the following:

Caused by: org.springframework.jdbc.UncategorizedSQLException: ConnectionCallback; uncategorized SQLException; SQL state [25P01]; error code [0]; 
    Cannot establish a savepoint in auto-commit mode.; nested exception is org.postgresql.util.PSQLException: Cannot establish a savepoint in auto-commit mode.
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:89) ~[spring-jdbc-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.jdbc.support.AbstractFallbackSQLExceptionTranslator.translate(AbstractFallbackSQLExceptionTranslator.java:81) ~[spring-jdbc-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.translateException(JdbcTemplate.java:1443) ~[spring-jdbc-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:336) ~[spring-jdbc-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at dev.lankydan.jdbc.PersonManager.processWithJdbcTemplate(Application.kt:73) ~[main/:na]
    at dev.lankydan.jdbc.PersonManager$$FastClassBySpringCGLIB$$56c95970.invoke(<generated>) ~[main/:na]
    at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:685) ~[spring-aop-5.2.0.RELEASE.jar:5.2.0.RELEASE]
    at dev.lankydan.jdbc.PersonManager$$EnhancerBySpringCGLIB$$d54410c6.processWithJdbcTemplate(<generated>) ~[main/:na]
    at dev.lankydan.jdbc.Application.run(Application.kt:32) ~[main/:na]
    at org.springframework.boot.SpringApplication.callRunner(SpringApplication.java:784) [spring-boot-2.2.0.RELEASE.jar:2.2.0.RELEASE]
    ... 6 common frames omitted

After that is handled, you can create a savepoint using Connection.setSavepoint.

SQL statements can then be executed as usual. Any pieces of code that you think are going to cause issues (but will only lead to soft failures) should be wrapped with a try/catch block. If any errors occur, they can then be handled within the catch. This is what the code example above demonstrates.

An attempt to save a record with an ID that already exists is made thus, throwing an exception. Without the savepoint, this would put the transaction into a terminal state, preventing it from ever successfully committing. Instead, as long as the exception is caught, the transaction can be rolled back to the savepoint. This is achieved by calling rollback while referencing the savepoint. The transaction can then continue as usual and achieve its goal of committing its contents to the database.

Before committing the transaction, its worth releasing the savepoint using releaseSavepont. Saying that, not all the JDBC drivers actually support releasing savepoints. The Javadocs note that SQLFeatureNotSupportedException is thrown by drivers that do not support releaseSavepoint. So, you might need to keep your eye out for these potential errors.

Savepoints with TransactionTemplate

TransactionTemplate works similarly to JdbcTemplate but with a greater focus on support for transactions, the name gives it away, right?

@Component
class PersonManager(
  private val jdbcTemplate: JdbcTemplate,
  private val transactionTemplate: TransactionTemplate
) {

  fun process() {
    transactionTemplate.execute { status: TransactionStatus ->
      jdbcTemplate.save(1, dan)
      jdbcTemplate.save(2, laura)

      val savepoint = status.createSavepoint()
      try {
        jdbcTemplate.save(1, george)
      } catch (e: DataAccessException) {
        // [DataAccessException] because spring converts the underlying [SQLException] inside of [update]
        log.info("There was an exception, rolling back to savepoint: ${e.message}")
        status.rollbackToSavepoint(savepoint)
      } finally {
        status.releaseSavepoint(savepoint)
      }
    }
  }

  private fun JdbcTemplate.save(id: Int, person: Person) {
    update("INSERT INTO people(id, name, age) VALUES (?, ?, ?)") { statement ->
      statement.setInt(1, id)
      statement.setString(2, person.name)
      statement.setInt(3, person.age)
    }
  }
}

This code is very similar to that shown in the JdbcTemplate example. The outcome is exactly the same, and with only a few differences between them. TransactionTemplate provides support for transactions which is not offered by JdbcTemplate without the code shown before.

To use savepoints with TransactionTemplate, you should interact with the TransactionStatus passed into the execute lambda. This allows you to create a savepoint using createSavepoint, rollback with rollbackToSavepoint and then release it by calling releaseSavepoint. Not much else to say really.

Note that JdbcTemplate is used within TransactionTemplate.execute and the underlying Connection is not being directly interacted with.

Savepoints with @Transactional

This option is the one that stands out from the others. Focusing on creating and committing transactions through the use of an annotation:

@Component
class PersonManager(
  private val jdbcTemplate: JdbcTemplate,
  private val transactionalAnnotationManager: TransactionalAnnotationPersonManager
) {

  @Transactional
  fun process() {
    jdbcTemplate.save(1, dan)
    jdbcTemplate.save(2, laura)

    try {
      transactionalAnnotationManager.processWithTransactionalAnnotationSavepoint()
    } catch (e: DataAccessException) {
      log.info("There was an exception, rolling back to savepoint: ${e.message}")
    }
  }
}

@Component
class TransactionalAnnotationPersonManager(private val jdbcTemplate: JdbcTemplate) {

  // By default, auto rollbacks for runtime exceptions but not checked exceptions
  // [DataAccessException] is a runtime exception, hence the rollback
  @Transactional(propagation = Propagation.NESTED)
  fun processWithSavepoint() {
    jdbcTemplate.save(1, george)
  }
}

private fun JdbcTemplate.save(id: Int, person: Person) {
  update("INSERT INTO people(id, name, age) VALUES (?, ?, ?)") { statement ->
    statement.setInt(1, id)
    statement.setString(2, person.name)
    statement.setInt(3, person.age)
  }
}

This example looks nothing like the first two.

@Transactional enables transaction management starting from the first function called with the annotation. Subsequent calls to JdbcTemplate functions will be executed within a transaction created by the function annotated with @Transactional. This seems to be the recommended way of enabling transactions within Spring JDBC.

Enabling savepoints with @Transactional is a bit strange. To explain why, you first need a bit of background context. When a Spring component contains a function annotated with @Transactional, a proxy instance is generated that stands in front of the original class (PersonManager and TransactionalAnnotationPersonManager in the example above). Calls to these classes will instead communicate with the proxy, which wraps the original function calls in transactions. The proxy only intercepts calls from external classes. Any calls from other functions within the class will not apply the transaction wrapping that the proxy does.

This makes it harder to enable savepoints when using @Transactional. If there is a call to a function marked with @Transactional(propagation = Propagation.NESTED) (which is what denotes the use of a savepoint), it will not do anything unless called from an external class (outside the proxy). This is why there are two classes in the example. One creates the original, overarching transaction. The other will create an inner transaction (using a savepoint) when called from an existing transaction.

When you call functions in this particular way, savepoints and the overall transaction will be managed for you.

Despite there being two classes, the amount of code isn’t particularly bad. That being said, its a bit confusing at first glance and it hides a lot behind a curtain while waving its hands and saying “behold Spring’s magic”. I am generally okay with the Spring’s magic, but this one was too much for me.

Summary

Spring provides you with multiple ways to manage savepoints within transactions:

Using JdbcTemplate and controlling the Connection gives you
Leveraging TransactionTemplate and the abstraction it provides over the above
Annotating functions with @Transactional(propagation = Propagation.NESTED) to have Spring sprinkle some magic for you

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Cancelling coroutines

Mon, 11 May 2020 00:00:00 GMT

Sooner or later, you will need to cancel a coroutine that you started. Let’s look at how you can do just that.

How to cancel a job

One of the functions available to a job is cancel. I don’t know about you, but that sounds like the function we need. We can also leverage join to wait until the job is finished cancelling.

The example below shows a job being cancelled:

runBlocking {
  val job = launch(Dispatchers.Default) {
    for (i in 0..1000) {
      delay(50)
      println("$i..")
    }
    println("Job is completed")
  }
  delay(500)
  println("Cancelling")
  job.cancel()
  job.join()
  println("Cancelled and done")
}

Outputting:

0..
1..
2..
3..
4..
5..
6..
7..
8..
Cancelling
Cancelled and done

Note, that Job is completed is never output as the job was cancelled before this could occur.

Job also provides cancelAndJoin to combine the two parts together. This will be used for the remainder of this post.

Cancellation is cooperative

As the Kotlin docs say, “coroutine cancellation is cooperative”. I really like this wording, and I think it goes a long way to describe what your part in writing coroutines that are cancellable.

I think this wording is so good, that I stole the heading from the Kotlin docs - Cancellation is cooperative… There is no better way to word what needs to be said here. If you want to leave and go through their docs instead, I understand.

For anyone that has decided to stay, let’s carry on.

A coroutine needs to cooperate to be cancellable. In other words, you need to take into account the contents of your coroutines to ensure that they can be cancelled. You can make your coroutines cancellable by following the two options below:

Calling any suspending functions from kotlinx.coroutines
Using CoroutineScope.isActive and handling the outcome appropriately

Both of these will be expanded in the following sections.

Kotlinx suspending functions are cancellable

All of the suspending functions provided by kotlinx.coroutines will check if the coroutine calling them is cancelled and throw a CancellationException if it has been.

The previous example demonstrated this. I have added it below again but tidied it up a little bit:

runBlocking {
  val job = launch(Dispatchers.Default) {
    for (i in 0..1000) {
      delay(50)
      println("$i..")
    }
  }
  delay(500)
  job.cancelAndJoin()
}

delay is a suspending function and therefore checks if the coroutine has been cancelled. The code for delay indicates how it handles cancellation:

/**
 * Delays coroutine for a given time without blocking a thread and resumes it after a specified time.
 * This suspending function is cancellable.
 * If the [Job] of the current coroutine is cancelled or completed while this suspending function is waiting, this function
 * immediately resumes with [CancellationException].
 *
 * Note that delay can be used in [select] invocation with [onTimeout][SelectBuilder.onTimeout] clause.
 *
 * Implementation note: how exactly time is tracked is an implementation detail of [CoroutineDispatcher] in the context.
 * @param timeMillis time in milliseconds.
 */
public suspend fun delay(timeMillis: Long) {
    if (timeMillis <= 0) return // don't delay
    return suspendCancellableCoroutine sc@ { cont: CancellableContinuation<Unit> ->
        cont.context.delay.scheduleResumeAfterDelay(timeMillis, cont)
    }
}

The Kdoc mentions what happens if the calling coroutine has been cancelled. The same documentation is also included in other suspending functions. Furthermore, the call to suspendCancellableCoroutine provides a few more clues into what happens, but we’re not going to continue this investigation any further.

You might have also noticed that the CancellationException thrown by delay is not caught in the previous example. It is handled by the coroutine and will not propagate to the calling thread. You can decide to catch it inside your coroutine, but be aware that any calls to built-in suspending functions will end up throwing another CancellationException. Some extra information around this topic that is not covered here can be found in the Kotlin docs - Closing resources with finally.

Checking isActive

Cancelling a coroutine will change a job’s isActive flag to false. This flag can then be used to check if a job is still running, has been cancelled or one of the other non-running states.

As discussed in the previous section, the built-in coroutines will handle cancellation for you. Therefore you’re going to need to check the isActive flag for the following reasons:

You aren’t using any suspending functions from kotlinx-coroutines inside your coroutine
You have parts of your coroutine that don’t call a suspending function
You are writing your own suspending function that should be cancellable

To explore the use of the isActive flag, we will focus on a set of examples consisting of coroutines that continually loop until they are cancelled.

Below are some methods you can use to handle cancellation:

Conditionally executing code using isActive as part of a while loop
Conditionally executing code using isActive and an if statement
Using return to escape
Throwing an exception using isActive
Escaping a coroutine using ensureActive

The following sub-sections contain examples of these.

Conditionally executing code using isActive as part of a while loop

isActive is a boolean property after all, therefore it can be used in a while loop:

runBlocking {
  val job = launch(Dispatchers.Default) {
    while (isActive) {
      Thread.sleep(50)
      println("I am still going..")
    }
  }
  delay(500)
  job.cancelAndJoin()
}

This example checks the isActive flag as part of the while loop. Once the job is cancelled, the value becomes false, and the loop ends.

Conditionally executing code using isActive and an if statement

In a very similar way to the while loop, you can check the state of isActive with an if statement to decide whether to execute some code:

runBlocking {
  val job = launch(Dispatchers.Default) {
    for (it in 0..1000) {
      if (isActive) {
        Thread.sleep(50)
        println("I am still going..")
      }
    }
  }
  delay(500)
  job.cancelAndJoin()
}

Using return to escape

You can use return to break out of a loop and stop further processing if the job has been cancelled:

runBlocking {
  val job = launch(Dispatchers.Default) {
    for (it in 0..1000) {
      if (!isActive) {
        return@launch
      }
      Thread.sleep(50)
      println("I am still going..")
    }
  }
  delay(500)
  job.cancelAndJoin()
}

Throwing an exception using isActive

This is similar to escaping using return, but changes it up a bit and throws an exception:

runBlocking {
  val job = launch(Dispatchers.Default) {
    for (it in 0..1000) {
      if (!isActive) {
        throw CancellationException("I have been cancelled")
      }
      Thread.sleep(50)
      println("I am still going..")
    }
  }
  delay(500)
  job.cancelAndJoin()
}

Escaping a coroutine using ensureActive

ensureActive is a helper function that throws a CancellationException if the coroutine has been cancelled:

runBlocking {
  val job = launch(Dispatchers.Default) {
    for (it in 0..1000) {
      ensureActive()
      Thread.sleep(50)
      println("I am still going..")
    }
  }
  delay(500)
  job.cancelAndJoin()
}

Calling ensureActive removes the need to call the earlier method that throws an exception. Furthermore, it has access to the exception passed into Job.cancel.

Summary

We have looked at how to cancel a job and how to write a coroutine that can be cancelled.

A job can be cancelled by calling cancel or cancelAndJoin.

It is important to remember that a coroutine must cooperate to be cancellable. You can make a coroutine cancellable by:

Calling any suspending functions from kotlinx.coroutines
Using CoroutineScope.isActive and handling the outcome appropriately

It is highly likely that your coroutines will call at least one of the provided suspending functions. Doing so removes the need to explicitly make your coroutines cancellable. That being said, it is an important subject to be aware of when you do eventually write a coroutine that does not call any of the kotlinx.coroutines suspending functions.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Waiting for coroutines

Sun, 26 Apr 2020 00:00:00 GMT

Coroutines allow you to execute tasks asynchronously without blocking the calling thread, such as the main thread. Great, but sometimes you will need to wait for all of these tasks to finish. In this post, we will look at how to wait for a coroutine to finish using join.

Note, async/await will not be covered here as I will cover that in a later post. For now, read the Kotlin docs - Concurrent using async if that is what you are interested in.

How to get a job

No, you didn’t misread the heading, I will show you how to get a job:

runBlocking {
  val job: Job = launch {
    delay(2000)
    println("and it seems to work!")
  }
  print("This is your first coroutine, ")
}

There, you now have a job. I have solved the world’s employment problems…

Ok, seriously, a Job is returned when you start a coroutine. In fact, from what I understand anyway, the words job and coroutine are interchangeable in this context. Really, they are the same thing, as seen below:

@InternalCoroutinesApi
public abstract class AbstractCoroutine<in T>(
    /**
     * The context of the parent coroutine.
     */
    @JvmField
    protected val parentContext: CoroutineContext,
    active: Boolean = true
) : JobSupport(active), Job, Continuation<T>, CoroutineScope

AbstractCoroutine implements the Job interface. So a coroutine is a job.

What to do once you have a job

You can control a coroutine through the functions available on the Job interface. Here are some of the functions (there are many more):

start
join
cancel

Further Job operations will be explored in future posts.

Waiting for a coroutine

To wait for a coroutine to finish, you can call Job.join.

join is a suspending function. Meaning that the coroutine calling it will be suspended until it is told to resume. At the point of suspension, the executing thread is released to any other available coroutines (that are sharing that thread or thread pool).

Below is a short example:

runBlocking {
  val job: Job = launch(context = Dispatchers.Default) {
    println("[${Thread.currentThread().name}] Launched coroutine")
    delay(100)
    println("[${Thread.currentThread().name}] Finished coroutine")
  }
  println("[${Thread.currentThread().name}] Created coroutine")
  job.join()
  println("[${Thread.currentThread().name}] Finished coroutine")
}

In the snippet above, a coroutine is launched, and a Job is returned. join is then called on the created job/coroutine to wait for it to finish before resuming. This leads to the following output:

[main] Created coroutine
[DefaultDispatcher-worker-1] Launched coroutine
[DefaultDispatcher-worker-1] Finished coroutine
[main] Finished coroutine

The main thread is blocked while it waits for the job/coroutine to finish. Note, that the main thread is used by runBlocking, while the child is launched using one of the default thread pools.

Just like futures and threads, many coroutines can be created and waited for through the use of join. This is also made slightly easier by the convenient joinAll extension function:

runBlocking {
  val jobs: List<Job> = (1..5).map {
    launch(context = Dispatchers.Default) {
      println("[${Thread.currentThread().name}] Launched coroutine: $it")
      delay(100)
      println("[${Thread.currentThread().name}] Finished coroutine: $it")
    }
  }
  println("[${Thread.currentThread().name}] Created all coroutines")
  jobs.joinAll()
  println("[${Thread.currentThread().name}] Finished all coroutines")
}

Something similar to the following would be output:

[DefaultDispatcher-worker-1] Launched coroutine: 1
[DefaultDispatcher-worker-2] Launched coroutine: 2
[DefaultDispatcher-worker-3] Launched coroutine: 3
[DefaultDispatcher-worker-4] Launched coroutine: 4
[main] Created all coroutines
[DefaultDispatcher-worker-5] Launched coroutine: 5
[DefaultDispatcher-worker-1] Finished coroutine: 1
[DefaultDispatcher-worker-4] Finished coroutine: 4
[DefaultDispatcher-worker-2] Finished coroutine: 3
[DefaultDispatcher-worker-3] Finished coroutine: 5
[DefaultDispatcher-worker-5] Finished coroutine: 2
[main] Finished all coroutines

I wonder if you can guess what joinAll’s implementation looks like. I don’t have any way to hide text, so I’ll just assume you actually guessed. I have added it below:

public suspend fun Collection<Job>.joinAll(): Unit = forEach { it.join() }

Joining is not always required

Joining all jobs running inside a coroutine is not a requirement to ensure their completion is waited for. By using the coroutineScope builder (a function), you can create a parent/child relationship between coroutines. More precisely, the coroutineScope builder will only progress once all of the coroutines inside it have completed. This is effectively doing a joinAll on all the jobs inside the coroutineScope.

The example below attempts to show this:

runBlocking {
  val jobs: List<Job> = (1..2).map { parentNumber ->
    // This coroutine is joined on inside [runBlocking] to allow the last [println]
    launch(context = Dispatchers.Default) {
      // The [coroutineScope] block cannot be left until the 2 corountines launched inside have finished
      coroutineScope {
        println("[${Thread.currentThread().name}] Launched parent: $parentNumber")
        (1..2).map { childNumber ->
          launch {
            println("[${Thread.currentThread().name}] Launched child: $parentNumber - $childNumber")
            delay(100)
            println("[${Thread.currentThread().name}] Finished child: $parentNumber - $childNumber")
          }
        }
      }
      println("[${Thread.currentThread().name}] Finished parent: $parentNumber")
    }
  }
  println("[${Thread.currentThread().name}] Created all coroutines")
  jobs.joinAll()
  println("[${Thread.currentThread().name}] Finished all coroutines")
}

Which outputs something like this:

[DefaultDispatcher-worker-1] Launched parent: 1
[main] Created all coroutines
[DefaultDispatcher-worker-2] Launched parent: 2
[DefaultDispatcher-worker-3] Launched child: 2 - 1
[DefaultDispatcher-worker-4] Launched child: 1 - 1
[DefaultDispatcher-worker-5] Launched child: 1 - 2
[DefaultDispatcher-worker-3] Launched child: 2 - 2
[DefaultDispatcher-worker-2] Finished child: 2 - 1
[DefaultDispatcher-worker-3] Finished child: 2 - 2
[DefaultDispatcher-worker-5] Finished child: 1 - 2
[DefaultDispatcher-worker-1] Finished child: 1 - 1
[DefaultDispatcher-worker-2] Finished parent: 2
[DefaultDispatcher-worker-1] Finished parent: 1
[main] Finished all coroutines

Each parent coroutine had to wait until their children finished. This was enabled by using coroutineScope, ensuring each coroutine launched inside it had completed before moving on.

I only touched on this subject to show that joining is not the only way to wait for jobs to finish. For some more information, have a look at the Kotlin docs - Scope builder.

Summary

Sometimes you will want to wait for several coroutines to complete before moving on. This behaviour can be achieved using join and joinAll provided by the Job interface. Doing so will suspend the calling coroutine until the joined jobs have concluded. Similar behaviour can also be obtained via the coroutineScope builder, ensuring that all jobs launched within it have ended before continuing.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Launching your coroutine knowledge

Fri, 17 Apr 2020 00:00:00 GMT

Coroutines are essentially lightweight threads that allow your applications to squeeze more juice out of the machines they are running on, without needing to manually manage the threads executing them. Coroutines add a new abstraction, further removing you (the developer) away from the lifecycle of threads. This abstraction allows a single thread to context switch between various tasks and therefore making better use of the thread’s time. It spends less time sitting around awaiting a result (from a potentially slow I/O task) and can instead continue to be useful by working on a task issued by another coroutine.

Let’s try thinking of a real-life example. Think of your day as a developer. You build your code, and it takes, let’s say, 3 mins to do so. During that time, what do you do? Well, yes, of course, some times you might just sit there and literally do nothing. But most of the time, you’ll check your emails, Slack, Twitter or go for a stretch. It doesn’t really matter what you’re doing, it’s the fact that you are doing something else. You didn’t just sit there and wait for the build to finish. You saturated your own processing and did something else while you waited. That is the concept that coroutines provide.

Yes, I did just compare you to a thread. You spawn, you live, and you die. Some of you are even killed… Ok, that metaphor got a bit dark.

One last comment before we dive into a small example. The concept of lightweight threads that coroutines introduce to Kotlin is not new. Many other languages include this ability, for example.

Erlang
Go has goroutines
Java is working on Project Loom

Quick setup

Coroutines are not included in Kotlin’s standard library (although some keywords are baked in). You will need to add a dependency on kotlinx-coroutines-core to access coroutines.

Gradle:

dependencies {
    compile group: 'org.jetbrains.kotlinx', name: 'kotlinx-coroutines-core', version: '1.3.5'
}

Maven:

<dependencies>
  <dependency>
      <groupId>org.jetbrains.kotlinx</groupId>
      <artifactId>kotlinx-coroutines-core</artifactId>
      <version>1.3.5</version>
  </dependency>
</dependencies>

The version will change as the library is updated (yes, I have to say this…)

A simple coroutine

As this is an introduction to coroutines, we will only go over an example with limited scope. I want to point out that a lot of the benefits of coroutines will not be highlighted by these examples. Moving forward, I will cover these topics which will show you more interesting use-cases.

Let’s jump right in:

fun main() {
  runBlocking {
    launch {
      delay(2000)
      println("and it seems to work!")
    }
    print("This is your first coroutine, ")
  }
}

In this example, we have some code that launches a coroutine with a delay before continuing to print out a message. Even in this short example, there are a few things we can extract from it. Let’s start from the top and work our way down.

runBlocking - Runs a new coroutine that blocks the current thread until everything inside of it (including other coroutines) has completed. This is only here to make the example work. Realistically, you should only use this to move from blocking to non-blocking code, such as in main (like the example) or in tests.
launch - Launches a new coroutine. Unlike runBlocking, launch does not block the current thread. Furthermore, it inherits the CoroutineContext from the current CoroutineScope, these magic words will be expanded on later.
delay - Suspends the coroutine it is executed in for a given amount of time. In one way it is similar to Thread.sleep in that it stops the coroutine from continuing for a while. But, it does a lot more than that. The thread that the coroutine is running on does not get blocked (like a Thread.sleep would). Instead, the coroutine suspends. Releasing its thread to allow another coroutine to continue executing. Once the delay period is up, and a thread is available, the coroutine will continue where it left off.

All functions shown above are part of kotlinx-coroutines-core

Quite a few words and concepts were thrown just thrown at you. I would be a monster to not go through them.

Suspending a coroutine - Conceptually, a coroutine suspending is like the passing of a baton in a never-ending relay race, allowing the next runner to progress. As you move into a multi-threaded environment though, this example becomes a bit weird as everyone is passing the baton to semi-random runners, but let’s ignore that. Coroutines share threads (for example, from a thread pool). Suspending a coroutine frees up the thread it was executing on to allow another to take over. This is the main benefit of coroutines. Allowing the number of running coroutines to significantly surpass the number of threads available to the application.
CoroutineContext - The persistent context of a coroutine, that decides what threads the asynchronous code found inside of it executes on.
CoroutineScope - Consists of a single CoroutineContext property that is provided to any new (child) coroutines that are launched inside it. Most functions that create new coroutines are extension functions of CoroutineScope, therefore requiring a scope to before anything useful can be done. In the example above, runBlocking is providing its CoroutineScope to the following call to launch.

This blog post by Roman Elizarov (Team Lead for Kotlin libraries) goes into the differences between CoroutineContext and CoroutineScope

Suspending functions

The topic of suspending a coroutine was touched on above, in this section, we will cover the suspend keyword.

You have already seen one suspending function from the example above, delay. Below are the contents of delay:

public suspend fun delay(timeMillis: Long) {
  if (timeMillis <= 0) return // don't delay
  return suspendCancellableCoroutine sc@ { cont: CancellableContinuation<Unit> ->
    cont.context.delay.scheduleResumeAfterDelay(timeMillis, cont)
  }
}

The function is short enough that I’ve included all of its contents, but don’t worry about understanding it all. We are focusing on the suspend keyword. Notice that I said keyword. suspend is built into the Kotlin language itself.

So, why do we need it in the first place?

All functions that suspend at some point during their execution must be annotated with suspend. I mean duh, right? Trying to go deeper than this to fully understand when a function suspends can quickly become confusing. Quite frankly, I haven’t use coroutines enough yet to give you a more in-depth explanation.

So for now, remember this is a getting started post of sorts and follow this rule of thumb. If you call a function prefaced with suspend, and the calling point is not inside a coroutine, then your function should also be marked with suspend.

For example:

Calling point outside a coroutine:

suspend fun printAfterDelay() {
  delay(2000)
  println("and it seems to work!")
}

Calling point inside a coroutine:

fun CoroutineScope.printAfterDelay() {
  launch {
    delay(2000)
    println("and it seems to work!")
  }
}

Both of these are a slight refactoring of the original example.

The compiler will help you in a lot of situations (a benefit of being a keyword). For example, trying to compile:

fun printAfterDelay() {
  delay(2000)
  println("and it seems to work!")
}

Note that the suspend is missing

Will lead to the following error:

Suspend function 'delay' should be called only from a coroutine or another suspend function

Which pretty much reiterates what I said previously.

I believe that should be enough of an explanation to get you started.

Summary

Kotlin’s coroutines are lightweight threads that share and release their underlying resources whenever a single coroutine reaches a suspension point. This allows better utilisation of an application’s resources, as threads are not necessarily blocked when performing long-running tasks. We looked at a rudimentary example consisting of a single coroutine and a suspending function. Finally, we took a modest look at the suspend keyword and how you add it to functions that suspend.

The real benefits of coroutines have not been touched on through the examples in this post. Instead, it has focused on laying the groundwork as you begin building your coroutine knowledge. Moving forward, I aim to publish more content on Kotlin coroutines. So look out for those, assuming I keep my word…

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Java friendly Kotlin - static functions

Wed, 18 Mar 2020 00:00:00 GMT

In this post, we will look at writing static functions in Kotlin that treat Java callers as first-class citizens.

Recap

There are three ways to define static functions in Kotlin:

Directly in a Kotlin file:

fun execute() {
  println("Executing from inside a kotlin file")
}

Called by:

execute()

Inside a Companion object:

class StaticFunctions {
  companion object {
    fun execute() {
      println("Executing from inside a companion object")
    }
  }
}

Called by:

StaticFunctions.execute()
StaticFunctions.Companion.execute() // If you really wanted to...

Inside a static object:

object StaticFunctions {
  fun execute() {
    println("Executing from inside a static object")
  }
}

Called by:

StaticFunctions.execute()

Non-friendly static functions

The functions defined above are not friendly. This is how you would call each of them from Java:

Directly in a Kotlin file:
```
StaticFunctionsKt.execute();
```
Inside a Companion object:
```
StaticFunctions.Companion.execute();
```
Inside a static object:
```
StaticFunctions.INSTANCE.execute();
```

They all look pretty bad. I would use more vulgar words if I wasn’t writing…

StaticFunctionsKt.execute is not too bad, it is relatively close to a standard Java static function. Except, that it’s clear that it came from a Kotlin file/library due to the Kt stuck onto the end.
StaticFunctions.Companion.execute is much worse as you have to reference the Companion object directly to access the function. Furthermore, mentioning the Companion object doesn’t bring any extra clarity to the code, so even more reason to try and remove it.
StaticFunctions.INSTANCE.execute has the same problem as the Companion version.

Writing Java friendly static functions

The three ways of defining static functions in Kotlin can each have their Java interoperability improved through different routes.

JvmName

The @file:JvmName annotation can be added to the top of a Kotlin file to allow you to manually specify the file’s name:

@file:JvmName("StaticFunctions")

package dev.lankydan

fun execute() {
  println("Executing from inside a kotlin file")
}

This allows a function to be called from Java using the specified file name rather than the original name. Any name can be chosen for the file, but most of the time, giving it the same name as the original file will suffice (the Kt part is dropped from the end).

This can then be called as such:

StaticFunctions.execute();

JvmStatic

The @JvmStatic annotation can be added to functions found in Companion and static objects:

Inside a Companion object:

class StaticFunctions {
  companion object {
    @JvmStatic
    fun execute() {
      println("Executing from inside a companion object")
    }
  }
}

Inside a static object:

object StaticFunctions {
  @JvmStatic
  fun execute() {
    println("Executing from inside a static object")
  }
}

@JvmStatic generates a new version of a function that is more suitable for Java callers.

You can then access the function just like any normal Java static function (both Companion and static objects can be called in the same way):

StaticFunctions.execute();

Which one should you use

This is a question I am still trying to answer myself.

I currently believe that if you are writing a purely static function, you should add the function directly to the file (not inside a Companion or static object). The inclusion of the class clutters things up.

On the other hand, you should use a Companion or static object if you need to fulfil the following criteria:

A class/object must be directly referenced (such as passing an instance into another function)
Provide static functions

Summary

By using the @file:JvmName and @JvmStatic annotations, you can write static Kotlin functions that can be used from Java in a nice and friendly way. It doesn’t matter how you define your static functions, as you have methods to make them all play better with Java.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Defined by failure - How failure fueled my improvement

Thu, 27 Feb 2020 00:00:00 GMT

I am going to go with a more personal blog post today. This is something that I have wanted to write for several years now. Better late than never is what people always say, right? I’ll keep it short since there isn’t too much to talk about anyway.

In this post, I am going to talk about the period of failure that defined my career.

I really struggled in my first role as a Software Developer after finishing university. No matter how hard I tried, I could not make sense of the problems I was facing. I managed to get a bit better over time, at least in my own opinion anyway.

Alas, after some feedback and modest strides in improvement. I was let go towards the end of my probation period.

This really broke me. I mean, I managed to hold it together until I had to leave later that day. But it didn’t take that long for it to really hit me.

I had failed. As I finally took steps into the real world, this was truly the first time I felt real failure.

But life goes on. I needed to get back on the horse, as they say. After some time, I found a new position.

There was a stark difference between how I felt after being let go, and now, at the beginning of a new journey. But, in the back of my mind, it was still there. The feeling of not meeting the bar, not having fixed my flaws and the need to prove that I was in fact, good enough.

I let these thoughts drive me. I made the advice and criticism I had received before focus me. And, over time, I improved. It became clear to me that I did indeed have flaws and really needed this kick up the butt to force me to focus on self-improvement.

Eventually, the end of my probation period came up. This time around, I passed. After spending the last 6 months working to improve myself, it was great to have confirmation that I had done so.

Moving the clock forward to when I started another new position. I didn’t come in with all the same worries as I did previously. They all left me, except for one. The fear of not being good enough and the need to prove myself. This drove me to continue to improve and do my best.

When I started yet another role sometime in the future, it was exactly the same. I had to prove that I belonged alongside these other great engineers. Actually, since I have now caught up to the present, I still make sure that I’m proving myself to my colleagues.

All of this story is a bit of drivel that I wanted to get out of my system. I wanted to write down the thoughts that I feel have driven me for a considerable time. If I did manage to wind some coherent point into my writing, then you will already have a good idea of what this conclusion will look like. I have been defined by my failures. One failure was so significant to me that it fueled my desire to be good enough. In fact, to some degree, I think it still does fuel me. I do not think this little voice in the back of my head is good for me. Nor, do I believe that it is a cancer to remove. I allow it to push me past the line when I don’t think I can cross it. But, I have forced it to evolve. To better serve what I need now. I no longer need to prove that I am just good enough. I now want to show what I can do.

To you, yes you, the person reading this post. Please take away one thing, from this somewhat self-absorbed biography I have written. Let your failures power you instead of weakening your resolve. Let them provide you with valuable lessons. Let them drive you to success in the future. Everyone suffers from failures and mistakes, and there is no way to abolish the awful feelings they bring. How you respond to these situations will play an influential role in how you grow throughout your career and, more generally, in life.

Calling Java Functional Interfaces from Kotlin

Mon, 10 Feb 2020 00:00:00 GMT

Basics

Below is a Functional Interface defined in Java:

// Standard Java interface
public interface Executor {
    void execute(Runnable command);
}

Note, that an interface does not need to be annotated with @FunctionalInterface to be treated as one.

In Kotlin, this can be implemented with the following code:

executor.execute { println("I am a runnable") }

As the Executor interface has only a single function, with a single input parameter, it can assign a type to the lambda passed to execute. This removes the need to define it explicitly as Runnable. This is known as SAM (Single Abstract Method) conversion, see the Kotlin docs for more information.

A more verbose way to achieve the same goal looks like:

executor.execute(Runnable { println("I am a runnable") })

If you are using Intellij, it will kindly guide you to the first solution.

Single-type generics

Let’s make it a bit more exciting and include some generics this time round. Taking the Java interface and a function to call it:

@FunctionalInterface
public interface MyJavaInterfaceWithGenerics<A> {
  A execute();
}

public static <A> A doStuff(MyJavaInterfaceWithGenerics<A> myJavaInterface) {
  return myJavaInterface.execute();
}

The doStuff function can be called with the following Kotlin code:

// Least simplified
doStuff(MyJavaInterfaceWithGenerics<String> { "hi" })
// A bit more simplified
doStuff(MyJavaInterfaceWithGenerics { "hi" })
// Specify generic type explicitly
doStuff<String> { "hi" }
// Let Kotlin do the work
doStuff { "hi" }

The return type of the most simplified code is determined by the lambda’s result, which in this case is a String.

Multi-type generics

What about when the generics get a bit more complicated? I have spiced up the example code a bit to demonstrate this:

@FunctionalInterface
public interface MyJavaInterfaceWithGenerics<A, B, C> {
  C execute(A a, B b);
}

public static <A, B, C> C doStuff(A a, B b,  MyJavaInterfaceWithGenerics<A, B, C> myJavaInterface) {
  return myJavaInterface.execute(a, b);
}

I actually had a bit of trouble thinking of an example to demonstrate both input and output generics. I believe this code is pretty ugly and is unlikely to represent genuine code. Still, it should be good enough for an example.

To call this you would use the code below:

// Least simplified
doStuff(1, 2L, MyJavaInterfaceWithGenerics<Int, Long, String> { a, b -> "hi" })
// Determine types from lambda instead of defining on the interface
doStuff(1, 2L, MyJavaInterfaceWithGenerics { a: Int, b: Long -> "hi" })
// Simplified some more
doStuff(1, 2L, { a, b -> "hi" })
// Fully simplified (extract lambda out of brackets)
doStuff(1, 2) { a, b -> "hi" }
// Determine type information from lambda (not all types had to be provided here)
// Removing the [Long] type will cause the compiler to choose an [Int] instead
doStuff(1, 2) { a, b: Long -> "hi" }

As I mentioned a second ago, realistically, the generic input types A and B in this example would be provided from an external source. Furthermore, the generic types would be specified on an overarching class or be locked down to specific types from the beginning.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Serializable Java Lambdas

Fri, 24 Jan 2020 00:00:00 GMT

Recently I was presented with the following error when serializing a lambda with Kryo:

com.esotericsoftware.kryo.KryoException: 
  java.lang.IllegalArgumentException: 
    Unable to serialize Java Lambda expression, unless explicitly declared e.g., 
    Runnable r = (Runnable & Serializable) () -> System.out.println("Hello world!");

If you do not recognise the (Runnable & Serializable) syntax, don’t worry, it is merely stating that the lambda must implement two types. This is called Type Intersection. Personally, I have never needed to use this myself, so have never really thought about it. Serializable is a bit of a unique interface in this regards, as there is nothing you actually need to implement.

Without making this cast, the lambda will be considered unserializable, which does not make Kryo happy.

As someone who doesn’t look at bytecode very often, I find it amazing how big the difference is when adding and extra casting of & Serializable. The examples below demonstrate this. For clarity, I used the following command to generate bytecode from the code snippets:

javap -c -p target.classes.dev.lankydan.IntersectionCasting

Before doing any casting:

public class IntersectionCasting {

  public static void main(String[] args) {
    Function<String, String> function = (message) -> "Kryo please serialize this message '" + message + "'";
  }
}

The generated bytecode is:

public class dev.lankydan.IntersectionCasting {
  public dev.lankydan.IntersectionCasting();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String[]);
    Code:
       0: invokedynamic #2,  0              // InvokeDynamic #0:apply:()Ljava/util/function/Function;
       5: astore_1
       6: return

  private static java.lang.String lambda$main$0(java.lang.String);
    Code:
       0: new           #3                  // class java/lang/StringBuilder
       3: dup
       4: invokespecial #4                  // Method java/lang/StringBuilder."<init>":()V
       7: ldc           #5                  // String Kryo please serialize this message '
       9: invokevirtual #6                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      12: aload_0
      13: invokevirtual #6                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      16: ldc           #7                  // String '
      18: invokevirtual #6                  // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      21: invokevirtual #8                  // Method java/lang/StringBuilder.toString:()Ljava/lang/String;
      24: areturn
}

After casting:

public class IntersectionCasting {

  public static void main(String[] args) {
    Function<String, String> function =
        (Function<String, String> & Serializable) (message) -> "Kryo please serialize this message '" + message + "'";
  }
}

The bytecode becomes:

public class dev.lankydan.IntersectionCasting {
  public dev.lankydan.IntersectionCasting();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String[]);
    Code:
       0: invokedynamic #2,  0              // InvokeDynamic #0:apply:()Ljava/util/function/Function;
       5: checkcast     #3                  // class java/io/Serializable
       8: checkcast     #4                  // class java/util/function/Function
      11: astore_1
      12: return

  private static java.lang.Object $deserializeLambda$(java.lang.invoke.SerializedLambda);
    Code:
       0: aload_0
       1: invokevirtual #5                  // Method java/lang/invoke/SerializedLambda.getImplMethodName:()Ljava/lang/String;
       4: astore_1
       5: iconst_m1
       6: istore_2
       7: aload_1
       8: invokevirtual #6                  // Method java/lang/String.hashCode:()I
      11: lookupswitch  { // 1
           -1657128837: 28
               default: 39
          }
      28: aload_1
      29: ldc           #7                  // String lambda$main$2cf54983$1
      31: invokevirtual #8                  // Method java/lang/String.equals:(Ljava/lang/Object;)Z
      34: ifeq          39
      37: iconst_0
      38: istore_2
      39: iload_2
      40: lookupswitch  { // 1
                     0: 60
               default: 135
          }
      60: aload_0
      61: invokevirtual #9                  // Method java/lang/invoke/SerializedLambda.getImplMethodKind:()I
      64: bipush        6
      66: if_icmpne     135
      69: aload_0
      70: invokevirtual #10                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceClass:()Ljava/lang/String;
      73: ldc           #11                 // String java/util/function/Function
      75: invokevirtual #12                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
      78: ifeq          135
      81: aload_0
      82: invokevirtual #13                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodName:()Ljava/lang/String;
      85: ldc           #14                 // String apply
      87: invokevirtual #12                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
      90: ifeq          135
      93: aload_0
      94: invokevirtual #15                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodSignature:()Ljava/lang/String;
      97: ldc           #16                 // String (Ljava/lang/Object;)Ljava/lang/Object;
      99: invokevirtual #12                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
     102: ifeq          135
     105: aload_0
     106: invokevirtual #17                 // Method java/lang/invoke/SerializedLambda.getImplClass:()Ljava/lang/String;
     109: ldc           #18                 // String dev/lankydan/IntersectionCasting
     111: invokevirtual #12                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
     114: ifeq          135
     117: aload_0
     118: invokevirtual #19                 // Method java/lang/invoke/SerializedLambda.getImplMethodSignature:()Ljava/lang/String;
     121: ldc           #20                 // String (Ljava/lang/String;)Ljava/lang/String;
     123: invokevirtual #12                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
     126: ifeq          135
     129: invokedynamic #2,  0              // InvokeDynamic #0:apply:()Ljava/util/function/Function;
     134: areturn
     135: new           #21                 // class java/lang/IllegalArgumentException
     138: dup
     139: ldc           #22                 // String Invalid lambda deserialization
     141: invokespecial #23                 // Method java/lang/IllegalArgumentException."<init>":(Ljava/lang/String;)V
     144: athrow

  private static java.lang.String lambda$main$2cf54983$1(java.lang.String);
    Code:
       0: new           #24                 // class java/lang/StringBuilder
       3: dup
       4: invokespecial #25                 // Method java/lang/StringBuilder."<init>":()V
       7: ldc           #26                 // String Kryo please serialize this message '
       9: invokevirtual #27                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      12: aload_0
      13: invokevirtual #27                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      16: ldc           #28                 // String '
      18: invokevirtual #27                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      21: invokevirtual #29                 // Method java/lang/StringBuilder.toString:()Ljava/lang/String;
      24: areturn
}

Now, I don’t really know how to read bytecode, but even I can see that there is a lot more going on in the version with the & Serializable cast.

If you can explain what is going on there, then be my guest and give me a shout.

One thing I can read from the bytecode above, is the references to SerializedLambda which were not there before. This class is used by compilers and libraries to ensure that lambdas deserialise correctly. Making the intersection cast of Function<String, String> & Serializable changes the underlying type of the lambda, allowing a library like Kryo to properly understand how to deserialise lambdas given to it.

Adding this extra casting of & Serializable is one possible solution to allow Kryo to deserialise lambdas. An alternative route involves creating a new interface that extends both the underlying Function type that you need, along with Serializable. This is useful when putting together a library or API for others to consume. Allowing them to focus purely on implementing their code, rather than worrying about providing the correct casting to satisfy the serialisation of their lambdas.

You could use an interface like the one below:

interface SerializableLambda extends Function<String, String>, Serializable {}

This can then be used to replace the casting in the previous example:

public class IntersectionCasting {

  public static void main(String[] args) {
    SerializableLambda function = (message) -> "Kryo please serialize this message '" + message + "'";
  }


  interface SerializableLambda extends Function<String, String>, Serializable {}
}

I have added the bytecode generated by this change below:

public class dev.lankydan.IntersectionCasting {
  public dev.lankydan.IntersectionCasting();
    Code:
       0: aload_0
       1: invokespecial #1                  // Method java/lang/Object."<init>":()V
       4: return

  public static void main(java.lang.String[]);
    Code:
       // NO CASTING                        // Mention of casting is removed and reference to new interface is added
       0: invokedynamic #2,  0              // InvokeDynamic #0:apply:()Ldev/lankydan/IntersectionCasting$SerializableLambda;
       5: astore_1
       6: return

  private static java.lang.Object $deserializeLambda$(java.lang.invoke.SerializedLambda);
    Code:
       0: aload_0
       1: invokevirtual #3                  // Method java/lang/invoke/SerializedLambda.getImplMethodName:()Ljava/lang/String;
       4: astore_1
       5: iconst_m1
       6: istore_2
       7: aload_1
       8: invokevirtual #4                  // Method java/lang/String.hashCode:()I
      11: lookupswitch  { // 1
           -1657128837: 28
               default: 39
          }
      28: aload_1
      29: ldc           #5                  // String lambda$main$2cf54983$1
      31: invokevirtual #6                  // Method java/lang/String.equals:(Ljava/lang/Object;)Z
      34: ifeq          39
      37: iconst_0
      38: istore_2
      39: iload_2
      40: lookupswitch  { // 1
                     0: 60
               default: 135
          }
      60: aload_0
      61: invokevirtual #7                  // Method java/lang/invoke/SerializedLambda.getImplMethodKind:()I
      64: bipush        6
      66: if_icmpne     135
      69: aload_0
      70: invokevirtual #8                  // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceClass:()Ljava/lang/String;
      73: ldc           #9                  // String dev/lankydan/IntersectionCasting$SerializableLambda
      75: invokevirtual #10                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
      78: ifeq          135
      81: aload_0
      82: invokevirtual #11                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodName:()Ljava/lang/String;
      85: ldc           #12                 // String apply
      87: invokevirtual #10                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
      90: ifeq          135
      93: aload_0
      94: invokevirtual #13                 // Method java/lang/invoke/SerializedLambda.getFunctionalInterfaceMethodSignature:()Ljava/lang/String;
      97: ldc           #14                 // String (Ljava/lang/Object;)Ljava/lang/Object;
      99: invokevirtual #10                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
     102: ifeq          135
     105: aload_0
     106: invokevirtual #15                 // Method java/lang/invoke/SerializedLambda.getImplClass:()Ljava/lang/String;
     109: ldc           #16                 // String dev/lankydan/IntersectionCasting
     111: invokevirtual #10                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
     114: ifeq          135
     117: aload_0
     118: invokevirtual #17                 // Method java/lang/invoke/SerializedLambda.getImplMethodSignature:()Ljava/lang/String;
     121: ldc           #18                 // String (Ljava/lang/String;)Ljava/lang/String;
     123: invokevirtual #10                 // Method java/lang/Object.equals:(Ljava/lang/Object;)Z
     126: ifeq          135
     129: invokedynamic #2,  0              // InvokeDynamic #0:apply:()Ldev/lankydan/IntersectionCasting$SerializableLambda;
     134: areturn
     135: new           #19                 // class java/lang/IllegalArgumentException
     138: dup
     139: ldc           #20                 // String Invalid lambda deserialization
     141: invokespecial #21                 // Method java/lang/IllegalArgumentException."<init>":(Ljava/lang/String;)V
     144: athrow

  private static java.lang.String lambda$main$2cf54983$1(java.lang.String);
    Code:
       0: new           #22                 // class java/lang/StringBuilder
       3: dup
       4: invokespecial #23                 // Method java/lang/StringBuilder."<init>":()V
       7: ldc           #24                 // String Kryo please serialize this message '
       9: invokevirtual #25                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      12: aload_0
      13: invokevirtual #25                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      16: ldc           #26                 // String '
      18: invokevirtual #25                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
      21: invokevirtual #27                 // Method java/lang/StringBuilder.toString:()Ljava/lang/String;
      24: areturn
}

Most of it is the same, with some new references to the SerializableLambda interface and the removal of the original intersection cast.

As mentioned before, this solution is ideal for library and API authors as it allows developers to write code as usual without having to worry about casting (for example, if the library uses Kryo under the hood). Furthermore, since the interface extends Function which is a @FunctionalInterface, developers can nicely write lambdas/functions and don’t even have to mention the interface if passing it directly into another function or constructor. I personally went down this route when designing a new API for Corda. I wanted to provide the most accessible API for developers to use, while still providing an API that works (I can’t let Kryo blow up…).

In conclusion, in this post which lacks a lot of information and is littered with extended snippets of bytecode, you need to take away two things. You can make a Java lambda/function serializable through type intersection, and you can ensure that your own APIs are clean by creating a new interface that extends both your desired function type and Serializable. These are both routes that should be considered when using a serialisation library like Kryo.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Responder flow validation (part 3 - overriding)

Mon, 06 Jan 2020 00:00:00 GMT

To finish off my trilogy of blog posts on including validation inside your responder flows, I present to you the final topic. How to override a responder flow with your own custom implementation. You will need to go down this route if you are leveraging an external CorDapp that does not allow you to extend their flows. It is not ideal, as you will not be able to make use of any of the responder code that was written. But, at least it will provide you with a way to implement your requirements while still allowing your version to communicate to the paired initiating flow. To cement what I am saying, I will say it one more time, you should only override if you cannot extend.

Before I get to the good stuff and show you what you need to do, if you really do need to go down this route, you should send an angry email to whoever wrote the CorDapp you are utilising. Moreover, you can tell them to read Responder flow validation (part 2 - extension) with some passive-aggressive message to go along with it 😎.

The content of this post is really a rehashing on what I already wrote in Extending and Overriding Flows from external CorDapps, except that I will focus precisely on overriding a responder flow.

When to override

So what sort of flow will you need to override?

Below is a flow that cannot be extended:

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

As this flow is written in Kotlin, the class is final and cannot be extended. In Java, the class would need to be marked as final explicitly to prevent you from extending it. In other words, all language rules around extending classes apply here and nothing else. If you tried to write a new flow that extended the one above, you would find a few compiler errors blocking your way.

Since this flow cannot be extended, the only solution is overriding it (or not using it at all).

How to override

To override the flow, you must do the following:

Write your own flow from scratch
Include @InitiatedBy and reference the initiating flow that your implementation should respond to
Add the flowOverrides configuration to your node.conf

Once these steps are taken, the node will start routing all calls to the original responder flow to your custom implementation.

Let’s look into each point a bit further:

Write your own flow from scratch

This could look like:

@InitiatedBy(SendMessageFlow::class)
class ValidatingSendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        val message = stx.coreTransaction.outputStates.single() as MessageState
        check(message.recipient == ourIdentity) { "I think you got the wrong person" }
        check(!message.contents.containsSwearWords()) { "Mind your language" }
        check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
        check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

Note that there is no mention of the original responder flow

Include @InitiatedBy and reference the initiating flow that your implementation should respond to

This was shown in the code block above:

@InitiatedBy(SendMessageFlow::class)
class ValidatingSendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

Add the flowOverrides configuration to your node.conf

Add the following to your node.conf:

flowOverrides {
  overrides=[
    {
      initiator="dev.lankydan.tutorial.flows.SendMessageFlow"
      responder="dev.lankydan.tutorial.flows.ValidatingSendMessageResponder"
    }
  ]
}

This links SendMessnageFlow to ValidatingSendMessageResponder instead of the original SendMessageResponder

You can also add this configuration to the deployNodes build task:

node {
  name "O=PartyB,L=London,C=GB"
  p2pPort 10003
  flowOverride(
    "dev.lankydan.tutorial.flows.SendMessageFlow",
    "dev.lankydan.tutorial.flows.ValidatingSendMessageResponder"
  )
}

When all these steps have been made, you should see a similar line in your node’s log file:

[INFO ] 2020-01-05T10:07:03,093Z [main] internal.NodeFlowManager. - Registered dev.lankydan.tutorial.flows.SendMessageFlow
 to initiate dev.lankydan.tutorial.flows.ValidatingSendMessageResponder (version 1)

This shows that the flow has been successfully switched out for the overridden version.

Forgetting to make step 2 or 3 will lead to errors.

Forgetting to add the @InitiatedBy annotation will cause the node to keep using the original flow

Forgetting to include the flowOverrides configuration will lead to an error due to multiple flows with the same @InitiatedBy annotation:

[ERROR] 2020-01-05T10:07:35,276Z [main] internal.NodeStartupLogging. - Exception during node startup: Unable to determine which flow to use when responding to:
dev.lankydan.tutorial.flows.SendMessageFlow. [dev.lankydan.tutorial.flows.SendMessageResponder, dev.lankydan.tutorial.flows.ValidatingSendMessageResponder] are all
registered with equal weight.

Conclusion

That brings us to the end of this trilogy of blog posts on responder flow validation.

Starting with why you should include validation in Responder flow validation
With the sequel detailing how to write flows that can be extended in Responder flow validation (part 2 - extension)
Finally ending with this post on overriding responder flows

You should now have a good understanding of why you should include validation in your responder flows, how to write extendable flows for other developers to leverage and how to override a flow if can’t be extended.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Responder flow validation (part 2 - extension)

Tue, 10 Dec 2019 00:00:00 GMT

In Responder flow validation I closed the post with the following:

How can I enforce the rules of my business when the logic inside a CorDapp is shared?

I will answer that question in my next post…

You can wait until then.

I’m not one to leave you hanging for too long. So, here I am, true to my word to show you how to enforce your own validation on a shared CorDapp.

Now, before I continue, I have a little disclaimer. What I will cover in this post requires cooperation from the developers of the CorDapp. Why? They are in control of what the CorDapp does. If they do not follow the necessary steps, then extending their CorDapp becomes much more difficult. It is still possible, but the chance of errors goes through the roof.

Obviously, this problem goes away if you are the one developing the CorDapp.

For the purpose of this post, I am going to assume that you are the CorDapp developer. I will show you how to write an extensible CorDapp that an external developer can leverage while adding a sprinkle of their own code. More specifically, allowing them to add extra verification to the responder flows that you create.

I will be skimming through some of the content in this post. For extra background information, I wrote Extending and Overriding Flows from external CorDapps earlier on this year. That post dives deeper down in regards to implementation compared to this post. Here we will only focus on custom validation.

Why extendable flows make sense for injecting validation

In Corda, organisations interact with each other through shared code, this means that each and every organisation executes the exact same code. The false assumption this model makes is that all businesses have identical processes. Many might be the same (there is a lot of duplication in this world), but the chance of them being exact copies is minuscule. To be honest, I would go as far as saying it’s impossible. Somewhere in the process, there will be a difference.

CorDapps need to be able to reflect this. Corda does its best to amalgamate processes amongst organisations. But, there is an appreciation that local customisation is necessary before companies can truly make the switch to distributed applications.

A vital example of the need for customisation is applying business rules to a shared application. Contracts and flows can provide a base level of validation. However, unique business rules still need to be applied before each individual organisation’s requirements are met.

As described in Extending and Overriding Flows from external CorDapps, a flow can be extended to apply additional logic defined inside of another CorDapp. Through the use of flow extension, you can design an extensible flow that applications can leverage to inject customised validation.

What a CorDapp developer must do

There are only a hand full of steps required to create an extensible flow:

The flow must be open (Kotlin) or not final (Java)
Provide a function(s) that can be overridden
The overridable function(s) must be called in relevant places

The following responder flow incorporates all the points above:

@InitiatedBy(SendMessageFlow::class)
// The flow is open
open class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  // An overridable function to contain validation is provided
  open fun checkTransaction(stx: SignedTransaction) {
    // To be implemented by sub type flows - otherwise do nothing
  }

  @Suspendable
  // [call] is final to prevent it from being overridden
  final override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        // The validation function is called
        this@SendMessageResponder.checkTransaction(stx)
        // Any other rules the CorDapp developer wants executed
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

I want to expand on the final step I mentioned above:

The overridable function(s) must be called in relevant places

I believe that SignTransactionFlow.checkTransaction is the best place to call an overridable function that contains custom validation. checkTransaction is triggered before signing a transaction, therefore it makes sense to add additional rules here. You could place it somewhere else, but the code becomes less efficient and harder to follow. I also discussed adding validation to checkTransaction in Responder flow validation.

An external developer now has two ways to use this flow:

Directly use the flow just like any normal flow. This means checkTransaction will execute without any code inside of it.
Leverage the flow by extending it and including their own custom verification.

Before we move on, I want to focus on a comment I included in the code:

// [call] is final to prevent it from being overridden
final override fun call(): SignedTransaction {

This might not be important. But, I personally think it is. By preventing external applications from altering the flow too much, you reduce the chance of the flow malfunctioning. For example, one unbalanced send or receive will break the flow. Making call final does not actually prevent this problem. However, by providing clear functions to implement, developers are less likely to make mistakes.

What an external developer must do

To extend and leverage an extensible flow, you must do the following:

Extend the flow
Include @InitiatedBy referencing the initiating flow
Implement/override provided functions

In other words, exactly what you would do to extend any average class in Kotlin or Java.

Below is what this could look like:

@InitiatedBy(SendMessageFlow::class)
class ValidatingSendMessageResponder(session: FlowSession) : SendMessageResponder(session) {

  override fun checkTransaction(stx: SignedTransaction) {
    val message = stx.coreTransaction.outputStates.single() as MessageState
    check(message.recipient == ourIdentity) { "I think you got the wrong person" }
    check(!message.contents.containsSwearWords()) { "Mind your language" }
    check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
    check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
  }
}

SendMessageResponder has been extended and checkTransaction has been implemented to include custom verification.

Any sessions that are initiated with the base SendMessageResponder will now be routed to ValidatingSendMessageResponder. You do not need to do anything else. Corda will handle it from here.

That’s really all there is. If the CorDapp developer has done the work in their flow to make it extendable, then external developers that leverage the flow should generally have a straight forward task when it comes to implementing their code.

Wrapping up

In this sequel to Responder flow validation, we looked into why flow extension provides an ideal solution for injecting custom validation and how to implement such a flow. If you continue to write CorDapps that are extensible, particularly regarding transaction validation, you will significantly increase both the usability and impact of your code. Businesses are far more likely to explore new solutions and migrate away from their existing linchpin/legacy applications if they are 100% confident that all of their needs can be met.

Just like the first episode in this trilogy, I will leave this post with a cliff hanger.

Now, before I continue, I have a little disclaimer. What I will cover in this post requires cooperation from the developers of the CorDapp that you wish to add your own rules to.

So, what can you do if the CorDapp developer does not play ball?

You’ll have to wait and see.

Responder flow validation

Mon, 25 Nov 2019 00:00:00 GMT

Do you want to prevent your node from accepting transactions that are not important to your business?

Do you want to prevent your node from accepting transactions that are invalid from your business’s perspective?

If you answered yes to either of these questions, then you might want to consider adding some validation to your responder flows.

For context, a responder flow is executed by a counterparty node when communicating with an initiating flow. Responder flows typically sign and record transactions sent to them. This means an organisation can end up in a situation where they did not have a large involvement in putting the contents of the transaction together, yet are expected to sign and potentially record it.

Surely, there should be some rules to prevent an organisation from being sent any old rubbish? And, yes, there are. The very bare minimum that needs to pass is contract validation. This ensures that a transaction is logically valid at a general level. What it does not do, is enforce local identity/roles or business level requirements.

To an extent, this is fine. A contract should only be ensuring that a transaction is logically valid.

Flows on the other hand, need to take charge of enforcing business level requirements on transactions.

Including validation

To include your own validation, you need to make use of SignTransactionFlow.checkTransaction when calling the SignTransactionFlow subflow. I guarantee that you have seen checkTransaction before (if you have worked with Corda already). If you have not, then please tell me how.

Assuming you have seen it, I bet there is a good chance that you also left it blank like this:

val stx = subFlow(object : SignTransactionFlow(session) {
  override fun checkTransaction(stx: SignedTransaction) {
    // no validation
  }
})

That is a code snippet from one of my other blog posts. Yes… I have sinned, and I am also a hypocrite. I have failed so you can learn from my mistakes! 👩‍🏫

Instead, it should look something like:

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      // [LedgerTransaction.verify] is called before [checkTransaction]
      override fun checkTransaction(stx: SignedTransaction) {
        val message = stx.coreTransaction.outputStates.single() as MessageState
        check(message.recipient == ourIdentity) { "I think you got the wrong person" }
        check(!message.contents.containsSwearWords()) { "Mind your language" }
        check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
        check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

Obviously, most of the things being checked here are pretty random, but it does show off what I have been talking about.

The recipient is checked to ensure that the MessageState is intended for the node it is being sent to.
```
check(message.recipient == ourIdentity) { "I think you got the wrong person" }
```
This is an essential step to ensure that transactions you receive are intended for you and the role you play in the current flow, or in general, within the network. Not doing this, will allow your node to accept anything it is asked to sign and record. Even if it has absolutely nothing to do with you.
A series of other rules are also included. Each additional criteria is contractually valid but is not a desired property of the state in the context of this flow.
```
check(!message.contents.containsSwearWords()) { "Mind your language" }
check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
```
I highly doubt that checking whether a message contains swear words would be contained inside of a contract. A message is still generally valid if it includes them. But, for the written flow, maybe it is a polite flow. Where any mention of a rude word must be beeped out (just like on pre-watershed TV). The same general gist applies to the other two rules.

How to validate

I have just shown you what sort of validation to include. What I didn’t do is explain how the code shown before leads to a transaction being rejected. You might have sussed it out, but let me be thorough and go through it properly.

To reject a transaction from within checkTransaction you need to throw an exception. Different exceptions will lead to different outcomes. This is due to the code below in SignTransactionFlow:

try {
  checkTransaction(stx)
} catch (e: Exception) {
  if (e is IllegalStateException || e is IllegalArgumentException || e is AssertionError)
    throw FlowException(e)
  else
    throw e
}

To properly indicate that a transaction is invalid, you should throw one of the following exceptions:

IllegalStateException
IllegalArguementException
AssertionError

These exceptions are converted to FlowExceptions, allowing them to be transferred to the counterparty with all details intact. You can also throw your own subclass of FlowException to cause the same effect.

Any other exception will become an UnexpectedFlowEndException with all of the reasons for the failure stripped out. From a counterparty’s perspective, this is the same as a random internal error that it could receive from its peers. Missing this vital information could have a significant effect on the outcome of the flow. It can be possible to propose a new transaction based on the information contained in the error. Receiving a UnexpectedFlowEndException makes it look like something just blew up.

To finally tie up this section, I want to give some suggestions on how to perform validation. This section is primarily focused on Kotlin developers since they provide some useful functions for this area (plus it’s what I code in every day).

I tend to use the following (some Kotlin basics coming up):

require - Throws an IllegalArgumentException

require(message.recipient == ourIdentity) { "I think you got the wrong person" }

check - Throws an IllegalStateException

check(message.recipient == ourIdentity) { "I think you got the wrong person" }

Which one you throw depends on the context (and possibly personal preference). I also don’t want to explain the difference between an IllegalArgumentException and IllegalStateException (as I would struggle). I personally prefer to use check here, but that’s just my opinion.

Both of these also have alternatives for null checking, requireNotNull and checkNotNull.

Corda also provides a built-in way to do similar validation using the requireThat DSL (Domain Specific Language):

requireThat {
  // <message if criteria fails> using <statement that must be true>
  "I think you got the wrong person" using (message.recipient == ourIdentity)
  // Non `using` lines can also be included
  val contents = message.contents
  "Mind your language" using (!contents.containsSwearWords())
  "Only serious messages are accepted" using (!contents.containsMemes())
  "Spam message detected" using (message.sender.name.organisation != "Nigerian Prince")
}

requireThat was built for use inside contracts but can be used anywhere. As the name suggests, it follows the same semantics as require. The difference is that it is a DSL, which makes use of using to define a message and criteria. Note, that you can add general code into the requireThat block. Only the using lines are going to (potentially) throw exceptions.

requireThat can also be used from Java (but looks a tad worse):

requireThat(require -> {
    require.using("I think you got the wrong person", message.getRecipient() == getOurIdentity());
    // Non `using` lines can also be included
    String contents = message.getContents()
    require.using("Mind your language", !contents.containsSwearWords());
    require.using("Only serious messages are accepted", !contents.containsMemes());
    require.using("Spam message detected", message.getSender().getName().getOrganisation() != "Nigerian Prince");
    return null;
});

It’s a bit more of a pain to use compared to Kotlin, but you might prefer to use it instead of other Java validation techniques.

Suggestions on what to validate

The following ideas are things that I think would be good to include. If I was writing a production-ready CorDapp, I might have some additional suggestions. For now, these are areas I think are important (some of these I touched on earlier):

Identity
- Is the transaction important to my business? Check that you are actually involved in this transaction in a meaningful way. For some scenarios, an organisation might be observing all transactions. Validation should change to accommodate this.
- What is my role in this transaction? You should validate that your business is participating in the transaction with the role that you expect. For example, a buyer or seller. Based on this, follow on validation should be included. Identity cannot be checked from within a contract (no access to ourIdentity), so it must be done inside of a flow.
Existing states
- Is this unique field actually unique compared to the states in my vault? It might be unique for the party proposing the transaction, it might not be for any of its peers though. The contents of a node’s vault cannot be accessed from within a contract. Therefore all validation that includes a vault query must be done from within a flow.
Business logic
- Does that state have an acceptable value for my business? You should validate that the states you are receiving meet whatever criteria your business has. Contracts should not enforce business rules. This is why flows should perform business validation.

Closing thoughts

I wrote this post to ensure that CorDapp developers add thorough validation to their applications. Ensuring that what they intend their CorDapps to do, is what they actually do. If there is even just a single hole in the hull of a ship, it is not a matter of whether it will sink, but when. Your applications should be put together with thorough consideration to reduce the chance (like a ship) that it sinks. Failures will still happen, but the chances of them happening and the repercussions that they cause will be significantly lower if your application is well defined and constrained. The validation inside of flows plays an essential part in fortifying the hull of your CorDapp.

While reading this, if you were particularly eagle-eyed and critical on what I wrote, you might have formulated the following question.

How can I enforce the rules of my business when the logic inside a CorDapp is shared?

I will answer that question in my next post…

You can wait until then.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Common CorDapp mistakes and how to fix them

Sat, 23 Nov 2019 00:00:00 GMT

This is the talk I gave at CordaCon in October.

Title: A tale of Corda Slack adventures - Common CorDapp mistakes and how to fix them

The talk covers the following topics:

Remembering the @Suspendable annotation
Contract validation
Responder flow validation
Balancing sends and receives in your flows

The laptop that ran the presentation didn’t support the emojis I added, so sorry for the random symbols you’ll see in the talk.

This video can also be found on the R3 website.

All other CordaCon videos can also be found here.

Sometimes, it's better to not do your work

Wed, 20 Nov 2019 00:00:00 GMT

It’s 11 o’clock and I am finally doing my own work…

I typically say this line to my colleague almost every day. I used to mean it (albeit satirically), as it could be draining coming into the office and instantly being dragged off to assist others. Now though, after saying that line a lot of times. I realised (after it was pointed out) that the impact I make while helping my team is probably higher than what I would achieve plugging away at my own work.

These are some of the things I get up to instead of focusing on my own work:

Answering slack questions (Open Source project)
Answering any other sort of questions
Debugging code with colleagues
Discussions about features (not necessarily ones close to me)
Reviewing blog posts

While I am sure there are more to list, my mind has gone completely blank now I am trying to write some down. If you were to put your own list together, I imagine a number of those would make the cut. It’s also highly likely there are things you do that I haven’t put here.

So, why am I even writing this? Partially for myself, to note down my stream of consciousness as I think about this subject. Primarily, for you, to feel more validated and correct in your decisions to assist your colleagues or anyone else you choose to support.

As I mentioned at the beginning of this post, you can have a more significant impact by assisting someone instead of focusing solely on your own work. Obviously, this is not a rule, and there will be times where you can have a large impact by going solo. But, in my opinion, those times are far fewer.

To see what impact can be made, let me expand on areas I tend to help others with:

Answering slack questions (Open Source project) - Very often, some questions are asked that I can answer with the snap of my fingers. But, that short time spent can be magnitudes smaller than what the asker might need to spend to figure it out themselves. They might have already spent hours trying to find the missing piece. Pointing them in the correct direction could unlock them to continue working productively. This is not just good for them. When participating in an Open Source project, it is also good for the project itself. As there is now one more person who is closer to successfully building a product on top of the project.
Answering any other sort of questions - Similar to the point above, but focusing on more internal questions from my own colleagues. The impact is the same, but the questions might be more precise. I might know the answer, and I am saving a lot of time by giving it to them or suggesting some things to look at.
Debugging code with colleagues - Being a living rubber duck, I can provide slightly more feedback than an actual rubber duck. Allowing colleagues to talk through a problem can break down the mental roadblock in their way. At the same time, it is an excellent opportunity to provide suggestions and general knowledge about the code that is being looked at. In doing so, knowledge is moved out of a single person and can continue to be shared. The colleague that received some help might now be able to help someone else further down the line. Heck, they’ll probably tell me what I told them after I have forgotten it.
Discussions about features (not necessarily ones close to me) - Again, this one is quite similar to the point above (debugging). I remember sitting with one colleague, for a few hours, with them talking at me. Every now and then I would ask a question. They might be stupid questions. I mean, I really was out of my depth and barely understood what we were discussing. But apparently, I am a really good rubber duck. Once we finished, they thanked me and claimed it was extremely helpful. So, somehow, it seemed that I made an impact without really grasping what we were talking about. Nice, I guess…
Reviewing blog posts - This is the one that is probably the least common from the list as it depends on your role or company. It involves reviewing the content and grammar of a post, along with suggesting other improvements. I find it quite a challenging task, but it is really satisfying seeing readers enjoy the writer’s post. The impact here is firstly the refined post that comes out. Secondly, by providing suggestions and indications on small mistakes that are often made, the original writer can produce better and better content. Allowing them to take the step from already creating good content, to potentially great and stimulating pieces.

Now, I realise I have spent the whole time talking about what I do…Unfortunately, it is the easiest way for me to get it down in writing. To correct this, I want you to rethink what you just read in the points above and switch all the “I”s with “you”s (or read it in first person from your perspective).

These points are things that you do.

These are areas that you impact others.

These are ways you are effectively spending your time to lift other people up.

What seems like small amounts of your time can allow others to move past the problem they are facing. More importantly, this time could have a significant impact on their output, outlook and ideas moving forward. You should realise how valuable your time is, and how sharing that time can spread out your reach far and wide.

To close this off, let me reuse the quote I opened with:

It’s 11 o’clock and I am finally doing my own work…

Go ahead and say it. But, don’t say it in annoyance. Say it as a statement highlighting the impact you make on others.

Augmenting a Spring Data repository through delegation

Sat, 14 Sep 2019 00:00:00 GMT

I have recently written several posts about Kotlin’s delegation. In doing so, I realised a useful way to apply it to Spring Data repositories. Which would allow Spring Data to continue sprinkling some magic while providing a route for customisation. The code shown in this post is in Kotlin, but is still relevant to Java.

This post uses R2DBC, but the content is generic enough to be applicable to any Spring Data module.

Reading Asynchronous RDBMS access with Spring Data R2DBC and Class delegation in Kotlin would be beneficial here if you do not have much background knowledge in these areas.

As a recap. What is the magic that Spring Data provides?

Spring Data allows you to write an interface where you are only required to define the queries that you need. It will then do all the work of creating the implementation and injecting dependencies for you. This looks something like:

@Repository
interface PersonRepository : R2dbcRepository<Person, Int> {

  @Query("SELECT * FROM people WHERE age > $1")
  fun findAllByAgeGreaterThan(age: Int): Flux<Person>
}

Since Spring Data R2DBC is being used, fully inferred queries are not yet fully supported. This is why the query is written out manually.

The downside to this is that it is creating an implementation based on an interface. Therefore, if you want to do any sort of customisation, you will need to create an instance of the interface yourself, inject in its dependencies and implement each query. For example:

class PersonRepositoryImpl(
  private val entity: RelationalEntityInformation<Person, Int>,
  private val databaseClient: DatabaseClient,
  converter: R2dbcConverter,
  private val accessStrategy: ReactiveDataAccessStrategy
) : SimpleR2dbcRepository<Person, Int>(entity, databaseClient, converter, accessStrategy),
  PersonRepository {

  override fun findAllByAgeGreaterThan(age: Int): Flux<Person> {

    val mapper: StatementMapper.TypedStatementMapper<Person> =
      accessStrategy.statementMapper.forType(entity.javaType)

    val selectSpec: StatementMapper.SelectSpec = mapper
      .createSelect(entity.tableName)
      .withProjection(accessStrategy.getAllColumns(entity.javaType))
      .withCriteria(Criteria.where("age").greaterThan(age))

    val operation: PreparedOperation<*> = mapper.getMappedObject(selectSpec)

    return databaseClient.execute().sql(operation).`as`(entity.javaType).fetch().all()
  }
}

Yes, that query code is probably terrible, and I am sure you could do better. You get my point though.

The pain of creating this class can be removed by delegating to the repository Spring implemented based on your interface. You can then add all the customisations you need.

In Kotlin, this would look like:

@Repository
class DelegatingPersonRepository(private val delegate: PersonRepository) :
  PersonRepository by delegate {

  override fun <S : Person> save(objectToSave: S): Mono<S> {
    // override `save` implementation
  }

  // any other overrides (kotlin provides delegated implementations)
}

In Java, it is a bit more cumbersome but still easily achievable:

@Repository
public class DelegatingPersonRepository implements PersonRepository {

  private final PersonRepository delegate;

  public DelegatingPersonRepository(PersonRepository delegate) {
    this.delegate = delegate;
  }

  @Override
  public Flux<Person> findAllByAgeGreaterThan(int age) {
    return delegate.findAllByAgeGreaterThan(age);
  }

  @Override
  public <S extends Person> Mono<S> save(S entity) {
    // override `save` implementation
  }

  // all other implementations of `PersonRepository` functions
}

In both versions, DelegatingPersonRepository calls the implementation of findAllByAgeGreaterThan defined in PersonRepository. So far, no effort has been directly spent on writing a function to query a database.

When the DelegatingPersonRepository is used, all function calls that are not overridden will delegate to the implementation of PersonRepository that Spring created.

For someone like me, who doesn’t really like putting together SQL queries and writing all the conversion code. Using delegation in this way really allows you to leverage the benefits of Spring Data while still giving you room to customise the outcome. The amount of code you save might not actually be that great. But, there is a considerable reduction in the effort required to put it together. Just let Spring do all the heavy lifting for you!

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Implementing multiple interfaces through delegation

Thu, 05 Sep 2019 00:00:00 GMT

In Kotlin, a class can implement multiple interfaces. This is common knowledge. A class can also use delegation to implement numerous interfaces where the implementations come from any delegated objects passed into the constructor. For example:

class GeneticExperiment(human: Human, animal: Animal) : Human by human, Animal by animal

interface Human {
  fun eat()
  fun sleep()
  fun poop()
}

interface Animal {
  fun bite()
}

This weird class inherits functionality from both the Human and Animal objects passed into the constructor. Without delegation, any functions on either of the interfaces will need to be written out manually within the class.

An important point to note is the fact that Human and Animal do not share a common parent interface.

If you do have interfaces that extend from a common interface, then there will be compilation errors due to conflicting implementations from the delegated objects. For example, the code:

class ShapeShifter(human: Human, weightLifter: WeightLifter) : Human by human, WeightLifter by weightLifter

interface Human {
  fun eat()
  fun sleep()
  fun poop()
}

interface WeightLifter : Human {
  fun liftHeavyStuff()
  fun pose()
}

Leads to the errors:

Delegation.kt:80:1: error: class 'ShapeShifter' must override public open fun eat(): 
Unit defined in dev.lankydan.ShapeShifter because it inherits many implementations of it
class ShapeShifter(human: Human, weightLifter: WeightLifter) : Human by human, WeightLifter by weightLifter
^
Delegation.kt:80:1: error: class 'ShapeShifter' must override public open fun sleep(): 
Unit defined in dev.lankydan.ShapeShifter because it inherits many implementations of it
class ShapeShifter(human: Human, weightLifter: WeightLifter) : Human by human, WeightLifter by weightLifter
^
Delegation.kt:80:1: error: class 'ShapeShifter' must override public open fun poop(): 
Unit defined in dev.lankydan.ShapeShifter because it inherits many implementations of it
class ShapeShifter(human: Human, weightLifter: WeightLifter) : Human by human, WeightLifter by weightLifter
^

As you can see in the errors, the ShapeShifter class does not know which delegated implementation to take. The compilation errors must be resolved by explicitly overriding the conflicting functions. By overriding the functions, you can decide which delegated implementation you wish to use, or you can provide a new one altogether. A fixed version of this class could look like:

class ShapeShifter(
  private val human: Human,
  private val weightLifter: WeightLifter
) : Human by human, WeightLifter by weightLifter {
  
  override fun eat() {
    human.eat()
  }

  override fun sleep() {
    weightLifter.sleep()
  }

  override fun poop() {
    println("💩")
  }
}

interface Human {
  fun eat()
  fun sleep()
  fun poop()
}

interface WeightLifter : Human {
  fun liftHeavyStuff()
  fun pose()
}

Only the functions declared in the Human interface need to manually overridden

Taking a slight step back from here. A class that implements a child interface and delegates its functionality to the child’s parent interface will compile as long as all of the child’s functions are implemented. Wording that was difficult. An example should help clarify what I am trying to say:

class ShapeShifter(copied: Human) : WeightLifter, Human by copied {

  override fun liftHeavyStuff() {
    println("🏋️‍♂️")
  }

  override fun pose() {
    println("💪")
  }
}

interface Human {
  fun eat()
  fun sleep()
  fun poop()
}

interface WeightLifter : Human {
  fun liftHeavyStuff()
  fun pose()
}

Only the functions declared in the WeightLifter interface need to be added manually

To summarise:

A class can implement multiple interfaces and delegate its functionality to one or more objects
A class with multiple interfaces that extend a common parent can delegate each interfaces’ implementation but must override functions defined in the parent
A class that implements a child interface can delegate to its parent and only needs to add functions defined in the child.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Streaming live updates from a reactive Spring Data repository

Thu, 29 Aug 2019 00:00:00 GMT

This post details a naive implementation of streaming updates from a database to any other components that are interested in that data. More precisely, how to alter a Spring Data R2DBC repository to emit events to relevant subscribers.

A little bit of background knowledge of R2DBC and Spring will be helpful for this post. My previous writings, Asynchronous RDBMS access with Spring Data R2DBC and Spring Data R2DBC for Microsoft SQL Server should help in that regard.

As I mentioned, this will be a naive implementation. Therefore, the code will not be anything fancy.

To do this, I hijacked a SimpleR2dbcRepository to create a repository implementation that emits an event every time a new record is persisted. New events are added to a DirectProcessor and sent to any Publishers subscribed to it. This looks like:

class PersonRepository(
  entity: RelationalEntityInformation<Person, Int>,
  databaseClient: DatabaseClient,
  converter: R2dbcConverter,
  accessStrategy: ReactiveDataAccessStrategy
) : SimpleR2dbcRepository<Person, Int>(entity, databaseClient, converter, accessStrategy) {

  private val source: DirectProcessor<Person> = DirectProcessor.create<Person>()
  val events: Flux<Person> = source

  override fun <S : Person> save(objectToSave: S): Mono<S> {
    return super.save(objectToSave).doOnNext(source::onNext)
  }
}

The only function from SimpleR2dbcRepository that needs to be overridden is save (saveAll delegates to save). doOnNext is added to the original save call, which pushes a new event to the source (the DirectorProcessor) by calling onNext.

The source is cast to a Flux to prevent classes from outside of the repository from adding new events. Technically they can still add events, but they will need to cast it themselves.

As you might have noticed, the repository is taking a load of parameters and passing them into SimpleR2dbcRepository. An instance of the repository needs to be created manually as some of its dependencies cannot be injected in automatically:

@Configuration
class RepositoryConfiguration {

  @Bean
  fun personRepository(
    databaseClient: DatabaseClient,
    dataAccessStrategy: ReactiveDataAccessStrategy
  ): PersonRepository {
    val entity: RelationalPersistentEntity<Person> = dataAccessStrategy
      .converter
      .mappingContext
      .getRequiredPersistentEntity(Person::class.java) as RelationalPersistentEntity<Person>
    val relationEntityInformation: MappingRelationalEntityInformation<Person, Int> =
      MappingRelationalEntityInformation(entity, Int::class.java)
    return PersonRepository(
      relationEntityInformation,
      databaseClient,
      dataAccessStrategy.converter,
      dataAccessStrategy
    )
  }
}

At this point, everything is set up and ready to use. Below is an example of it working:

personRepository.events
  .doOnComplete { log.info("Events flux has closed") }
  .subscribe { log.info("From events stream - $it") }
// insert people records over time
MARVEL_CHARACTERS
  .toFlux()
  .delayElements(Duration.of(1, SECONDS))
  .concatMap { personRepository.save(it) }
  .subscribe()

Which outputs:

29-08-2019 09:08:27.674 [reactor-tcp-nio-1]  From events stream - Person(id=481, name=Spiderman, age=18)
29-08-2019 09:08:28.550 [reactor-tcp-nio-2]  From events stream - Person(id=482, name=Ironman, age=48)
29-08-2019 09:08:29.555 [reactor-tcp-nio-3]  From events stream - Person(id=483, name=Thor, age=1000)
29-08-2019 09:08:30.561 [reactor-tcp-nio-4]  From events stream - Person(id=484, name=Hulk, age=49)
29-08-2019 09:08:31.568 [reactor-tcp-nio-5]  From events stream - Person(id=485, name=Antman, age=49)
29-08-2019 09:08:32.571 [reactor-tcp-nio-6]  From events stream - Person(id=486, name=Blackwidow, age=34)
29-08-2019 09:08:33.576 [reactor-tcp-nio-7]  From events stream - Person(id=487, name=Starlord, age=38)
29-08-2019 09:08:34.581 [reactor-tcp-nio-8]  From events stream - Person(id=488, name=Captain America, age=100)
29-08-2019 09:08:35.585 [reactor-tcp-nio-9]  From events stream - Person(id=489, name=Warmachine, age=50)
29-08-2019 09:08:36.589 [reactor-tcp-nio-10] From events stream - Person(id=490, name=Wasp, age=26)
29-08-2019 09:08:37.596 [reactor-tcp-nio-11] From events stream - Person(id=491, name=Winter Soldier, age=101)
29-08-2019 09:08:38.597 [reactor-tcp-nio-12] From events stream - Person(id=492, name=Black Panther, age=42)
29-08-2019 09:08:39.604 [reactor-tcp-nio-1]  From events stream - Person(id=493, name=Doctor Strange, age=42)
29-08-2019 09:08:40.609 [reactor-tcp-nio-2]  From events stream - Person(id=494, name=Gamora, age=29)
29-08-2019 09:08:41.611 [reactor-tcp-nio-3]  From events stream - Person(id=495, name=Groot, age=4)
29-08-2019 09:08:42.618 [reactor-tcp-nio-4]  From events stream - Person(id=496, name=Hawkeye, age=47)
29-08-2019 09:08:43.620 [reactor-tcp-nio-5]  From events stream - Person(id=497, name=Pepper Potts, age=44)
29-08-2019 09:08:44.627 [reactor-tcp-nio-6]  From events stream - Person(id=498, name=Captain Marvel, age=59)
29-08-2019 09:08:45.631 [reactor-tcp-nio-7]  From events stream - Person(id=499, name=Rocket Raccoon, age=30)
29-08-2019 09:08:46.637 [reactor-tcp-nio-8]  From events stream - Person(id=500, name=Drax, age=49)
29-08-2019 09:08:47.639 [reactor-tcp-nio-9]  From events stream - Person(id=501, name=Nebula, age=30)

A record is saved every second which matches up to the events coming out of the repository.

Note, that the doOnComplete event is never triggered. The source never closes and therefore never emits a completion event to any of its subscribers.

That’s all there is to it, at least for this basic implementation. I am sure there is a lot more that could be done, but I would need to figure out how to do that first… To summarise, with a few additions, you can stream data inserted into your database to components that are interested in the records being added.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

The potential traps in Kotlin's Data Classes

Sat, 17 Aug 2019 00:00:00 GMT

The aim of this post is not to point out some massive flaws in Kotlin’s design of data classes and show you how to get passed them. Actually, it is quite the opposite. The contents of this post are clearly documented in the Kotlin docs. I am merely here to highlight this information to anyone who has not noticed precisely how their data classes are working.

Data classes are convenient for us developers, especially those of us coming over from Java. They provide several generated functions allowing you to write a fully functional class with very little code. Data classes provide the following generated functions:

equals
hashCode
toString
copy
componentN() functions.

All of the above are generated for properties defined in a class’ primary constructor. Anything defined outside of this constructor is ignored. This is the potential trap. But, this is only a trap if you are not aware of how data classes work. As I mentioned earlier, it is clearly documented, you only need to be mindful of it. Which you now are, of course.

If you are not considerate in how you define your data classes, you are likely to find some bugs in your application. equals and hashCode are generally essential functions. If they do not work as expected, bugs are sure to follow.

Below is an example of this:

data class MyClass(val a: String, val b: Int) {
  // property defined outside of primary constructor
  lateinit var c: String
}

fun main() {
  // create two equal objects
  val myClass = MyClass("abc", 0)
  val myClass2 = myClass.copy()
  // check their hashCodes are the same and that they equal each other
  println("myClass hashCode: ${myClass.hashCode()}")
  println("myClass2 hashCode: ${myClass2.hashCode()}")
  println("myClass == myClass2: ${myClass == myClass2}")
  // set the lateinit variables
  myClass.c = "im a lateinit var"
  myClass2.c = "i have a different value"
  // have their hashCodes changed?
  println("myClass hashCode after setting lateinit var: ${myClass.hashCode()}")
  println("myClass2 hashCode after setting lateinit var: ${myClass2.hashCode()}")
  // are they still equal to each other?
  println("myClass == myClass2 after setting lateinit vars: ${myClass == myClass2}")
  // sanity check to make sure I'm not being stupid
  println("sanity checking myClass.c: ${myClass.c}")
  println("sanity checking myClass2.c: ${myClass2.c}")
}

Executing this example outputs:

myClass hashCode: 2986974
myClass2 hashCode: 2986974
myClass == myClass2: true
myClass hashCode after setting lateinit var: 2986974
myClass2 hashCode after setting lateinit var: 2986974
myClass == myClass2 after setting lateinit vars: true
sanity checking myClass.c: im a lateinit var
sanity checking myClass2.c: i have a different value

As you can see the hashCode of each object is the same, and they are both equal to each other, even though their c properties are different. If you tried to make use of MyClass inside of a Map or Set, the chance of entries colliding with each other increases. That being said, it really does depend on what you are trying to achieve. Maybe this is precisely what you want to happen. In which case, more power to you.

Placing c into the MyClass constructor would affect the hashCode and equals implementation. c would then be involved in any calls to hashCode, equals and the rest of the generated functions.

Another solution is to manually implement the generated functions. Rewriting the class as:

data class MyClass(val a: String, val b: Int) {
  lateinit var c: String

  override fun equals(other: Any?): Boolean {
    if (this === other) return true
    if (javaClass != other?.javaClass) return false

    other as MyClass

    if (a != other.a) return false
    if (b != other.b) return false
    if (c != other.c) return false

    return true
  }

  override fun hashCode(): Int {
    var result = a.hashCode()
    result = 31 * result + b
    result = 31 * result + c.hashCode()
    return result
  }

  override fun toString(): String {
    return "MyClass(a='$a', b=$b, c='$c')"
  }
}

These implementations were kindly provided by Intellij 👏. By specifying all the properties in each of the overridden functions, any properties not included in the primary constructor (c in this case) are now used. hashCode and equals now better represent the class and improves its use inside a Map or Set.

But, and a big but 😏. At least in in the code I have written. A bug has now been introduced. c is a lateinit var and each of the overridden functions now try to access it. If any of these functions are called before c is set you will get an exception:

Exception in thread "main" kotlin.UninitializedPropertyAccessException: lateinit property c has not been initialized
	at dev.lankydan.MyClass.hashCode(DataClasses.kt:60)
	at dev.lankydan.DataClassesKt.main(DataClasses.kt:72)
	at dev.lankydan.DataClassesKt.main(DataClasses.kt)

Rewriting equals, hashCode and toString to accommodate the lateinit var will resolve this error:

data class MyClass(val a: String, val b: Int) {
  lateinit var c: String

  override fun equals(other: Any?): Boolean {
    if (this === other) return true
    if (javaClass != other?.javaClass) return false

    other as MyClass

    if (a != other.a) return false
    if (b != other.b) return false
    if ((this::c.isInitialized && (other as MyClass)::c.isInitialized && c != other.c)
      || this::c.isInitialized != (other as MyClass)::c.isInitialized
    ) return false

    return true
  }

  override fun hashCode(): Int {
    var result = a.hashCode()
    result = 31 * result + b
    if (this::c.isInitialized) {
      result = 31 * result + c.hashCode()
    }
    return result
  }

  override fun toString(): String {
    return if (this::c.isInitialized) "MyClass(a='$a', b=$b, c='$c')"
    else "MyClass(a='$a', b=$b)"
  }
}

This implementation is safe to use, even if the lateinit var is not set.

Whether you want to do this or not depends on the requirements of your class. Using a data class like I have here inside a Map is probably not recommended. If you want to do this though, you can. Just be aware of how it all works.

If you haven’t done so already, I recommend that you take a quick look at the documentation on this subject. Highlighting this information was the goal of this post. It’s not some fancy code that does something magical. Instead, it is something more basic and fundamental to how Kotlin works. Being aware of how data classes work in this aspect can be vital to reducing the number of bugs in your application.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Connecting a Ktor web server to a Corda node

Mon, 12 Aug 2019 00:00:00 GMT

The preparation for this blog post began several weeks ago (probably over a month by now). Before I could write about melding Corda and Ktor together, I first needed to lay the groundwork and focus solely on Ktor. That is where my blog post, Ktor - a Kotlin web framework came into existence. If you haven’t used or seen Ktor before, I recommend taking a browse at that post either before or after reading this post. Reading it in advance is probably a better idea, but you are in control of your own life 🤷.

This post will focus on implementing a Ktor web server that connects to a Corda node. I am not going to talk about why you should use Ktor. That decision is up to you. What I am doing is providing you with some information and allowing you to formulate a decision by yourself (like anything you read on the internet 🙄).

Dependencies

buildscript {
  ext.ktor_version = '1.2.2'
  ext.kotlin_version_for_app = '1.3.41'

  repositories {
    mavenCentral()
    jcenter()
  }

  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version_for_app"
  }
}

apply plugin: 'java'
apply plugin: 'kotlin'

java {
  disableAutoTargetJvm()
}

dependencies {
  compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version_for_app"
  compile "$corda_release_group:corda-jackson:$corda_release_version"
  compile "$corda_release_group:corda-rpc:$corda_release_version"
  compile "$corda_release_group:corda:$corda_release_version"
  compile "io.ktor:ktor-server-netty:$ktor_version"
  compile "ch.qos.logback:logback-classic:1.2.3"
  implementation ("io.ktor:ktor-jackson:$ktor_version") {
    exclude group: 'com.fasterxml.jackson.module', module: 'jackson-module-kotlin'
  }
  compile project(":contracts")
  compile project(":workflows")
}

There are a few things to highlight here. First, the kotlin_version_for_app property. Ktor requires Kotlin 1.3 (as it uses coroutines) but, at the time of writing, Corda only supports Kotlin 1.2. Therefore, different versions need to be used for the web server code and the Corda node. Secondly, jackson-module-kotlin is excluded as it causes a runtime error due to a version mismatch.

High level look at the code

Below is a small snippet of code showing the starting point of the web server’s implementation:

fun main() {
  embeddedServer(
    Netty,
    port = System.getProperty("server.port").toInt(),
    module = Application::module
  ).start().addShutdownHook()
}

fun Application.module() {
  val connection: CordaRPCConnection = connectToNode()
  install(CallLogging) { level = Level.INFO }
  install(ContentNegotiation) { cordaJackson(connection.proxy) }
  routing { messages(connection.proxy) }
  addShutdownEvent(connection)
}

This code ties everything together as all the functionality of the server branches out from the functions above.

The contents of module will be explored in the following sections.

Connecting to the node

I bet you might have a good idea what connectToNode does. I hope you do anyway… Below are the contents of connectToNode:

fun connectToNode(
  host: String = System.getProperty("config.rpc.host"),
  rpcPort: Int = System.getProperty("config.rpc.port").toInt(),
  username: String = System.getProperty("config.rpc.username"),
  password: String = System.getProperty("config.rpc.password")
): CordaRPCConnection {
  val rpcAddress = NetworkHostAndPort(host, rpcPort)
  val rpcClient = CordaRPCClient(rpcAddress)
  return rpcClient.start(username, password)
}

If you have seen any of the Corda samples, then you will probably be familiar with this piece of code. Long story short, it connects to the node with the given connection details. I chose to generate the function’s default parameters from the application’s system properties. This implementation is not particularly important, it just connects to the node and could be written in several different ways.

A CordaRPCConnection is returned from the function. Initially, I wanted to return a CordaRPCOps as the connection itself doesn’t do too much. But, without returning the connection, there is no way to gracefully disconnect from a node. In other words, there needs to be a way to call notifyServerAndClose when the server stops. This is explored further down in the post.

Setting up Jackson

Some extra setup needs to be done to properly use Jackson with Corda:

fun ContentNegotiation.Configuration.cordaJackson(proxy: CordaRPCOps) {
  val mapper: ObjectMapper = JacksonSupport.createDefaultMapper(proxy)
  mapper.apply {
    setDefaultPrettyPrinter(DefaultPrettyPrinter().apply {
      indentArraysWith(DefaultPrettyPrinter.FixedSpaceIndenter.instance)
      indentObjectsWith(DefaultIndenter("  ", "\n"))
    })
  }
  val converter = JacksonConverter(mapper)
  register(ContentType.Application.Json, converter)
}

The Corda ObjectMapper is initialised with createDefaultMapper, allowing classes like Party or X509Certificate to be serialised or deserialised. This can be important depending on what is being returned from your own API.

The rest of the code is stolen from the ktor-jackson module. It alters the JSON output slightly to be more somewhat more desirable.

Creating the endpoints

HTTP requests are routed to these endpoints:

fun Routing.messages(proxy: CordaRPCOps) {
  route("/messages") {
    get("/") {
      call.respond(
        HttpStatusCode.OK,
        proxy.vaultQueryBy<MessageState>().states.map { it.state.data })
    }
    post("/") {
      val received = call.receive<Message>()
      try {
        val message = proxy.startFlow(
          ::SendMessageFlow,
          state(proxy, received, UUID.randomUUID())
        ).returnValue.getOrThrow().coreTransaction.outputStates.first() as MessageState
        call.respond(HttpStatusCode.Created, message)
      } catch (e: Exception) {
        call.respond(HttpStatusCode.InternalServerError, e.message ?: "Something went wrong")
      }
    }
  }
}

Logic wise, there is not much going on here. For a focused explanation of this code, I recommend reading Ktor - a Kotlin web framework as I mentioned earlier.

Gracefully disconnecting from a node

To gracefully disconnect from a node, the web server needs to call CordaRPCConnection.notifyServerAndClose. Implementing this required a bit of work that I wasn’t expecting. Below is the code that triggers notifyServerAndClose:

fun NettyApplicationEngine.addShutdownHook() {
  Runtime.getRuntime().addShutdownHook(Thread {
    stop(1, 1, TimeUnit.SECONDS)
  })
  Thread.currentThread().join()
}

fun Application.addShutdownEvent(connection: CordaRPCConnection) {
  environment.monitor.subscribe(ApplicationStopped) {
    connection.notifyServerAndClose()
  }
}

A shutdown hook is added to the server. As explained in Graceful shutdown of Ktor applications, subscribing to the ApplicationStopped event is not enough to execute code when terminating the application. The shutdown hook calls stop to gracefully close the NettyApplicationEngine that the server runs upon. Leading to the shutdown event being correctly triggered and executed.

That’s all there is

Yes, really, that is all. Implementing a super basic web server does not require much code at all. There isn’t really anything else to write. I have shown you that there is another web framework that can be used to connect to a Corda node. You don’t have to default to Spring just because the Corda samples use them. If you prefer Ktor, use Ktor. If you don’t, don’t. If you did like the look of Ktor, and if you haven’t already, I recommend looking at Ktor - a Kotlin web framework.

A lot of code was excluded from this post as I focused on the more important aspects of the implementation. If you are interested in the rest of the code, you can find it on my GitHub.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Flows can do anything

Tue, 30 Jul 2019 00:00:00 GMT

In Corda, Flows can do a lot more than proposing new transactions to record between organisations. Although, saying they can do anything is probably a bit far-reaching (it’s catchy though). What I really want to say, is that flows are the entry points into a node. Corda provides a series of functions to interact with a node via RPC. Currently, these cover the more straightforward use cases, such as querying the vault, but there is a limitation to what is provided. Flows cover any of the non-standard logic that needs to be triggered. So, if you want to expose an API from a Corda node that a client can trigger or consume, then this post is for you.

I will be exploring the use of flows as entry points to a node throughout this post. Flows that propose new transactions are shown in many other tutorials and have been excluded from this post. The limelight will be solely shone onto flows that provide different functionality.

The theory

As mentioned in the introduction, and I want to reiterate it, the only way to expose an endpoint or API from a Corda node is through a flow. The existing RPC endpoints will handle the standard functionality. In the future this might change, adding support for custom RPC endpoints. But, as of now (Corda 4), flows are the only way to run non-standard logic within a node.

In my opinion, exposing the functionality of a node through flows is very similar to writing HTTP endpoints for a web server. You must define what is accessible. Clients cannot request access to any internal code inside the web server. If there is not an endpoint, an error is sent back. The same concept is true for Corda. It is then more straightforward to reason about what a client can or cannot do when interacting with a node.

In regards to the implementation, the easiest way to get started is by showing you an example:

@StartableByRPC
class StupidSimpleQueryFlow(private val externalId: String) : FlowLogic<MessageState>() {
    // does not need to be suspendable?
    override fun call(): MessageState {
        return serviceHub.vaultService.queryBy<MessageState>(
            QueryCriteria.LinearStateQueryCriteria(
                externalId = listOf(externalId)
            )
        ).states.single().state.data
    }
}

If you know your stuff, then you will realise that you can do this via RPC. I chose this example to start you off with something familiar. You have probably done similar queries in your own flows. Although there is a good chance they were part of a more extensive process.

There are a few things to note here that differ compared to most flows:

No @InitiatingFlow annotation
No @Suspendable annotation
No matching responder flow

Why are they missing? Well, they are not needed. Think about it. Each point is related to communication with another node, which is not happening. The flow starts, does a query and returns the retrieved data. Let me expand on this:

@InitiatingFlow allows a flow to create new sessions to communicate with other nodes. Since no interactions are required, no sessions need to be created, and therefore the annotation can be removed.
@Suspendable is required on functions that suspend. Allowing a flow to create checkpoints that are loaded when the flow needs to wake up again. The most common place for a flow to suspend is during communication with another node. In this scenario, the annotation can be removed since the flow never needs to suspend.
Responder flows run on counterparty nodes and interact with your flows when you send data to them. Again, there is no communication, so there is no need for other nodes to have a flow installed that pairs with the flow above.

Most flows that do not interact with other nodes will follow this sort of structure (some flows can still suspend depending on what you are doing).

The flow defined above can then be accessed from an external client:

proxy.startFlow(::StupidSimpleQueryFlow, "asdas").returnValue.get()

Examples

Below are a few examples of flows that you could write and why you might use them.

Retrieving a value from a service

@StartableByRPC
class GetMeSomeValueFromAService : FlowLogic<Long>() {
  override fun call(): Long {
    return serviceHub.cordaService(IncrementingService::class.java).counter
  }
}

@CordaService
class IncrementingService(serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  var counter: Long = 0

  init {
    timer(period = TimeUnit.SECONDS.toMillis(1)) {
      counter += 1
    }
  }
}

If you have a service that stores some information that is useful to an external client, you will need a way to retrieve it.

Calling code inside the node to reduce code duplication

@StartableByRPC
class ExecuteSomeInternalNodeLogicYouDontWantToDuplicateFlow(private val recipient: Party) :
  FlowLogic<List<MessageState>>() {
  override fun call(): List<MessageState> {
    return serviceHub.cordaService(MessageRepository::class.java)
      .findAllMessagesByRecipient(recipient)
  }
}

@CordaService
class MessageRepository(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  fun findAllMessagesByRecipient(recipient: Party): List<MessageState> {
    return serviceHub.vaultService.queryBy<MessageState>(
      QueryCriteria.VaultCustomQueryCriteria(
        builder { MessageSchema.PersistentMessage::recipient.equal(recipient.name.toString()) }
      )
    ).states.map { it.state.data }
  }
}

Useful pieces of reusable code can be extracted into separate classes. More specifically to Corda, you could place logic within a flow or service.

To execute code inside a flow or service from the client:

Inside a flow - Just call the flow from the client
Inside a service - Add a new flow that delegates to the service

You could implement the query above in the client, but you will have two versions if you also use the same query inside the node.

Executing logic that can’t be done via RPC

@StartableByRPC
class DoSomethingComplicatedThatYouCantDoViaRpc(private val recipient: Party) :
    FlowLogic<List<MessageSchema.PersistentMessage>>() {
    override fun call(): List<MessageSchema.PersistentMessage> {
        return serviceHub.withEntityManager {
            createQuery(
                "SELECT m FROM $TABLE_NAME m WHERE m.recipient = ?1",
                MessageSchema.PersistentMessage::class.java
            ).setParameter(
                1,
                recipient.name.toString()
            ).resultList
        }
    }

    private companion object {
        val TABLE_NAME = MessageSchema.PersistentMessage::class.jvmName
    }
}

As mentioned earlier, there is a limit to what you can do via RPC. The code above is an example of this. It accesses the EntityManager which can execute lower level queries than what the vault allows.

Conclusion

Short conclusion for this post. Flows are the only entry points into a node that can run fully custom logic. Corda provides several APIs to interact with a node, but these provide limited functionality. To execute any sort of logic (that a node can run), you need to have a flow that is allowed to walk into the node and press the button for you.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Ktor - A Kotlin web framework (Ktor 1.2.2)

Wed, 24 Jul 2019 00:00:00 GMT

For an updated post using the more recent Ktor 2.0.3 see “Ktor - a Kotlin web framework updated”.

Ktor is an asynchronous web framework written in and designed for Kotlin. Allowing the more impressive features of Kotlin, such as coroutines, to not only be used but supported as a first-class citizen. Typically, Spring is my go-to general framework and usually what I use when I need to put a REST API together. But, after recently attending a London Kotlin meetup where there was a presentation on Ktor, I decided I’d try something new for once. That is how I ended up here, writing a blog post about Ktor. So, this post is a learning experience for both you and me. The content of this post will lack experienced advice but will instead document my journey as I play around with Ktor for the first time.

Here is a bit more background information on Ktor. It is backed by Jetbrains who are also the creators of Kotlin itself. Who better to make a Kotlin web framework than the men and women that work on the language.

Implementation

Dependencies

buildscript {
  ext.kotlin_version = '1.3.41'
  ext.ktor_version = '1.2.2'

  repositories {
    mavenCentral()
  }
  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version"
  }
}

apply plugin: 'java'
apply plugin: 'kotlin'

// might not be needed but my build kept defaulting to Java 12
java {
  disableAutoTargetJvm()
}

// Ktor uses coroutines
kotlin {
  experimental {
    coroutines "enable"
  }
}

compileKotlin {
  kotlinOptions.jvmTarget = "1.8"
}
compileTestKotlin {
  kotlinOptions.jvmTarget = "1.8"
}

dependencies {
  // Kotlin stdlib + test dependencies

  // ktor dependencies
  compile "io.ktor:ktor-server-netty:$ktor_version"
  compile "io.ktor:ktor-jackson:$ktor_version"
  // logback for logging
  compile group: 'ch.qos.logback', name: 'logback-classic', version: '1.2.3'
  // kodein for dependency injection
  compile group: 'org.kodein.di', name: 'kodein-di-generic-jvm', version: '6.3.0'
}

A few things are going on here.

Ktor requires a minimum version of Kotlin 1.3, so that coroutines can be leveraged.
Dependencies on ktor-server-netty and ktor-jackson are brought in. As the name suggests, this means Netty will be used for this post. Different underlying web servers can be used depending on which you choose to import. Currently, the remaining options are Jetty and Tomcat.
Logback is brought in to handle logging. This is not included in the Ktor dependencies and is needed if you plan on doing any sort of logging.
Kodein is a dependency injection framework written in Kotlin. I have used it loosely in this post, and due to the size of the code examples, I could probably remove it altogether. The main reason it is there is to provide me with another chance to use something other than Spring. Remember this is also one of the reasons that I am trying out Ktor.

Starting the web server

With the boring stuff out of the way, I can now run you through implementing a simple web server. The code below is all you need:

fun main() {
  embeddedServer(Netty, port = 8080, module = Application::module).start()
}

fun Application.module() {
  // code that does stuff which is covered later
}

Bam. There you have it. A simple web server running with Ktor and Netty. Ok, yes, it doesn’t really do anything, but we’ll expand on this in the following sections. The code is pretty self-explanatory. The only piece worth highlighting is the Application.module function. The module parameter of embeddedServer requires an extension function for Application. This is going to be the main function that makes the server do stuff.

In the following sections, we will expand the contents of Application.module so that your web server actually does something worthwhile.

Routing

At the moment, all incoming requests will be rejected since there are no endpoints to handle them. By setting up the routing, you can specify valid paths that requests can travel along and the functions that will process the requests when they reach their destinations.

This is done inside of a Routing block (or multiple Routing blocks). Inside of a block, routes to different endpoints are set up:

routing {
  // all routes defined inside are prefixed with "/people"
  route("/people") {
    // get a person
    get("/{id}") {
      val id = UUID.fromString(call.parameters["id"]!!)
      personRepository.find(id)?.let {
        call.respond(HttpStatusCode.OK, it)
      } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
    }
    // create a person
    post {
      val person = call.receive<Person>()
      val result = personRepository.save(person.copy(id = UUID.randomUUID()))
      call.respond(result)
    }
  }
}

routing is a little convenience function to make the code flow a little smoother. The context (a.k.a this) inside of routing is of type Routing. Furthermore, the functions route, get and post are all extension functions of Routing.

routing {
  // get a person
    get("/people/{id}") {
      val id = UUID.fromString(call.parameters["id"]!!)
      personRepository.find(id)?.let {
        call.respond(HttpStatusCode.OK, it)
      } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
    }
  // create a person
  post("/people) {
    val person = call.receive<Person>()
    val result = personRepository.save(person.copy(id = UUID.randomUUID()))
    call.respond(result)
  }
}

Before you move onto the next section, I want to show you how I actually implemented the routing:

fun Application.module() {
  val personRepository by kodein.instance<PersonRepository>()
  // route requests to handler functions
  routing { people(personRepository) }
}

// extracted to a separate extension function to tidy up the code
fun Routing.people(personRepository: PersonRepository) {
  route("/people") {
    // get a person
    get("/{id}") {
      val id = UUID.fromString(call.parameters["id"]!!)
      personRepository.find(id)?.let {
        call.respond(HttpStatusCode.OK, it)
      } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
    }
    // create a person
    post {
      val person = call.receive<Person>()
      val result = personRepository.save(person.copy(id = UUID.randomUUID()))
      call.respond(result)
    }
  }
}

I extracted the code to a separate function to decrease the contents of Application.module. This is going to be a good idea when you are trying to write a more significant application. Whether how I went about it is the Ktor way or not, is another question. From having a quick look at the Ktor docs, it looks like this is a decent solution. I believe I saw another way to do this, but I would need to spend more time with it.

Contents of a request handler

The code that executes when a request is routed to a request handler is obviously pretty important. The function needs to do something after all…

Each handler function executes within the context of a coroutine. I did not really make any use of this fact since each of the functions I have shown are fully synchronous. For a bit more information on this, the Ktor docs have an async example.

For the remainder of this post, I am going to try and not mention coroutines too much since they are not particularly important for this simple REST API.

In this section, the get function will be examined a little closer:

get("/{id}") {
  val id = UUID.fromString(call.parameters["id"]!!)
  personRepository.find(id)?.let {
    call.respond(HttpStatusCode.OK, it)
  } ?: call.respondText(status = HttpStatusCode.NotFound) { "There is no record with id: $id" }
}

{id} indicates that a path variable is expected in the request and its value will be stored as id. Multiple path variables can be included, but only one is needed for this example 👍. The value of id is retrieved from call.parameters which takes in the name of the variable you want to access.

call represents the context of the current request.
parameters is a list of the request’s parameters.

Using the id from the path variables, the database searches for the corresponding record. In this scenario, if it exists, the record is returned along with the appropriate 200 OK. If it doesn’t, an error response is returned. Both respond and respondText alter the underlying response of the current call. You could do this manually, for example, by using:

call.response.status(HttpStatusCode.OK)
call.response.pipeline.execute(call, it)

Installing extra features

In Ktor, extra features can be plugged in when needed. For example, Jackson JSON parsing can be added to handle and return JSON from your application. Below are the features installed to the example application:

fun Application.module() {
  install(DefaultHeaders) { header(HttpHeaders.Server, "My ktor server") }
  // controls what level the call logging is logged to
  install(CallLogging) { level = Level.INFO }
  // setup jackson json serialisation
  install(ContentNegotiation) { jackson() }
}

DefaultHeaders adds a header to every response with the name of the server.
CallLogging logs information about outgoing responses and specifies what level to log them at. A logging library needs to be included for this to work. The output will look something like:
```
INFO  ktor.application.log - 200 OK: GET - /people/302a1a73-173b-491c-b306-4d95387a8e36
```
ContentNegotiation tells the server to use Jackson for incoming and outbound requests. Remember this required including ktor-jackson as a dependency. You could also use GSON if you prefer.

For a long list of the other features that Ktor includes, here is a handy link to their docs.

Installing features ties all the way back to the routing done earlier. routing delegates down to install inside its implementation. So you could write:

install(Routing) {
  route("/people") {
    get {
      // implementation
    }
  }
}

Whatever floats your boat, but I’d just stick to using routing. Hopefully that helped you understand what is going on under the hood, even if it was just a little bit.

Brief mention for Kodein

val kodein = Kodein {
  bind<CqlSession>() with singleton { cassandraSession() }
  bind<PersonRepository>() with singleton { PersonRepository(instance()) }
}
val personRepository by kodein.instance<PersonRepository>()

Inside of the Kodein block, instances of the application’s classes are created. In this scenario, only one instance of each class is needed. Calling singleton denotes this. instance is a placeholder provided by Kodein to pass into a constructor instead of the actual object.

Outside of the Kodein block, an instance of PersonRespository is retrieved.

Yeah, I know, there isn’t really much point to the use of Kodein here since I could have replaced it with a single line…

val personRepository = PersonRepository(cassandraSession())

Instead, let’s think about it as a very concise example to understand 👍.

Closing thoughts

As someone that is extremely biased towards Spring, I found working with Ktor very different from what I am used to. It took me a bit longer than usual to work towards some sample code that I was happy with. That being said, I think the outcome looks ok, and I will need to spend some more time with Ktor to better understand precisely how to get the best out of it. At the moment, I am confident there is a lot more to squeeze out of Ktor. For more information on Ktor, I’ll have to refer you again to their documentation where they have plenty of samples and tutorials.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Saving transactions where only a subset of parties are signers

Fri, 05 Jul 2019 00:00:00 GMT

It took a while for me to think of a title that could summarise the contents of this post without becoming a full sentence itself. I think I have managed to choose something legible 😅. Either way, let me clarify what I am actually talking about.

I have seen several people ask questions like the one below in Slack:

In the example, it shows a responder flow when the node which is running the responder flow is one of the required signers. But how about the case when the node running the responder flow is not a required signer (e.g. one of the participants of a state involved in the tx)? Do I need to write responder flow for such node? If so, how should I write the responder flow?

In other words:

I have a state that has a set of participants. Some of them must sign the transaction, some must not. How do I structure my flows, especially the responder flow to cope with this?

Due to how responder flows work, where every counterparty runs the same responder code (unless overridden). Having a group of counterparties do one thing, and another do something else is not handled by the simple code found in samples. Your flows need to be constructed to handle this explicitly.

The code to do this is relatively simple, but might not be evident unless you have been developing with Corda for a while.

So far, I have two solutions to this problem. I am pretty sure these are the best solutions currently available, and I cannot think of any others that would work or are worth pursuing. These solutions are:

Sending a flag to counterparties to tell them whether they are signers or not
Using subflow’d @InitiatingFlows to collect signatures or to save the transaction

I will expand these in following sections.

Before I get to them, what happens if you don’t account for the difference in signers and participants? A typical responder flow will include:

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) { }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session))
  }
}

If an initiating flow triggers this responder for a non-signing counterparty, an error occurs:

net.corda.core.flows.UnexpectedFlowEndException: Tried to access ended session SessionId(toLong=3446769309292325575) with empty buffer
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventsUntilFlowIsResumed(FlowStateMachineImpl.kt:161) ~[corda-node-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.suspend(FlowStateMachineImpl.kt:407) ~[corda-node-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowSessionImpl.receive(FlowSessionImpl.kt:67) ~[corda-node-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowSessionImpl.receive(FlowSessionImpl.kt:71) ~[corda-node-4.0.jar:?]
    at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:294) ~[corda-core-4.0.jar:?]
    at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:198) ~[corda-core-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:290) ~[corda-node-4.0.jar:?]
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:314) ~[corda-core-4.0.jar:?]
    at dev.lankydan.tutorial.flows.SendMessageResponder.call(SendMessageFlow.kt:70) ~[main/:?]
    at dev.lankydan.tutorial.flows.SendMessageResponder.call(SendMessageFlow.kt:64) ~[main/:?]

This is because the non-signer is never sent the transaction to sign, but, alas, their code is sitting there waiting to sign a transaction that never comes. How sad 😿. I’m here to stop the counterparties of your flows from being sad like this one here.

This is also a good teaching moment. If you ever see a stack trace like the one above, it is most likely due to misplaced sends and receives. Either they are in the wrong order or there is a missing send or receive. Run through your code line by line and you should hopefully be able to pinpoint where the mismatch is.

Differentiating by flag

This solution is the one that came to me first as it is the easier one to understand.

A counterparty is notified telling them whether they need to sign the transaction or not. Their responder flow will then execute SignTransactionFlow or skip over it and go straight to ReceiveFinalityFlow. Both paths will always receive the flag and call ReceiveFinalityFlow.

An example can be found below:

@InitiatingFlow
class SendMessageFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val spy = serviceHub.identityService.partiesFromName("Spy", false).first()

    val tx = verifyAndSign(transaction(spy))

    // initiate sessions with each party
    val signingSession = initiateFlow(message.recipient)
    val spySession = initiateFlow(spy)

    // send signing flags to counterparties
    signingSession.send(true)
    spySession.send(false)

    val stx = collectSignature(tx, listOf(signingSession))

    // tell everyone to save the transaction
    return subFlow(FinalityFlow(stx, listOf(signingSession, spySession))
  }

  private fun transaction(spy: Party) =
    TransactionBuilder(notary()).apply {
      // the spy is added to the messages participants
      val spiedOnMessage = message.copy(participants = message.participants + spy)
      addOutputState(spiedOnMessage, MessageContract.CONTRACT_ID)
      addCommand(Command(Send(), listOf(message.recipient, message.sender).map(Party::owningKey)))
    }
}

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // receive the flag
    val needsToSignTransaction = session.receive<Boolean>().unwrap { it }
    // only sign if instructed to do so
    if (needsToSignTransaction) {
      subFlow(object : SignTransactionFlow(session) {
        override fun checkTransaction(stx: SignedTransaction) { }
      })
    }
    // always save the transaction
    return subFlow(ReceiveFinalityFlow(otherSideSession = session))
  }
}

Important points to the code above:

The spy (another party) is added to the state’s participants list
Flags are sent to the participants telling them whether to sign or not
The responder flow receives the flag and skips SignTransactionFlow if told to
ReceiveFinalityFlow is invoked waiting for the initiator to call FinalityFlow

I’ll leave the explanation at that as there is not much else to say.

Separating logic by extra initiating flows

This solution is a bit more involved due to the indirection caused by the different flows intermingling with each other. This solution really needs to be read before any explaining can be done:

@InitiatingFlow
@StartableByRPC
class SendMessageWithExtraInitiatingFlowFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    logger.info("Started sending message ${message.contents}")

    val spy = serviceHub.identityService.partiesFromName("Spy", false).first()

    val tx = verifyAndSign(transaction(spy))

    // collect signatures from the signer in a new session
    val stx = subFlow(CollectSignaturesInitiatingFlow(tx, listOf(message.recipient)))

    // initiate new sessions for all parties
    val sessions = listOf(message.recipient, spy).map { initiateFlow(it) }

    // tell everyone to save the transaction
    return subFlow(FinalityFlow(stx, sessions))
  }

  private fun transaction(spy: Party) =
    TransactionBuilder(notary()).apply {
      // the spy is added to the messages participants
      val spiedOnMessage = message.copy(participants = message.participants + spy)
      addOutputState(spiedOnMessage, MessageContract.CONTRACT_ID)
      addCommand(Command(Send(), listOf(message.recipient, message.sender).map(Party::owningKey)))
    }
}

@InitiatedBy(SendMessageWithExtraInitiatingFlowFlow::class)
class SendMessageWithExtraInitiatingFlowResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // save the transaction and nothing else
    return subFlow(ReceiveFinalityFlow(otherSideSession = session))
  }
}

@InitiatingFlow
class CollectSignaturesInitiatingFlow(
  private val transaction: SignedTransaction,
  private val signers: List<Party>
) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // create new sessions to signers and trigger the signing responder flow
    val sessions = signers.map { initiateFlow(it) }
    return subFlow(CollectSignaturesFlow(transaction, sessions))
  }
}

@InitiatedBy(CollectSignaturesInitiatingFlow::class)
class CollectSignaturesResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // sign the transaction and nothing else
    return subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) { }
    })
  }
}

The flow of the code above is as follows:

The spy (another party) is added to the state’s participants list
The signer’s signature is collected by calling CollectSignaturesInitiatingFlow
CollectSignaturesInitiatingFlow creates a new session and calls CollectSignaturesFlow
CollectSignaturesResponder signs the transaction sent by CollectSignaturesInitiatingFlow
More sessions are initiated for each participant
FinalityFlow is called which triggers the SendMessageWithExtraInitiatingFlowResponder linked to the original/top-level flow

The code above, is built upon the fact that any flow annotated with @InitiatingFlow will be routed to its @InitiatedBy partner and is done so in a new session. Leveraging this allows a responder flow to be added that is only triggered for required signers. Sessions are still created for the top level flow (SendMessageWithExtraInitiatingFlowFlow) and are used in FinalityFlow.

There are a few other things that happen under the covers, but they are not needed for the context of this post.

Which is better?

Hard to say at the moment. I would have to do a little performance testing and play around with the code some more…

My current opinion is the extra initiating flows works a bit better. It removes the need for an extra trip across the network from the initiator to each individual counterparty. It adds a bit of additional boilerplate code but also extracts the signing logic out from the rest of the responder/counterparty code.

To be honest, as I said a minute ago, I really need to use it and come up with more complex use cases before I can give a good answer. I doubt I’ll ever get round to that though…😩

Conclusion

Whichever route you go down or even if you manage to come up with another one, saving transactions where only a subset of parties are signers can 100% be done in Corda. I would be pretty disappointed if this were not possible. As long as you have a process in place to alter the logic in the responder flows to handle the different sends and receives that signing and non-signing parties need. You should be good to go.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Kotlin primitive and object arrays

Fri, 21 Jun 2019 00:00:00 GMT

I initially set out to write this post because I was playing around with some reflection code and thought I found something interesting. Alas, that was definitely not the case. Instead, it was just a basic feature of Kotlin that I haven’t needed to use or focus on yet. Although this post didn’t turn out the way I wanted it to be, I still think it is a nice little post to bring some clarity to this subject.

In Java, there is the concept of primitive types and their wrapped versions. Thanks to autoboxing and unboxing, types can be interchanged between their primitive and wrapped versions. In other words, in most situations you can use a long instead of a Long or a Long instead of a long. If you didn’t notice where the capitals were in that last sentence then I imagine it probably looked quite confusing. The wording in that sentence is also crucial. More specifically, the statement “in most situations”.

Autoboxing and unboxing does not work when attempting to interchange a primitive array and a wrapped (Object) array. For example,

public class PrimitiveArrays {

  public static void main(String args[]) {
    Long[] longArray = new Long[] {1L, 2L, 3L};
    takesInPrimitiveLongArray(longArray);
  }

  static void takesInPrimitiveLongArray(long[] array) {}
}

This does not work and attempting to compile it gives the following error:

error: incompatible types: Long[] cannot be converted to long[]
    takesInPrimitiveLongArray(longArray);

Switching the method to take in Long[] and passing in a long[] will also fail to compile for the same reasons. This is not something that most Java developers will find interesting but helps set the groundwork for the actual content of this post.

Kotlin needs to provide you with the equivalent of Java’s primitive arrays. But, Kotlin does not let you define arrays using the same syntax as Java. In Kotlin initialising an array looks like:

val array = Array<Long>(3)
// or
val array: Array<Long> = arrayOf(1,2,3)

The fact that you can see the Array uses generics should highlight that it is not a primitive array. This is a fact in both Java and Kotlin, that generic types cannot be primitives. Otherwise, it could be switched out for Array<long>, and we would all be happy. The code above compiles down to an object array of Long[] instead of a primitive long[].

This situation is somewhat unique to arrays. A Kotlin Long used by itself can compile to either a Long or long in JVM bytecode. The compiled type depends on the nullability of the field. Arrays are more explicit, so their types won’t change when compiled.

To circumvent this, Kotlin provides a selection of classes that become primitive arrays when compiled down to JVM bytecode.

These classes include:

Kotlin	Java
ByteArray	byte[]
CharArray	char[]
ShortArray	short[]
IntArray	int[]
LongArray	long[]
DoubleArray	double[]
FloatArray	float[]
BooleanArray	boolean[]

There are also further classes for arrays of unsigned types.

These classes can also be interchanged between Kotlin and Java without any extra effort.

As a final piece of evidence showing you the differences between primitive and wrapped/object arrays in Kotlin, I want to show you some Kotlin code that is converted to its Java counterpart:

@file:JvmName("PrimitiveArrays")
package dev.lankydan

fun main(args: Array<String>) {
  // long and Long arrays
  val longArray = longArrayOf(1,2,3,4)
  val arrayOfLongs = arrayOf<Long>(1,2,3,4)
  // int and Integer arrays
  val intArray = intArrayOf(1,2,3,4)
  val arrayOfInts = arrayOf<Int>(1,2,3,4)
  // boolean and Boolean arrays
  val booleanArray = booleanArrayOf(true, false)
  val arrayOfBooleans = arrayOf<Boolean>(true, false)
  // char and Character arrays
  val charArray = charArrayOf('a','b','c')
  val arrayOfChars = arrayOf<Char>('a', 'b', 'c')
}

Using Intellij’s Kotlin bytecode decompiler, the snippet decompiles to:

public final class PrimitiveArrays {
   public static final void main(String[] args) {
      Intrinsics.checkParameterIsNotNull(args, "args");
      // long and Long arrays
      long[] var10000 = new long[]{1L, 2L, 3L, 4L};
      Long[] var9 = new Long[]{1L, 2L, 3L, 4L};
      // int and Integer arrays
      int[] var10 = new int[]{1, 2, 3, 4};
      Integer[] var11 = new Integer[]{1, 2, 3, 4};
      // boolean and Boolean arrays
      boolean[] var12 = new boolean[]{true, false};
      Boolean[] var13 = new Boolean[]{true, false};
      // char and Character arrays
      char[] var14 = new char[]{'a', 'b', 'c'};
      Character[] var15 = new Character[]{'a', 'b', 'c'};
   }
}

Firstly, note that Kotlin provides you with useful initialisation functions for your arrays. Both for primitive and object arrays. Secondly, how they are compiled. For example, LongArray becomes long[] and Array<Long> becomes Long[].

You can now see the differences between these arrays. But, I have not mentioned which ones you should be utilising. You should defer to primitive types in the same way that Java does. This is due to the performance impact that autoboxing and unboxing can have on your application.

For smaller workloads, the result is likely to be negligible. On the other hand, for larger arrays in performance critical applications, this possibly small change can have a noticeable effect. Some more information on this subject can be found here.

If you need to store nulls in your arrays, then you will still need to refer back to a wrapped/object array. In most situations, I think you should be able to utilise primitive arrays, but there are always going to be times when you can’t. That being said, most of the time we all just use Lists, so none of this really matters 😛.

You should now have a better understanding of the differences between primitive arrays like LongArray and object arrays such as Array<Long>. If not, then I have failed you, and I apologise for that 😥.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Preventing invalid spending of broadcasted states

Wed, 12 Jun 2019 00:00:00 GMT

Corda is super flexible and will allow you to put together the code needed to write many complex workflows. This flexibility does come with one disadvantage. You, the developer, need to spend some time and thought on designing your application. No more blank contracts. No more responder flows that sign anything they receive. You’ll be glad you put in the effort when your application is working seamlessly in production. Luckily for you, I am here to protect your vaults from dodgy transactions and invalid spending of your states.

In my previous post, Broadcasting a transaction to external organisations, I showed you how to share transactions and the states within them with anyone in a network. Raising potential avenues for states to be spent in ways that were not foreseen. Possibly leading to the business process that a CorDapp(s) defines to fall apart.

I gave you the knowledge to end up in this situation, and now I am trying to prevent you from making stupid 🤦‍♀️, and potentially significant, mistakes. In other words, I gave you the keys to the car, and now I am trying to prevent you from crashing or running someone over 🚗💀.

Or even,

🕷 With great power comes great responsibility. 🕷

In this post, I will cover the sort of checks you should be adding to your contracts and flows to prevent other nodes from gaining advantages from potential oversights in your application. More precisely, preventing states received from broadcasted transactions from being spent in ways that were not anticipated. After making these changes, states can only be spent by parties that the application fully intended to allow.

I will be using the MessageState (a LinearState) that I have included in all of my Corda posts. Added below for clarity:

@BelongsToContract(MessageContract::class)
data class MessageState(
  val sender: Party,
  val recipient: Party,
  val contents: String,
  override val linearId: UniqueIdentifier,
  override val participants: List<Party> = listOf(sender, recipient)
) : LinearState

Using this state as an example, I can help you think about the design of your own contracts and the validation in your flows. The workflow we will be looking at is the process of replying to a message sent between two nodes.

Contract verification

I hope you already know this. Contract verification is important. This is the validation that is run by each signer of a transaction and anyone who needs to verify that same transaction in the future. States can only evolve in ways which are allowed by their contract’s design. Who can create them, who can spend them, what values they can have, how many of them per transaction, and so on. It is up to you to define these rules.

Below are some general requirements I believe are important to ensure a contract places the necessary boundaries on a state. Guiding the state along the possible routes it can take to be created and transformed:

override fun verify(tx: LedgerTransaction) {
  val commandWithParties: CommandWithParties<Commands> = tx.commands.requireSingleCommand()
  when (commandWithParties.value) {
    is Commands.Send -> // validation for sending
    is Commands.Reply -> requireThat {
      val inputPublicKeys = tx.inputs.flatMap { it.state.data.participants.map(AbstractParty::owningKey) }.toSet()
      "The input participant keys are a subset of the signing keys" using commandWithParties.signers.containsAll(inputPublicKeys)
      val outputPublicKeys = tx.outputStates.flatMap { it.participants.map(AbstractParty::owningKey) }.toSet()
      "The output participant keys are a subset of the signing keys" using commandWithParties.signers.containsAll(outputPublicKeys)
    }
  }
}

Although this validation is for a LinearState, I actually nicked the rules from ContractsDSL.verifyMoveCommand. Maybe this is an indication that my MessageState is actually an OwnableState (a state with a single owner), but let’s brush that under the rug 🙈🙊.

The rules in the snippet above, check that the transaction is going to be signed by all the participants of the input and output states. Ensuring all the relevant parties have a chance to validate the transaction. In other words, the flow must include the parties mentioned by the input and output states in the command’s required signers, and sessions opened to send them the transaction.

Thanks to these rules, there is no way for a transaction to be committed to anyone’s vault without the participants of the input and output states signing the transaction. Doing so, in the eyes of the contract, would be illegal. Let me reword that into a more manageable explanation. Any transactions consuming MessageStates cannot be created without notifying the original participants of the state. Furthermore, the reply cannot be created and stored in a party’s vault without their say so.

These rules are a good basis to build up your own contract’s validation. Explicitly stating which parties must sign a transaction is an important requirement in proving its validity. Further rules can be added as needed.

For example, the following checks could be added to the example above:

override fun verify(tx: LedgerTransaction) {
  val commandWithParties: CommandWithParties<Commands> = tx.commands.requireSingleCommand()
  val command = commandWithParties.value
  when (command) {
    is Commands.Send -> // validation for sending
    is Commands.Reply -> requireThat {
      // general requirements from previous snippet
      // precise requirements for `MessageState`
      "One input should be consumed when replying to a message." using (tx.inputs.size == 1)
      "Only one output state should be created when replying to a message." using (tx.outputs.size == 1)
      val output = tx.outputsOfType<MessageState>().single()
      val input = tx.inputsOfType<MessageState>().single()
      "Only the original message's recipient can reply to the message" using (output.sender == input.recipient)
      "The reply must be sent to the original sender" using (output.recipient == input.sender)
      // covered by the general requirements
      "The original sender must be included in the required signers" using commandWithParties.signers.contains(input.sender.owningKey)
      "The original recipient must be included in the required signers" using commandWithParties.signers.contains(input.recipient.owningKey)
    }
  }
}

These rules are specific to the MessageState’s use-case. Restricting the parties who can reply to a message. More precisely, the sender of the reply must be the recipient of the original message and can only be sent back to the original sender. These rules make sense in this scenario.

Even with these rules, dodgy transactions can still go through. The contract checks the signatures based on the transaction’s states as well as the contents of the states. But, there are still some ways around these.

That being said, some of these situations can only occur if a transaction has been sent to parties not originally involved in an exchange. This is the sort of scenario that can arise from broadcasting transactions as I mentioned in the introduction and in Broadcasting a transaction to external organisations.

Now that I have set the stage let me explain a hole in the contract above. There is nothing preventing a third-party from creating a mimicked reply to the original message. In that, the sender and recipient of the reply are valid, but the message is created by someone else that is not actually the sender. It is a bit to get your head around, so you might need to read that sentence a few times.

To prevent this from occurring, some logic needs to be added into the flow that creates this transaction. This will be the topic of the following section.

Before we move on, I want to highlight again that this situation can only occur when the original transaction has been shared around with other parties. Therefore, it could be a situation that never happens. But it could. Covering your bases in these scenarios really depends on your use-case and how paranoid you are of users being naughty.

Flow verification

Flows provide a space for you to add transaction verification that lives outside of the contract. These can be rules that are more specific to an individual organisation (I touched on this in Extending Flows to customise transaction validation), or rely on information not available to the contract.

Following on from the previous section where I mentioned there being a few holes in the contract verification. The responder flow below adds two restraints to patch them up:

@InitiatedBy(ReplyToMessageFlow::class)
class ReplyToMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        val message = stx.coreTransaction.outputsOfType<MessageState>().single()
        require(message.sender != ourIdentity) {
          "The sender of the new message cannot have my identity when I am not the creator of the transaction"
        }
        require(message.sender == session.counterparty) {
          "The sender of the reply must must be the party creating this transaction"
        }
      }
    })
    return subFlow(
      ReceiveFinalityFlow(
        otherSideSession = session,
        expectedTxId = stx.id
      )
    )
  }
}

The require blocks in this example are not possible from inside the contract as they need information that the contract does not contain. This extra context is vital. The contract can only validate based on the information it is given. On the other hand, the flow knows that it is on the receiving end of a proposed transaction and who is the counterparty sending it.

Using this knowledge, the flow can add two new rules.

The sender of the reply cannot have the flow’s identity. Preventing another party from impersonating their identity and attempting to send a reply on their behalf.
The sender of the reply must be the owner of the counterparty session. Preventing another party from mimicking the identity of the sender.

Although the second rule is a superset of the first, splitting them out provides an opportunity for a better error message. If the messages are not important to you, then removing the first require statement is ok.

Thanks to the contract, the transaction has to gather this party’s signature before it can be persisted. Providing the opportunity for their responder flow to add additional validation and decline it if needed. With the contract and flow working as a team, the chances of the flow being used illegally are vastly reduced.

Conclusion

Due to Corda’s flexibility, it is your responsibility as a CorDapp developer to restrain your application enough to prevent invalid usage. This is further complicated by the ability to share transactions with nodes not involved in original interactions. If the CorDapp is not put together thoughtfully, a transaction’s states could be spent by a party that does not have direct ownership over the states. The validation found inside your contracts and flows is essential to prevent these scenarios from occurring.

However, it is important to remember that it really does depend on your use-case. Maybe you want to allow parties to spend states they don’t own from transactions shared with them. In that case, you could relax the CorDapp’s restrictions. As long as it is by design rather than an oversight.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Broadcasting a transaction to external organisations

Fri, 31 May 2019 00:00:00 GMT

There is a misconception that Corda cannot broadcast data across a network. This is simply wrong. In fact, Corda can send anything between nodes in a network. What Corda does not do, is share unnecessary data (transactions) with nodes that have nothing to do with an individual interaction. Privacy by default is a central component of Corda’s design. Compared to other DLT (Distributed Ledger Technology) platforms and blockchains, this is indeed a big difference. Sharing data with non-transacting parties might not be part of Corda’s default behaviour, but it is definitely within its capabilities. In this post, I will demonstrate the small amount of code required to send a transaction to any desired node.

That being said, the code included in this post is for a trivial implementation. Moving forward, I aim to write further posts on this subject and move towards more sophisticated and useful implementations for broadcasting transactions.

Before I show you any code, we should have a quick talk about the need for broadcasting data to parties not originally involved in a transaction. The most common reason I have come across is the need to meet regulatory requirements. For certain workflows information has to be shared with third-parties. These parties can then validate that nothing dodgy is going on and might also be a link to the real world. I am not going to lie. This isn’t exactly my area of expertise. Luckily, if you are reading this because you have concerns about meeting regulatory requirements, then you probably know better than me as to why you need the ability to broadcast transactions.

There is also information in the docs around this subject.

Onto the code. Below is a flow that sends a SignedTransaction to any Party passed to it:

@InitiatingFlow
class BroadcastTransactionFlow(
  private val stx: SignedTransaction,
  private val recipients: List<Party>
) : FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    for (recipient in recipients) {
      val session = initiateFlow(recipient)
      subFlow(SendTransactionFlow(session, stx))
    }
  }
}

For this scenario, Corda does pretty much all of the heavy lifting from inside the platform. Therefore, the only thing you need to do is iterate through the passed in parties and send the transaction to each of them.

Each call to SendTransactionFlow will communicate with the responder flow found below:

@InitiatedBy(BroadcastTransactionFlow::class)
class BroadcastTransactionResponder(private val session: FlowSession) : FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    subFlow(ReceiveTransactionFlow(session, statesToRecord = StatesToRecord.ALL_VISIBLE))
  }
}

There is only a single line inside of call. So, due to nothing else being there, it must be important. ReceiveTransactionFlow is the counterpart of SendTransactionFlow who receives and persists the transaction sent to it. Furthermore, the statesToRecord property determines which states from the transaction should be stored in the vault. Since this is a broadcast (possibly for regulatory reasons) the contents of the transaction are going to be important. To achieve this, StatesToRecord.ALL_VISIBLE is used telling ReceiveTransactionFlow to record every state contained within the transaction. If, depending on your use-case, you do not require all of the transaction’s states, then you can use ONLY_RELEVANT or even NONE.

The last important line of code in these snippets is the @InitiatingFlow annotation on BroadcastTransactionFlow. The annotation allows the flow to work without requiring its caller to create sessions and before passing them into the flow. The flow handles that. Logically this makes sense since most of the time, you will not have initiated sessions with parties that don’t already have the transaction stored. Thanks to this, using this flow from inside another is nice and simple. For example, it could be added at the end of a flow:

subFlow(FinalityFlow(stx, sessions)).also {
  // sends to everyone in the network
  val broadcastToParties = serviceHub.networkMapCache.allNodes.map { node -> node.legalIdentities.first() }
    .minus(serviceHub.networkMapCache.notaryIdentities)
    .minus(message.recipient)
    .minus(message.sender)
  subFlow(BroadcastTransactionFlow(it, broadcastToParties))
}

At this point, after calling BroadcastTransactionFlow, the transaction will exist in the vault of each party passed to the flow. This includes the states contained within the transaction. Great, job done. You now know that Corda can broadcast transactions.

There is one concern that I need to address before you copy and paste this code and ship it straight to production. Parties who receive output states from broadcasted transactions also have the ability to consume them. Therefore, if you wish to share transactions like this, then you need to put in the required safeguards to prevent organisations from spending states that don’t really belong to them. Yes, that organisation might be a regulator, and the chances of them doing anything dodgy is low. Yes, it might be possible to settle these issues outside of Corda. However, neither of these assumptions are ideal. To circumvent this, your contracts and flows must be designed to prevent organisations from spending states that are not theirs. This responsibility falls onto you, the CorDapp developer. Not Corda. The flexibility Corda empowers you with must be controlled to prevent undesired outcomes. I will cover this topic in a future post.

To wrap up, it is a myth that Corda cannot broadcast transactions to parties not originally involved in the process. You can achieve this in your application by using the code that I have presented in this post. It is not the fastest implementation. But it works. Allowing data to be shared with organisations who require details of transactions occurring on the network who are not otherwise included in individual transactions themselves.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Running a Kotlin class as a subprocess

Sat, 25 May 2019 00:00:00 GMT

Last week I wrote a post on running a Java class as a subprocess. That post was triggered by my need to run a class from within a test without prebuilding a Jar. The only difference between what I wrote in that post and what actually happened was the language. I used Kotlin to write that test. Not Java. Therefore, I have decided to write this follow up post that builds upon what I previously wrote and focuses on starting a Kotlin subprocess instead of a Java subprocess.

Let’s start with an equivalent implementation of the exec from Running a Java class as a subprocess:

@Throws(IOException::class, InterruptedException::class)
fun exec(clazz: Class<*>, args: List<String> = emptyList(), jvmArgs: List<String> = emptyList()): Int {
  val javaHome = System.getProperty("java.home")
  val javaBin = javaHome + File.separator + "bin" + File.separator + "java"
  val classpath = System.getProperty("java.class.path")
  val className = clazz.name

  val command = ArrayList<String>()
  command.add(javaBin)
  command.addAll(jvmArgs)
  command.add("-cp")
  command.add(classpath)
  command.add(className)
  command.addAll(args)

  val builder = ProcessBuilder(command)
  val process = builder.inheritIO().start()
  process.waitFor()
  return process.exitValue()
}

Short explanation since I covered everything else in my previous post.

The path to the Java executable is retrieved and stored in javaBin. Using this along with the passed in clazz, args, jvmArgs and the process’s classpath a command is created and executed by the ProcessBuilder. The class is now successfully running as a subprocess.

The code above is pretty much a copy and paste of the Java implementation. The differences are the order of the function parameters. I decided to switch args and jvmArgs around so I can make full use of their default values. This is based on the assumption that you are more likely to provide args instead of jvmArgs. It isn’t really that much of a concern when running from Kotlin since you can name the parameters when calling the function. But, if any Java code needs to call this function then the order of the parameters might be helpful (as well as adding on @JvmOverloads).

Below are some ways of calling exec:

exec(MyProcess::class.java, listOf("3"), listOf("-Xmx200m"))
// jvmArgs = emptyList()
exec(MyProcess::class.java, listOf("3"))
// args = emptyList()
exec(MyProcess::class.java, jvmArgs = listOf("-Xmx200m"))
// both args and jvmArgs = emptyList()
exec(MyProcess::class.java)

A noticeable difference when writing in Kotlin instead of Java is the number of ways that you can define a main function. This is due to Kotlin providing many different routes to create a static function.

The function I provided above will run a main function in the two following scenarios:

Inside a companion object and annotated with @JvmStatic

class MyProcess {
  companion object {
    @JvmStatic
    fun main(args: Array<String>) {
      // do stuff
    }
  }
}

Inside an object and annotated with @JvmStatic

object MyProcess {
  @JvmStatic
  fun main(args: Array<String>) {
    // do stuff
  }
}

Another way to create a static function in Kotlin is to define it outside of a class:

@file:JvmName("MyProcess")

package dev.lankydan

fun main(args: Array<String>) {
  // do stuff
}

A name needs to be provided so Java knows what to do with it. Executing the main function in this snippet is one situation where this is needed. Unfortunately, the exec function won’t work in this situation, since the MyProcess class doesn’t actually exist. Preventing it from compiling. Funnily enough, calling MyProcess from Java code will compile without any changes.

To resolve the compilation error found in the Kotlin version, exec’s parameters need to change very slightly:

@Throws(IOException::class, InterruptedException::class)
fun exec(className: String, args: List<String> = emptyList(), jvmArgs: List<String> = emptyList()): Int {
  // className passed into command
}

This change sidesteps the compilation error the original exec function presents. But, this comes with the downside of not really being type-safe and might lead to a few errors here and there where an invalid string is passed in. As you can see below:

exec("dev.lankydan.MyProcess", listOf("argument"), listOf("-Xmx200m"))

Something about this change really rustles my jimmies. Therefore providing both overloads is probably the sanest thing to do.

To wrap up, although most of what I have to say about executing Kotlin classes as subprocesses was already covered in Running a Java class as a subprocess. There are a few differences due to Kotlin’s flexibility. A main function in Kotlin can be defined in multiple ways and unfortunately, one of one those does not get along with exec function that the rest are happy with. Luckily the solution is a small code change that can work well if there are overloads for both String and Class class names.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Extending Flows to customise transaction validation

Sat, 18 May 2019 00:00:00 GMT

Through the use of flow extension, nodes running the same CorDapp can include extra validation to ensure that a transaction meets their specific requirements. The validation inside contracts focuses on the rules that must be adhered to by all transacting parties. Because of this, they are more general and focus on ensuring that no one is putting together invalid transactions. This leaves out any checks that individual organisations require. By providing a base CorDapp between the organisations with the ability to add further checks, they can each tailor the CorDapp to meet their needs. Flow extension makes this possible. Continuing the push for organisations to communicate together through common CorDapps while still providing enough customisation to meet each of their specific requirements.

That introduction might have caught your attention (I hope it did anyway), but the code to achieve this is relatively simple. It follows the same concepts that I wrote about in Extending and Overriding Flows from external CorDapps. The extra transaction validation I have been talking about can be achieved by providing a single open function inside of a flow or responder flow.

I will skip over a lot of information in the following snippets since they were covered in Extending and Overriding Flows from external CorDapps. If you haven’t realised this by now, I heavily recommend that you read that post.

For extra validation in the initiating flow, I believe a base flow that looks like the example below is what you want:

@InitiatingFlow
open class SendMessageFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  open fun extraTransactionValidation(transaction: TransactionBuilder) {
    // to be implemented by subtype flows - otherwise do nothing
  }

  @Suspendable
  final override fun call(): SignedTransaction {
    // build transaction
    val tx = verifyAndSign(transaction)
    // collect signatures
    // save transaction
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    extraTransactionValidation(transaction)
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }
}

Calling extraTransactionValidation before the TransactionBuilder’s verify function forces the proposed transaction to meet an organisation’s personal requirements before running any shared transaction validation.

An implementation of extraTransactionValidation is provided by a extended flow. Its implementation might look like:

class ExtraValidationSendMessageFlow(message: MessageState) :
  SendMessageFlow(message) {

  override fun extraTransactionValidation(transaction: TransactionBuilder) {
    requireThat {
      val messages = transaction.toLedgerTransaction(serviceHub).outputsOfType<MessageState>()
      "There must be only one output message" using (messages.size == 1)
      val message = messages.single()
      "Message must contain the secret passphrase" using (message.contents.contains("I love Corda"))
    }
  }
}

This is pretty much the same sort of validation you would see inside of a contract. That is the point really. The validation is done in the same way, but the rules the transaction is checked against are customised.

In a similar fashion to the initiating flow. The base responder flow will look like the following snippet:

@InitiatedBy(SendMessageFlow::class)
open class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  open fun extraTransactionValidation(stx: SignedTransaction) {
    // to be implemented by subtype flows - otherwise do nothing
  }

  @Suspendable
  final override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        extraTransactionValidation(stx)
      }
    })
    // save transaction
  }
}

Adding the extraTransactionValidation to the body of SignTransactionFlow.checkTransaction makes perfect sense. checkTransaction and subsequently extraTransactionValidation runs before the general contract validation that SignTransactionFlow executes.

Below is an example flow extending the base flow and implementing extraTransactionValidation:

@InitiatedBy(SendMessageFlow::class)
class ExtraValidationSendMessageResponder(session: FlowSession) :
  SendMessageResponder(session) {

  override fun extraTransactionValidation(stx: SignedTransaction) {
    requireThat {
      val messages = stx.coreTransaction.outputsOfType<MessageState>()
      "There must be only one output message" using (messages.size == 1)
      val message = messages.single()
      "Message must contain the secret passphrase" using (message.contents.contains("I love Corda"))
    }
  }
}

Yes, this validation is also included in the initiating flow. This is what I am talking about. By adding this validation into both the initiating and responder flow, an organisation can ensure that any transactions that pass through their system meet their precise requirements.

Due to the nature of this validation, it could be moved out into a function shared between the flows:

private fun validate(transaction: BaseTransaction) {
  requireThat {
    val messages = transaction.outputsOfType<MessageState>()
    "There must be only one output message" using (messages.size == 1)
    val message = messages.single()
    "Message must contain the secret passphrase" using (message.contents.contains("I love Corda"))
  }
}

This Kotlin function has been moved out of a class, making it a static function. It can then be used in both flows.

One point I want to explore further before wrapping up this post is the additional validation in the initiating flow. It is highly likely that a transaction put together by an organisation will be found valid by that same organisation. So the question is, why add the extra checks into the flow? Similar validation could be done before the flow has been invoked. Which is a perfectly valid place to do it. The benefit of adding it to the flow is realised when using a third-party CorDapp. It is further compounded as the complexity of a CorDapp increases. Determining the states created by a complex flow might not be a simple thing to do. Therefore creating and immediately checking a transaction’s contents is the safest thing to do.

To conclude, placing additional validation inside of flows ensures that transacting organisations are completely happy with the contents of transactions before they commit them. Furthermore, it continues the push for organisations to utilise CorDapps implemented by third-party developers. By providing enough flexibility in CorDapps for organisations to add their own rules. It becomes easier to adopt them, due to the guarantees that transactions meet the requirements of any organisation in the network.

The rest of the code used in this post can be found on my Github.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Running a Java class as a subprocess

Fri, 17 May 2019 00:00:00 GMT

Running a Java class (not a jar) as a subprocess is something I needed to do this week. More precisely, I wanted to spawn a new process from within a test, instead of running it inside the test directly (in-process). I don’t think this is anything fancy or a complex thing to do. But, this is not something I have ever needed to do before and didn’t know the exact code to write.

Luckily, a quick google and a few Stack Overflow posts later. I found the answer I needed. Although the answer is there, I am rewriting it here for my own benefit and as well as yours.

class JavaProcess {

  private JavaProcess() {
  }

  public static int exec(Class clazz, List<String> jvmArgs, List<String> args) throws IOException,
        InterruptedException {
    String javaHome = System.getProperty("java.home");
    String javaBin = javaHome + File.separator + "bin" + File.separator + "java";
    String classpath = System.getProperty("java.class.path");
    String className = clazz.getName();

    List<String> command = new ArrayList<>();
    command.add(javaBin);
    command.addAll(jvmArgs);
    command.add("-cp");
    command.add(classpath);
    command.add(className);
    command.addAll(args);

    ProcessBuilder builder = new ProcessBuilder(command);
    Process process = builder.inheritIO().start();
    process.waitFor();
    return process.exitValue();
  }
}

This static function takes in the Class that you want to execute along with any JVM arguments and arguments that the class’s main method is expecting. Having access to both sets of arguments allows full control over the execution of the subprocess. For example, you might want to execute your class with a low heap space to see how it copes under memory pressure (which is what I needed it for).

Note, for this to work, the class that you want to execute needs to have a main method. 👈 This is kind of important.

Accessing the path of the Java executable (stored in javaBin) allows you to execute the subprocess using the same version of Java as the main application. If javaBin was replaced by "java", then you run the risk of executing the subprocess with your machine’s default version of Java. That is probably fine a lot of the time. But, there are likely to be situations where this is not desired.

Once the commands are all added to the command list, they are passed to the ProcessBuilder. The ProcessBuilder takes this list and uses each value contained in it to generate the command. Each value inside the command list is separated with spaces by the ProcessBuilder. There are other overloads of its constructor, one of which takes in a single string where you can manually define the whole command yourself. This removes the need for you to manually manage the addition of arguments to the command string.

The subprocess is started with its IO passing up to the process that executed it. This is required to see both any stdouts and stderrs it produces. inheritIO is a convenience method and can also be achieved by calling chaining the following code instead (also configures the stdin of the subprocess):

builder
    .redirectInput(ProcessBuilder.Redirect.INHERIT)
    .redirectOutput(ProcessBuilder.Redirect.INHERIT)
    .redirectError(ProcessBuilder.Redirect.INHERIT);

Finally waitFor tells the executing thread to wait for the spawned subprocess to finish. It does not matter if the process ends successfully or errors. As long as the subprocess finishes somehow. The main execution can carry on going. How the process finished is detailed by its exitValue. For example, 0 normally denotes a successful execution and 1 details an invalid syntax error. There are many other exit codes and they can all vary between applications.

Calling the exec method would look something like the below:

JavaProcess.exec(MyProcess.class, List.of("-Xmx200m"), List.of("argument"))

Which executes the following command (or something close to it):

/Library/Java/JavaVirtualMachines/jdk-12.0.1.jdk/Contents/Home/bin/java -cp /playing-around-for-blogs MyProcess "argument"

I have cut out a lot of the paths included classpath to keep it a bit tidier. Yours will probably look much longer than this. It really depends on your application really. The path in the command above is the bare minimum needed to get it to run (obviously customised for my machine).

The exec method is reasonably flexible and helpful in describing what is going on. Although, if you wish to make it more malleable and applicable in a wider range of situations, I recommend returning the ProcessBuilder itself from the method. Allowing you to reuse this piece of code in several places while providing the flexibility to configure the IO redirects as well as the power to decide whether to run the subprocess in the background or block and wait for it to finish. This would look something like:

public static ProcessBuilder exec(Class clazz, List<String> jvmArgs, List<String> args) {
  String javaHome = System.getProperty("java.home");
  String javaBin = javaHome + File.separator + "bin" + File.separator + "java";
  String classpath = System.getProperty("java.class.path");
  String className = clazz.getName();

  List<String> command = new ArrayList<>();
  command.add(javaBin);
  command.addAll(jvmArgs);
  command.add("-cp");
  command.add(classpath);
  command.add(className);
  command.addAll(args);

  return new ProcessBuilder(command);
}

By utilising either (or both) of these functions, you will now have the ability to run any class that exists in your application’s classpath. In my situation, this was very helpful in spawning subprocesses inside of an integration test without needing to pre-build any jars. This allowed control over JVM arguments, such as the memory of the subprocesses which would not be configurable if run directly inside the existing process.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Class delegation in Kotlin

Thu, 09 May 2019 00:00:00 GMT

In software engineering, the delegation pattern is an object-oriented design pattern that allows object composition to achieve the same code reuse as inheritance.

Yes, that is a textbook copy and paste from Wikipedia 😬. I went for that quote as an opening as it is a concise description of the Delegation Pattern. Favouring composition over inheritance is something that is recited to almost every developer working with an Object Oriented language like Java and Kotlin. Using this pattern allows you to keep on the good side of the inheritance hating developers while still secretly writing code that is implementing an interface.

Ok, I can see you twitching (not really 🙄). You want to know where Kotlin comes into play 🤔.

Kotlin makes following the Delegation Pattern easier by providing the by keyword. Using this keyword in your code allows you to implement (not a class) an interface while delegating all of its functions to the delegate that by has specified. All of the interface functions will then be satisfied without you needing to provide a single implementation yourself in your new class. One way to think about it is that the delegate basically becomes a pseudo abstract class.

You now can pick and choose which interface functions you want to provide your own implementation for. Any that you don’t provide are delegated down to the delegate. This is where the power comes in.

I think it is about time I showed you an example. Below is some code used in Corda (written entirely in Kotlin and is actually mentioned on the Kotlin homepage 👏👏). This is also the first time I saw by being used:

class RestrictedEntityManager(private val delegate: EntityManager) : EntityManager by delegate {

  override fun close() {
    throw UnsupportedOperationException("This method cannot be called via ServiceHub.withEntityManager.")
  }

  override fun clear() {
    throw UnsupportedOperationException("This method cannot be called via ServiceHub.withEntityManager.")
  }
}

The above class does everything I was just talking about. So let’s take a closer look. RestrictedEntityManager wants to implement EntityManager. To do so, it uses the implementations provided by the delegate property.

Note, the by keyword’s use here. To me, this says, EntityManager is implemented by the delegate property.

As the name RestrictedEntityManager suggests, this class wants to restrict some of the EntityManager’s functions. Overrides for close and clear are supplied and used instead of the versions provided by delegate.

This leaves you with a new class, RestrictedEntityManager that implements every function in the EntityManager interface (51 functions if I counted correctly 😳😵) while only writing 2 yourself. Not only does this allow you to be lazier, it is actually clearer and only relies on interfaces rather than concrete classes.

A piece of information from the Kotlin docs on this subject is rather important.

Note, however, that members overridden in this way do not get called from the members of the delegate object, which can only access its own implementations of the interface members

I said that the delegate was like an abstract class earlier. The statement above disproves that slightly, since the implementations you provide will not be used by the delegate at all. But, I still think the comparison is a nice way to think about it, you just need to remember this fact to prevent a mistake in the future.

Before I close this post, I want to compare the code above to a Java version. This will allow you to see how Kotlin provides you with tools to improve your code:

public class RestrictedEntityManager implements EntityManager {

  private EntityManager delegate;

  public RestrictedEntityManager(EntityManager delegate) {
    this.delegate = delegate;
  }

  @Override
  public void close() {
    throw new UnsupportedOperationException("This method cannot be called via ServiceHub.withEntityManager.");
  }

  @Override
  public void clear() {
    throw new UnsupportedOperationException("This method cannot be called via ServiceHub.withEntityManager.");
  }

  @Override
  public void persist(Object entity) {
    delegate.persist(entity);
  }

  @Override
  public <T> T merge(T entity) {
    return delegate.merge(entity);
  }

  // and another 47 functions/methods 😰😰😱
}

The inclusion of that code comment on the last line should be enough to convince you of the elegance of the Kotlin solution. Just for clarity, all the missing methods are just delegating to the delegate property. If you did make changes to some of the functions and did not order them nicely, you might find it hard to notice what you did. The Kotlin version on the other hand only includes the functions that have been altered and delegates the others by default.

To wrap up, this short post has taken a look at the by keyword provided by Kotlin and how it assists you in leveraging the Delegation Pattern. Helping you take one more step on your path to becoming a super developer. Just put down that Java ☕🤮 thing you are holding on to dearly and grab on to Kotlin with two hands 👌👏👏.

Just to protect myself. I have nothing against Java. Please don’t come and kill me! 😨😨

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Spring Data R2DBC for Microsoft SQL Server

Tue, 07 May 2019 00:00:00 GMT

This post is the sibling of another blog post I wrote that focused on Spring Data R2DBC and Postgres. Hopefully, you read the title before traversing to this page. So, you should know that I will be writing about Spring Data R2DBC’s integration with Microsoft SQL Server. As Spring provides a lot of magic 🧙‍♀️, most of the code from my previous Postgres post will still work and only requires a few changes to get up and running with MSSQL.

For more background information, I do recommend reading the Postgres version of this post as I do not want to repeat myself too much. That being said, below is its introduction which should provide enough context for us to continue. I will keep this post short by providing brief notes alongside the implementation for MSSQL.

Not too long ago, a reactive variant of the JDBC driver was released. Known as R2DBC. It allows data to be streamed asynchronously to any endpoints that have subscribed to it. Using a reactive driver like R2DBC together with Spring WebFlux allows you to write a full application that handles receiving and sending of data asynchronously. In this post, we will focus on the database. From connecting to the database and then finally saving and retrieving data. To do this, we will be using Spring Data. As with all Spring Data modules, it provides us with out of the box configuration. Decreasing the amount of boilerplate code that we need to write to get our application setup. On top of that, it provides a layer upon the database driver that makes doing the simple tasks easier and the more difficult tasks a little less painful.

Dependencies

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.2.0.M3</version>
</parent>
<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-r2dbc</artifactId>
    <version>1.0.0.M2</version>
  </dependency>
  <dependency>
    <groupId>io.r2dbc</groupId>
    <artifactId>r2dbc-mssql</artifactId>
    <version>0.8.0.M8</version>
  </dependency>
  <dependency>
    <groupId>io.projectreactor</groupId>
    <artifactId>reactor-core</artifactId>
  </dependency>
</dependencies>

<repositories>
  <repository>
    <id>repository.spring.milestone</id>
    <name>Spring Milestone Repository</name>
    <url>http://repo.spring.io/milestone</url>
  </repository>
</repositories>

These dependencies are the bare minimum you need for this post. With spring-data-r2dbc and r2dbc-mssql doing all the heavy lifting. Depending on which database you want to use r2dbc-mssql can be switched out for another database that has a R2DBC driver (for example r2dbc-postgresql).

Connecting to the database

Spring has your back here 🦸‍♀️. It allows you to get all the basic setup by only extending AbstractR2dbcConfiguration and implementing a ConnectionFactory bean:

@Configuration
@EnableR2dbcRepositories
class DatabaseConfiguration(
  @Value("\${spring.data.mssql.host}") private val host: String,
  @Value("\${spring.data.mssql.port}") private val port: Int,
  @Value("\${spring.data.mssql.database}") private val database: String,
  @Value("\${spring.data.mssql.username}") private val username: String,
  @Value("\${spring.data.mssql.password}") private val password: String
) : AbstractR2dbcConfiguration() {

  override fun connectionFactory(): ConnectionFactory {
    return MssqlConnectionFactory(
      MssqlConnectionConfiguration.builder()
        .host(host)
        .port(port)
        .database(database)
        .username(username)
        .password(password).build()
    )
  }
}

The only bean that you need to implement is the ConnectionFactory that contains (as the name suggests) details about the connection to the database. This bean is one component of the DatabaseClient. This is an important class, as it is central to the execution of SQL within the Spring Data R2DBC module. The rest of the beans the DatabaseClient is built from are created inside of AbstractR2dbcConfiguration and fed into the client. Let Spring take the wheel 🙌🚗 and you will be alright. In all seriousness extending AbstractR2dbcConfiguration and implementing connectionFactory will get your application up and running very quickly.

Finally, the @EnableR2dbcRepositories annotation instructs Spring to find any repository interfaces that extend Spring’s Repository interface. This is used as the base interface for instrumenting Spring Data repositories. We will look at this a little closer in the next section.

Creating a Spring Data Repository

R2DBC does not currently support the inferring of queries, which is one of the handy features that I am used to Spring Data giving me. Although it is missing, life can continue but this time it will need a tiny bit more code to get it done:

@Repository
interface PersonRepository : R2dbcRepository<Person, Int> {

  @Query("SELECT * FROM people WHERE name = @name")
  fun findAllByName(name: String): Flux<Person>

  @Query("SELECT * FROM people WHERE age = @age")
  fun findAllByAge(age: Int): Flux<Person>
}

By extending R2DBCRepository, Spring will pick up these queries and instrument them as well as providing some typical queries for you, such as findByAllId. This is where the @EnableR2dbcRepositories annotation added earlier comes into play. Without this annotation then this repository will not do anything for you and would instead require a completely manual implementation.

@Query is used to define the query a function will provide, with Spring providing the implementation itself. The inputs passed into the query are defined using the format of @<parameter_name>.

The Flux objects returned from these functions are basically the reactive version of a List. In other words, they return multiple values. These values will be returned and processed as they arrive as long as you have subscribed to the Flux once it has been created.

The entity

Although Spring Data R2DBC does not aim to be a fully fledged ORM, it does still provide entity mapping. Below is an entity class that doesn’t really need much explanation:

@Table("people")
data class Person(
  @Id val id: Int? = null,
  val name: String,
  val age: Int
)

I said it doesn’t need much explanation, so let me make this one point. id has been made nullable and has a default value of null to allow Postgres to generate the next suitable value itself. If this is not nullable and an id value is provided, Spring will actually try to run an update instead of an insert upon saving. There are other ways around this, but I think this is good enough. To be honest, this is a Kotlin specific problem. If you are using Java, then relax, you won’t need to worry about this. That being said, come to the light side and program in Kotlin, all your dreams will be fulfilled if you do (I cannot guarantee that will actually happen 🤦‍♂️).

This entity will map to the people table defined below:

CREATE TABLE people (
  id INT NOT NULL IDENTITY PRIMARY KEY,
  name VARCHAR(75) NOT NULL,
  age INTEGER NOT NULL
)

Seeing it all in action

All the setup has been done, the Application class below makes use of the queries written earlier as well as some that Spring Data provides out of the box.

@SpringBootApplication
class Application : CommandLineRunner {

  @Autowired
  private lateinit var personRepository: PersonRepository

  override fun run(vararg args: String?) {
    personRepository.saveAll(
      listOf(
        Person(name = "Dan Newton", age = 25),
        Person(name = "Laura So", age = 23)
      )
    ).log().subscribe()
    personRepository.findAll().subscribe { log.info("findAll - $it") }
    personRepository.findAllById(Mono.just(1)).subscribe { log.info("findAllById - $it") }
    personRepository.findAllByName("Laura So").subscribe { log.info("findAllByName - $it") }
    personRepository.findAllByAge(25).subscribe { log.info("findAllByAge - $it") }
  }
}

In the real implementation of this code, some sleeps have been added to ensure that the reactive code has a chance to work. Reactive applications are meant to do things asynchronously and therefore this application will process the function calls in different threads. Without blocking the main thread, these asynchronous processes might never fully execute. To keep everything tidy, I have removed the sleeps from the example.

The output for running the code above would look something like the below:

2019-05-06 18:05:04.766  INFO 23225 --- [           main] reactor.Flux.ConcatMap.1                 : onSubscribe(FluxConcatMap.ConcatMapImmediate)
2019-05-06 18:05:04.767  INFO 23225 --- [           main] reactor.Flux.ConcatMap.1                 : request(unbounded)
2019-05-06 18:05:15.451  INFO 23225 --- [actor-tcp-nio-1] reactor.Flux.ConcatMap.1                 : onNext(Person(id=1, name=Dan Newton, age=25))
2019-05-06 18:05:20.533  INFO 23225 --- [actor-tcp-nio-1] reactor.Flux.ConcatMap.1                 : onNext(Person(id=2, name=Laura So, age=23))
2019-05-06 18:05:20.533  INFO 23225 --- [actor-tcp-nio-1] reactor.Flux.ConcatMap.1                 : onComplete()
2019-05-06 18:05:25.550  INFO 23225 --- [actor-tcp-nio-2] com.lankydanblog.tutorial.Application    : findAll - Person(id=1, name=Dan Newton, age=25)
2019-05-06 18:05:25.550  INFO 23225 --- [actor-tcp-nio-2] com.lankydanblog.tutorial.Application    : findAll - Person(id=2, name=Laura So, age=23)
2019-05-06 18:05:30.554  INFO 23225 --- [actor-tcp-nio-3] com.lankydanblog.tutorial.Application    : findAllById - Person(id=1, name=Dan Newton, age=25)
2019-05-06 18:05:35.582  INFO 23225 --- [actor-tcp-nio-4] com.lankydanblog.tutorial.Application    : findAllByName - Person(id=2, name=Laura So, age=23)
2019-05-06 18:05:40.587  INFO 23225 --- [actor-tcp-nio-5] com.lankydanblog.tutorial.Application    : findAllByAge - Person(id=1, name=Dan Newton, age=25)

A few things to take away here:

onSubscribe and request occur on the main thread where the Flux was called from. Only saveAll outputs this since it has included the log function. Adding this to the other calls would have lead to the same result of logging to the main thread.
The execution contained within the subscribe function and the internal steps of the Flux are ran on separate threads.

This is not anywhere close to a real representation of how you would use Reactive Streams in a production application but hopefully demonstrates how to use them and gives a bit of insight into how they execute.

doOnComplete()

Yes, that is a smart pun using the name of a Flux function 🤣😒🤦‍♀️.

We have come to the close of this shortish post. I haven’t explained too much because my Postgres version did all that and I am too lazy to rewrite it. But, this is also because Spring Data provides a generic interface that will lay most of the paving for you. Leaving you with only a few steps that you need to do yourself. Firstly, deciding which database to use, whether this is Postgres, H2, Microsoft SQL Server (as shown in this post) or other databases that have R2DBC drivers in the future. Finally, on the code side, setting up the ConnectionFactory and repository queries which will slightly differ between databases. I should probably mention the fact that R2DBC enables you to move towards more reactive applications. I mean this, this post is about R2DBC after all 🤷‍♀️. Utilising databases with R2DBC drivers and Spring modules will allow you to build a fully reactive application. Starting from the core of your application (the database) to the edges and endpoints exposed to external clients. Spring’s push for reactive applications as well as the improvements in the R2DBC drivers that are yet to come, will make this sort of transition (if appropriate to your business) less painful and faster to implement.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Verifying a Contract with CSV data

Sun, 05 May 2019 00:00:00 GMT

Attachments in Corda can be more than just PDFs sent along with a transaction. They can actually be used programmatically when running a flow or even inside of a contract’s verify function. Why would you want to do that though? 🤔 The answer makes a lot of sense when you think about it. Let’s take an attachment containing CSV data as an example. Actually, that is what this post is about. Anyway. The attachment could contain all the valid IDs (or whatever else) that a state is allowed to have. Now, that could be done within your code, but this is not practical for a system that needs to change over time. Maybe there are new IDs that need to be added. If they live inside the code then an updated CorDapp needs to be compiled and distributed whenever new values are allowed. Not entirely practical. But, uploading a new version of the file containing the data allows the validation to change over time, without the need for recompilation. That is an idea that makes perfect sense 👏.

Now that we agree that validating the contents of a transaction against attachment data is a good idea. The next question is where is the best place to put this validation. As mentioned before, there are two options. Inside the flow or in the contract. Putting it inside the contract makes the most sense here because all parties receiving the transaction have to run the attachment’s validation. Guarantying that they all reach consensus on the validity of the transaction 🤝.

Knowing that you can use attachments to validate a contract is great and all, but there is still some code to write before it becomes a reality. Don’t worry though, I got you 👍.

Building a transaction with an Attachment

Going to keep this one short and sweet since more information on this can be found in the docs. Below is some code from a flow that builds a transaction containing an attachment (it assumes the attachment already exists):

private fun transaction(): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    val attachmentId = attachment()
    addOutputState(message)
    addCommand(Command(Send(attachmentId), message.participants.map(Party::owningKey)))
    addAttachment(attachmentId)
  }

private fun attachment(): AttachmentId {
  return serviceHub.attachments.queryAttachments(
    AttachmentQueryCriteria.AttachmentsQueryCriteria(
      filenameCondition = Builder.equal(
        attachmentName
      )
    )
  ).first()
}

Adding the attachment to the transaction does not include any fancy code. Retrieving the attachment is a bit more involved but is not hard to put together either. In this example the attachment is being queried by name and the returned AttachmentId is then passed to the TransactionBuilder’s addAttachment (it takes the AttachmentId not the attachment itself). Using the name of the attachment is my preference but passing in the AttachmentId is also possible, assuming you know it beforehand.

There is one other sneaky piece of code that you might not have caught. I passed in the AttachmentId into the Send command. Doing so allow the attachment’s hash to be known and makes retrieving it from the transaction later far easier. I’ll quickly show you how I did that:

class MessageContract : Contract {

  interface Commands : CommandData {
    class Send(attachmentId: AttachmentId) : CommandWithAttachmentId(attachmentId), Commands
  }

  abstract class CommandWithAttachmentId(val attachmentId: AttachmentId) : CommandData {
    override fun equals(other: Any?) = other?.javaClass == javaClass
    override fun hashCode() = javaClass.name.hashCode()
  }
}

Verifying the Contract

This is where the magic 🧙‍♀️ happens. By using the attachment previously added to the transaction, the data inside of it can be pulled out and used to validate the states in the transaction. Below is the contract’s verify function:

override fun verify(tx: LedgerTransaction) {
  val command = tx.commands.requireSingleCommand<Commands>()
  when (command.value) {
    is Commands.Send -> requireThat {
      "No inputs should be consumed when sending a message." using (tx.inputs.isEmpty())
      "Only one output state should be created when sending a message." using (tx.outputs.size == 1)
    }
    is Commands.Reply -> requireThat {
      "One input should be consumed when replying to a message." using (tx.inputs.size == 1)
      "Only one output state should be created when replying to a message." using (tx.outputs.size == 1)
    }
  }
  require(isMessageInCsv(tx)) {
    "The output message must be contained within the csv of valid messages. " +
            "See attachment with hash = ${tx.attachments.first().id} for its contents"
  }
}

private fun isMessageInCsv(tx: LedgerTransaction): Boolean {
  val message = tx.outputsOfType<MessageState>().first()
  val attachmentId = tx.commandsOfType<CommandWithAttachmentId>().single().value.attachmentId
  return tx.getAttachment(attachmentId).openAsJAR().use { zipInputStream: JarInputStream ->
    zipInputStream.nextJarEntry.name
    val csv = CSVFormat.DEFAULT.withHeader("valid_messages")
      .withFirstRecordAsHeader()
      .parse(InputStreamReader(zipInputStream))
    csv.records.any { row -> row.get("valid messages") == message.contents }
  }
}

The interesting content in this example is found inside the isMessageInCsv function. It might look a bit daunting, but perhaps I just need to tidy it up a bit 😩… Forget I said that 😉. It just needs a little explanation (or maybe some code comments 🤔).

Attachments can be retrieved from a transaction via its commands property or via the getAttachment function (taking a position or AttachmentId/SecureHash). In this example, the attachment containing the CSV data is retrieved using getAttachment along with the id that was previously passed into the command.

The attachment has been retrieved by this point. Now the CSV data inside of it needs to be parsed and compared to the transaction’s states. To make this much simpler, I have used Apache Commons CSV. The attachment is opened using openAsJAR (the attachment is stored as a zip) and the library is utilised to read the data inside of it. As each row is read, it checks whether the contents of the MessageState matches the current row. If any do, then great. The state passes the test and the contract is deemed valid. If none match, the following error is output and you will need to try again 😈.

net.corda.core.contracts.TransactionVerificationException$ContractRejection: Contract verification failed: 
The output message must be contained within the csv of valid messages. See attachment with hash = 3E3031BA98F3F01843E8FD0A1B34E21C599C9C8F09765C2F820E45D6E8770948 for its contents, 
contract: com.lankydanblog.tutorial.contracts.MessageContract, transaction: 5C8963497E493684A78C0A95A30E4C30029E6C36A792DE8A1B1733DE5358BB15

This code does assume that the contents of the CSV follows some sort of format. In other words, "valid messages" is hardcoded to be a header in the file. If it is not there, then the validation becomes a bit pointless. It will fail if this happens, which is actually a good thing but if you remember back to what I said in the introduction. This introduces more hard coding and reduces the flexibility of the contract. If you wanted to improve this, you could pass in the name of the row you wish to use into the command, in the same way I added the AttachmentId to it earlier on.

Wrapping up

As you have seen, validating a transaction using data from a CSV attachment does not require too much effort. In fact, I bet that if you wanted to implement this yourself, your code would look almost identical to mine. I mean, there are only so many ways to do this. By delegating to a library most of the work will be done for you.

To summarise the steps needs to validate a transaction with some CSV data:

Upload the CSV to the node
Inside the flow, retrieve it and add it to a transaction
Inside the contract, take the CSV from the transaction and finally compare the contents of the transaction to it.

Doing this will allow your verification to change over time without the need to recompile your CorDapp. That sounds great right? I think so. If you don’t, then you should, because it is definitely great 😎.

The code used in this post can be found on my GitHub.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Uploading and downloading attachments in Corda

Thu, 25 Apr 2019 00:00:00 GMT

This post is dedicated to the number of people I have seen in the Corda Slack channel asking how to upload and subsequently download attachments to and from a node. Although both of these can already be achieved via the Corda Shell. In this post, I will be focusing on writing a client application that interacts with a node to control the attachments.

Hopefully, this short post will save me a lot of time in the future as well as helping you, the reader, implement this feature without the need to request help from anyone else.

As a Spring Fanboy, I will obviously be implementing the client application in Spring. By leveraging spring-web, an application can be put together with little effort while Spring does the heavy lifting. What does this mean for you? Well, it allows you to send and receive files to and from the Spring application by adding a few annotations and using the right objects. Praise Spring! 🙌🙏

One last thing, I am sure there is better code that can do the same thing as what I wrote for this post. If so, you can always submit a Pull Request to change it 😎. The post can be found here if needed.

Let’s get on with it.

Uploading

private val proxy: CordaRPCOps = rpc.proxy

@PostMapping
fun upload(@RequestParam file: MultipartFile, @RequestParam uploader: String): ResponseEntity<String> {
  val filename = file.originalFilename
  require(filename != null) { "File name must be set" }
  val hash: SecureHash = if (!(file.contentType == "zip" || file.contentType == "jar")) {
    uploadZip(file.inputStream, uploader, filename!!)
  } else {
    proxy.uploadAttachmentWithMetadata(
      jar = file.inputStream,
      uploader = uploader,
      filename = filename!!
    )
  }
  return ResponseEntity.created(URI.create("attachments/$hash")).body("Attachment uploaded with hash - $hash")
}

private fun uploadZip(inputStream: InputStream, uploader: String, filename: String): AttachmentId {
  val zipName = "$filename-${UUID.randomUUID()}.zip"
  FileOutputStream(zipName).use { fileOutputStream ->
    ZipOutputStream(fileOutputStream).use { zipOutputStream ->
      val zipEntry = ZipEntry(filename)
      zipOutputStream.putNextEntry(zipEntry)
      inputStream.copyTo(zipOutputStream, 1024)
    }
  }
  return FileInputStream(zipName).use { fileInputStream ->
    val hash = proxy.uploadAttachmentWithMetadata(
      jar = fileInputStream,
      uploader = uploader,
      filename = filename
    )
    Files.deleteIfExists(Paths.get(zipName))
    hash
  }
}

As I mentioned earlier I’m sure much can be done to improve it, but hey, it does the job.

I don’t think there is too much to dive into here since a lot of the code is around checking and possibly converting the file into the right format.

Although, this does raise a question, why is the conversion needed in the first place? Well, only Zips or Jars (which are pretty much Zips) can be stored as attachments in Corda. Therefore, if a different file type is received, it must first be converted so it can be stored in the node.

The only Corda specific code here is the call to uploadAttachmentWithMetadata. Setting the filename becomes handy when attempting to retrieve and download the stored attachment.

The hash (AttachmentId, an alias for SecureHash) is returned from the endpoint. Just like the filename, this provides another way to retrieve the attachment later on.

A quick example of uploading the attachment can be seen in the image below:

Downloading

I have implemented two different versions for downloading attachments.

By Hash / AttachmentId

private val proxy: CordaRPCOps = rpc.proxy

@GetMapping("/{hash}")
fun downloadByHash(@PathVariable hash: String): ResponseEntity<Resource> {
  val inputStream = InputStreamResource(proxy.openAttachment(SecureHash.parse(hash)))
  return ResponseEntity.ok().header(
    HttpHeaders.CONTENT_DISPOSITION,
    "attachment; filename=\"$hash.zip\""
  ).body(inputStream)
}

This endpoint retrieves an attachment using the hash returned from the upload endpoint. openAttachment requires a hash to pull the attachment from the node.

The hash is passed in as a String since there doesn’t seem to be any default conversion to SecureHash that Spring sets up and it isn’t worth the effort to set up for this situation.

The correct header needs to be added to trigger the download of the attachment by the caller. Calling this endpoint will then return a result like the below:

By name

private val proxy: CordaRPCOps = rpc.proxy

@GetMapping
fun downloadByName(@RequestParam name: String): ResponseEntity<Resource> {
  val attachmentIds: List<AttachmentId> = proxy.queryAttachments(
    AttachmentQueryCriteria.AttachmentsQueryCriteria(filenameCondition = Builder.equal(name)),
    null
  )
  val inputStreams = attachmentIds.map { proxy.openAttachment(it) }
  val zipToReturn = if (inputStreams.size == 1) {
    inputStreams.single()
  } else {
    combineZips(inputStreams, name)
  }
  return ResponseEntity.ok().header(
    HttpHeaders.CONTENT_DISPOSITION,
    "attachment; filename=\"$name.zip\""
  ).body(InputStreamResource(zipToReturn))
}

private fun combineZips(inputStreams: List<InputStream>, filename: String): InputStream {
  val zipName = "$filename-${UUID.randomUUID()}.zip"
  FileOutputStream(zipName).use { fileOutputStream ->
    ZipOutputStream(fileOutputStream).use { zipOutputStream ->
      inputStreams.forEachIndexed { index, inputStream ->
        val zipEntry = ZipEntry("$filename-$index.zip")
        zipOutputStream.putNextEntry(zipEntry)
        inputStream.copyTo(zipOutputStream, 1024)
      }
    }
  }
  return try {
    FileInputStream(zipName)
  } finally {
    Files.deleteIfExists(Paths.get(zipName))
  }
}

This snippet, like the uploading example, has a lot of extra code that plays around with OutputStreams and InputStreams.

This endpoint makes use of the queryAttachments function to retrieve the AttachmentIds matching the input criteria. A call to openAttachment is still needed to return the attachment once the AttachmentIds/hashes are retrieved.

Going down this route makes much more sense when compared to the downloadByHash endpoint since a filename is more human-readable than a hash.

Most of the extra code in this example is handling the possibility of multiple files being returned. If there are multiple attachments with the same name, then they get zipped together and returned.

You can then make a request like the one below which can handle the eventuality of many attachments with the same name:

Note that the file size is larger than the by hash version since I uploaded another file with the same name.

Wrapping up

There isn’t really a proper conclusion to write here. Basically, all I have to say is that, if you want to upload and download attachments to and from a Corda node with Spring, then the code in this post will help get you on your way. I am sure there are tidier ways to handle all the InputStreams and OutputStreams but they seem to get the job done.

If you found this post helpful and want to keep up with my posts as I write them, then you can follow me on Twitter at @LankyDanDev.

Asking the right question, in the right way and in the right place

Sun, 14 Apr 2019 00:00:00 GMT

Is this the right place to post this? That probably isn’t the right question to be asking when I am writing this on my personal blog. Although this is not the sort of question lead me to write this post, I was hoping it would start the post in the right mood and perhaps even muster a chuckle from you. It did right? Another silly question. Of course it did, it was a great joke 😎.

Ok, I’ll get back to what I actually want to write about and be a bit more serious.

Asking the right questions and knowing where to ask them is something that has been on my mind for a quite while now. This mainly arises from the amount of tangential/borderline irrelevant or badly written question I see but still try to answer in a slack channel I spend a lot of time in. This makes it much harder to help the developers asking questions and possibly leaves them waiting for answers that will never come.

Now, rather than just continuing on this post complaining about how people ask the wrong questions in the wrong places. I am going to switch it around and instead focus on the person asking the question. The main reason for this is during my first attempt to write this post I realised just how whiney it sounded but also how hard it is to explain the problem I was facing. I am sure that the people who sit in a similar position to me have felt the frustrations I have and that there is no point actually walking through them point by point.

Instead, providing some assistance in how to ask the right questions will be a far better use of my time, as well as yours.

This post will focus on asking questions online, for example in Slack or Stack Overflow. Although I am sure the advice provided here will help in real life interactions, there are quite a few differences when compared to online interactions.

Understand the problem

I think this is the root cause of most questions that don’t get answered. Not understanding the problem you are facing leaves you possibly asking the wrong question, in the wrong way and in the wrong place. The exact opposite of the title of this post. It can be a hard issue to fix and, in my opinion, is heavily tied to experience.

Why is experience important here? Well. Knowing the boundaries of different technologies, frameworks or languages requires time and practice to fully grasp. I am facing this problem myself right now. Recently I have been spending time rebuilding my blog using Gatsby (a static site generator) that uses React under the hood. Trying to determine where Gatsby ends and React starts has required a bit of effort but I think I’m getting there.

Now, why do I mention this? The first questions I googled were not very helpful. Eventually, after playing around with the code a bit more and a few more Google searches I started to understand what is handled by Gatsby, React and Javascript itself. But, that being said, I have a few years being a Software Engineer under my belt so figuring these sort of things out has become easier. I’m not saying I can pick up anything and understand it, but it is massively easier than it was when I started out, maybe there is even a large difference between last year and now even.

What I’m trying to say in a convoluted way, is, that it gets easier. But, to do so, you need to look a bit closer at the technologies, frameworks or languages you spend time with. Eventually, you will see similarities in things you have never touched before to experiences you have already had. Not only will this help you achieve better results independently. For the times that you still need help, and I’m sure you will (I sure do), you can better formulate a question filled with relevant information. Allowing anyone kind enough to answer it to have a smoother experience as well as provide you with a clearer and more precise solution.

Spend time on it

Spending time trying to solve a problem yourself before asking a question is very important. It is heavily tied to understanding the problem as a lot of information is revealed as you work through it yourself. Therefore, when you do come to ask for help, the amount of useful information you can provide will greatly increase. Routes you followed that were dead ends, bugs that popped up and performance factors are just some examples of insights you can fill your question with. Now, when someone attempts to assist you, they will have a more detailed picture allowing their suggestions and advice to be better based on facts and experience, rather than taking a stab in the dark.

Furthermore, by actually spending a bit more time trying to fix it yourself. Well, you might fix it. There have been so many times that I have helped other developers by simply sending them the first Stack Overflow post I found on Google. If a bit more time was spent on these problems and possibly with some better Google searches, there is a good chance they could be solved without any assistance. This is something developers of all levels can do.

On the other hand, if you really can’t figure it out. It’s ok. From my perspective, I just want to know that you’ve tried and aren’t just trying to get me to do your work for you. Give me what you have found out. Let me know what didn’t work. I’m sure others feel the same way. We want to help you because we can and to raise you up, but we don’t want to be taken advantage of and do everything for you.

Provide the right about of detail

When I help people with questions they raise, one of my most common replies is “can you provide the stack trace”. This is a typical situation of useful information not being provided that is crucial to resolving an issue. For example, if you are asking for help in a Slack channel, please, please, please, do not just type the words “please help me, my code isn’t working”. Instead, add a short description of what you are trying to achieve, what you have done and what is going wrong. The description of what is going wrong can be a literal description, but more often than not, providing a stack trace or some log lines that you believe are important will go a long way in increasing the clarity of the question. More detail can always be provided at a later point when someone comes to aid you.

I think this is super important. Especially in Slack channels and the same most likely applies to Stack Overflow. Since it allows us to answer questions simply by browsing and responding to ones that catch our eyes. A stack trace or good description is much more likely to attract our attention. It is entirely possible that the question can be answered solely from the information provided.

Although, be careful of adding too much information. The advice from the previous two sections should help with this. Providing too much information is also harmful to the answerer. It becomes harder to extract what is useful from the question when there is a large wall of text with information of varying relevance. This why I am highlighting the previous sections as a cure for this illness. Understanding the problem and spending time trying to fix it yourself will provide you with the insight to filter out any noise and include the most relevant information, making it easier for us to answer your question.

Conclusion

When asking a question, try to properly understand where the problem you are facing lies. Spending time attempting to resolve it yourself will greatly help with this. Once you have done this, providing the right amount of details becomes far easier. You will have gained a superior insight into the precise location of the problem, as well as an improved understanding of the possible routes to get there. Working on these will not only allow us (the answerers) to have an easier time providing solutions and increase the chances of us attempting to help you but also improve the quality of the solution you receive. This leaves you with three parties that benefit from your effort; you, us, and anyone facing a similar issue that happens to see your question.

TLDR - Read the first three headings

Developing with Corda 4

Fri, 05 Apr 2019 00:00:00 GMT

This is an updated version of the Developing with Corda post that I wrote last year. Since then quite a lot has changed, but from your perspective, as a developer, it should feel very similar. Corda has changed a lot to increase its performance, improve security and provide a better developer experience. Touching on the last point, improving the developer experience requires adding better functionality while removing less desirable parts. But, to do so while maintaining backwards compatibility allows you to go through the process of working with Corda 3 to 4 come with little issues. That being said, there are a few differences. These are the things that will be highlighted throughout this post while mainly focusing on providing a base for any newer Corda developers reading this.

If you have already worked with Corda before, then skimming through this post is probably enough. Maybe you have worked with Corda 3 and want to upgrade to 4, then taking a quick look at the code examples and seeing if anything looks different will probably get you what you need out of this post. But, if you are a brand new to Corda, then I recommend you take some time to read through this as well as What is Corda? and watch the Corda key concepts.

From this point on you will be reading through an updated version of Developing with Corda.

In What is Corda? I gave an overview of what Corda is trying to achieve and the design decisions that were made to do so. That information is great to know so we can get some perspective of the platform but it’s not going to help us in writing a system that can leverage Corda. To do that we need to know how the components of Corda work and fit together, only then can we start writing an application that actually does something and works correctly from the perspective of Corda. I see this as the same as learning any other framework. But, we need the voice in the back of our heads to remind us every now and then that we are in fact designing upon a Distributed Ledger Technology platform. Steps need to be taken to ensure that the applications we create are properly designed.

In this post, we will look at writing a very simple Corda application.

Corda is compatible with any JVM language. I know you’re thinking it, but no, it’s not written in Java. It is actually written in Kotlin. This might require a little bit of extra learning to get to grips with Kotlin if you normally work with Java (like I do) but it shouldn’t take you long to be comfortable with it. Due to this, I personally suggest writing your own code in Kotlin to keep the whole stack in the same language. This will help when you need to dig down into Corda’s own code as it will look less alien compared to what you were just writing. Obviously, that is just my suggestion and you could instead use Clojure. But you’re going to find it hard to get a hold of any existing examples without first converting them into their Clojure equivalents.

It is a bit hard to dive straight into the code without first understanding the key concepts of Corda. Rather than going through these myself, I personally believe the documentation provide on this subject is very helpful and should get you most of the way there.

For the purpose of this post, I think it is best to leave out the configuration required to actually setup Corda nodes and instead we should fully focus on writing a Corda application.

Overview of the application

Before we can begin writing any code, we need to have an understanding of what the application is trying to achieve. Rather than coming up with my own example, I will lean upon r3’s training materials as the example is reasonably simple to understand. This example is the “IOU” process (I owe you), where someone requests for some money which will be paid back at a later date. In this post, we will just focus on the issuing of an IOU.

Below is a sequence diagram containing the very simplified steps involved in issuing an IOU between two people:

As you can see, the borrower asks for some money, if the lender agrees, then they both try and remember that the borrower owes the lender some money. This is process is simple, but even then, it has been simplified some more. In this process, we haven’t actually mentioned when and how the money is transferred between them. For the purpose of this tutorial, we shall leave this transfer out and just focus on saving the intent of borrowing and lending money between them.

So, how is this sort of process modelled in Corda? Again, before we move on, I want to mention Corda’s key concepts as they are extremely important for understanding what is going on properly.

To model this process in Corda, we need to replace some of the steps. Determining how much money to borrow, becomes the creation of a transaction containing this information. A party being happy with what is proposed is now represented by signing the transaction with their key. Communication between the borrower and lender is replaced with sending a transaction between them. Finally, remembering who owes who is now set in stone with both parties saving the transaction that occurred between them. By altering the simple diagram with this new information, a new version is created:

There are more steps, but the process is still pretty simple. The diagram really speaks for itself and I don’t think there is anything more I can add to it. What I will say, is that I have still further simplified the diagram a little bit by removing the Notary, but only to focus on the process we are trying to model. We’ll touch on what a Notary is very, very, very briefly later on but I will again suggest Corda’s documentation on the subject (I’ll do the same again later on).

These diagrams should provide us with enough guidance to put our Corda application together for issuing an IOU between two parties.

States

Onto the coding sections, let’s start with states. Information on states can be found here from the Corda documentation. States are passed around the network between nodes and eventually find themselves stored on the ledger. In terms of a transaction, a state can be an input or an output and many of these can exist on a single transaction. They are also immutable, which is required to build up the chain of states that are used within transactions.

As mentioned earlier, I am leaning upon r3 training materials. Below is the IOUState that will be passed around with the application:

@BelongsToContract(IOUContract::class)
data class IOUState(
  val amount: Amount<Currency>,
  val lender: Party,
  val borrower: Party,
  val paid: Amount<Currency> = Amount(0, amount.token),
  override val linearId: UniqueIdentifier = UniqueIdentifier()
) : LinearState {

  override val participants: List<Party> get() = listOf(lender, borrower)
}

Due to the “IOU” concept, pretty much all the fields make sense without much explanation; amount is the lent amount, lender the party lending the money and so on. The only property that needs explaining is linearId which is of type UniqueIdentifier, this class is basically a UUID, in fact, it’s internalId is generated from the UUID class.

The state extends LinearState which is one of the general types of state that Corda uses with another being FungibleState. Both of these are implementations of ContractState. LinearStates are used to represent states that, quoting its docs, evolves by superseding itself. As such, when the state is updated it should be included as an input of a transaction with a newer version being output. The old state will now be marked as CONSUMED from UNCONSUMED when saved to the vault (Corda’s abstraction over the database).

ContractState requires a property that returns the participants of the state.

The participants are a very important part of the state. The parties defined within this list determine who gets to have the state saved to their own vault/database. If you find that your state is not being saved to a party that should know about it, this is most likely the issue. I have personally run into and felt the pain of this mistake.

In the above code, the participants were not included in the constructor and instead relies on defining a get function that can be used to retrieve them. Here it returns the lender and the borrower since they are the only two parties involved in the transaction. If you wanted to, you could add the participants to the constructor like below:

@BelongsToContract(IOUContract::class)
data class IOUState(
  val amount: Amount<Currency>,
  val lender: Party,
  val borrower: Party,
  val paid: Amount<Currency> = Amount(0, amount.token),
  override val linearId: UniqueIdentifier = UniqueIdentifier(),
  override val participants: List<Party> = listOf(lender, borrower)
) : LinearState

This allows you to define participants that might not be included in the state. Which route you take depends on your use-case, for this tutorial either will do the job.

One quick little thing before moving onto the next section. The @BelongsToContract annotation, that was not mentioned before, is binding a state to a contract (which we will look at very soon). Another way to write the same code without the need for the annotation is to move the state into an inner class of the contract, this would look something like the below:

class IOUContract : Contract {

  data class IOUState(
    val amount: Amount<Currency>,
    val lender: Party,
    val borrower: Party,
    val paid: Amount<Currency> = Amount(0, amount.token),
    override val linearId: UniqueIdentifier = UniqueIdentifier(),
    override val participants: List<Party> = listOf(lender, borrower)
  ) : LinearState
}

If you don’t do this or add the annotation then you’re going to get the following warning:

[WARN] 16:41:55,620 [Test worker] contracts.TransactionState.<init> - State class net.corda.training.state.IOUState
is not annotated with @BelongsToContract, and does not have an enclosing class which implements Contract.
Annotate IOUState with @BelongsToContract(net.corda.training.contract.IOUContract.class) to remove this warning.

Lucky for you, it is a clear message and should point you in the correct direction. If you ever get around to looking into the Corda codebase then you will see that the inbuilt states that have their own contracts are written like the example above. More information on this can be found in the Corda 3 to 4 upgrade notes.

Contracts

Next up are contracts. Contracts are used to validate input and output states for a given command by all parties involved in the transaction. The command could be, for example, issuing the state or paying off owed money. We will look at commands slightly later in this section but we should be able to brush over them for now.

Contracts are quite nice to write due to their expressiveness. We are able to write conditions within the contract that must be met for a transaction to be valid. If any are not met then an exception will be thrown. This normally ends in terminating the current transaction since an involved party has found it invalid.

These conditions use the requireThat DSL that Corda defines to specify conditions along with error messages that detail what is wrong with the transaction. This makes it nice and easy to go through a contract and understand what it is doing since the code conditions are nicely complemented by the English messages (or whatever language you want to write them in).

Below is an example of a contract that is used to validate the IOUState defined above:

class IOUContract : Contract {

  companion object {
    @JvmStatic
    val IOU_CONTRACT_ID = "net.corda.training.contract.IOUContract"
  }

  interface Commands : CommandData {
    class Issue : TypeOnlyCommandData(), Commands
    class Transfer : TypeOnlyCommandData(), Commands
    class Settle : TypeOnlyCommandData(), Commands
  }

  override fun verify(tx: LedgerTransaction) {
    val command = tx.commands.requireSingleCommand<IOUContract.Commands>()
    when (command.value) {
      is Commands.Issue -> requireThat {
        "No inputs should be consumed when issuing an IOU." using (tx.inputs.isEmpty())
        "Only one output state should be created when issuing an IOU." using (tx.outputs.size == 1)
        val iou = tx.outputStates.single() as IOUState
        "A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)
        "The lender and borrower cannot have the same identity." using (iou.borrower != iou.lender)
        "Both lender and borrower together only may sign IOU issue transaction." using
                (command.signers.toSet() == iou.participants.map { it.owningKey }.toSet())
      }
      is Commands.Transfer -> requireThat {
        // more conditions
      }
      is Commands.Settle -> {
        // more conditions
      }
    }
  }
}

I have simplified the contract for the purpose of this post since we will only focus on implementing one command type. Let’s start from the top and work our way down.

The IOUContract implements Contract requiring it to now have a verify function that gets called to verify (hence the name) a transaction.

Contract class name

The class name of the contract has been included here:

companion object {
  @JvmStatic
  val IOU_CONTRACT_ID = "net.corda.contracts.IOUContract"
}

This is used in other parts of Corda when reflection is required. Another way to write this if you don’t want to put the string in directly is like below:

companion object {
  @JvmStatic
  val IOU_CONTRACT_ID = IOUContract::class.qualifiedName!!
}

Commands

Now we can talk about the commands that I briefly mentioned earlier on in this section. I have put them below again so you don’t need to scroll again:

interface Commands : CommandData {
  class Issue : TypeOnlyCommandData(), Commands
  class Transfer : TypeOnlyCommandData(), Commands
  class Settle : TypeOnlyCommandData(), Commands
}

These commands are used to specify the intention of the transaction. They have been put here due to their connection to the contract since they determine what conditions must be validated. That being said, you could put these commands somewhere else if you wanted, such as in their own file or outside of the contract class but within the same file. As long as your solution best describes your intention then you are probably going in the correct direction.

Since these commands are simple and are only used to specify intent, TypeOnlyCommandData is extended. Other abstract classes are available that specify commands that we might want to use, such as MoveCommand.

We will see how to use the commands in the next section.

Implementing verify

Here’s where most of the magic happens, the code has been copied and pasted below:

override fun verify(tx: LedgerTransaction) {
  val command = tx.commands.requireSingleCommand<IOUContract.Commands>()
  when (command.value) {
    is Commands.Issue -> requireThat {
      "No inputs should be consumed when issuing an IOU." using (tx.inputs.isEmpty())
      "Only one output state should be created when issuing an IOU." using (tx.outputs.size == 1)
      val iou = tx.outputStates.single() as IOUState
      "A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)
      "The lender and borrower cannot have the same identity." using (iou.borrower != iou.lender)
      "Both lender and borrower together only may sign IOU issue transaction." using
              (command.signers.toSet() == iou.participants.map { it.owningKey }.toSet())
    }
    is Commands.Transfer -> requireThat {
      // more conditions
    }
    is Commands.Settle -> {
      // more conditions
    }
  }
}

The verify function checks whether the proposed transaction is valid. If so, the transaction will continue forward and most likely be signed and committed to the ledger. But, if any of the conditions are not met an IllegalArgumentException is thrown thus leading to the termination of the proposed transaction.

Exceptions are generally how Corda deals with unmet requirements. When an exception is thrown, assuming nothing is trying to catch it, execution is terminated and propagated up until it is caught or it ends up in the console output. This provides a simple way to control the flow of the transaction since it can be stopped anywhere in its execution, even on the counterparty’s node, as the exception will be passed back to the initiator.

Onto the verification code itself. The command that the transaction is executing on its states is retrieved and depending on the type, different checks are made to check the validity of the transaction. The requireThat DSL that Corda provides allows you to write a condition that must be true to continue along an error message that is output if the condition is false.

Let’s look at one of the requireThat statements a bit more closely:

val iou = tx.outputStates.single() as IOUState
"A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)

There’s not much to explain here. The DSL takes care of the intent of the statement. What I will point out is the syntax:

<string message of what condition should be met> using <condition it must pass>

Quite simple. Although I will highlight this point. Something that used to stupidly catch me out when I started working with Corda, was the need for brackets in the using statement if the condition contains any spaces. Forgetting this will give you some errors 🤦‍. Finally, the DSL can contain code that is not in a condition expression, allowing you to initialise variables and such which can then be used in the actual conditions.

That’s enough of contracts for now. They will pop up again in the next section when we put the IOUContract into action.

Flows

In Corda, flows are the central point where we tie off all previous sections. States, contracts, and commands all come together to write the code that will propose a new transaction, send it to all counterparties to sign and commit it to the ledger if everyone is happy. You can do much more complicated things within flows but for this tutorial, we will stick with the basics.

Following on from the examples in the previous sections, we will now implement the IOUIssueFlow. Below is the code as a whole which we will then split and examine:

@InitiatingFlow
@StartableByRPC
class IOUIssueFlow(val state: IOUContract.IOUState) : FlowLogic<SignedTransaction>() {
  @Suspendable
  override fun call(): SignedTransaction {
    val transaction: TransactionBuilder = transaction()
    val signedTransaction: SignedTransaction = verifyAndSign(transaction)
    val sessions: List<FlowSession> = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet().toList()
    val transactionSignedByAllParties: SignedTransaction = collectSignature(signedTransaction, sessions)
    return recordTransaction(transactionSignedByAllParties, sessions)
  }

  private fun transaction(): TransactionBuilder {
    val notary: Party = serviceHub.networkMapCache.notaryIdentities.first()
    val issueCommand: Command<IOUContract.Commands.Issue> =
      Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
    val builder = TransactionBuilder(notary = notary)
    builder.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
    builder.addCommand(issueCommand)
    return builder
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }

  @Suspendable
  private fun collectSignature(
    transaction: SignedTransaction,
    sessions: List<FlowSession>
  ): SignedTransaction = subFlow(CollectSignaturesFlow(transaction, sessions))

  @Suspendable
  private fun recordTransaction(transaction: SignedTransaction, sessions: List<FlowSession>): SignedTransaction =
    subFlow(FinalityFlow(transaction, sessions))
}

@InitiatedBy(IOUIssueFlow::class)
class IOUIssueFlowResponder(val flowSession: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val signTransactionFlow = object : SignTransactionFlow(flowSession) {
      override fun checkTransaction(stx: SignedTransaction) = requireThat {
        val output = stx.tx.outputs.single().data
        "This must be an IOU transaction" using (output is IOUContract.IOUState)
      }
    }
    val signedTransaction = subFlow(signTransactionFlow)
    return subFlow(ReceiveFinalityFlow(otherSideSession = flowSession, expectedTxId = signedTransaction.id))
  }
}

The flow of this code (yes that is a pun 😅) is reasonably straight forward and will be possibly one of the typical flows that you write within your own application. I can say for most flows that I have written over the last year or so for both work and for my other blog posts have followed this sort of structure. To be honest, whenever I start a new one, I copy and paste one a flow that looks like this and then start from there. Hasn’t failed me yet 🙂.

You obviously don’t have to write your flows in the exact same way. I have split the call function into a load of smaller functions. It does actually make the code a lot longer. But, I don’t think it can get any easier to read than that 😎.

Anyway, enough about myself and back to the code.

What it does:

Creates a state
Adds the state to a new transaction
Verifies the transaction with a contract
Signs the transaction
Requests the signatures of the counterparties
Saves the transaction for all participants

Now that we know the steps that are made within this flow, we can go through and explain how this is done within the code.

The Initiating Flow

Here’s the code again with just the Initiating Flow extracted from it:

@InitiatingFlow
@StartableByRPC
class IOUIssueFlow(val state: IOUContract.IOUState) : FlowLogic<SignedTransaction>() {
  @Suspendable
  override fun call(): SignedTransaction {
    val transaction: TransactionBuilder = transaction()
    val signedTransaction: SignedTransaction = verifyAndSign(transaction)
    val sessions: List<FlowSession> = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet().toList()
    val transactionSignedByAllParties: SignedTransaction = collectSignature(signedTransaction, sessions)
    return recordTransaction(transactionSignedByAllParties, sessions)
  }

  private fun transaction(): TransactionBuilder {
    val notary: Party = serviceHub.networkMapCache.notaryIdentities.first()
    val issueCommand: Command<IOUContract.Commands.Issue> =
      Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
    val builder = TransactionBuilder(notary = notary)
    builder.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
    builder.addCommand(issueCommand)
    return builder
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }

  @Suspendable
  private fun collectSignature(
    transaction: SignedTransaction,
    sessions: List<FlowSession>
  ): SignedTransaction = subFlow(CollectSignaturesFlow(transaction, sessions))

  @Suspendable
  private fun recordTransaction(transaction: SignedTransaction, sessions: List<FlowSession>): SignedTransaction =
    subFlow(FinalityFlow(transaction, sessions))
}

Firstly, the flow class is annotated with @InitiatingFlow and extends FlowLogic. This combination is required by any flow that requests communication with a counterparty. FlowLogic contains one abstract function call that needs to be implemented by the flow. This is where all the magic happens. When the flow is triggered, which we will look at later, call is executed and any logic that put inside the function runs. FlowLogic is generic (FlowLogic<T>) where T determines the return type of call. In the above example, a SignedTransaction is returned but it is totally feasible to use FlowLogic<Unit> if you have no desire to return anything back to the caller of the flow.

Next up is the @StartableByRPC annotation. This allows the flow to be called from an RPC connection which is the interface between the outside of a Corda node and it’s internals. We’ll touch on this a bit more when we look at triggering the flow.

Yet another annotation popping up. @Suspendable actually originates from quasar-core instead of one of Corda’s own libraries. If you forget to add it then you will see the error below and your flow will hang:

java.lang.IllegalArgumentException: Transaction context is missing.
This might happen if a suspendable method is not annotated with @Suspendable annotation.

@Suspendable is needed on all functions that communicate with a counterparty. As the name suspendable suggests, the annotation allows the function to be suspended while the counterparty is dealing with their side of the transaction. Quite a bit of magic goes on here and it is touched on briefly in the Corda documentation on flows.

In my original version of this post, I talked about the error being confusing and unrelated. But, it looks like that is resolved and an error message explaining exactly what is wrong has been thrown. That being said, if you do ever get some weird errors, it is still worth double checking that you have got @Suspendable in the right place.

Linking this back to the example. call, collectSignatures and recordTransaction are all annotated with @Suspendable. Both collectSignatures and recordTransaction call subFlows that interact with counterparties. call on the other hand needs the annotation because it is delegating to our own functions which are suspending.

This is our first encounter with subFlows. I will put a little section further down to quickly cover this and give the topic the explanation it deserves.

Back to the task at hand.

Missing the @Suspendable annotation is one of the most common pitfalls I have seen other developers fall into. So let me reiterate this point again.

You need to add the @Suspendable annotation to any function that has an interaction with a counterparty

You don’t understand how hard it is to not simplify this all the way down to 🤦‍

You need to add the @Suspendable annotation to any function that suspends

Just read whichever one gets my point across.

Now we’re done with the annotations we can look more closely at the contents of the call function.

Creating the transaction

First, we will look at building the proposed transaction, the relevant code has been extracted below:

private fun transaction(): TransactionBuilder {
  val notary: Party = serviceHub.networkMapCache.notaryIdentities.first()
  val issueCommand: Command<IOUContract.Commands.Issue> =
    Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
  val builder = TransactionBuilder(notary = notary)
  builder.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
  builder.addCommand(issueCommand)
  return builder
}

For the purpose of this post, we will assume there is only one Notary which allows us to be lazy and just retrieve the first one from the list. If you do not know what a Notary is, like earlier I suggest reviewing the Corda Key Concepts for a good explanation on the topic. For now, I’ll provide you the bare minimum to carry on. A Notary is a node whose sole purpose is to validate that no double spends have occurred within a transaction sent to it. Extra validation can also be run if it is set up to do so.

The serviceHub comes provided since we extended FlowLogic; the function networkMapCache will then provide the identities of the parties on the network and notaryIdentities narrows it down even more. As I mentioned earlier, we’re going to be lazy and just retrieve the first one from this list. How you retrieve the Notary that you wish to use in a transaction might change depending on your requirements.

We then create a command that represents the intent of the transaction. In this case, we use the IOUContract.Commands.Issue that was defined earlier. In creating the command we also need to provide the public keys of the parties required to sign the transaction. it is a Party and owningKey represents their public key. The only signers in this transaction are contained within the states participants property but an independent list could be passed in instead.

All the components needed for the transaction have now been retrieved or created. Now we need to actually start putting it together. TransactionBuilder does just that. The Notary is added via one of the TransactionBuilder’s constructors but can be set later on if preferred. addOutputState takes in the state passed into the flow along with the contract name that will verify it. Remember I mentioned two ways to get this name; via a public property within the object (how Corda normally does it) or by manually adding the classes name yourself, either way, the end goal is the same. The final component we add to this transaction is the command we created.

Verifying and signing the transaction

The next block of code focuses on verifying and signing the transaction, again the relevant code has been pasted below:

private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
  transaction.verify(serviceHub)
  return serviceHub.signInitialTransaction(transaction)
}

Once we are happy that everything we want to include in the transaction has been included, we need to verify it. Simply call the verify function the TransactionBuilder provides. This function results in the validation inside the contract running against the transaction. As mentioned earlier in the contract section, if any of the conditions in the contract fail an exception is thrown. Since, in this code, there are no attempts to catch the exception, the flow will fail as the exception is propagated up the stack.

After the transaction has passed validation, as far as we (the initiator) are concerned, the transaction is valid and should be signed. To do this, serviceHub.signInitialTransaction is called. This creates a new SignedTransaction that is currently only signed by the initiator. Having this transaction signed will become important later when the Notary checks that the transaction has been signed by all the parties involved.

Collecting signatures from the counterparties

The transaction is now both verified and signed by the initiator. The next step is requesting the signatures of the counterparties involved in the transaction. Once that is done the transaction can be persisted to everyone’s vaults as they all agree that the transaction is correct and meets their needs.

Below is the code in charge of collecting signatures from counterparties:

@Suspendable
private fun collectSignature(
  transaction: SignedTransaction,
  sessions: List<FlowSession>
): SignedTransaction = subFlow(CollectSignaturesFlow(transaction, sessions))

The counterparties in this transaction are defined by the parties in the participants list. If we remember back to how the state’s participants field was constructed, only two parties were contained in it, therefore only two parties will need to sign the transaction. Although that statement is correct, the transaction has already been signed by the initiator so now only the single counterparty (the lender) needs to sign it.

To send the transaction to the counterparty we first need to create a session with them. The code below does just that and the sessions returned from it are the input to the function above:

val sessions: List<FlowSession> = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet().toList()

The initiateFlow function creates a session to another node, it takes in a Party and returns a FlowSession to be used for communication. As mentioned the initiator does not need to sign the transaction again through this construct, so in the code, they have been removed from the parties whose communication sessions are being created for. Due to us knowing who are involved in this transaction, the single FlowSession could be created with the code below instead:

val session = initiateFlow(state.lender)

Instead of relying on the participants list we instead just create a session for the lender as our knowledge of the state indicates that they are the only counterparty.

The FlowSession needs to be used inside of the CollectSignaturesFlow along with the SignedTransaction that is still only signed by the initiator at this point. CollectSignaturesFlow sends the SignedTransaction to the counterparty (counterparties if multiple need to sign) and awaits their response. We will look at how the response is sent back in a following section. Once returned, the SignedTransaction is now complete as it has been signed by everyone and can be persisted.

Persisting the signed transaction

The following code persists the transaction:

subFlow(FinalityFlow(transaction, sessions))

Although, for a one-liner, this piece of code packs quite a punch. FinalityFlow will most likely always be called at the end of your flows, at least for the simpler flows anyway.

Calling FinalityFlow will:

Send the transaction to the Notary (if required)
Record the transaction to the initiator’s vault
Broadcast to the participants of the transaction to record it to their vaults

Expanding slightly on the 1st point. A transaction is only sent to a Notary if it has any input states that are being consumed. This is due to Notaries checking for previously spent states to prevent double spends. Therefore, if there are no input states then there is nothing for the Notary to do. Not sending it to the Notary is mainly an optimisation that removes wasted work.

The last two steps depend on the Notary finding the transaction valid. If it does not, like usual, an exception is thrown thus leading to an exit from the flow.

The Responder Flow

Everything in the flow that we have looked at so far is on the initiator’s side of the process. There have been a few times during the example where the transaction was sent over to the counterparty and some “stuff” happened. In this brief section we will inspect the code that the counterparty would run:

@InitiatedBy(IOUIssueFlow::class)
class IOUIssueFlowResponder(val flowSession: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val signTransactionFlow = object : SignTransactionFlow(flowSession) {
      override fun checkTransaction(stx: SignedTransaction) = requireThat {
        val output = stx.tx.outputs.single().data
        "This must be an IOU transaction" using (output is IOUContract.IOUState)
      }
    }
    val signedTransaction = subFlow(signTransactionFlow)
    return subFlow(ReceiveFinalityFlow(otherSideSession = flowSession, expectedTxId = signedTransaction.id))
  }
}

As before this code was included earlier on in the post.

The most important line in this class is the @InitiatedBy annotation that specifies which flow it accepts requests from and responds back to. In this example, it is the IOUIssueFlow that we have already gone through.

Since IOUIssueFlowResponder is also a flow, it extends FlowLogic and will need to implement its own version of call. The FlowSession in the constructor is the session that was used by the initiator to communicate with this flow. @Suspendable is also used on call just like it was in the initiating flow.

SignTransactionFlow is the other half of the CollectSignaturesFlow that was called in the initiator. It is an abstract class that requires checkTransaction to be implemented. This contains any extra validation that the counterparty might want to run against the transaction. SignTransaction’s call function will still verify the transaction against the contract so this is the chance for anything else; ensuring that the transaction is up to the standards of the counterparty. Saying that, checkTransaction can also contain as little code as desired and could even be empty if the contract validation is enough. Rather than showing you what that would look like, I’ll let you use your vivid imagination to imagine an empty function…

subFlow is then called on the implementation of SignTransactionFlow leading to it being executed. The validation in the contract runs, followed by the contents of checkTransaction and if all the checks come back fine, the transaction is signed and sent back to where it came from.

The final step is the invocation of ReceiveFinalityFlow. This is different to how it previously worked in Corda 3, where you did not need to manually receive the transaction. Anyway, this subFlow needs to be called to record the transaction to the vault of the counterparty. The id of the expected transaction to record is extracted from the transaction that was signed beforehand. After this point, both the counterparty will have the transaction saved away in their vault, ready to use in future flows.

The code in this class could be as simple or complicated as it needs to be. For the example used in this tutorial, simple is good enough. This will change depending on your requirements and what must be communicated between the initiator and its responders.

Making using of subFlows

You will have seen the subFlow function scattered throughout the code examples. Finally, I think we have reached a point in the post where I can talk about them a bit more.

subFlows are flows, similar to the one we are looking at in this post, that are called from within other flows. For example, CollectSignaturesFlow and FinalityFlow cannot be triggered by themselves as they are not annotated with @InitiatingFlow, therefore they can only ever be used from within a subFlow. Most of the flows provided by Corda out of the box fall within this same category.

All subFlow does is run the flow passed into it, by calling the flow’s call function and returning whatever the flow would normally return. A flow does not require anything special to be passed into subFlow, if we ever needed to, IOUIssueFlow could be passed into it from another flow.

Corda provides flows for a lot of the typical operations that need to be repeated throughout our own flows. These are called via subFlow and include (and many more): CollectSignaturesFlow, SignTransactionFlow, SendTransactionFlow and ReceiveTransactionFlow.

Starting a Flow

In this final section, we will look at how to call a flow from outside of the Corda node.

There are a few ways to do this, each working slightly differently. But, let’s keep this short and sweet and only look at the bog standard startFlow function:

proxy.startFlow(::IOUIssueFlow, state)

That’s it. As I said, short and sweet. proxy is of type CordaRPCOps which contains a load of functions revolved around interacting with the Corda node via RPC. startFlow is one of those functions. It takes in the name of the flow class along with any arguments that are part of the flow’s constructor. So in this example, IOUIssueFlow’s call function will be invoked with an IOUState being passed in to be used within the flow.

A FlowHandle<T> is returned where T is the same generic type of the invoked flow, in this case, a SignedTransaction. returnValue can then be called to retrieve a CordaFuture, allowing the result to be retrieved as soon as it’s available. CordaFuture is a subtype of a standard Future with a few extra methods made available, one of which is toCompletableFuture that may or not be useful to you (this was useful to me anyway).

Wrapping up

Here we are, at the end at last.

This post should have hopefully given you some help in understanding how to go about developing with Corda. There is much more to learn as I have only covered the basics in this post. In this post, we implemented the process of an IOU while inspecting the components that are required to do so. States are facts that are shared among parties on the network, contracts are used to validate transactions and flows contain the logic to propose new transactions. With this information, you should be in a good place to start writing your own flows. There is much more you can do with flows that hasn’t been covered within this post, but these basics should serve you well through any flows that you try to write.

If you found this post helpful and want to keep up with my posts as I write them, then you can follow me on Twitter at @LankyDanDev.

You can also join the Corda slack channel if you have any questions or are interested in learning about Corda further after reading this post.

Removing elements from a Map in Java

Sun, 03 Mar 2019 00:00:00 GMT

Very short and simple post on removing elements from a Map in Java. We will be focusing on removing multiple elements and ignore the fact you can remove a single element using Map.remove.

The Map below will be used for this post:

Map<Integer, String> map = new HashMap<>();
map.put(1, "value 1");
map.put(2, "value 2");
map.put(3, "value 3");
map.put(4, "value 4");
map.put(5, "value 5");

There are a few ways to remove elements. You could loop through the code manually and remove them:

for(Iterator<Integer> iterator = map.keySet().iterator(); iterator.hasNext(); ) {
  Integer key = iterator.next();
  if(key != 1) {
    iterator.remove();
  }
}

This is how you would do it without access to Java 8+. The Iterator is needed to prevent ConcurrentModificationExceptions when removing elements from the Map.

If you do have access to newer versions of Java (8+) then you can choose from the below instead:

// remove by value
map.values().removeIf(value -> !value.contains("1"));
// remove by key
map.keySet().removeIf(key -> key != 1);
// remove by entry / combination of key + value
map.entrySet().removeIf(entry -> entry.getKey() != 1);

removeIf is a method available to Collections. Yes, a Map itself is not a Collection and does not have access to removeIf itself. But, by using: values, keySet or entrySet, a view of the Map‘s contents is returned. This view implements Collection allowing removeIf to be called on it.

The contents returned by values, keySet and entrySet are very important. Below is an extract of the JavaDoc for values:

* Returns a {@link Collection} view of the values contained in this map.
* The collection is backed by the map, so changes to the map are
* reflected in the collection, and vice-versa.
*
* The collection supports element removal, which removes the corresponding
* mapping from the map, via the {@code Iterator.remove},
* {@code Collection.remove}, {@code removeAll},
* {@code retainAll} and {@code clear} operations.

This JavaDoc explains that the Collection returned by values is backed by the Map and that changing either the Collection or the Map will alter the other. I don’t think I can explain what the JavaDoc is saying any better than what is already written there… So I’ll stop trying on that part now. I have only shown the documentation for values, but you can trust me when I say that keySet and entrySet are also both backed by the Map‘s contents. You can read the docs yourself if you don’t believe me.

This also links back to the first example using an older Java version. The documentation specifies that Iterator.remove can be used. This is what is used earlier. Furthermore, the implementation of removeIf is very similar to the Iterator example. After talking about it, I might as well show it:

default boolean removeIf(Predicate<? super E> filter) {
  Objects.requireNonNull(filter);
  boolean removed = false;
  final Iterator<E> each = iterator();
  while (each.hasNext()) {
    if (filter.test(each.next())) {
      each.remove();
      removed = true;
    }
  }
  return removed;
}

There is a bit extra to it. But, otherwise it is pretty much the same.

So, that’s that. Not much to conclude other than me telling you to remember that using: values, keySet, or entrySet will provide access to removeIf allowing easy removal of Map entries.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Extending and Overriding Flows from external CorDapps

Sat, 02 Mar 2019 00:00:00 GMT

Corda 4 was released last week (21st Feb) bringing with it a ton of new features to make Corda more enjoyable to work with. To be honest, I am kind of assuming there are a lot of new features. I had a quick browse through the changelog, mainly to see my contributions being referenced, but I remember seeing a lot of lines of text. That has to be a good thing right?

Anyway, one of these features is the ability to extend and override Flows. It doesn’t really sound very fancy when you realise that Corda is written in Kotlin and has inheritance completely baked into it (true for Java as well). But, there is more to it than that. Corda needs to map an Initiating Flow to the counterparty Flow that is responding to it.

This is fine when two parties are using the same CorDapp. There is no extra complexity added in this situation. If, on the other hand, the counterparty wanted to send some data to an external system upon receiving a transaction, how could they do that? The original CorDapp does not know or care about this system and therefore it cannot cater to these needs. Being able to solve this sort of problem allows developers to build upon existing CorDapps and adapt them to be more suitable for their use-case. Furthermore, once good practices are laid out, extending third-party CorDapps will become easier and remove the need for teams of developers to constantly reinvent the wheel when others have already solved part of a problem. Obviously, this assumes that there is access to these external CorDapps, but it is completely in the realm of possibility. Especially with the R3 Marketplace already showcasing a collection.

In this post, we will be focusing on extending and overriding Flows. Furthermore, we will take the perspective of two different viewpoints.

The developer/maintainer of a CorDapp
A developer wanting to use and adapt an existing CorDapp

For the process to work, both sides must put in the effort to write their applications in an appropriate manner so that the benefits can be leveraged.

We will start by looking at what the original CorDapp must contain and then what a developer must do to extend it.

Before we go any further, here is a link to the official documentation on extending and overriding flows.

Writing a base Flow to allow extension

Writing a CorDapp in a way that allows it to be easily extended will likely require a reasonable amount of thought. It largely depends on what a CorDapp maintainer is trying to achieve. Providing a way for developers to extend a CorDapp so that they can send data to external systems or add their own logging should pose no problems. On the other hand, allowing the contents of a transaction to be altered or who it is sent to will require more thought to ensure that a CorDapp is not misused. This is a subject I hope to explore a bit further in future posts.

For the purpose of this post, we will look at the simpler option. Let’s jump right in since there has a been a whole lot of text so far and no code. Below is the SendMessageFlow that will act as the “base” Flow that will be extended in a later section:

@InitiatingFlow
open class SendMessageFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  open fun preTransactionBuild() {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun preSignaturesCollected(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun postSignaturesCollected(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun postTransactionCommitted(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  @Suspendable
  final override fun call(): SignedTransaction {
    logger.info("Started sending message ${message.contents}")
    preTransactionBuild()
    val tx = verifyAndSign(transaction())
    preSignaturesCollected(tx)
    val sessions = listOf(initiateFlow(message.recipient))
    val stx = collectSignature(tx, sessions)
    postSignaturesCollected(stx)
    return subFlow(FinalityFlow(stx, sessions)).also {
      logger.info("Finished sending message ${message.contents}")
      postTransactionCommitted(it)
    }
  }

  // collectSignature

  // verifyAndSign

  // transaction
}

I have removed a few of the functions so we can focus on what is important.

The first and sometimes important step to allow this class to be extended is the fact it is open. This is more of a Kotlin thing rather than Java since all classes in Kotlin are final by default. If you are writing this in Java then just ignore the last few sentences!

Following on from that, there are a series of functions that are available to be overridden. Each function has been placed in an appropriate place inside the main execution of the Flow. They will then be called when the Flow runs. For now, they have been given empty implementations since they provide no use to the CorDapp developer.

In regards to the open functions. You can name them or place them wherever you want. These are functions that I thought could be useful for developers wanting to add extra traceability over what the base app provides.

Digging down into a bit more detail. The call function has been made final (the same as in Java) to prevent the whole contents of the Flow from being overridden. If anyone wants to take your Flow and completely replace its “main” functionality, then what is the point? To me, it seems kind of dodgy. To remove that possibility making it final is a smart move.

Later on, we will look at how this Flow can be subclassed.

Below is the SendMessageResponder that interacts with the SendMessageFlow. It follows the same concepts as above and therefore I will only show it as a reference for later:

@InitiatedBy(SendMessageFlow::class)
open class SendMessageResponder(private val session: FlowSession) : FlowLogic<Unit>() {

  open fun postTransactionSigned(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun postTransactionCommitted(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  @Suspendable
  final override fun call() {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {}
    })
    postTransactionSigned(stx)
    val committed = subFlow(
      ReceiveFinalityFlow(
        otherSideSession = session,
        expectedTxId = stx.id
      )
    )
    postTransactionCommitted(committed)
  }
}

Extending an existing Initiating Flow

In this section, we get to see how the developer can make use of the work done on the previous Flow. It already has all the needed functionality. The only thing that is missing is the small amount of extra traceability that the developer wants to add in. Thanks to the functions added to the base Flow. This should cause no problems.

Let’s start with extending an Initiating Flow. The requirements for doing so are as follows:

Extend the base @InitiatingFlow
Do not add @InitiatingFlow to the new Flow (errors will occur if you do)
Reference the base Flow’s constructor (super in Java)
Override any desired functions
Call the new Flow instead of the base Flow

After reading that list, you might have realised that this is pretty much a description of inheritance in Object Oriented languages (like Kotlin and Java). There might be more going on inside Corda to allow this to work but from your perspective, you are just writing normal Object Oriented code like usual.

Taking these requirements we can see what an extended Flow might look like:

@StartableByRPC
class CassandraSendMessageFlow(private val message: MessageState) :
  SendMessageFlow(message) {

  override fun preTransactionBuild() {
    serviceHub.cordaService(MessageRepository::class.java).save(
      message,
      sender = true,
      committed = false
    )
    logger.info("Starting transaction for message: $message")
  }

  override fun preSignaturesCollected(transaction: SignedTransaction) {
    val keys = transaction.requiredSigningKeys - ourIdentity.owningKey
    logger.info("Collecting signatures from $keys for transaction for message: $message")
  }

  override fun postSignaturesCollected(transaction: SignedTransaction) {
    logger.info("Collected signatures for transaction for message: $message")
  }

  override fun postTransactionCommitted(transaction: SignedTransaction) {
    serviceHub.cordaService(MessageRepository::class.java).save(
      message,
      sender = true,
      committed = true
    )
    logger.info("Committed transaction for message: $message")
  }
}

I have left in all the noisy functions that implement the extra traceability I was talking about, but that is due to how empty the class would be without them. Since call does not need to be implemented. This Flow only needs to override the open functions. To be honest, it doesn’t need to override them at all, they are optional. If desired, this Flow could override a single function and then be left empty.

Have all the requirements listed above been met?

CassandraSendMessageFlow extends SendMessageFlow
There is no @InitiatingFlow in sight
In Kotlin you must call the super constructor anyway, so that’s done
In this scenario, all the functions have overridden
We haven’t got this far

Ok, so that is 4/5 so far. That’s a pretty good start. To cross off the last item on the list we need to see how it’s called. Below are snippets that call the base SendMessageFlow and the CassandraSendMessageFlow extending Flow.

Starting with SendMessageFlow:

proxy.startFlow(::SendMessageFlow, messageState)

Followed by CassandraSendMessageFlow:

proxy.startFlow(::CassandraSendMessageFlow, messageState)

Notice the difference? In this scenario, only the name of the Flow has changed. Nothing else.

Both snippets are completely valid. Calling the original SendMessageFlow is still allowed. Remember from our perspective, it is just a normal Object Oriented code. It won’t have the fancy extra code added to the extending Flow but it will still execute without issues. Completing this step meets the last requirement for extending an @InitiatingFlow.

Before we end this section, here is an important piece of information to remember from the Corda documentation:

You must ensure the sequence of sends/receives/subFlows in a subclass are compatible with the parent.

I will put this into all of the following sections since failing to follow this will cause your Flows to fail.

Extending a Responder Flow

Extending a Responder Flow works in a very similar way to extending an @InitiatingFlow Flow. The only difference is how it is called. As stated in the documentation:

Corda would detect that both BaseResponder and SubResponder are configured for responding to Initiator. Corda will then calculate the hops to FlowLogic and select the implementation which is the furthest distance, ie: the most subclassed implementation.

The statement, “most subclassed” is the important takeaway from this text. Therefore from a developer’s viewpoint, all they need to do is extend the external base Responder Flow and that’s it. I quite liked previous the requirements list, so let’s go through another one for extending Responder Flows:

Extend the base @InitiatedBy / Responder Flow
Add @InitiatedBy to the new Flow
Reference the base Flow’s constructor (super in Java)
Override any desired functions

If you are vigilant, you might have noticed that there is no mention of how to call it. The extending Responder Flow does not need to be called or referenced anywhere else. Corda will do the work to route everything to the right location.

Just to be sure, let’s have a quick look at an example:

@InitiatedBy(SendMessageFlow::class)
class CassandraSendMessageResponder(session: FlowSession) :
  SendMessageResponder(session) {

  override fun postTransactionSigned(transaction: SignedTransaction) {
    val message = transaction.coreTransaction.outputsOfType<MessageState>().single()
    logger.info("Signed transaction for message: $message")
  }

  override fun postTransactionCommitted(transaction: SignedTransaction) {
    val message = transaction.coreTransaction.outputsOfType<MessageState>().single()
    serviceHub.cordaService(MessageRepository::class.java).save(
      message,
      sender = false,
      committed = true
    )
    logger.info("Committed transaction for message: $message")
  }
}

Furthermore, let’s look back at the statement “most subclassed” again. The CassandraSendMessageResponder is a subclass of SendMessageResponder and is therefore chosen by Corda to handle requests from the Initiating Flow. But, this could be taken a step further. If there was another class, say SuperSpecialCassandraSendMessageResponder, this Flow is now what Corda will start using. Although I do find this sort of scenario somewhat unlikely at the moment, it is definitely worth knowing about.

Copying and pasting this statement again so you don’t forget:

You must ensure the sequence of sends/receives/subFlows in a subclass are compatible with the parent.

Overriding a Responder Flow

This is purposely a separate section. Here we will talk specifically about overriding a Responder Flow rather than extending one. Why would you do this and what is the difference? Answering the first question, a developer may want to write a Responder Flow that greatly diverges from the original base Flow but still needs to interact with the specific Initiating Flow provided by an external CorDapp. To achieve this they can override the Flow. Another word to describe this could be “replace”. The original base Flow is completely replaced by the overriding Flow. There is no involvement of extension in this situation.

I think the Corda documentation’s wording on this subject is quite good:

Whilst the subclassing approach is likely to be useful for most applications, there is another mechanism to override this behaviour. This would be useful if, for example, a specific CordApp user requires such a different responder that subclassing an existing flow would not be a good solution.

Hopefully, this extract along with my earlier description will clarify the difference between extending and overriding Responder Flows.

So, what might an overriding Flow look like? Well, anything you want really, within reason. Maybe it might look like the below, although I doubt it:

@InitiatedBy(SendMessageFlow::class)
class OverridingResponder(private val session: FlowSession) :
  FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {}
    })
    logger.info("Screw the original responder. I'll build my own responder... with blackjack and hookers!")
    subFlow(
      ReceiveFinalityFlow(
        otherSideSession = session,
        expectedTxId = stx.id
      )
    )
  }
}

Since this Flow is completely replacing the original base Flow, it will look just like a normal Responder Flow. Since, well, it is one. That means it has @InitiatedBy referencing the Initiating Flow, extends FlowLogic and implements the call function.

Just putting this here one last time:

You must ensure the sequence of sends/receives/subFlows in a subclass are compatible with the parent.

This is even more prevalent here than in the previous sections. Since the whole call function is being overridden you must make sure that every send and receive is in the right place so interactions with the Initiating Flow run without errors.

Configuration wise there is a bit more to do than with extending a Flow. In this situation, we are trying to completely replace a Responder with another. To do so, we need a way to tell the node to redirect interactions from an Initiating Flow to a new overriding Responder Flow. Corda provides a way to do just that.

To specify the redirection add the following to your node.conf:

flowOverrides {
  overrides=[
    {
      initiator="com.lankydanblog.tutorial.base.flows.SendMessageFlow"
      responder="com.lankydanblog.tutorial.cassandra.flows.OverridingResponder"
    }
  ]
}

Obviously, change the classes referenced to your own…

So what is going on here? The config says that the SendMessageFlow which normally interacts with SendMessageResponder will now route to OverridingResponder instead.

To make everything a bit easier as well, the Cordform plugin provides the flowOverride method as part of deployNodes. This will then generate the configuration block above for you. For the example above, the following code was used:

node {
  name "O=PartyA,L=London,C=GB"
  p2pPort 10002
  rpcSettings {
    address("localhost:10006")
    adminAddress("localhost:10046")
  }
  rpcUsers = [[user: "user1", "password": "test", "permissions": ["ALL"]]]
  cordapp(project(':cordapp-contracts-states'))
  cordapp(project(':cordapp'))
  cordapp(project(':cordapp-extended-cassandra'))
  // the important part
  flowOverride("com.lankydanblog.tutorial.base.flows.SendMessageFlow",
    "com.lankydanblog.tutorial.cassandra.flows.OverridingResponder")
}

Now, after deployNodes has run and you have started your node, any requests coming from SendMessageFlow or any of its subclasses will now route communication to the OverridingResponder.

Conclusion

One of the handy features that Corda 4 provides is the ability to customise Flows from third-party CorDapps (or your own). This is done by two methods, extending or overriding.

Extending would be my first choice between the two but it does require a bit more effort on the side of the CorDapp developer. They must provide enough avenues for customisation without relinquishing control of the original functionality of their Flows. Providing little customisation might not deter other developers from using their CorDapp. But, developers could become unhappy with the lack of control of their own application. It is a slippery slope to control original intent with routes for customisation. On the other hand, actually extending a Flow does not require much work, making it easier for developers to adopt and adapt external Flows.

Overriding, on the other hand, requires no work for a CorDapp developer and instead everything is put onto the developer leveraging external Responder Flows. That is because the existing Flow is pretty much being thrown away and the only reference back to the original implementation is the link to the Initiating Flow.

By embracing both the extending and overriding of Flows, CorDapp developers will be able to leverage external CorDapps while still providing enough customisation to fulfil all business requirements they might have. As time progresses, developers will drive the adoption of reusing existing CorDapps as they provide access to additional customisation, soon taking the same position as Open Source libraries that we all already leverage in any work we do.

The code used in this post can be found on my GitHub. It contains the code for CassandraSendMessageFlow which sets up a connection to an external Cassandra database to save tracing style data. It also contains another module that sends HTTP requests as part of its extension of the base Flows. If you are still curious after reading this post, this repository might help.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Asynchronous RDBMS access with Spring Data R2DBC

Sat, 16 Feb 2019 00:00:00 GMT

Not too long ago, a reactive variant of the JDBC driver was released. Known as R2DBC. It allows data to be streamed asynchronously to any endpoints that have subscribed to it. Using a reactive driver like R2DBC together with Spring WebFlux allows you to write a full application that handles receiving and sending of data asynchronously. In this post, we will focus on the database. From connecting to the database and then finally saving and retrieving data. To do this, we will be using Spring Data. As with all Spring Data modules, it provides us with out of the box configuration. Decreasing the amount of boilerplate code that we need to write to get our application setup. On top of that, it provides a layer upon the database driver that makes doing the simple tasks easier and the more difficult tasks a little less painful.

For the content of this post, I am making use of a Postgres database. At the time of writing only Postgres, H2 and Microsoft SQL Server have their own implementations of R2DBC drivers.

I have previously written two posts about reactive Spring Data libraries, one on Mongo and another about Cassandra. You might have noticed that neither of these databases are RDBMS databases. Now there are other reactive drivers available for a long time (I wrote the Mongo post nearly 2 years ago) but at the time of writing a reactive driver for a RDBMS database is still a pretty new thing. This post will follow a similar format to those.

Furthermore, I have also written a post about using Spring WebFlux which I mentioned in the introduction. Feel free to have a look at that if you are interested in producing a fully reactive web application.

Dependencies

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.2.0.M3</version>
</parent>
<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter</artifactId>
  </dependency>
  <dependency>
    <groupId>org.springframework.data</groupId>
    <artifactId>spring-data-r2dbc</artifactId>
    <version>1.0.0.M2</version>
  </dependency>
  <dependency>
    <groupId>io.r2dbc</groupId>
    <artifactId>r2dbc-postgresql</artifactId>
    <version>0.8.0.M8</version>
  </dependency>
  <dependency>
    <groupId>io.projectreactor</groupId>
    <artifactId>reactor-core</artifactId>
  </dependency>
</dependencies>

<repositories>
  <repository>
    <id>repository.spring.milestone</id>
    <name>Spring Milestone Repository</name>
    <url>http://repo.spring.io/milestone</url>
  </repository>
</repositories>

There are a few things to point out here.

The more you use Spring Boot, the more you will get used to importing a single spring-boot-starter dependency for the cool thing that you want to do. For example, I hoped that there would have been a spring-boot-starter-r2dbc dependency, but unfortunately, there is not one. Yet. Simply put, this library is on the newer side and at the time of writing, does not have its own Spring Boot module that contains any dependencies it needs along with faster setup via auto-configuration. I am sure these things will come at some point and make setting up a R2DBC driver even easier.

For now, we will need to fill in a few extra dependencies manually.

Furthermore, the R2DBC libraries only have Milestone releases (more proof of them being new) so we need to make sure we bring in the Spring Milestone repository. I will probably need to update this post in the future when it gets a release version.

Connecting to the database

Thanks to Spring Data doing a lot of the work for us, the only Bean that needs to be created manually is the ConnectionFactory that contains the database’s connection details:

@Configuration
@EnableR2dbcRepositories
class DatabaseConfiguration(
  @Value("\${spring.data.postgres.host}") private val host: String,
  @Value("\${spring.data.postgres.port}") private val port: Int,
  @Value("\${spring.data.postgres.database}") private val database: String,
  @Value("\${spring.data.postgres.username}") private val username: String,
  @Value("\${spring.data.postgres.password}") private val password: String
) : AbstractR2dbcConfiguration() {

  override fun connectionFactory(): ConnectionFactory {
    return PostgresqlConnectionFactory(
      PostgresqlConnectionConfiguration.builder()
        .host(host)
        .port(port)
        .database(database)
        .username(username)
        .password(password).build()
    )
  }
}

The first thing to notice here is the extension of AbstractR2dbcConfiguration. This class contains a load of Beans that we no longer need to manually create. Implementing connectionFactory is the only requirement of the class as it is required to create the DatabaseClient Bean. This sort of structure is typical of Spring Data modules so it feels quite familiar when trying out a different one. Furthermore, I’d expect this manual configuration to be removed once auto-configuration is available and be solely driven via the application.properties.

I have included the port property here, but if you have not played around with your Postgres configuration then you can rely on the default value of 5432.

The four properties: host, database, username and password defined by the PostgresqlConnectionFactory are the bare minimum to get it working. Any less and you will experience exceptions during startup.

Using this configuration, Spring is able to connect to a running Postgres instance.

The final piece of noteworthy information from this example is the use of @EnableR2dbcRepositories. This annotation instructs Spring to find any repository interfaces that extend Spring’s Repository interface. This is used as the base interface for instrumenting Spring Data repositories. We will look at this a little closer in the next section. The main piece of information to take away from here is that you need to use the @EnableR2dbcRepositories annotation to fully leverage Spring Data’s capabilities.

Creating a Spring Data Repository

As touched on above, in this section we will look at adding a Spring Data Repository. These repositories are a nice feature of Spring Data, meaning that you don’t need to write out a load of extra code to simply write a query. Unfortunately, at least for now, Spring R2DBC cannot infer queries in the same way that other Spring Data modules currently do (I am sure this will be added at some point). This means that you will need to use the @Query annotation and write the SQL by hand. Let’s take a look:

@Repository
interface PersonRepository : R2dbcRepository<Person, Int> {

  @Query("SELECT * FROM people WHERE name = $1")
  fun findAllByName(name: String): Flux<Person>

  @Query("SELECT * FROM people WHERE age = $1")
  fun findAllByAge(age: Int): Flux<Person>
}

This interface extends R2dbcRepository. This in turn extends ReactiveCrudRepository and then down to Repository. ReactiveCrudRepository provides the standard CRUD functions and from what I understand, R2dbcRepository does not provide any extra functions and is instead an interface created for better situational naming.

R2dbcRepository takes in two generic parameters, one being the entity class that it takes as input and produces as output. The second being the type of the Primary Key. Therefore in this situation, the Person class is being managed by the PersonRepository (makes sense) and the Primary Key field inside Person is an Int.

The return types of functions in this class and the ones provided by ReactiveCrudRepository are Flux and Mono (not seen here). These are Project Reactor types that Spring makes use of as the default Reactive Stream types. Flux represents a stream of multiple elements whereas a Mono is a single result.

Finally, as I mentioned before the example, each function is annotated with @Query. The syntax is quite straight forward, with the SQL being a string inside the annotation. The $1 ($2, \$3, etc… for more inputs) represents the value input into the function. Once you have done this, Spring will handle the rest and pass the input(s) into their respective input parameter, gather the results and map it to the repository’s designated entity class.

A very quick look at the entity

Not going to say much here but simply show the Person class used by the PersonRepository.

@Table("people")
data class Person(
  @Id val id: Int? = null,
  val name: String,
  val age: Int
)

Actually, there is one point to make here. id has been made nullable and provided a default value of null to allow Postgres to generate the next suitable value itself. If this is not nullable and an id value is provided, Spring will actually try to run an update instead of an insert upon saving. There are other ways around this, but I think this is good enough.

This entity will map to the people table defined below:

CREATE TABLE people (
  id SERIAL PRIMARY KEY, 
  name VARCHAR NOT NULL, 
  age INTEGER NOT NULL
);

Seeing it all in action

Now let’s have a look at it actually doing something. Below is some code that inserts a few records and retrieves them in a few different ways:

@SpringBootApplication
class Application : CommandLineRunner {

  @Autowired
  private lateinit var personRepository: PersonRepository

  override fun run(vararg args: String?) {
    personRepository.saveAll(
      listOf(
        Person(name = "Dan Newton", age = 25),
        Person(name = "Laura So", age = 23)
      )
    ).log().subscribe()
    personRepository.findAll().subscribe { log.info("findAll - $it") }
    personRepository.findAllById(Mono.just(1)).subscribe { log.info("findAllById - $it") }
    personRepository.findAllByName("Laura So").subscribe { log.info("findAllByName - $it") }
    personRepository.findAllByAge(25).subscribe { log.info("findAllByAge - $it") }
  }
}

One thing I will mention about this code. There is a very real possibility that it executes without actually inserting or reading some of the records. But, when you think about it. It makes sense. Reactive applications are meant to do things asynchronously and therefore this application has started processing the function calls in different threads. Without blocking the main thread, these asynchronous processes might never fully execute. For this reason, there are some Thread.sleep calls in this code, but I removed them from the example to keep everything tidy.

The output for running the code above would look something like the below:

2019-02-11 09:04:52.294  INFO 13226 --- [           main] reactor.Flux.ConcatMap.1                 : onSubscribe(FluxConcatMap.ConcatMapImmediate)
2019-02-11 09:04:52.295  INFO 13226 --- [           main] reactor.Flux.ConcatMap.1                 : request(unbounded)
2019-02-11 09:04:52.572  INFO 13226 --- [actor-tcp-nio-1] reactor.Flux.ConcatMap.1                 : onNext(Person(id=35, name=Dan Newton, age=25))
2019-02-11 09:04:52.591  INFO 13226 --- [actor-tcp-nio-1] reactor.Flux.ConcatMap.1                 : onNext(Person(id=36, name=Laura So, age=23))
2019-02-11 09:04:52.591  INFO 13226 --- [actor-tcp-nio-1] reactor.Flux.ConcatMap.1                 : onComplete()
2019-02-11 09:04:54.472  INFO 13226 --- [actor-tcp-nio-2] com.lankydanblog.tutorial.Application    : findAll - Person(id=35, name=Dan Newton, age=25)
2019-02-11 09:04:54.473  INFO 13226 --- [actor-tcp-nio-2] com.lankydanblog.tutorial.Application    : findAll - Person(id=36, name=Laura So, age=23)
2019-02-11 09:04:54.512  INFO 13226 --- [actor-tcp-nio-4] com.lankydanblog.tutorial.Application    : findAllByName - Person(id=36, name=Laura So, age=23)
2019-02-11 09:04:54.524  INFO 13226 --- [actor-tcp-nio-5] com.lankydanblog.tutorial.Application    : findAllByAge - Person(id=35, name=Dan Newton, age=25)

A few things to take away here:

onSubscribe and request occur on the main thread where the Flux was called from. Only saveAll outputs this since it has included the log function. Adding this to the other calls would have lead to the same result of logging to the main thread.
The execution contained within the subscribe function and the internal steps of the Flux are ran on separate threads.

This is not anywhere close to a real representation of how you would use Reactive Streams in an actual application but hopefully demonstrates how to use them and gives a bit of insight into how they execute.

Conclusion

In conclusion, Reactive Streams have come to some RDBMS databases thanks to the R2DBC driver and Spring Data that builds a layer on top to make everything a bit tidier. By using Spring Data R2DBC we are able to create a connection to a database and start querying it without the need of to much code. Although Spring is already doing a lot for us, it could be doing more. Currently, it does not have Spring Boot auto-configuration support. Which is a bit annoying. But, I am sure that someone will get around to doing it soon and make everything even better than it already is.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Testing exceptions in Kotlin with `assertFailsWith`

Sat, 26 Jan 2019 00:00:00 GMT

I wanted to write this short post to highlight the assertFailsWith function available to Kotlin that makes testing exceptions a bit easier. Testing exceptions isn’t something fancy or new to JVM languages (from now on I will use Java for comparisons) but Kotlin comes with the nice extra benefit of providing this functionality as part of its standard library. Comparing this to Java, you are likely to bring AssertJ into the mix to achieve similar results.

The main purpose of this post is to make you aware of the assertFailsWith function. I personally did not know it existed for a while and defaulted to depending on AssertJ. Not that I have anything against AssertJ that is. There are many other features that the library provides but for this specific instance it might be possible to remove it (assuming you are not using it for anything else that is).

What is good about assertFailsWith and AssertJ in general? It provides better exception testing than the simple constructs that JUnit provides. More precisely, it allows you to specify which part of your test that you expect an exception to be thrown, instead of declaring that an exception will arise somewhere in the code. This could lead to exceptions being incorrectly swallowed by test at an incorrect point and tricking you into thinking it is working as you think it should.

Now I have that brief point out of the way, let’s get on with the main content of this post. Below is what assertFailsWith looks like inside a test:

@Test
fun `calling hereIsAnException will return an exception no matter what`() {
  assertFailsWith<IllegalArgumentException> { hereIsAnException() }
}

In this example, hereIsAnException is placed inside the body of assertFailsWith, who checks that an IllegalArgumentException is thrown. If one is not raised, then the assertion will fail. If one does occur, then the assertion will pass and the exception is caught.

Catching the exception allows the execution of the test code to continue if needed as well as allowing you to make further assertions on the state of the exception.

For example, is it a wrapper around another exception (what is the type of its cause property)?

@Test
fun `original cause for exception was IndexOutOfBoundsException`() {
  val exception = assertFailsWith<IllegalArgumentException> { hereIsAnException() }
  assertTrue(exception.cause is IndexOutOfBoundsException)
}

Is the message what you expect (not the most sturdy of checks)?

@Test
fun `exception has the correct message`() {
  val exception = assertFailsWith<IllegalArgumentException> { hereIsAnException() }
  assertEquals("I am a failure...", exception.message)
}

Only exceptions that are of the same type or sub type as specified by assertFailsWith will be caught. Any others will cause the test to to fail. Since it catches sub types, please don’t go around just specifying Exception or RuntimeException. Try to be precise so your tests are as useful as possible.

As touched on earlier, assertFailsWith will only catch an exception that is thrown within the body of the function. Therefore if this was written instead:

@Test
fun `calling hereIsAnException will return an exception no matter what`() {
  hereIsAnException()
  assertFailsWith<IllegalArgumentException> { hereIsAnException() }
}

The test would fail. hereIsAnException has thrown an exception, which has not been caught and leads to the test failing. I believe this is the best part of this sort of function over the previous ways this used to be done (e.g. asserting inside @Test that an exception would occur).

@Test
fun `calling hereIsAnException will return an exception no matter what`() {
  assertFailsWith<IllegalArgumentException>("This should throw an illegal argument exception") { 
    hereIsANormalReturnValue() 
  }
}

I personally have never really used the message part of an assertion. Maybe you do, so, I thought I’d at least let you know.

Before I wrap up the little amount of content in this post, let’s have a quick look at AssertJ so that we can draw a comparison between the two. Again, this is only for the case of catching exceptions which is only a small part of what AssertJ provides.

@Test
fun `calling hereIsAnException will return an exception no matter what`() {
  assertThatExceptionOfType(IllegalArgumentException::class.java).isThrownBy {
    hereIsAnException()
  }
}

This is slightly more “verbose” than the assertFailsWith version. But, that is made up for with the plethora of functions that AssertJ provides that makes any further checking of the returned exception much easier. More precisely, when using assertFailsWith I needed to write another assertion to check the message. In AssertJ this is just a function chained onto the end of the previous call.

To conclude, assertFailsWith is a nice little function to use in testing to ensure that a piece of code throws a specific type of exception. It is built in to the Kotlin standard library which removes the need to bring in an extra dependency to your project. That being said, it is a relatively simple function and does not bring the sort of functionality that a library like AssertJ would. It is likely to suffice until you want to write tests that contain a wide range or assertions as this is the point where it can get messy.

The official docs for assertFailsWith can be found here if you are interested Kotlin Docs - assertFailsWith.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Privately scoped variable in `when` block

Thu, 03 Jan 2019 00:00:00 GMT

Super short post, on a change introduced in Kotlin 1.3 (yes I know it has been out for a while now). We will take a quick look at capturing the subject of a when block into a scoped variable. This is a quality of life improvement that saves a line or so of code while making the role of the variable clearer.

Below is what you would write before the change:

val enum = myClass.enum
when (enum) {
  MyEnum.ONE -> println(enum.propertyOne)
  MyEnum.TWO -> println(enum.propertyTwo)
  else -> println(enum)
}

You can now write this instead:

when (val enum: MyEnum = myClass.enum) {
  MyEnum.ONE -> println(enum.propertyOne)
  MyEnum.TWO -> println(enum.propertyTwo)
  else -> println(enum)
}

The instantiation of the enum variable is merged with the declaration of the when block. enum is now scoped to the when block and cannot be accessed outside of it. Not a massive change, but it does makes the code look a little tidier.

Let’s look at one more example:

val obj = someObject()
when(obj) {
  is String -> println("This is a string and it says $obj")
  is Number -> println("This is a number and its value is $obj")
  else -> println("I don't know what $obj is")
}

Becomes:

when(val obj = someObject()) {
  is String -> println("This is a string and it says $obj")
  is Number -> println("This is a number and its value is $obj")
  else -> println("I don't know what $obj is")
}

Not really anything else to say on this subject as there isn’t much to talk about in the first place. You probably get the picture, the example from the Kotlin 1.3 Release Notes might make sense to you if I have failed to explain it myself.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Validating external data with an Oracle

Tue, 01 Jan 2019 00:00:00 GMT

I hang out in the Corda Slack channel quite a lot and try to answer questions when I can. A reasonable number of questions I have attempted to answer are related to Oracles. More specifically, when to use one. I feel like I can answer that, “Use an Oracle when you need to validate external data that can change frequently”. I probably wrote an answer similar to that at some point. What I could not do though… Was tell someone how to implement one. Therefore, to correct that. I have written this post to learn how to implement one myself and share that knowledge with you and my future self.

When to use an Oracle

Let’s start with expanding on when to use an Oracle. As I touched on a minute ago, you should use an Oracle when you need to validate external data that can change frequently. This could be data such as exchange rates, stock prices or even whether my blog is currently up or down (although I haven’t seen it be down yet!). I think the frequently part is important here. If data seldomly changes, it might be feasible to validate some data against an attachment containing the same sort of values the Oracle would be retrieving themselves. That is why validating data such as exchange rates, in my opinion, should only be done by an Oracle. All that being said, it really comes down to your particular use-case.

How to use an Oracle

How does the Oracle do this validation? Well, that is up to you. But, it will likely follow these steps:

Receive data from a node
Retrieve external data
Validate the received data against the external data
Provide a signature for the transaction

These are the steps that I think most Oracle implementations will comprise of. More steps could be added in and the validation that is done could be as complex or simple as the use-case demands. Although more steps could be added, I really doubt that there would be much use out of an Oracle that excluded any of the steps shown above.

All the steps shown above only show the process from the side of the Oracle. There is a bit more going on so I think a good diagram will help us out here. It will also go on to present the example that I will use for this post.

Quite a few of these steps are generic ones that are going to be made in whatever Flow you put together. In this section, I will expand and show the code involved to implement the Flow shown in the diagram. Therefore it is worth having a reasonable look at it… I also spent a lot of time making that look nice, so please look at it!!

Oh, another point before I continue. I want to highlight how helpful putting together sequence diagrams to model Corda Flows is. It really highlights who is involved, how many network hops need to be made and how much work each participant does. Furthermore, they go a good way of explaining what is going on to people who are only interested in the higher level processes that you are architecting and/or implementing.

Client / Not the Oracle side

As I mentioned before, some of the code here is generic code that you are likely to put into any Flow that you write. I have shown it all so there is no ambiguity around what is happening, but I will only expand on points that need to be highlighted as they contain code specific to interacting with an Oracle.

@InitiatingFlow
@StartableByRPC
class GiveAwayStockFlow(
  private val symbol: String,
  private val amount: Long,
  private val recipient: String
) :
  FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val recipientParty = party()
    val oracle = oracle()
    val transaction =
      collectRecipientSignature(
        verifyAndSign(transaction(recipientParty, oracle)),
        recipientParty
      )
    val allSignedTransaction = collectOracleSignature(transaction, oracle)
    return subFlow(FinalityFlow(allSignedTransaction))
  }

  private fun party(): Party =
    serviceHub.networkMapCache.getPeerByLegalName(CordaX500Name.parse(recipient))
      ?: throw IllegalArgumentException("Party does not exist")

  private fun oracle(): Party = serviceHub.networkMapCache.getPeerByLegalName(
    CordaX500Name(
      "Oracle",
      "London",
      "GB"
    )
  )
    ?: throw IllegalArgumentException("Oracle does not exist")

  @Suspendable
  private fun collectRecipientSignature(
    transaction: SignedTransaction,
    party: Party
  ): SignedTransaction {
    val signature = subFlow(
      CollectSignatureFlow(
        transaction,
        initiateFlow(party),
        party.owningKey
      )
    ).single()
    return transaction.withAdditionalSignature(signature)
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }

  private fun transaction(recipientParty: Party, oracle: Party): TransactionBuilder =
    TransactionBuilder(notary()).apply {
      val priceOfStock = priceOfStock()
      addOutputState(state(recipientParty, priceOfStock), StockContract.CONTRACT_ID)
      addCommand(
        GiveAway(symbol, priceOfStock),
        listOf(recipientParty, oracle).map(Party::owningKey)
      )
    }

  private fun priceOfStock(): Double =
    serviceHub.cordaService(StockRetriever::class.java).getCurrent(symbol).price

  private fun state(party: Party, priceOfStock: Double): StockGiftState =
    StockGiftState(
      symbol = symbol,
      amount = amount,
      price = priceOfStock * amount,
      recipient = party
    )

  private fun notary(): Party = serviceHub.networkMapCache.notaryIdentities.first()

  @Suspendable
  private fun collectOracleSignature(
    transaction: SignedTransaction,
    oracle: Party
  ): SignedTransaction {
    val filtered = filteredTransaction(transaction, oracle)
    val signature = subFlow(CollectOracleStockPriceSignatureFlow(oracle, filtered))
    return transaction.withAdditionalSignature(signature)
  }
  
  private fun filteredTransaction(
    transaction: SignedTransaction,
    oracle: Party
  ): FilteredTransaction =
    transaction.buildFilteredTransaction(Predicate {
      when (it) {
        is Command<*> -> oracle.owningKey in it.signers && it.value is GiveAway
        else -> false
      }
    })
}

@InitiatedBy(GiveAwayStockFlow::class)
class SendMessageResponder(val session: FlowSession) : FlowLogic<Unit>() {
  @Suspendable
  override fun call() {
    subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {}
    })
  }
}

First, let’s have a look at how the transaction is built:

private fun transaction(recipientParty: Party, oracle: Party): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    val priceOfStock = priceOfStock()
    addOutputState(state(recipientParty, priceOfStock), StockContract.CONTRACT_ID)
    addCommand(
      GiveAway(symbol, priceOfStock),
      listOf(recipientParty, oracle).map(Party::owningKey)
    )
  }

private fun priceOfStock(): Double =
  serviceHub.cordaService(StockRetriever::class.java).getCurrent(symbol).price

There is not too much different here from how I would create a transaction that does not involve an Oracle. The only two differences are, retrieving the stock price from an external source (hidden inside the StockRetriever service) and including the signature of the Oracle in the Command. These code additions line up with the reasons for using an Oracle. External data is included in the transaction and the Oracle needs to verify it is correct. To prove that the Oracle has deemed a transaction valid, we need its signature.

We will look closer at retrieving the external data separately.

Next up is collecting the recipients signature:

@Suspendable
private fun collectRecipientSignature(
  transaction: SignedTransaction,
  party: Party
): SignedTransaction {
  val signature = subFlow(
    CollectSignatureFlow(
      transaction,
      initiateFlow(party),
      party.owningKey
    )
  ).single()
  return transaction.withAdditionalSignature(signature)
}

Collecting the counterparty’s signature is not really an uncommon step of a Flow but what is done differently here is the use of CollectSignatureFlow rather than the CollectSignaturesFlow which is normally used (noticed the “s” missing in the middle). That is due to requiring the Oracle’s signature in the transaction. Calling the CollectSignaturesFlow will go off to retrieve signatures from all required signers, including the Oracle. This treats the Oracle like a “normal” participant. This is not what we want. Instead, we need to get the signature of the recipient and the Oracle individually and somewhat manually. The manual part is the use of transaction.withAdditionalSignature.

Now that the recipient has signed the transaction, the Oracle needs to sign it:

@Suspendable
private fun collectOracleSignature(
  transaction: SignedTransaction,
  oracle: Party
): SignedTransaction {
  val filtered = filteredTransaction(transaction, oracle)
  val signature = subFlow(CollectOracleStockPriceSignatureFlow(oracle, filtered))
  return transaction.withAdditionalSignature(signature)
}

private fun filteredTransaction(
  transaction: SignedTransaction,
  oracle: Party
): FilteredTransaction =
  transaction.buildFilteredTransaction(Predicate {
    when (it) {
      is Command<*> -> oracle.owningKey in it.signers && it.value is GiveAway
      else -> false
    }
  })

Before sending the transaction to the Oracle, it is recommended to filter it to remove any information that is not needed by the Oracle. This prevents information that shouldn’t be shared from being seen by the Oracle. Remember the Oracle is likely to be a node controlled by another organisation and is not a participant that you are attempting to share states and transactions with.

SignedTransaction provides the buildFilteredTransaction function that only includes objects that match the predicate passed in. In the example above it filters out everything but the GiveAway (command I created) command which must also have the Oracle as a signer.

This outputs a FilteredTransaction which is passed to CollectOracleStockPriceSignatureFlow:

@InitiatingFlow
class CollectOracleStockPriceSignatureFlow(
  private val oracle: Party,
  private val filtered: FilteredTransaction
) : FlowLogic<TransactionSignature>() {

  @Suspendable
  override fun call(): TransactionSignature {
    val session = initiateFlow(oracle)
    return session.sendAndReceive<TransactionSignature>(filtered).unwrap { it }
  }
}

All this code does is send the FilteredTransaction to the Oracle and awaits its signature. The code here could be put into the main Flow but it is quite nice to split code out when we can.

Finally, the TransactionSignature returned from the Oracle is added to the transaction in the same way the recipient’s signature was added earlier. At this point, the transaction is ready to be committed as all the required signers have done their part.

Oracle side

Now that we have covered the client side of the code, we need to have a look into how the Oracle validates the transaction. Below is the contents of the Oracle code:

@InitiatedBy(CollectOracleStockPriceSignatureFlow::class)
class OracleStockPriceSignatureResponder(private val session: FlowSession) : FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    val transaction = session.receive<FilteredTransaction>().unwrap { it }

    val key = key()

    val isValid = transaction.checkWithFun { element: Any ->
      when {
        element is Command<*> && element.value is GiveAway -> {
          val command = element.value as GiveAway
          (key in element.signers).also {
            validateStockPrice(
              command.symbol,
              command.price
            )
          }
        }
        else -> false
      }
    }

    if (isValid) {
      session.send(serviceHub.createSignature(transaction, key))
    } else {
      throw InvalidStockPriceFlowException("Transaction: ${transaction.id} is invalid")
    }
  }

  private fun key(): PublicKey = serviceHub.myInfo.legalIdentities.first().owningKey

  private fun validateStockPrice(symbol: String, price: Double) = try {
    serviceHub.cordaService(StockPriceValidator::class.java).validate(symbol, price)
  } catch (e: IllegalArgumentException) {
    throw InvalidStockPriceFlowException(e.message)
  }
}

Some of the code that should be here is hidden in the StockPriceValidator which retrieves the external stock price and compares it to the one passed to the Oracle. It doesn’t have much code and its validation is basic so I won’t elaborate on it. As it is short, I might as well show it now:

@CordaService
class StockPriceValidator(private val serviceHub: AppServiceHub) :
  SingletonSerializeAsToken() {

  fun validate(symbol: String, price: Double) =
    serviceHub.cordaService(StockRetriever::class.java).getCurrent(symbol).let {
      require(price == it.price) { "The price of $symbol is ${it.price}, not $price" }
    }
}

Back to the OracleStockPriceSignatureResponder. Firstly, receive is called to get the FilteredTransaction that the client sent. It is then checked using its checkWithFun function. This is a handy function that looks at each object and expects a Boolean in return. Using this, the transaction is deemed valid if all it contains are commands of GiveAway where the Oracle is the signer and most importantly checks that the external data contained in the command is correct. If you recall the code from earlier, the correct command and signers were passed in. The only remaining validation is on the external data. If that is all fine then the Oracle will accept the transaction and send its signature back to the client that requested it.

I chose to complete the validation via throwing exceptions (along with error messages) which are then propagated to the requesting party. I think this makes it easier on to understand what has gone wrong so it can be handled properly, rather than just a straight “failed validation” message. If the validation the Oracle is performing is complex, these error messages become even more valuable.

Retrieving external data

You should have seen the StockRetriever class pop up twice now. It was used in both the requesting party and the Oracle. I have shared this code between both types of nodes (normal nodes and Oracles) but this might not be suitable for your own use-case. Furthermore, how you choose to retrieve your external data is up to you, I am just providing a possible solution.

The code can be found below:

@CordaService
class StockRetriever(serviceHub: AppServiceHub) :
  SingletonSerializeAsToken() {

  private val client = OkHttpClient()
  private val mapper = ObjectMapper()

  fun getCurrent(symbol: String): Stock {
    val response = client.newCall(request(symbol)).execute()
    return response.body()?.let {
      val json = it.string()
      require(json != "Unknown symbol") { "Stock with symbol: $symbol does not exist" }
      val tree = mapper.readTree(json)
      Stock(
        symbol = symbol,
        name = tree["companyName"].asText(),
        primaryExchange = tree["primaryExchange"].asText(),
        price = tree["latestPrice"].asDouble()
      )
    } ?: throw IllegalArgumentException("No response")
  }

  private fun request(symbol: String) =
    Request.Builder().url("https://api.iextrading.com/1.0/stock/$symbol/quote").build()
}

The StockRetriever is a nice little service that uses an OkHttpClient (OkHttp) to make a HTTP request to an API (provided by IEX Trading using their Java library) that returns stock information when a stock symbol is provided. You can use whatever client you want to make the HTTP request. I saw this one in an example CorDapp and have taken it for my own. Personally, I’m too used to Spring so didn’t really know any clients other than their RestTemplate.

Once the response is returned, it is converted into a Stock object and passed back to the function’s caller. That’s all folks.

Conclusion

In conclusion, you should use an Oracle when your CorDapp requires frequently changing external data that needs to be validated before the transaction can be committed. Just like data held within states, external data is extremely important, probably the most important, as it is likely to determine the main contents of a transaction. Therefore all participants must feel comfortable that the data is correct and hasn’t just come out of thin air. To achieve this, an Oracle will also retrieve the external data and validate it against what the transaction says the data should be. At this point, the Oracle will either sign the transaction or throw an exception and deem it invalid. The implementation side of this is reasonably straightforward as there are not many steps that need to be taken. Retrieve the data, send a FilteredTransaction to the Oracle containing the data where it will be validated. Yes, as you have read this post, you will know there is a bit more to it. But, for a basic Flow that is pretty much it. As I said somewhere near the start, how the Oracle does its validation can be as simple or complicated as required. Although, I think most will follow the same sort of process shown here.

Now for the main conclusion… In conclusion, you now have the knowledge to answer questions in the slack channel about Oracles or know where to send them if you can’t!

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Throughput - A Corda story

Fri, 14 Dec 2018 00:00:00 GMT

I recently rolled off a project focusing on the performance of Corda for a particular use-case. The result of this project lead to us processing 115 million requests within a single day over a network of over 170 nodes. Furthermore, Corda was able to handle 6300 requests per second confirming that the highest requirement of the network was met. This is by far the largest Corda network that has been deployed and the highest throughput achieved to date. Proving that Corda can deliver in a very demanding environment.

The study was conducted by Accenture for DTCC, which also looked into another DLT platform, Digital Asset. Further information can be found in the press release.

DTCC Announces Study Results Demonstrating that DLT Can Support Trading Volumes in the US Equity… Groundbreaking study proves DLT can process more than 100 million trades per day New York/London/Hong…www.dtcc.com

In this post, I will use the experience I gained on this project to describe how you can also get the best out of Corda. I expect that there will be similar projects to what we did for DTCC coming up in the near future and I hope that the information here will help point other developers in the right direction.

So how did it go? Well, it wasn’t without its difficulties. But, I think we did well. More precisely, we proved that Corda can hit a high throughput if the network is thoughtfully architected and your CorDapps are well designed. Yes, I know that I’m basically saying if you do everything right then it will go well. It really is that important. As we tweaked our CorDapps throughout the project we found avenues to greatly improve the application’s performance. Making these changes allowed us to move closer and closer to the targets we were aiming for. But, none of this would have mattered if we had not architected our network in a particular way.

Corda Enterprise is needed to achieve the highest possible performance

This is the simplest way to increase your performance by a factor of 10 or by however many cores your machine has. Among other things, Corda Enterprise allows the number of Flow workers running within the node to increase from 1 to many. This affects the number of Flows that can run asynchronously within the node. But, this does not change the performance of individual Flows which run at the same speed on each version. Realistically, without Enterprise you will never achieve extremely high performance targets.

If achieving this sort of performance is not a requirement for your use-case, then the Open Source version will meet your needs. For example, we 100% needed to use Enterprise for our project since the volumes of requests that DTCC handle and the rate that they need to be processed is enormous. On the other hand, if we were dealing with the handling and processing of Loans. The rate that requests flow through a node would drop significantly compared to DTCC’s needs. In this situation, using Open Source would suffice.

Conveniently, Enterprise and Open Source code are compatible allowing you to switch over without too much pain. There are differences in deployment and it is extremely likely you will need to fiddle around with this side when making the change.

The compatibility between the Enterprise and Open Source versions will allow you to try both, allowing you to determine which best suits your needs. This allows you to start writing your application on Open Source until it is deemed necessary to switch to Enterprise.

Think about your Network

I really, really want to emphasise the importance of network architecture. I don’t even want to think about the performance we would have achieved if we stuck to our original design. We actually scrapped our original design since it was fundamentally flawed and would have prevented any hopes of reaching our goals. Personally, I think this section is half about Corda and half about architecting a good solution.

Sharding to improve performance at scale

At the time of writing this post, Corda does not support Load Balancing. Currently, a single node processes all of the work for the identity it represents. This is an area they are fully aware of and is something they have on their roadmap to work on in the near future. If this was there, then it might have been possible to simply rely on spinning up a ton of instances that support a single Corda node. This would have led to more compute power and therefore increasing throughput.

Due to load balancing not being ready and the fact that we had a single actor sitting in the middle of the network who was a massive bottleneck to the handling of requests, we had to approach the overall network design differently. To compensate, we had to think of a way to provide our own sort of horizontal scaling into the system as the single node sitting in the middle had to be removed. If these issues were not addressed, there was no way we could achieve the network throughput of 6300 trades per second.

Our solution? Sharding. We determined a way that this one actor could be logically split up into many, much smaller parts. Each handling requests in parallel to each other. This required a bit of extra logic to route the request to the right sharded node. But, this solution likely saved the project. We never tested the performance of the single bottleneck node, but, I am 100% sure we would not have reached out targets.

Below I have included two diagrams. One of an example process using the single node design and another of the sharded approach.

Single node

Sharded

I will let the diagrams speak for themselves. I will not delve any further into the implementation since that information is still confidential. There should be enough information there to understand why and what we did, but not how we achieved it.

As you can imagine the second design will yield a much higher throughput. It also comes with the benefit of scaling linearly as nodes are added to the network.

With the original design, the throughput might be acceptable with a small number of nodes. But, as soon as you hit larger numbers, say 100s, maybe even as small as 10, you are going to notice the levelling out of performance. This is completely due to the whole network relying on the rate at which the single bottleneck node can process requests.

Removing extra bottlenecks with multiple Notaries

Another area that will improve the network’s overall performance is the use of multiple Notaries. When the throughput of the network is already high, a single Notary will start to become the bottleneck of the workflow. By following the same idea as the previous section. A Notary can be sharded. Allowing each to process a smaller volume of transactions.

Every time I say “multiple Notaries”, I just feel like I need to clarify that I am not talking about a Notary Cluster.

I have already written a post, Increasing network throughput with multiple Notaries, covering this topic and rather than repeating myself, I will direct you there instead.

Increasing network throughput with multiple Notaries

Do you need a very high throughput Corda network? Has the network's throughput levelled out? Have you already squeezed…

Tweak those Cordapps

Onto Cordapps. There is a lot you can do to improve performance here. Most of it comes from trying to do as little as possible.

Do I need to do send all these transactions?
Does this other party really need to sign this transaction?
Do my transactions have too many states on them?
How many times are Flows jumping between the initiator and counterparties?

These are all questions that are important to the performance of your Flows. I’m sure there are other places to gain performance (one I will touch on later) but these are the only ones I can think of right now. I’m sure you got the picture though.

Let’s have a quick look at the last question.

How many times are Flows jumping between the initiator and counterparties?

This actually encompasses some of the other points I made. Anyway. Each time a jump across the network is made, the performance of the Flow goes down. It needs to travel from one Corda node to another and it likely needs to come back at some point. During this time, you are accumulating performance costs due to network latency and the process of checkpointing Flows to disk.

Network latency speaks for itself and doesn’t require further explanation. Checkpointing, on the other hand, requires a bit of fleshing out. Checkpointing is the process of serialising a Flow’s current execution so that it can be restarted from a specific point in case of failure. Doing so requires serialising the whole stack of the Flow which can be quite large and therefore a costly process to perform.

Taking this information into account, make sure you think about whether you really need to make these jumps. Try to keep them to a minimum. If you do this, I’m sure you will see your application’s performance increase.

Multi-threading? That is good for performance right?

Yes, yes, yes. Although, we didn’t measure the impact including multi-threading made I am sure it made a good improvement. But, be careful. If you don’t do it correctly then you might get bit in the arse. At of the time of writing, Corda does not support multi-threading within Flows. You will get some weird errors if you do. That being said, it is possible. You can do so from within a Corda Service which runs slightly outside of the territory of a Flow. By delegating some processing to a service, we were able to make use of threads to start new Flows which each run asynchronously, processing similar but separated tasks.

I have written about this in my earlier post Asynchronous Flow invocations with Corda Services which dives into this topic and to why you might end up falling over when trying this.

Asynchronous Flow invocations with Corda Services

How can I make my Flows faster? There's a good chance you have thought about this before if you have been working with…

What is it like using Corda

I found using Corda relatively simple. It does get more difficult as you try to implement more complex use-cases. But, for the most part, a lot of Flows can follow the same simple structure. Add some states to a transaction, verify it, get all required parties to sign it and commit the transaction.

As it gets more complicated, you need to keep in mind which party needs to do what. For example, spending cash. As an initiator, you can’t put other people’s cash states into a transaction. You need to send them some information and request them to add it to a transaction. Scenarios like this took me a while to get to grips with. As more developers spend time working with Corda, I am sure these concepts will become easier to understand. More examples will be published and knowledge of how to write good Flows will be distributed.

Furthermore, I stand by the Key Concepts that Corda have produced. Going through these and the documentation provided took me pretty far in my understanding of Corda.

Key concepts - R3 Corda V3.3 documentation

This section describes the key concepts and features of the Corda platform. It is intended for readers who are new to…docs.corda.net

Moving forward

Now, I don’t speak for Corda or R3, but since we worked closely with them throughout this project I can speak of possible improvements to the platform.

Making it easier to deploy multiple Corda nodes. R3 worked with us to produce a framework to deploy nodes more easily which can likely be adapted and generalised to work for a wider audience.
Performance. There were a few areas within the Corda code that could be tweaked to make way for good performance gains.
Better multi-threading. As I mentioned earlier, this can be done within Corda Services but it might be possible to shift some of this into Flows. Mainly focusing on starting multiple subFlows asynchronously and awaiting their completion.

Wrapping up

Towards the end of the project, it was definitely hectic but the massive performance increase that we were able to make within 1 month is crazy. Once we improved our CorDapps to eke every bit of performance out of them, our numbers went from “meh” to “wow”. Thankfully, we designed our network correctly to make these numbers possible. All the tweaking in the world would not have saved it if the network wasn’t put together in the way it was.

So, can you get good throughput with Corda? Yes. Yes, you can. Using Corda Enterprise will make higher performance targets more achievable and allow you to do so with less work on your end. But, that is not really the right mindset to be in. Using the information I covered in this post, you should have a better understanding of how to design a high performance Corda application or network.

Going forward, Corda’s performance is only going to get better. Combining that with a good idea of how to design your application, should allow your numbers to shoot through the roof.

Lastly, before I close this post, I just want to thank R3 and especially Stefano for working closely with us during this project.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Opinions and views found in my posts are my own and do not represent Accenture’s views on any subject.

Increasing network throughput with multiple Notaries

Sun, 18 Nov 2018 00:00:00 GMT

Do you need a very high throughput Corda network? Has the network’s throughput levelled out? Have you already squeezed out all the performance you could from other areas? If your answers to these questions are “yes”, then I may have some useful information for you. I have listed out these questions to decrease the chance of you prematurely optimising your Corda network/application. Switching to using multiple Notaries is only going to have a noticeable performance impact if it is one of the slowest parts involved in processing requests/transactions. It is highly likely that other areas need to be improved before looking into using multiple Notaries.

Before I continue. I really need to say this. I am not talking about using Notary Clusters in this post, which consist of Notaries that communicate with each other to reach consensus on whether states have been spent or not. I am talking about having multiple Notaries, each with their own identities, who only interact with the nodes that send them transactions to validate. This distinction needs to be made and should remove any confusion about exactly what I will be describing in this post.

At the time of writing, the current versions of Corda are:

Open Source 3.3
Enterprise 3.2

Why would I want to do this?

Ok, so. Let’s really dig into why you would want to use multiple Notaries. Diagrams do this best, so let’s use one:

This situation doesn’t look great. But, it might not actually be that bad. If the throughput of your network is not extremely high, this architecture should be able to handle the transactions passing through the Notary.

As mentioned in the introduction. It becomes an issue when the rate of transactions being sent to the Notary becomes very high. Once this point is reached, the Notary will start to lag behind. As it cannot validate the states in the transactions fast enough. If performance is important to the network, this is a good area to examine.

From a code perspective, this is the standard format that you have probably already been writing your CorDapps. You pick out a Notary based on particular criteria and send a transaction there. There might even be only a single Notary in the whole network that you have been dealing with. For example, in all my code examples I have produced before I have written code similar to the below that only relies on the single Notary in the network and blindly takes that one every time.

private fun notary(): Party = serviceHub.networkMapCache.notaryIdentities.first()

Switching to multiple Notaries

Moving from a network that relies on a single Notary to a design that consists of many, fundamentally, requires two things:

More than one Notary in the network
An algorithm to choose which Notary to send a transaction to

Furthermore, the chosen Notary for a transaction is then referenced by future transactions if consuming states. If you end up in a situation where input states from different Notaries are being consumed, then you must perform a Notary Change Transaction. I will cover this topic later on.

Below is how the previous design could be altered to use a few Notaries:

The best thing about this diagram is that it illustrates how simple it is to add another Notary to the network and redistribute the load among them. There is nothing stopping us from adding more and more Notaries to the network. But, there will be a point where adding more does not lead to a performance increase. This keeps coming back to what I have mentioned previously. That adding more Notaries will only increase throughput when the Notaries themselves are reaching saturation.

Choosing a Notary for issuance transactions

Below is a possible algorithm to choose which Notary to use:

private fun transaction(): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    addOutputState(message, MessageContract.CONTRACT_ID)
    addCommand(Send(), message.participants.map(Party::owningKey))
  }

private fun notary(): Party {
  val index = message.type.hashCode() % serviceHub.networkMapCache.notaryIdentities.size
  return serviceHub.networkMapCache.notaryIdentities.single { it.name.organisation == "Notary-$index" }
}

In this example, the transaction chooses the Notary to use based on the hashCode of one of the input state’s properties and the number of Notaries in the network.

How you choose the Notary could be as simple or complex as you need it to be. This will depend on requirements such as only a subset of Notaries being trusted for proposed transactions or resiliency to the Notaries in the network changing.

Choosing a Notary when consuming states from the same Notary

This is nice and simple… If all input states reference the same Notary. Below is what it looks like (this example only consumes a single input… because I am too lazy to write another version):

private fun transaction(response: MessageState): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    addInputState(message)
    addOutputState(response, MessageContract.CONTRACT_ID)
    addCommand(Reply(), response.participants.map(Party::owningKey))
  }

private fun notary(): Party = message.state.notary

As you can see, all the transaction does is retrieve the Notary that is related to the input state and uses it for itself. This information can be extracted because message is a StateAndRef and accessing its state property will return a TransactionState. Following this format. Creating new transactions that consume a state and produce a number of outputs is straightforward. This format is also valid for multiple input states. If, and only if, they all reference the same Notary.

So… With all this talk about input states with different Notaries. I should probably discuss it further.

Choosing a Notary when consuming states from different Notaries

Here we have to be careful or we will see errors like the one below:

java.lang.IllegalArgumentException: Input state requires notary "O=Notary-1, L=London, C=GB"
which does not match the transaction notary "O=Notary-0, L=London, C=GB".

The error shows that an input state does not have the same Notary as the transaction that contains it.

To solve this error we need to use a Notary Change Transaction. As per the docs:

A flow to be used for changing a state’s Notary. This is required since all input states to a transaction must point to the same notary.

I wanted to put that in there, just in case you think I am a lier!

The code to perform a Notary Change Transaction looks like the below:

@Suspendable
private fun notaryChange(
  message: StateAndRef<MessageState>,
  notary: Party
): StateAndRef<MessageState> =
  if (message.state.notary != notary) {
    subFlow(
      NotaryChangeFlow(
        message,
        notary
      )
    )
  } else {
    message
  }

I am sure you can figure out what is going on yourself, but to make myself feel smarter… I am going to tell you. message represents an input state and notary is the Notary that the new transaction will be using. If the Notaries are the same, then the state can be returned as nothing needs to be done to it. If they are indeed different, then call NotaryChangeFlow which takes in the two arguments passed into the original function. This will return a new StateAndRef which is then returned from the function.

The StateAndRef returned from this function can then be put into the transaction.

If you are not sure whether the states being passed into a transaction are from the same Notary, then I suggest sticking to the code in this section. Choose a Notary that the transaction will use, whether this is a specific one or one taken from the input states and perform a Notary Change Transaction on any that require it. For example, I think code similar to the below would make a generic and robust solution:

@Suspendable
private fun transaction(): TransactionBuilder {
  val messages = getMessageStates()
  val notary = notary()
  return TransactionBuilder(notary).apply {
    messages.forEach {
      addInputState(notaryChange(it, notary))
    }
    addCommand(
      Delete(),
      (messages.flatMap { it.state.data.participants }.toSet() + ourIdentity).map(Party::owningKey)
    )
  }
}

@Suspendable
private fun notaryChange(
  message: StateAndRef<MessageState>,
  notary: Party
): StateAndRef<MessageState> =
  if (message.state.notary != notary) {
    subFlow(
      NotaryChangeFlow(
        message,
        notary
      )
    )
  } else {
    message
  }

// however you want to choose your specific Notary
private fun notary(): Party =
  serviceHub.networkMapCache.notaryIdentities.single { it.name.organisation == "Notary-1" }

Here a specific Notary is chosen for the transaction, each input has its Notary changed to the chosen one if required and the signers comprise of all the participants of the consumed states. This might not suit your own use-case. Which is perfectly fine. But this should provide a good starting point when playing around with changing Notaries (mainly for performance).

Altering this solution slightly, we can instead choose the Notary based on the Notaries the input states reference. Since only the notary function really needs to change I have excluded the rest of the code from the example.

private fun notary(messages: List<StateAndRef<MessageState>>): Party =
  messages.map { it.state.notary }
    .groupingBy { it }
    .eachCount()
    .maxBy { (_, size) -> size }?.key ?: throw IllegalStateException("No Notary found")

The Notary chosen by this function is decided based on the most common Notary shared by the input states. By doing so, less Notary Change Transactions are required as the largest majority of the inputs will already reference the chosen Notary. This should provide the best performance if you do not know which Notaries the inputs reference.

Conclusion

Achieving high performance within a Corda network relies on removing bottlenecks out of the system and other general performance tweaks. One such bottleneck is the Notary. In a situation where a very high throughput is passing through the Notary, the network’s performance will start to plateau. The Notary cannot process the requests fast enough for the rate that they are coming in. Moving to use multiple Notaries that share the request load will allow the performance of the network to increase. This brings extra complexity in determining which Notary to use along with the possibility of needing Notary Change Transactions. But, if your network really needs to achieve a high throughput. This will be an area that is worth looking into.

One last comment I will throw in here. As the internal performance of Notaries increase, the need for this sort of architecture will decrease. There might even reach a point where a single Notary is able to completely handle a large rate of incoming requests. This is an area to keep an eye on as Corda continues to improve its overall performance.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Cached Prepared Statements with Spring Data Cassandra

Tue, 23 Oct 2018 00:00:00 GMT

Today I have a short post on using Prepared Statements in Spring Data Cassandra. Spring provides you with some utilities to make using Prepared Statements easier rather than relying on registering queries manually yourself with the Datastax Java Driver. The Spring code provides a cache to store prepared statements that are frequently used. Allowing you to execute your queries via the cache which either retrieves the prepared query from the cache or adds a new one before executing it.

To keep this short we should probably start looking at some code.

Dependencies

<parent>
  <groupId>org.springframework.boot</groupId>
  <artifactId>spring-boot-starter-parent</artifactId>
  <version>2.0.5.RELEASE</version>
</parent>

<dependencies>
  <dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-data-cassandra</artifactId>
  </dependency>
</dependencies>

Using Spring Boot 2.0.5.RELEASE will pull in 2.0.10.RELEASE of Spring Data Cassandra.

Using Prepared Statements

Let’s go straight in:

@Repository
public class PersonRepository extends SimpleCassandraRepository<Person, PersonKey> {

  private final Session session;
  private final CassandraOperations cassandraTemplate;
  private final PreparedStatementCache cache = PreparedStatementCache.create();

  public PersonRepository(
      Session session,
      CassandraEntityInformation entityInformation,
      CassandraOperations cassandraTemplate) {
    super(entityInformation, cassandraTemplate);
    this.session = session;
    this.cassandraTemplate = cassandraTemplate;
  }

  // using ORM
  public List<Person> findByFirstNameAndDateOfBirth(String firstName, LocalDate dateOfBirth) {
    return cassandraTemplate
        .getCqlOperations()
        .query(
            findByFirstNameAndDateOfBirthQuery(firstName, dateOfBirth),
            (row, rowNum) -> cassandraTemplate.getConverter().read(Person.class, row));
  }

  private BoundStatement findByFirstNameAndDateOfBirthQuery(
      String firstName, LocalDate dateOfBirth) {
    return CachedPreparedStatementCreator.of(
            cache,
            select()
                .all()
                .from("people_by_first_name")
                .where(eq("first_name", bindMarker("first_name")))
                .and(eq("date_of_birth", bindMarker("date_of_birth"))))
        .createPreparedStatement(session)
        .bind()
        .setString("first_name", firstName)
        .setDate("date_of_birth", toCqlDate(dateOfBirth));
  }

  private com.datastax.driver.core.LocalDate toCqlDate(LocalDate date) {
    return com.datastax.driver.core.LocalDate.fromYearMonthDay(
        date.getYear(), date.getMonth().getValue(), date.getDayOfMonth());
  }

  // without ORM
  public List<Person> findByFirstNameAndDateOfBirthWithoutORM(
      String firstName, LocalDate dateOfBirth) {
    return cassandraTemplate
        .getCqlOperations()
        .query(
            findByFirstNameAndDateOfBirthQuery(firstName, dateOfBirth),
            (row, rowNum) -> convert(row));
  }

  private Person convert(Row row) {
    return new Person(
        new PersonKey(
            row.getString("first_name"),
            toLocalDate(row.getDate("date_of_birth")),
            row.getUUID("person_id")),
        row.getString("last_name"),
        row.getDouble("salary"));
  }

  private LocalDate toLocalDate(com.datastax.driver.core.LocalDate date) {
    return LocalDate.of(date.getYear(), date.getMonth(), date.getDay());
  }
}

There is a reasonable amount of boilerplate code here so we can gain access to Spring Data’s ORM. I have also provided code to demonstrate how to achieve the same goal without using the ORM (well mapping straight from query to an object manually anyway).

Let’s look at one of the methods more closely:

public List<Person> findByFirstNameAndDateOfBirth(String firstName, LocalDate dateOfBirth) {
  return cassandraTemplate
      .getCqlOperations()
      .query(
          findByFirstNameAndDateOfBirthQuery(firstName, dateOfBirth),
          (row, rowNum) -> cassandraTemplate.getConverter().read(Person.class, row));
}

private BoundStatement findByFirstNameAndDateOfBirthQuery(
    String firstName, LocalDate dateOfBirth) {
  return CachedPreparedStatementCreator.of(
          cache,
          select()
              .all()
              .from("people_by_first_name")
              .where(eq("first_name", bindMarker("first_name")))
              .and(eq("date_of_birth", bindMarker("date_of_birth"))))
      .createPreparedStatement(session)
      .bind()
      .setString("first_name", firstName)
      .setDate("date_of_birth", toCqlDate(dateOfBirth));
}

CachedPreparedStatementCreator does exactly what it says… It creates cached Prepared Statements. The of method takes in the cache defined when the bean is instantiated and creates a new query as defined by the second parameter. If the query is one that has already been registered recently, i.e it is in the cache already. Then the query is pulled from there rather than going through the whole process of registering a new statement.

The query passed in is a RegularStatement which is converted to a PreparedStatement by calling createPreparedStatement (duh I guess). We are now able to bind values to query so it actually does something useful.

In terms of caching Prepared Statements, that is all you have to do. There are other ways to do it, for example, you could use the PreparedStatementCache manually or define your own cache implementation. Whatever floats your boat.

You have now reached the end of this short post, hopefully, it actually contained enough information to be useful…

In this post, we covered how to use the CachedPreparedStatementCreator to create and put Prepared Statements into a cache for faster execution at a later time. Using the classes Spring Data provides us, we can reduce the amount of code that we need to write.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Starting Flows with trackBy

Fri, 05 Oct 2018 00:00:00 GMT

Still continuing my trend of looking at Corda Services, I have some more tips to help your CorDapp work smoothly. This time around, we will focus on using trackBy to initiate Flows from inside a Service and the discrete problem that can arise if you are not careful.

This should be a relatively short post as I can lean upon the work in my previous posts: Corda Services 101 and Asynchronous Flow invocations with Corda Services. The content found in Asynchronous Flow invocations with Corda Services is very relevant to this post and will contain extra information not included within this post.

This post is applicable to both Corda Open Source and Enterprise. The versions at the time of writing are Open Source 3.2 and Enterprise 3.1.

A brief introduction to trackBy

trackBy allows you to write code that executes when a transaction containing states of a specified type completes. Whether they are included as inputs or outputs, the code will still trigger.

From here, you can decide what you want it to do. Maybe something very simple, like logging that a state has been received. Or, maybe something more interesting, such as initiating a new Flow. This use-case makes perfect sense for this feature. Once a node receives a new state or consumes one, they can start a new Flow that represents the next logical step in a workflow.

Furthermore, there are two versions of trackBy. One, the trackBy I keep mentioning, that can be used within a CorDapp. The other, vaultTrackBy, is called from outside of the node using RPC.

The problems presented in this post are only present in the CorDapp version, trackBy. Therefore, we will exclude vaultTrackBy for the remainder of this post.

What is this discrete problem?

Deadlock. When I word it that way, it isn’t very discrete. But, the way it happens is rather subtle and requires a good understanding of what is going on to figure it out. As mentioned before, this issue is very similar to the one detailed in Asynchronous Flow invocations with Corda Services. Furthermore, another shoutout to R3 for diagnosing this problem when I faced it in a project and I am sure they are going to iron this out. Until then, this post should save you some head scratching in case you run into the same problem.

I will quote what I wrote in my previous post as its explanation is only missing one point in regards to this post.

The Flow Worker queue looks after the order that Flows execute in and will fill and empty as Flows are added and completed. This queue is crucial in coordinating the execution of Flows within a node. It is also the source of pain when it comes to multi-threading Flows ourselves.

Why am I talking about this queue? Well, we need to be extra careful not to fill the queue up with Flows that cannot complete.

How can that happen? By starting a Flow within an executing Flow who then awaits its finish. This won’t cause a problem until all the threads in the queue’s thread pool encounter this situation. Once it does happen, it leaves the queue in deadlock. No Flows can finish, as they all rely on a number of queued Flows to complete.”

That marks the end of my copypasta. I am going to keep saying this though, really, I suggest you read through Asynchronous Flow invocations with Corda Services for a thorough explanation into this subject.

What has this got to do with trackBy? Calling trackBy from a Service will run each observable event on a Flow Worker thread. In other words, each event takes up a spot on the queue. Starting a Flow from here will add another item to the queue and suspend the current thread until the Flow finishes. It will stay in the queue until that time. If you end up in a situation where all the spots on the queue are held by the observable events, rather than actual Flows, I got one word for you. Deadlock. It is the exact same situation I’ve detailed before but starting from a different epicenter.

On the bright side, the solution is a piece of cake (where did this saying come from anyway?).

The section where the problem is fixed

Now that you know what the problem is. Altering a “broken” version to one shielded from deadlock will only require a few extra lines.

Let’s take a look at some code that is very similar to what lead me to step onto this landmine:

@CordaService
class MessageObserver(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  private companion object {
    val log = loggerFor<MessageObserver>()
  }

  init {
    replyToNewMessages()
    log.info("Tracking new messages")
  }

  private fun replyToNewMessages() {
    val ourIdentity = ourIdentity()
    serviceHub.vaultService.trackBy<MessageState>().updates.subscribe { update: Vault.Update<MessageState> ->
      update.produced.forEach { message: StateAndRef<MessageState> ->
        val state = message.state.data
        if (state.recipient == ourIdentity) {
          log.info("Replying to message ${message.state.data.contents}")
          serviceHub.startFlow(ReplyToMessageFlow(message))
        }
      }
    }
  }

  private fun ourIdentity(): Party = serviceHub.myInfo.legalIdentities.first()
}

This Service uses trackBy to start a new Flow whenever the node receives new MessageStates. For all the reasons mentioned previously, this code has the potential to deadlock. We don’t know when, or if it will ever happen. But, it could. So we should probably sort it out before it is an issue.

The code below will do just that:

@CordaService
class MessageObserver(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  private companion object {
    val log = loggerFor<MessageObserver>()
    // executor added
    val executor: Executor = Executors.newFixedThreadPool(8)!!
  }

  // init

  private fun replyToNewMessages() {
    val ourIdentity = ourIdentity()
    serviceHub.vaultService.trackBy<MessageState>().updates.subscribe { update: Vault.Update<MessageState> ->
      update.produced.forEach { message: StateAndRef<MessageState> ->
        val state = message.state.data
        if (state.recipient == ourIdentity) {
          // executor used
          executor.execute {
            log.info("Replying to message ${message.state.data.contents}")
            serviceHub.startFlow(ReplyToMessageFlow(message))
          }
        }
      }
    }
  }

  // ourIdentity
}

I have added a few comments to make it clearer what changed since only a few lines were added.

All this change does, is start the Flow on a new thread. This then allows the current thread to end. Remember, this is important because this thread holds onto a position in the queue. Allowing it to end, frees up a slot for whatever comes next. Whether it is another observable event from trackBy or a Flow. It does not matter. As long as the thread is released, the possibility of deadlock occurring due to this code is naught.

Releasing you from this thread

Please take a moment to bask in the glory of the pun I made in this sections header. Maybe it’s not that good, but I’m still proud of it.

In conclusion, using trackBy in a Corda Service is perfect for starting off new processes based on information being saved to the node. But, you need to be careful when starting new Flows from a trackBy observable. This is due to the observable holding onto a Flow Worker thread and therefore a spot in the queue. If your throughput reaches higher numbers, you risk the chance of your node deadlocking. You could end up in a situation where the queue is blocked by threads that are all waiting for a Flow to finish but with no actual Flows in the queue. By moving the Flow invocations onto a separate thread from the observable thread. You allow the once held spot on the queue to be released. There is now no chance of your trackBy code causing deadlock.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Asynchronous Flow invocations with Corda Services

Sat, 22 Sep 2018 00:00:00 GMT

How can I make my Flows faster? There’s a good chance you have thought about this before if you have been working with Corda for a while. You can make reasonable tweaks to eke out performance improvements by changing a few things: transaction size, optimising queries and reducing the number of network hops required throughout the Flow’s execution. There is one other possibility that probably also crossed your mind at some point. Multi-threading.

More specifically, asynchronously starting Flows/Sub Flows from an already executing Flow. Doing so has the potential to greatly improve your CorDapps performance.

If you tried this, you probably faced a similar exception to the one I got. Furthermore, as of now, Corda does not support threading of Sub Flows. But, it can still be done. We just need to be clever about it. That’s where multi-threading within Corda Services comes in. They can be called within Flows but are not prisoners to the strict rules that Flows put on them since an executing Flow will not suspend or checkpoint from within a service.

In this post, I will focus on multi-threading the starting of Flows from within a Service. There are other area’s that threading can be used within Corda, but this is an interesting area to that I want to look into deeper. On the other hand, starting Flows from a Service is also filled with a few gotchas. These need to be accounted for and traversed around. Otherwise, you are going to wake up one day and wonder why everything has stopped for no apparent reason.

Luckily for you, I am here to help. For me, well, I had to face this problem head-on.

Luckily for me, R3 was able to help.

For reference, I will be using Corda Enterprise 3.1 for this post. To actually gain any benefit from the contents of this post you will need to be using Enterprise. This is due to Enterprise supporting multiple Flows executing asynchronously. Open Source does not currently allow this.

I also recommend looking at my previous post Corda Services 101 as we will be building off the foundation laid there.

Scenario

Let’s start with outlining the scenario that we will be using for this post.

PartyA sends PartyB some messages over time. Each message comes from a single Flow.
PartyB responds to all messages sent to them. Each message comes from a single Flow, but they want a single place to execute the process.

A series of Flows can be quickly put together to satisfy this requirement. Doing this sequentially should prove absolutely zero problems (after we have fixed all the stupid mistakes we all make).

Although this scenario is a poor case for needing performance, it is a simple one to understand so we can focus on running this asynchronously.

The slow synchronous solution

Before we look at the asynchronous solution, it will be beneficial to have a quick look at the code we will be moving from. Below is the code from ReplyToMessagesFlow. I don’t want to go through all of the underlying code and instead only want to focus on the code relevant to this post:

@InitiatingFlow
@StartableByRPC
class ReplyToMessagesFlow : FlowLogic<List<SignedTransaction>>() {

  @Suspendable
  override fun call(): List<SignedTransaction> {
    return messages().map { reply(it) }
  }

  private fun messages() =
    repository().findAll(PageSpecification(1, 100))
      .states
      .filter { it.state.data.recipient == ourIdentity }

  private fun repository() = serviceHub.cordaService(MessageRepository::class.java)

  @Suspendable
  private fun reply(message: StateAndRef<MessageState>) =
    subFlow(SendMessageFlow(response(message), message))

  private fun response(message: StateAndRef<MessageState>): MessageState {
    val state = message.state.data
    return state.copy(
      contents = "Thanks for your message: ${state.contents}",
      recipient = state.sender,
      sender = state.recipient
    )
  }
}

If you did have a read through Corda Services 101 then you might have recognised this class. As I mentioned earlier, putting together a solution for the proposed problem is very easy. Retrieve the MessageStates from the Vault and start a subFlow to reply to them.

This code will happily chug along through the messages one by one.

So, can we take this code and make it faster?

A failed attempt at asynchronicity

Let’s try and make the current code faster by introducing threading! We will use CompletableFutures to do this:

@InitiatingFlow
@StartableByRPC
class ReplyToMessagesBrokenAsyncFlow : FlowLogic<List<SignedTransaction>>() {

  @Suspendable
  override fun call(): List<SignedTransaction> {
      return messages().map { CompletableFuture.supplyAsync { reply(it) }.join() }
  }

  // everything else is the same as before
}

Most of the code is the same as before and has therefore been excluded from the example.

The only change to the code is the addition of CompletableFuture and its supplyAsync method (comes from Java). It attempts to start executing the reply function for each message on a separate thread.

So why is this section named “A failed attempt”? I refer you to the stack trace you get when executing the above code:

java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Required value was null.
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_172]
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_172]
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592) ~[?:1.8.0_172]
    at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_172]
Caused by: java.lang.IllegalArgumentException: Required value was null.
    at net.corda.node.services.statemachine.FlowStateMachineImpl.checkDbTransaction(FlowStateMachineImpl.kt:201) ~[corda-node-3.1.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:192) ~[corda-node-3.1.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:271) ~[corda-node-3.1.jar:?]
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:312) ~[corda-core-3.1.jar:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.reply(ReplyToMessagesBrokenAsyncFlow.kt:57) ~[classes/:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.access$reply(ReplyToMessagesBrokenAsyncFlow.kt:19) ~[classes/:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$poop$inlined$map$lambda$1.get(ReplyToMessagesBrokenAsyncFlow.kt:46) ~[classes/:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$poop$inlined$map$lambda$1.get(ReplyToMessagesBrokenAsyncFlow.kt:19) ~[classes/:?]

You are going to get that, along with a lengthy list of checkpointing log lines that Corda is printing out. Furthermore, just to cover my ass and prove to you that this is not due to a problem with CompletableFutures, here is a different error you get when using an Executor threadpool:

Exception in thread "pool-29-thread-1" Exception in thread "pool-29-thread-2" java.lang.IllegalArgumentException: Required value was null.
    at net.corda.node.services.statemachine.FlowStateMachineImpl.checkDbTransaction(FlowStateMachineImpl.kt:201)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:192)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:271)
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:312)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.reply(ReplyToMessagesBrokenAsyncFlow.kt:48)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.access$reply(ReplyToMessagesBrokenAsyncFlow.kt:19)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$call$inlined$map$lambda$1.run(ReplyToMessagesBrokenAsyncFlow.kt:29)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalArgumentException: Required value was null.
    at net.corda.node.services.statemachine.FlowStateMachineImpl.checkDbTransaction(FlowStateMachineImpl.kt:201)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:192)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:271)
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:312)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.reply(ReplyToMessagesBrokenAsyncFlow.kt:48)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.access$reply(ReplyToMessagesBrokenAsyncFlow.kt:19)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$call$inlined$map$lambda$1.run(ReplyToMessagesBrokenAsyncFlow.kt:29)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Hopefully you believe me at this point. If not, refer back to what I said at the start. Corda does not currently support starting new Flows asynchronously from an executing Flow. It is something that I believe they are working towards. But, as of now. Don’t use this solution.

A working asynchronous solution

We have seen that threading inside a Flow is a no go. To carry on with our quest for performance, we will now look at threading from a Corda Service. This should come as no surprise since the title and opening paragraphs already discussed this…

Sarcastic comments aside. Delegating to a Service will require a bit of rework from the original solution but the bulk of the code will remain the same. Most of it will just be copied and pasted to another class. Taking the code from the Flow and putting it into a Service.

Below is the new MessageService that contains the code from the original ReplyToMessagesFlow, but with a few changes and addition of threading code:

@CordaService
class MessageService(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  private companion object {
    val executor: Executor = Executors.newFixedThreadPool(8)!!
  }

  fun replyAll() {
    messages().map {
      executor.execute {
        reply(it)
      }
    }
  }

  private fun messages() =
      repository().findAll(PageSpecification(1, 100))
          .states
          .filter { it.state.data.recipient == serviceHub.myInfo.legalIdentities.first() }

  private fun repository() = serviceHub.cordaService(MessageRepository::class.java)

  private fun reply(message: StateAndRef<MessageState>) =
      serviceHub.startFlow(SendMessageFlow(response(message), message))

  private fun response(message: StateAndRef<MessageState>): MessageState {
    val state = message.state.data
    return state.copy(
        contents = "Thanks for your message: ${state.contents}",
        recipient = state.sender,
        sender = state.recipient
    )
  }
}

As you can see most of the code is the same as it was in the ReplyToMessagesFlow.

The first point I want to highlight is the use of an Executor thread pool. I have not used CompletableFutures here for reasons that we will look at later.

So how does this all work? The replyAll function executes reply on a new system thread for each message retrieved from the Vault. This new thread, in turn, calls startFlow. Triggering a new Flow to be put onto the Flow Worker queue. This is where all the fun happens and everything starts to get messy.

The Flow Worker queue looks after the order that Flows execute in and will fill and empty as Flows are added and completed. This queue is crucial in coordinating the execution of Flows within a node. It is also the source of pain when it comes to multi-threading Flows ourselves.

Below is a diagram demonstrating a simplistic view of the queue:

Why am I talking about this queue? Well, we need to be extra careful not to fill the queue up with Flows that cannot complete.

How can that happen? By starting a Flow within an executing Flow who then awaits its finish. This won’t cause a problem until all the threads in the queue’s thread pool encounter this situation. Once it does happen, it leaves the queue in deadlock. No Flows can finish, as they all rely on a number of queued Flows to complete.

This is most likely to happen on a high throughput system that is triggering the same flow many times. The chance the queue is full of Flows waiting for other Flows to finish now shoots up.

This isn’t great and makes things a tad harder. But, as long as we are aware of this, we can accommodate for it.

This is also the reason for the Executor thread pool, rather than CompletableFutures. By starting new Flows and not waiting for their completion, deadlock can be avoided. This is also the downside of this solution. Without the results of the new Flows, its capabilities are extremely limited.

All that being said, if your use-case fits the sort of structure shown above then I definitely recommend using this solution.

In the next section, I will discuss using CompletableFutures.

A dangerous solution with CompletableFutures

There is one simple reason why this is dangerous. Deadlock. I recommend keeping clear of this solution. Unless your node has access to enough threads, to decrease the chance of filling the queue with threads that can’t finish. On the other hand, it is a much more ideal solution since you can await the results of the started Flows and do something with them. This makes for a much more useful solution.

Below is what the MessageService would look like with CompletableFutures:

@CordaService
class MessageService(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  fun replyAll(): List<SignedTransaction> =
    messages().map { reply(it).returnValue.toCompletableFuture().join() }

  // everything else is the same as before
}

The code is completely the same except for the replyAll function. The toCompletableFuture function that the returned CordaFuture provides, calls join to wait for the result of all futures and returns the overall result.

As I mentioned before, this solution could lead to deadlock. But, for your scenario, maybe it doesn’t. It is up to you to determine how likely it is to happen. If the odds are against you, it’s probably best to walk away. Either choosing to stick with a synchronous or an asynchronous solution similar to what I detailed in the previous section.

Do I really need to do this?

For now, yes, I believe you do.

Moving forward, I doubt that you will need to rely on the solution I proposed in this post.

I believe that Corda is working towards removing the need to even think about threading when starting Flows from within Flows. Instead, allowing you to simply call subFlow with an option to run it asynchronously. This would have allowed us to keep the original synchronous solution but with an option to make each subFlow run on a separate thread.

Joining the sections together

In conclusion, in Corda Enterprise 3, it is possible to initiate new Flows asynchronously within an executing Flow. This can provide good performance benefits depending on your use-case. There are downsides though. You cannot await the results of the asynchronous Flows without endangering your node with the threat of deadlock. The node’s underlying queue cannot deal with the situation it is being put in. Therefore, you need to be careful about how you go about introducing threads to your Flow invocations. Thankfully, as Corda progresses, it’s likely you won’t even need to worry about doing this yourself. It might even be as simple as adding a boolean function argument. That is the dream!

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Containerising a Spring Data Cassandra application

Sat, 08 Sep 2018 00:00:00 GMT

I’m continuing my journey of learning Docker. I am still keeping it simple at this point. This time around, I am going to tackle converting a Spring and Cassandra application to use containers instead of running locally on the host machine. More precisely, using Spring Data Cassandra to sort out the application.

I wish I looked at doing this change a while ago. I have written a fair amount of posts on Cassandra and each time I had to cd to the correct directory or have a shortcut to start it up. I guess it’s not that big of a deal, but there were a few other things involved. Such as, dropping and recreating keyspaces so I could test my application from scratch. Now, I just delete the container and restart it. To me anyway, this is helpful!

This post will be slightly different from my previous post, Using Docker to shove an existing application into containers. Instead, I will focus slightly more on the application side and remove the intermediate steps of using only Docker and instead will jump straight into Docker Compose.

Containers, containers, containers

I think it’s best to start on the container side of the project since the application depends on the configuration of the Cassandra container.

Let’s go!

FROM openjdk:10-jre-slim
LABEL maintainer="Dan Newton"
ARG JAR_FILE
ADD target/${JAR_FILE} app.jar
ENTRYPOINT ["java", "-jar", "/app.jar"]

There’s not much going on here. This Dockerfile builds the Spring application image that will be put into a container in a few moments.

Next up is the docker-compose file. This is will build both the Spring application and Cassandra containers:

version: '3'
services:
  app:
    build:
      context: .
      args:
        JAR_FILE: /spring-data-cassandra-docker-1.0.0.jar
    restart: always
  cassandra:
    image: "cassandra"

Again, there isn’t too much here. The app container builds the Spring application using the Dockerfile defined previously. The cassandra container instead relies on an existing image, appropriately named cassandra.

One thing that stands out is that the restart property is set to always. This was my lazy attempt to get past how long Cassandra takes to start and the fact that all the containers started with docker-compose start at the same time. This lead to a situation where the application is trying to connect to Cassandra without it being ready. Unfortunately, this leads to the application dying. I hoped that it would have some retry capability for initial connectivity built in… But it does not.

When we go through the code, we will see how to deal with the initial Cassandra connection programmatically instead of relying on the application dying and restarting multiple times. You will see my version of handling the connection anyway… I’m not really a fan of my solution but everything else I tried caused me much more pain.

A dash of code

I said this post would focus more on the application code, which it will, but we are not going to dive into everything I put within this application and how to use Cassandra. For that sort of information, you can have a look at my older posts, which I’ll link at the end. What we will do though, is examine the configuration code that creates the beans that connect to Cassandra.

First, let’s go through ClusterConfig which sets up the Cassandra cluster:

@Configuration
public class ClusterConfig extends AbstractClusterConfiguration {

  private final String keyspace;
  private final String hosts;

  ClusterConfig(
      @Value("${spring.data.cassandra.keyspace-name}") String keyspace,
      @Value("${spring.data.cassandra.contact-points}") String hosts) {
    this.keyspace = keyspace;
    this.hosts = hosts;
  }

  @Bean
  @Override
  public CassandraClusterFactoryBean cluster() {

    RetryingCassandraClusterFactoryBean bean = new RetryingCassandraClusterFactoryBean();

    bean.setAddressTranslator(getAddressTranslator());
    bean.setAuthProvider(getAuthProvider());
    bean.setClusterBuilderConfigurer(getClusterBuilderConfigurer());
    bean.setClusterName(getClusterName());
    bean.setCompressionType(getCompressionType());
    bean.setContactPoints(getContactPoints());
    bean.setLoadBalancingPolicy(getLoadBalancingPolicy());
    bean.setMaxSchemaAgreementWaitSeconds(getMaxSchemaAgreementWaitSeconds());
    bean.setMetricsEnabled(getMetricsEnabled());
    bean.setNettyOptions(getNettyOptions());
    bean.setPoolingOptions(getPoolingOptions());
    bean.setPort(getPort());
    bean.setProtocolVersion(getProtocolVersion());
    bean.setQueryOptions(getQueryOptions());
    bean.setReconnectionPolicy(getReconnectionPolicy());
    bean.setRetryPolicy(getRetryPolicy());
    bean.setSpeculativeExecutionPolicy(getSpeculativeExecutionPolicy());
    bean.setSocketOptions(getSocketOptions());
    bean.setTimestampGenerator(getTimestampGenerator());

    bean.setKeyspaceCreations(getKeyspaceCreations());
    bean.setKeyspaceDrops(getKeyspaceDrops());
    bean.setStartupScripts(getStartupScripts());
    bean.setShutdownScripts(getShutdownScripts());

    return bean;
  }

  @Override
  protected List<CreateKeyspaceSpecification> getKeyspaceCreations() {
    final CreateKeyspaceSpecification specification =
        CreateKeyspaceSpecification.createKeyspace(keyspace)
            .ifNotExists()
            .with(KeyspaceOption.DURABLE_WRITES, true)
            .withSimpleReplication();
    return List.of(specification);
  }

  @Override
  protected String getContactPoints() {
    return hosts;
  }
}

There isn’t too much there, but there would be even less if Spring would retry the initial connection to Cassandra. Anyway, let’s leave that part for a few minutes and focus on the other points in this class.

The original reason I created ClusterConfig was to create the keyspace that the application will use. To do this getKeyspaceCreations was overridden. When the application connects it will execute the query defined in this method to create the keyspace.

If this wasn’t needed and the keyspace was created in some other way, for example, a script executed as part of creating the Cassandra container, Spring Boot’s auto-configuration could be relied upon instead. This actually allows the whole application to be configured by the properties defined in application.properties and nothing else. Alas, it was not meant to be.

Since we have defined an AbstractClusterConfiguration, Spring Boot will disable its configuration in this area. Therefore, we need to define the contactPoints (I named the variable hosts) manually by overriding the getContactPoints method. Originally this was only defined in application.properties. I realised I needed to make this change once I started getting the following error:

All host(s) tried for query failed (tried: localhost/127.0.0.1:9042 
(com.datastax.driver.core.exceptions.TransportException: [localhost/127.0.0.1:9042] Cannot connect))

Before I created ClusterConfig the address was cassandra rather than localhost.

No other properties for the cluster need to be configured as Spring’s defaults are good enough for this scenario.

I have mentioned application.properties so much at this point, I should probably show you what is in it.

spring.data.cassandra.keyspace-name=mykeyspace
spring.data.cassandra.schema-action=CREATE_IF_NOT_EXISTS
spring.data.cassandra.contact-points=cassandra

keyspace-name and contact-points have already popped up since they are related to configuring the cluster. schema-action is needed to create tables based on the entities in the project. We don’t need to do anything else here as auto-configuration is still working in this area.

The fact that the contact-points value is set to cassandra is very important. This domain name originates from the name given to the container, in this case, cassandra. Therefore either cassandra can be used or the actual IP of the container. The domain name is definitely easier since it will always be static between deployments. Just to test this theory out, you can change the name of the cassandra container to whatever you want and it will still connect, as long as you change it in the application.properties as well.

Back to the ClusterConfig code. More precisely, the cluster bean. I have pasted the code below again so it’s easier to look at:

@Configuration
public class ClusterConfig extends AbstractClusterConfiguration {

  // other stuff

  @Bean
  @Override
  public CassandraClusterFactoryBean cluster() {

    RetryingCassandraClusterFactoryBean bean = new RetryingCassandraClusterFactoryBean();

    bean.setAddressTranslator(getAddressTranslator());
    bean.setAuthProvider(getAuthProvider());
    bean.setClusterBuilderConfigurer(getClusterBuilderConfigurer());
    bean.setClusterName(getClusterName());
    bean.setCompressionType(getCompressionType());
    bean.setContactPoints(getContactPoints());
    bean.setLoadBalancingPolicy(getLoadBalancingPolicy());
    bean.setMaxSchemaAgreementWaitSeconds(getMaxSchemaAgreementWaitSeconds());
    bean.setMetricsEnabled(getMetricsEnabled());
    bean.setNettyOptions(getNettyOptions());
    bean.setPoolingOptions(getPoolingOptions());
    bean.setPort(getPort());
    bean.setProtocolVersion(getProtocolVersion());
    bean.setQueryOptions(getQueryOptions());
    bean.setReconnectionPolicy(getReconnectionPolicy());
    bean.setRetryPolicy(getRetryPolicy());
    bean.setSpeculativeExecutionPolicy(getSpeculativeExecutionPolicy());
    bean.setSocketOptions(getSocketOptions());
    bean.setTimestampGenerator(getTimestampGenerator());

    bean.setKeyspaceCreations(getKeyspaceCreations());
    bean.setKeyspaceDrops(getKeyspaceDrops());
    bean.setStartupScripts(getStartupScripts());
    bean.setShutdownScripts(getShutdownScripts());

    return bean;
  }

  // other stuff
}

This code is only needed to allow retries on the initial Cassandra connection. It is annoying, but I could not come up with another simple solution. If you have a nicer one then please let me know!

What I have done is actually quite simple, but the code itself is not very nice. The cluster method is a carbon copy of the overridden version from AbstractClusterConfiguration, with the exception of the RetryingCassandraClusterFactoryBean (my own class). The original function used a CassandraClusterFactoryBean (Spring class) instead.

Below is the RetryingCassandraClusterFactoryBean:

public class RetryingCassandraClusterFactoryBean extends CassandraClusterFactoryBean {

  private static final Logger LOG =
      LoggerFactory.getLogger(RetryingCassandraClusterFactoryBean.class);

  @Override
  public void afterPropertiesSet() throws Exception {
    connect();
  }

  private void connect() throws Exception {
    try {
      super.afterPropertiesSet();
    } catch (TransportException | IllegalArgumentException | NoHostAvailableException e) {
      LOG.warn(e.getMessage());
      LOG.warn("Retrying connection in 10 seconds");
      sleep();
      connect();
    }
  }

  private void sleep() {
    try {
      Thread.sleep(10000);
    } catch (InterruptedException ignored) {
    }
  }
}

The afterPropertiesSet method in the original CassandraClusterFactoryBean takes its values and creates the representation of a Cassandra cluster by finally delegating to the Datastax Java driver. As I have mentioned throughout the post. If it fails to establish a connection, an exception will be thrown and if not caught will cause the application to terminate. That is the whole point of the above code. It wraps the afterPropertiesSet in a try-catch block specified for the exceptions that can be thrown.

The sleep is added to give Cassandra some time to actually start up. There is no point trying to reconnect straight away when the previous attempt failed.

Using this code the application will eventually connect to Cassandra.

At this point, I would normally show you some meaningless logs to prove that the application works, but in this situation, it really does not bring anything to the table. Just trust me when I say, if you run the below command:

mvn clean install && docker-compose up

Then the Spring application image is created and both containers are spun up.

Conclusion

We have had a look at how to put a Spring application that connects to a Cassandra database into containers. One for the application and another for Cassandra. The application image is built from the project’s code, whereas the Cassandra image is taken from Docker Hub. The image name is cassandra just to make sure no one forgets. In general connecting the two containers together was relatively simple but the application needed some adjustments to allow retries when connecting to Cassandra running in the other container. This made the code a bit uglier, but it works at least… Thanks to the code written in this post, I now have another application that I don’t need to set up on my own machine.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Links to my Spring Data Cassandra posts

Wow, I didn’t realise I wrote so many Cassandra posts.

All posts RSS Feed

Deep Work - A summary

Deep Work Is Valuable

Deep Work Is Rare

Deep Work Is Meaningful

Rule #1 Work Deeply

Rule #2 Embrace Boredom

Rule #3 Quit Social Media

Rule #4 Drain the Shallows

My Main takeaway

Ktor WebSockets

Including WebSockets

Writing a WebSocket handler

Outgoing data

Incoming data

Sending and receiving at the same time

Terminating a WebSocket

Ktor - A Kotlin web framework

Implementation

Dependencies

Starting the web server

Routing

Contents of a request handler

Installing plugins

Brief mention for Kodein

Summary

Reading and writing with a ConcurrentHashMap

Reading

Writing

A closer look at compute

Do not use get and put for atomic updates of existing mappings

Summary

I rarely need Stack Overflow anymore

Getting started with RSocket Kotlin

Dependencies

Building blocks of the inbound and backend services

Inbound service (RSocket client + HTTP server)

Backend service (RSocket server)

RSocket request types

Fire-and-forget

Client side

Server side

Request response

Client side

Server side

Request stream

Client side

Server side

Request channel

Client side

Server side

Summary

Should you write comments, revisited

Using GraphQL Java with Spring

Dependencies

The schema we’ll use

Fetching data with DataFetchers

The default PropertyDataFatcher

Writing a DataFetcher for a schema type’s field

Writing a DataFetcher for a query containing an argument

Setting up a GraphQL instance

Sending a GraphQL query to the application

Improving the registration of DataFetchers

Summary

Building a modular REST API with Javalin

Dependencies

Starting the Javalin server

Registering HTTP endpoints with Javalin

Writing an HTTP endpoint

Modular registration of endpoints

Summary

Documenting a Spring REST API following the OpenAPI specification

What is springdoc-openapi?

springdoc-openapi dependency

Taking your first steps with springdoc-openapi

Using OAS annotations for improved documentation

Extra tips after playing around with springdoc-openapi and the OAS annotations

Summary

Setting up a Polywork subdomain for a Netlify hosted site

Perfect is the enemy of good

Extra tips after playing around with `springdoc-openapi` and the OAS annotations