Corda posts RSS Feed

Responder flow validation (part 3 - overriding)

Mon, 06 Jan 2020 00:00:00 GMT

To finish off my trilogy of blog posts on including validation inside your responder flows, I present to you the final topic. How to override a responder flow with your own custom implementation. You will need to go down this route if you are leveraging an external CorDapp that does not allow you to extend their flows. It is not ideal, as you will not be able to make use of any of the responder code that was written. But, at least it will provide you with a way to implement your requirements while still allowing your version to communicate to the paired initiating flow. To cement what I am saying, I will say it one more time, you should only override if you cannot extend.

Before I get to the good stuff and show you what you need to do, if you really do need to go down this route, you should send an angry email to whoever wrote the CorDapp you are utilising. Moreover, you can tell them to read Responder flow validation (part 2 - extension) with some passive-aggressive message to go along with it 😎.

The content of this post is really a rehashing on what I already wrote in Extending and Overriding Flows from external CorDapps, except that I will focus precisely on overriding a responder flow.

When to override

So what sort of flow will you need to override?

Below is a flow that cannot be extended:

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

As this flow is written in Kotlin, the class is final and cannot be extended. In Java, the class would need to be marked as final explicitly to prevent you from extending it. In other words, all language rules around extending classes apply here and nothing else. If you tried to write a new flow that extended the one above, you would find a few compiler errors blocking your way.

Since this flow cannot be extended, the only solution is overriding it (or not using it at all).

How to override

To override the flow, you must do the following:

Write your own flow from scratch
Include @InitiatedBy and reference the initiating flow that your implementation should respond to
Add the flowOverrides configuration to your node.conf

Once these steps are taken, the node will start routing all calls to the original responder flow to your custom implementation.

Let’s look into each point a bit further:

Write your own flow from scratch

This could look like:

@InitiatedBy(SendMessageFlow::class)
class ValidatingSendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        val message = stx.coreTransaction.outputStates.single() as MessageState
        check(message.recipient == ourIdentity) { "I think you got the wrong person" }
        check(!message.contents.containsSwearWords()) { "Mind your language" }
        check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
        check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

Note that there is no mention of the original responder flow

Include @InitiatedBy and reference the initiating flow that your implementation should respond to

This was shown in the code block above:

@InitiatedBy(SendMessageFlow::class)
class ValidatingSendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

Add the flowOverrides configuration to your node.conf

Add the following to your node.conf:

flowOverrides {
  overrides=[
    {
      initiator="dev.lankydan.tutorial.flows.SendMessageFlow"
      responder="dev.lankydan.tutorial.flows.ValidatingSendMessageResponder"
    }
  ]
}

This links SendMessnageFlow to ValidatingSendMessageResponder instead of the original SendMessageResponder

You can also add this configuration to the deployNodes build task:

node {
  name "O=PartyB,L=London,C=GB"
  p2pPort 10003
  flowOverride(
    "dev.lankydan.tutorial.flows.SendMessageFlow",
    "dev.lankydan.tutorial.flows.ValidatingSendMessageResponder"
  )
}

When all these steps have been made, you should see a similar line in your node’s log file:

[INFO ] 2020-01-05T10:07:03,093Z [main] internal.NodeFlowManager. - Registered dev.lankydan.tutorial.flows.SendMessageFlow
 to initiate dev.lankydan.tutorial.flows.ValidatingSendMessageResponder (version 1)

This shows that the flow has been successfully switched out for the overridden version.

Forgetting to make step 2 or 3 will lead to errors.

Forgetting to add the @InitiatedBy annotation will cause the node to keep using the original flow

Forgetting to include the flowOverrides configuration will lead to an error due to multiple flows with the same @InitiatedBy annotation:

[ERROR] 2020-01-05T10:07:35,276Z [main] internal.NodeStartupLogging. - Exception during node startup: Unable to determine which flow to use when responding to:
dev.lankydan.tutorial.flows.SendMessageFlow. [dev.lankydan.tutorial.flows.SendMessageResponder, dev.lankydan.tutorial.flows.ValidatingSendMessageResponder] are all
registered with equal weight.

Conclusion

That brings us to the end of this trilogy of blog posts on responder flow validation.

Starting with why you should include validation in Responder flow validation
With the sequel detailing how to write flows that can be extended in Responder flow validation (part 2 - extension)
Finally ending with this post on overriding responder flows

You should now have a good understanding of why you should include validation in your responder flows, how to write extendable flows for other developers to leverage and how to override a flow if can’t be extended.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Responder flow validation (part 2 - extension)

Tue, 10 Dec 2019 00:00:00 GMT

In Responder flow validation I closed the post with the following:

How can I enforce the rules of my business when the logic inside a CorDapp is shared?

I will answer that question in my next post…

You can wait until then.

I’m not one to leave you hanging for too long. So, here I am, true to my word to show you how to enforce your own validation on a shared CorDapp.

Now, before I continue, I have a little disclaimer. What I will cover in this post requires cooperation from the developers of the CorDapp. Why? They are in control of what the CorDapp does. If they do not follow the necessary steps, then extending their CorDapp becomes much more difficult. It is still possible, but the chance of errors goes through the roof.

Obviously, this problem goes away if you are the one developing the CorDapp.

For the purpose of this post, I am going to assume that you are the CorDapp developer. I will show you how to write an extensible CorDapp that an external developer can leverage while adding a sprinkle of their own code. More specifically, allowing them to add extra verification to the responder flows that you create.

I will be skimming through some of the content in this post. For extra background information, I wrote Extending and Overriding Flows from external CorDapps earlier on this year. That post dives deeper down in regards to implementation compared to this post. Here we will only focus on custom validation.

Why extendable flows make sense for injecting validation

In Corda, organisations interact with each other through shared code, this means that each and every organisation executes the exact same code. The false assumption this model makes is that all businesses have identical processes. Many might be the same (there is a lot of duplication in this world), but the chance of them being exact copies is minuscule. To be honest, I would go as far as saying it’s impossible. Somewhere in the process, there will be a difference.

CorDapps need to be able to reflect this. Corda does its best to amalgamate processes amongst organisations. But, there is an appreciation that local customisation is necessary before companies can truly make the switch to distributed applications.

A vital example of the need for customisation is applying business rules to a shared application. Contracts and flows can provide a base level of validation. However, unique business rules still need to be applied before each individual organisation’s requirements are met.

As described in Extending and Overriding Flows from external CorDapps, a flow can be extended to apply additional logic defined inside of another CorDapp. Through the use of flow extension, you can design an extensible flow that applications can leverage to inject customised validation.

What a CorDapp developer must do

There are only a hand full of steps required to create an extensible flow:

The flow must be open (Kotlin) or not final (Java)
Provide a function(s) that can be overridden
The overridable function(s) must be called in relevant places

The following responder flow incorporates all the points above:

@InitiatedBy(SendMessageFlow::class)
// The flow is open
open class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  // An overridable function to contain validation is provided
  open fun checkTransaction(stx: SignedTransaction) {
    // To be implemented by sub type flows - otherwise do nothing
  }

  @Suspendable
  // [call] is final to prevent it from being overridden
  final override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        // The validation function is called
        this@SendMessageResponder.checkTransaction(stx)
        // Any other rules the CorDapp developer wants executed
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

I want to expand on the final step I mentioned above:

The overridable function(s) must be called in relevant places

I believe that SignTransactionFlow.checkTransaction is the best place to call an overridable function that contains custom validation. checkTransaction is triggered before signing a transaction, therefore it makes sense to add additional rules here. You could place it somewhere else, but the code becomes less efficient and harder to follow. I also discussed adding validation to checkTransaction in Responder flow validation.

An external developer now has two ways to use this flow:

Directly use the flow just like any normal flow. This means checkTransaction will execute without any code inside of it.
Leverage the flow by extending it and including their own custom verification.

Before we move on, I want to focus on a comment I included in the code:

// [call] is final to prevent it from being overridden
final override fun call(): SignedTransaction {

This might not be important. But, I personally think it is. By preventing external applications from altering the flow too much, you reduce the chance of the flow malfunctioning. For example, one unbalanced send or receive will break the flow. Making call final does not actually prevent this problem. However, by providing clear functions to implement, developers are less likely to make mistakes.

What an external developer must do

To extend and leverage an extensible flow, you must do the following:

Extend the flow
Include @InitiatedBy referencing the initiating flow
Implement/override provided functions

In other words, exactly what you would do to extend any average class in Kotlin or Java.

Below is what this could look like:

@InitiatedBy(SendMessageFlow::class)
class ValidatingSendMessageResponder(session: FlowSession) : SendMessageResponder(session) {

  override fun checkTransaction(stx: SignedTransaction) {
    val message = stx.coreTransaction.outputStates.single() as MessageState
    check(message.recipient == ourIdentity) { "I think you got the wrong person" }
    check(!message.contents.containsSwearWords()) { "Mind your language" }
    check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
    check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
  }
}

SendMessageResponder has been extended and checkTransaction has been implemented to include custom verification.

Any sessions that are initiated with the base SendMessageResponder will now be routed to ValidatingSendMessageResponder. You do not need to do anything else. Corda will handle it from here.

That’s really all there is. If the CorDapp developer has done the work in their flow to make it extendable, then external developers that leverage the flow should generally have a straight forward task when it comes to implementing their code.

Wrapping up

In this sequel to Responder flow validation, we looked into why flow extension provides an ideal solution for injecting custom validation and how to implement such a flow. If you continue to write CorDapps that are extensible, particularly regarding transaction validation, you will significantly increase both the usability and impact of your code. Businesses are far more likely to explore new solutions and migrate away from their existing linchpin/legacy applications if they are 100% confident that all of their needs can be met.

Just like the first episode in this trilogy, I will leave this post with a cliff hanger.

Now, before I continue, I have a little disclaimer. What I will cover in this post requires cooperation from the developers of the CorDapp that you wish to add your own rules to.

So, what can you do if the CorDapp developer does not play ball?

You’ll have to wait and see.

Responder flow validation

Mon, 25 Nov 2019 00:00:00 GMT

Do you want to prevent your node from accepting transactions that are not important to your business?

Do you want to prevent your node from accepting transactions that are invalid from your business’s perspective?

If you answered yes to either of these questions, then you might want to consider adding some validation to your responder flows.

For context, a responder flow is executed by a counterparty node when communicating with an initiating flow. Responder flows typically sign and record transactions sent to them. This means an organisation can end up in a situation where they did not have a large involvement in putting the contents of the transaction together, yet are expected to sign and potentially record it.

Surely, there should be some rules to prevent an organisation from being sent any old rubbish? And, yes, there are. The very bare minimum that needs to pass is contract validation. This ensures that a transaction is logically valid at a general level. What it does not do, is enforce local identity/roles or business level requirements.

To an extent, this is fine. A contract should only be ensuring that a transaction is logically valid.

Flows on the other hand, need to take charge of enforcing business level requirements on transactions.

Including validation

To include your own validation, you need to make use of SignTransactionFlow.checkTransaction when calling the SignTransactionFlow subflow. I guarantee that you have seen checkTransaction before (if you have worked with Corda already). If you have not, then please tell me how.

Assuming you have seen it, I bet there is a good chance that you also left it blank like this:

val stx = subFlow(object : SignTransactionFlow(session) {
  override fun checkTransaction(stx: SignedTransaction) {
    // no validation
  }
})

That is a code snippet from one of my other blog posts. Yes… I have sinned, and I am also a hypocrite. I have failed so you can learn from my mistakes! 👩‍🏫

Instead, it should look something like:

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      // [LedgerTransaction.verify] is called before [checkTransaction]
      override fun checkTransaction(stx: SignedTransaction) {
        val message = stx.coreTransaction.outputStates.single() as MessageState
        check(message.recipient == ourIdentity) { "I think you got the wrong person" }
        check(!message.contents.containsSwearWords()) { "Mind your language" }
        check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
        check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
      }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session, expectedTxId = stx.id))
  }
}

Obviously, most of the things being checked here are pretty random, but it does show off what I have been talking about.

The recipient is checked to ensure that the MessageState is intended for the node it is being sent to.
```
check(message.recipient == ourIdentity) { "I think you got the wrong person" }
```
This is an essential step to ensure that transactions you receive are intended for you and the role you play in the current flow, or in general, within the network. Not doing this, will allow your node to accept anything it is asked to sign and record. Even if it has absolutely nothing to do with you.
A series of other rules are also included. Each additional criteria is contractually valid but is not a desired property of the state in the context of this flow.
```
check(!message.contents.containsSwearWords()) { "Mind your language" }
check(!message.contents.containsMemes()) { "Only serious messages are accepted" }
check(message.sender.name.organisation != "Nigerian Prince") { "Spam message detected" }
```
I highly doubt that checking whether a message contains swear words would be contained inside of a contract. A message is still generally valid if it includes them. But, for the written flow, maybe it is a polite flow. Where any mention of a rude word must be beeped out (just like on pre-watershed TV). The same general gist applies to the other two rules.

How to validate

I have just shown you what sort of validation to include. What I didn’t do is explain how the code shown before leads to a transaction being rejected. You might have sussed it out, but let me be thorough and go through it properly.

To reject a transaction from within checkTransaction you need to throw an exception. Different exceptions will lead to different outcomes. This is due to the code below in SignTransactionFlow:

try {
  checkTransaction(stx)
} catch (e: Exception) {
  if (e is IllegalStateException || e is IllegalArgumentException || e is AssertionError)
    throw FlowException(e)
  else
    throw e
}

To properly indicate that a transaction is invalid, you should throw one of the following exceptions:

IllegalStateException
IllegalArguementException
AssertionError

These exceptions are converted to FlowExceptions, allowing them to be transferred to the counterparty with all details intact. You can also throw your own subclass of FlowException to cause the same effect.

Any other exception will become an UnexpectedFlowEndException with all of the reasons for the failure stripped out. From a counterparty’s perspective, this is the same as a random internal error that it could receive from its peers. Missing this vital information could have a significant effect on the outcome of the flow. It can be possible to propose a new transaction based on the information contained in the error. Receiving a UnexpectedFlowEndException makes it look like something just blew up.

To finally tie up this section, I want to give some suggestions on how to perform validation. This section is primarily focused on Kotlin developers since they provide some useful functions for this area (plus it’s what I code in every day).

I tend to use the following (some Kotlin basics coming up):

require - Throws an IllegalArgumentException

require(message.recipient == ourIdentity) { "I think you got the wrong person" }

check - Throws an IllegalStateException

check(message.recipient == ourIdentity) { "I think you got the wrong person" }

Which one you throw depends on the context (and possibly personal preference). I also don’t want to explain the difference between an IllegalArgumentException and IllegalStateException (as I would struggle). I personally prefer to use check here, but that’s just my opinion.

Both of these also have alternatives for null checking, requireNotNull and checkNotNull.

Corda also provides a built-in way to do similar validation using the requireThat DSL (Domain Specific Language):

requireThat {
  // <message if criteria fails> using <statement that must be true>
  "I think you got the wrong person" using (message.recipient == ourIdentity)
  // Non `using` lines can also be included
  val contents = message.contents
  "Mind your language" using (!contents.containsSwearWords())
  "Only serious messages are accepted" using (!contents.containsMemes())
  "Spam message detected" using (message.sender.name.organisation != "Nigerian Prince")
}

requireThat was built for use inside contracts but can be used anywhere. As the name suggests, it follows the same semantics as require. The difference is that it is a DSL, which makes use of using to define a message and criteria. Note, that you can add general code into the requireThat block. Only the using lines are going to (potentially) throw exceptions.

requireThat can also be used from Java (but looks a tad worse):

requireThat(require -> {
    require.using("I think you got the wrong person", message.getRecipient() == getOurIdentity());
    // Non `using` lines can also be included
    String contents = message.getContents()
    require.using("Mind your language", !contents.containsSwearWords());
    require.using("Only serious messages are accepted", !contents.containsMemes());
    require.using("Spam message detected", message.getSender().getName().getOrganisation() != "Nigerian Prince");
    return null;
});

It’s a bit more of a pain to use compared to Kotlin, but you might prefer to use it instead of other Java validation techniques.

Suggestions on what to validate

The following ideas are things that I think would be good to include. If I was writing a production-ready CorDapp, I might have some additional suggestions. For now, these are areas I think are important (some of these I touched on earlier):

Identity
- Is the transaction important to my business? Check that you are actually involved in this transaction in a meaningful way. For some scenarios, an organisation might be observing all transactions. Validation should change to accommodate this.
- What is my role in this transaction? You should validate that your business is participating in the transaction with the role that you expect. For example, a buyer or seller. Based on this, follow on validation should be included. Identity cannot be checked from within a contract (no access to ourIdentity), so it must be done inside of a flow.
Existing states
- Is this unique field actually unique compared to the states in my vault? It might be unique for the party proposing the transaction, it might not be for any of its peers though. The contents of a node’s vault cannot be accessed from within a contract. Therefore all validation that includes a vault query must be done from within a flow.
Business logic
- Does that state have an acceptable value for my business? You should validate that the states you are receiving meet whatever criteria your business has. Contracts should not enforce business rules. This is why flows should perform business validation.

Closing thoughts

I wrote this post to ensure that CorDapp developers add thorough validation to their applications. Ensuring that what they intend their CorDapps to do, is what they actually do. If there is even just a single hole in the hull of a ship, it is not a matter of whether it will sink, but when. Your applications should be put together with thorough consideration to reduce the chance (like a ship) that it sinks. Failures will still happen, but the chances of them happening and the repercussions that they cause will be significantly lower if your application is well defined and constrained. The validation inside of flows plays an essential part in fortifying the hull of your CorDapp.

While reading this, if you were particularly eagle-eyed and critical on what I wrote, you might have formulated the following question.

How can I enforce the rules of my business when the logic inside a CorDapp is shared?

I will answer that question in my next post…

You can wait until then.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Common CorDapp mistakes and how to fix them

Sat, 23 Nov 2019 00:00:00 GMT

This is the talk I gave at CordaCon in October.

Title: A tale of Corda Slack adventures - Common CorDapp mistakes and how to fix them

The talk covers the following topics:

Remembering the @Suspendable annotation
Contract validation
Responder flow validation
Balancing sends and receives in your flows

The laptop that ran the presentation didn’t support the emojis I added, so sorry for the random symbols you’ll see in the talk.

This video can also be found on the R3 website.

All other CordaCon videos can also be found here.

Connecting a Ktor web server to a Corda node

Mon, 12 Aug 2019 00:00:00 GMT

The preparation for this blog post began several weeks ago (probably over a month by now). Before I could write about melding Corda and Ktor together, I first needed to lay the groundwork and focus solely on Ktor. That is where my blog post, Ktor - a Kotlin web framework came into existence. If you haven’t used or seen Ktor before, I recommend taking a browse at that post either before or after reading this post. Reading it in advance is probably a better idea, but you are in control of your own life 🤷.

This post will focus on implementing a Ktor web server that connects to a Corda node. I am not going to talk about why you should use Ktor. That decision is up to you. What I am doing is providing you with some information and allowing you to formulate a decision by yourself (like anything you read on the internet 🙄).

Dependencies

buildscript {
  ext.ktor_version = '1.2.2'
  ext.kotlin_version_for_app = '1.3.41'

  repositories {
    mavenCentral()
    jcenter()
  }

  dependencies {
    classpath "org.jetbrains.kotlin:kotlin-gradle-plugin:$kotlin_version_for_app"
  }
}

apply plugin: 'java'
apply plugin: 'kotlin'

java {
  disableAutoTargetJvm()
}

dependencies {
  compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version_for_app"
  compile "$corda_release_group:corda-jackson:$corda_release_version"
  compile "$corda_release_group:corda-rpc:$corda_release_version"
  compile "$corda_release_group:corda:$corda_release_version"
  compile "io.ktor:ktor-server-netty:$ktor_version"
  compile "ch.qos.logback:logback-classic:1.2.3"
  implementation ("io.ktor:ktor-jackson:$ktor_version") {
    exclude group: 'com.fasterxml.jackson.module', module: 'jackson-module-kotlin'
  }
  compile project(":contracts")
  compile project(":workflows")
}

There are a few things to highlight here. First, the kotlin_version_for_app property. Ktor requires Kotlin 1.3 (as it uses coroutines) but, at the time of writing, Corda only supports Kotlin 1.2. Therefore, different versions need to be used for the web server code and the Corda node. Secondly, jackson-module-kotlin is excluded as it causes a runtime error due to a version mismatch.

High level look at the code

Below is a small snippet of code showing the starting point of the web server’s implementation:

fun main() {
  embeddedServer(
    Netty,
    port = System.getProperty("server.port").toInt(),
    module = Application::module
  ).start().addShutdownHook()
}

fun Application.module() {
  val connection: CordaRPCConnection = connectToNode()
  install(CallLogging) { level = Level.INFO }
  install(ContentNegotiation) { cordaJackson(connection.proxy) }
  routing { messages(connection.proxy) }
  addShutdownEvent(connection)
}

This code ties everything together as all the functionality of the server branches out from the functions above.

The contents of module will be explored in the following sections.

Connecting to the node

I bet you might have a good idea what connectToNode does. I hope you do anyway… Below are the contents of connectToNode:

fun connectToNode(
  host: String = System.getProperty("config.rpc.host"),
  rpcPort: Int = System.getProperty("config.rpc.port").toInt(),
  username: String = System.getProperty("config.rpc.username"),
  password: String = System.getProperty("config.rpc.password")
): CordaRPCConnection {
  val rpcAddress = NetworkHostAndPort(host, rpcPort)
  val rpcClient = CordaRPCClient(rpcAddress)
  return rpcClient.start(username, password)
}

If you have seen any of the Corda samples, then you will probably be familiar with this piece of code. Long story short, it connects to the node with the given connection details. I chose to generate the function’s default parameters from the application’s system properties. This implementation is not particularly important, it just connects to the node and could be written in several different ways.

A CordaRPCConnection is returned from the function. Initially, I wanted to return a CordaRPCOps as the connection itself doesn’t do too much. But, without returning the connection, there is no way to gracefully disconnect from a node. In other words, there needs to be a way to call notifyServerAndClose when the server stops. This is explored further down in the post.

Setting up Jackson

Some extra setup needs to be done to properly use Jackson with Corda:

fun ContentNegotiation.Configuration.cordaJackson(proxy: CordaRPCOps) {
  val mapper: ObjectMapper = JacksonSupport.createDefaultMapper(proxy)
  mapper.apply {
    setDefaultPrettyPrinter(DefaultPrettyPrinter().apply {
      indentArraysWith(DefaultPrettyPrinter.FixedSpaceIndenter.instance)
      indentObjectsWith(DefaultIndenter("  ", "\n"))
    })
  }
  val converter = JacksonConverter(mapper)
  register(ContentType.Application.Json, converter)
}

The Corda ObjectMapper is initialised with createDefaultMapper, allowing classes like Party or X509Certificate to be serialised or deserialised. This can be important depending on what is being returned from your own API.

The rest of the code is stolen from the ktor-jackson module. It alters the JSON output slightly to be more somewhat more desirable.

Creating the endpoints

HTTP requests are routed to these endpoints:

fun Routing.messages(proxy: CordaRPCOps) {
  route("/messages") {
    get("/") {
      call.respond(
        HttpStatusCode.OK,
        proxy.vaultQueryBy<MessageState>().states.map { it.state.data })
    }
    post("/") {
      val received = call.receive<Message>()
      try {
        val message = proxy.startFlow(
          ::SendMessageFlow,
          state(proxy, received, UUID.randomUUID())
        ).returnValue.getOrThrow().coreTransaction.outputStates.first() as MessageState
        call.respond(HttpStatusCode.Created, message)
      } catch (e: Exception) {
        call.respond(HttpStatusCode.InternalServerError, e.message ?: "Something went wrong")
      }
    }
  }
}

Logic wise, there is not much going on here. For a focused explanation of this code, I recommend reading Ktor - a Kotlin web framework as I mentioned earlier.

Gracefully disconnecting from a node

To gracefully disconnect from a node, the web server needs to call CordaRPCConnection.notifyServerAndClose. Implementing this required a bit of work that I wasn’t expecting. Below is the code that triggers notifyServerAndClose:

fun NettyApplicationEngine.addShutdownHook() {
  Runtime.getRuntime().addShutdownHook(Thread {
    stop(1, 1, TimeUnit.SECONDS)
  })
  Thread.currentThread().join()
}

fun Application.addShutdownEvent(connection: CordaRPCConnection) {
  environment.monitor.subscribe(ApplicationStopped) {
    connection.notifyServerAndClose()
  }
}

A shutdown hook is added to the server. As explained in Graceful shutdown of Ktor applications, subscribing to the ApplicationStopped event is not enough to execute code when terminating the application. The shutdown hook calls stop to gracefully close the NettyApplicationEngine that the server runs upon. Leading to the shutdown event being correctly triggered and executed.

That’s all there is

Yes, really, that is all. Implementing a super basic web server does not require much code at all. There isn’t really anything else to write. I have shown you that there is another web framework that can be used to connect to a Corda node. You don’t have to default to Spring just because the Corda samples use them. If you prefer Ktor, use Ktor. If you don’t, don’t. If you did like the look of Ktor, and if you haven’t already, I recommend looking at Ktor - a Kotlin web framework.

A lot of code was excluded from this post as I focused on the more important aspects of the implementation. If you are interested in the rest of the code, you can find it on my GitHub.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Flows can do anything

Tue, 30 Jul 2019 00:00:00 GMT

In Corda, Flows can do a lot more than proposing new transactions to record between organisations. Although, saying they can do anything is probably a bit far-reaching (it’s catchy though). What I really want to say, is that flows are the entry points into a node. Corda provides a series of functions to interact with a node via RPC. Currently, these cover the more straightforward use cases, such as querying the vault, but there is a limitation to what is provided. Flows cover any of the non-standard logic that needs to be triggered. So, if you want to expose an API from a Corda node that a client can trigger or consume, then this post is for you.

I will be exploring the use of flows as entry points to a node throughout this post. Flows that propose new transactions are shown in many other tutorials and have been excluded from this post. The limelight will be solely shone onto flows that provide different functionality.

The theory

As mentioned in the introduction, and I want to reiterate it, the only way to expose an endpoint or API from a Corda node is through a flow. The existing RPC endpoints will handle the standard functionality. In the future this might change, adding support for custom RPC endpoints. But, as of now (Corda 4), flows are the only way to run non-standard logic within a node.

In my opinion, exposing the functionality of a node through flows is very similar to writing HTTP endpoints for a web server. You must define what is accessible. Clients cannot request access to any internal code inside the web server. If there is not an endpoint, an error is sent back. The same concept is true for Corda. It is then more straightforward to reason about what a client can or cannot do when interacting with a node.

In regards to the implementation, the easiest way to get started is by showing you an example:

@StartableByRPC
class StupidSimpleQueryFlow(private val externalId: String) : FlowLogic<MessageState>() {
    // does not need to be suspendable?
    override fun call(): MessageState {
        return serviceHub.vaultService.queryBy<MessageState>(
            QueryCriteria.LinearStateQueryCriteria(
                externalId = listOf(externalId)
            )
        ).states.single().state.data
    }
}

If you know your stuff, then you will realise that you can do this via RPC. I chose this example to start you off with something familiar. You have probably done similar queries in your own flows. Although there is a good chance they were part of a more extensive process.

There are a few things to note here that differ compared to most flows:

No @InitiatingFlow annotation
No @Suspendable annotation
No matching responder flow

Why are they missing? Well, they are not needed. Think about it. Each point is related to communication with another node, which is not happening. The flow starts, does a query and returns the retrieved data. Let me expand on this:

@InitiatingFlow allows a flow to create new sessions to communicate with other nodes. Since no interactions are required, no sessions need to be created, and therefore the annotation can be removed.
@Suspendable is required on functions that suspend. Allowing a flow to create checkpoints that are loaded when the flow needs to wake up again. The most common place for a flow to suspend is during communication with another node. In this scenario, the annotation can be removed since the flow never needs to suspend.
Responder flows run on counterparty nodes and interact with your flows when you send data to them. Again, there is no communication, so there is no need for other nodes to have a flow installed that pairs with the flow above.

Most flows that do not interact with other nodes will follow this sort of structure (some flows can still suspend depending on what you are doing).

The flow defined above can then be accessed from an external client:

proxy.startFlow(::StupidSimpleQueryFlow, "asdas").returnValue.get()

Examples

Below are a few examples of flows that you could write and why you might use them.

Retrieving a value from a service

@StartableByRPC
class GetMeSomeValueFromAService : FlowLogic<Long>() {
  override fun call(): Long {
    return serviceHub.cordaService(IncrementingService::class.java).counter
  }
}

@CordaService
class IncrementingService(serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  var counter: Long = 0

  init {
    timer(period = TimeUnit.SECONDS.toMillis(1)) {
      counter += 1
    }
  }
}

If you have a service that stores some information that is useful to an external client, you will need a way to retrieve it.

Calling code inside the node to reduce code duplication

@StartableByRPC
class ExecuteSomeInternalNodeLogicYouDontWantToDuplicateFlow(private val recipient: Party) :
  FlowLogic<List<MessageState>>() {
  override fun call(): List<MessageState> {
    return serviceHub.cordaService(MessageRepository::class.java)
      .findAllMessagesByRecipient(recipient)
  }
}

@CordaService
class MessageRepository(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  fun findAllMessagesByRecipient(recipient: Party): List<MessageState> {
    return serviceHub.vaultService.queryBy<MessageState>(
      QueryCriteria.VaultCustomQueryCriteria(
        builder { MessageSchema.PersistentMessage::recipient.equal(recipient.name.toString()) }
      )
    ).states.map { it.state.data }
  }
}

Useful pieces of reusable code can be extracted into separate classes. More specifically to Corda, you could place logic within a flow or service.

To execute code inside a flow or service from the client:

Inside a flow - Just call the flow from the client
Inside a service - Add a new flow that delegates to the service

You could implement the query above in the client, but you will have two versions if you also use the same query inside the node.

Executing logic that can’t be done via RPC

@StartableByRPC
class DoSomethingComplicatedThatYouCantDoViaRpc(private val recipient: Party) :
    FlowLogic<List<MessageSchema.PersistentMessage>>() {
    override fun call(): List<MessageSchema.PersistentMessage> {
        return serviceHub.withEntityManager {
            createQuery(
                "SELECT m FROM $TABLE_NAME m WHERE m.recipient = ?1",
                MessageSchema.PersistentMessage::class.java
            ).setParameter(
                1,
                recipient.name.toString()
            ).resultList
        }
    }

    private companion object {
        val TABLE_NAME = MessageSchema.PersistentMessage::class.jvmName
    }
}

As mentioned earlier, there is a limit to what you can do via RPC. The code above is an example of this. It accesses the EntityManager which can execute lower level queries than what the vault allows.

Conclusion

Short conclusion for this post. Flows are the only entry points into a node that can run fully custom logic. Corda provides several APIs to interact with a node, but these provide limited functionality. To execute any sort of logic (that a node can run), you need to have a flow that is allowed to walk into the node and press the button for you.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Saving transactions where only a subset of parties are signers

Fri, 05 Jul 2019 00:00:00 GMT

It took a while for me to think of a title that could summarise the contents of this post without becoming a full sentence itself. I think I have managed to choose something legible 😅. Either way, let me clarify what I am actually talking about.

I have seen several people ask questions like the one below in Slack:

In the example, it shows a responder flow when the node which is running the responder flow is one of the required signers. But how about the case when the node running the responder flow is not a required signer (e.g. one of the participants of a state involved in the tx)? Do I need to write responder flow for such node? If so, how should I write the responder flow?

In other words:

I have a state that has a set of participants. Some of them must sign the transaction, some must not. How do I structure my flows, especially the responder flow to cope with this?

Due to how responder flows work, where every counterparty runs the same responder code (unless overridden). Having a group of counterparties do one thing, and another do something else is not handled by the simple code found in samples. Your flows need to be constructed to handle this explicitly.

The code to do this is relatively simple, but might not be evident unless you have been developing with Corda for a while.

So far, I have two solutions to this problem. I am pretty sure these are the best solutions currently available, and I cannot think of any others that would work or are worth pursuing. These solutions are:

Sending a flag to counterparties to tell them whether they are signers or not
Using subflow’d @InitiatingFlows to collect signatures or to save the transaction

I will expand these in following sections.

Before I get to them, what happens if you don’t account for the difference in signers and participants? A typical responder flow will include:

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) { }
    })
    return subFlow(ReceiveFinalityFlow(otherSideSession = session))
  }
}

If an initiating flow triggers this responder for a non-signing counterparty, an error occurs:

net.corda.core.flows.UnexpectedFlowEndException: Tried to access ended session SessionId(toLong=3446769309292325575) with empty buffer
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventsUntilFlowIsResumed(FlowStateMachineImpl.kt:161) ~[corda-node-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.suspend(FlowStateMachineImpl.kt:407) ~[corda-node-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowSessionImpl.receive(FlowSessionImpl.kt:67) ~[corda-node-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowSessionImpl.receive(FlowSessionImpl.kt:71) ~[corda-node-4.0.jar:?]
    at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:294) ~[corda-core-4.0.jar:?]
    at net.corda.core.flows.SignTransactionFlow.call(CollectSignaturesFlow.kt:198) ~[corda-core-4.0.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:290) ~[corda-node-4.0.jar:?]
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:314) ~[corda-core-4.0.jar:?]
    at dev.lankydan.tutorial.flows.SendMessageResponder.call(SendMessageFlow.kt:70) ~[main/:?]
    at dev.lankydan.tutorial.flows.SendMessageResponder.call(SendMessageFlow.kt:64) ~[main/:?]

This is because the non-signer is never sent the transaction to sign, but, alas, their code is sitting there waiting to sign a transaction that never comes. How sad 😿. I’m here to stop the counterparties of your flows from being sad like this one here.

This is also a good teaching moment. If you ever see a stack trace like the one above, it is most likely due to misplaced sends and receives. Either they are in the wrong order or there is a missing send or receive. Run through your code line by line and you should hopefully be able to pinpoint where the mismatch is.

Differentiating by flag

This solution is the one that came to me first as it is the easier one to understand.

A counterparty is notified telling them whether they need to sign the transaction or not. Their responder flow will then execute SignTransactionFlow or skip over it and go straight to ReceiveFinalityFlow. Both paths will always receive the flag and call ReceiveFinalityFlow.

An example can be found below:

@InitiatingFlow
class SendMessageFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val spy = serviceHub.identityService.partiesFromName("Spy", false).first()

    val tx = verifyAndSign(transaction(spy))

    // initiate sessions with each party
    val signingSession = initiateFlow(message.recipient)
    val spySession = initiateFlow(spy)

    // send signing flags to counterparties
    signingSession.send(true)
    spySession.send(false)

    val stx = collectSignature(tx, listOf(signingSession))

    // tell everyone to save the transaction
    return subFlow(FinalityFlow(stx, listOf(signingSession, spySession))
  }

  private fun transaction(spy: Party) =
    TransactionBuilder(notary()).apply {
      // the spy is added to the messages participants
      val spiedOnMessage = message.copy(participants = message.participants + spy)
      addOutputState(spiedOnMessage, MessageContract.CONTRACT_ID)
      addCommand(Command(Send(), listOf(message.recipient, message.sender).map(Party::owningKey)))
    }
}

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // receive the flag
    val needsToSignTransaction = session.receive<Boolean>().unwrap { it }
    // only sign if instructed to do so
    if (needsToSignTransaction) {
      subFlow(object : SignTransactionFlow(session) {
        override fun checkTransaction(stx: SignedTransaction) { }
      })
    }
    // always save the transaction
    return subFlow(ReceiveFinalityFlow(otherSideSession = session))
  }
}

Important points to the code above:

The spy (another party) is added to the state’s participants list
Flags are sent to the participants telling them whether to sign or not
The responder flow receives the flag and skips SignTransactionFlow if told to
ReceiveFinalityFlow is invoked waiting for the initiator to call FinalityFlow

I’ll leave the explanation at that as there is not much else to say.

Separating logic by extra initiating flows

This solution is a bit more involved due to the indirection caused by the different flows intermingling with each other. This solution really needs to be read before any explaining can be done:

@InitiatingFlow
@StartableByRPC
class SendMessageWithExtraInitiatingFlowFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    logger.info("Started sending message ${message.contents}")

    val spy = serviceHub.identityService.partiesFromName("Spy", false).first()

    val tx = verifyAndSign(transaction(spy))

    // collect signatures from the signer in a new session
    val stx = subFlow(CollectSignaturesInitiatingFlow(tx, listOf(message.recipient)))

    // initiate new sessions for all parties
    val sessions = listOf(message.recipient, spy).map { initiateFlow(it) }

    // tell everyone to save the transaction
    return subFlow(FinalityFlow(stx, sessions))
  }

  private fun transaction(spy: Party) =
    TransactionBuilder(notary()).apply {
      // the spy is added to the messages participants
      val spiedOnMessage = message.copy(participants = message.participants + spy)
      addOutputState(spiedOnMessage, MessageContract.CONTRACT_ID)
      addCommand(Command(Send(), listOf(message.recipient, message.sender).map(Party::owningKey)))
    }
}

@InitiatedBy(SendMessageWithExtraInitiatingFlowFlow::class)
class SendMessageWithExtraInitiatingFlowResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // save the transaction and nothing else
    return subFlow(ReceiveFinalityFlow(otherSideSession = session))
  }
}

@InitiatingFlow
class CollectSignaturesInitiatingFlow(
  private val transaction: SignedTransaction,
  private val signers: List<Party>
) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // create new sessions to signers and trigger the signing responder flow
    val sessions = signers.map { initiateFlow(it) }
    return subFlow(CollectSignaturesFlow(transaction, sessions))
  }
}

@InitiatedBy(CollectSignaturesInitiatingFlow::class)
class CollectSignaturesResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    // sign the transaction and nothing else
    return subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) { }
    })
  }
}

The flow of the code above is as follows:

The spy (another party) is added to the state’s participants list
The signer’s signature is collected by calling CollectSignaturesInitiatingFlow
CollectSignaturesInitiatingFlow creates a new session and calls CollectSignaturesFlow
CollectSignaturesResponder signs the transaction sent by CollectSignaturesInitiatingFlow
More sessions are initiated for each participant
FinalityFlow is called which triggers the SendMessageWithExtraInitiatingFlowResponder linked to the original/top-level flow

The code above, is built upon the fact that any flow annotated with @InitiatingFlow will be routed to its @InitiatedBy partner and is done so in a new session. Leveraging this allows a responder flow to be added that is only triggered for required signers. Sessions are still created for the top level flow (SendMessageWithExtraInitiatingFlowFlow) and are used in FinalityFlow.

There are a few other things that happen under the covers, but they are not needed for the context of this post.

Which is better?

Hard to say at the moment. I would have to do a little performance testing and play around with the code some more…

My current opinion is the extra initiating flows works a bit better. It removes the need for an extra trip across the network from the initiator to each individual counterparty. It adds a bit of additional boilerplate code but also extracts the signing logic out from the rest of the responder/counterparty code.

To be honest, as I said a minute ago, I really need to use it and come up with more complex use cases before I can give a good answer. I doubt I’ll ever get round to that though…😩

Conclusion

Whichever route you go down or even if you manage to come up with another one, saving transactions where only a subset of parties are signers can 100% be done in Corda. I would be pretty disappointed if this were not possible. As long as you have a process in place to alter the logic in the responder flows to handle the different sends and receives that signing and non-signing parties need. You should be good to go.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Preventing invalid spending of broadcasted states

Wed, 12 Jun 2019 00:00:00 GMT

Corda is super flexible and will allow you to put together the code needed to write many complex workflows. This flexibility does come with one disadvantage. You, the developer, need to spend some time and thought on designing your application. No more blank contracts. No more responder flows that sign anything they receive. You’ll be glad you put in the effort when your application is working seamlessly in production. Luckily for you, I am here to protect your vaults from dodgy transactions and invalid spending of your states.

In my previous post, Broadcasting a transaction to external organisations, I showed you how to share transactions and the states within them with anyone in a network. Raising potential avenues for states to be spent in ways that were not foreseen. Possibly leading to the business process that a CorDapp(s) defines to fall apart.

I gave you the knowledge to end up in this situation, and now I am trying to prevent you from making stupid 🤦‍♀️, and potentially significant, mistakes. In other words, I gave you the keys to the car, and now I am trying to prevent you from crashing or running someone over 🚗💀.

Or even,

🕷 With great power comes great responsibility. 🕷

In this post, I will cover the sort of checks you should be adding to your contracts and flows to prevent other nodes from gaining advantages from potential oversights in your application. More precisely, preventing states received from broadcasted transactions from being spent in ways that were not anticipated. After making these changes, states can only be spent by parties that the application fully intended to allow.

I will be using the MessageState (a LinearState) that I have included in all of my Corda posts. Added below for clarity:

@BelongsToContract(MessageContract::class)
data class MessageState(
  val sender: Party,
  val recipient: Party,
  val contents: String,
  override val linearId: UniqueIdentifier,
  override val participants: List<Party> = listOf(sender, recipient)
) : LinearState

Using this state as an example, I can help you think about the design of your own contracts and the validation in your flows. The workflow we will be looking at is the process of replying to a message sent between two nodes.

Contract verification

I hope you already know this. Contract verification is important. This is the validation that is run by each signer of a transaction and anyone who needs to verify that same transaction in the future. States can only evolve in ways which are allowed by their contract’s design. Who can create them, who can spend them, what values they can have, how many of them per transaction, and so on. It is up to you to define these rules.

Below are some general requirements I believe are important to ensure a contract places the necessary boundaries on a state. Guiding the state along the possible routes it can take to be created and transformed:

override fun verify(tx: LedgerTransaction) {
  val commandWithParties: CommandWithParties<Commands> = tx.commands.requireSingleCommand()
  when (commandWithParties.value) {
    is Commands.Send -> // validation for sending
    is Commands.Reply -> requireThat {
      val inputPublicKeys = tx.inputs.flatMap { it.state.data.participants.map(AbstractParty::owningKey) }.toSet()
      "The input participant keys are a subset of the signing keys" using commandWithParties.signers.containsAll(inputPublicKeys)
      val outputPublicKeys = tx.outputStates.flatMap { it.participants.map(AbstractParty::owningKey) }.toSet()
      "The output participant keys are a subset of the signing keys" using commandWithParties.signers.containsAll(outputPublicKeys)
    }
  }
}

Although this validation is for a LinearState, I actually nicked the rules from ContractsDSL.verifyMoveCommand. Maybe this is an indication that my MessageState is actually an OwnableState (a state with a single owner), but let’s brush that under the rug 🙈🙊.

The rules in the snippet above, check that the transaction is going to be signed by all the participants of the input and output states. Ensuring all the relevant parties have a chance to validate the transaction. In other words, the flow must include the parties mentioned by the input and output states in the command’s required signers, and sessions opened to send them the transaction.

Thanks to these rules, there is no way for a transaction to be committed to anyone’s vault without the participants of the input and output states signing the transaction. Doing so, in the eyes of the contract, would be illegal. Let me reword that into a more manageable explanation. Any transactions consuming MessageStates cannot be created without notifying the original participants of the state. Furthermore, the reply cannot be created and stored in a party’s vault without their say so.

These rules are a good basis to build up your own contract’s validation. Explicitly stating which parties must sign a transaction is an important requirement in proving its validity. Further rules can be added as needed.

For example, the following checks could be added to the example above:

override fun verify(tx: LedgerTransaction) {
  val commandWithParties: CommandWithParties<Commands> = tx.commands.requireSingleCommand()
  val command = commandWithParties.value
  when (command) {
    is Commands.Send -> // validation for sending
    is Commands.Reply -> requireThat {
      // general requirements from previous snippet
      // precise requirements for `MessageState`
      "One input should be consumed when replying to a message." using (tx.inputs.size == 1)
      "Only one output state should be created when replying to a message." using (tx.outputs.size == 1)
      val output = tx.outputsOfType<MessageState>().single()
      val input = tx.inputsOfType<MessageState>().single()
      "Only the original message's recipient can reply to the message" using (output.sender == input.recipient)
      "The reply must be sent to the original sender" using (output.recipient == input.sender)
      // covered by the general requirements
      "The original sender must be included in the required signers" using commandWithParties.signers.contains(input.sender.owningKey)
      "The original recipient must be included in the required signers" using commandWithParties.signers.contains(input.recipient.owningKey)
    }
  }
}

These rules are specific to the MessageState’s use-case. Restricting the parties who can reply to a message. More precisely, the sender of the reply must be the recipient of the original message and can only be sent back to the original sender. These rules make sense in this scenario.

Even with these rules, dodgy transactions can still go through. The contract checks the signatures based on the transaction’s states as well as the contents of the states. But, there are still some ways around these.

That being said, some of these situations can only occur if a transaction has been sent to parties not originally involved in an exchange. This is the sort of scenario that can arise from broadcasting transactions as I mentioned in the introduction and in Broadcasting a transaction to external organisations.

Now that I have set the stage let me explain a hole in the contract above. There is nothing preventing a third-party from creating a mimicked reply to the original message. In that, the sender and recipient of the reply are valid, but the message is created by someone else that is not actually the sender. It is a bit to get your head around, so you might need to read that sentence a few times.

To prevent this from occurring, some logic needs to be added into the flow that creates this transaction. This will be the topic of the following section.

Before we move on, I want to highlight again that this situation can only occur when the original transaction has been shared around with other parties. Therefore, it could be a situation that never happens. But it could. Covering your bases in these scenarios really depends on your use-case and how paranoid you are of users being naughty.

Flow verification

Flows provide a space for you to add transaction verification that lives outside of the contract. These can be rules that are more specific to an individual organisation (I touched on this in Extending Flows to customise transaction validation), or rely on information not available to the contract.

Following on from the previous section where I mentioned there being a few holes in the contract verification. The responder flow below adds two restraints to patch them up:

@InitiatedBy(ReplyToMessageFlow::class)
class ReplyToMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        val message = stx.coreTransaction.outputsOfType<MessageState>().single()
        require(message.sender != ourIdentity) {
          "The sender of the new message cannot have my identity when I am not the creator of the transaction"
        }
        require(message.sender == session.counterparty) {
          "The sender of the reply must must be the party creating this transaction"
        }
      }
    })
    return subFlow(
      ReceiveFinalityFlow(
        otherSideSession = session,
        expectedTxId = stx.id
      )
    )
  }
}

The require blocks in this example are not possible from inside the contract as they need information that the contract does not contain. This extra context is vital. The contract can only validate based on the information it is given. On the other hand, the flow knows that it is on the receiving end of a proposed transaction and who is the counterparty sending it.

Using this knowledge, the flow can add two new rules.

The sender of the reply cannot have the flow’s identity. Preventing another party from impersonating their identity and attempting to send a reply on their behalf.
The sender of the reply must be the owner of the counterparty session. Preventing another party from mimicking the identity of the sender.

Although the second rule is a superset of the first, splitting them out provides an opportunity for a better error message. If the messages are not important to you, then removing the first require statement is ok.

Thanks to the contract, the transaction has to gather this party’s signature before it can be persisted. Providing the opportunity for their responder flow to add additional validation and decline it if needed. With the contract and flow working as a team, the chances of the flow being used illegally are vastly reduced.

Conclusion

Due to Corda’s flexibility, it is your responsibility as a CorDapp developer to restrain your application enough to prevent invalid usage. This is further complicated by the ability to share transactions with nodes not involved in original interactions. If the CorDapp is not put together thoughtfully, a transaction’s states could be spent by a party that does not have direct ownership over the states. The validation found inside your contracts and flows is essential to prevent these scenarios from occurring.

However, it is important to remember that it really does depend on your use-case. Maybe you want to allow parties to spend states they don’t own from transactions shared with them. In that case, you could relax the CorDapp’s restrictions. As long as it is by design rather than an oversight.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Broadcasting a transaction to external organisations

Fri, 31 May 2019 00:00:00 GMT

There is a misconception that Corda cannot broadcast data across a network. This is simply wrong. In fact, Corda can send anything between nodes in a network. What Corda does not do, is share unnecessary data (transactions) with nodes that have nothing to do with an individual interaction. Privacy by default is a central component of Corda’s design. Compared to other DLT (Distributed Ledger Technology) platforms and blockchains, this is indeed a big difference. Sharing data with non-transacting parties might not be part of Corda’s default behaviour, but it is definitely within its capabilities. In this post, I will demonstrate the small amount of code required to send a transaction to any desired node.

That being said, the code included in this post is for a trivial implementation. Moving forward, I aim to write further posts on this subject and move towards more sophisticated and useful implementations for broadcasting transactions.

Before I show you any code, we should have a quick talk about the need for broadcasting data to parties not originally involved in a transaction. The most common reason I have come across is the need to meet regulatory requirements. For certain workflows information has to be shared with third-parties. These parties can then validate that nothing dodgy is going on and might also be a link to the real world. I am not going to lie. This isn’t exactly my area of expertise. Luckily, if you are reading this because you have concerns about meeting regulatory requirements, then you probably know better than me as to why you need the ability to broadcast transactions.

There is also information in the docs around this subject.

Onto the code. Below is a flow that sends a SignedTransaction to any Party passed to it:

@InitiatingFlow
class BroadcastTransactionFlow(
  private val stx: SignedTransaction,
  private val recipients: List<Party>
) : FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    for (recipient in recipients) {
      val session = initiateFlow(recipient)
      subFlow(SendTransactionFlow(session, stx))
    }
  }
}

For this scenario, Corda does pretty much all of the heavy lifting from inside the platform. Therefore, the only thing you need to do is iterate through the passed in parties and send the transaction to each of them.

Each call to SendTransactionFlow will communicate with the responder flow found below:

@InitiatedBy(BroadcastTransactionFlow::class)
class BroadcastTransactionResponder(private val session: FlowSession) : FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    subFlow(ReceiveTransactionFlow(session, statesToRecord = StatesToRecord.ALL_VISIBLE))
  }
}

There is only a single line inside of call. So, due to nothing else being there, it must be important. ReceiveTransactionFlow is the counterpart of SendTransactionFlow who receives and persists the transaction sent to it. Furthermore, the statesToRecord property determines which states from the transaction should be stored in the vault. Since this is a broadcast (possibly for regulatory reasons) the contents of the transaction are going to be important. To achieve this, StatesToRecord.ALL_VISIBLE is used telling ReceiveTransactionFlow to record every state contained within the transaction. If, depending on your use-case, you do not require all of the transaction’s states, then you can use ONLY_RELEVANT or even NONE.

The last important line of code in these snippets is the @InitiatingFlow annotation on BroadcastTransactionFlow. The annotation allows the flow to work without requiring its caller to create sessions and before passing them into the flow. The flow handles that. Logically this makes sense since most of the time, you will not have initiated sessions with parties that don’t already have the transaction stored. Thanks to this, using this flow from inside another is nice and simple. For example, it could be added at the end of a flow:

subFlow(FinalityFlow(stx, sessions)).also {
  // sends to everyone in the network
  val broadcastToParties = serviceHub.networkMapCache.allNodes.map { node -> node.legalIdentities.first() }
    .minus(serviceHub.networkMapCache.notaryIdentities)
    .minus(message.recipient)
    .minus(message.sender)
  subFlow(BroadcastTransactionFlow(it, broadcastToParties))
}

At this point, after calling BroadcastTransactionFlow, the transaction will exist in the vault of each party passed to the flow. This includes the states contained within the transaction. Great, job done. You now know that Corda can broadcast transactions.

There is one concern that I need to address before you copy and paste this code and ship it straight to production. Parties who receive output states from broadcasted transactions also have the ability to consume them. Therefore, if you wish to share transactions like this, then you need to put in the required safeguards to prevent organisations from spending states that don’t really belong to them. Yes, that organisation might be a regulator, and the chances of them doing anything dodgy is low. Yes, it might be possible to settle these issues outside of Corda. However, neither of these assumptions are ideal. To circumvent this, your contracts and flows must be designed to prevent organisations from spending states that are not theirs. This responsibility falls onto you, the CorDapp developer. Not Corda. The flexibility Corda empowers you with must be controlled to prevent undesired outcomes. I will cover this topic in a future post.

To wrap up, it is a myth that Corda cannot broadcast transactions to parties not originally involved in the process. You can achieve this in your application by using the code that I have presented in this post. It is not the fastest implementation. But it works. Allowing data to be shared with organisations who require details of transactions occurring on the network who are not otherwise included in individual transactions themselves.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Extending Flows to customise transaction validation

Sat, 18 May 2019 00:00:00 GMT

Through the use of flow extension, nodes running the same CorDapp can include extra validation to ensure that a transaction meets their specific requirements. The validation inside contracts focuses on the rules that must be adhered to by all transacting parties. Because of this, they are more general and focus on ensuring that no one is putting together invalid transactions. This leaves out any checks that individual organisations require. By providing a base CorDapp between the organisations with the ability to add further checks, they can each tailor the CorDapp to meet their needs. Flow extension makes this possible. Continuing the push for organisations to communicate together through common CorDapps while still providing enough customisation to meet each of their specific requirements.

That introduction might have caught your attention (I hope it did anyway), but the code to achieve this is relatively simple. It follows the same concepts that I wrote about in Extending and Overriding Flows from external CorDapps. The extra transaction validation I have been talking about can be achieved by providing a single open function inside of a flow or responder flow.

I will skip over a lot of information in the following snippets since they were covered in Extending and Overriding Flows from external CorDapps. If you haven’t realised this by now, I heavily recommend that you read that post.

For extra validation in the initiating flow, I believe a base flow that looks like the example below is what you want:

@InitiatingFlow
open class SendMessageFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  open fun extraTransactionValidation(transaction: TransactionBuilder) {
    // to be implemented by subtype flows - otherwise do nothing
  }

  @Suspendable
  final override fun call(): SignedTransaction {
    // build transaction
    val tx = verifyAndSign(transaction)
    // collect signatures
    // save transaction
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    extraTransactionValidation(transaction)
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }
}

Calling extraTransactionValidation before the TransactionBuilder’s verify function forces the proposed transaction to meet an organisation’s personal requirements before running any shared transaction validation.

An implementation of extraTransactionValidation is provided by a extended flow. Its implementation might look like:

class ExtraValidationSendMessageFlow(message: MessageState) :
  SendMessageFlow(message) {

  override fun extraTransactionValidation(transaction: TransactionBuilder) {
    requireThat {
      val messages = transaction.toLedgerTransaction(serviceHub).outputsOfType<MessageState>()
      "There must be only one output message" using (messages.size == 1)
      val message = messages.single()
      "Message must contain the secret passphrase" using (message.contents.contains("I love Corda"))
    }
  }
}

This is pretty much the same sort of validation you would see inside of a contract. That is the point really. The validation is done in the same way, but the rules the transaction is checked against are customised.

In a similar fashion to the initiating flow. The base responder flow will look like the following snippet:

@InitiatedBy(SendMessageFlow::class)
open class SendMessageResponder(private val session: FlowSession) : FlowLogic<SignedTransaction>() {

  open fun extraTransactionValidation(stx: SignedTransaction) {
    // to be implemented by subtype flows - otherwise do nothing
  }

  @Suspendable
  final override fun call(): SignedTransaction {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {
        extraTransactionValidation(stx)
      }
    })
    // save transaction
  }
}

Adding the extraTransactionValidation to the body of SignTransactionFlow.checkTransaction makes perfect sense. checkTransaction and subsequently extraTransactionValidation runs before the general contract validation that SignTransactionFlow executes.

Below is an example flow extending the base flow and implementing extraTransactionValidation:

@InitiatedBy(SendMessageFlow::class)
class ExtraValidationSendMessageResponder(session: FlowSession) :
  SendMessageResponder(session) {

  override fun extraTransactionValidation(stx: SignedTransaction) {
    requireThat {
      val messages = stx.coreTransaction.outputsOfType<MessageState>()
      "There must be only one output message" using (messages.size == 1)
      val message = messages.single()
      "Message must contain the secret passphrase" using (message.contents.contains("I love Corda"))
    }
  }
}

Yes, this validation is also included in the initiating flow. This is what I am talking about. By adding this validation into both the initiating and responder flow, an organisation can ensure that any transactions that pass through their system meet their precise requirements.

Due to the nature of this validation, it could be moved out into a function shared between the flows:

private fun validate(transaction: BaseTransaction) {
  requireThat {
    val messages = transaction.outputsOfType<MessageState>()
    "There must be only one output message" using (messages.size == 1)
    val message = messages.single()
    "Message must contain the secret passphrase" using (message.contents.contains("I love Corda"))
  }
}

This Kotlin function has been moved out of a class, making it a static function. It can then be used in both flows.

One point I want to explore further before wrapping up this post is the additional validation in the initiating flow. It is highly likely that a transaction put together by an organisation will be found valid by that same organisation. So the question is, why add the extra checks into the flow? Similar validation could be done before the flow has been invoked. Which is a perfectly valid place to do it. The benefit of adding it to the flow is realised when using a third-party CorDapp. It is further compounded as the complexity of a CorDapp increases. Determining the states created by a complex flow might not be a simple thing to do. Therefore creating and immediately checking a transaction’s contents is the safest thing to do.

To conclude, placing additional validation inside of flows ensures that transacting organisations are completely happy with the contents of transactions before they commit them. Furthermore, it continues the push for organisations to utilise CorDapps implemented by third-party developers. By providing enough flexibility in CorDapps for organisations to add their own rules. It becomes easier to adopt them, due to the guarantees that transactions meet the requirements of any organisation in the network.

The rest of the code used in this post can be found on my Github.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Verifying a Contract with CSV data

Sun, 05 May 2019 00:00:00 GMT

Attachments in Corda can be more than just PDFs sent along with a transaction. They can actually be used programmatically when running a flow or even inside of a contract’s verify function. Why would you want to do that though? 🤔 The answer makes a lot of sense when you think about it. Let’s take an attachment containing CSV data as an example. Actually, that is what this post is about. Anyway. The attachment could contain all the valid IDs (or whatever else) that a state is allowed to have. Now, that could be done within your code, but this is not practical for a system that needs to change over time. Maybe there are new IDs that need to be added. If they live inside the code then an updated CorDapp needs to be compiled and distributed whenever new values are allowed. Not entirely practical. But, uploading a new version of the file containing the data allows the validation to change over time, without the need for recompilation. That is an idea that makes perfect sense 👏.

Now that we agree that validating the contents of a transaction against attachment data is a good idea. The next question is where is the best place to put this validation. As mentioned before, there are two options. Inside the flow or in the contract. Putting it inside the contract makes the most sense here because all parties receiving the transaction have to run the attachment’s validation. Guarantying that they all reach consensus on the validity of the transaction 🤝.

Knowing that you can use attachments to validate a contract is great and all, but there is still some code to write before it becomes a reality. Don’t worry though, I got you 👍.

Building a transaction with an Attachment

Going to keep this one short and sweet since more information on this can be found in the docs. Below is some code from a flow that builds a transaction containing an attachment (it assumes the attachment already exists):

private fun transaction(): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    val attachmentId = attachment()
    addOutputState(message)
    addCommand(Command(Send(attachmentId), message.participants.map(Party::owningKey)))
    addAttachment(attachmentId)
  }

private fun attachment(): AttachmentId {
  return serviceHub.attachments.queryAttachments(
    AttachmentQueryCriteria.AttachmentsQueryCriteria(
      filenameCondition = Builder.equal(
        attachmentName
      )
    )
  ).first()
}

Adding the attachment to the transaction does not include any fancy code. Retrieving the attachment is a bit more involved but is not hard to put together either. In this example the attachment is being queried by name and the returned AttachmentId is then passed to the TransactionBuilder’s addAttachment (it takes the AttachmentId not the attachment itself). Using the name of the attachment is my preference but passing in the AttachmentId is also possible, assuming you know it beforehand.

There is one other sneaky piece of code that you might not have caught. I passed in the AttachmentId into the Send command. Doing so allow the attachment’s hash to be known and makes retrieving it from the transaction later far easier. I’ll quickly show you how I did that:

class MessageContract : Contract {

  interface Commands : CommandData {
    class Send(attachmentId: AttachmentId) : CommandWithAttachmentId(attachmentId), Commands
  }

  abstract class CommandWithAttachmentId(val attachmentId: AttachmentId) : CommandData {
    override fun equals(other: Any?) = other?.javaClass == javaClass
    override fun hashCode() = javaClass.name.hashCode()
  }
}

Verifying the Contract

This is where the magic 🧙‍♀️ happens. By using the attachment previously added to the transaction, the data inside of it can be pulled out and used to validate the states in the transaction. Below is the contract’s verify function:

override fun verify(tx: LedgerTransaction) {
  val command = tx.commands.requireSingleCommand<Commands>()
  when (command.value) {
    is Commands.Send -> requireThat {
      "No inputs should be consumed when sending a message." using (tx.inputs.isEmpty())
      "Only one output state should be created when sending a message." using (tx.outputs.size == 1)
    }
    is Commands.Reply -> requireThat {
      "One input should be consumed when replying to a message." using (tx.inputs.size == 1)
      "Only one output state should be created when replying to a message." using (tx.outputs.size == 1)
    }
  }
  require(isMessageInCsv(tx)) {
    "The output message must be contained within the csv of valid messages. " +
            "See attachment with hash = ${tx.attachments.first().id} for its contents"
  }
}

private fun isMessageInCsv(tx: LedgerTransaction): Boolean {
  val message = tx.outputsOfType<MessageState>().first()
  val attachmentId = tx.commandsOfType<CommandWithAttachmentId>().single().value.attachmentId
  return tx.getAttachment(attachmentId).openAsJAR().use { zipInputStream: JarInputStream ->
    zipInputStream.nextJarEntry.name
    val csv = CSVFormat.DEFAULT.withHeader("valid_messages")
      .withFirstRecordAsHeader()
      .parse(InputStreamReader(zipInputStream))
    csv.records.any { row -> row.get("valid messages") == message.contents }
  }
}

The interesting content in this example is found inside the isMessageInCsv function. It might look a bit daunting, but perhaps I just need to tidy it up a bit 😩… Forget I said that 😉. It just needs a little explanation (or maybe some code comments 🤔).

Attachments can be retrieved from a transaction via its commands property or via the getAttachment function (taking a position or AttachmentId/SecureHash). In this example, the attachment containing the CSV data is retrieved using getAttachment along with the id that was previously passed into the command.

The attachment has been retrieved by this point. Now the CSV data inside of it needs to be parsed and compared to the transaction’s states. To make this much simpler, I have used Apache Commons CSV. The attachment is opened using openAsJAR (the attachment is stored as a zip) and the library is utilised to read the data inside of it. As each row is read, it checks whether the contents of the MessageState matches the current row. If any do, then great. The state passes the test and the contract is deemed valid. If none match, the following error is output and you will need to try again 😈.

net.corda.core.contracts.TransactionVerificationException$ContractRejection: Contract verification failed: 
The output message must be contained within the csv of valid messages. See attachment with hash = 3E3031BA98F3F01843E8FD0A1B34E21C599C9C8F09765C2F820E45D6E8770948 for its contents, 
contract: com.lankydanblog.tutorial.contracts.MessageContract, transaction: 5C8963497E493684A78C0A95A30E4C30029E6C36A792DE8A1B1733DE5358BB15

This code does assume that the contents of the CSV follows some sort of format. In other words, "valid messages" is hardcoded to be a header in the file. If it is not there, then the validation becomes a bit pointless. It will fail if this happens, which is actually a good thing but if you remember back to what I said in the introduction. This introduces more hard coding and reduces the flexibility of the contract. If you wanted to improve this, you could pass in the name of the row you wish to use into the command, in the same way I added the AttachmentId to it earlier on.

Wrapping up

As you have seen, validating a transaction using data from a CSV attachment does not require too much effort. In fact, I bet that if you wanted to implement this yourself, your code would look almost identical to mine. I mean, there are only so many ways to do this. By delegating to a library most of the work will be done for you.

To summarise the steps needs to validate a transaction with some CSV data:

Upload the CSV to the node
Inside the flow, retrieve it and add it to a transaction
Inside the contract, take the CSV from the transaction and finally compare the contents of the transaction to it.

Doing this will allow your verification to change over time without the need to recompile your CorDapp. That sounds great right? I think so. If you don’t, then you should, because it is definitely great 😎.

The code used in this post can be found on my GitHub.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Uploading and downloading attachments in Corda

Thu, 25 Apr 2019 00:00:00 GMT

This post is dedicated to the number of people I have seen in the Corda Slack channel asking how to upload and subsequently download attachments to and from a node. Although both of these can already be achieved via the Corda Shell. In this post, I will be focusing on writing a client application that interacts with a node to control the attachments.

Hopefully, this short post will save me a lot of time in the future as well as helping you, the reader, implement this feature without the need to request help from anyone else.

As a Spring Fanboy, I will obviously be implementing the client application in Spring. By leveraging spring-web, an application can be put together with little effort while Spring does the heavy lifting. What does this mean for you? Well, it allows you to send and receive files to and from the Spring application by adding a few annotations and using the right objects. Praise Spring! 🙌🙏

One last thing, I am sure there is better code that can do the same thing as what I wrote for this post. If so, you can always submit a Pull Request to change it 😎. The post can be found here if needed.

Let’s get on with it.

Uploading

private val proxy: CordaRPCOps = rpc.proxy

@PostMapping
fun upload(@RequestParam file: MultipartFile, @RequestParam uploader: String): ResponseEntity<String> {
  val filename = file.originalFilename
  require(filename != null) { "File name must be set" }
  val hash: SecureHash = if (!(file.contentType == "zip" || file.contentType == "jar")) {
    uploadZip(file.inputStream, uploader, filename!!)
  } else {
    proxy.uploadAttachmentWithMetadata(
      jar = file.inputStream,
      uploader = uploader,
      filename = filename!!
    )
  }
  return ResponseEntity.created(URI.create("attachments/$hash")).body("Attachment uploaded with hash - $hash")
}

private fun uploadZip(inputStream: InputStream, uploader: String, filename: String): AttachmentId {
  val zipName = "$filename-${UUID.randomUUID()}.zip"
  FileOutputStream(zipName).use { fileOutputStream ->
    ZipOutputStream(fileOutputStream).use { zipOutputStream ->
      val zipEntry = ZipEntry(filename)
      zipOutputStream.putNextEntry(zipEntry)
      inputStream.copyTo(zipOutputStream, 1024)
    }
  }
  return FileInputStream(zipName).use { fileInputStream ->
    val hash = proxy.uploadAttachmentWithMetadata(
      jar = fileInputStream,
      uploader = uploader,
      filename = filename
    )
    Files.deleteIfExists(Paths.get(zipName))
    hash
  }
}

As I mentioned earlier I’m sure much can be done to improve it, but hey, it does the job.

I don’t think there is too much to dive into here since a lot of the code is around checking and possibly converting the file into the right format.

Although, this does raise a question, why is the conversion needed in the first place? Well, only Zips or Jars (which are pretty much Zips) can be stored as attachments in Corda. Therefore, if a different file type is received, it must first be converted so it can be stored in the node.

The only Corda specific code here is the call to uploadAttachmentWithMetadata. Setting the filename becomes handy when attempting to retrieve and download the stored attachment.

The hash (AttachmentId, an alias for SecureHash) is returned from the endpoint. Just like the filename, this provides another way to retrieve the attachment later on.

A quick example of uploading the attachment can be seen in the image below:

Downloading

I have implemented two different versions for downloading attachments.

By Hash / AttachmentId

private val proxy: CordaRPCOps = rpc.proxy

@GetMapping("/{hash}")
fun downloadByHash(@PathVariable hash: String): ResponseEntity<Resource> {
  val inputStream = InputStreamResource(proxy.openAttachment(SecureHash.parse(hash)))
  return ResponseEntity.ok().header(
    HttpHeaders.CONTENT_DISPOSITION,
    "attachment; filename=\"$hash.zip\""
  ).body(inputStream)
}

This endpoint retrieves an attachment using the hash returned from the upload endpoint. openAttachment requires a hash to pull the attachment from the node.

The hash is passed in as a String since there doesn’t seem to be any default conversion to SecureHash that Spring sets up and it isn’t worth the effort to set up for this situation.

The correct header needs to be added to trigger the download of the attachment by the caller. Calling this endpoint will then return a result like the below:

By name

private val proxy: CordaRPCOps = rpc.proxy

@GetMapping
fun downloadByName(@RequestParam name: String): ResponseEntity<Resource> {
  val attachmentIds: List<AttachmentId> = proxy.queryAttachments(
    AttachmentQueryCriteria.AttachmentsQueryCriteria(filenameCondition = Builder.equal(name)),
    null
  )
  val inputStreams = attachmentIds.map { proxy.openAttachment(it) }
  val zipToReturn = if (inputStreams.size == 1) {
    inputStreams.single()
  } else {
    combineZips(inputStreams, name)
  }
  return ResponseEntity.ok().header(
    HttpHeaders.CONTENT_DISPOSITION,
    "attachment; filename=\"$name.zip\""
  ).body(InputStreamResource(zipToReturn))
}

private fun combineZips(inputStreams: List<InputStream>, filename: String): InputStream {
  val zipName = "$filename-${UUID.randomUUID()}.zip"
  FileOutputStream(zipName).use { fileOutputStream ->
    ZipOutputStream(fileOutputStream).use { zipOutputStream ->
      inputStreams.forEachIndexed { index, inputStream ->
        val zipEntry = ZipEntry("$filename-$index.zip")
        zipOutputStream.putNextEntry(zipEntry)
        inputStream.copyTo(zipOutputStream, 1024)
      }
    }
  }
  return try {
    FileInputStream(zipName)
  } finally {
    Files.deleteIfExists(Paths.get(zipName))
  }
}

This snippet, like the uploading example, has a lot of extra code that plays around with OutputStreams and InputStreams.

This endpoint makes use of the queryAttachments function to retrieve the AttachmentIds matching the input criteria. A call to openAttachment is still needed to return the attachment once the AttachmentIds/hashes are retrieved.

Going down this route makes much more sense when compared to the downloadByHash endpoint since a filename is more human-readable than a hash.

Most of the extra code in this example is handling the possibility of multiple files being returned. If there are multiple attachments with the same name, then they get zipped together and returned.

You can then make a request like the one below which can handle the eventuality of many attachments with the same name:

Note that the file size is larger than the by hash version since I uploaded another file with the same name.

Wrapping up

There isn’t really a proper conclusion to write here. Basically, all I have to say is that, if you want to upload and download attachments to and from a Corda node with Spring, then the code in this post will help get you on your way. I am sure there are tidier ways to handle all the InputStreams and OutputStreams but they seem to get the job done.

If you found this post helpful and want to keep up with my posts as I write them, then you can follow me on Twitter at @LankyDanDev.

Developing with Corda 4

Fri, 05 Apr 2019 00:00:00 GMT

This is an updated version of the Developing with Corda post that I wrote last year. Since then quite a lot has changed, but from your perspective, as a developer, it should feel very similar. Corda has changed a lot to increase its performance, improve security and provide a better developer experience. Touching on the last point, improving the developer experience requires adding better functionality while removing less desirable parts. But, to do so while maintaining backwards compatibility allows you to go through the process of working with Corda 3 to 4 come with little issues. That being said, there are a few differences. These are the things that will be highlighted throughout this post while mainly focusing on providing a base for any newer Corda developers reading this.

If you have already worked with Corda before, then skimming through this post is probably enough. Maybe you have worked with Corda 3 and want to upgrade to 4, then taking a quick look at the code examples and seeing if anything looks different will probably get you what you need out of this post. But, if you are a brand new to Corda, then I recommend you take some time to read through this as well as What is Corda? and watch the Corda key concepts.

From this point on you will be reading through an updated version of Developing with Corda.

In What is Corda? I gave an overview of what Corda is trying to achieve and the design decisions that were made to do so. That information is great to know so we can get some perspective of the platform but it’s not going to help us in writing a system that can leverage Corda. To do that we need to know how the components of Corda work and fit together, only then can we start writing an application that actually does something and works correctly from the perspective of Corda. I see this as the same as learning any other framework. But, we need the voice in the back of our heads to remind us every now and then that we are in fact designing upon a Distributed Ledger Technology platform. Steps need to be taken to ensure that the applications we create are properly designed.

In this post, we will look at writing a very simple Corda application.

Corda is compatible with any JVM language. I know you’re thinking it, but no, it’s not written in Java. It is actually written in Kotlin. This might require a little bit of extra learning to get to grips with Kotlin if you normally work with Java (like I do) but it shouldn’t take you long to be comfortable with it. Due to this, I personally suggest writing your own code in Kotlin to keep the whole stack in the same language. This will help when you need to dig down into Corda’s own code as it will look less alien compared to what you were just writing. Obviously, that is just my suggestion and you could instead use Clojure. But you’re going to find it hard to get a hold of any existing examples without first converting them into their Clojure equivalents.

It is a bit hard to dive straight into the code without first understanding the key concepts of Corda. Rather than going through these myself, I personally believe the documentation provide on this subject is very helpful and should get you most of the way there.

For the purpose of this post, I think it is best to leave out the configuration required to actually setup Corda nodes and instead we should fully focus on writing a Corda application.

Overview of the application

Before we can begin writing any code, we need to have an understanding of what the application is trying to achieve. Rather than coming up with my own example, I will lean upon r3’s training materials as the example is reasonably simple to understand. This example is the “IOU” process (I owe you), where someone requests for some money which will be paid back at a later date. In this post, we will just focus on the issuing of an IOU.

Below is a sequence diagram containing the very simplified steps involved in issuing an IOU between two people:

As you can see, the borrower asks for some money, if the lender agrees, then they both try and remember that the borrower owes the lender some money. This is process is simple, but even then, it has been simplified some more. In this process, we haven’t actually mentioned when and how the money is transferred between them. For the purpose of this tutorial, we shall leave this transfer out and just focus on saving the intent of borrowing and lending money between them.

So, how is this sort of process modelled in Corda? Again, before we move on, I want to mention Corda’s key concepts as they are extremely important for understanding what is going on properly.

To model this process in Corda, we need to replace some of the steps. Determining how much money to borrow, becomes the creation of a transaction containing this information. A party being happy with what is proposed is now represented by signing the transaction with their key. Communication between the borrower and lender is replaced with sending a transaction between them. Finally, remembering who owes who is now set in stone with both parties saving the transaction that occurred between them. By altering the simple diagram with this new information, a new version is created:

There are more steps, but the process is still pretty simple. The diagram really speaks for itself and I don’t think there is anything more I can add to it. What I will say, is that I have still further simplified the diagram a little bit by removing the Notary, but only to focus on the process we are trying to model. We’ll touch on what a Notary is very, very, very briefly later on but I will again suggest Corda’s documentation on the subject (I’ll do the same again later on).

These diagrams should provide us with enough guidance to put our Corda application together for issuing an IOU between two parties.

States

Onto the coding sections, let’s start with states. Information on states can be found here from the Corda documentation. States are passed around the network between nodes and eventually find themselves stored on the ledger. In terms of a transaction, a state can be an input or an output and many of these can exist on a single transaction. They are also immutable, which is required to build up the chain of states that are used within transactions.

As mentioned earlier, I am leaning upon r3 training materials. Below is the IOUState that will be passed around with the application:

@BelongsToContract(IOUContract::class)
data class IOUState(
  val amount: Amount<Currency>,
  val lender: Party,
  val borrower: Party,
  val paid: Amount<Currency> = Amount(0, amount.token),
  override val linearId: UniqueIdentifier = UniqueIdentifier()
) : LinearState {

  override val participants: List<Party> get() = listOf(lender, borrower)
}

Due to the “IOU” concept, pretty much all the fields make sense without much explanation; amount is the lent amount, lender the party lending the money and so on. The only property that needs explaining is linearId which is of type UniqueIdentifier, this class is basically a UUID, in fact, it’s internalId is generated from the UUID class.

The state extends LinearState which is one of the general types of state that Corda uses with another being FungibleState. Both of these are implementations of ContractState. LinearStates are used to represent states that, quoting its docs, evolves by superseding itself. As such, when the state is updated it should be included as an input of a transaction with a newer version being output. The old state will now be marked as CONSUMED from UNCONSUMED when saved to the vault (Corda’s abstraction over the database).

ContractState requires a property that returns the participants of the state.

The participants are a very important part of the state. The parties defined within this list determine who gets to have the state saved to their own vault/database. If you find that your state is not being saved to a party that should know about it, this is most likely the issue. I have personally run into and felt the pain of this mistake.

In the above code, the participants were not included in the constructor and instead relies on defining a get function that can be used to retrieve them. Here it returns the lender and the borrower since they are the only two parties involved in the transaction. If you wanted to, you could add the participants to the constructor like below:

@BelongsToContract(IOUContract::class)
data class IOUState(
  val amount: Amount<Currency>,
  val lender: Party,
  val borrower: Party,
  val paid: Amount<Currency> = Amount(0, amount.token),
  override val linearId: UniqueIdentifier = UniqueIdentifier(),
  override val participants: List<Party> = listOf(lender, borrower)
) : LinearState

This allows you to define participants that might not be included in the state. Which route you take depends on your use-case, for this tutorial either will do the job.

One quick little thing before moving onto the next section. The @BelongsToContract annotation, that was not mentioned before, is binding a state to a contract (which we will look at very soon). Another way to write the same code without the need for the annotation is to move the state into an inner class of the contract, this would look something like the below:

class IOUContract : Contract {

  data class IOUState(
    val amount: Amount<Currency>,
    val lender: Party,
    val borrower: Party,
    val paid: Amount<Currency> = Amount(0, amount.token),
    override val linearId: UniqueIdentifier = UniqueIdentifier(),
    override val participants: List<Party> = listOf(lender, borrower)
  ) : LinearState
}

If you don’t do this or add the annotation then you’re going to get the following warning:

[WARN] 16:41:55,620 [Test worker] contracts.TransactionState.<init> - State class net.corda.training.state.IOUState
is not annotated with @BelongsToContract, and does not have an enclosing class which implements Contract.
Annotate IOUState with @BelongsToContract(net.corda.training.contract.IOUContract.class) to remove this warning.

Lucky for you, it is a clear message and should point you in the correct direction. If you ever get around to looking into the Corda codebase then you will see that the inbuilt states that have their own contracts are written like the example above. More information on this can be found in the Corda 3 to 4 upgrade notes.

Contracts

Next up are contracts. Contracts are used to validate input and output states for a given command by all parties involved in the transaction. The command could be, for example, issuing the state or paying off owed money. We will look at commands slightly later in this section but we should be able to brush over them for now.

Contracts are quite nice to write due to their expressiveness. We are able to write conditions within the contract that must be met for a transaction to be valid. If any are not met then an exception will be thrown. This normally ends in terminating the current transaction since an involved party has found it invalid.

These conditions use the requireThat DSL that Corda defines to specify conditions along with error messages that detail what is wrong with the transaction. This makes it nice and easy to go through a contract and understand what it is doing since the code conditions are nicely complemented by the English messages (or whatever language you want to write them in).

Below is an example of a contract that is used to validate the IOUState defined above:

class IOUContract : Contract {

  companion object {
    @JvmStatic
    val IOU_CONTRACT_ID = "net.corda.training.contract.IOUContract"
  }

  interface Commands : CommandData {
    class Issue : TypeOnlyCommandData(), Commands
    class Transfer : TypeOnlyCommandData(), Commands
    class Settle : TypeOnlyCommandData(), Commands
  }

  override fun verify(tx: LedgerTransaction) {
    val command = tx.commands.requireSingleCommand<IOUContract.Commands>()
    when (command.value) {
      is Commands.Issue -> requireThat {
        "No inputs should be consumed when issuing an IOU." using (tx.inputs.isEmpty())
        "Only one output state should be created when issuing an IOU." using (tx.outputs.size == 1)
        val iou = tx.outputStates.single() as IOUState
        "A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)
        "The lender and borrower cannot have the same identity." using (iou.borrower != iou.lender)
        "Both lender and borrower together only may sign IOU issue transaction." using
                (command.signers.toSet() == iou.participants.map { it.owningKey }.toSet())
      }
      is Commands.Transfer -> requireThat {
        // more conditions
      }
      is Commands.Settle -> {
        // more conditions
      }
    }
  }
}

I have simplified the contract for the purpose of this post since we will only focus on implementing one command type. Let’s start from the top and work our way down.

The IOUContract implements Contract requiring it to now have a verify function that gets called to verify (hence the name) a transaction.

Contract class name

The class name of the contract has been included here:

companion object {
  @JvmStatic
  val IOU_CONTRACT_ID = "net.corda.contracts.IOUContract"
}

This is used in other parts of Corda when reflection is required. Another way to write this if you don’t want to put the string in directly is like below:

companion object {
  @JvmStatic
  val IOU_CONTRACT_ID = IOUContract::class.qualifiedName!!
}

Commands

Now we can talk about the commands that I briefly mentioned earlier on in this section. I have put them below again so you don’t need to scroll again:

interface Commands : CommandData {
  class Issue : TypeOnlyCommandData(), Commands
  class Transfer : TypeOnlyCommandData(), Commands
  class Settle : TypeOnlyCommandData(), Commands
}

These commands are used to specify the intention of the transaction. They have been put here due to their connection to the contract since they determine what conditions must be validated. That being said, you could put these commands somewhere else if you wanted, such as in their own file or outside of the contract class but within the same file. As long as your solution best describes your intention then you are probably going in the correct direction.

Since these commands are simple and are only used to specify intent, TypeOnlyCommandData is extended. Other abstract classes are available that specify commands that we might want to use, such as MoveCommand.

We will see how to use the commands in the next section.

Implementing verify

Here’s where most of the magic happens, the code has been copied and pasted below:

override fun verify(tx: LedgerTransaction) {
  val command = tx.commands.requireSingleCommand<IOUContract.Commands>()
  when (command.value) {
    is Commands.Issue -> requireThat {
      "No inputs should be consumed when issuing an IOU." using (tx.inputs.isEmpty())
      "Only one output state should be created when issuing an IOU." using (tx.outputs.size == 1)
      val iou = tx.outputStates.single() as IOUState
      "A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)
      "The lender and borrower cannot have the same identity." using (iou.borrower != iou.lender)
      "Both lender and borrower together only may sign IOU issue transaction." using
              (command.signers.toSet() == iou.participants.map { it.owningKey }.toSet())
    }
    is Commands.Transfer -> requireThat {
      // more conditions
    }
    is Commands.Settle -> {
      // more conditions
    }
  }
}

The verify function checks whether the proposed transaction is valid. If so, the transaction will continue forward and most likely be signed and committed to the ledger. But, if any of the conditions are not met an IllegalArgumentException is thrown thus leading to the termination of the proposed transaction.

Exceptions are generally how Corda deals with unmet requirements. When an exception is thrown, assuming nothing is trying to catch it, execution is terminated and propagated up until it is caught or it ends up in the console output. This provides a simple way to control the flow of the transaction since it can be stopped anywhere in its execution, even on the counterparty’s node, as the exception will be passed back to the initiator.

Onto the verification code itself. The command that the transaction is executing on its states is retrieved and depending on the type, different checks are made to check the validity of the transaction. The requireThat DSL that Corda provides allows you to write a condition that must be true to continue along an error message that is output if the condition is false.

Let’s look at one of the requireThat statements a bit more closely:

val iou = tx.outputStates.single() as IOUState
"A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)

There’s not much to explain here. The DSL takes care of the intent of the statement. What I will point out is the syntax:

<string message of what condition should be met> using <condition it must pass>

Quite simple. Although I will highlight this point. Something that used to stupidly catch me out when I started working with Corda, was the need for brackets in the using statement if the condition contains any spaces. Forgetting this will give you some errors 🤦‍. Finally, the DSL can contain code that is not in a condition expression, allowing you to initialise variables and such which can then be used in the actual conditions.

That’s enough of contracts for now. They will pop up again in the next section when we put the IOUContract into action.

Flows

In Corda, flows are the central point where we tie off all previous sections. States, contracts, and commands all come together to write the code that will propose a new transaction, send it to all counterparties to sign and commit it to the ledger if everyone is happy. You can do much more complicated things within flows but for this tutorial, we will stick with the basics.

Following on from the examples in the previous sections, we will now implement the IOUIssueFlow. Below is the code as a whole which we will then split and examine:

@InitiatingFlow
@StartableByRPC
class IOUIssueFlow(val state: IOUContract.IOUState) : FlowLogic<SignedTransaction>() {
  @Suspendable
  override fun call(): SignedTransaction {
    val transaction: TransactionBuilder = transaction()
    val signedTransaction: SignedTransaction = verifyAndSign(transaction)
    val sessions: List<FlowSession> = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet().toList()
    val transactionSignedByAllParties: SignedTransaction = collectSignature(signedTransaction, sessions)
    return recordTransaction(transactionSignedByAllParties, sessions)
  }

  private fun transaction(): TransactionBuilder {
    val notary: Party = serviceHub.networkMapCache.notaryIdentities.first()
    val issueCommand: Command<IOUContract.Commands.Issue> =
      Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
    val builder = TransactionBuilder(notary = notary)
    builder.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
    builder.addCommand(issueCommand)
    return builder
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }

  @Suspendable
  private fun collectSignature(
    transaction: SignedTransaction,
    sessions: List<FlowSession>
  ): SignedTransaction = subFlow(CollectSignaturesFlow(transaction, sessions))

  @Suspendable
  private fun recordTransaction(transaction: SignedTransaction, sessions: List<FlowSession>): SignedTransaction =
    subFlow(FinalityFlow(transaction, sessions))
}

@InitiatedBy(IOUIssueFlow::class)
class IOUIssueFlowResponder(val flowSession: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val signTransactionFlow = object : SignTransactionFlow(flowSession) {
      override fun checkTransaction(stx: SignedTransaction) = requireThat {
        val output = stx.tx.outputs.single().data
        "This must be an IOU transaction" using (output is IOUContract.IOUState)
      }
    }
    val signedTransaction = subFlow(signTransactionFlow)
    return subFlow(ReceiveFinalityFlow(otherSideSession = flowSession, expectedTxId = signedTransaction.id))
  }
}

The flow of this code (yes that is a pun 😅) is reasonably straight forward and will be possibly one of the typical flows that you write within your own application. I can say for most flows that I have written over the last year or so for both work and for my other blog posts have followed this sort of structure. To be honest, whenever I start a new one, I copy and paste one a flow that looks like this and then start from there. Hasn’t failed me yet 🙂.

You obviously don’t have to write your flows in the exact same way. I have split the call function into a load of smaller functions. It does actually make the code a lot longer. But, I don’t think it can get any easier to read than that 😎.

Anyway, enough about myself and back to the code.

What it does:

Creates a state
Adds the state to a new transaction
Verifies the transaction with a contract
Signs the transaction
Requests the signatures of the counterparties
Saves the transaction for all participants

Now that we know the steps that are made within this flow, we can go through and explain how this is done within the code.

The Initiating Flow

Here’s the code again with just the Initiating Flow extracted from it:

@InitiatingFlow
@StartableByRPC
class IOUIssueFlow(val state: IOUContract.IOUState) : FlowLogic<SignedTransaction>() {
  @Suspendable
  override fun call(): SignedTransaction {
    val transaction: TransactionBuilder = transaction()
    val signedTransaction: SignedTransaction = verifyAndSign(transaction)
    val sessions: List<FlowSession> = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet().toList()
    val transactionSignedByAllParties: SignedTransaction = collectSignature(signedTransaction, sessions)
    return recordTransaction(transactionSignedByAllParties, sessions)
  }

  private fun transaction(): TransactionBuilder {
    val notary: Party = serviceHub.networkMapCache.notaryIdentities.first()
    val issueCommand: Command<IOUContract.Commands.Issue> =
      Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
    val builder = TransactionBuilder(notary = notary)
    builder.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
    builder.addCommand(issueCommand)
    return builder
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }

  @Suspendable
  private fun collectSignature(
    transaction: SignedTransaction,
    sessions: List<FlowSession>
  ): SignedTransaction = subFlow(CollectSignaturesFlow(transaction, sessions))

  @Suspendable
  private fun recordTransaction(transaction: SignedTransaction, sessions: List<FlowSession>): SignedTransaction =
    subFlow(FinalityFlow(transaction, sessions))
}

Firstly, the flow class is annotated with @InitiatingFlow and extends FlowLogic. This combination is required by any flow that requests communication with a counterparty. FlowLogic contains one abstract function call that needs to be implemented by the flow. This is where all the magic happens. When the flow is triggered, which we will look at later, call is executed and any logic that put inside the function runs. FlowLogic is generic (FlowLogic<T>) where T determines the return type of call. In the above example, a SignedTransaction is returned but it is totally feasible to use FlowLogic<Unit> if you have no desire to return anything back to the caller of the flow.

Next up is the @StartableByRPC annotation. This allows the flow to be called from an RPC connection which is the interface between the outside of a Corda node and it’s internals. We’ll touch on this a bit more when we look at triggering the flow.

Yet another annotation popping up. @Suspendable actually originates from quasar-core instead of one of Corda’s own libraries. If you forget to add it then you will see the error below and your flow will hang:

java.lang.IllegalArgumentException: Transaction context is missing.
This might happen if a suspendable method is not annotated with @Suspendable annotation.

@Suspendable is needed on all functions that communicate with a counterparty. As the name suspendable suggests, the annotation allows the function to be suspended while the counterparty is dealing with their side of the transaction. Quite a bit of magic goes on here and it is touched on briefly in the Corda documentation on flows.

In my original version of this post, I talked about the error being confusing and unrelated. But, it looks like that is resolved and an error message explaining exactly what is wrong has been thrown. That being said, if you do ever get some weird errors, it is still worth double checking that you have got @Suspendable in the right place.

Linking this back to the example. call, collectSignatures and recordTransaction are all annotated with @Suspendable. Both collectSignatures and recordTransaction call subFlows that interact with counterparties. call on the other hand needs the annotation because it is delegating to our own functions which are suspending.

This is our first encounter with subFlows. I will put a little section further down to quickly cover this and give the topic the explanation it deserves.

Back to the task at hand.

Missing the @Suspendable annotation is one of the most common pitfalls I have seen other developers fall into. So let me reiterate this point again.

You need to add the @Suspendable annotation to any function that has an interaction with a counterparty

You don’t understand how hard it is to not simplify this all the way down to 🤦‍

You need to add the @Suspendable annotation to any function that suspends

Just read whichever one gets my point across.

Now we’re done with the annotations we can look more closely at the contents of the call function.

Creating the transaction

First, we will look at building the proposed transaction, the relevant code has been extracted below:

private fun transaction(): TransactionBuilder {
  val notary: Party = serviceHub.networkMapCache.notaryIdentities.first()
  val issueCommand: Command<IOUContract.Commands.Issue> =
    Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
  val builder = TransactionBuilder(notary = notary)
  builder.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
  builder.addCommand(issueCommand)
  return builder
}

For the purpose of this post, we will assume there is only one Notary which allows us to be lazy and just retrieve the first one from the list. If you do not know what a Notary is, like earlier I suggest reviewing the Corda Key Concepts for a good explanation on the topic. For now, I’ll provide you the bare minimum to carry on. A Notary is a node whose sole purpose is to validate that no double spends have occurred within a transaction sent to it. Extra validation can also be run if it is set up to do so.

The serviceHub comes provided since we extended FlowLogic; the function networkMapCache will then provide the identities of the parties on the network and notaryIdentities narrows it down even more. As I mentioned earlier, we’re going to be lazy and just retrieve the first one from this list. How you retrieve the Notary that you wish to use in a transaction might change depending on your requirements.

We then create a command that represents the intent of the transaction. In this case, we use the IOUContract.Commands.Issue that was defined earlier. In creating the command we also need to provide the public keys of the parties required to sign the transaction. it is a Party and owningKey represents their public key. The only signers in this transaction are contained within the states participants property but an independent list could be passed in instead.

All the components needed for the transaction have now been retrieved or created. Now we need to actually start putting it together. TransactionBuilder does just that. The Notary is added via one of the TransactionBuilder’s constructors but can be set later on if preferred. addOutputState takes in the state passed into the flow along with the contract name that will verify it. Remember I mentioned two ways to get this name; via a public property within the object (how Corda normally does it) or by manually adding the classes name yourself, either way, the end goal is the same. The final component we add to this transaction is the command we created.

Verifying and signing the transaction

The next block of code focuses on verifying and signing the transaction, again the relevant code has been pasted below:

private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
  transaction.verify(serviceHub)
  return serviceHub.signInitialTransaction(transaction)
}

Once we are happy that everything we want to include in the transaction has been included, we need to verify it. Simply call the verify function the TransactionBuilder provides. This function results in the validation inside the contract running against the transaction. As mentioned earlier in the contract section, if any of the conditions in the contract fail an exception is thrown. Since, in this code, there are no attempts to catch the exception, the flow will fail as the exception is propagated up the stack.

After the transaction has passed validation, as far as we (the initiator) are concerned, the transaction is valid and should be signed. To do this, serviceHub.signInitialTransaction is called. This creates a new SignedTransaction that is currently only signed by the initiator. Having this transaction signed will become important later when the Notary checks that the transaction has been signed by all the parties involved.

Collecting signatures from the counterparties

The transaction is now both verified and signed by the initiator. The next step is requesting the signatures of the counterparties involved in the transaction. Once that is done the transaction can be persisted to everyone’s vaults as they all agree that the transaction is correct and meets their needs.

Below is the code in charge of collecting signatures from counterparties:

@Suspendable
private fun collectSignature(
  transaction: SignedTransaction,
  sessions: List<FlowSession>
): SignedTransaction = subFlow(CollectSignaturesFlow(transaction, sessions))

The counterparties in this transaction are defined by the parties in the participants list. If we remember back to how the state’s participants field was constructed, only two parties were contained in it, therefore only two parties will need to sign the transaction. Although that statement is correct, the transaction has already been signed by the initiator so now only the single counterparty (the lender) needs to sign it.

To send the transaction to the counterparty we first need to create a session with them. The code below does just that and the sessions returned from it are the input to the function above:

val sessions: List<FlowSession> = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet().toList()

The initiateFlow function creates a session to another node, it takes in a Party and returns a FlowSession to be used for communication. As mentioned the initiator does not need to sign the transaction again through this construct, so in the code, they have been removed from the parties whose communication sessions are being created for. Due to us knowing who are involved in this transaction, the single FlowSession could be created with the code below instead:

val session = initiateFlow(state.lender)

Instead of relying on the participants list we instead just create a session for the lender as our knowledge of the state indicates that they are the only counterparty.

The FlowSession needs to be used inside of the CollectSignaturesFlow along with the SignedTransaction that is still only signed by the initiator at this point. CollectSignaturesFlow sends the SignedTransaction to the counterparty (counterparties if multiple need to sign) and awaits their response. We will look at how the response is sent back in a following section. Once returned, the SignedTransaction is now complete as it has been signed by everyone and can be persisted.

Persisting the signed transaction

The following code persists the transaction:

subFlow(FinalityFlow(transaction, sessions))

Although, for a one-liner, this piece of code packs quite a punch. FinalityFlow will most likely always be called at the end of your flows, at least for the simpler flows anyway.

Calling FinalityFlow will:

Send the transaction to the Notary (if required)
Record the transaction to the initiator’s vault
Broadcast to the participants of the transaction to record it to their vaults

Expanding slightly on the 1st point. A transaction is only sent to a Notary if it has any input states that are being consumed. This is due to Notaries checking for previously spent states to prevent double spends. Therefore, if there are no input states then there is nothing for the Notary to do. Not sending it to the Notary is mainly an optimisation that removes wasted work.

The last two steps depend on the Notary finding the transaction valid. If it does not, like usual, an exception is thrown thus leading to an exit from the flow.

The Responder Flow

Everything in the flow that we have looked at so far is on the initiator’s side of the process. There have been a few times during the example where the transaction was sent over to the counterparty and some “stuff” happened. In this brief section we will inspect the code that the counterparty would run:

@InitiatedBy(IOUIssueFlow::class)
class IOUIssueFlowResponder(val flowSession: FlowSession) : FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val signTransactionFlow = object : SignTransactionFlow(flowSession) {
      override fun checkTransaction(stx: SignedTransaction) = requireThat {
        val output = stx.tx.outputs.single().data
        "This must be an IOU transaction" using (output is IOUContract.IOUState)
      }
    }
    val signedTransaction = subFlow(signTransactionFlow)
    return subFlow(ReceiveFinalityFlow(otherSideSession = flowSession, expectedTxId = signedTransaction.id))
  }
}

As before this code was included earlier on in the post.

The most important line in this class is the @InitiatedBy annotation that specifies which flow it accepts requests from and responds back to. In this example, it is the IOUIssueFlow that we have already gone through.

Since IOUIssueFlowResponder is also a flow, it extends FlowLogic and will need to implement its own version of call. The FlowSession in the constructor is the session that was used by the initiator to communicate with this flow. @Suspendable is also used on call just like it was in the initiating flow.

SignTransactionFlow is the other half of the CollectSignaturesFlow that was called in the initiator. It is an abstract class that requires checkTransaction to be implemented. This contains any extra validation that the counterparty might want to run against the transaction. SignTransaction’s call function will still verify the transaction against the contract so this is the chance for anything else; ensuring that the transaction is up to the standards of the counterparty. Saying that, checkTransaction can also contain as little code as desired and could even be empty if the contract validation is enough. Rather than showing you what that would look like, I’ll let you use your vivid imagination to imagine an empty function…

subFlow is then called on the implementation of SignTransactionFlow leading to it being executed. The validation in the contract runs, followed by the contents of checkTransaction and if all the checks come back fine, the transaction is signed and sent back to where it came from.

The final step is the invocation of ReceiveFinalityFlow. This is different to how it previously worked in Corda 3, where you did not need to manually receive the transaction. Anyway, this subFlow needs to be called to record the transaction to the vault of the counterparty. The id of the expected transaction to record is extracted from the transaction that was signed beforehand. After this point, both the counterparty will have the transaction saved away in their vault, ready to use in future flows.

The code in this class could be as simple or complicated as it needs to be. For the example used in this tutorial, simple is good enough. This will change depending on your requirements and what must be communicated between the initiator and its responders.

Making using of subFlows

You will have seen the subFlow function scattered throughout the code examples. Finally, I think we have reached a point in the post where I can talk about them a bit more.

subFlows are flows, similar to the one we are looking at in this post, that are called from within other flows. For example, CollectSignaturesFlow and FinalityFlow cannot be triggered by themselves as they are not annotated with @InitiatingFlow, therefore they can only ever be used from within a subFlow. Most of the flows provided by Corda out of the box fall within this same category.

All subFlow does is run the flow passed into it, by calling the flow’s call function and returning whatever the flow would normally return. A flow does not require anything special to be passed into subFlow, if we ever needed to, IOUIssueFlow could be passed into it from another flow.

Corda provides flows for a lot of the typical operations that need to be repeated throughout our own flows. These are called via subFlow and include (and many more): CollectSignaturesFlow, SignTransactionFlow, SendTransactionFlow and ReceiveTransactionFlow.

Starting a Flow

In this final section, we will look at how to call a flow from outside of the Corda node.

There are a few ways to do this, each working slightly differently. But, let’s keep this short and sweet and only look at the bog standard startFlow function:

proxy.startFlow(::IOUIssueFlow, state)

That’s it. As I said, short and sweet. proxy is of type CordaRPCOps which contains a load of functions revolved around interacting with the Corda node via RPC. startFlow is one of those functions. It takes in the name of the flow class along with any arguments that are part of the flow’s constructor. So in this example, IOUIssueFlow’s call function will be invoked with an IOUState being passed in to be used within the flow.

A FlowHandle<T> is returned where T is the same generic type of the invoked flow, in this case, a SignedTransaction. returnValue can then be called to retrieve a CordaFuture, allowing the result to be retrieved as soon as it’s available. CordaFuture is a subtype of a standard Future with a few extra methods made available, one of which is toCompletableFuture that may or not be useful to you (this was useful to me anyway).

Wrapping up

Here we are, at the end at last.

This post should have hopefully given you some help in understanding how to go about developing with Corda. There is much more to learn as I have only covered the basics in this post. In this post, we implemented the process of an IOU while inspecting the components that are required to do so. States are facts that are shared among parties on the network, contracts are used to validate transactions and flows contain the logic to propose new transactions. With this information, you should be in a good place to start writing your own flows. There is much more you can do with flows that hasn’t been covered within this post, but these basics should serve you well through any flows that you try to write.

If you found this post helpful and want to keep up with my posts as I write them, then you can follow me on Twitter at @LankyDanDev.

You can also join the Corda slack channel if you have any questions or are interested in learning about Corda further after reading this post.

Extending and Overriding Flows from external CorDapps

Sat, 02 Mar 2019 00:00:00 GMT

Corda 4 was released last week (21st Feb) bringing with it a ton of new features to make Corda more enjoyable to work with. To be honest, I am kind of assuming there are a lot of new features. I had a quick browse through the changelog, mainly to see my contributions being referenced, but I remember seeing a lot of lines of text. That has to be a good thing right?

Anyway, one of these features is the ability to extend and override Flows. It doesn’t really sound very fancy when you realise that Corda is written in Kotlin and has inheritance completely baked into it (true for Java as well). But, there is more to it than that. Corda needs to map an Initiating Flow to the counterparty Flow that is responding to it.

This is fine when two parties are using the same CorDapp. There is no extra complexity added in this situation. If, on the other hand, the counterparty wanted to send some data to an external system upon receiving a transaction, how could they do that? The original CorDapp does not know or care about this system and therefore it cannot cater to these needs. Being able to solve this sort of problem allows developers to build upon existing CorDapps and adapt them to be more suitable for their use-case. Furthermore, once good practices are laid out, extending third-party CorDapps will become easier and remove the need for teams of developers to constantly reinvent the wheel when others have already solved part of a problem. Obviously, this assumes that there is access to these external CorDapps, but it is completely in the realm of possibility. Especially with the R3 Marketplace already showcasing a collection.

In this post, we will be focusing on extending and overriding Flows. Furthermore, we will take the perspective of two different viewpoints.

The developer/maintainer of a CorDapp
A developer wanting to use and adapt an existing CorDapp

For the process to work, both sides must put in the effort to write their applications in an appropriate manner so that the benefits can be leveraged.

We will start by looking at what the original CorDapp must contain and then what a developer must do to extend it.

Before we go any further, here is a link to the official documentation on extending and overriding flows.

Writing a base Flow to allow extension

Writing a CorDapp in a way that allows it to be easily extended will likely require a reasonable amount of thought. It largely depends on what a CorDapp maintainer is trying to achieve. Providing a way for developers to extend a CorDapp so that they can send data to external systems or add their own logging should pose no problems. On the other hand, allowing the contents of a transaction to be altered or who it is sent to will require more thought to ensure that a CorDapp is not misused. This is a subject I hope to explore a bit further in future posts.

For the purpose of this post, we will look at the simpler option. Let’s jump right in since there has a been a whole lot of text so far and no code. Below is the SendMessageFlow that will act as the “base” Flow that will be extended in a later section:

@InitiatingFlow
open class SendMessageFlow(private val message: MessageState) :
  FlowLogic<SignedTransaction>() {

  open fun preTransactionBuild() {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun preSignaturesCollected(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun postSignaturesCollected(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun postTransactionCommitted(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  @Suspendable
  final override fun call(): SignedTransaction {
    logger.info("Started sending message ${message.contents}")
    preTransactionBuild()
    val tx = verifyAndSign(transaction())
    preSignaturesCollected(tx)
    val sessions = listOf(initiateFlow(message.recipient))
    val stx = collectSignature(tx, sessions)
    postSignaturesCollected(stx)
    return subFlow(FinalityFlow(stx, sessions)).also {
      logger.info("Finished sending message ${message.contents}")
      postTransactionCommitted(it)
    }
  }

  // collectSignature

  // verifyAndSign

  // transaction
}

I have removed a few of the functions so we can focus on what is important.

The first and sometimes important step to allow this class to be extended is the fact it is open. This is more of a Kotlin thing rather than Java since all classes in Kotlin are final by default. If you are writing this in Java then just ignore the last few sentences!

Following on from that, there are a series of functions that are available to be overridden. Each function has been placed in an appropriate place inside the main execution of the Flow. They will then be called when the Flow runs. For now, they have been given empty implementations since they provide no use to the CorDapp developer.

In regards to the open functions. You can name them or place them wherever you want. These are functions that I thought could be useful for developers wanting to add extra traceability over what the base app provides.

Digging down into a bit more detail. The call function has been made final (the same as in Java) to prevent the whole contents of the Flow from being overridden. If anyone wants to take your Flow and completely replace its “main” functionality, then what is the point? To me, it seems kind of dodgy. To remove that possibility making it final is a smart move.

Later on, we will look at how this Flow can be subclassed.

Below is the SendMessageResponder that interacts with the SendMessageFlow. It follows the same concepts as above and therefore I will only show it as a reference for later:

@InitiatedBy(SendMessageFlow::class)
open class SendMessageResponder(private val session: FlowSession) : FlowLogic<Unit>() {

  open fun postTransactionSigned(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  open fun postTransactionCommitted(transaction: SignedTransaction) {
    // to be implemented by sub type flows - otherwise do nothing
  }

  @Suspendable
  final override fun call() {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {}
    })
    postTransactionSigned(stx)
    val committed = subFlow(
      ReceiveFinalityFlow(
        otherSideSession = session,
        expectedTxId = stx.id
      )
    )
    postTransactionCommitted(committed)
  }
}

Extending an existing Initiating Flow

In this section, we get to see how the developer can make use of the work done on the previous Flow. It already has all the needed functionality. The only thing that is missing is the small amount of extra traceability that the developer wants to add in. Thanks to the functions added to the base Flow. This should cause no problems.

Let’s start with extending an Initiating Flow. The requirements for doing so are as follows:

Extend the base @InitiatingFlow
Do not add @InitiatingFlow to the new Flow (errors will occur if you do)
Reference the base Flow’s constructor (super in Java)
Override any desired functions
Call the new Flow instead of the base Flow

After reading that list, you might have realised that this is pretty much a description of inheritance in Object Oriented languages (like Kotlin and Java). There might be more going on inside Corda to allow this to work but from your perspective, you are just writing normal Object Oriented code like usual.

Taking these requirements we can see what an extended Flow might look like:

@StartableByRPC
class CassandraSendMessageFlow(private val message: MessageState) :
  SendMessageFlow(message) {

  override fun preTransactionBuild() {
    serviceHub.cordaService(MessageRepository::class.java).save(
      message,
      sender = true,
      committed = false
    )
    logger.info("Starting transaction for message: $message")
  }

  override fun preSignaturesCollected(transaction: SignedTransaction) {
    val keys = transaction.requiredSigningKeys - ourIdentity.owningKey
    logger.info("Collecting signatures from $keys for transaction for message: $message")
  }

  override fun postSignaturesCollected(transaction: SignedTransaction) {
    logger.info("Collected signatures for transaction for message: $message")
  }

  override fun postTransactionCommitted(transaction: SignedTransaction) {
    serviceHub.cordaService(MessageRepository::class.java).save(
      message,
      sender = true,
      committed = true
    )
    logger.info("Committed transaction for message: $message")
  }
}

I have left in all the noisy functions that implement the extra traceability I was talking about, but that is due to how empty the class would be without them. Since call does not need to be implemented. This Flow only needs to override the open functions. To be honest, it doesn’t need to override them at all, they are optional. If desired, this Flow could override a single function and then be left empty.

Have all the requirements listed above been met?

CassandraSendMessageFlow extends SendMessageFlow
There is no @InitiatingFlow in sight
In Kotlin you must call the super constructor anyway, so that’s done
In this scenario, all the functions have overridden
We haven’t got this far

Ok, so that is 4/5 so far. That’s a pretty good start. To cross off the last item on the list we need to see how it’s called. Below are snippets that call the base SendMessageFlow and the CassandraSendMessageFlow extending Flow.

Starting with SendMessageFlow:

proxy.startFlow(::SendMessageFlow, messageState)

Followed by CassandraSendMessageFlow:

proxy.startFlow(::CassandraSendMessageFlow, messageState)

Notice the difference? In this scenario, only the name of the Flow has changed. Nothing else.

Both snippets are completely valid. Calling the original SendMessageFlow is still allowed. Remember from our perspective, it is just a normal Object Oriented code. It won’t have the fancy extra code added to the extending Flow but it will still execute without issues. Completing this step meets the last requirement for extending an @InitiatingFlow.

Before we end this section, here is an important piece of information to remember from the Corda documentation:

You must ensure the sequence of sends/receives/subFlows in a subclass are compatible with the parent.

I will put this into all of the following sections since failing to follow this will cause your Flows to fail.

Extending a Responder Flow

Extending a Responder Flow works in a very similar way to extending an @InitiatingFlow Flow. The only difference is how it is called. As stated in the documentation:

Corda would detect that both BaseResponder and SubResponder are configured for responding to Initiator. Corda will then calculate the hops to FlowLogic and select the implementation which is the furthest distance, ie: the most subclassed implementation.

The statement, “most subclassed” is the important takeaway from this text. Therefore from a developer’s viewpoint, all they need to do is extend the external base Responder Flow and that’s it. I quite liked previous the requirements list, so let’s go through another one for extending Responder Flows:

Extend the base @InitiatedBy / Responder Flow
Add @InitiatedBy to the new Flow
Reference the base Flow’s constructor (super in Java)
Override any desired functions

If you are vigilant, you might have noticed that there is no mention of how to call it. The extending Responder Flow does not need to be called or referenced anywhere else. Corda will do the work to route everything to the right location.

Just to be sure, let’s have a quick look at an example:

@InitiatedBy(SendMessageFlow::class)
class CassandraSendMessageResponder(session: FlowSession) :
  SendMessageResponder(session) {

  override fun postTransactionSigned(transaction: SignedTransaction) {
    val message = transaction.coreTransaction.outputsOfType<MessageState>().single()
    logger.info("Signed transaction for message: $message")
  }

  override fun postTransactionCommitted(transaction: SignedTransaction) {
    val message = transaction.coreTransaction.outputsOfType<MessageState>().single()
    serviceHub.cordaService(MessageRepository::class.java).save(
      message,
      sender = false,
      committed = true
    )
    logger.info("Committed transaction for message: $message")
  }
}

Furthermore, let’s look back at the statement “most subclassed” again. The CassandraSendMessageResponder is a subclass of SendMessageResponder and is therefore chosen by Corda to handle requests from the Initiating Flow. But, this could be taken a step further. If there was another class, say SuperSpecialCassandraSendMessageResponder, this Flow is now what Corda will start using. Although I do find this sort of scenario somewhat unlikely at the moment, it is definitely worth knowing about.

Copying and pasting this statement again so you don’t forget:

You must ensure the sequence of sends/receives/subFlows in a subclass are compatible with the parent.

Overriding a Responder Flow

This is purposely a separate section. Here we will talk specifically about overriding a Responder Flow rather than extending one. Why would you do this and what is the difference? Answering the first question, a developer may want to write a Responder Flow that greatly diverges from the original base Flow but still needs to interact with the specific Initiating Flow provided by an external CorDapp. To achieve this they can override the Flow. Another word to describe this could be “replace”. The original base Flow is completely replaced by the overriding Flow. There is no involvement of extension in this situation.

I think the Corda documentation’s wording on this subject is quite good:

Whilst the subclassing approach is likely to be useful for most applications, there is another mechanism to override this behaviour. This would be useful if, for example, a specific CordApp user requires such a different responder that subclassing an existing flow would not be a good solution.

Hopefully, this extract along with my earlier description will clarify the difference between extending and overriding Responder Flows.

So, what might an overriding Flow look like? Well, anything you want really, within reason. Maybe it might look like the below, although I doubt it:

@InitiatedBy(SendMessageFlow::class)
class OverridingResponder(private val session: FlowSession) :
  FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    val stx = subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {}
    })
    logger.info("Screw the original responder. I'll build my own responder... with blackjack and hookers!")
    subFlow(
      ReceiveFinalityFlow(
        otherSideSession = session,
        expectedTxId = stx.id
      )
    )
  }
}

Since this Flow is completely replacing the original base Flow, it will look just like a normal Responder Flow. Since, well, it is one. That means it has @InitiatedBy referencing the Initiating Flow, extends FlowLogic and implements the call function.

Just putting this here one last time:

You must ensure the sequence of sends/receives/subFlows in a subclass are compatible with the parent.

This is even more prevalent here than in the previous sections. Since the whole call function is being overridden you must make sure that every send and receive is in the right place so interactions with the Initiating Flow run without errors.

Configuration wise there is a bit more to do than with extending a Flow. In this situation, we are trying to completely replace a Responder with another. To do so, we need a way to tell the node to redirect interactions from an Initiating Flow to a new overriding Responder Flow. Corda provides a way to do just that.

To specify the redirection add the following to your node.conf:

flowOverrides {
  overrides=[
    {
      initiator="com.lankydanblog.tutorial.base.flows.SendMessageFlow"
      responder="com.lankydanblog.tutorial.cassandra.flows.OverridingResponder"
    }
  ]
}

Obviously, change the classes referenced to your own…

So what is going on here? The config says that the SendMessageFlow which normally interacts with SendMessageResponder will now route to OverridingResponder instead.

To make everything a bit easier as well, the Cordform plugin provides the flowOverride method as part of deployNodes. This will then generate the configuration block above for you. For the example above, the following code was used:

node {
  name "O=PartyA,L=London,C=GB"
  p2pPort 10002
  rpcSettings {
    address("localhost:10006")
    adminAddress("localhost:10046")
  }
  rpcUsers = [[user: "user1", "password": "test", "permissions": ["ALL"]]]
  cordapp(project(':cordapp-contracts-states'))
  cordapp(project(':cordapp'))
  cordapp(project(':cordapp-extended-cassandra'))
  // the important part
  flowOverride("com.lankydanblog.tutorial.base.flows.SendMessageFlow",
    "com.lankydanblog.tutorial.cassandra.flows.OverridingResponder")
}

Now, after deployNodes has run and you have started your node, any requests coming from SendMessageFlow or any of its subclasses will now route communication to the OverridingResponder.

Conclusion

One of the handy features that Corda 4 provides is the ability to customise Flows from third-party CorDapps (or your own). This is done by two methods, extending or overriding.

Extending would be my first choice between the two but it does require a bit more effort on the side of the CorDapp developer. They must provide enough avenues for customisation without relinquishing control of the original functionality of their Flows. Providing little customisation might not deter other developers from using their CorDapp. But, developers could become unhappy with the lack of control of their own application. It is a slippery slope to control original intent with routes for customisation. On the other hand, actually extending a Flow does not require much work, making it easier for developers to adopt and adapt external Flows.

Overriding, on the other hand, requires no work for a CorDapp developer and instead everything is put onto the developer leveraging external Responder Flows. That is because the existing Flow is pretty much being thrown away and the only reference back to the original implementation is the link to the Initiating Flow.

By embracing both the extending and overriding of Flows, CorDapp developers will be able to leverage external CorDapps while still providing enough customisation to fulfil all business requirements they might have. As time progresses, developers will drive the adoption of reusing existing CorDapps as they provide access to additional customisation, soon taking the same position as Open Source libraries that we all already leverage in any work we do.

The code used in this post can be found on my GitHub. It contains the code for CassandraSendMessageFlow which sets up a connection to an external Cassandra database to save tracing style data. It also contains another module that sends HTTP requests as part of its extension of the base Flows. If you are still curious after reading this post, this repository might help.

If you enjoyed this post or found it helpful (or both) then please feel free to follow me on Twitter at @LankyDanDev and remember to share with anyone else who might find this useful!

Validating external data with an Oracle

Tue, 01 Jan 2019 00:00:00 GMT

I hang out in the Corda Slack channel quite a lot and try to answer questions when I can. A reasonable number of questions I have attempted to answer are related to Oracles. More specifically, when to use one. I feel like I can answer that, “Use an Oracle when you need to validate external data that can change frequently”. I probably wrote an answer similar to that at some point. What I could not do though… Was tell someone how to implement one. Therefore, to correct that. I have written this post to learn how to implement one myself and share that knowledge with you and my future self.

When to use an Oracle

Let’s start with expanding on when to use an Oracle. As I touched on a minute ago, you should use an Oracle when you need to validate external data that can change frequently. This could be data such as exchange rates, stock prices or even whether my blog is currently up or down (although I haven’t seen it be down yet!). I think the frequently part is important here. If data seldomly changes, it might be feasible to validate some data against an attachment containing the same sort of values the Oracle would be retrieving themselves. That is why validating data such as exchange rates, in my opinion, should only be done by an Oracle. All that being said, it really comes down to your particular use-case.

How to use an Oracle

How does the Oracle do this validation? Well, that is up to you. But, it will likely follow these steps:

Receive data from a node
Retrieve external data
Validate the received data against the external data
Provide a signature for the transaction

These are the steps that I think most Oracle implementations will comprise of. More steps could be added in and the validation that is done could be as complex or simple as the use-case demands. Although more steps could be added, I really doubt that there would be much use out of an Oracle that excluded any of the steps shown above.

All the steps shown above only show the process from the side of the Oracle. There is a bit more going on so I think a good diagram will help us out here. It will also go on to present the example that I will use for this post.

Quite a few of these steps are generic ones that are going to be made in whatever Flow you put together. In this section, I will expand and show the code involved to implement the Flow shown in the diagram. Therefore it is worth having a reasonable look at it… I also spent a lot of time making that look nice, so please look at it!!

Oh, another point before I continue. I want to highlight how helpful putting together sequence diagrams to model Corda Flows is. It really highlights who is involved, how many network hops need to be made and how much work each participant does. Furthermore, they go a good way of explaining what is going on to people who are only interested in the higher level processes that you are architecting and/or implementing.

Client / Not the Oracle side

As I mentioned before, some of the code here is generic code that you are likely to put into any Flow that you write. I have shown it all so there is no ambiguity around what is happening, but I will only expand on points that need to be highlighted as they contain code specific to interacting with an Oracle.

@InitiatingFlow
@StartableByRPC
class GiveAwayStockFlow(
  private val symbol: String,
  private val amount: Long,
  private val recipient: String
) :
  FlowLogic<SignedTransaction>() {

  @Suspendable
  override fun call(): SignedTransaction {
    val recipientParty = party()
    val oracle = oracle()
    val transaction =
      collectRecipientSignature(
        verifyAndSign(transaction(recipientParty, oracle)),
        recipientParty
      )
    val allSignedTransaction = collectOracleSignature(transaction, oracle)
    return subFlow(FinalityFlow(allSignedTransaction))
  }

  private fun party(): Party =
    serviceHub.networkMapCache.getPeerByLegalName(CordaX500Name.parse(recipient))
      ?: throw IllegalArgumentException("Party does not exist")

  private fun oracle(): Party = serviceHub.networkMapCache.getPeerByLegalName(
    CordaX500Name(
      "Oracle",
      "London",
      "GB"
    )
  )
    ?: throw IllegalArgumentException("Oracle does not exist")

  @Suspendable
  private fun collectRecipientSignature(
    transaction: SignedTransaction,
    party: Party
  ): SignedTransaction {
    val signature = subFlow(
      CollectSignatureFlow(
        transaction,
        initiateFlow(party),
        party.owningKey
      )
    ).single()
    return transaction.withAdditionalSignature(signature)
  }

  private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
    transaction.verify(serviceHub)
    return serviceHub.signInitialTransaction(transaction)
  }

  private fun transaction(recipientParty: Party, oracle: Party): TransactionBuilder =
    TransactionBuilder(notary()).apply {
      val priceOfStock = priceOfStock()
      addOutputState(state(recipientParty, priceOfStock), StockContract.CONTRACT_ID)
      addCommand(
        GiveAway(symbol, priceOfStock),
        listOf(recipientParty, oracle).map(Party::owningKey)
      )
    }

  private fun priceOfStock(): Double =
    serviceHub.cordaService(StockRetriever::class.java).getCurrent(symbol).price

  private fun state(party: Party, priceOfStock: Double): StockGiftState =
    StockGiftState(
      symbol = symbol,
      amount = amount,
      price = priceOfStock * amount,
      recipient = party
    )

  private fun notary(): Party = serviceHub.networkMapCache.notaryIdentities.first()

  @Suspendable
  private fun collectOracleSignature(
    transaction: SignedTransaction,
    oracle: Party
  ): SignedTransaction {
    val filtered = filteredTransaction(transaction, oracle)
    val signature = subFlow(CollectOracleStockPriceSignatureFlow(oracle, filtered))
    return transaction.withAdditionalSignature(signature)
  }
  
  private fun filteredTransaction(
    transaction: SignedTransaction,
    oracle: Party
  ): FilteredTransaction =
    transaction.buildFilteredTransaction(Predicate {
      when (it) {
        is Command<*> -> oracle.owningKey in it.signers && it.value is GiveAway
        else -> false
      }
    })
}

@InitiatedBy(GiveAwayStockFlow::class)
class SendMessageResponder(val session: FlowSession) : FlowLogic<Unit>() {
  @Suspendable
  override fun call() {
    subFlow(object : SignTransactionFlow(session) {
      override fun checkTransaction(stx: SignedTransaction) {}
    })
  }
}

First, let’s have a look at how the transaction is built:

private fun transaction(recipientParty: Party, oracle: Party): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    val priceOfStock = priceOfStock()
    addOutputState(state(recipientParty, priceOfStock), StockContract.CONTRACT_ID)
    addCommand(
      GiveAway(symbol, priceOfStock),
      listOf(recipientParty, oracle).map(Party::owningKey)
    )
  }

private fun priceOfStock(): Double =
  serviceHub.cordaService(StockRetriever::class.java).getCurrent(symbol).price

There is not too much different here from how I would create a transaction that does not involve an Oracle. The only two differences are, retrieving the stock price from an external source (hidden inside the StockRetriever service) and including the signature of the Oracle in the Command. These code additions line up with the reasons for using an Oracle. External data is included in the transaction and the Oracle needs to verify it is correct. To prove that the Oracle has deemed a transaction valid, we need its signature.

We will look closer at retrieving the external data separately.

Next up is collecting the recipients signature:

@Suspendable
private fun collectRecipientSignature(
  transaction: SignedTransaction,
  party: Party
): SignedTransaction {
  val signature = subFlow(
    CollectSignatureFlow(
      transaction,
      initiateFlow(party),
      party.owningKey
    )
  ).single()
  return transaction.withAdditionalSignature(signature)
}

Collecting the counterparty’s signature is not really an uncommon step of a Flow but what is done differently here is the use of CollectSignatureFlow rather than the CollectSignaturesFlow which is normally used (noticed the “s” missing in the middle). That is due to requiring the Oracle’s signature in the transaction. Calling the CollectSignaturesFlow will go off to retrieve signatures from all required signers, including the Oracle. This treats the Oracle like a “normal” participant. This is not what we want. Instead, we need to get the signature of the recipient and the Oracle individually and somewhat manually. The manual part is the use of transaction.withAdditionalSignature.

Now that the recipient has signed the transaction, the Oracle needs to sign it:

@Suspendable
private fun collectOracleSignature(
  transaction: SignedTransaction,
  oracle: Party
): SignedTransaction {
  val filtered = filteredTransaction(transaction, oracle)
  val signature = subFlow(CollectOracleStockPriceSignatureFlow(oracle, filtered))
  return transaction.withAdditionalSignature(signature)
}

private fun filteredTransaction(
  transaction: SignedTransaction,
  oracle: Party
): FilteredTransaction =
  transaction.buildFilteredTransaction(Predicate {
    when (it) {
      is Command<*> -> oracle.owningKey in it.signers && it.value is GiveAway
      else -> false
    }
  })

Before sending the transaction to the Oracle, it is recommended to filter it to remove any information that is not needed by the Oracle. This prevents information that shouldn’t be shared from being seen by the Oracle. Remember the Oracle is likely to be a node controlled by another organisation and is not a participant that you are attempting to share states and transactions with.

SignedTransaction provides the buildFilteredTransaction function that only includes objects that match the predicate passed in. In the example above it filters out everything but the GiveAway (command I created) command which must also have the Oracle as a signer.

This outputs a FilteredTransaction which is passed to CollectOracleStockPriceSignatureFlow:

@InitiatingFlow
class CollectOracleStockPriceSignatureFlow(
  private val oracle: Party,
  private val filtered: FilteredTransaction
) : FlowLogic<TransactionSignature>() {

  @Suspendable
  override fun call(): TransactionSignature {
    val session = initiateFlow(oracle)
    return session.sendAndReceive<TransactionSignature>(filtered).unwrap { it }
  }
}

All this code does is send the FilteredTransaction to the Oracle and awaits its signature. The code here could be put into the main Flow but it is quite nice to split code out when we can.

Finally, the TransactionSignature returned from the Oracle is added to the transaction in the same way the recipient’s signature was added earlier. At this point, the transaction is ready to be committed as all the required signers have done their part.

Oracle side

Now that we have covered the client side of the code, we need to have a look into how the Oracle validates the transaction. Below is the contents of the Oracle code:

@InitiatedBy(CollectOracleStockPriceSignatureFlow::class)
class OracleStockPriceSignatureResponder(private val session: FlowSession) : FlowLogic<Unit>() {

  @Suspendable
  override fun call() {
    val transaction = session.receive<FilteredTransaction>().unwrap { it }

    val key = key()

    val isValid = transaction.checkWithFun { element: Any ->
      when {
        element is Command<*> && element.value is GiveAway -> {
          val command = element.value as GiveAway
          (key in element.signers).also {
            validateStockPrice(
              command.symbol,
              command.price
            )
          }
        }
        else -> false
      }
    }

    if (isValid) {
      session.send(serviceHub.createSignature(transaction, key))
    } else {
      throw InvalidStockPriceFlowException("Transaction: ${transaction.id} is invalid")
    }
  }

  private fun key(): PublicKey = serviceHub.myInfo.legalIdentities.first().owningKey

  private fun validateStockPrice(symbol: String, price: Double) = try {
    serviceHub.cordaService(StockPriceValidator::class.java).validate(symbol, price)
  } catch (e: IllegalArgumentException) {
    throw InvalidStockPriceFlowException(e.message)
  }
}

Some of the code that should be here is hidden in the StockPriceValidator which retrieves the external stock price and compares it to the one passed to the Oracle. It doesn’t have much code and its validation is basic so I won’t elaborate on it. As it is short, I might as well show it now:

@CordaService
class StockPriceValidator(private val serviceHub: AppServiceHub) :
  SingletonSerializeAsToken() {

  fun validate(symbol: String, price: Double) =
    serviceHub.cordaService(StockRetriever::class.java).getCurrent(symbol).let {
      require(price == it.price) { "The price of $symbol is ${it.price}, not $price" }
    }
}

Back to the OracleStockPriceSignatureResponder. Firstly, receive is called to get the FilteredTransaction that the client sent. It is then checked using its checkWithFun function. This is a handy function that looks at each object and expects a Boolean in return. Using this, the transaction is deemed valid if all it contains are commands of GiveAway where the Oracle is the signer and most importantly checks that the external data contained in the command is correct. If you recall the code from earlier, the correct command and signers were passed in. The only remaining validation is on the external data. If that is all fine then the Oracle will accept the transaction and send its signature back to the client that requested it.

I chose to complete the validation via throwing exceptions (along with error messages) which are then propagated to the requesting party. I think this makes it easier on to understand what has gone wrong so it can be handled properly, rather than just a straight “failed validation” message. If the validation the Oracle is performing is complex, these error messages become even more valuable.

Retrieving external data

You should have seen the StockRetriever class pop up twice now. It was used in both the requesting party and the Oracle. I have shared this code between both types of nodes (normal nodes and Oracles) but this might not be suitable for your own use-case. Furthermore, how you choose to retrieve your external data is up to you, I am just providing a possible solution.

The code can be found below:

@CordaService
class StockRetriever(serviceHub: AppServiceHub) :
  SingletonSerializeAsToken() {

  private val client = OkHttpClient()
  private val mapper = ObjectMapper()

  fun getCurrent(symbol: String): Stock {
    val response = client.newCall(request(symbol)).execute()
    return response.body()?.let {
      val json = it.string()
      require(json != "Unknown symbol") { "Stock with symbol: $symbol does not exist" }
      val tree = mapper.readTree(json)
      Stock(
        symbol = symbol,
        name = tree["companyName"].asText(),
        primaryExchange = tree["primaryExchange"].asText(),
        price = tree["latestPrice"].asDouble()
      )
    } ?: throw IllegalArgumentException("No response")
  }

  private fun request(symbol: String) =
    Request.Builder().url("https://api.iextrading.com/1.0/stock/$symbol/quote").build()
}

The StockRetriever is a nice little service that uses an OkHttpClient (OkHttp) to make a HTTP request to an API (provided by IEX Trading using their Java library) that returns stock information when a stock symbol is provided. You can use whatever client you want to make the HTTP request. I saw this one in an example CorDapp and have taken it for my own. Personally, I’m too used to Spring so didn’t really know any clients other than their RestTemplate.

Once the response is returned, it is converted into a Stock object and passed back to the function’s caller. That’s all folks.

Conclusion

In conclusion, you should use an Oracle when your CorDapp requires frequently changing external data that needs to be validated before the transaction can be committed. Just like data held within states, external data is extremely important, probably the most important, as it is likely to determine the main contents of a transaction. Therefore all participants must feel comfortable that the data is correct and hasn’t just come out of thin air. To achieve this, an Oracle will also retrieve the external data and validate it against what the transaction says the data should be. At this point, the Oracle will either sign the transaction or throw an exception and deem it invalid. The implementation side of this is reasonably straightforward as there are not many steps that need to be taken. Retrieve the data, send a FilteredTransaction to the Oracle containing the data where it will be validated. Yes, as you have read this post, you will know there is a bit more to it. But, for a basic Flow that is pretty much it. As I said somewhere near the start, how the Oracle does its validation can be as simple or complicated as required. Although, I think most will follow the same sort of process shown here.

Now for the main conclusion… In conclusion, you now have the knowledge to answer questions in the slack channel about Oracles or know where to send them if you can’t!

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Throughput - A Corda story

Fri, 14 Dec 2018 00:00:00 GMT

I recently rolled off a project focusing on the performance of Corda for a particular use-case. The result of this project lead to us processing 115 million requests within a single day over a network of over 170 nodes. Furthermore, Corda was able to handle 6300 requests per second confirming that the highest requirement of the network was met. This is by far the largest Corda network that has been deployed and the highest throughput achieved to date. Proving that Corda can deliver in a very demanding environment.

The study was conducted by Accenture for DTCC, which also looked into another DLT platform, Digital Asset. Further information can be found in the press release.

DTCC Announces Study Results Demonstrating that DLT Can Support Trading Volumes in the US Equity… Groundbreaking study proves DLT can process more than 100 million trades per day New York/London/Hong…www.dtcc.com

In this post, I will use the experience I gained on this project to describe how you can also get the best out of Corda. I expect that there will be similar projects to what we did for DTCC coming up in the near future and I hope that the information here will help point other developers in the right direction.

So how did it go? Well, it wasn’t without its difficulties. But, I think we did well. More precisely, we proved that Corda can hit a high throughput if the network is thoughtfully architected and your CorDapps are well designed. Yes, I know that I’m basically saying if you do everything right then it will go well. It really is that important. As we tweaked our CorDapps throughout the project we found avenues to greatly improve the application’s performance. Making these changes allowed us to move closer and closer to the targets we were aiming for. But, none of this would have mattered if we had not architected our network in a particular way.

Corda Enterprise is needed to achieve the highest possible performance

This is the simplest way to increase your performance by a factor of 10 or by however many cores your machine has. Among other things, Corda Enterprise allows the number of Flow workers running within the node to increase from 1 to many. This affects the number of Flows that can run asynchronously within the node. But, this does not change the performance of individual Flows which run at the same speed on each version. Realistically, without Enterprise you will never achieve extremely high performance targets.

If achieving this sort of performance is not a requirement for your use-case, then the Open Source version will meet your needs. For example, we 100% needed to use Enterprise for our project since the volumes of requests that DTCC handle and the rate that they need to be processed is enormous. On the other hand, if we were dealing with the handling and processing of Loans. The rate that requests flow through a node would drop significantly compared to DTCC’s needs. In this situation, using Open Source would suffice.

Conveniently, Enterprise and Open Source code are compatible allowing you to switch over without too much pain. There are differences in deployment and it is extremely likely you will need to fiddle around with this side when making the change.

The compatibility between the Enterprise and Open Source versions will allow you to try both, allowing you to determine which best suits your needs. This allows you to start writing your application on Open Source until it is deemed necessary to switch to Enterprise.

Think about your Network

I really, really want to emphasise the importance of network architecture. I don’t even want to think about the performance we would have achieved if we stuck to our original design. We actually scrapped our original design since it was fundamentally flawed and would have prevented any hopes of reaching our goals. Personally, I think this section is half about Corda and half about architecting a good solution.

Sharding to improve performance at scale

At the time of writing this post, Corda does not support Load Balancing. Currently, a single node processes all of the work for the identity it represents. This is an area they are fully aware of and is something they have on their roadmap to work on in the near future. If this was there, then it might have been possible to simply rely on spinning up a ton of instances that support a single Corda node. This would have led to more compute power and therefore increasing throughput.

Due to load balancing not being ready and the fact that we had a single actor sitting in the middle of the network who was a massive bottleneck to the handling of requests, we had to approach the overall network design differently. To compensate, we had to think of a way to provide our own sort of horizontal scaling into the system as the single node sitting in the middle had to be removed. If these issues were not addressed, there was no way we could achieve the network throughput of 6300 trades per second.

Our solution? Sharding. We determined a way that this one actor could be logically split up into many, much smaller parts. Each handling requests in parallel to each other. This required a bit of extra logic to route the request to the right sharded node. But, this solution likely saved the project. We never tested the performance of the single bottleneck node, but, I am 100% sure we would not have reached out targets.

Below I have included two diagrams. One of an example process using the single node design and another of the sharded approach.

Single node

Sharded

I will let the diagrams speak for themselves. I will not delve any further into the implementation since that information is still confidential. There should be enough information there to understand why and what we did, but not how we achieved it.

As you can imagine the second design will yield a much higher throughput. It also comes with the benefit of scaling linearly as nodes are added to the network.

With the original design, the throughput might be acceptable with a small number of nodes. But, as soon as you hit larger numbers, say 100s, maybe even as small as 10, you are going to notice the levelling out of performance. This is completely due to the whole network relying on the rate at which the single bottleneck node can process requests.

Removing extra bottlenecks with multiple Notaries

Another area that will improve the network’s overall performance is the use of multiple Notaries. When the throughput of the network is already high, a single Notary will start to become the bottleneck of the workflow. By following the same idea as the previous section. A Notary can be sharded. Allowing each to process a smaller volume of transactions.

Every time I say “multiple Notaries”, I just feel like I need to clarify that I am not talking about a Notary Cluster.

I have already written a post, Increasing network throughput with multiple Notaries, covering this topic and rather than repeating myself, I will direct you there instead.

Increasing network throughput with multiple Notaries

Do you need a very high throughput Corda network? Has the network's throughput levelled out? Have you already squeezed…

Tweak those Cordapps

Onto Cordapps. There is a lot you can do to improve performance here. Most of it comes from trying to do as little as possible.

Do I need to do send all these transactions?
Does this other party really need to sign this transaction?
Do my transactions have too many states on them?
How many times are Flows jumping between the initiator and counterparties?

These are all questions that are important to the performance of your Flows. I’m sure there are other places to gain performance (one I will touch on later) but these are the only ones I can think of right now. I’m sure you got the picture though.

Let’s have a quick look at the last question.

How many times are Flows jumping between the initiator and counterparties?

This actually encompasses some of the other points I made. Anyway. Each time a jump across the network is made, the performance of the Flow goes down. It needs to travel from one Corda node to another and it likely needs to come back at some point. During this time, you are accumulating performance costs due to network latency and the process of checkpointing Flows to disk.

Network latency speaks for itself and doesn’t require further explanation. Checkpointing, on the other hand, requires a bit of fleshing out. Checkpointing is the process of serialising a Flow’s current execution so that it can be restarted from a specific point in case of failure. Doing so requires serialising the whole stack of the Flow which can be quite large and therefore a costly process to perform.

Taking this information into account, make sure you think about whether you really need to make these jumps. Try to keep them to a minimum. If you do this, I’m sure you will see your application’s performance increase.

Multi-threading? That is good for performance right?

Yes, yes, yes. Although, we didn’t measure the impact including multi-threading made I am sure it made a good improvement. But, be careful. If you don’t do it correctly then you might get bit in the arse. At of the time of writing, Corda does not support multi-threading within Flows. You will get some weird errors if you do. That being said, it is possible. You can do so from within a Corda Service which runs slightly outside of the territory of a Flow. By delegating some processing to a service, we were able to make use of threads to start new Flows which each run asynchronously, processing similar but separated tasks.

I have written about this in my earlier post Asynchronous Flow invocations with Corda Services which dives into this topic and to why you might end up falling over when trying this.

Asynchronous Flow invocations with Corda Services

How can I make my Flows faster? There's a good chance you have thought about this before if you have been working with…

What is it like using Corda

I found using Corda relatively simple. It does get more difficult as you try to implement more complex use-cases. But, for the most part, a lot of Flows can follow the same simple structure. Add some states to a transaction, verify it, get all required parties to sign it and commit the transaction.

As it gets more complicated, you need to keep in mind which party needs to do what. For example, spending cash. As an initiator, you can’t put other people’s cash states into a transaction. You need to send them some information and request them to add it to a transaction. Scenarios like this took me a while to get to grips with. As more developers spend time working with Corda, I am sure these concepts will become easier to understand. More examples will be published and knowledge of how to write good Flows will be distributed.

Furthermore, I stand by the Key Concepts that Corda have produced. Going through these and the documentation provided took me pretty far in my understanding of Corda.

Key concepts - R3 Corda V3.3 documentation

This section describes the key concepts and features of the Corda platform. It is intended for readers who are new to…docs.corda.net

Moving forward

Now, I don’t speak for Corda or R3, but since we worked closely with them throughout this project I can speak of possible improvements to the platform.

Making it easier to deploy multiple Corda nodes. R3 worked with us to produce a framework to deploy nodes more easily which can likely be adapted and generalised to work for a wider audience.
Performance. There were a few areas within the Corda code that could be tweaked to make way for good performance gains.
Better multi-threading. As I mentioned earlier, this can be done within Corda Services but it might be possible to shift some of this into Flows. Mainly focusing on starting multiple subFlows asynchronously and awaiting their completion.

Wrapping up

Towards the end of the project, it was definitely hectic but the massive performance increase that we were able to make within 1 month is crazy. Once we improved our CorDapps to eke every bit of performance out of them, our numbers went from “meh” to “wow”. Thankfully, we designed our network correctly to make these numbers possible. All the tweaking in the world would not have saved it if the network wasn’t put together in the way it was.

So, can you get good throughput with Corda? Yes. Yes, you can. Using Corda Enterprise will make higher performance targets more achievable and allow you to do so with less work on your end. But, that is not really the right mindset to be in. Using the information I covered in this post, you should have a better understanding of how to design a high performance Corda application or network.

Going forward, Corda’s performance is only going to get better. Combining that with a good idea of how to design your application, should allow your numbers to shoot through the roof.

Lastly, before I close this post, I just want to thank R3 and especially Stefano for working closely with us during this project.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Opinions and views found in my posts are my own and do not represent Accenture’s views on any subject.

Increasing network throughput with multiple Notaries

Sun, 18 Nov 2018 00:00:00 GMT

Do you need a very high throughput Corda network? Has the network’s throughput levelled out? Have you already squeezed out all the performance you could from other areas? If your answers to these questions are “yes”, then I may have some useful information for you. I have listed out these questions to decrease the chance of you prematurely optimising your Corda network/application. Switching to using multiple Notaries is only going to have a noticeable performance impact if it is one of the slowest parts involved in processing requests/transactions. It is highly likely that other areas need to be improved before looking into using multiple Notaries.

Before I continue. I really need to say this. I am not talking about using Notary Clusters in this post, which consist of Notaries that communicate with each other to reach consensus on whether states have been spent or not. I am talking about having multiple Notaries, each with their own identities, who only interact with the nodes that send them transactions to validate. This distinction needs to be made and should remove any confusion about exactly what I will be describing in this post.

At the time of writing, the current versions of Corda are:

Open Source 3.3
Enterprise 3.2

Why would I want to do this?

Ok, so. Let’s really dig into why you would want to use multiple Notaries. Diagrams do this best, so let’s use one:

This situation doesn’t look great. But, it might not actually be that bad. If the throughput of your network is not extremely high, this architecture should be able to handle the transactions passing through the Notary.

As mentioned in the introduction. It becomes an issue when the rate of transactions being sent to the Notary becomes very high. Once this point is reached, the Notary will start to lag behind. As it cannot validate the states in the transactions fast enough. If performance is important to the network, this is a good area to examine.

From a code perspective, this is the standard format that you have probably already been writing your CorDapps. You pick out a Notary based on particular criteria and send a transaction there. There might even be only a single Notary in the whole network that you have been dealing with. For example, in all my code examples I have produced before I have written code similar to the below that only relies on the single Notary in the network and blindly takes that one every time.

private fun notary(): Party = serviceHub.networkMapCache.notaryIdentities.first()

Switching to multiple Notaries

Moving from a network that relies on a single Notary to a design that consists of many, fundamentally, requires two things:

More than one Notary in the network
An algorithm to choose which Notary to send a transaction to

Furthermore, the chosen Notary for a transaction is then referenced by future transactions if consuming states. If you end up in a situation where input states from different Notaries are being consumed, then you must perform a Notary Change Transaction. I will cover this topic later on.

Below is how the previous design could be altered to use a few Notaries:

The best thing about this diagram is that it illustrates how simple it is to add another Notary to the network and redistribute the load among them. There is nothing stopping us from adding more and more Notaries to the network. But, there will be a point where adding more does not lead to a performance increase. This keeps coming back to what I have mentioned previously. That adding more Notaries will only increase throughput when the Notaries themselves are reaching saturation.

Choosing a Notary for issuance transactions

Below is a possible algorithm to choose which Notary to use:

private fun transaction(): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    addOutputState(message, MessageContract.CONTRACT_ID)
    addCommand(Send(), message.participants.map(Party::owningKey))
  }

private fun notary(): Party {
  val index = message.type.hashCode() % serviceHub.networkMapCache.notaryIdentities.size
  return serviceHub.networkMapCache.notaryIdentities.single { it.name.organisation == "Notary-$index" }
}

In this example, the transaction chooses the Notary to use based on the hashCode of one of the input state’s properties and the number of Notaries in the network.

How you choose the Notary could be as simple or complex as you need it to be. This will depend on requirements such as only a subset of Notaries being trusted for proposed transactions or resiliency to the Notaries in the network changing.

Choosing a Notary when consuming states from the same Notary

This is nice and simple… If all input states reference the same Notary. Below is what it looks like (this example only consumes a single input… because I am too lazy to write another version):

private fun transaction(response: MessageState): TransactionBuilder =
  TransactionBuilder(notary()).apply {
    addInputState(message)
    addOutputState(response, MessageContract.CONTRACT_ID)
    addCommand(Reply(), response.participants.map(Party::owningKey))
  }

private fun notary(): Party = message.state.notary

As you can see, all the transaction does is retrieve the Notary that is related to the input state and uses it for itself. This information can be extracted because message is a StateAndRef and accessing its state property will return a TransactionState. Following this format. Creating new transactions that consume a state and produce a number of outputs is straightforward. This format is also valid for multiple input states. If, and only if, they all reference the same Notary.

So… With all this talk about input states with different Notaries. I should probably discuss it further.

Choosing a Notary when consuming states from different Notaries

Here we have to be careful or we will see errors like the one below:

java.lang.IllegalArgumentException: Input state requires notary "O=Notary-1, L=London, C=GB"
which does not match the transaction notary "O=Notary-0, L=London, C=GB".

The error shows that an input state does not have the same Notary as the transaction that contains it.

To solve this error we need to use a Notary Change Transaction. As per the docs:

A flow to be used for changing a state’s Notary. This is required since all input states to a transaction must point to the same notary.

I wanted to put that in there, just in case you think I am a lier!

The code to perform a Notary Change Transaction looks like the below:

@Suspendable
private fun notaryChange(
  message: StateAndRef<MessageState>,
  notary: Party
): StateAndRef<MessageState> =
  if (message.state.notary != notary) {
    subFlow(
      NotaryChangeFlow(
        message,
        notary
      )
    )
  } else {
    message
  }

I am sure you can figure out what is going on yourself, but to make myself feel smarter… I am going to tell you. message represents an input state and notary is the Notary that the new transaction will be using. If the Notaries are the same, then the state can be returned as nothing needs to be done to it. If they are indeed different, then call NotaryChangeFlow which takes in the two arguments passed into the original function. This will return a new StateAndRef which is then returned from the function.

The StateAndRef returned from this function can then be put into the transaction.

If you are not sure whether the states being passed into a transaction are from the same Notary, then I suggest sticking to the code in this section. Choose a Notary that the transaction will use, whether this is a specific one or one taken from the input states and perform a Notary Change Transaction on any that require it. For example, I think code similar to the below would make a generic and robust solution:

@Suspendable
private fun transaction(): TransactionBuilder {
  val messages = getMessageStates()
  val notary = notary()
  return TransactionBuilder(notary).apply {
    messages.forEach {
      addInputState(notaryChange(it, notary))
    }
    addCommand(
      Delete(),
      (messages.flatMap { it.state.data.participants }.toSet() + ourIdentity).map(Party::owningKey)
    )
  }
}

@Suspendable
private fun notaryChange(
  message: StateAndRef<MessageState>,
  notary: Party
): StateAndRef<MessageState> =
  if (message.state.notary != notary) {
    subFlow(
      NotaryChangeFlow(
        message,
        notary
      )
    )
  } else {
    message
  }

// however you want to choose your specific Notary
private fun notary(): Party =
  serviceHub.networkMapCache.notaryIdentities.single { it.name.organisation == "Notary-1" }

Here a specific Notary is chosen for the transaction, each input has its Notary changed to the chosen one if required and the signers comprise of all the participants of the consumed states. This might not suit your own use-case. Which is perfectly fine. But this should provide a good starting point when playing around with changing Notaries (mainly for performance).

Altering this solution slightly, we can instead choose the Notary based on the Notaries the input states reference. Since only the notary function really needs to change I have excluded the rest of the code from the example.

private fun notary(messages: List<StateAndRef<MessageState>>): Party =
  messages.map { it.state.notary }
    .groupingBy { it }
    .eachCount()
    .maxBy { (_, size) -> size }?.key ?: throw IllegalStateException("No Notary found")

The Notary chosen by this function is decided based on the most common Notary shared by the input states. By doing so, less Notary Change Transactions are required as the largest majority of the inputs will already reference the chosen Notary. This should provide the best performance if you do not know which Notaries the inputs reference.

Conclusion

Achieving high performance within a Corda network relies on removing bottlenecks out of the system and other general performance tweaks. One such bottleneck is the Notary. In a situation where a very high throughput is passing through the Notary, the network’s performance will start to plateau. The Notary cannot process the requests fast enough for the rate that they are coming in. Moving to use multiple Notaries that share the request load will allow the performance of the network to increase. This brings extra complexity in determining which Notary to use along with the possibility of needing Notary Change Transactions. But, if your network really needs to achieve a high throughput. This will be an area that is worth looking into.

One last comment I will throw in here. As the internal performance of Notaries increase, the need for this sort of architecture will decrease. There might even reach a point where a single Notary is able to completely handle a large rate of incoming requests. This is an area to keep an eye on as Corda continues to improve its overall performance.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Starting Flows with trackBy

Fri, 05 Oct 2018 00:00:00 GMT

Still continuing my trend of looking at Corda Services, I have some more tips to help your CorDapp work smoothly. This time around, we will focus on using trackBy to initiate Flows from inside a Service and the discrete problem that can arise if you are not careful.

This should be a relatively short post as I can lean upon the work in my previous posts: Corda Services 101 and Asynchronous Flow invocations with Corda Services. The content found in Asynchronous Flow invocations with Corda Services is very relevant to this post and will contain extra information not included within this post.

This post is applicable to both Corda Open Source and Enterprise. The versions at the time of writing are Open Source 3.2 and Enterprise 3.1.

A brief introduction to trackBy

trackBy allows you to write code that executes when a transaction containing states of a specified type completes. Whether they are included as inputs or outputs, the code will still trigger.

From here, you can decide what you want it to do. Maybe something very simple, like logging that a state has been received. Or, maybe something more interesting, such as initiating a new Flow. This use-case makes perfect sense for this feature. Once a node receives a new state or consumes one, they can start a new Flow that represents the next logical step in a workflow.

Furthermore, there are two versions of trackBy. One, the trackBy I keep mentioning, that can be used within a CorDapp. The other, vaultTrackBy, is called from outside of the node using RPC.

The problems presented in this post are only present in the CorDapp version, trackBy. Therefore, we will exclude vaultTrackBy for the remainder of this post.

What is this discrete problem?

Deadlock. When I word it that way, it isn’t very discrete. But, the way it happens is rather subtle and requires a good understanding of what is going on to figure it out. As mentioned before, this issue is very similar to the one detailed in Asynchronous Flow invocations with Corda Services. Furthermore, another shoutout to R3 for diagnosing this problem when I faced it in a project and I am sure they are going to iron this out. Until then, this post should save you some head scratching in case you run into the same problem.

I will quote what I wrote in my previous post as its explanation is only missing one point in regards to this post.

The Flow Worker queue looks after the order that Flows execute in and will fill and empty as Flows are added and completed. This queue is crucial in coordinating the execution of Flows within a node. It is also the source of pain when it comes to multi-threading Flows ourselves.

Why am I talking about this queue? Well, we need to be extra careful not to fill the queue up with Flows that cannot complete.

How can that happen? By starting a Flow within an executing Flow who then awaits its finish. This won’t cause a problem until all the threads in the queue’s thread pool encounter this situation. Once it does happen, it leaves the queue in deadlock. No Flows can finish, as they all rely on a number of queued Flows to complete.”

That marks the end of my copypasta. I am going to keep saying this though, really, I suggest you read through Asynchronous Flow invocations with Corda Services for a thorough explanation into this subject.

What has this got to do with trackBy? Calling trackBy from a Service will run each observable event on a Flow Worker thread. In other words, each event takes up a spot on the queue. Starting a Flow from here will add another item to the queue and suspend the current thread until the Flow finishes. It will stay in the queue until that time. If you end up in a situation where all the spots on the queue are held by the observable events, rather than actual Flows, I got one word for you. Deadlock. It is the exact same situation I’ve detailed before but starting from a different epicenter.

On the bright side, the solution is a piece of cake (where did this saying come from anyway?).

The section where the problem is fixed

Now that you know what the problem is. Altering a “broken” version to one shielded from deadlock will only require a few extra lines.

Let’s take a look at some code that is very similar to what lead me to step onto this landmine:

@CordaService
class MessageObserver(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  private companion object {
    val log = loggerFor<MessageObserver>()
  }

  init {
    replyToNewMessages()
    log.info("Tracking new messages")
  }

  private fun replyToNewMessages() {
    val ourIdentity = ourIdentity()
    serviceHub.vaultService.trackBy<MessageState>().updates.subscribe { update: Vault.Update<MessageState> ->
      update.produced.forEach { message: StateAndRef<MessageState> ->
        val state = message.state.data
        if (state.recipient == ourIdentity) {
          log.info("Replying to message ${message.state.data.contents}")
          serviceHub.startFlow(ReplyToMessageFlow(message))
        }
      }
    }
  }

  private fun ourIdentity(): Party = serviceHub.myInfo.legalIdentities.first()
}

This Service uses trackBy to start a new Flow whenever the node receives new MessageStates. For all the reasons mentioned previously, this code has the potential to deadlock. We don’t know when, or if it will ever happen. But, it could. So we should probably sort it out before it is an issue.

The code below will do just that:

@CordaService
class MessageObserver(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  private companion object {
    val log = loggerFor<MessageObserver>()
    // executor added
    val executor: Executor = Executors.newFixedThreadPool(8)!!
  }

  // init

  private fun replyToNewMessages() {
    val ourIdentity = ourIdentity()
    serviceHub.vaultService.trackBy<MessageState>().updates.subscribe { update: Vault.Update<MessageState> ->
      update.produced.forEach { message: StateAndRef<MessageState> ->
        val state = message.state.data
        if (state.recipient == ourIdentity) {
          // executor used
          executor.execute {
            log.info("Replying to message ${message.state.data.contents}")
            serviceHub.startFlow(ReplyToMessageFlow(message))
          }
        }
      }
    }
  }

  // ourIdentity
}

I have added a few comments to make it clearer what changed since only a few lines were added.

All this change does, is start the Flow on a new thread. This then allows the current thread to end. Remember, this is important because this thread holds onto a position in the queue. Allowing it to end, frees up a slot for whatever comes next. Whether it is another observable event from trackBy or a Flow. It does not matter. As long as the thread is released, the possibility of deadlock occurring due to this code is naught.

Releasing you from this thread

Please take a moment to bask in the glory of the pun I made in this sections header. Maybe it’s not that good, but I’m still proud of it.

In conclusion, using trackBy in a Corda Service is perfect for starting off new processes based on information being saved to the node. But, you need to be careful when starting new Flows from a trackBy observable. This is due to the observable holding onto a Flow Worker thread and therefore a spot in the queue. If your throughput reaches higher numbers, you risk the chance of your node deadlocking. You could end up in a situation where the queue is blocked by threads that are all waiting for a Flow to finish but with no actual Flows in the queue. By moving the Flow invocations onto a separate thread from the observable thread. You allow the once held spot on the queue to be released. There is now no chance of your trackBy code causing deadlock.

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Asynchronous Flow invocations with Corda Services

Sat, 22 Sep 2018 00:00:00 GMT

How can I make my Flows faster? There’s a good chance you have thought about this before if you have been working with Corda for a while. You can make reasonable tweaks to eke out performance improvements by changing a few things: transaction size, optimising queries and reducing the number of network hops required throughout the Flow’s execution. There is one other possibility that probably also crossed your mind at some point. Multi-threading.

More specifically, asynchronously starting Flows/Sub Flows from an already executing Flow. Doing so has the potential to greatly improve your CorDapps performance.

If you tried this, you probably faced a similar exception to the one I got. Furthermore, as of now, Corda does not support threading of Sub Flows. But, it can still be done. We just need to be clever about it. That’s where multi-threading within Corda Services comes in. They can be called within Flows but are not prisoners to the strict rules that Flows put on them since an executing Flow will not suspend or checkpoint from within a service.

In this post, I will focus on multi-threading the starting of Flows from within a Service. There are other area’s that threading can be used within Corda, but this is an interesting area to that I want to look into deeper. On the other hand, starting Flows from a Service is also filled with a few gotchas. These need to be accounted for and traversed around. Otherwise, you are going to wake up one day and wonder why everything has stopped for no apparent reason.

Luckily for you, I am here to help. For me, well, I had to face this problem head-on.

Luckily for me, R3 was able to help.

For reference, I will be using Corda Enterprise 3.1 for this post. To actually gain any benefit from the contents of this post you will need to be using Enterprise. This is due to Enterprise supporting multiple Flows executing asynchronously. Open Source does not currently allow this.

I also recommend looking at my previous post Corda Services 101 as we will be building off the foundation laid there.

Scenario

Let’s start with outlining the scenario that we will be using for this post.

PartyA sends PartyB some messages over time. Each message comes from a single Flow.
PartyB responds to all messages sent to them. Each message comes from a single Flow, but they want a single place to execute the process.

A series of Flows can be quickly put together to satisfy this requirement. Doing this sequentially should prove absolutely zero problems (after we have fixed all the stupid mistakes we all make).

Although this scenario is a poor case for needing performance, it is a simple one to understand so we can focus on running this asynchronously.

The slow synchronous solution

Before we look at the asynchronous solution, it will be beneficial to have a quick look at the code we will be moving from. Below is the code from ReplyToMessagesFlow. I don’t want to go through all of the underlying code and instead only want to focus on the code relevant to this post:

@InitiatingFlow
@StartableByRPC
class ReplyToMessagesFlow : FlowLogic<List<SignedTransaction>>() {

  @Suspendable
  override fun call(): List<SignedTransaction> {
    return messages().map { reply(it) }
  }

  private fun messages() =
    repository().findAll(PageSpecification(1, 100))
      .states
      .filter { it.state.data.recipient == ourIdentity }

  private fun repository() = serviceHub.cordaService(MessageRepository::class.java)

  @Suspendable
  private fun reply(message: StateAndRef<MessageState>) =
    subFlow(SendMessageFlow(response(message), message))

  private fun response(message: StateAndRef<MessageState>): MessageState {
    val state = message.state.data
    return state.copy(
      contents = "Thanks for your message: ${state.contents}",
      recipient = state.sender,
      sender = state.recipient
    )
  }
}

If you did have a read through Corda Services 101 then you might have recognised this class. As I mentioned earlier, putting together a solution for the proposed problem is very easy. Retrieve the MessageStates from the Vault and start a subFlow to reply to them.

This code will happily chug along through the messages one by one.

So, can we take this code and make it faster?

A failed attempt at asynchronicity

Let’s try and make the current code faster by introducing threading! We will use CompletableFutures to do this:

@InitiatingFlow
@StartableByRPC
class ReplyToMessagesBrokenAsyncFlow : FlowLogic<List<SignedTransaction>>() {

  @Suspendable
  override fun call(): List<SignedTransaction> {
      return messages().map { CompletableFuture.supplyAsync { reply(it) }.join() }
  }

  // everything else is the same as before
}

Most of the code is the same as before and has therefore been excluded from the example.

The only change to the code is the addition of CompletableFuture and its supplyAsync method (comes from Java). It attempts to start executing the reply function for each message on a separate thread.

So why is this section named “A failed attempt”? I refer you to the stack trace you get when executing the above code:

java.util.concurrent.CompletionException: java.lang.IllegalArgumentException: Required value was null.
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:273) ~[?:1.8.0_172]
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:280) ~[?:1.8.0_172]
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1592) ~[?:1.8.0_172]
    at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1582) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692) ~[?:1.8.0_172]
    at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:157) ~[?:1.8.0_172]
Caused by: java.lang.IllegalArgumentException: Required value was null.
    at net.corda.node.services.statemachine.FlowStateMachineImpl.checkDbTransaction(FlowStateMachineImpl.kt:201) ~[corda-node-3.1.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:192) ~[corda-node-3.1.jar:?]
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:271) ~[corda-node-3.1.jar:?]
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:312) ~[corda-core-3.1.jar:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.reply(ReplyToMessagesBrokenAsyncFlow.kt:57) ~[classes/:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.access$reply(ReplyToMessagesBrokenAsyncFlow.kt:19) ~[classes/:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$poop$inlined$map$lambda$1.get(ReplyToMessagesBrokenAsyncFlow.kt:46) ~[classes/:?]
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$poop$inlined$map$lambda$1.get(ReplyToMessagesBrokenAsyncFlow.kt:19) ~[classes/:?]

You are going to get that, along with a lengthy list of checkpointing log lines that Corda is printing out. Furthermore, just to cover my ass and prove to you that this is not due to a problem with CompletableFutures, here is a different error you get when using an Executor threadpool:

Exception in thread "pool-29-thread-1" Exception in thread "pool-29-thread-2" java.lang.IllegalArgumentException: Required value was null.
    at net.corda.node.services.statemachine.FlowStateMachineImpl.checkDbTransaction(FlowStateMachineImpl.kt:201)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:192)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:271)
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:312)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.reply(ReplyToMessagesBrokenAsyncFlow.kt:48)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.access$reply(ReplyToMessagesBrokenAsyncFlow.kt:19)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$call$inlined$map$lambda$1.run(ReplyToMessagesBrokenAsyncFlow.kt:29)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalArgumentException: Required value was null.
    at net.corda.node.services.statemachine.FlowStateMachineImpl.checkDbTransaction(FlowStateMachineImpl.kt:201)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.processEventImmediately(FlowStateMachineImpl.kt:192)
    at net.corda.node.services.statemachine.FlowStateMachineImpl.subFlow(FlowStateMachineImpl.kt:271)
    at net.corda.core.flows.FlowLogic.subFlow(FlowLogic.kt:312)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.reply(ReplyToMessagesBrokenAsyncFlow.kt:48)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow.access$reply(ReplyToMessagesBrokenAsyncFlow.kt:19)
    at com.lankydanblog.tutorial.flows.ReplyToMessagesBrokenAsyncFlow$call$inlined$map$lambda$1.run(ReplyToMessagesBrokenAsyncFlow.kt:29)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Hopefully you believe me at this point. If not, refer back to what I said at the start. Corda does not currently support starting new Flows asynchronously from an executing Flow. It is something that I believe they are working towards. But, as of now. Don’t use this solution.

A working asynchronous solution

We have seen that threading inside a Flow is a no go. To carry on with our quest for performance, we will now look at threading from a Corda Service. This should come as no surprise since the title and opening paragraphs already discussed this…

Sarcastic comments aside. Delegating to a Service will require a bit of rework from the original solution but the bulk of the code will remain the same. Most of it will just be copied and pasted to another class. Taking the code from the Flow and putting it into a Service.

Below is the new MessageService that contains the code from the original ReplyToMessagesFlow, but with a few changes and addition of threading code:

@CordaService
class MessageService(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  private companion object {
    val executor: Executor = Executors.newFixedThreadPool(8)!!
  }

  fun replyAll() {
    messages().map {
      executor.execute {
        reply(it)
      }
    }
  }

  private fun messages() =
      repository().findAll(PageSpecification(1, 100))
          .states
          .filter { it.state.data.recipient == serviceHub.myInfo.legalIdentities.first() }

  private fun repository() = serviceHub.cordaService(MessageRepository::class.java)

  private fun reply(message: StateAndRef<MessageState>) =
      serviceHub.startFlow(SendMessageFlow(response(message), message))

  private fun response(message: StateAndRef<MessageState>): MessageState {
    val state = message.state.data
    return state.copy(
        contents = "Thanks for your message: ${state.contents}",
        recipient = state.sender,
        sender = state.recipient
    )
  }
}

As you can see most of the code is the same as it was in the ReplyToMessagesFlow.

The first point I want to highlight is the use of an Executor thread pool. I have not used CompletableFutures here for reasons that we will look at later.

So how does this all work? The replyAll function executes reply on a new system thread for each message retrieved from the Vault. This new thread, in turn, calls startFlow. Triggering a new Flow to be put onto the Flow Worker queue. This is where all the fun happens and everything starts to get messy.

The Flow Worker queue looks after the order that Flows execute in and will fill and empty as Flows are added and completed. This queue is crucial in coordinating the execution of Flows within a node. It is also the source of pain when it comes to multi-threading Flows ourselves.

Below is a diagram demonstrating a simplistic view of the queue:

Why am I talking about this queue? Well, we need to be extra careful not to fill the queue up with Flows that cannot complete.

How can that happen? By starting a Flow within an executing Flow who then awaits its finish. This won’t cause a problem until all the threads in the queue’s thread pool encounter this situation. Once it does happen, it leaves the queue in deadlock. No Flows can finish, as they all rely on a number of queued Flows to complete.

This is most likely to happen on a high throughput system that is triggering the same flow many times. The chance the queue is full of Flows waiting for other Flows to finish now shoots up.

This isn’t great and makes things a tad harder. But, as long as we are aware of this, we can accommodate for it.

This is also the reason for the Executor thread pool, rather than CompletableFutures. By starting new Flows and not waiting for their completion, deadlock can be avoided. This is also the downside of this solution. Without the results of the new Flows, its capabilities are extremely limited.

All that being said, if your use-case fits the sort of structure shown above then I definitely recommend using this solution.

In the next section, I will discuss using CompletableFutures.

A dangerous solution with CompletableFutures

There is one simple reason why this is dangerous. Deadlock. I recommend keeping clear of this solution. Unless your node has access to enough threads, to decrease the chance of filling the queue with threads that can’t finish. On the other hand, it is a much more ideal solution since you can await the results of the started Flows and do something with them. This makes for a much more useful solution.

Below is what the MessageService would look like with CompletableFutures:

@CordaService
class MessageService(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

  fun replyAll(): List<SignedTransaction> =
    messages().map { reply(it).returnValue.toCompletableFuture().join() }

  // everything else is the same as before
}

The code is completely the same except for the replyAll function. The toCompletableFuture function that the returned CordaFuture provides, calls join to wait for the result of all futures and returns the overall result.

As I mentioned before, this solution could lead to deadlock. But, for your scenario, maybe it doesn’t. It is up to you to determine how likely it is to happen. If the odds are against you, it’s probably best to walk away. Either choosing to stick with a synchronous or an asynchronous solution similar to what I detailed in the previous section.

Do I really need to do this?

For now, yes, I believe you do.

Moving forward, I doubt that you will need to rely on the solution I proposed in this post.

I believe that Corda is working towards removing the need to even think about threading when starting Flows from within Flows. Instead, allowing you to simply call subFlow with an option to run it asynchronously. This would have allowed us to keep the original synchronous solution but with an option to make each subFlow run on a separate thread.

Joining the sections together

In conclusion, in Corda Enterprise 3, it is possible to initiate new Flows asynchronously within an executing Flow. This can provide good performance benefits depending on your use-case. There are downsides though. You cannot await the results of the asynchronous Flows without endangering your node with the threat of deadlock. The node’s underlying queue cannot deal with the situation it is being put in. Therefore, you need to be careful about how you go about introducing threads to your Flow invocations. Thankfully, as Corda progresses, it’s likely you won’t even need to worry about doing this yourself. It might even be as simple as adding a boolean function argument. That is the dream!

The code used in this post can be found on my GitHub.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Corda Services 101

Sun, 19 Aug 2018 00:00:00 GMT

I’m in the mood to write a short and to the point post today. I’m actually curious how quickly I can get this published. So let’s go.

This post is about Corda Services (using Corda version 3.2). What are they? As a developer that uses Spring a lot, I would personally say they are like Beans. There is more than Spring Beans can do, but on a basic level, they are pretty similar. Anyway, let’s stop talking about Spring and focus on Corda.

The bare minimum you need to know

Corda Services are classes external to Flows, that currently, can only be called from within an executing Flow or from another service (which is in turn, called by a Flow). Similar to a subFlow, they allow you to reuse code but should be used for different reasons. Such as, a collection of vault query functions or initiating trackBy within a node. These are what I tend to use services for anyway.

Corda Services are defined by using the @CordaService annotation along with extending SingletonSerializeAsToken. Once this is done, when your Cordapp is loaded and the node starts up, the service that you have just defined will be initialised:

@CordaService
class MessageRepository(private val serviceHub: AppServiceHub) : SingletonSerializeAsToken() {

    private companion object {
        val log = loggerFor<MessageRepository>()
    }

    init {
        log.info("I am alive!")
    }

    fun findAll(pageSpec: PageSpecification): Vault.Page<MessageState> =
        serviceHub.vaultService.queryBy(QueryCriteria.LinearStateQueryCriteria(), pageSpec)
}

serviceHub provides access to everything you need. In this example, the service accesses the vaultService to retrieve states from the node’s vault.

It is now ready to be used from within your Flows or other services if desired. The snippet below is taken from one of my Flows:

private fun repository() = serviceHub.cordaService(MessageRepository::class.java)

serviceHub is available in all Flows and provides the cordaService function. For input, it requires the class of the service you are trying to retrieve. In this case, the MessageRepository is being loaded.

A tiny bit more information

That is all you need to start using Corda Services. But. I’ll give you a bit more information so you don’t make some of the same mistakes I made.

Lesson one. When calling a service from a Flow. Do not inject it inside the constructor of the Flow. Instead, call it from somewhere inside the call function or any others used from that point. If you don’t you’ll see the below error message:

java.lang.IllegalStateException: This can only be done after the flow has been started.

The above is the error you’ll get when calling the Flow from a test. If calling from RPC you’ll get something like this:

Caused by: java.lang.reflect.InvocationTargetException: null
Caused by: java.lang.IllegalStateException: This can only be done after the flow has been started.

Likely with a long stacktrace depending on your chosen web framework.

It isn’t entirely clear that injecting the service in at this point causes these errors and you might find they pop up for other reasons. But I think it’s safe to say, at least in Corda 3.2, that you should not do anything inside the constructor or during initialisation of a Flow.

Just to make this even clearer, below is the code that accompanied the earlier snippet where I injected the service:

@InitiatingFlow
@StartableByRPC
class ReplyToMessagesFlow : FlowLogic<List<SignedTransaction>>() {

    @Suspendable
    override fun call(): List<SignedTransaction> {
        return messages().map { reply(it) }
    }

    private fun messages() =
        repository().findAll(PageSpecification(1, 100))
            .states
            .filter { it.state.data.recipient == ourIdentity }

    private fun repository() = serviceHub.cordaService(MessageRepository::class.java)

    @Suspendable
    private fun reply(message: StateAndRef<MessageState>) = subFlow(SendMessageFlow(response(message), message))

    private fun response(message: StateAndRef<MessageState>): MessageState {
        val state = message.state.data
        return state.copy(
            contents = "Thanks for your message: ${state.contents}",
            recipient = state.sender,
            sender = state.recipient
        )
    }
}

As you can see, the service is injected within the repository function which is in turned called by call. Following this sort of structure everything will work just fine.

Lesson two. Do not forget to include serviceHub: AppServiceHub in your service’s constructor (you can call serviceHub whatever you like). If you don’t do this it won’t create the service and you’ll find the following error pops up when you try to access it:

Caused by: java.lang.IllegalArgumentException: Corda service com.lankydanblog.tutorial.services.MessageRepository does not exist

Although, there is a ray of hope in this situation… It is very unlikely you would do this. Because without an instance of AppServiceHub there isn’t really much you can do with your own service. You will not have access to the vault or any of the other inbuilt services. So, at the end of the day, this lesson is a bit pointless but I still fell into this trap…

Is that all?

Damn, I think I actually wrote a short post for once! Is that good or bad? I’m not 100% sure…

Anyway, I’m trying really hard to think of some more snippets of information. But I cant. The bare minimum to get a Corda Service working really is nice and easy.

That being said, in the last few weeks, I have learnt that there is some pretty cool and useful stuff that you can do within services that cannot be done within Flows. That is a subject I hope to cover at some point!

Conclusion

Corda Services allow you to create classes external to Flows where you can logically group code that isn’t directly related to the execution of a Flow. My favourite way to use a service is to group vault query functions into a single class (pretty much what I would do in the Spring world). There are a few steps you need to take to ensure you create your service correctly. Firstly, annotate it with @CordaService and extend SingletonSerializeAsToken. Secondly, make sure that you inject them into your Flows in the correct way, which is pretty much anywhere but the constructor (or init in Kotlin). Lastly, remember to include AppServiceHub in the service’s constructor. Once you are able to use Corda services, you will be able to separate code out of your Flows. Not only making the Flows shorter, but also making them easier to understand while increasing the reusability of the code that you spent your valuable time writing.

The code used for this post can be found on my GitHub. There is a lot more in that repository that was not included in this post.

If you found this post helpful, you can follow me on Twitter at @LankyDanDev to keep up with my new posts.

Streaming data out of a Corda node with Spring WebFlux

Wed, 25 Jul 2018 00:00:00 GMT

It’s been a while since my last post but I’m finally back! Since I am still on my project, I will be writing about using Corda again. This time, rather than focusing on Corda, we’ll look at using Spring with Corda. More specifically, Spring WebFlux. Why do this? One, because we can. Two, because it allows us to stream events coming out of the Corda node. This provides us with the possibility to track the progress of flows or retrieve updates to the vault and send them to any clients registered to the relevant endpoints. Using WebFlux with Corda did introduce a few problems. Some originating from Corda and some from Spring. Although, the Spring issues were to do with me expecting the Spring Boot + WebFlux combo to do more by default for me.

In this post, I’m going to assume you have some experience with Corda but if you do need some extra information on the subject I recommend reading through my previous posts: What is Corda and Developing with Corda. Furthermore, I also suggest taking a look at Doing stuff with Spring WebFlux as an introduction to WebFlux.

The 3.2 Open Source version of Corda will be used for the contents of this tutorial. I actually started writing this post based on 3.1 but the newer version was released during this time. Due to this, there are a few comments based on moving between these versions.

We will also be implementing everything in Kotlin but the contents of this post can be implemented in Java as well.

Introduction to the example application

We will be modelling a really simple application that doesn’t provide much use and is something I botched together for the sake of this post. The application will consist of one party sending a message (represented by the MessageState) to another party. To do this the SendMessageFlow will run and once it does, both parties will have a copy of the message and that’s it. Short and simple but should provide us with enough to demonstrate how WebFlux can work with Corda.

Structure

Normally I start by looking at the dependencies. Although, since I have split the code into separate modules, it would be best to first view the structure of the little example application.

.
+-- app
|   +-- {spring code}
|   +-- build.gradle
+-- cordapp
|   +-- {flow code}
|   +-- build.gradle
+-- contracts-and-states
|   +-- {contracts and states code}
|   +-- build.gradle
+-- build.gradle

That’s a quick view of the structure of the application. app will contain all the Spring code and will delegate to the Corda node via RPC. The cordapp module houses the flow logic and contracts-and-states does what the name suggests and contains the contract and state code. Both the cordapp and contracts-and-states modules are packaged up into Cordapp Jars and dumped into the Corda node.

Each of these modules contains a build.gradle file containing its relevant build information and dependencies. Since this post is not directly focusing on writing Corda code, we will not go on and look through every module and their build files in detail. Instead, we will only brush over the flow code at the end of the post so we can focus on the Spring implementation.

Dependencies for Spring module

Below is the build.gradle file of the app module (containing the Spring code):

buildscript {
    ext.spring_boot_version = '2.0.3.RELEASE'

    repositories {
        mavenCentral()
    }

    dependencies {
        classpath "org.jetbrains.kotlin:kotlin-allopen:$kotlin_version"
    }
}

// needed to resolve dependency error
// Constructor threw exception; nested exception is java.lang.NoSuchMethodError: io.netty.util.AsciiString.cached(Ljava/lang/String;)Lio/netty/util/AsciiString;
configurations.all {
    resolutionStrategy {
        force 'io.netty:netty-all:4.1.25.Final'
    }
}

repositories {
    mavenLocal()
    jcenter()
    mavenCentral()
    maven { url 'https://ci-artifactory.corda.r3cev.com/artifactory/corda-releases' }
}

apply plugin: 'kotlin'
apply plugin: 'net.corda.plugins.cordapp'
apply plugin: "kotlin-spring"

dependencies {
    compile "org.jetbrains.kotlin:kotlin-stdlib-jdk8:$kotlin_version"
    testCompile "org.jetbrains.kotlin:kotlin-test:$kotlin_version"
    testCompile "junit:junit:$junit_version"

    cordaCompile "$corda_release_group:corda-core:$corda_release_version"
    cordaCompile "$corda_release_group:corda-jackson:$corda_release_version"
    cordaCompile "$corda_release_group:corda-rpc:$corda_release_version"
    cordaRuntime "$corda_release_group:corda:$corda_release_version"

    compile group: 'org.springframework.boot', name: 'spring-boot-starter-webflux', version: "$spring_boot_version"

    // added since rxjava 1.xx does not implement publisher so cant be used by Spring WebFlux
    compile group: 'io.reactivex', name: 'rxjava-reactive-streams', version: '1.2.1'

    cordapp project(":cordapp-contracts-states")
    cordapp project(":cordapp")
}
// tasks to run web servers

I’m not an expert in Gradle, so there are probably some things in this snippet that could be done better, but it does what it needs to.

So, there are a few things I want to highlight. Spring Boot 2.0.3.RELEASE is being used and to go along with this the kotlin-spring plugin is used to add open to all Kotlin classes marked with certain Spring annotations. This is needed for quite a lot of situations since Spring requires some classes to be non-final. This isn’t a problem in Java but is problematic for Kotlin since all classes are final by default. More information on the plugin can be found at kotlinlang.org.

spring-boot-starter-webflux pulls in the WebFlux dependencies along with general Spring web server code to get everything up and running.

rxjava-reactive-streams, this is an interesting one that we will see come into play later on. Since Corda uses RxJava 1.x.x rather than the newer RxJava2, its Observables do not implement the Java 8 Publisher interface that Spring WebFlux uses to return reactive streams. This dependency converts these older Observables into Publishers so they are compatible with WebFlux. We will touch on this again later when we look at the code to do this conversion.

Finally, the netty-all version is forced to 4.1.25.Final to resolve a dependency issue.

Routing functions

WebFlux introduces a functional approach for routing requests to the functions that handle them. More information on this can be found in Doing stuff with Spring WebFlux. I don’t want to jump deep into how WebFlux is working but we will take a quick look at defining the routing functions. The main reason for this is due to using Kotlin instead of Java. Kotlin provides a different way to define the functions by using a DSL.

Below is the code to define the routing for this tutorial:

@Configuration
class MessageRouter {
    @Bean
    fun routes(handler: MessageHandler): RouterFunction<ServerResponse> = router {
        ("/messages").nest {
            (accept(MediaType.TEXT_EVENT_STREAM) and contentType(MediaType.APPLICATION_JSON)).nest {
                POST("/", handler::post)
            }
            accept(MediaType.APPLICATION_STREAM_JSON).nest {
                GET("/updates", handler::updates)
            }
        }
    }
}

The routes bean takes in the MessageHandler bean (which we will look at later) and maps two URI’s to functions found in that MessageHandler. The DSL allows for a slightly shorter version compared to the Java implementation. There are a few parts to focus on in this snippet.

("/messages") defines the base request path of the two routing functions. The DSL allows the functions to nest themselves from this base path and helps with the conveying the structure of the routes.

One function accepts TEXT_EVENT_STREAM (text/event-stream) in the response returned from sending the request while also specifying APPLICATION_JSON (application/stream+json) as the contents of the body. Since we have defined the Content-Type, in most cases we can assume that we will be sending a POST request (which we are). POST is further nested from the previous configuration and adds another MessageHandler function to accept requests.

The second function receives updates from the Corda node. To do this it returns APPLICATION_STREAM_JSON and expects a GET request to be sent to /messages/updates.

Handler functions

In this section, we will look at the MessageHandler that was mentioned a few times in the previous section. This class contains all the functions that perform the actual business logic. The routing was just a means to reach this point.

My previous post, Doing stuff with Spring WebFlux will explain the more WebFlux specific parts of these examples in more depth than I will in this post.

Below is the handler code:

@Component
class MessageHandler(rpc: NodeRPCConnection) {

    private val proxy: CordaRPCOps = rpc.proxy

    fun updates(request: ServerRequest): Mono<ServerResponse> {
        return ok().contentType(APPLICATION_STREAM_JSON)
            .body(trackNewMessages(), ParameterizedTypeReference.forType(Vault.Update::class.java))
    }

    private fun trackNewMessages(): Publisher<Vault.Update<MessageState>> =
        toPublisher(proxy.vaultTrackBy<MessageState>().updates)

    fun post(request: ServerRequest): Mono<ServerResponse> {
        val message = request.bodyToMono(Message::class.java)
        val id = UUID.randomUUID()
        return message.flatMap {
            created(UriComponentsBuilder.fromPath("messages/$id").build().toUri())
                .contentType(TEXT_EVENT_STREAM)
                .body(startTrackedFlow(it, id), String::class.java)
        }
    }

    private fun startTrackedFlow(message: Message, id: UUID): Publisher<String> =
        toPublisher(
            proxy.startTrackedFlow(
                ::SendMessageFlow, state(message, id)
            ).progress
        )

    private fun state(message: Message, id: UUID) =
        MessageState(
            sender = proxy.nodeInfo().legalIdentities.first(),
            recipient = parse(message.recipient),
            contents = message.contents,
            linearId = UniqueIdentifier(id.toString())
        )

    private fun parse(party: String) = proxy.wellKnownPartyFromX500Name(CordaX500Name.parse(party))
            ?: throw IllegalArgumentException("Unknown party name.")
}

First, we should highlight the NodeRPCConnection class and it’s property proxy of type CordaRPCOps. I stole NodeRPCConnection from an example Corda and Spring application. Long story short, NodeRPCConnection creates the RPC connection to the Corda node and proxy returns a CordaRPCOps. CordaRPCOps contains all the RPC operations that are available to use. This is the way that Spring will interact with the Corda node.

Let take a closer look at the updates function:

fun updates(request: ServerRequest): Mono<ServerResponse> {
    return ok().contentType(APPLICATION_STREAM_JSON)
        .body(trackNewMessages(), ParameterizedTypeReference.forType(Vault.Update::class.java))
}

private fun trackNewMessages(): Publisher<Vault.Update<MessageState>> =
    toPublisher(proxy.vaultTrackBy<MessageState>().updates)

This function returns new messages as they are saved to the vault. This sort of endpoint would be nice if you had an application that monitored updates coming from your Corda node.

The Corda related code in this snippet is all contained within the trackNewMessages function. It uses CordaRPCOps’s vaultTrackBy to access the vault service and starts tracking updates to any MessageStates. Since we have not passed any arguments to the function it will be tracking UNCONSUMED states only. vaultTrackBy returns a DataFeed object that can be used to either retrieve a snapshot of the vault via the snapshot property or by accessing the updates property an Observable will be returned allowing it’s update events to be subscribed to. This RxJava Observable is what we will use to stream data back to the caller.

This is the first instance where we need to use the rxjava-reactive-streams that I mentioned earlier. The toPublisher method takes in an Observable and converts it into a Publisher. Remember, WebFlux requires Java 8 compatible reactive streaming libraries that must implement Publisher. For example, Spring tends to make use of Reactor which provides the Mono and Flux classes.

After creating the Publisher it needs to be fed into a ServerResponse. As everything has gone well at this point we will return a 200 response via the ok method. The Content-Type is then set to APPLICATION_STREAM_JSON since it contains streaming data. Finally, the body of the response takes in the Publisher from trackNewMessages. The endpoint is now ready to be subscribed to by a requesting client.

The functionality to stream updates from the node to a client is now complete. What about actually saving a new message? Furthermore, is there any information that we can pass back to the sender about the executing flow? So let’s answer those two questions. Yes, we can save a new message using WebFlux. And yes, a flow can return its current progress.

Below is the code for the post function that saves a new message to both the sender’s and the recipient’s nodes while streaming the flow’s progress:

fun post(request: ServerRequest): Mono<ServerResponse> {
    val message = request.bodyToMono(Message::class.java)
    val id = UUID.randomUUID()
    return message.flatMap {
        created(UriComponentsBuilder.fromPath("messages/$id").build().toUri())
            .contentType(TEXT_EVENT_STREAM)
            .body(startTrackedFlow(it, id), String::class.java)
    }
}

private fun startTrackedFlow(message: Message, id: UUID): Publisher<String> =
    toPublisher(
        proxy.startTrackedFlow(
            ::SendMessageFlow, state(message, id)
        ).progress
    )

private fun state(message: Message, id: UUID) =
    MessageState(
        sender = proxy.nodeInfo().legalIdentities.first(),
        recipient = parse(message.recipient),
        contents = message.contents,
        linearId = UniqueIdentifier(id.toString())
    )

private fun parse(party: String) = proxy.wellKnownPartyFromX500Name(CordaX500Name.parse(party))
        ?: throw IllegalArgumentException("Unknown party name.")

proxy.startTrackedFlow starts a flow whose progress can be tracked by any ProgressTrackers added to the flow. The startTrackedFlow defined in this class delegates to the aforementioned function and returns its progress property; an Observable<String> whose events consist of the ProgressTracker’s progress.

The MessageState that’s passed into the flow is created from the Message object passed in from the request. This is to allow easier input of the message data to the endpoint since it contains less information than the MessageState itself. parse converts the string X500 name passed in the Message into a CordaX500Name and then into a Party within the network, assuming one exists.

This is then packaged into a response via the created method. The Content-Type is specified to tell the client that it contains text/event-stream. The path to the message uses the UUID that was created before the flow was executed. This could, for example, be used to retrieve a specific message but you’ll need to implement that yourself since I am too lazy to do that for this post.

Creating a client

Now that the endpoints are set up we should create a client that can send requests and consume the streams sent back to it. Later on, we will briefly look at the flow code to get a fuller understanding of whats going on.

To send requests to a reactive back-end, Spring WebFlux provides the WebClient class. After sending a request, the WebClient can react to each event sent in the response. The MessageClient below does just that:

@Component
class MessageClient(
    @Value("\${server.host}") private val host: String,
    @Value("\${server.port}") private val port: Int,
    private val decoder: Jackson2JsonDecoder
) {

    private val strategies = ExchangeStrategies
        .builder()
        .codecs { clientCodecConfigurer ->
            clientCodecConfigurer.defaultCodecs().jackson2JsonDecoder(decoder)
        }.build()

    private val client = WebClient.builder()
        .exchangeStrategies(strategies)
        .baseUrl("http://$host:$port")
        .build()

    fun doStuff() {
        val message = Message("O=PartyB,L=London,C=GB", "hello there")
        client
            .post()
            .uri("/messages")
            .body(Mono.just(message), Message::class.java)
            .accept(TEXT_EVENT_STREAM)
            .exchange()
            .flatMapMany { it.bodyToFlux(String::class.java) }
            .subscribe { println("STEP: $it") }

        client
            .get()
            .uri("/messages/updates")
            .accept(APPLICATION_STREAM_JSON)
            .exchange()
            .flatMapMany { it.bodyToFlux(Vault.Update::class.java) }
            .subscribe { println("UPDATE: $it") }

    }
}

The MessageClient wraps and uses a WebClient to send requests to the address specified in the WebClient’s builder. There is some extra configuration going on in this class around deserialisation, but I want to brush over that for now as there is a section further down covering that topic.

As before Doing stuff with Spring WebFlux provides in-depth explanations into the WebFlux specific methods.

So let’s look at each request individually, first up the POST request to the /messages endpoint:

client
    .post()
    .uri("/messages")
    .body(Mono.just(message), Message::class.java)
    .accept(TEXT_EVENT_STREAM)
    .exchange()
    .flatMapMany { it.bodyToFlux(String::class.java) }
    .subscribe { println("STEP: $it") }

The post method creates a builder that specifies the contents of the request. This should match up to an endpoint that we defined earlier. Once the request has been built, call the exchange method to send it to the server. The body of the response is then mapped to a Flux<String> allowing it to be subscribed to. That is the essence of using Reactive Streams. Once subscribing to the response, it is up to the client to perform whatever processing they wish to do on each event. In this scenario, it simply prints out the current step of the ProgressTracker.

If we sent a request via this piece of code we would receive the following:

STEP: Verifying
STEP: Signing
STEP: Sending to Counterparty
STEP: Collecting signatures from counterparties.
STEP: Verifying collected signatures.
STEP: Done
STEP: Finalising
STEP: Requesting signature by notary service
STEP: Broadcasting transaction to participants
STEP: Done
STEP: Done

These are the steps that the SendMessageFlow’s ProgressTracker defines. Yes, I know I haven’t shown you that code yet but just trust me on this. Not much else to this one really. As you can see, each string value returned from the stream attaches “STEP” to itself

Now onto the GET request to the /messages/update endpoint:

client
    .get()
    .uri("/messages/updates")
    .accept(APPLICATION_STREAM_JSON)
    .exchange()
    .flatMapMany { it.bodyToFlux(Vault.Update::class.java) }
    .subscribe { println("UPDATE: $it") }

Again there isn’t much to show at this point. But, behind the scenes, there is actually quite a bit of work required to get this to work. All the issues that I faced to get this call to work all revolved around serialisation and deserialisation. We will get into that in the next section.

The response to this request is as follows:

UPDATE: 0 consumed, 1 produced

Consumed:

Produced:
56781DF3CEBF2CDAFACE1C5BF04D4962B5483FBCD2C2E428352AD82BC951C686(0)
: TransactionState(data=MessageState(sender=O=PartyA, L=London, C=GB, 
recipient=O=PartyB, L=London, C=GB, contents=hello there, 
linearId=1afc6144-32b1-4265-a06e-73b6bb81aef3_b0fa8491-c9b9-418c-ba6e-8b7840faaf30, 
participants=[O=PartyA, L=London, C=GB, O=PartyB, L=London, C=GB]), 
contract=com.lankydanblog.tutorial.contracts.MessageContract, 
notary=O=Notary, L=London, C=GB, encumbrance=null, 
constraint=net.corda.core.contracts.WhitelistedByZoneAttachmentConstraint@4a1febb5)

The nice thing about this endpoint is that it now maintains a connection to the node which will keep sending any related updates back to this client. The above request was the update for the original POST message. Any new events received by the client would output an update on the client. This is what makes this sort of endpoint ideal for triggering a process or simply displaying up to date data on a front-end separate from the Corda node itself.

Serialisation and Deserialisation

In this section, I wanted to focus on setting up serialisation and deserialisation correctly. The data retrieved from the /messages/updates endpoint needs to serialise its data correctly to pass to the client, who also needs to be able to deserialise the response data.

Normally Spring does a lot of this for you, and it still does, but it seems with WebFlux there are some extra steps required to get it set up properly. Disclaimer, this is from my experience and if you know of better ways to do this I would be interested to hear from you.

Corda JacksonSupport

Spring tends to use Jackson by default and, very handily, Corda provides a lot of Jackson setup itself. The JacksonSupport.cordaModule provides some serialisation and deserialisation for classes such as Party and CordaX500Name. If you have some basic situations where you need to serialise or deserialise a Corda class this will probably suit your needs. In Spring you could create a bean that the default ObjectMapper will retrieve and add to itself.

@Bean
fun cordaModule() = JacksonSupport.cordaModule

But, this route has a few caveats. Some classes cannot be deserialised since the module relies on the ObjectMapper having access to node information, for example via the RPC client CordaRPCOps. Without this, deserialising a Party, AbstractParty or AnonymousParty will fail. Not only that, but this has now been deprecated from Corda 3.2 due to not being thread safe. JacksonSupport.cordaModule has also been moved into its own class (CordaModule).

The solution I give below is also the solution that Corda recommends taking from now on.

Below is the exception thrown when the MessageClient retrieves updates from the /messages/updates endpoint (for the rest of this section the same endpoint will be used):

com.fasterxml.jackson.databind.ObjectMapper cannot be cast to net.corda.client.jackson.JacksonSupport$PartyObjectMapper

From this, we can determine that our ObjectMapper is of the wrong type and actually needs to be the subtype PartyObjectMapper. Taking this a bit further we can see that this mapper is found in the JacksonSupport class as well. Now, all there is left to do is to create this mapper and use that instead of the default ObjectMapper.

So let’s see how to do that:

@Bean
fun rpcObjectMapper(rpc: NodeRPCConnection): ObjectMapper {
    return JacksonSupport.createDefaultMapper(rpc.proxy)
}

This will create a RpcObjectMapper which implements PartyObjectMapper and makes use of RPC to retrieve node information to make it possible to deserialise the various party classes. Inside the createDefaultMapper, the CordaModule from before is added and thanks to Spring, this will now be the default object mapper for most (note the most for later) instances where serialisation or deserialisation is needed.

Some more serialisation and deserialisation configuration

Now… I’m actually in quite a weird position. I wanted to go through all the other steps to get the endpoint working. But, no matter what I do, I cannot seem to recreate all the errors I used to run into before getting it to work. I don’t know what to say… Somewhere my exceptions are being swallowed and stopping me from seeing what is going on. Anyway, we must continue on. Thankfully I know why I added the rest of the code but I can no longer provide you with the exception that each change fixed…

Soooo, let’s look at the end product of the rpcObjectMapper that we started working on earlier:

@Bean
fun rpcObjectMapper(rpc: NodeRPCConnection): ObjectMapper {
    val mapper = JacksonSupport.createDefaultMapper(rpc.proxy)
    mapper.registerModule(jsonComponentModule())
    mapper.registerModule(MixinModule())
    return mapper
}

@Bean
fun jsonComponentModule() = JsonComponentModule()

class MixinModule : SimpleModule() {
    init {
        setMixInAnnotation(Vault.Update::class.java, VaultUpdateMixin::class.java)
        setMixInAnnotation(SecureHash::class.java, SecureHashMixin::class.java)
    }
}

abstract class VaultUpdateMixin {
    @JsonIgnore
    abstract fun isEmpty()
}

@JsonDeserialize(using = JacksonSupport.SecureHashDeserializer::class)
abstract class SecureHashMixin

There are a few additions here. The JsonComponentModule is added as a bean so that it picks up the defined @JsonSerializer and @JsonDeserializer custom components (in other classes). It seems that even if it is added to the mapper as a module, it still requires the bean itself to be created if it is going to find and register the custom JSON components.

Next is the MixinModule. This class resolves issues that arise when deserialising Vault.Update and SecureHash. Let’s take a closer look.

A Mixin allows us to add Jackson annotations onto a class without actually having access to the class itself which we obviously don’t control since this an object from within Corda’s codebase. The other option is that this is added to the CordaModule we discussed earlier but that is a different conversation.

Vault.Update needs this due to having a method called isEmpty, which doesn’t play nicely with Jackson who gets confused and thinks that isEmpty matches to a boolean field called empty. So when deserialising the JSON back into an object it tries to pass in a value for the field.

The MixinModule itself is simply a class whose constructor adds the VaultUpdateMixin and SecureHashMixin to itself. The mapper then adds the module just like any other module. Job done.

The Jackson annotation added to the VaultUpdateMixin was @JsonIgnore, which speaks for itself. When serialising or deserialising the isEmpty function will be ignored.

Next up is the SecureHashMixin:

@JsonDeserialize(using = JacksonSupport.SecureHashDeserializer::class)
abstract class SecureHashMixin

I have added this in after moving from 3.1 to 3.2. To me it looks like adding a Mixin for SecureHash has been forgotten. The CordaModule includes serialisation and deserialisation for SecureHash.SHA256 but not SecureHash. The above code is copy and paste from CordaModule with a different class being tied to the Mixin.

Once this is included, the differences between 3.1 and 3.2 will be resolved.

I think I’ll raise an issue for this!

Custom Serialisers and Deserialisers

To serialise Vault.Update only the AttachmentConstraint interface needs it’s own custom serialiser:

@JsonComponent
class AttachmentConstraintSerialiser : JsonSerializer<AttachmentConstraint>() {
    override fun serialize(value: AttachmentConstraint, generator: JsonGenerator, provider: SerializerProvider) {
        when (value) {
            is AlwaysAcceptAttachmentConstraint -> {
                generator.writeStartObject()
                generator.writeStringField("type", "AlwaysAcceptAttachmentConstraint")
                generator.writeEndObject()
            }
            is HashAttachmentConstraint -> {
                generator.writeStartObject()
                generator.writeStringField("type", "HashAttachmentConstraint")
                provider.defaultSerializeField("attachmentId", value, generator)
                generator.writeEndObject()
            }
            is WhitelistedByZoneAttachmentConstraint -> {
                generator.writeStartObject()
                generator.writeStringField("type", "WhitelistedByZoneAttachmentConstraint")
                generator.writeEndObject()
            }
            else -> {
                generator.writeStartObject()
                generator.writeStringField("type", "AutomaticHashConstraint")
                generator.writeEndObject()
            }
        }
    }
}

Not much to talk about since only the HashAttachmentConstraint actually has any fields. This matches up to the deserialiser later on which reads the type JSON field to determine which object is created.

The last two classes that need custom deserialisers are ContractState and AttachmentContract (matching the serialiser from before):

@JsonComponent
class ContractStateDeserialiser : JsonDeserializer<ContractState>() {
    override fun deserialize(p: JsonParser, ctxt: DeserializationContext): ContractState {
        return ctxt.readValue(p, MessageState::class.java)
    }
}

@JsonComponent
class AttachmentConstraintDeserialiser : JsonDeserializer<AttachmentConstraint>() {
    override fun deserialize(p: JsonParser, ctxt: DeserializationContext): AttachmentConstraint? {
        val tree = p.readValueAsTree<JsonNode>()
        val type = tree["type"]
        return when (type.asText()) {
            "AlwaysAcceptAttachmentConstraint" -> AlwaysAcceptAttachmentConstraint
            "HashAttachmentConstraint" -> HashAttachmentConstraint(
                SecureHash.parse(tree["attachmentId"].asText())
            )
            "WhitelistedByZoneAttachmentConstraint" -> WhitelistedByZoneAttachmentConstraint
            else -> AutomaticHashConstraint
        }
    }
}

The ContractStateDeserialiser is a pretty lazy implementation since only one state is being used in this tutorial. The AttachmentConstraintDeserialiser uses the type field defined in the serialiser to determine which implementation of AttachmentConstraint it should be converted into.

WebFlux specific configuration

This subsection goes over the extra required configuration due to using WebFlux. You have already seen some of the configuration within the MessageClient but there is a bit extra that needs to be done:

@Bean
fun decoder(rpcObjectMapper: ObjectMapper): Jackson2JsonDecoder =
    Jackson2JsonDecoder(rpcObjectMapper, MediaType.APPLICATION_JSON, MediaType.APPLICATION_STREAM_JSON)

The client needs this bean to be able to deserialise application/stream+json along with the objects returned in the response.

@Component
class MessageClient(
    @Value("\${server.host}") private val host: String,
    @Value("\${server.port}") private val port: Int,
    private val decoder: Jackson2JsonDecoder
) {

    private val strategies = ExchangeStrategies
        .builder()
        .codecs { clientCodecConfigurer ->
            clientCodecConfigurer.defaultCodecs().jackson2JsonDecoder(decoder)
        }.build()

    private val client = WebClient.builder()
        .exchangeStrategies(strategies)
        .baseUrl("http://$host:$port")
        .build()

    // do stuff
}

To make use of the Jackson2JsonDecoder defined in the configuration, the ExchangeStrategies of the WebClient must be specified. Unfortunately, the ExchangeStrategies class is not written to pick up the Jackson2JsonDecoder that we already created. I was hoping that this sort of configuration would work by default, but oh well. To add the ExchangeStrategies the WebClient builder must be used. Once that is done, we are finally there. All serialisation to package up the response and the deserialisation to use it from the client is complete.

That sums up all the Spring related code that I wish to go over in this post.

A quick look at the Flow code

Before concluding, I will briefly show the flow that I put together for the purpose of this tutorial:

@InitiatingFlow
@StartableByRPC
class SendMessageFlow(private val message: MessageState) : FlowLogic<SignedTransaction>() {

    private companion object {
        object CREATING : ProgressTracker.Step("Creating")
        object VERIFYING : ProgressTracker.Step("Verifying")
        object SIGNING : ProgressTracker.Step("Signing")
        object COUNTERPARTY : ProgressTracker.Step("Sending to Counterparty")
        object FINALISING : ProgressTracker.Step("Finalising")

        private fun tracker() = ProgressTracker(
            CREATING,
            VERIFYING,
            SIGNING,
            COUNTERPARTY,
            FINALISING
        )
    }

    override val progressTracker = tracker()

    @Suspendable
    override fun call(): SignedTransaction {
        val stx = collectSignature(verifyAndSign(transaction()))
        progressTracker.currentStep = FINALISING
        return subFlow(FinalityFlow(stx))
    }

    @Suspendable
    private fun collectSignature(
        transaction: SignedTransaction
    ): SignedTransaction {
        val session = initiateFlow(message.recipient)
        progressTracker.currentStep = COUNTERPARTY
        return subFlow(CollectSignaturesFlow(transaction, listOf(session)))
    }

    private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
        progressTracker.currentStep = VERIFYING
        transaction.verify(serviceHub)
        progressTracker.currentStep = SIGNING
        return serviceHub.signInitialTransaction(transaction)
    }

    private fun transaction() =
        TransactionBuilder(notary()).apply {
            progressTracker.currentStep = CREATING
            addOutputState(message, MessageContract.CONTRACT_ID)
            addCommand(
                Command(
                    MessageContract.Commands.Send(),
                    message.participants.map(Party::owningKey)
                )
            )
        }

    private fun notary() = serviceHub.networkMapCache.notaryIdentities.first()
}

@InitiatedBy(SendMessageFlow::class)
class SendMessageResponder(val session: FlowSession) : FlowLogic<Unit>() {
    @Suspendable
    override fun call() {
        subFlow(object : SignTransactionFlow(session) {
            override fun checkTransaction(stx: SignedTransaction) {}
        })
    }
}

It’s a pretty simple flow with the addition of a ProgressTracker that the /messages request used to follow the current state of the flow. Long story short, this flow takes the MessageState passed into it and sends it to the counterparty. While moving through the flow the ProgressTracker is updated to the relevant step. Further documentation on using a ProgressTracker can be found in the Corda docs.

Closing time

That was honestly a lot longer than I thought it would be and has taken me much longer to write than I hoped.

In conclusion, Spring WebFlux provides the capability to use reactive streams to handle response events whenever they arrive. When used with Corda, the progress of a flow can be tracked and a persistent stream of vault updates can be maintained ready to be acted upon as they arrive. To fully make use of WebFlux with Corda, we also had to look into ensuring that objects were serialised correctly by the server and then deserialised by the client so they can be made use of. Lucky Corda does provide some of this, but one or two classes or features are missing and we need to make sure that we use their provided the object mapper. Unfortunately, WebFlux requires a bit more configuration than I’m normally accustomed to when using Spring modules, but nothing that can’t be fixed.

The rest of the code for this post can be found on my GitHub

If you enjoyed this post, you can follow me on twitter at @LankyDanDev where I post updates of my new posts (although they have been slowing down a bit recently).

Developing with Corda

Tue, 05 Jun 2018 00:00:00 GMT

This post was written for Corda 3. If you are looking for a more up to date version, I recommend looking at my updated version of this post, Developing with Corda 4.

In my last post I gave an overview of what Corda is trying to achieve and the design decisions that were made to do so. That information is great to know so we can get some perspective of the platform but it’s not going to help us in writing a system that can leverage Corda. To do that we need to know how the components of Corda work and fit together, only then can we start writing an application that actually does something and works correctly from the perspective of Corda. I see this as the same as learning any other framework but we need the voice in the back of our heads to remind us every now and then that we are in fact designing upon a Distributed Ledger Technology platform so we can make sure that the applications we create are properly designed.

In this post we will look at writing a very simple Corda application.

Corda is compatible with any JVM language. I know your thinking it, but no, it’s not written in Java. It is actually written in Kotlin. This might require a little bit of extra learning to get to grips with Kotlin if you normally work with Java (like I do) but it shouldn’t take you long to be comfortable with it. Due to this, I personally suggest writing your own code in Kotlin to keep the whole stack in the same language so when you need to dig down into Corda’s own code it won’t look alien compared to what you were just writing. Obviously, that is just my suggestion and you could instead use Clojure but your going to find it hard to get a hold of any existing examples without first converting them into their Clojure equivalents.

It is a bit hard to dive straight into the code without first understanding the key concepts of Corda. Rather than going through these myself, I personally believe the documentation Corda provides on this subject is very helpful and should get you most of the way there.

For the purpose of this post, I think it is best to leave out the configuration required to actually setup Corda nodes and instead we should fully focus on writing a Corda application. In a later post I will go over the configuration needed to setup nodes so that they can interact with each other. For now your just going to have to trust me that this stuff works.

Overview of the application

Below is a sequence diagram containing the very simplified steps involved in issuing an IOU between two people:

As you can see, the borrower asks for some money, if the lender agrees, then they both try and remember that the borrower owes the lender some money. This is process is simple, but even then, it has been simplified some more. In this process we haven’t actually mentioned when and how the money is transferred between them. For the purpose of this tutorial we shall leave this transfer out and just focus on saving the intent of borrowing and lending money between them.

So, how is this sort of process modelled in Corda? Again, before we move on, I want to mention Corda’s key concepts as they are extremely important for understanding what is going on properly.

To model the process in Corda, we need to replace some of the steps. Determining how much money to borrow, becomes the creation of a transaction containing this information. A party being happy with what is proposed, is now represented by signing the transaction with their key. Communication between the borrower and lender, is replaced with sending a transaction between them. Finally, remembering who owes who, is now set in stone with both parties saving the transaction that occurred between them. By altering the simple diagram with this new information, a new version is created:

There are more steps but the process is still pretty simple. The diagram really speaks for itself and I don’t think there is anything more I can add to it. What I will say, is that I have still further simplified the diagram a little bit by removing the Notary, but only to focus on the process we are trying to model. We’ll touch on what a Notary is very, very, very briefly later on but I will again suggest Corda’s documentation on the subject (I’ll do the same again later on).

These diagrams should provide us with enough guidance to put our Corda application together for issuing an IOU between two parties.

States

As mentioned earlier, I am leaning upon r3 training materials. Below is the IOUState that will be passed around with the application:

data class IOUState(val amount: Amount<Currency>,
                    val lender: Party,
                    val borrower: Party,
                    val paid: Amount<Currency> = Amount(0, amount.token),
                    override val linearId: UniqueIdentifier = UniqueIdentifier()) : LinearState {
    
    override val participants: List<Party> get() = listOf(lender, borrower)
}

Due to the “IOU” concept, pretty much all the fields make sense without much explanation; amount is the lent amount, lender the party lending the money and so on. The only property that needs explaining is linearId which is of type UniqueIdentifier, this class is basically a UUID, in fact it’s internalId is generated from the UUID class.

The state extends LinearState which is one of the general types of state that Corda uses with another being FungibleState. Both of these are implementations of ContractState. LinearStates are use to represent states that, quoting it’s docs, “evolves by superseding itself”. As such, when the state is updated it should be included as an input of a transaction with a newer version being output. The old state will now be marked as CONSUMED from UNCONSUMED when saved to the vault (Corda’s abstraction over the database).

ContractState requires a property that returns the participants of the state.

data class IOUState(val amount: Amount<Currency>,
                    val lender: Party,
                    val borrower: Party,
                    val paid: Amount<Currency> = Amount(0, amount.token),
                    override val linearId: UniqueIdentifier = UniqueIdentifier(), 
                    override val participants: List<Party> = listOf(lender, borrower)) : LinearState

This allows you to define participants that might not be included in the state. Which route you take depends on your use-case, for this tutorial either will do the job.

Contracts

Next up are contracts. Contracts are used to validate input and output states for a given command by all parties involved in the transaction, the command could be, for example, issuing the state or paying off owed money. We will look at commands slightly later in this section but we should be able to brush over them for now.

Contracts are quite nice to write due to their expressiveness. We are able to write conditions within the contract that must be met for a transaction to be valid. If any are not met then an exception will be thrown which will normally end in terminating the current transaction since an involved party has found it invalid.

These conditions use the requireThat DSL that Corda defines to specify conditions along with error messages that details what is wrong with the transaction. This makes it nice and easy to go through a contract and understand what it is doing since the code conditions are nicely complemented by the English messages (or whatever language you want to write them in).

Below is an example of a contract that is used to validate the IOUState defined above, again this is taken from r3’s training materials:

class IOUContract : Contract {
    companion object {
        @JvmStatic
        val IOU_CONTRACT_ID = "net.corda.contracts.IOUContract"
    }

    interface Commands : CommandData {
        class Issue : TypeOnlyCommandData(), Commands
        class Transfer : TypeOnlyCommandData(), Commands
        class Settle : TypeOnlyCommandData(), Commands
    }

    override fun verify(tx: LedgerTransaction) {
        val command = tx.commands.requireSingleCommand<Commands>()
        when (command.value) {
            is Commands.Issue -> requireThat {
                "No inputs should be consumed when issuing an IOU." using (tx.inputs.isEmpty())
                "Only one output state should be created when issuing an IOU." using (tx.outputs.size == 1)
                val iou = tx.outputStates.single() as IOUState
                "A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)
                "The lender and borrower cannot have the same identity." using (iou.borrower != iou.lender)
                "Both lender and borrower together only may sign IOU issue transaction." using
                        (command.signers.toSet() == iou.participants.map { it.owningKey }.toSet())
            }
            is Commands.Transfer -> requireThat {
                // more conditions
            }
            is Commands.Settle -> {
               // more conditions
            }
        }
    }
}

I have simplified the contract for the purpose of this post since we will only focus on implementing one command type. Let’s start from the top and work our way down.

The IOUContract implements Contract requiring it to now have a verify function that gets called to verify (hence the name) a transaction.

Contract class name

The class name of the contract has been included here:

companion object {
    @JvmStatic
    val IOU_CONTRACT_ID = "net.corda.contracts.IOUContract"
}

This is used in other parts of Corda when reflection is required. r3’s training materials have done it this way but I personally think it’s a bit funky and should be done differently.

companion object {
    @JvmStatic
    val IOU_CONTRACT_ID = IOUContract::class.qualifiedName!!
}

That’s a bit better in my opinion. This solution removes the need to change the string’s value if the class is moved or renamed. That being said, Corda has followed the convention of using strings in their code so if you need to use the inbuilt contracts you can expect them to to follow this format. I’ll leave it up to you to decide which one you prefer.

Commands

Now we can talk about the commands that I briefly mentioned earlier on in this section. I have put them below again so you don’t need to scroll again:

interface Commands : CommandData {
    class Issue : TypeOnlyCommandData(), Commands
    class Transfer : TypeOnlyCommandData(), Commands
    class Settle : TypeOnlyCommandData(), Commands
}

These commands are used to specify the intention of the transaction. It has been put here due to it’s connection to the contract since they determine what conditions must be validated. That being said, you could put these commands somewhere else if you wanted, such as in its own file or outside of the contract class but within the same file. As long as your solution best describes your intention then you are probably going in the correct direction.

Since these commands are simple and are only used for to specify intent, TypeOnlyCommandData is extended. Other abstract classes are available that specify commands that we might want to use, such as MoveCommand.

We will see how to use the commands in the next section.

Implementing verify

Here’s where most of the magic happens, the code has been copied and pasted below:

override fun verify(tx: LedgerTransaction) {
    val command = tx.commands.requireSingleCommand<Commands>()
    when (command.value) {
        is Commands.Issue -> requireThat {
            "No inputs should be consumed when issuing an IOU." using (tx.inputs.isEmpty())
            "Only one output state should be created when issuing an IOU." using (tx.outputs.size == 1)
            val iou = tx.outputStates.single() as IOUState
            "A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)
            "The lender and borrower cannot have the same identity." using (iou.borrower != iou.lender)
            "Both lender and borrower together only may sign IOU issue transaction." using
                    (command.signers.toSet() == iou.participants.map { it.owningKey }.toSet())
        }
        is Commands.Transfer -> requireThat {
            // more conditions
        }
        is Commands.Settle -> {
            // more conditions
        }
    }
}

The verify function checks whether the proposed transaction is valid. If so, the transaction will continue forward and most likely be signed and committed to the ledger but if any of the conditions are not met an IllegalArgumentException is thrown thus likely leading to the termination of the proposed transaction.

Exceptions are generally how Corda deals with unmet requirements. When an exception is thrown, assuming nothing is trying to catch it, execution is terminated and propagated up until it is caught or it ends up in the console output. Using this, it provides a simple way to control the flow of the transaction since the it can be stopped anywhere in it’s execution, even on the counterparty’s node, as the exception will be passed back to the initiator.

Onto the verification code itself. The command that the transaction is executing on it’s states is retrieved and depending on the type, different checks are made to check the validity of the transaction. The requireThat DSL that Corda provides allows you to write a condition that must be true to continue along an error message that is output if the condition is false.

Let’s look at one of the requireThat statements a bit more closely:

val iou = tx.outputStates.single() as IOUState
"A newly issued IOU must have a positive amount." using (iou.amount.quantity > 0)

Theres not much to explain here. The DSL takes care of the intent of the statement. What I will point out is the syntax:

<string message of what condition should be met> using <condition it must pass>;

Quite simple. A point that stupidly caught me out a bit, if the condition contains spaces in it then it must be contained within brackets. Finally, the DSL can contain code that is not in a condition expression, allowing you to initialise variables and such which can then be used in the actual conditions.

That’s enough of contracts for now. They will pop up again in the next section when we put the IOUContract into action.

Flows

In Corda, flows are the central point where we tie of all previous sections. States, contracts and commands all come together to write the code that will propose a new transaction, send it to all counterparties to sign and commit it to the ledger if everyone is happy. You can do much more complicated things within flows but for this tutorial we will stick with the basics.

Following on from the examples in the previous sections, we will now implement the IOUIssueFlow. Again this is taken from r3’s training materials. Below is code as a whole which we will then split and examine:

@InitiatingFlow
@StartableByRPC
class IOUIssueFlow(private val state: IOUState) : FlowLogic<SignedTransaction>() {

    @Suspendable
    override fun call(): SignedTransaction {
        val notary = serviceHub.networkMapCache.notaryIdentities.first()
        val issueCommand = Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
        val transaction = TransactionBuilder(notary = notary)
        transaction.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
        transaction.addCommand(issueCommand)
        transaction.verify(serviceHub)
        val singleSignedTransaction = serviceHub.signInitialTransaction(transaction)
        val sessions = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet()
        val allSignedTransaction = subFlow(CollectSignaturesFlow(singleSignedTransaction, sessions))
        subFlow(FinalityFlow(allSignedTransaction))
        return allSignedTransaction
    }
}

@InitiatedBy(IOUIssueFlow::class)
class IOUIssueFlowResponder(private val flowSession: FlowSession) : FlowLogic<Unit>() {
    @Suspendable
    override fun call() {
        val signedTransactionFlow = object : SignTransactionFlow(flowSession) {
            override fun checkTransaction(stx: SignedTransaction) {
                requireThat {
                    val output = stx.tx.outputs.single().data
                    "This must be an IOU transaction" using (output is IOUState)
                }
            }
        }
        subFlow(signedTransactionFlow)
    }
}

The flow of this code (yes that is a pun) is reasonably straight forward and will be possibly one of the typical flows that you write within your own application.

All it does is:

Create a state
Add the state to a new transaction
Verify the transaction with a contract
Sign the transaction
Request the signatures of the counterparties
Save the transaction for all participants

Now that we know the steps that are made within this flow, we can go through and explain this is done within the code.

The Initiating Flow

Firstly, the flow class is annotated with @InitiatingFlow and extends FlowLogic. This combination is required by any flow that requests communication with a counterparty. FlowLogic contains one abstract function call that needs to be implemented by the flow. This is where all the magic happens. When the flow is triggered, which we will look at later, call is executed and any logic that we have put inside the function obviously runs. FlowLogic is generic (FlowLogic<T>) where T determines the return type of call. In the above example a SignedTransaction is returned but it is totally feasible to use FlowLogic<Unit> if you have no desire to return anything back to the caller of the flow.

Yet another annotation popping up. @Suspendable actually originates from quasar-core instead of one of Corda’s own libraries. This annotation is important and if you forget to add it you could run into errors that don’t necessarily indicate what is going wrong. It is needed on all functions that communicate with a counterparty. As the name “suspendable” suggests, the annotation allows the function to be suspended while the counterparty is dealing with their side of the transaction. Quite a bit of magic goes on here and it is touched on briefly in the Corda documentation on flows.

Now we’re done with the annotations we can look at the contents of call. I’ve pasted it below again to save you some energy on scrolling:

@Suspendable
override fun call(): SignedTransaction {
    val notary = serviceHub.networkMapCache.notaryIdentities.first()
    val issueCommand = Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
    val transaction = TransactionBuilder(notary = notary)
    transaction.addOutputState(state, IOUContract.IOU_CONTRACT_ID)
    transaction.addCommand(issueCommand)
    transaction.verify(serviceHub)
    val singleSignedTransaction = serviceHub.signInitialTransaction(transaction)
    val sessions = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet()
    val allSignedTransaction = subFlow(CollectSignaturesFlow(singleSignedTransaction, sessions))
    subFlow(FinalityFlow(allSignedTransaction))
    return allSignedTransaction
}

For this example I have just put everything into one function so we can see step by step what is happening; I will show another version of this function later on which splits it into smaller functions that I personally think conveys the purpose of the flow quite nicely.

Creating the transaction

First, we will look at building the proposed transaction, the relevant code has been extracted below:

val notary = serviceHub.networkMapCache.notaryIdentities.first()
val issueCommand = Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey })
val transaction = TransactionBuilder(notary = notary)
transaction.addOutputState(state, IOUContract.IOU_CONTRACT_ID.toString())
transaction.addCommand(issueCommand)

For the purpose of this post we will assume there is only one Notary which allows us to be lazy and just retrieve the first one from the list. If you do not know what a Notary is, like earlier I suggest reviewing the Corda Key Concepts for a good explanation on the topic. For now, I’ll provide you the bare minimum to carry on. A Notary is a node whose sole purpose is to validate that no double spends have occurred within a transaction sent to it and extra validation can also be run if it is setup to do so.

The serviceHub comes provided since we extended FlowLogic; the function networkMapCache will then provides us with the identities of the parties on the network and notaryIdentities narrows it down even more. As I mentioned earlier, we’re going to be lazy and just retrieve the first one from this list. How you retrieve the Notary that you wish to use in a transaction might change depending on your requirements.

We then create a command that represents the intent of the transaction. In this case, we use the IOUContract.Commands.Issue that we defined earlier. In creating the command we also need to provide the public keys of the parties required to sign the transaction. it is a Party and owningKey represents their public key. The only signers in this transaction are contained within the states participants property but an independent list could be passed in instead.

All the components we need for our transaction have now been retrieved or created. Now we need to actually start putting it together. TransactionBuilder does just that. The Notary that we retrieved can only be passed in via the TransactionBuilder constructor, whereas the others have various add methods as well as being included in the constructor. addOutputState takes in the state passed into the flow along with the contract name that will verify it. Remember I mentioned two ways to get this name; via a public property within the object, how Corda normally does it or by manually adding the classes name yourself, either way the end goal is the same. The final component we add to this transaction is the command we created.

Verifying and signing the transaction

The next block of code focuses on verifying and signing the transaction, again the relevant code has been pasted below:

transaction.verify(serviceHub)
val singleSignedTransaction = serviceHub.signInitialTransaction(transaction)

Once we are happy that everything we want to include in the transaction has been included, we need to verify it. Simply call the verify function the TransactionBuilder provides. This function results in the validation inside the contract being ran against the transaction. As mentioned in earlier in the contract section, if any of the conditions in the contract fail an exception is thrown. Since, in this code, there are no attempts to catch the exception, the flow will fail as the exception is propagated up the stack.

After the transaction has passed validation, as far as we (the initiator) are concerned, the transaction is ready to be shared with the other parties. To do this, serviceHub.signInitialTransaction is called. This leaves us with a new SignedTransaction that is currently only signed by us. Having this transaction signed will become important later when the Notary checks that the transaction has been signed by all the parties involved.

Collecting signatures of the counterparties

Below is the code in charge of collecting signatures:

val sessions = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet()
val allSignedTransaction = subFlow(CollectSignaturesFlow(singleSignedTransaction, sessions))

The counterparties in this transaction are defined by the parties in the participants list. If we remember back to how the participants field was constructed in the state, only two parties were contained in it, therefore only two parties will need to sign the transaction. Although that statement is correct, the transaction has already been signed by the initiator so now only the single counterparty (the lender) needs to sign it.

To send the transaction to the counterparty we first need to create a session with them. The initiateFlow does just that. It takes in a Party and returns a FlowSession to be used for communications. As mentioned the initiator does not need to sign the transaction again through this construct, so in the code they have been removed from the parties whose communication sessions are being created for. Due to us knowing who are involved in this transaction, the below could of been written instead:

val session = initiateFlow(state.lender)
val allSignedTransaction = subFlow(CollectSignaturesFlow(singleSignedTransaction, listOf(session)))

Instead of relying on the participants list we instead just create a session for the lender as our knowledge of the state indicates that they are the only counterparty.

The FlowSession needs to be used inside of the CollectSignaturesFlow along with the SignedTransaction that is still only signed by the initiator at this point. This is our first encounter with subFlows. These are flows, similar to the one we are looking at in this post, that are called from within another flow. CollectSignaturesFlow cannot be triggered by itself as it is not annotated with @InitiatingFlow, therefore it can only ever be used from within a subFlow. Most of the flows provided by Corda out of the box fall within this same category.

Anyway, back to the flow at hand! CollectSignaturesFlow sends the SignedTransaction to the counterparty and awaits their response. We will look at how the response is sent back in a following section. Once returned, the SignedTransaction is now complete as it has been signed by everyone and can now be saved.

Persisting the signed transaction

This is the smallest snippet we’ll see during this breakdown, a whole one line:

subFlow(FinalityFlow(allSignedTransaction))

Although, for a one liner, this piece of code packs quite a punch. FinalityFlow will most likely always be called at the end of your flows, at least for the simpler flows anyway.

Calling FinalityFlow will:

Send the transaction to the Notary (if required)
Save the transaction to the initiator’s vault
Broadcast to the participants of the transaction to save it to their vaults

The last two steps depend on the Notary finding the transaction valid. If it does not, like usual, an exception is throw thus leading to an exit from the flow. Finally (yes, another pun), you do not need to write any code to save the transaction for the counterparties as that all happens behind the scenes.

The Responding Flow

@InitiatedBy(IOUIssueFlow::class)
class IOUIssueFlowResponder(private val flowSession: FlowSession) : FlowLogic<Unit>() {
    @Suspendable
    override fun call() {
        val signedTransactionFlow = object : SignTransactionFlow(flowSession) {
            override fun checkTransaction(stx: SignedTransaction) {
                requireThat {
                    val output = stx.tx.outputs.single().data
                    "This must be an IOU transaction" using (output is IOUState)
                }
            }
        }
        subFlow(signedTransactionFlow)
    }
}

As before this code was included earlier on in the post.

The most important line in this class is the @InitiatedBy annotation that specifies which flow it accepts requests from and responds back to, in this example it is the IOUIssueFlow that we have already gone through.

Since IOUIssueFlowResponder is also a flow, it extends FlowLogic and will need to implement it’s own version of call. The FlowSession in the constructor is the session that was used by the initiator to communicate with this flow. @Suspendable is also used on call just like it was in the initiating flow.

Finally, subFlow is called on the implementation of SignTransactionFlow leading to it being executed. The validation in the contract is ran, followed by the contents of checkTransaction and if all the checks come back fine, the transaction is signed and sent back to where it came from.

How I would structure the Initiating Flow

This little section is just for me to give you a suggestion in how to structure the initiating flow that we used in the examples. This in my opinion is better, but you might have a differing opinion on the subject:

@InitiatingFlow
@StartableByRPC
class IOUIssueFlow(private val state: IOUState) : FlowLogic<SignedTransaction>() {
    @@Suspendable
    override fun call(): SignedTransaction {
        val stx =  collectSignatures(verifyAndSign(transaction()))
        return subFlow(FinalityFlow(stx))
    }

    @Suspendable
    private fun collectSignatures(transaction: SignedTransaction): SignedTransaction {
        val sessions = (state.participants - ourIdentity).map { initiateFlow(it) }.toSet()
        return subFlow(CollectSignaturesFlow(transaction, sessions))
    }

    private fun verifyAndSign(transaction: TransactionBuilder): SignedTransaction {
        transaction.verify(serviceHub)
        return serviceHub.signInitialTransaction(transaction)
    }

    private fun transaction() = TransactionBuilder(notary()).apply {
        addOutputState(state, IOUContract.IOU_CONTRACT_ID)
        addCommand(Command(IOUContract.Commands.Issue(), state.participants.map { it.owningKey }))
    }

    private fun notary() = serviceHub.networkMapCache.notaryIdentities.first()
}

I don’t think that I’ve done anything particularly special here but I think this separation makes each step clearer. call is reduced to two lines (one if you really wanted) and I won’t even bother explaining what each method does as we have already been through the code and the function names so accurately describe what they are doing. Anyway, if you prefer writing it this way, then great. If you dont, then do what you wish; I won’t be upset, I promise…

Starting a Flow

In this final section we will look at how to call a flow from outside of the Corda node.

There are a few ways to do this, each working slightly differently. But, let’s keep this short and sweet and only look at the bog standard startFlow function:

proxy.startFlow(::IOUIssueFlow, state)

That’s it. As I said, short and sweet. proxy is of type CordaRPCOps which contains a load of functions revolved around interacting with the Corda node via RPC. startFlow is one of those functions. It takes in the name of the flow class along with any arguments that are part of the flow’s constructor. So in this example IOUIssueFlow’s call function will be invoked with an IOUState being passed in to be used within the flow.

A FlowHandle<T> is returned where T is the same generic type of the invoked flow, in this case a SignedTransaction. returnValue can then be called to retrieve a CordaFuture, allowing the result to be retrieved as soon as it’s available. CordaFuture is a subtype of a standard Future with a few extra methods made available, one of which is toCompletableFuture that may or not be useful to you (this was useful to me anyway).

Wrapping up

Here we are, at the end at last.

This post should have hopefully given you some help in understanding how to go about developing with Corda. There is much more to learn as I have only covered the basics in this post (I also need to learn more myself first). In this post we implemented the process of an IOU while inspecting the components that are required to do so. States are facts that are shared among parties on the network, contracts are used to validate transactions and flows contain the logic to propose new transactions. With this information you should be in a good place to start writing your own flows. There is much more you can do with flows that hasn’t been covered within this post, but these basics should serve you well through any flows that you try to write.

I plan to write more posts on developing on Corda, focusing on more complicated features and diving deeper into whats going on behind the scenes.

If you found this post helpful and want to keep up with my posts as I write them, then you can follow me on Twitter at @LankyDanDev.

What is Corda?

Tue, 05 Jun 2018 00:00:00 GMT

I’ve recently started a project (top secret, can’t say anymore) which involves using Corda. So, what is it? If you’ve found this post on Google then you probably already have some sort of idea, but if you’ve come from most other sources then it most likely doesn’t sound familiar. Well, Corda is a Distributed Ledger Technology (DLT for short). Some might say this is a blockchain, but I definitely believe it has its differences from other blockchains like Ethereum and Bitcoin and is more similar to other DLTs such as Hyperledger Fabric and Quorum (which itself is a fork of Ethereum). These differences are important. They are what allow Corda to perform better for its particular use cases. In this post I will give a brief overview of the factors that influenced the design of Corda.

Expanding on what I said in the introduction, Corda is a Distributed Ledger Technology to be used by businesses, such as financial institutions, to keep a shared ledger of transactions and thus removing the need for the involved parties to constantly check that each of their books are in line after interacting with each other. This is the primary problem that Corda is trying to solve.

Following on from this, since it is designed to allow interacting parties to make sure they are all in line, it also removes the need for all parties on the Corda network to know about each and every transaction, as only those involved are interested in them. Both of these points really need to be expanded upon and to do this we need to take a proper look into the problems Corda is attempting to solve so we can fully understand why Corda chose to make these fundamental decisions.

Why are Distributed Ledgers a possible solution to keeping each party’s books in line without the need for constant checking from both sides?

That was quite a long question to write. Data consistency is one of the core features of Distributed Ledgers and is crucial to the problem that Corda is attempting to solve. Let’s look more closely into why Corda believes that using a Distributed Ledger will remove the extra time and effort that businesses need to put in to keep data from all parties consistent.

When a business or financial institution, a bank for this example, transfers money from a client’s account to the receiver’s account in another bank both must check that what was sent was valid, that the funds from the client have been sent and finally that they were received correctly. Ensuring that the funds have decreased in one account and increased in the other requires multiple checks from both sides of the transaction as they need to be 100% sure that money hasn’t magically disappeared or created out of thin air. Furthermore, a number of these checks are done manually, requiring even more time and effort to ensure consistency.

A Distributed Ledger could completely remove the need for this reconciliation, whether it is done by the interacting parties themselves or a third-party. This is due to the nature of Distributed Ledgers where all nodes (the parties) must be in the same state. More specifically to Corda, a transaction is only committed when all involved parties have accepted that the inputs and outputs of the proposed transaction are correct. If anyone disagrees then it doesn’t take place. Otherwise the transaction is committed and the funds (or whatever the inputs and outputs were) are moved between the parties as previously agreed.

Notice that I haven’t mentioned any need for manual checking of the transactions. There can still be manual interactions during the processing of a proposed transaction if, for example, it needs to be signed off before the process is complete.

Onto the next question.

Why did Corda decide to make transactions private to only the parties included within the transaction?

First, we need to understand why blockchains like Ethereum and Bitcoin make all of their transactions public. One of the main attractions of blockchains is being trustless, meaning that I don’t need to trust you personally (or anyone else) to still trust that all the transactions on the chain are valid. This is possible due to all transactions being public once they are included in a block and will remain that way permanently. For a bad actor to alter a transaction after it has been included in a block is almost impossible as it will require copies of the block on all other nodes in the network to also be changed before the next block is added to the chain.

The downside that comes with this is that every party using that blockchain must keep a local copy of the chain themselves so that when a new block is added to the chain all parties receive the update and kept in the same state.

Performance and scalability are greatly impacted by making a system trustless. For example, Ethereum currently requires every node to process all transactions and store the state of accounts and contract code. This reduces the possible throughput of all nodes on the network to that of a single node. If this model remains the same then as the number of users increases the average time that a user waits for their transaction to be mined can only go up.

Due to this problem, possible ways to improve the performance of the network are being looked into; the Ethereum Raiden Network and Bitcoin Lightning Network are both aiming to vastly increase both the performance and scalability of their respective systems.

Corda is not trustless. That does not mean it moves to the other end of the scale, but some trust is explicit as a party’s identity must be known to join an existing Corda network. Furthermore, the parties on the network must trust the issuers of assets onto the ledger that they themselves are moving around within transactions. This is where the requirement of trust ends. The parties themselves do not need to trust each other, between the ledger keeping everyone in step, the fact that each of their identities are known by the networks certificate authority provides some backup guarantee in case of any nefarious behaviour.

Additionally, even trustless blockchains or ledgers are not found at rock bottom of the trust scale as theirs is implicit within their consensus protocols and relies on a single actor (or group of actors) not controlling half of the mining power of the whole network. So, for now at least, we are going to have to settle for some trust in any system we choose to use.

By removing the need for consensus across all parties in the network and instead only between the participants of a transaction the time to complete one is reduced. The initial trust provided by the networks certificate authority provides the foundation for this decision. This is very important if distributed ledgers are going to replace the financial infrastructure that already exists.

Using the answers to these two questions we can determine that, as of this moment, Corda is not as decentralised as a blockchain like Bitcoin or Ethereum but it does allow businesses to transact without the need for a central point. This is in line with other Distributed Ledger Technologies such as Hyperledger Fabric.

Whether or not this is a good or bad point depends on your stance on the importance of decentralisation. Although, the distribution of contract code works in a similar way to a hard fork in a blockchain. All parties must agree to the new code before everyone upgrades; the old version of the code is then abandoned rather than having two different versions running independently. This sort of upgrade process moves it back up a tad on the decentralised scale.

I believe these are the sort of design decisions that need to be made if DLTs are going to move forward and become more widely used.

In conclusion, these are the factors that I believe greatly influenced the design of Corda. They chose to design a Distributed Ledger Technology that uses known identities to inject trust into the system and keeps transactions private between interacting parties, with the goal to reduce the effort required to maintain data consistency while still allowing it to scale well and handle a higher volume of transactions. I believe these decisions are needed if Distributed Ledger Technologies are going to replace how businesses interact with each other and possibly supplant current financial infrastructure.

If this post has piqued your interest and you want to have a look at Corda yourself, then go look at their documentation which provides tutorials on the concepts Corda is built on along with how to use the platform so you can play with the it yourself.

If you enjoyed this post or found it helpful (or both), you can keep up with my new posts as I write them by following me on Twitter at @LankyDanDev.