Workflow template factoring with records

At the very end of episode 2 of our new Introduction to Daml, @Levente_Barczy mentions that serious Daml projects will use custom record types extensively. I think Daml developers should consider adding records to their repertoire as soon as possible, because the structure of templates lends itself to an especially direct use of records.

Records and their templates

Every record is exactly as powerful as the fields in it, no more, no less. That means that if you have three fields in a template, perhaps in several places,

    foo: Foo
    bar: Bar
    baz: Baz

you may instead write

data Quux = Quux with
    foo: Foo
    bar: Bar
    baz: Baz
  deriving (Eq, Show)

and replace those three fields where they appear with

    quux: Quux

and, while the structure of the payload is different, there are no contracts that are possible in one form but not the other.

If you then inspect the Payment module from the video, you’ll find three occurrences of the same set of five fields. Sounds like a clear use case for factoring with data, right? However…

Every template has a record

Consider the first of those templates:

template Payable
  with
    amount: Decimal
    currency: Text
    from: Party
    to: Party
    reference: Text

This implies the existence of a record that looks like this:

data Payable = Payable with
    amount: Decimal
    currency: Text
    from: Party
    to: Party
    reference: Text
  deriving (Eq, Show)

This automatic definition is so fundamental that, as we describe it in the Daml-LF language specification, a template is just a record type with various definitions associated with it. But for convenience we typically think of the record type as the template’s associated record type.

Having a value of the associated record type does not mean “this is a contract on the ledger”

When I refer to a value of type ContractId Payable, I’m referring to a contract that actually may exist on the ledger. Of course, any contract can be archived at any time by the signatories, but it isn’t unreasonable to treat a contract ID as reference to a valid contract, failing if the contract doesn’t exist.

However, if you consider a typical create call,

  create Payable with
    from = newOwner
    -- ...

this is not any kind of “create syntax”. It is really of the form

  create (Payable with
    from = newOwner
    ...)

in which things happen in two steps.

  1. We create an ordinary value of record type Payable. This is not a contract yet, it is just a grouping of data in the associated record type’s fields.
  2. We call the ordinary function create, which effectively asks the ledger to create a contract whose payload is the record value.

To illustrate, we could even write this as the following:

do
  let it = Payable with
             from = newOwner
             -- ...
  create it

and in this context, it is an ordinary variable of type Payable. Until we actually call create on it, all we’re doing is rearranging data in memory.

Data-accumulative workflows

The Payment workflow, Payable ⇒ PaymentClaim ⇒ Receipt, follows a pattern I think is quite common to many workflows. The first step has the five fields listed above. The second step has all five fields, plus a new field transactionId. The third step has all six fields, plus the new field received. Instead of duplicating the fields, here is how you might express this:

template Payable
  with
    amount: Decimal
    currency: Text
    from: Party
    to: Party
    reference: Text
-- ...

template PaymentClaim
  with
    pymt: Payable
    transactionId: Text
-- ...

template Receipt
  with
    claim: PaymentClaim
    received: Time

What I have lost in explicitness of having each piece of data in the contract declared right there, I have gained in explicitness about how the data flows through the workflow, because, if you are comfortable with record types, it is immediately apparent that "Receipt is all the data in PaymentClaim plus a received time", and similarly for PaymentClaim and Payable.

However, the data is structured differently, even if it is impossible for that data to represent anything different. So there are some adaptations to the code that must be made.

Creating the next step

The ClaimPaid choice formerly looked like this:

    controller from can
      ClaimPaid: ContractId PaymentClaim
        with
          transactionId: Text
        do
          create PaymentClaim with ..

This doesn’t work because we aren’t passing any of the existing fields as standalone fields when creating PaymentClaim; instead, we want to pass a single value, the record that represents the contract prior to PaymentClaim in the workflow. To supply it, we can simply change the last line to

          create PaymentClaim with pymt = this; ..

In this form, .. is no longer doing the heavy lifting of passing lots of fields; it’s only responsible for passing the new data, transactionId.

Referring to indirect data

The signatories in Receipt ultimately derive from the Payable that started this little workflow. They were written

    signatory to, from

However, to and from are no longer in scope. Instead, the most trivial port to the new data structure is

    signatory claim.pymt.to, claim.pymt.from

Here is another demonstration that explicitness is in the eye of the beholder. Our original form was more explicit in passing along to and from, redeclaring in each template; this new form is more explicit in that it directly states that the provenance of to and from is “they come from the claim, which got them from the payable”. What is better? I do not think a single rule is wise; instead, you should exercise judgement and choose the form in all cases that you think best and most clearly expresses the meaning of your contracts.

Nevertheless, you do not have to rewrite every variable reference to suit this style. Just as .. packs variables into a record, it can also unpack them back into variables. We can write this signatory as

    signatory let Payable{..} = claim.pymt in [to, from]

Probably not worth it in this case. But consider the example of Token and TokenOffer.

template Token
  with
    -- 8 fields
-- ...

template TokenOffer
  with
    -- same 8 fields as in Token
    newOwner: Party
    price: Decimal
  where
    -- ...
    controller newOwner, userAdmin can
      AcceptToken: (ContractId Token, ContractId Payable, ContractId Payable)
        do
          -- many references to the 8 fields from Token

By my count, AcceptToken contains 13 references to the various fields that were originally derived from Token. So when I make the port as follows,

template TokenOffer
  with
    token: Token
    newOwner: Party
    price: Decimal

I might have to prefix all of those references with token., as well as expand the usages of .., which no longer have the variables they need in scope. Instead, I can simply pull every member of token into scope, by adding to the top of AcceptToken:

    controller newOwner, token.userAdmin can
      AcceptToken: (ContractId Token, ContractId Payable, ContractId Payable)
        do
          let Token{..} = token -- makes a variable of every token field
          -- many references continue working as before

Updating nested fields

One disadvantage of the packed format is that updating inner fields can be a little more complex. For example, we could add a choice for scribbling a note on a receipt to the original Receipt:

   controller from can
      Scribble: ContractId Receipt
        with
          note: Text
        do create this { reference = reference <> note }

But this no longer works, because Receipt doesn’t directly contain reference. Unpacking can help us with referring to the reference field but not updating the field. When we update a field like this, we have to repack by referring to the full field name (in SDK 1.13 or later):

    controller claim.pymt.from can
      Scribble: ContractId Receipt
        with
          note: Text
        do create this with 
             claim.pymt.reference = claim.pymt.reference <> note

For the specific use case of the accumulative data workflow, I don’t believe that you should encounter this very often, because typically you want the workflow to get each piece of data right when it is introduced, and not rely on later updates, because the type-checker won’t help you if you’ve forgotten an update.

What if fields are subtracted instead of added?

So it turns out that Token wants to pass on all eight fields to TokenOffer. What if we only wanted to pass seven fields, as one field was unused?

We actually have some examples of this workflow order in the video. An IssuerRequest leads into an Issuer as follows:

template Issuer
  with
    userAdmin: Party
    issuer: Party
-- ...

template IssuerRequest
  with
    userAdmin: Party
    issuer: Party
    reason: Text
  where
    -- ...
    controller userAdmin can
      GrantIssuerRights: ContractId Issuer
        do create Issuer with ..

Even though we’re removing the reason, we can still use record factoring; in fact, I think this example is a positive for clarity, since we’re asking the requestor to pass in the very structure they want to be created.

template IssuerRequest
  with
    issuer: Issuer
    reason: Text
  where
    -- ...
    controller issuer.userAdmin can
      GrantIssuerRights: ContractId Issuer
        do create issuer

And so we have come full circle. create is not syntax, and is perfectly happy to operate on an existing variable of type Issuer. The contract itself does not exist until GrantIssuerRights is exercised, even though the record structure does exist.

Interacting from UIs and other ledger clients

Changing the structure of contracts also changes how you interact with those contracts from JSON API, Java codegen, and any other ledger client. Again, the possible values do not change, but their shape does.

For example, our fully-nested Receipt looks like this in JSON:

{
  "claim": {
    "pymt": {
      "amount": "42",
      "currency": "CHF",
      "from": "Alice",
      "to": "Bob",
      "reference": ""
    },
    "transactionId": "abcdefg"
  },
  "received": "2021-05-18T22:08:22.892Z"
}

Just as Daml code that interacts with receipts may need changes to deal with a restructuring, your TypeScript code cannot pretend that this is the flat structure that the original Receipt encoded to JSON as.

Fortunately, just like Daml, TypeScript has good tools for unpacking JS objects used as records. Similarly, if you are using the TypeScript codegen to generate type annotations for your Daml code, the type-checker can guide you to references that need to be updated to reflect the factored structure. Plain JavaScript users will not be so lucky; this is the least of reasons that I strongly recommend all JavaScript developers writing Daml integrations incorporate TypeScript into their workflow.

Further nesting with your own records

As @Levente_Barczy said, serious Daml projects will likely include record types defined with the data keyword, containing whatever fields you want, and not simply rely on templates’ associated record types to clean up duplication. All of the techniques I’ve demonstrated above—packing, unpacking, nested updating—work just as well with your own record type definitions.

For example, my local copy of the nft project has a TokenKey data type, which has some nice hygiene and readability properties. But I will leave that sort of experimentation to your own Daml projects.

7 Likes