What are DAML best practices regarding bulk uploading to the ledger?

Questions
1

Hi guys,

I’m looking at initializing the ledger with data (or adding new functionality to a running ledger) and the requirement I have currently is to upload a very large number of objects to create contracts on the ledger.

So, for example, I have X clients with nodes on the ledger, and I have a large (six-digits) number of products across all clients which now need a contract in the ledger to represent them, and a contract representing each client must have a list of unique identifiers for the products belonging to them. Is there a best practice way to write the DAML in such a way to minimize the actions necessary on the ledger?

Would it be faster to simply iterate through the large list once, creating products and archiving/re-creating client contracts repeatedly to update it? Or would it be faster to iterate through the products list to get all products for one client, create those products and update the client once, then iterate again for the next client? Or is there another way?

1 Like
2

The best way I found after some investigation on this topic was to iterate through the large list twice, once to create all the relevant products, and a second time to update the clients. The issue I am currently having is with regards to extracting relevant information to update the client. The below is not working as expected:

let clientDetails = Map.empty: Map (Text, ClientType) [ProductIdentifier]
let processClientInfoFromProduct(productInfo: ProductInformation) = do
  let currentClientDetails = Map.lookup (productInfo.clientId, productInfo.clientType) clientDetails
  case currentClientDetails of
   None -> do Map.insert (productInfo.clientId, productInfo.clientType) [productInfo.productIdentifier] clientDetails
   Some details -> do
    let newDetails = dedup (productInfo.productIdentifier :: details)
    Map.insert (productInfo.clientId, productInfo.clientType) newDetails boDetails

map processClientInfoFromProduct productsInfo

I understand why it’s not working, it’s creating a list of maps because the map function expects to return the result of the method, but I can’t find in the documentation the best way to do what I’m trying to do, create single map of client information to the products created for them. Any advice would be appreciated!

1 Like
3

I can’t answer your first question, but for the pure Map manipulation, the issue here is that map applies to each element of its input list separately. What you want is to walk down the list, and handle each element while having access to the accumulated result of all the previous elements. The function for that is called foldl (assuming you want to process from left to right; if you want right to left, you can use foldr). So the code you’re looking for would be something like:

foldl (\acc el -> do
  let key = (el.clientId, el.clientType)
  let new_data = el.productIdentifier
  case Map.lookup key acc of
    None -> Map.insert key [new_data] acc
    Some (existing_data) -> Map.insert key (dedup $ new_data :: existing_data) acc)
       (Map.empty : (Text, ClientType) [ProductIdentifier]
       productsInfo

And here is a full, self-contained example with correct indentation etc.:

module Main where

import qualified DA.Map as Map
import qualified DA.List as List
import Daml.Script

data ClientType = ClientType
  deriving (Show, Ord, Eq)

data ProductIdentifier = A | B
  deriving (Show, Ord, Eq)

data ProductInfo = ProductInfo with
    clientId : Text
    clientType : ClientType
    productIdentifier : ProductIdentifier
  deriving (Show, Ord, Eq)


setup : Script ()
setup = script do
  let productsInfo = [ProductInfo "client1" ClientType A,
                      ProductInfo "client1" ClientType B,
                      ProductInfo "client2" ClientType A]
  debug $ foldl (\acc el ->
    do
      let key = (el.clientId, el.clientType)
      let new_data = el.productIdentifier
      case Map.lookup key acc of
        None -> Map.insert key [new_data] acc
        Some (existing_data) -> Map.insert key (List.dedup $ new_data :: existing_data) acc)
                (Map.empty : Map.Map (Text, ClientType) [ProductIdentifier])
                productsInfo
  return ()

The script will print (with added indentation):

Map [(("client1",ClientType),[B,A]),
     (("client2",ClientType),[A])]

which hopefully is what you wanted to get.

2 Likes
4

Thanks Gary, that worked great! I had a feeling it was foldl that I wanted, I just had trouble putting it together!

General question to anyone: Do you think it’s worthwhile leaving this thread up to talk about bulk uploads to the ledger in general? Or is there somewhere else to have this sort of discussion?

1 Like
5

If you want to talk about it in general you can start a new topic in the General section. Also might be worth checking out or following up to: On Attachments: What you should and shouldn't store on a ledger