Hi There, I was wondering if it is required to have a read-replica of the contract store in front of the DAML ledger to service the high volume read requests? To explain our usecase, We have a contract store which grows over the period of time and when a user tries to get a list of records from the ledger (a wild card query) it seems DAML is responding with all the records in the ledger instead of responding with a result set. So my question is can we configure the DAML in such a way that the records are filtered to suit the client request? If not what is the best way to handle such a situation as currently our java application which sits infront of the ledger is struggling to handle the big response coming back from Ledger.
Our setup is UI → Java Application → JSON API → DAML(daml-on-sql)
2 Likes
Hi @MayurDS and welcome to the forum.
Right now the JSON API has no built-in pagination. From what I can read from the use case you describe (and please do correct me if I’m wrong) it looks like you are using the v1/query endpoint. One possible approach that you can take to minimize the exchange of information with the JSON API is to use the streaming endpoint v1/stream/query, which opens a web socket which returns the set of active contracts active at the moment of the request (akin to what you would receive as part of the v1/query response) alongside with the offset at which this set of active contracts is valid, followed by messages that represent the creation of new contracts or the archival of existing ones. This should allow you to keep an up-to-date set of active contracts on your Java application.
The streaming part of the JSON API is documented here (valid for version 1.14.2).
2 Likes
Thanks for your reply @stefanobaghino-da. You are right, we are using /v1/query endpoint to get a list of contracts. But we have observed that along with pagination, filtering is also not possible through JSON API (let me know if my understanding is incorrect) which may be the reason DAML responding with all the records in the ledger.
For eg: We are getting a particular contract back if we provide all the necessary fields of the KEY required to identify it. But in case if the query is missing one of the mandatory value or if we want to search using wildcard field value (employeeName=“john*”) then the response contains all the contracts.
Also at the moment we are using version 1.11.1 and are planning to upgrade to the latest one soon.
1 Like
The JSON API does have some filtering capabilities based on the content of contracts. The query language is described here (for version 1.11.1 this time). However, full-text search or regex-based pattern matching as you mentioned are not currently supported.
1 Like
Thanks. Do you think that means, in case where it is required to get a list of contracts depending on the regex based patterns one has to use a read-replica of the contract store which provides this functionality (copying the contract store data to another datastore using a background process).
1 Like
At the current state, if you want the UI to only receive contracts where one of the field matches a regex, you would have to build this into your Java application. Based on your requirements and the trade-offs you may want to make, this can mean actually having a full read-replica that actually persists contracts for querying or simply making your Java application filter out the results which are not relevant for your UI before serving them. This is in a way what the JSON API does with the Ledger API, taking either route depending on whether you are using a query store or not (docs on the query store here).
2 Likes