getTransactionTrees cannot filter with template id

Questions
,
1

We are building an Java Application that subscribe to Daml Ledger for both ExercisedEvent and CreatedEvent, our subscriber looks like below

@Override
    public Disposable subscribe(String subscribeParty, Identifier templateId, Consumer<TransactionTree> consumer) {
        var transactions = ledger.getTransactionTree(templateId, new FiltersByParty(Collections.singletonMap(subscribeParty,
                new InclusiveFilter(Collections.singleton(templateId)))), LedgerOffset.LedgerEnd.getInstance());
        return transactions.forEach(consumer);
    }

Each of the listeners will call the above subscribe when they are created, and since each listener are built for one type of CreatedEvent/ExercisedEvent
(e.g. TransferProposal Creation, Confirmation Creation will have their own listeners)
We would like to filter by template id when we are querying the ledger instead of querying for all templates and filter afterward as the amount of contracts are really huge in our case.

While running the application we got below error

Template filtration is not supported on GetTransactionTrees RPC. To get filtered data, use the GetTransactions RPC.

I have noticed this issue Improve documentation regarding transaction filtering limitations for GetTransactionTrees · Issue #1483 · digital-asset/daml · GitHub
However in our case we cannot use GetTransaction as we want ExercisedEvent as well.
Can someone suggest how we should design our code and have a proper use of the api in our case?

Thanks a lot in advance.

2

Your only option at the moment is to not filter in your request to the ledger and then filter on the client side to the events you are interested in.

2 Likes
3

@cocreature thanks so much for the speedy reply.
Two more question:

  1. I wonder if ledger api query is like database query where pre filtering and post filtering can make a lot of difference in performance? (do we have an index on template id, maybe not lol)

  2. On the other hand, is there a reason that we are not doing filtering by template id with GetTransactionTree ?

One Little Suggestion
As mentioned in Improve documentation regarding transaction filtering limitations for GetTransactionTrees · Issue #1483 · digital-asset/daml · GitHub maybe it will be good to deprecate the api and remove the filter from below?
(As the api looks like it’s indicating that we can use a filter, and some naive noob like me might walk right into it :cry: )

Flowable<TransactionTree> getTransactionsTrees(LedgerOffset begin, LedgerOffset end, TransactionFilter filter, boolean verbose);

   Flowable<TransactionTree> getTransactionsTrees(LedgerOffset begin, LedgerOffset end, TransactionFilter filter, boolean verbose, String accessToken);

   Flowable<TransactionTree> getTransactionsTrees(LedgerOffset begin, TransactionFilter filter, boolean verbose);

   Flowable<TransactionTree> getTransactionsTrees(LedgerOffset begin, TransactionFilter filter, boolean verbose, String accessToken);
4
  1. Yes filtering in the database can often be more efficient. If you can use it, I would usually recommend it (e.g. for the flat transaction stream). Hard to quantify how big the difference is, I recommend benchmarking for your specific app if you’re interested.
  2. I’m not completely familiar with the history there but afaik the main reason is that it’s not entirely clear what filtering by template id would do: If you filter all events by template id your tree might no longer be connected because some events in the middle get filtered out. For some apps that may be fine but not in general. Another option is to try to avoid those holes by collapsing the child event ids until you get to an unfiltered event. Again, sometimes the right approach sometimes not. The third approach I’ve seen discussed is to filter only the root nodes. Again, sometimes what you want sometimes not. So overall, it’s not really clear what filtering we want to have. I do expect that at some point we add some filtering but probably in a more general way than trying to support only template id filters.

As for deprecating the functions that take a filter, I’ll defer to @stefanobaghino-da

1 Like
5

@cocreature Thanks so much for the help as always.

6

I think at the Java bindings level we can consider the idea of taking only a set of parties instead of a whole filter. However, I would also recommend to consider how we want filtering to happen on the Ledger API (where a transaction filter is currently accepted on the wire). /cc @bernhard (happy to discuss possible improvements to the Ledger API in case you agree and how to reflect those on the Java bindings)

1 Like