Data extractor for JSON API

Questions
,
1

Just wondering if there is a plan to create data extractor which listen to JSON API? Just like the existing data extractor working with gRPC Ledger API.

1 Like
2

Hi @Frankie, at this point we have no plans to provide support for running extractor over the JSON API or something like the extractor.

Could you explain why you are looking for that? To me, the main benefit of the JSON API is ease of use. However, in the case of extractor (similar arguments apply to triggers) the underlying ledger API is completely abstracted away from the user. In that case, the JSON API is just an extra component you have to deploy and operate and it’s not going to make things faster.

1 Like
3

There is also a fundamental way in which the JSON API is unsuitable as an extractor data source:

Ordinary ledger applications are meant to be built on the active contract set, as it is defined now. As a rule, they don’t have access to prior/archived contracts, and JSON API takes advantage of this to deliver an optimized experience for such ordinary applications.

There is a category of applications set aside from this one: audit applications. By contrast, JSON API is not suited to this kind of application. Extractor counts as such an audit application because it records the ledger as it has been historically, not just as it is “now”. JSON API doesn’t facilitate this kind of access to the ledger.

5 Likes
4

This is mainly for reporting purposes. The ledger operates on acs while the archived contracts are streamed out for historical reporting. Things like reporting on completed trades or delivered orders.

5

@Frankie are you looking for the extractor to give a different user facing interface than the current extractor, or for the current extractor to connect to the JSON API instead of the grpc API on the backend.

If the former (different UX) - what would you want to see in extractor?
If the latter (different backend) - can you shed more color on the reason to prefer JSON/http over grpc?

6

Hi @Shaul, I’m looking at the latter one. This question actually came out when me and my colleagues were discussing what API we should expose to our ledger users. The choices are JSON, gPRC or both. If we do gRPC, users can stream data out using Data Extractor and build report on top of it. For JSON user, there is no such tool exist at the moment. Maybe it is the best to leave it the user to build one fits into their requirements? Or maybe we should expose both interfaces?

7

As @Stephen said, the HTTP JSON API we provide has been designed specifically for ACS-based use-cases and is fundamentally unable to support audit-style use-cases. That is not to say it would be impossible to design a JSON API that supports auditing; it is just not what we have done.

So as of this point in time, if you want your users to be able to develop audit-style applications, you have two options:

  1. Expose the gRPC API to them directly.
  2. Build a gRPC client yourself that exposes whatever audit information you want on another protocol, which may happen to also be JSON over HTTP, but this will be a very different server from the HTTP JSON API we provide.
8

As for the general question of whether you should expose the gRPC API, the JSON API or both to your users I think there are a few different options depending on the usecase:

  1. Exposing the gRPC APi gives users the most flexibility and the chance to achieve the highest performance when interacting with your ledger. If you can do this, I’d strongly recommend to do this. However, in some cases exposing gRPC can be a bit tricky if there are proxies and firewalls involved in your setup.

  2. Exposing the JSON API is interesting. If you can expose the gRPC API this is in some sense redundant. Users could host their own JSON API and they might want to to be in control of the database and be able to tweak indices or other parameters. However, even if you do expose the gRPC API, exposing the JSON API can be very convenient since it removes the need for your users to host it themselves. If they just want to build a web UI they now only need a static file server to serve a react application talking to your ledger.
    If exposing gRPC is not an option, e.g., due to a proxy that does not support it, the JSON API can provide an escape hatch albeit with limited functionality.

So overall, I’d suggest to expose both to your users and let them chose if they have a usecase where hosting their own JSON API makes sense or whether they can get by with what you host.

1 Like