Timeout response from HTTP-JSON service

Questions
1

I received

The server was not able to produce a timely response to your request.
Please try again in a short while!

from the HTTP-JSON server, this error is not formatted per the documentation, is this to be expected?

2

This can happen when timeouts occur during command submission, when there are problems with the connection between the HTTP-JSON server and the gRPC server, specifically when no data is received within timeout in one direction, from HTTP-JSON server to the gRPC server, or vice versa.
You should have received a service unavailable 503 in this case.

Please see daml/FailureTests.scala at 7c23420876a9cd4de9ca815dacb5969d87f3dbb9 · digital-asset/daml · GitHub for specific test scenarios, where a toxic proxy emulates the issues between the HTTP-JSON server and the gRPC server.

This differs from the situation where there are no issues with the connection between the HTTP-JSON server and the gRPC server, and the gRPC server responds with a DEADLINE_EXCEEDED. This would result in the HTTP-JSON server responding with GATEWAY_TIMEOUT (504).

3

I would expect the HTTP-JSON server to not leak the grpc API abstraction and return one of status codes.

4

You should have received a service unavailable status (503).

5

@Raymond_Roestenburg Are these gRPC timeouts configurable from on the HTTP-JSON Api, akin to how they are configurable in Canton ?

6

You should be able to bump the HTTP timeout by starting the JSON API via something like

java -Dakka.http.server.request-timeout=5s -jar json-api.jar
7

@cocreature I tried it out, but I don’t think that setting prevents the gRPC connection of the Ledger API from closing.

8

I don’t think you get control over that direction, only JSON API <-> client not JSON API <-> gRPC API

9

I don’t think you can configure the gRPC timeouts in the JSON API.

10

Is there a technical reason we can’t expose them?

Something along the lines of it is cumbersome or if you really need to mess with those then you should write your own Ledger-API client are acceptable rebuttals to this request. I am on the fence as to whether my request is sensible or not and understand that I’m in a biased position at the moment.

11

@cocreature is this still the recommended way to bump the HTTP timeout?

I’ve increased the timeout parameters of canton as described here which now means the JSON API is the limiting factor and the above described method does not appear to solve the issue.

This was an error in testing. Can confirm the above does indeed increase the JSON API timeout

1 Like