Null offset in JSON API streaming endpoints

Questions
#1

The JSON API documentation mentions that it is possible to receive a message of the following form on the streaming endpoint:

{"events": [], "offset": null}

We also occasionally observe this in tests.

However, the documentation does not explain when offset will be null. Looking at the code, it seems to come from the offset being LedgerBegin. Can I trigger this reliably somehow? I’ve tried to run a query against a ledger with no transaction but the offset is still not null for both Sandbox and Sandbox-Classic.

1 Like
#2

https://docs.daml.com/json-api/index.html#contracts-query-stream

The stream is guaranteed to send an offset immediately at the beginning of this “live” data, which may or may not contain any events ; if it does not contain events and no events were emitted before, it may be null or a string; otherwise, it will be a string.

Yes, null is LedgerBegin…
I believe you have never seen null just because sandbox does not emit LedgerBegin even on totally empty ledger. That is sandbox implementation.

1 Like
#3

Thanks!
I see it in a test running against Sandbox so it does seem to emit it sometimes (or the JSON API emits null in other cases as well) but I see it only in ~1 out of 1000 test runs so not quite sure what makes that one special.

#4

I will check the JSON API implementation… I believe there might be a case when {"events": [], "offset": null} emitted by JSON API as part of the heart-beating.

1 Like
#5

@cocreature can you point me to this test?

I did a few manual tests, streaming /v1/stream/query from empty sandbox and sandbox-classic, did not get {"events": [], "offset": null}

daml sandbox-classic --ledgerid MyLedger ./.daml/dist/quickstart-0.0.1.dar

{"events":[],"offset":"0000000000000001"}
{"events":[],"offset":"0000000000000001"}
...

daml sandbox --ledgerid MyLedger ./.daml/dist/quickstart-0.0.1.dar

{"events":[],"offset":"00000000000000020000000000000000"}
{"events":[],"offset":"00000000000000020000000000000000"}
...

but technically {"events": [], "offset": null} is a valid message in this case.

#6

The test is daml/BUILD.bazel at 05d49b37c346855b5f793630be7c2926e0b9da67 · digital-asset/daml · GitHub. I noticed the issue because there is currently a bug in the JavaScript libraries where they fail if offset is null so the test fails. I’ve added some more debugging output in Debug build-and-lint tests by cocreature · Pull Request #6913 · digital-asset/daml · GitHub and ran the test 1000 times and managed to capture logs: https://gist.github.com/cocreature/48d049b1c4c182491c5804ccb7f3cd8b
The relevant part is

2020-07-30T09:24:25.4075810Z   console.log ../node_modules/@daml/ledger/index.js:503
2020-07-30T09:24:25.4076350Z     Message: {"events":[],"offset":null}

This is the first message, afterwards the test fails because there are no more messages. Judging from the fact that it’s the first message, I expect it comes from this query https://github.com/digital-asset/daml/blob/05d49b37c346855b5f793630be7c2926e0b9da67/language-support/ts/codegen/tests/ts/build-and-lint-test/src/tests/test.ts#L129.

#7

Maybe there is an asynchronous (as in it happens after the port file has been written and the ledger will accept other requests before it) process in the ledger (maybe the initial configuration?) which usually happens before but if you’re very quick you can send the query before it happens and the offset is null?

#8

I have found a way… start json-api with --websocket-config="maxDuration=120,heartBeatPer=0", and it will start heart-beating non-stop :), this way you will get a few {"events":[],"offset":null} messages, before JSON API completes the initial ACS retrieval.

$ daml json-api  --ledger-host=localhost --ledger-port=6865 --http-port=7575 \
 --package-reload-interval 5h --allow-insecure-tokens \
 --websocket-config="maxDuration=120,heartBeatPer=0"

here is the output from test websocket client:

{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":null}
{"events":[],"offset":"0000000000000001"}
{"events":[],"offset":"0000000000000001"}
...
#9

Perfect, thank you!