Some DAML on Fabric tests failed

I have run the DAML on Fabric tests described in the Step 8. Run Ledger API Test Tool section of the Quick Start Guide and some tests failed.

What can be the reason for that?

The full test report with successes and failures:


################################################################################
#
# TEST REPORT
#
################################################################################

### SUCCESSES

ActiveContractsServiceIT
- [ActiveContractsServiceIT:ACSarchivedContracts] The ActiveContractService does not return archived contracts ... Success (21076 ms)
- [ActiveContractsServiceIT:ACSusableOffset] The ActiveContractService should return a usable offset to resume streaming transactions ... Success (18632 ms)
- [ActiveContractsServiceIT:ACSverbosity] The ActiveContractService should emit field names only if the verbose flag is set to true ... Success (17987 ms)
- [ActiveContractsServiceIT:ACSmultiParty] The ActiveContractsService should return contracts for the requesting parties ... Success (24360 ms)
- [ActiveContractsServiceIT:ACSagreementText] The ActiveContractService should properly fill the agreementText field ... Success (3100 ms)
- [ActiveContractsServiceIT:ACSeventId] The ActiveContractService should properly fill the eventId field ... Success (2058 ms)
- [ActiveContractsServiceIT:ACSnoWitnessedContracts] The ActiveContractService should not return witnessed contracts ... Success (4296 ms)
- [ActiveContractsServiceIT:ACSnoDivulgedContracts] The ActiveContractService should not return divulged contracts ... Success (5392 ms)

LedgerConfigurationServiceIT
- [LedgerConfigurationServiceIT:ConfigSucceeds] Return a valid configuration for a valid request ... Success (61 ms)
- [LedgerConfigurationServiceIT:ConfigLedgerId] Return NOT_FOUND to invalid ledger identifier ... Success (46 ms)
- [LedgerConfigurationServiceIT:CSLSuccessIfMaxDedplicationTimeRight] Submission returns OK if deduplication time is within the accepted interval ... Success (1482 ms)
- [LedgerConfigurationServiceIT:CSLSuccessIfMaxDeduplicationTimeExceeded] Submission returns OK if deduplication time is too high ... Success (1293 ms)

PartyManagementServiceIT
- [PartyManagementServiceIT:PMNonEmptyParticipantID] Asking for the participant identifier should return a non-empty string ... Success (89 ms)
- [PartyManagementServiceIT:PMAllocateWithHint] It should be possible to provide a hint when allocating a party ... Success (1191 ms)
- [PartyManagementServiceIT:PMAllocateWithoutHint] It should be possible to not provide a hint when allocating a party ... Success (1245 ms)
- [PartyManagementServiceIT:PMAllocateWithoutDisplayName] It should be possible to not provide a display name when allocating a party ... Success (1054 ms)
- [PartyManagementServiceIT:PMAllocateDuplicateDisplayName] It should be possible to allocate parties with the same display names ... Success (27896 ms)
- [PartyManagementServiceIT:PMAllocateOneHundred] It should create unique party names when allocating many parties ... Success (26817 ms)
- [PartyManagementServiceIT:PMGetParties] It should get details for multiple parties, if they exist ... Success (28158 ms)
- [PartyManagementServiceIT:PMListKnownParties] It should list all known, previously-allocated parties ... Success (30204 ms)
- [PartyManagementServiceIT:PMGetPartiesIsLocal] GetParties should correctly report whether parties are local or not ... Success (5058 ms)

PackageServiceIT
- [PackageServiceIT:PackagesList] Listing packages should return a result ... Success (33 ms)
- [PackageServiceIT:PackagesGet] Getting package content should return a valid result ... Success (63 ms)
- [PackageServiceIT:PackagesGetUnknown] Getting package content for an unknown package should fail ... Success (33 ms)
- [PackageServiceIT:PackagesStatus] Getting package status should return a valid result ... Success (46 ms)
- [PackageServiceIT:PackagesStatusUnknown] Getting package status for an unknown package should fail ... Success (31 ms)

ContractKeysIT
- [ContractKeysIT:CKFetchOrLookup] Divulged contracts cannot be fetched or looked up by key by non-stakeholders ... Success (10739 ms)
- [ContractKeysIT:CKNoFetchUndisclosed] Contract Keys should reject fetching an undisclosed contract ... Success (11984 ms)
- [ContractKeysIT:CKMaintainerScoped] Contract keys should be scoped by maintainer ... Success (20617 ms)
- [ContractKeysIT:CKRecreate] Contract keys can be recreated in single transaction ... Success (4438 ms)
- [ContractKeysIT:CKTransients] Contract keys created by transient contracts are properly archived ... Success (6216 ms)
- [ContractKeysIT:CKExposedByTemplate] The contract key should be exposed if the template specifies one ... Success (2763 ms)
- [ContractKeysIT:CKExerciseByKey] Exercising by key should be possible only when the corresponding contract is available ... Success (3475 ms)

ClosedWorldIT
- [ClosedWorldIT:ClosedWorldObserver] Cannot execute a transaction that references unallocated observer parties ... Success (1985 ms)

CommandServiceIT
- [CommandServiceIT:CSsubmitAndWait] SubmitAndWait creates a contract of the expected template ... Success (2154 ms)
- [CommandServiceIT:CSsubmitAndWaitForTransactionId] SubmitAndWaitForTransactionId returns a valid transaction identifier ... Success (2404 ms)
- [CommandServiceIT:CSsubmitAndWaitForTransaction] SubmitAndWaitForTransaction returns a transaction ... Success (2257 ms)
- [CommandServiceIT:CSsubmitAndWaitForTransactionTree] SubmitAndWaitForTransactionTree returns a transaction tree ... Success (2295 ms)
- [CommandServiceIT:CSduplicateSubmitAndWait] SubmitAndWait should fail on duplicate requests ... Success (2167 ms)
- [CommandServiceIT:CSduplicateSubmitAndWaitForTransactionId] SubmitAndWaitForTransactionId should fail on duplicate requests ... Success (2175 ms)
- [CommandServiceIT:CSduplicateSubmitAndWaitForTransaction] SubmitAndWaitForTransaction should fail on duplicate requests ... Success (2393 ms)
- [CommandServiceIT:CSduplicateSubmitAndWaitForTransactionTree] SubmitAndWaitForTransactionTree should fail on duplicate requests ... Success (2653 ms)
- [CommandServiceIT:CSsubmitAndWaitForTransactionIdInvalidLedgerId] SubmitAndWaitForTransactionId should fail for invalid ledger ids ... Success (1361 ms)
- [CommandServiceIT:CSsubmitAndWaitForTransactionInvalidLedgerId] SubmitAndWaitForTransaction should fail for invalid ledger ids ... Success (1351 ms)
- [CommandServiceIT:CSsubmitAndWaitForTransactionTreeInvalidLedgerId] SubmitAndWaitForTransactionTree should fail for invalid ledger ids ... Success (1309 ms)
- [CommandServiceIT:CSRefuseBadParameter] The submission of a creation that contains a bad parameter label should result in an INVALID_ARGUMENT ... Success (1231 ms)
- [CommandServiceIT:CSReturnStackTrace] A submission resulting in an interpretation error should return the stack trace ... Success (3234 ms)
- [CommandServiceIT:CSDiscloseCreateToObservers] Disclose create to observers ... Success (8066 ms)
- [CommandServiceIT:CSDiscloseExerciseToObservers] Disclose exercise to observers ... Success (9021 ms)
- [CommandServiceIT:CSHugeCommandSubmission] The server should accept a submission with 15 commands ... Success (3205 ms)
- [CommandServiceIT:CSCallablePayout] Run CallablePayout and return the right events ... Success (8341 ms)
- [CommandServiceIT:CSReadyForExercise] It should be possible to exercise a choice on a created contract ... Success (3485 ms)
- [CommandServiceIT:CSBadNumericValues] Reject unrepresentable numeric values ... Success (1111 ms)
- [CommandServiceIT:CSCreateAndExercise] Implement create-and-exercise correctly ... Success (2279 ms)
- [CommandServiceIT:CSBadCreateAndExercise] Fail create-and-exercise on bad create arguments ... Success (1189 ms)
- [CommandServiceIT:CSCreateAndBadExerciseArguments] Fail create-and-exercise on bad choice arguments ... Success (1232 ms)
- [CommandServiceIT:CSCreateAndBadExerciseChoice] Fail create-and-exercise on invalid choice ... Success (1213 ms)

HealthServiceIT
- [HealthServiceIT:HScheck] The Health.Check endpoint reports everything is well ... Success (39 ms)
- [HealthServiceIT:HSwatch] The Health.Watch endpoint reports everything is well ... Success (1038 ms)

CommandDeduplicationIT
- [CommandDeduplicationIT:CDSimpleDeduplication] Deduplicate commands within the deduplication time window ... Success (8482 ms)
- [CommandDeduplicationIT:CDStopOnSubmissionFailure] Stop deduplicating commands on submission failure ... Success (1078 ms)
- [CommandDeduplicationIT:CDStopOnCompletionFailure] Stop deduplicating commands on completion failure ... Success (5103 ms)
- [CommandDeduplicationIT:CDSimpleDeduplicationCommandClient] Deduplicate commands within the deduplication time window using the command client ... Success (8841 ms)
- [CommandDeduplicationIT:CDDeduplicateSubmitter] Commands with identical submitter and command identifier should be deduplicated by the submission client ... Success (2374 ms)
- [CommandDeduplicationIT:CDDeduplicateSubmitterCommandClient] Commands with identical submitter and command identifier should be deduplicated by the command client ... Success (2280 ms)

CommandSubmissionCompletionIT
- [CommandSubmissionCompletionIT:CSCCompletions] Read completions correctly with a correct application identifier and reading party ... Success (1077 ms)
- [CommandSubmissionCompletionIT:CSCNoCompletionsWithoutRightAppId] Read no completions without the correct application identifier ... Success (5621 ms)
- [CommandSubmissionCompletionIT:CSCNoCompletionsWithoutRightParty] Read no completions without the correct party ... Success (5747 ms)
- [CommandSubmissionCompletionIT:CSCRefuseBadChoice] The submission of an exercise of a choice that does not exist should yield INVALID_ARGUMENT ... Success (721 ms)
- [CommandSubmissionCompletionIT:CSCSubmitWithInvalidLedgerId] Submit should fail for an invalid ledger identifier ... Success (614 ms)
- [CommandSubmissionCompletionIT:CSCDisallowEmptyTransactionsSubmission] The submission of an empty command should be rejected with INVALID_ARGUMENT ... Success (502 ms)
- [CommandSubmissionCompletionIT:CSCHandleMultiPartySubscriptions] Listening for completions should support multi-party subscriptions ... Success (1516 ms)

ConfigManagementServiceIT
- [ConfigManagementServiceIT:CMSetAndGetTimeModel] It should be able to get, set and restore the time model ... Success (1541 ms)

SemanticTests
- [SemanticTests:SemanticDoubleSpend] Cannot double spend across transactions ... Success (5844 ms)
- [SemanticTests:SemanticDoubleSpendSameTx] Cannot double spend within a transaction ... Success (4514 ms)
- [SemanticTests:SemanticDoubleSpendShared] Different parties cannot spend the same contract ... Success (12287 ms)
- [SemanticTests:SemanticPaintOffer] Conduct the paint offer workflow successfully ... Success (7238 ms)
- [SemanticTests:SemanticPaintCounterOffer] Conduct the paint counter-offer worflow successfully ... Success (7494 ms)
- [SemanticTests:SemanticPartialSignatories] A signatory should not be able to create a contract on behalf of two parties ... Success (1827 ms)
- [SemanticTests:SemanticAcceptOnBehalf] It should not be possible to exercise a choice without the consent of the controller ... Success (4171 ms)
- [SemanticTests:SemanticPrivacyProjections] Test visibility via contract fetches for the paint-offer flow ... Success (72938 ms)
- [SemanticTests:SemanticDivulgence] Respect divulgence rules ... Success (37663 ms)

PackageManagementServiceIT
- [PackageManagementServiceIT:PackageManagementEmptyUpload] An attempt at uploading an empty payload should fail ... Success (36 ms)
- [PackageManagementServiceIT:PackageManagementLoad] Concurrent uploads of the same package should be idempotent and result in the package being available for use ... Success (15262 ms)

TransactionServiceIT
- [TransactionServiceIT:TXBeginToBegin] An empty stream should be served when getting transactions from and to the beginning of the ledger ... Success (2251 ms)
- [TransactionServiceIT:TXTreesBeginToBegin] An empty stream of trees should be served when getting transactions from and to the beginning of the ledger ... Success (10963 ms)
- [TransactionServiceIT:TXEndToEnd] An empty stream should be served when getting transactions from and to the end of the ledger ... Success (2032 ms)
- [TransactionServiceIT:TXServeUntilCancellation] Items should be served until the client cancels ... Success (5626 ms)
- [TransactionServiceIT:TXServeTreesUntilCancellation] Trees should be served until the client cancels ... Success (8981 ms)
- [TransactionServiceIT:TXTreeBlinding] Trees should be served according to the blinding/projection rules ... Success (24595 ms)
- [TransactionServiceIT:TXRejectEmptyFilter] A query with an empty transaction filter should be rejected with an INVALID_ARGUMENT status ... Success (1818 ms)
- [TransactionServiceIT:TXCompleteOnLedgerEnd] A stream should complete as soon as the ledger end is hit ... Success (5354 ms)
- [TransactionServiceIT:TXCompleteTreesOnLedgerEnd] A stream of trees should complete as soon as the ledger end is hit ... Success (5183 ms)
- [TransactionServiceIT:TXProcessInTwoChunks] Serve the complete sequence of transactions even if processing is stopped and resumed ... Success (6358 ms)
- [TransactionServiceIT:TXParallel] The same data should be served for more than 1 identical, parallel requests ... Success (4127 ms)
- [TransactionServiceIT:TXNotDivulge] Data should not be exposed to parties unrelated to a transaction ... Success (3399 ms)
- [TransactionServiceIT:TXRejectBeginAfterEnd] A request with the end before the begin should be rejected with INVALID_ARGUMENT ... Success (2503 ms)
- [TransactionServiceIT:TXHideCommandIdToNonSubmittingStakeholders] A transaction should be visible to a non-submitting stakeholder but its command identifier should be empty ... Success (2528 ms)
- [TransactionServiceIT:TXFilterByTemplate] The transaction service should correctly filter by template identifier ... Success (2273 ms)
- [TransactionServiceIT:TXUseCreateToExercise] Should be able to directly use a contract identifier to exercise a choice ... Success (3478 ms)
- [TransactionServiceIT:TXContractIdFromExerciseWhenFilter] Expose contract identifiers that are results of exercising choices when filtering by template ... Success (3397 ms)
- [TransactionServiceIT:TXRejectOnFailingAssertion] Reject a transaction on a failing assertion ... Success (2266 ms)
- [TransactionServiceIT:TXCreateWithAnyType] Creates should not have issues dealing with any type of argument ... Success (2353 ms)
- [TransactionServiceIT:TXExerciseWithAnyType] Exercise should not have issues dealing with any type of argument ... Success (4296 ms)
- [TransactionServiceIT:TXVeryLongList] Accept a submission with a very long list (10,000 items) ... Success (7866 ms)
- [TransactionServiceIT:TXNotArchiveNonConsuming] Expressing a non-consuming choice on a contract should not result in its archival ... Success (9198 ms)
- [TransactionServiceIT:TXRequireAuthorization] Require only authorization of chosen branching signatory ... Success (2074 ms)
- [TransactionServiceIT:TXNotDiscloseCreateToNonSignatory] Not disclose create to non-chosen branching signatory ... Success (9287 ms)
- [TransactionServiceIT:TXDiscloseCreateToSignatory] Disclose create to the chosen branching controller ... Success (3140 ms)
- [TransactionServiceIT:TXNotDiscloseCreateToNonChosenBranchingController] Not disclose create to non-chosen branching controller ... Success (9742 ms)
- [TransactionServiceIT:TXDiscloseCreateToObservers] Disclose create to observers ... Success (4334 ms)
- [TransactionServiceIT:TXUnitAsArgumentToNothing] DAML engine returns Unit as argument to Nothing ... Success (2441 ms)
- [TransactionServiceIT:TXAgreementText] Expose the agreement text for templates with an explicit agreement text ... Success (2393 ms)
- [TransactionServiceIT:TXAgreementTextDefault] Expose the default text for templates without an agreement text ... Success (3069 ms)
- [TransactionServiceIT:TXStakeholders] Expose the correct stakeholders ... Success (3122 ms)
- [TransactionServiceIT:TXNoContractKey] There should be no contract key if the template does not specify one ... Success (3586 ms)
- [TransactionServiceIT:TXMultiActorChoiceOk] Accept exercising a well-authorized multi-actor choice ... Success (7608 ms)
- [TransactionServiceIT:TXMultiActorChoiceOkCoincidingControllers] Accept exercising a well-authorized multi-actor choice with coinciding controllers ... Success (7475 ms)
- [TransactionServiceIT:TXRejectMultiActorMissingAuth] Reject exercising a multi-actor choice with missing authorizers ... Success (4161 ms)
- [TransactionServiceIT:TXRejectMultiActorExcessiveAuth] Reject exercising a multi-actor choice with too many authorizers ... Success (5779 ms)
- [TransactionServiceIT:TXNoReorder] Don't reorder fields in data structures of choices ... Success (3284 ms)
- [TransactionServiceIT:TXSingleMultiSame] The same transaction should be served regardless of subscribing as one or multiple parties ... Success (4427 ms)
- [TransactionServiceIT:TXSingleMultiSameTrees] The same transaction trees should be served regardless of subscribing as one or multiple parties ... Success (4372 ms)
- [TransactionServiceIT:TXSingleMultiSameStakeholders] The same transaction should be served to all stakeholders ... Success (8838 ms)
- [TransactionServiceIT:TXSingleMultiSameTreesStakeholders] The same transaction trees should be served to all stakeholders ... Success (8893 ms)
- [TransactionServiceIT:TXFetchContractCreatedInTransaction] It should be possible to fetch a contract created within a transaction ... Success (3172 ms)
- [TransactionServiceIT:TXFlatTransactionsWrongLedgerId] The getTransactions endpoint should reject calls with the wrong ledger identifier ... Success (907 ms)
- [TransactionServiceIT:TXTransactionTreesWrongLedgerId] The getTransactionTrees endpoint should reject calls with the wrong ledger identifier ... Success (1067 ms)
- [TransactionServiceIT:TXTransactionTreeByIdWrongLedgerId] The getTransactionTreeById endpoint should reject calls with the wrong ledger identifier ... Success (1182 ms)
- [TransactionServiceIT:TXFlatTransactionByIdWrongLedgerId] The getFlatTransactionById endpoint should reject calls with the wrong ledger identifier ... Success (1106 ms)
- [TransactionServiceIT:TXTransactionTreeByEventIdWrongLedgerId] The getTransactionTreeByEventId endpoint should reject calls with the wrong ledger identifier ... Success (968 ms)
- [TransactionServiceIT:TXFlatTransactionByEventIdWrongLedgerId] The getFlatTransactionByEventId endpoint should reject calls with the wrong ledger identifier ... Success (1080 ms)
- [TransactionServiceIT:TXTransactionTreeByIdWrongLedgerId] The ledgerEnd endpoint should reject calls with the wrong ledger identifier ... Success (29 ms)
- [TransactionServiceIT:TXTransactionTreeById] Expose a visible transaction tree by identifier ... Success (4746 ms)
- [TransactionServiceIT:TXInvisibleTransactionTreeById] Do not expose an invisible transaction tree by identifier ... Success (9376 ms)
- [TransactionServiceIT:TXTransactionTreeByIdNotFound] Return NOT_FOUND when looking up an inexistent transaction tree by identifier ... Success (1438 ms)
- [TransactionServiceIT:TXTransactionTreeByIdNotFound] Return INVALID_ARGUMENT when looking up a transaction tree by identifier without specifying a party ... Success (27 ms)
- [TransactionServiceIT:TXTransactionTreeByIdSameAsTransactionStream] Expose the same events for each transaction as the output of getTransactionTrees ... Success (8365 ms)
- [TransactionServiceIT:TXFlatTransactionById] Expose a visible transaction by identifier ... Success (4233 ms)
- [TransactionServiceIT:TXInvisibleFlatTransactionById] Do not expose an invisible flat transaction by identifier ... Success (3471 ms)
- [TransactionServiceIT:TXFlatTransactionByIdNotFound] Return NOT_FOUND when looking up an inexistent flat transaction by identifier ... Success (1056 ms)
- [TransactionServiceIT:TXFlatTransactionByIdNotFound] Return INVALID_ARGUMENT when looking up a flat transaction by identifier without specifying a party ... Success (28 ms)
- [TransactionServiceIT:TXFlatTransactionByIdSameAsTransactionStream] Expose the same events for each transaction as the output of getTransactions ... Success (8380 ms)
- [TransactionServiceIT:TXTransactionTreeByEventId] Expose a visible transaction tree by event identifier ... Success (4196 ms)
- [TransactionServiceIT:TXInvisibleTransactionTreeByEventId] Do not expose an invisible transaction tree by event identifier ... Success (9633 ms)
- [TransactionServiceIT:TXTransactionTreeByEventIdInvalid] Return INVALID when looking up an invalid transaction tree by event identifier ... Success (1583 ms)
- [TransactionServiceIT:TXTransactionTreeByEventIdNotFound] Return NOT_FOUND when looking up an inexistent transaction tree by event identifier ... Success (1204 ms)
- [TransactionServiceIT:TXTransactionTreeByEventIdNotFound] Return INVALID_ARGUMENT when looking up a transaction tree by event identifier without specifying a party ... Success (27 ms)
- [TransactionServiceIT:TXFlatTransactionByEventId] Expose a visible flat transaction by event identifier ... Success (4280 ms)
- [TransactionServiceIT:TXInvisibleFlatTransactionByEventId] Do not expose an invisible flat transaction by event identifier ... Success (4316 ms)
- [TransactionServiceIT:TXFlatTransactionByEventIdInvalid] Return INVALID when looking up a flat transaction by an invalid event identifier ... Success (1237 ms)
- [TransactionServiceIT:TXFlatTransactionByEventIdNotFound] Return NOT_FOUND when looking up an inexistent flat transaction by event identifier ... Success (1041 ms)
- [TransactionServiceIT:TXFlatTransactionByEventIdNotFound] Return INVALID_ARGUMENT when looking up a flat transaction by event identifier without specifying a party ... Success (30 ms)
- [TransactionServiceIT:TXSingleSubscriptionInOrder] Archives should always come after creations when subscribing as a single party ... Success (59001 ms)
- [TransactionServiceIT:TXMultiSubscriptionInOrder] Archives should always come after creations when subscribing as more than on party ... Success (77134 ms)
- [TransactionServiceIT:TXFlatSubsetOfTrees] The event identifiers in the flat stream should be a subset of those in the trees stream ... Success (112752 ms)
- [TransactionServiceIT:TXFlatWitnessesSubsetOfTrees] The witnesses in the flat stream should be a subset of those in the trees stream ... Success (130052 ms)

IdentityIT
- [IdentityIT:IdNotEmpty] A ledger should return a non-empty string as its identity ... Success (37 ms)

### FAILURES

LotsOfPartiesIT
- [LotsOfPartiesIT:LOPseeTransactionsInMultipleSinglePartySubscriptions] Observers should see transactions in multiple single-party subscriptions ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [LotsOfPartiesIT:LOPseeTransactionsInSingleMultiPartySubscription] Observers should see transactions in a single multi-party subscription ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [LotsOfPartiesIT:LOPseeActiveContractsInMultipleSinglePartySubscriptions] Observers should see active contracts in multiple single-party subscriptions ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [LotsOfPartiesIT:LOPseeActiveContractsInSingleMultiPartySubscription] Observers should see active contracts in a single multi-party subscription ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

DivulgenceIT
- [DivulgenceIT:DivulgenceTx] Divulged contracts should not be exposed by the transaction service ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [DivulgenceIT:DivulgenceAcs] Divulged contracts should not be exposed by the active contract service ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [DivulgenceIT:DivulgenceKeys] Divulgence should behave as expected in a workflow involving keys ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

WitnessesIT
- [WitnessesIT:RespectDisclosureRules] The ledger should respect disclosure rules ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

WronglyTypedContractIdIT
- [WronglyTypedContractIdIT:WTExerciseFails] Exercising on a wrong type fails ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [WronglyTypedContractIdIT:WTFetchFails] Fetching of the wrong type fails ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

TransactionScaleIT
- [TransactionScaleIT:TXLargeCommand] Accept huge submissions with a large number of commands ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [TransactionScaleIT:TXManyCommands] Accept many, large commands at once ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

ActiveContractsServiceIT
- [ActiveContractsServiceIT:ACSinvalidLedgerId] The ActiveContractService should fail for requests with an invalid ledger identifier ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [ActiveContractsServiceIT:ACSemptyResponse] The ActiveContractService should succeed with an empty response if no contracts have been created for a party ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [ActiveContractsServiceIT:ACSallContracts] The ActiveContractService should return all active contracts ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [ActiveContractsServiceIT:ACSfilterContracts] The ActiveContractService should return contracts filtered by templateId ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

This is probably due to the fact that the guide suggests to use --all-tests, which is a legacy option. This also runs tests that mutate the global state of the ledger (when testing the configuration service) and perform operations that stress test the ledger (by allocating ~1k parties in a single go). Since these operations are currently using a common, hard-coded timeout (which we are already working toward removing), it’s quite likely that the “lots of parties” test is making the life difficult for all other tests.

More recent versions of the Ledger API Test Tool deprecate --all-tests and automatically run all the tests, making sure to schedule stress tests and global tests to run sequentially at the end of the test run.

For the time being I would recommend that you run the test tool without the --all-tests option and run the tests one further time specifying --include LotsOfPartiesIT,TransactionScaleIT,ConfigManagementServiceIT --concurrent-test-runs=1, which emulates the behavior of the more recent test tool versions.

4 Likes

Thank you!

2 Likes

Without the --all-tests option I have the following failures:

### FAILURES

PartyManagementServiceIT
- [PartyManagementServiceIT:PMAllocateDuplicateDisplayName] It should be possible to allocate parties with the same display names ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [PartyManagementServiceIT:PMAllocateOneHundred] It should create unique party names when allocating many parties ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [PartyManagementServiceIT:PMGetParties] It should get details for multiple parties, if they exist ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [PartyManagementServiceIT:PMListKnownParties] It should list all known, previously-allocated parties ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

TransactionServiceIT
- [TransactionServiceIT:TXFlatWitnessesSubsetOfTrees] The witnesses in the flat stream should be a subset of those in the trees stream ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

The second way resulted in all failures:

bgy@BGY fixture % java -jar ledger-api-test-tool.jar localhost:6865 --timeout-scale-factor=10 --include LotsOfPartiesIT,TransactionScaleIT,ConfigManagementServiceIT --concurrent-test-runs=1
12:33:48.504 INFO  c.d.l.a.t.i.p.ParticipantSession@[kka.actor.default-dispatcher-6] - Connecting to participant at localhost:6865...
12:33:48.509 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[main] - Running 7 tests, 1 at a time.
12:33:48.512 INFO  c.d.l.a.t.i.p.ParticipantSession@[kka.actor.default-dispatcher-6] - gRPC thread factory instantiated with pool 'grpc-event-loop-localhost-6865' (daemon threads: false)
12:33:48.682 INFO  c.d.l.a.t.i.p.ParticipantSession@[kka.actor.default-dispatcher-6] - gRPC event loop thread group instantiated with 4 threads using pool 'grpc-event-loop-localhost-6865'
12:33:48.917 INFO  c.d.l.a.t.i.p.ParticipantSession@[kka.actor.default-dispatcher-6] - Connection to participant at localhost:6865
12:33:50.569 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[kka.actor.default-dispatcher-6] - Started 'Observers should see transactions in multiple single-party subscriptions' with a timeout of 20 minutes.
12:34:20.828 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[kka.actor.default-dispatcher-6] - Finished 'Observers should see transactions in multiple single-party subscriptions'.
12:34:20.948 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-12] - Started 'Observers should see transactions in a single multi-party subscription' with a timeout of 20 minutes.
12:34:52.754 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-11] - Finished 'Observers should see transactions in a single multi-party subscription'.
12:34:52.969 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-11] - Started 'Observers should see active contracts in multiple single-party subscriptions' with a timeout of 20 minutes.
12:35:28.605 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[kka.actor.default-dispatcher-8] - Finished 'Observers should see active contracts in multiple single-party subscriptions'.
12:35:28.759 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-14] - Started 'Observers should see active contracts in a single multi-party subscription' with a timeout of 20 minutes.
12:35:59.046 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-15] - Finished 'Observers should see active contracts in a single multi-party subscription'.
12:35:59.058 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[kka.actor.default-dispatcher-8] - Started 'Accept huge submissions with a large number of commands' with a timeout of 5 minutes.
12:36:32.845 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-16] - Finished 'Accept huge submissions with a large number of commands'.
12:36:32.860 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-15] - Started 'Accept many, large commands at once' with a timeout of 5 minutes.
12:37:02.972 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-19] - Finished 'Accept many, large commands at once'.
12:37:03.037 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-20] - Started 'It should be able to get, set and restore the time model' with a timeout of 5 minutes.
12:37:33.482 INFO  c.d.l.a.t.i.LedgerTestCasesRunner@[ka.actor.default-dispatcher-21] - Finished 'It should be able to get, set and restore the time model'.
12:37:33.513 INFO  c.d.l.a.t.i.p.ParticipantSession@[ka.actor.default-dispatcher-22] - Disconnecting from participant at localhost:6865...
12:37:33.548 INFO  c.d.l.a.t.i.p.ParticipantSession@[ka.actor.default-dispatcher-22] - Connection to participant at localhost:6865 shut down.
12:37:33.563 INFO  c.d.l.a.t.i.p.ParticipantSession@[ka.actor.default-dispatcher-22] - Connection to participant at localhost:6865 closed.

################################################################################
#
# TEST REPORT
#
################################################################################

### FAILURES

ConfigManagementServiceIT
- [ConfigManagementServiceIT:CMSetAndGetTimeModel] It should be able to get, set and restore the time model ... Unexpected failure (StatusRuntimeException)
  ABORTED: Request timed out

TransactionScaleIT
- [TransactionScaleIT:TXLargeCommand] Accept huge submissions with a large number of commands ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [TransactionScaleIT:TXManyCommands] Accept many, large commands at once ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

LotsOfPartiesIT
- [LotsOfPartiesIT:LOPseeTransactionsInMultipleSinglePartySubscriptions] Observers should see transactions in multiple single-party subscriptions ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [LotsOfPartiesIT:LOPseeTransactionsInSingleMultiPartySubscription] Observers should see transactions in a single multi-party subscription ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [LotsOfPartiesIT:LOPseeActiveContractsInMultipleSinglePartySubscriptions] Observers should see active contracts in multiple single-party subscriptions ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [LotsOfPartiesIT:LOPseeActiveContractsInSingleMultiPartySubscription] Observers should see active contracts in a single multi-party subscription ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
bgy@BGY fixture % 

Both failures are quite unexpected, especially because to the best of my understanding the test tool is used as a suite on the DAML-on-Fabric CI pipeline. What setup are you testing this against? Are the index database and participant node co-located? What are their hardware specs?

I use the the default setup, specified in the config-local.yaml file. Must be a hardware limitation, I’m using my Macbook with 8 GB memory.

Will try in a more robust cloud environment where the production instance is deployed.

@gyorgybalazsi this is likely slowness as a result of things running on your local machine. Suggested that you shut down other programs while running fabric, or potentially allocated greater resources to local docker. The 30s timeout indicates that the local machine is likely overloaded and workflows are taking some time to process.

3 Likes

Thanks for the idea, I gave Docker Desktop 8 GB memory with no other apps running, still, these cases timed out:

### FAILURES

PartyManagementServiceIT
- [PartyManagementServiceIT:PMAllocateDuplicateDisplayName] It should be possible to allocate parties with the same display names ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [PartyManagementServiceIT:PMAllocateOneHundred] It should create unique party names when allocating many parties ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [PartyManagementServiceIT:PMGetParties] It should get details for multiple parties, if they exist ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.
- [PartyManagementServiceIT:PMListKnownParties] It should list all known, previously-allocated parties ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

TransactionServiceIT
- [TransactionServiceIT:TXFlatWitnessesSubsetOfTrees] The witnesses in the flat stream should be a subset of those in the trees stream ... Unexpected failure (StatusRuntimeException)
  INTERNAL: The stream has not been completed in 30 seconds.

I will try it in a more robust environment.

Otherwise, the app runs visibly faster with more memory, so the raise is useful even so for local experimentation.

1 Like

Hi @gyorgybalazsi I assume you are running the full suite of tests with default settings? The other configuration you can tweak is how many tests are run in parallel (–concurrent-test-runs=2)

By default the tests will be parallelized up to the number of available CPUs on your machine – this setting will limit it to 2 suites running concurrently.

2 Likes

Two individual test cases, not two test suites.

2 Likes

Thanks, will try. Currently Docker is set to 2CPUs, I don’t know if this setting makes a difference.

1 Like

We rely on the result from System.getRuntime().avalaibleProcessors() for that, which in containers may be subject to bugs (e.g.: here is the ticket that tracked the miscounting on OpenJDK 8 and 9).

2 Likes

With 8GB memory allocated in Docker Desktop and without the --all-tests option I can run the test successfully. This is the command:

java -jar ledger-api-test-tool.jar localhost:11111 --timeout-scale-factor=10 --concurrent-test-runs=2
2 Likes