Inconsistent behavior from the ledger

Questions Features & Bugs
1

We are experiencing inconsistent behavior with our app
A choice that was working previously stopped creating a certain contract, with out any change in the source DAR files.
The error we are getting is

{
    "errors": [
        "NOT_FOUND: CONTRACT_KEY_NOT_FOUND(11,5c7483a5): dependency error:
	couldn't find key: GlobalKey(44dd02ab2a7dc3292f33a3a7273f283e28bd4a5bdc0bbe8bb46386f32f1f5bae:
	Main.MinistryOfTreasury:BidInvitationOfferToBanker, ValueRecord(None,ImmArray((None,ValueParty(Ministry Of Finance)),
	(None,ValueParty(Bank Poalim Broker)),(None,ValueText(bid)))))"
    ],
    "status": 404

}

an excerpt from the log

24-03-2022 19:36:47.217 [http-json-ledger-api-akka.actor.default-dispatcher-12] INFO  com.daml.http.Endpoints - Incoming POST request on http://localhost:7575/v1/exercise from 127.0.0.1, context: {instance_uuid: "1a7ab657-6859-4737-8510-b93120b8c2f5", request_id: "13b28b1c-3ea6-4504-9a74-3fe83d88f530"}
24-03-2022 19:36:47.220 [http-json-ledger-api-akka.actor.default-dispatcher-5] INFO  com.daml.http.CommandService - Submitting exercise command, context: {command_id: "b672a303-90ef-4214-a717-fb17673ec2f9", ledger_id: "Hackathon", instance_uuid: "1a7ab657-6859-4737-8510-b93120b8c2f5", application_id: "Hackathon", read_as: "List(Bank Leumi Broker)", act_as: "List(Bank Leumi Broker)", request_id: "13b28b1c-3ea6-4504-9a74-3fe83d88f530", choice: "AcceptBidInvitationFromIssuer", template_id: "TemplateId(44dd02ab2a7dc3292f33a3a7273f283e28bd4a5bdc0bbe8bb46386f32f1f5bae,Main.MinistryOfTreasury,Banker)", contract_id: ""}     
24-03-2022 19:36:47.354 [http-json-ledger-api-akka.actor.default-dispatcher-5] ERROR com.daml.http.CommandService - exercise failure, context: {ledger_id: "Hackathon", instance_uuid: "1a7ab657-6859-4737-8510-b93120b8c2f5", application_id: "Hackathon", read_as: "List(Bank Leumi Broker)", act_as: "List(Bank Leumi Broker)", request_id: "13b28b1c-3ea6-4504-9a74-3fe83d88f530", choice: "AcceptBidInvitationFromIssuer", 
template_id: "TemplateId(44dd02ab2a7dc3292f33a3a7273f283e28bd4a5bdc0bbe8bb46386f32f1f5bae,Main.MinistryOfTreasury,Banker)", contract_id: ""}
io.grpc.StatusRuntimeException: NOT_FOUND: CONTRACT_KEY_NOT_FOUND(11,9423db13): dependency error: couldn't find key: GlobalKey(44dd02ab2a7dc3292f33a3a7273f283e28bd4a5bdc0bbe8bb46386f32f1f5bae:Main.MinistryOfTreasury:BidInvitationOfferToBanker, ValueRecord(None,ImmArray((None,ValueParty(Ministry Of Finance)),(None,ValueParty(Bank Leumi Broker)),(None,ValueText(bid)))))
        at io.grpc.Status.asRuntimeException(Status.java:535)
        at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:533)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:557)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:69)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:738)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:717)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
24-03-2022 19:36:47.359 [http-json-ledger-api-akka.actor.default-dispatcher-6] INFO  com.daml.http.Endpoints - Responding to client with HTTP 404 Not Found, context: {instance_uuid: "1a7ab657-6859-4737-8510-b93120b8c2f5", request_id: "13b28b1c-3ea6-4504-9a74-3fe83d88f530"}
24-03-2022 19:36:50.326 [http-json-ledger-api-akka.actor.default-dispatcher-11] WARN  akka.actor.ActorSystemImpl - Illegal header: Illegal 'origin' header: Illegal origin: Invalid input '/', expected DIGIT or 'EOI' (line 1, column 22): http://localhost:7575/

Sorry we can’t be more specific, as it is hard to pinpoint the issue

2

Does the code of your choice have a fetchByKey ?

3

Those error messages look very much like the choice you’re exercising depends, at some point in its dynamic execution, on a (possibly hardcoded?) fetchByKey, and the state of your ledger has changed such that that key is no longer available.