I don’t think the authorization of fetchByKey is wrong. It is exactly the same as fetch which I find very sensible. It is really lookupByKey that needs more restrictions since you have to differentiate authorization failures from negative key lookups. Note that the somewhat dynamic nature (not sure scope is the right term here) of authorization checks is unavoidable in DAML. You always need to first fetch the contract whenever you need signatories, observers or actors of a choice.
As for your question about lookupByKey, it really is more restrictive than fetchByKey:
Maintainers need to be signatories so if you have authorization from all maintainers, you definitely have authorization from a stakeholder.
That leaves you with the restriction that the submitter must be a stakeholder. This applies to both lookupByKey and fetchByKey (but admittedly the docs for lookupByKey could point this out more clearly).
What I meant was that I was expecting this would work:
template Secret
with
owner: Party
name: Text
secret: Text
where
signatory owner
key (owner, name): (Party, Text)
maintainer key._1
template AllowFetch
with
assetOwner: Party
obs: Party
where
signatory assetOwner
observer obs
controller obs can
nonconsuming FetchAsset: Secret
with
id: ContractId Secret
do
fetch @Secret id
testSecret = scenario do
alice <- getParty "Alice"
bob <- getParty "Bob"
secret <- submit alice do
create Secret with owner = alice, name = "label", secret = "my secret"
allow <- submit alice do
create AllowFetch with assetOwner = alice, obs = bob
submit alice do
fetch secret
submitMustFail bob do -- this is the one I expected to succeed
exercise allow FetchAsset with id = secret
return ()
because at the point of fetching secret, I do have the explicit authorization of alice. I suppose I need to adjust my mental model, but I will note that looking for information on fetch through the search function on the docs site is surprisingly unhelpful.
@Gary_Verhaegen Thank you for your reply. I follow your argument up to the paragraph starting with
if you’ll bear with me, hopefully you can explain where I’m confused.
One way to clear up this constraint, is for a NoSuchKey block to require that Alice delegate to Bob the ability to validate it. I’m purposefully using “delegate” to confer the same rights as what we would mean in a delegation contract. This doesn’t necessarily have to be explicit, but implicit in the semantics of DAML. If Alice is malicious this would mean that Bob would learn that. And if she is honest, she wants her transaction validated, so whatever reason led her to using lookupByKey as opposed to fetchByKey should merit that right.
Tangentially, I think that this implicit stakeholders right-of-validation is a desired feature of DAML. I assumed that it was everywhere in the language, so confusion around this is what has been driving my questions.
Here you raise many fine points. I think some are overstated (DoS) while others don’t really make sense from the perspective of Alice wanting her transaction validated, Bob clearly has something to do with the transaction if he’s validating it, so Carol should acknowledge that as a consequence of granting Alice authorization on the keyed contract in the first place. But at root, what I find confusing, is this piece of documentation that you recently committed (the paragraph just above here):
For the negative case, however, the transaction submitted for execution cannot say which contract it has not found (as, by definition, it has not found it, and it may not even exist). Still, validators have to be able to reproduce the result of not finding the contract, and therefore they need to be able to look for it, which means having the authorization to ask the maintainers about it.
This seems to contradict your points with respect to Carol. I will accept the more recent documentation as authoritative.
You’re right that it doesn’t need to be explicit in principle, but it looks like whoever designed lookupByKeydid make the choice of making this explicit. This is exactly what is currently happening: in order for the transaction to be valid, the lookupByKey call must have the authority to query the maintainers, i.e. the submitter is delegating to the validators the right to query the maintainers.
You may think implicit would have been better. I would disagree: Alice being malicious is not the only problematic case. Let’s say we want to make the choice that lookupByKey is special, and implicitly confers validators the right to check for key existence with the same rights as the submitter. What if the contract does exist but it is not visible to the submitter? We still want to refuse the transaction, but if the transaction is between Bob and Alice and the maintainer is Carol, nobody can tell either Alice or Bob that the contract does, in fact, exist. So we take the conservative approach of saying that, if the lookupByKey call does not explicitly have all the rights it would need to make sure that the contract does, in fact, not exist, then the transaction is invalid.
Would you mind clarifying where the contradiction lies? I’m not aware of any, but my understanding of this issue did evolve quite a bit over the past week or so so it’s definitely possible I don’t fully agree with my past self anymore.
@Gary_Verhaegen This is really great, I’m starting to understand much more technically about the argument and that our different interpretations stem from a difference of what I think lookupByKey should mean.
Now I understand Bernhard’s point about the LockDelegation. This makes explicit the differences between what we want; I’m arguing for an implicit delegation whereas y’all are for an explicit one.
My response in this case is why would we want to refuse it? If it is not visible to both Alice and Bob, how does its existence change the business logic that they would want to write in a contract between them? They’re told that “these are not the drones you seek” and continue doing what they’re doing.
W.r.t the contradiction, I think that you are using
as an argument against for why validating a non existence lookup is difficult, but then you document that you do have to pester Carol. I think that I am being dense here as you may be using different contexts.
After reading through some of the back issues that Moritz linked to I realized that lookupByKey was a stand alone idea, and not a wrapper for a potentially failing fetchByKey. So I should stop shoehorning it into that frame. I find the DoS arguments confusing, because couldn’t one fetchByKey out-of-band to perform this attack too, or to check for the existence of keys?
I still find the current abstraction awkward. I think that having implicit delegation for validation so that the authorization for the two functions is the same is a worthwhile trade-off, but I think that I’m in the minority on this opinion so I’ll stop.
To clarify: I am not trying to argue for anything here, and I am not presenting what I would want, I’m merely trying to explain how I think things currently work and what a reasoning for them working the way they do could be. In the following (and most of the above), when I say “we”, I mean the company / daml language design team; I was not involved in any of this until last week.
For better or worse, we have defined contract keys as globally unique over the entire ledger. Ideally, even when that ledger is a world-spanning, distributed, aggregated network of DAML ledgers. You may disagree with that, but I’m afraid that ship has sailed.
So that’s constraint 1. Constraint 2 is that we also do not want keys to be discoverable without the proper authorization, because the mere existence of a key could leak information about the corresponding contract. So we have made the decision that the existence of a key is a private thing that requires authorization to discover.
Alice may know that the contract does indeed exist; similarly, Alice may not know, but perhaps Bob does and Bob is the dishonest one. Either of them could learn of that contract afterwards. The ledger as a whole is not consistent with constraint 1 if we do not block that transaction, and being able to block that transaction without violating constraint 2 gets us back to needing the authorization of the maintainers.
Note that, as those things are currently defined, the maintainers of the contract could be observers, meaning you could have a transaction between Alice and Bob that checks for the existence of a contract (which they can’t see), and then, if it does not exist, attempts to create it (with Carol as observer/maintainer). What then?
I’m not claiming the current design is great UX, but every time I try to think of a different one, I hit a wall like that.
If Carol is the maintainer, asking her is the only way to know. So, yes, we do have to ask her, and therefore we need her authority before we can attempt the lookupByKey call.
If we were to allow lookupByKey without Carol’s authorization, we’d be stuck, because we would have to ask her and we couldn’t. Call this “Constraint 3”: we want DAML to be able to run truly distributed ledgers where different nodes are managed by different organizations, and you cannot make requests against a node without that node having given you consent first, through the mechanism of DAML signatories. Having her authority does not mean we don’t have to ask her; we still have to, but at least now we can (under constraint 3).
The way it is currently defined, fetchByKey actually resolves the key to a contract ID locally (this is possible because the submitting party must be a stakeholder), so it does not query anyone and cannot be used for DoS or discovery.
I agree with that sentiment, though I do hope that the switch of emphasis to visibleByKey helps a little bit with that.
I hope I’ve been able to at least convince you that this would not be compatible with three constraints we have set for ourselves. Whether those constraints are worth their cost is a separate discussion, I suppose, but not one I am well equipped to have.
which contract are you referring to here? If you mean the keyed one are you sure? The docs say that maintainers must be signatories. Or are you referring to maintainers being observers of the contract between Alice and Bob? In which case they’re not under obligation to help Alice and Bob.
“it” is the keyed contract? But then Alice and Bob couldn’t create it without Carol’s authorization. If they had Carol’s authorization for the query, they would be able to see the original contract.
I’m sorry to disappoint you but you have haven’t. I support your effort but I’d like to see a more full fledged example of the negative consequences of what I’m suggesting.
On a previous project I know that I ran into the fetchByKey vs lookupByKey distinction, but because of time pressure, I worked around it. I will try to resurrect that work into a meaningful example.
I have to say that I’m completely lost as to where the confusion is at this point. Let me try a completely fresh explanation.
There are two places where fetches and lookups can succeed or fail:
During interpretation, the Participant node of the requester (aka submitting party) looks inside their index database to see whether the requester knows of a contract with that key or not. Since we have to assume the Participant is malicious, any fetch or lookup can return just about anything in theory.
In practice, honest Participants currently succeed on fetch if the requestor is a witness on a contract, and succeeds on fetchByKey and lookupByKey if the requestor is a stakeholder in a contract with the given key.
Why the distinction, you ask? Because by witnessing creates, but not archives, you could be a witness on multiple contracts with the same key, and the events creating those keys can’t necessarily be ordered so there is no good way to decide what a key lookup should do if multiple key instances are witnessed.
During validation the result of that key or contractId resolution at interpretation time is validated by the signatories or maintainers. To see why authorization matters, let’s say there’s a template
template WeMeetTonightAt7UnderTheLimeTreeOnCentralPlaza
with
sigs : [Party]
obs : [Party]
where
signatory sigs
observer obs
key sigs : [Party]
maintainer key
The mere existence of a contract instance confers information, so I, bernhard, am interested to know whether you, leonid and gary have signed such a contract instance. I have a severe case of fomo.
Now let’s say I send out a transaction that is a single NoSuchKey node for the key [leonid, gary] on that template. If the two of you respond with “CONTENTION: Actually, there is such a key”, or “Yep, fine, go ahead and commit”, I got what I was looking for. I know whether to crash your party or not.
So you need an authorization rule that allows you to say “Na’uh! You are not invited, we are not telling you”.
The only information avaialble on the NoSuchKey node are the key and template. The only party-related information you can pull out of those is the set of maintainers. So the authorization rule cannot be based on stakeholders. You don’t know who the stakeholders of a non-existing contract are.
Ie The authorization rule cannot be the same as for fetch or fetchByKey.
Suppose we just need one maintainer. A lookupByKey with key [bernhard, leonid, gary] would now be well-authorized, as I submitted it. I could now submit an endless stream of negative lookups of that key. All of them are correct, all of them are well authorized, and each time the two of you have to go to your index database and check even though you have never agreed to anything that may possibly, maybe have led to an agreement to meet me.
Although I still have small questions about specific examples that are unanswered, I am not confused about why things are. But I am confused about why things cannot be different. For example,
What if [leonid, gary] respond with “No” (None) in the cases where bernhard is not authorized and when no such contract exists? What will bernhard do then? bernhard can’t force [leonid, gary] to be maintainers on this contract. Maybe bernhard could use that information to ask to meetup with [martin, moritz] instead?
Is that the only way that things can be? Could the NoSuchKey also contain the party who makes this claim (bernhard), so that to validate this transaction we have to take into account who is making this claim. To leonid this transaction makes sense (I don’t want you to know about our get together) and it also makes sense to martin who has the same information as bernhard.
Would this situation also be fixed if we stipulate any maintainer as long as it isn’t the one in the new NoSuchKey that I propose ?
If you response depends on the existence of the contract, I have learnt the information I was looking for. The response leonid and gary give must not reveal whether the key exists or not.
That should also answer your second question. The premise of my example is that bernhard must not be able to learn whether leonid and gary are meeting up or not. Don’t get hung up on the whole meeting thing, it’s just an example. The important thing is that bernhard is not entitled to find out whether gary and leonid have an active contract with a given key.
That would let the transaction submitter leak into the Ledger Model beyond top-level authorization, which harms composability. We try to guarantee that if bernhard has the right to perform an action a, bernhard can delegate that action to another party. If you let the submitter/requester leak into the auth model, that’s no longer the case.
But I also don’t fully understand what you have in mind. If you are saying the transaction NoSuchKey should just be accepted because bernhard doesn’t have the right to do the lookup, I’d infer that lookupByKey falls back to const None in any ill-authorized context, which is probably not what you had in mind.
I don’t understand this question. The key always contains all maintainers so the NoSuchKey node contains all maintainers.
In that case I don’t, but if you mean to say that you’d like lookupByKey to return None in all cases where it’s ill-authorized, I don’t think that’s a good idea. It’s better for things to fail than to behave in unexpected ways without warning.
How is that unexpected? The specification of DAML gets to define what the caller of lookupByKey should expect. I think being able to fetch something that I can’t lookup is unexpected.
Could you make precise what your proposal is? I may be misunderstanding. Under which circumstances should lookupByKey return what value during interpretation (evaluation of the command on the submitting node), under which circumstances should the resulting transaction be accepted, and what should the error message be if it doesn’t get accepted?
Current Behaviour:
Interpretation
Fails with authorization error if not authorized by all maintainers
Returns None if there is no matching contract the submitting party is a stakeholder on
Returns a Contract ID if there is a matching contract the submitting party is a stakeholder on
Validation
If returned None during Interpretateion
Fails with authorization error if not authorized by all maintainers
Fails with contention if the key does actually exist
Succeeds if there is (still) no contract with a matching key
If returned a ContractId during interpretation
Fails with authorization error if not authorized by all maintainers
Fails with contention if the ContractId doesn’t exist (anymore), or the referenced contract doesn’t match
Succeeds if the ContractId is still active and the key matches
Thank you for considering, and prompting me to write down this proposal. It isn’t necessarily something that I had clearly in my mind when I started asking questions; I was (still am) curious about the choices and trade-offs that have been made to arrive at the current abstraction. What do you think of:
Interpretation
Does contract of the requested type and given key exist ?
No -> None
Yes -> Is the submitter authorized by at least one stakeholder ?
No -> None
Yes -> Some (ContractId _)
Validation
If None
Succeeds
If Some (ContractId _)
Fails with authorization error if not authorized by at least one stakeholder
Fails with contention if the ContractId doesn’t exist (anymore), or the referenced contract doesn’t match
Succeeds if the ContractId is still active and the key matches
As I promised Gary, here is an example of where I want the power of lookupByKey to match fetchByKey:
template OpeningBid
with
bidder : Party
seller : Party
bid : Decimal
kId : Int
b_workers : [Party]
where
signatory bidder
observer seller, b_workers
key (bidder, kId) : (Party, Int)
maintainer key._1
controller seller can
Accept : ContractId Negotiation
with
offer : Decimal
s_workers : [Party]
do create Negotiation with ..
template Negotiation
with
bidder : Party
seller : Party
bid : Decimal
offer : Decimal
kId : Int
b_workers : [Party]
s_workers : [Party]
where
signatory bidder, seller
observer b_workers, s_workers
key (bidder, seller, kId) : (Party, Party, Int)
maintainer key._1
choice Agree : ()
with
who : Party
controller who
do
assert $ bid > offer
return ()
controller bidder can
MakeBid : ContractId Negotiation
with
newBid : Decimal
do
assert $ newBid > bid
create this with bid = newBid
controller seller can
MakeOffer : ContractId Negotiation
with
newOffer : Decimal
do
assert $ newOffer < offer
create this with offer = newOffer
template BidDelegate
with
bidder : Party
worker : Party
where
signatory bidder
controller worker can
nonconsuming WorkBidL : ContractId Negotiation
with
kId : Int
seller : Party
newBid : Decimal
do
nOpt <- lookupByKey @Negotiation (bidder, seller, kId)
case nOpt of
None -> abort "missing"
Some fId -> exercise fId MakeBid with ..
template OfferDelegate
with
seller : Party
worker : Party
where
signatory seller
controller worker can
nonconsuming WorkOfferL : ContractId Negotiation
with
kId : Int
bidder : Party
newOffer : Decimal
do
nOpt <- lookupByKey @Negotiation (bidder, seller, kId)
case nOpt of
None -> abort "missing"
Some fId -> exercise fId MakeOffer with ..
t1 = scenario do
[b, s, b1, s1] <- mapA getParty ["b", "s", "b1", "s1"]
ob <- b `submit` do
create OpeningBid with
bidder = b
seller = s
bid = 10.0
kId = 1
b_workers = [b1]
n <- s `submit` do
exercise ob Accept with
offer = 20.0
s_workers = [s1]
bd <- b `submit` do
create BidDelegate with
bidder = b
worker = b1
n <- b1 `submit` do
exercise bd WorkBidL with
kId = 1
seller = s
newBid = 15.0
sd <- s `submit` do
create OfferDelegate with
seller = s
worker = s1
-- I want this last submit to work.
n <- s1 `submit` do
exercise sd WorkOfferL with
kId = 1
bidder = b
newOffer = 14.0
pure ()
In this example, we describe a simple 2 sided market/negotiation, but with delegation. A couple of things combine to create the failure:
Delegation. This is fundamental to this example, but not necessarily to the original difference in authorizations.
Using lookupByKey as opposed to fetchByKey/ exerciseByKey. One can write the WorkOffer methods using the other Update's, and the scenario will succeed. But what I really want to write is this method:
nonconsuming WorkBid2L : Either Text (ContractId Negotiation)
with
kid_fst_choice : Int
kid_snd_choice : Int
seller : Party
by : Decimal
do
nOpt1 <- lookupByKey @Negotiation (bidder, seller, kid_fst_choice)
case nOpt1 of
Some fId1 -> do
f1 <- fetch fId1
Right <$> exercise fId1 MakeBid with newBid = f1.bid * by
None -> do
nOpt2 <- lookupByKey @Negotiation (bidder, seller, kid_snd_choice)
case nOpt2 of
Some fId2 -> do
f2 <- fetch fId2
Right <$> exercise fId2 MakeBid with newBid = f2.bid * by
None -> do return $ Left "Nothing"
I want to be able to write choices that have non-aborting functionality. This is pretty fundamental in finance where a lot of your exposure is hedged (though this is not that kind of example ) and making more than one transaction (trade) in a transaction (atomic DB event) is key (pun intended).
For fun consider changing the key maintainers on Negotiation to maintainer [key._1, key._2].
You don’t have this knowledge available. Data in DAML Ledgers is distributed. You don’t know whether a key exists, you only know whether you have local knowledge of it. The only people that are guarantee to have knowledge of key existence are its maintainers.
But let’s assume you keep this the same as current fetchByKey:
In this case, transaction submitters have a choice. Wherever there is a lookupByKey, I could choose to insert a None. This changes the nature of the function fundamentally. Have another look at my Lock Example above. It would no longer work.
Un- or partially checked queries like you propose are something that’s been discussed before, but there is currently no such thing in DAML, and that’s not what lookupByKey is. We currently have the following principle:
Given a an action (ie create, exercise, etc), and a ledger state, there is at most one valid transaction resulting from that action.
This would no longer hold under your proposal because no matter the state of a key, you can always submit None. Ie the None case is unchecked.
worker here is an observer (ie stakeholder) of the negotiations as part of b_workers or s_workers. Ie they know which negotiations exist. Why would they now pass in a kid_fst_choice that doesn’t exist? Ie in what case would you expect this to fall through to kid_snd_choice?
If you are trying to circumvent contention, that won’t work. Ie if you are saying “take the first choice if it’s still available, and otherwise go for my second choice”.
Keys are resolved on the submitters participant node during interpretation. Contention happens during validation. For example:
s1 looks what negotiations are active and sees two: kid_fst_choice and kid_snd_choice. They call WorkOfferL with those.
s1’s Participant interprets the command. Since kid_fst_choice was just queried moments before, it’s probably still there. The first lookupByKey resolves to a contract Id.
Another worker elsewhere does the same simultaneously. Their offer/bid,accept gets there first.
s1’s transaction will now fail due to contention.
The above is independent of authorization rules or whether we always accept None. Contract keys do not remove contention.
If you want to make this bidding model work, you need to remove contention from the Negotiation contract by collecting bids and offers in side contracts. The below version is contention free on a single negotiation. The only contention is between making bids/offers and a negotiation being agreed.
module Market where
import DA.Optional
import DA.Foldable
data NegotiationId = NegotiationId with
bidder : Party
seller : Party
n : Int
deriving (Eq, Show)
template OpeningBid
with
nId : NegotiationId
bid : Decimal
b_workers : [Party]
where
signatory nId.bidder
observer nId.seller, b_workers
key nId : NegotiationId
maintainer key.bidder
controller nId.seller can
Accept : ContractId Negotiation
with
offer : Decimal
s_workers : [Party]
do
create Bid with worker = nId.bidder; bid = Some bid; ..
forA b_workers (\w -> create Bid with worker = w; bid = None; ..)
create Offer with worker = nId.seller; offer = Some offer; ..
forA s_workers (\w -> create Offer with worker = w; offer = None; ..)
create Negotiation with ..
template Bid
with
bid : Optional Decimal
nId : NegotiationId
b_workers : [Party]
s_workers : [Party]
worker : Party
where
signatory nId.bidder
observer nId.seller, b_workers, s_workers
key (nId, worker) : (NegotiationId, Party)
maintainer key._1.bidder
controller worker can
ChangeBid : ContractId Bid
with newBid : Decimal
do create this with bid = Some newBid
template Offer
with
offer : Optional Decimal
nId : NegotiationId
b_workers : [Party]
s_workers : [Party]
worker : Party
where
signatory nId.seller
observer nId.bidder, b_workers, s_workers
key (nId, worker) : (NegotiationId, Party)
maintainer key._1.seller
controller worker can
ChangeOffer : ContractId Offer
with newOffer : Decimal
do create this with offer = Some newOffer
template Negotiation
with
nId : NegotiationId
b_workers : [Party]
s_workers : [Party]
where
signatory nId.bidder, nId.seller
observer b_workers, s_workers
key nId : NegotiationId
maintainer [key.bidder, key.seller]
let
getBid = do
let keys = map (\w -> (nId, w)) (nId.bidder::b_workers)
bids <- mapA (fetchByKey @Bid) keys
return (maximum (mapOptional (\(_, b) -> b.bid) bids))
getOffer = do
let keys = map (\w -> (nId, w)) (nId.seller::s_workers)
offers <- mapA (fetchByKey @Offer) keys
return (minimum (mapOptional (\(_, o) -> o.offer) offers))
choice Agree : ()
with
who : Party
controller who
do
bid <- getBid
offer <- getOffer
assert $ bid >= offer
forA (nId.bidder::b_workers) (\w -> exerciseByKey @Bid (nId, w) Archive)
forA (nId.seller::s_workers) (\w -> exerciseByKey @Offer (nId, w) Archive)
return ()
t1 = scenario do
[b, s, b1, s1, b2] <- mapA getParty ["b", "s", "b1", "s1", "b2"]
let nId = NegotiationId with
bidder = b
seller = s
n = 1
b `submit` do
create OpeningBid with
bid = 10.0
b_workers = [b1, b2]
nId
s `submit` do
exerciseByKey @OpeningBid nId Accept with
offer = 20.0
s_workers = [s1]
b1 `submit` do
exerciseByKey @Bid (nId, b1) ChangeBid with newBid = 13.0
s1 `submit` do
exerciseByKey @Offer (nId, s1) ChangeOffer with newOffer = 14.0
submitMustFail b do
exerciseByKey @Negotiation nId Agree with who = b
b2 `submit` do
exerciseByKey @Bid (nId, b2) ChangeBid with newBid = 14.0
submit b do
exerciseByKey @Negotiation nId Agree with who = b
pure ()
@Leonid_Rozenberg I’m not sure I fully understand what you’re getting at, so let me try to rephrase it.
I believe the disagreement/misunderstanding here is around the following two ideas: the meaning of NoSuchKey and how to interact with shared state. There are roughly two ways of (safely) interacting with shared state: optimistic updates (with a risk of rejection if state has changed in-between read and (attempted) write) and critical sections (with no risk of rejection but some need for queueing). Similarly, there are two possible meanings for NoSuchKey: no contract with that key exists globally, or no contract with that key was visible to the submitter at submission time.
It seems to me you are asking why we chose optimistic updates with globally-meaningful NoSuchKey nodes rather than critical sections with observer-dependent NoSuchKey nodes. Am I getting this right?
) and making more than one transaction (trade) in a transaction (atomic DB event) is key (pun intended).