Lookupbykey vs fetchbykey - why can I fetch a key but not do a lookup

Questions
1

I am following Aithorization pattern as described and in place of passing the contractid as per the example, I want to use contractKeys.

template AccreditedTenantToken
  with
    agency: Party
    tenant: Party
    landlord: Party
  where
    signatory agency
    observer landlord
    key (agency, landlord, tenant) : (Party, Party, Party)
    maintainer key._1
    controller agency can
      WithdrawAccreditedTenantToken : ()
        do
          return ()

where lanlord relies on agency to issue these tokens.
in another place landlord wants to check if these tokens exist before finalizing a contract…I am using a foldlA

          foldlA (\a b -> do
                    lookupByKey @AccreditedTenantToken (agency, finalContract.landlord, b )
                    return (a && True)
            )True finalContract.tenants

in here if I use lookupbyKey I get Authorization error which is inline with documentation but if I use fetchBykey I either can fetch the contracts if they do exist or fail which is fine and inline with documentation.

ignore the return part there…
My question is if I fetch a contract why I cannot do a lookup?

2 Likes
2 
          foldlA (\a b -> do
                    fetchByKey @AccreditedTenantToken (agency, finalContract.landlord, b )
                    return (a && True)
            )True finalContract.tenants

          -- gothrough the tenant list
          -- fetch tokens for each 
          -- if the token exist 
          res<-foldlA (\ a b -> do 
                      cid<- lookupByKey @AccreditedTenantToken (agency, finalContract.landlord, b )
                      case cid of
                        None -> return (a && False)
                        _-> return (a && True)) True finalContract.tenants
          assert (res == True )

image
image2474Ă—2104 471 KB

1 Like
3

lookupByKey and fetchByKey are authorized differently as described in the docs.

  • fetchByKey is authorized like fetch so it needs to be authorized by at least one stakeholder.
  • lookupByKey needs to be authorized by all maintainers of the contract you are trying to lookup.

The reason is that lookupByKey allows negative lookups, and validating any kind of lookup involves work. Imagine we didn’t have that restriction and both you and I were on a distributed ledger with a template

template T
  with
    sig : Party
  where
    signatory sig
    key = sig : Party
    maintainer = key

You could then hammer me with transactions containing lookupByKey @T bernhard without me ever having agreed to validate a key lookup for you.

6 Likes
4

Thanks that explains it well, may be we can add this explanation in the documentation as well.

1 Like
5

one more clarification… does this additional cost on lookup not exist on fetch or given its based on access check its much cheaper?

1 Like
6

Yes, but fetch only allows for positive lookups so unless you manipulate your participant node to be dishonest (which I can detect), I only need to do that work if I signed a contract that says I’m OK with that.

1 Like
7

@bernhard @Vivek_Srivastava - having lookupByKey behave differently to fetchByKey make it super confusing. I don’t think the benefit of avoiding spamming is worth the confusion that this causes. I’ve distilled this to a trivial example where I even make alice a signatory of the contract she is trying to lookup and fails to do so.

template Foo
  with
    admin : Party
    user : Party
    taker : Party
  where
    signatory admin, user
    key (admin, user) : (Party, Party)
    maintainer key._1

    controller taker can
      Foo_Take : ContractId Foo
        do create this with user = taker


lookupVsFetch = scenario do
  alice <- getParty "Alice"
  admin <- getParty "Admin"
  let foo = Foo with user = alice, taker = alice, ..

  -- create the foo
  fooCid <- admin `submit` create foo with user = admin

  -- alice becomes a signatory
  alice `submit` exercise fooCid Foo_Take

  -- alice can fetch it
  (fooCid, fooCdata) <- alice `submit` fetchByKey @Foo (admin, alice)
  assert $ fooCdata == foo

  -- she can even fetch if she has the contract id
  fooCdata2 <- alice `submit` fetch fooCid
  assert $ fooCdata2 == foo

  -- alice is a signatory of foo
  assert $ alice `elem` signatory foo

  -- but she can't look it up!
  -- does this make any sense?
  optFooCid <- alice `submitMustFail` lookupByKey @Foo (admin, alice)
  assert $ optFooCid == ()
1 Like
8

Alice is indeed a signatory, but she is not the maintainer so as per @bernhard 's comment, the lookupByKey @Foo (admin, alice) will fail unless you switch her with maintainer key._2 - but I can see how this might cause confusion.

1 Like
9

As much as I share your sentiment on this, I don’t know how to align the two, even ignoring the DoS angle.

One line of argument would be to say that positive and negative lookupByKey instances should have different authorization rules so that positive lookupByKey behaves like fetchByKey. That seems even worse and entirely pointless to me. lookupByKey then just falls back to fetchByKey in contexts where negative lookups are not properly authorized. The behaviour now becomes implicit from the context rather than being explicit through fetchByKey and lookupByKey.

Authorizing lookupByKey like fetchByKey is also not an option. fetchByKey needs the authority of at least one stakeholder (which then becomes the actor on the fetch node). In a negative lookupByKey, the stakeholders are unknown so this check can’t be performed.

One could then consider removing all authorization from negative lookupByKey. However, that would effectively make existence of keys public information. This is problematic as the existence of keys can convey valuable information (like two parties being in a relationship).

The only reasonable weakening I can imagine is to say we only require the authority of a single maintainer, not of all maintainers. I’m not sure that solves anything, though.

10

@bernhard (Or someone more knowledgeable than me :slight_smile:) can you please precisely define what you mean by “positive” and “negative” here? I’ve seen it in several parts of the documentation but nothing definitive.

My understanding by context is that a positive lookup is one for a contract that does exist and a negative is for one that does not. I’m asking because based on my understanding there would be some ex-post interpretation of the result that I find confusing. Ie.

Contract doesn’t exist, lookupByKey should be None, so why would be care about checking against the stakeholders?

11

Positive means lookupByKey returns Some cid during interpretation. Negative means lookupByKey returns None.

At the transaction level, the None becomes a NoSuchKey node in the transaction tree. A positive one becomes a simple Fetch with the maintainers as actors.

As a simple example say we implement a lock on a choice as a separate contract. Note that the below fails due to the authorization checks under discussion here.


template Lock
  with
    locker : Party
    obs : [Party]
  where
    signatory locker
    key locker : Party
    maintainer key

template Lockable
  with
    sig : Party
    con : Party
    locker : Party
  where
    signatory sig

    controller con can
      Guarded : ()
        do
          oLock <- lookupByKey @Lock locker
          case oLock of
            Some lock -> abort "Choice locked"
            None -> return ()

s : Scenario () = do
  [locker, sig, con] <- mapA getParty ["locker", "sig", "con"]
  
  lock <- submit locker do
    create Lock with
      locker
      obs = []
    
  locked <- submit sig do
    create Lockable with ..
  
  submitMustFail con do
    exercise locked Guarded

  submit locker do
    archive lock

  submit con do
    exercise locked Guarded

Neither sig nor con are stakeholders on the Lock, nor is locker a stakeholder on Lockable. When con submits the exercise locked Guarded, they fill in the NoSuchKey or Fetch nodes. Assuming they are malicious, they can submit the transaction with either. So unless sig can validate that con submits the right thing, the lock is worthless.

But asking locker is not an option. locker has given neither con, nor obs the authority to know about the existence of Lock.

So what if we give them that authority by adding them to the obs field (as in obs = [sig, con]? Now sig could validate the first submission attempt as they know Lock exists. But how about the second one? It may be that locker has created a new Lock with obs = [con], and con is maliciously submitting NoSuchKey. Again, the only way to be sure for sig is to ask locker, but locker hasn’t agreed to share any information about the existence of Lock.

So in short: to validate Guarded in the case where the lookup results in a NoSuchKey, locker needs to validate this. But doing so reveals information about the existence of Lock so authorization is needed. In other words: locker needs to authorize Guarded eg by delegating lookups.

template Lock
  with
    locker : Party
    obs : [Party]
  where
    signatory locker
    key locker : Party
    maintainer key

template LockDelegation
  with
    locker : Party
    con : Party
  where
    signatory locker
    key locker : Party
    maintainer key

    controller con can
      nonconsuming CheckLock : Optional (ContractId Lock)
        do
          lookupByKey @Lock locker

template Lockable
  with
    sig : Party
    con : Party
    locker : Party
  where
    signatory sig

    controller con can
      Guarded : ()
        do
          oLock <- exerciseByKey @LockDelegation locker CheckLock
          case oLock of
            Some lock -> abort "Choice locked"
            None -> return ()

s : Scenario () = do
  [locker, sig, con] <- mapA getParty ["locker", "sig", "con"]
  
  lock <- submit locker do
    create Lock with
      locker
      obs = [con]

  submit locker do
    create LockDelegation with ..
    
  locked <- submit sig do
    create Lockable with ..
  
  submitMustFail con do
    exercise locked Guarded

  submit locker do
    archive lock

  submit con do
    exercise locked Guarded

In the above, locker has agreed to validate the node resulting from lookupByKey so sig will have a guarantee that con can’t submit an incorrect result without being caught during validation.

12

I agree with @Leonid_Rozenberg that the difference between fetchByKey and lookupByKey is subtle and a source of confusion when learning DAML. Also I agree with @bernhard that the semantics and authority requirement of both updates are correct. But I do think a major source of confusion is that these functions are named and look almost the same, even their functionality is very similar - up to authorization. So here is a suggestion we might consider: We entangle the existence check from actually fetching the contract by

  • leaving fetchByKey as is
  • dropping lookupByKey and introduce doesContractExist: k -> Update Bool (or a better name)

This would make the distinction in functionality very clear and also why it needs different authority rules. We can reintroduce lookupByKey as a stdlib function:

lookupByKey k = do
  contractExists <- doesContractExist k
  if contractExists
    then 
      Some <$> fetchByKey k
    else 
      return None

Again, this makes it very clear why lookupByKey needs more authorization than fetchByKey. Also, this seems closer to what is actually written in the transaction.

13

If you think it helps, we could also implement

doesContractExist = fmap isSome lookupByKey

and favour that in the docs over lookupByKey. I don’t think there’s an important difference in performance, nor does it matter which function is native vs derived.

14

Yes, that’d be the other option. Would be interesting to know if this is more clear and intuitive to (new) users.

15

@bernhard Thank you for the detailed answer.

In the first scenario that you describe the server responds back with

Scenario execution failed on commit at Lock:52:3:
  1: lookup by key of Lock:Lock at DA.Internal.Prelude:381:26
     failed due to a missing authorization from 'locker'

which is odd since that contract should have been archived in the previous submit by locker. But I think this issue is beside your point? I will ignore it.

I am confused by

In the first clause, con is submitting an exercise and then you refer to they. Who is "they" ?

What I think that you are referring to is that when we perform an Update we need to translate that into a sequence of transactions that will be stored in the ledger (in order to maintain a consistent ledger). Each action against a key needs to be replaced with a NoSuchKey or Fetch node (or KeyLookup per this issue). Consequently, in order for both con and sig (the "they") to arrive at the same transaction the lookupByKey must resolve to the same node for both? (I think that I understood this part after I figured out the rest, but I want to clear it up for anyone else trying to follow along).

I think that you forgot to add observer = obs for Lock. Otherwise when I do set obs = [sig, con] the first submit by con fails with

Scenario execution failed on commit at Lock:46:3:
  Attempt to fetch, lookup or exercise a key associated with a contract not visible to the committer.
  Contract:  #0:0 (Lock:Lock)
  Key:  'locker'
  Committer: 'con'
  Stakeholders: 'locker'

Weirdly, after I do set the observer field on Lock and modify the abort to a benign debug I get the previous error (failed due to a missing authorization from 'locker') but this time on con’s first submit,

Is this in the scenario where con and sig are observers on Lock ? If both of those parties are observers I fail to see why we wouldn’t want the lock to function? Or more precisely, why we wouldn’t want the semantics of DAML to allow the lock to work.

Again, thank you for helping to elucidate this material.

16

Authorization checks happen before contracts are checked for being active to prevent privacy leaks. The whole point is that con is not entitled to learn whether the Lock is active or not.

I’m using they as the gender neutral pronoun. It refers to con.

I did and I didn’t. If you add them as observers, you could reasonably argue that sig could validate the positive lookup which is indeed true. If you switched to fetchByKey and added the observers, the first Guarded would work. But the negative lookup is still problematic. I went back and fourth over exactly how to structure the “hypothetical” first version. There’s really no point looking at the scenario error messages for it as the current implementation doesn’t allow it for a whole range of reasons.

The first version is a hypothetical one. It’s explicitly prevented by the current authorization rules, which is what that error is telling you.

As I said above, if both are observers, you could argue that the positive lookup should work. But in the negative case there are no observers so you can’t make it work. Having rules that allow for positive, but not negative lookups is pointless. You may as well use fetchByKey in that case. This is what I was saying further up in the thread:

17

I find this sentence and the one that follows (quoted later) confusing,

This might be the crux of why I don’t understand your explanation. If

and we’re in the negative case then what do the observers have to do with the return of Guarded? From the docs of lookupByKey I can reason that the Lock contract does not exist. If you meant to write that we’re in the negative case because there are no observers (stakeholders on the Lock to authorize), then why can’t we interpret that as “working”? Maybe our interpretation of “work” is different.

Don’t we want the authorization rules to determine whether lookupByKey is positive or negative, but in many of your examples, you assume a resulting state and talk about different authorization rules for those cases.

Isn’t that backwards? Or what do the authorization rules do after we’ve executed a lookupByKey and have a result?

Let me try a different line of inquiry. When lookupByKey was added to DAML, why did it need to have different authorization from fetchByKey ?

18

I’m not sure I understand all (or any) of this, but let me try. This is all speculation, but it sort of makes sense to me (now).

For both, you have three cases to consider:

  1. There is an active contract and you are allowed to see it.
  2. There is no active contract.
  3. There is an active contract and you are not allowed to see it.

At first glance, if you only think at the DAML level, there is no reason for them to be different. In the happy case (1) they return essentially the same thing; in the unhappy cases (2, 3), fetchByKey fails the transaction, whereas lookupByKey returns None. Both functions give you no way to know why things failed, which is great.

But DAML is not just a language, it’s also an architecture for distributed applications. From that perspective, every DAML choice is evaluated first in a “local” context (local to the submitting party) and then sent for confirmation to a “more global” context (validation by all the stakeholders; note that this is not (necessarily) a single global context).

Say Alice wants to exercise a choice, and Bob is a stakeholder on the contract. Bob will need to validate the transaction that Alice submits. The transaction says, simplifying a lot: “I have executed this choice on this contract, and the result is the following list (tree) of fetches, archives and creates”. Given the choice name and the contract, Bob has the exact code Alice should have run.

Let’s assume Alice is dishonest (because Bob has to).

How does authorization work for fetchByKey? Locally, Alice needs to create a transaction tree that contains the corresponding contract ID (in a Fetch node). fetchByKey explicitly only works if Alice is a stakeholder on the contract, so conceptually she (locally) scans all the contracts she knows about to find one with the correct type and key, and which is currently active.

If we’re in case 1, where the contract exists and Alice can see it, Alice knows about it because she is a stakeholder: if the contract had been archived, she would have been notified. There is of course a chance that contention would result in the transaction being rejected, but there is no opportunity for Alice to cheat here: either she can produce a transaction with the correct Fetch node, or she can’t. As a validator, you can ask Alice to prove that the given contract ID exists.

For cases 2 and 3, Alice does not have a contract ID, so she cannot produce a valid transaction to send. Further, fetchByKey is defined as only succeeding if the submitting party is a stakeholder on the corresponding contract, meaning that it would be trivial for any validating party to reject a fraudulent transaction Alice would try to create, either using obsolete IDs or invented ones.

Now, let’s think through how we should authorize lookupByKey. At first glance, the happy case stays the same: we validate it just like fetchByKey. If Alice is a stakeholder on the matching contract, she knows about it and can include the corresponding Fetch node.

For cases 2 and 3, however, things are less clear. Because lookupByKey does not fail the transaction when the contract cannot be found, we need to record, as part of the transaction, that the contract could not be found, so that other parties trying to validate the transaction can verify they get the same result.

So you’re Bob, and Alice sends you a transaction containing a NoSuchKey block. How do you validate that? As per the rules of fetchByKey, the only constraint here is that Alice be a stakeholder on the contract. There’s nothing about Bob, so it’s entirely possible that the contract exists and you don’t know about it (and Alice is lying to you). Now, based on the key (which you have), you know who the maintainers are; let’s say it’s Carol. You could ask her. But she may have nothing to do with the transaction you’re looking at. Not only do you not have authorization to pester Carol (possible DoS attack), you also should really not be telling Carol about this transaction you’re validating (possible information leak: Carol learning of the transaction), and she may think you really have no business knowing about the existence of that contract anyway (other possible information leak: what if the contract is between Carol and Alice, and Bob has nothing to do with it?).

So essentially, if we tried to keep the same permissions for lookupByKey as for fetchByKey, there would be no way to validate transactions in the negative case.

This is why fetchByKey and lookupByKey need different permissions for the negative case. From this point, we have a few options:

  • Make all keys public at all time. This is tantamount to maintaining a global lookup map for the entire ledger, so it’s not great from a performance perspective in a distributed system (or making everyone queryable at all times, which is also not great from a DoS perspective). It’s also not a great option for privacy, as keys are likely to contain privileged information along the lines of who is dealing with whom.
  • Have different permission requirements for positive lookupByKey (same as fetchByKey) and negative lookupByKey (as we evidently need more to validate those). This seems like a very messy model, as you need to know what the result of executing an action is before you know what permissions you need to run it.
  • Change the permissions for the positive lookupByKey to match what we need for the negative lookupByKey. This means lookupByKey is more restrictive than fetchByKey even though they superficially look like they do the same thing (especially in the positive case), but that seems better than the other two options.

So then, assuming you agree with our choice of the third option, what should the permissions for lookupByKey be? In other words, what does the validator need to be able to validate a negative lookup? Well, they need to be able to check if the contract exists, and for that they have to be able to talk about that key with at least one of the maintainers. The only way to resolve both the possible information leaks and the unwanted DoS attack is for the maintainer to be part of the transaction.

I’m not sure why “all the maintainers” rather than just any one, but I suppose if we assume Alice is dishonest we may not want to let her choose which maintainer is going to validate the transaction, especially if that could be herself.

3 Likes
19

Even ignoring validation and dishonest participants, a model where lookupByKey has the same authorization rules fetchByKey currently has doesn’t make sense conceptually:

  • fetchByKey needs to be authorized by a stakeholder.
  • To calculate stakeholders, you need the contract. There is no way to determine this from the key alone.
  • If there is no contract with a given key, there is no contract. Therefore you cannot calculate the stakeholders.
  • If you don’t know what the stakeholders are, you cannot possibly check if you have authorization from them.

Since the maintainers have to be inferable from the key, the authorization rules we currently have don’t have this issue: We can calculate maintainers just from the key alone without needing the contract.

This doesn’t answer why we have the exact authorization rules we have and others in this thread have done a much better job at answering this than I can. But it does provide a very simple answer as to why giving lookupByKey the same authorization rules as fetchByKey is not possible and the problem is more complex than that.

2 Likes
20

You make it sound like the issue really is with the permissions of fetchByKey.

I suppose now there is “backwards compatibility”, but ignoring that one, is there a fundamental reason why fetchByKey does not have the same requirements as lookupBuyKey? It seems like “all the maintainers” as lexical scope on the call to lookupByKey makes more sense than the dynamic scope of “the current submitter” used in fetchByKey.

Also am I reading it right that if I am not a stakeholder on the contract I’m trying to get, but I do have the required permissions in the current scope, fetchByKey would fail whereas lookupByKey would actually return me a contract ID? I.e. lookupByKey is not “more” restrictive, it’s just different restrictive?