One common source for this issue is a transaction where interpretation or validation takes a long time. This can happen even if you only have a send a single command and if it’s the first command caching won’t help either.
But there are some ways to make that work, take a look at the end of https://docs.daml.com/1.4.0-snapshot.20200729.4851.0.224ab362/concepts/time.html#assigning-ledger-time and Semantics of min_ledger_time for more information on this.