There are also several kinds of “group” or “role” patterns discussed in this thread. The linked post seems most relevant here.
The issue I point out there also holds for what you are trying to achieve here: As your group membership changes, you need to update read permissions (ie observers) on all contracts that should be visible to that group. That becomes a performance bottleneck if the relationship between group and contract is many-to-many.
Openwork Board shows this pattern in action, with read-write permissions restricted to groups of users. It uses a single contract per board to store all access levels, but it could be disentangled into access tokens of the style @Andrae suggests. However, DAML does not have transactional ranged queries so if you need access to the full list of users of a group, you do need to keep that list somewhere.