To avoid the case where someone modifies a cached item, there should be some way to either sign or encrypt (or both) cache data, and report when a cached item has been knowingly tampered with. Without this, it's possible to modify a cached item and insert, for example, an SSH key into a user record or fabricate an entire user.
It may be necessary to apply this broadly to memory caches as well as disk and memcache caches, but by the time someone is tampering with the memory cache, it's likely that they also have all the components needed to either fabricate realistic cache items or to simply point a ghaccountsd instance to a different token, organization, and team. So, memory caches aren't my main concern right now.