This is something I deal with often (and I'm working on right now), and unfortunately, thus far it's a completely manual process - we have an automated process that flags every file that might be modified as part of our initial automated audit for a new support client. We then have someone diff those files, and rule out any obvious false positives (i.e., whitespace changes).
Then, the fun part - a senior member of our team who has been working with Magento for quite some time has to take a look at the results to determine if any of the modified files could be the result of a patch. We've looked at updating our system to check against all of the patches we're aware of/can get our hands on, and that might work for CE, but on EE it's even more challenging, since EE support does sometimes issue patches directly to clients that are never released any other way or cataloged in a consistent manner.
So, when we do this level of review, we rely on past experience applying these patches + common sense (i.e., is it just a change to an API's endpoint? If so, is that changed endpoint present in the updated version? If so, it was a patch and can be ignored).
It would be theoretically simple to apply all of the patches available on the CE download page, etc., to every applicable version of CE and check against those (FYI, we don't use diff for the first pass - we use hashing, in part because we built this technology into a tool that can remotely check on a site without needing to download it first). That would rule out a majority of the patches, but it still doesn't help for any CE or EE patches that aren't posted to the public download area for CE or the client/protected download area for EE. That would require Magento to make a consistent policy that ALL patches be made available to ALL customers, and get those posted to where we could get to them.
So, I don't think there's a way to 100% automate this until changes happen on the Magento side of things, unfortunately.
