This is a very interesting question. There are millions of instruction sets out there, but only a handful of very commonly used ones.
The first thing I'd look at is the origin and intended use. If you suspect it was designed in the US, you'd primarily be targeting processors with datasheets available in english, for instance. If it was designed in Asia, then there are a number of processors they use for mass manufactured devices that US engineers rarely see. Even Europe has a few processors that are more common than others.
I'd then take a look at code size and functionality (assuming you know what the code does to some degree). If it's a few megabytes of code, you can pretty much discount most embedded 8 bit processors and start looking at larger devices with external memory. If it is a few kilobytes or less, then you'd instead want to focus on smaller, cheaper devices. If the functionality is simple, it might even be code for a four bit processor.
At this point it's worthwhile looking at the memory structure. There's likely to be a program section and a data section at minimum. If it's a binary file (versus intel hex or motorola s record) then you have little insight as to where in memory certain chunks of data are being placed. A hex editor might show some patterns. If it does come in a hex or s record format, you might have more information about the memory structure of the processor it's meant for. Some processors reset at program memory location 0, some at the highest memory location. The program might include EEPROM initial values in a separate memory location. If it's meant for a secure processor (as used in banking) it might even have security keys for an odd memory location.
Depending on the language it was programmed in, you might have some additional clues. If it was programmed in C or a similar procedural language, then functions will almost always start with a sequence of instructions to save certain registers to the stack (lots of pushes) then right before returning lots of pops to return the original values from the stack. If you can do some pattern recognition, you'll find many of these sequences throughout, and may be able to determine which instructions are most likely push/pop instructions, return, etc, which could narrow your choices down a bit.
If it's an embedded device with interrupts, it may have an interrupt vector table, which will look like a bunch of jumps to different memory locations all in a large block, probably at a convenient location (address of 0x???0 for instance). Jump tables are used elsewhere for other things as well, but if you can locate a sequence of instructions that look identical except for what would be the address to jump to, you might be able to infer what a jump instruction looks like, and again narrow your choices down.
At that point, I'd start with the most common processor architectures and see if anything correlates. x86, arm, mips, 8051, avr, pic, powerpc, Z80, 68k, 6502, etc, etc, etc. There are lists of common processors and instruction sets - at least in the english speaking world - that might prove helpful.
I'm not aware of any automated tools that would help with this, but MAME emulates a great many processor architectures, and one possible method is to run the code through a number of processors and watch the registers to see if anything clicks according to what you know about the design.