How would you play test an RPG before all of its rules are complete?

Playtesting should be similar to software testing (or any "product" testing).

1. Come up with the rules. Bang on it until you are reasonably sure you have the right idea.
2. Alpha test. Give it to someone else/unleash on players. Write down anything that goes wrong. Since all the players know they are playtesting something, in the case of your combat where no one could hit, I would personally tweak (if possible) on the fly. Otherwise, I would simply say, "OK, this part needs work, let's skip ahead. By hook or by crook you guys killed/subdued/whatever the bad guys, what now?" If someone hits you with a blind-side idea, by all means be open about it. Tell them that you hadn't thought around that contingency. Assuming you don't have munchkin testers they will likely have an idea on how the system should work. Ask them for their input. Most importantly, write down every question, issue, problem, gripe, "holy COW that's cool!", etc. Between sessions/during breaks in the test session, address any issues that come up. I'd categorize the issues as "darn dice" (nothing you can do), "darn rules" (the system is either difficult to understand and needs clarification, or is going to be something grumbled about. See D&D/Pathfinder's rules on grappling), and "darn GM" (you missed a rule or two). In fact, I probably wouldn't even bother writing a traditional session here. I would come in with a few combats, maybe a roleplaying scene or two, a few non-combat skill scenes, and finally a scene that really focuses on the classes you are testing (only testing fighters, don't worry about the magic system; only wizards, who cares about the stabby-part of combat). I would even go so far as to say that the alpha test shouldn't be "fun" unless you like disjointed and scatter-brained stuff.
3. Beta test. Here's the real test. Give the system to someone else and either get their feedback, or go to the session as a PLAYER. Work out with the new GM that you don't want him asking questions about the game system, sit back and jot down any rule questions that come up. Handle as already mentioned in the alpha test section. This should be the time when you stop really digging into how the players think the system should work and have a good idea yourself, even if you end up having to tweak certain rules. Continuing here, you should have a simple 1 session adventure here. Try to hit all of the points, but make sure that a long combat does not suck all of the fun out of the roleplaying scene before/after. Here I would have a simpler session that gives plenty of opportunities for both action and roleplay/combat avoidance. Don't go for anything deep and meaningful here, but a coherent plot would help most people to have fun here.
4. Release Candidate. Give the system to someone else and hopefully you don't get any significant issues to address. In any game I develop, this would be the first version that is printed out and not on a word document (assuming a traditional print release method). Likely this someone else would either have a starting adventure like some games do, or would make up a full adventure on their own.
5. To publish or not to publish?
6. Profit?

My initial answer is to use a depth-first testing mechanic. In effect, take a given aspect as far as you can before you rollback and try something else. For me, creating the system is a modular deal. Sure, there's a core mechanic, but I test each new bit on its own, then against what I already have in effect.

As far as the player ideas you don't expect, you have the GM's Perogative on your side. When a player suggests something you haven't anticipated, that's wonderful. That means you say to them "let's try Method A this time and see how it works. I might change it depending on what happens".

My favorite stress test is the Min/Max test. If I can optimize something and it's broken and/or the deficiencies don't seem to come up then it's a problem and needs to be addressed.

Some of the issues you mention could have been worked out without a playtest, like the odds of hitting: roll up some sample characters, then calculate what they'd have to roll to hit various targets, then look at what their target number to be hit is. Low-fidelity "combat" testing.

I was going to post that as a comment, but then it got me thinking. Maybe that's the real solution: run statistical analysis on as many things as you can before you even get playtesters involved, and then combine that with "ye can't get ye flask" when they go too far off the rails. Something like, "I'm testing the combat system here, so please don't try to parlay with the NPC, I need to know how combat works in a real situation."