It also encountered this issue, It looks very obvious the modal panel loading falls into a race condition and keeps waiting till time out.
I have tried many times, and found the solution is to hold the modal panel loading till it can be exactly found by webDriver and at the same time keep refresh the webDriver instance , then try to find WebElements within the modal panel.
So the solution is like follows:
e.g. MyModalPanel is your ModalPanel Id, then do the following
page.openModalPanel();
Assert.assertTrue(page.waitTillDisplay( "MyModalPanelContentDiv"), Wait.MODAL_PANEL));
page.fillInFormInModalpanel(formObj);
and the waitTillDisplay code can be found on WebDriver website, I will just paste my code here for your reference:
public Boolean waitTillDisplay(final String id, int waitSeconds){
WebDriverWait wait = new WebDriverWait(driver, waitSeconds);
Boolean displayed = wait.until(new ExpectedCondition<Boolean>() {
public Boolean apply(WebDriver driver) {
return driver.findElement(By.id(id)).isDisplayed();
}
});
return displayed;
}