In Firebug, in the Console tab,
watch for the reply from the server when the popup form is opened.
Then use the XML setting there to see if the reply is valid xml.
The most likely cause of the problem is that something being returned in the generated form is not xHTML compliant.
Cheers
Bruce