This due to this part of ajax.js
// We don't know what response.data contains: it might be a string of text
// without HTML, so don't rely on jQuery correctly iterpreting
// $(response.data) as new HTML rather than a CSS selector. Also, if
// response.data contains top-level text nodes, they get lost with either
// $(response.data) or $('<div></div>').replaceWith(response.data).
var new_content_wrapped = $('<div></div>').html(response.data);
var new_content = new_content_wrapped.contents();
// For legacy reasons, the effects processing code assumes that new_content
// consists of a single top-level element. Also, it has not been
// sufficiently tested whether attachBehaviors() can be successfully called
// with a context object that includes top-level text nodes. However, to
// give developers full control of the HTML appearing in the page, and to
// enable Ajax content to be inserted in places where DIV elements are not
// allowed (e.g., within TABLE, TR, and SPAN parents), we check if the new
// content satisfies the requirement of a single top-level element, and
// only use the container DIV created above when it doesn't. For more
// information, please see http://drupal.org/node/736066.
if (new_content.length != 1 || new_content.get(0).nodeType != 1) {
new_content = new_content_wrapped;
}
This ensures that text nodes are not lost when replacing contents. The second parts of the script checks if the data sent is a single top-level HTML node, and if not, wraps it in a <div>
Back to your problem, the solution is to make sure that theme('custom_theme_function', array('test_param' => 11,)) returns your content in a single HTML node (div, span, ul, li... as long as it is single)
