programmatically add an existing entity reference autocomplete field in custom form

If you are using the references module's node_reference field, then, in your custom form, you'll want to create a field that uses the node_reference's autocomplete, like so:

$form['referenced_field'] = array(
  '#title' => 'Referenced field title',
  '#description' => t('Referenced field description.'),
  '#type' => 'textfield',
  '#autocomplete_path' => 'node_reference/autocomplete',
  '#required' => TRUE,  // Optional depending on your circumstance.
);

See node_reference.module line 892 for more details on parameters, etc.