How to create an entity-reference field programmatically

Could I call field_create_instance() in hook_form_alter()?

field_create_instance() is normally called in hook_install(), or hook_enable() from a module, or an installation profile.

You don't call it in hook_form_alter(), especially if the hook is altering the node edit form: Fields are attached to bundles (content types), not to single nodes. When a node is created, its content type is already created, and any field that content type requires is created before its nodes are created.
If there is a setting form that allows users to decide which fields to attach to a content type, then field_create_instance() is called in its submission callback.

A call to field_create_instance() is normally coupled with a call to field_create_field(). To be sure the field doesn't exist, you should first call field_info_field(). It returns NULL when the field doesn't exists.