You cannot block direct access by IP. You must allow the connection but then decide what to do with it. This could be return a 403, a 404 or redirect them to the desired page. You can do this with mod_rewrite.
RewriteEngine On
RewriteCond %{HTTP_HOST} ^123\.123\.123\.123
RewriteRule ^(.*)$ - [F,L]
This will match the HTTP HOST header passed by the web client. All other requests would pass through.
However, you may want to normalize your URLs for SEO purposes.
With this approach you rewrite anything that does not match the desired result.
RewriteCond %{HTTP_HOST} !^fully\.qualified\.domain\.name [NC]
RewriteCond %{HTTP_HOST} !^$
RewriteCond %{SERVER_PORT} !^80$
RewriteRule ^/(.*) http://fully.qualified.domain.name:%{SERVER_PORT}/$1 [L,R]
Reference: https://httpd.apache.org/docs/2.0/rewrite/rewrite_guide.html#canonicalurl
Reference: http://en.wikipedia.org/wiki/URL_normalization