How to get the closest “visible” object from my “player”?

Typically, to get the closest object from the player, you do something like this (generic pseudo-code):

var listOfObjects = getAllObjects()
var closestObject = null
var distanceSquareOfClosest = infinite

for otherObject in listOfObjets:
  if otherObject != this:
    var distanceSquareOfThisObject = (otherObject.position - this.position).lengthSquare()
    if distanceSquareOfThisObject < distanceSquareOfClosest:
      distanceSquareOfClosest = distanceSquareOfThisObject
      closestObject = otherObject

I don't know how it's done in Unity, but I would assume there is something that lets you get all the objects, and there would be a way for you to check if they're visible or not. You'd modify the if otherObject != this to something like if otherObject != this && otherObject.isVisible.

To find all the game objects, you can take a look here.

As mentioned in the comments you probably want a Raycast.

An example of how to implement a forward-facing raycast in a Fixed Update cycle is spelled out exactly in the Unity Scripting API Documentation here. This will return true if the raycast strikes a collider in front of whatever object you attach the script to.

From the documentation above:

using UnityEngine;

public class ExampleClass : MonoBehaviour 
{
    void FixedUpdate() 
    {
        Vector3 fwd = transform.TransformDirection(Vector3.forward);

        if (Physics.Raycast(transform.position, fwd, 10)) 
            print("There is something in front of the object!");
    }
}

After that, to return the object you strike with your raycast, you use a Raycast.Hit to get the transform of whatever you're hitting. We do this because the transform has a variable of gameObject. SO:

        using UnityEngine;

public class ExampleClass : MonoBehaviour 
{

public GameObject storedGameObject; //create a bucket to store the game object in when we find it

    void FixedUpdate() 
    {
    RaycastHit hit; //define the raycasthit 

        Vector3 fwd = transform.TransformDirection(Vector3.forward); //define the forward Vector3

        if (Physics.Raycast(transform.position, fwd, hit)) //use the current transform position to start, shoot outward "fwd", and return a RaycastHit
            {
            storedGameObject = hit.transform.gameObject; //set the bucket to the value of the target's transform.gameobject
        }
    }
}

Full disclosure: I'm at work and can't test this code, but it should be approximately what you need.

You can use the dot product trick to get a list of instances actually visible by the player (thus lying inside the camera frustum), and check for the distance from your camera object to get the closest one.

First, you can test a certain number of instances so that their distance is low enough for them to be considered "near" the camera (or the player). Instead of testing every object, which can be overkill for your game performance, you can consider a volume around the camera (a sphere, for example) and test only object inside this space partition. The number of instances to work with is now lower, and more tests can be performed.

In mathematics, the dot product of two vectors gives as result a scalar number, which represents the projection of vector v₁ upon vector _v2. The length of projection depends on the cosine function of the angle between the two vectors. We can see it the other way: given two vectors, if their dot product is a value k, then their mutual orientation in the space (or plane, or any n-dimensional system) is such that the angle between them has a cosine value cos(θ) = k. So, the angle is θ = arccos(k).

If your camera has a FOV angle ranges from -β to β, assuming angle 0° is in front of the camera and that the frustum is just a cone, if θ is either lower than -β or greater than β, your object is out of sight.

This said, you consider two vectors, A and B. Vector A is the relative position of an object, and B is the "front" vector of the camera. If the dot product of A and B is lower than or equal to the cosine of β, such object is potentially visible to the player. Otherwise, its position is away from the sight cone.

The final step is to search the list of remaining objects to find the one whose distance is lowest among all. Here's your closest, visible object to focus your attention on!

To sum up: consider a list of candidate object, so that every object is closer than a certain distance from camera. Help comes from space volume instead of checking every single instance. Then, compute dot product of those instances, to get a list of only instances lying in front of the camera. Finally, look for the closest one.