Fixing wave-like movements when rotating an object using mouse input in Unity

I have encountered a rather odd issue with my program.

The setup:

In my scene, I have a GameObject which I have set as my player/character for this game. The player/character has a child GameObject, which is a cube, which represents the head, and is a child via the hierarchy. It also has a body for the character(both of these are just children of the main children/character).

When I move the mouse, rotation of the head should follow where the mouse is(the mouse is hidden, so it relies on mouse input). If the movement around the y-axis is larger than a set amount, the body of the character should turn as well. I have this all working fine.

The problem occurs with the actual movement. In certain circumstances, it will create a wave-like movement, while still following the mouse

Ie. if I move the mouse vertically, then back to being "flat", then move it horizontally(without any y-movement), it will create a wave movement horizontally. The same occurs with the other rotation, where if I move the mouse horizontally first, then move it up and down, it will create a wave, but only in the up/down movement.

Gyazo for GIF demonstrating issue

In the GIF above, I move the mouse slightly upwards, then left and right. The "wave-like" movement can clearly be seen, as it moves to the left, and to the right.

This is my code:

void moveView()
{
    //This method is called every Update(), after a method that locks the cursor using
    //Cursor.lockState = CursorLockMode.Locked, then sets it to Cursor.visible = false;
    //Raw inputs, ySen and xSen are public floats.
    toRotOnY = Input.GetAxisRaw("Mouse X") * xSen;
    toRotOnX = Input.GetAxisRaw("Mouse Y") * ySen;
    //I think its here, but I'm not sure exactly what the *= operator does to Quaternions. Also, the init values are set in Start()
    targetRotHead *= Quaternion.Euler(-toRotOnX, toRotOnY, 0f);
    //The 0f in the Z feild is to lock rotation to x/y only.
    //Should body rotate to match head?
    if (this.transform.eulerAngles.y >= targetRotHead.eulerAngles.y + 10 || this.transform.eulerAngles.y <= targetRotHead.eulerAngles.y - 10)
    {
    //If the body should rotate, do it here
        this.transform.rotation = Quaternion.Slerp(this.transform.rotation, Quaternion.Euler(0f, targetRotHead.eulerAngles.y, 0f), moveSpeed);

    }
    //Rotate to the wanted rotation
    //moveSpeed is a float which dictates the smoothing.
    headWithCamera.transform.rotation = Quaternion.Lerp(headWithCamera.transform.rotation, Quaternion.Euler(targetRotHead.eulerAngles.x, targetRotHead.eulerAngles.y, 0f), moveSpeed);


}