Calculating intersection of line segment with view frustum

I want to calculate the intersection of a line segment with the view frustum.

Here's what I tried so far. The code works perfectly when the look direction vector is orthogonal to the line. E.g. camera at (0,0,-100) looking at (0,0,0) and line from (-500,0,0) to (+500,0,0). However, as soon as I turn the camera sideways, the algorithm calculates intersection points that are not even on the line (something like -1000,0,0).

Where is the problem with my intersection method? Alternatively, is there an easier way to accomplish the same thing?

public static bool ViewFrustumLineSegmentIntersection(Vector3 lineStart,
    Vector3 lineEnd, Matrix viewProjection,
    out Vector3 firstIntersection, out Vector3 secondIntersection)
{
    // we do all calculations in projective space,
            // and afterwards convert the calculated intersection points back to worldspace.
    lineStart = Vector3.TransformCoordinate(lineStart, viewProjection);
    lineEnd = Vector3.TransformCoordinate(lineEnd, viewProjection);
    firstIntersection = secondIntersection = Vector3.Zero;

    BoundingBox viewFrustum = new BoundingBox(new Vector3(-1, -1, 0), new Vector3(1, 1, 1));

    var direction = lineEnd - lineStart;
    direction.Normalize();

    // First, we shoot a ray from lineStart to lineEnd to get the first intersection
    Ray r = new Ray(lineStart, direction);
    float lineLength = (lineEnd - lineStart).Length();
    float distance;
    if (!BoundingBox.Intersects(viewFrustum, r, out distance) || distance > lineLength)
        return false;
    firstIntersection = r.Position + r.Direction * distance;

    // now do the same from the other side to get the second intersection.
    r = new Ray(lineEnd, -direction);
    BoundingBox.Intersects(viewFrustum, r, out distance);
    secondIntersection = r.Position + r.Direction * distance;

    // Now convert the intersection points back to world space.
    var ivp = Matrix.Invert(viewProjection);
    firstIntersection = Vector3.TransformCoordinate(firstIntersection, ivp);
    secondIntersection = Vector3.TransformCoordinate(secondIntersection, ivp);

    return true;
}