Camera pan clips shadows?

[kpeeters]

I am trying to implement a simple 2D panning of the viewport by pre-multiplying the camera matrix with a translation; something like

auto cp = camera->getCullingProjectionMatrix();

math::mat4f trans(math::float4{1.0f,      0,      0,     0},
                  math::float4{0,      1.0f,      0,     0},
                  math::float4{0,         0,   1.0f,     0},
                  math::float4{pan.x, pan.y,      0,  1.0f} );

camera->setCustomProjection(trans * cp, camera_near, camera_far);

This pans the view fine, but it also clips shadows depending on how far I pan. Using getProjectionMatrix instead of getCullingProjectionMatrix does not change that.

Am I misunderstanding the logic or conventions followed in Filament for the camera matrix? I am doing something fairly similar in three.js, where it works as expected. Thanks!

(If there is a better way to pan the view then I'm all ears. Would like to avoid changing the viewport as it tends to be slower with three.js than changing the camera matrix, and would require me to change more backend code. In any case, I'd like to understand what I am doing wrong above).

[romainguy]

@pixelflinger can confirm, but if you are not updating the model matrix of the camera various internal computations will be incorrect. In particular Filament performs tight clipping of the shadow maps which may rely on the camera position somewhere. Other computations for lighting would be affected too.

[pixelflinger]

Yes, it's not documented properly, but the projection can't have a translation in it. To pan the camera, you can simply update its model matrix with your pan values.

[kpeeters]

But if I change the camera's model matrix the camera moves within the scene, seeing objects from a different angle. What I am after is simply a translation of the final rendered image on the screen, as if I had translated the viewport. So purely a 2D thing in screen space.

[pixelflinger]

Ah... I see. I'm going to have to investigate then. Can you attach a screenshot of what you're seeing?

[kpeeters]

The problem looks very similar to what I reported ages ago in #742 (which was my mistake, not Filament's, but I think this is different, so please read on). So my first response was to carefully check my bounding boxes once again. But I compute them with filament::RenderableManager::computeAABB now and I just don't see any error with them. Basically, I get that kind of effect, but with the camera located somewhere to the right, and the light shining towards the camera.

So without pan it looks like the view-from-the-right of the bottom figure (light shining towards the camera, the object casting a shadow towards the camera), but once I start panning to the right, the shadow gets clipped more and more, to look like the top figure there (but again viewed from a camera located to the right). Eventually the shadow disappears altogether.

Similar for moving up/down: if I pan the camera up, the shadow gets clipped from the bottom, and that clip moves further and further up as I pan more up. It is not just the shadow plane that is affected. Self-shadowing similarly clips. The clip is not a straight vertical line but seems to be influenced by perspective (that is to say, it's not just a 2D clip in screen space).

Was too busy today but I'll try to cook up a small example and some screenshots later if the above does not ring any bells. Let me know.

[pixelflinger]

Is it just the shadows that get clipped, or the geometry as well? What about dynamic lighting?

[pixelflinger]

I tried to reproduce this, but I didn't see any problem with the directional light's shadow. Did you see the problem with all types of shadows?
I implemented the sliding of the panel in our gltf_viewer with the code you gave above and I saw no problems with the shadows of either type.

[kpeeters]

Yes, all shadows. Sorry for the silence, very busy here, and I wanted to see if I could get it down to a minimal example. Thanks for the new 'shift' function, will use that for further tests. Will get back on this sometime next week.

[pixelflinger]

I've been thinking more about it and I think I may have an idea of why you're having some issues. I'll look into it on Monday.

I think this might be because I have some coffee that makes assumption about what's on the projection matrix in a few places.

A minimal example would be great though.

[pixelflinger]

@kpeeters would you be able to try this:

In Camera.cpp around line 205, replace the implementation of inverseProjection(const math::details::TMat44<T>& p) with return inverse(p);

template<typename T>
math::details::TMat44<T> inverseProjection(const math::details::TMat44<T>& p) noexcept {
    return inverse(p);
}

Does it solve the problem or make it better?

Thanks!

[kpeeters]

That doesn't seem to make any difference, unfortunately. I'll send you a video later today.

[pixelflinger]

I did more investigation today and as far as I can tell, by code inspection, it should work (famous last words).

[kpeeters]

The attached video is with the ma/shift branch, with the only difference between frames being a change in camera->setShift. The scene is some fixed geometry for the white object, a spotlight between the object and the camera, and a directional light above and behind it, casting a shadow on the shadow plane below. The geometry is unchanged for this entire clip.

I would expect the shift to just move the image, but instead the shadow gets clipped.

(The background is a fixed-size unlit rectangle, not a skybox, hence the appearance of the black bar on the left-hand side at some point; that is expected).

shift.mp4