How to make unit selection circles merge?

You could for example check in the decal rendering pixel shader if there is an intersection with any other decals and kill the pixel if so. It's quite efficient to do for this kind of circular decals.

Do you mean those blue circles on the ground?

There may be a smarter way to do this but consider my solution as an example how this can be done. Render your circles to a separate empty texture of the size of the screen. At this point you can blend them together, however you want. Depending on your needs you can write the full color or just the information that a circle is present at given pixel. However, you will definetely need a depth information for each pixel that you write if you want a Z-test effect.

When you render the terrain after the previous step, you can sample the previously created texture, using screen coordinates in pixel shader, as UVs. Now you know, that the circle should or shouldn't be projected at given pixel of the terrain. However, this way it will work as if the Z-test was disabled, so a terrain can't hide any circle. If you want a terrain to hide circles that are behind it, perform a depth test before blending a circle with terrain texture. You can use a small bias to enable rendering of circles that are just a little bit under the terrain but you would still like them to render.

Edit: Now in order to get the effect of merged circles what you need to do is: When rendering to the separate texture, you need to check if there is already a circle rendered where you are trying to render a new one. (Check if the texture has a value there) if so, erase this pixel. This should give the effect of "outline" when two or more circles overlap. For this to work correctly, however you will need to do this test only for the inside of the circle, not the border. So when rendering circles, write some specific value to the output texture, which indicates that it is a circle inside. Then you just need not to render anything on the circle inside and you should be good.

Edit2: For your simple red/green example: Before you render any pixel, check if this pixel is not already green. If it is, don't render. This will do the trick. In fact, while writing to the texture you can even use red/green for outside/inside of the circle, and then while rendering the terrain read those values and convert them to a desired color.

If my answer is too abstract by any chance, let me know.

There are a few options. As a general method, stencil buffers often come in very handy where certain drawing needs to be masked out, like the outline where the circles overlap in your example.

In this case, I think this can be done just as easily without a stencil buffer. You can use the depth buffer to eliminate the outline where the circles overlap. The idea is that you draw the interior of the circles into just the depth buffer (since we don't want to see the interior), and then draw the outline. This way, the part of the outline that overlaps with another circle will be eliminated by the depth test.

The only caveat is that you have to be careful about depth fighting. You can use a small offset to make sure that the outlines are in fact behind the interior, and get eliminated by the depth test. An alternative would be to use glPolygonOffset().

Let's say you have two circles that are parallel to the xy-plane, with centers at (x1, y1, z) and (x2, y2, z). And you have these draw functions:

// Draw interior part of circle, shown in green in the schematic in the question.
drawInterior(x, y, z);

// Draw outline of circle, shown in red in the schematic in the question.
drawOutline(x, y, z);

The sequence of drawing then looks like this, with delta being a small offset:

glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
drawInterior(x1, y1, z + delta);
drawInterior(x2, y2, z + delta);
glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
drawOutline(x1, y1, z);
drawOutline(x2, y2, z);

Off the top of my head, if these decals are drawn as discs, I would test each circle for intersection against each other circle, and put the intersection points into an array. Then draw arcs that are not inside an intersection. This is not a shader solution, though.