How do you maintain protocol of function return value in dynamic language?

People who prefer dynamic languages don't consider this to be a problem in the first place. the basic tools used to deal with this are simply good testing. Yes, you cannot detect a problem such as you describe without running the code, so the key is to run the code through a good test-suite, probably as often as you compile.

There are disadvantages to this, of course. If code doesn't get tested, the error goes undetected whereas a compiler would have caught it. But what you lose there, you gain in speed of development, and the ability to use looser coupling.

Your code will of course throw when norm is called. So if you run a basic test suite that calls norm with the results of make_vector2 at least once, you'll catch the error. If you run those tests as often as you would compile a C++ or Java program, then it really makes no difference. Either way, the bug gets caught.

The other part of the answer is to try to design the code in a way that the contract is more clear. For example:

class Vector:
    def __init__(self,x,y):
        self.x = x
        self.y = y

def make_vector2(a, b):
    return Vector(a,b)

def norm(v):
    return math.sqrt(v.x**2 + v.y**2)

That still has to be tested the same way as the original code, of course, but this documents the protocol in a way that the earlier version doesn't. It means that you can do a help(make_vector2) and see that it returns a Vector object with an x and a y attribute. This makes the error less likely in the first place.

In general, returning raw dictionaries is best when you are intentionally keeping things loosely coupled and therefore can handle more variant contents.