Empirically, software with higher complexity metrics, such as cyclomatic complexity, are harder to maintain. There is research supporting this dating back to the 1970s ("Program Complexity and Programmer Productivity", E.T. Chen). There is also work that suggests that complexity density, which is cyclomatic complexity over size of the system also relates to maintenance time ("Cyclomatic complexity density and software maintenance productivity", G.K. Gill, C.F. Kemerer), which is also available for free here. Unfortunately, an IEEE subscription is necessary to Chen's paper, but you can try to look it up on other sources if you're interested.
From a quality perspective, it's often worth it to spend some time refactoring, assuming you have a test framework in place to prevent the introduction of new defects. This will allow you to more easily implement new features to your system, add additional tests, and train new developers to work.
Ultimately, however, there is the pressure to deliver new functionality and added value. You need to balance refactoring with the implementation of new functionality and repair of defects.