How to delete a library which is not visible for package manager?

First you should check what version of mathplotlib you have installed. There should be __version__ variable in __init__.py which defines version of lib. Next you need to download sources of this lib from their homepage. After that:

tar xvf matplotlib-VERSION.tar.gz
mkdir ~/temp-mpl
cd matplotlib-VERSION
python setup.py build
python setup.py install --home ~/temp-mpl --record list.txt

After that in list.txt there should be list of files that are installed by mathplotlib in your version. So then you need to replace in it your prefix ~/temp-mpl to path where your mathplotlib is installed and remove for example in this way:

cat list.txt | xargs rm -rf

It will remove all files installed by your specific version of mathplotlib. If you installed some dependencies manually you need to know what it was to remove it.

Remember to remove ~/temp-mpl.

BTW, why you installed it from sources? There is python-matplotlib in repo.

Remove it by a judicious use of rm.

If you installed it outside your package manager, why do you expect the package manager to know how to remove it?

Note that this of course means that any programs that rely on this library won't work anymore, and since it's outside of package management nothing will warn you about it.