I finally was able to fix my site in our stage environment until I logged out of the admin section. Now the site has no css at all. All the html seem to be fine. If I login as admin, the site is still perfect.
I tried to clear cache, still same problem...
I tried to change the theme, the new theme has css. As soon as I get back to the original theme: no css as anonymous.
I looked at the source code and the includes are not there at all. So forget about a bad path.
I looked at my theme.info and it is fine.
What can possibly add tags in the while logged in as admin and not as anonymous?
Note: when there is no css, there is no javascript. So you could read this question and replace all "css" for "javascript"