Angularjs routing and html5

I have some routes for my application:

$routeProvider.
            when('/user', {templateUrl: 'user/partials/userlist.html',   controller: 'userListController'}).
            when('/user/:userName', {templateUrl: 'user/partials/userdetail.html', controller: 'userDetailController'}).
            when('/group',{templateUrl: 'group/partials/grouplist.html', controller: 'groupListController'}).
            when('/group/:groupName', {templateUrl: 'group/partials/groupdetail.html', controller: 'groupDetailController'}).
            when('/category',{templateUrl: 'category/partials/categorylist.html', controller: 'categoryListController'}).
            otherwise({redirectTo: '/user'});

the root of the application is /admin/ and I have my .htaccess set to default to index.html so the site looks like: www.domain.com/admin/#/user/username...etc... in the non HTML5 version.

Everything works fine when it's like that. When I turn on HTML 5 it redirects immediately to: www.domain.com/user/ and of course, it doesn't work. How can I tell my routing provider that the root of my module is /admin/?

EDIT:

I added: <base href="/admin/"></base> to the last line of my index.html head tags and now when I go to www.domain.com/admin it correctly redirects to www.domain.com/admin/user and the userlist template and controller are used. However, when I navigate directly to www.domain.com/admin/user it gives me a page not found error. I'm starting to wonder if this part of it has to do with my default rewrite rule:

RewriteRule ^$ index.html

EDIT:

Yup. Figured it out. Took out the Rewrite entirely and it worked for from the default redirect to user and then going to userdetail it worked, however typing in /user or any other route doesn't work...it seems like it only works with html5 is when I do the navigation from the application itself, I can't navigate directly to the different views. for instance when I try to go to group directly it doesn't work.

So, no rewrite rule to rewrite to index, the user default route works, group and category don't.

EDIT:

Here's what I've found. With no rewrite for the default index.html, the $location.html5mode(true) set, it works so long as when you navigate to the page you type in www.domain.com/#/user/ or www.domain.com/#/group. it immediately rewrites the URL as www.domain.com/user or www.domain.com/group. but if you try to navigate to www.domain.com/user directly it doesn't work. obviously, rewrite rules won't help here because it's the front end doing the processing of the route.

Is this standard behavior for angular routing?

EDITED TO ADD TO THE SOLUTION:

If you have any additional routes in place that redirect to subdirectories, besure to either use the [END] tag or make sure the directories have mod-rewrite turned off or it will process the generic rule again after the rewrite and everything will redirect to the index.