AngularJS and MVC Routing

I've been doing a lot of research on implementing a SPA with AngularJS on MVC, but I think I'm missing something. Do I need a MVC controller for each View still, or should angular be able to grab the cshtml or html file to render on the initial page without having an MVC controller?

Essentially I have

public class HomeController : Controller
{
    public ActionResult Index()
    {
        return View();
    }
}

Which returns my index page which is just

<div data-ng-view></div>

It is also loading a layout with the Index page that has my menu on it with all of the links, as well as all of the script files. When I click a link such as:

<a href="/Home/ChangeLog">Change Log</a>

Which contains

<h2>{{Changes.Title}}</h2>
<div>{{Changes.Version}}</div>

It fails to load anything because it's trying to reach out to the MVC controller. If I have an MVC controller returning a partial view, it returns just the partial view as a whole page load. My angular app file is this:

var myApp= angular.module("myApp.WebConsole", ['ngRoute']);

myApp.controller("IndexController", IndexController);

var configFunction = function ($routeProvider, $locationProvider, $httpProvider) {

$routeProvider.when("/Home/Index",
    {
        templateUrl: "Home/Index.cshtml",
        controller: IndexController
    });

$routeProvider.when("/Home/ChangeLog",
    {
        templateUrl: "Home/ChangeLog.cshtml",
        controller: IndexController
    });
$locationProvider.html5Mode(true);
}

configFunction.$inject = ["$routeProvider", "$locationProvider", "$httpProvider"];

myApp.config(configFunction);

And the indexcontroller.js file is:

var IndexController = function ($scope) {
    $scope.Heading = "Index Page";
    $scope.Changes = {
        Title: "Changes",
        Version: "1.0"
    };
}

IndexController.$inject = ['$scope'];

The js files are in a separate folder. The views are in the standard MVC folder structure.

What am I missing about how this works? I would expect that clicking the link would go to retrieve the cshtml file or even just an html file and render the html in the div with the data-ng-view tag on it. Should I not be using Razor views (perhaps part of the problem since I expect the MVC controller would need to render the code data, if any)? Though I still have the same issue of not finding the page if I use just html.

Edit 2: Ok, so I've partially figured this out, the problem is using

$locationProvider.html5Mode(true);

Without this, if I make the link on the menu

<a href="/#/Home/ChangeLog">Change Log</a>

Then everything seems to work ok. I believe that is also the expected format for links using Angular. Is there some magic to making MVC and Angular routing with html5Mode enabled?