1). EE is a CodeIgniter application and overrides the routing to the ee controller. To merge/use a custom application, you have to enable routing again. For to be able to update EE with less hassle, it is probably better to not merge a large app with libraries, helpers and controllers inside the ExpressionEngine directory. You can install/create your app in its own sub-directory, and set APPPATH accordingly.
- The manual is a good start: Further https://ellislab.com/codeigniter/user-guide/
- See also this answer on StackOverflow Expression Engine Controllers
2). If you also want to make use of EE functionality (members, backend-layout, etc.), and have a more tight integration in the backend control panel, you can develop an add-on. Extensions are class-methods that are called from certain points in other libraries, to alter data or do their own stuff, Modules are more or less standalone mini-apps inside EE. See the development manual
https://ellislab.com/expressionengine/user-guide/development/index.html