Do I really need to use a graphics API?

Practically its necessary, yes. It's necessary because unless you want to spend years writing what is essentially driver code for the multitude of different hardware configurations out there, you need to use an API that unifies against existing drivers written by GPU vendors for all popular operating systems and hardware.

The only realistic alternative is that you don't use 3D acceleration, and instead go for software rendering which, provided it's written in something truly portable like C, will be able to run on just about any system/device. That's fine for smaller games... something like SDL is suited to this purpose... there are others out there as well. But this lacks inherent 3D capabilities so you'd have to build them yourself... which is not a trivial task.

Also remember that CPU rendering is inefficient and suffers poor performance compared to GPUs.

I think a lot of the answers miss an important point: you can write apps that access hardware directly, but not on modern operating systems. It's not just a time problem, but a "you don't have a choice" problem.

Windows, Linux, OSX, etc. all ban direct hardware access to arbitrary applications. This is important for security reasons: you don't want any random app to be able to read arbitrary GPU memory for the same reason you don't want any random app to be able to read system memory. Things like the framebuffer for your bank account or whatnot live in GPU memory. You want that stuff isolated and protected and access controlled by your OS.

Only drivers can talk directly to most hardware, and the only way to talk to the driver is via the OS' exposed HAL and each driver's proprietary and incompatible interface. That driver interface will not only be different for each vendor but will even differ between versions of the driver itself, making it near impossible to talk directly to the interface in a consumer application. These layers are often covered by access controls that further restrict the ability for an application to access them.

So no, your game cannot just use the hardware directly, unless of course you're only targeting insecure operating systems like DOS, and your only feasible option for a game on modern consumer operating systems is to target a public API like DirectX or OpenGL.

APIs like OpenGL or DirectX are partialy implemented by the operating system and partially implemented by the graphic driver itself.

That means when you would want to create your own low-level API which makes use of the capabilities of modern GPUs, you essentially need to write an own graphic driver. Making sure that your driver works with all common graphic cards would be quite a challenge, especially because the vendors are often not very open with their specifications.

In summary: Theoretically you can, but it's unfeasible, and you won't get any advantage. The limitations APIs have today become less every day, you have CUDA and OpenCL and shaders. So having full control is no more a problem.

Fuller explanation:

Answering this is a boring yes. The real question is why?

I hardly imagine why would you want to do this other than for learning purposes. You have to know that at some point, developers had the freedom to do anything with their CPU rendering implementations, everyone had there own API, they were in control of everything in the "pipeline". With the introduction of fixed pipeline and GPUs, everybody switched..

With GPUs you get "better" performance, but lose a lot of the freedom. Graphics developers are pushing to get that freedom back. Hence, more customization pipeline everyday. You can do almost anything using CUDA/OpenCL and even shaders, without touching the drivers.

Boot your PC into MS-DOS. Then, using your copy of the PC Game Programmers Encyclopedia, you can write directly into the card's VESA registers and into video memory. I still have the code I wrote 20 years ago to do this and render rudimentary 3D in software.

Alternatively, you can just use DirectX; it's a really thin abstraction layer, and also lets you write directly into video buffers and then swap them to the screen.

There's also the extreme approach of building your own video hardware.

The other answers answer your main question quite nicely: technically, it's possible, but in practice, if your goal is to broaden your customer base, you're actually doing the opposite. While wasting a huge amount of work on supporting the thousands of different hardware and OS configurations you need to support.

However, that doesn't mean you have to be 100% dependent on one particular API. You can always code up your own abstraction layer, and then swap the more specific implementations (e.g. a DirectX implementation, an OpenGL implementation, ...) in and out.

Even then, it probably isn't worth it. Just pick something that works well enough for your purposes, and be done with it. You're an indie game maker - you need to keep yourself focused on the things that make the game, not on the minutae. Just make the game!

You want to talk to hardware. You need to use a way to talk to the hardware. That way is the interface to the hardware. That's what OpenGL, DirectX, CUDA, OpenCL and whatever else are.

If you get on a lower level you are just using a lower level interface, but you are still using an interface, only one that doesn't work with as many different cards as the higher level one.

To get 3D Hardware Acceleration without using a traditional API, you essentially need to write your own code to duplicate the functionality of the graphics driver. So the best way to learn how to do this is to look at the code of the graphics driver.

For NVIDIA cards, you should look at the open source nouveau project. They have a collection of great resources here.

For other brands of graphics cards, you can find similar projects and documentation.

Note that as has been mentioned in other answers, most modern operating systems will prevent user-space programs from directly accessing hardware, so you will need to write your code and install it as an operating system driver. Alternatively, you can use the FreeDOS operating system, which has no such restrictions, and should let you (in theory) directly access the hardware, and allow you to write a regular program that renders 3D hardware accelerated graphics. Note that, even leveraging code from an existing open source graphics driver, this would be a tremendous amount of non-trivial work (and as far as I'm aware, has never been done, and due to various reasons, might actually not be technically possible).