Representational State Transfer (REST) is based on the HTTP protocol and it uses HTTP verbs (GET, POST, DELETE, etc) to transfer representations of resources. The World Wide Web itself is actually a good example of a REST architecture in that you make a request for a resource (a web page) and the system returns a representation of that resource (HTML). As usual there is a pretty good article on Wikipedia that goes into a lot more detail. The term 'Restful Service' is used to describe a service that adheres to the principles of the REST pattern.
A Restful API is really just a published interface that allows control of a server or service programmatically - for example, with GeoServer it is possible to use the Rest API to manage its configuration. Often you will find that the Rest API is used by the supplied user interface or components - so for example all ArcGIS 10 Server clients utilise the Rest API that the core ArcGIS Server exposes.
I think you also need to consider the term "Server" in the context you have used it. A server in this context should really be considered the physical hardware on which software is running. In your examples the software is ArcGIS Server and GeoServer. These software (and there are many others) are what allows the "Server" to render the spatial data.
So, what makes them have the capability to render spatial data is the software that is running on them. The Restful APIs are simply a means by which other clients (OpenLayers, ArcGIS for Desktop, etc) can either control or request data from the server software.
