RFC: Improve ease of creating new sources and sinks #12025
Description
As it stands now, each of our sources and sinks is a bespoke piece of relatively complex code. However, many of them follow a similar pattern. For example, a large number of our sinks pull events from their input channel, generate a batch, encode the batch into a request, ship it over HTTP, and then trigger retries or acknowledgements based on the response code (or lack thereof). Similarly, we have sources that accept HTTP requests, decode the data, and generate responses.
Creating a new code base for each source or sink could become unscalable as we increase the number of sources and sinks Vector integrates with, both in terms of duplicated code and opportunities for bugs and build times in general.
Is there an opportunity for a new codec-like layer that could be used to simplify the creation of new sources or sinks by encoding this logic into more generic structures? More interestingly, could that be done with a non-compiled code base by leveraging an interpreter like VRL to describe how those sources and sinks work? Are the performance implications from this kind of approach manageable due to the interpreter being involved only at a higher level (as opposed to per-event handling)?