Transcript of a FAQ:

hey, is there any guidance on when it makes sense to use propulsion and when it makes sense to use fskafka?

FsKafka does:

a) basic logging of errors, support for periodically emitting lag stats per the broker
b) handling in Batches (BatchedConsumer.Start) - you provide a body that is not allowed to throw and takes care of any desired parallism
c) handing in Batches, grouped by Key - BatchedConsumer.StartByKey https://github.com/jet/FsKafka/blob/master/src/FsKafka/ConfluentKafka.fs#L484 - it takes care of max parallelism across the groups, but if you throw, the service stops
d) no throughput stats
e) producers (edited)

Propulsion does:

a) handlers like b/c above but with

• more detailed logging: latencies, successes, fails
• ability to emit outcomes from handlers and compose summary stats from them cleanly
• keeps a list of tasks and manages retries
• makes reading completely async from handling (which is not a big win for Kafka but is for other things)
  b) BatchedConsumer: ability to hold state of multiple streams for complex stateful consumers
  c) mix and match / swap ES vs CFP vs Kafka consumers with same semantics (edited)

Propulsion.Kafka does

a) depends on FsKafka
b) adapt Confluent.Kafka/FsKafka to ingest into Propulsion (using CK 1.x)
c) adds your Monitor stuff for 1.x (edited)

Propulsion.Kafka0 does

a) NO other dependencies aside from Propulsion
b) adapt Confluent.Kafka/FsKafka to ingest into Propulsion (using CK 0.11.3)
c) your monitor stuff for 0.x (edited)

Reasons not to use FsKafka

Things FsKafka currently has gaps on relative to Propulsion/Propulsion.Kafka aside from scheduling capabilities / retry management:

• supporting 0.x
• logging stats about batches wrt consuming or producing
  (some of the logging gaps will likely be plugged by some work currently being carried out)

Reasons to use Propulsion.Kafka

• One clear case for using Propulsion over FsKafka for me is when you are doing lots of parallel work that can fail or be rate limited and/or otherwise necessitate retries while maintaining good throughput - i.e. writing to Cosmos
• providing a place to hook in whitelisting/blacklisting/poison message management in a generic way

Rules of thumb

• If it can be done with FsKafka, its a damn good place to start in general - less Things. ... But if you take that too far, you're writing loads of untested snowflake infrastructure code against specific versions of client libraries with no easy way to upgrade things when breaking changes to underlying libraries come (vs Propulsion adding a binding that implements the same abstraction over a changed API contract from something like Confluent.Kafka / EventStore.ClientAPI / Microsoft.Azure.Documents etc)