The message-queue tag has no usage guidance.
0
votes
0answers
14 views
Message queues and triggered comms within a native app
I'm building a native app which currently has zero backend infrastructure.
With services like Firebase's Authentication, database and notifications means that all interactions are handled client-side ...
0
votes
0answers
24 views
Changing a rigid flow to a dynamic SOA approach: how to orchestrate and route?
The company I work for has a piece of software that deals with EDI (Electronic Data Interchange). It consists of a number of processing steps, which run concurrently but each of which individually ...
0
votes
0answers
50 views
How can I do recurring operations on a single resource in asp.net core?
I am building a WebApi targeting asp.net core on Linux. This WebApi depends on an external resource. I need to be able to access this resource from some of my API requests as well as doing recurring ...
0
votes
0answers
37 views
Best design for Azure worker queue process
Consider the situation.
A worker was running on Azure Worker role. It will constantly polling on an Azure queue when an item found it will process the item. The process will have some jobs like job1, ...
1
vote
1answer
153 views
Message driven architecture and horizontal scaling
During working on a pet project in microservice event driven architecture, I've faced a problem of delivering responses to a client.
Each microservice (API Gateway etc) has many replicas and has it's ...
7
votes
2answers
122 views
Microservices - compensate service failure with queue
We are using some sort of microservices approach in our app (although it is not really consequently adhered to).
When a service is down or throwing an exception, the approach is to put it into a ...
4
votes
1answer
56 views
Message publishing interface when deferring until transaction is committed
Suppose I have the following interface used for publishing messages onto a message queue:
interface IMessageProducer
{
void Publish<TMessage>(TMessage message);
}
Normally, an implementer ...
1
vote
0answers
43 views
Proof-of-concept of brokerless P2P NServiceBus with Request-Response. Worth trying?
I'm new to message queueing.
I'm trying to do a proof-of-concept of a messaging architecture for communication between multiple windows desktop apps of the same type. In simpler words, several ...
0
votes
1answer
71 views
AMQP routing-keys naming anti-patterns when using topics
I'm working on the specification for building a web services platform and AMQP is the messaging protocol used for the interaction between the components of this platform.
I'm using topic exchanges ...
0
votes
0answers
25 views
Synchronized computation task distribution
I have a system where a set of Operations needs to be done every day. Each Operation has the following metadata: Type, Entity and a Date (independent of current date). It reads something from the ...
0
votes
1answer
57 views
Where should I store stream wide metadata in stream processing?
I am building a stream processing architecture and was wondering what to do with metadata about the stream. For instance, every message of data coming from a source has the same attribution, as it ...
0
votes
0answers
18 views
Metrics Collector at Microservices
I am developing an application which is based on microservices architecture with Spring Boot. Beside my application jar I have a dashboard application jar too. I want to show some metrics at both my ...
3
votes
1answer
81 views
How to recognize consumers in AMQP architecture
I'm doing an project where I'm using a microservice architecture. All the services within this architecture communicate with each other using an AMQP broker (RabbitMQ).
When something happens in a ...
2
votes
1answer
87 views
Using messaging queue as async mysql writes
I read some articles about developers who takes rabbitmq and produce from PHP messages to write to mysql.
they do this to make the page speedup and not wait for mysql to return an answer.
So in my ...
8
votes
4answers
149 views
Handling changes in a event-driven microservice architecture
I'm doing an research-project where I'm researching the options to handle changes in an event-driven microservice architecture.
So, let's say we got an application where we got four different ...
2
votes
1answer
128 views
Best practice for delaying jobs
I have a system in PHP that works like a booking system.
Once a booking is confirmed there are a total of 7 emails/sms that needs to be send. The times of when they need to be send are already known.
...
0
votes
0answers
108 views
Service with task queue architecture
I have a REST endpoint that accepts a request from a client. This request needs to be done in steps and involves:
1) request to external service to perform a job
2) after this job is completed ...
0
votes
2answers
83 views
How can an update message from a rabbitmq broker reach all relevant clients via multiple websocket servers?
Lets say I have multiple websocket servers that maintain many connections to clients to send updates. These websocket servers will be pulling said updates from a rabbitmq broker, and broadcasting them ...
2
votes
1answer
73 views
What pattern should I set for handling event as working item?
I have this business event : "CustomerUpdated" published by my system.
I have many application listening to these events, and they do all king of action : sending email, updating the database, ...
4
votes
3answers
165 views
Slow throughput - still worth using a dedicated message queue?
Here are some requirements for a queue:
Every few days add ~100k tasks with various priorities
Workers will pull tasks at typically less than 10 / second
Tasks need to be completed by ~2 unique ...
0
votes
0answers
32 views
Integrating ActiveMQ in jetty
I was wondering how to integrate an ActiveMQ message broker into jetty, so that jetty starts up the broker (or ensures that it is running) before running the services.
Right now we have solved this ...
4
votes
2answers
173 views
Handling cases where the message queue is unreachable
We are using Apache Camel with ActiveMQ in a microservice architecture in a growing system of integration solutions. I was wondering how each service should react when they are unable to connect to ...
0
votes
0answers
50 views
Set up E-Mail message queue with digest system
We're currently building out a system to send out e-mail notifications in a digest format. We'll be delegating the actual sending to Sendgrid, but first we need to construct these messages using the ...
6
votes
1answer
150 views
Best practice for ensuring name uniqueness/correctness in message queues in a microservice architecture/distributed system
I was wondering what a good solution would be for ensuring that queue-names are entered correctly and are only used by the correct applications in a large system which uses message queues to exchange ...
0
votes
1answer
64 views
A WebSocket API, reply message
I am writing a Web application for copying files between servers.
I decided to write a WebSocket server which will receive and send relevant messages (that is communicate with a browser), start and ...
2
votes
2answers
74 views
AMQP broker and happen-before guarantees
I am trying to figure out how an AMQP broker, such as RabbitMQ, can be integrated in our architecture. This should allow to easily scale horizontally in the future.
For the sake of simplicity, let me ...
3
votes
1answer
267 views
Web API to Windows Service communication via ZeroMQ
I have an ASP.NET Web API 2 web service. This is the interface customers will use to submit data.
I have a Windows Service running on an internal server. I will be sending the customer's POST data to ...
4
votes
2answers
214 views
How to handle thousands of messages to C socket?
I currently have a setup looking like this:
__________________ _________ ________ ___________
| Front end server |----| Varnish |----| NodeJS |----| C-service |
‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾‾ ...
0
votes
1answer
190 views
Cache invalidation on multiple servers
I have 6 web servers which are giving me problems due to cache inconsistency. I am thinking of building a cache invalidation service such that there is a topic on which all the servers can publish a ...
1
vote
1answer
88 views
Message persistence in Message Queues
I have looked into several message queues as referenced in this link. Several of the larger and more popular ones are in memory only.
I can understand the need for this in some situations however in ...
5
votes
2answers
209 views
Master-Slave Cluster - How to make sure the master is really dead for the slave to take over?
I have a in-house messaging system, similar to a message broker. We have one master message broker and one slave message broker. A message broker just receives messages and sends them to all nodes. ...
0
votes
1answer
57 views
Message queues: Cluster of “send mail” and analytics consumers, ensure dispatch to only one mail-consumer and only one analytics-consumer
It seems there's lots of ways to create a task queue where something like a round-robin algo rotate tasks between consumers, but say I have a user queue, and I want to both send a welcome mail and do ...
3
votes
1answer
91 views
Simple messaging pattern to replace shared DB? [closed]
There are two modules A and B using the same DB. A imports B as a JAR. I read that messaging patterns can replace the tightly coupled solution of using a shared DB but I am not sure which message ...
0
votes
1answer
155 views
Eventing solutions for Java legacy applications too old for real JMS queue/topic eventing?
What are some architectural solutions that can mimic eventing for systems (Java) that are too old to implement eventing solutions, such as conventional Queue/Topic based JMS messaging?
I have a ...
19
votes
4answers
2k views
Using flat files vs database/API as a transport between a frontend and backend
I've got an application which has generated a rather heated discussion between a couple of the developers.
Basically, it's split into a web layer and a backend layer. The web layer collects ...
1
vote
0answers
109 views
Guaranteed message sending for messages of different priority for different modules
We have created a message que system using priority messages. The idea behind this is as follows:
We have a list of clients.
All clients know if (and what kind of priority) messages they have ...
2
votes
1answer
445 views
What is an appropriate pattern for handling a queue of network events in C++?
I'm writing an application that sends and receives messages over the network and pushes them into a std::deque queue. and I'm looking for the appropriate programming pattern to handle all of the ...
0
votes
0answers
98 views
Fault-tolerant, distributed, event dispatcher architecture?
I'm building an event handler/dispatcher for my monolith, aiming to slowly replace big parts of it by delegating jobs to other systems via events. For example, sending emails when a customer buys ...
1
vote
0answers
45 views
MVC WebApplication as MQ Subscriber
We have a Web (.Net MVC5) application that we are looking to start offloading some of the heavy processing off the Web Application to a distributed services layer. As part of this process we are ...
5
votes
2answers
572 views
Understanding the Single-Writer Principle
The article on the Single-Writer Principle of the Mechanical Sympathy blog explains how bad queues are (performance-wise), because they need to be able to receive messages from multiple produers, and ...
1
vote
2answers
89 views
How to guarantee web script execution sequence/order or not parallels queues
This is a very simplified version of the problem:
You have an integer saved on a server called "B" and you have an script saved on a server call "A" exposed to the internet. The script pulls the ...
4
votes
1answer
346 views
Is it always the best way to go to queue emails instead of sending them real-time? [duplicate]
If you had to decide between queueing emails and sending them real-time, is it always the best way to go putting them in a queue and sending them one by one instead of instantly as soon as a user ...
2
votes
1answer
81 views
Using short-lived REST requests to obtain messages from a message queue for clients
I would like to put together a system where clients obtain messages from a REST API, but with the producer being able to use the semantics of a message queue: e.g. ordering of messages, a destination ...
2
votes
3answers
366 views
Design pattern for large amounts of overflowing data?
Our current queues publish messages that consumed by 3rd party services with rate limits. Currently the messages are retried with exponential back-off. However there could be cases where data is ...
2
votes
2answers
1k views
Distributed transactions with Kafka
I need to implement a transaction that spans over loosely coupled (SOA+MOM) components.
When a particular event is received, FOO and BAR need to do a transactional operation: that is, both the ...
4
votes
1answer
245 views
Azure ServiceBus Queue: Consumer Application
I am running a .net MVC4 web application. I want to be able to hit an action on a controller, create an email and send it to my service bus queue to be processed (sent).
What are my options as far ...
4
votes
1answer
291 views
Message queue vs database for delayed tasks
I need to build a system that can handle a fairly high amount of delayed tasks (e.g. scheduled emails). For non-delayed tasks I would go for something like RabbitMQ.
But, is it ok to let tasks ...
2
votes
1answer
214 views
How do I handle long running third party calls from backing up my message queue?
Here is my exact scenario.
I must make requests to a third party service
The service takes ~15 seconds to respond
It also has no webhooks or any call back mechanisms
These requests are not made ...
8
votes
1answer
4k views
Does Akka obsolesce JMS/AMQP message brokers? [closed]
I spent the last week deep diving into the Akka docs and finally understand what actor systems are, and the problems that they solve.
My understanding (and experience with) traditional JMS/AMQP ...
2
votes
1answer
265 views
How many Topics (queues) to practically use in an ESB
In all the Service Bus examples and tutorials I've seen, they cover publishing a single message to a single Topic that is picked up by a single listening Subscriber. Whilst this is good for a simple ...
