Questions tagged [distributed-computing]
The distributed-computing tag has no usage guidance.
153
questions
2
votes
2answers
85 views
Is sequential consistency equivalent to performing memory accesses by a processes in program order and performing each memory access atomically?
In Fundamentals of Parallel Multicore Architecture, by Yan Solihin, p304 defines sequential consistency memory model:
Overall, we can express programmers’ implicit expectation of memory
access ...
4
votes
2answers
103 views
Do coherence and transactional isolation mean the same?
About coherence vs consistency, https://en.wikipedia.org/wiki/Consistency_model says
Coherence deals with maintaining a global order in which writes to a
single location or single variable are seen ...
7
votes
2answers
2k views
How does Elastic Search approach the problem of distributed membership and consensus?
I already understand that Elastic Search is supposed to be deployed in a distributed topology, in that you can have multiple nodes for a cluster of ES instances.
I like the API, and it looks promising....
4
votes
1answer
1k views
Using distributed locks in microservice environment
I'm developing distributed system and trying to use best practices of microservice architecture.
I was faced with a situation when I think I need something like distributed locks. Since I have not so ...
-1
votes
1answer
79 views
Pinning Stateful Tasks in Stateless microservice cluster
We have a microservice whose multiple instances are running for High Availablity and Scaling. But although we have multipe stateless microservice there are some tasks which needs to be singleton.
For ...
0
votes
1answer
66 views
Large file uploading in chunks
How to upload large(mb/gb) of video files from client to app server?
The solution as of now I know is pretty simple and widely used is to break file in chunks at at client side and send http post ...
3
votes
1answer
142 views
Database consistency for a distributed system
I'm working on a phone call monitoring project. The aim is to have one row in the calls table for each ongoing call. Each call instance may be updated by many different servers/threads as more ...
1
vote
2answers
87 views
Multiple sources of truth - Optimistic concurrency & Eventual consistency
I have a couple of web applications that write to their own databases. They also share a few entities, for example, the customer entity. My business case is such that the same field for the same ...
4
votes
3answers
278 views
How can adding redunancy adversely affect performance
I have been reading about system design and was going through this popular article - A Word on Scalability
The author here says:
An always-on service is said to be scalable if adding resources to
...
1
vote
1answer
69 views
Ordering of analytical events
My team is working on an analytics system for the web app. The web app has a service-oriented architecture, and all events are gathered in one centralized analytics service. The system is sending ...
2
votes
1answer
52 views
Manager node makes HTTP request to self?
I am working on a distributed application that uses the Manager Worker pattern. The application's manager node submits work to worker nodes through a WorkerNode object (allowing WorkerNode to be ...
0
votes
1answer
92 views
Implementing the microservice pattern
We are developing a system that runs a certain kind of simulation for our customers:
Every simulation runs on it's own server.
We typically have a two digit number of simulations running.
A client ...
0
votes
2answers
54 views
Cancelling a background service from another microservice in load balanced environment
I have 2 microservices [A], and [B]. These microservices run in a load balanced environment and there may be N instances if each microservice running. The microservices communicate via a message ...
1
vote
2answers
613 views
How do atomic updates work at scale?
I am working on a weird ID system where I generate all the IDs in advance and need to atomically pop() one from the database, so no ID is read twice at the same time (and so used more than once). This ...
0
votes
1answer
434 views
Building a serverless p2p CMS/System
I'm thinking about building an Open Source, serverless, offline-replicated p2p CMS, but I'm concerned about it really working in a real environment. How would you go about doing that, reducing the ...
0
votes
1answer
77 views
what is the difference between “distributed data processing” and “distributed computing”?
While going through learning hadoop and spark, I came across "distributed data processing" and "distributed computing".
Could you let me know if they both are same or referring to different concepts?
7
votes
1answer
588 views
How to prevent concurrency problems when using the repository pattern?
Considering that:
when using the repository pattern you deal with entities - those are the atomic units you persist (regardless of Hibernate or "manually")
when changing an entity you save it as a ...
-4
votes
1answer
290 views
What difference and relation are between fault tolerance and (high) availability?
From Coulouris' Distributed Systems 5ed
Chapter 18 Replication
18.1 Introduction
Increased availability: Users require services to be highly available. That is, the proportion ...
-3
votes
2answers
250 views
What is the difference between masking and tolerating failures?
Distributed Systems 5ed by Coulouris says on p21-22
1.5.5 Failure handling
Detecting failures: Some failures can be detected. For example, checksums can be used to detect corrupted data in a ...
0
votes
0answers
44 views
Effective data/rows queuing out of database
A component in a system (called Notifier) inserts hundreds of messages ~300 (SMSs) per second in a table in MSSQL database with PENDING status by flagging a column called status with int value 0.
...
-2
votes
1answer
94 views
In the distributed computing problem known as the “Byzantine fault”, what differences and relations are between commander, lieutenant, and general? [closed]
Distributed systems by Coulouris says
The Byzantine generals problem •
In the informal statement of the Byzantine generals problem [Lamport et al. 1982], three or more generals are to agree to ...
1
vote
1answer
480 views
Does the producer-consumer problem appear in both shared memory and distributed memory architectures?
I saw some books on operating system concepts mention the producer-consumer problem in the context of synchronizing concurrent accesses to shared resources. All seem to be in shared memory ...
2
votes
1answer
226 views
Do persistent/transient communication and temporal decoupling/coupling mean the same?
In Distributed Systems by Tanenbaum, p67 says
CHAPTER 2. ARCHITECTURES
2.1. ARCHITECTURAL STYLES
Cabri et al. [2000] provide a taxonomy of coordination models that can
be applied ...
-2
votes
1answer
350 views
How to understand two-tier and three-tier solutions for web applications?
From Distributed Systems by Coulouris:
Tiered architectures are complementary to layering. Whereas
layering deals with the vertical organization of services into layers
of abstraction, tiering ...
1
vote
3answers
164 views
Is shared disk architecture scaling up or scaling out?
In Design Data Intensive Applications,
If all you need is to scale to higher load, the simplest approach is to buy a more powerful machine (sometimes called vertical scaling or scaling up). ...
1
vote
1answer
106 views
Designing frontend experience to support backend async CQRS operations
I'm developing a microservice with CQRS and Event Sourcing. When an event is saved to the event store, the service currently also saves the updated aggregate root as a JSON object in a separate table. ...
3
votes
1answer
697 views
DynamoDB - Event Store on AWS
I'm designing an Event Store on AWS and I chose DynamoDB because it seemed the best option. My design seems to be quite good, but I'm facing some issues that I can't solve.
The Design
Events
Events ...
1
vote
2answers
143 views
Develop a distributed pointer in C++
I s it possible to develop a distributed memory pointer in C++ by using operator overload?
The idea would be that you pass ip address and port to the overloaded pointer *, &, and the pointer ...
1
vote
0answers
66 views
Consistency and Availability in distributed hashing Key value store
During the last semester of my college, we were asked to read the (chord paper) and implement it without fault tolerance. We built it with the following feature:
A new node can join.
A client can ...
2
votes
0answers
61 views
Distributing tasks to several clients
I describe a very very simple scenario. I try to develop a system where many machines can help another one (server). I tried RabbitMQ and Redis, but both don't fulfil all requirements.
I have 1-...
-1
votes
1answer
149 views
How do decentralised systems deal with all nodes being killed
In relation to DHT's (distributed hash tables), how do decentralised systems deal with all nodes disconnecting and thus having its routing meta data invalidated? i.e with the implementation of DHT's ...
3
votes
0answers
273 views
Is the concept of Rx debounce possible in a distributed environment?
The issue
I am building a server-side system that handles streams of time-series events from multiple users and I'd like to perform an action after a certain quiet period has elapsed in the event ...
1
vote
1answer
100 views
Would this system be considered decentralized or distributed?
I am currently working on a report regarding a matrix multiplication distribution system I have been working on. But after reading a lot of somewhat contradictory sources online I have become in doubt ...
1
vote
1answer
297 views
How to design a high-scale, reliable, distributed and periodic task (cronjob) execution service?
Assume you are tasked with implementing a "cronjob as a service" where you are supposed to be able to run potentially millions of periodical tasks (to simplify, making HTTP requests to URLs) with ...
0
votes
1answer
509 views
Distribute jobs evenly among a set of servers
I have the need to distribute a set of long running jobs across a cluster of containers in ECS. These jobs essentially would need to open a socket connection with a remote server and begin streaming ...
-2
votes
1answer
603 views
How to serve the front-end in a Spring microservice architecture?
I want to create the following distributed system: Spring back-end microservices containing the domain logic, a UAA (authentication) service, a Eureka service registry/discovery, a Spring Cloud Config ...
3
votes
2answers
1k views
Accepting the UUID collision risk based on number of clients
After reading some questions about the probability of UUID collisions it seems like collisions although unlikely, are still possible and a conflict solution is still needed. Therefore I am wondering ...
7
votes
1answer
266 views
Open Grid Engine or Akka/Something more fault tolerant?
My use case is that I have a pipeline of independent, stand alone programs, that I want to execute in a certain order on specific pieces of data that are output from previous pipeline stages.
The ...
3
votes
2answers
409 views
Architecting a distributed file processing system with leadership election
I'm in the planning phases of trying to build a distributed file processing system in Java and I'm looking for feedback and advice:
Problem : There are a large number of files continuously posted on ...
1
vote
0answers
43 views
Byzantine Failures - Client Voter and Replica on Same Processor
I need help understanding the following text from section 4.2 on this paper: Implementing Fault-Tolerant Services Using the State Machine
Approach: A Tutorial
Dependent-Failures Output ...
4
votes
3answers
324 views
Distributed database algorithm
I have a need for a distributed data store, where existing solutions may not work as the computers these will be running on will be extremely resource limited, for instance 64-128MB RAM. Plus, as a ...
12
votes
2answers
3k views
Patterns for maintaining consistency in a distributed, event sourced system?
I've been reading on event sourcing lately and really like the ideas behind it but am stuck with the following problem.
Let's say you have N concurrent processes which receive commands (e.g. web ...
16
votes
8answers
3k views
What exactly is distributed computing? [closed]
What exactly constitutes distributed computing, and how does it differ from parallelized / concurrent computing?
Does the use of mutexes and semaphores in multiple parallel threads trying to ...
5
votes
2answers
643 views
How can we keep sight of business flows in event driven architectures?
I am planning to set up an event driven architecture using Spring Boot apps that publish and read messages from a Kafka broker.
Let's suppose it were an e-commerce application with the usual events (...
4
votes
1answer
164 views
Is the actor model suited for an application dealing with elapsed time?
I need to create a distributed application with timers / periodically checking two timestamps for elapsed time.
Is it possible to achieve following behaviour with the actor model:
Spawn Child actor
...
2
votes
2answers
340 views
Distributed training of many small ML models
I have a data science application that involves training tens of thousands of small individual Gaussian models. By "small", I mean that any individual model can easily be trained on one of our worker ...
2
votes
0answers
158 views
Testing an algorithm for safety in mutual exclusion
I am attempting to write a code to test Lamport's Mutual Exclusion algorithm for safety as a correctness measure. I am running the alogrithm on a single core cpu machine with multiple processes ...
1
vote
2answers
271 views
How to avoid clashes in distributed computing when polling for work to do?
A big data job is split up into X partitions. The partitions are stored in a database. Status on each partition is also stored in the database and is used to ensure that each partition is only ...
-2
votes
1answer
579 views
How do I implement autocomplete functionality for a big website? [closed]
I am looking to implement a custom autocomplete functionality for a travel website with huge amounts of data. I am thinking of implementing it using a trie. But how do I handle it if the data cannot ...
1
vote
1answer
184 views
#Apache-flink: Stream processing or Batch processing using Flink
I am tasked with redesigning an existing catalog processor and the requirement goes as below
Requirement
I have 5 to 10 vendors(each vendor can have multiple stores) who would provide me with 'XML' ...
