utilizing more than one computer, connected to each other with a communication link to accomplish a common task.
5
votes
0answers
167 views
Bad version or endian-key in MATLAB parfor?
I am doing parallel computations with MATALB parfor. The code structure looks pretty much like
%%% assess fitness %%%
% save communication overheads
bitmaps = pop(1, new_indi_idices);
porosities = ...
4
votes
0answers
93 views
Are virtually all major distributed computing projects attempting to solve problems in NP?
Here's a huge list of distributed computing projects: http://distributedcomputing.info/projects.html
After a quick skim, I couldn't find any projects which weren't attempting to solve problems in NP ...
2
votes
0answers
63 views
Real world example of Apache Helix, Zookeeper, Mesos and Erlang?
I am new in
Apache ZooKeeper : ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Apache ...
2
votes
0answers
119 views
How to implement distributed processing
I have requests coming in for different samples (s1, s2, ..) that need to be processed in a linear fashion (i.e. only one say s1-request at the time can be processed). I have N-number of worker ...
2
votes
0answers
50 views
Rollback recovery for webservices
Im trying to do a Rollback for non-deterministic events(eg: receiving requests to a webservice) There are two approaches i have in my mind, One is checkpoint based, another one is log based. I chose ...
2
votes
0answers
113 views
WCF multiple instances of a service, 1 per thread
Im trying to set up a distributed computing environment using WCF. I have a large amount of data to process, which i have managed to split into discrete blocks.
Is it possible to create multiple ...
2
votes
0answers
318 views
Zookeeper priority queue
My problem description is follows:
I have n state based database infinite crawlers:
Currently how it is happening:
We are using single machine for crawling.
We have three level of priority queue. ...
2
votes
0answers
174 views
Windows RPC vs XML-RPC
Is there any benchmark about encoding/decoding certain common typed data in Microsoft RPC NDR engine (DCE 1.1) in comparison with that in XML-RPC-C/C++ in the de-facto C/C++ implementation in XML-RPC?
...
2
votes
0answers
67 views
Managing large objects across a cluster.
What kind of architecture should I use for the app below?
In our SAAS app each customer gets a directory of files. These files can be quite large, and need to be read and written to very quickly. ...
2
votes
0answers
60 views
RPC and shared persistent data space
I have a homework question that asks about an RPC system if we're using a shared persistent data space. Would having a shared persistent data space even change anything about the way you build an RPC ...
2
votes
0answers
297 views
Distributed algorithm for Minimum-Weight Spanning Trees
Does anyone know if there is available somewhere a Java implementation of the Gallagher-Humblet-Spira (GHS) algorithm?
2
votes
0answers
78 views
Good Open source avenues to work on OS / low level distributed system stuff
Or rather, if I'm reading "Understanding the Linux Kernel" and more of Linux/OS level or more low level dist. computing stuff, what kind of open source projects are worthing taking a look at - except ...
1
vote
0answers
13 views
Java client for XtreemFS
I am building a Java web application which will require storing and retrieving large files. I would like to be able to scale the application for the future so I was planning on using XtreemFS ...
1
vote
0answers
21 views
What difference is between monitoring and profiling?
What difference is between monitoring and profiling? Why some people say that tools for application monitoring shall not be used for profiling?
1
vote
0answers
29 views
Are there systems that are put on the cluster nodes and find each other by using UDP broadcast?
I want to automate the compilation of a list of IP addresses on which I can run the MPI program. I.e. get a list of all available servers automatically.
Are there cluster management systems that are ...
1
vote
0answers
16 views
Parallel processing in condor
I have a java program that will process 800 images.
I decided to use Condor as a platform for distributed computing, aiming that I can divide those images onto available nodes -> get processed -> ...
1
vote
0answers
46 views
How to edit matlabpool cluster configuration without using GUI
I am trying to edit the configuration properties of matlabpool on a cluster. I want to edit the ppn/nodes/walltime using a terminal because the GUI I am running is pretty slow. I know how to set the ...
1
vote
0answers
42 views
Extending IPython task scheduler for custom load balancing
I want to be able to implement my own load-calculating scheme for IPython parallel's LoadBalancedView, which uses a TaskScheduler.
I'm piecing through the source, and this looks like a likely ...
1
vote
0answers
72 views
In zookeeper distributed file system what to store in Zookeeper?
I'm trying to implement a file system using zookeeper with a mongo backend to store the data.
I'm trying to wrap my head around what all should be stored in zookeeper. ACL's ,file metadata, ...
1
vote
0answers
170 views
Looking for a WebCrawler API in Java
I am looking for a webcrawler in Java. I would like to modify the API in a way that it would basically have a distributed approach rather than working on a single server. I will use this to crawl a ...
1
vote
0answers
218 views
Structure of a multiplayer card game (Hearts) with Java RMI
I'm trying to figure out what would be the best practice for implementing a "Hearts" multiplayer game with Java RMI.
Here's what I thought so far:
A remote interface with the methods for the common ...
1
vote
0answers
66 views
Script for feeding data into Yahoo S4 (Now under Apache)
I know that we have to feed data stream into s4.
My question is that the script generating the input stream (that has to be feeded into s4) can be in python or it needs to be in Java only, since S4 ...
1
vote
0answers
95 views
Which are the Best Resources for the Gnutella P2P protocol?
I'm looking for using the Gnutella P2P protocol for something other than file-sharing. I want to implement distributed processing using it. For that, I need to learn it in detail.
Are there any good ...
1
vote
0answers
45 views
Datastore replication: help characterize legacy system topology
A new system has been built to gradually assume the responsibilities of the legacy one. Both collaborate in production.
To facilitate this, they share a replicated domain object model. Both use ...
1
vote
0answers
137 views
file sharing web site in asp.net like dropbox , currently all uploaded data stored on my machine but i want to store that on multiple machine?
i created web site and data base for that and works fine on single machine but now transform this on 2 machine in which website running on my PC but when end user upload the files it store on another ...
1
vote
0answers
77 views
Distributed computing using Apache with mod_pywebsocket
I'm quite new in Apache so please be patient.
I'm trying to write distributed application which uses WebSocket protocol to communicate with each node.
What is important it must work in client - ...
1
vote
0answers
26 views
Storing associations to remote object using URI's
While working on a new application I came across a new, for me, scenario. Our identity system is a stand alone application with it's own data store only accessible though REST APIs. This is the theme ...
1
vote
0answers
139 views
Distributed Multi-resource Pessimistic Locking Algorithm
I'm implementing pessimistic locking for a distributed key-value store. I have a rock-solid atomic compare-exchange (and increment and decrement) operation for implementing the locks. There's also ...
1
vote
0answers
54 views
Are there any overlay network libraries that DONT depend on rendevous nodes?
I have a (single-core) application that will run on several batch systems each consisting of many (1000s) of nodes. Though each instance will be separately scheduled, there's a lot of things I could ...
1
vote
0answers
348 views
Messaging, distributed task scheduling, cache replication framework (in Java)
I need a framework for clustering: messaging, cluster join/leave, distributed task scheduling and cache replication. Preferably all handled under the same framework, with initializition of IPs from ...
1
vote
0answers
90 views
How to auto-reconnect failed clients to another server in the server-cluster?
We have a client/server distributed application where server is written in .Net and clients could be in c++, php, .net, java, python etc. All the clients in the environment are connected to atmost 1 ...
1
vote
0answers
191 views
Real world high insert (more importantly updates that resize the value) nosql
I am working on a an IR problem that can be represented with two key-value tables.
Table Q : has a fixed size determined at time of insertion, updates to the table increment counters, the size can ...
1
vote
0answers
249 views
How to create simpile TCP stream proxy with 0MQ?
So we have N clients that want to have real TCP connection to a server. We have a server-Gateway TCP proxy we want to implement with 0MQ. And we have P real servers in a destributed system that has ...
0
votes
0answers
21 views
What technologies should I use for distributed computing with GPUs?
I have a three-plus servers with GPUs that I want to do some digital signals processing on. I plan on using CUDA to crunch the FFTs and do other DSP operations. In my past I have only done thick ...
0
votes
0answers
27 views
Best protocol for binary data in distributed network
I need to create a distributed computing network with simple following features:
receive client request containing some mime type (custom) and specific uri
calculate according to request binary data
...
0
votes
0answers
13 views
multithredin using dmtcp in python
I like to control the checkpointing process from inside the python code. DMTCP is a a framework for checkpointing in distributed systems. It allows checkpointing python [rograms. But I wish to ...
0
votes
0answers
15 views
Splitting data on Spark
I would like to train many decision trees on different parts of a large dataset. Each part ought to fit comfortably in memory. Is it possible in Spark to do something along the lines of (supposing I ...
0
votes
0answers
14 views
Framework similar to DMTCP for windows
DMTCP (Distributed MultiThreaded Checkpointing) is a tool to transparently checkpoint the state of multiple simultaneous applications, including multi-threaded and distributed applications. It is only ...
0
votes
0answers
25 views
Is there a way to use Cassandra to detect the topology of the network?
It's my understanding that Cassandra's snitch protocol enables all nodes to maintain a picture of the network topology (i.e., reachability) for all nodes in the cluster. The application I'm working on ...
0
votes
0answers
19 views
Apache Spark Streaming Change Time WIndow While Running
When we are running Apache Spark, is there a way to change the window length and slide interval while the system is running? If there is such a way, how would we do that? Thanks!
0
votes
0answers
13 views
ZooKeeper: Difference between WriteLock and Curator's InterProcessMutex?
What exactly is the difference between WriteLock recipe included with ZooKeeper and Curator's InterProcessMutex
Is it the same functionality (distributed locking) just using different APIs? Or are ...
0
votes
0answers
8 views
Ensuring a background job is only ran once over distributed architecture
I'm building a game in Rails 4 that will require multiple servers to be running in parallel. The database will contain records which will need to be updated regularly (every couple of hours).
I ...
0
votes
0answers
36 views
Find out the Number of Active Mappes and Reducers in Hadoop at a given Instant of Time
Is it Possible to know the number of Mappers and Reducers that are actively running at any given instant.
My Research work requires me to get some Performance metrics while performing Map-Reduce ...
0
votes
0answers
40 views
Corba ORBD error
I am developing a simple Remote Administration Tool using CORBA. This is my server code.
/*
* To change this template, choose Tools | Templates
* and open the template in the editor.
*/
package ...
0
votes
0answers
40 views
Hazelcast — Create a “coordinator” service
I'd like to implement a "coordinator" service in Hazelcast, with following requirements:
A single instance would exist in a cluster.
It would maintain a "coordination state", modify this state via ...
0
votes
0answers
14 views
Mobile P2P Simulator?
This question may not be a program related one. But for my final year project I need to find a good Mobile P2P simulators. Can anyone suggest me some good simulators for mobile P2P ?
0
votes
0answers
25 views
Why is the “initial value” not to be used on FLP impossibility?
I'm confused about the "initial value"(input register x) of FLP impossibility result, it did not seem to make any effort for consistency excepting for identifying different process state. However, the ...
0
votes
0answers
23 views
How do you achieve load balancing between n servers with i requests?
Describe how in the general case of n servers, each capable of serving s requests per second, you would allocate x requests from the queue to achieve load balancing. State any assumptions you make. ...
0
votes
0answers
20 views
What's the difference between sequential and causal consistency?
I am trying to solve this past paper, but I am not sure how to tackle this problem on both sides. It would be helpful if someone could show me how sequential and causal work step by step.
Further, ...
0
votes
0answers
40 views
Overcoming “Out of Memory” Errors in MATLAB Distributed Applications
Context: I'm running a job (consisting of 1 to 50 tasks) on a cluster, waiting for it to finish, and then pulling the results using:
outputArgs = getAllOutputArguements(myJob);
The Problem: The ...
