Tagged Questions
0
votes
1answer
23 views
distributed processing api like multiprocessing in python
I am considering the implementation of a light weight distributed processing framework, which has similar api with multiprocessing in Python. An example is as follow:
#! /usr/bin/env python
#
# ...
0
votes
1answer
47 views
python: how to run local code on cluster of remote servers? [closed]
I'm basically looking for something like picloud.com (shut down last year) but that is able to run on my own cluster of servers.
I'd imagine some kind of server running on each of the grunt work ...
1
vote
2answers
49 views
python using multiprocessing pool.map for distributed computing
The next code works to execute "someprogram" in parallel for the 50 inputfilenames, using a pool of 5 workers. How is it possible to use 5 workers on
this computer, and 7 on another computer, using ...
1
vote
1answer
345 views
Using Titan Graph database from Python
I am attempting to model a network topology using Titan Graph DB.I want to specify the topology from a python application.
I have a java interface file that uses tinkertop frames annotation.An ...
1
vote
1answer
44 views
Celery not using all concurrent slots
I have a Celery cluster made up of machines with 8-core processors. Each machine has one worker that is set to a concurrency factor of 8 (-c8).
I often see nodes with a lot of reserved tasks, but ...
1
vote
1answer
115 views
Framework for message-passing distributed algorithms
I've recently taken a course in distributed computing where we learned about distributed algorithms for problems such as consensus, leader election, etc, and now I would like to implement some of ...
1
vote
2answers
43 views
Loading python modules through a computing cluster
I have an account to a computing cluster that uses Scientific Linux. Of course I only have user access. I'm working with python and I need to run python scripts, so I need to import some python ...
1
vote
4answers
75 views
How to get N computers in a network to start an activity simultaneously?
I'm trying to create a distributed application that requires all the computers in a network to perform an event simultaneously.
Let's say we have two arrays of equal length.
labels[] = ["label1", ...
0
votes
1answer
43 views
How can I share a class definition over multiple server instances?
The solution I am working on consists of modules that are decoupled across multiple virtual server instances. All of the modules require the exact same DTO (Data transfer object) classes. Currently I ...
1
vote
0answers
40 views
Extending IPython task scheduler for custom load balancing
I want to be able to implement my own load-calculating scheme for IPython parallel's LoadBalancedView, which uses a TaskScheduler.
I'm piecing through the source, and this looks like a likely ...
1
vote
2answers
109 views
Token passing in ring algorithm in python
I am trying to implement a ring algorithm for token passing. The algorithm will be later tested using different nodes in a cluster. I am new to python and network programming. I want to start simple ...
1
vote
1answer
232 views
Python MapReduce on Sun Grid Engine
I am relatively new to distributed computing, so forgive me if I misunderstand some of the basic concepts here. I am looking for a (preferably) Python-based alternative to Hadoop for processing large ...
1
vote
1answer
32 views
MPI distributed, unordered work
I would like to write a MPI program where the master thread continuously submits new job to the workers (i.e. not just at the start, like in the MapReduce pattern).
Initially, lets say, I submit 100 ...
1
vote
1answer
228 views
ZeroRPC pub/sub aggregate results
I am designing a simple distributed database in python. I consider to implement a communication layer with ZeroRPC. The key lookup is implemented by the DHT protocol with req/rep pattern. However, I ...
1
vote
1answer
109 views
RPC Protocols comparison
I have to select a protocol/technology to use for communicating a client-server architecture, with support both for Python and C. The main requirements are:
Symmetrical communication in between ...
8
votes
1answer
997 views
Clarification of use-cases for Hadoop versus RabbitMQ+Celery
I know that there are similar questions to this, such as:
Pros and cons of celery vs disco vs hadoop vs other distributed computing packages
Differentiate celery, kombu, PyAMQP and RabbitMQ/ironMQ
...
0
votes
3answers
67 views
Multiply read text files files don't seem to exist?
I'm running batch jobs on a RHEL5 lustre filesystem. Many jobs (13k) read the same text file, which is used to direct each job to a different dataset. The code looks like this:
with ...
0
votes
0answers
94 views
python job queueing where worker nodes must ssh into controller, not the other way around
There are lots of distributed job queuing libraries for Python (see http://wiki.python.org/moin/ParallelProcessing). All of the ones I've looked at so far seem to share a design model where a ...
-1
votes
2answers
61 views
Environment to simulate classic distributed computation model on single machine [closed]
I'm searching for a tool to simulate the classic distributed computation model on a single machine to implement several algorithms in the purpose of the paper that I'm working on. Thus, performance ...
1
vote
1answer
90 views
Any python library for parallel and distributed tasks?
I am looking for a Python library that can distribute the tasks across a few servers. The task would be similar to what can be parallelized by the subprocess library in a single machine.
I know that ...
3
votes
2answers
189 views
Running a Python Application on BOINC
I am currently in the process of setting up my first BOINC project and have configured the server and site to the point where the next step is my application.
I have no experience in C programming ...
1
vote
0answers
66 views
Script for feeding data into Yahoo S4 (Now under Apache)
I know that we have to feed data stream into s4.
My question is that the script generating the input stream (that has to be feeded into s4) can be in python or it needs to be in Java only, since S4 ...
1
vote
4answers
220 views
data bridge between Java and Python daemons
I have two background processes running on linux machine. One is Java and second one is in Python. What would be most efficient way to exchange data between these two apps ? I am talking about text / ...
0
votes
1answer
88 views
Best way to distributed massive ammounts on zip files
I haven't much experience in distributed computing, but I would like to create a script within python to distribute unzip jobs centrally to x amount of servers, saving me from the hassle of routinely ...
2
votes
1answer
204 views
Updating shared read-only data with Python multiprocessing
I am attempting to use Python's multiprocessing library to experiment with distributed neural networks. At the moment, I have it set up so that a server process creates the neural network and chunks ...
0
votes
1answer
269 views
“Global array” parallel programming on distributed memory clusters with python
I am looking for a python library which extends the functionality of numpy to operations on a distributed memory cluster: i.e. "a parallel programming model in which the programmer views an array as a ...
1
vote
1answer
97 views
Query userproperty attribute user_id [closed]
I have +- the next model
class Users(db.Model):
user = db.UserProperty()
date = db.DateProperty(auto_now_add=True)
and, I need search users by user_id
q = Users.all()
q.filter("user.user_id =", ...
google-app-engine
2
votes
1answer
184 views
Durable architecture in Python distributed application
Wondering about durable architectures for distributed Python applications.
This question I asked before should provide a little guidance about the sort of application it is. We would like to have the ...
1
vote
1answer
136 views
Is using Celery for task management in cluster good idea?
I'm going to use Celery to manage tasks in cluster. There will be one master server and some worker servers. Master sends tasks to the worker servers (any number) and gets the result. Task state ...
0
votes
1answer
182 views
Monitor python scrapper programs on multiple Amazon EC2 servers with a single web interface written in Django
I have a web-scrapper (command-line scripts) written in Python that run on 4-5 Amazon-EC2 instances.
What i do is place the copy of these python scripts in these EC2 servers and run them.
So the ...
1
vote
2answers
301 views
How to stop python multiprocessing server when processes are done
I'm trying to use python's multiprocessing module to run a distributed task over a couple of machines, and I've been using this blog post as a reference.
However, this post's task uses a job queue, ...
0
votes
1answer
337 views
How to structure code that distributes jobs to threads/nodes in Python?
I have python code that takes a bunch of tasks and distributes them to either different threads or different nodes on a cluster. I always end up writing a main script driver.py, that takes two command ...
2
votes
1answer
279 views
Can Redis or Memcached be used to “lock” a resource such as a blobstore like S3?
Would like to "lock" an external resource, in this case it will be a "blob" stored on rackspace cloud servers, similar so Amazon S3. (Using Python)
Would like a race condition to be handled like ...
0
votes
1answer
60 views
decorators to make a sequential set of tasks in Python?
I am writing a program that executes a set of tasks sequentially. Each task is a function that outputs a new file, but any given task should not execute if the filename already exists. I find myself ...
1
vote
1answer
286 views
Celery: long dedicated monolithic task vs short multiple tasks
In my solution I use distributed tasks to monitor hardware instances for a period of time (say, 10 minutes). I have to do some stuff when:
I start this monitoring session
I finish this monitoring ...
0
votes
2answers
183 views
Python - Calculate field density in csv/tsv file
Hi suppose if i have a tab seperated file like this (each field separated by tab spaces):
Name ID Country GPA
Tom id1 USA 3.4
Jon id2 Canada
Amy UK ...
5
votes
2answers
723 views
Where to begin with Distributed Computing / Parallel Processing? (Python / C) [closed]
I'm currently investigating topics for graduate studies in Computer Science and I've found a relatively large area of interest, Distributed Computing, that I'd like to get more information on. There ...
3
votes
2answers
128 views
Is there a Python equivalent of Ruby's officer gem?
Is there a Python equivalent to Ruby's officer gem? It is a distributed locking server, basically allows you to have locks across a network rather than just between threads.
I've been looking at ...
1
vote
1answer
665 views
Data analysis using MapReduce in MongoDb vs a Distributed Queue using Celery & RabbitMq
I am currently working on a project which involves performing a lot of statistical calculations on many relatively small datasets. Some of these calculations are as simple as computing a moving ...
mongodb
1
vote
2answers
483 views
What is the right ZMQ architecture for a webserver sending fire-and-forget tasks to a bunch of webservers?
I have a website which sends out heavy processing tasks to a worker server. Right now, there is only one worker server however in the future more will be added. These jobs are quite time-consuming ...
7
votes
2answers
2k views
Connect two TCP sockets without defining client/server role [closed]
Question
I want to connect two processes via TCP but I don't want have to specify which of them is the server and which is the client but they know the IP and host of each other. They should decide ...
7
votes
2answers
453 views
Implementing distributed lock using files
I have a network drive (Z:\) which is shared by multiple Windows computers. Is it possible to implement a cross-machine lock by simply creating/deleting files on this network drive?
For example, two ...
0
votes
2answers
262 views
Celery appropriate for this task?
This query is related to my Google Summer of Code project. Basically, I have a master node which delegates build tasks to a number of specified image building nodes. Right now, I do it using a manual, ...
1
vote
1answer
153 views
Distributed server model
Lets say I have 100 servers each running a daemon - lets call it server - that server is responsible for spawning a thread for each user of this particular service (lets say 1000 threads per server). ...
1
vote
2answers
131 views
Any Python solution for having distributed processes interact with relational databases in the most decoupled way possible?
Let's say I have Python process 1 on machine 1 and Python process 2 on machine 2. Both processes are the same and process data sent by a load balancer.
Both processes need to interact with a database ...
1
vote
1answer
101 views
distributed-computing library in the standard library of python [closed]
I want to implement a program that will be used by a lot of co-workers, the problem is that I will use sqlserver and pyodbc in my program but in the same time I must not oblige the others who want to ...
2
votes
2answers
214 views
I am interested in disproving some graph theory conjectures in python, what is the most efficient library/server set up to use?
I am interested in implementing and running some heavy graph-theory algorithms for the purpose of (hopefully) finding counterexamples for some conjecture.
What is the most efficient libraries, server ...
5
votes
1answer
1k views
Why does this Python 0MQ script for distributed computing hang at a fixed input size?
I recently started learning 0MQ. Earlier today, I ran into a blog, Python Multiprocessing with ZeroMQ. It talked about the ventilator pattern in the 0MQ Guide that I read about, so I decided to give ...
1
vote
1answer
316 views
Implementing recursion with a deep copy
How can I implement recursion in a deep copy function object? This is the relevant code (if you want more then please ask):
PS: I would like the recursion to iterate through a filtered list of ...
3
votes
1answer
196 views
How to secure a Python distributed computing layer
These modules are designed to facilitate a layer of computational capacity across multiple computers. They are by no means ready for publication, nor has the required funding for hardware arrived. ...
