I am engaging in redesigning a linux c++ server application. The application acts as file relayer in the form of receiving file packets (incuding control packets and data packets) from client A, writing these packets into a local data file then create and update index file, finally start many threads to read the file data and forward these data packets to its receivers such as client B, client C, client D, etc. Flow chart is like: Client A -> Server -> Client B, Client C, Client D, ...
Currently the application is using Producer-Consumer pattern to pass data packets among modules, here are some modules we have: SocketModule(contains a thread for listening and reading incoming data of all sockets), SessionModule(for managing user tcp sessions), FileRelayModule(contains many threads for doing jobs like processing incoming packets and writing into local data file, reading local data files and updating index files and forwarding these data packets to receivers via TCP sockets).
The problem is that its performance is so bad, take an example, it takes about 15 minutes for Client A to send a file of size 200M to Client B, but I did a p2p file sending and receiving between different machines it takes only 20 seconds.
We think the main cause of bad performance is that we store sockets information in a global std::map, then every thread who wants to send out data (to clients) needs to wait for the lock for std::map, thus the application acts like single-threaded application.
As the original design of this application doesn't follow principles of design pattern and this results in many issues, so we want redesign it using MVC pattern, adding roles of thread manager and file manager. The first item I think of is to use a ThreadPool, then dispatch every accepted socket into the thread who has the least clients(sockets number). Is this possible or reasonable for this application? What would you do if you were the designer of this project?
[update]
1. I will check the project and figure out which part is the real bottleneck of
performance
2. No matter which part is the real bottleneck, we need to reconstruct the
application for it to manage the resources such as threads and memory and
be easy to maintain.
I am gonna do things like following:
1. Create many classes: CSessionManager for managing tcp session data (contains
a thread), CFileManager for doing file relaying and managing all data files and index files (contains a ThreadPool with 32 threads of which contains a sockets map, and a listening socket), CFileServer(contains the CSessionManager and CFileManager). Here CFileServer acts as Controller, the CFileManager acts as Model.
2. The CFileManager will create a thread pool with 32 threads initialized at the startup. The CFileServer creates a listening socket and puts every accepted socket to CFileManager where the the new accepted socket is set to a thread in the threadpool by querying who has the least sockets.
[update2]
The threadpool will be designed as following(is it reasonable?):
std::list<CThread*> workers;
//each thread would be
class CThread{
public:
PostMsg(CMessage*);
static void Run();
private:
map<int, socketinfo*> sockets;
std::deque<CMessage*> messageQueue;
};
...
Other objects will call PostMsg to inform message
to this CThread and in the run function:
void CThread::Run(){
....
while(1){
....
//firstly process all messages
//these messages includes adding socket,
//deleting socket, or go offline
CMessage* msg = messageQueue.pop_front;
if (msg != NULL) {
process(msg);
}
//after processing the messages,
//now read/write the sockets data
//set readfd and writefd
ret = select(max, readfd, writefd, NULL, timeout);
//now process readfd and writefd
//read data from readfd, then write to local
//file read data from local file then write
//to wirtefd or post a message to controller
// to remove fd from array.
......
}
}