My server receives requests in one thread and responds in another. I am using thread pools of pthread-s for both receiving and sending. The performance is not that bad, however...
When I am doing the same algorithm in one thread only(MULTITHREAD not defined) so that both recvs and sends are in the same worker thread, my app runs about 10% better.
Strangely, when I am using separate threads(worker and operation) but include printf
in any operation, the performance raises (yes, raises - not declines) some 20%. My printf
s goes over SSH.(*)
(Packet analyses shows some 100 micros of delay of ACKing responses (not separate ACKs). This delay does not happen with the printf
s, neither it happens in a single-thread mode. It seems like SSH triggers something in the TCP stack. )
I tried to use the TCP_QUICKACK option after every recv()
but the performance goes down!(**)
I am using Centos7. The application is very concurrent - it is SMB server.
The algorithm is as follow:
One thread that get the message with
epoll_wait()
and then execute the first worker thread, this thread parse the message and launch the operation thread(create/read/write etc.).
main thread:
static void mainThreadBodySocketTransport(CSTransport * transport)
{
struct epoll_event e;
int res;
while (true)
{
void * context = NULL; /* to pass to thread */
res = epoll_wait(transport->epoll, &e, 1, (int)*60 * 12 * 1000));
if (res == 1)
{
*context = e.data.ptr;
}
/* wrapper to execute the worker thread */
cmThreadPoolExecute(transport->threadPool, context);
}
}
worker thread
static void threadBodySocketTransport(void * context)
{
msgBuffer * message;
message = parseMSG(context);
switch (message->cmd)
{
case WRITE:
doWriteOp(message);
break;
case READ:
doReadOp(message);
break;
case CREATE:
doCreateOp(message);
break;
}
}
void doWriteOp(msgBuffer * message)
{
#ifdef MULTITHREAD
/* wrapper to execute the worker thread */
cmThreadPoolExecute(threadPoolWrite, (void *)message);
#else
/* write to file */
threadBodyWrite(message);
#endif
}
operation thread
static void threadBodyWrite(void * context)
{
msgBuffer * message = (msgBuffer *)context;
int fd;
.
.(get the fd)
.
/* printf("w\n");*/
write(fd, message->write.data , message->write.len);
sendResponse(message->cmd, true);
}
pthread create
void threadStart(SYThread *taskIdPtr, void (*startpoint)(void), NQ_BOOL background)
{
pthread_attr_t attr;
pthread_attr_init(&attr);
pthread_attr_setdetachstate(&attr, PTHREAD_CREATE_DETACHED);
pthread_create(taskIdPtr, &attr, (void * (*)(void *))startpoint, NULL)
pthread_attr_destroy(&attr);
}
Maybe I need to use different thread attribute??
Packet sequence:
(*)ack time - request time = ~50 micros
request
response(builtin ack)
request
request
ack
respons
(**)ack time - request time = ~50 micros
request
ack
response
request
request
ack
respons