We don't use k8s exactly for this reason. We deploy three docker containers using docker-compose.

Does this mean tinode shall not be deployed on Kubernetes? I believe, the application should not restrict the deployment infrastructure. Due to increased adoption of Kubernetes, it would be better to publish the way to deploy on Kubernetes, Possible ways of cluster discovery

1. Can use service dns to discover all the IP addresses which serving at a point in time.
2. A discovery announcer agent can be used as a side can which frequent;y poll either service or pods for getting all the cluster nodes which are not joined the cluster and can add to all the cluster.
3. A state can be maintained separately using things like consul or etcd and can figureout cluster using the state from them.

As I wrote earlier, deploying Tinode with k8s makes no sense to me because k8s expects stateless identical pods. K8s is a wrong tool for the job. But of course you are welcome to deploy it with k8s if you see a use case for yourself.

1. This is true but not relevant.
2. Cluster nodes must be configured ahead of time to discover each other. Tinode is not stateless like a web server.
3. I don't think so.

@or-else we are going to use tinode soon. And I'd like to deploy it in k8s. I'm going to use external DB for storage, s3 for assets. what other stateful stuff do you have that you say it's not possible to deploy it in k8s?

If you are deploying a cluster then every node maintains a state

Each node is different from every other cluster node. Like when you run a mysql server, you don't deploy multiple pods with the same database, right? Same here.

If you are deploying a standalone server, then you can use k8s to deploy a single instance of it.

@or-else I found a guide here that tells that we need names for each node.
My suggestion is to deploy it using k8s stateful sets with fixed names.

Sure, please do it and share your experience.

@or-else sure, I will. Does tinode have something stored in FS so I need put it to volume?

If you use s3 for user-generated content, then the only thing it reads from the file system is static config (read-only).

1. We will be using External storage for data
2. s3 for asset

So why do we really need even stateful sets, if we can manage the cluster to work with auto-discovery of nodes?

Actually this means tinode is not stateful by design. Can you make correct if is m missing something? As per my experiments on configs, only the node hostnames and name for it is what changes between config also. Am I missing something?

@or-else Thank you for reopening the ticket btw. More than an error and trial I am looking for the design paradigm that causing us to not run it as stateless? Given that stateless would be easy to maintain, Can you call out all the possible problems while running it as stateless, that you are aware of.

So why do we really need even stateful sets, if we can manage the cluster to work with auto-discovery of nodes?

Because data is sharded between nodes. Each node serves a shard of data. When user connects to a node N and subscribes to topic T, T may be located at another node, say node M. So, node N will act as a proxy for node M.

Actually this means tinode is not stateful by design

I'm not sure what meaning you assign to this statement.

Because data is sharded between nodes. Each node serves a shard of data. When user connects to a node N and subscribes to topic T, T may be located at another node, say node M. So, node N will act as a proxy for node M.

So even when we use external data storage like Postgres, the in-flight data sharded between nodes? I am missing something on the internal architecture, I guess.

I also worried, what happens in case of a node failure. If the nodes are maintaining shards of data, (nowhere mentioned replication or leader for the topics) how the system handles high availability of topics?

So even when we use external data storage like Postgres

DB is used as dumb storage. All the logic is in the code, all live topics, sessions are in memory.

I also worried, what happens in case of a node failure

Cluster reconfigures itself, reshards the data and continues to run.

Then a stateful set should work well, let me try it.