Can’t get k8s deployment up and running

[Supporterino]

Hi,
my plan was do deploy and anytype stack to my local k8s cluster for use with my devices across my family. I used the dockercompose and ansible repos to build a kustomize app. In the namespace I created the requirements of minio,mongodb and redis. All is working fine so far. All services are up and running and not logging any errors just the regular delete schedule. But once I create an account and the initial sync tries to run I get "cid not found" error in the file-node and it gets stuck forever. Any ideas what is wrong with my filenode.

[fb929]

I think it same problem - anyproto/any-sync-dockercompose#10

[Supporterino]

All server are exposed with their respective IP adresses which are set inside the clients config. Or do you mean that the ips inside client and server confs need to be the same. If that is the case I can try that. My load balancer can handle that.

[fb929]

configuration distributed from mongo, example:

rs0 [direct: secondary] coordinator>   db.getMongo().setReadPref('primaryPreferred'); db.nodeConf.find().sort( { _id: -1 } ).limit(1)
[
  {
    _id: ObjectId("65a58d038f0e2ecc5a1f3192"),
    networkId: 'N6JipqvGFPiy3bNvf9hggJTeQzac6cHHPSJCVVFTaif9Vnf3',
    nodes: [
      {
        peerId: '12D3KooWHMZHedpKyUxbwFsR2Qt8C2K95XRg3xJ4KnKGDTH61hgL',
        addresses: [ 'any-sync-node-1:1001' ],
        types: [ 'tree' ]
      },
      {
        peerId: '12D3KooWGtFSGEYPy8d2Q2RvewPZmbVjpRnvy6aYUCAzdKMAtvEd',
        addresses: [ 'any-sync-node-2:1002' ],
        types: [ 'tree' ]
      },
      {
        peerId: '12D3KooWLdLHY5UAvZ1VzhDqZ8G2q7jb7Bkm1LThjtt36Vigea7D',
        addresses: [ 'any-sync-node-3:1003' ],
        types: [ 'tree' ]
      },
      {
        peerId: '12D3KooWDVuUW2PFBvzo5s3hE5FM7QZ7Gwk3NE5uJhbFpc9q9Gnv',
        addresses: [ 'any-sync-coordinator:1004' ],
        types: [ 'coordinator' ]
      },
      {
        peerId: '12D3KooWAizgNeqiPjSfNFCTmN4dmdh4ZxwQ7iQrKzpx9o7BBDSd',
        addresses: [ 'any-sync-filenode:1005' ],
        types: [ 'file' ]
      },
      {
        peerId: '12D3KooWMh2fv7mLavFCtTX1yNTLNQ64NckYPykL7dU2W5GrMNVn',
        addresses: [ 'any-sync-consensusnode:1006' ],
        types: [ 'consensus' ]
      }
    ],
    creationTime: ISODate("2024-01-15T19:52:35.719Z"),
    enable: true
  }
]
rs0 [direct: secondary] coordinator>

all hosts in cluster (clients and servers hosts) receive the same configuration.
it mean, the example address 'any-sync-node-1:1001' should be available for clients, but by default, this address available only in docker-compose network.

for fix it, you can configure your load balancer or add additional external address in mongo network config
example:

addresses: [ 'any-sync-node-1:1001', '192.168.0.10:1001' ]

also I recommend use dns hostnames instead of IP, if it possible.

I'm added special settings in .env file for easier configure this:
https://github.com/anyproto/any-sync-dockercompose/blob/main/.env#L14

[winston0410]

@Supporterino I am trying to do the same and building a kustomization.yaml based on the offical docker-compose.yaml. Would it be possible for you to share your existing manifest, so I can contribute on it and avoid reinventing the wheel?

[Supporterino]

@winston0410 here is a snippet to my setup I used a while back. I think this was the working one. I quickly migrated away from it sadly since the deployment in k8s wasn't that straight forward. Feel free to ask if anything is unclear. One side note a loadbalancer is strictly required as stated in the snippet.

https://git.roth-kl.de/-/snippets/2

[winston0410]

Hey @Supporterino thank you for your response, it is really helpful. Just read through your manifest there, and when you say the deployment is not straight forward, do you mean you need to kubectl apply it multiple times to make it work? Haven't worked with UDP in the past yet, but I guess using a domain name can replace the ip of each load balancer(i.e. 10.2.0.20, 10.2.0.21, etc...) and simplify the config?

Also I wonder where did you migrated away to? Did you just stick with docker-compose at the end?

[Supporterino]

Hey @winston0410,
yes just kubectl apply won't do the trick here. First you need to install the two helm charts with commands like this:

helm repo add bitnami https://charts.bitnami.com/bitnami
helm install anytype-cache bitnami/redis -f redis.yml
helm install my-mongodb bitnami/mongodb -f mongodb.yml

In those file the storage class would need to be adjusted to suite yours. After this each of the other files is more like a folder with comments indicating the actual file path. So the config.yml would result in a folder named configs with the following files inside:

• consensus.yml
• coordinator.yml
• network.yml
• file_1.yml
• sync_1.yml
• sync_2.yml
• sync_3.yml

This needs to be done for each file. After this you could run kubectl apply -k in the folder to deploy the kustomization app. If you mean you can use ingresses instead of the loadBalancer IPs that isn't possible since the nodes need to talk to each other via TCP and UDP on a specific port not 80/443. Probably the new GatewayAPI could realize it with UDP- and TCPRoutes Manifests. You need an active LoadBalancer here like MetalLB with BGP which advertises the IPs to your local network and then yes you can adjust the IPs to suite your CIDR.
I wonder where you would simplify since this is pretty much a bare bones setup don't think you can leave out any part. And you still might need a MinIO setup if I remember correctly or use the filesystem option for the file node with a PVC.
If interested I could do a little write up of how to completely setup anytype in K8S with a real helm chart or kustomization app as a repo.

For now I switched to outline as my documentation space as maintaining anytype was still a bit inconvenient but I am keeping a close eye here since the app looks awesome.