Implement central catalog

[vkuznet]

PhEDEx uses central catalog, while transfer2go does not. We need feasibility studies if we can use distributed model at scale. E.g., what will happen if we need to update catalog, corrupted catalog at one site, etc.

[rishiloyola]

I guess we need to test this thing to implement push and pull model. Can you tell me more about this test?

[vkuznet]

Since TFC (trivial file catalog) now resides in each agent we need to test what will happen in agent will go down or its catalog get corrupted. There are few ways to avoid this problem. We may follow "amazon" approach for distributed store where we keep a copy of catalog elsewhere, e.g. each agent keep a catalog of its neighbors. Or, we may still sync agent's catalog to central one. But in any case this will be second priority until you implement push model and we perform integration tests.

…

On 0, Rishi ***@***.***> wrote: I guess we need to test this thing to implement push and pull model. Can you tell me more about this test? -- You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub: #16 (comment)

[vkuznet]

The main-agent should be able to answer questions about participating agents. For that we need to have central catalog. The main-agent will periodically collect data from agents and merge them into central catalog. The system should work similar to git model where there is a central repository and forked ones. The forked repositories (agent's catalog) will keep current status, while central repository (main-agent) will keep info from all of them