GO-3913 separate getting identities into multiple calls to identity #1790
Conversation
Signed-off-by: AnastasiaShemyakinskaya <[email protected]>
Signed-off-by: AnastasiaShemyakinskaya <[email protected]>
Coverage provided by https://github.com/seriousben/go-patch-cover-action |
| } else { | ||
| requestedIdentities = identities[j:] | ||
| } | ||
| repoIdentities, err = s.identityRepoClient.IdentityRepoGet(ctx, requestedIdentities, []string{identityRepoDataKind}) |
| } else { | ||
| resultIdentities = append(resultIdentities, repoIdentities...) | ||
| } | ||
| j += identityBatch |
There was a problem hiding this comment.
maybe move it to the loop expression?
for j := 0; j < len(identities); j+= identityBatch {
| } | ||
| repoIdentities, err = s.identityRepoClient.IdentityRepoGet(ctx, requestedIdentities, []string{identityRepoDataKind}) | ||
| if err != nil { | ||
| log.Error("error getting identity repo data", zap.Error(err)) |
There was a problem hiding this comment.
I'm concerned about the number of log messages. Is it possible that we will log unnecessary errors?
| @@ -280,6 +278,32 @@ func (s *service) getIdentitiesDataFromRepo(ctx context.Context, identities []st | |||
| return res, nil | |||
| } | |||
|
|
|||
| func (s *service) batchIdentities(ctx context.Context, identities []string) ([]*identityrepoproto.DataWithIdentity, []string) { | |||
| resultIdentities := make([]*identityrepoproto.DataWithIdentity, 0, len(identities)) | |||
There was a problem hiding this comment.
Just summing all the comments, there should be one abstract function, which splits the []T to several batches of size <= N (take N as 200 pls), this function should be in package slice or whatever and tested
and then call in parallel some named func for every batch which returns results using waitgroup, gather everything and return.
This func should either get stuff from remote or if nothing in remote get stuff from local.
I would do it like this
Description: https://linear.app/anytype/issue/GO-3913/separate-getting-identities-into-multiple-calls-to-identity-repo