-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
$match in pipeline uses indexes, but first subscription is really slow. #19
Comments
Reactive Aggregate uses the standard MongoDB aggregation method. It will use any indexes you have defined if it can. How have you defined your indexes? |
Rob:
And this is how it looks
|
@robfallows is there any way to explain() ReactiveAggregate? |
Does this happen in development as well as production? |
@Hernanm0g : please see #20 and test to see if this fixes the problem. Thanks you. |
Hi Rob. Havent tested the new "enable_debug" yet, but adding:
helped a lot. I will test the new features and give you a feedback |
Hi @robfallows , thanks for the debugging option. I finally confirmed that my subscription is using the indexes i created, so thats not the problem.
And outputs:
which is great. But i found something weird in the timestamps, please check this out: The first time the publication is made it lasts a lot (30 seconds), but when its done, the second time it lasts less than 1 second (which is what we expect in the first time too). Check the timestamp: First pub: (30 secs)
Second pub: (1 sec)
Third piub and subsequent (1 sec)
Whenever i refresh or restart meteor, the first pub lasts a lot, and the subsequent are very fast. Im testing this on development, maybe in production it will fix as you suggested before. Any thoughts? |
@robfallows: confirmed, in production, the first publication lasts 30 secs as well. |
@robfallows, check this out:
But ReactiveAggregate:
|
@robfallows: I've finally found what was delaying the publication, and was the observers intialization. this might be helpful for any user thats having the same issue. I've added a few time trackers (in seconds) to aggregations.js to find out what was causing the delay and got this:
As you can see, the first time the observers are initialized, they take a long time to be setted. (28.323 seconds). So i decided to set noAutomaticObservers option to true, just for testing, and got this:
So its really, REALLY important that, if a collection is medium or large size (> 1000 docs), to set the "noAutomaticObservers" option to true, and to set in the "observers" object the filtered query that reduces the observer size. So now my publish looks like this
And the debug results are:
Thanks a lot for the help. This achievement would be impossible without your fast response. |
@Hernanm0g : Awesome research and really interesting results. I'm going to do a little more work on this to discover why the initial observer setup is so slow. I really don't think it should be. Glad you have a solution you're happy with 🙂 |
Hey, was playing around with implementing this on a large collection (1M+ documents) and encountered the same issue -- my app would crash because of heap allocation issues. So, I followed @Hernanm0g's tip and it solved my problem as well. Doesn't seem ideal, though. Did you did you ever make any discoveries @robfallows? |
Following up, one problem that emerges when using For example:
If the For complex documents that has an array field such as Not sure the best solution there? Setting Looks like #36 has discussion about this as well. Perhaps I need to create a new |
For large collections it's crucial to ensure that good indexes are used, or there will be long delays and possibly other performance-related issues, especially where reactive changes are causing re-runs before the previous aggregation has completed. The MongoDB engine will then be running multiple aggregations at the same time, meaning likely excessive memory (heap and stack) problems. I would suggest starting with the MongoDB shell:
The aggregation pipeline is expensive and Meteor observers are expensive, so tuning the
|
Hi Rob.
Thans a lot for this amazing tool.
Im using it in a project and im wondering why a simple ReactiveAggregate lasts 9 seconds.
My "Evaluations" Collection has ~ 5000 docs. I want to publish some of them in a ClientCollection called EvaluationsList
I have setted an index for the Evaluation collection that works pretty well on normal Pub queries.
And The following Publication lasts less than 1 sec
But when i use Reactive Aggregation it lasts more than 9 secs:
I think that maybe ReactiveAggregate doesnt use Mongodb Indexes. ¿Can you confirm this?
Thanks and regards.
The text was updated successfully, but these errors were encountered: