Try to avoid intermittent errors when checking links #86

douglascamata · 2022-07-11T16:09:02Z

Related to #22.

Stop relying on the http.DefaultHTTPTransport: it's effectively global state. It can be
modified by any other package. It has different default configuration depending on the
Go compiler version and we cannot trust that these defaults are good for us.
Considering this is a tool often ran in resource constrained
environments, like Github Actions, the HTTP client timeout was bumped
to 30 seconds.
Colly was configured with a depth of 1, to avoid crawling too much
information from the websites
The parallelism configuration of colly has been brought down from 100
to 10. This setting means that at most 10 requests will be sent in
parallel for the matching domains, instead of 100. This way we are
more friendly to the servers and hopefully they will be happier with us too,
returning more responses with 200 status code faster.
Added a random delay of up to 1 second to create new requests to
matching domains, again to be more friendly with the servers, etc, etc.

How did I test this?

I compiled a binary and ran the make check-docs target of the https://github.com/thanos-io/thanos project a couple of times without my changes and with them.

Without the changes it's easy to reproduce the failures indicated by #22.

With the changes, I couldn't reproduce them.

How does it affect run time?

Locals runs were taking:

Without changes: Failed local runs that failed were taking 2m48s. Successful local runs could be as fast as 1m31s.
With changes
- With 1s random delay: 3m20s.
- Without random delay: 1m30s.

There are a lot of variables involved though, it isn't a reliable measurement.

I think I could be able to get rid of the random delay to make this faster while still avoiding the intermittent failures, if that's desirable. WDYT, @bwplotka?

It's a global tranpsort and could be changed by any dependency. Signed-off-by: Douglas Camata <[email protected]>

This is an effort to avoid random errors when checking a bunch of links across many files: - Considering this is a tool often ran in resource constrained environments, like Github Actions, the HTTP client timeout was bumped to 30 seconds. - Colly was configured with a depth of 1, to avoid crawling too much information from the websites - The parallelism configuration of `colly` has been brought down from 100 to 10. This setting means that at most 10 requests will be sent in parallel for the same matching domain. This way we are more friendly to the servers and hopefully they will be happier with us too, returning more responses with 200 status code. - Added a random delay of up to 1 second to create new requests to matching domains, again to be more friendly with the servers, etc, etc. Signed-off-by: Douglas Camata <[email protected]>

`DialContext` is the new way to go. Signed-off-by: Douglas Camata <[email protected]>

bwplotka

Nice, some comments - it's though tuning play we have to until we do proper caching with ~hours/days jitter as we have planned in #54

Thanks for helping here! 💪🏽

bwplotka · 2022-07-12T11:56:49Z

pkg/mdformatter/linktransformer/link.go

@@ -241,7 +257,8 @@ func NewValidator(ctx context.Context, logger log.Logger, linksValidateConfig []
 	}
 	if err := v.c.Limit(&colly.LimitRule{
 		DomainGlob:  "*",
-		Parallelism: 100,
+		Parallelism: 10,


The parallelism configuration of colly has been brought down from 100
to 10. This setting means that at most 10 requests will be sent in
parallel for the matching domains, instead of 100. This way we are
more friendly to the servers and hopefully they will be happier with us too,
returning more responses with 200 status code faster.

Hope based development 🙈

I think this makes sense generally, but I wonder if it makes sense to perhaps expose those options so we can tune it to every environment. Right now we are bit guessing with programmatic change that takes time to modify and release.

We can only hope when it comes to checking third-party links (on remote servers we don't own) from third-party servers (Github Actions runners), sending many concurrent requests . 😂

I'll try to make some important parameters configurable via the command line 👍

Better, I am moving them to the config file, like the timeout.

Added some parameters in the config file via 23f1dec.

bwplotka · 2022-07-12T11:57:38Z

pkg/mdformatter/linktransformer/link.go

@@ -241,7 +257,8 @@ func NewValidator(ctx context.Context, logger log.Logger, linksValidateConfig []
 	}
 	if err := v.c.Limit(&colly.LimitRule{
 		DomainGlob:  "*",
-		Parallelism: 100,
+		Parallelism: 10,
+		RandomDelay: 1 * time.Second,


It might take forever to validate things 🤔 Again, useful to expose an option?

It doesn't always take forever to validate things. The delays are a random number between 0 and 1s, not always 1s. I have some runs that are just 1 minute slower than without the random delay, but again, it's hard to measure and say with confidence why a run was slow (was my network slower this time? was somebody else's server slower? which one? how slow?).

bwplotka · 2022-07-12T11:58:27Z

pkg/mdformatter/linktransformer/link.go

+	collector.SetClient(&http.Client{
+		Timeout: 30 * time.Second,
+	})
+	transport := &http.Transport{


This is great, but also before tuning we should probably expose the options we want to tune to outside of binary. The problem is that running it locally almost never reproduces what's will happen on GH actions.

Which configuration from the transport do you think would be valuable to expose as configuration, @bwplotka? I think the MaxIdleConns could be useful. But the other ones all seem like too much/too deep.

Actually I don't think tuning MaxIdleConns to different values will be useful. But I found out MaxConnsPerHost (by default 2), which can be useful. For example, Colly might be trying to concurrently send 10 requests to Github, but the transport will only hold 2 connections to Github.

Fixed in 23f1dec.

saswatamcode

Thanks a lot for taking this up! 💪🏻

Some tiny comments.

pkg/mdformatter/linktransformer/link.go

saswatamcode · 2022-07-12T12:16:42Z

pkg/mdformatter/linktransformer/link.go

+	linktransformerMetrics := newLinktransformerMetrics(reg)
+	collector := colly.NewCollector(colly.Async(), colly.StdlibContext(ctx), colly.MaxDepth(1))
+	collector.SetClient(&http.Client{
+		Timeout: 30 * time.Second,


mdox already has a parameter in config which allows users to customize this, as different users will likely have different types of docs (how many links/type of links). 🙂

We use this parameter in prometheus-operator too.

@saswatamcode if the user doesn't provide a timeout, we would end up relying on the default timeout of Colly: 10 seconds. Are we fine with that? Or do we prefer to have a higher default timeout?

Fixed in a02848d

Signed-off-by: Douglas Camata <[email protected]>

These options are: * Colly's HTTP parallelism * Colly's random delay between HTTP requests * HTTP transport's max connections per host

Signed-off-by: Douglas Camata <[email protected]>

douglascamata · 2022-07-12T14:50:18Z

I made some parameters configurable via the validate config file and updated the example in the README to include them and the timeout parameter (it wasn't documented yet).

Signed-off-by: Douglas Camata <[email protected]>

Bad editor! Signed-off-by: Douglas Camata <[email protected]>

saswatamcode

Thanks! 🎉

bwplotka

Amazing, thanks!

douglascamata added 3 commits July 11, 2022 16:00

Avoid using the default HTTP transport

1082ee4

It's a global tranpsort and could be changed by any dependency. Signed-off-by: Douglas Camata <[email protected]>

Move away from Dial (it's deprecated)

5ec34fc

`DialContext` is the new way to go. Signed-off-by: Douglas Camata <[email protected]>

bwplotka reviewed Jul 12, 2022

View reviewed changes

saswatamcode reviewed Jul 12, 2022

View reviewed changes

douglascamata added 4 commits July 12, 2022 14:24

Remove maxDepth and default timeout of 30s

a02848d

Signed-off-by: Douglas Camata <[email protected]>

Make a few HTTP options configurable

23f1dec

These options are: * Colly's HTTP parallelism * Colly's random delay between HTTP requests * HTTP transport's max connections per host

Fix MaxConnsPerHost usage

1cf8fac

Signed-off-by: Douglas Camata <[email protected]>

Update example validate configuration

9e2940e

Signed-off-by: Douglas Camata <[email protected]>

douglascamata requested review from saswatamcode and bwplotka July 12, 2022 14:50

douglascamata added 3 commits July 12, 2022 14:53

Add documentation about default config values

9b4b3c8

Signed-off-by: Douglas Camata <[email protected]>

Handle parallelism config more safely

d58f181

Signed-off-by: Douglas Camata <[email protected]>

Revert editor's autoformat

d2f9cf3

Bad editor! Signed-off-by: Douglas Camata <[email protected]>

saswatamcode approved these changes Jul 12, 2022

View reviewed changes

bwplotka approved these changes Jul 12, 2022

View reviewed changes

bwplotka merged commit d39d57f into bwplotka:main Jul 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Try to avoid intermittent errors when checking links #86

Try to avoid intermittent errors when checking links #86

douglascamata commented Jul 11, 2022 •

edited

Loading

bwplotka left a comment

bwplotka Jul 12, 2022

douglascamata Jul 12, 2022 •

edited

Loading

douglascamata Jul 12, 2022

douglascamata Jul 12, 2022

bwplotka Jul 12, 2022

douglascamata Jul 12, 2022 •

edited

Loading

bwplotka Jul 12, 2022

douglascamata Jul 12, 2022

douglascamata Jul 12, 2022 •

edited

Loading

douglascamata Jul 12, 2022

saswatamcode left a comment •

edited

Loading

saswatamcode Jul 12, 2022

douglascamata Jul 12, 2022

douglascamata Jul 12, 2022

douglascamata commented Jul 12, 2022

saswatamcode left a comment

bwplotka left a comment

Try to avoid intermittent errors when checking links #86

Try to avoid intermittent errors when checking links #86

Conversation

douglascamata commented Jul 11, 2022 • edited Loading

How did I test this?

How does it affect run time?

bwplotka left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

douglascamata Jul 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

douglascamata Jul 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

douglascamata Jul 12, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

saswatamcode left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

douglascamata commented Jul 12, 2022

saswatamcode left a comment

Choose a reason for hiding this comment

bwplotka left a comment

Choose a reason for hiding this comment

douglascamata commented Jul 11, 2022 •

edited

Loading

douglascamata Jul 12, 2022 •

edited

Loading

douglascamata Jul 12, 2022 •

edited

Loading

douglascamata Jul 12, 2022 •

edited

Loading

saswatamcode left a comment •

edited

Loading