These are community-contributed definitions for search engine and social network detections list maintained and used by Matomo (formerly Piwik), the leading open source web analytics platform.
Social networks are defined in YAML format in the file Socials.yml
The definitions contain the name of the social network, as well as a list of one or more urls.
"My Social Network":
- my-social-network.com
- mysocial.org
Search engines are defined in YAML format in the file SearchEngines.yml
Definitions of search engines contain several parameters that are required to be able to detect which search engine and which search keywords are included in a given url.
Those parameters are:
- name of the engine
- URLs of the engine
- request parameters (or regexes), that can be used to get the search keyword
- hidden keyword paths
- backlink pattern, that can be used to create a valid link back to the search engine (with the keyword)
- charsets that might be used to convert keyword to UTF-8
For each search engine (name) it is possible to define multiple configurations. Each configuration needs to include one or more urls, one or more parameters/regexes and may include a backlink and one or more charsets.
Each configuration needs to contain one ore more urls. Please only define the hostname.
You can use {}
as a placeholder for country shortcodes in subdomains or tld.
{}.searchengine.com
would also matchde.searchengine.com
ornl.searchengine.com
searchengine.{}
would also matchsearchengine.de
orsearchengine.nl
- For TLDs only
{}
would also match combined TLDs likeco.uk
. (Full listcom.*, org.*, net.*, co.*, it.*, edu.*
) - The first URL will be used for the icon, so the most popular/representative URL should be placed there
Each configuration needs to contain one or more params. A param is a name of a request param that might be available in the url. As many search engines do not use query parameters to handle the keywords, but include them in the url structure, it is also possible to define a regex. A regex need to be encapsulated by '/'
SearchEngine:
-
urls:
- searchengine.com
params:
- q
- '/search\/[^\/]+\/(.*)/'
The example above would first try to get the keyword with the request param q
. If that is not available it would use the regex '/search\/[^\/]+\/(.*)/'
to get it.
This regex would match an url like 'http://searchengine.com/search/web/matomo'
A backlink will be used to generate a link back to the search engine including the given keyword. backlinks may be defined per configuration and need to include {k}
as placeholder for the keyword.
SearchEngine:
-
urls:
- searchengine.com
params:
- q
backlink: '/search?q={k}'
For the configuration above the generated backlink would look like searchengine.com/search?q=matomo
(assuming that matomo
is the keyword).
The backlink will always be generated using the first defined url in this configuration block.
hiddenkeyword
More and more search engines started to hide keywords in referrers for privacy reasons. hiddenkeyword
allows to define if the search engines refers from paths that may not contain/provide a keyword.
If a search engine always refers from the path /do/search
that path should be added. If the path might vary regexes can be added with strings, starting and ending with /
, e.g. /\/search[0-9]*/
NOTE: The path matched against will also include the referrers query string and hash. So if the referrer might contain a query you might use a regex like /search(\?.*)?/
SearchEngine:
-
urls:
- searchengine.com
params: []
hiddenkeyword:
- '/^$/'
- '/'
- '/search'
The configuration above would allow an empty keyword for searchengine.com
, searchengine.com/
and searchengine.com/search
Charsets can be defined if search engines are using charsets other than UTF-8. The provided charset will be used to convert any detected search keyword to UTF-8.
A simple defintion of a search eninge might look like this:
SearchEngine:
-
urls:
- searchengine.com
- search-engine.org
params:
- q
- as_q
The example above would match for the hosts searchengine.com
and search-engine.org
and use the request parameters q
and as_q
(in this order) to detect the search keyword.
A simple definition of a search engine with multiple configurations might look like this:
SearchEngine:
-
urls:
- searchengine.com
params:
- as_q
-
urls:
- search-engine.org
params:
- q
The definition above would again match for the hosts searchengine.com
and search-engine.org
. However the request parameter q
will be used specifically for search-engine.org
while the request parameter as_q
will be used specifically for searchengine.com
.
A complete definition (including all optionals) of a search engine might look like this:
SearchEngine:
-
urls:
- searchengine.com
params:
- q
backlink: '/search?q={k}'
charsets:
- windows-1250
-
urls:
- search-engine.org
params:
- as_q
-
urls:
- search-engine.fr
params: []
hiddenkeyword:
- '/'
- '/^search.*/
In this case, a backlink and charset is only defined for the first configuration. Which means there is no backlink nor charset set for search-engine.org
.
We welcome your contributions and Pull requests at github.com/matomo-org/searchengine-and-social-list!