-### CRuX Countries (Optional)
+### CrUX Countries (Optional)
In case you want to change in which countries crux data will be extracted, you can also alter the COUNTRIES array, adding or removing countries using the [ISO 3166-1 alpha-2](https://en.wikipedia.org/wiki/ISO_3166-1_alpha-2) format.
```
@@ -114,9 +117,53 @@ In case you want to change in which countries crux data will be extracted, you c
// data will be extracted
```
-### Deploy
+### Deployment
After finishing the setup, upload the file into the same GCP bucket, overwriting the previous file. After that, the setup is finished and you'll start collecting performance data on your BQ tables!
+# Dashboard
+After installing and seting up the urls and countries, you'll have the "back-end" done and ready to work, but you still won't be able to visualize the data you are collecting. In order to do that, you'll have to follow through the steps below.
+
+## Duplicating Data Sources
+In the dataset created in your GCP project, you can find three tables:
+
+- crux_table
+- psi_metrics_results
+- psi_suggestion_results
+
+Each table will compose a different Data Studio data source and all these datasources will compose the dashboard, that, in turn, will create the data visualization. You'll have to duplicate all the data sources below:
+
+- [CrUX Table - Template [Data Source - Bigquery]](https://datastudio.google.com/datasources/521b4106-d002-45d4-b428-f359ea9d4b33)
+- [PSI Table Results - Template [Data Source - Bigquery]](https://datastudio.google.com/datasources/4184e227-e6bf-4323-b224-f684ad58ee9b)
+- [PSI Table Suggestions - Template [Data Source - Bigquery]](https://datastudio.google.com/datasources/1313adbe-412a-406e-beef-e7313238c43b)
+
+To duplicate, follow the steps below:
+1. Click the
+
icon
+on the top menu of each data source and then click "Copy Data Source".
+
+2. Select the GCP project in which you created your site speed dashboard BigQuery tables
+
+3. Select the BigQuery data source created when executing the terraform script
+
+4. Select the corresponding table that matches the Data Studio data source selected
+
+At the end, you'll have 3 new data sources connected to your tables.
+
+## Duplicating the dashboard template
+
+After duplicating the data sources, you'll need to duplicate the dashboard itself. To do that, follow this steps:
+
+1. Open the [Site Speed Dashboard - Template](https://datastudio.google.com/reporting/19dd4850-55a1-42ef-9de1-43f776b9ddbb/page/p_3exd7jginc) dashboard.
+
+2. Duplicate the dashboard by clicking at the
+
icon on the top menu and then at *Make a Copy* option.
+
+3. Link each Original Data Source to your newly created data sources
+
+4. Click *Copy Report*.
+
+Now you have a beautiful dashboard ready to be used!
+
# How to contribute
Pull requests are welcome! We will love help to evolve this module. Feel free to browse open issues looking for something you can do. If you have a new feature or bug, please open a new issue to be followed up by our team.
@@ -129,7 +176,8 @@ Only contributions that meet the following requirements will be accepted:
## Api Docs
-- [Index.js](https://github.com/dp6/site-speed-dashboard/blob/master/docs/index.md)
+- [PSI](https://github.com/dp6/site-speed-dashboard/blob/master/functions/psi/README.md)
+- [CrUX](https://github.com/dp6/site-speed-dashboard/blob/master/functions/crux/README.md)
# Support:
diff --git a/docs/index.md b/docs/index.md
deleted file mode 100644
index 269c5f1..0000000
--- a/docs/index.md
+++ /dev/null
@@ -1,117 +0,0 @@
-# Overview
-
-Gets urls and processes data
Gets all URLs with strategy = desktop
Gets all URLs with strategy = mobile
Requests api results for the given urls and strategies
Processes PSI API responses
Persists data on BigQuery via Stream
Loads configuration file stored into Google Cloud Storage
Sends log into stdout only if the variable debugging = true
String | Dados estruturados no padrão de persistência do BQ
-| res | String | Dados estruturados no padrão de persistência do BQ
-|
-
-
-
-
-## getUrlsDesktop(strategy => 'desktop')
-
-Gets all URLs with strategy = desktop
-
-**Kind**: global function
-
-
-
-## getUrlsMobile(strategy => 'mobile')
-
-Gets all URLs with strategy = mobile
-
-**Kind**: global function
-
-
-
-## makeRequest(urls, strategy)
-
-Requests api results for the given urls and strategies
-
-**Kind**: global function
-
-| Param | Type | Description |
-| ----- | ------------------ | -------------------------------------------------- |
-| urls | Array | URL base information (url, brand, page)
-| strategy | Array | Strategy that will be used in the request (desktop, mobile)
-|
-
-
-
-## processPsiData()
-
-Processes PSI API responses
-
-**Kind**: global function
-
-
-
-## insertRowsAsStream(data)
-
-Persists data on BigQuery via Stream
-
-**Kind**: global function
-
-| Param | Type | Description |
-| ----- | ------------------ | --------------------- |
-| data | Array | Structured data in the persistency standard of BigQuery
-|
-
-
-
-## loadProjectConfig()
-
-Loads configuration file stored into Google Cloud Storage
-
-**Kind**: global function
-
-
-## trace(log)
-
-Sends log into stdout only if the variable debugging = true
-
-**Kind**: global function
-
-| Param | Type | Description |
-| ----- | ------------------- | ------------------------------ |
-| log | Object | Text that will be shown in stdout
-|
diff --git a/functions/crux/README.md b/functions/crux/README.md
index e69de29..4a7ca7c 100644
--- a/functions/crux/README.md
+++ b/functions/crux/README.md
@@ -0,0 +1,76 @@
+# main.py
+## Overview
+
+Checks if CrUX's last month table already exists and, if it does, copies the data to your BigQuery's CrUX table
String | Context in which the request will be executed |
+
+# util_class.py
+
+Gets table suffix for the last month in CrUX's public dataset tables format
Gets number of rows in your BigQuery's CrUX table
Checks if last month's CrUX table already exists
Checks if last month's CrUX data has already been inserted in your BigQuery's CrUX table
Gets list of URLs in the config.json file
Gets list of countries in the config.json file
Update your BigQuery's CrUX table
Formats and executes query to retrieve data from CrUX's public dataset tables
List | List of countries |
+| domains | List | List of URLs |
+| table_suffix | String | Suffix of the CrUX table|
diff --git a/functions/crux/main.py b/functions/crux/main.py
index 5739875..43762f5 100644
--- a/functions/crux/main.py
+++ b/functions/crux/main.py
@@ -3,7 +3,7 @@
from util_class import Crux
import logging
import os
-
+
logging.basicConfig(level=logging.INFO)
storage_client = storage.Client()
diff --git a/functions/crux/util_class.py b/functions/crux/util_class.py
index c801287..c748d30 100644
--- a/functions/crux/util_class.py
+++ b/functions/crux/util_class.py
@@ -38,9 +38,9 @@ def check_table_crux(self):
def check_last_month(self):
try:
response = self.bigquery_client.query("""
- SELECT if(count(*) > 0,true,false) as check_rows FROM `{table}`
- where year_month = '{year_month}'
-
+ SELECT if(count(*) > 0,true,false) as check_rows
+ FROM `{table}`
+ WHERE year_month = '{year_month}'
""".format(year_month = self.table_suffix(), table = self.crux_table)
)
return list(response.result())[0].values()[0]
diff --git a/functions/psi/README.md b/functions/psi/README.md
index e69de29..f2df892 100644
--- a/functions/psi/README.md
+++ b/functions/psi/README.md
@@ -0,0 +1,112 @@
+# Overview
+
+Gets urls and processes data
Gets all URLs with strategy = desktop
Gets all URLs with strategy = mobile
Requests api results for the given urls and strategies
Processes PSI API responses
Persists data on BigQuery via Stream
Loads configuration file stored into Google Cloud Storage
Sends log into stdout only if the variable debugging = true
String | Request for the PSI API |
+| res | String | Response of the PSI API |
+
+
+
+## getUrlsDesktop(strategy => 'desktop')
+
+Gets all URLs with strategy = desktop
+
+**Kind**: global function
+
+
+
+## getUrlsMobile(strategy => 'mobile')
+
+Gets all URLs with strategy = mobile
+
+**Kind**: global function
+
+
+
+## makeRequest(urls, strategy)
+
+Requests api results for the given urls and strategies
+
+**Kind**: global function
+
+| Param | Type | Description |
+| ----- | ------------------ | -------------------------------- |
+| urls | Array | URL base information (url, brand, page) |
+| strategy | Array | Strategy that will be used in the request (desktop, mobile) |
+
+
+
+## processPsiData()
+
+Processes PSI API responses
+
+**Kind**: global function
+
+
+
+## insertRowsAsStream(data)
+
+Persists data on BigQuery via Stream
+
+**Kind**: global function
+
+| Param | Type | Description |
+| ----- | ------------------ | ---------------------------------- |
+| data | Array | Structured data in the persistency standard of BigQuery |
+
+
+
+## loadProjectConfig()
+
+Loads configuration file stored into Google Cloud Storage
+
+**Kind**: global function
+
+
+## trace(log)
+
+Sends log into stdout only if the variable debugging = true
+
+**Kind**: global function
+
+| Param | Type | Description |
+| ----- | ------------------- | -------------------------------- |
+| log | Object | Text that will be shown in stdout|
diff --git a/terraform/scripts/download-project.sh b/terraform/scripts/download-project.sh
deleted file mode 100644
index 98f8687..0000000
--- a/terraform/scripts/download-project.sh
+++ /dev/null
@@ -1,28 +0,0 @@
-
-#!/bin/bash
-echo "Executando $0 com os parâmetros versão=$1 e bucket=$2"
-curl "https://codeload.github.com/DP6/site-speed-dashboard/zip/$1" --output code.zip
-unzip code.zip
-rm -f code.zip
-
-FILE_NAME_PSI="$1.zip"
-FILE_NAME_CRUX="crux_data.zip"
-
-cd site-speed-dashboard-"$1"/functions/psi
-zip -r $FILE_NAME_PSI package.json index.js README.md
-echo "Movendo Zip para terraform/files-copy-to-gcs/config/"
-mv $FILE_NAME_PSI ./../../terraform/files-copy-to-gcs/config/
-
-cd ../crux
-zip -r $FILE_NAME_CRUX *
-echo "Movendo Zip para terraform/files-copy-to-gcs/config/"
-mv $FILE_NAME_CRUX ./../../terraform/files-copy-to-gcs/config/
-
-echo "Entrando nas pasta terraform para copiar os arquivos"
-cd ./../../terraform
-echo "Iniciando copia para GCP"
-gsutil cp -r ./files-copy-to-gcs/* "gs://$2"
-echo "excluindo zip"
-cd ./files-copy-to-gcs/config
-rm -rf *.zip
-echo "FIM script $0"
\ No newline at end of file
diff --git a/terraform/scripts/using-local-project.sh b/terraform/scripts/using-local-project.sh
deleted file mode 100644
index d400cb7..0000000
--- a/terraform/scripts/using-local-project.sh
+++ /dev/null
@@ -1,31 +0,0 @@
-
-#!/bin/bash
-echo "Executando $0 com os parâmetros versão=$1 e bucket=$2"
-
-FILE_NAME_PSI="$1.zip"
-FILE_NAME_CRUX="crux_data.zip"
-
-echo "Acessando diretorio raiz da Cloud Function PSI do projeto"
-cd ../functions/psi
-echo "Criando Zip PSI"
-zip -r $FILE_NAME_PSI package.json index.js README.md
-echo "Movendo Zip para terraform/files-copy-to-gcs/config/"
-mv $FILE_NAME_PSI ./../../terraform/files-copy-to-gcs/config/
-
-echo "Acessando diretorio raiz da Cloud Function CRUX do projeto"
-cd ../crux
-zip -r $FILE_NAME_CRUX *
-echo "Movendo Zip para terraform/files-copy-to-gcs/config/"
-mv $FILE_NAME_CRUX ./../../terraform/files-copy-to-gcs/config/
-
-echo "Entrando nas pasta terraform para copiar os arquivos"
-cd ./../../terraform
-
-echo "Iniciando copia para GCP"
-gsutil cp -r ./files-copy-to-gcs/* "gs://$2"
-
-echo "Excluindo zip"
-cd ./files-copy-to-gcs/config
-rm -rf *.zip
-
-echo "FIM script $0"
\ No newline at end of file
diff --git a/terraform/variables.tf b/terraform/variables.tf
index e5d6837..1d3cc87 100644
--- a/terraform/variables.tf
+++ b/terraform/variables.tf
@@ -9,12 +9,12 @@ locals {
gcs_bucket_folder_name = "config"
cf_name_psi = "${var.project_prefix}-sitespeed-psi"
- cf_name_crux = "${var.project_prefix}-sitespeed-crux"
+ cf_name_crux = "${var.project_prefix}-sitespeed-crux"
cf_entry_point_psi = "getUrls"
cf_entry_point_crux = "main"
- scheduler_name_psi = "${var.project_prefix}-sitespeed-psi-job"
- scheduler_name_crux = "${var.project_prefix}-sitespeed-crux-job"
+ scheduler_name_psi = "${var.project_prefix}-sitespeed-psi-job"
+ scheduler_name_crux = "${var.project_prefix}-sitespeed-crux-job"
scheduler_time_zone = "America/Sao_Paulo"
scheduler_attempt_deadline = "320s"
From 7271d569265e145070f44ecf285cfd2da78a7b63 Mon Sep 17 00:00:00 2001
From: Gian Gabriel Panacioni <66138840+GianGP@users.noreply.github.com>
Date: Wed, 23 Feb 2022 16:16:00 -0300
Subject: [PATCH 2/2] docs: :memo: Update api links in main readme
Update links of the api docs in the root folder's README
---
README.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/README.md b/README.md
index 903dc1e..19ae90b 100644
--- a/README.md
+++ b/README.md
@@ -176,8 +176,8 @@ Only contributions that meet the following requirements will be accepted:
## Api Docs
-- [PSI](https://github.com/dp6/site-speed-dashboard/blob/master/functions/psi/README.md)
-- [CrUX](https://github.com/dp6/site-speed-dashboard/blob/master/functions/crux/README.md)
+- [PSI](https://github.com/dp6/site-speed-dashboard/blob/master/functions/psi)
+- [CrUX](https://github.com/dp6/site-speed-dashboard/blob/master/functions/crux)
# Support: