Merge branch 'latest' into 3508-docs-rfc-reorg-tiering-policy-section…

…s-into-manage-tiering
timescale · Nov 1, 2024 · 8fd8769 · 8fd8769
2 parents dd53546 + a96d7c0
commit 8fd8769
Show file tree

Hide file tree

Showing 10 changed files with 219 additions and 44 deletions.
diff --git a/about/changelog.md b/about/changelog.md
@@ -8,6 +8,25 @@ keywords: [changelog, upgrades, updates, releases]
 
 All the latest features and updates to Timescale products.
 
+## 🎃 New AI, data integration, and performance enhancements
+<Label type="date">October 31, 2024</Label>
+
+### Pgai Vectorizer: vector embeddings as database indexes (early access)
+This early access feature enables you to automatically create, update, and maintain embeddings as your data changes. Just like an index, Timescale handles all the complexity: syncing, versioning, and cleanup happen automatically.
+This means no manual tracking, zero maintenance burden, and the freedom to rapidly experiment with different embedding models and chunking strategies without building new pipelines.
+Navigate to the AI tab in your service overview and follow the instructions to add your OpenAI API key and set up your first vectorizer or read our [guide to automate embedding generation with pgai Vectorizer](https://github.com/timescale/pgai/blob/main/docs/vectorizer.md) for more details.
+
+![Vectorizer setup](https://s3.amazonaws.com/assets.timescale.com/docs/images/vectorizer-setup.png)
+
+### PostgreSQL-to-PostgreSQL foreign data wrappers: 
+Fetch and query data from multiple PostgreSQL databases, including time-series data in hypertables, directly within Timescale Cloud using [foreign data wrappers (FDW)](https://docs.timescale.com/use-timescale/latest/schema-management/foreign-data-wrappers/). No more complicated ETL processes or external tools—just seamless integration right within your SQL editor. This feature is ideal for developers who manage multiple PostgreSQL and time-series instances and need quick, easy access to data across databases.
+
+### Chunk interval recommendations
+Timescale Console now shows recommendations for services with too many small chunks in their hypertables. 
+Recommendations for new intervals that improve service performance are displayed for each underperforming service and hypertable. Users can then change their chunk interval and boost performance within Timescale Console.
+
+![Chunk interval recommendation](https://s3.amazonaws.com/assets.timescale.com/docs/images/chunk-interval-recommendation.png)
+
 ## 💡 Help with hypertables and faster notebooks
 <Label type="date">October 18, 2024</Label>
 

diff --git a/api/create_materialized_view.md b/api/create_materialized_view.md
@@ -48,8 +48,14 @@ Continuous aggregates have some limitations of what types of queries they can
 support. For more information, see the
 [continuous aggregates section][cagg-how-tos].
 
-The settings for continuous aggregates are in the
-[informational views][info-views].
+For services running TimescaleDB v2.17.1 and greater, to dramatically decrease the amount
+of data written on a continuous aggregate in the presence of a small number of changes,
+reduce the i/o cost of refreshing a continuous aggregate, and generate fewer Write-Ahead
+Logs (WAL), set the`timescaledb.enable_merge_on_cagg_refresh`
+[configuration parameter][modify-parameters] to `TRUE`. This enables continuous aggregate
+refresh to use merge instead of deleting old materialized data and re-inserting.
+
+For more settings for continuous aggregates, see [timescaledb_information.continuous_aggregates][info-views].
 
 ## Parameters
 
@@ -111,4 +117,4 @@ WITH (timescaledb.continuous) AS
 [cagg-how-tos]: /use-timescale/:currentVersion:/continuous-aggregates/
 [real-time-aggregates]: /use-timescale/:currentVersion:/continuous-aggregates/real-time-aggregates/
 [refresh-cagg]: /api/:currentVersion:/continuous-aggregates/refresh_continuous_aggregate/
-[info-views]: /api/:currentVersion:/informational-views/
+[info-views]: /api/:currentVersion:/informational-views/continuous_aggregates/
diff --git a/api/refresh_continuous_aggregate.md b/api/refresh_continuous_aggregate.md
@@ -30,6 +30,9 @@ zone, so, if `window_start` and `window_end` is specified in the local time
 zone, any time zone shift relative UTC needs to be accounted for when refreshing
 to align with bucket boundaries.
 
+To improve performance for continuous aggregate refresh, see 
+[CREATE MATERIALIZED VIEW ][create_materialized_view].
+
 ### Required arguments
 
 |Name|Type|Description|
@@ -89,3 +92,7 @@ BEGIN
 END
 $$;
 ```
+
+
+[modify-parameters]: /use-timescale/:currentVersion/configuration/customize-configuration/
+[create_materialized_view]: /api/:currentVersion:/continuous-aggregates/create_materialized_view/
diff --git a/quick-start/index.md b/quick-start/index.md
@@ -13,5 +13,5 @@ These pages help you to quickly integrate Timescale Cloud and TimescaleDB into y
 * [Golang](/quick-start/:currentVersion:/golang/)
 * [Java](/quick-start/:currentVersion:/java/)
 
-You are not limited to these languages. Timescale Cloud is based on PosgreSQL, you can interface
+You are not limited to these languages. Timescale Cloud is based on PostgreSQL, you can interface
 with TimescaleDB and Timescale Cloud using any [PostgreSQL client driver](https://wiki.postgresql.org/wiki/List_of_drivers).
diff --git a/self-hosted/configuration/timescaledb-config.md b/self-hosted/configuration/timescaledb-config.md
@@ -6,6 +6,8 @@ keywords: [configuration, settings]
 tags: [tune]
 ---
 
+import MultiNodeDeprecation from "versionContent/_partials/_multi-node-deprecation.mdx";
+
 # TimescaleDB configuration and tuning
 
 Just as you can tune settings in PostgreSQL, TimescaleDB provides a number of configuration
@@ -36,8 +38,16 @@ Enables or disables the vectorized optimizations in the query executor. For
 example, the `sum()` aggregation function on compressed chunks can be optimized
 in this way.
 
+### `timescaledb.enable_merge_on_cagg_refresh  (bool)`
+
+Set to `TRUE` to dramatically decrease the amount of data written on a continuous aggregate 
+in the presence of a small number of changes, reduce the i/o cost of refreshing a 
+[continuous aggregate][continuous-aggregates], and generate fewer Write-Ahead Logs (WAL)
+
 ## Distributed hypertables
 
+<MultiNodeDeprecation />
+
 ### `timescaledb.enable_2pc (bool)`
 
 Enables two-phase commit for distributed hypertables. If disabled, it
@@ -119,3 +129,5 @@ Records last time `timescaledb-tune` ran.
 ### `timescaledb.last_tuned_version (string)`
 
 Version of `timescaledb-tune` used to tune when it runs.
+
+[continuous-aggregates]: /use-timescale/:currentVersion:/continuous-aggregates/
diff --git a/use-timescale/compression/modify-compressed-data.md b/use-timescale/compression/modify-compressed-data.md
@@ -20,14 +20,22 @@ This feature requires PostgreSQL 14 or later
 
 <Tab title="TimescaleDB&nbsp;2.11 and later">
 
-In TimescaleDB&nbsp;2.11 and later, you can insert data into compressed chunks.
+In TimescaleDB v2.11 and later, you can insert data into compressed chunks.
 This works even if the data you are inserting has unique constraints, and those
 constraints are preserved during the insert operation. This is done by using a
 PostgreSQL function that decompresses relevant data during the insert to check
 if the new data breaks unique checks. This means that any time you insert data
 into a compressed chunk, a small amount of data is decompressed to allow a
 speculative insertion, and block any inserts which could violate constraints.
 
+For TimescaleDB v2.17.0 and later there is improved delete performance on compressed 
+hypertables when a large amount of data is affected. When you delete whole segments of 
+data, filter your deletes by `segment_by` column(s) instead of separate deletes. 
+This considerably increase performance by skipping the decompression step. 
+
+
+
+
 </Tab>
 
 <Tab title="TimescaleDB&nbsp;2.3-2.10">

diff --git a/use-timescale/extensions/index.md b/use-timescale/extensions/index.md
@@ -13,48 +13,50 @@ supported extensions:
 
 <!-- vale Vale.Spelling = NO -->
 
-|Extension|Description|
-|-|-|
-|bloom|Bloom access method - signature file based index|
-|btree_gin|Support for indexing common datatypes in GIN|
-|btree_gist|Support for indexing common datatypes in GiST|
-|citext|Data type for case-insensitive character strings|
-|cube|Data type for multidimensional cubes|
-|dict_int|Text search dictionary template for integers|
-|dict_xsyn|Text search dictionary template for extended synonym processing|
-|fuzzystrmatch|Determine similarities and distance between strings|
-|hstore|Data type for storing sets of (key, value) pairs|
-|intarray|Functions, operators, and index support for 1-D arrays of integers|
-|isn|Data types for international product numbering standards|
-|lo|Large Object maintenance|
-|ltree|Data type for hierarchical tree-like structures|
-|pg_stat_statements|Track execution statistics of all SQL statements executed|
-|pg_trgm|Text similarity measurement and index searching based on trigrams|
-|pgaudit|Detailed session and/or object audit logging|
-|[pgcrypto][pgcrypto]|Cryptographic functions|
-|pgpcre|Perl-compatible RegEx|
-|pgrouting|pgRouting Extension|
-|pgstattuple|Obtain tuple-level statistics|
-|[pgvector][pgvector]|Vector similarity search for PostgreSQL|
-|plpgsql|SQL procedural language|
-|[postgis][postgis]|PostGIS geometry and geography spatial types and functions|
-|postgis_raster|PostGIS raster types and functions|
-|postgis_sfcgal|PostGIS SFCGAL functions|
-|postgis_tiger_geocoder|PostGIS tiger geocoder and reverse geocoder|
-|postgis_topology|PostGIS topology spatial types and functions|
-|seg|data type for representing line segments or floating-point intervals|
-|tablefunc|Functions that manipulate whole tables, including crosstab|
-|tcn|Triggered change notifications|
-|timescaledb_toolkit|TimescaleDB Toolkit|
-|timescaledb|Enables scalable inserts and complex queries for time-series data|
-|tsm_system_rows|TABLESAMPLE method which accepts number of rows as a limit|
-|tsm_system_time|TABLESAMPLE method which accepts time in milliseconds as a limit|
-|unaccent|Text search dictionary that removes accents|
-|unit|SI Units for PostgreSQL|
-|uuid-ossp|Generate universally unique identifiers (UUIDs)|
+| Extension                    |Description|
+|------------------------------|-|
+| bloom                        |Bloom access method - signature file based index|
+| btree_gin                    |Support for indexing common datatypes in GIN|
+| btree_gist                   |Support for indexing common datatypes in GiST|
+| citext                       |Data type for case-insensitive character strings|
+| cube                         |Data type for multidimensional cubes|
+| dict_int                     |Text search dictionary template for integers|
+| dict_xsyn                    |Text search dictionary template for extended synonym processing|
+| fuzzystrmatch                |Determine similarities and distance between strings|
+| hstore                       |Data type for storing sets of (key, value) pairs|
+| intarray                     |Functions, operators, and index support for 1-D arrays of integers|
+| isn                          |Data types for international product numbering standards|
+| lo                           |Large Object maintenance|
+| ltree                        |Data type for hierarchical tree-like structures|
+| pg_stat_statements           |Track execution statistics of all SQL statements executed|
+| pg_trgm                      |Text similarity measurement and index searching based on trigrams|
+| pgaudit                      |Detailed session and/or object audit logging|
+| [pgcrypto][pgcrypto]         |Cryptographic functions|
+| pgpcre                       |Perl-compatible RegEx|
+| pgrouting                    |pgRouting Extension|
+| pgstattuple                  |Obtain tuple-level statistics|
+| [pgvector][pgvector]         |Vector similarity search for PostgreSQL|
+| plpgsql                      |SQL procedural language|
+| [postgis][postgis]           |PostGIS geometry and geography spatial types and functions|
+| postgis_raster               |PostGIS raster types and functions|
+| postgis_sfcgal               |PostGIS SFCGAL functions|
+| postgis_tiger_geocoder       |PostGIS tiger geocoder and reverse geocoder|
+| postgis_topology             |PostGIS topology spatial types and functions|
+| [postgres_fdw][postgres-fdw] |Foreign data wrappers|
+| seg                          |data type for representing line segments or floating-point intervals|
+| tablefunc                    |Functions that manipulate whole tables, including crosstab|
+| tcn                          |Triggered change notifications|
+| timescaledb_toolkit          |TimescaleDB Toolkit|
+| timescaledb                  |Enables scalable inserts and complex queries for time-series data|
+| tsm_system_rows              |TABLESAMPLE method which accepts number of rows as a limit|
+| tsm_system_time              |TABLESAMPLE method which accepts time in milliseconds as a limit|
+| unaccent                     |Text search dictionary that removes accents|
+| unit                         |SI Units for PostgreSQL|
+| uuid-ossp                    |Generate universally unique identifiers (UUIDs)|
 
 <!-- vale Vale.Spelling = YES -->
 
 [pgvector]: /use-timescale/:currentVersion:/extensions/pgvector/
 [pgcrypto]: /use-timescale/:currentVersion:/extensions/pgcrypto/
 [postgis]: /use-timescale/:currentVersion:/extensions/postgis/
+[postgres-fdw]: /use-timescale/:currentVersion:/schema-management/foreign-data-wrappers/
diff --git a/use-timescale/page-index/page-index.js b/use-timescale/page-index/page-index.js
@@ -237,6 +237,10 @@ module.exports = [
             href: "json",
             excerpt: "Using JSON data types in a hypertable",
           },
+          {
+            title: "Foreign data wrappers",
+            href: "foreign-data-wrappers",
+          },
           {
             title: "Troubleshoot schema management",
             href: "troubleshooting",

diff --git a/use-timescale/schema-management/foreign-data-wrappers.md b/use-timescale/schema-management/foreign-data-wrappers.md
@@ -0,0 +1,113 @@
+---
+title: Foreign data wrappers
+excerpt: Query PostgreSQL databases within or outside Timescale
+products: [cloud, mst, self_hosted]
+keywords: [hypertables, schemas, alter]
+tags: [change]
+---
+
+# Foreign data wrappers
+
+You use foreign data wrappers (FDWs) to query external data sources from a Timescale Cloud service. These data sources can be one of the following:
+
+- Other Timescale Cloud services.
+- PostgreSQL databases outside of Timescale Cloud.
+
+If you are using [VPC peering][vpc-peering], you can create FDWs in your Customer VPC for a service in Timescale VPC or the same project. However, you can't create FDWs from Timescale VPC to Customer VPC.
+
+FDWs are particularly useful if you manage multiple different Timescale Cloud service types, and need to seamlessly access and merge regular and time-series data.
+
+## Query another data source
+
+You create FDWs with the `postgres_fdw` extension, which is enabled by default. 
+
+<Procedure>
+
+To query another data source, run the following queries in the [SQL editor][sql-editor]:
+
+1. **Create a server:**
+
+   ```sql
+   CREATE SERVER myserver 
+   FOREIGN DATA WRAPPER postgres_fdw 
+   OPTIONS (host 'serviceID.projectID.tsdb.cloud.timescale.com', dbname 'tsdb', port '30702');
+   ```
+
+1. **Create user mapping:**
+
+   ```sql
+   CREATE USER MAPPING FOR tsdbadmin 
+   SERVER myserver 
+   OPTIONS (user 'tsdbadmin', password 'mysupersecurepassword');
+   ```
+
+1. **Import a foreign schema (recommended) or create a foreign table:**
+
+    - Import the whole schema:
+
+      ```sql
+      CREATE SCHEMA foreign_stuff;
+      
+      IMPORT FOREIGN SCHEMA public 
+      FROM SERVER myserver 
+      INTO foreign_stuff ;
+      ```
+
+    - Alternatively, import a limited number of tables: 
+
+      ```sql
+      CREATE SCHEMA foreign_stuff;
+      
+      IMPORT FOREIGN SCHEMA public 
+      LIMIT TO (table1, table2) 
+      FROM SERVER myserver 
+      INTO foreign_stuff;
+      ```
+
+    - Create a foreign table. Skip if you are importing a schema:
+
+      ```sql
+      CREATE FOREIGN TABLE films (
+          code        char(5) NOT NULL,
+          title       varchar(40) NOT NULL,
+          did         integer NOT NULL,
+          date_prod   date,
+          kind        varchar(10),
+          len         interval hour to minute
+      )
+      SERVER film_server;
+      ```
+
+</Procedure>
+
+
+A user with the `tsdbadmin` role assigned already has the required `USAGE` permission to create FDWs. You can enable another user, without the `tsdbadmin` role assigned, to query foreign data. To do so, explicitly grant the permission: 
+
+```sql
+CREATE USER grafana;
+       
+GRANT grafana TO tsdbadmin;
+
+CREATE SCHEMA fdw AUTHORIZATION grafana;
+
+CREATE SERVER db1 FOREIGN DATA WRAPPER postgres_fdw 
+OPTIONS (host 'serviceID.projectID.tsdb.cloud.timescale.com', dbname 'tsdb', port '30702');
+
+CREATE USER MAPPING FOR grafana SERVER db1 
+OPTIONS (user 'tsdbadmin', password 'mysupersecurepassword');
+
+GRANT USAGE ON FOREIGN SERVER db1 TO grafana;
+
+SET ROLE grafana;
+
+IMPORT FOREIGN SCHEMA public 
+       FROM SERVER db1 
+       INTO fdw;
+```
+
+[vpc-peering]: /use-timescale/:currentVersion:/vpc/
+[sql-editor]: /getting-started/:currentVersion:/run-queries-from-console/#ops-mode-sql-editor/
+
+
+
+
diff --git a/use-timescale/schema-management/index.md b/use-timescale/schema-management/index.md
@@ -23,6 +23,8 @@ significant performance improvements.
 *   [Create an index][schema-indexing] to speed up your queries.
 *   [Create triggers][schema-triggers] to propagate your schema changes to chunks.
 *   [Use JSON and JSONB][schema-json] for semi-structured data.
+*   [Query external databases][foreign-data-wrappers] with foreign data wrappers.
+*   [Troubleshoot][troubleshoot-schemas] your schemas. 
 
 [about-constraints]: /use-timescale/:currentVersion:/schema-management/about-constraints
 [about-indexing]: /use-timescale/:currentVersion:/schema-management/about-indexing
@@ -32,3 +34,5 @@ significant performance improvements.
 [schema-indexing]: /use-timescale/:currentVersion:/schema-management/indexing
 [schema-json]: /use-timescale/:currentVersion:/schema-management/json
 [schema-triggers]: /use-timescale/:currentVersion:/schema-management/triggers
+[foreign-data-wrappers]: /use-timescale/:currentVersion:/schema-management/foreign-data-wrappers
+[troubleshoot-schemas]: /use-timescale/:currentVersion:/schema-management/troubleshooting