-
Notifications
You must be signed in to change notification settings - Fork 97
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reorg tiering policy sections into manage tiering #3524
base: latest
Are you sure you want to change the base?
Changes from 4 commits
20822a0
e44aadc
09f9863
e897b89
6ad34ad
373f6c5
c572d39
be82728
c34591a
0a643a2
ca3f4df
6b90d0b
ee380c4
d44a4a4
6d574e3
9d2b86e
dd53546
8fd8769
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
@@ -1,6 +1,6 @@ | ||||||
--- | ||||||
title: Enabling the object storage tier | ||||||
excerpt: How to enable the object storage tier | ||||||
title: Manage tiering | ||||||
excerpt: How to enable and use object storage tiering | ||||||
products: [cloud] | ||||||
keywords: [tiered storage] | ||||||
tags: [storage, data management] | ||||||
|
@@ -9,27 +9,24 @@ cloud_ui: | |||||
- [services, :serviceId, overview] | ||||||
--- | ||||||
|
||||||
# Tier data to the object storage tier | ||||||
# Manage tiering | ||||||
|
||||||
Enable tiered storage to begin migrating rarely used data from Timescale's standard high-performance storage tier | ||||||
to the object storage tier to save on storage costs. | ||||||
You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. With tiered storage enabled, you then either create automated tiering policies or manually tier and untier data. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
## Enabling the object storage tier | ||||||
Data on the object storage tier cannot be modified - so inserts, updates, and deletes will not work on tiered data. Make sure that you are not tiering data that is being <b>actively modified</b> to the object storage tier. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
You can enable tiered storage from the Services Overview page in the Timescale | ||||||
console. | ||||||
## Enable tiered storage | ||||||
|
||||||
<Procedure> | ||||||
You enable tiered storage from the `Overview` tab in Console. | ||||||
|
||||||
### Enabling tiered storage | ||||||
<Procedure> | ||||||
|
||||||
1. In the Timescale console, from the `Services` list, click the name of | ||||||
1. In Timescale Console, from the `Services` list, click the name of | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
the service you want to modify. | ||||||
1. In the `Overview` tab, locate the `Tiered Storage` card, and click | ||||||
1. In the `Overview` tab, locate the `Tiered Storage` card and click | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
`Enable tiered storage`. Confirm the action. | ||||||
1. Tiered storage can take a few seconds to turn on and once activated shows the amount of | ||||||
data that has been tiered. Once enabled, data can be tiered by manually tiering | ||||||
a chunk or by creating a tiering policy. | ||||||
1. Tiered storage can take a few seconds to turn on and, once activated, shows the amount of | ||||||
data that has been tiered. | ||||||
|
||||||
<img class="main-content__illustration" | ||||||
src="https://assets.timescale.com/docs/images/enable-data-tiering-ga.png" | ||||||
|
@@ -38,9 +35,181 @@ console. | |||||
|
||||||
</Procedure> | ||||||
|
||||||
After tiered storage is enabled you must either [manually tier data][manual-tier-chunk] or [setup a tiering policy][creating-data-tiering-policy] | ||||||
to begin tiering data from your hypertables. | ||||||
## Automate tiering with policies | ||||||
|
||||||
A tiering policy automatically moves data to the object storage tier. Any chunks that only contain data | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
older than the `move_after` threshold are moved. This works similarly to a | ||||||
[data retention policy][data-retention], but chunks are moved rather than deleted. You can add tiering policies to hypertables, including continuous aggregates. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
A tiering policy schedules a job that runs periodically to migrate eligible chunks. The migration is asynchronous. The chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view. Tiering does not influence your ability to query the chunks. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
### Add a tiering policy | ||||||
|
||||||
To add a tiering policy, call `add_tiering_policy`: | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sql | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. remove There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. remove what? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The bits that are moved to the updates i suggested previously. |
||||||
SELECT add_tiering_policy(hypertable REGCLASS, move_after INTERVAL, if_not_exists BOOL = false); | ||||||
``` | ||||||
|
||||||
For example, tier chunks that are more than three days old in the `example` hypertable in the following way: | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sql | ||||||
SELECT add_tiering_policy('example', INTERVAL '3 days'); | ||||||
``` | ||||||
|
||||||
### Remove a tiering policy | ||||||
|
||||||
To remove an existing tiering policy, call `remove_tiering_policy`: | ||||||
|
||||||
```sql | ||||||
SELECT remove_tiering_policy(hypertable REGCLASS, if_exists BOOL = false); | ||||||
``` | ||||||
|
||||||
For example, remove the tiering policy from the `example` hypertable in the following way: | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sql | ||||||
SELECT remove_tiering_policy('example'); | ||||||
``` | ||||||
|
||||||
If you remove a tiering policy, new scheduled chunks will not be tiered. However, already tiered chunks won't be untiered. You can [untier chunks manually](#manually-tier-and-untier-chunks) to the local storage. | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Suggested change
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This suggestion reverses the meaning There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You are right, updated. |
||||||
|
||||||
## Manually tier and untier chunks | ||||||
|
||||||
If tiering policies do not meet your current needs, you can tier and untier chunks manually. | ||||||
|
||||||
### Tier chunks | ||||||
|
||||||
Tiering a chunk is an asynchronous process that schedules the chunk to be tiered. In this example, you use a hypertable called `example` and tier chunks older than three days. You then proceed to list tiered chunks. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
<Procedure> | ||||||
|
||||||
1. At the psql prompt, select all chunks in the table `example` that are older | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
than three days: | ||||||
|
||||||
```sql | ||||||
SELECT show_chunks('example', older_than => INTERVAL '3 days'); | ||||||
``` | ||||||
|
||||||
This returns a list of chunks. Take a note of the chunk names: | ||||||
|
||||||
```sql | ||||||
|1|_timescaledb_internal_hyper_1_2_chunk| | ||||||
|2|_timescaledb_internal_hyper_1_3_chunk| | ||||||
``` | ||||||
|
||||||
1. Call the `tier_chunk` function to manually tier each chunk: | ||||||
|
||||||
```sql | ||||||
SELECT tier_chunk( '_timescaledb_internal_hyper_1_2_chunk'); | ||||||
``` | ||||||
|
||||||
1. Repeat for all chunks you want to tier. | ||||||
|
||||||
1. To see which chunks are tiered into the object storage tier, use the `tiered_chunks` informational view: | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_osm.tiered_chunks; | ||||||
``` | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
Tiering a chunk schedules it for migration to the object storage tier, but the migration won't happen immediately. Chunks are tiered one at a time in order to minimize database resource consumption. You can continue to query a chunk during migration. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
To see which chunks are scheduled for tiering either by policy or by a manual call, but have not yet been tiered, use this view: | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_osm.chunks_queued_for_tiering ; | ||||||
``` | ||||||
|
||||||
### Untier chunks | ||||||
|
||||||
Tiered data is immutable. To update data in a tiered chunk, move it back to local storage, that is, Timescale's standard high-performance storage tier. You can do so by using the `untier_chunk` stored procedure. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
Untiering chunks is a synchronous process. Chunks are renamed when the data is untiered. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
<Procedure> | ||||||
|
||||||
1. At the `psql` prompt, check which chunks are currently tiered: | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_osm.tiered_chunks ; | ||||||
``` | ||||||
|
||||||
The output looks something like this: | ||||||
|
||||||
```sql | ||||||
hypertable_schema | hypertable_name | chunk_name | range_start | range_end | ||||||
-------------------+-----------------+------------------+------------------------+------------------------ | ||||||
public | sample | _hyper_1_1_chunk | 2023-02-16 00:00:00+00 | 2023-02-23 00:00:00+00 | ||||||
(1 row) | ||||||
``` | ||||||
|
||||||
1. Run `untier_chunk`: | ||||||
|
||||||
```sql | ||||||
CALL untier_chunk('_hyper_1_1_chunk'); | ||||||
``` | ||||||
|
||||||
1. See the details of the chunk with the | ||||||
`timescaledb_information.chunks` function: | ||||||
|
||||||
```sql | ||||||
SELECT * FROM timescaledb_information.chunks; | ||||||
``` | ||||||
|
||||||
The output looks something like this: | ||||||
|
||||||
```sql | ||||||
-[ RECORD 1 ]----------+------------------------- | ||||||
hypertable_schema | public | ||||||
hypertable_name | sample | ||||||
chunk_schema | _timescaledb_internal | ||||||
chunk_name | _hyper_1_4_chunk | ||||||
primary_dimension | ts | ||||||
primary_dimension_type | timestamp with time zone | ||||||
range_start | 2023-02-16 00:00:00+00 | ||||||
range_end | 2020-03-23 00:00:00+00 | ||||||
range_start_integer | | ||||||
range_end_integer | | ||||||
is_compressed | f | ||||||
chunk_tablespace | | ||||||
data_nodes | | ||||||
``` | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
|
||||||
## Disable tiering | ||||||
|
||||||
If you no longer want to use tiered storage for a particular hypertable, you | ||||||
can drop the associated metadata by calling the `disable_tiering` function. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
<Procedure> | ||||||
|
||||||
1. Call `remove_tiering_policy` and drop any tiering policy associated with this hypertable. | ||||||
atovpeko marked this conversation as resolved.
Show resolved
Hide resolved
|
||||||
|
||||||
1. Make sure that there is no tiered data associated with this hypertable: | ||||||
|
||||||
1. List the tiered chunks associated with this hypertable: | ||||||
|
||||||
```sql | ||||||
select * from timescaledb_osm.tiered_chunks | ||||||
``` | ||||||
|
||||||
1. If you have any tiered chunks, either untier this data, or drop these chunks from tiered storage. | ||||||
|
||||||
1. Use `disable_tiering` to drop all tiering-related metadata for the hypertable: | ||||||
|
||||||
```sql | ||||||
select disable_tiering('my_hypertable_name'); | ||||||
``` | ||||||
|
||||||
1. Verify that tiering has been disabled by listing the hypertables that have tiering enabled: | ||||||
|
||||||
```sql | ||||||
select * from timescaledb_osm.tiered_hypertables; | ||||||
``` | ||||||
|
||||||
</Procedure> | ||||||
|
||||||
[manual-tier-chunk]: /use-timescale/:currentVersion:/data-tiering/manual-tier-chunk/ | ||||||
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/creating-data-tiering-policy/ | ||||||
[data-retention]: /use-timescale/:currentVersion:/data-retention/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe the title should explain more clearly what we explain. Manage automatic and manual tiering?