Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reorg tiering policy sections into manage tiering #3524

Open
wants to merge 18 commits into
base: latest
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 4 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
207 changes: 188 additions & 19 deletions use-timescale/data-tiering/enabling-data-tiering.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Enabling the object storage tier
excerpt: How to enable the object storage tier
title: Manage tiering
excerpt: How to enable and use object storage tiering
products: [cloud]
keywords: [tiered storage]
tags: [storage, data management]
Expand All @@ -9,27 +9,24 @@ cloud_ui:
- [services, :serviceId, overview]
---

# Tier data to the object storage tier
# Manage tiering
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe the title should explain more clearly what we explain. Manage automatic and manual tiering?


Enable tiered storage to begin migrating rarely used data from Timescale's standard high-performance storage tier
to the object storage tier to save on storage costs.
You use tiered storage to save on storage costs. Specifically, you can migrate rarely used data from Timescale's standard high-performance storage to the object storage. With tiered storage enabled, you then either create automated tiering policies or manually tier and untier data.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

## Enabling the object storage tier
Data on the object storage tier cannot be modified - so inserts, updates, and deletes will not work on tiered data. Make sure that you are not tiering data that is being <b>actively modified</b> to the object storage tier.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

You can enable tiered storage from the Services Overview page in the Timescale
console.
## Enable tiered storage

<Procedure>
You enable tiered storage from the `Overview` tab in Console.

### Enabling tiered storage
<Procedure>

1. In the Timescale console, from the `Services` list, click the name of
1. In Timescale Console, from the `Services` list, click the name of
atovpeko marked this conversation as resolved.
Show resolved Hide resolved
the service you want to modify.
1. In the `Overview` tab, locate the `Tiered Storage` card, and click
1. In the `Overview` tab, locate the `Tiered Storage` card and click
atovpeko marked this conversation as resolved.
Show resolved Hide resolved
`Enable tiered storage`. Confirm the action.
1. Tiered storage can take a few seconds to turn on and once activated shows the amount of
data that has been tiered. Once enabled, data can be tiered by manually tiering
a chunk or by creating a tiering policy.
1. Tiered storage can take a few seconds to turn on and, once activated, shows the amount of
data that has been tiered.

<img class="main-content__illustration"
src="https://assets.timescale.com/docs/images/enable-data-tiering-ga.png"
Expand All @@ -38,9 +35,181 @@ console.

</Procedure>

After tiered storage is enabled you must either [manually tier data][manual-tier-chunk] or [setup a tiering policy][creating-data-tiering-policy]
to begin tiering data from your hypertables.
## Automate tiering with policies

A tiering policy automatically moves data to the object storage tier. Any chunks that only contain data
atovpeko marked this conversation as resolved.
Show resolved Hide resolved
older than the `move_after` threshold are moved. This works similarly to a
[data retention policy][data-retention], but chunks are moved rather than deleted. You can add tiering policies to hypertables, including continuous aggregates.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

A tiering policy schedules a job that runs periodically to migrate eligible chunks. The migration is asynchronous. The chunks are considered tiered once they appear in the `timescaledb_osm.tiered_chunks` view. Tiering does not influence your ability to query the chunks.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

### Add a tiering policy

To add a tiering policy, call `add_tiering_policy`:
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

```sql
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove what?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The bits that are moved to the updates i suggested previously.

SELECT add_tiering_policy(hypertable REGCLASS, move_after INTERVAL, if_not_exists BOOL = false);
```

For example, tier chunks that are more than three days old in the `example` hypertable in the following way:
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

```sql
SELECT add_tiering_policy('example', INTERVAL '3 days');
```

### Remove a tiering policy

To remove an existing tiering policy, call `remove_tiering_policy`:

```sql
SELECT remove_tiering_policy(hypertable REGCLASS, if_exists BOOL = false);
```

For example, remove the tiering policy from the `example` hypertable in the following way:
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

```sql
SELECT remove_tiering_policy('example');
```

If you remove a tiering policy, new scheduled chunks will not be tiered. However, already tiered chunks won't be untiered. You can [untier chunks manually](#manually-tier-and-untier-chunks) to the local storage.
Copy link
Contributor

@billy-the-fish billy-the-fish Oct 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
If you remove a tiering policy, new scheduled chunks will not be tiered. However, already tiered chunks won't be untiered. You can [untier chunks manually](#manually-tier-and-untier-chunks) to the local storage.
If you remove a tiering policy, new scheduled chunks are not be tiered. However, chunks in tiered storage are not untiered. You [untier chunks manually](#manually-tier-and-untier-chunks) to local storage.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This suggestion reverses the meaning

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You are right, updated.


## Manually tier and untier chunks

If tiering policies do not meet your current needs, you can tier and untier chunks manually.

### Tier chunks

Tiering a chunk is an asynchronous process that schedules the chunk to be tiered. In this example, you use a hypertable called `example` and tier chunks older than three days. You then proceed to list tiered chunks.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

<Procedure>

1. At the psql prompt, select all chunks in the table `example` that are older
atovpeko marked this conversation as resolved.
Show resolved Hide resolved
than three days:

```sql
SELECT show_chunks('example', older_than => INTERVAL '3 days');
```

This returns a list of chunks. Take a note of the chunk names:

```sql
|1|_timescaledb_internal_hyper_1_2_chunk|
|2|_timescaledb_internal_hyper_1_3_chunk|
```

1. Call the `tier_chunk` function to manually tier each chunk:

```sql
SELECT tier_chunk( '_timescaledb_internal_hyper_1_2_chunk');
```

1. Repeat for all chunks you want to tier.

1. To see which chunks are tiered into the object storage tier, use the `tiered_chunks` informational view:

```sql
SELECT * FROM timescaledb_osm.tiered_chunks;
```

</Procedure>

Tiering a chunk schedules it for migration to the object storage tier, but the migration won't happen immediately. Chunks are tiered one at a time in order to minimize database resource consumption. You can continue to query a chunk during migration.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

To see which chunks are scheduled for tiering either by policy or by a manual call, but have not yet been tiered, use this view:

```sql
SELECT * FROM timescaledb_osm.chunks_queued_for_tiering ;
```

### Untier chunks

Tiered data is immutable. To update data in a tiered chunk, move it back to local storage, that is, Timescale's standard high-performance storage tier. You can do so by using the `untier_chunk` stored procedure.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

Untiering chunks is a synchronous process. Chunks are renamed when the data is untiered.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

<Procedure>

1. At the `psql` prompt, check which chunks are currently tiered:
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

```sql
SELECT * FROM timescaledb_osm.tiered_chunks ;
```

The output looks something like this:

```sql
hypertable_schema | hypertable_name | chunk_name | range_start | range_end
-------------------+-----------------+------------------+------------------------+------------------------
public | sample | _hyper_1_1_chunk | 2023-02-16 00:00:00+00 | 2023-02-23 00:00:00+00
(1 row)
```

1. Run `untier_chunk`:

```sql
CALL untier_chunk('_hyper_1_1_chunk');
```

1. See the details of the chunk with the
`timescaledb_information.chunks` function:

```sql
SELECT * FROM timescaledb_information.chunks;
```

The output looks something like this:

```sql
-[ RECORD 1 ]----------+-------------------------
hypertable_schema | public
hypertable_name | sample
chunk_schema | _timescaledb_internal
chunk_name | _hyper_1_4_chunk
primary_dimension | ts
primary_dimension_type | timestamp with time zone
range_start | 2023-02-16 00:00:00+00
range_end | 2020-03-23 00:00:00+00
range_start_integer |
range_end_integer |
is_compressed | f
chunk_tablespace |
data_nodes |
```

</Procedure>


## Disable tiering

If you no longer want to use tiered storage for a particular hypertable, you
can drop the associated metadata by calling the `disable_tiering` function.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

<Procedure>

1. Call `remove_tiering_policy` and drop any tiering policy associated with this hypertable.
atovpeko marked this conversation as resolved.
Show resolved Hide resolved

1. Make sure that there is no tiered data associated with this hypertable:

1. List the tiered chunks associated with this hypertable:

```sql
select * from timescaledb_osm.tiered_chunks
```

1. If you have any tiered chunks, either untier this data, or drop these chunks from tiered storage.

1. Use `disable_tiering` to drop all tiering-related metadata for the hypertable:

```sql
select disable_tiering('my_hypertable_name');
```

1. Verify that tiering has been disabled by listing the hypertables that have tiering enabled:

```sql
select * from timescaledb_osm.tiered_hypertables;
```

</Procedure>

[manual-tier-chunk]: /use-timescale/:currentVersion:/data-tiering/manual-tier-chunk/
[creating-data-tiering-policy]: /use-timescale/:currentVersion:/data-tiering/creating-data-tiering-policy/
[data-retention]: /use-timescale/:currentVersion:/data-retention/
25 changes: 1 addition & 24 deletions use-timescale/page-index/page-index.js
Original file line number Diff line number Diff line change
Expand Up @@ -488,46 +488,23 @@ module.exports = [
"A quick tour of tiered storage",
},
{
title: "Enabling the object storage tier",
title: "Manage tiering",
href: "enabling-data-tiering",
excerpt:
"How to enable the object storage tier",
},
{
title: "Manually tier data",
href: "manual-tier-chunk",
excerpt:
"How to manually tier data to the object storage tier",
},
{
title: "Creating tiering policies",
href: "creating-data-tiering-policy",
excerpt:
"How to create a tiering policy",
},
{
title: "Querying tiered data",
href: "querying-tiered-data",
excerpt:
"How to query tiered data",
},
{
title: "Manually untier data",
href: "untier-data",
excerpt: "How to manualy untier data from the object storage tier",
},
{
title: "Replicas and forks with tiered data",
href: "tiered-data-replicas-forks",
excerpt:
"How tiered data works on replicas and forks",
},
{
title: "Disabling tiering for a hypertable",
href: "disabling-data-tiering",
excerpt:
"How to disable tiering for a hypertable",
},
{
title: "Troubleshooting",
href: "troubleshooting",
Expand Down
Loading