beta-2023-11-29

Release Notes

ReadySet is currently released as a docker container. See the Install with Docker documentation for instructions.

What's New

• 3679fb7: ReadySet will now automatically restore caches even if there is a backwards-incompatible upgrade that invalidates the persisted dataflow graph.

Fixes

• 66f2d9f: Fixed an issue where ReadySet failed to start up if the instance was passed a --replication-server-id.
• fc15d1b: Fixed a race condition that could result in a deadlock.
• c67bb95: Fixed an issue where ReadySet didn't start up correctly if its replication slot was dropped while it was down.

beta-2023-10-26

Release Notes

ReadySet is currently released as a docker container. See the Install with Docker documentation for instructions.

What's New

• 8b81da2: Added domain packet queueing metrics
• b67b44b: Improved write performance during replication by writing data to disk periodically in the background.
• a3f88c5: Added a config option to allow users to configure the interval upon which we report our current position to Postgres.
• 24bdd0c: Remove support for straddled joins (joins with partial keys which trace to each side of the join), which can sometimes be slow to execute.
• 7e68db8: Added support for queries which have non equi-join-key filters in left joins, as long as those filters mention columns from only one side of the join.
• 9b2fc49: Support correlated subqueries on the right-hand side of IN and NOT IN in the column list.
• 4ee8dba: Replace latency histograms in SHOW PROXIED QUERIES and SHOW CACHES with a simple counter.
• 456eb17: Limited the number of queries returned from SHOW PROXIED QUERIES to 100,000.
• e2aac68: Added metric for /metrics payload size.
• 99e6fb8: Added memory allocator metrics.

Fixes

• fb6da0f: Fixed an issue where ReadySet failed to shut down in certain situations.
• d5e0ef5: ReadySet now correctly closes upstream prepared statements when the client requests prepared statements to be closed over either the MySQL or PostgreSQL protocols.
• 377df52: Fixed an issue where ReadySet would panic if it received an EOF packet from a mysql upstream that had gone away.

beta-2023-09-28

Release Notes

ReadySet is currently released as a docker container. See the Install with Docker documentation for instructions.

What's New

• 847c0cd: Add support for correlated and uncorrelated NOT EXISTS in the WHERE clause of queries.
• c948a59: Add a new FOR CACHE <q> clause to the end of the EXPLAIN GRAPHVIZ SQL command to allow limiting the generated graphviz to only the nodes for a named cached query.
• 9496016: In SHOW READYSET STATUS, ReadySet will display the number of queries that failed to parse if --prometheus-metrics is enabled.
• e6afd5f: ReadySet will display the process start time in SHOW READYSET STATUS if --prometheus-metrics is enabled.
• 3d6800b: ReadySet will display latency quantiles in SHOW PROXIED QUERIES and SHOW CACHES if --prometheus-metrics is enabled.
• 4bb7a17: Displays the last error encountered by the replicator in SHOW READYSET STATUS if the error prevents the replicator from entering the main replication loop. The error is cleared once replication is able to begin.
• d723221: Add a --no-color flag to the readyset and readyset-server binaries, which allows disabling colored output in logs
• 309bf55: You can now optionally display only supported proxied queries with the command SHOW PROXIED SUPPORTED QUERIES. Thanks, @altmannmarcelo!
• dafad7a: Adds a status field for the connection between the ReadySet adapter and the upstream database to SHOW READYSET STATUS.
• 0f66130: Added a --replication-tables-ignore flag that allows you to replicate all tables other than the ones explicity ignored. Thanks, @altmannmarcelo!
• 1ce1282: Adds a SHOW READYSET MIGRATION STATUS <id> SQL extension to query the status of a migration initiated via CREATE CACHE CONCURRENTLY. This command will return the status of the migration, which is either "Completed", "Pending", or "Failed with error: ". Executing this statement with an id that has already returned a non-pending status (Completed or Failed) will lead to undefined behavior.
• 81340d8: Adds an optional CONCURRENTLY keyword to the CREATE CACHE syntax which triggers a non-blocking migration. If the keyword is present, the CREATE CACHE statement will return an id can be used to query the state of the migration via the Controller's /migration_status endpoint. The new CREATE CACHE syntax is CREATE CACHE [CONCURRENTLY] [ALWAYS] [<cache_name>] FROM {<select_statement> | <query_id>}
• a23caf3: In SHOW READYSET TABLES, ReadySet will display a description column as to why a table is not replicated.

Fixes

• 864e5a2: Fix proxying of Postgres queries that use the Postgres text protocol to send query parameters when those parameters use types that aren't natively supported by ReadySet.
• 51b7d21: ReadySet now correctly returns metadata about the table and column for queries that reference columns from tables directly. This allows client libraries that use that metadata to know which columns come from which tables in a query, either proxied or cached.
• 12a09b5: Fixed an issue where ReadySet got stuck in a retry loop if the upstream's WAL rotated.
• 4c5d2cf: Fixed an issue where ReadySet's replication slot could grow without bound in the absence of writes to its upstream database.
• 46a1120: Fixed an issue where we were incorrectly assuming that LSNs in replication events from Postgres would always increase monotonically.

beta-2023-08-23

Release Notes

ReadySet is currently released as a docker container. See the Install with Docker documentation for instructions.

What's New

• e166f17: Provided a better error message and guidance for passwords that include special characters, which must be percent-encoded.
• c92db77: Removed unused and hidden fallback_cache feature.
  ec5ee70: Removes references to arguments that are either not well support or not recommended for general use from the output of --help.
• f245f4c: DROP CACHE now indicates whether a cache was removed via a DELETE result with a row count of either 1 or 0.
• 1a97b33: By default, ReadySet now does not support creating fully materialized nodes. There is a flag --allow-full-materialization that allows this behavior to be configured, which replaces the previous --forbid-full-materialization when this behavior was reversed. This default prevents issues when a full materialization would have needed to allocate more memory than the system has available.
• 07ef2a8: Add support for queries with a IN and NOT IN expressions with subqueries on the right-hand side in the projected field list
• 3c87547: Add support for queries with a WHERE clause containing NOT IN with subqueries on the right-hand side
• 380b258: Add support for queries with a WHERE clause containing IN with subqueries on the right-hand side
• 66b4a54: --query-log and --query-log-ad-hoc are now enabled by default if --prometheus-metrics is set, which effectively enables all query metrics by default if metrics are enabled.
• 70756bf: The deployment name is no longer a required argument for the ReadySet adapter or server, and now defaults to "tmp-readyset". Users should take care to set the same deployment ID for all nodes in a deployment.
• 0491795: Changes the default listening address of a ReadySet adapter to 127.0.0.1:3307 or 127.0.0.1:5433 when connected to a MySQL or Postgres upstream database respectively.
• adfd2fc: Renamed NORIA_MEMORY_BYTES to READYSET_MEMORY_LIMIT (and --memory to --memory-limit, similarly), and updated the documentation provided by --help to be more accurate regarding the function of the parameter.

Performance Improvements

• e662a9d: Improved performance of Postgres cached query response encoding by avoiding an unnecessary copy of the results in memory.

Bug Fixes

• 4c07dc1: Fixed an issue where ReadySet could crash or return incorrect data in certain situations when multiple rows shared the same value in a join column for a cached query.
• 854aabb: Fixed an issue with Postgres replication that could cause dropped replication events if ReadySet fell behind while replicating
• 9b64d46: Fix a bug that led to table replication failures on Postgres when we tried to replicate temporal data that contained leap days in years prior to the year 1 CE.
• f16dfb0: Fixed a recent regression which broke support for OID types in proxied Postgres queries.
• fcbeb08: Renames the OUTPUTS_POLLING_INTERVAL env variable to VIEWS_POLLING_INTERVAL. This renaming was missed when the option was renamed to --views-polling-interval.

beta-2023-07-26

2023-07-26 Release Notes

ReadySet is currently released as a docker container. See the Quickstart documentation for instructions on using it.

What's New

• Add support for more complex conditions in the ON clause of INNER JOINs (504b5c9)
• Added a mechanism to inspect per-thread memory usage. (f1bbc02)

Performance Improvements

• Significantly reduce maximum resident memory usage for fully materialized queries by streaming records out of base tables (de3231f)
• Optimize message flushing in PG extended query protocol implementation (0f56f97)
• Improve Postgres proxy performance by eliminating an intermediate conversion step (396f41c)

Bug Fixes

• Fixed a bug which would prevent ReadySet from being able to cache certain queries which perform aggregates over subqueries (a53ab66)
• Fix an issue where certain queries which use the OR operator between conditions in the WHERE clause could sometimes return the wrong results (f7ffbe6)
• Fixed a Postgres regression that could trigger a resnapshotting loop when tables were dropped shortly after altering the database schema. (#106)
• Fixed a Postgres issue that could cause errors when executing prepared statements that referenced enum values that were added to existing enum types.
• Fixed a similar Postgres issue that triggered on arrays of enum values.

beta-2023-06-29

2023-06-29 Release Notes

Included in this release are binaries for the following platforms:

• x86_64-unknown-linux-gnu / Amazon Linux
• aarch64-apple-darwin

What's New

• Helm chart improvements. [c6ded74, cc1cf3f]
• The custom SHOW READYSET STATUS SQL statement now includes the minimum and maximum replication offset. da50579
• Apache Zookeeper is no longer supported. 83ced41
• Reduced the time ReadySet waits before retrying after a replication error from 30s to 1s. 2649e2f
• SQL support:
  • Allow spaces between and around arguments to aggregate functions. 529ba86
  • MySQL:
    • Allow arbitrary expressions as arguments to GROUP_CONCAT. 6f16afe
    • Support bit literals (already supported for Postgres). 071678b (thanks @cameronbraid!)
  • Postgres:
    • Support COMMENT statements. a2e9302
• Added the custom EXPLAIN DOMAINS SQL statement, which gives a list of domain replica shards and the URLs of the workers they're running on, sorted by the worker URL. [8e402e7, 12090f8]

Performance Improvements

• RocksDB-based persistent storage now uses the block-based SST format, improving post-snapshot compaction speed by 10x for large tables (100+ GiB), and improving get/put performance by 7-28% across current benchmarks. [8a32b30a, d0bec66]

Bug Fixes

• Update to rust-openssl 0.10.55 to patch RUSTSEC-2023-0044. 5fb5e88

• Correctly handle concurrent updates to Consul keys. cba739a

beta-2023-05-31

2023-05-31 Release Notes

What's New

• Revamped helm chart with changes to align with best practices and support
  multiple cloud platforms. a51ac40 b739acb 931d0b9 c4ff836
• Created the proptest-stateful
  crate and added extensive stateful property tests.
• ReadySet can now infer the database_type from an upstream_db_url,
  making the --database-type argument optional. If --database-type is
  provided with an upstream_db_url, readyset will validate the database
  type. 8ad4482
• Improve error messages for why something is unsupported when running
  without an upstream database. 41f3532
• Add --controller-address option that exports prometheus metrics for the
  readyset-server when running in standalone mode. 688ab93
• Add replication-server-id flag to allow multiple readyset deployments to
  connect to the same upstream db. Each deployment must have a unique
  replication server id. 896b423
• Add logging that uses row estimates to report snapshotting progress.
  9f85776
• Added an experimental feature to automatically inline
  placeholders in a query with literal values when the placeholders are
  unsupported, so that the query can be run against ReadySet. This
  feature is enabled with the --experimental-placeholder-inlining flag.
  This feature is experimental as it could degrade the performance of
  the ReadySet instance if too many inlined instances of a query are
  created. It is recommended that you drop the cache of any query with
  inlined placeholders with high cardinality. 4a31acd
• Add support for INT2, INT4, and INT8 postgres type aliases. e9518f1

Performance Improvements

• Controller requests now run asynchronously. 120fb30
• RocksDB compaction is now done as a background process. 51d5a4c
• Tuned the retry behavior of extend_recipe. 1b77183
• Initialize persistent state in a different thread. c78bd61
• Numerous miscelaneous query performance optimizations.

Bug Fixes

• Update metrics-rs crate to 0.21 to fix a bug in summary
  calculations where they
  were not being calculated over sliding windows of
  time.
  1557db0
• Fixed a bug where DDL actions were lost when we encountered a particular
  error in replication. 1101215
• Fixed an issue where we returned an empty value instead of BEGIN for
  certain transactional queries. fd08ff1
• Fixed an incorect timezone conversion that prevented lookups into a date
  field using timestamps that had the same date. 33990c7
• Fixed a bug with optimized string serialization. 5dec2e1
• Fixed a panic if offset is longer than result set, specifically in the case
  that a query, has ORDER BY, LIMIT, and OFFSET, has aggregates with a
  group by, and doesn't filter by a primary or unique column. 5d9c5ce
• Fixed a panic that could happen if a connection failed during a migration.
  020f58a
• Fixed a bug where the ordering of project and distinct nodes would return
  incorrect results. 236ed82
• Properly mark PostgreSQL partitioned tables as unsupported. 2abffe1
• Fix handling of a partial replay where a source node has been dropped.
  94be3fc
• Properly handle creating tables with FKs. f10319e
• Properly handle replication offsets for uninitialized base tables. 582b68f

beta-2023-04-12

• Fixed an issue where ReadySet would crash when replicating an ALTER TYPE statement referencing a type that was mentioned in a column of a dropped table.
• Fixed formatting of table identifier with schema (if initially provided during parsing) for ALTER TABLE, DELETE FROM, INSERT INTO, and UPDATE statements.
• ReadySet running in postgresql mode now defaults to SCRAM-SHA-256 authentication for all clients. To force ReadySet to use cleartext authentication (the previous behavior) instead, pass --postgres-authentication-method=cleartext or set the POSTGRES_AUTHENTICATION_METHOD environment variable to cleartext when running the readyset binary

beta-2023-03-15

• Fixed PostgreSQL table/column quoting for SHOW PROXIED QUERIES to use double quotes instead of backticks.
• Fixed quoting of table/column identifiers in PostgreSQL snapshotting.
• Added support for the PostgreSQL functions json_object and jsonb_object.
• Fixed a bug where we were not correctly handling netmasks in the Postgres INET type.
• When upgrading to a new version of ReadySet with an incompatible format for user configuration, ReadySet will now automatically wipe out the cluster state, where previously manual intervention was required. Note that this does mean that upgrading to a new version of ReadySet can currently result in losing CREATE CACHE statements.
• Added a new '--statement-logging' flag to enable logging statements received by ReadySet from the client, and via replication from the primary database. Logs are optionally written to the path provided by '--statement-log-path' or _statements.log.
• ReadySet now defaults to explicit query caching mode, rather than implicit
• ReadySet will now automatically re-snapshot the snapshotted copy of base table state if necessary when upgrading to a new version.
• ReadySet now automatically runs ALTER TABLE REPLICA IDENTITY FULL on PostgreSQL tables that ReadySet is configured to replicate from, and which don't have primary keys or already-configured non-default replica identities. This avoids an issue where ReadySet replicating from these tables would cause PostgreSQL to return an error message for updates and deletes to these tables

beta-2023-01-18

replication: incremental updates on compaction

Fix the incremental updates on compaction that were
broken due to the change in RocksDB log format.

Change-Id: Icd711dad5dceea1049c9ef79cf9548b497a560bb
Reviewed-on: https://gerrit.readyset.name/c/readyset/+/4160
Tested-by: Buildkite CI
Reviewed-by: Luke Osborne <[email protected]>
Reviewed-by: Peter Farr <[email protected]>