v1.0.0-rc.9

What's Changed

• Put Ion 1.1 support behind a feature flag by @popematt in #842
• Update conformance testing DSL to support unknown symbols by @nirosys in #833
• TDL syntax updates by @zslayton in #845
• Treat end-of-stream as a potential unstarted, incomplete value in StreamingRawReader by @popematt in #846
• Adds read/write support for v1.1 system symbols by @zslayton in #847
• Version bump to v1.0.0-rc.9 by @zslayton in #848

Full Changelog: v1.0.0-rc.8...v1.0.0-rc.9

v1.0.0-rc.8

What's Changed

• Adds support for eexps with 12- and 20-bit addresses by @zslayton in #825
• Adds make_sexp macro, enabling output of system values by @zslayton in #827
• Add conformance testing DSL by @nirosys in #826
• Adds support for qualified refs to $ion, $ion_encoding by @zslayton in #830
• Renames ImmutableBuffer/TextBufferView to BinaryBuffer/TextBuffer by @zslayton in #831
• Adds TDL expr groups, append_macros system macro by @zslayton in #835
• Adds OpcodeKind, LengthType to Opcode by @zslayton in #836
• Updates read_many_structs benchmark by @zslayton in #837
• Adds AsRef<[Element]> for Sequence, List, and SExp by @popematt in #841
• Version bump to v1.0.0-rc.8 by @popematt in #843

Full Changelog: v1.0.0-rc.7...v1.0.0-rc.8

v1.0.0-rc.7

This release does not change any APIs that are being stabilized. The primary changes for existing users are:

• The is_human_readable() method on the serde Serializer/Deserializer now works as expected.
• Improved detection of incomplete text Ion streams
• A fix to decimal precision calculation
• Several optimizations to reading

This release also includes the first preview implementation of Ion 1.1's macros. While this implementation is functional, more work is needed to make the APIs for defining macros both straightforward and safe. We encourage users to try it out and open issues with any feedback you may have!

What's Changed

• Shift opcodes by @zslayton in #786
• Implements flexsym annotation reading, adds binary 1.1 roundtrip unit tests by @zslayton in #788
• Adds binary 1.1 read support for e-expressions, macro expansion by @zslayton in #789
• Implement serde Serializer/Deserializer::is_human_readable() by @desaikd in #792
• Fixes logic error in Decimal::precision by @popematt in #797
• Fix certain symbols being incorrectly identified as incomplete boolean values by @popematt in #798
• Support for encoding directives, arg groups, length-prefixed e-expressions by @zslayton in #796
• Adds the (annotate ...) form and bin read support for flex_uint parameters by @zslayton in #801
• Add support for delimited containers to the 1.1 binary reader by @nirosys in #787
• Adds additional tooling APIs to enable Ion 1.1 support in the CLI's inspect command by @zslayton in #802
• Adds a 1.1 ValueWriter setting for writing symbol text inline by @zslayton in #804
• Adds text 1.1 read support for eexps with a numeric address by @zslayton in #813
• Adds binary 1.1 read support for symbol address annotations by @zslayton in #814
• Allow ? and * parameters to be omitted in tail position by @zslayton in #815
• Adds config option for writing struct field names as text/sid by @zslayton in #816
• Adds incompleteness detection tests by @zslayton in #819
• Support 1.1 LST append, binary e-exps in field value position by @zslayton in #820
• Configures Miri to use tree borrows, fixes remaining errors by @zslayton in #821
• Enables dependent macros by @zslayton in #822
• Incomplete text detection tests for v1.0 by @zslayton in #823
• Version bump to v1.0.0-rc.7 by @zslayton in #824

Full Changelog: v1.0.0-rc.6...v1.0.0-rc.7

v1.0.0-rc.6

This release does not contain API changes, only bugfixes.

Bugfixes

Detecting incomplete values in streaming text

In binary encodings, stream items contain enough data for the reader to tell whether they are complete. In text encodings, it's possible for the buffer to end with data that looks like a complete item but is not. The only way to be certain is to try to read again from the input source to confirm there's no more data. Consider the following examples in which Ion is being pulled from a File into a Vec<u8>:

foo /* comment */   ::bar::baz::1000
└────────┬───────┘ └────────┬───────┘
  buffer contents   remaining in File

              $ion _1_0
└────────┬───────┘ └────────┬───────┘
  buffer contents   remaining in File

               75 1.20
└───────┬───────┘ └────────┬───────┘
  buffer contents   remaining in File

To avoid misinterpreting the data, the StreamingRawReader now performs a final check for text readers who have emptied their buffer: it does not consider the item complete unless the input source is exhausted. (See #784.)

Correct indentation for annotated, nested values.

Prior to PR #784, the new 'pretty' text writer implementation would indent values properly but not their annotations, resulting in unexpected output like the following:

[
foo::bar::    1,
baz::quux::    2,
quuz::    3
]

This has been corrected, and the writer now emits:

[
    foo::bar::1,
    baz::quux::2,
    quuz::3
]

PRs

• Enables the StreamingRawReader to detect incomplete text by @zslayton in #784
• Version bump to 1.0.0-rc.6, improvements to IonEncoding by @zslayton in #785

Full Changelog: v1.0.0-rc.5...v1.0.0-rc.6

v1.0.0-rc.5

This release fixes a bug in v1.0.0-rc4 that could emit invalid binary Ion.

What's Changed

• Prevent writing empty annotations sequences by @zslayton in #781
• Version bump to rc5 by @zslayton in #782

Full Changelog: v1.0.0-rc.4...v1.0.0-rc.5

v1.0.0-rc.4

Caution

This release contains a bug that can cause it to emit invalid binary data if a value is explicitly assigned an empty annotations sequence. It has been fixed in v1.0.0-rc.5; users should upgrade to that version instead.

This release closes out the list of known blocking issues for version 1.0. It includes substantial changes to the experimental streaming reader and writer APIs, but only relatively small changes to the Element API being stabilized in ion-rs v1.0.

Breaking changes to the Element API

The minimum Rust version has been bumped to 1.67

Details

This allowed us to benefit from the stabilization of the ilog10 operation, which greatly simplified much of the code for our Decimal and Int types

Ints and Decimal coefficients are now limited to the i128 range

Details

The Ion data model does not impose any limitation on the range of integers that it can represent. Previously, the Int and Coefficient types would fall back to heap-allocated space to enable the representation of arbitrarily-sized integers. However, in practice there has been no call to support integers that require 17 or more bytes to represent. This simplification allowed us to remove our dependency on BigInt and to remove many branches from the codebase. We saw reading benchmark improvements in the 3-6% range across the board.

Element encoding methods have been replaced

Details

The following Element methods have been removed:

• write_as (encode data as Ion and write it to an io::Write impl)
• to_binary (encode data as binary Ion and return a newly allocated Vec<u8>)
• to_text (encode data as text Ion and return a newly allocated String)

These methods did not offer a means to configure the way the data was encoded beyond a coarse-grained choice of format. In particular, there was no room in their signatures to allow users to specify a version of Ion to use, which will become necessary when Ion 1.1 is released.

To address this, we have added types that represent the available Ion encodings:

• ion_rs::v1_0::Binary
• ion_rs::v1_0::Text

v1_0 refers to the Ion specification version, not the crate's version.

These types can be passed to methods to specify an encoding to use. They also serve as entry points to a builder API for the new WriteConfig type which allows users to specify how their data is encoded.

Together with Element's new encode_as and encode_to methods, users will be able to fully specify how their data is encoded with a list of settings that will grow over time. WriteConfig's builder-style API gives us an evolution path.

encode_as

use ion_rs::v1_0::{Binary, Text};

let element: Element = Element::string("hello");

// Encode the element as binary Ion.
let binary_buffer: Vec<u8> = element.encode_as(Binary)?;
assert_eq!(element, Element::read_one(binary_buffer)?);

// Encode the element as text Ion, further specifying that the text should be generously spaced ("pretty").
let text_ion: String = element.encode_as(Text.with_format(TextFormat::Pretty))?;
assert_eq!(element, Element::read_one(text_ion)?);

Notice that using encode_as with a text encoding results in a String while using a binary encoding results in a Vec<u8>.

encode_to

use ion_rs::v1_0::{Binary, Text};

let element: Element = Element::string("hello");

// Encode the element as binary Ion and write it to an `io::Write` implementation
let mut buffer = Vec::new();
element.encode_to(&mut buffer, Binary)?;
assert_eq!(element, Element::read_one(buffer)?);

// Encode the element as pretty text Ion and write it to an `io::Write` implementation
let mut buffer = Vec::new();
let text_ion: String = element.encode_to(Text.with_format(TextFormat::Pretty))?;
assert_eq!(element, Element::read_one(text_ion)?);

Changes that do not affect the Element API

The experimental IonReader and IonWriter traits have been replaced

Details

The IonReader and IonWriter APIs mimicked the stateful streaming IonReader/IonWriter APIs used in ion-java. Each method call would modify the state of the reader or writer, potentially changing the operations that were legal to call afterward. On the reading side, it was nearly impossible to return to data that had already been visited, which made handling struct fields (which can arrive in any order) painful.

Because development of the IonReader and IonWriter traits predated the availability of GATs, there were also many places in the API where it was necessary to use Box<dyn> to generalize over different encodings and layers of abstraction. Box<dyn> requires heap allocation and vtable lookups to function, which negatively affected performance.

These traits have been replaced by a Reader type and a Writer type that are generic over the encoding you wish to use.

Reader

Details

A reader instance only offers a few methods, the most central of which is next(). Each call to next() returns a LazyValue representing the next top-level value in the stream.

let ion_data = "1 foo::true 2024T";
let mut reader = Reader::new(ion_data);

while let Some(value) = reader.next()? {
  println!("It's a(n) {:?}", value.ion_type());
}

In the above example, the reader visits each value in the stream but--because value is a LazyValue--does not read the value. A LazyValue can tell you the value's data type, whether it's null, its annotations, and upon request, its data.

The old IonReader trait had read_TYPE methods for each Ion type (read_bool, read_int, etc). These methods would fail if the reader was not positioned on a value of the correct type or if the value was null, requiring applications to inspect its state ahead of time. Once it was confirmed, however, there was not a way to avoid having to check the Result wrapping the read_TYPE method's output.

The StreamItem enum's Null and (non-null) Value variants were distinct to allow users to call read_TYPE with the confidence that the value returned was non-null.

This combination of characteristics required unwieldy code like the following, which recursively reads and counts the values at all levels of depth in the stream:

    fn read_all_values<R: IonReader<Item = StreamItem>>(reader: &mut R) -> IonResult<usize> {
        use IonType::*;
        use StreamItem::{Nothing, Null as NullValue, Value};
        let mut count: usize = 0;
        loop {
            match reader.next()? {
                NullValue(_ion_type) => { // null values get their own code path
                    count += 1;
                    continue;
                }
                Value(ion_type) => {
                    count += 1;
                    // We need to match against the IonType of this value to know what read_* method to call
                    match ion_type {
                        String => {
                            // Each scalar read method has a `?` that handles both invalid data and
                            // the case where the reader is on a valid value of an unexpected type.
                            let _string = reader.read_str()?;
                        }
                        Symbol => {
                            // This demonstration code would read the value and discard it for timing purposes.
                            let _symbol_id = reader.read_symbol()?;
                        }
                        Int => {
                            let _int = reader.read_i64()?;
                        }
                        Float => {
                            let _float = reader.read_f64()?;
                        }
                        Decimal => {
                            let _decimal = reader.read_decimal()?;
                        }
                        Timestamp => {
                            let _timestamp = reader.read_timestamp()?;
                        }
                        Bool => {
                            let _boolean = reader.read_bool()?;
                        }
                        Blob => {
                            let _blob = reader.read_blob()?;
                        }
                        Clob => {
                            let _clob = reader.read_clob()?;
                        }
                        Null => {
                            // Matching against the IonType requires us to handle `Null` even though our StreamItem
                            // variants make that impossible.
                        }
                        // Reading a container requires you to mutate the state of the reader and then continue the loop,
                        // which can be difficult for developers to mentally model.
                        Struct | List | SExp => reader.step_in()?,
                    }
                }
                Nothing if reader.depth() > 0 => {
                    reader.step_out()?;
                }
                _ => break,
            }
        }
        Ok(count)
    }

In contrast, when you call LazyValue::read()?, it returns a ValueRef--an enum of the possible types it can return. Here's updated code that does the same thing as the ...

v1.0.0-rc.3

What's Changed

• Fixes bug in reading shared symbol table import by @desaikd in #714
• Adds implementation of finish for IonWriter by @desaikd in #720

Experimental (feature gated) changes

Lazy writer implementation

• Fixes ivm-after-nop in the lazy reader by @zslayton in #708
• Removes kludge import in integration test by @zslayton in #710
• Implements writing length-prefixed and delimited structs by @zslayton in #709
• Implements writing FlexSym annotation sequences by @zslayton in #711

Full Changelog: v1.0.0-rc.2...v1.0.0-rc.3

v1.0.0-rc.2

Changes that affect users upgrading from v1.0.0-rc1

This release does not introduce any breaking changes for users of v1.0.0-rc1. However, it does include a variety of bug fixes and performance improvements.

• Fix comparison for cross representations in Int by @desaikd in #640
• Adds test for PartialEq of Symbol for &str by @desaikd in #702
• Fix to support finding an IVM after a NOP in binary v1.0 by @zslayton in #706
• Broke integration tests into modules by @zslayton in #679
• Varuint optimization by @zslayton in #682
• Version bump to v1.0.0-rc.2 by @zslayton in #707

Upcoming breaking changes

In order to support the upcoming Ion v1.1, the read_* and write_* methods in the Element API need to be able to specify additional options. For example: readers need to be able to specify a Catalog implementation to use and writers need to be able to specify which version of Ion to produce. @desaikd has been working on future-proof new WriteConfig and ReadConfig types that will be added to those methods as arguments in an upcoming rc3.

At this time, this is the only remaining API change to address before v1.0.0 can be cut.

• Adds implementation of writer configuration by @desaikd in #685
• Adds implementation for reader builder with catalog by @desaikd in #700

Experimental (feature gated) changes

Text impl of the Lazy Reader API

The lazy reader is now available for both text and binary Ion 1.0, offering a more ergonomic API that is both faster and exposes fewer
illegal states that could result in surprising errors. In the near future, the existing readers (which are themselves feature gated) will
be replaced by the lazy reader.

• Initial raw lazy text reader (top-level nulls, bools, ints) by @zslayton in #609
• Adds support for floats to the LazyRawTextReader by @zslayton in #612
• Adds LazyRawTextReader support for comments by @zslayton in #613
• Adds LazyRawTextReader support for reading strings by @zslayton in #614
• Adds LazyRawTextReader support for reading symbols by @zslayton in #616
• Adds LazyRawTextReader support for reading lists by @zslayton in #617
• Adds LazyRawTextReader support for structs by @zslayton in #619
• Adds LazyRawTextReader support for reading IVMs by @zslayton in #620
• Initial impl of a LazyRawAnyReader by @zslayton in #621
• Adds lazy reader support for reading annotations by @zslayton in #622
• Adds lazy reader support for timestamps by @zslayton in #623
• Lazy reader support for s-expressions by @zslayton in #627
• Adds lazy reader support for decimals by @zslayton in #628
• Adds lazy reader support for blobs by @zslayton in #629
• Adds lazy reader support for long strings by @zslayton in #630
• Adds lazy reader support for reading clobs by @zslayton in #638
• Adds ion-tests integration for the lazy reader by @zslayton in #639
• Incorporates pending feedback from lazy reader PRs by @zslayton in #642

"Lazy" writer

Work is underway to offer an improved writer API that has ergonomics and safety improvements that parallel the lazy reader.
The name is a placeholder; there's nothing especially lazy about the writer's implementation.

• Introduces a new writer API by @zslayton in #680
• Adds sexp and struct writers to the LazyRawTextWriter_1_0 by @zslayton in #684
• Binary 1.0 impl for new writer API by @zslayton in #686

Ion v1.1 prototype implementation

Text reader (incl. macro evaluation)

• Stub out types for RawTextReader_1_1 by @zslayton in #643
• Initial implementation of the Ion 1.1 text reader by @zslayton in #645
• Recursive macro expansion in TDL containers by @zslayton in #647
• Incorporates feedback from PR #645 by @zslayton in #652
• Fixes parsing of macro invocations in Lists and SExps and some cases … by @popematt in #654
• Adds integration testing for Ion 1.1 by @popematt in #673
• Evaluation of template macros by @zslayton in #674
• Constant time struct field access by @zslayton in #676

Binary writer

• Skeleton impl of binary 1.1 writer by @zslayton in #688
• Implements reading/writing FlexInt, FlexUInt by @zslayton in #690
• Implements reading/writing FixedInt/FixedUInt by @zslayton in #694
• Implements writing v1.1 nulls, bools, ints, floats by @zslayton in #695
• Implements writing v1.1 strings, symbols, and SIDs by @zslayton in #696
• Implements writing v1.1 decimals by @zslayton in #697
• Implements writing v1.1 timestamps by @zslayton in #699
• Implements writing delimited and length-prefixed sequence types by @zslayton in #701
• Implements writing blobs and clobs by @zslayton in #704

serde support

• Adds experimental serde feature by @desaikd in #633

Full Changelog: v1.0.0-rc.1...v1.0.0-rc.2

v1.0.0 Release Candidate 1

This release is the first release candidate for ion-rust v1.0. It stabilizes the Element API for reading, writing, and manipulating Ion data. Please open issues for any API concerns that cannot be addressed in a backwards-compatible manner. We still intend to offer features like serde support and stabilize streaming readers and writers in future releases.

Breaking changes from v0.18

Experimental features

Portions of the API that are not ready to be stabilized have been moved into opt-in crate features.

Warning
Types requiring the experimental- features are still subject to breaking changes between minor versions.

This includes:

• The streaming reader (experimental-reader)
• The streaming writer (experimental-writer)
• Ion hash (experimental-ion-hash).

Most inner modules are now private

Nearly all inner modules are now private. In almost all cases, types that were previously imported from paths like ion_rs::types::, ion_rs::element::, etc have been re-exported at the top level.

// Before
use crate::element::Element;
use crate::result::IonResult;
use crate::types::Int;

// After
use crate::{Element, Int, IonResult};

Int and UInt are now opaque structs, not enums

In order to minimize the number of third-party types exposed in the public API, the Int and UInt types are now opaque structs. This prevents users from encountering version mismatches between their own dependency on num-bigint and ion_rs's dependency on num-bigint.

// Before
let int = Int::from(5);

match int {
  Int::I64(i64_value) => {...},
  Int::BigInt(big_int_value) => {...},
};

// After
let int = Int::from(5);

let value: i64 = int.try_into()?;
// or
let value: BigInt = int.into()?;

Into/TryInto have been added for Int/UInt to and from most Rust integer types, including i128/u128.

Vec<Element> has been replaced by Sequence

In order to make implementation changes in the future, APIs that previously returned Vec<Element> now return Sequence, an opaque wrapper around Vec<Element>.

IonError is now non_exhaustive

We may need to add new error variants to IonError in the future, so it is now marked non_exhaustive.

IonError's variants are now opaque structs

In order to improve error handling and enable future implementation changes, each of the error variants is now an opaque struct.

match Element::read_one(...) {
  Ok(element) => {...},
  Err(Decoding(e)) => {...},
  Err(Encoding(e)) => {...},
  Err(Io(e)) => {...},
  Err(IllegalOperation(e)) = {...}
  // ...
}

Methods referring to 3rd party types have been removed

The streaming reader and writer (both now marked experimental) previously supported chrono DateTimes and BigDecimals, which were holdovers from before we implemented Timestamp and Decimal.

The TimestampBuilder API is now simpler

See #588; most method names have remained the same, but the types they return have changed, potentially leading to breakage.

Pull Requests

• [User]Reader wraps SystemReader, not RawReader by @zslayton in #567
• Makes UInt an opaque struct type by @zslayton in #568
• Signed type impls of Into are now fallible by @zslayton in #570
• Makes Int an opaque struct by @zslayton in #571
• Replaces Vec<Element> with Sequence by @zslayton in #572
• Adds Into, Into impl for all ints by @zslayton in #574
• Adds "experimental-reader" feature by @zslayton in #577
• Adds write_value to ElementWriter by @popematt in #579
• Adds an "experimental-writer" feature by @zslayton in #580
• Makes IonError variants wrap opaque types by @zslayton in #584
• Removes IonDataSource and vestigial read() impls by @zslayton in #591
• Adds From<Int> and From` impls for Decimal by @zslayton in #592
• Adds Decimal methods to access the coefficient and exponent by @zslayton in #593
• Removes vestigial dependency on BigDecimal by @zslayton in #594
• Module cleanup, adds Element::expect_*, removes IntAccess by @zslayton in #595
• Clean up TimestampBuilder by @popematt in #588
• ion-hash cleanup by @zslayton in #597
• Adds traits for the lazy reader API to abstract over format by @zslayton in #596
• Makes element and types private by @zslayton in #598
• Removes Coefficient and Sign from the root by @zslayton in #599
• Adds UInt conversions to Rust unsigned int types by @zslayton in #600
• adds changes for shared symbol table support in reader by @desaikd in #578
• Error types now hold a Cow instead of requiring a String by @zslayton in #601
• Removes SymbolTable and Catalog from the public API by @zslayton in #602
• Adds doc comments, Element::write_all_as method by @zslayton in #604
• Result doc comments by @zslayton in #605
• Update readme by @zslayton in #606
• Version bump to 1.0.0-rc.1 by @zslayton in #608

Full Changelog: v0.18.1...v1.0.0-rc.1

v0.18.1

What's Changed

• Hotfix v0.18.1: Exposes raw bytes accessors in the SystemReader by @zslayton in #566

Full Changelog: v0.18.0...v0.18.1