From 36c5061c42c1c99d0ae6b630d4db7a3b0dfe2d9a Mon Sep 17 00:00:00 2001 From: chirsz-ever Date: Fri, 17 Jan 2025 18:09:20 +0800 Subject: [PATCH 1/5] Add `ignore_trailing_commas` option Added examples and modified the corresponding documents and unit tests. Signed-off-by: chirsz-ever --- README.md | 12 +- docs/mkdocs/docs/api/basic_json/accept.md | 11 +- docs/mkdocs/docs/api/basic_json/parse.md | 39 +++++- docs/mkdocs/docs/api/basic_json/sax_parse.md | 10 +- docs/mkdocs/docs/examples/comments.cpp | 31 +++++ docs/mkdocs/docs/examples/comments.output | 12 ++ docs/mkdocs/docs/examples/trailing_commas.cpp | 37 ++++++ .../docs/examples/trailing_commas.output | 12 ++ docs/mkdocs/docs/features/comments.md | 53 +------- docs/mkdocs/docs/features/trailing_commas.md | 39 ++++++ docs/mkdocs/mkdocs.yml | 1 + include/nlohmann/detail/input/parser.hpp | 63 ++++++---- include/nlohmann/json.hpp | 50 ++++---- single_include/nlohmann/json.hpp | 113 +++++++++++------- tests/src/unit-class_parser.cpp | 38 +++++- 15 files changed, 379 insertions(+), 142 deletions(-) create mode 100644 docs/mkdocs/docs/examples/comments.cpp create mode 100644 docs/mkdocs/docs/examples/comments.output create mode 100644 docs/mkdocs/docs/examples/trailing_commas.cpp create mode 100644 docs/mkdocs/docs/examples/trailing_commas.output create mode 100644 docs/mkdocs/docs/features/trailing_commas.md diff --git a/README.md b/README.md index 0d98fb931c..ead12212c3 100644 --- a/README.md +++ b/README.md @@ -1763,7 +1763,17 @@ This library does not support comments by default. It does so for three reasons: 3. It is dangerous for interoperability if some libraries would add comment support while others don't. Please check [The Harmful Consequences of the Robustness Principle](https://tools.ietf.org/html/draft-iab-protocol-maintenance-01) on this. -However, you can pass set parameter `ignore_comments` to true in the `parse` function to ignore `//` or `/* */` comments. Comments will then be treated as whitespace. +However, you can set set parameter `ignore_comments` to true in the `parse` function to ignore `//` or `/* */` comments. Comments will then be treated as whitespace. + +### Trailing commas + +Trailing commas in arrays and objects are also not part of the [JSON specification](https://tools.ietf.org/html/rfc8259), and this library does not support it by default. + +Like comments, you can set parameter `ignore_trailing_commas` to true in the `parse` function to ignore trailing commas in arrays and objects. Note that a single comma as the only content of the array or object (`[,]` or `{,}`) is not allowed, and multiple trailing commas (`[1,,]`) are not allowed either. + +This library does not add trailing commas when serializing JSON data. + +For more information, see [JWCC](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html) (JSON With Commas and Comments). ### Order of object keys diff --git a/docs/mkdocs/docs/api/basic_json/accept.md b/docs/mkdocs/docs/api/basic_json/accept.md index 43d5eff982..7670a9fb38 100644 --- a/docs/mkdocs/docs/api/basic_json/accept.md +++ b/docs/mkdocs/docs/api/basic_json/accept.md @@ -4,12 +4,14 @@ // (1) template static bool accept(InputType&& i, - const bool ignore_comments = false); + const bool ignore_comments = false, + const bool ignore_trailing_commas = false); // (2) template static bool accept(IteratorType first, IteratorType last, - const bool ignore_comments = false); + const bool ignore_comments = false, + const bool ignore_trailing_commas = false); ``` Checks whether the input is valid JSON. @@ -50,6 +52,10 @@ Unlike the [`parse()`](parse.md) function, this function neither throws an excep : whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error (`#!cpp false`); (optional, `#!cpp false` by default) +`ignore_trailing_commas` (in) +: whether trailing commas in arrays or objects should be allowed (`#!cpp true`) or yield a parse error + (`#!cpp false`); (optional, `#!cpp false` by default) + `first` (in) : iterator to start of character range @@ -102,6 +108,7 @@ A UTF-8 byte order mark is silently ignored. - Added in version 3.0.0. - Ignoring comments via `ignore_comments` added in version 3.9.0. - Changed [runtime assertion](../../features/assertions.md) in case of `FILE*` null pointers to exception in version 3.11.4. +- Added `ignore_trailing_commas` in version 3.11.4. !!! warning "Deprecation" diff --git a/docs/mkdocs/docs/api/basic_json/parse.md b/docs/mkdocs/docs/api/basic_json/parse.md index 69d412f977..eacdb5a3cc 100644 --- a/docs/mkdocs/docs/api/basic_json/parse.md +++ b/docs/mkdocs/docs/api/basic_json/parse.md @@ -6,14 +6,16 @@ template static basic_json parse(InputType&& i, const parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false); + const bool ignore_comments = false, + const bool ignore_trailing_commas = false); // (2) template static basic_json parse(IteratorType first, IteratorType last, const parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false); + const bool ignore_comments = false, + const bool ignore_trailing_commas = false); ``` 1. Deserialize from a compatible input. @@ -56,6 +58,10 @@ static basic_json parse(IteratorType first, IteratorType last, : whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error (`#!cpp false`); (optional, `#!cpp false` by default) +`ignore_trailing_commas` (in) +: whether trailing commas in arrays or objects should be allowed (`#!cpp true`) or yield a parse error + (`#!cpp false`); (optional, `#!cpp false` by default) + `first` (in) : iterator to start of character range @@ -189,6 +195,34 @@ A UTF-8 byte order mark is silently ignored. --8<-- "examples/parse__allow_exceptions.output" ``` +??? example "Effect of `ignore_comments` parameter" + + The example below demonstrates the effect of the `ignore_comments` parameter in the `parse()` function. + + ```cpp + --8<-- "examples/comments.cpp" + ``` + + Output: + + ``` + --8<-- "examples/comments.output" + ``` + +??? example "Effect of `ignore_trailing_commas` parameter" + + The example below demonstrates the effect of the `ignore_trailing_commas` parameter in the `parse()` function. + + ```cpp + --8<-- "examples/trailing_commas.cpp" + ``` + + Output: + + ``` + --8<-- "examples/trailing_commas.output" + ``` + ## See also - [accept](accept.md) - check if the input is valid JSON @@ -200,6 +234,7 @@ A UTF-8 byte order mark is silently ignored. - Overload for contiguous containers (1) added in version 2.0.3. - Ignoring comments via `ignore_comments` added in version 3.9.0. - Changed [runtime assertion](../../features/assertions.md) in case of `FILE*` null pointers to exception in version 3.11.4. +- Added `ignore_trailing_commas` in version 3.11.4. !!! warning "Deprecation" diff --git a/docs/mkdocs/docs/api/basic_json/sax_parse.md b/docs/mkdocs/docs/api/basic_json/sax_parse.md index e2ac1b41d9..6c207b8f85 100644 --- a/docs/mkdocs/docs/api/basic_json/sax_parse.md +++ b/docs/mkdocs/docs/api/basic_json/sax_parse.md @@ -7,7 +7,8 @@ static bool sax_parse(InputType&& i, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false); + const bool ignore_comments = false, + const bool ignore_trailing_commas = false); // (2) template @@ -15,7 +16,8 @@ static bool sax_parse(IteratorType first, IteratorType last, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false); + const bool ignore_comments = false, + const bool ignore_trailing_commas = false); ``` Read from input and generate SAX events @@ -65,6 +67,10 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index : whether comments should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error (`#!cpp false`); (optional, `#!cpp false` by default) +`ignore_trailing_commas` (in) +: whether trailing commas in arrays or objects should be allowed (`#!cpp true`) or yield a parse error + (`#!cpp false`); (optional, `#!cpp false` by default) + `first` (in) : iterator to start of character range diff --git a/docs/mkdocs/docs/examples/comments.cpp b/docs/mkdocs/docs/examples/comments.cpp new file mode 100644 index 0000000000..fc8a0ac34a --- /dev/null +++ b/docs/mkdocs/docs/examples/comments.cpp @@ -0,0 +1,31 @@ + +#include +#include + +using json = nlohmann::json; + +int main() +{ + std::string s = R"( + { + // update in 2006: removed Pluto + "planets": ["Mercury", "Venus", "Earth", "Mars", + "Jupiter", "Uranus", "Neptune" /*, "Pluto" */] + } + )"; + + try + { + json j = json::parse(s); + } + catch (json::exception& e) + { + std::cout << e.what() << std::endl; + } + + json j = json::parse(s, + /* callback */ nullptr, + /* allow exceptions */ true, + /* ignore_comments */ true); + std::cout << j.dump(2) << '\n'; +} diff --git a/docs/mkdocs/docs/examples/comments.output b/docs/mkdocs/docs/examples/comments.output new file mode 100644 index 0000000000..6e0d1c4622 --- /dev/null +++ b/docs/mkdocs/docs/examples/comments.output @@ -0,0 +1,12 @@ +[json.exception.parse_error.101] parse error at line 3, column 9: syntax error while parsing object key - invalid literal; last read: ' { /'; expected string literal +{ + "planets": [ + "Mercury", + "Venus", + "Earth", + "Mars", + "Jupiter", + "Uranus", + "Neptune" + ] +} diff --git a/docs/mkdocs/docs/examples/trailing_commas.cpp b/docs/mkdocs/docs/examples/trailing_commas.cpp new file mode 100644 index 0000000000..32bf6874eb --- /dev/null +++ b/docs/mkdocs/docs/examples/trailing_commas.cpp @@ -0,0 +1,37 @@ +#include +#include + +using json = nlohmann::json; + +int main() +{ + std::string s = R"( + { + "planets": [ + "Mercury", + "Venus", + "Earth", + "Mars", + "Jupiter", + "Uranus", + "Neptune", + ] + } + )"; + + try + { + json j = json::parse(s); + } + catch (json::exception& e) + { + std::cout << e.what() << std::endl; + } + + json j = json::parse(s, + /* callback */ nullptr, + /* allow exceptions */ true, + /* ignore_comments */ false, + /* ignore_trailing_commas */ true); + std::cout << j.dump(2) << '\n'; +} diff --git a/docs/mkdocs/docs/examples/trailing_commas.output b/docs/mkdocs/docs/examples/trailing_commas.output new file mode 100644 index 0000000000..3b2c49eb77 --- /dev/null +++ b/docs/mkdocs/docs/examples/trailing_commas.output @@ -0,0 +1,12 @@ +[json.exception.parse_error.101] parse error at line 11, column 9: syntax error while parsing value - unexpected ']'; expected '[', '{', or a literal +{ + "planets": [ + "Mercury", + "Venus", + "Earth", + "Mars", + "Jupiter", + "Uranus", + "Neptune" + ] +} diff --git a/docs/mkdocs/docs/features/comments.md b/docs/mkdocs/docs/features/comments.md index e99cceb49d..b6508008c4 100644 --- a/docs/mkdocs/docs/features/comments.md +++ b/docs/mkdocs/docs/features/comments.md @@ -11,7 +11,9 @@ This library does not support comments *by default*. It does so for three reason 3. It is dangerous for interoperability if some libraries would add comment support while others don't. Please check [The Harmful Consequences of the Robustness Principle](https://tools.ietf.org/html/draft-iab-protocol-maintenance-01) on this. -However, you can pass set parameter `ignore_comments` to `#!c true` in the parse function to ignore `//` or `/* */` comments. Comments will then be treated as whitespace. +However, you can set parameter `ignore_comments` to `#!cpp true` in the [`parse`](../api/basic_json/parse.md) function to ignore `//` or `/* */` comments. Comments will then be treated as whitespace. + +For more information, see [JWCC](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html) (JSON With Commas and Comments). !!! example @@ -28,56 +30,11 @@ However, you can pass set parameter `ignore_comments` to `#!c true` in the parse When calling `parse` without additional argument, a parse error exception is thrown. If `ignore_comments` is set to `#! true`, the comments are ignored during parsing: ```cpp - #include - #include "json.hpp" - - using json = nlohmann::json; - - int main() - { - std::string s = R"( - { - // update in 2006: removed Pluto - "planets": ["Mercury", "Venus", "Earth", "Mars", - "Jupiter", "Uranus", "Neptune" /*, "Pluto" */] - } - )"; - - try - { - json j = json::parse(s); - } - catch (json::exception &e) - { - std::cout << e.what() << std::endl; - } - - json j = json::parse(s, - /* callback */ nullptr, - /* allow exceptions */ true, - /* ignore_comments */ true); - std::cout << j.dump(2) << '\n'; - } + --8<-- "examples/comments.cpp" ``` Output: ``` - [json.exception.parse_error.101] parse error at line 3, column 9: - syntax error while parsing object key - invalid literal; - last read: ' { /'; expected string literal - ``` - - ```json - { - "planets": [ - "Mercury", - "Venus", - "Earth", - "Mars", - "Jupiter", - "Uranus", - "Neptune" - ] - } + --8<-- "examples/comments.output" ``` diff --git a/docs/mkdocs/docs/features/trailing_commas.md b/docs/mkdocs/docs/features/trailing_commas.md new file mode 100644 index 0000000000..baf38d7186 --- /dev/null +++ b/docs/mkdocs/docs/features/trailing_commas.md @@ -0,0 +1,39 @@ +# Trailing Commas + +Like comments, this library does not support trailing commas in arrays and objects *by default*. + +You can set parameter `ignore_trailing_commas` to `#!cpp true` in the [`parse`](../api/basic_json/parse.md) function to allow trailing commas in arrays and objects. Note that a single comma as the only content of the array or object (`[,]` or `{,}`) is not allowed, and multiple trailing commas (`[1,,]`) are not allowed either. + +This library does not add trailing commas when serializing JSON data. + +For more information, see [JWCC](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html) (JSON With Commas and Comments). + +!!! example + + Consider the following JSON with trailing commas. + + ```json + { + "planets": [ + "Mercury", + "Venus", + "Earth", + "Mars", + "Jupiter", + "Uranus", + "Neptune", + ] + } + ``` + + When calling `parse` without additional argument, a parse error exception is thrown. If `ignore_trailing_commas` is set to `#! true`, the trailing commas are ignored during parsing: + + ```cpp + --8<-- "examples/trailing_commas.cpp" + ``` + + Output: + + ``` + --8<-- "examples/trailing_commas.output" + ``` diff --git a/docs/mkdocs/mkdocs.yml b/docs/mkdocs/mkdocs.yml index 7a89219fbc..24588816a2 100644 --- a/docs/mkdocs/mkdocs.yml +++ b/docs/mkdocs/mkdocs.yml @@ -67,6 +67,7 @@ nav: - features/binary_formats/ubjson.md - features/binary_values.md - features/comments.md + - features/trailing_commas.md - Element Access: - features/element_access/index.md - features/element_access/unchecked_access.md diff --git a/include/nlohmann/detail/input/parser.hpp b/include/nlohmann/detail/input/parser.hpp index c856d11675..8227f8fe72 100644 --- a/include/nlohmann/detail/input/parser.hpp +++ b/include/nlohmann/detail/input/parser.hpp @@ -71,10 +71,12 @@ class parser explicit parser(InputAdapterType&& adapter, parser_callback_t cb = nullptr, const bool allow_exceptions_ = true, - const bool skip_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas_ = false) : callback(std::move(cb)) - , m_lexer(std::move(adapter), skip_comments) + , m_lexer(std::move(adapter), ignore_comments) , allow_exceptions(allow_exceptions_) + , ignore_trailing_commas(ignore_trailing_commas_) { // read first token get_token(); @@ -384,11 +386,17 @@ class parser if (states.back()) // array { // comma -> next value + // or end of array (ignore_trailing_commas = true) if (get_token() == token_type::value_separator) { // parse a new value get_token(); - continue; + + // if ignore_trailing_commas and last_token is ], we can continue to "closing ]" + if (!(ignore_trailing_commas && last_token == token_type::end_array)) + { + continue; + } } // closing ] @@ -417,32 +425,39 @@ class parser // states.back() is false -> object // comma -> next value + // or end of object (ignore_trailing_commas = true) if (get_token() == token_type::value_separator) { - // parse key - if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::value_string)) - { - return sax->parse_error(m_lexer.get_position(), - m_lexer.get_token_string(), - parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr)); - } + get_token(); - if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string()))) + // if ignore_trailing_commas and last_token is }, we can continue to "closing }" + if (!(ignore_trailing_commas && last_token == token_type::end_object)) { - return false; - } + // parse key + if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string)) + { + return sax->parse_error(m_lexer.get_position(), + m_lexer.get_token_string(), + parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr)); + } - // parse separator (:) - if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator)) - { - return sax->parse_error(m_lexer.get_position(), - m_lexer.get_token_string(), - parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr)); - } + if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string()))) + { + return false; + } - // parse values - get_token(); - continue; + // parse separator (:) + if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator)) + { + return sax->parse_error(m_lexer.get_position(), + m_lexer.get_token_string(), + parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr)); + } + + // parse values + get_token(); + continue; + } } // closing } @@ -513,6 +528,8 @@ class parser lexer_t m_lexer; /// whether to throw exceptions in case of errors const bool allow_exceptions = true; + /// whether trailing commas in objects and arrays should be ignored (true) or signaled as errors (false) + const bool ignore_trailing_commas = false; }; } // namespace detail diff --git a/include/nlohmann/json.hpp b/include/nlohmann/json.hpp index c8b7626067..a114bce5ec 100644 --- a/include/nlohmann/json.hpp +++ b/include/nlohmann/json.hpp @@ -134,11 +134,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec InputAdapterType adapter, detail::parser_callback_tcb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false + const bool ignore_comments = false, + const bool ignore_trailing_commas = false ) { return ::nlohmann::detail::parser(std::move(adapter), - std::move(cb), allow_exceptions, ignore_comments); + std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); } private: @@ -4043,10 +4044,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static basic_json parse(InputType&& i, parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { basic_json result; - parser(detail::input_adapter(std::forward(i)), std::move(cb), allow_exceptions, ignore_comments).parse(true, result); // cppcheck-suppress[accessMoved,accessForwarded] + parser(detail::input_adapter(std::forward(i)), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); // cppcheck-suppress[accessMoved,accessForwarded] return result; } @@ -4058,10 +4060,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec IteratorType last, parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { basic_json result; - parser(detail::input_adapter(std::move(first), std::move(last)), std::move(cb), allow_exceptions, ignore_comments).parse(true, result); // cppcheck-suppress[accessMoved] + parser(detail::input_adapter(std::move(first), std::move(last)), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); // cppcheck-suppress[accessMoved] return result; } @@ -4070,10 +4073,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static basic_json parse(detail::span_input_adapter&& i, parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { basic_json result; - parser(i.get(), std::move(cb), allow_exceptions, ignore_comments).parse(true, result); // cppcheck-suppress[accessMoved] + parser(i.get(), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); // cppcheck-suppress[accessMoved] return result; } @@ -4081,26 +4085,29 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec /// @sa https://json.nlohmann.me/api/basic_json/accept/ template static bool accept(InputType&& i, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { - return parser(detail::input_adapter(std::forward(i)), nullptr, false, ignore_comments).accept(true); + return parser(detail::input_adapter(std::forward(i)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); } /// @brief check if the input is valid JSON /// @sa https://json.nlohmann.me/api/basic_json/accept/ template static bool accept(IteratorType first, IteratorType last, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { - return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments).accept(true); + return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); } JSON_HEDLEY_WARN_UNUSED_RESULT JSON_HEDLEY_DEPRECATED_FOR(3.8.0, accept(ptr, ptr + len)) static bool accept(detail::span_input_adapter&& i, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { - return parser(i.get(), nullptr, false, ignore_comments).accept(true); + return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); } /// @brief generate SAX events @@ -4110,11 +4117,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static bool sax_parse(InputType&& i, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { auto ia = detail::input_adapter(std::forward(i)); return format == input_format_t::json - ? parser(std::move(ia), nullptr, true, ignore_comments).sax_parse(sax, strict) + ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict) : detail::binary_reader(std::move(ia), format).sax_parse(format, sax, strict); } @@ -4125,11 +4133,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static bool sax_parse(IteratorType first, IteratorType last, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { auto ia = detail::input_adapter(std::move(first), std::move(last)); return format == input_format_t::json - ? parser(std::move(ia), nullptr, true, ignore_comments).sax_parse(sax, strict) + ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict) : detail::binary_reader(std::move(ia), format).sax_parse(format, sax, strict); } @@ -4144,12 +4153,13 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static bool sax_parse(detail::span_input_adapter&& i, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { auto ia = i.get(); return format == input_format_t::json // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) - ? parser(std::move(ia), nullptr, true, ignore_comments).sax_parse(sax, strict) + ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict) // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) : detail::binary_reader(std::move(ia), format).sax_parse(format, sax, strict); } diff --git a/single_include/nlohmann/json.hpp b/single_include/nlohmann/json.hpp index f0063fe8ec..68447268f7 100644 --- a/single_include/nlohmann/json.hpp +++ b/single_include/nlohmann/json.hpp @@ -12863,10 +12863,12 @@ class parser explicit parser(InputAdapterType&& adapter, parser_callback_t cb = nullptr, const bool allow_exceptions_ = true, - const bool skip_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas_ = false) : callback(std::move(cb)) - , m_lexer(std::move(adapter), skip_comments) + , m_lexer(std::move(adapter), ignore_comments) , allow_exceptions(allow_exceptions_) + , ignore_trailing_commas(ignore_trailing_commas_) { // read first token get_token(); @@ -13176,11 +13178,17 @@ class parser if (states.back()) // array { // comma -> next value + // or end of array (ignore_trailing_commas = true) if (get_token() == token_type::value_separator) { // parse a new value get_token(); - continue; + + // if ignore_trailing_commas and last_token is ], we can continue to "closing ]" + if (!(ignore_trailing_commas && last_token == token_type::end_array)) + { + continue; + } } // closing ] @@ -13209,32 +13217,39 @@ class parser // states.back() is false -> object // comma -> next value + // or end of object (ignore_trailing_commas = true) if (get_token() == token_type::value_separator) { - // parse key - if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::value_string)) - { - return sax->parse_error(m_lexer.get_position(), - m_lexer.get_token_string(), - parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr)); - } + get_token(); - if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string()))) + // if ignore_trailing_commas and last_token is }, we can continue to "closing }" + if (!(ignore_trailing_commas && last_token == token_type::end_object)) { - return false; - } + // parse key + if (JSON_HEDLEY_UNLIKELY(last_token != token_type::value_string)) + { + return sax->parse_error(m_lexer.get_position(), + m_lexer.get_token_string(), + parse_error::create(101, m_lexer.get_position(), exception_message(token_type::value_string, "object key"), nullptr)); + } - // parse separator (:) - if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator)) - { - return sax->parse_error(m_lexer.get_position(), - m_lexer.get_token_string(), - parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr)); - } + if (JSON_HEDLEY_UNLIKELY(!sax->key(m_lexer.get_string()))) + { + return false; + } - // parse values - get_token(); - continue; + // parse separator (:) + if (JSON_HEDLEY_UNLIKELY(get_token() != token_type::name_separator)) + { + return sax->parse_error(m_lexer.get_position(), + m_lexer.get_token_string(), + parse_error::create(101, m_lexer.get_position(), exception_message(token_type::name_separator, "object separator"), nullptr)); + } + + // parse values + get_token(); + continue; + } } // closing } @@ -13305,6 +13320,8 @@ class parser lexer_t m_lexer; /// whether to throw exceptions in case of errors const bool allow_exceptions = true; + /// whether trailing commas in objects and arrays should be ignored (true) or signaled as errors (false) + const bool ignore_trailing_commas = false; }; } // namespace detail @@ -20082,11 +20099,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec InputAdapterType adapter, detail::parser_callback_tcb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false + const bool ignore_comments = false, + const bool ignore_trailing_commas = false ) { return ::nlohmann::detail::parser(std::move(adapter), - std::move(cb), allow_exceptions, ignore_comments); + std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas); } private: @@ -23991,10 +24009,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static basic_json parse(InputType&& i, parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { basic_json result; - parser(detail::input_adapter(std::forward(i)), std::move(cb), allow_exceptions, ignore_comments).parse(true, result); // cppcheck-suppress[accessMoved,accessForwarded] + parser(detail::input_adapter(std::forward(i)), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); // cppcheck-suppress[accessMoved,accessForwarded] return result; } @@ -24006,10 +24025,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec IteratorType last, parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { basic_json result; - parser(detail::input_adapter(std::move(first), std::move(last)), std::move(cb), allow_exceptions, ignore_comments).parse(true, result); // cppcheck-suppress[accessMoved] + parser(detail::input_adapter(std::move(first), std::move(last)), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); // cppcheck-suppress[accessMoved] return result; } @@ -24018,10 +24038,11 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static basic_json parse(detail::span_input_adapter&& i, parser_callback_t cb = nullptr, const bool allow_exceptions = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { basic_json result; - parser(i.get(), std::move(cb), allow_exceptions, ignore_comments).parse(true, result); // cppcheck-suppress[accessMoved] + parser(i.get(), std::move(cb), allow_exceptions, ignore_comments, ignore_trailing_commas).parse(true, result); // cppcheck-suppress[accessMoved] return result; } @@ -24029,26 +24050,29 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec /// @sa https://json.nlohmann.me/api/basic_json/accept/ template static bool accept(InputType&& i, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { - return parser(detail::input_adapter(std::forward(i)), nullptr, false, ignore_comments).accept(true); + return parser(detail::input_adapter(std::forward(i)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); } /// @brief check if the input is valid JSON /// @sa https://json.nlohmann.me/api/basic_json/accept/ template static bool accept(IteratorType first, IteratorType last, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { - return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments).accept(true); + return parser(detail::input_adapter(std::move(first), std::move(last)), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); } JSON_HEDLEY_WARN_UNUSED_RESULT JSON_HEDLEY_DEPRECATED_FOR(3.8.0, accept(ptr, ptr + len)) static bool accept(detail::span_input_adapter&& i, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { - return parser(i.get(), nullptr, false, ignore_comments).accept(true); + return parser(i.get(), nullptr, false, ignore_comments, ignore_trailing_commas).accept(true); } /// @brief generate SAX events @@ -24058,11 +24082,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static bool sax_parse(InputType&& i, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { auto ia = detail::input_adapter(std::forward(i)); return format == input_format_t::json - ? parser(std::move(ia), nullptr, true, ignore_comments).sax_parse(sax, strict) + ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict) : detail::binary_reader(std::move(ia), format).sax_parse(format, sax, strict); } @@ -24073,11 +24098,12 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static bool sax_parse(IteratorType first, IteratorType last, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { auto ia = detail::input_adapter(std::move(first), std::move(last)); return format == input_format_t::json - ? parser(std::move(ia), nullptr, true, ignore_comments).sax_parse(sax, strict) + ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict) : detail::binary_reader(std::move(ia), format).sax_parse(format, sax, strict); } @@ -24092,12 +24118,13 @@ class basic_json // NOLINT(cppcoreguidelines-special-member-functions,hicpp-spec static bool sax_parse(detail::span_input_adapter&& i, SAX* sax, input_format_t format = input_format_t::json, const bool strict = true, - const bool ignore_comments = false) + const bool ignore_comments = false, + const bool ignore_trailing_commas = false) { auto ia = i.get(); return format == input_format_t::json // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) - ? parser(std::move(ia), nullptr, true, ignore_comments).sax_parse(sax, strict) + ? parser(std::move(ia), nullptr, true, ignore_comments, ignore_trailing_commas).sax_parse(sax, strict) // NOLINTNEXTLINE(hicpp-move-const-arg,performance-move-const-arg) : detail::binary_reader(std::move(ia), format).sax_parse(format, sax, strict); } diff --git a/tests/src/unit-class_parser.cpp b/tests/src/unit-class_parser.cpp index b48b800047..a71f9b8d03 100644 --- a/tests/src/unit-class_parser.cpp +++ b/tests/src/unit-class_parser.cpp @@ -206,6 +206,7 @@ class SaxCountdown : public nlohmann::json::json_sax_t json parser_helper(const std::string& s); bool accept_helper(const std::string& s); void comments_helper(const std::string& s); +void trailing_comma_helper(const std::string& s); json parser_helper(const std::string& s) { @@ -225,6 +226,8 @@ json parser_helper(const std::string& s) comments_helper(s); + trailing_comma_helper(s); + return j; } @@ -259,10 +262,11 @@ bool accept_helper(const std::string& s) // 6. check if this approach came to the same result CHECK(ok_noexcept == ok_noexcept_cb); - // 7. check if comments are properly ignored + // 7. check if comments or trailing commas are properly ignored if (ok_accept) { comments_helper(s); + trailing_comma_helper(s); } // 8. return result @@ -302,6 +306,38 @@ void comments_helper(const std::string& s) } } +void trailing_comma_helper(const std::string& s) +{ + json _; + + // parse/accept with default parser + CHECK_NOTHROW(_ = json::parse(s)); + CHECK(json::accept(s)); + + // parse/accept while allowing trailing commas + CHECK_NOTHROW(_ = json::parse(s, nullptr, false, false, true)); + CHECK(json::accept(s, false, true)); + + // note: [,] and {,} are not allowed + if (s.size() > 1 && (s.back() == ']' || s.back() == '}') && !_.empty()) + { + std::vector json_with_trailing_commas; + json_with_trailing_commas.push_back(s.substr(0, s.size() - 1) + " ," + s.back()); + json_with_trailing_commas.push_back(s.substr(0, s.size() - 1) + "," + s.back()); + json_with_trailing_commas.push_back(s.substr(0, s.size() - 1) + ", " + s.back()); + + for (const auto& json_with_trailing_comma : json_with_trailing_commas) + { + CAPTURE(json_with_trailing_comma) + CHECK_THROWS_AS(_ = json::parse(json_with_trailing_comma), json::parse_error); + CHECK(!json::accept(json_with_trailing_comma)); + + CHECK_NOTHROW(_ = json::parse(json_with_trailing_comma, nullptr, true, false, true)); + CHECK(json::accept(json_with_trailing_comma, false, true)); + } + } +} + } // namespace TEST_CASE("parser class") From 1e1d7b4786f75b41bf00ed8578f382ad998ee879 Mon Sep 17 00:00:00 2001 From: chirsz Date: Mon, 27 Jan 2025 18:32:37 +0800 Subject: [PATCH 2/5] Update README.md Co-authored-by: Niels Lohmann Signed-off-by: chirsz --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ead12212c3..6fd7d450e7 100644 --- a/README.md +++ b/README.md @@ -1773,7 +1773,7 @@ Like comments, you can set parameter `ignore_trailing_commas` to true in the `pa This library does not add trailing commas when serializing JSON data. -For more information, see [JWCC](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html) (JSON With Commas and Comments). +For more information, see [JSON With Commas and Comments (JWCC)](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html). ### Order of object keys From 44a50e7e2aa377234e165d12fc130d8ab976e8f8 Mon Sep 17 00:00:00 2001 From: chirsz-ever Date: Mon, 27 Jan 2025 18:46:42 +0800 Subject: [PATCH 3/5] Fix JWCC links Signed-off-by: chirsz-ever --- docs/mkdocs/docs/features/comments.md | 2 +- docs/mkdocs/docs/features/trailing_commas.md | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/mkdocs/docs/features/comments.md b/docs/mkdocs/docs/features/comments.md index b6508008c4..85da595f63 100644 --- a/docs/mkdocs/docs/features/comments.md +++ b/docs/mkdocs/docs/features/comments.md @@ -13,7 +13,7 @@ This library does not support comments *by default*. It does so for three reason However, you can set parameter `ignore_comments` to `#!cpp true` in the [`parse`](../api/basic_json/parse.md) function to ignore `//` or `/* */` comments. Comments will then be treated as whitespace. -For more information, see [JWCC](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html) (JSON With Commas and Comments). +For more information, see [JSON With Commas and Comments (JWCC)](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html). !!! example diff --git a/docs/mkdocs/docs/features/trailing_commas.md b/docs/mkdocs/docs/features/trailing_commas.md index baf38d7186..d3adbe3dd0 100644 --- a/docs/mkdocs/docs/features/trailing_commas.md +++ b/docs/mkdocs/docs/features/trailing_commas.md @@ -6,7 +6,7 @@ You can set parameter `ignore_trailing_commas` to `#!cpp true` in the [`parse`]( This library does not add trailing commas when serializing JSON data. -For more information, see [JWCC](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html) (JSON With Commas and Comments). +For more information, see [JSON With Commas and Comments (JWCC)](https://nigeltao.github.io/blog/2021/json-with-commas-comments.html). !!! example From 86d28891f50d3453e2de3d72d3ac0863b9c5e649 Mon Sep 17 00:00:00 2001 From: chirsz-ever Date: Mon, 27 Jan 2025 18:51:06 +0800 Subject: [PATCH 4/5] Improve documentation Signed-off-by: chirsz-ever --- README.md | 2 +- docs/mkdocs/docs/api/basic_json/accept.md | 2 +- docs/mkdocs/docs/api/basic_json/parse.md | 2 +- docs/mkdocs/docs/api/basic_json/sax_parse.md | 2 +- 4 files changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 6fd7d450e7..133bff54fe 100644 --- a/README.md +++ b/README.md @@ -1767,7 +1767,7 @@ However, you can set set parameter `ignore_comments` to true in the `parse` func ### Trailing commas -Trailing commas in arrays and objects are also not part of the [JSON specification](https://tools.ietf.org/html/rfc8259), and this library does not support it by default. +The JSON specification does not allow trailing commas in arrays and objects, and hence this library is treating them as parsing errors by default. Like comments, you can set parameter `ignore_trailing_commas` to true in the `parse` function to ignore trailing commas in arrays and objects. Note that a single comma as the only content of the array or object (`[,]` or `{,}`) is not allowed, and multiple trailing commas (`[1,,]`) are not allowed either. diff --git a/docs/mkdocs/docs/api/basic_json/accept.md b/docs/mkdocs/docs/api/basic_json/accept.md index 7670a9fb38..dfebb59e77 100644 --- a/docs/mkdocs/docs/api/basic_json/accept.md +++ b/docs/mkdocs/docs/api/basic_json/accept.md @@ -53,7 +53,7 @@ Unlike the [`parse()`](parse.md) function, this function neither throws an excep (`#!cpp false`); (optional, `#!cpp false` by default) `ignore_trailing_commas` (in) -: whether trailing commas in arrays or objects should be allowed (`#!cpp true`) or yield a parse error +: whether trailing commas in arrays or objects should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error (`#!cpp false`); (optional, `#!cpp false` by default) `first` (in) diff --git a/docs/mkdocs/docs/api/basic_json/parse.md b/docs/mkdocs/docs/api/basic_json/parse.md index eacdb5a3cc..12a8eed72f 100644 --- a/docs/mkdocs/docs/api/basic_json/parse.md +++ b/docs/mkdocs/docs/api/basic_json/parse.md @@ -59,7 +59,7 @@ static basic_json parse(IteratorType first, IteratorType last, (`#!cpp false`); (optional, `#!cpp false` by default) `ignore_trailing_commas` (in) -: whether trailing commas in arrays or objects should be allowed (`#!cpp true`) or yield a parse error +: whether trailing commas in arrays or objects should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error (`#!cpp false`); (optional, `#!cpp false` by default) `first` (in) diff --git a/docs/mkdocs/docs/api/basic_json/sax_parse.md b/docs/mkdocs/docs/api/basic_json/sax_parse.md index 6c207b8f85..effb8eef6f 100644 --- a/docs/mkdocs/docs/api/basic_json/sax_parse.md +++ b/docs/mkdocs/docs/api/basic_json/sax_parse.md @@ -68,7 +68,7 @@ The SAX event lister must follow the interface of [`json_sax`](../json_sax/index (`#!cpp false`); (optional, `#!cpp false` by default) `ignore_trailing_commas` (in) -: whether trailing commas in arrays or objects should be allowed (`#!cpp true`) or yield a parse error +: whether trailing commas in arrays or objects should be ignored and treated like whitespace (`#!cpp true`) or yield a parse error (`#!cpp false`); (optional, `#!cpp false` by default) `first` (in) From 7fb2ff29889040d47e2e22cca76466171bf6162c Mon Sep 17 00:00:00 2001 From: chirsz Date: Mon, 27 Jan 2025 19:17:04 +0800 Subject: [PATCH 5/5] Update docs/mkdocs/docs/features/trailing_commas.md Co-authored-by: Niels Lohmann Signed-off-by: chirsz --- docs/mkdocs/docs/features/trailing_commas.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/mkdocs/docs/features/trailing_commas.md b/docs/mkdocs/docs/features/trailing_commas.md index d3adbe3dd0..583e7cbffc 100644 --- a/docs/mkdocs/docs/features/trailing_commas.md +++ b/docs/mkdocs/docs/features/trailing_commas.md @@ -1,6 +1,6 @@ # Trailing Commas -Like comments, this library does not support trailing commas in arrays and objects *by default*. +Like [comments](comments.md), this library does not support trailing commas in arrays and objects *by default*. You can set parameter `ignore_trailing_commas` to `#!cpp true` in the [`parse`](../api/basic_json/parse.md) function to allow trailing commas in arrays and objects. Note that a single comma as the only content of the array or object (`[,]` or `{,}`) is not allowed, and multiple trailing commas (`[1,,]`) are not allowed either.