GH-51395: [C++][Docs] Replace RapidJSON with simdjson in row converter example by Reranko05 · Pull Request #51396 · apache/arrow

Reranko05 · 2026-09-19T19:44:20Z

Rationale for this change

This PR continues the simdjson migration by replacing RapidJSON usage in the C++ example demonstrating row-to-columnar and columnar-to-row conversion using simdjson and Apache Arrow.

Changes

Add simdjson_row_converter.cc for JSON parsing, Arrow array construction, and JSON serialization.
Delete rapidjson_row_converter.cc.
Update example build configuration and documentation.

Was AI used for this PR?

In accordance to the AI generation guidelines, please disclose below whether and how AI was used in this PR.

PR code and description written by:

Human
AI

Reviewed before submission by:

Human
AI
Not reviewed

GitHub Issue: [C++][Docs] Replace RapidJSON with simdjson in row converter example #51395

Reranko05 · 2026-09-20T18:48:59Z

Will mark this PR Ready to Review after #51038 is merged.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

pitrou · 2026-10-08T15:36:48Z

+//  * arrow::TableBatchReader and Arrow iterators for converting Arrow tables back
+//    into row-based JSON data
+
+namespace arrow {


Example code shouldn't be in the arrow namespace because it showcases how third-party code using Arrow C++ could look like.

Ah, okay. I got it.

pitrou · 2026-10-08T15:37:01Z

+//  * arrow::internal::JsonWriter for writing JSON values
+//  * arrow::internal::ParseJsonObject and related helpers for parsing JSON


Examples should not use any internal Arrow APIs.

Okay, understood.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-10-11T09:52:15Z

I have not got the time to review this myself after changing it into plain simdjson. I will mark it as Ready for review after I review this.

Copilot

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

Reranko05 · 2026-10-16T11:20:48Z

@pitrou @kou Could you review this when you have time? (CI failures seem unrelated)

kou · 2026-10-16T13:58:11Z

+  // Parse each row and append its values to the corresponding Arrow builders.
+  for (const auto& json : rows) {
+    ARROW_ASSIGN_OR_RAISE(auto element,
+                          Unwrap(parser.parse(json), "Failed to parse JSON row"));
+    ARROW_ASSIGN_OR_RAISE(auto object,
+                          Unwrap(element.get_object(), "Expected a JSON object"));
+
+    for (int i = 0; i < schema->num_fields(); ++i) {
+      const auto& field = schema->field(i);
+      arrow::ArrayBuilder* builder = batch_builder->GetField(i);
+
+      ARROW_ASSIGN_OR_RAISE(auto value, GetField(object, field->name()));
+
+      if (!value.has_value()) {
+        ARROW_RETURN_NOT_OK(builder->AppendNull());
+      } else {
+        ARROW_RETURN_NOT_OK(AppendJsonValue(*value, field->type(), builder));
+      }
+    }
+  }


Can we keep column based iteration (not row based iteration) like the RapidJSON version so that we demonstrate how to use a visitor to convert row based data to Arrow data?

I changed the impl to do column based iteration.

kou · 2026-10-16T14:03:30Z

+  for (int64_t i = 0; i < batch->num_rows(); ++i) {
+    ARROW_ASSIGN_OR_RAISE(auto row, ConvertRowToJson(*batch, i));
+    rows.push_back(std::make_shared<std::string>(std::move(row)));
+  }


Can we keep column based iteration (not row based iteration) like the RapidJSON version so that we demonstrate how to use a visitor to convert Arrow data to row based data?

As said above.

Reranko05 · 2026-10-22T04:24:29Z

[1342/2536] Building CXX object _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o
FAILED: [code=1] _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o 
/opt/homebrew/bin/ccache /usr/bin/c++ -DARROW_HAVE_NEON -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/address_sorting/include -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/ext/upb-gen -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/ext/upbdefs-gen -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/upb -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/utf8_range -I/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/third_party/xxhash -I/Users/runner/work/arrow/arrow/build/cpp/_deps/cares-build -I/Users/runner/work/arrow/arrow/build/cpp/_deps/cares-src -I/Users/runner/work/arrow/arrow/build/cpp/_deps/cares-src/include -isystem /Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/include -isystem /Users/runner/work/arrow/arrow/build/cpp/_deps/re2-src -isystem /Users/runner/work/arrow/arrow/build/cpp/zlib_ep/src/zlib_ep-install/include -isystem /Users/runner/work/arrow/arrow/build/cpp/_deps/absl-src -isystem /opt/homebrew/opt/openssl@4/include -fno-aligned-new  -I/opt/homebrew/Cellar/libiodbc/3.52.16/include -Qunused-arguments -fcolor-diagnostics  -Wall -Wno-unknown-warning-option -Wno-pass-failed -march=armv8-a  -Wno-attributes -Wno-format-security -Wno-unknown-warning-option  -g -O0 -ggdb -g1 -Wno-error -std=c++20 -arch arm64 -fPIC -Wno-error=deprecated-declarations -MD -MT _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o -MF _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o.d -o _deps/grpc-build/CMakeFiles/grpc.dir/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc.o -c /Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc
/Users/runner/work/arrow/arrow/build/cpp/_deps/grpc-src/src/core/credentials/transport/tls/grpc_tls_crl_provider.cc:59:14: error: cannot initialize a variable of type 'X509_NAME *' (aka 'X509_name_st *') with an rvalue of type 'const X509_NAME *' (aka 'const X509_name_st *')
   59 |   X509_NAME* issuer = X509_CRL_get_issuer(crl);
      |              ^        ~~~~~~~~~~~~~~~~~~~~~~~~

@kou @pitrou @taepper I am a bit confused about whether this failure is related to my PR, since another PR passes the same CI job even after I rebased mine. Could someone clarify?

Reranko05 · 2026-10-22T04:35:58Z

BTW, I have not yet updated the .rst doc file, I will do that once the simdjson_row_converter.cc changes are approved.

taepper · 2026-10-22T09:31:58Z

Hi, yes, I can take a look at this

Can you quickly confirm my understanding re/ the commit history because I am a little bit confused (and the history has since been rebased & force-pushed)

The initial commit Replace example was the first draft and the subsequent commits (which change the structure considerably), were the iterations based on review feedback?

@taepper I am a bit confused about whether this failure is related to my PR, since another #54839 passes the same CI job even after I rebased mine. Could someone clarify?

Yes, I would also think that your change to CMakeLists.txt is unrelated

pitrou · 2026-10-22T09:35:31Z

@kou @pitrou @taepper I am a bit confused about whether this failure is related to my PR

It's not, we see it on other PRs as well.

Reranko05 · 2026-10-22T14:51:28Z

Oh okay, thanks for confirming :)
This is ready for review.
cc @kou @pitrou @taepper

Reranko05 · 2026-10-22T15:04:05Z

were the iterations based on review feedback?

@taepper I initially used internal Arrow APIs to write the example code, then I got to know that I should write it in plain simdjson and then it was that my impl was row based but we needed column based iteration, so yeah they were considerable feedback changes 😅

Reranko05 · 2026-10-26T12:17:56Z

Hi @kou @pitrou @taepper, could you please review this when you get a chance? Once this is reviewed, I will open the final cleanup PR (#55266). Thanks!

kou

Could you update the documents and the PR description?

taepper

(I actually found the row-based conversion to arrow more readable [and it would also allow usage of the simdjson's ondemand backend]. The visit on arrow arrays can still be demonstrated for the back-conversion to json in JsonColumnWriter.)

Splitting the two conversions json->arrow and arrow->json into two separate files would make this more well-organized? I would also imagine any user that wants to follow this code example in their own project should organise it in separate files as well

taepper · 2026-10-26T13:59:31Z

-      ARROW_RETURN_NOT_OK(
-          AppendJsonValue(*child, field->type(), builder->child_builder(i).get()));
+// A JSON value, or std::nullopt if it is missing.
+using JsonValue = std::optional<simdjson::dom::element>;


Ugh, do we really need this special type? Can't we append a simdjson null document in those cases instead?

Also, the code comment below mentions:

Missing fields, and fields of null objects, are returned as std::nullopt

but I do not think this is true? The way I understand it is that fields of null objects are returned as null, but in case of missing fields this:

ARROW_ASSIGN_OR_RAISE(auto object, Unwrap(value->get_object(), "Failed to get JSON object")); ARROW_ASSIGN_OR_RAISE(auto child, GetField(object, name));

will raise an error instead?

Overall, I think avoiding this special type could make the example easier to understand

I tested with a missing metrics field, and conversion succeeds, producing an Arrow null without errors. I think using a simdjson null value instead might require more code than the current std::optional approach. Could you explain how you'd implement this using the DOM API?

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

Reranko05 · 2026-10-26T18:55:53Z

Could you update the documents and the PR description?

@kou Addressed feedbacks and updated docs and PR description.

Reranko05 · 2026-10-26T18:57:31Z

(I actually found the row-based conversion to arrow more readable [and it would also allow usage of the simdjson's ondemand backend]. The visit on arrow arrays can still be demonstrated for the back-conversion to json in JsonColumnWriter.)

I am fine with any approach. @kou @pitrou what do you think?

Splitting the two conversions json->arrow and arrow->json into two separate files would make this more well-organized?

Same as above

Copilot

Copilot was unable to review this pull request because the user who requested the review has insufficient remaining quota.

github-actions Bot added Component: C++ awaiting review Awaiting review labels Sep 19, 2026

Reranko05 added the CI: Extra: C++ Run extra C++ CI label Sep 19, 2026

Reranko05 changed the title ~~GH-51395: [C++] Replace RapidJSON with Simdjson in row converter example~~ GH-51395: [C++] Replace RapidJSON with simdjson in row converter example Sep 19, 2026

Reranko05 force-pushed the example-doc branch from 3842058 to 9813fc2 Compare September 19, 2026 20:23

github-actions Bot added the Component: Documentation label Sep 19, 2026

Reranko05 changed the title ~~GH-51395: [C++] Replace RapidJSON with simdjson in row converter example~~ GH-51395: [C++][Docs] Replace RapidJSON with simdjson in row converter example Sep 19, 2026

Reranko05 force-pushed the example-doc branch from ebf072f to 4934519 Compare October 8, 2026 08:29

Reranko05 marked this pull request as ready for review October 8, 2026 09:37

Reranko05 requested a review from pitrou as a code owner October 8, 2026 09:37

Copilot AI balanced review requested due to automatic review settings October 8, 2026 09:37

Copilot AI reviewed Oct 8, 2026

Reranko05 requested a review from kou October 8, 2026 09:37

pitrou reviewed Oct 8, 2026

View reviewed changes

github-actions Bot added awaiting committer review Awaiting committer review and removed awaiting review Awaiting review labels Oct 8, 2026

Copilot AI balanced review requested due to automatic review settings October 9, 2026 09:20

Copilot AI reviewed Oct 9, 2026

Copilot AI balanced review requested due to automatic review settings October 9, 2026 09:40

Reranko05 force-pushed the example-doc branch from 84ed127 to 7b885c3 Compare October 9, 2026 09:40

Copilot AI reviewed Oct 9, 2026

Reranko05 marked this pull request as draft October 11, 2026 09:52

Reranko05 marked this pull request as ready for review October 16, 2026 11:19

Copilot AI balanced review requested due to automatic review settings October 16, 2026 11:19

Copilot AI reviewed Oct 16, 2026

kou reviewed Oct 16, 2026

View reviewed changes

Reranko05 added 7 commits October 22, 2026 09:40

Replace example

f8212f5

Update rst doc

646b0ef

Minor fixes

025a091

Self Review

3bc2678

Use plain simdjson

6470cc1

Add get_uint64()

5dac63e

Change to column based iteration

3673c4d

Reranko05 force-pushed the example-doc branch from 9519b33 to 3673c4d Compare October 22, 2026 04:11

Reranko05 mentioned this pull request Oct 26, 2026

GH-55265: [C++][Cleanup] Remove remaining RapidJSON references #55266

Draft

3 of 5 tasks

kou reviewed Oct 26, 2026

View reviewed changes

Comment thread cpp/examples/arrow/CMakeLists.txt Outdated

Comment thread cpp/examples/arrow/simdjson_row_converter.cc Outdated

Comment thread cpp/examples/arrow/simdjson_row_converter.cc Outdated

github-actions Bot added awaiting changes Awaiting changes and removed awaiting change review Awaiting change review labels Oct 26, 2026

taepper reviewed Oct 26, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings October 26, 2026 18:46

Copilot AI reviewed Oct 26, 2026

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 26, 2026

Address Feedback

a5d38cd

Reranko05 force-pushed the example-doc branch from a9f6790 to a5d38cd Compare October 26, 2026 19:14

Copilot AI balanced review requested due to automatic review settings October 26, 2026 19:14

Copilot AI reviewed Oct 26, 2026

		// * arrow::internal::JsonWriter for writing JSON values
		// * arrow::internal::ParseJsonObject and related helpers for parsing JSON

Conversation

Reranko05 commented Sep 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

Changes

Was AI used for this PR?

Uh oh!

Reranko05 commented Sep 20, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 11, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

taepper commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pitrou commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 22, 2026

Uh oh!

Reranko05 commented Oct 26, 2026

Uh oh!

kou left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

taepper left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Reranko05 commented Oct 26, 2026

Uh oh!

Reranko05 commented Oct 26, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Reranko05 commented Sep 19, 2026 •

edited

Loading

taepper commented Oct 22, 2026 •

edited

Loading