![Key code javascript](https://cdn1.cdnme.se/5447227/9-3/22_64e61dfde087c31b3e70ff43.png)
For CloudQuery this is important as it makes it much easier to develop source or destination plugins in different languages. Cross-language with extensive libraries for different languages - The format is defined via flatbuffers in such way that you can parse it in any language and already has extensive support in C/C++, C#, Go, Java, JavaScript, Julia, Matlab, Python, R, Ruby and Rust (at the time of writing).Apache Arrow defines a language-independent columnar format for flat and hierarchical data, and brings the following advantages: Also, performance-wise, lots of the time spent in an ELT process is around converting data from one format to another, so we wanted to take a step back and see if we can avoid this famous XKCD (by building yet another format): For example, in database to database replication, we needed to support many more types, including nested types. This served us well, but we started to hit limitations in various use-cases.
![key code javascript key code javascript](https://www.exeideas.com/wp-content/uploads/2014/05/List-Of-All-Keyboard-Keys-JavaScript-Char-Codes-Key-Codes.jpg)
Why Arrow?īefore Arrow, we used our own type system that supported more than 14 types. In Arrow terminology, these are a schema and a record batch. So to recap, the source plugin sends mainly two things to a destination: 1) the schema 2) the records that fit the defined schema.
![key code javascript key code javascript](https://www.codegrepper.com/codeimages/insert-value-to-html-input-with-javascript-variable.png)
The destination plugin can then easily create the schema for its database and transform the incoming data to the destination types. Source plugins extract information from APIs in the most performant way possible, defining a schema and then transforming the result from the API (JSON or any other format) to a well-defined type system. This is crucial to allowing the addition of new destinations and updating old destinations without requiring updates to source plugin code (which otherwise would introduce an unmaintainable architecture).
![key code javascript key code javascript](https://www.codegrepper.com/codeimages/what-is-keycode-32-javascript.png)
Sources and destinations are decoupled and communicate via gRPC.
![Key code javascript](https://cdn1.cdnme.se/5447227/9-3/22_64e61dfde087c31b3e70ff43.png)