Generating Julia code from Google's Flatbuffers schema

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Generating Julia code from Google's Flatbuffers schema

Douglas Bates
I have been looking at http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia
Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Jeffrey Sarnoff
For others: FlatBuffers are Google's developer's revisit of Protocol Buffers; structured information serialization in "performance-critical applications" with many rereads of the schema-guided data serialization.
Why FlatBuffers? for design perspective,  facebook on the phone chooses FlatBuffers  for pragmatic application, FlatBuffers home for guides details and source code.  It is licensed under Apache v2.

On Friday, April 1, 2016 at 2:05:06 PM UTC-4, Douglas Bates wrote:
I have been looking at <a href="http://google.github.io/flatbuffers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fgoogle.github.io%2Fflatbuffers\46sa\75D\46sntz\0751\46usg\75AFQjCNHdUfAMW9LelJtbbYZ7xWcCkBRFFA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fgoogle.github.io%2Fflatbuffers\46sa\75D\46sntz\0751\46usg\75AFQjCNHdUfAMW9LelJtbbYZ7xWcCkBRFFA&#39;;return true;">http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (<a href="https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fwesm%2Ffeather&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNEpvZ9gX4KIQ_YtCjqvq8vU3HundA" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fwesm%2Ffeather\46sa\75D\46sntz\0751\46usg\75AFQjCNEpvZ9gX4KIQ_YtCjqvq8vU3HundA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fwesm%2Ffeather\46sa\75D\46sntz\0751\46usg\75AFQjCNEpvZ9gX4KIQ_YtCjqvq8vU3HundA&#39;;return true;">https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia
Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Tom Breloff
I just want to say that I am interested in this... probably more so for using raw FlatBuffers than the associated dataframes-type libraries.  Is there a FlatBuffers.jl?  I may get involved with development at some point if nothing exists.

On Wed, Apr 6, 2016 at 3:04 AM, Jeffrey Sarnoff <[hidden email]> wrote:
For others: FlatBuffers are Google's developer's revisit of Protocol Buffers; structured information serialization in "performance-critical applications" with many rereads of the schema-guided data serialization.
Why FlatBuffers? for design perspective,  facebook on the phone chooses FlatBuffers  for pragmatic application, FlatBuffers home for guides details and source code.  It is licensed under Apache v2.

On Friday, April 1, 2016 at 2:05:06 PM UTC-4, Douglas Bates wrote:
I have been looking at http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia

Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Douglas Bates
On Wednesday, April 6, 2016 at 7:59:37 AM UTC-5, Tom Breloff wrote:
I just want to say that I am interested in this... probably more so for using raw FlatBuffers than the associated dataframes-type libraries.  Is there a FlatBuffers.jl?  I may get involved with development at some point if nothing exists.

A Flatbuffers.jl package would be ideal but probably beyond my abilities.  The way I understand the process there would need to be extensions to the flatc compiler to be able to produce Julia code from schema.   Currently supported languages are C++, Go, Java, JavaScript, C#, Python and PHP. Each language also has a library written in that language that provides definitions of the top-level classes.
 
Wes McKinney added a C API to libfeather so I used that in Feather.jl package instead of trying to write code directly from the Flatbuffers schema.


On Wed, Apr 6, 2016 at 3:04 AM, Jeffrey Sarnoff <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="qSddlPoWAwAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jeffrey...@...> wrote:
For others: FlatBuffers are Google's developer's revisit of Protocol Buffers; structured information serialization in "performance-critical applications" with many rereads of the <a href="https://google.github.io/flatbuffers/flatbuffers_grammar.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgoogle.github.io%2Fflatbuffers%2Fflatbuffers_grammar.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFacvK62fNaEhJrYbG2MRh4kkCn9g&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgoogle.github.io%2Fflatbuffers%2Fflatbuffers_grammar.html\46sa\75D\46sntz\0751\46usg\75AFQjCNFacvK62fNaEhJrYbG2MRh4kkCn9g&#39;;return true;">schema-guided data serialization.
<a href="https://google.github.io/flatbuffers/flatbuffers_white_paper.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgoogle.github.io%2Fflatbuffers%2Fflatbuffers_white_paper.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHW8SRtvOA3tmd4kiEvWbADPpMoqQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgoogle.github.io%2Fflatbuffers%2Fflatbuffers_white_paper.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHW8SRtvOA3tmd4kiEvWbADPpMoqQ&#39;;return true;">Why FlatBuffers? for design perspective,  <a href="https://code.facebook.com/posts/872547912839369/improving-facebook-s-performance-on-android-with-flatbuffers/" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fcode.facebook.com%2Fposts%2F872547912839369%2Fimproving-facebook-s-performance-on-android-with-flatbuffers%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGORi4e4aZLL25C8EcnFTyPfhubrQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fcode.facebook.com%2Fposts%2F872547912839369%2Fimproving-facebook-s-performance-on-android-with-flatbuffers%2F\46sa\75D\46sntz\0751\46usg\75AFQjCNGORi4e4aZLL25C8EcnFTyPfhubrQ&#39;;return true;">facebook on the phone chooses FlatBuffers  for pragmatic application, <a href="https://google.github.io/flatbuffers/index.html" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgoogle.github.io%2Fflatbuffers%2Findex.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHv48k4lFuXzEAXeLbbXZ2_tL33VQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgoogle.github.io%2Fflatbuffers%2Findex.html\46sa\75D\46sntz\0751\46usg\75AFQjCNHv48k4lFuXzEAXeLbbXZ2_tL33VQ&#39;;return true;">FlatBuffers home for guides details and <a href="https://github.com/google/flatbuffers" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fgoogle%2Fflatbuffers\46sa\75D\46sntz\0751\46usg\75AFQjCNGfm8n-qXkxWyBZdDJW6snAkIfKKQ&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fgoogle%2Fflatbuffers\46sa\75D\46sntz\0751\46usg\75AFQjCNGfm8n-qXkxWyBZdDJW6snAkIfKKQ&#39;;return true;">source code.  It is licensed under Apache v2.

On Friday, April 1, 2016 at 2:05:06 PM UTC-4, Douglas Bates wrote:
I have been looking at <a href="http://google.github.io/flatbuffers" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fgoogle.github.io%2Fflatbuffers\46sa\75D\46sntz\0751\46usg\75AFQjCNHdUfAMW9LelJtbbYZ7xWcCkBRFFA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\75http%3A%2F%2Fgoogle.github.io%2Fflatbuffers\46sa\75D\46sntz\0751\46usg\75AFQjCNHdUfAMW9LelJtbbYZ7xWcCkBRFFA&#39;;return true;">http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (<a href="https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Fwesm%2Ffeather&amp;sa=D&amp;sntz=1&amp;usg=AFQjCNEpvZ9gX4KIQ_YtCjqvq8vU3HundA" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fwesm%2Ffeather\46sa\75D\46sntz\0751\46usg\75AFQjCNEpvZ9gX4KIQ_YtCjqvq8vU3HundA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2Fwesm%2Ffeather\46sa\75D\46sntz\0751\46usg\75AFQjCNEpvZ9gX4KIQ_YtCjqvq8vU3HundA&#39;;return true;">https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia

Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Tom Breloff
If I understand correctly, a proper FlatBuffers.jl implementation would have 2 stages:

  1. Pure-Julia code to define FlatBuffers structs, with a getter/setter API
  2. A code-generation tool which would produce boilerplate Julia code (#1) for a given FlatBuffers schema file
So one could use/interact with FlatBuffers data once we have #1.  If anyone starts work on this, please post here!

On Wed, Apr 6, 2016 at 11:02 AM, Douglas Bates <[hidden email]> wrote:
On Wednesday, April 6, 2016 at 7:59:37 AM UTC-5, Tom Breloff wrote:
I just want to say that I am interested in this... probably more so for using raw FlatBuffers than the associated dataframes-type libraries.  Is there a FlatBuffers.jl?  I may get involved with development at some point if nothing exists.

A Flatbuffers.jl package would be ideal but probably beyond my abilities.  The way I understand the process there would need to be extensions to the flatc compiler to be able to produce Julia code from schema.   Currently supported languages are C++, Go, Java, JavaScript, C#, Python and PHP. Each language also has a library written in that language that provides definitions of the top-level classes.
 
Wes McKinney added a C API to libfeather so I used that in Feather.jl package instead of trying to write code directly from the Flatbuffers schema.


On Wed, Apr 6, 2016 at 3:04 AM, Jeffrey Sarnoff <[hidden email]> wrote:
For others: FlatBuffers are Google's developer's revisit of Protocol Buffers; structured information serialization in "performance-critical applications" with many rereads of the schema-guided data serialization.
Why FlatBuffers? for design perspective,  facebook on the phone chooses FlatBuffers  for pragmatic application, FlatBuffers home for guides details and source code.  It is licensed under Apache v2.

On Friday, April 1, 2016 at 2:05:06 PM UTC-4, Douglas Bates wrote:
I have been looking at http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia


Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Kevin Squire
Likely not directly useful, but Ragel does something similar, and Daniel Jones spent some time getting it to produce julia from a spec.


It won't read FlatBuffers spec files, though.

   Kevin

On Wed, Apr 6, 2016 at 8:23 AM, Tom Breloff <[hidden email]> wrote:
If I understand correctly, a proper FlatBuffers.jl implementation would have 2 stages:

  1. Pure-Julia code to define FlatBuffers structs, with a getter/setter API
  2. A code-generation tool which would produce boilerplate Julia code (#1) for a given FlatBuffers schema file
So one could use/interact with FlatBuffers data once we have #1.  If anyone starts work on this, please post here!

On Wed, Apr 6, 2016 at 11:02 AM, Douglas Bates <[hidden email]> wrote:
On Wednesday, April 6, 2016 at 7:59:37 AM UTC-5, Tom Breloff wrote:
I just want to say that I am interested in this... probably more so for using raw FlatBuffers than the associated dataframes-type libraries.  Is there a FlatBuffers.jl?  I may get involved with development at some point if nothing exists.

A Flatbuffers.jl package would be ideal but probably beyond my abilities.  The way I understand the process there would need to be extensions to the flatc compiler to be able to produce Julia code from schema.   Currently supported languages are C++, Go, Java, JavaScript, C#, Python and PHP. Each language also has a library written in that language that provides definitions of the top-level classes.
 
Wes McKinney added a C API to libfeather so I used that in Feather.jl package instead of trying to write code directly from the Flatbuffers schema.


On Wed, Apr 6, 2016 at 3:04 AM, Jeffrey Sarnoff <[hidden email]> wrote:
For others: FlatBuffers are Google's developer's revisit of Protocol Buffers; structured information serialization in "performance-critical applications" with many rereads of the schema-guided data serialization.
Why FlatBuffers? for design perspective,  facebook on the phone chooses FlatBuffers  for pragmatic application, FlatBuffers home for guides details and source code.  It is licensed under Apache v2.

On Friday, April 1, 2016 at 2:05:06 PM UTC-4, Douglas Bates wrote:
I have been looking at http://google.github.io/flatbuffers because the schema for the feather binary file storage of data frames (https://github.com/wesm/feather) is defined using it.  I wanted to check if anyone has thought about Flatbuffers and Julia



Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Douglas Bates
In reply to this post by Tom Breloff


On Wednesday, April 6, 2016 at 10:23:18 AM UTC-5, Tom Breloff wrote:
If I understand correctly, a proper FlatBuffers.jl implementation would have 2 stages:

  1. Pure-Julia code to define FlatBuffers structs, with a getter/setter API
  2. A code-generation tool which would produce boilerplate Julia code (#1) for a given FlatBuffers schema file
So one could use/interact with FlatBuffers data once we have #1.  If anyone starts work on this, please post here!
 
It seems from the code in https://github.com/google/flatbuffers that it would be necessary to create src/idl_gen_julia.cpp
for #2 and a separate julia directory for #1
Reply | Threaded
Open this post in threaded view
|

Re: Generating Julia code from Google's Flatbuffers schema

Douglas Bates
I have a preliminary Julia implementation of the reader in https://github.com/dmbates/FlatBuffers.jl