Julia2C initial release

classic Classic list List threaded Threaded
45 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Julia2C initial release

Hongbo Rong
This is to announce the release of Julia2C, a source-to-source translator from Julia to C. This initial version converts basic Julia types and expressions into the corresponding C types and statements. It is open sourced at   https://github.com/IntelLabs/julia/tree/j2c. You can download and start playing with the code. 

Enjoy!

Hongbo
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Jey Kottalam
Interesting. What are the sorts of usecases envisioned for this?

On Wed, Nov 12, 2014 at 11:22 AM, Hongbo Rong <[hidden email]> wrote:
This is to announce the release of Julia2C, a source-to-source translator from Julia to C. This initial version converts basic Julia types and expressions into the corresponding C types and statements. It is open sourced at   https://github.com/IntelLabs/julia/tree/j2c. You can download and start playing with the code. 

Enjoy!

Hongbo

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tim Holy
In reply to this post by Hongbo Rong
Interesting! I'd be even happier about a translator that works in the opposite
direction. Have you given that any thought?

Best,
--Tim

On Wednesday, November 12, 2014 11:22:51 AM Hongbo Rong wrote:
> This is to announce the release of Julia2C, a source-to-source translator
> from Julia to C. This initial version converts basic Julia types and
> expressions into the corresponding C types and statements. It is open
> sourced at   https://github.com/IntelLabs/julia/tree/j2c. You can download
> and start playing with the code.
>
> Enjoy!
>
> Hongbo

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
In reply to this post by Jey Kottalam
By translating Julia to C, we leverage the high-level abstractions (matrix, vector, ..), which are easier to analyze, and can potentially add the rich extensions of C (like openmp, tbb, ...). 

The tool may also extend Julia to new architectures where the only available tool chain is for C

On Wednesday, November 12, 2014 2:06:13 PM UTC-8, Jey Kottalam wrote:
Interesting. What are the sorts of usecases envisioned for this?

On Wed, Nov 12, 2014 at 11:22 AM, Hongbo Rong <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="KfIqnTKsqIMJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">rongh...@...> wrote:
This is to announce the release of Julia2C, a source-to-source translator from Julia to C. This initial version converts basic Julia types and expressions into the corresponding C types and statements. It is open sourced at   <a href="https://github.com/IntelLabs/julia/tree/j2c" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FIntelLabs%2Fjulia%2Ftree%2Fj2c\46sa\75D\46sntz\0751\46usg\75AFQjCNHRXfiNE7By2nwf4Htq1qalaN9cyw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FIntelLabs%2Fjulia%2Ftree%2Fj2c\46sa\75D\46sntz\0751\46usg\75AFQjCNHRXfiNE7By2nwf4Htq1qalaN9cyw';return true;">https://github.com/IntelLabs/julia/tree/j2c. You can download and start playing with the code. 

Enjoy!

Hongbo

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
In reply to this post by Tim Holy
Translation from C to Julia might be harder. Some issues have to be addressed: (1) how to recognize and lift up the abstraction level of data structures and their operations? For example, how to raise a uint* to Vector object, and a loop of a[i]+b[i] to vectorA + vectorB? (2) Memory management: All alloc and free have to be replaced and hooked to garbage collector. If the C code uses customized memory allocators, memory pools, etc. it can cause problems. (3) Extensions like pragmas, embedded assembly, intrinsics, etc. could bring some implementation restrictions.

On Wednesday, November 12, 2014 2:11:31 PM UTC-8, Tim wrote:
Interesting! I'd be even happier about a translator that works in the opposite
direction. Have you given that any thought?

Best,
--Tim

On Wednesday, November 12, 2014 11:22:51 AM Hongbo Rong wrote:
> This is to announce the release of Julia2C, a source-to-source translator
> from Julia to C. This initial version converts basic Julia types and
> expressions into the corresponding C types and statements. It is open
> sourced at   <a href="https://github.com/IntelLabs/julia/tree/j2c" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FIntelLabs%2Fjulia%2Ftree%2Fj2c\46sa\75D\46sntz\0751\46usg\75AFQjCNHRXfiNE7By2nwf4Htq1qalaN9cyw';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FIntelLabs%2Fjulia%2Ftree%2Fj2c\46sa\75D\46sntz\0751\46usg\75AFQjCNHRXfiNE7By2nwf4Htq1qalaN9cyw';return true;">https://github.com/IntelLabs/julia/tree/j2c. You can download
> and start playing with the code.
>
> Enjoy!
>
> Hongbo

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Keith Campbell
In reply to this post by Jey Kottalam
Maybe a way to get to a standalone executable?

On Wednesday, November 12, 2014 5:06:13 PM UTC-5, Jey Kottalam wrote:
Interesting. What are the sorts of usecases envisioned for this?


Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tom Short

Wow. That's a lot of work. It's very entwined in the core Julia code. Is there hope that it could be separated into a package?

Or, are you hoping to get it merged into Julia?

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tony Kelman-2
If I were the Mathworks, this would scare the crap out of me. Simulink Real-time Workshop and C code generation for embedded real-time control applications is a substantial industrial use case for Matlab/Simulink that hasn't really been replicated by any high-level open source language as of yet.

I'll have to look at this in more detail and try it out. Does this expand macros? Include code used from Julia packages? Does the generated C require any pieces of the libjulia runtime library?


On Wednesday, November 12, 2014 6:46:58 PM UTC-8, Tom Short wrote:

Wow. That's a lot of work. It's very entwined in the core Julia code. Is there hope that it could be separated into a package?

Or, are you hoping to get it merged into Julia?

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
In reply to this post by Keith Campbell

Yes, this can be a possible usage.

So far, the concept is: the user specifies 1 Julia function to be translated into native C; then in the Julia code generation phase, we recursively compile this function and all its direct and indirect callees into C; that is, the whole call graph is translated. Then a C compiler is invoked to translate the C code into a shared library, and the original call to the user Julia function is replaced by a call to this shared library.

If the user specifies the main function to be translated into C, he/she should get a standalone executable as you imagined.


On Wednesday, November 12, 2014 5:21:42 PM UTC-8, Keith wrote:
Maybe a way to get to a standalone executable?

On Wednesday, November 12, 2014 5:06:13 PM UTC-5, Jey Kottalam wrote:
Interesting. What are the sorts of usecases envisioned for this?


Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
In reply to this post by Tom Short

There are two main parts of the code:

(1) traverse the AST tree of a function. This is embedded into codegen.cpp and a few other files to minimize our coding efforts. But a (much) better way would be to make it a self-contained module.

(2) map Julia types and AST nodes to C types and statements. This is in j2c.cpp.

Since the code works at Julia code generation phase, it seems more appropriate to merge it into Julia, instead of as an external package.  

We hope that this initial version will give the community a starting point for Julia to C translation, and people can re-shape it to be better. 


On Wednesday, November 12, 2014 6:46:58 PM UTC-8, Tom Short wrote:

Wow. That's a lot of work. It's very entwined in the core Julia code. Is there hope that it could be separated into a package?

Or, are you hoping to get it merged into Julia?

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
In reply to this post by Tony Kelman-2
All macros and included Julia code have been processed before J2C happens. The generated C is  standalone: a call graph rooted at a user-specified function is translated, including any Julia runtime function called. So it is a kind of whole-program translation. The C does not call back to libjulia during execution. It may call C runtime instead (for printf, for example).

On Wednesday, November 12, 2014 7:47:49 PM UTC-8, Tony Kelman wrote:
If I were the Mathworks, this would scare the crap out of me. Simulink Real-time Workshop and C code generation for embedded real-time control applications is a substantial industrial use case for Matlab/Simulink that hasn't really been replicated by any high-level open source language as of yet.

I'll have to look at this in more detail and try it out. Does this expand macros? Include code used from Julia packages? Does the generated C require any pieces of the libjulia runtime library?


On Wednesday, November 12, 2014 6:46:58 PM UTC-8, Tom Short wrote:

Wow. That's a lot of work. It's very entwined in the core Julia code. Is there hope that it could be separated into a package?

Or, are you hoping to get it merged into Julia?

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Jameson Nash
Wow. That is quite a large change set in that commit.

It's perhaps worth pointing out that Julia has been emitting native executables for a little over a year now. This translator is different in that it adds an intermediate representation in C, rather than emitting the binaries directly as we do now.

Julia can be compiled for any computer which can be targeted by the LLVM/Clang compiler. It is theoretically already possible to compile Julia directly for an embedded application, although it is poorly exposed by the current frontend (libjulia would be a very small library in such a situation, due to the removal of the libLLVM libraries, so as to be nearly insignificant. Regardless, every programming language includes a similar such set of standard routines: libc, libgfortran, libpython, etc., so it isn't particularly meaningful or beneficial to remove it).

Since this is a source transform (codegen.ccp is also just a source transform in this respect), it could easily be implemented in a pure julia package (inference.jl is another quite similar source transform) just by recursively calling expand() on every function call and emitting an equivalent C file. I think you could even leverage libclang to pretty-print your C AST in that case. Note that a significant portion of Julia code cannot be emitted as a direct transform to C. You will need the jl_apply_generic function from the libjulia library to handle the general case of calling an unknown function (such as when using first-class functions or when calling map or when type inference is incomplete).

On Wed Nov 12 2014 at 11:03:16 PM Hongbo Rong <[hidden email]> wrote:
All macros and included Julia code have been processed before J2C happens. The generated C is  standalone: a call graph rooted at a user-specified function is translated, including any Julia runtime function called. So it is a kind of whole-program translation. The C does not call back to libjulia during execution. It may call C runtime instead (for printf, for example).

On Wednesday, November 12, 2014 7:47:49 PM UTC-8, Tony Kelman wrote:
If I were the Mathworks, this would scare the crap out of me. Simulink Real-time Workshop and C code generation for embedded real-time control applications is a substantial industrial use case for Matlab/Simulink that hasn't really been replicated by any high-level open source language as of yet.

I'll have to look at this in more detail and try it out. Does this expand macros? Include code used from Julia packages? Does the generated C require any pieces of the libjulia runtime library?


On Wednesday, November 12, 2014 6:46:58 PM UTC-8, Tom Short wrote:

Wow. That's a lot of work. It's very entwined in the core Julia code. Is there hope that it could be separated into a package?

Or, are you hoping to get it merged into Julia?

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Viral Shah
In reply to this post by Hongbo Rong
Hi Hongbo

First of all - a big thanks to Intel for this contribution. Large organisations are not easy to convince on open source, and this is huge.

A question for everyone is if we should open have j2c as a PR? How should we integrate it? There is a fair amount of work to do before it is complete.

There is a case for embedded deployments where a certified C compiler is required to compile and this gives a way to deploy Julia in such cases if we can make the support first class.
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tom Short
I tried compiling j2c on Arch Linux. In the j2c branch, I get the
following error:

codegen.cpp:46:32: fatal error: llvm/ADT/OwningPtr.h: No such file or directory
 #include "llvm/ADT/OwningPtr.h"
                                ^

The master branch compiles fine. In both cases, I was using this
invocation (taken from the AUR script that compiles Julia):

make USE_SYSTEM_LLVM=1     USE_SYSTEM_LIBUNWIND=1
USE_SYSTEM_PCRE=1     USE_SYSTEM_LIBM=1     USE_SYSTEM_OPENLIBM=0
USE_SYSTEM_OPENSPECFUN=0     USE_SYSTEM_DSFMT=0     USE_SYSTEM_BLAS=1
   USE_SYSTEM_LAPACK=1     USE_SYSTEM_FFTW=1     USE_SYSTEM_GMP=1
USE_SYSTEM_MPFR=1     USE_SYSTEM_ARPACK=1     USE_SYSTEM_SUITESPARSE=0
    USE_SYSTEM_RMATH=0     USE_SYSTEM_LIBUV=0
USE_SYSTEM_UTF8PROC=0     USE_SYSTEM_MOJIBAKE=0     USE_INTEL_MKL=0
 USE_BLAS64=0     USE_LLVM_SHLIB=0


On Thu, Nov 13, 2014 at 12:25 AM, Viral Shah <[hidden email]> wrote:
> Hi Hongbo
>
> First of all - a big thanks to Intel for this contribution. Large organisations are not easy to convince on open source, and this is huge.
>
> A question for everyone is if we should open have j2c as a PR? How should we integrate it? There is a fair amount of work to do before it is complete.
>
> There is a case for embedded deployments where a certified C compiler is required to compile and this gives a way to deploy Julia in such cases if we can make the support first class.
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
There is a condition before that include: 

#if defined(LLVM_VERSION_MAJOR) && LLVM_VERSION_MAJOR == 3 && LLVM_VERSION_MINOR >= 5

So it looks like the issue is related with the LLVM version used.

Note: the J2C branch is behind the main Julia branch in commits after April 10th, 2014. We have not updated Julia since then. So the compilation of the main Julia and J2C may have differences.

On Friday, November 14, 2014 10:37:41 AM UTC-8, Tom Short wrote:
I tried compiling j2c on Arch Linux. In the j2c branch, I get the
following error:

codegen.cpp:46:32: fatal error: llvm/ADT/OwningPtr.h: No such file or directory
 #include "llvm/ADT/OwningPtr.h"
                                ^

The master branch compiles fine. In both cases, I was using this
invocation (taken from the AUR script that compiles Julia):

make USE_SYSTEM_LLVM=1     USE_SYSTEM_LIBUNWIND=1
USE_SYSTEM_PCRE=1     USE_SYSTEM_LIBM=1     USE_SYSTEM_OPENLIBM=0
USE_SYSTEM_OPENSPECFUN=0     USE_SYSTEM_DSFMT=0     USE_SYSTEM_BLAS=1
   USE_SYSTEM_LAPACK=1     USE_SYSTEM_FFTW=1     USE_SYSTEM_GMP=1
USE_SYSTEM_MPFR=1     USE_SYSTEM_ARPACK=1     USE_SYSTEM_SUITESPARSE=0
    USE_SYSTEM_RMATH=0     USE_SYSTEM_LIBUV=0
USE_SYSTEM_UTF8PROC=0     USE_SYSTEM_MOJIBAKE=0     USE_INTEL_MKL=0
 USE_BLAS64=0     USE_LLVM_SHLIB=0


On Thu, Nov 13, 2014 at 12:25 AM, Viral Shah <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="8V2t_Lzz83YJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">vi...@...> wrote:
> Hi Hongbo
>
> First of all - a big thanks to Intel for this contribution. Large organisations are not easy to convince on open source, and this is huge.
>
> A question for everyone is if we should open have j2c as a PR? How should we integrate it? There is a fair amount of work to do before it is complete.
>
> There is a case for embedded deployments where a certified C compiler is required to compile and this gives a way to deploy Julia in such cases if we can make the support first class.

Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tom Short
I got julia to compile using USE_SYSTEM_LLVM=0. Now, my problem is
that when I run `../julia j2c.jl`, it creates an empty out.cpp file in
../j2c/, and there's an error message saying it can't find function
j2c_sumOfThree.
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
I cannot repro this error. Here is what I tried, with a fresh checkout, build, and test:

git clone -b j2c https://github.com/IntelLabs/julia.git
cd julia/
make debug

export JULIA_ROOT=/home/hrong1/j2c-release/julia
cd test
../julia j2c.jl

On Friday, November 14, 2014 12:17:03 PM UTC-8, Tom Short wrote:
I got julia to compile using USE_SYSTEM_LLVM=0. Now, my problem is
that when I run `../julia j2c.jl`, it creates an empty out.cpp file in
../j2c/, and there's an error message saying it can't find function
j2c_sumOfThree.
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tom Short
I tried it again, this time in a Mint Linux virtual machine. I got the
same issue: a zero-length out.cpp file.


On Fri, Nov 14, 2014 at 6:51 PM, Hongbo Rong <[hidden email]> wrote:

> I cannot repro this error. Here is what I tried, with a fresh checkout,
> build, and test:
>
> git clone -b j2c https://github.com/IntelLabs/julia.git
> cd julia/
> make debug
>
> export JULIA_ROOT=/home/hrong1/j2c-release/julia
> cd test
> ../julia j2c.jl
>
> On Friday, November 14, 2014 12:17:03 PM UTC-8, Tom Short wrote:
>>
>> I got julia to compile using USE_SYSTEM_LLVM=0. Now, my problem is
>> that when I run `../julia j2c.jl`, it creates an empty out.cpp file in
>> ../j2c/, and there's an error message saying it can't find function
>> j2c_sumOfThree.
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Hongbo Rong
Can you show the complete error messages, if any?

On Sunday, November 16, 2014 9:14:59 AM UTC-8, Tom Short wrote:
I tried it again, this time in a Mint Linux virtual machine. I got the
same issue: a zero-length out.cpp file.


On Fri, Nov 14, 2014 at 6:51 PM, Hongbo Rong <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="Dmi21F-J-V0J" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">rongh...@...> wrote:

> I cannot repro this error. Here is what I tried, with a fresh checkout,
> build, and test:
>
> git clone -b j2c <a href="https://github.com/IntelLabs/julia.git" target="_blank" onmousedown="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FIntelLabs%2Fjulia.git\46sa\75D\46sntz\0751\46usg\75AFQjCNEREKkA5n6xNBo1QcS__60nxjPiog';return true;" onclick="this.href='https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FIntelLabs%2Fjulia.git\46sa\75D\46sntz\0751\46usg\75AFQjCNEREKkA5n6xNBo1QcS__60nxjPiog';return true;">https://github.com/IntelLabs/julia.git
> cd julia/
> make debug
>
> export JULIA_ROOT=/home/hrong1/j2c-release/julia
> cd test
> ../julia j2c.jl
>
> On Friday, November 14, 2014 12:17:03 PM UTC-8, Tom Short wrote:
>>
>> I got julia to compile using USE_SYSTEM_LLVM=0. Now, my problem is
>> that when I run `../julia j2c.jl`, it creates an empty out.cpp file in
>> ../j2c/, and there's an error message saying it can't find function
>> j2c_sumOfThree.
Reply | Threaded
Open this post in threaded view
|

Re: Julia2C initial release

Tom Short
****Matrix size: 1000*1000
ERROR: ccall: could not find function sumOfThree_ in library
/home/tshort/julia2c/j2c/libout.so.1.0
 in j2c_sumOfThree at /home/tshort/julia2c/test/j2c.jl:65
 in anonymous at no file:73
 in include_from_node1 at loading.jl:128
while loading /home/tshort/julia2c/test/j2c.jl, in expression starting
on line 70

I tried pasting the code from j2c.jl. Here's the only thing that I was
that might indicate something. I'm not sure what the return value
means.

julia> offload(sumOfThree, (Int,))
2



On Sun, Nov 16, 2014 at 2:11 PM, Hongbo Rong <[hidden email]> wrote:

> Can you show the complete error messages, if any?
>
> On Sunday, November 16, 2014 9:14:59 AM UTC-8, Tom Short wrote:
>>
>> I tried it again, this time in a Mint Linux virtual machine. I got the
>> same issue: a zero-length out.cpp file.
>>
>>
>> On Fri, Nov 14, 2014 at 6:51 PM, Hongbo Rong <[hidden email]> wrote:
>> > I cannot repro this error. Here is what I tried, with a fresh checkout,
>> > build, and test:
>> >
>> > git clone -b j2c https://github.com/IntelLabs/julia.git
>> > cd julia/
>> > make debug
>> >
>> > export JULIA_ROOT=/home/hrong1/j2c-release/julia
>> > cd test
>> > ../julia j2c.jl
>> >
>> > On Friday, November 14, 2014 12:17:03 PM UTC-8, Tom Short wrote:
>> >>
>> >> I got julia to compile using USE_SYSTEM_LLVM=0. Now, my problem is
>> >> that when I run `../julia j2c.jl`, it creates an empty out.cpp file in
>> >> ../j2c/, and there's an error message saying it can't find function
>> >> j2c_sumOfThree.
123