Pros/Cons of various autodiff packages

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Pros/Cons of various autodiff packages

Benjamin Deonovic
What are pros/cons of the various auto differentiation packages available in Julia? Is there a benefit to using ReverseDiffSource vs ForwardDiff? Thank you

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Pros/Cons of various autodiff packages

Andrei Zh
Forward mode AD works faster for functions R^m -> R^n where m << n, e.g. when you have a function of 1 input and return tuple or vector of multiple outputs.
Reverse mode AD works faster for functions R^m -> R^n where m >> n, e.g. when you have a function of a vector of inputs and return a single output (say, typical loss function in machine learning). 

As for packages, ReverseDiffSource generates expressions for derivatives, but currently supports only simple `for` loops and the author works on adding support for `if`s. AFAIK, ForwardDiff doesn't have any restrictions on a program structure and should work with any type of loops, conditions, recursion, etc., but can't be used for code generation (e.g. for GPU). 

I'm also currently working on Espresso.jl - package for expression manipulation and hybrid differentiation (AD pipeline, symbolic output) where I drop support for any kind of conditions and loops in favor of higher flexibility (e.g. differentiation for nested function calls which is an hard to achieve in ReverseDiffSource). The package is in early testing stage yet, but differentiation over expressions with numbers and simple vectors seem to work fine, and I'm working hard to add derivatives of higher-order tensors. 


On Sunday, August 21, 2016 at 6:36:48 AM UTC+3, Benjamin Deonovic wrote:
What are pros/cons of the various auto differentiation packages available in Julia? Is there a benefit to using ReverseDiffSource vs ForwardDiff? Thank you

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Pros/Cons of various autodiff packages

Alex Williams
Also keep an eye on: https://github.com/jrevels/ReverseDiffPrototype.jl

(As the name suggests, they are still prototyping it.)


On Sun, Aug 21, 2016 at 11:59 PM, Andrei Zh <[hidden email]> wrote:
Forward mode AD works faster for functions R^m -> R^n where m << n, e.g. when you have a function of 1 input and return tuple or vector of multiple outputs.
Reverse mode AD works faster for functions R^m -> R^n where m >> n, e.g. when you have a function of a vector of inputs and return a single output (say, typical loss function in machine learning). 

As for packages, ReverseDiffSource generates expressions for derivatives, but currently supports only simple `for` loops and the author works on adding support for `if`s. AFAIK, ForwardDiff doesn't have any restrictions on a program structure and should work with any type of loops, conditions, recursion, etc., but can't be used for code generation (e.g. for GPU). 

I'm also currently working on Espresso.jl - package for expression manipulation and hybrid differentiation (AD pipeline, symbolic output) where I drop support for any kind of conditions and loops in favor of higher flexibility (e.g. differentiation for nested function calls which is an hard to achieve in ReverseDiffSource). The package is in early testing stage yet, but differentiation over expressions with numbers and simple vectors seem to work fine, and I'm working hard to add derivatives of higher-order tensors. 


On Sunday, August 21, 2016 at 6:36:48 AM UTC+3, Benjamin Deonovic wrote:
What are pros/cons of the various auto differentiation packages available in Julia? Is there a benefit to using ReverseDiffSource vs ForwardDiff? Thank you

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "julia-stats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Loading...