Quantcast

Updated performance tips?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Updated performance tips?

Jonathan Goldfarb
I've really been enjoying writing Julia code as a user, and following the language as it develops, but I have noticed that over time, previously fast code sometimes gets slower, and (impressively) previously slow code will sometimes get faster, with updates to the Julia codebase. No complaint here in general; I really appreciate the work all of the Core and package developers do, and variations in performance of different codes it to be expected.
My question is this: has anyone in the Julia community thought about updated performance tips for writing high performance code? Obviously, using the profiler, along with many of the tips at https://github.com/JuliaLang/julia/commits/master/doc/manual/performance-tips.rst still apply, but I am wondering more about general/structural ideas to keep in mind in Julia v0.4, as well as guidance on how best to take advantage of recent changes on master. I know that document hasn't been stagnant in any sense, but relatively "big in any case, I'd be happy to help make some updates in a PR if there's anything we come up with.
Best Regards,
Max
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Milan Bouchet-Valat
Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
> I've really been enjoying writing Julia code as a user, and following
> the language as it develops, but I have noticed that over time,
> previously fast code sometimes gets slower, and (impressively)
> previously slow code will sometimes get faster, with updates to the
> Julia codebase.
Code is not supposed to get slower with newer releases. If this
happens, please report the problem here or on GitHub (if possible with
a reproducible example). This will be very helpful to help avoiding
regressions.

> No complaint here in general; I really appreciate the work all of the
> Core and package developers do, and variations in performance of
> different codes it to be expected.
> My question is this: has anyone in the Julia community thought about
> updated performance tips for writing high performance code?
> Obviously, using the profiler, along with many of the tips
> at https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
> rmance-tips.rst still apply, but I am wondering more about
> general/structural ideas to keep in mind in Julia v0.4, as well as
> guidance on how best to take advantage of recent changes on master. I
> know that document hasn't been stagnant in any sense, but relatively
> "big in any case, I'd be happy to help make some updates in a PR if
> there's anything we come up with.
I've just skimmed through this page, and I don't think any of the
advice given there is outdated. What's new in master is that anonymous
functions (and therefore map) are now fast, but that wasn't previously
mentioned in the tips as a performance issue anyway.

The only small sentence which should likely be removed is "for example,
currently it’s not possible to infer the return type of an anonymous
function". Type inference seems to work fine now on master with
anonymous functions. I'll leave others confirm this.

Anyway, do you have any specific points in mind?


Regards
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Stefan Karpinski
Yes, ideally code should not get slower with new releases – unfortunately, keeping track of performance regressions can be a bit of a game of whack-a-mole. Having examples of code whose speed has regressed is very helpful. Thanks to Jarrett Revels excellent work, we now have some great performance regression tracking infrastructure, but of course we always need more things to test!

On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]> wrote:
Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
> I've really been enjoying writing Julia code as a user, and following
> the language as it develops, but I have noticed that over time,
> previously fast code sometimes gets slower, and (impressively)
> previously slow code will sometimes get faster, with updates to the
> Julia codebase.
Code is not supposed to get slower with newer releases. If this
happens, please report the problem here or on GitHub (if possible with
a reproducible example). This will be very helpful to help avoiding
regressions.

> No complaint here in general; I really appreciate the work all of the
> Core and package developers do, and variations in performance of
> different codes it to be expected.
> My question is this: has anyone in the Julia community thought about
> updated performance tips for writing high performance code?
> Obviously, using the profiler, along with many of the tips
> at https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
> rmance-tips.rst still apply, but I am wondering more about
> general/structural ideas to keep in mind in Julia v0.4, as well as
> guidance on how best to take advantage of recent changes on master. I
> know that document hasn't been stagnant in any sense, but relatively
> "big in any case, I'd be happy to help make some updates in a PR if
> there's anything we come up with.
I've just skimmed through this page, and I don't think any of the
advice given there is outdated. What's new in master is that anonymous
functions (and therefore map) are now fast, but that wasn't previously
mentioned in the tips as a performance issue anyway.

The only small sentence which should likely be removed is "for example,
currently it’s not possible to infer the return type of an anonymous
function". Type inference seems to work fine now on master with
anonymous functions. I'll leave others confirm this.

Anyway, do you have any specific points in mind?


Regards

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Jonathan Goldfarb
Yes, understood about difficulty keeping track of regressions. I was originally going to send a message relating up to 2x longer test time on the same code on Travis, but it appears as though something has changed in the nightly build available to CI that now gives significantly faster builds, even though the previous poor performance had been dependable... Evidently that build is not as up-to-date as I thought. Our code is currently not open source, but should be soon after which I can share an example.

Thanks for your comments, and thanks again for your work on Julia.

-Max


On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski wrote:
Yes, ideally code should not get slower with new releases – unfortunately, keeping track of performance regressions can be a bit of a game of whack-a-mole. Having examples of code whose speed has regressed is very helpful. Thanks to Jarrett Revels excellent work, we now have some great performance regression tracking infrastructure, but of course we always need more things to test!

On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="uf472ZsJAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">nali...@...> wrote:
Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
> I've really been enjoying writing Julia code as a user, and following
> the language as it develops, but I have noticed that over time,
> previously fast code sometimes gets slower, and (impressively)
> previously slow code will sometimes get faster, with updates to the
> Julia codebase.
Code is not supposed to get slower with newer releases. If this
happens, please report the problem here or on GitHub (if possible with
a reproducible example). This will be very helpful to help avoiding
regressions.

> No complaint here in general; I really appreciate the work all of the
> Core and package developers do, and variations in performance of
> different codes it to be expected.
> My question is this: has anyone in the Julia community thought about
> updated performance tips for writing high performance code?
> Obviously, using the profiler, along with many of the tips
> at <a href="https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FJuliaLang%2Fjulia%2Fcommits%2Fmaster%2Fdoc%2Fmanual%2Fperfo\46sa\75D\46sntz\0751\46usg\75AFQjCNHFDD9YIIB8FjvK5O3B0OJYTk78Kw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\75https%3A%2F%2Fgithub.com%2FJuliaLang%2Fjulia%2Fcommits%2Fmaster%2Fdoc%2Fmanual%2Fperfo\46sa\75D\46sntz\0751\46usg\75AFQjCNHFDD9YIIB8FjvK5O3B0OJYTk78Kw&#39;;return true;">https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
> rmance-tips.rst still apply, but I am wondering more about
> general/structural ideas to keep in mind in Julia v0.4, as well as
> guidance on how best to take advantage of recent changes on master. I
> know that document hasn't been stagnant in any sense, but relatively
> "big in any case, I'd be happy to help make some updates in a PR if
> there's anything we come up with.
I've just skimmed through this page, and I don't think any of the
advice given there is outdated. What's new in master is that anonymous
functions (and therefore map) are now fast, but that wasn't previously
mentioned in the tips as a performance issue anyway.

The only small sentence which should likely be removed is "for example,
currently it’s not possible to infer the return type of an anonymous
function". Type inference seems to work fine now on master with
anonymous functions. I'll leave others confirm this.

Anyway, do you have any specific points in mind?


Regards

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Stefan Karpinski
I'm glad that particular slow case got faster! If you want to submit some reduced version of it as a performance test, we could still include it in our perf suite. And of course, if you find that anything else has ever slowed down, please don't hesitate to file an issue.

On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <[hidden email]> wrote:
Yes, understood about difficulty keeping track of regressions. I was originally going to send a message relating up to 2x longer test time on the same code on Travis, but it appears as though something has changed in the nightly build available to CI that now gives significantly faster builds, even though the previous poor performance had been dependable... Evidently that build is not as up-to-date as I thought. Our code is currently not open source, but should be soon after which I can share an example.

Thanks for your comments, and thanks again for your work on Julia.

-Max


On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski wrote:
Yes, ideally code should not get slower with new releases – unfortunately, keeping track of performance regressions can be a bit of a game of whack-a-mole. Having examples of code whose speed has regressed is very helpful. Thanks to Jarrett Revels excellent work, we now have some great performance regression tracking infrastructure, but of course we always need more things to test!

On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]> wrote:
Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
> I've really been enjoying writing Julia code as a user, and following
> the language as it develops, but I have noticed that over time,
> previously fast code sometimes gets slower, and (impressively)
> previously slow code will sometimes get faster, with updates to the
> Julia codebase.
Code is not supposed to get slower with newer releases. If this
happens, please report the problem here or on GitHub (if possible with
a reproducible example). This will be very helpful to help avoiding
regressions.

> No complaint here in general; I really appreciate the work all of the
> Core and package developers do, and variations in performance of
> different codes it to be expected.
> My question is this: has anyone in the Julia community thought about
> updated performance tips for writing high performance code?
> Obviously, using the profiler, along with many of the tips
> at https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
> rmance-tips.rst still apply, but I am wondering more about
> general/structural ideas to keep in mind in Julia v0.4, as well as
> guidance on how best to take advantage of recent changes on master. I
> know that document hasn't been stagnant in any sense, but relatively
> "big in any case, I'd be happy to help make some updates in a PR if
> there's anything we come up with.
I've just skimmed through this page, and I don't think any of the
advice given there is outdated. What's new in master is that anonymous
functions (and therefore map) are now fast, but that wasn't previously
mentioned in the tips as a performance issue anyway.

The only small sentence which should likely be removed is "for example,
currently it’s not possible to infer the return type of an anonymous
function". Type inference seems to work fine now on master with
anonymous functions. I'll leave others confirm this.

Anyway, do you have any specific points in mind?


Regards


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Harish Kumar
I am using Version 0.3.12 calling from python (pyjulia). I do LME fit with 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+ data transfer time). Any tips?
      using MixedModels
      modelREML = lmm({formula}, dataset)
      reml!(modelREML,true)
      lmeModel = fit(modelREML)
      fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,estimate = coeftable(lmeModel).mat[:,1],
                     stdError = coeftable(lmeModel).mat[:,2],zVal = coeftable(lmeModel).mat[:,3])

On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski wrote:
I'm glad that particular slow case got faster! If you want to submit some reduced version of it as a performance test, we could still include it in our perf suite. And of course, if you find that anything else has ever slowed down, please don't hesitate to file an issue.

On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="0g20kR9VAQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">jgol...@...> wrote:
Yes, understood about difficulty keeping track of regressions. I was originally going to send a message relating up to 2x longer test time on the same code on Travis, but it appears as though something has changed in the nightly build available to CI that now gives significantly faster builds, even though the previous poor performance had been dependable... Evidently that build is not as up-to-date as I thought. Our code is currently not open source, but should be soon after which I can share an example.

Thanks for your comments, and thanks again for your work on Julia.

-Max


On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski wrote:
Yes, ideally code should not get slower with new releases – unfortunately, keeping track of performance regressions can be a bit of a game of whack-a-mole. Having examples of code whose speed has regressed is very helpful. Thanks to Jarrett Revels excellent work, we now have some great performance regression tracking infrastructure, but of course we always need more things to test!

On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]> wrote:
Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
> I've really been enjoying writing Julia code as a user, and following
> the language as it develops, but I have noticed that over time,
> previously fast code sometimes gets slower, and (impressively)
> previously slow code will sometimes get faster, with updates to the
> Julia codebase.
Code is not supposed to get slower with newer releases. If this
happens, please report the problem here or on GitHub (if possible with
a reproducible example). This will be very helpful to help avoiding
regressions.

> No complaint here in general; I really appreciate the work all of the
> Core and package developers do, and variations in performance of
> different codes it to be expected.
> My question is this: has anyone in the Julia community thought about
> updated performance tips for writing high performance code?
> Obviously, using the profiler, along with many of the tips
> at <a href="https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FJuliaLang%2Fjulia%2Fcommits%2Fmaster%2Fdoc%2Fmanual%2Fperfo\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHFDD9YIIB8FjvK5O3B0OJYTk78Kw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FJuliaLang%2Fjulia%2Fcommits%2Fmaster%2Fdoc%2Fmanual%2Fperfo\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHFDD9YIIB8FjvK5O3B0OJYTk78Kw&#39;;return true;">https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
> rmance-tips.rst still apply, but I am wondering more about
> general/structural ideas to keep in mind in Julia v0.4, as well as
> guidance on how best to take advantage of recent changes on master. I
> know that document hasn't been stagnant in any sense, but relatively
> "big in any case, I'd be happy to help make some updates in a PR if
> there's anything we come up with.
I've just skimmed through this page, and I don't think any of the
advice given there is outdated. What's new in master is that anonymous
functions (and therefore map) are now fast, but that wasn't previously
mentioned in the tips as a performance issue anyway.

The only small sentence which should likely be removed is "for example,
currently it’s not possible to infer the return type of an anonymous
function". Type inference seems to work fine now on master with
anonymous functions. I'll leave others confirm this.

Anyway, do you have any specific points in mind?


Regards


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Mauro
Julia 0.3.12, that's a stone-age version of Julia.  You should move to 0.5!

On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]> wrote:

> I am using Version 0.3.12 calling from python (pyjulia). I do LME fit with
> 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+ data
> transfer time). Any tips?
>       using MixedModels
>       modelREML = lmm({formula}, dataset)
>       reml!(modelREML,true)
>       lmeModel = fit(modelREML)
>       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,estimate
> = coeftable(lmeModel).mat[:,1],
>                      stdError = coeftable(lmeModel).mat[:,2],zVal =
> coeftable(lmeModel).mat[:,3])
>
> On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski wrote:
>>
>> I'm glad that particular slow case got faster! If you want to submit some
>> reduced version of it as a performance test, we could still include it in
>> our perf suite. And of course, if you find that anything else has ever
>> slowed down, please don't hesitate to file an issue.
>>
>> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <[hidden email]
>> <javascript:>> wrote:
>>
>>> Yes, understood about difficulty keeping track of regressions. I was
>>> originally going to send a message relating up to 2x longer test time on
>>> the same code on Travis, but it appears as though something has changed in
>>> the nightly build available to CI that now gives significantly faster
>>> builds, even though the previous poor performance had been dependable...
>>> Evidently that build is not as up-to-date as I thought. Our code is
>>> currently not open source, but should be soon after which I can share an
>>> example.
>>>
>>> Thanks for your comments, and thanks again for your work on Julia.
>>>
>>> -Max
>>>
>>>
>>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski wrote:
>>>>
>>>> Yes, ideally code should not get slower with new releases –
>>>> unfortunately, keeping track of performance regressions can be a bit of a
>>>> game of whack-a-mole. Having examples of code whose speed has regressed is
>>>> very helpful. Thanks to Jarrett Revels excellent work, we now have some
>>>> great performance regression tracking infrastructure, but of course we
>>>> always need more things to test!
>>>>
>>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]>
>>>> wrote:
>>>>
>>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
>>>>> > I've really been enjoying writing Julia code as a user, and following
>>>>> > the language as it develops, but I have noticed that over time,
>>>>> > previously fast code sometimes gets slower, and (impressively)
>>>>> > previously slow code will sometimes get faster, with updates to the
>>>>> > Julia codebase.
>>>>> Code is not supposed to get slower with newer releases. If this
>>>>> happens, please report the problem here or on GitHub (if possible with
>>>>> a reproducible example). This will be very helpful to help avoiding
>>>>> regressions.
>>>>>
>>>>> > No complaint here in general; I really appreciate the work all of the
>>>>> > Core and package developers do, and variations in performance of
>>>>> > different codes it to be expected.
>>>>> > My question is this: has anyone in the Julia community thought about
>>>>> > updated performance tips for writing high performance code?
>>>>> > Obviously, using the profiler, along with many of the tips
>>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
>>>>> > rmance-tips.rst still apply, but I am wondering more about
>>>>> > general/structural ideas to keep in mind in Julia v0.4, as well as
>>>>> > guidance on how best to take advantage of recent changes on master. I
>>>>> > know that document hasn't been stagnant in any sense, but relatively
>>>>> > "big in any case, I'd be happy to help make some updates in a PR if
>>>>> > there's anything we come up with.
>>>>> I've just skimmed through this page, and I don't think any of the
>>>>> advice given there is outdated. What's new in master is that anonymous
>>>>> functions (and therefore map) are now fast, but that wasn't previously
>>>>> mentioned in the tips as a performance issue anyway.
>>>>>
>>>>> The only small sentence which should likely be removed is "for example,
>>>>> currently it’s not possible to infer the return type of an anonymous
>>>>> function". Type inference seems to work fine now on master with
>>>>> anonymous functions. I'll leave others confirm this.
>>>>>
>>>>> Anyway, do you have any specific points in mind?
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>
>>>>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Harish Kumar

Will it support Python 3.4 ? I am calling this from pyjulia interface


On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
Julia 0.3.12, that's a stone-age version of Julia.  You should move to 0.5!

On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]> wrote:
> I am using Version 0.3.12 calling from python (pyjulia). I do LME fit with
> 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+ data
> transfer time). Any tips?
>       using MixedModels
>       modelREML = lmm({formula}, dataset)
>       reml!(modelREML,true)
>       lmeModel = fit(modelREML)
>       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,estimate
> = coeftable(lmeModel).mat[:,1],
>                      stdError = coeftable(lmeModel).mat[:,2],zVal =
> coeftable(lmeModel).mat[:,3])
>
> On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski wrote:
>>
>> I'm glad that particular slow case got faster! If you want to submit some
>> reduced version of it as a performance test, we could still include it in
>> our perf suite. And of course, if you find that anything else has ever
>> slowed down, please don't hesitate to file an issue.
>>
>> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <[hidden email]
>> <javascript:>> wrote:
>>
>>> Yes, understood about difficulty keeping track of regressions. I was
>>> originally going to send a message relating up to 2x longer test time on
>>> the same code on Travis, but it appears as though something has changed in
>>> the nightly build available to CI that now gives significantly faster
>>> builds, even though the previous poor performance had been dependable...
>>> Evidently that build is not as up-to-date as I thought. Our code is
>>> currently not open source, but should be soon after which I can share an
>>> example.
>>>
>>> Thanks for your comments, and thanks again for your work on Julia.
>>>
>>> -Max
>>>
>>>
>>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski wrote:
>>>>
>>>> Yes, ideally code should not get slower with new releases –
>>>> unfortunately, keeping track of performance regressions can be a bit of a
>>>> game of whack-a-mole. Having examples of code whose speed has regressed is
>>>> very helpful. Thanks to Jarrett Revels excellent work, we now have some
>>>> great performance regression tracking infrastructure, but of course we
>>>> always need more things to test!
>>>>
>>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]>
>>>> wrote:
>>>>
>>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
>>>>> > I've really been enjoying writing Julia code as a user, and following
>>>>> > the language as it develops, but I have noticed that over time,
>>>>> > previously fast code sometimes gets slower, and (impressively)
>>>>> > previously slow code will sometimes get faster, with updates to the
>>>>> > Julia codebase.
>>>>> Code is not supposed to get slower with newer releases. If this
>>>>> happens, please report the problem here or on GitHub (if possible with
>>>>> a reproducible example). This will be very helpful to help avoiding
>>>>> regressions.
>>>>>
>>>>> > No complaint here in general; I really appreciate the work all of the
>>>>> > Core and package developers do, and variations in performance of
>>>>> > different codes it to be expected.
>>>>> > My question is this: has anyone in the Julia community thought about
>>>>> > updated performance tips for writing high performance code?
>>>>> > Obviously, using the profiler, along with many of the tips
>>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/manual/perfo
>>>>> > rmance-tips.rst still apply, but I am wondering more about
>>>>> > general/structural ideas to keep in mind in Julia v0.4, as well as
>>>>> > guidance on how best to take advantage of recent changes on master. I
>>>>> > know that document hasn't been stagnant in any sense, but relatively
>>>>> > "big in any case, I'd be happy to help make some updates in a PR if
>>>>> > there's anything we come up with.
>>>>> I've just skimmed through this page, and I don't think any of the
>>>>> advice given there is outdated. What's new in master is that anonymous
>>>>> functions (and therefore map) are now fast, but that wasn't previously
>>>>> mentioned in the tips as a performance issue anyway.
>>>>>
>>>>> The only small sentence which should likely be removed is "for example,
>>>>> currently it’s not possible to infer the return type of an anonymous
>>>>> function". Type inference seems to work fine now on master with
>>>>> anonymous functions. I'll leave others confirm this.
>>>>>
>>>>> Anyway, do you have any specific points in mind?
>>>>>
>>>>>
>>>>> Regards
>>>>>
>>>>
>>>>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Mauro
On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]> wrote:
> Will it support Python 3.4 ? I am calling this from pyjulia interface

https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
but it doesn't say that 3.4 is not supported.  So you should try.

> On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>
>> Julia 0.3.12, that's a stone-age version of Julia.  You should move to 0.5!
>>
>> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>> wrote:
>> > I am using Version 0.3.12 calling from python (pyjulia). I do LME fit
>> with
>> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+
>> data
>> > transfer time). Any tips?
>> >       using MixedModels
>> >       modelREML = lmm({formula}, dataset)
>> >       reml!(modelREML,true)
>> >       lmeModel = fit(modelREML)
>> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>> estimate
>> > = coeftable(lmeModel).mat[:,1],
>> >                      stdError = coeftable(lmeModel).mat[:,2],zVal =
>> > coeftable(lmeModel).mat[:,3])
>> >
>> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>> wrote:
>> >>
>> >> I'm glad that particular slow case got faster! If you want to submit
>> some
>> >> reduced version of it as a performance test, we could still include it
>> in
>> >> our perf suite. And of course, if you find that anything else has ever
>> >> slowed down, please don't hesitate to file an issue.
>> >>
>> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <[hidden email]
>> >> <javascript:>> wrote:
>> >>
>> >>> Yes, understood about difficulty keeping track of regressions. I was
>> >>> originally going to send a message relating up to 2x longer test time
>> on
>> >>> the same code on Travis, but it appears as though something has
>> changed in
>> >>> the nightly build available to CI that now gives significantly faster
>> >>> builds, even though the previous poor performance had been
>> dependable...
>> >>> Evidently that build is not as up-to-date as I thought. Our code is
>> >>> currently not open source, but should be soon after which I can share
>> an
>> >>> example.
>> >>>
>> >>> Thanks for your comments, and thanks again for your work on Julia.
>> >>>
>> >>> -Max
>> >>>
>> >>>
>> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski
>> wrote:
>> >>>>
>> >>>> Yes, ideally code should not get slower with new releases –
>> >>>> unfortunately, keeping track of performance regressions can be a bit
>> of a
>> >>>> game of whack-a-mole. Having examples of code whose speed has
>> regressed is
>> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now have
>> some
>> >>>> great performance regression tracking infrastructure, but of course we
>> >>>> always need more things to test!
>> >>>>
>> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
>> >>>>> > I've really been enjoying writing Julia code as a user, and
>> following
>> >>>>> > the language as it develops, but I have noticed that over time,
>> >>>>> > previously fast code sometimes gets slower, and (impressively)
>> >>>>> > previously slow code will sometimes get faster, with updates to the
>> >>>>> > Julia codebase.
>> >>>>> Code is not supposed to get slower with newer releases. If this
>> >>>>> happens, please report the problem here or on GitHub (if possible
>> with
>> >>>>> a reproducible example). This will be very helpful to help avoiding
>> >>>>> regressions.
>> >>>>>
>> >>>>> > No complaint here in general; I really appreciate the work all of
>> the
>> >>>>> > Core and package developers do, and variations in performance of
>> >>>>> > different codes it to be expected.
>> >>>>> > My question is this: has anyone in the Julia community thought
>> about
>> >>>>> > updated performance tips for writing high performance code?
>> >>>>> > Obviously, using the profiler, along with many of the tips
>> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>> manual/perfo
>> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as well as
>> >>>>> > guidance on how best to take advantage of recent changes on
>> master. I
>> >>>>> > know that document hasn't been stagnant in any sense, but
>> relatively
>> >>>>> > "big in any case, I'd be happy to help make some updates in a PR if
>> >>>>> > there's anything we come up with.
>> >>>>> I've just skimmed through this page, and I don't think any of the
>> >>>>> advice given there is outdated. What's new in master is that
>> anonymous
>> >>>>> functions (and therefore map) are now fast, but that wasn't
>> previously
>> >>>>> mentioned in the tips as a performance issue anyway.
>> >>>>>
>> >>>>> The only small sentence which should likely be removed is "for
>> example,
>> >>>>> currently it’s not possible to infer the return type of an anonymous
>> >>>>> function". Type inference seems to work fine now on master with
>> >>>>> anonymous functions. I'll leave others confirm this.
>> >>>>>
>> >>>>> Anyway, do you have any specific points in mind?
>> >>>>>
>> >>>>>
>> >>>>> Regards
>> >>>>>
>> >>>>
>> >>>>
>> >>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Harish Kumar
Thank you. I agree on python.. but my question was did they update the Pyjulia libraries for latest Julia version? . We tried with 0.4.3 which failed 6 months back. So we revered to 0.3.4. Or is this library remain same for all Julia versions?

Any suggestion on this?

On Sat, Nov 19, 2016 at 7:38 PM, Mauro <[hidden email]> wrote:
On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]> wrote:
> Will it support Python 3.4 ? I am calling this from pyjulia interface

https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
but it doesn't say that 3.4 is not supported.  So you should try.

> On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>
>> Julia 0.3.12, that's a stone-age version of Julia.  You should move to 0.5!
>>
>> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>> wrote:
>> > I am using Version 0.3.12 calling from python (pyjulia). I do LME fit
>> with
>> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+
>> data
>> > transfer time). Any tips?
>> >       using MixedModels
>> >       modelREML = lmm({formula}, dataset)
>> >       reml!(modelREML,true)
>> >       lmeModel = fit(modelREML)
>> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>> estimate
>> > = coeftable(lmeModel).mat[:,1],
>> >                      stdError = coeftable(lmeModel).mat[:,2],zVal =
>> > coeftable(lmeModel).mat[:,3])
>> >
>> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>> wrote:
>> >>
>> >> I'm glad that particular slow case got faster! If you want to submit
>> some
>> >> reduced version of it as a performance test, we could still include it
>> in
>> >> our perf suite. And of course, if you find that anything else has ever
>> >> slowed down, please don't hesitate to file an issue.
>> >>
>> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <[hidden email]
>> >> <javascript:>> wrote:
>> >>
>> >>> Yes, understood about difficulty keeping track of regressions. I was
>> >>> originally going to send a message relating up to 2x longer test time
>> on
>> >>> the same code on Travis, but it appears as though something has
>> changed in
>> >>> the nightly build available to CI that now gives significantly faster
>> >>> builds, even though the previous poor performance had been
>> dependable...
>> >>> Evidently that build is not as up-to-date as I thought. Our code is
>> >>> currently not open source, but should be soon after which I can share
>> an
>> >>> example.
>> >>>
>> >>> Thanks for your comments, and thanks again for your work on Julia.
>> >>>
>> >>> -Max
>> >>>
>> >>>
>> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski
>> wrote:
>> >>>>
>> >>>> Yes, ideally code should not get slower with new releases –
>> >>>> unfortunately, keeping track of performance regressions can be a bit
>> of a
>> >>>> game of whack-a-mole. Having examples of code whose speed has
>> regressed is
>> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now have
>> some
>> >>>> great performance regression tracking infrastructure, but of course we
>> >>>> always need more things to test!
>> >>>>
>> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <[hidden email]
>> >
>> >>>> wrote:
>> >>>>
>> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit :
>> >>>>> > I've really been enjoying writing Julia code as a user, and
>> following
>> >>>>> > the language as it develops, but I have noticed that over time,
>> >>>>> > previously fast code sometimes gets slower, and (impressively)
>> >>>>> > previously slow code will sometimes get faster, with updates to the
>> >>>>> > Julia codebase.
>> >>>>> Code is not supposed to get slower with newer releases. If this
>> >>>>> happens, please report the problem here or on GitHub (if possible
>> with
>> >>>>> a reproducible example). This will be very helpful to help avoiding
>> >>>>> regressions.
>> >>>>>
>> >>>>> > No complaint here in general; I really appreciate the work all of
>> the
>> >>>>> > Core and package developers do, and variations in performance of
>> >>>>> > different codes it to be expected.
>> >>>>> > My question is this: has anyone in the Julia community thought
>> about
>> >>>>> > updated performance tips for writing high performance code?
>> >>>>> > Obviously, using the profiler, along with many of the tips
>> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>> manual/perfo
>> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as well as
>> >>>>> > guidance on how best to take advantage of recent changes on
>> master. I
>> >>>>> > know that document hasn't been stagnant in any sense, but
>> relatively
>> >>>>> > "big in any case, I'd be happy to help make some updates in a PR if
>> >>>>> > there's anything we come up with.
>> >>>>> I've just skimmed through this page, and I don't think any of the
>> >>>>> advice given there is outdated. What's new in master is that
>> anonymous
>> >>>>> functions (and therefore map) are now fast, but that wasn't
>> previously
>> >>>>> mentioned in the tips as a performance issue anyway.
>> >>>>>
>> >>>>> The only small sentence which should likely be removed is "for
>> example,
>> >>>>> currently it’s not possible to infer the return type of an anonymous
>> >>>>> function". Type inference seems to work fine now on master with
>> >>>>> anonymous functions. I'll leave others confirm this.
>> >>>>>
>> >>>>> Anyway, do you have any specific points in mind?
>> >>>>>
>> >>>>>
>> >>>>> Regards
>> >>>>>
>> >>>>
>> >>>>
>> >>
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Mauro
On Sat, 2016-11-19 at 20:48, Harish Kumar <[hidden email]> wrote:
> Thank you. I agree on python.. but my question was did they update the
> Pyjulia libraries for latest Julia version? . We tried with 0.4.3 which
> failed 6 months back. So we revered to 0.3.4. Or is this library remain
> same for all Julia versions?
>
> Any suggestion on this?

They are testing against the latest release, i.e. 0.5:
https://github.com/JuliaPy/pyjulia/blob/master/.travis.yml

You should try and file an issue if it doesn't work.  6 months are a
long time at the current julia development pace.

>
> On Sat, Nov 19, 2016 at 7:38 PM, Mauro <[hidden email]> wrote:
>
>> On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]>
>> wrote:
>> > Will it support Python 3.4 ? I am calling this from pyjulia interface
>>
>> https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
>> but it doesn't say that 3.4 is not supported.  So you should try.
>>
>> > On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>> >
>> >> Julia 0.3.12, that's a stone-age version of Julia.  You should move to
>> 0.5!
>> >>
>> >> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>> >> wrote:
>> >> > I am using Version 0.3.12 calling from python (pyjulia). I do LME fit
>> >> with
>> >> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+
>> >> data
>> >> > transfer time). Any tips?
>> >> >       using MixedModels
>> >> >       modelREML = lmm({formula}, dataset)
>> >> >       reml!(modelREML,true)
>> >> >       lmeModel = fit(modelREML)
>> >> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>> >> estimate
>> >> > = coeftable(lmeModel).mat[:,1],
>> >> >                      stdError = coeftable(lmeModel).mat[:,2],zVal =
>> >> > coeftable(lmeModel).mat[:,3])
>> >> >
>> >> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>> >> wrote:
>> >> >>
>> >> >> I'm glad that particular slow case got faster! If you want to submit
>> >> some
>> >> >> reduced version of it as a performance test, we could still include
>> it
>> >> in
>> >> >> our perf suite. And of course, if you find that anything else has
>> ever
>> >> >> slowed down, please don't hesitate to file an issue.
>> >> >>
>> >> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <
>> [hidden email]
>> >> >> <javascript:>> wrote:
>> >> >>
>> >> >>> Yes, understood about difficulty keeping track of regressions. I was
>> >> >>> originally going to send a message relating up to 2x longer test
>> time
>> >> on
>> >> >>> the same code on Travis, but it appears as though something has
>> >> changed in
>> >> >>> the nightly build available to CI that now gives significantly
>> faster
>> >> >>> builds, even though the previous poor performance had been
>> >> dependable...
>> >> >>> Evidently that build is not as up-to-date as I thought. Our code is
>> >> >>> currently not open source, but should be soon after which I can
>> share
>> >> an
>> >> >>> example.
>> >> >>>
>> >> >>> Thanks for your comments, and thanks again for your work on Julia.
>> >> >>>
>> >> >>> -Max
>> >> >>>
>> >> >>>
>> >> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski
>> >> wrote:
>> >> >>>>
>> >> >>>> Yes, ideally code should not get slower with new releases –
>> >> >>>> unfortunately, keeping track of performance regressions can be a
>> bit
>> >> of a
>> >> >>>> game of whack-a-mole. Having examples of code whose speed has
>> >> regressed is
>> >> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now have
>> >> some
>> >> >>>> great performance regression tracking infrastructure, but of
>> course we
>> >> >>>> always need more things to test!
>> >> >>>>
>> >> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <
>> [hidden email]
>> >> >
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit
>> :
>> >> >>>>> > I've really been enjoying writing Julia code as a user, and
>> >> following
>> >> >>>>> > the language as it develops, but I have noticed that over time,
>> >> >>>>> > previously fast code sometimes gets slower, and (impressively)
>> >> >>>>> > previously slow code will sometimes get faster, with updates to
>> the
>> >> >>>>> > Julia codebase.
>> >> >>>>> Code is not supposed to get slower with newer releases. If this
>> >> >>>>> happens, please report the problem here or on GitHub (if possible
>> >> with
>> >> >>>>> a reproducible example). This will be very helpful to help
>> avoiding
>> >> >>>>> regressions.
>> >> >>>>>
>> >> >>>>> > No complaint here in general; I really appreciate the work all
>> of
>> >> the
>> >> >>>>> > Core and package developers do, and variations in performance of
>> >> >>>>> > different codes it to be expected.
>> >> >>>>> > My question is this: has anyone in the Julia community thought
>> >> about
>> >> >>>>> > updated performance tips for writing high performance code?
>> >> >>>>> > Obviously, using the profiler, along with many of the tips
>> >> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>> >> manual/perfo
>> >> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>> >> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as well
>> as
>> >> >>>>> > guidance on how best to take advantage of recent changes on
>> >> master. I
>> >> >>>>> > know that document hasn't been stagnant in any sense, but
>> >> relatively
>> >> >>>>> > "big in any case, I'd be happy to help make some updates in a
>> PR if
>> >> >>>>> > there's anything we come up with.
>> >> >>>>> I've just skimmed through this page, and I don't think any of the
>> >> >>>>> advice given there is outdated. What's new in master is that
>> >> anonymous
>> >> >>>>> functions (and therefore map) are now fast, but that wasn't
>> >> previously
>> >> >>>>> mentioned in the tips as a performance issue anyway.
>> >> >>>>>
>> >> >>>>> The only small sentence which should likely be removed is "for
>> >> example,
>> >> >>>>> currently it’s not possible to infer the return type of an
>> anonymous
>> >> >>>>> function". Type inference seems to work fine now on master with
>> >> >>>>> anonymous functions. I'll leave others confirm this.
>> >> >>>>>
>> >> >>>>> Anyway, do you have any specific points in mind?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Regards
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>
>> >>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Harish Kumar
I found the cause for this ... When i run julia 0.3.2 or 0.5 as standalone (mix model) it uses all the available cores from my server, so it was fast.

If i call Julia from Python (Pyjulia), i see only one core is busy with python process (100% cpu) and all other cores are free.  Can you help me how can i force Pyjulia/python to use available cores from my server?

Regards,
 Harish




On Sat, Nov 19, 2016 at 8:32 PM, Mauro <[hidden email]> wrote:
On Sat, 2016-11-19 at 20:48, Harish Kumar <[hidden email]> wrote:
> Thank you. I agree on python.. but my question was did they update the
> Pyjulia libraries for latest Julia version? . We tried with 0.4.3 which
> failed 6 months back. So we revered to 0.3.4. Or is this library remain
> same for all Julia versions?
>
> Any suggestion on this?

They are testing against the latest release, i.e. 0.5:
https://github.com/JuliaPy/pyjulia/blob/master/.travis.yml

You should try and file an issue if it doesn't work.  6 months are a
long time at the current julia development pace.

>
> On Sat, Nov 19, 2016 at 7:38 PM, Mauro <[hidden email]> wrote:
>
>> On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]>
>> wrote:
>> > Will it support Python 3.4 ? I am calling this from pyjulia interface
>>
>> https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
>> but it doesn't say that 3.4 is not supported.  So you should try.
>>
>> > On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>> >
>> >> Julia 0.3.12, that's a stone-age version of Julia.  You should move to
>> 0.5!
>> >>
>> >> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>> >> wrote:
>> >> > I am using Version 0.3.12 calling from python (pyjulia). I do LME fit
>> >> with
>> >> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+
>> >> data
>> >> > transfer time). Any tips?
>> >> >       using MixedModels
>> >> >       modelREML = lmm({formula}, dataset)
>> >> >       reml!(modelREML,true)
>> >> >       lmeModel = fit(modelREML)
>> >> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>> >> estimate
>> >> > = coeftable(lmeModel).mat[:,1],
>> >> >                      stdError = coeftable(lmeModel).mat[:,2],zVal =
>> >> > coeftable(lmeModel).mat[:,3])
>> >> >
>> >> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>> >> wrote:
>> >> >>
>> >> >> I'm glad that particular slow case got faster! If you want to submit
>> >> some
>> >> >> reduced version of it as a performance test, we could still include
>> it
>> >> in
>> >> >> our perf suite. And of course, if you find that anything else has
>> ever
>> >> >> slowed down, please don't hesitate to file an issue.
>> >> >>
>> >> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <
>> [hidden email]
>> >> >> <javascript:>> wrote:
>> >> >>
>> >> >>> Yes, understood about difficulty keeping track of regressions. I was
>> >> >>> originally going to send a message relating up to 2x longer test
>> time
>> >> on
>> >> >>> the same code on Travis, but it appears as though something has
>> >> changed in
>> >> >>> the nightly build available to CI that now gives significantly
>> faster
>> >> >>> builds, even though the previous poor performance had been
>> >> dependable...
>> >> >>> Evidently that build is not as up-to-date as I thought. Our code is
>> >> >>> currently not open source, but should be soon after which I can
>> share
>> >> an
>> >> >>> example.
>> >> >>>
>> >> >>> Thanks for your comments, and thanks again for your work on Julia.
>> >> >>>
>> >> >>> -Max
>> >> >>>
>> >> >>>
>> >> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski
>> >> wrote:
>> >> >>>>
>> >> >>>> Yes, ideally code should not get slower with new releases –
>> >> >>>> unfortunately, keeping track of performance regressions can be a
>> bit
>> >> of a
>> >> >>>> game of whack-a-mole. Having examples of code whose speed has
>> >> regressed is
>> >> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now have
>> >> some
>> >> >>>> great performance regression tracking infrastructure, but of
>> course we
>> >> >>>> always need more things to test!
>> >> >>>>
>> >> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <
>> [hidden email]
>> >> >
>> >> >>>> wrote:
>> >> >>>>
>> >> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit
>> :
>> >> >>>>> > I've really been enjoying writing Julia code as a user, and
>> >> following
>> >> >>>>> > the language as it develops, but I have noticed that over time,
>> >> >>>>> > previously fast code sometimes gets slower, and (impressively)
>> >> >>>>> > previously slow code will sometimes get faster, with updates to
>> the
>> >> >>>>> > Julia codebase.
>> >> >>>>> Code is not supposed to get slower with newer releases. If this
>> >> >>>>> happens, please report the problem here or on GitHub (if possible
>> >> with
>> >> >>>>> a reproducible example). This will be very helpful to help
>> avoiding
>> >> >>>>> regressions.
>> >> >>>>>
>> >> >>>>> > No complaint here in general; I really appreciate the work all
>> of
>> >> the
>> >> >>>>> > Core and package developers do, and variations in performance of
>> >> >>>>> > different codes it to be expected.
>> >> >>>>> > My question is this: has anyone in the Julia community thought
>> >> about
>> >> >>>>> > updated performance tips for writing high performance code?
>> >> >>>>> > Obviously, using the profiler, along with many of the tips
>> >> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>> >> manual/perfo
>> >> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>> >> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as well
>> as
>> >> >>>>> > guidance on how best to take advantage of recent changes on
>> >> master. I
>> >> >>>>> > know that document hasn't been stagnant in any sense, but
>> >> relatively
>> >> >>>>> > "big in any case, I'd be happy to help make some updates in a
>> PR if
>> >> >>>>> > there's anything we come up with.
>> >> >>>>> I've just skimmed through this page, and I don't think any of the
>> >> >>>>> advice given there is outdated. What's new in master is that
>> >> anonymous
>> >> >>>>> functions (and therefore map) are now fast, but that wasn't
>> >> previously
>> >> >>>>> mentioned in the tips as a performance issue anyway.
>> >> >>>>>
>> >> >>>>> The only small sentence which should likely be removed is "for
>> >> example,
>> >> >>>>> currently it’s not possible to infer the return type of an
>> anonymous
>> >> >>>>> function". Type inference seems to work fine now on master with
>> >> >>>>> anonymous functions. I'll leave others confirm this.
>> >> >>>>>
>> >> >>>>> Anyway, do you have any specific points in mind?
>> >> >>>>>
>> >> >>>>>
>> >> >>>>> Regards
>> >> >>>>>
>> >> >>>>
>> >> >>>>
>> >> >>
>> >>
>>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Mauro
On Tue, 2016-11-22 at 14:12, Harish Kumar <[hidden email]> wrote:
> I found the cause for this ... When i run julia 0.3.2 or 0.5 as standalone
> (mix model) it uses all the available cores from my server, so it was fast.
>
> If i call Julia from Python (Pyjulia), i see only one core is busy with
> python process (100% cpu) and all other cores are free.  Can you help me
> how can i force Pyjulia/python to use available cores from my server?

I can't help you there.  If no-one answers here try posting a new
question on discourse.julialang.org

> Regards,
>  Harish
>
>
>
>
> On Sat, Nov 19, 2016 at 8:32 PM, Mauro <[hidden email]> wrote:
>
>> On Sat, 2016-11-19 at 20:48, Harish Kumar <[hidden email]>
>> wrote:
>> > Thank you. I agree on python.. but my question was did they update the
>> > Pyjulia libraries for latest Julia version? . We tried with 0.4.3 which
>> > failed 6 months back. So we revered to 0.3.4. Or is this library remain
>> > same for all Julia versions?
>> >
>> > Any suggestion on this?
>>
>> They are testing against the latest release, i.e. 0.5:
>> https://github.com/JuliaPy/pyjulia/blob/master/.travis.yml
>>
>> You should try and file an issue if it doesn't work.  6 months are a
>> long time at the current julia development pace.
>>
>> >
>> > On Sat, Nov 19, 2016 at 7:38 PM, Mauro <[hidden email]> wrote:
>> >
>> >> On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]>
>> >> wrote:
>> >> > Will it support Python 3.4 ? I am calling this from pyjulia interface
>> >>
>> >> https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
>> >> but it doesn't say that 3.4 is not supported.  So you should try.
>> >>
>> >> > On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>> >> >
>> >> >> Julia 0.3.12, that's a stone-age version of Julia.  You should move
>> to
>> >> 0.5!
>> >> >>
>> >> >> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>> >> >> wrote:
>> >> >> > I am using Version 0.3.12 calling from python (pyjulia). I do LME
>> fit
>> >> >> with
>> >> >> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model
>> (+
>> >> >> data
>> >> >> > transfer time). Any tips?
>> >> >> >       using MixedModels
>> >> >> >       modelREML = lmm({formula}, dataset)
>> >> >> >       reml!(modelREML,true)
>> >> >> >       lmeModel = fit(modelREML)
>> >> >> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>> >> >> estimate
>> >> >> > = coeftable(lmeModel).mat[:,1],
>> >> >> >                      stdError = coeftable(lmeModel).mat[:,2],zVal
>> =
>> >> >> > coeftable(lmeModel).mat[:,3])
>> >> >> >
>> >> >> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>> >> >> wrote:
>> >> >> >>
>> >> >> >> I'm glad that particular slow case got faster! If you want to
>> submit
>> >> >> some
>> >> >> >> reduced version of it as a performance test, we could still
>> include
>> >> it
>> >> >> in
>> >> >> >> our perf suite. And of course, if you find that anything else has
>> >> ever
>> >> >> >> slowed down, please don't hesitate to file an issue.
>> >> >> >>
>> >> >> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <
>> >> [hidden email]
>> >> >> >> <javascript:>> wrote:
>> >> >> >>
>> >> >> >>> Yes, understood about difficulty keeping track of regressions. I
>> was
>> >> >> >>> originally going to send a message relating up to 2x longer test
>> >> time
>> >> >> on
>> >> >> >>> the same code on Travis, but it appears as though something has
>> >> >> changed in
>> >> >> >>> the nightly build available to CI that now gives significantly
>> >> faster
>> >> >> >>> builds, even though the previous poor performance had been
>> >> >> dependable...
>> >> >> >>> Evidently that build is not as up-to-date as I thought. Our code
>> is
>> >> >> >>> currently not open source, but should be soon after which I can
>> >> share
>> >> >> an
>> >> >> >>> example.
>> >> >> >>>
>> >> >> >>> Thanks for your comments, and thanks again for your work on
>> Julia.
>> >> >> >>>
>> >> >> >>> -Max
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan
>> Karpinski
>> >> >> wrote:
>> >> >> >>>>
>> >> >> >>>> Yes, ideally code should not get slower with new releases –
>> >> >> >>>> unfortunately, keeping track of performance regressions can be a
>> >> bit
>> >> >> of a
>> >> >> >>>> game of whack-a-mole. Having examples of code whose speed has
>> >> >> regressed is
>> >> >> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now
>> have
>> >> >> some
>> >> >> >>>> great performance regression tracking infrastructure, but of
>> >> course we
>> >> >> >>>> always need more things to test!
>> >> >> >>>>
>> >> >> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <
>> >> [hidden email]
>> >> >> >
>> >> >> >>>> wrote:
>> >> >> >>>>
>> >> >> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a
>> écrit
>> >> :
>> >> >> >>>>> > I've really been enjoying writing Julia code as a user, and
>> >> >> following
>> >> >> >>>>> > the language as it develops, but I have noticed that over
>> time,
>> >> >> >>>>> > previously fast code sometimes gets slower, and
>> (impressively)
>> >> >> >>>>> > previously slow code will sometimes get faster, with updates
>> to
>> >> the
>> >> >> >>>>> > Julia codebase.
>> >> >> >>>>> Code is not supposed to get slower with newer releases. If this
>> >> >> >>>>> happens, please report the problem here or on GitHub (if
>> possible
>> >> >> with
>> >> >> >>>>> a reproducible example). This will be very helpful to help
>> >> avoiding
>> >> >> >>>>> regressions.
>> >> >> >>>>>
>> >> >> >>>>> > No complaint here in general; I really appreciate the work
>> all
>> >> of
>> >> >> the
>> >> >> >>>>> > Core and package developers do, and variations in
>> performance of
>> >> >> >>>>> > different codes it to be expected.
>> >> >> >>>>> > My question is this: has anyone in the Julia community
>> thought
>> >> >> about
>> >> >> >>>>> > updated performance tips for writing high performance code?
>> >> >> >>>>> > Obviously, using the profiler, along with many of the tips
>> >> >> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>> >> >> manual/perfo
>> >> >> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>> >> >> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as
>> well
>> >> as
>> >> >> >>>>> > guidance on how best to take advantage of recent changes on
>> >> >> master. I
>> >> >> >>>>> > know that document hasn't been stagnant in any sense, but
>> >> >> relatively
>> >> >> >>>>> > "big in any case, I'd be happy to help make some updates in a
>> >> PR if
>> >> >> >>>>> > there's anything we come up with.
>> >> >> >>>>> I've just skimmed through this page, and I don't think any of
>> the
>> >> >> >>>>> advice given there is outdated. What's new in master is that
>> >> >> anonymous
>> >> >> >>>>> functions (and therefore map) are now fast, but that wasn't
>> >> >> previously
>> >> >> >>>>> mentioned in the tips as a performance issue anyway.
>> >> >> >>>>>
>> >> >> >>>>> The only small sentence which should likely be removed is "for
>> >> >> example,
>> >> >> >>>>> currently it’s not possible to infer the return type of an
>> >> anonymous
>> >> >> >>>>> function". Type inference seems to work fine now on master with
>> >> >> >>>>> anonymous functions. I'll leave others confirm this.
>> >> >> >>>>>
>> >> >> >>>>> Anyway, do you have any specific points in mind?
>> >> >> >>>>>
>> >> >> >>>>>
>> >> >> >>>>> Regards
>> >> >> >>>>>
>> >> >> >>>>
>> >> >> >>>>
>> >> >> >>
>> >> >>
>> >>
>>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Douglas Bates
In reply to this post by Harish Kumar
On Tuesday, November 22, 2016 at 1:12:26 PM UTC-6, Harish Kumar wrote:
> I found the cause for this ... When i run julia 0.3.2 or 0.5 as standalone (mix model) it uses all the available cores from my server, so it was fast.

Fitting a linear mixed effects model only uses multiple threads for the BLAS (Basic Linear Algebra Subroutine) calls and a few LAPACK calls.  In Julia v0.5 you may be able to set the number of threads for the BLAS by calling, say,

BLAS.set_num_threads(4)

(or some other number) before trying to fit a model.  Be aware that increasing the number of threads doesn't always make things faster.  You may need to do a bit of experimentation to determine a suitable number of threads.

Can you describe the formula you are using?  If you are trying to fit a "maximal" model with large-dimensional vector-valued random effects for crossed grouping factors you should be aware that most of the time fitting such models is just a convenient way of burning up a lot of computing time.

 

> If i call Julia from Python (Pyjulia), i see only one core is busy with python process (100% cpu) and all other cores are free.  Can you help me how can i force Pyjulia/python to use available cores from my server?
>
>
> Regards,
>  Harish
>
>
>
>
>
>
>
>
> On Sat, Nov 19, 2016 at 8:32 PM, Mauro <[hidden email]> wrote:
> On Sat, 2016-11-19 at 20:48, Harish Kumar <[hidden email]> wrote:
>
> > Thank you. I agree on python.. but my question was did they update the
>
> > Pyjulia libraries for latest Julia version? . We tried with 0.4.3 which
>
> > failed 6 months back. So we revered to 0.3.4. Or is this library remain
>
> > same for all Julia versions?
>
> >
>
> > Any suggestion on this?
>
>
>
> They are testing against the latest release, i.e. 0.5:
>
> https://github.com/JuliaPy/pyjulia/blob/master/.travis.yml
>
>
>
> You should try and file an issue if it doesn't work.  6 months are a
>
> long time at the current julia development pace.
>
>
>
>
>
> >
>
> > On Sat, Nov 19, 2016 at 7:38 PM, Mauro <[hidden email]> wrote:
>
> >
>
> >> On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]>
>
> >> wrote:
>
> >> > Will it support Python 3.4 ? I am calling this from pyjulia interface
>
> >>
>
> >> https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
>
> >> but it doesn't say that 3.4 is not supported.  So you should try.
>
> >>
>
> >> > On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>
> >> >
>
> >> >> Julia 0.3.12, that's a stone-age version of Julia.  You should move to
>
> >> 0.5!
>
> >> >>
>
> >> >> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>
> >> >> wrote:
>
> >> >> > I am using Version 0.3.12 calling from python (pyjulia). I do LME fit
>
> >> >> with
>
> >> >> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+
>
> >> >> data
>
> >> >> > transfer time). Any tips?
>
> >> >> >       using MixedModels
>
> >> >> >       modelREML = lmm({formula}, dataset)
>
> >> >> >       reml!(modelREML,true)
>
> >> >> >       lmeModel = fit(modelREML)
>
> >> >> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>
> >> >> estimate
>
> >> >> > = coeftable(lmeModel).mat[:,1],
>
> >> >> >                      stdError = coeftable(lmeModel).mat[:,2],zVal =
>
> >> >> > coeftable(lmeModel).mat[:,3])
>
> >> >> >
>
> >> >> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>
> >> >> wrote:
>
> >> >> >>
>
> >> >> >> I'm glad that particular slow case got faster! If you want to submit
>
> >> >> some
>
> >> >> >> reduced version of it as a performance test, we could still include
>
> >> it
>
> >> >> in
>
> >> >> >> our perf suite. And of course, if you find that anything else has
>
> >> ever
>
> >> >> >> slowed down, please don't hesitate to file an issue.
>
> >> >> >>
>
> >> >> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <
>
> >> [hidden email]
>
> >> >> >> <javascript:>> wrote:
>
> >> >> >>
>
> >> >> >>> Yes, understood about difficulty keeping track of regressions. I was
>
> >> >> >>> originally going to send a message relating up to 2x longer test
>
> >> time
>
> >> >> on
>
> >> >> >>> the same code on Travis, but it appears as though something has
>
> >> >> changed in
>
> >> >> >>> the nightly build available to CI that now gives significantly
>
> >> faster
>
> >> >> >>> builds, even though the previous poor performance had been
>
> >> >> dependable...
>
> >> >> >>> Evidently that build is not as up-to-date as I thought. Our code is
>
> >> >> >>> currently not open source, but should be soon after which I can
>
> >> share
>
> >> >> an
>
> >> >> >>> example.
>
> >> >> >>>
>
> >> >> >>> Thanks for your comments, and thanks again for your work on Julia.
>
> >> >> >>>
>
> >> >> >>> -Max
>
> >> >> >>>
>
> >> >> >>>
>
> >> >> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski
>
> >> >> wrote:
>
> >> >> >>>>
>
> >> >> >>>> Yes, ideally code should not get slower with new releases –
>
> >> >> >>>> unfortunately, keeping track of performance regressions can be a
>
> >> bit
>
> >> >> of a
>
> >> >> >>>> game of whack-a-mole. Having examples of code whose speed has
>
> >> >> regressed is
>
> >> >> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now have
>
> >> >> some
>
> >> >> >>>> great performance regression tracking infrastructure, but of
>
> >> course we
>
> >> >> >>>> always need more things to test!
>
> >> >> >>>>
>
> >> >> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <
>
> >> [hidden email]
>
> >> >> >
>
> >> >> >>>> wrote:
>
> >> >> >>>>
>
> >> >> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit
>
> >> :
>
> >> >> >>>>> > I've really been enjoying writing Julia code as a user, and
>
> >> >> following
>
> >> >> >>>>> > the language as it develops, but I have noticed that over time,
>
> >> >> >>>>> > previously fast code sometimes gets slower, and (impressively)
>
> >> >> >>>>> > previously slow code will sometimes get faster, with updates to
>
> >> the
>
> >> >> >>>>> > Julia codebase.
>
> >> >> >>>>> Code is not supposed to get slower with newer releases. If this
>
> >> >> >>>>> happens, please report the problem here or on GitHub (if possible
>
> >> >> with
>
> >> >> >>>>> a reproducible example). This will be very helpful to help
>
> >> avoiding
>
> >> >> >>>>> regressions.
>
> >> >> >>>>>
>
> >> >> >>>>> > No complaint here in general; I really appreciate the work all
>
> >> of
>
> >> >> the
>
> >> >> >>>>> > Core and package developers do, and variations in performance of
>
> >> >> >>>>> > different codes it to be expected.
>
> >> >> >>>>> > My question is this: has anyone in the Julia community thought
>
> >> >> about
>
> >> >> >>>>> > updated performance tips for writing high performance code?
>
> >> >> >>>>> > Obviously, using the profiler, along with many of the tips
>
> >> >> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>
> >> >> manual/perfo
>
> >> >> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>
> >> >> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as well
>
> >> as
>
> >> >> >>>>> > guidance on how best to take advantage of recent changes on
>
> >> >> master. I
>
> >> >> >>>>> > know that document hasn't been stagnant in any sense, but
>
> >> >> relatively
>
> >> >> >>>>> > "big in any case, I'd be happy to help make some updates in a
>
> >> PR if
>
> >> >> >>>>> > there's anything we come up with.
>
> >> >> >>>>> I've just skimmed through this page, and I don't think any of the
>
> >> >> >>>>> advice given there is outdated. What's new in master is that
>
> >> >> anonymous
>
> >> >> >>>>> functions (and therefore map) are now fast, but that wasn't
>
> >> >> previously
>
> >> >> >>>>> mentioned in the tips as a performance issue anyway.
>
> >> >> >>>>>
>
> >> >> >>>>> The only small sentence which should likely be removed is "for
>
> >> >> example,
>
> >> >> >>>>> currently it’s not possible to infer the return type of an
>
> >> anonymous
>
> >> >> >>>>> function". Type inference seems to work fine now on master with
>
> >> >> >>>>> anonymous functions. I'll leave others confirm this.
>
> >> >> >>>>>
>
> >> >> >>>>> Anyway, do you have any specific points in mind?
>
> >> >> >>>>>
>
> >> >> >>>>>
>
> >> >> >>>>> Regards
>
> >> >> >>>>>
>
> >> >> >>>>
>
> >> >> >>>>
>
> >> >> >>
>
> >> >>
>
> >>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Updated performance tips?

Harish Kumar
I have copied my code in the below link (Julia user group), let me know if you cant access that


On Wed, Nov 23, 2016 at 2:03 PM, Douglas Bates <[hidden email]> wrote:
On Tuesday, November 22, 2016 at 1:12:26 PM UTC-6, Harish Kumar wrote:
> I found the cause for this ... When i run julia 0.3.2 or 0.5 as standalone (mix model) it uses all the available cores from my server, so it was fast.

Fitting a linear mixed effects model only uses multiple threads for the BLAS (Basic Linear Algebra Subroutine) calls and a few LAPACK calls.  In Julia v0.5 you may be able to set the number of threads for the BLAS by calling, say,

BLAS.set_num_threads(4)

(or some other number) before trying to fit a model.  Be aware that increasing the number of threads doesn't always make things faster.  You may need to do a bit of experimentation to determine a suitable number of threads.

Can you describe the formula you are using?  If you are trying to fit a "maximal" model with large-dimensional vector-valued random effects for crossed grouping factors you should be aware that most of the time fitting such models is just a convenient way of burning up a lot of computing time.


> If i call Julia from Python (Pyjulia), i see only one core is busy with python process (100% cpu) and all other cores are free.  Can you help me how can i force Pyjulia/python to use available cores from my server?
>
>
> Regards,
>  Harish
>
>
>
>
>
>
>
>
> On Sat, Nov 19, 2016 at 8:32 PM, Mauro <[hidden email]> wrote:
> On Sat, 2016-11-19 at 20:48, Harish Kumar <[hidden email]> wrote:
>
> > Thank you. I agree on python.. but my question was did they update the
>
> > Pyjulia libraries for latest Julia version? . We tried with 0.4.3 which
>
> > failed 6 months back. So we revered to 0.3.4. Or is this library remain
>
> > same for all Julia versions?
>
> >
>
> > Any suggestion on this?
>
>
>
> They are testing against the latest release, i.e. 0.5:
>
> https://github.com/JuliaPy/pyjulia/blob/master/.travis.yml
>
>
>
> You should try and file an issue if it doesn't work.  6 months are a
>
> long time at the current julia development pace.
>
>
>
>
>
> >
>
> > On Sat, Nov 19, 2016 at 7:38 PM, Mauro <[hidden email]> wrote:
>
> >
>
> >> On Sat, 2016-11-19 at 18:36, Harish Kumar <[hidden email]>
>
> >> wrote:
>
> >> > Will it support Python 3.4 ? I am calling this from pyjulia interface
>
> >>
>
> >> https://github.com/JuliaPy/pyjulia says that it is tested against 3.5,
>
> >> but it doesn't say that 3.4 is not supported.  So you should try.
>
> >>
>
> >> > On Nov 19, 2016 4:58 PM, "Mauro" <[hidden email]> wrote:
>
> >> >
>
> >> >> Julia 0.3.12, that's a stone-age version of Julia.  You should move to
>
> >> 0.5!
>
> >> >>
>
> >> >> On Sat, 2016-11-19 at 16:42, Harish Kumar <[hidden email]>
>
> >> >> wrote:
>
> >> >> > I am using Version 0.3.12 calling from python (pyjulia). I do LME fit
>
> >> >> with
>
> >> >> > 2.8 M rows and 60-70 Variables. It is taking 2 hours just to model (+
>
> >> >> data
>
> >> >> > transfer time). Any tips?
>
> >> >> >       using MixedModels
>
> >> >> >       modelREML = lmm({formula}, dataset)
>
> >> >> >       reml!(modelREML,true)
>
> >> >> >       lmeModel = fit(modelREML)
>
> >> >> >       fixedDF = DataFrame(fixedEffVar = coeftable(lmeModel).rownms,
>
> >> >> estimate
>
> >> >> > = coeftable(lmeModel).mat[:,1],
>
> >> >> >                      stdError = coeftable(lmeModel).mat[:,2],zVal =
>
> >> >> > coeftable(lmeModel).mat[:,3])
>
> >> >> >
>
> >> >> > On Tuesday, February 23, 2016 at 9:16:47 AM UTC-6, Stefan Karpinski
>
> >> >> wrote:
>
> >> >> >>
>
> >> >> >> I'm glad that particular slow case got faster! If you want to submit
>
> >> >> some
>
> >> >> >> reduced version of it as a performance test, we could still include
>
> >> it
>
> >> >> in
>
> >> >> >> our perf suite. And of course, if you find that anything else has
>
> >> ever
>
> >> >> >> slowed down, please don't hesitate to file an issue.
>
> >> >> >>
>
> >> >> >> On Tue, Feb 23, 2016 at 9:55 AM, Jonathan Goldfarb <
>
> >> [hidden email]
>
> >> >> >> <javascript:>> wrote:
>
> >> >> >>
>
> >> >> >>> Yes, understood about difficulty keeping track of regressions. I was
>
> >> >> >>> originally going to send a message relating up to 2x longer test
>
> >> time
>
> >> >> on
>
> >> >> >>> the same code on Travis, but it appears as though something has
>
> >> >> changed in
>
> >> >> >>> the nightly build available to CI that now gives significantly
>
> >> faster
>
> >> >> >>> builds, even though the previous poor performance had been
>
> >> >> dependable...
>
> >> >> >>> Evidently that build is not as up-to-date as I thought. Our code is
>
> >> >> >>> currently not open source, but should be soon after which I can
>
> >> share
>
> >> >> an
>
> >> >> >>> example.
>
> >> >> >>>
>
> >> >> >>> Thanks for your comments, and thanks again for your work on Julia.
>
> >> >> >>>
>
> >> >> >>> -Max
>
> >> >> >>>
>
> >> >> >>>
>
> >> >> >>> On Monday, February 22, 2016 at 11:12:58 AM UTC-5, Stefan Karpinski
>
> >> >> wrote:
>
> >> >> >>>>
>
> >> >> >>>> Yes, ideally code should not get slower with new releases –
>
> >> >> >>>> unfortunately, keeping track of performance regressions can be a
>
> >> bit
>
> >> >> of a
>
> >> >> >>>> game of whack-a-mole. Having examples of code whose speed has
>
> >> >> regressed is
>
> >> >> >>>> very helpful. Thanks to Jarrett Revels excellent work, we now have
>
> >> >> some
>
> >> >> >>>> great performance regression tracking infrastructure, but of
>
> >> course we
>
> >> >> >>>> always need more things to test!
>
> >> >> >>>>
>
> >> >> >>>> On Mon, Feb 22, 2016 at 9:58 AM, Milan Bouchet-Valat <
>
> >> [hidden email]
>
> >> >> >
>
> >> >> >>>> wrote:
>
> >> >> >>>>
>
> >> >> >>>>> Le lundi 22 février 2016 à 06:27 -0800, Jonathan Goldfarb a écrit
>
> >> :
>
> >> >> >>>>> > I've really been enjoying writing Julia code as a user, and
>
> >> >> following
>
> >> >> >>>>> > the language as it develops, but I have noticed that over time,
>
> >> >> >>>>> > previously fast code sometimes gets slower, and (impressively)
>
> >> >> >>>>> > previously slow code will sometimes get faster, with updates to
>
> >> the
>
> >> >> >>>>> > Julia codebase.
>
> >> >> >>>>> Code is not supposed to get slower with newer releases. If this
>
> >> >> >>>>> happens, please report the problem here or on GitHub (if possible
>
> >> >> with
>
> >> >> >>>>> a reproducible example). This will be very helpful to help
>
> >> avoiding
>
> >> >> >>>>> regressions.
>
> >> >> >>>>>
>
> >> >> >>>>> > No complaint here in general; I really appreciate the work all
>
> >> of
>
> >> >> the
>
> >> >> >>>>> > Core and package developers do, and variations in performance of
>
> >> >> >>>>> > different codes it to be expected.
>
> >> >> >>>>> > My question is this: has anyone in the Julia community thought
>
> >> >> about
>
> >> >> >>>>> > updated performance tips for writing high performance code?
>
> >> >> >>>>> > Obviously, using the profiler, along with many of the tips
>
> >> >> >>>>> > at https://github.com/JuliaLang/julia/commits/master/doc/
>
> >> >> manual/perfo
>
> >> >> >>>>> > rmance-tips.rst still apply, but I am wondering more about
>
> >> >> >>>>> > general/structural ideas to keep in mind in Julia v0.4, as well
>
> >> as
>
> >> >> >>>>> > guidance on how best to take advantage of recent changes on
>
> >> >> master. I
>
> >> >> >>>>> > know that document hasn't been stagnant in any sense, but
>
> >> >> relatively
>
> >> >> >>>>> > "big in any case, I'd be happy to help make some updates in a
>
> >> PR if
>
> >> >> >>>>> > there's anything we come up with.
>
> >> >> >>>>> I've just skimmed through this page, and I don't think any of the
>
> >> >> >>>>> advice given there is outdated. What's new in master is that
>
> >> >> anonymous
>
> >> >> >>>>> functions (and therefore map) are now fast, but that wasn't
>
> >> >> previously
>
> >> >> >>>>> mentioned in the tips as a performance issue anyway.
>
> >> >> >>>>>
>
> >> >> >>>>> The only small sentence which should likely be removed is "for
>
> >> >> example,
>
> >> >> >>>>> currently it’s not possible to infer the return type of an
>
> >> anonymous
>
> >> >> >>>>> function". Type inference seems to work fine now on master with
>
> >> >> >>>>> anonymous functions. I'll leave others confirm this.
>
> >> >> >>>>>
>
> >> >> >>>>> Anyway, do you have any specific points in mind?
>
> >> >> >>>>>
>
> >> >> >>>>>
>
> >> >> >>>>> Regards
>
> >> >> >>>>>
>
> >> >> >>>>
>
> >> >> >>>>
>
> >> >> >>
>
> >> >>
>
> >>


Loading...