Loading packages is very slow...

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

Loading packages is very slow...

Jose Augusto
Hi

Loading of julia modules (0.3.0-rc1) is a slow process (compared to other languages). This is more visible when in the process of running/debugging.
Is there any way of speeding it up? 

TIA

Jose

Code:

 
t1=time()
println("start time: ",0)
println("using DataFrames... slow loading...")
using DataFrames
t2=time()
println("load time: ",t2-t1," secs")
println("using PyPlot... slow loading...")
using PyPlot
t3=time()
println("load time: ",t3-t2," secs")
println("using Winston... slow loading...")
using Winston
t4=time()
println("load time: ",t4-t3," secs")
 

Result: (julia 0.3.0-rc1, AMD A8-3850 on Windows 7, 64 bits)

C:\Users\...>julia timeLoading.jl
start time: 0.0
using DataFrames... slow loading...
load time: 12.457000017166138 secs
using PyPlot... slow loading...
INFO: Loading help data...
load time: 13.289999961853027 secs
using Winston... slow loading...
load time: 6.801000118255615 secs
Reply | Threaded
Open this post in threaded view
|

Loading packages is very slow...

Ivar Nesje
If you compile Julia yourself, you can use the undocumented usherimg.jl trick to cache the packages in the system image.  Search for userimg
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

John Myles White
For example, I used to have a base/userimg.jl file that read as follows:

Base.require("Distributions.jl")
Base.require("Optim.jl")
Base.require("DataFrames.jl")
Base.require("Gadfly.jl”)

 — John

On Jul 30, 2014, at 9:13 PM, Ivar Nesje <[hidden email]> wrote:

> If you compile Julia yourself, you can use the undocumented usherimg.jl trick to cache the packages in the system image.  Search for userimg

Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Ivar Nesje
Sorry for the typo usherimg.jl should be "userimg.jl". I typed it correctly, but somehow swiftkey managed to change it one more time than I corrected it.

kl. 07:40:20 UTC+2 torsdag 31. juli 2014 skrev John Myles White følgende:
For example, I used to have a base/userimg.jl file that read as follows:

Base.require("Distributions.jl")
Base.require("Optim.jl")
Base.require("DataFrames.jl")
Base.require("Gadfly.jl”)

 — John

On Jul 30, 2014, at 9:13 PM, Ivar Nesje <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="4CHQB2PPUxsJ" onmousedown="this.href='javascript:';return true;" onclick="this.href='javascript:';return true;">iva...@...> wrote:

> If you compile Julia yourself, you can use the undocumented usherimg.jl trick to cache the packages in the system image.  Search for userimg

Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Ivar, John, thank you.

I tried to compile julia with mingw a couple weeks ago (when talking with devs about openblas issues) and I didn't succeed, so I can't implement the solution you point out  (compilation from sources). I undesrtand the slowness is due to repeated JIT compilation of the module/package each time julia is called and 'using' is executed. I was looking for the existence of some middle-way solution such as .pyc files in Python.

Another quirk: the loading time is not very important if one is using Julia REPL-- it is imported only once; but when using the command line it becomes annoying because its done in each run. So I tried the following: wrapping the code for loading modules inside a function:

function mainLoad() 
    ### code for importing modules here...
end

an then in the REPL I have an error. Apparently we cannot call "using..." inside a function:

julia> include("timeLoading.jl")
mainLoad (generic function with 1 method)
julia> mainLoad()
ERROR: error compiling mainLoad: unsupported or misplaced expression using in function mainLoad
julia>

Finally, to conclude, I think the more efficient way (which I'm using now) is launching the REPL, importing only once the packages, developing everything in the script as a function and load it with include("script.jl")... at least in the developing stage...

Thanks again

Jose


On Thursday, July 31, 2014 4:45:08 AM UTC+1, Jose Augusto wrote:
Hi

Loading of julia modules (0.3.0-rc1) is a slow process (compared to other languages). This is more visible when in the process of running/debugging.
Is there any way of speeding it up? 

TIA

Jose

Code:

 
t1=time()
println("start time: ",0)
println("using DataFrames... slow loading...")
using DataFrames
t2=time()
println("load time: ",t2-t1," secs")
println("using PyPlot... slow loading...")
using PyPlot
t3=time()
println("load time: ",t3-t2," secs")
println("using Winston... slow loading...")
using Winston
t4=time()
println("load time: ",t4-t3," secs")
 

Result: (julia 0.3.0-rc1, AMD A8-3850 on Windows 7, 64 bits)

C:\Users\...>julia timeLoading.jl
start time: 0.0
using DataFrames... slow loading...
load time: 12.457000017166138 secs
using PyPlot... slow loading...
INFO: Loading help data...
load time: 13.289999961853027 secs
using Winston... slow loading...
load time: 6.801000118255615 secs
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Ivar Nesje
It seems likely that we will get something like .pyc files in the future, but first we need a 0.3 release. It is on the roadmap for 0.4, but it is hard to promise working code that has not yet been written.

Ivar

kl. 16:37:03 UTC+2 torsdag 31. juli 2014 skrev Jose Augusto følgende:
Ivar, John, thank you.

I tried to compile julia with mingw a couple weeks ago (when talking with devs about openblas issues) and I didn't succeed, so I can't implement the solution you point out  (compilation from sources). I undesrtand the slowness is due to repeated JIT compilation of the module/package each time julia is called and 'using' is executed. I was looking for the existence of some middle-way solution such as .pyc files in Python.

Another quirk: the loading time is not very important if one is using Julia REPL-- it is imported only once; but when using the command line it becomes annoying because its done in each run. So I tried the following: wrapping the code for loading modules inside a function:

function mainLoad() 
    ### code for importing modules here...
end

an then in the REPL I have an error. Apparently we cannot call "using..." inside a function:

julia> include("timeLoading.jl")
mainLoad (generic function with 1 method)
julia> mainLoad()
ERROR: error compiling mainLoad: unsupported or misplaced expression using in function mainLoad
julia>

Finally, to conclude, I think the more efficient way (which I'm using now) is launching the REPL, importing only once the packages, developing everything in the script as a function and load it with include("script.jl")... at least in the developing stage...

Thanks again

Jose


On Thursday, July 31, 2014 4:45:08 AM UTC+1, Jose Augusto wrote:
Hi

Loading of julia modules (0.3.0-rc1) is a slow process (compared to other languages). This is more visible when in the process of running/debugging.
Is there any way of speeding it up? 

TIA

Jose

Code:

 
t1=time()
println("start time: ",0)
println("using DataFrames... slow loading...")
using DataFrames
t2=time()
println("load time: ",t2-t1," secs")
println("using PyPlot... slow loading...")
using PyPlot
t3=time()
println("load time: ",t3-t2," secs")
println("using Winston... slow loading...")
using Winston
t4=time()
println("load time: ",t4-t3," secs")
 

Result: (julia 0.3.0-rc1, AMD A8-3850 on Windows 7, 64 bits)

C:\Users\...>julia timeLoading.jl
start time: 0.0
using DataFrames... slow loading...
load time: 12.457000017166138 secs
using PyPlot... slow loading...
INFO: Loading help data...
load time: 13.289999961853027 secs
using Winston... slow loading...
load time: 6.801000118255615 secs
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Tony Kelman
In reply to this post by Jose Augusto
Technically you don't need a from-source build to rebuild the system image. You should be able to do this with a binary install too, and ironically it's a little easier on Windows because there are some batch files to do it for you. Rename lib/julia/sys.ji to something else to back it up, then run bin/prepare-julia-env.bat to regenerate the new system image. It should include userimg.jl that way too.

This is doable manually on Linux/Mac from a binary install too, but it involves an invocation of julia --build that I never remember the particulars of. See the Makefile rules for $(build_private_libdir)/sys0.o and $(build_private_libdir)/sys.o. Also $(build_private_libdir)/sys%$(SHLIB_EXT) for the fast-startup precompiled so/dylib version (not recommended on Windows since it causes problems with backtraces, and it relies on a MinGW installation).


On Thursday, July 31, 2014 7:37:03 AM UTC-7, Jose Augusto wrote:
Ivar, John, thank you.

I tried to compile julia with mingw a couple weeks ago (when talking with devs about openblas issues) and I didn't succeed, so I can't implement the solution you point out  (compilation from sources). I undesrtand the slowness is due to repeated JIT compilation of the module/package each time julia is called and 'using' is executed. I was looking for the existence of some middle-way solution such as .pyc files in Python.

Another quirk: the loading time is not very important if one is using Julia REPL-- it is imported only once; but when using the command line it becomes annoying because its done in each run. So I tried the following: wrapping the code for loading modules inside a function:

function mainLoad() 
    ### code for importing modules here...
end

an then in the REPL I have an error. Apparently we cannot call "using..." inside a function:

julia> include("timeLoading.jl")
mainLoad (generic function with 1 method)
julia> mainLoad()
ERROR: error compiling mainLoad: unsupported or misplaced expression using in function mainLoad
julia>

Finally, to conclude, I think the more efficient way (which I'm using now) is launching the REPL, importing only once the packages, developing everything in the script as a function and load it with include("script.jl")... at least in the developing stage...

Thanks again

Jose


On Thursday, July 31, 2014 4:45:08 AM UTC+1, Jose Augusto wrote:
Hi

Loading of julia modules (0.3.0-rc1) is a slow process (compared to other languages). This is more visible when in the process of running/debugging.
Is there any way of speeding it up? 

TIA

Jose

Code:

 
t1=time()
println("start time: ",0)
println("using DataFrames... slow loading...")
using DataFrames
t2=time()
println("load time: ",t2-t1," secs")
println("using PyPlot... slow loading...")
using PyPlot
t3=time()
println("load time: ",t3-t2," secs")
println("using Winston... slow loading...")
using Winston
t4=time()
println("load time: ",t4-t3," secs")
 

Result: (julia 0.3.0-rc1, AMD A8-3850 on Windows 7, 64 bits)

C:\Users\...>julia timeLoading.jl
start time: 0.0
using DataFrames... slow loading...
load time: 12.457000017166138 secs
using PyPlot... slow loading...
INFO: Loading help data...
load time: 13.289999961853027 secs
using Winston... slow loading...
load time: 6.801000118255615 secs
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Thanks Tony,

I'm finishing a .bat script to automate the task and that includes building also a slightly modified 'prepare-julia-env.bat'.
I'll call it 'redo-julia-env.bat'

To finish it, I need two pieces of information:

First, in the prepare-julia-env.bat' script, julia is used with a --build option which I cannot find documented.
Here is the line with the issue (the vars are the correct dirs, etc...):

cd "%JULIA_HOME%..\share\julia\base" && ^
"%JULIA%" --build "%JULIA_HOME%..\lib\julia\sys0" sysimg.jl && ^
"%JULIA%" --build "%JULIA_HOME%..\lib\julia\sys" -J sys0.ji sysimg.jl && ^
popd && pushd "%cd%" )

However, when I run Julia with that switch I get an error. (Perhaps before the 'prepare-julia-env.bat' script is run for the first time in installation, there is that option available...?)

D:\Math\Julia-0.3.0-rc1\bin>julia --version
julia version 0.3.0-rc1
D:\Math\Julia-0.3.0-rc1\bin>julia --build
ERROR: unknown option: --build
 in process_options at client.jl:288

The second issue is that I don't have the 'sys0' file also shown in that line, but since I don't know what --build does, sys0 is perhaps being built in that command... So, in the end I think the issue is lacking the --build command option;

Appreciate any help.

TIA and regards.

Jose

Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Isaiah Norton

Sys0 is the uninferred image, which needs to be built by the first invocation, then it needs to be run again with -J to cache an inferred version. --build apparently only works with arguments. What is the error from the bat script? You could try running it manually first

cd base
../julia --build ../usr/lib/julia/sys0 sysimg.jl
../julia --build ../usr/lib/julia/sys -J ../usr/lib/julia/sys0.ji

On Jul 31, 2014 9:02 PM, "Jose Augusto" <[hidden email]> wrote:
Thanks Tony,

I'm finishing a .bat script to automate the task and that includes building also a slightly modified 'prepare-julia-env.bat'.
I'll call it 'redo-julia-env.bat'

To finish it, I need two pieces of information:

First, in the prepare-julia-env.bat' script, julia is used with a --build option which I cannot find documented.
Here is the line with the issue (the vars are the correct dirs, etc...):

cd "%JULIA_HOME%..\share\julia\base" && ^
"%JULIA%" --build "%JULIA_HOME%..\lib\julia\sys0" sysimg.jl && ^
"%JULIA%" --build "%JULIA_HOME%..\lib\julia\sys" -J sys0.ji sysimg.jl && ^
popd && pushd "%cd%" )

However, when I run Julia with that switch I get an error. (Perhaps before the 'prepare-julia-env.bat' script is run for the first time in installation, there is that option available...?)

D:\Math\Julia-0.3.0-rc1\bin>julia --version
julia version 0.3.0-rc1
D:\Math\Julia-0.3.0-rc1\bin>julia --build
ERROR: unknown option: --build
 in process_options at client.jl:288

The second issue is that I don't have the 'sys0' file also shown in that line, but since I don't know what --build does, sys0 is perhaps being built in that command... So, in the end I think the issue is lacking the --build command option;

Appreciate any help.

TIA and regards.

Jose

Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Thanks Isiah,

I have seen that the errors I got resulted from not running the .bat script in julia/bin directory... After locating it there, the --build switch worked well.

But I have some more news. The (naive) .bat script I used is below. The script shall be located in julia/bin directory for correct working. 
After the script I report about the current status of rebuilding the sys.ji image with added modules...

@echo off
REM -- Script SHALL be located in julia /bin dir, near julia.exe
REM -- JULIA_DIR MUST be julia 0.3.0-rc1 base/install dir 
set JULIA_DIR=..

REM -- Define variables for the various files involved
REM -- sys files... original ... and backup copy
set sysjlori=%JULIA_DIR%\lib\julia\sys.ji  
set sysjlcopy=%JULIA_DIR%\lib\julia\sys.bak.ji  
REM -- userimg.jl shouldn't exist in the original distribution! It's created
set userimjl=%JULIA_DIR%\share\julia\base\userimg.jl
REM -- Build batch
set buildBatch=%JULIA_DIR%\bin\prepare-julia-env.bat

REM  -- Only backups the first time it runs so it really preserves...
REM      the original sys.jl from julia's binary distribution
if not exist %sysjlcopy% ( 
copy %sysjlori% %sysjlcopy% 
del %sysjlori%  
)

REM -- Echoes packages to be "compiled" for userimg.jl
REM -- The packages should already have been added to julia's system...
REM     by running 'Pkg.add()' in julia's REPL, e.g. like 'Pkg.add("Winston")' 

REM  #######  USER DEFINITIONS #######

echo Base.require("Distributions.jl") > %userimjl%
echo Base.require("Optim.jl") >> %userimjl%
echo Base.require("DataFrames.jl") >> %userimjl%
REM echo Base.require("Gadfly.jl") >> %userimjl%
echo Base.require("Winston.jl")  >> %userimjl%

REM  #######  END USER DEFINITIONS #######
 
call %buildBatch%

Notice that I've commented the inclusion of the Gadfly module. This module triggered two errors (but didn't stop the process and a sys.ji image was created)

......
profile.jl
precompile.jl
LoadError("sysimg.jl",287,LoadError("E:\\math\\Julia-0.3.0-rc1\\share\\julia\\base\\userimg.jl",4,Lo
adError("C:\\Users\\JAugusto\\.julia\\v0.3\\Gadfly\\src\\Gadfly.jl",3,LoadError("C:\\Users\\JAugusto
\\.julia\\v0.3\\Compose\\src\\Compose.jl",45,LoadError("C:\\Users\\JAugusto\\.julia\\v0.3\\Compose\\
src\\table.jl",346,could not spawn `git '--work-tree=C:\Users\JAugusto\.julia\v0.3\JuMP' '--git-dir=
C:\Users\JAugusto\.julia\v0.3\JuMP\.git' rev-parse HEAD`: no such file or directory (ENOENT))))))
exports.jl
base.jl
........

.............
graphics.jl
profile.jl
precompile.jl
Warning: New definition
    cat_aes_var!(AbstractArray{T,N},AbstractArray{U<:Union(DataType,Function),N}) at C:\Users\JAugus
to\.julia\v0.3\Gadfly\src\aesthetics.jl:262
is ambiguous with:
    cat_aes_var!(AbstractArray{T<:Union(DataType,Function),N},AbstractArray{U,N}) at C:\Users\JAugus
to\.julia\v0.3\Gadfly\src\aesthetics.jl:257.
To fix, define
    cat_aes_var!(AbstractArray{_<:Union(DataType,Function),N},AbstractArray{_<:Union(DataType,Functi
on),N})
before the new definition.
Warning: New definition
    cat_aes_var!(AbstractArray{T,N},AbstractArray{U<:Union(DataType,Function),N}) at C:\Users\JAugus
to\.julia\v0.3\Gadfly\src\aesthetics.jl:262
is ambiguous with:
    cat_aes_var!(AbstractArray{T,N},AbstractArray{T,N}) at C:\Users\JAugusto\.julia\v0.3\Gadfly\src\
aesthetics.jl:249.
To fix, define
    cat_aes_var!(AbstractArray{T,N},AbstractArray{_<:Union(DataType,Function),N})
before the new definition.

After removing Gadfly the sys.ji image was built without a glitch :-)

But the worst part was yet to come :-(
See what happens when running the new image of julia below...

First time only happens when using Winston (output slightly truncated, but errors are similar...)

C:\Users\JAugusto>julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc1 (2014-07-14 02:04 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ build
|__/                   |  x86_64-w64-mingw32

julia> using Winston
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_profile_is_running at ???:1840910295
jl_profile_is_running at ???:1840909971
jl_profile_is_running at ???:1840909971
.......................
jl_profile_is_running at ???:1840909971
jl_profile_is_running at ???:1840909223
jl_profile_is_running at ???:1840909210
jl_gc_collect at ???:1840912205
jl_gc_new_weakref at ???:1840915381
jl_array_grow_end at ???:1840861344
append! at array.jl:474
??? at ???:12959840
??? at ???:84503184
??? at ???:12960480
??? at ???:84503184
??? at ???:544320768
??? at ???:84503184
jl_match_method at ???:1840590031
??? at ???:544320768
??? at ???:12960144
_methods at reflection.jl:103
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
_methods at reflection.jl:103
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
_methods at reflection.jl:103
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.


Second time, it happens just after launching the REPL window:

C:\Users\JAugusto>julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc1 (2014-07-14 02:04 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ build
|__/                   |  x86_64-w64-mingw32

julia> Please submit a bug report with steps to reproduce this fault, and any error messages that f
llow (in their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_profile_is_running at ???:1840910295
jl_profile_is_running at ???:1840909971
jl_profile_is_running at ???:1840909971
jl_profile_is_running at ???:1840909971
.....................
jl_profile_is_running at ???:1840909223
jl_profile_is_running at ???:1840909210
jl_gc_collect at ???:1840912205
allocobj at ???:1840916005
jl_alloc_tuple at ???:1840824419
jlcall__uv_hook_alloc_buf;676 at ???:37702739
??? at ???:540770640
??? at ???:12971248
??? at ???:12970992
jl_trampoline at ???:1840617882
??? at ???:542795656
??? at ???:544546480
utf8proc_NFKC at ???:1844016070
utf8proc_NFKC at ???:1844024345
jl_apply_generic at ???:1840587164
wait_readbyte at stream.jl:308
??? at ???:102591680
??? at ???:542795656
jl_undefined_var_error at ???:1840614687
??? at ???:542795656
??? at ???:102591680
??? at ???:4
??? at ???:12971976
??? at ???:56500160
??? at ???:56500160
??? at ???:106210256
??? at ???:10
??? at ???:105263592
??? at ???:101708464
??? at ???:102591680
??? at ???:542795656
??? at ???:10
??? at ???:12972016
readuntil at stream.jl:715
jl_undefined_var_error at ???:1840614687
??? at ???:542795656
??? at ???:102591680
??? at ???:2
??? at ???:538883888
??? at ???:2
??? at ???:12972072
??? at ???:105263592
??? at ???:2
jl_pgcstack at ???:1854104592
??? at ???:12972176
readuntil at io.jl:153
??? at ???:538883936
ZdlPv at ???:1875744924
readuntil at stream.jl:715
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
readuntil at stream.jl:715
jl_undefined_var_error at ???:1840614687
??? at ???:542795656
??? at ???:102591680
??? at ???:2
??? at ???:538883888
??? at ???:2
??? at ???:12972072
??? at ???:105263592
??? at ???:2
jl_pgcstack at ???:1854104592
??? at ???:12972176
readuntil at io.jl:153
??? at ???:538883936
ZdlPv at ???:1875744924
readuntil at io.jl:153
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
readuntil at io.jl:153
??? at ???:538883936
ZdlPv at ???:1875744924
readuntil at Terminals.jl:185
??? at ???:55922416
??? at ???:543471360
??? at ???:540764448
??? at ???:12973328
??? at ???:2
??? at ???:12972384
??? at ???:106210256
??? at ???:12973328
??? at ???:12972304
jlcall_readuntil;662 at ???:37697022
??? at ???:2
??? at ???:12973328
??? at ???:2
??? at ???:101708464
??? at ???:12972448
jl_apply_generic at ???:1840587290
??? at ???:12973328
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
??? at ???:12973328
??? at ???:2
??? at ???:101708464
??? at ???:12972448
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290
Please submit a bug report with steps to reproduce this fault, and any error messages that follow (i
n their entirety). Thanks.
Exception: EXCEPTION_ACCESS_VIOLATION at 0x6dba0fd7 -- jl_profile_is_running at ???:1840910295
jl_apply_generic at ???:1840587290

So I reverted to the original julia image :-(

Hope the bug reports be helpful. Sorry for being so verbose in the crash output. BTW, this was running in a AMD Piledriver architecture (don't know if it helps).
Also I don't know if it is a particular module who is triggering the errors, or if any of the modules.
I can try to redo the thing only with one module (e.g. DataFrames).

TIA

Jose




Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Just an update...

I just run the script without extra modules (basically that is equivalent to re-running prepare-julia-env.bat) and have a working new image sys.ji.
However the size of the original image and of the current image are different (13139 KB vs. 13084). Don't know the reason, but I added several packages, so perhaps some files (dependencies) have been updated since the raw rc1 installation.

I'll be using this new image from now on and report if something strange occurs... If it proves it is OK, then perhaps the errors I reported in the previous post are due to some (or only one) of the added modules.

Regards

Jose (aka @jasax)


Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Corrected .bat script...

I was deleting manually some files while running the batch files, so if sys.ji exists in the lib/julia dir then prepare-julia-env.bat wouldn't run. The change/correction consists in moving del %sysjlori% to outside of the if block (in red below).

So use this version: 

@echo off
REM -- Script SHALL be located in julia /bin dir, near julia.exe
REM -- JULIA_DIR MUST be julia 0.3.0-rc1 base/install dir 
set JULIA_DIR=..

REM -- Define variables for the various files involved
REM -- sys files... original ... and backup copy
set sysjlori=%JULIA_DIR%\lib\julia\sys.ji  
set sysjlcopy=%JULIA_DIR%\lib\julia\sys.bak.ji  
REM -- userimg.jl shouldn't exist in the original distribution! It's created
set userimjl=%JULIA_DIR%\share\julia\base\userimg.jl
REM -- Build batch
set buildBatch=%JULIA_DIR%\bin\prepare-julia-env.bat

REM  -- Only backups the first time it runs so it really preserves...
REM      the original sys.jl from julia's binary distribution
if not exist %sysjlcopy% copy %sysjlori% %sysjlcopy% 
del %sysjlori%  

REM -- Echoes packages to be "compiled" for userimg.jl
REM -- The packages should already have been added to julia's system...
REM     by running 'Pkg.add()' in julia's REPL, e.g. like 'Pkg.add("Winston")' 

REM  #######  USER DEFINITIONS #######

REM echo Base.require("Distributions.jl") > %userimjl%
REM echo Base.require("Optim.jl") >> %userimjl%
REM echo Base.require("DataFrames.jl") >> %userimjl%
REM echo Base.require("Gadfly.jl") >> %userimjl%
REM echo Base.require("Winston.jl")  >> %userimjl%

REM  #######  END USER DEFINITIONS #######
 
call %buildBatch%
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Updating info...

Results with building sys.ji image with external module DataFrames (only), so userimg.jl has inside:

Base.require("DataFrames.jl") 

Running the prepare-julia-env.bat script ends with an error/warning?

.................
graphics.jl
profile.jl
precompile.jl
'DEFINITIONS' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the path specified.

Then using DataFrames triggers an error (but Winston, which was loaded from the exterior, worked OK):

C:\Users\JAugusto>julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc1 (2014-07-14 02:04 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ build
|__/                   |  x86_64-w64-mingw32

julia> using Winston

julia> plot([1:10],rand(10))

julia> using Dataframes
Warning: replacing module DataFrames
Warning: Method definition dump(IO,AbstractDataArray{T,1},Int64,Any) in module DataFrames at C:\User
s\JAugusto\.julia\v0.3\DataFrames\src\dataframe\dataframe.jl:710 overwritten in module DataFrames at
 C:\Users\JAugusto\.julia\v0.3\Dataframes\src\dataframe\dataframe.jl:710.
Warning: Method definition zeros(Type{T<:String},Any...) in module DataFrames at C:\Users\JAugusto\.
julia\v0.3\DataFrames\src\dataframe\dataframe.jl:845 overwritten in module DataFrames at C:\Users\JA
ugusto\.julia\v0.3\Dataframes\src\dataframe\dataframe.jl:845.
Warning: Method definition ord(Expr,) in module DataFrames at C:\Users\JAugusto\.julia\v0.3\DataFram
es\src\statsmodels\formula.jl:117 overwritten in module DataFrames at C:\Users\JAugusto\.julia\v0.3\
Dataframes\src\statsmodels\formula.jl:117.
Warning: Method definition ord(Any,) in module DataFrames at C:\Users\JAugusto\.julia\v0.3\DataFrame
s\src\statsmodels\formula.jl:118 overwritten in module DataFrames at C:\Users\JAugusto\.julia\v0.3\D
ataframes\src\statsmodels\formula.jl:118.
Warning: Method definition reorder(Function,PooledDataArray{T,R<:Integer,N},AbstractArray{T,1}...) i
n module DataFrames at C:\Users\JAugusto\.julia\v0.3\DataFrames\src\deprecated.jl:235 overwritten in
 module DataFrames at C:\Users\JAugusto\.julia\v0.3\Dataframes\src\deprecated.jl:235.
Warning: requiring "Dataframes" did not define a corresponding module.

Regards

Jose

Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Kevin Squire
Hi Jose,

I'm not sure if this will solve your problems, but be sure to keep the capitalization consistent.  "using Dataframes" should be "using DataFrames". 

Cheers, Kevin

On Friday, August 1, 2014, Jose Augusto <[hidden email]> wrote:
Updating info...

Results with building sys.ji image with external module DataFrames (only), so userimg.jl has inside:

Base.require("DataFrames.jl") 

Running the prepare-julia-env.bat script ends with an error/warning?

.................
graphics.jl
profile.jl
precompile.jl
'DEFINITIONS' is not recognized as an internal or external command,
operable program or batch file.
The system cannot find the path specified.

Then using DataFrames triggers an error (but Winston, which was loaded from the exterior, worked OK):

C:\Users\JAugusto>julia
               _
   _       _ _(_)_     |  A fresh approach to technical computing
  (_)     | (_) (_)    |  Documentation: http://docs.julialang.org
   _ _   _| |_  __ _   |  Type "help()" to list help topics
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 0.3.0-rc1 (2014-07-14 02:04 UTC)
 _/ |\__'_|_|_|\__'_|  |  Official http://julialang.org/ build
|__/                   |  x86_64-w64-mingw32

julia> using Winston

julia> plot([1:10],rand(10))

julia> using Dataframes
Warning: replacing module DataFrames
Warning: Method definition dump(IO,AbstractDataArray{T,1},Int64,Any) in module DataFrames at C:\User
s\JAugusto\.julia\v0.3\DataFrames\src\dataframe\dataframe.jl:710 overwritten in module DataFrames at
 C:\Users\JAugusto\.julia\v0.3\Dataframes\src\dataframe\dataframe.jl:710.
Warning: Method definition zeros(Type{T<:String},Any...) in module DataFrames at C:\Users\JAugusto\.
julia\v0.3\DataFrames\src\dataframe\dataframe.jl:845 overwritten in module DataFrames at C:\Users\JA
ugusto\.julia\v0.3\Dataframes\src\dataframe\dataframe.jl:845.
Warning: Method definition ord(Expr,) in module DataFrames at C:\Users\JAugusto\.julia\v0.3\DataFram
es\src\statsmodels\formula.jl:117 overwritten in module DataFrames at C:\Users\JAugusto\.julia\v0.3\
Dataframes\src\statsmodels\formula.jl:117.
Warning: Method definition ord(Any,) in module DataFrames at C:\Users\JAugusto\.julia\v0.3\DataFrame
s\src\statsmodels\formula.jl:118 overwritten in module DataFrames at C:\Users\JAugusto\.julia\v0.3\D
ataframes\src\statsmodels\formula.jl:118.
Warning: Method definition reorder(Function,PooledDataArray{T,R<:Integer,N},AbstractArray{T,1}...) i
n module DataFrames at C:\Users\JAugusto\.julia\v0.3\DataFrames\src\deprecated.jl:235 overwritten in
 module DataFrames at C:\Users\JAugusto\.julia\v0.3\Dataframes\src\deprecated.jl:235.
Warning: requiring "Dataframes" did not define a corresponding module.

Regards

Jose

Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Hi Kevin,

You're right. Thanks. I was expecting to see a failure :-( and when it happened I didn't check to see if it was my sloppiness (and indeed it was...).

I'll try to check again ASAP -- I'm in another place and in another machine right now :-(

Regards

Jose
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Joshua Job
In reply to this post by Jose Augusto
I've been trying to precompile my most used packages, but I keep getting the error "UndefVarError(var=:STDERR)". 

example error message:

error during bootstrap: LoadError(at "sysimg.jl" line 287: LoadError(at "/Users/joshuajob/julia/base/userimg.jl" line 1: LoadError(at "/Users/joshuajob/.julia/v0.3/Homebrew/src/Homebrew.jl" line 339: UndefVarError(var=:STDERR))))

Any idea what's going wrong?
On Wednesday, July 30, 2014 8:45:08 PM UTC-7, Jose Augusto wrote:
Hi

Loading of julia modules (0.3.0-rc1) is a slow process (compared to other languages). This is more visible when in the process of running/debugging.
Is there any way of speeding it up? 

TIA

Jose

Code:

 
t1=time()
println("start time: ",0)
println("using DataFrames... slow loading...")
using DataFrames
t2=time()
println("load time: ",t2-t1," secs")
println("using PyPlot... slow loading...")
using PyPlot
t3=time()
println("load time: ",t3-t2," secs")
println("using Winston... slow loading...")
using Winston
t4=time()
println("load time: ",t4-t3," secs")
 

Result: (julia 0.3.0-rc1, AMD A8-3850 on Windows 7, 64 bits)

C:\Users\...>julia timeLoading.jl
start time: 0.0
using DataFrames... slow loading...
load time: 12.457000017166138 secs
using PyPlot... slow loading...
INFO: Loading help data...
load time: 13.289999961853027 secs
using Winston... slow loading...
load time: 6.801000118255615 secs
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Ivar Nesje
Sounds to me that not all packages support precompilation. I think there are some peculiarities when building the system image, and I would not be surprised if a undefined STDOUT, was one of those.
Reply | Threaded
Open this post in threaded view
|

Re: Loading packages is very slow...

Jose Augusto
In reply to this post by Jose Augusto
Hello,

I built an image with DataFrames (only) as an extra module, and the loading of it is now much faster.
Here is the new timing of module loading (PyPlot and Winston are not in the sys image).
 
C:\Users\me>julia timeLoading.jl
start time: 0
using DataFrames... loading...
load time: 0.6729998588562012 secs
using PyPlot... loading...
INFO: Loading help data...
load time: 15.241000175476074 secs
using Winston... loading...
load time: 5.635999917984009 secs

In REPL 'using DataFrames' don't causes crashes or other odd behavior (in my previous post I screwed up with 'Dataframes' instead of 'FataFrames' and didn't notice the typo...).
I didn't do exhaustive tests to DataFrames now but a small program I'm developing that uses some of its features is working. I'll inform about any odd behavior I detect.

Meanwhile I also built an image with both DataFrames and Winston. But now 'using Winston' was accepted in the REPL (and loaded fast, of course), but when running plot(), nothing is displayed and the REPL is closed silently.    

So, in the end it seems that in Windows 7, 64 bits, some Julia modules are able to be included in the sys.ji image (DataFrames, for instance) while other aren't (Gadfly, Winston). Don't know how this relates with Linux/OSx installations... or what happens with other modules. 

Regards

Jose