NPM style app/script distribution and installation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

NPM style app/script distribution and installation

Ben Ward
Hi,

I'd just like to draw anyone who is interested to this BioJulia issue: https://github.com/BioJulia/Bio.jl/issues/267

Essentially we are wondering if apps could be distributed using julia and Pkg, by having packages include bin scripts that could be symlinked into the path (much like NPM does https://docs.npmjs.com/files/package.json#bin) during install of the package.

At the moment the suggestion is perhaps to just use deps/build.jl to do this. But a specific hook in the Pkg install process could also be cool.

Advice or suggestions welcome here or the issue, I'm not 100% familiar with the Pkg internals, I have only read the Pkg.update function source.

Many Thanks,
Ben.
Reply | Threaded
Open this post in threaded view
|

Re: NPM style app/script distribution and installation

Stefan Karpinski
Seems like a plausible thing to do, although we'd have to have a discussion about the security implications. I'll keep it in mind for the Pkg3 revamp of the package manager.

On Sat, Aug 20, 2016 at 6:30 AM, Ben Ward <[hidden email]> wrote:
Hi,

I'd just like to draw anyone who is interested to this BioJulia issue: https://github.com/BioJulia/Bio.jl/issues/267

Essentially we are wondering if apps could be distributed using julia and Pkg, by having packages include bin scripts that could be symlinked into the path (much like NPM does https://docs.npmjs.com/files/package.json#bin) during install of the package.

At the moment the suggestion is perhaps to just use deps/build.jl to do this. But a specific hook in the Pkg install process could also be cool.

Advice or suggestions welcome here or the issue, I'm not 100% familiar with the Pkg internals, I have only read the Pkg.update function source.

Many Thanks,
Ben.

Reply | Threaded
Open this post in threaded view
|

Re: NPM style app/script distribution and installation

Ben Ward
Thanks Stefan,

From our brief discussion in the issue, we considered maybe there would be some hook that gets run during Pkg.add and Pkg.rm, bit how the build.jl script of each package gets run during install.
In fact you could use a build.jl script to achieve the kind of effect:
import Glob
for script in Glob.glob("bin/*.jl")
   
symlink(script, joinpath("/usr/local/bin", splitdir(script)[2]))
end

But of course without a corresponding Pkg.rm operation to undo that, you'll have broke links all up in your /usr/local/bin :/

I tried to read the Pkg. source today to see if I could see where in the add process for example the package repository is downloaded, and build.jl run and so on.
I got as far as the REQUIRE stuff getting resolved and edited by Entry.add, which executes Entry.edit, but then Entry.edit calls resolve, and there's graph stuff and at that point I got lost so couldn't really determine where in the entire process something like a symlinking of "app scripts" in a package would occur (somewhere just after the package is downloaded, and build.jl and so on is run for the package is my guess where this fits). 

I know a lot of people I work with, and bioinformaticians in general would really like this: we very often create libraries which can be loaded in scripts and other future projects, but we also need that code quickly runnable in day to day pipelines and jobs, so often have small apps or scripts very thinly wrapping some of the functions in those libraries or packages.


On Saturday, August 20, 2016 at 4:49:08 PM UTC+1, Stefan Karpinski wrote:
Seems like a plausible thing to do, although we'd have to have a discussion about the security implications. I'll keep it in mind for the Pkg3 revamp of the package manager.

On Sat, Aug 20, 2016 at 6:30 AM, Ben Ward <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="WBfeTUnVBAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">axolotl...@...> wrote:
Hi,

I'd just like to draw anyone who is interested to this BioJulia issue: <a href="https://github.com/BioJulia/Bio.jl/issues/267" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FBioJulia%2FBio.jl%2Fissues%2F267\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoRyNqWR8xV0OyMQEofcKcMCbsEg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgithub.com%2FBioJulia%2FBio.jl%2Fissues%2F267\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGoRyNqWR8xV0OyMQEofcKcMCbsEg&#39;;return true;">https://github.com/BioJulia/Bio.jl/issues/267

Essentially we are wondering if apps could be distributed using julia and Pkg, by having packages include bin scripts that could be symlinked into the path (much like NPM does <a href="https://docs.npmjs.com/files/package.json#bin" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.npmjs.com%2Ffiles%2Fpackage.json%23bin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlZw2DyoWKR_MsPEDG-Nm6VDMitg&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.npmjs.com%2Ffiles%2Fpackage.json%23bin\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHlZw2DyoWKR_MsPEDG-Nm6VDMitg&#39;;return true;">https://docs.npmjs.com/files/package.json#bin) during install of the package.

At the moment the suggestion is perhaps to just use deps/build.jl to do this. But a specific hook in the Pkg install process could also be cool.

Advice or suggestions welcome here or the issue, I'm not 100% familiar with the Pkg internals, I have only read the Pkg.update function source.

Many Thanks,
Ben.

Reply | Threaded
Open this post in threaded view
|

Re: NPM style app/script distribution and installation

Chris Foster
It might also be worth looking at the python "entry points" mechanism
as a source of inspiration. Here's a summary as I understand it:

The idea is to associate a script name (which will go in the path)
with a function inside your module.  At install time, the python setup
system then generates a tiny thunk script with the given name and puts
it in the install path for the python environment. The thunk deals
with using the correct python interpreter path to import the correct
module and call the desired function.

This can be nice as any platform-specific annoyances regarding setting
up paths and other environment details can go into the autogenerated
script and the package author doesn't need to deal with them. It also
allows for a layer of renaming to abstract the details of what is
considered callable from the shell on different systems.  Eg, a batch
script could be generated on windows (I don't know if this is what
actually happens), while on linux you'd want the executable flag set
and avoid a filename extension. It also avoids the need for a script
per entry point which might be a good or bad thing depending on your
point of view.

Downsides as I see them: 1) the install directory can't be trivially
copied to a different path - this can be bad if you want to package an
entire environment 2) Calling the entry points during package
development is different from after they've been installed, and this
is annoying.

Is there any documentation or code for the proposed Pkg3 design other
than the juliacon talk?


On Sun, Aug 21, 2016 at 3:10 AM, Ben Ward <[hidden email]> wrote:

> Thanks Stefan,
>
> From our brief discussion in the issue, we considered maybe there would be
> some hook that gets run during Pkg.add and Pkg.rm, bit how the build.jl
> script of each package gets run during install.
> In fact you could use a build.jl script to achieve the kind of effect:
>
> import Glob
> for script in Glob.glob("bin/*.jl")
>     symlink(script, joinpath("/usr/local/bin", splitdir(script)[2]))
> end
>
>
> But of course without a corresponding Pkg.rm operation to undo that, you'll
> have broke links all up in your /usr/local/bin :/
>
> I tried to read the Pkg. source today to see if I could see where in the add
> process for example the package repository is downloaded, and build.jl run
> and so on.
> I got as far as the REQUIRE stuff getting resolved and edited by Entry.add,
> which executes Entry.edit, but then Entry.edit calls resolve, and there's
> graph stuff and at that point I got lost so couldn't really determine where
> in the entire process something like a symlinking of "app scripts" in a
> package would occur (somewhere just after the package is downloaded, and
> build.jl and so on is run for the package is my guess where this fits).
>
> I know a lot of people I work with, and bioinformaticians in general would
> really like this: we very often create libraries which can be loaded in
> scripts and other future projects, but we also need that code quickly
> runnable in day to day pipelines and jobs, so often have small apps or
> scripts very thinly wrapping some of the functions in those libraries or
> packages.
>
>
> On Saturday, August 20, 2016 at 4:49:08 PM UTC+1, Stefan Karpinski wrote:
>>
>> Seems like a plausible thing to do, although we'd have to have a
>> discussion about the security implications. I'll keep it in mind for the
>> Pkg3 revamp of the package manager.
>>
>> On Sat, Aug 20, 2016 at 6:30 AM, Ben Ward <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I'd just like to draw anyone who is interested to this BioJulia issue:
>>> https://github.com/BioJulia/Bio.jl/issues/267
>>>
>>> Essentially we are wondering if apps could be distributed using julia and
>>> Pkg, by having packages include bin scripts that could be symlinked into the
>>> path (much like NPM does https://docs.npmjs.com/files/package.json#bin)
>>> during install of the package.
>>>
>>> At the moment the suggestion is perhaps to just use deps/build.jl to do
>>> this. But a specific hook in the Pkg install process could also be cool.
>>>
>>> Advice or suggestions welcome here or the issue, I'm not 100% familiar
>>> with the Pkg internals, I have only read the Pkg.update function source.
>>>
>>> Many Thanks,
>>> Ben.
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: NPM style app/script distribution and installation

Stefan Karpinski
On Sat, Aug 20, 2016 at 9:20 PM, Chris Foster <[hidden email]> wrote:
Is there any documentation or code for the proposed Pkg3 design other
than the juliacon talk?

 Not yet but there will be a Julep write up in a bit, at which point we'll have a public discussion.
Reply | Threaded
Open this post in threaded view
|

Re: NPM style app/script distribution and installation

Chris Foster
On Mon, Aug 22, 2016 at 5:24 AM, Stefan Karpinski <[hidden email]> wrote:
> On Sat, Aug 20, 2016 at 9:20 PM, Chris Foster <[hidden email]> wrote:
>>
>> Is there any documentation or code for the proposed Pkg3 design other
>> than the juliacon talk?
>
>  Not yet but there will be a Julep write up in a bit, at which point we'll
> have a public discussion.

Great, I'm looking forward to this.  At the moment we use a hacked up
combination
of normal Pkg, DeclarativePackages.jl and custom shell scripts for
managing projects
and packaging up a julia environment for moving to a different
machine.  This works ok
for pure julia code, but binary dependencies are a mess, especially
when the system
package manager is involved.