equivalent of numpy newaxis?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

equivalent of numpy newaxis?

Neal Becker
Some time ago I asked this question
http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis

As a more interesting example, here is some real python code I use:
dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])

where demod_out, const.map are each vectors, mag_sqr performs element-wise
euclidean distance, and the result is a 2D array whose 1st axis matches the
1st axis of demod_out, and the 2nd axis matches the 2nd axis of const.map.


From the answers I've seen, julia doesn't really have an equivalent
functionality.  The idea here is, without allocating a new array, manipulate
the strides to cause broadcasting.

AFAICT, the best for Julia would be just forget the vectorized code, and
explicitly write out loops to perform the computation.  OK, I guess, but
maybe not as readable.

Is there any news on this front?

Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Matt Bauman
It's pretty close. In Julia 0.5, we have all the parts that are required to make this a possibility.  We have index types that specify both how many indices in the source array should be consumed (CartesianIndex{N} spans N dimensions) and types that determine what the dimensionality of the output should be (the dimensionality of the result is the sum of the dimensionalities of the indices).

That means that we now know exactly what the Julia equivalent of `np.newaxis` should be: [CartesianIndex{0}()].  This is really cool, and something I never considered before.

We're not quite there yet; views don't fully support CartesianIndices (fix in progress), and indexing by default creates a copy.

On Monday, September 12, 2016 at 2:29:15 PM UTC-5, Neal Becker wrote:
Some time ago I asked this question
<a href="http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fstackoverflow.com%2Fquestions%2F25486506%2Fjulia-broadcasting-equivalent-of-numpy-newaxis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmFR-KaZh8xfJ_rhiZvvCEqqe7MA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fstackoverflow.com%2Fquestions%2F25486506%2Fjulia-broadcasting-equivalent-of-numpy-newaxis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmFR-KaZh8xfJ_rhiZvvCEqqe7MA&#39;;return true;">http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis

As a more interesting example, here is some real python code I use:
dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])

where demod_out, const.map are each vectors, mag_sqr performs element-wise
euclidean distance, and the result is a 2D array whose 1st axis matches the
1st axis of demod_out, and the 2nd axis matches the 2nd axis of const.map.


From the answers I've seen, julia doesn't really have an equivalent
functionality.  The idea here is, without allocating a new array, manipulate
the strides to cause broadcasting.

AFAICT, the best for Julia would be just forget the vectorized code, and
explicitly write out loops to perform the computation.  OK, I guess, but
maybe not as readable.

Is there any news on this front?

Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Tim Holy
In reply to this post by Neal Becker
I'm not certain I understand what `np.newaxis` does, but doesn't `reshape` do
the same thing? (newaxis does look like a convenient way to specify shape,
though.)

Best,
--Tim

On Monday, September 12, 2016 3:28:56 PM CDT Neal Becker wrote:

> Some time ago I asked this question
> http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of
> -numpy-newaxis
>
> As a more interesting example, here is some real python code I use:
> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])
>
> where demod_out, const.map are each vectors, mag_sqr performs element-wise
> euclidean distance, and the result is a 2D array whose 1st axis matches the
> 1st axis of demod_out, and the 2nd axis matches the 2nd axis of const.map.
>
>
> From the answers I've seen, julia doesn't really have an equivalent
> functionality.  The idea here is, without allocating a new array, manipulate
> the strides to cause broadcasting.
>
> AFAICT, the best for Julia would be just forget the vectorized code, and
> explicitly write out loops to perform the computation.  OK, I guess, but
> maybe not as readable.
>
> Is there any news on this front?


Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Neal Becker
I haven't studied it, but I guess that newaxis increases the dimensionality,
while specifying 0 for the stride.  Can reshape do that?

Tim Holy wrote:

> I'm not certain I understand what `np.newaxis` does, but doesn't `reshape`
> do the same thing? (newaxis does look like a convenient way to specify
> shape, though.)
>
> Best,
> --Tim
>
> On Monday, September 12, 2016 3:28:56 PM CDT Neal Becker wrote:
>> Some time ago I asked this question
>> http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of
>> -numpy-newaxis
>>
>> As a more interesting example, here is some real python code I use:
>> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])
>>
>> where demod_out, const.map are each vectors, mag_sqr performs
>> element-wise euclidean distance, and the result is a 2D array whose 1st
>> axis matches the 1st axis of demod_out, and the 2nd axis matches the 2nd
>> axis of const.map.
>>
>>
>> From the answers I've seen, julia doesn't really have an equivalent
>> functionality.  The idea here is, without allocating a new array,
>> manipulate the strides to cause broadcasting.
>>
>> AFAICT, the best for Julia would be just forget the vectorized code, and
>> explicitly write out loops to perform the computation.  OK, I guess, but
>> maybe not as readable.
>>
>> Is there any news on this front?


Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Tim Holy

julia> a = rand(3)
3-element Array{Float64,1}:
0.47428  
0.505429
0.198919

julia> reshape(a, (3,1))
3×1 Array{Float64,2}:
0.47428  
0.505429
0.198919

julia> reshape(a, (1,3))
1×3 Array{Float64,2}:
0.47428  0.505429  0.198919

Is that what you want? (Note that for both of them, the result is 2-dimensional.)

 

--Tim

 

On Monday, September 12, 2016 6:47:04 PM CDT Neal Becker wrote:

> I haven't studied it, but I guess that newaxis increases the dimensionality,

> while specifying 0 for the stride. Can reshape do that?

>

> Tim Holy wrote:

> > I'm not certain I understand what `np.newaxis` does, but doesn't `reshape`

> > do the same thing? (newaxis does look like a convenient way to specify

> > shape, though.)

> >

> > Best,

> > --Tim

> >

> > On Monday, September 12, 2016 3:28:56 PM CDT Neal Becker wrote:

> >> Some time ago I asked this question

> >> http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent

> >> -of -numpy-newaxis

> >>

> >> As a more interesting example, here is some real python code I use:

> >> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])

> >>

> >> where demod_out, const.map are each vectors, mag_sqr performs

> >> element-wise euclidean distance, and the result is a 2D array whose 1st

> >> axis matches the 1st axis of demod_out, and the 2nd axis matches the 2nd

> >> axis of const.map.

> >>

> >>

> >> From the answers I've seen, julia doesn't really have an equivalent

> >> functionality. The idea here is, without allocating a new array,

> >> manipulate the strides to cause broadcasting.

> >>

> >> AFAICT, the best for Julia would be just forget the vectorized code, and

> >> explicitly write out loops to perform the computation. OK, I guess, but

> >> maybe not as readable.

> >>

> >> Is there any news on this front?

 

 

Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Bob Nnamtrop
I use a simple function for this:

function newdim(A::AbstractArray, d::Integer)
    @assert 0 < d <= ndims(A)+1
    dim = size(A)
    reshape(A, dim[1:d-1]..., 1, dim[d:end]...)
end
 
But having syntax for a newaxis would be great. See also:

https://github.com/JuliaLang/julia/issues/5405
https://github.com/JuliaLang/julia/issues/4774 (search for newaxis)

Bob

On Mon, Sep 12, 2016 at 5:00 PM, Tim Holy <[hidden email]> wrote:

julia> a = rand(3)
3-element Array{Float64,1}:
0.47428  
0.505429
0.198919

julia> reshape(a, (3,1))
3×1 Array{Float64,2}:
0.47428  
0.505429
0.198919

julia> reshape(a, (1,3))
1×3 Array{Float64,2}:
0.47428  0.505429  0.198919

Is that what you want? (Note that for both of them, the result is 2-dimensional.)

 

--Tim

 

On Monday, September 12, 2016 6:47:04 PM CDT Neal Becker wrote:

> I haven't studied it, but I guess that newaxis increases the dimensionality,

> while specifying 0 for the stride. Can reshape do that?

>

> Tim Holy wrote:

> > I'm not certain I understand what `np.newaxis` does, but doesn't `reshape`

> > do the same thing? (newaxis does look like a convenient way to specify

> > shape, though.)

> >

> > Best,

> > --Tim

> >

> > On Monday, September 12, 2016 3:28:56 PM CDT Neal Becker wrote:

> >> Some time ago I asked this question

> >> http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent

> >> -of -numpy-newaxis

> >>

> >> As a more interesting example, here is some real python code I use:

> >> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])

> >>

> >> where demod_out, const.map are each vectors, mag_sqr performs

> >> element-wise euclidean distance, and the result is a 2D array whose 1st

> >> axis matches the 1st axis of demod_out, and the 2nd axis matches the 2nd

> >> axis of const.map.

> >>

> >>

> >> From the answers I've seen, julia doesn't really have an equivalent

> >> functionality. The idea here is, without allocating a new array,

> >> manipulate the strides to cause broadcasting.

> >>

> >> AFAICT, the best for Julia would be just forget the vectorized code, and

> >> explicitly write out loops to perform the computation. OK, I guess, but

> >> maybe not as readable.

> >>

> >> Is there any news on this front?

 

 


DNF
Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

DNF
In reply to this post by Neal Becker
For your particular example, it looks like what you want is (and I am just guessing what mag_sqr means):
dist = abs2.(x .- y.')
The performance should be the similar to a hand-written loop on version 0.5.

You can read about it here: http://docs.julialang.org/en/release-0.5/manual/functions/#dot-syntax-for-vectorizing-functions


On Monday, September 12, 2016 at 9:29:15 PM UTC+2, Neal Becker wrote:
Some time ago I asked this question
<a href="http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fstackoverflow.com%2Fquestions%2F25486506%2Fjulia-broadcasting-equivalent-of-numpy-newaxis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmFR-KaZh8xfJ_rhiZvvCEqqe7MA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fstackoverflow.com%2Fquestions%2F25486506%2Fjulia-broadcasting-equivalent-of-numpy-newaxis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmFR-KaZh8xfJ_rhiZvvCEqqe7MA&#39;;return true;">http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis

As a more interesting example, here is some real python code I use:
dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])

where demod_out, const.map are each vectors, mag_sqr performs element-wise
euclidean distance, and the result is a 2D array whose 1st axis matches the
1st axis of demod_out, and the 2nd axis matches the 2nd axis of const.map.


From the answers I've seen, julia doesn't really have an equivalent
functionality.  The idea here is, without allocating a new array, manipulate
the strides to cause broadcasting.

AFAICT, the best for Julia would be just forget the vectorized code, and
explicitly write out loops to perform the computation.  OK, I guess, but
maybe not as readable.

Is there any news on this front?

Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Neal Becker
So you're saying that abs2.(x .- y.') will not allocate a 2d array and then
pass to abs2?  That's great!  But how would I know that?

DNF wrote:

> For your particular example, it looks like what you want is (and I am just
> guessing what mag_sqr means):
> dist = abs2.(x .- y.')
> The performance should be the similar to a hand-written loop on version
> 0.5.
>
> You can read about it here:
> http://docs.julialang.org/en/release-0.5/manual/functions/#dot-syntax-for-vectorizing-functions
>
>
> On Monday, September 12, 2016 at 9:29:15 PM UTC+2, Neal Becker wrote:
>>
>> Some time ago I asked this question
>>
>> http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis
>>
>> As a more interesting example, here is some real python code I use:
>> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])
>>
>> where demod_out, const.map are each vectors, mag_sqr performs
>> element-wise euclidean distance, and the result is a 2D array whose 1st
>> axis matches the
>> 1st axis of demod_out, and the 2nd axis matches the 2nd axis of
>> const.map.
>>
>>
>> From the answers I've seen, julia doesn't really have an equivalent
>> functionality.  The idea here is, without allocating a new array,
>> manipulate
>> the strides to cause broadcasting.
>>
>> AFAICT, the best for Julia would be just forget the vectorized code, and
>> explicitly write out loops to perform the computation.  OK, I guess, but
>> maybe not as readable.
>>
>> Is there any news on this front?
>>
>>


Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

Mauro
On Tue, 2016-09-13 at 14:26, Neal Becker <[hidden email]> wrote:
> So you're saying that abs2.(x .- y.') will not allocate a 2d array and then
> pass to abs2?  That's great!  But how would I know that?

The operators do not do the fusing yet, check right at the bottom of the
linked manual section.  I think you can work around it by using their
functional form:

x .+ y # not fused
(+).(x,y) # fused

So:

out .= abs2.((-).(x, y.'))

> DNF wrote:
>
>> For your particular example, it looks like what you want is (and I am just
>> guessing what mag_sqr means):
>> dist = abs2.(x .- y.')
>> The performance should be the similar to a hand-written loop on version
>> 0.5.
>>
>> You can read about it here:
>> http://docs.julialang.org/en/release-0.5/manual/functions/#dot-syntax-for-vectorizing-functions
>>
>>
>> On Monday, September 12, 2016 at 9:29:15 PM UTC+2, Neal Becker wrote:
>>>
>>> Some time ago I asked this question
>>>
>>> http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis
>>>
>>> As a more interesting example, here is some real python code I use:
>>> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])
>>>
>>> where demod_out, const.map are each vectors, mag_sqr performs
>>> element-wise euclidean distance, and the result is a 2D array whose 1st
>>> axis matches the
>>> 1st axis of demod_out, and the 2nd axis matches the 2nd axis of
>>> const.map.
>>>
>>>
>>> From the answers I've seen, julia doesn't really have an equivalent
>>> functionality.  The idea here is, without allocating a new array,
>>> manipulate
>>> the strides to cause broadcasting.
>>>
>>> AFAICT, the best for Julia would be just forget the vectorized code, and
>>> explicitly write out loops to perform the computation.  OK, I guess, but
>>> maybe not as readable.
>>>
>>> Is there any news on this front?
>>>
>>>
DNF
Reply | Threaded
Open this post in threaded view
|

Re: equivalent of numpy newaxis?

DNF
Oh, yeah. I forgot that .+, .-, etc are not along for the ride yet. I understand it is pretty much ready for inclusion in 0.6.

There is also a one extra allocation, namely y.', which would not be necessary in a loop. But this is hardly worse than numpy, right?


On Tuesday, September 13, 2016 at 2:42:35 PM UTC+2, Mauro wrote:
On Tue, 2016-09-13 at 14:26, Neal Becker <<a href="javascript:" target="_blank" gdf-obfuscated-mailto="dyESCWf5AQAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">ndbe...@...> wrote:
> So you're saying that abs2.(x .- y.') will not allocate a 2d array and then
> pass to abs2?  That's great!  But how would I know that?

The operators do not do the fusing yet, check right at the bottom of the
linked manual section.  I think you can work around it by using their
functional form:

x .+ y # not fused
(+).(x,y) # fused

So:

out .= abs2.((-).(x, y.'))

> DNF wrote:
>
>> For your particular example, it looks like what you want is (and I am just
>> guessing what mag_sqr means):
>> dist = abs2.(x .- y.')
>> The performance should be the similar to a hand-written loop on version
>> 0.5.
>>
>> You can read about it here:
>> <a href="http://docs.julialang.org/en/release-0.5/manual/functions/#dot-syntax-for-vectorizing-functions" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdocs.julialang.org%2Fen%2Frelease-0.5%2Fmanual%2Ffunctions%2F%23dot-syntax-for-vectorizing-functions\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJPrAuqnWcEtB_tcn2b4A3wUspgg&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fdocs.julialang.org%2Fen%2Frelease-0.5%2Fmanual%2Ffunctions%2F%23dot-syntax-for-vectorizing-functions\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHJPrAuqnWcEtB_tcn2b4A3wUspgg&#39;;return true;">http://docs.julialang.org/en/release-0.5/manual/functions/#dot-syntax-for-vectorizing-functions
>>
>>
>> On Monday, September 12, 2016 at 9:29:15 PM UTC+2, Neal Becker wrote:
>>>
>>> Some time ago I asked this question
>>>
>>> <a href="http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fstackoverflow.com%2Fquestions%2F25486506%2Fjulia-broadcasting-equivalent-of-numpy-newaxis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmFR-KaZh8xfJ_rhiZvvCEqqe7MA&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2Fstackoverflow.com%2Fquestions%2F25486506%2Fjulia-broadcasting-equivalent-of-numpy-newaxis\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNHmFR-KaZh8xfJ_rhiZvvCEqqe7MA&#39;;return true;">http://stackoverflow.com/questions/25486506/julia-broadcasting-equivalent-of-numpy-newaxis
>>>
>>> As a more interesting example, here is some real python code I use:
>>> dist = mag_sqr (demod_out[:,np.newaxis] - const.map[np.newaxis,:])
>>>
>>> where demod_out, const.map are each vectors, mag_sqr performs
>>> element-wise euclidean distance, and the result is a 2D array whose 1st
>>> axis matches the
>>> 1st axis of demod_out, and the 2nd axis matches the 2nd axis of
>>> const.map.
>>>
>>>
>>> From the answers I've seen, julia doesn't really have an equivalent
>>> functionality.  The idea here is, without allocating a new array,
>>> manipulate
>>> the strides to cause broadcasting.
>>>
>>> AFAICT, the best for Julia would be just forget the vectorized code, and
>>> explicitly write out loops to perform the computation.  OK, I guess, but
>>> maybe not as readable.
>>>
>>> Is there any news on this front?
>>>
>>>