push! function and multidimensional arrays

[john pollack]

Hi. I want to create a 2-column array.It should be empty at first, then I should be able to add rows to it. I was able to do this by : 

A = Array(Float64,(0,2)) 

A = vcat(A , [ 1 2 ] ) 

However, I couldn't use push! function for this. Is it possible to use push! function for this aim, and are there any substantial performance differences between push! and vcat functions  ?

Thanks for any help.

[Ivar Nesje]

You can't use push! on multidimentional arrays. You can push onto a 1d array, and then reshape. Another way is to collect all the row/column vectors and use hcat(c...) or vcat(c...) to create an array.

Ivar

kl. 21:22:01 UTC+2 onsdag 23. juli 2014 skrev john pollack følgende:

Hi. I want to create a 2-column array.It should be empty at first, then I should be able to add rows to it. I was able to do this by : 

A = Array(Float64,(0,2)) 

A = vcat(A , [ 1 2 ] ) 

However, I couldn't use push! function for this. Is it possible to use push! function for this aim, and are there any substantial performance differences between push! and vcat functions  ?

Thanks for any help.

[Kevin Squire]

Some additional comments:

1. vcat will always create a new array, and copies the contents of the arrays to be concatenated into the new one.
   
2. Julia stores arrays in column-major order, so vcat(A, [1 2]) does the following
   
• copy column 1 of A to temp array
  
• copy the "1" from [1 2] to temp
  
• copy column 2 of A to temp
  
• copy the "2" from [1 2] to temp

This is rather inefficent.  In contrast, hcat(A, [1,2])

• copies all of A to temp
• copies [1,2] to temp

The first method Ivar recommended would be to do the following:

a = Float64[]
for i = 1:100
    push!(a, rand(2)...)
end

A = reshape(a, (2, div(length(a, 2)))

Cheers,

   Kevin

On Wed, Jul 23, 2014 at 12:33 PM, Ivar Nesje <[hidden email]> wrote:

You can't use push! on multidimentional arrays. You can push onto a 1d array, and then reshape. Another way is to collect all the row/column vectors and use hcat(c...) or vcat(c...) to create an array.

Ivar

kl. 21:22:01 UTC+2 onsdag 23. juli 2014 skrev john pollack følgende:

Hi. I want to create a 2-column array.It should be empty at first, then I should be able to add rows to it. I was able to do this by : 

A = Array(Float64,(0,2)) 

A = vcat(A , [ 1 2 ] ) 

However, I couldn't use push! function for this. Is it possible to use push! function for this aim, and are there any substantial performance differences between push! and vcat functions  ?

Thanks for any help.

[Alireza Nejati]

John, just to give some explanation: push! is there as an efficient push operation - one that ideally takes O(1) time because it simply extends the vector in-place rather than copying everything to a new vector. (In practice, though, it takes slightly longer than O(1) time because of the overhead of occasionally allocating a new array). Julia stores arrays in column-major order, so you can push a new column on to a matrix by pushing the column to a 1d array and then reshaping, as Ivar said. But you can't do the same with rows, because there is no way to append a new row to a matrix in an in-place fashion. You have to shift all the array elements around in memory.

The suggestions above are both good, and another way would be to simply create row matrix by appending columns instead. At the end, just transpose the matrix. The transpose operation does add O(n) overhead but depending on what you're doing a single transpose at the end could be much more efficient than cat'ing at each iteration.



On Thursday, July 24, 2014 7:22:01 AM UTC+12, john pollack wrote:

Hi. I want to create a 2-column array.It should be empty at first, then I should be able to add rows to it. I was able to do this by : 

A = Array(Float64,(0,2)) 

A = vcat(A , [ 1 2 ] ) 

However, I couldn't use push! function for this. Is it possible to use push! function for this aim, and are there any substantial performance differences between push! and vcat functions  ?

Thanks for any help.

[Stefan Karpinski-2]

This explanation should maybe go straight in our FAQ :-)

On Jul 23, 2014, at 3:12 PM, Alireza Nejati <[hidden email]> wrote:

John, just to give some explanation: push! is there as an efficient push operation - one that ideally takes O(1) time because it simply extends the vector in-place rather than copying everything to a new vector. (In practice, though, it takes slightly longer than O(1) time because of the overhead of occasionally allocating a new array). Julia stores arrays in column-major order, so you can push a new column on to a matrix by pushing the column to a 1d array and then reshaping, as Ivar said. But you can't do the same with rows, because there is no way to append a new row to a matrix in an in-place fashion. You have to shift all the array elements around in memory.

The suggestions above are both good, and another way would be to simply create row matrix by appending columns instead. At the end, just transpose the matrix. The transpose operation does add O(n) overhead but depending on what you're doing a single transpose at the end could be much more efficient than cat'ing at each iteration.



On Thursday, July 24, 2014 7:22:01 AM UTC+12, john pollack wrote:

Hi. I want to create a 2-column array.It should be empty at first, then I should be able to add rows to it. I was able to do this by : 

A = Array(Float64,(0,2)) 

A = vcat(A , [ 1 2 ] ) 

However, I couldn't use push! function for this. Is it possible to use push! function for this aim, and are there any substantial performance differences between push! and vcat functions  ?

Thanks for any help.