# Multinomial Logistic Regression

13 messages
Open this post in threaded view
|

## Multinomial Logistic Regression

 Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic <bdeo...@...> wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 I got the following error:julia> model = ScikitLearn.fit!(LogisticRegression(multi_class="multinomial"), X, y)ERROR: PyError (:PyObject_Call) ValueError('Solver liblinear does not support a multinomial backend.',)  File "/home/benjamin/bin/miniconda/lib/python2.7/site-packages/sklearn/linear_model/logistic.py", line 1148, in fit    self.dual, sample_weight)  File "/home/benjamin/bin/miniconda/lib/python2.7/site-packages/sklearn/linear_model/logistic.py", line 412, in _check_solver_option    "a multinomial backend." % solver) [inlined code] from /home/benjamin/.julia/v0.4/PyCall/src/exception.jl:81 in pycall at /home/benjamin/.julia/v0.4/PyCall/src/PyCall.jl:344 in call at /home/benjamin/.julia/v0.4/PyCall/src/PyCall.jl:372 in fit! at /home/benjamin/.julia/v0.4/ScikitLearn/src/Skcore.jl:75On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 I figured out I just need to change the solver from "liblinear" to "lbfgs"On Wednesday, April 6, 2016 at 4:30:27 PM UTC-5, Benjamin Deonovic wrote:I got the following error:julia> model = ScikitLearn.fit!(LogisticRegression(multi_class="multinomial"), X, y)ERROR: PyError (:PyObject_Call) ValueError('Solver liblinear does not support a multinomial backend.',)  File "/home/benjamin/bin/miniconda/lib/python2.7/site-packages/sklearn/linear_model/logistic.py", line 1148, in fit    self.dual, sample_weight)  File "/home/benjamin/bin/miniconda/lib/python2.7/site-packages/sklearn/linear_model/logistic.py", line 412, in _check_solver_option    "a multinomial backend." % solver) [inlined code] from /home/benjamin/.julia/v0.4/PyCall/src/exception.jl:81 in pycall at /home/benjamin/.julia/v0.4/PyCall/src/PyCall.jl:344 in call at /home/benjamin/.julia/v0.4/PyCall/src/PyCall.jl:372 in fit! at /home/benjamin/.julia/v0.4/ScikitLearn/src/Skcore.jl:75On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 In reply to this post by Cedric St-Jean-2 How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 Great thanks! I'm having trouble reproducing some R results for multinomial regression. Could you take a look if I am doing things right with the SciKitLearn package? http://stackoverflow.com/questions/36463072/julia-multinomial-regression-with-time-series-lagged-valuesOn Wednesday, April 6, 2016 at 5:06:09 PM UTC-5, Cedric St-Jean wrote:All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 The ScikitLearn.jl call looks OK. Some ideas:- scikit-learn's regression is regularized by default. You can disable it with LogisticRegression(C=1.e9)- you convert y to Vector{Int64}, might want to try Vector{Float64}- isn't multinomial logistic regression non-identifiable under certain parametrizations? I would try computing the output (i.e. predict(model, X)) in both models and see if they differOtherwise, trying a third package might be useful in figuring out which one is wrong.CédricOn Wednesday, April 6, 2016 at 6:28:57 PM UTC-4, Benjamin Deonovic wrote:Great thanks! I'm having trouble reproducing some R results for multinomial regression. Could you take a look if I am doing things right with the SciKitLearn package? http://stackoverflow.com/questions/36463072/julia-multinomial-regression-with-time-series-lagged-valuesOn Wednesday, April 6, 2016 at 5:06:09 PM UTC-5, Cedric St-Jean wrote:All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 Note that not all of the $\beta_k$ vectors of coefficients are uniquely identifiable. This is due to the fact that all probabilities must sum to 1, making one of them completely determined once all the rest are known. As a result there are only $k-1$ separately specifiable probabilities, and hence $k-1$ separately identifiable vectors of coefficients. One way to see this is to note that if we add a constant vector to all of the coefficient vectors, the equations are identical:From Wikipedia. So it's expected that the coefficients are different, even if the predictions are the same. What do you want to use the coefficients for?On Wednesday, April 6, 2016 at 7:09:09 PM UTC-4, Cedric St-Jean wrote:The ScikitLearn.jl call looks OK. Some ideas:- scikit-learn's regression is regularized by default. You can disable it with LogisticRegression(C=1.e9)- you convert y to Vector{Int64}, might want to try Vector{Float64}- isn't multinomial logistic regression non-identifiable under certain parametrizations? I would try computing the output (i.e. predict(model, X)) in both models and see if they differOtherwise, trying a third package might be useful in figuring out which one is wrong.CédricOn Wednesday, April 6, 2016 at 6:28:57 PM UTC-4, Benjamin Deonovic wrote:Great thanks! I'm having trouble reproducing some R results for multinomial regression. Could you take a look if I am doing things right with the SciKitLearn package? http://stackoverflow.com/questions/36463072/julia-multinomial-regression-with-time-series-lagged-valuesOn Wednesday, April 6, 2016 at 5:06:09 PM UTC-5, Cedric St-Jean wrote:All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
Open this post in threaded view
|

## Re: Multinomial Logistic Regression

 Thanks, I don't need the coefficients but the estimated probabilityes, (in wikipedia's notation) $P(Y_i = k) = exp(X\beta_k)/(1 + \sum_u exp(X\beta_u))$. These should be identifiable, correct? On Wednesday, April 6, 2016 at 6:19:38 PM UTC-5, Cedric St-Jean wrote:Note that not all of the  vectors of coefficients are uniquely identifiable. This is due to the fact that all probabilities must sum to 1, making one of them completely determined once all the rest are known. As a result there are only  separately specifiable probabilities, and hence  separately identifiable vectors of coefficients. One way to see this is to note that if we add a constant vector to all of the coefficient vectors, the equations are identical:From Wikipedia. So it's expected that the coefficients are different, even if the predictions are the same. What do you want to use the coefficients for?On Wednesday, April 6, 2016 at 7:09:09 PM UTC-4, Cedric St-Jean wrote:The ScikitLearn.jl call looks OK. Some ideas:- scikit-learn's regression is regularized by default. You can disable it with LogisticRegression(C=1.e9)- you convert y to Vector{Int64}, might want to try Vector{Float64}- isn't multinomial logistic regression non-identifiable under certain parametrizations? I would try computing the output (i.e. predict(model, X)) in both models and see if they differOtherwise, trying a third package might be useful in figuring out which one is wrong.CédricOn Wednesday, April 6, 2016 at 6:28:57 PM UTC-4, Benjamin Deonovic wrote:Great thanks! I'm having trouble reproducing some R results for multinomial regression. Could you take a look if I am doing things right with the SciKitLearn package? http://stackoverflow.com/questions/36463072/julia-multinomial-regression-with-time-series-lagged-valuesOn Wednesday, April 6, 2016 at 5:06:09 PM UTC-5, Cedric St-Jean wrote:All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
 Yeah. Use predict_proba(model, X) to get them.On Thu, Apr 7, 2016 at 3:00 PM, Benjamin Deonovic wrote:Thanks, I don't need the coefficients but the estimated probabilityes, (in wikipedia's notation) $P(Y_i = k) = exp(X\beta_k)/(1 + \sum_u exp(X\beta_u))$. These should be identifiable, correct? On Wednesday, April 6, 2016 at 6:19:38 PM UTC-5, Cedric St-Jean wrote:Note that not all of the  vectors of coefficients are uniquely identifiable. This is due to the fact that all probabilities must sum to 1, making one of them completely determined once all the rest are known. As a result there are only  separately specifiable probabilities, and hence  separately identifiable vectors of coefficients. One way to see this is to note that if we add a constant vector to all of the coefficient vectors, the equations are identical:From Wikipedia. So it's expected that the coefficients are different, even if the predictions are the same. What do you want to use the coefficients for?On Wednesday, April 6, 2016 at 7:09:09 PM UTC-4, Cedric St-Jean wrote:The ScikitLearn.jl call looks OK. Some ideas:- scikit-learn's regression is regularized by default. You can disable it with LogisticRegression(C=1.e9)- you convert y to Vector{Int64}, might want to try Vector{Float64}- isn't multinomial logistic regression non-identifiable under certain parametrizations? I would try computing the output (i.e. predict(model, X)) in both models and see if they differOtherwise, trying a third package might be useful in figuring out which one is wrong.CédricOn Wednesday, April 6, 2016 at 6:28:57 PM UTC-4, Benjamin Deonovic wrote:Great thanks! I'm having trouble reproducing some R results for multinomial regression. Could you take a look if I am doing things right with the SciKitLearn package? http://stackoverflow.com/questions/36463072/julia-multinomial-regression-with-time-series-lagged-valuesOn Wednesday, April 6, 2016 at 5:06:09 PM UTC-5, Cedric St-Jean wrote:All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups "julia-stats" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/julia-stats/2pHziogT9LI/unsubscribe. To unsubscribe from this group and all its topics, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.
 Thanks Cedric, you've been a great help. It looks like all of the predicted probabilities are the same across all the different implementations (R, Regression.jl, and ScikitLearn) thanks for taking your time to help me with this. On Thursday, April 7, 2016 at 2:06:24 PM UTC-5, Cedric St-Jean wrote:Yeah. Use predict_proba(model, X) to get them.On Thu, Apr 7, 2016 at 3:00 PM, Benjamin Deonovic <bdeo...@...> wrote:Thanks, I don't need the coefficients but the estimated probabilityes, (in wikipedia's notation) $P(Y_i = k) = exp(X\beta_k)/(1 + \sum_u exp(X\beta_u))$. These should be identifiable, correct? On Wednesday, April 6, 2016 at 6:19:38 PM UTC-5, Cedric St-Jean wrote:Note that not all of the  vectors of coefficients are uniquely identifiable. This is due to the fact that all probabilities must sum to 1, making one of them completely determined once all the rest are known. As a result there are only  separately specifiable probabilities, and hence  separately identifiable vectors of coefficients. One way to see this is to note that if we add a constant vector to all of the coefficient vectors, the equations are identical:From Wikipedia. So it's expected that the coefficients are different, even if the predictions are the same. What do you want to use the coefficients for?On Wednesday, April 6, 2016 at 7:09:09 PM UTC-4, Cedric St-Jean wrote:The ScikitLearn.jl call looks OK. Some ideas:- scikit-learn's regression is regularized by default. You can disable it with LogisticRegression(C=1.e9)- you convert y to Vector{Int64}, might want to try Vector{Float64}- isn't multinomial logistic regression non-identifiable under certain parametrizations? I would try computing the output (i.e. predict(model, X)) in both models and see if they differOtherwise, trying a third package might be useful in figuring out which one is wrong.CédricOn Wednesday, April 6, 2016 at 6:28:57 PM UTC-4, Benjamin Deonovic wrote:Great thanks! I'm having trouble reproducing some R results for multinomial regression. Could you take a look if I am doing things right with the SciKitLearn package? http://stackoverflow.com/questions/36463072/julia-multinomial-regression-with-time-series-lagged-valuesOn Wednesday, April 6, 2016 at 5:06:09 PM UTC-5, Cedric St-Jean wrote:All Python attributes are accessible with [:attribute_name]. model[:coef_] and model[:intercept_] should work.On Wednesday, April 6, 2016 at 5:40:31 PM UTC-4, Benjamin Deonovic wrote:How do I extract the coefficient values after I have fit the model? On Sunday, March 27, 2016 at 12:34:13 PM UTC-5, Cedric St-Jean wrote:Hi Benjamin,You can use the scikit-learn version via ScikitLearn.jl if you don't mind the (moderate) cost of sending your data to Python.using ScikitLearn@sk_import linear_model: LogisticRegressionmodel = fit!(LogisticRegression(multi_class="multinomial"), X_train, y_train)predict(model, X_test)It was released recently, don't hesitate to file an issue if you get a problem with it.CédricOn Friday, March 25, 2016 at 12:58:43 PM UTC-4, Andreas Noack wrote:I think that Dahua doesn't have as much time for Julia package development as he used to have. I'm not aware of other implementations but there could easily be some that I don't know of. For now, it will probably be easiest to roll your own version or fork Dahua's package.On Fri, Mar 25, 2016 at 12:51 PM, Benjamin Deonovic wrote:Can someone point me to the best place to do multinomial logistic regression in Julia? I've tried with https://github.com/lindahua/Regression.jl but I am not getting the right results according to my simulation, I'm not sure how to interpret the output from that package (documentation is rather lacking), and I am not getting any responses on their issues (https://github.com/lindahua/Regression.jl/issues/14). -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to a topic in the Google Groups "julia-stats" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/julia-stats/2pHziogT9LI/unsubscribe. To unsubscribe from this group and all its topics, send an email to julia-stats...@googlegroups.com. For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "julia-stats" group. To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email]. For more options, visit https://groups.google.com/d/optout.