Hi
For regression model evaluation, can we make a decision based on comparing the R-squared values of different algorithms ?
For instance R-squared for Multiple linear regression is 0.8 and for Polynomial Linear regression is 0.6!
Therefore we should choose Multiple linear regression!
Is it an appropriate approach?
I am going to paraphrase your question and make it more general by stating it as follows. Is it a good idea to use a particular metric to train your model when you know that a different evaluation metric will be used on the test set? In your specific case you want to use R-squared on the training set knowing that MAE is used on the test set.
The simple answer is that using a different evaluation metric between train and test creates a risk that your model is fitting to the wrong thing, so better to avoid it if possible. Based on the fact that you are asking the question, I think you may have already known that. So I’ll offer more.
One way to mitigate the risk and/or clarify the differences created by using a different metric would be to use a holdout set (in this case you could use the evaluation set that we provided). Train your models with R-squared and then compare their prediction accuracy on the evaluation set using MAE. That would allow you to make a comparison of the models based on MAE and to understand what the differences are between the model outputs in terms of MAE. Of course in this competition you could also just submit results from both models and see which scores better.
I hope this helps. If not, post again and I’ll try again.