Transparency in Evaluation of Machine Learning Models
About Andy Troiano
Hi! I’m Andy Troiano. I currently work as a VP of Risk and Analytics at CAN Capital during the day and GridironAI, at night.
In the world of fantasy sports predictions, there is little accountability for performance.
Our evaluation process examines the performance against leading fantasy sites. We found a low model MSE does not always correlate to good performance. This is due to the complexity of the number of outputs and how those outputs get translated into a fantasy score. We evaluate our models against our competitors and provide our community with those metrics. This allows us to be transparent and honest with our users.
We also learned something valuable in this process, if you have the best score overall while predictions on the best players each week is miserable, your predictions aren’t good.
You can find me on twitter Andytroiano where I mostly tweet about Philly sports and R.