{"id":4136,"date":"2023-04-07T09:35:46","date_gmt":"2023-04-07T16:35:46","guid":{"rendered":"https:\/\/jeremywhittaker.com\/?p=4136"},"modified":"2023-04-07T09:47:14","modified_gmt":"2023-04-07T16:47:14","slug":"mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation","status":"publish","type":"post","link":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/","title":{"rendered":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation"},"content":{"rendered":"\n<p>I was running some machine learning models and was struggling to find a metric that could capture errors relative to volatility in the underlying model that I was trying to predict so I created MARDE. Commonly used metrics like Mean Absolute Error (MAE), Mean Absolute Percentage Error (MAPE), Mean Squared Error (MSE), and R-squared (R2) have limitations that failed to capture my model&#8217;s performance.<\/p>\n\n\n\n<p>I created a new metric called Mean Absolute Relative Difference Error (MARDE), which addresses some of the shortcomings of the existing metrics. MARDE measures the average absolute error between the true and predicted values, relative to the consecutive differences in the true values. By considering the relative differences, MARDE offers a more nuanced evaluation of model performance, particularly in situations with varying volatility or fluctuating data patterns.<\/p>\n\n\n\n<p>MARDE = (1\/n) * \u03a3(abs_y_error_i \/ \u0394y_true_i) * 100, for i=1 to n<\/p>\n\n\n\n<p>Here, \u0394y_true represents the absolute differences between consecutive elements in y_true, and abs_y_error represents the absolute differences between y_true and y_pred.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\ndef mean_absolute_volatility_error(y_true: Union&#91;List&#91;float], np.ndarray], y_pred: Union&#91;List&#91;float], np.ndarray]) -&gt; float:\n    \"\"\"\n    Calculate the mean absolute volatility error for the given true and predicted values.\n\n    :param y_true: List or numpy array of true values\n    :param y_pred: List or numpy array of predicted values\n    :return: Mean absolute volatility error (in percentage)\n    \"\"\"\n    # Convert input lists to numpy arrays\n    y_true, y_pred = np.array(y_true), np.array(y_pred)\n\n    # Check if there are less than two elements in y_true\n    if len(y_true) &lt; 2:\n        return np.nan\n\n    # Calculate the absolute differences between consecutive elements in y_true\n    if len(y_true) == 2:\n        abs_diff_y_true = np.abs(y_true&#91;1] - y_true&#91;0])\n    else:\n        abs_diff_y_true = np.abs(np.diff(y_true))\n        # Insert the first difference value at the beginning of the array\n        abs_diff_y_true = np.insert(abs_diff_y_true, 0, abs_diff_y_true&#91;0])\n\n    # Calculate the absolute differences between y_true and y_pred\n    abs_y_error = np.abs(y_true - y_pred)\n\n    # Calculate mean absolute volatility error (in percentage)\n    return np.mean(abs_y_error \/ abs_diff_y_true * 100)\n\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">Comparing MARDE to existing metrics:<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Mean Absolute Error (MAE): MAE measures the average absolute difference between the true and predicted values. However, it does not account for the inherent volatility or fluctuations in the data. MARDE considers this aspect by normalizing the error with respect to the consecutive differences in the true values.\n<ul class=\"wp-block-list\">\n<li>MAE = (1\/n) * \u03a3|y_true_i &#8211; y_pred_i|, for i=1 to n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Mean Absolute Percentage Error (MAPE): MAPE is similar to MAE, but it normalizes the error by dividing it by the true value. While MAPE is useful for expressing errors in percentage terms, it can lead to biased results when dealing with very small or zero true values. MARDE overcomes this issue by using consecutive differences as the normalization factor.\n<ul class=\"wp-block-list\">\n<li>MAPE = (1\/n) * \u03a3(|(y_true_i &#8211; y_pred_i) \/ y_true_i|) * 100, for i=1 to n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>Mean Squared Error (MSE): MSE measures the average squared difference between the true and predicted values. It tends to penalize larger errors more heavily, which can be useful in some cases but might also overemphasize outliers. MARDE, on the other hand, focuses on the absolute differences and is less sensitive to extreme values.\n<ul class=\"wp-block-list\">\n<li>MSE = (1\/n) * \u03a3(y_true_i &#8211; y_pred_i)^2, for i=1 to n<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li>R-squared (R2): R2 measures the proportion of variance in the true values that can be explained by the model. While R2 is useful for assessing the overall goodness-of-fit, it does not provide a direct measure of the prediction error. MARDE, as a metric focused on errors, offers a complementary perspective to R2 in evaluating model performance.\n<ul class=\"wp-block-list\">\n<li>R\u00b2 = 1 &#8211; (SSR \/ SST)<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>SSR (Sum of Squared Residuals) is the sum of the squared differences between the actual dependent variable values (y) and the predicted values (\u0177) from the model;<\/li>\n\n\n\n<li>SST (Total Sum of Squares) is the sum of the squared differences between the actual dependent variable values (y) and the mean of the dependent variable (\u0233).<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<p>The Mean Absolute Relative Difference Error (MARDE) is a valuable addition to the set of evaluation metrics used in forecasting and prediction tasks. By taking into account the consecutive differences in the true values, MARDE provides a more granular assessment of model performance, particularly in situations with fluctuating data patterns. It overcomes some of the limitations of existing metrics and can be employed alongside them for a comprehensive evaluation of prediction models.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>I was running some machine learning models and was struggling to find a metric that could capture errors relative to volatility in the underlying model that I was&#8230;<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-4136","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.2 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation - Jeremy Whittaker<\/title>\n<meta name=\"robots\" content=\"noindex, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation\" \/>\n<meta property=\"og:description\" content=\"I was running some machine learning models and was struggling to find a metric that could capture errors relative to volatility in the underlying model that I was...\" \/>\n<meta property=\"og:url\" content=\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/\" \/>\n<meta property=\"og:site_name\" content=\"Jeremy Whittaker\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/WhittakerJeremy\" \/>\n<meta property=\"article:author\" content=\"https:\/\/www.facebook.com\/WhittakerJeremy\" \/>\n<meta property=\"article:published_time\" content=\"2023-04-07T16:35:46+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2023-04-07T16:47:14+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/04\/MARDE-1.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1089\" \/>\n\t<meta property=\"og:image:height\" content=\"653\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"JeremyWhittaker\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/04\/MARDE-1.png\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"JeremyWhittaker\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/\"},\"author\":{\"name\":\"JeremyWhittaker\",\"@id\":\"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c\"},\"headline\":\"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation\",\"datePublished\":\"2023-04-07T16:35:46+00:00\",\"dateModified\":\"2023-04-07T16:47:14+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/\"},\"wordCount\":574,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/\",\"url\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/\",\"name\":\"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation - Jeremy Whittaker\",\"isPartOf\":{\"@id\":\"https:\/\/new.jeremywhittaker.com\/#website\"},\"datePublished\":\"2023-04-07T16:35:46+00:00\",\"dateModified\":\"2023-04-07T16:47:14+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/new.jeremywhittaker.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/new.jeremywhittaker.com\/#website\",\"url\":\"https:\/\/new.jeremywhittaker.com\/\",\"name\":\"Jeremy Whittaker\",\"description\":\"Research, software, markets, housing, and energy\",\"publisher\":{\"@id\":\"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/new.jeremywhittaker.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c\",\"name\":\"JeremyWhittaker\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g\",\"caption\":\"JeremyWhittaker\"},\"logo\":{\"@id\":\"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g\"},\"sameAs\":[\"http:\/\/www.jeremywhittaker.com\",\"https:\/\/www.facebook.com\/WhittakerJeremy\",\"https:\/\/www.linkedin.com\/in\/jeremywhittaker\/\"],\"url\":\"https:\/\/new.jeremywhittaker.com\/index.php\/author\/jeremywhittaker\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation - Jeremy Whittaker","robots":{"index":"noindex","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"og_locale":"en_US","og_type":"article","og_title":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation","og_description":"I was running some machine learning models and was struggling to find a metric that could capture errors relative to volatility in the underlying model that I was...","og_url":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/","og_site_name":"Jeremy Whittaker","article_publisher":"https:\/\/www.facebook.com\/WhittakerJeremy","article_author":"https:\/\/www.facebook.com\/WhittakerJeremy","article_published_time":"2023-04-07T16:35:46+00:00","article_modified_time":"2023-04-07T16:47:14+00:00","og_image":[{"width":1089,"height":653,"url":"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/04\/MARDE-1.png","type":"image\/png"}],"author":"JeremyWhittaker","twitter_card":"summary_large_image","twitter_title":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation","twitter_image":"https:\/\/new.jeremywhittaker.com\/wp-content\/uploads\/2023\/04\/MARDE-1.png","twitter_misc":{"Written by":"JeremyWhittaker","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#article","isPartOf":{"@id":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/"},"author":{"name":"JeremyWhittaker","@id":"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c"},"headline":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation","datePublished":"2023-04-07T16:35:46+00:00","dateModified":"2023-04-07T16:47:14+00:00","mainEntityOfPage":{"@id":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/"},"wordCount":574,"commentCount":0,"publisher":{"@id":"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c"},"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/","url":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/","name":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation - Jeremy Whittaker","isPartOf":{"@id":"https:\/\/new.jeremywhittaker.com\/#website"},"datePublished":"2023-04-07T16:35:46+00:00","dateModified":"2023-04-07T16:47:14+00:00","breadcrumb":{"@id":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/new.jeremywhittaker.com\/index.php\/2023\/04\/07\/mean-absolute-relative-difference-error-marde-metric-for-improved-forecasting-evaluation\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/new.jeremywhittaker.com\/"},{"@type":"ListItem","position":2,"name":"Mean Absolute Relative Difference Error (MARDE) Metric for Improved Forecasting Evaluation"}]},{"@type":"WebSite","@id":"https:\/\/new.jeremywhittaker.com\/#website","url":"https:\/\/new.jeremywhittaker.com\/","name":"Jeremy Whittaker","description":"Research, software, markets, housing, and energy","publisher":{"@id":"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/new.jeremywhittaker.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/new.jeremywhittaker.com\/#\/schema\/person\/ed0edfdefb3e180693efef453372980c","name":"JeremyWhittaker","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g","url":"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g","caption":"JeremyWhittaker"},"logo":{"@id":"https:\/\/secure.gravatar.com\/avatar\/c8ac20e6dfa86b5f27ce9bffee4851099770cbea5ae7338a274865bfbc8c0218?s=96&d=retro&r=g"},"sameAs":["http:\/\/www.jeremywhittaker.com","https:\/\/www.facebook.com\/WhittakerJeremy","https:\/\/www.linkedin.com\/in\/jeremywhittaker\/"],"url":"https:\/\/new.jeremywhittaker.com\/index.php\/author\/jeremywhittaker\/"}]}},"_links":{"self":[{"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/posts\/4136","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/comments?post=4136"}],"version-history":[{"count":0,"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/posts\/4136\/revisions"}],"wp:attachment":[{"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/media?parent=4136"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/categories?post=4136"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/new.jeremywhittaker.com\/index.php\/wp-json\/wp\/v2\/tags?post=4136"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}