Evaluator

Evaluator

bigdl.chronos.metric.forecast_metrics.mae(y_label, y_predict)

Calculate the mean absolute error (MAE).

\[\text{MAE} = \frac{1}{n}\sum_{t=1}^n |y_t-\hat{y_t}|\]

Parameters

• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.mse(y_label, y_predict)

Calculate the mean squared error (MSE).

\[\text{MSE} = \frac{1}{n}\sum_{t=1}^n (y_t-\hat{y_t})^2\]

Parameters

• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.rmse(y_label, y_predict)

Calculate square root of the mean squared error (RMSE).

\[\text{RMSE} = \sqrt{(\frac{1}{n}\sum_{t=1}^n (y_t-\hat{y_t})^2)}\]

Parameters

• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.mape(y_label, y_predict)

Calculate mean absolute percentage error (MAPE).

\[\text{MAPE} = \frac{100\%}{n}\sum_{t=1}^n |\frac{y_t-\hat{y_t}}{y_t}|\]

Parameters

• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.smape(y_label, y_predict)

Calculate Symmetric mean absolute percentage error (sMAPE).

\[\text{sMAPE} = \frac{100\%}{n} \sum_{t=1}^n \frac{|y_t-\hat{y_t}|}{|y_t|+|\hat{y_t}|}\]

Parameters

• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 0.0).

bigdl.chronos.metric.forecast_metrics.r2(y_label, y_predict)

Calculate the r2 score.

\[R^2 = 1-\frac{\sum_{t=1}^n (y_t-\hat{y_t})^2}{\sum_{t=1}^n (y_t-\bar{y})^2}\]

Parameters

• y_label – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_predict – Array-like of shape = (n_samples, *). Estimated target values.

Returns

Ndarray of floats. An array of non-negative floating point values (the best value is 1.0).

class bigdl.chronos.metric.forecast_metrics.Evaluator

Bases: object

Evaluate metrics for y_true and y_pred.

static evaluate(metrics, y_true, y_pred, aggregate='mean')

Evaluate a specific metrics for y_true and y_pred. :param metrics: String or list in [‘mae’, ‘mse’, ‘rmse’, ‘r2’, ‘mape’, ‘smape’] for built-in

metrics. If callable function, it signature should be func(y_true, y_pred), where y_true and y_pred are numpy ndarray.

Parameters

• y_true – Array-like of shape = (n_samples, *). Ground truth (correct) target values.

• y_pred – Array-like of shape = (n_samples, *). Estimated target values.

• aggregate – aggregation method. Currently, “mean” and None are supported, ‘mean’ represents aggregating by mean, while None will return the element-wise result. The value defaults to ‘mean’.

Returns

Float or ndarray of floats. A floating point value, or an array of floating point values, one for each individual target.

static get_latency(func, *args, num_running=100, **kwargs)

Return the time cost in milliseconds of a specific function by running multiple times.

Parameters

• func – The function to be tested for latency.

• args – arguments for the tested function.

• num_running – Int and the value is positive. Specify the running number of the function and the value defaults to 100.

• kwargs – other arguments for the tested function.

Returns

Dictionary of str:float. Show the information of the time cost in milliseconds.

Example

>>> # to get the inferencing performance of a trained TCNForecaster
>>> x = next(iter(test_loader))[0]
>>> # run forecaster.predict(x.numpy()) for len(tsdata_test.df) times
>>> # to evaluate the time cost
>>> latency = Evaluator.get_latency(forecaster.predict, x.numpy(),num_running = len(tsdata_test.df))
>>> # an example output:
>>> # {"p50": 3.853, "p90": 3.881, "p95": 3.933, "p99": 4.107}