Robust Methods for Soft Clustering of Multidimensional Time Series ^†

Abstract

Three robust algorithms for clustering multidimensional time series from the perspective of underlying processes are proposed. The methods are robust extensions of a fuzzy C-means model based on estimates of the quantile cross-spectral density. Robustness to the presence of anomalous elements is achieved by using the so-called metric, noise and trimmed approaches. Analyses from a wide simulation study indicate that the algorithms are substantially effective in coping with the presence of outlying series, clearly outperforming alternative procedures. The usefulness of the suggested methods is also highlighted by means of a specific application.

1. Introduction

Clustering of time series is a pivotal problem in statistics with several applications [1,2]. Generally, the goal is to divide collection of unlabelled time series into uniform groups so that intra-cluster similarity is maximized wheres the inter-cluster similarity is minimized. Most of the current techniques deal with univariate time series (UTS), while clustering of multidimensional time series (MTS) has received limited attention. This paper proposes three robust clustering methods for MTS. All of them are aimed at neutralizing the effect of outlying series while detecting the underlying grouping structure.

2. Robust Clustering Methods for Multivariate Time Series

Let $\{{\mathit{X}}_t,t\in \mathbb{Z}\}=\{(X_{t,1},\dots ,X_{t,d}),t\in \mathbb{Z}\}$ be a d-variate real-valued strictly stationary stochastic process. Let $F_j$ the marginal distribution function of $X_{t,j}$, $j=1,\dots ,d$, and let $q_j\left(\tau \right)=F_j^{-1}\left(\tau \right)$, $\tau \in [0,1]$, the corresponding quantile function. Fixed $l\in \mathbb{Z}$ and an arbitrary couple of quantile levels $(\tau ,{\tau }^{\prime })\in {[0,1]}^2$, consider the cross-covariance of the indicator functions $I\left\{X_{t,j_1}\le q_{j_1}\left(\tau \right)\right\}$ and $I\left\{X_{t+l,j_2}\le q_{j_2}\left({\tau }^{\prime }\right)\right\}$

$${\gamma }_{j_1,j_2}(l,\tau ,{\tau }^{\prime })=\mathrm{Cov}\left(I\left\{X_{t,j_1}\le q_{j_1}\left(\tau \right)\right\},I\left\{X_{t+l,j_2}\le q_{j_2}\left({\tau }^{\prime }\right)\right\}\right),$$

(1)

for $1\le j_1,j_2\le d$. Taking $j_1=j_2=j$, the function ${\gamma }_{j,j}(l,\tau ,{\tau }^{\prime })$, with $(\tau ,{\tau }^{\prime })\in {[0,1]}^2$, so-called quantile autocovariance function (QAF) of lag l, generalizes the traditional autocovariance function.

For the multivariate process $\{{\mathit{X}}_t,t\in \mathbb{Z}\}$, we can consider the $d\times d$ matrix $\mathsf{\Gamma }(l,\tau ,{\tau }^{\prime })={\left({\gamma }_{j_1,j_2}(l,\tau ,{\tau }^{\prime })\right)}_{1\le j_1,j_2\le d}$, which simultaneously gives information about both the cross-dependence (when $j_1\ne j_2$) and the serial dependence (since there is a lag l).

Under appropriate summability conditions (mixing conditions), we can define the the Fourier transform of the cross-covariances. In this regards, the quantile cross-spectral density is given by

$${\mathfrak{f}}_{j_1,j_2}(\omega ,\tau ,{\tau }^{\prime })=(1/2\pi )\sum _{l=-\infty }^{\infty }{\gamma }_{j_1,j_2}(l,\tau ,{\tau }^{\prime })e^{-il\omega },$$

(2)

for $1\le j_1,j_2\le d$, $\omega \in \mathbb{R}$ and $\tau ,{\tau }^{\prime }\in [0,1]$. Note that ${\mathfrak{f}}_{j_1,j_2}(\omega ,\tau ,{\tau }^{\prime })$ is complex-valued.

The quantile cross-spectral density contains information about the general dependence patterns of a given stochastic process. For a specific realization of the process, this quantity can be consistently estimated by means of the so-called smoothed CCR-periodogram, ${\widehat{G}}_{T,R}^{j_1,j_2}(\omega ,\tau ,{\tau }^{\prime })$, proposed by [3].

Based on previous remarks, a simple dissimilarity measure between two realizations of the d-variate process (MTS) can be defined as follows. Given the i-th MTS, ${\mathit{X}}_t^{\left(i\right)}$, consider the set $G^{\left(i\right)}=\{{\widehat{G}}_{T,R}^{j_1,j_2}(\omega ,\tau ,{\tau }^{\prime }),j_1,j_2=1,\dots ,d,\omega \in \mathsf{\Omega },\tau ,{\tau }^{\prime }\in \mathcal{T}\}$, where $\mathsf{\Omega }$ is the set of Fourier frequencies and $\mathcal{T}=\{0.1,0.5,0.9\}$. Let ${\mathsf{\Psi }}^{\left(i\right)}$ be the vector formed by concatenating the elements of the set $G^{\left(i\right)}$. The dissimilarity measure between the series ${\mathit{X}}_t^{\left(1\right)}$ and ${\mathit{X}}_t^{\left(2\right)}$ is defined as the Euclidean distance between the complex vectors ${\mathsf{\Psi }}^{\left(1\right)}$ and ${\mathsf{\Psi }}^{\left(2\right)}$. We call this dissimilarity $d_{QCD}$.

The dissimilarity $d_{QCD}$ is used to develop three robust fuzzy clustering methods. All of them assume that we want to group n MTS into C clusters, and are based on the traditional fuzzy C-means clustering algorithm. They look for the set of centroids $\overline{\mathsf{\Psi }}=\{{\overline{\mathsf{\Psi }}}^{\left(1\right)},\dots ,{\overline{\mathsf{\Psi }}}^{\left(C\right)}\}$, and the $n\times C$ matrix of fuzzy coefficients, $\mathbf{U}=\left(u_{ic}\right)$, $i=1,\dots ,n$, $c=1,\dots ,C$, which define the solution of a given minimization problem. The quantity $u_{ic}$ represents the membership degree of the i-th MTS in the c-th cluster. The minimization problem for the first method is the following:

$$\underset{\overline{\mathsf{\Psi }},\mathbf{U}}{min}\sum _{i=1}^n\sum _{c=1}^C{u}_{ic}^m\left[1-\exp \left\{-\beta {∥{\mathsf{\Psi }}^{\left(i\right)}-{\overline{\mathsf{\Psi }}}^{\left(c\right)}∥}_2^2\right\}\right]\text{w.r.t}\sum _{c=1}^C{u}_{ic}=1 \mathrm{and} u_{ic}\ge 0,$$

where $\beta$ is an hyperparameter that needs to be set in advance and m is a parameter which determines the fuzziness of the partition, frequently called the fuziness parameter.

The exponential distance is used in the previous model because it is capable of neutralizing the effect of outlying series by spreading out their membership degrees between the different clusters [4].

The second robust procedure follows the noise cluster approach, and takes into account the following minimization problem:

$${\displaystyle \underset{\overline{\mathsf{\Psi }},\mathbf{U}}{min}\sum _{i=1}^n\sum _{c=1}^{C-1}{u}_{ic}^m{∥{\mathsf{\Psi }}^{\left(i\right)}-{\overline{\mathsf{\Psi }}}^{\left(c\right)}∥}_2^2+\sum _{i=1}^n{\delta }^2{\left(1-\sum _{c=1}^{C-1}{u}_{ic}\right)}^m\text{w.r.t.}\sum _{c=1}^C{u}_{ic}=1 \mathrm{and} u_{ic}\ge 0,}$$

where $\delta >0$ is the a parameter known as the noise distance, which has to be specified in advance.

The previous model includes C groups, but only $(C-1)$ are “real” clusters. The noise cluster is artificially created for outlier identification purposes. The aim is to locate the outliers and place them in the noise cluster, which is represented by a fictitious prototype that has a constant distance from every MTS (the noise distance, $\delta$).

The third technique can be expressed by means of the minimization problem:

$${\displaystyle \underset{\mathbf{Y},\mathbf{U}}{min}\sum _{i=1}^{H\left(\alpha \right)}\sum _{c=1}^C{u}_{ic}^m{∥{\mathsf{\Psi }}^{\left(i\right)}-{\overline{\mathsf{\Psi }}}^{\left(c\right)}∥}^2\text{w.r.t.}\sum _{c=1}^C{u}_{ic}=1 \mathrm{and} u_{ic}\ge 0.}$$

where $\mathbf{Y}$ ranges on all the subsets of $\mathsf{\Psi }=\{{\mathsf{\Psi }}^{\left(1\right)},\dots ,{\mathsf{\Psi }}^{\left(n\right)}\}$ of size $H\left(\alpha \right)=\lfloor n(1-\alpha )\rfloor$. The model attains its robustness by removing a certain proportion of the series and requires the specification of the fraction $\alpha$ of the data to be trimmed.

The three previously presented robust models have been analysed by means of a broad simulation study containing a wide variety of generating processes. Two alternative dissimilarities were taken into account for comparison purposes [5,6]. In all cases, the three proposed algorithms outperformed the competitors.

3. Application to real data

The three techniques proposed in Section 2 were applied to perform clustering in a real MTS database. Specifically, we considered daily stock returns and trading volume of the top 20 companies of the S&P 500 index, thus obtaining 20 bivariate MTS.

Table 1. Membership degrees for the top 20 companies in the S&P 500 index by considering the trimmed approach and a 6-cluster partition.

Company	${\mathit{C}}_1$	${\mathit{C}}_2$	${\mathit{C}}_3$	${\mathit{C}}_4$	${\mathit{C}}_5$	${\mathit{C}}_6$
AAPL	0.083	0.146	0.299	0.365	0.066	0.041
MSFT	0.107	0.049	0.213	0.356	0.099	0.176
AMZN	0.865	0.017	0.051	0.032	0.010	0.025
GOOGL	0.682	0.032	0.092	0.128	0.025	0.040
GOOG	0.902	0.010	0.031	0.028	0.008	0.022
FB	0.002	0.983	0.006	0.004	0.003	0.002
TSLA	0.023	0.012	0.056	0.885	0.013	0.010
BRK.B	-	-	-	-	-	-
V	0.004	0.014	0.015	0.017	0.941	0.009
JNJ	0.004	0.015	0.019	0.013	0.937	0.013
WMT	-	-	-	-	-	-
JPM	0.002	0.001	0.003	0.003	0.002	0.989
MA	0.005	0.006	0.968	0.010	0.005	0.006
PG	0.015	0.012	0.028	0.016	0.019	0.909
UNH	0.006	0.924	0.026	0.013	0.022	0.008
DIS	0.020	0.038	0.772	0.099	0.042	0.030
NVDA	0.025	0.020	0.085	0.804	0.043	0.024
HD	-	-	-	-	-	-
PYPL	0.155	0.301	0.297	0.115	0.057	0.075
BAC	0.076	0.086	0.225	0.067	0.060	0.485

shows the membership degrees of the series concerning the trimmed approach.

The symbols in bold correspond to the companies which were trimmed away, Berkshire Hathaway (BRK.B), Walmart (WMT) and Home Depot (HD). Similar clustering solutions were obtained with the remaining two methods.

4. Conclusions

This work proposes three robust methods to perform fuzzy clustering of MTS. They are based on the so-called exponential, noise and trimmed ideas. Each approach attains robustness to outlying series in a different way. The three procedures have been presented and assessed through a wide simulation study, substantially outperforming alternative approaches. A real data application has been also carried out in order to show the usefulness of the presented techniques.