Loading...
Please wait, while we are loading the content...
Similar Documents
Implemetation of Centroid Decomposition Algorithm on Big Data Platforms — Apache Spark vs . Apache Flink
| Content Provider | Semantic Scholar |
|---|---|
| Author | Liu, Qian |
| Copyright Year | 2016 |
| Abstract | The Centroid Decomposition (CD) algorithm is the approximation of the Singular Value Decomposition (SVD) algorithm, which is one of the most used matrix decomposition techniques to deal with real world data analysis tasks. CD algorithm is based on a greedy algorithm, termed the Scalable Sign Vector (SSV), that efficiently determines vectors that are consisted of 1s and -1s as elements, called sign vectors. CD algorithm is generally applied for data analysis tasks that involve long time series, i.e. where the number of rows (observations) is much larger than the number of columns (time series). The goal of this thesis is to implement the CD algorithm on two Big Data platforms, i.e., Apache Spark and Apache Flink. The proposed implementation compares two different data structures for both platforms. The first data structure is the per-element data structure, which distributively transforms the matrix based on every single element. The second data structure, the per-vector data structure, executes every transformation on the basis of each row or column vector. We empirically evaluate the efficiency of the non-streamed Spark and Flink CD implementations respectively. To simulate the streams of time series, we use Apache Kafka to periodically produce new matrix data to a broker and Spark Streaming and Flink Data Streaming to regularly fetch the data and run the CD algorithm. |
| File Format | PDF HTM / HTML |
| Alternate Webpage(s) | https://exascale.info/assets/pdf/students/2016-Qian_CD_Flink-vs-Spark.pdf |
| Language | English |
| Access Restriction | Open |
| Content Type | Text |
| Resource Type | Article |
