My name is Philipp C. Heckel and I write about nerdy things.

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


  • May 20 / 2013
  • 3
Uncategorized

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


Contents

1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion		 A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography

Download as PDF: This article is a web version of my Master’s thesis. Feel free to download the original PDF version.

F. Best Algorithms by CPU Usage

The following table lists the best algorithm configurations with regard to the processor usage. The list is sorted by the average CPU usage (last column). All values are in percent.

Rank Algorithm Configuration A B C D Average
1 Custom-125-30/Fixed-16/Cipher 67.12 78.58 65.28 54.30 66.05
2 Custom-125-30/Fixed-8/Cipher 75.44 80.86 66.14 53.55 66.85
3 Custom-125-30/Fixed-16/Gzip-Cipher 70.36 81.92 70.02 52.50 68.15
4 Custom-125-30/Fixed-4/Cipher 72.28 82.70 67.84 57.45 69.33
5 Custom-125-30/Fixed-8/Gzip-Cipher 79.82 83.26 70.56 55.60 69.81
6 Custom-125-30/Fixed-4/Gzip-Cipher 76.42 84.04 71.88 56.85 70.92
7 Custom-250-30/Fixed-8/Cipher 85.22 85.72 76.60 54.15 72.16
8 Custom-250-30/Fixed-16/Cipher 70.88 84.02 76.20 58.20 72.81
9 Custom-250-30/Fixed-4/Cipher 78.50 87.50 77.90 58.90 74.77
10 Custom-250-30/Fixed-16/Gzip-Cipher 74.96 86.92 80.18 59.05 75.38
11 Custom-250-30/Fixed-4/Gzip-Cipher 80.98 89.12 81.50 56.50 75.71
12 Custom-250-30/Fixed-8/Gzip-Cipher 84.98 87.76 80.52 60.30 76.19
13 Custom-125-30/Fixed-4/Bzip2-Cipher 89.52 86.70 84.72 62.40 77.94
14 Custom-125-30/TTTD-8-PLAIN/Cipher 75.20 86.54 82.34 65.25 78.04
15 Custom-125-30/TTTD-16-PLAIN/Cipher 79.20 86.40 81.52 68.10 78.67
16 Custom-125-30/Fixed-16/Bzip2-Cipher 81.40 86.50 84.38 65.75 78.88
17 Custom-125-30/TTTD-16-PLAIN/Gzip-Cipher 83.78 87.90 83.14 67.55 79.53
18 Custom-125-30/TTTD-8-PLAIN/Gzip-Cipher 77.50 88.02 83.86 67.55 79.81
19 Custom-125-30/Fixed-8/Bzip2-Cipher 93.76 88.10 84.34 67.70 80.05
20 Custom-125-30/TTTD-4-PLAIN/Gzip-Cipher 79.22 88.62 85.88 67.95 80.82
21 Custom-125-30/TTTD-4-PLAIN/Cipher 78.64 87.42 84.60 72.60 81.54
22 Custom-250-30/Fixed-8/Bzip2-Cipher 96.16 88.32 90.66 68.15 82.38
23 Custom-125-30/TTTD-8-Adler32/Cipher 89.06 88.94 88.86 69.35 82.38
24 Custom-125-30/TTTD-4-Adler32/Cipher 93.16 89.28 88.98 68.90 82.39
25 Custom-250-30/Fixed-4/Bzip2-Cipher 90.76 89.16 90.76 68.65 82.86
26 Custom-250-30/Fixed-16/Bzip2-Cipher 87.72 91.32 90.60 66.80 82.91
27 Custom-125-30/TTTD-16-Rabin/Cipher 77.28 90.76 85.26 72.75 82.92
28 Custom-125-30/TTTD-8-Adler32/Gzip-Cipher 92.48 90.42 89.62 68.90 82.98
29 Custom-125-30/TTTD-16-Adler32/Cipher 95.70 89.54 88.78 71.55 83.29
30 Custom-125-0/Fixed-8/Cipher 126.18 81.96 99.24 68.95 83.38
31 Custom-250-30/TTTD-16-PLAIN/Cipher 83.46 90.56 88.36 71.65 83.52
32 Custom-125-30/TTTD-4-Rabin/Cipher 75.72 90.88 85.44 74.75 83.69
33 Custom-250-30/TTTD-4-PLAIN/Cipher 83.74 90.80 90.24 70.05 83.70
34 Custom-250-30/TTTD-8-PLAIN/Cipher 80.18 90.56 88.92 72.50 83.99
35 Custom-125-30/TTTD-16-Adler32/Gzip-Cipher 93.86 90.54 89.50 72.15 84.06
36 Custom-125-30/TTTD-8-Rabin/Cipher 78.42 91.60 85.54 75.40 84.18
37 Custom-125-30/TTTD-4-Adler32/Gzip-Cipher 93.38 90.22 90.00 72.70 84.31
38 Custom-125-30/TTTD-4-Rabin/Gzip-Cipher 78.04 91.56 86.40 75.15 84.37
39 Custom-125-30/TTTD-16-Rabin/Gzip-Cipher 78.90 91.80 86.34 75.00 84.38
40 Custom-125-0/Fixed-16/Cipher 148.94 84.86 99.22 69.55 84.54
41 Custom-250-30/TTTD-8-PLAIN/Gzip-Cipher 81.78 91.86 90.06 72.75 84.89
42 Custom-250-30/TTTD-16-PLAIN/Gzip-Cipher 89.42 91.96 89.68 73.90 85.18
43 Custom-125-30/TTTD-8-Rabin/Gzip-Cipher 79.96 92.02 86.60 78.10 85.57
44 Custom-250-30/TTTD-4-PLAIN/Gzip-Cipher 84.52 92.40 91.20 73.15 85.58
45 Custom-250-30/TTTD-4-Adler32/Cipher 94.30 92.70 93.20 71.50 85.80
46 Custom-250-30/TTTD-8-Adler32/Cipher 93.54 92.78 92.98 72.10 85.95
47 Custom-125-0/Fixed-8/Bzip2-Cipher 119.10 91.66 99.34 67.80 86.27
48 Custom-250-0/Fixed-4/Bzip2-Cipher 111.06 93.32 99.42 66.20 86.31
49 Custom-250-30/TTTD-16-Adler32/Cipher 95.64 92.40 92.82 73.75 86.32
50 Custom-125-30/TTTD-16-PLAIN/Bzip2-Cipher 91.98 92.24 89.56 77.65 86.48
51 Custom-250-30/TTTD-4-Rabin/Cipher 84.22 93.88 91.04 75.05 86.66
52 Custom-125-30/TTTD-4-PLAIN/Bzip2-Cipher 87.90 89.88 90.68 79.60 86.72
53 Custom-125-30/TTTD-8-PLAIN/Bzip2-Cipher 85.20 92.30 89.80 78.20 86.77
54 Custom-250-0/Fixed-4/Cipher 123.78 96.26 99.20 65.15 86.87
55 Custom-250-30/TTTD-16-Adler32/Gzip-Cipher 95.84 93.64 93.36 74.55 87.18
56 Custom-250-0/Fixed-8/Bzip2-Cipher 114.80 93.08 99.38 69.10 87.19
57 Custom-125-0/Fixed-16/Bzip2-Cipher 105.40 90.98 99.34 71.65 87.32
58 Custom-125-0/Fixed-4/Cipher 122.10 95.92 101.72 64.80 87.48
59 Custom-250-30/TTTD-16-Rabin/Cipher 84.68 93.60 90.58 78.75 87.64
60 Custom-250-30/TTTD-8-Adler32/Gzip-Cipher 91.94 93.80 93.60 75.60 87.67
61 Custom-250-0/Fixed-16/Cipher 147.18 95.58 99.26 68.35 87.73
62 Custom-250-30/TTTD-4-Adler32/Gzip-Cipher 95.72 93.84 93.78 75.75 87.79
63 Custom-250-0/Fixed-4/Gzip-Cipher 112.78 97.32 99.36 66.70 87.79
64 Custom-250-0/Fixed-16/Bzip2-Cipher 106.18 93.86 99.34 70.30 87.83
65 Custom-125-30/TTTD-4-Adler32/Bzip2-Cipher 94.72 92.28 92.86 78.55 87.90
66 Custom-125-30/TTTD-8-Adler32/Bzip2-Cipher 94.06 92.18 92.78 78.85 87.94
67 Custom-250-30/TTTD-4-Rabin/Gzip-Cipher 85.76 94.82 91.86 77.20 87.96
68 Custom-125-0/Fixed-8/Gzip-Cipher 123.02 97.44 99.16 67.50 88.03
69 Custom-125-0/Fixed-4/Gzip-Cipher 115.54 97.94 99.58 66.60 88.04
70 Custom-250-0/Fixed-8/Gzip-Cipher 122.80 97.02 99.28 68.05 88.12
71 Custom-250-30/TTTD-8-PLAIN/Bzip2-Cipher 87.44 93.54 94.06 76.80 88.13
72 Custom-250-30/TTTD-16-Rabin/Gzip-Cipher 85.96 94.66 91.54 78.35 88.18
73 Custom-125-0/Fixed-16/Gzip-Cipher 142.22 96.08 99.24 69.45 88.26
74 Custom-250-30/TTTD-8-Rabin/Gzip-Cipher 86.12 95.08 91.80 78.00 88.29
75 Custom-250-30/TTTD-8-Rabin/Cipher 85.04 94.76 90.98 79.15 88.30
76 Custom-125-30/TTTD-4-Rabin/Bzip2-Cipher 87.74 92.78 91.08 81.45 88.44
77 Custom-125-30/TTTD-16-Adler32/Bzip2-Cipher 94.34 93.30 92.76 79.30 88.45
78 Custom-125-30/TTTD-16-Rabin/Bzip2-Cipher 86.86 93.12 91.36 81.10 88.53
79 Custom-250-0/Fixed-16/Gzip-Cipher 140.76 95.68 99.38 70.70 88.59
80 Custom-125-30/TTTD-8-Rabin/Bzip2-Cipher 87.60 93.02 91.24 82.40 88.89
81 Custom-125-0/Fixed-4/Bzip2-Cipher 109.48 94.12 99.20 73.55 88.96
82 Custom-250-30/TTTD-4-PLAIN/Bzip2-Cipher 90.44 93.36 94.46 79.60 89.14
83 Custom-250-0/Fixed-8/Cipher 123.12 96.14 99.22 73.30 89.55
84 Custom-250-30/TTTD-16-PLAIN/Bzip2-Cipher 93.58 94.98 94.02 81.10 90.03
85 Custom-125-0/TTTD-4-PLAIN/Gzip-Cipher 107.14 98.92 99.24 74.15 90.77
86 Custom-250-30/TTTD-4-Adler32/Bzip2-Cipher 96.06 94.56 95.70 82.25 90.84
87 Custom-250-0/TTTD-4-Adler32/Cipher 107.78 98.36 99.22 74.95 90.84
88 Custom-250-0/TTTD-4-PLAIN/Cipher 115.40 98.36 99.36 75.40 91.04
89 Custom-250-30/TTTD-8-Adler32/Bzip2-Cipher 93.90 95.80 95.60 81.80 91.07
90 Custom-250-30/TTTD-16-Adler32/Bzip2-Cipher 96.30 95.84 95.54 82.00 91.13
91 Custom-125-0/TTTD-8-PLAIN/Gzip-Cipher 103.16 98.50 99.22 75.95 91.22
92 Custom-250-0/TTTD-4-PLAIN/Gzip-Cipher 106.54 99.10 99.40 75.20 91.23
93 Custom-125-0/TTTD-4-PLAIN/Bzip2-Cipher 104.66 95.42 99.22 79.30 91.31
94 Custom-125-0/TTTD-8-Adler32/Cipher 109.36 99.10 99.22 75.85 91.39
95 Custom-250-0/TTTD-16-PLAIN/Gzip-Cipher 107.86 98.46 99.28 76.60 91.45
96 Custom-125-0/TTTD-4-PLAIN/Cipher 113.54 96.78 99.44 78.40 91.54
97 Custom-125-0/TTTD-8-PLAIN/Bzip2-Cipher 101.02 96.46 99.36 79.00 91.61
98 Custom-125-0/TTTD-8-PLAIN/Cipher 111.90 96.54 99.28 79.05 91.62
99 Custom-125-0/TTTD-16-PLAIN/Cipher 111.36 97.92 99.24 77.75 91.64
100 Custom-125-0/TTTD-16-Adler32/Cipher 112.96 98.30 99.22 77.40 91.64
101 Custom-250-30/TTTD-4-Rabin/Bzip2-Cipher 92.64 94.68 94.80 85.45 91.64
102 Custom-250-0/TTTD-8-PLAIN/Gzip-Cipher 102.74 98.52 99.32 77.55 91.80
103 Custom-250-0/TTTD-16-Adler32/Cipher 111.08 98.50 99.24 77.70 91.81
104 Custom-250-0/TTTD-4-PLAIN/Bzip2-Cipher 104.88 96.18 99.40 80.15 91.91
105 Custom-250-30/TTTD-8-Rabin/Bzip2-Cipher 90.70 94.82 94.94 86.10 91.95
106 Custom-250-30/TTTD-16-Rabin/Bzip2-Cipher 90.48 95.02 94.92 86.15 92.03
107 Custom-250-0/TTTD-16-Adler32/Bzip2-Cipher 110.42 97.62 99.40 79.10 92.04
108 Custom-125-0/TTTD-16-Adler32/Gzip-Cipher 110.00 98.90 99.16 78.20 92.09
109 Custom-250-0/TTTD-16-PLAIN/Cipher 110.50 97.88 99.40 79.05 92.11
110 Custom-250-0/TTTD-8-PLAIN/Bzip2-Cipher 100.64 98.32 99.48 78.70 92.17
111 Custom-250-0/TTTD-4-Adler32/Bzip2-Cipher 108.12 98.06 99.28 79.75 92.36
112 Custom-125-0/TTTD-8-Adler32/Gzip-Cipher 108.96 98.90 99.34 78.95 92.40
113 Custom-125-0/TTTD-16-Adler32/Bzip2-Cipher 109.98 97.86 99.34 80.00 92.40
114 Custom-125-0/TTTD-4-Adler32/Bzip2-Cipher 105.60 97.36 99.16 80.80 92.44
115 Custom-125-0/TTTD-16-PLAIN/Bzip2-Cipher 109.36 97.88 99.42 80.05 92.45
116 Custom-125-0/TTTD-4-Adler32/Cipher 108.72 98.54 99.74 79.10 92.46
117 Custom-250-0/TTTD-16-Adler32/Gzip-Cipher 111.40 98.76 99.26 79.65 92.56
118 Custom-125-0/TTTD-16-PLAIN/Gzip-Cipher 110.96 98.64 99.24 79.80 92.56
119 Custom-125-0/TTTD-4-Adler32/Gzip-Cipher 108.66 99.04 99.32 79.50 92.62
120 Custom-250-0/TTTD-8-PLAIN/Cipher 112.72 98.06 99.38 80.55 92.66
121 Custom-125-0/TTTD-8-Adler32/Bzip2-Cipher 107.26 96.78 99.30 82.40 92.83
122 Custom-250-0/TTTD-8-Adler32/Bzip2-Cipher 104.18 97.28 99.26 82.15 92.90
123 Custom-125-0/TTTD-4-Rabin/Cipher 112.18 95.70 99.80 83.35 92.95
124 Custom-250-0/TTTD-8-Adler32/Gzip-Cipher 108.10 99.14 99.32 80.60 93.02
125 Custom-250-0/TTTD-4-Adler32/Gzip-Cipher 107.62 99.00 99.32 81.00 93.11
126 Custom-250-0/TTTD-16-PLAIN/Bzip2-Cipher 109.80 98.08 99.34 82.40 93.27
127 Custom-250-0/TTTD-16-Rabin/Cipher 107.00 98.78 99.18 82.05 93.34
128 Custom-125-0/TTTD-4-Rabin/Gzip-Cipher 106.26 98.98 99.44 81.95 93.46
129 Custom-250-0/TTTD-4-Rabin/Gzip-Cipher 106.84 98.90 99.22 82.55 93.56
130 Custom-125-0/TTTD-8-Rabin/Cipher 106.16 96.28 99.62 84.85 93.58
131 Custom-125-0/TTTD-4-Rabin/Bzip2-Cipher 103.90 97.62 99.68 84.65 93.98
132 Custom-250-0/TTTD-8-Adler32/Cipher 109.46 98.76 98.96 84.30 94.01
133 Custom-125-0/TTTD-16-Rabin/Cipher 107.16 98.52 99.40 84.50 94.14
134 Custom-250-0/TTTD-16-Rabin/Bzip2-Cipher 101.06 96.86 99.84 86.70 94.47
135 Custom-250-0/TTTD-8-Rabin/Bzip2-Cipher 100.66 97.38 99.78 86.25 94.47
136 Custom-125-0/TTTD-8-Rabin/Bzip2-Cipher 100.88 97.88 99.84 85.85 94.52
137 Custom-125-0/TTTD-16-Rabin/Bzip2-Cipher 101.04 98.04 99.86 85.70 94.53
138 Custom-250-0/TTTD-4-Rabin/Bzip2-Cipher 103.40 97.82 99.82 86.15 94.60
139 Custom-125-0/TTTD-16-Rabin/Gzip-Cipher 105.90 99.02 99.34 85.60 94.65
140 Custom-125-0/TTTD-8-Rabin/Gzip-Cipher 103.82 99.52 99.40 85.05 94.66
141 Custom-250-0/TTTD-16-Rabin/Gzip-Cipher 103.80 99.08 99.24 86.05 94.79
142 Custom-250-0/TTTD-8-Rabin/Cipher 106.78 99.04 99.14 87.15 95.11
143 Custom-250-0/TTTD-4-Rabin/Cipher 111.20 97.98 99.40 88.35 95.24
144 Custom-250-0/TTTD-8-Rabin/Gzip-Cipher 103.24 99.60 99.22 87.70 95.51

Table F.1: Average CPU usage over all datasets, ordered by the average (values of dataset A ignored in ranking and average).

>> Next chapter: Bibliography

1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion		 A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography
Pages:<12 ... 1314 15>

3 Comments

  1. JP

    Hi,

    I would love to see a ebook version of your thesis (epub or mobi). Would that be possible ?

    thanks



  3. Thiruven Madhavan

    Hi Philipp:
    Good Morning. Possible to receive pdf version of your thesis.
    cheers
    Madhavan