My name is Philipp C. Heckel and I write about nerdy things.

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


  • May 20 / 2013
  • 3
Uncategorized

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


Contents

1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion		 A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography

Download as PDF: This article is a web version of my Master’s thesis. Feel free to download the original PDF version.

E. Best Algorithms by Duration

The following table lists the best algorithm configurations with regard to the overall duration. The list is sorted by the average duration (last column). All values are in seconds.

Rank Algorithm Configuration A B C D Average
1 Custom-250-0/Fixed-8/Cipher 0.10 9.09 1.70 12.92 4.72
2 Custom-250-0/Fixed-16/Cipher 0.10 8.60 1.64 14.58 4.75
3 Custom-125-0/Fixed-16/Cipher 0.09 10.62 1.64 15.16 5.42
4 Custom-250-0/Fixed-4/Cipher 0.10 10.88 1.88 16.82 5.76
5 Custom-250-0/Fixed-16/Gzip-Cipher 0.14 11.14 2.22 15.56 5.80
6 Custom-125-0/Fixed-8/Cipher 0.10 11.99 1.71 15.13 5.84
7 Custom-125-0/Fixed-4/Cipher 0.10 10.84 1.96 17.72 5.88
8 Custom-125-0/Fixed-16/Gzip-Cipher 0.14 11.00 2.20 16.91 5.91
9 Custom-250-0/Fixed-8/Gzip-Cipher 0.14 11.66 2.28 15.89 6.01
10 Custom-125-0/Fixed-8/Gzip-Cipher 0.14 11.52 2.28 17.41 6.15
11 Custom-125-0/Fixed-4/Gzip-Cipher 0.15 13.33 2.43 18.46 6.85
12 Custom-250-0/Fixed-4/Gzip-Cipher 0.15 13.51 2.45 18.33 6.90
13 Custom-250-30/Fixed-16/Cipher 0.19 12.53 2.63 20.29 6.90
14 Custom-250-30/Fixed-8/Cipher 0.19 12.97 2.69 20.12 7.03
15 Custom-250-30/Fixed-4/Cipher 0.20 14.98 2.89 20.66 7.74
16 Custom-250-30/Fixed-16/Gzip-Cipher 0.23 14.94 3.21 22.08 8.00
17 Custom-250-30/Fixed-8/Gzip-Cipher 0.24 15.62 3.27 21.37 8.14
18 Custom-250-0/TTTD-8-PLAIN/Cipher 0.19 17.22 4.65 20.16 8.86
19 Custom-125-30/Fixed-16/Cipher 0.25 16.39 3.56 26.00 9.00
20 Custom-125-0/TTTD-16-PLAIN/Cipher 0.19 17.24 4.69 21.28 9.01
21 Custom-250-0/TTTD-16-PLAIN/Cipher 0.19 17.44 4.67 21.24 9.06
22 Custom-125-30/Fixed-8/Cipher 0.26 16.97 3.63 25.16 9.09
23 Custom-250-30/Fixed-4/Gzip-Cipher 0.25 17.59 3.47 24.68 9.17
24 Custom-250-0/TTTD-4-PLAIN/Cipher 0.19 17.43 4.64 22.92 9.24
25 Custom-125-0/TTTD-4-PLAIN/Cipher 0.19 19.50 3.49 21.81 9.38
26 Custom-125-0/TTTD-8-PLAIN/Cipher 0.19 19.10 4.66 20.90 9.50
27 Custom-125-30/Fixed-4/Cipher 0.26 18.83 3.82 25.11 9.69
28 Custom-125-0/TTTD-16-PLAIN/Gzip-Cipher 0.24 19.56 5.24 23.05 10.08
29 Custom-250-0/TTTD-8-PLAIN/Gzip-Cipher 0.23 19.68 5.21 23.31 10.13
30 Custom-125-0/TTTD-8-PLAIN/Gzip-Cipher 0.23 19.59 5.21 23.79 10.16
31 Custom-250-0/TTTD-16-PLAIN/Gzip-Cipher 0.24 19.77 5.25 24.26 10.28
32 Custom-125-30/Fixed-8/Gzip-Cipher 0.31 19.75 4.26 26.87 10.31
33 Custom-250-0/TTTD-4-PLAIN/Gzip-Cipher 0.23 19.82 5.11 24.93 10.33
34 Custom-125-30/Fixed-16/Gzip-Cipher 0.30 19.06 4.20 29.33 10.38
35 Custom-125-0/TTTD-4-PLAIN/Gzip-Cipher 0.23 19.82 5.10 25.41 10.39
36 Custom-250-0/TTTD-8-Adler32/Cipher 0.22 21.72 5.24 22.09 10.59
37 Custom-125-0/TTTD-16-Adler32/Cipher 0.22 21.79 5.25 22.96 10.72
38 Custom-250-0/TTTD-16-Adler32/Cipher 0.22 21.79 5.21 23.28 10.74
39 Custom-125-0/TTTD-4-Adler32/Cipher 0.25 22.32 4.47 24.50 10.84
40 Custom-250-30/TTTD-8-PLAIN/Cipher 0.27 21.19 5.59 24.62 10.85
41 Custom-125-0/TTTD-8-Adler32/Cipher 0.22 21.50 5.26 25.43 10.93
42 Custom-250-30/TTTD-16-PLAIN/Cipher 0.28 21.04 5.63 26.40 11.03
43 Custom-125-30/Fixed-4/Gzip-Cipher 0.31 21.54 4.44 28.53 11.09
44 Custom-250-30/TTTD-4-PLAIN/Cipher 0.28 21.48 5.45 26.91 11.17
45 Custom-250-0/TTTD-4-Adler32/Cipher 0.25 22.36 5.29 26.96 11.38
46 Custom-250-0/TTTD-8-Adler32/Gzip-Cipher 0.27 23.76 5.62 25.58 11.73
47 Custom-250-0/TTTD-16-Adler32/Gzip-Cipher 0.24 24.20 5.61 24.99 11.78
48 Custom-125-0/TTTD-16-Adler32/Gzip-Cipher 0.24 24.09 5.62 25.34 11.79
49 Custom-125-0/TTTD-8-Adler32/Gzip-Cipher 0.24 23.79 5.61 26.11 11.79
50 Custom-250-0/TTTD-4-Adler32/Gzip-Cipher 0.27 24.71 5.67 25.41 12.01
51 Custom-250-30/TTTD-8-PLAIN/Gzip-Cipher 0.32 23.68 6.15 27.21 12.07
52 Custom-125-0/TTTD-4-Adler32/Gzip-Cipher 0.27 24.86 5.65 26.19 12.13
53 Custom-250-30/TTTD-16-PLAIN/Gzip-Cipher 0.33 23.55 6.23 27.93 12.14
54 Custom-250-30/TTTD-4-PLAIN/Gzip-Cipher 0.33 23.95 5.94 27.96 12.18
55 Custom-250-30/TTTD-16-Adler32/Cipher 0.27 25.74 5.91 27.21 12.59
56 Custom-250-30/TTTD-8-Adler32/Cipher 0.30 25.45 5.92 28.31 12.65
57 Custom-125-30/TTTD-4-PLAIN/Cipher 0.34 25.03 6.22 30.24 12.85
58 Custom-250-30/TTTD-4-Adler32/Cipher 0.30 26.15 5.97 28.50 12.89
59 Custom-125-30/TTTD-8-PLAIN/Cipher 0.33 24.91 6.48 31.69 13.06
60 Custom-125-30/TTTD-16-PLAIN/Cipher 0.35 24.82 6.58 31.81 13.08
61 Custom-250-30/TTTD-16-Adler32/Gzip-Cipher 0.29 27.98 6.32 28.83 13.56
62 Custom-250-30/TTTD-8-Adler32/Gzip-Cipher 0.32 27.73 6.32 30.42 13.69
63 Custom-125-0/TTTD-16-Rabin/Cipher 0.31 29.41 6.79 26.83 13.89
64 Custom-250-30/TTTD-4-Adler32/Gzip-Cipher 0.32 28.68 6.36 30.21 13.96
65 Custom-250-0/TTTD-16-Rabin/Cipher 0.31 29.25 6.77 27.84 13.96
66 Custom-125-30/TTTD-4-PLAIN/Gzip-Cipher 0.39 27.60 6.69 32.79 14.05
67 Custom-125-30/TTTD-8-PLAIN/Gzip-Cipher 0.37 27.48 7.05 33.23 14.17
68 Custom-125-30/TTTD-16-Adler32/Cipher 0.29 29.16 6.52 31.78 14.32
69 Custom-125-30/TTTD-16-PLAIN/Gzip-Cipher 0.39 27.34 7.16 34.55 14.33
70 Custom-250-0/Fixed-16/Bzip2-Cipher 0.50 29.68 6.98 29.38 14.39
71 Custom-250-0/Fixed-8/Bzip2-Cipher 0.50 30.41 6.91 29.15 14.55
72 Custom-125-30/TTTD-8-Adler32/Cipher 0.29 29.11 6.59 33.76 14.56
73 Custom-250-0/TTTD-4-Rabin/Cipher 0.34 31.68 6.89 28.60 14.81
74 Custom-125-0/TTTD-16-Rabin/Gzip-Cipher 0.36 31.03 7.38 29.15 14.83
75 Custom-250-0/TTTD-16-Rabin/Gzip-Cipher 0.36 31.12 7.41 28.97 14.85
76 Custom-125-30/TTTD-4-Adler32/Cipher 0.33 30.05 6.57 34.20 14.89
77 Custom-125-0/Fixed-16/Bzip2-Cipher 0.50 31.18 7.05 30.08 14.93
78 Custom-125-0/Fixed-8/Bzip2-Cipher 0.50 31.41 6.99 30.80 15.07
79 Custom-125-30/TTTD-16-Adler32/Gzip-Cipher 0.31 31.27 6.91 33.54 15.27
80 Custom-125-0/Fixed-4/Bzip2-Cipher 0.51 32.62 7.09 30.50 15.42
81 Custom-125-0/TTTD-8-Rabin/Cipher 0.36 33.69 6.81 29.19 15.45
82 Custom-250-0/Fixed-4/Bzip2-Cipher 0.50 32.72 6.96 31.44 15.52
83 Custom-125-30/TTTD-4-Adler32/Gzip-Cipher 0.34 32.25 6.98 33.31 15.56
84 Custom-125-30/TTTD-8-Adler32/Gzip-Cipher 0.34 31.20 6.98 36.23 15.59
85 Custom-250-0/TTTD-8-Rabin/Cipher 0.36 35.58 6.81 28.23 15.90
86 Custom-125-0/TTTD-4-Rabin/Cipher 0.34 34.80 6.80 31.25 16.01
87 Custom-250-30/TTTD-16-Rabin/Cipher 0.41 33.28 7.80 32.68 16.05
88 Custom-250-0/TTTD-4-Rabin/Gzip-Cipher 0.42 33.80 7.45 33.63 16.21
89 Custom-125-0/TTTD-4-Rabin/Gzip-Cipher 0.42 33.70 7.47 33.90 16.22
90 Custom-250-30/Fixed-16/Bzip2-Cipher 0.59 33.20 7.97 34.29 16.32
91 Custom-250-0/TTTD-8-Rabin/Gzip-Cipher 0.41 36.04 7.37 30.58 16.49
92 Custom-125-0/TTTD-8-Rabin/Gzip-Cipher 0.42 35.98 7.36 31.70 16.60
93 Custom-250-30/Fixed-8/Bzip2-Cipher 0.59 34.63 7.90 33.59 16.63
94 Custom-250-30/Fixed-4/Bzip2-Cipher 0.60 36.25 7.95 34.02 17.18
95 Custom-250-30/TTTD-16-Rabin/Gzip-Cipher 0.47 35.73 8.42 35.12 17.25
96 Custom-250-30/TTTD-4-Rabin/Cipher 0.48 35.40 7.90 37.67 17.31
97 Custom-250-0/TTTD-4-PLAIN/Bzip2-Cipher 0.58 38.40 8.91 33.04 17.97
98 Custom-250-0/TTTD-8-PLAIN/Bzip2-Cipher 0.56 38.04 9.67 33.19 18.10
99 Custom-250-30/TTTD-8-Rabin/Cipher 0.47 39.72 7.80 34.38 18.16
100 Custom-125-30/TTTD-16-Rabin/Cipher 0.49 36.97 8.71 39.07 18.18
101 Custom-125-0/TTTD-4-PLAIN/Bzip2-Cipher 0.59 39.27 9.02 33.64 18.33
102 Custom-250-30/TTTD-4-Rabin/Gzip-Cipher 0.56 37.84 8.45 38.96 18.36
103 Custom-125-0/TTTD-8-PLAIN/Bzip2-Cipher 0.57 39.11 9.73 33.80 18.51
104 Custom-250-0/TTTD-16-PLAIN/Bzip2-Cipher 0.60 37.98 10.11 36.21 18.58
105 Custom-125-30/Fixed-8/Bzip2-Cipher 0.67 38.41 8.95 39.46 18.77
106 Custom-125-30/Fixed-16/Bzip2-Cipher 0.67 38.36 9.01 40.30 18.87
107 Custom-250-30/TTTD-8-Rabin/Gzip-Cipher 0.53 40.20 8.37 38.36 18.96
108 Custom-125-0/TTTD-16-PLAIN/Bzip2-Cipher 0.61 38.71 10.15 38.00 19.02
109 Custom-250-0/TTTD-4-Adler32/Bzip2-Cipher 0.41 41.40 8.73 35.79 19.08
110 Custom-125-30/TTTD-16-Rabin/Gzip-Cipher 0.54 39.36 9.31 40.32 19.22
111 Custom-250-0/TTTD-8-Adler32/Bzip2-Cipher 0.41 41.83 8.86 36.24 19.29
112 Custom-125-30/TTTD-4-Rabin/Cipher 0.60 39.51 8.85 42.43 19.39
113 Custom-250-0/TTTD-16-Adler32/Bzip2-Cipher 0.39 41.58 8.97 37.53 19.40
114 Custom-125-0/TTTD-4-Adler32/Bzip2-Cipher 0.41 42.16 8.82 36.51 19.41
115 Custom-125-0/TTTD-8-Adler32/Bzip2-Cipher 0.41 41.98 8.90 36.97 19.43
116 Custom-125-0/TTTD-16-Adler32/Bzip2-Cipher 0.39 42.07 9.01 37.43 19.54
117 Custom-250-30/TTTD-4-PLAIN/Bzip2-Cipher 0.67 42.01 9.75 35.89 19.64
118 Custom-125-30/Fixed-4/Bzip2-Cipher 0.67 40.70 8.99 42.16 19.78
119 Custom-250-30/TTTD-8-PLAIN/Bzip2-Cipher 0.64 41.89 10.61 36.86 19.97
120 Custom-125-30/TTTD-8-Rabin/Cipher 0.57 43.54 8.72 40.67 20.32
121 Custom-250-30/TTTD-16-PLAIN/Bzip2-Cipher 0.69 41.53 11.07 40.51 20.44
122 Custom-125-30/TTTD-4-Rabin/Gzip-Cipher 0.67 41.79 9.40 44.14 20.45
123 Custom-250-30/TTTD-8-Adler32/Bzip2-Cipher 0.46 44.40 9.54 39.76 20.68
124 Custom-250-30/TTTD-4-Adler32/Bzip2-Cipher 0.46 45.33 9.43 38.12 20.73
125 Custom-125-30/TTTD-8-Rabin/Gzip-Cipher 0.62 43.96 9.28 41.78 20.76
126 Custom-250-30/TTTD-16-Adler32/Bzip2-Cipher 0.44 45.12 9.67 39.50 20.89
127 Custom-125-30/TTTD-4-PLAIN/Bzip2-Cipher 0.75 46.75 10.57 39.89 21.77
128 Custom-125-30/TTTD-8-PLAIN/Bzip2-Cipher 0.71 45.70 11.56 40.84 21.86
129 Custom-125-30/TTTD-16-PLAIN/Bzip2-Cipher 0.77 46.26 12.05 46.51 22.85
130 Custom-125-30/TTTD-4-Adler32/Bzip2-Cipher 0.49 49.71 10.15 43.68 22.89
131 Custom-125-30/TTTD-16-Adler32/Bzip2-Cipher 0.46 48.91 10.31 45.48 22.90
132 Custom-125-30/TTTD-8-Adler32/Bzip2-Cipher 0.45 49.20 10.23 45.19 22.93
133 Custom-250-0/TTTD-16-Rabin/Bzip2-Cipher 0.79 50.41 12.68 42.20 23.75
134 Custom-250-0/TTTD-8-Rabin/Bzip2-Cipher 0.89 50.44 12.32 44.22 23.92
135 Custom-125-0/TTTD-16-Rabin/Bzip2-Cipher 0.80 51.08 12.67 43.17 24.07
136 Custom-125-0/TTTD-8-Rabin/Bzip2-Cipher 0.91 50.91 12.35 45.47 24.22
137 Custom-250-0/TTTD-4-Rabin/Bzip2-Cipher 0.99 51.45 12.25 44.99 24.32
138 Custom-125-0/TTTD-4-Rabin/Bzip2-Cipher 0.99 52.05 12.32 47.36 24.80
139 Custom-250-30/TTTD-16-Rabin/Bzip2-Cipher 0.90 54.34 13.69 46.29 25.72
140 Custom-250-30/TTTD-8-Rabin/Bzip2-Cipher 1.02 54.32 13.31 47.99 25.84
141 Custom-250-30/TTTD-4-Rabin/Bzip2-Cipher 1.13 56.06 13.27 48.94 26.48
142 Custom-125-30/TTTD-16-Rabin/Bzip2-Cipher 0.99 57.88 14.60 53.20 27.87
143 Custom-125-30/TTTD-8-Rabin/Bzip2-Cipher 1.11 58.72 14.24 54.79 28.23
144 Custom-125-30/TTTD-4-Rabin/Bzip2-Cipher 1.26 60.13 14.27 56.16 28.86

Table E.1: Average overall duration over all datasets, ordered by the average.

>> Next chapter: Appendix “Best Algorithms by CPU Usage”

1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion		 A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography
Pages:<12 ... 1213 1415>

3 Comments

  1. JP

    Hi,

    I would love to see a ebook version of your thesis (epub or mobi). Would that be possible ?

    thanks



  3. Thiruven Madhavan

    Hi Philipp:
    Good Morning. Possible to receive pdf version of your thesis.
    cheers
    Madhavan