My name is Philipp C. Heckel and I write about nerdy things.
This site moved here from blog.philippheckel.com/blog.heckel.xyz!

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


Cloud Computing, Distributed Systems, Security, Synchronization

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


Contents


1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion
A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography

Download as PDF: This article is a web version of my Master’s thesis. Feel free to download the original PDF version.


D. Best Algorithms by Deduplication Ratio

The following table lists the best algorithm configurations with regard to the deduplication ratio. The list is sorted by the average temporal deduplication ratio (last column). All values are deduplication ratios (ratio:1).

Rank Algorithm Configuration A B C D Average
1 Custom-250-*/TTTD-4-Adler32/Bzip2-Cipher 15.35 12.88 15.43 32.67 16.68
2 Custom-125-*/TTTD-4-Adler32/Bzip2-Cipher 15.33 12.84 15.40 30.92 16.45
3 Custom-250-*/TTTD-8-Adler32/Bzip2-Cipher 15.13 12.80 15.00 30.56 16.22
4 Custom-250-*/TTTD-4-Adler32/Gzip-Cipher 15.37 12.82 15.37 28.31 16.14
5 Custom-125-*/TTTD-4-Adler32/Gzip-Cipher 15.39 12.82 15.38 27.99 16.11
6 Custom-125-*/TTTD-8-Adler32/Bzip2-Cipher 15.12 12.75 14.96 29.00 16.01
7 Custom-250-*/TTTD-16-Adler32/Bzip2-Cipher 14.55 12.67 14.68 28.74 15.71
8 Custom-250-*/TTTD-8-Adler32/Gzip-Cipher 15.16 12.73 14.94 26.37 15.70
9 Custom-125-*/TTTD-8-Adler32/Gzip-Cipher 15.17 12.73 14.94 26.11 15.67
10 Custom-125-*/TTTD-16-Adler32/Bzip2-Cipher 14.53 12.63 14.66 27.29 15.51
11 Custom-250-*/TTTD-16-Adler32/Gzip-Cipher 14.57 12.59 14.61 24.68 15.19
12 Custom-125-*/TTTD-16-Adler32/Gzip-Cipher 14.58 12.59 14.63 24.44 15.17
13 Custom-250-*/TTTD-4-PLAIN/Bzip2-Cipher 6.73 12.21 12.72 34.19 13.33
14 Custom-125-*/TTTD-4-PLAIN/Bzip2-Cipher 6.72 12.15 12.68 32.33 13.08
15 Custom-250-*/TTTD-8-PLAIN/Bzip2-Cipher 6.79 12.14 11.75 32.91 12.90
16 Custom-250-*/TTTD-4-Adler32/Cipher 15.12 12.07 13.54 7.00 12.80
17 Custom-125-*/TTTD-4-Adler32/Cipher 15.12 12.07 13.54 7.00 12.80
18 Custom-250-*/TTTD-4-PLAIN/Gzip-Cipher 6.74 12.12 12.68 29.71 12.77
19 Custom-125-*/TTTD-4-PLAIN/Gzip-Cipher 6.74 12.12 12.67 29.37 12.73
20 Custom-125-*/TTTD-8-PLAIN/Bzip2-Cipher 6.78 12.09 11.71 31.09 12.65
21 Custom-250-*/TTTD-8-Adler32/Cipher 14.90 11.95 13.08 6.45 12.50
22 Custom-125-*/TTTD-8-Adler32/Cipher 14.90 11.95 13.07 6.45 12.50
23 Custom-250-*/TTTD-8-PLAIN/Gzip-Cipher 6.80 12.06 11.61 28.50 12.32
24 Custom-125-*/TTTD-8-PLAIN/Gzip-Cipher 6.80 12.05 11.61 28.13 12.27
25 Custom-250-*/Fixed-4/Bzip2-Cipher 6.64 12.13 11.39 27.90 12.15
26 Custom-250-*/Fixed-8/Bzip2-Cipher 6.61 12.16 11.41 27.76 12.14
27 Custom-250-*/TTTD-16-Adler32/Cipher 14.33 11.80 12.74 5.98 12.14
28 Custom-125-*/TTTD-16-Adler32/Cipher 14.33 11.80 12.74 5.98 12.14
29 Custom-250-*/Fixed-16/Bzip2-Cipher 6.54 12.16 11.41 27.17 12.05
30 Custom-125-*/Fixed-4/Bzip2-Cipher 6.63 12.08 11.35 26.39 11.94
31 Custom-125-*/Fixed-8/Bzip2-Cipher 6.60 12.10 11.37 26.23 11.93
32 Custom-250-*/TTTD-16-PLAIN/Bzip2-Cipher 6.39 12.16 11.37 26.26 11.89
33 Custom-125-*/Fixed-16/Bzip2-Cipher 6.53 12.11 11.37 25.76 11.86
34 Custom-250-*/TTTD-8-Rabin/Bzip2-Cipher 5.32 12.10 11.73 27.28 11.78
35 Custom-250-*/TTTD-16-Rabin/Bzip2-Cipher 6.09 12.10 11.11 26.82 11.77
36 Custom-125-*/TTTD-16-PLAIN/Bzip2-Cipher 6.38 12.10 11.34 24.98 11.71
37 Custom-250-*/Fixed-4/Gzip-Cipher 6.65 12.05 11.26 24.13 11.65
38 Custom-250-*/Fixed-8/Gzip-Cipher 6.62 12.07 11.28 23.90 11.63
39 Custom-125-*/Fixed-4/Gzip-Cipher 6.65 12.05 11.25 23.85 11.62
40 Custom-125-*/TTTD-16-Rabin/Bzip2-Cipher 6.07 12.05 11.08 25.51 11.59
41 Custom-125-*/Fixed-8/Gzip-Cipher 6.62 12.07 11.27 23.60 11.59
42 Custom-125-*/TTTD-8-Rabin/Bzip2-Cipher 5.30 12.05 11.69 25.87 11.59
43 Custom-250-*/Fixed-16/Gzip-Cipher 6.55 12.08 11.27 23.32 11.54
44 Custom-125-*/Fixed-16/Gzip-Cipher 6.56 12.07 11.27 23.08 11.51
45 Custom-250-*/TTTD-16-PLAIN/Gzip-Cipher 6.40 12.07 11.23 22.43 11.37
46 Custom-250-*/TTTD-4-Rabin/Bzip2-Cipher 4.02 12.12 11.47 27.50 11.36
47 Custom-125-*/TTTD-16-PLAIN/Gzip-Cipher 6.41 12.06 11.23 22.23 11.35
48 Custom-250-*/TTTD-8-Rabin/Gzip-Cipher 5.33 12.01 11.56 23.48 11.26
49 Custom-250-*/TTTD-16-Rabin/Gzip-Cipher 6.09 12.01 10.94 22.99 11.25
50 Custom-125-*/TTTD-8-Rabin/Gzip-Cipher 5.32 12.01 11.55 23.23 11.23
51 Custom-125-*/TTTD-16-Rabin/Gzip-Cipher 6.10 12.01 10.93 22.79 11.22
52 Custom-125-*/TTTD-4-Rabin/Bzip2-Cipher 4.01 12.07 11.43 26.02 11.15
53 Custom-250-*/TTTD-4-Rabin/Gzip-Cipher 4.03 12.04 11.34 23.77 10.85
54 Custom-125-*/TTTD-4-Rabin/Gzip-Cipher 4.03 12.03 11.33 23.49 10.82
55 Custom-250-*/TTTD-4-PLAIN/Cipher 6.66 11.44 11.30 7.37 9.51
56 Custom-125-*/TTTD-4-PLAIN/Cipher 6.66 11.44 11.30 7.37 9.51
57 Custom-250-*/TTTD-8-PLAIN/Cipher 6.71 11.31 10.10 6.99 9.09
58 Custom-125-*/TTTD-8-PLAIN/Cipher 6.71 11.31 10.10 6.99 9.09
59 Custom-250-*/Fixed-4/Cipher 6.58 11.33 9.82 5.92 8.85
60 Custom-125-*/Fixed-4/Cipher 6.58 11.33 9.82 5.91 8.85
61 Custom-250-*/Fixed-8/Cipher 6.54 11.34 9.80 5.79 8.82
62 Custom-125-*/Fixed-8/Cipher 6.54 11.34 9.80 5.79 8.82
63 Custom-250-*/Fixed-16/Cipher 6.47 11.32 9.78 5.62 8.77
64 Custom-125-*/Fixed-16/Cipher 6.47 11.32 9.78 5.62 8.77
65 Custom-250-*/TTTD-16-PLAIN/Cipher 6.31 11.28 9.71 5.37 8.66
66 Custom-125-*/TTTD-16-PLAIN/Cipher 6.31 11.28 9.71 5.37 8.66
67 Custom-250-*/TTTD-16-Rabin/Cipher 6.00 11.23 9.31 5.53 8.46
68 Custom-125-*/TTTD-16-Rabin/Cipher 6.00 11.23 9.31 5.53 8.46
69 Custom-250-*/TTTD-8-Rabin/Cipher 5.26 11.24 9.81 5.67 8.40
70 Custom-125-*/TTTD-8-Rabin/Cipher 5.26 11.24 9.81 5.67 8.40
71 Custom-250-*/TTTD-4-Rabin/Cipher 3.99 11.27 9.74 5.79 8.03
72 Custom-125-*/TTTD-4-Rabin/Cipher 3.98 11.27 9.74 5.79 8.03

Table D.1: Average temporal deduplication ratio over all datasets, ordered by the average.

>> Next chapter: Appendix “Best Algorithms by Duration”


1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion
A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography

Pages:<12 ... 1112 131415>

3 Comments

  1. JP

    Hi,

    I would love to see a ebook version of your thesis (epub or mobi). Would that be possible ?

    thanks



  2. Thiruven Madhavan

    Hi Philipp:
    Good Morning. Possible to receive pdf version of your thesis.
    cheers
    Madhavan


Leave a comment

I'd very much like to hear what you think of this post. Feel free to leave a comment. I usually respond within a day or two, sometimes even faster. I will not share or publish your e-mail address anywhere.