My name is Philipp C. Heckel and I write about nerdy things.

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


  • May 20 / 2013
  • 3
Uncategorized

Minimizing remote storage usage and synchronization time using deduplication and multichunking: Syncany as an example


Contents

1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion		 A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography

Download as PDF: This article is a web version of my Master’s thesis. Feel free to download the original PDF version.

D. Best Algorithms by Deduplication Ratio

The following table lists the best algorithm configurations with regard to the deduplication ratio. The list is sorted by the average temporal deduplication ratio (last column). All values are deduplication ratios (ratio:1).

Rank Algorithm Configuration A B C D Average
1 Custom-250-*/TTTD-4-Adler32/Bzip2-Cipher 15.35 12.88 15.43 32.67 16.68
2 Custom-125-*/TTTD-4-Adler32/Bzip2-Cipher 15.33 12.84 15.40 30.92 16.45
3 Custom-250-*/TTTD-8-Adler32/Bzip2-Cipher 15.13 12.80 15.00 30.56 16.22
4 Custom-250-*/TTTD-4-Adler32/Gzip-Cipher 15.37 12.82 15.37 28.31 16.14
5 Custom-125-*/TTTD-4-Adler32/Gzip-Cipher 15.39 12.82 15.38 27.99 16.11
6 Custom-125-*/TTTD-8-Adler32/Bzip2-Cipher 15.12 12.75 14.96 29.00 16.01
7 Custom-250-*/TTTD-16-Adler32/Bzip2-Cipher 14.55 12.67 14.68 28.74 15.71
8 Custom-250-*/TTTD-8-Adler32/Gzip-Cipher 15.16 12.73 14.94 26.37 15.70
9 Custom-125-*/TTTD-8-Adler32/Gzip-Cipher 15.17 12.73 14.94 26.11 15.67
10 Custom-125-*/TTTD-16-Adler32/Bzip2-Cipher 14.53 12.63 14.66 27.29 15.51
11 Custom-250-*/TTTD-16-Adler32/Gzip-Cipher 14.57 12.59 14.61 24.68 15.19
12 Custom-125-*/TTTD-16-Adler32/Gzip-Cipher 14.58 12.59 14.63 24.44 15.17
13 Custom-250-*/TTTD-4-PLAIN/Bzip2-Cipher 6.73 12.21 12.72 34.19 13.33
14 Custom-125-*/TTTD-4-PLAIN/Bzip2-Cipher 6.72 12.15 12.68 32.33 13.08
15 Custom-250-*/TTTD-8-PLAIN/Bzip2-Cipher 6.79 12.14 11.75 32.91 12.90
16 Custom-250-*/TTTD-4-Adler32/Cipher 15.12 12.07 13.54 7.00 12.80
17 Custom-125-*/TTTD-4-Adler32/Cipher 15.12 12.07 13.54 7.00 12.80
18 Custom-250-*/TTTD-4-PLAIN/Gzip-Cipher 6.74 12.12 12.68 29.71 12.77
19 Custom-125-*/TTTD-4-PLAIN/Gzip-Cipher 6.74 12.12 12.67 29.37 12.73
20 Custom-125-*/TTTD-8-PLAIN/Bzip2-Cipher 6.78 12.09 11.71 31.09 12.65
21 Custom-250-*/TTTD-8-Adler32/Cipher 14.90 11.95 13.08 6.45 12.50
22 Custom-125-*/TTTD-8-Adler32/Cipher 14.90 11.95 13.07 6.45 12.50
23 Custom-250-*/TTTD-8-PLAIN/Gzip-Cipher 6.80 12.06 11.61 28.50 12.32
24 Custom-125-*/TTTD-8-PLAIN/Gzip-Cipher 6.80 12.05 11.61 28.13 12.27
25 Custom-250-*/Fixed-4/Bzip2-Cipher 6.64 12.13 11.39 27.90 12.15
26 Custom-250-*/Fixed-8/Bzip2-Cipher 6.61 12.16 11.41 27.76 12.14
27 Custom-250-*/TTTD-16-Adler32/Cipher 14.33 11.80 12.74 5.98 12.14
28 Custom-125-*/TTTD-16-Adler32/Cipher 14.33 11.80 12.74 5.98 12.14
29 Custom-250-*/Fixed-16/Bzip2-Cipher 6.54 12.16 11.41 27.17 12.05
30 Custom-125-*/Fixed-4/Bzip2-Cipher 6.63 12.08 11.35 26.39 11.94
31 Custom-125-*/Fixed-8/Bzip2-Cipher 6.60 12.10 11.37 26.23 11.93
32 Custom-250-*/TTTD-16-PLAIN/Bzip2-Cipher 6.39 12.16 11.37 26.26 11.89
33 Custom-125-*/Fixed-16/Bzip2-Cipher 6.53 12.11 11.37 25.76 11.86
34 Custom-250-*/TTTD-8-Rabin/Bzip2-Cipher 5.32 12.10 11.73 27.28 11.78
35 Custom-250-*/TTTD-16-Rabin/Bzip2-Cipher 6.09 12.10 11.11 26.82 11.77
36 Custom-125-*/TTTD-16-PLAIN/Bzip2-Cipher 6.38 12.10 11.34 24.98 11.71
37 Custom-250-*/Fixed-4/Gzip-Cipher 6.65 12.05 11.26 24.13 11.65
38 Custom-250-*/Fixed-8/Gzip-Cipher 6.62 12.07 11.28 23.90 11.63
39 Custom-125-*/Fixed-4/Gzip-Cipher 6.65 12.05 11.25 23.85 11.62
40 Custom-125-*/TTTD-16-Rabin/Bzip2-Cipher 6.07 12.05 11.08 25.51 11.59
41 Custom-125-*/Fixed-8/Gzip-Cipher 6.62 12.07 11.27 23.60 11.59
42 Custom-125-*/TTTD-8-Rabin/Bzip2-Cipher 5.30 12.05 11.69 25.87 11.59
43 Custom-250-*/Fixed-16/Gzip-Cipher 6.55 12.08 11.27 23.32 11.54
44 Custom-125-*/Fixed-16/Gzip-Cipher 6.56 12.07 11.27 23.08 11.51
45 Custom-250-*/TTTD-16-PLAIN/Gzip-Cipher 6.40 12.07 11.23 22.43 11.37
46 Custom-250-*/TTTD-4-Rabin/Bzip2-Cipher 4.02 12.12 11.47 27.50 11.36
47 Custom-125-*/TTTD-16-PLAIN/Gzip-Cipher 6.41 12.06 11.23 22.23 11.35
48 Custom-250-*/TTTD-8-Rabin/Gzip-Cipher 5.33 12.01 11.56 23.48 11.26
49 Custom-250-*/TTTD-16-Rabin/Gzip-Cipher 6.09 12.01 10.94 22.99 11.25
50 Custom-125-*/TTTD-8-Rabin/Gzip-Cipher 5.32 12.01 11.55 23.23 11.23
51 Custom-125-*/TTTD-16-Rabin/Gzip-Cipher 6.10 12.01 10.93 22.79 11.22
52 Custom-125-*/TTTD-4-Rabin/Bzip2-Cipher 4.01 12.07 11.43 26.02 11.15
53 Custom-250-*/TTTD-4-Rabin/Gzip-Cipher 4.03 12.04 11.34 23.77 10.85
54 Custom-125-*/TTTD-4-Rabin/Gzip-Cipher 4.03 12.03 11.33 23.49 10.82
55 Custom-250-*/TTTD-4-PLAIN/Cipher 6.66 11.44 11.30 7.37 9.51
56 Custom-125-*/TTTD-4-PLAIN/Cipher 6.66 11.44 11.30 7.37 9.51
57 Custom-250-*/TTTD-8-PLAIN/Cipher 6.71 11.31 10.10 6.99 9.09
58 Custom-125-*/TTTD-8-PLAIN/Cipher 6.71 11.31 10.10 6.99 9.09
59 Custom-250-*/Fixed-4/Cipher 6.58 11.33 9.82 5.92 8.85
60 Custom-125-*/Fixed-4/Cipher 6.58 11.33 9.82 5.91 8.85
61 Custom-250-*/Fixed-8/Cipher 6.54 11.34 9.80 5.79 8.82
62 Custom-125-*/Fixed-8/Cipher 6.54 11.34 9.80 5.79 8.82
63 Custom-250-*/Fixed-16/Cipher 6.47 11.32 9.78 5.62 8.77
64 Custom-125-*/Fixed-16/Cipher 6.47 11.32 9.78 5.62 8.77
65 Custom-250-*/TTTD-16-PLAIN/Cipher 6.31 11.28 9.71 5.37 8.66
66 Custom-125-*/TTTD-16-PLAIN/Cipher 6.31 11.28 9.71 5.37 8.66
67 Custom-250-*/TTTD-16-Rabin/Cipher 6.00 11.23 9.31 5.53 8.46
68 Custom-125-*/TTTD-16-Rabin/Cipher 6.00 11.23 9.31 5.53 8.46
69 Custom-250-*/TTTD-8-Rabin/Cipher 5.26 11.24 9.81 5.67 8.40
70 Custom-125-*/TTTD-8-Rabin/Cipher 5.26 11.24 9.81 5.67 8.40
71 Custom-250-*/TTTD-4-Rabin/Cipher 3.99 11.27 9.74 5.79 8.03
72 Custom-125-*/TTTD-4-Rabin/Cipher 3.98 11.27 9.74 5.79 8.03

Table D.1: Average temporal deduplication ratio over all datasets, ordered by the average.

>> Next chapter: Appendix “Best Algorithms by Duration”

1. Introduction
2. Related Work
3. Deduplication
4. Syncany
5. Implications of the Architecture
6. Experiments
7. Future Research
8. Conclusion		 A. List of Configurations
B. Pre-Study Folder Statistics
C. List of Variables Recorded
D. Best Algorithms by Deduplication Ratio
E. Best Algorithms by Duration
F. Best Algorithms by CPU Usage
Bibliography
Pages:<12 ... 1112 131415>

3 Comments

  1. JP

    Hi,

    I would love to see a ebook version of your thesis (epub or mobi). Would that be possible ?

    thanks



  3. Thiruven Madhavan

    Hi Philipp:
    Good Morning. Possible to receive pdf version of your thesis.
    cheers
    Madhavan