Data Deduplication for general user in Application layer

Abstract

This project mainly focuses on the deduplication and create cloud storage client application. This project does not analyze new algorithm for deduplication chunking or focusing on deduplication performance, it just a comprehensive deduplication application with Nginx, HTTPS, API server and fully functional deduplication works for general user. The application works as a proxy between cloud storage client and cloud storage. All file upload is being deduplicated to binary data and uploads to the cloud storage. User benefits from reduces the storage usage on cloud storage. The client would work as same as a normal cloud storage client. We are only focusing on Dropbox cloud storage in this project. We implemented Fast CDC and fixed size chunking, Highway hash, Murmur3 hash and Sip hash for the hashing function, Rocks DB offers LSM tree and Bloom filter for hash store, compression with lz4, encryption with AES/CBC, custom simply chunk store format and finally chunk store compaction prevents fragmentation on delete chunks. This is all this project implemented for deduplication. For cloud storage client, we use ReactJS creates a Progressive Web App (PWA) which works in both web browser and mobile environment. We developed OAuth for Dropbox user authentication. This client only supports simple functions like create, retrieve, update, delete (CRUD) operations. It doesn't support function like content search and any other advanced feature in this stage. Finally, we create the API server in Nodejs and Express.js framework with HTTPS. It is a simple proxy server calling Dropbox API for file manipulation. We also use Redis for session control, SQLite for relational state storage.