Deduplication layer on Filecoin

The pricing models of web2 storage providers like Dropbox or Goolge drive charge a user based on per GB basis. Imagine a scenario where you and your friend both subscribed to Dropbox and storing the same file on respective Dropboxes. In this case, both of you will get charged from the Dropbox. Although, Filecoin is a web3 solution, it is still following the pricing models of web2 storage providers. In our solution, if multiple users are storing the same file on Filecoin, they all can share the cost of storing the file and there will be only copy (excluding fault-tolerance requirement) of that file exsists in the network.

We have faced multiple issues:

• Market API Abstratction We initially though of using the mock deals provided by Market API, however we wanted to store in actual Filecoin testnet rather tham simulating it through mocks.
• No real-time storage deals on testnet We wanted fto ind and have deal with storage provider on testnet, but no such thing existed. Therefore, we migrated to using web3.storage for storing the files on Filecoin.
• Limited documention for making storage deal on either mainnet or testnet Migrated to web3.storage
• Different CIDs The CIDs produced by IPFS and web3.storage are different. We tired with different versions of CIDs, including changing the encoding functions but still not able to figure out why it is happening. We migrated to using only the CID provided by web3.storage.