Data Deduplication File Systems

Nicholas Leippe nick at leippe.com
Mon Feb 25 00:27:27 MST 2013


On Sun, Feb 24, 2013 at 11:45 PM, DANIEL DAVID EGLI
<ddavidegli at gmail.com> wrote:
> *Question for you guys. Someone recently told me about a new set of Linux
> file systems that include data-deduplication. They sent me an old article
> from Linux Journal before they stopped printing it and went web only. The
> article was very informative, except in one aspect. It only dealt with one
> file system, called lessfs, which (at least the way they showed it) could
> not be mounted via fstab. You had to run a separate binary to mount it. Has
> anyone heard of any other file systems that implement data deuplication?
> I'd be curious to check it out. I think it could be especially handy on my
> MythTV box because when you get all these commercials over and over the
> system would only need to store the entire video sequence for the
> commercial once. That's handy because as an example I was watching
> Underworld on spike Saturday night and must have seen the same commercials
> for places like Burger King over and over. If I'm going to record shows I
> like the idea of not having to save that space. Yes, on a multi-terabyte
> disk drive space is probably not going to be an issue. But I'd at least
> like to try it out anyway, IF I can find a file system that includes
> data-deduplication and can be mounted at boot via fstab/rc.sysinit.*
>
> *--- Dan*

I wasn't aware that mythtv cut out commercials and recorded them as
separate files.
If so:
1) why not simply *delete* them?
2) wouldn't it have to also record the *exact* same file contents? I
suppose there may be an mpeg-diff that could determine that 99% of two
files are the same video content. If not, could be a good project for
someone.


More information about the PLUG mailing list