Remove x number of lines from beginning of file

Jeff Anderson jefferya at programmerq.net
Thu Oct 25 16:40:24 MDT 2007


The only way that I would know of to accomplish something like that
would be to move/change the file header on disk to reflect the new state
of the file. It would mark the bits representing the first 10 lines as
free space.

I am not aware of a utility that does this, but if you were to do this
operation several times a second, the time involved in writing, testing,
and debugging a low level utility might be worth it.

Jeff Anderson

Jon Gale wrote:
> PLUGers,
>
> What would be the most efficient way to remove x number of lines from a very
> large (many gigabyte) file?
>
> Preliminary tests show that sed is not a good option:
>
> [root at linuxboy root]# time sed -i -e '1,4d' test2.txt
>
> real    0m0.016s
> user    0m0.000s
> sys     0m0.010s
> [root at linuxboy root]# time sed -i -e '1,4d' test.txt
>
> real    1m13.596s
> user    0m18.020s
> sys     0m52.930s
>
> test2.txt is only 100 lines.  test.txt is 10,000,000 lines.
>
> Is there a better way to do this that won't necessitate scanning the entire
> file just to remove the first x number of lines?
>
> Thanks,
>
> Jon
>
>   


-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 252 bytes
Desc: OpenPGP digital signature
Url : http://plug.org/pipermail/plug/attachments/20071025/7c637d9f/attachment.bin 


More information about the PLUG mailing list