Remove Duplicate Lines
Remove repeated lines from text while keeping the first occurrence.
Written by Anil Gautam · Updated
When are two lines treated as duplicates?
Only an exact, character-for-character match counts. The tool splits your text at each line break, keeps the first copy of every distinct line and drops the later ones. Capital letters, spaces at either end and invisible characters all make lines different, so Apple, apple and apple followed by a space are three separate lines. Windows line endings are handled, so a list exported from a Windows program behaves the same way.
Take a six-line email list: anna@example.com, ben@example.com, Anna@example.com, anna@example.com, ben@example.com with a trailing space, and cara@example.com. The output has five lines. Only the second anna@example.com is gone, while Anna@example.com and the padded ben@example.com both stay.
Why did the blank lines between my paragraphs disappear?
An empty line is a line like any other, so only the first one survives. Paste three paragraphs separated by blank lines and the output keeps the gap after the first paragraph, while the second and third end up on consecutive lines. A line holding only spaces is distinct from an empty one and is kept once as well. When the spacing matters, dedupe the list part on its own and put the surrounding paragraphs back afterwards.
Why does my list still show repeats after cleaning?
Lines that still look alike differ in at least one character, such as a capital letter, a trailing space or an invisible character. Duplicate Item Counter helps you find them: it lists each line with its count and has switches to trim spaces, ignore blank lines and turn case sensitivity off. With trimming on and case sensitivity off, the six-line email list above shrinks to 3 unique items. Switch Trim spaces and Ignore blank lines off and leave Case sensitive on, and its Total lines and Unique items figures match the lines you pasted here and the lines this tool keeps.
Sort Lines Alphabetically has Trim whitespace and Remove duplicates options that catch copies with stray spaces, though sorting changes the order. To drop duplicates that differ only in case while keeping the order, convert the list to lowercase with Case Converter first and paste the result here. Every line ends up lowercase, so this suits data where case carries no meaning, such as a list of domain names. For two separate lists, Compare Lists shows what is in one list only or in both, and its Send to menu can pass a result straight into this tool. The whole check runs in your browser in one pass over the lines, and nothing is uploaded.