Remove Duplicate Lines
Paste a list and keep only the first occurrence of each line, in the order you pasted it.
- lines in
- 1
- lines out
- 0
- removed
- 1
Output
Paste or type above to see the result here.
Your files stay on your device
This tool runs entirely in your browser. What you paste is processed on your device and is never sent to a server, so it is safe to use with tokens, keys and unpublished text.
How it works
- Paste your list with one item per line: email addresses, URLs, SKUs, keywords, log lines.
- Decide how strict the comparison should be. Ignore case treats Apple and apple as the same; Trim ignores spaces and tabs at the ends of each line.
- Leave Remove empty lines on to drop blank lines as well, or turn it off to keep one blank line where several were.
- Press Remove duplicates. The output keeps the first occurrence of each line in its original position, and the count of removed lines is shown.
Frequently asked questions
- Does deduplication change the order of my lines?
- No. The first occurrence of each line stays where it was and later repeats are removed. If you want a sorted, unique list, use the Sort Lines tool with the Unique option instead.
- Are lines that differ only in spaces or capital letters treated as duplicates?
- Only if you enable the corresponding option. With Trim on, trailing spaces and tabs are ignored. With Ignore case on, comparison is case-insensitive. Without those options, comparison is exact, character for character.
- How large a list can I paste?
- Hundreds of thousands of lines are fine in a modern browser; the work is done with a single pass and a hash set in memory. Very large files may be slow to paste into the textarea itself rather than slow to process.
- Is Windows CRLF line ending handled?
- Yes. Lines are split on \n, \r\n and lone \r, so text copied from Windows, macOS and Linux is treated the same. The output uses \n line endings.
- Is my list uploaded?
- No. Everything runs in your browser, so customer email lists and internal logs never leave your device.
Related tools
• Sort Lines Put a list in alphabetical, natural, length or random order without leaving the page. • Word Counter Live word, character, sentence and paragraph counts for essays, posts, ads and anything with a length limit. • Text Diff Paste two versions of a text and see exactly which lines changed. • Case Converter Switch between UPPER, lower, Title, Sentence, camel, Pascal, snake and kebab case in one click.