Remove Duplicate Lines
Paste a list and keep only the first occurrence of each line, in the order you pasted it.
- lines in
- 1
- lines out
- 0
- removed
- 1
Output
Paste or type above to see the result here.
Your files stay on your device
This tool runs entirely in your browser. What you paste is processed on your device and is never sent to a server, so it is safe to use with tokens, keys and unpublished text.
Example
Example: merging two newsletter sign-up lists with stray spaces and capitals
A village hall committee merges the paper sign-up sheet from the summer fair with its online form: 10 lines, including two blank lines, an address with two spaces after it, and three people typed twice with different capitals (Olivia.Hart@, R.Kaur@Example.com) or exactly the same (j.mensah@).
- The removed count includes blank lines: of the 4 lines removed by default, 2 are empty, 1 is the repeated j.mensah@ and 1 is sam.price@ once its trailing spaces are trimmed.
- Addresses are compared letter for letter unless Ignore case is on. Mail services in practice deliver Olivia.Hart@ and olivia.hart@ to the same inbox, so tick it for email lists.
- The first spelling wins: R.Kaur@Example.com stays as typed and the lower-case copy further down goes.
| Settings | Lines in | Lines out | Removed | What is left |
|---|---|---|---|---|
| Default: Trim and Remove empty lines on | 10 | 6 | 4 | Olivia Hart and R. Kaur still twice, in two spellings |
| Ignore case on as well | 10 | 4 | 6 | olivia.hart@example.co.uk, j.mensah@example.com, sam.price@example.org, R.Kaur@Example.com |
| Trim off | 10 | 7 | 3 | sam.price@example.org twice: one copy ends in two spaces |
| Ignore case on, Remove empty lines off | 10 | 5 | 5 | the first blank line stays where it was |
Four addresses remain, in the order they first appeared, ready to paste into the BCC field or a mail-merge sheet.
How it works
- Paste your list with one item per line: email addresses, URLs, SKUs, keywords, log lines.
- Decide how strict the comparison should be. Ignore case treats Apple and apple as the same; Trim ignores spaces and tabs at the ends of each line.
- Leave Remove empty lines on to drop blank lines as well. Turned off, only the first blank line in the whole text stays: every later one counts as a duplicate of it, wherever it is.
- The result appears as you type. The output keeps the first occurrence of each line in its original position, and the count of removed lines is shown.
Frequently asked questions
- Does deduplication change the order of my lines?
- No. The first occurrence of each line stays where it was and later repeats are removed. If you want a sorted, unique list, use the Sort Lines tool with the Unique option instead.
- Are lines that differ only in spaces or capital letters treated as duplicates?
- Only if you enable the corresponding option. With Trim on (the default), spaces and tabs at the start and end of each line are ignored, and the lines come out trimmed. With Ignore case on, comparison is case-insensitive. Without those options, comparison is exact, character for character.
- How large a list can I paste?
- Hundreds of thousands of lines are fine in a modern browser; the work is done with a single pass and a hash set in memory. Very large files may be slow to paste into the textarea itself rather than slow to process.
- Is Windows CRLF line ending handled?
- Yes. Lines are split on \n, \r\n and lone \r, so text copied from Windows, macOS and Linux is treated the same. The output uses \n line endings.
- How do I remove duplicate email addresses from a list?
- Paste one address per line and turn on Ignore case, so Olivia.Hart@example.co.uk and olivia.hart@example.co.uk count as one. Keep Trim on so a space after an address does not make a copy look different. The first spelling is kept, in its place.
- Why does the removed count include blank lines?
- With Remove empty lines on, every blank line is dropped and counted as removed along with the repeats. Ten lines with two blank ones and two repeats show 4 removed; turn the option off to keep the first blank line.
- Is my list uploaded?
- No. Everything runs in your browser, so customer email lists and internal logs never leave your device.
Related tools
• Sort Lines Put a list in alphabetical, natural, length or random order without leaving the page. • Word Counter Live word, character, sentence and paragraph counts for essays, posts, ads and anything with a length limit. ≠ Text Diff Paste two versions of a text and see exactly which lines changed. • Case Converter Switch between UPPER, lower, Title, Sentence, camel, Pascal, snake and kebab case in one click.