remove duplicate lines in text file

yinkajewole

Active Member
Joined
Nov 23, 2018
Messages
281
how can i use a macro to remove duplicate lines in my text file such that the last one will remain, for instance
the boy is good
great thinking
great thinking
he lives
the boy is good
what's up
great thinking

it should now be like this
he lives
the boy is good
what's up
great thinking
 
Last edited:

Excel Facts

Test for Multiple Conditions in IF?
Use AND(test, test, test, test) or OR(test, test, test, ...) as the logical_test argument of IF.
Since duplicate lines are going to be deleted, it shouldn't matter which instance should remain. Or am I missing something?
 
Upvote 0
Maybe....
Code assumes that your data is in column A and you have a header in A1.

Code:
Sub DelDupsKeepLast()

    Dim myRng As Range, myCell As Range
    Dim RngDel As Range, LstRw As Long

    Set myRng = Range("A2:A" & Range("A" & Rows.Count).End(xlUp).Row)
    LstRw = Range("A" & Rows.Count).End(xlUp).Row

    For Each myCell In myRng

        If myCell.Row < LstRw Then
            If Not myCell.Offset(1, 0).Resize(LstRw - myCell.Row).Find(What:=myCell.Value, Lookat:=xlWhole) Is Nothing Then
                If RngDel Is Nothing Then
                    Set RngDel = myCell
                Else
                    Set RngDel = Application.Union(RngDel, myCell)
                End If
            End If
        End If

    Next myCell

    If Not RngDel Is Nothing Then RngDel.EntireRow.Delete

End Sub
 
Upvote 0
Maybe....
Code assumes that your data is in column A and you have a header in A1.

Code:
Sub DelDupsKeepLast()

    Dim myRng As Range, myCell As Range
    Dim RngDel As Range, LstRw As Long

    Set myRng = Range("A2:A" & Range("A" & Rows.Count).End(xlUp).Row)
    LstRw = Range("A" & Rows.Count).End(xlUp).Row

    For Each myCell In myRng

        If myCell.Row < LstRw Then
            If Not myCell.Offset(1, 0).Resize(LstRw - myCell.Row).Find(What:=myCell.Value, Lookat:=xlWhole) Is Nothing Then
                If RngDel Is Nothing Then
                    Set RngDel = myCell
                Else
                    Set RngDel = Application.Union(RngDel, myCell)
                End If
            End If
        End If

    Next myCell

    If Not RngDel Is Nothing Then RngDel.EntireRow.Delete

End Sub

the data is not in excel, it's in a text file (.txt)
 
Upvote 0
You do realize this is an Excel site? Import the data into Excel run the code and save as a text file.
 
Upvote 0
No, you can't remove duplicates while still in a text file, what is so difficult about importing it into Excel and then saving it?
 
Upvote 0
Try this on a COPY of your text file. It asks for the file name, opens it, reads it, processes the data, and writes back the results to the same file.

Rich (BB code):
Sub RemoveDups()
Dim FileName As String, MyData As String, MyResult As String, i As Long, MyLines As Variant
Dim Kept As Long, Lost As Long

    FileName = Application.GetOpenFilename
    
    Open FileName For Input As #1 
    MyData = Input(LOF(1), 1)
    Close #1 
    
    MyResult = vbCrLf
    MyLines = Split(MyData, vbCrLf)
    For i = UBound(MyLines) To 0 Step -1
        If InStr(1, MyResult, vbCrLf & MyLines(i) & vbCrLf, vbTextCompare) = 0 Then
            MyResult = vbCrLf & MyLines(i) & MyResult
            Kept = Kept + 1
        Else
            Lost = Lost + 1
        End If
    Next i
    
    Open FileName For Output As #1 
    Print #1 , Mid(MyResult, 2)
    Close #1 
    
    MsgBox Kept & " lines were kept" & vbCrLf & Lost & " lines were removed"
    
End Sub
 
Upvote 0
Try this on a COPY of your text file. It asks for the file name, opens it, reads it, processes the data, and writes back the results to the same file.

Rich (BB code):
Sub RemoveDups()
Dim FileName As String, MyData As String, MyResult As String, i As Long, MyLines As Variant
Dim Kept As Long, Lost As Long

    FileName = Application.GetOpenFilename
    
    Open FileName For Input As #1 
    MyData = Input(LOF(1), 1)
    Close #1 
    
    MyResult = vbCrLf
    MyLines = Split(MyData, vbCrLf)
    For i = UBound(MyLines) To 0 Step -1
        If InStr(1, MyResult, vbCrLf & MyLines(i) & vbCrLf, vbTextCompare) = 0 Then
            MyResult = vbCrLf & MyLines(i) & MyResult
            Kept = Kept + 1
        Else
            Lost = Lost + 1
        End If
    Next i
    
    Open FileName For Output As #1 
    Print #1 , Mid(MyResult, 2)
    Close #1 
    
    MsgBox Kept & " lines were kept" & vbCrLf & Lost & " lines were removed"
    
End Sub

this is too real to be true... Exactly what I was looking for. Many thanks you all
 
Upvote 0

Forum statistics

Threads
1,213,521
Messages
6,114,104
Members
448,548
Latest member
harryls

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top