remove duplicate lines in text file

yinkajewole

Board Regular
Joined
Nov 23, 2018
Messages
212
how can i use a macro to remove duplicate lines in my text file such that the last one will remain, for instance
the boy is good
great thinking
great thinking
he lives
the boy is good
what's up
great thinking

it should now be like this
he lives
the boy is good
what's up
great thinking
 
Last edited:

Domenic

MrExcel MVP
Joined
Mar 10, 2004
Messages
19,008
Since duplicate lines are going to be deleted, it shouldn't matter which instance should remain. Or am I missing something?
 

MARK858

MrExcel MVP
Joined
Nov 12, 2010
Messages
11,487
Office Version
365, 2010
Platform
Windows, Mobile
Maybe....
Code assumes that your data is in column A and you have a header in A1.

Code:
Sub DelDupsKeepLast()

    Dim myRng As Range, myCell As Range
    Dim RngDel As Range, LstRw As Long

    Set myRng = Range("A2:A" & Range("A" & Rows.Count).End(xlUp).Row)
    LstRw = Range("A" & Rows.Count).End(xlUp).Row

    For Each myCell In myRng

        If myCell.Row < LstRw Then
            If Not myCell.Offset(1, 0).Resize(LstRw - myCell.Row).Find(What:=myCell.Value, Lookat:=xlWhole) Is Nothing Then
                If RngDel Is Nothing Then
                    Set RngDel = myCell
                Else
                    Set RngDel = Application.Union(RngDel, myCell)
                End If
            End If
        End If

    Next myCell

    If Not RngDel Is Nothing Then RngDel.EntireRow.Delete

End Sub
 

yinkajewole

Board Regular
Joined
Nov 23, 2018
Messages
212
Maybe....
Code assumes that your data is in column A and you have a header in A1.

Code:
Sub DelDupsKeepLast()

    Dim myRng As Range, myCell As Range
    Dim RngDel As Range, LstRw As Long

    Set myRng = Range("A2:A" & Range("A" & Rows.Count).End(xlUp).Row)
    LstRw = Range("A" & Rows.Count).End(xlUp).Row

    For Each myCell In myRng

        If myCell.Row < LstRw Then
            If Not myCell.Offset(1, 0).Resize(LstRw - myCell.Row).Find(What:=myCell.Value, Lookat:=xlWhole) Is Nothing Then
                If RngDel Is Nothing Then
                    Set RngDel = myCell
                Else
                    Set RngDel = Application.Union(RngDel, myCell)
                End If
            End If
        End If

    Next myCell

    If Not RngDel Is Nothing Then RngDel.EntireRow.Delete

End Sub
the data is not in excel, it's in a text file (.txt)
 

MARK858

MrExcel MVP
Joined
Nov 12, 2010
Messages
11,487
Office Version
365, 2010
Platform
Windows, Mobile
You do realize this is an Excel site? Import the data into Excel run the code and save as a text file.
 

MARK858

MrExcel MVP
Joined
Nov 12, 2010
Messages
11,487
Office Version
365, 2010
Platform
Windows, Mobile
No, you can't remove duplicates while still in a text file, what is so difficult about importing it into Excel and then saving it?
 

Eric W

MrExcel MVP
Joined
Aug 18, 2015
Messages
8,981
Try this on a COPY of your text file. It asks for the file name, opens it, reads it, processes the data, and writes back the results to the same file.

Rich (BB code):
Sub RemoveDups()
Dim FileName As String, MyData As String, MyResult As String, i As Long, MyLines As Variant
Dim Kept As Long, Lost As Long

    FileName = Application.GetOpenFilename
    
    Open FileName For Input As #1 
    MyData = Input(LOF(1), 1)
    Close #1 
    
    MyResult = vbCrLf
    MyLines = Split(MyData, vbCrLf)
    For i = UBound(MyLines) To 0 Step -1
        If InStr(1, MyResult, vbCrLf & MyLines(i) & vbCrLf, vbTextCompare) = 0 Then
            MyResult = vbCrLf & MyLines(i) & MyResult
            Kept = Kept + 1
        Else
            Lost = Lost + 1
        End If
    Next i
    
    Open FileName For Output As #1 
    Print #1 , Mid(MyResult, 2)
    Close #1 
    
    MsgBox Kept & " lines were kept" & vbCrLf & Lost & " lines were removed"
    
End Sub
 

yinkajewole

Board Regular
Joined
Nov 23, 2018
Messages
212
Try this on a COPY of your text file. It asks for the file name, opens it, reads it, processes the data, and writes back the results to the same file.

Rich (BB code):
Sub RemoveDups()
Dim FileName As String, MyData As String, MyResult As String, i As Long, MyLines As Variant
Dim Kept As Long, Lost As Long

    FileName = Application.GetOpenFilename
    
    Open FileName For Input As #1 
    MyData = Input(LOF(1), 1)
    Close #1 
    
    MyResult = vbCrLf
    MyLines = Split(MyData, vbCrLf)
    For i = UBound(MyLines) To 0 Step -1
        If InStr(1, MyResult, vbCrLf & MyLines(i) & vbCrLf, vbTextCompare) = 0 Then
            MyResult = vbCrLf & MyLines(i) & MyResult
            Kept = Kept + 1
        Else
            Lost = Lost + 1
        End If
    Next i
    
    Open FileName For Output As #1 
    Print #1 , Mid(MyResult, 2)
    Close #1 
    
    MsgBox Kept & " lines were kept" & vbCrLf & Lost & " lines were removed"
    
End Sub
this is too real to be true... Exactly what I was looking for. Many thanks you all
 

Forum statistics

Threads
1,082,066
Messages
5,362,986
Members
400,704
Latest member
mhan

Some videos you may like

This Week's Hot Topics

  • populate from drop list with multiple tables
    Hi All, i have a drop list that displays data, what i want is when i select one of those from the list to populate text from different tables on...
  • Find list of words from sheet2 in sheet1 before a comma and extract text vba
    Hi Friends, Trying to find the solution on my task. But did not find suitable one to the need. Here is my query and sample file with details...
  • Dynamic Formula entry - VBA code sought
    Hello, really hope one of you experts can help with this - i've spent hours on this and getting no-where. .I have a set of data (more rows than...
  • Listbox Header
    Have a named range called "AccidentsHeader" Within my code I have: [CODE]Private Sub CommandButton1_Click() ListBox1.RowSource =...
  • Complex Heat Map using conditional formatting
    Good day excel world. I have a concern. Below link have a list of countries that carries each country unique data. [URL...
  • Conditional formatting
    Hi good morning, hope you can help me please, I have cells P4:P54 and if this cell is equal to 1 then i want row O to say "Fully Utilised" and to...
Top