Multiple data import from text files

Nils_Junker

Board Regular
Joined
Jun 2, 2023
Messages
80
Office Version
  1. 365
Platform
  1. Windows
Hi everybody,

I got the following problem:

I need to get data from round about 500 text files.
Problem one is that I just need a part of each text file, (the part I need always stays the same)
Problem two is that I dont have enough time to manually convert all the text files one by one.

The following link shows all the files I want to include in my excel


Also how many lines can i insert in Excel?

If you need further Information, please send me an info!

Thanks for your help!
 
Put this into a seperate workbook.

All you have to do is change one line.

strFolder = "C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\"

This is the folder containing the text files.

Only the text files should be in this folder.

Remember the "\" at the end.

It takes about 25 minutes to run.

It splits the data between a number of worksheets.

VBA Code:
Public Sub subAppendDataFromTextFiles()
Dim fsoLibrary As FileSystemObject
Dim fsoFolder As Object
Dim sFileName As Object
Dim s As String
Dim arrFileName() As String
Dim dteStart As Date
Dim dteEnd As Date
Dim i As Integer
Dim WsProcess As Worksheet
Dim lngProcessLastRow As Long
Dim lngDestinationLastRow As Long
Dim rng As Range
Dim lngCount As Long
Dim Ws As Worksheet
Dim WsDestination As Worksheet
Dim WsLog As Worksheet
Dim strFolder As String
Dim intWorksheetUsedCount As Integer
Dim Wb As Workbook

' On Error GoTo Err_Handler
    
    strFolder = "C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\"

    ActiveWorkbook.Save

    intWorksheetUsedCount = 1
    
    Set Wb = ActiveWorkbook

    Application.ScreenUpdating = False

    On Error Resume Next
    Application.DisplayAlerts = False
    For Each Ws In ThisWorkbook.Worksheets
        If Ws.Name Like "ImportedData*" Or Ws.Name = "FileLog" Then
            Ws.Delete
        End If
    Next Ws
    Application.DisplayAlerts = True
    On Error GoTo 0
    
    Worksheets.Add after:=Worksheets(Sheets.Count)
    Set WsLog = ActiveSheet
    With WsLog
        .Name = "FileLog"
        .Range("A1:B1").Value = Array("File Name", "Count")
    End With
    
    intWorksheetUsedCount = 1
    Worksheets.Add after:=Worksheets(Sheets.Count)
    Set WsDestination = ActiveSheet
    With WsDestination
        .Name = "ImportedData" & intWorksheetUsedCount
        .Range("A1:D1").Value = Array("STATIONS_ID", "MESS_DATUM", "TT_10", "File Name")
    End With
    
    Set fsoLibrary = New FileSystemObject
    
    Set fsoFolder = fsoLibrary.GetFolder(strFolder)
    
    'Loop through each file in a folder.
    For Each sFileName In fsoFolder.Files
    
        On Error Resume Next
        Application.DisplayAlerts = False
        Wb.Worksheets("Process").Delete
        Application.DisplayAlerts = True
        On Error GoTo 0
            
        Workbooks.OpenText Filename:=sFileName, DataType:=xlDelimited, Semicolon:=True, DecimalSeparator:=",", ThousandsSeparator:="."
        
        With ActiveWorkbook
            .Sheets(1).Copy after:=Wb.Sheets(Wb.Sheets.Count)
            .Close
        End With
        
        Set WsProcess = Wb.Sheets(Sheets.Count)
         
        WsProcess.Name = "Process"
        
        WsProcess.Range("A1").CurrentRegion.Columns(2).NumberFormat = "00"
                
        WsProcess.Range("$C$1,$D$1,$F$1,$G$1,$H$1,$I$1").EntireColumn.Delete
        ' "STATIONS_ID", "MESS_DATUM" and "TT_10"
        
        ' Count the number of rows between 202305010010 and 202305312350.
        lngProcessLastRow = WsProcess.Cells(Rows.Count, 1).End(xlUp).Row
        Set rng = WsProcess.Range("B2:B" & lngProcessLastRow)
        lngCount = WorksheetFunction.CountIfs(rng, ">=" & 202305010010#, rng, "<=" & 202305312350#)
        
        If lngCount > 0 Then
        
            i = i + 1
            
            WsLog.Range("A" & WsLog.Cells(Rows.Count, 1).End(xlUp).Row + 1).Resize(1, 2).Value = Array(Replace(sFileName, strFolder, "", 1), lngCount)
                        
            lngDestinationLastRow = WsDestination.Cells(Rows.Count, 1).End(xlUp).Row
        
            ' Is there enough room to paste the data?
            If (lngDestinationLastRow + lngCount) > 500000 Then     ' WsDestination.Rows.Count Then
                Call subFormatWorksheet(WsDestination)
                intWorksheetUsedCount = intWorksheetUsedCount + 1
                Worksheets.Add after:=Worksheets(Sheets.Count)
                Set WsDestination = ActiveSheet
                WsDestination.Name = "ImportedData" & intWorksheetUsedCount
                WsDestination.Range("A1:D1").Value = Array("STATIONS_ID", "MESS_DATUM", "TT_10", "File Name")
                lngDestinationLastRow = 1
            End If
            
            If intWorksheetUsedCount >= 4000 Then
                Exit Sub
            End If
            
            With WsProcess
                .Range("$D$1").Value = "File Name"
                .Range("$D$2:$D" & lngProcessLastRow).Value = Replace(sFileName, strFolder, "", 1)
                .Range("$A$1:$D" & lngProcessLastRow).AutoFilter Field:=2, Criteria1:= _
                    ">=202305010010", Operator:=xlAnd, Criteria2:="<=202305312350"
                .Range("A2:D" & lngProcessLastRow).SpecialCells(xlCellTypeVisible).Copy
            End With
            
            WsDestination.Range("A" & lngDestinationLastRow + 1).PasteSpecial xlPasteValues
        
            Application.CutCopyMode = False
        
        End If
                     
    Next
    
    'Release the memory.
    Set fsoLibrary = Nothing
    Set fsoFolder = Nothing
    
    Call subFormatWorksheet(WsDestination)
    
    Call subFormatWorksheet(WsLog)
    
    Application.ScreenUpdating = True
    
    MsgBox "All files have been imported.", vbOKOnly, "Confirmation"
    
Exit_Handler:

    Exit Sub
    
Err_Handler:

    MsgBox Err.Number & vbCrLf & _
       Err.Description

    Resume Exit_Handler
    
End Sub

Private Sub subFormatWorksheet(Ws As Worksheet)

    With Ws.Range("A1").CurrentRegion
        
        .Columns(2).NumberFormat = "00"
        .Font.Size = 16
        .EntireColumn.AutoFit
        .RowHeight = 30
        .VerticalAlignment = xlCenter
        
        With .Rows(1)
            .Interior.Color = RGB(213, 213, 213)
            .Font.Bold = True
        End With
        
        With .Borders
            .LineStyle = xlContinuous
            .Weight = xlThin
            .ColorIndex = vbBlack
        End With
        
    End With

End Sub
 
Upvote 0

Excel Facts

Bring active cell back into view
Start at A1 and select to A9999 while writing a formula, you can't see A1 anymore. Press Ctrl+Backspace to bring active cell into view.
Put this into a seperate workbook.

All you have to do is change one line.

strFolder = "C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\"

This is the folder containing the text files.

Only the text files should be in this folder.

Remember the "\" at the end.

It takes about 25 minutes to run.

It splits the data between a number of worksheets.

VBA Code:
Public Sub subAppendDataFromTextFiles()
Dim fsoLibrary As FileSystemObject
Dim fsoFolder As Object
Dim sFileName As Object
Dim s As String
Dim arrFileName() As String
Dim dteStart As Date
Dim dteEnd As Date
Dim i As Integer
Dim WsProcess As Worksheet
Dim lngProcessLastRow As Long
Dim lngDestinationLastRow As Long
Dim rng As Range
Dim lngCount As Long
Dim Ws As Worksheet
Dim WsDestination As Worksheet
Dim WsLog As Worksheet
Dim strFolder As String
Dim intWorksheetUsedCount As Integer
Dim Wb As Workbook

' On Error GoTo Err_Handler
   
    strFolder = "C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\"

    ActiveWorkbook.Save

    intWorksheetUsedCount = 1
   
    Set Wb = ActiveWorkbook

    Application.ScreenUpdating = False

    On Error Resume Next
    Application.DisplayAlerts = False
    For Each Ws In ThisWorkbook.Worksheets
        If Ws.Name Like "ImportedData*" Or Ws.Name = "FileLog" Then
            Ws.Delete
        End If
    Next Ws
    Application.DisplayAlerts = True
    On Error GoTo 0
   
    Worksheets.Add after:=Worksheets(Sheets.Count)
    Set WsLog = ActiveSheet
    With WsLog
        .Name = "FileLog"
        .Range("A1:B1").Value = Array("File Name", "Count")
    End With
   
    intWorksheetUsedCount = 1
    Worksheets.Add after:=Worksheets(Sheets.Count)
    Set WsDestination = ActiveSheet
    With WsDestination
        .Name = "ImportedData" & intWorksheetUsedCount
        .Range("A1:D1").Value = Array("STATIONS_ID", "MESS_DATUM", "TT_10", "File Name")
    End With
   
    Set fsoLibrary = New FileSystemObject
   
    Set fsoFolder = fsoLibrary.GetFolder(strFolder)
   
    'Loop through each file in a folder.
    For Each sFileName In fsoFolder.Files
   
        On Error Resume Next
        Application.DisplayAlerts = False
        Wb.Worksheets("Process").Delete
        Application.DisplayAlerts = True
        On Error GoTo 0
           
        Workbooks.OpenText Filename:=sFileName, DataType:=xlDelimited, Semicolon:=True, DecimalSeparator:=",", ThousandsSeparator:="."
       
        With ActiveWorkbook
            .Sheets(1).Copy after:=Wb.Sheets(Wb.Sheets.Count)
            .Close
        End With
       
        Set WsProcess = Wb.Sheets(Sheets.Count)
        
        WsProcess.Name = "Process"
       
        WsProcess.Range("A1").CurrentRegion.Columns(2).NumberFormat = "00"
               
        WsProcess.Range("$C$1,$D$1,$F$1,$G$1,$H$1,$I$1").EntireColumn.Delete
        ' "STATIONS_ID", "MESS_DATUM" and "TT_10"
       
        ' Count the number of rows between 202305010010 and 202305312350.
        lngProcessLastRow = WsProcess.Cells(Rows.Count, 1).End(xlUp).Row
        Set rng = WsProcess.Range("B2:B" & lngProcessLastRow)
        lngCount = WorksheetFunction.CountIfs(rng, ">=" & 202305010010#, rng, "<=" & 202305312350#)
       
        If lngCount > 0 Then
       
            i = i + 1
           
            WsLog.Range("A" & WsLog.Cells(Rows.Count, 1).End(xlUp).Row + 1).Resize(1, 2).Value = Array(Replace(sFileName, strFolder, "", 1), lngCount)
                       
            lngDestinationLastRow = WsDestination.Cells(Rows.Count, 1).End(xlUp).Row
       
            ' Is there enough room to paste the data?
            If (lngDestinationLastRow + lngCount) > 500000 Then     ' WsDestination.Rows.Count Then
                Call subFormatWorksheet(WsDestination)
                intWorksheetUsedCount = intWorksheetUsedCount + 1
                Worksheets.Add after:=Worksheets(Sheets.Count)
                Set WsDestination = ActiveSheet
                WsDestination.Name = "ImportedData" & intWorksheetUsedCount
                WsDestination.Range("A1:D1").Value = Array("STATIONS_ID", "MESS_DATUM", "TT_10", "File Name")
                lngDestinationLastRow = 1
            End If
           
            If intWorksheetUsedCount >= 4000 Then
                Exit Sub
            End If
           
            With WsProcess
                .Range("$D$1").Value = "File Name"
                .Range("$D$2:$D" & lngProcessLastRow).Value = Replace(sFileName, strFolder, "", 1)
                .Range("$A$1:$D" & lngProcessLastRow).AutoFilter Field:=2, Criteria1:= _
                    ">=202305010010", Operator:=xlAnd, Criteria2:="<=202305312350"
                .Range("A2:D" & lngProcessLastRow).SpecialCells(xlCellTypeVisible).Copy
            End With
           
            WsDestination.Range("A" & lngDestinationLastRow + 1).PasteSpecial xlPasteValues
       
            Application.CutCopyMode = False
       
        End If
                    
    Next
   
    'Release the memory.
    Set fsoLibrary = Nothing
    Set fsoFolder = Nothing
   
    Call subFormatWorksheet(WsDestination)
   
    Call subFormatWorksheet(WsLog)
   
    Application.ScreenUpdating = True
   
    MsgBox "All files have been imported.", vbOKOnly, "Confirmation"
   
Exit_Handler:

    Exit Sub
   
Err_Handler:

    MsgBox Err.Number & vbCrLf & _
       Err.Description

    Resume Exit_Handler
   
End Sub

Private Sub subFormatWorksheet(Ws As Worksheet)

    With Ws.Range("A1").CurrentRegion
       
        .Columns(2).NumberFormat = "00"
        .Font.Size = 16
        .EntireColumn.AutoFit
        .RowHeight = 30
        .VerticalAlignment = xlCenter
       
        With .Rows(1)
            .Interior.Color = RGB(213, 213, 213)
            .Font.Bold = True
        End With
       
        With .Borders
            .LineStyle = xlContinuous
            .Weight = xlThin
            .ColorIndex = vbBlack
        End With
       
    End With

End Sub
Hi the first line of the code directly gives me an error
1687241507363.png

does my worksheet got to have a special name?

why do i have to change the line? in the code it is already what you described?
 
Upvote 0
Hi the first line of the code directly gives me an error
View attachment 93911
does my worksheet got to have a special name?

why do i have to change the line? in the code it is already what you described?
Where are you putting this code?

How are you runnning it?

You need to change the line to indicate where your text files are,.

Mine are in this folder : C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\
Yours are elsewhere.
 
Upvote 0
Solution
Where are you putting this code?

How are you runnning it?

You need to change the line to indicate where your text files are,.

Mine are in this folder : C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\
Yeah i did that already its also now working
but let me check if this is the data i need

But thanks already man!
 
Upvote 0
Where are you putting this code?

How are you runnning it?

You need to change the line to indicate where your text files are,.

Mine are in this folder : C:\Dump\Download Zip Files And Unzip\Downloads\Unzipped\
Yours are elsewhere.
Okay so thats exactly what i need.
but my runtime to download the files ist just a pair of seconds for 10 files thats perfect.

you really are the best man i know.

thanks a lot!!!!!!
i hope you're doing well
let me know if i can do anything to make your day better as you did mine!
 
Upvote 0

Forum statistics

Threads
1,215,606
Messages
6,125,811
Members
449,262
Latest member
hideto94

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top