SQL Insert Yields Fewer Records Than Anticipated

goss

Active Member
Joined
Feb 2, 2004
Messages
372
Hi all,

My SQL INSERT snippet is only yielding 28,739 records. However there are 421,956 record on my Excel worksheet:
Code:
    'Load to Access table
        'SQL
            strSQL9 = "INSERT INTO tmp_Table SELECT * FROM [tmpf$] IN '" _
            & ThisWorkbook.FullName & "' 'Excel 8.0;'"
            
        'Do it
            cnt.Execute (strSQL9)

Any idea why this may be?
Full code below
thx
w
Code:
Option Explicit

Sub ImportRecords()
    '
    'Imports all records from the specified table
    'Uses Microsoft ActiveX Data Objects 2.7 Library
    '
    'Date       Developer       Action
    '---------------------------------------------
    '01/20/12   ws              Created
    '01/20/12   ws              Temorary worksheet delete process is commented out
   
    Dim wb As Workbook
    Dim ws As Worksheet
    Dim wstmp As Worksheet
    Dim wstmpf As Worksheet
    Dim wsTitle As Worksheet
    Dim strDBPath As String
    Dim strDb As String
    Dim strDBTable As String
    Dim strDBPathFile As String
    Dim cnt As ADODB.Connection
    Dim rst1, rst2, rst3, rst4, rst5, rst6, rst7, rst8 As ADODB.Recordset
    Dim strSQL1, strSQL2, strSQL3, strSQL4, strSQL5, strSQL6, strSQL7, strSQL8, strSQL9 As String
    Dim stConn As String
    Dim strFormula As String
    Dim lngRows As Long
    
    'Initialize
        With Application
            .ScreenUpdating = False
            .Calculation = xlCalculationManual
            .DisplayAlerts = False
        End With
        lngRows = 0
        strFormula = "=C2&""_""&A2"
      
    'Instantiate objects
        Set cnt = New ADODB.Connection
        Set rst1 = New ADODB.Recordset
        Set rst2 = New ADODB.Recordset
        Set rst3 = New ADODB.Recordset
        Set rst4 = New ADODB.Recordset
        Set rst5 = New ADODB.Recordset
        Set rst6 = New ADODB.Recordset
        Set rst7 = New ADODB.Recordset
        Set rst8 = New ADODB.Recordset
        Set wb = ThisWorkbook
        Set ws = wb.Worksheets("ImportRecs")
        Set wsTitle = wb.Worksheets("iTitle")
        
    'Temprary worksheets to handle data
        wb.Worksheets.Add(After:=Worksheets(Worksheets.Count)).Name = "tmp"
        wb.Worksheets.Add(After:=Worksheets(Worksheets.Count)).Name = "tmpf"
        Set wstmp = wb.Worksheets("tmp")
        Set wstmpf = wb.Worksheets("tmpf")

    'Get database string values
        With ws
            strDBPath = .Range("C4")
            strDb = .Range("C5")
            strDBTable = .Range("C6")
        End With
        
    ' Get the database name.
        strDBPathFile = strDBPath
        If Right$(strDBPathFile, 1) <> "\" Then strDBPathFile = strDBPathFile & _
            "\"
        strDBPathFile = strDBPathFile & strDb

    'Connection String
        stConn = "Provider=Microsoft.Jet.OLEDB.4.0;" _
        & "Data Source=" & strDBPathFile & ";"
    
    'SQL-statements to be executed.
        strSQL1 = " SELECT OrderID FROM Orders "
        strSQL2 = " SELECT CustomerID FROM Orders "
        strSQL3 = " SELECT OrderDate FROM Orders "
        strSQL4 = " SELECT ShipName FROM Orders "
        strSQL5 = " SELECT CompanyName" & _
                 " FROM Customers " & _
                 " LEFT OUTER JOIN Orders " & _
                 " ON Customers.CustomerID " & _
                 " = Orders.CustomerID"
        strSQL6 = " SELECT ShipPostalCode FROM Orders "
        strSQL7 = " SELECT ContactName" & _
                 " FROM Customers " & _
                 " LEFT OUTER JOIN Orders " & _
                 " ON Customers.CustomerID " & _
                 " = Orders.OrdersID"
        strSQL8 = " SELECT Status FROM Orders "
        
    With cnt
        .Open (stConn) 'Open the connection.
        .CursorLocation = adUseClient 'Necessary to disconnect the recordset.
    End With
        
    With rst1
        .Open strSQL1, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst2
        .Open strSQL2, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst3
        .Open strSQL3, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst4
        .Open strSQL4, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst5
        .Open strSQL5, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst6
        .Open strSQL6, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst7
        .Open strSQL7, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With
    
    With rst8
        .Open strSQL8, cnt 'Create the recordset.
        Set .ActiveConnection = Nothing 'Disconnect the recordset.
    End With

        
    'Copy recordsets
        With wstmp
            .Cells(2, 1).CopyFromRecordset rst1
            .Cells(2, 2).CopyFromRecordset rst2
            .Cells(2, 3).CopyFromRecordset rst3
            .Cells(2, 4).CopyFromRecordset rst4
            .Cells(2, 5).CopyFromRecordset rst5
            .Cells(2, 6).CopyFromRecordset rst6
            .Cells(2, 7).CopyFromRecordset rst7
            .Cells(2, 8).CopyFromRecordset rst7
        End With

    'Add Customer Site (Cust_Site)
        With wstmp
            .Range("D1").EntireColumn.Insert
            lngRows = .Cells(Rows.Count, 1).End(xlUp).Row 'Find last Row
            .Range("D2:D" & lngRows).Formula = strFormula 'Create Cust_Site
        End With
        
    'Add Header Row
        wsTitle.Range("D8:L8").Copy Destination:=wstmp.Range("A1")

    'Copy data to sheet tmpf for final export to Access
        wstmp.Range("A1:I" & lngRows).Copy
        wstmpf.Range("A1").PasteSpecial (xlPasteValues)
        wstmpf.Range("A1").PasteSpecial (xlPasteFormats)

    'Format sheet tmpf
        wstmpf.Activate
        ActiveWindow.Zoom = 75
        Rows("2:2").Select
        ActiveWindow.FreezePanes = True
        wstmpf.Columns("A:I").EntireColumn.AutoFit 'RAD
        
    'Load to Access table
        'SQL
            strSQL9 = "INSERT INTO Pegging_Table SELECT * FROM [tmpf$] IN '" _
            & ThisWorkbook.FullName & "' 'Excel 8.0;'"
            
        'Do it
            cnt.Execute (strSQL9)
    
    'Tidy up
        'Delete temporary worksheets
        'After RAD, remove comments
'            wb.Worksheets.Delete ("tmp")
'            wb.Worksheets.Delete ("tmpf")
            
        'Destroy Objects
            Set wb = Nothing
            Set ws = Nothing
            Set wsTitle = Nothing
            Set wstmp = Nothing
            Set wstmpf = Nothing
            Set cnt = Nothing
            Set rst1 = Nothing
            Set rst2 = Nothing
            Set rst3 = Nothing
            Set rst4 = Nothing
            Set rst5 = Nothing
            Set rst6 = Nothing
            Set rst7 = Nothing
            Set rst8 = Nothing
        
        'Reset Excel environment
            With Application
                .ScreenUpdating = False
                .Calculation = xlCalculationManual
                .DisplayAlerts = False
            End With
        
End Sub
 

Excel Facts

Pivot Table Drill Down
Double-click any number in a pivot table to create a new report showing all detail rows that make up that number
I don't see why either (yet). But ADO has problems when used with Excel to query the same workbook the code is in - the "temp" worksheet should be in a "temp" workbook to avoid potential problems down the road. Without seeing your source data, one possibility is simply that the full data range is not being scanned (this happens if you are querying Excel and there's a blank row somewhere).

This seems like it could be simplified considerably. Whats the data source (excel, access, text file)? And what's the ultimate destination (excel, access, text file?). It seems like you could use one query rather than 8, one recordset, and just push the data right to the destination in one go rather than using a temporary worksheet. What's this code doing?
 
Last edited:
Upvote 0
Thanks Xenou,

The data comes from 2 tables that are linked to external ODBC databases
I'm bringing in the required fields, performing 1 transformation and loading back to a different table.

It appears there are a few blanks rows, but they occur far down below 400,000.
I wasn't sure about the the number of rst's to use.
If I output to .cells(1,1) will it automatically put the data in deifferent columns?

I'll try a tmpworkbook and deleteing all blank rows to see if that take care of the problem

thx
w
 
Upvote 0
Your SQL statement should work like this:
'SQL-statements to be executed.
Code:
strSQL = ""
strSQL = strSQL & " SELECT o.OrderID, o.CustomerID, o.OrderDate, o.ShipName, "
strSQL = strSQL & " c.CompanyName, o.ShipPostalCode, c.ContactName, o.Status "
strSQL = strSQL & " FROM CUSTOMERS c LEFT JOIN Orders o"
strSQL = strSQL & " ON c.CustomerID = o.CustomerID"

Then when you write the values to the worksheet:
Code:
.Cells(2,1).CopyFromRecordset rst

Actually you don't need to use Excel to create the Cust_Site record if it's a concatenation of two other fields. Simply in your sql:
Code:
strSQL = ""
strSQL = strSQL & " SELECT o.OrderID, o.CustomerID, o.OrderDate,"
[COLOR="Blue"]strSQL = strSQL & o.OrderDate & "_" & o.OrderID AS Cust_Site,"
[/COLOR]strSQL = strSQL & "o.ShipName, "
strSQL = strSQL & " c.CompanyName, o.ShipPostalCode, c.ContactName, o.Status "
strSQL = strSQL & " FROM CUSTOMERS c LEFT JOIN Orders o"
strSQL = strSQL & " ON c.CustomerID = o.CustomerID"
If I have correctly interpreted your formula from the column you insert.

I don't see why you wouldn't just go right from the first database (?) to the second database (?). Why involve Excel here?
 
Upvote 0
Hi all,

Have not been able to successfully insert all records from an Excel Worksheet to an Access table

I am using a temporary worksheet. There are 421,952 records on the Excel sheeet.
There are only 28,736 in the Access table after the INSERT statement:

No error messages, just not the correct number of records.

Code:
 'Load to Access table
        'SQL
            strSQL9 = "INSERT INTO myTbl SELECT * FROM [tmpf$] IN '" _
            & wbtmp.FullName & "' 'Excel 8.0;'"
            
        'Do it
            cnt.Execute (strSQL9)

thx
w
 
Upvote 0
Can you confirm that the 28,736 that are inserted are actually the first 28,736 records? Are there any indexes in the table?
 
Upvote 0
Hi Xenou,

The first record in the desination table is row 701 on the worksheet - seems odd.

There are 10 fields in the table
These are indexed (Duplicates OK)
Field1, Field3, Field7

thx
w
 
Upvote 0
That's bizarre alright - not sure why Excel would be skipping (or rejecting) rows. If you want to send a sample to me I can try it out to (depends on the sensitivity of your data). What's in the fields? What version of Excel? What type/version of database are you loading to? Is this all on a local network or a remote connection?
 
Upvote 0
Hi Xenou,

The data is sensitive, I'll see if I can generate similar random data to produce similar results.

I created a new workbook and copied in nothing but headers and saved it.
I then went to Access and used Table Wizard to create a new table by importing the newly created workbook.

I stepped through the wizard. Seemed odd that it automatically wanted to add indices to the same three fields even though there was no data, I removed all indices and removed the primary key. Saved.

Back to my control workbook, changed the name of the target table, same result 28,736 records.

The data: there are some blanks in some of the columns, though none in Column A.

thx
w
 
Last edited:
Upvote 0

Forum statistics

Threads
1,215,051
Messages
6,122,872
Members
449,097
Latest member
dbomb1414

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top