Problem with manually added column in a table connected to a query

rudiklein

New Member
Joined
Jan 10, 2017
Messages
2
I have this weird issue with an additional column in a table that is linked to a query. It is a bit hard to explain.

This is the case:
I have loaded data from a CSV file. The file contains bank records. The data is presented in a table on my sheet (standard procedure).
I then add a column at the end of the table (not within the query, but straight into the table with "Insert/Table columns...).
I use the fields in this column to tag bank records (mortgage, utilities etc.)

Now I can use this table to create a pivot table which aggregates all records.
When I refresh the data, the table is updated. The 'tags' stay in the correct rows.
When I sort the table, the rows are correctly sorted including the tag fields.

But here is the snag:
When I sort the table first and then refresh the data, the tag column goes bezirk. With every refresh it is (randomly?) sorted and the tags are no longer at the correct row.

I am flabbergasted.

Anyone?

Bye,

Rudi
 

Excel Facts

Easy bullets in Excel
If you have a numeric keypad, press Alt+7 on numeric keypad to type a bullet in Excel.
from all my mistakes and trial and error, you cannot add your own columns to a query, they will not be recognized.
there is a round-about way arond it that i use successfully
i will copy it here exactly as i saw it posted. i can no longer find original post to give the site.
i use this sucessfully, its a bit complicated but it works

Marcel Beug did a great job, finding the solution and taking the time to edit a video with an example.

After testing it positively, here is my rehash of his explanations, with more written details in order to hopefully help people not yet very familiar with Excel Power Query.

Situation:

Excel workbook with existing sheet (e.g. “SQLdbOrigin”) containing Original SQL Query data

You wish to add to this original data extra columns with manually entered values, keeping their relationship to the queried data upon refreshes of the original SQL query data

Procedure:

From a cell in the existing original data table (obtained from a DB query for instance), Power query this table to a new output table:

Menu: ‘Data > From Table’ (original data table will be auto named ‘Table1’)

A Power Query Editor window opens, displaying the queried data

In ‘Applied Steps’: Rename ‘Changed Type’ (via F2) to e.g. ‘SQLdata’, and query name to ‘SQLoutput’

Click on menu ‘Close&Load’, which auto creates a new sheet with output table ‘SQLoutput’

One can optionally Rename the sheet and the table

Add (lines outside the table, formats everywhere, and) columns for manual fields attached to ‘SQLoutput’, and fill in data

NB: the definition of the table will auto adjust to include these new columns

From a cell in table ‘SQLoutput’, Power query this output data table:

Menu: ‘Data > From Table’

In ‘Applied Steps’: Rename ‘Changed Type’ (via F2) to e.g. ‘FullData’, and query name to ‘SQLoutput’

Click on ‘Advanced Editor’, which opens a new window for SQLoutput (2)

Copy the 2 lines between ‘let’ and ‘in’

Click on ‘Done’

Close the Power Query Editor window and click ‘Discard’

In the ‘Workbook Queries’ column, right-click on ‘SQLoutput’ query > ‘Edit’

In this Power Query Editor window, modify the query for SQLoutput to merge with its own output table FullData:

Click on ‘Advanced Editor’

Add a comma at the end of the line just before the ‘in’

Paste, just before the line with ‘in’, the 2 lines previously copied (from the query of Table1_2)

Rename ‘Source’ to e.g. ‘Source2’ in the 2 newly added lines (2 times)

Rename the table after ‘in’ from ‘SQLdata’ into ‘FullData’ (as chosen before)

Click on ‘Done’ (and observe the column ‘Query Settings’ on the right)

Click on ‘Merge Queries’

Choose the (only possible) table ‘SQLoutput (Current)’ in the dropdown box

Select the Key column(s) corresponding in the two tables

Select ‘Join Kind’: ‘Left Outer’

Click on ‘OK’

In the Formula Bar of the Power Query Editor (if needed, set checkbox in Menu > View):

Rename the first ‘FullData’ into ‘SQLdata’ (to redefine the two tables to join)

In the header of the new rightmost column, click on the ‘expand’ icon ‘<>’

Deselect the first checkbox ‘(Select All Columns)’

Select in the list the added columns (with the manual values)

Deselect checkbox ‘Use original column name as prefix’

Click ‘OK’

Menu: ‘Close & Load’

Test:

Add and/or delete line(s) in the table of the sheet SQLdbOrigin to simulate a change in the db

Refresh the query (via right click in the table; the ‘Workbook Queries’ on the right can be closed)

Observe that the results are ok

Smile.

Notes:

For calculated cells to retain their functions, add them in the original SQLdbOrigin sheet; Beware: they will behave as expected only for cells within the original table

For calculated cells to work correctly in the SQLoutput table, add first an empty column to the right of the table (which can then be hidden) and add column(s) of calculated fields thereafter. They will not be changed by the queries refresh, and will keep their functions unaltered (for relative addresses, of course)

Do not rename column heads in the original Table1 table in sheet SQLdbOrigin, as they are used in the queries, unless you update all queries accordingly…

Careful also not to change the names of the queries or table, for the same reason…

Formatting of the SQLoutput table seems to hold across refreshes, except for row height always reset.
 
Upvote 0
from all my mistakes and trial and error, you cannot add your own columns to a query, they will not be recognized.
there is a round-about way arond it that i use successfully
i will copy it here exactly as i saw it posted. i can no longer find original post to give the site.
i use this sucessfully, its a bit complicated but it works

Marcel Beug did a great job, finding the solution and taking the time to edit a video with an example.

After testing it positively, here is my rehash of his explanations, with more written details in order to hopefully help people not yet very familiar with Excel Power Query.

Situation:

Excel workbook with existing sheet (e.g. “SQLdbOrigin”) containing Original SQL Query data

You wish to add to this original data extra columns with manually entered values, keeping their relationship to the queried data upon refreshes of the original SQL query data

Procedure:

From a cell in the existing original data table (obtained from a DB query for instance), Power query this table to a new output table:

Menu: ‘Data > From Table’ (original data table will be auto named ‘Table1’)

A Power Query Editor window opens, displaying the queried data

In ‘Applied Steps’: Rename ‘Changed Type’ (via F2) to e.g. ‘SQLdata’, and query name to ‘SQLoutput’

Click on menu ‘Close&Load’, which auto creates a new sheet with output table ‘SQLoutput’

One can optionally Rename the sheet and the table

Add (lines outside the table, formats everywhere, and) columns for manual fields attached to ‘SQLoutput’, and fill in data

NB: the definition of the table will auto adjust to include these new columns

From a cell in table ‘SQLoutput’, Power query this output data table:

Menu: ‘Data > From Table’

In ‘Applied Steps’: Rename ‘Changed Type’ (via F2) to e.g. ‘FullData’, and query name to ‘SQLoutput’

Click on ‘Advanced Editor’, which opens a new window for SQLoutput (2)

Copy the 2 lines between ‘let’ and ‘in’

Click on ‘Done’

Close the Power Query Editor window and click ‘Discard’

In the ‘Workbook Queries’ column, right-click on ‘SQLoutput’ query > ‘Edit’

In this Power Query Editor window, modify the query for SQLoutput to merge with its own output table FullData:

Click on ‘Advanced Editor’

Add a comma at the end of the line just before the ‘in’

Paste, just before the line with ‘in’, the 2 lines previously copied (from the query of Table1_2)

Rename ‘Source’ to e.g. ‘Source2’ in the 2 newly added lines (2 times)

Rename the table after ‘in’ from ‘SQLdata’ into ‘FullData’ (as chosen before)

Click on ‘Done’ (and observe the column ‘Query Settings’ on the right)

Click on ‘Merge Queries’

Choose the (only possible) table ‘SQLoutput (Current)’ in the dropdown box

Select the Key column(s) corresponding in the two tables

Select ‘Join Kind’: ‘Left Outer’

Click on ‘OK’

In the Formula Bar of the Power Query Editor (if needed, set checkbox in Menu > View):

Rename the first ‘FullData’ into ‘SQLdata’ (to redefine the two tables to join)

In the header of the new rightmost column, click on the ‘expand’ icon ‘<>’

Deselect the first checkbox ‘(Select All Columns)’

Select in the list the added columns (with the manual values)

Deselect checkbox ‘Use original column name as prefix’

Click ‘OK’

Menu: ‘Close & Load’

Test:

Add and/or delete line(s) in the table of the sheet SQLdbOrigin to simulate a change in the db

Refresh the query (via right click in the table; the ‘Workbook Queries’ on the right can be closed)

Observe that the results are ok

Smile.

Notes:

For calculated cells to retain their functions, add them in the original SQLdbOrigin sheet; Beware: they will behave as expected only for cells within the original table

For calculated cells to work correctly in the SQLoutput table, add first an empty column to the right of the table (which can then be hidden) and add column(s) of calculated fields thereafter. They will not be changed by the queries refresh, and will keep their functions unaltered (for relative addresses, of course)

Do not rename column heads in the original Table1 table in sheet SQLdbOrigin, as they are used in the queries, unless you update all queries accordingly…

Careful also not to change the names of the queries or table, for the same reason…

Formatting of the SQLoutput table seems to hold across refreshes, except for row height always reset.
Hi!

This was really helpful. I have one question. I´m querying from csv files I get from daily exporting contacts from LinkedIn and dump in one folder, just to keep track of people contacted. My problem was solved as I had the rows in the new columns changing places. But this made it a two step refresh or am I wrong? I first have to refresh Query 1 (LinkedinExports) and then refresh newly created SQLoutput so new columns keep rows linked to right data. Is it like this? There's no way to create a one refresh solution only in Query 1? Or did I missed some step?
Thanks!
 
Upvote 0
to enable and update all tables i have to do refresh all
sometimes even 6 times as i have about 9 queried queries in the workbook
 
Upvote 0
Hi @rjmdc

I tried to follow your instructions but it is not working for me. I need to extract data from two SharePoint lists that are connected to each other (one extract data from the other). It is for an complicated Org Chart.
I have two data sources - Employees and Organizational Units. I did a extra queries for today's date. Then I did my DataOrgChart query.

I need to add two extra columns so my HR dept. can add the exception such as a second report to or special conditions.

I did follow exactly your steps but when I filter, the data entered in the exception columns doesn't follow the associated rows.


At the line In this Power Query Editor window, modify the query for SQLoutput to merge with its own output table FullData: I am not sure I fully understand what you mean.

I am not a programmer, I am just a regular user who like to challenges herself so I don't have any programming knowledge.

Can you help me?
 
Upvote 0
it wasnt my isnstructions
it was verbatim followed form what i was given
it alkso took me about 5 tries till i got the instructions correct
first i created my worksheet

then i queried this worksheet to create another one called data_prep
i refrenced dataprep and loaded to data_full
i added my columns to data_full
then i merged data_prep and data_full extracting my new columns

this leaves a lot of worksheets in the workbook
 
Upvote 0

Forum statistics

Threads
1,214,875
Messages
6,122,047
Members
449,064
Latest member
scottdog129

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top