Which Data Import Method to Use in this Scenario ?

happy1001

New Member
Joined
Jan 17, 2016
Messages
14
Hello

I have got a few Excel Files in which I need to import the data from a few different queries from a database in SQL Server installed on my local machine. Then I intend to use this imported data inside Power Pivot Data Model.

I am not sure which particular data import method should be used, which will provide me the following flexibility -

1. Ability to change / modify the SOURCE data query which resides inside SQL Server. I will have to add or remove columns from that query based on the testing work that I am doing. The New Columns that I add to that SQL query in the future, should be able to get imported inside Excel.

2. Auto Refresh - I need to auto refresh the data that I will getting inside excel. Because the data is such that it will keep on getting modified everyday within the SQL Server Database and I need to have the LATEST Data inside Excel.

I have tried to import the data from within Power Pivot Data Model, but it does not import the latest data columns that I add within the SQL Server later on.

I am interested in using the Power Query based data import method, but I am not sure if that will help me in the above 2 requirements.

Please share any possible solutions for data import, which would satisfy the above conditions.

And if there are any links available about the BEST PRACTICES for creating such Excel and SQL Server type of Data Connections, then please share those links as well.

I am using the following software versions -
Microsoft SQL Server Management Studio version- 12.0.2000.8,
Microsoft Office Professional Plus 2016 x64
and Windows 7 x64

Thanks a lot for any help.
Regards

PS: Please do not suggest that method in which there is a risk for the Data Model to get broken because of changes in the SQL Server data source query. Because then I loose all the work that I have done within the Data Model and I have to recreate everything from start.
 

ImkeF

Well-known Member
Joined
Feb 1, 2015
Messages
544
If you put Power Query between your SQL Server and your Power Pivot Model you’re basically creating sort of a logical DWH which enables you to separate your business logic that is reflected in your reports (and is modelled in Power Pivot) from your sources (your SQL server today, Oracle or csv files tomorrow…).

You just have to check for consistent output to Power Pivot and everything should be fine. Meaning: No renaming of fields that are transferred to Power Pivot already. If a column name changes in your source you simply add a renaming step in Power Query so that the original column name for the output to Power Pivot will remain.

If you edit your Power Query query properties you have several options like: Automatic refresh when opening the file or scheduled refreshes.

Prerequisite: Excel 2013 or higher. There still are technical issues in 2010 between PQ and PP.
 

happy1001

New Member
Joined
Jan 17, 2016
Messages
14
If you put Power Query between your SQL Server and your Power Pivot Model you’re basically creating sort of a logical DWH which enables you to separate your business logic that is reflected in your reports (and is modelled in Power Pivot) from your sources (your SQL server today, Oracle or csv files tomorrow…).

You just have to check for consistent output to Power Pivot and everything should be fine. Meaning: No renaming of fields that are transferred to Power Pivot already. If a column name changes in your source you simply add a renaming step in Power Query so that the original column name for the output to Power Pivot will remain.

If you edit your Power Query query properties you have several options like: Automatic refresh when opening the file or scheduled refreshes.

Prerequisite: Excel 2013 or higher. There still are technical issues in 2010 between PQ and PP.
Thank you so much for your reply ImkeF

If I understand it correctly, then you mean that –



  1. If I keep the same column names, which have been earlier imported inside the Data Model “by doing modifications inside the Power Query” then the Data Model will not break, even if I add any NEW Columns to it, by making changes in the original SQL query from within the database ? —– If my understanding is correct, then please confirm it once.
  2. If I use the power query method for data import into the Data Model, then any new columns that I add to the original SQL query from within the database, will get reflected inside the Data Model. —— I tried this approach earlier, but that did not work for some reason, maybe I should try again.
Thanks a lot for your help.

PS: we had this discussion on other place as well, but let us keep it here now.
 

ImkeF

Well-known Member
Joined
Feb 1, 2015
Messages
544
Did you experience problems in the past like described here?: Breaking Power Query via Power Pivot is a thing of the pastThe Ken Puls (Excelguru) Blog
They are solved fortunately (at least for 2013 upwards).

So 1) shouldn't give you any problem now

Not sure what you mean with 2): It's basically up to you if you let new columns from your SQL DB pass through your PQ queries into your PP data model: Any new SQL columns can be retrieved by PQ. They might not show up autmatically in your existing PQ queries like you expect them if your query contains column filters somewhere. They can be defined as "Keep the following columns only" (Table.SelectColumns) or "Remove These columns" (Table.RemoveColumns). Only the latter will let new columns automatically show up, otherwise you have to include the new column in the filter expression.
 

happy1001

New Member
Joined
Jan 17, 2016
Messages
14
Thank you so much ImkeF. It is great that most of these common problems are solved in the new updates.


Best Regards
 

Forum statistics

Threads
1,082,369
Messages
5,365,049
Members
400,819
Latest member
Gossow

Some videos you may like

This Week's Hot Topics

  • populate from drop list with multiple tables
    Hi All, i have a drop list that displays data, what i want is when i select one of those from the list to populate text from different tables on...
  • Find list of words from sheet2 in sheet1 before a comma and extract text vba
    Hi Friends, Trying to find the solution on my task. But did not find suitable one to the need. Here is my query and sample file with details...
  • Dynamic Formula entry - VBA code sought
    Hello, really hope one of you experts can help with this - i've spent hours on this and getting no-where. .I have a set of data (more rows than...
  • Listbox Header
    Have a named range called "AccidentsHeader" Within my code I have: [CODE]Private Sub CommandButton1_Click() ListBox1.RowSource =...
  • Complex Heat Map using conditional formatting
    Good day excel world. I have a concern. Below link have a list of countries that carries each country unique data. [URL...
  • Conditional formatting
    Hi good morning, hope you can help me please, I have cells P4:P54 and if this cell is equal to 1 then i want row O to say "Fully Utilised" and to...
Top