Excel 2019: Clean Data with Power Query
October 31, 2019 - by Bill Jelen
Power Query is built in to Windows versions of Office 365, Excel 2016, Excel 2019 and is available as a free download in Windows versions of Excel 2010 and Excel 2013. The tool is designed to extract, transform, and load data into Excel from a variety of sources. The best part: Power Query remembers your steps and will play them back when you want to refresh the data. This means you can clean data on Day 1 in 80% of the normal time, and you can clean data on Days 2 through 400 by simply clicking Refresh.
I say this about a lot of new Excel features, but this really is the best feature to hit Excel in 20 years.
I tell a story in my live seminars about how Power Query was invented as a crutch for SQL Server Analysis Services customers who were forced to use Excel in order to access Power Pivot. But Power Query kept getting better, and every person using Excel should be taking the time to learn Power Query.
Get Power Query
You may already have Power Query. It is in the Get & Transform group on the Data tab.
But if you are in Excel 2010 or Excel 2013, go to the Internet and search for Download Power Query. Your Power Query commands will appear on a dedicated Power Query tab in the Ribbon.
Clean Data the First Time in Power Query
To give you an example of some of the awesomeness of Power Query, say that you get the file shown below every day. Column A is not filled in. Quarters are going across instead of down the page.
To start, save that workbook to your hard drive. Put it in a predictable place with a name that you will use for that file every day.
In Excel, select Get Data, From File, From Workbook.
Browse to the workbook. In the Preview pane, click on Sheet1. Instead of clicking Load, click Edit. You now see the workbook in a slightly different grid—the Power Query grid.
Now you need to fix all the blank cells in column A. If you were to do this in the Excel user interface, the unwieldy command sequence is Home, Find & Select, Go To Special, Blanks, Equals, Up Arrow, Ctrl+Enter.
In Power Query, select Transform, Fill, Down.
All of the null values are replaced with the value from above. With Power Query, it takes three clicks instead of seven.
Next problem: The quarters are going across instead of down. In Excel, you can fix this with a Multiple Consolidation Range pivot table. This requires 12 steps and 23+ clicks.
In Power Query select the two columns that are not quarters. Open the Unpivot Columns dropdown on the Transform tab and choose Unpivot Other Columns, as shown below.
Right-click on the newly created Attribute column and rename it Quarter instead of Attribute. Twenty-plus clicks in Excel becomes five clicks in Power Query.
Now, to be fair, not every cleaning step is shorter in Power Query than in Excel. Removing a column still means right-clicking a column and choosing Remove Column. But to be honest, the story here is not about the time savings on Day 1.
But Wait: Power Query Remembers All of Your Steps
Look on the right side of the Power Query window. There is a list called Applied Steps. It is an instant audit trail of all of your steps. Click any gear icon to change your choices in that step and have the changes cascade through the future steps. Click on any step for a view of how the data looked before that step.
When you are done cleaning the data, click Close & Load as shown below.
If your data is more than 1,048,576 rows, you can use the Close & Load dropdown to load the data directly to the Power Pivot Data Model, which can accommodate 995 million rows if you have enough memory installed on the machine.
In a few seconds, your transformed data appears in Excel. Awesome.
The Payoff: Clean Data Tomorrow With One Click
But again, the Power Query story is not about the time savings on Day 1. When you select the data returned by Power Query, a Queries & Connections panel appears on the right side of Excel, and on it is a Refresh button. (We need an Edit button here, but because there isn't one, you have to right-click the original query to view or make changes to the original query).
It is fun to clean data on Day 1. I love doing something new. But when my manager sees the resulting report and says “Beautiful. Can you do this every day?” I quickly grow to hate the tedium of cleaning the same data set every day.
So, to demonstrate Day 400 of cleaning the data, I have completely changed the original file. New products, new customers, smaller numbers, more rows, as shown below. I save this new version of the file in the same path and with the same filename as the original file.
If I open the query workbook and click Refresh, in a few seconds, Power Query reports 92 rows instead of 68 rows.
Cleaning the data on Day 2, Day 3, Day, 4,...Day 400,...Day Infinity now takes two clicks.
This one example only scratches the surface of Power Query. If you spend two hours with the book, M is for (Data) Monkey by Ken Puls and Miguel Escobar, you will learn about other features, such as these:
- Combining all Excel or CSV files from a folder into a single Excel grid
- Converting a cell with Apple;Banana;Cherry;Dill;Eggplant to five rows in Excel
- Doing a VLOOKUP to a lookup workbook as you are bringing data into Power Query
- Making a single query into a function that can be applied to every row in Excel
For a complete description of Power Query, check out M Is for (Data) Monkey by Ken Puls and Miguel Escobar. By late 2019, the retitled second edition, Master Your Data, will be available.
Title Photo: pan xiaozhen at Unsplash.com