Back in February 2016, I posted a step by step article (found here) on how to loop through a website and dynamically pull down multiple pages into a single table in Power Query. I was recently approached by a gentleman who had read my blog, wanting to enhance the looping process. The requirements were:
Current Problem:
– The user can currently only loop through Year, having to hard code the Page and Month.
Requirements:
– Dynamically loop through three webpage parameters and output them into a single table – highlighted below:
http://www.boxofficemojo.com/monthly/?page=1&view=calendargross&yr=2017&month=1&p=.htm
– Specify the Page, Year and Month to loop through
o Years 2010 – 2017
o Months 1 – 12
o Pages 1 – 10
Parameters
Whilst GUI Parameters are not a new concept in Power Query, they did not exist when I wrote my initial blog. As a result, I had to code a parameter into the M query and then define it as a function. A similar concept will be adopted here, but the out of the box parameter functionality makes life so much easier now.
I previously wrote an article explaining how to set up parameters (here), so I will simply show you the output of them below:
Note – all three parameters are set to Text. This is needed in order to pass the parameters into the website address later on.
Web Connection
The first step is to import the data from the website (shown above) using the web connector. To pass our parameter values into the URL, click the cog in the first Applied Step – typically called ‘Source’.
This will open a new window, where by design, the hardcoded URL is shown under the Basic radio button. Select Advanced and you will see additional configuration capabilities.
URL Parts enable you to build up elements of the webpage, whilst dynamically passing in parameters. The URL Preview will show you how the URL looks with any parameters or URL Parts. All other options can be ignored for this example.
The URL Parts had to be broken down quite significantly and I will show you both the code and how it looks in the Advanced window.
User Interface:
The sections highlighted in red are the parameters and sit in between some of the hard-coded URL text
Code Breakdown:
– Text = http://www.boxofficemojo.com/monthly/?page=
– Parameter = [Page]
– Text = &view=calendargross&yr=
– Parameter = [Year]
– Text = &month=
– Parameter = [Month]
– Text = &p=.htm
URL Preview:
http://www.boxofficemojo.com/monthly/?page={Page}&view=calendargross&yr={Year}&month={Month}&p=.htm
Year/Month/Page Permutations Table
Before creating/invoking the function, a table that contains the different years, months and pages is needed. There are many ways to achieve this, but I used a cross join technique (found here) with three disparate tables. I followed the end user requirements and end up with a table like below:
This will be the final table where the function is called.
Function
Let’s go back to the original table, where the parameters have been applied to the Web URL. Right click the table and select Create Function. Nothing else is required (apart from naming it), as Power Query will automatically package up the table and you end up with a new Function like below:
The last step is to invoke the function. Go back to the Permutations table (called ‘All Data’ in my example), and go to Add Column > Invoke Custom Function.
When prompted with the next screen, map each parameter to the equivalent column. The red highlighted values are the columns and yellow contain the parameters. Both are named the same in my example and make mapping even easier.
The function will loop through each URL permutation, dynamically changing the parameters on execution. Once it has finished, tidy up any errors (for Years, Pages and/or Months) that do not exist. Apply all transformations within the Query Editor and that is it – the data is ready for Reporting! You may need to configure some of the measures (e.g. Total Gross) to be numbers, but that is easy enough to do within the Query Editor.
The example in this blog can be adapted in many ways and illustrates how dynamic Power Query can be – both in Excel and Power BI.
Further Reading
o Cross Join in Power BI – http://bit.ly/2ij8zJ1
o Reza Rad Custom Functions – http://bit.ly/2zz6X2f
Contact Me
If you would like a copy of the PBIX workbook or have any general questions, feel free to leave a comment below.
Twitter: @DataVizWhizz
Introduction to Data Wrangler in Microsoft Fabric
What is Data Wrangler? A key selling point of Microsoft Fabric is the Data Science
Jul
Autogen Power BI Model in Tabular Editor
In the realm of business intelligence, Power BI has emerged as a powerful tool for
Jul
Microsoft Healthcare Accelerator for Fabric
Microsoft released the Healthcare Data Solutions in Microsoft Fabric in Q1 2024. It was introduced
Jul
Unlock the Power of Colour: Make Your Power BI Reports Pop
Colour is a powerful visual tool that can enhance the appeal and readability of your
Jul
Python vs. PySpark: Navigating Data Analytics in Databricks – Part 2
Part 2: Exploring Advanced Functionalities in Databricks Welcome back to our Databricks journey! In this
May
GPT-4 with Vision vs Custom Vision in Anomaly Detection
Businesses today are generating data at an unprecedented rate. Automated processing of data is essential
May
Exploring DALL·E Capabilities
What is DALL·E? DALL·E is text-to-image generation system developed by OpenAI using deep learning methodologies.
May
Using Copilot Studio to Develop a HR Policy Bot
The next addition to Microsoft’s generative AI and large language model tools is Microsoft Copilot
Apr