Scenario:
We got a requirement to download the Excel files on a daily basis from the webpage and need to process the same automatically (window scheduled jobs) and share the report with my supervisors. If we manage this on daily basis by manual and it will occupy some resource bandwidth each day and there is a chance to have some manual error, to avoid this we thought to spend time with automation to complete this task.
In simple, If you need to download files from the web by repeatedly clicking links and processing the data in it on daily basis then you will probably want to automate the task. Windows PowerShell and PowerShell come with file-download capabilities. Using PowerShell to download files is a matter of knowing which cmdlets and .NET classes to use and how to use them
Download File Using PowerShell:
In this discussion, I am going to show how to download a file from a URL using PowerShell and this can be achieved using the Invoke-WebRequest method. We can use Invoke-WebRequest/Invoke-RestMethod/Start-BitsTransfer to download the files in the PowerShell. Whichever one of these methods you use, the logic and components to make them work are the same. To download a file we need to know the source URL and give up a destination for the file that we want to download. If required by the webserver, you need to enter the credentials as well. You can also download the files by using Invoke-RestMethod & Start-BitsTransfer
For our example, I am going to use the Invoke-WebRequest method for downloading the files from the Web. If the source location requires users to log in, the Invoke-WebRequest cmdlet can handle requests with credentials as well.
To download a file, the syntax below shows the minimum parameters required to achieve our requirement. To download the file, the parameter -OutFile is required. You don’t need to enter the full path, but a file name is required. The Outfile expects a path and a filename.
Invoke-WebRequest -Uri <source> -OutFile <destination>
STEP #1 Get the Souce path
# Source URL
$sourceURL = “https://filesamples.com/samples/document/txt/sample3.txt”
STEP #2 Get the Destination path
# Destation file
$destPath = “C:\Thiyagu Disk\dotnet-helpsers\output.txt”
STEP #3 Download the file using Invoke Method.
# Download the file
Invoke-WebRequest -Uri $sourceURL -OutFile $destPath
Full Code:
|
1 2 3 4 5 |
# Assign SoruceURL and Destination Path $sourceURL = "https://filesamples.com/samples/document/txt/sample3.txt" $destPath = "C:\Thiyagu Disk\BLOG_2022\output.txt" # USe Invoke Method to download the File Invoke-WebRequest -Uri $sourceURL -OutFile $destPath |
Output:
Authentication with Invoke-WebRequest
Some URLs require you to log in before you can access/download the files. With the Invoke-WebRequest cmdlet, we can provide the credentials that are needed for downloading the files. For this example, i am using the Credential directly in the script. If you are creating a script for automation (that will need to run automatically), then you need to store the credentials in the script itself.
Full Code:
|
1 2 3 4 5 6 7 8 9 10 11 12 |
# Assign SoruceURL and Destination Path $sourceURL = "https://filesamples.com/samples/document/txt/sample3.txt" $destPath = "C:\Thiyagu Disk\BLOG_2022\output.txt" # Assign the Username and Pwd to access the $sourceURL $username = 'XXXXX' $password = 'XXXXX' # Convert to SecureString $secPassword = ConvertTo-SecureString $password -AsPlainText -Force # Create Credential Object $credObject = New-Object System.Management.Automation.PSCredential ($username, $secPassword) # USe Invoke Method to download the File Invoke-WebRequest -Uri $sourceURL -OutFile $destPath |
- The Invoke-WebRequest doesn’t perform a check if the target file exists. If the file already exists, it is overwritten without any warning.
- If you need to authenticate under the current user on a remote website (via NTLM or Kerberos), you need to add the –UseDefaultCredentials parameter
- You can also download the files by using Invoke-RestMethod & Start-BitsTransfer