All posts by Thiyagu

Quickly extracting all links from a web page using the PowerShell

We are maintaining 500+ client websites in our environment. Some day before we received a request to get the list of links/Images used on each home page. We knew that it will be very tricky to get the list of links/URLs mapped in the 500+ pages and you are also aware that the annual work will not give 100% results.

So we decided to use Powershell Links in the Invoke-WebRequest method to reduce manual effort. In this post, we will discuss the same with a simple example using a single URL. For checking the multiple URLs, please refer to a similar article which helps to read from excel and loop it –

https://dotnet-helpers.com/powershell/powershell-script-for-website-availability-monitoring-with-excel-report-as-output

PowerShell’s Invoke-WebRequest is a powerful cmdlet that allows you to download, parse, and scrape web pages. The Invoke-WebRequest cmdlet is used to download files from the web via HTTP and HTTPS. However, this cmdlet enables you to do more than download files. You can use this cmdlet for analyzing the contents of web pages.

Example: Get the list of URLs

The below script will grab the innerText in addition to the corresponding links

(Invoke-WebRequest -Uri “https://dotnet-helpers.com/powershell”).Links | sort-object href -Unique | Format-List innerText, href

Example: Get the list of URLs This gets the list of links with grid view control

The grid view control lets you filter URLs with keyword search and you will copy the listings to the clipboard by using the Ctrl + C option.

(Invoke-WebRequest -Uri “www.lantus.com”).Links.Href | Sort-Object | Get-Unique | out-gridview

Example: Get the list of Image URLs

To fetch the list of image URLs from the page, you can run the below cmdlet

(Invoke-WebRequest -Uri “https://dotnet-helpers.com”).Images | Select-Object src

Run BAT File From PowerShell Script

To Run BAT File from PowerShell Script, you can run it manually from the PowerShell (or make the PowerShell execution in the windows scheduler run at a certain time). 

A batch file is a series of commands or a script in the Windows Operating System that executes a series of tasks on the local or remote machines and it has a. Bat extension. Although PowerShell and Batch are different languages, both can be integrated into each other and helps to call and execute each other.

This article will illustrate different ways to run a Batch file from a PowerShell script.

Step #1 Create a .bat file with commands

For time being, I only used the echo command in the bat file, you can write your logic command as required. For this example, I save the file as CallMe.bat.

echo Write something, it will be used in the command “echo”
pause

Step #2 Create a Powershell script file & call the .bat file

There are many approaches we can call the .bat file from PowerShell, which we can choose based on our purpose. This step will guide you on how to Add the Batch file to a PowerShell script to run automatically when the PowerShell script is being run.

Method: 1

One way of running a Batch file from the PowerShell script is using the Start-Process cmdlet. The Start-Process cmdlet allows you to run one or multiple processes on your computer from within PowerShell.
It’s designed to run a process asynchronously or to run an application/script elevated (with administrative privileges).

To run the Batch file, add the following line of code to the PowerShell script:

Filepath specifies the path of the Batch file.
NoNewWindow starts the process in the current window (add this at end of the script to mention not to open the cmd window).

Start-Process -FilePath ‘C:\blog\callme.bat’

To run the Batch file as administrator, add -verb run as in the above code. This command is useful when your .bat file contains commands which require administrator privileges to run on execution.

Start-Process -FilePath ‘C:\blog\callme.bat’ -Verbose Runas -NoNewWindow

Output

Method: 2

/c referred to Carries out My Command and then terminates

cmd.exe /c ‘C:\blog\callme.bat’

Method : 3

& C:\blog\callme.bat

How to Create Your First simple Jenkins Pipeline

In our first Jenkins tutorial for beginners, we focused on how to install and configure Jenkins. In this tutorial, How To Create Your First simple Jenkins Pipeline. We’ll keep it simple and avoid Maven or Git at this juncture. We’ll just create a Jenkins freestyle job that invokes the JDK’s runtime instance and prints out the version of the JRE that is currently running in the Jenkins machine (location of Jenkins installation).

STEP: 1 Login into Jenkins and go to Jenkins dashboard

To create a Jenkins freestyle job, log on to your Jenkins dashboard by visiting your Jenkins installation path. Usually, it will be hosted on localhost at http://localhost:8080 If you have installed Jenkins in another path, use the appropriate URL to access your dashboard as shown in the below Jenkins job creation example.

STEP: 2 Create New Item (New job/pipeline)

The first step to creating a Jenkins build job is to click the New Item link in the top left-hand corner of the admin console and enter the Item name & click Ok.

Note: If you are unable to see this icon, it means that you don’t have sufficient privileges. In the next window, type the name of the job such as the first job, select job type as freestyle job, and then click ok:

STEP 3: Configure the new job details

After clicking OK ( in STEP 3), the configuration page for the freestyle Jenkins job will appear as shown in the below snapshot. Notice there are a number of options to configure, including build triggers, source code management options, Jenkins build job steps, and post-build actions. As we mentioned in starting of the post, we are only going to create simple jobs without doing any build or deployment.

In the Jenkins build job, we will change the description to “My_First_JenkinsJob” and Under the “Source Code Management” section, for this example, we are not going to use any GIT URL to download the solution so you can select “None”

In this job/pipeline, we going the check the java version that was installed in our local/Jenkins machine. For this, in the “Build section” choose “Execute Windows batch command” from the drop-down and type the “java -version” in the window batch command section.

STEP 4: Save the job and click on the Build Now link.

Now your Jenkins Job is ready for checking the version of Java installed on the Jenkins machine.

STEP 5: Save the job, click on the Build icon & Check the status.

To run the created job, click on the job which you need to build. Once the new job is opened as shown in the below snapshot, click the “Build Now” to start the job execution. You can check the build status under the “Build History section” at the left bottom of the screen.

STEP 6: View the log for checking the output

In the window batch command, we placed the cmd to check the version of java.  So the same has been executed and you can able to see the output in the log file as shown in the below image.

Based on this simple example, I now hope you can able to Create Your First simple Jenkins Pipeline.

Install Jenkins on Windows – A Step-By-Step Guide

In this article, we will go through the steps to download and install Jenkins on Windows. Jenkins is a free and open-source automation software used for building, testing, and deploying code to achieve the end goal of Continuous Delivery and Continuous Integration. It provides faster and more efficient code deployment in multiple environments. Jenkins supports a wide range of plugins due to which it can deploy almost any kind of code to any environment.

What is Jenkins?

Jenkins is a self-contained, open source (DevOps tool) automation server which can be used to automate all sorts of tasks related to building, testing and delivering or deploying software. Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with a Java Runtime Environment (JRE) installed.

Jenkins may be installed on either Windows/Unix/supported platforms, but we will focus only install Jenkins on a Windows machine in the article (the below steps explain the installation in the standalone machine).

STEP 1: Prerequisites

Before you proceed to install Jenkins in your windows/Unix system, there are some prerequisites for Jenkins to install Jenkins on your computer.

Hardware requirements:

  • Hardware requirements (Minimum): 256 MB of RAM, 1 GB of drive space (although 10 GB is a recommended minimum if running Jenkins as a Docker container)
  • Hardware configuration for a small team (Recommended) : 4 GB+ of RAM & 50 GB+ of drive space

Software Requirements:

  • You should have the latest Java software installed as a prerequisite. Since Jenkins runs on Java, you need either the latest version of Java Development Kit (JDK) or Java Runtime Environment (JRE).
  • You should have access to install Software on Windows Server.

You can refer to the prerequisites of Jenkins in Jenkins.io

STEP 2: Choose the type of Jenkins download 

Jenkins releases two types of versions based on the organization’s needs. The first one is the Long-term support release & Weekly release. First, you need to download the latest Jenkins software from Download Page. At the time of writing this article, Jenkins 2.289.3 is the latest version. This might be different for you.

Long-term support releases are available every 12 weeks. They are stable and are widely tested. This release is intended for end users.

Weekly releases are made available every week by fixing bugs in its earlier version. These releases are intended towards plugin developers.

For this article, we will use the LTS, and more of the steps will remain the same for the Weekly release.

STEP 3: Download the Jenkins tool

First you need to download the latest Jenkins software from Download Page. At the time of writing this article, Jenkins 2.346.2 is the latest version (May you will find the different version during your time).

STEP 4: Double-click on Downloaded setup

Go to download location from the local computer (unzip) and Double-click on jenkins.msi. You can also Jenkin using a WAR (Web application archive) but that is not recommended.

STEP 5: In the Jenkin Setup screen, click Next.

STEP 6: Choose the Installation location

Choose the location where you want to have the Jenkins instance installed (default location is C:\Program Files (x86)\Jenkins), then click on the Next button.

STEP 8: Service Logon

You need to provide user account credentials to run Jenkins as Independent Windows Service. These can be done using two different ways – run service as LocalSystem or run service as local or domain user. Usually, it is recommended to use a local or domain user to run the Jenkins Service but here we will run the service as LocalSystem. Then Click on Next.

STEP 9: Choose your port

By default Port 8080 will be used for running Jenkins Service but you can always change this port as per your need. Once port detail is given you can quickly check its availability by clicking on Test Port. If testing goes successful then Click on Next.

SETP 10: Select Java Home Directory

Since Jenkins requires Java Runtime Environment to run so here you need to provide the JRE path to proceed with the installation. 

STEP 11: Choose a Custom setup

If you want to install any other features during the installation process then you can select them from here and click on Next. You can also select and install other features post-Jenkins Installation.

STEP 12: Start the installation

If you see the below window then it means Jenkins is finally ready to install. You can now just click on Install to begin the installation process.

Post-installation you can verify the Jenkins service is running in the services. If you have any separate service account then you can choose “Run service as local or domain user” option during STEP 8 and provide the service account username &password instead of “run service as LocalSystem”.

Step 13: Unlock Jenkins

Type http://localhost:portno (here I configured with 8080 during the installation) in the browser. To ensure Jenkins is started securely by an administrator account, a password is written on C:\Windows\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\secrets\initialAdminPassword file needs to be given in the below screen to Continue.

Step 14: Choose the Customize Jenkins

You can choose either “Install suggested plugin” or “Select Plugins to install ” ( this option will not install any plugin and you can install based on your organization’s needs). For my setup, I have chosen the first option.

Step 15: Create an Admin User

You can also create an admin user account in case you don’t want to proceed as an admin. Here we are creating a user cyber hub and going to use the same for accessing Jenkins Server. In case you don’t want to create any user, you can click on Skip and continue as admin.

Step 16: Instance Configuration

Then you need to set up the Jenkins URL below Window. Here we will use the below default URL. So we leave it as it is and then click on Save and Finish.

Step 17: Using Jenkins in the browser

You will see a Jenkins Dashboard like below logged in with cyberithub account and now you can start creating your Job.

 

 

How to Convert YAML to JSON / JSON to YAML using PowerShell

A few days back my team member asked, Is there an easy way to convert YAML to JSON and JSON to YAML using scripts? As expected, my answer was: Yes, with PowerShell!

In some scenarios like in configuration management and Infrastructure as Code (IaC) spaces, use JSON or YAML files to store configuration data. This article is for you if you need to convert data from YAML to JSON format. In this article, you will learn ways to convert data YAML to JSON format & JSON to YAML format. To create the YAML to JSON conversion PowerShell script, follow these instructions. This PowerShell module is a thin wrapper on top of YamlDotNet that serializes and un-serializes simple PowerShell objects to and from YAML.

Converting YAML to JSON format:

STEP:1 Get the YAML file path and assign it to Variable

$YAMLFilePath = “C:\dotnet-helper\Build&Release.yml”

STEP:2 Get the YAML Content using Get-Content Cmdlet

$YAMLContent = Get-Content -Path $YAMLFilePath

STEP:3 Convert the YAML file to Hash Value using ConvertFrom-Yaml cmdlet

Read the YAML file using Get-Content and convert the data to a hashtable using ConvertFrom-Yaml (as shown below script).

hash_Value = ($YAMLContent | ConvertFrom-Yaml)

STEP:4 Convert the hashtable object to JSON format using ConvertTo-Json cmdlet & Finally save the file using Set-Content.

Set-Content -Path “C:\dotnet-helper\Build&Release.json” -Value ($hash_Value| ConvertTo-Json)

Full Code:

#Get the YAML file path and assign to Variable
$YAMLFilePath = "C:\dotnet-helpers\YML_Build&Release.yml"

#Get the YAML Content using Get-Content Cmdlet
$YAMLContent = Get-Content -Path $YAMLFilePath

#Convert the YAML file to Hash Value using ConvertFrom-Yaml cmdlet
$hash_Value = ($YAMLContent | ConvertFrom-Yaml)

#Convert the hashtable object to JSON format using ConvertTo-Json
#Finally save the save the file using Set-Content.
Set-Content -Path "C:\dotnet-helpers\JSON_Build&Release.json" -Value ($hash_Value| ConvertTo-Json)

OUTPUT:

Converting JSON to YAML format:

STEP:1 Get the JSON file path and assign it to Variable

$JsonFilePath = “C:\dotnet-helpers\JSON_Build&Release.JSON”

STEP:2 Get the JSON Content using Get-Content Cmdlet

$JsonContent = Get-Content -Path $JsonFilePath

STEP:3 Convert the JSON file to Hash Value using the ConvertFrom-Json cmdlet

Read the JSON file using Get-Content and convert the data to a hashtable using ConvertFrom-Json (as shown below script).

$hash_Value = ($JsonContent | ConvertFrom-Json)

STEP:4 Convert the hashtable object to YML format using ConvertTo-Yaml cmdlet & Finally save the file using Set-Content.

Set-Content -Path “C:\dotnet-helpers\YML_Build&Release.yml” -Value ($hash_Value| ConvertTo-Yaml)

Full Code:

#Get the JSON file path and assign to Variable
$JsonFilePath = "C:\dotnet-helpers\JSON_Build&Release.JSON"

#Get the YAML Content using Get-Content Cmdlet
$JsonContent = Get-Content -Path $JsonFilePath

#Convert the YAML file to Hash Value using ConvertFrom-Json cmdlet
$hash_Value = ($JsonContent | ConvertFrom-Json)

#Convert the hashtable object to YML format using ConvertTo-Yaml
Set-Content -Path "C:\dotnet-helpers\YML_Build&Release.yml" -Value ($hash_Value| ConvertTo-Yaml)

OUTPUT:

Passing Local Variables to Remote PowerShell session

While working with automation in remote computers using PowerShell, you’ll come across many scenarios where you may want to use local variables or input data in a remote session. But when you try to use a local variable in a remote session (inside the PowerShell script block), it may not work or may throw an error because the local variable doesn’t exist in the remote session. The same scenario I got in my automation which leads to writing this post.

To overcome the above scenario, PowerShell provides 3 way of methods to pass local variables to remote sessions. By using the below method, you can use them with Invoke-Command and New-PSSession to run the specific scripts inside the remote machine.

The script runs in a different scope on the remote system and it is not aware of the variables in the local session. In the example above, $local is actually $null in the remote session because it was never defined in that scope.

Note: Variables defined outside of Invoke-Command will not be available in the Invoke-Command scriptblock unless you explicitly pass them through or reference them.

  • -ArgumentList parameter
  • param() block
  • $using scope modifier

-ArgumentList parameter & Param block

One way to pass local variables to a remote scriptblock is to use the Invoke-Command ArgumentList parameter. This parameter allows you to pass local variables to the parameter and replace local variable references in the scriptblock with placeholders.

The $args variable is an automatic variable that contains an array of the undeclared parameters or parameter values passed to a function or a scriptblock. You can access the parameters passed to the scriptblock just like elements of an array( ArgumentList parameter is an object collection. Object collections allow you to pass one or more objects at a time.) In this example, I’m just passing one.

Example: 1 Using the $args automatic variable

$username = $serviceUsername $password = $servicePwd 
$buildNo = $BUILD_NUMBER 
$userPassword = ConvertTo-SecureString -String 
$password -AsPlainText -Force 
$userCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $userPassword

$arguments = @("dotnet-helpers","powershell")
Invoke-Command -ComputerName $iP -ScriptBlock {
$args
foreach ($value in $args)
{
write-host $value
}
} -credential $userCredential -ArgumentList $arguments

Note: The first parameter passed to the scriptblock is $args[0], the second is $args[1], and so on. 

Example: 2 Using Param with -ArgumentList (Parameterized scriptblocks using param())

You can use param() blocks inside a function to accept parameters, but the only difference between a function and a scriptblock is that a function is named scriptblock. That means that just like in a function, we can use a param() block in a scriptblock. Then we will use this scriptblock with the Invoke-Command cmdlet and pass parameters through the -ArgumentList cmdlet.

$username = $serviceUsername
$password = $servicePwd
$buildNo = $BUILD_NUMBER

$userPassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$userCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $userPassword

Invoke-Command -ComputerName $iP -ScriptBlock {

param($buildNo)
Write-host $buildNo

} -credential $userCredential -ArgumentList $buildNo

$Using Construct

From PowerShell 3, you can use local variables in remote sessions by utilizing the $Using modifier. The command uses the Using scope modifier to identify a local variable in a remote command.

$buildNo variable that is prefixed by the Using scope modifier to indicate that it was created in the local session, not in the remote session.

Example:

$buildNo = "2010"

Invoke-Command -ComputerName Server01 -ScriptBlock {

Write-host $Using:buildNo

}

 

How to remotely get computer CPU and memory usage

Requirement: One of my clients requested to check the status of the CPU utilization and Memory usage before starting the deployment. If any utilization is beyond the threshold (by taking the N samples of utilization with N intervals), the deployment process needs to stop in the remote machine and share the email notification with the support team.

Post by automation, though to write the post on the same and with sample examples. In Windows PowerShell, there is no exclusive cmdlet to find out the CPU and memory utilization rates. You can use the get-wmi object cmdlet along with the required parameters to fetch the Memory results.

STEP #1: Get the IP and hostname of the Remote server 

Param ( [string]$iP, [string]$hostName )

STEP #2: Assign the username and password to connect the server to execute the remote script

$username = $env:serviceUsername
$password = $env:servicePwd
$userPassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$userCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $userPassword

STEP #3: Use Invoke-Command to connect the server and execute the script inside the connected Machine.

In the Invoke-Command pass the IP of the remote server which going to check the CPU & Memory usage.

Invoke-Command -ComputerName $iP -ScriptBlock {
#…Enter your Logic
} -credential $userCredential

STEP 4: Get the sample CPU usage with an interval of 2 seconds and Find the average.

We can achieve this by using the Get-Counter cmdlet in Powershell. It will get performance counter data directly from the performance monitoring instrumentation in the Windows family of operating systems. Get-Counter gets performance data from a local computer or remote computers. For this example, we fetched the CPU samples 5 times at 2-sec Intervals.

$CPUAveragePerformance = (GET-COUNTER -Counter “\Processor(_Total)\% Processor Time” -SampleInterval 2 -MaxSamples 5 |select -ExpandProperty countersamples | select -ExpandProperty cookedvalue | Measure-Object -Average).average

Write-Host “Average of CPU usage (calculated with 5 Sample with an interval of 2 sec):” $CPUAveragePerformance

STEP 5: Get the Memory usage using Get-WmiObject cmdlet.

$ComputerMemory = Get-WmiObject -ComputerName $env:COMPUTERNAME -Class win32_operatingsystem -ErrorAction Stop

$Memory = ((($ComputerMemory.TotalVisibleMemorySize – $ComputerMemory.FreePhysicalMemory)*100)/ $ComputerMemory.TotalVisibleMemorySize)

$RoundMemory = [math]::Round($Memory, 2)
Write-Host “Memory usage percentage of the system :” $RoundMemory

You can save this script as .PS1 (DEVMachine_ServerHealthCheck.ps1) and call in automation bypassing the IP & Hostname as parameters to get the result.

Full Code:

#Check the CPU & Memory Usage in the Remote Machine

Param ( [string]$iP, [string]$hostName  )

$username = $env:serviceUsername
$password = $env:servicePwd
$userPassword = ConvertTo-SecureString -String $password -AsPlainText -Force
$userCredential = New-Object -TypeName System.Management.Automation.PSCredential -ArgumentList $username, $userPassword

try {

Invoke-Command -ComputerName $iP -ScriptBlock { 
$CPUAveragePerformance = (GET-COUNTER -Counter "\Processor(_Total)\% Processor Time" -SampleInterval 2 -MaxSamples 5 |
select -ExpandProperty countersamples | 
select -ExpandProperty cookedvalue | Measure-Object -Average).average
Write-Host "Average of CPU usage (calculated with 5 interval of 2 sec) :" $CPUAveragePerformance

$ComputerMemory = Get-WmiObject -ComputerName $hostName -Class win32_operatingsystem
$Memory = ((($ComputerMemory.TotalVisibleMemorySize - 
$ComputerMemory.FreePhysicalMemory)*100)/ $ComputerMemory.TotalVisibleMemorySize) 
$RoundMemory = [math]::Round($Memory, 2)
Write-Host "Memory usage percentage of the system :" $RoundMemory

}  -credential $userCredential
}
catch{
Write-Host $error[0]
}

Output:

Downloading a file with PowerShell

Scenario:

We got a requirement to download the Excel files on a daily basis from the webpage and need to process the same automatically (window scheduled jobs) and share the report with my supervisors. If we manage this on daily basis by manual and it will occupy some resource bandwidth each day and there is a chance to have some manual error, to avoid this we thought to spend time with automation to complete this task.

In simple, If you need to download files from the web by repeatedly clicking links and processing the data in it on daily basis then you will probably want to automate the task. Windows PowerShell and PowerShell come with file-download capabilities. Using PowerShell to download files is a matter of knowing which cmdlets and .NET classes to use and how to use them

Download File Using PowerShell:

In this discussion, I am going to show how to download a file from a URL using PowerShell and this can be achieved using the Invoke-WebRequest method. We can use Invoke-WebRequest/Invoke-RestMethod/Start-BitsTransfer to download the files in the PowerShell. Whichever one of these methods you use, the logic and components to make them work are the same. To download a file we need to know the source URL and give up a destination for the file that we want to download. If required by the webserver, you need to enter the credentials as well. You can also download the files by using Invoke-RestMethod & Start-BitsTransfer

For our example, I am going to use the Invoke-WebRequest method for downloading the files from the Web. If the source location requires users to log in, the Invoke-WebRequest cmdlet can handle requests with credentials as well.

To download a file, the syntax below shows the minimum parameters required to achieve our requirement. To download the file, the parameter -OutFile is required. You don’t need to enter the full path, but a file name is required. The Outfile expects a path and a filename.

Invoke-WebRequest -Uri <source> -OutFile <destination>

STEP #1 Get the Souce path

# Source URL
$sourceURL = “https://filesamples.com/samples/document/txt/sample3.txt”

STEP #2 Get the Destination path

# Destation file
$destPath = “C:\Thiyagu Disk\dotnet-helpsers\output.txt”

STEP #3  Download the file using Invoke Method.

# Download the file
Invoke-WebRequest -Uri $sourceURL -OutFile $destPath

Full Code:

# Assign SoruceURL and Destination Path
$sourceURL = "https://filesamples.com/samples/document/txt/sample3.txt"
$destPath = "C:\Thiyagu Disk\BLOG_2022\output.txt"
# USe Invoke Method to download the File
Invoke-WebRequest -Uri $sourceURL -OutFile $destPath

Output: 

Authentication with Invoke-WebRequest

Some URLs require you to log in before you can access/download the files. With the Invoke-WebRequest cmdlet, we can provide the credentials that are needed for downloading the files. For this example, i am using the Credential directly in the script. If you are creating a script for automation (that will need to run automatically), then you need to store the credentials in the script itself.

Full Code:

# Assign SoruceURL and Destination Path
$sourceURL = "https://filesamples.com/samples/document/txt/sample3.txt"
$destPath = "C:\Thiyagu Disk\BLOG_2022\output.txt"
# Assign the Username and Pwd to access the $sourceURL
$username = 'XXXXX'
$password = 'XXXXX'
# Convert to SecureString
$secPassword = ConvertTo-SecureString $password -AsPlainText -Force
# Create Credential Object
$credObject = New-Object System.Management.Automation.PSCredential ($username, $secPassword)
# USe Invoke Method to download the File
Invoke-WebRequest -Uri $sourceURL -OutFile $destPath

Points to Remember:

  • The Invoke-WebRequest doesn’t perform a check if the target file exists. If the file already exists, it is overwritten without any warning.
  • If you need to authenticate under the current user on a remote website (via NTLM or Kerberos), you need to add the –UseDefaultCredentials parameter
  • You can also download the files by using Invoke-RestMethod & Start-BitsTransfer

 

 

Where to use the –replace operator and Replace() method & its difference

In this post, you’re going to learn where to use the PowerShell replace() method and PowerShell replace operator. The tutorial will cover the basics and even drive into some regular expressions.

.Replace is a .NET method and -replace is a PowerShell operator that uses regular expressions. In another word, the .Replace() method comes from the .NET String class whereas the -Replace operator is implemented using System.Text.RegularExpressions.Regex.Replace().

The -replace operator takes a regular expression (regex) replacement rule as input and replaces every match with the replacement string.

When and where to use it?

Like other languages, PowerShell can work with strings and text. One of those useful features is to use PowerShell to replace characters, strings, or even text inside of files. In PowerShell, Replace() method and -replace operator is used to finding specified characters and replace them with a new string. To perform simple replacements, you can use the replace() method but if you need to match and replace anything more advanced, always use the replace operator.

.Replace Method:

Example 1: Replace characters in strings.

$string = ‘hello, dotnet-helpers.com’

In the above code, we gave the string that likes to replace. From the above string value, we would like to replace “hello” with “Welcome”. To do this in PowerShell, first you need to figure out the matched text. Once it’s found, need to replace that text with a user-defined value.

$string = ‘hello, dotnet-helpers.com’
$string.replace(‘hello’,’Welcome’)

The replace() method has two arguments; the string to find and the string to replace. As shown above, the “hello” string is going to replace with “Welcome”.

Points to Remember:

You can call the replace() method on any string to replace any literal string with another. If the string-to-be-replaced isn’t found, the replace() method returns nothing.

Example 2: Replacing multiple strings

You aware that replace() method returns a string, to replace another instance, you can append another replace() method at the end ( .replace(‘dotnet-helpers’,’dotnet-helpers.com!!!’) ). In the previous example, we try to replace “hello” with “Welcome”, in this example we trying to replace another string with one more .replace method as shown below.

$string = ‘hello, dotnet-helpers’
$string.replace(‘hello’,’welcome’).replace(‘dotnet-helpers’,’dotnet-helpers.com!!!’)

Points to Remember:

You can chain together as many replace() method calls as necessary

-Replace Operator:

The replace operator is similar to the .Replace method (in that you provide a string to find and replace). But, it has one big advantage; the ability to use regular expressions to find matching strings.

Example 1: Replacing single string

$string = ‘hello, dotnet-helpers.com’
$string -replace ‘hello,’, ‘Welcome to’

Example 2: Replacing multiple strings

Like the replace() method, you can also chain together usages of the replace operator.

$string = ‘hello, dotnet-helpers’
$string -replace ‘hello,’,’Welcome to’ -replace ‘dotnet-helpers’,’dotnet-helpers.com!!!’

-Replace Operator with Regex:

Replacing strings in PowerShell with the replace() method works but it’s limited. You are constrained to only using literal strings. You cannot use wildcards or regex. If you’re performing any kind of intermediate or advanced replacing, you should use the replace operator.

The -replace operator takes a regex (regular expression) replacement rule as input and replaces every match with the replacement string. The operator itself is used as shown in the following examples : <input string> -replace <replacement rule>,<replacement string>

 

Example 1: With Simple Regex

In this example, you can use the expression hello|hi to match both required strings using the regex “or” (|) character as you can see below. In the below regex, it finds the match for a string like “hello” or “hi” and if a match is found it will replace with the given string.

$string = ‘hi, dotnet-helpers.com’
$string -replace ‘hello|hi’,’Good day’

Example 2: Direct Replace of special character

As per the below example, you need to replace text in a string. That string contains a couple of regex special characters like a bracket and an exclamation mark. If you try to replace the string [dotnethelpers!] with dotnet-helpers.com as shown below, then it will not work as expected because the characters will have special meaning in regex language.

$string = “hi, [dotnethelpers!]”
$string -replace ‘[dotnethelpers!]’,’dotnet-helpers.com’

The problem is you often need to replace “[]”, “!” or other characters that have special meaning in regex language. One way to achieve this is to escape every special character by “\”.

To overcome this problem, you have two options. You can either escape these special characters by prepending a backslash to the front of each character or using the Escape() method (([regex]::Escape(‘[dotnethelpers]’)).

Points to Remember:

If you try to replace any special characters directly from the string using Replace operator then it won’t work correctly as characters will have special meaning in regex language.

Conclusion :

Replacing characters or words in a string with PowerShell is easily done using either the replace method or -replace operator. When working with special characters, like [ ], \ or $ symbols, it’s often easier to use the replace() method than the operator variant. Because this way you don’t need to escape the special character.

To perform simple replacements, you can use the replace() method but if you need to match and replace anything more advanced, always use the replace operator.

How to Extract Specific Files from ZIP Archive using PowerShell

When is this required (Real-time Scenario)?

In our blog, we already discussed the zip and unzip files using Powershell. In this post, we are going to discuss how to extract specific files from the zip. I have several zip files that Contain multiple file types and my team got one requirement for extracting only specific files (like .txt/.wav/.xls, file name filter) alone from the Zip on weekly basis and need to ignore all other file types in the zips. If we go for manual then it will be time taking process to check all the zip files on a weekly basis. so we decided to make this automated.

Sample Folder structure used for this example:

How to achieve?

In this example, we are going to use .NET method. To make this process of separating the specific files from the Zip, We need to specify the ‘entry’ object within the .zip file object and pass that to the ExtractToFile() method. Below are the 4 main steps for extracting specific files from the Zip.

  • Fetch the ZIP file and open it for reading.
  • Identifies all files inside the ZIP file.
  • Fetch the files based on the given extension/any matched filters & copy them to the destination folder.
  • Finally, close the zip.

STEP #1: Fetch the ZIP file and open it for reading.

First, we need to open a zip archive for reading (As we are using .NET, we must first open it for reading) at the specified path using the dotnet ZipFile.OpenRead(String) Method (Namespace: System.IO.Compression).

Add-Type -Assembly System.IO.Compression.FileSystem
$zipFile = [IO.Compression.ZipFile]::OpenRead($sourceFilePath)

STEP #2: Identifies all files inside the ZIP file

Next, we need to get all the entries and start identifying/filtering the files. In this example, we are going to fetch all the txt files from the zip.

$zipFile.Entries | where {$_.Name -like ‘*.txt’}

STEP #3: Fetch all the files based on the filters & copy them to the destination folder.

Now you can extract the selected items from the ZIP archive and copy them to the output folder.

foreach {
$FileName = $_.Name
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, “$destPath\$FileName”, $true)
}

Note: To only extract a single file from a .zip file, things get a little trickier. We need to specify the ‘entry’ ($zipFile.Entries[0]) object within the .zip file object and pass that to the ExtractToFile() method.   

[System.IO.Compression.ZipFileExtensions]::ExtractToFile($zipFile.Entries[0], “$destPath\ExtractMe1.txt”, $true)

To extract multiple files the target file needs to be composed for each.

$zipFile.Entries | Where-Object Name -like *.txt | ForEach-Object{[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, “$extractPath\$($_.Name)”, $true)}

STEP #4: Close/Dispose the Zip

$zipFile.Dispose()

Final Code:

 
############################################################################################
#Project: How to Extract Specific Files from ZIP Archive in PowerShell
#Developer: Thiyagu S (dotnet-helpers.com)
#Tools : PowerShell 5.1.15063.1155 [irp]
#E-Mail: mail2thiyaguji@gmail.com
############################################################################################

$destPath = "C:\dotnet-helpers\Destination\"
$sourcePath = 'C:\dotnet-helpers\Source\ExtractMe.zip'

# load ZIP methods
Add-Type -Assembly System.IO.Compression.FileSystem

# open ZIP archive for reading
$zipFile = [IO.Compression.ZipFile]::OpenRead($sourcePath)

#Find all files in ZIP that match the filter (i.e. file extension)
#Use the Entries property to retrieve the entire collection of entries

$zipFile.Entries | where {$_.Name -like '*.txt'} | foreach {$FileName = $_.Name
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, "$destPath\$FileName", $true)}

# close ZIP file
$zipFile.Dispose()

Output