We are maintaining 500+ client websites in our environment. Some day before we received a request to get the list of links/Images used on each home page. We knew that it will be very tricky to get the list of links/URLs mapped in the 500+ pages and you are also aware that the annual work will not give 100% results.
So we decided to use Powershell Links in the Invoke-WebRequest methodto reduce manual effort. In this post, we will discuss the same with a simple example using a single URL. For checking the multiple URLs, please refer to a similar article which helps to read from excel and loop it –
PowerShell’s Invoke-WebRequest is a powerful cmdlet that allows you to download, parse, and scrape web pages. The Invoke-WebRequest cmdlet is used to download files from the web via HTTP and HTTPS. However, this cmdlet enables you to do more than download files. You can use this cmdlet for analyzing the contents of web pages.
Example: Get the list of URLs
The below script will grab the innerText in addition to the corresponding links
A batch file is a series of commands or a script in the Windows Operating System that executes a series of tasks on the local or remote machines and it has a. Bat extension. Although PowerShell and Batch are different languages, both can be integrated into each other and helps to call and execute each other.
This article will illustrate different ways to run a Batch file from a PowerShell script.
Step #1 Create a .bat file with commands
For time being, I only used the echo command in the bat file, you can write your logic command as required. For this example, I save the file as CallMe.bat.
echo Write something, it will be used in the command “echo” pause
Step #2 Create a Powershell script file & call the .bat file
There are many approaches we can call the .bat file from PowerShell, which we can choose based on our purpose. This step will guide you on how to Add the Batch file to a PowerShell script to run automatically when the PowerShell script is being run.
Method: 1
One way of running a Batch file from the PowerShell script is using the Start-Process cmdlet. The Start-Process cmdlet allows you to run one or multiple processes on your computer from within PowerShell. It’s designed to run a process asynchronously or to run an application/script elevated (with administrative privileges).
To run the Batch file, add the following line of code to the PowerShell script:
–Filepath specifies the path of the Batch file. –NoNewWindow starts the process in the current window (add this at end of the script to mention not to open the cmd window).
Start-Process -FilePath ‘C:\blog\callme.bat’
To run the Batch file as administrator, add -verb run as in the above code. This command is useful when your .bat file contains commands which require administrator privileges to run on execution.
In our first Jenkins tutorial for beginners, we focused on how to install and configure Jenkins. In this tutorial, How To Create Your First simple Jenkins Pipeline. We’ll keep it simple and avoid Maven or Git at this juncture. We’ll just create a Jenkins freestyle job that invokes the JDK’s runtime instance and prints out the version of the JRE that is currently running in the Jenkins machine (location of Jenkins installation).
STEP: 1 Login into Jenkins and go to Jenkins dashboard
To create a Jenkins freestyle job, log on to your Jenkins dashboard by visiting your Jenkins installation path. Usually, it will be hosted on localhost at http://localhost:8080 If you have installed Jenkins in another path, use the appropriate URL to access your dashboard as shown in the below Jenkins job creation example.
STEP: 2 Create New Item (New job/pipeline)
The first step to creating a Jenkins build job is to click the New Item link in the top left-hand corner of the admin console and enter the Item name & click Ok.
Note: If you are unable to see this icon, it means that you don’t have sufficient privileges. In the next window, type the name of the job such as the first job, select job type as freestyle job, and then click ok:
STEP 3: Configure the new job details
After clicking OK ( in STEP 3), the configuration page for the freestyle Jenkins job will appear as shown in the below snapshot. Notice there are a number of options to configure, including build triggers, source code management options, Jenkins build job steps, and post-build actions. As we mentioned in starting of the post, we are only going to create simple jobs without doing any build or deployment.
In the Jenkins build job, we will change the description to “My_First_JenkinsJob” and Under the “Source Code Management” section, for this example, we are not going to use any GIT URL to download the solution so you can select “None”
In this job/pipeline, we going the check the java version that was installed in our local/Jenkins machine. For this, in the “Build section” choose “Execute Windows batch command” from the drop-down and type the “java -version” in the window batch command section.
STEP 4: Save the job and click on the Build Now link.
Now your Jenkins Job is ready for checking the version of Java installed on the Jenkins machine.
STEP 5: Save the job, click on the Build icon & Check the status.
To run the created job, click on the job which you need to build. Once the new job is opened as shown in the below snapshot, click the “Build Now” to start the job execution. You can check the build status under the “Build History section” at the left bottom of the screen.
STEP 6: View the log for checking the output
In the window batch command, we placed the cmd to check the version of java. So the same has been executed and you can able to see the output in the log file as shown in the below image.
Based on this simple example, I now hope you can able to Create Your First simple Jenkins Pipeline.
In this article, we will go through the steps to download and install Jenkins on Windows. Jenkins is a free and open-source automation software used for building, testing, and deploying code to achieve the end goal of Continuous Delivery and Continuous Integration. It provides faster and more efficient code deployment in multiple environments. Jenkins supports a wide range of plugins due to which it can deploy almost any kind of code to any environment.
What is Jenkins?
Jenkins is a self-contained, open source (DevOps tool) automation server which can be used to automate all sorts of tasks related to building, testing and delivering or deploying software. Jenkins can be installed through native system packages, Docker, or even run standalone by any machine with a Java Runtime Environment (JRE) installed.
Jenkins may be installed on either Windows/Unix/supported platforms, but we will focus only install Jenkins on a Windows machine in the article (the below steps explain the installation in the standalone machine).
STEP 1: Prerequisites
Before you proceed to install Jenkins in your windows/Unix system, there are some prerequisites for Jenkins to install Jenkins on your computer.
Hardware requirements:
Hardware requirements (Minimum): 256 MB of RAM, 1 GB of drive space (although 10 GB is a recommended minimum if running Jenkins as a Docker container)
Hardware configuration for a small team (Recommended) : 4 GB+ of RAM & 50 GB+ of drive space
Software Requirements:
You should have the latest Java software installed as a prerequisite. Since Jenkins runs on Java, you need either the latest version of Java Development Kit (JDK) or Java Runtime Environment (JRE).
You should have access to install Software on Windows Server.
You can refer to the prerequisites of Jenkins in Jenkins.io
STEP 2: Choose the type of Jenkins download
Jenkins releases two types of versions based on the organization’s needs. The first one is the Long-term support release & Weekly release. First, you need to download the latest Jenkins software from Download Page. At the time of writing this article, Jenkins 2.289.3 is the latest version. This might be different for you.
Long-term support releases are available every 12 weeks. They are stable and are widely tested. This release is intended for end users.
Weekly releases are made available every week by fixing bugs in its earlier version. These releases are intended towards plugin developers.
For this article, we will use the LTS, and more of the steps will remain the same for the Weekly release.
STEP 3: Download the Jenkins tool
First you need to download the latest Jenkins software from Download Page. At the time of writing this article, Jenkins 2.346.2 is the latest version (May you will find the different version during your time).
STEP 4: Double-click on Downloaded setup
Go to download location from the local computer (unzip) and Double-click on jenkins.msi. You can also Jenkin using a WAR (Web application archive) but that is not recommended.
STEP 5: In the Jenkin Setup screen, click Next.
STEP 6: Choose the Installation location
Choose the location where you want to have the Jenkins instance installed (default location is C:\Program Files (x86)\Jenkins), then click on the Next button.
STEP 8: Service Logon
You need to provide user account credentials to run Jenkins as Independent Windows Service. These can be done using two different ways – run service as LocalSystem or run service as local or domain user. Usually, it is recommended to use a local or domain user to run the Jenkins Service but here we will run the service as LocalSystem. Then Click on Next.
STEP 9: Choose your port
By default Port 8080 will be used for running Jenkins Service but you can always change this port as per your need. Once port detail is given you can quickly check its availability by clicking on Test Port. If testing goes successful then Click on Next.
SETP 10: Select Java Home Directory
Since Jenkins requires Java Runtime Environment to run so here you need to provide the JRE path to proceed with the installation.
STEP 11: Choose a Custom setup
If you want to install any other features during the installation process then you can select them from here and click on Next. You can also select and install other features post-Jenkins Installation.
STEP 12: Start the installation
If you see the below window then it means Jenkins is finally ready to install. You can now just click on Install to begin the installation process.
Post-installation you can verify the Jenkins service is running in the services. If you have any separate service account then you can choose “Run service as local or domain user” option during STEP 8 and provide the service account username &password instead of “run service as LocalSystem”.
Step 13: Unlock Jenkins
Type http://localhost:portno (here I configured with 8080 during the installation) in the browser. To ensure Jenkins is started securely by an administrator account, a password is written on C:\Windows\system32\config\systemprofile\AppData\Local\Jenkins\.jenkins\secrets\initialAdminPassword file needs to be given in the below screen to Continue.
Step 14: Choose the Customize Jenkins
You can choose either “Install suggested plugin” or “Select Plugins to install ” ( this option will not install any plugin and you can install based on your organization’s needs). For my setup, I have chosen the first option.
Step 15: Create an Admin User
You can also create an admin user account in case you don’t want to proceed as an admin. Here we are creating a user cyber hub and going to use the same for accessing Jenkins Server. In case you don’t want to create any user, you can click on Skip and continue as admin.
Step 16: Instance Configuration
Then you need to set up the Jenkins URL below Window. Here we will use the below default URL. So we leave it as it is and then click on Save and Finish.
Step 17: Using Jenkins in the browser
You will see a Jenkins Dashboard like below logged in with cyberithub account and now you can start creating your Job.
A few days back my team member asked, Is there an easy way to convert YAML to JSON and JSON to YAML using scripts? As expected, my answer was: Yes, with PowerShell!
In some scenarios like in configuration management and Infrastructure as Code (IaC) spaces, use JSON or YAML files to store configuration data. This article is for you if you need to convert data from YAML to JSON format. In this article, you will learn ways to convert data YAML to JSON format & JSON to YAML format. To create the YAML to JSON conversion PowerShell script, follow these instructions. This PowerShell module is a thin wrapper on top of YamlDotNet that serializes and un-serializes simple PowerShell objects to and from YAML.
Converting YAML to JSON format:
STEP:1 Get the YAML file path and assign it to Variable
#Get the YAML file path and assign to Variable
$YAMLFilePath = "C:\dotnet-helpers\YML_Build&Release.yml"
#Get the YAML Content using Get-Content Cmdlet
$YAMLContent = Get-Content -Path $YAMLFilePath
#Convert the YAML file to Hash Value using ConvertFrom-Yaml cmdlet
$hash_Value = ($YAMLContent | ConvertFrom-Yaml)
#Convert the hashtable object to JSON format using ConvertTo-Json
#Finally save the save the file using Set-Content.
Set-Content -Path "C:\dotnet-helpers\JSON_Build&Release.json" -Value ($hash_Value| ConvertTo-Json)
OUTPUT:
Converting JSON to YAML format:
STEP:1 Get the JSON file path and assign it to Variable
#Get the JSON file path and assign to Variable
$JsonFilePath = "C:\dotnet-helpers\JSON_Build&Release.JSON"
#Get the YAML Content using Get-Content Cmdlet
$JsonContent = Get-Content -Path $JsonFilePath
#Convert the YAML file to Hash Value using ConvertFrom-Json cmdlet
$hash_Value = ($JsonContent | ConvertFrom-Json)
#Convert the hashtable object to YML format using ConvertTo-Yaml
Set-Content -Path "C:\dotnet-helpers\YML_Build&Release.yml" -Value ($hash_Value| ConvertTo-Yaml)
While working with automation in remote computers using PowerShell, you’ll come across many scenarios where you may want to use local variables or input data in a remote session. But when you try to use a local variable in a remote session (inside the PowerShell script block), it may not work or may throw an error because the local variable doesn’t exist in the remote session. The same scenario I got in my automation which leads to writing this post.
To overcome the above scenario, PowerShell provides 3 way of methods to pass local variables to remote sessions. By using the below method, you can use them with Invoke-Command and New-PSSession to run the specific scripts inside the remote machine.
The script runs in a different scope on the remote system and it is not aware of the variables in the local session. In the example above, $local is actually $null in the remote session because it was never defined in that scope.
Note: Variables defined outside of Invoke-Command will not be available in the Invoke-Command scriptblock unless you explicitly pass them through or reference them.
-ArgumentList parameter
param() block
$using scope modifier
-ArgumentList parameter & Param block
One way to pass local variables to a remote scriptblock is to use the Invoke-Command ArgumentList parameter. This parameter allows you to pass local variables to the parameter and replace local variable references in the scriptblock with placeholders.
The $args variable is an automatic variable that contains an array of the undeclared parameters or parameter values passed to a function or a scriptblock. You can access the parameters passed to the scriptblock just like elements of an array( ArgumentList parameter is an object collection. Object collections allow you to pass one or more objects at a time.) In this example, I’m just passing one.
Note: The first parameter passed to the scriptblock is $args[0], the second is $args[1], and so on.
Example: 2 Using Param with -ArgumentList (Parameterized scriptblocks using param())
You can use param() blocks inside a function to accept parameters, but the only difference between a function and a scriptblock is that a function is named scriptblock. That means that just like in a function, we can use a param() block in a scriptblock. Then we will use this scriptblock with the Invoke-Command cmdlet and pass parameters through the -ArgumentList cmdlet.
From PowerShell 3, you can use local variables in remote sessions by utilizing the $Using modifier. The command uses the Using scope modifier to identify a local variable in a remote command.
$buildNo variable that is prefixed by the Using scope modifier to indicate that it was created in the local session, not in the remote session.
Requirement: One of my clients requested to check the status of the CPU utilization and Memory usage before starting the deployment. If any utilization is beyond the threshold (by taking the N samples of utilization with N intervals), the deployment process needs to stop in the remote machine and share the email notification with the support team.
Post by automation, though to write the post on the same and with sample examples. In Windows PowerShell, there is no exclusive cmdlet to find out the CPU and memory utilization rates. You can use the get-wmi object cmdlet along with the required parameters to fetch the Memory results.
STEP #1: Get the IP and hostname of the Remote server
Param ( [string]$iP, [string]$hostName )
STEP #2: Assign the username and password to connect the server to execute the remote script
STEP 4: Get the sample CPU usage with an interval of 2 seconds and Find the average.
We can achieve this by using the Get-Counter cmdlet in Powershell. It will get performance counter data directly from the performance monitoring instrumentation in the Windows family of operating systems. Get-Counter gets performance data from a local computer or remote computers. For this example, we fetched the CPU samples 5 times at 2-sec Intervals.
$RoundMemory = [math]::Round($Memory, 2) Write-Host “Memory usage percentage of the system :” $RoundMemory
You can save this script as .PS1 (DEVMachine_ServerHealthCheck.ps1) and call in automation bypassing the IP & Hostname as parameters to get the result.
We got a requirement to download the Excel files on a daily basis from the webpage and need to process the same automatically (window scheduled jobs) and share the report with my supervisors. If we manage this on daily basis by manual and it will occupy some resource bandwidth each day and there is a chance to have some manual error, to avoid this we thought to spend time with automation to complete this task.
In simple, If you need to download files from the web by repeatedly clicking links and processing the data in it on daily basis then you will probably want to automate the task. Windows PowerShell and PowerShell come with file-download capabilities. Using PowerShell to download files is a matter of knowing which cmdlets and .NET classes to use and how to use them
Download File Using PowerShell:
In this discussion, I am going to show how to download a file from a URL using PowerShell and this can be achieved using the Invoke-WebRequest method. We can use Invoke-WebRequest/Invoke-RestMethod/Start-BitsTransfer to download the files in the PowerShell. Whichever one of these methods you use, the logic and components to make them work are the same. To download a file we need to know the source URL and give up a destination for the file that we want to download. If required by the webserver, you need to enter the credentials as well. You can also download the files by using Invoke-RestMethod & Start-BitsTransfer
For our example, I am going to use the Invoke-WebRequest method for downloading the files from the Web. If the source location requires users to log in, the Invoke-WebRequest cmdlet can handle requests with credentials as well.
To download a file, the syntax below shows the minimum parameters required to achieve our requirement. To download the file, the parameter -OutFile is required. You don’t need to enter the full path, but a file name is required. The Outfile expects a path and a filename.
# Download the file Invoke-WebRequest -Uri $sourceURL -OutFile $destPath
Full Code:
# Assign SoruceURL and Destination Path
$sourceURL = "https://filesamples.com/samples/document/txt/sample3.txt"
$destPath = "C:\Thiyagu Disk\BLOG_2022\output.txt"
# USe Invoke Method to download the File
Invoke-WebRequest -Uri $sourceURL -OutFile $destPath
Output:
Authentication with Invoke-WebRequest
Some URLs require you to log in before you can access/download the files. With the Invoke-WebRequest cmdlet, we can provide the credentials that are needed for downloading the files. For this example, i am using the Credential directly in the script. If you are creating a script for automation (that will need to run automatically), then you need to store the credentials in the script itself.
Full Code:
# Assign SoruceURL and Destination Path
$sourceURL = "https://filesamples.com/samples/document/txt/sample3.txt"
$destPath = "C:\Thiyagu Disk\BLOG_2022\output.txt"
# Assign the Username and Pwd to access the $sourceURL
$username = 'XXXXX'
$password = 'XXXXX'
# Convert to SecureString
$secPassword = ConvertTo-SecureString $password -AsPlainText -Force
# Create Credential Object
$credObject = New-Object System.Management.Automation.PSCredential ($username, $secPassword)
# USe Invoke Method to download the File
Invoke-WebRequest -Uri $sourceURL -OutFile $destPath
Points to Remember:
The Invoke-WebRequest doesn’t perform a check if the target file exists. If the file already exists, it is overwritten without any warning.
If you need to authenticate under the current user on a remote website (via NTLM or Kerberos), you need to add the –UseDefaultCredentials parameter
You can also download the files by using Invoke-RestMethod & Start-BitsTransfer
In this post, you’re going to learn where to use the PowerShell replace() method and PowerShell replace operator. The tutorial will cover the basics and even drive into some regular expressions.
.Replace is a .NET method and -replace is a PowerShell operator that uses regular expressions. In another word, the .Replace() method comes from the .NET String class whereas the -Replace operator is implemented using System.Text.RegularExpressions.Regex.Replace().
The -replace operator takes a regular expression (regex) replacement rule as input and replaces every match with the replacement string.
When and where to use it?
Like other languages, PowerShell can work with strings and text. One of those useful features is to use PowerShell to replace characters, strings, or even text inside of files. In PowerShell, Replace() method and -replace operator is used to finding specified characters and replace them with a new string. To perform simple replacements, you can use the replace() method but if you need to match and replace anything more advanced, always use the replace operator.
.Replace Method:
Example 1: Replace characters in strings.
$string = ‘hello, dotnet-helpers.com’
In the above code, we gave the string that likes to replace. From the above string value, we would like to replace “hello” with “Welcome”. To do this in PowerShell, first you need to figure out the matched text. Once it’s found, need to replace that text with a user-defined value.
The replace() method has two arguments; the string to find and the string to replace. As shown above, the “hello” string is going to replace with “Welcome”.
Points to Remember:
You can call the replace() method on any string to replace any literal string with another. If the string-to-be-replaced isn’t found, the replace() method returns nothing.
Example 2: Replacing multiple strings
You aware that replace() method returns a string, to replace another instance, you can append another replace() method at the end ( .replace(‘dotnet-helpers’,’dotnet-helpers.com!!!’) ). In the previous example, we try to replace “hello” with “Welcome”, in this example we trying to replace another string with one more .replace method as shown below.
You can chain together as many replace() method calls as necessary
-Replace Operator:
The replace operator is similar to the .Replace method (in that you provide a string to find and replace). But, it has one big advantage; the ability to use regular expressions to find matching strings.
Replacing strings in PowerShell with the replace() method works but it’s limited. You are constrained to only using literal strings. You cannot use wildcards or regex. If you’re performing any kind of intermediate or advanced replacing, you should use the replace operator.
The -replace operator takes a regex (regular expression) replacement rule as input and replaces every match with the replacement string. The operator itself is used as shown in the following examples : <input string> -replace <replacement rule>,<replacement string>
Example 1: With Simple Regex
In this example, you can use the expression hello|hi to match both required strings using the regex “or” (|) character as you can see below. In the below regex, it finds the match for a string like “hello” or “hi” and if a match is found it will replace with the given string.
As per the below example, you need to replace text in a string. That string contains a couple of regex special characters like a bracket and an exclamation mark. If you try to replace the string [dotnethelpers!] with dotnet-helpers.com as shown below, then it will not work as expected because the characters will have special meaning in regex language.
The problem is you often need to replace “[]”, “!” or other characters that have special meaning in regex language. One way to achieve this is to escape every special character by “\”.
To overcome this problem, you have two options. You can either escape these special characters by prepending a backslash to the front of each character or using the Escape() method (([regex]::Escape(‘[dotnethelpers]’)).
Points to Remember:
If you try to replace any special characters directly from the string using Replace operator then it won’t work correctly as characters will have special meaning in regex language.
Conclusion :
Replacing characters or words in a string with PowerShell is easily done using either the replace method or -replace operator. When working with special characters, like [ ], \ or $ symbols, it’s often easier to use the replace() method than the operator variant. Because this way you don’t need to escape the special character.
To perform simple replacements, you can use the replace() methodbut if you need to match and replace anything more advanced, always use the replace operator.
In our blog, we already discussed the zipand unzipfiles using Powershell. In this post, we are going to discuss how to extract specific files from the zip. I have several zip files that Contain multiple file types and my team got one requirement for extracting only specific files (like .txt/.wav/.xls, file name filter) alone from the Zip on weekly basis and need to ignore all other file types in the zips. If we go for manual then it will be time taking process to check all the zip files on a weekly basis. so we decided to make this automated.
Sample Folder structure used for this example:
How to achieve?
In this example, we are going to use .NET method. To make this process of separating the specific files from the Zip, We need to specify the ‘entry’ object within the .zip file object and pass that to the ExtractToFile() method. Below are the 4 main steps for extracting specific files from the Zip.
Fetch the ZIP file and open it for reading.
Identifies all files inside the ZIP file.
Fetch the files based on the given extension/any matched filters & copy them to the destination folder.
Finally, close the zip.
STEP #1: Fetch the ZIP file and open it for reading.
First, we need to open a zip archive for reading (As we are using .NET, we must first open it for reading) at the specified path using the dotnet ZipFile.OpenRead(String) Method (Namespace: System.IO.Compression).
Note: To only extract a single file from a .zip file, things get a little trickier. We need to specify the ‘entry’ ($zipFile.Entries[0]) object within the .zip file object and pass that to the ExtractToFile() method.
To extract multiple files the target file needs to be composed for each.
$zipFile.Entries | Where-Object Name -like *.txt | ForEach-Object{[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, “$extractPath\$($_.Name)”, $true)}
STEP #4: Close/Dispose the Zip
$zipFile.Dispose()
Final Code:
############################################################################################
#Project: How to Extract Specific Files from ZIP Archive in PowerShell
#Developer: Thiyagu S (dotnet-helpers.com)
#Tools : PowerShell 5.1.15063.1155 [irp]
#E-Mail: mail2thiyaguji@gmail.com
############################################################################################
$destPath = "C:\dotnet-helpers\Destination\"
$sourcePath = 'C:\dotnet-helpers\Source\ExtractMe.zip'
# load ZIP methods
Add-Type -Assembly System.IO.Compression.FileSystem
# open ZIP archive for reading
$zipFile = [IO.Compression.ZipFile]::OpenRead($sourcePath)
#Find all files in ZIP that match the filter (i.e. file extension)
#Use the Entries property to retrieve the entire collection of entries
$zipFile.Entries | where {$_.Name -like '*.txt'} | foreach {$FileName = $_.Name
[System.IO.Compression.ZipFileExtensions]::ExtractToFile($_, "$destPath\$FileName", $true)}
# close ZIP file
$zipFile.Dispose()
To provide the best experiences, we use technologies like cookies to store and/or access device information. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.