Category Archives: Devops

What Are Environment Variables in Linux?

Environment Variables in Linux are dynamic values that the operating system and various applications use to determine information about the user environment. They are essentially variables that can influence the behavior and configuration of processes and programs on a Linux system. These variables are used to pass configuration information to programs and scripts, allowing for flexible and dynamic system management.

These variables, often referred to as global variables, play a crucial role in tailoring the system’s functionality and managing the startup behavior of various applications across the system. On the other hand, local variables are restricted and accessible from within the shell in which they’re created and initialized.

Linux environment variables have a key-value pair structure, separated by an equal (=) sign. Note that the names of the variables are case-sensitive and should be in uppercase for instant identification.

Key Features of Environment Variables

Dynamic Values: They can change from session to session and even during the execution of programs.
System-Wide or User-Specific: Some variables are set globally and affect all users and processes, while others are specific to individual users.
Inheritance: Environment variables can be inherited by child processes from the parent process, making them useful for configuring complex applications.

Common Environment Variables

Here are some commonly used environment variables in Linux:

HOME: Indicates the current user’s home directory.
PATH: Specifies the directories where the system looks for executable files.
USER: Contains the name of the current user.
SHELL: Defines the path to the current user’s shell.
LANG: Sets the system language and locale settings.

Setting and Using Environment Variables

Temporary Environment Variables in Linux

You can set environment variables temporarily in a terminal session using the export command: This command sets an environment variable named MY_VAR to true for the current session. Environment variables are used to store information about the environment in which programs run.

export MY_VAR=true
echo $MY_VAR

Example 1: Setting Single Environment Variable

For example, the following command will set the Java home environment directory.

export JAVA_HOME=/usr/bin/java

Note that you won’t get any response about the success or failure of the command. As a result, if you want to verify that the variable has been properly set, use the echo command.

echo $JAVA_HOME

The echo command will display the value if the variable has been appropriately set. If the variable has no set value, you might not see anything on the screen.

Example 2: Setting Multiple Environment Variables

You can specify multiple values for a multiple variable by separating them with space like this:

export VAR1="value1" VAR2="value2" VAR3="value3"

Example 3: Setting Multiple value for single Environment Variable

You can specify multiple values for a single variable by separating them with colons like this: <NAME>=<VALUE1>:<VALUE2>:<VALUE3>

export PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"

The PATH variable contains a list of directories where the system looks for executable files. Multiple directories are separated by colons.

Permanent Environment Variables in Linux

To make MY_VAR available system-wide, follow these steps:

This command appends the line MY_VAR=”True” to the /etc/environment file, which is a system-wide configuration file for environment variables.

By adding this line, you make the MY_VAR variable available to all users and sessions on the system.

The use of sudo ensures that the command has the necessary permissions to modify /etc/environment

Example 1: Setting Single Environment Variable for all USERS

export MY_VAR=true
echo 'MY_VAR="true"' | sudo tee /etc/environment -a

Breakdown of the Command

echo ‘MY_VAR=”true”‘: This command outputs the string MY_VAR=”true”. Essentially, echo is used to display a line of text.

| (Pipe): The pipe symbol | takes the output from the echo command and passes it as input to the next command. In this case, it passes the string MY_VAR=”true” to sudo tee.

sudo tee /etc/environment -a: sudo: This command is used to run commands with superuser (root) privileges. Since modifying /etc/environment requires administrative rights, sudo is necessary.

tee: The tee command reads from the standard input (which is the output of the echo command in this case) and writes it to both the standard output (displaying it on the terminal) and a file.

/etc/environment: This is the file where tee will write the output. The /etc/environment file is a system-wide configuration file for environment variables.

-a: The -a (append) option tells tee to append the input to the file rather than overwriting its contents. This ensures that any existing settings in /etc/environment are preserved and the new line is simply added to the end of the file.

This command is used to add a new environment variable (MY_VAR) to the system-wide environment variables file (/etc/environment). By appending it, you ensure that the new variable is available to all users and sessions across the entire system.

Example 2: Setting Multiple value for single Environment Variable for all USERS

You can specify multiple values for a single variable by separating them with colons like this: <NAME>=<VALUE1>:<VALUE2>:<VALUE3>

export MY_PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin"

echo MY_PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin" | sudo tee /etc/environment -a

Devops

Cross-Subscription Code Integration: How to Access External Azure DevOps Repos Like a Pro

January 15, 2025 Thiyagu Leave a comment

Introduction

Efficiently integrating code from external Azure DevOps repositories is crucial for collaborative projects and streamlined development workflows. This comprehensive guide provides a step-by-step approach to accessing and utilizing external repositories within your Azure DevOps pipelines (Checkout External Repositories). We’ll cover essential steps, including creating Personal Access Tokens (PATs), configuring service connections, and referencing external repositories in your YAML pipelines. By following these instructions, you’ll enhance your development process by seamlessly incorporating code from various sources across different subscriptions.

Accessing an External Azure DevOps Repository Across Subscriptions

Accessing a repository from another Azure DevOps subscription can be essential for projects where resources are distributed across different organizations or accounts. This article provides a step-by-step guide on using a Personal Access Token (PAT) and a service connection to access an external repository within an Azure DevOps pipeline. By following these instructions, you’ll be able to integrate code from another subscription seamlessly.

Where it required?

In scenarios where you need to access resources (like repositories) that belong to a different Azure DevOps organization or subscription, you need to configure cross-subscription access. This setup is commonly required in the following situations:

Shared Repositories Across Teams: Teams working on interconnected projects in different organizations or subscriptions often need to share code. For example, a core library or shared services might be maintained in one subscription and used across multiple other projects.
Centralized Code Management: Large enterprises often centralize codebases for specific functionalities (e.g., CRM services, microservices). If your pipeline depends on these centralized repositories, you must configure access.
Multi-Subscription Projects: When an organization spans multiple Azure subscriptions, projects from one subscription might need to integrate code or services from another, necessitating secure cross-subscription access.
Dependency Management: A project may depend on another repository’s codebase (e.g., APIs, SDKs, or CI/CD templates) that resides in a different Azure DevOps subscription.
Separate Environments: Development and production environments might exist in separate subscriptions for security and compliance. For example, accessing a production-ready repository for release from a different subscription’s development repository.

Step-by-Step Guide

Step 1: Create a Personal Access Token (PAT) in External ADO

Navigate to the Azure DevOps organization containing the external repository.
Click on your profile picture in the top-right corner and select Personal Access Tokens.
Click on New Token and:

Provide a name (e.g., External Repo Access).
Set the Scope to Code (Read) (or higher if required).
Specify the expiration date.
Generate the PAT and copy it. Store it securely as you won’t be able to view it again.

Step 2: Create a Service Connection in your ADO

A service connection allows your pipeline to authenticate with the external repository.

Go to the Azure DevOps project where you’re creating the pipeline.
Navigate to Project Settings > Service Connections.
Click on New Service Connection and select Azure Repos/Team Foundation Server.

In the setup form:

Repository URL: Enter the URL of the external repository.
Authentication Method: Select Personal Access Token.
PAT: Paste the PAT you generated earlier.

Give the service connection a name (e.g., CRM Service Connection) and save it.

Step 3: Reference the External Repository in Your Pipeline

The repository keyword lets you specify an external repository. Use a repository resource to reference an additional repository in your pipeline. Add the external repository to your pipeline configuration.

SYNTAX

repositories:
- repository: string #Required as first property. Alias for the repository.
  endpoint: string #ID of the service endpoint connecting to this repository.
  trigger: none | trigger | [ string ] # CI trigger for this repository(only works for Azure Repos).
  name: string #repository name (format depends on 'type'; does not accept variables).
  ref: string #ref name to checkout; defaults to 'refs/heads/main'. The branch checked out by default whenever the resource trigger fires.
  type: string #Type of repository: git, github, githubenterprise, and bitbucket.

Update your pipeline YAML file to include:

resources:
  repositories:
  - repository: externalRepo
    type: git
    name: myexternal_project/myexternal_repo
    ref: external-ProductionBranch #Branch reference
    endpoint: dotnet Service Connection #Service connection name

References the external repository under resources.repositories.
name: mention your external project and Repo name
ref: Specifies the branch (external-ProductionBranch)
endpoint: service connection (dotnet Service Connection).

Step 4: Checkout the External Repository

Include a checkout step in your pipeline: This ensures the external repository is cloned into the pipeline workspace for subsequent tasks.

steps:
- checkout: externalRepo

Step 5: Define the Build Pipeline

Add steps for building and packaging the code. In my case, the external project is dotnet core so i have added the build steps for the same as shown in below.

- script: |
    dotnet --version
    nuget restore ProjectSrc/dotnethelpers.FunctionApp.csproj
  displayName: 'Restore NuGet Packages'
- task: DotNetCoreCLI@2
  inputs:
    command: 'build'
    projects: '**/dotnethelpers.FunctionApp.csproj'
    arguments: '--output $(Build.BinariesDirectory)/publish_output'
- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)/publish_output'
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
    replaceExistingArchive: true
- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

Full YAML

resources:
  repositories:
  - repository: externalRepo
    type: git
    trigger: 
    - external-ProductionBranch
    name: myexternal_project/myexternal_repo
    ref: external-ProductionBranch # Branch reference
    endpoint:dotnet Service Connection # Service connection name
pool:
  vmImage: windows-latest
steps:
- checkout: externalRepo
- task: UseDotNet@2
  displayName: 'Install .NET SDK'
  inputs:
    packageType: 'sdk'
    version: '8.0.x'
    installationPath: $(Agent.ToolsDirectory)/dotnet
- script: |
    dotnet --version
    nuget restore ProjectSrc/dotnethelpers.FunctionApp.csproj
  displayName: 'Restore NuGet Packages'

- task: DotNetCoreCLI@2
  inputs:
    command: 'build'
    projects: '**/dotnethelpers.FunctionApp.csproj'
    arguments: '--output $(Build.BinariesDirectory)/publish_output'
- task: ArchiveFiles@2
  inputs:
    rootFolderOrFile: '$(Build.BinariesDirectory)/publish_output'
    includeRootFolder: false
    archiveType: 'zip'
    archiveFile: '$(Build.ArtifactStagingDirectory)/$(Build.BuildId).zip'
    replaceExistingArchive: true
  
- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

Conclusion

Successfully accessing and integrating external Azure DevOps repositories requires careful authentication and configuration. By following the steps outlined in this guide, including creating PATs, establishing service connections, and effectively referencing external repositories within your YAML pipelines, you can seamlessly integrate code from various sources. This streamlined approach fosters enhanced collaboration, improved efficiency, and a more robust development process for your projects.

Devops

How To Copy Secrets From KeyVault To Another KeyVault In Azure

December 25, 2024 Thiyagu 1 Comment

Introduction

Azure Key Vault is a secure cloud service for managing secrets, encryption keys, and certificates. In modern multi-region deployments, ensuring that application secrets are consistently available across regions is essential for high availability and disaster recovery. However, manually copying secrets from one Key Vault to another can be tedious, error-prone, and time-consuming, especially when dealing with numerous secrets.

This blog post demonstrates how to automate the process of copying secrets from one Azure Key Vault to another using a PowerShell script. By following this guide, you can efficiently replicate secrets between regions, ensuring consistency and reducing manual intervention.

Use Case:

In our application setup, we aimed to configure high availability by deploying the application in two Azure regions. The primary Key Vault in region 1 contained numerous secrets, which we needed to replicate to the Key Vault in region 2. Manually moving each secret one by one was impractical and error-prone.

To overcome this, we developed an automated process using PowerShell to copy all secrets from the source Key Vault to the destination Key Vault. This approach eliminates human errors, saves time, and ensures seamless secret replication for high availability.

e. This blog will help you to understand How To Copy Secrets From KeyVault To Another In Azure using PowerShell script.

To clone a secret between key vaults, we need to perform two steps:

Retrieve/export the secret value from the source key vault.
Import this value into the destination key vault.

You can also refer below link to learn how to maintain your secrets in key vault and access in YAML pipeline

Using secrets from Azure Key Vault in a pipeline

Step 1: Install Azure AZ module

Use the below cmdlet to Install the Azure PowerShell module if not already installed

# Install the Azure PowerShell module if not already installed
  Install-Module -Name Az -Force -AllowClobber

Step 2: Set Source and destination Key Vault name

# Pass both Source and destination Key Vault Name
Param( [Parameter(Mandatory)] 
[string]$sourceKvName, 
[Parameter(Mandatory)] 
[string]$destinationKvName )

Step 3: Connect the Azure portal to access the Key Vault (non-interactive mode)

As we are doing the automation, so you can’t use Connect-AzAccount (which will make the popup to authenticate), if want to execute without any manual intervention then use az login with non-interactive mode as shown in below.

# Connect to Azure portal (you can also use Connect-AzAccount)
az login --service-principal -u "0ff3664821-0c94-48e0-96b5-7cd6422f46" -p "XACccAV2jXQrNks6Lr3Dac2B8z95BAt~MTCrP" --tenant "116372c23-ba4a-223b-0339-ff8ba7883c2"

Step 4: Get the all the secrets name from the source KV

# Get all the Source Secret keys
$secretNames = (Get-AzKeyVaultSecret -VaultName $sourceKvName).Name

Step 5: Copy Secrets From source to destination KV.

The below script will loop based on the number of key names to fetch both name of the key and its value from the source key Vault and started to set the key and value in the destination KvName.

# Loop the Secret Names and copy the key/value pair to the destination key vault
$secretNames.foreach{
Set-AzKeyVaultSecret -VaultName $destinationKvName -Name $_ `
-SecretValue (Get-AzKeyVaultSecret -VaultName $sourceKvName -Name $_).SecretValue
}

Full code

# Pass both Source and destination Key Vault Name
Param(
[Parameter(Mandatory)]
[string]$sourceKvName,
[Parameter(Mandatory)]
[string]$destinationKvName
)
# Connect to Azure portal (you can also use Connect-AzAccount)
az login --service-principal -u "422f464821-0c94-48e0-96b5-7cd60ff366" -p "XACccAV2jXQrNks6Lr3Dac2B8z95BAt~MTCrP" --tenant "116372c23-ba4a-223b-0339-ff8ba7883c2"
# Get all the Source Secret keys
$secretNames = (Get-AzKeyVaultSecret -VaultName $sourceKvName).Name
# Loop the Secret Names and copy the key/value pair to the destination key vault
$secretNames.foreach{
Set-AzKeyVaultSecret -VaultName $destinationKvName -Name $_ `
-SecretValue (Get-AzKeyVaultSecret -VaultName $sourceKvName -Name $_).SecretValue
}

Conclusion

Managing secrets across multiple Azure regions can be challenging but is crucial for ensuring high availability and disaster recovery. Automating the process of copying secrets between Key Vaults not only streamlines the operation but also enhances reliability and reduces the risk of errors.

By following the steps outlined in this blog, you can easily replicate secrets between Azure Key Vaults using PowerShell. This solution ensures that your applications in different regions are configured with consistent and secure credentials, paving the way for robust and scalable deployments.

Implement this process to save time, minimize errors, and focus on scaling your applications while Azure handles secure secret management for you.

Devops

How to Delete a Blob from an Azure Storage using PowerShell

November 26, 2024 Thiyagu Leave a comment

In one of my automation (Delete a Blob), I need to delete the previously stored reports (reports will always append with timestamp) on daily basis in Azure storage account in automated way in the specific container. So i need to ensure my container is available before start deleting my report. This article will have detail explain about How to Delete a Blob from an Azure Storage Account using PowerShell.

New to storage account?

One of the core services within Microsoft Azure is the Storage Account service. There are many service that utilize Storage Accounts for storing data, such as Virtual Machine Disks, Diagnostics logs (specially application log), SQL backups and others. You can also use the Azure Storage Account service to store your own data; such as blobs or binary data.

As per MSDN, Azure blob storage allows you to store large amounts of unstructured object data. You can use blob storage to gather or expose media, content, or application data to users. Because all blob data is stored within containers, you must create a storage container before you can begin to upload data.

Delete a Blob from an Azure Storage

Step: 1 Get the prerequisite inputs

As in this example i am going to delete the one the sql db (backup/imported to the storage) stored as bacpac format in the container called SQL…

## prerequisite Parameters
$resourceGroupName="rg-dgtl-strg-01"
$storageAccountName="sadgtlautomation01"
$storageContainerName="sql"
$blobName = "core_2022110824.bacpac"

Step: 2 Connect to your Azure subscription

Using the az login command with a service principal is a secure and efficient way to authenticate and connect to your Azure subscription for automation tasks and scripts. In scenarios where you need to automate Azure management tasks or run scripts in a non-interactive manner, you can authenticate using a service principal. A service principal is an identity created for your application or script to access Azure resources securely.

## Connect to your Azure subscription
az login --service-principal -u "210f8f7c-049c-e480-96b5-642d6362f464" -p "c82BQ~MTCrPr3Daz95Nks6LrWF32jXBAtXACccAV" --tenant "cf8ba223-a403-342b-ba39-c21f78831637"

Step: 3 Get the storage account to Check the container exit or not

When working with Azure Storage, you may need to verify if a container exists in a storage account or create it if it doesn’t. You can use the Get-AzStorageContainer cmdlet to check for the existence of a container.

## Get the storage account to check container exist or need to be create
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName
## Get the storage account context
$context = $storageAccount.Context

Step: 4 Check the container exist before deleting the blob

We need to use Remove-AzStorageBlob cmdlet to delete a blob from the Azure storage container

## Check if the storage container exists
if(Get-AzStorageContainer -Name $storageContainerName -Context $context -ErrorAction SilentlyContinue)
{
Write-Host -ForegroundColor Green $storageContainerName ", the requested container exit,started deleting blob"
## Create a new Azure Storage container
Remove-AzStorageBlob -Container $storageContainerName -Context $context -Blob $blobName
Write-Host -ForegroundColor Green $blobName deleted
}
else
{
Write-Host -ForegroundColor Magenta $storageContainerName "the requested container does not exist"
}

Full Code:

## Delete a Blob from an Azure Storage
## Input Parameters
$resourceGroupName="rg-dgtl-strg-01"
$storageAccountName="sadgtlautomation01"
$storageContainerName="sql"
$blobName = "core_2022110824.bacpac"
## Connect to your Azure subscription
az login --service-principal -u "210f8f7c-049c-e480-96b5-642d6362f464" -p "c82BQ~MTCrPr3Daz95Nks6LrWF32jXBAtXACccAV" --tenant "cf8ba223-a403-342b-ba39-c21f78831637"
## Function to create the storage container
Function DeleteblogfromStorageContainer
{
## Get the storage account to check container exist or need to be create
$storageAccount = Get-AzStorageAccount -ResourceGroupName $resourceGroupName -Name $storageAccountName
## Get the storage account context
$context = $storageAccount.Context

## Check if the storage container exists
if(Get-AzStorageContainer -Name $storageContainerName -Context $context -ErrorAction SilentlyContinue)
{
Write-Host -ForegroundColor Green $storageContainerName ", the requested container exit,started deleting blob"
## Remove the blob in Azure Storage container
Remove-AzStorageBlob -Container $storageContainerName -Context $context -Blob $blobName
Write-Host -ForegroundColor Green $blobName deleted
}
else
{
Write-Host -ForegroundColor Magenta $storageContainerName "the requested container does not exist"
}
}
#Call the Function
DeleteblogfromStorageContainer

Output:

Devops

How to view the secret variables in Azure DevOps

November 7, 2024 Thiyagu Leave a comment

Today, I will be taking about a technique using which you can view the secret variables in Azure DevOps.

Introduction

Azure DevOps supports us to store secrets within Azure DevOps variable Groups which could be used with the Pipelines. These secret variables couldn’t be viewed by us manually from the portal. Sometimes, we may want to view the password to perform some other activities.

Note: The best practice to have the secrets in Azure Key Vault and same you can read and execute in Azure pipeline in very secured way. Still some legacy projects are maintaining the secrets in Azure Variable group, so this article focus on them. You can read this to use Key Vault to handle the secrets

Using secrets from Azure Key Vault in a pipeline

What are Secrets Variables in Azure Pipelines?

Secret Variables are placeholders for values which you want to store in an encrypted format and use while using running a pipeline. Secret Variables can be used for values like username, password, API key etc. Secret variables are encrypted variables that you can use in pipelines without exposing their value. Secret variables can be used for private information like passwords, IDs, and other identifying data that you wouldn’t want to have exposed in a pipeline. Secret variables are encrypted at rest with a 2048-bit RSA key and are available on the agent for tasks and scripts to use.

How to set Secret in Azure Variable group?

Set secret variables in the UI for a pipeline. Secret variables set in the pipeline settings UI for a pipeline are scoped to the pipeline where they are set. So, you can have secrets that only visible to users with access to that pipeline. Set secrets in a variable group. Variable groups follow the library security model. You can control who can define new items in a library, and who can use an existing item.

Let’s create a Secret variable in a Variable Group as shown below and make sure that you set it as a secret by locking it.

Once you mark it a secret (by clicking on the open lock icon as shown in below image), save the Variable Group, no one including admin will be able to view the secret. Let’s now understand how to view the secret with the help of Azure DevOps – Pipelines.

View the secret variables from Variable Group

You can create a simple Pipeline which has the below tasks to view the secrets in pipeline execution.

PowerShell task which outputs a text (along with secret) into a file names ViewSecretValue.Txt
Publish the ViewSecretValue.txt into Azure Pipeline artifacts.

Run the pipeline with with below PowerShell task

variables:
- group: Demo_VariableGroup
steps:
- powershell: |
    "The secretkey value is : $(secretkey)" | Out-File -FilePath  $(Build.ArtifactStagingDirectory)\ViewSecretValue.txt
- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

Now, click on the ViewSecretValue.txt file to download the file. Once you download it, view that in a Notepad which should below.

Conclusion

In summary, handling secret variables securely is crucial for maintaining data confidentiality in DevOps processes. Azure DevOps provides built-in features and best practices to keep sensitive data protected, making it a powerful platform for secure CI/CD pipeline management. Integrating with tools like Azure Key Vault can further strengthen your security posture and simplify secret management across multiple projects.

Devops

How to pass objects between tasks in Azure pipeline

October 24, 2024 Thiyagu Leave a comment

In our previous post, we already discussed about “How to pass values between Tasks in a Pipeline” where we can pass single value from one task/job/stage to another in pipeline. But incase, if you want to share the object from one task/job/stage to another instead of single value then we need to perform some trick to achieve this. In this quick post we will discuss about trick that I have found recently to store an object in an Azure Pipelines variable from a PowerShell script (How to pass objects between tasks in Azure pipeline).

The problem

Setting a variable in a script for later use in an Azure DevOps’ pipeline is possible using the task.setvariable command as described in below post.

How to pass values between Tasks in a Pipeline using task.setvariable Command

This works great for simple variables like below:

steps:
- pwsh: |
Write-Host "##vso[task.setvariable variable=variableName]variableValue"
- pwsh: |
Write-Host "Value from previous step: $(variableName)"

But it is bit trickier for sharing the complex variables likes objects, arrays, or arrays of objects between taks/stage/jobs in pipeline.

The solution

As an example, let’s we try to retrieve the name, type and resource group of all the resources in an Azure subscription as shown in the below script. Let we see how can pass this value of $resources in pipeline.

$azure_resources = Get-AzResource | Select-Object -Property Name,Type,ResourceGroupName

First you can store an object in an Azure Pipelines variable using the PowerShell task. Next, you can simply serialize it in JSON and apply in single input like below: -Compress flag which conver JSON to a single line.

$azure_resourcesJson = $azure_resources | ConvertTo-Json -Compress

Pass objects between tasks in Azure pipeline

pool:
name: devopsagent-win-pprd
steps:
- task: AzurePowerShell@5
inputs:
azureSubscription: 'Azure_Digital'
azurePowerShellVersion: LatestVersion
ScriptType: InlineScript
Inline: |
$azure_resources = Get-AzResource | Select-Object -Property Name,Type,ResourceGroupName -First 3
$azure_resourcesJson = $azure_resources | ConvertTo-Json -Compress
Write-Host "##vso[task.setvariable variable=resources]$azure_resourcesJson"
- pwsh: |
$resources = '$(resources)' | ConvertFrom-Json
Write-Host "There are $($resources.Count) resources in the list"
Write-Host "There are resources are" $resources.ResourceGroupName

OUTPUT

Devops, linux

Linux Environment Variables

September 25, 2024 Thiyagu Leave a comment

What Are Linux Environment Variables?

Linux environment variables are dynamic values that the operating system and various applications use to determine information about the user environment. They are essentially variables that can influence the behavior and configuration of processes and programs on a Linux system. These variables are used to pass configuration information to programs and scripts, allowing for flexible and dynamic system management.

Key Features of Environment Variables

Dynamic Values: They can change from session to session and even during the execution of programs.
System-Wide or User-Specific: Some variables are set globally and affect all users and processes, while others are specific to individual users.
Inheritance: Environment variables can be inherited by child processes from the parent process, making them useful for configuring complex applications.

Common Environment Variables

Here are some commonly used environment variables in Linux:

HOME: Indicates the current user’s home directory.
PATH: Specifies the directories where the system looks for executable files.
USER: Contains the name of the current user.
SHELL: Defines the path to the current user’s shell.
LANG: Sets the system language and locale settings.

Setting and Using Environment Variables

Temporary Environment Variables

export MY_VAR=true
echo $MY_VAR

Example 1: Setting Single Environment Variable

For example, the following command will set the Java home environment directory.

export JAVA_HOME=/usr/bin/java

Note that you won’t get any response about the success or failure of the command. As a result, if you want to verify that the variable has been properly set, use the echo command.

echo $JAVA_HOME

The echo command will display the value if the variable has been appropriately set. If the variable has no set value, you might not see anything on the screen.

Example 2: Setting Multiple Environment Variables

You can specify multiple values for a multiple variable by separating them with space like this:

export VAR1="value1" VAR2="value2" VAR3="value3"

Example 3: Setting Multiple value for single Environment Variable

You can specify multiple values for a single variable by separating them with colons like this: <NAME>=<VALUE1>:<VALUE2>:<VALUE3>

export PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"

The PATH variable contains a list of directories where the system looks for executable files. Multiple directories are separated by colons.

Permanent Environment Variables

To make DOTNET_HOME available system-wide, follow these steps:

This command appends the line MY_VAR=”True” to the /etc/environment file, which is a system-wide configuration file for environment variables. By adding this line, you make the MY_VAR variable available to all users and sessions on the system. The use of sudo ensures that the command has the necessary permissions to modify /etc/environment

Example 1: Setting Single Environment Variable for all USERS

export DOTNET_HOME=true
echo 'DOTNET_HOME="true"' | sudo tee /etc/environment -a

Example 2: Setting Multiple value for single Environment Variable for all USERS

You can specify multiple values for a single variable by separating them with colons like this: <NAME>=<VALUE1>:<VALUE2>:<VALUE3>

export PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin"

echo PATH="/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin" | sudo tee /etc/environment -a

Breakdown of the Command

echo ‘DOTNET_HOME=”true”‘: This command outputs the string DOTNET_HOME=”/dotnet-helpers/execute”. Essentially, echo is used to display a line of text.

| (Pipe): The pipe symbol | takes the output from the echo command and passes it as input to the next command. In this case, it passes the string DOTNET_HOME=”/dotnet-helpers/execute” to sudo tee.

sudo tee /etc/environment -a: sudo: This command is used to run commands with superuser (root) privileges. Since modifying /etc/environment requires administrative rights, sudo is necessary.

tee: The tee command reads from the standard input (which is the output of the echo command in this case) and writes it to both the standard output (displaying it on the terminal) and a file.

/etc/environment: This is the file where tee will write the output. The /etc/environment file is a system-wide configuration file for environment variables.

This command is used to add a new environment variable (DOTNET_HOME) to the system-wide environment variables file (/etc/environment). By appending it, you ensure that the new variable is available to all users and sessions across the entire system.

Devops

Delete File or Directory in Linux with Scheduled Azure DevOps Pipeline

August 22, 2024 Thiyagu Leave a comment

In my working environment, we are managing more Linux based Agent machines for building the solution to create artifacts and we got required to clean the build artifacts on regular manner in automatic way so we though to write a bash scrip and make this as scheduled in release pipeline (Delete File or Directory in Linux). Base on my automation, though to write a post to explain How to Delete File or Directory in Linux with Scheduled Azure DevOps Pipeline

You can also read

How to Delete a Folder or File using PowerShell

step 1: Find the disk space usage

The df -h command is used to display information about disk space usage on a Unix-like system. When you run this command in a terminal, it will show the disk space usage in a human-readable format.

For more clear, disk free also known as `df`, which is a powerful utility that provides valuable information on disk space utilization. The df command displays information about file system disk space usage on the mounted file system. The -h flag makes the sizes human-readable, using units like KB, MB, GB, etc.

df -h

STEP 2:Get list of directories/files and assign to variable

Before we can remove a folder or director, we must first know its name. Therefore, we must first execute the “ls” command in the terminal to find a folder or directory, or to view all of the folders. In Linux and other operating systems based on Unix, the “ls” command is used to display the files or folders.

As i am going to clean my agent folder so path will be /agent/_work.
We are assigning the output of the command ls /agent/_work/ | grep [0-9] to the variable directorylist. This command lists the contents of the /agent/_work/ directory and filters the results to include only lines that contain numbers (as my agent machine folder will create with name as numbers).

directorylist=$(ls /agent/_work/ | grep [0-9])

STEP 3: Loop the list of directories and delete

Next, we need to loop the directory list one by one in the while loop as shown in below script. while read line is a loop that reads each line of the processed output ( is used for bash shell to read a file using while loop).The option ‘-r’ in the above-mentioned syntax passed to read command that avoids the backslash escapes from being interpreted

tr ‘ ‘ ‘\n’: one of the use of tr command is to find and replace, here it will replace spaces with newline characters.
The loop body (between do and done) is where you can put your processing logic for each line. I’ve included a simple echo statement as an example.

echo $directorylist | tr ' ' '\n' | while read -r line
do
........... You logic to delete ...........
Done

STEP 4: Remove the directory/file from the list

We can Delete File or Directory in Linux by using rm command and -rf will be used to remove fore fully as shown below.

echo "removing folder $line"
rm -rf /agent/_work/$line

Full code: Delete File or Directory in Linux

# Find the disk space usage
df -h
echo "Running a disk space clean up"
#Get list of directories/files and assign to variable
directorylist=$(ls /agent/_work/ | grep [0-9])
#Loop the list of directries and delete
echo $directorylist | tr ' ' '\n' | while read line
do
echo "removing folder $line"
rm -rf /agent/_work/$line
done

How to implement delete directory (above script) script in scheduled manner Azure DevOps pipeline?

First enable the “Scheduled release trigger” as shown below in release pipeline. In same pipeline, create a new stage with the Bash task with the above script which shown to Delete File or Directory in Linux .
Select the stage and click “pre-deployment condition” and schedule the pipeline condition when it need to execute and save. Post this action, the pipeline will ran on specific time and execute the cleanup task.

Devops

How to Drop SQL database using PowerShell

August 8, 2024 Thiyagu Leave a comment

My Scenario:

As a System Admin/DevOps Guys, we got urgent cost optimization process and it need to done in very less time frame. From the cost optimization list, one of the action is to clean the unused database and backup file across all the environment. So we having 100+ database in each environment and its difficult to make manual clean up as it will take more time and there is lot of chance to wrong deletion of used database’s. So to avoid this we thought to automate this process to clean up database’s across all the environment. In this post we will discuss about how to How to drop SQL database using PowerShell

If you are working with Azure SQL Databases and want to use Azure PowerShell (Az module), you can use the Get-AzSqlDatabase cmdlet to retrieve information about SQL databases in an Azure SQL Server. Here’s an example script to get the list of all SQL database names:

Step 1: Declare SQL and resource details

#Assign the variables
$resourcegroup = "rg-dgtl-strg-prd-we-01"
$dbserverName = "sqlsrvr-dgtl-prd-we"
$username = "sqlprd01"
$password = "Srdc4$wm2t1F"

Step 2: Connect to the database using Get-AzSqlDatabase cmdlet

The Get-AzSqlDatabase cmdlet is used in Azure PowerShell to retrieve information about SQL databases (as shown in the below snap shot) in an Azure SQL Server. It’s part of the Az module, which is the recommended module for managing Azure resources. Below is a brief explanation of how to use the Get-AzSqlDatabase cmdlet:

SYNTAX:

Get-AzSqlDatabase
[[-DatabaseName] <String>]
[-ExpandKeyList]
[-KeysFilter <String>]
[-ServerName] <String>
[-ResourceGroupName] <String>
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]

#Get the all the database for specific SQL server using -ServerName parameter
$SQLdbs = Get-AzSqlDatabase -ServerName $dbserverName -ResourceGroupName $resourcegroup

Step 3: Retrieve all database’s details Using foreach.

In step 1, we got all the database details which present inside sqlsrvr-dgtl-prd-we SQL server and as i said on top, I am having 100+ database present in the sql server so below loop will loop one by one to process the db’s details. Below, i am getting only database name for db using property name like “DatabaseName”

#Loop the list of databases and check on one by one
foreach ($SQLdb in $SQLdbs){
$SQLdb = $SQLdb.DatabaseName.ToString()
if (..) { check databasename contains or -eq to find activie dbs. }
else { logic to delete the non-active dbs. }
}

Step 4: Remove the database using Remove-AzSqlDatabase

The Remove-AzSqlDatabase cmdlet removes an Azure SQL database. This cmdlet is also supported by the SQL Server Stretch Database service on Azure.

SYNTAX :

Remove-AzSqlDatabase
[-DatabaseName] <String>
[-Force]
[-ServerName] <String>
[-ResourceGroupName] <String>
[-DefaultProfile <IAzureContextContainer>]
[-WhatIf]
[-Confirm]
[<CommonParameters>]

#Remove the database based on -DatabaseName parameter
Remove-AzSqlDatabase -ResourceGroupName $resourcegroup -ServerName $dbserverName -DatabaseName $dbName

Points to remember:

I am running above script inside my jump/AVD machine so if required please use -DefaultProfile parameter in the Get-AzSqlDatabase / Remove-AzSqlDatabase to authenticate the SQL server. This -DefaultProfile parameter used with credentials, tenant and subscription used for communication with azure.

Devops

Pull and Push Docker Image to Azure Container Registry using Azure Devops pipeline

August 1, 2024 Thiyagu Leave a comment

When you want to develop and implement the container application in Azure. The first and main step you would execute is to build the images and push them into the our own private Registry (ex: Azure Container registry). In this post, I will explain how to Pull and Push Docker Image to Azure Container Registry using Azure DevOps pipeline

If your solution is going to use base image from public repo then best practice in DevOps to pull & push the trusted public image to ACR, post that we need to use same in our custom solution build.

what is Azure container Registry (ACR)

Azure Container Registry also is the similar as hub.docker.com but is provided by azure cloud. The Azure Container registry can be private and can be used by only one team or users who have access. So, users with access can push and pull images.

It provides geo-replication so that images pushed in one datacenter in one region gets replicated in all the connected configured datacenters and gets deployed simultaneously to Kubernetes clusters in respective locations.

Pull and push Docker Image

The purpose of this article is to provide steps to guide how to pull the image from public repository and provide commands to push and pull images from registry using the Azure DevOps pipeline.

There can be two options when you want to push the container images into ACR.

Option 1: Import the pre-existing Docker image from the docker hub (docker.io)/public registry and deploy it to AKS.

Option 2: Create a new custom image based on our solution (we can use push and pull other public registry and use in our solution as base image to build our solution), push it to ACR, and then deploy it to AKS.

Note: If you are using Azure default Agent or your own Agent, then decide which type of image your pulling and pushing. If the image is build on windows then the window Agent need to use for the push and pull or linux Agent if image is build with linux as base image. In my case, i am pulling the linux based image from registry.k8s.io to my ACR. Post this action, we will refer the same image during the nginx ingress installation in my AKS

Push Image to Azure Container Registry

Step 1 : Login to Azure Container Registry with Non-Interactive mode

Syntax: docker login –username demo –password example

- bash: |
docker login crdgtlshared02.azurecr.io -u crdgtlshared010 -p rtvGwd6X2YJeeKhernclok=UHRceH7rls

Step 2 : Pull the image and tag the image with registry prefix

In my case, I need to pull the image from public repository (registry.k8s.io) and from my ACR i need to refer this image during the ingress installation in AKS cluster. To be able to push Docker images to Azure Container Registry, they need to be tagged with the login Server name of the Registry. These tags are used for routing purposes when we push these Docker images to Azure. In simple words, Docker tags convey useful information about a specific image version/variant

Syntax: docker pull [OPTIONS] NAME[:TAG|@DIGEST]
docker tag SOURCE_IMAGE[:TAG] TARGET_IMAGE[:TAG]

- bash: |
docker pull registry.k8s.io/ingress-nginx/controller:v1.3.0
docker tag registry.k8s.io/ingress-nginx/controller:v1.3.0 crdgtlshared02.azurecr.io/nginx-baseimage/controller:v1.3.0
displayName: 'push ingnix base image'
enabled: false

Pull Image to Azure Container Registry

Step 3 : Pull the image with registry name prefix

Now that the image is tagged (in step 2), we can use the “docker push” command to push this image to Azure Container Registry;

Syntax: docker push [OPTIONS] NAME[:TAG|@DIGEST]

- bash: |
docker push crdgtlshared02.azurecr.io/nginx-baseimage/controller:v1.3.0
displayName: 'push ingnix base image'
enabled: false

This operation might take a few minutes and you will se the image being uploaded to Azure Container Registry in the console.

Note: To pull image directly onto docker-compose, kubernetes yml files, use appropriate logins. Usually in these scenarios, docker login is the first step before docker-compose up is called, so that images get pulled successfully

For this above example, to explain in step by step i used bash task for each action but we can do all to gether in single bask task in pipeline as shown below.

Full YAML code for Pipeline

- bash: |
docker login crdgtlshared02.azurecr.io -u crdgtlshared02 -p gbHdlo6X2YJeeKhaxjnlok=UHRceT9NR
docker pull registry.k8s.io/ingress-nginx/controller:v1.3.0 docker tag registry.k8s.io/ingress-nginx/controller:v1.3.0 crdgtlshared02.azurecr.io/nginx-baseimage/controller:v1.3.0
docker push crdgtlshared02.azurecr.io/nginx-baseimage/controller:v1.3.0 
displayName: 'push ingnix base image' 
enabled: false