Skip to main content

Removing duplicate objects from an set of objects

The question that I took this evening involved a user who has some functional PowerShell code.  The needed a way to filter out any duplicate objects.  They were looking at using several loops.  In my college days, that would have been the answer.  As my favorite professor, Dan Matthews, put it, “Never Reinvent the Wheel”

PowerShell has built in functionality to remove duplicate items by using the Sort-Object cmdlet.  You need to use the –Property parameter to tell PowerShell which proper of the object you are looking for duplicates on and also you need to use the –Unique parameter to tell PowerShell to only leave unique objects (remove duplicates).

Below is some sample code to generate a set of 11 objects.  Two of those objects will have a duplicate value in Prop1.

# Create a dynamic array to hold the test objects.

$Array = @()

 

# Create 10 objects in the dynamic array.

For ($X = 0;$X -lt 10;$X++)

{

    $Obj = New-Object PSObject

    $Obj | Add-Member -MemberType NoteProperty -Name "Prop1" -Value $X

    $Obj | Add-Member -MemberType NoteProperty -Name "Prop2" -Value $True

    $Obj | Add-Member -MemberType NoteProperty -Name "Prop3" -Value $False

 

    # Add the object to the array

    $Array += $Obj

}

 

# Add a duplicate to the array.  The duplication will be on the property

# Prop1 with a value of 3

$Obj = New-Object PSObject

$Obj | Add-Member -MemberType NoteProperty -Name "Prop1" -Value 3

$Obj | Add-Member -MemberType NoteProperty -Name "Prop2" -Value $True

$Obj | Add-Member -MemberType NoteProperty -Name "Prop3" -Value $False

 

# Add the object to the array

$Array += $Obj

 

# View the contents of the array.  Notice the duplicate value in Prop 1

$Array | FT –autosize

 

# Divider between the two sets of data.

Write-host "-----------------------------------------------------------"

 

# Use the -Unique parameter of the Sort-Object cmdlet.

# Note the duplicate object is gone.

$Array | Sort-Object -Property Prop1 -Unique | FT -autosize

 

This output will display two sets of data.  One set will show all 11 objects (including the duplicate).  The second set was filtered using Select-Object in the last line of the code.

Prop1 Prop2 Prop3

----- ----- -----

    0  True False

    1  True False

    2  True False

    3  True False

    4  True False

    5  True False

    6  True False

    7  True False

    8  True False

    9  True False

    3  True False

 

 

-----------------------------------------------------------

 

Prop1 Prop2 Prop3

----- ----- -----

    0  True False

    1  True False

    2  True False

    3  True False

    4  True False

    5  True False

    6  True False

    7  True False

    8  True False

    9  True False

Notice in the second set that was filtered with Sort-Object, the duplicate value of 3 in Prop1 has been removed.

Comments

JoeM said…
Thanks! Was helpful.

Popular posts from this blog

Adding a Comment to a GPO with PowerShell

As I'm writing this article, I'm also writing a customization for a PowerShell course I'm teaching next week in Phoenix.  This customization deals with Group Policy and PowerShell.  For those of you who attend my classes may already know this, but I sit their and try to ask the questions to myself that others may ask as I present the material.  I finished up my customization a few hours ago and then I realized that I did not add in how to put a comment on a GPO.  This is a feature that many Group Policy Administrators may not be aware of. This past summer I attended a presentation at TechEd on Group Policy.  One organization in the crowd had over 5,000 Group Policies.  In an environment like that, the comment section can be priceless.  I always like to write in the comment section why I created the policy so I know its purpose next week after I've completed 50 other tasks and can't remember what I did 5 minutes ago. In the Group Policy module for PowerShell V3, th

Return duplicate values from a collection with PowerShell

If you have a collection of objects and you want to remove any duplicate items, it is fairly simple. # Create a collection with duplicate values $Set1 = 1 , 1 , 2 , 2 , 3 , 4 , 5 , 6 , 7 , 1 , 2   # Remove the duplicate values. $Set1 | Select-Object -Unique 1 2 3 4 5 6 7 What if you want only the duplicate values and nothing else? # Create a collection with duplicate values $Set1 = 1 , 1 , 2 , 2 , 3 , 4 , 5 , 6 , 7 , 1 , 2   #Create a second collection with duplicate values removed. $Set2 = $Set1 | Select-Object -Unique   # Return only the duplicate values. ( Compare-Object -ReferenceObject $Set2 -DifferenceObject $Set1 ) . InputObject | Select-Object – Unique 1 2 This works with objects as well as numbers.  The first command creates a collection with 2 duplicates of both 1 and 2.   The second command creates another collection with the duplicates filtered out.  The Compare-Object cmdlet will first find items that are diffe

How to list all the AD LDS instances on a server

AD LDS allows you to provide directory services to applications that are free of the confines of Active Directory.  To list all the AD LDS instances on a server, follow this procedure: Log into the server in question Open a command prompt. Type dsdbutil and press Enter Type List Instances and press Enter . You will receive a list of the instance name, both the LDAP and SSL port numbers, the location of the database, and its status.