Find Duplicate Files - Windows - Powershell

[PenkethBoy 2068]

This is a simple one line script to highlight duplicate files - By NAME

How to use the script

Very briefly - I am assuming you know how to run a PS Script - if you dont Google it.

Run this script from the root of say your movies directory via the powershell console window (not the Powershell ISE if using PS5) - script is recursive so will find all movies

It may take a few seconds or minutes to look through all directories - depending on the number of files and speed of the pc.

File types searched for mkv, mp4, avi and ts - but if necessary you can add others

It will produce a txt file called ... drum roll ...Duplicates.txt - the file will be located in the same location as the script file and will get overwritten with each run.

If the file is empty after the script completes - then no duplicates found - otherwise it will contain the duplicate movies  (by name)

example output of a duplicate

with a duplicate "count" will be 2 or higher etc - "Group" will attempt to list the paths to each duplicate but if you have many files it will get truncated eventually.

If like me you have numerous extras - some are likely to be named the same - e.g. Behind the Scenes - so you will get false positives.

I have not filtered the search to exclude trailers, extras directories as you could have genuine duplicates there or misplaced files etc but it can easily be done.

Also the search is case insensitive

Duplicates.zip

Unzip the above and find your duplicates!

Have Fun

PenkethBoy 15/10/2020

Edited October 15, 2020 by PenkethBoy

[PenkethBoy 2068]

reserved

[PenkethBoy 2068]

reserved

[PenkethBoy 2068]

reserved

[rbjtech 5282]

Amazing for a 1 line script - and fast too.  Great work @PenkethBoy .. 

I'm now going to butcher it so it automatically scans all my media areas not just the directory it's run from .. 

[PenkethBoy 2068]

2 lines is your challenge?

[Jcheinaman 19]

This may be a silly question, but if I put this in my media directory, will it find movies inside of subfolders? All of my movies are organized into folders such as the example below.

 

Media/H/Halloween (2018) - 1080p/Halloween (2018) - 1080p.mkv

 

Basically, if I put the script in the Media folder, would it scan down into the other folders to find the duplicates?

Thanks!!

[PenkethBoy 2068]

thats what recursive means

[Jcheinaman 19]

On 2/9/2021 at 3:32 PM, PenkethBoy said:

thats what recursive means

Thanks

[HawkXP71 112]

The problem here is you only check by name.  If you have 2 files that are identical, there will not be a match. 

This is an app I wrote that uses MD5 on the files to compare

https://github.com/towel42-com/FindDupe

[Hispanico957 23]

In relation to finding duplicates, I wonder if there is a utility that can find similar files.

Let me explain, I have many movies both normal (1080) and with the same name in 4K, like for example:

The Mist (2007).mkv
The Mist (2007) 4K.mkv
The Others (2001).mkv
The Others (2001) 4K.mkv

How can I identify these films that have the same name but with a 4K ending ?

 

Thank

Hispa

[Luke 42077]

25 minutes ago, Hispanico957 said:

In relation to finding duplicates, I wonder if there is a utility that can find similar files.

Let me explain, I have many movies both normal (1080) and with the same name in 4K, like for example:

The Mist (2007).mkv
The Mist (2007) 4K.mkv
The Others (2001).mkv
The Others (2001) 4K.mkv

How can I identify these films that have the same name but with a 4K ending ?

 

Thank

Hispa

@rbjtechmay have some tips.

[rookies 12]

On 10/15/2020 at 1:09 PM, rbjtech said:

Amazing for a 1 line script - and fast too.  Great work @PenkethBoy .. 

I'm now going to butcher it so it automatically scans all my media areas not just the directory it's run from .. 

example of how to scan all locations with specific paths.

On 10/11/2023 at 12:11 PM, Hispanico957 said:

In relation to finding duplicates, I wonder if there is a utility that can find similar files.

Let me explain, I have many movies both normal (1080) and with the same name in 4K, like for example:

The Mist (2007).mkv
The Mist (2007) 4K.mkv
The Others (2001).mkv
The Others (2001) 4K.mkv

How can I identify these films that have the same name but with a 4K ending ?

 

Thank

Hispa

and find duplicates based on full name or on partial name with a delimiter.

On 8/21/2022 at 1:01 PM, HawkXP71 said:

The problem here is you only check by name.  If you have 2 files that are identical, there will not be a match. 

This is an app I wrote that uses MD5 on the files to compare

https://github.com/towel42-com/FindDupe

It even does checksums.  ( commented out because if your media is on a secondary storage location, say a nas and your run this from your workstation, it essentially transfers all the data from the nas to the workstation to generate the checksum.  In my case many TB of data.  Which is not feasible from a network activity and run time perspective. But it would in fact work if you waited long enough.

The script is not a simple one liner like the original, but gives you a workable framework to create your own versions, and add your own functionality.   

Be aware I have 0, no duplicates in my library because I only ever pull in content as I decrypt the disks.  That is a task I would only ever do once.  So I can't actually test this other than to say,  of course it works!  I wrote it!

It takes a list of library locations:

$path = @('Z:\Media\Movies', 'Z:\Media1\Movies', 'Z:\Media2\Movies', 'Z:\Media1\Movies4K', 'Z:\Media2\Movies4K')

It allows for a partial name delimiter. (anything to the left of this delimiter is check for a duplicate of the same name)  

$partialNameDelimiter = '\)' #use \ to delimit special characters

it also allows for exclusions of specific folders:

$excludeFolders = @("Featurettes", "Trailers", "Deleted Scenes")

 

it should output a text file with line 1 as the key string it was checking,  and the following lines directly below it would be the full path for all duplicates based on that string.

Have fun with it, and modify it for your own specific purposes!

 

Another Duplicates Finder Script.zip