Faster formula for identifying duplicates

kwp004

Board Regular
Joined
Dec 27, 2016
Messages
93
I have a large excel file that I work with (>250K records). I have a need to remove duplicates. Normally, I use the formula below to identify the first unique value (then I drop the others), but it takes forever!

=(COUNTIF($C$2:$C2,$C2)=1)+0

The way the data is organized any duplicates are sequential (i.e. always next to each other). I'm extremely confident there will never be more than 25 unique duplicates in the dataset. Is there another formula I can use that would work faster? I think excel is searching all 250K+ records each time for a duplicate, when it only needs to search 25 - 50.

Thanks.
 

Attachments

  • 1601342367767.png
    1601342367767.png
    3.7 KB · Views: 10

Excel Facts

Is there a shortcut key for strikethrough?
Ctrl+S is used for Save. Ctrl+5 is used for Strikethrough. Why Ctrl+5? When you use hashmarks to count |||| is 4, strike through to mean 5.
Would be helpful if you could post a sample of your data using XL2BB.
Have you considered using Data>Data Tools>Remove Duplicates ?? As an alternative, Data>Sort & Filter>Advanced Filter which can produce unique values.

I've never used either on more than a few thousand rows of data so don't know if either would be faster.
 
Upvote 0
If the values are grouped and beginning in C2 this should be faster
=--(C2<>C1)
copy down

M.
 
Upvote 0
Thank you both. Marcelo -- your approach worked like a charm! 20 minutes to 2 seconds :)
 
Upvote 0

Forum statistics

Threads
1,215,063
Messages
6,122,935
Members
449,094
Latest member
teemeren

We've detected that you are using an adblocker.

We have a great community of people providing Excel help here, but the hosting costs are enormous. You can help keep this site running by allowing ads on MrExcel.com.
Allow Ads at MrExcel

Which adblocker are you using?

Disable AdBlock

Follow these easy steps to disable AdBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the icon in the browser’s toolbar.
2)Click on the "Pause on this site" option.
Go back

Disable AdBlock Plus

Follow these easy steps to disable AdBlock Plus

1)Click on the icon in the browser’s toolbar.
2)Click on the toggle to disable it for "mrexcel.com".
Go back

Disable uBlock Origin

Follow these easy steps to disable uBlock Origin

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back

Disable uBlock

Follow these easy steps to disable uBlock

1)Click on the icon in the browser’s toolbar.
2)Click on the "Power" button.
3)Click on the "Refresh" button.
Go back
Back
Top