Hi,
Background: Previous consultant did not clean the data and did not create duplicate detection rules before import.
Situation: The current client has over 600k contacts (most of them duplicates). Sadly, every single contact record has important activities/notes/related custom entities attached to them (yes, weird, the way they exported data from the old system was flawed), so deletion is not an option and we will need to be merge.
Possible solutions (thinking out loud):
- Scrap the system: The system is live and being used by 700+ users on a daily basis. I don't really think it's possible that can we export, clean and re-import the data into a live system over a weekend.
- Brute force: Merging 600k contacts manually is not a realistic option.
- Fixing via the Database side: Yes, this is frowned upon, but we will back up the whole server first.
- Fixing via CRM (coding): Create a custom workflow to merge records.
Solution (3) and (4) can only take care of exact duplicates, and a small subset of records that are "similar" using regex/fuzzy logic/etc (i.e. first and last 10 characters match, etc). So there will still be a lot of manually merging afterwards as the data is super dirty :(
Please share any ideas on how to tackle this problem, any ideas will be much appreciated!!!! Thanks!