Allow merging when duplicates found on contact import
anonymous
shared this idea
AdminDuncan Stockdill
(Admin, Capsule)
responded
we have looked into merging and concluded it’s not quite as straight forward as it might appear. Various implementations we’ve looked at aren’t without their problems.
I’d like some feedback from supporters of this feature on the following ideas and concerns:
Our initial thought was to match based on email address. However, if there are already multiple contacts in the database, what to do… do we ignore, or do we update each? If the name is different on the record being imported do we overwrite? Would it be better to match on name + email? Many of you are looking to import 100+ records at a time so presenting a choice or showing differences per record isn’t really viable.
If the updated data in the import contains a ‘work phone’ column, should we update existing ‘work phone’ records or should we add a secondary ‘work phone’. What if the ‘work phone’ column is blank for a particular customer, but populated on their contact record in Capsule? Should we remove the existing number or ignore? Same thing applies to other contact details, tags and also custom fields, but for custom fields we don’t have the option to add a secondary value, we can only overwrite.
I’m also concerned about the destructive nature of an import that overwrites. e.g. a user has a file that accidentally has data in the wrong columns such as an email address in the phone number field. Possibly we should be building an undo function alongside the import. However this would significantly increase the development effort. My current thinking is the user would need to resolve by re-running the import merge on the corrected file. Anyone see any issue with this?
56 comments
-
@Alec Kinnear Thanks for the feedback on this. We're best to continue this conversation via email and see if we can help you out. I am already replying to your other message around the FreshBooks integration so I'll add my comments to that. Thanks!
-
Alec Kinnear
commented
Hi Duncan,
We use the Freshbooks integration very heavily and have run into problems with merging contacts (you have to merge in the Freshbooks one, not merge to). It makes merging very fussy. Could you set up the merging to grab the Freshbooks match and hang onto it except in case of conflict please? Ideally then there would be a question about which Freshbooks account to match.
Thanks!
-
Web-ideas
commented
I've spent a while checking Capsule to make sure it meets my client's requirements and it looks great. However I came upon this stumbling block today. They need to regularly upload updates to contacts and I'd assumed that Import would do the job but alas no! I'm a developer so I'm going to have to implement my own Update facility (for People) via the API. Would this be of use to anyone else?
-
Pippa Russell
commented
Definately you need to create the ability to merge. Updating files singuarly in Capsule is too time consuming. Currently we are exporting lists and updating details in the CSV and want to reimport them. If a field is blank then it should retain the current inserted text. If there is text on any imports to merge it should overwrite the current data.
Alternatively come up with a faster easier way for us to update many files rather than singluarly.
-
Kai
commented
One use case is the export of some contacts, edit them in excel, and then import the changes. In this case it would be sufficient to include some sort of internal ID/Number which can be used to match the imported contacts to existing ones.
-
gsb
commented
Even if it has a potential for large scale distruction I would prefer a simple implementation to none. You could offer an option for handling duplicate contacts, individually and for all duplicates:
- ignore
- import and create duplicate
- import and overwrite existing
- import and merge by adding new info
(You can think of shorter names)A second option
Mac users have an easy way to merge contacts: export contacts as a vcard, import them to Addressbook. Addressbook does a very decent job of merging the contacts. I do not know whether Windows Outlook does this too, but I assume it does.
Then you could reimport the changed contacts and update all info. This way you would leave the painful merging buiness to Apple/Microsoft.Another idea:
Another program we use asks you to create a contact list with its name, in this case capsule to your Addressbook. With a function "import from addressbook" it reads all contacts in this grop and updates the info. Very convenient! -
Zahra Mamoojee
commented
Would it not be easier for the system to automatically upload but automatically create a list of the possible duplicates? This way it would avoid the need to change each record to "Import" and note it outside of the system and merge after import. The user can then check them once loaded. Perhaps a merge all function would suit some users with the option to uncheck and merge individually.
-
Mark Williams
commented
Just to update this, since there hasn't been a comment in a few months. Any progress at all here yet? As an interim measure, how about introducing an "Import All" button. I can sort of cope with the duplicates, what I definitely can not cope wit is you forcing me to scroll through a very small box painstakingly looking for re warnings and needing to set them to "Import" - because I'd rather load duplicates than miss data to be honest. This would be particularly useful for Contacts,where your system checks on names - a very strange thing to do.
-
Ian Weatherhogg
commented
It would make life MUCH easier if, at the preview step of an import, you stated how many duplicates had been found and gave us a single button that would flip them all from "Ignore" to "Import".
....when we're importing mailing lists with 1000's of entries, there are hundreds of apparent duplicates which are just coincidences, and it's extremely painful to have to scroll down and find each one. -
Perch Ten Design
commented
Here's a thought:
* Run the import dry. My guess is that email is the most sensible id to use.
* Make a record of all the conflicts you find, with each conflict also recording the type of conflict (e.g. name clash, multiple email addresses etc.).
* Present the user a list of of the conflict types and give them the option in the UI to select ones to batch solve (e.g. overwrite all names, lossless add email addresses). The options available for each merge type can be displayed in a select box.
* Also provide a UI to allow the user to drill down into each conflict type and make per-record choices to override the batch selection if they wish. Again, select box against each record.
* Store the user preferences as a map of options keyed against a suitable unique conflict identifier
* Run the actual import. Where a conflict is found check the option map for a specific user choice, if not found fall back to the batch solution.With each conflict type you find in the code as you're programming you will be able to identify the possible solutions. These are the options in the select boxes. They could include things like merge, ignore, overwrite, keep both records etc. Or with a custom field, if it is not possible to have multiple entries the options are simply 'overwrite, ignore'.
I don't think this is actually as hard as it sounds in principle. I think the trick is to accept that you need to let the user decide from the word go and build your solution around efficiently gathering the data you need from the user about their choices. Simply asking the user for each record as and when you find it during the import is going to hurt. A lot. Better to build that map.
Alternatively, just delegate it all to a third party like google contacts and call it integration :D
Hope that helps.
-
Fräulein
commented
Please also consider duplicates on a basis of phone numbers. We are using the CRM for telemarketing and would like to have a dublicate review as we are entering a new contact based on phone number... Just to throw one more in.
-
Mark
commented
Hi, I have just started using Capsule and uploaded 2600 potential clients (schools in my country) from a spreadsheet. I will need to update information about them on an annual basis from a csv file to Capsule or even upload new info into a newly created field assigned to each client. You really need to have the ability to do this. I have read the comments here about unique emails and phone nrs etc. For my client organisations each has a unique organisation name or better still a unique identifier number field. When uploading why not have a allow the upload to assign a "key" field to assign the information to. This is a normal database function and surely cannot be that difficult. Then a mass assign or merge could take place. I have already mass uploaded and deleted three times to get my initial client list right. I cannot afford to muck around here.
Thanks -
jami
commented
Duplicate Check & Merge can be a separate report, not done on import. Although if you're adding a single contact, perhaps a pop-up to signal that there is a similar record already in the system.
Duplicate merge would ideally show two records side by side, along with date entered or updated for each, and allow you to select radial buttons on either record to choose which info to keep. This way you can actually blend the data and not do a straight overwrite.
-
Karey Bacon
commented
In gmail in the Contacts page, you can check 2 (or more?) of your contacts, then click on Merge Contacts, and it adds it all together. Could it be this simple? If you just have it add everything together, then we can go in and delete the info that's no longer relevant.
-
Revenson
commented
We need a merge feature! I don't think it needs to be limited to only be available when contacts are imported. Why couldn't it be a "check entire database for duplicates" at any time? We could still accomplish the merge duplicates after import but also have the ability to clean up an existing database. If there are concerns about lost data, maybe it could be a batch system that finds 50 duplicates at a time, allows the user to see which contacts are about to be merged, and confirms the batch. If there are more than 2 duplicates of the same contact, those will be combined in a subsequent batch. Is there any progress on this?
-
Josh
commented
I'd like to see it produce a list of the duplicates and a quick merge then a review screen before it processes it.
So it would list the duplicate in column format with fields that are different highlighted in red so you can select the one that you want to keep for the new single record.
-
Tom Brereton-Downs commented
I concur with all Rob's points here. It would be great to see this happen. It makes working with large numbers of records infinitely easier and opens up all sorts of workarounds (including exporting data, working with it on a spreadsheet and then sucking it back in).
-
Rob
commented
@Capsule Support and @Duncan Stockdill -- I can't tell you how thrilled I am to read your responses. Thanks!
@Support - I suggest a balanced approach, as you said, make Capsule the best for customers old and new. How focused right now are you on making it the best for new customers? I hear ya that this is tricky
@Duncan and @Dave, you bring up good discussion points. @Dave, it sounds like you want the tools to do whatever task you want. As long as you are told what will happen, you will be okay with it. @Duncan, it sounds like your ideal situation would be to build one method of importing data that will work for everyone.
As a user, I'd just like some way to do this, even if it's not ideal (ship it!). I'd suggest starting with whatever is the simplest method of importing and handling duplicate contacts, and let users know exactly what will happen. Perhaps the user can select which fields to match on (name + address or email or phone ... ) and then warn that any existing contact info will be overwritten whenever there is data in a cell being imported. So if John exists in the database with a phone # 555-1234 and no email, and I import a John with email john@abc.com , I click "match on first name", and then the result is I have John, # 555-1234 , john@abc.com. In the result of a conflict where data exists in both, perhaps give the user the option to keep new or old across the board. There is no manual work involved, the user is told what will happen, and it's all backend so minimal extra interface needed.
Some people will obviously mess this up and not read the directions. And they're paying customers... regarding the "undo" feature.. don't you guys offer a one-click export? Does the opposite (import) exist? Why not suggest that the user export all their data first? Or save that export on your end as a means to perform the undo?
-
AdminDuncan Stockdill
(Admin, Capsule)
commented
@Dave, appreciate your views on this, so thanks for the feedback. Agree with the choice per data set of "import / ignore / merge" on duplicate. However I was hoping to avoid the user making record-by-record choices when merging since we have to build the UI around that and it's a bit manual for the user obviously. If we can find some common ground in the add/overwrite rules on merge that would be my preference since it will make the implementation a bit easier. But food for thought, so thanks.
-
Dave
commented
thanks Duncan for that feedback, I'm sure many of the posters were beginning to think we weren't being heard.
For me I want to be able to map my data to the capsule fields and then be presented with three options per import data set, import, dont import, merge.
Import would create new contacts regardless of duplicates and could be set for all contacts
dont import individual contacts that have duplicates set on an individual contact basis
merge which would compare the old and new contact with an option to select which state wins for each field. Even if I have to do this for every one of 100 contacts then it is better than creating duplicates or not importing new data to existing contacts.
For me users must be given the option to select which data to use it seems like too much to expect the software to judge which is the correct data.hope that helps rather than confuses the issue.
