Allow merging when duplicates found on contact import
We are new users of Capsule so may have missed something. It appears that I can import initially from, say, a CSV file but if I then update records in the CSV (ie bulk updates) then re-import said CSV then Capsule identifies possible duplicate and then give the opportunity to Import or Ignore - I would like an Update option.
It seems to me that by far the most efficient way to update the information in my capsule database would be to export the info to csv, edit the table in excel, save back to csv, and import into Capsule. This process could eliminate so many clicks versus trying to bulk update on the web within Capsule. Since every entry has an ID number, I see no reason why you shouldn't be able to edit the information associated with an ID number in excel, and then import and OVERWRITE the old info without creating duplicates that then have to be merged. Having to create and then merge duplicates is a major shortcoming. There are lots of cases where being able to overwrite could be extremely useful, and being able to do so would go a long way to help users maintain a "clean" database rather than one full of compounded errors and inconsistencies.
Right now, when I import a list of contacts and if any of them are already in the system, the only option that it gives right now is to overrwrite or ignore. There should be a third option, to merge the data together. This is hugely important!
We have implemented a new accounting system that requires unique contact ID numbers.
We have over 3000 customers on our database thus it would be great to export all contacts from Capsule (including any notes etc affiliated) and then open the .csv file into Excel, insert sequential numbering and then import the file back into Capsule with the function to "update duplicate records".
This function would be helpful to anyone (Administrators only - obviously) who would like to do a grand update of their database without the painful manual process which Capsule presently is!
When importing contacts, I'd like to be able to update certain details of their record, rather than rewriting the contact, or ignoring the contact. I'd like to be able to choose which datapoints to add to the duplicate record.
Ability to Update Contacts, this would be helpful in managing contacts. In salesforce, they allow you to use the contact's email as the master key for a contact since an email is unique. This would be helpful to manage contact, like last time they ordered etc
I would like to be able to import to existing contacts, instead of creating new ones then merging them, or otherwise having to manually enter new information on existing contacts. This wouold be great along with an opportunities import
we have looked into merging and concluded it’s not quite as straight forward as it might appear. Various implementations we’ve looked at aren’t without their problems.
I’d like some feedback from supporters of this feature on the following ideas and concerns:
Our initial thought was to match based on email address. However, if there are already multiple contacts in the database, what to do… do we ignore, or do we update each? If the name is different on the record being imported do we overwrite? Would it be better to match on name + email? Many of you are looking to import 100+ records at a time so presenting a choice or showing differences per record isn’t really viable.
If the updated data in the import contains a ‘work phone’ column, should we update existing ‘work phone’ records or should we add a secondary ‘work phone’. What if the ‘work phone’ column is blank for a particular customer, but populated on their contact record in Capsule? Should we remove the existing number or ignore? Same thing applies to other contact details, tags and also custom fields, but for custom fields we don’t have the option to add a secondary value, we can only overwrite.
I’m also concerned about the destructive nature of an import that overwrites. e.g. a user has a file that accidentally has data in the wrong columns such as an email address in the phone number field. Possibly we should be building an undo function alongside the import. However this would significantly increase the development effort. My current thinking is the user would need to resolve by re-running the import merge on the corrected file. Anyone see any issue with this?
Mark E Felling commented
The complexity of your questions comes down to different usage scenarios. SIMPLIFY. Import Profiles that contain predefined default of import logic for ease-of-use but also full flexibility (MIGRATION, UPDATE, or ADVANCED). These two different primary usage scenarios is what is causing all of the confusion and differences in user responses. Advanced allows your user to select an import profile for defaults then a pulldown box for each data field with selection to: (Overwrite, Ignore, Add Secondary). This then accommodates all usage scenarios including custom fields and custom data tags.
Ability to undo an import mistake is important but is easy to solve with your current UI. Add "Imported" Data Tag to all imported records with a [Date - Time] data tag value. In this way on UI front end, all imported records will be easily identified, and using your search tool one can select all records imported at a specific date and time to "Undo Import" en masse if there was an error. During the import, make a copy in a separate "import backup's table" of all records that are modified (overwritten or values added) during the import with the same "Imported" Data Tag with date and time. To undo you simply first delete all records with the matching data tag and then restore the records from the backup table with the matching data tag. Or even better give the user the ability to do each of these two steps so they can choose to restore only overwritten records and leave all new records imported.
Name + E-mail match. Before import prompt user if e-mail matches but Name does not, (i.e. Bob Smith vs. Bob G. Smith vs. Smith, Bob) whether to create new contact or after import show a list of conflicts and allow user to select which one is correct. User can decide based on size of import. If name matches exactly but e-mail does not, same question for user: create additional contact or add e-mail to existing contact.
All other fields Add Secondary by default (when possible) and overwrite custom fields and data tags by default. Or user can select Advanced and change any field to overwrite existing and not add secondary if they know the data being imported is more current or to ignore values being imported if they know the data in capsule is more current.
-If Name Matchs but e-mail does not, then (User Specified before import: update existing e-mail or add secondary).
-If e-mail matches but name does not then update name.
-If neither name nor e-mail match create new Contact record
-For all other records, overwrite existing by default. Or user can specify in Advanced to Add Secondary (when possible) or ignore for custom fields and data tags.
OK, I think I have solved your problem for all possible scenarios. Future import logic can be handled by adding a new import scenario without modifying the rest of the code. Now you guys get it implemented so we can all use it! Otherwise only option is to use Nutshell CRM or Podbox/tool to pull together/clean data first before importing (at least for migration scenario)
To me the answer ie easy. Give the responsibility to the user (us) with three choices
1) - For new emails: Add contact + opportunity
2) - For existing emails: Update contact fields + Add opportunity
3) - For existing emails: Update EMPTY contact fields + Add opportunity
If the name is different on the record being imported do we overwrite?
NO definitely match name + email, add secondary value if they are different from the one in existing record
If the updated data in the import contains a ‘work phone’ column, should we update existing ‘work phone’ records or should we add a secondary ‘work phone’
ADD a 2nd
What if the ‘work phone’ column is blank for a particular customer, but populated on their contact record in Capsule? Should we remove the existing number or ignore?
I would also like to be able to do this - and cannot understand why it is not possible to automate a refresh of existing records, given that each contact, organisation and opportunity has a Unique Reference Number (URN) / ID.
Is there a reason that the IDs are not visible within each record on screen?
Is the ability to be able to import batch updates to existing contacts in development please?
We need some forward progress on this. Something is better than the current options.
Maybe there could be a score based match. The more pieces of data that match (name, email, phone, address, etc), the higher the score. Then we get to chose. If a user selectable score for match exists, corresponding to multiple pieces of data matching, then merge.
Please make some improvement!
@Alec Kinnear Thanks for the feedback on this. We're best to continue this conversation via email and see if we can help you out. I am already replying to your other message around the FreshBooks integration so I'll add my comments to that. Thanks!
Alec Kinnear commented
We use the Freshbooks integration very heavily and have run into problems with merging contacts (you have to merge in the Freshbooks one, not merge to). It makes merging very fussy. Could you set up the merging to grab the Freshbooks match and hang onto it except in case of conflict please? Ideally then there would be a question about which Freshbooks account to match.
I've spent a while checking Capsule to make sure it meets my client's requirements and it looks great. However I came upon this stumbling block today. They need to regularly upload updates to contacts and I'd assumed that Import would do the job but alas no! I'm a developer so I'm going to have to implement my own Update facility (for People) via the API. Would this be of use to anyone else?
Pippa Russell commented
Definately you need to create the ability to merge. Updating files singuarly in Capsule is too time consuming. Currently we are exporting lists and updating details in the CSV and want to reimport them. If a field is blank then it should retain the current inserted text. If there is text on any imports to merge it should overwrite the current data.
Alternatively come up with a faster easier way for us to update many files rather than singluarly.
One use case is the export of some contacts, edit them in excel, and then import the changes. In this case it would be sufficient to include some sort of internal ID/Number which can be used to match the imported contacts to existing ones.
Even if it has a potential for large scale distruction I would prefer a simple implementation to none. You could offer an option for handling duplicate contacts, individually and for all duplicates:
- import and create duplicate
- import and overwrite existing
- import and merge by adding new info
(You can think of shorter names)
A second option
Mac users have an easy way to merge contacts: export contacts as a vcard, import them to Addressbook. Addressbook does a very decent job of merging the contacts. I do not know whether Windows Outlook does this too, but I assume it does.
Then you could reimport the changed contacts and update all info. This way you would leave the painful merging buiness to Apple/Microsoft.
Another program we use asks you to create a contact list with its name, in this case capsule to your Addressbook. With a function "import from addressbook" it reads all contacts in this grop and updates the info. Very convenient!
Zahra Mamoojee commented
Would it not be easier for the system to automatically upload but automatically create a list of the possible duplicates? This way it would avoid the need to change each record to "Import" and note it outside of the system and merge after import. The user can then check them once loaded. Perhaps a merge all function would suit some users with the option to uncheck and merge individually.
Mark Williams commented
Just to update this, since there hasn't been a comment in a few months. Any progress at all here yet? As an interim measure, how about introducing an "Import All" button. I can sort of cope with the duplicates, what I definitely can not cope wit is you forcing me to scroll through a very small box painstakingly looking for re warnings and needing to set them to "Import" - because I'd rather load duplicates than miss data to be honest. This would be particularly useful for Contacts,where your system checks on names - a very strange thing to do.
Ian Weatherhogg commented
It would make life MUCH easier if, at the preview step of an import, you stated how many duplicates had been found and gave us a single button that would flip them all from "Ignore" to "Import".
....when we're importing mailing lists with 1000's of entries, there are hundreds of apparent duplicates which are just coincidences, and it's extremely painful to have to scroll down and find each one.
Perch Ten Design commented
Here's a thought:
* Run the import dry. My guess is that email is the most sensible id to use.
* Make a record of all the conflicts you find, with each conflict also recording the type of conflict (e.g. name clash, multiple email addresses etc.).
* Present the user a list of of the conflict types and give them the option in the UI to select ones to batch solve (e.g. overwrite all names, lossless add email addresses). The options available for each merge type can be displayed in a select box.
* Also provide a UI to allow the user to drill down into each conflict type and make per-record choices to override the batch selection if they wish. Again, select box against each record.
* Store the user preferences as a map of options keyed against a suitable unique conflict identifier
* Run the actual import. Where a conflict is found check the option map for a specific user choice, if not found fall back to the batch solution.
With each conflict type you find in the code as you're programming you will be able to identify the possible solutions. These are the options in the select boxes. They could include things like merge, ignore, overwrite, keep both records etc. Or with a custom field, if it is not possible to have multiple entries the options are simply 'overwrite, ignore'.
I don't think this is actually as hard as it sounds in principle. I think the trick is to accept that you need to let the user decide from the word go and build your solution around efficiently gathering the data you need from the user about their choices. Simply asking the user for each record as and when you find it during the import is going to hurt. A lot. Better to build that map.
Alternatively, just delegate it all to a third party like google contacts and call it integration :D
Hope that helps.
Please also consider duplicates on a basis of phone numbers. We are using the CRM for telemarketing and would like to have a dublicate review as we are entering a new contact based on phone number... Just to throw one more in.
Hi, I have just started using Capsule and uploaded 2600 potential clients (schools in my country) from a spreadsheet. I will need to update information about them on an annual basis from a csv file to Capsule or even upload new info into a newly created field assigned to each client. You really need to have the ability to do this. I have read the comments here about unique emails and phone nrs etc. For my client organisations each has a unique organisation name or better still a unique identifier number field. When uploading why not have a allow the upload to assign a "key" field to assign the information to. This is a normal database function and surely cannot be that difficult. Then a mass assign or merge could take place. I have already mass uploaded and deleted three times to get my initial client list right. I cannot afford to muck around here.
Duplicate Check & Merge can be a separate report, not done on import. Although if you're adding a single contact, perhaps a pop-up to signal that there is a similar record already in the system.
Duplicate merge would ideally show two records side by side, along with date entered or updated for each, and allow you to select radial buttons on either record to choose which info to keep. This way you can actually blend the data and not do a straight overwrite.
Karey Bacon commented
In gmail in the Contacts page, you can check 2 (or more?) of your contacts, then click on Merge Contacts, and it adds it all together. Could it be this simple? If you just have it add everything together, then we can go in and delete the info that's no longer relevant.
We need a merge feature! I don't think it needs to be limited to only be available when contacts are imported. Why couldn't it be a "check entire database for duplicates" at any time? We could still accomplish the merge duplicates after import but also have the ability to clean up an existing database. If there are concerns about lost data, maybe it could be a batch system that finds 50 duplicates at a time, allows the user to see which contacts are about to be merged, and confirms the batch. If there are more than 2 duplicates of the same contact, those will be combined in a subsequent batch. Is there any progress on this?