Allow merging when duplicates found on contact import
Our situation is this..we need to tag our organisations on a weekly basis, i.e. with the date that a mailshot has been sent. Doing this manually is practically impossible as it takes an inordinate amount of time. Our solution is quite simple. Instead of creating duplicates when you import a CSV file, allow us to overwrite the existing record. A current work around is to re-import the CSV file, a tedious task, as the default is "Ignore", so every single imported record has to be manually accepted and then merged. Which is OK for 2 or 3 records. However, when you are doing, say, 500, it's takes too long. A simple solution would be to overwrite a record. I'm sure many people would use it. When we upload a CSV to our eMarketing platform with a simple tick box, we can overwrite all the records in a queue.
make it possible to first export my contacts to a excelsheet using a .csv file, then making changes, then importing them and automaticly merge them with the existing contacts. Because it's much easier to see missing data from contacts in an excelsheet.
We are new users of Capsule so may have missed something. It appears that I can import initially from, say, a CSV file but if I then update records in the CSV (ie bulk updates) then re-import said CSV then Capsule identifies possible duplicate and then give the opportunity to Import or Ignore - I would like an Update option.
It seems to me that by far the most efficient way to update the information in my capsule database would be to export the info to csv, edit the table in excel, save back to csv, and import into Capsule. This process could eliminate so many clicks versus trying to bulk update on the web within Capsule. Since every entry has an ID number, I see no reason why you shouldn't be able to edit the information associated with an ID number in excel, and then import and OVERWRITE the old info without creating duplicates that then have to be merged. Having to create and then merge duplicates is a major shortcoming. There are lots of cases where being able to overwrite could be extremely useful, and being able to do so would go a long way to help users maintain a "clean" database rather than one full of compounded errors and inconsistencies.
Right now, when I import a list of contacts and if any of them are already in the system, the only option that it gives right now is to overrwrite or ignore. There should be a third option, to merge the data together. This is hugely important!
We have implemented a new accounting system that requires unique contact ID numbers.
We have over 3000 customers on our database thus it would be great to export all contacts from Capsule (including any notes etc affiliated) and then open the .csv file into Excel, insert sequential numbering and then import the file back into Capsule with the function to "update duplicate records".
This function would be helpful to anyone (Administrators only - obviously) who would like to do a grand update of their database without the painful manual process which Capsule presently is!
When importing contacts, I'd like to be able to update certain details of their record, rather than rewriting the contact, or ignoring the contact. I'd like to be able to choose which datapoints to add to the duplicate record.
Ability to Update Contacts, this would be helpful in managing contacts. In salesforce, they allow you to use the contact's email as the master key for a contact since an email is unique. This would be helpful to manage contact, like last time they ordered etc
I would like to be able to import to existing contacts, instead of creating new ones then merging them, or otherwise having to manually enter new information on existing contacts. This wouold be great along with an opportunities import
we have looked into merging and concluded it’s not quite as straight forward as it might appear. Various implementations we’ve looked at aren’t without their problems.
I’d like some feedback from supporters of this feature on the following ideas and concerns:
Our initial thought was to match based on email address. However, if there are already multiple contacts in the database, what to do… do we ignore, or do we update each? If the name is different on the record being imported do we overwrite? Would it be better to match on name + email? Many of you are looking to import 100+ records at a time so presenting a choice or showing differences per record isn’t really viable.
If the updated data in the import contains a ‘work phone’ column, should we update existing ‘work phone’ records or should we add a secondary ‘work phone’. What if the ‘work phone’ column is blank for a particular customer, but populated on their contact record in Capsule? Should we remove the existing number or ignore? Same thing applies to other contact details, tags and also custom fields, but for custom fields we don’t have the option to add a secondary value, we can only overwrite.
I’m also concerned about the destructive nature of an import that overwrites. e.g. a user has a file that accidentally has data in the wrong columns such as an email address in the phone number field. Possibly we should be building an undo function alongside the import. However this would significantly increase the development effort. My current thinking is the user would need to resolve by re-running the import merge on the corrected file. Anyone see any issue with this?
Mary Butlin commented
Just to add to my previous comment - perhaps there could be two matching options: Email and Capsule ID number. Email address would be the simple and most used version. But if a user wished to put in more time to matching up to existing Capsule records, they could decide for themselves whether to match on name, postal address etc. They'd do this via a full export from Capsule, then matching their data using Excel/alternative to create an Import file with the new data matched to Capsule IDs. I've used other systems that work this way and it's really helpful to allow the advanced user more options - it also means the onus is on the user to get the matching right, not on Capsule.
Mary Butlin commented
I agree with Robin on this - a really helpful addition would be 'Default behaviour for a list import' - i.e. either ignore or add the potential duplicates. I spend a lot of time clicking individual records to make sure potential dupes get imported (for subsequent manual merging) which is really annoying. I hope this would be a simple addition?
In terms of auto-merging, my feeling would be to match on email address - if there are more than one matches, don't merge but spit out a CSV file the organisation can work through manually. I think it should be a setting on whether or not to over-write the Name field with the new data, or keep it as per Capsule record (in which case add the alternative name as a note). For the other queries, work phone, tags, custom fields etc. I think it should be a setting whether you over-write existing data fields or keep Capsule as is, most of the time it's about appending more information rather than over-writing existing information.
Lastly re-running the import merge on the corrected file (in the event of an error) sounds like a good alternative to an 'undo' function which I realise would be a lot of development work.
As per other comments, best to make decisions on all of the above and release some functionality rather than leaving us with no functionality - i.e. having to click through hundreds of records so that duplicate data still gets imported, and then having to manually merge many records.
A CSV/Excel report of those that didn't get imported would be really useful.
Robin Carswell commented
Two user stories
1. Select default behaviour for a list
As a list uploader, I would like to select a default selected behaviour for the list I upload (ignore / add), so that I don't have to click through the default behaviour currently implemented (a really long click marathon)
2. Allow merge as default behaviour
As a list uploader, I would like the option of merging "suspected duplicates", so that I don't have to manually merge each one later (another click marathon). This would expand the list above to allow auto-merge as a default option
Hi Duncan, I'm with Ian Curry on this, it's a serious weakness. Whatever you call it (merge, bulk update, whatever) the impact is that once your contact / org data is loaded you cannot execute any changes (apart form add a tag, or delete) to the data except one at a time. For items with data tags it makes even less sense. The whole point of a data tag is that it can be used programmatically somehow. The solution might be to write a bulk update tool that uses the API, we're looking at this at the moment. If that works we'll licence it to other capsule users.
Ian Curry commented
New users to Capsule and seriously disappointed with the inability to import data to update people/organisations. A very simple implementation with warnings would be far better than doing nothing for 5 years!!
So the scenario is I have multiple files that contain a column that is a key field e.g. email address. I have another column that I then want to load into a custom field or data tag where the key field matches.
We are still in trial period and this is seriously impacting on our ability to add value to the data and ultimately run our business.
Mark E Felling commented
The complexity of your questions comes down to different usage scenarios. SIMPLIFY. Import Profiles that contain predefined default of import logic for ease-of-use but also full flexibility (MIGRATION, UPDATE, or ADVANCED). These two different primary usage scenarios is what is causing all of the confusion and differences in user responses. Advanced allows your user to select an import profile for defaults then a pulldown box for each data field with selection to: (Overwrite, Ignore, Add Secondary). This then accommodates all usage scenarios including custom fields and custom data tags.
Ability to undo an import mistake is important but is easy to solve with your current UI. Add "Imported" Data Tag to all imported records with a [Date - Time] data tag value. In this way on UI front end, all imported records will be easily identified, and using your search tool one can select all records imported at a specific date and time to "Undo Import" en masse if there was an error. During the import, make a copy in a separate "import backup's table" of all records that are modified (overwritten or values added) during the import with the same "Imported" Data Tag with date and time. To undo you simply first delete all records with the matching data tag and then restore the records from the backup table with the matching data tag. Or even better give the user the ability to do each of these two steps so they can choose to restore only overwritten records and leave all new records imported.
Name + E-mail match. Before import prompt user if e-mail matches but Name does not, (i.e. Bob Smith vs. Bob G. Smith vs. Smith, Bob) whether to create new contact or after import show a list of conflicts and allow user to select which one is correct. User can decide based on size of import. If name matches exactly but e-mail does not, same question for user: create additional contact or add e-mail to existing contact.
All other fields Add Secondary by default (when possible) and overwrite custom fields and data tags by default. Or user can select Advanced and change any field to overwrite existing and not add secondary if they know the data being imported is more current or to ignore values being imported if they know the data in capsule is more current.
-If Name Matchs but e-mail does not, then (User Specified before import: update existing e-mail or add secondary).
-If e-mail matches but name does not then update name.
-If neither name nor e-mail match create new Contact record
-For all other records, overwrite existing by default. Or user can specify in Advanced to Add Secondary (when possible) or ignore for custom fields and data tags.
OK, I think I have solved your problem for all possible scenarios. Future import logic can be handled by adding a new import scenario without modifying the rest of the code. Now you guys get it implemented so we can all use it! Otherwise only option is to use Nutshell CRM or Podbox/tool to pull together/clean data first before importing (at least for migration scenario)
To me the answer ie easy. Give the responsibility to the user (us) with three choices
1) - For new emails: Add contact + opportunity
2) - For existing emails: Update contact fields + Add opportunity
3) - For existing emails: Update EMPTY contact fields + Add opportunity
If the name is different on the record being imported do we overwrite?
NO definitely match name + email, add secondary value if they are different from the one in existing record
If the updated data in the import contains a ‘work phone’ column, should we update existing ‘work phone’ records or should we add a secondary ‘work phone’
ADD a 2nd
What if the ‘work phone’ column is blank for a particular customer, but populated on their contact record in Capsule? Should we remove the existing number or ignore?
I would also like to be able to do this - and cannot understand why it is not possible to automate a refresh of existing records, given that each contact, organisation and opportunity has a Unique Reference Number (URN) / ID.
Is there a reason that the IDs are not visible within each record on screen?
Is the ability to be able to import batch updates to existing contacts in development please?
We need some forward progress on this. Something is better than the current options.
Maybe there could be a score based match. The more pieces of data that match (name, email, phone, address, etc), the higher the score. Then we get to chose. If a user selectable score for match exists, corresponding to multiple pieces of data matching, then merge.
Please make some improvement!
@Alec Kinnear Thanks for the feedback on this. We're best to continue this conversation via email and see if we can help you out. I am already replying to your other message around the FreshBooks integration so I'll add my comments to that. Thanks!
Alec Kinnear commented
We use the Freshbooks integration very heavily and have run into problems with merging contacts (you have to merge in the Freshbooks one, not merge to). It makes merging very fussy. Could you set up the merging to grab the Freshbooks match and hang onto it except in case of conflict please? Ideally then there would be a question about which Freshbooks account to match.
I've spent a while checking Capsule to make sure it meets my client's requirements and it looks great. However I came upon this stumbling block today. They need to regularly upload updates to contacts and I'd assumed that Import would do the job but alas no! I'm a developer so I'm going to have to implement my own Update facility (for People) via the API. Would this be of use to anyone else?
Pippa Russell commented
Definately you need to create the ability to merge. Updating files singuarly in Capsule is too time consuming. Currently we are exporting lists and updating details in the CSV and want to reimport them. If a field is blank then it should retain the current inserted text. If there is text on any imports to merge it should overwrite the current data.
Alternatively come up with a faster easier way for us to update many files rather than singluarly.
One use case is the export of some contacts, edit them in excel, and then import the changes. In this case it would be sufficient to include some sort of internal ID/Number which can be used to match the imported contacts to existing ones.
Even if it has a potential for large scale distruction I would prefer a simple implementation to none. You could offer an option for handling duplicate contacts, individually and for all duplicates:
- import and create duplicate
- import and overwrite existing
- import and merge by adding new info
(You can think of shorter names)
A second option
Mac users have an easy way to merge contacts: export contacts as a vcard, import them to Addressbook. Addressbook does a very decent job of merging the contacts. I do not know whether Windows Outlook does this too, but I assume it does.
Then you could reimport the changed contacts and update all info. This way you would leave the painful merging buiness to Apple/Microsoft.
Another program we use asks you to create a contact list with its name, in this case capsule to your Addressbook. With a function "import from addressbook" it reads all contacts in this grop and updates the info. Very convenient!
Zahra Mamoojee commented
Would it not be easier for the system to automatically upload but automatically create a list of the possible duplicates? This way it would avoid the need to change each record to "Import" and note it outside of the system and merge after import. The user can then check them once loaded. Perhaps a merge all function would suit some users with the option to uncheck and merge individually.
Mark Williams commented
Just to update this, since there hasn't been a comment in a few months. Any progress at all here yet? As an interim measure, how about introducing an "Import All" button. I can sort of cope with the duplicates, what I definitely can not cope wit is you forcing me to scroll through a very small box painstakingly looking for re warnings and needing to set them to "Import" - because I'd rather load duplicates than miss data to be honest. This would be particularly useful for Contacts,where your system checks on names - a very strange thing to do.
Ian Weatherhogg commented
It would make life MUCH easier if, at the preview step of an import, you stated how many duplicates had been found and gave us a single button that would flip them all from "Ignore" to "Import".
....when we're importing mailing lists with 1000's of entries, there are hundreds of apparent duplicates which are just coincidences, and it's extremely painful to have to scroll down and find each one.
Perch Ten Design commented
Here's a thought:
* Run the import dry. My guess is that email is the most sensible id to use.
* Make a record of all the conflicts you find, with each conflict also recording the type of conflict (e.g. name clash, multiple email addresses etc.).
* Present the user a list of of the conflict types and give them the option in the UI to select ones to batch solve (e.g. overwrite all names, lossless add email addresses). The options available for each merge type can be displayed in a select box.
* Also provide a UI to allow the user to drill down into each conflict type and make per-record choices to override the batch selection if they wish. Again, select box against each record.
* Store the user preferences as a map of options keyed against a suitable unique conflict identifier
* Run the actual import. Where a conflict is found check the option map for a specific user choice, if not found fall back to the batch solution.
With each conflict type you find in the code as you're programming you will be able to identify the possible solutions. These are the options in the select boxes. They could include things like merge, ignore, overwrite, keep both records etc. Or with a custom field, if it is not possible to have multiple entries the options are simply 'overwrite, ignore'.
I don't think this is actually as hard as it sounds in principle. I think the trick is to accept that you need to let the user decide from the word go and build your solution around efficiently gathering the data you need from the user about their choices. Simply asking the user for each record as and when you find it during the import is going to hurt. A lot. Better to build that map.
Alternatively, just delegate it all to a third party like google contacts and call it integration :D
Hope that helps.