Core data migration: Set a unique constraint to a parameter and avoid duplicates

Coredata migrations are easier said than done, isn’t it? One wrong step and we are doomed. 😥 

Coming from an Android background (where the platform allows writing raw SQLite queries in the migration), it feels tough to handle manual migration on iOS. Although I love the fact that lightweight migration works out of the box, when it comes to other types of migrations, it’s just too many tasks to remember, and it feels like I’m defusing a bomb.

Context

We have recently seen duplicate records saved in one of our entities. Our investigation made it clear that the uniqueId parameter (column) we have in the entity isn’t unique and is an optional parameter. 

Though we had implemented a code logic that handles upsert operation (update/insert the record based on the record availability in the entity), it was not sufficient. The duplicate record will still be saved if there’s a race condition. So, we decided to solve this at the database level. 

When I started working on it, I referred to a lot of documentation & resources but couldn’t get complete information. Most of the articles only talked about lightweight migration. Now that we have successfully migrated, I am sharing my learnings here and hope that it will help somebody who is dealing with a similar problem. 

Let’s jump right into it.

HL6SkIzfdbueOZDLxkaFKjz U7Ot0OwVtR1VDx3LiodvkmJ ibTMHyw6e1c9LcsOvupolOd7pGY jx7cunPchNcsi 4tT3w9ptOkv6aSWSMzmkbMHz7sI5d6n5Fgb31tXCOl4SgtnBv4Os3dy4MwtgV9R3FQlLKpogsiiBlf ShnxPiMpk zKmp5uA
Photo by Vitolda Klein on Unsplash

Existing Set-up

For better understanding, I’ve created a sample project with a single entity, UserEntity, in its data model. 

H0ATtCL2V29Xldc3tHhb NfaxMepKYtpF6v2K37iyVJSvswXg83DHyIWEsnWe AXOaj9B TQA0tEW4OOg1y8HoV2Eb8wVnMgoRuMnxVMApmTsPGtqWOKSvNjLD6PYv9YUAzR2GH wgPbvp6CLIUZ03 h4iYTr7XVbw3UBZJQQzCAu8rUWBVTHyc39A

UserEntity with its parameters. Observe UniqueId is an optional parameter here.

CoreData creates a table in SQLite under the hood when we run the project. The generated table structure can be seen below. A minor detail to observe here is the naming convention — table name & parameter names starts with Z, and PrimaryKey Z_PK is internally created and handled. 

9I2oj6c0 0cBwkGNr3xHe zT8yjs0UPsBBINaz7HERrDLMWKK upWk9 UJ9aFVLl86f1BO36j7HrOBnZah4fUSF50PqViPx5cz3LZMSsizqjaBVtIflYlFUDDueMn5XlWof0SpzNBz5Ogv2kcPjbk1GgcAeBDX90wHL8nY4geBpbqsM5M 6qZrNB7g

UserEntity table created in SQLite under the hood.

CREATE TABLE ZUSERENTITY ( Z_PK INTEGER PRIMARY KEY, Z_ENT INTEGER, Z_OPT INTEGER, ZAGE INTEGER, ZNAME VARCHAR, ZUNIQUEID VARCHAR )

With this setup, every time we insert a record, CoreData treats it as a new record. Hence it can store duplicate records (even when uniqueId is the same) as no unique constraint has been set. 

If the app is still under development, we could make the necessary changes on UserEntity to not accept duplicate records. What if this implementation has already been rolled-out to production, users already using the app, and they see duplicate records? Now, migration will be not able to fix the issue.

Coredata migration process

  1. Create a second version of the data model and set it as a current model. This creates a copy of the existing data model.
KFhVRDA3KYBZNdnzqwZjv6t7Fcu5RjjqIIBF7VPNgS hVcRPISg EUNahw6OEOxXWqmbfb69uzfQZwE823OvyCEQhIrRwaG7eLShyMV vEDRBEAW3RRf8 eAv 35JvnGnVwwiu pckGpptXKjEU0Ill5hZmNso71XeEd7XQ ofqNiJCHWHtundppXg

2. Add uniqueId parameter in UserEntity’s Constraints section in the data model v2. This adds a unique constraint to the parameter.

hkZnkxrvLkvs30l6RB 2PBF2if qlqPo2zqmFcd7Jmv81cTiGoYzKfRyUQmITvvSYctXU xq2GskBdO1DrnKyVoq7gY409wjUZk0KrqXURTlB1jcx51QlFTdQGpeoBu60geWAqE8nDTid4waIHw2hCURYW1aV1Leeh78MyJoYFsc ACAItW2NAbI g

3. Set NSMergePolicy to handle the merge conflict while inserting the record. Use context.mergePolicy = NSMergePolicy.mergeByPropertyObjectTrump. This MergePolicy asks Core Data to merge duplicate objects based on their properties. So, if a record with the same uniqueId is already present, CoreData will update the existing record instead of inserting a new one.

6zdBwMbzomFuthQBAeB SLEt5ZmDJmEDbsJ59FFwb YiF19EQHPBx4ny8veszwOz0ghZvqYkqS0giSsh3p2zeef4Tg10VAbZTPrvV3OBZJjK1JLVpDjlV2dOzHxLC E6Y9h A bo79v8a5qm Zn8IZp17BPsk0RuRhUk89Tu fJjgC42iik3JBZ3A

A sample implementation of NSMergePolicy

When we run the app, the migration would fail as lightweight migration cannot handle the unique constraint migration. This needs a manual migration with the help of the mapping model.

NSUnderlyingException=Constraint unique violation: UNIQUE constraint failed: ZUSERENTITY.ZUNIQUEID, reason=constraint violation during attempted migration, NSExceptionOmitCallstacks=true}}}, [“reason”: Cannot migrate store in-place: constraint violation during attempted migration, “NSUnderlyingError”: Error Domain=NSCocoaErrorDomain Code=134111 “(null)”

4. Let’s create a mapping model by selecting v1 as the source data model and v2 as the target data model.

Creating a mapping model is insufficient because it only maps the old records to the new data model. If we run the app, it would still crash by showing the same error. So, how do we solve the unique constraint exception? 

The answer is to use NSEntityMigrationPolicy

The app crashes when we run the app because CoreData wouldn’t know how to handle the existing duplicate records (if any). Since we have set the unique constraint to uniqueId, we should remove the existing duplicate records from the source data model before the migration takes place.

5. So, let’s create a custom MigrationPolicy that deletes the duplicate records. 

gH0EcopVvGnCak8JtxudimY px4XcL8WzHfC1jId0o3MlOtwSDrwaL4dIIv7DYL6MXFx2MNGKYHP1j2BuCaJunBLeBesiAyKReCQQfRY6SVtStvf q1kx0mOw6YJdS162KgPA20jwQDCrQhxtqHI4YC2hHziqAB9JcJ 3ZHZj0 w5Cp4jOu2BtijHQ

Sample code snippet to remove duplicate records from the source context

NSEntityMigrationPolicy has a begin(mapping:manager) function that the migration manager invokes at the start of the given entity mapping. We are making use of this function to run our code snippet. This code snippet gets all the records from sourceContext’s UserEntity and deletes duplicate records if any. 

6. The final step is to tell the CoreData to use this MigrationPolicy when migrating the UserEntity table. We can do so by specifying the fully name-spaced class name of MigrationPolicy_v1_v2 in the MappingModel. Make sure the entity mappingType changes from copy to custom once you add the custom policy.

jdSU7bWf8qHBUxiY8JDZBKfg4BOxafsV7YSc2lshWgEROFnfG27mHAsz3bvtrVQcc3rDMxbAbF6g97kQGgdcqHwwRYE2jJGWXt7goXVyBkgIkdzT42LKO6WoG71klDVzRCkBX7Eva33p F5By4OTezI6XkkZwZHp1F2QfC4liGSJslIx657 lvAQWQ

Final words

Were we able to run the migration successfully? Oh yeah! 

Rgiv6chhewVmKLBvfIfs 0RanGSsjJ8a5YIJAAqIThWtCiKPMYGTnY7f2i cXqNp3KghsbXYrJxhuoCON32OUxB5sxMyUGMJHaNxvKU0Rvp97YFyeFeXjs 4KqG5n2HugjUOg6wDQhrk6aKoiSsOvPkcO

If we do all these steps correctly and run the app, coredata migration succeeds and removes the existing duplicate records as well. Any future insertions will avoid duplicates by updating the existing record if it is already available in the table.

I have created a sample app to implement & test the functionality. You can refer to the source code for a detailed understanding.

Recommended Reads