Upgrading Dynamics CRM can be a big task. Everyone from the developers to the end users need to be aware of the scope of changes with the new release, the impact it will have on their current deployment, and the time it will take to implement, test and deploy. So, upgrading across multiple releases can be downright scary. Multi-release upgrades can magnify the effort several times depending on the complexity of the solution.
Many companies can understandably find themselves in a multi version upgrade situation. Organizations may not be able to afford the time it takes to upgrade their system given constraints around development resources, scheduling downtime, and active customer feature requests. So upgrades are put off repeatedly and the they fall behind in releases. With Microsoft’s recent switch to a more aggressive release schedule for Dynamics CRM, just keeping up on the changes can be a full-time job.
I have been assisting a customer with upgrades from CRM 2011 On Premise to CRM 2016 On Premise. They previously upgraded a highly customized (and in many ways unsupported) CRM 4 deployment to a nearly 100% supported CRM 2011 implementation. They have multiple CRM deployments and the upgrade/migration for each had their own challenges but each team did a great job in planning for subsequent upgrades.
With all of the general preparation, each upgrade still requires a solid plan for execution. In this and subsequent posts, I’d like to share some of the planning approaches, challenges, and “gotchas” that we’ve seen during an ongoing migration. The system that we are currently migrating is a very large CRM 2011 On Premise xRM solution, leveraging much of the out of the box capabilities while greatly expanding with custom code and custom entities.
Planning for your upgrade
Before diving into my customer’s experience, we can take a look at some very high level topics to consider. This is by no means the definitive list but it should be a nice starting point for the discussion.
- What resources will you need?
Your team will need some environments to do your work. With an On Premise deployment, this can mean new servers, new hosted virtual machines, or new developer virtual machines. Setting up these environments can be costly in both dollars and time, so it needs to be considered as early as possible
- How does this affect your current production system?
What happens to the current production system while you upgrade? Will you provide new features and updates while the upgrade effort is taking place? With large migrations, this can cause a LOT of churn and possible issues with merging updates. Your management team will need to consider a code freeze at some point during the process
- What steps will you need to take in your migration?
If you have a large solutions that includes a large amount of code with a complex schema, how do you sync the customizations and source code? This does not seem like it would be complicated, but planning out the stages/steps in advance can save time for your development team
- What about all of the cool new features?
With major releases, you usually see some new feature that you want to leverage. Some new features may not be relevant while some may be extremely easy to implement. However, other features might require data migration and process changes. Planning time for your team to review new capabilities during the migration can be key to making the most out of the CRM platform
- How will you test your migration?
If your current solution is supported and you follow all of the steps provided by Microsoft, it should just work, right? Well, sure… but that is an awful big leap of faith to take with this project! A full regression test of your system will likely be required, so pulling in your end users and QA team as early as possible can help greatly
- How will you complete the final migration and deployment?
When everything is complete, your solution is migrated and tested, how will you go about the final deployment? As with any system roll out, your team should plan for down time, end-user testing, remediation, and acceptance. If your upgrade included any data migration or transformation steps, this will need some extra testing and validation
My customer experience, part one
The first step in planning my customer upgrade was a review overall solution and choose an upgrade approach based on Microsoft recommendations. We know that the solution is large and complex, so we knew that we needed to take a staged approach. Fortunately, you can find a solid amount of material on upgrade planning with Microsoft online at the following link:
https://technet.microsoft.com/en-us/library/hh699716.aspx#BKMK_Upgradeoptions The first thing that Microsoft offers are three approaches to your upgrade. Our team chose the first option listed:
Migrate by using a new instance of SQL Server. We recommend this option for upgrading Microsoft Dynamics CRM Server. Although this option requires a different computer for Microsoft Dynamics CRM 2016 and a different instance of SQL Server, it provides the least amount of potential downtime for users since the existing deployment can remain functioning until the upgrade is completed and verified.
What this means choice for our migration: our team took the approach of leveraging the Deployment Administrator Organization Import feature to upgrade the database on a new machine. Right away, we had infrastructure resource requirements: a new CRM 2016 instance on which we would perform the upgrade. Unfortunately, a CRM 2011 database cannot be imported to a CRM 2016 instance, so we also needed an instance of CRM 2013 and CRM 2015. My customer has an excellent team that provides virtual infrastructure so they provided us a few new virtual machines for the upgrade, such as the CRM 2013 and 2015 instances required for import, three CRM 2016 dev sandboxes, and a CRM 2016 testing sandbox. The development team quickly had the resources to begin testing some of the initial migration steps and reviewing new CRM 2016 functionality.
A staged approach
Another decision the team made was to simply port the system and get a functional CRM 2016 instance with the current solution customizations. This would allow us to get a working solution in front of our customers very early and get some feedback. This also meant that we did not need the code ported and working to get the system in front of the users. We would disable any of the scripts that caused errors or required migration, disable any plugins and workflows that would cause problems, etc. Essentially, we wanted to present users with the coming attractions.
This approach is important because with the CRM 2013 release, we knew that the user interface saw a massive overhaul, so the user experience would change drastically. So my customer wanted to get their users on board and familiar with the upgraded system as soon as possible. We aimed to avoid major shock by the end-users when they saw the new user interface for the first time while providing some early training, gathering early feedback, and to simply gathering ideas by the customer. It’s great to see users participate with conversations often sounding like, “Oh, that’s cool. Can you also make it do this too?”
While the end users are reviewing the updated system, the development team is able to proceed with the code migration analysis, including plugins, workflows, scripts, etc. This exercise allowed planning our iterative releases to the user team and QA team. Basically, how do we roll out the migrated features to the end users? Even though we are simply porting over the current functionality, some of the areas are very complex. Performing an analysis of the code base allowed the team a chance to review features and functionality that they may not have addressed in some time, provide insight into a testing and release planning, while at the same time, deciding what new functionality can be leveraged with the new release. For example, would Business Process Flows benefit the end users? What JavaScript can be ported over to business rules? Where can we use Actions effectively? All of these items would be tracked in preparation for a later stage in the migration.
All coming together…
So, with some of these decisions for approach, our big picture plan started taking shape. Very high level steps for the migration included:
- Migrate the CRM database from CRM 2011 to CRM 2016
- Disable any broken functionality
- Present to the end-user team
- Begin code migration and upgrade
- Evaluate new capabilities available in CRM 2016
- Begin iterative releases to QA and end-user testing
With this high level plan, we have some actionable steps to take as a team a plan that gets us in front of the customer early and often!
Next steps?
In my next post, I will go into some of the details on the database migration steps. These are fairly well documented but we ran into a few situations that are worth a discussion!
========================================================================
In my first post,
Dynamics CRM Upgrade: 2011 to 2016, Part 1, we looked at some general considerations with a CRM 2011 to CRM 2016 On Premise migration. This touched on resource planning, assessing new features, test and release planning, and the final roll out. We began discussing my current customer experience with an enterprise system migration of a very complex, xRM heavy CRM 2011 on premise system to CRM 2016 on premise. The next step in the migration focuses on bringing the database through the upgrade steps.
Migrating the Database
Fortunately, upgrading the database is relatively easy. The
Microsoft Dynamics CRM Deployment Manager will apply all of the database upgrade scripts for the you. Simply point the Deployment Manager to your database and away we go!
Ok, as I am sure you guessed, it’s not that simple. Almost all of the work on the database is still going to be done for you, but you still need to jump through a few hoops. As mentioned, we are not going to upgrade the database in place, which is good since we actually CAN’T do an in place migration. This is because we are upgrading across several versions of Dynamics CRM and the Deployment Manager can only handle one version upgrade at a time. Meaning, we can’t just install Dynamics CRM 2016 and use the Deployment Manager to upgrade our database. We have a few steps to take first. This limitation has a big impact on resource and schedule planning as well as your detailed migration steps.
Database Migration Steps
Steps for upgrading a Dynamics CRM database by one version are relatively straightforward. Assuming we are in the same domain, the steps are essentially as follows:
- Install new CRM version on target system
- Perform a Full DB backup of current system
- Restore DB to target system database server
- Import Organization using Deployment Manager Organization Import Wizard on target system
Once the Organization Import Wizard completes, you should have a functioning system on your target environment, upgraded to the new version of Dynamics CRM. All instance database data should be migrated, such as system user records, Accounts, Contacts, custom entity records, etc. Users on your current system should be able to login to this new instance with the same credentials and see the same data as they saw just before the full database backup. You may see some issues with your system depending on the customizations made, such as some JavaScript or plugin errors. This will all depend on your current solution. But the general idea is that if we were to take a baseline CRM instance and upgrade from one version to the next, the system should be fully functional post Organization Import Wizard.
Since we are moving from CRM 2011 to CRM 2016, we will need to repeat these steps for each version through which we are upgrading:
- CRM 2011 production -> CRM 2013 staging
- CRM 2013 staging -> CRM 2015 staging
- CRM 2015 staging -> CRM 2016 staging
- CRM 2016 staging -> CRM 2016 production
These steps are a bit simplified because there is a lot of work to be done before finally deploying to the CRM 2016 production instance. But I think it illustrates the steps required and the impact on your upgrade planning.
Migration planning (again!)
As you can see from the Technet articles, the steps for the Import Wizard are relatively straightforward. But your organization will need an environment for each interim version of CRM listed above. This can require significant time and resources depending on your organization’s current capabilities. My customer’s infrastructure team set up virtual environments for the development team to accomplish each of these steps. Each virtual environment is self contained with full installations, meaning they ran SQL Server, SQL Server Reporting Services, and Dynamics CRM with all of the server roles. This allows the dev team to handle resources as needed, such as beefing up memory for the migration or shutting down the entire virtual machine when not in use. Fortunately, my customer has an excellent infrastructure team in place. They set up the environments, provided access and managed resources to each, and provided the correct roles and accounts required for the migration steps. This work saves the development team a big chunk of time during our migration stages.
Once you have the infrastructure in place, this process can also take a lot of time! Your team will need to schedule accordingly for this stage and the time required depends on the size of your organization’s database and the complexity of your customizations. My customer’s database is approximately 50GB and includes a large number of attachments, significant record sharing, and a complex set of entity customizations to accommodate their xRM system. Running this database through the Organization Import wizard from CRM 2011 to CRM 2013 can take a few hours depending on the resources assigned to the server performing the upgrade. With the steps outlined above, you can see that we need to perform this Organization Import wizard three times. This ALSO includes performing the database backup, copy from one system to the next, and a full database restore on the target system before running the Organization Import Wizard. Assuming all goes smoothly, this process can take up an entire day for our team.
So your team should plan accordingly:
- Do you have the sufficient infrastructure in place to upgrade?
- Do you have people on your team to manage the infrastructure?
- Do you have the time required to upgrade the database included in your overall project plan?
Answers to each of these questions will entirely depend on your organization’s situation but asking and answering them before diving in can be critical to keeping your migration on track.
In following posts, I think we have a few more things to discuss around the database migration. I will discuss some detailed issues we encountered during the database migration steps and how we addressed them, and a few things we did to streamline our development process. I will also discuss what actually happens to your database once you upgrade and how that can impact subsequent stages of the upgrade process.
========================================================================
A quick recap of parts 1 and 2 of this series, Dynamics CRM Upgrade: 2011 to 2016: In
part 1, we looked at some general considerations with a CRM 2011 to 2016 On Premise migration, such as resource planning, assessing new features, test and release planning, and the final roll out. I introduced my current customer experience with an enterprise system migration of a very complex, xRM heavy CRM 2011 on premise system to CRM 2016 on premise. In my latest post with
part 2, we focused on the database portion of the migration, such as the staged approach and moving the upgrade through different iterations. We also discussed the impact of these requirements on resources, timing, and overall planning.
In this post, we can take a look at some of the issues faced when backing up, restoring, and importing our CRM 2011 database. We can also take a look at a few things that might help streamline the process.
Some bumps in the road
The staged approach entails importing our organization database through multiple versions of CRM in order to complete the full database migration. As mentioned in
part 2, once the organization import is complete, we should have a working CRM 2013 system. As discussed, my customer has a heavily customized xRM system, which means many plugins have been registered and deployed to the database. With the import and upgrade, these plugins are still registered and SDK message processing steps are ready to fire. Unfortunately, a completely new CRM version means that our old plugin code will not work! Because this is a major version release, our CRM 2011 plugin assemblies will fail to load causing exceptions. If we do not trigger an event for which a plugin is registered, such as an on Create plugin, this should really not pose an issue. But my customer registered a Retrieve Multiple plugin that fires each time we brought up a list of records. This meant that we could not work with most of the system, either viewing our customer data or accessing the administrative screens.
In order to avoid this, we simply disabled the Retrieve Multiple plugin before performing the Organization import on CRM 2013. In our case, we did not want to impact the production system so we did not disable the plugin step on CRM 2011. Once our database was restored to the CRM 2013 SQL Server, we run a SQL script to disable the plugin execution steps BEFORE we ran the organization import. With this SDK message processing step was disabled, we could navigate our system and only ran into issues if we triggered the other plugins, such as an On Create or On Update plugin.
Another issue we faced early on occurred the SECOND time we ran the import. We pulled a new backup from production because my customer made some code and configuration updates and it was time to merge our updates. I had hoped to preserve my existing tenant on my development virtual machine in case we had issues, so I restored the new backup as a new database on SQL server. When attempting to import the organization using the Deployment Manager Organization Import Wizard, I received an error indicating that the Organization GUID already existed. Because I was importing a backup from the same source system, the Organization GUID clashed with the existing Organization registered in the configuration database. I have seen a few articles on how to change the Organization GUID so that you can avoid this clash. I don’t want to post any examples here, since I have personally not attempted any of the methods I’ve found and I really do not believe these are supported! Also, because my customer wants to replicate the database import steps as close as possible to the final go-live migration, we take the step of deleting the existing database and performing a new restore each time we pull in a new Production backup. My customer’s infrastructure team also provide some additional development environments that made our work much easier!
Another odd issue that we faced occurred during the CRM 2015 database import. We had successfully imported the database into our CRM 2013 instance and after the backup and restore to our CRM 2015 instance, we kicked off the Deployment Manager Import wizard. The import failed and after some digging into the log files provided by the import wizard, we saw that one of the Activity Feed plugins attempted to fire. It seems that these were updated during the CRM 2013 import and they now caused issues with the CRM 2015 upgrade process. What was also confusing is that the Activity Feed plugin SDK message processing steps are not displayed by default within the Default Solution. This was a relatively easy fix once we realized what was going on. The steps we took:
- Navigate to Settings -> Customizations and choose Customize the System
- View SDK Message Processing Steps and select View:All to see the Activity Feed plug-in
- Select All SDK Message Processing Steps and Deactivate
Once we performed these steps, we backed up the database again and the import on CRM 2015 worked great.
Streamline the process
During the development period of upgrading the system, my customer had to run the migration several times. In order to speed things up a bit during the migration, we took a few steps that seemed to save us some time. As mentioned in my earlier posts, even just copying the files between servers can take some time! So a few of these approaches might help.
- Disable plugins and workflows – I had assumed that when performing an Organization import, only the database would be affected and the rest of the system was essentially offline. But after running into the issue with the Activity Feeds plugin mentioned earlier, I am assuming that this is not the case! So if you have the option of disabling these during your import steps, I am thinking that it would reduce some of the time to complete. This is one of those items for which I would like to get some clarification. If anyone knows in the meantime, please drop me a line, and I if I get an answer, I’ll update the post.
- Cleanup your DB! – I am sure most reading this have some experience with your CRM databases increasing in size over time. One very popular culprit is the AsyncOperationBase table. The following support article provides details on the issue and some scripts will help to clean things up: Performance is slow if the AsyncOperationBase table becomes too large in Microsoft Dynamics CRM. If your organization has run into this issue, you likely have some of these steps already in place to run periodically. Another clean up script that can clean up some space is to clean up Outlook client sync data. This article by Sean McNellis provides a good summary and the script: Cleaning up CRM Sync Entry tables. These are older posts, but remember that we are running against a CRM 2011 database. I run each of these scripts, and a few other specific to my customer’s system, right after I import the database on the CRM 2013 SQL Server. Here is yet another blog post by Ben Hosking about database cleanup and maintenance: CRM 2011 – Cutting my CRM database down to size This is another topic on which feedback and suggestions are welcome.
- Prune some data – The cleanup steps already mentioned will clean up your database, but what if your system has a LOT of old data that is no longer being used? Does your system have an archive or data warehouse to which you periodically push data? If so, this is an additional step that could shrink your DB and simply give the import process less data on which it needs to work, and moving the backups. This step is completely dependent on your organization’s situation and may simply not be an option because of internal or external regulations, but it is worth investigating early on as it may have an impact on your delivery. Even if it’s only to force your team to review its data, processes, and long-term data management strategies, it’s a good exercise if you can spare the time!
- Just drop some stuff – Once again it’s about reducing the size of the database, but here, you can drop some data temporarily. My customer deals with a large number of attachments and annotations. They regularly use the Note attachments for moving around reports, sharing documents, etc, in addition to their syncing of email with their system that might contain attachments. So the dev team decided that during the their development cycle, we would simply drop all annotations from the system prior to import. Unfortunately, I don’t have the exact number, but during one migration test, the compressed database backup shrank from 50GB to 9GB. This really cut down on moving files around as well as the database restore, and each stage of the import from server to server. For multiple iterations, this really saved us time, but during the actual migration, we will not be dropping the annotations.
- Bump up resources – Throw hardware at it! If you have the resources available, I would suggest bumping up the CPUs and RAM on the target environments. The database restore and the Deployment Manager import wizard can really use a lot of memory and CPU, so an increase during running these times. In my customer’s case, they have a virtualized environment, so they have some flexibility in managing resources. And during the actual go live, planning for these additional resources can definitely reduce your downtime. Even if the calendar time remains, say you block off a weekend for the upgrade, giving the migration team a few extra hours to deal with issues or run tests can be an enormous help.
- Time it accordingly – One point was brought up just today by my customer that I had not even considered, but it can really make a difference, so I definitely want to note it here. During our weekly call, she asked about the timing of when I was testing out latest backup because of possible network latency. Their organization has nightly batch processing that can chew up network bandwidth. Even though the CRM system will be offline during the actual migration, other systems will not. If we decide to perform our migration when the network is slow, it can really eat into our time to complete the upgrade. So this may not be an issue given other networks, but it is definitely a great point to keep in mind when planning!
Up next
Once these migration steps are completed on your database and you have run through gauntlet of database conversions, we get to dive into the rest of the system. In the next post, we will start looking at the next steps in bringing the system up to speed on CRM 2016. This includes looking at the changes to the database and its impact on our solution. new CRM components such as an updated entity model, and then diving into the code changes for our solution.
