PART TWO: IT’S NOT PARANOIA IF THEY’RE REALLY OUT TO GET YOU (OR, STEP-BY-STEP INSTRUCTIONS INCLUDING DISASTER MITIGATION)
This post is to walk you through applying X++ application updates for Dynamics 365 for Finance and Operations. As of this writing, there is a Microsoft wiki page documenting how to do so, but it’s slightly out of date… and, more significantly, it does not cover the preventative measures I recommend to avoid hoarking your development environment (or worse your source control) if you need to yank the hotfix back out.
Before reading this, you should understand what application update hotfixes (versus binary update hotfixes) are, and the general documented steps for applying them. You should also be aware that things can go wrong, and sometimes you need to remove them from your development VM, or pull them back out of source control even after they’ve been deployed. Part one of this blog post series covers some of these subjects, but I assume basic competence with source control.
Step “Zero”: Have Microsoft application code in source control
There might be some disagreement with this, and welcome discussion in the comments. But, as you are probably aware, every developer VM has a gigabyte or more (depending what you count) of code, in tens of thousands of files, in Microsoft-controlled models like Application Suite. Since you can’t directly change it, and you know it’s already on every shipped “onebox,” it probably seems like a waste of time and space to check all that in.
But, when you apply an application hotfix, you are changing some of that code. The changed files will be checked in; that’s how they get built and distributed. If you discover a need to roll back that hotfix, and you want it to roll back to an older version (instead of doing a deletion)… it’s way easier if the known good previous version was already there.
This is what the “prepare” step is for, by the way. (Something that took me about a year to figure out. Yes, it’s documented. Yes, I miss things sometimes.) But what if you forget the “prepare” step?
So, if you are starting a brand new project (or have a time machine), I recommend doing this as your very first changeset: Add, at minimum, the contents of ApplicationSuite/Foundation to source control.
Step One: Download the hotfixes you need
I think Microsoft would love it if we all kept up on hotfixes, constantly applying them as soon as they came out. Given the effort involved, I’m quite impressed if anybody manages to do that. Seriously… if you do this, please comment below and talk to me about how many hours you spend each week on doing so.
Although most Microsoft documentation talks about the “tiles” in the environment pages of Lifecycle Services, in my experience, we have been using “Issue Search” (possibly with guidance from direct contacts at Microsoft) to identify hotfixes that address specific, pressing problems.
For what it’s worth, as of this writing, going through the “tiles” sometimes lets you see the dependent hotfixes for a given hotfix; it also lets you download multiple hotfixes in a single package. I don’t know that the latter is always desirable; I find hotfixes problematic enough that I prefer to be able to apply them one at a time, with hotfixes isolated to indivdual changesets, in order to narrow down problems. It does, however, make sure the hotfixes you see are applicable to your application version; if you go through issue search, you need to check the noted “Release.”
Either way, you need to download and unzip these in your development VM. I like to keep these downloads in a specific directory, each in a subdirectory that has the hotfix KB number in its name.
Step Two: Have a fallback plan for your development VM (“onebox”)
There is a small, but non-zero, chance that the hotfix will fail and be very difficult to scrape off your dev VM. So: after you download the hotfix locally, and “get” the latest dev branch changesets from source control, and you’re about ready to apply… stop.
If possible, checkpoint your VM. This is the fastest and easiest way to get back to a good state if things go south. Checkpoints are not meant as backups, and you can’t keep a long string of them without eating a ton of disk space, so, you’ve got to regularly delete your old checkpoints… but before applying a hotfix, you should have a current checkpoint you can go back to.
If a checkpoint is not possible, I offer two other suggestions: first, make a copy of PackagesLocalDirectory (that idea is inspired by this nice blog post). Second, have a second development VM at the same version on standby, from which you can copy out files.
Step Three: Prepare and check in
To be very clear about the point of the “prepare” step: this makes it easy to check in all the files that the hotfix will change, in their current state, to make it easy to roll the hotfix back. If you didn’t go with “step zero” above (or if you did but didn’t include every single Microsoft model), this step is crucial if you need to roll back the hotfix later. I’ve needed to do this multiple times. Don’t blow off this step.
You have two choices for your tool here: command line or GUI.
Choice 1: Prepare with SCDPBundleInstall (command line)
I’ve had trouble with SCDPBundleInstall. Others think it is more reliable than the VS addin. If you choose this command line tool, you’ll use a command like this:
SCDPBundleInstall.exe -prepare -packagepath=C:\AXHotfixes\MicrosoftDynamicsAX_KB4058584\HotfixPackageBundle.axscdppkg -metadatastorepath= c:\AOSService\PackagesLocalDirectory -tfsworkspacepath= c:\AOSService\PackagesLocalDirectory -tfsprojecturi=https://myaccount.visualstudio.com/defaultcollection
I couldn’t find thorough documentation for SCDPBundleInstall, but there’s a little in the official instructions for hotfixes.
Choice 2: Prepare in Visual Studio (GUI addin)
If nothing else, this tool is a nice way to see which hotfixes are already installed. If you go the GUI route, go to Dynamics 365 > Addins > Apply hotfix; browse to the hotfix; and click “Prepare.” Do not click “Apply” yet!
For large hotfixes, this might take a little time; as of this writing, there are no progress bars or indicators, so you pretty much just have to wait until the GUI is responsive again.
ALWAYS: Check the “prepare” files in
Whether you used SCDPBundleInstall or the Visual Studio GUI addin, you need to check the prepared files in BEFORE you apply the hotfix. Otherwise, trying to roll the hotfix back will be a nightmare.
I won’t walk you through this. If you can’t manage a VSTS checkin with a sensible comment, you are not ready to manage hotfixes.
Step Four: Apply/Install
Once again, you choose between SCDPBundleInstall or the Visual Studio GUI addin.
Choice 1: Install with SCDPBundleInstall (command line)
If you choose this command line tool, you’ll use a command like this:
SCDPBundleInstall.exe -install -packagepath=C:\AXHotfixes\MicrosoftDynamicsAX_KB4058584\HotfixPackageBundle.axscdppkg -metadatastorepath= c:\AOSService\PackagesLocalDirectory -tfsworkspacepath= c:\AOSService\PackagesLocalDirectory -tfsprojecturi=https://myaccount.visualstudio.com/defaultcollection
Choice 2: Prepare in Visual Studio (GUI addin)
If you used this tool for the “Prepare” step, this is pretty intuitive. Just click “Apply” instead of “Prepare” this time.
NEVER: DON’T CHECK IN YET!
Unlike the “Prepare” step, you do not check files in immediately after you Apply/Install the hotfix.
However, you should look at your “Pending Changes” and make sure that any changed files it lists were checked in during the “Prepare” step. Any added files obviously will not have been there.
Step Five: Resolve conflicts and build
If you have any customizations/overlays on the objects changed by the hotfix, there might be conflicts. Use Dynamics > Addins > Create project from conflicts and check every VAR, ISV, etc. model where you’ve got custom code. If you’re feeling like a cowboy, you might try just doing build first, counting on that to raise an error if there are unresolved conflicts.
Customizations/overlays or not, I have seen hotfixes break extensions. I have also seen them have dependencies on other hotfixes that are not included. You should ALWAYS do a build that includes every model the hotfix touched; and, probably, you should throw in all your in-house extension models as well. If you don’t know how to determine what models the hotfix touched, or if you are unsure and the hotfix contains more than one or two files, consider taking the time to do a full build of ALL models.
If this raises errors, hopefully you have the expertise to deal with them. But if you need to back out… this is why I recommended a fallback plan in step two. You can probably just undo the changes… probably. But it’ll feel good to have insurance that you can just restore a checkpoint.
Step Six: Check in, etc.
If/when you get a clean build on your dev VM, you are ready to check in. If you’re managing hotfixes, you should know how to do that. If you’re responsible for builds and/or deployments, know that those application hotfixes will be included just like any other custom code you write. It is, hopefully smooth sailing at this point.
Step Seven: Oh Crap Something Went Wrong
Hopefully you rarely/never need this step. But, maybe you checked something in and it broke the build for some reason; or it has some awful side effect when deployed. Either way, you need to get it back out.
If you did step “zero” or step three correctly, and all the hotfix files have checked-in “known good” versions before the hotfix, you do a rollback like you would any other code. (Don’t know how to do a rollback? You probably shouldn’t be in charge of this stuff. You’ll need some remedial VSTS help for your job.)
Of course, if you applied multiple hotfixes simultaneously instead of doing them one at a time, I hope you’re rolling them all back simultaneously. Otherwise… honestly, you might want to roll them all back, and re-apply the ones you want to keep. Separately, one at a time, this time around.
If you did not do step “zero” or step three, you’ve probably ruined the rest of your day. Probably your evening, and tomorrow, too. But, I can share some hard-won tricks with you. There might be other ways, and I welcome feedback in the comments.
NOTE: WHAT FOLLOWS IS NOT NECESSARILY THE BEST ADVICE. THIS WILL BE UPDATED SOON WITH NEW LEARNING.
Step 7.1: Get a baseline
What you need to do is get your hands on a good baseline of the code before the hotfix. Your best bet for this is to set up a new clean “onebox” dev VM (if you don’t have one on standby) and “get” up to the changeset right before the hotfix. Then, unfortunately, you have the tedious work of looking over the changest containing the hotfix; individually reviewing each file; and determining whether it was a newly-added file, or a change to an existing file.
Step 7.2: Delete “new” files added in the hotfix
Start by rolling back the changeset. This will give you a “delete” for most of the files in it. If you’ve determined that the file did not exist before applying the hotfix, this is just what you want. If a file DID exist before applying the hotfix, undo that deletion in your “Pending Changes,” and go to step 7.3.
Step 7.3: Replace “previously existing” files changed in the hotfix
Using the baseline copy of the code, copy any files that the hotfix changed into your dev VM, replacing the changed version. When you do this, the file you copy over will have an older timestamp. This will make it difficult to check it in over the version with a newer timestamp. To trick VSTS, you can change the timestamp on the files you copy over. There are many ways to do this; I have a saved Powershell script. You might prefer to use some other utility.
Step 7.4: Check in
You should now have in your “Pending changes” a list of files identical to the changeset containing the bad hotfix. Considering all the work you’ve done, you might want to re-check it. You definitely want to make sure you can get a full local build of all models. After you check it in, follow up with a careful “compare” and “get” of the latest, and make sure you can build locally as well run the shared build without error.
And next time, I’m sure you’ll remember to take care during the “Prepare” step. Here’s hoping you won’t need to revisit this blog post (or at least this step) again soon!