Installing Application X++ Updates Still Sucks (part two)


This post is to walk you through applying X++ application updates for Dynamics 365 for Finance and Operations. As of this writing, there is a Microsoft wiki page documenting how to do so, but it’s slightly out of date… and, more significantly, it doesn’t emphasize the preventative measures you can use to avoid hoarking your development environment (or worse your source control) if you need to yank the hotfix back out.

Before reading this, you should understand what application update hotfixes (versus binary update hotfixes) are, and the general documented steps for applying them. You should also be aware that things can go wrong, and sometimes you need to remove them from your development VM, or pull them back out of source control even after they’ve been deployed. Part one of this blog post series covers some of these subjects, but I assume basic competence with source control.

Step “Zero”: Have Microsoft application code in source control

There might be some disagreement with this, and welcome discussion in the comments. As time has gone on, I’ve become less convinced that there is a need for this. It definitely is not “best practices.”

But, as you are probably aware, every developer VM has a gigabyte or more (depending what you count) of code, in tens of thousands of files, in Microsoft-controlled models like Application Suite. Since you can’t directly change it, and you know it’s already on every shipped “onebox,” it probably seems like a waste of time and space to check all that in.

But, when you apply an application hotfix, you are changing some of that code. The changed files will be checked in; that’s how they get built and distributed. If you discover a need to roll back that hotfix, and you want it to roll back to an older version (instead of doing a deletion)… it’s way easier if the known good previous version was already there.

This is what the “prepare” step (which I’ll discuss below) is for, by the way. (Something that took me about a year to figure out. Yes, it’s documented. Yes, I miss things sometimes.) But what if you forget or have a problem with the “prepare” step?

So, if you are starting a brand new project (or have a time machine), you might consider doing this as your very first changeset: Add, at minimum, the contents of ApplicationSuite/Foundation to source control.

Possibly a better idea: at least make a copy of the “pristine” ApplicationSuite/Foundation somewhere. Or keep an “as shipped” onebox available. Basically, somewhere to get the original code when it is overwritten.

Step One: Download the hotfixes you need

I think Microsoft would love it if we all kept up on hotfixes, constantly applying them as soon as they came out. Given the effort involved, I’m quite impressed if anybody manages to do that. Seriously… if you do this, please comment below and talk to me about how many hours you spend each week on doing so.

Although most Microsoft documentation talks about the “tiles” in the environment pages of Lifecycle Services, in my experience, we have been using “Issue Search” (possibly with guidance from direct contacts at Microsoft) to identify hotfixes that address specific, pressing problems.

For what it’s worth, as of this writing, going through the “tiles” sometimes lets you see the dependent hotfixes for a given hotfix; it also lets you download multiple hotfixes in a single package. I don’t know that the latter is always desirable; I find hotfixes problematic enough that I prefer to be able to apply them one at a time, with hotfixes isolated to indivdual changesets, in order to narrow down problems. Using the “tiles” does, however, make sure the hotfixes you see are applicable to your application version; if you go through issue search, you need to check the noted “Release.”

Either way, you need to download and unzip these in your development VM. I like to keep these downloads in a specific directory, each in a subdirectory that has the hotfix KB number in its name. Like so:

hotfix directory structure

Step Two: Have a fallback plan for your development VM (“onebox”)

There is a small, but non-zero, chance that the hotfix will fail and be very difficult to scrape off your dev VM. So: after you download the hotfix locally, and “get” the latest dev branch changesets from source control, and you’re about ready to apply… stop.

If possible, checkpoint your VM. This is the fastest and easiest way to get back to a good state if things go south. Checkpoints are not meant as backups, and you can’t keep a long string of them without eating a ton of disk space, so, you’ve got to regularly delete your old checkpoints… but before applying a hotfix, you should have a current checkpoint you can go back to.

If a checkpoint is not possible, I offer two other suggestions: first, make a copy of PackagesLocalDirectory (that idea is inspired by this nice blog post). Second, have a second development VM at the same version on standby, from which you can copy out files.

Step Three: Prepare and check in

To be very clear about the point of the “prepare” step: this makes it easy to check in all the files that the hotfix will change, in their current (pre-hotfix) state, to make it easy to roll the hotfix back. If you didn’t go with “step zero” above (or if you did but didn’t include every single Microsoft model), this step is crucial if you need to roll back the hotfix later. I’ve needed to do this multiple times. Don’t blow off this step.

You have two choices for your tool here: command line or GUI.

Choice 1: Prepare with SCDPBundleInstall (command line)

I’ve had trouble with SCDPBundleInstall. Others think it is more reliable than the VS addin. If you choose this command line tool, you’ll use a command like this:

SCDPBundleInstall.exe -prepare -packagepath=C:\AXHotfixes\MicrosoftDynamicsAX_KB4058584\HotfixPackageBundle.axscdppkg -metadatastorepath= c:\AOSService\PackagesLocalDirectory -tfsworkspacepath= c:\AOSService\PackagesLocalDirectory -tfsprojecturi=

I couldn’t find thorough documentation for SCDPBundleInstall, but there’s a little in the official instructions for hotfixes.

Choice 2: Prepare in Visual Studio (GUI addin)

If nothing else, this tool is a nice way to see which hotfixes are already installed. If you go the GUI route, go to Dynamics 365 > Addins > Apply hotfix; browse to the hotfix; and click “Prepare.” Do not click “Apply” yet!

Prepare hotfix

For large hotfixes, this might take a little time; as of this writing, there are no progress bars or indicators, so you pretty much just have to wait until the GUI is responsive again.

ALWAYS: Check the “prepare” files in

Whether you used SCDPBundleInstall or the Visual Studio GUI addin, you need to check the prepared files in BEFORE you “apply” the hotfix. Otherwise, trying to roll the hotfix back will be a nightmare.

I won’t walk you through this. If you can’t manage a VSTS checkin with a sensible comment, you are not ready to manage hotfixes.

Step Four: Apply/Install

Once again, you choose between SCDPBundleInstall or the Visual Studio GUI addin.

Choice 1: Install with SCDPBundleInstall (command line)

If you choose this command line tool, you’ll use a command like this:

SCDPBundleInstall.exe -install -packagepath=C:\AXHotfixes\MicrosoftDynamicsAX_KB4058584\HotfixPackageBundle.axscdppkg -metadatastorepath= c:\AOSService\PackagesLocalDirectory -tfsworkspacepath= c:\AOSService\PackagesLocalDirectory -tfsprojecturi=

Choice 2: Prepare in Visual Studio (GUI addin)

If you used this tool for the “Prepare” step, this is pretty intuitive. Just click “Apply” instead of “Prepare” this time.

Apply hotfix


Unlike the “Prepare” step, you do not check files in immediately after you Apply/Install the hotfix.

However, you should look at your “Pending Changes” and make sure that any changed files it lists were checked in during the “Prepare” step. There might be a couple of files under “AxUpdate” that you can ignore (they are used by the system to track which hotfixes have been installed); but anything else should be an “Edit,” not an “Add.”

Step Five: Refresh models, resolve conflicts, and build

Occasionally Microsoft might include new models in a hotfix. It can’t hurt to start by going to Dynamics 365 > Model Management > Refresh Models before you continue.

If you have any customizations/overlays on the objects changed by the hotfix, there might be conflicts. Use Dynamics 365 > Addins > Create project from conflicts and check every VAR, ISV, etc. model where you’ve got custom code. (See my blog post on resolving conflicts for a few tips.) If you’re feeling like a cowboy, you might try just doing a build first, counting on that to raise an error if there are unresolved conflicts. (Later versions of D365 will eliminate customizations, but as of this writing, we’re not all there yet.)

Customizations/overlays or not, I have seen hotfixes break extensions. I have also seen them have dependencies on other hotfixes that are not included. You should ALWAYS do a build that includes every model the hotfix touched; and, probably, you should throw in all your in-house extension models as well. If you don’t know how to determine what models the hotfix touched, or if you are unsure and the hotfix contains more than one or two files, consider taking the time to do a full build of ALL models.

If this raises errors, hopefully you have the expertise to deal with them. But if you need to back out… this is why I recommended a fallback plan in step two. You can probably just undo the changes… probably. But it’ll feel good to have insurance that you can just restore a checkpoint.

Step Six: Check in, etc.

If/when you get a clean build on your dev VM, you are ready to check in. If you’re managing hotfixes, you should know how to do that. If you’re responsible for builds and/or deployments, know that those application hotfixes will be included just like any other custom code you write. It is, hopefully, smooth sailing at this point.

Step Seven: Oh Crap Something Went Wrong

Hopefully you rarely/never need this step. But, maybe you checked something in and it broke the build for some reason; or it has some awful side effect when deployed. Either way, you need to get it back out.

If you did step “zero” or step three correctly, and all the hotfix files have checked-in “known good” versions before the hotfix, you do a rollback like you would any other code. (Don’t know how to do a rollback? You probably shouldn’t be in charge of this stuff. You’ll need some remedial VSTS help for your job.)

Of course, if you applied multiple hotfixes simultaneously instead of doing them one at a time, I hope you’re rolling them all back simultaneously. Otherwise… honestly, you might want to roll them all back, and re-apply the ones you want to keep. Separately, one at a time, this time around.

If you did not do step “zero” or step three, you’ve probably ruined the rest of your day. Probably your evening, and tomorrow, too. But, I can share some hard-won tricks with you. There might be other ways, and I welcome feedback in the comments.

Step 7.1: Stop other devs from doing a “get”

Until you finish cleaning this up, warn other developers to avoid doing a “get” from source control, or they’ll have a ruined day too.

Step 7.2: If you don’t have a checkpoint, get a baseline

Rolling back this hotfix will probably hoark your dev VM.

If you don’t have a checkpoint, what you need to do is get your hands on a good baseline of the code before the hotfix. Your best bet for this is to set up a new clean “onebox” dev VM (if you don’t have one on standby) and “get” up to the changeset right before the hotfix.

Of course, if you’re doing all that work, you might want to just abandon your old VM and switch to a new one. Your call on what will ruin your day the least.

Step 7.3: Roll back the hotfix

Start by rolling back the changeset containing the bad hotfix(es). This will give you a “delete” for most of the files in it. If you’ve determined that the file did not exist before applying the hotfix, this is just what you want.

If a file DID exist before applying the hotfix, but it wasn’t checked in, the “delete” in this rollback will wipe it off of your dev VM. If any other developer did a “get” of the changeset with the hotfix, then later does a “get” of the deletion, I’m not sure what happens; but it might wipe the file off their dev VM too, ruining their day as well. That’s why we try to stop them from doing a “get” until we’ve cleaned up.

Step 7.4: Check in the rollback

Getting the rollback checked in means that developers can “get” again. As long as they “get” the rollback changeset in the same action as the hotfix(es) changeset, it doesn’t delete the system files off their VM.

Step 7.5: Apply your checkpoint OR replace “previously existing” files changed in the hotfix

Ideally you have a pre-hotfix checkpoint. Or can just switch to a different dev VM. Otherwise…

Using the baseline copy of the code you acquired in Step 7.2, copy any files that the hotfix changed into your dev VM. You won’t need to overwrite existing files, but, this will replace the ones you wiped off your system.

Whatever you did, you probably want to “get” the latest, maybe verify with a “compare,” and probably do a full build of all models to make sure your dev VM is in a working state.

And next time, I’m sure you’ll remember to take care during the “Prepare” step.

Here’s hoping you won’t need to revisit this blog post (or at least this step) again soon!

Installing Application X++ Updates Still Sucks (part one)


In the future, Microsoft is going to “seal” off all customizations, and soon after they’ll be automatically rolling out updates without us having to worry about it. But since that hypothetical rosy future could be years off for slow upgraders (and because I have suspicions about how rosy it will be), some of us are going to have to keep applying application hotfixes. And applying hotfixes sucks.

It’s not quite as bad as it used to be, but it’s still bad.

For sake of these blog posts, I’m going to assume you have a general idea what an application hotfix is for Dynamics 365 for Finance and Operations (what they’re naming the product this week) and that you know how it is different from a binary hotfix. In general:

  • Application hotfix: This is changes to X++ code in the Microsoft-controlled models (Application Foundation, Application Suite, etc.). When you look at or extend classes, tables, etc., in these models, this is the code that an application hotfix can change. When we get “application updates” (7.1/”1611″, 7.2/”July 2017″, 7.3/”December 2017″, 8.0/???), it includes all of these and more.
  • Binary hotfix: This is a change to binary code we can’t see, customize, or extend. They are always cumulative. When we get “platform updates,” it includes these and more. We won’t be talking about these right now.

I’ll also assume you know how you’re supposed to apply an application hotfix, and maybe you’ve even done so. (Note: as of this writing, the linked wiki documentation still uses the command line tool to prepare and apply the hotfix. There is now a tool in the Visual Studio GUI that makes it a little easier, under Dynamics 365 > Addins > Apply hotfix) The general steps are:

  1. Download the hotfix to a development VM (“onebox”) from Lifecycle Services, either through “tiles” or “issue search.”
  2. Prepare the hotfix in the development VM, either through SCDPBundleInstall or the GUI.
  3. Check in the “prepare” files. This is your insurance policy if things go bad.
  4. Apply the hotfix in the development VM, either through SCDPBundleInstall or the GUI.
  5. Resolve conflicts; and build/compile locally. If you have customized/extended the objects changed in the hotfix (or the hotfix has dependencies on other hotfixes), you can’t take this step for granted.
  6. Check the hotfix in to source control.
  7. Build & deploy to sandboxes, test, promote to MAIN/production… same as any other code.

It is easy and works fine… like, 95% of the time. But if you are here, something has probably gone wrong, or you are afraid something might (good instincts, you). You might be wondering if you did something wrong. Maybe you DID do something wrong. But maybe not.

After all, in one year of working with D365, I have seen application hotfixes that:

  • Were marked “binary” instead of “application update” and couldn’t be downloaded until Microsoft fixed that marking.
  • Broke customizations/overlays. (Expected.)
  • Broke extensions. (Surprise!)
  • Required followup actions in the GUI, which were not documented on the hotfix page (or possibly anywhere?), after deployment.
  • Had undocumented dependencies on other hotfixes.
  • Just plain didn’t do what they were supposed to do.

When a hotfix failed, I have lost hours, or even days (if it got into source control), fixing the damage. Arguably, I was dumb, and it didn’t need to be so bad. No reason YOU need to be dumb too, though.

In my next blog post, I’m going to talk about the steps you can use, proactively, to minimize the likelihood of problems when applying application hotfixes.

Installing Application X++ updates when SCDPBundleInstall.exe sucks

(UPDATE TO ORIGINAL POST: I’ve written more on this subject, and there is now a GUI tool in Visual Studio for applying hotfixes; you don’t need to use the SCDPBundleInstall.exe command line tool. Take a look at Installing Application X++ Updates Still Sucks (part one) and Installing Application X++ Updates Still Sucks (part two) instead of this, and only come back if you feel you still need to use SCDPBundleInstall.exe and you’re having problems. Installing Application X++ updates still sucks, but it does suck significantly less now.)

If you’re reading this, you have probably already tried to install some Application X++ updates you downloaded from Lifecycle Services, and failed. I’m assuming you’ve gone to this page:

…and tried to follow it, and run into one or both of these problems:

(1) As of this writing, the instructions on that page are out of date, and have one or two mistakes.

(2) SCDPBundleInstall kind of sucks. It doesn’t seem to always get dependencies right when the updates stack up, running them in the right order; and in some cases -prepare doesn’t create needed directories, although -install does.

For example, if you included hotfix 3208224, you probably got an error like this:

Could not find a part of the path 'C:\AOSService\PackagesLocalDirectory\AccountsPayableMobile\AccountsPayableMobile\AxLabelFile\LabelResources'.
 at Microsoft.Dynamics.AX.Servicing.SCDPBundleInstall.PrepareCommand.RunCommand()
 at Microsoft.Dynamics.AX.Servicing.SCDPBundleInstall.Program.Main(String[] args)

As such, you need to follow some steps that include a couple of corrections and workarounds. These will get you a good safe changeset to roll back to in source control if things go awry; it will also help make sure you have a deployable package that will install correctly through Lifecycle Services.

After much suffering on my end, I want to save you the trouble, and step you through what to do. This came from my own trial and error, and I haven’t seen these steps laid out anywhere else; if they have been, please let me know so I can link there and/or give credit. If there is a better way, or this gets fixed, please let me know and I’ll update this.

Part I: Prepare your dev/onebox

Step 1: Have the hotfixes

I assume that you have downloaded the package of hotfixes, which is fairly well documented elsewhere (and pretty intuitive anyway). You should have a file with a .axscdppkg extension on your dev/onebox, with a name/folder location that is clear and recognizable. (Do yourself a favor: have a folder where you put all the hotfixes, and create subfolders that include the VM name, date, number of hotfixes, and the text “X++ Hotfixes,” or something similar.)

Step 2: Your dev/onebox needs to be healthy

By the time you’ve done your web searches and gotten here, it might be too late, and your dev/onebox might be messed up. I can’t help you fix it here. Maybe you’re lucky and have a good checkpoint to go back to, and a good point in version control to pull from. I have had to reinstall my dev/onebox, and it stinks; if you do too, then you have my sympathy.

Temporarily, you should probably not have any mapping to your main/production/build branch in your dev/onebox. If you do, strongly consider removing that mapping for now. You’ll probably need to put it back if you are doing branch merges from your box. But, having mapping to multiple branches will probably confuse SCDPBundleInstall; running it will at least take longer, as it creates the same changes in multiple branches.

Step 3: Get version control to a healthy state

If you’ve hoarked your version control, you need to get back to a healthy state, and I can’t help with that. You need to work with someone who understands version control.

This will be easiest/simplest if your production/build branch is up to date and in sync with your dev/working branch. Consider doing a compare before proceeding.

Locally, on your dev/onebox, this will be easiest if you start from a point where you have nothing in Pending Changes > Included Changes. (If you don’t know how to find that easily yet– in Visual Studio, under Team Explorer– you are swimming in the deep end now!)

Step 4: Checkpoint your VM

If you screwed up your dev/onebox before finding this page, you already appreciate the value of frequent checkpoints. If not… be thankful and do a checkpoint now anyway.

If you don’t know how to do a checkpoint, you should probably spend a little time poking around whatever tool you use to manage virtual machines. In Hyper-V Manager on Windows, right-click the VM and choose “Checkpoint.”

Part II: Create the directory structure required to make SCDPBundleInstall -prepare work properly

Step 5: Do SCDPBundleInstall -install

If you are reading these instructions, it’s probably because -prepare failed for you. As of this writing, -prepare does not always create necessary folders, but -install does. That’s why you’re doing all these convoluted steps. As much time as it wastes, we are going to do the -install first solely to get the directory structure created.

This step will take a while, especially if you have  multiple branches mapped. Note my earlier advice that you (at least temporarily) remove mappings for anything but your dev/working branch.

Note also that, as of this writing, the syntax of the -tfsprojecturi switch is not documented correctly, at least not on the wiki page noted above. You just use a URL, not a project.  Here is some sample syntax (anybody with sufficient WordPress expertise to help me format this better, please comment!):

C:\AOSService\PackagesLocalDirectory\Bin>SCDPBundleInstall -install
-packagepath="C:\AXHotfixes\X++ hotfixes 2017-02-22 (142)\HotfixPackageBundle.axscdppkg" 

If you get an error, try re-running the command, and it might work! This does not seem to do things in the right order, or consistently, and sometimes it runs fine after a retry or two. The important thing is that it eventually completes, creating the directory structure we need.

If you absolutely can’t get this to finish, go back to your VM checkpoint and quit following these instructions. Sorry, but you’re beyond the help I can give here.

Step 6: Undo the file changes via source control

This is where we get tricky. In Visual Studio, in Team Explorer, look under Pending Changes and undo all the Included Changes. Not exclude; undo. We are rolling back the file changes that the -prepare just made. (But give them a once over first to make sure you’ve excluded any active development you still have going. Ideally you don’t have anything outstanding, but we don’t live in a perfect world, or you wouldn’t be here.)

After completing this step, you should be at a point where the directory structure you need for the hotfixes has been created; but none of the files have been changed.

Step 7: Copy the directory structure

We want to take a copy of just the directory structure (the files inside would be overkill), which we can effectively do with this command:

C:\AOSService\PackagesLocalDirectory>xcopy * "C:\PLDstructure" /T /E /X /I /H

If run this in your packages folder as shown, this is going to copy out JUST the folder structure, with no files, to C:\PLDstructure. Adjust the command as necessary if you are using another directory.

(It might be possible to speed this up by also using the /D:m-d-y switch, but this should not take too long anyway.)

After doing this, copy the C:\PLDstructure folder (or whatever you named it) out of your VM and into your host operating system. We need it backed up outside the VM, because we are just about to…

Step 8: Apply the checkpoint you took in Step #4

Something might have gotten screwed up on your dev/onebox by doing this. SCDPBundleInstall can’t be trusted. Let’s be safe and apply our checkpoint, restoring our box to the healthy state before we ran it.

Step 9: Restore the needed folder structure

Copy the C:\PLDstructure (or whatever you named  it) back into your VM.

Now create whatever empty folders you need in your packages directory by doing this (adjust the locations as necessary):

xcopy "C:\PLDstructure" "C:\AOSService\PackagesLocalDirectory" /T /E /X /I /H

Part III: Back on track!

We are now at a place where we can mostly return to following the wiki.


Step 10: Do SCDPBundleInstall -prepare

This may take a couple of minutes, but nearly as long as the -install. Remember the different syntax for -tfsprojecturi. You are pretty much using the same command that you used in Step #5, other than replacing -install with -prepare.

Although the -prepare probably failed before, it hopefully works now that the directory structure it needs exists. If not… you might be beyond the help I can give.

Step 11: Check in the touched files

All the files you see in Pending Changes > Included Changes should be files that currently exist but will be changed by running the actual install. They should not have been changed, just “touched” so that they are picked up as pending changes for version control. (There are less of them than what the -install will create since you are just getting the pre-existing files. -install will create all new files, but we don’t need them in the checkpoint we are creating, since rolling back to it in an emergency should delete files that did not exist.)

You now want to check all these in, with a clear note that this changeset is your pre-hotfix rollback point. If things go south, we can theoretically roll back to this point to get our VM back to a good state.

Make sure to merge your dev/working branch into your main/production/build branch so you have somewhere to fall back there, too. Watch the changesets; I got some false matches of older ones.

You might want to run a build in Lifecycle Services at this point, just to make sure nothing unexpected broke. Theoretically, there is nothing new being introduced, and you should have no problems. But that requires more trust of SCDPBundleInstall than I have.

Step 11: Do SCDPBundleInstall -install (again)

Once again, you might need to run it multiple times, if dependencies and/or run orders aren’t right.

This will take a while, especially if you have  multiple branches mapped. Note my earlier advice that you (at least temporarily) remove mappings for anything but your dev/working branch.

Step 12: Review “Pending Changes”

Take a look at your Pending Changes. If you have multiple branches mapped, you might have redundancies you need to clean up by excluding/undoing them. Ideally, you see a nice clean set of hotfix changes, all contained in your packages directory (i.e. C:\AOSService\PackagesLocalDirectory).

Step 13: Resolve Conflicts

This is covered pretty well in the original wiki:

Step 14: Do a local build

Before doing this, refresh the list of models, so any new models are picked up by your build! Go to Dynamics 365 > Model Management > Refresh Models.

Do a local build. I usually just include all models at this point; be sure to “Synchronize Database” under Options, too! Because the build’s analysis of dependencies isn’t perfect, you might get build errors that resolve themselves if you do the build one or two more times. I ended up running the local build four or five times; as long as you are getting different errors each time, it means you’re making progress. (If you have a deeper understanding of this than me, please let me know, and I’ll add detail.) Eventually, you should hit a point where it builds without error, and all you get are the 1000+ warnings you are used to ignoring with every build.

If you keep getting the exact same errors over and over, you are outside the scope of help I can offer in this blog post.

Step 15: Check in, Synchronize, Deploy

The original wiki picks up fine at this point:

It may go without saying, but you should be sure to use check-in comments and deployable package names that are clear about their contents. I try to include at least the date, number of hotfixes, and the text “X++ Hotfixes,” or something similar. If you have a small enough set of hotfixes, actually including the hotfix number(s) makes sense!