Installing Application X++ Updates Still Sucks (part two)


This post is to walk you through applying X++ application updates for Dynamics 365 for Finance and Operations. As of this writing, there is a Microsoft wiki page documenting how to do so, but it’s slightly out of date… and, more significantly, it does not cover the preventative measures I recommend to avoid hoarking your development environment (or worse your source control) if you need to yank the hotfix back out.

Before reading this, you should understand what application update hotfixes (versus binary update hotfixes) are, and the general documented steps for applying them. You should also be aware that things can go wrong, and sometimes you need to remove them from your development VM, or pull them back out of source control even after they’ve been deployed. Part one of this blog post series covers some of these subjects, but I assume basic competence with source control.

Step “Zero”: Have Microsoft application code in source control

There might be some disagreement with this, and welcome discussion in the comments. But, as you are probably aware, every developer VM has a gigabyte or more (depending what you count) of code, in tens of thousands of files, in Microsoft-controlled models like Application Suite. Since you can’t directly change it, and you know it’s already on every shipped “onebox,” it probably seems like a waste of time and space to check all that in.

But, when you apply an application hotfix, you are changing some of that code. The changed files will be checked in; that’s how they get built and distributed. If you discover a need to roll back that hotfix, and you want it to roll back to an older version (instead of doing a deletion)… it’s way easier if the known good previous version was already there.

This is what the “prepare” step is for, by the way. (Something that took me about a year to figure out. Yes, it’s documented. Yes, I miss things  sometimes.) But what if you forget the “prepare” step?

So, if you are starting a brand new project (or have a time machine), I recommend doing this as your very first changeset: Add, at minimum, the contents of ApplicationSuite/Foundation to source control.

Step One: Download the hotfixes you need

I think Microsoft would love it if we all kept up on hotfixes, constantly applying them as soon as they came out. Given the effort involved, I’m quite impressed if anybody manages to do that. Seriously… if you do this, please comment below and talk to me about how many hours you spend each week on doing so.

Although most Microsoft documentation talks about the “tiles” in the environment pages of Lifecycle Services, in my experience, we have been using “Issue Search” (possibly with guidance from direct contacts at Microsoft) to identify hotfixes that address specific, pressing problems.

For what it’s worth, as of this writing, going through the “tiles” sometimes lets you see the dependent hotfixes for a given hotfix; it also lets you download multiple hotfixes in a single package. I don’t know that the latter is always desirable; I find hotfixes problematic enough that I prefer to be able to apply them one at a time, with hotfixes isolated to indivdual changesets, in order to narrow down problems. It does, however, make sure the hotfixes you see are applicable to your application version; if you go through issue search, you need to check the noted “Release.”

Either way, you need to download and unzip these in your development VM. I like to keep these downloads in a specific directory, each in a subdirectory that has the hotfix KB number in its name.

hotfix directory structure

Step Two: Have a fallback plan for your development VM (“onebox”)

There is a small, but non-zero, chance that the hotfix will fail and be very difficult to scrape off your dev VM. So: after you download the hotfix locally, and “get” the latest dev branch changesets from source control, and you’re about ready to apply… stop.

If possible, checkpoint your VM. This is the fastest and easiest way to get back to a good state if things go south. Checkpoints are not meant as backups, and you can’t keep a long string of them without eating a ton of disk space, so, you’ve got to regularly delete your old checkpoints… but before applying a hotfix, you should have a current checkpoint you can go back to.

If a checkpoint is not possible, I offer two other suggestions: first, make a copy of PackagesLocalDirectory (that idea is inspired by this nice blog post). Second, have a second development VM at the same version on standby, from which you can copy out files.

Step Three: Prepare and check in

To be very clear about the point of the “prepare” step: this makes it easy to check in all the files that the hotfix will change, in their current state, to make it easy to roll the hotfix back. If you didn’t go with “step zero” above (or if you did but didn’t include every single Microsoft model), this step is crucial if you need to roll back the hotfix later. I’ve needed to do this multiple times. Don’t blow off this step.

You have two choices for your tool here: command line or GUI.

Choice 1: Prepare with SCDPBundleInstall (command line)

I’ve had trouble with SCDPBundleInstall. Others think it is more reliable than the VS addin. If you choose this command line tool, you’ll use a command like this:

SCDPBundleInstall.exe -prepare -packagepath=C:\AXHotfixes\MicrosoftDynamicsAX_KB4058584\HotfixPackageBundle.axscdppkg -metadatastorepath= c:\AOSService\PackagesLocalDirectory -tfsworkspacepath= c:\AOSService\PackagesLocalDirectory -tfsprojecturi=

I couldn’t find thorough documentation for SCDPBundleInstall, but there’s a little in the official instructions for hotfixes.

Choice 2: Prepare in Visual Studio (GUI addin)

If nothing else, this tool is a nice way to see which hotfixes are already installed. If you go the GUI route, go to Dynamics 365 > Addins > Apply hotfix; browse to the hotfix; and click “Prepare.” Do not click “Apply” yet!

Prepare hotfix

For large hotfixes, this might take a little time; as of this writing, there are no progress bars or indicators, so you pretty much just have to wait until the GUI is responsive again.

ALWAYS: Check the “prepare” files in

Whether you used SCDPBundleInstall or the Visual Studio GUI addin, you need to check the prepared files in BEFORE you apply the hotfix. Otherwise, trying to roll the hotfix back will be a nightmare.

I won’t walk you through this. If you can’t manage a VSTS checkin with a sensible comment, you are not ready to manage hotfixes.

Step Four: Apply/Install

Once again, you choose between SCDPBundleInstall or the Visual Studio GUI addin.

Choice 1: Install with SCDPBundleInstall (command line)

If you choose this command line tool, you’ll use a command like this:

SCDPBundleInstall.exe -install -packagepath=C:\AXHotfixes\MicrosoftDynamicsAX_KB4058584\HotfixPackageBundle.axscdppkg -metadatastorepath= c:\AOSService\PackagesLocalDirectory -tfsworkspacepath= c:\AOSService\PackagesLocalDirectory -tfsprojecturi=

Choice 2: Prepare in Visual Studio (GUI addin)

If you used this tool for the “Prepare” step, this is pretty intuitive. Just click “Apply” instead of “Prepare” this time.

Apply hotfix


Unlike the “Prepare” step, you do not check files in immediately after you Apply/Install the hotfix.

However, you should look at your “Pending Changes” and make sure that any changed files it lists were checked in during the “Prepare” step. Any added files obviously will not have been there.

Step Five: Resolve conflicts and build

If you have any customizations/overlays on the objects changed by the hotfix, there might be conflicts. Use Dynamics > Addins > Create project from conflicts and check every VAR, ISV, etc. model where you’ve got custom code. If you’re feeling like a cowboy, you might try just doing build first, counting on that to raise an error if there are unresolved conflicts.

Customizations/overlays or not, I have seen hotfixes break extensions. I have also seen them have dependencies on other hotfixes that are not included. You should ALWAYS do a build that includes every model the hotfix touched; and, probably, you should throw in all your in-house extension models as well. If you don’t know how to determine what models the hotfix touched, or if you are unsure and the hotfix contains more than one or two files, consider taking the time to do a full build of ALL models.

If this raises errors, hopefully you have the expertise to deal with them. But if you need to back out… this is why I recommended a fallback plan in step two. You can probably just undo the changes… probably. But it’ll feel good to have insurance that you can just restore a checkpoint.

Step Six: Check in, etc.

If/when you get a clean build on your dev VM, you are ready to check in. If you’re managing hotfixes, you should know how to do that. If you’re responsible for builds and/or deployments, know that those application hotfixes will be included just like any other custom code you write. It is, hopefully smooth sailing at this point.

Step Seven: Oh Crap Something Went Wrong

Hopefully you rarely/never need this step. But, maybe you checked something in and it broke the build for some reason; or it has some awful side effect when deployed. Either way, you need to get it back out.

If you did step “zero” or step three correctly, and all the hotfix files have checked-in “known good” versions before the hotfix, you do a rollback like you would any other code. (Don’t know how to do a rollback? You probably shouldn’t be in charge of this stuff. You’ll need some remedial VSTS help for your job.)

Of course, if you applied multiple hotfixes simultaneously instead of doing them one at a time, I hope you’re rolling them all back simultaneously. Otherwise… honestly, you might want to roll them all back, and re-apply the ones you want to keep. Separately, one at a time, this time around.

If you did not do step “zero” or step three, you’ve probably ruined the rest of your day. Probably your evening, and tomorrow, too. But, I can share some hard-won tricks with you. There might be other ways, and I welcome feedback in the comments.


Step 7.1: Get a baseline

What you need to do is get your hands on a good baseline of the code before the hotfix. Your best bet for this is to set up a new clean “onebox” dev VM (if you don’t have one on standby) and “get” up to the changeset right before the hotfix. Then, unfortunately, you have the tedious work of looking over the changest containing the hotfix; individually reviewing each file; and determining whether it was a newly-added file, or a change to an existing file.

Step 7.2: Delete “new” files added in the hotfix

Start by rolling back the changeset. This will give you a “delete” for most of the files in it. If you’ve determined that the file did not exist before applying the hotfix, this is just what you want. If a file DID exist before applying the hotfix, undo that deletion in your “Pending Changes,” and go to step 7.3.

Step 7.3: Replace “previously existing” files changed in the hotfix

Using the baseline copy of the code, copy any files that the hotfix changed into your dev VM, replacing the changed version. When you do this, the file you copy over will have an older timestamp. This will make it difficult to check it in over the version with a newer timestamp. To trick VSTS, you can change the timestamp on the files you copy over. There are many ways to do this; I have a saved Powershell script. You might prefer to use some other utility.

Step 7.4: Check in

You should now have in your “Pending changes” a list of files identical to the changeset containing the bad hotfix. Considering all the work you’ve done, you might want to re-check it. You definitely want to make sure you can get a full local build of all models. After you check it in, follow up with a careful “compare” and “get” of the latest, and make sure you can build locally as well run the shared build without error.

And next time, I’m sure you’ll remember to take care during the “Prepare” step. Here’s hoping you won’t need to revisit this blog post (or at least this step) again soon!


Installing Application X++ Updates Still Sucks (part one)


In the future, Microsoft is going to “seal” off all customizations, and soon after they’ll be automatically rolling out updates without us having to worry about it. But since that hypothetical rosy future could be years off for slow upgraders (and because I have suspicions about how rosy it will be), some of us are going to have to keep applying application hotfixes. And applying hotfixes sucks.

It’s not quite as bad as it used to be, but it’s still bad.

For sake of these blog posts, I’m going to assume you have a general idea what an application hotfix is for Dynamics 365 for Finance and Operations (what they’re naming the product this week) and that you know how it is different from a binary hotfix. In general:

  • Application hotfix: This is changes to X++ code in the Microsoft-controlled models (Application Foundation, Application Suite, etc.). When you look at or extend classes, tables, etc., in these models, this is the code that an application hotfix can change. When we get “application updates” (7.1/”1611″, 7.2/”July 2017″, 7.3/”December 2017″), it includes all of these and more.
  • Binary hotfix: This is a change to binary code we can’t see, customize, or extend. They are always cumulative. When we get “platform updates,” it includes these and more. We won’t be talking about these right now.

I’ll also assume you know how you’re supposed to apply an application hotfix, and maybe you’ve even done so. (Note: as of this writing, the linked wiki documentation still uses the command line tool to prepare and apply the hotfix. There is now a tool in the Visual Studio GUI that makes it a little easier, under Dynamics 365 > Addins > Apply hotfix) The general steps are:

  1. Download the hotfix to a development VM (“onebox”) from Lifecycle Services, either through “tiles” or “issue search.”
  2. Prepare & apply the hotfix in the development VM, either through SCDPBundleInstall or the GUI.
  3. Build/compile locally. Because you might have customized/extended the objects changed in the hotfix (or it might have dependencies on other hotfixes), this might not be successful, and you’ve got work on your hands.
  4. Check the hotfix in to source control.
  5. Build & deploy to sandboxes, test, promote to MAIN/production… same as any other code.

It is easy and works fine… like, 95% of the time. But if you are here, something has probably gone wrong, or you are afraid something might (good instincts, you). You might be wondering if you did something wrong. Maybe you DID do something wrong. But maybe not.

After all, in one year of working with D365, I have seen application hotfixes that:

  • Were marked “binary” instead of “application update” and couldn’t be downloaded until Microsoft fixed that marking.
  • Broke customizations/overlays. (Expected.)
  • Broke extensions. (Surprise!)
  • Required followup actions in the GUI, which were not documented on the hotfix page (or possibly anywhere?), after deployment.
  • Had undocumented dependencies on other hotfixes.
  • Just plain didn’t do what they were supposed to do.

When a hotfix failed, I have lost hours, or even days (if it got into source control), fixing the damage. Arguably, I was dumb, and it didn’t need to be so bad. No reason YOU need to be dumb too, though.

In my next blog post, I’m going to talk about the steps you can use, proactively, to minimize the likelihood of problems when applying application hotfixes.

Configuring Remote Desktop (RDP) to a Dynamics 365 for Operations sandbox in Lifecycle Services

Sometimes you need to use Remote Desktop (RDP) to get into your cloud-hosted sandboxes for Dynamics 365 for Operations. Whether it’s to poke into a SQL database, clean off a full drive on a build machine, or… well, I can’t think of a lot of other reasons, but those are good ones.

As you should already know, to do this, you want to go to the Lifecycle Services (LCS) page for your VM and use the links (and usernames and passwords) there to RDP and log in. Those basics are out of scope for this blog post… but they should be pretty intuitive anyway.

But: For security reasons, you are now required to whitelist the IP addresses that are allowed to connect. If you don’t do that, you’ll get an error that looks like “Remote Desktop can’t connect to the remote computer…”:

RDP error

Adding your IP address seems pretty easy at first. So, you happily pull up a Command Prompt, run IPCONFIG, and on the environments LCS page go to Maintain > Enable access:

LCS whitelist

LCS whitelist 2

“Easy!,” you think.

Then it still doesn’t work.

Here’s the deal: Most internet service providers (whether they be your home provider or your work/office) use different internal IP addresses than the ones they use out on the internet. To get the actual IP address that Microsoft’s filters think you are using, you need to ask the internet outside of where you are. Go to a site like , or just search Google for “what is my ip address” to get it:

what is my ip address

Plug in this IP address instead, and you’ll be good to go.

Just hope Microsoft doesn’t accidentally erase them. If you are managing this list for a bunch of other people, maybe keep a copy somewhere.

Checking in references to other models

As you might be aware, sometimes you need to add a reference to other models in your own models. You do this by going to Dynamics 365 > Model Management > Update model parameters, selecting your model, and checking the box for the model you want to reference.

Addinf references to a model

For example, here you see references added for FiscalBooks, GeneralLedger, Ledger, and Measurement. (A full discussion of references is too much for this blog post, though!)

I discovered, to my surprise, that even experienced developers sometimes don’t know how to check in these changes. If code that requires the reference is checked in, but the new reference is not, the build breaks with potentially mysterious and hard-to-diagnose errors!

How do you check in changes to references? You need to check in the DESCRIPTOR file for the model. It will be an XML file named after the model, and it will be in \AOSService\PackagesLocalDirectory\<ModelName>\Descriptor; it might excluded by default, and you need to make sure to include it in your changes. For example, if you add a reference to a model named “Czar Extensions Model,” you want to see something like this in your pending changes for the check-in:

Pending changes with Descriptor file

I hope this helps. May all your builds be “green”!


Upgrade development VM from Windows evaluation version to “full” version

If you’re anything like me, you’ve got a lot of ISVs to shuffle, making it very difficult to keep up with the current rapid release cadence of Dynamics 365 for Operations. Maybe things will get easier as the cadence slows down, or when they lock out customizations/overlayering. But for now– and, I expect, probably for another couple of years– I’m not easily able to change platform updates every month or two.

(If you’re curious: I’m still on application update 1611 and platform update 7, much to my chagrin; as of this writing there was a big July 2017 application update, and platform update 11 just dropped a couple of days ago.)

Point is, the development VM comes on an evaluation copy of Windows Server 2016 Datacenter, and after a while it starts nagging you– with increasing frequency– that the evaluation is about to end. To work uninterrupted (or work at all beyond a point), you need to switch to a full version, and it isn’t really obvious how.

Step one: you need a valid key for Windows Server 2016 Datacenter. You didn’t think that was a magic freebie, did you? Fortunately, you’ve got an MSDN subscription (or you sure as heck SHOULD if you’re developing for Dynamics 365 for Operations!) which gives you five keys. In MSDN, you can go to your account, choose “Use your benefits,” and then click “Product keys.” As of this writing, you can also try going straight to this URL: Search for Windows Server 2016 Datacenter. There, you can claim a key. (I’m not sure yet if there is a way to deactivate or return the key when you finally move to a new version and can get rid of the old VM… please comment below if you do!)

Step two: Pretty much follow these instructions: (Give this kind fellow the traffic and leave them a nice comment!) Although their writeup is nicer and easier to follow, for sake of posterity, you need to open an Administrator command prompt and run:

DISM /online /Set-Edition:ServerDatacenter /ProductKey:XXXXX-XXXXX-XXXXX-XXXXX-XXXXX /AcceptEula

…filling in the product key. The Edition value of “ServerDatacenter” might change in the future, so check the link for full instructions on what to do.

Step three? Wait a while. The process is slow, and will be prompted to reboot at least once. Then just reset your desktop wallpaper to a soothing plain black background, and enjoy the lack of evaluation expiration pop-up warnings. 🙂

Resolving X++ and metadata conflicts when merging code

In the near future, Microsoft will seal off customizations/overlayering in Dynamics 365 for Operations. Once that happens, and VAR/ISVs catch up, you won’t have the same problems with conflicts while merging code from various sources (including Microsoft’s own X++ hotfixes). Until then, though… I found the process of properly resolving conflicts to be not completely intuitive, and ended up doing unnecessary manual work and comparison for a while. Here’s how to do it the right way (also the easier way), with a little-known secret or two to help.

You probably already know that after merging in new models, you need to go to Dynamics 365 > Addins > Create project from conflicts in Visual Studio to check for conflicts.

Resolve Conflicts 0a

After doing so, you need to select which model to look for conflicts in. Yes, you need to do them one at a time. Yes, the dialog disappears after each check, making it a hassle to use if you are juggling models from a bunch of different ISVs.

Resolve Conflicts 0b

To save time, be smart about which models you check. You only need to check third party “customization” models, not “extension” models. (Comment if you know of a way conflicts can appear in extension models!) If you’re savvy, you might be able to limit it further, if you know who is changing which objects. But play it safe; you don’t want to miss any conflicts!

Assuming you actually hit a conflict… it’s going to need to create a project. I like to use a naming scheme like ModelNameConflicts_YYYYMMDD to make it easy to clean up later.

The project will be created, and it will contain any and all objects that have conflicts. Here’s where it got confusing for me; intuitively, I thought I’d want to right-click the object and choose “Compare and Merge Code” or something. Nope! Start out by just opening the object.

Resolve Conflicts 1

Once you open it, it won’t be obvious where the conflicts are. They’re marked, but if it’s a big complex object, it might be hard to find them.

The secret?… put “cf:” in the search bar and hit enter.

Resolve Conflicts 2

The object will be filtered to show you the conflicts. (See the little red double arrows icon?)

What you do next depends on whether the conflict is in metadata or in code. It’s not that different, though. One at a time, right click each object, and choose either “Resolve property conflicts” or “Resolve code conflicts.”

If it’s a property conflict:

Resolve Conflicts 3

If you are resolving a property conflict, you’ll get a GUI that you hopefully find intuitive.

Resolve Conflicts 4

If it’s a code conflict:

Resolve Conflicts 5

If you are resolving a code conflict, you’ll be given a DIFF tool to do the merge. This post is not meant to fully explain code merges, but it might be intuitive for you anyway, even if you aren’t already familiar with the concept.

Resolve Conflicts 6

Whether you’ve done a property merge or a code merge, you’ll be able to mark the conflict as resolved, so it won’t come up if you do another check for conflicts.

However, re-merging models might cause the conflict to pop up again, if your ISVs (or whomever you are merging from) do not integrate the merged changes. So, be prepared to re-check in the future, and possibly correct the same conflicts repeatedly!

Doing a code merge when a VAR/ISV has renamed an object

This is probably general to Visual Studio (or any other development tool and source control), and not specific to Dynamics 365 for Operations; but after it ate up several hours of my life figuring out why a build wasn’t working, I thought I should share.

What might be unique (or at least more prevalent) in our world of Dynamics 365 for Operations is the need to do a “code merge” from multiple ISVs or VARs. Chances are it will become almost routine for you, and unless Visual Studio gets better at it, after a few months you’ll be routinely promoting changes that aren’t automatically picked up in “Pending Changes.”

Unfortunately, if you get used to promoting both adds and deletes all over, you might miss that renames should be treated specially. You might inadvertently end up with two copies of the object in source control, one with the old name and one with the new name… and, unless you’re sharper than me (certainly possible!) you’ll spend a lot of time puzzling out why your build won’t work but your local compile does, even though your source seems synched. (But, you might eventually notice, the names in your Application Explorer don’t match your file names…!)

Anyway, the meat of the matter is: make sure to watch out for renames, and promote them by highlighting the relevant “delete” and “add” together, right-clicking, and choosing “Promote as Rename.”

Promote as Rename


And as a tip  to those working for an ISV/VAR: try to avoid renaming objects once they’ve been checked in, and pass along warning if you do. Maybe your client will have no problem if you don’t… but there’s a chance you’ll be saving them a long night if you do!