Last week, around June 30 2018, this bug showed up from nowhere. It’s thus a problem in editable grids, related to currency numbers apparently.
After searching the web and finding nothing, I reached the conclusion that it was a Microsoft bug… not always easy to convince the customers though. However, since this bug is online and showed up right after the update to Dynamics 365 Version 1612 (18.104.22.1680).
This is the link to the page describing the update in question, service update 14 for Dyamics 365 8.2.2.
And, on July 2nd, a new thread appeared on community.dynamics.com discussing the issue, only confirming it.
So as you can see, the only thing to do here is open a ticket to Microsoft support, and wait…
I hope they react quickly as for many customers, this is a show stopper for editable grids.
Maybe you’ve already heard of SPNs, also known as Service Principal Names. Most CRM technical people have heard of them, but not many can really explain WHY they are necessary. Some can say it is for ensuring “mutual authentication”, but few can really explain how this mechanism works.
Today, let me explain to you my understanding of SPN’s, based on this article.
The Kerberos mechanism by which a client authenticates a service works as follows:
- When a service is installed, a service installer, running with administrator privileges, registers one or more unique SPNs for each service instance.
- The names are registered in the Active Directory Domain Controller (DC) on the user or computer account object that the service instance will use to log on.
- When a client requests a connection to a service, it composes an SPN for a service instance, using known data or data provided by the user.
- The client then uses the SSPI negotiate package to present the SPN to the Key Distribution Center (KDC) for the client domain account.
- The KDC searches the forest for a user or computer account on which that SPN is registered.
- If the SPN is registered on more than one account, the authentication fails.
- Otherwise, the KDC encrypts a message using the password of the account on which the SPN was registered.
- The KDC passes this encrypted message to the client, which in turn passes it to the service instance.
- The service uses the SSPI negotiate package to decrypt the message, which it passes back to the client and on to the client’s KDC.
- The KDC authenticates the service if the decrypted message matches its original message.
Let me rephrase this in the CRM world:
- When CRM is installed, SPNs are registered for the identity of the application pool CRMAppPool. It can be Network Service or some other account.
- That’s why, when changing the identity of the application pool, you need to remove existing SPNs and create new ones.
- When a client requests a connection to CRM, it composes an SPN for it.
- That’s why you need to register SPNs for both server the server name and the FQDN, because you can’t be sure how the SPN will be composed by the client.
- (1) The client presents the SPN to the KDC for the client domain account
- (2) The KDC searches the domain for a user or computer account on which that SPN is registered
- Here, the KDC must find the identity of the CRMAppPool
- If the SPN is registered on more than one account, the authentication will fail
- Indeed, because the KDC will not know which account to use
- The KDC encrypts a message with the password of the CRMAppPool identity
- (3 & 4) The KDC passes the message to the client, which passes it to the service instance, ie to the CRM server
- (5 & 6) The CRM server can decrypt the message, passes it back to the client which passes it back to the KDC
So that’s why we talk about mutual authentication: not only is the client identified (this aspect is not described here), but also the service.
I found on this post a very interesting thing: the list of all the currently released rollups for CRM 2011 (there’s also a list for CRM 4.0 for those interested). I reproduce the CRM 2011 list below. For more information about UR6, see that same post.
The post mentions the following important information:
If you did new installs with the Update Rollup 6 “slipstream” packages that were released (these would be the packages that include the full install bits as well as Update Rollup 6) you will get an error trying to patch those components to the new Update Rollup 6 build. This Knowledge Base article describes the issue:
This Knowledge Base article is live, as well as the updated Update Rollup 6 KB article:
The only workaround is to reinstall if you happened to use the Update Rollup 6 slipstream builds. For server you can connect to existing databases on the reinstall. The reinstall goes for any of the components that you installed with the Update Rollup 6 slipstream builds, (Client, Router, etc.). If you do not reinstall you will not be able to patch to the re-released Update Rollup 6 packages or future Update Rollups, as Update Rollup 6 represents a new baseline.
What this means is that:
- if you made a fresh install of CRM, that includes UR6, then you will need to reinstall everything.
- if you installed the UR6 patch to an existing install, you should be able to install the UR6 on top of it without problem
Second part of the series: the Application tier.
- 2.1 Optimizing and Maintaining Microsoft Windows Server
- 2.2 Optimizing and Maintaining the Microsoft .NET Framework and Microsoft .NET Applications
- 2.3 Optimizing and Maintaining Internet Information Services
- 2.4 Optimizing the Performance of Microsoft Dynamics CRM Server 2011
- 2.5 Optimizing Microsoft Dynamics CRM Reporting Services
- 2.6 Optimizing Report Performance
- 2.7 Best Practices for Optimizing Workflow
2. Optimizing and Maintaining the Application Tier
2.1 Optimizing and Maintaining Microsoft Windows Server
- See the Performance Tuning Guidelines for Windows Server 2008 R2
- Increasing the Ephemeral TCP Port Limit:
- In some situations, you may want to reserve a range of ports so that a program or process that requests a random port will not be assigned a port that is in the reserved range. The ports in that range are called ephemeral ports. In rare cases there may be no free ephemeral ports available, which will cause the connection open to fail or time out.
- To view the current range of ephemeral ports: netsh int <ipv4|ipv6> show dynamicport <tcp|udp>
- To change the current range of ephemeral ports: netsh int <ipv4|ipv6> set dynamic <tcp|udp> start=number num=range
- To increase the maximum number of ephemeral TCP ports: navigate to HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParameters.Create a registry entry by using the following information:
Value Name: MaxUserPort
Value Type: DWORD
Value data: 65534
Valid Range: 5000-65534 (decimal)
Default: 0x1388 (5000 decimal)
Description: Controls the maximum port number used when a program requests any available user port from the system. Typically, ephemeral (short-lived) ports are allocated between the values of 1024 and 5000, inclusive.
- Monitoring the Performance of Microsoft Windows Server: Microsoft Windows Server provides performance counters that you can use to help identify potential performance bottlenecks associated with memory and the cache, processors (and multi-processor computers), physical disks, and the network infrastructure. Remember to monitor each disk and processor used by the operating system. Check the TechNet article Performance and Reliability Monitoring Step-by-Step Guide for Windows Server 2008.
2.2 Optimizing and Maintaining the Microsoft .NET Framework and Microsoft .NET Applications
- Optimizing the Performance of the Microsoft .NET Framework: Configuring the .NET Framework for optimal performance involves tuning the common language runtime (CLR) and then, depending on the nature of any specific application, tuning the associated .NET Framework technology, for example ASP.NET-connected applications, Web services, Enterprise Services, and ADO.NET code. Consider the potential issues: (see here for more information)
||Creating too many objects, or failing to properly release resources, pre-allocate memory, or explicitly force garbage collection can prevent the CLR from efficiently managing memory, which can lead to an increased working set size.
||Implementing finalizers unnecessarily, failing to suppress finalization in the Dispose method, or failing to release unmanaged resources can lead to unnecessary delays in reclaiming resources and can potentially create resource leaks.
|Improper use of threads
||Creating threads on a per-request basis and not sharing threads using thread pools can cause performance and scalability bottlenecks for server applications.
|Abusing shared resources
||Creating resources per request can lead to resource pressure, and failing to properly release shared resources can cause delays in reclaiming them.
||Implicit type conversions and mixing value and reference types leads to excessive boxing and unboxing operations.
|Misuse of collections
||Each collection type in the .NET Framework class library is designed to meet specific storage and access requirements; they may not perform optimally outside of those requirements.
||Looping magnifies even the slightest coding inefficiency, and loops that access an object’s properties are a common culprit of performance bottlenecks, particularly if the object is remote or the property getter performs significant work.
2.3 Optimizing and Maintaining Internet Information Services
- Optimizing the Performance of Internet Information Services: Optimizing the performance of Microsoft Internet Information Services (IIS) within a Microsoft Dynamics CRM 2011 implementation benefits not only the overall system, but also any custom applications, plug-ins, or add-ins that have been developed by using the Microsoft Dynamics CRM 2011 SDK.
- Optimizing the Performance of Integrated Windows Authentication and Kerberos Authentication
- In Internet Explorer, ensure that “Enable Integrated Windows Authentication” is set to enable use of Kerberos for integrated authentication.
- There is a known issue where users are members of too many groups to be communicated in a UDP packet. It is possible to get Kerberos to use TCP rather than UDP, which enables larger packets of information.
- To determine whether users are connecting via Kerberos, in the registry, enable Kerberos logging. If necessary, to help diagnose whether the user can obtain and use a Kerberos ticket, use applications such as Kerbtray
- On IIS, tracing and debugging, while disabled by default, may cause performance issues if enabled.
- Optimizing Microsoft .NET ThreadPool Settings: if each .aspx page makes a Web service call to a single IP address, it is recommended to adjust these parameters as shown in the following table.
||12*n (where n is the number of CPUs)
||50 (manually add this parameter and value to the file)
- Monitoring the Performance of Internet Information Services: One of the key counters to monitor and measure against a baseline is the %Process Time for the inetinfo (IIS).
2.4 Optimizing the Performance of Microsoft Dynamics CRM Server 2011
- Disable platform tracing
- Verify that the computers meet the hardware and software requirements
- Enhancing Performance by Distributing Server Roles on Multiple Servers
- Throttling Client Synchronization Processes
- Limiting the Number of Records Returned by Aggregate Queries
- Applying RetrieveMultiple Query Optimizations for large data sets
- Setting the Default View to a smaller view to limit the records that are displayed
- Customizing Quick Find Views by Limiting Search Columns
- Leveraging teams instead of an excessively complex business hierarchy because teams will provide better performance with a lower penalty for security checks
- Use Field Level Security (FLS) wisely, since there is a performance impact associated with using FLS
- When changing the order of records returned by a saved query, consider adding an index based on the new ordering to improve the performance of the query
- Use an iterative process to determine which index best optimizes query performance
- Disabling Auto-Complete on Lookups
- To optimize the performance of queries on custom entities, ensure that all columns on the ORDER BY clause derive from a single table, and build an index that satisfies the ORDER BY requirements and as much of the query’s
- A specific recommendation for any custom application is to limit any columns and rows retrieved to those required to achieve the application’s business goals.WHERE clause selection criteria as possible.
2.5 Optimizing Microsoft Dynamics CRM Reporting Services
- Report processing and rendering are memory intensive operations, so ensure that the computer hosting the report server includes ample memory.
- Host the report server and the report server database on separate computers rather than hosting both on a single high-end computer.
- If all reports are processing slowly, consider a scale-out deployment with multiple report server instances. For best results, use load balancing software and hardware to distribute requests evenly across multiple report servers in the deployment. If load balancing is used, it is advisable to use client affinity to avoid loading the session cache in memory on multiple servers.
- If a single report is processing slowly, tune the query if the report must run on demand. You might also consider caching the report or running it as a snapshot.
- If all reports process slowly in a specific format (for example, while rendering to PDF), consider file share delivery, adding more memory, or using another format.
2.6 Optimizing Report Performance
- Configure reports to display data from a specified time frame, for example the previous 90 days, rather than displaying all records in the Microsoft Dynamics CRM database.
- Reports with a large dataset or a complex SQL query should not be available to all users on-demand. Instead, schedule a snapshot in Report Manager during a period when the system is lightly loaded.
- Deploy reports through Microsoft Dynamics CRM, and then use Report Manager to run the reports and have the results posted at a scheduled time.
- Reports should access the fewest datasets possible to meet business requirements.
- When possible, use fetch based reports, which are much more efficient than SQL-based reports, to run against Filtered Views.
- Consider using subreports. Show aggregated information in a report initially, and then use subreports for drilling down on aggregates to show non-aggregated values. Using this technique will prevent Report Server aggregations and improve performance.
- Use explicit paging for reports that require bringing large amounts of data to the Reporting Services middle tier. Build reports so that they show only a page of the total records at one time and have explicit clickable links to bring in data for further pages. Although Report Viewer control shows paging control on reports, all the data needed for the full report has already been pulled from the middle tier.
- Use SQL “Group By”: This helps to prevent the computer running Microsoft SQL Server from being overtaxed with gathering, transmitting, and then processing large volumes of data. Instead, it uses the natural indexing and grouping ability of SQL Server to massively reduce this overhead.
- Making Reports Pre-Filterable (CRMAF_)
- Using Dynamic Excel or Filtered View Queries: make reports restrictive, verify that a non-clustered index exists on fields in the where clause
- Throttling Resources used for Reports and Data Visualizations: If a query is sent for reporting purposes, it is tagged with MSCRMReportsGroup. If it is sent for data visualization, it is tagged with MSCRMVisualizationsGroup. See Managing SQL Server Workloads with Resource Governor.
2.7 Best Practices for Optimizing Workflow
- Determine the appropriate security/permissions model for workflow.
- Use the Scope property judiciously.
- Review workflow logic carefully.
- To improve performance in large deployments, scale out as necessary
- Monitor the Microsoft Dynamics CRM 2011 database for excess workflow log records.
This is a summary of this white paper, to use as a quick reference when necessary.
This summary will come in different parts. I’ll update this posts with links to the new posts as they come.
Part 1. Server-Side Techniques for Optimizing the Client Tier
1.1 Using Compression Techniques
- Configuring HTTP Compression. Note: Compression of dynamic application responses can affect CPU resources because IIS does not cache compressed versions of dynamic output. So use it only on servers that have slow network connections but that have CPU time to spare. Note that caching compressed static responses does not consume CPU resources.
- Using WAN Accelerator Hardware: To address latency issues, several vendors offer web-acceleration appliances that improve the performance of applications such as Microsoft Dynamics CRM, and using these devices can greatly improve Microsoft Dynamics CRM performance over the WAN. Using WAN accelerator hardware can help improve performance especially in on-premises scenarios with a geographically distributed deployment in which users are distributed around the world and performance for users in a specific location is not satisfactory.
1.2. Configuring Proxy Server Settings
- Configuring Proxy Server Settings: For on-premises deployments of Microsoft Dynamics CRM 2011 within a local area network, client computers can achieve much higher throughput by completely bypassing the proxy server, as long as the fully qualified domain name of the Microsoft Dynamics CRM server is listed as an exception.
1.3 Reducing E-Mail Traffic by Modifying Outlook Rules:
- For deployments with the E-mail router configured to use a forward mailbox strategy, when users only track Microsoft Dynamics CRM e-mails, then configure Outlook to forward only messages that include CRM: in the subject or body.
Shawn Dieken has just posted here what I think is a great list of links to white papers, technical documents and other stuff about Microsoft Dynamics CRM. So for my own reference, I just copied this list to add it here, so I can find it back more easily. Thank you for the great job Shawn!
Brad Wilson, who was since 2005 General manager of the customer relationship management (CRM) business with the Microsoft Dynamics organization, is handing over his responsibilities to Dennis Michalis, who officially took over as general manager of the product line last week. The process began approximately 2 months ago.
Before joining Microsoft, he worked for Deloitte, Infor Global Solutions, XALT Technologies, a Fujitsu subsidiary, PWC, etc. With more than 20 years of experience as manager, he describes himself as “leading companies and build teams that won’t accept anything other than growth, profitability and fanatical customer loyalty”. He believes that “the key to great business results is to get great people, then do cool stuff to maximize their motivation, performance and effectiveness once you have them in the door”.
Hopefully, he will do a great job for Dynamics CRM, and help the product get even more growth, recognition and attractive features.
Thanks to Brad Wilson for the great job he accomplished as GM for Dynamics CRM. He helped bring the product from version 1.2 to version 2011. As a reminder, when Brad joined the team in 2005, he is the person who decider to skip entirely version 2.0 and to move directly to version 3.0. At the time, when version 2.0 was almost ready, he decided to go further with the Web Services architecture, and to provide the ability to create new objects with no coding.