D365, Flow and Twilio

In this post I will walk through the steps to use Microsoft Flow and Twilio to provide two factor authentication for registering with a Microsoft D365 portal. For the purpose of this post I am not actually going to go through the portal registration process as there are plenty of posts out there on how to configure the D365 portal for registration. Instead, I will simulate the process by creating my own Contact record and using Flow to deal with the messaging.

The first task is to sign up for a Twilio account. The great thing is they provide free trial to check the services are going to meet your needs. For our needs we are creating a Verify project.

Step-1

Step-2

The next step is to verify a phone number against your own Twilio account. This option also allows you to set 2FA on your own Twilio account.

Step-3

We then add a Friendly Name for the application.

Step-4

Add a phone number to make your first request

Step-5

Pressing the Make Request button will send the message to the designated mobile number with the Friendly Name of your project in the content of the SMS message. Make sure you select the correct Country Code!

Step-6

To verify the code supplied in the SMS message you can complete the process by supplying the verification code.

Step-7

If you supplied the correct verification code then you will get the following response.

Step-8

Having completed the setup for the Verify project we can now turn our attention to the Microsoft Flow configuration. The process will need to send the verification code when a new Contact record is created that contains a value in the Mobile field.

The first step is to create a trigger that is going to start the operation. For this we need a Dynamics 365 trigger called When a record is created.

Step-1Step-2

Make sure you give your trigger an appropriate name before you add any other actions because this can’t be changed once actions are attached to the trigger.

Step-3

Enter the name of the D365 Organization and the select the relevant entity for the trigger, in our case its the Contacts entity. After that we add our first Action which is a Condition to check that the Mobile Phone field has a value. If it doesn’t the workflow will just finish.

Step-4

If a Mobile Phone number is supplied in the new Contact record then we can create the request to our Twilio application. This is done by creating an HTTP Action that uses the POST method to send the request.

Step-5

Twilio has some good documentation on the API requests that can be made for the Verify API, including simple examples. – Verify Phone Verification API. Using this API we need to provide a couple of header values. The first is normal Content-type (application/json) and the important one is the X-Authy-API-Key which is available in the General Settings of your Verify Project.

Step-6

The POST body names to contain the key-value pairs for parameters listed in the API documentation. For the phone_number we select the dynamic value from the Contact Mobile Phone field. To test our Flow we can use the Test option and create a quick Contact record in D365.

Step-7

Step-8

Once you click the Test button then you need to create your Contact record and fill out the Mobile Phone field. The mobile phone number can be entered without spaces, with spaces or with hyphens.

Step-9

You should then get a response on your Flow to say that everything ran successfully, plus you should get you SMS on the mobile phone you entered in the Contact record with a verification code.

Step-10

You can also review the Flow history within the Flow dashboard, under My Flows, to see which requests were successful. Opening the specific run will provide details of when the trigger was fired, which Actions were completed and details about the POST request and response.

Step-11

In my next post I will provide a walk through of how to verify the code that was supplied in the SMS message.

D365 Authentication – connect my apps

With the constant push to make data more secure D365 now includes Application Users that can be configured to allow access to D365 from external applications. There are numerous articles around explaining how to create your Application User in D365 using Azure AD App Registrations – this one from PowerObjects has a good explanation of steps required to achieve this.

In this post I wanted to briefly describe the next steps for using that authentication method in your app to perform CRUD operations. To demonstrate this I created a simple Console App to connect to a D365 instance and perform some simple operations using both the OrganizationWebProxyClient and the CrmServiceClient. For reference, I added a package from NuGet called Microsoft.CrmSdk.XrmTooling.CoreAssembly which contains the official Microsoft.Xrm.Tooling.Connector assembly. The same NuGet package also contains the Microsoft.CrmSdk.CoreAssemblies dependency which contains the Microsoft.Xrm.Sdk.dll. Within the console app I created a class that manages to authentication.

using Microsoft.IdentityModel.Clients.ActiveDirectory;
using System;
using System.Collections.Generic;

namespace OrgService.Connect
{
    class AuthHook : Microsoft.Xrm.Tooling.Connector.IOverrideAuthHookWrapper
    {
        // In memory cache of access tokens
        Dictionary<string, AuthenticationResult> accessTokens = new Dictionary<string, Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationResult>();

        public void AddAccessToken(Uri orgUri, Microsoft.IdentityModel.Clients.ActiveDirectory.AuthenticationResult accessToken)
        {
            // Access tokens can be matched on the hostname,
            // different endpoints in the same organization can use the same access token
            accessTokens[orgUri.Host] = accessToken;
        }

        public string GetAuthToken(Uri connectedUri)
        {
            // Check if you have an access token for this host
            if (accessTokens.ContainsKey(connectedUri.Host) && accessTokens[connectedUri.Host].ExpiresOn > DateTime.Now)
            {
                return accessTokens[connectedUri.Host].AccessToken;
            }
            else
            {
                accessTokens[connectedUri.Host] = GetAccessTokenFromAzureAD(connectedUri);
            }
            return accessTokens[connectedUri.Host].AccessToken;
        }

        private AuthenticationResult GetAccessTokenFromAzureAD(Uri orgUrl)
        {
            string organizationUrl = "https://myD365instance.crm6.dynamics.com";
            string clientId = "00000000-0000-0000-0000-000000000001";    // This is the Application ID from your App Registration
            string appKey = "<the client secret for your app>";          // The Client Secret from your App Registration
            string aadInstance = "https://login.microsoftonline.com/";
            string tenantID = "00000000-0000-0000-0000-000000000001";    // The GUID of your Azure Tenant ID. See the article above for details on finding this value.

            ClientCredential clientcred = new ClientCredential(clientId, appKey);
            AuthenticationContext authenticationContext = new AuthenticationContext(aadInstance + tenantID);
            AuthenticationResult authenticationResult = authenticationContext.AcquireToken(organizationUrl, clientcred);

            accessTokens[new Uri(organizationUrl).Host] = authenticationResult;

            return authenticationResult;
        }
    }
}

To access the D365 instance using the OrganizationWebProxyClient we can use the following code

                string organizationUrl = "https://myD365instance.crm6.dynamics.com";

                var hook = new AuthHook();

                var requestedToken = hook.GetAuthToken(new Uri(organizationUrl));

                using (var webProxyClient = new OrganizationWebProxyClient(new Uri($"{organizationUrl}/XRMServices/2011/Organization.svc/web"), false))
                {
                    webProxyClient.HeaderToken = requestedToken;

                    // Test with a basic WhoAmI request first
                    OrganizationRequest request1 = new OrganizationRequest()
                    {
                        RequestName = "WhoAmI"
                    };
                    OrganizationResponse response1 = webProxyClient.Execute(request1);

                    // We are also able to create an instance of the OrganizationService and run queries against it
                    IOrganizationService organizationService = webProxyClient as IOrganizationService;

                    Entity entity = organizationService.Retrieve("account", new Guid("92348762-0D32-E611-80EC-B38A27891203"), new Microsoft.Xrm.Sdk.Query.ColumnSet("name", "preferredcontactmethodcode"));

                }

Using the Microsoft.Xrm.Tooling.Connector we require a slightly different approach when using the authenticated Application User. It still uses the AuthHook class but through the AuthOverrideHook property. This uses the same AuthHook object we created earlier for the OrganizationWebProxyClient.

                // Register the hook with the CrmServiceClient
                CrmServiceClient.AuthOverrideHook = hook;

                // Create a new instance of CrmServiceClient, pass your organization url and make sure useUniqueInstance = true!
                var client = new CrmServiceClient(new Uri(organizationUrl), useUniqueInstance: true);

                // Test with a basic WhoAmI request first
                OrganizationRequest request2 = new OrganizationRequest()
                {
                    RequestName = "WhoAmI"
                };
                OrganizationResponse response2 = client.Execute(request2);

                Entity entity1 = client.Retrieve("account", new Guid("92348762-0D32-E611-80EC-B38A27891203"), new Microsoft.Xrm.Sdk.Query.ColumnSet("name", "preferredcontactmethodcode"));

SharePoint Online List View Thresholds

Running CAML queries against List Views with a large number of items (> 5000) can fail with the following error.

The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

To overcome this problem we have to ensure our query meets these requirements when running against SharePoint Online.

  • Run the query in a loop a number of times using the ListItemCollectionPosition to keep track of the looping process.
  • Ensure that the columns specified in any Query Where clause and the OrderBy are already indexed on the list view. If you include columns in either of these parts of the query that are not indexed then you will still get the same error.
  • Ensure the Query AllowIncrementalResults is set to true
  • If you want to restrict the Query to a particular folder in the List View set the FolderServerRelativeUrl property.

I used PowerShell to test out my CAML query.

#Load SharePoint CSOM Assemblies
$folder = "C:\Nuget\packages\Microsoft.SharePointOnline.CSOM.16.1.8412.1200\lib\net45"
Add-Type -Path "$folder\Microsoft.SharePoint.Client.dll"
Add-Type -Path "$folder\Microsoft.SharePoint.Client.Runtime.dll"
   
#Variables for Processing
$SiteUrl = "https://mysharepoint.sharepoint.com/sites/mysite"
$ListName="MyList"
 
$UserName="me@mysharepoint.onmicrosoft.com"
$Password ="nottellingyou"
  
#Setup Credentials to connect
$Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName,(ConvertTo-SecureString $Password -AsPlainText -Force))
  
#Set up the context
$Context = New-Object Microsoft.SharePoint.Client.ClientContext($SiteUrl) 
$Context.Credentials = $credentials

try {
   
    #Get the List
    $List = $Context.web.Lists.GetByTitle($ListName)

    $Context.Load($List)
    $Context.ExecuteQuery()

    $position = $null

    $allItems = @()

    Do {
        $Query = New-Object Microsoft.SharePoint.Client.CamlQuery;
        # Define the starting position
        $Query.ListItemCollectionPosition = $position

        $Query.ViewXml = "<View Scope='RecursiveAll'>  
                            <Query> 
                                <Where>
                                    <And>
                                        <And>
                                            <And>
                                                <Eq><FieldRef Name='ReportName' /><Value Type='Text'>Monthly Report</Value></Eq>
                                                <Eq><FieldRef Name='Category' /><Value Type='TaxonomyFieldType'>Statement</Value></Eq>
                                            </And>
                                            <Eq><FieldRef Name='ProductLine' /><Value Type='TaxonomyFieldType'>Credits</Value></Eq>
                                        </And>
                                        <Eq><FieldRef Name='ContentType' /><Value Type='Computed'>Report</Value></Eq>
                                    </And>
                                </Where>
                            </Query> 
                        </View>"

        # Define a folder for the starting point of the recursive search
        $Query.FolderServerRelativeUrl = "/sites/MySite/MyList/Folder1"
        $Query.AllowIncrementalResults = $true

        $ListItems = $List.GetItems($Query) 
        $Context.Load($ListItems)

        $Context.ExecuteQuery()    
    
        # Increment the next position for the search
        $position = $ListItems.ListItemCollectionPosition
    
        $allItems += $ListItems
    }
    Until($position -eq $null)
}
Catch {
    Write-Error $_.Exception.Message
    Break
} 


write-host "Total Number of List Items found:"$allItems.Count
 
#Loop through each item
$allItems | ForEach-Object {
    #Get the Title field value

    $Context.Load( $_.File)
    $Context.Load( $_.File.ListItemAllFields)
    $Context.ExecuteQuery()
    write-host $_.File.Name
}  

D365 – Web API – Custom Actions & Entity Reference Inputs

Using v9.0 of the D365 Web Api we are able to trigger Actions. Using a standard URL structure we can call those Actions from client-side script, for example – POST /api/data/v9.0/incidents(00000000-0000-0000-0000-000000000000)/Microsoft.Dynamics.CRM.new_CustomIncidentAction.

When these Actions have Input parameters they can be added to the body of the request but knowing how to structure the body can be a bit of a challenge. For numeric, character and boolean values it’s just a case of constructing the relevant JSON object. For EntityReference’s there is a specific structure which will require you to find some details in the ODataV4Metadata.xml file. This file can be downloaded by going to your D365 instance under Settings | Customizations | Developer Resources and clicking on the Download OData Metadata link under the Instance Web API section. Search the file for the name of your Action you want to call and you should get something similar to this…

<Action Name="new_CustomIncidentAction" IsBound="true">
<Parameter Name="entity" Type="mscrm.incident" Nullable="false" />
<Parameter Name="TeamId" Type="mscrm.team" Nullable="false" />
</Action>

The first is the GUID of the incident we are running the Action against. The second is the one we are interested in as this is our input EntityReference.

To construct the JSON body of our request we need three pieces of information

  1. Parameter name – this we can get from the XML (above). The name is case sensitive.
  2. The OData type – this is also specified in the XML, but needs to be defined in a slightly different format than is defined in the XML.
  3. The EntityReference Id – the GUID value for the entity

The JSON object for our EntityReference input parameter can be specified like this.

{ "TeamId": {
	"@odata.type": "Microsoft.Dynamics.CRM.team",
	"teamid": "2ab7a6d5-9a36-e611-80e7-c4346bc48ef4" 
	} 
}

D365 JavaScript Web Resource Library Usage

It’s fairly straight forward to find which Forms use a Web Resource with the ‘Show Dependencies’ option, but that will list Forms where the JavaScript library has been added to the form in the Form Libraries. This doesn’t give you any indication whether the functions in the JavaScript file have actually been used in the form.

Obviously, you can scroll through each Tab and Field in the Event Handlers to look for occurrences of an event, but if you are doing maintenance on a form you didn’t originally develop then it could take a while to scroll though all the possible events. The Event Libraries and Event Handlers are stored in the formjson column of the systemform entity so with a simple FetchXML query you can get a list of Forms that actually use functions from a particular library. The example below will return all Forms where the msdyn_/Utils/head.js Web Resource has been used.The key part is the Filter Condition that is looking for a LibraryName that matches our JavaScript library. The formjson content contains an array of EventHandlers that define the EventName, FunctionName and LibraryName.

<fetch>
  <entity name='systemform' >
    <attribute name='name' />
    <attribute name='formjson' />
    <attribute name='formactivationstate' />
    <attribute name='type' />
    <attribute name='objecttypecode' />
    <filter>
      <condition attribute='formjson' operator='like' 
         value='%&quot;LibraryName&quot;:&quot;msdyn_/Utils/head.js%' />
    </filter>
  </entity>
</fetch>

The next step is to figure out which EventHandlers, if any, use that library in those forms. To do that I created a simple LINQPad script to return a list of the functions in each form.

void Main()
{
	string url = "https://myorgname.crm6.dynamics.com"; 
	string username = "my.user@myorgname.onmicrosoft.com";
	string password = Util.GetPassword("d365-admin");
	
	CrmServiceClient conn = new CrmServiceClient($"Url={url};Username={username};Password={password}; AuthType=Office365");

	conn.OrganizationServiceProxy.Timeout = new System.TimeSpan(0, 3, 0);
	
	conn.OrganizationServiceProxy.EnableProxyTypes();
	
	IOrganizationService orgService = conn.OrganizationWebProxyClient != null ? (IOrganizationService)conn.OrganizationWebProxyClient : (IOrganizationService)conn.OrganizationServiceProxy;

	string pagingCookie = null;
	
	EntityCollection formsCol = null;
	
	List<Entity> forms = new List<Microsoft.Xrm.Sdk.Entity>();
	
	do
	{
		string fetchXML = $@"<fetch>
		  <entity name='systemform' >
		    <attribute name='name' />
		    <attribute name='formjson' />
		    <attribute name='formactivationstate' />
		    <attribute name='type' />
		    <attribute name='objecttypecode' />
		    <filter>
		      <condition attribute='formjson' operator='like' value='%&quot;LibraryName&quot;:&quot;msdyn_/Utils/head.js%' />
		    </filter>
		  </entity>
		</fetch>";

		formsCol = orgService.RetrieveMultiple(new FetchExpression(fetchXML));
		
		if (formsCol.Entities.Count > 0)
		{
			forms.AddRange(formsCol.Entities);	
		}
		pagingCookie = formsCol.PagingCookie;
	}
	while (formsCol.MoreRecords);

	forms.Select(e => new
	{
		Name = e.GetAttributeValue<string>("name"),
		FormJSON = e.GetAttributeValue<string>("formjson"),
		Functions = System.Text.RegularExpressions.Regex.Matches(e.GetAttributeValue<string>("formjson"), "(\"EventName\":(.*?)},)", RegexOptions.None).Cast<Match>().Select(m => m.Value).ToList()
	}).Dump();
}

Since the formjson content is just a string, I used a Regex pattern to find multiple occurrences of the Event Handlers for each form. The result is a list of the EventNames, FunctionNames and LibraryNames for each form that utilises the JavaScript library in the FetchXML statement.

Azure DevOps – Restore deleted pipelines

If you have deleted a release definition, either intentionally or by mistake, currently there is no way to restore it through the UI. However, there is a way to recover it using a simple PowerShell script. The details for the script can be found here.

A couple of points to note about running the script.

  • The script will only work within 4 weeks of the definition being deleted.
  • You will need to create a Personal Access Token (PAT) within your DevOps account. Details for creating your PAT are available here.
  • You do not need to encode the $accountName, $projectName and $definitionNameToRecover variables. Values containing spaces will work.
  • The script will recover the release definition but not the release history

D365 Portals – Language agnostic Content Snippets

When dealing with multi-language websites Content Snippets are a great way to separate out your language specific portal content from your Web Templates and Page Copy.

If you want to use a single Content Snippet across all of your portal languages, for example with generic HTML that is language agnostic, then all you need to do is ensure the value in the Content Snippet Language field is left blank.