SharePoint Online List View Thresholds

Running CAML queries against List Views with a large number of items (> 5000) can fail with the following error.

The attempted operation is prohibited because it exceeds the list view threshold enforced by the administrator.

To overcome this problem we have to ensure our query meets these requirements when running against SharePoint Online.

  • Run the query in a loop a number of times using the ListItemCollectionPosition to keep track of the looping process.
  • Ensure that the columns specified in any Query Where clause and the OrderBy are already indexed on the list view. If you include columns in either of these parts of the query that are not indexed then you will still get the same error.
  • Ensure the Query AllowIncrementalResults is set to true
  • If you want to restrict the Query to a particular folder in the List View set the FolderServerRelativeUrl property.

I used PowerShell to test out my CAML query.

#Load SharePoint CSOM Assemblies
$folder = "C:\Nuget\packages\Microsoft.SharePointOnline.CSOM.16.1.8412.1200\lib\net45"
Add-Type -Path "$folder\Microsoft.SharePoint.Client.dll"
Add-Type -Path "$folder\Microsoft.SharePoint.Client.Runtime.dll"
   
#Variables for Processing
$SiteUrl = "https://mysharepoint.sharepoint.com/sites/mysite"
$ListName="MyList"
 
$UserName="me@mysharepoint.onmicrosoft.com"
$Password ="nottellingyou"
  
#Setup Credentials to connect
$Credentials = New-Object Microsoft.SharePoint.Client.SharePointOnlineCredentials($UserName,(ConvertTo-SecureString $Password -AsPlainText -Force))
  
#Set up the context
$Context = New-Object Microsoft.SharePoint.Client.ClientContext($SiteUrl) 
$Context.Credentials = $credentials

try {
   
    #Get the List
    $List = $Context.web.Lists.GetByTitle($ListName)

    $Context.Load($List)
    $Context.ExecuteQuery()

    $position = $null

    $allItems = @()

    Do {
        $Query = New-Object Microsoft.SharePoint.Client.CamlQuery;
        # Define the starting position
        $Query.ListItemCollectionPosition = $position

        $Query.ViewXml = "<View Scope='RecursiveAll'>  
                            <Query> 
                                <Where>
                                    <And>
                                        <And>
                                            <And>
                                                <Eq><FieldRef Name='ReportName' /><Value Type='Text'>Monthly Report</Value></Eq>
                                                <Eq><FieldRef Name='Category' /><Value Type='TaxonomyFieldType'>Statement</Value></Eq>
                                            </And>
                                            <Eq><FieldRef Name='ProductLine' /><Value Type='TaxonomyFieldType'>Credits</Value></Eq>
                                        </And>
                                        <Eq><FieldRef Name='ContentType' /><Value Type='Computed'>Report</Value></Eq>
                                    </And>
                                </Where>
                            </Query> 
                        </View>"

        # Define a folder for the starting point of the recursive search
        $Query.FolderServerRelativeUrl = "/sites/MySite/MyList/Folder1"
        $Query.AllowIncrementalResults = $true

        $ListItems = $List.GetItems($Query) 
        $Context.Load($ListItems)

        $Context.ExecuteQuery()    
    
        # Increment the next position for the search
        $position = $ListItems.ListItemCollectionPosition
    
        $allItems += $ListItems
    }
    Until($position -eq $null)
}
Catch {
    Write-Error $_.Exception.Message
    Break
} 


write-host "Total Number of List Items found:"$allItems.Count
 
#Loop through each item
$allItems | ForEach-Object {
    #Get the Title field value

    $Context.Load( $_.File)
    $Context.Load( $_.File.ListItemAllFields)
    $Context.ExecuteQuery()
    write-host $_.File.Name
}  
Advertisements

D365 – Web API – Custom Actions & Entity Reference Inputs

Using v9.0 of the D365 Web Api we are able to trigger Actions. Using a standard URL structure we can call those Actions from client-side script, for example – POST /api/data/v9.0/incidents(00000000-0000-0000-0000-000000000000)/Microsoft.Dynamics.CRM.new_CustomIncidentAction.

When these Actions have Input parameters they can be added to the body of the request but knowing how to structure the body can be a bit of a challenge. For numeric, character and boolean values it’s just a case of constructing the relevant JSON object. For EntityReference’s there is a specific structure which will require you to find some details in the ODataV4Metadata.xml file. This file can be downloaded by going to your D365 instance under Settings | Customizations | Developer Resources and clicking on the Download OData Metadata link under the Instance Web API section. Search the file for the name of your Action you want to call and you should get something similar to this…

<Action Name="new_CustomIncidentAction" IsBound="true">
<Parameter Name="entity" Type="mscrm.incident" Nullable="false" />
<Parameter Name="TeamId" Type="mscrm.team" Nullable="false" />
</Action>

The first is the GUID of the incident we are running the Action against. The second is the one we are interested in as this is our input EntityReference.

To construct the JSON body of our request we need three pieces of information

  1. Parameter name – this we can get from the XML (above). The name is case sensitive.
  2. The OData type – this is also specified in the XML, but needs to be defined in a slightly different format than is defined in the XML.
  3. The EntityReference Id – the GUID value for the entity

The JSON object for our EntityReference input parameter can be specified like this.

{ "TeamId": {
	"@odata.type": "Microsoft.Dynamics.CRM.team",
	"teamid": "2ab7a6d5-9a36-e611-80e7-c4346bc48ef4" 
	} 
}

D365 JavaScript Web Resource Library Usage

It’s fairly straight forward to find which Forms use a Web Resource with the ‘Show Dependencies’ option, but that will list Forms where the JavaScript library has been added to the form in the Form Libraries. This doesn’t give you any indication whether the functions in the JavaScript file have actually been used in the form.

Obviously, you can scroll through each Tab and Field in the Event Handlers to look for occurrences of an event, but if you are doing maintenance on a form you didn’t originally develop then it could take a while to scroll though all the possible events. The Event Libraries and Event Handlers are stored in the formjson column of the systemform entity so with a simple FetchXML query you can get a list of Forms that actually use functions from a particular library. The example below will return all Forms where the msdyn_/Utils/head.js Web Resource has been used.The key part is the Filter Condition that is looking for a LibraryName that matches our JavaScript library. The formjson content contains an array of EventHandlers that define the EventName, FunctionName and LibraryName.

<fetch>
  <entity name='systemform' >
    <attribute name='name' />
    <attribute name='formjson' />
    <attribute name='formactivationstate' />
    <attribute name='type' />
    <attribute name='objecttypecode' />
    <filter>
      <condition attribute='formjson' operator='like' 
         value='%&quot;LibraryName&quot;:&quot;msdyn_/Utils/head.js%' />
    </filter>
  </entity>
</fetch>

The next step is to figure out which EventHandlers, if any, use that library in those forms. To do that I created a simple LINQPad script to return a list of the functions in each form.

void Main()
{
	string url = "https://myorgname.crm6.dynamics.com"; 
	string username = "my.user@myorgname.onmicrosoft.com";
	string password = Util.GetPassword("d365-admin");
	
	CrmServiceClient conn = new CrmServiceClient($"Url={url};Username={username};Password={password}; AuthType=Office365");

	conn.OrganizationServiceProxy.Timeout = new System.TimeSpan(0, 3, 0);
	
	conn.OrganizationServiceProxy.EnableProxyTypes();
	
	IOrganizationService orgService = conn.OrganizationWebProxyClient != null ? (IOrganizationService)conn.OrganizationWebProxyClient : (IOrganizationService)conn.OrganizationServiceProxy;

	string pagingCookie = null;
	
	EntityCollection formsCol = null;
	
	List<Entity> forms = new List<Microsoft.Xrm.Sdk.Entity>();
	
	do
	{
		string fetchXML = $@"<fetch>
		  <entity name='systemform' >
		    <attribute name='name' />
		    <attribute name='formjson' />
		    <attribute name='formactivationstate' />
		    <attribute name='type' />
		    <attribute name='objecttypecode' />
		    <filter>
		      <condition attribute='formjson' operator='like' value='%&quot;LibraryName&quot;:&quot;msdyn_/Utils/head.js%' />
		    </filter>
		  </entity>
		</fetch>";

		formsCol = orgService.RetrieveMultiple(new FetchExpression(fetchXML));
		
		if (formsCol.Entities.Count > 0)
		{
			forms.AddRange(formsCol.Entities);	
		}
		pagingCookie = formsCol.PagingCookie;
	}
	while (formsCol.MoreRecords);

	forms.Select(e => new
	{
		Name = e.GetAttributeValue<string>("name"),
		FormJSON = e.GetAttributeValue<string>("formjson"),
		Functions = System.Text.RegularExpressions.Regex.Matches(e.GetAttributeValue<string>("formjson"), "(\"EventName\":(.*?)},)", RegexOptions.None).Cast<Match>().Select(m => m.Value).ToList()
	}).Dump();
}

Since the formjson content is just a string, I used a Regex pattern to find multiple occurrences of the Event Handlers for each form. The result is a list of the EventNames, FunctionNames and LibraryNames for each form that utilises the JavaScript library in the FetchXML statement.

Azure DevOps – Restore deleted pipelines

If you have deleted a release definition, either intentionally or by mistake, currently there is no way to restore it through the UI. However, there is a way to recover it using a simple PowerShell script. The details for the script can be found here.

A couple of points to note about running the script.

  • The script will only work within 4 weeks of the definition being deleted.
  • You will need to create a Personal Access Token (PAT) within your DevOps account. Details for creating your PAT are available here.
  • You do not need to encode the $accountName, $projectName and $definitionNameToRecover variables. Values containing spaces will work.
  • The script will recover the release definition but not the release history

D365 Portals – Language agnostic Content Snippets

When dealing with multi-language websites Content Snippets are a great way to separate out your language specific portal content from your Web Templates and Page Copy.

If you want to use a single Content Snippet across all of your portal languages, for example with generic HTML that is language agnostic, then all you need to do is ensure the value in the Content Snippet Language field is left blank.

D365 Portals – Parent Web Links and Authenticated Users

When you need to display a multi-level web link set the parent level web link does not need to have a page defined. However, this means that the parent level web link will always be visible. If the child web links are behind Restrict-Read web pages, then unauthenticated users will always see the parent level menu option, even if they can’t see the children. Although this is not a major issue as the menu item does not go to a page, it can be confusing to the users if they click on the menu item and it doesn’t go anywhere.

To overcome this problem we can assign a blank page to the parent web link using the Blank Page Page Template. The screenshot below shows the configuration for the parent web link page.
Blank Page

After creating the page we create an Access Control Rule (Restrict Read) that is linked to the Authenticated Users Web Role.
Blank Page - Access Control Rule

Having configured the page, all that is left to do is link the page to the Web Link.
Web Link

Now the Parent Web Link is only visible once the user has logged into the portal and not displayed to unauthenticated users.

D365 Portals – tokenhtml

If you need to refresh the __RequestVerificationToken on your page, this is stored as a hidden input field but can be generated on demand by loading /_layout/tokenhtml.

In my case, I apply the token to a form posting to ensure the request is valid.

    $("#antiforgerytoken").load("/_layout/tokenhtml", function() {
        console.log("tokenhtml loaded");
        $("form>input[name='__RequestVerificationToken']").val($("div#antiforgerytoken>input[name='__RequestVerificationToken']").val());
});