[BUG] Remove-PnPListItem: Property Id was not yet loaded

[seriavio]

Reporting an Issue or Missing Feature

I'm experiencing sporadic issues with Remove-PnPListItem cmdlet (In scope of a big custom PS Module as follows: Remove-PnPListItem -List $script:ListName -Identity $Id -Batch $Script:UpdateBatch) in one of my scheduled scripts, where it produces an error message saying Property Id was not yet loaded.

What's strange is that this script is scheduled to run daily and is written in a way that if it fails to delete something from the list today, it will try to delete it again tomorrow. And I've seen it multiple times already where it was failing to delete one or more items from the list and eventually it was succeeding with removal on the next day. Initially, I thought it could have been an issue in the older module version (1.12) so I updated it to the latest one (2.5.0) and recently got this one again (once) but it worked fine next day.

Expected behavior

The cmdlet should not throw an exception

Actual behavior

Sometimes, the cmdlet produces an exception as follows:

PSMessageDetails      : 
Exception             : System.Management.Automation.PSInvalidOperationException: Property Id was not yet loaded
                           at PnP.PowerShell.Commands.Base.PnPConnectedCmdlet.ProcessRecord() in c:\build\src\Commands\Base\PnPConnectedCmdlet.cs:line 101
                           at PnP.PowerShell.Commands.PnPSharePointCmdlet.ProcessRecord() in c:\build\src\Commands\Base\PnPSharePointCmdlet.cs:line 131
                           at System.Management.Automation.CommandProcessor.ProcessRecord()
TargetObject          : 
CategoryInfo          : InvalidOperation: (:) [Remove-PnPListItem], PSInvalidOperationException
FullyQualifiedErrorId : InvalidOperation,PnP.PowerShell.Commands.Lists.RemoveListItem
ErrorDetails          : 
InvocationInfo        : System.Management.Automation.InvocationInfo
ScriptStackTrace      : <REDACTED - leads to a line in a custom module that contains Remove-PnPListItem>
PipelineIterationInfo : {}

Steps to reproduce behavior

Remove-PnPListItem -List $script:ListName -Identity $Id -Batch $Script:UpdateBatch, where $Id is taken from an object that was returned by Get-PnPListItem

What is the version of the Cmdlet module you are running?

2.5.0

Which operating system/environment are you running PnP PowerShell on?

• Windows
• Linux
• MacOS
• Azure Cloud Shell
• Azure Functions
• Other : please specify

[jackpoz]

Does the $Id parameter have a valid value at that point ? Just to be sure the exception is not happening when reading the $Id returned by Get-PnPListItem .

If you pass -ErrorAction:Stop as additional parameter to Remove-PnPListItem , it will show the actual callstack instead of the generic PnP.PowerShell.Commands.Base.PnPConnectedCmdlet.ProcessRecord() one.

[seriavio]

Does the $Id parameter have a valid value at that point ? Just to be sure the exception is not happening when reading the $Id returned by Get-PnPListItem .

Unfortunately, currently I'm not checking the presence of the value in $Id assuming that each List item has one.

If you pass -ErrorAction:Stop as additional parameter to Remove-PnPListItem , it will show the actual callstack instead of the generic PnP.PowerShell.Commands.Base.PnPConnectedCmdlet.ProcessRecord() one.
I'll update the code for Remove-PnPListItem to see if we can pull something additional out of it.

Do I understand it correctly that it seems like the value for the Id property of the ListItem may not be populated yet (as it is being done asynchronously under the hood) by the time it is passed into Remove-PnPListItem? If so, what's the best way to make sure that the value is ready? Shall I just try to parse it as integer when saving it into $Id or is there a better way? If I'll update the script to log the value, would it force the result to be loaded?

[jackpoz]

Chances are that Remove-PnPListItem under the hoods is doing some queries that don't include the Id field and then the code tries to read that and throws that error. That's why having the actual callstack would help as we could see the exact line where the Id is used.

[gautamdsheth]

Moving this to a discussion as it seems to be an issue with the custom script and not the cmdlet itself.

[seriavio]

@gautamdsheth, it seems to me like there's something wrong within the cmdlet as I was able to confirm that the proper value is provided. Can you please move it back to an issue?

[seriavio]

Hi @jackpoz, I've updated the script as suggested (added -ErrorAction:Stop to Remove-PnPListItem and added a verbose message to output the value in $Id) and recently have received another exception from the execution on past Sunday:

Exception             : PnP.Core.ClientError
                        PnP.Core.ClientException: Property Id was not yet loaded
                           at PnP.Core.Model.TransientObject.GetValue[T](String propertyName)
                           at PnP.Core.Model.SharePoint.ListItem.get_Id()
                           at PnP.Core.Model.SharePoint.ListItem.get_Key()
                           at PnP.Core.Model.BaseDataModelCollection`1.<>c__DisplayClass63_0.<DeleteByIdImplementationAsync>b__0(TModel i)
                           at System.Collections.Generic.List`1.FindIndex(Int32 startIndex, Int32 count, Predicate`1 match)
                           at PnP.Core.Model.BaseDataModelCollection`1.DeleteByIdImplementationAsync(Boolean viaBatchMethod, Int32 intId, String stringId, Guid guidId)
                           at PnP.Core.Model.BaseDataModelCollection`1.DeleteByIdBatchAsync(Batch batch, Int32 id)
                           at PnP.Core.Model.BaseDataModelCollection`1.DeleteByIdBatch(Batch batch, Int32 id)
                           at PnP.PowerShell.Commands.Lists.RemoveListItem.ExecuteCmdlet() in c:\build\src\Commands\Lists\RemoveListItem.cs:line 80
                           at PnP.PowerShell.Commands.Base.PnPConnectedCmdlet.ProcessRecord() in c:\build\src\Commands\Base\PnPConnectedCmdlet.cs:line 53
TargetObject          : 
CategoryInfo          : WriteError: (:) [Remove-PnPListItem], ClientException
FullyQualifiedErrorId : EXCEPTION,PnP.PowerShell.Commands.Lists.RemoveListItem
ErrorDetails          : Property Id was not yet loaded
InvocationInfo        : System.Management.Automation.InvocationInfo

It's worth to mention that according to the logs, the $Id variable had a valid integer value. Strangely, when the script rerun yesterday it was able to successfully remove this List Item.

Any idea what's going on here? In both cases, the batch was supposed to have just a single List Item.

[jackpoz]

Looking at the code, it seems to happen when calling Remove-PnPListItem with -Batch but without having loading the list item yet, with the issue happening in PnP Core. I will do some tests to prove this theory and let you know how they go.

[jackpoz]

To reproduce this issue, create a batch, add a list item to the batch and then remove a list item. This will throw the same error.

$batch = New-PnPBatch
Add-PnPListItem -List "Test simple list" -Batch $batch
Remove-PnPListItem -List "Test simple list" -Identity 1 -Batch $batch

It doesn't matter which id is passed to -Identity as the issue is that the new list item in the batch doesn't have an Id yet and that throws the exception.

@seriavio which operations do you do in the batch ?

[seriavio]

@jackpoz, generally, I'm doing combinations of Add/Set/Remove-PnPListItem against multiple list items in a batch (one of the operations per list item), but I'll have to go through the logs to find out what actions were done in those particular occasions when I was getting those exceptions

[seriavio]

@jackpoz, it is worth to mention that Remove operations are invoked against List items that exist in SharePoint, because my script reads the entire list (with the corresponding list item IDs) prior to removing some of those.

[jackpoz]

Oh the error doesn't happen because the item doesn't exist, you can see that in my case I don't even call Invoke-PnPBatch . The error happens because Add-PnPListItem adds to the batch an item without any Id, as it will be generated by SPO once the batch is submitted, and this item is checked by Remove-PnPListItem when deleting items. It doesn't matter which id you pass :)

The issue itself happens in PnP Core at https://github.com/pnp/pnpcore/blob/331b154a0696768b84acdf131dbc4ccc5001abc8/src/sdk/PnP.Core/Model/Base/BaseDataModelCollection.cs#L469 .

As a temporary workaround you could avoid mixing Remove-PnPListItem calls with Add ones, or append the Add calls after all Remove calls. This is just for you to be able to run your scripts in the time being.

[seriavio]

Ah, makes sense now. I've gone through all the logs for the executions where I had this problem and in all of the cases I indeed had at least one new List Item created. Would it work if I put Add calls into a dedicated batch from Set/Remove calls?

[jackpoz]

I would say so, but I haven't tested if Set causes issues. It shouldn't as it already needs an Id anyway so it should be fine.