Failed to load virtual machine templates for subscription

We encountered problems with Windows Azure Pack and tenants loading subscriptions / templates.
And we are not the only one, in the Technet forum there is also a busy thread about it: here

THIS ISSUE HAS BEEN FIXED IN VMM, SPF, WAP UR4

THIS ISSUE HAS BEEN FIXED IN VMM, SPF, WAP UR4

THIS ISSUE HAS BEEN FIXED IN VMM, SPF, WAP UR4

————————————————————————————

Also have seen a blog by Markus Lassfolk at http://www.isolation.se/azure-pack-failed-to-load-virtual-machine-templates-for-subscription/ going by on Twitter with a workaround for this problem..

Time to investigate!

The error we’re getting in WAP:
subfail

On the Tenant portal server we are seeing this error in the eventviewer:
subfail_event1

Failed to load list of VM Templates for SubscriptionId: '1b045552-b48b-4081-a51b-087579feb9ad', Exception: 'DataServiceQueryException: An error occurred while processing this request.

And on the SPF server in the eventviewer we even see 3 errors:

subfail_event2

Operation manager plugin method 'GetAssociatedInstances' for resource name 'VMM.VMTemplate' failed with error messsage 'Exception getting "VirtualHardDisks": "You do not have permission to access one or more of the objects required by this operation.

subfail_event3

Custom module operation invocation failed exception.
Module name: Microsoft.SystemCenter.Foundation.Psws.Spf.SpfOperationManager
Method Name: GetAssociatedInstances. Exception message: Exception
Message = Exception getting "VirtualHardDisks": "You do not have permission to access one or more of the objects required by this operation.

subfail_event4

Web Service has got a callback from OData framework about an error.
Exception message = Invoking method GetAssociatedInstances of type Microsoft.SystemCenter.Foundation.Psws.Spf.SpfOperationManager failed. Cause of the problem: Exception getting "VirtualHardDisks": "You do not have permission to access one or more of the objects required by this operation.

This error seems to occurs every time the VMM applicationpool on the SPF server is recycled.
After a change in the plan, or a resync of the plan everything goes back to “normal”.
So, we can trigger this behavior with a applicationpool recycle, time for some VMM Debug Tracing

In the VMM Debug trace I see a error that matches the WAP and SPF errors.
But more important, its telling me which object causes the problems

1
2
3
4
5
6
7
8
9
10
11
12
13
Microsoft.VirtualManager.Utils.ObjectNotAccessibleException: <strong>You do not have permission to access one or more of the objects required by this operation.</strong>
Contact the Virtual Machine Manager administrator to obtain the appropriate permissions.
   at Microsoft.VirtualManager.Engine.Remoting.ClientConnection.FetchObjectInternal(CarmineObjectType type, Guid id, ConnectionProperties conn)
   at Microsoft.VirtualManager.Engine.Remoting.ClientConnection.FetchObject(CarmineObjectType type, Guid id, OnBehalfOf obo)
   at SyncInvokeFetchObject(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
*** Carmine error was: ObjectsInaccessible (11418)
[s#18782a61] ClientConnection.FetchObject(this: (ClientConnection#14e), type: CarmineObjectType.VirtualHardDisk, id: (guid) 9c7bce89-5aa1-4452-bf63-370fccd54304, obo: (OnBehalfOf#addbc3a) Microsoft.VirtualManager.Remoting.OnBehalfOf) in EventingImpl.cs:line 42
Full call stack from when the exception was thrown:   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

So I obviously ran

Get-SCVirtualHardDisk -ID 9c7bce89-5aa1-4452-bf63-370fccd54304

And quickly came to the discovery this is a VirtualHardDisk that my VM Template is dependent on.
I opened the VirtualHardDisk properties in VMM and these 2 settings caught my eye:

subfail5

After changing the settings of the VirtualHardDisk to settings you would expect…

subfail6

Or with Powershell:

$libraryObject = Get-SCVirtualHardDisk -ID "9c7bce89-5aa1-4452-bf63-370fccd54304"
$os = Get-SCOperatingSystem -Name "64-bit edition of Windows Server 2012 Datacenter"
Set-SCVirtualHardDisk -VirtualHardDisk $libraryObject -VirtualizationPlatform "Hyperv" -Operatingsystem $OS

The error went away, after a VMM Applicationpool recycle on the SPF server.

And I can now recycle as much as I want.. did not see this error anymore.
It seems like is has something to do with the VirtualizationPlatform setting, similar (or maybe connected?) to the “Cloud Capability Profile” under VM Templates.

I dont know for sure this is the long-term solution, please leave a comment if the above did not fix your problems.
Hope it helps, Otherwise we’ll have to stick with the resync workarround….

Update:

Argh! the problem still persist…. so did some more troubleshooting 🙂
I think the solution was in plain sight all the time, but went to much on a deepdive with troubleshooting.

After setting the VM Template Capability Profile to “Hyper-V” and also the dependent VirtualHardDisk I got the following error:

1
2
3
4
5
6
7
8
9
10
11
12
13
Microsoft.VirtualManager.Utils.ObjectNotAccessibleException: You do not have permission to access one or more of the objects required by this operation.
Contact the Virtual Machine Manager administrator to obtain the appropriate permissions.
   at Microsoft.VirtualManager.Engine.Remoting.ClientConnection.FetchObjectInternal(CarmineObjectType type, Guid id, ConnectionProperties conn)
   at Microsoft.VirtualManager.Engine.Remoting.ClientConnection.FetchObject(CarmineObjectType type, Guid id, OnBehalfOf obo)
   at SyncInvokeFetchObject(Object , Object[] , Object[] )
   at System.ServiceModel.Dispatcher.SyncMethodInvoker.Invoke(Object instance, Object[] inputs, Object[]& outputs)
   at System.ServiceModel.Dispatcher.DispatchOperationRuntime.InvokeBegin(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage5(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(MessageRpc& rpc)
   at System.ServiceModel.Dispatcher.MessageRpc.Process(Boolean isOperationContextSet)
*** Carmine error was: ObjectsInaccessible (11418)
[s#1c9257f0] ClientConnection.FetchObject(this: (ClientConnection#1fb), type: CarmineObjectType.CapabilityProfile, id: (guid) ca240bb5-013a-4f58-9afe-be10f56bb87a, obo: (OnBehalfOf#addbc3a) Microsoft.VirtualManager.Remoting.OnBehalfOf) in EventingImpl.cs:line 42
Full call stack from when the exception was thrown:   at System.Environment.GetStackTrace(Exception e, Boolean needFileInfo)

The error is not mentioning the VirtualHardDisk anymore, but a CapabilityProfile Object.
*BINGO* The VM Template has CapabilityProfile set on HyperV, the VirtualHardDisk has a CapabilityProfile set to “HyperV” and the Cloud that is attached to the plan also has the CapabilityProfile set to Hyper-V. And still I got a error on the CapabilityProfile “Hyper-V”.

Hans Vredevoort pointed me to the fact that for VM Roles you may not set the CapabilityProfile on the cloud your deploying to.
The next step is to uncheck the CapabilityProfile on the VM Template, VirtualHardDisk and Cloud.
I haven’t seen the error since, but said this before 🙂
To be continued..

Update 1th december 2014:
I’ve just got confirmation from a Program Manager of Windows Azure Pack this is fixed in UR4.
All components needs to be updated; VMM, SPF, WAP

2 thoughts to “Failed to load virtual machine templates for subscription”

  1. i was able to fix it by doing the opposite way. Setting and removing the virtulization plattform did not help.

    Adding the Hyper-V capability Profile to the VM Template seems to fix the error..

    1. Hi Mike,

      Thanks for the feedback!

      Yes, so it seems the problem is fixed when all the capability profiles are set the same on VM Template, Cloud, VirtualHardDisk.
      The recommendations for Windows Azure Pack is not to set a capability profile.. did you uncheck it on all three?

Leave a Reply

Your email address will not be published. Required fields are marked *