As is often the case with software troubleshooting, the story has some surprising twists. Do stay through to the end, to see how Itay figured out the problem, then came up with a solution. Itay kindly gave me permission to publish this tale, enjoy!
From: Itay Sent: Friday, March 01, 2013 7:05 PM To: Paul Braren Subject: Constantly high disk I/O with WSE 2012
...My box that runs WSE2012 at my home lab shows constantly high read/write rates on my iSCSI attached storage (Drive E:). I believe this is excessive by all means and I did not experience this with WHS2011.
I assume this is the work of the Volume Shadow Copy service, as I have the following output on the command line
C:\Windows\system32>vssadmin list shadowstorage
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.
Shadow Copy Storage association
For volume: (\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963})\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963}\
Shadow Copy Storage volume: (\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963})\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963}\
Used Shadow Copy Storage space: 528 KB (0%)
Allocated Shadow Copy Storage space: 32.0 MB (9%)
Maximum Shadow Copy Storage space: 35.0 MB (10%)
Shadow Copy Storage association
For volume: (E:)\?\Volume{e4d96878-e9d9-11e1-b542-000c29e24105}\
Shadow Copy Storage volume: (E:)\?\Volume{e4d96878-e9d9-11e1-b542-000c29e24105}\
Used Shadow Copy Storage space: 399 GB (7%)
Allocated Shadow Copy Storage space: 402 GB (7%)
Maximum Shadow Copy Storage space: 512 GB (10%)
Shadow Copy Storage association
For volume: (\?\Volume{d76a722a-9e07-458d-bce7-ac4d9083590b})\?\Volume{d76
a722a-9e07-458d-bce7-ac4d9083590b}\
Shadow Copy Storage volume: (\?\Volume{d76a722a-9e07-458d-bce7-ac4d9083590b})\?\Volume{d76a722a-9e07-458d-bce7-ac4d9083590b}\
Used Shadow Copy Storage space: 5.52 GB (3%)
Allocated Shadow Copy Storage space: 8.45 GB (4%)
Maximum Shadow Copy Storage space: UNBOUNDED (100%)
Shadow Copy Storage association
For volume: (\?\Volume{0553b479-5e80-11e2-93fa-005056ad4ce7})\?\Volume{055
3b479-5e80-11e2-93fa-005056ad4ce7}\
Shadow Copy Storage volume: (\?\Volume{0553b479-5e80-11e2-93fa-005056ad4ce7})\?\Volume{0553b479-5e80-11e2-93fa-005056ad4ce7}\
Used Shadow Copy Storage space: 0 bytes (0%)
Allocated Shadow Copy Storage space: 0 bytes (0%)
Maximum Shadow Copy Storage space: 5.00 GB (0%)
Shadow Copy Storage association
For volume: (C:)\?\Volume{156a45f8-55d4-11e2-93ed-806e6f6e6963}\Shadow Copy Storage volume: (C:)\?\Volume{156a45f8-55d4-11e2-93ed-806e6f6e6963}\
Used Shadow Copy Storage space: 12.7 GB (5%)
Allocated Shadow Copy Storage space: 13.9 GB (5%)
Maximum Shadow Copy Storage space: 25.0 GB (10%)
C:\Windows\system32>
Check this screenshot
The WSE2012 box is one of the VMs on my ESX box. also CPU usage is quite high at all times. Usually shows 40% on Task Manager on the VM and consumes about 6GHz on the ESX host. I have allocated this box 8GB of memory (all locked) and 4 vCPUs.
My ESX host has 32GB of RAM and sports an i7-2600 CPU which makes for 4x3.400GHz.
I figured it'll take some time for the server to "settle" with my storage but it's been this way for months - ever since I attached the server to the iSCSI storage. I'm quite worried about the high CPU usage (mostly because of the energy impact) and the high I/O that my [consumer grade disks] have to constantly suffer on the storage box...
Itay
From: Itay Sent: Friday, March 08, 2013 4:23 PM To: Paul Braren Subject: Re: Constantly high disk I/O with WSE 2012
Hi Paul,
Just a quick update on this matter - seems like I was able to solve it. Using Sysinternals Process Explorer I found out the rundll32 was running the following process - aepdu.dll,AePduRunUpdate
Apparently, I opted-in to the Customer Experience Improvement Program during the installation of WSE2012. As part of the CEIP process, a scheduled task called ProgramDataUpdater on the Task Scheduler Library/Microsoft/Windows/Application Experience was always running, consuming over 30% CPU usage.
HA! No more Microsoft customer improvement stuff from me!
I guess I was way off regarding my suspicion of VSS and iSCSI.
Regards,
--
Itay
From: Itay Sent: Saturday, March 09, 2013 6:45 AM To: Paul Braren Subject: Re: Constantly high disk I/O with WSE 2012
Resource monitor revealed to me that rundll32 process was consuming the CPU time, just as Process Explorer did. However, knowing that rundll32 was not the process itself but rather a method of launching a functionality inside a DLL file, I knew that in itself, the rundll32 process was not the culprit and I have to dig further. Showing the properties of the process in question using PE, I revealed the running function inside the aepdu.dll.
I later learned that the magnificent Task Manager on Windows 2012 / 8 provides functionality to show the command line that executed a process - by right-clicking the columns on the process tab and ticking the 'Command Line'. This easily reveals the DLL file in question and which function was called. The rest of the knowledge of what this function does can be checked on an app's documentation or your favorite search engine. :)
BTW, I had to forcefully disable this task since it insisted in running again after a while even though I disabled CIEP. I honestly don't know why they insist on running this when I said "no". I don't even know why windows was crawling my storage in the name of CIEP. This doesn't make a lot of sense.
You can freely use any of this info in your blog. I would have loved to write a post myself but I'm not sure I can do this well enough...
Have a wonderful weekend!
--
Itay
May 11 2013 corrected section appears below (added after original publication of this article, thank you Itay!):
From: Itay Sent: Saturday, May 11 06:10 PM To: Paul Braren Subject: Re: Constantly high disk I/O with WSE 2012Hi Paul, just a small addition -
As stated in one of my emails to you, unchecking the "Help us improve quality, reliablity [yada yada]" didn't solve the problem. What did solve it for me was disabling a scheduled task called ProgramDataUpdater, found under -
Task Scheduler > Task Scheduler Library > Microsoft > Windows > Application Experience.
--
Itay
So there you have it, turns out Disabling ProgramDataUpdater in the Task Scheduler saved Itay's day. You can see from the screenshot below it's related to the Microsoft Customer Experience Improvement Program.
Perhaps this tip will save you to, should you find yourself in a similar situation. Please comment below, letting us all know how it goes!
TinkerTry - PCs, EVs, home tech, efficiency and more, including virtualization. My opinions here, not my employer's.
After 6 successful years testing then shipping well over 1,000 Xeon D Bundles, Wiredzone had to stop selling them in mid-2021 due to cost, supply, and logistics challenges. So far, Xeon D-1700/2700 (Ice Lake D) was a minor refresh for 2023, with Xeon D-1800/2800 (Granite Rapids D) refresh looking a little better for 2024. More cores, but still just PCIe Gen4 and DDR4. Looking ahead, I'm glad it's Pat Gelsinger at Intel's helm. I'm also grateful to have had the honor of working at VMware when Pat was the CIO there. I'll leave it at that, given the whole Broadcom thing.
