Constantly high disk I/O with Windows Server 2012 Essentials, here's one possible reason, and a fix

Posted by Paul Braren on May 5 2013 in
  • ESXi
  • HomeServer
  • Windows
  • Virtualization
  • "A problem well stated is a problem half solved." - Charles Kettering, inventor, engineer

    Itay, a fellow enthusiast and TinkerTry follower, and currently the top commenter at TinkerTry (click Activity here), sent in this interesting story. Itay tells the tale of how he solved his mysterious high CPU utilization on Windows Server 2012 Essentials, using Resource Monitor to help with his investigation, see also:
    Using Task Manager and Resource Monitor to figure out what’s going behind the scenes in Windows Server 2012 / Windows 8.

    As is often the case with software troubleshooting, the story has some surprising twists. Do stay through to the end, to see how Itay figured out the problem, then came up with a solution. Itay kindly gave me permission to publish this tale, enjoy!

    From: Itay
    Sent: Friday, March 01, 2013 7:05 PM
    To: Paul Braren
    Subject: Constantly high disk I/O with WSE 2012

    ...My box that runs WSE2012 at my home lab shows constantly high read/write rates on my iSCSI attached storage (Drive E:). I believe this is excessive by all means and I did not experience this with WHS2011.

    I assume this is the work of the Volume Shadow Copy service, as I have the following output on the command line

    C:\Windows\system32>vssadmin list shadowstorage
    vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
    (C) Copyright 2001-2012 Microsoft Corp.

    Shadow Copy Storage association
    For volume: (\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963})\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963}\
    Shadow Copy Storage volume: (\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963})\?\Volume{156a45f7-55d4-11e2-93ed-806e6f6e6963}\
    Used Shadow Copy Storage space: 528 KB (0%)
    Allocated Shadow Copy Storage space: 32.0 MB (9%)
    Maximum Shadow Copy Storage space: 35.0 MB (10%)

    Shadow Copy Storage association
    For volume: (E:)\?\Volume{e4d96878-e9d9-11e1-b542-000c29e24105}\
    Shadow Copy Storage volume: (E:)\?\Volume{e4d96878-e9d9-11e1-b542-000c29e24105}\
    Used Shadow Copy Storage space: 399 GB (7%)
    Allocated Shadow Copy Storage space: 402 GB (7%)
    Maximum Shadow Copy Storage space: 512 GB (10%)

    Shadow Copy Storage association
    For volume: (\?\Volume{d76a722a-9e07-458d-bce7-ac4d9083590b})\?\Volume{d76
    a722a-9e07-458d-bce7-ac4d9083590b}\
    Shadow Copy Storage volume: (\?\Volume{d76a722a-9e07-458d-bce7-ac4d9083590b})\?\Volume{d76a722a-9e07-458d-bce7-ac4d9083590b}\
    Used Shadow Copy Storage space: 5.52 GB (3%)
    Allocated Shadow Copy Storage space: 8.45 GB (4%)
    Maximum Shadow Copy Storage space: UNBOUNDED (100%)

    Shadow Copy Storage association
    For volume: (\?\Volume{0553b479-5e80-11e2-93fa-005056ad4ce7})\?\Volume{055
    3b479-5e80-11e2-93fa-005056ad4ce7}\
    Shadow Copy Storage volume: (\?\Volume{0553b479-5e80-11e2-93fa-005056ad4ce7})\?\Volume{0553b479-5e80-11e2-93fa-005056ad4ce7}\
    Used Shadow Copy Storage space: 0 bytes (0%)
    Allocated Shadow Copy Storage space: 0 bytes (0%)
    Maximum Shadow Copy Storage space: 5.00 GB (0%)

    Shadow Copy Storage association
    For volume: (C:)\?\Volume{156a45f8-55d4-11e2-93ed-806e6f6e6963}\Shadow Copy Storage volume: (C:)\?\Volume{156a45f8-55d4-11e2-93ed-806e6f6e6963}\
    Used Shadow Copy Storage space: 12.7 GB (5%)
    Allocated Shadow Copy Storage space: 13.9 GB (5%)
    Maximum Shadow Copy Storage space: 25.0 GB (10%)

    C:\Windows\system32>

    Check this screenshot

    Resource-Monitor-high-disk-io-with-ws2012e

    The WSE2012 box is one of the VMs on my ESX box. also CPU usage is quite high at all times. Usually shows 40% on Task Manager on the VM and consumes about 6GHz on the ESX host. I have allocated this box 8GB of memory (all locked) and 4 vCPUs.

    My ESX host has 32GB of RAM and sports an i7-2600 CPU which makes for 4x3.400GHz.

    I figured it'll take some time for the server to "settle" with my storage but it's been this way for months - ever since I attached the server to the iSCSI storage. I'm quite worried about the high CPU usage (mostly because of the energy impact) and the high I/O that my [consumer grade disks] have to constantly suffer on the storage box...

    Itay


    From: Itay
    Sent: Friday, March 08, 2013 4:23 PM
    To: Paul Braren
    Subject: Re: Constantly high disk I/O with WSE 2012

    Hi Paul,

    Just a quick update on this matter - seems like I was able to solve it. Using Sysinternals Process Explorer I found out the rundll32 was running the following process - aepdu.dll,AePduRunUpdate

    Apparently, I opted-in to the Customer Experience Improvement Program during the installation of WSE2012. As part of the CEIP process, a scheduled task called ProgramDataUpdater on the Task Scheduler Library/Microsoft/Windows/Application Experience was always running, consuming over 30% CPU usage.

    HA! No more Microsoft customer improvement stuff from me!

    I guess I was way off regarding my suspicion of VSS and iSCSI.

    Regards,

    --
    Itay


    From: Itay
    Sent: Saturday, March 09, 2013 6:45 AM
    To: Paul Braren
    Subject: Re: Constantly high disk I/O with WSE 2012

    Resource monitor revealed to me that rundll32 process was consuming the CPU time, just as Process Explorer did. However, knowing that rundll32 was not the process itself but rather a method of launching a functionality inside a DLL file, I knew that in itself, the rundll32 process was not the culprit and I have to dig further. Showing the properties of the process in question using PE, I revealed the running function inside the aepdu.dll.

    I later learned that the magnificent Task Manager on Windows 2012 / 8 provides functionality to show the command line that executed a process - by right-clicking the columns on the process tab and ticking the 'Command Line'. This easily reveals the DLL file in question and which function was called. The rest of the knowledge of what this function does can be checked on an app's documentation or your favorite search engine. :)

    BTW, I had to forcefully disable this task since it insisted in running again after a while even though I disabled CIEP. I honestly don't know why they insist on running this when I said "no". I don't even know why windows was crawling my storage in the name of CIEP. This doesn't make a lot of sense.

    You can freely use any of this info in your blog. I would have loved to write a post myself but I'm not sure I can do this well enough...

    Have a wonderful weekend!

    --

    Itay


    May 11 2013 corrected section appears below (added after original publication of this article, thank you Itay!):

    From: Itay
    Sent: Saturday, May 11 06:10 PM
    To: Paul Braren
    Subject: Re: Constantly high disk I/O with WSE 2012Hi Paul, just a small addition -

    As stated in one of my emails to you, unchecking the "Help us improve quality, reliablity [yada yada]" didn't solve the problem. What did solve it for me was disabling a scheduled task called ProgramDataUpdater, found under -

    Task Scheduler > Task Scheduler Library > Microsoft > Windows > Application Experience.

    --

    Itay


    So there you have it, turns out Disabling ProgramDataUpdater in the Task Scheduler saved Itay's day. You can see from the screenshot below it's related to the Microsoft Customer Experience Improvement Program.

    Perhaps this tip will save you to, should you find yourself in a similar situation. Please comment below, letting us all know how it goes!

    ProgramDataUpdater