Hardware DEP has a backdoor

Introduction :
Hardware DEP (Data Execution Prevention) is a feature that protects Windows XP SP2 from buffer overruns. When a compatible CPU is present, Windows XP SP2 DEP interface offers 2 options. The default one protects only the core Windows components and the ones manually added by the user (OPTIN). The most secure one protects all the programs on your systems except the ones manually added by the user (OPTOUT).
For more informations on how to activate hardware DEP, visit my DEP to keep buffer overrun at bay blog post.

Investigating hardware DEP :
I activated DEP on my 2 compatible PCs in mid 2006, it was protecting my PCs in the background so I forgot about it. Recently the security guru Steve Gibson of grc.com in it's Security Now podcasts mentioned that IrfanView refused to run in ALWAYSON DEP mode. While it would run perfectly in in OPTOUT mode. Theoretically OPTOUT with an empty exclusion list should behave exactly like ALWAYSON. The only possible explainations were either IrfanView was doing a nasty trick to evade DEP or Microsoft lied about the definition of OPTOUT.

I ran a bunch of queries on Google to find more information on the subject... nothing really impressive. Then a contributor on the GRC security newsgroup posted about packed executable disabling DEP. Back in late 2005, when the WMF exploit surfaced Kaspersky pointed out that IrfanView and XnView were running without protection even in OPTOUT mode. Leaving their users with their pants on the ankles. It clearly stated:

However, when testing the latest builds of third party image viewers like Irfanview and XnView HW DEP didn’t prevent exploitation, even with HW DEP enabled for all programs. This is because both Irfanview and XnView are packed with ASPack and Windows disables HW DEP for ASPack packed files.
— Roel Schouwenberg, Kaspersky

ASPack allows to compress executable files (EXE, DLL, OCX,...) so their footprint on the hard disk is smaller. The file is decompressed on the fly at runtime. The CPUs are faster than hard disk so in the end running these apps should be faster. I never used DriveSpace/DoubleSpace (back in the DOS days) or RAID0... mainly for low reliability and lack of decent performance boost. So I have a hard time believing that saving 2MB on my laptop 160GB hard disk is a useful trick. Especially when it comes at the expense of DEP protection.

So this new keyword unleashed the doors of very interesting stuff on Google (query ASPack DEP Kaspersky). I quickly found out that IrfanView was not using any trick to evade DEP... Microsoft just coded a backdoor used only in OPTOUT. Bascially Microsoft checks the executable header for a section matching one of the 3 strings. If one these strings is found, DEP will be turned OFF for this application by windows. To find these strings all you have to do is use a hex editor to open ntdll.dll found in your sytem32 directory.

The 3 backdoor strings in ntdll.dll :

By order of appearance, the strings are:

  • aspack : from ASPack. Common programs that use ASPack are the 2 image viewers IrfanView and XnView... but also the web browser Opera.
  • pcle : (source not yet identified) I put my 0.02€ on Pinnacle System for their long history of crashing apps and their NASDAQ ticker.
  • sforce : from Star-Force the highly unpopular game/app copy protection. The list of current Star-Force protected games can be found on the Boycott StarForce website.

The easiest way to identify the DEP status of an application is to run Process Explorer. In order to display the DEP status you'll need to go to the View menu, click on Select columns and then tick the "DEP status" checkbox.

Process Explorer window :

The compressed programs appear in purple. You can see that packed apps are common but only the ASPack ones disable DEP. Both IrfanView and Opera run with DEP off. In this case, I had Skype (proprietary packer???) and FastStone Capture (UPX). I also use uTorrent which uses PECompact.

From my point of view the worst case is Opera. By definition a web browser is on the front line it has to be backed up by DEP. I would barely tolerate that a local widget that displays an analog clock disables DEP... but a browser is way beyond my threshold. If the widget requires DEP to be turned off to run, I want to be in charge of doing so. Just like I want to control the opened ports in my routers and therefore turn off UPnP, I want to have the control of my hardware DEP protection.

Conclusion :
I want to believe that Microsoft will eventually release a properly named OPTOUT mode. If the current backdoored version was to stay it should be named with clear unsecured in its description. If some spin doctor wanted to pull "Hybrid" out of his sleeves he should take this PR stunt and *censored*. On the programs side, it would be a good time for the ASPack users to switch to UPX which is open source and DEP compliant. Or simply dump packed executables for good. I don't care if my web browser takes 200 extra milliseconds to launch. But I do care to have DEP blindly protecting my programs from undisclosed buffer overruns.

Some will say that ALWAYSON locks this backdoor... but in my case being able to OPTOUT one program in beta stage is priceless. Therefore I'd rather replace the strings in ntdll.dll by some random gibberish so that no other app can slip thru my fingers. Last year, for speed reasons, I switched from IrfanView to FastStone Image Viewer. I'll now use Opera only as the last resort and for testing purpose on known websites.

Edits :

    • IrfanView 4.x+ now uses UPX packer so it's fully hardware DEP compliant.
    • A bunch of PhotoShop compatible plugins require switching DEP off for the hosting app (PhotoShop, Paint Shop Pro or others).
    • These plugins don't trigger a DEP interception, they just fail to properly run and crash the host application. Here are some plugins that crash: Xero Graphics, PhotoWiz and plugins built using FilterMeister. The FilterMeister devs are aware of the issue and are working on a solution.
    • Opera from version 9.50 alpha and up is now DEP compliant (FYI: it uses UPX to pack its executable).

    Useful links :

    DEP to keep buffer overrun at bay

    Since the service pack 2 (august 2004), Windows XP supports the hardware DEP (Data Execution Prevention). This feature is hardwired in the modern CPUs, it allows to intercept and lock programs before buffer overruns lead to issues. It requires a compatible BIOS, CPU and operating system.

    By default the DEP is only applied only to the Windows core programs and services. The main reason behind this default behavior is that back in august 2004 a bunch of legit apps were locked by hardware DEP (Adobe Reader 6, Jasc Paint Shop Pro 8, Pinnacle Systems Studio 9,...). These programs were patched out of the bugs / performance tricks to be hardware DEP compliant.

    If your gear supports hardware DEP, it's highly recommended to activate it. To do so, you'll need:

    • Open the system applet in the control panel.
    • Select the advanced tab then click on parameters in the Performance section.
    • Select the Data Execution Prevention tab.
    • Select the Turn on DEP for all programs and services except those I select.
    • Click on Ok then reboot to apply the changes.

    DEP options:


    From now on your PC is protected against most buffers overruns. If a program tries to execute code in a forbidden area, the following window will appear.

    DEP warning:


    In this example, Windows Explorer is not behaving properly... If a legit program is stopped by DEP, you still have the option to add it to the Opt-out list. But you'll have to understand that disabling DEP leaves a breach in the security wall created by DEP.
    In case you don't know if your PC supports hardware DEP, Steve Gibson from grc.com made a program called Securable. Securable checks your system for the following security features:

    • Hardware DEP: for Windows XP SP2 32/64 bits and up.
    • 64 bits: for the Kernel Patch Protection of Windows Vista 64 bits.
    • Virtualization: for the future HyperVisor of Windows Vista 64 bits.

    Securable ran on my laptop:


    Additional informations: for the advanced users
    The DEP mode is a parameter of the operating system in the boot.ini file.

    [boot loader]
    operating systems]
    multi(0)disk(0)rdisk(0)partition(1)\\WINDOWS=\"Microsoft Windows XP Professionnel\" /fastdetect /NoExecute=OptOut

    The possible values for the NoExecute parameter are:

    • ALWAYSOFF turns DEP off.
    • OPTIN activates DEP only for the main Windows programs and services and those added to the list. (Windows XP SP2 default mode)
    • OPTOUT activates DEP for all the programs et services except those added to the list. (best mode available from the DEP options window)
    • ALWAYSON activates DEP for all programs and services. (only available by manually editing the boot.ini file)

    Remark: If a bug in a critical driver triggers a DEP lock in OPTOUT or ALWAYSON modes it might prevent your PC from booting. In that case you'll have to access the recovery console and edit the boot.ini file to switch to OPTIN mode.