Response to Concerns Regarding eDellroot Certificate

SHARE:
Copied!

Today we became aware that a certificate (eDellRoot), installed by our Dell Foundation Services application on our PCs, unintentionally introduced a security vulnerability. The certificate was implemented as part of a support tool and intended to make it faster and easier for our customers to service their system. Customer security and privacy is a top concern and priority for Dell; we deeply regret that this has happened and are taking steps to address it.

The certificate is not malware or adware. Rather, it was intended to provide the system service tag to Dell online support allowing us to quickly identify the computer model, making it easier and faster to service our customers. This certificate is not being used to collect personal customer information. It’s also important to note that the certificate will not reinstall itself once it is properly removed using the recommended Dell process.

We have posted instructions to permanently remove the certificate from your system here. We will also push a software update starting on November 24 that will check for the certificate, and if detected remove it. Commercial customers who reimaged their systems without Dell Foundation Services are not affected by this issue. Additionally, the certificate will be removed from all Dell systems moving forward.

Your trust is important to us and we are actively working to address this issue. We thank customers such as Hanno Böck, Joe Nord and Kevin Hicks, aka rotorcowboy, who brought this to our attention. If you ever find a potential security vulnerability in any Dell product or software, we encourage you to visit this site to contact us immediately.

*****UPDATE 11/25/2015*****

Since Monday, our teams have been working hard to address the security issue caused by the eDellRoot certificate. When we became aware of the issue, we immediately dug into all our applications that get pre-loaded on our PCs. We can confirm we have found no other root certificates on our factory installed PC images. What we did find was that the Dell System Detect application and its DSDTestProvider root certificate had similar characteristics to eDellRoot. Thank you again Hanno Böck for calling this to our attention, as well as topg who commented below.

In the case of Dell System Detect, our customer opts to download the software proactively to interact with our support website so we can provide a better and more personalized experience. Like eDellRoot, the certificate in question was designed to make it faster and easier for our customers to get support. Unlike eDellRoot, this certificate is not related to software that was pre-installed on our systems.

The impact from Dell System Detect is limited to customers who used the “detect product” functionality on our support site between October 20 and November 24, 2015. The application in question was removed from the support site on November 24 and a replacement application without the certificate is now available. We are proactively pushing a software update to address the issue for our consumer systems. Our commercial customers can either manually remove the certification or use their system management tools like SCCM to do so (we will be providing instructions on this shortly). If you choose the manual option, we have updated instructions on our site http://www.dell.com/support/edellroot to permanently remove this certificate. Note, these are updated instructions for removing both eDellRoot and DSDTestProvider from any folders where they may be stored. If you previously uninstalled eDellRoot, we recommend you go through the process again to ensure a thorough sweep.

WIRED has noted that “security is far easier to promise than it is to achieve.” We know that your trust is harder to win than it is to lose. Once we know we have addressed these issues and our customers have what they need to ensure their systems are safe, we will provide an account of how the issues were introduced – not only for your information, but so we can improve our processes.

In today’s world of ever-increasing cybersecurity threats, we all need to be vigilant. And that is the promise that we make – Dell will remain ever vigilant against security threats and we will respond with the utmost speed and accuracy when we become aware of issues that can impact our customers.

*****UPDATE 11/30/2015*****

Today Microsoft released Security Advisory 3119884 that will place both the eDellRoot and DSDTestProvider certificates into the Windows Certified Trust List (CTL) as non-trusted certificates, so even if the certificates are installed, they cannot be used.  CTL updates are automatically pushed to both consumer and commercial Windows PCs. Most systems with Internet access should pick up the update within the next 24 hours. For more information, see the Security Advisory. This security step is in addition to actions already taken by Dell, as outlined in this post early last week, and by partners like Microsoft and Intel who added the patch to their anti-virus, anti-malware tools on November 25 to ensure both certificates were no longer usable.

Jeff Clarke, our vice chairman and president of Client Solutions, came into the studio to tape a short message to our customers and the security community to underscore our commitment to your security, to getting these issues resolved, and to being forthcoming with information as we have it.

With this latest important step by Microsoft and the proactive security updates driven by Dell complete, we are now turning our full attention to understanding what happened and how to prevent it in the future. We will be sure to update you here when we have more information to share.

Continue Reading
Would you like to read more like this?

Related Posts

Blogging Dead? Not at Dell!

Do blogs even matter now? Richard Macmanus, the founder of ReadWriteWeb, recently asked that question in his annual post to commemorate his first blog post 14 years ago. It’s been … READ MORE

Laura Pevehouse May 7, 2017
Click to Load More
  • Ivorygate

    @DanielHarsh

    That works, except the Word doc instructions Dell provides shows more steps.  Specifically, when the 'Dell Foundation Services' service first starts, it re-installs the cert.  So, first, one has to stop that service and delete the Dell.Foundation.Agent.Plugins.eDell.dll" file (or uninstall the software completely).

  • topg

    What about the equally problematic DSDTestProvider root certificate that seems to have been installed by Dell System Detect on my XPS 13? It has the same properties as eDellRoot & also includes a private key …

  • LezLezLez

    I use XP SP3 on a Dell Vostro 200 PC.  Can I assume this does not apply to me?

  • DELL-Laura

    @Louis D – Any commercial and consumer systems that received an update to Dell Foundation Services beginning in August 2015 were impacted. This update was removed on 11/23 and was replaced by a new update that will eliminate the root certificate from systems. Commercial customers who reimaged their systems without the Dell Foundation Services application were not impacted.

  • DELL-Laura

    @gerben_z – Sorry to hear of the issue. For technical assistance contact Dell Customer Service or Dell Technical Support (links above under "Content Reminder"), or if you are on Twitter you can reach out to @DellCaresPro for assistance.

  • DELL-Laura

    @topg – Wanted to let you know that our team is looking into this, as well, and we should have an update to address it very soon.

  • chilinux

    > "Today we became aware that a certificate (eDellRoot), installed by our Dell Foundation Services application on our PCs, unintentionally introduced a security vulnerability."

    Dell is selling equipment to customer accounts with full knowledge the customers have SAS 70, SSAE 16, SOC or HIPAA certification requirements.  Dell also has the ability to set the policies of SonicWall, EMC and VMware.  The amount of confidential information, both enterprise and personal, that are impacted by Dell's policies is staggering.  Therefore, it is extremely alarming that Dell is claiming incompetence is the reason for installing a rogue root certificate authority with a publically available private key.  If Dell has reached the level of incompetence that they truly do not understand the basic concepts of the impact the root certificate authority entry has on security then how can Dell be trusted to actually be following a security criteria required by its customers?

    Why doesn't Dell have a responsible computer security researcher on staff to monitor and sign off on change to the OS security layers?  What exactly does Dell SecureWorks do and what hope is there of them having positive results for customers if their own house is in a state of such disarray?

    At this point, Dell might want to consider paying Bruce Schneier or Daniel Bernstein whatever it takes to get them to join the company because I think that is the extent that Dell will need to go to recover its image.

    > "This certificate is not being used to collect personal customer information."

    FALSE FALSE FALSE FALSE FALSE!!!!

    This statement further illustrates the degree to which Dell does not understand the impact it has.

    The private key for this certificate is available publicly.  Dell does not have complete control over over the *use* of this certificate.  Instead, any third party can leverage this certificate to perform a man in the middle attack to view traffic from Dell customers.  Once the third party takes advantage of the vulnerability against Dell customers, it will be "this certificate" which is used to assist in the collection of personal customer information including the possibility of collecting information related to online banking and healthcare.  The fact that Dell itself isn't using the certificate to collect personal customer information does make your statement any less false.

    If Dell really understood the magnitude of liability Dell has created for itself and for its customers, a warning would be issued on Dell's primary web page instead of buried in a community forum.  Dell's handling of this issue, both on twitter and here, has been extremely upsetting given the amount of trust people have placed in Dell.

  • Dodss

    😀

  • DanielHarsh

    @Ivorygate

    Whoops! I missed that. That's crazy that it reinstalls constantly. Well – nevermind my seemingly temporary fix then; that's why I wanted someone to test it. Cheers 🙂

  • Vladimir Jirasek

    What this indicates is that the certificate was generated outside an HSM; or was exported from HSM  or was exported from it. Either would be very bad certificate management practice.

  • uControllerMan

    My Services does not contain an entry for "Dell Foundation Services," and my directory c:Program FilesDell does not contain the subdirectory "Dell Foundation Services." Does that mean I don't need this fix?

  • Louis D

    I read about this vulnerability in an article on BBC online.  Has not hit the US news?  

    Big issue:  no where (even in the linked instructions) does it say which PCs are affected?  The instructions imply that only Windows 8.1/10 machines are affected?  Desktops and laptops?

  • ameyer117

    Is there a silent switch for the automatic removal tool method? We would like to be able to push it out via SCCM.

    For example: eDellRootCertFix.exe /quiet

  • alunj

    For the sake of the rest of the industry, please publish an account of why this happened – the security world would like to understand the failures in process and thought that led to the creation of a relatively complex use of cryptographic components in a way that would ring alarm bells for most practiced cryptography users. If it "just seemed like a good idea" to some developer who didn't speak to a security professional, or it was reviewed by a security team, those are different problems, and which one it was will add to the corpus of knowledge used by application security professionals to improve security involvement in development process. Don't let this languish in the realm of "oops, we did a bad thing, and now we made it better".

    You have a responsibility to tell us what the bad thing was, and what steps you did to make it better – otherwise, the only responsible conclusion to draw is that you're idiots, you employ idiots, and you'll continue to employ idiots who do idiotic things.

    I'm fairly certain that's not the case, but I think the security community needs you to explain why that's not true.

  • DanielHarsh

    @ameyer117 Try this command for SCCM deployment:

    certutil.exe -delstore root "6b c5 7b 95 18 93 aa 97 4b 62 4a c0 88 fc 3b b6"

    That's the eDellRoot serial number.

  • donchichi

    after running the .exe I've noticed that eDellRoot still exist in the personal certificate store in local computer certificates. Why does it not delete the certificate?

    Also we'd like to deploy it via GPO, any chance of a .MSI?

  • gerben_z

    I ran the eDellRootCertFix.exe on my new (just last week) Dell OptiPlex 7020 running Win10.  Now I can't access my start menu.  I get a big green box that says "Critical Error.  Your start menu isn't working.  We'll try to fix it the next time you sign in."  Logging out and back in again does not fix the problem.  Rebooting my computer does not fix the problem.  Dell's RootCertFix program has made my computer inoperable.  Any suggestions on how to make my computer work again?

  • DNeuwir

    This is one of those reasons I always uninstall anything with "Dell" in the name from newly acquired systems, including "Dell Foundation Services".  Unfortunately, I don't know whether simply uninstalling Dell Foundation Services actually removes the offending certificate.

    The "fix-tool" released today appears to solve the problem, but for those of us that have to manage hundreds or even thousands of systems, it is "noisy" popping up "you're not affected" or "you've been fixed" boxes which are intrusive to the user.  So we can't script a rollout of that tool.  

    Can we get a silent version that simply writes log results to a file so IT administrators can simply retrieve the file to interpret the success/failure results?  Or even just a silent version?  This would be a 3-minute issue for me to fix by rolling out to the 100+ companies we support if it weren't for the non-silent nature of the current fix tool.

  • FDIPC

    As of 3 pm EST Nov. 25, the link to http://www.dell.com/…/SLN300321 is not working.

  • donchichi

    @Laura P. Thomas

    after running the .exe I've noticed that eDellRoot still exist in the personal certificate store in local computer certificates. Why does it not delete the certificate?

    Can you clarify?

  • Greg00

    I haven't received the software update which was stated as being released on November 24? I have clicked several times on check for updates but nothing?

  • hanson53

    This is only a problem for those running Windows OS. If you are running only Linux then it is not a problem.

  • Aquimoon

    There are possibilities to have this certificate installed on a Dell Inspiron 15, 3000 Series  3552,  with Ubuntu 14.04 preinstalled?

  • chilinux

    To start, "this certificate is not being used to collect personal customer information" is still not something Dell can ensure.  If I leave my car parked since August 2015 with the door unlocked and the keys in the ignition, I can't then claim "this car" could never have been used in a drive-by shooting.  The reason I can't make this claim is because by handing over control of the keys, I lost the ability to control how the car gets used. This is such a fundamental concept!  Yet Dell still continues to this day to claim control over how "this certificate" is used.

    > "And that is the promise that we make – Dell will remain ever vigilant against security threats and we will respond with the utmost speed and accuracy when we become aware of issues that can impact our customers."

    According to @hanno on Twitter, Dell's security team was notified regarding eDellRoot on November 9th and still hasn't responded to him.  Is what you mean that Dell will respond with the utmost speed when it become *publically* aware of issues?

    Also, what exactly is Dell willing to provide when it breaks its promises?  As pointed out in the Wired article, Dell has had a promise of of policies to keep security concerns similar to Superfish from happening.  Then Dell violates the same basic tenet of asymmetric cryptography: the private key must be kept private.  The industry also already learned that obfuscating the private key does not keep it private.  How did the industry learn this?  Superfish obfuscated the private key just like Dell did.

    Dell has committed the same basic violations of computer security as Superfish with the XPS 15 laptop which Dell advertised as having policies to keep that from happening.  So will there be refunds offered to people that bought the laptop on the fraudulent premise of promises in the Dell advertising?  I have yet to hear from anyone that there will be any such refund provided.  Even if someone no longer feels comfortable with using Dell at all for the tasks they purchased the equipment, there is no indication a refund will be given.  As such, the Dell "promise" rings just as empty as Dell's statement of what they will "ensure."  Dell states that it is making promises because Dell has no existing sales to loose in continuing to make fraudulent claims.

    While there will be a patch issued to address the two existing abuses of the Certificate Authority Root database, Dell's pre-load software and the software Dell encourages to be downloaded as part of its technical support both make changes to the file system ACLs, to the registry and other configuration changes which can impact security.  Who is auditing all of the changes?  Why shouldn't we expect eDellRoot to be just the rip of the iceberg?  Why haven't we heard from any member of the Dell SecureWorks team yet?

    The answer to all of this is: we can still trust Dell because Dell has empty promises backed by nothing.

  • C.M.R.

    This is not a complete fix.  If you have to reinstall your OS, it will also reinstall infected certificates.  Additionally, the certificates were found in the "User Certificates" and "Computer Certificates."  Your fix needs work.

  • steve.mills1

    Powershell Script, set it as a computer startup script. We have already made the move to Windows 10 but this should work from Windows 7 onwards.

    Stop-Service -Name 'Dell Foundation Services'

    Set-Service -Name 'Dell Foundation Services' -StartupType Disabled

    Remove-Item 'C:Program FilesDellDell Foundation ServicesDell.Foundation.Agent.Plugins.eDell.dll' -Force

    certutil.exe -delstore root "6b c5 7b 95 18 93 aa 97 4b 62 4a c0 88 fc 3b b6"

  • ekimchi

    I have read that this affects newer laptops and desktops. Can you be more specific as to what is meant by "newer", and what systems this affects? Also, does this affect servers?

    Our organization has many dell servers, desktops, and laptops and it would be very cumbersome to check every single one

  • huntnyc

    I ran the removal "eDellRootCertFix.exe" command from the Dos Prompt with different commands /silent or /quiet and there still a pop up message.  How can I deploy this solution silently?  This is a big issue that needs a WAN solution.  Please help!!!

  • DELL-Laura

    @ameyer117 @DNeuwir and @huntnyc

    For our commercial customers looking for a silent install via System Management tools, download and use the following patches:

    DSDTestProvider Certificate Removal: dellupdater.dell.com/…/DSDCertFixSilent.exe

    eDellRoot Certificate Removal:  dellupdater.dell.com/…/eDellRootCertSilent.exe

    Note: Both packages are silent and do not require additional switches. Return codes for the packages are:

                 0: Patch applied successfully or no issue detected.

                 1: Failure installing the patch.

  • DELL-Laura

    @ekimchi – Any commercial and consumer systems that received an update to Dell Foundation Services beginning in August 2015 were impacted.  This update was removed on 11/23 and was replaced by a new update that will eliminate the root certificate from systems. Commercial customers who reimaged their systems without the Dell Foundation Services application were not impacted.

  • DELL-Laura

    @LezLezLez – If you have used the “detect product” functionality on our support site between Oct. 20 and Nov. 24, 2015, you should follow the updated instructions to remove DSDTestProvider.

  • DELL-Laura

    uControllerMan – If you want to be certain, I suggest contacting our Technical Support Team.

  • DELL-Laura

    @donchichi and @C.M.R. – There are updated instructions for removing both eDellRoot and DSDTestProvider from any folders where they may be stored now available at http://www.dell.com/…/edellroot. If you previously uninstalled eDellRoot, we recommend you go through the process again to ensure a thorough sweep.

  • DELL-Laura

    @FDIPC – Apologies that the page was briefly unavailable earlier today.

  • DELL-Laura

    @Aquimoon – Any commercial and consumer systems that received an update to Dell Foundation Services beginning in August 2015 were impacted.  This update was removed on 11/23 and was replaced by a new update that will eliminate the root certificate from systems. Commercial customers who reimaged their systems without the Dell Foundation Services application were not impacted.

  • The__Machine

    It would be helpfulu if the article linked the actual KB and removal tool for this.  What a mess.

    Knowledge Base Article:

    http://www.dell.com/…/SLN300321

    Removal Tool:

    dellupdater.dell.com/…/DellCertFix.exe

  • MikeRenna

    Sorry, but I don't even know what Foundation Services is for.  I haven't been able to find anything about it on the web.  I just uninstall it. But would love to be able to explain it / protected workspace? and other dell installed things to clients.

    Is there somewhere on the dell website that explains all these items?

  • DELL-Laura

    @MikeRenna – Dell Support Foundation is an application, designed to provide a better, faster and easier customer support experience by allowing us to quickly identify the customer's computer model.

  • CostiTechc

    Can we have a msi file? I've tested the 2 executable files that you provide us ( DSDCertFixSilent.exe and eDellRootCertSilent.exe) and they're not silent at all. If you have the certificate, you'll be prompted if you're sure that you want t certificate.

  • bill mpls

    Is there an easy way to check if the eDell or DSDTestProvider certificates are installed?  There are a lot of cert directories that I don't want to stare at on the screen.

  • ajmelmv

    "Dell will also push a software update starting November 24, that will check for the certificate, and if detected will remove it"

    where I can get the software update mentioned above (not the removal tool)?

  • Ivorygate

    Can anyone post the DSDTestProvider cert serial number (for use with certutil.exe)?

  • IanFM

    Where's the fix for the latest reported security issue on Dell systems?

  • DELL-Laura

    @IanFM – If you choose the manual option, we have updated instructions on our site http://www.dell.com/…/edellroot to permanently remove this certificate. Note, these are updated instructions for removing both eDellRoot and DSDTestProvider from any folders where they may be stored.

  • DELL-Laura

    ajmelmv – The software update we are proactively pushing to address the issue for our consumer systems will take several days to reach everyone. If you wish to take action faster, the manual option instructions are at http://www.dell.com/…/edellroot.

  • DELL-Laura

    bill mpls – The tool we provide at http://www.dell.com/…/edellroot does both — let’s you know if you don’t have it, and if present removes it.

  • BernardAncie

    Nice post, I learn a lot about eDellroot, I'll have to practice it in my society soon

  • allheart55

    Why would Dell post  the instructions in .docx format? You will need MS Word 2007 or above to even read the instructions.;

  • Aane

    Laura, This morning McAfee flagged the following as a "potentially unwanted program" on my Dell XPS desktop –  Program Name: "DeCertA" quarantined from C:Program FilesDellDell Foundation ServicesDell Foundation Agent Plugins,eDell.dll  Is this the file being sent out by Dell to correct the eDellroot certifiate issue and if so, should I instruct McAfee to allow it to run?  Or is the problem file and should I iinstruct to McAfee to remove it?  Thank you.

  • tpaljr

    I agree with CostiTech:

    "Can we have a msi file? I've tested the 2 executable files that you provide us ( DSDCertFixSilent.exe and eDellRootCertSilent.exe) and they're not silent at all. If you have the certificate, you'll be prompted if you're sure that you want t certificate."

    We need silent switches, there is a popup to let the user know if there is or is not a problem.

    Also a log switch because a lot of the smaller companies may or may not have SCCM to deploy the fix.

  • paul_sibert

    Just to reinforce the request to provide silent removal tools for SMB's that do not use System Center. This is a must have feature for clients and we need it asap.

  • DELL-Laura

    @Aane – We have worked with Intel security to embed our patch into their anti-virus software, so yes, you can allow McAfee to remove the file.

  • DELL-Laura

    @allheart55 – For the sake of speed, the very first set of instructions were posted in .docx, but they were soon re-created in web format at http://www.dell.com/…/edellroot.

  • clr512atx

    Please make the certificates available for download; this entire issue would be easily mitigated via GPO and moving the certs in question to the "Untrusted Certificates" folder.

  • MichaelD4012

    I had a problem with running the following removal tool on 32-bit Windows 7 Enterprise computersl:

    eDellRoot Certificate Removal:  dellupdater.dell.com/…/eDellRootCertSilent.exe

    It runs fine on 64-bit computers. When run on a 32-bit machine, I get the following error: The version of this file is not compatible with the version of Windows you're running……….

    Can you provide a 32-bit version of the silent eDellRoot Certificate Removal tool?

    The DSDTestProvider Certificate Removal tool works fine on both 32- and 64-bit machines.

  • DELL-Laura

    @MichaelD4012  and @mccolganm – You'll be happy to know we have updated the link to include a removal tool that will identify and remove on either 32 or 64 bit. And, you might also be interested in learning more here about what Windows 10 can offer your office. 🙂 en.community.dell.com/…/dell-helping-windows-10-get-down-to-business

  • andrey42

    The system quality is defined not only by the errors/flaws discovered, everyone can make them, but by the honest & correct & quick reaction on them. So far I admire reaction of Dell Security team on the issue, way to go.

  • DELL-Laura

    @andrey42 – Thank you!

  • mccolganm

    My organization uses a K1000 Management system and i am interested in using the Certificate removal tool (DellCertFix.exe) with a script on computers we have ID's as having affected versions of Dell Foundation Services and or Dell System Detect onboard.  the version of DellCertFix.exe that is published seems to be designed for 64bit Windows only.  Can a 32bit version compatible with Windows 7 Pro SP1 be created?  I have potentially 60+ computers that are affected by these bad certificates.  Also is there a way to run the DellCertFix.exe silently so no message are displayed on screen?

    Many Thanks

  • Celine03

    Thanks for information

  • cathaleen

    Despite his polished public speaking skills, I find his recorded video message to be insincere and quite frankly insulting.  He says "You have my commitment that we will be transparent and forthcoming with information as quickly as we have all of the details…" but it is now December 4th, and I have yet to be contacted directly by Dell regarding this pre-installed security violation that was learned about in the news and tested using third party tools that Dell did not even bother to publish.  As a Securities Exchange Commission regulated business we take security seriously and Dell's sloppy handing of this error has made our compliance office question if we should buy from Dell again.

  • cathaleen

    Here is a clear and simple test and repair of this Dell pre-installed security flaw.

    Dells instructions are horrible and do not include a test to see if your system is infected with this flaw or repaired.

    This flaw does impact desktops as well as laptops.  I have found in on XPS 8900 Dekstops…

    http://tinyurl.com/qgj4qoz

  • delver

    There has been no response to Aquimoon's post on the 25 Nov 2015:

    There are possibilities to have this certificate installed on a Dell Inspiron 15, 3000 Series  3552,  with Ubuntu 14.04 preinstalled?

    I have also bought a Inspiron 15 3000 Series (3551) at the end of July 2015, with Ubuntu 14.04 pre-installed, and would like to know if my laptop can be affected by this issue.

    Thank you.

  • DELL-Laura

    @delver – Apologies I missed answering @Aquimoon's question about this earlier. I have now confirmed that neither eDellRoot, nor DSDTestProvider, impact users of Linux.

  • cathaleen

    What a disgusting attempt at damage control.  Publishing a statement that is so unclear and directions on how to check for and remedy the problem that only makes a bad situation worse.  And on top of that invite people to comment only to have your censor the meaningful comments that may actually help the customers that are victims of this blunder.  Shame on you Dell.  You are showing all the signs of a company more concerned about public image and self preservation than you are about caring for customers and earning their respect, respect that garners loyalty.  You best days are most certainly behind you.

  • DELL-Laura

    @cathaleen – Your comments on December 4th were not censored, but are published above. On December 7, I did indeed reach out privately to see if you wished to send me your company name so that I could locate your account representative and have them reach out directly to assist your organization with the eDellRoot issue. You declined that offer on December 10, so I regret to hear that you remain unsatisfied with our response. I do hope that you will give us another opportunity to assist you as we start a new year.

  • Bob Foster

    Great post!!!

    Thanks to author…

  • samerfifa

    thank you for this post

    good luck