Recovering your Dedeuped Files on Windows 10

Were you one of those who installed the server 2012 binaries into your Windows 8.1 to enable Disk Deduplication? Did you turn on dedupe on all of your drives, saving hundreds of gigs of storage space, then upgrade to Windows 10?

Upon boot, were you greeted with frequent ‘the machine cannot access the file’ errors? If so, then this is the guide for you!

This fixes the error 0x80070780: The file cannot be accessed by the system

What happened to my stuff? Did I lose it all?

NO!  You did not lose your files.  What happened when you ran deduplication on your files, is that Windows gradually scrubbed all of the common elements out of many files in order to compress them, much like what happens when you put files into a .zip or .rar archive.

All of your files still live on that disk or volume, happily compressed in their Optimized state and sitting snuggled up to their brethren.  However, until you ‘Unoptimize’ them, or connect the disk to a system with Windows Server Deduplication enabled, you cannot access any items bigger than 32KB (the minimum size for block level dedupe).

You have two methods to recover your files

These are both very similar, in the outcome, only the beginning differs.

Method 1 – Move the disk to another computer

Physically take the disks that are hosed and attach them to another system with ‘Data Deduplication’ enabled, could be any OS which supports it, even 8.1.  Once you’ve done this, proceed.

Method 2 – use Hyper-V to virtually move the disk

This is the method I used.

    1. Stand up another VM (in my case Server 2016 Tech Preview, though you could use 2012, 2012 R2, or even Windows 8.1) and installed the Storage -> Data Deduplication role on it.  Make sure to add one or more SCSI Hard Drive Controllers, one for each drive you need to ‘redupe’.Dedupe/redupe is incredibly IO and memory intensive.  Bump up the VMs stats to super level if you want this to be done quickly.
    2. Take the disks you need to ‘redupe’ offline.  I then went to my affected volumes in Disk Manager on my local machine and took the disks offline.  This is necessary in order to make this physical drive a ‘Passthru’ disk, which essentially connects the spinning disk to your VM.

      Once the disk is offline, we can attach it to a VM.  Do NOT accidentally format your drive here.
      Once the disk is offline, we can attach it to a VM. Do NOT accidentally format your drive here.
    3. In Hyper-V, attach the disk to your VM, by attaching it as a physical disk.redupe2
    4. Go into your VM and run Disk Management, and bring the disk online.  Do NOT accidentally format your disk here either.

      If you format your drive here, you'll now have even more problems.
      If you format your drive here, you’ll now have even more problems.
    5. Once the disks are all online in your VM, launch PowerShell as an admin and then run Get-DedupStatus to verify that all of your drives are listed.

      redupe4
      Say goodbye to all of this optimized space…
    6. For each drive, run the following Cmdlet.
      Start-DedupJob -Volume "D:" -Type Unoptimization

      If you only have one drive, run the cmdlet with -Wait so you can see how incredibly long this is going to take.

    7. Wait forever.  It takes a REALLY, REALLY REALLY really super long time to ‘redupe’ your files.
      It's been stuck at 0% for ten minutes!
      It’s been stuck at 0% for ten minutes!

      If you want to know how long it will take if you didn’t run this in -Wait mode, you can run

      Get-DedupJob

      to see how long it will take, but don’t sit around.  It will take forever.

    8. Send Microsoft an e-mail or vote for this issue on User Voice.  I’ve been to Redmond, and I can confidently say that they really do pay attention to user feedback on issues like this.  Seriously, Windows Desktop Enterprise and Pro could use some more distinguishing features.  Vote this up and hopefully we’ll get Dedupe as a standard feature with Threshold in October.

Big thanks to fellow Microsoft MVP Mike F Robbins for his tweets and post on the matter, and to Microsoft Storage Program Manager Ran Kalanch for helping me to understand exactly what I’d done to my drives at midnight of the Windows 10 release.
 

Advertisements

39 thoughts on “Recovering your Dedeuped Files on Windows 10

  1. Dirk Becker August 2, 2015 / 8:05 am

    After Unoptimization run chkdsk on your drive!
    chkdsk.exe D: /F /R /X /B /offlinescanandfix
    (where D: is the drive to check)

    I had some corupted files with 0 Byte size which were repaired this way.

    And for the status of the process, after 4 hours I had 0% .. 5 Minutes later it was 56%. So do not trust the status from Get-DedupJob, just check from time to time.

    • Dirk Becker August 2, 2015 / 2:09 pm

      OK, here ist the correct way to deactivate deduplication:

      First disable Dedup:
      Disable-DedupVolume D:

      Undupe:
      Start-DedupJob -Volume D: -Type Unoptimization

      Clean up:
      Start-DedupJob -Volume D: -Type GarbageCollection

      finally check drive (optional)
      chkdsk.exe D: /F /R /X /B /offlinescanandfix

      Good Luck 🙂

      • FoxDeploy August 2, 2015 / 8:05 pm

        There’s actually no reason to run the Garbage Collection job as far as I can tell. When you’ve finished the Unoptimization job, the dedup service removes all the metadata and gunk that a GarbageCollection would have removed.

        I personally haven’t needed to do the Chkdsk method, but YMMV.

  2. matthewf01 August 5, 2015 / 10:31 pm

    Wow, didn’t expect to find a post on this!!!
    I have a desktop at home, which was running Windows 8.1, and using Storage Spaces to carve up a whole bunch of 3TB disks into several virtual volumes, backing everything from Plex multimedia to VMs. On one of my volumes, for backups and ISOs, I had found the trick for enabling dedupe on Win8. Was getting great dedupe results!
    Upgraded to Windows 10 last weekend, today noticing I can’t mount any of my ISOs — strange. Error always the same. Also can’t copy out to other disks – Teracopy reporting the same error. Searched it, and here we are!

    So, Storage Spaces obviously kind of complicates things, due to the layers. Methods 1 and 2 of your post would require me to pull my disks out of the Storage Spaces array, then take a chance by expecting that the ‘fixer’ machine would be able to mount the whole array properly, and the subsequent provisioned volumes.

    Is Dirk Becker’s comment above an _alternative_ to your methods posted above? I’m wondering if in my scenario it may be the cleaner solution, as I can target only the virtual volumes on which I enabled dedupe in the first place, and I don’t have to deal with detaching my entire Storage Spaces pool (terrifying!).

    Thanks! Will follow your blog!

  3. Ben August 12, 2015 / 11:08 am

    I’m hoping there’s an alternate method as well. I had DeDupe turned on in 8.1 when I performed the upgrade to 10 almost everything is inaccessible. Unfortunately I only have essentials running here so I can’t rehydrate the data with that and of course it won’t rehydrate as an external. The disk just shows up as a RAW volume. I’m not sure how to proceed but losing the data isn’t an option. I can run the drive in the original machine and write new files and access some of the previous files for the time being until there’s a way for DeDupe to be turned on in 10. But anyone has any additional guesses I am all ears. As it stands I can’t even copy the files off.

    I wonder if a dedupe aware backup would allow me to backup and restore in full….

    • FoxDeploy August 12, 2015 / 11:47 am

      Can you enable Hyper-V and then make a VM with full server installed?

      Official word from MS as it stands is that enabling dedupe was a hack (which it was) and unsupported. Don’t hold your breath waiting for dedupe to ever ship on desktop Windows.

      • Ben September 10, 2015 / 9:16 am

        I’ll give it a shot! Anything is worth a shot at this point. I should’ve been backing up files more often! That said, I haven’t been able to get the volume to show up as anything other than raw at this point.

      • Ben September 10, 2015 / 9:18 am

        Ok so the answer is no. Essentials doesn’t allow for Hyper-V OOTB. I get a message that it’s disabled in the firmware….

        • FoxDeploy September 10, 2015 / 10:09 am

          If you have another PC handy, just install Server 2012 preview on it, move your hard drive over, turn on dedupe and use it to disable dedupe on your volume.

          Not pretty, but pretty much our only workaround, today.

  4. Alex August 14, 2015 / 11:55 pm

    I have the same problem, running a Windows 8.1 as a Homeserver with two deduped 3TB drives as Backup and Media Server.
    I first tried to Rollback to 8.1, but W10 told me that there is no way back, because files are missing (which is not possible, because I nothing changed after the Installation and the windolws.old files are still there).
    Now, after 6 hours Hyper-V Server 2016 unoptimization the progress is 60%.
    I bought two cheap 8TB drives and then I need dedup no more for the next time…

  5. Toto August 15, 2015 / 2:45 pm

    MS are bastard on this. They should give us a dedup package for Windows 10.

    • FoxDeploy August 15, 2015 / 2:53 pm

      Use the link and vote it up! Hell, share it with your friends and get more people to vote!

      • Toto August 16, 2015 / 8:56 am

        I did 🙂

    • FoxDeploy August 16, 2015 / 9:04 am

      Do you mind reposting that driver in Dropbox or elsewhere?

        • FoxDeploy August 16, 2015 / 10:06 am

          Cool, I’ll try this on a VM and write up some instructions!

      • SLRVision39 August 16, 2015 / 10:14 am

        Run Install.bat and InstallPack.bat (it’s still a work in progress :))

        • FoxDeploy August 16, 2015 / 10:20 am

          So it’s a driver that lets you read from deduped volumes. I’m assuming that you cannot write to them though. Without the Powershell cmdlets to disable dedupe, I imagine you’d need to copy the files onto another volume.

      • SLRVision39 August 16, 2015 / 10:59 am

        That’s right.
        We are still trying to fully enable dedup but it’s a first answer to the problem you’re describing in your blog. I haven’t tried to write on this disk… but that’s going to be my next test.
        I’m still dual booting W8.1 and W10 because of lack of RSAT on W10 so it’s not a big deal right now but I don’t want to screw up my VMs 🙂
        I have a 1 To SSD with, at least 20 W2K12 R2 VMs. I gain not less than 62% of space on it thanks to deduplication on Windows 8.1 so I’ll keep trying to enable dedup in Windows 10.
        My other solution would be to triple boot W2K16TP3 but I’ll miss a lot of things (beginning with Bluetooth and it’s a problem with my Apple Magic Mouse :().
        I’m just disappointed by the fact that TP3 is not a 10240 build but it was expected.

        • FoxDeploy August 16, 2015 / 11:42 am

          Windows 10 has 2016 Hyper-V, which has nested virtualization. So you could run a 2016 Hyper-V host VM with dedupe, if you wanted minimal work. Plus it would be cool. 🙂

      • SLRVision39 August 16, 2015 / 12:04 pm

        Yes, I have, of course, enabled HV 🙂
        I have copied a small amount of my VMs to a new 1 To SSD I bought specially for this 🙂 But I can only fit 9 of the 20 I have on my deduped disk 😦

      • SLRVision39 August 17, 2015 / 2:52 pm

        Hi,
        This is a totally working package…
        Have fun

    • Ben September 10, 2015 / 9:11 am

      Did this stop working? The mega links seem to be down and if I understand what your saying here it sounds like you can copy out of a deduped volume directly. That would solve my problem immediately!

  6. ssd vds NatcoWeb.com August 18, 2015 / 10:22 am

    Thank you for the post about the recovery files on Dedeuped Windows 10, you helped me a lot.

  7. Matthew August 21, 2015 / 3:25 am

    How would you advise to recover the file if you have an dedupped partition instead of a dedupped disk?

  8. Wayne February 22, 2016 / 5:11 pm

    You sir are a genius! Thank you so much for this. You have saved my job.

    • FoxDeploy February 22, 2016 / 5:43 pm

      Next time, don’t run hacky stuff in production. Actually who am I kidding, I had the same issue.

      Imagine if dedupe worked on the OS partition. We would have been really hosed!

  9. Shai October 5, 2016 / 11:09 pm

    Hi Guys , thank you very much for this post and your comments, I tried to do all the steps as you described to do (FOXDEPLOY & Dirk Becker) but when I tired to run the command “”Start-DedupJob -Volume D: -Type Unoptimization”” in the event log I receiving the following error : Data Deduplication service could not unoptimize file “”E:\DOWNLOAD\TEST.ZIP”. Error 0x80070519, “”The revision level is unknown.

    I tried anything but nothing work 😦

    BTW all the mess started when Upgrade windows 10 ver 1511 to ver 1607 …

  10. shaismit October 6, 2016 / 8:21 am

    Hi all. Stephen , thank you very much for your post it’s very useful, I thought that the worst is behind me .. but it seems not …
    I ran a deduplication feature on my Windows 10 O\S … but I didn’t took in account what will happen if I will upgrade my Win 10 OS from ver 1511 to 1607 …I lost all the ability to reach and retrieve my files (important files).

    In a short , I followed your instruction how to unoptimize, but unfortunately the situation now that while I’m trying to unoptimize the deduplicated partition I receiving the following error (at the event log) :

    “Data Deduplication service could not unoptimize file “`E:\downloads\[file name]”. error 0x80070519, the revision level is unknown”

    Please please if someone have any idea how to resolve this it will be a miracle for me ! …It’s seems that I lost all my data ;(

    • FoxDeploy October 8, 2016 / 11:30 pm

      Did you try attaching the volume to a VM with dedupe? I’ve seen this message, just trying to recall how to fix it.

    • FoxDeploy April 29, 2017 / 10:05 am

      This is not Windows dedupe but is their own product instead.

      • Mark Jennings April 29, 2017 / 10:18 am

        100 % compatible with Windows dedup. Activate dedup on a drive in windows 10 with this program, and test it on windows server 2016. It works: all files are accessable…

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s