Data Erasure Concerns: What everyone should know

[Xenocrates]

Synopsis:

Did you know that operating system based methods of erasing data from your hard drive don't actually erase the physical data on your hard drive? Even if you format your disk or delete the partitions on your hard drive, the data still remains!

The Situation

1. Deleting a file only deletes the pointer to that file in the File Allocation Table.
   
   
2. Quick-Formatting only resets the File Allocation Table.
   
   
3. Formatting not only resets the FAT, but also each file cluster pointer. Formatting a disk merely erases the indices of each file cluster of each sector on the disk. The data is still there, intact and relatively unharmed. It's just that there are no pointers to the data on your hard drive. As such, your operating system won't detect their presence.
   
   
4. Deleting a partition erases the FAT for the logical segment of a hard drive. Again, the data still remains.

NONE of these readily accessible methods actually erases the physical data - i.e., the 0's and 1's that make up the essence of the file itself. In all three cases, the data remains. So long as none of its segments have been overwritten, it's CRC (i.e. it's cyclic redundancy check) will remain intact and thus is perfectly retrievable by forensic or file recovery tools, such as "GetDataBack". Such tools are used by the FBI to retrieve data from the erased hard drives of software pirates!

[more...]

[Xenocrates]

About Pointers / Indices

For those of us who are unfamiliar with disk architecture, all of these jargons:

1. Partitions
2. Logical Drives
3. File Allocation Tables
4. Sectors
5. Clusters

...are forms of indices or pointers, arranged in hierarchies. If you are familiar with C programming, then you understand why pointers are necessary. They add the "random access" element to data, enabling the direct-access component of all forms of RAM. Whether this is permanent RAM (disks) or volatile RAM (VRAM, main RAM etc.), pointers are used to index the data for fast retrieval. These pointers eliminate the need for sequentially accessing the medium, which would be a very slow and cumbersome process. Therefore:

When you delete, format, or remove the partition of a disk or erase a CD-RW or DVD-RW, you are not actually erasing the data. You are only erasing the pointers to the data, not the data itself. The data still remains.

That's why programs like "GetDataBack" can retrieve the data from your hard drive, even after multiple formats. Even changing the file system won't erase the data. File Systems only effectively differ in the cluster size and the nature of the indices. But the concept remains the same across all file systems in virtually every operating system.

Note: A full erase on optical media will erase the actual data on CD-RWs and DVD-RWs.

[more...]

[Xenocrates]

How the data is retrieved

Essentially, formatting your hard drive reduces it to the same thing as a sequential access medium (like tape for example). The data is there, but without the file cluster indices, the hard drive has to be read sequentially (i.e. from sector 0 to the very last sector, from inside-out on the disk) to find all of the data.

Since your operating system (doesn't matter which one) only locates data based on these indices, deleting the index to a file makes the file "invisible" to the Operating System - giving the impression that the file has been deleted. But anyone with a forensic tool can find the data (albeit, sequentially), so long as that tool knows how the file clusters are stored (as per the disk's file system). This has some serious security implications.

Why is this potentially dangerous?

Can you imagine if someone were to access your hard drive for whatever reason, and bring back the data that you thought you deleted? What about things like company secrets, or personal emails, etc. that were "deleted" from a hard drive? Do you realise that anyone can access forensic tools such as "GetDataBack" which sequentially read your hard drive for file clusters without indices, and look at the data you supposedly "deleted"?

This is essentially why programs like "Norton Erase" are necessary if you want to completely remove that data from your hard drive. Since deleting or formatting won't erase this data, you have to use a program which exhaustively overwrites every 1 with a 0 for every single bit of the file on the disk. Norton Systemworks comes with such a program. No techie should be without a tool as crucial as this.

[more...]

[Xenocrates]

Why are Operating Systems Designed like this?

Quite simply? Efficiency in Design. There are three issues with this scenario:

• The Good - If you accidentally delete a file, it can be retrieved.
  
  
• The Bad - If you actually want a file gone, it can still be retrieved.
  
  
• The Ugly - Most operating systems don't ship with a tool to actually erase the physical data. Most operating sysems only delete the indices - which is faster than actually erasing the data.

If all operating systems were to completely delete every file that is "deleted", the actual deleting process would take just as long as creating the file. Can you imagine deleting a 4.3 gigabyte DVD image from your hard drive? It would take just as long as it did to copy that image to your hard drive in the first place. This would severely slow down the performance of your computer system - especially when your OS occassionally creates thousands of temporary files. This is why operating systems only delete the index to the file, and not the file itself. That's why you can take 2 minutes to copy a 700 mb CD image to your drive, and a fraction of a second to delete it.

[more...]

[Xenocrates]

Conclusion

End Users

Be careful of what data you store on company hard drives. Don't put anything there you would be embarrassed to have retrieved by a local administrator - or worse, by a BSA operative. Keep your private e-mails at home. Keep your loot off the company PC. If you are about to leave an organisation, it would be a good idea to use a tool such as Norton Erase to completely wipe the free space on your disk.

If you must, buy a USB-To-IDE adapter (such as a drive enclosure) and walk with an extra hard drive, or buy a personal USB drive (aka Key drive or Jump drive). Keep your loot, personal data, etc., on those external mediums.

Administrators / Techies

Having tools such as Norton Erase (to completely remove the data) and GetDataBack (to retrieve "lost" data) are indispensable to your software toolset. Make sure you have software in your CD-pouch (or USB drive as the case may be) that can do either function. When you're in the business of working with data, (any data, really) you just never know when these tools could save a life (or crucify one) if necessary.

The implications of this data storage paradigm are absolutely stupefying!

[zRo ToLeRaNcE]

Nice!!!

Already knew the stuff you were talking about but still learnt one thing...

Note: A full erase on optical media will erase the actual data on CD-RWs and DVD-RWs.

[Billerg]

wonderful article xeno .

[BlaqMale]

knew this stuff to, i use drive shredder that comes with system mechanic to erase hard drives or the respective manufacturer's tool. i use handy recovery 2.0 to recover deleted files even off a thumb drive. i kind of like the fact that not many people know this so that they leave sensitive data on a hard drive b4 i get it (right chef?)

[willy]

nice read xeno .... got anymore?

[digitalchef]

very true blaqmale. however i have nothing to hide on my system.