dup for Synology systems

dup is a small package I created for my Synology NAS. Having it loaded with 500Gb drives I soon ran out of space, and having additional drives lying about with more data on them so I decided to fork out and upgrade the DS2411+ to have 12 3TB drives and move all my data I’ve collected over the last 10+ years into one central location.

While in the process of moving my data I noticed I had the same Files/Folders on different drives but copied them over anyway and thought I’d run a program to find the duplicates and then remove them. While moving the data I noticed that this will take a long time, even with a GB connection to the system, and a computer would also be required to run the software from. This is where Dup came from!

I am (at the time of this post) working on another project called nzb (might be renamed) which I grabbed some basic code from and as my data is more important, and I decided to complete the dup project so I can start using it. I also thought I’d share it. It’s not perfect by a long shot, but it’s quick and light and fulfills my needs. Below will be some basic information about it and how to use it, including it’s features.

 

dup is written in php and requires an additional package for working on your Synology system, you can get it here.

dup also requires you to make some changes to the way php is configured on your Synology device.

There changes are, adding /volume1:/volume2(etc) to your open_basedir

This is required for dup to scan the folders for files. you should add all other volumes you might want to scan with dup.

safe_exec_dir must also be disabled.

This is required for dup to be able to use the ls -lRe command on your system to list the files and folders, also required for the unlink and rename functions for deleting and renaming/moving files.

Once you have setup the correct PHP setting and installed dup your main screen will look like this until you have set it up.

Here you need to run the setup script.

After running the Setup script you have 3 fields to enter.

Password : for user root on MySQL

Default path to start searching, eg /volume1/Films/

Default folder to move duplicates to, eg /volume1/Data/Moved/

Note, these 2 paths must exists! If they don’t, defaults paths will be entered simulare to above.

Once your setup, the basic interface is as below, settings is always at the top, here you can make changes, add new profiles and down the bottom the home link and search function.

Here are a few things to keep in mind when using Dup.

Don’t have the folders to be scanned too large (if around 100,000 files I’d break it up, eg, /Music/Rock, /Music/Pop instead of just /Music), although dup can gather the file listing quick enough, it’s the MySQL operations that take the longest amount of time due to the MySQL being limited to 24MB. (If you know a way to change this like you can with some php options please let me know)

I usually scan (check Scan path for files and save settings) with CRC32 disabled first, reason for this is the matches will be based on the file size first, not very accurate but gives you an idea of how long the next step will take. Once you have gotten your file list (check CRC32 and don’t check Scan path for files and save settings), this will then only perform a CRC32 update on the found files that match in size. This can take up to 5 minutes for about 1000 matches (2000+ files to scan).

You will always know when the script has completed as the settings block will always disappear on a page load! theres no loading times or progress indicators at current, maybe something for an update though!

The CRC32 is not a complete CRC, hence why it’s so fast at computing them. The crc is made up for 3 stages, the first 150 kb of the file is scanned, the middle 150 kb is scanned and the last 150 kb of the file is scanned (up to 2GB in size, php limit!) and the 3 crc’s are added together and that string is crc’d again to give the final CRC32 result!

The minimum file size that can be scanned is 1MB , there’s no limit on the largest but the PHP functions can only read 2GB into the file on 32 bit systems. I messed about and tried to get this to work for large files but decided to leave it as is, it will display large files sizes without any issues though.

There are default extensions that you can change, or set to the found ext’s after performing a scan (Update option). Setting this blank and saving will load the defaults again. These are the file types that get entered into the database only.

When you click Add profile it will copy the current profile settings, so you will need to select the new profile again after the page reloads (I may change this to automatically select it for you in a later version)

Down the bottom of the file list are 3 options, to check all duplicates found (based on file size or crc, depending on how your viewing the list in settings) and a delete and move options.

There pretty self explained but the delete option removes the file for good, and does not move the files to the recycling bin (again this may change in a newer version, if I can find out how other then renameing the file to the bin).

You can search the current profile only or select the Global option to search all profiles. Results are displayed showing case matches and non case mattches, also aranged by either size or crc32, depending on the profiles the search was performed from.

Also note, there is no security check in Dup, so if your nas is used by lots of users it might not be a good idea to install it (they could delete duplicats files found!), this will be added in a later version.

[dm]16[/dm]

1/11/2012 Updated to version .2

php limited to 1000 POST variables, sql querys limited to sets of 500 instead of getting user to edit php.ini file with max_input_vars=5000 etc, ini_set() cannot be used to set this. Pages links added to output to view addition records.
Added profile ‘auto select’ new profile feature (addprofile.php updated).
Won’t try to rename a file in the recycling bin but will show it.
Profile variable wasn’t getting passed properly in a header() call in action.php
fsize function updated to use ls -le format instead (safer)
ProcesslsStr() updated to remove echo statement (used for debugging)
ScanFiles() updated to not dump ls output into text file (used for debugging)
profile POST variable catch added in action.php
setup.php updated, 2 paths must exists in order to continue.

 

5/11/2012 Updated to version .3

php vars limit code updated to display under 1000 items total instead of limiting to 500 matches as this could still go over and casue issues.
action.php updated to rename files if already in the Moved to folder, skips code if action var not sent
setup.php code updated to make sure set folders exists better (without the ending slash)
die.php updated to also display php last error aswell as mysql’s
search.php updated to display file size with comms
few code tweeks here and there.

 

4/12/2012 Updated to version .4

Updated the fsize(), crc_file() and UpdateCRC() functions in dp.php, possible chance of returning wrong size for files, also hashing function updated to use hex values instead of unsigned ints and sql statement for updating crc’s updated.
Few display updates here and there.

 

4/12/2012 Updated to version .5

Progress popup window added to show scripts progress.

04/07/2013 Update coming soon, read more here.

09/07/2013 dup 0.6 released

  dup 0.6 (248.0 KiB, 83 hits)

31/01/2014 dup 0.6.3 Beta 2 released 

  dup 0.6.3 beta 2 (338.0 KiB, 131 hits)