As I may be busy over the next few weeks fixing little things here and there I thought that a database upgrade option would be handy, so, if you have already upgraded to version 0.6 or higher you no longer need to start from scratch and can keep your data.
The default setup script will now give you an option to upgrade, but if the validate fails then it will reset the database by default!
[dm]18[/dm]
Just a little update, dup requires the below packages and service to run.
Init_3rdparty
MySQL must be enabled on your NAS with a root password set. (This will change with version 0.6.3 with local file database option)
HTTP service must be enabled on your NAS.
hey arden,
I’ve installed this pkg and it won’t get passed the setup.php screen no matter what i do. i’ve followed the screenshots and it’s still not working. i think there is a bug in your 0.6.2 package and I couldn’t download any of the previous versions to test it myself.
as of right now, the app doesn’t work because you can’t get passed the setup.php screen no matter what is entered or clicked on.
Hi, I too have this problem but my mySQL password has definitely been changed and I’m required to input it for phpMyAdmin – did the OP eventually solve this, or any other ideas how I can troubleshoot? I’ve downloaded the latest (0.6.2) package and installed MySql specifically for this, so I have no reference point to guarantee it’s working beforehand, and like the OP, didn’t have anything earlier than 0.6.2 to install. Would be grateful for any pointers where to check next, as I can’t get past the setup.php page. Thanks!
Hey,
Did you perform the following command via ssh?
chmod 777 /volume1/web/dup/db.php
or you could do it through the DSM File Manager.
db.php needs to be writeable by everyone in order to be updated via the setup script.
Hi,
If it reloads to the setup screen it’s because the mysql password was wrong or empty.
Did you just enable the MySQL now for this package or was it already enabled and working before hand? Have you set a root password for your MySQL yet?
You can test by installing the phpMyAdmin (in the package center) and if that can connect to your MySQL you know it’s related to dup, if not then it’s related to your system setup.
On a side note I just noticed the file for 0.6.2 was really 0.6, I uploaded the wrong one. I’ve fixed this now, but it should have still worked. Try downloading again and install, but I’d also do the phpMyAdmin test to verify connections can be made to your MySQL.
Hi Arden,
First of all thanks for all the effort you put in this package!
It looks like the scanning proces won’t start.
In the “profiles” the status says “starting scan..”, cpu stays low and i don’t see any relevant processen taking up any resources.
info:
– Fresh install 0.6.2
– dub did create database.
– Customize PHP open_basedir – none (otherwise not able to browse folders while adding profile)
– Used the PHP settings posted in the setup screen
– created /Data/Moved dir
any thoughs?
Hey,
I just downloaded the 0.6.2 version and installed and it’s working fine for me, so I’d say the best bet is to check the logs, set to All and Debug, look for something like the following,
Debug Scan: shell_exec(“php -q scan.php 1 >/dev/null 2>/dev/null &”); – /volume1/web/dup/scanfiles.php
Note dup can log a lot of data so your best off clearing the logs first and then trying to run a scan again.
Also for testing set the scan to a small folder of say about 200 or so files.
scanfiles.php is where the scan kicks in and what sets the status to ‘Staring Scan…’ so I’d say the issue is around here.
As it’s working for me and your the first to have an issue that I know of like this I’d try restarting your unit if you changed any php settings just in case it’s a permissions issue and the command fails.
Thanks for your help Arden,
I restarted DS.
created&chose folder with 20 files
started scan but after few hours still status “starting scan” and no files scaned.
Checked permission for /volume1/web: all 777 recursive
I check the log and saved it after creating a profile and after starting the scan, i’m not sure what to look out for. Here’s a short piece after starting the scan:
2013-07-18 11:20:31 Info+ GetProfiles() Gets profile ordered by passed pram - /volume1/web/dup/db.php2013-07-18 11:20:31 Info+ GetStats() Gets Profile Status - /volume1/web/dup/db.php
2013-07-18 11:20:31 Info+ PageFooter() HTML footer - /volume1/web/dup/dup.php
2013-07-18 11:20:31 Info+ GetStat() Get profile status via pram - /volume1/web/dup/db.php
2013-07-18 11:20:31 Info+ PageHeader() HTML Header - /volume1/web/dup/dup.php
2013-07-18 11:20:31 Info Home - /volume1/web/dup/index.php
2013-07-18 11:20:31 Info+ GetProfiles() Gets profiles based on order parm - /volume1/web/dup/dup.php
2013-07-18 11:20:31 Debug Scan: shell_exec("php -q scan.php 2 >/dev/null 2>/dev/null &"); - /volume1/web/dup/scanfiles.php
2013-07-18 11:20:31 Debug MySQL: 'UPDATE profiles SET scan=1 WHERE id=2' - /volume1/web/dup/scanfiles.php
2013-07-18 11:20:31 Debug MySQL: 'UPDATE status SET value='Starting Scan...', data='/volume1/foto/test' WHERE profileid=2;' - /volume1/web/dup/scanfiles.php
2013-07-18 11:20:31 Info ScanProfile '/volume1/foto/test' - /volume1/web/dup/scanfiles.php
ok, it looks as though the command to run the script in another thread is failing, .eg
shell_exec(“php -q scan.php 2 >/dev/null 2>/dev/null &”);
In version 0.6.3 I’ll put in full paths to both php and the script to bypass any issues it might have caused.
Ca you log into you nas via ssh and try the following commands
cd /volume1/web/dup
php -q scan.php 2 >/dev/null 2>/dev/null &
nothing should be displayed in the terminal, but you should have new log entries!
here we go:
2013-07-19 10:27:00 Info+ PageHeader() HTML Header – /volume1/web/dup/dup.php
2013-07-19 10:27:00 Info Logs – /volume1/web/dup/viewlog.php
2013-07-19 10:26:58 Info+ GetProfiles() Gets profile ordered by passed pram – /volume1/web/dup/db.php
2013-07-19 10:26:58 Info+ GetStats() Gets Profile Status – /volume1/web/dup/db.php
2013-07-19 10:26:58 Info+ PageFooter() HTML footer – /volume1/web/dup/dup.php
2013-07-19 10:26:58 Info+ PageHeader() HTML Header – /volume1/web/dup/dup.php
2013-07-19 10:26:58 Info Logs – /volume1/web/dup/viewlog.php
2013-07-19 10:26:55 Info+ GetProfiles() Gets profile ordered by passed pram – /volume1/web/dup/db.php
2013-07-19 10:26:55 Info+ GetStats() Gets Profile Status – /volume1/web/dup/db.php
2013-07-19 10:26:55 Info+ PageFooter() HTML footer – /volume1/web/dup/dup.php
2013-07-19 10:26:55 Info+ PageHeader() HTML Header – /volume1/web/dup/dup.php
2013-07-19 10:26:55 Info Logs – /volume1/web/dup/viewlog.php
2013-07-19 10:26:52 Debug MySQL: UPDATE profiles SET files = (SELECT COUNT(id) FROM files WHERE files.profile = 2) WHERE id=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE profiles SET runs=1 WHERE id=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE profiles SET scan=0 WHERE id=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE status SET value=’Refreshing’, data=’Refreshing’ WHERE profileid=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE profiles SET dups=0 WHERE id=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: SELECT COUNT(size) FROM files WHERE profile=2 GROUP BY size HAVING COUNT(size)>1;
– /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE status SET value=’Idel’, data=’0 Files Added’ WHERE profileid=2;
– /volume1/web/dup/scan.php
2013-07-19 10:26:52 Info+ CheckStatus() Check a profiels status via passed pram – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Info+ ProcesslsStr() Process ls string via pram – /volume1/web/dup/db.php
2013-07-19 10:26:52 Debug MySQL: DELETE FROM files WHERE profile=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE status SET value=’Processing Scan’, data=’Looping through 22 entrys and sorting’ WHERE profileid=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Info+ Scan: shell_exec(“ls -lRe ‘/volume1/foto/test'”); – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Debug MySQL: UPDATE status SET value=’Scanning Files’, data='”ls -lRe ‘/volume1/foto/test'”‘ WHERE profileid=2; – /volume1/web/dup/scan.php
2013-07-19 10:26:52 Info Scan – /volume1/web/dup/scan.php
2013-07-19 10:26:44 Info+ GetProfiles() Gets profile ordered by passed pram – /volume1/web/dup/db.php
2013-07-19 10:26:44 Info+ GetStats() Gets Profile Status – /volume1/web/dup/db.php
2013-07-19 10:26:44 Info+ PageFooter() HTML footer – /volume1/web/dup/dup.php
2013-07-19 10:26:44 Info+ PageHeader() HTML Header – /volume1/web/dup/dup.php
2013-07-19 10:26:44 Info Logs – /volume1/web/dup/viewlog.php
2013-07-19 10:26:44 Debug MySQL: TRUNCATE TABLE logs; – /volume1/web/dup/clearlogs.php
hmm..sorry for messing up the forum with code..
Cool, so it looks like it performed the scan via SSH then, just not via the web interface which is very strange!
I see no dups were found and only 22 ls entries were found, although I don’t see any being added. Does your profile show as having any files in it on the home page? Maybe add a few more files!
When you added the Profile, did you select the correct type, as in dup will only scan/add files over 1mb in size, but will also only process files with the correct extension, so if the files are of .iso and you selected Films, no files will get added/processed as the Films will be defined as avi,mkv,…….etc
You can email the logs from now using the feedback section in dup, just copy and paste!
Please recheck your php settings so they match the screen shots, as its looking like the shell_exec(“php -q scan.php 2 >/dev/null 2>/dev/null &”); is not running.
i dont see my comment, or isnt aproveed ?
i will copy mi last comment 🙁
Hi everyone, i have the same problem,
It looks like the scanning proces won’t start.
In the “profiles” the status says “starting scan..”, cpu stays low and i don’t see any relevant processen taking up any resources.
1 have 2 synology servers
1 ds211j ( 1 hdd 2tb / 1 volume
1 ds410j( 4 hdd 3tb each/ 1 volume shr 1 drive proteccion)
dsm 4.2-3211
first i installed, test and use dup in de ds211j, and only when i consider that its working ok, then i instales dub in the other srv, 410j
in the beginning its runing ok, in the 2 srv , making goods search+
but in a while, the dup program, make me the same that ouwe
i make inthe fisrt times small folders, then bigger ones, now again smaler, but its not working like in the beggining
the folder have
1407.2 gb, 460,087 files, 32,503 folders
any solution?, i will apreciate any info, thnks!
how i copy the log??
Sorry to hear that guys, I really can’t think of why it’s not working except for it being a permissions issue. From ouwe’s logs I can see it’s finding 22 possible entry’s, that means the “ls -lRe ‘/volume1/foto/test'” command returned 22 lines, yet the script (scan.php) only processed one line and yet it wasn’t a valid file entry line and then the script exited.
Can you perform that command and send its output via the feedback form please, just in case the ls output format is different to mine (DS2411+)
I’m currently on a 2 week holiday so will reply when I can, and version 0.6.3 which does address some issues including local file DB instead of MySQL wont be ready until I’m back.
Hi,
i think that you should provide complete and consistent installation instructions.
For the first version there should be added /volume1 and 2 to the basedir, in a later version you set it to none, and a new user has no clue what is correct. I am a real power user with good skills, but this is something I can not deal with.
I can not read for hours for using this package.
The parts you mentioned under: “Just a little update, dup requires the below packages and service to run.”
is simply not enough.
We need the complete steps with every required setting for the corresponding version (0.6.2 p.e.).
Just my two cents.
What I did: setting the http settings (both, none and the ones with the volumes added), installed 3rd party, installed dup, checked mysql root password, is set and known as correct, but still,
– stuck in setup.
Don’t get me wrong, i REALLY appreciate your work, but I want to save others from that frustration that I am experiencing, because I think it’s worth it.
Hey erdnuesse,
Thanks for the feedback! With regards to your endless loop of the setup script, I think I may have found the issue. Go into your file manager and select the file db.php in your /web/dup folder and set the permissions to 777, I’ve noticed on some systems the file was not getting the right permissions set and therefor the setup script could not edit the file.
dup 0.6.3 has addressed this issue and will be release shortly. As for the install process, as of dup version 0.6, it’s always notified the user that both the web station and Init_3rdparty are required before it can be installed. Once installed and on the setup pages the checked options have always informed the user of the SQL requirement along with the requires values for open_basedir for basic usage, the new value of none just makes things easier so the user doesn’t have to add every single path he wants php allowed access too.
dup 0.6.3 has much better error reporting/logging too so hopefully finding issues like the will be a lot easier!
First, Thank you for your response.
unfortunately, it chmodding dup.php to 777 did not solve the issue for me.
I restarted web station and mysql, even upgraded my syno and restarted it. Still nothing.
So I’m waiting for 0.6.3 – because I need to find my doublettes – consolidated three external hard disks to the syno – twice. So I’m probably wasting about a third of my disk space…
Regards,
erdnuesse
Hey erdnuesse,
Just a FYI, it’s db.php that needs to be set to 777 not dup.php
eg. chmod 777 /volume1/web/dup/db.php
Note I recently updated to DSM 4.3-2 myself and this has a load of new permissions issues. The account the webserver starts with is nobody, and has been severely limited compared to older DSM versions, meaning additional updates and I’ve been getting side tracked quite a lot lately too.
If your on 4.3-2 the delete/move/rename functions will more then likely stop working until 0.6.3 is released which has a work around to su root or what ever account will have access to perform the actions on the files, this will also include symlinking, but the reporting should be fine!
I have been looking at trying to find a way to locate and delete my duplicate files on my Synology, and i have a TON of them. The feature that Synology has added is nice, but it only looks for duplicate by filename, which is not helpful. A lot of times I will download photos from an SD card and then format the SD card and take more pictures. Well the files all have the same name, but are different photos. I see that you use a CRC32 for hash verification. I am familiar with MD5 and SHA hash values, but how accurate at detecting duplicates, versus false duplicates is the CRC32 you use?
Thank you for your hard work on this project!
Hey musicguy,
dup doesn’t currently compare files by filename. dup has 3 different methods for identifying dup’s.
1. File size (fastest but least secure)
2. CRC32 Part (reads the first 150KB, the middle 150KB and the very last 150KB of the file, generates a CRC32 hash for all 3 parts and then hashes the 3 results added together, faster and more secure, note, dup will still only compare/hash files of the same size, if the file is less then 450KB in size then the full files is hashed)
3. Full CRC32 (slowest but most secure)
CRC32 Part should be good enough for image files, as it really only hashes 450KB from 3 parts of the file, but if your worried select Full CRC32 but it will take longer, especially if your files are 5+ MB in size!
MD5 would be more secure but also slower, that’s what I selected CRC32. Some more information about CRC can be found here http://en.wikipedia.org/wiki/Cyclic_redundancy_check
Thank you so much for your quick reply. I will give dup a try.
Thank you,
What is the best way to install or verify that MySQL is installed?
Thanks,
Install the phpMyAdmin package and then you can connect to it, make sure to set a password for the root account too!
Thank you. I did so and now get the message, “Please install Init_3rdparty=1.5 before installing this package.
Suggestions?
You can download it from here and install manually.
http://www.cphub.net/?id=37