What might cause Files to abort or fail a big copy?

Question

Context; saving Mac backups using another PC with eOS boot SSD to store duplicates. Repeatable failure using Files to copy music archive, about 15K item count and 125GB total. The source is iTunes storage in the style of Music/Artist/Album/Song:

• external 2TB cap HDD over USB, fast enough, feels like a five minute copy if it could finish
• scroll dialogue opens with correct count/size
• scroll stops reporting around the 1st GB, probably hung
• system monitor reports about 50% CPU then squats there, probably hung
• file destination quickly reaches about 30GB, but best attempt was 64GB
• directory check shows some jumping around, but somewhat alphabetical with gaps
• new attempts repeatably copy or skip the same subdirectories, although stopping at varied points
• some system files without read permission skipped, as expected
• not asking me to skip unreadable after first attempt, maybe cached response?
• similar results whether drag-drop or context menu copy-paste
• no problems with copied heirarchy structure
• can successfully copy missing, nested directories individually
• can successfully copy smaller samples, up to 6GB so far

I'm proceeding with my manual workaround, a few GB at a time so I can check results before moving on; for example, the "As" were about 3GB and I could count subdirectories on my fingers and toes. Smaller bites won't take long but I'm asking about Files behaviour because the unexpected failure is disturbing. The silver lining is that it's obvious something is wrong by checking top level count/size if the dialogue appears frozen.

Does Files have known hard limits or throughput problems? Please don't worry about my expectations; I still recall swapping 5MB pie platters and 10MB tapes, so copying 15K file systems is magic. I didn't find other complaints or recommended Files replacements online, which suggests my issue might be local.

Answer 1

There are a number of outstanding issues reported against Files in relation to file operations involving very large number of files. Files is currently best used for small to modest amounts or data (<10K items?). A crash might be caused by a memory leak for example. One of the proposals for this year's GSOC is the improvement of the aspect of Files.

Note also that because the system might cache file operations, Files may think that the operation is complete but it is still continuing in the background.

I would recommend using a specialist program like Grsync for backing up large volumes of data rather than a general purpose file manager. Or you could install another filemanager from the AppCenter. If you otherwise like elementaryos then there are certainly solutions other than switching to another distro.