Author Topic: FTP upload issues on most Stock Sites (Read 7096 times)

dnavarrojr · « **on:** May 06, 2009, 11:00 »

I am running into major upload issues with most of the stock video sites. Because video files are so large, they can take a VERY long time to upload depending on your connection speed.

Using CuteFTP or FileZilla, I run into an issue that if left unattended, they will think that a file has not been successfully uploaded and will attempt to re-upload the file. On a number of sites (such as Shutterstock) this results in many duplicate copies uploaded of the same file.

Since I am a programmer, I decided to write my own FTP program to try and diagnose the issue. And what I discovered is that on those sites where I have the problem, the FTP server closes the control connection while the file is uploading. So when the upload is done, the client software gets a time-out error trying to talk to the FTP server through the closed connection. Most FTP clients assume this means the file was not successfully uploaded and attempt to upload the file again.

The root of the problem is actually the fault of the Stock Sites. FTP servers are NEVER supposed to close a control connection if there is an active data connection. The stock sites are not enabling (or disabling) this feature in their FTP servers.

I have tried to discuss this with the sites I have these issues with: Shutterstock, Fotolia, Clip Canvas, Can Stock and Dreamstime with little success. Either they don't understand what I am talking about, or they don't care.

So my only solution is to try and find a client program that does not attempt to re-upload files.

Does anyone have any suggestions? This is not so much an issue with photos as they are smaller and can be uploaded before the control connection is closed. Only when you start uploading 200 megabyte files and larger does this issue rear its ugly head.

alias · « **Reply #1 on:** May 06, 2009, 11:16 »

What if you just open the terminal application and use ftp direct from the bash command line ?

donnelt · « **Reply #2 on:** May 06, 2009, 11:19 »

Have you tried the 'keep-alive' option in Filezilla->Edit->Settings->FTP?

dnavarrojr · « **Reply #3 on:** May 06, 2009, 11:34 »

Quote from: alias on May 06, 2009, 11:16

What if you just open the terminal application and use ftp direct from the bash command line ?

Same basic problem.. The control connection is closed by the server and the FTP utility closes with an error after uploading the first file. The good news is that duplicates aren't uploaded, but you have to write a script (or batch file as the case may be) to upload batches to account for the error.

dnavarrojr · « **Reply #4 on:** May 06, 2009, 11:38 »

Quote from: donnelt on May 06, 2009, 11:19

Have you tried the 'keep-alive' option in Filezilla->Edit->Settings->FTP?

The problem with FileZilla (and CuteFTP and a half-dozen other programs I tried) is that it does FTP transfers multi-threaded. So keep-alives work in the primary thread on some servers (not all) but in the data thread the control connection is still closed and it still causes FZ to re-upload the file.

I have tried to bring this to the attention of the FZ author but because the underlying issue is a bug or misconfiguration of the server software, he refuses to do anything about it. He's a brilliant programmer, but an incredible ass with very poor social skills.

donnelt · « **Reply #5 on:** May 06, 2009, 11:52 »

Set the time-out to zero and that may force the server to keep the control connection open. (Edit->settings->transfers).

My control connections drop regularly during upload but I don't get a transfer failure at the end of the upload. I assume you are using Passive?

dnavarrojr · « **Reply #6 on:** May 06, 2009, 11:57 »

Quote from: donnelt on May 06, 2009, 11:52

Set the time-out to zero and that may force the server to keep the control connection open. (Edit->settings->transfers).

My control connections drop regularly during upload but I don't get a transfer failure at the end of the upload. I assume you are using Passive?

Yup, passive mode. I'll give that a try, but on many sites keep-alives don't work because the server closes any control connection that does not initiate a transfer within 3 to 5 minutes. So, if I am uploading a file that takes 30 minutes to upload and FZ (or my own test program) is sending keep-alive commands the control connection is still closed because it doesn't initiate a new transfer.

For the moment I am just using my own program that I wrote and having it ignore the connection error. That pretty much works, but I wrote it as a command line utility, so I have to stick a GUI on it to make it more useful. In the meantime, I'm still hopeful of finding something else that works.

stockastic · « **Reply #7 on:** May 06, 2009, 18:53 »

This sounds like a 'watchdog' timer on the server side that enforces an overall maximum time on a file transfer. It if hasn't completed in that time the connection is closed, even if data is still being transferred. I don't know about such a setting, just guessing.

Of course someone on the microstock site would have to get involved but if it's just a configuration setting, it would be easy to change.