Topic: FTP file upload

I have an existing file management rails app that uses http for file upload (using attachment_fu).  The files that people are sending are getting too large for an http process.  Does anyone know how make these uploads instead go to our ftp server?

Thanks,
Zack

But they that wait upon the Lord shall renew their strength

Re: FTP file upload

Hi Zack... how big are the files?

Ryan Heneise  |  Art of Mission  |  Now with extra-strong Donor Tools mojo

Re: FTP file upload

They are now trying to upload architectural drawings... They can be up to 500 megs.

But they that wait upon the Lord shall renew their strength

Re: FTP file upload

Holy schmoley - that's big.

You might end up having to use Net/ftp (http://www.ruby-doc.org/stdlib/libdoc/n … index.html) to write your own upload routine. I don't know how this would work though, because you'd have to initiate the FTP connection on the client side - Net/ftp would only work if the file was already on the server.

Really, the file size limitation is a limitation of the web server. You should be able to achieve this by tweaking your server settings. Apache 2.2 should support large file uploads up to 2GB: http://httpd.apache.org/docs/2.2/new_features_2_2.html

One other thing you'll probably want to do is open a socket and/or run extra mongrels - one for each upload. Otherwise, you'll tie up your mongrel processes waiting for the upload to finish.

Some semi-helpful links:

http://lists.apple.com/archives/Java-de … 00216.html
http://mail.python.org/pipermail/python … 43710.html

Ryan Heneise  |  Art of Mission  |  Now with extra-strong Donor Tools mojo

Re: FTP file upload

Thank you Ryan!

I've started writing an FTP upload routine, but I'm getting Input/Output errors right now.  I'll work on it some more and see if I can't get it working.  If I can't get it working, I might just use extra mongrels and optimize Apache, but I'm sure I'll come across someone trying to upload a package of drawings, and those might exceed 2GB, so I think I'll have to get this working with FTP, or some other way.

The other thing I've been thinking about is using MERB.  Does anyone have any experience with MERB, and file uploads?

Thanks,
Zack

Last edited by zreed20 (2007-10-14 16:42:40)

But they that wait upon the Lord shall renew their strength

Re: FTP file upload

Check out http://www.webappers.com/category/components/upload/ <- one of those might work for ftp uploads.

Ryan Heneise  |  Art of Mission  |  Now with extra-strong Donor Tools mojo

Re: FTP file upload

ryenski,  Thanks for the upload links.  Some of those look very good, but I'm dealing with people that might get confused by an Ajax / Flash form (Scary, considering that they're Architects). So, I've got this working using MERB.  I've sent files between 500MB and 1GB and haven't had a hiccup yet.  But I can't for the life of me figure out how to get the uploaded files back from the server.  It has a send_file method, and I tried to use it in my controller, but I keep getting can't convert nil into String - (TypeError messages).

Here's what I have in the controller

def file
    send_file params[:filename]
end

I've tried passing it the id, the filename, the whole path to the file, but it's not working.  This the MERB documentation on the method...

     # File lib/merb/mixins/controller_mixin.rb, line 168
     def send_file(file, opts={})
       opts.update(Merb::Const::DEFAULT_SEND_FILE_OPTIONS.merge(opts))
       disposition = opts[:disposition].dup || 'attachment'
       disposition << %(; filename="#{opts[:filename] ? opts[:filename] : File.basename(file)}")
       headers.update(
         'Content-Type'              => opts[:type].strip,  # fixes a problem with extra '\r' with some browsers
         'Content-Disposition'       => disposition,
         'Content-Transfer-Encoding' => 'binary',
         'X-SENDFILE'                => file
       )
       return
    end

It looks like it just requires the filename, but I can't seem to get it working.  Any help would be greatly appreciated.

Last edited by zreed20 (2007-10-15 10:07:53)

But they that wait upon the Lord shall renew their strength

Re: FTP file upload

I wouldn't use send_file, because streaming a 2GB file through Ruby might rip a whole in the space-time continuum.  Your best bet is probably to link directly to the file for download by HTTP.

If you're storing the files in your public/ folder, than it's as easy as directly linking to that file. If you're storing them outside of public/, you still have a lot of options. For example, you could set up a rewrite rule on the server.

I'm REALLY curious how you implemented your FTP upload solution. What are the chances of posting a tutorial?

Ryan Heneise  |  Art of Mission  |  Now with extra-strong Donor Tools mojo

Re: FTP file upload

ryenski,

I really don't want to rip a hole through the space-time continium, so I'll avoid send_file and stick with your advice and do a straight HTTP download from the public folder.

I didn't end up using FTP, I'm using MERB (Mongrel + ERB) temporarily.  I'm still working on the FTP backend, NET::FTP is very simple, but passing the stream through Rails is the tough part. I don't know if I'll ever get it get it working, but it has been fun working on it.

For the record, MERB seems very capable of handling huge file uploads / downloads.  I just uploaded a 3 gb file to it, with no problem.  The next phase will be to integrate this into my existing Rails App.

Thanks again,
Zack

Last edited by zreed20 (2007-10-15 11:37:18)

But they that wait upon the Lord shall renew their strength

Re: FTP file upload

zreed20,

+1 request for a tutorial smile

I will soon be working on a project that requires massive uploads as well and was thinking of MERB, but have no idea how to integrate it. I assume you just set up a REESTful service and then your Rails app consumes it, but some code examples/ detailed tutorial would just be lovely!

Re: FTP file upload

I'd highly suggest using BackgrounDRB (http://backgroundrb.rubyforge.org/) to handling the upload and use an ajax call back to keep the user informed of the progress. This will send the uploading process to a background process freeing up your rails app.

Re: FTP file upload

Ace of Dubs,

MERB is actually really easy.  Unfortunately, I don' have any examples of integrating it into Rails because, my previous app was so simple, I just re-wrote it completely as a a MERB app.  MERB now supports RESTful routes, including nesting, so it made much of my code, copy and paste.

All you'd need to do is have Rails do it's upload post request to your MERB server, it runs on port 4000 by default, and it should be that easy.

***WARNING: Much of this code is taken from the attachment_fu plugin. ***

Here is a little example of my upload action...

   @upload = UserFile.new
   @upload.attributes = {
     :filename      =>  params[:data][:filename],
     :filesize      =>  params[:data][:size],
     :content_type  =>  params[:data][:content_type],
     :user_id       =>  current_user
   }
   @upload.save!
   destination = MERB_ROOT+"/public/uploads/#{@upload.filename}"
   FileUtils.mv params[:data][:tempfile].path, destination
   render_no_layout

And my Model code, to clean up the IE file path bugs...

class UserFile < ActiveRecord::Base
  before_save :sanitize

  def sanitize
     self.filename.gsub!(/^.*(\\|\/)/, '')
     self.filename.gsub!(/[^\w\.\-]/,'_') 
  end
end

I'm still refactoring my code, so don't beat me up on it too bad.  I'm very impressed with MERB, I've tested it with files as large as 1.5GB, and it handled it with no problems at all, while still managing other uploads (Something that's impossible in Rails).

As Robert suggests, backgrounddb is great, but I just found MERB to be easier to implement in this case, and it's been something I've been wanting to mess around with for a while.

Zack

Last edited by zreed20 (2007-10-22 06:47:30)

But they that wait upon the Lord shall renew their strength

Re: FTP file upload

Thanks Zack!

Ryan Heneise  |  Art of Mission  |  Now with extra-strong Donor Tools mojo

Re: FTP file upload

Awesome post Zack!

You finally gave me the push I needed to try it out. Who knows... I may just end up doing my whole app in Merb!

Re: FTP file upload

Well I found this by searching form very same exact thing. Though I have not retried Merb yet, I would rather understand for to get the upload to ftp portion working first before I hop to merb again.

So for me i simply have this
ftp.putbinaryfile(self.temp_path, self.name, 1024)
for the old code of File.open that works fine, but I just can't figure out how to make the self.see_the_client_side_file
All I get back are reports that I have no file named "so and so" If it is not the the uploaded path already . But if it's in the upload path already it will upload, so I know it works, it is just seeing it incorrectly located.

On further investigation I guess CGI is streaming the location of the object when in HTTP mode and net/ftp mode does not have that ? I know thats wrong.

Any ideas ? All I want to do is replicate the form upload but use the ftp instead of File.open.write.read

Re: FTP file upload

Hi Zack! I have been struggling with this exact same issue myself.

file_field does not seem to be able to pass on a 'filepath' - maybe for security reasons?

Were you ever able to figure this on out?

Thanks!
Adam

Re: FTP file upload

Adam,

I kind of gave up on using Net/FTP and went the MERB route.  It's worked very well for us over the last few months.  I just integrated it into our existing public Rails app.  This tutorial walks you through it.

http://www.scribelabs.com/2007/10/18/fi … rt-1-of-3/

But they that wait upon the Lord shall renew their strength