Transferring and hosting files and images

From Helpful
Jump to: navigation, search
This hasn't been updated for a while, so could be outdated (particularly if it's about something that evolves constantly, such as software).
This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Stuff vaguely related to storing, hosting, and transferring files and media:

Transfer

SSH and such

Note: When setting up a server for this, you may be interested in Allowing SCP/SFTP but not login

SCP

SCP refers to utilities using SSH for data transfer.

scp
can be use as something of a drop-in of
cp
that can take remote-host path specifications in the form user@host:path (don't omit the colon; scp would consider it a local filename).

Since it copies via SSH mechanims, you get the same login/account details, which can be rather convenient (in terms of firewalling, authentication) where those are already set up.


Examples:

scp some.txt fork@example.com:~/text/
scp fork@example.com:~/text/ .


Notes:

  • While scp can work on wildcards, it launches a process per resolved file(verify), so will be slow when you are transferring many smaller files.
  • Note that if you are copying large amounts of small files, you may want to use the ssh/tar trick instead.
  • In target specs, the default directory is the user's home directory. Also, tildes are understood, so the following are equivalent
scp some.txt fork@example.com:
scp some.txt fork@example.com:~
scp some.txt fork@example.com:~myusername
scp some.txt fork@example.com:/home/myusername



SFTP

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

Uses SSH for data transfer (so uses same login/account details).

SFTP refers to a somewhat more copy-geared protocol (than SCP) that can be used as a secure file transfer protocol.

ssh as a pipe

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

You can (ab)use the fact that when you hand a command to ssh, stdin and stdout on both ends go through the tunnel and are connected to stdin and stdout of the command on the other end.

You can e.g. combine this with tar to get file transfer, like

# tars local directory (with compression), sends to remote end, untars within a given directory
tar -czf - /some/path/ | ssh user@host "tar -xzf - -C /destination"
 
# tars local directory, sends to remote end (avoids storing file locally. 'streaming backup')
tar -cjf - /some/path/ | ssh user@backuphost "cat > /raiddisk/backups/2012-01-01.tgz"


When copying many small files, this can be faster than scp, sftp, and other methods that handle each file separately. When making backups or verbatim copies, it avoids having intermediate temporary (and possibly large) archives on-disk as part of the process.


Remote-to-local transfers are similar. You give a command to run remotely instead of a shell, which outputs data that you can use. For example, to extract that last backup archive to the current directory:

ssh user@backuphost "cat /backups/2009-01-01.tgz" | tar xvf -

FTP

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

GridFTP

P2P

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

The decentralized nature of P2P programs can be convenient when you need to make many copies something large on a LAN sort of scale. For example, when distributing a dataset to all nodes in a processing cluster (or games in a LAN party), this can be an easy and network-friendly/efficient method.


You could e.g. use torrent with uTorrent's built-in tracker, though there are probably more user-friendly alternatives.

Limited lifespan web storage

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)
This hasn't been updated for a while, so could be outdated (particularly if it's about something that evolves constantly, such as software).

Note: All of these sites and rules/figures may change at any time. Take prices and sizes as estimates.


  • Yousendit [1]
  • wetransfer (verify)
    • 2GB max size
  • [4]
    • 1GB for free, single-user account. Various larger-scale plans
    • win+osx
  • mediafire [5]
  • magicVORTEX [6]
  • sendspace [8]
  • MegaUpload [10] [11]
    • Free users have to wait ~45sec for download and can download one file at a time (incentives to pay)


Backup and sync, more personal

This article/section is a stub — probably a pile of half-sorted notes, is not well-checked so may have incorrect bits. (Feel free to ignore, fix, or tell me)

http://en.wikipedia.org/wiki/List_of_online_backup_services


  • Dropbox [14]
    • sync tool between systems, backup; online storage
    • win+mac+lin app, plus web interface
    • Free: 2GB storage; Paid: 50/100GB max storage
    • public/private (can share somethings publicly (also use as email attachment replacement), can share specific folders with other dropbox users (automatically synched))
  • IDrive (also/formerly i-drive, Anuvio)
    • win, 2GB free, a few plans up to <1TB[15]
  • IDriveSync
    • win, 2GB free (≤5 computers), $5/month for unlimited (≤5 computers)(verify)


  • Mozy
    • win+mac, 2GB free, $5/month/computer for unlimited(verify)
  • Jungle Disk [16]
    • Simplest plan: 2$ basic fee, 5GB free, each dollar/month buys ~6.6GB more.


  • ZumoDrive [17] [18]
    • win+lin+osx (also some mobile)
  • drop.io [26] [27]
    • Free: 100MB max storage,
    • Paid: USD10.00year/GB (max 25 GB)

Media Hosting

Images and/or video

See e.g.:



Files at home

Note that things like dropbox can also be handy for this .


SMB (Windows sharing, samba)

NAS and SAN

Media sharing

See: