Transferring and hosting files and images

From Helpful
Jump to navigation Jump to search
This hasn't been updated for a while, so could be outdated (particularly if it's about something that evolves constantly, such as software or research).
This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Stuff vaguely related to storing, hosting, and transferring files and media:

Transfer protocols

SSH and such

Note: When setting up a server for this, you may be interested in Allowing SCP/SFTP but not login

scp

scp refers to utilities using SSH for data transfer.


scp is a near-drop-in of cp that can take remote-host path specifications in the form user@host:path (don't omit the colon; scp would consider it a local filename).

Since it copies via SSH, login/account details work the same way. If you already had access this way host, this is great (in terms of firewalling, authentication).


Examples:

scp some.txt fork@example.com:~/text/
scp fork@example.com:~/text/ .


Notes:

  • While scp can work on wildcards, it launches a process per resolved file(verify), so will be slow when you are transferring many smaller files.
  • Note that if you are copying large amounts of small files, you may want to use the ssh/tar trick instead.
  • In target specs, the default directory is the user's home directory. Also, tildes are understood, so the following are equivalent
scp some.txt fork@example.com:
scp some.txt fork@example.com:~
scp some.txt fork@example.com:~myusername
scp some.txt fork@example.com:/home/myusername
  • in various cases, rsync gives more control over what exactly to do, and if using the rsync-over-SSH flavour it works almost identically



SFTP

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Uses SSH for data transfer (so uses same login/account details).

SFTP refers to a somewhat more copy-geared protocol (than SCP) that can be used as a secure file transfer protocol.

ssh as a pipe

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

You can (ab)use the fact that when you hand a command to ssh, stdin and stdout on both ends go through the tunnel and are connected to stdin and stdout of the command on the other end.

You can e.g. combine this with tar to get file transfer, like

# tars local directory (with compression), sends to remote end, untars within a given directory
tar -czf - /some/path/ | ssh user@host "tar -xzf - -C /destination"

# tars local directory, sends to remote end (avoids storing file locally. 'streaming backup')
tar -cjf - /some/path/ | ssh user@backuphost "cat > /backups/2012-01-01.tgz"


When copying many small files, this can be faster than scp, sftp, and other methods that handle each file separately. When making backups or verbatim copies, it avoids having intermediate temporary (and possibly large) archives on-disk as part of the process.


Remote-to-local transfers are similar. You give a command to run remotely instead of a shell, which outputs data that you can use. For example, to extract that last backup archive to the current directory:

ssh user@backuphost "cat /backups/2009-01-01.tgz" | tar xvf -

FTP

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

FTPS

TFTP

GridFTP

Specific

Rsync

WebDAV

Cloud storage

P2P

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

The decentralized nature of P2P programs can be convenient when you need to make many copies something large on a LAN sort of scale. For example, when distributing a dataset to all nodes in a processing cluster (or games in a LAN party), this can be an easy and network-friendly/efficient method.


You could e.g. use torrent with uTorrent's built-in tracker, though there are probably more user-friendly alternatives.

Limited lifespan web storage

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)
This hasn't been updated for a while, so could be outdated (particularly if it's about something that evolves constantly, such as software or research).

Note: All of these sites may vary in rules, cost, and existence any time. Take everything as estimates.


  • wetransfer [1] [2]
    • Free: 2GB max size, expire earlier
    • Paid: 200GB max size


  • DropSend [3] [4]
    • win+osx
    • Free: 5 files per month, 250 MB/2 GB of storage space (verify)
    • Paid: a few different plans


  • Hightail [5] [6], previously Yousendit
    • Free: 100MB max size, 2GB total
    • Paid: multiple plans
    • size limited, time limited(verify)




Online backup and/or multiple-copies syncing

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

http://en.wikipedia.org/wiki/List_of_online_backup_services


Distinctions/features you may care about

  • hard-to-touch backup copies is in some ways very distinct with change-anytime sync.
so choose which you need
  • clients for win, lin, osx - and android, ios
  • secure storage
and whether that means it's actually secure from the company who stores it
  • ease of exposing specific files/folders for download
  • ease of syncing files/folders
  • ease of use at work
some are specifically aimed at project management
  • how much storage you get for free or reasonably little - and how stable that price is)
  • limitations of free/cheap variants (e.g. a low number of devices is common)
  • whether there are costs for transfer (instead of, or even on top of, storage size)


Most products will be "Accounts at a site" model - of which there are now many (with some shifts - a bunch don't exist anymore)



Self-hosted

When you don't care about having one copy in a datacenter, or having a size limit that feels like it's from the nineties, or just prefer the control/security/trust of hosting it yourself.


  • seaFile
  • ownCloud
  • nextCloud (fork of ownCloud)


and many other variations on this idea

Media Hosting

Images and/or video

See e.g.:



Files at home

Note that things like dropbox can also be handy for this .


SMB (Windows sharing, samba)

NAS and SAN

Media sharing

See:



Image hosting notes

Use notes

Seafile notes

Windows client

CLI

huge seafile-data

Owncloud notes

This article/section is a stub — probably a pile of half-sorted notes and is probably a first version, is not well-checked, so may have incorrect bits. (Feel free to ignore, or tell me)

Access:

  • web interface
interactive, and upload size is limited, so not useful for large-volume data
download seems flawed?


  • DAV
  • Desktop client (win, lin, osx)
builds on DAV interface, seems the easier way to use it


  • CLI (client)
single sync run
https://doc.owncloud.org/desktop/1.8/owncloudcmd.1.html
https://www.ossramblings.com/headless-owncloud-linux-sync


Errors

Could not authenticate to server: rejected Basic challenge

...when trying to mount via davfs2.

Seems to mean access denied, which e.g. includes an incorrect username or password.


ERROR committing to the database: "database is locked" Database Transaction is running, not starting another one!

Seems to mean two ownCloud processes are accessing the same database, or it wasn't unlocked after a crash?


Mostly brought out by trying to sync to a network share. ownCloud has mentioned they do not support this.

Semi-sorted

iRODS