The following are some useful tips for moving around x-ray data and processing in a highly platform-independent way. We begin with the "easy" tools available at 8.3.1 and then move on to a deeper explanation of how they work.
WARNING: If you moved your files after collecting them, the download program will not find them! But if your home computer is equipped with rsync, you can use that:
unix% cd /some/big/disk unix% rsync -azv mcfuser@gateway.bl831.als.lbl.gov:/data/mcfuser/your/stuff/ ./
Firewire
If you have a Mac-formatted firewire drive, just plug it into the firewire cable marked "iMac", open up a terminal, and type this:
[iMac:~]% df
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/disk0s5 117221400 31035432 8567...The following are some useful tips for moving around x-ray data and processing in a highly platform-independent way. We begin with the "easy" tools available at 8.3.1 and then move on to a deeper explanation of how they work.
WARNING: If you moved your files after collecting them, the download program will not find them! But if your home computer is equipped with rsync, you can use that:
unix% cd /some/big/disk unix% rsync -azv mcfuser@gateway.bl831.als.lbl.gov:/data/mcfuser/your/stuff/ ./
Firewire
If you have a Mac-formatted firewire drive, just plug it into the firewire cable marked "iMac", open up a terminal, and type this:
[iMac:~]% df
Filesystem 512-blocks Used Avail Capacity Mounted on
/dev/disk0s5 117221400 31035432 85673968 26% /
devfs 180 180 0 100% /dev
fdesc 2 2 0 100% /dev
<volfs> 1024 1024 0 100% /.vol
automount -fstab [326] 0 0 0 100% /Network/Servers
automount -static [326] 0 0 0 100% /automount
//mcfuser@BL831/DATA 1341095936 858587136 482508800 64% /Volumes/mcfuser
/dev/disk2s2 234441296 787160 233654128 0% /Volumes/Untitled
[iMac:~]% rsync -av server:/data/mcfuser/you/ /Volumes/Untitled/831_2-14-04
This will recursively transfer everything under the "you" directory into /Volumes/Untitled/831_2-14-04. The 831_2-14-04 folder will be created if neccesary. The rsync program will only transfer files you havn’t already got, so just run this same command every so often to keep your firewire drive synched-up with data collection. Alternately, if you want to just lauch something and forget about it, you can run this:
[iMac:~]% rsh server "download holton" | tar xvBCf /Volumes/Untitled -
This will transfer all data files containing "holton" to the firewire disk named "Untitled" in real time. Make sure your name and your firewire disk’s name are entered appropriately.
If you have a PC-formatted firewire drive, just plug it into the firewire or USB2 cable marked "PC".
Then go to a terminal on graphics1 and type this:
graphics1% ssh server
bl831% cd /data/mcfuser/yourname
bl831% backup_firewire.com .
This will check /data/mcfuser/yourname at least every ten minutes and transfer any new files to your firewire drive. The contents of your firewire drive will appear as /firewire on "graphics1" and "server". So, you can check on it periodically with:
graphics1% ls /firewire
When you’re done, check that all your data are on the firewire disk, and then just pull the plug and go home.
The Hard Way
(understanding file-transfer commands)
Copying one file at a time:
unix% cp test_0_001.img /somewhere/else/test_0_001.img
for hundreds of files, this can get boring really fast. The "cp" command also has the unfortunate habit of changing the date stamps on files, and this can present problems latter on when you’re trying to figure out how the data were collected, etc. You can do this (on some systems):
unix% cp -p test_0_001.img /somewhere/else/test_0_001.img
to preserve the dates, but you are still stuck with the tedium of typing in a lot of cp commands. This is especially annoying if you need to rename or renumber the files. However, there is a better way:
Renaming a bunch of files:
The following commands may seem like a lot of fancy typing, but it really is the most flexible way of doing this. Just cut-and-paste from this web page into your terminal and you’ll never have to understand how it works. If you do want to understand, have a look at the tips n tricks page.
unix% foreach num ( `awk 'BEGIN{for(num=1;num<=180;++num) printf "%03d ", num; exit}'` )
foreach? mv stupidname${num}.img bettername_${num}.img
foreach? end
re-names all the x-ray images from stupidname001.img through stupidname180.img to bettername_001.img through bettername_180.img. I’ll bet there have been plenty of times you wished you knew how to do that!
If you also want to re-number your images, you can add one line:
unix% foreach num ( `awk 'BEGIN{for(num=12;num<=180;++num) printf "%03d ", num; exit}'` )
foreach? set newnum = `echo ${num} | awk '{printf "%03d ", $1-11}'`
foreach? mv stupidname${num}.img bettername_${newnum}.img
foreach? end
re-names all the x-ray images from stupidname012.img through stupidname180.img to bettername_001.img through bettername_169.img.
Whatever you do, DO NOT DO THIS -> mv frame_*.img bettername_*.img <- NEVER EVER DO THIS!
The above command may seem like a good idea, but it will actually destroy all but the last of your images!
Copying a whole directory system:
A popular way of doing this is to use the "-r" flag in cp.
unix% cp -r /data/wrong/place /data/right/
This will recurse into subdirectories and copy them too. However, this will usually mess up your file dates, change the file permissions (read,write,execute) and usually change the ownership. If you want yo make sure everything about the files is preserved, you want to do something like this:
unix% cd /data/wrong/place
unix% tar cf /tmp/deleteme.tar .
unix% cd /data/right/place
unix% tar xvf /tmp/deleteme.tar
unix% rm /tmp/deleteme.tar
Will copy every file in /data/wrong/place to /data/right/place.
This command sequence may seem like a lot more work than "cp -r", but the advantages to using "tar" will become apparent below. The tar command also has the nice feature of recursing into all subdirectories but preseres all file dates, permissons and ownerships. The "tar" command (tape archiver) is also one of the oldest backup/restore programs so you can move the deleteme.tar file to just about any kind of computer system (Unix,Windows,Mac) and be able to restore all your data. The tar options "cf" mean "create a new archive" and "put it in this file" where "this" file is the next argument. The "xvf" options mean "extract" and "verbose" and "from this file" where "this" file is the next argument.
The main distadvantage of the above five commands is that the copy can take twice as long as cp -r, because you are generating a huge intermediate deleteme.tar file. You also might not have enough disk space. The following command solves both these problems:
unix% cd /data/wrong/place
unix% tar cf - . | ( cd /data/right/place ; tar xvf - )
Will copy every file in /data/wrong/place to /data/right/place.
In this case, instead of "deleteme.tar", we use the special filename "-" which means "dump to screen" or "take from screen" depending on wether you are creating or restoring. Running the tar cf - . command by itself would probably be a bad idea. However, unix allows you to send a programs "screen" output to another program by using a pipe "|". Therefore, the "screen" output of tar cf - . becomes the "screen" input to the tar xvf - command. The bytes that would have gone into "deleteme.tar" are still created and extracted, but only exists in the etherial world of unix pipes. Unix pipes consume no extra bandwidth and no disk space.
The "()" just allow you to change diretories before running the tar xvf - command. Some tar programs allow you to use "tar xvCf /data/right/place -" instead of "( cd /data/right/place ; tar xvf - )", but not all systems support this. the "()" thing will always work.
Copying a whole directory system from one computer to another:
The tar program has real advantages when you need to move data from one computer to another.
unix% cd /data/wrong/place
unix% tar cBf - . |\
ssh -n -l username othercomputer.college.edu "cd /data/right/place ; tar xvBf -"
This command shuttles the data through an ssh login session. (This is how scp works). The commands in "" are executed on othercomputer, and their "screen input" is taken from the "screen input" of the ssh program (the output of tar cBf - .). In this way a tar file can be "streamed" over a secure connection to any other unix system that you have an account on. You will still be able to enter your password on the usual prompt, and this will not interfere with the data transfer.
The additional "B" in each tar command is a universal way to tell each computer’s tar program to use the same block size. Omitting the "B" can lead to problems if you are transferring data between different flavors of unix.
In our modern world of firewalls, you may find yourself on the wrong side of one. Your data is on a remote computer, you can log into it, but you can’t connect back to your own system. In that case, use this command:
home% cd /data/right/place
home% ssh -n -l username bcsb-downolad.als.lbl.gov "cd /data/bl822/wrong/place ; tar cBf - ." |\
tar xvBf -
In this case, you want to get data you collected into /data/wrong/place on beamline 8.2.2. The files will be expanded into /data/right/place on your current (home) machine.
For slow connections, you might want to use compression:
unix% cd /data/wrong/place
unix% tar cBf - . |\
compress -c |\
ssh -l username othercomputer.college.edu "cd /data/right/place ; uncompress -c | tar xvBf -"
This command adds a compress filter into the pipe, so that the bytes sent over the network are in compressed form. For x-ray images, this usually means a factor of 2-3 increase in transfer rate. On the other end, the uncompress command restores the archive stream to its original size and tar behaves just as it does in the last example.
The reverse direction would be this:
home% cd /data/right/place
home% ssh -n -l username bcsb-download.als.lbl.gov "cd /data/bl822/dcsuser/wrongplace ; tar cBf - . | compress -c" |\
uncompress -c | tar xvBf -
Note: you will only see the increase in speed for slow networks. If you do this over gigabit, the CPU overhead from compression and decompression will only slow things down.
Copying the part of a whole directory system that didn’t make it on the first try:
Network connections can break, and when they do you are left with the problem of figuring out which files made it and which files still need to be transferred. This can be a little tricky if you didn’t plan ahead.
1) Plan ahead:
unix% cd /data/wrong/place
unix% find . -type f -name ‘*.img’ >! ~/2bsent.txt
unix% tar cTBf ~/2bsent.txt - |
ssh -l username othercomputer.college.edu "cd /data/right/place ; tar xvBf -"
The find command will recurse into all subdirectories, looking for files that match certain criteria. In this case the rules are: it has to be a "regular file" (not a directory or other special file), and it must end in ".img". The output of the find command will be a one-line-per-file listing of each file, pathnames included. It’s just like "ls -1", expect that it recurses into subdirectories. The T flag to tar means to take the list of files to archive from the contents of the next file, which is "~/2bsent.txt". The "T" is the option used under Linux. On an SGI, this is an "L" instead of a "T" and on a Sun it’s an "I".
Creating ~/2bsent.txt insures that you know the order in which the files will be sent. This lets you "continue" the transfer smoothly if it ever gets interrupted. For example, if the connection breaks and the last file you see is "random_0_034.img", type this command to continue the transfer:
unix% awk ‘/random_0_034.img/{p=1} p==1{print}’ ~/2bsent.txt |
tar cTBf - - |
ssh -l username othercomputer.college.edu "cd /data/right/place ; tar xvBf -"
The short awk program will only pass lines in ~/2bsent.txt after or including a line containing the text "random_0_034.img". This will repeat the transmission of random_0_024.img (which is probably incomplete anyway) and commence sending every file that was not already sent.
2) Didn’t plan ahead: Okay, so you didn’t plan ahead. No problem. You may also find yourself in the situation if you have collected more data since you started your last transmission, and the 2bsent.txt file above has become out-of-date already. In either case, you’d probably rather not waste time transferring a bunch of files that you already have. To make sure you don’t do this, you need to make a list of all the files you want to transfer. Then make a list of all the files that have transferred sucessfully. Then, finally, take the difference between these lists.
First, lets find out what we already did:
unix% cd
unix% ssh -l username othercomputer "cd /data/right/place ; find . -type f -exec ls -ln \{\} \;" |\
awk '{print $5,$NF}' |\
sort >! alreadysent.txt
This will create a list of files and their sizes in alreadysent.txt. Like this: 10617344 ./fpp/tetPEG1_fpp_1_001.img 10617344 ./fpp/tetPEG1_fpp_1_002.img 10617344 ./fpp/tetPEG1_fpp_1_003.img 10617344 ./fpp/tetPEG1_fpp_1_004.img 10617344 ./fpp/tetPEG1_fpp_1_005.img 10617344 ./fpp/tetPEG1_fpp_1_006.img etc... This example takes advantage of the -exec flag for find. This will run a given command on each found file. The filename is substituted for the {} thing. Here we call the ls command to get size information about each file. You want to include the size so you can detect the one file that was only partially transferred.
Now let’s make a list of all files that should, eventually, be transferred.
unix% cd /data/wrong/place
unix% find . -type f -exec ls -ln {} ; |
awk ‘{print $5,$NF}’ |
sort >! ~/shouldbesent.txt
Now that you have both lists, you just need to see the differences between them unix% cd unix% diff shouldbesent.txt alreadysent.txt | awk ‘/^</{print $NF}’ >! 2bsent.txt
Now 2bsent.txt contains the names of all remaining files to send. The diff program shows the differences between two files, and it prints out a "<" character before lines that occur in the left file. The awk program looks for lines beginning with "<" and prints out the last word on the line (the name of the untransferred file).
Now you can just send these files via tar and ssh. Just like the first example.
unix% tar cTBf ~/2bsent.txt - |
ssh -l username othercomputer.college.edu "cd /data/right/place ; tar xvBf -"
Another increasingly popular way of doing incremental transfers like this is to use rsync. You can man rsync to find out more. However, rsync usually requires unacceptably low security for transferring data from synchrotron to home.
Live transfer of data as it is collected
Even over a fast link like firewire or gigabit ethernet, multiple-gigabyte data transfers can take a really long time. For example, transferring 40Gb of data onto a firewire drive over a 100BaseT link will take about 6 hours (in reality). However, data can usually be transferred much faster than it is collected. Why not just transfer as you go? Unfortunately, "ordinary" data transfer programs like ftp or scp will only do batch transfers of files that already exist. However, tar can (usually) be fooled into generating the archive stream on-the-fly. This works on Linux and SGI but not Suns or Alphas.
1) follow the beamline control system
On beamlines running BLU-ICE, you can generate a "live" list of collected images like this:
unix% xos3_channel.tcl | awk '/image_ready/{print $NF}'
This list of filenames can be fed into tar just like we did above.
unix% xos3_channel.tcl | awk '/image_ready/{print $NF}' |\
tar cBTf - - |\
ssh -l user@othercomputer.college.edu "cd /some/big/disk ; tar xvBf -"
This will send all new files to /some/big/disk on othercomputer. You can also do this from the "other direction"
unix% cd /some/big/disk
unix% ssh -l mcfuser gateway.bl831.als.lbl.gov "xos3_channel.tcl | awk '/image_ready/{print $NF}' | tar cBTf - - " |\
tar xvBf -
On beamlines running MCF/DCS, you might be able to do a "live" file transfer like this:
bl822c% mkfifo fifo$$
bl822c% tail -f /prod/DCS/log/hwserv.log | awk '/img/{print $NF}' >> fifo$$ &
bl822c% tar cBf - -I fifo$$ |\
ssh -l user@othercomputer.college.edu "cd /some/big/disk ; tar xvBf -"
2) poll the file system
The following command will poll the filesystem every five minutes for new files and list them on the screen
unix% echo "/data/mcfuser semet_1_E1_025.img" |\
awk '{framedir=$1;firstfile=$NF;\
# start sending everything if no firstfile specified\
if(firstfile == "") sending=1;\
if(firstfile != "") firstfile="-cnewer "firstfile;\
cmd="find "framedir" -type f "$firstfile;\
while(1){\
# keep searching the directory every 300 seconds \
while(cmd | getline){\
# ignore entries before the first file \
if($NF == firstfile){sending=1};\
# pretend earlier entries were already sent \
if(! sending) sent[$0]=1;\
if((sending)&&(! sent[$0])){\
# keep an internal record of files we have sent \
print $NF; ; sent[$0]=1;};\
};\
# wait five minutes between polls \
close(cmd); system(" sleep 300 ")}}'
This is kind of complicated, but it does work. This will soon be packaged into new_files.com.
Firewire disks
If you have shared your firewire disk, as demonstrated above, you can shuttle data to it using smbclient like this:
unix% xos3_channel.tcl | awk '/image_ready/{print $NF}' |\
tar cBTf - - |\
smbclient //192.168.10.5/share -U "username%password" -Tx -
Where 192.168.10.5 is the IP address of the computer that is sharing the firewire disk as "share". Under Windows, you can use the "ipconfig" command under "Programs:Acessories:Command Prompt" to see what your IP address is.
However, it is probably easier to use backup_firewire.com.
This page is not finished. It will never be finished, and neither will yours. Admit it.