
平台论坛博客文库

论坛徽章:: 7

11楼 [报告]

发表于 2005-05-25 21:47 |只看该作者

comp.unix.shell FAQ(转载http://home.comcast.net/~j.p.h/)

24. How do I use shell variables in awk scripts
Short answer = either of these, where "svar" is a shell variable
and "avar" is an awk variable:
awk -v avar="$svar" '... avar ...' file
awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}... avar ...' "$svar" file
depending on your requirements for handling backslashes and
handling ARGV[] if it contains a null string (see below for details).
Long answer = There are several ways of passing the values of
shell variables to awk scripts depending on which version of awk
(and to a much lesser extent which OS) you're using. For this
discussion, we'll consider the following 4 awk versions:
oawk (old awk, /usr/bin/awk and /usr/bin/oawk on Solaris)
nawk (new awk, /usr/bin/nawk on Solaris)
sawk (non-standard name for /usr/xpg4/bin/awk on Solaris)
gawk (GNU awk, downloaded from [url]http://www.gnu.org/software/gawk)[/url]
If you wanted to find all lines in a given file that match text
stored in a shell variable "svar" then you could use one of the
following:
a) awk -v avar="$svar" '$0 == avar' file
b) awk -vavar="$svar" '$0 == avar' file
c) awk '$0 == avar' avar="$svar" file
d) awk 'BEGIN{avar=ARGV[1];ARGV[1]=""}$0 == avar' "$svar" file
e) awk 'BEGIN{avar=ARGV[1];ARGC--}$0 == avar' "$svar" file
f) svar="$svar" awk 'BEGIN{avar=ENVIRON["svar"]}$0 == avar' file
g) awk '$0 == '"$svar"'' file
The following list shows which version is supported by which
awk on Solaris (which should also apply to most other OSs):
oawk = c, g
nawk = a, c, d, f, g
sawk = a, c, d, f, g
gawk = a, b, c, d, f, g
Notes:
1) Old awk only works with forms "c" and "g", both of which have
problems.
2) GNU awk is the only one that works with form "b" (no space
between "-v" and "var="). Since gawk also supports form "a",
as do all the other new awks, you should avoid form "b" for
portability between newer awks.
3) In form "c", ARGV[1] is still getting populated, but
because it contains an equals sign (=), awk changes it's normal
behavior of assuming that arguments are file names and now instead
assumes this is a variable assignment so you don't need to clear
ARGV[1] as in form "d".
4) In light of "3)" above, this raises the interesting question of
how to pass awk a file name that contains an equals sign - the
answer is to do one of the following:
i) Specify a path, e.g. for a file named "abc=def" in the
current directory, you'd use:
awk '...' ./abc=def
Note that that won't work with older versions of gawk or with
sawk.
ii) Redirect the input from a file so it's opend by the shell
rather than awk having to parse the file name as an argument
and then open it:
awk '...' < abc=def
Note that you will not have access to the file name in the
FILENAME variable in this case.
5) An alternative to setting ARGV[1]="" in form "d" is to delete
that array entry, e.g.:
awk 'BEGIN{avar=ARGV[1];delete ARGV[1]}$0 == avar' "$svar" file
This is slightly misleading, however since although ARGV[1]
does get deleted in the BEGIN section and remains deleted
for any files that preceed the deleted variable assignment,
the ARGV[] entry is recreated by awk when it gets to that
argument during file processing, so in the case above when
parsing "file", ARGV[1] would actually exist with a null
string value just like if you'd done ARGV[1]="". Given that
it's misleading and introduces inconsistency of ARGV[]
settings between files based on command-line order, it is
not recommended.
6) An alternative to setting svar="$svar" on the command line
prior to invoking awk in form "f" is to export svar first,
e.g.:
export svar
awk 'BEGIN{avar=ENVIRON["svar"]}$0 == avar' file
Since this forces you to export variables that you wouldn't
normally export and so risk interfering with the environment
of other commands invoked from your shell, it is not recommended.
7) When you use form "d", you end up with a null string in
ARGV[1], so if at the end of your program you want to print
out all the file names then instead of doing:
END{for (i in ARGV) print ARGV[i]}
you need to check for a null string before printing. or
store FILENAMEs in a different array during processing.
Note that the above loop as written would also print the
script name stored in ARGV[0].
8) When you use form "a", "b", or "c", the awk variable
assignment gets processed during awks lexical analaysis
stage (i.e. when the internal awk program gets built) and
any backslashes present in the shell variable may get
expanded so, for example, if svar contains "hi\there"
then avar could contain "hi<tab>there" with a literal tab
character. This behavior depends on the awk version as
follows:
oawk: does not print a warning and sets avar="hi\there"
sawk: does not print a warning and sets avar="hi<tab>here"
nawk: does not print a warning and sets avar="hi<tab>here"
gawk: does not print a warning and sets avar="hi<tab>here"
If the backslash preceeds a character that has no
special meaning to awk then the backslash may be discarded
with or without a warning, e.g. if svar contained "hi\john"
then the backslash preceeds "j" and "\j" has no special
meaning so the various new awks each would behave differently
as follows:
oawk: does not print a warning and sets avar="hi\john"
sawk: does not print a warning and sets avar="hi\john"
nawk: does not print a warning and sets avar="hijohn"
gawk: prints a warning and sets avar="hijohn"
9) None of the awk versions discussed here work with form "e" but
it is included above as there are older (i.e. pre-POSIX) versions
of awk that will treat form "d" as if it's intended to access a
file named "" so you instead need to use form "e". If you find
yourself with that or any other version of "old awk", you need
to get a new awk to avoid future headaches and they will not be
discussed further here.
So, the forms accepted by all 3 newer awks under discussion (nawk,
sawk, and gawk) are a, c, d, f, and g. The main differences between
each of these forms is as follows:
|-------|-------|----------|-----------|-----------|--------|
| BEGIN | files | requires | accepts | expands | null |
| avail | set | access | backslash | backslash | ARGV[] |
|-------|-------|----------|-----------|-----------|--------|
a) | y | all | n | n | y | n |
c) | n | sub | n | n | y | n |
d) | y | all | n | n | n | y |
f) | y | all | y | n | n | n |
g) | y | all | n | y | n/a | n |
|-------|-------|----------|-----------|-----------|--------|
where the columns mean:
BEGIN avail = y: variable IS available in the BEGIN section
BEGIN avail = n: variable is NOT available in the BEGIN section
files set = all: variable is set for ALL files regardless of
command-line order.
files set = sub: variable is ONLY set for those files subsequent
to the definition of the variable on the command line
requires access = y: variable DOES need to be exported or set on
the command line
requires access = n: shell variable does NOT need to be exported
or set on the command line
accepts backslash = y: variable CAN contain a backslash without
causing awk to fail with a syntax error
accepts backslash = n: variable can NOT contain a backslash without
causing awk to fail with a syntax error
expands backslash = y: if the variable contains a backslash, it IS
expanded before execution begins
expands backslash = n: if the variable contains a backslash, it is
NOT expanded before execution begins
null ARGV[] = y: you DO end up with a null entry in the ARGV[]
array
null ARGV[] = n: you do NOT end up with a null entry in the ARGV[]
array
For most applications, form "a" and "d" provide the most intuitive
functionality. The only functional differences between the 2 are:
1) Whether or not backslashes get expanded on variable assignment.
2) Whether or not ARGV[] ends up containing a null string.
so which one you choose to use depends on your requirements for
these 2 situations.
======================================================================
25. How do I get input from the user with a timeout?
In bash or ksh93 you can use the read built-in with the "-t"
option.
In zsh, use the zsh/zselect module.
You can also use your terminal capability to do that.
{
s=$(stty -g)
stty -icanon min 0 time 100
var=$(head -n 1)
stty "$s"
}
For a 10 second timeout (reset at each key press).
======================================================================

复制代码

[ 本帖最后由 r2007 于 2006-4-20 22:03 编辑 ]

实战分享：从技术角度谈机器学习入门| 【大话IT】RadonDB低门槛向MySQL集群下战书 | ChinaUnix打赏功能已上线！ | 新一代分布式关系型数据库RadonDB知多少？

r2007

广告杀手

论坛徽章:: 7

12楼 [报告]

发表于 2005-05-25 21:47 |只看该作者

comp.unix.shell FAQ(转载http://home.comcast.net/~j.p.h/)

26. How do I get one character input from the user?

In bash this can be done with the "-n" option to read.
In ksh93 it's read -N
In zsh it's read -k

More portably:

OLDSTTY=$(stty -g)  # save our terminal settings
stty cbreak  # enable independent processing of each input character
ONECHAR=$(dd bs=1 count=1 2>/dev/null)  # read one byte from standard in
stty "$OLDSTTY"  # restore the terminal settings

Use the `something` format if your shell doesn't understand
$(something).  This reads from standard input, which may or may not
be desirable.  If you want to read from the terminal regardless of
where standard input is, add "if=$(tty)" to the dd command.

======================================================================

27. why isn't my .profile read?

~/.profile is only read for login shells.  In short if you don't
see a login prompt then your ~/.profile isn't being read.  You
can fix this by either porting all of the things in your
~/.profile to /etc/profile or your shells rc script such as
~/.bashrc or ~/.zshrc.

You may have to set the ENV variable in your login shell to get
the .*rc shell read. See the man page for your shell to understand
how it works.

======================================================================

28. why do I get "[5" not found in "[$1 -eq 2]"?

Because you didn't RTFM :-)

"[" is an alias for the "test" command. As such, it's called by a
script like any other command (this applies even if test is
builtin). Since the command line uses spaces to separate a command
from its arguments, you have to put a space between '[' and its
argument. So:

   $ [ -f xxx ] isn't the same as
   $ [-f xxx ]

In the latter case, the shell will think that "[-f" is the
command, not "[" with arguments "-f xxx ]

======================================================================

29. How do I exactly display the content of $var (with a \n appended).

A: on POSIX systems or with shells with builtin printf (bash2,
ksh93, zsh4.1, dash...)

printf '%s\n' "$var"

(except for memory/environment full errors, should be expected
to work at least if $var is not longer than LINE_MAX (supposed
to be at least _POSIX2_LINE_MAX == 2048), no hardcoded limits in
zsh/ksh/bash/dash builtins)

ksh, zsh:
print -r -- "$var"

zsh:
echo -E - "$var"

Other bourne like shells:

cat << EOF
$var
EOF
(creates a temporary file and forks a process)

expr "x$var" : 'x$.*$'
(limited to 126 characters with some exprs, may return a
non-null exit code).

With ash:
(unset a; ${a?$var}) 2>&1

printf %s "$var"    # posix
print -rn -- "$var" # zsh/ksh
echo -nE - "$var"    # zsh

awk 'NR>1{print ""}{printf("%s",$0)}' << EOF
$var
EOF

======================================================================

30. How do I split a pathname into the directory and file?

The most portable way of doing this is to use the external
commands dirname(1) and basename(1), as in

   pathname='/path/to/some/file'
   dir=`dirname "$pathname"`
   file=`basename "$pathname"`

However, since this executes an external command, it's slower than
using shell builtins (if your shell has them). For ksh, bash, zsh
and POSIX shells the following will do the same thing more
efficiently:

   pathname=/path/to/some/file
   file=${pathname##*/}

To get the directory using the shell builtin, you should first
ensure that the path has a '/' in it.

   case $pathname in
      */*) dir=${pathname%/*};;
      *) dir=''
   esac

In zsh, (abd csh, tcsh), you have

   ${pathname:h} (head) ${pathname:t} (tail).

======================================================================

[ 本帖最后由 r2007 于 2005-12-3 17:07 编辑 ]

r2007

广告杀手

论坛徽章:: 7

13楼 [报告]

发表于 2005-05-25 21:48 |只看该作者

comp.unix.shell FAQ(转载http://home.comcast.net/~j.p.h/)

31. How do I make an alias take an argument?

In Bourne-derived shells aliases cannot take arguments, so if you
need to be able to do that, define a shell function rather than
an alias.

Aliases are often used to reduce the need to type long command
lines:

   alias prog='/opt/bin/prog -x -y -z --long-option'

Or to make a command default to using certain parameters:

   alias ls='ls -F'

Shell functions must be used when arguments are needed. For
example, this will move one or more files to a Trash directory:

   trash() { mv -- "$@" ~/.Trash; }

======================================================================

32. How do I deal with a file whose name begins with a weird character

Do something to hide the weird character from the command being
used. Assuming that command is rm, try things like

   rm ./-foo
   rm -- -foo
   rm -i -- * (and then decide what you want to delete interactively)

   If the weird character is not printable, the last option may be
   your best bet. Another possibility in that case is to pipe the
   output of ls into od -c and figure out what the weird character
   is. Then use sed to isolate it and nuke it. However, the rm -i
   approach is probably much safer.

For more particulars, see the rm man page for your system.

======================================================================

33. Why do I lose the value of global variables that are set in a loop.

Given the following program

   #!/bin/sh
   x="this is the initial value of x"
   cat dataFile | while read line;do
      x="$line"
   done
   echo x = $x

   You may get the following for output

   x = this is the initial value of x

   This is because in the Bourne shell redirected control structures
   run in a subshell, so the value of x only gets changed in the
   subshell, and is lost when the loop ends.

   In other shells the same result may be seen because of the way
   pipelines are handled. In shells other than ksh (not pdksh) and
   zsh elements of a pipeline are run in subshells. In ksh and zsh,
   the last element of the pipeline is run in the current shell.

   An alternative for non-Bourne shells is to use redirection
   instead of the pipeline

   #!/bin/sh
   x="this is the initial value of x"
   while read line;do
      x="$line"
   done < dataFile
   echo x = $x

With a Bourne shell you need to reassign file descriptors, so no
pipline or redirection in the loop is involved.

   exec 3<&0       # save stdin
   exec < file
   while read line; do
      x=$line
   done
   exec 0<&3       # restore stdin

Note that putting #!/bin/sh at the top of a script doesn't
guarantee you're using the Bourne shell. Some systems link /bin/sh
to some other shell. Check your system documentation to find out
what shell you're really getting in this case.

======================================================================

34. How do I batch a FTP download/upload?

The best way to handle this is with ncftpput and ncftpget which
are part of the ncftp program.  ncftpput -u username -p password
somewebsite.com /pics *jpg The above usage of the username and
password is not recomend though as it will be seen by anyone using
"ps" while the script is running.  ncftp has a way to handle that
as well.  Just create a file with the information in the following
formate:

            host somewebsite.com
            user username
            pass password

Then just use the -f option on the ncftp program:
ncftpput -f /home/username/somefile somewebsite.com /pics *jpg

ncftp can be found at  http://freshmeat.net/projects/ncftp/

If you want to do this interactively, there's no need to keep the
password in a file. For example, if you're building a program on
one machine, but testing it on another, and you have to keep
ftp'ing the files, you can cut down on typing by doing something
like

#!/bin/ksh
ftp -n $remote_host <<EOF
user <username>
cd <path to build directory>
prompt
bin
mget "$@"
bye
EOF

The ftp program will automatically ask you for the password, then
do the rest for you.

[ 本帖最后由 r2007 于 2005-12-3 17:08 编辑 ]

r2007

广告杀手

论坛徽章:: 7

14楼 [报告]

发表于 2005-05-25 21:49 |只看该作者

comp.unix.shell FAQ(转载http://home.comcast.net/~j.p.h/)

Appendix A: Examples

  Web sites:
  -------------------------------------------

Heiner Steven's Shelldorado site has quite a few example scripts,
tutorials, and links to other such places.

   http://www.shelldorado.com/

  Arbitrary Date Arithmetic
  -------------------------------------------

From: Tapani Tarvainen (tt@it.jyu.fi)
Subject: Re: yesterday's date under the shell
View: Complete Thread (8 articles)
Original Format
Newsgroups: comp.unix.shell
Date: 2002-02-12 07:45:05 PST

"Jean-No?l" <Jean@freckles.de> writes:

> To determine the yesterday date i do it so:
> TZ=PST+24 date +%d
> it work well but my question is:
> does this work on all systems and all shells
> or should i do it otherwise ???

No, it does not work on all systems at all times.
In some it will work practically always,
on others never, on most it works sometimes
and sometimes not. I would recommend against it.

Unfortunately there is no short and sweet portable
solution. If you have or can install Gnu date it
will do it cleanly, otherwise you can find a number
of solutions posted in this group in the past.
For a general solution you could try the following,
which should work with POSIXy shells (I've only tested
it with HP's which is essentially ksh88, though):

#! /usr/bin/sh

# Date calculations using POSIX shell
# Tapani Tarvainen July 1998, February 2001 (POSIXified)
# This code is in the public domain.

# Julian Day Number from calendar date
date2julian() #  day month year
{
   day=$1;  month=$2;  year=$3
   tmpmonth=$((12 * year + month - 3))
   tmpyear=$((tmpmonth / 12))
   print $(( (734 * tmpmonth + 15) / 24 -  2 * tmpyear + \
      tmpyear/4 - tmpyear/100 + tmpyear/400 + day + 1721119 ))
}

# Calendar date from Julian Day Number
julian2date() # julianday
{
   tmpday=$(($1 - 1721119))
   centuries=$(( (4 * tmpday - 1) / 146097))
   tmpday=$((tmpday + centuries - centuries/4))
   year=$(( (4 * tmpday - 1) / 1461))
   tmpday=$((tmpday - (1461 * year) / 4))
   month=$(( (10 * tmpday - 5) / 306))
   day=$((tmpday - (306 * month + 5) / 10))
   month=$((month + 2))
   year=$((year + month/12))
   month=$((month % 12 + 1))
   print $day $month $year
}

# Day of week, Monday=1...Sunday=7
dow() # day month year
{
   print $(( $(date2julian $1 $2 $3) % 7 + 1))
}

##################### The End ########################

Those allow rather arbitrary date computations.
For example, yesterday's date can be computed like this:

julian2date $(( $(date2julian $(date +"%d %m %Y") ) - 1 ))

--
Tapani Tarvainen

======================================================================

[ 本帖最后由 r2007 于 2005-12-3 17:09 编辑 ]