Wpisy oznaczone tagiem bash

bash parallel processing and error handling

Napisany przez Lukasz w Lukasz dnia 2011/08/31

Another note to myself.

Here is a way how I process some program execution which is sending xml to standard output.

This part of script executes, checks and compress program xml output in parallel.

Tested for output sizes > 10GB.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

 
$JAVA_HOME/bin/java -jar $jarFile \                                                                                                             
    | tee >(bzip2  > $archFile ) \                                                                                                          
    | (xmllint --stream --noout - )                                                                                                         
 
DUMPSTATUS=(${PIPESTATUS[*]})                                                                                                               
 
commandResult=${DUMPSTATUS[0]};                                                                                                         
bzipResult=${DUMPSTATUS[1]};                                                                                                                
checkResult=${DUMPSTATUS[2]};                                                                                                               
 
if [ "$commandResult" -ne 0 ]; then                                                                                                     
    die 'ERROR::could not execute command'                                                                                        
else                                                                                                                                        
    echo 'INFO::command successfully executed'                                                                                           
fi                                                                                                                                          
 
if [ "$checkResult" -ne 0 ]; then                                                                                                           
    die 'ERROR::xml is not correct'                                                                                                  
else                                                                                                                                        
    echo 'INFO::xml is correct'                                                                                         
fi                                                                                                                                          
 
if [ "$bzipResult" -ne 0 ]; then                                                                                                            
    die 'ERROR::could not create bzip file from output'                                                                                       
else                                                                                                                                        
    echo 'INFO::compression is successful'                                                                                             
fi

, , , , , , , ,

Brak komentarzy

uploading whole directory to WebDav endpoint using curl

Napisany przez Lukasz w Lukasz dnia 2011/08/15

Note to myself :)

upload whole directory structure to webdav endpoint using curl only:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

#!/bin/bash
 
#read -p "username: " username
#stty -echo
username=
 
#read -p "Password: " password; echo
#stty echo
password=
 
dest="https://host/path/"
sourceDir="./sourceDir"
user="${username}:${password}"
 
echo "cleaning up ${dest}"
curl --request DELETE --user ${user} "${dest}" -o /dev/null
curl --request MKCOL  --user ${user} "${dest}" -o /dev/null
 
echo "uploading from ${sourceDir} to ${dest}"
for file in $(find ${sourceDir} )
do
    target="${dest}/${file#"${sourceDir}"}"
 
    echo "$file -> $target"
 
    if [ -d "$file" ]
    then
        curl --request MKCOL --user "${user}" "${target}" -o /dev/null
    else
        curl --upload-file "$file" --user "${user}" "${target}" -o /dev/null
    fi
 
done

, , , ,

1 komentarz

lockpicking production

Napisany przez Lukasz w Lukasz dnia 2011/07/13

i.e. accessing jmx through the firewall

All the following is extension of this great article which I’ve found while searching for a way to connect jconsole on my local pc to JVM running on some server to which I have only ssh access. This works fast, stable and without any problems.

Scripts

This is nice bash function which i’ve put into my ~/.bashrc

1
2
3
4
5
6
7
8
9
10

function jc {
jmx_host=$1
jmx_port=${2:-5000}
proxy_host=${3:-$jmx_host}
proxy_port=${4:-8123}
echo "connecting jconsole to $jmx_host:$jmx_port via SOCKS proxy $proxy_host using local port $proxy_port"
ssh -f -ND $proxy_port $proxy_host
jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=${proxy_port} service:jmx:rmi:///jndi/rmi://${jmx_host}:${jmx_port}/jmxrmi
kill $(ps ax | grep "[s]sh -f -ND $proxy_port" | awk '{print $1}')
}

For example You have box on prod1 running java process with jmx listening on port 5000 and have ssh access to proxy1 which not separated by firewall from prod1 try

% jc prod1 5000 proxy1

Where prod1 is production host name resolvable from proxy1.

It can be the same host i.e. you have ssh access to prod than do:

% jc prod1 5000

or

% jc localhost 5000 proxy1

Here is another one for Visual VM if someone find it more useful.

1
2
3
4
5
6
7
8
9
10

function jvvm {
jmx_host=$1
jmx_port=${2:-5000}
proxy_host=${3:-$jmx_host}
proxy_port=${4:-8123}
echo "connecting jvisualvm to $jmx_host:$jmx_port via SOCKS proxy $proxy_host using local port $proxy_port"
ssh -f -ND $proxy_port $proxy_host
jvisualvm -J-Dnetbeans.system_socks_proxy=localhost:${proxy_port} -J-Djava.net.useSystemProxies=true --openjmx  service:jmx:rmi:///jndi/rmi://${jmx_host}:${jmx_port}/jmxrmi
kill $(ps ax | grep "[s]sh -f -ND $proxy_port" | awk '{print $1}')
}

If there are two (or more) hosts on You way

to prod still no problem.
Given topology

yourPc -> proxy1 -> proxy2 -> prod

Assuming that you can successfully execute :

ssh usernameOnProxy1@proxy1

on Your local box
and 

ssh usernameOnProxy2@proxy2

on proxy1

put this in Your ~/.ssh/config

Host proxy2
Hostname proxy2
User usernameOnProxy2
ProxyCommand ssh -q usernameOnProxy1@proxy1 nc -w 180 %h %p

this will allow You to execute ssh proxy2 and seamlessly connect to proxy2 via proxy1 invoking only this one command.

Given prod1 is destination box resolvable from proxy2

you should be able to simply execute

% jc prod1 5000 proxy2

And have Your local jconsole connected to prod via proxy1 and proxy2

proxy on Your way

still possible. First install corkscrew and put following in Your ~/.ssh/config

Host proxy2
HostName proxy2
User usernameOnProxy2
ProxyCommand corkscrew httpProxy1 8080 %h %p

than again

% jc prod1 5000 proxy2

(assumptions are the same as in previous example, only difference is that proxy1 is now your http proxy on port 8080 (httpProxy1) )

Enjoy!!

, , , , , , ,

1 komentarz

mysql backup restore on the fly using ssh

Napisany przez Lukasz w Lukasz dnia 2011/05/29

Some time ago I wanted to restore big mysql database from remote server without storing backup file locally.
I also wanted to see progress of this process since it can be long by displaying progress using great pv utility.

This is bash script what I came up with:


SOURCE_DB_NAME=
DESTINATION_DB_HOST=
DESTINATION_DB_NAME=
DESTINATION_DB_USER=
DESTINATION_DB_PASS=
DUMP_HOST=

sourceFile=/tmp/someMysqlDumpfileOnRemoteMachine.sql.gz

fileSize=`ssh $LOGS_HOST "stat -c %s \"$sourceFile\""`

echo "restoring $sourceFile of size $fileSize"

ssh $DUMP_HOST "cat $sourceFile" | pv -s $fileSize | gzip -d | mysql -u$DESTINATION_DB_USER -p$DESTINATION_DB_PASS -h$DESTINATION_DB_HOST $DESTINATION_DB_NAME

, , , , ,

1 komentarz