Wpisy oznaczone tagiem bash
bash parallel processing and error handling
Another note to myself.
Here is a way how I process some program execution which is sending xml to standard output.
This part of script executes, checks and compress program xml output in parallel.
Tested for output sizes > 10GB.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 | $JAVA_HOME/bin/java -jar $jarFile \ | tee >(bzip2 > $archFile ) \ | (xmllint --stream --noout - ) DUMPSTATUS=(${PIPESTATUS[*]}) commandResult=${DUMPSTATUS[0]}; bzipResult=${DUMPSTATUS[1]}; checkResult=${DUMPSTATUS[2]}; if [ "$commandResult" -ne 0 ]; then die 'ERROR::could not execute command' else echo 'INFO::command successfully executed' fi if [ "$checkResult" -ne 0 ]; then die 'ERROR::xml is not correct' else echo 'INFO::xml is correct' fi if [ "$bzipResult" -ne 0 ]; then die 'ERROR::could not create bzip file from output' else echo 'INFO::compression is successful' fi |
uploading whole directory to WebDav endpoint using curl
Note to myself
upload whole directory structure to webdav endpoint using curl only:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 | #!/bin/bash #read -p "username: " username #stty -echo username= #read -p "Password: " password; echo #stty echo password= dest="https://host/path/" sourceDir="./sourceDir" user="${username}:${password}" echo "cleaning up ${dest}" curl --request DELETE --user ${user} "${dest}" -o /dev/null curl --request MKCOL --user ${user} "${dest}" -o /dev/null echo "uploading from ${sourceDir} to ${dest}" for file in $(find ${sourceDir} ) do target="${dest}/${file#"${sourceDir}"}" echo "$file -> $target" if [ -d "$file" ] then curl --request MKCOL --user "${user}" "${target}" -o /dev/null else curl --upload-file "$file" --user "${user}" "${target}" -o /dev/null fi done |
lockpicking production
i.e. accessing jmx through the firewall
All the following is extension of this great article which I’ve found while searching for a way to connect jconsole on my local pc to JVM running on some server to which I have only ssh access. This works fast, stable and without any problems.
Scripts
This is nice bash function which i’ve put into my ~/.bashrc
1 2 3 4 5 6 7 8 9 10 | function jc { jmx_host=$1 jmx_port=${2:-5000} proxy_host=${3:-$jmx_host} proxy_port=${4:-8123} echo "connecting jconsole to $jmx_host:$jmx_port via SOCKS proxy $proxy_host using local port $proxy_port" ssh -f -ND $proxy_port $proxy_host jconsole -J-DsocksProxyHost=localhost -J-DsocksProxyPort=${proxy_port} service:jmx:rmi:///jndi/rmi://${jmx_host}:${jmx_port}/jmxrmi kill $(ps ax | grep "[s]sh -f -ND $proxy_port" | awk '{print $1}') } |
For example You have box on prod1
running java process with jmx listening on port 5000
and have ssh access to proxy1
which not separated by firewall from prod1
try
% jc prod1 5000 proxy1
Where prod1
is production host name resolvable from proxy1
.
It can be the same host i.e. you have ssh access to prod
than do:
% jc prod1 5000
or
% jc localhost 5000 proxy1
Here is another one for Visual VM if someone find it more useful.
1 2 3 4 5 6 7 8 9 10 | function jvvm { jmx_host=$1 jmx_port=${2:-5000} proxy_host=${3:-$jmx_host} proxy_port=${4:-8123} echo "connecting jvisualvm to $jmx_host:$jmx_port via SOCKS proxy $proxy_host using local port $proxy_port" ssh -f -ND $proxy_port $proxy_host jvisualvm -J-Dnetbeans.system_socks_proxy=localhost:${proxy_port} -J-Djava.net.useSystemProxies=true --openjmx service:jmx:rmi:///jndi/rmi://${jmx_host}:${jmx_port}/jmxrmi kill $(ps ax | grep "[s]sh -f -ND $proxy_port" | awk '{print $1}') } |
If there are two (or more) hosts on You way
to prod
still no problem.
Given topology
yourPc -> proxy1 -> proxy2 -> prod
Assuming that you can successfully execute :
ssh usernameOnProxy1@proxy1
on Your local box
and
ssh usernameOnProxy2@proxy2
on proxy1
put this in Your ~/.ssh/config
Host proxy2 Hostname proxy2 User usernameOnProxy2 ProxyCommand ssh -q usernameOnProxy1@proxy1 nc -w 180 %h %p
this will allow You to execute ssh proxy2
and seamlessly connect to proxy2 via proxy1
invoking only this one command.
Given prod1
is destination box resolvable from proxy2
you should be able to simply execute
% jc prod1 5000 proxy2
And have Your local jconsole connected to prod
via proxy1
and proxy2
proxy on Your way
still possible. First install corkscrew and put following in Your ~/.ssh/config
Host proxy2 HostName proxy2 User usernameOnProxy2 ProxyCommand corkscrew httpProxy1 8080 %h %p
than again
% jc prod1 5000 proxy2
(assumptions are the same as in previous example, only difference is that proxy1
is now your http proxy on port 8080 (httpProxy1
) )
Enjoy!!
mysql backup restore on the fly using ssh
Some time ago I wanted to restore big mysql database from remote server without storing backup file locally.
I also wanted to see progress of this process since it can be long by displaying progress using great pv utility.
This is bash script what I came up with:
SOURCE_DB_NAME=
DESTINATION_DB_HOST=
DESTINATION_DB_NAME=
DESTINATION_DB_USER=
DESTINATION_DB_PASS=
DUMP_HOST=
sourceFile=/tmp/someMysqlDumpfileOnRemoteMachine.sql.gz
fileSize=`ssh $LOGS_HOST "stat -c %s \"$sourceFile\""`
echo "restoring $sourceFile of size $fileSize"
ssh $DUMP_HOST "cat $sourceFile" | pv -s $fileSize | gzip -d | mysql -u$DESTINATION_DB_USER -p$DESTINATION_DB_PASS -h$DESTINATION_DB_HOST $DESTINATION_DB_NAME