Difference between revisions of "Reboot Machines by Monitoring email Instructions"
Michael.mast (talk | contribs) |
Michael.mast (talk | contribs) |
||
(15 intermediate revisions by the same user not shown) | |||
Line 11: | Line 11: | ||
==Process== | ==Process== | ||
Here I am using Centos 7 with a basic install. Epel-release is installed as well. | Here I am using Centos 7 with a basic install. Epel-release is installed as well. | ||
− | ===Fetchmail=== | + | ===Email Config=== |
+ | ====Sendmail==== | ||
+ | This is so we can send confirmation emails back to the user. | ||
+ | <pre> | ||
+ | yum -y install sendmail sendmail-cf | ||
+ | echo "xxx.xxx.xxx.xxx youremail.relay" >> /etc/hosts | ||
+ | sed 's/dnl\ define\(\`SMART_HOST\'\,\ \`smtp\.your\.provider\'\)/define\(\`SMART_HOST\'\,\ \`youremail\.relay\'\)/' /etc/mail/sendmail.mc | ||
+ | systemctl restart sendmail | ||
+ | </pre> | ||
+ | ====Fetchmail==== | ||
*Install fetchmail | *Install fetchmail | ||
yum -y install fetchmail | yum -y install fetchmail | ||
Line 58: | Line 67: | ||
fetchmail: outlook.office365.com fingerprints match. | fetchmail: outlook.office365.com fingerprints match. | ||
fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits | fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits | ||
− | fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready. | + | fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready. |
fetchmail: IMAP> A0001 CAPABILITY | fetchmail: IMAP> A0001 CAPABILITY | ||
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+ | fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+ | ||
Line 79: | Line 88: | ||
fetchmail: normal termination, status 1 | fetchmail: normal termination, status 1 | ||
</pre> | </pre> | ||
+ | |||
===Parse Received Email=== | ===Parse Received Email=== | ||
In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools. | In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools. | ||
<br> | <br> | ||
<br> | <br> | ||
− | In this sample we want to pull the email address and subject from the email. | + | |
+ | ====Resolving user names from email==== | ||
+ | |||
+ | =====LDAP Search===== | ||
+ | *Install openldap client <ref>https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/conf_examplesofusingldapsearch_t.html</ref> | ||
+ | <pre>yum -y install openldap-clients</pre> | ||
+ | |||
+ | =====Parsing directly from email===== | ||
+ | In this sample we want to pull the email address and subject from the email.<ref>https://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos</ref><ref>https://stackoverflow.com/questions/10408816/how-do-i-use-the-nohup-command-without-getting-nohup-out</ref> | ||
<pre> | <pre> | ||
#!/bin/bash | #!/bin/bash | ||
− | + | filetag1="/tmp/email_$(date +%s)" | |
− | + | filetag2="/tmp/parse_$(date +%s)" | |
− | + | filetag3="/tmp/address_$(date +%s)" | |
− | + | emailsource="<address>@<tld>" | |
− | + | subject="Subject: Reboot Request $(date +%m%d%y%M)" | |
+ | mail1="/home/<rebootuser>/templates/mail1.txt" | ||
+ | fetchmail --sslproto TLS1.2+ --sslcertck | ||
+ | if [ "$?" -ne "1" ] && [ "$?" -ne "0" ]; then | ||
+ | /usr/bin/echo "$(date) : Failed to download emails" >> /home/<rebootuser>/fetch_log | ||
+ | else | ||
+ | /usr/bin/cp /var/mail/<rebootuser> $filetag1 | ||
+ | grep -E Subject:\|From: $filetag1 | sed 'N;s/\n/ /' > $filetag2 | ||
+ | :> /var/mail/email | ||
+ | /usr/bin/rm -f $filetag1 | ||
+ | while read line; do | ||
+ | from1=$(grep -Eo "From:.*[a-zA-Z_'-']+@[a-zA-Z_'-']+.com>" <<<$line | sed -e 's/From:\ [A-Za-z]\+\ [A-Za-z]\+\ <//g; s/>//g') | ||
+ | if grep -qiE domain1.tld\|domain2.tld <<< $from1 ; then | ||
+ | echo "$subject" | cat - "$mail1" | sendmail -f $emailsource $from1 | ||
+ | /home/<rebootuser>/restart.sh $from1 & | ||
+ | fi | ||
+ | done < $filetag2 | ||
+ | /usr/bin/rm $filetag2 | ||
+ | fi | ||
+ | sleep 10 | ||
+ | nohup /home/<rebootuser>/reademail.sh >/dev/null & | ||
+ | exit | ||
+ | |||
</pre> | </pre> | ||
− | ===Restart Windows Machines Using Native Tools=== | + | |
+ | ===Restarting Machines=== | ||
+ | ====Restart Windows Machines Using Native Tools==== | ||
*Install samba-common-tools<ref>https://lifehacker.com/5275652/shut-down-your-windows-pc-remotely-from-linux</ref> | *Install samba-common-tools<ref>https://lifehacker.com/5275652/shut-down-your-windows-pc-remotely-from-linux</ref> | ||
<pre>yum -y install samba-common-tools</pre> | <pre>yum -y install samba-common-tools</pre> | ||
*A command like the following will work<ref>https://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/</ref> | *A command like the following will work<ref>https://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/</ref> | ||
<pre>net rpc shutdown -r -W <domain> -S <hostname> --user <username> </pre> | <pre>net rpc shutdown -r -W <domain> -S <hostname> --user <username> </pre> | ||
+ | =====Full Script===== | ||
+ | *In this example I am looking through a directory in which a file is written every time a user logs in. The file name has the computer name and the samaccountname.<br> | ||
+ | *It also assumes the samaccountname is first.last.<br> | ||
+ | *Also that your primary SMTP proxy address is the address you are sending from.<br> | ||
+ | *This script is called by the email script, with the email address as input. | ||
+ | *The email feedback is important to keep users happy. <ref>https://stackoverflow.com/questions/3005457/combining-echo-and-cat-on-unix</ref> | ||
+ | <pre> | ||
+ | #!/bin/bash | ||
+ | netreboot_log="/home/<rebootuser>/netreboot_log" | ||
+ | filelocation="<mount point>" | ||
+ | ldapsearch_log="/home/<rebootuser>/ldapsearch_log" | ||
+ | emailsource="<account>@<domain.tld>" | ||
+ | subject="Subject: Reboot Request $(date +%m%d%y%M)" | ||
+ | mail2="/home/<rebootuser>/templates/mail2.txt" | ||
+ | mail3="/home/<rebootuser>/templates/mail3.txt" | ||
+ | mail4="/home/<rebootuser>/templates/mail4.txt" | ||
+ | mail5="/home/<rebootuser>/templates/mail5.txt" | ||
+ | mail6="/home/<rebootuser>/templates/mail6.txt" | ||
+ | mail7="/home/<rebootuser>/templates/mail7.txt" | ||
+ | mail8="/home/<rebootuser>/templates/mail8.txt" | ||
+ | |||
+ | user=$(ldapsearch -x -h <domain.tld> -D "cn=rebootuser,ou=Users,DC=<domain>,DC=<tld>" -w <password> -b OU=Users,DC=<domain>,DC=<tld> "(&(objectClass=user)(proxyaddresses=SMTP:$1))" samaccountname | grep -iEo sAMAccountName:\ [a-z]+\.[a-z]+$ | sed 's/sAMAccountName:\ //') | ||
+ | |||
+ | |||
+ | if ! grep -qiE [a-z]+\.[a-z]+$ <<< $user; then | ||
+ | echo "$(date) : Failed to retrieve user associated with $1" >> $ldapsearch_log | ||
+ | echo "$subject" | cat - "$mail2" | sendmail -f $emailsource $1 | ||
+ | else | ||
+ | image=$(echo $(ls -t1 $filelocation | grep -i <machine name prefix> | grep -i $user | head -1) | grep -Eio csp-vdi-[0-9]\{1,2\}) | ||
+ | net rpc shutdown -r -t 0 -W domain.tld -S $image --user <reboot user>%<password> | ||
+ | if [ "$?" -ne "0" ]; then | ||
+ | echo "$(date) : Reboot of $image failed" >> $netreboot_log | ||
+ | echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1 | ||
+ | else | ||
+ | ping="0%" | ||
+ | count="0" | ||
+ | until [ "$ping" = "100%" ]; | ||
+ | do | ||
+ | if [ "$count" -eq "120" ]; then | ||
+ | echo "$(date) : $image has not gone down after 2 minutes" >> $netreboot_log | ||
+ | ping="100%" | ||
+ | testout="failed" | ||
+ | echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1 | ||
+ | else | ||
+ | count=$((count + 1)) | ||
+ | ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%) | ||
+ | sleep 1 | ||
+ | fi | ||
+ | done | ||
+ | if [ "$testout" != "failed" ]; then | ||
+ | ping="100%" | ||
+ | count="0" | ||
+ | testout="other" | ||
+ | until [ "$ping" = "0%" ]; | ||
+ | do | ||
+ | if [ "$count" -eq "300" ]; then | ||
+ | echo "$(date) : $image has not come back up after 5 minutes" >> $netreboot_log | ||
+ | ping="0%" | ||
+ | testout="failed" | ||
+ | echo "$subject" | cat - "$mail4" | sendmail -f $emailsource $1 | ||
+ | else | ||
+ | count=$((count + 1)) | ||
+ | ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%) | ||
+ | fi | ||
+ | done | ||
+ | echo "$subject" | cat - "$mail5" | sendmail -f $emailsource $1 | ||
+ | sleep 120 | ||
+ | fi | ||
+ | if [ "$testout" != "failed" ]; then | ||
+ | echo "$subject" | cat - "$mail6" | sendmail -f $emailsource $1 | ||
+ | fi | ||
+ | fi | ||
+ | fi | ||
+ | exit | ||
+ | </pre> | ||
+ | |||
+ | ====Restart VMs in vsphere==== | ||
+ | <ref>https://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx</ref> | ||
+ | ===Systemd Unit File=== | ||
+ | The key here was the "ReaminAfterExit" which was nice to learn about. Red Hat dropped the ball and did not have this on their how-to.<ref>http://tuxgraphics.org/npa/systemd-scripts/</ref><br> | ||
+ | To have the script start at boot: | ||
+ | <pre> | ||
+ | [Unit] | ||
+ | Description=Reboot VC Application | ||
+ | After=network.target | ||
+ | |||
+ | [Service] | ||
+ | Type=oneshot | ||
+ | User=<rebootuser> | ||
+ | ExecStart=/home/<rebootuser>/reademail.sh | ||
+ | RemainAfterExit=true | ||
+ | |||
+ | [Install] | ||
+ | WantedBy=default.target | ||
+ | </pre> |
Latest revision as of 10:11, 26 April 2018
Contents
Purpose
To monitor for emails, then execute commands. In this case we want to
- Look for emails from users
- Verify the user sent the email
- Reboot their remote desktop
- Check for errors
- Report to the user the desktop is ready
For this run we want a Linux server to be the middle man to watch an inbox hosted on Office365, then tell Windows machines to reboot. If the Windows machine does not respond we want to use powercli to forcibly reboot using vsphere (and eventually KVM, which is ironically much easier to do. But when execs want to spend money on the main stream....)
Process
Here I am using Centos 7 with a basic install. Epel-release is installed as well.
Email Config
Sendmail
This is so we can send confirmation emails back to the user.
yum -y install sendmail sendmail-cf echo "xxx.xxx.xxx.xxx youremail.relay" >> /etc/hosts sed 's/dnl\ define\(\`SMART_HOST\'\,\ \`smtp\.your\.provider\'\)/define\(\`SMART_HOST\'\,\ \`youremail\.relay\'\)/' /etc/mail/sendmail.mc systemctl restart sendmail
Fetchmail
- Install fetchmail
yum -y install fetchmail
- Create a non-root user dedicated to the task
useradd mailuser passwd mailuser su mailuser cd ~/
- Create .fetchmailrc and add the following contents
poll outlook.office365.com protocol imap port 993 username "user@login.tld" password "password" ssl sslfingerprint "97:08:33:5A:74:09:CC:EA:28:2D:9C:A4:49:3B:A2:C7"
openssl s_client -connect outlook.office365.com:993 -showcerts | openssl x509 -fingerprint -noout -md5
- At this point you can run the following. If you created a new inbox your output should look similar[3]
[emailuser@testserver ~]$ fetchmail -v --sslproto TLS1.2+ --sslcertck fetchmail: 6.3.24 querying outlook.office365.com (protocol IMAP) at Tue 17 Apr 2018 02:29:42 PM EDT: poll started Trying to connect to 40.97.100.50/993...connected. fetchmail: Server certificate: fetchmail: Issuer Organization: DigiCert Inc fetchmail: Issuer CommonName: DigiCert Cloud Services CA-1 fetchmail: Subject CommonName: outlook.com fetchmail: Subject Alternative Name: *.clo.footprintdns.com fetchmail: Subject Alternative Name: *.nrb.footprintdns.com fetchmail: Subject Alternative Name: *.hotmail.com fetchmail: Subject Alternative Name: *.internal.outlook.com fetchmail: Subject Alternative Name: *.live.com fetchmail: Subject Alternative Name: *.office.com fetchmail: Subject Alternative Name: *.office365.com fetchmail: Subject Alternative Name: *.outlook.com fetchmail: Subject Alternative Name: *.outlook.office365.com fetchmail: Subject Alternative Name: attachment.outlook.live.net fetchmail: Subject Alternative Name: attachment.outlook.office.net fetchmail: Subject Alternative Name: attachment.outlook.officeppe.net fetchmail: Subject Alternative Name: ccs.login.microsoftonline.com fetchmail: Subject Alternative Name: ccs-sdf.login.microsoftonline.com fetchmail: Subject Alternative Name: hotmail.com fetchmail: Subject Alternative Name: mail.services.live.com fetchmail: Subject Alternative Name: office365.com fetchmail: Subject Alternative Name: outlook.com fetchmail: Subject Alternative Name: outlook.office.com fetchmail: Subject Alternative Name: substrate.office.com fetchmail: Subject Alternative Name: substrate-sdf.office.com fetchmail: outlook.office365.com key fingerprint: 97:08:33:5A:74:09:CC:EA:28:2D:9C:A4:49:3B:A2:C7 fetchmail: outlook.office365.com fingerprints match. fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready. fetchmail: IMAP> A0001 CAPABILITY fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+ fetchmail: IMAP< A0001 OK CAPABILITY completed. fetchmail: IMAP> A0002 LOGIN "******@*******.com" * fetchmail: IMAP< A0002 OK LOGIN completed. fetchmail: IMAP> A0003 SELECT "INBOX" fetchmail: IMAP< * 0 EXISTS fetchmail: IMAP< * 0 RECENT fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent) fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags fetchmail: IMAP< * OK [UIDVALIDITY 14] UIDVALIDITY value fetchmail: IMAP< * OK [UIDNEXT 4] The next unique identifier value fetchmail: IMAP< A0003 OK [READ-WRITE] SELECT completed. fetchmail: No mail for ******@*****.com at outlook.office365.com fetchmail: IMAP> A0004 LOGOUT fetchmail: IMAP< * BYE Microsoft Exchange Server IMAP4 server signing off. fetchmail: IMAP< A0004 OK LOGOUT completed. fetchmail: 6.3.24 querying outlook.office365.com (protocol IMAP) at Tue 17 Apr 2018 02:29:51 PM EDT: poll completed fetchmail: normal termination, status 1
Parse Received Email
In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools.
Resolving user names from email
LDAP Search
- Install openldap client [4]
yum -y install openldap-clients
Parsing directly from email
In this sample we want to pull the email address and subject from the email.[5][6]
#!/bin/bash filetag1="/tmp/email_$(date +%s)" filetag2="/tmp/parse_$(date +%s)" filetag3="/tmp/address_$(date +%s)" emailsource="<address>@<tld>" subject="Subject: Reboot Request $(date +%m%d%y%M)" mail1="/home/<rebootuser>/templates/mail1.txt" fetchmail --sslproto TLS1.2+ --sslcertck if [ "$?" -ne "1" ] && [ "$?" -ne "0" ]; then /usr/bin/echo "$(date) : Failed to download emails" >> /home/<rebootuser>/fetch_log else /usr/bin/cp /var/mail/<rebootuser> $filetag1 grep -E Subject:\|From: $filetag1 | sed 'N;s/\n/ /' > $filetag2 :> /var/mail/email /usr/bin/rm -f $filetag1 while read line; do from1=$(grep -Eo "From:.*[a-zA-Z_'-']+@[a-zA-Z_'-']+.com>" <<<$line | sed -e 's/From:\ [A-Za-z]\+\ [A-Za-z]\+\ <//g; s/>//g') if grep -qiE domain1.tld\|domain2.tld <<< $from1 ; then echo "$subject" | cat - "$mail1" | sendmail -f $emailsource $from1 /home/<rebootuser>/restart.sh $from1 & fi done < $filetag2 /usr/bin/rm $filetag2 fi sleep 10 nohup /home/<rebootuser>/reademail.sh >/dev/null & exit
Restarting Machines
Restart Windows Machines Using Native Tools
- Install samba-common-tools[7]
yum -y install samba-common-tools
- A command like the following will work[8]
net rpc shutdown -r -W <domain> -S <hostname> --user <username>
Full Script
- In this example I am looking through a directory in which a file is written every time a user logs in. The file name has the computer name and the samaccountname.
- It also assumes the samaccountname is first.last.
- Also that your primary SMTP proxy address is the address you are sending from.
- This script is called by the email script, with the email address as input.
- The email feedback is important to keep users happy. [9]
#!/bin/bash netreboot_log="/home/<rebootuser>/netreboot_log" filelocation="<mount point>" ldapsearch_log="/home/<rebootuser>/ldapsearch_log" emailsource="<account>@<domain.tld>" subject="Subject: Reboot Request $(date +%m%d%y%M)" mail2="/home/<rebootuser>/templates/mail2.txt" mail3="/home/<rebootuser>/templates/mail3.txt" mail4="/home/<rebootuser>/templates/mail4.txt" mail5="/home/<rebootuser>/templates/mail5.txt" mail6="/home/<rebootuser>/templates/mail6.txt" mail7="/home/<rebootuser>/templates/mail7.txt" mail8="/home/<rebootuser>/templates/mail8.txt" user=$(ldapsearch -x -h <domain.tld> -D "cn=rebootuser,ou=Users,DC=<domain>,DC=<tld>" -w <password> -b OU=Users,DC=<domain>,DC=<tld> "(&(objectClass=user)(proxyaddresses=SMTP:$1))" samaccountname | grep -iEo sAMAccountName:\ [a-z]+\.[a-z]+$ | sed 's/sAMAccountName:\ //') if ! grep -qiE [a-z]+\.[a-z]+$ <<< $user; then echo "$(date) : Failed to retrieve user associated with $1" >> $ldapsearch_log echo "$subject" | cat - "$mail2" | sendmail -f $emailsource $1 else image=$(echo $(ls -t1 $filelocation | grep -i <machine name prefix> | grep -i $user | head -1) | grep -Eio csp-vdi-[0-9]\{1,2\}) net rpc shutdown -r -t 0 -W domain.tld -S $image --user <reboot user>%<password> if [ "$?" -ne "0" ]; then echo "$(date) : Reboot of $image failed" >> $netreboot_log echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1 else ping="0%" count="0" until [ "$ping" = "100%" ]; do if [ "$count" -eq "120" ]; then echo "$(date) : $image has not gone down after 2 minutes" >> $netreboot_log ping="100%" testout="failed" echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1 else count=$((count + 1)) ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%) sleep 1 fi done if [ "$testout" != "failed" ]; then ping="100%" count="0" testout="other" until [ "$ping" = "0%" ]; do if [ "$count" -eq "300" ]; then echo "$(date) : $image has not come back up after 5 minutes" >> $netreboot_log ping="0%" testout="failed" echo "$subject" | cat - "$mail4" | sendmail -f $emailsource $1 else count=$((count + 1)) ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%) fi done echo "$subject" | cat - "$mail5" | sendmail -f $emailsource $1 sleep 120 fi if [ "$testout" != "failed" ]; then echo "$subject" | cat - "$mail6" | sendmail -f $emailsource $1 fi fi fi exit
Restart VMs in vsphere
Systemd Unit File
The key here was the "ReaminAfterExit" which was nice to learn about. Red Hat dropped the ball and did not have this on their how-to.[11]
To have the script start at boot:
[Unit] Description=Reboot VC Application After=network.target [Service] Type=oneshot User=<rebootuser> ExecStart=/home/<rebootuser>/reademail.sh RemainAfterExit=true [Install] WantedBy=default.target
- ↑ https://michaelwiki.geekgalaxy.com/w/index.php/Check_IMAP_and_POP_with_OpenSSL
- ↑ http://www.aerus.net/2017/04/29/updatefix-fetchmail-unix-connection-to-outlook-365/
- ↑ http://www.fetchmail.info/fetchmail-man.html#2
- ↑ https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/conf_examplesofusingldapsearch_t.html
- ↑ https://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos
- ↑ https://stackoverflow.com/questions/10408816/how-do-i-use-the-nohup-command-without-getting-nohup-out
- ↑ https://lifehacker.com/5275652/shut-down-your-windows-pc-remotely-from-linux
- ↑ https://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/
- ↑ https://stackoverflow.com/questions/3005457/combining-echo-and-cat-on-unix
- ↑ https://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx
- ↑ http://tuxgraphics.org/npa/systemd-scripts/