Difference between revisions of "Reboot Machines by Monitoring email Instructions"

From Michael's Information Zone
Jump to navigation Jump to search
 
(13 intermediate revisions by the same user not shown)
Line 11: Line 11:
 
==Process==
 
==Process==
 
Here I am using Centos 7 with a basic install. Epel-release is installed as well.
 
Here I am using Centos 7 with a basic install. Epel-release is installed as well.
−
===Fetchmail===
+
===Email Config===
 +
====Sendmail====
 +
This is so we can send confirmation emails back to the user.
 +
<pre>
 +
yum -y install sendmail sendmail-cf
 +
echo "xxx.xxx.xxx.xxx youremail.relay" >> /etc/hosts
 +
sed 's/dnl\ define\(\`SMART_HOST\'\,\ \`smtp\.your\.provider\'\)/define\(\`SMART_HOST\'\,\ \`youremail\.relay\'\)/' /etc/mail/sendmail.mc
 +
systemctl restart sendmail
 +
</pre>
 +
====Fetchmail====
 
*Install fetchmail
 
*Install fetchmail
 
  yum -y install fetchmail
 
  yum -y install fetchmail
Line 58: Line 67:
 
fetchmail: outlook.office365.com fingerprints match.
 
fetchmail: outlook.office365.com fingerprints match.
 
fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits
 
fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits
−
fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready. [QgBOADMAUABSADAAMwBDAEEAMAAxADAAOAAuAG4AYQBtAHAAcgBkADAAMwAuAHAAcgBvAGQALgBvAHUAdABsAG8AbwBrAC4AYwBvAG0A]
+
fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
 
fetchmail: IMAP> A0001 CAPABILITY
 
fetchmail: IMAP> A0001 CAPABILITY
 
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
 
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
Line 79: Line 88:
 
fetchmail: normal termination, status 1
 
fetchmail: normal termination, status 1
 
</pre>
 
</pre>
 +
 
===Parse Received Email===
 
===Parse Received Email===
 
In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools.
 
In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools.
Line 87: Line 97:
  
 
=====LDAP Search=====
 
=====LDAP Search=====
−
*Install openldap client
+
*Install openldap client <ref>https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/conf_examplesofusingldapsearch_t.html</ref>
 
<pre>yum -y install openldap-clients</pre>
 
<pre>yum -y install openldap-clients</pre>
 +
 
=====Parsing directly from email=====
 
=====Parsing directly from email=====
−
In this sample we want to pull the email address and subject from the email. At the moment this only works with a single email at a time for obvious reasons. Does not scale well.
+
In this sample we want to pull the email address and subject from the email.<ref>https://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos</ref><ref>https://stackoverflow.com/questions/10408816/how-do-i-use-the-nohup-command-without-getting-nohup-out</ref>
 
<pre>
 
<pre>
 
#!/bin/bash
 
#!/bin/bash
−
parse1=$(grep -a1 -B7 -iE ^Message-ID:$ /var/mail/email)
+
filetag1="/tmp/email_$(date +%s)"
−
from=$(grep From: <<<$parse1 | grep -Eo [a-zA-Z_'-']+@[a-zA-Z_'-']+.com )
+
filetag2="/tmp/parse_$(date +%s)"
−
user=$(grep Subject: <<<$parse1 | awk 'BEGIN {FS=","} {print $2}')
+
filetag3="/tmp/address_$(date +%s)"
−
echo $user
+
emailsource="<address>@<tld>"
−
echo $from
+
subject="Subject: Reboot Request $(date +%m%d%y%M)"
 +
mail1="/home/<rebootuser>/templates/mail1.txt"
 +
fetchmail --sslproto TLS1.2+ --sslcertck
 +
if [ "$?" -ne "1" ] && [ "$?" -ne "0" ]; then
 +
/usr/bin/echo "$(date) : Failed to download emails" >> /home/<rebootuser>/fetch_log
 +
else
 +
/usr/bin/cp /var/mail/<rebootuser> $filetag1
 +
grep -E Subject:\|From: $filetag1 | sed 'N;s/\n/ /' > $filetag2
 +
:> /var/mail/email
 +
/usr/bin/rm -f $filetag1
 +
while read line; do
 +
from1=$(grep -Eo "From:.*[a-zA-Z_'-']+@[a-zA-Z_'-']+.com>" <<<$line | sed -e 's/From:\ [A-Za-z]\+\ [A-Za-z]\+\ <//g; s/>//g')
 +
if grep -qiE domain1.tld\|domain2.tld <<< $from1 ; then
 +
echo "$subject" | cat - "$mail1" | sendmail -f $emailsource $from1
 +
/home/<rebootuser>/restart.sh $from1 &
 +
fi
 +
done < $filetag2
 +
/usr/bin/rm $filetag2
 +
fi
 +
sleep 10
 +
nohup /home/<rebootuser>/reademail.sh >/dev/null &
 +
exit
 +
 
 
</pre>
 
</pre>
  
Line 106: Line 139:
 
*A command like the following will work<ref>https://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/</ref>
 
*A command like the following will work<ref>https://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/</ref>
 
<pre>net rpc shutdown -r -W <domain> -S <hostname> --user <username> </pre>
 
<pre>net rpc shutdown -r -W <domain> -S <hostname> --user <username> </pre>
 +
=====Full Script=====
 +
*In this example I am looking through a directory in which a file is written every time a user logs in. The file name has the computer name and the samaccountname.<br>
 +
*It also assumes the samaccountname is first.last.<br>
 +
*Also that your primary SMTP proxy address is the address you are sending from.<br>
 +
*This script is called by the email script, with the email address as input.
 +
*The email feedback is important to keep users happy. <ref>https://stackoverflow.com/questions/3005457/combining-echo-and-cat-on-unix</ref>
 +
<pre>
 +
#!/bin/bash
 +
netreboot_log="/home/<rebootuser>/netreboot_log"
 +
filelocation="<mount point>"
 +
ldapsearch_log="/home/<rebootuser>/ldapsearch_log"
 +
emailsource="<account>@<domain.tld>"
 +
subject="Subject: Reboot Request $(date +%m%d%y%M)"
 +
mail2="/home/<rebootuser>/templates/mail2.txt"
 +
mail3="/home/<rebootuser>/templates/mail3.txt"
 +
mail4="/home/<rebootuser>/templates/mail4.txt"
 +
mail5="/home/<rebootuser>/templates/mail5.txt"
 +
mail6="/home/<rebootuser>/templates/mail6.txt"
 +
mail7="/home/<rebootuser>/templates/mail7.txt"
 +
mail8="/home/<rebootuser>/templates/mail8.txt"
 +
 +
user=$(ldapsearch -x -h <domain.tld> -D "cn=rebootuser,ou=Users,DC=<domain>,DC=<tld>" -w <password> -b OU=Users,DC=<domain>,DC=<tld> "(&(objectClass=user)(proxyaddresses=SMTP:$1))" samaccountname | grep -iEo sAMAccountName:\ [a-z]+\.[a-z]+$ | sed 's/sAMAccountName:\ //')
 +
 +
 +
if ! grep -qiE [a-z]+\.[a-z]+$  <<< $user; then
 +
echo "$(date) : Failed to retrieve user associated with $1" >> $ldapsearch_log
 +
echo "$subject" | cat - "$mail2" | sendmail -f $emailsource $1
 +
else
 +
image=$(echo $(ls -t1 $filelocation | grep -i <machine name prefix> | grep -i $user | head -1) | grep -Eio csp-vdi-[0-9]\{1,2\})
 +
net rpc shutdown -r -t 0 -W domain.tld -S $image --user <reboot user>%<password>
 +
if [ "$?" -ne "0" ]; then
 +
echo "$(date) : Reboot of $image failed" >> $netreboot_log
 +
echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1
 +
else
 +
ping="0%"
 +
count="0"
 +
until [ "$ping" = "100%" ];
 +
do
 +
if [ "$count" -eq "120" ]; then
 +
echo "$(date) : $image has not gone down after 2 minutes" >> $netreboot_log
 +
ping="100%"
 +
testout="failed"
 +
echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1
 +
else
 +
count=$((count + 1))
 +
ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%)
 +
sleep 1
 +
fi
 +
done
 +
if [ "$testout" != "failed" ]; then
 +
ping="100%"
 +
count="0"
 +
testout="other"
 +
until [ "$ping" = "0%" ];
 +
do
 +
if [ "$count" -eq "300" ]; then
 +
echo "$(date) : $image has not come back up after 5 minutes" >> $netreboot_log
 +
ping="0%"
 +
testout="failed"
 +
echo "$subject" | cat - "$mail4" | sendmail -f $emailsource $1
 +
else
 +
count=$((count + 1))
 +
ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%)
 +
fi
 +
done
 +
echo "$subject" | cat - "$mail5" | sendmail -f $emailsource $1
 +
sleep 120
 +
fi
 +
if [ "$testout" != "failed" ]; then
 +
echo "$subject" | cat - "$mail6" | sendmail -f $emailsource $1
 +
fi
 +
fi
 +
fi
 +
exit
 +
</pre>
 +
 
====Restart VMs in vsphere====
 
====Restart VMs in vsphere====
 
<ref>https://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx</ref>
 
<ref>https://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx</ref>
 +
===Systemd Unit File===
 +
The key here was the "ReaminAfterExit" which was nice to learn about. Red Hat dropped the ball and did not have this on their how-to.<ref>http://tuxgraphics.org/npa/systemd-scripts/</ref><br>
 +
To have the script start at boot:
 +
<pre>
 +
[Unit]
 +
Description=Reboot VC Application
 +
After=network.target
 +
 +
[Service]
 +
Type=oneshot
 +
User=<rebootuser>
 +
ExecStart=/home/<rebootuser>/reademail.sh
 +
RemainAfterExit=true
 +
 +
[Install]
 +
WantedBy=default.target
 +
</pre>

Latest revision as of 10:11, 26 April 2018

Purpose

To monitor for emails, then execute commands. In this case we want to

  • Look for emails from users
  • Verify the user sent the email
  • Reboot their remote desktop
  • Check for errors
  • Report to the user the desktop is ready



For this run we want a Linux server to be the middle man to watch an inbox hosted on Office365, then tell Windows machines to reboot. If the Windows machine does not respond we want to use powercli to forcibly reboot using vsphere (and eventually KVM, which is ironically much easier to do. But when execs want to spend money on the main stream....)

Process

Here I am using Centos 7 with a basic install. Epel-release is installed as well.

Email Config

Sendmail

This is so we can send confirmation emails back to the user. 

yum -y install sendmail sendmail-cf
echo "xxx.xxx.xxx.xxx youremail.relay" >> /etc/hosts
sed 's/dnl\ define\(\`SMART_HOST\'\,\ \`smtp\.your\.provider\'\)/define\(\`SMART_HOST\'\,\ \`youremail\.relay\'\)/' /etc/mail/sendmail.mc
systemctl restart sendmail

Fetchmail

  • Install fetchmail
yum -y install fetchmail
  • Create a non-root user dedicated to the task
useradd mailuser
passwd mailuser
su mailuser
cd ~/
  • Create .fetchmailrc and add the following contents
poll outlook.office365.com protocol imap port 993 username "user@login.tld" password "password" ssl sslfingerprint "97:08:33:5A:74:09:CC:EA:28:2D:9C:A4:49:3B:A2:C7"
  • The fingerprint can be obtained using openssl[1][2]
openssl s_client -connect outlook.office365.com:993 -showcerts | openssl x509 -fingerprint -noout -md5
  • At this point you can run the following. If you created a new inbox your output should look similar[3]
[emailuser@testserver ~]$ fetchmail -v --sslproto TLS1.2+ --sslcertck
fetchmail: 6.3.24 querying outlook.office365.com (protocol IMAP) at Tue 17 Apr 2018 02:29:42 PM EDT: poll started
Trying to connect to 40.97.100.50/993...connected.
fetchmail: Server certificate:
fetchmail: Issuer Organization: DigiCert Inc
fetchmail: Issuer CommonName: DigiCert Cloud Services CA-1
fetchmail: Subject CommonName: outlook.com
fetchmail: Subject Alternative Name: *.clo.footprintdns.com
fetchmail: Subject Alternative Name: *.nrb.footprintdns.com
fetchmail: Subject Alternative Name: *.hotmail.com
fetchmail: Subject Alternative Name: *.internal.outlook.com
fetchmail: Subject Alternative Name: *.live.com
fetchmail: Subject Alternative Name: *.office.com
fetchmail: Subject Alternative Name: *.office365.com
fetchmail: Subject Alternative Name: *.outlook.com
fetchmail: Subject Alternative Name: *.outlook.office365.com
fetchmail: Subject Alternative Name: attachment.outlook.live.net
fetchmail: Subject Alternative Name: attachment.outlook.office.net
fetchmail: Subject Alternative Name: attachment.outlook.officeppe.net
fetchmail: Subject Alternative Name: ccs.login.microsoftonline.com
fetchmail: Subject Alternative Name: ccs-sdf.login.microsoftonline.com
fetchmail: Subject Alternative Name: hotmail.com
fetchmail: Subject Alternative Name: mail.services.live.com
fetchmail: Subject Alternative Name: office365.com
fetchmail: Subject Alternative Name: outlook.com
fetchmail: Subject Alternative Name: outlook.office.com
fetchmail: Subject Alternative Name: substrate.office.com
fetchmail: Subject Alternative Name: substrate-sdf.office.com
fetchmail: outlook.office365.com key fingerprint: 97:08:33:5A:74:09:CC:EA:28:2D:9C:A4:49:3B:A2:C7
fetchmail: outlook.office365.com fingerprints match.
fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits
fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
fetchmail: IMAP< A0001 OK CAPABILITY completed.
fetchmail: IMAP> A0002 LOGIN "******@*******.com" *
fetchmail: IMAP< A0002 OK LOGIN completed.
fetchmail: IMAP> A0003 SELECT "INBOX"
fetchmail: IMAP< * 0 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
fetchmail: IMAP< * OK [UIDVALIDITY 14] UIDVALIDITY value
fetchmail: IMAP< * OK [UIDNEXT 4] The next unique identifier value
fetchmail: IMAP< A0003 OK [READ-WRITE] SELECT completed.
fetchmail: No mail for ******@*****.com at outlook.office365.com
fetchmail: IMAP> A0004 LOGOUT
fetchmail: IMAP< * BYE Microsoft Exchange Server IMAP4 server signing off.
fetchmail: IMAP< A0004 OK LOGOUT completed.
fetchmail: 6.3.24 querying outlook.office365.com (protocol IMAP) at Tue 17 Apr 2018 02:29:51 PM EDT: poll completed
fetchmail: normal termination, status 1

Parse Received Email

In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools.

Resolving user names from email

LDAP Search
  • Install openldap client [4]
yum -y install openldap-clients
Parsing directly from email

In this sample we want to pull the email address and subject from the email.[5][6] 

#!/bin/bash
filetag1="/tmp/email_$(date +%s)"
filetag2="/tmp/parse_$(date +%s)"
filetag3="/tmp/address_$(date +%s)"
emailsource="<address>@<tld>"
subject="Subject: Reboot Request $(date +%m%d%y%M)"
mail1="/home/<rebootuser>/templates/mail1.txt"
fetchmail --sslproto TLS1.2+ --sslcertck
if [ "$?" -ne "1" ] && [ "$?" -ne "0" ]; then
	/usr/bin/echo "$(date) : Failed to download emails" >> /home/<rebootuser>/fetch_log
else
	/usr/bin/cp /var/mail/<rebootuser> $filetag1
	grep -E Subject:\|From: $filetag1 | sed 'N;s/\n/ /' > $filetag2
	:> /var/mail/email
	/usr/bin/rm -f $filetag1
	while read line; do
	from1=$(grep -Eo "From:.*[a-zA-Z_'-']+@[a-zA-Z_'-']+.com>" <<<$line | sed -e 's/From:\ [A-Za-z]\+\ [A-Za-z]\+\ <//g; s/>//g')
	if grep -qiE domain1.tld\|domain2.tld <<< $from1 ; then
		echo "$subject" | cat - "$mail1" | sendmail -f $emailsource $from1
		/home/<rebootuser>/restart.sh $from1 &
	fi
	done < $filetag2
	/usr/bin/rm $filetag2
fi
sleep 10
nohup /home/<rebootuser>/reademail.sh >/dev/null &
exit

Restarting Machines

Restart Windows Machines Using Native Tools

  • Install samba-common-tools[7]
yum -y install samba-common-tools
  • A command like the following will work[8]
net rpc shutdown -r -W <domain> -S <hostname> --user <username>
Full Script
  • In this example I am looking through a directory in which a file is written every time a user logs in. The file name has the computer name and the samaccountname.
  • It also assumes the samaccountname is first.last.
  • Also that your primary SMTP proxy address is the address you are sending from.
  • This script is called by the email script, with the email address as input.
  • The email feedback is important to keep users happy. [9]
#!/bin/bash
netreboot_log="/home/<rebootuser>/netreboot_log"
filelocation="<mount point>"
ldapsearch_log="/home/<rebootuser>/ldapsearch_log"
emailsource="<account>@<domain.tld>"
subject="Subject: Reboot Request $(date +%m%d%y%M)"
mail2="/home/<rebootuser>/templates/mail2.txt"
mail3="/home/<rebootuser>/templates/mail3.txt"
mail4="/home/<rebootuser>/templates/mail4.txt"
mail5="/home/<rebootuser>/templates/mail5.txt"
mail6="/home/<rebootuser>/templates/mail6.txt"
mail7="/home/<rebootuser>/templates/mail7.txt"
mail8="/home/<rebootuser>/templates/mail8.txt"

user=$(ldapsearch -x -h <domain.tld> -D "cn=rebootuser,ou=Users,DC=<domain>,DC=<tld>" -w <password> -b OU=Users,DC=<domain>,DC=<tld> "(&(objectClass=user)(proxyaddresses=SMTP:$1))" samaccountname | grep -iEo sAMAccountName:\ [a-z]+\.[a-z]+$ | sed 's/sAMAccountName:\ //')


if ! grep -qiE [a-z]+\.[a-z]+$  <<< $user; then
	echo "$(date) : Failed to retrieve user associated with $1" >> $ldapsearch_log
	echo "$subject" | cat - "$mail2" | sendmail -f $emailsource $1
else
	image=$(echo $(ls -t1 $filelocation | grep -i <machine name prefix> | grep -i $user | head -1) | grep -Eio csp-vdi-[0-9]\{1,2\})
	net rpc shutdown -r -t 0 -W domain.tld -S $image --user <reboot user>%<password>
	if [ "$?" -ne "0" ]; then
		echo "$(date) : Reboot of $image failed" >> $netreboot_log
		echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1
	else
		ping="0%"
		count="0"
		until [ "$ping" = "100%" ];
		do
		if [ "$count" -eq "120" ]; then
			echo "$(date) : $image has not gone down after 2 minutes" >> $netreboot_log
			ping="100%"
			testout="failed"
			echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1
		else
			count=$((count + 1))
			ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%)
			sleep 1
		fi
		done
		if [ "$testout" != "failed" ]; then
			ping="100%"
			count="0"
			testout="other"
			until [ "$ping" = "0%" ];
			do
			if [ "$count" -eq "300" ]; then
				echo "$(date) : $image has not come back up after 5 minutes" >> $netreboot_log
				ping="0%"
				testout="failed"
				echo "$subject" | cat - "$mail4" | sendmail -f $emailsource $1
			else
				count=$((count + 1))
				ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%)
			fi
			done
		echo "$subject" | cat - "$mail5" | sendmail -f $emailsource $1
		sleep 120
		fi
		if [ "$testout" != "failed" ]; then
			echo "$subject" | cat - "$mail6" | sendmail -f $emailsource $1
		fi
	fi
fi
exit

Restart VMs in vsphere

[10]

Systemd Unit File

The key here was the "ReaminAfterExit" which was nice to learn about. Red Hat dropped the ball and did not have this on their how-to.[11]
To have the script start at boot: 

[Unit]
Description=Reboot VC Application
After=network.target

[Service]
Type=oneshot
User=<rebootuser>
ExecStart=/home/<rebootuser>/reademail.sh
RemainAfterExit=true

[Install]
WantedBy=default.target
  1. ↑ https://michaelwiki.geekgalaxy.com/w/index.php/Check_IMAP_and_POP_with_OpenSSL
  2. ↑ http://www.aerus.net/2017/04/29/updatefix-fetchmail-unix-connection-to-outlook-365/
  3. ↑ http://www.fetchmail.info/fetchmail-man.html#2
  4. ↑ https://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/conf_examplesofusingldapsearch_t.html
  5. ↑ https://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos
  6. ↑ https://stackoverflow.com/questions/10408816/how-do-i-use-the-nohup-command-without-getting-nohup-out
  7. ↑ https://lifehacker.com/5275652/shut-down-your-windows-pc-remotely-from-linux
  8. ↑ https://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/
  9. ↑ https://stackoverflow.com/questions/3005457/combining-echo-and-cat-on-unix
  10. ↑ https://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx
  11. ↑ http://tuxgraphics.org/npa/systemd-scripts/
Retrieved from "https://wiki.1701technology.com/index.php?title=Reboot_Machines_by_Monitoring_email_Instructions&oldid=717"