Difference between revisions of "Reboot Machines by Monitoring email Instructions"

Latest revision as of 10:11, 26 April 2018

Purpose

To monitor for emails, then execute commands. In this case we want to

Look for emails from users
Verify the user sent the email
Reboot their remote desktop
Check for errors
Report to the user the desktop is ready

For this run we want a Linux server to be the middle man to watch an inbox hosted on Office365, then tell Windows machines to reboot. If the Windows machine does not respond we want to use powercli to forcibly reboot using vsphere (and eventually KVM, which is ironically much easier to do. But when execs want to spend money on the main stream....)

Process

Here I am using Centos 7 with a basic install. Epel-release is installed as well.

Email Config

Sendmail

This is so we can send confirmation emails back to the user.

yum -y install sendmail sendmail-cf
echo "xxx.xxx.xxx.xxx youremail.relay" >> /etc/hosts
sed 's/dnl\ define\(\`SMART_HOST\'\,\ \`smtp\.your\.provider\'\)/define\(\`SMART_HOST\'\,\ \`youremail\.relay\'\)/' /etc/mail/sendmail.mc
systemctl restart sendmail

Fetchmail

Install fetchmail

yum -y install fetchmail

Create a non-root user dedicated to the task

useradd mailuser
passwd mailuser
su mailuser
cd ~/

Create .fetchmailrc and add the following contents

poll outlook.office365.com protocol imap port 993 username "user@login.tld" password "password" ssl sslfingerprint "97:08:33:5A:74:09:CC:EA:28:2D:9C:A4:49:3B:A2:C7"

The fingerprint can be obtained using openssl^[1]^[2]

openssl s_client -connect outlook.office365.com:993 -showcerts | openssl x509 -fingerprint -noout -md5

At this point you can run the following. If you created a new inbox your output should look similar^[3]

[emailuser@testserver ~]$ fetchmail -v --sslproto TLS1.2+ --sslcertck
fetchmail: 6.3.24 querying outlook.office365.com (protocol IMAP) at Tue 17 Apr 2018 02:29:42 PM EDT: poll started
Trying to connect to 40.97.100.50/993...connected.
fetchmail: Server certificate:
fetchmail: Issuer Organization: DigiCert Inc
fetchmail: Issuer CommonName: DigiCert Cloud Services CA-1
fetchmail: Subject CommonName: outlook.com
fetchmail: Subject Alternative Name: *.clo.footprintdns.com
fetchmail: Subject Alternative Name: *.nrb.footprintdns.com
fetchmail: Subject Alternative Name: *.hotmail.com
fetchmail: Subject Alternative Name: *.internal.outlook.com
fetchmail: Subject Alternative Name: *.live.com
fetchmail: Subject Alternative Name: *.office.com
fetchmail: Subject Alternative Name: *.office365.com
fetchmail: Subject Alternative Name: *.outlook.com
fetchmail: Subject Alternative Name: *.outlook.office365.com
fetchmail: Subject Alternative Name: attachment.outlook.live.net
fetchmail: Subject Alternative Name: attachment.outlook.office.net
fetchmail: Subject Alternative Name: attachment.outlook.officeppe.net
fetchmail: Subject Alternative Name: ccs.login.microsoftonline.com
fetchmail: Subject Alternative Name: ccs-sdf.login.microsoftonline.com
fetchmail: Subject Alternative Name: hotmail.com
fetchmail: Subject Alternative Name: mail.services.live.com
fetchmail: Subject Alternative Name: office365.com
fetchmail: Subject Alternative Name: outlook.com
fetchmail: Subject Alternative Name: outlook.office.com
fetchmail: Subject Alternative Name: substrate.office.com
fetchmail: Subject Alternative Name: substrate-sdf.office.com
fetchmail: outlook.office365.com key fingerprint: 97:08:33:5A:74:09:CC:EA:28:2D:9C:A4:49:3B:A2:C7
fetchmail: outlook.office365.com fingerprints match.
fetchmail: SSL/TLS: using protocol TLSv1.2, cipher ECDHE-RSA-AES256-GCM-SHA384, 256/256 secret/processed bits
fetchmail: IMAP< * OK The Microsoft Exchange IMAP4 service is ready.
fetchmail: IMAP> A0001 CAPABILITY
fetchmail: IMAP< * CAPABILITY IMAP4 IMAP4rev1 AUTH=PLAIN AUTH=XOAUTH2 SASL-IR UIDPLUS MOVE ID UNSELECT CHILDREN IDLE NAMESPACE LITERAL+
fetchmail: IMAP< A0001 OK CAPABILITY completed.
fetchmail: IMAP> A0002 LOGIN "******@*******.com" *
fetchmail: IMAP< A0002 OK LOGIN completed.
fetchmail: IMAP> A0003 SELECT "INBOX"
fetchmail: IMAP< * 0 EXISTS
fetchmail: IMAP< * 0 RECENT
fetchmail: IMAP< * FLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)
fetchmail: IMAP< * OK [PERMANENTFLAGS (\Seen \Answered \Flagged \Deleted \Draft $MDNSent)] Permanent flags
fetchmail: IMAP< * OK [UIDVALIDITY 14] UIDVALIDITY value
fetchmail: IMAP< * OK [UIDNEXT 4] The next unique identifier value
fetchmail: IMAP< A0003 OK [READ-WRITE] SELECT completed.
fetchmail: No mail for ******@*****.com at outlook.office365.com
fetchmail: IMAP> A0004 LOGOUT
fetchmail: IMAP< * BYE Microsoft Exchange Server IMAP4 server signing off.
fetchmail: IMAP< A0004 OK LOGOUT completed.
fetchmail: 6.3.24 querying outlook.office365.com (protocol IMAP) at Tue 17 Apr 2018 02:29:51 PM EDT: poll completed
fetchmail: normal termination, status 1

Parse Received Email

In this poor example I will be parsing the emails through a custom script. It would probably be better to do this using established tools.

Resolving user names from email

LDAP Search

Install openldap client ^[4]

yum -y install openldap-clients

Parsing directly from email

In this sample we want to pull the email address and subject from the email.^[5]^[6]

#!/bin/bash
filetag1="/tmp/email_$(date +%s)"
filetag2="/tmp/parse_$(date +%s)"
filetag3="/tmp/address_$(date +%s)"
emailsource="<address>@<tld>"
subject="Subject: Reboot Request $(date +%m%d%y%M)"
mail1="/home/<rebootuser>/templates/mail1.txt"
fetchmail --sslproto TLS1.2+ --sslcertck
if [ "$?" -ne "1" ] && [ "$?" -ne "0" ]; then
	/usr/bin/echo "$(date) : Failed to download emails" >> /home/<rebootuser>/fetch_log
else
	/usr/bin/cp /var/mail/<rebootuser> $filetag1
	grep -E Subject:\|From: $filetag1 | sed 'N;s/\n/ /' > $filetag2
	:> /var/mail/email
	/usr/bin/rm -f $filetag1
	while read line; do
	from1=$(grep -Eo "From:.*[a-zA-Z_'-']+@[a-zA-Z_'-']+.com>" <<<$line | sed -e 's/From:\ [A-Za-z]\+\ [A-Za-z]\+\ <//g; s/>//g')
	if grep -qiE domain1.tld\|domain2.tld <<< $from1 ; then
		echo "$subject" | cat - "$mail1" | sendmail -f $emailsource $from1
		/home/<rebootuser>/restart.sh $from1 &
	fi
	done < $filetag2
	/usr/bin/rm $filetag2
fi
sleep 10
nohup /home/<rebootuser>/reademail.sh >/dev/null &
exit

Restarting Machines

Restart Windows Machines Using Native Tools

Install samba-common-tools^[7]

yum -y install samba-common-tools

A command like the following will work^[8]

net rpc shutdown -r -W <domain> -S <hostname> --user <username>

Full Script

In this example I am looking through a directory in which a file is written every time a user logs in. The file name has the computer name and the samaccountname.
It also assumes the samaccountname is first.last.
Also that your primary SMTP proxy address is the address you are sending from.
This script is called by the email script, with the email address as input.
The email feedback is important to keep users happy. ^[9]

#!/bin/bash
netreboot_log="/home/<rebootuser>/netreboot_log"
filelocation="<mount point>"
ldapsearch_log="/home/<rebootuser>/ldapsearch_log"
emailsource="<account>@<domain.tld>"
subject="Subject: Reboot Request $(date +%m%d%y%M)"
mail2="/home/<rebootuser>/templates/mail2.txt"
mail3="/home/<rebootuser>/templates/mail3.txt"
mail4="/home/<rebootuser>/templates/mail4.txt"
mail5="/home/<rebootuser>/templates/mail5.txt"
mail6="/home/<rebootuser>/templates/mail6.txt"
mail7="/home/<rebootuser>/templates/mail7.txt"
mail8="/home/<rebootuser>/templates/mail8.txt"

user=$(ldapsearch -x -h <domain.tld> -D "cn=rebootuser,ou=Users,DC=<domain>,DC=<tld>" -w <password> -b OU=Users,DC=<domain>,DC=<tld> "(&(objectClass=user)(proxyaddresses=SMTP:$1))" samaccountname | grep -iEo sAMAccountName:\ [a-z]+\.[a-z]+$ | sed 's/sAMAccountName:\ //')


if ! grep -qiE [a-z]+\.[a-z]+$  <<< $user; then
	echo "$(date) : Failed to retrieve user associated with $1" >> $ldapsearch_log
	echo "$subject" | cat - "$mail2" | sendmail -f $emailsource $1
else
	image=$(echo $(ls -t1 $filelocation | grep -i <machine name prefix> | grep -i $user | head -1) | grep -Eio csp-vdi-[0-9]\{1,2\})
	net rpc shutdown -r -t 0 -W domain.tld -S $image --user <reboot user>%<password>
	if [ "$?" -ne "0" ]; then
		echo "$(date) : Reboot of $image failed" >> $netreboot_log
		echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1
	else
		ping="0%"
		count="0"
		until [ "$ping" = "100%" ];
		do
		if [ "$count" -eq "120" ]; then
			echo "$(date) : $image has not gone down after 2 minutes" >> $netreboot_log
			ping="100%"
			testout="failed"
			echo "$subject" | cat - "$mail3" | sendmail -f $emailsource $1
		else
			count=$((count + 1))
			ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%)
			sleep 1
		fi
		done
		if [ "$testout" != "failed" ]; then
			ping="100%"
			count="0"
			testout="other"
			until [ "$ping" = "0%" ];
			do
			if [ "$count" -eq "300" ]; then
				echo "$(date) : $image has not come back up after 5 minutes" >> $netreboot_log
				ping="0%"
				testout="failed"
				echo "$subject" | cat - "$mail4" | sendmail -f $emailsource $1
			else
				count=$((count + 1))
				ping=$(ping -c1 -W1 $image | grep -Eo [0-9]+\%)
			fi
			done
		echo "$subject" | cat - "$mail5" | sendmail -f $emailsource $1
		sleep 120
		fi
		if [ "$testout" != "failed" ]; then
			echo "$subject" | cat - "$mail6" | sendmail -f $emailsource $1
		fi
	fi
fi
exit

Restart VMs in vsphere

^[10]

Systemd Unit File

The key here was the "ReaminAfterExit" which was nice to learn about. Red Hat dropped the ball and did not have this on their how-to.^[11]
To have the script start at boot:

[Unit]
Description=Reboot VC Application
After=network.target

[Service]
Type=oneshot
User=<rebootuser>
ExecStart=/home/<rebootuser>/reademail.sh
RemainAfterExit=true

[Install]
WantedBy=default.target

[1] ttps://michaelwiki.geekgalaxy.com/w/index.php/Check_IMAP_and_POP_with_OpenSSL

[2] ttp://www.aerus.net/2017/04/29/updatefix-fetchmail-unix-connection-to-outlook-365/

[3] ttp://www.fetchmail.info/fetchmail-man.html#2

[4] ttps://www.ibm.com/support/knowledgecenter/en/SSKTMJ_9.0.1/admin/conf_examplesofusingldapsearch_t.html

[5] ttps://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos

[6] ttps://stackoverflow.com/questions/10408816/how-do-i-use-the-nohup-command-without-getting-nohup-out

[7] ttps://lifehacker.com/5275652/shut-down-your-windows-pc-remotely-from-linux

[8] ttps://www.peterdavehello.org/2015/05/remotely-shutdownrestart-windows-via-linux-on-debianubuntu-based-linux/

[9] ttps://stackoverflow.com/questions/3005457/combining-echo-and-cat-on-unix

[10] ttps://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx

[11] ttp://tuxgraphics.org/npa/systemd-scripts/

[1]

[2]

[3]

[4]

[5]

[6]

[7]

[8]

[9]

[10]

[11]

@@ Line 101: / Line 101: @@
 =====Parsing directly from email=====
-In this sample we want to pull the email address and subject from the email.<ref>https://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos</ref>
+In this sample we want to pull the email address and subject from the email.<ref>https://serverfault.com/questions/513358/how-to-delete-all-mails-of-user-in-linuxcentos</ref><ref>https://stackoverflow.com/questions/10408816/how-do-i-use-the-nohup-command-without-getting-nohup-out</ref>
 <pre>
 #!/bin/bash
@@ Line 144: / Line 144: @@
 *Also that your primary SMTP proxy address is the address you are sending from.<br>
 *This script is called by the email script, with the email address as input.
+*The email feedback is important to keep users happy. <ref>https://stackoverflow.com/questions/3005457/combining-echo-and-cat-on-unix</ref>
 <pre>
 #!/bin/bash
@@ Line 204: / Line 205: @@
 			done
 		echo "$subject" | cat - "$mail5" | sendmail -f $emailsource $1
-		echo "Waiting 2 minutes for $image to finish booting"
 		sleep 120
 		fi
@@ Line 217: / Line 217: @@
 ====Restart VMs in vsphere====
 <ref>https://virtualizationreview.com/articles/2017/06/01/how-to-install-and-use-powershell-and-powercli-on-linux.aspx</ref>
+===Systemd Unit File===
+The key here was the "ReaminAfterExit" which was nice to learn about. Red Hat dropped the ball and did not have this on their how-to.<ref>http://tuxgraphics.org/npa/systemd-scripts/</ref><br>
+To have the script start at boot:
+<pre>
+[Unit]
+Description=Reboot VC Application
+After=network.target
+[Service]
+Type=oneshot
+User=<rebootuser>
+ExecStart=/home/<rebootuser>/reademail.sh
+RemainAfterExit=true
+[Install]
+WantedBy=default.target
+</pre>

Difference between revisions of "Reboot Machines by Monitoring email Instructions"

Latest revision as of 10:11, 26 April 2018

Contents

Purpose

Process

Email Config

Sendmail

Fetchmail

Parse Received Email

Resolving user names from email

LDAP Search

Parsing directly from email

Restarting Machines

Restart Windows Machines Using Native Tools

Full Script

Restart VMs in vsphere

Systemd Unit File

Navigation menu

Personal tools

Namespaces

Variants

Views

More

Search

Navigation

Tools