Scripting file downloads from secure sites - Revision history

Michael.mast: /* Process */

2018-08-17T17:50:14Z

Process

Michael.mast: /* Process */

2018-08-17T12:37:19Z

Process

Michael.mast: Created page with "==Purpose== I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well. ==Process== I started..."

2018-08-17T12:22:32Z

Created page with "==Purpose== I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well. ==Process== I started..."

New page

==Purpose==
I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well.

==Process==
I started with using Lynx as I could log into the web form. This allowed me to save session cookies, but I had an issue with the trusted device cookie. Curl seems to be the answer as I can post data, save cookies, and create an additional "session" using the saved data. Though I was looking forward to using Lynx, learning curl will be more beneficial in the long run.
<ref>https://www.linuxquestions.org/questions/programming-9/click-%27submit%27-in-a-remote-web-form-from-bash-really-882839/</ref><ref>https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies</ref><ref>https://williamjxj.wordpress.com/2010/12/17/curl-vs-wget-vs-lynx/</ref><ref>https://stackoverflow.com/questions/51882330/curl-or-lynx-scripting-with-chrome-cookie</ref>
<br>
<br>
I figured out I need to do the following
#POST login information to the login form, while sending the trusted device cookie.
#Save the session cookie.
#Download the files using both the session and trusted device cookie.
So far I have the following
<pre>
curl -c temp.file -b xxxxx=xxxxxxx -d 'username=xxx&password=xxx&Login.x=27&Login.y=8&login-form-type=pwd&CUR_ST_PAGE=file+login&CUR_ST_ERROR&CUR_ST_ERROR_CODE&CUR_ST_USERNAME=unauthenticated&CUR_ST_LOCATION'
</pre>

@@ Line 11: / Line 11: @@
 #Save the session cookie.
 #Download the files using both the session and trusted device cookie.
-So far I have the following
+I was able to figure out what was needed by using a browser and selecting copy curl, trimming the output, and entering the fields needed. Then I was able to get the session cookie using the device cookie. Unlike what the curl man page said
 <pre>
-curl -c temp.file -b xxxxx=xxxxxxx -d 'username=xxx&password=xxx&Login.x=27&Login.y=8&login-form-type=pwd&CUR_ST_PAGE=file+login&CUR_ST_ERROR&CUR_ST_ERROR_CODE&CUR_ST_USERNAME=unauthenticated&CUR_ST_LOCATION'
+curl -b "xxx=xxx"
 </pre>

@@ Line 4: / Line 4: @@
 ==Process==
 I started with using Lynx as I could log into the web form. This allowed me to save session cookies, but I had an issue with the trusted device cookie. Curl seems to be the answer as I can post data, save cookies, and create an additional "session" using the saved data. Though I was looking forward to using Lynx, learning curl will be more beneficial in the long run.
-<ref>https://www.linuxquestions.org/questions/programming-9/click-%27submit%27-in-a-remote-web-form-from-bash-really-882839/</ref><ref>https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies</ref><ref>https://williamjxj.wordpress.com/2010/12/17/curl-vs-wget-vs-lynx/</ref><ref>https://stackoverflow.com/questions/51882330/curl-or-lynx-scripting-with-chrome-cookie</ref>
+<ref>https://www.linuxquestions.org/questions/programming-9/click-%27submit%27-in-a-remote-web-form-from-bash-really-882839/</ref><ref>https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies</ref><ref>https://williamjxj.wordpress.com/2010/12/17/curl-vs-wget-vs-lynx/</ref><ref>https://stackoverflow.com/questions/51882330/curl-or-lynx-scripting-with-chrome-cookie</ref><ref>http://arquivo.splat-n.com/scripts/lynx-saving-cookies-and-scripting/</ref>
 <br>
 <br>