Scripting file downloads from secure sites

From Michael's Information Zone
Jump to navigation Jump to search

Purpose

I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well.

Process

I started with using Lynx as I could log into the web form. This allowed me to save session cookies, but I had an issue with the trusted device cookie. Curl seems to be the answer as I can post data, save cookies, and create an additional "session" using the saved data. Though I was looking forward to using Lynx, learning curl will be more beneficial in the long run. [1][2][3][4][5]

I figured out I need to do the following

  1. POST login information to the login form, while sending the trusted device cookie.
  2. Save the session cookie.
  3. Download the files using both the session and trusted device cookie.

I was able to figure out what was needed by using a browser and selecting copy curl, trimming the output, and entering the fields needed. Then I was able to get the session cookie using the device cookie. Unlike what the curl man page said

Instead of 

curl -b "xxx=xxx"

I needed 

curl  -H "Cookie: XXXX="%"2Fwpsnew; xxx=xxx"

Where XXXX="%"2Fwpsnew is requesting a new session cookie, and xxx=xxx is the trusted cookie. By adding "-c cookie.file" I can save the generated session cookie and reuse it to download the files on another page.

  1. ↑ https://www.linuxquestions.org/questions/programming-9/click-%27submit%27-in-a-remote-web-form-from-bash-really-882839/
  2. ↑ https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies
  3. ↑ https://williamjxj.wordpress.com/2010/12/17/curl-vs-wget-vs-lynx/
  4. ↑ https://stackoverflow.com/questions/51882330/curl-or-lynx-scripting-with-chrome-cookie
  5. ↑ http://arquivo.splat-n.com/scripts/lynx-saving-cookies-and-scripting/
Retrieved from "https://wiki.1701technology.com/index.php?title=Scripting_file_downloads_from_secure_sites&oldid=900"