Scripting file downloads from secure sites

From Michael's Information Zone
Jump to navigation Jump to search

Purpose

I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well.

Process

I started with using Lynx as I could log into the web form. This allowed me to save session cookies, but I had an issue with the trusted device cookie. Curl seems to be the answer as I can post data, save cookies, and create an additional "session" using the saved data. Though I was looking forward to using Lynx, learning curl will be more beneficial in the long run. [1][2][3][4][5]

I figured out I need to do the following

  1. POST login information to the login form, while sending the trusted device cookie.
  2. Save the session cookie.
  3. Download the files using both the session and trusted device cookie.

So far I have the following 

curl -c temp.file -b xxxxx=xxxxxxx -d 'username=xxx&password=xxx&Login.x=27&Login.y=8&login-form-type=pwd&CUR_ST_PAGE=file+login&CUR_ST_ERROR&CUR_ST_ERROR_CODE&CUR_ST_USERNAME=unauthenticated&CUR_ST_LOCATION'
  1. ↑ https://www.linuxquestions.org/questions/programming-9/click-%27submit%27-in-a-remote-web-form-from-bash-really-882839/
  2. ↑ https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies
  3. ↑ https://williamjxj.wordpress.com/2010/12/17/curl-vs-wget-vs-lynx/
  4. ↑ https://stackoverflow.com/questions/51882330/curl-or-lynx-scripting-with-chrome-cookie
  5. ↑ http://arquivo.splat-n.com/scripts/lynx-saving-cookies-and-scripting/
Retrieved from "https://wiki.1701technology.com/index.php?title=Scripting_file_downloads_from_secure_sites&oldid=899"