Scripting file downloads from secure sites

From Michael's Information Zone
Revision as of 07:22, 17 August 2018 by Michael.mast (talk | contribs) (Created page with "==Purpose== I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well. ==Process== I started...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Purpose

I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well.

Process

I started with using Lynx as I could log into the web form. This allowed me to save session cookies, but I had an issue with the trusted device cookie. Curl seems to be the answer as I can post data, save cookies, and create an additional "session" using the saved data. Though I was looking forward to using Lynx, learning curl will be more beneficial in the long run. [1][2][3][4]

I figured out I need to do the following

  1. POST login information to the login form, while sending the trusted device cookie.
  2. Save the session cookie.
  3. Download the files using both the session and trusted device cookie.

So far I have the following 

curl -c temp.file -b xxxxx=xxxxxxx -d 'username=xxx&password=xxx&Login.x=27&Login.y=8&login-form-type=pwd&CUR_ST_PAGE=file+login&CUR_ST_ERROR&CUR_ST_ERROR_CODE&CUR_ST_USERNAME=unauthenticated&CUR_ST_LOCATION'
  1. ↑ https://www.linuxquestions.org/questions/programming-9/click-%27submit%27-in-a-remote-web-form-from-bash-really-882839/
  2. ↑ https://developer.mozilla.org/en-US/docs/Web/HTTP/Cookies
  3. ↑ https://williamjxj.wordpress.com/2010/12/17/curl-vs-wget-vs-lynx/
  4. ↑ https://stackoverflow.com/questions/51882330/curl-or-lynx-scripting-with-chrome-cookie
Retrieved from "https://wiki.1701technology.com/index.php?title=Scripting_file_downloads_from_secure_sites&oldid=898"