Scripting file downloads from secure sites
I need to automate downloading files from a site requiring credentials. However, the site also uses a "Trusted this device" cookie as well.
I started with using Lynx as I could log into the web form. This allowed me to save session cookies, but I had an issue with the trusted device cookie. Curl seems to be the answer as I can post data, save cookies, and create an additional "session" using the saved data. Though I was looking forward to using Lynx, learning curl will be more beneficial in the long run.
I figured out I need to do the following
- POST login information to the login form, while sending the trusted device cookie.
- Save the session cookie.
- Download the files using both the session and trusted device cookie.
I was able to figure out what was needed by using a browser and selecting copy curl, trimming the output, and entering the fields needed. Then I was able to get the session cookie using the device cookie. Unlike what the curl man page said
curl -b "xxx=xxx"
curl -H "Cookie: XXXX="%"2Fwpsnew; xxx=xxx"
Where XXXX="%"2Fwpsnew is requesting a new session cookie, and xxx=xxx is the trusted cookie. By adding "-c cookie.file" I can save the generated session cookie and reuse it to download the files on another page.