To download as HTML in specific location:
curl -L -o /Users/xxx/test/microsoft.html https://www.website.com/en/partners/microsoft
-L or --location - Follow redirects
-o or --output - Write to file
-i or --include - Include the HTTP-header in the output
-I or --head - Fetch the HTTP-header only
-X or --request - Specify a custom request method to use when communicating with the HTTP server
-H or --header - Pass custom header(s) to server
-d or --data - HTTP POST data
-F or --form - Specify multipart POST data
-O or --remote-name - Write output to a file named as the remote file
-u or --user - Specify user and password
-A or --user-agent - Send User-Agent header to server
-v or --verbose - Make the operation more talkative
-s or --silent - Silent mode
-k or --insecure - Allow connections to SSL sites without certs
-x or --proxy - Use the specified HTTP proxy
-X or --request - Specify a custom request method to use when communicating with the HTTP server
-T or --upload-file - Transfer file
-H or --header - Pass custom header(s) to server
-b or --cookie - Send cookies from string/file
-c or --cookie-jar - Write cookies to file after operation
-L or --location - Follow redirects
-i or --include - Include the HTTP-header in the output
-I or --head - Fetch the HTTP-header only
-v or --verbose - Make the operation more talkative
-s or --silent - Silent mode
-k or --insecure - Allow connections to SSL sites without certs
-x or --proxy - Use the specified HTTP proxy
-T or --upload-file - Transfer file
-H or --header - Pass custom header(s) to server
-b or --cookie - Send cookies from string/file
Script to download JSON files from a website with a loop:
#!/bin/bash
# Create the directory if it doesn't exist
mkdir -p path/to/directory
for i in `seq 1 159`; do
echo "Fetching page $i";
curl -o path/to/directory/`printf "%03d" $i`.json 'https://xxxx.com/search/' \
-H 'accept: */*' \
-H 'accept-language: en-US,en;q=0.9,nl;q=0.8,de;q=0.7' \
-H 'content-type: application/json' \
-H 'cookie: [see below for how to get this] \
-H 'referer: https://xxxx.com/listings?page=minions&pageNumber=1' \
-H 'sec-ch-ua: "Google Chrome";v="123", "Not:A-Brand";v="8", "Chromium";v="123"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "macOS"' \
-H 'sec-fetch-dest: empty' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-site: same-origin' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36' \
-H 'x-auth-token: 115987af9dsfafasd9faf7a477a25' \
-H 'x-lang: en' \
--data-raw '{"page":'`printf "%d" $i`',"limit":30}'
done
To get the cookier header, got to Developer Tools, Network, reload the page, identify the cookie and right click here:
