In my previous post, https://ssscripting.wordpress.com/2009/02/24/how-to-scrape-data-from-sites-you-cant-log-into/, I showed how you can use a cookie from the web browser to speed up the scraping process. However, due to the fact that a website can have more than a cookie, this process is a bit error prone ( because you have to concatenate all the values in a string ). While testing something with netcat, I found a faster way, that is virtually without errors.
All you have to do is to start netcat and set it to listen on a specific port. Here’s how you do that:
nc -l -p 9000
Next, you have to configure your browser’s proxy to “localhost” , port 9000 ( or whatever port you specified ). Here is how you do it in firefox . Go to Options/Preferences, and then get to this screen:
From here, click on Settings, and fill in the proxy related details. After you’ve done this, visit the site to which you want to find the cookie. Look in the terminal/console you opened nc in and you will see the HTTP request. Look for the Cookie header, and copy it. From here on, you can follow the steps in the other article.