Recommended Tools




Unix shell

  • bash website (though not necessary for Mac/Linux systems)
  • Git bash for Windows


For Web Scraping

  • Chickenfoot: A FireFox extension that allows you to script web page manipulation and the ability to intelligently download data into a useable file from your browser.
  • DownThemAll: A FireFox add-on that allows you to download all the files or images in a webpage and more.

In Python

  • Requests allows you to send HTTP/1.1 requests easily.
  • lxml is an easy-to-use library for processing XML and HTML in the Python language.
  • Beautiful Soup is a Python library designed for quick turnaround projects like screen-scraping.
  • Selenium and Headless Chrome is useful if you need to interact with a browser or the site needs to run JavaScript.

In R