Objective

To get the data! Having the data is the first part of everything behind including analysis/dashboard/automated alerts/actions etc.

Methodology

Scrapy, Selenium, Puppet, Requests, XPath, Pandas, Distributed programming.

Insight

Well. That's just a crawler I replicate the one I built for my previous company, and of course the most critical optimization part cannot be posted on Github :)