If you can get away without a JS environment, do so. Something like scrapy will be much easier than a full browser environment. If you cannot, don’t bother going halfway and just go straight for headless chrome or Firefox. Unfortunately Selenium seems to be past its useful life as Firefox dropped support and chrome has a chrome driver which wraps around it. Phantom.js is woefully out of date and since it’s a different environment than your target site was designed for just leads to problems.
I manage the WebDriver work at Mozilla making Firefox work with Selenium. I can categorically State we haven’t killed Selenium. We, over the last few years, have invested more in Selenium than other browsers.
Selenium IDE no longer works in Firefox for a number of reasons;
1) Selenium IDE didn’t have a maintainer
2) Selenium IDE is a Firefox add on and Mozilla changed how adding worked. They did this for numerous security reasons.
My apologies, I was mistaken, but I can't edit my post now. It looks like the selenium code has moved into something called geckodriver, which I suppose is a wrapper around the underlying Marionette protocol.
Firefox did not drop support for Selenium. Selenium IDE, a record/playback test creation tool, stopped working in newer versions of Firefox, but a) Selenium IDE is only one part of the Selenium project, and b) The Selenium team is working on a new version of IDE compatible with the new Firefox add-on APIs.
