Also good is RoboBrowser which combines beautifulsoup with Requests to get a nice 'Browser' abstraction. It also has good built-in functionality for filling in forms.
Any details on this anywhere, or is it not for public consumption? I'm just getting started in Python and want to do something with Gumtree and eBay as an idea to help me in a different sphere.
It's not really for public consumption because it's embarrassingly badly written :)
It's pretty dumb really. Just figured out the search URLs and then parse the list responses. It then stores the auctions/ad IDs it has seen in a tiny redis instance with 60 days' expiry on each ID it inserts. If there are any items it hasn't seen each time it runs, it compiles them in a list and emails them to me via AWS SNS. Runs every 5 minutes from cron on a Raspberry Pi Zero plugged into the back of my XBox 360 as a power supply and my router via a USB/ethernet cable.
The main bulk of the work went into the searches to run which are a huge list of typos on things with a high return. I tend to buy, test, then reship them for profit. Not much investment gives a very good return - pays for the food bill every month :)
Thanks for the info - I'm sure mine will be of lower quality when I do write it - hoping to compile real-world info on sold vehicles by scraping info from eBay and Gumtree, but that will take time and more skills than I currently possess. Good to hear someone's made something out of a similar idea, though.
