I love getting free stuff. Who doesn’t. The problem is that there’s no reliable way to continually get free stuff. I picture a sea anonenomie, waiting for plankton to swim into its mouth as my technique for acquiring things in the past. Now however, I have identified a way to jack directly into the community of people who have excess items to get rid of. Now I am like the darwins apex sea anonenomie, free from the coral, hunting the plankton in the open ocean.
Finding that community of people who want to get rid of stuff is the easy part. In my neck of the woods there’s only 2 games in town. Facebook Marketplace and Craigslist. The choice was easy for me… I loathe facebook marketplace with a passion. So this free stuff hack is done - you really don’t need any more technology than just going to the free section of craigslist and waiting for something interesting to get posted. So you could sit there and refresh the browser endlessly OR… you could use python to do it for you. The implementation I envisioned is I run my python script indefinitely on a server at home and I receive alerts to my phone in real time as free stuff gets posted.
So there’s a library for this… here but I couldn’t get it to work for me. Looks like Craigslist has fortified their website against plain requests which is what this library is using under the hood. You get a html page that looks like it would be the correct craigslist page, except theres no posts and inside the script tag theres this message from the developers:
EDIT: I just pulled this error from an issue on the github repo because when I tried to reproduce it just now, I got the whole kitchen sink from craigslist - with no error!! I specifically remember getting this error 3 months ago when I was playing with this project. I will perhpas do some more investigation to see what’s going on but for now I am going to keep writing my blog post as if I didn’t see this.
I wrote this solution on an Arm macbook and then deployed it on a raspberry pi 4 running 64bit raspbian (I am informed that 32 bit doesn’t have a geckodriver or chrome driver). I wanted to practice my databasing skills so I ran a postgres database and interfaced with it using sqlalchemy. To bypass the issue above, I followed the advice of the error message and scraped it by running an actual browser with selenium.
Here is my models.py file:
The funkiness that is the get_db function is a result of the postgres tablename being available at runtime based on the config file passed to the script.
Then the actual script that does the scraping:
Where the config file looks like:
The url in the config file comes from going to the craigslist website and using their GUI to refine your search, in my case I used the map to circle a radius of 5.1 miles around my neighborhood. Any farther and I received way too many text messages.
I noticed while running this script that Craigslist batch posts ads with the rollout distributed instead of being universally available for everyone all at once. Looks like they post the ads about every 15 minutes or so, and I have seen posts on my phone posted up to 3 minutes before my bot notifies me that they’re available and vice versa. So it’s not quite fish in a barrel when trying to get an item, there’s still some lottery luck that plays into it. After missing out on a TV I wanted, I made an iOS shortcut that will allows me to use a template to send emails so I can be that much faster and have that much more likihood of being the first emailer.
I’ve had trouble sharing the shortcut so i’ll just include a screenshot of it so you can replicate it yourself.
Now I finally am the junkyard king of my neighborhood in seattle. If I see something that gets posted - I will have it. Since running the bot paired with my shortcut, I have only fumbled one connection. Usually I get a response that looks like this, which I receieved in response to picking up a Sony OLED 55XBR TV:
Note the “you’re the first to respond - pretty quick” - when I tell you I felt smug, oh boy… This is how the people who used a bot to get a 4080 during the GPU crisis must have felt.