Automate the [Luxury] Stuff with Python

I've been in the market for a new mountain bike for a few months now. The pandemic has put a strain on a lot of supply chains, including the bike market. Shops are selling out quickly and I wanted to stop having to manually refresh my local bike shop's ecommerce page every hour or so to see if they came into more stock. Que Python, AWS Lambda and SMS. ‍

The structure of this system was built using python, scraping my local bike shop for it's inventory in my price range and pushing that information to my phone via SMS through a SMTP server (gmail). Lastly, this code was plopped into AWS Lambda and set to run at specific time intervals, twice a day.

Virtual Environment

• Create a project folder and start and activate a virtual env. On Windows:
• Start cmd: py -m venv env
• Activate: .env\Scripts\Activate
• Install necessary dependencies via pip, bs4 and requests
• Deactivate the environment cmd: deactivate
• Create a python file and place inside env -> - Lib -> sitepackages

Scraping

• Two import functions are necessary here, one to scrape the site for their bike stock and one to send that string to my phone
• Find the HTML tags and DOM hierarchy to pull the title of the bike, inspect the desired element. In this case, a div named 'seProductTitle', within this div there is an href tag and we need the text from it

• Normalize by removing newlines, add to larger string and send to the SMS function
• Encase code in a function called lambda_function(event,context) taking in two arguments. This is necessary for the Lambda function to call and execute the script

def get_bike_items(url,price_range):
    response = get(url)

    html_soup = BeautifulSoup(response.text, 'html.parser')
    all_bikes = html_soup.find_all('div', class_ = 'seProductTitle')

    bike_names ="Price Range " + price_range
    for bikes in all_bikes:
        text = bikes.a.text
        biken = text.replace("\n","")

        bike_names+= " " + biken + "||"

---

SMS

• Create Gmail account solely dedicated to sending out emails
• Establish email protocol and authorize gmail account
• Send message

def send(message):
        # Replace the number with your own, or consider using an argument\dict for multiple people.
    to_number = '0000000000{}'.format(carriers['att'])
    auth = ('**email**', '**password**')

    # Establish a secure session with gmail's outgoing SMTP server using your gmail account
    server = smtplib.SMTP( "smtp.gmail.com", 587 )
    server.starttls()
    server.login(auth[0], auth[1])

    # Send text message through SMS gateway of destination number
    server.sendmail( auth[0], to_number, message)

---

Serverless Computing

• Zip all of the contents in the sitepackages folder. The most important note here is that all packages and python files must be in the same root, zipped together
• Create a new function in AWS Lambda, assign a new basic role permissions
• Upload the zip file from step 11, choose your runtime. Python 3.8 in my case
• Under Runtime settings In the handler role, input the name of your file followed by the name of the function, in our case lambda_function, separated by a period -> Edit

• Increase the timeout time under general configuration -> Edit

• Add a trigger to instruct AWS when and how often to run the packaged code
• In this case, I wanted to grab data early in the morning and early evening, 4AM EST and 4PM EST. Note that crons time are in UTC so convert accordingly
• Add trigger -> search for EventBridge(CloudWatch Events) -> add a name and a cron expression. cron(0 08,20 ? *) for my desired time. Cron calculator ‍
• Test function to ensure it's working properly, troubleshoot if not. The most common error has to do with incorrectly installed dependencies, mainly due to different versions. i.e. using Python 3.7, but selected runtime 3.8 selected and packages are outdated
• Finished!

No longer do I have to tirelessly refresh my page manually! 😝

---

Update

Bike acquired :)

---