Every browser automation script makes a promise it never writes down: that the page will look tomorrow the way it looked on the day the script was written. Websites never agreed to that, which is why scripted browsers have a reputation for breaking. Most of the reputation is earned.
The Workflow Automation Tool on my projects page is built with Python, Selenium and REST APIs, and that pairing of browser and API is how I think about the whole problem. Call an API when you can, drive a browser when you have to, and write the browser parts on the assumption that the page will change under you.
Two reasons scripts break
The first is layout. A script finds things on a page by description: the third button in the form, the element with a class like css-1x9fk2. Then the form is redesigned, a build step regenerates the class names, or a cookie banner lands on top of the button. The site still works perfectly for people. Your script is the only visitor that noticed.
The second is timing. A person waits for a page without thinking about it. A script does exactly what it is told, as fast as it can. If it clicks before the button exists, or reads a table before the data arrives, it fails, but only sometimes, which is worse than failing every time. The usual patch is a sleep: pause three seconds and hope. Three seconds is too long on a good day and too short on a bad one.
First, check whether you need a browser at all
Before writing any Selenium, look for an API. Plenty of systems that seem to have only a web interface also have a documented REST API, a scheduled export or a report that can be emailed. An API returns data in a fixed structure, tends to be versioned, and usually changes with notice. A web page changes whenever someone improves the design.
Sometimes the page itself fetches its data as JSON from an address you can see in the browser's developer tools. Calling that directly often works, but it is undocumented and may be against the site's terms, so read them first. A documented API beats a scripted browser every time. Keep the browser for the steps that have no other way in.
Selectors that survive a redesign
When you do need the browser, choose how you find things as carefully as what you do with them. In rough order of preference:
- A dedicated test attribute such as
data-testid. If the system is yours, add them; they exist for exactly this. - An
idornamethat means something.name="invoice_number"will outlive any generated class. - Visible text or an accessible label, for buttons and links. "Download report" changes less often than the markup around it.
Avoid the fragile ones: long XPath expressions copied from the browser's menu, positions such as "the fourth row", and generated class names. Each describes today's page in far more detail than you need, and every extra detail is one more thing that can change. Keep all the selectors in one place, a small dictionary or one class per screen, so a redesign means editing one line rather than hunting through the script.
Wait for something, not for a while
Replace every sleep with an explicit wait: a condition and a time limit. Selenium's WebDriverWait checks the condition repeatedly until it is true or the limit passes. On a fast day it moves on the moment the element is ready. On a slow day it waits as long as it needs, up to the limit. And when the limit passes, it fails with an error about what never appeared, rather than a confusing one three steps later.
import time
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome()
wait = WebDriverWait(driver, timeout=20)
try:
driver.get("https://example.com/reports")
# clickable, not merely present
button = wait.until(EC.element_to_be_clickable(
(By.CSS_SELECTOR, "[data-testid='download-report']")))
button.click()
# wait for the result of the click, not for a fixed time
status = wait.until(EC.visibility_of_element_located(
(By.ID, "export-status")))
print(status.text)
except TimeoutException:
stamp = time.strftime("%Y%m%d-%H%M%S")
driver.save_screenshot(f"failure-{stamp}.png")
with open(f"failure-{stamp}.html", "w", encoding="utf-8") as f:
f.write(driver.page_source)
raise
finally:
driver.quit()Wait for the result of each action too, such as a confirmation message or a new row, before the next step. And use explicit waits instead of Selenium's implicit wait setting, not alongside it: mixed together, they make timeouts hard to predict.
Fail loudly, with evidence
A script that runs unattended will fail when nobody is watching. The job is to make that failure explain itself.
- A screenshot at the moment of failure. The snippet above saves one. It answers most questions at a glance: a new layout, an unexpected login screen, an error page, a banner covering the button.
- The page source beside it, for the cases the picture does not explain.
- A log line for every step, with the time. Opened the page, signed in, found the report, downloaded it. When it breaks, the last line tells you where.
- An alert to a person. A failure that only reaches a log file nobody reads gets discovered at month-end, along with the missing figures.
Retries need care. Retrying a page that loaded slowly is harmless. Retrying a step that submits a form may submit it twice. Make each step safe to repeat, or check whether it already happened before trying again.
Breaking well
Sites change, and scripts that drive them will always need some attention. What you can choose is how they break: quietly, passing wrong data downstream, or loudly, with a screenshot of the new layout and one selector to update. The second kind can be handed over with a plain-English runbook and fixed by whoever inherits it, without a call to the person who wrote it.
The Workflow Automation Tool is not a public link; demo access is on request through my contact page.
