chealion.ca

A couple weeks back, a Twitter bot I run stopped working correctly. It’s been on my list for a long time to revamp and clean up - if only because the server that it was running on was very overdue for being rebuilt. So when Eco Counter changed their private API and broke the script it provided a great chance to rewrite.

Fun items learned along the way:

Python 3 Upgrade Notes

• Changing print is straight forward.
• f strings are awesome
• Cleaning up data is super time consuming, but cleans things up so nicely.

GitHub Actions

As part of the rewrite I wanted to explore ways to stop managing a server in order to run the bot. As it turns out GitHub actions has a schedule feature and could work quite well for both testing and running the script. They also provide a fantastic learning tutorial at https://lab.github.com/githubtraining/github-actions:-hello-world.

It gave me a great easy chance to get into actually setting up a simplistic CD system for myself, and to get my hands dirty. This coupled with act made it easy to test and run the script through being rewritten.

The whole process is summarized in a sentence, but it took some time and I hope will pay off in spades in the future.

You can find the workflow used on GitHub.

Repurposing or Extending yycbike_count

For those who are interested in taking the work in yycbike_count to use for their own city - please do. There are a couple things to mention if you want to re-use most of the work I did.

Counter Config

The counter config is hard coded into twitterBot.py - sorry.

GitHub Actions

If you want to use the workflow you’ll need to make 4 GitHub secrets correspondding to the environment variables (TWITTER_TOKEN, TWITTER_TOKEN_KEY, etc.). If you want to use act, you’ll also need to add them to the .secrets file.

act

If you want to use act to run the workflow locally you’ll need to add the following to your .actrc file:

-P ubuntu-latest=catthehacker/ubuntu:act-latest

On April 8th I stopped redirecting calgary.bike to Bike Calgary[1] to start showing off the aggregated data that I was pulling together from the 3 Eco-Counter installations. With the source on GitHub, I thought it’d be worth explaining a little of the why and how.

At the start of January, the City of Calgary made public the web page for bike counter on the Peace Bridge with promises of making more available including at least 10 more during the upcoming cycle track pilot. The Peace Bridge counter had data stretching back to April 24th, 2014 and by default was always showing the entire daily data set.

My first curiousity was whether I can could have a bookmark to just show the last week or so worth of numbers which led me to figuring out how the webapp worked. (Good ol’ WebKit developer tools)

After that in tandem with some projects I was looking into for work I decided to start seeing about scrapping the data and storing it somewhere to compare numbers (different installations, averages, weather) more easily. So a big thank you for the people at the City and Eco-Counter for not telling me to “get lost and don’t use things inappropriately”.

As for how - the Python scripts just ask Environment Canada and the counters once a day for their last day’s worth of new data (if possible) and store it in Graphite. Interacting with the data is Grafana 2 behind nginx. All hosted on a tiny instance on some publicly available free compute resources that I just happen to also manage as part of my day job. Funnily, most of the script writing was done during an all nighter at a Denny’s in Kamloops waiting for 4 AM to roll around so I could swap some power cables in a maintenance window.

It’s nothing fancy but it’s fun to see what might come of it when data is made available.

1 - I had registered the domain last year and figured that was a good place to point until I had a better idea of how to use it.