Every year, the digital operations team at the Daily Texan builds an interactive candidate explorer for students to get information on the people running for student government. I had built the project in 2016, so it seemed natural that I would work on it again for 2017.
The previous approach used was bootstrapped from the strategy the digital manager took in 2015, because I didn't have a lot of experience in working on something so interactive and data-driven at that point in time. We would receive responses on a Google form, which would then end up in a spreadsheet. A python script would parse that spreadsheet and collect that data into python, where it would be rendered via a flask server. The front-end would then delegate what page to load based on the URL.
The 2016 plan, however, resulted in messy code, a complicated schema, and an unclear data flow that was kept going solely because of manual data cleaning. I knew I needed to do better in 2017.
Using a Django backend, I built a fully interactive, Postgres-backed web app that would take survey results on the site itself, which removed the need for manual data scraping. The results would then be sent to the team to moderate and filter out unsuitable responses, or to follow up for more information. Once that editorial process was complete, we would flag for publish and the candidate would be instantly displayed on the site.
Because we had less control over the asset pipeline (candidates would upload 20MB images that were uncompressed and at 18 megapixels), I chose to use Amazon S3 to handle larger uploads that would have bucked the server we were hosting on. I then wrote a script that would enter our S3 bucket, use the TinyPNG API to compress and resize images down to a web format, overwrite those images, and tag those images so that they would not be recompressed. We then ran this script once a day, which meant that our images were compressed readily, reducing front-end load for the end user.
Originally, the explorer was hosted on DigitalOcean in a Ubuntu box, which caused all sorts of headaches when it came to configuration and deployment. Recently (for archival purposes) I took down the DigitalOcean server and redeployed the explorer with Heroku instead, following the deployment pipeline of the site you're reading this on. I took a dump of the original database and imported it into Heroku PostgreSQL, and so the data on the site is exactly the same as the original one that students would have seen—just that now it's archived for my portfolio.