The Portfolio of Derek Brooks

Scraping

Though I don't have a whole lot to show for it in my portfolio I've done quite a bit of scraping. Everything from web page scraping to scraping poorly formatted CSV exports.

Here are 4 projects that I've worked on tagged Scraping.

Iowa Alcoholic Beverages Division

Screenshot of Iowa Alcoholic Beverages Division
Iowa Alcoholic Beverages Division is responsible for the regulation and control of all alcohol and tobacco in the state of Iowa. The division has generated hundreds of millions of dollars for the state of Iowa. Our company, Red 5 Interactive had hosted IABD for years. Up until this rebuild, they were one of the very last Java based sites running on an ancient server stuck in a closet in the back of our office. As time went on, it became scarier and scarier to let their site remain live. Once showing IABD the physical server they were running on, talking them into a complete rebuild was no challenge at all. One of the most daunting parts of this rebuild was the task of migrating all their content. I heavily modified our internal Site Spider to accomplish this task. The client had around 400 pages and 700 files that were linked to (mostly PDF press releases). I had to completely crawl their site, import everything into our CMS and fix every link to work correctly. Given the fact that they had FTP access to the old server, things were a mess, and this was not a simple task. I'm talking about some serious site scraping madness! Being such a large site, this client came up with all kinds of awesome ideas for our CMS, which I was able to add-on and commit back to the repository for our other clients. After setting up our internal CMS and importing their hundreds of pages/files, I had to build a Javascript Fund counter to display on their homepage. Basically, the client takes their projected income, enters it into a custom tool that I built in SiteMan, and my JS did the math to determine how much money was being transferred to the state every second. It then dynamically animated a count-up on the homepage. I also built some fancy, custom (and totally dynamic) javascript dropdown navigation. Finally, I had to model and build a custom plugin to allow IABD to import CSV files of their alcohol products, stores, and tobacco compliance data. Then, I built functionality to browse and search all of this data on the front-end. At the time, Iowa Alcoholic Beverages Division was basically a test of all my skills. Sys admin, back-end development, front-end development, scraping, importing, and finally deployment.

Site Spider

Site Spider is an internal application that we wrote at Red 5 Interactive. Its purpose was to crawl an entire website and save everything (pages, stylesheets, javascripts and images) locally. We originally wrote the app when a client's previous web host was unwilling to surrender their data. It was incredibly useful for our company to scoop up new business from uncooperative providers at the time.

NAPA Sales Driver

Screenshot of NAPA Sales Driver
Edwards Graphics Arts (a partner to NAPA) hired our company to build a web application for distributing posters and flyers to NAPA's retail stores. Each store could purchase any of several different promotional products. This is a process that NAPA and it's 3rd party designers had previously handled over the phone. I built a web application to help streamline this process and eliminate phone ordering. This app allowed Edwards Graphic Arts to upload and edit designs for the stores to order. Retail store managers could then visit the site and order promotional products that fit their demographic. This site was very javascript heavy with a lot of asynchronous calls to provide a quick user experience. I also integrated it with PayPal's Pay Flow Pro to allow for online payment.

Glimcher

Screenshot of Glimcher
Glimcher is a premier real estate investment trust that manages malls and shopping centers across the United States. They came to my company, Red 5 Interactive, seeking a corporate website. My company had already built and launched this site by the time I began working on it, so my job was simply to enhance some features on both the admin and the front-end. Glimcher wanted a near real-time display of their stock price on the homepage. However, there weren't any great services to grab this information at the time, so I built a CRON scheduled script to scrape a couple quote sites and grab their stock price. Whenever the price was successfully grabbed, the price was written to a local file that was read by the website. Glimcher also wanted real-time stats on their leasing informational pages, so I added some functionality to display updates on leasable square feet, property count, etc. Finally, they wanted to be able to quickly show/hide all dynamic content on their website. I added some simple toggling as well as drag/drop sorting to any and all data in the CMS.