Analyzing S3 and CloudFront access logs with AWS RedShift

Note that I originally wrote this article for theScore Tech Blog.

It’s not really a surprise that we deal with fairly high volumes of user traffic at theScore. We use AWS for pretty much all of our hosting needs and their two object distribution services – CloudFront and S3 – are two of the most heavily used offerings in our environment. S3 provides file storage for essentially unlimited data, split across as many “buckets” as you would like to create. Separate from that is CloudFront, which is a content distribution network similar to Akamai or Fastly. CloudFront points at one or more origins, and adheres to standard HTTP caching headers. Typically we will deploy S3 buckets or application servers behind CloudFront “distributions” – this allows the majority of HTTP requests to our properties be soaked up by CloudFront edge servers.

Read full article

Posted in AWS | Leave a comment

Splunk deployment retrospective

Note that this article was written for the Points.com blog.

It’s been slightly over a year now since Points.com rolled out Splunk across our systems. For those not in the know, Splunk is basically a datastore for unstructured data – generally it’s described as centralized logging, and it’ll certainly fill that role but it’s capable of a lot more. This post is sort of an overview of things I wish I had planned better beforehand.

Read the full article

Posted in Tech | Leave a comment

Site refresh.

Nothing earth-shattering here, just a quick refresh with a new theme and URL. I’m not sure if that means I’ll blog again in the near- or far-future, but the things I have put here have some value to me so this site isn’t going offline anytime soon.

Posted in Uncategorized | Leave a comment

I can’t believe I ever managed…

I can’t believe I ever managed to get by without the awesomeness of modern SSDs.

Posted in Twitter | 2 Responses

Fan Expo 2009

Derek and us went to Fan Expo 2009. We all had a good time, and I posted a bunch of pictures on Flickr.

Posted in Uncategorized | Leave a comment

Vacation photos.

So Chie and I came back this week from a tour of Atlantic Canada, and took a billion pictures. I’ve posted some of them on Flickr if anyone’s interested!

Posted in Travel | 1 Response

Technology hell.

This has been the weekend from hell. On Thursday we lost our Internet access and I spent close to 3 hours on the phone with various Sympatico people trying to diagnose the problem. I can’t keep a connection to their service. Their conclusion is that suddenly all three of our phone jacks have “gone bad”. How this happens is a mystery to me, but whatever. So then I have to call Bell and schedule a repair person to come who’ll show up between 8AM and 4PM on Thursday.

So because I work on call and have to connect to work from home, I grabbed the cheapest package Rogers has for the Internet. It works, but it’s about 15 kilobytes per second in each direction. It’s fine if I’m not using BitTorrent on it (which Rogers fucking throttles anyways).

Now in the midst of plugging in the new cable modem my router, a Linksys WRT54G pre-VXWare fiasco, fries. As in suddenly it resets and when it comes back none of the network interfaces on it work anymore. So I factory restore it, change the firmware, and so on and nothing. It responds just fine on the wireless LAN but will transmit no data on it’s wired jacks.

Pissed off, I go and replace the stupid router and get a new power bar with a surge protector for the cable line and the phone line, on the off chance that the power is fucked up enough to damage equipment.

To add to the whole mess I went into work on Saturday morning to upgrade our firewall there (since it’s pretty much the one thing I can’t do remotely) and it should have been seamless. It was almost seamless, except post-upgrade it wouldn’t accept any more inbound VPN connections due to a bug in the configuration update system. A bug that their engineers confirmed, yet no one else has had before. Luckily rolling back the firmware and restoring was no big deal.

And to top everything off, over the last couple weeks my Xbox 360 has been sounding more and more like a goddamned jet engine spooling up in our living room, getting so loud that I have to keep increasing the volume on the games to drown it out. Well I don’t have to worry about that anymore, since now it won’t read discs! I’ve called and they’re replacing it and sending a box this week. For anyone keeping score this is the second time since January that a 360 has croaked on me.

So yeah, the last few days have been last than fun.

Posted in Games, Tech | 16 Responses

(Untitled)

I finished my first day today as at my new job – I’m one of two Senior Unix Administrators at Points.com. It actually went really well. I don’t want to get ahead of myself since it was one day, so I’ll probably post more on Friday or the weekend or something.

Speaking of the weekend we’re heading up north for a week on Saturday to visit the parents. I can’t wait to freeze my ass off.

Posted in Uncategorized | 4 Responses

We’re back.

We made it back on the weekend, and then guess what – I got sick! I mean really sick, like horribly violently sick. It sucked. I had a couple of interviews earlier this week that I had to reschedule to today and yesterday because I couldn’t leave our place.

Other than that the trip went well and we had a blast. Sometime soon all of us Guitar Hero people gotta get together for a session. You know who!

Posted in Uncategorized | 1 Response

(Untitled)

Our family here went on a three day trip up to Izu this weekend to go to an Onsen resort (hot spa kind of thing). I posted a ton more photos to Flickr and some are private; if anyone wants access just either comment with you Yahoo ID or e-mail me.

Also if Yahoo phases out Flickr in favor of their subpar “Gallery” site I’ll be totally pissed, what with having just upgraded to the premium version.

Posted in Uncategorized | 2 Responses