Over the last few years I've started to become a lot more aware of various carbon emissions created by myself and the people around me. From non obvious things like the food we eat, to the worst offenders such as travelling. In the 21st century it's increasingly hard to not have a negative impact on the planet by simply just existing and going about our daily lives. I've curated a list of the ways I'm trying to offset my impact with small lifestyle changes.

These are in no particular order, but generally the ones I use more often are closer to the top.

## Ecosia

https://www.ecosia.org

For over a year now I've been using Ecosia as my primary search engine. It was a little disorienting at first, having used Google for so long, but I slowly grew to love it and now Google feels a little alien to me. The beauty of this passive approach is that every time I search (hundreds and hundreds per day) it contribute towards tree planting and funding projects that help our planet. About every 45 searches plants one tree, so I can sometimes plant many in just one day by making a tiny adjustment to my search behaviour.


## Ecosia Shop

https://ecosia.teemill.com

A quicker way to increase that tree count and also add to your environmentally friendly wardrobe, is to check out the Ecosia shop. I have so many of these now, and they're such high quality. New designs are released regularly and each t-shirt plants 20 tress. There's also hoodies and jumpers, if t-shirts aren't quite to your liking.


## Compensaid

https://compensaid.com

If you do a lot of flying, this is a great way to pay back the emissions debt by also planting trees. The extra twist with Compensaid if you can offset instantly by funding synthetic fuel for use in passenger and commercial airlines. A slightly more costly option, but it's a much quicker ROI than the 15-20+ year it takes for a full tree to grow and absorb enough Co2.


## Offset Earth

https://offset.earth/marcqualie?r=5d3d8c8df7e0190010e0cf13

Shortly after setting this up I completely forgot about it, and I now (at the time of writing) have 798 trees planted with a total of 46.64 tonnes of carbon offset. This cost me very little, required no effort, and had a decent impact on the planet.

Use my [referral link](https://offset.earth/?r=5d3d8c8df7e0190010e0cf13) and we'll both get bonus trees in our forests.



Feel free to share the carbon offsetting initiatives you're part of and I'll also continue to expand this list regularly as I find more services.

Carbon Offsetting

Configuring IAM is essential for any AWS account. The most common use case is granting access to EC2 instances, or allowing web apps to read/write to S3. While it's not recommended to make any direct changes in the AWS console, it's sometimes helpful to allow humans to browse around and view configurations.

To simplify management of these user accounts, it's helpful to allow individuals to manage their own credentials and configure things like MFA devices. Below is the policy I created as a drop in starting point for allowing any human users to login to the console. Note, this does not grant any access to any resources inside AWS, only for a user to manage their own password, access keys and MFA devices. Permissions for resources still need to be added otherwise the console will be a very boring place.

The easist way to apply this policy to all humans is to create an IAM group called `humans`. Whenever you wish to allow a user to login the AWS console and manage their own account, simply add them to the `humans` group.



### Allow viewing of account status

Users need to be able to see what their account settings currently are.

```json
{
  "Sid": "AllowViewAccountInfo",
  "Effect": "Allow",
  "Action": [
    "iam:GetAccountPasswordPolicy",
    "iam:GetAccountSummary",
    "iam:GetUser",
    "iam:ListVirtualMFADevices"
  ],
  "Resource": "*"
}
```



### Change Password

A user should be allowed to change their own password without contacting their administrator.

```json
{
  "Sid": "AllowManageOwnPasswords",
  "Effect": "Allow",
  "Action": [
    "iam:ChangePassword"
  ],
  "Resource": "arn:aws:iam::*:user/${aws:username}"
}
```



### Access Key Management

If users want to interact with the AWS APIs, they'll need to generate access keys that inherit their policy permissions.

```json
{
  "Sid": "AllowManageOwnAccessKeys",
  "Effect": "Allow",
  "Action": [
    "iam:CreateAccessKey",
    "iam:DeleteAccessKey",
    "iam:GetAccessKeyLastUsed",
    "iam:ListAccessKeys",
    "iam:UpdateAccessKey"
  ],
  "Resource": "arn:aws:iam::*:user/${aws:username}"
}
```



### Manage MFA

Using MFA (Multi-Factor Authentication) is highly encouraged, so all users should be able to manage their own virtual MFA devices.

```json
{
  "Sid": "AllowManageOwnUserMFA",
  "Effect": "Allow",
  "Action": [
    "iam:DeactivateMFADevice",
    "iam:EnableMFADevice",
    "iam:GetUser",
    "iam:ListMFADevices",
    "iam:ResyncMFADevice"
  ],
  "Resource": "arn:aws:iam::*:user/${aws:username}"
}
```



### Full Policy

The full policy combined is dsiplayed below in full.

```json
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AllowViewAccountInfo",
      "Effect": "Allow",
      "Action": [
        "iam:GetAccountPasswordPolicy",
        "iam:GetAccountSummary",
        "iam:GetUser",
        "iam:ListVirtualMFADevices"
      ],
      "Resource": "*"
    },
    {
      "Sid": "AllowManageOwnPasswords",
      "Effect": "Allow",
      "Action": [
        "iam:ChangePassword"
      ],
      "Resource": "arn:aws:iam::*:user/${aws:username}"
    },
    {
      "Sid": "AllowManageOwnAccessKeys",
      "Effect": "Allow",
      "Action": [
        "iam:CreateAccessKey",
        "iam:DeleteAccessKey",
        "iam:GetAccessKeyLastUsed",
        "iam:ListAccessKeys",
        "iam:UpdateAccessKey"
      ],
      "Resource": "arn:aws:iam::*:user/${aws:username}"
    },
    {
      "Sid": "AllowManageOwnUserMFA",
      "Effect": "Allow",
      "Action": [
        "iam:DeactivateMFADevice",
        "iam:EnableMFADevice",
        "iam:GetUser",
        "iam:ListMFADevices",
        "iam:ResyncMFADevice"
      ],
      "Resource": "arn:aws:iam::*:user/${aws:username}"
    }
  ]
}
```

IAM Policy for humans

I've lost count of the amount of times I've had to research this on a fresh install or different machine. Since python is not a language I use day to day, I constantly forget the environment configuration so I'm putting it here for quick reference when I will inevitably need it in a month or two.

Full credit goes to [@tarrex](https://github.com/tarrex) for the original comment on [this issue thread](https://github.com/pyenv/pyenv/issues/1219#issuecomment-448658430). I'm simply surfacing the important parts from that thread for quick reference, mainly for my own needs.

For reference, my current system enviroment at time of writing (2019-04-10) is:

- macOS Mojave 10.14.3
- pyenv 1.2.9
- zlib 1.2.11

You're likely to end up here if you searched for an error message along the lines of:

```
zipimport.ZipImportError: can't decompress data; zlib not available
make: *** [install] Error 1
```

Your first instinct (as mine was) would be to `brew install zlib`, but you'll soon find that won't resolve your issue. The problem lies in fact with the compiler not knowing where `zlib` is located on your machine, rather than it not being installed. Fixing this is, thankfully, quite trivial.

First, you'll want to make sure pyenv and zlib are actually installed and up to date:

``` bash
brew install pyenv zlib
```

Next, you'll need to grab the compiler exports which can be found via the `info` command:

``` bash
brew info
```

This is likely to yield something like the following:

``` bash
zlib: stable 1.2.11 (bottled) [keg-only]
General-purpose lossless data-compression library
https://zlib.net/
/usr/local/Cellar/zlib/1.2.11 (12 files, 373KB)
  Poured from bottle on 2019-04-10 at 09:34:14
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/zlib.rb
==> Caveats
zlib is keg-only, which means it was not symlinked into /usr/local,
because macOS already provides this software and installing another version in
parallel can cause all kinds of trouble.

For compilers to find zlib you may need to set:
  export LDFLAGS="-L/usr/local/opt/zlib/lib"
  export CPPFLAGS="-I/usr/local/opt/zlib/include"

For pkg-config to find zlib you may need to set:
  export PKG_CONFIG_PATH="/usr/local/opt/zlib/lib/pkgconfig"
```

As the output suggests, you want to make sure that `LDFLAGS` and `CPPFLAGS` are set for the compiler (python-build):

``` bash
export LDFLAGS="-L/usr/local/opt/zlib/lib"
export CPPFLAGS="-I/usr/local/opt/zlib/include"
```

You can even take this a step further and setup these environment variables as part of your bash or zsh profile, so you never have to worry about this again.

Finally, you are now ready to install your desired version of python!

``` bash
pyenv install 3.7.2
pyenv local 3.7.2
```

You can now verify your new version of python is activated:

``` bash
which python # $HOME/.pyenv/shims/python
python --version # Python 3.7.2
```

If everything went smoothly, you should now have a working version of python 3.7.2.

pyenv install with zlib

Migrating a schema when deploying code is a fairly solved problem in Rails, and many other languages/frameworks. What hasn't quite been solved, or at least agreed upon, is the ability to make changes to data based on those migrations. I'm not going to cover the dos, donts, pros and cons of each method as many people before me have covered this in great detail. What I do want to do however, is share a nice simple solution I created for simplifying this process, and making the most out of previous methods.

What I came up with is a very simplified solution from a combination of my favorite findings around the internet. First of all, I agree that data should never be ran at the time of schema migration. The links I've included cover this in a lot of detail. The TL;DR of it all is you'll lock up your database, cause downtime and learn a valuable career lesson.

Let's start with a basic migration. Nothing fancy, we just want to keep a copy of the timestamp a user last performed something. This is very helpful for being able to filter and sort users based on this field, without joins and crazy nested ActiveModel helpers. I use this technique often for fields that are queried extremely often, but join on a table with millions of rows and frequent writes. Do the work once at write-time, then read-time performance is greatly improved.

``` ruby
class AddLastSmiledAtColumnToUsers < ActiveRecord::Migration[5.1]
 def change
 add_column :users, :last_smiled_at, :datetime
 add_index :users, :last_smiled_at
 end
end
```

Now that we have this column, what we want to do is populate all users with the most recent value from the join table so that our app is usable. Naive instinct (I was guilty of this a few years ago) is to just pop a loop inside the change method to update all of the users. As I mentioned, the linked articles cover this in detail but the summary is you'll end up with downtime on even small datasets.

You want the data to be separate, but ideally also linked for reference. A gem called [migration_data](https://github.com/ka8725/migration_data) attempts to solve this by adding a `data` method. This solves the problem of having the code separate from the change method, but since it's ran at the same time as the schema migrations, it is still prone to the downtime problem.

My solution is very similar, but I actually embed a whole extra class onto the migration object to separate it even further.

``` ruby
class AddLastSmiledAtColumnToUsers < ActiveRecord::Migration[5.1]
 def change
 add_column :users, :last_smiled_at, :datetime
 add_index :users, :last_smiled_at
 end

 class Data
 def up
 User.all.find_in_batches(batch_size: 250).each do |group|
 ActiveRecord::Base.transaction do
 group.each do |user|
 user.last_smiled_at = user.smiles.last.created_at
 user.save if user.changed?
 end
 end
 end
 end
 end
end
```

While this doesn't appear obviously different, it has a few advantages. It is not run as part of the schema migrations, nor is it part of the actual migration class. It's a raw ruby class, so writing a test for this is extremely easy. You can detect N+1 performance using [bullet](https://github.com/flyerhzm/bullet), ensure the data is correct post-migration and much more without even initializing the ActiveRecord::Migration base. All of that, and it's also tightly linked to the migration for reference and code review.

A final reason I chose this over the `data` method is at some point in the future, I have a feeling ActiveRecord core will attempt to handle data migration and that method may become in use. Keeping our code outside of the core class prevents this.

Now that we can write, test, commit and review our data migration code we need to be able to run it. I avoid including gems as much as possible when you can achieve the same with a few lines of (sometimes hacky) code, so I have a quick two-liner that is responsible for running this data migration.

``` ruby
Dir.glob("#{Rails.root}/db/migrate/*.rb").each { |file| require file }
AddLastSmiledAtColumnToUsers::Data.new.up
```

There are plenty of clever ways to only include the file for the migration you are interested in, but if migrations are written correctly then there's little to no overhead of including them all as they only define class constants. You're probably going to be running this in a one off dyno or equivalent, so the memory is thrown out afterwards anyways.

Another way you could simplify the running of this, and even automate it is to use [active_job](https://guides.rubyonrails.org/active_job_basics.html) and have a `DataMigrationRunnerJob`. Ideally you would want to keep track of which data migrations have ran (similar to how schema migrations are tracked) and implement detailed logging in case something goes wrong.

This has been my personal preference for managing data migrations because it doesn't require any gems, you are in full control of how and when the data script is ran and the code is also committed alongside your schema migration code.

After writing this, I came across the [data-migrate rubygem](https://github.com/ilyakatz/data-migrate) which seems like a very well put together solution for huge applications. My personal preference still remains, but if you're a gem collector or want a full end to end solution with rake tasks and tests, then maybe `data-migrate` is for you.

Feel free to share your own solutions and/or offer feedback on what I've come up with on `Twitter` or in the comments below.

## Resources

- Helpful articles about Rails data migration
 - [https://robots.thoughtbot.com/data-migrations-in-rails](https://robots.thoughtbot.com/data-migrations-in-rails)
 - [https://railsguides.net/change-data-in-migrations-like-a-boss/](https://railsguides.net/change-data-in-migrations-like-a-boss/)

Rails Data Migrations

Although what I'm about to talk about is well documented<a href="#git-push-repository">[1]</a>, I've never come across it in my 10 year career using Git. I stumbled across it out of sheer curiosity to see what would happen if I used a repository instead of a remote name. In theory Git would treat them the same, since everything is technically a ref, and it did. The idea is simply to be able to push your code anywhere, without having to first add the remote to your local repository. You're probably asking why you would want to do this, or why anyone would even care that this is possible. I have a few use cases below which are now way easier day to day.

## Lots of remotes

Git, by design, is decentralised. However, nearly all use cases I've come across have used git in a centralised way. Teams or groups all put their code onto a hub, such as Github or Bitbucket, and each member pulls down the latest code from a single place. When one of these hubs go down, software engineers pretty much declare themselves a free bank holiday.

The irony is that Git is designed in a way where these hubs are completely none-essential in order to actually be used a version control system. With the correct network setup and workflow in place, any user can push and pull code directly between themselves and any other user, without a hub at all. You could push your latest change directly to someone for code review, who could then push it directly to a CI server. That CI server could then push directly to a build system or deployment server. Obviously complex workflows would have to be in place to ensure all team members get all revisions, which isn't trivial, via some sort of eventually consistent mesh topology.

In reality it's rare to use Git like this, but it's entirely possible. if anyone were to use Git is such a way, they would need a remote setup for each team member in order to push and pull code. `git remote add john-smith me@johnscomputer/secretapp1.git` etc. While this could be automated quite easily, it'd still be a pain to keep all of these named origins locally. Likely, you're only going to want to push/pull code to a certain team member once or twice, but they're origin is always around. What if you have a team of 10,000 potential team members? Multiple people with similar names? You see my point. Un-named origins solve this:

``` bash
git push me@johnscomputer/secretapp1.git backdoor-branch
```

A great use case for this in the real world would be to push/pull large changes over a local network in an office environment. For example if there isn't an internal hub, teammates could achieve much higher transfer speeds by direct communication rather than first pushing then pulling from a hub. In my case I push directly to my Macbook from my iMac over SSH before I leave the office. This was mostly useful when I had a 3G internet connection at home and Git would sometimes drop connections or take a long while to pull changes.


## Multi-stage Heroku apps

A more common and realistic problem that I've come across many times myself is multi stage Heroku apps. This can also apply to apps where you run a mini version on your own account rather than the real one for a client. The way heroku-cli works by default is it assumes you have a single remote called `heroku`. You can actually call it whatever you want as internally it checks the remote for heroku style syntax such as `https://git.heroku.com/secretapp2.git`. As long as you have this single endpoint, you can use all commands without having to specify an application or remote everything using `-a secretapp2` or `-r herokuremote`.

Likely, you're going to have `secretapp2-staging` and `secretapp2-production`. As soon as you add that second remote, every single command now needs the `-a` or `-r` suffix which is a pain to type out. 95% of the time, you are going to be running commands on the staging app; Checking logs, shell-ing in to the rails console, testing buildpack configurations. The production remote is really just there for deployments and/or shell-ing in to rails to change a piece of customer data you forgot to make available through the admin panel but customer support team need it right now at 4am or the client will cry themselves to sleep.

Realistically what you want is the single remote to make development commands easier to type. But now do you need to add the production remote, deploy, then remove it again for local CLI zen? No! You can use direct push like above; Since Heroku is just another decentralised node you can push directly to it without a name.

``` bash
git push https://git.heroku.com/secretapp2-production.git master
```

The advantage to this is you can't accidentally deploy to the wrong app when you forget which terminal tab you're in (we've all been there). If you had a named remote then `git push heroku-production master` would deploy master to production, but for the wrong app; Disaster. In an ideal world you could never deploy to production from your local machine. That would all be handled via nicely configured CI that never goes wrong. In the real world we prototype and we work on hobby apps. Many hobby apps; So things like this are bound to happen eventually. Luckily, git has built in protection for this if you use the remote explicitly as above. You will actually get a rejection message because you are pushing content that doesn't match the remote's hash signature.

Side note: *Never* use `-f` (force push flag) as it will bypass said protection. That should be also be general rule for good practice. Although it doesn't help in the Heroku case, I recommend always enabling branch protection on your remote. <a href="#branch-protection">[2]</a>



## Conclusion

So there you have it. A quick and dirty way to push and pull your way around any remote without all the **commit**ment issues of their named origins taking up precious bytes on your machine.


## Resources

- <a name="git-push-repository">[1]</a>
 [Git push repository documentation](https://git-scm.com/docs/git-push#git-push-ltrepositorygt)
- <a name="branch-protection">[2]</a>
 Protecting Git Branches
 - [Github Branch Protection](https://help.github.com/articles/about-protected-branches/)
 - [Bitbucket Branch Permissions](https://confluence.atlassian.com/bitbucketserver/using-branch-permissions-776639807.html)
 - [Gitlab Protected Branches](https://docs.gitlab.com/ce/user/project/protected_branches.html)

Git push without a named remote

A few weeks ago I decided to take the leap and implement Webpacker into an existing Rails project. Initially this seemed like a great idea and almost too easy thanks to the [rails/webpacker](https://github.com/rails/webpacker) project. After a couple of hours I had everything working and some existing React components migrated from a custom compile system to the asset pipeline. Deploying to staging had a few minor hiccups, but nothing major and the project was ready for sign-off and production release.

For this specific project I'm using Heroku coupled with the pipelines workflow. For those not familiar, pipelines allow you to "promote" a slug almost instantly to production that is identical to staging. Everything worked as expected, assets were being served and development was now much simpler thanks to the Webpack-dev-server and access to tons of ES6 and JSX helpers out of the box.

A few days later I started seeing errors in [Sentry](https://sentry.io) from our frontend, complaining that libraries were unavailable and JavaScript files could not be downloaded from the CDN. After inspection it appeared production assets were being loaded from the staging CDN URL. In hindsight this was pretty obvious, but I want to share the findings so others can either avoid this or find a solution much faster.

Webpacker works based on the concept of a manifest, which writes **absolute** URLs to disk, which Rails then reads to figure out the pack file URL to render. With the asset pipeline, this manifest uses relative file paths and the `config.action_controller.asset_host` variable is used at **run-time** to build the asset URL. However, since Webpacker writes an absolute URL path at **compile-time** this is transferred to production via the pipeline promote workflow.

This wasn't initially obvious, but I really should have understood the Webpacker internals before pushing to production. This also didn't surface until a few days later because the assets were still on the staging CDN. As soon as new code was pushed to staging, and the fingerprinted hashes changed then the production URLs started receiving 404s once the file origin expiration came around.

This all happened in the middle of the night for us, as do all production issues, but the middle of the day for our users who are primarily in a different time zone. In the interest of getting this resolved as fast as possible I decided that bypassing the pipeline workflow all together and digging into the source code of Webpack after some sleep was the best approach. Without the pipeline workflow, pushing to Heroku production directly causes the whole build to run from the beginning which takes much longer, but means the absolute URLs written to disk will be the production URLs rather than staging.

Once this decision was made, I now needed to figure out how to force rebuild the current production servers. One approach is to push an empty commit (`git commit --allow-empty -m "Nothing to see here"`) but I prefer to keep the git history clean. The code should not be coupled to deployments in this way in my opinion. Previously I'd used a plugin called `heroku:repo` which luckily still works, although I'd read in various places it was now deprecated. It turns out it was actually moved from it's original location to the Heroku Github organization, rather than removed.

Before being able to rebuild, I actually needed to reconfigure the project buildpacks to get the latest version of yarn, since builds were failing due to a new version of Webpack in the code. When migrating the initial code, Webpack worked fine with the ruby buildpack alone. Since then it now requires yarn >=v0.25, but the ruby buildpack only has v0.22. There is a pull request open to fix this, but until then the nodejs is the best approach. I also prefer using the nodejs buildpack going forward to protect against future dependency requirements of Webpack. The nodejs buildpack is also way more likely to have the latest version of yarn, including security patches. The most important thing to note is that the ruby buildpack should come after the nodejs buildpack so that the correct version of yarn is available.

``` bash
heroku buildpacks:set heroku/nodejs -a yourapp
heroku buildpacks:set heroku/ruby --index 2 -a yourapp
```

Here are the commands to quickly rebuild the production servers. This can be used on any Heroku deployment, in any stage whether it uses pipelines or not.

``` bash
heroku plugins:install heroku-repo
heroku buildpacks:clear -a yourapp
heroku buildpacks -a yourapp
heroku repo:reset -a yourapp
git push heroku-yourapp master
```

The moral of the story here is always be careful when mixing multiple language build processes that require compilation, and understand as much of the internals of the build process as possible. In this case better knowledge of Webpack manifests would have helped hugely. I'm sure there is a way to configure relative URLs in the manifest, but I'm yet to find it.

Rails + Webpacker Deployments on Heroku

Time and time again I've found myself needing to limit access to S3 repositories via write-only. Read-Only access is widely used for public repositories, such as CDNs. A highly common use case for write-only access is allowing users to upload new files, but not modify any that currently exist.

Here is an example of a common policy I use. It will allow read access to the entire bucket, but allow write-only access to anything within the /blog/* path. This is currently being used by at least 2 clients for uploading new files to their bucket, without being able to accidentally delete or modify any existing images.

``` json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BucketPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::media.marcqualie.com"
            ]
        },
        {
            "Sid": "BaseObjectReadPermissions",
            "Effect": "Allow",
            "Action": [
                "s3:GetObject",
                "s3:GetObjectAcl",
            ],
            "Resource": [
                "arn:aws:s3:::media.marcqualie.com/*"
            ]
        },
        {
            "Sid": "BlogWritePermissions",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::media.marcqualie.com/blog/*"
            ]
        }
    ]
}
```

It's worth pointing out that if you apply this as a *bucket policy* then your bucket will open to the world. This will allow anyone to upload anything to your bucket, which could cause you legal or cost problems. A more sensible option is to add this to an IAM user, which applies these permissions only to that user when they are authenticated.

I've labelled each policy section to be pretty self explanatory, but I'm happy to help if you have any questions in the comments.

Write only S3 permissions

I've lost count of the amount of times I've forgotten the syntax for creating users, databases and permissions within postgresql. After spending so many years working with MySQL it became muscle memory and for some reason I can't shake those commands. Unlike MySQL, which I used to host myself and understand the internals of security/performance, I've only ever used hosted versions of PostgreSQL where all of this is taken care of.

Create a new database and a user to manage it:

``` sql
CREATE DATABASE marcs_stash;
CREATE USER marc WITH PASSWORD 's3cr3t';
GRANT ALL PRIVILEGES ON DATABASE marcs_stash TO marc;
ALTER DATABASE marcs_stash OWNER TO marc;
```

I also find myself creating read-only users for contractors, internal teams and third part analytics tools so I'll share those here as well.

``` sql
CREATE USER spy WITH ENCRYPTED PASSWORD 's3cr3t';
GRANT CONNECT ON DATABASE stuff TO spy;
GRANT USAGE ON SCHEMA public TO spy;
GRANT SELECT ON ALL SEQUENCES IN SCHEMA public TO spy;
GRANT SELECT ON ALL TABLES IN SCHEMA public TO spy;
```

I'll likely be updating this as I go and learn more about PostgreSQL permissions and security. Comments and improvement suggestions are always welcome.

PostgreSQL Create User

Today, I made the mistake of running `gem pristine` while helping someone debug an issue. Not only did this not resolve their issue, but now my ruby projects were now also unusable. By re-installing all of my gems it re-compiled Nokogiri which was now, for some reason, not compiling against the system libraries.

I'd used Nokogiri for weeks on macOS Sierra without an issue. In fact I'd even created new projects with fresh bundles just hours prior to this issue. It appears that Nokogiri (and all other native gems) has actually compiled against the previous version of OS X system libraries and that's what had been in use up until now.

By default, Nokogiri comes with it's own packaged version of libxml2 which appears incompatible with macOS. After googling I was able to figure out the two following commands to get everything back to normal. Change the version of Nokogiri to match the version you want. This docs recommend against this, which is ironic because their recommended version is broken and this one works. Hopefully this will be fixed in the future and this workaround will no longer be needed.

``` bash
brew install libxml2 libxslt
gem install nokogiri -v '1.6.8.1' -- --use-system-libraries --with-xml2-include=/usr/local/opt/libxml2/include/libxml2
```

It's also possible to do this at the Bundler level; Helpful if you have multiple different versions of Nokogiri across your projects.

``` bash
bundle config build.nokogiri --use-system-libraries --with-xml2-include=/usr/local/opt/libxml2/include/libxml2
bundle install
```

I hope this saves someone some time. I know I'll almost certainly need to refer back to this if I do a fresh system install or run pristine again.

Install Nokogiri on macOS Sierra

Due to using to [Github flow](https://guides.github.com/introduction/flow/) quite heavily, I end up with a lot of rogue branches locally that I'm always forgetting to cleanup. The snippet below is a quick one-liner I have aliased to `cleanup-branches` which saves me some time and keeps me out of branch spaghetti when autocompleting.

```
git --no-pager branch -vv | grep gone | awk '{print $1}' | xargs git branch -D
```

Remove Deleted Git Branches

I'm currently building an application which makes heavy use of JSONB in PostgreSQL. After a while of searching I was unable to find any solution to people's questions about distinct JSONB keys across an entire table. Here are my findings and I hope it helps someone else.

For the application I"m building, it's highly critical upfront to know which keys exist in the column in order to customize the UI. Data stored in the JSONB column is completely dynamic and has no schema associated, similar to what you'd expect from MongoDB. Due to this restraint, without such query, the application in unable to know which keys it can query.

This is the simplified query I came up with which gives a unique list of every key stored across the entire table. I tested it across with a few million records and returned in a few miliseconds so it more than serves it's performance needs and it can even be easily cached as a flat string array on billions of records.

``` sql
SELECT
  DISTINCT field
FROM (
  SELECT jsonb_object_keys(column_name) AS field
  FROM table_name
) AS subquery
```

The above query uses a sub statement to unfold the keys as a queryable array, which then uses `DISTINCT` to only bring back unique values from the sub array. Since I'm using Ruby on Rails, I combined it with ActiveRecord in order to give me an array of values to then use in the UI:

``` ruby
result = ActiveRecord::Base.connection.execute(query_string)
fields = result.map { |row| row['field'] }
```

I managed to build this query using documentation from [JSONB Functions and Operations](https://www.postgresql.org/docs/9.4/static/functions-json.html) which is quite in depth.
I hope this helps someone looking for a similar use case as it seems to be a common question among the fairly new JSONB column types. Comment below with any questions or suggestions for improvement.

PostgreSQL: Distinct JSONB Keys

Today, I was preparing a [Docker](https://www.docker.com) container for production release and found managing environment variables was a bit of a pain. As a ruby developer it's natural to me to use [12factor](https://12factor.net/config) methodology for my environment so I wanted to use that with Docker.

**EDIT:** After I figured out my obscene way of doing it, I thought to check the docker helper page and sure enough, support is backed right in. You can simply pass in the location of the file using `docker run --env-file=.env`. That being said, I'm still publishing this to share what I learned about the shell and to serve as a reminder that I should always check the documentation more thoroughly.

Most tutorials described using docker-compose, but I'm deliberately not using that as I only want a single service. I'm also trying to learn as much about native Docker as possible before adding abstractions. After a few iterations I came up with the following shell command which I then put into a Makefile for easy re-use:

``` bash
docker run `<.env xargs -I % echo "-e %"`
```

What that above code is doing is:

- Reading the file **.env** (I previously used `cat .env |`)
- Iterating over each line and pass it to **xargs**
- xargs then outputs **-e {LINE}**

Using backslashes in-line with the Docker run command injects this concatenated string directly as if you were passing them yourself with multiple `-e` arguments.

There you have it, a highly overcomplicated way of passing a **.env** file to a Docker container. Although this isn't needed, I learned a lot about shell piping and concatenation using xargs.

<blockquote>
 Simplicity is prerequisite for reliability.
 <footer>
 <a href="https://en.wikiquote.org/wiki/Edsger_W._Dijkstra" target="_blank">Edsger W. Dijkstra</a>
 </footer>
</blockquote>

Docker Dotenv

I spent a lot my time this weekend looking into a long overdue infrastructure upgrade for all of our hosted GoStatic sites and found some awesome tools a long the way. Something I've wanted to learn for a long while is the lua programming language due to it's small footprint and simplicity.

For the last 18 months, all GoStatic clients have ran from a very simple file system with Nginx in front. While that served it's purpose, it was very inflexible, slow to deploy and hard to distribute globally. I've always envisioned how I wanted the setup to look, but I just never had the time to implement it.

<blockquote>
 OpenResty (aka. ngx_openresty) is a full-fledged web application server by bundling the standard Nginx core, lots of 3rd-party Nginx modules, as well as most of their external dependencies.
 <footer>OpenResty website</footer>
</blockquote>

I was set on the idea of sticking with Nginx; I just needed more flexibility, performance and scalability. OpenResty offers all of this as it comes packaged as an application server with helpful modules like Lua and Redis built right in. It took a few hours but I was able to put together a reverse proxy backed by Amazon S3 and Redis controlled routing. With these changes in place I was able to change the config of any live client site within miliseconds in multiple regional locations.

Here is the complete stack I now have:

- OpenResty (Nginx, Redis + Lua modules) to handle the logic
- Redis for host configuration
- Amazon S3 to store all the actual files
- Amazon EC2 for the application servers
- Tutum/Docker for deployment

Amazon S3 isn't known for it's speed so all of the files are cached at the application layer in RAM. The cache for a client it then instantly invalidated on each deploy by changing the hashed key in Redis. My second reason for this kind of caching is that I can now distribute these servers closer to end users from geographical regions closer to them, regardless of distance from the origin (S3). This gives all GoStatic clients instance CDN capabilities for their entire sites, not just their assets.

Here is a snippet that returns a platform level 404 if the requested domain doesn't route to a client's project:

``` lua
local host_config, err = client:hmget("gonestatic:host:" .. ngx.var.host, "appid", "cache", "build", "redirect")
if host_config[1] == ngx.null then
 ngx.log(ngx.INFO, "PLATFORM 404")
 res = ngx.location.capture("/404-platform.html")
 ngx.header['Content-Type'] = "text/html"
 ngx.header['Content-Length'] = string.len(res.body)
 ngx.status = 404
 ngx.say(res.body)
 return
end
```

This checks Redis (using a Hash data type) to see if the host key maps to a project configuration. If it does not, then a subrequest is made to a custom 404 page and the request is then halted. The routing logic is as follows:

- Host is not configured: Platform level 404 encouraging user to claim the URL
- Host is configured to redirect: Variables such as `$uri` allow things like *https://watson.marcqualie.com* -> *https://marcqualie.com/watson*
- File is found in the local cache: Served out of RAM with custom headers
- File is not found in the local cache: Served from S3 using proxy buffering then cached
- File is not found: A custom 404 page is served from the project (cache or S3)
- Custom 404 is not found: We served a template application specific 404 page

The last peice of the puzzle is master<>slave Redis setup as that's now the only component that isn't automatically distributed. Hopefully I'll have time to implement that next weekend.

I plan to write more about the stack in detail in more specific posts, so do let me know if you have any questions or thoughts about this setup as I've had a lot of fun experimenting with it and learned a lot along the way.

Also, incase you were curious, this site is running entirely on the new setup!

Hosted GoStatic now runs on OpenResty

One of the huge down-sides of having a static site is the in-ability to do search. It's not impossible, but the solutions out there are no where near as good as their dynamic CMS competitors. Searching a static site is hard. GoStatic now supports the ability to search your entire project without having to do any extra configuration on your end.

There are many solutions out there and I'm going to cover each one of them in a bit of detail and explain why GoStatic's solution out weighs them and why it is a great choice for your next search-enabled static project.

## Solution 1: Client Side Search

This is the approach I used to use for this blog before I switched to GoStatic. I love this implementation because it doesn't require any extra build configuration and it's always up to date with your content. While this solution is cheap, easy and fast to implement it does come with a few downsides. First of all, all of your content has to be bundled into a js file which is either included in your main page, or loaded in via XHR when you land on the search page. This does allow for instant search result but adds weight to every page load when there are a lot of posts and the build/deploy process is slowed down as a consequence. The other main downside is this approach is proprietary to whatever static CMS you are using due to the fact it is part of the build process and makes it a pain to switch provider.

## Solution 2: Third Party Provider

Using a third party provider can be a nice solution for a lot of projects because it moves all of the complexity to another service and you never have to worry about maintaining your dependencies or libraries. The reason I didn't use this solution compared to the client side implementation is simply cost. These services normally charge on a per query basis which can become quite expensive if you have a lot of traffic on your site. You are also adding one more dependency and cost to your stack which is the problem GoStatic aims to solve in the first place. The one big advantage I see to this solution is that you are not tied to one single provider as the indexing and storage is handled entirely by the third party, normally via a web scraper that indexes whenever you update your content via a web hook.

## Solution 3: GoStatic

GoStatic combines the best of both of these solutions by allowing you to shift the complexity of the indexing to a third party while keeping the cost benefits of your own client side implementation. If you are already hosting your site on GoStatic then search is included by default as part of your package. You can either use the provided SDK (coming soon) or roll your own using the REST API. Since posts are indexed during site compilation, the index is always up to date with your latest content unlike a lot of providers.

In conclusion I personally think a third party solution is the way to go, whether that be GoStatic or another provider. Search is a core feature of most sites and it's something that has to be fast, accurate and flexible which just isn't possible on a static site when compiled at build time. You can request a free BETA invite to GoStatic in the comments below.

GoStatic Search API Beta

Today, I spent longer than I feel comfortable admitting solving a supposedly simple problem in my EmberJS application. Since this particular issue was so frustrating and undocumented I'm going to share my experiences in case someone else comes across the same hair-pulling nightmare in the future.

### Back Story

The application in question is a private dashboard system which allows clients to create and manage their own business profiles. It's getting close to launch so I decided it was time to spice up the UI with some nice error messages that are helpful to the user. The idea was to highlight the fields that have errors and display a message underneath explaining why there was an error. In Rails (or any other framework) this would be very easy or even built right into the form helper in some cases. In EmberJS? Seemingly straightforward, but it turned into an afternoon of rabbit holes and head-banging against my desk.

### Ember Models

Ember, or rather [Ember Data](https://emberjs.com/api/data/) has error helpers built right into the models. Actually figuring out how to use those helpers is another story altogether. I wasn't able to find any tutorials and the API documentation wasn't clear on what to use. There was also no indication as to what formatting errors should be in. The Rails Application I was developing against output data in the following format:

``` json
{
 "attribute_1": ["error message 1", "error message 2"],
 "attribute_2": ["error_message 2"]
}
```

I'm not overly fond of the layout, but it seems fairly standard in rails applications. Checking my models for errors was bringing back absolutely nothing so it was time to do some digging around the source code. An assumption lead me to search the [Rest Adapter](https://github.com/emberjs/data/blob/bce78d5d5b28fd468b7b4f8346cf51f0326cf54f/packages/ember-data/lib/adapters/rest_adapter.js#L524) file since that's where all the Ajax requests are handled.

``` javascript
App.ApplicationAdapter = DS.RESTAdapter.extend({
 ajaxError: function(jqXHR) {
 var error = this._super(jqXHR);
 if (jqXHR && jqXHR.status === 422) {
 var jsonErrors = Ember.$.parseJSON(jqXHR.responseText)["errors"];
 return new DS.InvalidError(jsonErrors);
 } else {
 return error;
 }
 }
});
```

Sure enough, from the code above you can see that anything in the ```errors``` attribute is returned as a DS.InvalidError. Unfortunately, this code is not part of the adapter by default. It's actually commented out and put there as suggested functionality if you need it. After patching that code into my adapter I then had to update the API output. Luckily this version of the API has never been released to the public yet and our applications that currently use it internally are in private Beta and heavily tested. I added a helper to my Rails application controller which is easily tested and keeps the output consistent.

``` ruby
class ApplicationController < ActionController::Base
 def respond_with_errors(errors, status)
 respond_with { errors: errors }, status: (status || 500)
 end
end
```

A very simple snippet of code. All it does it make sure the errors field is set. This allows us to change it to any other field name in the future to update all errors output in one go. It also defaults to a 500 http status code unless one is passed through. To use the code is very straight forward.

``` ruby
class SomeController < ApplicationController
 def update
 if @model.save
 respond_with @model
 else
 respond_with_errors @model.errors, 422
 end
 end
end
```

### Almost Working

Okay, so now I've figured out the issue and patched the adapter. The API is outputting the correct formatting now so everything should just work as expected. Wrong. You didn't think Ember would be that kind, did you? Here is the below pseudo code that I expected to work after these changes:

``` html
{% raw %}
<div {{bind-attr class=":form-group errors.name:has-error"}}>
 {{input value=name}}
 {#if errors.name}}
 {{#each errors.name}}
 {{error}}
 {{/each}}
 {{/if}}
</div>
{% endraw %}
```

The actual code is a lot more complex, but this demonstrates that I was essentially iterating over each of the errors on the ```name``` field and displaying it inside a label tag. The actual output in the application was ```[Object object]```. The experienced JavaScript developers reading this will recognise this as an object being incorrectly type-casted to a string. I was quite confused because the API had been updated to the format that EmberJS requires to this didn't make any sense.

### Digging Deeper

I started inspecting the Ajax response objects in the chrome console and the output was as I expected. The ```DS.InvalidError``` object that I was returning was in the following format:

``` json
{
 "name": ["can't be blank", "must be unique"]
}
```

This is exactly what I expected. I even triple checked my ```each``` block and it didn't make sense why "can't be blank" was coming out as "[Object object]". I went digging around the source code some more and turns out the DS.InvalidError object is not actually used, and instead is used to create a ```DS.Error``` object which then gets attached to ```Model.errors```. The ```DS.Error``` object is in the following format.

``` json
{
 "name": [
 { "attribute": "name", "message": "can't be blank" },
 { "attribute": "name", "message": "must be unique" }
 ]
}
```

I'm not 100% sure of the design choice behind that object formatting. All I can assume is that each error block is global and assigned to the fields when needed which is why they have the attribute attached. Once I figured this out the fix was quite simple.

``` html
{% raw %}
{{#if errors.name}}
 {{#each errors.name}}
 <label class="label label-error">{{error.message}}</label>
 {{/each}}
{{/if}}
{% endraw %}
```

### Grey Hairs Ahoy!

Finally everything was working as expected. This was originally a quick form to add a new Business profile which should have taken no more than 10-20 minutes of my time. I ended up taking closer to 2 hours which pushed back some other features from getting launched. On the upside, if I ever need to use these kind of model error validations again on other projects I will be able to write them instantly without any worries.

I think I covered all the major points but feel free to comment if you're having the same issue and this doesn't solve it. Also, if you managed to solve this another way without cutting years off your life I'd love to hear about it.

Model Errors in EmberJS

I've had the same dull blog layout for years. Even in my transition from [Blogger](https://marcqualie.com/2012/06/post-200-new-blog) to [Wordpress](https://marcqualie.com/2012/10/no-more-wordpress) and finally [Jekyll](https://marcqualie.com/2014/01/jekyll-is-no-longer-hyde-ing) I've still kept the look similar. While I'm the first to admit that my design skills aren't great, I still thought it would be a fun idea to spent the day making the layout more content focused and less cluttered.

First of all, the black header had to go. Every time I loaded my page it just hit me in the face; Dark colours and too text-oriented. Not only did I change the colours but it now spans the full browser width, contains a search bar, my avatar and colourful social buttons linking to my profiles which are designed to stand out and draw the eye.

Onto the content I wanted it to be more focused and streamlined. I dropped the sidebar and centred the post list with generous white space on either side. When people land on a page they want to be influenced on what to read or click next. By eliminating the sidebar and other distractions they are forced to start reading my posts instantly. The increased font-size and faded out interactions (such as tags) keep the user reading just the content.

As for Typography, I've always stuck to using browser built-in defaults because I'm a software engineer which makes me naturally boring and prefer functionality to aesthetics. After recently being amazed by [Thoughtbot](https://www.thoughtbot.com)'s [Refills](https://thoughtbot.github.io/refills/) I took some inspiration for my new look from them and decided to go with the Titillium Web as the font of choice. Their subtle use of colours and border-radius also helped me style the header.

Finally, the whole thing remains built upon Bootstrap 3. I'm sure I'm not the only one sick of seeing the same old default Bootstrap themes around the Internet so I tried to stray away from the defaults a bit. The best way to do this is to use the Sass version and customize things using the variables and mixins. I learned a lot about Jekyll and various techniques to managing assets which I'm going to share in my next post.

I hope you like the new design and find it easier to enjoy my content. Feedback is always welcome so feel free to comment below. This whole site is also [Open Source](https://github.com/marcqualie/marcqualie.com) if you want to browse around the code and suggest some improvements. On a side note, one of the best things about writing with Jekyll is that as soon as I merge the [design's pull request](https://github.com/marcqualie/marcqualie.com/pull/3) this post and the new design will be live as exactly the same time!

April 2014 Redesign

The community tap I created for custom Nginx modules within Homebrew just got accepted into the official [Homebrew](https://github.com/Homebrew) organization. This is huge news for the project as it means there should be way more usage and contributions thanks to the extra exposure. This is also a good time to share a couple of unannounced updates I added in the last couple of weeks.

Thanks to the new [Travis](https://travis-ci.org/Homebrew/homebrew-nginx) tests we can now detect and prevent errors before they are even merged. There's also a cool new site dedicated to listing all of the available modules and their current versions. None of these updates should impact you as a user, just continue using it and watch out for great new features and updates. Instructions for installing are over at [https://github.com/Homebrew/homebrew-nginx](https://github.com/Homebrew/homebrew-nginx) where you can also fork the code and contribute modules.

Nginx Tap moves to Homebrew Organization

Not only does Jekyll power this blog but it has been one of the main tools I use on a daily basis for almost 2 months now.
Due to it's growing role in our company I was starting to get fed up with having to manually pull down repositories and ask other team members to do the same, especially when all of our ruby projects are managed via [Boxen](https://boxen.github.com). This post is to share my knowledge of how I was able to very easily modify a boxen project to work with an install of Jekyll instead of the default ruby setup.

To start with, you're going to want to navigate to your boxen repo directory which is installed at ```/opt/boxen/repo``` by default.
Once there, create a new project in ```modules/projects/manifests``` called ```jekyll.pp```.
You may want to name this file the same as your project name (especially if you end up having multiple projects) but to keep things simple I will be referring to this file as jekyll.pp to make it easier to follow.
The file contents should be as follows.

``` puppet
class projects::jekyll {
 boxen::project { 'jekyll':
 server_name => 'jekyll.dev',
 source => 'yourname/jekyll-project',
 ruby => '2.0.0-p353',
 nginx => "projects/shared/nginx.jekyll.conf.erb"
 }
}
```

I could have gone the route of building a custom puppet module but this way is way more straight forward and easier to work with.
Each of these settings should be straight forward to anyone who's set up a boxen project before but for those who haven't I will explain what each one does.

setting | description
------------| -----------
server_name | Unique hostname you use to access the site in your browser
source | Location of your repository on Github
ruby | Ruby version you wish to use Jekyll with
nginx | Path to your custom nginx template

Next, create a file in ```modules/projects/templates/shared/nginx.jekyll.conf.erb```.
You will notice it looks very similar to the path you supplied above but they are purposefully different.
I found this out the hard way when my boxen run kept complaining the file was missing.
The file contents should be as follows.

```
server {

 listen 80;
 root <%= @repo_dir %>/_site;
 server_name <%= @server_name %>;

 # Logging
 access_log <%= scope.lookupvar "nginx::config::logdir" %>/<%= @name %>.access.log main;
 access_log <%= scope.lookupvar "nginx::config::logdir" %>/<%= @name %>.error.log main;

 # Variables
 index index.html;
 client_max_body_size 1M;

 # Error Pages
 error_page 500 502 503 504 /50x.html;

 # Proxy
 try_files $uri/index.html $uri =404;

}
```

You can customize this template a bit more to suite your needs using the variables below.

variable | description
----------------------|------------
@repo.dir | Document root of your application code
@server_name | server_name setting you provided earlier
nginx::config::logdir | Internal logging path which nginx is setup to use
@name | Name of your project class above


Now run the ```boxen``` command to install the project.
Once that completes you should compile your site if you haven't already using ```jekyll build```.
You can watch for file changes and auto build using the ```--watch``` flag.
There is more information on [Jekyll usage](https://jekyllrb.com/docs/usage/) in the documentation.
If everything goes correctly there should be no errors and you will be able to access your new compiled boxen site from the vhost you setup earlier (jekyll.dev).
If you have changed the _config.yml file to have a different destination folder then make sure you set the correct root directive in your template instead of _site.

That is all you should need to get a default install of Jekyll running.
If you run into any troubles or have any questions then please contact me in the comments below.

Jekyll with Boxen

After convincing myself all year that I will finally Go Static and get rid of my custom CMS for this blog, I decided to finally launch it on the very last day of the year; Talk about leaving things until the last minute! This isn't a complete coincidence though, as I've been working long hours on a new project which is related to Jekyll development. I'll blog about this separately further into the year. This post is going to cover why I chose Jekyll, and why it's going to be so powerful moving into 2014.

### Why Jekyll over custom CMS?

This is the question on almost everyone's mind when I talk to them about Jekyll. A custom CMS can do almost anything you need it do, so why change? There are hundreds of them actively developed by the open source community which brings tons of great software to the market. This is an awesome thing, however the whole work-flow gets on my nerves for many reasons. For my use case, Jekyll has solved almost all of these annoyances and the ones it didn't solve I created custom plug-ins for.

### Almost too easy to set up

For those of you who have set-up a CMS (e.g Wordpress) you know it can be difficult to both initially set up and to maintain for a client. Setting up a secure database, installing a cache, plug-ins, version upgrades, security patches and theme dependencies are just some of the things you have to battle with when maintaining a traditional CMS. What happens when your new theme doesn't work with one of the plug-ins that are running but an alternative plug-in doesn't work with another plug-in that is installed? Imagine if all of that was gone. Since Jekyll is entirely flat file based there is no database or cache needed to get high performance. There is no dynamic scripting language so it's virtually impossible to get hacked via your web server. Setting up Jekyll takes a matter of minutes rather than hours or even days in some cases.

### No backup strategy needed

If you are using version control (if not, why not?) then you already have a backup system in place with easy rollback functionality. You no longer have to worry about paying for storage elsewhere or make sure that crons run to backup the data. You also know that your data will be 100% up-to-date, unlike a system where you backups could be as much as 24 hours behind in a major outage where your main database is corrupted. This is a huge win for both always consistent content and ease of maintainability. Your content and your site scaffolding are also backed up in the same place which is not the case with most other CMSs.

### Cheaper to maintain

Since there are no complex run-times or scripting languages to run, the compiled site can be run out of almost any standard HTTP web server. I am currently using Nginx due to it's great scalability and high performance, but it's just as easy with Apache or other alternative. A site which would maybe hold a couple of hundred concurrent users with PHP could now easily scale to 10s of thousands as long as there is enough bandwidth and RAM on the machine. You can even serve your site directly out of Amazon S3 where you only pay for the storage and bandwidth used instead of keeping a web server running.

### CDN All The Things

Why serve just your assets out of a CDN when your entire site could be served from Edge cache? Imagine every single request for a page on your domain coming from a server in the same country as your visitors. This is something that is simply not possible without the site being 100% static or having stale data that has to be updated. Not only does this speed things up but it also another way to not have to worry about hosting management. The CDN is responsible for serving content to your users from data centers around the globe which means your site availability will greatly increase and your site will feel almost instantaneous.

### Personal Editing Interface

I'm currently using a mixture custom fork of [Prose](https://prose.io) and Sublime Text 3 to write this post, but nothing is stopping you using any editor of your choice. Since all files are plain text markdown (by default, the engine can be changed) then you can use your preferred editor of choice on your own machine. If you need to edit on the go to keep the site updated then go for an online editor such as Prose, Cloud9 or various other alternatives. If you're a little code-savvy you could even code up something nice for yourself and host it somewhere like Heroku to have a fully custom on-line editing experience.

### Drawbacks

Jekyll isn't for everyone. It's mainly aimed at developers due to the fact you are editing within the project code directory and there's no nice control panel like you get with Wordpress. For most developers this will be fine, but for non-techies this can be a huge drawback. Since everything has to be pre-compiled, there's no way to pull in data from a database in real-time like you might be used to. If you change any content you will have to re-deploy to your server, which can turn out to be more hassle than simply editing via a control panel. There are ways around this such as integrating with [Github Pages](https://pages.github.com) or [Go Static](https://gostatic.io).

### Conclusion

Overall Jekyll is a great platform for managing your content but it's still maturing. The project I'm working on will be aimed at bridging the gap between flat file version controlled markdown with a beautiful and easy to use control panel which requires no technical knowledge to set up and use. Use the comments below if you have any questions or thoughts around Jekyll or Static Site Managing in general. I'd love to get your feedback!

Jekyll is no longer Hyde-ing

As some of you may know I recently left my position at Performance Horizon to take on a Senior Developer role with <a href="https://www.givey.com" target="_blank">Givey</a>. This is a quick post I'm writing to clear up the many questions people have had on why I chose to move and what my first week was like at this growing North East based start up.

Over the past year with PHG my skill set has largely grown, but also diverged quite a lot from what I was previously doing. When I joined the company I had quite a lot of experience with full stack development and optimisation, ranging from linux server administration to making website animations smoother when transitioning pages. Givey is doing something amazing with their product and hopefully changing the world which is why this position was so appealing. On top of that I would also get to go back into the world of full stack and use my skills to the full to drive them forward in their mission.

Now that you have a bit of background on why I joined, I'd love to share my experience on what it's actually been like for the first five days. One thing I noticed is the fast pace of an early startup. There's constant code being merged and shipped, lively discussions and real-time feedback on what's happening around the office; Both in person and on-line via the many tools that are integrated into our workflow. This works so well because of the very small but highly organised and family-like atmosphere.

After just one week I already have lots of code waiting to be deployed into production at the end of the sprint thanks to <a href="https://github.com/blog/1557-github-flow-in-the-browser" target="_blank">Github Flow</a>. I was also able to design, build and test a new database infrastructure which will be integrated at the end of this sprint also which should bring huge performance enhancements coupled with various code changes.

One of the biggest challenges I found was going from a PHP-driven history into the scary world of Ruby on Rails. The code itself is quite straight forward, it's the workflow, environment and culture around the tools that fascinate me. There seems to be a gem for everything and Gemfiles stacked to the brim with goodies. Testing is so elegant thanks to the story-like structure and how easy it easy to analyse and debug complex issues in a codebase that spans various repositories.

Lastly, my morning routine and working location have largely changed. Instead of a short drive to a secluded office building I now work in the <a href="https://www.ignite.io/" target="_blank">Ignite Loft</a> which is a city centre based tech incubator. The place feels alive and I've found I actually work better with sounds of the city and sirens blazing past the window every 5 minutes. I'm also lucky enough to share the early morning commute with the lovely metro-dwellers of Tyne and Wear, which is always a fun experience.

All in all this week has been busy, exciting and enlightening. I'm really looking forward to my continued work here and contributions to an amazing product that will change the world in years to come. On a final note, I've made it my new mission in life to beat <a href="https://twitter.com/pootsbook" target="_blank">@pootsbook</a> at Ping Pong before the end of the year which will mean a few more hours on the office ping pong table!

If you're feeling generous, head over to my page on Givey and see what's it's all about: <a href="https://www.givey.com/marcqualie" target="_blank">givey.com/marcqualie</a>

My First Week at Givey

Lately there has been a lot of buzz around security when using web services. Everyone seems to be very paranoid all of a sudden and encryption is the answer to everything internet based. I wouldn't say I'm too overly paranoid, but I took this opportunity to add some extra security to my personal website. This post will cover the things I learned along the way through my research and also explain how quick and easy it can be to add SSL Encryption to your website.


#### Why use SSL?

Some people may argue that you don't need SSL encryption on your personal website, but I beg to differ. My personal website has logins, client details and API access. All of these things contain important information which I don't want people to be able to <a href="https://en.wikipedia.org/wiki/Packet_analyzer" target="_blank">sniff</a>. I login to my site from various shared networks that I don't control, such as WiFi access points, which poses an obvious threat. The other not so obvious threat is your workplace. 9 times out of 10 your company will have network monitoring software which a human has access to. I don't like the idea of someone I work with (or anyone for that matter) seeing my passwords and personal data I store on my server transferred across the wire. All of the above issues can be very easily avoided by adding a SSL Certificate to your server.


#### Obtaining a Certificate

You can get certificates cheap or even <a href="https://www.startssl.com" target="_blank">free</a> in some cases. I use PossitiveSSL which are fairly cheap from my hosting provider, Namecheap. Each provider has their own instructions on how to setup and install them so I'm not going to go into detail here on how to actually generate your server keys and get your key from the provider. I know almost all providers will have very detailed instructions on how the process works. Once you have your certificate, make sure you keep it safe. SSL only protects from man-in-the-middle attacks if you keep your key file private. If an attacker was able to somehow get your key, they could impersonate your server (unless you buy a high end certificate which is very expensive) and your users would still see the green https icon in their browser.


#### HSTS

Some of you may have come across <a href="https://en.wikipedia.org/wiki/HTTP_Strict_Transport_Security" target="_blank">HTTP Strict Transport Security</a> which I currently use on this site. There are many arguments against this, and also many for. I'd like share my reasons for using it on marcqualie.com.


#### Consistency

My main reason is it keeps things consistent. Previously I had a lot of business logic within the site code which controls how links are generated and whether they should be SSL or not. There was also logic which checked if the page required SSL (such as logins and admin) and redirected to a secure version of the page if need be. With global SSL I can simply do a single check at the web server level and the application can just assume SSL is enabled.


#### HTTPS Only Cookies

For my logins I require https only cookies. This protects against packet sniffers accessing the cookies as they are transferred over the wire and attempting <a href="https://en.wikipedia.org/wiki/Man-in-the-middle_attack" target="_blank">man-in-the-middle attacks</a> which have happened to <a href="https://codebutler.com/firesheep/" target="_blank">Facebook and Twitter</a> before. The problems this caused was that visiting the site from a non-https connection would cause the application to always assume the user to think they were logged out, which was annoying for certain pages which change their content based on whether a user has an authenticated session.


#### Less Redirection Overhead

Simply forcing HTTPS at the web server level isn't simply enough. Using the HSTS header means search engines know to always link to the secure version and once a user has been on the site their browser won't ever try to connect to the non encrypted version. This is a tiny amount of overhead, but causes a nicer user experience to a frequent user.


#### Caveats

Of course, there are downsides. The huge one is that all static assets much be served from a secure url, which means they can't be cached. For a media site this could be a big issue, and it also causes a lot of extra bandwidth to be used. Luckily for me, I don't have much media and I have unlimited bandwidth thanks to <a href="https://www.digitalocean.com/?refcode=4f6e23440060" target="_blank">Digital Ocean</a>'s grandfather account. Another downside is that your server will see increased CPU usage. If you have a high traffic site you may need to increase your server specifications or expand to more servers. It's worth noting that even in Nginx (which is known for it's non blocking nature) will block a worker process when handling SSL handshakes causing a backlog if you get hit by a lot of traffic in one go on a single server. I've never seen this cause issues myself, but it's something to watch out for if you have a lot of concurrent connections.


#### Conclusion

Always on SSL isn't for everyone. I may even find that it's not the right use case for me in a few months, but at least I will have tried it. I always encourage improved security, but make sure you're doing it for the right reasons and understand the downsides as well as the benefits. I'm happy to answer any questions about my experiences in the comments, but I'm also interested in hearing what you guys have to say about SSL and why you chose to use it, or not use it on your own applications.

SSL All The Things

I've been asked a few times how I back up my servers at [Digital Ocean](https://www.digitalocean.com/?refcode=4f6e23440060). It seems this topic is quite popular due to the fact they just started charging for automated backups on the 1st of July. In this article I'm going to go through the process of using [s3cmd](s3tools.org/s3cmd) with [Amazon S3](https://aws.amazon.com/s3/) to easily backup and restore your servers. Although I use Digital Ocean as my own hosting company, and this is where this backup system is in place, it can just as easily be transferred to any hosting provider.

First of all, you're going to want to install s3cmd which is part of s3tools. This is a simple command line utility that takes away all of the complexity around transferring files to S3. All of these instructions below are going to be targeted at an Ubuntu based operating system. The only thing that mainly differs is the installation of the s3cmd tool. Once it's installed, all commands are the same. Run the following code to install the utility.

``` bash
wget -O- -q https://s3tools.org/repo/deb-all/stable/s3tools.key | sudo apt-key add -
wget -O/etc/apt/sources.list.d/s3tools.list https://s3tools.org/repo/deb-all/stable/s3tools.list
apt-get update
apt-get install -y  s3cmd
```

You will need a bucket to store you data in S3. If you don't already have an account you will need to create one. Visit the [AWS Console](https://console.aws.amazon.com/s3/home?region=us-east-1) and create a new bucket. This name must be unique to you. Selecting a location is permanent and cannot be changed after creation to it's best to pick a location close to your server will reduce transfer times.

Now that the tool is installed and a bucket is setup, you will want to configure it with your AWS credentials. I recommend using Amazon AMI, but that is out of the scope of this tutorial. You will require valid AWS security credentials with permission to read and modify contents of S3 buckets. Run the following command and follow the instructions.

``` bash
s3cmd --configure
```

Once everything is configured and the test checks have passed you will want to create a backup script to actually send your data to S3. There are many different ways of doing this and you might even have your own scripts to use. In general I try to grab all of my data into once place then sync that directory. Below I'm going to show you a couple of basic scripts to backup your web directory and MySQL. The following code should go in `~/backup/mysql`.

``` python
#!/usr/bin/env python
import os
import time
username = 'backup'
password = 'password'
hostname = 'localhost'
filestamp = time.strftime('%Y%m%d')
database_list_command = "mysql -u%s -p%s -h%s --silent -N -e 'show databases'" % (username, password, hostname)
for database in os.popen(database_list_command).readlines():
    database = database.strip()
    if database == 'information_schema' or database == 'performance_schema' or database == 'mysql':
        continue
    filename = "/backup/mysql/%s-%s.sql" % (database, filestamp)
    print "Backing up %s" % filename
    os.popen("mysqldump -u%s -p%s -h%s -e --opt -c %s | gzip -c -9 > %s.gz" % (username, password, hostname, database, filename))
    print ".. done"
```

All that does is grab a list of all your databases, dump them, then gzip then into a the ```/backup/mysql``` directory. I recommend using a /backup directory and splitting it into categories such as "www" and "mysql". This makes syncing back to S3 very easy, which make more sense in the final section.

Now that all of your MySQL data is backed up and compressed locally, we want to do the same for your source code and web applications. I'm going to assume that your web applications are installed at ```/var/www```. You may or may not have multiple websites hosted here, this tutorial will work regardless as you're going to be backing up the entire directory. The following script will compress that entire directory into one tar.gz file which saves on space and allows point-in-time backups. You should put the following code in ```~/backup/www```.

``` bash
#!/bin/sh
tar -czvf /backup/www/`date +%Y%m%d`.tar.gz /var/www
```

This will store the entire directory in one tar.gz file which will be your single day snapshot. Now that you have your mysql and www data in /backup it's time to write the s3cmd part of the system. Put the following code into ```~/backup/s3```.

``` bash
#!/bin/sh
s3cmd sync /backup/ s3://yourbucket/`hostname`/
```

You should replace ```yourbucket``` with the bucket you created earlier. You will notice I inserted hostname in there; This is incase you want to backup multiple machines to the same bucket for easier management. I currently do this for clients so they're all in one place with one set of credentials so it's easy to manage and everything is in one central place with organisation.

All parts of the backup system are now in place, you simply need to make them run automatically. Before we can do this you will need to run the 3 following commands to get your system ready. Nothing advanced, they simple create the directories where the backups will be stored.

``` bash
chmod +x ~/backup/*
mkdir -p /backup/mysql
mkdir -p /backup/www
```

Putting all of the parts together is quite simple. Create a script at ```~/backup/run``` with the following content.

``` bash
#!/bin/sh
~/backup/mysql
~/backup/www
~/backup/s3
```

That script now allows us to setup a daily cron to automatically backup mysql, then your web directory, then sync everything back to Amazon S3. To do that will will want to run ```crontab -e``` and insert the following code.

``` bash
MAILTO=cron@yourserver.com
0 0 * * * user /home/user/backup/run
```

You should replace ```user``` with the username that should run the backups (also where the user where you stored the backup scripts in ~). Once that is done everything is now ready and your system has fully automated backups to Amazon S3! You can give it a quick test by running:

``` bash
~/backup/run
```

You should see the output in your console and the files appear in your s3 bucket. Any errors that happens along the way should be printed out to the console for you to debug, and if they were to happen during execution they will also be emailed to your as part of the cron mailer.

To restore content you can simple download the tar.gz files from Amazon S3 onto the server, or a new server, extract them and move them to the correct place. In case of MySQL you simply run the mysql importer on the extracted .sql files to get your system back to the point the backup was created. My next article will be on advanced restoration topics and how to build a fresh server automatically from a backup in S3. Please comment if you have any feedback or questions.

All of the code for the above scripts can be found at the following [Gist](https://gist.github.com/marcqualie/6099324).

Server Backups with s3cmd

I've recently been working on an <a href="https://github.com/marcqualie/homebrew-nginx" target="_blank">Nginx Tap</a> for Mac OS X Homebrew and now I think it's time to share with the world. Originally I created a pull request on the original repo but due to the custom nature of the modules I needed it was reccomended that I create a tap and maintain my own repository (tap). The main goal behind the tap is to create a version of Nginx with as many modules as possible so people can create fully custom builds very easily.

The main advantage of using this tap over the native one inside homebrew is the power of the extra modules. For some reason they don't want to accept custom modules very often so when you're developing custom builds of Nginx you can't really use homebrew. My tap fixes this by accepting pull requests for any module. I've modified the original formula quite a lot to make submitting extra modules extremely easy by just editing a list at the top of the file.

Currently I've covered every non-default module and I'm now working on powering through the list of 3rd party modules that are popular and add real power to Nginx. Currently a couple of my favorites are <a href="https://github.com/chaoslawful/lua-nginx-module" target="_blank">Luajit</a> and <a href="https://wiki.nginx.org/HttpSecureLinkModule" target="_blank">Secure Link</a>.

``` bash
brew tap marcqualie/nginx
brew unlink nginx
brew install nginx-full --with-lua-module --with-secure-link --with-spdy
```

You can try out the tap in seconds by running the code above. I'm currently looking for feedback on which modules you think I should add and how well the current ones are implemented. Pull requests for new modules and documentation is also greatly appreciated.

Homebrew Nginx Tap

For those of you that follow my blog reguarly, a few weeks ago I announced I was undertaking my <a href="/2013/05/open-source-challenge" target="_blank">Open Source Challenge</a>. Today marks the 32nd day of consecutive code pushes, so I successfully completed my goal! I've attached a list of the projects I contributed to for you guys to check out.


#### Community Projects

- <a href="https://github.com/CHH/heroku-buildpack-php" target="_blank">CHH/heroku-buildpack-php</a>
- <a href="https://github.com/mxcl/homebrew" target="_blank">mxcl/homebrew</a>
- <a href="https://github.com/josegonzalez/homebrew-php" target="_blank">josegonzalez/homebrew-php</a>
- <a href="https://github.com/fish-shell/fish-shell" target="_blank">fish-shell/fish-shell</a>
- <a href="https://github.com/ppi/framework" target="_blank">ppi/framework</a>
- <a href="https://github.com/disqus/python-phabricator" target="_blank">disqus/python-phabricator</a>
- <a href="https://github.com/scottymeuk/installer" target="_blank">scottymeuk/installer</a>

#### Personal Projects

- <a href="https://github.com/marcqualie/hoard" target="_blank">marcqualie/hoard</a>
- <a href="https://github.com/marcqualie/hoard-php-client" target="_blank">marcqualie/hoard-php-client</a>
- <a href="https://github.com/marcqualie/php-statsd" target="_blank">marcqualie/php-statsd</a>
- <a href="https://github.com/marcqualie/homebrew-nginx" target="_blank">marcqualie/homebrew-nginx</a>
- <a href="https://github.com/marcqualie/bashdown" target="_blank">marcqualie/bashdown</a>
- <a href="https://github.com/marcqualie/mongominify" target="_blank">marcqualie/mongominify</a>
- <a href="https://github.com/marcqualie/hiptask" target="_blank">marcqualie/hiptask</a>
- <a href="https://github.com/marcqualie/pimple-aware-controller" target="_blank">marcqualie/pimple-aware-controller</a>


I didn't contribute to as many community projects as I'd hoped, but I tried to make up for it with lot of work on my own which I've had great feedback on. I've decided to try to continue the streak for another 32 days as I've still got a few projects I want to contribute too, and it's actually quite a lot of fun writing this much code.

Feel free to add comments suggesting which projects I should work on next, or feedback on my own projects.

Full Progress: <a href="https://github.com/marcqualie?tab=contributions&period=monthly" target="_blank">https://github.com/marcqualie?tab=contributions&period=monthly</a>

Open Source Challenge Complete!

Today, I just created a <a href="https://github.com/etsy/statsd/" target="_blank">StatsD</a> PHP Library and released it on <a href="https://github.com/marcqualie/php-statsd" target="_blank">Github </a>. Before you start with "but there are already loads..", I do realise this, but they didn't suite my needs. This is a short post to let you know why I opted to create my own, and what I hope to achieve from it.

First of all, I was torn between two packages. The first one, <a href="https://github.com/domnikl/statsd-php" target="_blank">domnikl/statsd-php</a>, looked very feature-full, but felt way too complex to do such a simple task. For the ones of you that are familiar with StatsD, the concept is meant to be simple and efficient. I didn't quite like the idea of adding a huge library and bootstrap code to my projects just to send out metrics.

The second library, <a href="https://github.com/till/php-statsd" target="_blank">till/php-statsd</a> is actually the one I am currently using, along with a few other friends and collegues. This is my library of choice due to it's simplicity in bootstrapping and efficiency. However, it didn't quite fit my needs on a bigger project and I'm not too fond of static only based programming. The main let-down of this library is the fact it can't support multiple servers within the same project (not without re-initialising before each call), so I simply can't use it for projects where I have multiple StatsD servers.

After evaluating these two options (and many others) I decided that creating my own would be the best option. I was heavily inspired by Till's implementation and took a lot of the same structure, but implemented it in a more object oriented fashion. I am still toying with the idea of making it backwards compatible, but chances are it'll be easier to just replace the old library and calls across my projects as I come to them. I also went with an optional factory-based initialisation system to allow sharing multiple instances with dependency injection.

I would love your feedback on what you think of this and if you chose to use it, how wel lit compares to other libraries you've used in the past.

Project Source: <a href="https://github.com/marcqualie/php-statsd" target="_blank">https://github.com/marcqualie/php-statsd</a>

StatsD PHP Library

It's bank Holiday Monday and I think it's about time for a new challenge. I've been working a lot with Open Source lately, both contributing to other people's projects and refactoring/improving my own. My current contribution streak on Github is 8 consecutive days of published code, which is pretty awesome considering my previous best was 2 days from about 6 months ago. Not only is this making me a better programmer but it's also extremely fun. With this in mind I've decided the new challenge is to try and create a 32 day streak.

To many of you that may not seem a lot, but think about it a bit closer. Every single day for the next 24 days I have to push at least 1 commit publicly. When I was thinking about the details of this challenge over breakfast I decided the hardest part was finding projects to contribute to and what kind of contributions I could actually make. Another daunting factor is the fact I've just bought a house, I work full time and have social commitments. Fitting all of this together and still writing code for fun is going to be hard work, but worth it in the end I think.

What do I hope to get out of this? Like most things in life, people are driven by what they can benefit from and this challenge is no different. However, unlike most of the common reasons people do crazy projects or work silly hours the goal here is not to make money or have any kind of benefit other than self improvement. Hopefully by the end of this I'll be able to patch, contribute and create pull requests in my sleep. My greatest advantage here is that I'm quite quite young and I have the opportunity to put the time in and learn all of these skills now before I get too many real life commitments.

I think the only way I'm going to complete this challenge, or even come close is to plan out at least 2 days in advance what I'm going to work on and do the harder projects first, with an option to fall back on easier projects if time is an issue or unplanned events happen. This is your chance to help me out. Do you know any open source projects which would benefit from patches or enhancements? Your own work? A library you use daily? I'm open to all suggestions. I'll be creating a second post later in the week detailing the personal projects I'll be working on, some of them I think you'll find really interesting!

You can track my progress and check out the projects I'm working on at the following link: <a href="https://github.com/marcqualie" target="_blank">github.com/marcqualie</a>


#### Project List so Far

- <a href="https://marcqualie.com/marcqualie/hoard" target="_blank">marcqualie.com/marcqualie/hoard</a>
- <a href="https://marcqualie.com/marcqualie/mongominify" target="_blank">marcqualie.com/marcqualie/mongominify</a>
- <a href="https://github.com/marcqualie/homebrew-nginx" target="_blank">github.com/marcqualie/homebrew-nginx</a>
- <a href="https://github.com/manishval/pearl" target="_blank">github.com/manishval/pearl</a>
- <a href="https://github.com/marcqualie/hiptask" target="_blank">github.com/marcqualie/hiptask</a>
- <a href="https://github.com/scottymeuk/installer" target="_blank">github.com/scottymeuk/installer</a>

Open Source Challenge

In the last few years I've used a lot of hosting providers. Until now I've never actually found any that have been able to satisfy my needs at the right price. Sure, there are huge companies such as AWS, Rackspace and Softlayer to name a few, but who has the budget to use these kinds of providers for research or personal projects outside of a big enterprise company. Previously I used Linode. I'm going to let you guys know why I switched and why <a href="https://www.digitalocean.com/?refcode=4f6e23440060" target="_blank">Digital Ocean</a> is my host of choice right now.



#### Overview

If you're reading this article then you've most likely come from a search engine because you've heard about this cool new hosting company and you're looking for reasons to use Digital Ocean over what you currently have. I'm hoping you've come to the right place. If you're not familiar with VPS hosting and using Linux via only the command line then Digital Ocean probably isn't the right hosting provider for you. However, if you're keen to learn and really don't mind breaking things a lot then there is a huge <a href="https://www.digitalocean.com/community" target="_blank" title="Digital Ocean Community">community</a> to help you get started if you run into trouble. Terminology is slightly different for what most VPS providers would call a "node" or "instance". These are referred to as "droplets", so when I mention them later you will know what I mean.


#### Price

Obviously, price is the first thing that draws people to Digital Ocean. Who can complain with a $5 VPS with SSD and instant deployment? I'll be honest I was sceptical even until fairly recently. If cheap is what you want then why not go to shared hosting to get unlimited resources? When I first joined Digital Ocean in January this was my initial impression. Thankfully, in this case, cheap does not mean low quality.


#### Network transfer

The main problem with most VPS providers is that they don't offer much in terms of bandwidth allowance. For most people this is great, but when you're running media sites or your own custom built CDN at low cost then bandwidth starts to become an issue. When I switched from Linode they had just launched their network upgrades which increased their monthly bandwidth allowance by 10 fold; increasing from 200MB to 2TB!. This was a pretty impressive upgrade and a hard deal to turn down. Digital Ocean offer 1TB per node, even on the cheapest plan. I was lucky enough to have an account before everyone was Grandfathered, so I get truly unlimited bandwidth on my account. Really, unlimited, I contacted them just to make sure.


#### Stability

I've been using these servers for around 5 months now and not had any issues. I'm currently running a 5 server MongoDB cluster, 2 web servers and a generic Minecraft/IRC server for messing about. The servers which get the most load are the MongoDB servers. They currently power a few of my applications, one of which is Hoard (with millions of tracked events over a lot of buckets) and another is a server monitoring tool I built which monitors around 60 services every minute for various metrics. Safe to say this intense work would cripple a lot of VPSs, especially single core machines with 512mb ram, but Digital Ocean just keeps on swimming.


#### Hardware

As I mentioned earlier, Digital Ocean are one of the only VPS hosts to offer SSD based storage (at very low prices aswel). Not only that but RAM is much cheaper in comparison to other hosts and can be upgraded/downgraded at any time during the lifecycle of your VPS. You get a lot of storage space for your money and the CPUs aren't over-crowded by overselling. All in all, these machines pack a pretty big punch for your money.


#### API
Almost every hosting provider I've used has had an API. Maybe not the most functional, responsive or even nicely written API but it exists. What I've found with Digital Ocean is that their API is awesome. It's simple. It's functional. Most importantly, you can do anything that would be able to do through the console, which is great for integrating into deployment and build tools. I've contributed to an open source wrapper for the API and I've also fully integrated it into a custom build system which deploys a new droplet (55 seconds or less), configures it and syncs a codebase. If there are problems with the machine, simply launch a new one. Within minutes I can have lots of new servers up and running without ever logging into their console.


#### Conclusion

After using Digital Ocean as my main hosting provider for 5 months I would definitely recommend them to any friends and developers who are looking for VPS hosting. They're cheap, reliable and the IRC channel is always filled with helpful staff and community members. Let me know if you have any questions or think I may have missed out some important information to help you decide.



Referral Link: <a href="https://www.digitalocean.com/?refcode=4f6e23440060" target="_blank">digitalocean.com/?refcode=4f6e23440060</a>

Why I use Digital Ocean Hosting

For a while now I've been hearing a lot of people taking about the programming language Ruby. Not that it's anything new, it just seems to be a lot more popular in my circle of friends and my Twitter feed. Naturally, as a professional PHP developer, I should hate the idea of Ruby and start some crazy argument about how PHP is far superior and Ruby should die (blah blah etc.. boring). Instead of this approach, I took it upon myself to learn Ruby from the ground up and understand why so many people enjoy using it to power their applications. Too many people argue over things without understanding both points of view, which always annoys me.


#### Really, learn all of Ruby?

Ruby as a language is obviously quite a large codebase due to it's maturity and years of existence but my goal here isn't to learn Ruby to the same standard of PHP which I have 7+ years of commercial experience with. Instead, I simply want to understand the language syntax, quirks and helpers that seem to make developers very happy and churn out apps like there's no tomorrow. I started by using a site called <a href="https://www.codecademy.com/marcqualie" target="_blank">Code Academy</a> which I used to think was for noobs but it's actually really helpful and I managed to do the entire course and have a published application in less than 24 hours. That an assumption I made without actually trying out the service. Never assume.


#### Comparisons

I'm not about to start another PHP vs Ruby war, this is simply me pointing out the most obvious differences in the languages and what my thoughts are. Firstly, everything is an object. It's majorly consistent and for that reason I think that's why it's so easy to pick up. Even though I hadn't read a lot of the course material I was still able to do most of the exercises by simply guessing the syntax or method names since a lot of it is common sense. Although PHP technically has Hashes and Arrays (both called just Arrays), they are more obvious in Ruby and you get a slight performance boost by using "Symbols" as Hash keys. There also seems to be a huge obsession with trying to get all code on one line in the Ruby community, especially with breaks, logical statements and raising errors. The more I write, the more natural it feels. There are many many more but that's not what this post is meant to focus on (maybe a separate post?) so I will let you explore the details more yourself if you're interested. The main point I would like to emphasise is that Ruby is easy. Almost too easy in a lot of cases. By that I mean there is about 3 or 4 ways to do almost every task within Ruby which is either a really good or a really bad thing.


#### Applying The Knowledge

What's the point in learning all of this new tech and not putting it to a good use? After I learned Ruby I decided to start an application from scratch to test out my newly acquired skills. Since Ruby is known as the language of Hipsters and I am a huge fan of <a href="http://todotxt.com/" target="_blank" title="Todo.txt">todo.txt</a> I decided to build a similar tool for the command line called <a href="https://github.com/marcqualie/hiptask">Hiptask</a>. I learned a lot of skills that weren't covered in Code Academy such as Unit Testing, dependency management, exception handling and publishing my own gem. Try it out and let me know what you think as I plan to continue development and use this myself on a daily basis.


#### Conclusion

Would I recommend Ruby? Hell yeah. Would I replace PHP with it? Not in the foreseeable future. Just because it's a great programming language and it's possible to write applications extremely fast it still can't replace a language I've been working with for best part of a decade. This is 100% my own personal opinion which may or may not change over time. If you are going to take anything away from this post then let it be this. Don't be afraid to learn new languages, techniques and skills. This is how some of the best code is born and how people innovate, develop as programmers and people. So what language should be next on my list?

Why I learned Ruby

Today, I decided to explore Cloudflare features in a bit more depth. I currently manage quite a few applications, both personal and business related on the platform. One feature which I'd never really looked into before was Page Rules. I'd read about them, but never actually applied them to any sites. Due to how much of an affect it actually had by applying a few rules, I feel like I should spread the word for anyone who isn't aware of what they are and how amazing they can be when used correctly.

First off, I'd like to explain what pushed me to look into page rules more. I had a job come up where someone wanted me to improve their site performance. Their traffic was increasing and couldn't scale to keep on par with demand. The main reason for this was due to the fact they were running from a single server and serving large amounts of media from it aswel as the main website. I actually did a lot of backend system tuning before turning to Cloudflare, which I will write as a series of posts later in the year as it's quite an interesting case study.

You may be asking, if I had already tuned the system then why would Cloudflare be needed at all? The answer is simple. It's not. I'm not going to be passing Cloudflare off a silver bullet that will solve all of your problems. If you are you going to rely entirely on a 3rd party for your site to be performant then you're doing something wrong. In this case, the page rules are simply going to offer enhancement and costs savings for very little work.


#### What are Page Rules?

You can think of them as basically rules that Cloudflare will follow when it matches a pattern you specify. There are so many rules such as caching, redirection, minification, optimizations, Always On™ and many more. The huge advtange here is they are outside of your application code and can be modified, changed and removed at any time. They can also be applied to a specific URL, or a bunch of paths that match a pattern. They're also not limited to web pages, you can apply rules to static files or any other resource your backend can serve.


#### What did I use them for?

In my particular use case, I had already setup Cloudflare to act as a CDN and serve static content from various edge servers so the load was reduced a due to the fact the server was serving less traffic. What I now needed to do was find a way to make the actual application load better and not always hit the servers if possible. After optimising the code to use more caching, and splitting it across multiple machines the load still wasn't going down without having a few powerful machines running the site. Luckily, this particular application doesn't use any authentication and all users get the same content, so caching the entire page as one html block is feasible. I then implemented nginx with memcached backstore to cache the returned data which works great and reduces the load by extreme amounts compared to PHP rendering. But, I wanted more control over the content, and to hopefully get the request off the server entirely. This is where page rules come in. I started applying rules to match the most heavily loaded pages to serve the cached html page out to clients.


#### Why do Page Rules work so well?

Cloudflare is essentially doing the same thing I was doing with Nginx, and runs on the same software under the hood, albeit a lot more customised and advanced. So why did I get such amazing performance out of Cloudflare compared to my custom system? Technically, in terms of server load, I didn't. Nginx is very fast and powerful and after implementing the html caching with memcached I could have happily stopped optimising there and everything would be fine. However, remember earlier when I mentioned that all of the static content was being served out of a CDN? The beauty of Cloudflare is they have a global CDN which can cache any content, including HTML! By implementing Page Rules I can basically serve out any page of the application directly from a global CDN without any code changes or the application even having to be aware that it's happening. Obvious speed improvements for users due to the fact they get the site from the closest server, and now these requests don't actually go through to the application servers, so in the end I was able to keep the original server and not need to scale horizontally, saving lots of money.


#### A few things to consider

You may be thinking this is the perfect solution, and rush out to go and start caching your sites and serving them out of Cloudflare right this second but there are a few downsides that you should consider first. Above, I mentioned that this application was auth-less; If it required user logins then unfortunately any pages which were unique to a user could not make use of Page Rules. Secondly, server side access logs are affected due to the fact the requests aren't going through to the servers anymore. A lot of site owners only use Google Analytics to track users, but more technical web masters use access logs and various other server side tracking tools to run data reports in the background. You can get around this by adding your own tracking client side, but that requires extra code and more work which you may not want to do. It also doesn't catch everything a server side tracker would catch, such as spiders and non-javascript browsers. Lastly, any changes you make to your site won't happen in real time. Since everything is cached via Cloudflare's edge network, you'll have to wait for their Cache to expire, or manually flush the cache every time you make code changes. I get around this by implementing Cloudflare's API into my deploy process to flush the cache that way.


#### Conclusion

If you're wanting a quick boost to auth-less pages on your website without any code changes or effort then Page Rules are perfect for you. There are so many other use cases for Page Rules which I didn't cover above, such as Ignoring certain static files, making certain pages redirect, disabling protection for pages or even simply disabling features for parts of your site such the admin panel. Feel free to comment if you have any questions, I'm also looking forward to seeing what you are using Page Rules for yourself.


On a side note, I can't actually share the particular website with you but I did enable the same features on one of my smaller personal sites, <a href="https://molehillgaming.com" title="Molehill Gaming" target="_blank">Molehill Gaming</a>, as a testing playground. Go check it out and see how fast it loads!

Using Cloudflare Page Rules

Today I finally released the beta version of my MongoDB compression library for PHP. There still a lot of work to be done, but it's stable enough to use as a drop replacement for standard usage such as saving and retrieving data. Not only have I released the source code, but I've also submitted it as a composer package on <a href="https://packagist.org" target="_blank">packagist.org</a> so everyone can include it in their projects very easily.

The library is fully unit tested and has been benchmarked with millions of documents yielding impressive results. This is my first real open source contribution, and also my first fully packed and tested library. I am looking for feature requests and bug reports for as many use cases as possible before releasing it as version 0.1, so the code is currently on dev-master by design; not because I'm too lazy to version it.

Currently there is no support for the aggregation framework or map reduce, but this is in the pipeline and will be available shortly. These options are still available manually through the $native-> interface on each component of the library with manual attributes.

An overall goal for this project is to be a 100% drop-in replacement for the native MongoDB PHP driver, which means it will eventually support all features that does with a few extra optimisations and helpers. Any feedback is welcome and I'd love to know what you guys think. Below are the links to grab the code and start using it right away.

- Source: <a href="https://github.com/marcqualie/mongominify" target="_blank">https://github.com/marcqualie/mongominify</a>
- Packagist: <a href="https://packagist.org/packages/marcqualie/mongominify" target="_blank">https://packagist.org/packages/marcqualie/mongominify</a>

MongoMinify Launched

The other day I stumbled across an amazing feature of Sublime Text 2 that doesn't require a plugin to work, yet somehow after 1 year of using the software I hadn't come across it. If you ever find yourself rewriting huge code blocks such as configs, or copying/pasting lots of text and replacing certain parts for folder names for example then Snippets are the perfect tool! This is a short post to share awareness of this awesome feature.

The <a href="https://sublime-text-unofficial-documentation.readthedocs.io/en/latest/extensibility/snippets.html" target="_blank">Snippets Documentation</a> is very simple to follow and writing your own couldn't be easier. How I have survived this long without this tool is crazy, they speed up productivity so much when programming. This feature should be emphasised more as it adds so much functionality right out of the box which is normally only obtained via the package system. Quote from the documentation "Snippets can be stored under any package’s folder, but to keep it simple while you’re learning, you can save them to your Packages/User folder". This means it's very simple to get started in writing them with no previous setup. I am tempted to write a plugin to manage snippets and share them better, but something makes me think one may already exist so I'll do some research into that.

The quickest way to get started with snippets is to go to "Tools -> New Snippet" from inside the Sublime Text 2 menu system. I find it ironic how there isn't a keyboard shortcut for that. From here you can start writing the content for your snippet which can be as simple as copy/paste text block (I have one for standard HTML5 with bootstrap layout) or a more advanced template such as what I use for <a href="https://gist.github.com/4402490" target="_blank">Nginx Configs</a> during development. I'm going to be sharing a lot more of these via the Github Gist system over the new few weeks because they have helped me so much!

Feel free to share your own or others in the comments or use mine as you please.

Sublime Text 2 Snippets

I've had a few people asking about the specs on my new gaming machine, so here they are! This upgrade should have came about a year ago but thanks to a trusty desk fan I managed to keep my old computer going for this long. The hardware was still running all of the latest games perfectly fine, but it was always overheating and the case was tiny with not much run to expand above 8GB RAM on the motherboard.

<a href="//cdn.marcqualie.com/blog/img/20121226_161652.jpg" target="_blank"><img class="pull-left" src="//cdn.marcqualie.com/blog/img/20121226_161652_preview.jpg" style="width:120px;height:90px;margin:0 5px 0 0"/></a> The processor is the <a href="https://www.amazon.co.uk/gp/product/B007RUZKK6/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B007RUZKK6" target="_blank">Intel 3rd Generation Core i5-3570K CPU (4 x 3.40GHz, Ivy Bridge, Socket 1155, 6Mb L3 Cache, Intel Turbo Boost Technology 2.0)</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B007RUZKK6" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> which is currently the fastest i5 on the market, and can be over clocked easily to about 5.2ghz without damaging or overheating. If you're wondering why I went for the i5 instead of i7 then it's because this is a gaming machine and the processor doesn't play as much as part as the GPU, so the price difference (£100+) wasn't worth the upgrade to i7.

<a href="//cdn.marcqualie.com/blog/img/20121226_161747.jpg" target="_blank"><img class="pull-left" src="//cdn.marcqualie.com/blog/img/20121226_161747_preview.jpg" style="width:120px;height:90px;margin:0 5px 0 0"/></a> I went for the <a href="https://www.amazon.co.uk/gp/product/B007KZQE7Q/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B007KZQE7Q" target="_blank">Asus P8Z77-V LX</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B007KZQE7Q" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> as the motherboard because of the price and the features. I needed USB 3.0 support and this has 4 + 2 for case, which was perfect. It also comes with 2 + 2 USB 2.0 ports which always come in handy for various gadgets. The is also the <a href="https://www.amazon.co.uk/gp/product/B007KZQEGC/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B007KZQEGC" target="_blank">Asus P8Z77-V</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B007KZQEGC" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> which has SLI support and built in WiFI for an extra £40 which I didn't need, but others might find handy.

<a href="//cdn.marcqualie.com/blog/img/20121226_161720.jpg" target="_blank"><img src="//cdn.marcqualie.com/blog/img/20121226_161720_preview.jpg" class="pull-left" style="width:120px;height:90px;margin:0 5px 0 0;"/></a> For the RAM I went for <a href="https://www.amazon.co.uk/gp/product/B004QBUL1C/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B004QBUL1C" target="_blank">Corsair 8GB DDR3 Vengeance Memory Module Kit</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B004QBUL1C" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> which I will be upgrading to 16gb tomorrow to fill up the extra slots on the motherboard and future proof the system a little more. The board actually support 32GB which is pretty awesome if you're going for a really intense setup.

<a href="//cdn.marcqualie.com/blog/img/20121226_180136.jpg" target="_blank"><img class="pull-left" src="//cdn.marcqualie.com/blog/img/20121226_180136_preview.jpg" style="width:120px;height:90px;margin:0 5px 0 0"/></a> For the case I decided to go for something fancy since my last one kinda sucked. The <a href="https://www.amazon.co.uk/gp/product/B0067FFYXU/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B0067FFYXU" target="_blank">Cooler Master 690 II Nvidia Edition</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B0067FFYXU" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> looked amazing in the shop and even better on my desktop with the lights on the front! I'm also considering buying an extra light for inside to make the side graphic glow more. This case is huge and has a removable bay for SSD and 3 case fans built right in for great air flow. To my surprise it actually runs silent even when playing games like Borderlands 2 on full graphics. There are rubber pads on the base and the disk bays are made from rubbery plastic material to soften sound if you had spinning disks. My favourite feature when building it was the hole in the base to filter wires through the back and out of site for better cable management.

It's all powered by a <a href="https://www.amazon.co.uk/gp/product/B007E7O7TC/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B007E7O7TC" target="_blank">Cooler Master GX-Lite 600W Power Supply</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B007E7O7TC" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> that runs silent and comes with plenty of connections to power all the internals. I was falsely told this was a modular supply but it isn't. I didn't mind this too much when paired with the case's built in cable management system, though.

My graphics card I already owned, so I kept that since it's still very powerful even for the latest games. It's a <a href="https://www.amazon.co.uk/gp/product/B0099AOD50/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B0099AOD50" target="_blank">GeForce GTX 660 SC 2GB GDDR5 Graphics Card</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B0099AOD50" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> and I haven't had any issues running games on highest settings since I bought it in October 2011 so I would highly recommend.

For storage I already had an <a href="https://www.amazon.co.uk/gp/product/B004Z0S6S4/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B004Z0S6S4" target="_blank">OCZ 240G Agility 3 SSD</a><img src="https://www.assoc-amazon.co.uk/e/ir?t=marqua-21&l=as2&o=2&a=B004Z0S6S4" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;" /> so I simply transfered that over with all of my system already installed and running Windows 8 with all of my steam games. I also bought an extra 1tb drive for my server so move some more movies over and clear up more space for games!

My new Gaming Rig Specs

I've put together this simple install script that I compiled from the various tutorials on the rabbitmq site. It's not intended to be anything new or that isn't easily accessible; It's simply the quickest way I've found to get up and running with the web interface out of the box.

``` bash
echo "deb https://www.rabbitmq.com/debian/ testing main" > /etc/apt/sources.list.d/rabbitmq.list
wget https://www.rabbitmq.com/rabbitmq-signing-key-public.asc
sudo apt-key add rabbitmq-signing-key-public.asc
apt-get update
apt-get install rabbitmq-server -y
service rabbitmq-server start
rabbitmq-plugins enable rabbitmq_management
rabbitmqctl add_user admin password
rabbitmqctl set_user_tags admin administrator
rabbitmqctl set_permissions -p / admin ".*" ".*" ".*"
rabbitmqctl delete_user guest
service rabbitmq-server restart
```

Make sure you change the word "password" to the actual password you want to login with in the add_user line before running. The default port for the admin interface is 15672. You can use the management interface to finish off your configuration such as adding exchanges, queues, users and vhosts. Make sure to open the correct firewall ports (15672, 5672) if you are going to be accessing the box outside of a VPN or localhost.

Install RabbitMQ on Ubuntu 12.04 Quickly

I recently just passed my probation period and got hired full time at Performance Horizon which means I get to work on awesome technology with a great team for the foreseeable future! Over the last 3 months I've met really interesting people, attended conferences and meetups, worked on complex edge cases and compiled quite an inventory of tools I use in my daily work. This post is pretty much a summary of those tools and my opinion on why I think you should give them a try. I've included all the links to the mentioned projects at the bottom of this post.


#### Sublime Text 2

Simply the best IDE I've ever used, hands down. In fact I love it so much I purchased a license even when it's free. There are hundreds of plugins via the package manager and I use quite a lot of them, but one that stands out the most right now is Origami. It allows you to split your workspace into smaller chunks and move them (including idividual files within) around the screen in any manner; All with really simple to follow keyboard shortcuts. Since I do this inside Terminal for managing multiple connections in the same window, it was a natural extension to Sublime's already great interface. Any developers not currently using Sublime should really check it out!


#### Fabric

I've recently started becoming more and more interested in Python, and Fabric has a huge part to play in that decision. Previously I've been managing my servers through a custom built bash deployment script which can run hooks and upload the current git commit. This is great, but not very flexible when I want to add new features or do more advanced actions. Fabric is perfect for this as it allows me to assign roles (I manage a lot of servers) to run actions on multiple servers at once. Not only can I deploy my code and run actions on a per project basis now, I can also restart services, install software and even run backup scripts and build an entirely new system from one file. I much prefer this to some of the more fully featured and in my opinion bloated options such as Puppet and Chef, personal preference only, I like simplicity.


#### Bitbucket / Github

Not really development tools but I feel like I should mention them as they're part of my daily routine. If you are a developer and aren't using one of the above (or self hosted version control like Gitlab) then you are missing out on a lot of the great features they offer such as Wikis, Issues, Pull requests and easy collaboration. There are also other options such as Codebase and Gitorious that I haven't personally used, but my point is that hosted version control is awesome! Our team recently passed 10,000 commits and it's great to be able to role back to any point in time over the entire development period.


#### Silex Framework

Until 3 months I was completely against using open source frameworks because over the years I had developed my own which was a lot more optimised and does exactly what I want. Also, being head of development I wanted my team to know the code inside out and learn how to build the insides of a framework as aposed to building on top to build better programming skills. However, as I'm moving away from actual development and more into architecure and ops, I have less time to maintain such frameworks so I've tried a lot of open source options. My current company uses Kohana, and it's awesome! It's not my personal choice though, as the documentation sucks it doesn't feel as simple to use as it could be. After my visit to PHPNW12 (PHP Conference in Manchester) I was enlightened to Silex, and Symfony components. This website and blog is actually running off both of those and only took 2 days to rebuild the entire backend including my API and admin interfaces. Building on Silex is so easy, well documented and very extendible with one of my favourite features being the Twig templating framework.


#### Composer

Following on from Silex, Composer is essentially a long overdue package manager for PHP. Ever since I discovered it I've found any excuse to incorporate it into my projects. I've even built a few vendor modules to reuse in my various projects, one of them I intend to fully open source as part of a bigger library. The community around composer is thriving right now and people are building great libraries, including the entire symfony component library is on there which is why installing and running Silex is so simple. Any PHP developers who aren't already using composer really should be, and I would recommend at least installing the Silex framework to try it out as it's a great way to version control your libraries and share them in seconds to everyone else who is using them. I will probably write a post in the future going into more detail about managing your libraries and dependencies with Composer, let me know in the comments if this would interest you.


#### Supervisord

A bit of a confession here, I only found out about this in the last few days but it's helped so much already and after seeing it in action I feel I have to share it! Supervisord is a python (I know, more python) tool that manages your services and can spawn instances and keep them running. I'm currently using it to manage services on my Macbook Pro as my current development environment involes multiple instances of MongoDB, Couchbase, Memcache, Redis, Gearman Consumers, Producers, MySQL, Nginx, PHP FPM (.. and many more) so having one config file to manage the starting stopping and status of each of these with inbuilt logging is great. It also comes with a simple nice looking web interface to make managing the services even easier.


#### Vagrant

Since in my daily routine I'm often switching between projects and environments I was recommended an app called Vagrant. This amazing tool allows me to have an entirely isolated virtual machine (running on a Headless Virtual Box) that I can spin up when I need it without any kind of code modifications or workspace changes. I've heard so many developers also using it in their daily routine due to the fact you can share entire environments between team members and system configs can be passed around easily. My favorite feature is how you can spin up an exact copy of your live OS image and test your code against it full before deployment or having extra hosted machines to testing code, much cheaper.


#### Source Links

I've combined all of the links below so you can check out each of the projects easily, let me know if I missed any.

- Sublime Text 2 - <a href="https://www.sublimetext.com/2" target="_blank">sublimetext.com</a>
- Origami - <a href="https://github.com/SublimeText/Origami" target="_blank">github.com/SublimeText/Origami</a>
- Bitbucket - <a href="https://bitbucket.org/" target="_blank">bitbucket.org</a>
- Github - <a href="https://github.com/" target="_blank">github.com</a>
- Fabric - <a href="https://www.fabfile.org/" target="_blank">fabfile.org</a>
- Silex - <a href="https://silex.symfony.com/" target="_blank">silex.sensiolabs.org</a>
- Composer - <a href="https://getcomposer.org/" target="_blank">getcomposer.org</a>
- Supervisord - <a href="http://supervisord.org/" target="_blank">supervisord.org/</a>
- Python - <a href="https://python.org/" target="_blank">python.org</a>
- Vagrant - <a href="https://vagrantup.com/" target="_blank">vagrantup.com</a>
- Linode (Web/Database Hosting)- <a href="https://www.linode.com/?r=dfdb6f0a4cbe69831090eb51f6186e6aa2e060cb" target="_blank" rel="nofollow">linode.com</a>
- VPS.net (Hoard/Message Queue Hosting) - <a href="https://manage.aff.biz/z/11599/CD18136/" target="_blank" rel="nofollow">vps.net</a>

I'm always on the lookout for new tools and cool projects so feel free to let me know about any you guys are using in the comments, or share them through <a href="https://twitter.com/marcqualie" target="_blank">twitter @marcqualie</a> as that's where I find most of these anyways!

Awesome Tools I've Been Using

I've been meaning to try out <a href="https://pages.github.com/" target="_blank">Github Pages</a> for quite a while, but never had a project that was suitable as nearly all of my work is built on a custom back end server. I also wanted to get a few old projects online to share the code so people can maybe do something cool with it.

The first project I wanted to upload was my <a href="https://marcqualie.com/2012/04/javascript-racing-game-prototype">Racing.js Prototype</a>, mainly because I enjoyed working on it and with the increased interest in HTML5 gaming, there's a possibility it could be used for something useful. The great thing about this project is it's entirely frontend, so only HTML and JavaScript is required to run, no server required. Enter Github Pages. All of the content is hosted like a static website, similar to how Amazon are doing with S3.

My favorite part of the system is how it's simply and orphan branch inside of your standard git repository, which you would already be hosting there anyway. No extra management or uploads to update the repo, simply commit to the gh-pages branch. Another great advantage is that it's away from your standard infrastructure and keeps the source code and demos all in one place for your project.

A great way I've seen people using this is to host their Blog using <a href="http://octopress.org" target="_blank">Octopress</a>. Since you compile the entire locally into static html files, it can be stored in a git repo and served directly out of the repository. Demo pages are also popping up everywhere, which is quite interesting to see them hosted in the same place as the source code itself meaning they are often updated at the same time, and the horrible days outdated demo pages are being more rare.

This post is in no way a Tutorial or Guide, but simply here to highlight that Github Pages exists and that you should try it out if you haven't already. I currently use the manual deploy method which you can find <a href="https://help.github.com/articles/creating-project-pages-manually" target="_blank">here</a>. Hope someone finds this helpful and has an awesome use for Github Pages.

Free Hosting with Github Pages

I've decided that having my Blog hosted on 3rd party software is no longer an option. After 3+ years of posting originally on Blogger, then on Wordpress I've decided to finally integrate it into my main site, which I rebuilt from the ground up a few days ago.

There will most likely be a few teething problems after the switch as there are a lot of things I haven't fully finished writing yet such as the feed system and pagination. For now, there is simple search at the top which will also be replaced in coming days. Right now I'm just happy to be not using Wordpress anymore as it was awful to update, horrible coding, not very modular and so, so slow hogging a lot of my server resources.

Now that I finally have the foundation for my own custom blogging platform I can start writing a lot more frequently as I have a huge backlog of cool projects that I want to share with you guys. I'm also going to start doing request posts and reviews on new applications, software and tech so suggestions are welcome!

No more Wordpress!

Today I found a major flaw with a piece of software I use heavily every single day, as do a lot of people I know. Logmein Hamachi is currently using the 5.0.0.0/8 subnet, which has recently (Novermber 2011) been assigned to real users and conflicts are beginning to happen which is a major issue. Basically, the problem makes any service running in this subnet to appear invisible to Hamachi users, and webhosts are starting to assign these addresses to real users by the looks of it. I stumbled across this issue when connecting to a demo server at the company I now work for. Other people in the office could connect, yet I couldn't. The machine just appeared dead until I noticed the IP was in the 5.0.0.0/8 range!

Of course the easy way to fix this is to enable IPv6 only mode from within Hamachi. Luckily for me, all of my servers and Macbooks support this so the transition was easy, however a lot of people who use Hamachi use it for legacy gaming for LAN matches, which certainly won't support IPv6 as many other services won't yet or ever will. The worrying thing right now is this is actually the <a href="https://community.logmein.com/t5/Hamachi/5-x-x-x-Address-Range/td-p/83996" target="_blank">official “fix”</a> from Logmein for the issue. From my research it turns out other services such as EasyVPN are also using this address range, and it was common ground for this subnet to be used for VPNs so I can imagine quite a few connectivity issues arising when more of these IPs are given to services. People are reporting problems with people connecting to their websites, which is never good.

So next time a site is down for no apparent reason, try pinging it to see if the IP is in the above address range, if it is and you have Hamachi (or other VPN service) you might want to think about converting to IPv6 or disabling it all together. Hope this helps someone avoid hours of confusion, as there is no way to really “debug” this unless you are familiar with the fact all Hamachi IPs have the same subnet.

Have any of you guys noticed problems with other VPN providers? Feel free to share your thoughts in the comments.

Beware of Hamachi IPv4 Subnet

Where has the time gone! I've been working at my new job for almost a whole month now and it still feels like I've just started! For those of you that don't know I'm now a programmer at <a href="https://www.performancehorizon.com/" title="Performance Horizon" target="_blank">Performance Horizon</a> which is pretty awesome; quite the change from flash gaming! It's also the main reason I've been neglecting my personal projects and blog updates to concentrate all the cool things I've been working on.

My backlog of projects to upload to Github is quite big so this week I'm going to be uploading them, starting with the full source code to my <a href="https://marcqualie.com/2012/04/javascript-racing-game-prototype.html" title="JavaScript Racing Game Prototype" target="_blank">JavaScript Racing Engine</a> which I plan to do more work on in the future and maybe make it into a mini mobile game. I also have a great JavaScript gaming engine which I built, unrelated to the racing engine which has all the standard features from the Actionscript 3.0 library such as asset embedding, rotation, sprite animation, audio and more!

Secondly, I've done so much work on my <a href="https://github.com/marcqualie/hoard" title="Hoard" target="_blank">Hoard</a> project which I'm still putting together as it's integrated deeply inside one of my other projects. It has great new features including more control over user permissions, custom data types, import/export and a dashboard reporting system with an example of how to build a simple web analytics engine ontop of the core code. I'm also working on making it run out of the box with Nginx servers, but I'm still new to tweaking those for performance and stability.

I've also got a few server modules I've been building for custom configs and I've found some pretty cool new tech to share with you guys once I've drafted together some tutorials and tips on how to install and use them. As usual, I build all of my sites ontop of the <a href="https://manage.aff.biz/z/146/CD18136/" title="VPS Cloud Hosting" target="_blank">VPS.net</a> network, but I've recently found a great new tool called Vagrant which is making server development so much easier (and cheaper) then deploying custom servers. Obviously there is the disadvantage of slow networking and no public access unless you configure your router which will hog your home connection.

Lastly, I've just updated my blog to use Bootstrap 2.1.0 which I think gives it a much cleaner and brighter feel than before. If you haven't tried it out, I recommend using it as it speeds up development so much if you're a developer and just want to throw together a quick prototype before you get a designer to polish it up. Feel free to comment if you have any requests for which projects I should open source first, or have any suggestions for new posts.

It's September already?

Today I finally got around to upgrading my home media server to Ubuntu 12.04. I'd been putting it off for a while, mainly because I was busy and didn't have time to run extra backups. Anyways, everything ran smoothly until I forgot it was running and closed my Macbook lid, which resulted in no connection and a broken SSH pipe. This is the first time it's happened to me so I quickly did some googling and found the best solution.

Luckily the creators of Ubuntu planned ahead for this kind of thing and run a separate SSH Deamon (mine was port 1022) so I could connect to that using <code>ssh me@myserver.com -p 1022</code>. Now that you're connected, you will notice everything is locked down. The entire file system will be locked down incluing apptitude, and you can't even resume using <code>dpkg --configure -a</code> or <code>aptitude t -f dist-upgrade</code>.

There is a screen instance running as the root user on th machine, which you can easily reconnect to using <code>screen -d -r</code>. Once you are connected to this, you can easily resume wherever you were disconnected from. If you aren't logged in as root, which is good practice when you need to <code>sudo su</code> to get to the root user, or <code>su root</code> then enter the root password if you don't have sudo permissions. Thankfully that's all I needed to do to resume my upgrade.

Recover from Interrupted Ubuntu Release Upgrade

I mainly write these blog posts for my own personal reference, so I don't forget how to install all of these various packages, but also because I learn a lot of my stuff from other people's posts and like to contribute back where I can. Below is my install script for GeoIP on CentOS 5.6+, and so far it has never failed. It should work on any Redhat based system with the yum package manager, but I haven't tested on any system below 5.6. I normally create scripts in a personal install directory and run as sh ~/install/geoip.sh.

``` bash
mkdir /usr/share/GeoIP && cd /usr/share/GeoIP
wget "https://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz"
wget "https://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz"
gunzip GeoIP.dat.gz
gunzip GeoLiteCity.dat.gz
yum update
yum install php53u-devel php53u-pear geoip geoip-devel
pecl channel-update pecl.php.net
pecl install geoip
echo "extension=geoip.so" > /etc/php.d/geoip.ini
service httpd restart
```

The parts what a lot of people miss out are the devel packages which are needed to compile the php extensions. This works right out of the box, which means you don't need the GeoIP.dat files in your webroot, or the GeoIP.inc file. The function names are slightly different, but work in the same manner and a lot faster.

Installing GeoIP on CentOS

Today, I attended the workshops leading up to the <a href="https://www.10gen.com/events/mongodb-uk" target="_blank">MongoDB UK</a> event and learned so much from the amazing solutions experts over at 10gen. I can't wait to apply everything I learned such as replica sets, sharding and forward planned Schema Design. I say Schema Deisgn, but since Mongo is a Schema-less database I actually mean an implied data structure on how to highly optimize applications for different use cases.

Before attending I had intermediate understanding of how mongo worked in a production environment, but now I feel confident I could multiple machines, back them up and shard them with ease. It was also quite fun being a physical shard as part of <a href="https://twitter.com/cj_harris5" target="_blank">Chris</a>&#8216;s role play example which helped to visual how data is actually managed and passed around the machines.

I'm even more excited for MongoDB UK tomorrow, which is of course, the main event. With speakers from AOL, Nokia, Cloud Foundry and gov.uk I'm sure there will be some amazing use cases to be learned from. I'll hopefully try and getting some posts written while I'm there tomorrow, and maybe even write up some coding examples from what I learn. Also looking forward to the Sharded pub crawl afterwards, but let's not tell the boss about that one!

MongoDB UK Workshops

Here is a small script I whipped up while I was bored in the Travelodge waiting for <a href="https://twitter.com/search/%23mongodbuk" target="_blank">#MongoDBUK</a>. It's basically a drop in script to make sure MongoDB is installed with the default settings, and up to date. Very simple, but it saves running the commands manually each time. I may update this in the future when I find a better way to check if a key is already added before always inserting it

``` bash
#!/usr/bin/env sh

# Sudo is required to install everything properly

if [ "$(whoami)" != "root" ]; then

	echo "You need to use sudo to run this command"

	exit 0;

fi

echo "Installing MongoDB, please be patient, this could take a few minutes.."

REPOSOURCE="deb https://downloads-distro.mongodb.org/repo/ubuntu-upstart dist 10gen"

REPOFILE=/etc/apt/sources.list

# Make sure packages are up to date and key exists

apt-get update -qq && sudo apt-get upgrade -qq -y

apt-key adv --keyserver keyserver.ubuntu.com --recv 7F0CEB10

# Append to repo

if ! grep -Fxq "$REPOSOURCE" $REPOFILE

then

	echo ": Adding $REPOSOURCE to $REPOFILE"

	echo "$REPOSOURCE" >> $REPOFILE

	apt-get update -qq

fi

# Run install

apt-get install -y mongodb-10gen
```

I normally put this in my home folder at <code>~/install-mongodb</code> then run it using <code>sudo ./install-mongodb</code>. Note that sudo is actually a requirement, and the script will warn you if it you don't have root permissions. I will add more scripts in the next few hours as I go through the MongoDB UK workshops to have simple ways of switching configs for different scenarios and replacing the defaults to enable auth, for example. If you have any cool things you'd like me to write about regarding MongoDB please feel free to ask in the comments.

Install MongoDB on Ubuntu 12.04

<a href="https://lh4.googleusercontent.com/-QQCepJIhU4c/T9zR6HxDqII/AAAAAAAAEZI/yraCndSouGk/w497-h373/12%2B-%2B1">
 <img style="float:left;margin-right:15px" title="Samsung S3" src="https://lh4.googleusercontent.com/-QQCepJIhU4c/T9zR6HxDqII/AAAAAAAAEZI/yraCndSouGk/w497-h373/12%2B-%2B1" alt="" width="170" height="100" />
</a>

I am now the proud owner of the Samsung S3. In fact, I'm actually writing this post from it using the WordPress for android application which looks beautiful on the large and vibrant 4.8 inch screen.

I haven't had much chance to push it to it's limits yet, but so far I'm extremely impressed. Gaming and app switching is so smooth and the OS level motion gestures make using the phone so fast and easy. I was disappointed when I couldn't run Dead Space, but that's due to EA not updating it, not the phone.

It also came with 50gb of <a href="https://db.tt/QAzmwzS" target="_blank">Dropbox</a> space which is a huge bonus. Wifi and Network speeds are stupidly fast for a mobile device where I managed to get upto <code>7mb</code> download speeds while tethered. Also, with a ping of <code>121ms</code> it's very easy to manage servers over ssh without a Wifi connection.

It does come bundled with some Samsung bloatware, but it stays out of the way nicely and never forces me to use it unlike HTC Sense in my 3 previous android phones. Coming from a HTC Desire, the size was a bit overwhelming at first but now it feels natural and comfy.

NFC is built in, but nowhere in England seems to really support it yet so I'm going to have to wait to try that out at a later date. The torch is very powerful and the camera is amazing quality and feature packed including 1080p video recording, smile detection and self portraits with the front facing camera.

Overall, it's an amazing device and if you're looking for a new phone then I would seriously recommend going for the Samsung S3.

Samsung S3

Since <a title="MongoDB" href="https://mongodb.org" target="_blank">MongoDB</a> is technically a schema-less database, there aren't fields or columns that you are used to in things such as MySQL, and certainly no decent applications for managing data easily. Luckily, MongoDB has a lovely command shell which can be queried using simple JavaScript. The below command will rename a field, if you named it wrong in an initial stage of your app, like I've done a few times while learning.

``` javascript
db.post.update({}, { $rename: {"old_name": "new_name"}}, false, true);
```

The first parameter is your standard query, which I've left blank to match all documents by default. Please note that the 3rd and fourth paramters are actually required, or this won't work as by default only one document will be matched and an upsert will be applied, which isn't what you want in this situation.
You can find the full documentation and explanation of the update function and the $rename command over at <a title="Updating - MongoDB" href="https://www.mongodb.org/display/DOCS/Updating#Updating-%24rename" target="_blank">mongodb.org/display/DOCS/Updating#Updating-%24rename</a>

How to Rename a Field in MongoDB

I've just finished working on the first prototype for my up and coming racing engine written in JavaScript. It's nowhere near finished, but it gets the simple concept across of where this could be taken. Right now it's not a game at all, it's simply an engine which can be used to make a game. I wrote a basic one track map, with dumb AI cars and a player controlled car to show some of the functionality. Please excuse the bad graphics, I will be improving them later as the project develops.

I will upload the code to Github as a fully open source project once the project reaches a mature stage. Right now it's pretty messy since it only took me one day to write. This is also a chance to show off a new kind of Framework I have been thinking about, where JavaScript games can take over and become as popular as Flash games by getting rid of the one thing holding them back, but I will write more about that in another post once that project has been launched. I've embdeded the game below for you to play around with.

<iframe src="https://marcqualie.github.com/racing.js/" frameborder="no" width="640" height="480"></iframe>

Make sure you press the spawn car button to view the simple AI I started to write for it. I may draw up a simple gravel track so it looks a bit nicer later on tonight if I get time, but right now I'm focusing on the code only. Let me know what you think in the comments?

Update: The code is now live over at <a href="https://github.com/marcqualie/racing.js" target="_blank">github.com/marcqualie/racing.js</a> and the demo above is actually hosted from that repo on Github Pages.

JavaScript Racing Game Prototype

I recently just discovered that Mac OSX doesn't have <a href="https://github.com/mirror/wget" target="_blank"">wget</a> functionality, and that's one of the most useful things about the command line to quickly download files without a UI. The most common time I use it is when I want to download .swf files, which isn't very streamlined through the browser. The quickest way to download a file would be to run:

``` bash
wget -O ~/Downloads/eggstinction.swf https://www.freeonlinegames.com/games/21945/game.swf
```

in the command and your file would simply be in your Downloads folder. Since wget isn't available, this is where <a href="https://curl.haxx.se/" target="_blank">curl</a> comes in. To do this in Mac OSX (or any other OS with Curl installed) you would simply run:

``` bash
curl https://www.freeonlinegames.com/games/21945/game.swf > ~/Downloads/eggstinction.swf
```

Very short post, but straight to the point. Downloading files in this mannor is very useful, one of which is storing large files in a personal cloud system then only downloading them when you need them. Very useful if you write a script to download batches using curl, but that's another post.

Using Curl to Download Files via Command Line

A pet hate of mine on Mac is the inability to order directories with folders at the top. It makes perfectly logical sense to put things in Alphabetical order, but when you're a developer and working with lots of folders which you have to navigate through and huge directories it's just a pain.


Unfortunately, it's not as simple as checking a box. You're going to have to rewrite a tiny bit of one of the files. First of all you're going to want to convert the text file into XML so that you can edit it, then open it with Nano.

``` bash
cd /System/Library/CoreServices/Finder.app/Contents/Resources/English.lproj
sudo plutil -convert xml1 InfoPlist.strings
sudo nano InfoPlist.strings
```

Once you have the file open, navigate to the section which has Folder in, about a 3rd of the way down which will look like this

``` xml
<key>Folder</key>
<string>Folder</string>
```

Simply put a space before Folder in the String field so it looks like this.

``` xml
<key>Folder</key>
<string> Folder</string>
```

Once you've done that, you need to convert the file back to binary format so your system won't break once it's restarted.

``` bash
sudo plutil -convert binary1 InfoPlist.strings
```

All done, simply restart finder and then order your folder by "kind" type.

``` bash
sudo killall Finder
```

Order By Folders First in Mac OSX Lion

It's quite annoying how non-simple OSX is to Linux, so nearly all of the commands I'm used to don't actually work anymore so I have a lot more to learn. One of the basics was the lack of <code>usermod</code> which I use quite a lot while setting up permissions for things like client FTP, shared folders in Samba and vhosts in Apache2. Seems there is no simple way to do this on Mac, and the command is somewhat long winded, but here it is for anyone looking for a solution that works.

``` bash
dseditgroup -o edit -u adminusername -p -a username -t user group
```

Not quite as easy to remember as usermod, but I'm sure after a few attempts it will sink in. <code>adminusername</code> is the name of your admin account and <code>username</code> is the name of the account you wish to ammend. <code>group</code> is the name of the group you wish for the user to go into.

Adding User to a Group on OSX Lion

I'm so excited for the launch of Mass Effect 3 that I felt like I had to write about it to get it out of my system. I was extremely disappointed when the launch got delayed, as it was planned to launch a few days after my Birthday last October. However, after hearing about the reasons behind it, I didn't care too much.

Kinect Voice Integration is going to be so awesome, yet extremely nerdy at the same time. Shouting at your Xbox, giving virtual characters commands as you blast your way through space and stop the Galaxy from being destroyed? Hell yeah!

The Demo is going to be launched on February 14th (Unlucky for our girlfriends, or boyfriends for you gamer girls!) and the game will be fully launched to all platforms on March 9th so if you haven't planned to play it yet then get the date set.

I have no idea what the Multilayer will be like but I'm sure it's going to be insane and huge.

Now I'm off to go and play through Mass Effect 2 again, since I've completed the first game twice already.. see you guys next month, maybe?

Mass Effect Addiction

Are you still using FTP for website deployment? Don't. If you're like me, and think there isn't a better solution then look no further than Git deployment. This topic has interested me for a very long time, but I'd never found a way to do it efficiently with the least amount of overhead and set up complexity. Before now I was currently using a mixture of FTP and Rsync to manage uploads, but both of these had major drawbacks. 

#### FTP

Where do I start with FTP? It's an outdated protocol which isn't very good for syncing changes, and unless you have some kind of automatic syncing software, it's a tedious manual job. Until Recently I was using this for nearly all of projects, and I'm quite embarrassed about it to be honest. As a developer who is always trying out new technology across the board and always up to date, there is just no excuse for still using FTP. The advantage FTP has is the transfer speed, but when you factor in how insecure, heavy and failure-prone it is, it is still the worst choice. 


#### Rsync

Rsync is a beautiful way to sync updates to projects and websites. It's fast, only updates files which have changed and if you write a script it can be entirely automated. The main reason I use Rsync is for replicating changes through multiple servers and data centres. The script detects the files changes which need to be pushed, and updates them accordingly on a schedule, or when it detects the files have been changed in a similar way to how Dropbox works. The reason I don't use it for everything is I can't find a decent Windows solution and I still use Win7 quite a lot for development both at work and home. 


#### Git

Now we're on to the powerful stuff you all came to see. What are the main advantages of Git Deployment then? First of all, for many of you guys, git will be your main source of version control and where you host your projects if you're working in a team. One of the largest examples of this would be <a href="https://github.com" target="_blank" title="Github">Github</a>, which I'm sure you've all heard of. Personally, I use my own custom built Git system on a VPS, mainly for customization, security and cost reduction due to the large amount of projects I manage. I will write a post on that later on, explaining how to host your own git repositories without any additional cost.

To start with, I am going to assume you have sufficient basic experience with the Linux command line. All of these commands are done through a remote server, and depend upon you have <a href="https://marcqualie.com/2011/07/improving-ssh-security-on-linux.html">public keys</a> set up. Go ahead and login to your server and create a folder where you will keep your repositories (e.g <code>/var/www/git</code>) and then create another folder specific to your first project (e.g <code>/var/www/git/project1</code>). Navigate into this directory and type the following commands, where <code>$worktree</code> is an absolute path to the folder where you keep your actual site files (e.g /var/www/html). The worktree part is important, or it won't deploy to the correct directory. Also, If you're curious why I'm doing it in a separate folder and modifying the worktree rather than the same directory as the site, is so that the .git folder is completely separate for backup purposes, and there are less files in the document root. It's especially useful when deploying on multiple servers. I deploy via git to a master server per cluster, then internally Rsync takes over and only syncs the files it needs to the internal cluster, thus saving overhead on deploying to 30 different machines.

``` bash
$ git init
$ git config receive.denycurrentbranch ignore
$ git config core.worktree $worktree
$ sudo nano .git/hooks/post-receive
#!/bin/sh
env -i git checkout -f
echo "Deployed!"
$ sudo chmod +x .git/hooks/post-receive
```

Then on your local machine you will want to add another remote to your git repo.

``` bash
$ git remote add live ssh://yourserver/var/www/git/project1
```

As I mentioned before, you will have to have already setup public keys for access to your server via SSH or else this method won't work. I've read through a lot of articles on deploying through git, and tried them all and this method was the best Hybrid I found. It doesn't require a bare repository, you can push/pull from it like a normal repo and it doesn't destroy and static files you may have in your document root on your server such as file uploads from users.

When you're ready, simply commit then push to it like a normal remote. I create a .bat or .sh file with the following content, for automatic deployment without having to do this every time.

``` bash
$ git add .
$ git commit -am "Automatic Deployment"
$ git push live master
```

Another great use for this kind of deployment I've found it the ability to have seperate remotes pointing to different machines for testing purposes. If I'm not 100% sure that something will work live, I can send it to a sandbox machine first via a different remote, then once it works there, simply repeat the process with the live remote and within seconds the site is live.

If you have any questions or suggestions I'm happy to chat with anyone interested in this kind of deployment or similar.

Deploying Websites with Git

Recently I've been looking into a lot of Technology and Software to improve the performance and scalability of some of the projects I've been working on. This has been proving difficult as we implement more and more features, adding extra strain to databases, web servers and budget. One of the major problems we kept seeing was overloaded CPUs, bottlenecked Databases and huge ammounts of bandwidth being consumed, so a new plan was needed. Over the next few weeks I'll be posting articles on how to save money, improve performance and even improve security of your sites and applications using <a href="https://www.varnish-cache.org/" target="_blank">Varnish Cache</a>.

One of the main reasons I am so inclined to use it is the fact Facebook have chosen to adopt it to power some of their features, so it obviously scales rather impressively. Case studies will include things like Multiple Backends, Traffic Routing, Proxying, Load Balancing, Static Caching and maybe even building a personal CDN service for readers working with clusters and large amounts of static content. Feel free to contact me if you have any questions about Varnish or other server performance enhancing techniques. I've already started using it on a few sites, and I can see performance improvements already with minimal configuration.

Varnish Cache Accelerator Introduction

This is a quick post explaning how to quickly install the CURL module on PHP5 in preperation for my next few posts. Below is the code to run from the command line. Please make all your packages are up to date and <a href="https://marcqualie.com/2011/11/upgrading-to-ubuntu-1110-using-command.html" target="_blank">upgraded</a> to the latest version before continuing. This isn't essential, but I always reccomend doing this before installing new software. Now the actual code.

``` bash
sudo apt-get install php5 openssl php5-curl
```

I included openssl above, too. This is because you may want to do requests to https resources and this is required to do so. I realise this post is short and sweet, but please be assured it's only a pre-post for what is to come in the next few days.

Installing CURL on Ubuntu with PHP5

Here is a very simple script to update from Ubuntu 11.04 to 11.10. I have just launched a few new servers and this was the first thing I had to do on all of them, so I thought I'd share.

``` bash
apt-get update
apt-get autoclean
do-release-upgrade
```

Some of the above code might not be needed on a fresh install, but I always like to make sure the packages are all up to date before upgrading to a new distribution. This will work on both Desktop and Server editions of Ubuntu 11. Also, remember to use <code>sudo</code> if you're not logged in as root.

Upgrading to Ubuntu 11.10 using Command Line

This is a quick post demonstrating the quickest way to get <a href="https://nodejs.org" target="_blank">node.js</a> running on Ubuntu 11.10. I've tested this on both Desktop and Server editions, aswel as 11.04. From my experience, this is the quickest way to guarantee your installation of Node to be a success and no errors will occur, and install all required packages before hand.

Simply put this code into a shell script, and make it executable so you can easily install/update your Node installation from one simple command.

``` bash
apt-get install -y make git git-core g++ curl libssl-dev apache2-utils python
git clone https://github.com/joyent/node.git && cd node
./configure && make && make install
```

There you have it. A brand new Installation of Node. You can test that everything is working by checking out the examples in my <a href="https://marcqualie.com/2011/11/introduction-to-nodejs-building-web.html">Introduction to Node.js</a> post. Contact me if you have any feedback, or have an problems getting set up! Update: I also added a ready made script that I will keep updated with the most current and best way to install Node. Simply run the command below to install.

``` bash
curl https://www.marcqualie.com/labs/node/install.sh | sh
```

Installing NodeJS on Ubuntu 11.10

Lately I've been looking into different technologies for making real time communication better for my sites, games and apps. <a href="https://nodejs.org" target="_blank">Node.js</a> has been around for a while now, but it still doesn't seem to be picking up any traction in terms of it becoming widely adopted or used in big applications as far as I can tell. I've wanted to use it for quite a long time now, but never had the time. Lucky for me, I will be working on a project at work which requires real time communication so this is the perfect time to learn node and share my experience!

My first annoyance when it came down to getting into the code was lack of examples and useful documentation. This wasn't much of an issue once I got used to the syntax; I could just figure things out logically and break it until it worked. Let's start with the basic hello world example. Firstly, create a file called helloworld.js and put the below code into that file. Next you will want to run <code>node helloworld.js</code> from your console to start listening for connections.

``` javascript
var http = require('http');
http.createServer(function (req, res) {
 res.writeHead(200, {'Content-Type': 'text/plain'});
 res.end('Hello World\n');
}).listen(8080);
```

This example is also available from the nodejs website, however I modified it to listen for external connections instead of just locally since I develop on an internal server, and not localhost. As nice as that is, you're going to want to do more that just say hello to your users. Next we will start playing with the file handling features of node to create our own simple version of an Apache web server.

However, before we jump in at the deep end and start making full blown servers, let's start with the basics of file handling. Below is a nice and simple Hello World demo, with a twist of reading the output from a .html file.

``` javascript
var http	= require('http'),
	fs		= require('fs'),
	url		= require('url'),
	path	= require('path');
http.createServer(function(req, res) {
	var filename = '.' + url.parse(req.url).pathname;
	if (filename == './') filename = 'index.html';
	path.exists(filename, function(exists) {
		if (!exists) {
			res.writeHeader(404, {"Content-type": "text/plain"});
			res.write('Error 404: File not found');
			res.end();
			return;
		}
		fs.readFile(filename, 'utf8', function(error, file) {
			if (error) {
				res.writeHeader(500, {"Content-type": "text/plain"});
				res.write('Internal File Error');
				res.end();
				return;
			}
			res.writeHeader(200, {"Content-type": "text/html"});
			res.write(file, 'utf8');
			res.end();
		});
	});
}).listen(8080);
```

The above code simply checks the URI you've specified, and looks for a file with that name in the current directory, then displays it to the user.

This next example is rather complex so instead of posting the code here I decided to host it on <a href="https://github.com/marcqualie/node-examples" target="_blank">Github</a> so you can access and download the code. As you read through the example you will see I've implemented a very basic templating engine, which is the start of a project I wish to pursue. Basically the goal will be to allow people to create web pages without every needing anything other than JavaScript and HTML knowledge, including interaction with the backend including database and file management! If you would like to take the code, or contribute to it then you can access the code on <a href="https://github.com/marcqualie/node-examples" target="_blank">Github</a>.

Here is a small example of the template system, in case you're interested without viewing the entire project

``` html
<ul class="files">
<node:files dir="/images">
	<li><a href="<file:url/>" title="<file:name/>"><file:name/></a></li>
</node:files>
</ul>
```

The above code will be parsed by node and a list of files will be printed using the above enclosed markup. Pretty simple, but allows you to create powerful client side displays with simple html markup. With a little though I'm sure you could come up with something a lot more exciting with the posibilites of a node templating engine!

Introduction to Node.js: Building a Web Server

I'm working on a simple game to refresh my canvas knowledge. I decided to built the game entirly in JavaScript, which isn't easy to do due to the fact it isn't designed for that purpose. There are no nice little helpers like drawing, object rendering, and most importantly collision detection. I've decided it would probably be easier to use an engine, but since you all know I hate using libraries or engines unless really necessary, I've decided to go about doing all of the code entirely myself from scratch. After a bit of playing around, and some memories from maths lessons in school I figured out how to do collision detection in raw JavaScript code, which I'm going to share with you

First of all, I'd like to point out that since JavaScript is nothing like Actionscript in the sense that there isn't a stage, I am using complete code logic to move my objects around, then rendering them separately by drawing them out onto the canvas using virtual coordinates as reference points. I'm going to write a lot more tutorials on game development with JavaScript very soon, as a whole series; but for now, here is the collision code.

``` javascript
if (
 obj1.x <= obj2.x + ((obj1.width + obj2.width) / 2)
&& obj1.x >= obj2.x - ((obj1.width + obj2.width) / 2)
&& obj1.y >= obj2.y - ((obj1.height + obj2.height) / 2)
&& obj1.y >= obj2.y - ((obj1.height + obj2.height) / 2)
) {
 console.log('Hit!');
}
```

I'd like to point out that the above code only works on rectangular based objects, but if done right, it's pixel perfect. Feel free to suggest any improvements, or ask for any more help to do with JavaScript Game Development.

Pixel Perfect Collision Detection in JavaScript

Here is a quick snippet for updating the time zone on Ubuntu Servers via the command line. This will be part of the series I'm writing on getting set up as fast as possible with an Amazon EC2 Instance running Ubuntu Server.

``` bash
echo "Europe/London" | sudo tee /etc/timezone
sudo dpkg-reconfigure - frontend noninteractive tzdata
```

You can check if this has worked by typing date into the command line, which should show the correct date and time for your location. If for some reason the time is wrong, make sure you've typed the right timezone, or manually update your server's clock using:

``` bash
date -s "08/22/2011 14:25:00"
```

Hopefully this will save you some time wondering why your times are wrong on your PHP scripts, or Cron scripts run at obscure times.

Updating Timezone on Ubuntu Server

Recently I've been thinking a lot about open source, and whether or not it's a good idea for me. I can see why it benefits groups of developers wanting to collaborate on a project, build for the community or draw attention to an individual as a developer. I've been trying to get into services such as <a href="https://github.com/marcqualie/">Github</a> to get a better idea of what's happening in the Open Source development world. Here's just a few thoughts on what I'm planning in the coming months.

First of all, I really want to release the code to a few of the projects I use internally but my biggest worry is that people will come along and pass off my code as their own. I've come to realise there isn't really a way around this, no matter what line of work someone is in. If something is available, there will be someone, somewhere that wants to steal it. Luckily, this only accounts for a small minority of the people I'm trying to attract.

Most of my projects have some sort of financial gain for me, such as my <a href="https://marcqualie.com/projects/css-compressor">CSS Compressor</a>, and by releasing the source code I'm essentially risking the chance of loosing potential income. However, I'm not interested in making direct money from some of work my work; Instead I simply want recognition and more work to put on my portfolio, which to be honest, if quite bare right now.

I'm slowly releasing projects one at a time to my Github account, but it's a time consuming process and it's hard to find time. Another option is to collaborate on an already existing project, but I'm finding it hard to find anything I'm interested in that doesn't already have the features I'd want. Hopefully I'll find more time to update more projects, but for now you can check out my <a href="https://marcqualie.com/projects/css3-buttons">CSS3 Buttons</a> and <a href="https://github.com/marcqualie/feed-parser">Feed Parser</a> for an idea of what I'm working on.

Feel free to request any projects, or suggest any ways I can get into the open source community a bit easier based on your personal experience.

Moving to Open Source

This week a colleague needed to work from home to finish up some code for our new internal site launch next week. Unfortunately, since we develop on Linux at work, and he uses windows at home, this wasn't such a smooth process of getting the files to render correctly on the Windows platform. He was using <a href="https://www.apachefriends.org/en/xampp.html" target="_blank">Xampp</a> which doesn't come pre configured with quite a few components we use to keep our site running fast and stable. One of these programs was Memcache; a distributed memory object caching system. Below I will show you the steps I took to get Memcache working on windows with the latest version of Xampp (1.7.4 at the time of writing).

First of all, you're going to need to download the files which I am hosting below. The zip file contains the Memcache service, a module for PHP 5.3 and some .bat files I created to run it as a service.

<a href="https://dl.dropbox.com/u/3471240/memcache.zip" target="_blank">https://dl.dropbox.com/u/3471240/memcache.zip</a>

Once you have the files, unzip them, you are going to be moving them around your system. First of all, you can put your memcache.exe and .bat files anywhere, but I normally put mine in C:\Program Files\Memcache just because that's where windows normally stores things. Once you've copied those files over, run install.bat and this will install Memcache as a service that starts with your computer. Only use this option if you plan to develop a lot on this machine, other wise there is no need to have it run on startup. Another option is to create a desktop shortcut to the .exe file and just click it when you need to run memcache related programs.

Up until now memcache will work, if you already have your language of choice compiled with a memcache module or plugin you will be able to connect to it on port 11211 like normal. If you don't have this, I'll show you how to get php on xampp working. You will need to <a href="https://www.apachefriends.org/en/xampp-windows.html#641" target="_blank">download Xampp</a> if you haven't already. This has only been tested with version 1.7.4, let me know if it works with any other versions you have.

Next we will be install the php module, this is where things might get a little tricky if you're never done this before. Navigate to your Xampp directory, mine is located at C:\Xampp. Inside there will be a php folder, open up a file inside there called php.ini. Find a part in the file where there are loads of lines starting with extension= and add this line at the bottom:

```
extension=php_memcache.dll
```

Once you've done that there will be a folder called ext inside your php folder. Copy and paste the .dll file included in my download into here. Once you've done that you're ready to restart Apache. This can be done via the Xampp Control Panel or via the normal window's services area.

That's all there is to it. Simple, right? If you have any problems you may have mistyped something in your php.ini, so double check it and try resaving. You can test your install with this php snippet:

``` php
$mem = new Memcache;
$mem->connect(&#8216;localhost', 11211);
$mem->set(&#8216;test', &#8216;Hello, World!');
echo $mem->get(&#8216;test');
```

If the code above doesn't output Hello, World! onto the screen then something is wrong. Make sure the Memcache.exe file is running if you haven't installed as a service, or else php won't have anything to connect to.

You can always contact me if you need help with anything, I always reply to my emails and I'm happy to answer any questions you may have. Comments work too.

Installing Memcache on Windows

This morning, I just stumbled across a cool site which teaches people who to code, without it being an entirely tedious task. The site tries to engage people by actually letting them code in a console window while they are learning, getting feedback as they progress through the stages. The site is <a href="https://www.codecademy.com/" target="_blank">codecademy.com</a> and I advise you try it out if you're wanting to get into basic coding, or you just want to play around with an interactive console window for a while.

Codecademy: Learn to Code the fun way

I recently just setup a new SSL Certificate on <a href="https://marcqualie.com/">marcqualie.com</a> and ran into a little glitch that wasn't explained in the setup process, but luckily I already knew how to fix as I've installed these many times before. When you generate the .key file you must put a password on it, which is a good idea, but impractical for use on an Apache server. All goes well until your restart, during which is asks for the password before the server can start. This is a huge problem, especially if your server crashes, or restarts and there's no one around to type in the password, your entire site will be down because the Apache process will be locked up waiting for this password. Luckily there is a simple workaround. Make sure you backup your original key first, just in case!

``` bash
cp server.key server.pass.key
openssl rsa -in server.pass.key -out server.key
```

It really is as simple as that. Next time you restart Apache you won't be prompted for the password. Hope this helped someone out. Keep checking back for new snippets, or feel free to ask me for help.

Removing Password from SSL Key

Today I woke up with the vision of watching <a href="https://www.amazon.co.uk/gp/product/B004CYECF4/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B004CYECF4" target="_blank">Dexter</a> while working on my laptop because I'm sick of sitting at my desk. This left me with 4 options; Stream from Desktop, Stream from Laptop, Use USB Stick or Stream from new Linux Server. Each of these options has it downfalls, and one best solution, which I'm going to describe below.


#### Option 1: Stream from Desktop
The main reason I don't want to do this is because this computer consumes a lot of power due to the graphics card and high wattage power supply. Even though I've already setup streaming from here, I'd rather not use it and reerve it only for gaming.


#### Option 2: Stream from Laptop
Since I'd be working from my laptop, I don't want to drain the battery by streaming the content from here. Simple as that, no other reason other than battery life.


#### Option 3: Use a USB Stick
This is my usual method for watching my media on the TV, but it's very annoying. I have to manually copy the files to the USB Stick each time I want to watch something new, which is very inconvenient.


#### Option 4: Stream from Linux Server

This is my ultimate goal, since I recently just bought a new <a href="https://www.amazon.co.uk/gp/product/B004FPOLCO/ref=as_li_ss_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=19450&creativeASIN=B004FPOLCO" target="_blank">HP Proliant Microserver</a> that I plan to always have turned on, which is now hosting all of my files and working as my internal web server and test environment. Unfortunetly Linux doesn't have native support for streaming to Xbox so I had to do some researching. Turns out there are a few options, none of them seeming to work from the comments I was reading, or there would be an odd occasion of success on certain operating systems. After an hour or so of searching I decided to use my instinct and go with <a href="https://ushare.geexbox.org/" target="_blank">ushare</a>; A free UPnP A/V & DLNA Media Server for Linux.

To my surprise, ushare is actually part of the Ubuntu packages, so this makes things extremely easy. I am running Ubuntu Server 11.04. You can simply run the command:

``` bash
sudo apt-get install ushare
```

I haven't tested this on Fedora based systems like CentOS, but I've read the RPM installer works just the same via the yum command. Ushare stores it's config file in /etc/ushare.conf so this is our next stop. Luckily, there are only a few slight modifications you have to make to this file which I've listed below.


```
USHARE_NAME=SERVER
USHARE_IFACE=eth0
USHARE_PORT=49200
USHARE_TELNET_PORT=1337
USHARE_DIR=/storage/media
USHARE_OVERRIDE_ICONV_ERR=
USHARE_ENABLE_WEB=yes
USHARE_ENABLE_TELNET=no
USHARE_ENABLE_XBOX=yes
USHARE_ENABLE_DLNA=no
```

That is the exact configuration file I used and it worked straight away. I simply saved the file, started the daemon and turned on my Xbox. All my media files were instantly there, no errors, no messing around with configurations. You can easily start and stop the daemon using:

```
/etc/init.d/ushare start
```

Hopefully this will work as easily for you as it did for me. Feel free to comment or ask questions if you have trouble or any suggestions.

Streaming Media from Linux to Xbox 360

This is just a quick post to hopefully help you out if you're having the same problem as me. I recently purchased a <a href="https://www.amazon.co.uk/gp/product/B004FPOLCO/ref=as_li_qf_sp_asin_il_tl?ie=UTF8&tag=marqua-21&linkCode=as2&camp=1634&creative=6738&creativeASIN=B004FPOLCO" target="_blank">HP Proliant Microserver</a> and decided to go with Ubuntu Server 11.04 instead of my usual CentOS distro. It took a while to get used to but I got everything up and running within a few hours and everything was working fine; until I wanted to shutdown!

I do realise this is a server, and shouldn't be shut down rechnically, but since it's a home machine for my personal use, and has wake on lan functionality I don't see why I should keepit on while I'm sleeping, unless I'm processing lots of info, which will be a post in the future!

Basically the problem I was having was when I used the shutdown command via ssh, the machine would shutdown, but not power off.. meaning I couldn't get back into SSH or do anything really, and had to hold down the power button.. which if isn't good for the machine and is impossible if I'm away from my desk.

Luckily this was a very easy fix, even though it took hours of Google searching! I was using the following command:

``` bash
sudo shutdown 0
```

when I should have been using:

``` bash
sudo shutdown -h now
```

The only real different being that I didn't use the -h parameter. As far as I can tell, this tells the system to halt then power off where as the previous command only halts the system. What is the point in that? Anyways, I hope this helps someone. Let me know if you have any other tips or suggestions?

Shutdown Ubuntu Server 11.04

For way too long now I've taken advantage of the amazing bind method build into the Mootools core framework, so when I started using jQuery for a client, I was shocked at how basic the code was in comparison. I'm going to write a whole new article on that topic tomorrow, but for now my focus is object binding. I'm going to explain what it is, where to use it, and how to get it to work in jQuery without breaking anything!

If you haven't already checked out the <a href="https://mootools.net" target="_blank">Mootools</a> framework, please do. It's an amazing library that I've used for many years, and swear by when coding complex applications. As I said above, a main feature I make use of is the ability to bind objects to functions as I call them, which helps a lot when creating custom classes and objects that are linked together, or I'm not sure what it's name space will be.

In Mootools, the syntax is as follows:

```
element.addEvent(&#8216;click', clickFunction.bind(thisObj));
```

The <code>thisObj</code> variable is a variable that you wish to be accessible through <code>function.this</code> from within the function. Here, I am assuming you have JavaScript knowledge and know what the <code>this</code> variable is. I may write some tutorials on core JavaScript later this month, but for now you're just going to have to learn JavaScript the old fashioned way; Google!

Unfortunately, this functionality isn't available in jQuery, but luckily it is baked right in the JavaScript Prototype Core as long as you know what you're doing. JavaScript is a very powerful language, that most developers don't even know it can do half of the things it is capable of. My favourite feature is it's <a href="https://en.wikipedia.org/wiki/JavaScript#Prototype-based" target="_blank">prototype-based</a> style of coding that is available, but rarely used by day-to-day developers. I'll let you do you're own reading for the full details,. but basically this allows you to inject custom code right into the core of the JavaScript language and add extremely powerful functionality.

Here is a basic example of prototyping.

``` javascript
String.prototype.displayAlert = function() { alert(this); };
("This will be displayed via alert").displayAlert();
(".. and this will be displayed too").displayAlert();
```

What happens above, is you are injecting the displayAlert function onto the String object of the language, meaning any object with this type has access to the function you created. Pretty neat, huh?

Back to my main topic, object binding. You can use prototype to bind methods to all functions too, so I will use the code below (which I took from Mootools Core and modified slightly) to modify the this variable, without needing Mootools!

``` javascript
Function.prototype.bind=function(a,b){var c=this;return function(){if(!b&&!arguments.length)return c.call(a);return c.apply(a,b||arguments)}};
```

I've trimmed the code down for you to copy and paste straight into your projects, and it can be used exactly the same way as you use the Mootools bind function I displayed above. Obviously if you already code using the Moootols framework, you don't need this function. I made this for use in projects without a Framework, or a none prototype based Framework such as jQuery.

I hope this helped someone by opening theirs eyes to a new way of coding or even by simply bringing a cool feature of Mootools into the jQuery Framework!

Advanced Object Binding in JavaScript

This is a quick post to inform you of some ways to keep your Linux machines well protected from hackers and bots. Today I realised some of these so called hackers had been brute forcing my web servers for the last few days; luckily they didn't prevail but it got me thinking. I'm going to tell you the steps I took to secure my server even further than it already was, and hopefully help you avoid the same troubles.


#### Deny Root Logins

This might seem like an obvious one, but most servers allow root logins by default. There are no reasons that anyone should be remote logging in using root account. Leaving this account open for logins is very dangerous and allows for easy brute force attacks to guess your password. The best part about this option is it's extremely easy. Use your favourite editor to open /etc/ssh/ssh_config and change PermitRootLogin to no. Make sure at least one other user on your machine has SSH if you are currently logging in as root as you won't be able to log back in after restarting the SSH daemon otherwise.


#### Limit Login Attempts

By limiting the amount of failed authentication attempts you are stopping bots from repeatedly attacking your connection and maybe eventually guessing your password. Again, this is a very easy change. Inside /etc/ssh/ssh_config change MaxAuthTries to 5. Any more than 5 wrong password attempts is almost definitely a hacking attempt, don't set the number too low or if you accidentally forget your password, or miss a key or too while typing like a maniac you may find yourself locked out of your own machine.


#### Use Keys instead of Passwords

If you're new to linux you may not know what keys are, but trust me, you want to. Keys allow you to login to your server without a password. "OH NO!?" I hear you say? Thankfully it's not as simple as that, access will be permitted to users who posses your private key file, which you must guard with your life! No need to remember long complex passwords, although it's still recommended, and it means hackers can't ever guess your password because your machine won't even be checking for them without this special key. First of all you're going to need to generate a key pair. Simply run the command below

``` bash
$ ssh-keygen -v -t rsa
```

You will see an output similar to:

```
Generating public/private rsa key pair.
Enter file in which to save the key (.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
18:c3:71:2d:56:13:e7:ff:34:e5:19:f8:2e:af:f3:94 user@localhost</div>
```

Make sure that the keys you generated are in your home directory inside the .ssh directory. Notice that you are allowed to enter a blank password. This is nice and convinient, but I highly recommend against it, even if it's a simple 8 character password. This is because if your key were to end up in the hands of someone unfortunate then your server's security is now compromised as they can now login as you without a password!

Now to get your key onto your server. The easiest way to do this is to simply inject your new public key into your server's authorized_keys file.

``` bash
$ cat id_rsa.pub | ssh user@host "cat >> ~/.ssh/authorized_keys"
```

The above command will put your newly generated public key onto the server, meaning you can now login with your private keys. Give this a quick test before you disable password logins to make sure it works.

``` bash
$ ssh user@host -i id_rsa
```

If that works and you can login then you can go ahead and modify your <code>/etc/ssh/ssh_config</code> file with the below options.

```
PasswordAuthentication no
ChallengeResponseAuthentication no
```


#### Enforce Password Protected Keys

Unfortunately there isn't a way on the server to force using password protected keys, so as the server administrator you must enforce this yourself. Make sure that any user with access to your machine uses a strong password on their keys or else your machine is only as secure as your users private keys and where they are stored.


#### Change your port number

Another good way to defend against basic attacks is to change your port to something other than 22. Most bots and hackers will only try to attack this port and ignore others, but you can get around this by changing yours. Simply change <code>PortNumber</code> to any number above 1024 as ports below this are reserved for other services. Make sure you update your firewall to allow this new port or else you'll be blocked out, so be warned!


#### Summary

In all honesty, you will never be 100% secure against hackers if they are determined enough, but you can really make it harder for them and always keep on top of patches and check your logs regularly. There are also tools like DenyHosts which help blacklist hackers before they even get to your server. Feel free to post your own security tips in the comments, or suggest anything I might have missed in my article.

Improving SSH Security on Linux

I recently just dusted down an old laptop to use as at work, and I felt Ubuntu was the smart option since I'm becoming very fond of Linix lately. I also want all of my files to be my own, and not be on my work machine when I'm not there (a whole new Blog post on network mounting coming soon). However, since I installed 64-bit version of Ubuntu 10.04 I ran into an issue when it came to using Flash Player in Chrome.

Basically my problem is that I had to install 64-bit version of Chrome .deb package, which doesn't support 32 bit flash plugins. Annoying. Luckily there is a native Flash Player build that can help out, it's called <a href="https://labs.adobe.com/technologies/flashplayer10/square/" target="_blank">Flash Player Square</a>. Problem is, it's not very helpful on how to Install. I assume most Linux users would already know, but since I'm new to Ubuntu and Linux having a GUI then I wasn't sure what to do with the .so file. Turns out it's very simple.

All you have to do is copy libflashplayer.so file into the <code>/usr/lib64/mozilla/plugins</code> directory. Once you've done that simply restart chrome and visit <code>about:plugins</code> in your address bar and you should see Flash is enabled. Seriously, that's all you have to do. Personally, I was a bit let down because I love the challenge of hacking things and this way too simple to think possible! Looks like I'm going to have to find something else to hack to finish up my weekend; Triple Gateway Secured Proxy, Personal SVN Server and Spotify on Linux? Sounds fun..

Installing 64-bit Flash Player on Ubuntu

Today I decided to build myself a small, yet very secure proxy tunnel to use at work. Not that I have anything to hide, I'd just prefer my boss not to know what sites I browse in my lunch hour, or get curious as to why I enjoy watching stand up comedy while I work. Luckily most of the services I use are https enabled anyways, but for the few that aren't, this is my workaround.


I'm going to start off by saying this option isn't cheap if you don't have a dedicated machine at home, which a lot of people don't, and you wish to host this service on a 3rd party. You have a few options and I will outline them all below, letting you decide which is best for you. You may also be asking "can't I just use a free proxy website?" and my answer would be "only if you want do the opposite of secure your web data". Although using free web proxy's get around things like content filters, but they also have the option of looking at any data that passes through them, and won't be secured.

Step 1. Running entirely  from an always-on home PC or Remote Server
This is the cheapest and easiest way to set-up a secure proxy. You will need an SSH Client if you're using windows, but on Linux you can do this via the command line. On your home machine you will need to have a Linux install. I'm sure this can be done with windows too, but this tutorial focuses on Squid using the Linux User login system. You can install Squid using yum install squid (apt-get install squid for Debian). Once you have that installed you're going to need to change the configuration files a little. nano /etc/squid/squid.conf should do the trick. Once you're in there you're going to want to give yourself access to your IP. Since we're going to be logging in with SSH we don't need to add a username/password to the config right now; just make sure you don't tell anyone your host or port and you will be fine with this method. Adding acl users src x.x.x.x where x.x.x.x is the IP address where you want to be able to access the proxy. You can add as many lines as you like for different IPs or IP ranges. By default Squid already has access from local address's in case you need to do internal routing. Now you will need to add http_access allow users to your config file. Save your changes then start Squid (/etc/init.d/squid start). All done, however very basic, you can play around with Squid yourself later on.


Step 2: Setting up a SSH Connection from your client machine
Now comes the part where you connect from your local machine. Personally I am using Putty on Windows 7 as I like my games too much to switch fully to Linux for personal use. One day when Linux supports Steam properly. or I switch entirely to Xbox 360 then maybe I'll run Linux are my primary OS, but for now.. Windows 7! Inside Putty you're going to want to create a new connection like you normally would to your server. This is no different to a standard SSH log in, whether it require username/password and/or private key files. Once that's done, there's one extra thing you have to do. Inside the "Auth" directory, you will see a section called "Tunnels". In here you must add the port you wish to listen locally (we will use 8080) in the first box, then your host:port (default Squid port is 3128) configuration in the next box. Once you've added that, save your session for easy access next time. Click connection and login!


Step 3: Modifying your connection settings in your browser
I use Google Chrome as my primary browser, but many of you may not. Changing proxy settings can be done either via your actual connection on your network, or within the browser settings panels, which are pretty easy to find in all browsers. For ease of use I am going to refer you to <a href="https://chrome.google.com/extensions/detail/caehdcpeofiiigpdhbabniblemipncjj" target="_blank">Proxy Switchy!</a>, which is an amazing tool for quickly switching between multiple proxy configurations. In the host section you will want to put localhost, and for port you will put your local listening port (I used 8080 in my example). Activate your proxy and navigate to a website. If you setup everything correctly you should see the website load fine. If not then double check everything is running and your firewall is allowing access to the right ports.

I realise this tutorial is very basic, but it's almost 3am here and I've had a very long day! If you have any questions at all please leave them in the comments below and I will do my best to assist you! If you want a more simple set-up you can use another tutorial I wrote a few weeks ago using <a href="https://marcqualie.com/2011/02/secure-proxy-with-hamachi.html">Hamachi and Privoxy</a>. This is just another way of doing what I mentioned above, with less configuration and more likely to work straight off.

Secure Proxy with Squid and SSH Tunnels

Hey guys! It's a while since I posted, but it's all for a good cause, I promise. Recently I joined forces with a friend and we become a single identity known as "The Mole".

"Why The Mole?", I hear you you ask; well it's simple, we just launched <a href="https://molehillgaming.com">Molehill Gaming</a>!

Molehill Gaming is a new Blog, Portal and Forum for all things related to the Adobe Molehill API. If you don't already know what this is then head over to the site and check out some of the amazing demos! We update the site daily with new content and our forums are a great place to get involved in the community if you're a flash developer or an amazed onlooker.

Molehill Gaming Launched!

I really want to make something like this so I thought I'd share.

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/sLVXmsbVwUs?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Laser Beam Techno Harp

Today I received my Kinect Cable (oh yeah!) so I could start playing around with some hacks. Unfortunately, I was in a lazy mood, so Instead of learning the C#, Java or Python versions of these SDK's, I went with the Actionscript version since I am much more fluent in coding this. Turns out, FlashDevelop doesn't have TouchEvent!

If I compiled my code using Adobe Flash CS5, it runs fine; If I run my code using FlashDevelop I get a nasty "Error: Definition flash.events:TouchEvent could not be found". This was a problem for me because I don't like using Adobe Flash IDE. I prefer to make all of my projects using only code and embedded assets; mainly because it compiles faster and I prefer the interface and auto complete performance.

Now to my point, which is why you're here, right? Flex SDK 4.1 does not support TouchEvent for some reason, however the Actionscript 3.0 library included with Flash CS5 does. If you want to enable Touch Events in FlashDevelop simply copy <code>C:\Program Files (x86)\Adobe\Adobe Flash CS5\Common\Configuration\ActionScript 3.0\FP10\playerglobal.swc</code> to your project folder and select "Add To Library" from FlashDevelop. I hope this helps someone save some time, it took me quite a while to figure this out.

P.S. The OpenKinect Actionscript library has TouchEvents in for some reason, and that's why I was looking for this. Basically I was too lazy to modify the code to take the Touch Events out, but I will write my own library in the future.

TouchEvent in FlashDevelop?

<a href="https://2.bp.blogspot.com/-1fRyZ3kk9d4/TWXAu99XvCI/AAAAAAAAA2c/XBQBNn1uXKs/s1600/logmein_hamachi-437849-1293507014.jpg">
 <img border="0" src="https://2.bp.blogspot.com/-1fRyZ3kk9d4/TWXAu99XvCI/AAAAAAAAA2c/XBQBNn1uXKs/s1600/logmein_hamachi-437849-1293507014.jpg" />
</a>

Recently I read an article on how to<a href="https://lifehacker.com/#!5763217/how-to-secure-and-encrypt-your-web-browsing-on-public-networks-with-hamachi-and-privoxy" target="_blank"> secure my public browsing</a> using LogMeIn Hamachi Client. I was using Hamachi for a while prior to this to to Sync my work network with my home network for easy file sharing. I also use LogMeIn free to access my PCs and I've been very happy with the service. I want to show you how you can have your very own provate tunnel back to your home or work network for secure browsing, no matter where you are. I won't go into detail on what this is, or how to set it up on windows (as the article is very in depth). I will be demonstrating how to get this same system running on a CentOS 5 machine.

I have this setup on both CentOS 5.4 and Amazon AMI 1.0, which is the default for basic AWS linux servers. If you don't know what any of the above means, or you think CentOS is a cheaper version of DollarOS then this tutorial isn't for you! Let's get started by installing Hamachi. Unfortunately you can't simply use <code>yum install hamachi2</code>, so we have to a manual install, which isn't too difficult. After a few hours of messing around with different server configurations I found the code below will work on all systems and is the quickest way to install and get your server running.

``` bash
sudo yum install lsb
wget https://secure.logmein.com/labs/logmein-hamachi-2.0.1.13-x86.tgz
tar xfz logmein-hamachi-2.0.1.13-x86.tgz
cd logmein-hamachi-2.0.1.13-x86
sudo make install
sudo ./hamachi-init
sudo /etc/init.d/logmein-hamachi start
hamachi set-nick yourservername
hamachi join ***-***-*** proxickle
```

You will want to replace ***-***-*** with your actual network ID which you created, and yourservername will be the name you see in your remote Hamachi Client.

Next you will want to install privoxy on the server where you are running your proxy network.

``` bash
sudo yum install privoxy
sudo nano /etc/privoxy/config
sudo /etc/init.d/privoxy start
```

Luckily privoxy is in the Fedora repository so you can use yum and life is very very easy. You can modify the config files if you want, but the defaults work just fine, and don't need changing to get you up and running. Now all you do is follow the original article and connect to your new server using your Hamachi client.

I hope this helped someone, you can always leaves comments or contact me privately for any information or questions you have.

Secure Proxy with Hamachi2 on Linux

Today, I installed <a href="https://varnish-cache.org/" target="_blank">Varnish Cache</a> onto my Amazon EC2 server to test drive it's performance. I had recently discovered sites such as Facebook are using Varnish as a way of delivering the billions of photo requests they receive daily; which must mean this thing can really take the load of of standard http servers. Unfortunately I was unable to find a decent tutorial on how to do so, which led me to write my own.

To start with, you should know that I'm currently working with the default Amazon AMI install, which is pretty much a basic CentOS install as far as I can tell. Things shouldn't be much different for other version of Linux, such as Ubuntu or FreeBSD. I should also note that Varnish is currently not compatible with Windows platforms, which is annoying for me as I use a Windows test server at home, and at work.

Let's get started. First of all you're going to have to configure your Apache server to run from a different port. The examples below show port 8080, but you can choose any port you like.

```
Listen 8080
<virtualhost *:8080>
 ServerName marcqualie.com
</virtualhost>
```

Installing Varnish Cache on CentOS

A collegue brought up an interesting subject today at work; how do you detect which radio button out of set has been selected in JavaScript? This is the first time in a few years I've been stumped on a JavaScript question, so I was quite annoyed! The last time I used a radio button must have been on a MySpace voting pole widget back in the day. So I had my challenge, and I was determined to find out.

My first attempt was to access the button via it's name inside of a form, but this just seemed to "IE6" to me. Then I started thinking about IDs, but then each button would need an id, and that's just overkill. Surely there is a more simple way then a reference to the form and looping through each input with the name then checking for .selected==true; Luckily, thanks to <a href="https://mootools.net/" target="_blank">Mootools</a> there is!

A month or two I upgraded the company website to use the latest stable version of Mootools 1.3, and I'm so glad I did! This version of the framework includes the Slick Parser, which has to be one of my most loved and life saving pieces of script, which I use on a daily basis. I started to think of radio selection as an event, or a state, rather than a variable. Many of you reading this will know that Slick supports <a href="https://www.w3.org/TR/selectors-3/" target="_blank">CSS3 Selectors</a>, which includes pseudo events.

``` javascript
$$('#formid input[type=radio][name=test]:selected')[0]
```

Thanks to pseudo, I was able to grab the selected radio button via the above code. That snippet will get the reference to the radio input with name "test" in the form with ID "formid". There are many possibilities what you could use this for, but this is just an example. Hope this helps someone save some time.

Detect Selected Radio Input in JavaScript [Mootools]

Today I was experimenting with various caching techniques in PHP and Apache to try to speed up my site, while still keep all of my content real-time and efficient. I will be writing a full post on my experiment and my results later on tonight, but for now I just want to write a quick post about APC.

First of all, APC can only run on one server, and can't communicate with other servers in a cluster, or remotely; this makes it useless for scaling, but useful for other scenarios as I will explain. Mainly, APC can only be used for small amounts of data that need to be read extremely fast. As far as I can tell, APC uses the shared memory of the running Apache process, and not it's own server.

My problem with the software is that if you can't access it via cron, unless you use something like curl or wget, which isn't really tidy, but doable. I found out while experimenting earlier, that if you up to the cache using apc_store via a script running from the command line directly in PHP, then an Apache request (standard web request) can't access the data, because it was not ran through Apache. This is quite annoying due to the fact I update my cache via crons so my site is extremely responsive, but I will also write a separate post about that.

All in all, APC is an amazing piece of technology, and I plan to use it a lot in my projects from now on, just not in the way I had hoped. It also speeds up Apache by default once enabled, without having to manually use the cache via PHP. It's a very negligible, but noticeable difference in performance, I wonder how it performs under heavy load?

APC (Alternate PHP Cache) Drawback

Thanks to Adobe's newest release of Flash Player 10.2, web video will use much less of your processor, and improve battery life on mobile devices. Most of video on the Internet is powered by Flash Player, so this is quite a substantial update that we've been waiting a long time for.

The secret to the new improved performance is an element called Stage Video.Users who upgrade to the new version won't see any sort of change unless the website video provider has specifically implemented the new Stage Video; which luckily for you, the major video sites such as Youtube and Vimeo already have started doing. A performance increase of up to 34% is expected from the new update.

Another new feature of 10.2 is the ability to have flash running in full screen on one monitor, while you complete other tasks using another monitor. This might seem like a simple enough task, but any of you who have tried doing this before will have become easily frustrated at the fact the full screen deactivates when any interaction is made on your other screen. Luckily I had a plug-in which counter-acted this, but that was ideal and also had to be re-hacked every time I updated chrome; which is a lot due to the fact I use the developer version. Thankfully flash player now supports this feature natively, and I'm using it right now to watch Youtube videos! Yay!

Flash 10.2 gives web video a Boost

<a href="https://2.bp.blogspot.com/_SCmPP6x1F1A/TTDxXtfAlYI/AAAAAAAAAqM/h-T5c6URj_A/s1600/tczgrande1.jpg">
 <img src="https://2.bp.blogspot.com/_SCmPP6x1F1A/TTDxXtfAlYI/AAAAAAAAAqM/h-T5c6URj_A/s200/tczgrande1.jpg" />
</a>

I've been designing a 3D Flash Gallery that can render .3ds models fully in Flash. The models aren't mine, they belong to a friend, who I will credit when he finally gets his site finished! I'm building the demo to showcase his models, and also showcase the 3D capabilities of Flash. These models are extremely detailed as they are designed for pc games so I'm having quite a bit of trouble getting flash to perform well when animation them. Some of the models between 30,000 - 60,000 polygons per frame which is almost one million polygons per second. Not much these days for graphics cards running pc games, but on 32 bit single threaded virtual sandbox like Flash, it's a whole different world.

3D Car Gallery in Flash

<a href="https://2.bp.blogspot.com/_SCmPP6x1F1A/TTDf1KcvU8I/AAAAAAAAAqE/pYOD1YTby-o/s1600/local.marcqualie.png">
 <img src="https://2.bp.blogspot.com/_SCmPP6x1F1A/TTDf1KcvU8I/AAAAAAAAAqE/pYOD1YTby-o/s1600/local.marcqualie.png" />
</a>

I got bored and decided to change my profile photo on my social networks to a QR code pointing to my website. I've been interested in them for a while, but never been able to find any solid documentation on how to create them. Fortunately I found an <a href="https://phpqrcode.sourceforge.net/" target="_blank">open source library</a> to do it for me! Now I have my very own QR code that you can scan with your mobiles. Not only that, but it looks sort of cool aswel!

My Personal QR Code

We've all met people like this. I meet meet people like this on a daily basis due to the nature of my work and I think they should all be banned from the Internet for being a waste of oxygen. Anyone who I work with will get the reference to the title.

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/fa9DLxDtPtc?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Go To Google, Type in Google..

Below is an article from MSDN, where a Microsoft employee openly mocks [Google's decision to no longer support H.264 video]({% post_url 2011-01-12-google-is-to-drop-support-for-h-264-video-format %}) which I wrote about earlier this week. I found it quite amusing, so thought I would share.

The world’s ability to communicate with one another is a key factor in its rapid evolution and economic growth. The [Esperanto language](https://www.webmproject.org/) was invented last century as a politically neutral language that would foster peace and international understanding. Since the launch, we’ve seen first-hand the benefits of a constructed language:

  - A pure form of communication that is unsullied by cultural context;
  - Broad adoption by as many as 10,000 speakers
  - [Independent](https://en.wikipedia.org/wiki/Esperantido) (yet mostly compatible) dialects that not only bring additional choice for speakers also foster healthy competition and innovation

We expect even more communication between people in the coming year and are therefore focusing our investments in languages that are created based on constructed language principles. To that end, [we are changing the spoken and written language of this nation](https://blog.chromium.org/2011/01/html-video-codec-support-in-chrome.html) to make it consistent with the form of speech already supported by the [Language Creation Society](https://www.whatwg.org/). Specifically, we are supporting the [Esperanto](https://www.webmproject.org/) and [Klingon](https://theora.org/) languages, and will consider adding support for other high-quality constructed languages in the future. Though [English](https://en.wikipedia.org/wiki/H.264) plays an important role in speech today, as our goal is to enable open innovation, its further use as a form of communication in this country will be prohibited and [our resources](https://www.youtube.com) directed towards languages that are untainted by real-world usage.

These changes will occur in the next couple months but we are announcing them now to give citizens using other languages an opportunity to translate the [libraries of the world](https://techcrunch.com/2010/05/01/h-264-66-percent-web-video/) into Esperanto.

**Dankon, nedankinde!**

Source: [MSDN Blog](https://blogs.msdn.com/b/tims/archive/2011/01/11/an-open-letter-from-the-president-of-the-united-states-of-google.aspx)

Microsoft Mocks Google using Failed Esperanto Language

Google has announced that it will be dropping support for the H.264 video format in favour of more open standards. They will be pushing people to use open sourced codecs such as WebM and Theora.

To that end, we are changing Chrome's HTML5 support to make it consistent with the codecs already supported by the open Chromium project. Specifically, we are supporting the WebM (VP8) and Theora [video codecs](https://gizmodo.com/tag/videocodecs/), and will consider adding support for other high-quality open codecs in the future. Though H.264 plays an important role in video, as our goal is to enable open innovation, support for the codec will be removed and our resources directed towards completely open codec technologies.

There are arguments around the internet on whether this is a good move or not. It could potentially make or break Chrome's take off to be the next biggest browser. Currently all browsers, even mobiles (and even flash) support the H.264 standard, however Google's move to change this could mean other browsers have the edge when it comes to online video; one of the biggest parts of the Internet today.

There is also the argument that Flash is built right into chrome now. Not only does Flash support the format, but it is also a closed service, as in it's not open sourced. Could not supporting flash be Google's next move? I doubt it. Needless to say, they seem to be a bit inconsistent when they choose what standards they choose to support, and their reasoning behind them.

Google is to drop support for H.264 video format

I would love to be able to draw like this. This must have taken the guys hours, and it is clearly a masterpiece!

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/5faZrck75L8?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Drawing Minecraft using HTML5

This hacker is using a blender model and Kinect to replicate his movements in real-time. Quite impressive how accurate it is.

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/GCu8KTrC4sc?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Kinect Hack: Real-time Motion Capture

Not really much to say about this one apart from how clearly awesome this thing could be. I'd loved to be able to compose music without ever playing a note on a keyboard or twanging my guitar. It also brings memories of when I used to use FLStudio (Fruityloops back then) quite a lot to make insane dance music. Keep the hacks coming, and also an adapter so I can start making my own! My about that in my next post.

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/tKzVMvosK34?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Kinect hack Midi Controller

As many of you may know, Xbox Kinect has been hacked many times for various different reasons on the PC. Now the millions of gamers can play their favourite Wow experiences using body gestures rather than a keyboard. This enhanced use of Kinect on one of the world's most popular online computer games demonstrates exactly how powerful the future of gaming is going to be. Maybe when the "official" PC drivers are released (as Microsoft have hinted at the possibility of this), more top gaming companies will embrace this new way of controlling games. I am buying my own Kinect in the morning and can't wait to start hacking away at different projects for myself and others.

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/62wj8eJ0FHw?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Kinect hack for World Of Warcraft

Thanks to the Open Source Community over at Apache, Google Wave will now be able to live on in it's new home where many eager developers wait to pick up where Google went wrong. Personally I was quite disappointed when I heard Google wave would be shutting down, as I believed it had much potential. You can read the full story over at the <a href="https://googlewavedev.blogspot.com/2010/12/introducing-apache-wave.html" target="_blank">Google Wave Blog</a>.

Google Wave is now Apache Wave

PHP, while known for it's simplicity and scalability, is an overall slow and memory consuming language. I my 5 years of programming, I've managed to build myself a fully custom and highly optimized set of libraries which I use in a lot of my projects. These libraries range from Geo Location and Authentication, to Sessions Handling and Web Crawling. While these libraries are highly optimized, they still don't have the performance I am hoping for from my applications. PHP consumes high amounts of memory, and isn't the fastest language by any means. I've spent the last few months trying out new ways of upgrading my applications to be more powerful and faster, even learning new languages and techniques along the way.

My first idea was to learn Java. Java, which is cross platform like PHP, is a <a href="https://en.wikipedia.org/wiki/Compiler">compiled language</a>, which means it naturally runs faster due to the fact it doesn't read through the raw code each time a program is run. To my disappointment though, Java isn't as straightforward to develop for as PHP. This is mainly due to my habits, as I use raw text editing for my projects, rather than an IDE. I am sure if I spent the time to learn Java as well as I know PHP, then this would be a different story, but unfortunately I don't have the time. So ultimately, Java is not the option I chose for building the back end to my web apps. On a side note, I also tried installing Apache tomcat to run .jsp files from my server. Since this failed, this was the main reason I decided not to put more time into learning Java, maybe in the future.

Secondly, I decided to go for Python. Python isn't compiled language, but due to it's simplicity, is used for many real-time applications and web apps. Personally, I don't like the syntax and find it a bit dry to work with. Also I couldn't get mod_python working on my server, which meant I couldn't run web apps via Apache. This is another project for a later date.

I then came to realise, I know PHP extremely well. I could write any application straight off without any guidance, documentation or code suggestion. Why should I learn a new language? My answer is simple. I won't! I recently came across a project called HotSpot which is used by Facebook internally to make PHP work for their billions of monthly page views. Their initial answer was to code PHP extensions using C++, which runs as native code, speeding up many aspects of their applications. They then went on to build a project to convert PHP to C++, so that PHP can essentially become a compiled language. People had done this before on a small scale, but never converted a site the size of Facebook.

So my conclusion is this. I will learn small snippets of new Applications for various parts of projects, rather than coding my entire project from scratch in a new language. For example. I will use Java or Python for real-time aspects of the applications. I will use C++ and build extensions for features I use all of the time for things like caching and reading large sets of data. The more you learn, the better you become in every area of programming. Since I learned AS3 I have realized many different ways to optimize my heavy JavaScript applications and vice versa. I will post more information on how to actually put these above tips into practice in the next week. Contact me if you have any questions

Convenience vs Performance in PHP

<a href="https://1.bp.blogspot.com/_SCmPP6x1F1A/TNW_ncfEcnI/AAAAAAAAAjs/tW8Of69sXSk/s1600/Random+Bird+Thing.jpg">
 <img border="0" src="https://1.bp.blogspot.com/_SCmPP6x1F1A/TNW_ncfEcnI/AAAAAAAAAjs/tW8Of69sXSk/s1600/Random+Bird+Thing.jpg" />
</a>

I decided to do something constructive with my weekend. I have a lot of projects lying around unfinished, and random optimizations to do on my sites, so I tidied things up a little. To start with, I upgraded the coding on <a href="https://marcqualie.com/">marcqualie.com</a>, added a share bar and made the site load faster by tidying up loose ends. Then I remembered <a href="https://squigglecanvas.com/">Squiggle Canvas</a>.

For those of you who have used this tool before, you will have noticed it's been offline for quite a while now, and since I missed it, I brought it back. I have huge plans for this project, as I enjoy working on it, and it's useful for all people to use. I am currently working on a fully redesigned system for Squiggle Canvas, however I don't have time to complete it right now, so I have uploaded the old version, with a few enhancements.


#### Changes I Have Made:

- Upgraded to use Mootools 1.2.5 (I was going to use 1.3, as I use that on all of my projects now, however the internal engine would have taken too much recoding)
- Optimized the way the mouse events are handled. I noticed there was quite a bit of lag when drawing on large documents. This was due to the way I was handling mouse events and DOM interaction. This has been fixed, so lag should be minimal in all browsers.
- Google Image Search has been improved so there is no lag when dragging and resizing images. Again, this was to my lazy implementation of mouse event handling.
- You can no longer modify your projects after you have closed them, nor can you modify other people's projects. I have my reasons for doing this, but this feature will be back in the future sometime.
- Server speed and bandwidth increased. Now I can store huge huge drawings for you guys to go wild. I had a few issues with the server hanging when saving massive Squiggle's, but now this isn't an issue.

I think those are the main changes I made, if there are anymore I'm sure you'll find them. For now I'm going to continue working on Squiggle Canvas, and it's many impressive variations, such as a fully integrated Desktop version for all platforms. Try out <a href="https://squigglecanvas.com/">Squiggle Canvas</a> and let me know what you think of it so I can add improvements to the new system?

Squiggle Canvas Returns!

As you may know I <a href="https://marcqualie.com/2010/11/squiggle-canvas-returns">recently relaunched Squiggle Canvas</a>. Whilst re-coding the site I noticed the entire engine was very glitchy, and the JavaScript hang time for drawing was between 500-1000ms. This was the time needed to Chrome to take some input, run a function, draw something to the canvas, update the UI, and get ready to redraw the next line. All of this happens when you move the mouse one pixel.

Many of you may think this is a lot to do every time you move the mouse, but in fact, when programming is concerned, it's actually not much at all, so more than half a second to complete this task is out of order. I ran Speed Tracer and found out the exact cause of the problem. Turns out, any interaction with the canvas was causing huge latency, and the entire page to freeze. I assumed this was my own coding, as Squiggle Canvas was made in a hurry, but being this bad was out of the question. Then I suddenly realised I had enabled one of Chrome's new Experimental features, GPU Acceleration for Canvas. In a lot of cases, this improves performance significantly, and even outdoes IE9 in some areas on the IE9 Demo page.

However on my project it made it entirely unusable. Once I disabled this feature, Squiggle Canvas ran as smooth as porn star's lower regions. Just a word of warning to anyone using Chrome's new experimental features on the Dev channel, they aren't completely stable, and break a lot of applications, including mine. More updates when they make changes this feature.

Canvas GPU Acceleration in Chrome Fails

<a href="https://1.bp.blogspot.com/_SCmPP6x1F1A/TNP5R2St4GI/AAAAAAAAAjk/p9oqwaoCBY0/s1600/url.jpg">
 <img border="0" height="132" src="https://1.bp.blogspot.com/_SCmPP6x1F1A/TNP5R2St4GI/AAAAAAAAAjk/p9oqwaoCBY0/s200/url.jpg" width="200" />
</a>

I'm assuming you have all heard of <a href="https://www.minecraft.net/en-us/" target="_blank">Minecraft</a>, as it's one of the most popular games with nerds on the Internet right now. It's basically a game where you can build and destroy anything in the entire world, with huge, huge maps to explore and create. Follow the link above if you're not sure what it is.

If you've read my Blog before you will know I use Symbolic links a lot to synchronise all of my game data across computers. This works all fine and dandy until Minecraft is introduced into the equation. The game seems to save hundreds if not thousands of files, all of time whilst the game is running. This wouldn't normally be a problem, as it means your data is always saved if the game crashes or you need to leave suddenly (at work for example..).

But when Dropbox is trying to instantly Sync this data across about 5 of my computers all at once, problems begin to occur. Just a word of warning to anyone syncing large sets of data that is updated frequently, not only does this add huge bandwidth to your network, but also slows down performance as Dropbox has to index and sync this data constantly while the game is running.

In the future I will develop a manual Sync system for such files, which either updates the files every 5 minutes, or I can press a button to manually push the files to all of my computers. Who knows, I may even make my own Dropbox!

Minecraft is a Dropbox Overkill

This is a quick tutorial on how to code basic pagination using a combination of PHP and MySQL. Pagination is basically splitting a list of content between multiple pages for easier navigation and less scrolling on a website. To start with you are going to need a few variables.

``` php
$page_number = intval($_GET['p']) ? $_GET['p'] : 1; // Current page you are viewing
$page_limit = 10; // Items per page
$page_offset = ($page_number - 1) * $page_limit; // Where to start MySQL reading
```

Once you have the above variables you can then call the data from MySQL, like so:

``` php
$result = mysql_query("SELECT * FROM table LIMIT {$page_offset},{$page_limit}");
```

Now to create the actual links (1 &#8211; 10):

``` php
for ($i = 1; $i < 10; $i++) {
 echo "<a href="?p={$i}\">{$i}</a>";
 if ($i != 10) {
 echo " | ";
 }
}
```

There you have it, extremely basic pagination.

Pagination with PHP and MySQL

Hi guys! It's been a long time since I've posted on here, but I've been a very busy little worker bee! I will explain more about that in my next post, however. Right now I want to share some code that has been annoying me for the past 4 hours, as maybe it will help someone in the future. I'm currently working on an arcade racing game, just so I have something challenging to keep me out of trouble, but it's proving more difficult than I thought.

My basic problem was I needed to check if the car was currently hitting anything apart from the track, which in this case was the grass, but I also needed to know which part of the car was hitting the object. I started off looking through adobe live docs, as I have used something called hitTestPoint and hitTestObject before. Unfortunately, these are very limited due to the fact my track is a very complex and weird shape. hitTestPoint and hitTestObject only work on either a point, or the bounding box of the object. Somehow I managed to get hitTestObject working on curved lines in a previous project, but I think that may have been AS2. Luckily there is a bitmapData API which allows for deep integration into Bitmap Collision Detection. It all sounds fine and dandy, until you actually start using the code.

This is the basic Syntax for using the function:

``` actionscript
hitTest(firstPoint:Point, firstAlphaThreshold:uint, secondObject:Object, secondBitmapDataPoint:Point = null, secondAlphaThreshold:uint = 1):Boolean
```

I also found various examples of people using this technique around the Internet, but after an hour of trying I still couldn't get mine working in the game! Turns out, which Adobe fails to mention, that it seems specific on which way round you test the objects. In my example I was testing the front left wheel, against the grass that goes around the entire track.

My Code was:

``` actionscript
var Map:GameMap = Main.instance.Scene.current.Map;
var Grass:Bitmap = Map.CollisionLayer;
var WheelPoint:Point = Wrap.localToGlobal(new Point(Wheel.x, Wheel.y));
var GrassPoint:Point = Map.localToGlobal(new Point(0, 0));
if (Wheel.bitmapData.hitTest(WheelPoint, 255, Grass.bitmapData, GrassPoint, 255)) {
   // Hit
} else {
  // No Hit
}
```

Looking at the documentation, and others' examples, the code above should have worked! I even tried replacing the Wheel variables with the Mouse Position. Also, when I traced the global values out, and drove the car to the top left of where the grass starts (Grass Point), the global values were Identical. The most annoying thing is, I was sure I had tried switching them around (I've learned from past experience programs can be very picky when it comes to the order in which objects are referenced), but I hadn't. Something in my head triggered me to flip the values round and there you have it, working, "Pixel Perfect Collision" using bitmapData.

Here is my final code if you're interested:

``` actionscript
var Map:GameMap = Main.instance.Scene.current.Map;
var Grass:Bitmap = Map.CollisionLayer;
var WheelPoint:Point = Wrap.localToGlobal(new Point(Wheel.x, Wheel.y));
var GrassPoint:Point = Map.localToGlobal(new Point(0, 0));
if (Grass.bitmapData.hitTest(GrassPoint, 255, Wheel.bitmapData, WheelPoint, 255)) {
 // Hit
} else {
 // No Hit
}
```

There is still quite a bit of optimization to go now that I've figured out the basics, such as testing the entire object's bounding box first, rather than doing pixel precision detection on every frame. The game is still a work in progress, but I'm hoping to post an early version online for you guys to play with and give me feedback on by the end of the week.

Pixel Perfect Collision Detection AS3

Did you know you can also create symlinks using PHP? Now when you have a lot of files you want to symlink in bulk, such as Apache project files which I described in a previous Blog post, you can simply create a php file to do all the work for you.

I plan to build a backup script, which will copy all my project files and certain programs from my OS drive, to my external drive so I can use them in multiple locations easily, and also make it easier to backup if I wish to do a System Restore or something. You can find all the information on the <a href="https://us.php.net/manual/en/function.symlink.php" target="_blank">php website</a> for how to use the function. It is very similar to the <a href="https://marcqualie.com/2010/09/symbolic-links-in-windows-7.html">command line code</a> I put in a previous post.

Create Symlinks using PHP

Over at Google they are always trying to make the internet faster, and just generally better for all of us Speed Craving Humanoids. They weren't happy with just customizing the web pages themselves (even though they released many tools for this such as Google Page Speed and even created a new Web Protocol SPDY), they wanted to optimize every aspect of the assets inside the pages too, which includes Images.

Yes, you guessed it. Google is building a new Image format known as WebP. This new format was tested against 1,000,000 random Images from the Internet to get an average of 39% compression ratio, which is quite impressive on unoptimized images.They have released an open source version of this format for us developers to get our hands on and test for ourselves. I'm gonna try it out later next week when I get some time, but if you're into that sort of thing, you should check it out.

Full details over at the <a href="https://googlecode.blogspot.com/2010/09/webp-new-image-format-for-web.html" target="_blank">Google Code Blog</a>.

Ever wanted images to be smaller and load faster?

<a href="https://www.dropbox.com/referrals/NTM0NzEyNDA5">
 <img border="0" height="200" src="https://4.bp.blogspot.com/_SCmPP6x1F1A/TKJdFKzzpkI/AAAAAAAAAjA/GRd_w0I3qQI/s200/Dropbox.jpg" width="200" />
</a>

Do you have certain files you need to keep synced up on many different machines? For example, home work and a spare laptop? Well with <a href="https://www.dropbox.com/referrals/NTM0NzEyNDA5" target="_blank">Dropbox</a> it's very easy to do this, simply drop the files into the folder, and it'll be synced over all your computers. That all sounds nice and dandy, but what if you have a lot of files, in different locations, that you need to sync on a daily basis? This is where syncing gets tricky, who wants to continuously copy files over all the time from folder to folder? This is where I found an amazing practical use for symbolic links. We will take game save data for example.

Most game save data to the my documents folder, and are normally saved under weird names, which isn't helpful. Say you play a game at work, and you want to carry on at home, where you left off. Your first option is to manually locate the game save folder, and copy the files to Dropbox, then when you get home, you can copy the dropbox files to overwrite the files on your home machine. This is time consuming, and also has the scenario where you forgot to copy your files one day, and suddenly you can't carry on from another location. This is where the symbolic links come into play. Simply copy the game save folder to your Dropbox, anywhere you like.

I normally save mine under <code>/Games/[Game_Name]/Save Data</code>. Then, you create a symbolic link to this folder, where your save files are meant to be on the system. As the game loads, it will see the symbolic link as the actual files, since the operating system follows the loop hole transparently. You also do the same at home, however you simply create the symlink to the folder you have already created in your dropbox.

Now whenever you play the game from any location you have the symlinks setup, the save data will automatically be saved in your Dropbox. There are many, many more scenarios where you can use this method, so I hope this helps save someone some valuable time.

Symbolic Links and Dropbox

Animation Vs. Animator II [Flash]

Animation Vs. Animator [Flash]

<a href="https://4.bp.blogspot.com/_SCmPP6x1F1A/TJvSihEAOpI/AAAAAAAAAdA/Ehyk7lv3jMo/s1600/imgres.jpg">
 <img border="0" height="126" src="https://4.bp.blogspot.com/_SCmPP6x1F1A/TJvSihEAOpI/AAAAAAAAAdA/Ehyk7lv3jMo/s200/imgres.jpg" width="200" />
</a>

Well, Super WiFi, is in fact, standard WiFi. It will still give you the same data, and be accessed the same way, but on a different frequency. Basically since TV has gone digital, there is now so called “white-space” which is the space between the signal for your TV channels. As far as I can gather from various posts around the net, this WiFi 2.0 will operate from 500mhz – 700mhz rather than the standard 2.4ghz – 5ghz. Even though everyone seems excited about this right now, don't expect it anytime soon. However, when it does land, the possibilities for mobile networking will be vastly increased.

What is Super WiFi?

If this were a real phone, I would most likely sell everything I own just to have one. Seriously, amazing.

<iframe class="d-block mx-auto" width="560" height="420" src="https://www.youtube.com/embed/oG3tLxEQEdg?color=white&theme=light" frameborder="0" allowfullscreen></iframe>

Mozilla Concept Phone

<a href="https://3.bp.blogspot.com/_SCmPP6x1F1A/TJaek5dshuI/AAAAAAAAAco/2nqN5l4HjCA/s1600/imgres.jpg">
 <img border="0" height="149" src="https://3.bp.blogspot.com/_SCmPP6x1F1A/TJaek5dshuI/AAAAAAAAAco/2nqN5l4HjCA/s200/imgres.jpg" width="200" />
</a>

The other day I posted an article about <a href="https://marcqualie.com/2010/09/symbolic-links-in-windows-7.html">Symbolic Links in Windows 7</a>. I just want to write an update on how amazing these things actually are, especially for working on multiple projects, across multiple locations. Currently I'm running a <a href="https://www.amazon.co.uk/Western-Digital-VelociRaptor-150GB-Internal/dp/B001EQQW1A/ref=sr_1_1?ie=UTF8&s=computers&qid=1284939517&sr=8-1">10k VelociRaptor Drive</a>, mainly due to the fact I am inpatient and want my system to startup as soon as possible, without splashing out for a SSD.

Back to my point, I only have 150GB, and I have my local Apache Server running from the OS drive. Now, loading files from a remote drive over a network, or even just an external hard drive can really make Apache angry, so all my files are on my OS drive for my projects too. Great, right? Wrong. This east up valuable space, which I could be using for more programs and games, which could really benefit from the speed boost. This is where the Symbolic links come in handy. I can now place all of my project files on an external hard drive, which can then be monitored and backed up easily, and even picked up and carried arround without having to copy files first. Now all I do is set up symbolic links in the Apache projects file, rather than the actual files and there you have it; Apache sees the file locally, and you have your files on removable media for easy transportation and backup, without eating into your computers hard drive space. Hope this helps someone, it's made my life so much easier.

Symbolic Links are my new Best Friend

Recently I purchased a new Keyboard and Mouse desk set, which I have been very impressed with so far. It's a <a href="https://www.amazon.co.uk/Microsoft-Wireless-Desktop-3000-MFC-00007/dp/B00284AM90/ref=sr_1_fkmr0_2?ie=UTF8&qid=1284815074&sr=8-2-fkmr0">Microsoft Wireless Deskset 3000</a>. The best thing I have to say about the setup is the fact the wireless signal never drops. In the past I've always had issues with signal dropping on wireless devices, especially keyboards. It is also very responsive for gaming, so I can still waste time playing games just as much as I used to.

However, going off topic here, due to the fact this post is meant to be a rant! As the keyboard and mouse may be very well made pieces of hardware, unfortunately the software that accompanies it, isn't. IntelliType Pro 7.0 is possibly the most annoying thing ever. While it does some things great, such as on screen prompts of certain keys like volume indicator and hot key interaction notifiers, it fails on one of the most simple tasks. This would be controlling media via any player other than Windows Media Player.

Personally, I use <a href="https://spotify.com">Spotify</a> due to the fact I can listen to my music anywhere without having to carry it with me. IntelliType Pro seems to think the only media player anyone uses would be Media Player, due to the fact whenever I press the play/pause button it opens Media Player, and I can only control Spotify using the media controls if Media Player is open in the background, and doesn't have focus. Get your act together Microsoft, just when I was starting to get faith back in you thanks to Windows 7.

Marc Qualie