James Crisp

Software dev, tech, mind hacks and the occasional personal bit

Category: Technical (Page 1 of 13)

WordPress.com vs Self Hosted WordPress

By

On September 5, 2017

In Technical

Previously, this blog was hosted on a Virtual Private Server (VPS) with Linode. No problem there, it worked for many years. However, the VPS has other uses, and this blog needed to move.

WordPress.com
Wordpress.com looked like the most attractive option. Automatic updates, automatic backups, automatic CDN etc. So I exported my blog and imported into WordPress.com with a $5 AUD/month personal account. This price gives you a limited setup (no custom plugins), but that was OK as their JetPack plugin provided anti-spam for comments and a form plugin to make a contact form. The price was reasonable too. Alas, the import was very slow eventually imported only 50% of my media (about 40mb of 80mb). URLs to images were also incorrect and showed up as broken even for media that was imported, as the URL structure changed on WordPress.com (links go to a CDN instead). WordPress.com support promised that the image URLs would be updated by a batch job during the night. This never happened. On the subject of the missing media, support told me I needed to upload my media in ~5mb chunks, as the import process could time out but didn’t give any output to the user or support. Ie, they recommended I do 16 manual exports and imports from old blog to new to avoid timeouts. At this point, things felt too creaky to continue, and I asked for a refund, which WordPress.com support kindly agreed to.

Hosting on WordPress.com has some other minor differences. The comment form shows up differently (maybe from jetpack), maybe a little less attractive than my theme, but not a show stopper. There are two admin consoles, the normal WordPress one, and their custom one. There is some overlap between. It is a workable setup. You can’t add Google Analytics though without upgrading your plan, and there’s no way to get everything out easily (you can get out your WordPress xml export, but not media files or settings).

In conclusion, WordPress.com was mainly unsuccessful for me as it was unable to import my old blog. With no access to files or database, it was also impossible for me to manually correct any issues. Perhaps if you were starting a new blog, it could be a good option.

Shared Account with InMotion
After some research online, I went with inmotion.com shared hosting for WordPress. They promise to update your WordPress automatically (I also installed an additional plugin to keep the other plugins up to date). They offer a full and automated WordPress install (all plugins, themes etc available), backup functionality (though you have to ask them to restore), and full ssh access. Speed is good so far, and they are very reasonably priced ($2.95 USD/month if you sign up for 3 years with this coupon). They offer a 90 day back money guarantee.

I tried the same import from my old blog. It was more successful than wordpress.com but didn’t get all media and some links were still broken. I rsync-ed the missing files across (handy to have ssh access) and ran a quick DB update to fix links. InMotion even had a support page for this. Their online support was helpful when I needed to know the IP of my shared host.

So far so good, my blog is now set up and working and I’d recommend InMotion.

Scalability / Performance Tip – Use Cloudflare
I use Cloudflare to provide a CDN, HTTPS and reverse proxy caching for my blog. Their free account is sufficient and means that if my blog ever gets super busy, the load will be handled by Cloudflare, rather than overloading my shared host. Their CDN setup also means the blog loads faster from non-US locations.

Solving Calendar Sync problems on Android 7 Nougat

By

On August 26, 2017

In Android, Personal, Technical

Recently, the phone calendar on my Samsung Galaxy S6 stopped synchronising with Google calendar. When I went to Google Accounts Sync in Settings, Calendar had the spinner next to it, but it was didn’t spin. Meanwhile, the calendar didn’t sync, and the battery was being chewed through more quickly than usual.

How to fix? Well the first thing I tried was deleting Calendar Storage. This worked for a day or two, and then the problem reoccurred. Next, tried deleting all my Google accounts and adding them back. That worked for about a week.

Finally, by a stroke of good luck, I was looking at the sync screen when I’d just plugged in the phone to charge – it synced fine. A lead at last! Likely something related to power settings!

I’d already poked around in the usual Device Maintenance > Battery > App power monitor screen, and all Calendar related apps where in the ‘Unmonitored’ list so wouldn’t be put to sleep. This section wasn’t the cause of the problem. I finally found the solution, in an additional hidden set of power saving options.

So.. To fix, go to Settings > Device maintenance > Battery > Battery Usage button > Vertical … at top right > Optimise battery usage. Choose All Apps from the drop down. Then disable Optimise for Calendar storage and your calendar apps. Voila! Finally calendar will sync reliably again!

UPDATE – Other syncs

I also had a similar issue with Google Sheets, Google Docs and Google Drive Sync. The same change in ‘Optimise battery usage’ settings for each of these fixed their sync as well.

Talk: Winning at HTTPS

By

On October 18, 2016

In ALT.NET, Cloud, Design / Architecture, Talks, Technical

For the first time in a little while, I’ll be giving a talk at the Sydney ALT.NET user group:

HTTPS is ever more pervasive, with few sites still using plain HTTP. Want to be the guy or girl on the team who actually understands HTTPS, can set up certificates and fix issues that come up? Sometimes this is left to an ops team, but there are benefits and impacts that cannot be ignored in development.

James has migrated several sites from HTTP to HTTPS and has tips and tricks to share.

Tues 25 October from 6pm at ThoughtWorks Sydney office, Lvl8 51 Pitt St, Sydney.
RSVP on Meetup (for pizza and beer!)

You can find the slides from the night here.

Talk: Add a billion row data warehouse to your App.. with Redshift, sql and duct tape!

By

On June 16, 2014

In ALT.NET, Cloud, Design / Architecture, Talks, Technical

Come along to Sydney ALT.NET for a BIG data night.

I’ll be giving a talk on Redshift:

Started to hit the point where your transactional database is not the right place for running reporting queries and experimental data science? Keen to chuck in more data from web logs, CRMs, facebook, etc so you can start learning more about your users? Come along to Sydney ALT.NET on June 24th to see an easy way to do it with AWS Redshift, mapping SQL and some simple scripting duct tape.

We also have a co-presented talk on Azure’s Hadoop implementation, HD Insights and Power BI: The Power of the Elephant in the Microsoft Cloud given by Jibin Johnson and Simon Waight from the Azure User Group.

From 6pm at ThoughtWorks Sydney office Lvl8 51 Pitt St, 24 June 14.
Remember to get your free ticket. See you there!

Moving to HTTPS, Rails force_ssl and Rollback

By

On August 4, 2013

In GetUp, Ruby / Rails, Technical

Background
We recently moved the Getup site from mixed HTTP/HTTPS to completely HTTPS. The primary driver was to ensure that sessions were never sent in plain text over the wire, to avoid session hijacking. There are other benefits too, such as protecting personal details from eavesdropping over the wire, proving site authenticity and generally simplifying the code. Checking around the web, Twitter is HTTPS only, and even Google search is all HTTPS (when you are logged in).

Rails HTTPS and force_ssl
The easiest and simplest way of moving a Rails 3 app to all HTTPS is to simply set force_ssl = true in relevant environment files. This then causes Rack ssl middleware to be loaded as the first middleware. As you can see from the code, this middleware does a variety of good stuff, that really ensures once HTTPS, always HTTPS!

  • 301 permanent redirect to HTTPS (cached forever in most browsers so you never hit the http url again)
  • Secure on the cookies so they can never be sent over HTTP (important they don’t accidentally go with a redirect for example!)
  • HSTS to ensure that in supported browsers, nobody can ever go to HTTP again!

HSTS
HSTS (HTTP Strict Transport Security) is offered by Chrome and Firefox ensures that for a given time (usually a long time, eg 20 years in the case of Twitter!), it is not possible to go to the site over HTTP again. To activate this, the site only needs to send the Strict-Transport-Security header once. You can check and manage what’s in your HSTS store in Chrome with chrome://net-internals/#hsts

Rollback
When moving to HTTPS, we wanted to ensure we could rollback to a previous release in case of problems. Using force_ssl out of the box precludes this – if you roll back after a 301 redirect or HSTS loaded by a client browser, your site will no longer be accessible!

We used a small monkey patch which turns off HSTS and uses a 302 temporary redirect, rather than a 301. This means that rollback to a previous release works fine. Here’s the patch:

require 'rack/ssl'

module ::Rack
  class SSL

    def redirect_to_https(env)
      req        = Request.new(env)
      url        = URI(req.url)
      url.scheme = "https"
      url.host   = @host if @host
      status     = 302
      headers    = { 'Content-Type'  => 'text/html',
                     'Location'      => url.to_s,
                     'Cache-Control' => 'no-cache, no-store'
                   }
      [status, headers, []]
    end

    def hsts_headers
      {}
    end

  end
end

This patch is only needed temporarily, until you decide that you no longer would want to deploy a release before force_ssl.

Performance
So is HTTPS slower than HTTP? It is to start with in initiating the first request, as ssl needs to be negotiated and set up. This leads to a few more round trips. If your clients and servers are in same country, this is pretty insignificant. Round trips from Australia to USA for example are more significant but not a major stumbling block, as long as you use Keep-Alive on the connection to ensure that later requests re-use the set up from the first request.

Assets are fully cacheable over HTTPS using the usual HTTP headers, and have been even since early Internet Explorer versions. You do need to make sure that you always load HTTPS assets though, to ensure you don’t get mixed-mode warnings in the web browser. We found some useful HTTPS performance tips here.

Load on the servers was not an issue for us as we are using Amazon Elastic Load Balancers (ELB) for our HTTPS implementation. The web/app servers don’t get involved as they are just reverse proxied by the ELB, which manages the HTTPS sessions.

Redirect Gotchas!
We have a few other domains which simply redirect to the canonical www.getup.org.au domain. Out of the box, the rack ssl middleware loads first, before our redirect middleware. This meant that for these additional domains, we got a nasty certificate warning in the browser as it is sent to HTTPS first (on the wrong domain), and then gets the redirect to the canonical domain that has the valid certificate. Changing the order of middleware to do redirects first, and then HTTPS is an easy solution.

Conclusion
The move to full HTTPS has gone smoothly and we didn’t end up needing rollback. However, it was worth having the monkey patch so that rollback was possible as an insurance policy against unexpected major problems.

Talk on Tues: Moving to HTTPS

By

On July 28, 2013

In ALT.NET, C#, GetUp, Ruby / Rails, Talks, Technical

I’ll be giving a talk at Sydney ALT.NET on Tues:

After recently moving the Getup site fully to HTTPS, James will share with you security pitfalls, the justification for the move from mixed HTTP/HTTPS, lessons learnt, and performance tips. A romp through the protocols of the web with riffs on status codes, HSTS, domain verification, and interesting headers. This talk could save your bacon.

From 6pm at ThoughtWorks Sydney office on Pitt St. Remember to RSVP on the Sydney ALT.NET site to help with catering. See you there!

iAwards Win for CommunityRun / ControlShift!

By

On June 19, 2013

In GetUp, Personal, Technical, ThoughtWorks

Very pleased to announce that CommunityRun and the ControlShift platform have won the NSW iAwards in the Community Category.

Many thanks to everyone from GetUp, ThoughtWorks and ControlShift labs for all the hard work and perseverance. I’m proud to have been part of the team to build a tool that lets anyone start and run their own campaign to improve things in their community.

Talk tonight: Responsive Layout with HTML5

By

On November 27, 2012

In ALT.NET, GetUp, Talks, Technical

I’ll be giving a talk at Sydney ALT.NET tonight:

Want to build a web application which dynamically changes layout to best suit the client, be it mobile, tablet or desktop with the same HTML? Fun times with HTML5, Bootstrap, HAML and Sass. You’ll get to see it in action, and the code behind the magic.

From 6pm at ThoughtWorks Sydney office on Pitt St. Remember to RSVP on the Sydney ALT.NET site to help with catering. See you there!

VPS Performance Comparison and Benchmarking

By

On June 20, 2012

In Ruby / Rails, Technical

What VPS / cloud instance provider gives you the best bang for buck for a small linux server? I’ve been focusing on evaluating providers at the low cost end, and reasonable latency to Australia – Linode Tokyo, Octopus Australia, RailsPlayground USA and Amazon Singapore. From the tech specs on a VPS, most providers look similar. However, when you have an instance, you often find huge variation in performance due to contention for CPU and disk on the node. For a simple test, I ran a small ruby script on each server every 20 minutes on cron, and logged the results to a CSV file over the same 5 days.

Here is the script:

start = Time.now

`man ls`
`ls /`

disk_time = Time.now - start

start = Time.now

(1..200000).each { |i| i*i/3.52345*i*i*i%34 }

cpu_time = Time.now - start

total = disk_time + cpu_time
puts "#{total},#{disk_time},#{cpu_time},#{start.strftime('%Y-%m-%d')},#{start.strftime('%H:%M:%S')}"

It uses ls and man to test wait for disk and then some made up maths to test the CPU. It’s not perfect but gives a good indication of performance.

Results
All times are in seconds, and you can download the full data to get a better picture. I’ve provided a summary and notes for each provider below.

Linde 512mb Tokyo (Zen)

  Average Max
Total time 0.81 1.74
Disk 0.03 0.13
CPU 0.78 1.63

Full Results

This is my second Linode VPS. I asked to be moved to a different node as the first one started fast when I got it but performance degraded (due to contention on the node?) within a few days. Nothing else is running on the VPS besides the script. Overall this is consistent, good performance in both CPU and disk. Latency to this VPS is around ~130ms from my ADSL in Sydney.

Octopus 650mb Australia (VMWare)

  Average Max
Total time 0.74 3.40
Disk 0.19 2.88
CPU 0.54 1.08

Full Results

Octopus is running a site for me with a significant cron job on the hour. I’ve therefore removed results collected on the hour. Despite running my live site, Octopus has the fastest average performance of all VPS tested. The higher max time could have been caused by load on my site. Octopus costs a bit more but is hosted in Australia so has very low latency of around 20ms from my ADSL in Sydney.

Amazon Small Instance 1.7gb Singapore (Zen)

  Average Max
Total time 1.25 2.42
Disk 0.20 0.40
CPU 1.04 2.03

Full Results

Amazon EC2 Small instances are now in the same ballpark cost as small VPS instances, when you go for high usage reserved. Many people think that EC2 disk is poor. However, from my tests (and experience) it is not super fast, but it is consistent and reliable. Amazon is also very generous with memory and provides other benefits like free transfer to S3. Where it falls down is processor speed, which, although fairly consistent, is about 50% slower than Linode and Octopus. Latency from Sydney ADSL is around 120ms.

Rails Playground 640mb USA (Virtuozzo)

  Average Max
Total time 3.53 24.12
Disk 2.44 23.42
CPU 1.09 2.66

Full Results

My RailsPlayground VPS is running various services and sites for me but none of them are high load. As you can see, the CPU performance is similar to Amazon and doesn’t vary too much. The problem is disk which varies hugely and can sometimes lead to unresponsive sites and services while waiting for IO. Latency from Sydney ADSL is around 200ms.

Conclusion
If you want a fast VPS with low latency in Australia and are willing to pay a bit more than the other providers listed, Octopus VPS will serve you well.

For lots of memory but slower CPU, Amazon small instance will be good.

For faster CPU but less memory, Linode is your best choice.

It’s worth testing out any new VPS and keeping an eye on performance over time. Contention on your node could change dramatically without you being aware of it, dramatically impacting performance of your VPS.

Jetstar Review: Booking a holiday package

By

On June 11, 2011

In Personal, Technical

With some holiday leave coming up, my wife and I decided to go to Queensland for a warm break from the Sydney winter. After a little online research into Virgin and Jetstar packages, and booking directly, it was clear that going with a Jetstar package for the flight and hotel was several hundred dollars cheaper. The hotel on offer also looked quite good and only had availability for our dates when booked through Jetstar.

Tues 7 June
Went to the Jetstar site. The first time I tried entering our dates for the search, I got a message about the site being overloaded and please try again. I did so, and got to the next page, allowing me to customise flight times and choose accommodation options. The hotel we wanted to stay at had the same room type listed 3 times, at different prices. I had called Jetstar about this earlier in the day as I’d seen it, and been confused, when researching prices earlier. They had explained that it was the same room type – just if the cheapest price for the room type was already full, you would have to buy the next one up and so on. Following this advice, I chose the cheapest price for the room type offered.

I then tried to click the ‘Reviews and Information’ link which had worked earlier when I’d been doing the price comparison. This time, I got an exception message:


For the technical minded, you can see that the link from the package configuration page was faulty and did not include a ‘hotelId’ (bug?). Next, the hotel information page did not check the input parameters at all (bug and security risk) and exploded. Lastly, you can see that the Jetstar site has not been configured for production error messages. Instead, a developer error message was displayed which exposes implementation and technology details to the casual observer (security risk and poor user experience).

Somewhat concerned, I continued with the booking process. Choosing flights worked well, except I did wonder why one of the flight choices was listed as 4 hour duration, but left at 7am and arrived at 10am:


We next entered personal details for me and my wife. We had to try several times to submit the form as there must have been validation rules that our data was not meeting, but no error messages were displayed to help us. We fiddled around with the details we’d entered a bit (different formats for phone numbers etc), and then it finally worked.

Transitioning to the next page was very slow, but eventually came up with a chance to choose seats and travel insurance. My wife was sitting with me at the computer and we decided to choose seats. It was certainly not clear that we would have to pay extra for this. So I went ahead and chose some nice seats near the front of the plane (quick exit seats they were called) and then noticed that this would add an extra ~$64 on to the holiday cost! Realising our mistake, we chose regular seats a little further back. This was still an extra $16 on to the cost of the holiday. We tried everything we could think of to try and de-select our seat choices but could not find a way to do so that would actually stick (you could de-select temporarily but this could not be saved). Somewhat annoyed, we continued with the booking.

Next I entered my credit card details. It turned out there was an extra $30 booking and service charge that suddenly appeared at that point. Not happy but keen to finish the booking process, we next hit the pay button. The site took a while to respond and then reloaded the payment page, showing that there was a balance of $-907.40 with a proceed button below it. Quite a mystery. Had my credit card been charged? Had anything been booked? What did the mysterious $-907.40 mean?

I rang the call centre. Unfortunately it was either overloaded or down and I couldn’t even get to the “Press 1 for English…” prompt. I waited 15 minutes and rang again. I left the phone on speaker and went on with other things. I was on hold for 54 minutes before I gave up and went to bed.

Wed 8 June
In the morning, I range the call centre again, and got through in about 20 minutes to the reservations section. I explained the situation to the customer service person and she found my booking. It turned out that flights were booked, the hotel was not booked, and my itinerary was not sent. I explained which hotel was meant to be booked and she went to talk to her supervisor. I was on hold for a further 30 minutes or so with a few other brief questions during this time. She was able to send me the flight itinerary without the hotel which she did. She was unable to help with the hotel booking, but sent an internal request to the holidays section and said they would call me back within 24-48 hours to help with the hotel booking. I asked if I had any assurance that the hotel was booked. She said I didn’t have any assurance. I asked if there was any way to make this faster, as I needed to know if I should book my own accommodation. She sounded annoyed and said that there was nothing she could do, it was a different section of the business and she couldn’t help me further, just wait for the phone call in 24-48 hours. I did receive a copy of the itinerary for flights without hotel within 24 hours as promised. Meanwhile I had tweeted to @JetstarAirways about the experience so far, and they asked for my booking reference via DM to expedite things.

Thurs 9 June
Finally received a phone call from Jetstar holiday section. The guy said that he had looked into the case and sorted out the hotel booking. Great news! I asked if he could do anything about the $16 seat booking charge from earlier, but he said that it was clearly explained on the site, and he couldn’t do anything about it. He promised that the complete itinerary would arrive including the hotel booking within 24 hours and I gave him my email address (not sure why they didn’t have it from earlier, I’d already received the flight itinerary via email).

Fri 10 June
Waiting for itinerary. It did not arrive.

Sat 11 June
Wondering what had happened, I rang the call centre again. This time, I didn’t go on hold at all and got straight through to a helpful lady that confirmed my email address – it had been entered wrongly. That was probably why I hadn’t got an email yet. She fixed the typo in my email address, and I received the complete and correct itinerary within 10 minutes. The booking process was now complete!

Conclusion
A transaction I had expected to take about 20 minutes booking through the site took about 4 days from start to end, including about 2 hours of time on the phone to the call centre. The Jetstar price was still good compared with other options, but the unexpected charges of $16 for choosing seats and $30 for booking fee were a major turn off. The bugs in the site (especially the validation of personal details without showing error messages) must lose many potential customers. The failure of the purchase process which left the customer confused but charged without services being booked was a major issue. The call centre delays were a time waster and the silo-ed nature of the business between flights and holidays made solving problems slower and harder. Jetstar’s twitter account was potentially a help in resolving problems.

Post-Holiday Update
Happy to say that the flights and hotel worked well after all the booking was sorted out. Jetstar also upgraded us to a better room at the hotel which was a nice touch.

Page 1 of 13

Next →

Categories

Powered by WordPress & Theme by Anders Norén