Jekyll2024-03-17T15:10:59+00:00http://www.jdwilkins.co.uk/feed.xmlJeremy WilkinsFull stack software developer living in Cumbria who loves all things Ruby and Docker. Outdoor Enthusiast who climbs and bikes and loves the Lake District.
Jeremy Wilkinsjeremy@jdwilkins.co.ukHappy New Year2022-01-12T15:29:00+00:002022-01-12T15:29:00+00:00http://www.jdwilkins.co.uk/2022/01/12/tools-i-have-learned<p>Well since I've restarted this blog, again, so I should probably write something.</p>
<p>First up Happy New Year - here's to hoping for a less Covid infested year.</p>
<h2>Things I have learnt</h2>
<p>I thought I'd take stock of the random assortment of tools I've learnt working on GDS projects over the last couple of years.</p>
<p>RSpec - I kinda knew this one anyway but most projects I'd previously worked on had been minitest + shoulda - these days I'd describe myself as pretty knowledgeable on writing good rspec friendly tests.</p>
<p>Azure and ARM templates - I'd not used Azure as a hosting platform prior to doing GDS projects. Some parts are really nice (ACI), some less so - SSL certificate mgmt still seems to be a mess (more so with ACI oddly).</p>
<p>Whilst I know a lot of projects like to use Terraform to build out their infrastructure, at DfE they were keen on ARM templates. The syntax is horrible (Bicep looks 'a bit' better) but it does work well and having the infrastructure track the state makes far more sense then mucking around with state files, buckets and Terraform.</p>
<p>GovUK Frontend - the canonical implementation of the GovUK Design System</p>
<p>Stimulus - I'd not used this prior to the GDS projects but I am now a big convert. It seems to be a much better fit for any Rails projects that aren't trying to build a client side app. Previously I'd used both JQuery and React (and Preact) for client side 'widgetry' - React was nice but gets complicated fast and you can easily end up duplicating templating. JQuery is simpler but enforces no structure so gets messy over time.</p>
<p>Jest (+ Babel) - Partly related to using Stimulus. Previously I'd relied on capybara/system testing to verify javascript functionality - unit testing JS controllers makes a lot of sense and speeds up the development a lot. By the point your integrating your Stimulus controllers you can be sure they work as intended, anything not working on a page will almost certainly be something simple like a typo on a target or action name.</p>
<p>ElasticSearch - I'd used this before but not for a while, my current project is using ElasticSearch so its been good to bring myself back up to speed.</p>
<p>GitHub Actions - definitely my preferred CI system - seems to work really well and the ability to hook into different events within GitHub allows for nice integrations between different parts of the teams workflow.</p>
<p>Azure Pipelines - I'd worked with this whilst at DfE - the best I can say for it is 'it works' but the limitations were fairly major and the rate of feature development painfully slow.</p>
<p>CircleCI - we're using this on my current project, I used to really like this but GitHub Actions has spoiled me and it now feels quite limited in comparison. Still does the job fine though.</p>
<p>Sequel - alternative ORM to ActiveRecord. I know some people rave about Sequel, but whilst its more flexible then ActiveRecord (and its flexibility is probably a necessity on our current project) - it definitely suffers for being a small team and not particularly invested in the Rails ecosystem.</p>
<p>GovUK PaaS and Cloud Foundry - GDS has built their own hosting service - GovUK PaaS, this is a centrally managed Cloud Foundry instance. This makes it very easy to spin up new apps in their cloud, heroku style - and I used this to add Review Apps to our current project (one for a future blog post).</p>
<h2>Things I want to learn in 2022</h2>
<ul>
<li>Rust - I've got the basics on this but its a complex language and there's a lot more still to learn</li>
<li>Gtk4 - I've done some limited Gtk app development in the past but I'd like to become proficient at this. I'm hoping to work on a Gemtext editor a la Apostrophe (very nice Markdown editor for Gnome) - we'll see if I ever make time for this.</li>
</ul>Jeremy Wilkinsjeremy@jdwilkins.co.ukWell since I've restarted this blog, again, so I should probably write something.Blogging again2021-12-30T10:53:16+00:002021-12-30T10:53:16+00:00http://www.jdwilkins.co.uk/2021/12/30/blogging<p>Well I've managed 2 blog posts in the last 9 years - the New Year's coming around, and New Years resolutions with it, so I thought I'd try and start blogging again. A lot of the work I do these days is open source so no excuses this time!</p>
<p>Since the last blog post I've found myself working on several Government Digital Services projects - first for Dept for Education and now for HMRC. For those not 'in the know', these are the various services you'll find on the GOV.UK website, all using the same consistent look and feel.</p>
<h2>Get school experience service</h2>
<p>For Dept for Education we first built a booking service for prospective teachers to arrange time in schools to gain experience of teaching in classrooms. As with many GDS services with was quite simple for a user perspective with plenty of complexity hidden away behind the scenes with integrations to the DfE internal authentication system and with data from the public held within one of the departments Microsoft Dynamics instances.</p>
<p>
<a href="https://github.com/DFE-Digital/schools-experience">Get school experience</a>
</p>
<h2>Get into teaching</h2>
<p>Another DFE project, Get into teaching was a promotional website to drive up teacher recruitment numbers. As with Get school experience this project integrated with the Microsoft Dynamics system behind the scenes for data management. This was a much more UI focused project so didn't use the GOV.UK look but followed a similar feel and made extensive use of Stimulus and Rails Components to build a nice dynamic feeling website which our content authors could easily edit and update.</p>
<p>
<a href="https://github.com/DFE-Digital/get-into-teaching-app">Get into teaching</a>
</p>
<h2>Get teacher training adviser service</h2>
<p>A GOV.UK service using the standard GDS Design System to apply for a teacher training adviser, who can help guide prospective teachers through the process of becoming a teacher. Multi-step wizard process using the same pattern I'd built for the wizards in Get into Teaching.</p>
<p>
<a href="https://github.com/DFE-Digital/get-teacher-training-adviser-service">Get teacher training adviser service</a>
</p>
<h2>HMRC Online Trade Tariff</h2>
<p>This is 4 separate ruby applications providing trade tariff information to traders who wish to import or export goods with the rest of the world. Quite an interesting set of applications - they predate many of the current tooling around the GOV.UK Design System but are steadily being ported to take advantage of it. The backend uses the Sequel ORM in place of ActiveRecord because the nightly data feed from Dept for International Trade expects to populate a table with a log/append style set records of changes - eg INSERTing a 'delete' record to mark an earlier recorded as deleted. These are then 'presented' via a database view, which in turn is then exposed to the rails app via the Sequel ORM. The applications themselves are all Rails 6.1 but have a mixture of quite modern code and some legacy Rails 2/3 era code, though the old code is steadily getting modernised with every improving test coverage.</p>
<p>
<a href="https://github.com/trade-tariff">HMRC Online Trade Tariff repos</a>
</p>
<h2>Personal projects</h2>
<p>As well as working on the GDS projects, I've also been worked on a couple of features for Gnome-Builder - my current text editor - adding support for ERB files (both HTML and JS) and adding Rubocop support for inline linting.</p>
<p>Prior to switching back to using Linux I'd been a TextMate user for over a decade and whilst there was some initial learning curve I've settled right in to using Gnome-Builder. I'd encourage any other TextMate enthusiasts switching to Gnome to give it a go. It feels native to Gnome in the same way TextMate does on a mac and offers many of the same features.</p>
<p>The other project I've been working on relates to Geminispace but I'll save that for another post.</p>Jeremy Wilkinsjeremy@jdwilkins.co.ukWell I've managed 2 blog posts in the last 9 years - the New Year's coming around, and New Years resolutions with it, so I thought I'd try and start blogging again. A lot of the work I do these days is open source so no excuses this time!Building Docker Images with GitLab CI2018-10-19T14:49:16+00:002018-10-19T14:49:16+00:00http://www.jdwilkins.co.uk/2018/10/19/gitlab-ci<p>On a recent project I’ve been working to standardise the deployment
infrastructure across an assortment of Ruby, PHP and Java apps. Some on their
own boxes, some using boxes shared across projects.</p>
<p>It was decided to standardise the deployment process using Docker Images. I’ll
blog about the deployment side in the future but this post is about automating
building those Docker Images.</p>
<p>After some research it was decided to move all the repositories to GitLab. I’d
used GitLab for some of my own projects previously and can whole heartly
recommend it.</p>
<p>As with most CI systems, the actual build gets triggered by the presence of a
YAML file, called <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code>, describing what should be run and how.</p>
<p>GitLab’s CI system runs all the commands from within a Docker Image. You can
specify the actual image - this can be something like the minimal
Ubuntu or Debian images or one of your own images held within a registry you control.</p>
<p>I’ve seen some blogs where developers run the tests, then build a Docker Image only if
all tests pass. To me this seems self defeating - the Docker Image is there to provide
a consistent environment, so tests should be run from within that same environment.</p>
<p>This means we have a 3 step process.</p>
<ol>
<li>Build the git repo into a Docker Image</li>
<li>Run unit tests from within the Docker Image, if the project has a test suite</li>
<li>If you want to deploy the image, then upload it to a Registry
with an appropriate tag.</li>
</ol>
<h2 id="building-the-image">Building the Image</h2>
<p>So Step 1 meant getting an environment that can build Docker Images, and
building from the <code class="language-plaintext highlighter-rouge">Dockerfile</code> in the Git repo.</p>
<p><code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code></p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">image</span><span class="pi">:</span> <span class="s">docker:stable</span>
<span class="na">variables</span><span class="pi">:</span>
<span class="na">DOCKER_HOST</span><span class="pi">:</span> <span class="s">tcp://docker:2375/</span>
<span class="na">DOCKER_DRIVER</span><span class="pi">:</span> <span class="s">overlay2</span>
<span class="na">IMAGE_TAG</span><span class="pi">:</span> <span class="s">$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG</span>
<span class="na">services</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">docker:dind</span>
<span class="na">build_and_test</span><span class="pi">:</span>
<span class="na">stage</span><span class="pi">:</span> <span class="s">test</span>
<span class="na">script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">docker build --pull -t $IMAGE_TAG .</span>
</code></pre></div></div>
<p>The YAML above tells GitLab to use the ‘Docker’ Docker image, which contains just the Docker client program.
It also says to run the ‘Docker In Docker’ service giving us
access to a functional Docker daemon. The two environment variables are to tell
the Docker client how to connect to the daemon and to use the overlay2 driver to speed
things up.</p>
<p>GitLab’s CI can run an arbitrary number of jobs, which will be run in parallel. We
only need one job though, since the image can’t be tested until its built, and it
can’t be uploaded until its been tested.</p>
<p>I also found that multiple jobs may get run on different Runners (servers), but we need
tests to be run on the same Runner which built the Image.</p>
<p>Helpfully GitLab provides a Registry and sets up access tokens. It then sets
environment variables with access details of the registry and a normalised tag name.
This <code class="language-plaintext highlighter-rouge">CI_COMMIT_REF_SLUG</code> is the branch or tag name in Git but with any invalid
characters substituted for compatibility with Docker.</p>
<h2 id="testing-the-image">Testing the Image</h2>
<p>Next up I needed to get tests running from within the Docker Image. This
particular project was a Ruby on Rails app which meant things like database
connections could be controlled from environment variables.</p>
<p>The hardest thing to figure out was the need to run your own MySQL instead of
GitLab’s default service, and that when running the tests you’ll need it to be
within the same GitLab Job as the build process.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">build_and_test</span><span class="pi">:</span>
<span class="na">stage</span><span class="pi">:</span> <span class="s">test</span>
<span class="na">script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">docker run --name=mysql -d -e MYSQL_DATABASE -e MYSQL_ROOT_PASSWORD mysql:5.7</span>
<span class="pi">-</span> <span class="s">docker build --pull -t $IMAGE_TAG .</span>
<span class="pi">-</span> <span class="s">docker run --rm --link=mysql:mysql -e RAILS_ENV=test -e DATABASE_URL=mysql2://root:$MYSQL_ROOT_PASSWORD@mysql/$MYSQL_DATABASE $IMAGE_TAG rake db:test:prepare</span>
<span class="pi">-</span> <span class="s">docker run --rm --link=mysql:mysql -e RAILS_ENV=test -e DATABASE_URL=mysql2://root:$MYSQL_ROOT_PASSWORD@mysql/$MYSQL_DATABASE $IMAGE_TAG rake test</span>
</code></pre></div></div>
<p>First we run start the MySQL instance, this will pull down the MySQL Image automatically, then
daemonise. In theory this is racy but in practical terms its always going to
take longer to build the Image for our project, then MySQL will take to boot and
start accepting connections.</p>
<p>The MySQL Container will create the database automatically on start, so I just
need to load the schema and any fixtures with <code class="language-plaintext highlighter-rouge">rake db:test:prepare</code> then run the
tests with <code class="language-plaintext highlighter-rouge">rake test</code>.</p>
<p>One potential gotcha is making sure you install your testing Gems when building
your Docker Image. I’d created these Dockerfiles myself with this use case in
mind so they already were included.</p>
<h2 id="publishing-the-image">Publishing the Image</h2>
<p>If the build and tests are successful, then the final step is to publish the
Image - I’m using GitLab’s Registry since they helpfully provide token access
from the CI environment as Env Vars.</p>
<p>I only wanted to publish Images which are approved for release. Thats a simply
process of tagging within Git. Any tag be beginning with <code class="language-plaintext highlighter-rouge">release-</code> is assumed
to be intended for release. Pushing that tag up to GitLab will result in the
the CI running as normal but the image also being published to the Registry.</p>
<div class="language-yaml highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="na">image</span><span class="pi">:</span> <span class="s">docker:stable</span>
<span class="na">variables</span><span class="pi">:</span>
<span class="na">DOCKER_HOST</span><span class="pi">:</span> <span class="s">tcp://docker:2375/</span>
<span class="na">DOCKER_DRIVER</span><span class="pi">:</span> <span class="s">overlay2</span>
<span class="na">IMAGE_TAG</span><span class="pi">:</span> <span class="s">$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG</span>
<span class="na">MYSQL_ROOT_PASSWORD</span><span class="pi">:</span> <span class="s">secretpassword</span>
<span class="na">MYSQL_DATABASE</span><span class="pi">:</span> <span class="s">testingdb</span>
<span class="na">services</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">docker:dind</span>
<span class="na">build_and_test</span><span class="pi">:</span>
<span class="na">stage</span><span class="pi">:</span> <span class="s">test</span>
<span class="na">script</span><span class="pi">:</span>
<span class="pi">-</span> <span class="s">docker run --name=mysql -d -e MYSQL_DATABASE -e MYSQL_ROOT_PASSWORD mysql:5.7</span>
<span class="pi">-</span> <span class="s">docker build --pull -t $IMAGE_TAG .</span>
<span class="pi">-</span> <span class="s">docker run --rm --link=mysql:mysql -e RAILS_ENV=test -e DATABASE_URL=mysql2://root:$MYSQL_ROOT_PASSWORD@mysql/$MYSQL_DATABASE $IMAGE_TAG rake db:test:prepare</span>
<span class="pi">-</span> <span class="s">docker run --rm --link=mysql:mysql -e RAILS_ENV=test -e DATABASE_URL=mysql2://root:$MYSQL_ROOT_PASSWORD@mysql/$MYSQL_DATABASE $IMAGE_TAG rake test</span>
<span class="pi">-</span> <span class="s">docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY</span>
<span class="pi">-</span> <span class="s">echo $CI_COMMIT_REF_SLUG | egrep '^release-' && docker push $IMAGE_TAG || </span><span class="no">true</span>
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">docker login</code> was straight forward, but publishing only release tags was
more convoluted. This was because the Docker client image only has <code class="language-plaintext highlighter-rouge">ash</code> as a
shell instead of the full <code class="language-plaintext highlighter-rouge">bash</code>.</p>
<p>In short if <code class="language-plaintext highlighter-rouge">egrep</code> spits anything out, then <code class="language-plaintext highlighter-rouge">docker push</code> gets run, and if not
<code class="language-plaintext highlighter-rouge">/bin/true</code> is run to avoid the line being treated as a test failure by CI.</p>
<h2 id="summary">Summary</h2>
<p>With the above 20 lines of YAML any developer with suitable access can publish a new
fully tested release to the Registry ready for the SysAdmins to deploy.</p>
<p>Actually releasing just needs tagging a SHA in Git and pushing the tag upstream.</p>
<div class="language-bash highlighter-rouge"><div class="highlight"><pre class="highlight"><code>laptop:src <span class="o">(</span>master<span class="o">)</span>: git tag <span class="nt">-a</span> release-3
laptop:src <span class="o">(</span>master<span class="o">)</span>: git push gitlab release-3
</code></pre></div></div>
<p>The <code class="language-plaintext highlighter-rouge">.gitlab-ci.yml</code> file was largely identical for most projects. Some required
additional services like Redis but those were launched using standard Docker
Images, the same as MySQL.</p>Jeremy Wilkinsjeremy@jdwilkins.co.ukOn a recent project I’ve been working to standardise the deployment infrastructure across an assortment of Ruby, PHP and Java apps. Some on their own boxes, some using boxes shared across projects.Restarting this blog2018-10-04T13:19:08+00:002018-10-04T13:19:08+00:00http://www.jdwilkins.co.uk/2018/10/04/restarting-this-blog<p>After a 7 year hiatus its probably about time I started blogging again, I’ve a
few ideas for blog posts from some of the projects I’ve been working on recently.
Hopefully I can make the time to continue posting.</p>
<p>As to what I’ve been up to over the last couple of years - in a word, coding!</p>
<p>After several years of self employment, with ever increasing success of a product
for one particular client, I took on the role of Chief Technology Officer for
that client and helped build a team and workflow around the product
I’d built for them.</p>
<p>As is often the case with these roles I gradually ended up doing more
management and less ‘architecting’ and software development. Eventually I
decided after working on that product for a decade it was time for a break and
to try something new.</p>
<p><img class="fit image" src="/images/spain2018/IMG_20180216_182428.jpg" /></p>
<p>Ellie and I took some time off, went to Spain for a couple of months over the
winter, dodging the worst of the weather from here.</p>
<p>Now I’m back self employed, writing code and creating great products for clients
again.</p>Jeremy Wilkinsjeremy@jdwilkins.co.ukAfter a 7 year hiatus its probably about time I started blogging again, I’ve a few ideas for blog posts from some of the projects I’ve been working on recently. Hopefully I can make the time to continue posting.