15 Jan 2016
What happens when you want a private non-routable network such that you would use for a cluster heartbeat or in this case, a vertica spread network, on AWS?
If like me, you use DHCP when attaching an ENI to your EC2 instance, then you will find that when your linux (centos or redhat distro) executes ifup-eth1, the default dhcp options will mean the default route will be set to this network.
The second network card in this configuration is on its own private non-routable network, and with AWS there is no way to remove the default gateway for this network. The best I could find was an article stating you could use ACL (access control list) to restrict the network to that subnet. This is of little use if your default gateway has been set to a router that blocks all traffic.
There is an answer however. The DHCP client looks for config files, and if they are not found, uses defaults.
The fix for my private non-routable network, create a file that looks like this:
/etc/dhcp/dhclient-eth1.conf
request subnet-mask, broadcast-address, time-offset,
interface-mtu;
This little file means only request the basic info from the DHCP Server. If we don't ask for the router, then we aren't going to get it.
(Read more...)
05 Jan 2016
http://twitter.com/ProJavaScript/status/606262588725456897/photo/1?ref_src=twsrc%5Etfw
Technical debt is created when the 'proper way' to do things is not followed, or to put it another way, get it working and we'll fix it up later.
Does that 'later' ever come along. I know in the race to deliver value, these small items can be overlooked, especially if the question is 'fix debt' or 'ship new feature'.
Unfortunately these small debts can add up until the status quo is tipped in the balance of technical debt dragging your team down.
This can look like
- more support calls coming into the helpdesk
- more support impacting the call out rota
- longer delivery times when adding new features due to work arounds for that bug that will be fixed in the next version
- bad publicity due to slow bug fixing
- staff turnover due to quality team members losing faith in the team.
The first step in managing this technical debt, is to make it visible. Add it to the issues list and backlog, and help the product owner understand what a pain it creating during the development process.
Once it is visible, it can then be prioritised or perhaps the decision can be made to live with it. At least it is a conscious choice.
This series on the topic by the sharp folks over at 18F draws a nice, clear picture of what it is and why you need to do something about it.
(Read more...)
31 Dec 2015
For the last couple of weeks I have been working on a puppet manifest to enable more resilience and build of vertica nodes on AWS.
Using automation tools has allowed the fast prototyping and development of many software solutions. Using puppet and Jenkins, a full environment is provided for the Data Analysts to use within about 1 hour.
Part of what I've been working on is to build vertica nodes whilst retaining the data. In addition to this, building in the logic to enable a node to fail, recover and join the cluster with next to no manual work.
Monitoring and detection is useful in this phase, so the database admins can double check the health of the data during and after the failure. Part of the solution includes automatically configuring the Vertica Management Console ready for use.
More progress is expected in the next month as I build auto scaling on top of this resiliency.
(Read more...)
30 Dec 2015
Within a project team you require a number of skills usually found across a number of team members.
Developer - This is the programming and design skill required for the end product this project is sponsored to deliver. For a web application, this might be Javascript, PHP, Paintshop etc. For a data analysis BI team, these skills will be more like R, tableaux or Qlikview.
Testing - These skills are about ensuring the best experience for the end user or customer of the deliverables. Performance (the speed of the product), usability (if it actually works as designed) and load testing (how many users or how much data can be provided) are important measurements in this area.
Operations - Providing the IT bits that the project and application rely on to function and serve. These are concerned with the servers, networks and other dependencies.
In terms of what DevOps engineers do, well that spans the developer and operations skills. The reason it is so specialist, is the blend of these skills. Someone who understand programming and infrastructure.
My team and I spend the majority of our time, writing and refining puppet modules. These are definitions of what a server needs to do, in order to provide a server, service or piece of infrastructure. They are programs in their own right.
We also spend time on writing tools to help the developers and testers move through the deployment processes. We don't want to slow the development or project down, so providing tools to enable the developers and testers to 'self serve' allows that flexibility.
Automation of the project steps and providing tools, buttons, one-click actions, to move the project and help the developers write, test and deploy code as easily as possible.
(Read more...)
21 Dec 2015
What is DevOps?
The best description I have seen is from Kurt Bittner;
"DevOps optimizes the software delivery pipeline, all the steps that you have to go through between when you have an idea and when a customer starts benefiting from that idea. In the traditional delivery processes, you have lots of hand-offs, lots of stops and starts. You have relatively inefficient processes, and it can take months -- and sometimes years -- to go from idea to having somebody get a benefit."
Optimising, automating and streamlining that process is what we are all about. Embracing Agile with open communications to help your team deliver value to your customers, faster and with higher quality. From Idea to usable and valuable product.
Infrastructure -> Development -> Testing -> Delivery
How do I get started?
Unfortunately it's not any one product or training course you can buy. It's an idea and ethos, which requires executive management across silos. You can however start with a few steps.
Step one; Environments.
Changing the way that environments are provisioned. That includes getting environments provisioned on-demand, using techniques like infrastructure-as-code using technologies such as Puppet to automatically generate environments based on configuration settings so that you can have an environment anytime you need it. This needs a virtualised infrastructure such as VMWare or Amazon Web Services. That removes a lot of friction and a lot of delays.
Step two; CI and Testing
Implementation of techniques like continuous integration and then, after that, test automation, based on APIs. There's a shift to APIs on an integrated architecture for the applications, and then usually deployment automation comes after that. Once you have environments provisioned in code that you can put into those environments, you need a way to move that code between environments.
At this point you may find Silo inside the organisation can slow or even stop some tasks from taking place. Resource contention and task priorities can get in the way of delivering the release to your customers. Spotify offer an approach to this called Tribes, Squads, Chapters and Guilds.
There's a shift in team structure to become more product-oriented with dedicated resources to a product, so that you can release, and do release after release most effectively. That tends to break the organization silos down and start shifting to a more product-centric organization and away from a functionally oriented organization.
Step three; Continuous Improvement
We are very much evolving within the DevOps space, and your organisation will too. There are many products to help and many tools your teams could create, adopt or improve upon.
For more information how we could help you become more agile and proactive, contact us now.
(Read more...)
