DevOps in Database Development

When we speak of applications development today, we assume DevOps is an integral part of the software development cycle. Modern microservices-based architectures facilitate the use of DevOps and the benefits of this are well known – agile development, quicker defect resolution, better collaboration, etc. Through containerisation using platforms such as Docker and container orchestrators such as Kubernetes and DC/OS, continuous integration and deployment become essential and not optional steps in daily activities. PaaS offerings in Microsoft Azure like AKS (Azure Kubernetes Service) make management of the platforms even simpler and thereby encourage uptake.

However, while DevOps practices have become mature in the applications development sphere, the same cannot be said when it comes to database development. To be able to build a true DataOps team that can integrate agile engineering processes encompassing IT and data teams, a DevOps mindset is essential. Many large enterprises as well as small organisations continue to follow age-old practices for developing data-related artefacts and as a result, we still see a lack of agility and at times, poor quality.

Microsoft has invested heavily to ensure that database developers can also leverage the benefits that have been reaped by application developers. Today’s SQL Server development IDE, SQL Server Data Tools (SSDT), comes loaded with features that enable a development team to collaborate and follow good programming practices. When combined with Visual Studio Team Services (VSTS), we get the environment needed to engender a DevOps-focused development culture.

Six Steps to DevOps

At Talos, we believe DevOps is a foundational step in ensuring high-quality outcomes for our clients as part of a Modern Data Platform. Therefore, we make use of the toolsets made available by Microsoft in our development activities and adhere to strict policies, which are enforced by the tools. If you are looking to enable a similar culture in your database development team, consider the following guidelines –

1. Version Control – Use a distributed version control system like Git for your database code. Git is ingrained in SSDT and VSTS, and for those who prefer the command line, Git can be used in a PowerShell window. Once you’ve set-up a VSTS environment, make use of a SQL Server database project in SSDT for your database development and sync it with Git.
2. Branching Strategy – Start with a simple branching strategy in Git. There is no one-size-fits-all approach for this, so you’ll need to pick a strategy based on the complexity of the project and the size of the team. As an example, in addition to the master branch, create a dev branch and have the development team work of this branch. Create pull requests to merge the changes into the master branch. Ensure that the master branch is always stable.
3. Development Environment – Consider making use of SQL Server 2017 hosted on Linux in Docker as a development instance. The containerised SQL Server instance is quick to boot, tear down & replace. PowerShell can be used to issue docker commands, or Kitematic can be used if the preference is for a GUI. 
4. Continuous Integration – VSTS can be configured for automated builds which can be triggered when changes are committed. Configure continuous integration on the dev branch to ensure that the database builds successfully on every commit. 
5. Continuous Deployment – Automate publishing changes to QA environment. This will allow testing to commence as soon as changes are committed successfully. When the process becomes mature, deployment to production can also be automated.
6. Policies – Ensure access to the branches is only given to those who need it. Apply strict policies such as requiring a successful build as a prerequisite for a pull request to succeed. Automatically include code reviewers who would need to approve the changes before pull requests can be completed.

These initial steps will ease the team into the DevOps culture. Look to get these steps right before moving to more advanced areas like automated unit testing, NuGet packaging, coupling database with application changes, etc.

Through the use of a combination of mature tools and strict practices, a DevOps pipeline for database-related development activities is no longer a pipe dream. As MapR’s Chief Architect Ted Dunning has predicted, a sophisticated DataOps team comprising of data-focused developers and data scientists will be the way of the future (MapR press release). Sound DevOps practices will be the first step towards getting there.