What is the role of the DBA in a world where Cloud is being adopted more and more each day? This is a question I’ve had to answer multiple times in various sessions and trainings as well as consulting. The fact is the answer is very straight forward but at the same time not an easy one to hear. I’ve spent the better part of 18 years as a purely Microsoft SQL Server DBA. Over the years I’ve learned and implemented a number of solutions that were focused primarily on in-house or on-premise systems. My first interaction with the “Cloud” was in 2011 admittedly not the first wave and also on AWS but at the same time very soon I found out that Microsoft’s Azure was making a massive push to the Cloud as well. After the initial teething troubles, I found the Microsoft Cloud to be a far more user friendly and enterprise ready solution then AWS. Frankly I have no preference for one over the other and my decisions are based mainly on the client’s needs rather than my own personal opinions. The question here is not about cloud in Azure versus AWS. It’s more about the change in job role for the database professional as the databases are moved to the cloud.
First let’s talk about the role of a DBA. I’ve seen that the database professional has had to go through a massive change in the definition of what it means to be a DBA. Initially we only had to worry about OLTP systems. When I started my career, it was all about writing the SQL queries and creating databases and designing tables. In a short while I moved on to creating reports and dashboards, data flow transformations as well as analysis services and crystal reports. Before I knew it, I was doing BI. After a decent stint doing BI and being fascinated with data, I moved onto administration which included things like backups, tuning, HA and DR implementations etc. At this point I was considered a fairly senior resource and I was deluding myself into thinking I understood SQL server well enough. The truth was I barely scratched the surface and I felt that there’s no way I can keep up unless I kept learning. And so, I did. I delve deeper and deeper into internals, design best practises, security best practises and much more. The more I dug the more I realised that SQL server is a product on which I can base an entire career on! The myriad of tuning options itself was enough to guarantee a career. And so, I started my company providing database consulting services and BI. Little did I know in a span of two years the vigour with which Microsoft would embrace the cloud would make most of the things I did much easier to do as well as hassle free. It started out with a simple backup in another SQL PaaS database. A client who shifted to the cloud didn’t have to worry about DR solution and capacity planning / provisioning; it was taken care of by Microsoft. As I had just started my company, I figured well that’s just one among the many things that’s SQL Server can do. I’m still safe. There was a lot of reporting to be done and I’m sure that nobody is going to be able to take that away from me. Suddenly I started hearing rumours about a product called PowerBI. This was when I was still creating reports in SSRS. I figured PowerBI is going to be one of those niche products that very few customers would actually end up using after all; why should they when they get SSRS for free. Once again, I was proven wrong. Before I knew it more and more of my customers were inquiring about dashboards and reports in PowerBI. I figured it’s OK it’s just reports there’s still a lot of database design and tuning that I can to do to keep work streaming in.
The more I thought about it the more I was convinced that database design definitely cannot be taken away from the database professional. But I was seeing more and more systems being implemented using entity framework or other ORM tools where design certainly was being implemented by .Net professionals. I am not going to comment on the efficacy of these designs and only intend to point out how things don’t always turnout the way you assume. I figured well databases are always going to be around and as long as database are around, I’m sure that the best among us will always have something to do. And suddenly I started hearing things like big data non-relational databases and NOSQL being used more and more. I wondered. This new type of database design principle surely could not be applied to the majority of the database systems available in the market today. After all the number of companies that could justify a proper database system that truly is BIG DATA are few and far between. I ignored the big data trend to my own detriment for a very long time I assume that majority of my customers being start-ups or small organisations wouldn’t necessarily have the use case scenarios that big data caters. Boy was I wrong.
While I can still console myself every once in a while, the Big Data didn’t really align with the road map that I had for my company the fact is the revolution that followed Big Data was definitely well within what I thought my company should be doing. The advent of big data unleashed a whole new job role called the data scientist or data engineer. This was something that I’ve always been doing using integration services, analysis services and reporting services however these tools suddenly felt very out of date compared to their cloud counterparts. The flexibility and convenience offered by cloud-based solutions when handling very large volumes of data was miles ahead of anything on-premise systems could do. Just as quickly as Big Data had become a sensation suddenly AI and Machine Learning was starting to grab the imagination. I used to think what’s so special about AI and machine learning we’ve had these features in Analysis services way back in 2005. It took some time to connect the dots between a technology that was way ahead of its time and the democratisation of machine learning. Suddenly TSQL was being replaced by Python. Analysis services models were being replaced by Libraries like SciKit Learn.
Hopefully by now you can see the pattern. When it comes to being a database professional there is no steady state, there is no finish line. The reason I say this; is because like most people who ask the question at the beginning of this post, I’ve done the same. It was extremely challenging for me to accept the time and energy I spent developing SQL skills for the better part of my career was slowly becoming irrelevant. With every release of SQL Server Microsoft was bringing in more automation and more convenience that you really didn’t need as many DBAs as you used to. It’s only natural that Microsoft would do this considering the number of databases they need to manage in the cloud. At some point you reach the cut off where the number of DBAs you hire to maintain the infrastructure far exceed what you would be willing to spend maintaining those services. Automation is a trend we see not just in the cloud but everywhere. It brings efficiency, reliability and cost savings. Any company that can say no to these three benefits is probably not ready for the future. An unfortunate side effect of automation is obviously the toll it takes on the working professional. In fact, the best DBAs I know have already been doing automation by scripting out routine tasks and implementing packages and schedules to do their job for them. The cloud has just taken it to the next level. So, the question becomes if there is a shelf life for the DBA what should he/she do? As you can see the field has changed significantly over the years and anybody who learns to keep up will find themselves in a position to be considered for the next opening. It’s only common sense.
There are a number of popular authors who have talked about how the role of the DBA changes into a more niche role specifically around performance tuning etc. But that’s a very short-term outlook. The fact is the whole database landscape is changing drastically and there are two types of people the first type who still stick to the prior assumptions of database design like a deer in headlights. Or those who can see the change in tides and are already rowing in the direction of the future. As you can see, I have made some of the mistakes that I am pointing out others have done. The key difference being a willingness to understand a mistake and take corrective action. If you’re under the assumption that it’s too late to change then most likely it is. On the other hand, if you’re willing to see this as an opportunity to see how you would fit in a new world order then the world is your oyster. It reminds me of the time I saw my first computer long before any other family I knew had one. My father somehow was able to understand the significance of a computer on the Internet before others from his generation. That kind of fundamental shift is what’s happening in the industry right now and database professionals the captains steering the different boats as industries adapt two AI and machine learning which in turn is being enabled but the adoption of cloud.
Here are some stats for you to consider.
Source https://hostingtribunal.com/blog/cloud-computing-statistics/
After a COVID like pandemic, adoption of cloud-based solutions might not even be an option for most companies as they struggle to cut costs and adjust to a new way of working and living. The flexibility the cloud has provided as helped a number of companies adapt faster than their rivals.
So, what is a DBA to do in a world where the cloud is becoming a predominant force? The simple answer is “stop calling yourself a dba” and start thinking data. Be the guy who your company comes to for all things related to data not just relational databases not just tuning and administration but everything, basically, anything that relates to data. You might ask isn’t that biting off more than you can chew it probably is but at least this way you won’t starve.
Don’t get me wrong I still Love MS SQL Server and RDMBs but I also loved the feeling I got when I saw a PlayStation for the first time. I could never go back to my Atari. If you are a DBA don’t be disheartened; the game is still the same and the rules too haven’t changed much. We have just moved to a bigger arena.