Development and Operations are both critical to IS/IT usage in an organization, but operations often look at development as something alien – from another planet (and developers have the same view on operations). Having worked in both development and operations I often claim that development doesn’t know operations and operations doesn’t know development.
Let’s have a look at some common operation opinions about development (yes it’s exactly the same list as the developers’ opinions about operations).
- They don’t understand deployment.
- What they do is unqualified compared to what we do.
- How hard is it really to build a proper backup solution?
- Just get those firewall rules right!
They don’t Understand Deployment
In any large scale operations environment deployment has to be extremely well structured. There are often hundreds of systems installed across hundreds of servers. To save some money, systems are preferred to share servers, e.g. a web server. To get that working it is critical to know the exact requirements of each system. If one system requires an update to .NET 4.0.3, it might break something else running on the same server requiring .NET 4.0.0. Developers are working hard to get one system at a time working. Operations are responsible for keeping all systems working at the same time all the time.
What developers also often doesn’t understand is that large scale deployment is highly automated. There is no space for manual fixing during installation. Even if it is really simple to do on one machine when logged in, it has to be automated to work in a large scale environment. To correctly package different software into installation packages that can be automatically and repeatedly pushed out to multiple servers or client computers is hard.
What They do is Unqualified Compare to what We Do
Managing a large operations environment is complex, there are often hundreds of servers and there are thousands of lines of configuration files for central network equipment. To make things even more complex everything is live at all times. In operations there is often no way to test run a new firewall configuration. Each network segment has unique addressing (otherwise they would not be able to communicate) so each router or firewall needs to be individually configured. It is not possible to test a complete configuration in a test environment and copy it to the production environment. The unique configuration also has to be done on live equipment and there should of course be no downtime to the users…
The responsibility of keeping everything running all the time is the reason why any operations team worth their salt would never give developers access to the production environment. If things go wrong and a critical system stops operations are the ones who are blamed. They have the responsibility and should be the ones having complete control of the environment. As a developer I feel safe when operations don’t give me administrative access to the production environment.
How Hard is it Really to Build a Proper Backup Solution?
I’ve been in charge of budgeting for a small data center. What I quickly found out was that even though hard drives are nowadays cheap, professional storage isn’t. Professional storage with a proper backup solution is expensive. Designing a good backup solution is hard and it becomes even harder if the developers can’t tell exactly what files in what places that need backup. A good piece of software thoroughly separates application files (that can be reinstalled with the installation package), data files that require backups and temporary files that shouldn’t be backed up. Unfortunately many systems are written without operations in mind, placing critical files in the most unexpected places (what about a critical config file in the profile directory of the user that installed the system?)
Backups of a database is a separate issue. The recovery model has to be taken into account, to get proper backups and truncation of transaction log files.
Just get Those Firewall Rules Right!
A good operations department of course splits the network into different security zones. When a system is first deployed there are often problems with the firewall configuration. If operations are lucky, the developers at least left a list of services that their system requires. Unfortunately that is not how firewalls are configured. A firewall only knows about protocols and port numbers. If the developers has specified “database access” operations may assume that TCP/1433 is required to be open from the web server to the database server. When the application mysteriously fails on some data updates (but not all) it is found out that MSDTC is required which requires both OS and firewall configuration. Without information from the developers it’s impossible for operations to know that they need to enable MSDTC.
Operations is different from Development
Having worked with both development and operations it is clear to me that both are complex, but yet very different. Development dives deeply into the complexity of a single system. Operations take all the complexity of getting different system working in a common environment, while having everything running continuously.
To improve the cooperation between operations and development a better understanding for the other side is needed. I find DevOps really interesting as it removes the traditional division between operations and development. I’m sure that it can really improve the entire IS/IT service offered to an organization. I’m also sure that it’s hard to do properly as it requires a level of cross-competence between operations and development that is very uncommon.
These two articles have been great! Even as just a developer, I’ve seen so much of this personally – right now I’m working as a developer in a software company, but previously I was on a development team embedded within operational IT.
One of the constant challenges we faced was that the overall management structure of the IT department was completely operations-focused. Huge amounts of knowledge on server administration, networks, desktop support, etc, very little on SDLC or any kind of development process. In fact, the CIO’s official description of our team was “application administrators” for IIS and ASP.Net on our intranet web servers.
Change control was impossible – there was actually a very efficient change management process, but it was all based around things like “Windows updates are scheduled on the primary exchange server – who does that affect, what’s the rollout plan, what’s the backup plan, when will it be scheduled, etc” rather than “we’ve been asked to introduce new search functionality to this custom web application and we need to plan a release date and dev cycle for it”.
I to have enjoyed reading the two posts. It’s alot i do agree with, especially the thing that the understanding of eachothers world isn’t good enough.
I don’t agee with the “What they do is unqualified…”, I’m convinced that most people that have some experience in computers understand that everything is complex. But in different ways… :)