Vision is the hardest

I started off my career thinking that coding was the hardest thing I had to do every day. That fixing bugs, building features, and writing documentation was the toughest part of building software. It’s what I spent all my time studying in school, so I figured that once I knew how to code well enough to make things work, the rest of the job would be easy. I heard “The first programing language is the hardest”, so I expected to struggle with the basics. I figured that classes would teach how to be a successful programmer, but then I saw enough fellow students struggling to get projects working or even compiling that I assumed that the implementation details were individually more difficult than what could be taught. I thought I’d eventually run into a line of code or piece of hardware that could only be overcome by pure “programming skill’. So I memorized algorithms and learned every layer of the stack so that I’d be unstoppable. Eventually I got enough experience building different kinds of software and realised that technical limitations very rarely make the project difficult. Most projects can be accomplished without knowing any fancy algorithms or any details outside the current layer. The layers of complexity weren’t leaky enough to sink a project by themselves. It just took time to learn a few tricks to get from one step to the next and eventually finish. The projects I saw that did push the limit on technical accomplishments weren’t all that much more complex than other business applications; they used the same tricks in more clever or constrained ways. It was difficult to maintain a level of technical excitement as I looked at the big fancy projects, the more they looked like lot of other less shiny software that wasn’t doing anything amazing. The magic behind programming lifted as I became more comfortable behind the curtain. I started to run out of the “wow, I didn’t even know you could do something like that” moments. I’m still impressed by cleverness in code, but I’m rarely surprised in a good way. It was never the code itself that was impressive, it was what the code was designed to do that made it impressive or annoying.

So with some level of technical understanding behind me, I started looking for new fronts to explore. I understood that no individual snippet of code made a project tough, but I knew there was a level of complexity that I wasn’t grasping. I knew that building good software had to be tough in some sense or there wouldn’t be so many failures! I took the lessons in bug hunting up a level and noticed that it was how the technical details were organized in the large that caused problems: the architecture and design. This was clearly tougher than any individual piece, since most of the bugs I saw weren’t in a single component but in how many separate pieces interacted. These higher-level aspects were much more difficult to understand because the abstraction was often subjective. Unless the class name contained the design pattern, it would have to recognized by its interface or place within the system. UML doesn’t execute and as a road map it was only as good as the map makers and bricklayers were diligent. Sometimes the design and implementation were practically in conflict! This was a tough hurdle to climb in comparison to just knowing syntax and some debugging strategies. It was more difficult to learn from these abstractions because they occurred less often and were often subtle in how they affected the quality of implementation. It was difficult to decide if it was the design or implementation that was the key to understanding a specific behavior of the system. Often counterintuitively the business model had little in common with the technical model. I frequently ran into models that made perfect sense on paper, but were unrecognizable in code and beautiful code that made no sense in the bigger picture. While having an implementation in mind ensures the feasibility of the design, it sometimes blurs the line between what is inherent to the design of the system and what is just a convenient abstraction for programming the cases at hand. Coding in the large is where knowing only one language or a few tricks causes the whole plan to be just another layer of pseudocode instead of a mental framework for understanding and modeling the problem at hand. Outlining the final code is good, just as higher level abstractions are good, but just applying a cookbook of patterns only works if the patterns already fit the problem. And even then, applying the same pattern over and over without improvement is short sighted for the huge gains that software has taken in some spaces. These problems of both philosophy and practicality don’t arise when coding one line at a time. The ability to look and think beyond that initial scope is a valuable skill to practice, even if it is difficult because of the lack of opportunities to test and receive feedback on what works and what hurts later.

Good design in any field is challenging, but that wasn’t what made programming systems difficult in my experience. I began to see that many people didn’t struggle with the technical details even when the big technical picture was a mess, but instead failed at the human aspects of planning and communication. These weren’t directly technical at all, but have a massive impact on the technology as it is put to use. The ability to organize and execute are tied together for everything but the smallest of teams. Without proper communication (where the definition of proper is already difficult), the design has no chance to be realized and the implementers will be stuck with the tough problems from the start. Organizational critiques to determine effective communication can be tough to understand because each one is situational. I didn’t have a large enough personal dataset to make any generalizations on how or why these sorts of problems came to be and how best to solve them. The planning and resource estimation component of project management is well understood in industries outside of software, so I’ve been able to better grasp those patterns of solutions by expanding my horizons. The individual aspect of time estimation alone is difficult, and compounding that project management in software engineering is still a hot topic for both the technical and non-technical sides. So I figured that this is why software is so hard. But once I started managing that complexity, I saw and executed a few project plans from nothing to a steady-state of maintenance and sunset. It was a hard problem because dealing with groups of people can be difficult, but it wasn’t what made software engineering difficult.

The more I looked for an answer to why software development was specifically difficult in comparison with other areas of technical project management the more I had to look at the motivations of a particular application. I eventually saw that the project planning aspect was a byproduct of the business’s vision. Having little experience in setting successful business goals or vision, I haven’t yet overcome this hurdle. Vision is the technical and product foresight to understand where the software product needs to go. The manner in which it can get there is then dependent on all of the other steps: management, design, and implementation. If you don’t have a good vision, it’s very difficult to recover the quality in any following steps. The resource planning fails because the scope is unclear, the design becomes messy as the assumptions and constraints change, and the implementation is then incomplete and can’t even meet the minimum of the vision. That top down view has a massive hidden impact on the smallest detail, where you could almost infer what the vision was by working up from the implementation. Looking at what makes the implementation special can then help guide the vision with a long feedback loop. “This has all the features of a note-taking app, but it’s not branded as such, maybe we should start thinking of this product differently’ or “this application isn’t leveraging any of the features from the big gui framework that it started with, maybe it would be better as a service with a separate UI product”. Those sorts of insights require a full view of the technical stack. Just because a product functions in one way, doesn’t mean it’s modeled that way underneath. There are plenty of applications that could probably support a full existing scripting language if they looked at their internal DSLs that are built on top of them. (Greenspun’s tenth rule) With this I’m now focused on the holistic aspects of software engineering, every component is important, but it all comes from a good vision. What makes a good vision? I’m still working that one out.