Suggested Reading: Programming
Today I have a few goodies for you. The first a well thought out response to the increasing complexity of PHP frameworks. The second, a now well known reactionary response to the evolution of software engineering and project management methodologies.
PHP has been around for quite some time. As an open source programming / scripting language that snaps into the world’s most popular open source web server, it is easy to see why it has become the world’s most popular web application platform. With its absolutely massive developer community, rich documentation and examples and tutorials abound it also happens to be one of the easiest languages to learn. With this low learning curve, you can get into PHP rapidly and start cranking out all kinds of strange things with security issues you have never even imagined. People see this problem as well as issues with repetitious code and they tend to gravitate toward application frameworks. Over time these frameworks, beloved to their users grow and grow. ”Now it can do everything!” an excited user base will chant. But what they tend to not see so well is that as these frameworks grow, their barrier to entry skyrockets. The conversations go from, “Hey you should try this tool out. Its easy and reduces the code you have to write.” to, “In order to do that you should do, this… Make use of that pattern… Oh, you’re not supposed to use it that way… ” Funkatron posed their response quite eloquently at http://funkatron.com/posts/the-microphp-manifesto.html. Are thin frameworks too much to ask? I’d say that most things start thin, but rarely remain so for long.
Application design and programming are both something I like to call creative engineering. They are both also active processes. These activities tend to appeal to a certain type of person. A person who gets excited building things. Someone who demands results. They do not write code for the satisfaction of sitting in front of a computer for days on end or for the ever fulfilling notion of growing their asses. These are a means to an end, some deterministic result. We grow up in a self driven, wholly unstructured environment where we learn how to externalize ideas, turn tasks into algorithms and really create from our own Will. Then it happens. Grow into a world where we work with other people, and we all work for someone else. The creative engineering process meets communication constraints and business expectations. Rarely does it happen that any one person can understand all three worlds of thought. In recent years we have all witnessed the world of project management and software engineering practices evolve more rapidly than ever before. The shortfalls of every step in that evolution have both helped to shape the next evolution and further soured everyone involved. I’m willing to state that I’m rather optimistic about the whole thing these days. In my opinion things are a world better than they have been in the past, and that I would do anything to keep from reliving the dark days where management and engineers were separated by an anoxic barrier of mistrust and contempt. For those with a more purist view Zed A. Shaw presents to you, an alternative project management and software engineering practices philosophy, Programming Motherfucker (http://programming-motherfucker.com/). Its both humorous and a pointed counter movement of creative engineers who are tired of elongating the process to get to their precious results. Who can blame them. While managers and executives measure the cost of development in man months, they measure it in ass growth. Who do you think is more motivated to finish a project?
Spoils of Success and Artificial Availability: SOA Anti-Patterns
I'm afraid. I'm afraid that I've been working hard to solve the wrong problems. Its not just me. I'm afraid that a ton of engineers have been working on solving the wrong problems. I mean, they feel so right. Over the last few weeks I've learned a few things that I had previously either assumed I understood or had never thought about long enough.
Artificial limiting factors
We build things because we want people to use them. We work very hard to write the software, buy hardware, market our product, evangelize adoption, assist with migrations, and provide support. And then, sometimes, when we've succeeded, we start to forget how hard we worked. We start to see growth or increased adoption or unexpected use as an enemy. As an engineer, you start to think something along the lines of, "Well our customers shouldn't be using this so much." Or maybe something like, "This is an abuse of the system." That is when it happens, and you start thinking, "How could we build a system that protects us from our customers?" Building systems that enable you to meter, throttle, or rate limit your service are an admission of defeat in all but the most extreme circumstances. With very few exceptions, if you find yourself reaching for one of these solutions, you are considering the wrong problem. Beyond the obvious poor experience you are creating, you're "protecting" your systems from innovation.
Pride
Take pride in your work. If you built a system that is so successful, that people want to use it, make it even better. Scaling issues are the best issues to have. Take those challenges head on. Partition, shard, pre-compute and cache your way to a solution. Make sacrifices to do more work, but deliver more consistent customer experiences. Raise rate limits, reduce customer expenses, and deliver an experience that customers trust.
Scarcity, Exclusivity and Adoption
When the thing that you're gaiting is the adoption of your service or software, be aware of the sweet spot. This practice is common and a reasonable mechanism for protecting QoS for customers of a new operation during scaling phases. But there is a sweet spot. Grow as rapidly as you can handle, and be loose with viral personal referral growth. Referral invites bring more committed customers than marketed or media references. Aim to grow exponentially. If it takes too long for interested parties to get in, the likelihood that they will be satisfied with your product is slim.
Actual limiting factors
If you must limit use, consider a business solution like tiered pricing. In these cases, the limits are set by the customer. They know how much they want to spend, and in B2B scenarios, you'll find that your customers are happy to make some monetary sacrifices.
An Example: Wireless Networks
Lets consider the wireless network example. Over the last decade or so, with the advent of the smart phone and mobile computing, wireless networks have experienced this huge story arch that takes their character from innovative liberators all the way to hideous tyrants in the eyes of their customers. In the early days, market forces pushed reasonably priced unlimited data plans. We were all free. Customers could go anywhere and access our data. The iPhone and subsequent Android boom enabled us to consume more and more of this data, and we fell in love. Then something odd happened. The networks realized that they were having a difficult time keeping up with demand for bandwidth. They didn't just want more money. I mean, we were hooked. They could have simply raised their prices if they wanted more money. Instead, they started instituting data caps. Beyond which the cost of data is so high, that nobody would reasonably want to pay for it. The effect of such a policy is behavior correction. They are so spoiled by demand, that they see no reason that they should improve their service. They would rather create a poor customer experience, and that is what they have done. This problem has only become worse since the adoption of 4G. Many phones do not simply prefer WIFI over wireless, they prefer the connection that will deliver data the fastest. In many areas, the winning network is wireless. The result is unsuspecting customers with massive phone bills at the end of the month.
Artificial Availability
I want to talk about failure modes. Regardless of how far your service or software is from a customer facing edge, you likely built it with the intention that it should be maximally accessible and available. Furthermore, your service or software likely has some service or data source dependencies. These two facets are really at odds. As cliché as it is to point a finger, dependencies are a real bitch for your availability metrics. If one goes, so does your service. So, what do we do? Well, sometimes we put forth a huge effort to try to make the chain stronger than its weakest link. Some might try to move to an asynchronous processing approach for to service data mutation requests. Others might just indefinitely retry internal service requests indefinitely, or until the client goes away. But both of these coping mechanisms actually do more to damage the system than good. This is a case where we are solving the wrong problem.
Lying to a client is bad
Advertising a service call as a synchronous call, and then handling it asynchronously internally is lying. The reason is simple: asynchronous processing is shit without a reliable feedback channel. Fake-synchronous processing and internal backlogging in order to handle dependency outages will destroy customer expectations and forms the foundation for unstable systems that fail big. This is all about managing customer expectations. Systems fail, it is unavoidable. Your customers will be much more appreciative if you clearly let them know when it happens, and how long you expect the outage to be rather than pretending it doesn't or never did happen. If your software is worth using in the first place, they will try again. Especially where expectations have been clearly managed. Take the outage. Communicate. If you're going to put effort into improving the system, put that effort towards improving the accessibility and availability of your dependencies.
What do we do instead?
What you should do really depends on the software. On accessor type operations, delivering cached results might be an acceptable option. Maybe you could even leverage more available but eventually consistent data stores. On mutation operations, just do everyone a favor and fail fast. The faster the better. If your database is down, don't bother it with continued customer polling. Short-circuit those requests until service has been restored. Get creative, but make sure you're addressing the real problems.
The Right Problems
The right problems are the ones that impact your customer first. The right solutions improve the experience for your customer first. These solutions also have the benefit of being the most innovative.