Today, we're diving into an interesting topic: friction in development processes. I recently had a conversation with Tom about this, and it brought some new perspectives to light.

An Initial Confusion

During a call with our engineering team, our CEO Tom suggested adding a bit of friction to our processes. This was strange because I always want to remove friction to make things smoother and faster. Velocity is crucial for us— Tom harps on how we want to move quickly and efficiently. So, why would we want to add friction?

Friction Generates Heat

While friction does slow things down, it also generates benefits.

For example, we conduct code reviews, which slow down development but lead to increased stability, better product alignment, and reduced technical debt. These processes, though seemingly slowing us down, actually contribute to a more sustainable and team-involved development

This reminded me of a decision made at at a security awareness company I used to work for. We conducted typically annual security awareness training on a monthly basis. We had data from our customers that conducting these tests/lessons quarterly had a significant improvement over annual training, but monthly training didn’t really yield better results than quarterly training. So why did we do it monthly?

First, we wanted to make sure we never got hacked. It was an existential threat to the business if one of our own employees fell for a phishing attack.

Second, we wanted to make a point of how important this was and to build it into the identity of the company. It became part of the culture that our courses were espousing to our customers.

In essence, adding friction in certain areas can yield significant benefits. It's like in physics—more friction can generate beneficial heat. For us, it's about strategically introducing pauses and checks to enhance the overall process.

Slowing Down to Move Faster

Sometimes, we need to slow down in order to move fast. At first this makes no sense but there is a simple analogy that makes it blindly obvious: Formula 1.

The software development process isn’t a drag strip where you can put the pedal to the metal and go in a straight line. There are many twists and turns and you don’t even know where or when they’ll show up!

The prime example here is writing tests. Writing tests can seem like an additional, time-consuming step that slows down the development process. However, by integrating tests early and consistently, developers can identify bugs and issues before they become costly problems. This proactive approach not only improves the stability and reliability of the software but also enhances the team's confidence in their codebase. The initial slowdown caused by writing tests is offset by the time saved in debugging and fixing errors later on, leading to faster, more efficient development cycles in the long run.

In this sense, adding friction strategically to ensure you’re not driving off a cliff, or in software terms, bring down production actually lets you move faster than if there were no guardrails.

The Cost and Risk of Adding Friction

One challenge is maintaining a balance. Large companies often introduce excessive friction, leading to bureaucratic slowdowns. It’s important to focus on velocity while continuously re-evaluating and fine-tuning the process. Tight feedback loops and minimizing unnecessary latency are crucial.

Recognizing and Implementing Useful Friction

It's crucial to recognize when friction is necessary. Friction should not be added randomly but should serve a clear purpose. For example, implementing a mandatory pause between merging and deploying code forces developers to reflect and catch potential issues.

You should also continuously evaluate any added friction. If it's not yielding the desired benefits, remove it. Optimizing for velocity ensures that unnecessary friction doesn't become a permanent hindrance.

Final Thoughts

Friction can be incredibly beneficial when applied thoughtfully. It helps prevent errors, enhances security, and fosters a culture of careful consideration. However, you need to strike a balance to make sure you’re not grinding to a halt. If you add friction and slow down your process in the right places, you’ll be amazed at how much faster you can go!

Here is the full video interview:

What is Secrets Management?

Secrets management involves the secure handling of sensitive information, such as API keys, passwords, certificates, and access tokens. These secrets are necessary for various parts of an infrastructure to interact securely. For instance, databases need to connect to applications, and servers need to communicate with each other. Secrets management solutions ensure that these interactions happen securely and efficiently, without exposing sensitive information to unauthorized parties.

What is the Problem?

As software systems become more complex, the challenge of securely managing and distributing secrets intensifies. Different parts of a system—whether they are databases, applications, or servers—need to be securely interconnected. The traditional methods of managing these secrets are often manual, prone to errors, and not scalable. This can lead to significant security vulnerabilities, such as secret leaks or unauthorized access, which can have severe repercussions, including data breaches and compliance issues.

Different Approaches

There are several approaches to secrets management:

1. Manual Management: Some companies opt to manage secrets manually, using secure storage solutions like Key Management Systems (KMS). While this method can be straightforward, it often lacks automation and scalability, leading to potential security gaps.

2. Automated Solutions: Tools like Terraform Vault have become popular, offering automated secrets management. These tools provide a more structured and scalable way to manage secrets but can be complex to implement and maintain.

3. Dynamic Secrets: More advanced solutions offer dynamic secrets management, where secrets are generated on-the-fly and are valid for a limited time or a specific number of requests. This approach minimizes the risk of secret leaks and unauthorized access.

4. Integrated Solutions: Platforms like Infisical offer integrated secrets management solutions that provide seamless access control, secret rotation, and automation across different environments (development, staging, production).

Types of Companies, When They Adopt a Solution, and Their Motivations

The adoption of secrets management solutions varies based on the company's size, industry, and stage of development:

1. Early-Stage Startups: These companies often adopt secrets management solutions early, especially if they operate in highly regulated industries like finance or healthcare. The motivation here is to establish strong security foundations from the beginning.

2. Growth-Stage Companies: Companies at this stage (Series B and beyond) adopt secrets management solutions to improve operational efficiency and enhance their security posture. As their infrastructure becomes more complex, managing secrets manually becomes impractical.

3. Large Enterprises and Governments: These organizations have strict security requirements and often have specific policies on how secrets should be managed. Their motivation is not only security but also compliance with regulatory standards.

Security

Security is the primary driver behind secrets management. Effective secrets management minimizes the risk of unauthorized access and data breaches. It ensures that sensitive information is only accessible to those who need it, and it is done so in a secure and controlled manner. Implementing features like automatic secret rotation and dynamic secret generation further enhances security by reducing the attack surface and limiting the potential damage of a compromised secret.

Why Isn’t This Solved Already?

Despite the existence of secrets management solutions, many companies still experience secret leaks and security breaches. This is often due to the complexity of these solutions, which can be challenging for developers to implement and use effectively. If a secrets management tool is overly complicated, developers might bypass it, leading to security vulnerabilities. Moreover, the dynamic nature of modern infrastructures, with multiple environments and a myriad of integration points, adds to the complexity.

At Infisical, the focus is on making secrets management more accessible and user-friendly for developers, ensuring that security practices are embedded into their workflows without adding unnecessary friction. By providing robust tools and seamless integrations, Infisical aims to reduce the complexity and enhance the security of secrets management.

Conclusion

Secrets management is a critical aspect of modern cybersecurity, ensuring that sensitive information is handled securely and efficiently across different environments. While there are various approaches and solutions available, the key to effective secrets management lies in balancing security with ease of use. By understanding the unique needs and motivations of different types of companies, we can better tailor secrets management solutions to address their specific challenges and enhance their overall security posture.

Incident management is a critical aspect of maintaining the reliability and integrity of systems. Recently, we had the pleasure of speaking with Chris Evans, Co-founder and CPO of Incident.io, to learn how incident response has evolved, and how engineers can prepare themselves for modern incident response. You can see a recording of the full conversation here.

The Centralized Operations Team

Traditionally, incident management was handled by a centralized operations team. These experts monitored dashboards, identified issues, and coordinated responses. However, the advent of DevOps has decentralized this function, distributing the responsibility across the entire engineering team. Today, the very people who build and deploy software are also tasked with managing incidents, which introduces both challenges and opportunities.

The Decentralization of Incident Response

The advent of DevOps has distributed the responsibility of incident response across the entire engineering team. Today, the engineers who build and deploy software are also tasked with managing incidents. This shift has led to a broader, more inclusive approach to incident management, where everyone in the organization is involved in the response process.

Why the Shift?

This shift is largely driven by the need for faster, more flexible incident response. As software development cycles accelerate, the traditional centralized model struggles to keep pace. By involving the entire engineering team in incident management, organizations can respond more quickly to issues as they arise.

This shift to distributed incident management also means more people need to be trained in incident response. While this democratization of incident management can introduce complexities, it also enables organizations to be more resilient. When every team member is prepared to handle incidents, the organization as a whole can respond more swiftly and effectively.

Pros and Cons of Each Approach

The centralized operations team approach has its advantages, such as consistent processes and highly trained experts handling incidents. However, it can also be slower and less adaptable to rapid changes.

On the other hand, the decentralized model offers greater flexibility and faster response times, but it can introduce inconsistencies and require more extensive training across a larger group of people.

Hybrid Approaches

Many organizations are now adopting hybrid approaches that combine elements of both models. For instance, they may still have a central team that handles major incidents and coordinates responses across the entire organization (Marketing, Support, Customer Success, PR, etc.), while also empowering individual engineering teams to manage smaller, more routine incidents. This approach maintains consistency and leverages specialized expertise when needed, while benefiting from the speed and flexibility of a decentralized model.

What does this mean for engineers?

Redefining What Constitutes an Incident. For engineers, this evolution means a shift in how incidents are defined and managed. For example, at Incident.io, an incident is defined as anything that disrupts planned work with a degree of urgency. This can range from a minor bug affecting a key customer to a complete database failure. By broadening the definition, organizations can gain valuable insights into their operations and better prioritize their efforts.

Documenting and Communicating During Incidents. Engineers should document their debugging process in real-time, creating a clear trail of their actions and thoughts. This practice helps in troubleshooting and serves as a learning resource for other team members. Clear documentation can streamline the handover process if the incident escalates and requires additional support.

Leveraging AI for Incident Management. AI can automatically summarize incident channels, making it easier for teams to stay updated on ongoing issues. However, it's still important to keep humans in the loop. While AI is good at summarizing information, they’re not perfect. AI-generated summaries should be reviewed by incident leads to ensure accuracy and prevent the dissemination of incorrect information.

Running Effective Post-Incident Reviews. Post-incident reviews are crucial for learning and improvement. Serious incidents should always be followed by a face-to-face review. This allows for comprehensive discussions, clarifying any uncertainties and ensuring that the entire team benefits from the insights gained.

Shifting the Culture Around Incidents. A key takeaway from our conversation is the cultural shift required to handle incidents effectively. Organizations must foster an environment where incidents are not seen as failures but as opportunities to learn and improve. Encouraging open communication and collaboration across all departments, from engineering to PR and Legal, is essential.

For the full details and first hand suggestions from Chris, check out our video interview.

Because this contract can't change unilaterally, being precise and strict is extremely important. Undefined behavior is de-facto defined behavior, and if you are too lenient with what you accept, you will forever be stuck supporting all the unpredictable ways people use your APIs.

When talking about APIs I often hear people quoting the robustness principle: "be conservative in what you send, be liberal in what you accept". I strongly disagree with this principle, as it leads to fragile, insecure, and inefficient systems.

For example, consider an API endpoint that expects a query parameter called sort which according to the docs accepts the values asc for ascending, and desc for descending.

Here is an example psuedo-code implementation:

function list_items(sort: string) {
    if (sort === "asc") {
        return fetch_sorted_asc(...);
    } else {
        return fetch_sorted_desc(...);
    }
}

This function, as you can see accepts sort as a string, and then checks the value to sort the data accordingly. The problem is that the input is not validated to be one of two values, so in practice Desc, descending, and dscnding will all lead to the input correctly being sorted in a descending manner. Which in turn means that anyone that uses this API incorrectly like this will see it working correctly and therefore will start relying on this behavior.

This means we are now stuck supporting endless variations of desc until the end of time without even realizing. Which most likely also means we are going to break some implementations by accident when we change this code without even realizing it.

Additionally, it's easy to become more lenient later, but it's impossible to become more strict. So if you start more strict you keep your options open, if you start more lenient you don't.

Formulating the contract

As I said above, APIs are a contract, which is why it's important to formulate that contract as well as humanly (computerly?) possible. Therefore having an OpenAPI spec for your API goes a long way. Preferably also having validation that the actual implementation is compatible with the spec.

You should also be as specific as you can with both the definition, and therefore the validation of the fields. For example, the Svix user defined IDs (uids) follow a specific pattern: they have to be between 1 and 256 characters long, and match this regex ^[a-zA-Z0-9\-_.]+$. This is both documented in the spec and validated at runtime.

The advantage of having it documented in the spec, is that you can generate rich and specific documentation for your customers. Your customers don't need to guess what values are allowed for a uid, they can see it right there.

Validating the spec at runtime, when API calls are made, means that even if someone made a mistake and accidentally passed the wrong value, this will be caught immediately and they would be able to rectify the mistake.

Another advantage of having an OpenAPI spec, is that your API consumers can also generate validation on their end. Having schemas type checked on their end at compile time, so issues never hit production.

Tagging IDs with their type

Many APIs use uuids internally as their ID representation, which they then expose in their APIs. Those uuids usually look something like this: 12455207-ab83-5602-8d93-67999e204cff. First of all, I think the base62 representation (not a typo, it's 62) is much nicer: 2bd6zcCxa8v3fx1nj9XVlQp3WR3. It's more concise and more aesthetically pleasing, but to each their own.

The problem with using uuids, whether in their standard string representation or the base62 one, is that they are just generic IDs. This means that you can accidentally use an ID in the wrong place (which will usually return a 404) and be very confused on why it doesn't work.

A much better system is to tag IDs with their type, so the example above when tagged would look like this: app_2bd6zcCxa8v3fx1nj9XVlQp3WR3 for an "application" and msg_2bd6zcCxa8v3fx1nj9XVlQp3WR3 for a message.

Tagged IDs mean that you can have strict validation in your API and great examples in your docs that show exactly the kind of ID that's expected. Passing the wrong ID will no longer result in a 404, but rather with a validation error that explains exactly what's going on.

This makes debugging much much easier, but it also makes support much easier. Because it makes it very obvious when someone is accidentally passing the wrong IDs.

It's a wrap

I hope you enjoyed this post! For more content like this subscribe to the 5x9s newsletter.

It will be a source to help engineers overcome the challenges they face day to day to make their systems more reliable and scalable. Some articles will be technical guides while others may be interviews with industry experts (if you’ve built and scaled a software product we’d love to talk!).

If you’re an engineer looking to learn from others who’ve built and scaled products to millions of users, subscribe here and join our community on Slack. We look forward to seeing you in the Five 9’s community!

Five 9’s

Subscribe now