Designing/Building systems - Part 1

This is first article in series of articles about principles which can be used in Designing/Architecting systems

Some of below are from my experiences in building various systems

Some of it represents my mental model from book - Building Secure and Reliable Systems

Broadly I want to talk about below topics

• Reliability and Security
• Architecture
• Change
• Automated Testing/CI/CD

Design Principles

Security and Reliability

Generally when building systems , in addition to functional requirements ( which are defacto honoured) importance should be given to Security and Reliability. Although these can be retrofitted later, it becomes costly to include it at later stage. Below are some cost that could be incurred if these are not taken at start of product/project

• Increased Effort in development and testing
• Financial loss,
• Increased code complexity
• Incorrect implementation or
• Difficult to change existing project/product already Live

Good example is to look at early days of internet (Http /TCP/DNS) where emphasis was given on reliability and not in security. Subsequently those has been included in layers( Like SSL) but it has taken time and effort.

So its better to include these as requirements from start itself

Reliability

Reliability in general sense signifies anything which you will have to include in your product to ensure that it has uptime per expected SLA

Below are some of features (not limited) which can be considered as part of Reliability.

• Framework of choice for communication- http, REST, rpc, tcp , event buses
• Microservices- Granularity of services , How big or small are services defined
• Testing - Unit testing, Functional testing and Load testing
• Monitoring and Auditing
• Rate Limiting, Load Balancing, Retry Logic
• Transaction Journal
• Log Shipping
• TimeOuts, deadlines
• CI/CD

Security

Security represents service boundaries as to how services interact with each other

Below are some of features(not limited) which can be considered as part of Security features

• Trust relationship between services when they interact — Authentication/Authorization etc
• Choice of frameworks/programming languages etc
• Security testing, Penetration testing, Validation testing
• Security monitoring, audit logging and anomaly detection

Architecture

When Architecting systems below factors should be given considerations

• Centralised responsibility for Security and Reliability
• Frameworks
• Interfaces/Services
• Identities, Authentication and Access Control

Centralised responsibility for Security and Reliability

Many features pertaining to security and reliability are needed across the system and not limited to one subsystem. Below represents some of them.

• cascading failures under load, timeouts, deadlines
• authentication, authorisation and logging

So these should be included as part of framework which can be monitored, understood, changed, reasoned about easily.

Frameworks

Anything Global should be baked into framework so that one can reason/understand about it easily and it stays invariant with Life of System

Frameworks provide centralised place where features pertaining to Overall system should be included. Below are some of features which can be included in Frameworks

• Authentication & Authorization
• Logging and Error reporting
• Data Encryption
• Rate Limiting/ Retry Logic/Governor
• Deadline propagation
• Validations
• Health Management, Monitoring, Diagnostics
• Quota Enforcement
• Dashboards and alerts
• Transaction Management
• Abstractions pertaining to Queries
• Utilities
• Request/response transformations
• Caching
• Transaction journal

Interfaces/Services

Interfaces/Services should be small , easily understandable . Below are some guidelines

• Interfaces/Services should be narrow, clear and should leave no room for interpretation
• TradeOffs should be considered when selecting how interfaces communicate. eg- rpc vs REST
• Clear boundaries should be defined
• Access Policies should be defined

Identities , Authentication and Access Control

When services interact with each other, ensure identities are passed across and authorization policies are clearly defined. Below are some techniques for authentication and authorization

• OAuth Token
• JWT
• X.509
• SAML

Identify whats best in your situation and enforce/implement same.

Identities which are used should be non-spoofable (example certificates), non reusable and understandable

Services should communicate using TLS

Change

Any system should be easily adaptable for changes. Changes could be due to many things. Some factors are mentioned below

• Product or Feature Change
• Change response to security incident
• Change to improve perception
• Change in response to vulnerabilities
• Regulatory changes

Below are some considerations to provide flexibility

• Microservices based Architecture
• Using Containers

Microservices based Architecture

Below are some of advantages of using Microservices based architecture

• Smaller services which are easy to manage/maintain/discover
• Independent release/scalability needs
• Provides defence in depth, each layer /service provides its own defence mechanism
• Each service has its own access policies

Containers

Below are some benefits of using Containers at scale

• Binaries /Libraries are decoupled from host OS
• Containers being immutable, you cannot patch/update live server. Patch is applied in registry and then propagated
• Easy to manage/monitor patches
• Host OS is smaller
• Application can be more portable

Of course if you have 2–3 servers running, containers would be overkill. It only makes sense at huge scale

Automated Testing/CI/CD

Another item which is ignored at start is CI/CD or Automated testing. This could have impact on velocity when implemented from start. However as project grows , testing and deployment becomes frequent this can help enormously