Implementing microservices in production environments

Informatec Digital » Resources » Implementing microservices in production environments

Adopting a microservices architecture in a real-world environment isn't just about breaking down a monolith into smaller pieces; it involves rethinking infrastructure, teams, processes, data, security, and operations . When the system moves from theory to a production cluster, issues arise regarding service discovery, contracts between teams, CI/CD, observability, resilience, and scalability. If these issues aren't addressed properly, they can turn microservices into distributed chaos.

The good news is that today we have a wealth of accumulated experience from organizations like Netflix, Amazon, Google, and other large corporations that run hundreds of microservices in production . Building on these lessons, along with best practices in enterprise environments using Kubernetes and OpenShift, we can develop a very robust approach to designing, deploying, and operating microservices at scale without losing control.

Why deploy microservices to production (and when it's not worth it)

A well-designed microservices architecture allows you to work with small, autonomous, and cross-functional teams that take ownership of an end-to-end service. Each team operates within a well-defined context, can deploy frequently, and assume full responsibility for its service, reducing development cycle time and accelerating the delivery of new features.

Another key benefit is independent scaling per service . You don't have to oversize the entire application if only the catalog, checkout, or public API experiences traffic spikes. You can adjust each microservice horizontally or vertically according to its load pattern, accurately measure the cost of each feature, and maintain availability even if a specific area experiences a surge in consumption.

The way these services are packaged and deployed facilitates continuous, low-risk implementation . Releasing each microservice independently makes testing new ideas and reverting problematic versions much simpler: canary deployments, blue/green rollbacks, and automated rollbacks reduce the cost of failure and provide room for experimentation.

From a technological standpoint, microservices foster the freedom to choose languages, frameworks, and databases for each service. Not all needs fit into the same technology stack: you might have business services in .NET or Java, data processing in Scala/Spark, specialized services in Python or F#, or AI microservices in R. This controlled diversity allows you to use the right tool for each case, without forcing the entire application into a global technological shift.

Furthermore, breaking down the system into small, well-defined pieces facilitates the reuse of functionalities as building blocks . A microservice initially created as part of a larger functionality can later be reused as a dependency of other parts of the system without rewriting the logic. And because the services are isolated, a failure in one of them usually results in partial system degradation, not a complete system outage, provided that resilience was designed in from the outset.

Architectural and service design

For microservices to work well in production, it's essential to start with a careful design of service boundaries and responsibilities . In practical terms, this usually begins by identifying coarse-grained services within the existing monolith: large functional areas or business domains (e.g., orders, catalog, users, billing) that already have some logical separation.

Starting with these large building blocks, the process involves refining the design to obtain finely granular microservices that operate on a coherent dataset , own their own model, and know exactly what they need to read from or write to other services. This process typically relies on domain-driven design (DDD) concepts and bounded contexts, preventing a microservice from becoming a "mini monolith."

The APIs that expose these services must have well-defined and stable contracts . This implies rigorous documentation (REST with OpenAPI, gRPC with .proto files, etc.), explicit versioning, maintaining backward compatibility where possible, and automating contract validation to detect breaking changes before they reach production.

In environments with dozens or hundreds of services, it is crucial to incorporate resilience patterns from the design stage, so that the system is prepared for partial failures . Patterns such as circuit breakers, retries with backoff, well-defined timeouts, bulkheads, and backpressure help prevent the failure of one service from bringing down the rest. Chaos engineering tools like ChaosMonkey or Gremlin are useful for practically testing how the platform behaves under simulated outages.

Many complex systems combine relatively simple CRUD services with more sophisticated ones that handle evolving business rules. Not all microservices require a complex internal architecture : some can be simple HTTP controllers with basic data access, while others, such as order or billing services, can leverage more advanced patterns (DDD, CQRS, domain events, etc.).

Production infrastructure: cloud, containers and Kubernetes/OpenShift

Real-world experience shows that microservices perform much better when deployed on cloud infrastructure with containers and orchestration than on isolated virtual machines. Platforms like Kubernetes and OpenShift provide the necessary primitives to package services as containers, scale, update, load balance, and manage high availability.

Typically, each microservice is packaged in a container image based on a corporate base image (for example, OpenJDK 21 for Java services) managed by the infrastructure team. This base image is kept up-to-date with security patches, and when a new version is released, the development teams are responsible for rebuilding and redeploying their services in the corresponding environments.

In Kubernetes/OpenShift, the basic deployment unit is the pod, which encapsulates one or more containers . Typically, a microservice corresponds to a pod type and is deployed using resources such as Deployments (for stateless services) or StatefulSets (when there is associated state). From the outset, a minimum number of replicas per environment is defined so that test, pre-production, and production environments have availability levels appropriate to their criticality.

Automatic scaling is implemented using HorizontalPodAutoscaler (HPA) , which adjusts the number of replicas based on metrics such as CPU, memory, or other custom metrics. The platform must also configure pod anti-affinity rules to distribute replicas of the same service across different nodes, preventing a single node failure from taking down all instances.

Regarding vertical sizing, resources.requests and resources.limits are used to define the range of CPU and memory that a pod can consume. For example, reserving a minimum of 100MB of CPU and 256MB of memory, and allowing up to 500MB and 2GB respectively for a Java service, adjusting the JVM (Xms, Xmx, Xss) to make good use of the container's resources.

State management: stateless and stateful microservices

Most business microservices are designed as stateless services . This means that the pod does not store information that needs to survive reboots; state is persisted in external databases, message queues, or other storage. This approach facilitates dynamic horizontal scaling and frictionless deployments, as any replica can handle any request.

However, there are scenarios where there is no alternative but to have stateful microservices supported by persistent volumes . This is the case for some databases, distributed file systems, or components that require maintaining local data. These pods are typically deployed with StatefulSets, linked to PersistentVolumes using PersistentVolumeClaims, and scale vertically rather than horizontally.

When a microservice needs persistent storage, a PersistentVolumeClaim (PVC) is requested with its size, access mode, and intended use , and the operations team provisions it according to platform policies. This PVC is referenced in the deployment manifest and mounted on the pod so the service can read and write data persistently.

While stateful models may be necessary in specific cases, the general recommendation is to keep as many services as possible stateless . This simplifies deployment, scaling, resilience, and disaster recovery, and reduces operational complexity in environments with many microservices.

Data decentralization and service sovereignty

In traditional infrastructures, it's common to centralize databases and storage to maximize efficiency. With microservices, this approach conflicts with team autonomy and decoupling . If many services share the same relational schema, any structural change can block multiple teams and unintentionally break compatibility.

Therefore, the recommended practice is for each microservice to own its own data model and database , although in a development environment that database runs as a container within the cluster to simplify deployment. In production, cloud-managed instances or other high-availability database servers are typically used, always maintaining a clear ownership boundary.

This doesn't mean there's no data integration; it means that consistency between services is managed with events and asynchronous messaging , accepting eventual consistency when reasonable. It's common to use event buses (RabbitMQ, Azure Service Bus, Kafka, etc.) to propagate state changes between microservices, reducing strong dependencies on a single database.

The cloud platform makes it easy for teams to choose the optimal database type for each service (relational, document, key-value, time-series, etc.), without imposing a single technology. The key is that the design considers the possibility of migrating schemas and structures without breaking contracts with other services, and that data decisions are made in alignment with the domain boundaries of each microservice.

Distributed governance, teams and organization

Moving to microservices without changing the organization is asking for trouble. Instead of the classic functional silos of networks, systems, databases, development, and operations , a structure based on product teams is encouraged, bringing together profiles from development, QA, DevOps, and, where applicable, business or data analysts.

Each team is responsible for one or more microservices within the same functional domain, handling both development and operation (you build it, you run it) . This means the team manages its CI/CD pipelines, collaborates with infrastructure for specific needs, and participates in monitoring and incident response. The infrastructure and cloud platform focus on providing common and standardized services.

To prevent this distributed governance from descending into anarchy, it is crucial to define lightweight standards and shared catalogs : approved base images, deployment patterns, naming conventions for namespaces and services, API guidelines, Dockerfile and Kustomize templates, etc. These guidelines serve as "guardrails" that orient teams without blocking their ability to make decisions.

In many enterprise environments, separate namespaces are used for each project or domain , with at least one per environment (development, pre-production, production). A large project can distribute its microservices across several namespaces, provided that internal communications are properly configured and security rules are respected.

CI/CD, automation and the GitOps model

When an architecture comprises dozens or hundreds of microservices, the only way to keep them operational is to invest heavily in end-to-end automation . This includes consistent CI/CD pipelines, declarative deployment definitions, automated testing, and automatic rollback mechanisms.

A typical continuous integration and delivery pipeline handles compiling code, running tests, analyzing quality with tools like SonarQube , building the container image from the corporate Dockerfile, and updating deployment manifests. From there, a system like ArgoCD or similar applies the changes to the cluster using a GitOps approach.

Each microservice repository typically includes a standardized Dockerfile, a pipeline configuration file (e.g., ci.json) , properties for quality analysis, and a deployment directory with Kubernetes definitions (Kustomize or Helm) separated by environment. The repository's webhooks trigger the pipeline when events such as tag pushes or merge requests occur.

The GitOps pattern establishes the Git repository as the source of truth for infrastructure and deployment . Manifests for Deployments, Services, ConfigMaps, PVCs, SealedSecrets, and other resources are versioned there, and specific tools handle synchronizing the cluster state with what's defined in Git. This provides traceability, pull request reviews, and easy rollback capabilities.

Settings, secrets and security

In a mature microservices platform, configuration management relies on ConfigMaps for non-sensitive parameters and Secrets for confidential information . Each microservice typically has its own environment-specific ConfigMap, which stores properties such as URLs of dependent services, functionality flags, and tuning parameters.

Secrets (credentials, keys, tokens, certificates) are handled with strict security policies . In less critical environments, it may be acceptable to keep them in plain text managed by the development team, but in pre-production and production environments, it is recommended to encrypt them using tools such as Sealed Secrets or specific cloud-based external managers.

When a secret needs to be shared among multiple services (for example, OTEL Collector credentials or a common keystore ), it can be centralized in a configuration repository per namespace. Projects sharing that namespace coordinate to update it as needed, maintaining control over who can read or modify these resources.

In terms of communications security, the dominant pattern is Zero Trust : nothing is taken for granted simply because the traffic is "internal." All calls between services, both internal and external, must be authenticated and authorized, ideally with mTLS, JWT tokens, or other equivalent mechanisms. Microservices do not blindly delegate security to the API Manager or the network; they also perform their own checks.

Communication between microservices, APIs, and messaging

In a mature microservices architecture, the communication layer is divided into several cases. For traffic from clients (browsers, mobile apps, third parties) to the back end, published APIs governed by an API Manager are used . These APIs are typically RESTful (often using OpenAPI) or, in some cases, gRPC exposed through a gateway.

Calls between microservices residing in the same namespace, or even across multiple namespaces within the same project, are typically handled by internal Kubernetes services with internal DNS . These calls bypass the public API Manager but adhere to security, authentication, and authorization policies. For these scenarios, a service mesh or internal gateways enforcing common policies can be used.

When microservices belong to different functional domains or projects , communication is considered "public" at the organizational level. In these cases, it's common practice to use an API Manager or an interoperability bus, where contracts, quotas, security, versioning, and auditing are managed, preventing direct coupling between independent clusters or namespaces.

Regarding integration with legacy or external systems, which may not always expose modern APIs, it's common to rely on specific connectors over an interoperability bus . This way, microservices speak a common language (for example, events or internal REST APIs), and the connector handles the translation to and from the legacy system, always with enhanced security.

In addition to synchronous communication, asynchronous messaging plays a key role . It is used to decouple processes, absorb spikes, propagate business events between services, and improve resilience. Each event typically has a well-defined and versioned scheme, with tracking mechanisms to prevent breakdowns between producers and consumers as they evolve.

Observability, OTEL Collector and operation

In a system composed of many microservices, diagnosing a problem without good observability is nearly impossible. That's why metrics, centralized logging, and distributed traces are integrated from the design stage , allowing for an understanding of what's happening at both the service and platform levels.

A central component of this scheme is the OpenTelemetry Collector (OTEL Collector) , which is deployed in the namespace or centrally to collect metrics, logs, and traces from all components. Microservices only need to know that they should send their telemetry to the Collector; the Collector then forwards it to the observability systems (Prometheus, Grafana, Jaeger, Elastic, etc.) without the service needing to know the details.

For the infrastructure layer, node-level collectors and exporters are used to gather CPU, memory, disk, network, and log metrics from the pods, sending them to Prometheus and Elasticsearch, respectively. Tools like Grafana and Kibana are used to visualize this information, build dashboards, and define alerts with smart thresholds and associated runbooks.

When a project needs very specific processing of its metrics or traces, it can deploy its own instance of OTEL Collector in its namespace, provided it has operational approval and the production maintenance model is clear.

Testing strategy, contracts, and local development experience

Testing a distributed microservices architecture requires a more sophisticated testing strategy than testing a monolith. Unit tests remain essential, but contract tests (for APIs and events), integration tests between services, and end-to-end tests that traverse complete flows are becoming increasingly important.

To prevent compatibility issues, techniques such as consumer-oriented contract testing are used , where clients define API expectations and service providers fulfill them. Every contract change undergoes automated testing within the CI pipelines, preventing deployments that break any known consumers.

When the number of services grows beyond one hundred, replicating the entire system locally becomes impractical. Therefore, development relies on simulations of dependent services or tunneling to remote environments . Developers typically launch only a subset of microservices and stub the rest with mocks, fakes, or simulators, or redirect certain calls to a shared integration environment.

End-to-end testing increasingly relies on ephemeral environments or "previews" created from feature branches , which establish an isolated environment with the services relevant to that functionality. This minimizes friction between teams, reduces the "it works on my machine" effect, and detects integration problems before reaching more expensive environments like pre-production.

Microservices deployment patterns in production

Beyond Kubernetes, there are several microservices deployment patterns in production that are worth knowing because they address different scenarios of isolation, cost, and maturity . One of the oldest patterns is multiple service instances per host, where a single physical or virtual host runs several instances of different services, typically on a shared application server.

In the per-VM service instance pattern , each service is packaged as a VM image (for example, an EC2 AMI) and runs on its own instance. This offers strong isolation at the cost of higher resource consumption and slower startup times. Tools like Packer or cloud provider-specific solutions make it easy to generate production-ready VM images.

The most widespread pattern today is the service instance per container , where each microservice is built as a container image and deployed on an orchestrator (Kubernetes, OpenShift, etc.). Containers are lighter than VMs, start up very quickly, and allow you to package everything needed for the service, simplifying deployments and enabling automatic scaling.

Finally, serverless approaches, such as AWS Lambda , have gained popularity. These packages functions that respond to HTTP requests or events from other services (S3, DynamoDB, queues, etc.), with users paying only for what they use. This pattern is particularly well-suited to very small microservices or short-lived event-driven tasks, although it introduces additional considerations regarding observability, cold starts, and execution limits.

In practice, many organizations end up with a hybrid ecosystem: the core part of the system runs on containers and orchestrators, while certain auxiliary components are implemented as serverless functions or as specialized VMs, always with clear interfaces and well-defined protocols to integrate them into the whole.

When it comes to bringing all of this to production, what makes the difference isn't just the chosen technology, but having built an architecture that tolerates faults, scales where needed, deploys automatically, and is observable . With product-aligned teams, well-managed contracts, decentralized data, and a robust cloud platform, microservices go from being a promise to becoming an effective and sustainable way to evolve complex applications for years.