Implementing microservices in production environments

Informatec Digital » Resources » Implementing microservices in production environments

Adopting a microservices architecture in a real-world environment is not just about dividing a monolith into smaller pieces: it involves rethink infrastructure, equipment, processes, data, security and operationsWhen the system moves from theory to the production cluster, problems arise regarding service discovery, contracts between teams, CI/CD, observability, resilience, and scalability, which, if not addressed properly, turn microservices into distributed chaos.

The good news is that today we have a lot of accumulated experience from organizations like Netflix, Amazon, Google, or large corporations that run Hundreds of microservices in productionBased on these lessons, plus the best practices in enterprise environments on Kubernetes and OpenShift, a very solid approach can be drawn up to design, deploy and operate microservices at scale without losing control.

Why deploy microservices to production (and when it's not worth it)

A well-designed microservices architecture allows you to work with small, autonomous, and multifunctional teams that take ownership of an end-to-end service. Each team operates in a well-defined context, can deploy frequently, and assume full responsibility for its service, reducing development cycle time and accelerating the delivery of new features.

Another key benefit is the independent scaling per serviceYou don't need to oversize the entire application if only the catalog, checkout, or public API experiences traffic spikes. You can scale each microservice horizontally or vertically based on its load pattern, accurately measure the cost of each feature, and maintain availability even if a specific area experiences a surge in consumption.

The way these services are packaged and deployed facilitates a continuous implementation with low riskIf each microservice is released independently, testing new ideas and reverting problematic versions is much simpler: canary deployments, blue/green deployments, and automatic rollbacks reduce the cost of error and allow room for experimentation.

From a technological standpoint, microservices promote the freedom to choose languages, frameworks, and databases per service. Not all needs fit into the same technology stack: you might have business services in .NET or Java, data processing in Scala/Spark, specialized services in Python or F#, or AI microservices in R. This controlled diversity allows you to use the right tool for each case, without dragging the entire application into a global technology change.

Furthermore, breaking it down into small, well-defined pieces facilitates reuse of functionalities as building blocksA microservice initially created as part of a feature can be reused later as a dependency of other parts of the system without rewriting the logic. And because the services are isolated, a failure in one of them usually results in partial system degradation, not a total system outage, provided resilience was designed in from the start.

Architectural and service design

For microservices to work well in production, you have to start with a careful design of service boundaries and responsibilitiesIn practical terms, it usually begins by identifying coarse-grained services within the existing monolith: large functional areas or business domains (e.g., orders, catalog, users, billing) that already have some logical separation.

Starting from those large blocks, you have to refine until you obtain fine-granular microservices that work on a coherent datasetThey should own their own model and know exactly what they need to read or write to other services. This process is usually supported by domain-driven design (DDD) concepts and bounded contexts, preventing a microservice from becoming a "mini monolith".

The APIs that expose these services must have well-defined and stable contractsThis involves rigorous documentation (REST with OpenAPI, gRPC with .proto files, etc.), explicit versioning, maintaining backward compatibility where possible, and automating contract validation to detect breaking changes before they reach production.

In environments with dozens or hundreds of services, it is crucial to incorporate resilience patterns from the design stage, so that the system be prepared for partial failuresPatterns such as circuit breakers, retries with backoff, well-defined timeouts, bulkheads, and backpressure help prevent the failure of one service from affecting the rest. Chaos engineering tools like Chaos Monkey or Gremlin are used to practically test how the platform behaves under simulated outages.

In many complex systems, relatively simple CRUD services coexist with more sophisticated ones that concentrate changing business rules. Not all microservices need a complex internal architectureSome may be simple HTTP controllers with basic data access, while others, such as the order or billing service, may take advantage of more advanced patterns (DDD, CQRS, domain events, etc.).

Production infrastructure: cloud, containers and Kubernetes/OpenShift

Real-world experience shows that microservices fit much better when deployed on cloud infrastructure with containers and orchestration than on isolated virtual machines. Platforms like Kubernetes and OpenShift provide the necessary primitives to package services as containers, scale, upgrade, load balance, and manage high availability.

Typically, each microservice is packaging in a container image based on a corporate base image (for example, OpenJDK 21 for Java services) governed by the infrastructure team. This base image is kept up-to-date with security patches, and when a new version is released, the development teams are responsible for rebuilding and redeploying their services in the corresponding environments.

In Kubernetes/OpenShift, the basic deployment unit is the pod, which encapsulates one or more containersTypically, a microservice corresponds to a type of pod and is deployed using resources such as Deployments (for stateless services) or StatefulSets (when there is associated state). From the outset, a minimum number of replicas per environment is defined so that test, pre-production, and production environments have availability levels appropriate to their criticality.

Automatic scaling is implemented with HorizontalPod Autoscaler (HPA)This adjusts the number of replicas based on metrics such as CPU, memory, or other custom metrics. The platform must also configure pod anti-affinity rules so that replicas of the same service are distributed across different nodes, preventing a node failure from taking down all instances.

Regarding vertical dimensioning, it plays with resources.requests and resources.limits To define the range of CPU and memory that a pod can consume. For example, reserving a minimum of 100m of CPU and 256Mi of memory, and allowing up to 500m and 2Gi respectively in a Java service, adjusting the JVM (Xms, Xmx, Xss) to make good use of the container's resources.

State management: stateless and stateful microservices

Most business microservices are designed as stateless servicesThis means the pod does not store information that needs to survive reboots; state is persisted in external databases, message queues, or other storage. This approach facilitates dynamic horizontal scaling and frictionless deployments, as any replica can handle any request.

However, there are scenarios where there is no other option but to have stateful microservices supported by persistent volumesThis is the case for some databases, distributed file systems, or components that require maintaining local data. These pods are typically deployed with StatefulSets, linked to PersistentVolumes using PersistentVolumeClaims, and scale vertically rather than horizontally.

When a microservice needs persistent storage, a PersistentVolumeClaim (PVC) with size, access mode, and intended useThe operations team is responsible for provisioning it according to platform policies. This PVC is referenced in the deployment manifest and mounted on the pod so that the service can persistently read and write data.

Although a stateful model may be necessary in specific cases, the general recommendation is to try to make the as many services as possible remain statelessThis simplifies deployment, scaling, resilience, and disaster recovery, and reduces operational complexity in environments with many microservices.

Data decentralization and service sovereignty

In traditional infrastructures, it's common to centralize databases and storage to maximize efficiency. With microservices, this approach is different. This conflicts with team autonomy and decoupling.If many services share the same relational schema, any structural change can block multiple teams and unintentionally break compatibility.

Therefore, the recommended practice is that each microservice be owner of their own data model and their own databaseAlthough in a development environment this database might run as a container within the cluster to simplify deployment, in production, cloud-managed instances or other high-availability database servers are typically used, always maintaining a clear ownership boundary.

This does not mean there is no integration between data: it means that the Consistency between services is managed with events and asynchronous messagingAccepting eventual consistency when reasonable. It is common to use event buses (RabbitMQ, Azure Service Bus, Kafka, etc.) to propagate state changes between microservices, reducing strong dependencies on a single database.

The cloud platform makes it easier for teams to choose the optimal database type for each service (relational, document-based, key-value, time-series, etc.), without imposing a single technology. The important thing is that the design takes into account the possibility of migrating schemas and structures without breaking contracts with other services, and that data decisions are made in alignment with the domain boundaries of each microservice.

Distributed governance, teams and organization

Moving to microservices without changing the organization is asking for trouble. Instead of the classic Functional silos for networks, systems, databases, development, and operationsA structure based on product teams is encouraged, bringing together development, QA, DevOps profiles and, where applicable, business or data analysts.

Each team is responsible for one or more microservices within the same functional domain, and assumes both the development as an operation (you build it, you run it)This means the team manages its CI/CD pipelines, collaborates with infrastructure for specific needs, and participates in incident monitoring and response. The infrastructure and cloud platform areas focus on providing common and standardized services.

To prevent this distributed governance from leading to anarchy, it is key to define lightweight standards and shared catalogsApproved base images, deployment patterns, naming conventions for namespaces and services, API guidelines, Dockerfile and Kustomize templates, etc. These guidelines serve as "guardrails" that guide teams without blocking their ability to decide.

Many business environments use namespaces separated by project or domainwith at least one per environment (development, pre-production, production). A large project can distribute its microservices across several namespaces, provided that internal communications are properly configured and security rules are respected.

CI/CD, automation and the GitOps model

When an architecture has dozens or hundreds of microservices, the only way to keep them operational is to invest heavily in the end to end automationThis includes consistent CI/CD pipelines, declarative deployment definition, automated testing, and automatic rollback mechanisms.

A typical continuous integration and continuous delivery pipeline handles compile the code, run tests, analyze quality with tools like SonarQubeBuild the container image from the corporate Dockerfile and update the deployment manifests. From there, a system like ArgoCD or similar applies the changes to the cluster following a GitOps approach.

Each microservice repository typically includes a standardized Dockerfile, a configuration file for the pipeline (e.g., ci.json)Properties for quality analysis and a deployment directory with Kubernetes definitions (Kustomize or Helm) separated by environment. Repository webhooks trigger the pipeline when events such as tag pushes or merge requests occur.

The GitOps pattern states that the The source of truth for infrastructure and deployment is the Git repositoryThe manifests for Deployments, Services, ConfigMaps, PVCs, SealedSecrets, and other resources are versioned there, and specific tools handle synchronizing the cluster state with what's defined in Git. This provides traceability, pull request reviews, and easy rollback capabilities.

Settings, secrets and security

In a mature microservices platform, configuration management relies on ConfigMaps for non-sensitive parameters and Secrets for confidential informationEach microservice usually has its own ConfigMap per environment, which stores properties such as URLs of dependent services, functionality flags, or tuning parameters.

Secrets (credentials, keys, tokens, certificates) are handled with strict security policiesIn less critical environments, it may be acceptable to keep them in plain text managed by the development team, but in pre-production and production, it is recommended to encrypt them using tools such as Sealed Secrets or specific external cloud managers.

When a secret needs to be shared between several services (for example, the OTEL Collector credentials or a common keystoreThis can be centralized in a configuration repository per namespace. Projects that share this namespace coordinate to update it when necessary, maintaining control over who can read or modify these resources.

In the area of ​​communications security, the dominant pattern is the Zero TrustNothing is taken for granted just because the traffic is "internal." All calls between services, both internal and external, must be authenticated and authorized, ideally with mTLS, JWT tokens, or other equivalent mechanisms. Microservices do not blindly delegate security to the API Manager or the network; they also perform their own checks.

Communication between microservices, APIs, and messaging

In a mature microservices architecture, the communication layer is divided into several cases. For traffic from clients (browsers, mobile apps, third parties) to the back-end, the following are used: APIs published and governed through an API ManagerThese APIs are usually REST (often using OpenAPI) or, in some cases, gRPC exposed through a gateway.

Calls between microservices residing in the same namespace, or even in multiple namespaces of the same project, are usually handled by Internal Kubernetes services with internal DNSThey don't go through the public API Manager, but they still follow security, authentication, and authorization policies. For these cases, a service mesh or internal gateways that enforce common policies can be used.

When microservices belong to different functional domains or different projectsCommunication is considered "public" at the organizational level. In these cases, it is normal to go through an API Manager or an interoperability bus, where contracts, quotas, security, versioning, and auditing are controlled, avoiding direct coupling between independent clusters or namespaces.

Regarding integration with legacy or external systems, which cannot always expose modern APIs, it is common to rely on specific connectors on an interoperability busIn this way, microservices speak a common language (for example, events or internal REST APIs) and the connector is responsible for translating to and from the legacy system, always with reinforced security.

In addition to synchronous communication, the Asynchronous messaging plays a key roleIt is used to decouple processes, absorb spikes, propagate business events between services, and improve resilience. Each event typically has a well-defined and versioned scheme, with tracking mechanisms to prevent breakdowns between producers and consumers as they evolve.

Observability, OTEL Collector and operation

In a system composed of many microservices, diagnosing a problem without good observability is nearly impossible. That's why they are integrated from the design stage. metrics, centralized logging, and distributed traces, that allow us to understand what is happening at the service and platform level.

A central piece in this scheme is the OpenTelemetry Collector (OTEL Collector)This is deployed in the namespace or centrally to collect metrics, logs, and traces from all components. Microservices only need to know that they must send their telemetry to the Collector; the Collector then forwards it to observability systems (Prometheus, Grafana, Jaeger, Elastic, etc.) without the service needing to know the details.

For the infrastructure plan, the following are used node-level collectors and exporters These systems collect CPU, memory, disk, network, and log metrics from the pods, sending them to Prometheus and Elasticsearch respectively. Tools like Grafana and Kibana are used to visualize this information, build dashboards, and define alerts with smart thresholds and associated runbooks.

When a project needs very specific processing of its metrics or traces, it can deploy its own instance of OTEL Collector in its namespace, provided it has operational approval and the production maintenance model is clear.

Testing strategy, contracts, and local development experience

Testing a distributed microservices architecture requires a more elaborate testing strategy than in a monolith. Unit tests remain fundamental, but contract tests (for APIs and events), integration tests between services, and end-to-end tests that traverse complete flows are gaining importance.

To avoid changes that break compatibility, techniques such as consumer-oriented contract testingwhere customers define API expectations and service providers fulfill them. Every contract change goes through automated tests that run in CI pipelines, preventing deployments that break any known consumers.

When the number of services grows above one hundred, replicating the entire system locally becomes impractical. Therefore, the development experience relies on simulations of dependent services or tunneling to remote environmentsDevelopers typically deploy only a subset of microservices and stubby the rest with mocks, fakes, or simulators, or redirect certain calls to a shared integration environment.

End-to-end testing increasingly relies on ephemeral environments or “previews” created from feature branchesThese tools create an isolated environment with the services relevant to that functionality. This minimizes friction between teams, reduces the "it works on my machine" effect, and detects integration problems before they reach more expensive environments like pre-production.

Microservices deployment patterns in production

Beyond Kubernetes, there are several microservices deployment patterns in production that are worth knowing, because they address different scenarios of isolation, cost, and maturityOne of the oldest patterns is that of multiple service instances per host, where the same physical or virtual host runs several instances of different services, usually on a shared application server.

In the pattern of service instance per virtual machineEach service is packaged as a VM image (e.g., an EC2 AMI) and runs in its own instance. This provides strong isolation at the cost of higher resource consumption and slower startup times. Tools like Packer or cloud provider-specific solutions make it easy to generate production-ready VM images.

The most widespread pattern today is that of service instance per containerwhere each microservice is built as a container image and deployed on an orchestrator (Kubernetes, OpenShift, etc.). Containers are lighter than VMs, start up very quickly, and allow you to package everything needed for the service, simplifying deployments and automatic scaling.

Finally, approaches have gained popularity serverless, like AWS LambdaThis pattern involves packaging functions that respond to HTTP requests or events from other services (S3, DynamoDB, queues, etc.), with users paying only for what they use. It's particularly well-suited for very small microservices or short-lived event-driven tasks, although it introduces additional considerations regarding observability, cold starts, and execution limits.

In practice, many organizations end up with a hybrid ecosystem: the core part of the system runs on containers and orchestrators, while certain auxiliary components are implemented as serverless functions or as specialized VMs, always with clear interfaces and well-defined protocols to integrate them into the whole.

When it comes to bringing all of this into production, what makes the difference is not just the technology chosen, but having built an architecture that It is fault-tolerant, scales where needed, deploys automatically, and is observable.With product-aligned teams, well-managed contracts, decentralized data, and a robust cloud platform, microservices are moving from a promise to becoming an effective and sustainable way to evolve complex applications for years.