Top 5 Kubernetes production challenges – Try HyScale’s approach

Blog

Top 5 Kubernetes production challenges - Try HyScale’s approach

March 11, 2020 - by admin | appeared on Faun

Kubernetes in production

Kubernetes is the technology that ran production workloads for Google for over fifteen years. Since then, it has been open-sourced, and hundreds of members of the community have come together to make it better. There is no doubt that Kubernetes is a production-grade container orchestration system.

Yet, one of the most common misconceptions about Kubernetes adoption is that if it works on dev / QA / staging, it’ll work on prod. This is hardly ever true. Firing up a k8s environment and deploying your microservice to it for dev / QA / staging is simple and relatively developer-friendly. Migrating it to an enterprise-grade production environment brings with it several complexities of performance, security, interoperability, and governance.

In this blog post, we’ll discuss the differences in using Kubernetes across environments, the challenges you are likely to face.

#1 Making Kubernetes deployments work at scale on demand.


One of the first concerns for the IT/Ops teams while deploying to Kubernetes in production is setting up dynamic scalability. Don’t get me wrong, Kubernetes is built for scalability. There are several in-built tools that ensure infrastructure- and application-level scaling needs and load balancing. But enabling demand-based auto-scaling in production requires Ops teams to work harder on setting things up correctly.


  • You might have to configure a load balancer like HAProxy or NGINX, if you’re deploying k8s anywhere other than the Google Cloud Engine.
  • You can not afford to skip specifications like resource or request limits.
  • You must implement graceful pod termination to downscale safely.
  • You must design your autoscaling in a manner that Horizontal Pod Autoscaler (HPA) and Vertical Pod Autoscaler (VPA) don’t apply together.

#2 Ensuring reliability


When a developer fires up a development environment and begins to code, their primary concern is function and agility. As long as the code works, interacts with other services, and tests right, they’re happy. But in production, enterprise-grade applications — their k8s pods — are to work at significantly higher standards for performance, high-availability, disaster recovery etc.

This requires your IT Ops teams to plan the architecture and k8s deployment definitions accordingly.


  • You need multi-master setups for high-availability. And then build redundancies at application and infra-level for good measure.
  • You must plan zero-downtime environment upgrades. You must also be patching applications and upgrading Kubernetes to its latest version, while carefully maintaining compatibility between the components and k8s.
  • You must set up a CI/CD toolchain that not only expedites releases, but also ensures their quality, without additional efforts from your DevOps teams.

#3 Providing enterprise-grade security in production.


In development or staging environments, security is often not a primary concern. But security and resilience from external attacks is fundamental to the app in production. To ensure unbreakable security,IT/Ops teams must set up processes and failsafes across the board.


  • Infrastructure design, development, and deployment processes need to keep security in mind.
  • You must control kubesprawl, which can result in a larger attack surface for your application.
  • You must have clear visibility over access control, multi-factor authentication, anonymous authentication etc.
  • All unnecessary open network connections need to be closed.
  • You need to change custom images used in dev environments into trusted images for production.
  • You must run patches and upgrades on time.

#4 Enabling in-depth and end-to-end governance.


For any enterprise-grade application, governance is essential — at pod, cluster, application and infrastructure levels. Unlike in dev or test environments, containers in prod need to be monitored all the time. Enabling this requires IT/Ops teams to make focussed and persistent efforts.


  • You need to set up an automated audit trail for your production deployments.
  • You must monitor infra elements like CPU, RAM etc. as well as abstractions like pods, replica sets etc.
  • Version control for configurations, policies, containers and even infrastructure is crucial.
  • You must have systems to generate reports for resource usage, utilization and saturation metrics to ensure cost management.

#5 Bringing consistency and visibility across multi-cloud environments.


Even though Kubernetes is consistent in providing environments, there can be differences across cloud vendors. Depending on how some of your services like load balancers and firewalls — which are not a native capability in open source k8s — are structured, your containers might work differently in different cloud environments.

DevOps teams must take care that these application services run effectively across multi-cloud deployments, which involves:


  • Finding distribution that supports multi-cloud deployments. You must also make sure it accommodates the needs of each cloud platform.
  • Ensuring you configure your cluster topology not just for the needs of your application, but also for multi-cloud environments.
  • Minimizing inconsistencies among environments, often achieved through declarative approach, for smooth CI/CD.

What we've presented here is hardly an exhaustive checklist. It is just a few of the many things that you must keep in mind while using k8s in production. In our experience, to ensure smooth deployment of apps to k8s in prod environments, we've seen application teams need two things: Firstly, automation that makes DevOps simple, secure, reliable, manageable and consistent at scale. As your enterprise application grows in scale and complexity, you will need a platform that provides an app-centric experience, for specific tasks like CI/CD, tracking, monitoring, multi-cloud visibility etc.

Secondly, and perhaps just as important if you’re only now adopting Kubernetes for production, you also need a solutioning expert, who will eliminate the bumps in your path. They will assess your readiness, identify requirements, set up your K8s operations, build your application deployment strategy and work with you on migrating your applications.

With the right platform and a capable partner, there will be nothing stopping you from taking your #app2k8s.