26 September, 2019
Recently, we’ve organised a second online webinar on microservices – it focused on the most popular design patterns for microservice architecture. While Adrian Zmenda (our Lead Developer) was conducting the lecture, I tried to quickly answer all your questions in the chat window. Since the participants’ questions were so on point, we decided to publish the entire Q&A here – I’m sure that lots of developers and CTOs will find all this microservices-related knowledge useful too. Hey ho, let’s go!
1. Will you share the webinar presentation?
If you’d rather just read the presentation itself, it’s available below:
2. How to handle a situation when you have REST API Gateway or BFF and you want to do something async? I want to send 10 emails and return success or error information to the user when the process is completed. Maybe I should use WebSocket/messaging communication instead of rest Gateway/BFF?
Backend for Frontend/Gateway doesn’t need to be a REST API. We have a project where outside (SPA) communication is done by WebSockets and inner one using REST and GRPC. So yeah, that’s definitely an option here.
3. If the API Gateway does the authentication then should individual microservice still need to do authentication?
You don’t need to authenticate inner services, only those that can be called directly without a gateway. Of course, you still need a token because it might carry some useful information, like user ID, user roles etc.
4. In order to talk to services, we have our classes. Is it a good idea to have a shared library of these classes between different microservices, or rather keep that separate for each microservice?
First of all, duplication is not a bad thing. For example, I wouldn’t create shared lib for “clients” to each service, but rather build them on runtime using some contract – per service. On the other hand, things like loggers, tracers and other standardized things are good to be put into a shared lib.
5. How do you manage to log for the full end-to-end request?
It’s much better to use tracing instead, and pass some REQID field for easier tracing.
6. If configs are external, what happens if there is a network error when fetching the config? How often do you update configs and how?
If this is at the start of service, you can try to retry a few times and then restart the service. However, if the service is already fully working, then I would assume you have some “working” config copy (cache). Honestly, we don’t use “config service” most of the time. We’d rather pass config into the service and restart/update it if necessary, rather than have another possible point of failure.
7. For the container orchestration, should I go with basic orchestration tool like Docker Swarm first, rather than using complex orchestration tool like Kubernetes?
If you have previous experience with Kubernetes (and micro/SOA infrastructure) then I would stick to it. Otherwise, I would use a less complex solution – at least during the first phase.
8. I’ve noticed that you prefer monorepo. How do you guys resolve the case when A and B features merged/deployed to dev but then you need only feature A go to master/prod?
It all depends, but one of the answers is – feature flags. Everything can be merged go to prod, but by default, it’s switched off. Our approach is that everything on dev goes to prod. We may keep things on feature branches, but we’d rather keep it in code and use feature flags.
9. Should I use separate DB per service?
It all depends. We worked with both approaches and they both have their pros and cons. You can have a single DB and separate tables – it makes it a little bit easier to handle (since you can access data from other services directly) or single DB per service (a different DB type per service is much more flexible).
10. Is it a bad practice to have HTTP(s) protocol between services that are closed from the client and used only BE other services? It’s kinda slow in comparison to other protocols…
It’s not a bad practice, but personally I would rather use some messaging/binary protocols.
11. Is Domain-Driven Design compatible with microservices?
It’s quite common to convert bounded context into separate microservices. However, it’s not like you have to use DDD with microservices. Some of them might be a simple CRUD, and DDD wouldn’t be useful there.
12. How do you know that request is valid without the authentication? You will add a token to the request but how microservice decide that it is a valid token.
Most of the time, we have a separate security service that has information about the private key for tokens. Every time we send a request from the outside (SPA) we pass a token and in API Gateway we make a single call to security service to check if the token is valid. That way we only need to check it once, and then you’re allowed to go inside.
13. Which tool would you use for centralized logging, provided you can have service written in PHP, NodeJS or Python?
We require logs to be passed to standard output so all of them can be passed to a single source of logging. Then we can use some visualization tools like Kibana/Grafana/Greylog to check them. The only important thing is a common format for logs.
14. How do you handle a service that needs to be used in several places? For example, a service that has users’ information and you need that data in several other microservices.
One approach is to use messages as a way to pass that information and keep a local copy. Every time it changes, you just push an event and subscribe for that event in specific services. Other option might be to use a single DB and access to tables from other services. It’s risky, but we’ve seen such communication working pretty well too.
15. How to avoid the risk of the message broker becoming the single point of failure?
Scale it and replicate it. Most of the message brokers support this.
16. How to minimize operation latency?
There are a couple of options:
- reduce the number of hops by adding a local cache for common requests. Instead of asking for “user details” you can have a local “slim” version of user details, using Kafka as a data source or some kind of a stream.
- use a faster protocol – binary instead of HTTP.
17. Should API gateway aggregate data from all services without having services to communicate with each other and without services knowing about each other?
We need to separate outside and inside communication. Outside is when gateways are useful – so from SPA to our system. As far as inside is concerned, we may use events (Kafka, Rabbit etc.) or direct communication (REST, RPC).
18. Can you mention any recommended packages/approaches to code sharing across NodeJS microservices? Would having a monorepo and/or submodules be the way to go or separate packages referenced as dependencies?
In TSH, we are fans of monorepo, so I obviously recommend it. But it’s not the only option.
19. Let’s say we have a message broker that should notify three services. If the message comes in, how to keep it available exactly for three services (so it won’t be picked by only one service ) and then get rid of it.
We would introduce a special piece of code called saga, responsible for making sure that all of those services got their message.
20. Serverless (e.g. AWS lambda) vs. Microservices?
We’re fans of both. It all depends on what problem you’re facing – you need some “state” or is it just a fully stateless API.
21. If service is set up in a way that it is going to be restarted, in case of fail should I still need a health-check?
Yes, because you want to know when it happens, how often and then find what was the cause. You can think of it as a kind of “alert”.
22. Do you have any strategies for when service should be separated from your mono app? How to migrate data from databases? Fix routing?
This is why having a gateway in the first place is important. If you provide a layer before the actual app, you’re free to proxy requests for specific routes to a specific app (some goes to the monolith and the rest to your new microservice). In the case of data, it all comes down to a simple question – do need to separate databases? Splitting tables is always a complex task, and unfortunately, there is no ready-to-go method for it.
23. Do you use some tool in a monorepo for speeding up a build? I use Lerna and if we modularize the code we can run parallel. Could there be a way to have multiple services and builds that call separate builds?
We use Lerna or tools like Plop to build our custom CLI for building services. If you use Jenkins for the build, it’s absolutely possible. You might have several jobs that work only on specific services.
24. Why versioning is better in monorepo? Since services are independently deployable we should have separate version numbers increased in a different time. Could you explain?
I wouldn’t say it’s better, but it is easier to keep all services in sync because you can add new functionality and update other services at once in a single PR. In the case of separate repositories, you would need to create multiple PRs, wait for specific services to be merged, deployed etc.
25. AWS Lambda: When to use a separate function and when to have a routing inside a function to serve several endpoints?
There is no simple answer. Separate functions are easier to debug but make the system more complex. On the other hand, having the routing inside function requires more code, makes debugging a little bit easier but your architecture will be simpler. Another downside is a fact that you’re in fact building the monolithic app and may hit deployment limit.
26. Let’s say your communication is based on message broker and one of the services is down while there are plenty of messages awaiting for this service availability. What should happen to those messages in case we timeout the request for the user? Other services may not even know what channel should be purged.
A solution that pops into my mind is to have a separate channel for specific types of messages. It will allow you to specify TTL/retention time for that information, so when it hits timeout it will be removed automatically.
27. If the container orchestration tool provides the service discovery feature then do you still need to use third party service discovery tool like Consul?
Probably not. You should be fine, as long as you don’t hardcode addresses to services.
28. How can a centralized ACL service know about the specific permissions that need to be enforced in another service that is dictated by that service’s specific business model? Isn’t that mixing the responsibilities of the services?
The business model has access to security service, so you still are able to check if you can perform a specific business operation in a given service. Security service knows how to validate token and has access to ACL table, while other services might have a local copy of it (synchronised every time it changes in a security service) or call security method to check access for a specific resource directly. It removes a need for duplication of ACL logic in many places.
29. So the API Gateway is responsible for checking whether a token is valid or not. How other microservices knows that request comes from API Gateway and is not a direct request to the REST API of the microservice?
They don’t need to know. All of our services are private and the only way to communicate with them is the gateway. So it doesn’t matter if it’s a call from a gateway or other service.
30. What do you think about the auth mechanism?
We are big fans of token-based authentication. On the other hand, when it comes to authorisation, we like to use attribute-based access control or role-based access control.
31. Should logging (events, debug data) be a separate service? What tools to use gather analyse log files?
Mostly, we take everything from standard output, then it’s taken from container and passed to elasticsearch DB. From the developer’s point of view, the only thing we need to take care of is the same logs format between different languages. Most of the time we use Grafana to visualize logs, but before that, we used to work with Kibana/Graylog.
32. When it is okay to have inter-service communications and when all services should talk to each other through an entry point?
As long as your services are in the same private network, there is no need to communicate through the entry point. If you have multiple apps (set of microservices) hosted in separate networks, then the communication should go through gateways.
Phew, that’s it! Thanks to you all for constructive questions. 🙌 I hope that my advice was able to help you a little. If you missed our previous webinars but you love what we’re doing here, follow us on social media to make sure you’ll attend the next one.