4 June, 2019
It’s a Monday morning, and you need to set up a new project. You install all of the required npm packages, then start working on some features. Sooner or later you’ll find an issue requiring some exceptional coding, so you turn to Google and look for an answer – probably a package. You install it and create a production package image. Then you take a look at its size and… that’s how you get a heart attack. “How did I end up with 1GB node_modules?!” – the struggle is real, and believe me, at TSH we’ve been there a multiple times. But fear not, something can be done to reduce that size! We’ve created a few simple steps to reduce its size. Just follow me, please…
Our node_modules backstory
Not so long ago, we’ve been working on this British fintech project, where we had to migrate the system to a microservice architecture using Node.js. When we were introduced to the platform, we got two requirements from the client – it had to be efficient and light. Performance is not a problem for Node.js (unless you accidentally block the event loop), however, lightweight can be. The starting image was 32MB, which is a very nice result, but it quickly turned out that the real problem is node_modules.
Contrary to what you may think, the size of the application is important in a couple of areas:
- performance – an application that is light and runs faster, takes up fewer resources so it works more efficiently during development,
- costs – some light apps require fewer resources which means that the infrastructure costs drop and you save up a lot of money.
Developers like to simplify their work to do everything faster – that’s why they include additional libraries. Unfortunately, this results in an increased size of the entire application that can go out of control very quickly. This is how our adventure with “slimming down” the thing began. So what did we do to shrink them?
Interested in developing microservices? 🤔 Make sure to check out our State of Microservices 2020 report – based on opinions of 650+ microservice experts!
Reduce the number of dependencies
This one sounds obvious but before we get into more invasive solutions, maybe you should think about the packages you install. Do you really need all of them?
At TSH we are using sites like npm.broofa or npm.anvaka to actually see a whole graph of dependencies for a single package and then we discuss if this is something we’re looking for.
A lot of times I see people installing Jest just for simple unit tests (about 460+ dependencies, +/-60MB) when there are smaller libs that will do exactly the same (Mocha – 115 dependencies, 12MB).
I know what you’re thinking – it’s just a dev dependency. Yeah, it is, yet by reducing the number of modules you also speed up your development machine. Fewer things stored in memory, fewer libs to follow by IDE all of this makes us develop faster. It all comes to simple props and cons. In our case, we started with 700MB of dependencies and after a few exchanges and removals, we ended up with just 256MB.
Use production flag
Another obvious (but sometimes forgotten) method is to use `–production` flag on npm install. It will skip all of the devDependencies and use production ones only. Trust me it’s worth it. Most of our projects have seen +-33% reduction of node_modules size after we’ve used it (176MB).
Remove unnecessary files
Have you ever thought about stuff being installed when you type npm install? I’m not talking about the whole tree of dependencies, but a lot of trash that’s inside. Docs, tests, markdowns, images, sources, a lot of files that aren’t useful during development at all (tell me when was the last time, you dived into node_modules to read docs, huh?).
At TSH we found two cool libraries – node-prune and ModClean. Both of them have the same purpose to remove everything that is not necessary for a package. ModClean comes with three patterns – safe, caution and danger – and sometimes removes too many files.
That’s why most of the time we use node-prune because it’s not only safer but also it allows us to reduce node_modules by another 30% from the production version.
For better visualisation: we started with 256MB of those modules for dev version. After production we went down to about 176MB and then after node-prune, we are at about 126MB. It’s more than half of initial size!
See also: Dependency injection in Node.js
Seek and clean
Even after using node-prune, there’s still something that can be done. At some point, we started to check the size of each module, to see which ones are the largest. There is a very simple command in case of MacOS and Linux:
It’ll print every module that has a size of at least 1MB. By doing this you will know which modules take up the most space. It was quite helpful because until then we didn’t realize that RxJS / gRPC and AWS aren’t so slim. What’s more, you can use it on a specific module to find which directory is to blame. This allowed us to find that RxJS has a version for ts, es5, es2015 and UMD package, even though we’re using dist directly.
Those additional packages took 7MB from 11MB of the whole package. More than 50%!
The same story was with gRPC (there was a directory of multiple third parties that we didn’t use at all – about 12MB) and AWS (you get a version for web and native bundled – about 10MB). Because of that, we created a special shell script that did the cleaning.
By doing it we went down by another 17%, so from 126MB to about 104MB.
The final result
We started with 700MB of dependencies. Then we did some chopping and left only the modules we actually needed. The rest of them were exchanged for smaller modules or even replaced by our custom code. We ended up with 256MB. After that, we used the production flag and made it into 176MB. By using node-prune we could go down to 126MB and then cut it down to 104MB after some additional cleaning.
From 700MB to 104MB! It sounds great, doesn’t it?
Believe it or not, but all of those steps (except production flag) are usable on dev version too, so we tried that as well. We ended up with 161MB for dev version. It’s less than using production flag only.
And just like that, we lost 539 MB in total! Your orca-sized node_modules are not doing any favour to your application – they worsen the performance, slow down the processes, and devour infrastructure (which means you virtually lose a lot of money on additional resources). All in all, I highly recommend slimming down the modules in your project for the sake undeniable of business benefits. If you’d rather like a specialist to take care of it, you can give our Node.js team a shout. Good luck!