Everything You Wanted To Know About package-lock.json But Were Too Afraid To Ask

Introduction

So you’ve updated Node Package Manager (npm) to v5.x.x, and everything seems to be going fine. But wait, what’s this? A new file was created automatically. package-lock.json. If you open it, it looks sort of like the dependencies in package.json, but more verbose. You decide to ignore it and go along your way developing your project. Eventually, you run into problems with a dependency. It can’t be found or the wrong version seems to be installed. Most people just end up deleting the package-lock.json and running npm install. So why even have it? What is it supposed to do? What does it actually do?

Summary

• If you’re using npm ^5.x.x, by default a package-lock.json will be generated for you
• You should use package-lock to ensure a consistent install and compatible dependencies
• You SHOULD commit your package-lock to source control
• As of npm ^5.1.x, package.json is now able to trump package-lock.json, so you should experience much less of a headache
• No more deleting that package-lock just to run npm install and regenerate it;
• Use semver if your app offers an API, and adhere to the rules of semver.

Background

Semantic Versioning

Before you can understand the package-lock and even package.json, you have to understand semantic versioning (semver). It’s the genius behind npm, and what has made it even more successful. You can read more about how npm uses it here. In a nutshell, if you are building an application with which other applications interface, you should communicate how the changes that you make will affect the third party’s ability to interact with your application. This is done via semantic versioning. A version is made up of three parts: X,Y,Z where those are major, minor and patch versions respectively. An example would be 1.2.3, or major version 1, minor version 2, patch 3. A change in patch represents a bugfix that doesn’t break anything. A change in minor version represents a new functionality that doesn’t break anything. A change in major version represents a large change that breaks compatibility. If users don’t adapt to a major version change, stuff won’t work.

Managing Packages

npm exists to make managing packages easy. Your projects might have hundreds of dependencies, each of those with a hundred others. To keep your mind away from dependency hell, npm was created so that with some simple commands, you could install and manage these dependencies and hardly ever have to think about them.

When you install a package with npm (and save it), an entry is added to your package.json containing the package name, and the semver that should be used. npm supports some wildcards in this semver definition however. By default, npm installs the latest version, and prepends a caret e.g. “^1.2.12”. This signifies that at a minimum, version 1.2.12 should be used, but any version higher than that is OK, as long as it has the same major version, 1. Since minor and patch numbers only represent bugfixes and non-breaking additions, you should be safe to use any higher same-major version. Read more about semver wildcards and play with npm’s cool semver calculator here.

Shared Projects

The real benefit to having dependencies defined like this in package.json, is that anybody who has access to the package.json could create a dependency folder that contains the modules needed to run your application. But let’s take a look at a specific way where things might go wrong.

Let’s say we create a new project that is going to use express. After running npm init, we install express: npm install express — save. At the time of writing, the latest express version is 4.15.4. So “express”: “^4.15.4” is added as a dependency within my package.json and that exact version is installed on my machine. Now maybe tomorrow, the maintainers of express release a bug fix, and so the latest version becomes 4.15.5. Then if someone were to want to contribute to my project, they would clone it, and run `npm install.’ Since 4.15.5 is a higher version with the same major version, that is installed for them. We both have express, but we have two different versions. Theoretically, they should still be compatible, but maybe that bugfix affected functionality that we are using, and our application would produce different results when run with express version 4.15.4 compared to 4.15.5.

Package-lock

The Goal

The purpose of the package-lock is to avoid the situation described above, where installing modules from the same package.json results in two different installs. Package-lock.json was added in npm version 5.x.x, so if you are using major version 5 or higher, you will see it generated unless you disabled it.

The Format

Package-lock is a large list of each dependency listed in your package.json, the specific version that should be installed, the location of the module (URI), a hash that verifies the integrity of the module, the list of packages it requires, and a list of dependencies. Let’s take a look at what the entry for express is:

"express": {
      "version": "4.15.4",
      "resolved": "https://registry.npmjs.org/express/-/express-4.15.4.tgz",
      "integrity": "sha1-Ay4iU0ic+PzgJma+yj0R7XotrtE=",
      "requires": {
        "accepts": "1.3.3",
        "array-flatten": "1.1.1",
        "content-disposition": "0.5.2",
        "content-type": "1.0.2",
        "cookie": "0.3.1",
        "cookie-signature": "1.0.6",
        "debug": "2.6.8",
        "depd": "1.1.1",
        "encodeurl": "1.0.1",
        "escape-html": "1.0.3",
        "etag": "1.8.0",
        "finalhandler": "1.0.4",
        "fresh": "0.5.0",
        "merge-descriptors": "1.0.1",
        "methods": "1.1.2",
        "on-finished": "2.3.0",
        "parseurl": "1.3.1",
        "path-to-regexp": "0.1.7",
        "proxy-addr": "1.1.5",
        "qs": "6.5.0",
        "range-parser": "1.2.0",
        "send": "0.15.4",
        "serve-static": "1.12.4",
        "setprototypeof": "1.0.3",
        "statuses": "1.3.1",
        "type-is": "1.6.15",
        "utils-merge": "1.0.0",
        "vary": "1.1.1"
      }
    },

Equivalent entries can be found for every package listed in the “requires” section.

The idea then becomes that instead of using package.json to resolve and install modules, npm will use the package-lock.json. Because the package-lock specifies a version, location and integrity hash for every module and each of its dependencies, the install it creates will be the same, every single time. It won’t matter what device you are on, or when in the future you install, it should give you the same result every time, which is very useful.

The Controversy

So if package-lock is supposed to solve a common problem, why are the top search results for it (other than npm documentation) all about disabling it or questioning the role that it plays?

Before npm 5.x.x, package.json was the source of truth for a project. What lived in package.json was law. npm users liked this model and grew very accustomed to maintaining their package file. However, when package-lock was first introduced, it acted contrary to how many people expected it to. Given a pre-existing package and package-lock, a change to the package.json (what many users considered the source of truth) was not reflected in the package-lock.

Example: Package A, version 1.0.0 is in the package and package-lock. In package.json, A is manually edited to version 1.1.0. If a user who considers package.json to be the source of truth runs npm install, they would expect version 1.1.0 to be installed. However, version 1.0.0 is installed, despite the fact that v1.1.0 is listed is the package.json.

Example: A module does not exist in the package-lock, but it does exist in the package.json. As a user who looks to package.json as the source of truth, I would expect for my module to be installed. However since the module is not present in package-lock, it isn’t installed, and my code fails because it cannot find the module.

Much of the time, because they couldn’t figure out why their dependencies weren’t being installed correctly, users either deleted package-lock and reinstalled, or would disable the package-lock altogether.

This conflict between expect and real behavior sparked a very interesting issue thread in the npm repo. Some people thought that the package.json should be the source of truth, some people thought that since package-lock is what npm uses to create the install, that should be considered the source of truth. The resolution to this controversy lies in npm PR #17508. Npm maintainers added a change that causes package.json to overrule the package-lock if package.json has been updated. Now in both above scenarios, the packages that a user would expect to be installed are installed correctly. This change was released as a part of npm v5.1.0, which went live on July 5th, 2017.

Written by James Quigley, an SRE/DevOps Engineer, and general tech nerd. Views and opinions are my own. Check out my YouTube Channel or follow me on Twitter!