Devblog #8 - Improved DX

Introduction

The main focus this time was on improving the developer experience. Some of it was planned, like the report parameter validation rework and some of it came naturally, like the improved data collection from the dependency tree.

info

This devblog is about the packageanalyzer project.

A framework to introspect Node.js packages.

Please find a short introduction and the motivation for this project here.

A built in way to collect custom data

There is now an easy way to traverse the dependency tree and collect custom data.

Problem

While working on a new feature to identify preinstall and postinstall scripts I realized there is no built in way to traverse the dependency tree and collect custom data.

There is the visit method on the Package class but this will only allow a callback to be executed for every dependency. Any data that you want to collect during this traversal needs to be saved manually.

So if we want to collect all postinstall entries, the code might look something like this:

// hold all the data we want to collect
const postinstallMap: Map<string, string> = new Map();

// iterate through the dependency tree
root.visit(pkg => {
    const postinstallEntry = pkg.getData("scripts.postinstall");

    if (postinstallEntry) postinstallMap.set(pkg.fullName, postinstallEntry);
}, true /* start from root */);

While this doesn't look too bad, it's possible that the same dependency (same version) exists multiple times in the dependency tree. If we want to collect all instances (because we want to display a path for each entry the above code wouldn't work as the key is the name + version.

To track all instances we need the key to be the actual Package:

const postinstallMap: Map<Package, string> = new Map();

root.visit(pkg => {
    const postinstallEntry = pkg.getData("scripts.postinstall");

    if (postinstallEntry) postinstallMap.set(pkg, postinstallEntry);
}, true);

Now we get all possible instances where there is a postinstall script. But since how the data is saved is totally up to the developer, another approach would be to use a tuple:

const postinstallTuples: [Package, string][] = [];

root.visit(pkg => {
    const postinstallEntry = pkg.getData("scripts.postinstall");

    if (postinstallEntry) postinstallTuples.push([pkg, postinstallEntry]);
}, true);

That would work as well but what if we want to group same dependencies together? Whoops, yet another implementation is needed.

You see, it can get out of hand real quick real fast.

The problem is that there are multiple ways to collect the data, the burden is on the developer.

On top of figuring out what data he wants to collect, he also needs to figure out how to collect that data.

What's more, traversing and collecting data from the dependecy tree is such a common theme that there really ought to be an official way to do it.

Solution

The Package class now contains a collect method that lets you iterate through the dependency tree and collect custom data along the way.

It is itself nested in structure, containing accessors to the parent as well as to the children (dependencies).

Additionally it contains a flatten method to turn the nested structure into an easy to consume data structure, automatically grouping same dependencies (dependencies with same version).

The collect method takes a callback where the return value of the callback is the data that is getting collected.

The implementation of the postinstall example can now look like this:

// collect postinstall entries and group same dependencies
const postinstallMap = root.collect(pkg => pkg.getData("scripts.postinstall")).flatten();

Much more straight forward 🙌.

If we want to print out the information we can do it with a simple for of loop:

// print out
for (const [[pkg, ...rest], postinstall] of postinstallMap) {
    if (postinstall) console.log(`Found postinstall entry for ${pkg.fullName}: "${postinstall}"`);
    console.log(`${pkg.fullName} exists ${rest.length + 1}x in the dependency tree`);
}

This would produce output similar to:

Found postinstall entry for somelibrary@1.2.3: "node ./postinstall.js"
somelibrary@1.2.3 exists 1x in the dependency tree

If we want to also add preinstall entries we can simply add it to the collect return value:

Here utilizing an interface to describe the return value, so the implementation returns the correct data

interface IScripts {
    preinstall?: string;
    postinstall?: string;
}

const scriptMap = root
    .collect<IScripts>(pkg => ({
        preinstall: pkg.getData("scripts.preinstall"),
        postinstall: pkg.getData("scripts.postinstall")
    }))
    .flatten();

🙌

Now collecting custom data is super easy as you only need to worry about what data you want to collect and not how you collect it.

Improved parameter validation

The Report functionality contains an optional validate method.

This method (if present) is used to validate the input parameters for a Report at runtime.

If not implemented, it will fall back to the TypeScript defined types but since the parameters could come from user input it's not the safest approach, that's why there is a dedicated validate method to validate the parameters. The extra validation also allows to provide good error messages when the wrong parameters are received.

Up until now the validate method relied on io-ts for its validation. It's an awesome library, however its focus on functional programming makes it a little hard to get into, aka the developer experience is not where I want it to be. As said its an awesome library and its strength lies in functional programming but there is rarely a need for it in this project. Most of the time you just want to validate a simple enough schema, not pipe it further etc.

Then last month I stumbled upon Zod. A validation library that focuses less on functional programming aka is more aligned with the desired developer experience.

Comparison

To illustrate, here's a little comparison between io-ts and Zod, suppose we want to validate the following schema:

interface IDiffReportParams {
    from: string;
    to: string;
    type: "dependencies" | "devDependencies";
}

io-ts

To validate the type attribute we need to define a custom type:

type _DependencyType = "dependencies" | "devDependencies";

export const dependencyType = new t.Type<_DependencyType>(
    "dependencyType",
    (input: unknown): input is _DependencyType =>
        input === "dependencies" || input === "devDependencies",
    (input, context) => {
        if (input === "dependencies" || input === "devDependencies") {
            return t.success(input);
        }

        return t.failure(
            input,
            context,
            `Expected "dependencies" or "devDependencies" but got "${input}"`
        );
    },
    t.identity
);

type DependencyTypes = t.TypeOf<typeof dependencyType>;

That's a lot of code just to validate if something is of type "dependencies" | "devDependencies". To be fair though, there is a reason why it needs all that code but that functionality is not needed in this project.

To validate the to and from attribute we can use a built in type:

const FromParamenter = t.type({
    from: t.string
});

const ToParamenter = t.type({
    to: t.string
});

Then we can assemble the schema and generate the corresponding interface:

const DiffParams = t.intersection([FromParamenter, ToParamenter, TypeParameter]);

type IDiffReportParams = t.TypeOf<typeof DiffParams>;

Zod

Here's the same thing with Zod, first we define the type attribute:

const dependencyTypes = z.union([z.literal(`dependencies`), z.literal(`devDependencies`)], {
    invalid_type_error: `type must be "dependencies" or "devDependencies"`
});

Much more straight forward than its io-ts counterpart!

Then to and from:

const FromParameter = z.object({
    from: z.string()
});

const To = z.object({
    to: z.string()
});

Lastly, we generate the whole schema and corresponding interface:

const DiffParams = FromParameter.merge(ToParameter).merge(TypeParameter);

type IDiffReportParams = z.infer<typeof DiffParams>;

Zod overall requires less code, this is also highlighted by the fact that the pull request for replacing io-ts with Zod removed more code than it added:

On top it's also easier to grasp.

Validation

Now let's look at how validation works with their respective schemas:

io-ts

const result = DiffParams.decode(someinput);

if (isRight(result)) {
    // valiation was successful, result.right holds the validated data
    const params: IDiffReportParams = result.right;
} else if (isLeft(result)) {
    // validation was not successful
}

isLeft and isRight are functions provided by io-ts and are due to its focus on functional programming.

Zod

const result = DiffParams.safeParse(someinput);

if (result.success) {
    // validation was successful, result.data holds the validated data
    const params: IDiffReportParams = result.data;
} else {
    // validation was not successful
}

Again Zod is more straight forward and thus makes it easier to write validation code for your Reports.

And that is the end of this devblog!