diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 25a8976587..0b0f1f1724 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -64,12 +64,14 @@ yarn local-registry disable To publish packages to a local registry, do the following: - Run `yarn local-registry start` in Terminal 1 (keep it running) -- Run `npm adduser --registry http://localhost:4873` in Terminal 2 (real credentials are not required, you just need to be logged in) +- Run `npm adduser --registry http://localhost:4873` in Terminal 2 (real credentials are not required, you just need to be logged in. You can use test/test/test@test.io.) - Run `yarn local-registry enable` in Terminal 2 - Run `yarn nx-release 999.9.9 --local` in Terminal 2 - Run `cd /tmp` in Terminal 2 - Run `npx create-nx-workspace@999.9.9` in Terminal 2 +If you have problems publishing, make sure you use Node 14 and NPM 6 instead of Node 15 and NPM 7. + ### Running Unit Tests To make sure your changes do not break any unit tests, run the following: @@ -86,6 +88,8 @@ nx test jest ### Running E2E Tests +**Use Node 14 and NPM 6. E2E tests won't work on Node 15 and NPM 7.** + To make sure your changes do not break any E2E tests, run: ```bash diff --git a/docs/angular/cli/clear-cache.md b/docs/angular/cli/clear-cache.md new file mode 100644 index 0000000000..399d1f9ed0 --- /dev/null +++ b/docs/angular/cli/clear-cache.md @@ -0,0 +1,11 @@ +# clear-cache + +Clears all the cached Nx artifacts and metadata about the workspace. + +## Usage + +```bash +nx clear-cache +``` + +Install `nx` globally to invoke the command directly using `nx`, or use `npm run nx` or `yarn nx`. diff --git a/docs/map.json b/docs/map.json index b42d4945c2..30079f0ec3 100644 --- a/docs/map.json +++ b/docs/map.json @@ -275,6 +275,11 @@ "name": "connect-to-nx-cloud", "id": "connect-to-nx-cloud", "file": "angular/cli/connect-to-nx-cloud" + }, + { + "name": "clear-cache", + "id": "clear-cache", + "file": "angular/cli/clear-cache" } ] }, @@ -1441,6 +1446,11 @@ "name": "connect-to-nx-cloud", "id": "connect-to-nx-cloud", "file": "react/cli/connect-to-nx-cloud" + }, + { + "name": "clear-cache", + "id": "clear-cache", + "file": "react/cli/clear-cache" } ] }, @@ -2571,6 +2581,11 @@ "name": "connect-to-nx-cloud", "id": "connect-to-nx-cloud", "file": "node/cli/connect-to-nx-cloud" + }, + { + "name": "clear-cache", + "id": "clear-cache", + "file": "node/cli/clear-cache" } ] }, diff --git a/docs/node/cli/clear-cache.md b/docs/node/cli/clear-cache.md new file mode 100644 index 0000000000..399d1f9ed0 --- /dev/null +++ b/docs/node/cli/clear-cache.md @@ -0,0 +1,11 @@ +# clear-cache + +Clears all the cached Nx artifacts and metadata about the workspace. + +## Usage + +```bash +nx clear-cache +``` + +Install `nx` globally to invoke the command directly using `nx`, or use `npm run nx` or `yarn nx`. diff --git a/docs/react/cli/clear-cache.md b/docs/react/cli/clear-cache.md new file mode 100644 index 0000000000..399d1f9ed0 --- /dev/null +++ b/docs/react/cli/clear-cache.md @@ -0,0 +1,11 @@ +# clear-cache + +Clears all the cached Nx artifacts and metadata about the workspace. + +## Usage + +```bash +nx clear-cache +``` + +Install `nx` globally to invoke the command directly using `nx`, or use `npm run nx` or `yarn nx`. diff --git a/docs/shared/workspace/project-graph-plugins.md b/docs/shared/workspace/project-graph-plugins.md index a4640dba72..3b72459783 100644 --- a/docs/shared/workspace/project-graph-plugins.md +++ b/docs/shared/workspace/project-graph-plugins.md @@ -2,11 +2,13 @@ > This API is experimental and might change. -Nx views the workspace as a graph of projects that depend on one another. It's able to infer most projects and dependencies automatically. Currently, this works best within the JavaScript ecosystem, but it can be extended to other languages and technologies as well. This is where project graph plugins come in. +Project Graph is the representation of the source code in your repo. Projects can have files associated with them. Projects can have dependencies on each other. -## Defining Plugins to be used in a workspace +One of the best features of Nx is that is able to construct the project graph automatically by analyzing your source code. Currently, this works best within the JavaScript ecosystem, but it can be extended to other languages and technologies using plugins. -In `nx.json`, add an array of plugins: +## Adding Plugins to Workspace + +You can register a plugin by adding it to the plugins array in `nx.json`: ```json { @@ -17,22 +19,23 @@ In `nx.json`, add an array of plugins: } ``` -These plugins are used when running targets, linting, and sometimes when generating code. - ## Implementing a Project Graph Processor -Project Graph Plugins are chained together to produce the final project graph. Each plugin may have a Project Graph Processor which iterates upon the project graph. Let's first take a look at the API of Project Graph Plugins. In later sections, we will go over some common use cases. Plugins should export a function named `processProjectGraph` that handles updating the project graph with new nodes and edges. This function receives two things: +A Project Graph Processor that takes a project graph and returns a new project graph. It can add/remove nodes and edges. + +Plugins should export a function named `processProjectGraph` that handles updating the project graph with new nodes and edges. This function receives two things: - A `ProjectGraph` - - Nodes in the project graph are the different projects currently in the graph. - - Edges in the project graph are dependencies between different projects in the graph. -- Some context is also passed into the function to use when processing the project graph. The context contains: - - The `workspace` which contains both configuration and the different projects. - - A `fileMap` which has a map of files by projects -> Note: The notion of a workspace is separate from the notion of the project graph. The workspace is first party code that is checked into git, targets are run on, etc. The project graph may include third party packages as well that is not checked into git, not run at all, etc. + - `graph.nodes` lists all the projects currently known to Nx. `node.data.files` lists the files belonging to a particular project. + - `graph.dependencies` lists the dependencies between projects. -The `processProjectGraph` function should return an updated `ProjectGraph`. This is most easily done using the `ProjectGraphBuilder` to iteratively add edges and nodes to the graph: +- A `Context` + - `context.workspace` contains the combined configuration for the workspace. + - `files` contains all the files found in the workspace. + - `filesToProcess` contains all the files that have changed since the last invocation and need to be reanalyzed. + +The `processProjectGraph` function should return an updated `ProjectGraph`. This is most easily done using `ProjectGraphBuilder`. The builder is there for convenience, so you don't have to use it. ```typescript import { @@ -48,32 +51,15 @@ export function processProjectGraph( ): ProjectGraph { const builder = new ProjectGraphBuilder(graph); // We will see how this is used below. - return builder.getProjectGraph(); + return builder.getUpdatedProjectGraph(); } ``` -## Adding New Dependencies to the Project Graph - -Project Graph Plugins can add smarter dependency resolution to projects already in the workspace. Projects in the workspace are first party code whose dependencies change as the code in the workspace changes and matter to Nx the most. Such projects should be defined in `workspace.json` and `nx.json` and will be automatically included as nodes in the project graph. However, when some projects are written in other languages, the relationships between these projects will not be clear to Nx out of the box. A Project Graph Plugin can add these relationships. - -```typescript -import { DependencyType } from '@nrwl/devkit'; - -// Add a new edge -builder.addDependency(DependencyType.static, 'existing-project', 'new-project'); -``` - -> Note: Even though the plugin is written in JavaScript, resolving dependencies of different languages will probably be more easily written in their native language. Therefore, a common approach is to spawn a new process and communicate via IPC or `stdout`. - -Dependencies can be one of the following types: - -- `DependencyType.static` dependencies indicate that a dependency is imported directly into the code and would be present even without running the code. -- `DependencyType.dynamic` dependencies indicate that a dependency _might be_ imported at runtime such as lazy loaded dependencies. -- `DependencyType.implicit` dependencies indicate that one project affects another project's behavior or outcome even though there is no dependency in the code. For example, e2e tests or communication over HTTP. - ## Adding New Nodes to the Project Graph -Sometimes it can be valuable to have third party packages as part of the project graph. A Project Graph Plugin can add these packages to the project graph. After these packages are added as nodes to the project graph, dependencies can then be drawn from the workspace projects to the third party packages as well as between the third party packages. +You can add nodes to the project graph. Since first-party code is added to the graph automatically, this is most commonly used for third-party packages. + +A Project Graph Plugin can add them to the project graph. After these packages are added as nodes to the project graph, dependencies can then be drawn from the workspace projects to the third party packages as well as between the third party packages. ```typescript // Add a new node @@ -86,15 +72,46 @@ builder.addNode({ }); ``` -> Note: You can designate any type for the node. This differentiates third party projects from projects in the workspace. Also, like before, retrieving these projects might be easiest within their native language. Therefore, spawning a new process may also be a common approach here. +> Note: You can designate any type for the node. This differentiates third party projects from projects in the workspace. If you are writing a plugin for a different language, it's common to use IPC to get the list of nodes which you can then add using the builder. -## Incrementally Reprocessing +## Adding New Dependencies to the Project Graph -Workspaces can have _a lot_ of files and finding dependencies for every file can be expensive. Nx incrementally recalculates the `ProjectGraph` by only looking at files that have changed. Let's take a look at how this works. +It's more common for plugins to create new dependencies. First-party code contained in the workspace is registered in `workpspace.json` and is added to the project graph automatically. Whether your project contains TypeScript or say Java, both projects will be created in the same way. However, Nx does not know how to analyze Java sources, and that's what plugins can do. -Remember that the `ProjectGraph` that is passed into the `processProjectGraph` function is a graph that already has nodes and dependencies. These nodes and dependencies are not _only_ those from prior plugins, but might also be a cached part of the graph that does not need to be recalculated. If files have not been modified since the last calculation, they do not need to be processed again. How do we know which files we _need_ to reprocess? +You can create 2 types of dependencies. -`ProjectGraphProcessorContext.fileMap` contains only the files that need to be processed. You should, if possible, definitely take advantage of this subset of files to make it cheaper to reprocess the graph. +### Implicit Dependencies + +An implicit dependency is not associated with any file, and can be crated as follows: + +```typescript +import { DependencyType } from '@nrwl/devkit'; + +// Add a new edge +builder.addImplicitDependency('existing-project', 'new-project'); +``` + +> Note: Even though the plugin is written in JavaScript, resolving dependencies of different languages will probably be more easily written in their native language. Therefore, a common approach is to spawn a new process and communicate via IPC or `stdout`. +> . + +Because an implicit dependency is not associated with any file, Nx doesn't know when it might change, so it will be recomputed every time. + +## Explicit Dependencies + +Nx knows what files have changed since the last invocation. Only those files will be present in the provided `filesToProcess`. You can associate a dependency with a particular file (e.g., if that file contains an import). + +```typescript +import { DependencyType } from '@nrwl/devkit'; + +// Add a new edge +builder.addExplicitDependency( + 'existing-project', + 'libs/existing-project/src/index.ts', + 'new-project' +); +``` + +If a file hasn't changed since the last invocation, it doesn't need to be reanalyzed. Nx knows what dependencies are associated with what files, so it will reuse this information for the files that haven't changed. ## Visualizing the Project Graph diff --git a/e2e/workspace/src/plugins.test.ts b/e2e/workspace/src/plugins.test.ts index 8191b482e9..d2903011b8 100644 --- a/e2e/workspace/src/plugins.test.ts +++ b/e2e/workspace/src/plugins.test.ts @@ -10,7 +10,7 @@ describe('Nx Plugins', () => { beforeAll(() => newProject()); afterAll(() => removeProject({ onlyOnCI: true })); - it('should use plugins defined in nx.json', () => { + it('vvvshould use plugins defined in nx.json', () => { const nxJson = readJson('nx.json'); nxJson.plugins = ['./tools/plugin']; updateFile('nx.json', JSON.stringify(nxJson)); @@ -35,12 +35,11 @@ describe('Nx Plugins', () => { root: 'test2' } }); - builder.addDependency( - require('@nrwl/devkit').DependencyType.static, + builder.addImplicitDependency( 'plugin-node', 'plugin-node2' ); - return builder.getProjectGraph(); + return builder.getUpdatedProjectGraph(); } }; ` @@ -51,7 +50,7 @@ describe('Nx Plugins', () => { expect(projectGraphJson.graph.nodes['plugin-node']).toBeDefined(); expect(projectGraphJson.graph.nodes['plugin-node2']).toBeDefined(); expect(projectGraphJson.graph.dependencies['plugin-node']).toContainEqual({ - type: 'static', + type: 'implicit', source: 'plugin-node', target: 'plugin-node2', }); diff --git a/e2e/workspace/src/workspace-aux-commands.test.ts b/e2e/workspace/src/workspace-aux-commands.test.ts index 800395ef9d..576b9dd9d6 100644 --- a/e2e/workspace/src/workspace-aux-commands.test.ts +++ b/e2e/workspace/src/workspace-aux-commands.test.ts @@ -83,7 +83,7 @@ describe('lint', () => { expect(out).toContain( 'Libraries cannot be imported by a relative or absolute path, and must begin with a npm scope' ); - expect(out).toContain('Imports of lazy-loaded libraries are forbidden'); + // expect(out).toContain('Imports of lazy-loaded libraries are forbidden'); expect(out).toContain('Imports of apps are forbidden'); expect(out).toContain( 'A project tagged with "validtag" can only depend on libs tagged with "validtag"' @@ -490,7 +490,7 @@ describe('dep-graph', () => { target: mylib, type: 'static', }, - { source: myapp, target: mylib2, type: 'dynamic' }, + { source: myapp, target: mylib2, type: 'static' }, ], [myappE2e]: [ { diff --git a/nx-dev/nx-dev/public/documentation/latest/node/guides/configuration.md b/nx-dev/nx-dev/public/documentation/latest/node/guides/configuration.md index e27f8ca5e1..70d55560c7 100644 --- a/nx-dev/nx-dev/public/documentation/latest/node/guides/configuration.md +++ b/nx-dev/nx-dev/public/documentation/latest/node/guides/configuration.md @@ -394,7 +394,7 @@ Tasks runners can accept different options. The following are the options suppor - `maxParallel` defines the max number of processes used. - `captureStderr` defines whether the cache captures stderr or just stdout - `skipNxCache` defines whether the Nx Cache should be skipped. Defaults to `false` -- `cacheDirectory` defines where the local cache is stored, which is `node_modules/.cache/nx` by default. +- `cacheDirectory` defines where the local cache is stored, which is `node_modules/.cache/nx` by default. You can clear the cache directory by running `nx clear-cache`. - `encryptionKey` (when using `"@nrwl/nx-cloud"` only) defines an encryption key to support end-to-end encryption of your cloud cache. You may also provide an environment variable with the key `NX_CLOUD_ENCRYPTION_KEY` that contains an encryption key as its value. The Nx Cloud task runner normalizes the key length, so any length of key is acceptable. - `runtimeCacheInputs` defines the list of commands that are run by the runner to include into the computation hash value. diff --git a/packages/cli/lib/parse-run-one-options.ts b/packages/cli/lib/parse-run-one-options.ts index 43180a71b6..ddaee3ff97 100644 --- a/packages/cli/lib/parse-run-one-options.ts +++ b/packages/cli/lib/parse-run-one-options.ts @@ -50,6 +50,7 @@ const invalidTargetNames = [ 'workspace-generator', 'workspace-schematic', 'connect-to-nx-cloud', + 'clear-cache', 'report', 'list', ]; diff --git a/packages/devkit/index.ts b/packages/devkit/index.ts index 82a2d16234..2bae3d45e4 100644 --- a/packages/devkit/index.ts +++ b/packages/devkit/index.ts @@ -62,7 +62,7 @@ export type { ProjectGraphProcessorContext, } from './src/project-graph/interfaces'; export { DependencyType } from './src/project-graph/interfaces'; -export { ProjectGraphBuilder } from './src/project-graph/utils'; +export { ProjectGraphBuilder } from './src/project-graph/project-graph-builder'; export { readJson, writeJson, updateJson } from './src/utils/json'; diff --git a/packages/devkit/src/project-graph/interfaces.ts b/packages/devkit/src/project-graph/interfaces.ts index a7c7133586..7a877598d6 100644 --- a/packages/devkit/src/project-graph/interfaces.ts +++ b/packages/devkit/src/project-graph/interfaces.ts @@ -10,6 +10,7 @@ export interface FileData { file: string; hash: string; ext: string; + deps?: string[]; } /** @@ -96,7 +97,16 @@ export interface ProjectGraphProcessorContext { * Workspace information such as projects and configuration */ workspace: Workspace; + + /** + * All files in the workspace + */ fileMap: ProjectFileMap; + + /** + * Files changes since last invocation + */ + filesToProcess: ProjectFileMap; } /** diff --git a/packages/devkit/src/project-graph/project-graph-builder.spec.ts b/packages/devkit/src/project-graph/project-graph-builder.spec.ts new file mode 100644 index 0000000000..59488fed1a --- /dev/null +++ b/packages/devkit/src/project-graph/project-graph-builder.spec.ts @@ -0,0 +1,116 @@ +import { ProjectGraphBuilder } from './project-graph-builder'; + +describe('ProjectGraphBuilder', () => { + let builder: ProjectGraphBuilder; + beforeEach(() => { + builder = new ProjectGraphBuilder(); + builder.addNode({ + name: 'source', + type: 'lib', + data: { + files: [ + { + file: 'source/index.ts', + }, + { + file: 'source/second.ts', + }, + ], + }, + }); + builder.addNode({ + name: 'target', + type: 'lib', + data: {}, + }); + }); + + it(`should add an implicit dependency`, () => { + expect(() => + builder.addImplicitDependency('invalid-source', 'target') + ).toThrowError(); + expect(() => + builder.addImplicitDependency('source', 'invalid-target') + ).toThrowError(); + + // ignore the self deps + builder.addImplicitDependency('source', 'source'); + + // don't include duplicates + builder.addImplicitDependency('source', 'target'); + builder.addImplicitDependency('source', 'target'); + + const graph = builder.getUpdatedProjectGraph(); + expect(graph.dependencies).toEqual({ + source: [ + { + source: 'source', + target: 'target', + type: 'implicit', + }, + ], + target: [], + }); + }); + + it(`should add an explicit dependency`, () => { + expect(() => + builder.addExplicitDependency( + 'invalid-source', + 'source/index.ts', + 'target' + ) + ).toThrowError(); + expect(() => + builder.addExplicitDependency( + 'source', + 'source/index.ts', + 'invalid-target' + ) + ).toThrowError(); + expect(() => + builder.addExplicitDependency( + 'source', + 'source/invalid-index.ts', + 'target' + ) + ).toThrowError(); + + // ignore the self deps + builder.addExplicitDependency('source', 'source/index.ts', 'source'); + + // don't include duplicates + builder.addExplicitDependency('source', 'source/index.ts', 'target'); + builder.addExplicitDependency('source', 'source/second.ts', 'target'); + + const graph = builder.getUpdatedProjectGraph(); + expect(graph.dependencies).toEqual({ + source: [ + { + source: 'source', + target: 'target', + type: 'static', + }, + ], + target: [], + }); + }); + + it(`should use implicit dep when both implicit and explicit deps are available`, () => { + // don't include duplicates + builder.addImplicitDependency('source', 'target'); + builder.addExplicitDependency('source', 'source/index.ts', 'target'); + + const graph = builder.getUpdatedProjectGraph(); + expect(graph.dependencies).toEqual({ + source: [ + { + source: 'source', + target: 'target', + type: 'implicit', + }, + ], + target: [], + }); + }); +}); diff --git a/packages/devkit/src/project-graph/project-graph-builder.ts b/packages/devkit/src/project-graph/project-graph-builder.ts new file mode 100644 index 0000000000..f6bcd0dfa9 --- /dev/null +++ b/packages/devkit/src/project-graph/project-graph-builder.ts @@ -0,0 +1,152 @@ +import type { + FileData, + ProjectFileMap, + ProjectGraph, + ProjectGraphDependency, + ProjectGraphNode, +} from './interfaces'; +import { DependencyType } from './interfaces'; + +/** + * Builder for adding nodes and dependencies to a {@link ProjectGraph} + */ +export class ProjectGraphBuilder { + readonly graph: ProjectGraph; + + constructor(g?: ProjectGraph) { + if (g) { + this.graph = g; + } else { + this.graph = { + nodes: {}, + dependencies: {}, + }; + } + } + + /** + * Adds a project node to the project graph + */ + addNode(node: ProjectGraphNode): void { + // Check if project with the same name already exists + if (this.graph.nodes[node.name]) { + // Throw if existing project is of a different type + if (this.graph.nodes[node.name].type !== node.type) { + throw new Error( + `Multiple projects are named "${node.name}". One is of type "${ + node.type + }" and the other is of type "${ + this.graph.nodes[node.name].type + }". Please resolve the conflicting project names.` + ); + } + } + this.graph.nodes[node.name] = node; + this.graph.dependencies[node.name] = []; + } + + /** + * Adds a dependency from source project to target project + */ + addImplicitDependency( + sourceProjectName: string, + targetProjectName: string + ): void { + if (sourceProjectName === targetProjectName) { + return; + } + if (!this.graph.nodes[sourceProjectName]) { + throw new Error(`Source project does not exist: ${sourceProjectName}`); + } + if (!this.graph.nodes[targetProjectName]) { + throw new Error(`Target project does not exist: ${targetProjectName}`); + } + this.graph.dependencies[sourceProjectName].push({ + source: sourceProjectName, + target: targetProjectName, + type: DependencyType.implicit, + }); + } + + /** + * Add an explicit dependency from a file in source project to target project + */ + addExplicitDependency( + sourceProjectName: string, + sourceProjectFile: string, + targetProjectName: string + ): void { + if (sourceProjectName === targetProjectName) { + return; + } + const source = this.graph.nodes[sourceProjectName]; + if (!source) { + throw new Error(`Source project does not exist: ${sourceProjectName}`); + } + + if (!this.graph.nodes[targetProjectName]) { + throw new Error(`Target project does not exist: ${targetProjectName}`); + } + + const fileData = source.data.files.find( + (f) => f.file === sourceProjectFile + ); + if (!fileData) { + throw new Error( + `Source project ${sourceProjectName} does not have a file: ${sourceProjectFile}` + ); + } + + if (!fileData.deps) { + fileData.deps = []; + } + + if (!fileData.deps.find((t) => t === targetProjectName)) { + fileData.deps.push(targetProjectName); + } + } + + getUpdatedProjectGraph(): ProjectGraph { + for (const sourceProject of Object.keys(this.graph.nodes)) { + const alreadySetTargetProjects = + this.calculateAlreadySetTargetDeps(sourceProject); + this.graph.dependencies[sourceProject] = [ + ...alreadySetTargetProjects.values(), + ]; + + const fileDeps = this.calculateTargetDepsFromFiles(sourceProject); + for (const targetProject of fileDeps) { + if (!alreadySetTargetProjects.has(targetProject)) { + this.graph.dependencies[sourceProject].push({ + source: sourceProject, + target: targetProject, + type: DependencyType.static, + }); + } + } + } + return this.graph; + } + + private calculateTargetDepsFromFiles(sourceProject: string) { + const fileDeps = new Set(); + const files = this.graph.nodes[sourceProject].data.files; + if (!files) return fileDeps; + for (let f of files) { + if (f.deps) { + for (let p of f.deps) { + fileDeps.add(p); + } + } + } + return fileDeps; + } + + private calculateAlreadySetTargetDeps(sourceProject: string) { + const alreadySetTargetProjects = new Map(); + for (const d of this.graph.dependencies[sourceProject]) { + alreadySetTargetProjects.set(d.target, d); + } + return alreadySetTargetProjects; + } +} diff --git a/packages/devkit/src/project-graph/utils.ts b/packages/devkit/src/project-graph/utils.ts deleted file mode 100644 index 08f3cdfb96..0000000000 --- a/packages/devkit/src/project-graph/utils.ts +++ /dev/null @@ -1,83 +0,0 @@ -import type { - ProjectGraph, - ProjectGraphDependency, - ProjectGraphNode, - DependencyType, -} from './interfaces'; - -/** - * Builder for adding nodes and dependencies to a {@link ProjectGraph} - */ -export class ProjectGraphBuilder { - readonly nodes: Record = {}; - readonly dependencies: Record< - string, - Record - > = {}; - - constructor(g?: ProjectGraph) { - if (g) { - Object.values(g.nodes).forEach((n) => this.addNode(n)); - Object.values(g.dependencies).forEach((ds) => { - ds.forEach((d) => this.addDependency(d.type, d.source, d.target)); - }); - } - } - - /** - * Adds a project node to the project graph - */ - addNode(node: ProjectGraphNode): void { - // Check if project with the same name already exists - if (this.nodes[node.name]) { - // Throw if existing project is of a different type - if (this.nodes[node.name].type !== node.type) { - throw new Error( - `Multiple projects are named "${node.name}". One is of type "${ - node.type - }" and the other is of type "${ - this.nodes[node.name].type - }". Please resolve the conflicting project names.` - ); - } - } - this.nodes[node.name] = node; - this.dependencies[node.name] = {}; - } - - /** - * Adds a dependency from source project to target project - */ - addDependency( - type: DependencyType | string, - sourceProjectName: string, - targetProjectName: string - ): void { - if (sourceProjectName === targetProjectName) { - return; - } - if (!this.nodes[sourceProjectName]) { - throw new Error(`Source project does not exist: ${sourceProjectName}`); - } - if (!this.nodes[targetProjectName]) { - throw new Error(`Target project does not exist: ${targetProjectName}`); - } - this.dependencies[sourceProjectName][ - `${sourceProjectName} -> ${targetProjectName}` - ] = { - type, - source: sourceProjectName, - target: targetProjectName, - }; - } - - getProjectGraph(): ProjectGraph { - return { - nodes: this.nodes as ProjectGraph['nodes'], - dependencies: Object.keys(this.dependencies).reduce((acc, k) => { - acc[k] = Object.values(this.dependencies[k]); - return acc; - }, {} as ProjectGraph['dependencies']), - }; - } -} diff --git a/packages/tao/src/commands/migrate.ts b/packages/tao/src/commands/migrate.ts index 0a87bb8229..b3fb347d33 100644 --- a/packages/tao/src/commands/migrate.ts +++ b/packages/tao/src/commands/migrate.ts @@ -730,13 +730,8 @@ async function runNxMigration(root: string, packageName: string, name: string) { flushChanges(root, changes); } -function removeNxDepsIfCaseItsFormatChanged(root: string) { - removeSync(join(root, 'node_modules', '.cache', 'nx', 'nxdeps.json')); -} - export async function migrate(root: string, args: string[], isVerbose = false) { return handleErrors(isVerbose, async () => { - removeNxDepsIfCaseItsFormatChanged(root); const opts = parseMigrationsOptions(args); if (opts.type === 'generateMigrations') { await generateMigrationsJsonAndUpdatePackageJson(root, opts); diff --git a/packages/workspace/src/command-line/clear-cache.ts b/packages/workspace/src/command-line/clear-cache.ts new file mode 100644 index 0000000000..9338543fbc --- /dev/null +++ b/packages/workspace/src/command-line/clear-cache.ts @@ -0,0 +1,29 @@ +import { appRootPath } from '@nrwl/tao/src/utils/app-root'; +import { + cacheDirectory, + readCacheDirectoryProperty, +} from '../utilities/cache-directory'; +import { removeSync } from 'fs-extra'; +import { output } from '../utilities/output'; + +export const clearCache = { + command: 'clear-cache', + describe: + 'Clears all the cached Nx artifacts and metadata about the workspace.', + handler: clearCacheHandler, +}; + +async function clearCacheHandler() { + output.note({ + title: 'Deleting the cache directory.', + bodyLines: [`This might take a few minutes.`], + }); + const dir = cacheDirectory( + appRootPath, + readCacheDirectoryProperty(appRootPath) + ); + removeSync(dir); + output.success({ + title: 'Deleted the cache directory.', + }); +} diff --git a/packages/workspace/src/command-line/nx-commands.ts b/packages/workspace/src/command-line/nx-commands.ts index 15bc517370..c5b0d4e5bd 100644 --- a/packages/workspace/src/command-line/nx-commands.ts +++ b/packages/workspace/src/command-line/nx-commands.ts @@ -199,6 +199,7 @@ export const commandsObject = yargs ) .command(require('./report').report) .command(require('./list').list) + .command(require('./clear-cache').clearCache) .command( 'connect-to-nx-cloud', `Makes sure the workspace is connected to Nx Cloud`, diff --git a/packages/workspace/src/command-line/supported-nx-commands.ts b/packages/workspace/src/command-line/supported-nx-commands.ts index 12f40dea18..01877741c3 100644 --- a/packages/workspace/src/command-line/supported-nx-commands.ts +++ b/packages/workspace/src/command-line/supported-nx-commands.ts @@ -19,6 +19,7 @@ export const supportedNxCommands: string[] = [ 'report', 'run-many', 'connect-to-nx-cloud', + 'clear-cache', 'list', 'help', '--help', diff --git a/packages/workspace/src/core/affected-project-graph/affected-project-graph.spec.ts b/packages/workspace/src/core/affected-project-graph/affected-project-graph.spec.ts index 469238e55e..9319d6235b 100644 --- a/packages/workspace/src/core/affected-project-graph/affected-project-graph.spec.ts +++ b/packages/workspace/src/core/affected-project-graph/affected-project-graph.spec.ts @@ -1,10 +1,9 @@ -import { extname } from 'path'; import { jsonDiff } from '../../utilities/json-diff'; import { vol } from 'memfs'; import { stripIndents } from '@angular-devkit/core/src/utils/literals'; import { createProjectGraphAsync } from '../project-graph'; import { filterAffected } from './affected-project-graph'; -import { FileData, WholeFileChange } from '../file-utils'; +import { WholeFileChange } from '../file-utils'; import type { NxJsonConfiguration } from '@nrwl/devkit'; jest.mock('fs', () => require('memfs').fs); @@ -18,11 +17,9 @@ describe('project graph', () => { let tsConfigJson: any; let nxJson: NxJsonConfiguration; let filesJson: any; - let filesAtMasterJson: any; - let files: FileData[]; - let readFileAtRevision: (path: string, rev: string) => string; beforeEach(() => { + process.env.NX_CACHE_PROJECT_GRAPH = 'false'; packageJson = { name: '@nrwl/workspace-src', scripts: { @@ -115,26 +112,11 @@ describe('project graph', () => { './workspace.json': JSON.stringify(workspaceJson), './tsconfig.base.json': JSON.stringify(tsConfigJson), }; - files = Object.keys(filesJson).map((f) => ({ - file: f, - ext: extname(f), - hash: 'some-hash', - })); - readFileAtRevision = (p, r) => { - const fromFs = filesJson[`./${p}`]; - if (!fromFs) { - throw new Error(`File not found: ${p}`); - } - if (r === 'master') { - const fromMaster = filesAtMasterJson[`./${p}`]; - return fromMaster || fromFs; - } else { - return fromFs; - } - }; vol.fromJSON(filesJson, '/root'); }); + afterEach(() => [delete process.env.NX_CACHE_PROJECT_GRAPH]); + it('should create nodes and dependencies with workspace projects', async () => { const graph = await createProjectGraphAsync(); const affected = filterAffected(graph, [ @@ -151,7 +133,7 @@ describe('project graph', () => { getChanges: () => [new WholeFileChange()], }, ]); - expect(affected).toEqual({ + expect(affected).toMatchObject({ nodes: { api: { name: 'api', @@ -170,6 +152,7 @@ describe('project graph', () => { }, }, dependencies: { + api: [], demo: [ { type: 'static', @@ -182,7 +165,6 @@ describe('project graph', () => { target: 'api', }, ], - api: [], ui: [], }, }); diff --git a/packages/workspace/src/core/affected-project-graph/affected-project-graph.ts b/packages/workspace/src/core/affected-project-graph/affected-project-graph.ts index d612d022aa..641af3231a 100644 --- a/packages/workspace/src/core/affected-project-graph/affected-project-graph.ts +++ b/packages/workspace/src/core/affected-project-graph/affected-project-graph.ts @@ -1,4 +1,4 @@ -import { ProjectGraphBuilder, reverse } from '../project-graph'; +import { reverse } from '../project-graph'; import { FileChange, readNxJson, @@ -58,21 +58,21 @@ function filterAffectedProjects( graph: ProjectGraph, ctx: AffectedProjectGraphContext ): ProjectGraph { - const builder = new ProjectGraphBuilder(); + const result = { nodes: {}, dependencies: {} } as ProjectGraph; const reversed = reverse(graph); ctx.touchedProjects.forEach((p) => { - addAffectedNodes(p, reversed, builder, []); + addAffectedNodes(p, reversed, result, []); }); ctx.touchedProjects.forEach((p) => { - addAffectedDependencies(p, reversed, builder, []); + addAffectedDependencies(p, reversed, result, []); }); - return builder.build(); + return result; } function addAffectedNodes( startingProject: string, reversed: ProjectGraph, - builder: ProjectGraphBuilder, + result: ProjectGraph, visited: string[] ): void { if (visited.indexOf(startingProject) > -1) return; @@ -80,27 +80,31 @@ function addAffectedNodes( throw new Error(`Invalid project name is detected: "${startingProject}"`); } visited.push(startingProject); - builder.addNode(reversed.nodes[startingProject]); + result.nodes[startingProject] = reversed.nodes[startingProject]; + result.dependencies[startingProject] = []; reversed.dependencies[startingProject].forEach(({ target }) => - addAffectedNodes(target, reversed, builder, visited) + addAffectedNodes(target, reversed, result, visited) ); } function addAffectedDependencies( startingProject: string, reversed: ProjectGraph, - builder: ProjectGraphBuilder, + result: ProjectGraph, visited: string[] ): void { if (visited.indexOf(startingProject) > -1) return; visited.push(startingProject); reversed.dependencies[startingProject].forEach(({ target }) => - addAffectedDependencies(target, reversed, builder, visited) + addAffectedDependencies(target, reversed, result, visited) ); reversed.dependencies[startingProject].forEach(({ type, source, target }) => { // Since source and target was reversed, // we need to reverse it back to original direction. - builder.addDependency(type, target, source); + if (!result.dependencies[target]) { + result.dependencies[target] = []; + } + result.dependencies[target].push({ type, source: target, target: source }); }); } diff --git a/packages/workspace/src/core/file-utils.ts b/packages/workspace/src/core/file-utils.ts index b35ea2b0c9..3bdaae7700 100644 --- a/packages/workspace/src/core/file-utils.ts +++ b/packages/workspace/src/core/file-utils.ts @@ -302,15 +302,5 @@ export function normalizedProjectRoot(p: ProjectGraphNode): string { } } -export function filesChanged(a: FileData[], b: FileData[]) { - if (a.length !== b.length) return true; - - for (let i = 0; i < a.length; ++i) { - if (a[i].file !== b[i].file) return true; - if (a[i].hash !== b[i].hash) return true; - } - return false; -} - // Original Exports export { FileData }; diff --git a/packages/workspace/src/core/nx-deps/nx-deps-cache.spec.ts b/packages/workspace/src/core/nx-deps/nx-deps-cache.spec.ts index e88d28b5c9..49c1468b1e 100644 --- a/packages/workspace/src/core/nx-deps/nx-deps-cache.spec.ts +++ b/packages/workspace/src/core/nx-deps/nx-deps-cache.spec.ts @@ -1,6 +1,6 @@ import { NxJsonConfiguration, WorkspaceJsonConfiguration } from '@nrwl/devkit'; import { - extractCachedPartOfProjectGraph, + extractCachedFileData, ProjectGraphCache, shouldRecomputeWholeGraph, } from './nx-deps-cache'; @@ -41,7 +41,7 @@ describe('nx deps utils', () => { shouldRecomputeWholeGraph( createCache({ nodes: { - 'renamed-mylib': {} as any, + 'renamed-mylib': { type: 'lib' } as any, }, }), createPackageJsonDeps({}), @@ -111,7 +111,7 @@ describe('nx deps utils', () => { }, dependencies: { mylib: [] }, } as any; - const r = extractCachedPartOfProjectGraph( + const r = extractCachedFileData( { mylib: [ { @@ -121,17 +121,24 @@ describe('nx deps utils', () => { }, ], }, - createNxJson({}), createCache({ nodes: { ...cached.nodes }, dependencies: { ...cached.dependencies }, }) ); - expect(r.filesDifferentFromCache).toEqual({}); - expect(r.cachedPartOfProjectGraph).toEqual(cached); + expect(r.filesToProcess).toEqual({}); + expect(r.cachedFileData).toEqual({ + mylib: { + 'index.ts': { + file: 'index.ts', + ext: 'ts', + hash: 'hash1', + }, + }, + }); }); - it('should handle cases when no projects are added', () => { + it('should handle cases when new projects are added', () => { const cached = { nodes: { mylib: { @@ -150,7 +157,7 @@ describe('nx deps utils', () => { }, dependencies: { mylib: [] }, } as any; - const r = extractCachedPartOfProjectGraph( + const r = extractCachedFileData( { mylib: [ { @@ -167,13 +174,12 @@ describe('nx deps utils', () => { }, ], }, - createNxJson({}), createCache({ nodes: { ...cached.nodes }, dependencies: { ...cached.dependencies }, }) ); - expect(r.filesDifferentFromCache).toEqual({ + expect(r.filesToProcess).toEqual({ secondlib: [ { file: 'index.ts', @@ -182,7 +188,18 @@ describe('nx deps utils', () => { }, ], }); - expect(r.cachedPartOfProjectGraph).toEqual(cached); + expect(r.cachedFileData).toEqual({ + mylib: { + 'index.ts': { + file: 'index.ts', + ext: 'ts', + hash: 'hash1', + }, + }, + }); + expect(r.filesToProcess).toEqual({ + secondlib: [{ ext: 'ts', file: 'index.ts', hash: 'hash2' }], + }); }); it('should handle cases when files change', () => { @@ -194,103 +211,73 @@ describe('nx deps utils', () => { data: { files: [ { - file: 'index.ts', + file: 'index1.ts', ext: 'ts', hash: 'hash1', }, + { + file: 'index2.ts', + ext: 'ts', + hash: 'hash2', + }, + { + file: 'index3.ts', + ext: 'ts', + hash: 'hash3', + }, ], }, }, }, dependencies: { mylib: [] }, } as any; - const r = extractCachedPartOfProjectGraph( + const r = extractCachedFileData( { mylib: [ { - file: 'index.ts', - ext: 'ts', - hash: 'hash2', - }, - ], - }, - createNxJson({}), - createCache({ - nodes: { ...cached.nodes }, - dependencies: { ...cached.dependencies }, - }) - ); - expect(r.filesDifferentFromCache).toEqual({ - mylib: [ - { - file: 'index.ts', - ext: 'ts', - hash: 'hash2', - }, - ], - }); - expect(r.cachedPartOfProjectGraph).toEqual({ - nodes: {}, - dependencies: {}, - }); - }); - - it('should handle cases when implicits change', () => { - const cached = { - nodes: { - mylib: { - name: 'mylib', - type: 'lib', - data: { - files: [ - { - file: 'index.ts', - ext: 'ts', - hash: 'hash1', - }, - ], - implicitDependencies: ['otherlib'], - }, - }, - }, - dependencies: { - mylib: [{ type: 'static', source: 'mylib', target: 'otherlib' }], - }, - } as any; - const r = extractCachedPartOfProjectGraph( - { - mylib: [ - { - file: 'index.ts', + file: 'index1.ts', ext: 'ts', hash: 'hash1', }, + { + file: 'index2.ts', + ext: 'ts', + hash: 'hash2b', + }, + { + file: 'index4.ts', + ext: 'ts', + hash: 'hash4', + }, ], }, - createNxJson({ - projects: { - mylib: { - implicitDependencies: [], - }, - }, - }), createCache({ nodes: { ...cached.nodes }, dependencies: { ...cached.dependencies }, }) ); - expect(r.filesDifferentFromCache).toEqual({ + expect(r.filesToProcess).toEqual({ mylib: [ { - file: 'index.ts', + file: 'index2.ts', ext: 'ts', - hash: 'hash1', + hash: 'hash2b', + }, + { + file: 'index4.ts', + ext: 'ts', + hash: 'hash4', }, ], }); - expect(r.cachedPartOfProjectGraph).toEqual({ - nodes: {}, - dependencies: {}, + expect(r.cachedFileData).toEqual({ + mylib: { + 'index1.ts': { + file: 'index1.ts', + ext: 'ts', + hash: 'hash1', + }, + }, }); }); }); @@ -307,7 +294,7 @@ describe('nx deps utils', () => { }, nxJsonPlugins: [{ name: 'plugin', version: '1.0.0' }], nodes: { - mylib: {} as any, + mylib: { type: 'lib' } as any, }, dependencies: { mylib: [] }, }; diff --git a/packages/workspace/src/core/nx-deps/nx-deps-cache.ts b/packages/workspace/src/core/nx-deps/nx-deps-cache.ts index 187f73b7f2..02047d7e57 100644 --- a/packages/workspace/src/core/nx-deps/nx-deps-cache.ts +++ b/packages/workspace/src/core/nx-deps/nx-deps-cache.ts @@ -1,5 +1,5 @@ -import { FileData, filesChanged } from '../file-utils'; import type { + FileData, NxJsonConfiguration, ProjectGraph, ProjectGraphDependency, @@ -29,6 +29,9 @@ export interface ProjectGraphCache { pathMappings: Record; nxJsonPlugins: { name: string; version: string }[]; nodes: Record; + + // this is only used by scripts that read dependency from the file + // in the sync fashion. dependencies: Record; } @@ -116,8 +119,7 @@ export function shouldRecomputeWholeGraph( if ( Object.keys(cache.nodes).some( (p) => - cache.nodes[p].type != 'app' && - cache.nodes[p].type != 'lib' && + (cache.nodes[p].type === 'app' || cache.nodes[p].type === 'lib') && !workspaceJson.projects[p] ) ) { @@ -157,46 +159,58 @@ This can only be invoked when the list of projects is either the same or new projects have been added, so every project in the cache has a corresponding project in fileMap */ -export function extractCachedPartOfProjectGraph( +export function extractCachedFileData( fileMap: ProjectFileMap, - nxJson: NxJsonConfiguration, c: ProjectGraphCache ): { - filesDifferentFromCache: ProjectFileMap; - cachedPartOfProjectGraph: ProjectGraph; + filesToProcess: ProjectFileMap; + cachedFileData: { [project: string]: { [file: string]: FileData } }; } { + const filesToProcess: ProjectFileMap = {}; const currentProjects = Object.keys(fileMap).filter( (name) => fileMap[name].length > 0 ); - - const filesDifferentFromCache: ProjectFileMap = {}; - // Re-compute nodes and dependencies for projects whose files changed + const cachedFileData = {}; currentProjects.forEach((p) => { - if (!c.nodes[p] || filesChanged(c.nodes[p].data.files, fileMap[p])) { - filesDifferentFromCache[p] = fileMap[p]; - delete c.dependencies[p]; - delete c.nodes[p]; - } - }); - - // Re-compute nodes and dependencies for projects whose implicit deps changed - Object.keys(nxJson.projects || {}).forEach((p) => { - if ( - nxJson.projects[p]?.implicitDependencies && - JSON.stringify(c.nodes[p].data.implicitDependencies) !== - JSON.stringify(nxJson.projects[p].implicitDependencies) - ) { - filesDifferentFromCache[p] = fileMap[p]; - delete c.dependencies[p]; - delete c.nodes[p]; - } + processProjectNode(p, c.nodes[p], cachedFileData, filesToProcess, fileMap); }); return { - filesDifferentFromCache, - cachedPartOfProjectGraph: { - nodes: c.nodes, - dependencies: c.dependencies, - }, + filesToProcess, + cachedFileData, }; } + +function processProjectNode( + name: string, + cachedNode: ProjectGraphNode, + cachedFileData: { [project: string]: { [file: string]: FileData } }, + filesToProcess: ProjectFileMap, + fileMap: ProjectFileMap +) { + if (!cachedNode) { + filesToProcess[name] = fileMap[name]; + return; + } + + const fileDataFromCache = {} as any; + for (let f of cachedNode.data.files) { + fileDataFromCache[f.file] = f; + } + + if (!cachedFileData[name]) { + cachedFileData[name] = {}; + } + + for (let f of fileMap[name]) { + const fromCache = fileDataFromCache[f.file]; + if (fromCache && fromCache.hash == f.hash) { + cachedFileData[name][f.file] = fromCache; + } else { + if (!filesToProcess[cachedNode.name]) { + filesToProcess[cachedNode.name] = []; + } + filesToProcess[cachedNode.name].push(f); + } + } +} diff --git a/packages/workspace/src/core/project-graph/build-dependencies/build-dependencies.ts b/packages/workspace/src/core/project-graph/build-dependencies/build-dependencies.ts deleted file mode 100644 index 62961502d1..0000000000 --- a/packages/workspace/src/core/project-graph/build-dependencies/build-dependencies.ts +++ /dev/null @@ -1,13 +0,0 @@ -import { - AddProjectDependency, - ProjectGraphContext, - ProjectGraphNodeRecords, -} from '../project-graph-models'; - -export interface BuildDependencies { - ( - ctx: ProjectGraphContext, - nodes: ProjectGraphNodeRecords, - addDependency: AddProjectDependency - ): void; -} diff --git a/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.spec.ts b/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.spec.ts index f54e1028be..c04b559079 100644 --- a/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.spec.ts +++ b/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.spec.ts @@ -1,13 +1,12 @@ import { buildExplicitPackageJsonDependencies } from '@nrwl/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies'; import { vol } from 'memfs'; -import { - AddProjectDependency, - DependencyType, - ProjectGraphContext, - ProjectGraphNode, -} from '../project-graph-models'; +import { DependencyType, ProjectGraphNode } from '../project-graph-models'; import { createProjectFileMap } from '../../file-graph'; import { readWorkspaceFiles } from '../../file-utils'; +import { + ProjectGraphBuilder, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; jest.mock('fs', () => require('memfs').fs); jest.mock('@nrwl/tao/src/utils/app-root', () => ({ @@ -15,7 +14,7 @@ jest.mock('@nrwl/tao/src/utils/app-root', () => ({ })); describe('explicit package json dependencies', () => { - let ctx: ProjectGraphContext; + let ctx: ProjectGraphProcessorContext; let projects: Record; let fsJson; beforeEach(() => { @@ -60,10 +59,12 @@ describe('explicit package json dependencies', () => { vol.fromJSON(fsJson, '/root'); ctx = { - workspaceJson, - nxJson, - fileMap: createProjectFileMap(workspaceJson, readWorkspaceFiles()), - }; + workspace: { + workspaceJson, + nxJson, + }, + filesToProcess: createProjectFileMap(workspaceJson, readWorkspaceFiles()), + } as any; projects = { proj: { @@ -88,27 +89,14 @@ describe('explicit package json dependencies', () => { }); it(`should add dependencies for projects based on deps in package.json`, () => { - const dependencyMap = {}; - const addDependency = jest - .fn, Parameters>() - .mockImplementation( - (type: DependencyType, source: string, target: string) => { - const depObj = { - type, - source, - target, - }; - if (dependencyMap[source]) { - dependencyMap[source].push(depObj); - } else { - dependencyMap[source] = [depObj]; - } - } - ); + const builder = new ProjectGraphBuilder(); + Object.values(projects).forEach((p) => { + builder.addNode(p); + }); - buildExplicitPackageJsonDependencies(ctx, projects, addDependency); + buildExplicitPackageJsonDependencies(ctx, builder); - expect(dependencyMap).toEqual({ + expect(builder.getUpdatedProjectGraph().dependencies).toEqual({ proj: [ { source: 'proj', @@ -121,6 +109,8 @@ describe('explicit package json dependencies', () => { type: DependencyType.static, }, ], + proj2: [], + proj3: [], }); }); }); diff --git a/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.ts b/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.ts index 09cbbfbe85..af91ec0fbb 100644 --- a/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.ts +++ b/packages/workspace/src/core/project-graph/build-dependencies/explicit-package-json-dependencies.ts @@ -1,21 +1,20 @@ -import { - AddProjectDependency, - DependencyType, - ProjectGraphContext, - ProjectGraphNodeRecords, -} from '../project-graph-models'; +import { ProjectGraphNodeRecords } from '../project-graph-models'; import { defaultFileRead } from '../../file-utils'; -import { joinPathFragments, parseJson } from '@nrwl/devkit'; +import { + joinPathFragments, + parseJson, + ProjectGraphBuilder, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; export function buildExplicitPackageJsonDependencies( - ctx: ProjectGraphContext, - nodes: ProjectGraphNodeRecords, - addDependency: AddProjectDependency + ctx: ProjectGraphProcessorContext, + builder: ProjectGraphBuilder ) { - Object.keys(ctx.fileMap).forEach((source) => { - Object.values(ctx.fileMap[source]).forEach((f) => { - if (isPackageJsonAtProjectRoot(nodes, f.file)) { - processPackageJson(source, f.file, nodes, addDependency); + Object.keys(ctx.filesToProcess).forEach((source) => { + Object.values(ctx.filesToProcess[source]).forEach((f) => { + if (isPackageJsonAtProjectRoot(builder.graph.nodes, f.file)) { + processPackageJson(source, f.file, builder); } }); }); @@ -35,15 +34,14 @@ function isPackageJsonAtProjectRoot( function processPackageJson( sourceProject: string, fileName: string, - nodes: ProjectGraphNodeRecords, - addDependency: AddProjectDependency + builder: ProjectGraphBuilder ) { try { const deps = readDeps(parseJson(defaultFileRead(fileName))); deps.forEach((d) => { // package.json refers to another project in the monorepo - if (nodes[d]) { - addDependency(DependencyType.static, sourceProject, d); + if (builder.graph.nodes[d]) { + builder.addExplicitDependency(sourceProject, fileName, d); } }); } catch (e) { diff --git a/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.spec.ts b/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.spec.ts index 7b4d2823ee..bfdaeedfac 100644 --- a/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.spec.ts +++ b/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.spec.ts @@ -4,18 +4,17 @@ jest.mock('@nrwl/tao/src/utils/app-root', () => ({ })); import { vol } from 'memfs'; -import { - AddProjectDependency, - ProjectGraphContext, - ProjectGraphNode, - DependencyType, -} from '../project-graph-models'; +import { ProjectGraphNode, DependencyType } from '../project-graph-models'; import { buildExplicitTypeScriptDependencies } from './explicit-project-dependencies'; import { createProjectFileMap } from '../../file-graph'; import { readWorkspaceFiles } from '../../file-utils'; +import { + ProjectGraphBuilder, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; describe('explicit project dependencies', () => { - let ctx: ProjectGraphContext; + let ctx: ProjectGraphProcessorContext; let projects: Record; let fsJson; beforeEach(() => { @@ -111,10 +110,12 @@ describe('explicit project dependencies', () => { vol.fromJSON(fsJson, '/root'); ctx = { - workspaceJson, - nxJson, - fileMap: createProjectFileMap(workspaceJson, readWorkspaceFiles()), - }; + workspace: { + ...workspaceJson, + ...nxJson, + } as any, + filesToProcess: createProjectFileMap(workspaceJson, readWorkspaceFiles()), + } as any; projects = { proj3a: { @@ -122,7 +123,7 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj3a', - files: [], + files: [{ file: 'libs/proj3a/index.ts' }], }, }, proj2: { @@ -130,7 +131,7 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj2', - files: [], + files: [{ file: 'libs/proj2/index.ts' }], }, }, proj: { @@ -138,7 +139,7 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj', - files: [], + files: [{ file: 'libs/proj/index.ts' }], }, }, proj1234: { @@ -146,7 +147,11 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj1234', - files: [], + files: [ + { file: 'libs/proj1234/index.ts' }, + { file: 'libs/proj1234/a.b.ts' }, + { file: 'libs/proj1234/b.c.ts' }, + ], }, }, proj123: { @@ -154,7 +159,7 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj123', - files: [], + files: [{ file: 'libs/proj123/index.ts' }], }, }, proj4ab: { @@ -162,7 +167,7 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj4ab', - files: [], + files: [{ file: 'libs/proj4ab/index.ts' }], }, }, 'proj1234-child': { @@ -170,34 +175,21 @@ describe('explicit project dependencies', () => { type: 'lib', data: { root: 'libs/proj1234-child', - files: [], + files: [{ file: 'libs/proj1234-child/index.ts' }], }, }, }; }); it(`should add dependencies for projects based on file imports`, () => { - const dependencyMap = {}; - const addDependency = jest - .fn, Parameters>() - .mockImplementation( - (type: DependencyType, source: string, target: string) => { - const depObj = { - type, - source, - target, - }; - if (dependencyMap[source]) { - dependencyMap[source].push(depObj); - } else { - dependencyMap[source] = [depObj]; - } - } - ); + const builder = new ProjectGraphBuilder(); + Object.values(projects).forEach((p) => { + builder.addNode(p); + }); - buildExplicitTypeScriptDependencies(ctx, projects, addDependency); + buildExplicitTypeScriptDependencies(ctx, builder); - expect(dependencyMap).toEqual({ + expect(builder.getUpdatedProjectGraph().dependencies).toEqual({ proj1234: [ { source: 'proj1234', @@ -214,14 +206,19 @@ describe('explicit project dependencies', () => { { source: 'proj', target: 'proj3a', - type: DependencyType.dynamic, + type: DependencyType.static, }, { source: 'proj', target: 'proj4ab', - type: DependencyType.dynamic, + type: DependencyType.static, }, ], + proj123: [], + 'proj1234-child': [], + proj2: [], + proj3a: [], + proj4ab: [], }); }); }); diff --git a/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.ts b/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.ts index f83e080154..f4e3bc364c 100644 --- a/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.ts +++ b/packages/workspace/src/core/project-graph/build-dependencies/explicit-project-dependencies.ts @@ -1,31 +1,29 @@ -import { - AddProjectDependency, - DependencyType, - ProjectGraphContext, - ProjectGraphNodeRecords, -} from '../project-graph-models'; +import { DependencyType } from '../project-graph-models'; import { TypeScriptImportLocator } from './typescript-import-locator'; import { TargetProjectLocator } from '../../target-project-locator'; +import { + ProjectGraphBuilder, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; export function buildExplicitTypeScriptDependencies( - ctx: ProjectGraphContext, - nodes: ProjectGraphNodeRecords, - addDependency: AddProjectDependency + ctx: ProjectGraphProcessorContext, + builder: ProjectGraphBuilder ) { const importLocator = new TypeScriptImportLocator(); - const targetProjectLocator = new TargetProjectLocator(nodes); - Object.keys(ctx.fileMap).forEach((source) => { - Object.values(ctx.fileMap[source]).forEach((f) => { + const targetProjectLocator = new TargetProjectLocator(builder.graph.nodes); + Object.keys(ctx.filesToProcess).forEach((source) => { + Object.values(ctx.filesToProcess[source]).forEach((f) => { importLocator.fromFile( f.file, (importExpr: string, filePath: string, type: DependencyType) => { const target = targetProjectLocator.findProjectWithImport( importExpr, f.file, - ctx.nxJson.npmScope + ctx.workspace.npmScope ); if (source && target) { - addDependency(type, source, target); + builder.addExplicitDependency(source, f.file, target); } } ); diff --git a/packages/workspace/src/core/project-graph/build-dependencies/implicit-project-dependencies.ts b/packages/workspace/src/core/project-graph/build-dependencies/implicit-project-dependencies.ts index 6f1209a7e6..3970703641 100644 --- a/packages/workspace/src/core/project-graph/build-dependencies/implicit-project-dependencies.ts +++ b/packages/workspace/src/core/project-graph/build-dependencies/implicit-project-dependencies.ts @@ -1,20 +1,17 @@ import { - AddProjectDependency, - DependencyType, - ProjectGraphContext, - ProjectGraphNodeRecords, -} from '../project-graph-models'; + ProjectGraphBuilder, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; export function buildImplicitProjectDependencies( - ctx: ProjectGraphContext, - nodes: ProjectGraphNodeRecords, - addDependency: AddProjectDependency + ctx: ProjectGraphProcessorContext, + builder: ProjectGraphBuilder ) { - Object.keys(ctx.nxJson.projects).forEach((source) => { - const p = ctx.nxJson.projects[source]; + Object.keys(ctx.workspace.projects).forEach((source) => { + const p = ctx.workspace.projects[source]; if (p.implicitDependencies && p.implicitDependencies.length > 0) { p.implicitDependencies.forEach((target) => { - addDependency(DependencyType.implicit, source, target); + builder.addImplicitDependency(source, target); }); } }); diff --git a/packages/workspace/src/core/project-graph/build-dependencies/index.ts b/packages/workspace/src/core/project-graph/build-dependencies/index.ts index b93a91df31..eec030f38c 100644 --- a/packages/workspace/src/core/project-graph/build-dependencies/index.ts +++ b/packages/workspace/src/core/project-graph/build-dependencies/index.ts @@ -1,4 +1,3 @@ -export * from './build-dependencies'; export * from './implicit-project-dependencies'; export * from './explicit-project-dependencies'; export * from './explicit-package-json-dependencies'; diff --git a/packages/workspace/src/core/project-graph/build-nodes/build-nodes.ts b/packages/workspace/src/core/project-graph/build-nodes/build-nodes.ts deleted file mode 100644 index 2e5cbd2467..0000000000 --- a/packages/workspace/src/core/project-graph/build-nodes/build-nodes.ts +++ /dev/null @@ -1,5 +0,0 @@ -import { AddProjectNode, ProjectGraphContext } from '../project-graph-models'; - -export interface BuildNodes { - (ctx: ProjectGraphContext, addNode: AddProjectNode): void; -} diff --git a/packages/workspace/src/core/project-graph/build-nodes/index.ts b/packages/workspace/src/core/project-graph/build-nodes/index.ts index e8e4a7f353..faff1905e7 100644 --- a/packages/workspace/src/core/project-graph/build-nodes/index.ts +++ b/packages/workspace/src/core/project-graph/build-nodes/index.ts @@ -1,3 +1,2 @@ -export * from './build-nodes'; export * from './workspace-projects'; export * from './npm-packages'; diff --git a/packages/workspace/src/core/project-graph/build-nodes/npm-packages.ts b/packages/workspace/src/core/project-graph/build-nodes/npm-packages.ts index 8395cf2f5e..5b07a2fce7 100644 --- a/packages/workspace/src/core/project-graph/build-nodes/npm-packages.ts +++ b/packages/workspace/src/core/project-graph/build-nodes/npm-packages.ts @@ -1,19 +1,15 @@ -import { AddProjectNode, ProjectGraphContext } from '../project-graph-models'; -import { readJsonFile } from '@nrwl/devkit'; +import { ProjectGraphBuilder, readJsonFile } from '@nrwl/devkit'; import { join } from 'path'; import { appRootPath } from '@nrwl/tao/src/utils/app-root'; -export function buildNpmPackageNodes( - ctx: ProjectGraphContext, - addNode: AddProjectNode -) { +export function buildNpmPackageNodes(builder: ProjectGraphBuilder) { const packageJson = readJsonFile(join(appRootPath, 'package.json')); const deps = { ...packageJson.dependencies, ...packageJson.devDependencies, }; Object.keys(deps).forEach((d) => { - addNode({ + builder.addNode({ type: 'npm', name: `npm:${d}`, data: { diff --git a/packages/workspace/src/core/project-graph/build-nodes/workspace-projects.ts b/packages/workspace/src/core/project-graph/build-nodes/workspace-projects.ts index 07660417d9..66867a9459 100644 --- a/packages/workspace/src/core/project-graph/build-nodes/workspace-projects.ts +++ b/packages/workspace/src/core/project-graph/build-nodes/workspace-projects.ts @@ -1,5 +1,8 @@ -import { AddProjectNode, ProjectGraphContext } from '../project-graph-models'; import { defaultFileRead } from '../../file-utils'; +import { + ProjectGraphBuilder, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; export function convertNpmScriptsToTargets(projectRoot: string) { try { @@ -24,18 +27,15 @@ export function convertNpmScriptsToTargets(projectRoot: string) { } export function buildWorkspaceProjectNodes( - ctx: ProjectGraphContext, - addNode: AddProjectNode + ctx: ProjectGraphProcessorContext, + builder: ProjectGraphBuilder ) { const toAdd = []; - - Object.keys(ctx.fileMap).forEach((key) => { - const p = ctx.workspaceJson.projects[key]; + Object.keys(ctx.workspace.projects).forEach((key) => { + const p = ctx.workspace.projects[key]; if (!p.targets) { p.targets = convertNpmScriptsToTargets(p.root); } - - // TODO, types and projectType should allign const projectType = p.projectType === 'application' ? key.endsWith('-e2e') @@ -43,8 +43,8 @@ export function buildWorkspaceProjectNodes( : 'app' : 'lib'; const tags = - ctx.nxJson.projects && ctx.nxJson.projects[key] - ? ctx.nxJson.projects[key].tags || [] + ctx.workspace.projects && ctx.workspace.projects[key] + ? ctx.workspace.projects[key].tags || [] : []; toAdd.push({ @@ -66,7 +66,7 @@ export function buildWorkspaceProjectNodes( }); toAdd.forEach((n) => { - addNode({ + builder.addNode({ name: n.name, type: n.type, data: n.data, diff --git a/packages/workspace/src/core/project-graph/index.ts b/packages/workspace/src/core/project-graph/index.ts index 28d9346f3b..4003cf7270 100644 --- a/packages/workspace/src/core/project-graph/index.ts +++ b/packages/workspace/src/core/project-graph/index.ts @@ -3,8 +3,5 @@ export { createProjectGraphAsync, readCurrentProjectGraph, } from './project-graph'; -export { BuildDependencies } from './build-dependencies'; -export { BuildNodes } from './build-nodes'; -export { ProjectGraphBuilder } from './project-graph-builder'; export * from './project-graph-models'; export * from './operators'; diff --git a/packages/workspace/src/core/project-graph/operators.ts b/packages/workspace/src/core/project-graph/operators.ts index d82f30907a..b85ee5af43 100644 --- a/packages/workspace/src/core/project-graph/operators.ts +++ b/packages/workspace/src/core/project-graph/operators.ts @@ -1,24 +1,24 @@ -import { ProjectGraphBuilder } from './project-graph-builder'; -import type { ProjectGraph, ProjectGraphNode } from '@nrwl/devkit'; +import { ProjectGraph, ProjectGraphNode } from '@nrwl/devkit'; const reverseMemo = new Map(); export function reverse(graph: ProjectGraph): ProjectGraph { - let result = reverseMemo.get(graph); - if (!result) { - const builder = new ProjectGraphBuilder(); - Object.values(graph.nodes).forEach((n) => { - builder.addNode(n); - }); - Object.values(graph.dependencies).forEach((byProject) => { - byProject.forEach((dep) => { - builder.addDependency(dep.type, dep.target, dep.source); + const resultFromMemo = reverseMemo.get(graph); + if (resultFromMemo) return resultFromMemo; + + const result = { nodes: graph.nodes, dependencies: {} } as ProjectGraph; + Object.keys(graph.nodes).forEach((n) => (result.dependencies[n] = [])); + Object.values(graph.dependencies).forEach((byProject) => { + byProject.forEach((dep) => { + result.dependencies[dep.target].push({ + type: dep.type, + source: dep.target, + target: dep.source, }); }); - result = builder.build(); - reverseMemo.set(graph, result); - reverseMemo.set(result, graph); - } + }); + reverseMemo.set(graph, result); + reverseMemo.set(result, graph); return result; } @@ -26,22 +26,23 @@ export function filterNodes( predicate: (n: ProjectGraphNode) => boolean ): (p: ProjectGraph) => ProjectGraph { return (original) => { - const builder = new ProjectGraphBuilder(); + const graph = { nodes: {}, dependencies: {} } as ProjectGraph; const added = new Set(); Object.values(original.nodes).forEach((n) => { if (predicate(n)) { - builder.addNode(n); + graph.nodes[n.name] = n; + graph.dependencies[n.name] = []; added.add(n.name); } }); Object.values(original.dependencies).forEach((ds) => { ds.forEach((d) => { if (added.has(d.source) && added.has(d.target)) { - builder.addDependency(d.type, d.source, d.target); + graph.dependencies[d.source].push(d); } }); }); - return builder.build(); + return graph; }; } @@ -81,18 +82,21 @@ export function withDeps( original: ProjectGraph, subsetNodes: ProjectGraphNode[] ): ProjectGraph { - const builder = new ProjectGraphBuilder(); + const res = { nodes: {}, dependencies: {} } as ProjectGraph; const visitedNodes = []; const visitedEdges = []; Object.values(subsetNodes).forEach(recurNodes); Object.values(subsetNodes).forEach(recurEdges); - return builder.build(); + return res; // --------------------------------------------------------------------------- function recurNodes(node) { if (visitedNodes.indexOf(node.name) > -1) return; - builder.addNode(node); + res.nodes[node.name] = node; + if (!res.dependencies[node.name]) { + res.dependencies[node.name] = []; + } visitedNodes.push(node.name); original.dependencies[node.name].forEach((n) => { @@ -106,7 +110,10 @@ export function withDeps( const ds = original.dependencies[node.name]; ds.forEach((n) => { - builder.addDependency(n.type, n.source, n.target); + if (!res.dependencies[n.source]) { + res.dependencies[n.source] = []; + } + res.dependencies[n.source].push(n); }); ds.forEach((n) => { diff --git a/packages/workspace/src/core/project-graph/project-graph-builder.spec.ts b/packages/workspace/src/core/project-graph/project-graph-builder.spec.ts deleted file mode 100644 index 44191e5806..0000000000 --- a/packages/workspace/src/core/project-graph/project-graph-builder.spec.ts +++ /dev/null @@ -1,85 +0,0 @@ -import { ProjectGraphBuilder } from './project-graph-builder'; -import { DependencyType, ProjectGraphNode } from './project-graph-models'; - -describe('ProjectGraphBuilder', () => { - it('should generate graph with nodes and dependencies', () => { - const builder = new ProjectGraphBuilder(); - const myapp = createNode('myapp', 'app'); - const libA = createNode('lib-a', 'lib'); - const libB = createNode('lib-b', 'lib'); - const libC = createNode('lib-c', 'lib'); - const happyNrwl = createNode('happy-nrwl', 'npm'); - builder.addNode(myapp); - builder.addNode(libA); - builder.addNode(libB); - builder.addNode(libC); - builder.addNode(libC); // Idempotency - builder.addNode(happyNrwl); - - expect(() => { - builder.addDependency(DependencyType.static, 'fake-1', 'fake-2'); - }).toThrow(); - - builder.addDependency(DependencyType.static, myapp.name, libA.name); - builder.addDependency(DependencyType.static, myapp.name, libB.name); - builder.addDependency(DependencyType.static, libB.name, libC.name); - builder.addDependency(DependencyType.static, libB.name, libC.name); // Idempotency - builder.addDependency(DependencyType.static, libB.name, libB.name); - builder.addDependency(DependencyType.static, libC.name, happyNrwl.name); - - const graph = builder.build(); - - expect(graph).toMatchObject({ - nodes: { - [myapp.name]: myapp, - [libA.name]: libA, - [libB.name]: libB, - [libC.name]: libC, - [happyNrwl.name]: happyNrwl, - }, - dependencies: { - [myapp.name]: [ - { - type: DependencyType.static, - source: myapp.name, - target: libA.name, - }, - { - type: DependencyType.static, - source: myapp.name, - target: libB.name, - }, - ], - [libB.name]: [ - { type: DependencyType.static, source: libB.name, target: libC.name }, - ], - [libC.name]: [ - { - type: DependencyType.static, - source: libC.name, - target: happyNrwl.name, - }, - ], - }, - }); - }); - - it('should throw an error when there are projects with conflicting names', () => { - const builder = new ProjectGraphBuilder(); - const projA = createNode('proj', 'app'); - const projB = createNode('proj', 'lib'); - builder.addNode(projA); - - expect(() => { - builder.addNode(projB); - }).toThrow(); - }); -}); - -function createNode(name: string, type: string): ProjectGraphNode { - return { - type, - name, - data: null, - }; -} diff --git a/packages/workspace/src/core/project-graph/project-graph-builder.ts b/packages/workspace/src/core/project-graph/project-graph-builder.ts deleted file mode 100644 index e9e7e6fc23..0000000000 --- a/packages/workspace/src/core/project-graph/project-graph-builder.ts +++ /dev/null @@ -1,7 +0,0 @@ -import { ProjectGraphBuilder as DevkitProjectGraphBuilder } from '@nrwl/devkit'; - -export class ProjectGraphBuilder extends DevkitProjectGraphBuilder { - build() { - return super.getProjectGraph(); - } -} diff --git a/packages/workspace/src/core/project-graph/project-graph-models.ts b/packages/workspace/src/core/project-graph/project-graph-models.ts index 0a2ddf1bdf..69d43a1569 100644 --- a/packages/workspace/src/core/project-graph/project-graph-models.ts +++ b/packages/workspace/src/core/project-graph/project-graph-models.ts @@ -1,9 +1,5 @@ -import type { ProjectFileMap } from '../file-graph'; -import type { - ProjectGraphNode, - DependencyType, - NxJsonConfiguration, -} from '@nrwl/devkit'; +import type { DependencyType, ProjectGraphNode } from '@nrwl/devkit'; + export { ProjectGraph, ProjectGraphDependency, @@ -13,20 +9,6 @@ export { export type ProjectGraphNodeRecords = Record; -export type AddProjectNode = (node: ProjectGraphNode) => void; - -export type AddProjectDependency = ( - type: DependencyType | string, - source: string, - target: string -) => void; - -export interface ProjectGraphContext { - workspaceJson: any; - nxJson: NxJsonConfiguration; - fileMap: ProjectFileMap; -} - export enum ProjectType { app = 'app', e2e = 'e2e', diff --git a/packages/workspace/src/core/project-graph/project-graph.spec.ts b/packages/workspace/src/core/project-graph/project-graph.spec.ts index cbaf19025a..779b886eda 100644 --- a/packages/workspace/src/core/project-graph/project-graph.spec.ts +++ b/packages/workspace/src/core/project-graph/project-graph.spec.ts @@ -1,4 +1,5 @@ import { vol, fs } from 'memfs'; + jest.mock('fs', () => require('memfs').fs); jest.mock('@nrwl/tao/src/utils/app-root', () => ({ appRootPath: '/root', @@ -158,34 +159,37 @@ describe('project graph', () => { }, }, }); - expect(graph.dependencies).toMatchObject({ - api: [ - { type: DependencyType.static, source: 'api', target: 'npm:express' }, - ], - 'demo-e2e': [], + expect(graph.dependencies).toEqual({ + api: [{ source: 'api', target: 'npm:express', type: 'static' }], demo: [ - { type: DependencyType.static, source: 'demo', target: 'ui' }, + { source: 'demo', target: 'api', type: 'implicit' }, { - type: DependencyType.static, source: 'demo', - target: 'shared-util-data', + target: 'ui', + type: 'static', }, + { source: 'demo', target: 'shared-util-data', type: 'static' }, { - type: DependencyType.dynamic, source: 'demo', target: 'lazy-lib', + type: 'static', }, - { type: DependencyType.implicit, source: 'demo', target: 'api' }, - ], - ui: [ - { type: DependencyType.static, source: 'ui', target: 'shared-util' }, - { type: DependencyType.dynamic, source: 'ui', target: 'lazy-lib' }, ], + 'demo-e2e': [], + 'lazy-lib': [], + 'npm:@nrwl/workspace': [], + 'npm:express': [], + 'npm:happy-nrwl': [], 'shared-util': [ + { source: 'shared-util', target: 'npm:happy-nrwl', type: 'static' }, + ], + 'shared-util-data': [], + ui: [ + { source: 'ui', target: 'shared-util', type: 'static' }, { - type: DependencyType.static, - source: 'shared-util', - target: 'npm:happy-nrwl', + source: 'ui', + target: 'lazy-lib', + type: 'static', }, ], }); @@ -229,7 +233,7 @@ describe('project graph', () => { target: 'shared-util', }, { - type: DependencyType.dynamic, + type: DependencyType.static, source: 'ui', target: 'lazy-lib', }, diff --git a/packages/workspace/src/core/project-graph/project-graph.ts b/packages/workspace/src/core/project-graph/project-graph.ts index 69a644731f..c5605ff4e0 100644 --- a/packages/workspace/src/core/project-graph/project-graph.ts +++ b/packages/workspace/src/core/project-graph/project-graph.ts @@ -22,19 +22,17 @@ import { } from '../file-utils'; import { normalizeNxJson } from '../normalize-nx-json'; import { - extractCachedPartOfProjectGraph, + extractCachedFileData, readCache, shouldRecomputeWholeGraph, writeCache, } from '../nx-deps/nx-deps-cache'; import { - BuildDependencies, buildExplicitPackageJsonDependencies, buildExplicitTypeScriptDependencies, buildImplicitProjectDependencies, } from './build-dependencies'; import { - BuildNodes, buildNpmPackageNodes, buildWorkspaceProjectNodes, } from './build-nodes'; @@ -64,6 +62,9 @@ export function createProjectGraph( const projectFileMap = createProjectFileMap(workspaceJson, workspaceFiles); const packageJsonDeps = readCombinedDeps(); const rootTsConfig = readRootTsConfig(); + + let filesToProcess = projectFileMap; + let cachedFileData = {}; if ( cache && cache.version === '3.0' && @@ -73,31 +74,24 @@ export function createProjectGraph( workspaceJson, normalizedNxJson, rootTsConfig - ) + ) && + cacheEnabled ) { - const diff = extractCachedPartOfProjectGraph(projectFileMap, nxJson, cache); - const ctx = { - workspaceJson, - nxJson: normalizedNxJson, - fileMap: diff.filesDifferentFromCache, - }; - const projectGraph = buildProjectGraph(ctx, diff.cachedPartOfProjectGraph); - if (cacheEnabled) { - writeCache(packageJsonDeps, nxJson, rootTsConfig, projectGraph); - } - return addWorkspaceFiles(projectGraph, workspaceFiles); - } else { - const ctx = { - workspaceJson, - nxJson: normalizedNxJson, - fileMap: projectFileMap, - }; - const projectGraph = buildProjectGraph(ctx, null); - if (cacheEnabled) { - writeCache(packageJsonDeps, nxJson, rootTsConfig, projectGraph); - } - return addWorkspaceFiles(projectGraph, workspaceFiles); + const fromCache = extractCachedFileData(projectFileMap, cache); + filesToProcess = fromCache.filesToProcess; + cachedFileData = fromCache.cachedFileData; } + const context = createContext( + workspaceJson, + normalizedNxJson, + projectFileMap, + filesToProcess + ); + const projectGraph = buildProjectGraph(context, cachedFileData); + if (cacheEnabled) { + writeCache(packageJsonDeps, nxJson, rootTsConfig, projectGraph); + } + return addWorkspaceFiles(projectGraph, workspaceFiles); } export function readCurrentProjectGraph(): ProjectGraph | null { @@ -112,24 +106,29 @@ function addWorkspaceFiles( return { ...projectGraph, allWorkspaceFiles }; } -type BuilderContext = { - nxJson: NxJsonConfiguration; - workspaceJson: WorkspaceJsonConfiguration; - fileMap: ProjectFileMap; -}; - -function buildProjectGraph(ctx: BuilderContext, projectGraph: ProjectGraph) { +function buildProjectGraph( + ctx: ProjectGraphProcessorContext, + cachedFileData: { [project: string]: { [file: string]: FileData } } +) { performance.mark('build project graph:start'); - const builder = new ProjectGraphBuilder(projectGraph); - const addNode = builder.addNode.bind(builder); - const addDependency = builder.addDependency.bind(builder); - buildWorkspaceProjectNodes(ctx, addNode); - buildNpmPackageNodes(ctx, addNode); - buildExplicitTypeScriptDependencies(ctx, builder.nodes, addDependency); - buildExplicitPackageJsonDependencies(ctx, builder.nodes, addDependency); - buildImplicitProjectDependencies(ctx, builder.nodes, addDependency); - const initProjectGraph = builder.getProjectGraph(); + const builder = new ProjectGraphBuilder(); + + buildWorkspaceProjectNodes(ctx, builder); + buildNpmPackageNodes(builder); + for (const proj of Object.keys(cachedFileData)) { + for (const f of builder.graph.nodes[proj].data.files) { + const cached = cachedFileData[proj][f.file]; + if (cached) { + f.deps = cached.deps; + } + } + } + + buildExplicitTypeScriptDependencies(ctx, builder); + buildExplicitPackageJsonDependencies(ctx, builder); + buildImplicitProjectDependencies(ctx, builder); + const initProjectGraph = builder.getUpdatedProjectGraph(); const r = updateProjectGraphWithPlugins(ctx, initProjectGraph); @@ -143,35 +142,43 @@ function buildProjectGraph(ctx: BuilderContext, projectGraph: ProjectGraph) { return r; } -function updateProjectGraphWithPlugins( - ctx: BuilderContext, - initProjectGraph: ProjectGraph -) { - const plugins = (ctx.nxJson.plugins || []).map((path) => { - const pluginPath = require.resolve(path, { - paths: [appRootPath], - }); - return require(pluginPath) as NxPlugin; - }); - - const projects = Object.keys(ctx.workspaceJson.projects).reduce( +function createContext( + workspaceJson: WorkspaceJsonConfiguration, + nxJson: NxJsonConfiguration, + fileMap: ProjectFileMap, + filesToProcess: ProjectFileMap +): ProjectGraphProcessorContext { + const projects = Object.keys(workspaceJson.projects).reduce( (map, projectName) => { map[projectName] = { - ...ctx.workspaceJson.projects[projectName], - ...ctx.nxJson.projects[projectName], + ...workspaceJson.projects[projectName], + ...nxJson.projects[projectName], }; return map; }, {} as Record ); - const context: ProjectGraphProcessorContext = { + return { workspace: { - ...ctx.workspaceJson, - ...ctx.nxJson, + ...workspaceJson, + ...nxJson, projects, }, - fileMap: ctx.fileMap, + fileMap, + filesToProcess, }; +} + +function updateProjectGraphWithPlugins( + context: ProjectGraphProcessorContext, + initProjectGraph: ProjectGraph +) { + const plugins = (context.workspace.plugins || []).map((path) => { + const pluginPath = require.resolve(path, { + paths: [appRootPath], + }); + return require(pluginPath) as NxPlugin; + }); return plugins.reduce((graph, plugin) => { if (!plugin.processProjectGraph) { diff --git a/packages/workspace/src/core/target-project-locator.spec.ts b/packages/workspace/src/core/target-project-locator.spec.ts index b45a502f49..2a5cdc300e 100644 --- a/packages/workspace/src/core/target-project-locator.spec.ts +++ b/packages/workspace/src/core/target-project-locator.spec.ts @@ -1,6 +1,8 @@ import { vol } from 'memfs'; -import { ProjectGraphContext } from './project-graph'; -import type { ProjectGraphNode } from '@nrwl/devkit'; +import type { + ProjectGraphNode, + ProjectGraphProcessorContext, +} from '@nrwl/devkit'; import { TargetProjectLocator } from './target-project-locator'; jest.mock('@nrwl/tao/src/utils/app-root', () => ({ @@ -9,7 +11,7 @@ jest.mock('@nrwl/tao/src/utils/app-root', () => ({ jest.mock('fs', () => require('memfs').fs); describe('findTargetProjectWithImport', () => { - let ctx: ProjectGraphContext; + let ctx: ProjectGraphProcessorContext; let projects: Record; let fsJson; let targetProjectLocator: TargetProjectLocator; @@ -56,8 +58,10 @@ describe('findTargetProjectWithImport', () => { vol.fromJSON(fsJson, '/root'); ctx = { - workspaceJson, - nxJson, + workspace: { + ...workspaceJson, + ...nxJson, + } as any, fileMap: { proj: [ { @@ -109,7 +113,7 @@ describe('findTargetProjectWithImport', () => { }, ], }, - }; + } as any; projects = { proj3a: { @@ -209,22 +213,22 @@ describe('findTargetProjectWithImport', () => { const res1 = targetProjectLocator.findProjectWithImport( './class.ts', 'libs/proj/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); const res2 = targetProjectLocator.findProjectWithImport( '../index.ts', 'libs/proj/src/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); const res3 = targetProjectLocator.findProjectWithImport( '../proj/../proj2/index.ts', 'libs/proj/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); const res4 = targetProjectLocator.findProjectWithImport( '../proj/../index.ts', 'libs/proj/src/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(res1).toEqual('proj'); @@ -237,12 +241,12 @@ describe('findTargetProjectWithImport', () => { const proj2 = targetProjectLocator.findProjectWithImport( '@proj/my-second-proj', 'libs/proj1/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); const proj3a = targetProjectLocator.findProjectWithImport( '@proj/project-3', 'libs/proj1/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(proj2).toEqual('proj2'); @@ -253,12 +257,12 @@ describe('findTargetProjectWithImport', () => { const result1 = targetProjectLocator.findProjectWithImport( '@ng/core', 'libs/proj1/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); const result2 = targetProjectLocator.findProjectWithImport( 'npm-package', 'libs/proj1/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(result1).toEqual('npm:@ng/core'); @@ -269,7 +273,7 @@ describe('findTargetProjectWithImport', () => { const proj4ab = targetProjectLocator.findProjectWithImport( '@proj/proj4ab', 'libs/proj1/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(proj4ab).toEqual('proj4ab'); @@ -278,21 +282,21 @@ describe('findTargetProjectWithImport', () => { const proj = targetProjectLocator.findProjectWithImport( '@proj/proj123', '', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(proj).toEqual('proj123'); const childProj = targetProjectLocator.findProjectWithImport( '@proj/proj1234-child', '', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(childProj).toEqual('proj1234-child'); const parentProj = targetProjectLocator.findProjectWithImport( '@proj/proj1234', '', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(parentProj).toEqual('proj1234'); }); @@ -301,14 +305,14 @@ describe('findTargetProjectWithImport', () => { const similarImportFromNpm = targetProjectLocator.findProjectWithImport( '@proj/proj123-base', 'libs/proj/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(similarImportFromNpm).toEqual('npm:@proj/proj123-base'); const similarDeepImportFromNpm = targetProjectLocator.findProjectWithImport( '@proj/proj123-base/deep', 'libs/proj/index.ts', - ctx.nxJson.npmScope + ctx.workspace.npmScope ); expect(similarDeepImportFromNpm).toEqual('npm:@proj/proj123-base'); }); diff --git a/packages/workspace/src/generators/convert-to-nx-project/convert-to-nx-project.ts b/packages/workspace/src/generators/convert-to-nx-project/convert-to-nx-project.ts index 9d5d8ac49d..da3601306a 100644 --- a/packages/workspace/src/generators/convert-to-nx-project/convert-to-nx-project.ts +++ b/packages/workspace/src/generators/convert-to-nx-project/convert-to-nx-project.ts @@ -19,7 +19,6 @@ import { import { Schema } from './schema'; import { getProjectConfigurationPath } from './utils/get-project-configuration-path'; -import { workspaceConfigName } from '@nrwl/tao/src/shared/workspace'; export const SCHEMA_OPTIONS_ARE_MUTUALLY_EXCLUSIVE = '--project and --all are mutually exclusive';