Table of contents
tldr;
This section will go over how you can combine NVTX filtering, targeted metric selection, and kernel selection to reduce ncu profiling overhead on vLLM from 18+ hours to under 1 second — while still extracting roofline metrics for specific kernels, without isolating or reproducing them.
What causes ncu’s high profiling overheads
The overheads of using ncu on a large code base like vLLM directly comes from 3 sources:
- The sheer number of kernels profiled.
- The number of execution replay passes that are run per kernel
- Cost of each replay
This post will go over how we can use various flags that ncu exposes to reduce each of these costs.m
What is execution replay
When ncu is profiling a workload it creates sub-group…
Table of contents
tldr;
This section will go over how you can combine NVTX filtering, targeted metric selection, and kernel selection to reduce ncu profiling overhead on vLLM from 18+ hours to under 1 second — while still extracting roofline metrics for specific kernels, without isolating or reproducing them.
What causes ncu’s high profiling overheads
The overheads of using ncu on a large code base like vLLM directly comes from 3 sources:
- The sheer number of kernels profiled.
- The number of execution replay passes that are run per kernel
- Cost of each replay
This post will go over how we can use various flags that ncu exposes to reduce each of these costs.m
What is execution replay
When ncu is profiling a workload it creates sub-groups of the overall set of metrics that you request and depending on the number of groups created ncu replays either a single kernel, a group of kernels or the entire application for each group of metrics created.
When is a replay triggered
Replay grouping is necessary as there are a limited number of profiling registers available to collect runtime data. Requesting a large number of metrics causes multiple replays to occur as each replay collects a different subset of the metrics requested in the available registers.
What happens during a replay
If ncu needs to trigger a replay, the following steps occur. On the first pass of the kernel, ncu saves all GPU memory. After the first pass, it detects the regions of memory that were updated by the kernel and on subsequent replays, only stores this subset.
In the subseuqent pass, depending on how much memory needs to be backed up, ncu first checks if it can be backed up to GPU memory (which is fast). If there isn’t sufficient free GPU memory available, it then falls back to backing the data up to CPU memory which incurs the penalty of CPU-GPU data transfers.