Skip to content

Add NVTX support for performance analysis - #144

Merged
ajschmidt8 merged 5 commits into
rapidsai:branch-21.12from
gigony:support_nvtx
Nov 30, 2021
Merged

ajschmidt8 merged 5 commits into
rapidsai:branch-21.12from
gigony:support_nvtx

Conversation

@gigony

@gigony gigony commented Nov 9, 2021 •

Copy link
Copy Markdown
Contributor

Need to rebase once #141 is merged.

  • Instrumented important methods in cuCIM with NVTX so can see performance bottlenecks easily with NSight systems.
  • Tracing can be enabled through config file or environment variable or through API and less than 1% performance overheads in normal execution.

Enabling Tracing

Through .cucim.json file

{
        "profiler" : { "trace": true }
}

Through Environment variable

CUCIM_TRACE=1 python

Through API

from cucim import CuImage

CuImage.profiler(trace=True)
#or
CuImage.profiler().trace(True)

CuImage.profiler().config
# {'trace': True}
CuImage.profiler().trace()
# True
CuImage.is_trace_enabled # this is simpler method.
# True

Profiling with NVIDIA Nsight Systems

nsys profile -f true -t cuda,nvtx,osrt -s cpu -x true --trace-fork-before-exec true -o my_profile `which python` benchmark.py
# can add `--stats true`

Then, execute nsight-sys to open the profile results (my_profile.qdrep).

image

closes #137

@gigony gigony added the feature request New feature or request label Nov 9, 2021
@gigony gigony added this to the v21.12.00 milestone Nov 9, 2021
@gigony gigony self-assigned this Nov 9, 2021
@gigony
gigony requested review from a team as code owners November 9, 2021 23:48
@gigony
gigony requested a review from jakirkham November 9, 2021 23:48
@gigony
gigony force-pushed the support_nvtx branch 2 times, most recently from 60bc2ff to 8dddd15 Compare November 10, 2021 02:14
@gigony gigony added the non-breaking Introduces a non-breaking change label Nov 10, 2021
@gigony
gigony force-pushed the support_nvtx branch 9 times, most recently from 0faa489 to b83c51b Compare November 12, 2021 19:02
@jakirkham

Copy link
Copy Markdown
Member

PR ( #141 ) is in. So this is ready for rebasing

@gigony gigony changed the title WIP: Add NVTX support for performance analysis Add NVTX support for performance analysis Nov 24, 2021
@gigony

gigony commented Nov 24, 2021

Copy link
Copy Markdown
Contributor Author

PR ( #141 ) is in. So this is ready for rebasing

@jakirkham
Please help merge this when you have time.

@gigony gigony mentioned this pull request Nov 29, 2021
Comment thread cpp/cmake/deps/nvtx3.cmake Outdated
deps-nvtx3
GIT_REPOSITORY https://github.com/NVIDIA/NVTX.git
GIT_TAG 3c98c8425b0376fd8653aac7cfc6a864f3897752
# GIT_SHALLOW TRUE # Uncomment this when the official release of nvtx3-cpp is available.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we raise an issue to track this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @jakirkham!

Created #168 and updated the comment in nvtx3.cmake.

Comment thread cpp/cmake/deps/nvtx3.cmake Outdated
@jakirkham

Copy link
Copy Markdown
Member

Had a couple questions above. Otherwise LGTM

@ajschmidt8

Copy link
Copy Markdown
Member

CI is passing. Merging.

@ajschmidt8
ajschmidt8 merged commit fb85f01 into rapidsai:branch-21.12 Nov 30, 2021
@jakirkham

Copy link
Copy Markdown
Member

Thanks AJ! 😄

prateek9623 pushed a commit to AMD-Ecosystem/hipCIM that referenced this pull request Sep 22, 2026
…6515) (rapidsai#144)

Promotes the documentation fixes from AIOSS-6515 (merged as rapidsai#142) from
`amd-integration-26.06.00` to `release/26.06.00`.

Documentation only — no source or build-system changes.

```
1bc661e  docs: apply doc team review feedback
d157d0d  docs: simplify the GPU architecture note
c58f509  docs: fix install/test instructions found during 26.06 validation
```

`README.md`, `docs/installation/installation.rst`, `scripts/README.md` —
+102/-44. Reviewed and approved by the doc team on rapidsai#142.

**Please approve only — do not merge from the UI.** This will be
fast-forwarded manually so the individual commits are preserved
(`release/26.06.00` is at `cebc52ce`, the direct parent of `c58f5098`,
so a fast-forward applies cleanly).

Refs: AIOSS-6515
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature request New feature or request non-breaking Introduces a non-breaking change

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[FEA] Support profiling with NVTX

3 participants