Skip to content

[enhancement] Vendoring OpenMPI as an MPI implementation within TheRock - #2599

Merged
marbre merged 17 commits into
ROCm:mainfrom
umarinkovic:enhancement/openmpi
Mar 6, 2026
Merged

marbre merged 17 commits into
ROCm:mainfrom
umarinkovic:enhancement/openmpi

Conversation

@umarinkovic

Copy link
Copy Markdown
Contributor

Motivation

Issue: #1284

Technical Details

Added OpenMPI as an MPI implementation within TheRock

Test Plan

Run the build locally using OpenMPI as an MPI implementation

Test Result

Build successful

Submission Checklist

@umarinkovic

Copy link
Copy Markdown
Contributor Author

Creating this as a regular PR because it looks pretty ready to me apart from lacking a mirror of OpenMPI with all its submodules, which will have to be handled from your side.

@stellaraccident

Copy link
Copy Markdown
Collaborator

Hi @umarinkovic - this has been waiting a few weeks. Does this need a design review before we proceed? Would a f2f meeting help to discuss the approach?

@umarinkovic

Copy link
Copy Markdown
Contributor Author

Hi @umarinkovic - this has been waiting a few weeks. Does this need a design review before we proceed? Would a f2f meeting help to discuss the approach?

Hi, thanks for checking in. I modeled this approach on similar projects and features already included in TheRock. I’d be happy to do a design review to walk through the approach and trade-offs; async or in a f2f, whichever suits you.

@marbre

marbre commented Jan 14, 2026

Copy link
Copy Markdown
Member

Thanks for picking this up @umarinkovic! I unfortunately missed this so far. @HereThereBeDragons can you please do a first review? Will do mine following up then. This would also unblock #2721.

@HereThereBeDragons HereThereBeDragons left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thank you so much for this PR!

it already looks quite good.

some question that came up but @marbre is better in saying if we need it or not are:

  • do we need to change the symbols/name of the lib as we do for the sysdeps?
  • use s3 bucket instead of git clone?
  • parallelism during build even though it is background job

Comment thread third-party/OpenMPI/CMakeLists.txt Outdated
endif()

execute_process(
COMMAND git clone --recursive --branch v5.0.9 https://github.com/open-mpi/ompi.git "${_source_dir}"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we want to git clone this or clone it once into a s3 bucket and just pull it from there as zip/tar?

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to # TODO: mirror full release and use therock_subproject_fetch, this is for TESTING ONLY this is the intention :) I uploaded 5.0.9 to our S3 bucket: https://rocm-third-party-deps.s3.us-east-2.amazonaws.com/openmpi-5.0.9.tar.bz2, mirrored from https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.9.tar.bz2.

set(MPI_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/install")
set(MPI_INSTALL_DESTINATION "${CMAKE_INSTALL_PREFIX}")

include(ProcessorCount)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

above you declare mpi as a background job. that means it already runs in parallel with other jobs.
maybe @marbre can clarify if that is here something we want to have background parallelism + build parallelism

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is what we have / do for other projects as well but would need to check how we actually handle it in non-CMake third-party deps.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got this pattern from looking at other make projects.
From what I understand:

  • 1 command in cmake = 1 job
  • Passing BACKGROUND_BUILD ensures that no more than therock_background jobs are started for all projects marked with the flag.

For example, let's say that therock_background jobs is set to 2, that means that no more than 2 projects marked with the flag can be built concurrently. Conversely, on a machine with 8 cores, when building a make project we scoop the core-count and use it as the number of jobs to run the build. So in this scenario, at any given point during the build we will at most have 16 jobs building the projects marked with BACKGROUND_BUILD. Not passing the -j flag to the make build command would result in the project being built with only one job.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeopendent of this pr we should have some recommendations of BACKGROUND_BUILD and other parallelism in cmake. i know there were some effforts to make it more clear, also how to enforce to just build sequentially iwth a single thread. but i think those efforts got lost somewhere

Comment thread third-party/OpenMPI/mpi-config.cmake.in Outdated

set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}")
set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include")
set_and_check(MPI_LIBRARIES "${MPI_ROOT}/lib/lib/libmpi.so")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is this a good pattern /lib/lib? otherwise maybe the install instructions must be adjusted

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are also building for windows, you cannot check for libmpi.so

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

also looking on cmake i found the following:

MPI_LIBRARIES
Deprecated since version 2.8.5: Use the MPI_<lang>_LIBRARIES instead.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Profilers is disabled on Windows

TheRock/CMakeLists.txt

Lines 208 to 212 in 8115618

# On some platforms (WIN32 presently), the profiler is always disabled.
cmake_dependent_option(
THEROCK_FLAG_INCLUDE_PROFILER
"Allows the profiler to be manually disabled by setting to OFF"
ON "NOT WIN32" OFF)

and as far as I remember RCCL does not support Windows as well. Thus landing this for Linux only is fine for now but we need to make sure to not accidently build for Windows when looking for an .so. Furthermore, I agree that lib/lib is odd and should be avoided.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if we are also building for windows, you cannot check for libmpi.so

Native Windows support for OpenMPI doesn't exist anymore. It can be compiled with Cygwin's gcc, though considering marbre's comments as well it seems to me to be redundant at this point and OpenMPI should probably be excluded from the Windows build for now.

From: https://docs.vllm.ai/en/v0.6.5/getting_started/amd-installation.html#option-2-build-from-source

We primarily develop Open MPI on Linux and MacOS.
Other operating systems are supported, however. The exact list of operating systems supported has changed over time (e.g., native Microsoft Windows support was added in v1.3.3, and although it was removed prior to v1.8, is still supported through Cygwin). See the Platform Notes section for a listing of the OSes that that version supports.

Comment thread third-party/OpenMPI/mpi-config.cmake.in Outdated
set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include")
set_and_check(MPI_LIBRARIES "${MPI_ROOT}/lib/lib/libmpi.so")

set_and_check(MPIEXEC "${MPI_ROOT}/lib/bin/mpiexec")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

here the same

MPIEXEC
Deprecated since version 3.10: Use MPIEXEC_EXECUTABLE instead.

i would remove it as we are building with a higher cmake version anyway.

Comment thread third-party/OpenMPI/mpi-config.cmake.in Outdated
@PACKAGE_INIT@

set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}")
set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this does not seem to exist

MPI_INCLUDE_PATH
Deprecated since version 2.8.5: Use the MPI_<lang>_INCLUDE_DIRS instead.

with

MPI_<lang>_INCLUDE_DIRS
Include path(s) for MPI header.

langs should be C and CXX

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think whatever we define does exist but this is not what FindMPI expects. As Laura suggested, let's follow https://cmake.org/cmake/help/latest/module/FindMPI.html here and go with

MPI_<lang>_INCLUDE_DIRS

here.

Comment thread third-party/OpenMPI/mpi-config.cmake.in Outdated

# Backward compatibility variables
set(MPI_FOUND TRUE)
set(MPI_INCLUDE_PATH "${MPI_INCLUDE_DIRS}")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see above, this is deprecated. dont think we need it

Comment thread third-party/openmpi/mpi-config.cmake.in

@marbre marbre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR again! I think there are a few things to address (added to Laura's comments) but this isn't too far from landing if CI passes.

Comment thread third-party/OpenMPI/CMakeLists.txt Outdated
endif()

execute_process(
COMMAND git clone --recursive --branch v5.0.9 https://github.com/open-mpi/ompi.git "${_source_dir}"

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to # TODO: mirror full release and use therock_subproject_fetch, this is for TESTING ONLY this is the intention :) I uploaded 5.0.9 to our S3 bucket: https://rocm-third-party-deps.s3.us-east-2.amazonaws.com/openmpi-5.0.9.tar.bz2, mirrored from https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.9.tar.bz2.

set(MPI_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/install")
set(MPI_INSTALL_DESTINATION "${CMAKE_INSTALL_PREFIX}")

include(ProcessorCount)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is what we have / do for other projects as well but would need to check how we actually handle it in non-CMake third-party deps.

Comment thread third-party/OpenMPI/mpi-config.cmake.in Outdated
@PACKAGE_INIT@

set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}")
set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think whatever we define does exist but this is not what FindMPI expects. As Laura suggested, let's follow https://cmake.org/cmake/help/latest/module/FindMPI.html here and go with

MPI_<lang>_INCLUDE_DIRS

here.

Comment thread third-party/OpenMPI/mpi-config.cmake.in Outdated

set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}")
set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include")
set_and_check(MPI_LIBRARIES "${MPI_ROOT}/lib/lib/libmpi.so")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Profilers is disabled on Windows

TheRock/CMakeLists.txt

Lines 208 to 212 in 8115618

# On some platforms (WIN32 presently), the profiler is always disabled.
cmake_dependent_option(
THEROCK_FLAG_INCLUDE_PROFILER
"Allows the profiler to be manually disabled by setting to OFF"
ON "NOT WIN32" OFF)

and as far as I remember RCCL does not support Windows as well. Thus landing this for Linux only is fine for now but we need to make sure to not accidently build for Windows when looking for an .so. Furthermore, I agree that lib/lib is odd and should be avoided.

Comment thread third-party/openmpi/mpi-config.cmake.in
@umarinkovic

Copy link
Copy Markdown
Contributor Author

@marbre @HereThereBeDragons

Thanks for the attention you've given this PR and for helping me clean it up, I am working on pushing the changes tonight.

Two questions arose from my side:

  1. The release of OpenMPI mirrored (5.0.9) was configured with autoconf 1.16.5. The manylinux image provided by TheRock uses autoconf 1.16.1 (incompatible, see logs below)
[therock-openmpi] checking if gcc  supports C11 atomic_fetch_xor_explicit... configure.ac:106: error: version mismatch.  This is Automake 1.16.1,
[therock-openmpi] configure.ac:106: but the definition used by this AM_INIT_AUTOMAKE
[therock-openmpi] configure.ac:106: comes from Automake 1.16.5.  You should recreate
[therock-openmpi] configure.ac:106: aclocal.m4 with aclocal and run automake again.
[therock-openmpi] yes

I've added a reconfigure step to the build and it works this way, however this is not the way the release is intended to be built. Is this okay, or do we need a different solution (installing a newer autoconf on the manylinux image, using an older release?, probing why automake is re-ran in the first place?)

  1. Do we need to build OpenMPI with Fortran support as well? From what I've seen in the install tree, this minimal configuration does come with Fortran libs / headers / mod files. Do we need to include these in the config.in file as well? Do we need to pass any flags to the configure step to get a full(er?) Fortran build? Or should it perhaps be explicitly excluded if downstream projects only require C/CXX MPI?

@marbre

marbre commented Jan 21, 2026

Copy link
Copy Markdown
Member

@marbre @HereThereBeDragons

Thanks for the attention you've given this PR and for helping me clean it up, I am working on pushing the changes tonight.

Two questions arose from my side:

  1. The release of OpenMPI mirrored (5.0.9) was configured with autoconf 1.16.5. The manylinux image provided by TheRock uses autoconf 1.16.1 (incompatible, see logs below)
[therock-openmpi] checking if gcc  supports C11 atomic_fetch_xor_explicit... configure.ac:106: error: version mismatch.  This is Automake 1.16.1,
[therock-openmpi] configure.ac:106: but the definition used by this AM_INIT_AUTOMAKE
[therock-openmpi] configure.ac:106: comes from Automake 1.16.5.  You should recreate
[therock-openmpi] configure.ac:106: aclocal.m4 with aclocal and run automake again.
[therock-openmpi] yes

I've added a reconfigure step to the build and it works this way, however this is not the way the release is intended to be built. Is this okay, or do we need a different solution (installing a newer autoconf on the manylinux image, using an older release?, probing why automake is re-ran in the first place?)

If I remember correctly @jayhawk-commits might have had a similar problem or at least needed to touch the pre-installed toolchain in the manylinux docker and can eventually give a recommendation. I think we might want to update.

  1. Do we need to build OpenMPI with Fortran support as well? From what I've seen in the install tree, this minimal configuration does come with Fortran libs / headers / mod files. Do we need to include these in the config.in file as well? Do we need to pass any flags to the configure step to get a full(er?) Fortran build? Or should it perhaps be explicitly excluded if downstream projects only require C/CXX MPI?

For now I would exclude Fortran and disable it. As far as I know the project in only use the C/CXX API. Furthermore we would like to switch from gfortran to flang at some point. Right now a third-party dep cannot compiled with our own toolchain due to dependency ordering issues (addressed in #2045).

@umarinkovic
umarinkovic force-pushed the enhancement/openmpi branch 2 times, most recently from f0501a6 to 1611e85 Compare January 26, 2026 09:07
@umarinkovic

Copy link
Copy Markdown
Contributor Author

@marbre @HereThereBeDragons

Hi guys, sorry for the delay, GH issues...

This PR is back up and I've pushed the changes I'd been working on in the meantime.

@HereThereBeDragons HereThereBeDragons left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for all your effort :) hope you got your github issues resolved!

i only see one more thing that needs to be fixed. otherwise i leave the final approval to marius

Comment thread CMakeLists.txt Outdated
@umarinkovic

Copy link
Copy Markdown
Contributor Author

@HereThereBeDragons @marbre

I see that the CI is failing on RCCL tests:

INFO root:test_rccl.py:59 
++ Exec [/__w/TheRock/TheRock]$ ./build/bin/all_gather_perf ./build/bin/all_gather_perf: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory

Should we add something like:

therock_test_validate_shared_lib(
  PATH dist/lib
  LIB_NAMES
    libmpi.so
)

therock_provide_artifact(openmpi
  TARGET_NEUTRAL
  DESCRIPTOR openmpi.toml
  COMPONENTS
    ...
  SUBPROJECT_DEPS
    therock-openmpi
)

@marbre

marbre commented Feb 3, 2026

Copy link
Copy Markdown
Member

@HereThereBeDragons @marbre

I see that the CI is failing on RCCL tests:

INFO root:test_rccl.py:59 
++ Exec [/__w/TheRock/TheRock]$ ./build/bin/all_gather_perf ./build/bin/all_gather_perf: error while loading shared libraries: libmpi.so.40: cannot open shared object file: No such file or directory

Should we add something like:

therock_test_validate_shared_lib(
  PATH dist/lib
  LIB_NAMES
    libmpi.so
)

therock_provide_artifact(openmpi
  TARGET_NEUTRAL
  DESCRIPTOR openmpi.toml
  COMPONENTS
    ...
  SUBPROJECT_DEPS
    therock-openmpi
)

There have been some issues with RCCL but we switched over to rocm-systems to consume the sources form there now. We actually should be able to now run RCCL tests without installing Open MPI in a dedicated Docker image before (fyi @araravik-psd ). The issue here most likely is that the RCCL tests do not download the Open MPI artifact we now have but that we need those (I haven't checked closely). If it is not that, @araravik-psd hopefully can add to this as I am not super familiar with those tests.

@marbre

marbre commented Feb 12, 2026

Copy link
Copy Markdown
Member

@umarinkovic can you rebase on latest main and resolve the conflicts on profiler/CMakeLists.txt.

@umarinkovic

Copy link
Copy Markdown
Contributor Author

can you rebase on latest main and resolve the conflicts on profiler/CMakeLists.txt.

Pushed rebase.

@marbre marbre mentioned this pull request Feb 16, 2026

@marbre marbre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This misses a artifacts-openmpi.toml that makes sure libmpi.so and other files find their way into the sysdeps artifacts. If you like, we can also land this and iterate but in that case we need to THEROCK_ENABLE_MPI later and leave this set to OFF for now.

ScottTodd added a commit that referenced this pull request Feb 16, 2026
## Motivation

This allows third-party deps to be built using our own amd-llvm
toolchain. This is a step towards a larger bootstrapping effort where we
should first build the compiler than use it consistently for all
subprojects.

* Fixes #2003 by building spdlog
with amd-llvm (instead of MSVC on Windows, which fails to build it)
* Unblocks #2599 since OpenMPI has
issues building using the system toolchain in our manylinux build
container that using our own amd-llvm toolchain may avoid

## Technical Details

This splits third-party into sysdeps and application libraries:
* sysdeps are needed to build the compiler and must be built using the
system toolchain (MSVC on Windows)
* application libraries can/should be built using a clang toolchain

For building spdlog on Windows, see how it sets the `/Zc:__cplusplus`
flag for MSVC (see [the code
here](https://github.com/gabime/spdlog/blob/3f7e50285936d8ba91d17650d94a2ab48d33169f/CMakeLists.txt#L296-L307))
and we don't want that propagating to later builds using clang.

## Test Plan

Try building spdlog and rocroller on Windows.

> [!NOTE]
> rocroller also needed these changes to try building on Windows:
main...ScottTodd:TheRock:rocroller-windows-testing

## Test Result

* `therock-spdlog_configure.log` before:
  ```
  0.5	-- The CXX compiler identification is MSVC 19.44.35222.0
  0.7	-- Detecting CXX compiler ABI info
  1.7	-- Detecting CXX compiler ABI info - done
1.8 -- Check for working CXX compiler: C:/Program Files/Microsoft Visual
Studio/2022/Community/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe -
skipped
  ```
* ``therock-spdlog_configure.log`` after:
  ```
0.3 -- The CXX compiler identification is Clang 22.0.0 with GNU-like
command-line
  0.3	-- Detecting CXX compiler ABI info
  0.5	-- Detecting CXX compiler ABI info - done
0.5 -- Check for working CXX compiler:
D:/projects/TheRock/build/compiler/amd-llvm/dist/lib/llvm/bin/clang++.exe
- skipped
  ```
* `rocRoller_build.log` before
(https://gist.github.com/ScottTodd/791e5addeb02950e542cabf6b93e3ef0#file-rocroller_build_before_spdlog_toolchain_changes-log):
  ```
0.1 FAILED: [code=1]
GPUArchitectureGenerator/CMakeFiles/GPUArchitectureGenerator.dir/__/lib/source/GPUArchitecture.cpp.obj
0.1 ccache
D:\projects\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe ...
--dependent-lib=msvcrt /Zc:__cplusplus -MD -MT ...
  0.1	clang++: error: no such file or directory: '/Zc:__cplusplus'
  ```
* `rocRoller_build.log` after
(https://gist.github.com/ScottTodd/791e5addeb02950e542cabf6b93e3ef0#file-rocroller_build_after_spdlog_toolchain_changes-log):
  ```
4.7 [8/441] Building CXX object
GPUArchitectureGenerator/CMakeFiles/GPUArchitectureGenerator.dir/__/lib/source/Assemblers/InProcessAssembler.cpp.obj
4.7 FAILED: [code=1]
GPUArchitectureGenerator/CMakeFiles/GPUArchitectureGenerator.dir/__/lib/source/Assemblers/InProcessAssembler.cpp.obj
4.7 ccache
D:\projects\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe ...
--dependent-lib=msvcrt -MD -MT ...
4.7 In file included from
D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/source/Assemblers/InProcessAssembler.cpp:33:
4.7 In file included from
D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/GPUArchitecture/GPUArchitectureLibrary.hpp:41:
4.7 In file included from
D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/GPUArchitecture/GPUArchitecture.hpp:39:
4.7 In file included from
D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/DataTypes/DataTypes.hpp:41:
4.7
D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/DataTypes/DataTypes_E8M0.hpp:57:17:
error: unknown type name 'uint'
  4.7	   57 |                 uint mantissa : 23;
  4.7	      |                 ^
  ```

## Submission Checklist

- [x] Look over the contributing guidelines at
https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
@amd-aakash

Copy link
Copy Markdown
Contributor

@umarinkovic : Will we clean up the https://github.com/ROCm/TheRock/blob/main/dockerfiles/build_manylinux_rccl_x86_64.Dockerfile after landing this PR ? This kept in place to aid the RCCL CI workflows but with OpenMPI being provided as part of third-party - we dont need this Dockerfile anymore am I right?

@marbre

marbre commented Feb 24, 2026

Copy link
Copy Markdown
Member

@umarinkovic : Will we clean up the https://github.com/ROCm/TheRock/blob/main/dockerfiles/build_manylinux_rccl_x86_64.Dockerfile after landing this PR ? This kept in place to aid the RCCL CI workflows but with OpenMPI being provided as part of third-party - we dont need this Dockerfile anymore am I right?

This is what we intend, yes.

@marbre marbre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is another merge conflict which needs to be resolved (and there are likely more to come). Let's land this asap and iterate once it was merged.

Comment thread CMakeLists.txt Outdated
Comment on lines +119 to +121
option(THEROCK_ENABLE_MPI "Enables building components with Message Passing Interface (MPI) support" OFF)
cmake_dependent_option(THEROCK_ENABLE_MPI
"Enables building components with Message Passing Interface (MPI) support"
ON "NOT WIN32" OFF)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As mentioned before, let's disable this for now :) With THEROCK_ENABLE_MPI, rccl-tests will behave as before. Enabling MPI will need some more CI changes:

  • rccl-tests should be single-node / non-MPI in TheRock
  • rccl-tests should use the bundled MPI when building in rocm-systems

Easiest is therefore to set THEROCK_ENABLE_MPI to OFF for now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Got it, I've set the default to OFF, and added a TODO for future reference.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

Comment thread BUILD_TOPOLOGY.toml
artifact_deps = []
feature_group = "CORE"

[artifacts.openmpi]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we further need to make sure that rccl / rccl-tests properly pull in openmpi. As mentioned in my other comment, we can land (if the build passes) but tend to set THEROCK_ENABLE_MPI to OFF for now.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did you want openmpi added to the artifact deps of rccl in this PR? Or should that be left for when the additional changes land?

[artifacts.rccl]
artifact_group = "comm-libs"
type = "target-specific"
artifact_deps = ["core-runtime", "core-hip", "hipify", "rocprofiler-sdk", "core-amdsmi", "openmpi"]
disable_platforms = ["windows"]

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No need to yet but if it is already it doesn’t hurt :)

@umarinkovic
umarinkovic requested a review from marbre February 27, 2026 08:54
@umarinkovic

Copy link
Copy Markdown
Contributor Author

@marbre What's left to do to get this merged? Re-run the build on the CI with the new changes (USE_MPI flag reverted to OFF?)

@marbre marbre left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies for yet more delay, the time I can offer for review is unfortunately too limited. LGTM, this is very much appreciated! I approved a CI run.

@HereThereBeDragons please merge if CI passes.

Third-party deps were added twice and were thus breaking the CMake build tree during the config stage
@umarinkovic

Copy link
Copy Markdown
Contributor Author

Okay, I found the error. Apparently during rebasing on a merge-conflict I had accidentally included some old code for including the sysdeps dir which was supposed to have been moved elsewhere so that duplicate was causing these errors:

-- Including subproject therock-frugally-deep (from B:/build/third-party/frugally-deep/source)
-- Including subproject therock-spdlog (from B:/build/third-party/spdlog/source)
-- Including subproject therock-host-blas (from C:/home/runner/_work/TheRock/TheRock/third-party/host-blas/.)
CMake Error at third-party/CMakeLists.txt:50 (add_subdirectory):
  The binary directory

    B:/build/third-party/sysdeps/windows

  is already used to build a source directory.  It cannot be used to build
  source directory

    C:/home/runner/_work/TheRock/TheRock/third-party/sysdeps/windows

  Specify a unique binary directory name.

and on Linux:

-- Including subproject therock-host-blas (from /__w/TheRock/TheRock/third-party/host-blas/.)
-- Including subproject therock-SuiteSparse (from /__w/TheRock/TheRock/build/third-party/SuiteSparse/source)
CMake Error at third-party/CMakeLists.txt:48 (add_subdirectory):
  The binary directory

    /__w/TheRock/TheRock/build/third-party/sysdeps/linux

  is already used to build a source directory.  It cannot be used to build
  source directory

    /__w/TheRock/TheRock/third-party/sysdeps/linux

  Specify a unique binary directory name.

I started a build locally on Linux after these changes and the problem disappeared (build completed successfully), so if someone can restart the CI to get this merged

@marbre
@HereThereBeDragons

@marbre

marbre commented Mar 6, 2026

Copy link
Copy Markdown
Member

CI passed, thanks!

@marbre
marbre merged commit 82348d8 into ROCm:main Mar 6, 2026
113 of 117 checks passed
@github-project-automation github-project-automation Bot moved this from TODO to Done in TheRock Triage Mar 6, 2026
@marbre

marbre commented Mar 6, 2026

Copy link
Copy Markdown
Member

Merged, thanks again for your contribution!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants