Repository navigation
[enhancement] Vendoring OpenMPI as an MPI implementation within TheRock - #2599
Conversation
|
Creating this as a regular PR because it looks pretty ready to me apart from lacking a mirror of OpenMPI with all its submodules, which will have to be handled from your side. |
|
Hi @umarinkovic - this has been waiting a few weeks. Does this need a design review before we proceed? Would a f2f meeting help to discuss the approach? |
Hi, thanks for checking in. I modeled this approach on similar projects and features already included in TheRock. I’d be happy to do a design review to walk through the approach and trade-offs; async or in a f2f, whichever suits you. |
|
Thanks for picking this up @umarinkovic! I unfortunately missed this so far. @HereThereBeDragons can you please do a first review? Will do mine following up then. This would also unblock #2721. |
HereThereBeDragons
left a comment
There was a problem hiding this comment.
thank you so much for this PR!
it already looks quite good.
some question that came up but @marbre is better in saying if we need it or not are:
- do we need to change the symbols/name of the lib as we do for the sysdeps?
- use s3 bucket instead of git clone?
- parallelism during build even though it is background job
| endif() | ||
|
|
||
| execute_process( | ||
| COMMAND git clone --recursive --branch v5.0.9 https://github.com/open-mpi/ompi.git "${_source_dir}" |
There was a problem hiding this comment.
do we want to git clone this or clone it once into a s3 bucket and just pull it from there as zip/tar?
There was a problem hiding this comment.
According to # TODO: mirror full release and use therock_subproject_fetch, this is for TESTING ONLY this is the intention :) I uploaded 5.0.9 to our S3 bucket: https://rocm-third-party-deps.s3.us-east-2.amazonaws.com/openmpi-5.0.9.tar.bz2, mirrored from https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.9.tar.bz2.
| set(MPI_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/install") | ||
| set(MPI_INSTALL_DESTINATION "${CMAKE_INSTALL_PREFIX}") | ||
|
|
||
| include(ProcessorCount) |
There was a problem hiding this comment.
above you declare mpi as a background job. that means it already runs in parallel with other jobs.
maybe @marbre can clarify if that is here something we want to have background parallelism + build parallelism
There was a problem hiding this comment.
I think this is what we have / do for other projects as well but would need to check how we actually handle it in non-CMake third-party deps.
There was a problem hiding this comment.
I got this pattern from looking at other make projects.
From what I understand:
- 1 command in cmake = 1 job
- Passing
BACKGROUND_BUILDensures that no more thantherock_backgroundjobs are started for all projects marked with the flag.
For example, let's say that therock_background jobs is set to 2, that means that no more than 2 projects marked with the flag can be built concurrently. Conversely, on a machine with 8 cores, when building a make project we scoop the core-count and use it as the number of jobs to run the build. So in this scenario, at any given point during the build we will at most have 16 jobs building the projects marked with BACKGROUND_BUILD. Not passing the -j flag to the make build command would result in the project being built with only one job.
There was a problem hiding this comment.
indeopendent of this pr we should have some recommendations of BACKGROUND_BUILD and other parallelism in cmake. i know there were some effforts to make it more clear, also how to enforce to just build sequentially iwth a single thread. but i think those efforts got lost somewhere
|
|
||
| set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}") | ||
| set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include") | ||
| set_and_check(MPI_LIBRARIES "${MPI_ROOT}/lib/lib/libmpi.so") |
There was a problem hiding this comment.
is this a good pattern /lib/lib? otherwise maybe the install instructions must be adjusted
There was a problem hiding this comment.
if we are also building for windows, you cannot check for libmpi.so
There was a problem hiding this comment.
also looking on cmake i found the following:
MPI_LIBRARIES
Deprecated since version 2.8.5: Use the MPI_<lang>_LIBRARIES instead.
There was a problem hiding this comment.
Profilers is disabled on Windows
Lines 208 to 212 in 8115618
and as far as I remember RCCL does not support Windows as well. Thus landing this for Linux only is fine for now but we need to make sure to not accidently build for Windows when looking for an .so. Furthermore, I agree that lib/lib is odd and should be avoided.
There was a problem hiding this comment.
if we are also building for windows, you cannot check for
libmpi.so
Native Windows support for OpenMPI doesn't exist anymore. It can be compiled with Cygwin's gcc, though considering marbre's comments as well it seems to me to be redundant at this point and OpenMPI should probably be excluded from the Windows build for now.
From: https://docs.vllm.ai/en/v0.6.5/getting_started/amd-installation.html#option-2-build-from-source
We primarily develop Open MPI on Linux and MacOS.
Other operating systems are supported, however. The exact list of operating systems supported has changed over time (e.g., native Microsoft Windows support was added in v1.3.3, and although it was removed prior to v1.8, is still supported through Cygwin). See the Platform Notes section for a listing of the OSes that that version supports.
| set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include") | ||
| set_and_check(MPI_LIBRARIES "${MPI_ROOT}/lib/lib/libmpi.so") | ||
|
|
||
| set_and_check(MPIEXEC "${MPI_ROOT}/lib/bin/mpiexec") |
There was a problem hiding this comment.
here the same
MPIEXEC
Deprecated since version 3.10: Use MPIEXEC_EXECUTABLE instead.
i would remove it as we are building with a higher cmake version anyway.
| @PACKAGE_INIT@ | ||
|
|
||
| set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}") | ||
| set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include") |
There was a problem hiding this comment.
this does not seem to exist
MPI_INCLUDE_PATH
Deprecated since version 2.8.5: Use the MPI_<lang>_INCLUDE_DIRS instead.
with
MPI_<lang>_INCLUDE_DIRS
Include path(s) for MPI header.
langs should be C and CXX
There was a problem hiding this comment.
I think whatever we define does exist but this is not what FindMPI expects. As Laura suggested, let's follow https://cmake.org/cmake/help/latest/module/FindMPI.html here and go with
MPI_<lang>_INCLUDE_DIRS
here.
|
|
||
| # Backward compatibility variables | ||
| set(MPI_FOUND TRUE) | ||
| set(MPI_INCLUDE_PATH "${MPI_INCLUDE_DIRS}") |
There was a problem hiding this comment.
see above, this is deprecated. dont think we need it
marbre
left a comment
There was a problem hiding this comment.
Thanks for this PR again! I think there are a few things to address (added to Laura's comments) but this isn't too far from landing if CI passes.
| endif() | ||
|
|
||
| execute_process( | ||
| COMMAND git clone --recursive --branch v5.0.9 https://github.com/open-mpi/ompi.git "${_source_dir}" |
There was a problem hiding this comment.
According to # TODO: mirror full release and use therock_subproject_fetch, this is for TESTING ONLY this is the intention :) I uploaded 5.0.9 to our S3 bucket: https://rocm-third-party-deps.s3.us-east-2.amazonaws.com/openmpi-5.0.9.tar.bz2, mirrored from https://download.open-mpi.org/release/open-mpi/v5.0/openmpi-5.0.9.tar.bz2.
| set(MPI_PREFIX "${CMAKE_CURRENT_BINARY_DIR}/install") | ||
| set(MPI_INSTALL_DESTINATION "${CMAKE_INSTALL_PREFIX}") | ||
|
|
||
| include(ProcessorCount) |
There was a problem hiding this comment.
I think this is what we have / do for other projects as well but would need to check how we actually handle it in non-CMake third-party deps.
| @PACKAGE_INIT@ | ||
|
|
||
| set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}") | ||
| set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include") |
There was a problem hiding this comment.
I think whatever we define does exist but this is not what FindMPI expects. As Laura suggested, let's follow https://cmake.org/cmake/help/latest/module/FindMPI.html here and go with
MPI_<lang>_INCLUDE_DIRS
here.
|
|
||
| set_and_check(MPI_ROOT "${PACKAGE_PREFIX_DIR}") | ||
| set_and_check(MPI_INCLUDE_DIRS "${MPI_ROOT}/include") | ||
| set_and_check(MPI_LIBRARIES "${MPI_ROOT}/lib/lib/libmpi.so") |
There was a problem hiding this comment.
Profilers is disabled on Windows
Lines 208 to 212 in 8115618
and as far as I remember RCCL does not support Windows as well. Thus landing this for Linux only is fine for now but we need to make sure to not accidently build for Windows when looking for an .so. Furthermore, I agree that lib/lib is odd and should be avoided.
|
Thanks for the attention you've given this PR and for helping me clean it up, I am working on pushing the changes tonight. Two questions arose from my side:
[therock-openmpi] checking if gcc supports C11 atomic_fetch_xor_explicit... configure.ac:106: error: version mismatch. This is Automake 1.16.1,
[therock-openmpi] configure.ac:106: but the definition used by this AM_INIT_AUTOMAKE
[therock-openmpi] configure.ac:106: comes from Automake 1.16.5. You should recreate
[therock-openmpi] configure.ac:106: aclocal.m4 with aclocal and run automake again.
[therock-openmpi] yesI've added a reconfigure step to the build and it works this way, however this is not the way the release is intended to be built. Is this okay, or do we need a different solution (installing a newer autoconf on the manylinux image, using an older release?, probing why automake is re-ran in the first place?)
|
If I remember correctly @jayhawk-commits might have had a similar problem or at least needed to touch the pre-installed toolchain in the manylinux docker and can eventually give a recommendation. I think we might want to update.
For now I would exclude Fortran and disable it. As far as I know the project in only use the C/CXX API. Furthermore we would like to switch from gfortran to flang at some point. Right now a third-party dep cannot compiled with our own toolchain due to dependency ordering issues (addressed in #2045). |
b721073 to
af64418
Compare
f0501a6 to
1611e85
Compare
|
Hi guys, sorry for the delay, GH issues... This PR is back up and I've pushed the changes I'd been working on in the meantime. |
HereThereBeDragons
left a comment
There was a problem hiding this comment.
thanks for all your effort :) hope you got your github issues resolved!
i only see one more thing that needs to be fixed. otherwise i leave the final approval to marius
|
I see that the CI is failing on RCCL tests: Should we add something like: therock_test_validate_shared_lib(
PATH dist/lib
LIB_NAMES
libmpi.so
)
therock_provide_artifact(openmpi
TARGET_NEUTRAL
DESCRIPTOR openmpi.toml
COMPONENTS
...
SUBPROJECT_DEPS
therock-openmpi
) |
There have been some issues with RCCL but we switched over to rocm-systems to consume the sources form there now. We actually should be able to now run RCCL tests without installing Open MPI in a dedicated Docker image before (fyi @araravik-psd ). The issue here most likely is that the RCCL tests do not download the Open MPI artifact we now have but that we need those (I haven't checked closely). If it is not that, @araravik-psd hopefully can add to this as I am not super familiar with those tests. |
|
@umarinkovic can you rebase on latest main and resolve the conflicts on |
bd51140 to
f93b022
Compare
Pushed rebase. |
marbre
left a comment
There was a problem hiding this comment.
This misses a artifacts-openmpi.toml that makes sure libmpi.so and other files find their way into the sysdeps artifacts. If you like, we can also land this and iterate but in that case we need to THEROCK_ENABLE_MPI later and leave this set to OFF for now.
## Motivation This allows third-party deps to be built using our own amd-llvm toolchain. This is a step towards a larger bootstrapping effort where we should first build the compiler than use it consistently for all subprojects. * Fixes #2003 by building spdlog with amd-llvm (instead of MSVC on Windows, which fails to build it) * Unblocks #2599 since OpenMPI has issues building using the system toolchain in our manylinux build container that using our own amd-llvm toolchain may avoid ## Technical Details This splits third-party into sysdeps and application libraries: * sysdeps are needed to build the compiler and must be built using the system toolchain (MSVC on Windows) * application libraries can/should be built using a clang toolchain For building spdlog on Windows, see how it sets the `/Zc:__cplusplus` flag for MSVC (see [the code here](https://github.com/gabime/spdlog/blob/3f7e50285936d8ba91d17650d94a2ab48d33169f/CMakeLists.txt#L296-L307)) and we don't want that propagating to later builds using clang. ## Test Plan Try building spdlog and rocroller on Windows. > [!NOTE] > rocroller also needed these changes to try building on Windows: main...ScottTodd:TheRock:rocroller-windows-testing ## Test Result * `therock-spdlog_configure.log` before: ``` 0.5 -- The CXX compiler identification is MSVC 19.44.35222.0 0.7 -- Detecting CXX compiler ABI info 1.7 -- Detecting CXX compiler ABI info - done 1.8 -- Check for working CXX compiler: C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.44.35207/bin/Hostx64/x64/cl.exe - skipped ``` * ``therock-spdlog_configure.log`` after: ``` 0.3 -- The CXX compiler identification is Clang 22.0.0 with GNU-like command-line 0.3 -- Detecting CXX compiler ABI info 0.5 -- Detecting CXX compiler ABI info - done 0.5 -- Check for working CXX compiler: D:/projects/TheRock/build/compiler/amd-llvm/dist/lib/llvm/bin/clang++.exe - skipped ``` * `rocRoller_build.log` before (https://gist.github.com/ScottTodd/791e5addeb02950e542cabf6b93e3ef0#file-rocroller_build_before_spdlog_toolchain_changes-log): ``` 0.1 FAILED: [code=1] GPUArchitectureGenerator/CMakeFiles/GPUArchitectureGenerator.dir/__/lib/source/GPUArchitecture.cpp.obj 0.1 ccache D:\projects\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe ... --dependent-lib=msvcrt /Zc:__cplusplus -MD -MT ... 0.1 clang++: error: no such file or directory: '/Zc:__cplusplus' ``` * `rocRoller_build.log` after (https://gist.github.com/ScottTodd/791e5addeb02950e542cabf6b93e3ef0#file-rocroller_build_after_spdlog_toolchain_changes-log): ``` 4.7 [8/441] Building CXX object GPUArchitectureGenerator/CMakeFiles/GPUArchitectureGenerator.dir/__/lib/source/Assemblers/InProcessAssembler.cpp.obj 4.7 FAILED: [code=1] GPUArchitectureGenerator/CMakeFiles/GPUArchitectureGenerator.dir/__/lib/source/Assemblers/InProcessAssembler.cpp.obj 4.7 ccache D:\projects\TheRock\build\core\clr\dist\lib\llvm\bin\clang++.exe ... --dependent-lib=msvcrt -MD -MT ... 4.7 In file included from D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/source/Assemblers/InProcessAssembler.cpp:33: 4.7 In file included from D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/GPUArchitecture/GPUArchitectureLibrary.hpp:41: 4.7 In file included from D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/GPUArchitecture/GPUArchitecture.hpp:39: 4.7 In file included from D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/DataTypes/DataTypes.hpp:41: 4.7 D:/projects/TheRock/rocm-libraries/shared/rocroller/lib/include\rocRoller/DataTypes/DataTypes_E8M0.hpp:57:17: error: unknown type name 'uint' 4.7 57 | uint mantissa : 23; 4.7 | ^ ``` ## Submission Checklist - [x] Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.
|
@umarinkovic : Will we clean up the https://github.com/ROCm/TheRock/blob/main/dockerfiles/build_manylinux_rccl_x86_64.Dockerfile after landing this PR ? This kept in place to aid the RCCL CI workflows but with OpenMPI being provided as part of third-party - we dont need this Dockerfile anymore am I right? |
This is what we intend, yes. |
marbre
left a comment
There was a problem hiding this comment.
There is another merge conflict which needs to be resolved (and there are likely more to come). Let's land this asap and iterate once it was merged.
| option(THEROCK_ENABLE_MPI "Enables building components with Message Passing Interface (MPI) support" OFF) | ||
| cmake_dependent_option(THEROCK_ENABLE_MPI | ||
| "Enables building components with Message Passing Interface (MPI) support" | ||
| ON "NOT WIN32" OFF) |
There was a problem hiding this comment.
As mentioned before, let's disable this for now :) With THEROCK_ENABLE_MPI, rccl-tests will behave as before. Enabling MPI will need some more CI changes:
- rccl-tests should be single-node / non-MPI in TheRock
- rccl-tests should use the bundled MPI when building in rocm-systems
Easiest is therefore to set THEROCK_ENABLE_MPI to OFF for now.
There was a problem hiding this comment.
Got it, I've set the default to OFF, and added a TODO for future reference.
| artifact_deps = [] | ||
| feature_group = "CORE" | ||
|
|
||
| [artifacts.openmpi] |
There was a problem hiding this comment.
I think we further need to make sure that rccl / rccl-tests properly pull in openmpi. As mentioned in my other comment, we can land (if the build passes) but tend to set THEROCK_ENABLE_MPI to OFF for now.
There was a problem hiding this comment.
Did you want openmpi added to the artifact deps of rccl in this PR? Or should that be left for when the additional changes land?
[artifacts.rccl]
artifact_group = "comm-libs"
type = "target-specific"
artifact_deps = ["core-runtime", "core-hip", "hipify", "rocprofiler-sdk", "core-amdsmi", "openmpi"]
disable_platforms = ["windows"]There was a problem hiding this comment.
No need to yet but if it is already it doesn’t hurt :)
to fit latest CMake standards
… preserve correct RPATH
Co-authored-by: Laura Promberger <lpromber@amd.com>
…ed name downstream
Add artifact-openmpi.toml, update BUILD_TOPOLOGY.toml to include openmpi artifacts and update openmpi/CMakeLists.txt to call therock_provide_artifact.
e90d4ad to
48a5b37
Compare
|
@marbre What's left to do to get this merged? Re-run the build on the CI with the new changes (USE_MPI flag reverted to OFF?) |
marbre
left a comment
There was a problem hiding this comment.
Apologies for yet more delay, the time I can offer for review is unfortunately too limited. LGTM, this is very much appreciated! I approved a CI run.
@HereThereBeDragons please merge if CI passes.
Third-party deps were added twice and were thus breaking the CMake build tree during the config stage
|
Okay, I found the error. Apparently during rebasing on a merge-conflict I had accidentally included some old code for including the sysdeps dir which was supposed to have been moved elsewhere so that duplicate was causing these errors: -- Including subproject therock-frugally-deep (from B:/build/third-party/frugally-deep/source)
-- Including subproject therock-spdlog (from B:/build/third-party/spdlog/source)
-- Including subproject therock-host-blas (from C:/home/runner/_work/TheRock/TheRock/third-party/host-blas/.)
CMake Error at third-party/CMakeLists.txt:50 (add_subdirectory):
The binary directory
B:/build/third-party/sysdeps/windows
is already used to build a source directory. It cannot be used to build
source directory
C:/home/runner/_work/TheRock/TheRock/third-party/sysdeps/windows
Specify a unique binary directory name.and on Linux: -- Including subproject therock-host-blas (from /__w/TheRock/TheRock/third-party/host-blas/.)
-- Including subproject therock-SuiteSparse (from /__w/TheRock/TheRock/build/third-party/SuiteSparse/source)
CMake Error at third-party/CMakeLists.txt:48 (add_subdirectory):
The binary directory
/__w/TheRock/TheRock/build/third-party/sysdeps/linux
is already used to build a source directory. It cannot be used to build
source directory
/__w/TheRock/TheRock/third-party/sysdeps/linux
Specify a unique binary directory name.I started a build locally on Linux after these changes and the problem disappeared (build completed successfully), so if someone can restart the CI to get this merged |
|
CI passed, thanks! |
|
Merged, thanks again for your contribution! |
Motivation
Issue: #1284
Technical Details
Added OpenMPI as an MPI implementation within TheRock
Test Plan
Run the build locally using OpenMPI as an MPI implementation
Test Result
Build successful
Submission Checklist