Question about compile & usage on iOS/macOS platform

Hello. I'm a iOS developer and recently I'm build a iOS-library which integrate libheif to a third-party lib SDWebImageHEIFCoder. Which support HEIF/HEIC decoding in old version iOS/macOS firmware. (Since iOS 11 & macOS 10.13 support HEIC natively)

But since I'm not an expert in image processing field, here are some questions I faced during the development. Some questions may be silly and I hope this will not bother someone :)

Readme

It seems some part of Readme is out-of-date.

heif_context_new_heic this API is not available now. Maybe we can update the demo code about encoding ?
frame-parallel branch described in the compile part seems out-of-date. I try this branch of libde265 but this can't compile with the master branch of libheif or latest v1.1.0 tag. So I then use the latest v1.0.3 tag of libde265 with v1.1.0 libheif tag to compile to iOS/macOS platform.

Compile

I find that your're also the maintainer of libde265, So I put the question here.

To compile to iOS platform, I'm using Cmake with ios-cmake. For macOS, just using the official Cmake. All of following problem is about Cmake compile issue. (Not using autoconf)

libde265 master branch or latest tag can not compile with the build-in config. It seems this API memalign is not availble on macOS & iOS platform.

I solve this by passing -DHAVE_POSIX_MEMALIGN flag to Cmake to use posix_memalign instead. However, since Cmake support feature detect, maybe this flag can be automatically config instead of manual set?

libde265 master branch or latest tag can not compile with the build-in config. It seems the SSE4.1 detect cause some issue. See the log here:

[ 34%] Building CXX object libde265/x86/CMakeFiles/x86_sse.dir/sse-motion.cc.o
/Users/lizhuoli/Documents/GitHub/libde265/libde265/x86/sse-motion.cc:746:29: error: 
      always_inline function '_mm_packus_epi32' requires target feature
      'sse4.1', but would be inlined into function
      'ff_hevc_weighted_pred_avg_8_sse4' that is compiled without support for
      'sse4.1'
                       r0 = _mm_packus_epi32(r0, r2);
                            ^
/Users/lizhuoli/Documents/GitHub/libde265/libde265/x86/sse-motion.cc:747:29: error: 
      always_inline function '_mm_packus_epi32' requires target feature
      'sse4.1', but would be inlined into function
      'ff_hevc_weighted_pred_avg_8_sse4' that is compiled without support for
      'sse4.1'
                       r1 = _mm_packus_epi32(r1, r3);

I solve this by passing -DDISABLE_SSE flag to Cmake to disable it. However, since ARM64 support SSE4.1, I guess this is not a good solution.

Usage

After I compile the binary for iOS & macOS platform and start using libheif. Though I can successfully run the demo and see the HEIF image decoding works. But there are some question I want to know.

Does libheif support progressive/Incremental image decoding ? Because my lib is a plugin to an image loading system, which support rendering partial image during large image download.
libheif's this encoding API heif_context_write seems a little wired. I think this may be some misunderstanding from me but libheif library (See the code comments below). I use this API to write the encoded heif data to a memory representation but not file.

static struct heif_error WriteImageData(struct heif_context* ctx,
                                                const void* data, size_t size, void* userdata) {
    __autoreleasing NSData **imageData = (__autoreleasing NSData **)userdata;
    NSCParameterAssert(imageData);
    
    heif_error error;
    if (!data) {
        // Can this rgba data be NULL ?
        // And how can I get the error information about this because there are no public API or input arg to grab the error information
        error.code = heif_error_Encoder_plugin_error;
        return error;
    }
    *imageData = [NSData dataWithBytes:data length:size];
    
    error.code = heif_error_Ok;
    return error;
}

// test code
void test() {
    NSData *data;
    heif_writer *writer;
    writer->writer_api_version = 1;
    writer->write = WriteImageData; // This is a function pointer
    
    error = heif_context_write(ctx, writer, &data);
}

Another question. After I try to integrate the libx265 and libheif to support HEIF encoding, I find a really strange issue about this heif_image_add_plane API.

It looks like heif_image_add_plane will modify some memory outside addreess and cause issue. This demo code show the result. After that line of function call, a external variable (not related to libheif all of class) was modified.

CGImageRef imageRef = image.CGImage; // This is iOS/macOS platform's Core Graphics represent image

// code to fill in the image
heif_chroma chroma = hasAlpha ? heif_chroma_interleaved_RGBA : heif_chroma_interleaved_RGB;
heif_image* img;
error = heif_image_create(width, height, heif_colorspace_RGB, chroma, &img);
if (error.code != heif_error_Ok) {
    heif_context_free(ctx);
    heif_encoder_release(encoder);
    return nil;
}

// -> Preview the image before this call

error = heif_image_add_plane(img, heif_channel_interleaved, width, height, 24);

// -> Preview the image after this call (using step over)

The two preview of image (Xcode's feature to view the debug image) show that the original CGImageRef internal storage was modified by the heif function call.

Before:

After:

Concerning your invalid memory access. My guess is that it is because of this line:
heif_image_add_plane(img, heif_channel_interleaved, width, height, 24);
When the image has an alpha channel, you should allocate 32 bits per pixel instead of only 24.

Concerning

I solve this by passing -DDISABLE_SSE flag to Cmake to disable it. However, since ARM64 support SSE4.1, I guess this is not a good solution.

No, ARM64 does not support SSE. SSE is Intel x86 only. You are mixing this up with ARM64's NEON.

Does libheif support progressive/Incremental image decoding ?

Currently, it does not. Maybe I will add this later at least for tiled images, but we first have to see if there is a real need for it.

libheif's this encoding API heif_context_write seems a little wired

Basically, you are doing things right. 'data' will never be null, so you can skip that test. However, you will have to take care that the write-callback can be called several times for an image.

frame-parallel branch described in the compile part seems out-of-date

Actually, the 'frame-parallel' branch is the newer branch, which will become libde265 version 2. The API is not stable yet, however. libheif should also understand the version 2 API, but it might be that I broke things a bit since I am still making changes to libde265 version 2.

PS: I will take care of the other issues later.

@farindk

@farindk Thanks for your reply. I'll try to figure out the encoding issue again. (Maybe I need to use 4 planar channel with R/G/B/A instead of that heif_channel_interleaved).

I recently find this libheif repo and want to have a try. It seems that this repo is still during quick development. I'll follow the changes for it. :)

heif_channel_interleaved is ok, you just have to adjust the number of bits like this:
heif_image_add_plane(image, heif_channel_interleaved, width, height, has_alpha ? 32 : 24);

@farindk

@farindk Actually...The previous code I use bitsPerPixel (For non-alpha is 24) to feed bit_depth arg. However, that still no luck.

Here I grab the internal bitmap data stored in CGImageRef, run again and using lldb to print the result. After that function call (using Step Over), the bitmap data was modified. (See the output in the bottom right of Xcode). Quite strange....That uint8_t *rgba or CGImageRef variable even not been passed to any function call of libheif.

You can try using an address sanitizer tool (e.g., valgrind, ...) to check where the memory is actually modified.

It seems that std::vector's resize cause this issue ? (I update my project to use the source code instead of compiled framework)

After resize was called, the original imageRef's internal bitmap vector from the call stack was modified...

OK...I got it works. After I convert the original image to RGB888/RGBA8888 before call heif_image_add_plane, and test again. It now works...And there are no memory address issue. Though this is a little strange. 😅

And one interesting thing, during my experience, I found that there are no ways to manual add R/G/B/A 4 planar to do HEIC encoding. Because of current HEVC only support YCbCr420, and it will try to convert RGBA to YCbCr. However, during convert, it does not check heif_channel_R, but only assume heif_channel_interleaved. Like this:

//heif_image.cc

// If I only add four R/G/B/A channel to `heif_image` without `heif_channel_interleaved`, this will return a nullptr
uint8_t* HeifPixelImage::get_plane(enum heif_channel channel, int* out_stride)
{
  auto iter = m_planes.find(channel);
  if (iter == m_planes.end()) {
    return nullptr;
  }

  if (out_stride) {
    *out_stride = iter->second.stride;
  }

  return iter->second.mem.data();
}

// This convert will call `get_plane`
std::shared_ptr<HeifPixelImage> HeifPixelImage::convert_RGB24_32_to_YCbCr420() const
{
//...
in_p  = get_plane(heif_channel_interleaved,  &in_stride);
// This `in_p` will become nullptr

  if (!has_alpha) {
    for (int y=0;y<m_height;y++) {
      for (int x=0;x<m_width;x++) {
        uint8_t r = in_p[y*in_stride + x*3 +0];   // <----------Crash here
        uint8_t g = in_p[y*in_stride + x*3 +1];
        uint8_t b = in_p[y*in_stride + x*3 +2];
        out_y[y*out_y_stride + x] = clip(r*0.299f + g*0.587f + b*0.114f);
      }
    }
}

But however, thanks for your reply. Now I can successful decoding/encoding HEIC in macOS & iOS platform without the dependency of Apple :)

And maybe another small question. Is there any public API to detect the current registered decoder/encoder plugin ? Sometime I need to check whether specify type is availble and then to provide the image data to decode.

You can list all encoders with heif_context_get_encoder_descriptors().

There is currently no way to check the available decoder codecs, you will simply get a heif_suberror_Unsupported_codec error when you try to decode an image that is not supported.
I am not sure whether it makes sense to provide such an API for the decoder, since you would have to parse the HEIF file yourself to see what format is used and ask libheif for decoder support.
If you have a convincing use-case for that, I might reconsider, though.

Updated the example code. heif_context_new_heic is not required anymore.

Any further point that from here that needs taken care of? Otherwise, I will close this issue.
BTW: in the future, please open a separate thread for each issue. That is much easier to track.

@farindk

@farindk Thanks for your reply. Now the above questions has been solved. I'll close this issue :)

@dreampiggy

@dreampiggy Hello, I'm encountering many issues while compiling and cannot successfully compile. My source code, toolchain, Xcode SDK, and macOS system are all up-to-date. I'm really at a loss on how to proceed with the compilation. Could you please share your compilation parameters with me? Thank you.

@dreampiggy

@dreampiggy Hello, I'm encountering many issues while compiling and cannot successfully compile. My source code, toolchain, Xcode SDK, and macOS system are all up-to-date. I'm really at a loss on how to proceed with the compilation. Could you please share your compilation parameters with me? Thank you.

Just be stupid and use my intergration with Xcode : https://github.com/SDWebImage/libheif-Xcode

This use C implementation on macOS x86_64, NEON on arm64, it's fast enough.

@dreampiggy

@dreampiggy Thank you for your response, but it seems the link you provided is for an iOS project? What I have here is a pure C++ project. The ultimate goal is to modify it and export it as an iOS dynamic library, and I'm not sure how to use it.

It's not only iOS.

It's macOS iOS tvOS watchOS or even visionOS, all Apple OS

You can just open that and compile to binary ( change the MACHO_LIB_TYPE to statclib if you need) and then integrate to you other project.

@dreampiggy

@dreampiggy First of all, thank you! I am not an iOS developer, and I am not familiar with Xcode either. I followed your instructions to clone and try to compile the project, but encountered an error. Additionally, what is MACHO_LIB_TYPE? I couldn't find this string throughout the entire project. This is the error pic:

MACHO_LIB_TYPE

For compile error, be sure you clone it with git submodule, or git submodule update
MACH_O_TYPE typo, by default it build dynamic lib, if you don't need static lib ignore this : https://stackoverflow.com/questions/13082149/in-xcode-project-target-build-settings-what-is-mach-o-type

@dreampiggy

@dreampiggy Thank you for your patient response. When building the necessary base project libde265, I have already modified the 'Mach-O Type' as you suggested, but the result is still a framework instead of the .a static library. Is there any other special setting needed? Additionally, if the .a library is successfully built, how can I use it in libheif-Xcode ？
This is my build result pic:

Framework is Package Way, not about how MachO binary is.

Means:

Static MachO + Framework Package == Static Framework
Dynamic MachO + Framework Package == Dynamic Framework
Static MachO + No Package == libXXX.a
Dynamic MachO + No Package == libXXX.dylib

You see that the MachO binary in .framework you can just rename it into libXXX.a, then use it.
You can even use file XXX.framework/XXX command to check the binary format

@dreampiggy

@dreampiggy You're welcome! I'm glad you learned something new.

but the result is still a framework instead of the .a static library. Is there any other special setting needed? Additionally, if the .a library is successfully built, how can I use it in libheif-Xcode ？

Because libheif dependency libde265, right ?

Now come to Package Manager problem, which is used to maintain the dependency.

That Xcodeproj, use Carthage as default package manager. You can also use Swift Package Manager as well.

For example, in Carthage, you do actuall this on your App repo (not even libheif or libde265), for example, I has a https://github.com/SDWebImage/SDWebImageHEIFCoder/blob/master/Cartfile

Create a Cartfile, contains the version you want

github "SDWebImage/libde265-Xcode" ~> 1.0.9
github "SDWebImage/libheif-Xcode" >= 1.6.0

Run carthage build, if you care about iOS only, run carthage build --platform iOS
The built .framework will inside the Carthage/Build folder, you will see libheif.framework, liblibde265.framework. But it's dynamic framework
If you need static framework (Then rename it with libXXX.a), you need some magic like here: https://github.com/Carthage/Carthage/blob/master/Documentation/StaticFrameworks.md

#!/bin/sh -e

xcconfig=$(mktemp /tmp/static.xcconfig.XXXXXX)
trap 'rm -f "$xcconfig"' INT TERM HUP EXIT
echo "MACH_O_TYPE = staticlib" >> $xcconfig

export XCODE_XCCONFIG_FILE="$xcconfig"

carthage build "$@"

You see this MACH_O_TYPE = staticlib do what's magic changes

And another way, use Swift Package Manager as replacement. Note Swift Package Manager Always use staticlib, no dynamic lib support...WTF

Just see what I did on shell terminal, re-produce this by yourself:

➜  ~ cd Desktop
➜  mkdir Test
➜  cd Test
➜  swift package init
Creating library package: Test
Creating Package.swift
Creating README.md
Creating .gitignore
Creating Sources/
Creating Sources/Test/Test.swift
Creating Tests/
Creating Tests/TestTests/
Creating Tests/TestTests/TestTests.swift

➜  cat Package.swift
// swift-tools-version: 5.7
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "Test",
    platforms: [.iOS(.v11)], // change here for you minOS support, like iOS 11+
    products: [
        // Products define the executables and libraries a package produces, and make them visible to other packages.
        .library(
            name: "Test",
            targets: ["Test"]),
    ],
    dependencies: [
        .package(url: "https://github.com/SDWebImage/libheif-Xcode.git", from: "1.6.1") // Change here for your version
    ],
    targets: [
        // Targets are the basic building blocks of a package. A target can define a module or a test suite.
        // Targets can depend on other targets in this package, and on products in packages this package depends on.
        .target(
            name: "Test",
            dependencies: [.product(name: "libheif", package: "libheif-Xcode")]),
        .testTarget(
            name: "TestTests",
            dependencies: ["Test"]),
    ]
)

➜  Test swift build -c release
Building for production...
[15/15] Compiling Test Test.swift
Build complete! (2.65s)

ls -lash .build

Or be stupid, you can double click on that libheif-Xcode repo's Package.swift

Then change the destination to Any iOS device, click Project -> Archive

The binary (not ar archive yet, but simple MachO Object file) is under temp folder, you can found it in Log, like here:

To create ar archive from MachO Object file, use libtool

libtool -static libde265.o -o libde265.a
libtool -static libheif.o -o libheif.a

@dreampiggy

@dreampiggy Thank you so much for your beginner-friendly guidance. I've successfully compiled it. Thanks again! Best wishes~~

@dreampiggy

@dreampiggy Thanks for helping out.

Navigation Menu

Repository navigation

Question about compile & usage on iOS/macOS platform #16

Readme

Compile

Usage

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Question about compile & usage on iOS/macOS platform #16

Description

Readme

Compile

Usage

Activity

dreampiggy commented on May 9, 2018

farindk commented on May 9, 2018

dreampiggy commented on May 9, 2018

farindk commented on May 9, 2018

dreampiggy commented on May 9, 2018

farindk commented on May 9, 2018

dreampiggy commented on May 9, 2018

dreampiggy commented on May 9, 2018

dreampiggy commented on May 9, 2018

dreampiggy commented on May 10, 2018

farindk commented on May 10, 2018

farindk commented on May 22, 2018

dreampiggy commented on May 23, 2018

parcool commented on Jul 24, 2023

dreampiggy commented on Jul 25, 2023

parcool commented on Jul 25, 2023

dreampiggy commented on Jul 25, 2023

parcool commented on Jul 25, 2023

dreampiggy commented on Jul 25, 2023

parcool commented on Jul 26, 2023

dreampiggy commented on Jul 26, 2023

parcool commented on Jul 26, 2023

dreampiggy commented on Jul 26, 2023

dreampiggy commented on Jul 26, 2023

dreampiggy commented on Jul 26, 2023

parcool commented on Jul 26, 2023

farindk commented on Jul 26, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions