Skip to content

/ onnxruntime

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

onnxruntime_test_all --gtest_filter=ActivationOpTest.Relu fails, cross-compile onnxruntime v1.3.1 against ACL v19.08 #4463

Closed
dhunt-ambarella opened this issue Jul 9, 2020 · 9 comments
Closed

onnxruntime_test_all --gtest_filter=ActivationOpTest.Relu fails, cross-compile onnxruntime v1.3.1 against ACL v19.08 #4463

dhunt-ambarella opened this issue Jul 9, 2020 · 9 comments
Assignees
@prabhat00155
Labels
ep:ACL type:bug

Comments

@dhunt-ambarella
Copy link

@dhunt-ambarella dhunt-ambarella commented Jul 9, 2020

Note: Google Test filter = ActivationOpTest.Relu
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from ActivationOpTest
[ RUN ] ActivationOpTest.Relu
free(): invalid pointer
Aborted (core dumped)

All tests preceding this test (hundreds) are OK. Some tests after fail also.

Also, but possibly not related:

onnxruntime_perf_test -e acl testdata/squeezenet/model.onnx result.csv

free(): invalid pointer
Aborted (core dumped)

onnxruntime v1.3.1 cross compiled with linaro-gcc8.2.

In onnxruntime, I changed these files:

onnxruntime/core/providers/acl/acl_execution_provider.cc

onnxruntime/core/providers/acl/acl_common.cc

onnxruntime/core/providers/acl/nn/conv.cc

In onnxruntime/core/providers/acl/acl_execution_provider.cc, pretend to use the return value from Register():

  • Status ret_val;
  • ret_val =
    kernel_registry.Register(BuildKernelCreateInfo<ONNX_OPERATOR_KERNEL_CLASS_NAME(kAclExecutionProvider, kOnnxDomain, 6, Relu)>());
  • ret_val =
    kernel_registry.Register(BuildKernelCreateInfo<ONNX_OPERATOR_VERSIONED_KERNEL_CLASS_NAME(kAclExecutionProvider, kOnnxDomain, 7, 9, Gemm)>());

In onnxruntime/core/providers/acl/acl_common.cc, fixed an unused-variable warning:

--- a/onnxruntime/core/providers/acl/acl_common.cc
+++ b/onnxruntime/core/providers/acl/acl_common.cc
@@ -70,7 +70,8 @@ void importDataFromTensor(arm_compute::Tensor* tensor, T* data){
// copy input tensor into the larger buffer
arm_compute::execute_window_loop(
aclInpuWindow,

  •  [&](const arm_compute::Coordinates& co) {

+// [&](const arm_compute::Coordinates& co) {

  •  [&](const arm_compute::Coordinates&) {

In onnxruntime/core/providers/acl/nn/conv.cc, fixed 'used uninitialized' error:

  • arm_compute::ActivationLayerInfo::ActivationFunction acl_activ_func;
  • arm_compute::ActivationLayerInfo::ActivationFunction acl_activ_func = arm_compute::ActivationLayerInfo::ActivationFunction::RELU;

Also, to compile and link, there were changes to my environment.

In .bashrc, I added ACL include files to CPATH:

export ACL_VER="19.08"

export CPATH="/dump/dump201/dhunt/arm_compute-v${ACL_VER}-bin-linux/include:/dump/dump201/dhunt/arm_compute-v${ACL_VER}-bin-linux:$CPATH"

And for linking, I added this to environment:

export LD_RUN_PATH="/dump/dump201/dhunt/arm_compute-v${ACL_VER}-bin-linux/lib/linux-arm64-v8a-neon/"
export LD_LIBRARY_PATH="/dump/dump201/dhunt/arm_compute-v${ACL_VER}-bin-linux/lib/linux-arm64-v8a-neon/"

I provided soft links to the ACL libraries so onnxruntime could find them.

dhunt@ubu-sw4:/dump/dump201/dhunt/onnxruntime$ ln -s /dump/dump201/dhunt/arm_compute-v19.08-bin-linux/lib/linux-arm64-v8a-neon/libarm_compute.so libarm_compute.so

dhunt@ubu-sw4:/dump/dump201/dhunt/onnxruntime$ ln -s /dump/dump201/dhunt/arm_compute-v19.08-bin-linux/lib/linux-arm64-v8a-neon/libarm_compute_core.so libarm_compute_core.so

dhunt@ubu-sw4:/dump/dump201/dhunt/onnxruntime$ ln -s /dump/dump201/dhunt/arm_compute-v19.08-bin-linux/lib/linux-arm64-v8a-neon/libarm_compute_graph.so libarm_compute_graph.so

I used this cmake command, and 'make -j 6' to build:

cmake -DCMAKE_TOOLCHAIN_FILE=/dump/dump201/dhunt/tool.cmake -Donnxruntime_CROSS_COMPILING=ON -Donnxruntime_RUN_ONNX_TESTS=OFF -Donnxruntime_DEV_MODE=ON -DPYTHON_EXECUTABLE=/opt/amba/cv2_third/tv2/exe/python3 -Donnxruntime_USE_CUDA=OFF -Donnxruntime_USE_NSYNC=OFF -Donnxruntime_CUDNN_HOME= -Donnxruntime_USE_JEMALLOC=OFF -Donnxruntime_ENABLE_PYTHON=OFF -Donnxruntime_BUILD_CSHARP=OFF -Donnxruntime_BUILD_SHARED_LIB=ON -Donnxruntime_USE_EIGEN_FOR_BLAS=ON -Donnxruntime_USE_OPENBLAS=OFF -Donnxruntime_USE_DNNL=OFF -Donnxruntime_USE_MKLML=OFF -Donnxruntime_USE_OPENMP=ON -Donnxruntime_USE_TVM=OFF -Donnxruntime_USE_LLVM=OFF -Donnxruntime_ENABLE_MICROSOFT_INTERNAL=OFF -Donnxruntime_USE_BRAINSLICE=OFF -Donnxruntime_USE_NUPHAR=OFF -Donnxruntime_USE_EIGEN_THREADPOOL=OFF -Donnxruntime_BUILD_UNIT_TESTS=ON -DCMAKE_BUILD_TYPE=RelWithDebInfo -Dprotobuf_WITH_ZLIB=OFF -DONNX_CUSTOM_PROTOC_EXECUTABLE=/dump/dump201/dhunt/protoc_3.11.3/bin/protoc -DCMAKE_VERBOSE_MAKEFILE=ON -Donnxruntime_USE_ACL=ON -Donnxruntime_USE_ACL_1902=OFF -Donnxruntime_USE_ACL_1905=OFF -Donnxruntime_USE_ACL_1908=ON -Donnxruntime_USE_ACL_2005=OFF cmake

cat tool.cmake

SET(CMAKE_SYSTEM_NAME Linux)
 SET(CMAKE_SYSTEM_VERSION 1)
 SET(CMAKE_C_COMPILER /usr/local/linaro-aarch64-2018.08-gcc8.2/bin/aarch64-linux-gnu-gcc)
 SET(CMAKE_CXX_COMPILER /usr/local/linaro-aarch64-2018.08-gcc8.2/bin/aarch64-linux-gnu-g++)

After all that, the compile and link succeed.
@hariharans29 hariharans29 added the ep:ACL label Jul 9, 2020
@askhade askhade added the type:bug label Jul 10, 2020
@prabhat00155
Copy link
Member

@prabhat00155 prabhat00155 commented Jul 14, 2020 •
edited

Looks like import_memory()

onnxruntime/onnxruntime/core/providers/acl/acl_common.cc

Line 58 in ee5ca27

return allocator->import_memory(memory);
(called from

onnxruntime/onnxruntime/core/providers/acl/activation/activations.cc

Line 34 in ee5ca27

ACLImportMemory(in.allocator(), (void*)src_data, X->Shape().Size() * 4);
) is causing this issue.

import_memory() is defined in ACL compute library(https://github.com/ARM-software/ComputeLibrary/blob/6a7771e460abeac7d401d6d38a0fcf0a0d2c3cbe/src/runtime/TensorAllocator.cpp#L154), could be an issue with the way it's getting freed in the class.
@jywu-msft
Copy link
Member

@jywu-msft jywu-msft commented Jul 15, 2020

@Andrews548, @mikecaraman any ideas what could be going wrong here?
@jywu-msft
Copy link
Member

@jywu-msft jywu-msft commented Jul 15, 2020

maybe this is the reason?
ARM-software/ComputeLibrary#744 (comment)

are you building ACL from source or using pre-built binaries?
@prabhat00155
Copy link
Member

@prabhat00155 prabhat00155 commented Jul 15, 2020

This works when ACL is built from source(just tried it). You only run into error when using pre-built binaries.
@dhunt-ambarella
Copy link
Author

@dhunt-ambarella dhunt-ambarella commented Jul 15, 2020

Yes. Using prebuilt libraries.

I will ask my administrator to install scons.

Any chance someone can share the libraries that will work?

Also, I use v8a-neon libraries. Not sure how to know which ones I should be using:
dhunt@ubu-sw4:/dump/dump201/dhunt/arm_compute-v19.08-bin-linux/lib$ ls -l
total 168
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-cl
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-cl-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-cl-debug
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-cl-gles
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-cl-gles-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-cl-gles-debug
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-gles
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-gles-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-gles-debug
drwxrws--- 2 dhunt engr3 4096 Jul 3 12:42 linux-arm64-v8a-neon
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-cl
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-cl-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-cl-debug
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-cl-gles
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-cl-gles-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-cl-gles-debug
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-debug
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-gles
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-gles-asserts
drwxrws--- 2 dhunt engr3 4096 Sep 2 2019 linux-arm64-v8a-neon-gles-debug
@prabhat00155
Copy link
Member

@prabhat00155 prabhat00155 commented Jul 16, 2020

I did a native build of ACL library, so it may not work on a different type of machine. You can use ACL 19.08 with onnxruntime using the following steps:

  1. Download the ACL v19.08 source code from here: https://github.com/ARM-software/ComputeLibrary/releases/tag/v19.08 and extract it:
wget https://github.com/ARM-software/ComputeLibrary/archive/v19.08.tar.gz
gunzip v19.08.tar.gz
tar -xvf v19.08.tar
  1. Build ACL:
cd ComputeLibrary-19.08/
sudo apt install scons
sudo apt install g++-arm-linux-gnueabihf
scons -j8 arch=arm64-v8a  Werror=1 debug=0 asserts=0 neon=1 opencl=1 examples=1 build=native
  1. Set environment variables(assuming you are under ComputeLibrary-19.08):
export CPATH=`pwd`/include/:`pwd`
export LD_LIBRARY_PATH=`pwd`/build/
  1. Building onnxruntime with ACL 19.08:
./build.sh --config Release --use_acl ACL_1908
  1. Run onnxruntime_perf_test with acl:
./onnxruntime_perf_test -x 4 -r times -n 1 -e acl ~/Documents/models/onnx/squeezenet1.1/squeezenet1.1.onnx result
@stale
Copy link

@stale stale bot commented Sep 14, 2020

This issue has been automatically marked as stale due to inactivity and will be closed in 7 days if no further activity occurs. If further support is needed, please provide an update and/or more details.
@stale stale bot added the stale label Sep 14, 2020
@Dan5002
Copy link

@Dan5002 Dan5002 commented Sep 14, 2020

Native build of ACL libraries was the fix.
@stale stale bot removed the stale label Sep 14, 2020
@Dan5002
Copy link

@Dan5002 Dan5002 commented Sep 14, 2020

Seems I can't close this myself. It can be closed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@prabhat00155 prabhat00155

Labels
ep:ACL type:bug
Projects
None yet
Milestone
No milestone
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
6 participants
@askhade @prabhat00155 @hariharans29 @jywu-msft @dhunt-ambarella @Dan5002
You can’t perform that action at this time.