Some notes about building Caffe RC3 with Mac OS X 10.11.3, Anaconda, CUDA 7.5, cuDNN 4, Intel MKL and MATLAB R2015b

Environment: Mac OS X 10.11.3, Xcode 7.2, Anaconda Python 2.7.11, CUDA 7.5, cuDNN 4, Intel parallel_studio_xe_2016.1.043 mkl, homebrew boost 1.6.0, homebrew OpenCV 2.4.12, MATLAB R2015b.

Note 1: If your run into this problem afterwards "ld: warning: directory not found for option '-L/opt/intel/mkl/lib/intel64'", you can solve this problem as:
cd /opt/intel/mkl/lib/
sudo ln -s . /opt/intel/mkl/lib/intel64
Suppose the environment is setup as above, first follow the official Caffe OS X Installation guide step by step.

Then modify your .bash_profile file as:
export PATH=/usr/local/bin:/Applications/MATLAB_R2015b.app/bin:/Developer/NVIDIA/CUDA-7.5/bin:$PATH
export PATH="/Users/ylzhao/anaconda/bin:$PATH"
export DYLD_LIBRARY_PATH=/Developer/NVIDIA/CUDA-7.5/lib:$DYLD_LIBRARY_PATH
export DYLD_FALLBACK_LIBRARY_PATH=/usr/local/cuda/lib::$HOME/anaconda/lib:/usr/local/lib:/usr/lib:$DYLD_FALLBACK_LIBRARY_PATH
export PYTHONPATH=/usr/local/lib/python2.7/site-packages:$PYTHONPATH
Note 2: If you run into this problem when invoking "make runtest"afterwards:
dyld: Library not loaded: @rpath/libcudart.7.0.dylib
Referenced from: .../caffe/.build_release/tools/caffe
Reason: image not found
make: *** [runtest] Trace/BPT trap: 5
That is because on Mac OS X El Capitan, DYLD_FALLBACK_LIBRARY_PATH is cleared by the new System Integrity Protection feature of El Capitan. See the discussions 2320 and 3628. One possible way to solve this problem is disabled System Integrity Protection as:
    1. Boot to Recovery OS by restarting your machine and holding down the Command and R keys at startup.
    2. Launch Terminal from the Utilities menu.
    3. Enter the following command: csrutil disable
    4. restart your computer
     Then follow the official Caffe compilation with Make guide, my 'Makefile.config' file is:
    ## Refer to http://caffe.berkeleyvision.org/installation.html
    # Contributions simplifying and improving our build system are welcome!
    # cuDNN acceleration switch (uncomment to build with cuDNN).
    USE_CUDNN := 1
    # CUDA directory contains bin/ and lib/ directories that we need.
    CUDA_DIR := /usr/local/cuda
    # BLAS choice:
    # atlas for ATLAS (default)
    # mkl for MKL
    # open for OpenBlas
    BLAS := mkl

    # This is required only if you will compile the matlab interface.
    # MATLAB directory should contain the mex binary in /bin.
    # MATLAB_DIR := /usr/local
    MATLAB_DIR := /Applications/MATLAB_R2015b.app

    # NOTE: this is required only if you will compile the python interface.
    # We need to be able to find Python.h and numpy/arrayobject.h.
    # PYTHON_INCLUDE := /usr/include/python2.7 \
    # /usr/lib/python2.7/dist-packages/numpy/core/include
    # Anaconda Python distribution is quite popular. Include path:
    # Verify anaconda location, sometimes it's in root.
    ANACONDA_HOME := $(HOME)/anaconda
    PYTHON_INCLUDE := $(ANACONDA_HOME)/include \
    $(ANACONDA_HOME)/include/python2.7 \
    $(ANACONDA_HOME)/lib/python2.7/site-packages/numpy/core/include \

    # We need to be able to find libpythonX.X.so or .dylib.
    # PYTHON_LIB := /usr/lib
    PYTHON_LIB := $(ANACONDA_HOME)/lib

    # Uncomment to support layers written in Python (will link against Python libs)
    # WITH_PYTHON_LAYER := 1

    # Whatever else you find you need goes here.
    INCLUDE_DIRS := $(PYTHON_INCLUDE) /usr/local/include
    LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
    My modification parts are labelled as bold.

    Then you can start the compilation in Terminal:
    make all
    make test
    make runtest
    Note 3: If your "LIBRARY_DIRS" like these:
    LIBRARY_DIRS := $(PYTHON_LIB) $(MATLAB_DIR)/bin/maci64 usr/local/lib  /usr/lib
    you may run into this protobuf problem afterwards:
    Undefined symbols for architecture x86_64:
     "google::protobuf::io::CodedOutputStream::WriteStringWithSizeToArray(std::__1::basic_string, std::__1::allocator > const&, unsigned char*)", referenced from:
    caffe::Datum::SerializeWithCachedSizesToArray(unsigned char*) const in caffe.pb.o
    This is because Caffe tries to link with MATLAB's protobuf instead of homebrew protobuf, and this will also happen with the boost library and CUDA. (More detail: Caffe was seeing Matlab's internal libraries earlier in the library path than the local homebrew libraries in /usr/local/lib. Caffe might be trying to link Matlab's version of the boost library, which, needless to say, isn't compatible with Caffe.) See discussion 915.

    So before compile the Caffe Matlab interface, your "LIBRARY_DIRS" should like:
    LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib /usr/lib
    After the commands "make all", "make test", "make runtest" successfully, you should modify the "LIBRARY_DIRS" like:
    LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib $(MATLAB_DIR)/bin/maci64 /usr/lib
    Thus we can avoid the conflict between homebrew's protobuf/boost/cuda libraries and MATLAB's protobuf/boost/cuda libraries.

    Then you can build the Caffe Matlab interface as:
    make matcaffe
    make mattest 
    Note 4: If you forget to add "(MATLAB_DIR)/bin/maci64" in "LIBRARY_DIRS" line, you may run into this problem:
    Undefined symbols for architecture x86_64:
      "_mxArrayToString", referenced from:
          get_solver(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          solver_restore(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          get_net(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          net_copy_from(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          net_save(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          read_mean(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          write_mean(int, mxArray_tag**, int, mxArray_tag const**) in caffe_.o
          ...
    That why we need to change the LIBRARY_DIRS := ... line in your Makefile.config so that the /usr/local/lib directory is before your Matlab library directory as above described.

    Note 5: when you compile Caffe Python interface like:
    make pycaffe
    make pytest 
    you may run into this problem:
    ======================================================================
    ERROR: test_forward_backward (test_net.TestNet)
    ----------------------------------------------------------------------
    Traceback (most recent call last):
      File "/Users/ylzhao/Project/caffe/python/caffe/test/test_net.py", line 45, in setUp
        size=self.net.blobs['label'].data.shape)
      File "caffe/pycaffe.py", line 28, in _Net_blobs
        return OrderedDict(zip(self._blob_names, self._blobs))
    TypeError: No to_python (by-value) converter found for C++ type: boost::shared_ptr >
    It seems that this problem is related to Boost 1.60, not 1.59. You can find the discussion at 3494 and 3575, and the fix is at 3575. The commit you want to apply is at 3575 commit.

    Unfortunately, this commit has not been merged into Caffe master branch, so you should do the merge by hand.

    That's all, have fun with Caffe building!

    评论

    Awe-inspiring bequest! Your site is actually attention-grabbing. Personally i think love for this.
    mobile apps developers USA
    Unknown说…
    hello, it‘s very helpful.
    but when I got to make matcaffe
    I follow exactly LIBRARY_DIRS := $(PYTHON_LIB) /usr/local/lib $(MATLAB_DIR)/bin/maci64 /usr/lib,but got the same error , do your have any idea?
    thanks a lot

    此博客中的热门博文

    BLAS & LAPACK for Windows

    从特征描述符到深度学习:计算机视觉发展20年

    One note for building Caffe RC3 on Ubuntu 14.04