Conda Install Osx

  1. Conda Install Xz
  2. Conda Install Osx Software

Anaconda 5.0 switched from OS-provided compiler tools to our own toolsets. Thisallows improved compiler capabilities, including better security andperformance. This page describes how to use these tools and enable thesebenefits.

Before Anaconda 5.0, compilers were installed using system tools such as XCode or yum install gcc. Now there are conda packages for Linux and macOS compilers. Unlike the previous GCC 4.8.5 packages that included GCC, g, and GFortran all in the same package, these conda packages are split into separate compilers: macOS: clangosx-64.

Any Linux, Mac OS X, or Windows computer with a web browser (preferably Google Chrome) should be suitable. 16GB of RAM is required for some of the examples, but most will run fine in 4GB. If you don’t already have conda on your machine, you can get it from conda.io, and then open a terminal window with the conda environment activated. To install additional conda packages, it is best to recreate the environment. Store conda and pip requirements in text files. Package requirements can be passed to conda via the -file argument. Pip accepts a list of Python packages with -r or -requirements. Conda env will export or create environments based on a file with conda and pip.

Compiler packages¶

Before Anaconda 5.0, compilers were installed using system tools such as XCodeor yuminstallgcc. Now there are conda packages for Linux and macOScompilers. Unlike the previous GCC 4.8.5 packages that included GCC, g++, andGFortran all in the same package, these conda packages are split into separatecompilers:

macOS:

  • clang_osx-64.

  • clangxx_osx-64.

  • gfortran_osx-64.

Linux:

  • gcc_linux-64.

  • gxx_linux-64.

  • gfortran_linux-64.

A compiler's 'build platform' is the platform where the compiler runs andbuilds the code.

A compiler's 'host platform' is the platform where the built code will finallybe hosted and run.

Notice that all of these package names end in a platform identifier whichspecifies the host platform. All compiler packages are specific to both thebuild platform and the host platform.

Using the compiler packages¶

The compiler packages can be installed with conda. Because they are designedwith (pseudo) cross-compiling in mind, all of the executables in a compilerpackage are 'prefixed.' Instead of gcc, the executable name of the compileryou use will be something like x86_64-conda_cos6-linux-gnu-gcc. These fullcompiler names are shown in the build logs, recording the host platform andhelping prevent the common mistake of using the wrong compiler.

Many build tools such as make and CMake search by default for acompiler named simply gcc, so we set environment variables to point thesetools to the correct compiler.

We set these variables in conda activate.d scripts, so any environment inwhich you will use the compilers must first be activated so the scripts willrun. Conda-build does this activation for you using activation hooks installedwith the compiler packages in CONDA_PREFIX/etc/conda/activate.d, so noadditional effort is necessary.

You can activate the root environment with the command condaactivateroot.

macOS SDK¶

The macOS compilers require the macOS 10.9 SDK or above. The SDK license preventsit from being bundled in the conda package. We know of 2 current sources for themacOS SDKs:

We usually install the 10.10 SDK at /opt/MacOSX10.10.sdk but you may installit anywhere. Edit your conda_build_config.yaml file to point to it, like this:

At Anaconda, we have this configuration setting in a centralizedconda_build_config.yaml at the root of our recipe repository. Since we runbuild commands from that location, the file and the setting are used for allrecipes. The conda_build_config.yaml search order is described further atCreating conda-build variant config files.

Build scripts for macOS should make use of the variablesMACOSX_DEPLOYMENT_TARGET and CONDA_BUILD_SYSROOT, which are set byconda-build (see Environment variables). These variables should be translated intocorrect compiler arguments, e.g. for Clang this would be:

Most build tools, e.g. CMake and distutils (setuptools), will automatically pickup MACOSX_DEPLOYMENT_TARGET but you need to pass CONDA_BUILD_SYSROOTexplicitly. For CMake, this can be done with the option-DCMAKE_OSX_SYSROOT=${CONDA_BUILD_SYSROOT}. When building Python extensionswith distutils, one should always extend CFLAGS before callingsetup.py:

When building C++ extensions with Cython, CXXFLAGS must be similarly modified.

Backward compatibility¶

Some users want to use the latest Anaconda packages but do not yet want to usethe Anaconda compilers. To enable this, the latest Python package builds havea default _sysconfigdata file. This file sets the compilers provided by thesystem, such as gcc and g++, as the default compilers. This way allows legacyrecipes to keep working.

Python packages also include an alternative _sysconfigdata file that setsthe Anaconda compilers as the default compilers. The Anaconda Python executableitself is made with these Anaconda compilers.

The compiler packages set the environment variable_PYTHON_SYSCONFIGDATA_NAME, which tells Python which _sysconfigdata fileto use. This variable is set at activation time using the activation hooksdescribed above.

The new _sysconfigdata customization system is only present in recentversions of the Python package. Conda-build automatically tries to use thelatest Python version available in the currently configured channels, whichnormally gets the latest from the default channel. If you're using somethingother than conda-build while working with the new compilers, conda does notautomatically update Python, so make sure you have the correct_sysconfigdata files by updating your Python package manually.

Anaconda compilers and conda-build 3¶

The Anaconda 5.0 compilers and conda-build 3 are designed to work together.

Conda-build 3 defines a special jinja2 function, compiler(), to make iteasy to specify compiler packages dynamically on many platforms. Thecompiler function takes at least 1 argument, the language of the compilerto use:

'Cross-capable' recipes can be used to make packages with a host platformdifferent than the build platform where conda-build runs. To writecross-capable recipes, you may also need to use the 'host' section in therequirements section. In this example we set 'host' to 'zlib' to tellconda-build to use the zlib in the conda environment and not the systemzlib. This makes sure conda-build uses the zlib for the host platformand not the zlib for the build platform.

Generally, the build section should include compilers and other build tools andthe host section should include everything else, including shared libraries,Python, and Python libraries.

An aside on CMake and sysroots¶

Anaconda's compilers for Linux are built with something called crosstool-ng.They include not only GCC, but also a 'sysroot' with glibc, as well as the restof the toolchain (binutils). Ordinarily, the sysroot is something that yoursystem provides, and it is what establishes the libc compatibility bound foryour compiled code. Any compilation that uses a sysroot other than the systemsysroot is said to be 'cross-compiling.' When the target OS and the build OSare the same, it is called a 'pseudo-cross-compiler.' This is the case fornormal builds with Anaconda's compilers on Linux.

Unfortunately, some software tools do not handle sysroots in intuitive ways.CMake is especially bad for this. Even though the compiler itself understandsits own sysroot, CMake insists on ignoring that. We've filed issues at:

Additionally, this Stack Overflow issue has some more information: https://stackoverflow.com/questions/36195791/cmake-missing-sysroot-when-cross-compiling

In order to teach CMake about the sysroot, you must do additional work. As anexample, please see our recipe for libnetcdf athttps://github.com/AnacondaRecipes/libnetcdf-feedstock/tree/master/recipe

In particular, you'll need to copy the cross-linux.cmake file there, and reference it in your build.sh file:

Customizing the compilers¶

Conda Install Xz

The compiler packages listed above are small packages that only include theactivation scripts and list most of the software they provide as runtimedependencies.

This design is intended to make it easy for you to customize your own compilerpackages by copying these recipes and changing the flags. You can then edit theconda_build_config.yaml file to specify your own packages.

Conda Install Osx Software

We have been careful to select good, general purpose, secure, and fast flags.We have also used them for all packages in Anaconda Distribution 5.0.0, exceptfor some minor customizations in a few recipes. When changing these flags,remember that choosing the wrong flags can reduce security, reduce performance,and cause incompatibilities.

With that warning in mind, let's look at good ways to customize Clang.

  1. Download or fork the code from https://github.com/anacondarecipes/aggregate.The Clang package recipe is in the clang folder. The main material is in thellvm-compilers-feedstock folder.

  2. Edit clang/recipe/meta.yaml:

    The name here does not matter but the output names below do. Conda-buildexpects any compiler to follow the BASENAME_PLATFORMNAME pattern, so it isimportant to keep the {{target_platform}} part of the name.

    {{version}} is left as an intentionally undefined jinja2 variable. Itis set later in conda_build_config.yaml.

  3. Before any packaging is done, run the build.sh script:https://github.com/AnacondaRecipes/aggregate/blob/master/clang/build.sh

    In this recipe, values are changed here. Those values are inserted into theactivate scripts that are installed later.

  4. With those changes to the activate scripts in place, it's time to move on toinstalling things. Look back at the clang folder's meta.yaml. Here'swhere we change the package name. Notice what comes before the{{target_platform}}.

    The script reference here is another place you might add customization.You'll either change the contents of those install scripts or change thescripts that those install scripts are installing.

    Note that we make the package clang in the main material agree in versionwith our output version. This is implicitly the same as the top-levelrecipe. The clang package sets no environment variables at all, so itmay be difficult to use directly.

  5. Let's examine the script install-clang.sh:

    Nothing here is too unusual.

    Activate scripts are named according to our package name so they won'tconflict with other activate scripts.

    The symlink for Clang is a Clang implementation detail that sets the hostplatform.

    We define macos_machine in aggregate's conda_build_config.yaml:https://github.com/AnacondaRecipes/aggregate/blob/master/conda_build_config.yaml#L79

    The activate scripts that are being installed are where we actually set theenvironment variables. Remember that these have been modified by build.sh.

  6. With any of your desired changes in place, go ahead and build the recipe.

    You should end up with a super_duper_clang_osx-64 package. Or, if you're noton macOS and are modifying a different recipe, you should end up with anequivalent package for your platform.

Using your customized compiler package with conda-build 3¶

Remember the Jinja2 function, {{compiler('c')}}? Here's where that comesin. Specific keys in conda_build_config.yaml are named for the languageargument to that jinja2 function. In your conda_build_config.yaml, addthis:

Note that we're not adding the target_platform part, which is separate. Youcan define that key, too:

With those two keys defined, conda-build will try to use a compiler packagenamed super_duper_clang_win-64. That package needs to exist for your nativeplatform. For example, if you're on macOS, your native platform is osx-64.

The package subdirectory for your native platform is the build platform. Thebuild platform and the target_platform can be the same, and they are thesame by default, but they can also be different. When they are different,you're cross-compiling.

If you ever needed a different compiler key for the same language, rememberthat the language key is arbitrary. For example, we might want differentcompilers for Python and for R within one ecosystem. On Windows, the Pythonecosystem uses the Microsoft Visual C compilers, while the R ecosystem uses theMingw compilers.

Let's start in conda_build_config.yaml:

In Python recipes, you'd have:

In R recipes, you'd have:

This example is a little contrived, because the m2w64-gcc_win-64 package isnot available. You'd need to create a metapackage m2w64-gcc_win-64 topoint at the m2w64-gcc package, which does exist on the msys2 channel onrepo.anaconda.com.

Anaconda compilers implicitly add RPATH pointing to the conda environment¶

You might want to use the Anaconda compilers outside of conda-buildso that you use the same versions, flags, and configuration, for maximumcompatibility with Anaconda packages (but in a case where you want simpletarballs, for example). In this case, there is a gotcha.

Even if Anaconda compilers are used from outside of conda-build, the GCCspecs are customized so that, when linking an executable or a shared library,an RPATH pointing to lib/ inside the current enviroment prefix directory($CONDA_PREFIX/lib) is added. This is done by changing thelink_libgcc: section inside GCC specs file, and this change is doneso that LD_LIBRARY_PATH isn't required for basic libraries.

conda-build knows how to make this automatically relocatable, so thatthis RPATH will be changed to point to the environment where the packageis being installed (at installation time, by conda). But if you only packthis binary in a tarball, it will continue containing this hardcoded RPATHto an environment in your machine. In this case, it is recommended to manuallyremove the RPATH.

Using Conda

Conda is a package manager for Python, CPP and other packages.

Currently, SINGA has conda packages for Linux and MacOSX.Miniconda3 is recommended to use with SINGA.After installing miniconda, execute the one of the following commands to installSINGA.

It is preferable to use python 3.6 as the conda virtual environment.

  1. CPU only
Conda

Please note that we need to specify the version of singa-cpu during the installation.

  1. GPU with CUDA and cuDNN (CUDA driver >=384.81 is required)

Please note that we need to specify the version of singa-gpu during the installation.

  1. Install a specific version of SINGA. The following command lists all theavailable SINGA packages.

The following command installs a specific version of SINGA,

If there is no error message from

then SINGA is installed successfully.

Using Pip

  1. CPU only

You can install a specific version of SINGA via singa<version>, where the<version> field should be replaced, e.g., 3.2.0. The available SINGAversions are listed at the link.

To install the latest develop version, replace the link withhttp://singa.apache.org/docs/next/wheel-cpu-dev.html

  1. GPU With CUDA and cuDNN

You can also configure SINGA version and the CUDA version, likesinga3.2.0+cuda10.2. The available combinations of SINGA version and CUDAversion are listed at the link.

To install the latest develop version, replace the link withhttp://singa.apache.org/docs/next/wheel-gpu-dev.html

Note: the Python version of your local Python environment will be used to findthe corresponding wheel package. For example, if your local Python is 3.6, thenthe wheel package compiled on Python 3.6 will be selected by pip and installed.In fact, the wheel file's name include SINGA version, CUDA version and Pythonversion. Therefore, pip knows which wheel file to download and install.

Refer to the comments at the top of the setup.py file for how to build thewheel packages.

Using Docker

Install Docker on your local host machine following theinstructions. Add your user into thedocker group to rundocker commands without sudo.

  1. CPU-only.
  1. With GPU enabled. InstallNvidia-Docker after installDocker.
  1. For the complete list of SINGA Docker images (tags), visit thedocker hub site. For each dockerimage, the tag is named as
TagDescriptionExample value
versionSINGA version'2.0.0-rc0', '2.0.0', '1.2.0'
cputhe image cannot run on GPUs'cpu'
gputhe image can run on Nvidia GPUs'gpu', or 'cudax.x-cudnnx.x' e.g., 'cuda10.0-cudnn7.3'
develindicator for developmentif absent, SINGA Python package is installed for runtime only; if present, the building environment is also created, you can recompile SINGA from source at '/root/singa'
OSindicate OS version number'ubuntu16.04', 'ubuntu18.04'

From source

You can build and install SINGA from the source code using nativebuilding tools or conda-build, on local host OS or in a Docker container.

FAQ

  • Q: Error from from singa import tensor

    A: Check the detailed error from

    The folder of _singa_wrap.so is like~/miniconda3/lib/python3.7/site-packages/singa. Normally, the error iscaused by the mismatch or missing of dependent libraries, e.g. cuDNN orprotobuf. The solution is to create a new virtual environment and installSINGA in that environment, e.g.,

  • Q: When using virtual environment, every time I install SINGA, numpy would bereinstalled. However, the numpy is not used when I run import numpy

    A: It could be caused by the PYTHONPATH environment variable which should beset to empty when you are using virtual environment to avoid the conflictswith the path of the virtual environment.

  • Q: When I run SINGA in Mac OS X, I got the error 'Fatal Python error:PyThreadState_Get: no current thread Abort trap: 6'

    A: This error happens typically when you have multiple versions of Python inyour system, e.g, the one comes with the OS and the one installed by Homebrew.The Python linked by SINGA must be the same as the Python interpreter. You cancheck your interpreter by which python and check the Python linked by SINGAvia otool -L <path to _singa_wrap.so>. This problem should be resolved ifSINGA is installed via conda.