fuchencong

The way I am

0%

Envoy ubuntu 22.04 编译

发表于 分类于

Envoy 的编译还是比较复杂的，这里将在 ubuntu:22.04 上构建 Envoy。

准备源码

这里我们使用 Envoy 最新的 tag v1.24.1：

1
2
3
# git clone https://github.com/envoyproxy/envoy.git
# cd envoy
# git checkout -b v1.24.1 v1.24.1

准备构建环境

下载 Docker 镜像

1
# docker pull ubuntu:22.04

运行 Docker 容器

1
2
# docker run  -d -it --name envoy_build --network="host" -v $PWD:$PWD ubuntu:22.04  /bin/bash
# docker exec -it envoy_build /bin/bash

安装基础的构建工具

1
2
3
4
5
6
7
8
9
10
11
12
13
14
# apt-get -y install \
   autoconf \
   automake \
   cmake \
   curl \
   libtool \
   make \
   ninja-build \
   patch \
   python3-pip \
   unzip \
   virtualenv \
   git \
   wget

安装 Bazelisk

Envoy 使用 Bazel 作为构建系统。官方推荐使用 Bazelisk 作为 bazel

1
2
# wget -O /usr/local/bin/bazel https://github.com/bazelbuild/bazelisk/releases/latest/download/bazelisk-linux-$([ $(uname -m) = "aarch64" ] && echo "arm64" || echo "amd64")
# chmod +x /usr/local/bin/bazel

编译

开始编译

到这一步就可以开始编译了，此时默认将使用 gcc 编译：

1
# bazel build --jobs=2 --copt=-Wno-error=vla-parameter envoy

生成交付件

在编译成功后，生成的交付件位于 bazel-bin/source/exe/envoy-static。执行如下命令，确认编译成功。

1
2
3
# bazel-bin/source/exe/envoy-static --version

bazel-bin/source/exe/envoy-static  version: 69958e4fe32da561376d8b1d367b5e6942dfba24/1.24.1/Clean/DEBUG/BoringSSL

bazel-bin/ 目录其实是一个符号连接，它连接到 $(bazel info bazel-genfiles) 目录：

1
2
3
4
5
6
# ls -l bazel-bin
lrwxrwxrwx 1 root root 105 Jan  7 09:36 bazel-bin -> /root/.cache/bazel/_bazel_root/4c4ce5c670eb1080771f6be707bd2d39/execroot/envoy/bazel-out/k8-fastbuild/bin

# echo $(bazel info bazel-genfiles)
WARNING: info command does not support starlark options. Ignoring options: [--@com_googlesource_googleurl//build_config:system_icu=0]
/root/.cache/bazel/_bazel_root/4c4ce5c670eb1080771f6be707bd2d39/execroot/envoy/bazel-out/k8-fastbuild/bin

快速测试 Envoy

使用如下方式对生成的 Envoy 交付件进行快速功能测试：

1
# $(bazel info bazel-genfiles)/source/exe/envoy-static --config-path /root/data/code/envoy/envoy_test.yaml

配置文件 envoy_test.ymal 的内容如下：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
admin:
  address:
    socket_address:
      protocol: TCP
      address: 0.0.0.0
      port_value: 9901
static_resources:
  listeners:
  - name: listener_0
    address:
      socket_address:
        protocol: TCP
        address: 0.0.0.0
        port_value: 10000
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          scheme_header_transformation:
            scheme_to_overwrite: https
          stat_prefix: ingress_http
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: local_nginx
          http_filters:
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
  clusters:
  - name: local_nginx
    connect_timeout: 30s
    type: LOGICAL_DNS
    # Comment out the following line to test on v6 networks
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    load_assignment:
      cluster_name: local_nginx
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address:
                address: 172.17.0.2
                port_value: 80

该配置主要实现了以下功能：

  • 提供了一个管理站点，站点端口为 9901
  • 通过 静态资源方式 配置了一个 HTTP 代理站点，站点的端口为 10000，上游集群为 local_nginx，且集群中只有一个 server，即 172.17.0.2:80

所以通过访问宿主机的 9901 端口，可以访问 envoy 的管理界面，访问宿主机的 10000 端口，即可访问 172.17.0.2:80：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
# curl http://127.0.0.1:9901/clusters
local_nginx::observability_name::local_nginx
local_nginx::default_priority::max_connections::1024
local_nginx::default_priority::max_pending_requests::1024
local_nginx::default_priority::max_requests::1024
local_nginx::default_priority::max_retries::3
local_nginx::high_priority::max_connections::1024
local_nginx::high_priority::max_pending_requests::1024
local_nginx::high_priority::max_requests::1024
local_nginx::high_priority::max_retries::3
local_nginx::added_via_api::false
local_nginx::172.17.0.2:80::cx_active::2
local_nginx::172.17.0.2:80::cx_connect_fail::0
local_nginx::172.17.0.2:80::cx_total::4
local_nginx::172.17.0.2:80::rq_active::0
local_nginx::172.17.0.2:80::rq_error::0
local_nginx::172.17.0.2:80::rq_success::5
local_nginx::172.17.0.2:80::rq_timeout::0
local_nginx::172.17.0.2:80::rq_total::5
local_nginx::172.17.0.2:80::hostname::172.17.0.2
local_nginx::172.17.0.2:80::health_flags::healthy
local_nginx::172.17.0.2:80::weight::1
local_nginx::172.17.0.2:80::region::
local_nginx::172.17.0.2:80::zone::
local_nginx::172.17.0.2:80::sub_zone::
local_nginx::172.17.0.2:80::canary::false
local_nginx::172.17.0.2:80::priority::0
local_nginx::172.17.0.2:80::success_rate::-1
local_nginx::172.17.0.2:80::local_origin_success_rate::-1
1
2
3
4
5
6
7
8
9
10
# curl http://127.0.0.1:10000/ -I
HTTP/1.1 200 OK
server: envoy
date: Mon, 09 Jan 2023 07:42:59 GMT
content-type: text/html
content-length: 615
last-modified: Tue, 19 Jul 2022 14:05:27 GMT
etag: "62d6ba27-267"
accept-ranges: bytes
x-envoy-upstream-service-time: 0

其他

生成调试版本

使用如下方式生成调试版本，方便我们在 gdb 下调试 Envoy 代码：

1
# TEST_TMPDIR="$PWD/envoy_compile_db" bazel build -c dbg --jobs=2 --copt=-Wno-error=vla-parameter envoy

另外这里有一点需要特别注意，由于 Envoy 生成调试版本时默认会生成独立的调试信息文件（.dwp 文件），在我的环境中没法完全调试（看不到函数参数，行号等信息），得益于这个 commit，我们可以关闭这种调试信息生成方式。修改源码目录下的 .bazelrc 文件，将如下两行注释掉：

1
2
build:linux --fission=dbg,opt
build:linux --features=per_object_debug_info

这样就可以对生成的交付件调试了：

1
2
3
4
5
6
7
#0  Envoy::OptionsImpl::OptionsImpl(int, char const* const*, std::function<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > (bool)> const&, spdlog::level::level_enum) (this=0x7917fcfc338, argc=1, argv=0x7fffffffe0f8, hot_restart_version_cb=...,
    default_log_level=spdlog::level::info) at source/server/options_impl.cc:40
#1  0x0000555555c1dfc6 in Envoy::MainCommon::MainCommon (this=0x7917fcfc000, argc=1, argv=0x7fffffffe0f8) at source/exe/main_common.cc:206
#2  0x0000555555c34e19 in std::make_unique<Envoy::MainCommon, int&, char**&> () at /usr/include/c++/11/bits/unique_ptr.h:962
#3  0x0000555555c1e5f7 in Envoy::MainCommon::main(int, char**, std::function<void (Envoy::Server::Instance&)>) (argc=1, argv=0x7fffffffe0f8,
    hook=...) at source/exe/main_common.cc:235
#4  0x0000555555bc5f14 in main (argc=1, argv=0x7fffffffe0f8) at source/exe/main.cc:24

清理交付件

执行 bazel clean 清理生成的交付件。

使用 CI docker 镜像构建

也可以使用 Envoy 的 CI 容器镜像来进行构建，此时可以免去各种依赖安装问题。执行如下命令会使用 CI 容器进行构建：

1
# ./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.dev'

生成的交付件默认位于 /tmp/envoy-docker-build/envoy/source/exe/envoy-fastbuild。通过设置 ENVOY_DOCKER_BUILD_DIR 环境变量可以修改交付件的目录。

关于 CI docker 镜像的使用，可以参考这里。

生成调试版本：

1
./ci/run_envoy_docker.sh './ci/do_ci.sh bazel.debug.server_only'

同样需要通过上述方式关闭生成独立的调试文件。要不然也无法直接用 gdb 调试生成的交付件，会出现错误：

1
2
Reading symbols from envoy...
Dwarf Error: DW_FORM_strx1 found in non-DWO CU [in module /build/envoy/source/exe/envoy/envoy]

生成编译数据库

使用如下命令生成 编译数据库，之后就可以在 Vim 编辑器中以支持 LSP（Language Server Protocol）的方式阅读 Envoy 源代码了：

1
2
# mkdir envoy_compile_db
# TEST_TMPDIR="$PWD/envoy_compile_db" BAZEL_BUILD_OPTIONS='--copt=-Wno-error=vla-parameter' python3 tools/gen_compilation_database.py --include_all  //source/exe:envoy

实际测试时使用该方法生成的 compile_commands.json，代码定义无法跳转。还尝试过通过 bear 来生成 compile_commands.json，仍然不行：

1
bear -- bazel build --spawn_strategy=local -c dbg --jobs=2 --copt=-Wno-error=vla-parameter envoy

尝试了一个开源项目 bazel-compile-commands-extractor，通过如下方法生成 compile_commands.json：

在 WORKSPACE 文件中添加如下内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")


# Hedron's Compile Commands Extractor for Bazel
# https://github.com/hedronvision/bazel-compile-commands-extractor
http_archive(
    name = "hedron_compile_commands",

    # Replace the commit hash in both places (below) with the latest, rather than using the stale one here.
    # Even better, set up Renovate and let it do the work for you (see "Suggestion: Updates" in the README).
    url = "https://github.com/hedronvision/bazel-compile-commands-extractor/archive/ed994039a951b736091776d677f324b3903ef939.tar.gz",
    strip_prefix = "bazel-compile-commands-extractor-ed994039a951b736091776d677f324b3903ef939",
    # When you first run this tool, it'll recommend a sha256 hash to put here with a message like: "DEBUG: Rule 'hedron_compile_commands' indicated that a canonical reproducible form can be obtained by modifying arguments sha256 = ..."
)
load("@hedron_compile_commands//:workspace_setup.bzl", "hedron_compile_commands_setup")
hedron_compile_commands_setup()

在 BUILD 文件中添加如下内容：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
load("@hedron_compile_commands//:refresh_compile_commands.bzl", "refresh_compile_commands")

refresh_compile_commands(
    name = "refresh_compile_commands",

    # Specify the targets of interest.
    # For example, specify a dict of targets and any flags required to build.
    targets = {
      "//source/exe:envoy": "-c dbg --jobs=2 --copt=-Wno-error=vla-parameter",
    },
    # No need to add flags already in .bazelrc. They're automatically picked up.
    # If you don't need flags, a list of targets is also okay, as is a single target string.
    # Wildcard patterns, like //... for everything, *are* allowed here, just like a build.
      # As are additional targets (+) and subtractions (-), like in bazel query https://docs.bazel.build/versions/main/query.html#expressions
    # And if you're working on a header-only library, specify a test or binary target that compiles it.
)

之后执行如下命令，生成 compile_commands.json 文件：

1
TEST_TMPDIR="$PWD/envoy_compile_db" bazel run :refresh_compile_commands

但该方法生成的 compile_commands.json 也无法完全正常工作，从 LSP 报错信息来看，主要是一些头文件找不到，通过如下脚本添加了一些符号链接：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
from pathlib import Path

TARGET_DIR = "/root/docker_share/code/envoy/envoy_compile_db/_bazel_root/fb60d7d0260d7a99ab8b2dd78a7e47d8/external"
LINK_DIR = "/root/docker_share/code/envoy/envoy_compile_db/_bazel_root/fb60d7d0260d7a99ab8b2dd78a7e47d8/execroot/envoy/external"

target_dir = Path(TARGET_DIR)
for d in target_dir.iterdir():
    if d.is_dir():
        print(d.name)
        link = Path(LINK_DIR).joinpath(d.name)
        if link.is_symlink():
            link.unlink()

        link.symlink_to(d.absolute())

貌似完全工作了，有些折腾。

Reference

Building Envoy with Bazel
Bazel 输出目录布局
libc++ 与 libstdc++ 是什么关系？
Clangd Getting started