多层和双层压缩
解耦归并排序和 Flush
下图展示了一个用户线程插入小 KV 对应的随时间变化的性能,顺序和随机。CPU-core 利用率为 100% 的线程分别只占用 SSD IO 全带宽的 1/6 和 1/20。连续的随机写操作会触发相应的后台线程执行周期性的刷新和压缩,这些线程会消耗大约 25% 的 CPU-core 利用率。
当 CPU-core 利用率约70%的用户线程连续插入随机大KV对(即1KB)时,后台线程的周期性 compaction 操作仅消耗23% IO带宽和60% CPU-core利用率,如图4b所示。
以往的研究大多认为LSMtree压缩是严重影响整体性能的主要因素,因为LSMtree压缩具有写停顿和写放大的高IO强度[1,3,18,41,43,46,50,54]。事实上,使用小型KV对的写工作负载会使 CPU 核过载,但IO带宽利用不足,而使用大型KV对的写工作负载则恰恰相反
下图展示了不同线程在单个实例下的延迟分解,写流程分成五个步骤,WAL,Memtable,WAL Lock,Mmetable Lock and Others.
锁开销:随着写入器数量的增加,WAL 和 MemTable 的 CPU 周期百分比从单个线程时的 90% 下降到 32 个线程时的16.3%,而总锁开销(即WAL锁和MemTable 锁)从几乎为零增加到 81.4%。更多的写入器会对共享数据结构(如日志和MemTable)引入更严重的争用。特别是,只有 8 个线程的 WAL-lock 占用了一半以上的延迟。根据 Amdahl 定律,在小型KV对的高并发工作负载下,对特定日志和索引结构的优化不再有效,因为序列化瓶颈占延迟的较大比例。
造成高锁开销的主要原因有三个。首先,
现象:单实例,随着线程数增加,延迟也不断增加,其中加锁操作带来的开销比例也不断增加
如3.4节所示,对于小的写操作,请求批处理是减少IO和CPU开销的有效方法。此外,一些kvs(如RocksDB)对读类型请求有很好的并行优化。这些特性有助于提高每个 worker 的整体表现。
为了提升内部并行性,引入一个基于队列的请求批处理调度技术,简称 OBM,如下图所示。
当工作线程处理请求时,用户线程将请求添加到请求队列中。当worker完成一个请求的处理后,它会检查请求队列。如果有两个或更多连续的相同请求类型的传入请求(例如,读类型GET或写类型PUT、UPDATE和DELETE),它们将合并成一个批处理请求,然后作为一个整体处理,如算法1所示。
对于写类型的请求,工作者将其作为WriteBatch处理。与 IO 级别的批处理(如RocksDB组日志记录)相比,这种请求级别的批处理更有利于减少线程同步开销。
对于读类型的请求,工作人员在KVS上并发查询它们。RocksDB提供了一个multiget接口,这是一个非常优化的操作来处理内部并发的键搜索,我们在实现中使用这个接口来处理读类型的批处理请求。
注意,worker 不会主动等待来捕获传入的请求。因此,这种批处理是机会主义的
在表2中,我们展示了在p2KVS和其他kvs上处理100M随机写操作时内存和cpu的使用情况。
在所有kvs中,平均内存使用量小于1.5 GB。p2KVS-4和p2KVS-8的CPU消耗分别超过单核的 7x和12x。
p2KVS的这些较高的CPU使用率来自于4或8个工作线程和额外的后台线程。用户线程在提交请求后休眠,只消耗很少的CPU资源。
RocksDB中的每个用户线程都会重载几乎整个CPU核,导致16个线程时巨大的CPU占用。然而,由于频繁的线程同步和锁开销,它的吞吐量很低。因此,后台压缩线程占用的CPU资源较少。
因为PebblesDB没有针对并发写进行优化,所以大多数并发用户线程都处于等待状态,只占总CPU资源的一小部分。
p2KVS的内存消耗来自底层RocksDB实例的内存使用总和。这是可以接受的,也是稳定的,因为RocksDB实例的内存使用量不会随着数据量的增加而增加
接下来,我们评估请求延迟作为负载强度的函数。我们在RocksDB和p2KVS上以不同的请求强度进行1M随机写操作。图13a显示了p2KVS和RocksDB在轻负荷下的平均延迟非常接近。
下图展示了多个 workers 和 OBM 下的点查询性能。我们向RocksDB和p2kv发起10M GET请求。
如图14a所示,在没有OBM的情况下,p2KVS的性能与RocksDB基本相同。
通过利用RocksDB和OBM p2KVS的读优化,可以实现几乎线性的可伸缩性,如图14b所示。启用OBM的p2kv -8的QPS比禁用情况高出7.5倍,RocksDB高出5.4倍。
范围查询。我们加载100M 128字节KV对对系统进行预热,然后使用单个用户线程执行不同扫描大小的1M RANGE或SCAN操作。如图15所示,在RANGE查询中,
我们还记录并比较p2kv -8和KVell-8在连续100M随机写工作负载下的IO带宽、内存和CPU的利用率。KVell-8和p2KVS-8的吞吐量分别为2.5 MQPS和3.0 MQPS。
MultiGet Performance · facebook/rocksdb Wiki
读路径
MultiGet 性能优化
客户端元数据缓存一致性维护的开销变得非常大,因为非常大规模的文件系统通常需要为大量并发的客户端服务
路径解析需要从根遍历目录树,并依次检查路径中所有中间目录的权限。这将导致大量读取接近根目录,即使是为了平衡元数据操作工作负载。文件系统的吞吐量将受到存储近根目录的服务器的限制。在本文中,我们称之为近根热点。许多分布式文件系统依赖于客户端元数据缓存来缓解接近根的热点
我们发现,以前的客户端缓存机制在大量客户端的大规模场景中不能很好地工作。
局部性的考虑
负载均衡的考虑
数据的存储
INFINIFS 如何生成并维护 可以在未来根据 pathname 预测的目录 ID
Creating:
Renaming:
ID 唯一性
注意:我们的目标和依赖文件之间要使用冒号分隔开,命令的开始一定要使用Tab键。
name: dependencies
commands
# or
targets : prerequisites
commands
# or
targets : prerequisites; command
command
#include <iostream>
int main(int argc, const char *argv[])
{
std::cout << "Hello world!" << std::endl;
return 0;
}
hello: main.cpp
$(CXX) -o hello main.cpp
echo "OK"
$ make hello
$ ./hello
make all
时对应跳转到执行 make hello
,再跳转到执行 make main.o
,相应地将 main.cpp
编译成了 main.o
,然后 hello 的 command 将对应的目标文件链接起来构成对应的可执行文件 hellomake clean
删除对应的编译的中间文件和可执行文件#
# := 用于给变量赋值,除此之外还有 =、?=、+= 等不同的赋值方式。
#
# 一般全大写变量用来表示允许调用 make 的时候传入的变量,
# 全小写变量表示仅内部使用的变量。
#
# 这里 CC 和 CXX 指定了要使用的 C 和 C++ 编译器。
#
CC := clang
CXX := clang++
#
# Makefile 中的核心概念是 target(目标),定义 target 的基本
# 格式是(注意每一行 command 是必须用 tab 缩进的):
#
# name: dependencies
# commands
#
# 要构建某个 target 时,使用如下命令:
#
# make target_name
#
# 下面 all 是一个 target,它依赖另一个 target:hello,
# 意味着要构建 all,首先需要构建 hello。而 all 的 commands
# 部分为空,表示构建 all 不需要额外命令。
#
# .PHONY 表示 all 不是一个真实会生成的文件,而是一个“伪目标”。
#
.PHONY: all
all: hello
#
# 由于后面需要多次使用 main.o 等目标文件列表,这里赋值给变量
# objects。
#
objects := main.o
#
# hello 是我们最终要生成的可执行文件名,它依赖 objects 中的
# 所有目标文件。
#
# 它的 commands 部分使用 CXX 指定的编译器将所有目标文件链接
# 成 hello 可执行文件。
#
hello: $(objects)
$(CXX) -o $@ $(objects)
# main.o 目标文件依赖 main.cpp 源文件。
main.o: main.cpp
$(CXX) -c main.cpp
#
# clean 用于清除构建生成的临时文件、目标文件和可执行文件。
# 和 all 类似,它是一个“伪目标”。
#
.PHONY: clean
clean:
rm -f hello $(objects)
#pragma once
namespace answer {
int find_the_ultimate_answer();
} // namespace answer
#include "answer.hpp"
namespace answer {
int find_the_ultimate_answer() {
return 42;
}
} // namespace answer
#include <iostream>
#include "answer.hpp"
int main(int argc, const char *argv[]) {
int expected_answer = answer::find_the_ultimate_answer();
for (;;) {
std::cout << "What is the ultimate answer?" << std::endl;
int answer;
std::cin >> answer;
if (answer == expected_answer) {
std::cout << "Correct!" << std::endl;
break;
}
}
return 0;
}
make answer.o
需要添加对应的步骤,因为有对应的规范,所以 Make 其实可以通过对应的目标名推断出同名的 .cpp 文件,只需要指定目标文件所依赖的头文件,使头文件变动时可以重新编译对应目标文件。CC := clang
CXX := clang++
.PHONY: all
all: answer
# 在这里添加了 answer.o 目标文件。
objects := main.o answer.o
answer: $(objects)
$(CXX) -o $@ $(objects)
#
# Make 可以自动推断 .o 目标文件需要依赖同名的 .cpp 文件,
# 所以其实不需要在依赖中指定 main.cpp 和 answer.cpp,
# 也不需要写编译 commands,它知道要用 CXX 变量制定的命令
# 作为 C++ 编译器。
#
# 这里只需要指定目标文件所依赖的头文件,使头文件变动时可以
# 重新编译对应目标文件。
#
main.o: answer.hpp
answer.o: answer.hpp
.PHONY: clean
clean:
rm -f answer $(objects)
.PHONY:clean
clean:
rm -rf *.o test
test:*.c
gcc -o $@ $^
# 不能使用 OBJ=*.c
# 要使用一个函数 "wildcard",这个函数在我们引用变量的时候,会帮我们展开
OBJ=$(wildcard *.c)
test:$(OBJ)
gcc -o $@ $^
# "%.o" 把我们需要的所有的 ".o" 文件组合成为一个列表,
# 从列表中挨个取出的每一个文件,"%" 表示取出来文件的文件名(不包含后缀)
# 然后找到文件中和 "%"名称相同的 ".c" 文件
# 然后执行下面的命令,直到列表中的文件全部被取出来为止
test:test.o test1.o
gcc -o $@ $^
%.o:%.c
gcc -o $@ $^
# "$@" 代表的是目标文件test
# “$^” 代表的是依赖的文件
test:test.o test1.o test2.o
gcc -o $@ $^
# “$<” 代表的是依赖文件中的第一个
test.o:test.c test.h
gcc -o $@ $<
test1.o:test1.c test1.h
gcc -o $@ $<
test2.o:test2.c test2.h
gcc -o $@ $<
# 库文件的制作依赖于这三个文件。当修改了其中的某个依赖文件,
# 在命令行执行 make 命令,库文件 "lib" 就会自动更新。
# "$?" 表示修改的文件
lib:test.o test1.o test2.o
ar r $?
cmake -G Ninja
CMakeLists.txt
基本格式:cmake_minimum_required(VERSION 3.9)
project(answer)
add_executable(answer main.cpp answer.cpp)
# 指定最小 CMake 版本要求
cmake_minimum_required(VERSION 3.9)
# 设置项目名称
project(answer)
#[[
添加可执行文件 target,类似于原来 Makefile 的:
answer: main.o answer.o
main.o: main.cpp answer.hpp
answer.o: answer.cpp answer.hpp
CMake 会自动找到依赖的头文件,因此不需要特别指定,
当头文件修改的时候,会重新编译依赖它的目标文件。
#]]
# 当前头文件就在当前目录,所以无需特别指定
# 如需指定,可以使用 include_directories
# 格式 add_executable(target srcfiles)
add_executable(answer main.cpp answer.cpp)
#[[
使用如下命令构建本项目:
cmake -B build # 生成构建目录
cmake --build build # 执行构建
./build/answer # 运行 answer 程序
#]]
cmake -B <buildPath>
使用"-B"参数指定生成目录,这样CMake生成的文件都会集中在这个文件cmake --build <buildPath>
执行类似于 Makefile 中的 make 的效果,同时支持底层为 Ninja 时的 build 操作,类似于一个封装./build/answer
执行对应的目标程序cmake -B build # 生成构建目录,-B 指定生成的构建系统代码放在 build 目录
cmake --build build # 执行构建
./build/answer # 运行 answer 程序
# 添加一个 libanswer 的静态库
add_library(libanswer STATIC answer.cpp)
# 依然指定可执行文件
add_executable(answer main.cpp)
# 链接对应的库
target_link_libraries(answer libanswer)
cmake_minimum_required(VERSION 3.9)
project(answer)
# 添加 libanswer 库目标,STATIC 指定为静态库
add_library(libanswer STATIC answer.cpp)
add_executable(answer main.cpp)
# 为 answer 可执行目标链接 libanswer
target_link_libraries(answer libanswer)
#[[
使用如下命令构建本项目:
cmake -B build # 生成构建目录
cmake --build build # 执行构建
./build/answer # 运行 answer 程序
#]]
.
├── answer
│ ├── answer.cpp
│ ├── CMakeLists.txt
│ └── include
│ └── answer
│ └── answer.hpp
├── CMakeLists.txt
└── main.cpp
add_subdirectory
添加 answer 子目录cmake_minimum_required(VERSION 3.9)
project(answer)
# 添加 answer 子目录
add_subdirectory(answer)
add_executable(answer_app main.cpp)
target_link_libraries(answer_app libanswer)
#[[
使用如下命令构建本项目:
cmake -B build # 生成构建目录
cmake --build build # 执行构建
./build/answer_app # 运行 answer_app 程序
#]]
CMAKE_CURRENT_SOURCE_DIR
是 CMake 内置变量,表示当前 CMakeLists.txt 文件所在目录,此处其实可以省略target_include_directories
PUBLIC 参数表示这个包含目录是 libanswer 的公开接口一部分,链接 libanswer 的 target 可以 #include 该目录中的文件。
#include <answer/answer.hpp>
add_library(libanswer STATIC answer.cpp)
#[[
message 可用于打印调试信息或错误信息,除了 STATUS
外还有 DEBUG WARNING SEND_ERROR FATAL_ERROR 等。
#]]
message(STATUS "Current source dir: ${CMAKE_CURRENT_SOURCE_DIR}")
#[[
给 libanswer 库目标添加 include 目录,PUBLIC 使
这个 include 目录能被外部使用者看到。
当链接 libanswer 库时,这里指定的 include 目录会被
自动添加到使用此库的 target 的 include 路径中。
#]]
target_include_directories(libanswer PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
answer.hpp
#pragma once
#include <string>
namespace answer {
namespace v1 {
int find_the_ultimate_answer();
} // namespace v1
namespace v2 {
std::string find_the_ultimate_answer();
} // namespace v2
using namespace v2;
} // namespace answer
answer.cpp
#include "answer/answer.hpp"
#include <curl/curl.h>
namespace answer {
namespace v1 {
int find_the_ultimate_answer() {
return 42;
}
} // namespace v1
namespace v2 {
std::string find_the_ultimate_answer() {
// 使用 CURL 调用 WolframAlpha API 获得答案
// 注:这里的 appid 是演示用的,只有免费的 2000 次/天调用额度,如有实际需要请自行申请
const auto url = "https://api.wolframalpha.com/v1/result?appid=YAPKJY-8XT9VEYPX9&i=what+is+ultimate+answer";
const auto curl = curl_easy_init();
curl_easy_setopt(curl, CURLOPT_URL, url);
curl_easy_setopt(curl, CURLOPT_FOLLOWLOCATION, 1L);
const auto write_func = [](char *ptr, size_t size, size_t nmemb, void *userdata) {
auto &result = *static_cast<std::string *>(userdata);
result.append(ptr, size * nmemb);
return size * nmemb;
};
using WriteFunction = size_t (*)(char *, size_t, size_t, void *);
curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, static_cast<WriteFunction>(write_func));
std::string result = "";
curl_easy_setopt(curl, CURLOPT_WRITEDATA, &result);
curl_easy_perform(curl); // 暂时不考虑 API 请求失败的情况
curl_easy_cleanup(curl);
return result;
}
} // namespace v2
} // namespace answer
find_package
寻找已经安装的第三方库的头文件和库文件的位置,参数 REQUIRED 要求必须找到,没找到就报错。target_link_libraries
相应地链接 libcurl,使用了 PRIVATE 表明只有当前子模块 answer 能使用对应的接口。#[[
find_package 用于在系统中寻找已经安装的第三方库的头文件和库文件
的位置,并创建一个名为 CURL::libcurl 的库目标,以供链接。
#]]
find_package(CURL REQUIRED)
add_library(libanswer STATIC answer.cpp)
target_include_directories(libanswer PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
#[[
为 libanswer 库链接 libcurl,这里 PRIVATE 和 PUBLIC 的区别是:
CURL::libcurl 库只会被 libanswer 看到,根级别的 main.cpp 中
无法 include curl 的头文件。
#]]
target_link_libraries(libanswer PRIVATE CURL::libcurl)
.
├── answer
│ ├── answer.cpp
│ ├── CMakeLists.txt
│ └── include
│ └── answer
│ └── answer.hpp
├── CMakeLists.txt
├── curl_wrapper
│ ├── CMakeLists.txt
│ ├── curl_wrapper.cpp
│ └── include
│ └── curl_wrapper
│ └── curl_wrapper.hpp
├── main.cpp
├── wolfram
│ ├── alpha.cpp
│ ├── CMakeLists.txt
│ └── include
│ └── wolfram
│ └── alpha.hpp
# CMakeLists.txt
add_subdirectory(answer)
add_subdirectory(curl_wrapper)
add_subdirectory(wolfram)
add_executable(answer_app main.cpp)
target_link_libraries(answer_app libanswer)
# answer/CMakeLists.txt
add_library(libanswer STATIC answer.cpp)
target_include_directories(libanswer PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
target_link_libraries(libanswer PRIVATE wolfram)
# wolfram/CMakeLists.txt
add_library(wolfram STATIC alpha.cpp)
target_include_directories(wolfram PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
target_link_libraries(wolfram PRIVATE curl_wrapper)
# curl_wrapper/CMakeLists.txt
find_package(CURL REQUIRED)
add_library(curl_wrapper STATIC curl_wrapper.cpp)
target_include_directories(curl_wrapper
PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
target_link_libraries(curl_wrapper PRIVATE CURL::libcurl)
-D<para_name>
,但是需要在 CMakeLists.txt 中添加相应的支持。set(WOLFRAM_APPID "" CACHE STRING "WolframAlpha APPID")
注意各个字段的含义target_compile_definitions
:要让 C++ 代码能够拿到 CMake 中的变量,可添加编译时宏定义cmake -B build -DWOLFRAM_APPID=xxx
:命令行参数传递ccmake
:直接 TUI 修改变量传递参数#[[
创建一个可配置的变量,可以由上级 CMakeLists 或 cmake 命令指定变量值。
这里由于 APPID 是一个应该藏好、不应该放在代码里的值,所以建议在 cmake
命令中通过 -D 参数传入。
#]]
# 格式 参数名
# 默认值
# 变量类型 CACHE
# 数据类型 STRING
# 参数描述
set(WOLFRAM_APPID
""
CACHE STRING "WolframAlpha APPID")
if(WOLFRAM_APPID STREQUAL "")
message(SEND_ERROR "WOLFRAM_APPID must not be empty")
endif()
add_library(libanswer STATIC answer.cpp)
target_include_directories(libanswer PUBLIC ${CMAKE_CURRENT_SOURCE_DIR}/include)
#[[
将 WOLFRAM_APPID 添加到编译 .cpp 文件时的 definition 列表,从而
可在 C++ 代码中使用。宏定义名 WOLFRAM_APPID
#]]
target_compile_definitions(libanswer PRIVATE WOLFRAM_APPID="${WOLFRAM_APPID}")
target_link_libraries(libanswer PRIVATE wolfram)
#pragma once
#include <string>
#include <wolfram/alpha.hpp>
// header-only 库的所有实现代码均在头文件中
namespace answer {
namespace v1 {
int find_the_ultimate_answer() {
return 42;
}
} // namespace v1
namespace v2 {
std::string find_the_ultimate_answer() {
return wolfram::simple_query(WOLFRAM_APPID, "what is the ultimate answer?");
}
} // namespace v2
using namespace v2;
} // namespace answer
target_xxx
给 INTERFACE library 添加属性都要用 INTERFACE。add_library(libanswer INTERFACE)
target_include_directories(libanswer
INTERFACE ${CMAKE_CURRENT_SOURCE_DIR}/include)
target_compile_definitions(libanswer INTERFACE WOLFRAM_APPID="${WOLFRAM_APPID}")
target_link_libraries(libanswer INTERFACE wolfram)
target_compile_features
CMAKE_CXX_STANDARD
会应用于所有能看到这个变量的 target,而 target_compile_features
只应用于单个 targettarget_compile_features
可以指定更细粒度的 C++ 特性,例如 cxx_auto_type
、cxx_lambda
等。这里使用了 cxx20 的标准是为了使用 require 的关键字# CMakeLists.txt
set(CMAKE_CXX_STANDARD 11)
# answer/CMakeLists.txt
#[[
指明 libanswer 要求 C++20。
这里和直接设置 CMAKE_CXX_STANDARD 的区别是:
1. 设置 CMAKE_CXX_STANDARD 之后,从设置它的那一级开始
include 的 subdirectory 都会继承这个变量,且应用于
所有能看到这个变量的 target;而 target_compile_features
只应用于单个 target。
2. target_compile_features 可以指定更细粒度的 C++ 特性,
例如 cxx_auto_type、cxx_lambda 等。
#]]
target_compile_features(libanswer INTERFACE cxx_std_20)
answer::check_the_answer
// main.cpp
int main(int argc, const char *argv[]) {
for (;;) {
std::cout << "What is the ultimate answer?" << std::endl;
std::string answer;
std::cin >> answer;
auto expected_answer = answer::find_the_ultimate_answer();
if (answer::check_the_answer(answer, expected_answer)) {
std::cout << "Correct!" << std::endl;
break;
}
}
return 0;
}
answer::check_the_answer
实现 namespace v2 {
std::string find_the_ultimate_answer() {
return wolfram::simple_query(WOLFRAM_APPID, "what is the ultimate answer?");
}
// 下面是非常 fancy 的两个函数,使用了 C++14 的 auto 返回类型、
// C++17 的 if constexpr 和 C++20 的 constraints。
namespace impl {
template <typename T>
auto to_string(T &&t) {
if constexpr (requires { std::to_string(t); }) {
return std::to_string(std::forward<T>(t));
} else if constexpr (requires { std::string(t); }) {
return std::string(std::forward<T>(t));
}
}
} // namespace impl
template <typename T, typename U>
requires requires(T &&t, U &&u) {
impl::to_string(std::forward<T>(t));
impl::to_string(std::forward<U>(u));
}
auto check_the_answer(T &&given, U &&expected) {
return impl::to_string(std::forward<T>(given)) == impl::to_string(std::forward<U>(expected));
}
} // namespace v2
using namespace v2;
root@aep-shunzi:/shunzi/modern-cmake-by-example# tree
.
├── answer
│ ├── CMakeLists.txt
│ ├── include
│ │ └── answer
│ │ └── answer.hpp
│ └── tests
│ ├── CMakeLists.txt
│ └── test_check_the_answer.cpp
├── CMakeLists.txt
├── curl_wrapper
│ ├── CMakeLists.txt
│ ├── curl_wrapper.cpp
│ └── include
│ └── curl_wrapper
│ └── curl_wrapper.hpp
├── main.cpp
├── wolfram
│ ├── alpha.cpp
│ ├── CMakeLists.txt
│ └── include
│ └── wolfram
│ └── alpha.hpp
include(CTest)
主 CMakeLists 需要添加对 ctest 的支持BUILD_TESTING
在引入 CTest 之后将会引入一个默认的 CACHE 变量来标识是否编译 Test,默认值为 ON,也可以通过 -D
参数传递来修改add_subdirectory(tests)
在 answer 模块中添加新的测试目录作为子目录,此时的文件结构add_test
添加相应的测试用例
<name>
指定本测试的名称Debug/Release
选项可以控制在不同的编译版本下是否进行测试WORKING_DIRECTORY
设置工作路径command
表示可运行程序add_test(NAME <name> [CONFIGURATIONS [Debug|Release|...]]
[WORKING_DIRECTORY dir]
COMMAND <command> [arg1 [arg2 ...]])
# CMakeLists.txt
cmake_minimum_required(VERSION 3.14) # 提高了 CMake 版本要求
project(answer)
#[[
判断当前目录是否是 CMake 调用的 top-level,如果是,
引入 CTest 支持。
这会引入一个 BUILD_TESTING 选项(类似之前的 CACHE
STRING,这是一个 CACHE BOOL),默认值为 ON,可以在
之后的 CMake 脚本中通过该选项判断是否需要 include
测试用例子目录。
#]]
if(CMAKE_PROJECT_NAME STREQUAL PROJECT_NAME)
include(CTest)
endif()
# answer/CMakeLists.txt
if(BUILD_TESTING)
add_subdirectory(tests)
endif()
# answer/tests/CMakeLists.txt
#[[
add_test 添加 CTest 可以识别到的测试程序,建议使用项目名前缀,
方便在运行测试时和别的第三方库的测试区分。
#]]
add_executable(test_some_func test_some_func.cpp)
add_test(NAME answer.test_some_func COMMAND test_some_func)
answer/tests/test_check_the_answer.cpp
#include <catch2/catch_test_macros.hpp>
#include <answer/answer.hpp>
using namespace answer;
// 使用 Catch2 编写测试用例
TEST_CASE("Can compare string and string", "[check_the_answer]") {
REQUIRE(check_the_answer("Hello", "Hello") == true);
REQUIRE(check_the_answer("Hello", "world") == false);
REQUIRE(check_the_answer("13", std::string("13")) == true);
}
TEST_CASE("Can compare string and integer", "[check_the_answer]") {
REQUIRE(check_the_answer("13", 13) == true);
REQUIRE(check_the_answer("13", 14) == false);
REQUIRE(check_the_answer(13, "13") == true);
REQUIRE(check_the_answer(13, std::string("13")) == true);
REQUIRE(check_the_answer(13, std::string("14")) == false);
}
find_package
找到系统中安装的第三方库,也可通过 CMake 3.11 新增的 FetchContent 功能下载使用第三方库,使用步骤如下
include(FetchContent)
首先导入该功能FetchContent_Declare
定义依赖的库对应的信息
SOURCE_DIR
Declare 中可以指定要安装的目录FetchContent_MakeAvailable
根据定义信息对应下载构建库并进行依赖
FetchContent_MakeAvailable
要求 CMake 3.14,如果要支持更旧版本,或者需要更细粒度的控制,可以使用如下替代:FetchContent_GetProperties(catch2)
if(NOT catch2_POPULATED)
FetchContent_Populate(catch2)
add_subdirectory(${catch2_SOURCE_DIR} ${catch2_BINARY_DIR})
endif()
target_link_libraries(${TEST_NAME} PRIVATE Catch2::Catch2WithMain)
相应地在后面依赖该库# 导入 FetchContent 相关命令
include(FetchContent)
# 描述如何获取 Catch2
FetchContent_Declare(
catch2 # 建议使用全小写
GIT_REPOSITORY https://github.com/catchorg/Catch2.git
GIT_TAG v3.0.0-preview3)
# 一条龙地下载、构建 Catch2
FetchContent_MakeAvailable(catch2)
#[[
FetchContent 要求 CMake 3.11 或更高版本,在此之间
可以使用 Git submodule + add_subdirectory 的方式
使用没有安装在系统中的第三方库。即使支持 FetchContent
也可以选择使用 Git submodule,各有优劣。
FetchContent_MakeAvailable 要求 CMake 3.14,如果
要支持更旧版本,或者需要更细粒度的控制,可以使用如下替代:
FetchContent_GetProperties(catch2)
if(NOT catch2_POPULATED)
FetchContent_Populate(catch2)
add_subdirectory(${catch2_SOURCE_DIR} ${catch2_BINARY_DIR})
endif()
#]]
# macro(宏)类似于 C/C++ 中的宏
macro(answer_add_test TEST_NAME)
add_executable(${TEST_NAME} ${ARGN}) # ${ARGN} 类似于 C/C++ 中的 __VA_ARGS__
#[[
add_test 添加 CTest 可以识别到的测试程序,建议使用项目名前缀,
方便在运行测试时和别的第三方库的测试区分。
#]]
add_test(NAME answer.${TEST_NAME} COMMAND ${TEST_NAME})
target_link_libraries(${TEST_NAME} PRIVATE libanswer)
#[[
链接 Catch2::Catch2WithMain 以使用 Catch2 提供的宏,链接
Catch2WithMain 时,测试程序中不需要手动编写 main 函数。
#]]
target_link_libraries(${TEST_NAME} PRIVATE Catch2::Catch2WithMain)
endmacro()
# 调用上面的 macro 添加测试程序
answer_add_test(test_check_the_answer test_check_the_answer.cpp)
macro
类似于定义了一个宏。
macro(answer_add_test TEST_NAME)
对应定义了函数名 answer_add_test
和函数参数 TEST_NAME
add_executable(${TEST_NAME} ${ARGN})
本例中相应地使用对应的参数创建了对应的测试程序,其中 ${ARGN}
类似于 C/C++ 中的 __VA_ARGS__
,即可变数量的参数数组add_test(NAME answer.${TEST_NAME} COMMAND ${TEST_NAME})
添加 CTest 可以识别到的测试程序target_link_libraries(${TEST_NAME} PRIVATE libanswer)
给相应的测试程序链接对应的库answer_add_test(test_check_the_answer test_check_the_answer.cpp)
相应地调用对应的宏定义,本例中对应了一个测试程序,test_check_the_answer 以及其其所依赖的源文件 test_check_the_answer.cppmacro(answer_add_test TEST_NAME)
add_executable(${TEST_NAME} ${ARGN}) # ${ARGN} 类似于 C/C++ 中的 __VA_ARGS__
add_test(NAME answer.${TEST_NAME} COMMAND ${TEST_NAME})
target_link_libraries(${TEST_NAME} PRIVATE libanswer)
target_link_libraries(${TEST_NAME} PRIVATE Catch2::Catch2WithMain)
endmacro()
answer_add_test(test_check_the_answer test_check_the_answer.cpp)
answer_add_test(test_another_function test_another_function.cpp)
ctest --test-dir build -R "^answer."
运行 ctest,制定相应的目录和对应的 test 名称${CMAKE_CURRENT_FUNCTION}
当前函数名称${CMAKE_CURRENT_FUNCTION_LIST_DIR}
当前函数路径${CMAKE_CURRENT_FUNCTION_LIST_FILE}
当前函数所属文件${CMAKE_CURRENT_FUNCTION_LIST_LINE}
当前函数定义的起始行数make build WOLFRAM_APPID=xxx
简化 build 过程,并传递相关参数make test
简化 test 运行,执行 CTestmake run
简化运行过程,直接执行make clean
简化重编译的过程,删除 build 目录WOLFRAM_APPID :=
.PHONY: build
build: configure
cmake --build build
.PHONY: configure
configure:
cmake -B build -DWOLFRAM_APPID=${WOLFRAM_APPID}
.PHONY: run
run:
./build/answer_app
.PHONY: test
test:
ctest --test-dir build -R "^answer."
.PHONY: clean
clean:
rm -rf build
#
# 可以使用 Make 来更方便地调用 CMake 命令:
#
# make build WOLFRAM_APPID=xxx
# make test
# make run
# make clean
#
$ make defconfig # 有各种 xxxconfig
$ make menuconfig # TUI 界面修改 config
$ make # 构建 Linux 内核
$ make modules_install # 安装内核模块
$ make install # 安装内核
list (subcommand <list> [args...])
:subcommand
为具体的列表操作子命令,GET/APPEND/INSERT/REMOVE_ITEM/REMOVE_AT/REMOVE_DUPLICATES/REVERSE/SORT
。<list>
为待操作的列表变量,[args...]
为对列表变量操作需要使用的参数表,不同的子命令对应的参数也不一致。
CMAKE_MODULE_PATH
默认情况下为空,它是由项目设置的。include
: 用来载入并运行来自于文件或模块的 CMake 代码。即引入其他的 cmake 配置代码,这里主要是承接上一个命令对应的其他 cmake 模块include(ReadVersion)
对应了上图中的 ReadVersion 模块,该模块定义了获取当前 RocksDB 版本的 function get_rocksdb_version,该函数将从 "${CMAKE_CURRENT_SOURCE_DIR}/include/rocksdb/version.h
文件中读取对应的版本号,使用正则表达式去匹配,然后得到相应的返回值 version_varinclude(GoogleTest)
对应了 CMake 中自带的 GoogleTest 模块,https://cmake.org/cmake/help/latest/module/GoogleTest.htmlproject
描述项目基本信息,名称,版本,语言CMP0042
https://cmake.org/cmake/help/v3.0/policy/CMP0042.htmlCMAKE_BUILD_TYPE
CMake 编译的相关配置,根据时候被作为参数传递进来以及是否包含 git 目录来采取不同的设置,可能的配置如下。设置完后 set cache string 变量 CMAKE_BUILD_TYPE
find_program
寻找相应的程序,并将结果存储在变量中。option
定义编译选项,可以在编译时指定参数 -Dxxx 传递add_definitions
为源文件的编译添加由-D定义的标志$ENV{NAME}
调用系统的环境变量CMAKE_SYSTEM_NAME
:CMake 要构建的操作系统的名称CMAKE_CXX_STANDARD
:CXX 标准设置include(CMakeDependentOption)
:引入 CMAKE_DEPENDENT_OPTION 对应的模块string
cmake 中的字符串操作 https://cmake.org/cmake/help/latest/command/string.htmlexecute_process
执行对应的系统命令。注意几个变量的设置
# Linux:
#
# 1. Install a recent toolchain if you're on a older distro. C++17 required (GCC >= 7, Clang >= 5)
# 2. mkdir build; cd build
# 3. cmake ..
# 4. make -j
cmake_minimum_required(VERSION 3.10)
# 添加模块目录到 CMAKE_MODULE_PATH
list(APPEND CMAKE_MODULE_PATH "${CMAKE_CURRENT_LIST_DIR}/cmake/modules/")
# 引入 ReadVersion 对应的 function
include(ReadVersion)
# 引入 CMake 自带的 GoogleTest
include(GoogleTest)
# 获取对应的 RocksDB 版本信息,rocksdb_VERSION
get_rocksdb_version(rocksdb_VERSION)
# 描述项目的基础信息
project(rocksdb
VERSION ${rocksdb_VERSION}
LANGUAGES CXX C ASM)
# MACOSX_RPATH 默认启用
if(POLICY CMP0042)
cmake_policy(SET CMP0042 NEW)
endif()
# 如果没有指定 CMAKE_BUILD_TYPE
if(NOT CMAKE_BUILD_TYPE)
# 判断是否包含 git 目录,包含的话开启 debug 模式,否则开启 RelWithDebInfo
if(EXISTS "${CMAKE_SOURCE_DIR}/.git")
set(default_build_type "Debug")
else()
set(default_build_type "RelWithDebInfo")
endif()
# 设置相应的参数
set(CMAKE_BUILD_TYPE "${default_build_type}" CACHE STRING
"Default BUILD_TYPE is ${default_build_type}" FORCE)
endif()
# 查找编译器程序 ccache 编译器缓存
find_program(CCACHE_FOUND ccache)
# 如果找到了该程序,设置属性,将 ccache 作为编译命令和链接命令的启动器
if(CCACHE_FOUND)
set_property(GLOBAL PROPERTY RULE_LAUNCH_COMPILE ccache)
set_property(GLOBAL PROPERTY RULE_LAUNCH_LINK ccache)
endif(CCACHE_FOUND)
# 定义几个依赖库的编译选型,并设置默认值
option(WITH_JEMALLOC "build with JeMalloc" OFF)
option(WITH_LIBURING "build with liburing" ON)
option(WITH_SNAPPY "build with SNAPPY" OFF)
option(WITH_LZ4 "build with lz4" OFF)
option(WITH_ZLIB "build with zlib" OFF)
option(WITH_ZSTD "build with zstd" OFF)
option(WITH_WINDOWS_UTF8_FILENAMES "use UTF8 as characterset for opening files, regardles of the system code page" OFF)
if (WITH_WINDOWS_UTF8_FILENAMES)
add_definitions(-DROCKSDB_WINDOWS_UTF8_FILENAMES)
endif()
# 如果环境变量中配置了 CIRCLECI,一款持续集成的工具
if ($ENV{CIRCLECI})
message(STATUS "Build for CircieCI env, a few tests may be disabled")
add_definitions(-DCIRCLECI)
endif()
# 如果操作系统是 Linux 或者 win
# third-party/folly is only validated to work on Linux and Windows for now.
# So only turn it on there by default.
if(CMAKE_SYSTEM_NAME MATCHES "Linux|Windows")
# 判断 MSVC 版本,并对 Folly 进行配置(folly, Facebook Open-source Library)
if(MSVC AND MSVC_VERSION LESS 1910)
# Folly does not compile with MSVC older than VS2017
option(WITH_FOLLY_DISTRIBUTED_MUTEX "build with folly::DistributedMutex" OFF)
else()
option(WITH_FOLLY_DISTRIBUTED_MUTEX "build with folly::DistributedMutex" ON)
endif()
else()
option(WITH_FOLLY_DISTRIBUTED_MUTEX "build with folly::DistributedMutex" OFF)
endif()
# 如果没制定 CXX 标准,指定 C++17
if( NOT DEFINED CMAKE_CXX_STANDARD )
set(CMAKE_CXX_STANDARD 17)
endif()
# 引入专门的条件 Option 模块,CMAKE_DEPENDENT_OPTION
include(CMakeDependentOption)
# 根据不同的操作系统/不同的编译器为 option 赋值
if(MSVC)
option(WITH_GFLAGS "build with GFlags" OFF)
option(WITH_XPRESS "build with windows built in compression" OFF)
include(${CMAKE_CURRENT_SOURCE_DIR}/thirdparty.inc)
else()
if(CMAKE_SYSTEM_NAME MATCHES "FreeBSD" AND NOT CMAKE_SYSTEM_NAME MATCHES "kFreeBSD")
# FreeBSD has jemalloc as default malloc
# but it does not have all the jemalloc files in include/...
set(WITH_JEMALLOC ON)
else()
if(WITH_JEMALLOC)
# 查询 Jemalloc 的库
find_package(JeMalloc REQUIRED)
# 为源文件的编译添加由-D定义的标志。
add_definitions(-DROCKSDB_JEMALLOC -DJEMALLOC_NO_DEMANGLE)
# 将对应的依赖库名称追加到变量 THIRDPARTY_LIBS
list(APPEND THIRDPARTY_LIBS JeMalloc::JeMalloc)
endif()
endif()
if(MINGW)
option(WITH_GFLAGS "build with GFlags" OFF)
else()
option(WITH_GFLAGS "build with GFlags" ON)
endif()
set(GFLAGS_LIB)
if(WITH_GFLAGS)
# Config with namespace available since gflags 2.2.2
option(GFLAGS_USE_TARGET_NAMESPACE "Use gflags import target with namespace." ON)
find_package(gflags CONFIG)
if(gflags_FOUND)
if(TARGET ${GFLAGS_TARGET})
# Config with GFLAGS_TARGET available since gflags 2.2.0
set(GFLAGS_LIB ${GFLAGS_TARGET})
else()
# Config with GFLAGS_LIBRARIES available since gflags 2.1.0
set(GFLAGS_LIB ${gflags_LIBRARIES})
endif()
else()
find_package(gflags REQUIRED)
set(GFLAGS_LIB gflags::gflags)
endif()
include_directories(${GFLAGS_INCLUDE_DIR})
list(APPEND THIRDPARTY_LIBS ${GFLAGS_LIB})
add_definitions(-DGFLAGS=1)
endif()
# 如果开启了 WITH_SNAPPY
if(WITH_SNAPPY)
find_package(Snappy CONFIG)
if(NOT Snappy_FOUND)
find_package(Snappy REQUIRED)
endif()
add_definitions(-DSNAPPY)
list(APPEND THIRDPARTY_LIBS Snappy::snappy)
endif()
if(WITH_ZLIB)
find_package(ZLIB REQUIRED)
add_definitions(-DZLIB)
list(APPEND THIRDPARTY_LIBS ZLIB::ZLIB)
endif()
option(WITH_BZ2 "build with bzip2" OFF)
if(WITH_BZ2)
find_package(BZip2 REQUIRED)
add_definitions(-DBZIP2)
if(BZIP2_INCLUDE_DIRS)
include_directories(${BZIP2_INCLUDE_DIRS})
else()
include_directories(${BZIP2_INCLUDE_DIR})
endif()
list(APPEND THIRDPARTY_LIBS ${BZIP2_LIBRARIES})
endif()
if(WITH_LZ4)
find_package(lz4 REQUIRED)
add_definitions(-DLZ4)
list(APPEND THIRDPARTY_LIBS lz4::lz4)
endif()
if(WITH_ZSTD)
find_package(zstd REQUIRED)
add_definitions(-DZSTD)
include_directories(${ZSTD_INCLUDE_DIR})
list(APPEND THIRDPARTY_LIBS zstd::zstd)
endif()
endif()
# 生成对应的时间戳 TS
string(TIMESTAMP TS "%Y-%m-%d %H:%M:%S" UTC)
# 设置相应的参数
set(BUILD_DATE "${TS}" CACHE STRING "the time we first built rocksdb")
# 搜索依赖 Git
find_package(Git)
# 如果找到了 Git 并且当且目录包含 Git 目录
if(GIT_FOUND AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/.git")
# 执行系统命令 git rev-parse HEAD,得到了对应的 CommitID 保存到了 GIT_SHA (标准输出)
execute_process(WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" OUTPUT_VARIABLE GIT_SHA COMMAND "${GIT_EXECUTABLE}" rev-parse HEAD )
# 执行命令 git diff-index HEAD --quiet 比较树与工作树或索引并禁止输出,保存命令执行结果到 GIT_MOD
execute_process(WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" RESULT_VARIABLE GIT_MOD COMMAND "${GIT_EXECUTABLE}" diff-index HEAD --quiet)
# 执行命令 git log -1 --date=format:"%Y-%m-%d %T" --format="%ad" 输出最近的 git 提交的时间到 GIT_DATE
execute_process(WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" OUTPUT_VARIABLE GIT_DATE COMMAND "${GIT_EXECUTABLE}" log -1 --date=format:"%Y-%m-%d %T" --format="%ad")
# 执行命令 git symbolic-ref -q --short HEAD OUTPUT_STRIP_TRAILING_WHITESPACE
# 输出对应的 tag 到 GIT_TAG 和命令执行结果 rv
execute_process(WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" OUTPUT_VARIABLE GIT_TAG RESULT_VARIABLE rv COMMAND "${GIT_EXECUTABLE}" symbolic-ref -q --short HEAD OUTPUT_STRIP_TRAILING_WHITESPACE)
# 如果 rv != 0
if (rv AND NOT rv EQUAL 0)
# git describe --tags --exact-match OUTPUT_STRIP_TRAILING_WHITESPACE
# 结果输出到 GIT_TAG
execute_process(WORKING_DIRECTORY "${CMAKE_CURRENT_SOURCE_DIR}" OUTPUT_VARIABLE GIT_TAG COMMAND "${GIT_EXECUTABLE}" describe --tags --exact-match OUTPUT_STRIP_TRAILING_WHITESPACE)
endif()
else()
set(GIT_SHA 0)
set(GIT_MOD 1)
endif()
# 使用正则表达式格式化对应的 GIT_SHA 和 GIT_DATE
string(REGEX REPLACE "[^0-9a-fA-F]+" "" GIT_SHA "${GIT_SHA}")
string(REGEX REPLACE "[^0-9: /-]+" "" GIT_DATE "${GIT_DATE}")
option(WITH_MD_LIBRARY "build with MD" ON)
if(WIN32 AND MSVC)
if(WITH_MD_LIBRARY)
set(RUNTIME_LIBRARY "MD")
else()
set(RUNTIME_LIBRARY "MT")
endif()
endif()
set(BUILD_VERSION_CC ${CMAKE_BINARY_DIR}/build_version.cc)
configure_file(util/build_version.cc.in ${BUILD_VERSION_CC} @ONLY)
if(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /Zi /nologo /EHsc /GS /Gd /GR /GF /fp:precise /Zc:wchar_t /Zc:forScope /errorReport:queue")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /FC /d2Zi+ /W4 /wd4127 /wd4800 /wd4996 /wd4351 /wd4100 /wd4204 /wd4324")
else()
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -W -Wextra -Wall -pthread")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wsign-compare -Wshadow -Wno-unused-parameter -Wno-unused-variable -Woverloaded-virtual -Wnon-virtual-dtor -Wno-missing-field-initializers -Wno-strict-aliasing")
if(CMAKE_SYSTEM_PROCESSOR MATCHES "x86_64")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -Wstrict-prototypes")
endif()
if(MINGW)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wno-format -fno-asynchronous-unwind-tables")
add_definitions(-D_POSIX_C_SOURCE=1)
endif()
if(NOT CMAKE_BUILD_TYPE STREQUAL "Debug")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-omit-frame-pointer")
include(CheckCXXCompilerFlag)
CHECK_CXX_COMPILER_FLAG("-momit-leaf-frame-pointer" HAVE_OMIT_LEAF_FRAME_POINTER)
if(HAVE_OMIT_LEAF_FRAME_POINTER)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -momit-leaf-frame-pointer")
endif()
endif()
endif()
include(CheckCCompilerFlag)
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64")
CHECK_C_COMPILER_FLAG("-mcpu=power9" HAS_POWER9)
if(HAS_POWER9)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mcpu=power9 -mtune=power9")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcpu=power9 -mtune=power9")
else()
CHECK_C_COMPILER_FLAG("-mcpu=power8" HAS_POWER8)
if(HAS_POWER8)
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mcpu=power8 -mtune=power8")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mcpu=power8 -mtune=power8")
endif(HAS_POWER8)
endif(HAS_POWER9)
CHECK_C_COMPILER_FLAG("-maltivec" HAS_ALTIVEC)
if(HAS_ALTIVEC)
message(STATUS " HAS_ALTIVEC yes")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -maltivec")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -maltivec")
endif(HAS_ALTIVEC)
endif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64")
if(CMAKE_SYSTEM_PROCESSOR MATCHES "arm64|aarch64|AARCH64")
CHECK_C_COMPILER_FLAG("-march=armv8-a+crc+crypto" HAS_ARMV8_CRC)
if(HAS_ARMV8_CRC)
message(STATUS " HAS_ARMV8_CRC yes")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -march=armv8-a+crc+crypto -Wno-unused-function")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=armv8-a+crc+crypto -Wno-unused-function")
endif(HAS_ARMV8_CRC)
endif(CMAKE_SYSTEM_PROCESSOR MATCHES "arm64|aarch64|AARCH64")
if(CMAKE_SYSTEM_PROCESSOR MATCHES "s390x")
CHECK_C_COMPILER_FLAG("-march=native" HAS_S390X_MARCH_NATIVE)
if(HAS_S390X_MARCH_NATIVE)
message(STATUS " HAS_S390X_MARCH_NATIVE yes")
endif(HAS_S390X_MARCH_NATIVE)
endif(CMAKE_SYSTEM_PROCESSOR MATCHES "s390x")
option(PORTABLE "build a portable binary" OFF)
option(FORCE_SSE42 "force building with SSE4.2, even when PORTABLE=ON" OFF)
option(FORCE_AVX "force building with AVX, even when PORTABLE=ON" OFF)
option(FORCE_AVX2 "force building with AVX2, even when PORTABLE=ON" OFF)
if(PORTABLE)
# MSVC does not need a separate compiler flag to enable SSE4.2; if nmmintrin.h
# is available, it is available by default.
if(FORCE_SSE42 AND NOT MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -msse4.2 -mpclmul")
endif()
if(MSVC)
if(FORCE_AVX)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /arch:AVX")
endif()
# MSVC automatically enables BMI / lzcnt with AVX2.
if(FORCE_AVX2)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /arch:AVX2")
endif()
else()
if(FORCE_AVX)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mavx")
endif()
if(FORCE_AVX2)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -mavx2 -mbmi -mlzcnt")
endif()
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^s390x")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=z196")
endif()
endif()
else()
if(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /arch:AVX2")
else()
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^s390x" AND NOT HAS_S390X_MARCH_NATIVE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=z196")
elseif(NOT CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64" AND NOT HAS_ARMV8_CRC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -march=native")
endif()
endif()
endif()
include(CheckCXXSourceCompiles)
set(OLD_CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS})
if(NOT MSVC)
set(CMAKE_REQUIRED_FLAGS "-msse4.2 -mpclmul")
endif()
CHECK_CXX_SOURCE_COMPILES("
#include <cstdint>
#include <nmmintrin.h>
#include <wmmintrin.h>
int main() {
volatile uint32_t x = _mm_crc32_u32(0, 0);
const auto a = _mm_set_epi64x(0, 0);
const auto b = _mm_set_epi64x(0, 0);
const auto c = _mm_clmulepi64_si128(a, b, 0x00);
auto d = _mm_cvtsi128_si64(c);
}
" HAVE_SSE42)
if(HAVE_SSE42)
add_definitions(-DHAVE_SSE42)
add_definitions(-DHAVE_PCLMUL)
elseif(FORCE_SSE42)
message(FATAL_ERROR "FORCE_SSE42=ON but unable to compile with SSE4.2 enabled")
endif()
# Check if -latomic is required or not
if (NOT MSVC)
set(CMAKE_REQUIRED_FLAGS "--std=c++17")
CHECK_CXX_SOURCE_COMPILES("
#include <atomic>
std::atomic<uint64_t> x(0);
int main() {
uint64_t i = x.load(std::memory_order_relaxed);
bool b = x.is_lock_free();
return 0;
}
" BUILTIN_ATOMIC)
if (NOT BUILTIN_ATOMIC)
#TODO: Check if -latomic exists
list(APPEND THIRDPARTY_LIBS atomic)
endif()
endif()
if (WITH_LIBURING)
find_package(uring)
if (uring_FOUND)
add_definitions(-DROCKSDB_IOURING_PRESENT)
list(APPEND THIRDPARTY_LIBS uring::uring)
endif()
endif()
# Reset the required flags
set(CMAKE_REQUIRED_FLAGS ${OLD_CMAKE_REQUIRED_FLAGS})
# thread_local is part of C++11 and later (TODO: clean up this define)
add_definitions(-DROCKSDB_SUPPORT_THREAD_LOCAL)
option(WITH_IOSTATS_CONTEXT "Enable IO stats context" ON)
if (NOT WITH_IOSTATS_CONTEXT)
add_definitions(-DNIOSTATS_CONTEXT)
endif()
option(WITH_PERF_CONTEXT "Enable perf context" ON)
if (NOT WITH_PERF_CONTEXT)
add_definitions(-DNPERF_CONTEXT)
endif()
option(FAIL_ON_WARNINGS "Treat compile warnings as errors" ON)
if(FAIL_ON_WARNINGS)
if(MSVC)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} /WX")
else() # assume GCC
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Werror")
endif()
endif()
option(WITH_ASAN "build with ASAN" OFF)
if(WITH_ASAN)
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=address")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=address")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=address")
if(WITH_JEMALLOC)
message(FATAL "ASAN does not work well with JeMalloc")
endif()
endif()
option(WITH_TSAN "build with TSAN" OFF)
if(WITH_TSAN)
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=thread -pie")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=thread -fPIC")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=thread -fPIC")
if(WITH_JEMALLOC)
message(FATAL "TSAN does not work well with JeMalloc")
endif()
endif()
option(WITH_UBSAN "build with UBSAN" OFF)
if(WITH_UBSAN)
add_definitions(-DROCKSDB_UBSAN_RUN)
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fsanitize=undefined")
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fsanitize=undefined")
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -fsanitize=undefined")
if(WITH_JEMALLOC)
message(FATAL "UBSAN does not work well with JeMalloc")
endif()
endif()
option(WITH_NUMA "build with NUMA policy support" OFF)
if(WITH_NUMA)
find_package(NUMA REQUIRED)
add_definitions(-DNUMA)
include_directories(${NUMA_INCLUDE_DIR})
list(APPEND THIRDPARTY_LIBS NUMA::NUMA)
endif()
option(WITH_TBB "build with Threading Building Blocks (TBB)" OFF)
if(WITH_TBB)
find_package(TBB REQUIRED)
add_definitions(-DTBB)
list(APPEND THIRDPARTY_LIBS TBB::TBB)
endif()
# Stall notifications eat some performance from inserts
option(DISABLE_STALL_NOTIF "Build with stall notifications" OFF)
if(DISABLE_STALL_NOTIF)
add_definitions(-DROCKSDB_DISABLE_STALL_NOTIFICATION)
endif()
option(WITH_DYNAMIC_EXTENSION "build with dynamic extension support" OFF)
if(NOT WITH_DYNAMIC_EXTENSION)
add_definitions(-DROCKSDB_NO_DYNAMIC_EXTENSION)
endif()
option(ASSERT_STATUS_CHECKED "build with assert status checked" OFF)
if (ASSERT_STATUS_CHECKED)
message(STATUS "Build with assert status checked")
add_definitions(-DROCKSDB_ASSERT_STATUS_CHECKED)
endif()
if(DEFINED USE_RTTI)
if(USE_RTTI)
message(STATUS "Enabling RTTI")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DROCKSDB_USE_RTTI")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -DROCKSDB_USE_RTTI")
else()
if(MSVC)
message(STATUS "Disabling RTTI in Release builds. Always on in Debug.")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DROCKSDB_USE_RTTI")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /GR-")
else()
message(STATUS "Disabling RTTI in Release builds")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -fno-rtti")
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -fno-rtti")
endif()
endif()
else()
message(STATUS "Enabling RTTI in Debug builds only (default)")
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} -DROCKSDB_USE_RTTI")
if(MSVC)
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /GR-")
else()
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} -fno-rtti")
endif()
endif()
# Used to run CI build and tests so we can run faster
option(OPTDBG "Build optimized debug build with MSVC" OFF)
option(WITH_RUNTIME_DEBUG "build with debug version of runtime library" ON)
if(MSVC)
if(OPTDBG)
message(STATUS "Debug optimization is enabled")
set(CMAKE_CXX_FLAGS_DEBUG "/Oxt")
else()
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /Od /RTC1")
# Minimal Build is deprecated after MSVC 2015
if( MSVC_VERSION GREATER 1900 )
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /Gm-")
else()
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /Gm")
endif()
endif()
if(WITH_RUNTIME_DEBUG)
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /${RUNTIME_LIBRARY}d")
else()
set(CMAKE_CXX_FLAGS_DEBUG "${CMAKE_CXX_FLAGS_DEBUG} /${RUNTIME_LIBRARY}")
endif()
set(CMAKE_CXX_FLAGS_RELEASE "${CMAKE_CXX_FLAGS_RELEASE} /Oxt /Zp8 /Gm- /Gy /${RUNTIME_LIBRARY}")
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} /DEBUG")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} /DEBUG")
endif()
if(CMAKE_COMPILER_IS_GNUCXX)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-builtin-memcmp")
endif()
option(ROCKSDB_LITE "Build RocksDBLite version" OFF)
if(ROCKSDB_LITE)
add_definitions(-DROCKSDB_LITE)
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -fno-exceptions -Os")
endif()
if(CMAKE_SYSTEM_NAME MATCHES "Cygwin")
add_definitions(-fno-builtin-memcmp -DCYGWIN)
elseif(CMAKE_SYSTEM_NAME MATCHES "Darwin")
add_definitions(-DOS_MACOSX)
elseif(CMAKE_SYSTEM_NAME MATCHES "Linux")
add_definitions(-DOS_LINUX)
elseif(CMAKE_SYSTEM_NAME MATCHES "SunOS")
add_definitions(-DOS_SOLARIS)
elseif(CMAKE_SYSTEM_NAME MATCHES "kFreeBSD")
add_definitions(-DOS_GNU_KFREEBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "FreeBSD")
add_definitions(-DOS_FREEBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "NetBSD")
add_definitions(-DOS_NETBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "OpenBSD")
add_definitions(-DOS_OPENBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "DragonFly")
add_definitions(-DOS_DRAGONFLYBSD)
elseif(CMAKE_SYSTEM_NAME MATCHES "Android")
add_definitions(-DOS_ANDROID)
elseif(CMAKE_SYSTEM_NAME MATCHES "Windows")
add_definitions(-DWIN32 -DOS_WIN -D_MBCS -DWIN64 -DNOMINMAX)
if(MINGW)
add_definitions(-D_WIN32_WINNT=_WIN32_WINNT_VISTA)
endif()
endif()
if(NOT WIN32)
add_definitions(-DROCKSDB_PLATFORM_POSIX -DROCKSDB_LIB_IO_POSIX)
endif()
option(WITH_FALLOCATE "build with fallocate" ON)
if(WITH_FALLOCATE)
CHECK_CXX_SOURCE_COMPILES("
#include <fcntl.h>
#include <linux/falloc.h>
int main() {
int fd = open(\"/dev/null\", 0);
fallocate(fd, FALLOC_FL_KEEP_SIZE, 0, 1024);
}
" HAVE_FALLOCATE)
if(HAVE_FALLOCATE)
add_definitions(-DROCKSDB_FALLOCATE_PRESENT)
endif()
endif()
CHECK_CXX_SOURCE_COMPILES("
#include <fcntl.h>
int main() {
int fd = open(\"/dev/null\", 0);
sync_file_range(fd, 0, 1024, SYNC_FILE_RANGE_WRITE);
}
" HAVE_SYNC_FILE_RANGE_WRITE)
if(HAVE_SYNC_FILE_RANGE_WRITE)
add_definitions(-DROCKSDB_RANGESYNC_PRESENT)
endif()
CHECK_CXX_SOURCE_COMPILES("
#include <pthread.h>
int main() {
(void) PTHREAD_MUTEX_ADAPTIVE_NP;
}
" HAVE_PTHREAD_MUTEX_ADAPTIVE_NP)
if(HAVE_PTHREAD_MUTEX_ADAPTIVE_NP)
add_definitions(-DROCKSDB_PTHREAD_ADAPTIVE_MUTEX)
endif()
include(CheckCXXSymbolExists)
if(CMAKE_SYSTEM_NAME MATCHES "^FreeBSD")
check_cxx_symbol_exists(malloc_usable_size malloc_np.h HAVE_MALLOC_USABLE_SIZE)
else()
check_cxx_symbol_exists(malloc_usable_size malloc.h HAVE_MALLOC_USABLE_SIZE)
endif()
if(HAVE_MALLOC_USABLE_SIZE)
add_definitions(-DROCKSDB_MALLOC_USABLE_SIZE)
endif()
check_cxx_symbol_exists(sched_getcpu sched.h HAVE_SCHED_GETCPU)
if(HAVE_SCHED_GETCPU)
add_definitions(-DROCKSDB_SCHED_GETCPU_PRESENT)
endif()
check_cxx_symbol_exists(getauxval auvx.h HAVE_AUXV_GETAUXVAL)
if(HAVE_AUXV_GETAUXVAL)
add_definitions(-DROCKSDB_AUXV_GETAUXVAL_PRESENT)
endif()
check_cxx_symbol_exists(F_FULLFSYNC "fcntl.h" HAVE_FULLFSYNC)
if(HAVE_FULLFSYNC)
add_definitions(-DHAVE_FULLFSYNC)
endif()
include_directories(${PROJECT_SOURCE_DIR})
include_directories(${PROJECT_SOURCE_DIR}/include)
if(WITH_FOLLY_DISTRIBUTED_MUTEX)
include_directories(${PROJECT_SOURCE_DIR}/third-party/folly)
endif()
find_package(Threads REQUIRED)
# Main library source code
set(SOURCES
cache/cache.cc
cache/cache_entry_roles.cc
cache/cache_key.cc
cache/cache_reservation_manager.cc
cache/clock_cache.cc
cache/lru_cache.cc
cache/sharded_cache.cc
db/arena_wrapped_db_iter.cc
db/blob/blob_fetcher.cc
db/blob/blob_file_addition.cc
db/blob/blob_file_builder.cc
db/blob/blob_file_cache.cc
db/blob/blob_file_garbage.cc
db/blob/blob_file_meta.cc
db/blob/blob_file_reader.cc
db/blob/blob_garbage_meter.cc
db/blob/blob_log_format.cc
db/blob/blob_log_sequential_reader.cc
db/blob/blob_log_writer.cc
db/blob/prefetch_buffer_collection.cc
db/builder.cc
db/c.cc
db/column_family.cc
db/compaction/compaction.cc
db/compaction/compaction_iterator.cc
db/compaction/compaction_picker.cc
db/compaction/compaction_job.cc
db/compaction/compaction_picker_fifo.cc
db/compaction/compaction_picker_level.cc
db/compaction/compaction_picker_universal.cc
db/compaction/sst_partitioner.cc
db/convenience.cc
db/db_filesnapshot.cc
db/db_impl/compacted_db_impl.cc
db/db_impl/db_impl.cc
db/db_impl/db_impl_write.cc
db/db_impl/db_impl_compaction_flush.cc
db/db_impl/db_impl_files.cc
db/db_impl/db_impl_open.cc
db/db_impl/db_impl_debug.cc
db/db_impl/db_impl_experimental.cc
db/db_impl/db_impl_readonly.cc
db/db_impl/db_impl_secondary.cc
db/db_info_dumper.cc
db/db_iter.cc
db/dbformat.cc
db/error_handler.cc
db/event_helpers.cc
db/experimental.cc
db/external_sst_file_ingestion_job.cc
db/file_indexer.cc
db/flush_job.cc
db/flush_scheduler.cc
db/forward_iterator.cc
db/import_column_family_job.cc
db/internal_stats.cc
db/logs_with_prep_tracker.cc
db/log_reader.cc
db/log_writer.cc
db/malloc_stats.cc
db/memtable.cc
db/memtable_list.cc
db/merge_helper.cc
db/merge_operator.cc
db/output_validator.cc
db/periodic_work_scheduler.cc
db/range_del_aggregator.cc
db/range_tombstone_fragmenter.cc
db/repair.cc
db/snapshot_impl.cc
db/table_cache.cc
db/table_properties_collector.cc
db/transaction_log_impl.cc
db/trim_history_scheduler.cc
db/version_builder.cc
db/version_edit.cc
db/version_edit_handler.cc
db/version_set.cc
db/wal_edit.cc
db/wal_manager.cc
db/write_batch.cc
db/write_batch_base.cc
db/write_controller.cc
db/write_thread.cc
env/composite_env.cc
env/env.cc
env/env_chroot.cc
env/env_encryption.cc
env/file_system.cc
env/file_system_tracer.cc
env/fs_remap.cc
env/mock_env.cc
env/unique_id_gen.cc
file/delete_scheduler.cc
file/file_prefetch_buffer.cc
file/file_util.cc
file/filename.cc
file/line_file_reader.cc
file/random_access_file_reader.cc
file/read_write_util.cc
file/readahead_raf.cc
file/sequence_file_reader.cc
file/sst_file_manager_impl.cc
file/writable_file_writer.cc
logging/auto_roll_logger.cc
logging/event_logger.cc
logging/log_buffer.cc
memory/arena.cc
memory/concurrent_arena.cc
memory/jemalloc_nodump_allocator.cc
memory/memkind_kmem_allocator.cc
memory/memory_allocator.cc
memtable/alloc_tracker.cc
memtable/hash_linklist_rep.cc
memtable/hash_skiplist_rep.cc
memtable/skiplistrep.cc
memtable/vectorrep.cc
memtable/write_buffer_manager.cc
monitoring/histogram.cc
monitoring/histogram_windowing.cc
monitoring/in_memory_stats_history.cc
monitoring/instrumented_mutex.cc
monitoring/iostats_context.cc
monitoring/perf_context.cc
monitoring/perf_level.cc
monitoring/persistent_stats_history.cc
monitoring/statistics.cc
monitoring/thread_status_impl.cc
monitoring/thread_status_updater.cc
monitoring/thread_status_util.cc
monitoring/thread_status_util_debug.cc
options/cf_options.cc
options/configurable.cc
options/customizable.cc
options/db_options.cc
options/options.cc
options/options_helper.cc
options/options_parser.cc
port/stack_trace.cc
table/adaptive/adaptive_table_factory.cc
table/block_based/binary_search_index_reader.cc
table/block_based/block.cc
table/block_based/block_based_filter_block.cc
table/block_based/block_based_table_builder.cc
table/block_based/block_based_table_factory.cc
table/block_based/block_based_table_iterator.cc
table/block_based/block_based_table_reader.cc
table/block_based/block_builder.cc
table/block_based/block_prefetcher.cc
table/block_based/block_prefix_index.cc
table/block_based/data_block_hash_index.cc
table/block_based/data_block_footer.cc
table/block_based/filter_block_reader_common.cc
table/block_based/filter_policy.cc
table/block_based/flush_block_policy.cc
table/block_based/full_filter_block.cc
table/block_based/hash_index_reader.cc
table/block_based/index_builder.cc
table/block_based/index_reader_common.cc
table/block_based/parsed_full_filter_block.cc
table/block_based/partitioned_filter_block.cc
table/block_based/partitioned_index_iterator.cc
table/block_based/partitioned_index_reader.cc
table/block_based/reader_common.cc
table/block_based/uncompression_dict_reader.cc
table/block_fetcher.cc
table/cuckoo/cuckoo_table_builder.cc
table/cuckoo/cuckoo_table_factory.cc
table/cuckoo/cuckoo_table_reader.cc
table/format.cc
table/get_context.cc
table/iterator.cc
table/merging_iterator.cc
table/meta_blocks.cc
table/persistent_cache_helper.cc
table/plain/plain_table_bloom.cc
table/plain/plain_table_builder.cc
table/plain/plain_table_factory.cc
table/plain/plain_table_index.cc
table/plain/plain_table_key_coding.cc
table/plain/plain_table_reader.cc
table/sst_file_dumper.cc
table/sst_file_reader.cc
table/sst_file_writer.cc
table/table_factory.cc
table/table_properties.cc
table/two_level_iterator.cc
table/unique_id.cc
test_util/sync_point.cc
test_util/sync_point_impl.cc
test_util/testutil.cc
test_util/transaction_test_util.cc
tools/block_cache_analyzer/block_cache_trace_analyzer.cc
tools/dump/db_dump_tool.cc
tools/io_tracer_parser_tool.cc
tools/ldb_cmd.cc
tools/ldb_tool.cc
tools/sst_dump_tool.cc
tools/trace_analyzer_tool.cc
trace_replay/block_cache_tracer.cc
trace_replay/io_tracer.cc
trace_replay/trace_record_handler.cc
trace_replay/trace_record_result.cc
trace_replay/trace_record.cc
trace_replay/trace_replay.cc
util/coding.cc
util/compaction_job_stats_impl.cc
util/comparator.cc
util/compression_context_cache.cc
util/concurrent_task_limiter_impl.cc
util/crc32c.cc
util/dynamic_bloom.cc
util/hash.cc
util/murmurhash.cc
util/random.cc
util/rate_limiter.cc
util/ribbon_config.cc
util/slice.cc
util/file_checksum_helper.cc
util/status.cc
util/string_util.cc
util/thread_local.cc
util/threadpool_imp.cc
util/xxhash.cc
utilities/backupable/backupable_db.cc
utilities/blob_db/blob_compaction_filter.cc
utilities/blob_db/blob_db.cc
utilities/blob_db/blob_db_impl.cc
utilities/blob_db/blob_db_impl_filesnapshot.cc
utilities/blob_db/blob_dump_tool.cc
utilities/blob_db/blob_file.cc
utilities/cache_dump_load.cc
utilities/cache_dump_load_impl.cc
utilities/cassandra/cassandra_compaction_filter.cc
utilities/cassandra/format.cc
utilities/cassandra/merge_operator.cc
utilities/checkpoint/checkpoint_impl.cc
utilities/compaction_filters.cc
utilities/compaction_filters/remove_emptyvalue_compactionfilter.cc
utilities/counted_fs.cc
utilities/debug.cc
utilities/env_mirror.cc
utilities/env_timed.cc
utilities/fault_injection_env.cc
utilities/fault_injection_fs.cc
utilities/fault_injection_secondary_cache.cc
utilities/leveldb_options/leveldb_options.cc
utilities/memory/memory_util.cc
utilities/merge_operators.cc
utilities/merge_operators/bytesxor.cc
utilities/merge_operators/max.cc
utilities/merge_operators/put.cc
utilities/merge_operators/sortlist.cc
utilities/merge_operators/string_append/stringappend.cc
utilities/merge_operators/string_append/stringappend2.cc
utilities/merge_operators/uint64add.cc
utilities/object_registry.cc
utilities/option_change_migration/option_change_migration.cc
utilities/options/options_util.cc
utilities/persistent_cache/block_cache_tier.cc
utilities/persistent_cache/block_cache_tier_file.cc
utilities/persistent_cache/block_cache_tier_metadata.cc
utilities/persistent_cache/persistent_cache_tier.cc
utilities/persistent_cache/volatile_tier_impl.cc
utilities/simulator_cache/cache_simulator.cc
utilities/simulator_cache/sim_cache.cc
utilities/table_properties_collectors/compact_on_deletion_collector.cc
utilities/trace/file_trace_reader_writer.cc
utilities/trace/replayer_impl.cc
utilities/transactions/lock/lock_manager.cc
utilities/transactions/lock/point/point_lock_tracker.cc
utilities/transactions/lock/point/point_lock_manager.cc
utilities/transactions/lock/range/range_tree/range_tree_lock_manager.cc
utilities/transactions/lock/range/range_tree/range_tree_lock_tracker.cc
utilities/transactions/optimistic_transaction_db_impl.cc
utilities/transactions/optimistic_transaction.cc
utilities/transactions/pessimistic_transaction.cc
utilities/transactions/pessimistic_transaction_db.cc
utilities/transactions/snapshot_checker.cc
utilities/transactions/transaction_base.cc
utilities/transactions/transaction_db_mutex_impl.cc
utilities/transactions/transaction_util.cc
utilities/transactions/write_prepared_txn.cc
utilities/transactions/write_prepared_txn_db.cc
utilities/transactions/write_unprepared_txn.cc
utilities/transactions/write_unprepared_txn_db.cc
utilities/ttl/db_ttl_impl.cc
utilities/wal_filter.cc
utilities/write_batch_with_index/write_batch_with_index.cc
utilities/write_batch_with_index/write_batch_with_index_internal.cc)
list(APPEND SOURCES
utilities/transactions/lock/range/range_tree/lib/locktree/concurrent_tree.cc
utilities/transactions/lock/range/range_tree/lib/locktree/keyrange.cc
utilities/transactions/lock/range/range_tree/lib/locktree/lock_request.cc
utilities/transactions/lock/range/range_tree/lib/locktree/locktree.cc
utilities/transactions/lock/range/range_tree/lib/locktree/manager.cc
utilities/transactions/lock/range/range_tree/lib/locktree/range_buffer.cc
utilities/transactions/lock/range/range_tree/lib/locktree/treenode.cc
utilities/transactions/lock/range/range_tree/lib/locktree/txnid_set.cc
utilities/transactions/lock/range/range_tree/lib/locktree/wfg.cc
utilities/transactions/lock/range/range_tree/lib/standalone_port.cc
utilities/transactions/lock/range/range_tree/lib/util/dbt.cc
utilities/transactions/lock/range/range_tree/lib/util/memarena.cc)
message(STATUS "ROCKSDB_PLUGINS: ${ROCKSDB_PLUGINS}")
if ( ROCKSDB_PLUGINS )
string(REPLACE " " ";" PLUGINS ${ROCKSDB_PLUGINS})
foreach (plugin ${PLUGINS})
add_subdirectory("plugin/${plugin}")
foreach (src ${${plugin}_SOURCES})
list(APPEND SOURCES plugin/${plugin}/${src})
set_source_files_properties(
plugin/${plugin}/${src}
PROPERTIES COMPILE_FLAGS "${${plugin}_COMPILE_FLAGS}")
endforeach()
foreach (path ${${plugin}_INCLUDE_PATHS})
include_directories(${path})
endforeach()
foreach (lib ${${plugin}_LIBS})
list(APPEND THIRDPARTY_LIBS ${lib})
endforeach()
foreach (link_path ${${plugin}_LINK_PATHS})
link_directories(AFTER ${link_path})
endforeach()
set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} ${${plugin}_CMAKE_SHARED_LINKER_FLAGS}")
set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} ${${plugin}_CMAKE_EXE_LINKER_FLAGS}")
endforeach()
endif()
if(HAVE_SSE42 AND NOT MSVC)
set_source_files_properties(
util/crc32c.cc
PROPERTIES COMPILE_FLAGS "-msse4.2 -mpclmul")
endif()
if(CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64")
list(APPEND SOURCES
util/crc32c_ppc.c
util/crc32c_ppc_asm.S)
endif(CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64")
if(HAS_ARMV8_CRC)
list(APPEND SOURCES
util/crc32c_arm64.cc)
endif(HAS_ARMV8_CRC)
if(WIN32)
list(APPEND SOURCES
port/win/io_win.cc
port/win/env_win.cc
port/win/env_default.cc
port/win/port_win.cc
port/win/win_logger.cc
port/win/win_thread.cc)
if(WITH_XPRESS)
list(APPEND SOURCES
port/win/xpress_win.cc)
endif()
if(WITH_JEMALLOC)
list(APPEND SOURCES
port/win/win_jemalloc.cc)
endif()
else()
list(APPEND SOURCES
port/port_posix.cc
env/env_posix.cc
env/fs_posix.cc
env/io_posix.cc)
endif()
if(WITH_FOLLY_DISTRIBUTED_MUTEX)
list(APPEND SOURCES
third-party/folly/folly/detail/Futex.cpp
third-party/folly/folly/synchronization/AtomicNotification.cpp
third-party/folly/folly/synchronization/DistributedMutex.cpp
third-party/folly/folly/synchronization/ParkingLot.cpp
third-party/folly/folly/synchronization/WaitOptions.cpp)
endif()
set(ROCKSDB_STATIC_LIB rocksdb${ARTIFACT_SUFFIX})
set(ROCKSDB_SHARED_LIB rocksdb-shared${ARTIFACT_SUFFIX})
option(ROCKSDB_BUILD_SHARED "Build shared versions of the RocksDB libraries" ON)
if(WIN32)
set(SYSTEM_LIBS ${SYSTEM_LIBS} shlwapi.lib rpcrt4.lib)
else()
set(SYSTEM_LIBS ${CMAKE_THREAD_LIBS_INIT})
endif()
add_library(${ROCKSDB_STATIC_LIB} STATIC ${SOURCES} ${BUILD_VERSION_CC})
target_link_libraries(${ROCKSDB_STATIC_LIB} PRIVATE
${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
if(ROCKSDB_BUILD_SHARED)
add_library(${ROCKSDB_SHARED_LIB} SHARED ${SOURCES} ${BUILD_VERSION_CC})
target_link_libraries(${ROCKSDB_SHARED_LIB} PRIVATE
${THIRDPARTY_LIBS} ${SYSTEM_LIBS})
if(WIN32)
set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
COMPILE_DEFINITIONS "ROCKSDB_DLL;ROCKSDB_LIBRARY_EXPORTS")
if(MSVC)
set_target_properties(${ROCKSDB_STATIC_LIB} PROPERTIES
COMPILE_FLAGS "/Fd${CMAKE_CFG_INTDIR}/${ROCKSDB_STATIC_LIB}.pdb")
set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
COMPILE_FLAGS "/Fd${CMAKE_CFG_INTDIR}/${ROCKSDB_SHARED_LIB}.pdb")
endif()
else()
set_target_properties(${ROCKSDB_SHARED_LIB} PROPERTIES
LINKER_LANGUAGE CXX
VERSION ${rocksdb_VERSION}
SOVERSION ${rocksdb_VERSION_MAJOR}
OUTPUT_NAME "rocksdb${ARTIFACT_SUFFIX}")
endif()
endif()
if(ROCKSDB_BUILD_SHARED AND NOT WIN32)
set(ROCKSDB_LIB ${ROCKSDB_SHARED_LIB})
else()
set(ROCKSDB_LIB ${ROCKSDB_STATIC_LIB})
endif()
option(WITH_JNI "build with JNI" OFF)
# Tests are excluded from Release builds
CMAKE_DEPENDENT_OPTION(WITH_TESTS "build with tests" ON
"CMAKE_BUILD_TYPE STREQUAL Debug" OFF)
option(WITH_BENCHMARK_TOOLS "build with benchmarks" ON)
option(WITH_CORE_TOOLS "build with ldb and sst_dump" ON)
option(WITH_TOOLS "build with tools" ON)
if(WITH_TESTS OR WITH_BENCHMARK_TOOLS OR WITH_TOOLS OR WITH_JNI OR JNI)
include_directories(SYSTEM ${PROJECT_SOURCE_DIR}/third-party/gtest-1.8.1/fused-src)
endif()
if(WITH_JNI OR JNI)
message(STATUS "JNI library is enabled")
add_subdirectory(${CMAKE_CURRENT_SOURCE_DIR}/java)
else()
message(STATUS "JNI library is disabled")
endif()
# Installation and packaging
if(WIN32)
option(ROCKSDB_INSTALL_ON_WINDOWS "Enable install target on Windows" OFF)
endif()
if(NOT WIN32 OR ROCKSDB_INSTALL_ON_WINDOWS)
if(CMAKE_INSTALL_PREFIX_INITIALIZED_TO_DEFAULT)
if(${CMAKE_SYSTEM_NAME} STREQUAL "Linux")
# Change default installation prefix on Linux to /usr
set(CMAKE_INSTALL_PREFIX /usr CACHE PATH "Install path prefix, prepended onto install directories." FORCE)
endif()
endif()
include(GNUInstallDirs)
include(CMakePackageConfigHelpers)
set(package_config_destination ${CMAKE_INSTALL_LIBDIR}/cmake/rocksdb)
configure_package_config_file(
${CMAKE_CURRENT_LIST_DIR}/cmake/RocksDBConfig.cmake.in RocksDBConfig.cmake
INSTALL_DESTINATION ${package_config_destination}
)
write_basic_package_version_file(
RocksDBConfigVersion.cmake
VERSION ${rocksdb_VERSION}
COMPATIBILITY SameMajorVersion
)
install(DIRECTORY include/rocksdb COMPONENT devel DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}")
install(DIRECTORY "${PROJECT_SOURCE_DIR}/cmake/modules" COMPONENT devel DESTINATION ${package_config_destination})
install(
TARGETS ${ROCKSDB_STATIC_LIB}
EXPORT RocksDBTargets
COMPONENT devel
ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}"
INCLUDES DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
)
if(ROCKSDB_BUILD_SHARED)
install(
TARGETS ${ROCKSDB_SHARED_LIB}
EXPORT RocksDBTargets
COMPONENT runtime
ARCHIVE DESTINATION "${CMAKE_INSTALL_LIBDIR}"
RUNTIME DESTINATION "${CMAKE_INSTALL_BINDIR}"
LIBRARY DESTINATION "${CMAKE_INSTALL_LIBDIR}"
INCLUDES DESTINATION "${CMAKE_INSTALL_INCLUDEDIR}"
)
endif()
install(
EXPORT RocksDBTargets
COMPONENT devel
DESTINATION ${package_config_destination}
NAMESPACE RocksDB::
)
install(
FILES
${CMAKE_CURRENT_BINARY_DIR}/RocksDBConfig.cmake
${CMAKE_CURRENT_BINARY_DIR}/RocksDBConfigVersion.cmake
COMPONENT devel
DESTINATION ${package_config_destination}
)
endif()
option(WITH_ALL_TESTS "Build all test, rather than a small subset" ON)
if(WITH_TESTS OR WITH_BENCHMARK_TOOLS)
add_subdirectory(third-party/gtest-1.8.1/fused-src/gtest)
add_library(testharness STATIC
test_util/mock_time_env.cc
test_util/testharness.cc)
target_link_libraries(testharness gtest)
endif()
if(WITH_TESTS)
set(TESTS
db/db_basic_test.cc
env/env_basic_test.cc
)
if(WITH_ALL_TESTS)
list(APPEND TESTS
cache/cache_reservation_manager_test.cc
cache/cache_test.cc
cache/lru_cache_test.cc
db/blob/blob_counting_iterator_test.cc
db/blob/blob_file_addition_test.cc
db/blob/blob_file_builder_test.cc
db/blob/blob_file_cache_test.cc
db/blob/blob_file_garbage_test.cc
db/blob/blob_file_reader_test.cc
db/blob/blob_garbage_meter_test.cc
db/blob/db_blob_basic_test.cc
db/blob/db_blob_compaction_test.cc
db/blob/db_blob_corruption_test.cc
db/blob/db_blob_index_test.cc
db/column_family_test.cc
db/compact_files_test.cc
db/compaction/clipping_iterator_test.cc
db/compaction/compaction_job_stats_test.cc
db/compaction/compaction_job_test.cc
db/compaction/compaction_iterator_test.cc
db/compaction/compaction_picker_test.cc
db/compaction/compaction_service_test.cc
db/comparator_db_test.cc
db/corruption_test.cc
db/cuckoo_table_db_test.cc
db/db_with_timestamp_basic_test.cc
db/db_block_cache_test.cc
db/db_bloom_filter_test.cc
db/db_compaction_filter_test.cc
db/db_compaction_test.cc
db/db_dynamic_level_test.cc
db/db_flush_test.cc
db/db_inplace_update_test.cc
db/db_io_failure_test.cc
db/db_iter_test.cc
db/db_iter_stress_test.cc
db/db_iterator_test.cc
db/db_kv_checksum_test.cc
db/db_log_iter_test.cc
db/db_memtable_test.cc
db/db_merge_operator_test.cc
db/db_merge_operand_test.cc
db/db_options_test.cc
db/db_properties_test.cc
db/db_range_del_test.cc
db/db_secondary_test.cc
db/db_sst_test.cc
db/db_statistics_test.cc
db/db_table_properties_test.cc
db/db_tailing_iter_test.cc
db/db_test.cc
db/db_test2.cc
db/db_logical_block_size_cache_test.cc
db/db_universal_compaction_test.cc
db/db_wal_test.cc
db/db_with_timestamp_compaction_test.cc
db/db_write_test.cc
db/dbformat_test.cc
db/deletefile_test.cc
db/error_handler_fs_test.cc
db/obsolete_files_test.cc
db/external_sst_file_basic_test.cc
db/external_sst_file_test.cc
db/fault_injection_test.cc
db/file_indexer_test.cc
db/filename_test.cc
db/flush_job_test.cc
db/listener_test.cc
db/log_test.cc
db/manual_compaction_test.cc
db/memtable_list_test.cc
db/merge_helper_test.cc
db/merge_test.cc
db/options_file_test.cc
db/perf_context_test.cc
db/periodic_work_scheduler_test.cc
db/plain_table_db_test.cc
db/prefix_test.cc
db/range_del_aggregator_test.cc
db/range_tombstone_fragmenter_test.cc
db/repair_test.cc
db/table_properties_collector_test.cc
db/version_builder_test.cc
db/version_edit_test.cc
db/version_set_test.cc
db/wal_manager_test.cc
db/wal_edit_test.cc
db/write_batch_test.cc
db/write_callback_test.cc
db/write_controller_test.cc
env/env_test.cc
env/io_posix_test.cc
env/mock_env_test.cc
file/delete_scheduler_test.cc
file/prefetch_test.cc
file/random_access_file_reader_test.cc
logging/auto_roll_logger_test.cc
logging/env_logger_test.cc
logging/event_logger_test.cc
memory/arena_test.cc
memory/memory_allocator_test.cc
memtable/inlineskiplist_test.cc
memtable/skiplist_test.cc
memtable/write_buffer_manager_test.cc
monitoring/histogram_test.cc
monitoring/iostats_context_test.cc
monitoring/statistics_test.cc
monitoring/stats_history_test.cc
options/configurable_test.cc
options/customizable_test.cc
options/options_settable_test.cc
options/options_test.cc
table/block_based/block_based_filter_block_test.cc
table/block_based/block_based_table_reader_test.cc
table/block_based/block_test.cc
table/block_based/data_block_hash_index_test.cc
table/block_based/full_filter_block_test.cc
table/block_based/partitioned_filter_block_test.cc
table/cleanable_test.cc
table/cuckoo/cuckoo_table_builder_test.cc
table/cuckoo/cuckoo_table_reader_test.cc
table/merger_test.cc
table/sst_file_reader_test.cc
table/table_test.cc
table/block_fetcher_test.cc
test_util/testutil_test.cc
trace_replay/block_cache_tracer_test.cc
trace_replay/io_tracer_test.cc
tools/block_cache_analyzer/block_cache_trace_analyzer_test.cc
tools/io_tracer_parser_test.cc
tools/ldb_cmd_test.cc
tools/reduce_levels_test.cc
tools/sst_dump_test.cc
tools/trace_analyzer_test.cc
util/autovector_test.cc
util/bloom_test.cc
util/coding_test.cc
util/crc32c_test.cc
util/defer_test.cc
util/dynamic_bloom_test.cc
util/file_reader_writer_test.cc
util/filelock_test.cc
util/hash_test.cc
util/heap_test.cc
util/random_test.cc
util/rate_limiter_test.cc
util/repeatable_thread_test.cc
util/ribbon_test.cc
util/slice_test.cc
util/slice_transform_test.cc
util/timer_queue_test.cc
util/timer_test.cc
util/thread_list_test.cc
util/thread_local_test.cc
util/work_queue_test.cc
utilities/backupable/backupable_db_test.cc
utilities/blob_db/blob_db_test.cc
utilities/cassandra/cassandra_functional_test.cc
utilities/cassandra/cassandra_format_test.cc
utilities/cassandra/cassandra_row_merge_test.cc
utilities/cassandra/cassandra_serialize_test.cc
utilities/checkpoint/checkpoint_test.cc
utilities/memory/memory_test.cc
utilities/merge_operators/string_append/stringappend_test.cc
utilities/object_registry_test.cc
utilities/option_change_migration/option_change_migration_test.cc
utilities/options/options_util_test.cc
utilities/persistent_cache/hash_table_test.cc
utilities/persistent_cache/persistent_cache_test.cc
utilities/simulator_cache/cache_simulator_test.cc
utilities/simulator_cache/sim_cache_test.cc
utilities/table_properties_collectors/compact_on_deletion_collector_test.cc
utilities/transactions/optimistic_transaction_test.cc
utilities/transactions/transaction_test.cc
utilities/transactions/lock/point/point_lock_manager_test.cc
utilities/transactions/write_prepared_transaction_test.cc
utilities/transactions/write_unprepared_transaction_test.cc
utilities/transactions/lock/range/range_locking_test.cc
utilities/ttl/ttl_test.cc
utilities/write_batch_with_index/write_batch_with_index_test.cc
)
endif()
if(WITH_FOLLY_DISTRIBUTED_MUTEX)
list(APPEND TESTS third-party/folly/folly/synchronization/test/DistributedMutexTest.cpp)
endif()
set(TESTUTIL_SOURCE
db/db_test_util.cc
monitoring/thread_status_updater_debug.cc
table/mock_table.cc
utilities/cassandra/test_utils.cc
)
enable_testing()
add_custom_target(check COMMAND ${CMAKE_CTEST_COMMAND})
set(TESTUTILLIB testutillib${ARTIFACT_SUFFIX})
add_library(${TESTUTILLIB} STATIC ${TESTUTIL_SOURCE})
target_link_libraries(${TESTUTILLIB} ${ROCKSDB_LIB})
if(MSVC)
set_target_properties(${TESTUTILLIB} PROPERTIES COMPILE_FLAGS "/Fd${CMAKE_CFG_INTDIR}/testutillib${ARTIFACT_SUFFIX}.pdb")
endif()
set_target_properties(${TESTUTILLIB}
PROPERTIES EXCLUDE_FROM_DEFAULT_BUILD_RELEASE 1
EXCLUDE_FROM_DEFAULT_BUILD_MINRELEASE 1
EXCLUDE_FROM_DEFAULT_BUILD_RELWITHDEBINFO 1
)
foreach(sourcefile ${TESTS})
get_filename_component(exename ${sourcefile} NAME_WE)
add_executable(${exename}${ARTIFACT_SUFFIX} ${sourcefile})
set_target_properties(${exename}${ARTIFACT_SUFFIX}
PROPERTIES EXCLUDE_FROM_DEFAULT_BUILD_RELEASE 1
EXCLUDE_FROM_DEFAULT_BUILD_MINRELEASE 1
EXCLUDE_FROM_DEFAULT_BUILD_RELWITHDEBINFO 1
OUTPUT_NAME ${exename}${ARTIFACT_SUFFIX}
)
target_link_libraries(${exename}${ARTIFACT_SUFFIX} testutillib${ARTIFACT_SUFFIX} testharness gtest ${THIRDPARTY_LIBS} ${ROCKSDB_LIB})
if(NOT "${exename}" MATCHES "db_sanity_test")
gtest_discover_tests(${exename} DISCOVERY_TIMEOUT 120)
add_dependencies(check ${exename}${ARTIFACT_SUFFIX})
endif()
endforeach(sourcefile ${TESTS})
if(WIN32)
# C executables must link to a shared object
if(ROCKSDB_BUILD_SHARED)
set(ROCKSDB_LIB_FOR_C ${ROCKSDB_SHARED_LIB})
else()
set(ROCKSDB_LIB_FOR_C OFF)
endif()
else()
set(ROCKSDB_LIB_FOR_C ${ROCKSDB_LIB})
endif()
if(ROCKSDB_LIB_FOR_C)
set(C_TESTS db/c_test.c)
add_executable(c_test db/c_test.c)
target_link_libraries(c_test ${ROCKSDB_LIB_FOR_C} testharness)
add_test(NAME c_test COMMAND c_test${ARTIFACT_SUFFIX})
add_dependencies(check c_test)
endif()
endif()
if(WITH_BENCHMARK_TOOLS)
add_executable(db_bench${ARTIFACT_SUFFIX}
tools/simulated_hybrid_file_system.cc
tools/db_bench.cc
tools/db_bench_tool.cc)
target_link_libraries(db_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} ${THIRDPARTY_LIBS})
add_executable(cache_bench${ARTIFACT_SUFFIX}
cache/cache_bench.cc
cache/cache_bench_tool.cc)
target_link_libraries(cache_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} ${GFLAGS_LIB})
add_executable(memtablerep_bench${ARTIFACT_SUFFIX}
memtable/memtablerep_bench.cc)
target_link_libraries(memtablerep_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} ${GFLAGS_LIB})
add_executable(range_del_aggregator_bench${ARTIFACT_SUFFIX}
db/range_del_aggregator_bench.cc)
target_link_libraries(range_del_aggregator_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} ${GFLAGS_LIB})
add_executable(table_reader_bench${ARTIFACT_SUFFIX}
table/table_reader_bench.cc)
target_link_libraries(table_reader_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} testharness ${GFLAGS_LIB})
add_executable(filter_bench${ARTIFACT_SUFFIX}
util/filter_bench.cc)
target_link_libraries(filter_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} ${GFLAGS_LIB})
add_executable(hash_table_bench${ARTIFACT_SUFFIX}
utilities/persistent_cache/hash_table_bench.cc)
target_link_libraries(hash_table_bench${ARTIFACT_SUFFIX}
${ROCKSDB_LIB} ${GFLAGS_LIB})
endif()
if(WITH_CORE_TOOLS OR WITH_TOOLS)
add_subdirectory(tools)
add_custom_target(core_tools
DEPENDS ${core_tool_deps})
endif()
if(WITH_TOOLS)
add_subdirectory(db_stress_tool)
add_custom_target(tools
DEPENDS ${tool_deps})
endif()
option(WITH_EXAMPLES "build with examples" OFF)
if(WITH_EXAMPLES)
add_subdirectory(examples)
endif()
option(WITH_BENCHMARK "build benchmark tests" OFF)
if(WITH_BENCHMARK)
add_subdirectory(${PROJECT_SOURCE_DIR}/microbench/)
endif()
]]> sh csh ksh bash tcsh zsh rc es
Job control N Y Y Y Y Y N N
Aliases N Y Y Y Y Y N N
Shell functions Y(1) N Y Y N Y Y Y
"Sensible" Input/Output redirection Y N Y Y N Y Y Y
Directory stack N Y Y Y Y Y F F
Command history N Y Y Y Y Y L L
Command line editing N N Y Y Y Y L L
Vi Command line editing N N Y Y Y(3) Y L L
Emacs Command line editing N N Y Y Y Y L L
Rebindable Command line editing N N N Y Y Y L L
User name look up N Y Y Y Y Y L L
Login/Logout watching N N N N Y Y F F
Filename completion N Y(1) Y Y Y Y L L
Username completion N Y(2) Y Y Y Y L L
Hostname completion N Y(2) Y Y Y Y L L
History completion N N N Y Y Y L L
Fully programmable Completion N N N N Y Y N N
Mh Mailbox completion N N N N(4) N(6) N(6) N N
Co Processes N N Y N N Y N N
Builtin artithmetic evaluation N Y Y Y Y Y N N
Can follow symbolic links invisibly N N Y Y Y Y N N
Periodic command execution N N N N Y Y N N
Custom Prompt (easily) N N Y Y Y Y Y Y
Sun Keyboard Hack N N N N N Y N N
Spelling Correction N N N N Y Y N N
Process Substitution N N N Y(2) N Y Y Y
Underlying Syntax sh csh sh sh csh sh rc rc
Freely Available N N N(5) Y Y Y Y Y
Checks Mailbox N Y Y Y Y Y F F
Tty Sanity Checking N N N N Y Y N N
Can cope with large argument lists Y N Y Y Y Y Y Y
Has non-interactive startup file N Y Y(7) Y(7) Y Y N N
Has non-login startup file N Y Y(7) Y Y Y N N
Can avoid user startup files N Y N Y N Y Y Y
Can specify startup file N N Y Y N N N N
Low level command redefinition N N N N N N N Y
Has anonymous functions N N N N N N Y Y
List Variables N Y Y N Y Y Y Y
Full signal trap handling Y N Y Y N Y Y Y
File no clobber ability N Y Y Y Y Y N F
Local variables N N Y Y N Y Y Y
Lexically scoped variables N N N N N N N Y
Exceptions N N N N N N N Y
$ sudo find -L /sys/class/backlight -maxdepth 2 -name '*brightness*'
/sys/class/backlight/thinkpad_screen/brightness
$ cd /sys/class/backlight/thinkpad_screen
$ sudo echo 3 > brightness
An error occurred while redirecting file 'brightness'
open: Permission denied
$ echo 3 | sudo tee brightness
$ echo 1 | sudo tee /sys/class/leds/input6::scrolllock/brightness
$ command 2>file
$ command > file 2>&1
$ command > /dev/null
mkfifo
或 mknod
命令来创建一个命名管道。创建出来的文件类型比较特殊,是p类型。表示这是一个管道文件。有了这个管道文件,系统中就有了对一个管道的全局名称,于是任何两个不相关的进程都可以通过这个管道文件进行通信[root@iZ8vb09drusrfas7dkou1tZ ~]# mkfifo pipe
[root@iZ8vb09drusrfas7dkou1tZ ~]# ls -l
total 8
drwxr-xr-x 8 root root 4096 Dec 8 17:58 oneinstack
prw-r--r-- 1 root root 0 Feb 8 16:12 pipe
-rw-r--r-- 1 root root 475 Dec 8 17:58 ReadMe
[root@iZ8vb09drusrfas7dkou1tZ ~]# ls -l pipe
prw-r--r-- 1 root root 0 Feb 8 16:12 pipe
[root@iZ8vb09drusrfas7dkou1tZ ~]# echo xxxx > pipe
pipe()
,而创建命名管道的函数是 mkfifo()
。使用 mknod()
系统调用并指定文件类型为为 S_IFIFO
也可以创建一个FIFO。#include <unistd.h>
int pipe(int pipefd[2]);
#include <sys/types.h>
#include <sys/stat.h>
int mkfifo(const char *filename, mode_t mode);
int mknod(const char *filename, mode_t mode | S_IFIFO, (dev_t)0);
root@aep-shunzi:~/tut_0# cat run_exp.sh
echo "Dummy Experiement Output"
echo "Throughput $RANDOM ops/s"
root@aep-shunzi:~/tut_0# ./run_exp.sh
Dummy Experiement Output
Throughput 18447 ops/s
root@aep-shunzi:~/tut_0# ./run_exp.sh
Dummy Experiement Output
Throughput 18224 ops/s
root@aep-shunzi:~/tut_0# cat run_dummy.sh
# 输入试验次数作为参数
exp_times=$1
# 输入最终的输出文件作为参数
result_file=$2
# 清空输出文件
echo "" > $result_file
# 循环执行
for i in `seq 1 $exp_times`
do
# 执行吞吐量统计
# sed 取对应的第二行
# awk 输出对应的第二列值,即吞吐量对应的数值
# 重定向到输出文件
bash ./run_exp.sh | sed -n '2 p' | awk '{print $2}' >> $result_file
done
# 执行求平均值的函数
awk 'BEGIN{cnt=0} {sum+=$1;cnt++;} END {print sum/cnt}' $result_file
root@aep-shunzi:~/tut_0# ls
run_dummy.sh run_exp.sh test.txt
root@aep-shunzi:~/tut_0# ./run_dummy.sh 100 avg-thr.txt
16249.1
[root@iZ8vb09drusrfas7dkou1tZ ~]# cat rand_req.sh
# 随机生成延迟
echo "$RANDOM us"
[root@iZ8vb09drusrfas7dkou1tZ ~]# cat run_req.sh
# 循环执行 10001 次
for i in `seq 0 10000`
do
bash ./rand_req.sh
done
[root@iZ8vb09drusrfas7dkou1tZ ~]# cat run_exp.sh
# 重定向到 req_time 文件
bash ./run_req.sh > req_time
# 接受输入参数作为读取的文件
resultfile=$1
# 按照数字进行排序得到对应排序后的文件
sort -g $resultfile > $resultfile-sort
# 统计 总共的行数
# 提取 wordcount 的输出的第一列 得到行数
lineno=`wc -l $resultfile | awk '{print $1}'`
echo $lineno
# 计算 P99 所在的位置
p99line=$[lineno * 99 / 100]
# 输出 P99 所在的行答应对应的延迟
p99lat=`sed -n "$p99line p " $resultfile-sort`
echo $p99lat
# 统计 CDF 计算概率密度分布
# 读取对应的排序后的延迟文件,从第一行开始,间隔一行取样
# 输出对应的行号所对应的比例值 和 对应的延迟
# 重定向到新的文件中
awk -v total="$lineno" 'BEGIN{line=0} {line++;if(line%2 == 0) print (line/total" "$1)}' $resultfile-sort > $resultfile-cdf
]]>$ sudo find -L /sys/class/backlight -maxdepth 2 -name '*brightness*'
/sys/class/backlight/thinkpad_screen/brightness
$ cd /sys/class/backlight/thinkpad_screen
$ sudo echo 3 > brightness
An error occurred while redirecting file 'brightness'
open: Permission denied
$ echo 3 | sudo tee brightness
$ echo 1 | sudo tee /sys/class/leds/input6::scrolllock/brightness
foo=bar
echo "$foo"
# 打印 bar
echo '$foo'
# 打印 $foo
for file in $(ls)
l首先将调用ls ,然后遍历得到的这些返回值diff <(ls foo) <(ls bar)
会显示文件夹 foo 和 bar 中文件的区别#!/bin/bash
echo "Starting program at $(date)" # date会被替换成日期和时间
echo "Running program $0 with $# arguments with pid $$"
# 遍历参数列表
for file in "$@"; do
grep foobar "$file" > /dev/null 2> /dev/null
# 如果模式没有找到,则grep退出状态为 1
# 我们将标准输出流和标准错误流重定向到Null,因为我们并不关心这些信息
# 前面的 grep 命令没有找到 foobar 的时候,追加一个 foorbar
if [[ $? -ne 0 ]]; then
echo "File $file does not have any foobar, adding one"
echo "# foobar" >> "$file"
fi
done
cp /path/to/project/{foo,bar,baz}.sh /newpath
mv *{.py,.sh} folder
touch {foo,bar}/{a..h}
ShellCheck
# 查找文件
# 查找所有名称为src的文件夹
find . -name src -type d
# 查找所有名称为src的文件夹,不区分大小写
find . -name src -iname -type d
# 查找所有文件夹路径中包含test的python文件
find . -path '*/test/*.py' -type f
# 查找前一天修改的所有文件
find . -mtime -1
# 查找所有大小在500k至10M的tar.gz文件
find . -size +500k -size -10M -name '*.tar.gz'
# 操作文件
# 删除全部扩展名为.tmp 的文件
find . -name '*.tmp' -exec rm {} \;
# 查找全部的 PNG 文件并将其转换为 JPG
find . -name '*.png' -exec convert {} {}.jpg \;
# 查找和pwd相关的所有文件
locate pwd
# 搜索etc目录下所有以sh开头的文件
locate /etc/sh
# 搜索etc目录下,所有以i开头的文件
locate /etc/i
# 指定显示数量,限定结果个数
locate -n 3 passwd
# 查找不需要区分大小写时,使用 -i 选项
locate -i -n 5 passwd
# 正则表达式匹配
locate -r ^/var/lib/rpm
# 查看数据库统计信息
locate -S
# 查看passwd统计数量
locate -c passwd
# update相关配置文件
vim /etc/updatedb.conf
# 查找所有使用了 requests 库的文件
rg -t py 'import requests'
# 查找所有没有写 shebang 的文件(包含隐藏文件)
rg -u --files-without-match "^#!"
# 查找所有的foo字符串,并打印其之后的5行
rg foo -A 5
# 打印匹配的统计信息(匹配的行和文件的数量)
rg --stats PATTERN
ls -ahtl --color=auto
-a
所有文件包括隐藏文件-h
文件打印以人类可以理解的格式输出 (例如,使用454M 而不是 454279954)-t
文件以最近访问顺序排序--color=auto
以彩色文本显示输出结果find . -type f -name "*.html" | xargs -d '\n' tar -cvzf html.zip
# 查找 .html 文件并打包成 zip
xargs
使用标准输入中的内容作为参数,主要用于 不接受标准输入作为参数的命令
-d
更改分隔符来分解标准输入,默认分隔符为空格。tar
打包
find . -type f -mmin -60 -print0 | xargs -0 ls -lt | head -10
# 递归的查找文件夹中最近使用的文件
-mmin
60
print
和 print0
-print
在每一个输出后会添加一个回车换行符-print0
则不会-xarg 0
xargs 也用 NULL 字符来作为记录的分隔符i
R
v
一般 V
行 Ctrl-V
块:
移动 (正常模式下)
hjkl
(左, 下, 上, 右)w
(下一个词), b
(词初), e
(词尾)0
(行初), ^
(第一个非空格字符), $
(行尾)H
(屏幕首行), M
(屏幕中间), L
(屏幕底部)Ctrl-u
(上翻), Ctrl-d
(下翻)gg
(文件头), G
(文件尾):{行数}<CR>
或者 {行数}G
({行数}为行数)%
(找到配对,比如括号或者 /* */ 之类的注释对)f{字符}
, t{字符}
, F{字符}
, T{字符}
,
/ ;
用于导航匹配/{正则表达式}
, n
/ N
用于导航匹配编辑
i
进入插入模式O
/ o
在之上/之下插入行d{移动命令}
删除 {移动命令}
dw
删除词d$
删除到行尾d0
删除到行头c{移动命令}
改变 {移动命令}
cw
改变词x
删除字符(等同于 dl
)s
替换字符(等同于 xi
)+
操作
d
删除 或者 c
改变u
撤销, <C-r>
重做y
复制 / “yank” (其他一些命令比如 d 也会复制)p
粘贴计数:你可以用一个计数来结合“名词”和“动词”,这会执行指定操作若干次。
修饰语:你可以用修饰语改变“名词”的意义。修饰语有 i,表示“内部”或者“在内“,和 a, 表示”周围“。
ci(
改变当前括号内的内容ci[
改变当前方括号内的内容da'
删除一个单引号字符串, 包括周围的单引号" Comments in Vimscript start with a `"`.
" If you open this file in Vim, it'll be syntax highlighted for you.
" Vim is based on Vi. Setting `nocompatible` switches from the default
" Vi-compatibility mode and enables useful Vim functionality. This
" configuration option turns out not to be necessary for the file named
" '~/.vimrc', because Vim automatically enters nocompatible mode if that file
" is present. But we're including it here just in case this config file is
" loaded some other way (e.g. saved as `foo`, and then Vim started with
" `vim -u foo`)."
" Vim基于Vi。设置' nocompatible '从默认的Vi兼容模式切换,并启用有用的Vim功能.
set nocompatible
" 语法高亮"
" Turn on syntax highlighting.
syntax on
" 禁用默认的Vim启动消息"
" Disable the default Vim startup message.
set shortmess+=I
" Show line numbers.
set number
" 这使得可以使用相对行编号模式。 启用number和relativenumber后,当前行显示真实的行号,而所有其他行(上面和下面)都相对于当前行编号。 这是很有用的,因为你可以通过{count}k向上或{count}j向下,一眼就能知道向上或向下跳转到特定行需要哪些计数。 "
" This enables relative line numbering mode. With both number and
" relativenumber enabled, the current line shows the true line number, while
" all other lines (above and below) are numbered relative to the current line.
" This is useful because you can tell, at a glance, what count is needed to
" jump up or down to a particular line, by {count}k to go up or {count}j to go
" down.
set relativenumber
" 总是在底部显示状态行,即使你只有一个窗口打开"
" Always show the status line at the bottom, even if you only have one window open.
set laststatus=2
" The backspace key has slightly unintuitive behavior by default. For example,
" by default, you can't backspace before the insertion point set with 'i'.
" This configuration makes backspace behave more reasonably, in that you can
" backspace over anything.
set backspace=indent,eol,start
" By default, Vim doesn't let you hide a buffer (i.e. have a buffer that isn't
" shown in any window) that has unsaved changes. This is to prevent you from "
" forgetting about unsaved changes and then quitting e.g. via `:qa!`. We find
" hidden buffers helpful enough to disable this protection. See `:help hidden`
" for more information on this.
set hidden
" 默认情况下,退格键有一些不太直观的行为。 例如,在默认情况下,不能在使用'i'设置的插入点之前进行退格操作。 这个配置使退格操作更合理,因为你可以在任何东西上退格。"
" This setting makes search case-insensitive when all characters in the string
" being searched are lowercase. However, the search becomes case-sensitive if
" it contains any capital letters. This makes searching more convenient.
set ignorecase
set smartcase
"在输入时启用搜索,而不是等到按回车键 "
" Enable searching as you type, rather than waiting till you press enter.
set incsearch
" 解绑定一些无用的/恼人的默认键绑定"
" Unbind some useless/annoying default key bindings.
nmap Q <Nop> " 'Q' in normal mode enters Ex mode. You almost never want this.
" Disable audible bell because it's annoying.
set noerrorbells visualbell t_vb=
" 禁用鼠标支持,因为它很烦人。"
" Enable mouse support. You should avoid relying on this too much, but it can
" sometimes be convenient.
set mouse+=a
"尽量避免像使用方向键移动这样的坏习惯。 这不是唯一可能的坏习惯。 例如,按住h/j/k/l键移动,而不是使用更有效的移动命令,也是一个坏习惯。 前者可以通过.vimrc执行,而我们不知道如何防止后者。 在正常模式下做这个… "
" Try to prevent bad habits like using the arrow keys for movement. This is
" not the only possible bad habit. For example, holding down the h/j/k/l keys
" for movement, rather than using more efficient movement commands, is also a
" bad habit. The former is enforceable through a .vimrc, while we don't know
" how to prevent the latter.
" Do this in normal mode...
nnoremap <Left> :echoe "Use h"<CR>
nnoremap <Right> :echoe "Use l"<CR>
nnoremap <Up> :echoe "Use k"<CR>
nnoremap <Down> :echoe "Use j"<CR>
" ...and in insert mode
inoremap <Left> <ESC>:echoe "Use h"<CR>
inoremap <Right> <ESC>:echoe "Use l"<CR>
inoremap <Up> <ESC>:echoe "Use k"<CR>
inoremap <Down> <ESC>:echoe "Use j"<CR>
set editing-mode vi
:s
(替换)命令
%s/foo/bar/g
在整个文件中将 foo 全局替换成 bar%s/\[.*\](\(.*\))/\1/g
将有命名的 Markdown 链接替换成简单 URLs:sp
/ :vsp
来分割窗口q{字符}
来开始在寄存器{字符}中录制宏q
停止录制@{字符}
重放宏{计数}@{字符}
执行一个宏{计数}次q{字符}q
清除宏@{字符}
来递归调用该宏 (在录制完成之前不会有任何操作)$ ssh -p 7777 root@localhost 'journalctl | grep sshd | grep "Disconnected from"' | less
$ ssh myserver 'journalctl | grep sshd | grep "Disconnected from"' > ssh.log
$ less ssh.log
ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed 's/.*Disconnected from //'
.
除换行符之外的”任意单个字符”*
匹配前面字符零次或多次+
匹配前面字符一次或多次[abc]
匹配 a, b 和 c 中的任意一个(RX1|RX2)
任何能够匹配RX1 或 RX2的结果^
行首$
行尾/.*Disconnected from /
这个正则表达式可以匹配任何以若干任意字符开头,并接着包含”Disconnected from “的字符串。"Jan 17 03:13:00 thesquareplanet.com sshd[2631]: Disconnected from invalid user Disconnected from 46.97.239.16 port 55920 [preauth]
perl -pe 's/.*?Disconnected from //'
| sed -E 's/.*Disconnected from (invalid |authenticating )?user .* [^ ]+ port [0-9]+( \[preauth\])?$//'
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
sort
排序uniq -c
把连续出现的行折叠为一行并使用出现次数作为前缀ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
| sort | uniq -c
sort -n
会按照数字顺序对输入进行排序(默认情况下是按照字典序排序 -k1,1
则表示“仅基于以空格分割的第一列进行排序”。,n
部分表示“仅排序到第n个部分”,默认情况是到行尾head
来代替 tail
,或者使用sort -r
来进行倒序排序。ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
| sort | uniq -c
| sort -nk1,1 | tail -n10
awk
处理文本,'{print $2}' 即为打印第二个部分paste
合并行(-s
),并指定一个分隔符进行分割 (-d
)ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
| sort | uniq -c
| sort -nk1,1 | tail -n10
| awk '{print $2}' | paste -sd,
$1 == 1
匹配 uniq -c 计数,即只是登陆一次$2 ~ /^c[^ ]*e$/
匹配对应的正则表达式print $2
打印出该用户名wc -l
统计输出结果的行数 | awk '$1 == 1 && $2 ~ /^c[^ ]*e$/ { print $2 }' | wc -l
BEGIN { rows = 0 }
$1 == 1 && $2 ~ /^c[^ ]*e$/ { rows += $1 }
END { print rows }
| paste -sd+ | bc -l
# or
echo "2*($(data | paste -sd+))" | bc -l
ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
| sort | uniq -c
| awk '{print $1}' | R --slave -e 'x <- scan(file="stdin", quiet=TRUE); summary(x)'
ssh myserver journalctl
| grep sshd
| grep "Disconnected from"
| sed -E 's/.*Disconnected from (invalid |authenticating )?user (.*) [^ ]+ port [0-9]+( \[preauth\])?$/\2/'
| sort | uniq -c
| sort -nk1,1 | tail -n10
| gnuplot -p -e 'set boxwidth 0.5; plot "-" using 1:xtic(2) with boxes'
rustup toolchain list | grep nightly | grep -vE "nightly-x86" | sed 's/-x86.*//' | xargs rustup toolchain uninstall
ffmpeg -loglevel panic -i /dev/video0 -frames 1 -f image2 -
| convert - -colorspace gray -
| gzip
| ssh mymachine 'gzip -d | tee copy.jpg | env DISPLAY=:0 feh -'
Ctrl-C
Ctrl-\
可以发送该信号。#!/usr/bin/env python
import signal, time
def handler(signum, time):
print("\nI got a SIGINT, but I am not stopping")
signal.signal(signal.SIGINT, handler)
i = 0
while True:
time.sleep(.1)
print("\r{}".format(i), end="")
i += 1
^
是我们在终端输入 Ctrl
时的表示形式:$ python sigint.py
24^C
I got a SIGINT, but I am not stopping
26^C
I got a SIGINT, but I am not stopping
30^\[1] 39913 quit python sigint.pyƒ
kill
命令, 它的语法是: kill -TERM <PID>
。Ctrl-Z
会让 shell 发送 SIGTSTP
信号。kill -STOP
fg
或 bg
命令恢复暂停的工作。它们分别表示在前台继续或在后台继续。jobs
命令会列出当前终端会话中尚未完成的全部任务。您可以使用 pid
引用这些任务(也可以用 pgrep
找出 pid)。更加符合直觉的操作是您可以使用百分号 + 任务编号(jobs 会打印任务编号)来选取该任务。如果要选择最近的一个任务,可以使用 $!
这一特殊参数。&
后缀可以让命令在直接在后台运行,这使得您可以直接在 shell 中继续做其他操作,不过它此时还是会使用 shell 的标准输出,这一点有时会比较恼人(这种情况可以使用 shell 重定向处理)。Ctrl-Z
,然后紧接着再输入 bg
。注意,后台的进程仍然是您的终端进程的子进程,一旦您关闭终端(会发送另外一个信号 SIGHUP
),这些后台的进程也会终止。为了防止这种情况发生,您可以使用 nohup (一个用来忽略 SIGHUP 的封装) 来运行程序。针对已经运行的程序,可以使用 disown 。除此之外,您可以使用终端多路复用器来实现,下一章节我们会进行详细地探讨。$ sleep 1000
^Z
[1] + 18653 suspended sleep 1000
$ nohup sleep 2000 &
[2] 18745
appending output to nohup.out
$ jobs
[1] + suspended sleep 1000
[2] - running nohup sleep 2000
$ bg %1
[1] - 18653 continued sleep 1000
$ jobs
[1] - running sleep 1000
[2] + running nohup sleep 2000
$ kill -STOP %1
[1] + 18653 suspended (signal) sleep 1000
$ jobs
[1] + suspended (signal) sleep 1000
[2] - running nohup sleep 2000
$ kill -SIGHUP %1
[1] + 18653 hangup sleep 1000
$ jobs
[2] + running nohup sleep 2000
$ kill -SIGHUP %2
$ jobs
[2] + running nohup sleep 2000
$ kill %2
[2] + 18745 terminated nohup sleep 2000
$ jobs
tmux
这类的终端多路复用器可以允许我们基于面板和标签分割出多个终端窗口,这样您便可以同时与多个 shell 会话进行交互。<C-b> x
这样的组合,即需要先按下 Ctrl+b
,松开后再按下 x
。tmux 中对象的继承结构如下:
tmux
开始一个新的会话tmux new -s NAME
以指定名称开始一个新的会话tmux ls
列出当前所有会话<C-b> d
,将当前会话分离tmux a
重新连接最后一个会话。您也可以通过 -t
来指定具体的会话<C-b> c
创建一个新的窗口,使用 <C-d>
关闭<C-b> N
跳转到第 N 个窗口,注意每个窗口都是有编号的<C-b> p
切换到前一个窗口<C-b> n
切换到下一个窗口<C-b> ,
重命名当前窗口<C-b> w
列出当前所有窗口<C-b> "
水平分割<C-b> %
垂直分割<C-b> <方向>
切换到指定方向的面板,<方向>
指的是键盘上的方向键<C-b> z
切换当前面板的缩放<C-b> [
开始往回卷动屏幕。您可以按下空格键来开始选择,回车键复制选中的部分<C-b> <空格>
在不同的面板排布间切换alias alias_name="command_to_alias arg1 arg2"
# 创建常用命令的缩写
alias ll="ls -lh"
# 能够少输入很多
alias gs="git status"
alias gc="git commit"
alias v="vim"
# 手误打错命令也没关系
alias sl=ls
# 重新定义一些命令行的默认行为
alias mv="mv -i" # -i prompts before overwrite
alias mkdir="mkdir -p" # -p make parent dirs as needed
alias df="df -h" # -h prints human readable format
# 别名可以组合使用
alias la="ls -A"
alias lla="la -l"
# 在忽略某个别名
\ls
# 或者禁用别名
unalias la
# 获取别名的定义
alias ll
# 会打印 ll='ls -lh'
~/.vimrc
。也正因为此,它们默认是隐藏文件,ls并不会显示它们)。.bashrc
或 .bash_profile
来进行配置。在文件中您可以添加需要在启动时执行的命令,例如上文我们讲到过的别名,或者是您的环境变量。export PATH="$PATH:/path/to/program/bin"
的命令,这样才能确保这些程序能够被 shell 找到。~/.bashrc
, ~/.bash_profile
~/.gitconfig
~/.vimrc
和 ~/.vim
目录~/.ssh/config
~/.tmux.conf
if [[ "$(uname)" == "Linux" ]]; then {do_something}; fi
# 使用和 shell 相关的配置时先检查当前 shell 类型
if [[ "$SHELL" == "zsh" ]]; then {do_something}; fi
# 您也可以针对特定的设备进行配置
if [[ "$(hostname)" == "myServer" ]]; then {do_something}; fi
[include]
path = ~/.gitconfig_local
~/.gitconfig_local
来包含与该设备相关的特定配置。您甚至应该创建一个单独的代码仓库来管理这些与设备相关的配置。# Test if ~/.aliases exists and source it
if [ -f ~/.aliases ]; then
source ~/.aliases
fi
ssh
的一个经常被忽视的特性是它可以直接远程执行命令。 ssh foobar@server ls
可以直接在用 foobar
的命令下执行 ls
命令。 想要配合管道来使用也可以, ssh foobar@server ls | grep PATTERN
会在本地查询远端 ls
的输出而 ls | ssh foobar@server grep PATTERN
会在远端对本地 ls 输出的结果进行查询。ssh-keygen
ssh-keygen -y -f /path/to/key
.cat .ssh/id_ed25519.pub | ssh foobar@remote 'cat >> ~/.ssh/authorized_keys'
ssh-copy-id -i .ssh/id_ed25519.pub foobar@remote
ssh+tee
最简单的方法是执行 ssh 命令,然后通过这样的方法利用标准输入实现 cat localfile | ssh remote_server tee serverfile
。回忆一下,tee 命令会将标准输出写入到一个文件;scp
: 当需要拷贝大量的文件或目录时,使用scp 命令则更加方便,因为它可以方便的遍历相关路径。语法如下:scp path/to/local_file remote_host:path/to/remote_file
;rsync
:对 scp 进行了改进,它可以检测本地和远端的文件以防止重复拷贝。它还可以提供一些诸如符号连接、权限管理等精心打磨的功能。甚至还可以基于 --partial
标记实现断点续传。rsync 的语法和scp类似;ssh -L 9999:localhost:8888 foobar@remote_server
。这样只需要访问本地的 localhost:9999 即可。Host vm
User foobar
HostName 172.16.174.141
Port 2222
IdentityFile ~/.ssh/id_ed25519
LocalForward 9999 localhost:8888
# 在配置文件中也可以使用通配符
Host *.mit.edu
User foobaz
~/.ssh/config
文件来创建别名,类似 scp、rsync和mosh的这些命令都可以读取这个配置并将设置转换为对应的命令行选项。~/.ssh/config
文件也可以被当作配置文件,而且一般情况下也是可以被导入其他配置文件的。不过,如果您将其公开到互联网上,那么其他人都将会看到您的服务器地址、用户名、开放端口等等。这些信息可能会帮助到那些企图攻击您系统的黑客,所以请务必三思。/etc/ssh/sshd_config
。您可以在这里配置免密认证、修改 ssh 端口、开启 X11 转发等等。 您也可以为每个用户单独指定配置。<root> (tree)
|
+- foo (tree)
| |
| + bar.txt (blob, contents = "hello world")
|
+- baz.txt (blob, contents = "git is wonderful")
o <-- o <-- o <-- o
^
\
--- o <-- o
o <-- o <-- o <-- o <---- o
^ /
\ v
--- o <-- o
// 文件就是一组数据
type blob = array<byte>
// 一个包含文件和目录的目录
type tree = map<string, tree | blob>
// 每个提交都包含一个父辈,元数据和顶层树
type commit = struct {
parent: array<commit>
author: string
message: string
snapshot: tree
}
type object = blob | tree | commit
objects = map<string, object>
def store(object):
id = sha1(object)
objects[id] = object
def load(id):
return objects[id]
git cat-file -p 698281bc680d1995c5f4caaf3359721a5a58d48d
来进行可视化),看上去是这样的:100644 blob 4448adbf7ecd394f42ae135bbeed9676e894af85 baz.txt
040000 tree c68d233a33c5c06e0340e4c224f0afca87c8ce87 foo
git cat-file -p 4448adbf7ecd394f42ae135bbeed9676e894af85
,即通过哈希值查看 baz.txt 的内容,会得到以下信息:git is wonderful
references = map<string, string>
def update_reference(name, id):
references[name] = id
def read_reference(name):
return references[name]
def load_reference(name_or_id):
if name_or_id in references:
return load(references[name_or_id])
else:
return load(name_or_id)
这样,Git 就可以使用诸如 “master” 这样人类可读的名称来表示历史记录中某个特定的提交,而不需要在使用一长串十六进制字符了。
有一个细节需要我们注意, 通常情况下,我们会想要知道“我们当前所在位置”,并将其标记下来。这样当我们创建新的快照的时候,我们就可以知道它的相对位置(如何设置它的“父辈”)。在 Git 中,我们当前的位置有一个特殊的索引,它就是 “HEAD”。
对象
和 引用
。git checkout master
; git reset --hard 5d83f9e
)git log --all --graph --decorate
可视化历史记录(有向无环图)git diff <revision> <filename>
显示某个文件两个版本之间的差异git mergetool
使用工具来处理合并冲突git rebase
将一系列补丁变基(rebase)为新的基线git clone --depth=1
浅克隆(shallow clone),不包括完整的版本历史信息git add -p
交互式暂存git rebase -i
交互式变基git blame
查看最后修改某行的人git bisect
通过二分查找搜索历史记录ANSI escape codes
,它是一系列的特殊字符,可以使您的 shell 改变输出结果的颜色。例如,执行 echo -e "\e[38;2;255;0;0mThis is red\e[0m"
会打印红色的字符串:This is red 。只要您的终端支持真彩色。如果您的终端不支持真彩色(例如 MacOS 的 Terminal.app),您可以使用支持更加广泛的 16 色,例如:”\e[31;1mThis is red\e[0m”
。#!/usr/bin/env bash
for R in $(seq 0 20 255); do
for G in $(seq 0 20 255); do
for B in $(seq 0 20 255); do
printf "\e[38;2;${R};${G};${B}m█\e[0m";
done
done
done
logger "Hello Logs"
# On macOS
log show --last 1m | grep Hello
# On Linux
journalctl --since "1m ago" | grep Hello
strace
,在 macOS 和 BSD 中可以使用 dtrace。dtrace 用起来可能有些别扭,因为它使用的是它自有的 D 语言,但是我们可以使用一个叫做 dtruss 的封装使其具有和 strace (更多信息参考 这里)类似的接口strace
或 dtruss 来显示ls 执行时,对stat 系统调用进行追踪对结果。若需要深入了解 strace,这篇文章 值得一读。# On Linux
sudo strace -e lstat ls -l > /dev/null
4
# On macOS
sudo dtruss -t lstat64_extended ls -l > /dev/null
import time
def foo():
return 42
for foo in range(5):
print(foo)
bar = 1
bar *= 0.2
time.sleep(60)
print(baz)
$ pyflakes foobar.py
foobar.py:6: redefinition of unused 'foo' from line 3
foobar.py:11: undefined name 'baz'
$ mypy foobar.py
foobar.py:6: error: Incompatible types in assignment (expression has type "int", variable has type "Callable[[], Any]")
foobar.py:9: error: Incompatible types in assignment (expression has type "float", variable has type "int")
foobar.py:11: error: Name 'baz' is not defined
Found 3 errors in 1 file (checked 1 source file)
import time, random
n = random.randint(1, 10) * 100
# 获取当前时间
start = time.time()
# 执行一些操作
print("Sleeping for {} ms".format(n))
time.sleep(n/1000)
# 比较当前时间和起始时间
print(time.time() - start)
# Output
# Sleeping for 500 ms
# 0.5713930130004883
$ time curl https://missing.csail.mit.edu &> /dev/null`
real 0m2.561s
user 0m0.015s
sys 0m0.012s
#!/usr/bin/env python
import sys, re
def grep(pattern, file):
with open(file, 'r') as f:
print(file)
for i, line in enumerate(f.readlines()):
pattern = re.compile(pattern)
match = pattern.search(line)
if match is not None:
print("{}: {}".format(i, line), end="")
if __name__ == '__main__':
times = int(sys.argv[1])
pattern = sys.argv[2]
for i in range(times):
for file in sys.argv[3:]:
grep(pattern, file)
$ python -m cProfile -s tottime grep.py 1000 '^(import|\s*def)[^,]*$' *.py
[omitted program output]
ncalls tottime percall cumtime percall filename:lineno(function)
8000 0.266 0.000 0.292 0.000 {built-in method io.open}
8000 0.153 0.000 0.894 0.000 grep.py:5(grep)
17000 0.101 0.000 0.101 0.000 {built-in method builtins.print}
8000 0.100 0.000 0.129 0.000 {method 'readlines' of '_io._IOBase' objects}
93000 0.097 0.000 0.111 0.000 re.py:286(_compile)
93000 0.069 0.000 0.069 0.000 {method 'search' of '_sre.SRE_Pattern' objects}
93000 0.030 0.000 0.141 0.000 re.py:231(compile)
17000 0.019 0.000 0.029 0.000 codecs.py:318(decode)
1 0.017 0.017 0.911 0.911 grep.py:3(<module>)
[omitted lines]
#!/usr/bin/env python
import requests
from bs4 import BeautifulSoup
# 这个装饰器会告诉行分析器
# 我们想要分析这个函数
@profile
def get_urls():
response = requests.get('https://missing.csail.mit.edu')
s = BeautifulSoup(response.content, 'lxml')
urls = []
for url in s.find_all('a'):
urls.append(url['href'])
if __name__ == '__main__':
get_urls()
$ kernprof -l -v a.py
Wrote profile results to urls.py.lprof
Timer unit: 1e-06 s
Total time: 0.636188 s
File: a.py
Function: get_urls at line 5
Line # Hits Time Per Hit % Time Line Contents
==============================================================
5 @profile
6 def get_urls():
7 1 613909.0 613909.0 96.5 response = requests.get('https://missing.csail.mit.edu')
8 1 21559.0 21559.0 3.4 s = BeautifulSoup(response.content, 'lxml')
9 1 2.0 2.0 0.0 urls = []
10 25 685.0 27.4 0.1 for url in s.find_all('a'):
11 24 33.0 1.4 0.0 urls.append(url['href'])
像 C 或者 C++ 这样的语言,内存泄漏会导致您的程序在使用完内存后不去释放它。为了应对内存类的 Bug,我们可以使用类似 Valgrind 这样的工具来检查内存泄漏问题。
对于 Python 这类具有垃圾回收机制的语言,内存分析器也是很有用的,因为对于某个对象来说,只要有指针还指向它,那它就不会被回收。
下面这个例子及其输出,展示了 memory-profiler 是如何工作的(注意装饰器和 line-profiler 类似)。
@profile
def my_func():
a = [1] * (10 ** 6)
b = [2] * (2 * 10 ** 7)
del b
return a
if __name__ == '__main__':
my_func()
$ python -m memory_profiler example.py
Line # Mem usage Increment Line Contents
==============================================
3 @profile
4 5.97 MB 0.00 MB def my_func():
5 13.61 MB 7.64 MB a = [1] * (10 ** 6)
6 166.20 MB 152.59 MB b = [2] * (2 * 10 ** 7)
7 13.61 MB -152.59 MB del b
8 13.61 MB 0.00 MB return a
perf
命令将 CPU 的区别进行了抽象,它不会报告时间和内存的消耗,而是报告与您的程序相关的系统事件。perf list
- 列出可以被 pref 追踪的事件;perf stat COMMAND ARG1 ARG2
- 收集与某个进程或指令相关的事件;perf record COMMAND ARG1 ARG2
- 记录命令执行的采样信息并将统计数据储存在perf.data中;perf report
- 格式化并打印 perf.data 中的数据。使用分析器来分析真实的程序时,由于软件的复杂性,其输出结果中将包含大量的信息。人类是一种视觉动物,非常不善于阅读大量的文字。因此很多工具都提供了可视化分析器输出结果的功能。
对于采样分析器来说,常见的显示 CPU 分析数据的形式是 火焰图,火焰图会在 Y 轴显示函数调用关系,并在 X 轴显示其耗时的比例。火焰图同时还是可交互的,您可以深入程序的某一具体部分,并查看其栈追踪(您可以尝试点击下面的图片)。
调用图和控制流图可以显示子程序之间的关系,它将函数作为节点并把函数调用作为边。将它们和分析器的信息(例如调用次数、耗时等)放在一起使用时,调用图会变得非常有用,它可以帮助我们分析程序的流程。 在 Python 中您可以使用 pycallgraph 来生成这些图片。
$ hyperfine --warmup 3 'fd -e jpg' 'find . -iname "*.jpg"'
Benchmark #1: fd -e jpg
Time (mean ± σ): 51.4 ms ± 2.9 ms [User: 121.0 ms, System: 160.5 ms]
Range (min … max): 44.2 ms … 60.1 ms 56 runs
Benchmark #2: find . -iname "*.jpg"
Time (mean ± σ): 1.126 s ± 0.101 s [User: 141.1 ms, System: 956.1 ms]
Range (min … max): 0.975 s … 1.287 s 10 runs
Summary
'fd -e jpg' ran
21.89 ± 2.33 times faster than 'find . -iname "*.jpg"'
hash(value: array<byte>) -> vector<byte, N> (N对于该函数固定)
$ printf 'hello' | sha1sum
aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d
$ printf 'hello' | sha1sum
aaf4c61ddcc5e8a2dabede0f3b482cd9aea9434d
$ printf 'Hello' | sha1sum
f7ff9e8b7bb2e09b70935a5d785e0cc5d9d0abf0
F(password + salt)
KDF(input + salt)
,并与存储的哈希值对比。keygen() -> key (这是一个随机方法)
encrypt(plaintext: array<byte>, key) -> array<byte> (输出密文)
decrypt(ciphertext: array<byte>, key) -> array<byte> (输出明文)
encrypt()
输出的密文ciphertext 很难在不知道key的情况下得出明文plaintext。decrypt()
有明显的正确性。因为功能要求给定密文及其密钥,解密方法必须输出明文:decrypt(encrypt(m, k), k) = m
。key = KDF(passphrase)
,然后存储 encrypt(file, key)
。keygen() -> (public key, private key) (这是一个随机方法)
encrypt(plaintext: array<byte>, public key) -> array<byte> (输出密文)
decrypt(ciphertext: array<byte>, private key) -> array<byte> (输出明文)
sign(message: array<byte>, private key) -> array<byte> (生成签名)
verify(message: array<byte>, signature: array<byte>, public key) -> bool (验证签名是否是由和这个公钥相关的私钥生成的)
decrypt(encrypt(m, public key), private key) = m
。verify(message, signature, public key)
返回为真的签名。verify(message, sign(message, private key), public key) = true
。整体结构
ChameleonDB 是一个 Value 存储在存储日志上的 KV 存储,同时 keys(或者他们的 hash 值)以及对应值的地址信息被存储在相应的持久性索引中,如下图所示。
KV 项根据请求到达顺序被批量写入到 Value Log 中,持久性索引是一个有多个 shards 的高度并行的结构,每个 shard 有自己的多层结构以及自己的 compaction 操作。Keys 根据相应的 hash 值分布在这些 shards
压缩涉及昂贵的合并操作。我们对压缩应用了三种优化: 数据重用、异步 I/O 和 FPGA 加载。
数据重用。我们在压缩过程中重用区段和数据块,以减少合并两个相邻级别(即Leveli和Leveli+1)所需的 I/O 数量。为了增加重用的机会并使其有效,我们将区段的大小减少到 2 MB,并进一步将区段划分为多个 16 KB 的数据块。如果压缩所涉及的一个区段的键范围与其他区段的键范围不重叠,则只需更新其相应的元数据索引(如第3.1.3节所介绍的)就可以重用它,而不需要在磁盘上实际移动它。我们在图8中展示了一个示例,其中Level1的3个区段(键范围为[1,35]、[80,200]和[210,280])被压缩为来自Level2的5个区段(键范围为[1,30]、[50,70]、[100,130]、[150,170]和[190,205])。下面我们列出了不同的重用情况:
Asynchronous I/O:在区段级别,压缩操作由三个不相交的阶段组成:
第一和第三阶段是I/O阶段,而第二阶段是计算密集型阶段。我们在第一和第三阶段发出异步I/O请求。第二阶段实现为第一I/O阶段的回调函数。当多个压缩并行运行时,第二阶段的执行与其他阶段的执行重叠,以隐藏I/O
FPGA offloading:我们引入 FPGA 来加速压缩,并减少cpu上的资源消耗。通过上面介绍的两个优化,在cpu上运行的压缩仍然会消耗多个线程。因为压缩工作在lsm树的两个连续级别的独立区段对上,所以压缩任务在这些区段对的粒度上理论上能并行的。因此,它可以被分割成多个小任务。我们把这样小的任务放到 FPGA 上,并以流的方式处理它们。每个压缩区段都被传输回磁盘。通过这样的加载,CPU 线程从合并区段的沉重负担中得到释放。因此,我们能够分配更多的线程来处理并发事务。这种基于 FPGA 的压实加速的细节超出了本文的范围,将另行探讨。