Rui Li

简单记录MongoDB从6升级到8的过程

Mon, 27 Oct 2025 22:30:27 +0800

好久没打开MongoDB Compass浏览数据库了，这次登陆后提示MongoDB 6生命周期已经结束了，遂决定升级至8。自部署的MongoDB是一个三节点的ReplicaSet，需要同时升级三个机器。

步骤1：先用MongoDump备份数据。

步骤2：ssh到每个机器上docker compose down掉每个节点的mongodb容器。

步骤3：先修改docker-compose.yaml中的mongodb版本从6到7。

步骤4：启动每个机器上的mongodb，docker compose up -d。

步骤5：启动成功后，在主节点上执行rs.status()查看各个节点的状态，没有问题后执行db.adminCommand({setFeatureCompatibilityVersion: '7.0'})设置MongoDB的FCV(Feature Compatibility Version)到7，这时候会告知你开弓没有回头箭，需要二次确认。

步骤6：执行db.adminCommand({getParameter: 1, featureCompatibilityVersion: 1})检查FCV是不是设置到7了。

步骤7：down掉每个节点，设置docker镜像为8.0，启动，设置FCV到8.0，检查是否正常。

升级成功！数据无价，谨慎操作。

dnSpyEx MCP插件

Tue, 21 Oct 2025 00:06:06 +0800

最近在用dnSpy逆向配合Claude Code开发BepInEx插件的时候，感觉手动告诉Claude Code有哪些变量、成员函数以及代码非常麻烦。既然现在有MCP这个协议，为什么不能让整个流程自动化起来呢？

于是就花了一个晚上让Claude Code搓了一个dnSpyEx MCP插件出来，插件里面内置了一些工具用于获取二进制信息、反编译源码、某一类型的路径等，同时提供了BepInEx的文档作为Resources。

插件开源在dnSpy.Extension.MCP，由于dnSpyEx是使用GPL-3.0 license，所以本项目也沿用了GPL-3.0 license。

使用方法

首先从dnSpyEx下载最新的dnSpyEx二进制，截至目前是v6.5.1。

下载完成后，前往dnSpy.Extension.MCP的Release下载最新版本的插件文件，通常来说你应该下载dnSpy.Extension.MCP-net48.dll。下载完成后，将dll重命名并放置在dnSpy的bin/Extensions/dnSpy.Extension.MCP/dnSpy.Extension.MCP.x.dll路径下，你可能需要手动创建两级目录。然后启动dnSpyEx，你应该可以在设置中看到插件被加载。

插件默认监听http://localhost:3000，同时第一次启动你需要手动启用MCP服务器。勾选启用后，你应该可以使用MCP Inspector工具来调试MCP服务器，传输类型选择Streamable HTTP，URL填写http://localhost:3000/。

有了MCP之后，你可以让Claude Code全自动帮你编写BepInEx插件，不过最好还是自己探索一下关键的变量和类名告诉大模型，效果会更好。

`1`	`claude mcp add --transport http dnspy http://localhost:3000/`

Have fun!

写在信用卡被盗刷并成功找回损失之后

Sun, 15 Dec 2024 20:16:45 +0800

记一次MongoDB灾难恢复

Wed, 13 Mar 2024 20:13:30 +0800

TL;DR 参考官方文档：Reconfigure a Replica Set with Unavailable Members

在今天查看报告的时候，发现数据库的数据有异常，一开始还未是客户端的磁盘满了，登录客户端发现磁盘还有空间，但是日志说因为网络问题连不上MongoDB ReplicaSet。日志类似：

`1`	`No replica set members match selector "Primary()", Timeout: 30s`

网络超时就很奇怪，因为目前架构下Tailscale组的网不应该出现网络问题。于是登上一个MongoDB服务器，尝试使用MongoSh登录发现也提示超时。看日志如下：

`1`	`"msg":"Not starting an election, since we are not electable","attr":{"reason":"Not standing for election because I cannot see a majority (mask 0x1)"}`

目前ReplicaSet里面有4个节点，其中1个节点忘了续费，1个节点的提供商跑路了。这下难崩了😅，看来是主节点突然掉了，剩下活着的两个节点因为数量没超过集群节点数的一半，于是不知道咋办了。这时候还是可以MongoSh登录的，但是不能加replicaSet参数，否则连上的节点会尝试去找Primary，进而网络超时。

解决方案就是在活着的服务器上MongoSh登录集群管理员账户，但是不加replicaSet参数。登录后强制重新配置MongoDB ReplicaSet，删除失效节点即可。

实现SSH登录消息推送

Mon, 29 Jan 2024 11:28:57 +0800

最近有一些安全焦虑，虽然在我的服务器上一般都做了如下配置：

ufw limit ssh
fail2ban sshd
sshd_config禁用密码登录、对于root实行prohibit-password

但是还是想做一个SSH登录后自动向我推送消息的功能。经过一番调查发现PAM Plugin可以实现，但是自己用C写一个模组感觉比较麻烦，于是选择了现成的项目：jeroennijhof/pam_script。pam script是一个现成的pam plugin，可以在配置文件中指定被调用时需要执行的脚本/程式位置，同时通过环境变量传入远端IP等信息。结合fly.io上托管的Webhook接口，就可以比较方便实现SSH登录推送。

整体来看技术难度并不高，以Ubuntu 22.04为例需要做如下配置：

安装libpam-script包
修改/etc/pam.d/common-auth，在底部添加：auth optional pam_script.so
将需要执行的脚本或者二进制文件放置在/usr/share/libpam-script/pam_script_ses_open

一个Rust的二进制例子如下：

use serde::{Deserialize, Serialize};
use reqwest;

#[derive(Serialize, Deserialize)]
struct Payload {
    hostname: String,
    pam_service: String,
    pam_type: String,
    pam_user: String,
    pam_ruser: String,
    pam_rhost: String,
    pam_tty: String,
}

#[tokio::main]
async fn main() {
    let payload = Payload {
        hostname: gethostname::gethostname().into_string().unwrap(),
        pam_service: std::env::var("PAM_SERVICE").unwrap_or("".to_string()),
        pam_type: std::env::var("PAM_TYPE").unwrap_or("".to_string()),
        pam_user: std::env::var("PAM_USER").unwrap_or("".to_string()),
        pam_ruser: std::env::var("PAM_RUSER").unwrap_or("".to_string()),
        pam_rhost: std::env::var("PAM_RHOST").unwrap_or("".to_string()),
        pam_tty: std::env::var("PAM_TTY").unwrap_or("".to_string()),
    };

    let client = reqwest::Client::new();
    let _ = client.post(env!("WEBHOOK_URL"))
        .json(&payload)
        .send()
        .await;
}

而部署在FaaS平台上的Webhook编程语言我使用的是C#，可以比较简单实现多线程以及跨线程的消息队列。同时使用了IPinfo.io的SDK来获取IP对应的物理地址一并推送。这个方案有个缺点是除了ssh以外，cron等服务也会调用pam进行验证从而推送消息，可能尝试在服务端过滤PAM_SERVICE。

在Modal上部署Bert-VITS2模型

Sun, 07 Jan 2024 13:29:08 +0800

受制于原仓库开源协议，本文涉及到的代码全部使用AGPL-3.0协议开源。

在很早之前得知了Modal平台融资的消息，当时就训练了一个Bert-VITS2模型部署到了上面，后来随着模型的版本快速迭代就没去管了。现在终于到了V2.3一个比较稳定的版本，于是训练了三天模型又花了半天时间给部署到了Modal平台上面，现在写篇博客介绍一下，同时开源代码。

Modal是个FaaS平台，目前主要支持Python代码，可以在云端部署函数或者Flask API后，通过HTTP API和SDK去调用对应的函数。FaaS平台并不是一个新鲜东西，但是Modal创新点在于支持GPU且可以按秒计费。这就非常适合我们部署一些模型来按需调用，模型不需要一直占用我们的GPU显存，且调用成本会非常低，单次text2speech大概0.02美元左右。对于一些大模型，平台也提供了A100等大显存GPU。

另外，每个用户每月似乎是有30美元的免费额度。

部署

仓库开源在：KernelErr/Bert-VITS2的modal分支上。主要代码如下：

import os
import modal
import fastapi
import fastapi.staticfiles
from modal_const import CACHE_PATH
from pydantic import BaseModel
from fastapi.responses import Response

stub = modal.Stub("bert-vits2")
web_app = fastapi.FastAPI()


def download_model_weights():
    import requests
    from huggingface_hub import snapshot_download
    from transformers.utils import move_cache

    model_files = ["config.json", "D_88000.pth", "DUR_88000.pth", "G_88000.pth", "WD_88000.pth"]
    os.makedirs(CACHE_PATH, exist_ok=True)
    for model_file in model_files:
        rsp = requests.get("HOSTURL" + model_file)
        rsp.raise_for_status()
        with open(os.path.join(CACHE_PATH, model_file), "wb") as f:
            for chunk in rsp.iter_content(chunk_size=1024*1024): 
                if chunk:
                    f.write(chunk)

    os.makedirs(CACHE_PATH + "/bert/deberta-v2-large-japanese-char-wwm", exist_ok=True)
    snapshot_download(
        "ku-nlp/deberta-v2-large-japanese-char-wwm",
        local_dir=CACHE_PATH + "/bert/deberta-v2-large-japanese-char-wwm",
    )
    move_cache()

    os.makedirs(CACHE_PATH + "/bert/chinese-roberta-wwm-ext-large", exist_ok=True)
    snapshot_download(
        "hfl/chinese-roberta-wwm-ext-large",
        local_dir=CACHE_PATH + "/bert/chinese-roberta-wwm-ext-large",
    )
    move_cache()

    os.makedirs(CACHE_PATH + "/bert/deberta-v3-large", exist_ok=True)
    snapshot_download(
        "microsoft/deberta-v3-large",
        local_dir=CACHE_PATH + "/bert/deberta-v3-large",
    )
    move_cache()

    os.makedirs(CACHE_PATH + "/bert/deberta-v2-large-japanese", exist_ok=True)
    snapshot_download(
        "ku-nlp/deberta-v2-large-japanese",
        local_dir=CACHE_PATH + "/bert/deberta-v2-large-japanese",
    )
    move_cache()

    os.makedirs(CACHE_PATH + "/bert/bert-base-japanese-v3", exist_ok=True)
    snapshot_download(
        "cl-tohoku/bert-base-japanese-v3",
        local_dir=CACHE_PATH + "/bert/bert-base-japanese-v3",
    )
    move_cache()

    import nltk
    nltk.download('averaged_perceptron_tagger')
    nltk.download('cmudict')


image = (
    modal.Image.debian_slim(python_version="3.10")
        .pip_install(
            "librosa==0.9.2",
            "matplotlib",
            "numpy",
            "numba",
            "phonemizer",
            "scipy",
            "tensorboard",
            "Unidecode",
            "amfm_decompy",
            "jieba",
            "transformers",
            "pypinyin",
            "cn2an",
            "gradio==3.50.2",
            "av",
            "mecab-python3",
            "loguru",
            "unidic-lite",
            "cmudict",
            "fugashi",
            "num2words",
            "PyYAML",
            "requests",
            "pyopenjtalk-prebuilt",
            "jaconv",
            "psutil",
            "GPUtil",
            "vector_quantize_pytorch",
            "g2p_en",
            "sentencepiece",
            "pykakasi",
            "langid",
            "torch",
            "torchvision",
            "torchaudio",
        )
        .run_function(download_model_weights)
)

@stub.function(
    gpu="l4",
    image=image,
    retries=3,
    mounts=[
        modal.Mount.from_local_python_packages("config"),
        modal.Mount.from_local_python_packages("tools"),
        modal.Mount.from_local_python_packages("utils"),
        modal.Mount.from_local_python_packages("infer"),
        modal.Mount.from_local_python_packages("re_matching"),
        modal.Mount.from_local_python_packages("modal_const"),
        modal.Mount.from_local_python_packages("commons"),
        modal.Mount.from_local_python_packages("text"),
        modal.Mount.from_local_python_packages("models"),
        modal.Mount.from_local_python_packages("modules"),
        modal.Mount.from_local_python_packages("transforms"),
        modal.Mount.from_local_python_packages("attentions"),
        modal.Mount.from_local_python_packages("monotonic_align"),
        modal.Mount.from_local_python_packages("oldVersion"),
        modal.Mount.from_local_file("config.yml", CACHE_PATH + "/config.yml"),
        modal.Mount.from_local_file("bert/bert_models.json", CACHE_PATH + "/bert/bert_models.json"),
    ]
)
def speech(
    text,
    speaker,
    sdp_ratio,
    noise_scale,
    noise_scale_w,
    length_scale,
    language,
    style_text,
    style_weight,
):
    import torch
    import utils
    from infer import infer, latest_version, get_net_g
    import gradio as gr
    import numpy as np
    from config import config
    import wave
    import tempfile

    net_g = None
    device = config.webui_config.device
    hps = utils.get_hparams_from_file(config.webui_config.config_path)
    version = hps.version if hasattr(hps, "version") else latest_version
    net_g = get_net_g(
        model_path=config.webui_config.model, version=version, device=device, hps=hps
    )
    speaker_ids = hps.data.spk2id
    # speakers = list(speaker_ids.keys())
    # languages = ["ZH", "JP", "EN", "mix", "auto"]

    def generate_audio(
        text,
        sdp_ratio,
        noise_scale,
        noise_scale_w,
        length_scale,
        speaker,
        language,
        reference_audio,
        emotion,
        style_text,
        style_weight,
        skip_start=False,
        skip_end=False,
    ):
        slices = text.split("|")
        audio_list = []
        # silence = np.zeros(hps.data.sampling_rate // 2, dtype=np.int16)
        with torch.no_grad():
            for idx, piece in enumerate(slices):
                skip_start = idx != 0
                skip_end = idx != len(slices) - 1
                audio = infer(
                    piece,
                    reference_audio=reference_audio,
                    emotion=emotion,
                    sdp_ratio=sdp_ratio,
                    noise_scale=noise_scale,
                    noise_scale_w=noise_scale_w,
                    length_scale=length_scale,
                    sid=speaker,
                    language=language,
                    hps=hps,
                    net_g=net_g,
                    device=device,
                    skip_start=skip_start,
                    skip_end=skip_end,
                    style_text=style_text,
                    style_weight=style_weight,
                )
                audio16bit = gr.processing_utils.convert_to_16_bit_wav(audio)
                audio_list.append(audio16bit)
        return np.concatenate(audio_list)

    res = generate_audio(text, sdp_ratio, noise_scale, noise_scale_w, length_scale, speaker, language, None, "Happy", style_text, style_weight)
    data = res.tobytes()
    tempfd, temppath = tempfile.mkstemp()
    with wave.open(temppath, "wb") as wav_file:
        wav_file.setparams((1, 2, 44100, 0, 'NONE', 'NONE'))
        wav_file.writeframes(data)

    ret = b""
    with open(temppath, "rb") as wav_file:
        ret = wav_file.read()

    os.close(tempfd)

    return ret

class SpeechReq(BaseModel):
    text: str
    speaker: str
    sdp_ratio: float = 0.5
    noise_scale: float = 0.6
    noise_scale_w: float = 0.9
    length_scale: float = 1.0
    language: str = "ZH"
    style_text: str = ""
    style_weight: float = 0.7

@web_app.post("/submit")
async def submit(req: SpeechReq):
    speech = modal.Function.lookup("bert-vits2", "speech")
    call = speech.spawn(
        req.text,
        req.speaker,
        req.sdp_ratio,
        req.noise_scale,
        req.noise_scale_w,
        req.length_scale,
        req.language,
        req.style_text,
        req.style_weight,
    )
    return {"call_id": call.object_id}

@web_app.get("/result/{call_id}")
async def poll_results(call_id: str):
    from modal.functions import FunctionCall

    function_call = FunctionCall.from_id(call_id)
    try:
        result = function_call.get(timeout=0)
    except TimeoutError:
        return fastapi.responses.JSONResponse(content="Still running", status_code=202)

    return Response(content=result, media_type="audio/x-wav")

@stub.function()
@modal.asgi_app()
def wrapper():
    return web_app

如果不想管细节，只需要修改download_model_weights函数中下载模型文件部分即可。我在自己的服务器上托管了模型权重，你需要把model_files修改为你的模型配置、权重文件名称，然后修改HOSTURL为模型下载地址。如果模型是存在S3或者其他地方，按需修改即可。

然后将自己训练时的config.yml放到项目根目录下面，注意修改路径（这里没列出全部配置）：

dataset_path: "/root/model_cache"

# webui webui配置
# 注意， “:” 后需要加空格
webui:
  # 推理设备
  device: "cuda"
  # 模型路径
  model: "G_88000.pth"
  # 配置文件路径
  config_path: "config.json"
  # 端口号
  port: 7860
  # 是否公开部署，对外网开放
  share: false
  # 是否开启debug模式
  debug: false
  # 语种识别库，可选langid, fastlid
  language_identification_library: "langid"

我们的模型都下载到/root/model_cache文件夹里的，webui配置里面的模型路径只需要填模型名称即可。

搞定后在本地安装并配置modal（但是我推荐使用pipenv来管理虚拟环境）:

1
2

python3 -m pip install modal
python3 -m modal setup

完成后部署即可：

`1`	`modal deploy modal_deploy.py`

谈谈细节

image定义了镜像，其中使用了debian slim镜像并指定Python版本为3.10，之后pip install了需要的包，随后调用我们的模型下载函数。镜像构建好后如果没有修改定义则不需要重新构建镜像，只需要同步代码即可，可以加快部署。

stub可以理解为一个app，一个项目之类的东西。我们在函数上加修饰可以定义一些细节，比如speech函数我们指定了GPU使用L4，使用我们之前构建的镜像，重试次数为3次。因为我们不是单个Python文件，所以这里通过mount将本地的包和配置文件给上传上去。

效果

部署好后在终端和Modal网页上可以看到https://***.modal.run的地址，我们使用Postman调用API：

这样我们的请求就进入到队列中等待处理了，可以拿submit返回的call id调用result接口获取结果：

如果还没完成则会返回202。

谈谈数据交换格式

Sun, 02 Jul 2023 19:36:20 +0800

这周末我们来看看数据交换格式：JSON（BSON）、Protocol Buffers和MessagePack。每当客户端和服务器、后端服务与服务之间通信时候，都会交换数据，如何将本机程序内存中的数据交给远端程序使用，同时不受语言、架构等限制就成为了一个问题。我们将对象转化为可以存储或传输的形式这个过程称为序列化，而其逆向过程称为反序列化。

JSON

这或许是大家最熟悉的一种格式了，JSON格式广泛用于C/S架构中通信，其具有良好的可读性，一段样例JSON如下：

{
    "a": "a string",
    "b": 2,
    "c": {
        "d": [1, 2, 3]
    },
    "e": true
}

JSON可以传递数字、布尔、数组、字符串、对象、空类型的数据，但是对于二进制数据不太方便，除非你进行编码（如Base64）或者转换成数组。由于是字符串格式，JSON可读性较好，但是处理速度并不理想。

顺带，JSON发展于2000年初。

BSON

BSON目前主要运用于MongoDB中，BSON的含义即“二进制JSON”，相较于JSON，BSON使用二进制格式来进行编码，参考自JSON and BSON，一个BSON和JSON的样例如下：

{"hello": "world"} →
\x16\x00\x00\x00           // total document size
\x02                       // 0x02 = type String
hello\x00                  // field name
\x06\x00\x00\x00world\x00  // field value
\x00                       // 0x00 = type EOO ('end of object')

相较于JSON，BSON通过二进制表示了数据的类型和长度，这加快了遍历速度。同时BSON支持了JSON不支持的日期和二进制数据。

Protocol Buffers

Protocol Buffer是2008年由Google开发的序列化协议，其设计目标就是简单和性能。Protocol Buffer需要双方约定好消息的定义（.proto文件），一个示例如下：

1
2
3

message Test1 {
  optional int32 a = 1;
}

假如a的值为150，你会得到如下的序列化结果：

1
2

08 96 01
00001000 10010110 00000001

我们怎么来看呢？对于第一个字节代表Tag和类型，前5位代表Tag即上面的1，之后000三个bit代表其类型是VARINT，作为整数、无符号整数、布尔、枚举类型的统称。

10010110 00000001
拆分为
1 0010110 0 0000001
其中第一位表示接下来的字节是否继续表示该数据
除去MSB后，我们将数据变为小端序
10010110 = 150

对于类型和编码更多详细内容可以阅读官方文档：Encoding。

我们可以发现ProtoBuf是用tag来表示数据的Key的，和其真正名称无关。ProtoBuf是二进制格式不具有可读性，但是传输开销和（反）序列化开销较小。

MessagePack

MessagePack目前最新的版本是2009年发布的0.3.3版本，MessagePack专注于紧凑和简单。一个JSON到MessagePack的样例如下：

{"compact":true,"schema":0}

82 a7 63 6f 6d 70 61 63 74 c3 a6 73 63 68 65 6d 61 00

82: 1000XXXX 表示2*N个元素的map，这里的N是0010，表示4个元素的Map。
a7：101XXXXX 表示长度为XXXXX的固定字符串，这里长度为7，即后续的63 6f 6d 70 61 63 74代表compact
c3: 固定表示true
a6: 类似于a7，对应后续的73 63 68 65 6d 61
00: 0xxxxxxx 表示xxxxxxx正整数

对于更加详细的编码规则可以参考spec。MessagePack对应于JSON，但是支持二进制数组等JSON本不支持的格式。由于是二进制格式，其仍然可读性不佳，但是性能较JSON好。

搭建属于你的Anycast任播网络

Fri, 23 Jun 2023 20:58:46 +0800

很早之前我就在好奇DNS是怎么做到全球访问同一个IP但是延迟都这么低，Serverless平台是怎么做到全球多地部署同一个应用共享同一个IP等问题。后来我申请下来了一个ASN和/44的IPv6地址段，开始了BGP之旅，终于亲手搭建了一个Anycast网络，并在全球四个地区部署了使用同一IP地址的相同应用。下面让我来分享一下如何搭建属于你的Anycast网络，关于如何申请ASN并且BGP广播出去可以参考我之前的文章：成为互联网自治系统之一。

如果你有学习、研究、开发等用途，希望拿到一个免费的IPv6 /64任播地址并接入我的网络，欢迎联系我，免费提供给有正当需求（~~和我比较熟~~）的人。(●’◡’●)

更新数据库

上篇文章我们讲到BGP后只简单提到了RPKI和ROA，后来在研究的过程中我发现我遗漏了IRR相关的内容。在拿到网段后，我们最好先去RIPE数据库中为我们的网段创建inet6num和route6对象，分别作为Whois和IRR。

inet6num中我们可以填入IP段的名称、所属国家、维护者等信息，这些信息会在第三方查询工具中以Whois展示出来。IP地址库也会参考这里面的所属国家做定位。这也是为什么有些网上的人的IP地址会显示其他国家的原因。

route6中我们需要填入网段和允许的ASN来源，这可以让其他人验证广播是否有效。

BGP

BGP这里也不做过多讲述，之前的文章介绍过怎么在Vultr上进行BGP，他们的官方文档（Configuring BGP on Vultr）中也有比较详细的教程。新手BGP一定记住：过滤、过滤、过滤！不要把不该export的路由给播出去了。

对于IPv6，我们想做到Anycast的话只需要在不同机器上广播同一网段即可，等到广播在互联网上传播开来就好。之后其他人访问你时会选择跳数最少的访问过去，正因为跳数最少所以不是延迟最低（缺乏考证）。

WireGuard

我们在不同国家地区都部署了PoP后，想接入服务器有两种方法：

每个PoP都建立一个WireGuard链接到一台服务器，这些服务器使用同一个IPv6，那么我们需要n台服务器才行。
接入至少一台服务器，让每个PoP都通过WireGuard连起来，然后通过动态路由协议更新它们的路由表。

是不是第二种看上去经济挺多？那我们就这么干！WireGuard的配置教程很多，我就不班门弄斧了。但是，代价是什么呢？

Babel

让我们用点新东西，Babel是近几年出来的动态路由协议，它能够快速收敛、避免环路。其不仅考虑到了有线网络，也对无线网络Mesh网络进行了思考，它可以用跳数、丢包率、延迟等作为指标，这允许了我们选择更加实用的指标来进行路由。

Babeld默认工作在6696端口，使用UDP作为传输层协议，默认的IPv6的组播地址是ff02::1:6。提到组播后有的人要头皮发麻了，是的，这意味着你得创建n(n-1)/2个Wireguard链接，当然你也可以选择牺牲健壮性，将几个关键节点成对连接起来，只要保证至少有一条通路可以走遍所有节点即可。

在Wg的配置文件中你需要加一个fe80::/64的本地地址，以便组播。

Babeld的配置文件在/etc/babeld.conf，我的样例配置文件如下：

default type tunnel # 表明链路类型是隧道，默认在Hello和IHU消息中发送时间戳便于计算RTT
default split-horizon true # 开启水平分割
default faraway true # 表明网络节点直接是“很远”的，其背后的网络和其之前的网络没有关系
default unicast true # 向所有邻居发送除Hello外的多份TLV

local-port 23333 # 提供给client连接的端口

interface wg0 # 工作在wg0 if上
interface wg1 # 工作在wg1 if上

in ip 2a0c:xxxx:xxxx::/48 eq 64 allow # 我们接收2a0c:xxxx:xxxx::/48段长度为64的prefix，根据metrics判断是否加入路由表
in ip fddc:xxxx:xxxx:xxxx::/64 eq 128 allow # 我们接收fddc:xxxx:xxxx:xxxx::/64段长度为128的prefix，根据metrics判断是否加入路由表
in deny # 其余prefix默认拒绝

redistribute ip 2a0c:xxxx:xxxx::/48 eq 64 allow # 我们会重播2a0c:xxxx:xxxx::/48段长度为64的prefix，告诉别人可以经过我们到达
redistribute ip fddc:xxxx:xxxx:xxxx::/64 eq 128 allow # 我们会重播fddc:xxxx:xxxx:xxxx::/64段长度为128的prefix，告诉别人可以经过我们到达
redistribute local deny # 其余prefix默认拒绝

babeld启动后，我们可以通过ncat看看其状态：ncat :: 23333。

输入dump并回车：查看当前的路由信息，可以观察每条prefix可以经过谁可达，指标的值是多少，是否加入了我们的路由表等等。
输入monitor并回车：实时输出邻居和路由的变动信息，非常便于调试。

如果没有问题的话，这时候我们将一台服务器wg到任意一个PoP，其他人访问任意PoP终会走WireGuard跳到我们的服务器。

Why I Switch From Netbird to Tailscale

Sat, 10 Jun 2023 20:55:53 +0800

Last year, I used NetBird to create an inner network with WireGuard VPN. That really provides me with easier network management and safer access control. However, I started to switch to Tailscale recently. Here’s why.

Why is NetBird no longer suitable for me?

NetBird is under actively development, in most perspective, it’s a good thing. This means they really care about it, and paid many attentions to it. That’s what we should appreciate with. But this also means their version iteration is fast, I don’t have much willing to log in to every server and do an upgrade.
Since NetBird 0.20 version, I could not use it on my Windows 11 machine to connect to other Linux machine. I’m uncertain if it’s because I have many versions running on different machines. So, I must downgrade to 0.18, and it works.
NetBird is not ready for IPv6, at least when I use it. If you have IPv6 only VPS, NetBird could not connect to the TURN server.
No Android app yet.

How about Tailscale?

I have been using it since a week ago. It means work well. It supports IPv6, free plan supports more devices, the desktop UI can see all devices’ IP. But there are also drawbacks:

You should use a third-party login by default, I want to log in with my email of my domain.
No NetBird’s route feature, you cannot easily access your school network or work network by only a few clicks.

I will update this post when I find anything new.

力扣第346场周赛思路

Thu, 25 May 2023 16:13:24 +0800

第二次打周赛，这次拿到了全国160多名的成绩，前面三道题不到二十分钟可以做完，思路也比较好想。最后一道图相关的hard题暴露了短板，以后图还是得多看看。

删除子串后的字符串最小长度

题目链接：2696. 删除子串后的字符串最小长度

思路：根据题目的描述，嗯造就完了。

代码：

class Solution {
public:
    int minLength(string s) {
        while(true) {
            bool modified = false;
            while (true) {
                int pos = s.find("AB");
                if (pos == string::npos) {
                    break;
                }
                s.erase(pos, 2);
                modified = true;
            }
            while (true) {
                int pos = s.find("CD");
                if (pos == string::npos) {
                    break;
                }
                s.erase(pos, 2);
                modified = true;
            }
            if (!modified) {
                break;
            }
        }
        return s.length();
    }
};

字典序最小回文串

题目链接：2697. 字典序最小回文串

思路：双指针，如果遇到不相同的字母则修改为字典序更小的一方。

代码：

class Solution {
public:
    string makeSmallestPalindrome(string s) {
        int n = s.size();
        int left = 0, right = n - 1;
        while (left < right) {
            if (s[left] != s[right]) {
                if (s[left] < s[right]) {
                    s[right] = s[left];
                } else {
                    s[left] = s[right];
                }
            }
            left++;
            right--;
        }
        return s;
    }
};

求一个整数的惩罚数

题目链接：2698. 求一个整数的惩罚数

思路：比较简单的搜索，搜索的时候判断是否需要将当前数字加到划分里面，将积累的待加数放到acc里面。

代码：

class Solution {
    bool search(const string &s, int cur, int target, int sum, int acc) {
        if (sum > target) {
            return false;
        }
        if (cur == s.size()) {
            return (sum + acc) == target;
        }
        bool valid = false;
        valid = valid || search(s, cur + 1, target, sum + (acc * 10 + s[cur] - '0'), 0);
        valid = valid || search(s, cur + 1, target, sum, acc * 10 + s[cur] - '0');
        return valid;
    }

public:
    int punishmentNumber(int n) {
        int ans = 0;
        for (int i = 1; i <= n; i++) {
            int x = i * i;
            string s = to_string(x);
            if (search(s, 0, i, 0, 0)) {
                cout << i << endl;
                ans += x;
            }
        }
        return ans;
    }
};

力扣第345场周赛思路

Sun, 14 May 2023 13:58:56 +0800

第一次参加了周赛，侥幸碰到了比较简单的一场，离大佬们还有很大差距，以后尽量每周都来写写代码。每次打完比赛后都分享一下自己的思路吧，很可能不是最优的思路，还希望谅解。

找出转圈游戏输家

题目链接：6430. 找出转圈游戏输家

思路：模拟题，题目说任意玩家第二次接到球游戏就结束，于是可以创一个set来表存历史位置，如果当前位置已经出现过，游戏结束。开始寻找未出现在set中的位置。

时间复杂度：$$O(n)$$

空间复杂度：$$O(n)$$

代码：

class Solution {
public:
    vector<int> circularGameLosers(int n, int k) {
        unordered_set<int> get_ball;
        int pos = 0;
        int round = 1;

        auto next_pos = [&](int pos) {
            return (pos + round * k) % n;
        };
        
        get_ball.insert(0);
        while (true) {
            pos = next_pos(pos);
            if (get_ball.count(pos)) {
                break;
            }
            get_ball.insert(pos);
            round++;
        }

        vector<int> ans;
        for (int i = 0; i < n; i++) {
            if (!get_ball.count(i))
                ans.emplace_back(i + 1);
        }
        
        return ans;
    }
};

相邻值的按位异或

题目链接：6431. 相邻值的按位异或

思路：这道题反而是最后来看的，因为我对位运算或者数字规律有点不太行，后续得加强一下。这道题拿到在纸上画了画，异或比较常用的性质是：

$$a⊕b=c$$

$$a⊕c=b$$

$$b⊕c=a$$

然而这道题用不上这个，我一开始画了画发现算不出来，于是往找规律上面靠。我们可以发现其实中间的数字是无所谓的，因为异或是“同0异1”，对于中间的数字我们总是可以构造出来符合的数字。我们可以假设原数组第一个数字是0，通过derived的值可以知道0和1的翻转情况，比如derived[0]是1，说明第二个数字和第一个不相同，那么原数组第二个数字就是1，以此类推，遍历数组。最后看我们得到的原数组的最后一个数字和第一个数字的值是否满足derived[n - 1]，如果满足则可以构建，不满足则无法构建。

时间复杂度：$$O(n)$$

空间复杂度：$$O(1)$$

代码：

class Solution {
public:
    bool doesValidArrayExist(vector<int>& derived) {
        int n = derived.size();
        bool last = (accumulate(derived.begin(), derived.end(), 0) - derived[n - 1]) % 2;
        if (derived[n - 1] == 0) {
            return !last;
        } else {
            return last;
        }
    }
};

矩阵中移动的最大次数

题目链接：6433. 矩阵中移动的最大次数

思路：下意识写了个搜索，发现过不了，于是加上记忆化，即保存历史走过当前点的步数，如果现在走到那一格的步数还没有之前高就不继续了，还是超时。于是放弃搜索，开始动规。结果发现这道题就是常规的动规，先走列，再走行，中途还可以利用上之前说到的记忆化及时剪枝。

时间复杂度：$$O(n)$$

空间复杂度：$$O(n)$$

代码：

class Solution {
public:
    int ans;
    int m, n;

    vector<vector<int>> dp;
    const vector<pair<int, int>> direction = {
        {-1, 1},
        {0, 1},
        {1, 1},
    };

    inline bool valid(int x, int y) {
        return (x >= 0 && x < m && y >= 0 && y < n);
    }

    int maxMoves(vector<vector<int>>& grid) {
        ans = 0;
        m = grid.size();
        n = grid[0].size();
        dp.resize(m, vector<int>(n, 0));

        for (int j = 0; j < n - 1; j++) {
            for (int i = 0; i < m; i++) {
                if (j != 0 && dp[i][j] == 0)
                    continue;
                for (auto [dx, dy] : direction) {
                    int x = i + dx;
                    int y = j + dy;
                    if (valid(x, y) && grid[x][y] > grid[i][j]) {
                        dp[x][y] = max(dp[x][y], dp[i][j] + 1);
                        ans = max(ans, dp[x][y]);
                    }
                }
            }
        }

        return ans;
    }
};

统计完全连通分量的数量

题目链接：6432. 统计完全连通分量的数量

思路：先把整个图的连通分量找出来，保存起来。然后检查每个连通分量中的点的边数是否等于连通分量点的个数，如果都相等则是完全连通分量。

时间复杂度：$$O(n)$$

空间复杂度：$$O(n)$$

代码：

class Solution {
public:
    int countCompleteComponents(int n, vector<vector<int>>& edges) {
        vector<vector<int>> graph(n);
        for (auto edge : edges) {
            graph[edge[0]].emplace_back(edge[1]);
            graph[edge[1]].emplace_back(edge[0]);
        }
        vector<bool> visited(n, false);
        int ans = 0;
        vector<vector<int>> components;
        for (int i = 0; i < n; i++) {
            if (!visited[i]) {
                vector<int> component;
                queue<int> q;
                q.push(i);
                component.emplace_back(i);
                visited[i] = true;
                while (!q.empty()) {
                    int cur = q.front();
                    q.pop();
                    for (auto next : graph[cur]) {
                        if (!visited[next]) {
                            q.push(next);
                            visited[next] = true;
                            component.emplace_back(next);
                        }
                    }
                }
                components.emplace_back(component);
            }
        }
        for (auto component : components) {
            bool is_complete = true;
            for (auto node : component) {
                if (graph[node].size() != component.size() - 1) {
                    is_complete = false;
                    break;
                }
            }
            if (is_complete) {
                ans++;
            }
        }
        return ans;
    }
};

成为互联网自治系统之一

Tue, 02 May 2023 16:04:10 +0800

最近心血来潮，从RIPE NCC的LIR申请到了一个ASN和IPv6 /44前缀地址：AS198764。这里记录一下我的经历，分享给有需要的朋友。

写在前面：IPv4资源已经消耗殆尽，这是IPv6的时代。;)

申请ASN

RIR（区域互联网注册机构）负责管理世界上特定地区的互联网资源，我们需要向他们去申请互联网资源。目前世界上有五个运作的机构：

ARIN：美洲
RIPE NCC：欧洲
APNIC：亚洲
LACNIC：拉丁美洲
AfriNIC：欧洲

看了一圈下来似乎欧洲稍微友好些，你可以根据实际情况更新RIPE数据库中的信息，而不需要审核之类的，费用看上去也还行。当决定向RIPE申请互联网资源后，有两种方式去申请：

每年交大笔费用成为RIPE NCC Member，对象可以是法律实体或者是自然人。
找到一位RIPE NCC Member作为赞助你的LIR（本地互联网注册机构），让他来赞助你ASN和IP资源。

由于没钱，我选择了第二种方法，每年向LIR支付一定费用来让他们赞助我ASN和资源。这里我并不打算推荐任何LIR，搜索LIR Service应该可以找到大批商家。

申请RIPE NCC的资源前，你需要做下面的准备：

可以提供你的身份证或者护照照片。
需要证明你在欧洲的存在，即你需要可以ping通的欧洲服务器，然后至少有两个Transit（上游）为你提供服务，比如你用Vultr(推广链接)创建了一个欧洲VPS，使用他们的免费BGP Session，那么你就有了一个Transit，你还得找一个。有些LIR可以把满足要求的VPS为你打包销售，省得麻烦。
在RIPE Database中创建Role and maintainer pair对象，这里直接登录RIPE NCC官网进入Database自行创建即可。创建的时候Role和Person没有多大区别，可以选择创建Role，然后把你的mntner和role对象ID记下来即可，创建完成后建议截个图。
钱。

准备好后，就可以去联系欧洲的LIR啦，他们会让你确认订单、提供身份信息、缴费等等，差不多一周左右你就可以惊喜地在RIPE Database中看到你的ASN和IPv6啦！缴费前建议去RIPE官方公布的Member列表中确认一下对方是不是真的LIR。

广播

当拿到一个ASN和IPv6资源后，我们可以通过BGP将其广播出去，让我们可以被路由到。BGP Daemon一般使用bird，bird支持多种路由协议，如OSPF、RIP(RIP确实RIP了)、BGP等等。你也可以选择Golang或者Rust的相关实现。

你还需要准备支持BGP的服务器，不是所有卖VPS的都支持你BGP。假如是Vultr或者BuyVM，我们需要先联系客服，客服会发送验证邮件到你在数据库中留下的联系邮箱确认你的所有权，然后为你设置BGP Sessioon。一切妥当后，让我们开始广播！

这里推荐先阅读Soha Jin编写的BIRD 与 BGP 的新手开场。我的Vultr服务器上的Bird配置如下（请注意替换大括号内的内容）：

log syslog all;
log "/var/log/bird.log" {info};

router id {VPS的公网IPv4};
define OWN_ASN = {你的ASN};

protocol device {}

protocol kernel {
    ipv6 {
        export all;
    };
}

protocol static {
    ipv6;
    route {你准备宣告的段，建议最小/48} via {VPS的IPv6地址};
}

# 过滤掉太长的段
function net_len_too_long(){
    case net.type {
        NET_IP4: return net.len > 24;
        NET_IP6: return net.len > 48;
        else: print "net_len_too_long: unexpected net.type ", net.type, " ", net; return false;
    }
}

function bgp_export() {
    if net_len_too_long() then return false;
    if source != RTS_STATIC then return false;
    # 这里为了防止泄露路由，我们只允许我们的段export出去
    if net !~ [{你的IPv6段}+] then return false;
    return true;
}

protocol bgp vultr {
    local as OWN_ASN;
    source address {VPS的IPv6地址};
    ipv6 {
        import none;
        export filter {
            if !bgp_export() then reject;
            accept;
        };
    };
    graceful restart on;
    multihop 2;
    neighbor {ISP提供的BGP IPv6} as {ISP提供的BGP ASN};
    password "{ISP提供的BGP 密码}";
}

搞定之后，一定再三确定不会出现泄露路由的情况，即把不该广播的地址广播出去了！BGP并不安全，详见CloudFlare专门做的网站Is BGP safe yet?。确定之后重启Bird即可。

当我们通过birdc show protocol all vultr查看信息，有类似下面的信息：

Routes:         0 imported, 1 exported, 0 preferred
Route change stats:     received   rejected   filtered    ignored   accepted
    Import updates:        2043363          0    2043363          0          0
    Import withdraws:       206668          0        ---     206669          0
    Export updates:              1          0          0        ---          1
    Export withdraws:            0        ---        ---        ---          0

有一条路由Export成功了，我们接下来要做的事情就是等待这条路由慢慢传播了，一般24小时之后可以在各种BGP工具上查询到。

RPKI

其实完成上面的步骤后，我们的段就可以用了，当你创建一个dummy网卡，配置成你的IPv6地址后就能ping通。但是这里我还想讲讲RPKI和ROA。

上面提到了BGP并不安全，为了缓解这个问题引入了基于证书的系统，名为Resource Public Key Infrastructure (RPKI)。我们可以创建ROA（路由源授权），每个ROA包含三个信息：ASN、地址段、最长允许长度。其他人收到我们的路由信息后可以通过软件去验证这条路由信息，判断是否符合ROA，防止路由劫持、泄露。

我们不需要关心密码学的东西太多，如果需要配置ROA联系LIR即可。（这里没有对PA和PI地址进行说明，一般LIR赞助我们的是PA资源，由LIR来管理，所以需要他们去配置）

今天就写到这里，下次我会介绍怎么搭建全球任播网络。Global Anycast Network!

直面我的焦虑

Fri, 28 Apr 2023 12:22:08 +0800

不知道为什么最近心中总是感到烦躁，没有办法静下心来做事情，这种感觉大概是前几天开始的，这几天感觉明明没做什么事情就是感觉疲惫，想睡觉。对于等待的事情，越来越没有耐心，想得越来越多。总是说人生是自己的旅途，但实际上多多少少会被他人影响。最近也有种倾向不想和其他人沟通，只要说出去的话经常会去想是不是不应该这么说。

希望一切都顺利吧，也许过了这段时间就好了。最近会写一篇文章介绍我申请下来了ASN并且搭建Anycast网络，还请期待，感谢。

为什么Offer邮件会丢失（简述电子邮件）

Fri, 07 Apr 2023 08:14:10 +0800

今天我们来谈谈一个熟悉却又陌生的东西——电子邮件，我们每天都在接收或者发送电子邮件，有些邮件可以到达我们的收件箱，而有些邮件会出现在我们邮箱的Junk/Spam目录，甚至会被邮件服务器直接拒收。作者这两天参加春招就遇到这么一件事情，被告知Offer邮件已经发送到我的邮箱，等了许久却没有收到，检查邮件服务器的日志发现邮件状态是DEFER，被邮件服务器延迟接收。为什么会这样？电子邮件是怎么工作的？让我们来简单看看。（本文简单聊聊电子邮件，如有错误，还请批评指正。🥰）

SMTP

像许多网络协议一样，电子邮件相关协议的岁数比我还大。或许反直觉的是，电子邮件本身是不加密的，电子邮件就是一堆纯文本在复杂的网络上传来传去。发送一封邮件到对方服务器常用的是SMTP协议，这个协议一开始便基于ASCII纯文本，后来才使用MIME标准让图像、声音等二进制可以在电子邮件中传输。对于这么一个协议，如果对方的SMTP服务器没有加密，我们可以使用nc直接发一封邮件：

这里，我们向www.mail-tester.com发送了一封测试邮件，带有状态码的行是对方给我们的回复，除此之外都是我们发送的数据。发送一封电子邮件是不是比我们想得简单？是的，这样对方就收到了我们的邮件。（如果对方使用了SSL/TLS，可以考虑使用swaks或者openssl，不过大同小异）但是，这封邮件的命运如何呢？

毫不意外，我们的邮件永远不会出现在对方的收件箱中，为什么会这样？Mail tester给出了以下理由：

MISSING_DATE Missing Date: header
MISSING_FROM Missing From: header
MISSING_HEADERS Missing To: header
MISSING_MID Missing Message-Id: header
RDNS_NONE Delivered to internal network by a host with no rDNS
SPF_FAIL SPF: sender does not match SPF record (fail)
SPF_HELO_FAIL SPF: HELO does not match SPF record (fail)

让我们加一点Header：

Date: Fri, 07 Apr 2023 01:10:32 +0000 (UTC)
From: Test <sender@example.com>
To: Test <test-enjevs04y@srv1.mail-tester.com>
Message-ID: <5161dbc0-16d0-7a08-20ad-ba315ac2d1f0@example.com>
Subject: Test Email

Hello, world!

.

评分从0.1上升到了3.8，看上去还行！让我们继续。

Headers

相比较第一封邮件，我们添加了一些邮件Header，让我们问问GPT-4有哪些常见的Header：

From：发件人的电子邮件地址。这个字段是必需的，表示邮件的发送者。
To：收件人的电子邮件地址。这个字段也是必需的，表示邮件的接收者。
Cc（抄送）：向其他收件人发送邮件副本的电子邮件地址。这个字段是可选的。
Bcc（密件抄送）：向其他收件人发送邮件副本的电子邮件地址，但其他收件人看不到这些 Bcc 地址。这个字段是可选的。
Subject：邮件的主题。这个字段通常包含邮件的简短摘要，以帮助收件人理解邮件的内容。
Date：邮件发送的日期和时间。这个字段通常由邮件客户端自动填充。
Message-ID：邮件的唯一标识符。这个字段通常由邮件客户端自动生成，用于跟踪和引用特定的邮件。
Content-Type：邮件正文的 MIME 类型，例如 text/plain（纯文本）或 text/html（HTML 格式）。
Content-Transfer-Encoding：邮件正文的编码方式，例如 7bit、8bit、binary、quoted-printable 或 base64。
MIME-Version：邮件使用的 MIME（多用途因特网邮件扩展）协议版本，通常为 1.0。
Reply-To：指定回复邮件时应发送到的电子邮件地址。这个字段是可选的，如果没有指定，回复通常会发送到发件人地址。
In-Reply-To：引用或回复的原始邮件的 Message-ID。这个字段用于将相关邮件组织成线程。
References：一个包含其他相关邮件 Message-ID 的列表，用于将邮件组织成线程。
X-*：自定义 Header 字段，用于添加特定于应用程序的信息。这些字段通常以 X- 开头，以区分它们与标准 Header 字段。

对于其中的Message-ID和In-Reply-To，是判断邮件Thread的重要依据，如果In-Reply-To的ID和之前Message-ID相同，这会帮助邮件客户端正确处理邮件回复。

SPF

就算我们添加了Header，邮件分数还是不是很高，这时候我们来看看SPF。SPF是一种特殊的DNS记录，可用于确保邮件由授权的服务器发出，否则谁都可以连接到邮件服务器，声称自己是example.com或者是其他域名。让我们找到一个不怎么用的域名，给它加上SPF记录，然后再来看看。假如我们的域名是example.com，我们想允许IP是1.1.1.1的服务器能够代表我们发信，那么我们给@加一条TXT：v=spf1 ip4:1.1.1.1 -all，具体的规则可以看看参考资料的百科页面。设置好后，我们再来发信：

不错不错！除了SPF，还有一些安全措施。

DKIM

DKIM也是为了验证电子邮件是否真正来自发件人生成的域的手段，我们来看看GPT-4的解释其工作原理：

设置DKIM记录：首先，发件人域的管理员需要生成一对公钥和私钥。公钥将作为一个DNS TXT记录发布在域的DNS设置中，而私钥则需要妥善保管，以便发件服务器使用。
签名邮件：当发件服务器准备发送电子邮件时，它会使用私钥对邮件的某些部分（通常是邮件头和正文的一部分）进行数字签名。签名会包含在一个名为DKIM-Signature的邮件头字段中。
接收服务器验证：收件服务器在收到带有DKIM-Signature的邮件时，首先从该签名中获取发件人域的相关信息。然后，收件服务器会查询发件人域的DNS记录，获取相应的公钥。
验证签名：收件服务器使用获取到的公钥对DKIM-Signature中的签名进行验证。如果验证成功，说明邮件确实来自声称的发件人域，并且在传输过程中没有被篡改。否则，邮件可能被视为伪造或者被篡改。

DMARC

DMARC建立在SPF和DKIM之上，设置了如果未通过以上两种方法验证该如何处理电子邮件，可以查看DMARC Overview。一般我们设置为v=DMARC1; p=quarantine;。

Blacklist

评价就是不要用黑名单，为什么？我想在这里引用部分来自Messages might go to Junk - Migadu的话，Migadu是我比较喜欢的邮件服务器提供商：使用黑名单不是发件人的问题，而是收件人的问题，黑名单只是一个判断垃圾邮件的指标，更有部分黑名单需要收取删除费用，让他们成为勒索者。正常邮件就是正常邮件，和它是否从来自黑名单IP中发出没有关系。

Greylist

终于，我们可以来讲讲之前的Offer邮件为什么没有收到了。收到的Offer邮件经过邮件服务器的检查后，会给出一个“分数”，我们可以在邮件的Header中看到X-Spam-Score，如果负数越小说明更不像是垃圾邮件，正数越大越像是垃圾邮件。如果垃圾邮件分数介于正常邮件和垃圾邮件之间，邮件服务器会暂时拒绝这封邮件，告诉发信方重试发送。这时候有能力的送信服务器会将邮件保留在队列中重试发送，而大量垃圾邮件发送者不会这么做。如果重试发送了，邮件终将会送达收件箱。所以我怀疑是对方公司的邮件服务器没有做重试，导致邮件没有收到。以后看来得设置白名单看看能不能解决这个问题了。

参考资料

ARM下的CAS

Sun, 02 Apr 2023 10:58:07 +0800

简单分享记录一下最近在ARM下探索的CAS(compare and swap)实现，语言使用的是C++。我们知道ARM是RISC精简指令集，它不像x86下面有cmpxchg这么一条指令就可以做到，我们需要用多条指令来锁内存、写入值等操作。

这里是我根据参考资料写的一个弱CAS操作函数：

// Atomic compare and swap, return 0 if success, otherwise return 1
static inline unsigned atomic_compare_and_swap(volatile unsigned* ptr,
                                               unsigned old_value,
                                               unsigned new_value) {
  asm volatile(
      "ldaxr %w0, [%1]\n"
      "cmp %w0, %w2\n"
      "b.ne 1f\n"
      "stlxr %w0, %w3, [%1]\n"
      "1: clrex"
      : "=&r"(old_value)
      : "r"(ptr), "r"(old_value), "r"(new_value)
      : "memory");

  return old_value;
}

GPT-4 对这段代码的理解

这段代码看起来是一个用于 ARM 架构的 CAS（Compare and Swap）操作的实现。CAS 是一种原子操作，用于在多线程环境中确保数据的一致性。这段代码使用了 ARMv8-A 架构的汇编指令。现在我来解释这段代码的每一条汇编指令：

ldaxr %w0, [%1]: 这是一条 Load-Acquire Exclusive Register 指令，将内存地址 [%1]（ptr 指针所指向的内存）中的值加载到寄存器 %w0（old_value）中。这是一个独占加载操作，表示此操作会尝试获取一个独占访问权限，以确保后续的操作不会被其他处理器或内核干扰。

cmp %w0, %w2: 这是一条 Compare 指令，将寄存器 %w0（old_value）和寄存器 %w2（old_value）的值进行比较。注意这里的 %w0 和 %w2 都表示 old_value，但在第一条指令之后，%w0 的值实际上是从内存加载的值。

b.ne 1f: 这是一条 Branch 指令，当上一条 Compare 指令的结果不等（Not Equal）时，跳转到标签 1（即下面的 1: clrex）。如果从内存加载的值不等于预期的 old_value，这将跳过后续的 Store-Exclusive 操作并清除独占访问权限。

stlxr %w0, %w3, [%1]: 这是一条 Store-Exclusive Register 指令。如果独占访问权限仍然有效，那么将寄存器 %w3（new_value）的值存储到内存地址 [%1]（ptr 指针所指向的内存）。同时，将存储是否成功的结果写入寄存器 %w0。如果存储成功，%w0 的值将为 0，否则为非零值。

1: clrex: 这是一条 Clear Exclusive 指令，用于清除当前处理器的独占访问权限。这样可以确保后续不再需要独占访问权限的指令不会受到影响。

这段代码看起来没有问题。当调用这个函数时，如果在内存地址 ptr 的值等于 old_value，那么将会把 new_value 存储到该内存地址，并返回原始的 old_value。如果在内存地址 ptr 的值不等于 old_value，那么什么都不会发生，直接返回从内存加载的值。

我们再来看看这段代码

有些细节是在我写这个函数的时候不太明白的，也查阅了一些资料，这里跟着说明一下。

asm后面的volatile告诉编译器禁用对这段汇编的优化，保持原样。汇编代码结束后第一行我们定义了有哪些OutputOperands，形如“=&r”叫constraint，这里的=表明会覆盖已有变量的值，&表明这个值在指令使用完InputOperands前就会被写入，r表明使用一个通用寄存器。

这里我们没有对传入的值使用asmSymbolicName，因此每个操作数和汇编里面的从零开始的操作数一一对应：

old_value对应了%0
ptr对应了%1
old_value对应了%2
new_value对应%3

可以看到按照顺序可以对应。最后我们编译器我们改变了内存，这样编译器可以处理好读写内存的屏障。

这里和x86有一点不同，如果写入成功，我们的函数返回0，否则返回1。

参考资料

Migration Journey on the Cloud

Thu, 02 Feb 2023 14:12:47 +0800

As you might have noticed, the CDN provider of this blog has been changed to Bunny CDN. It’s a CDN provider from Slovenia, a country in Central Europe. Furthermore, it has a satisfying performance according to CDNPerf website.

In the meanwhile, I also chose Sendinblue as my transactional email service, BlazingCDN as a backup CDN, Scaleway as my cloud computing service, Better Stack as my status page service. The reason to pick digital products from Europe is that they give you a feeling of respecting your privacy at first sight. And some of them are cost-effective than providers from other places.

But before migrating data to those providers, I have to address that there are still some flaw among their services. Like serverless function service from one cloud provider, I tried two days, but failed to make Rust function to be built successfully on their platform. One CDN service could not handle video generated by Emby correctly. The origin server sent a whole image, but CDN server were unable to transit the complete data to the end user. While other CDN does not have this issue.

Anyway, you should consider the advantages and drawbacks of using services from Europe. If you want to find a complete solution provider, you may like Azure or AWS more. But if you plan to find a place for your cloud infrastructure, why not have a try?

Celebrate Server Storage Reaches 10 TB

Mon, 23 Jan 2023 18:56:15 +0800

Today, I want to celebrate all storage resource of my personal cloud infrastructure sum up to over 10 terabytes. It consists of some storage VPSs and the home NAS, and monthly bill is less than 15 USD. A one gigabyte network port seems to be a minimal requirement for VPS nowadays.

So, you may be wondering why we need storage servers instead of some network drive. Firstly, it could make you feel like my data is in my control. You can manage your data. Other people can’t delete or spy on it. Secondly, some network drives may limit the download speed to force users to charge VIP. If you have a NAS or high speed VPS, you won’t wait years for downloading.

Storing files on a real disk rather than mounting network drives via rclone also brings you another advantage. You can use BitTorrent download and set up your own Emby server. Emby can extract thumbnails from videos, but if you use network drive, extracting thumbnails and playing video will depend on the speed of down stream. That might cause inconsistent user experience and a longer loading time.

Furthermore, I want to mention BackBlaze here. It has been over one year since I start to use object storage service from B2. It’s cheap and safe. However, one of the drawbacks is you may need to use Cloudflare worker to avoid a download transmission fee.

The weather is getting warmer. At the end of this post, I sincerely hope the global economy will, like the coming spring, recover soon.

写在2022年末

Sat, 24 Dec 2022 01:20:03 +0800

终于磕磕绊绊走到了2022年的末尾，回头看今年收获了很多，也留下了很多遗憾，这里想先列举几点：

今年完成了两段实习，非常感谢Second State和Microsoft，真诚祝愿他们越来越好
今年下半年一共投递了27个岗位，其中面试了3家公司，1家止于一面，2家止于二面，0个offer
为Linux贡献了多篇文档翻译和一个Cleanup，目前已经被合并进了Linus的主线

今年上半年感觉精神状态还是挺好的，该实习实习，该面试面试，该玩游戏就玩游戏，还是挺无忧无虑。从今年下半年开始，我逐渐开始感到焦虑和无助，究其根源无非来自于身边同伴的peer pressure，未来就业的压力和不确定性，经济的不景气。

以至于现在我发现自己会产生排斥互联网的情绪，不想看到别人的评论，不想看到别人输出意识形态，不想看到别人的观点。今年曾经有几个月我注册了脉脉天天看看里面的社区帖子，后来我逐渐对于网上贩卖的焦虑感到恶心和排斥，于是删号卸载。我感觉自己在某些方面算是个急性子，情绪也容易受到影响，或许少看些是对我有好处的。

要问对未来的打算，说不上来，没有打算。大概就是活着，悲观地活着。

今年读了哪些书

Linux设备驱动程序：Corbet和Greg K-H大佬的书，膜拜。
操作系统导论：我发现这类书就是看的时候一时爽，看完后啥都不记得。
System Design Interview：学习到不少系统设计相关的知识，对于架构挺有帮助，但是目前还没遇到面试考这些。
Understanding Distributed Systems：分布式系统的介绍书籍，还没读完，感觉不错。
eBPF之巅：看了前面几章，还是感觉得实践实践。

明年准备做什么

想找到一份暑期实习和毕业后的工作
想去旅游
希望心态可以好起来
继续翻译Linux文档，再看看代码

写在最后

没啥好说的，啥也不是。祝愿我们都有美好未来，好日子还在后头呢！

Create Your Network With NetBird

Sat, 12 Nov 2022 14:16:51 +0800

It’s recommended to read my new post: Why I Switch From Netbird to Tailscale.

Recently, I am introducing a change to the cloud infrastructure of myself. That is, installing NetBird on all my servers and change firewall configuration to ensure some essential services can only be visited by trusted devices. So, why did I this, and what’s the problem it can resolve?

Long existing problem

For a very long period of time, I am exposing my services’ ports directly to the Internet. To avoid MITM (Man-in-the-middle attack) and other kinds of security threats, I choose to use forced TLS encryption connection for my database instances and access some services use SSH tunnel instead of expose them. Moreover, I really like ufw and use it to limit SSH connection attempts. However, with the number of services goes larger and larger, I felt it’s a burden for me to manage all the certificates (although there is auto-renewal mechanism). And it obviously is far from best practice.

What’s more, I have a NAS hosted around one thousand kilometers away from me. And I use IPv6 to access it. If there is anything goes wrong with DDNS, I could not know the address of that machine. It’s a single point of failure.

NetBird

NetBird is generally a service for managing your WireGuard VPN’s secret and provide some user-friendly ways to control your network.

They provide some APIs to authenticate and issue VPN connection configuration like key pair and peers’ information. If you are interested in how it really works when a NetBird client goes online, you could see: General Flow Overview.

After setting up all clients, your network may look like this:

And every server will own a IPv4 address in 100.64.0.0/10. If both sides have public addresses or are in a same local network, they can use P2P connection. Otherwise, NetBird could offer a relay service when P2P connection is not possible. What’s more interesting, they released a new feature named network routes. Let me use a list to illustrate how it works.

Create a route with specified CIDR of your private network and routing node.
NetBird client will add an entry in all servers’ route table, let all traffic send to the target CIDR goes to the NetBird’s network interface.
Traffic is redirected to the selected routing node.
The routing node will replace the IP address of the received package with its IP address in the private network. And send the package to the target IP address.
After received the response, things are quite similar. But the direction is opposite.

You can also disable masquerade to make your NetBird network transparent to your private network. But you should add route manually on the servers in your private network. Otherwise, the server does not know which route it can send the package back. The default gateway does not work. For more information, please check Routing Traffic to Private Networks.

*: Images are from the official NetBird document. The GitHub repository is at netbirdio/docs.

Better network

So now, I can use the Web UI to check the IP address and status of my nodes. I can also add ufw rules like this:

`1`	`ufw allow in on wt0 from 100.64.0.0/10 to any port 80`

The 80 port could only be visited within the NetBird network. As the connection itself is encrypted, we are no longer need to use TLS. :)

I am looking forward to share more things with you, see you later.

迈出为Linux贡献的第一步

Wed, 09 Nov 2022 20:25:03 +0800

本文以GPL-2.0协议授权。

Licensed under GPL-2.0.

忘了从什么时候开始，我喜欢将自己的用户名叫和Kernel相关的，就像我的GitHub用户名一样（KernelErr）。突然有一天我突发奇想，既然我叫Kernel，能不能给Linux这个大家都知道的Kernel做些贡献？于是我开始搜索相关的入门教程，阅读Linux相关书籍。

目前我已有一个驱动的cleanup和多个中文文档翻译的PATCH被采用，用中文写这篇博文的目的也在于呼吁大家有空也可以加入到内核开发的行列中来。从我的视角来看，Linux社区非常友好，如果你犯了错，大家通常会友好地指出来，不必有心理负担。同时Linux也是一个非常有名的开源项目，你的贡献总有一天可以被他人看到。

相信你可以

前段时间从Inside Rust博客中看到了这么一篇文章：Imposter Syndrome。文章介绍了“冒名顶替”综合征这一软件社区中的问题，我们每个人知道的事物实际上是不同的，所以不必过度认为自己“不够格”或者“一无所知”。我们每个人都不是完美的，都犯过错。

从哪儿开始

在内核文档中有这么一部分：Getting started with Kernel development - How the development process works，介绍了如何上手内核开发。这里我引用时奎亮 Alex Shi的中文翻译：

“可以理解的是，单个开发人员往往对起步感到茫然。从一个大型项目开始可能会很吓人；人们往往想先用一些较小的东西来试试水。由此，一些开发人员开始创建修补拼写错误或轻微编码风格问题的补丁。不幸的是，这样的补丁会产生一定程度的噪音，这会分散整个开发社区的注意力，因此，它们越来越被人不看重。希望向社区介绍自己的新开发人员将无法通过这些方式获得他们期待的反响。

…

在没有明显问题需要解决的情况下，通常建议开发人员查看当前的回归和开放缺陷列表。从来都不缺少需要解决的问题；通过解决这些问题，开发人员将从该过程获得经验，同时与开发社区的其他成员建立相互尊重。”

很遗憾，我的水平有限，所以我的第一个PATCH选择了做cleanup（清除不必要的大括号），从那以后我开始翻译Linux文档，个人认为文档也是一个好的贡献的点。如果你实在觉得无从下手，可以先关注一下Linux邮件列表，你可以从Majordomo lists at VGER.KERNEL.ORG中选择一个或者多个自己感兴趣的邮件列表订阅，订阅的方法就是点击subscribe，会发从一封邮件到majordomo@vger.kernel.org，其正文是订阅邮件列表的命令。注意，发送邮件时需要保证邮件的Content-Type为纯文本，许多在线邮件客户端是带有样式的，需要手动指定为纯文本。过几分钟你会收到来自服务器的确认邮件，之后你就可以收到任何发到订阅的邮件列表的邮件。

直接订阅linux-kernel列表需要三思，你的邮箱很可能一天收到几百封邮件！

订阅邮件列表的好处是你可以第一时间知道内核相关方向的动态，知道最新的PATCH，也可以参与其中。

第一个PATCH

在邮件列表中观望许久的你终于决定修改一些代码/文档，并提交第一个PATCH，这里我们以文档为例。

正确的git仓库

Linux有许多仓库，在发版的时候Linus会从各个仓库merge到他的仓库。假如我想给内核文档做贡献，我需要先知道我该clone下来哪个仓库，你可以在Linux源码树的根目录下找到MAINTAINERS文件，其中写明了源码树不同区域的维护者和仓库信息，就像这样：

DOCUMENTATION
M:	Jonathan Corbet <corbet@lwn.net>
L:	linux-doc@vger.kernel.org
S:	Maintained
P:	Documentation/doc-guide/maintainer-profile.rst
T:	git git://git.lwn.net/linux.git docs-next
F:	Documentation/
F:	scripts/documentation-file-ref-check
F:	scripts/kernel-doc
F:	scripts/sphinx-pre-install
X:	Documentation/ABI/
X:	Documentation/admin-guide/media/
X:	Documentation/devicetree/
X:	Documentation/driver-api/media/
X:	Documentation/firmware-guide/acpi/
X:	Documentation/i2c/
X:	Documentation/power/
X:	Documentation/spi/
X:	Documentation/userspace-api/media/

那么你就可以从git://git.lwn.net/linux.git仓库的docs-next中找到最新的文档源码，并在此基础上做修改。

检查更改

Linux提供了一个脚本来检查代码、PATCH中常见的错误，其位于scripts/checkpatch.pl。作出修改后请务必先使用该脚本看看文件有没有问题：

`1`	`./scripts/checkpatch.pl -f --terse --show-types FILE_NAME`

没有任何输出说明没有问题，如果有WARNING或者ERROR就需要非常注意了。脚本的具体用法可以查看文档。

commit message

当你做出你的修改并测试无误，准备好提交后，这时候需要git commit：

1
2

git add files_you_edited
git commit -s -v

你需要使用-s来添加你的签名（What is the Sign Off feature in Git for?）。之后你需要编写一个言简意赅的title和详细说明更改的body：

需要注意的是Linux强调“one commit do one thing”，这方便维护者来审阅你的更改。所以如果你做了多个不同方向的更改，请给每个更改一个commit，而不是合成一个大commit提交。多个commit叫做“PATCH SET”。

打包 commit

git commit后，你需要打包commit并生成PATCH文件，如果你只有一个commit，可以执行下面的命令：

`1`	`git format-patch -o /tmp/ HEAD^`

如果你有多个commit需要打包成一个PATCH SET，可以执行下面的命令，需要替换FIRST COMMIT ID为包含的第一个commit id，LAST COMMIT ID为最后一个commit id：

`1`	`git format-patch -o /tmp/ --cover-letter -n --thread=shallow [FIRST COMMIT ID]^..[LAST COMMIT ID]`

发送PATCH

终于，我们准备好提交PATCH，但是等等！在哪儿提交pull request（误）？

Linux不同于GitHub上面的开源项目，你需要通过邮件发送PATCH，而不是提交pull request/merge requst。这颇有一种美好的年代感🙂。

邮件服务器的要求

首先请不要使用email relay service来发送邮件，实测表明这类服务并不能很好处理一些邮件头部信息，也不能正确处理好邮件的Message-Id。

邮件客户端可以通过Message-Id和In-Reply-To来正确折叠邮件成topic的样子，而不是平行显示邮件。

其次你的邮箱服务器需要支持SMTP协议发信，非常不推荐直接通过网页邮箱来发送PATCH，这非常可能导致你的PATCH不是纯文本的！

你的邮箱服务器最好正确配置了SPF和DKIM，SPF记录可以确定邮件发自正确的邮件服务器，DKIM可以进一步确保邮件发件人地址不是伪造的。如果你的邮件服务器存在问题，这会给维护者使用b4工具审阅PATCH时造成困难。

邮件客户端的要求

推荐阅读：Linux邮件客户端配置信息。目前我正在使用Thunderbird，还没遇到问题，使用推荐的邮件客户端的好处是在回复邮件时，可以以正确的格式引用原文，而不是以类似outlook的方式折叠原文。

Rui Li <me@lirui.org> writes:

> 嘟嘟嘟~

啦啦啦~

原文需要用>来折叠，Thunderbird等客户端会自动帮你处理好。

发送给谁

如果你不确定PATCH格式方面有没有问题，可以在发送PATCH前先发到我的邮箱 me@lirui.org ，我看到后会帮你检查格式。

Linux提供了一个脚本告诉你文件的维护者信息，运行下面的命令可以获取文件的维护者：

`1`	`perl scripts/get_maintainer.pl --separator , --nokeywords --nogit --nogit-fallback --norolestats -f FILE_NAME`

通常来说邮件的To应该是直接维护者，CC应该是邮件列表等。中文文档翻译你可以To Alex和Yanteng，CC给其他邮件地址。

发送邮件

这里我们使用git send-email来发送邮件，如果你是Debian系系统，可以安装git-email。git send-email可以自动帮你处理好邮件的ID等细节，非常方便。如何配置还请参阅Git - git-send-email Documentation，里面有以gmail为例子的配置文件。

`1`	`git send-email --to "..." --cc "..." PATCH_FILES`

to和cc对应上面的邮件地址，PATCH_FILES可以是之前format-patch生成的一个文件，或者对应PATCH SET的多个文件。你可以先发给我试试😊：

`1`	`git send-email --to "Rui Li <me@lirui.org>" PATCH_FILES`

收到审阅

当你提交PATCH后可能有两种情况：

收到Reviewed-by签名，这说明你的PATCH已经被审阅，之后请耐心等待PATCH被采用。
审阅者提出了修改意见。

如果提交后几天都没有消息，还请耐心等待，大家都很忙。对于收到了修改意见的情况，还请回复修改意见邮件，表示你的态度，是否作出修改，这对其他人也有参考意义。

发送修改后的PATCH

做出修改后，需要重新写git commit message，发送更新后的PATCH。这时候除了原来的message，你还需要在签名后添加一个change log，用三个短横线分割：

docs/zh_CN: Add userspace-api/seccomp_filter Chinese translation

Translate the following documents into Chinese:

- userspace-api/seccomp_filter.rst

Also adjust index order according to the original index file.

Signed-off-by: Rui Li <me@lirui.org>
---
Changes since v3:
- Improve translation.

Changes since v2:
- Fix some typo.

Changes since v1:
- Add a note for Seccomp.
- Fix some typo.

重新发送PATCH的方式大体上没有区别，但你需要在PATCH后添加版本号：

`1`	`git format-patch --subject-prefix="PATCH v2" -o /tmp/ HEAD^`

通过--subject-prefix可以修改前缀，添加版本号，通常第二个版本就是v2，需要添加v1的change log，告诉审阅者你做了什么修改。

遗漏的部分

如果本文遗漏了什么内容，或者你有什么疑问，欢迎通过邮件联系我，工作时间大概率秒回~（如果求职邮件可以被秒回就好了🥹）

有用的链接

邮件列表：http://vger.kernel.org/vger-lists.html
邮件归档：https://lore.kernel.org/
内核文档：https://docs.kernel.org/
git仓库（大多数）：https://git.kernel.org/

写在最后

路漫漫其修远兮，吾将上下而求索。万事开头难。

Explore GitHub Codespaces

Fri, 14 Oct 2022 21:41:40 +0800

There are less than 3 months left for this year. And I felt a little disappointed and anxious about the future of mine and software industry, especially for undergraduate employment. Anyway, things have to be moved on. So let’s explore the GitHub Codespaces today.

In my perspective, it’s a cloud hosted VS Code instance with customizable configuration. You can set what VS Code plugin should be installed, what feature like SSH server should be chosen, and even bring your Dockerfile. This feature is still in preview, if you have access to it, you may see this green bottom in the code section.

Configuration of Codespaces is placed at the .devcontainer directory. Configuration of Codespaces is placed at the container directory. You may want to check the introduction to dev containers document. The JSON file contains the configuration options like opened feature, Dockerfile location, run arguments and other staffs. The format detailed is available at https://aka.ms/devcontainer.json. And here is my example configuration for the Rust demo project.

// For format details, see https://aka.ms/devcontainer.json. For config options, see the README at:
// https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/rust
{
	"name": "Rust",
	"build": {
		"dockerfile": "Dockerfile",
		"args": {
			// Use the VARIANT arg to pick a Debian OS version: buster, bullseye
			// Use bullseye when on local on arm64/Apple Silicon.
			"VARIANT": "buster"
		}
	},
	"runArgs": [
		"--cap-add=SYS_PTRACE",
		"--security-opt",
		"seccomp=unconfined"
	],

	// Configure tool-specific properties.
	"customizations": {
		// Configure properties specific to VS Code.
		"vscode": {
			// Set *default* container specific settings.json values on container create.
			"settings": { 
				"lldb.executable": "/usr/bin/lldb",
				// VS Code don't watch files under ./target
				"files.watcherExclude": {
					"**/target/**": true
				},
				"rust-analyzer.checkOnSave.command": "clippy"
			},
			
			// Add the IDs of extensions you want installed when the container is created.
			"extensions": [
				"vadimcn.vscode-lldb",
				"mutantdino.resourcemonitor",
				"rust-lang.rust-analyzer",
				"tamasfe.even-better-toml",
				"serayuzgur.crates"
			]
		}
	},

	// Use 'forwardPorts' to make a list of ports inside the container available locally.
	// "forwardPorts": [],

	// Use 'postCreateCommand' to run commands after the container is created.
	// "postCreateCommand": "rustc --version",

	// Comment out to connect as root instead. More info: https://aka.ms/vscode-remote/containers/non-root.
	"remoteUser": "vscode",
	"features": {
		"git": "latest",
		"github-cli": "latest",
		"azure-cli": "latest",
		"sshd": "latest"
	}
}

My Dockerfile looks much more straight. We only defined the Debian version of base image and where the base image is.

# See here for image contents: https://github.com/microsoft/vscode-dev-containers/tree/v0.245.2/containers/rust/.devcontainer/base.Dockerfile

# [Choice] Debian OS version (use bullseye on local arm64/Apple Silicon): buster, bullseye
ARG VARIANT="buster"
FROM mcr.microsoft.com/vscode/devcontainers/rust:0-${VARIANT}

# [Optional] Uncomment this section to install additional packages.
# RUN apt-get update && export DEBIAN_FRONTEND=noninteractive \
#     && apt-get -y install --no-install-recommends <your-package-list-here>

Finally, we can click the code button mentioned before and create our instance.

You can select which branch and configuration file is used. And Codespaces are available in the following four regions right now:

US East
Southeast Asia
Europe West
US West

There are also four machine type to choose:

2 Core 4GB RAM 32GB
4 Core 8GB RAM 32GB
8 Core 16GB RAM 64GB
16 Core 32GB RAM 128GB

This time I’d like to use the machine with 4 cores, as our workload is not heavy. This machine is quite decent. After clicking the create button, you will be redirected to a new website, and it’ll show you the progress of building the container.

After the building progress is finished, we can inter the online VS Code and begin to write some code. I will use the hello world from poem web framework this time.

What’s more interesting, if there is a new port opened to accept connections, Codespaces will create a forward for it. Then you can access it via a URL. If you visit that URL with cURL, you may find a redirection. That’s because the opened port must be accessible after authed with your GitHub account.

You can also push the commit directly, as you have logged in with your GitHub account.

You can also manage all instances of a specific project and delete unused ones if you like.

At the end, I am very excited to introduce the prebuild feature to you. You can prebuild the dev container image and make it available in all regions.

Next time, if we start Codespaces, it will download the prebuilt image and save your time. :)

Upgrading Debian Kernel From Source

Fri, 07 Oct 2022 22:08:10 +0800

Recently, I decided to dive deeper into Linux Kernel. So, I subscribed to some email lists and want to create a VM dedicated for running latest kernel. My first choice is Ubuntu, but it appears that Ubuntu (or Debian I think) has an annotation file limiting you to choose values of flags. Then I installed a Debian and make it running with a Linux 6.0.0-rc4+ kernel.

The step was quite straight and simple, we just need to:

Clone the Kernel source code
Copy the config of the current kernel
Make some changes in menu config
Compile and install

There are some essential commands I ran:

sudo apt install build-essential linux-source bc kmod cpio flex libncurses5-dev libelf-dev libssl-dev dwarves bison fakeroot
sudo apt-get build-dep linux
mkdir -p ~/git/kernels; cd git/kernels
git clone -b staging-testing git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/staging.git
cd staging
sudo cp /boot/config-5.10.0-18-amd64 .config
make menuconfig
make deb-pkg -j8

If you meet some errors like a subprocess returned an exit status. And the detailed error message is:

`1`	`make[4]: *** No rule to make target 'debian/certs/debian-uefi-certs.pem', needed by 'certs/x509_certificate_list'. Stop.`

You can get the file from Debian’s GitLab: debian-uefi-certs.pem, and put it to the right location. You may also want to enable some debug flags to make kernel development easier.

Most debug flags are located at Kernel Hacking menu, whiling some flags like driver debug is at Device Drivers menu.

If build succeed, you can find these file and install them by apt or dpkg.

It highly recommended that backup your machine before booting into the new kernel.

After a reboot, we are using the new kernel!

Write a Simple Linux Kernel Module

Tue, 04 Oct 2022 15:13:36 +0800

Note that all code showed in this article is licensed under GPL-2.0 and MPL-2.0.

Recently, I am interested in Linux kernel space and want to develop some simple kernel module. Although Rust for Linux is merging into the main line of Linux git repository, and will become available at Linux 6.1 release. I still want to let it be in C, and it’s more native and doesn’t need to pay attention to type conversion.

What we will do is create a custom char device and this device will send all written data to a remote cloud server in UDP protocol.

First, I assume you are running an Ubuntu or Debian distribution, other Linux flavors are also supported. But you should ensure Linux kernel header package is available. In my test, Debian in Windows Subsystem for Linux and Debian for Raspberry Pi is incapable. You may install the packages need by the following command:

`1`	# apt install kmod build-essential linux-headers-`uname -r`

And then let’s create the following file:

Makefile

obj-m += repipe.o

PWD := $(CURDIR)

all:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) modules

clean:
	make -C /lib/modules/$(shell uname -r)/build M=$(PWD) clean

chardev.h

#ifndef CHARDEV_H
#define CHARDEV_H

#include <linux/ioctl.h>

#define DEVICE_FILE_NAME "repipe"
#define DEVICE_PATH "/dev/repipe"

#endif

repipe.c

#include <linux/init.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/moduleparam.h> 
#include <linux/sched.h>
#include <linux/tty.h>
#include <linux/cdev.h>
#include <linux/socket.h>
#include <linux/net.h>
#include <linux/ip.h>
#include <linux/in.h>
#include <linux/uio.h>
#include <linux/inet.h>

MODULE_LICENSE("Dual MPL/GPL");
MODULE_AUTHOR("Rui Li <me@lirui.org>");
MODULE_DESCRIPTION("Create a char device which sends data to remote server");

static char *remote_host = "localhost";
static int remote_port = 1040;

module_param(remote_host, charp, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
MODULE_PARM_DESC(remote_host, "Remote host to connect to");
module_param(remote_port, int, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
MODULE_PARM_DESC(remote_port, "Remote port to connect to");

#include "chardev.h"

#define DEVICE_NAME "repipe"
#define BUF_LEN 1024

static int major;
static atomic_t pipe_inuse = ATOMIC_INIT(0);
static char msg[BUF_LEN + 1];
static struct class *cls;
static struct sockaddr_in remote_addr;
static struct socket *remote_sock;

enum {
    REPIPE_FREE = 0,
    REPIPE_INUSE = 1,
};

static int device_open(struct inode *inode, struct file *file);
static int device_release(struct inode *inode, struct file *file);
static ssize_t device_read(struct file *filp, char __user *buff, size_t len, loff_t *off);
static ssize_t device_write(struct file *filp, const char __user *buff, size_t len, loff_t *off);

static struct file_operations fops = {
    .read = device_read,
    .write = device_write,
    .open = device_open,
    .release = device_release,
};

static int __init repipe_init(void) {
    int ret_val;

    major = register_chrdev(0, DEVICE_NAME, &fops);

    if (major < 0) {
        pr_alert("Registering repipe device failed with %d", major);
        return major;
    }

    cls = class_create(THIS_MODULE, DEVICE_FILE_NAME);
    device_create(cls, NULL, MKDEV(major, 0), NULL, DEVICE_FILE_NAME);

    pr_info("Repipe device registered with major number %d on %s", major, DEVICE_PATH);

    memset(&remote_addr, 0, sizeof(remote_addr));
    remote_addr.sin_family = AF_INET;
    remote_addr.sin_port = htons(remote_port);
    remote_addr.sin_addr.s_addr = in_aton(remote_host);

    remote_sock = sock_alloc();

    ret_val = sock_create_kern(&init_net, AF_INET, SOCK_DGRAM, IPPROTO_UDP, &remote_sock);
    if (ret_val < 0) {
        pr_alert("Repipe failed to create socket: %d", ret_val);
        return ret_val;
    }

    ret_val = remote_sock->ops->connect(remote_sock, (struct sockaddr *)&remote_addr, sizeof(remote_addr), 0);
    if (ret_val < 0) {
        pr_alert("Repipe failed to connect to remote host: %d", ret_val);
        return ret_val;
    }

    return 0;
}

static void __exit repipe_exit(void) {
    device_destroy(cls, MKDEV(major, 0));
    class_destroy(cls);
    sock_release(remote_sock);
    unregister_chrdev(major, DEVICE_NAME);
    pr_info("Repipe device unregistered");
}

static int device_open(struct inode *inode, struct file *file) {
    if (atomic_cmpxchg(&pipe_inuse, REPIPE_FREE, REPIPE_INUSE) == REPIPE_INUSE) {
        return -EBUSY;
    }

    try_module_get(THIS_MODULE);

    return 0;
}

static int device_release(struct inode *inode, struct file *file) {
    atomic_set(&pipe_inuse, REPIPE_FREE);

    module_put(THIS_MODULE);

    return 0;
}

static ssize_t device_read(struct file *filp, char __user *buff, size_t len, loff_t *off) {
    return -EINVAL;
}

static ssize_t device_write(struct file *filp, const char __user *buff, size_t len, loff_t *off) {
    static struct msghdr msg_hdr;
    static struct kvec vec;
    int i;
    int ret_val;

    for (i = 0; i < len && i < BUF_LEN; i++) {
        get_user(msg[i], buff + i);
    }

    msg_hdr.msg_name = &remote_addr;
    msg_hdr.msg_namelen = sizeof(remote_addr);
    vec.iov_base = msg;
    vec.iov_len = i;

    ret_val = kernel_sendmsg(remote_sock, &msg_hdr, &vec, 1, i);

    if (ret_val < 0) {
        pr_alert("Repipe failed to send message: %d", ret_val);
        return ret_val;
    }

    return i;
}

module_init(repipe_init);
module_exit(repipe_exit);

There are about 160 lines of code, and the main logic is concentrated on repipe.c. The first lines we defined the license, author name and description of this module using macro. And then there are two parameters it takes: the remote host and port. That’s fairly straight and simple, those macros are defined at include/linux/module.h. The third parameter of module_param function is the permission of accordant file located at sysfs. In this example, you can find the parameter also available at /sys/module/repipe/parameters/remote_host.

MODULE_LICENSE("Dual MPL/GPL");
MODULE_AUTHOR("Rui Li <me@lirui.org>");
MODULE_DESCRIPTION("Create a char device which sends data to remote server");

static char *remote_host = "localhost";
static int remote_port = 1040;

module_param(remote_host, charp, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
MODULE_PARM_DESC(remote_host, "Remote host to connect to");
module_param(remote_port, int, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH);
MODULE_PARM_DESC(remote_port, "Remote port to connect to");

The repipe_init function is called when loading the module, it will be responsible for creating the char device and initialize the socket. On Linux, devices have a major number and a minor number, here is a list for it. We use 0 as our major number, so Linux would assign one for us. That’s could avoid conflict.

static int __init repipe_init(void) {
    int ret_val;

    major = register_chrdev(0, DEVICE_NAME, &fops);

    if (major < 0) {
        pr_alert("Registering repipe device failed with %d", major);
        return major;
    }

    cls = class_create(THIS_MODULE, DEVICE_FILE_NAME);
    device_create(cls, NULL, MKDEV(major, 0), NULL, DEVICE_FILE_NAME);

    pr_info("Repipe device registered with major number %d on %s", major, DEVICE_PATH);

    memset(&remote_addr, 0, sizeof(remote_addr));
    remote_addr.sin_family = AF_INET;
    remote_addr.sin_port = htons(remote_port);
    remote_addr.sin_addr.s_addr = in_aton(remote_host);

    remote_sock = sock_alloc();

    ret_val = sock_create_kern(&init_net, AF_INET, SOCK_DGRAM, IPPROTO_UDP, &remote_sock);
    if (ret_val < 0) {
        pr_alert("Repipe failed to create socket: %d", ret_val);
        return ret_val;
    }

    ret_val = remote_sock->ops->connect(remote_sock, (struct sockaddr *)&remote_addr, sizeof(remote_addr), 0);
    if (ret_val < 0) {
        pr_alert("Repipe failed to connect to remote host: %d", ret_val);
        return ret_val;
    }

    return 0;
}

Another important function is device_write. When we write data into /dev/repipe, device_open is called first, and then is device_write. device_open ensure there is only one process is writing data. device_write will copy data from user space to kernel space and send data. The struct msghdr is used to describe the message to send, the most essential fields are msg_name (remote address) and its length. And the kvec struct is quite like iovec but it’s for kernel space. Finally, we can send our data:

static ssize_t device_write(struct file *filp, const char __user *buff, size_t len, loff_t *off) {
    static struct msghdr msg_hdr;
    static struct kvec vec;
    int i;
    int ret_val;

    for (i = 0; i < len && i < BUF_LEN; i++) {
        get_user(msg[i], buff + i);
    }

    msg_hdr.msg_name = &remote_addr;
    msg_hdr.msg_namelen = sizeof(remote_addr);
    vec.iov_base = msg;
    vec.iov_len = i;

    ret_val = kernel_sendmsg(remote_sock, &msg_hdr, &vec, 1, i);

    if (ret_val < 0) {
        pr_alert("Repipe failed to send message: %d", ret_val);
        return ret_val;
    }

    return i;
}

Here is a screenshot from a real machine.

By the way, seems that VM in Hyper-V has a very high UDP loss rate. Or it could not send UDP data gram at all. If you want to send data to other machines, don’t use Hyper-V VM. I tested on my VPS, it works fine.

That’s all for today. Thanks for reading. Hopefully, I will share more knowledge about Linux.

Air Quality Monitor With 4G Network

Sat, 24 Sep 2022 19:22:30 +0800

After moving to new office and rummage my luggage, I found a 4G development board with a Air724UG model chip and a PM 2.5 detector module with friendly UART output. So, why not make it sending real-time air quality data like PM 2.5 and temperature to somewhere and visualize it?

Hardware

I think it’s one of the best development board I’ve ever bought, as it supports LTE Cat.1, Bluetooth, Wi-Fi, camera and many other interface and features. But it only needs 69 CNY when on sale. But one of the back draws is that it lacks of well-organized document. And it only supports Lua and C development. They provide high-level APIs with Lua, and the usage of extent and standard is split into several documents. Their IDE integrated with VS Code also seems not working well.

The sensor I will use is 702 which you can easily find on many online shopping platforms. One of the interesting points is many resellers have this sensor and rename it with their name. That would be like they are actually the upstream factory. According to my test, its error is acceptable. Except, carbon dioxide and formaldehyde concentrations are calculated from TVOC rather than measurement value.

Software

The infrastructure would be like the image showed. The SIM card specially used for IoT does not support connecting to common ports like 80 and 443. To simplify the debug and development process, I will use a relay server rather than connecting the IoT platform directly with MQTT protocol.

The sensor will start to send data once connected with power supply, but carbon dioxide and other data will only be accurate after warming up. The communication method is protocol and decode script is simple:

    -- Check data length
    if string.len(data) ~= 17 then
        log.error("airmonitor::parse_uart - invalid data length")
        return
    end

    -- Byte 17 is checksum
    local checksum = string.byte(data, 17)
    log.debug("airmonitor::parse_uart - checksum: " .. checksum)
    -- Add all bytes together
    local sum = 0
    for i = 1, 16 do sum = sum + string.byte(data, i) end
    -- Checksum is the last byte
    if sum % 256 == checksum then
        log.debug("airmonitor::parse_uart - checksum ok")
    else
        log.error("airmonitor::parse_uart - checksum error")
        return
    end

    -- Byte 3 and 4 is eCO2
    local eco2 = string.byte(data, 3) * 256 + string.byte(data, 4)
    log.debug("airmonitor::parse_uart - eCO2: " .. eco2)
    -- Byte 5 and 6 is eCH2O
    local ech2o = string.byte(data, 5) * 256 + string.byte(data, 6)
    log.debug("airmonitor::parse_uart - eCH2O: " .. ech2o)
    -- Byte 7 and 8 is TVOC
    local tvoc = string.byte(data, 7) * 256 + string.byte(data, 8)
    log.debug("airmonitor::parse_uart - TVOC: " .. tvoc)
    -- Byte 9 and 10 is pm2.5
    local pm25 = string.byte(data, 9) * 256 + string.byte(data, 10)
    log.debug("airmonitor::parse_uart - pm2.5: " .. pm25)
    -- Byte 11 and 12 is pm10
    local pm10 = string.byte(data, 11) * 256 + string.byte(data, 12)
    log.debug("airmonitor::parse_uart - pm10: " .. pm10)
    -- Byte 13 and 14 is temperature
    local temperature = string.byte(data, 13) + string.byte(data, 14) / 100
    log.debug("airmonitor::parse_uart - temperature: " .. temperature)
    -- Byte 15 and 16 is humidity
    local humidity = string.byte(data, 15) + string.byte(data, 16) / 100
    log.debug("airmonitor::parse_uart - humidity: " .. humidity)

After decoded the data, we can encrypt it with RSA algorithm and encode it with base64:

    while true do
        while not DATA do sys.wait(1000) end

        if socket.isReady() then
            if udp_client == nil then
                udp_client = socket.udp()
                udp_client:connect("foo", "bar")
            else
                if last_timestamp ~= DATA.timestamp then
                    log.info("airmonitor::send_data", "Sending data to server")
                    data_crypted =
                        crypto.rsa_encrypt("PUBLIC_KEY", [[-----BEGIN PUBLIC KEY-----
                        foo
                        -----END PUBLIC KEY-----]], 2048, "PUBLIC_CRYPT", json.encode(DATA))
                    data_encoded =
                        crypto.base64_encode(data_crypted, #data_crypted)
                    udp_client:send(data_encoded)
                    last_timestamp = DATA.timestamp
                    sys.wait(2000)
                end
            end
            -- udp_client:close()
        end
    end

Once our server received the UDP diagram, we can parse the message and relay it to the OneNET IoT platform powered by China Mobile:

#[tokio::main]
async fn main() -> io::Result<()> {
    let sock = UdpSocket::bind("0.0.0.0:12345").await?;
    let mut buf = [0; 1024];
    loop {
        let (len, addr) = sock.recv_from(&mut buf).await?;
        println!("{:?} bytes received from {:?}", len, addr);
        let decoded = ignore!(decode(&buf[..len]));

        // RSA decryption
        let private_key = RsaPrivateKey::from_pkcs8_pem(PRIVATE_KEY).unwrap();
        let padding = PaddingScheme::new_pkcs1v15_encrypt();
        let decrypted = ignore!(private_key.decrypt(padding, &decoded));
        let decrypted = String::from_utf8(decrypted).unwrap();

        // Deserialize
        let message: Message = serde_json::from_str(&decrypted).unwrap();
        println!("{:?}", message);

        // Push data to OneNET
        ignore!(onenet::push_data(message).await);
    }
}

Here is my code used to authorize OneNET API:

pub fn get_token() -> String {
    let current_time = SystemTime::now()
        .duration_since(UNIX_EPOCH)
        .unwrap()
        .as_secs();

    let version = "2018-10-31";
    let et = current_time + 3600;
    let res = format!("products/{}/devices/{}", PRODUCT_ID, DEVICE_NAME);
    let method = "sha1";

    let string_for_sign = format!("{}\n{}\n{}\n{}", et, method, res, version);

    let key_decoded = base64::decode(PRODUCT_KEY).unwrap();
    let mut mac = HmacSha1::new_from_slice(&key_decoded).unwrap();
    mac.update(string_for_sign.as_bytes());
    let result = mac.finalize();
    let signature = base64::encode(result.into_bytes());

    format!(
        "version={}&res={}&et={}&method={}&sign={}",
        version, encode(&res), et, method, encode(&signature)
    )
}

Finally, we can check all the data on OneNET platform.

Improvements

Current infrastructure is not elegant enough. I will try directly connect the OneNET platform with MQTT later. In previous tests, the socket is closed unexpected after first packages sent to the MQTT server. That’s still under investigation. Or we can send data to Prometheus and visualize it with Grafana.

Observability With Grafana and Vector

Thu, 15 Sep 2022 14:41:43 +0800

Recently, I am migrating my personal cloud infrastructure including some VMs and databases. For a better understanding how much work load is on each machine and if databases or nginx under high pressure. I choose to use Vector to collect metrics and Grafana products to save and visualize logs and metrics.

Vector

Vector is a high-performance observability data pipeline according to its GitHub description. In my understanding it consists of tree parts:

Sources: Vector supports logs and metrics input, they are called sources. For example, access logs from Nginx, host metrics like CPU and network usage, and simply a text file.
Transforms: After collected logs and metrics, we could write some scripts with VRL or use built-in rules. We could change hostname tag in this step or change any data.
Sinks: That’s where our data finally goes! You can select Loki, Prometheus, Kafka and other destinations.

Vector is not the only software could do log forwarding stuff. But I think Vector has some advantages you may like:

It’s written by Rust, build the binary is simple, and memory safe is out of the box.
It’s faster in some cases, check their performance test.
Vector could check if data is received by some sinks. The file offset should not be advanced if data was failed to be delivered to destinations. Vector has a comparison with other similar software products in data correctness.

Drawbacks

Although the Vector is under active maintaining. There are still some bugs and pain points with it.

Could not support customized Nginx log format well. Seems that Vector uses a built-in expression of log format to parse them. You may need to change some code and build your binary to get customized log format supported.
Some MongoDB metrics datatype is not correct. That would cause an overflow problem when the number is bigger than the maximum value of a 32-bit integer. I am following this issue and created some PR. After my investigation, MongoDB returns dynamic datatype, if the number overflows the maximum value of a 32-bit integer, returned data in BSON will be a 64-bit integer. So, we’d better change all integer to 64-bit. In MongoDB 4, one field is a float number, but it’s an integer in MongoDB 5. Mapping all correct datatype may be a hard work.

Grafana Cloud

Grafana Cloud consists of the following parts:

Grafana
Prometheus
Loki
Graphite
Alerts
Tempo

I am currently using the free tier service. It’s fairly enough for an individual with few servers and services. Allow me to introduce some services I used:

Loki: It’s basically a log aggregation system without indexing the whole log content. In my practice, Nginx access and error logs will be presented to Loki by Vector. And Loki would use some tags from Vector like hostname, app, forward to index the logs. You may also add more tags by VRL scripts.

Prometheus: This is a more well-known product by Grafana Labs. We usually use it to monitoring Docker, Kubernetes, and our applications. Vector could send host metrics (CPU, network, file system etc.) and Nginx metrics (active connection number) to it.

Grafana: Grafana allows us to explore and visualize data collected by Prometheus and Loki.

Integration

Configuration of Vector is simple and straight. Just following the official guide to write a config TOML file and pass its path to Vector.

Here is an example of config file:

data_dir = "/var/lib/vector"

[api]
enabled = true

[sources.host_metrics]
type = "host_metrics"
scrape_interval_secs = 15

[transforms.host_metrics_trans]
type = "remap"
inputs = [ "host_metrics" ]
source = """
.tags.host = "example-1"
"""

[sinks.host_metric_prom]
type = "prometheus_remote_write"
inputs = [ "host_metrics_trans" ]
endpoint = "https://prometheus-endpoint/"

    [sinks.host_metric_prom.auth]
    strategy = "basic"
    user = "53847239"
    password = "XJdG6A9b9ayNKijYoAMty"

We use transforms to change the host tag in host metrics, and send it to Prometheus. Then we could visualize and monitor them on Grafana!

To add more sources like Nginx or log tail is quite similar. That’s all, I’d like to share more things about observability if I have more findings later. Technologies like eBPF now are high-profile these days!

Start New Blog

Wed, 14 Sep 2022 21:09:06 +0800

Due to some reasons, I decided to start a new blog and use English as the main language of articles. The old blog (https://lirui.tech) will be kept, but no longer be updated. This new blog will use CDN service to better reader experience for global user.

After two internship experience and meet some professional people, hopefully I will create more in-depth contents about computer and how things really works. It’s essential to improve our skills, especially at this period of time - many tech companies in China are decreasing head count of the new campus recruitment session.

In this COVID-19 pandemic, we face more hard challenges than even before. Many sad things are playing out as well. We could see the economy downturn that takes toll on employment. I hope that we are all doing well, and our future will still be bright. :)

About

Mon, 01 Jan 0001 00:00:00 +0000

Yet another normal backend engineer.

This blog is using forked version of theme Stack.

Contact

Email: me[at]lirui.org (replace [at] with @)
GitHub: @KernelErr

Search

Mon, 01 Jan 0001 00:00:00 +0000

周末記事

Mon, 01 Jan 0001 00:00:00 +0000