aws Distro for OpenTelemetry 可观测性workshop记录

news2025/7/16 19:23:21

参考资料

https://aws-otel.github.io/docs/introduction
https://aws-otel.github.io/docs/introduction

aws distro for opentelemetry 官方提供了不同语言不同使用场景下完善的使用实例和相关配置。

AWS Distro for OpenTelemetrics 由以下部分组成，用于向后端服务发送数据

SDK
Auto-instrumentation agent，自动注入代理
OpenTelemetry Collector，收集器

使用opentelemetry的一般架构如以下ecs示例。将opentelemetry collector作为sidecar运行。

opentelemetry基础

所有的opentelemery关注点，比如跟踪和度量，都共享一个底层 Context 机制，用于在分布式事务的整个生命周期中存储状态和访问数据

trace，通过span隐式定义trace，trace是span组成的有向无环图
span，span代表事务中的一个操作，每个span封装以下状态：
- 事件，每个事件都是元组（时间戳，名称，属性）
  - 操作名称
  - 开始和结束的时间戳
  - 属性（键值对）
- span之间的因果关系链接

Http追踪

服务端追踪

使用gorilla/mux HTTP 多路复用器。通过opentelemetry库包装 HTTP 处理程序。对这些端点的所有调用都将根据现有的采样规则自动跟踪并发送到 AWS X-Ray

import(
	go.opentelemetry.io/contrib/instrumentation/github.com/gorilla/mux/otelmux
	github.com/gorilla/mux
	...
)
...
func MakeHttpHandler(s Service, Logger log.logger) http.Handler{
    r := mux.NewRouter()
    r.Use(otelmux.Middleware("petlistadoptions"))
    r.Methods("GET").Path("/api/adoptionlist/").Handler(handlerFunc)
    return r
}

客户端追踪

使用go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp封装客户端的http调用

client := http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
req, _ := http.NewRequestWithContext(ctx, "GET", url, nil)

对于没有提供自动追踪检测支持的库调用，可以通过span和go context创建自定义追踪

tracer := otel.GetTracerProvider().Tracer("petlistadoptions")
_, span := tracer.Start(ctx, "PGSQL Query", trace.WithSpanKind(trace.SpanKindClient))

sql := `SELECT pet_id, transaction_id, adoption_date FROM transactions ORDER BY id DESC LIMIT 25`

// injecting custom attributes
span.SetAttributes(
  label.String("sql", sql),
  label.String("url", r.safeConnStr),
)

rows, err := r.db.Query(sql)
if err != nil {
  handleErr(err)
}
span.End()

java追踪代理

https://github.com/aws-samples/one-observability-demo

使用java代理并修改JVM启动参数，使得自动台历能够收集追踪信息

ADD https://github.com/aws-observability/aws-otel-java-instrumentation/releases/download/version/aws-opentelemetry-agent.jar /app/aws-opentelemetry-agent.jar
ENV JAVA_TOOL_OPTIONS "-javaagent:/app/aws-opentelemetry-agent.jar"

其他的opentelemetry配置

# OpenTelemetry agent configuration
ENV OTEL_TRACES_SAMPLER "always_on"
ENV OTEL_PROPAGATORS "tracecontext,baggage,xray"
ENV OTEL_RESOURCE_ATTRIBUTES "service.name=PetSearch" # 资源属性
ENV OTEL_IMR_EXPORT_INTERVAL "10000" # 将OTLP指标导出到collector的频率
ENV OTEL_EXPORTER_OTLP_ENDPOINT "http://localhost:4317"	 # collector监听断电

像aws sdk和流行框架都得到了opentelemetry的支持，对于java来说任何经由spring MVC的请求和对aws服务的调用都会被追踪和发布。

lambda追踪

lambda提供了专门的ADTO lambda layer，自动检测lambda函数，使用步骤如下

添加ADOT lambda层
在lambda函数中添加环境变量AWS_LAMBDA_EXEC_WRAPPER = /opt/otel-instrument
在lambda函数中启动Active Tracing

示例cdk代码如下

var adotLayerArn = "arn:aws:lambda:"+ process.env.CDK_DEFAULT_REGION + ":901920570463:layer:aws-otel-python-amd64-ver-1-11-1:2"
var adotlayer = lambda.LayerVersion.fromLayerVersionArn(this,'otelPythonLambdaLayer',adotLayerArn);
var layers: lambda.ILayerVersion[] = [adotlayer]

// lambda自动追踪请求，业务代码中没有相关逻辑
new pythonlambda.PythonFunction(this, lambdaFileName, {
      entry: './resources/stepfn_lambdas/',
      index: lambdaFileName + '.py',
      handler: 'lambda_handler',
      memorySize: 128,
      runtime: lambda.Runtime.PYTHON_3_9,
      role: lambdaRole,
      layers: lambdalayers, //加入otel层
      tracing: Tracing.ACTIVE // 启用主动追踪
    });
pythonFn.addEnvironment("AWS_LAMBDA_EXEC_WRAPPER", "/opt/otel-instrument") //添加环境变量

python示例

https://github.com/aws-samples/one-observability-demo/tree/main/PetAdoptions/petadoptionshistory-py

https://catalog.workshops.aws/observability/en-US/aws-managed-oss/adot/python-instrumentation/tracing

手动追踪

配置环境

pip3 install opentelemetry-api
pip3 install opentelemetry-sdk
pip3 install opentelemetry-distro
pip3 install opentelemetry-exporter-otlp-proto-grpc
pip3 install opentelemetry-sdk-extension-aws
pip3 install opentelemetry-propagator-aws-xray

导入依赖

# OTLP Tracing
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource, get_aggregated_resources

# Exporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter

# Propagation
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.aws import AwsXRayPropagator

# AWS X-Ray ID Generator
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator

# Resource detector
from opentelemetry.sdk.extension.aws.resource.eks import AwsEksResourceDetector

手动追踪，最终去哦那个球

# Setup AWX X-Ray Propagator
# 跨 AWS 服务注入或提取跟踪上下文
set_global_textmap(AwsXRayPropagator())

# Setup AWS EKS resource detector 
# 从 EKS 环境中检测到的额外资源字段(例如集群名称和容器 ID)添加到单个跟踪中
resource = get_aggregated_resources(
    [
        AwsEksResourceDetector(), 
    ]
)

# Setup tracer provider with the X-Ray ID generator
tracer_provider = TracerProvider(resource=resource, id_generator=AwsXRayIdGenerator())
processor = BatchSpanProcessor(OTLPSpanExporter())
tracer_provider.add_span_processor(processor)

# Sets the global default tracer provider
trace.set_tracer_provider(tracer_provider)

# Creates a tracer from the global tracer provider
tracer = trace.get_tracer(__name__)

在flask中手动创建span追踪，注意start_as_current_span

@app.route('/api/home/transactions', methods=['GET'])
def transactions_get():
    with tracer.start_as_current_span("transactions_get") as transactions_span:
        transactions_get_counter.add(1)
        transactions = repository.list_transaction_history(db)
        return jsonify(transactions)

自动追踪

自动检测 boto3和 psycopg2之类的库调用来生成跟踪

pip3 install opentelemetry-instrumentation-botocore
pip3 install opentelemetry-instrumentation-flask

封装调用

# Instrumentation
from opentelemetry.instrumentation.botocore import BotocoreInstrumentor
from opentelemetry.instrumentation.flask import FlaskInstrumentor

# Instrumentation
BotocoreInstrumentor().instrument()

# Setup flask app
app = Flask(__name__)
FlaskInstrumentor().instrument_app(app)
...