【WebRTC-13】是在哪，什么时候，创建编解码器？

Android-RTC系列软重启，改变以往细读源代码的方式改为带上实际问题分析代码。增加实用性，方便形成肌肉记忆。同时不分种类、不分难易程度，在线征集问题切入点。

问题：编解码器的关键实体类是什么？在哪里&什么时候创建的？

这个问题是在分析webrtc如何增加第三方外置的编解码库时额外提出来的，在找答案的过程中领略webrtc内部代码结构组织的划分。废话不多，这个问题的关键可以想到之前的一个问题 webrtc是如何确定双端的编解码类型？是在sdp交换信息后，local和remote的description两者结合确认。那么可以在这基础上继续寻找，也就是去SdpOfferAnswerHandler找答案。

我们直接定位到 SdpOfferAnswerHandler:: ApplyLocalDescription / ApplyRemoteDescription。

RTCError SdpOfferAnswerHandler::ApplyLocalDescription(
    std::unique_ptr<SessionDescriptionInterface> desc,
    const std::map<std::string, const cricket::ContentGroup*>&  bundle_groups_by_mid) 
{
  pc_->ClearStatsCache();

  RTCError error = PushdownTransportDescription(cricket::CS_LOCAL, type);

  if (IsUnifiedPlan()) {
    UpdateTransceiversAndDataChannels(...)
  } else {
    ... ...
  }

  UpdateSessionState(type, cricket::CS_LOCAL,
                             local_description()->description(),
                             bundle_groups_by_mid);
  // Now that we have a local description, we can push down remote candidates.
  UseCandidatesInRemoteDescription();

  ... ...
}

大致的逻辑如上，这里关注 UpdateSessionState，继续深入。

RTCError SdpOfferAnswerHandler::UpdateSessionState(
    SdpType type,  cricket::ContentSource source,
    const cricket::SessionDescription* description,
    const std::map<std::string, const cricket::ContentGroup*>&  bundle_groups_by_mid) {
  // If this is answer-ish we're ready to let media flow.
  if (type == SdpType::kPrAnswer || type == SdpType::kAnswer) {
    EnableSending();
  }
  // Update the signaling state according to the specified state machine (see
  // https://w3c.github.io/webrtc-pc/#rtcsignalingstate-enum).
  if (type == SdpType::kOffer) {
    ChangeSignalingState(source == cricket::CS_LOCAL
                             ? PeerConnectionInterface::kHaveLocalOffer
                             : PeerConnectionInterface::kHaveRemoteOffer);
  } else if (type == SdpType::kPrAnswer) {
    ChangeSignalingState(source == cricket::CS_LOCAL
                             ? PeerConnectionInterface::kHaveLocalPrAnswer
                             : PeerConnectionInterface::kHaveRemotePrAnswer);
  } else {
    RTC_DCHECK(type == SdpType::kAnswer);
    ChangeSignalingState(PeerConnectionInterface::kStable);
    if (ConfiguredForMedia()) {
      transceivers()->DiscardStableStates();
    }
  }
  // Update internal objects according to the session description's media descriptions.
  return PushdownMediaDescription(type, source, bundle_groups_by_mid);
}

根据输入的type改变信令状态。注意最后的 PushdownMediaDescription，这里看函数名字有点奇怪，其核心功能是检索新的sdp信息，更新 rtp_transceivers 的channel

RTCError SdpOfferAnswerHandler::PushdownMediaDescription(
    SdpType type,  cricket::ContentSource source,
    const std::map<std::string, const cricket::ContentGroup*>&  bundle_groups_by_mid) 
{
  const SessionDescriptionInterface* sdesc =
      (source == cricket::CS_LOCAL ? local_description() : remote_description());
  // Push down the new SDP media section for each audio/video transceiver.
  auto rtp_transceivers = transceivers()->ListInternal();

std::vector<std::pair<cricket::ChannelInterface*, const MediaContentDescription*>> 
 channels;

  for (const auto& transceiver : rtp_transceivers) {
    const ContentInfo* content_info =
        FindMediaSectionForTransceiver(transceiver, sdesc);
    cricket::ChannelInterface* channel = transceiver->channel();

    const MediaContentDescription* content_desc = content_info->media_description();

    channels.push_back(std::make_pair(channel, content_desc));
  }


  for (const auto& entry : channels) {
    std::string error;
    bool success = context_->worker_thread()->BlockingCall([&]() {
      return (source == cricket::CS_LOCAL)
                 ? entry.first->SetLocalContent(entry.second, type, error)
                 : entry.first->SetRemoteContent(entry.second, type, error);
    });
  }
  return RTCError::OK();
}

（这里的channel以后介绍）伪代码如上。可以看到关于ChannelInterface的关键方法 SetLocalContent 、SetRemoteContent。

文件位置  src/pc/channel.cc
bool BaseChannel::SetLocalContent(const MediaContentDescription* content,
                                  SdpType type, std::string& error_desc) {
  RTC_DCHECK_RUN_ON(worker_thread());
  TRACE_EVENT0("webrtc", "BaseChannel::SetLocalContent");
  return SetLocalContent_w(content, type, error_desc);
}
bool BaseChannel::SetRemoteContent(const MediaContentDescription* content,
                                   SdpType type, std::string& error_desc) {
  RTC_DCHECK_RUN_ON(worker_thread());
  TRACE_EVENT0("webrtc", "BaseChannel::SetRemoteContent");
  return SetRemoteContent_w(content, type, error_desc);
}

SetLocalContent_w / SetRemoteContent_w又到具体的媒体通道类VoiceChannel / VideoChannel实现，以VideoChannel为例，精简核心代码如下。

bool VideoChannel::SetLocalContent_w(const MediaContentDescription* content,
                                     SdpType type, std::string& error_desc) 
{
  RtpHeaderExtensions header_extensions =
      GetDeduplicatedRtpHeaderExtensions(content->rtp_header_extensions());

  media_send_channel()->SetExtmapAllowMixed(content->extmap_allow_mixed());

  VideoReceiverParameters recv_params = last_recv_params_;
  VideoSenderParameters send_params = last_send_params_;

  MediaChannelParametersFromMediaDescription(
      content, header_extensions,
      webrtc::RtpTransceiverDirectionHasRecv(content->direction()),
      &recv_params);
  
  media_receive_channel()->SetReceiverParameters(recv_params);
  media_send_channel()->SetSenderParameters(send_params);

  UpdateLocalStreams_w(content->streams(), type, error_desc)

  UpdateMediaSendRecvState_w();
}

这里的UpdateLocalStreams_w又回到了BaseChannel。这里有一大段注释比较关键，主要描述了：在媒体协商的过程中SSRC与StreamParams相关联，构成安全的 local_stream_成员对象。

bool BaseChannel::UpdateLocalStreams_w(const std::vector<StreamParams>& streams,
                                       SdpType type, std::string& error_desc) 
{
  // In the case of RIDs (where SSRCs are not negotiated), this method will
  // generate an SSRC for each layer in StreamParams. That representation will
  // be stored internally in `local_streams_`.
  // In subsequent offers, the same stream can appear in `streams` again
  // (without the SSRCs), so it should be looked up using RIDs (if available)
  // and then by primary SSRC.
  // In both scenarios, it is safe to assume that the media channel will be
  // created with a StreamParams object with SSRCs. However, it is not safe to
  // assume that `local_streams_` will always have SSRCs as there are scenarios
  // in which niether SSRCs or RIDs are negotiated.
  ... ...
  media_send_channel()->AddSendStream(new_stream);
}

到这里我们先停顿一下，因为发现这里出现了众多 Channel 对象，ChannelInterface、BaseChannel、VoiceChannel/VideoChannel、media_send_channel/media_receive_channel。它们究竟是什么关系？我绘制了一张简易的UML图，这张图概括了Channel 以及之后要介绍的 Stream的内部关系，提取了核心代码的常见方法。大家一定要放大仔细看看！

小结：这部分阐述了 webrtc如何从sdp提取信息，根据ssrc创建并绑定网络传输通器中的 Channel。并通过代码，由BaseChannel->Video/VoiceChannel承接rtp数据包。

有了上面的UML图预热，在进入 media_send_channel()->AddSendStream的流程之前，我们要搞清楚这个 media_send_channel是怎么来的。

回到文章最开始的 SdpOfferAnswerHandler::ApplyLocalDescription 的UpdateTransceiversAndDataChannels。关键代码逻辑如下，可以看到涉及ChannelInterface的通道，是由RtpTransceiver内部创建的。

RTCError SdpOfferAnswerHandler::UpdateTransceiversAndDataChannels(
    cricket::ContentSource source,
    const SessionDescriptionInterface& new_session,
    const SessionDescriptionInterface* old_local_description,
    const SessionDescriptionInterface* old_remote_description,
    const std::map<std::string, const cricket::ContentGroup*>& bundle_groups_by_mid) 
{
    const ContentInfos& new_contents = new_session.description()->contents();
    
    for (size_t i = 0; i < new_contents.size(); ++i) {
      const cricket::ContentInfo& new_content = new_contents[i];

      auto transceiver_or_error =
          AssociateTransceiver(source, new_session.GetType(), i, new_content,
                               old_local_content, old_remote_content);
      auto transceiver = transceiver_or_error.Value();

      RTCError error= UpdateTransceiverChannel(transceiver, new_content, bundle_group);
    }
}

RTCError SdpOfferAnswerHandler::UpdateTransceiverChannel(
    rtc::scoped_refptr<RtpTransceiverProxyWithInternal<RtpTransceiver>> transceiver,
    const cricket::ContentInfo& content,
    const cricket::ContentGroup* bundle_group) 
{
    cricket::ChannelInterface* channel = transceiver->internal()->channel();
    if (!channel) {
      auto error = transceiver->internal()->CreateChannel(...);
    }
}

RtpTransceiver的CreateChannel内部，其核心是调用media_engine去创建对应的SendChannel / ReceiveChannel，最终组成RtpTransceiver的 VideoChannel / VoiceChannel。

RTCError RtpTransceiver::CreateChannel(
    absl::string_view mid,
    Call* call_ptr,
    const cricket::MediaConfig& media_config,
    bool srtp_required,
    CryptoOptions crypto_options,
    const cricket::AudioOptions& audio_options,
    const cricket::VideoOptions& video_options,
    VideoBitrateAllocatorFactory* video_bitrate_allocator_factory,
    std::function<RtpTransportInternal*(absl::string_view)> transport_lookup) 
{
  std::unique_ptr<cricket::ChannelInterface> new_channel;

  if (media_type() == cricket::MEDIA_TYPE_VIDEO) {
    std::unique_ptr<cricket::VideoMediaSendChannelInterface>
          media_send_channel = media_engine()->video().CreateSendChannel(...);

    std::unique_ptr<cricket::VideoMediaReceiveChannelInterface>
          media_receive_channel = media_engine()->video().CreateReceiveChannel(...);

    new_channel = std::make_unique<cricket::VideoChannel>(
          worker_thread(), network_thread(), signaling_thread(), 
          std::move(media_send_channel), std::move(media_receive_channel), ...);
  } else {
    // media_type() == cricket::MEDIA_TYPE_AUDIO
  }

  SetChannel(std::move(new_channel), transport_lookup);
  return RTCError::OK();
}

根据以往的文章，我们就可以快速定位到 src/media/engine/webrtc_video_engine / webrtc_audio_engine，找到SendChannel ReceiveChannel。至此，我们正式定位到了media_send_channel()的具体实现。

// 以media_type==video为例
std::unique_ptr<VideoMediaSendChannelInterface>
WebRtcVideoEngine::CreateSendChannel(
    webrtc::Call* call,
    const MediaConfig& config,
    const VideoOptions& options,
    const webrtc::CryptoOptions& crypto_options,
    webrtc::VideoBitrateAllocatorFactory* video_bitrate_allocator_factory) {
  return std::make_unique<WebRtcVideoSendChannel>(
      call, config, options, crypto_options, encoder_factory_.get(),
      decoder_factory_.get(), video_bitrate_allocator_factory);
}
std::unique_ptr<VideoMediaReceiveChannelInterface>
WebRtcVideoEngine::CreateReceiveChannel(
    webrtc::Call* call,
    const MediaConfig& config,
    const VideoOptions& options,
    const webrtc::CryptoOptions& crypto_options) {
  return std::make_unique<WebRtcVideoReceiveChannel>(
      call, config, options, crypto_options, decoder_factory_.get());
}

小结：根据m=session创建RtpTransceiver，Video/VoiceChannel由webrtc_medie_engine创建，并保存在RtpTransceiver网络传器者的成员变量。Video/VoiceChannel 内包含SendChannel和ReceiveChannel。

回头再看media_send_channel()->AddSendStream(new_stream)，即WebRtcVideoSendChannel::AddSendStream，其核心逻辑很简单：

bool WebRtcVideoSendChannel::AddSendStream(const StreamParams& sp) 
{
  WebRtcVideoSendStream* stream = new WebRtcVideoSendStream(
      call_, sp, std::move(config), default_send_options_,
      video_config_.enable_cpu_adaptation, bitrate_config_.max_bitrate_bps,
      send_codec(), send_rtp_extensions_, send_params_);

  uint32_t ssrc = sp.first_ssrc();
  send_streams_[ssrc] = stream;
}

WebRtcVideoSendStream 的构造函数内容比较多，但都是属性赋值。我们这里只关心文章提出的问题，也就是构造函数里唯一调用的成员函数SetCodec。

// src/media/engine/webrtc_video_engine.cc
void WebRtcVideoSendChannel::WebRtcVideoSendStream::SetCodec(
    const VideoCodecSettings& codec_settings) 
{
  parameters_.encoder_config = CreateVideoEncoderConfig(codec_settings.codec);
  parameters_.config.rtp = ...
  parameters_.codec_settings = codec_settings;
  // TODO(bugs.webrtc.org/8830): Avoid recreation, it should be enough to call  ReconfigureEncoder.
  RTC_LOG(LS_INFO) << "RecreateWebRtcStream (send) because of SetCodec.";
  RecreateWebRtcStream();
}

void WebRtcVideoSendChannel::WebRtcVideoSendStream::RecreateWebRtcStream() 
{
  if (stream_ != NULL) {
    call_->DestroyVideoSendStream(stream_);
  }
  stream_ = call_->CreateVideoSendStream(std::move(config),
                                         parameters_.encoder_config.Copy());
  // Attach the source after starting the send stream to prevent frames from
  // being injected into a not-yet initializated video stream encoder.

  // rtc::VideoSourceInterface<webrtc::VideoFrame>* source_
  if (source_) {
    stream_->SetSource(source_, GetDegradationPreference());
  }
}

具体实现还要到Call::CreateVideoSendStream。这里有个细节，stream_->SetSource(webrtc::VideoFrame )出现了VideoFrame，显然路是找对了，继续往下。

//  src/call/call.cc
webrtc::VideoSendStream* Call::CreateVideoSendStream(
    webrtc::VideoSendStream::Config config,
    VideoEncoderConfig encoder_config,
    std::unique_ptr<FecController> fec_controller) 
{
  VideoSendStreamImpl* send_stream = new VideoSendStreamImpl(...);

  for (uint32_t ssrc : ssrcs) {
    RTC_DCHECK(video_send_ssrcs_.find(ssrc) == video_send_ssrcs_.end());
    video_send_ssrcs_[ssrc] = send_stream;
  }
  video_send_streams_.insert(send_stream);
  video_send_streams_empty_.store(false, std::memory_order_relaxed);
}

// src/video/video_send_stream_impl.cc
VideoSendStreamImpl::VideoSendStreamImpl(
    RtcEventLog* event_log,
    VideoSendStream::Config config,
    VideoEncoderConfig encoder_config,
    std::unique_ptr<FecController> fec_controller,
    const FieldTrialsView& field_trials,
    std::unique_ptr<VideoStreamEncoderInterface> video_stream_encoder_for_test)
//构造赋值
: video_stream_encoder_(
          video_stream_encoder_for_test
              ? std::move(video_stream_encoder_for_test)
              : CreateVideoStreamEncoder(...) 
  ), ... ...
//构造VideoStreamEncoder
std::unique_ptr<VideoStreamEncoderInterface> CreateVideoStreamEncoder() {
  std::unique_ptr<TaskQueueBase, TaskQueueDeleter> encoder_queue =
      task_queue_factory->CreateTaskQueue("EncoderQueue",
                                          TaskQueueFactory::Priority::NORMAL);
  TaskQueueBase* encoder_queue_ptr = encoder_queue.get();
  return std::make_unique<VideoStreamEncoder>(...std::move(encoder_queue), ...);
}

到这里我们就找到了webrtc的视频编码实体类VideoStreamEncoder（src/video/video_stream_encoder.cc），相对应的解码实体类VideoStreamDecoder。这里放一些关键方法，此类是个宝库，任何关于视频编码功能的细节，都可以在这找到参考。

总结答案：整篇文章跟踪的代码逻辑如下，归纳了从Sdp->RtpTransceiver->VideoChannel/VoiceChannel->Send&ReceiveChannel-> 然后根据sdp::ssrc创建VideoSendStream-> VideoStreamEncoder。

还待挖掘的细节非常多，奈何篇幅有限有所侧重。在写本文章的时候，其实是在研究icecandidate，stun服务端已经搭建起来了。希望有兴趣的同学联系我，一起深入成长。

SdpOfferAnswerHandler:: ApplyLocalDescription / ApplyRemoteDescription（sdp信息）
SdpOfferAnswerHandler::UpdateTransceiversAndDataChannels -> UpdateTransceiverChannel（创建RtpTransceiver->Video/VoiceChannel）
SdpOfferAnswerHandler::UpdateSessionState
SdpOfferAnswerHandler::PushdownMediaDescription
BaseChannel::SetLocalContent(const MediaContentDescription* content, ..)
VoiceChannel/VideoChannel::SetLocalContent_w
BaseChannel::UpdateLocalStreams_w(const std::vector<StreamParams>& streams, ..)

WebRtcVideoSendChannel::AddSendStream
WebRtcVideoSendChannel::WebRtcVideoSendStream::WebRtcVideoSendStream（Constructor）
WebRtcVideoSendChannel::WebRtcVideoSendStream::SetCodec|::RecreateWebRtcStream|::SetSenderParameters|::ReconfigureEncoder
Call::CreateVideoSendStream
VideoSendStreamImpl() -> VideoStreamEncoder(Interface)