Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid redundant shutdown in TracerProvider::drop when already shut down #2197

Open
wants to merge 15 commits into
base: main
Choose a base branch
from
Open
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
88 changes: 78 additions & 10 deletions opentelemetry-sdk/src/trace/provider.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,22 +36,29 @@ static NOOP_TRACER_PROVIDER: Lazy<TracerProvider> = Lazy::new(|| TracerProvider
span_limits: SpanLimits::default(),
resource: Cow::Owned(Resource::empty()),
},
is_shutdown: AtomicBool::new(true),
Copy link
Contributor

@utpilla utpilla Oct 21, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not added in this PR but why have we initialized the no-op tracer provider as an already shut down provider?

I know this wouldn't make much difference in functionality but semantically it would be weird if I call shutdown on the global provider and get an error saying it has already been shut down.

}),
is_shutdown: Arc::new(AtomicBool::new(true)),
});

/// TracerProvider inner type
#[derive(Debug)]
pub(crate) struct TracerProviderInner {
processors: Vec<Box<dyn SpanProcessor>>,
config: crate::trace::Config,
is_shutdown: AtomicBool,
}

impl Drop for TracerProviderInner {
fn drop(&mut self) {
for processor in &mut self.processors {
if let Err(err) = processor.shutdown() {
global::handle_error(err);
if self
.is_shutdown
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
.is_ok()
{
cijothomas marked this conversation as resolved.
Show resolved Hide resolved
for processor in &self.processors {
if let Err(err) = processor.shutdown() {
global::handle_error(err);
}
}
}
}
Expand All @@ -65,7 +72,6 @@ impl Drop for TracerProviderInner {
#[derive(Clone, Debug)]
pub struct TracerProvider {
inner: Arc<TracerProviderInner>,
is_shutdown: Arc<AtomicBool>,
}

impl Default for TracerProvider {
Expand All @@ -79,7 +85,6 @@ impl TracerProvider {
pub(crate) fn new(inner: TracerProviderInner) -> Self {
TracerProvider {
inner: Arc::new(inner),
is_shutdown: Arc::new(AtomicBool::new(false)),
}
}

Expand All @@ -101,7 +106,7 @@ impl TracerProvider {
/// true if the provider has been shutdown
/// Don't start span or export spans when provider is shutdown
pub(crate) fn is_shutdown(&self) -> bool {
self.is_shutdown.load(Ordering::Relaxed)
self.inner.is_shutdown.load(Ordering::Relaxed)
}

/// Force flush all remaining spans in span processors and return results.
Expand Down Expand Up @@ -153,6 +158,7 @@ impl TracerProvider {
/// Note that shut down doesn't means the TracerProvider has dropped
pub fn shutdown(&self) -> TraceResult<()> {
if self
.inner
.is_shutdown
.compare_exchange(false, true, Ordering::SeqCst, Ordering::SeqCst)
.is_ok()
Expand Down Expand Up @@ -215,7 +221,7 @@ impl opentelemetry::trace::TracerProvider for TracerProvider {
}

fn library_tracer(&self, library: Arc<InstrumentationLibrary>) -> Self::Tracer {
if self.is_shutdown.load(Ordering::Relaxed) {
if self.inner.is_shutdown.load(Ordering::Relaxed) {
return Tracer::new(library, NOOP_TRACER_PROVIDER.clone());
}
Tracer::new(library, self.clone())
Expand Down Expand Up @@ -292,7 +298,12 @@ impl Builder {
p.set_resource(config.resource.as_ref());
}

TracerProvider::new(TracerProviderInner { processors, config })
let is_shutdown = AtomicBool::new(false);
TracerProvider::new(TracerProviderInner {
processors,
config,
is_shutdown,
})
}
}

Expand Down Expand Up @@ -391,6 +402,7 @@ mod tests {
Box::from(TestSpanProcessor::new(false)),
],
config: Default::default(),
is_shutdown: AtomicBool::new(false),
});

let results = tracer_provider.force_flush();
Expand Down Expand Up @@ -534,6 +546,7 @@ mod tests {
let tracer_provider = super::TracerProvider::new(TracerProviderInner {
processors: vec![Box::from(processor)],
config: Default::default(),
is_shutdown: AtomicBool::new(false),
});

let test_tracer_1 = tracer_provider.tracer("test1");
Expand All @@ -554,14 +567,69 @@ mod tests {

// after shutdown we should get noop tracer
let noop_tracer = tracer_provider.tracer("noop");

// noop tracer cannot start anything
let _ = noop_tracer.start("test");
assert!(assert_handle.started_span_count(2));
// noop tracer's tracer provider should be shutdown
assert!(noop_tracer.provider().is_shutdown.load(Ordering::SeqCst));
assert!(noop_tracer.provider().is_shutdown());

// existing tracer becomes noops after shutdown
let _ = test_tracer_1.start("test");
assert!(assert_handle.started_span_count(2));
}

#[test]
fn test_tracer_provider_inner_drop_shutdown() {
// Test 1: Already shutdown case
{
let processor = TestSpanProcessor::new(true);
let assert_handle = processor.assert_info();
let provider = super::TracerProvider::new(TracerProviderInner {
processors: vec![Box::from(processor)],
config: Default::default(),
is_shutdown: AtomicBool::new(false),
});

// Create multiple providers sharing same inner
let provider2 = provider.clone();
let provider3 = provider.clone();

// Shutdown explicitly first
assert!(provider.shutdown().is_ok());

// Drop all providers - should not trigger another shutdown in TracerProviderInner::drop
drop(provider);
drop(provider2);
drop(provider3);

// Verify shutdown was called exactly once
assert!(assert_handle.0.is_shutdown.load(Ordering::SeqCst));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How does this verify that shutdown was called only once? It looks like it's only verifying that shutdown was called (could have been called once or multiple times)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point. I think I should be using CountingShutdownProcessor which was added in #2195.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have made this use CountingShutdownProcessor now.

}

// Test 2: Not shutdown case
{
let processor = TestSpanProcessor::new(true);
let assert_handle = processor.assert_info();
let provider = super::TracerProvider::new(TracerProviderInner {
processors: vec![Box::from(processor)],
config: Default::default(),
is_shutdown: AtomicBool::new(false),
});

// Create multiple providers sharing same inner
let provider2 = provider.clone();
let provider3 = provider.clone();

// Drop providers without explicit shutdown
drop(provider);
drop(provider2);

utpilla marked this conversation as resolved.
Show resolved Hide resolved
// Last drop should trigger shutdown in TracerProviderInner::drop
drop(provider3);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add assert_eq!(shutdown_count.load(Ordering::SeqCst), 1); here as well to ensure that explicitly calling shutdown on an already shutdown TracerProvider doesn't call shutdown again.

// Verify shutdown was called exactly once
assert!(assert_handle.0.is_shutdown.load(Ordering::SeqCst));
}
}
}
Loading