Threading issues with Puma

[cbenning]

Started seeing these:

#<NoMethodError: undefined method `borrow_or_take' for nil:NilClass> /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/aws-xray-sdk-0.11.2/lib/aws-xray-sdk/sampling/default_sampler.rb:85:in `process_matched_rule' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/aws-xray-sdk-0.11.2/lib/aws-xray-sdk/sampling/default_sampler.rb:54:in `sample_request?' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/aws-xray-sdk-0.11.2/lib/aws-xray-sdk/facets/helper.rb:36:in `should_sample?' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/aws-xray-sdk-0.11.2/lib/aws-xray-sdk/facets/rack.rb:30:in `call' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/puma-4.0.1/lib/puma/configuration.rb:228:in `call' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/puma-4.0.1/lib/puma/server.rb:657:in `handle_request' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/puma-4.0.1/lib/puma/server.rb:467:in `process_client' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/puma-4.0.1/lib/puma/server.rb:328:in `block in run' /usr/local/rbenv/versions/2.4.0/lib/ruby/gems/2.4.0/gems/puma-4.0.1/lib/puma/thread_pool.rb:135:in `block in spawn_thread'

However we aren't using sampling so I'm not sure whats up.

[ss2305]

@cbenning The sampling decision coming from trace_header always has the highest precedence. If the trace header doesn't contain sampling decision then it checks to see if sampling is enabled or no in the recorder. If not enabled it returns 'true'.

When you don't use sampling, the sampler looks for previously made decision based on default rule if
no path-based rule has been matched. I think that's why you see this error.

[cbenning]

So I guess that means that sampling: true by default and I'm using it without realizing it?

[ss2305]

@cbenning The default rule traces the first request each second, and five percent of any additional requests across all services sending traces to X-Ray. If the SDK can't reach X-Ray to get sampling rules, it reverts to a default local rule of the first request each second, and five percent of any additional requests per host. This can occur if the host doesn't have permission to call sampling APIs, or can't connect to the X-Ray daemon, which acts as a TCP proxy for API calls made by the SDK.

You can find more in the documentation here.

[cbenning]

This error only happens occasionally, as far as we can tell it is working fine otherwise. Are you suggesting this is a connectivity/latency issue with the local xray daemon?

[ss2305]

@cbenning If it's sporadic in how this breaks then yes. However if you can reproduce the issue consistently and share with us a sample it would be greatly appreciated.

[cbenning]

Ok @ss2305 I can't reproduce it reliably so I will just treat it as intermittent for now and keep an eye on it.

thanks

[cbenning]

@ss2305 Also, this triggers alerts for us, what is a safe way to suppress them? Can I add a default sampling rule? It still feels to me that this should not be stacktracing if this is just business-as-usual in this situation.

Would increasing the Concurrency setting improve this potentially? The x-ray daemon in our instance is using default config, but I don't see why it would be unreachable

[chanchiem]

We're taking a deeper look at this and we will update it when we have any new findings. Thanks for letting us know about this issue.

[cbenning]

FYI Increasing Concurrency from 8 -> 24 had no effect

[thegorgon]

We're experiencing the same issue - sporadically, requests will fail while making a sampling decision.

Looks like this issue is quite stale. Was there any update or mitigation discovered?

[cbenning]

@thegorgon not that we have discovered. We've basically just turned the sampling rate down to almost nothing and have basically stopped using x-ray with ruby.

[willarmiros]

@thegorgon I know this is a sporadic issue, but are there any patterns with which you've noticed this consistently fails? Any help in reproducing this like a sample app (even if the error only occurs intermittently) would be very useful. For some context from an initial inspection, it looks like this error must be happening because a sampling rule doesn't have a reservoir. Given that SamplingRules are always initialized with a non-Nil Reservoir, I'd guess something weird may be happening in this rule merge logic.

Are you using Centralized sampling? If so can you describe your use case?