Why would CPython logging use a Lock for each handler rather than one Lock per Logger?












2














While developing my own logging library, I studied the source code of the standard logging module of CPython.



One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.



I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.



What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?



I see two main advantages:




  • Better performances as there is only one lock acquired and released

  • Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1) to happen)


In my personal logging library, can I safely do something like this per logger:



with self.lock:

    for handler in self.handlers:

        handler.handle(message)  # No lock used in "handle()"


Or am I overlooking something else?






python multithreading logging locks







share|improve this question









New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
    – Delgan
    1 hour ago


















2














While developing my own logging library, I studied the source code of the standard logging module of CPython.



One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.



I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.



What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?



I see two main advantages:




  • Better performances as there is only one lock acquired and released

  • Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1) to happen)


In my personal logging library, can I safely do something like this per logger:



with self.lock:

    for handler in self.handlers:

        handler.handle(message)  # No lock used in "handle()"


Or am I overlooking something else?






python multithreading logging locks







share|improve this question









New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.




















  • Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
    – Delgan
    1 hour ago
















2












2








2


0





While developing my own logging library, I studied the source code of the standard logging module of CPython.



One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.



I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.



What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?



I see two main advantages:




  • Better performances as there is only one lock acquired and released

  • Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1) to happen)


In my personal logging library, can I safely do something like this per logger:



with self.lock:

    for handler in self.handlers:

        handler.handle(message)  # No lock used in "handle()"


Or am I overlooking something else?






python multithreading logging locks







share|improve this question









New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











While developing my own logging library, I studied the source code of the standard logging module of CPython.



One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.



I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.



What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?



I see two main advantages:




  • Better performances as there is only one lock acquired and released

  • Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1) to happen)


In my personal logging library, can I safely do something like this per logger:



with self.lock:

    for handler in self.handlers:

        handler.handle(message)  # No lock used in "handle()"


Or am I overlooking something else?






python multithreading logging locks




python multithreading logging locks






share|improve this question









New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.











share|improve this question









New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









share|improve this question




share|improve this question








edited 1 hour ago





















New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.









asked 2 hours ago









Delgan

1166




1166




New contributor




Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.





New contributor





Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.






Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.












  • Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
    – Delgan
    1 hour ago




















  • Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
    – Delgan
    1 hour ago


















Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago






Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago












1 Answer
1






active

oldest

votes


















3














Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).



Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.






share|improve this answer





















  • I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
    – Delgan
    2 hours ago












  • Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
    – Lie Ryan
    1 hour ago












  • Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
    – Delgan
    1 hour ago










  • And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
    – Delgan
    1 hour ago













Your Answer








StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "131"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});






Delgan is a new contributor. Be nice, and check out our Code of Conduct.










draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsoftwareengineering.stackexchange.com%2fquestions%2f384505%2fwhy-would-cpython-logging-use-a-lock-for-each-handler-rather-than-one-lock-per-l%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









3














Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).



Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.






share|improve this answer





















  • I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
    – Delgan
    2 hours ago












  • Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
    – Lie Ryan
    1 hour ago












  • Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
    – Delgan
    1 hour ago










  • And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
    – Delgan
    1 hour ago


















3














Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).



Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.






share|improve this answer





















  • I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
    – Delgan
    2 hours ago












  • Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
    – Lie Ryan
    1 hour ago












  • Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
    – Delgan
    1 hour ago










  • And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
    – Delgan
    1 hour ago
















3












3








3






Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).



Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.






share|improve this answer












Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).



Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.







share|improve this answer












share|improve this answer



share|improve this answer










answered 2 hours ago









Lie Ryan

6,45811825




6,45811825












  • I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
    – Delgan
    2 hours ago












  • Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
    – Lie Ryan
    1 hour ago












  • Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
    – Delgan
    1 hour ago










  • And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
    – Delgan
    1 hour ago




















  • I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
    – Delgan
    2 hours ago












  • Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
    – Lie Ryan
    1 hour ago












  • Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
    – Delgan
    1 hour ago










  • And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
    – Delgan
    1 hour ago


















I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
– Delgan
2 hours ago






I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing logger.log() anyway, right?
– Delgan
2 hours ago














Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
1 hour ago






Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
1 hour ago














Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago




Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago












And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago






And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago












Delgan is a new contributor. Be nice, and check out our Code of Conduct.










draft saved

draft discarded


















Delgan is a new contributor. Be nice, and check out our Code of Conduct.













Delgan is a new contributor. Be nice, and check out our Code of Conduct.












Delgan is a new contributor. Be nice, and check out our Code of Conduct.
















Thanks for contributing an answer to Software Engineering Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.





Some of your past answers have not been well-received, and you're in danger of being blocked from answering.


Please pay close attention to the following guidance:


  • Please be sure to answer the question. Provide details and share your research!

But avoid …



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsoftwareengineering.stackexchange.com%2fquestions%2f384505%2fwhy-would-cpython-logging-use-a-lock-for-each-handler-rather-than-one-lock-per-l%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







-

Popular posts from this blog

Accessing regular linux commands in Huawei's Dopra Linux

Image
4 Backstory: I have a Huawei HG8245 router in my house and I want to change the default username/password. I've tried following this guide but the config file always gets reset. My ISP isn't helping either so I'm looking for other ways to accomplish this. The router has telnet access so this seems like a way to change the credentials. The first shell I gain access to looks like this: WAP> Pressing "?" gives me a list of available commands which I've pasted here, the two interesting ones are "su" and "shell". I can't seem to get su access, even after logging in as root (challenge password prompt, and then nothing happens), but shell gives me a proper linux shell. WAP>shell BusyBox v1.18.4 (2017-12-26 17:06:34 CST) built-in shell (ash) Enter 'hel...
Read more

Can't connect RFCOMM socket: Host is down

Image
up vote 3 down vote favorite I have bluez 5.4.3 and I am attempting to communicate with a ble device through minicom. These two deceives (my computer and a bluno nano) are alreadt paired and this has been my method so far. #bluetoothctl # pair xx:xx:xx:xx:xx:xx # exit Everything is working well so far then... #sudo rfcomm connect /dev/rfcomm0 xx:xx:xx:xx:xx:xx 1 & [1] 3665 Can't connect RFCOMM socket: Host is down Does anyone know how to resolve this issue? Any help would be very much appreciated. bluetooth bluez share | improve this question asked Feb 9 '17 at 18:46 Brandon Knape 46 2 ...
Read more

Kernel panic - not syncing: Fatal Exception in Interrupt

Image
0 Every day this error occured on my pc, kernel panic error on interrupt, what will be the problem? linux-mint share asked 6 mins ago Vinod Selvin Vinod Selvin 1 1 New contributor Vinod Selvin is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct. add a comment ...
Read more