0

Question

I am trying to use gevent to improve my programe to interact with redis, but when use gevent + redis-py in multi threading scenario, i found that the performence is so pool. When i comment the monkey.patch_all(), the performence is good, i cannot understand why.

ENV and dependency

  • python: 2.7
  • redis-py: 3.0.1
  • gevent: 1.2.1

Example code:

from gevent import monkey
monkey.patch_all()   # comment this code, the result will be diffrent

import redis
import time
import threading
import logging

logging.basicConfig(format="%(asctime)s: %(message)s", level=logging.INFO, datefmt="%H:%M:%S")

# Redis will use default connection pool
conn = redis.Redis(host="127.0.0.1", port=6379, socket_timeout=5, socket_connect_timeout=5)

conn.set("key", "value", ex=50000)


def get_key():
    t = time.time()
    conn.get("key")
    t1 = time.time() - t
    if t1 > 1:
        logging.info("!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!  get key fron redis to slow %s" % t1)
    else:
        logging.info("----------------------------------  get key cost time is %s " % t1)


for i in range(1000):
    x = threading.Thread(target=get_key, args=())
    x.start()


logging.info("Main    : all done")

Test result

without monkey.patch_all()

from gevent import monkey
# monkey.patch_all()

without monkey.patch_all(), the redis delay is very small.

13:45:10: ----------------------------------  get key cost time is 0.000197887420654 
13:45:10: ----------------------------------  get key cost time is 0.000231981277466 
13:45:10: ----------------------------------  get key cost time is 0.000230073928833 
13:45:10: ----------------------------------  get key cost time is 0.000216960906982 
13:45:10: ----------------------------------  get key cost time is 0.000287055969238 
13:45:10: ----------------------------------  get key cost time is 0.000190019607544 
13:45:10: ----------------------------------  get key cost time is 0.000202178955078 
13:45:10: ----------------------------------  get key cost time is 0.000118970870972 
13:45:10: ----------------------------------  get key cost time is 0.000120878219604 
13:45:10: Main    : all done

with monkey.patch_all()

from gevent import monkey
monkey.patch_all()

the redis delay is very big, and if i adjust the 1000 threading to bigger, the delay will be more bigger.

13:46:19: ----------------------------------  get key cost time is 0.217961072922 
13:46:19: ----------------------------------  get key cost time is 0.217952013016 
13:46:19: ----------------------------------  get key cost time is 0.218060970306 
13:46:19: ----------------------------------  get key cost time is 0.216876029968 
13:46:19: ----------------------------------  get key cost time is 0.216850996017 
13:46:19: ----------------------------------  get key cost time is 0.216836929321 
13:46:19: ----------------------------------  get key cost time is 0.216396093369 
13:46:19: ----------------------------------  get key cost time is 0.216381072998
Improve this question
4
  • This answer may help you: stackoverflow.com/a/10663498/1055722 Commented Jan 14, 2022 at 16:37
  • thank you very much. yes, i also searched this issue, and i also had try it, but the result is same, it do not improve the performence. Commented Jan 19, 2022 at 12:27
  • Have you tried accumulating your timing results and print them at the end rather than logging from each thread/greenlet and seeing what the difference is? Commented Jan 19, 2022 at 14:28
  • yes, it is. the result is same. Commented Jan 26, 2022 at 4:27
Related questions
1
Speed up Redis server inputs
22
redis + gevent - Poor performance - what am I doing wrong?
2
Why only one connection to redis was made in this gevent program?
Load 7 more related questions Show fewer related questions

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.