Python + OpenCV 抓桌面動作

woff

I'm trying to process live screen. There is a game about catching fish. You have to click on fish when it is in circle. I think that I can process my screen with opencv, find fish and click on it with pyautogui.

I did it but problem is program not fast enough to click. By the way game is a mini game in Metin 2 mmorpg. It is like a hack or bot but I just wondering if can I do that.

Here is my code:

1. import numpy as np
   
2. import cv2
   
3. from PIL import ImageGrab
   
4. import pyautogui
   
5. import time
   
6. 
   
7. while True:
   
8. 
   
9.     img=ImageGrab.grab(bbox=(341,208,430,290))
   
10.     img_np=np.array(img)
    
11.     #gray=cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)
    
12. 
    
13.     #lower=np.array([57,91,120])
    
14.     #upper=np.array([65,95,160])
    
15. 
    
16.     #mask=cv2.inRange(gray,95,130)
    
17. 
    
18.     #sonuc=cv2.bitwise_and(gray,gray,mask=mask)
    
19.     #cv2.imshow('frame',mask)
    
20. 
    
21. 
    
22.     degsk=np.argwhere(img_np==[123,90,57])
    
23.     if len(degsk)!=0:
    
24.         #print(degsk)
    
25.         yerx=341+degsk[int(len(degsk)/2),1]
    
26.         yery=208+degsk[int(len(degsk)/2),0]
    
27.         #pyautogui.click(x=yerx, y=yery)
    
28.         time.sleep(0.8)
    
29. 
    
30.     if cv2.waitKey(1)&0xFF==ord('q'):
    
31.         break
    
32. 
    
33. cv2.destroyAllWindows()

複製代碼

As you can see, first I tried mask the screen than I realise that it is not necessary so I found BGR value of fish and programed to find it in numpy array than I took value middle in the array and than i used mouse move function. As I said this is not fast enough to catch fish.

So the program is working but delayed for catch fish. How can I make faster this program?

[color=var(--blue-700)]Game Screen Here

I find IPython best for timing things, so if you start IPython and paste in the following code:

1. from PIL import ImageGrab
   
2. 
   
3. img=ImageGrab.grab(bbox=(341,208,430,290))

複製代碼

You can then time a statement with:

1. %timeit img=ImageGrab.grab(bbox=(341,208,430,290))

複製代碼

and I get this:

1. 552 ms ± 5.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

複製代碼

So, grabbing the screen takes over 500ms, so you are only going to get under 2 frames/second - without even processing it.

If you want to grab the screen faster, I would suggest ffmpeg. I installed it on my iMac running macOS with homebrew using:

1. brew install ffmpeg

複製代碼

I can then see the list of available video sources and find what I need to record the screen with:

1. ffmpeg -f avfoundation -list_devices true -i ""

複製代碼

Sample Output

1. [AVFoundation input device @ 0x7fa7dcf05b40] AVFoundation video devices:
   
2. [AVFoundation input device @ 0x7fa7dcf05b40] [0] FaceTime HD Camera
   
3. [AVFoundation input device @ 0x7fa7dcf05b40] [1] Capture screen 0             <--- THIS ONE IS THE SCREEN
   
4. [AVFoundation input device @ 0x7fa7dcf05b40] AVFoundation audio devices:
   
5. [AVFoundation input device @ 0x7fa7dcf05b40] [0] MacBook Pro Microphone
   
6. [AVFoundation input device @ 0x7fa7dcf05b40] [1] CalDigit Thunderbolt 3 Audio

複製代碼

So I know I need input 1 for the screen.

So, if I want to record the screen from top-left corner (0,0) at a width of 400px and height of 200px at 20fps for 10s and pass RGBA8888 data to my fishing program, I can do this:

1. fmpeg -y -pix_fmt bgr0 -f avfoundation -r 20 -t 10 -i 1 -filter:v "crop=400:200:0:0" -f rawvideo - | ./fish.py

複製代碼

I can now use the following as my fishing program:

1. #!/usr/bin/env python3
   
2. 
   
3. import numpy as np
   
4. import pyautogui
   
5. import time
   
6. import os, sys
   
7. 
   
8. # width, height
   
9. w, h = 400, 200
   
10. 
    
11. # Bytes per frame - assumes bgr0, i.e. 8-bits of blue, 8-bits red. 8-bits green and 8-bits junk
    
12. bytesPerFrame = w * h * 4
    
13. 
    
14. while True:
    
15.     img = sys.stdin.buffer.read(bytesPerFrame)
    
16.     if len(img) != bytesPerFrame:
    
17.         break
    
18.     # Process your video here

複製代碼

Keywords: pyautogui, screen grab, screen-grab, screengrab, slow, Mac, macOS, Python, capture, screen capture, ffmpeg, PIL, OpenCV

文章出處