查看: 204|回復: 0

[教學] Python + OpenCV 抓桌面動作

發表於 2020-9-29 11:48:42 | 顯示全部樓層 |閱讀模式
Push to Facebook Push to Plurk  
I'm trying to process live screen. There is a game about catching fish. You have to click on fish when it is in circle. I think that I can process my screen with opencv, find fish and click on it with pyautogui.

I did it but problem is program not fast enough to click. By the way game is a mini game in Metin 2 mmorpg. It is like a hack or bot but I just wondering if can I do that.

Here is my code:
  1. import numpy as np
  2. import cv2
  3. from PIL import ImageGrab
  4. import pyautogui
  5. import time

  6. while True:

  7.     img=ImageGrab.grab(bbox=(341,208,430,290))
  8.     img_np=np.array(img)
  9.     #gray=cv2.cvtColor(img_np, cv2.COLOR_BGR2GRAY)

  10.     #lower=np.array([57,91,120])
  11.     #upper=np.array([65,95,160])

  12.     #mask=cv2.inRange(gray,95,130)

  13.     #sonuc=cv2.bitwise_and(gray,gray,mask=mask)
  14.     #cv2.imshow('frame',mask)

  15.     degsk=np.argwhere(img_np==[123,90,57])
  16.     if len(degsk)!=0:
  17.         #print(degsk)
  18.         yerx=341+degsk[int(len(degsk)/2),1]
  19.         yery=208+degsk[int(len(degsk)/2),0]
  20.         #pyautogui.click(x=yerx, y=yery)
  21.         time.sleep(0.8)

  22.     if cv2.waitKey(1)&0xFF==ord('q'):
  23.         break

  24. cv2.destroyAllWindows()

As you can see, first I tried mask the screen than I realise that it is not necessary so I found BGR value of fish and programed to find it in numpy array than I took value middle in the array and than i used mouse move function. As I said this is not fast enough to catch fish.
So the program is working but delayed for catch fish. How can I make faster this program?
[color=var(--blue-700)]Game Screen Here

I find IPython best for timing things, so if you start IPython and paste in the following code:
  1. from PIL import ImageGrab

  2. img=ImageGrab.grab(bbox=(341,208,430,290))

You can then time a statement with:
  1. %timeit img=ImageGrab.grab(bbox=(341,208,430,290))

and I get this:
  1. 552 ms ± 5.91 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)

So, grabbing the screen takes over 500ms, so you are only going to get under 2 frames/second - without even processing it.
If you want to grab the screen faster, I would suggest ffmpeg. I installed it on my iMac running macOS with homebrew using:
  1. brew install ffmpeg

I can then see the list of available video sources and find what I need to record the screen with:
  1. ffmpeg -f avfoundation -list_devices true -i ""

Sample Output
  1. [AVFoundation input device @ 0x7fa7dcf05b40] AVFoundation video devices:
  2. [AVFoundation input device @ 0x7fa7dcf05b40] [0] FaceTime HD Camera
  3. [AVFoundation input device @ 0x7fa7dcf05b40] [1] Capture screen 0             <--- THIS ONE IS THE SCREEN
  4. [AVFoundation input device @ 0x7fa7dcf05b40] AVFoundation audio devices:
  5. [AVFoundation input device @ 0x7fa7dcf05b40] [0] MacBook Pro Microphone
  6. [AVFoundation input device @ 0x7fa7dcf05b40] [1] CalDigit Thunderbolt 3 Audio

So I know I need input 1 for the screen.
So, if I want to record the screen from top-left corner (0,0) at a width of 400px and height of 200px at 20fps for 10s and pass RGBA8888 data to my fishing program, I can do this:
  1. fmpeg -y -pix_fmt bgr0 -f avfoundation -r 20 -t 10 -i 1 -filter:v "crop=400:200:0:0" -f rawvideo - | ./fish.py

I can now use the following as my fishing program:
  1. #!/usr/bin/env python3

  2. import numpy as np
  3. import pyautogui
  4. import time
  5. import os, sys

  6. # width, height
  7. w, h = 400, 200

  8. # Bytes per frame - assumes bgr0, i.e. 8-bits of blue, 8-bits red. 8-bits green and 8-bits junk
  9. bytesPerFrame = w * h * 4

  10. while True:
  11.     img = sys.stdin.buffer.read(bytesPerFrame)
  12.     if len(img) != bytesPerFrame:
  13.         break
  14.     # Process your video here

Keywords: pyautogui, screen grab, screen-grab, screengrab, slow, Mac, macOS, Python, capture, screen capture, ffmpeg, PIL, OpenCV


您需要登錄後才可以回帖 登錄 | 註冊 |



GMT+8, 2020-10-21 15:22 , Processed in 0.053982 second(s), 21 queries .

本論壇言論純屬發表者個人意見,與 TShopping綜合論壇 立場無關 如有意見侵犯了您的權益 請寫信聯絡我們。

Powered by Discuz! X3.2

© 2001-2013 Comsenz Inc.

快速回復 返回頂部 返回列表