Automating video screenshots

02 Oct 2022

When I review videos and want to take notes about it with illustrations - be it Youtube videos I downloaded or business meeting recordings - it's painful to have to use several keyboard shortcuts and mouse actions to get to the point where I can just copy in my note the relevant image (ie HTML code pointing to the path of the saved image).

A script to do this end-to-end would be a time-saver and improve my workflows so as to use illustrations more.

Need to figure out how to do it - AppleScript seems more appropriate than Python for this, considering the need to target a specific app to be screeshot.

Objective

App I want to focus on is my media player of choice: IINA !apps/iina

Single command to trigger: keyboard shortcut and/or Streamdeck key !home-office/streamdeck and /or Contour Shuttle key -ideal !home-office/contour-shuttle

Automates:

  • puts IINA at the foreground
  • capture screenshot of app only
  • saved to a dedicated folder

Output:

  • saved image file with video title and timestamp.
  • HTML code in clipboard, ready to be pasted, with path to image and CSS styling (see !projects/clipee/#clipee-note)

Could be a multi-tool solution, ie shell script + AppleScript + Hazel !apps/hazel.

Gathering infos

property N : 0
set N to N + 1
set picPath to ((POSIX path of (path to desktop)) & "Picture_" & N & ".png") as string
do shell script "screencapture -tjpg " & quoted form of picPath
set {l, t, r, b} to {911, 173, 1738, 794}
set {x, y, w, h} to {l, t, r - l, b - t}
screencapture(x, y, w, h)

on screencapture(x, y, w, h)
    (*
        number x, y, w, h: rectangle parameters
            (x, y) = x, y-coordinates of origin point
            (w, h) = width and height of rectangle

        * screen shot is saved as ~/Desktop/yyyy-mm-dd hh.mm.ss.png
    *)
    set args to ""
    repeat with a in {x, y, w, h}
        set args to args & space & ("" & a)'s quoted form
    end repeat
    considering numeric strings
        if (system info)'s system version < "10.9" then
            set ruby to "/usr/bin/ruby"
        else
            set ruby to "/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby"
        end if
    end considering
    do shell script ruby & " <<'EOF' - " & args & "
require 'osx/cocoa'
include OSX
raise ArgumentError, \"Usage: #{File.basename($0)} x, y, w, h\" unless ARGV.length == 4
x, y, w, h = ARGV.map {|a| a.to_f}
outfile = File.expand_path(%x[date +'%F %H.%M.%S.png'].chomp, '~/Desktop')
img = CGDisplayCreateImageForRect(CGMainDisplayID(), CGRectMake(x, y, w, h))
brep = NSBitmapImageRep.alloc.initWithCGImage(img)
data = brep.objc_send(
    :representationUsingType, NSPNGFileType,
    :properties, {})
data.objc_send(
    :writeToFile, outfile,
    :atomically, false)
EOF"
end screencapture

The command to size a window in AppleScript is:

tell application "Finder" to set the bounds of the front window to {25, 50, 1025, 650}

active application:

activate application "Safari"

Work in progress

1st working script

working:

  • put IINA at foreground
  • screenshot
  • saves file

script-screenshot-app/Picture_1.png

to figure out:

  1. capture app only, not full screen ➤ shell script syntax
  2. capture video file name ➤ AppleScript
  3. capture timestamp ➤ possible? 🤔
  4. generate HTML code ➤ I know how to do it with Python, not sure how to connect both or if I have to redo logic in AppleScript? 🤔

For 1) looking at command line tools that would help, eg.:

  • gnome-screenshot
  • scrot
  • ImageMagick

doesn't seem to be available for macOS

To silently take a screen shot just use the -x flag:

screencapture -x quiet.jpg

Learning of the day: make use more often of the -h (help) switch when a shell command is known. screencapture -h gives:

xxx@xxxx ~ % screencapture -h
screencapture: illegal option -- h
usage: screencapture [-icMPmwsWxSCUtoa] [files]
  -c         force screen capture to go to the clipboard
  -b         capture Touch Bar - non-interactive modes only
  -C         capture the cursor as well as the screen. only in non-interactive modes
  -d         display errors to the user graphically
  -i         capture screen interactively, by selection or window
               control key - causes screen shot to go to clipboard
               space key   - toggle between mouse selection and
                             window selection modes
               escape key  - cancels interactive screen shot
  -m         only capture the main monitor, undefined if -i is set
  -D<display> screen capture or record from the display specified. -D 1 is main display, -D 2 secondary, etc.
  -o         in window capture mode, do not capture the shadow of the window
  -p         screen capture will use the default settings for capture. The files argument will be ignored
  -M         screen capture output will go to a new Mail message
  -P         screen capture output will open in Preview or QuickTime Player if video
  -I         screen capture output will open in Messages
  -B<bundleid> screen capture output will open in app with bundleid
  -s         only allow mouse selection mode
  -S         in window capture mode, capture the screen not the window
  -J<style>  sets the starting of interfactive capture
               selection       - captures screen in selection mode
               window          - captures screen in window mode
               video           - records screen in selection mode
  -t<format> image format to create, default is png (other options include pdf, jpg, tiff and other formats)
  -T<seconds> take the picture after a delay of <seconds>, default is 5
  -w         only allow window selection mode
  -W         start interaction in window selection mode
  -x         do not play sounds
  -a         do not include windows attached to selected windows
  -r         do not add dpi meta data to image
  -l<windowid> capture this windowsid
  -R<x,y,w,h> capture screen rect
  -v        capture video recording of the screen
  -V<seconds> limits video capture to specified seconds
  -g        captures audio during a video recording using default input.
  -G<id>    captures audio during a video recording using audio id specified.
  -k        show clicks in video recording mode
  -U        Show interactive toolbar in interactive mode
  -u        present UI after screencapture is complete. files passed to command line will be ignored
  files   where to save the screen capture, 1 file per screen

so need:

screencapture -w -d -i -o -x -tjpg

-i is interactive so mouse click required to select windown 😕

or trying

screencapture -wdioxtjpg

Working solution seems to be using Shortcuts 🤯

need to figure out if I can get timestamp and video name.

To run: shortcuts run "Screenshot IINA"

Works even when window is not at the foreground 😁

Took a different turn, but now have it so that a key on my Contour Shuttle !home-office/contour-shuttle takes the screenshot and adds to a folder with video name.

Need to figure out how to rename files with no spaces and generate HTML. Probably triggering a Python script with Hazel - will be the quickest.

Enough for today though 😁

Final solution

03 Oct 2022

macOS Shortcuts
+ Hazel
+ Python

Workflow:

when I plan to take screenshots of a video, only manual step is to go in Hazel (Cmd Space then haz) and change the output folder as needed:

script-screenshot-app/221003-1252_hazel_process_iina.jpg

then:

if I am actively in IINA, I have a key on my Contour Shuttle to take the screenshot:

script-screenshot-app/contour-iina.jpg

if I'm outside of IINA - even if it is hidden behind another app! - I use a key on my Streamdeck to take the screenshot:

script-screenshot-app/streamdeck-iina.jpg

which triggers the macOS Shortcut:

script-screenshot-app/iina-macos-shortcut.jpg

in any case, 3s later, I have the HTML snippet in my clipboard, ready to be pasted in my note, like so:

<img class="screenshot" src="https://ik.imagekit.io/vhucnsp9j1u/images/221003-125026-tesla-ai-day-2022.png" alt="221003-125026-tesla-ai-day-2022"/>

Python script used for reference:

'''
Complement script to the "Screenshot IINA" Apple Shortcuts. 
- copy HTML to clipboard
'''

import subprocess
from inspect import currentframe
import os

import time
start_time = time.time()

import sys
import pymsgbox

output_folder = '/Users/path/to/content/images'

### Functions

def write_to_clipboard(output):
    process = subprocess.Popen(
        'pbcopy', env={'LANG': 'en_US.UTF-8'}, stdin=subprocess.PIPE)
    process.communicate(output.encode('utf-8'))
    print(f"\nOUTPUT COPIED TO CLIPBOARD\n")

def process_iina_screenshot(filepath, v=False, test=False):

    image_path = filepath.replace(output_folder, '')[1:] # remove trailing /

    if image_path not in [None, 'None', '']:

        image_name = image_path.split('.')[0]
        if image_name.endswith('-'):
            image_name.replace('-', '')

        output = f"<img class=\"screenshot\" src=\"https://ik.imagekit.io/vhucnsp9j1u/images/{image_path}\" alt=\"{image_name}\"/>"  

        write_to_clipboard(output)

run_time = round((time.time() - start_time), 1)
print(f'finished in {run_time}s.\n')

if __name__ == '__main__':
    filepath = sys.argv[1]
    process_iina_screenshot(filepath)    

I'm happy with this new workflow 😁

First use of the workflow: !interests/tesla-ai-day-2022

14 Mar 2023

Issue with using IINA is that it has poor video playback control - jog & frame-by-frame - making screen capture difficult.

Switched the workflow to VLC today !apps/vlc , without code:

  • Assign the native "Snapshot" control in VLC (Cmd + Option + S by default) to a key on my ShuttlePro & save to a specific folder (in VLC settings) !home-office/contour-shuttle

script-screenshot-app/230314-contour-shuttle-settings-vlc.jpg

  • Hazel rule to rename & move any image file hitting that folder to my notes / images folder.

script-screenshot-app/230314-2251-hazel-vlc-settings.jpg

Works much better. But frame-by-frame is only possible forward (with E)? 🤔

15 Mar 2023

Backwards frame-by-frame is not possible in VLC 😕 !apps/vlc

Workaround:

  • use Very short backwards jump (Cmd + Ctrl + Right) of 3s
  • hold Next Frame (E) to go forward frame-by-frame

Updated Contour settings:

script-screenshot-app/230315-0721-contour-settings.jpg

create HTML snippet

Next step: script + Hazel logic to create the HTML snippet needed for my notes, ready to be pasted from clipboard.

Seems to be working with:

'''
Complement script to the "Generate HTML snippet for last image" Hazel rule. 
- get full URL from Hazel
- generate HTML snippet
- copy HTML to clipboard
'''

import subprocess
import sys


def generate_short_path(filepath, v=False, test=False):
    if filepath.startswith('/local/path/to/folder/images/') and '/logos/' not in filepath:
        filepath = filepath.replace('/local/path/to/folder/images/', '')
    return filepath

def generate_html_snippet(short_filepath, v=False, test=False):

    html_snippet = f"<img class=\"screenshot\" src=\"https://ik.imagekit.io/vhucnsp9j1u/images/{short_filepath}\" alt=\"{short_filepath}\"/>"  

    return html_snippet


def write_to_clipboard(output):
    process = subprocess.Popen(
        'pbcopy', env={'LANG': 'en_US.UTF-8'}, stdin=subprocess.PIPE)
    process.communicate(output.encode('utf-8'))
    print(f"\nOUTPUT COPIED TO CLIPBOARD\n")


def process(filepath, v=False, test=False):

    short_path = generate_short_path(filepath)

    output = generate_html_snippet(short_path)

    write_to_clipboard(output)


if __name__ == '__main__':
    filepath = sys.argv[1] # get filepath from Hazel
    process(filepath)       

In Hazel:

script-screenshot-app/230315-0754-hazel-settings.jpg

links

social