OS X

xrayspx's picture

DVD Ripper

Download the dvdrip bash script

This is the correct way (for me) to rip hundreds of DVDs. I still wish there was a global hash table of discs whereby we could automatically name individual files, but this does the job and I'll describe my overall workflow. Ripping TV shows is stupidly time consuming compared to audio CDs and I've done everything I can to reduce the time wastery involved. It's not perfect, but I can just feed disks through my machine all day then take an hour or so a week and rename everything I've done.

xrayspx's picture

Have some content

Music: 

It seems like I really don't write very much, but that's kind of a massive misconception. I don't write "much", but I had a bunch of blog entries that were at least 60% written and were missing like, screenshots or links or tags or I need to make new tags for things like XScreensaver and BSDs. Haiku. Shit lots of stuff.

xrayspx's picture

NoMachine NX Key Based Auth

Music: 

This will likely be updated.

I saw a request for some help in setting up key-based auth in NoMachine NX tonight just as I was going to bed and decided to do that instead.  I believe the request is for Mac-to-Mac, but for the moment I'm doing bi-directional Linux-to-Mac and Mac-to-Linux.  If I make any changes at all in how I set the Mac side up vs the Linux side I'll note them of course.

So here is the basic Linux client to Mac server.  In testing I set this up Linux-to-Linux.  The commands I used were exactly the same on the Linux and Mac servers.

Tags:
xrayspx's picture

Two Step Remote Assistance Tool

Music: 

My mom has a Mac, and occasionally something will fuck up in a way that is best fixed by me having some control over her machine.  I had one of those cases last week and it was embarrassing that there was no good way for me to get remote access.  Google Meet doesn't cut it, but there's a whole other Chrome Remote Desktop app, but that was a lot of hoops to install and gave up any hope of walking my mother through the install process.

xrayspx's picture

Linux Needs To Be Ashamed

Music: 

I'm a 25 year Linux user, 22 as my primary desktop. I like pain, and that's OK. But do I consider myself any kind of "expert"? No.

xrayspx's picture

Lattice of Convenience - MP3 Playlists

Music: 

Underworld - Kittens

Hopefully everyone can live in the future someday.

We do a lot with MP3 playlists. I run Airsonic for streaming around the house and in the car, and we have a playlist-based FM transmitter setup, etc. So I have scripts which run every night and generate playlists based on star ratings and other things (GET THE LED OUT ANYONE?).

Previously what I've done is dump the contents of a bunch of Smart Playlists in Clementine to a file and use those files to generate the randomized 200 track daily playlists. The downside to that is that every time I add music or change star ratings, I'd have to refresh these "base" files like some kind of animal. I had base playlists for "3+ stars", "4+ Stars" and "5 Stars", among others.

Today I decided to fix all that. Clementine uses a SQLite3 database, so now I'm just querying it instead, and it seems to be working well. For example, my "5 star" playlist in Clementine results in 10800 or so tracks. The same one built from the DB ends up with a couple hundred more tracks, but is pretty close. I'm not entirely sure what the difference there is just yet, but "close enough". What it looks like to me is I probably need to enable Samba case sensitivity.

The DB records ratings as decimal numbers from 0.0 (Zero stars) through 1.0 (5 Stars). So to build a "4-Star +" playlist, searching for rating >= "0.8", you get ratings like this:

1
1.10000002384186
0.800000011920929
0.800000011920929
0.800000011920929
1.10000002384186
1.10000002384186
0.800000011920929
0.800000011920929
1
1
1
1

! Caveat: Prior to Clementine 1.4.0rc1-533-gf4e70face there was a bug where it was possible to give a song a higher than 5 star rating (higher than 1.0 in the DB) as you can see above, so know that if you have Clementine from the repositories, it's likely you have that bug. For instance in the UI, if you want to show all 5 star songs, use "Rating is Greater Than 4.5 Stars" rather than "Rating is Equal to 5 Stars".

Now I can just have a cron job to copy the master Clementine DB once a day to my server and drop it in next to the playlist generation scripts.

The downside to all this is speed. When using the Clementine-Generated base playlists, I could be sure all the files actually exist on disk. However while Clementine will only show you files that exist in the UI, it doesn't seem to do a very good job of cleaning the database of stale files which no longer exist. So if you move or rename files, the old DB entries stick around unless you purge it completely and start over from scratch. That means I have to test every single file as I add it to the playlist, which takes time. It takes about 5-8 seconds to generate my 200 track 5-Star M3U file.

The 5-Star.sh script is below if you'd like to play along at home:

  


#!/bin/bash

rm /Volumes/Filestore/CDs/playlists/5\ Stars.m3u

i=1

while [ $i -le 200 ]
do
 file=$(sqlite3 ./clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

 ### Clementine data encodes special characters and accent marks and stuff so I'm using
 ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649
 
 data=$(urldecode.sh "$file")
 if [ -f "$data" ]
 then
  ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
  ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

  escaped=$(echo "$data" | sed 's/\[/\\[/g')
  #echo "$escaped"

  ### Avoid duplicates
  match=$(grep -i "$escaped" /Volumes/Filestore/CDs/playlists/5\ Stars.m3u)
  if [ -z "$match" ]
  then
   echo "$data" >> /Volumes/Filestore/CDs/playlists/5\ Stars.m3u
   ((i++))
  fi
 fi
done

For the 3+ and 4+ lists, I repeat this main block, but instead each rating dumps into a text file that I randomize into an .m3u at the end. So for the 3-Star + script below, I collect 130 5-star tracks, 45 4-star, and 25 3-star, push them out to a temp file and then cat temp.m3u | sort -R > "./3 Star +.m3u". I could do all this by creating a new table in the database and stuffing tracks into that, but this was faster for me to write and it works well enough:


#!/bin/bash

rm /Volumes/Filestore/CDs/playlists/3\ Stars\ +.m3u

i=1

while [ $i -le 130 ]
do
 file=$(sqlite3 ./clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

 ### Clementine data encodes special characters and accent marks and stuff so I'm using
 ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649
 
 data=$(urldecode.sh "$file")
 if [ -f "$data" ]
 then
  ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
  ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

  escaped=$(echo "$data" | sed 's/\[/\\[/g')
  #echo "$escaped"

  ### Avoid duplicates
  match=$(grep -i "$escaped" ./3-star-tmp.m3u)
  if [ -z "$match" ]
  then
   echo "$data" >> ./3-star-tmp.m3u
   ((i++))
  fi
 fi
done

i=1

while [ $i -le 45 ]
do
  file=$(sqlite3 ./clementine.db "select filename from songs where rating >= "0.8" and rating

  ### Clementine data encodes special characters and accent marks and stuff so I'm using
  ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

  data=$(urldecode.sh "$file")
  if [ -f "$data" ]
  then
   ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
   ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

   escaped=$(echo "$data" | sed 's/\[/\\[/g')
   #echo "$escaped"

   ### Avoid duplicates
   match=$(grep -i "$escaped" ./3-star-tmp.m3u)
   if [ -z "$match" ]
   then
    echo "$data" >> ./3-star-tmp.m3u
    ((i++))
   fi
  fi
done

i=1

while [ $i -le 25 ]
do
  file=$(sqlite3 ./clementine.db "select filename from songs where rating >= "0.6" and rating

  ### Clementine data encodes special characters and accent marks and stuff so I'm using
  ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

  data=$(urldecode.sh "$file")
  if [ -f "$data" ]
  then
   ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
   ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

   escaped=$(echo "$data" | sed 's/\[/\\[/g')
   #echo "$escaped"

   ### Avoid duplicates
   match=$(grep -i "$escaped" ./3-star-tmp.m3u)
   if [ -z "$match" ]
   then
    echo "$data" >> ./3-star-tmp.m3u
    ((i++))
   fi
  fi
done

cat ./3-star-tmp.m3u | sort -R > /Volumes/Filestore/CDs/playlists/3\ Stars\ +.m3u

rm ./3-star-tmp.m3u

xrayspx's picture

Caching Password passer

Similar to the RDP Launcher, I occasionally need to grab passwords that I use all the time from KeePassXC to paste into various forms or prompts. Basically anything I use more than once per day, I have defined in this script for quick access. I don't want to be able to remember these, and I also don't want to have to interact with the password manager UI if I'm in a shell.

xrayspx's picture

DVD Ripping

Music: 

The Wipeouters - Ravin' Surf

Another note for myself for later, and boy this is dumb.

RDPLauncher

TL;DR: Here's the Link:
RDPLauncher

I use RDP a lot and had some scripts to let me launch lots of RDP sessions without having to enter my random-generated passwords over and over. I wasn't happy with how I was handling those passwords so I've made it more secure using gpg and KeePassXC. Last night I made it compatible with Windows and MSTSC which will be uploaded here shortly once it's cleaned up a bit.

Basically I'll click a shortcut for whatever host, which runs my launcher. I get prompted for my GPG passphrase, which reads from an encrypted file containing my KeePassXC passphrase, which is then used to retrieve the user password for launching the RDP session.

Gpg-agent uses a cache-TTL to "hold the door open" for 10 minutes by default, so I can launch a bunch of sessions and only type my passphrase once.

Requirements:

- gpg client and running gpg-agent (gpg4win, etc) with a private key set up, etc.
- cygwin if you're running Windows
- KeePassXC (or some other key-store that has a command-line interface
to query the database. In the beginning I was just using the gpg file
with user/password pairs, so that works too)

The tool has a few neat features:

- If run from the command line with no arguments, it will prompt for user/pass/host/domain, good for one-off sessions to machines I won't log into much. That's great since I spend all my time in terminal windows and this stops me having to go back and forth to the mouse and keyboard while entering credentials.

- If launched with -b, it prompts you for information for a one-off connection, but will also build a new shortcut launcher from a template. So like for the first connection to a machine you know you're going to use a lot. (Linux/Mac only)

- Automatically tunnel sessions over ssh. This means I can launch RDP sessions on my Mac and they'll seamlessly proxy through my work laptop to the VPN.

For tunneling, I am taking an arbitrary range of 200 ports and incrementing them based on what's currently listening. If there's already a process listening on port 6201, then try 6202 etc until there's an open one. So I can easily open 20-30 ssh tunneled sessions each with its own ssh process which will close down when the RDP window closes. 200 is "probably overkill", which means it might just be barely enough in the real world.

The launcher shortcut mechanics are a bit different on my Linux and Mac machines so I split the -b script builder piece out based on OS. On Linux, I use KDE/Plasma, and so I generate these as KDE desktop files which look like this:

#!/usr/bin/env xdg-open
[Desktop Entry]
Comment[en_US]=
Comment=
Exec=/home/xrayspx/bin/rdplauncher.sh -h it-host.xrayspx.com -d xdomainx -u xrayspx
GenericName[en_US]=
GenericName=host.xrayspx.com
Icon=remmina
MimeType=
Name[en_US]=
Name=host.xrayspx.com
Path=
StartupNotify=true
Terminal=false
TerminalOptions=
Type=Application
X-DBUS-ServiceName=host.xrayspx.com
X-DBUS-StartupType=
X-KDE-SubstituteUID=false
X-KDE-Username=

On the Mac side, I use shell scripts with the extension .rdp (which conflicts with Microsoft's client, but I don't care since I never use their client anyway). Those just launch using Terminal, so it does pop a terminal for a fraction of a second, but I really don't have a problem with that. To get the Terminal window to close (and I do associate these files with Terminal.app specifically rather than iTerm2), open Terminal.app, go to the Terminal menu -> Settings -> Profiles (tab) -> "Basic" or whatever profile is your default -> Shell (tab). Choose what action to take when the shell exits. I have it set to "Close if the shell exited cleanly" and "ask before closing" set to "only if there are processes other than the login shell..."

The launcher for that looks like:

#! /bin/bash
rdplauncher.sh -h host.xrayspx.com -d xdomainx -u xrayspx &

I generate those from the KDE .desktop files with a command like this:

for host in $(ls | grep "\.desktop$" | awk -F ".desktop" '{print $1}'); do cmd=$(grep Exec $host.desktop | awk -F "xrayspx/bin/" '{print $2}'); echo "\!#/bin/zsh" >> $host.rdp; echo "$cmd &" >> $host.rdp; done

That creates .rdp files in the same directory as the .desktop files, so now they can be moved around, have chmod set, etc.

If I call it with AppleScript or Automator instead of a bash script as above, none of the password retrieval process works. I think it short circuits and sends the output back to the AppleScript rather than the bash script which ran the command. If I can get that working that would be ideal.

The mechanics on Windows are similar to the Mac method. a .bat file which launches the bash script via Cygwin:

C:\cygwin64\bin\mintty.exe -w hide -e /bin/bash -l -c '/home/user/bin/rdplauncher.sh -h host -u username -d domain'

On Windows at least the Cygwin window it creates is hidden from the user, so that's nice.

xrayspx's picture

Rippin' DVDs

Music: 

Dana Carvey - Choppin' Broccoli

Today in Lattice of Convenience news, here's how to rip DVDs.

I barely understand the mencoder command that is the backbone of this thing, and there are many better ways to do lots of the stuff in this script, in fact I know several of those better ways, and looking at it fresh, I see some redundant stuff that cancels out other stuff. But it runs, and I use it, so here goes.

Ripping DVDs isn't fun, the disk labels are iffy at best, even within a single box set you might go from the Gold Standard "TV Show - S1D1" to "DVD_VIDEO" as a disk label. So it can get kind of ugly. To mitigate that I create an output folder based on the DVD disk label + a timestamp. If you get a run of disks with the same name, at least they're not overwriting each others files because the timestamp will shift. I currently have a dvdrip-output directory with the following DVDs in it:

...
DVD_VIDEO-090720202337
DVD_VIDEO-090820201025
DVD_VIDEO-090820201027
DVD_VIDEO-090820201142
I_LOVE_LUCY_S2_D1-090520202354
I_LOVE_LUCY_S2_D3-090620201047
LUCY_S1D1-090520201043
LUCY_S1D2-090520201043
LUCY_S1D3-090520201359
...

Those are all from the same box set. So that's 3 naming conventions from one series. To be fair I think that while it's the same company producing them they probably came as separate "season" boxes rather than one big set. Still. Come on. Jesus.

Another big gotcha I've hit, again mainly with TV series box sets, a single show might exist on the disk as many as THREE times. Once as a "standalone episode", once as "episode with commentary track" and once as part of a massive concatenated file of all the episodes on that disk. In the case of the commentary track, that audio seems to be separate, so the actual episode rips to exactly the same filesize, the commentary track seems not to be something I have access to, so you just get two identical files at the end.

So as you're ripping, that's going to triple the rip time.

The way I'm trying to fix that is to rip the first 30 seconds of every Title on the disk, then do a SHA sum on those ripped sample files. As a Title rips, when it's done I'll drop its clip checksum into a "rippedchecksums" file. The next TItle starts the first thing it does is check to see if its checksum has already been ripped. If it has, skip it. It seems to catch 100% of repeated Titles, and probably 70% of the "Big Concatenated File" cases will match the sum for Title 1. Saves a shitload of time.

In this case, Title 1 is a standalone episode, and Title 21 is the Big Concatenated File of all the episodes on the disk. Title 21 will be skipped. Since I get about 70 or 80 FPS on my Mac Pro, that probably saved 90 minutes of rip time or so with 3 hours of video on the disk:

763b6035c4bf239b4425fb8f484018387574baca /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/1-sample.avi
59cca1b18759647e13e3e1b6a4facace0520fc06 /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/10-sample.avi
125add4181b9dc6eee57c32c07568765b8e4483b /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/11-sample.avi
4daae35d014032964fe57e70e2cc3450f7dac4e5 /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/12-sample.avi
a942f31a9ee42c5839772f733b2c666195397ad5 /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/13-sample.avi
8c9473a940a9bc685d84e0ac29c66f53efa6667d /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/14-sample.avi
29d2200d8c46ac11417119b4b7179e4b526d99cf /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/15-sample.avi
466860b79bba6d132fcc97d6dc7c0c3a20dd771c /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/16-sample.avi
f4ae11cca0752956c4d6025a8760a260a59fe79b /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/17-sample.avi
00753d529f4bbf4081f647056cf44db7c630c198 /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/18-sample.avi
b7f9c9087fed6b00d22de5033c153f9ffb3cd3b1 /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/19-sample.avi
14efcb6164f1424b894cc28200ab621ec805ecd0 /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/2-sample.avi
6c411c8869f1e6bc9a6ec298ba9b6a5c9eefc9ae /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/20-sample.avi
763b6035c4bf239b4425fb8f484018387574baca /Volumes/Filestore/dvdrip-output/DVD_VIDEO-090720202337/21-sample.avi

At the end of it, I still end up with just a directory full of files labeled 1 through whatever.avi. I have to take a few seconds per file to get it to "TV Show - S01E01.avi". But from there FileBot can mass-rename them with episode titles.

So here's the full ugliness. You'll want to adjust all the paths. I should have made variables, but I don't care, I maybe have 3 or 4 ripping trays running at a time on various machines, so I don't mind just changing the paths for each host. Works on OSX and Linux, and probably Windows with Cygwin, but I don't care about Windows so I'm not going to test it.


#! /bin/bash

timestamp=`date +%m%d%Y%H%M`

id=$(drutil status |grep -m1 -o '/dev/disk[0-9]*')

if [ -z "$id" ]; then
echo "No Media Inserted"
else
name=`df | grep "$id" |grep -o /Volumes.* | awk -F "Volumes\/" '{print $2}' | sed 's/ /_/g'`

fi
name=`df | grep "$id" |grep -o /Volumes.* | awk -F "Volumes\/" '{print $2}' | sed 's/ /_/g'`
echo $name
dir="$name-$timestamp"
mkdir /Volumes/Filestore/dvdrip-output/$dir

maxtitle=`/Applications/mencoder dvd://100 -o bob | grep "titles on this DVD" | awk '{print $3}'`

for title in {1..100}
do
if [ $title -le $maxtitle ]
then
/Applications/mencoder dvd://$title -alang en -ovc lavc -lavcopts vcodec=mpeg4:vhq:vbitrate="1200" -vf scale -zoom -xy 720 -oac mp3lame -lameopts br=128 -endpos 30 -o /Volumes/Filestore/dvdrip-output/$dir/$title-sample.avi
shasum /Volumes/Filestore/dvdrip-output/$dir/$title-sample.avi > /Volumes/Filestore/dvdrip-output/$dir/$title-checksum
touch /Volumes/Filestore/dvdrip-output/$dir/rippedchecksums.txt
fi
done

cat /Volumes/Filestore/dvdrip-output/$dir/*checksum >> /Volumes/Filestore/dvdrip-output/$dir/allchecksums.txt

for title in {1..100}
do
if [ $title -gt $maxtitle ]
then
chmod -R 775 /Volumes/Filestore/dvdrip-output/$dir
sleep 3
drutil tray eject
exit 0
fi
sum=`cat /Volumes/Filestore/dvdrip-output/$dir/$title-checksum | awk '{print $1}'`
match=`grep $sum /Volumes/Filestore/dvdrip-output/$dir/rippedchecksums.txt`
if [ -z $match ]
then
echo "CURRENTLY RIPPING TITLE #$title"
/Applications/mencoder dvd://$title -alang en -ovc lavc -lavcopts vcodec=mpeg4:vhq:vbitrate="1200" -vf scale -zoom -xy 720 -oac mp3lame -lameopts br=128 -o /Volumes/Filestore/dvdrip-output/$dir/$title.avi
echo $sum >> /Volumes/Filestore/dvdrip-output/$dir/rippedchecksums.txt
rm /Volumes/Filestore/dvdrip-output/$dir/$title-checksum
rm /Volumes/Filestore/dvdrip-output/$dir/$title-sample.avi
fi
done
chmod -R 775 /Volumes/Filestore/dvdrip-output/$dir

Pages

Subscribe to RSS - OS X