It seems like I really don't write very much, but that's kind of a massive misconception. I don't write "much", but I had a bunch of blog entries that were at least 60% written and were missing like, screenshots or links or tags or I need to make new tags for things like XScreensaver and BSDs. Haiku. Shit lots of stuff.

I had a request to share some playlist management stuff so I thought I should explain myself. I've got a significant CD collection, and a somewhat-significant collection of TV shows. This is fine on its own, but lots of media is pretty worthless without well curated playlists that you really don't have to think about. So I built Spotify, MTV and Syndicated TV.

* NOTE: If you have a better way to do any of this let me know and I'll fix it. I particularly have the sense, which is not backed up by my testing, that "sort -R" isn't great.

Music's easier so we'll start there. I use Strawberry to manage my music. This was all running under Clementine and aside from some DB schema changes, the scripts are portable between them.

Until relatively recently I was never a big fan of "star" or "heart" ratings, but Clementine/Strawberry will store this metadata in the MP3 itself so I should be able to quickly recover if I lose my music database. In the app I have a few Smart Playlists like 3-Stars, 3 Stars + (This is 3, 4 and 5 star tracks), 4-Stars, 4-Star + and 5 Stars. To use 4 Star as an example, the rules look like this:

Match every search term (AND)
Rating - Greater than - 3.5 Stars
Rathing - Less than - 5 Stars
Ratin - Not Equals - 5 Stars
Length - Greater Than - 8 Seconds

That results in a playlist of 8423 songs with ratings between 4 and 4.99 stars. There was a bug in Clementine which I got fixed where ratings could exceed 5, so I'm a little careful to deal with weirdo cases, but it's pretty simple. I also have a bunch of manually selected playlists, so like an '80s one, '90s, and "Barn Radio". Barn Radio is our catch-all for the ubiquitous music we heard from the late '70s through late '80s. For Natalie that was largely with her dad in the dairy barn, for me it was the music of my 2 hours on the bus every day.

Anyway, I have all these .m3us stored in a folder along with my MP3s called "playlists_base". These are used by a nightly playlist generator that pulls ~200 tracks and makes daily playlists running 8 or 10 hours each. The reason for this is that streaming software such as Airsonic-Advanced kind of chokes on massive playlists. It could be Airsonic itself, it could be populating the mobile client, I don't really know or care, other than to say it works great with list sizes under about 1000 tracks or so, so I keep them shorter.

The x-Star playlists are all built from the database like this 4 Star + playlist below. You can see it do a couple of different Star Rating DB queries, dump out the tracks to $playlist_tmp.m3u, then cat that file and do a random sort to generate the final version. It's pretty easy to adjust the mix based on ratings, so if I wanted to weight high-rated tracks I could do that by adjusting how many tracks of the 200 are returned by each search:


#!/bin/bash

rm /Volumes/Filestore/CDs/playlists/4\ Stars\ +.m3u

i=1

while [ $i -le 100 ]
do

### Switching from Clementine to Strawberry ###
#       file=$(sqlite3 /var/tmp/clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')
        file=$(sqlite3 /var/tmp/strawberry.db "select url from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

        ### Clementine data encodes special characters and accent marks and stuff so I'm using
        ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

        data=$(/home/xrayspx/bin/urldecode.sh "$file")
        if [ -f "$data" ]
        then
                ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
                ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

                escaped=$(echo "$data" | sed 's/\[/\\[/g')
                #echo "$escaped"

                ### Avoid duplicates
                match=$(grep -i "$escaped" /var/tmp/4-star-tmp.m3u)
                if [ -z "$match" ]
                then
                        echo "$data" >> /var/tmp/4-star-tmp.m3u
                        ((i++))
                fi
        fi
done

i=1

while [ $i -le 100 ]
do
### Switching from Clementine to Strawberry ###
#        file=$(sqlite3 /var/tmp/clementine.db "select filename from songs where rating >= "0.8" and rating          file=$(sqlite3 /var/tmp/strawberry.db "select url from songs where rating >= "0.8" and rating 

        ### Clementine data encodes special characters and accent marks and stuff so I'm using
        ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

        data=$(/home/xrayspx/bin/urldecode.sh "$file")
        if [ -f "$data" ]
        then
                ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
                ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

                escaped=$(echo "$data" | sed 's/\[/\\[/g')
                #echo "$escaped"

                ### Avoid duplicates
                match=$(grep -i "$escaped" /var/tmp/4-star-tmp.m3u)
                if [ -z "$match" ]
                then
                        echo "$data" >> /var/tmp/4-star-tmp.m3u
                        ((i++))
                fi
        fi
done

cat /var/tmp/4-star-tmp.m3u | sort -R > /Volumes/Filestore/CDs/playlists/4\ Stars\ +.m3u

rm /var/tmp/4-star-tmp.m3u

Those Star Rating lists are called at the beginning of my overall static playlist script, but the Barn playlist and other manually selected ones are built from the "playlists_base" directory. I basically just edit those .m3us in place with Strawberry as we add CDs. They just the files, do a random sort and pull the top 200. This will use any .m3u in .../playlists_base/ and make a daily file from it:


#!/bin/bash

#scp xrayspx@pro:~/.config/Clementine/clementine.db /var/tmp/

### Switching between Clementine and Strawberry ###
#cp /Volumes/Filestore/CDs/playlists_base/clementine.db /var/tmp/

cp /Volumes/Filestore/CDs/playlists_base/strawberry.db /var/tmp/

/home/xrayspx/bin/3-star-playlist.sh
/home/xrayspx/bin/4-star-playlist.sh
/home/xrayspx/bin/5-star-playlist.sh
/home/xrayspx/bin/get-the-led-out.sh

ls /Volumes/Filestore/CDs/playlists_base/*.m3u > /Volumes/Filestore/CDs/playlists_base/m3us.txt

while IFS= read -r file
do

        filename=$(echo $file | awk -F "/Volumes/Filestore/CDs/playlists_base/" '{print $2}')

        echo Filename: $file

        rm "$file.full"
        rm "$file.scratch"
        rm "/Volumes/Filestore/CDs/playlists/$filename"

        ###Testing a change since Strawberry creates playlists without EXTINF lines ###
#        array=`grep EXTINF "$file" | sort | uniq`
        array=`grep -v EXTINF "$file" | sort | uniq`

        printf '%s\n' "${array[@]}" | sort -R > "$file.full"
        head -n 200 "$file.full" > "/Volumes/Filestore/CDs/playlists_base/$filename.scratch"

        n=0
        while IFS= read -r extinfo
        do
#       echo $extinfo
                term=`echo $extinfo` # | cut -d "," -f 2-`
#       echo $term

 ###Testing a change since Strawberry creates playlists without EXTINF lines ###
 # grep -A 1 -m 1 "$term" "$file" >> "/Volumes/Filestore/CDs/playlists/$filename"

        grep -m 1 "$term" "$file" >> "/Volumes/Filestore/CDs/playlists/$filename"
        done 

        rm "$file.full"
        rm "$file.scratch"

done 

rm /var/tmp/clementine.db
rm /var/tmp/strawberry.db

For TV shows it's a bit more complicated. I've got individual scripts for things like Sitcoms, Saturday Morning Cartoons, Buddy-Cop shows, Nick-at-Nite, etc. Each script uses a text file which just lists the relative path to the directories I want to randomize. I just read in that text file then scan each directory and build an array that again I sort -R and dump in an m3u. You'll see a couple of my conventions here, like the "dvd_extras" folders I use for any extras that I want to keep but don't want to have show up in the mix, as well as a bunch of other crap I grep out.

This script references "./.sitcoms.txt", which looks like this:


./Archer (2009)
./30 Rock
./Absolutely Fabulous
./Alexei Sayle's Stuff



#! /bin/bash

array=$(
while read line
do
        find "$line" -type f;
done < .sitcoms.txt
)

printf '%s\n' "${array[@]}" | sort -R | grep -v -w "batch" | grep -v dvd_extras | grep -v "./$" | grep -v "\.m3u" | grep -v -i ds_store |
 grep -v "\.nzb" | grep -v "\.nfo" | grep -v "\.sub" | grep -v "\.sfv" | grep -v "\.srt" | grep -v -i "\.ifo" | grep -v -i "\.idx" |
 sed 's/^/..\//' > ./1\ -\ Playlists/Sitcoms.m3u

This dumps out to a folder called "1 - Playlists" inside my TV Shows directory, just so it shows up first. There's a folder in there for Blocks as well, in which I create blocks of 10 random episodes of a bunch of shows. This is built to replicate like TBS/TNT/USA in the evening where you just sit and watch a block of whatever is on. In practice I do this wrong and tend to be too picky about these and just watch blocks until I've worked my way through a whole series and wind up tired of it forever.

One thing I do for things like Nick at Nite and overall Sitcom lists and stuff is that I mix in commercials. I don't do this very well though, I just treat my directory of commercials like any other TV show. I'd rather do "pull a TV show, toss in two commercials, repeat", but I'm not there yet I guess.

The last type of lists I build are for music videos. I break this into a few different playlists, one overall catchall that pulls in all videos, a playlist for MTV 120 Minutes, and one for "Arcade / Pizzeria" music. Basically the ubiquitous music you'd hear in a pizza shop or arcade in the '80s or '90s. I do the same commercial thing here as well.

Example:


#! /bin/bash

array=`find ../120\ Minutes -type f;
find ../../../Commercials -type f`

printf '%s\n' "${array[@]}" | sort -R | grep -v dvd_extras | grep -v "./$" | grep -v "ERRORS$" | grep -v "\.sh" | grep -v "\.m3u" |
 grep -v -i ds_store | grep -v ".nzb" | grep -v ".srt" > 120\ Minutes.m3u

Hopefully everyone can live in the future someday.

We do a lot with MP3 playlists. I run Airsonic for streaming around the house and in the car, and we have a playlist-based FM transmitter setup, etc. So I have scripts which run every night and generate playlists based on star ratings and other things (GET THE LED OUT ANYONE?).

Previously what I've done is dump the contents of a bunch of Smart Playlists in Clementine to a file and use those files to generate the randomized 200 track daily playlists. The downside to that is that every time I add music or change star ratings, I'd have to refresh these "base" files like some kind of animal. I had base playlists for "3+ stars", "4+ Stars" and "5 Stars", among others.

Today I decided to fix all that. Clementine uses a SQLite3 database, so now I'm just querying it instead, and it seems to be working well. For example, my "5 star" playlist in Clementine results in 10800 or so tracks. The same one built from the DB ends up with a couple hundred more tracks, but is pretty close. I'm not entirely sure what the difference there is just yet, but "close enough". What it looks like to me is I probably need to enable Samba case sensitivity.

The DB records ratings as decimal numbers from 0.0 (Zero stars) through 1.0 (5 Stars). So to build a "4-Star +" playlist, searching for rating >= "0.8", you get ratings like this:

1
1.10000002384186
0.800000011920929
0.800000011920929
0.800000011920929
1.10000002384186
1.10000002384186
0.800000011920929
0.800000011920929
1
1
1
1

! Caveat: Prior to Clementine 1.4.0rc1-533-gf4e70face there was a bug where it was possible to give a song a higher than 5 star rating (higher than 1.0 in the DB) as you can see above, so know that if you have Clementine from the repositories, it's likely you have that bug. For instance in the UI, if you want to show all 5 star songs, use "Rating is Greater Than 4.5 Stars" rather than "Rating is Equal to 5 Stars".

Now I can just have a cron job to copy the master Clementine DB once a day to my server and drop it in next to the playlist generation scripts.

The downside to all this is speed. When using the Clementine-Generated base playlists, I could be sure all the files actually exist on disk. However while Clementine will only show you files that exist in the UI, it doesn't seem to do a very good job of cleaning the database of stale files which no longer exist. So if you move or rename files, the old DB entries stick around unless you purge it completely and start over from scratch. That means I have to test every single file as I add it to the playlist, which takes time. It takes about 5-8 seconds to generate my 200 track 5-Star M3U file.

The 5-Star.sh script is below if you'd like to play along at home:


#!/bin/bash

rm /Volumes/Filestore/CDs/playlists/5\ Stars.m3u

i=1

while [ $i -le 200 ]
do
 file=$(sqlite3 ./clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

 ### Clementine data encodes special characters and accent marks and stuff so I'm using
 ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649
 
 data=$(urldecode.sh "$file")
 if [ -f "$data" ]
 then
  ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
  ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

  escaped=$(echo "$data" | sed 's/\[/\\[/g')
  #echo "$escaped"

  ### Avoid duplicates
  match=$(grep -i "$escaped" /Volumes/Filestore/CDs/playlists/5\ Stars.m3u)
  if [ -z "$match" ]
  then
   echo "$data" >> /Volumes/Filestore/CDs/playlists/5\ Stars.m3u
   ((i++))
  fi
 fi
done

For the 3+ and 4+ lists, I repeat this main block, but instead each rating dumps into a text file that I randomize into an .m3u at the end. So for the 3-Star + script below, I collect 130 5-star tracks, 45 4-star, and 25 3-star, push them out to a temp file and then cat temp.m3u | sort -R > "./3 Star +.m3u". I could do all this by creating a new table in the database and stuffing tracks into that, but this was faster for me to write and it works well enough:


#!/bin/bash

rm /Volumes/Filestore/CDs/playlists/3\ Stars\ +.m3u

i=1

while [ $i -le 130 ]
do
 file=$(sqlite3 ./clementine.db "select filename from songs where rating > "0.9" order by random() limit 1;" | awk -F "file://" '{print $2}')

 ### Clementine data encodes special characters and accent marks and stuff so I'm using
 ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649
 
 data=$(urldecode.sh "$file")
 if [ -f "$data" ]
 then
  ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
  ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

  escaped=$(echo "$data" | sed 's/\[/\\[/g')
  #echo "$escaped"

  ### Avoid duplicates
  match=$(grep -i "$escaped" ./3-star-tmp.m3u)
  if [ -z "$match" ]
  then
   echo "$data" >> ./3-star-tmp.m3u
   ((i++))
  fi
 fi
done

i=1

while [ $i -le 45 ]
do
  file=$(sqlite3 ./clementine.db "select filename from songs where rating >= "0.8" and rating

  ### Clementine data encodes special characters and accent marks and stuff so I'm using
  ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

  data=$(urldecode.sh "$file")
  if [ -f "$data" ]
  then
   ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
   ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

   escaped=$(echo "$data" | sed 's/\[/\\[/g')
   #echo "$escaped"

   ### Avoid duplicates
   match=$(grep -i "$escaped" ./3-star-tmp.m3u)
   if [ -z "$match" ]
   then
    echo "$data" >> ./3-star-tmp.m3u
    ((i++))
   fi
  fi
done

i=1

while [ $i -le 25 ]
do
  file=$(sqlite3 ./clementine.db "select filename from songs where rating >= "0.6" and rating

  ### Clementine data encodes special characters and accent marks and stuff so I'm using
  ### Joel Parker Henderson's urldecode.sh to undo that: https://gist.github.com/cdown/1163649

  data=$(urldecode.sh "$file")
  if [ -f "$data" ]
  then
   ### Have to escape leading brackets because grep treated it as a range and would allow duplicates ###
   ### Can't do that in "data" because \[ isn't in the filename so they'll fail ###

   escaped=$(echo "$data" | sed 's/\[/\\[/g')
   #echo "$escaped"

   ### Avoid duplicates
   match=$(grep -i "$escaped" ./3-star-tmp.m3u)
   if [ -z "$match" ]
   then
    echo "$data" >> ./3-star-tmp.m3u
    ((i++))
   fi
  fi
done

cat ./3-star-tmp.m3u | sort -R > /Volumes/Filestore/CDs/playlists/3\ Stars\ +.m3u

rm ./3-star-tmp.m3u

Anyone have any Deep Thoughts about how videos should be categorized? If not, skip it, this is really that boring.

--

Let's say for arguments sake that I'm building a playlist of
videos from 120 minutes (Like say from this comprehensive list right
here.

I've already decided that any band that gets one of their songs on 120 Minutes one time gets all of their songs in this folder. Because I don't want to have 3 different places where I can find songs of one band. It gets unruly. The only exception to this is the "Arcade Pizza" folder. These are songs that ubiquitous on the radio when I was a kid, especially in arcades and pizzerias of the '80s and '90s. For that case I have /videos/Arcade Pizza, as well as /videos/120 Minutes/Arcade Pizza.

Question is, should I only put stuff that appeared on the actual show, or should I put bands that /should/ have been on 120 minutes, but weren't, because MTV could show neither the full name of the band nor the full name of the song involved?

Or what if they're too new, like this video philosophically belongs to 120 Minutes, but it's only a year and a half old:

I think they should go in, but I'm holding off. Teddybears would have been HEAVY ROTATION on 120 minutes if they'd existed then.

Should I kick Evan Dando out because he spoiled my Juliana? These are questions that require fucking answers.

I'm nearing 3000 music vids now, so these things are starting to become problems I have to think about. I need to nip this shit in the bud before I have 20,000 videos and no damn plan at all.

The Bonus Question is: Do I change the name of the Youtube video to fit a rational style, or leave it alone? For instance:

I Was A Teenage Zombie (2016) [heHh9EIlAbw].mp4

Should be renamed to:

The Fleshtones - I Was A Teenage Zombie (2016) [heHh9EIlAbw].mp4

The "[heHh9EIlAbw]" is the only actually important part of that filename anyway, since that's the video ID on Youtube, so it'll be youtube.com?v=heHh9EIlAbw. That is there for pattern matching, so I think that makes it OK to rename shit.

Right?

A couple of years ago, Natalie and I canceled cable since we found it had literally been a year since we watched anything live on TV. I've built a pretty good "lattice of convenience" to store a media library of "Crap we like" and conveniently stream it pretty much anywhere.

Over the years, we've collected maybe 3000 CDs and a several hundred DVDs, including many box sets of TV series we like. I feel like we spent a TON on CDs when we were younger that most people didn't.