Rely on a DB table for artwork scans to prevent repeated disk access

[michaelherger]

When we crawl the file system for music files, keep a list of artwork files as well. This way we don't have to repeatedly read full folders to find potential artwork.

• add scanned_pics table in the scanner while looking for audio files
• when looking for file based artwork query this table instead of reading the file system
• look for an exact template match in the music file's folder first (if defined), then cover, album, folder, thumb
• check the template in the artwork folder (if defined) second
• last check any artwork file in the music file's folder

@darrell-k I took inspiration from cffd307 (#1536) to further look into using a db table to store artwork information. Thanks for that work!

At this point (without other modifications to the scanner) I can't see any performance improvement or decrease. It's almost identical in my current test environment (much faster NAS than last week's tests), with none being consistently faster or slower. Most likely this is because there's none of that repeated artwork lookup like with the changes for box sets.

I believe the DB table approach might prove to be super helpful for other use cases, too, eg. contributor picture lookups. But I haven't gone there yet.

I'll use this branch to further look into the box set challenge. I don't know yet where exactly that will best be implemented...

Signed-off-by: Michael Herger michael@herger.net

[darrell-k]

@michaelherger Makes sense to separate this out from the box sets work.

It would be good if @mikeysas updated #1430 with the additional requirements as he offered yesterday in his #1536 comment. Then we can take that issue as the statement of box set requirements.

Most likely this is because there's none of that repeated artwork lookup like with the changes for box sets.

I wouldn't say there were repeated lookups in #1536 (at least there shouldn't be!), but we do need to look for candidates for each folder and disc number within an album to potentially update the albums/tracks tables with new/different image urls.

The logic for the box sets enhancement is, I believe, all there in #1536. I'll try to merge these changes with that. Unless you want to take over from here?

Some questions on this PR:

1. I was wondering if we need to also change AIO.pm?
2. Should we DELETE FROM scanned_pics... in Slim/Utils/Scanner/Local.pm as I had in Alternative artwork/box set approach #1536 ?
3. Also, should we be checking the artwork when rescanning a single album?

[darrell-k]

I thought that here we would want to recognise case differences?

[michaelherger]

I copied this 1:1 from schema_11_up.sql. It's what we've had before.

[darrell-k]

Yes, I did the same to start with, but I don't think the issue mentioned in the comment applies. And on a non-Windows system we could potentially have Cover.jpg and cover.jpg in an album folder. Maybe it wouldn't matter in that case which one we picked up , but I don't think case sensitivity would do any harm.

[mherger]

...and I'm handling upper/lower case file extensions in code already. I'll remove this. Thanks!

[michaelherger]

• I was wondering if we need to also change AIO.pm?

Good question! I just don't know how to test it 🤦🏻. I think it's Linux only, and under very specific circumstances, too? I'd have to test in a VM or container. Would you know how to easiest test this?

• Should we DELETE FROM scanned_pics... in Slim/Utils/Scanner/Local.pm as I had in Alternative artwork/box set approach #1536 ?

No. Unlike the audio files, which we process one after the other until all have been "consumed", we don't iterate over that list. And the same file can be used by multiple tracks.

There's actually a nice use case: if you set the cover format to %ARTIST and point the artwork folder at the folder with your artist pictures, tracks which otherwise wouldn't have an album cover or something might use an artist picture. Just a possibility, if all conditions are met. But it was quite useful to test with.

• Also, should we be checking the artwork when rescanning a single album?

To be verified indeed. Thanks.

Edit: the code would currently only use the DB when run in the scanner. It should still work in other cases, but then checking against the file system, rather than the DB. Single albums are run in the server process (IIRC). Therefore this should not be impacted by this change.

[michaelherger]

Would https://github.com/LMS-Community/slimserver/blob/public/9.2/Slim/Schema.pm#L1376-L1382 be a candidate where to look for "box" or album level artwork?

[darrell-k]

• I was wondering if we need to also change AIO.pm?

Good question! I just don't know how to test it 🤦🏻. I think it's Linux only, and under very specific circumstances, too? I'd have to test in a VM or container. Would you know how to easiest test this?

No. But I'm all Linux here, so I'll investigate.

• Should we DELETE FROM scanned_pics... in Slim/Utils/Scanner/Local.pm as I had in Alternative artwork/box set approach #1536 ?

No. Unlike the audio files, which we process one after the other until all have been "consumed", we don't iterate over that list. And the same file can be used by multiple tracks.

Understood. I also saw that you're clearing the new table in the optimize SQL script but of course there's a possibility that doesn't run (crashed or cancelled scan) and I'm thinking it should always reflect the current images on disc, so should always be empty at the start of the scan.

There's actually a nice use case: if you set the cover format to %ARTIST and point the artwork folder at the folder with your artist pictures, tracks which otherwise wouldn't have an album cover or something might use an artist picture. Just a possibility, if all conditions are met. But it was quite useful to test with.

• Also, should we be checking the artwork when rescanning a single album?

To be verified indeed. Thanks.

Edit: the code would currently only use the DB when run in the scanner. It should still work in other cases, but then checking against the file system, rather than the DB. Single albums are run in the server process (IIRC). Therefore this should not be impacted by this change.

You're keeping the code which finds images directly from the disk?

[darrell-k]

Would https://github.com/LMS-Community/slimserver/blob/public/9.2/Slim/Schema.pm#L1376-L1382 be a candidate where to look for "box" or album level artwork?

No, because we need to scan for new/changed artwork for all tracks/albums, not just those that have changed in a n&c scan.

The driver is this:

slimserver/Slim/Music/Artwork.pm

Lines 273 to 286 in 2f7076c

	my $sql = qq{
	SELECT
	tracks.id,
	tracks.url,
	tracks.cover,
	tracks.coverid,
	albums.id AS albumid,
	albums.title AS album_title,
	albums.artwork AS album_artwork
	FROM tracks
	JOIN albums ON (tracks.album = albums.id)
	WHERE $where
	GROUP BY tracks.cover, tracks.album
	};

which in my branch is enhanced to this:
https://github.com/darrell-k/slimserver/blob/87ce2c4fa264ea3ec804f02e8059537b4c8570b4/Slim/Music/Artwork.pm#L272-L290

So we get a result set row returned for every album/discnumber/directory combination. I then have further logic where the $processAlbum sub is called from within the $work sub when the album number changes (and one final time at the end to process the last album)

The bigger size of this result set, and the extra calls for each album, are the reasons why there are many more calls of findStandaloneArtwork in the box set branch.

I did wonder if we could use the mtime from the new table to loop through that in order to determine which albums /tracks needed updating, but that wouldn't handle the case where an image had been removed, and we'd still need to be processing such tracks/albums.

I can zip up my test data with an explanation of each test case, but it won't be today.

Can I take it that you are taking over the coding the box set functionality? I'm perfectly fine with that, and of course remain on hand for support and testing.

[mikeysas]

It would be good if @mikeysas updated #1430 with the additional requirements as he offered yesterday in his #1536 comment. Then we can take that issue as the statement of box set requirements.

Done. #1430 (comment)

[darrell-k]

I'm still trying to invoke the AIO scan, it looks like we need to trigger a directory scan in the main process, but I can't yet work out how to do that.

But I found a bug: when Slim::Music::Artwork::findStandaloneArtwork is called via Slim::Schema::_checkValidity (eg for a now-playing track that has a new mtime) it wipes the artwork from the track.

[michaelherger]

I'm still trying to invoke the AIO scan, it looks like we need to trigger a directory scan in the main process, but I can't yet work out how to do that.

I think one way is to use the AutoRescan plugin.

But I found a bug: when Slim::Music::Artwork::findStandaloneArtwork is called via Slim::Schema::_checkValidity (eg for a now-playing track that has a new mtime) it wipes the artwork from the track.

Thanks! Looking into this.

Are you using standalone artwork for that particular file? Any custom artwork format string? Could you please enable logging for artwork and send me the relevant snippet?

[michaelherger]

You're keeping the code which finds images directly from the disk?

Basically yes. It's not the exact same code, but functionally it should be. Under certain circumstances it doesn't make sense to initialise the DB and everything, eg. when dealing with a single, known file.

[michaelherger]

Would public/9.2/Slim/Schema.pm#L1376-L1382 be a candidate where to look for "box" or album level artwork?

No, because we need to scan for new/changed artwork for all tracks/albums, not just those that have changed in a n&c scan.

Why would we have to do so?

[darrell-k]

Would public/9.2/Slim/Schema.pm#L1376-L1382 be a candidate where to look for "box" or album level artwork?

No, because we need to scan for new/changed artwork for all tracks/albums, not just those that have changed in a n&c scan.

Why would we have to do so?

We already do. External artwork can change without music file changes.

[darrell-k]

I'm still trying to invoke the AIO scan, it looks like we need to trigger a directory scan in the main process, but I can't yet work out how to do that.

I think one way is to use the AutoRescan plugin.

I already tried that, it runs in a separate scanner process which doesn't look for AIO capability.

But I found a bug: when Slim::Music::Artwork::findStandaloneArtwork is called via Slim::Schema::_checkValidity (eg for a now-playing track that has a new mtime) it wipes the artwork from the track.

Thanks! Looking into this.

Are you using standalone artwork for that particular file? Any custom artwork format string? Could you please enable logging for artwork and send me the relevant snippet?

There is a single image in the album directory, with a random name (ie not one of the preferred names). No custom format. Not near a computer right now, but I guess it's going into the "else" after "if scanner" block which uses the new table.

[michaelherger]

I already tried that, it runs in a separate scanner process which doesn't look for AIO capability.

Disable all plugins that provide scanner features...

I guess we can assume it's a rare case 🤣.

There is a single image in the album directory, with a random name (ie not one of the preferred names). No custom format. Not near a computer right now, but I guess it's going into the "else" after "if scanner" block which uses the new table.

Oh, good catch! The disk lookup for random name got lost in my re-factor... Great catch, thanks!

[michaelherger]

There is a single image in the album directory, with a random name (ie not one of the preferred names). No custom format. Not near a computer right now, but I guess it's going into the "else" after "if scanner" block which uses the new table.

This should be fixed. Thanks!

[darrell-k]

I wasn't suggesting removing the nocase from scanned_files. Does #705 no longer apply?

[michaelherger]

Heh... what were you suggesting then?

[darrell-k]

Just remove it from scanned_pics.

[michaelherger]

Ok. Got it. Thanks for the clarification.

[michaelherger]

Any objections to merging this?

[darrell-k]

I haven't tested all scenarios (eg album rescan, and still haven't managed to trigger the AIO scan).

[darrell-k]

Also I haven't looked at the variable artwork naming.

[michaelherger]

Also I haven't looked at the variable artwork naming.

That part actually is in 9.2 already.

[michaelherger]

I haven't tested all scenarios (eg album rescan, and still haven't managed to trigger the AIO scan).

album rescan wouldn't work yet

Album rescan does work. I was confused by "boxset" idea, which is not part of this PR yet.

[darrell-k]

Would it be worth looking at Slim::Music::Artwork::generateImageId because that uses -e ?

Could we use the new table instead?

(the code in Slim::Music::Artwork uses Slim::Schema::Track->generateCoverId but that just points back to the above function)

[darrell-k]

Here's an odd issue:

I've got two albums (in adjacent folders) which have the same image (different file paths, same image name/size) and under this PR the tracks for both albums have tracks.cover set to the image from the second folder.

[darrell-k]

The problem I just mentioned in the main thread is here. I reverted to LIKE (just for an experiment) and the issue was resolved.

The directories in question are:

/home/darrell/Music/testmusic/stripped_test3/Bill Evans/Waltz For Debby (Copy)
and
/home/darrell/Music/testmusic/stripped_test3/Bill Evans/Waltz For Debby