Tracks Skipped when importing with selected fields

Lexicon version: 1.1.26
Operating system (remove one): Windows

Bug description: When running an import using the “Select Fields” dialog to selectively import specific fields a large number of tracks are failing to import. Tracks that fail to import are removed from the Lexicon playlist. An example of the messages seen in the logs is below:

[2022-09-01T04:05:25.205Z] [warn]: could not find track 268437850 in trackId map
[2022-09-01T04:05:25.205Z] [warn]: could not find track 268437841 in trackId map
[2022-09-01T04:05:25.205Z] [warn]: could not find track 268437837 in trackId map
[2022-09-01T04:05:25.205Z] [warn]: could not find track 268437834 in trackId map
[2022-09-01T04:05:25.209Z] [warn]: could not find track 268437707 in trackId map
[2022-09-01T04:05:25.209Z] [warn]: could not find track 268437761 in trackId map

This doesn’t appear to happen when you run the same import with all fields selected and all tracks are imported successfully.

This also exposes another issue that I feel needs addressing in that there’s no message in the UI to let the user know that the import had a problem and skipped tracks. In this test I had a playlist with 194 tracks in RB. Lexicon just sais “Import Complete” in the UI with no error despite it only managing to import 98/194 tracks successfully. Unless the user actually checks their playlist and notices that tracks are missing in Lexicon they could pretty easily end up run a sync back to RB on that playlist causing the playlist in both applications to now be missing those tracks.

Obviously, this specific import bug will likely be resolved but bugs do happen and a user should be adequately notified in the UI of any issues occurring on import that result in tracks being removed from their existing Lexicon playlists.

Can you show me your Specific Fields settings?

Does a notification help if there were tracks missing? I’m just concerned that some users will always have “missing” tracks because of a variety of valid reasons. They would get a warning but there isn’t really anything they can do about it. For example, you could have weird VirtualDJ tracks (you can add almost anything to VDJ) or Windows paths in a Mac DJ app. There are valid reasons and mostly you should ignore that when they happen.

This specific example I’m referencing I only had ratings selected, everything else unchecked. I believe the other time I saw this I had only ratings/cue points checked.

It might make sense to ignore import errors when importing a brand new playlist that doesn’t already exist in Lexicon where errors can’t have a bearing on existing Lexicon data. But I strongly believe that when importing an existing playlist with merge selected that all import errors should be reported to the user as such errors can directly impact a user’s Lexicon data negatively and if unchecked, result in data loss (valid Lexicon tracks disappearing from playlists).

Thinking about this some more, in addition to a notification about the failed import, it might also be worth having a dialog when an import error is detected giving you to optionally not update the Lexicon playlist. So in my example, Lexicon does the import, 96 tracks fail to import it prompts me and I choose not to update the playlist. The successful 98 individual tracks are updated as per the import, but my 196-track playlist in Lexicon remains untouched. Personally, I’d be happy with this as the default behaviour and just never update an existing LXC Playlist unless the entire import was successful but for less savvy users who might not know how to manually resolve an issue a dialog box gives them the option to just drop the affected tracks.

Well I do agree some more feedback would be good and I could add a text report to the import process. But I don’t think hidden behavior or potential unhelpful/confusion notifications is the way to go. The goal is to not have good tracks failing to import of course! But yeah, they sometimes happen in this complex environment with changing DJ apps.

For the Specific Fields bug, can you upload your RB master.db file? http://upload.lexicondj.com
And LXC database upload

Thanks!

But I don’t think hidden behavior or potential unhelpful/confusion notifications is the way to go.

Hidden behaviour is exactly what’s happening now. Users are potentially having tracks vanish from their Lexicon playlists with no feedback.

The goal is to not have good tracks failing to import of course! But yeah, they sometimes happen in this complex environment with changing DJ apps.

I totally acknowledge “shit happens” sometimes. What Lexicon does is complex and has many moving parts from different applications. What I’m advocating for is transparency so that when shit happens the user is aware of it and isn’t losing data in Lexicon with no warning or knowledge.

I’m really struggling to see your point of view on this. I really do understand wanting to keep things simple but do you really not see losing tracks from your Lexicon playlists on a merge import with no warning or notification as a terrible thing to happen for a user’s experience? Lexicon is promoted as your master library and database. If I can’t trust it not to randomly remove tracks from my playlists with no warning or explanation when a random import error occurs how can I trust it as my master database?

can you upload your RB master.db and LXC db?

Done, “Import Specific Fields Issue” was the name on the upload. Playlist in question was “LXCUpdate”

Is a text report not a good compromise then? With playlist names and amount of tracks that it failed to import.

The reason I’m hesitant is because Lexicon should gravitate towards 100% reliability with every bug I fix.
Then there’s only rare DJ app changes to account for but they need to be pretty fundamental before it breaks import. And even then I think it’s exceedingly rare that a DJ app changes in such a way that it would break import but not error in Lexicon at all.

Looking into the bug.

The reason I’m hesitant is because Lexicon should gravitate towards 100% reliability with every bug I fix.
Then there’s only rare DJ app changes to account for but they need to be pretty fundamental before it breaks import. And even then I think it’s exceedingly rare that a DJ app changes in such a way that it would break import but not error in Lexicon at all.

In my mind, this is an argument in favour of more aggressive error reporting, not less. As things gravitate to 100% reliability the error messaging fades away because the bugs and quirks all get resolved. But if/when shit hits the fan from a DJ app making an unexpected change or testing a beta version of Lexicon or just a user doing something completely wonky it should be made abundantly clear to the user what has happened inside of Lexicon so that they don’t lose data without even knowing it.

I’m sorry to keep belaboring this point but I feel very strongly that maintaining data integrity and user trust is a higher priority than a clean UX. Even though database backups are taken and things could be recovered in most cases I could very easily see a user sync/import back and forward from their DJ app a bunch of times then realise a week/month later that it’s been dropping tracks on the imports and look and find their playlists decimated and quit using Lexicon as a result. I don’t want that to happen for anyone.

Is a text report not a good compromise then? With playlist names and amount of tracks that it failed to import.

Yes, a text report detailing the import failure is a fair compromise.

I do believe that some functionality around importing tracks without changing Lexicon playlists would also be really useful but I’ll certainly settle for the transparency of a report on imports that contained errors.

I’ll add a report and if there is anything to report, the import notification will say so. If there was nothing to report, it will just be the usual green “import success” notification.