Arrangeable FlowLayout Panel

I found this piece of code and when I ran it, I was surprised how usable it actually was.

At the time, I needed something that would allow me to sort an array of things by dragging and dropping.  The things I was working with were photos, but this class handles pretty much everything.

You can grab an item and drag it into a new position.  A caret shows you where the new position will be.  Even better, you can multi-select items and drag them to a new position.  When you need the new order, just iterate through the child controls.

I don’t actually have a use for it right now, but I need to save this so I have it for the future.

Public Class ArrangeableFlowLayoutPanel
    Inherits FlowLayoutPanel

    Protected mouseDownPoint As Point
    Protected insertCaret As PictureBox
    Protected isMultiSelectOn As Boolean
    Protected isRangeSelectOn As Boolean

    Public Property AllowReordering As Boolean = True
    Public Property CaretColor As Color = Color.Green
    Public Property CaretWidth As Integer = 3
    Public Property CaretPadding As Padding = New Padding(2, 0, 2, 0)
    Public Property SelectionColor As Brush = Brushes.Black
    Public Property SelectionWidth As Integer = 1
    Friend Property SelectedControls As New Generic.List(Of Control)
    Public Property DragTolerance As Integer = 40

    Public Event ItemOrderChanged(sender As Object, e As EventArgs)

    Public Sub New()
        Me.AllowDrop = True
        Me.AutoScroll = True

        CreateCaret()

    End Sub

    Private Sub Form_Key(sender As Object, e As KeyEventArgs)
        isMultiSelectOn = e.Control
        isRangeSelectOn = e.Shift
    End Sub

    Private Sub ArrangeableFlowLayoutPanel_ControlAdded(sender As Object, e As ControlEventArgs) Handles Me.ControlAdded
        AddHandler e.Control.MouseDown, AddressOf Item_MouseDown
        AddHandler e.Control.MouseUp, AddressOf Item_MouseUp
        AddHandler e.Control.MouseMove, AddressOf Item_MouseMove
        AddHandler e.Control.Paint, AddressOf Item_Paint

    End Sub

    Private Sub ArrangeableFlowLayoutPanel_ControlRemoved(sender As Object, e As ControlEventArgs) Handles Me.ControlRemoved
        RemoveHandler e.Control.MouseDown, AddressOf Item_MouseDown
        RemoveHandler e.Control.MouseUp, AddressOf Item_MouseUp
        RemoveHandler e.Control.MouseMove, AddressOf Item_MouseMove
        RemoveHandler e.Control.Paint, AddressOf Item_Paint

        If SelectedControls.Contains(e.Control) Then SelectedControls.Remove(e.Control)

    End Sub

    Private Sub ArrangeableFlowLayoutPanel_ParentChanged(sender As Object, e As EventArgs) Handles Me.ParentChanged
        Dim f As Form

        f = Me.FindForm
        If f IsNot Nothing AndAlso Not f.KeyPreview Then f.KeyPreview = True

        RemoveHandler Me.FindForm.KeyDown, AddressOf Form_Key
        RemoveHandler Me.FindForm.KeyUp, AddressOf Form_Key

        AddHandler Me.FindForm.KeyDown, AddressOf Form_Key
        AddHandler Me.FindForm.KeyUp, AddressOf Form_Key
    End Sub

    Private Sub ArrangeableFlowLayoutPanel_Disposed(sender As Object, e As EventArgs) Handles Me.Disposed
        insertCaret.Dispose()

        RemoveHandler Me.FindForm.KeyDown, AddressOf Form_Key
        RemoveHandler Me.FindForm.KeyUp, AddressOf Form_Key

    End Sub

    Private Sub ArrangeableFlowLayoutPanel_DragDrop(sender As Object, e As DragEventArgs) Handles Me.DragDrop
        Dim dropIndex As Integer

        For i As Integer = 0 To SelectedControls.Count - 1
            dropIndex = Me.Controls.GetChildIndex(insertCaret)
            Me.Controls.SetChildIndex(SelectedControls(i), dropIndex + 1)
        Next

        Me.Controls.Remove(insertCaret)
        RaiseEvent ItemOrderChanged(Me, New EventArgs)

    End Sub

    Private Sub ArrangeableFlowLayoutPanel_DragLeave(sender As Object, e As EventArgs) Handles Me.DragLeave
        Dim topBorderY As Integer
        Dim bottomBorderY As Integer
        Dim mousePositionY As Integer
        Dim hostForm As Form

        Me.Controls.Remove(insertCaret)

        hostForm = Me.FindForm
        topBorderY = hostForm.PointToClient(Me.Parent.PointToScreen(Me.Location)).Y
        bottomBorderY = Me.Height + topBorderY
        mousePositionY = hostForm.PointToClient(MousePosition).Y

        Do While mousePositionY >= bottomBorderY ' Below bottom of control
            If Me.VerticalScroll.Value <= Me.VerticalScroll.SmallChange + Me.VerticalScroll.Maximum Then
                Me.VerticalScroll.Value += Me.VerticalScroll.SmallChange
            Else
                Me.VerticalScroll.Value = Me.VerticalScroll.Maximum
            End If

            mousePositionY = hostForm.PointToClient(MousePosition).Y
            Me.Refresh()

        Loop

        Do While mousePositionY <= topBorderY ' Above top of control
            If Me.VerticalScroll.Value >= Me.VerticalScroll.SmallChange - Me.VerticalScroll.Minimum Then
                Me.VerticalScroll.Value -= Me.VerticalScroll.SmallChange
            Else
                Me.VerticalScroll.Value = Me.VerticalScroll.Minimum
            End If

            mousePositionY = hostForm.PointToClient(MousePosition).Y
            Me.Refresh()

        Loop

    End Sub

    Private Sub ArrangeableFlowLayoutPanel_DragOver(sender As Object, e As DragEventArgs) Handles Me.DragOver
        Dim ctl As Control
        Dim dropControlPosition As Point
        Dim dropIndex As Integer

        If e.Data IsNot Nothing Then
            e.Effect = DragDropEffects.Move
            ctl = Me.GetChildAtPoint(Me.PointToClient(New Point(e.X, e.Y)))

            If ctl IsNot Nothing AndAlso ctl IsNot insertCaret Then
                dropControlPosition = ctl.PointToClient(New Point(e.X, e.Y))

                If dropControlPosition.X <= ctl.Width \ 2 Then
                    dropIndex = Me.Controls.GetChildIndex(ctl) - 1
                Else
                    dropIndex = Me.Controls.GetChildIndex(ctl) + 1
                End If

                If dropIndex < 0 Then dropIndex = 0

                If Not Me.Controls.Contains(insertCaret) Then
                    insertCaret.Height = ctl.Height
                    Me.Controls.Add(insertCaret)
                End If

                Me.Controls.SetChildIndex(insertCaret, dropIndex)

            End If

        End If

    End Sub

    Private Sub Item_MouseDown(sender As Object, e As MouseEventArgs)
        If e.Button = System.Windows.Forms.MouseButtons.Left Then
            mouseDownPoint = e.Location
        End If

    End Sub

    Private Sub Item_MouseUp(sender As Object, e As MouseEventArgs)
        Dim ctl As Control
        Dim startIndex As Integer
        Dim endIndex As Integer
        Dim newCtl As Control

        If e.Button = System.Windows.Forms.MouseButtons.Left Then
            ctl = DirectCast(sender, Control)

            ' Choosing individual items or the first of a range
            If isMultiSelectOn OrElse (isRangeSelectOn And SelectedControls.Count = 0) Then
                If SelectedControls.Contains(ctl) Then
                    SelectedControls.Remove(ctl)
                Else
                    SelectedControls.Add(ctl)
                End If

                ctl.Invalidate()

                ' Choosing the end of a range
            ElseIf isRangeSelectOn Then
                startIndex = Me.Controls.GetChildIndex(SelectedControls(SelectedControls.Count - 1))
                endIndex = Me.Controls.GetChildIndex(ctl)

                For i As Integer = startIndex To endIndex Step CInt(IIf(startIndex < endIndex, 1, -1))
                    newCtl = DirectCast(Me.Controls(i), Control)

                    If Not SelectedControls.Contains(newCtl) Then
                        SelectedControls.Add(newCtl)
                        newCtl.Invalidate()
                    End If

                Next

            Else
                SelectedControls.Clear()
                SelectedControls.Add(ctl)
                For Each c As Control In Me.Controls
                    c.Invalidate()
                Next

            End If ' single or multi-select

        End If ' Left button only

    End Sub

    Private Sub Item_MouseMove(sender As Object, e As MouseEventArgs)
        Dim ctl As Control
        Dim rect As Rectangle
        Dim rectPoint As Point

        If AllowReordering AndAlso e.Button = System.Windows.Forms.MouseButtons.Left Then
            ctl = DirectCast(sender, Control)

            ' create a range before dragging activates
            rectPoint = New Point(mouseDownPoint.X, mouseDownPoint.Y)
            rectPoint.Offset(0 - (Me.DragTolerance \ 2), 0 - (Me.DragTolerance \ 2))

            rect = New Rectangle(rectPoint, New Size(Me.DragTolerance, Me.DragTolerance))

            ' See if we've dragged outside the tolerance area
            If Not rect.Contains(e.Location) Then

                ' dragged item is not in selection, include it if ctrl is held
                ' otherwise, clear the selection and only use the dragged item
                If Not SelectedControls.Contains(ctl) Then
                    If isMultiSelectOn Then
                        SelectedControls.Add(ctl)
                        ctl.Invalidate()

                    Else
                        SelectedControls.Clear()
                        SelectedControls.Add(ctl)
                        For Each c As Control In Me.Controls
                            c.Invalidate()
                        Next

                    End If ' Ctrl held down

                End If ' Not in current selection

                Me.DoDragDrop(SelectedControls, DragDropEffects.Move)

            End If ' Outside drag buffer area

        End If ' mouse button down

    End Sub

    Private Sub Item_Paint(sender As Object, e As PaintEventArgs)
        Dim ctl As Control

        ctl = DirectCast(sender, Control)

        If SelectedControls.Contains(ctl) Then
            ' Draw outline
            e.Graphics.DrawRectangle(New Pen(Me.SelectionColor, Me.SelectionWidth), Me.SelectionWidth \ 2, Me.SelectionWidth \ 2, ctl.Width - Me.SelectionWidth, ctl.Height - Me.SelectionWidth)
        End If

    End Sub

    Private Sub CreateCaret()
        insertCaret = New PictureBox
        With insertCaret
            .Name = "caret"
            .Height = 1
            .Width = CaretWidth
            .Margin = CaretPadding
            .Padding = New Padding(0)
            .BackColor = Me.CaretColor
        End With

    End Sub

End Class

VS2015 Hangs/Freezes When Adding MVC Controller or View

This issue came to me today at work where two devs said they couldn’t add a new view to a project.  I repro’d it right away and began investigating.

The problem was isolated to one project in the solution.  I compared references between projects and compared the underlying project files.  Each change that I made to bring them closer into alignment didn’t help.

As said many times, the trick to successful debugging is isolation.  So I unloaded every project except for the web project being tested.  The Add dialogs loaded immediately.  So at this point, I had a workaround.  The projects referenced by the web project (only one) could be unloaded when a new view or controller was needed.

Although this was a workaround, it’s hardly a solution.  I continued down that path of figuring out why the MVC Add dialogs were causing VS to hang.

At one point, I was researching online and I noticed that the Add Controller dialog did appear.  It took over 5 minutes to get to that point, but it did show.  So I tried the Add View dialog and it eventually did show as well.  The dropdown list for model classes was populated with every class from the referenced project.  That might have been the cause of the slowdown.

Something about the dialog struck me funny.  They were different between the two projects.  This suggested to me that they were using different libraries.  I loaded up the NuGet package manager between the two projects and saw the functional project used Razor v2.0, but the failing project used v3.2.3.

I downgraded the failing project to Razor 2.0.  This caused a new problem.  I no longer had context menu options to directly add controllers or views.  Fortunately, I knew where to go for this fix.

In the project file is a node called ProjectTypeGuid, which tells VS what menu options to provide.  I could see the non-working project was missing one of  the guids.  I quick copy between the projects restored the context menu options.

TFS 2012 Fails To Create New Team Project

Today, I went to add a new team project to TFS.  I did the usual steps by giving it a name and choosing the template type (not really relevant to me since I’m a solo programmer).  Visual Studio worked for a bit, then failed with the following error in the log file:

Event Description: TF30162: Task "LinkTypes" from Group "WorkItemTracking" failed
Exception Type: Microsoft.TeamFoundation.Client.PcwException
Exception Message: Page not found.

The Internet had plenty of advice to resolve TF30162 errors, but nothing useful for the specific LinkTypes error.  The only suggestion was to reload the project templates.  I was not able to do that because in TFS 2012, those templates are locked and unable to be uploaded and overwritten.

So I attempted a Repair install of TFS 2012.  That had no effect

I then made sure I was on the latest version of TFS, so I installed Update 4.  My scrum project template went from 2.0 to 2.2, but I still got the exact same error.

I then tried to create a new Project Collection and create a team project in that.  Still, no change.

Then I did a repair install on Visual Studio 2015.  This took a very long time and required two restarts.  And after that, no improvement.

So far, I’ve eliminated the TFS binaries, the TFS database, and the VS binaries.  There’s only one element left – IIS.  And that one seems to have potential because of the PageNotFound error.

Poking around the IIS log files, there’s entries in there like this:

2017-01-09 14:27:23 ::1 GET /tfs/_apis/connectionData connectOptions=IncludeServices&lastChangeId=-1&lastChangeId64=-1 8080 MicrosoftAccount\myaccount@domain.com ::1 VSServices/14.98.25331.0+(devenv.exe+,Pro,+SKU:31) – 404 0 0 1719

Looking at IIS log files prior to when my problem happened,  this 404 error has happened before, too.  but there was also other calls to the service like:

2016-03-25 12:43:46 ::1 OPTIONS /tfs/defaultcollection/_apis/ – 8080 MicrosoftAccount\myaccount@domain.com ::1 Team+Foundation+(devenv.exe,+14.0.24712.0,+Pro,+SKU:31)+VSServices/14.0.24712.0+(devenv.exe+,Pro,+SKU:31) – 404 0 0 8

Two suspects come up in Internet searches: Git and IIS plugins.  Disabling some of the plugins in C:\Program Files\Microsoft Team Foundation Server 11.0\Application Tier\Web Services\bin\Plugins made no difference.

At this point, I suspect an incompatibility between VS2015 and TFS 2012, so I upgrade my TFS install to 2015.  What else could it be?  A call is being made to a REST service that doesn’t exist, or is different enough that the inbound parameters don’t match up.  This TFS upgrade requires SQLExpress to be upgraded as well.  The rabbit hole keeps going deeper and deeper.

After that entire install and restart, I was finally able to create a new Team Project.  The new default project template is “Agile”, not that I care much.

So, the short solution for my error: Upgrade TFS from 2012 to 2015.  It must’ve become incompatible after a VS update somewhere along the way.

Yahoo Security Is Still A Joke

Today I got an email from Yahoo.  You know, they’ve been having some security issues lately with millions of accounts being compromised.

The email said they noticed I hadn’t changed my password in a while.  That was kind of odd since I thought I got this email not long ago and humored them by changing my password.  I don’t use Yahoo for anything important anymore, so I didn’t really care.

image

When I looked at where the email was sent to, it was sent to a non-yahoo email address.  It was sent to an email address I used to log in to Facebook.  Putting it together, this was an account I used to log in to Flickr using the Facebook account login option.  It’s been a while since Yahoo gave the middle finger to external logins because they didn’t want to support them anymore.  But that doesn’t mean they cleared all that data out.

So let’s get this right.  I used to log in to Flickr using a Facebook login.  Yahoo discontinued Facebook login ability.  I can’t remember if I converted to a Yahoo account or abandoned it, but regardless, there is an account in Yahoo’s system that has a Facebook email in it.  I can’t find that email anywhere in my password manager.  Even if I did, there’s no Yahoo password for me to change.  It’s a Facebook login.

I used to be pretty neutral on Yahoo.  I didn’t care one way or the other about them.  That’s changed.  I really want them to close up shop.  They are not doing the world any favors with their lack of security and perpetually changing services.

Handling JPEG XR (HD Photos/Windows Media Photos)

I recently found some images on the internet in a format that I couldn’t directly support in my applications.  The images were in JPEG XR, with a .jxr extension.  These images would fail with a “The parameter is incorrect” error when loading the byte array into a System.Drawing.Bitmap (using Image.FromStream).

A little bit of research showed that the standard Drawing classes wouldn’t support this format.  I would need to use classes from the WPF library.  I added references to PresentationCore and WindowsBase, then wrote this function to convert the bytes of a JXR image to a standard bitmap:

 

    Private Function JXRToBitmap(bytes() As Byte) As Bitmap
        Dim s As IO.MemoryStream
        Dim dec As Windows.Media.Imaging.WmpBitmapDecoder
        Dim enc As Windows.Media.Imaging.BmpBitmapEncoder
        Dim bmp As Bitmap

        s = New IO.MemoryStream
        dec = New Windows.Media.Imaging.WmpBitmapDecoder(New IO.MemoryStream(bytes), Windows.Media.Imaging.BitmapCreateOptions.None, Windows.Media.Imaging.BitmapCacheOption.Default)
        enc = New Windows.Media.Imaging.BmpBitmapEncoder

        enc.Frames.Add(Windows.Media.Imaging.BitmapFrame.Create(dec.Frames(0)))
        enc.Save(s)

        bmp = New Bitmap(s)

        s.Dispose()

        Return bmp

    End Function

Since I didn’t know in advance what the file format was, the best I could do was attempt to load the bitmap and if it threw an exception, I would try to load it again using this function in the Catch block.  If that failed, then it would be a true exception.

The Minimum For Integration Logins

When you want to have your site support login from other services like Google, Facebook, or Microsoft, you use the Owin libraries.  The base MVC template sets all this up for you, but it also has a lot of stuff that your site probably already has, like login and a user database and whatnot.

So, what’s the minimum amount of code you need to add to handle the extra login sources.  You need two things: a class to initialize Owin and an MVC controller to handle the login redirect and the login response from the external site.

This is the class to initialize Owin:

Imports Microsoft.Owin
Imports Microsoft.Owin.Security
Imports Owin.Security.Providers

' Nuget Packages needed:
' Owin
' Microsoft.Owin
' Microsoft.Owin.Security
' Microsoft.Owin.Security.Coookies
' Microsoft.Owin.Host.SystemWeb
' Microsoft.Owin.Security.Facebook (for FB Login)
' Microsoft.Owin.Security.Google (for Google Login)
' Microsoft.Owin.Security.MicrosoftAccount (for MS Login)
' Owin.Security.Providers (for many other Logins)

<Assembly: Microsoft.Owin.OwinStartup(GetType(Startup))>
Public Class Startup
    Public Sub Configuration(app As Owin.IAppBuilder)
        Dim opt As Cookies.CookieAuthenticationOptions

        opt = New Cookies.CookieAuthenticationOptions With {.LoginPath = New PathString("/Account/Login")}

        Owin.CookieAuthenticationExtensions.UseCookieAuthentication(app, opt)
        AppBuilderSecurityExtensions.SetDefaultSignInAsAuthenticationType(app, opt.AuthenticationType)

        ' Google - Signup https://developers.google.com
        Owin.GoogleAuthenticationExtensions.UseGoogleAuthentication(app, New Google.GoogleOAuth2AuthenticationOptions With {.ClientId = "", .ClientSecret = ""})

        ' Facebook - Signup https://developers.facebook.com/
        Owin.FacebookAuthenticationExtensions.UseFacebookAuthentication(app, New Facebook.FacebookAuthenticationOptions With {.AppId = "", .AppSecret = ""})

        ' Microsoft - Signup https://account.live.com/developers/applications/index (RedirectURL in app settings must be http://domain.com/signin-microsoft) 
        Owin.MicrosoftAccountAuthenticationExtensions.UseMicrosoftAccountAuthentication(app, New MicrosoftAccount.MicrosoftAccountAuthenticationOptions With {.ClientId = "", .ClientSecret = ""})

        ' Yahoo - Signup http://developer.yahoo.com (not working; 401 errors) 
        Yahoo.YahooAuthenticationExtensions.UseYahooAuthentication(app, New Yahoo.YahooAuthenticationOptions With {.ConsumerKey = "", .ConsumerSecret = ""})

        ' Flickr - Signup https://www.flickr.com/services/api/keys/
        Flickr.FlickrAuthenticationExtensions.UseFlickrAuthentication(app, New Flickr.FlickrAuthenticationOptions With {.AppKey = "", .AppSecret = ""})

    End Sub

End Class

And this is the controller class to manage the logins:

Imports System.Web.Mvc

Namespace Controllers
    Public Class OwinController
        Inherits Controller

        Function Login(provider As String, returnURL As String) As ActionResult
            Return New ChallengeResult(provider, Url.Action("Callback", "Owin", New With {.ReturnURL = returnURL}))
        End Function

        Function Callback(returnURL As String)
            Dim userID As String
            Dim email As String

            ' capture credentials
            With DirectCast(My.User.CurrentPrincipal.Identity, Security.Claims.ClaimsIdentity)
                userID = .Claims.Where(Function(x) x.Type.EndsWith("/nameidentifier")).DefaultIfEmpty(New Security.Claims.Claim("", "")).First.Value
                email = .Claims.Where(Function(x) x.Type.EndsWith("/emailaddress")).DefaultIfEmpty(New Security.Claims.Claim("", "")).First.Value
            End With

            Debug.WriteLine(My.User.CurrentPrincipal.Identity.Name)

            Return New RedirectResult(returnURL)

        End Function

        Private Class ChallengeResult
            Inherits HttpUnauthorizedResult

            Public Property LoginProvider As String
            Public Property RedirectURL As String

            Public Sub New(provider As String, url As String)
                Me.LoginProvider = provider
                Me.RedirectURL = url
            End Sub

            Public Overrides Sub ExecuteResult(context As ControllerContext)
                Dim prop As New Microsoft.Owin.Security.AuthenticationProperties With {.RedirectUri = Me.RedirectURL}
                context.HttpContext.GetOwinContext.Authentication.Challenge(prop, Me.LoginProvider)
            End Sub

        End Class

    End Class

End Namespace

The Callback method in the OwinController is where you would look up the user by their provider/nameidentifier combination or store that info in an existing logged-in user profile or create a new user profile for the new login.

When you want to offer external logins, you call the Owin controller:

@Html.ActionLink("Google Login", "Login", "Owin", New With {.Provider = "Google", .ReturnURL = Url.Action("Secure")}, Nothing)
<br />
@Html.ActionLink("Facebook Login", "Login", "Owin", New With {.Provider = "Facebook", .ReturnURL = Url.Action("Secure")}, Nothing)
<br />
@Html.ActionLink("Yahoo Login", "Login", "Owin", New With {.Provider = "Yahoo", .ReturnURL = Url.Action("Secure")}, Nothing)
<br />
@Html.ActionLink("Flickr Login", "Login", "Owin", New With {.Provider = "Flickr", .ReturnURL = Url.Action("Secure")}, Nothing)
<br />
@Html.ActionLink("Microsoft Login", "Login", "Owin", New With {.Provider = "Microsoft", .ReturnURL = Url.Action("Secure")}, Nothing)

This assumes there is an action named “Secure” that you want to come back to.

Outlook Rules Utility

Way back in 2013, I was getting annoyed with the rules manager in Outlook.  See, I have many rules.  I have one folder for every person in my inbox.  The rules move their messages to their respective folders.  So at work, I have 67 rules.  At home, I have 267 rules.  I like to keep my rules sorted, so I can find one quickly if I need to.  So if I add a rule for Zillow.com, I have to click the “move down” arrow hundreds of times to get to the bottom of the list.  There has to be a better way.

I focused on the rules export, which creates a binary file of all your rules.  I just had to reverse-engineer the file structure and I could create an app to modify the order of the rules.  Over a span of time, I figured out what I needed and the results were somewhat workable.  I uploaded the source code to CodePlex and didn’t really use it much after that.

Two and a half years later, I was annoyed at my rules again and fired up this code to fix my rules.  I discovered the code didn’t work like I expected, so I set to work fixing the problems.  Once it was working properly, I decided that the CodePlex project deserved to have a binary for download.  It wasn’t doing a lot of good to people who weren’t programmers and who could compile the code on their own.

After that update, I did a quick search to see if anything new had been learned about the Outlook rules file since I last worked on it.  Nothing.  But what I did stumble on was a post asking how to parse the rules file.  In the post, the author explained where the rules file was stored in the mail store file.  Taking that information, I was able to add the ability for my utility to read and write the rules directly from Outlook.  No more export/import hassle!  That is a huge step forward.

This unexpected progress has inspired me to work a little harder on the project.  There are random bits of code and comments that handle the different criteria and the different actions.  These need to be clarified and once that is done, this utility could be an entire replacement for the Rules Manager in Outlook.

The project is at http://rwzreader.codeplex.com.

GrooveManager 1.0 Release

I’ve been posting for a little while on Groove Music’s database.  While investigating, I wrote up some test utilities to read and write to the Groove Music database.  These utilities have been combined and simplified into a utility called GrooveManager.

At this point, the utility is extremely limited.  It will only edit Artist names and the Artist image.  And the Artist image feature itself is limited in that you can’t use external images.  It has to be an artist with a profile on Xbox Music.  That limitation aside, there are still times when you have a legitimate need to use an artist that isn’t spelled the same as you have it in your collection.  For example, Sergei Rachmaninoff seems to have many different spellings and variations.

The project page is at https://groovemanager.codeplex.com and has simple documentation on the website.

Groove Metadata (database)

Here’s some more info on the structure of the Groove Music database.

Schema

As mentioned the database is in ESE format.  The tables within are pretty much just like any other database.  And in Groove Music’s case, you have the expected elements of Artist, Album, Tracks, and Genre.  Most of the fields are what you would expect them to be.  Here’s a brief analysis on some things you might be curious about.

tblPerson

This table holds the Artist info, but it’s more than just the band name.  It also includes credits in each song’s ID3 tag, like composer, artist, albumartist, etc.  This means you’re going to have a lot more “Persons” than bands in your library.  Especially when dealing with compilation albums.  Other fields:

tblAudioAlbum

This table is the list of albums in the library.

tblTrack

This table lists the songs.  There’s lots of interesting info in here that needs more exploration.  I have three albums in the cloud.  Two I purchased back from the Zune days, and one I just uploaded to OneDrive from my hard drive.  I’ll reference these in the interesting fields.

  • CloudCollectionContentId – This seems to be a guid to use in a link to an online file.  This is populated for both my purchased and uploaded albums
  • BlockFromCloud – Always true
  • ActionableMediaId – I assume this is a Guid that points to a matched online track. It’s blank on my more obscure albums.
  • InCloudCollection – True for the three albums I have in OneDrive and Store-purchased
  • CatalogId – The same as ActionableMediaId, but there are more CatalogIds than ActionableMediaIds.  Maybe ActionableMediaId means it can be used online instead of just having information?
  • AlbumImageId – A Guid that is used to get the album art from web services
  • AlbumImageSource – A number that likely represents the URL to get the image from (zuneimages, music.microsoft.com, or musicimages.xbox.com).
  • UniqueFileId – These values are similar to what Windows Media Player used to store in the ID3 tag when it would apply its metadata from its store.

So still, plenty to learn about this database.

Groove Metadata (album art)

As a previous post mentioned, the metadata for Groove is in an ESENT database.  One of the big questions is where does it get its artwork from?

Album Artwork

Groove will get its album artwork from one of two places: from the file metadata or from xbox catalog web services at http://musicimage.xboxlive.com  (probably former Zune services).  It does not use folder.jpg, nor does it use ZuneArt_{guid}.jpg.  It will only access the web services if you have the Media Info option set to automatically retrieve missing metadata.

In order to get good album artwork, your metadata needs to be pristine and has to match what’s in the xbox metadata services.  That’s unreliable at best.  The best solution is to embed your artwork in your media files.

If you want to wipe out your library and rebuild from scratch, you can rename the randomly-(or not)-named folder in:

%userprofile%\AppData\Local\Packages\Microsoft.ZuneMusic_8wekyb3d8bbwe\LocalState\Database

Why wipe out the library?  If you have a massive amount of artwork that originally was downloaded from the xbox services and now you have all that artwork in your media files, it’s faster to let Groove rescan your library than to update each album’s artwork.

As Groove is collecting the album art, it keeps a cache of the images in:

%userprofile%\AppData\Local\Packages\Microsoft.ZuneMusic_8wekyb3d8bbwe\LocalState\ImageStore

It doesn’t seem to matter if the image comes from the file or from the web services.

Artist Images

How, when and where does Groove get its artist info?

Although the ID3 specification does provide a way to embed an artist image, there isn’t currently a common utility that lets you do it.  Doesn’t matter anyway since Groove doesn’t read the metadata fro artist images.

Like the album art cache, artists are stored in:

%userprofile%\AppData\Local\Packages\Microsoft.ZuneMusic_8wekyb3d8bbwe\LocalState\ImageCache\20

And the way Groove gets the artist images is by searching the music.xbox.com site based on your metadata.  If there’s no match,m or if music.xbox.com doesn’t have that artist in the marketplace, that’s about where it ends.  There doesn’t appear to be a way to override this yet.