With the current interface of Serato DJ, the recommended display resolution is 1280x720. With the release of Mac's Retina displays, these computers are capable of resolutions upwards of 2560×1600, from 400%-1400% of the recommended display.
If you do run Serato DJ using the largest scale resolution of a Retina screen:
Serato DJ may appear as follows:
To avoid this, we would recommend setting your display to the 3rd resolution available:
With Serato DJ then showing as follows:
Now you've got Serato DJ looking good, you can now tweak the Library Text Size (to personal taste), through the "Library & Display" tab of the Setup screen.