Game of Thrones seasons

I am a fan of the series and am eager to see the story continue but one thing has annoyed me from the start.  How do these people keep track of time?  Winter has been coming for several years.  Without seasons, how does one know how long a year is?  There is much talk of something having happened in the time of the Andals or the Fall of Valeria but there is no associated year like we have.  Charlemagne ruled from 768 to 814 but Aegon the Conqueror invaded Westeros '300 hundred years ago.'  Really?  How do you know that?  Initially, I thought the series just wasn't going to bog down the viewer with made-up dates but, having read the first four books in the series, it isn't addressed there either.  Bran Stark was born after the last winter and is seven years old in the initial episode.  Again, how does this world keep track of time if the seasons come and go in an arbitrary fashion and last an uncertain number of years?  As far as fantasy calendars, I think only Tolkien really outlined a history of his world with specific years.  For instance, Bilbo Baggins was born in the 2,989th year of the Third Age, or the year 1389 by Shire Reckoning.  Such specific dates are never mentioned in any of the movies - as far as I recall - but it is there as a foundation and, as Middle Earth has ordinary seasons, such isn't really necessary.  My guess is that Martin began writing his epic and the issue of how years are determined didn't really come to play until he was too far along to rectify it.  Therefore, the issue is ignored.  The historian in me cringes at the absence of a dating system.