David Bowie II (classifying his songs)

last article in my blog was dedicated to David Bowie.

I want to extend it searching relationships in his songs using as a features just his lyrics. Using ununstructured text analysis tools to find the distance between songs (I will use Euclidean distances).

Background

We have lyrics of 254 songs.

Like in last article, we have to convert lyrics to terms, discarding, if possible, the differences between plurals and verbs.

Términos

First of all, we create a word list to discard. Articles, pronoums, ...

i, me, my, myself, we, our, ours, ourselves, you, your, yours, yourself, yourselves, he, him, his, himself, she, her, hers, herself, it, its, itself, they, them, their, theirs, themselves, what, which, who, whom, this, that, these, those, am, is, are, was, were, be, been, being, have, has, had, having, do, does, did, doing, a, an, the, and, but, if, or, because, as, until, while, of, at, by, for, with, about, against, between, into, through, during, before, after, above, below, to, from, up, down, in, out, on, off, over, under, again, further, then, once, here, there, when, where, why, how, all, any, both, each, few, more, most, other, some, such, no, nor, not, only, own, same, so, than, too, very, s, t, can, will, just, don, should, now, d, ll, m, o, re, ve, y, ain, aren, couldn, didn, doesn, hadn, hasn, haven, isn, ma, mightn, mustn, needn, shan, shouldn, wasn, weren, won, wouldn, chorus, 'm, 's, 'll, n't, 'd, ca, wo, oh, yeah, x2.

Then we tokenize al lyrics, e.g., Blackstar lyrics:

in the villa of ormen in the villa of ormen stands a solitary candle ah-ah ah-ah in the centre of it all in the centre of it all your eyes on the day of execution on the day of execution only women kneel and smile ah-ah ah-ah at the centre of it all at the centre of it all your eyes your eyes ah-ah-ah ah-ah-ah in the villa of ormen in the villa of ormen stands a solitary candle ah-ah ah-ah in the centre of it all in the centre of it all your eyes ah-ah-ah something happened on the day he died spirit rose a metre and stepped aside somebody else took his place and bravely cried i’m a blackstar i’m a blackstar how many times does an angel fall how many people lie instead of talking tall he trod on sacred ground he cried loud into the crowd i’m a blackstar i’m a blackstar i’m not a gangster i can’t answer why i’m a blackstar just go with me i’m not a filmstar i’m-a take you home i’m a blackstar take your passport and shoes i’m not a popstar and your sedatives boo i’m a blackstar you’re a flash in the pan i’m not a marvel star i’m the great i am i’m a blackstar i’m a blackstar way up oh honey i’ve got game i see right so white so open-heart it’s pain i want eagles in my daydreams diamonds in my eyes i’m a blackstar i’m a blackstar something happened on the day he died spirit rose a metre and stepped aside somebody else took his place and bravely cried i’m a blackstar i’m a star star i’m a blackstar i can’t answer why i’m not a gangster but i can tell you how i’m not a flam star we were born upside-down i’m a star star born the wrong way ‘round i’m not a white star i’m a blackstar i’m not a gangster i’m a blackstar i’m a blackstar i’m not a pornstar i’m not a wandering star i’m a blackstar i’m a blackstar in the villa of ormen stands a solitary candle ah-ah ah-ah at the centre of it all your eyes on the day of execution only women kneel and smile ah-ah ah-ah at the centre of it all your eyes your eyes ah-ah-ah

Now we supress diferences between similar words, discarded words, etc. e.g. Blackstar:

villa ormen villa ormen stand solitari candl ah-ah ah-ah centr centr eye day execut day execut women kneel smile ah-ah ah-ah centr centr eye eye ah-ah-ah ah-ah-ah villa ormen villa ormen stand solitari candl ah-ah ah-ah centr centr eye ah-ah-ah someth happen day die spirit rose metr step asid somebodi els took place brave cri i'm blackstar i'm blackstar mani time angel fall mani peopl lie instead talk tall trod sacr ground cri loud crowd i'm blackstar i'm blackstar i'm gangster can't answer i'm blackstar go i'm filmstar i'm-a take home i'm blackstar take passport shoe i'm popstar sedat boo i'm blackstar you'r flash pan i'm marvel star i'm great i'm blackstar i'm blackstar way honey i'v got game see right white open-heart it pain want eagl daydream diamond eye i'm blackstar i'm blackstar someth happen day die spirit rose metr step asid somebodi els took place brave cri i'm blackstar i'm star star i'm blackstar can't answer i'm gangster tell i'm flam star born upside-down i'm star star born wrong way round i'm white star i'm blackstar i'm gangster i'm blackstar i'm blackstar i'm pornstar i'm wander star i'm blackstar i'm blackstar villa ormen stand solitari candl ah-ah ah-ah centr eye day execut women kneel smile ah-ah ah-ah centr eye eye ah-ah-ah

We do this with all lyrics. At the end we select the list of more frequent words. We will measure the distance between songs using de frequency of words. These are the 30 most frequent words:

freq        tok      word
---------  -----  -------
16.082115   love     love
13.795538   like     like
11.673021   look     look
11.623957    got      got
11.104668   know     know
11.050255   time     time
10.515605    day      day
10.002770    say     says
 9.683665    eye     eyes
 9.519472  could    could
 9.511280    one      one
 9.362057  never    never
 9.320950    let  letting
 9.214944   come     come
 9.207492   want     want
 9.036289    get      get
 8.794052    man      man
 8.770926   life     life
 8.730971    boy      boy
 8.549849  thing   things
 8.469426     go       go
 8.214135    see      see
 8.118021   feel     feel
 8.037416   live    lives
 7.983238  littl   little
 7.935172   girl    girls
 7.839930   take     take
 7.751650   noth  nothing
 7.165395   make     make
 7.109152   well     well
(...)

We saw this frequent words with the wordcloud of the last article.

Now we generate the distance matrix between songs from the matrix of frequencies of terms and documents. With this distance matrix clusterize songs using the ward method, to form a tree that allows you to see the potential clusters.