Log in

10 entrées précédentes

07 juin 2008

purple WH


I have abandoned livejournal to run things myself at hoenir.himinbi.org.

I regret the loss of the privacy protections, but wanted to explore the medium a bit more than livejournal would let me.

If you're interested in still keeping track of me (though I'm currently being a bit more professional in my writing), I'd recommend using a RSS feed reader. I'm using Google Reader to follow about two dozen blogs (including all my livejournal friends) and it does a good job of letting me manage everything.

15 mar 2008

purple WH


I've been contemplating and discussing the idea of a pervasive recommender system. I'm find most people find it creepy moreso than intriguing. I don't know if it would always have been like this, but particularly in this modern age of corporate malfeasance and secret government torture facilities people just don't trust organizations to act responsibly.

I'll be honest that I don't necessarily like it all that much myself. It kinda makes me skirmish to guess how much Google knows about me.

Privacy has been called the ability to pretend to be like everyone else. As a researcher I want as much raw data as possible. What steps can be taken to make people feel comfortable?

Two main things come to mind:

  1. Allow a complete separation of a person's recreational and professional personas. If I'm a respected writer by day and dress up in a bunny suit for sexual pleasure at night, privacy is the capacity to keep those two lives separate.
  2. Give the user complete control over their data. At any time if I can see the specific information I have entered into the system from any source and I should be able to modify or at least delete it. If there are derived attributes that is the organizations discretion to disclose. Some dating sites create an emotional stability score. They don't tell members because having a computer tell you you're unstable is slightly insulting to many.

14 mar 2008

purple WH


As I was preparing the last post I looked up a link for my Chase Freedom credit card.

The basic idea is that you get the usual 1% cash back on most stuff, and for you top three spending categories you get a bonus 3%. In their description they say: "3% at the top 3 types of places you spend the most each month. There are 15 categories, more than any other card!"

I wonder what percentage of people will realize that the larger the number of categories the smaller the number of things in each division will be. That exclamation point threw me for a minute. It seemed like they were excited about it so surely I should be as well. I think they're just excited because they now have to give me less money. ☺
purple WH

Junk Mail

I have a Chase Freedom credit card. I use it to buy most of my stuff and the balance is automatically paid in whole from my HSBC Direct savings account. I get a bit of cashback and it keeps a larger balance in my savings account. Both of these are negligible bits of money, but who am I do turn down free money?

The one thing that has been irritating me is the huge amount of crap I get in the mail from Chase. They like those promotions which say, "Here's a check for $20 just for being you (btw, cashing this check enrolls you in our $15/month identity protection program)."

I sent them a message today saying, "Please leave me alone Chase. I'm tired of having to destroy all the things you send me that are much more attractive to thieves than they are to me."

They didn't entirely agree to it, but they said they'd leave me alone for five years or until I changed my address. Not perfect, but I reckon I'll take what I can get.

13 mar 2008

purple WH

The Role of Recommendations

So, on blogging, I was reading an post by Kurt Jacobson on avoiding mob rule in determining music quality. I have thoughts on the issue and want to have a conversation. I don't actually know the guy, so through the medium of blogging we just both shout our ideas out into the ether.

The basic idea he's hitting on is social proof. It's essentially the fact (and I say fact because there's enough research to make it pretty much indisputable) that human perception and preference is highly malleable, and one of the biggest influences is what they see other people doing.

In music recommendation this played out in a recent study on recommendations in an artificial culture. One group was given a bunch of songs and asked to rank them. Another group was given the same task, but they were given feedback on how much other people liked the same songs.

The results from the first group were fairly consistent. This suggests that there are some inherent qualities of songs that are consistent across listeners. (The test was conducted on a "teen-interest" site, so there was probably some perceptual preference similarity among samples.) With the introduction of social influence, scores became much more erratic and predictability dropped.

Jacobson's argument is essentially Britney Spears is crap and if we let the mob decide what good music is, we'll have a sort of dysgenic collapse as the masses all follow each other into a collective mediocracy.

There's certainly something to that argument. The consistency in preferences without social influence says that there is some sort of inherent quality to music. (Whether this quality would hold up under a more diverse sample population is debatable.) If I am wanting to recommend a song to someone who has none of the social feedback, I am going to want to use the consistent sample set because I can predict with less error that they're going to like the song.

The idea though that technology should somehow try to correct for humanity's herding instinct is philosophically dangerous and ultimately counter productive. Most people don't want you to correct for their weaknesses. They want you to give them what they want. In fact, the bulk of people would find the technologist's assertion that they need their preferences accommodated for slightly insulting.

One of the things I would have liked to see in the study was means for the data. They put data in terms of market share which is interesting, but I would like to know if the average rating was higher. There was the recent study on how a $2.50 placebo is 25% more effective than a $.10 one. I wouldn't be surprised if the user enjoyment of music with high community approval was higher than when the song was heard in isolation. It isn't just that people take cues from their environment, but that their actual experience is different.

So instead of trying to somehow fight against social proof, instead use it and try to introduce systemic constraints to encourage novelty.

Imagine a site where people nominate songs for song of the day. A song of the day can never have been a song of the day before. (Or to have more variety, disallow the top 10 each day from future consideration.) The criteria for deciding the song for the day is based on votes from all the users. There's some sort of mechanism where the more often a user is correct the more their opinion counts. I'd have to think on the details, but you essentially develop experts within a particular genre whose natural preference profiles are very similar to a stereotype. This would bias the results toward reproducibility.

The details would need some thinking, but the basic idea is you want to use social proof to increase a user's enjoyment while somehow encouraging the introduction of novel ideas. There's already a hugely implemented system dealing with this exact problem: the stock market. You want to find a resource before anyone else knows about it (find novel music) and after you've invested in it, you want to see it perform as well as possible (have as many people rank it as possible). I think a site based around that concept would be interesting.

I don't disagree that there is such a thing as great art. I'm just not convinced that computers are capable of recognizing that on their own. The best that we can do is set things up so that hopefully the human capability to recognize beauty is captured as much as possible and leveraged.
purple WH


So, I'm lining up summer plans and looking to do something music informatics.

One of the front runners for places I would like to end up is at Sun interning with Project Aura (which is a meld of Sun's Search Inside the Music and Advanced Search Technology).

I had a phone interview earlier this week and one of the questions was the specific areas I'm interested in investigating. I attempted a literatue survey and discovered the field of MIR is large and active. As I was contemplating the question of the most interesting parts last night, I remembered a list of MIR research blogs.

I'm realizing the extent to which the internet is growing to play a role in certain areas of research. My adviser was saying it can take up to a year to get a paper from submission to publication. Blogs in particular let a community of researchers keep up with each other. From reading the MIR blogs, I can't imagine a researcher really being current in the field without paying at least a little bit of attention to them.

I've been wanting to set up a feed aggregator. I figured why not use this as an opportunity to deepen my relationship with my gracious electronic overlord. So far I've been pretty happy with Google Reader. I understand blogging a bit better both as a community building tool, but also as a medium for communication.

I have a bit on blogs, MIR and technologists, but I'm going to split it out into another entry

28 fév 2008

purple WH


This is just a brief note to any command line users such as myself to remember the handiness of the little utility dc.

I was wandering the web and came across an estimate of basal metabolism. Now I know such thinks are affected by fitness level and whatnot, but I wanted to know quickly what the result was and I wanted to see all the numbers at once so I could know if I was copying them correctly. On the command line, this is just:

echo 195 2.2 / 11.6 \* 879 + p | dc

Where 195 is my weight, 2.2 is the pounds in a kilogram and 11.6 and 879 are the line parameters.

As I look at this I start to recognize that most people would find calculator more intuitive. ☺ This is for the geekier among you.


06 fév 2008

purple WH

Why I Like MusicBrainz

Recent years have seen an increasing amount of interest in music recommendations. Music is being created and published at a rate that far outstrips the ability of any listener to keep up with. How then, if I'm an artist, can I get my awesome new song to the ears of listeners? Or, if I'm a music connoisseur, how can I sort through the thousands of songs to find the few I'll love?

The oldest recommendation system is still going strong: word of mouth. "Dude, you've got to hear this…" will always be a vector for music to spread. Another old-school system is one where I pick a bunch of songs I like and send them out over the airways to anyone who would like to hear. The internet is changing the ballgame, however. The advent of computers, in addition to fundamentally altering how people create and distribute music, also promises to change how they discover it.

Some of the earliest systems were what are known as "collaborative filters." A computer system, like Amazon, knows thousands of albums that various people have purchased. Assuming that people liked the albums they bought, if Bob and I both bought ten of the same albums, when Bob buys an 11th, there's a better than average chance that I'll like that album as well.

Systems like Pandora work in a similar way, but not only have they broken things down from albums to individual songs, they also look at specific characteristics of songs. So, now the system can say, not only, "people who liked the songs you like also liked…," but also, "you have liked 9 of 10 R&B songs I played for you with melodic tones and three male singers, here's another song that sounds like those."

Pandora's knowledge of music comes from the Music Genome Project. Human listeners have spent countless hours reviewing "tens of thousands of artists" and cataloging God-only-knows how many characteristics. That's cool, but not really an undertaking the average person has time for, so there's quite a bit of research into how to get computer programs to listen to the songs for us.

Systems employ a variety of different types of signal analyses, most taking into account human perceptive characteristics (we are more sensitive to high frequency noises than low ones). They not only attempt to determine traditional characteristics such as beats per minute, key, and timing; they also represent "timbral characteristics," like the signal strength 3-4kHz, that human listener would never use to describe music.

There's quite a bit of research in this area including the International Conferences on Music Information Retrieval and Related Activities, the Laboratory for the Recognition and Organization of Speech and Audio at Columbia, and the Music Technology Group at the Universitat Pompeu Fabra. Research is also driven by a growing rapidly group of private commercial interests too numerous to list.

The more interesting developments for me, a solo programmer and graduate student in an unrelated field, are open-source projects. These are areas where I can take part and help create something interesting. One of the more promising is Paul Lamere's Search Inside the Music project with Sun Microsystems.

Which brings us to a crucial issue for every computer program ever written: data. Systems like Last.fm will integrate with your music player so that every song you play goes toward helping the system build a profile of preferences. Accessing actual songs to play with spectral analysis on a large scale is a bit more complex. The options are fourfold:

  1. Make a deal with the record companies gaining some sort of license — Unlikely
  2. Buy tens of thousands of songs online — I'm poor
  3. Use creative commons sources such as Magnatune — Limits selection
  4. Steal as much music as the internet will allow — Illegal

Each of these has advantages, but ultimately with all of them I run out of space to store all these songs. Also, part of the point is community building, and the point at which I try to open the system to other people (other than with the creative commons sources), I'm in legal trouble. What would really be nice is if I had an online system that I could query about the characteristics of songs (or a database of info I could download). The tricky part is building that database.

In the last couple weeks, a couple systems have come online (EchoNest & Vienna MIR) that let you send in a song and get back an XML description of the audio characteristics. That's a fine idea, but it limits the songs I can do computations on to songs I have a copy of. It also means I need to transfer whole files over which means if I want to experiment on a 5,000 song set, I need to start about a week early just so I can send all my songs over.

Right now I'm listening to The Nature by Talib Kweli. If someone else has already analyzed this song, I don't need to send the song over. I just need to tell the system I want The Nature and it can send me the data I want. This saves both the time to transfer the song and the time to analyze it.

Now the problem becomes how to tell the system what song I want. For many rap and folk bands there are often myriad versions of a particular song. Lots of new XML based technologies use URIs, and that makes sense, but how to make a unique one for each song? Enter MusicBrainz. They are a huge (more than 6,300,000 tracks and growing) community maintained database of songs, and they've assigned a unique identifier to each one. If I know the MusicBrainz id for a particular song then I can refer unambiguously about it to anyone else.

MusicBrainz has some limitations that would eventually be a problem. Mostly that they are "album-based" meaning they don't deal with DJ mixes or bootlegs. Also so far as a completely open process, they use MusicDNS which is partially encumbered. That would matter eventually, but those concerns are outweighed by the huge amount of data and community support you'd get.

There may well be other good solutions to the data problem. I like the MusicBrainz idea because it gives a project like Search Inside the Music a pre-built community and support structure. It would also draw users to help build MusicBrainz as a resource for both tagging and music recommendation applications.

purple WH


I went with the_archange1 to see Uncounted a couple nights ago. The thesis of the movie is that modern electronic voting machines are not only open to hacking, but there is extremely strong evidence that voter fraud has been committed in several elections including the 2004 Bush/Kerry presidential election.

Honestly, when are conspiracy theories ever true? Their case looked pretty solid though.

One of the more interesting statistics were anomalies in undervoting. An undervote is when a voter comes to the poll, votes in some races, but expresses no preference in others. This makes sense at times. Yesterday I could easily not have had an opinion on who the Democrat candidate for city council should be.

In presidential elections there tends to be an undervote of 1-3%. In New Mexico in 2004 in certain Native American and Hispanic precincts there were undervotes of up to 80%. 4 out of 5 people went to the poll and just didn't have an opinion on who the President should be. It's particularly interesting that it happened in Democratic strongholds.

There was also an analysis of a comparison between the exit polls and the actual results. I'm actually in a stats class right now and I have a newfound respect for the degree to which studies and statistical analysis have a serious mathematical basis. The issue wasn't simply that the polling data disagreed with the result. There could have been some unseen factor to push error beyond any previous amount of error they've seen. It was that over and over the errors favored Bush.

There's quite a bit more about things like the suspicious proportioning of voting machines that had primarily low-income minorities standing in line to vote in Ohio for up to 12 hours. I'd highly recommend watching the movie if you have a chance.

If you happen to live in Tennessee, you can support SB1363/HB1256 which would remove Tennessee from the 12 states that allow voting machines with no paper trail whatsoever. (With no paper trail the vote count is whatever the machine tells you it is and there is no method for verification. If nothing else, anyone who has dealt with a computer should know better than to do this. Absolutely no programmer writes bug-free code all the time.)

Even if you don't think there's been a conspiracy, there's no reason not to prevent one in the future.

26 jan 2008

purple WH


This is complete geekitude, so feel free to disregard…

I want to take a DVD, stick it in the drive and at some point have the smallest file possible that contains as much of the original structure as possible.

There are a few problems with most of how this is done most of the time. When you download a video off the internet there are several parts that go into that file. There's the sound and the video streams and then a container that joins them all together. The container syncs the streams and says, this second of audio goes with this second of video.

The most common container format in use currently is AVI from Microsoft. AVI is a fairly old format and it has several limitations. One of the most important for my purposes is it pairs one video with one audio track. That's fine when you're encoding your home movies, but it can't handle dubbing and director's commentaries for DVDs.

There are a couple new formats to consider, but the one I like the most is Matroska. It's flexible and supports not only the multiple audio streams but also multiple subtitles.

Inside the container there are even more standards at work. The raw data for audio waveforms and video images is huge. Different compressions schemes, called codecs, are used to compress the data and make it take up less space. They operate in a variety of ways, but ultimately the goal is a tradeoff between the degradation of the image or sound quality and the size of the output file.

Audio compression has seen a bit of improvement over the old workhorse MP3, but the real improvements in technology in recent years has been in the realm of video compression. H.264 is able to produce a file that is nearly visually identical to the MPEG-2 used on DVDs in fraction of the space.

There are some programs, notably HandBrake, that do a pretty good job of doing what I want. Before HandBrake was released however, I started on a ripping program of my own. I've been playing with it for a bit and this morning I had a bit of a breakthrough.

When I list the contents of the DVD, it tells me in general that the video is in NTSC. That makes sense, since NTSC is what we use here in America. The encoder was complaining though about dropping frames which shouldn't happen if I was encoding at the right rate.

It turns out that though NTSC is 60000/1001 frames per second and interlaced every other frame NTSC is 30000/1001, movies are filmed at 24000/1001 and that is how they're stored on progressive DVDs. In running my encoding previously at 29.97 and ending up with a file around 1500mb. I switched to 23.976 and the file size dropped by 500mb.

10 entrées précédentes