Why Does Japan Accept So Few Refugees?

For a country with a negative birthrate and an aging population, it's a fair question to ask. There's always bound to be concerns about security, culture, etc. and I think these are valid concerns, but we're not talking about thousands of refugees here, we're talking about single digits. 

Also, I'd be remiss if I didn't point out, although Japan's treatment of asylum seekers seems harsher than is probably reasonable, it's clearly NOT Guantanamo Bay. Let's back peddle that just a tad.