You said DH works across the street--is it a street that is easy to cross? Could your DS get to him quickly? I think it depends a lot on the neighborhood and the kid's responsibility. I don't remember when I was left alone, but I started babysitting in 6th grade, so it was before that. I also remember that since early elementary I could recite what to do in case of a fire, stranger, someone comes to the door, etc.
Actually, now that I think about it, it was probably about 3rd or 4th grade that I was left alone for small periods---my mom was at a neighbor's house (but we had a big neighborhood, so that put her a bit away, not just next door) and I was home alone with my younger brother. We had a strict No Answering Doors rule. So when the doorbell rings, my brother and I went into Commando Mode. We were in the back of the house and got on our stomachs and were inching across the kitchen floor to try to get to a position to see who it was. It ended up being a neighbor, who went away when no one answered. We were so proud of our recon moves!