I'm not (yet) sensitive to poison ivy. When I was a toddler I'd go out in the back yard and crawl around in heaps of it and come back inside and my mom's arms would just about fall off from her reaction to me.
So, yeah, it could totally be what he's sleeping in or wandering through.