You make a valid point. Yes, St. James was around the time of St. Peter. However, as I have outlined to Angelus, the earliest form of confirmation - before oils were introduced - was the laying of hands. It was the Church - the Apostles - who decided that chrism would be used as the "matter" instead.
Well, we are talking about the Last Rites re the link you provided on the new sacrament of anointing of the sick, not confirmation. You are trying to tie two different sacraments to the same exception. I opine that if the Church still had the Apostles as bishops instead of just their successors, then the imposition of hands would still be the matter.
But you have to realize that when the Church spoke is when the Council of Florence decreed the matter to be olive oil, then the matter is olive oil for all time. When that same council decreed if olive oil is lacking that the sacrament is not effected, then without olive oil the sacrament is not effected for all time. This stands forever, nobody can modify or in any way change this. Any more than anyone can change the matter of chrism in Confirmation, or water in baptism.
The Church does not change the matter of her sacraments, the new church does this. We had our Pentecost, that was when the Church was born, the birthday of the Church. The conciliar church had their Pentecost at V2, that's when their church was born. The conciliar church is built on change, the Catholic Church is built on a rock - it don't change for nuthin' or nobody.