Context. The context the word is used in makes a huge difference. I don't feel that the word "trap" (or in this case more the term) in and of itself in a vacuum is offensive, but in the right context it can be.
Obviously a word like trap already has a "normal" irl definition, that, while the internet term is based off of, does mean something else. This already shows context is important. From this point onwards, I'm just going to refer to the internet term.
Calling a male feminine/androgynous character that dresses like a girl in anime/manga/other media that self identifies as male a trap I think is fine. They are fictional idividuals that are designed to fit this "made up" term. While it's origins are fetish based, I know a few of them have seeped into stuff whoose primary purpose is not this. Still, I consider traps to be a made up term to describe made up people, and even if the media they are in doesn't play up the fetish aspect, its still an overtone.
The closest I would consider anyone in real life to a trap would be people who are nearly always cross-dressing (of course females in traditionally male clothing isn't genreally seen as cross dressing, but most male looking characters who were really female in anime/manga/other media I also don't see being called traps, at most, occasionally "reverse trap". This is starting to get into another topic though). Even then, cross dressing is an already existing term that exist explicitly to describe this exact thing and, while is certainly has connotations around it many find negative, the same could be said of any term for any group. Cross dressing would at least be an accurate term and does not have the connotations as fictional manga/anime fetish bait like trap does.
As for context and how it can be considered offensive depending on it, how about something else as an example. Jewish is a term that is used to describe followers of Judaism, and belifs or objects relating to the religion. Additionally, unlike many other religions there is also racial component. Still, in this context, using it has no problem. However, most people online know of "Happy Merchamt" and other anti-Semitic jokes that abound. There are people who use the term "Jewish" or simmilar terms to describe anything that fits these anti-Semitic stereotypes. Someone describing a thing as "Jewish" it could be that it does actually have something to do with Judaism. However, it's possible the person holds bigoted stereotypes about Jews and considers said thing to fit those same steryotypes, using it as a derogatory term.
Using trap to refer to trans people is using a "made up" term to describe fictional characters of one definition in a largly sexual context, and ascribing it to real people who do not meet this definition. I can guarantee trans people ( or every real person for that matter) have more complicated lives that just being print on paper for people to get off to.
Sure, some people in certain communities do have positive opinions on traps, and I have no doubts that at least one trans individual at some point in time didn't mind having the term trap ascribed to them, in the same way some black people personally don't mind if anybody wants to use the n-word. I would never say that this is anything close to a large number of people, but people are individuals. By far, the vast majority of people do not like being referred to by a term that has inaccurate and generally negative connotations.