Secrecy is a long standing tradition in the martial arts. When you analyze how most arts developed, it's easy to understand why.
Take kenjutsu (sword arts) of the Samurai for example. A kenjutsu headmaster was in charge of training his disciples with the deadliest skills he could. Those men would then take their techniques to the battlefield and fight for their daimyo. Unfortunately, during the Warring States Period, backstabbing and side switching was so common it was almost unremarkable. The daimyo changed allegiances depending on what elevated their status or saved them from annihilation. They also intermarried among clans, which shifted the power of alliances. The Samurai were often doomed to follow along, even fighting against clans they once considered allies.
These switches could happen in a hurry, even in the middle of a war. The outcome of one of the most famous battles in Japanese history, the Battle of Sekigahara, was significantly influenced by a few properly persuaded commanders right before the battle took place.
Imagine now if a skilled kenjutsu instructor was wide open with his teachings, sharing all he knew with anyone who came to him. How soon would that information be used against him?
This secretive behavior persisted even after the Samurai were less needed on the battlefield. Sword schools transitioned from focus on large battles to more individual development. It was at this time musha shugyo became more popular. Roaming Samurai desired to test their skills against headmasters and make a name for themselves. If a headmaster was loose with his knowledge and word of his tactics got out, it could mean an early demise.
Karate and kobudo also posses a history of secrecy. After the invasion of the Satsuma warriors, quite a bit of tode and kobujutsu training was conducted away from prying eyes. The Satsuma had roving metsuke (armed informants) who would be all too happy too report or outright extinguish suspicious behavior.
It wasn't just threat of death that kept the lips of instructor's tightly sealed. Both Japan and Okinawa developed under Confucian philosophy. In Confucianism, there is little tolerance for questioning of those in higher stations of respect than oneself. Further, it is expected of seniors to maintain a certain level of aloofness above those below them. The result was a lot of show with little tell.
Our modern era has it's own excuses for secrecy. Luckily the amount of "roving fighters" has gone down (but not disappeared thanks to the everpresence of ego), so it's not a particular need for tactical secrecy that keeps teachers quiet. More often than not, it is a need to retain students. If a modern teacher with limited skillset teaches everything they know, then students will inevitably grow tired of the training. Some may have the natural ability to surpass the teacher and go beyond their lessons, but then the instructor may become concerned about the student opening a school nearby and taking students away (or just becoming better, which hurts the ego). One safe bet to keep students interested and at a comparatively low skill level is to institute varying degrees of witholdance and secrecy.
The Wayward Student
There's one reason for concealment that we haven't discussed, and it's perhaps the most important. Whether in the early days of karate or today in a modern dojo, the possibility of "wayward students" exists. Despite a teacher's experience and intuition, some people are capable of hiding what's in their hearts. On the surface a student might seem dedicated and cautious and honorable, but deep down they could be manipulative and devious. Some of the most important martial artists in recorded history have fallen prey to charasmatic "disciples" who have siphoned power and influence from them.
One or two of these deep burns is enough to make any teacher pack his/her belt away for awhile.
Aside from the political aspects, teaching a wayward student the most debilitating, deadly, and effective aspects of an art is not only regrettable but dangerous to society as a whole.
Truly, wouldn't it be safer just to teach the bare basics and not risk it?
Giving the Bleeding Edge
The task of finding an honorable student and helping them to higher levels is a monumental task. After getting to a certain point, the instinct and tradition of letting things "coast" is very strong. After all, it's been done that way for generations.
However, in this modern era, it is appropriate (and even vital) for teachers to draw students to the edge of their own understanding***.
Today, it is very unlikely that martial artists of the same school will ever appear on opposite sides of a battlefield. Nor is it likely they will face other martial artists who have received inside information about their tactics. Instead, students are probably going to encounter street violence. Technique foresight is not an issue; the training will work or not work based on quality, not on secrecy.
Therefore, it is the duty of an instructor to give all they can in order to help students protect themselves and their loved ones. After years of getting to know each pupil the instructor can make an educated decision about sharing deeper knowledge in the hopes the student will carry on the desired "way".
Teaching to the very edge of skill level is also critical for an instructor's growth. "Toping out" in regards to skill and knowledge while strategically keeping students at a lower level is a very effective way to never grow. If, however, a teacher actively pulls students up as far as he/she can, the teacher will then be challenged to improve even more in order to continue sharing and helping the students.
Personally speaking, a lot of new ideas and developments in my own training begin as feelings. I sense that something is coming within reach and manifesting into potential improvement. However, it isn't until I attempt to verbalize and demonstrate what I'm thinking that it takes ahold in my own skillset and becomes available to me in a more complete way.
This very blog post is part of that process. If I don't truly understand it, then I can't help others understand it. So in order for me to understand it, you have to understand it.
See, now you know all my secrets! (or do you??).
***(please keep in mind I write this as an instructor with limited understanding and experience. Kyoshi and Hanshi level practitioners may possess a different perspective given their experience, but I do currently believe this advice applies forever in one's training).