Four carboxylic acids with different acidities and chemical structures were applied to produce carboxylic cellulose nanocrystals (CNCs) with the assistance of ultrasonication. The effects of acid concentration, hydrolysis time, acidity (pKa), and chemical structure on the preparation of CNCs and their properties were systematically evaluated and discussed to preliminarily investigate the mechanisms influencing the preparation process and properties of the CNCs. Without the introduction of any strong mineral acids, a high yield of up to 60% of pure carboxylic CNCs with good nanoscales has been successfully produced. Acidity played a key role mainly in the yield and crystalline degree of CNCs. Stronger carboxylic acid (lower pKa), which can hydrolyze the amorphous part of cellulose more efficiently, prepared CNCs with a higher yield and crystalline degree. Other than acidity, the chemical structure of carboxylic acids was also a very important factor that clearly affected the particle size and carboxylic group contents of CNCs, correspondingly affecting their thermal and dispersion stabilities. This work further developed a green and efficient method to produce carboxylic CNCs without addressing environmental and economic problematic issues. The findings presented in this work not only provide a proper guideline in selecting appropriate carboxylic acid for CNC preparation but also a preliminary suggestion of the mechanisms that were possibly involved in the whole CNC preparation processes. The work built a fundamental for further deeper studies about the mechanisms related to CNC preparation using carboxylic acids.