– Due to the ever-increasing volume and complexity of digital transactions, financial fraud represents a difficult problem for FinTech platforms. Imbalanced dataset, high-dimensional features, and change of feedback immediately in fraudulent trends aspects are prevalent challenges with current detection methods. This proposed study will introduce a GA RF hybrid model to improve feature selection and hyperparameter optimisation in order to perform more accurate fraud detection. The Genetic Algorithm selects the informative subsets of features and optimal Random Forest parameters, whereas classifiers like Random Forest use ensemble learning for regularized predictions. We used publicly available Kaggle datasets for performing the experiments, such as IEEE-CIS Fraud Detection, PaySim and BankSim with millions of simulated & real transactions. The hybrid approach was shown to outperform the baseline models—Logistic Regression, SVM, Decision Tree, and Xgboost—with an accuracy of 99.3%, precision of 98.7%, recall of 97.9% F1-score: 98.3% and AUC-ROC: 0.996 These results illustrate the framework's ability to mitigate class imbalance, reduce the number of false positives, and improve model interpretability.