A fine power control policy and beam alignment is required between the base station (BS) and user equipment (UE) to achieve the promising performance of massive multiple input multiple output (MIMO) in millimeter wave (mmWave) communications. However, obtaining the channel state information (CSI) of mmWave – massive MIMO systems is challenging. In this paper, the beam-steering technique is used to estimate the signal strength from the BS to the user. We propose a novel learning framework to determine the suitable beam for a specific user and the transmit power for minimizing the cost including the transmit power and the unsatisfied rate when the channel is unknown. In addition, we address the missing data problem, and then employ the long-short term memory (LSTM) on the temporal processed inputs to select the suitable beam. Furthermore, we design a learning agent to predict the proper transmit power from the transmitted SSBs taking into account the required transmission rate. We then validate the proposed learning framework on the Deep MIMO dataset constructed based on accurate ray-tracing channels. Numerical results show our proposed framework outperforms the state-of-the-art prediction strategies, and approximates the best performance which is obtained when the CSI is available.