Budgets, taxes, and debt are the fundamental cycle of government finance. Expenditures are financed through taxes or debt. When spending exceeds tax revenue, the budget deficit must be bridged by borrowing, increasing the national debt.
Conversely, when tax revenue is more than the expenditure, as was the case in Kenya in 2004, you have a budget surplus. The surplus can retire some existing debt, or finance new expenditure. A balanced budget is achieved when total inflows equal total outflows.