To sell to the federal government, you must be registered in the System for Award Management (SAM). There are a number of requirements in the SAM system.
You must have been in business for at least 2 years.
You must have a DUNS number to complete the process. You can obtain one for free from Dun and Bradstreet if you are getting it specifically to bid for government contracts.

Small business specialists for federal agencies

Government agencies provide P-Cards or Purchasing Cards to their employees for expenditures under $2,500. If you would like to accept P-Cards, you must be able to handle Level III credit card processing.

Indian Incentive Program when selling to the Department of Defense
Once You Have a Contract:
Each contract will have its own requirements. Be sure you understand them before you get started and then honor those requirements. Some examples:
If your contract requires a certified payroll statement, you can see that form here.
Federal prime contractors with more than 50 employees must also file an annual EEOC (Equal Employment Opportunity Commission) report.
Federal contractors must also post additional employee posters. Click here for the posters.