A startup is a company that is in the first stage of its operations. These companies are often initially bankrolled by their entrepreneurial founders as they attempt to capitalize on developing a product or service for which they believe there is a demand. It is a young company founded by one or more entrepreneurs in order to develop a unique product or service and bring it to market. By its nature, the typical startup tends to be a shoestring operation, with initial funding from the founders or their families.
Age, profitability, and etc may not be the right criteria for defining a startup. People tend to think that startups are pretty young, not-so-profitable and pretty "fun" overall. However, these are completely arbitrary and only because a large number of tech startups (which tend to be covered extensively by the press) are like that. It isn't by pure coincidence that most of these startups have similar traits but by no means do they define a startup.
Paul Graham: A startup is a company designed to grow fast.
Notice that both these definitions are not limited to tech companies. If you open a new chain of cafeteria and want to grow as fast as possible, then that cafeteria is a startup. If you open a cafeteria with no plans to open another one and are satisfied with the current state of affairs with your current cafeteria, then it's not a startup.
In my view, both the definitions are highly related. You aren't going to grow quickly if you don't have a scalable business model and the startup's goal is to "exit" from the startup stage as soon as possible. Every startup has to do a ton of soul searching. They are always trying out which product/service/feature is going to generate them the maximum revenue and allow them to stay afloat for the next decade (or just the next year). A startup fails at the point when it doesn't discover a scalable business model before it runs out of cash to support the ship.
Even Instagram is a startup! It had a high growth but it was still searching for a scalable business model.