SCRUM driven Drupal development
I'm kinda assuming you already know what Scrum is. There is a lot of buzz about scrum over the past year, including discussion on howto fit development of drupal 8 itself using an agile methodology. Other projects, like Aegir, have weekly scrums you can read online. And there are numerous presentations on drupalcamps and drupalcons about the topic. You can find a lot of resources about scrum online and I gathered some of the better Drupal related ones in the footer of this article.
Scrum all boils down to release complete software in short cycles (sprints) and prioritizing using business value. Those features who return the highest return on investestment get higher on the list. The development team ideally consists of allrounders who work closely together with the client and have short daily scrum standups in which everybody involved tells the other what he did yesterday, what he will be doing tomorrow and if there are things which block his/her progress.
There are a lot of other details and rules involving the Scrum process, and here is a good presentation which describes the whole process from a Drupal development point of view.
We at Merge are currently working together with ezCompany on a large Drupal project using scrum, and have almost finished our second sprint. Overall I'm very positive about Scrum and will try to eplain why I think Scrum fits larger/more complex Drupal projects.
The scope of larger Drupal projects tends to change along the way
I think most current web development studios started out using a waterfall process for their projects: quoting fixed price and trying to specify every aspect of the entire project in a (huge) technical document, wireframes and designs.
The problem with this approach and (drupal) web development is that for larger, more complex projects the entire scope of the project is unclear. There often are a lot of uncertainties and dependencies for functionality down the road.
Planning is guessing
In a waterfall process, deadlines are set in the far future. Sometimes 2-3 months. Most of the time this is just a wild guess, based on previous projects and experience. Clients demand deadlines but often fail to meet them themselves. Using scrum a deadline is never further then 3 weeks away. And you can oversee things a lot better for such a timespan.
Also, related to planning, the typical "scrumboard" (see picture) may seem old fashioned in this digital age, but I find it to be very helpfull. Glance at it and you will have an immediate overview of the status of the current sprint.
Release quick, release often
Every sprint, a complete shippable product is released. Something that is finished and actually usable. This is probably one of the best things about scrum: iterative development. We use GIT as a version control system and developers push code to a test server. At the end of each sprint, a demo (and retrospective) is performed and the client can accept (or reject) the result on a acceptation environment.
Designers are part of the team
We sometimes had projects where at the start of "our part" of the project - building it - the company who did the projects is already working on something else and not available for iterations. (or unwilling because they already got paid for what they delivered). In Scrum, ideally, the (UX) designers are part of the team and only design the stuff which is needed for every iteration.
Designers tend to have poor knowledge of the inner workings of Drupal - but we are experts. And scrum allows us to easily share this knowledge to the designers. There are a lot of design patterns in Drupal, and designers shouldn't have to reinvent the wheel if a good solution is already available.
Great communications and feeling of teamwork
The daily standup is short and sweet. Everybody gets to tell what's done and it helps keeping everybody on the same page. Thanks to this daily standup, nasty surprises at the end of a sprint are impossible. The whole process is super-transparant and if somebody has got a problem with something, others can quickly help out. Another benefit of this is everybody get to learn from each other (ideally you want allroudners on your scrum team, not specialists).
The client will know what to expect thanks to the sprint planning (poker planning) at the beginning of each sprint (which includes the client) and the demo's, retrospective and many contact moments,.
Sounds great, so what are the downsides?
While scrum is great, some things are harder. One of the biggest hurdles is convincing the client to use scrum. For clients who are unfamiliar with scrum it may sound really scary if they are used to waterfall. Another issue is the transition from waterfall. You will need to spend resources to learn and implement scrum in your team. And not everyone might be easily convinced. Also, people tend to stick with old habits and scrum requires quite a new mindset. We prepared by all reading a (thin) book about scrum and having someone over with a lot of experience with scrum (thanks @dickstegeman!).