Growing Venture Solutions - GVS - conference organizing distribution

Selecting conference session proposals: popular vote? selection committee?

Greg — Wed, 03 Aug 2011 22:32:43 +0000

I was on the "Ecosystem" track session selection team for Drupalcon London, which motivated me to finally do some more analysis on the traditional pre-selection session voting. Specifically, I wanted to compare the votes a session receives against the evaluations submitted after the conference.

By the way, if you have the opportunity, I highly suggest going to a Drupalcon; they are always great events.

Here are some conclusions based on analysis of the evaluation and voting data from DrupalCon Chicago:

Voting was not a useful predictor of high quality sessions!
The pre-selected sessions did not fare better in terms of evaluation than the other sessions (though they may have served a secondary goal of getting attendees to sign up earlier).
We should re-evaluate how we do panels. They tend to get lower scores in the evaluation.
The number of evaluations submitted increased 10% compared to San Francisco, which seems great (Larry Garfield theorizes it is related to the mobile app, I think there are a lot of factors involved)

Is voting a good way to judge conference session submissions?

Drupalcon has historically used a voting and committee system for session selection that is pretty common. This is also the default workflow for sites based on the Conference Organizing Distribution.

Typical system:

Users register on the site
They propose sessions (and usually there is a session submission cutoff date before voting)
Voting begins: people (sometimes registered users, sometimes limited to attendees) can vote on their favorite sessions
During steps 2 and 3, a session selection committee is encouraging submissions and contacting the session proposers to improve their session descriptions
Selection begins: Voting closes and the session selection committee does their best to choose the right sessions based on factors like appropriateness of content to the audience, the number of votes, their knowledge of the presenter's skill, diversity of ideas
???
Profit

Drupalcon Chicago (the event I'm basing this analysis on) had a few changes to that model. They pre-selected some sessions from the people they knew would submit sessions and get accepted (see their blog post on that and the faq). This allows us to see whether pre-selecting actually brought in sessions that were more valuable to people which seems like a decent proxy for whether or not the committee's choices are right.

The pre-conference voting had 5 stars with the following labels:

I have no interest in this session
I would probably not attend this session
I might attend this session
I would probably attend this session
I totally want to see this session

The post-session evaluations had 5 stars with the following criteria:

Overall evaluation of this session
Speaker's ability to control discussions and keep session moving
Speaker's knowledge of topic
Speaker's presentation skills
Content of speaker's slides/visual aids

I've previously looked at the percent of the attendee population that actually gets to vote and the distribution of votes (1 to 5) to see if that was actually used in a meaningful way in Chicago (that analysis is on groups.drupal.org). Given the distribution of votes in Chicago across the entire 1 to 5 spectrum, I believe it is useful to use a 5 star system as a rating on a session. However, I don't think the resulting value is directly useful by the session selection committee when they choose individual sessions (more on that later).

My analysis method was to create a nice spreadsheet with the average and count of votes on sessions from the pre-conference period where votes were used to help determine which sessions to include. Then I added in the votes (from 1 to 5 stars) which covered several categories.

I graphed the pre-conference votes compared to the post-conference evaluations and used the "correl" function to see how correlated the data is. I expect a straight line correlation: the higher the average votes, the higher the post-conference evaluation scores. In fact, there was basically no correlation.

What I found was that there is basically no correlation between the pre-conference voting and the post-session evaluations. Here is a table that shows the axis (i.e. one of those 5 elements above) and the correlation between that axis and pre-conference session votes.

Axis	Correlation (r)
overall	-0.00615455064217481
control	0.0528419859853818
knowledge	0.0907826506020892
presentation	0.00493457701973411
visuals	-0.0216904506593498

As a graph, the overall data looks like:

This data is not correlated. Just look at it, spaghetti soup!

I graphed it along with a random line that has a correlation value of .95. As you can see, the overall evaluation is not at all correlated to the outcome evaluations.

It isn't surprising that votes don't correlate to session quality. Voting tends to be done by a minority of event attendees who are "insiders" to the event. They are likely to be swayed by friendships, employers, and social media campaigns.

Comparing pre-selected sessions to regular sessions

I also took an average of the evaluation scores across non-pre-selected-sessions and the pre-selected sessions. The average overall evaluation score for non-pre-selected sessions was 80.9 vs. 80.7 for pre-selected sessions. The other axes show similar results except for knowledge and visuals, though it's not clear if those are statistically significant.

Axis	Pre-selected average evaluation score	Non-pre-selected average evaluation score
overall	81	81
control	83	83
knowledge	93	91
presentation	80	81
visuals	78	75

So, we can see that regularly selected sessions got very similar scores to the pre-selected ones. I'm not suggesting that pre-selecting is flawed (it didn't produce lower results, anyway), but I do think we should carefully consider who we pre-select.

The third bit of analysis I did was to look at overall score, and the number of presenters for that session. Here's the average per decile where decile 1 is the 9 sessions that were ranked highest. Seems like a pretty clear trend from nearly 1 person for the top rated sessions to 2.5 people for the bottom rated sessions.

Average # of presenters	Decile
1.11	1
1.67	2
1.89	3
1.44	4
1.67	5
2.33	6
2.00	7
2.00	8
2.44	9
2.50	10

I believe there are two big reasons for this. First, panel presentations are rarely done in a well-coordinated manner and the panel members usually don't take time to practice as a group (our distributed community makes that hard). Second, Drupalcon session selection committees often suggest similar topics get merged into one panel. I think we should stop merging independent presenters. The result is often that people who may not have the same story to tell end up putting 45 minutes of information into one-half or one-third of the time.

What can we do to improve session quality and session selection?

One of the great tools for session selection committee members at Drupalcon London was the availability of evaluation data from previous conferences. If a proposed session got a lot of votes (perhaps due to a campaign on twitter or within a large company) but the presenter had horrible evaluations from a previous conference then the evaluator has an easy job: just say "no thanks".

The only problem with using previous conference evaluations to judge sessions is that it can lead to stagnation among the presenters. Part of the value of a conference is in hearing new ideas. This can be reduced by having free-for-all BOF sessions, but I think in the Drupal world that part of the solution is to use Drupalcamps as a ramp into Drupalcon: presenters should give their session at a camp and mention that (and any evaluations from the camp, any video from the camp) in their session proposal. With approval from presenters, Drupalcamp Colorado published our evaluations - we hope this helps other camps and that they will do the same. It's not surprise that some feature requests for COD will help make the process of gathering this information and getting it to the right people much easier.

See also a great discussion on groups.drupal.org: On popular voting and merit-based selection of sessions.

What else can improve session quality?

So far I've talked about identifying good sessions, but I think the nature is more complex. It's also about encouraging and inspiring the presenters to do great work on their sessions. We can tell them "please practice it 10 times" but nobody will do it if they aren't motivated. Sending reminder mails to presenters like "we expect 3,000 attendees including key decision makers from companies like Humongo Inc." could help. There's also the possibility of compensating presenters. Drupalcon Chicago gave a mix of cash and non-cash benefits (massage chair, faster check-in line).

Scott Berkun gives some tips on how to improve the presenter experience at a conference in An open letter to conference organizers. He recommends a lot of things including sharing the results of the evaluation data. I'm in favor of that as well...(provide default terms of attendance).

Extra note: Want to see your evaluations from Chicago? Just needs more code

There were evaluations in Chicago, but the speakers have not seen this data. I got access to it as part of my role on the London session selection team and my work on the infrastructure team/Chicago sites.

However, the fact that presenters can't see it is a result of a bug in software that you can help fix. The organizers of Drupalcon want to share that information, but the code to do that isn't fully working. If you can help make it work then all session presenters will be able to see their evaluations.

COD Beta1 Released, Packed with Features

Ezra — Wed, 13 Jul 2011 18:27:06 +0000

We're excited to announce that COD Beta1 has been released and is loaded with great new features, including:

Enhanced conference administration menu
Automated sponsorship sales
Birds of a Feather scheduling tool
Automated speaker confirmation and contact
More granular control over the event registration workflow
Ability to collect profile information for free events
Session editing for multiple speakers
Improved attendee check-in
Better Ubercart reporting for purchased registrations
Integration with RegOnline and Etouches

Marketing, beauty, the senses, and Agile at Big Apple Redux 2011

lisa — Thu, 02 Jun 2011 01:14:27 +0000

In May, I had the pleasure of attending the IXDA NYC's "redUX", showcasing some of the talks from Interaction 11 conference earlier this year (where possible, I've linked to the video from the Interaction 11 session).

Ray DeLaPena (@rayraydel) did an excellent writeup of the talks in Big Apple Redux Recap, so I'm going to focus on the talks that I could relate to the most (and they were the ones where I captured the most notes).

Marketing is not a 4 Letter Word by Megan Grocki

[video]

Many folks, particularly designers and developers, have a negative reaction to the word "marketing".

Megan says, "Marketing is like matchmaking". You want people to know about your work, your portfolio, your company or your product. It's a nice little ecosystem where something that is well designed gets promoted, and then you're more likely to get more work like it.

Who is doing marketing right? The Disney experience is immersive and enchanting and when you get home, they remind you to book your next trip! Netflix has great branding and also immersive. It's pretty easy to spend ages in their site rating, reviewing and being shown suggestions for things you might like. I agree with both those examples. Megan also cited Zipcar as a product with excellent user experience.

During Megan's talks, it really dawned on me that at GVS, we don't spend much time marketing ourselves, particularly our work on event and conference sites, yet we should. Given the choice, we've always done client work and improvements to the Conference Organizing Distribution (COD) itself, rather than putting portfolios and case studies together. But we are focusing on conference and event websites, and we need to raise the awareness our focus and abilities to potential clients. And those clients always help fund direct improvements to COD.

Therefore, I'm glad to say we've got several new conference project listings in our portfolio.

Beautiful Interactions: Codifying Aesthetics in Interaction Design

by Callie Neylan [video]

Callie told us that beauty is visceral, behavioral and reflective. Beauty is symmetry and patterns and mathematics. It's the Fibonacci sequence and golden ratio. Humans have evolved to find these patterns beautiful. The new Twitter site layout was based on the Golden Ratio.

Vision is the dominant sense, but interactions and experience design aren't just limited to visuals either. Touch is the other primary interaction that we design for. Buy there's audio: pink noise is filtration of white noise. It helps humans get into a flow. And smell. Did you know, the Katy Perry's CD Teenage Dream was scented with cotton candy, as will her upcoming concert tour (if anyone goes, please let me know if it enhanced the experience!).

Humans have a preference for patterns. Beautiful relationships are honesty, trust, support, communication and humor. These are all working within a healthy relationship. Callie then went on to rank the beauty of Seattle and Baltimore based on all of these criteria.

I love her quote “Interaction designers are mixtures of aesthetic equalizer".

An image that embodies marketing, Drupal and the senses touch, sight and smell! Photo by @rupl

Agile’s Secret Step: Discovery (and Planning!)

by Lis Hubert [slides]

In Agile, or any other project management system, the first step is the discovery and planning phase.
But in Agile, teams don't always know where UX fits in. In waterfall, UX had it's own "design" phase.

Lis pointed out the hard truth: UX slows down Agile, but without it, we don't know who users are, how they need to use it, etc. She says, the problem is we're not using Agile correctly, and used the assembly-line analogy: there is a backlog and there is a priority.

During discovery, you discover the what. You conduct user research, business research. You put the product roadmap in priority order, you plan, analyze and prioritize continuously. Discovery is figuring out what to build in what order. Discovery and planning often happens outside of Agile.

Usability testing decreases risk.

Agile, yay!

This talk resonated with me because we (GVS) are Agile as much as possible. Even when clients think it's totally weird, and make us do tons of documentation upfront, we always end up adopting some Agile techniques. Early feedback is a critical one. During the requirements phase, it's nearly impossible for clients and development team to think of every single scenario, and we'd rather not. We'd rather start building, getting feedback, tweaking, building, feedback, etc.

We're building UX into projects. I recently had pleasure of interviewing 14 readers of the product we were converting to Drupal. It was invaluable in understanding their unique needs and desires. Many of the readers are 40+ years old, and less confident about interacting with the social web (specifically, they were nervous about posting comments!) The product is now 90% built, and I'm ready to start usability testing. I hope that we have designed a reassuring interface.

Conclusion

For a one day mini conference, this was $30 very well spent. I met loads of people in the NYC interaction design community, and several of them were familiar with Drupal. I really value and appreciate time spent with smart folks!

Additionally, local Drupal groups should consider hosting their own DrupalCon Redux. Invite a few speakers from a recent DrupalCon, and let your local community hear some of the great sessions live. And hey, you can use COD to put together your Redux site!

Branding Exercise Leaves Fish Dead, Owl Satisfied, Drupalers sad

Ezra — Fri, 01 Apr 2011 15:24:20 +0000

A routine branding exercise went awry early Friday in Brooklyn when Growing Venture Solutions performed a "mind map" exercise to aid in the creation of logos for two of their flagship products, the Scout hosted sercurity review service and COD, the Conference Organizing Distribution for Drupal.

"Connecting to the unintellectualized, visceral, gut responses we get from each logo helps us maximize branding potential so that we can produce marketing collateral that's sure to engage members of our target market segments, helping them to connect to each brand at an emotional level, which results in increased conversions" said social media expert Robert H. McJellyPants. He added, "Tachyon converter beam subspace electron resonance tuning."

Unfortunately, while exercise participants were discussing their friendly, communal associations with schools of smiling fish in the COD logo, as well as some of the more stern, defensive associations evoked by the Scout owl logo, the owl took flight and picked up the fish, instantly crushing the fish's vital internal organs with its beak.

Illustration by Carl Wiedemann.

Needless to say, participants were horrified at the sudden attack by the owl but impressed by its swift, decisive action in the face of what the owl saw as a potential security risk to its personal website, SupercuteCatsWearingWigsandSmallDressesTailoredEspciallyforCatsNoThisisnotajoke.com/website.

McJellypants expressed concern, saying "This is highly unusual for a branding exercise, and the attack could be a step in the wrong direction for both brands. It may cause people to construe the owl as reckless and aggressive, rather than defensive and wise." He pointed out that "[i]t makes the COD logo more similar to the previous one, which was also a dead fish," referring to the public domain image of an Atlantic cod that previously represented the Conference Organizing Distribution.

COD has been used to power many feature-rich conference websites for DrupalCamps, the recent DrupalCon Chicago conference, as well as non-Drupal events in the United States, India and Australia.

When asked to justify the attack, the owl provided participants with helpful information about the specific security vulnerability it claimed was presented by the fish. However, for more information, it directed them to its "Scout Automated Plus" and "Scout Enterprise" solutions, where Drupal security experts explain potential vulnerabilities on specific sites—and the steps to mediate them—in great detail.

Despite having clear expertise in Drupal security, it was reported in several tweets on Friday that the owl's personal site had the full HTML input format enabled for anonymous commenters early Friday morning. The issue appears to be resolved as of the publication of this article, though one comment on the site by "IttyBittyPrettyWittyKittyCommitteeinNewYorkCitySingingaDitty36" appeared to show a properly escaped cross-site scripting attack probe, and read, <script>alert('XSS Vulnerable Meeeeow');</script>

When reached for comment, the owl tried to bite off my face.

Two of the mind-map exercise attendees posted a re-enactment of the event to YouTube. Note: the mind map was done on a boat.

DrupalCon Copenhagen wrapup and recovery

lisa — Tue, 14 Sep 2010 19:49:55 +0000

This year's European Drupal conference has sadly come and gone. Ezra Gildesgame, Ben Jeavons, Lisa Rex and Carl Weidemann were in Copenhagen representing Growing Venture Solutions. Here are a few of our highlights. Links to the videos will be added when they are available.

Theme Preprocess Functions

Carl presented Theme Preprocess Functions: an Introduction, where he showed a packed room how to bring and flexibility into the hands of the themer. Preprocess functions are a key component of Drupal theming and when first learned, open up a whole new world of possibility. Carl received some positive feedback on the session and hopes to continue to give it at future conferences, adapting it to stay current with future versions of Drupal.

COD (Conference Organizing Distribution)

Ezra (our COD ringleader) and Ben co-presented Building Conference & Event Websites in Drupal with COD, with Lisa there to lend support. COD is a Drupal distrubution for building event websites.

The primary out-of-the-box benefits of using COD are:

Proposing and voting on sessions
Setting a session schedule for the event and for each attendee
Making it easy for attendees to register, pay and provide profile information
Managing a waiting list of attendees
Collecting and displaying sponsor information

Drupal Security

Ben and Gábor Hojtsy co-presented Drupal Security - Configuration and Process. The session covered:

Common vulnerabilities and what they mean
Drupal defaults and why you should care
Trusting site visitors
How to recover from attack
Is Drupal secure?

Drupal.org Redesign

Lisa co-presented The Future of Drupal.org with fellow project manager Kieran Lal, former Blue Cheese theme lead Todd Nienkerk and the Drupal.org redesign technical architect Neil Drumm. They covered:

Why Drupal.org needed a redesign
How the Drupal.org redesign project is also a redevelopment, and the technical challenges we face
What project management strategies worked, and what didn't, and what we're learning along the way
Why the Drupal Association had to hire people
Overview of our contributors

Scrum and Drupal at the Economist

Ezra co-presented On the importance of DONE: Scrum and Drupal at the Economist with the lovely Rob Purdie, SCRUM Master at The Economist in New York City. Topics covered include:

Introduction to Scrum
Why a complete, shared Definition of Done is critical
Benefits & challenges of having 3 teams working in 3 timezones
Launchpad and Bazaar workflow for code review and management, pushing configuration changes to production
Measuring progress
Getting products 100% Done

Watch the Economist Scrum session video

Views Basics

In Views: The Key to the Drupal Castle, this informative beginners' session, Ezra showed attendees how to create:

Dynamic photo galleries based on a tag
A Map of user locations
A list of blog posts sorted by post date and more

Watch the Views session video

Taking it for the team

On Tuesday, in addition to co-presenting The Future of Drupal.org, Lisa was involved in the Redesign BOF, and appeared in the live Day One wrap up on Drupalradar.

On Wednesday, Ezra was involved in a record three sessions in a single afternoon. Can you say extreme?

The post-conference sprint was quite productive. Carl focused on Drupal.org redesign theming and Lisa contributed more Drupal.org redesign content-related issues and direction. Ben worked with Jakub Suchy and Gerhard Killesreiter on updates to security.drupal.org.

OH: What's funkplaza

You also may have seen tweets with the hashtag #funkplaza. Let us explain. Our wifi account in our apartment was creatively named funkplaza (the original name from Ezra's personal blog), and it quickly became the name for the two apartments that GVS was sharing.

What's next

Post-conference, Ben went to Stockholm, Carl went to Berlin and Lisa went to Tallinn (that's Estonia). Ezra flew home to attend a special concert -- details and link to review forthcoming.

Once we're back to routine, expect more great things from GVS.

Drupalcamp Atlanta and best practices for event websites in Drupal

Greg — Thu, 02 Sep 2010 00:58:11 +0000

Drupalcamp Atlanta recently launched their new site for the 2010 Camp. I reviewed some of their features and found it to be a solid site. As the GVS team increases our focus on selling events with Drupal, I wanted to hear more about the backend of the site. So, I got in touch with Brent Ratliff who was the lead developer on the site, to find out how the site was built.

Adding Signups and Payment to an Event Site: UC Signup

In 2009, the site was a standard build relying mostly on Views and content types. For 2010 they needed all of the features from the previous year but also needed to charge an entry fee for the event, for individual sponsorships, as well as the ability to submit and vote on sessions. The Atlanta Drupal Users Group decided to base their 2010 site on the acclaimed 2009 Drupalcamp LA site that was released as a zip file. The LA code, updated with new module releases, handled the voting, some nice Views, helpful theme functions, context, and some of the "attendee logic," but not the e-commerce portion. Brent was familiar with using the UC Node Checkout module for building an event site, but ultimately decided to go with the Ubercart Signup integration module along with Rules and custom hooks to handle workflow. UC_Signup allows them to collect profile information from users during checkout and helps keep track of attendees using the Signup module. They also grant roles to users based on which products they have purchased.

Panels vs. Context? Both

On the topic of Panels vs. Context, Brent said he follows the practice of using both modules. Context is responsible for block placement on the site and changing the active menu item on some pages of the site to give a more consistent feeling as you navigate around. Panels is responsible for the main "content" area in a few parts of the site, allowing the site to pull in Views and other content into complex landing pages without having to write much code. Arguments can also easily be passed from the Panel directly to the Views and the Panels can be exported along with features for version control while remaining theme independent. Panels also allow Skinr styles to be applied to content via the UI and for content editors to easily add or remove relevant components as the camp timeline changes, such as removing session voting and adding a session videos after the camp. One example of this is the sponsors page built with four Views embedded into a Panel.

Call to Action on Views for Usability

They are using some PHP in the Views headers and block bodies to present appropriate call-to-action blocks to site visitors. For example, on the sessions view, anonymous users see a suggestion to register for the camp. If you are logged in and registered you'll see a request to "submit a session". The same is true for the voting widgets and any other privileged callouts that require the "attendee" role, triggered by a successful Ubercart checkout. In the LA camp site, as it was a free event, any registered user had these abilities.

Performance and troubles with Pressflow

The Drupalcamp Atlanta site is using Pressflow and Varnish to maintain a fast response time for their visitors. In fact, Brent is giving a presentation on Using Pressflow and Mercury to achieve a super fast site. However, Pressflow caused some problems with the Ubercart shopping cart block so Brent had to use some custom code and the Ajax cart module to let anonymous users see the cart contents.

Schedule of Events: Always a challenge

One area where they want to do more is the Schedule. Currently it's just a Google Spreadsheet embedded into the page, but ideally it could be built based on Views. As part of GVS work on the COD - the Conference Organizing Distrubution distribution, we will be releasing a Views+CCK based schedule builder in the next few weeks.

Conference Organizing Distribution vs. Build your own

When I asked Brent about why they didn't use COD he said it was simply a lack of knowledge of COD's availability. He said they based some of the site on the features from Drupalcamp Colorado and just didn't know that they had been packaged up into the COD_Support features.

For the future, though, he said he would definitely base his work on the COD and COD_Support packages since they are a modular, features-based system that make it easy to pick and choose the parts you need and then contribute back enhancements, while maintaining the ability to modify workflow through customization. Brent also feels it's much better using a distribution where you start off with a fresh installation and turning on compartmentalized features as you need them rather than using a database export which may have cruft in it.

A few weeks after our conversation, Brent got back in touch with me to say:

We're running into groups who want one account to set up
multiple attendees. Live and learn. Take it as validation to the COD
approach and uc_signup.

That was a feature that GVS built into UC Signup from the beginning and which was used on the Drupalcamp Colorado site by several large companies, one of which purchased tickets for their 30 attendees all at once. Brent disabled the feature at the cart level to make their site simpler.

Sharing Events Best Practices

It's great to hear about best practices in building event management sites and to share those ideas with other camp builders.

Last weekend I participated in a call with the fun folks from LA Drupal which is now available: LA Drupal Podcast 8: Rocking Drupal where we discussed the future of camps vs. summits vs. cons and the idea of a distribution like COD.
This coming Wednesday, September 8th, GVS will be hosting a Free COD support hour in IRC (#drupal-cod) and Skype. This is just the first of what will hopefully be many events where Drupal site builders can get together to share tips and tricks and get support.