Review of progress on the website

By Ozzie Gooen · Published March 11th, 2014

NOTE: This piece is now out of date. More current information on our plans and impact can be found on our Evaluations page.

Summary

This document outlines the website updates from the period of August
2013 to January 2014. It’s part of our annual review, the rest of which will be released on the blog over the next month.

The most significant change was a site redesign,
followed by several new features such as a hierarchical categories
system, a research rating system, and newly designed pages.

Throughout this period we’ve decided on having a more focussed
brand and website in the future. In the next year, technical development
will primarily support website maintenance and organizational research.

Website Redesign

In early 2013, the 80,000 Hours team decided to redesign the brand and website. These were to be launched simultaneously for consistency. The team chose an initial completion date of approximately October 1st 2013, close to the start of the academic year in the UK.

The redesign was comprehensive, resulting in changes to the 80,000 Hours color scheme, logo, branding, and website. It was carried through and most of the intended changes were completed.

Important documents

Several redesign analysis documents were written by the 80,000 Hours team, and
other design documents were created by an external design team. Most of the analysis documents and select design documents are below.

Redesign goals

During the process of the redesign the team sought feedback upon what could be improved. Some specific areas of significant opportunity for improvement included the following:

The color scheme wasn’t seen as very professional or credible.
The website was relatively unresponsive to mobile devices.
The blog looked relatively ‘chaotic and disorganized’.
The organizational message or purpose was not incredibly clear.
There were several user interface bugs.
The codebase included lots of features that were not complete or were not getting attention.
The donate page was difficult to access.

Metric gathering

In order to focus the redesign to best fulfill the organizations’ purpose, we brainstormed on the most important priorities for 80,000 hours which could be assisted by the website. This resulted in a draft of possible metrics to ‘optimize’, in order of importance. We didn’t end up measuring these metrics, but they still helped provide focus.

Research Readership
- Hours spent reading blog articles
- Hours spent reading Quality Adjusted blog articles
- Minutes spent reading QABA (quality adjusted blog articles) per reader
Money Donated via Website
- Percentage of visitors who donate
- Average donation of donors
Case Study Requests
- Percent of visitors who apply
- Average quality of applications
Credibility
- Standardized survey with rubrics on Mechanical Turk users and website members
Mailing Lists
- Percentage of visitors who sign up
Feedback
- Percentage of visitors who post comment or idea
Social media sharing
- Viral Coefficient

We initially intended to measure these before and after the redesign. This didn’t happen because the team had a tight deadline, and this experiment had low decision value. It was unlikely that we would choose to revert the brand redesign, so measuring against the previous design at the time of change wasn’t important. The metrics could be useful to later optimize a design, rather than compare an old and completely different new design.

After the redesign was completed, we went back and implemented a few methods to gather metrics. These are discussed in the Metrics Gathering section.

Logo and color scheme

The original 80,000 Hours brand was primarily magenta and black. We decided to change to a color scheme with a more modern and professional look.

We hired two external designers to design a new logo, brand, and create
important design elements. Color scheme experimentation began with the
logo redesign, which went through dozens of
iterations. the 80,000 hours team worked with them and decided upon different shades of green and blue. once a logo design was chosen, we viewed it in approximately 10 different gradients between blue and teal. pure blue was considered professional but boring, while teal was seen as playful but less professional. eventually we decided on a color in between blue and teal for our logo.

Old 80,000 Hours Logo

New 80,000 Hours Logo (Basic Version)

When designing the website, we tried out and liked a blue-green
gradient. This was copied for the logo on the website.

Mobile optimization

The first website was poorly optimized for mobile devices. Simple tests and mobile analytics indicated that our homepage, main pages, and blog posts were all quite difficult to access. This was improved by upgrading from using Twitter Bootstrap 2.0 to Twitter Bootstrap 3.0. Bootstrap 3.0 is a front end web framework with a mobile first fluid grid system, which was useful for making the website adjustable to mobile devices.

Old website simulated on mobile devices

New website simulated on mobile devices

Social network changes

Originally 80,000 Hours considered focussing on the social network as part of its main strategy. During the time of the redesign, this focus had shifted towards research and coaching. A strong social network thus made sense in the beginning of 80,000 Hours, but made less sense during the time of redesign.

The social network required a large codebase and would have been time consuming to both redesign and maintain. Parts of it were breaking and it took significant maintenance to keep running well. The social network made up over ½ of the entire codebase, so it could have easily taken ½ of all maintenance time. Maintenance time took a total of approximately one full a day a week by a good developer, occasionally more when large issues would arise.

Few of the social network features received much use. In the month
of August 2013, the forum experienced only 4 posts and the donation
tracker was used by 4 people. A few members reported that they found
and contacted other members through the service, but this was only a
small aspect of the site. See our code complexity
document
for more information on this issue.

80,000 Hours website database schema with social network components highlighted (from our code complexity document)

Because of the costs, we decided to substantially reduce the social network functionality. We removed the user forum, user comments, user votes, user sign-up, and user donation tracking.

The custom commenting system was ported to Disqus. Pictures were lost in the process from users with email addresses not corresponding to Gravitar profiles. Our previous comment system gave us some flexibility and an enhanced karma system, but was frustrating to maintain. The Disqus system requires significantly less maintenance and is more usable. The script used to convert our comment data to the Disqus importable format is here.

User up votes and the existing Karma system were removed. The codebase
for these was quite complex, and we decided that similar information
would come from social media buttons and Google analytics view count data. Later we added a quality rating system as well.

The donation tracking system and forum were both removed without replacements. In the future we intend to establish another forum using existing software like Facebook Groups or Google Groups. Giving What We Can has created a donation tracker recently and we will encourage our audience to use it.

Profiles and user sign-up were both put on hold, meaning they were not
updated or linked to from the main website. These pages are still
accessible via their URLs. We temporarily replaced the sign up process
with a Google form for email addresses (over 100 have been collected so far). At the time of change we intended to re-release the profiles within a few weeks, with a scaled-back social network.

This change has not yet been attempted. We are currently considering if and when we want to re-launch a simplified version of member profiles.

Static page removal

We redesigned navigation to be simpler in preparation of new content. In doing so we removed links to old or obsolete pages.

Previously these static pages made up much of the website outside of the blog. These pages were useful to provide broad advice that wouldn’t change over time. However, the organizational strategy and advice had changed significantly since they were written. Many of these pages proved difficult to maintain. Much of the content was outdated or incomplete.

In the future we intend to bring back content and add more to our website on careers and causes. We also intend to remove all the dead pages (ones that aren’t linked to from other sources) that we decide not to bring back.

Some static pages that we no longer support (and will be either
re-integrating or deleting) include:

Our Findings
Which Careers Make the Most Difference?
Planning Your Career
Cause Effectiveness
Recommendations
What Should I Read if I’m New?
Causes
Jobs with 80,000 Hours
Volunteering
Happiness and Your Career
Research Overview
Index of Career Resources
Rules of Thumb
Career Strategy
Types of Careers
- Research
- Innovating
- Improving
- Advocacy
- Donating
- Volunteering
- Global Poverty
- Future Generations
- Scientific Research
- High Impact Research
Career Profiles
Career Advice
Our Features

All of our static pages, including these old ones, are saved in Markdown and PDF. You can see these here.

Completion success

The team successfully changed the logo, changed the color scheme,
changed the website design, and removed static pages. We did not
refactor the social network, fully complete the home page design, or complete
the blog categories changes within the intended period of the redesign
(approximately August 2013 until October 2013). Overall we were happy
with the progress and resulting feedback.

Anecdotal results

The team received received several comments about the website, almost all of which were mostly positive or very positive. The praise was primarily towards our brand, web design, and mobile compatibility.

There were a few specific complaints about the logo and mobile functionality. In addition, 5-15 people mentioned that they were disappointed that the user profiles functionality was not included in the revision. We heard one complaint that the static pages were removed. Several specific bugs and issues were brought up on our new uservoice page, and these have since been addressed.

Statistical results

It’s very difficult to accurately estimate the impact of the redesign on
our key web metrics.

Originally we expected that the redesign would come with positive impacts on the average visit duration, bounce rate, and number of visits. In practice it was difficult to both improve these metrics and measure if we were doing so.

Full Site Traffic & Average Visit Duration, Jan 2013 to Feb 2014

The graph above shows the 80,000 Hours visits and average visit duration per month from January 2013 to February 2014. The website redesign launched in early October 2014, and work continued until December 2013.

There was a large surge of traffic in May and June 2013, much in response to Peter Singer’s TED presentation on Effective Altruism. This increase in visits seemed to have a corresponding increase in average visit duration, time on page, and bounce rate. This may have been because many of these users were new users, skewing the balance of total users to new people, who on average stayed on the website more. This traffic may have been particularly high-relevance for our brand, which could have resulted in a heightening of these statistics.

There was a small spike of additional traffic once the new website redesign was launched in early October, followed by a minor decline in traffic after October. It seems like the greatest predictive factor in traffic variation has historically been the popularity of our new blog posts month to month. November and December were relatively slow months for blog posts, which is one reason why traffic may have declined in this time. We also lost some traffic because of an issue with organic users (explained later in this document) during the month of December.

Average visit duration, average pages per visit, and average bounce rate
did not appear to change significantly after the redesign launch.
One reason for this may have been the removal of static pages and social
network aspects. This loss in content could have countered the benefits in other sections of the website.

Home Page Average Time on Page and Bounce Rate

We were able to notice differences to specific pages, specifically on the home page. The graph above shows the average time on page and bounce rate for the homepage over this same time period. There appears to be definite changes in October 2013 and onwards, approximately the period of the redesign. The average time on page declined, and the bounce rate declined. The old homepage featured a 7-minute video, while the new homepage featured text and icons with a much cleaner layout. We think that the old homepage had a high average time on page because users would often watch the video, while the new homepage instead is cleaner and seems to direct people to other parts of the website more (hence a lower bounce rate and exit rate). On the whole, this seems quite positive.

Metrics gathering

We wanted to continuously generate metrics for long term website
optimization. The technical ideal was one automatic system which would
update counts for dozens of important metrics in real time. After
doing some study on similar systems, it seemed like this required a data
warehouse. Unfortunately data warehouses were too expensive to be practical.

One alternative was to create simple dashboards for the main metrics.
We created a research
dashboard for quick
reference for the use of our blog articles, which are the most
time-intensive and important feature of the website. Similar dashboards
may later be created for other metrics, although many of the important
metrics not on this dashboard exist in other places. For example,
Google Analytics stores our web traffic data, and it would be
complicated to import much of this to the 80,000 Hours blog.

We decided to manually query and record data each week for several
different variables. The weekly website
metrics
includes the number of case study applicants per week, the total visits
and average time spent on site per week, and the total visits and
average time spent on the the blog per week. These helped us
understand responses to our blog posts, and occationally indicated
large problems.

Categories system

Analysis indicated that user time on site was not correlated with the
number of old blog posts. Most blog posts over two weeks old were
getting very little traffic. This was frustrating because many of them
were still relevant. Our blog acts as a research repository that holds
almost all of our work.

Total website visits and average visit duration from original site
launch (Oct 2011) to now (Feb 2014).

The graph above shows our traffic in the last two years. Note that the
average visit duration doesn’t seem to increase after August 2012. This
was strange because the total number of blog posts increased linearly.
This trend was true even for new visitors. We did not measure
engagement metrics like user retention rates and average total lifespan,
which may be better indicators of content quality but were more difficult to analyze.

One reason for this lack of a correlation was that old blog posts were
difficult to find. We experienced this ourselves and have heard complaints
from others. We had a tagging system, but this became difficult to
us because our tags were not consistent between posts.

After some consideration we decided to implement a categories system for our blog posts and
categorize all of our previous posts and future. We decided to
categorize posts in two distinct ways, in ‘Types’ or the ‘types of content’, and
‘Categories’ or the ‘subject categories’. This has been implemented on
the website and now users can find all posts in certain
subjects much easier. However, there is still work to
do to clarify and encourage users to investigate more of our content.

Types

First, we came up with a list of ‘types’ of blog posts. Some of our posts were introductory content aimed at new readers, some were in-depth research pieces intended for dedicated users, and others were updates aimed at supporters who wanted to follow our organisational progress. The combination of these were confusing, so we sought to classify the articles into ‘types’. We decided on the types ‘Updates’, ‘Discussion’, ‘Research’, ‘Case Study’, and ‘Interview’. Articles would generally be only one type, but could technically be categorized with multiple types.

Navigation tree

We set up a navigation tree to our article categories and types on the top of the main blog page. When users click specific categories, the page filters for articles with that category, and the tree then shows subcategories of that category (with the corresponding number of posts in each). On the right of this is a selection tool to choose the article ‘type’ (in this case, ‘Interviews’).

The tree was useful for navigation by the team and for some people we talked to. However, it seemed like many others didn’t realize that it was usable in this way. We intend to later release updates to make this more understandable and usable.

Categories challenges

Our hierarchical solution is very flexible, but may be difficult to
maintain. We did not find a ruby gem for this purpose, so had to write
the hierarchal system using a custom Rails model. Making changes to the tree structure currently requires use of the ruby console instead of the administrative editor. If we eventually add images and descriptions to categories it would require extra Rails code and thus more complexity.

The existing hierarchical category system seems portable to other blog infrastructures, but future features may complicate this. WordPress 3.0+, for instance, does feature a robust hierarchical category system, but previous versions and other similar systems do not. It includes category fields like name and description, but not image.

Rating system

We created an online rating system for our blog content. A rubric was created for our articles, an online implementation was built to rate on our website, a dashboard was created to view all of the ratings, and we selected a small team of individuals to rate our content. The system was built in late October 2013 and has been in practice since.

The rubric system is under revision and will be iterated upon shortly. After this iteration, we plan to write a document specifically about the implementation and outcomes.

Issue: lost Google traffic

At the end of November 2013 we noticed a steep decline in our organic
traffic from Google. Organic traffic represented 40% of our total
traffic in 2013, so this was very significant. After spending a significant amount of time trying to understand the issue, we eventually fixed it.

Total and organic visits to our website from September 2013 to February
2014

It turned out that the Memcached add-on service we were using shut down. The presence of caching was temporarily removed from most of our website during the redesign, so we originally didn’t think this was an urgent issue. However, Mamcached was necessary for the websites’ robots.txt file to be accessed without a cookie to our website. Google was repeatedly having errors in their attempts to access our robots.txt file, and subsequently stopped indexing our website, leading to reduced traffic. Once the issue was diagnosed, we changed our Memcached service and got the robots.txt fully working again. Shortly afterwards our organic traffic went up to normal levels. This is showed by a steep dip in organic traffic in December 2013, followed by an increase in January 2014.

Other additions

We made several bug fixes and small improvements during the past 6 months. For a full list of all of the commits to do these, check out our Github page here. Some notable changes include:

A website SEO analysis
Redesigned the donations page
Redesigned the research page
Redesign of the coaching page
Added Google metadata to articles
Significant design improvements to blog index and blog pages
Fixed OG tags for Facebook
Set up duplicate homepage for Google Adwords
Social impact coaching application setup with Wufoo

Lessons learned

Feature creation time is often significantly underestimated

The expected costs of software development during this work have been substantially lower than the actual costs. If we would have understood the actual costs before deciding to implement features, we may have aimed for substantially less than we did.

For instance, the categories feature was expected to take approximately
5 days to implement. Instead it took around 7-12 full days of work,
which was spread out over 30 days due to coordination challenges.
Originally we did not intend for it to be hierarchical, but came to
realize that may be necessary during development. The rubric was also
supposed to take a week or two, but took most of one full month to fully
implement, including planning and technical development. Coming up with
a rubric took far more time than expected, and later we added more
functionality than we originally assumed we would need. Both features
eventually became better than originally intended, but took longer. We
initially underestimated the necessary functionality, and later experienced feature creep.

This suggested that innovative features are simply difficult to implement well. It may make sense to plan for less features in the future, and designate more time and attention to pursuing them.

The website should focus on assisting the research

When 80,000 Hours was established, there was uncertainty whether the organization should focus on community building, research, or coaching. For the next year, rather than directly aiming to build a community or primarily doing coaching, we have decided to focus on research. This is a flexible strategy because research could prepare us well for community building or other work later on.

The purpose of the website should be aligned with the purpose of the organization. This means that for the next year, if the organization will focus on research, the website should primarily assist the research.

The website could help research in several ways. The website could explain and increase transparency of our research process. Feedback collected through comments and ratings could be used to improve future posts. Perhaps most important, it should make this research accessible to readers, specifically those looking for thorough and innovative research. Overall, it seems like the website has the potential to help 80,000 Hours focus on developing high quality online content, and shouldn’t deviate far from that.

Future website metrics should focus on research

We tried several metrics strategies during the last six months. First we established a set of metrics for the redesign, as described in the redesign section. These were very extensive and unrealistic in implementation, but were useful to consider in order to align the goals of the website.

After the redesign the team set up a weekly metrics document that was manually completed each week. The most actionable thing learned from this document was that visits fell dramatically in December 2013, causing us to diagnose and fix the issue. Total website visits and average time on site didn’t seem highly correlated with other actionable things, except for writing good blog posts. There was surprisingly little correlation between weekly website visits and social impact coaching requests.

While the weekly website dashboard was nice to have, it’s not obvious how useful it was. The costs were relatively minor (10-50 minutes per team member per week), but the benefits also seemed relatively minor. Most of the website metrics came from Google Analytics, which is fairly easy to check on a weekly basis for important issues. In the beginning we assumed that website improvements would be very visible in user time on site, but in practice this was not the case. This may have been because relatively little time was spent improving the website design compared to other issues that came up (website repairs, coming up with a rubric, designing a category system, writing documentation, etc).

The website purpose and metrics should be reevaluated if 80,000 Hours focusses on research in the next year. It may not be worth doing heavy optimization on website readability if we have a relatively small audience and are primarily focussed on producing high quality research. The main uses of the website should be things that promote the research, but these can be fairly difficult to measure.

Rails works for now, but we should consider changing after several months

It is not clear if Rails is the best platform for the 80,000 Hours’ website, especially if the website will act primarily as a blog. Alternatives would include existing blogging platforms like WordPress and Drupal.

There are substantial costs to using Rails for our website. Rails blog content can be difficult to edit. Administrative editing takes customized tools, which are difficult to write and maintain (even with helpful gems like ActiveAdmin). We currently have custom methods for administrators to edit static pages and blog pages, but not to edit custom designed pages (like the home page or donations page).

Second, maintaining the site requires significant work. If we were using an open source solution, then bugs could be dealt with by the open source contributors. In this case much of our software is customized, so requires continue fixes from our team. Ozzie Gooen has estimated that the website could take 2-12 hours per week to maintain at normal use.

Third, Rails as a blog is less popular than blogging frameworks like WordPress. Rails programmers may be more difficult to find and/or expensive than those who can maintain a WordPress site or similar.

On the other hand, the use of Rails does present a few benefits. Our software can be heavily modified and extended to add social networking functionality or similar. Rails programmers will be better at working with Rails than other systems, and it seems like within the Effective Altruist community there are several people who know at least some Rails. It may also be that interns or hires would be more interested in using Rails than other systems, in part because they may be able to get more career capital from doing so.

Migrating our existing website to Wordpress or another system would take a lot of time, perhaps 1-4 full months. If the organization is still unsure what it will do in a year, it may be best to leave the system as is for now (with regular maintenance) and re-evaluate the situation later.

Our brand could become more focussed

The existing website was focussed on a ‘social entrepreneur’ brand that leaned towards business, international nonprofits, social entrepreneurship, and related careers. Our brand values were ‘aspirational’, ‘inspirational’, and ‘idealistic’.

While our brand has a ‘social entrepreneurship’ style, our research and coaching is often aimed at a different type of person. The recent coaching application analysis indicated:
1. We’re appealing to a significant number of ‘globalists’ who don’t
know about effective altruism, which may be being caused by our branding.
2. We tend not to coach these people, preferring to work with those with more technical backgrounds (e.g. software engineers, people in finance, people with quantitative degrees or philosophy degrees).

Unless we’re going to broaden our audience, it may be worth changing our branding so that we appeal less to ‘globalists’ and more to the people with technical backgrounds we tend to work with.

If we later decide to target a tech or finance audience (as mentioned in our Coaching Applications Analysis), for example, we may want to focus on intelligence rather than goodwill in our brand. The brand design could take cues from organizations known for their intelligence, like Jane Street, Hacker School, and Y Combinator. In contrast, our current branding has taken cues from organizations like TED, Charity:Water, and Purpose Generation. It seems likely that such a brand change may have to happen across our content; the logo, color scheme, website, and accessories. However, the benefits could be quite significant if done well.

Existing web services are often superior to custom solutions

We used Wufoo for the Social Impact Coaching applications. While this
could have been implemented directly in our codebase, doing so would
have taken several days to implement and then significant time
afterwards for iteration and maintenance. We found Wufoo to be very
flexible and the cost ($14.95 per month) was quite reasonable as well.

Disqus was similar. It was free, has been trivial to maintain, and
seems to have worked quite well for our commenting.

In summary, our experiences with custom solutions have been quite
positive, especially compared to our experiences with custom Rails
software. In the future we intend to write less code and rely more on
existing services instead.

Conclusion

This 6 month period of website work with 80,000 Hours have been very
productive, though many issues have arisen and mistakes were made. Much
of the work done in this period produced substantial progress, but will
require more work to be fully fleshed out as intended. The redesign
should have updated static pages, the rubric needs revamping, and the
category system should be further improved. While we hope that these completions will be relatively simple, historically work has taken longer than expected.

In the coming year much of these loose ends will need to get completed
and regular website maintenance will be necessary. Users should expect
the website to incrementally improve without significant changes in
infrastructure this year.

Next year the team will re-evaluate its plans. If 80,000 Hours later
decides to move focus more towards outreach, as intended, the website
will probably change a lot then. This will not be simple, but the team is becoming steadily better at estimating website changes, costs, and benefits.