2016 was a transformative year for the MetaBrainz team -- the financial changes in 2015 improved our situation and allowed us to hire Laurent Monin (Zas) as our part time system administrator. The timing of this was perfect, because as Zas was getting up to speed on our ancient systems, we saw a serious uptick in traffic at the end of 2015.
In January of 2016 it became a consistent problem for us that showed no signs of letting up. In response, we fired up three servers we had held in reserve for just such a case, but sadly, we discovered that these servers all had became unreliable as they sat unused, further complicating our hosting scenario and not allowing us to improve the quality of our service. It would take us many months to finally figure out this mysterious uptick on out traffic; where the traffic came from and how to mitigate it.
In January 2016 we also hired Christina Smith to help establish a Spanish representative organization in an effort to setup a new headquarters office in Barcelona. Christina researched different Spanish corporate models that would allow us to have a pass-thru organization that would be piggy backed off the California 501(c)3 non-profit.
In January we continued with our participation in the Google Code-In project, which had actually started in December of 2015. A number of high-school aged students from various countries claimed and completed tasks for MetaBrainz in the course of the program. Code-in, compared to Summer of Code, focuses on less technically challenging tasks, which allows the mentors and students to complete a lot of relatively small tasks can still provide a lot of polish for our projects.
One of the big winners of our Google Code-in participation was our own BookBrainz project. A lot of students contributed many tasks and that led to a release of BookBrainz in February. In the same month we were also accepted to the Google’s Summer of Code internship program for 2016!
We kicked off March by moving our forums to the Discourse platform. Discourse was intended to combine the defunct mailing lists that had been seeing a decrease in traffic over the years and our old online forums. Previously it has been hard for us combine both of these as their respective user groups were never interested in using the other technology, which split our community into two. Discourse is of high enough quality that it allowed us to consolidate both mailing lists and forums into one technology that works well.
Also in March we found that the site Six Degrees of Vince Gill is powered by MusicBrainz data! For a less fun aspect, we had to deal with a questionable web organization that aimed to whitewash the origin of one of their artists. When MusicBrainz was founded, we had no idea that we would get involved in tricky issues like this one!
In April we finally signed a lease on an office in Barcelona and started the slow process of getting the utilities set up and finding furniture for the office. It would take until nearly June to get the office ready for people to actually work there. For a while we even powered our office router off a solar panel!
In May we had a difficult schema change release. Most of what happened was laid out in our blog, but to sum it up: This release was drastically complicated by our aging infrastructure, further fueling the need to moving to a new hosting provider.
After the schema change release was complete, we called an all-hands-on-deck emergency phase for our project to plan and execute a move to a new hosting provider. We started planning and evaluating service providers; eventually we decided to move to Hetzner in Germany and to move our hosting infrastructure to the Docker software containerization system.
All of the sudden we had a ton of work to do with a great sense of urgency -- we had to get our services stable soon, so we put our heads down and worked. And you can see this happening as the number of blog posts on our blog died down in the latter half of 2016, since we had nothing new to report while we feverishly worked towards modernizing our hosting.
In July Christina left the team after a number of conflicts that ultimately required the help of the board of directors to sort out and resolve. In August we published our Google Summer of Code results -- GSoC seemed to be the only relief we got from working on our move.
In November it was finally time to make the move to Hetzner -- a very challenging move was further complicated by the loss of a file in the migration, which made maintaining our Live Data Feed a challenge, to say the least. Overall the site was migrated to Hetzner and our old hosting setup in California broken down. MusicBrainz remained in read-only state for a few days as we ensured that the new infrastructure was stable enough for us to enable writes to our database again.
We spent the remainder of the year working to get our new services cleaned up at Hetzner -- we had left quite a few loose ends that needed wrapping up. Between that and learning how our new infrastructure worked we were kept busy through the end of the year.
2016 was a good year for picking up new supporters. Our new website from 2015 was starting to bear fruit and we saw a number of companies join our supporters. In June Barcelona based BMAT and US based Wolfram Research both became Bronze supporters. In September the London based Soundmouse became a Silver supporter and Ticketmaster joined in November and SiriusXM in December, both at Unicorn level.
The new heavy hitting supporters toward the end of the year allowed us to start thinking about growing them team further!
At the end of 2015 we had a sharp uptick in traffic that caused our servers to overload -- at that point in time it was unclear where that traffic was coming from and we had no means to really analyze our situation. As 2016 got under way, we started making concrete plans to find a new hosting provider and what it would entail to move our services.
We still had not been able to setup our traffic analysis software in 2016, so we only have a rough sketch of what was happening to our traffic. For most of 2016 we had a lot of traffic hitting us that we didn’t know where it was coming from. Zas spent quite a few weeks trying to make sense of this mysterious traffic. He started blocking the traffic that we could not identify and waited for people to complain that they couldn't reach MusicBrainz.
As blocked people started contacting us for help, Zas slowly started picking up on the fact that all blocked users had a QNAP NAS in their local home network. After a lot of digging it became clear that QNAP’s NAS product had started using our API without ever having a discussion about this with us. We proceeded to block their traffic more effectively and started to find a way to discuss the matter with QNAP directly.
Before our move to Hetzner we had old servers coupled expensive bandwidth and loads of traffic that we couldn't quite cope with -- we were limited to 35 mbit/s of outgoing traffic. After our move to Hetzner our traffic jumped to 120 mbits/s! Blocking the QNAP traffic and improving the controls of bad traffic to our API allowed us to make our systems relatively stable again, with bandwidth usage leveling off at 70 mbit/s.
After the move we had vastly more servers at our disposal and we immediately increased the rate at which we started serving successful traffic. Our services immediately became more stable serving traffic more constantly. The response time on our web pages and our API started improving as we got the new system to be stable. At this point we started collecting a different set of metrics that measure the performance of our systems, but we don’t have enough data for 2016 to provide a better thumbnail of our traffic. We hope that our 2017 report will change that significantly.
In 2016 we earned the following income:
| Income | |
|---|---|
| Donations - PayPal | $ 6,177.44 |
| Donations - General | $ 43,521.36 |
| Consulting - General | $ 11,800.00 |
| Bank - Interest | $ 53.28 |
| Affiliate - Amazon | $ 97.95 |
| Affiliate - Tagger | $ 4,194.31 |
| Sponsorship - Summit | $ 4,189.15 |
| Support - Bronze | $ 24,830.54 |
| Support - Silver | $ 18,049.11 |
| Support - Gold | $ 6,000.00 |
| Support - Unicorn | $ 165,579.88 |
| Support - Mobile | $ 245.54 |
| Support - Development | $ 39,600.00 |
| Total income: | $ 324,338.56 |
| Expenses | |
|---|---|
| Bank - General | $ 984.84 |
| Bank - BBVA | $ 326.41 |
| Bank - PayPal | $ 770.84 |
| Rent - Higuera Office | $ 707.83 |
| Hardware | $ 844.84 |
| Travel | $ 6,222.73 |
| Internet | $ 1,142.54 |
| Marketing | $ 2,003.75 |
| Depreciation | $ 7,786.00 |
| Development - Contractor | $ 28,558.13 |
| Officer Salary | $ 80,602.86 |
| Community management | $ 22,272.57 |
| Development - 1099 | $ 46,695.36 |
| Customer service | $ 8,009.64 |
| Events | $ 536.97 |
| Hosting - DWNI | $ 20,325.00 |
| Filing Fees | $ 150.00 |
| Software | $ 363.96 |
| Entertainment | $ 1,156.81 |
| Insurance | $ 4,333.00 |
| Accounting | $ 2,100.00 |
| Shipping | $ 157.77 |
| Bank - WePay | $ 12.84 |
| Hosting - Hetzner | $ 7,621.16 |
| Bank - Stripe | $ 142.25 |
| Hosting - Google | $ 1,702.79 |
| Sysadmin | $ 25,235.53 |
| Biz Dev - Contractor | $ 15,036.54 |
| Office Buenaventura Muñoz 15 | $ 11,001.08 |
| Rent Buenaventura Muñoz | $ 5,715.57 |
| Utilities - Buenaventura Muñoz office | $1,436.32 |
| Loss/gain equip sale | $ 1,439.23 |
| Exchange Gain or Loss | $ 230.05 |
| Total expenses: | $302,286.65 |
| Excess income: | $18,713.35 |
Our revenue increased over the course of 2016, removing more financial pressure, allowing us some more breathing room. Moving to Hetzner initially saved money and expected our expenditures there would be far less for a couple of years to come, while the services we’re receiving are considerably greater than what we had available in California.
At the end of 2016 we had gotten rid of all of our servers and are now renting servers from Hetzner in Germany. Going forward this means that we no longer have servers to fiscally depreciate or otherwise worry about. If a server fails we submit a ticket and someone else has to worry about fixing the server. This was a game changer for our entire team!
In the course of 2016 we spent $844.84 on hardware, mostly related to preparing for our move to Hetzner. We spent $20,325.00 on hosting in California and $7,621.16 on our new infrastructure. Including $1,702.79 for some Google Cloud hosting, the total hosting cost came to $29,648.95 for 2016.
This compares to $24,375.00 in 2015, which means that our costs of operating the service were a little higher in 2016. Most of this increased cost was related to starting up new infrastructure in Germany; paying setup-costs and for duplicated hosting infrastructures for almost two months.
In 2016 we spent $75,253.49 on salaries for developers, down from $81,378.25 in 2016. But we also hired a system administrator, so our total expenditure on technical contractors grew to $100,489.02. After a complete year of having a community manager, we spent $22,272.57. We also spent $8,009.64 for customer support and our costs for administration was $80,602.86.
2016 was the first full year after changing how we charge for support; here is the breakdown of how we generated revenue from our open data:
| Income | |
|---|---|
| Support - Bronze | $ 24,830.54 |
| Support - Silver | $ 18,049.11 |
| Support - Gold | $ 6,000.00 |
| Support - Unicorn | $ 165,579.88 |
| Support - Mobile | $ 245.54 |
| Support - Development | $ 39,600.00 |
In 2015 we had earned only $146,130.00 from licensing our data and support payments. In 2016, this grew to $254,305.07! The two largest factors that led to this growth are that we asked more companies for support and that our data has been growing and becoming more relevant to the industry.
The Bronze category has the largest number of customers, with almost 20 supporters in 2016, yet it doesn’t earn a lot of money, $24,830.54, since many of those supporters have limited income or make limited use of our data. If you compare this to our Unicorn level where large companies normally come in, you’ll see far fewer supporters, but with considerably much more income at $165,579.88. Finally, the Development category was an anomaly for 2016 -- in 2016 we received payment from Google for the JSON-LD contract we completed for them in 2015.
You can also see that the mobile tier didn’t pan out as we had thought -- in fact no mobile application has ever generated any real income. Just from these numbers we can deduce that mobile isn’t a big market for us and also that if we wish to increase income, we should go Unicorn hunting, rather than searching for Bronze.
Our end user donations via PayPal came to $6,177.44, down from $7,985.93 in 2015. This is far less than our donations of 2012, when they exceeded $14,000. We still have no explanation for this! Direct/large donations added up to $43,521.36, including $40,000 annual support from Google’s Open Source Programs Office and some cash donations from smaller donors. Google also sponsored our annual summit for $4,189.15. Thank you Google!
Number of editors who made at least one edit in 2016, including accounts that were deleted: 20,989. This represents almost no change to 2015, when it was 21,004 editors. The number of voters who made at least one vote in 2015, including accounts that were deleted was 1037. This is down from 1,198 in 2015.
Note: All of the statistics may include editors whose accounts have been deleted in the course of the year.
| Top editors | |||||
|---|---|---|---|---|---|
| 1 | TheBookkeeper | 400,583 | 13 | drsaunde | 89,384 |
| 2 | reosarevok | 288,316 | 14 | ProfChris | 82,515 |
| 3 | ListMyCDs.com | 218,531 | 15 | goch | 72,899 |
| 4 | stupidname | 172,009 | 16 | Senax | 72,530 |
| 5 | HibiscusKazeneko | 159,490 | 17 | fmera | 67,135 |
| 6 | loujin | 156,069 | 18 | pomes27 | 60,850 |
| 7 | tigerman325 | 123,722 | 19 | rikuj | 53,478 |
| 8 | Yurim | 118,564 | 20 | MrH | 50,156 |
| 9 | CyberSkull | 97,286 | 21 | chabreyflint | 50,059 |
| 10 | wonder_al | 94,498 | 22 | 221bbs | 49,736 |
| 11 | colbydray | 93,901 | 23 | vzell | 49,313 |
| 12 | MajorLurker | 93,528 | 24 | monxton | 47,527 |
| Top Voters | |||||
|---|---|---|---|---|---|
| 1 | TheBookkeeper | 65,837 | 13 | rdswift | 14,038 |
| 2 | fmera | 56,913 | 14 | Cheezmo | 13,907 |
| 3 | jpmoss | 47,514 | 15 | sbontrager | 11,424 |
| 4 | Jazzy Jarilith | 43,658 | 16 | JJLvK | 10,920 |
| 5 | chabreyflint | 31,117 | 17 | salo.rock | 10,058 |
| 6 | monxton | 25,131 | 18 | drsaunde | 9,281 |
| 7 | 221bbs | 23,968 | 19 | Alioth | 8,867 |
| 8 | otters61 | 23,849 | 20 | ProfChris | 7,743 |
| 9 | MrH | 17,163 | 21 | Hawke | 6,242 |
| 10 | jesus2099 | 16,000 | 22 | Senax | 5,390 |
| 11 | steinbdj | 15,970 | 23 | ListMyCDs.com | 3,682 |
| 12 | chiark | 15,399 | 24 | PhantomOTO | 3,613 |
Sadly, with the move to Hetzner in Germany where we rent servers and have no physical access to them, we cannot show you a pretty picture of our servers.
Our new hosting infrastructure uses artist names for servers, with some heavy hitters naming some of our most important servers:
This is a total of 12 machines that are handling vastly more traffic than the 21 machines that we had in service in California. Admittedly, the servers we had in California were a mixed bag -- the better ones were reasonably modern, but we had a large array of donated servers that had their best days behind them. It was never an ideal setup, but it allowed us to grow from 3 servers in 2006 to 21 servers in 2018 without spending a huge amount of money on hosting/servers.
We started the year with 35mbit/s of traffic, peaked at 120mbit/s of traffic shortly after our move and finally settled on 70mbit/s of traffic towards the end of the year. The uptime of all of our services drastically improved and the response times of all of our services dropped significantly after our move. Our search servers were still relics from a past era that continued to impact our uptime on a regular basis, but overall site reliability was drastically improved by our move to Hetzner.
Many thanks to our editors, voters, peer reviewers, bug watchers and other members of our community -- without you MetaBrainz would not be what it is today!
We'd like to also thank our developers that pushed out dozens of releases of the site, Picard, client libraries, AcousticBrainz, BookBrainz, CritiqueBrainz, ListenBrainz and the Cover Art Archive. All of your work is critical to enabling the MetaBrainz community to do its job.
We'd also like to thank our awesome Board of Directors and all of the donors who contributed money and all of our supporters who make it possible for us to employ a team of open source hackers scattered across the globe.
In particular we'd like to thank Google for its annual support and the support of our summit. These large contributions allow us to carry on with our mission of making MusicBrainz the most comprehensive music encyclopedia out there and pressing on with making our other projects more relevant.
Thank you to everyone who contributed in 2016!