WordPress.com’s Job System – Cron for PHP in Distributed Environment

Colleague Demitrious Kelly (meech, Apokalyptik) earlier this month open sourced the (Unix process) jobs system he (primarily) has been developing for WordPress.com. Not that I really understand it, but “jobs” is described as

A fast, distributed, horizontally scalable system built upon linux, php5.2, and mysql 5.1 wherein work can be stored in a database, and processed outside the flow of script execution. Examples of common things that are part of script execution but not necessary to the rendering of the response to the user might be spam scanning, statistical analysis, email notification sending, processing input data, etc. Also included is an equally distributed cron mechanism to remove single servers as a point of failure for scheduled jobs.

There are a lot of terms I like in there like fast, distributed, horizontally scalable, scheduled. It could probably benefit from robust and fault tolerant. Any others?

Anyway, this crontab-like system for PHP scripts is an essential part of our WordPress.com infrastructure, and I’m really excited to see it open sourced!

Check out code.trac.wordpress.org for other pieces of our puzzle that don’t have pretty project pages.

New Project to Find Movable Type Community’s Melody

Interesting development today in the blog publishing space with the announcement of Melody and the Open Melody Software Group.

Melody is a new WordPress competitor — bring it! ;-)

Based on Movable Type Open Source (MTOS), Byrne Reese writes “[the project's] focus initially is consciously not about features, but rather upon laying the groundwork through a well-documented set of processes by which future features and contributions can be made.” to live up to it’s tag line “Community Powered Publishing”.

The tag line seems to directly take aim at Movable Type for not being community powered, though in interview Byrne suggests that may be part of the overhead of Movable Type being an enterprise product.

From my position looking over the fence, I’m sympathetic to how the Movable Type community has suffered since “in 2008 [when] the hyper dedicated Movable Type product manager, Byrne Reese, was laid off from Six Apart”. Sure, the MT community isn’t just that one person, but he sure was a catalyst and one of the only open channels to the inners of Six Apart. Since then there doesn’t seem to have been anyone there for the developer community, or for me, as a member of another project, to collaborate with. Even Byrne’s own recent email to the MTOS-dev list asking “Who is the lead engineer of MTOS?” went unanswered. Here is that email:

“I hate to ask such a seemingly odd question, but I have recently had questions I wanted to address to the lead engineer of MTOS — offlist, but am honestly not sure who that might be right now. Who is the best person to address questions about governance and process to? Is there one?”

Mark Carey writes today on mt-hacks.com:

“Over two years ago, Six Apart, the creator of Movable Type open sourced the code for the core Movable Type application. While its was an exciting and bold move, the announcement and product naming choices were confusing to many — the differences between Movable Type Open Source and the Movable Type Commercial product and closed source add-ons sold by Six Apart weren’t easy to grasp, and some even disputed the newly open source nature of core application.”

Although Six Apart promised that they would  continue “fighting for openness” when they announced “Open Source Movable Type ” at the end of 2007, Melody is now the hope for a Movable Type-based openly developed product. The Open Melody FAQs includes:

“The community created Melody out a shared passion for Movable Type and a shared desire to see it flourish as a platform. We felt that the best and quickest way to achieve that goal was to create a product in which the community was inherently entrusted with a greater degree of control over its direction, communication channels and roadmap, and rewarded with more transparency and a greater sense of belonging.”

Serdar Yegulalp writes “To see a new way for the same framework to be improved, and to allow for feedback and suggestions that stem from my own use, is deeply heartening”

I’m very interested to see how the source code flows. The greatest gift of open source isn’t the right to fork, but the ability to merge.

Wih founding members and leadership including the likes of Byrne, Tim Appnel, Jay Allen , and Jesse Gardner, Open Melody is off to an incredible start. ((By incorporating as a US non-profit there commitment is beyond doubt — if only in surviving the painful process that the WordPress Foundation has recently come out the other end of.)) The web site looks great, and they’ve chosen open and friendly development tools.

What is good for blogging and open source is good for WordPress, and Melody seems very good for both:

  • I’m eager to put my frustrations trying to collaborate with the often opaque Six Apart behind me, and collaborate through the Open Melody conduit.
  • I can’t wait to see a leaner, more modular open source MT based product emerges that is also more feature rich — further confirmation of WordPress’s own approaches, and more good open source products are great for open source.

If you love blogging or open source, then Melody needs our love, participate! (hence this post)

SourceForge Projects Moving to WordPress

Open source project hosting has long been a topic that interests me. I stopped recommending venerable SourceForge.net some years ago, as it’s proprietary stack (open source prior to 2001) became crufty and fell behind some of the newer and more agile offerings.

Well, I think it’s time to revisit.

I received an email on Tuesday from the “SourceForge.net Team” with the geeky and lame title of “SourceForge.net feature deprecation upcoming: forums, DocManager, TaskManager, Diary/Notes”. Deprecated should be deprecated from email subject lines, as should negative sounding email subjects.

The meat of the email is the second half (emphasis mine):

We will provide an easy-to-use migration path to move the data to the provided replacements.  We will also provide dumps of this data in case
projects want to do something different with their data.  Additional information on how to obtain or migrate your data will be provided when the
timeline is announced, in a future mailing.

The following applications are due to be deprecated, replaced by high-quality Open Source applications we have in our Hosted Apps offering:

* TaskManager will be replaced by TaskFreak!, dotProject and Trac
(tickets).
* DocManager will be replaced by MediaWiki and Trac (wiki).
* Discussion Forums will be replaced by phpBB.
* Diary and Notes will be replaced by WordPress.

To solicit your feedback on how the migration should be handled, and alternate options you would like us to consider, we are running a survey
for the next 30 days for the user base of each of these applications.  For links to the surveys, please see our Site Status post at:

http://tinyurl.com/q3g8o3

Trac and WordPress (really!) are two of my favorite open source web applications. And all of the applications in that list are highly regarded.

It’s fantastic to see SourceForge getting back to open source — ironic, no. It will be interesting to see how active they are in particulating in those projects. I dont’ think I’ve seen any SorceForge team members participating recently on WordPress’s Bug Tracker (Trac).

It’s also fantastic to see SourceForge engaging their community by posting this on a WordPress powered blog and also using surveys to solicite feedback about the migrations and the tool choices.

I would love to find out how long this change has been in the works and what people made it happen.

I also notice that SourceForge’s own documentation is now in a trac wiki. Wow, as well as Subversion, you can also use either Mercurial or Git.

The SourceForge team has done an incredible amount for open source, and I’m excited to reconsider using SourceForge.net again to future projects.

WordCamp SF Here I Come!

Only two more weeks until WordCamp San Francisco 2009. I can’t wait!

This is the original WordCamp. Every year has been fantastic!  There is no other event that brings so many of WordPress‘s elite together.

They’re friendly people to boot! Thankfully, the elite are welcoming and generous with their time, knowledge, talent and bad jokes (puns). It’s a great learning environment.

The best kept secret about WordCamp is that the speaker line up includes many technology and web luminaries – Matt Cutts, Philip Greenspun, Tim Ferriss, Tara Hunt to name a few of the incredible speakers.

Can you believe it is near free — the $25 cost doesn’t even cover the cost of the food. Thank you sponsors!

Even if WordPress wasn’t your thing (shame!), if you’re in the web, it should be a must attend event. It’s a great event for all bloggers and anyone publishing online.

There is also an after party celebrating the sixth anniversary of WordPress!

And Sunday, we’re hosting a barcamp-esque WordPress developer day:

…expect more hardcore geek content like heavy WordPress performance optimization, BuddyPress internals, an intro to Erlang, a guide to secure coding, WordPress-as-CMS discussions, and more. If there’s a topic you’d like to lead start thinking about it now…

There are only 64 spaces left for WordCamp, so sign up now! (Already 536 people have registered.)

I hope to meet you there.

Budd Family helping at Genius Bar WordCamp SF 2008

Budd Family helping at Genius Bar WordCamp SF 2008, as seen at http://central.wordcamp.org/

Daniel Jalkut, WordPress Hero!

Daniel Jalkut, the proprietor of Red Sweater Software, is the developer of the excellent Mac desktop blog editor MarsEdit. He is also a code contributor to WordPress.

I became acquainted with him about two years ago when he started submitting detailed bug tickets (often with patches!) to WordPress.

I’m the developer of MarsEdit, a Mac desktop blogging application that works with WordPress. When my customers have trouble with the “upload image” functionality, the error responses from WordPress are lackluster. I tracked this down to a simple failure case in xmlrpc.php where the verbose error returned by wp_upload_bits is not being propagated out to the response text.

I’m attaching a proposed diff which addresses this. The diff is against /trunk/ as of today, but I would really love to see this integrated into the 2.1.x branch, because it will have a major impact on my customers’ ability to debug MarsEdit interoperability with their WordPress blog.

Daniel Jalkut
Red Sweater Software

That’s the description of ticket #3981: “Patch: Improve error propagation from newMediaObject failure in xmlrpc.php”.

Starting with #3981 he has participated in 50+ bug reports and fixes in the XML-RPC and ATOMPub areas. He has literally participating every month for the last two year! He collaborates regularly with Joseph Scott and the other members of the WordPress XML-RPC and AtomPub community.

As I have been navigating around the edges of the Mac development communities lately, I’ve learned what a well regarded, active participant Daniel is the indie Mac developer community. I highly recommend his and Manton Reece‘s podcast Core Intuition if you are interested in Mac development.

Daniel comes across as a pragmatic, thoughtful, eloquent person. As the bug above suggests, it’s in the interest of giving his customers fantastic service that he participates in WordPress development. And WordPress is much better software thanks to his participation!

He also regularly recommends the WordPress.com service or the WordPress.org software depending in his customers’s needs and tech savviness.

Daniel Jalkut is a WordPress developer and advocate. He is a WordPress Hero!

Open Source Software Project Management Web Service?

Wow, the title “Open Source Software Project Management Web Service?” is a mouthful.

Working on a small project with a friend (mentioned in “Learning by doing something else“) is a great opportunity to check out tools in other parts of the web development stack. I’ve been interested in trying out some of the software project management services and software.

(At Automattic, we mostly use prologue and trac. This is a fantastic combination!)

Likely the best approach for this project would be to first identify our needs and wants and then work through the available web services to find the closest matches.

After it being a web service, the next big want is for it to be open source software.

I have not yet been able to find any software project management web service that is open source software. Any hints?

Happy Birthday GNU!

The GNU Project is 25 years young! And Richard Stallman and crew are working as hard as ever. Thank you!

The GNU Project is most famous for it’s versions of UNIX utilities and the GPL family of licenses. But whatever you think of Richard Stallman and the Free Software Foundation‘s hard-line approaches and politics, everyone has benefited from their work. Every computerized device, if it doesn’t run GNU software, likely contains software that was influenced by the GNU project or software released under a GPL license. And the GNU Project’s influence extends beyond software to most areas where technology meets freedom.

Thanks again GNU!

Quebec’s Anti- Open Source Policy

Peter Nowak wrote a CBC article yesterday about Quebec being sued for not accepting software contract bids:

“Government buyers are using an exception in provincial law that allows them to buy directly from a proprietary vendor when there are no options available, but Facil said that loophole is being abused…”

The article has the tantalizing title of “Quebec government sued for buying Microsoft software“, but the heart of the matter is that Quebec is being sued for not accepting other bids. This policy is an anti-competitive business practice, and by effect anti- open source.

Continue reading

Drupal Camp Victoria next Weekend

Drupal Camp Victoria LogoDrupal Camp Victoria is happening all day Friday and Saturday Sept 5th and 6th. It’s hosted by North Studio at their training center, 301-771 Vernon Ave (plaza just coming into Victoria, near Save-On-Foods, Walmart).

It’s a free event, and if your work is any way related to the web, you will be missing out if you don’t attend. Register now before the event is full.

Besides, we need to support the grass roots Victoria tech scene — stop the Vancouver tech drain ;-)

Dave Olson points out that it’s “strategically scheduled for the same weekend as The Great Canadian Beer Festival”. So it seems safe to expect a good turnout from Drucouver. I know Boris Mann will be in attendance, and participating.

I hope to at least make an appearance. I’ll be looking for opportunies for WordPress collaboration. But me making it there is based on the whim of my baby son — and I won’t have it any other way.

Movable Type Pro, Setting Social Networking Free, Vaporware, WordPress, BuddyPress

Six Apart VP Anil’s response today on the official Six Apart blog to my Movable Type Pro Introduction video parody doesn’t surprise me, but where is the link love?

Continue reading