dan
/
jekyll
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Merge branch 'master' into issue-653

This commit is contained in:
Matt Rogers 2013-02-25 13:08:23 -06:00
parent f460a6cf71 83675ee095
commit d2c79a8841
56 changed files with 661 additions and 1876 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
.gitignore vendored
View File

@ -10,3 +10,4 @@ _site/
bbin/
gh-pages/
site/_site/
coverage

2
.ruby-version
View File

@ -1 +1 @@
1.9.3-p362
1.9.3-p374

54
CONTRIBUTING.md
View File

@ -1,20 +1,35 @@
Contribute
==========
So you've got an awesome idea to throw into Jekyll. Great! Please keep the following in mind:
So you've got an awesome idea to throw into Jekyll. Great! Please keep the
following in mind:
* **Contributions will not be accepted without tests.**
* If you're creating a small fix or patch to an existing feature, just a simple test will do. Please stay in the confines of the current test suite and use [Shoulda](http://github.com/thoughtbot/shoulda/tree/master) and [RR](http://github.com/btakita/rr/tree/master).
* If it's a brand new feature, make sure to create a new [Cucumber](https://github.com/cucumber/cucumber/) feature and reuse steps where appropriate. Also, whipping up some documentation in your fork's wiki would be appreciated, and once merged it will be transferred over to the main wiki.
* If you're creating a small fix or patch to an existing feature, just a simple
test will do. Please stay in the confines of the current test suite and use
[Shoulda](http://github.com/thoughtbot/shoulda/tree/master) and
[RR](http://github.com/btakita/rr/tree/master).
* If it's a brand new feature, make sure to create a new
[Cucumber](https://github.com/cucumber/cucumber/) feature and reuse steps
where appropriate. Also, whipping up some documentation in your fork's wiki
would be appreciated, and once merged it will be transferred over to the main
wiki.
* If your contribution changes any Jekyll behavior, make sure to update the
documentation. It lives in site/_posts. If the docs are missing information,
please feel free to add it in. Great docs make a great project!
* Please follow the [Github Ruby Styleguide](https://github.com/styleguide/ruby) when modifying Ruby code.
Test Dependencies
-----------------
To run the test suite and build the gem you'll need to install Jekyll's dependencies. Jekyll uses Bundler, so a quick run of the bundle command and you're all set!
To run the test suite and build the gem you'll need to install Jekyll's
dependencies. Jekyll uses Bundler, so a quick run of the bundle command and
you're all set!
$ bundle
Before you start, run the tests and make sure that they pass (to confirm your environment is configured properly):
Before you start, run the tests and make sure that they pass (to confirm your
environment is configured properly):
$ rake test
$ rake features
@ -23,23 +38,30 @@ Workflow
--------
Here's the most direct way to get your work merged into the project:
* Fork the project
* Clone down your fork ( `git clone git://github.com/<username>/jekyll.git` )
* Create a topic branch to contain your change ( `git checkout -b my_awesome_feature` )
* Fork the project.
* Clone down your fork ( `git clone git://github.com/<username>/jekyll.git` ).
* Create a topic branch to contain your change ( `git checkout -b my_awesome_feature` ).
* Hack away, add tests. Not necessarily in that order.
* Make sure everything still passes by running `rake`
* If necessary, rebase your commits into logical chunks, without errors
* Push the branch up ( `git push origin my_awesome_feature` )
* Create an issue with a description and link to your branch
* Make sure everything still passes by running `rake`.
* If necessary, rebase your commits into logical chunks, without errors.
* Push the branch up ( `git push origin my_awesome_feature` ).
* Create a pull request against mojombo/jekyll and describe what your change
does and the why you think it should be merged.
Gotchas
-------
* If you want to bump the gem version, please put that in a separate commit. This way, the maintainers can control when the gem gets released.
* Try to keep your patch(es) based from the latest commit on mojombo/jekyll. The easier it is to apply your work, the less work the maintainers have to do, which is always a good thing.
* Please don't tag your GitHub issue with [fix], [feature], etc. The maintainers actively read the issues and will label it once they come across it.
* If you want to bump the gem version, please put that in a separate commit.
This way, the maintainers can control when the gem gets released.
* Try to keep your patch(es) based from the latest commit on mojombo/jekyll.
The easier it is to apply your work, the less work the maintainers have to do,
which is always a good thing.
* Please don't tag your GitHub issue with [fix], [feature], etc. The maintainers
actively read the issues and will label it once they come across it.
Finally...
----------
Thanks! Hacking on Jekyll should be fun, and if for some reason it's a pain to do let us know so we can fix it.
Thanks! Hacking on Jekyll should be fun. If you find any of this hard to figure
out, let us know so we can improve our process or documentation!

25
History.txt
View File

@ -1,7 +1,18 @@
== HEAD
* Major Enhancements
* Refactored jekyll commands into subcommands: build, serve, and migrate. (#690)
* Removed importers/migrators from main project, migrated to jekyll-import sub-gem (#793)
* Minor Enhancements
* Add 'gist' liquid tag to core (#822)
* New format of Jekyll output (#795)
* Reinstate --limit_posts and --future switches (#788)
* Remove ambiguity from command descriptions (#815)
* Fix SafeYAML Warnings (#807)
* Relaxed Kramdown version to 0.14 (#808)
* Aliased `jekyll server` to `jekyll serve`. (#792)
* Updated gem versions for Kramdown, Rake, Shoulda, Cucumber, and RedCarpet. (#744)
* Refactored jekyll subcommands into Jekyll::Commands submodule, which now contains them (#768)
* Rescue from import errors in Wordpress.com migrator (#671)
* Massively accelerate LSI performance (#664)
* Truncate post slugs when importing from Tumblr (#496)
* Add glob support to include, exclude option (#743)
@ -11,10 +22,22 @@
* Add source and destination directory protection (#535)
* Better YAML error message (#718)
* Bug Fixes
* Prevent custom destination from causing continuous regen (#528)
* Tumblr Migrator: finds _posts dir correctly, fixes truncation of long
post names (#775)
* Force Categories to be Strings (#767)
* Safe YAML plugin to prevent vulnerability (#777)
* Add SVG support to Jekyll/WEBrick. (#407, #406)
* Prevent custom destination from causing continuous regen on watch (#528, #820)
* Site Enhancements
* Bring site into master branch with better preview/deploy (#709)
* Redesigned site (#583)
* Development fixes
* Changed Ruby version for development to 1.9.3-p374 from p362 (#801)
* Including a link to the GitHub Ruby style guide in CONTRIBUTING.md (#806)
* Added script/bootstrap (#776)
* Running Simplecov under 2 conditions: ENV(COVERAGE)=true and with Ruby version
of greater than 1.9 (#771)
* Switch to Simplecov for coverage report (#765)
== 0.12.0 / 2012-12-22
* Minor Enhancements

39
Rakefile
View File

@ -55,14 +55,6 @@ Rake::TestTask.new(:test) do |test|
test.verbose = true
end
desc "Generate RCov test coverage and open in your browser"
task :coverage do
require 'rcov'
sh "rm -fr coverage"
sh "rcov test/test_*.rb"
sh "open coverage/index.html"
end
require 'rdoc/task'
Rake::RDocTask.new do |rdoc|
rdoc.rdoc_dir = 'rdoc'
@ -71,32 +63,6 @@ Rake::RDocTask.new do |rdoc|
rdoc.rdoc_files.include('lib/**/*.rb')
end
desc "Open an irb session preloaded with this library"
task :console do
sh "irb -rubygems -r ./lib/#{name}.rb"
end
#############################################################################
#
# Custom tasks (add your own tasks here)
#
#############################################################################
namespace :migrate do
desc "Migrate from mephisto in the current directory"
task :mephisto do
sh %q(ruby -r './lib/jekyll/migrators/mephisto' -e 'Jekyll::Mephisto.postgres(:database => "#{ENV["DB"]}")')
end
desc "Migrate from Movable Type in the current directory"
task :mt do
sh %q(ruby -r './lib/jekyll/migrators/mt' -e 'Jekyll::MT.process("#{ENV["DB"]}", "#{ENV["USER"]}", "#{ENV["PASS"]}")')
end
desc "Migrate from Typo in the current directory"
task :typo do
sh %q(ruby -r './lib/jekyll/migrators/typo' -e 'Jekyll::Typo.process("#{ENV["DB"]}", "#{ENV["USER"]}", "#{ENV["PASS"]}")')
end
end
begin
require 'cucumber/rake/task'
Cucumber::Rake::Task.new(:features) do |t|
@ -109,6 +75,11 @@ rescue LoadError
end
end
desc "Open an irb session preloaded with this library"
task :console do
sh "irb -rubygems -r ./lib/#{name}.rb"
end
#############################################################################
#
# Site tasks - http://jekyllrb.com

22
bin/jekyll
View File

@ -19,22 +19,26 @@ global_option '--layouts', 'Layouts directory (defaults to ./_layouts)'
command :build do |c|
c.syntax = 'jekyll build [options]'
c.description = 'Build your site with the option of auto-renegeration'
c.description = 'Build your site'
c.option '--future', 'Publishes posts with a future date'
c.option '--limit_posts MAX_POSTS', 'Limits the number of posts to parse and publish'
c.option '-w', '--watch', 'Watch for changes and rebuild'
c.option '--lsi', 'Use LSI for improved related posts'
c.action do |args, options|
options.defaults :serving => false
options = Jekyll.configuration(options.__hash__)
Jekyll::BuildCommand.process(options)
Jekyll::Commands::Build.process(options)
end
end
command :serve do |c|
c.syntax = 'jekyll serve [options]'
c.description = 'Serve your site locally with the option of auto-regeneration'
c.description = 'Serve your site locally'
c.option '--future', 'Publishes posts with a future date'
c.option '--limit_posts MAX_POSTS', 'Limits the number of posts to parse and publish'
c.option '-w', '--watch', 'Watch for changes and rebuild'
c.option '--lsi', 'Use LSI for improved related posts'
@ -49,10 +53,11 @@ command :serve do |c|
:serving => true
options = Jekyll.configuration(options.__hash__)
Jekyll::BuildCommand.process(options)
Jekyll::ServeCommand.process(options)
Jekyll::Commands::Build.process(options)
Jekyll::Commands::Serve.process(options)
end
end
alias_command :server, :serve
command :import do |c|
c.syntax = 'jekyll import <platform> [options]'
@ -65,6 +70,11 @@ command :import do |c|
c.option '--host', 'Host address to use when migrating'
c.action do |args, options|
Jekyll::MigrateCommand.process(args.first, options)
begin
require 'jekyll-import'
rescue
abort "You must install the 'jekyll-import' gem before continuing."
end
Jekyll::Commands::Import.process(args.first, options)
end
end

30
jekyll.gemspec
View File

@ -27,21 +27,24 @@ Gem::Specification.new do |s|
s.add_runtime_dependency('classifier', "~> 1.3")
s.add_runtime_dependency('directory_watcher', "~> 1.1")
s.add_runtime_dependency('maruku', "~> 0.5")
s.add_runtime_dependency('kramdown', "~> 0.13.4")
s.add_runtime_dependency('kramdown', "~> 0.14")
s.add_runtime_dependency('pygments.rb', "~> 0.3.2")
s.add_runtime_dependency('commander', "~> 4.1.3")
s.add_runtime_dependency('safe_yaml', "~> 0.7")
s.add_development_dependency('rake', "~> 0.9")
s.add_development_dependency('rake', "~> 10.0.3")
s.add_development_dependency('rdoc', "~> 3.11")
s.add_development_dependency('redgreen', "~> 1.2")
s.add_development_dependency('shoulda', "~> 2.11")
s.add_development_dependency('shoulda', "~> 3.3.2")
s.add_development_dependency('rr', "~> 1.0")
s.add_development_dependency('cucumber', "1.1")
s.add_development_dependency('cucumber', "~> 1.2.1")
s.add_development_dependency('RedCloth', "~> 4.2")
s.add_development_dependency('rdiscount', "~> 1.6")
s.add_development_dependency('redcarpet', "~> 2.1.1")
s.add_development_dependency('redcarpet', "~> 2.2.2")
s.add_development_dependency('launchy', "~> 2.1.2")
s.add_development_dependency('simplecov', "~> 0.7")
s.add_development_dependency('simplecov-gem-adapter', "~> 1.0.1")
# = MANIFEST =
s.files = %w[
.travis.yml
@ -67,7 +70,6 @@ Gem::Specification.new do |s|
lib/jekyll.rb
lib/jekyll/command.rb
lib/jekyll/commands/build.rb
lib/jekyll/commands/migrate.rb
lib/jekyll/commands/serve.rb
lib/jekyll/converter.rb
lib/jekyll/converters/identity.rb
@ -80,20 +82,6 @@ Gem::Specification.new do |s|
lib/jekyll/generator.rb
lib/jekyll/generators/pagination.rb
lib/jekyll/layout.rb
lib/jekyll/migrators/csv.rb
lib/jekyll/migrators/drupal.rb
lib/jekyll/migrators/enki.rb
lib/jekyll/migrators/joomla.rb
lib/jekyll/migrators/marley.rb
lib/jekyll/migrators/mephisto.rb
lib/jekyll/migrators/mt.rb
lib/jekyll/migrators/posterous.rb
lib/jekyll/migrators/rss.rb
lib/jekyll/migrators/textpattern.rb
lib/jekyll/migrators/tumblr.rb
lib/jekyll/migrators/typo.rb
lib/jekyll/migrators/wordpress.rb
lib/jekyll/migrators/wordpressdotcom.rb
lib/jekyll/page.rb
lib/jekyll/plugin.rb
lib/jekyll/post.rb

19
lib/jekyll.rb
View File

@ -18,7 +18,7 @@ require 'rubygems'
# stdlib
require 'fileutils'
require 'time'
require 'yaml'
require 'safe_yaml'
require 'English'
# 3rd party
@ -48,6 +48,8 @@ require_all 'jekyll/converters'
require_all 'jekyll/generators'
require_all 'jekyll/tags'
SafeYAML::OPTIONS[:suppress_warnings] = true
module Jekyll
VERSION = '0.12.0'
@ -129,13 +131,18 @@ module Jekyll
# Get configuration from <source>/_config.yml
config_file = File.join(source, '_config.yml')
begin
config = YAML.load_file(config_file)
raise "Invalid configuration - #{config_file}" if !config.is_a?(Hash)
$stdout.puts "Configuration from #{config_file}"
config = YAML.safe_load_file(config_file)
raise "Configuration file: (INVALID) #{config_file}" if !config.is_a?(Hash)
$stdout.puts "Configuration file: #{config_file}"
rescue SystemCallError
# Errno:ENOENT = file not found
$stderr.puts "Configuration file: none"
config = {}
rescue => err
$stderr.puts "WARNING: Could not read configuration. " +
$stderr.puts " " +
"WARNING: Error reading configuration. " +
"Using defaults (and options)."
$stderr.puts "\t" + err.to_s
$stderr.puts "#{err}"
config = {}
end

6
lib/jekyll/command.rb
View File

@ -1,14 +1,12 @@
module Jekyll
class Command
def self.globs(source)
def self.globs(source, destination)
Dir.chdir(source) do
dirs = Dir['*'].select { |x| File.directory?(x) }
dirs -= ['_site']
dirs -= [destination]
dirs = dirs.map { |x| "#{x}/**/*" }
dirs += ['*']
end
end
end
end

133
lib/jekyll/commands/build.rb
View File

@ -1,76 +1,81 @@
module Jekyll
module Commands
class Build < Command
def self.process(options)
site = Jekyll::Site.new(options)
class BuildCommand < Command
def self.process(options)
site = Jekyll::Site.new(options)
source = options['source']
destination = options['destination']
source = options['source']
destination = options['destination']
if options['watch']
self.watch(site, options)
else
self.build(site, options)
end
end
# Private: Build the site from source into destination.
#
# site - A Jekyll::Site instance
# options - A Hash of options passed to the command
#
# Returns nothing.
def self.build(site, options)
source = options['source']
destination = options['destination']
puts "Building site: #{source} -> #{destination}"
begin
site.process
rescue Jekyll::FatalException => e
puts
puts "ERROR: YOUR SITE COULD NOT BE BUILT:"
puts "------------------------------------"
puts e.message
exit(1)
end
puts "Successfully generated site: #{source} -> #{destination}"
end
# Private: Watch for file changes and rebuild the site.
#
# site - A Jekyll::Site instance
# options - A Hash of options passed to the command
#
# Returns nothing.
def self.watch(site, options)
require 'directory_watcher'
source = options['source']
destination = options['destination']
puts "Auto-Regenerating enabled: #{source} -> #{destination}"
dw = DirectoryWatcher.new(source)
dw.interval = 1
dw.glob = self.globs(source)
dw.add_observer do |*args|
t = Time.now.strftime("%Y-%m-%d %H:%M:%S")
puts "[#{t}] regeneration: #{args.size} files changed"
site.process
if options['watch']
self.watch(site, options)
else
self.build(site, options)
end
end
dw.start
# Private: Build the site from source into destination.
#
# site - A Jekyll::Site instance
# options - A Hash of options passed to the command
#
# Returns nothing.
def self.build(site, options)
source = options['source']
destination = options['destination']
puts " Source: #{source}"
puts " Destination: #{destination}"
print " Generating... "
begin
site.process
rescue Jekyll::FatalException => e
puts
puts "ERROR: YOUR SITE COULD NOT BE BUILT:"
puts "------------------------------------"
puts e.message
exit(1)
end
puts "done."
end
unless options['serving']
trap("INT") do
puts "Stopping auto-regeneration..."
exit 0
# Private: Watch for file changes and rebuild the site.
#
# site - A Jekyll::Site instance
# options - A Hash of options passed to the command
#
# Returns nothing.
def self.watch(site, options)
require 'directory_watcher'
source = options['source']
destination = options['destination']
puts " Source: #{source}"
puts " Destination: #{destination}"
puts " Auto-regeneration: enabled"
dw = DirectoryWatcher.new(source)
dw.interval = 1
dw.glob = self.globs(source, destination)
dw.add_observer do |*args|
t = Time.now.strftime("%Y-%m-%d %H:%M:%S")
print " Regenerating: #{args.size} files at #{t} "
site.process
puts "...done."
end
loop { sleep 1000 }
dw.start
unless options['serving']
trap("INT") do
puts " Halting auto-regeneration."
exit 0
end
loop { sleep 1000 }
end
end
end
end
end

47
lib/jekyll/commands/migrate.rb
View File

@ -1,47 +0,0 @@
module Jekyll
class MigrateCommand < Command
MIGRATORS = {
:csv => 'CSV',
:drupal => 'Drupal',
:enki => 'Enki',
:mephisto => 'Mephisto',
:mt => 'MT',
:posterous => 'Posterous',
:textpattern => 'TextPattern',
:tumblr => 'Tumblr',
:typo => 'Typo',
:wordpressdotcom => 'WordpressDotCom',
:wordpress => 'WordPress'
}
def self.process(migrator, options)
abort 'missing argument. Please specify a migrator' if migrator.nil?
migrator = migrator.downcase
cmd_options = []
[ :file, :dbname, :user, :pass, :host, :site ].each do |p|
cmd_options << "\"#{options[p]}\"" unless options[p].nil?
end
if MIGRATORS.keys.include?(migrator)
app_root = File.expand_path(
File.join(File.dirname(__FILE__), '..', '..', '..')
)
require "#{app_root}/lib/jekyll/migrators/#{migrator}"
if Jekyll.const_defiend?(MIGRATORS[migrator.to_sym])
puts 'Importing...'
migrator_class = Jekyll.const_get(MIGRATORS[migrator.to_sym])
migrator_class.process(*cmd_options)
exit 0
end
end
abort 'invalid migrator. Please specify a valid migrator'
end
end
end

39
lib/jekyll/commands/serve.rb
View File

@ -1,28 +1,29 @@
module Jekyll
module Commands
class Serve < Command
def self.process(options)
require 'webrick'
include WEBrick
class ServeCommand < Command
def self.process(options)
require 'webrick'
include WEBrick
destination = options['destination']
destination = options['destination']
FileUtils.mkdir_p(destination)
FileUtils.mkdir_p(destination)
mime_types = WEBrick::HTTPUtils::DefaultMimeTypes
mime_types.store 'js', 'application/javascript'
mime_types.store 'svg', 'image/svg+xml'
mime_types = WEBrick::HTTPUtils::DefaultMimeTypes
mime_types.store 'js', 'application/javascript'
s = HTTPServer.new(
:Port => options['port'],
:BindAddress => options['host'],
:MimeTypes => mime_types
)
s = HTTPServer.new(
:Port => options['port'],
:BindAddress => options['host'],
:MimeTypes => mime_types
)
s.mount(options['baseurl'], HTTPServlet::FileHandler, destination)
t = Thread.new { s.start }
trap("INT") { s.shutdown }
t.join()
s.mount(options['baseurl'], HTTPServlet::FileHandler, destination)
t = Thread.new { s.start }
trap("INT") { s.shutdown }
t.join()
end
end
end
end

4
lib/jekyll/converter.rb
View File

@ -1,5 +1,4 @@
module Jekyll
class Converter < Plugin
# Public: Get or set the pygments prefix. When an argument is specified,
# the prefix will be set. If no argument is specified, the current prefix
@ -46,5 +45,4 @@ module Jekyll
self.class.pygments_suffix
end
end
end
end

29
lib/jekyll/converters/identity.rb
View File

@ -1,22 +1,21 @@
module Jekyll
module Converters
class Identity < Converter
safe true
class IdentityConverter < Converter
safe true
priority :lowest
priority :lowest
def matches(ext)
true
end
def matches(ext)
true
def output_ext(ext)
ext
end
def convert(content)
content
end
end
def output_ext(ext)
ext
end
def convert(content)
content
end
end
end

270
lib/jekyll/converters/markdown.rb
View File

@ -1,149 +1,149 @@
module Jekyll
module Converters
class Markdown < Converter
safe true
class MarkdownConverter < Converter
safe true
pygments_prefix "\n"
pygments_suffix "\n"
pygments_prefix "\n"
pygments_suffix "\n"
def setup
return if @setup
case @config['markdown']
when 'redcarpet'
begin
require 'redcarpet'
def setup
return if @setup
case @config['markdown']
when 'redcarpet'
begin
require 'redcarpet'
@renderer ||= Class.new(Redcarpet::Render::HTML) do
def block_code(code, lang)
lang = lang && lang.split.first || "text"
output = add_code_tags(
Pygments.highlight(code, :lexer => lang, :options => { :encoding => 'utf-8' }),
lang
)
end
@renderer ||= Class.new(Redcarpet::Render::HTML) do
def block_code(code, lang)
lang = lang && lang.split.first || "text"
output = add_code_tags(
Pygments.highlight(code, :lexer => lang, :options => { :encoding => 'utf-8' }),
lang
)
def add_code_tags(code, lang)
code = code.sub(/<pre>/,'<pre><code class="' + lang + '">')
code = code.sub(/<\/pre>/,"</code></pre>")
end
end
def add_code_tags(code, lang)
code = code.sub(/<pre>/,'<pre><code class="' + lang + '">')
code = code.sub(/<\/pre>/,"</code></pre>")
@redcarpet_extensions = {}
@config['redcarpet']['extensions'].each { |e| @redcarpet_extensions[e.to_sym] = true }
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install redcarpet'
raise FatalException.new("Missing dependency: redcarpet")
end
when 'kramdown'
begin
require 'kramdown'
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install kramdown'
raise FatalException.new("Missing dependency: kramdown")
end
when 'rdiscount'
begin
require 'rdiscount'
@rdiscount_extensions = @config['rdiscount']['extensions'].map { |e| e.to_sym }
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install rdiscount'
raise FatalException.new("Missing dependency: rdiscount")
end
when 'maruku'
begin
require 'maruku'
if @config['maruku']['use_divs']
require 'maruku/ext/div'
STDERR.puts 'Maruku: Using extended syntax for div elements.'
end
if @config['maruku']['use_tex']
require 'maruku/ext/math'
STDERR.puts "Maruku: Using LaTeX extension. Images in `#{@config['maruku']['png_dir']}`."
# Switch off MathML output
MaRuKu::Globals[:html_math_output_mathml] = false
MaRuKu::Globals[:html_math_engine] = 'none'
# Turn on math to PNG support with blahtex
# Resulting PNGs stored in `images/latex`
MaRuKu::Globals[:html_math_output_png] = true
MaRuKu::Globals[:html_png_engine] = @config['maruku']['png_engine']
MaRuKu::Globals[:html_png_dir] = @config['maruku']['png_dir']
MaRuKu::Globals[:html_png_url] = @config['maruku']['png_url']
end
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install maruku'
raise FatalException.new("Missing dependency: maruku")
end
@redcarpet_extensions = {}
@config['redcarpet']['extensions'].each { |e| @redcarpet_extensions[e.to_sym] = true }
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install redcarpet'
raise FatalException.new("Missing dependency: redcarpet")
end
when 'kramdown'
begin
require 'kramdown'
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install kramdown'
raise FatalException.new("Missing dependency: kramdown")
end
when 'rdiscount'
begin
require 'rdiscount'
@rdiscount_extensions = @config['rdiscount']['extensions'].map { |e| e.to_sym }
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install rdiscount'
raise FatalException.new("Missing dependency: rdiscount")
end
when 'maruku'
begin
require 'maruku'
if @config['maruku']['use_divs']
require 'maruku/ext/div'
STDERR.puts 'Maruku: Using extended syntax for div elements.'
end
if @config['maruku']['use_tex']
require 'maruku/ext/math'
STDERR.puts "Maruku: Using LaTeX extension. Images in `#{@config['maruku']['png_dir']}`."
# Switch off MathML output
MaRuKu::Globals[:html_math_output_mathml] = false
MaRuKu::Globals[:html_math_engine] = 'none'
# Turn on math to PNG support with blahtex
# Resulting PNGs stored in `images/latex`
MaRuKu::Globals[:html_math_output_png] = true
MaRuKu::Globals[:html_png_engine] = @config['maruku']['png_engine']
MaRuKu::Globals[:html_png_dir] = @config['maruku']['png_dir']
MaRuKu::Globals[:html_png_url] = @config['maruku']['png_url']
end
rescue LoadError
STDERR.puts 'You are missing a library required for Markdown. Please run:'
STDERR.puts ' $ [sudo] gem install maruku'
raise FatalException.new("Missing dependency: maruku")
end
else
STDERR.puts "Invalid Markdown processor: #{@config['markdown']}"
STDERR.puts " Valid options are [ maruku | rdiscount | kramdown ]"
raise FatalException.new("Invalid Markdown process: #{@config['markdown']}")
end
@setup = true
end
def matches(ext)
rgx = '(' + @config['markdown_ext'].gsub(',','|') +')'
ext =~ Regexp.new(rgx, Regexp::IGNORECASE)
end
def output_ext(ext)
".html"
end
def convert(content)
setup
case @config['markdown']
when 'redcarpet'
@redcarpet_extensions[:fenced_code_blocks] = !@redcarpet_extensions[:no_fenced_code_blocks]
@renderer.send :include, Redcarpet::Render::SmartyPants if @redcarpet_extensions[:smart]
markdown = Redcarpet::Markdown.new(@renderer.new(@redcarpet_extensions), @redcarpet_extensions)
markdown.render(content)
when 'kramdown'
# Check for use of coderay
if @config['kramdown']['use_coderay']
Kramdown::Document.new(content, {
:auto_ids => @config['kramdown']['auto_ids'],
:footnote_nr => @config['kramdown']['footnote_nr'],
:entity_output => @config['kramdown']['entity_output'],
:toc_levels => @config['kramdown']['toc_levels'],
:smart_quotes => @config['kramdown']['smart_quotes'],
:coderay_wrap => @config['kramdown']['coderay']['coderay_wrap'],
:coderay_line_numbers => @config['kramdown']['coderay']['coderay_line_numbers'],
:coderay_line_number_start => @config['kramdown']['coderay']['coderay_line_number_start'],
:coderay_tab_width => @config['kramdown']['coderay']['coderay_tab_width'],
:coderay_bold_every => @config['kramdown']['coderay']['coderay_bold_every'],
:coderay_css => @config['kramdown']['coderay']['coderay_css']
}).to_html
else
# not using coderay
Kramdown::Document.new(content, {
:auto_ids => @config['kramdown']['auto_ids'],
:footnote_nr => @config['kramdown']['footnote_nr'],
:entity_output => @config['kramdown']['entity_output'],
:toc_levels => @config['kramdown']['toc_levels'],
:smart_quotes => @config['kramdown']['smart_quotes']
}).to_html
end
when 'rdiscount'
rd = RDiscount.new(content, *@rdiscount_extensions)
html = rd.to_html
if rd.generate_toc and html.include?(@config['rdiscount']['toc_token'])
html.gsub!(@config['rdiscount']['toc_token'], rd.toc_content)
end
html
when 'maruku'
Maruku.new(content).to_html
STDERR.puts "Invalid Markdown processor: #{@config['markdown']}"
STDERR.puts " Valid options are [ maruku | rdiscount | kramdown ]"
raise FatalException.new("Invalid Markdown process: #{@config['markdown']}")
end
@setup = true
end
def matches(ext)
rgx = '(' + @config['markdown_ext'].gsub(',','|') +')'
ext =~ Regexp.new(rgx, Regexp::IGNORECASE)
end
def output_ext(ext)
".html"
end
def convert(content)
setup
case @config['markdown']
when 'redcarpet'
@redcarpet_extensions[:fenced_code_blocks] = !@redcarpet_extensions[:no_fenced_code_blocks]
@renderer.send :include, Redcarpet::Render::SmartyPants if @redcarpet_extensions[:smart]
markdown = Redcarpet::Markdown.new(@renderer.new(@redcarpet_extensions), @redcarpet_extensions)
markdown.render(content)
when 'kramdown'
# Check for use of coderay
if @config['kramdown']['use_coderay']
Kramdown::Document.new(content, {
:auto_ids => @config['kramdown']['auto_ids'],
:footnote_nr => @config['kramdown']['footnote_nr'],
:entity_output => @config['kramdown']['entity_output'],
:toc_levels => @config['kramdown']['toc_levels'],
:smart_quotes => @config['kramdown']['smart_quotes'],
:coderay_wrap => @config['kramdown']['coderay']['coderay_wrap'],
:coderay_line_numbers => @config['kramdown']['coderay']['coderay_line_numbers'],
:coderay_line_number_start => @config['kramdown']['coderay']['coderay_line_number_start'],
:coderay_tab_width => @config['kramdown']['coderay']['coderay_tab_width'],
:coderay_bold_every => @config['kramdown']['coderay']['coderay_bold_every'],
:coderay_css => @config['kramdown']['coderay']['coderay_css']
}).to_html
else
# not using coderay
Kramdown::Document.new(content, {
:auto_ids => @config['kramdown']['auto_ids'],
:footnote_nr => @config['kramdown']['footnote_nr'],
:entity_output => @config['kramdown']['entity_output'],
:toc_levels => @config['kramdown']['toc_levels'],
:smart_quotes => @config['kramdown']['smart_quotes']
}).to_html
end
when 'rdiscount'
rd = RDiscount.new(content, *@rdiscount_extensions)
html = rd.to_html
if rd.generate_toc and html.include?(@config['rdiscount']['toc_token'])
html.gsub!(@config['rdiscount']['toc_token'], rd.toc_content)
end
html
when 'maruku'
Maruku.new(content).to_html
end
end
end
end
end

84
lib/jekyll/converters/textile.rb
View File

@ -1,50 +1,50 @@
module Jekyll
module Converters
class Textile < Converter
safe true
class TextileConverter < Converter
safe true
pygments_prefix '<notextile>'
pygments_suffix '</notextile>'
pygments_prefix '<notextile>'
pygments_suffix '</notextile>'
def setup
return if @setup
require 'redcloth'
@setup = true
rescue LoadError
STDERR.puts 'You are missing a library required for Textile. Please run:'
STDERR.puts ' $ [sudo] gem install RedCloth'
raise FatalException.new("Missing dependency: RedCloth")
end
def matches(ext)
rgx = '(' + @config['textile_ext'].gsub(',','|') +')'
ext =~ Regexp.new(rgx, Regexp::IGNORECASE)
end
def output_ext(ext)
".html"
end
def convert(content)
setup
# Shortcut if config doesn't contain RedCloth section
return RedCloth.new(content).to_html if @config['redcloth'].nil?
# List of attributes defined on RedCloth
# (from http://redcloth.rubyforge.org/classes/RedCloth/TextileDoc.html)
attrs = ['filter_classes', 'filter_html', 'filter_ids', 'filter_styles',
'hard_breaks', 'lite_mode', 'no_span_caps', 'sanitize_html']
r = RedCloth.new(content)
# Set attributes in r if they are NOT nil in the config
attrs.each do |attr|
r.instance_variable_set("@#{attr}".to_sym, @config['redcloth'][attr]) unless @config['redcloth'][attr].nil?
def setup
return if @setup
require 'redcloth'
@setup = true
rescue LoadError
STDERR.puts 'You are missing a library required for Textile. Please run:'
STDERR.puts ' $ [sudo] gem install RedCloth'
raise FatalException.new("Missing dependency: RedCloth")
end
r.to_html
def matches(ext)
rgx = '(' + @config['textile_ext'].gsub(',','|') +')'
ext =~ Regexp.new(rgx, Regexp::IGNORECASE)
end
def output_ext(ext)
".html"
end
def convert(content)
setup
# Shortcut if config doesn't contain RedCloth section
return RedCloth.new(content).to_html if @config['redcloth'].nil?
# List of attributes defined on RedCloth
# (from http://redcloth.rubyforge.org/classes/RedCloth/TextileDoc.html)
attrs = ['filter_classes', 'filter_html', 'filter_ids', 'filter_styles',
'hard_breaks', 'lite_mode', 'no_span_caps', 'sanitize_html']
r = RedCloth.new(content)
# Set attributes in r if they are NOT nil in the config
attrs.each do |attr|
r.instance_variable_set("@#{attr}".to_sym, @config['redcloth'][attr]) unless @config['redcloth'][attr].nil?
end
r.to_html
end
end
end
end

2
lib/jekyll/convertible.rb
View File

@ -30,7 +30,7 @@ module Jekyll
if self.content =~ /\A(---\s*\n.*?\n?)^(---\s*$\n?)/m
self.content = $POSTMATCH
self.data = YAML.load($1)
self.data = YAML.safe_load($1)
end
rescue => e
puts "Error reading file #{File.join(base, name)}: #{e.message}"

4
lib/jekyll/errors.rb
View File

@ -1,6 +1,4 @@
module Jekyll
class FatalException < StandardError
end
end
end

6
lib/jekyll/filters.rb
View File

@ -1,7 +1,6 @@
require 'uri'
module Jekyll
module Filters
# Convert a Textile string into HTML output.
#
@ -10,7 +9,7 @@ module Jekyll
# Returns the HTML formatted String.
def textilize(input)
site = @context.registers[:site]
converter = site.getConverterImpl(Jekyll::TextileConverter)
converter = site.getConverterImpl(Jekyll::Converters::Textile)
converter.convert(input)
end
@ -21,7 +20,7 @@ module Jekyll
# Returns the HTML formatted String.
def markdownify(input)
site = @context.registers[:site]
converter = site.getConverterImpl(Jekyll::MarkdownConverter)
converter = site.getConverterImpl(Jekyll::Converters::Markdown)
converter.convert(input)
end
@ -124,6 +123,5 @@ module Jekyll
"#{array[0...-1].join(', ')}, #{connector} #{array[-1]}"
end
end
end
end

5
lib/jekyll/generator.rb
View File

@ -1,7 +1,4 @@
module Jekyll
class Generator < Plugin
end
end
end

94
lib/jekyll/generators/pagination.rb
View File

@ -1,55 +1,56 @@
module Jekyll
module Generators
class Pagination < Generator
# This generator is safe from arbitrary code execution.
safe true
class Pagination < Generator
# This generator is safe from arbitrary code execution.
safe true
# Generate paginated pages if necessary.
#
# site - The Site.
#
# Returns nothing.
def generate(site)
site.pages.dup.each do |page|
paginate(site, page) if Pager.pagination_enabled?(site.config, page.name)
end
end
# Paginates the blog's posts. Renders the index.html file into paginated
# directories, e.g.: page2/index.html, page3/index.html, etc and adds more
# site-wide data.
#
# site - The Site.
# page - The index.html Page that requires pagination.
#
# {"paginator" => { "page" => <Number>,
# "per_page" => <Number>,
# "posts" => [<Post>],
# "total_posts" => <Number>,
# "total_pages" => <Number>,
# "previous_page" => <Number>,
# "next_page" => <Number> }}
def paginate(site, page)
all_posts = site.site_payload['site']['posts']
pages = Pager.calculate_pages(all_posts, site.config['paginate'].to_i)
(1..pages).each do |num_page|
pager = Pager.new(site.config, num_page, all_posts, pages)
if num_page > 1
newpage = Page.new(site, site.source, page.dir, page.name)
newpage.pager = pager
newpage.dir = File.join(page.dir, paginate_path(site, num_page))
site.pages << newpage
else
page.pager = pager
# Generate paginated pages if necessary.
#
# site - The Site.
#
# Returns nothing.
def generate(site)
site.pages.dup.each do |page|
paginate(site, page) if Pager.pagination_enabled?(site.config, page.name)
end
end
end
private
def paginate_path(site, num_page)
format = site.config['paginate_path']
format.sub(':num', num_page.to_s)
# Paginates the blog's posts. Renders the index.html file into paginated
# directories, e.g.: page2/index.html, page3/index.html, etc and adds more
# site-wide data.
#
# site - The Site.
# page - The index.html Page that requires pagination.
#
# {"paginator" => { "page" => <Number>,
# "per_page" => <Number>,
# "posts" => [<Post>],
# "total_posts" => <Number>,
# "total_pages" => <Number>,
# "previous_page" => <Number>,
# "next_page" => <Number> }}
def paginate(site, page)
all_posts = site.site_payload['site']['posts']
pages = Pager.calculate_pages(all_posts, site.config['paginate'].to_i)
(1..pages).each do |num_page|
pager = Pager.new(site.config, num_page, all_posts, pages)
if num_page > 1
newpage = Page.new(site, site.source, page.dir, page.name)
newpage.pager = pager
newpage.dir = File.join(page.dir, paginate_path(site, num_page))
site.pages << newpage
else
page.pager = pager
end
end
end
private
def paginate_path(site, num_page)
format = site.config['paginate_path']
format.sub(':num', num_page.to_s)
end
end
end
class Pager
@ -115,5 +116,4 @@ module Jekyll
}
end
end
end

2
lib/jekyll/layout.rb
View File

@ -1,5 +1,4 @@
module Jekyll
class Layout
include Convertible
@ -40,5 +39,4 @@ module Jekyll
self.ext = File.extname(name)
end
end
end

26
lib/jekyll/migrators/csv.rb
View File

@ -1,26 +0,0 @@
module Jekyll
module CSV
# Reads a csv with title, permalink, body, published_at, and filter.
# It creates a post file for each row in the csv
def self.process(file = "posts.csv")
FileUtils.mkdir_p "_posts"
posts = 0
FasterCSV.foreach(file) do |row|
next if row[0] == "title"
posts += 1
name = row[3].split(" ")[0]+"-"+row[1]+(row[4] =~ /markdown/ ? ".markdown" : ".textile")
File.open("_posts/#{name}", "w") do |f|
f.puts <<-HEADER
---
layout: post
title: #{row[0]}
---
HEADER
f.puts row[2]
end
end
"Created #{posts} posts!"
end
end
end

103
lib/jekyll/migrators/drupal.rb
View File

@ -1,103 +0,0 @@
require 'rubygems'
require 'sequel'
require 'fileutils'
require 'yaml'
# NOTE: This converter requires Sequel and the MySQL gems.
# The MySQL gem can be difficult to install on OS X. Once you have MySQL
# installed, running the following commands should work:
# $ sudo gem install sequel
# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
module Jekyll
module Drupal
# Reads a MySQL database via Sequel and creates a post file for each post
# in wp_posts that has post_status = 'publish'. This restriction is made
# because 'draft' posts are not guaranteed to have valid dates.
QUERY = "SELECT n.nid, \
n.title, \
nr.body, \
n.created, \
n.status \
FROM node AS n, \
node_revisions AS nr \
WHERE (n.type = 'blog' OR n.type = 'story') \
AND n.vid = nr.vid"
def self.process(dbname, user, pass, host = 'localhost', prefix = '')
db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host, :encoding => 'utf8')
if prefix != ''
QUERY[" node "] = " " + prefix + "node "
QUERY[" node_revisions "] = " " + prefix + "node_revisions "
end
FileUtils.mkdir_p "_posts"
FileUtils.mkdir_p "_drafts"
# Create the refresh layout
# Change the refresh url if you customized your permalink config
File.open("_layouts/refresh.html", "w") do |f|
f.puts <<EOF
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8" />
<meta http-equiv="refresh" content="0;url={{ page.refresh_to_post_id }}.html" />
</head>
</html>
EOF
end
db[QUERY].each do |post|
# Get required fields and construct Jekyll compatible name
node_id = post[:nid]
title = post[:title]
content = post[:body]
created = post[:created]
time = Time.at(created)
is_published = post[:status] == 1
dir = is_published ? "_posts" : "_drafts"
slug = title.strip.downcase.gsub(/(&|&amp;)/, ' and ').gsub(/[\s\.\/\\]/, '-').gsub(/[^\w-]/, '').gsub(/[-_]{2,}/, '-').gsub(/^[-_]/, '').gsub(/[-_]$/, '')
name = time.strftime("%Y-%m-%d-") + slug + '.md'
# Get the relevant fields as a hash, delete empty fields and convert
# to YAML for the header
data = {
'layout' => 'post',
'title' => title.to_s,
'created' => created,
}.delete_if { |k,v| v.nil? || v == ''}.to_yaml
# Write out the data and content to file
File.open("#{dir}/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
# Make a file to redirect from the old Drupal URL
if is_published
aliases = db["SELECT dst FROM #{prefix}url_alias WHERE src = ?", "node/#{node_id}"].all
aliases.push(:dst => "node/#{node_id}")
aliases.each do |url_alias|
FileUtils.mkdir_p url_alias[:dst]
File.open("#{url_alias[:dst]}/index.md", "w") do |f|
f.puts "---"
f.puts "layout: refresh"
f.puts "refresh_to_post_id: /#{time.strftime("%Y/%m/%d/") + slug}"
f.puts "---"
end
end
end
end
# TODO: Make dirs & files for nodes of type 'page'
# Make refresh pages for these as well
# TODO: Make refresh dirs & files according to entries in url_alias table
end
end
end

49
lib/jekyll/migrators/enki.rb
View File

@ -1,49 +0,0 @@
# Adapted by Rodrigo Pinto <rodrigopqn@gmail.com>
# Based on typo.rb by Toby DiPasquale
require 'fileutils'
require 'rubygems'
require 'sequel'
module Jekyll
module Enki
SQL = <<-EOS
SELECT p.id,
p.title,
p.slug,
p.body,
p.published_at as date,
p.cached_tag_list as tags
FROM posts p
EOS
# Just working with postgres, but can be easily adapted
# to work with both mysql and postgres.
def self.process(dbname, user, pass, host = 'localhost')
FileUtils.mkdir_p('_posts')
db = Sequel.postgres(:database => dbname,
:user => user,
:password => pass,
:host => host,
:encoding => 'utf8')
db[SQL].each do |post|
name = [ sprintf("%.04d", post[:date].year),
sprintf("%.02d", post[:date].month),
sprintf("%.02d", post[:date].day),
post[:slug].strip ].join('-')
name += '.textile'
File.open("_posts/#{name}", 'w') do |f|
f.puts({ 'layout' => 'post',
'title' => post[:title].to_s,
'enki_id' => post[:id],
'categories' => post[:tags]
}.delete_if { |k, v| v.nil? || v == '' }.to_yaml)
f.puts '---'
f.puts post[:body].delete("\r")
end
end
end
end
end

53
lib/jekyll/migrators/joomla.rb
View File

@ -1,53 +0,0 @@
require 'rubygems'
require 'sequel'
require 'fileutils'
require 'yaml'
# NOTE: This migrator is made for Joomla 1.5 databases.
# NOTE: This converter requires Sequel and the MySQL gems.
# The MySQL gem can be difficult to install on OS X. Once you have MySQL
# installed, running the following commands should work:
# $ sudo gem install sequel
# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
module Jekyll
module Joomla
def self.process(dbname, user, pass, host = 'localhost', table_prefix = 'jos_', section = '1')
db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host, :encoding => 'utf8')
FileUtils.mkdir_p("_posts")
# Reads a MySQL database via Sequel and creates a post file for each
# post in wp_posts that has post_status = 'publish'. This restriction is
# made because 'draft' posts are not guaranteed to have valid dates.
query = "SELECT `title`, `alias`, CONCAT(`introtext`,`fulltext`) as content, `created`, `id` FROM #{table_prefix}content WHERE state = '0' OR state = '1' AND sectionid = '#{section}'"
db[query].each do |post|
# Get required fields and construct Jekyll compatible name.
title = post[:title]
slug = post[:alias]
date = post[:created]
content = post[:content]
name = "%02d-%02d-%02d-%s.markdown" % [date.year, date.month, date.day,
slug]
# Get the relevant fields as a hash, delete empty fields and convert
# to YAML for the header.
data = {
'layout' => 'post',
'title' => title.to_s,
'joomla_id' => post[:id],
'joomla_url' => post[:alias],
'date' => date
}.delete_if { |k,v| v.nil? || v == '' }.to_yaml
# Write out the data and content to file
File.open("_posts/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
end
end
end
end

52
lib/jekyll/migrators/marley.rb
View File

@ -1,52 +0,0 @@
require 'yaml'
require 'fileutils'
module Jekyll
module Marley
def self.regexp
{ :id => /^\d{0,4}-{0,1}(.*)$/,
:title => /^#\s*(.*)\s+$/,
:title_with_date => /^#\s*(.*)\s+\(([0-9\/]+)\)$/,
:published_on => /.*\s+\(([0-9\/]+)\)$/,
:perex => /^([^\#\n]+\n)$/,
:meta => /^\{\{\n(.*)\}\}\n$/mi # Multiline Regexp
}
end
def self.process(marley_data_dir)
raise ArgumentError, "marley dir #{marley_data_dir} not found" unless File.directory?(marley_data_dir)
FileUtils.mkdir_p "_posts"
posts = 0
Dir["#{marley_data_dir}/**/*.txt"].each do |f|
next unless File.exists?(f)
#copied over from marley's app/lib/post.rb
file_content = File.read(f)
meta_content = file_content.slice!( self.regexp[:meta] )
body = file_content.sub( self.regexp[:title], '').sub( self.regexp[:perex], '').strip
title = file_content.scan( self.regexp[:title] ).first.to_s.strip
prerex = file_content.scan( self.regexp[:perex] ).first.to_s.strip
published_on = DateTime.parse( post[:published_on] ) rescue File.mtime( File.dirname(f) )
meta = ( meta_content ) ? YAML::load( meta_content.scan( self.regexp[:meta]).to_s ) : {}
meta['title'] = title
meta['layout'] = 'post'
formatted_date = published_on.strftime('%Y-%m-%d')
post_name = File.dirname(f).split(%r{/}).last.gsub(/\A\d+-/, '')
name = "#{formatted_date}-#{post_name}"
File.open("_posts/#{name}.markdown", "w") do |f|
f.puts meta.to_yaml
f.puts "---\n"
f.puts "\n#{prerex}\n\n" if prerex
f.puts body
end
posts += 1
end
"Created #{posts} posts!"
end
end
end

84
lib/jekyll/migrators/mephisto.rb
View File

@ -1,84 +0,0 @@
# Quickly hacked together my Michael Ivey
# Based on mt.rb by Nick Gerakines, open source and publically
# available under the MIT license. Use this module at your own risk.
require 'rubygems'
require 'sequel'
require 'fastercsv'
require 'fileutils'
require File.join(File.dirname(__FILE__),"csv.rb")
# NOTE: This converter requires Sequel and the MySQL gems.
# The MySQL gem can be difficult to install on OS X. Once you have MySQL
# installed, running the following commands should work:
# $ sudo gem install sequel
# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
module Jekyll
module Mephisto
#Accepts a hash with database config variables, exports mephisto posts into a csv
#export PGPASSWORD if you must
def self.postgres(c)
sql = <<-SQL
BEGIN;
CREATE TEMP TABLE jekyll AS
SELECT title, permalink, body, published_at, filter FROM contents
WHERE user_id = 1 AND type = 'Article' ORDER BY published_at;
COPY jekyll TO STDOUT WITH CSV HEADER;
ROLLBACK;
SQL
command = %Q(psql -h #{c[:host] || "localhost"} -c "#{sql.strip}" #{c[:database]} #{c[:username]} -o #{c[:filename] || "posts.csv"})
puts command
`#{command}`
CSV.process
end
# This query will pull blog posts from all entries across all blogs. If
# you've got unpublished, deleted or otherwise hidden posts please sift
# through the created posts to make sure nothing is accidently published.
QUERY = "SELECT id, \
permalink, \
body, \
published_at, \
title \
FROM contents \
WHERE user_id = 1 AND \
type = 'Article' AND \
published_at IS NOT NULL \
ORDER BY published_at"
def self.process(dbname, user, pass, host = 'localhost')
db = Sequel.mysql(dbname, :user => user,
:password => pass,
:host => host,
:encoding => 'utf8')
FileUtils.mkdir_p "_posts"
db[QUERY].each do |post|
title = post[:title]
slug = post[:permalink]
date = post[:published_at]
content = post[:body]
# Ideally, this script would determine the post format (markdown,
# html, etc) and create files with proper extensions. At this point
# it just assumes that markdown will be acceptable.
name = [date.year, date.month, date.day, slug].join('-') + ".markdown"
data = {
'layout' => 'post',
'title' => title.to_s,
'mt_id' => post[:entry_id],
}.delete_if { |k,v| v.nil? || v == ''}.to_yaml
File.open("_posts/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
end
end
end
end

86
lib/jekyll/migrators/mt.rb
View File

@ -1,86 +0,0 @@
# Created by Nick Gerakines, open source and publically available under the
# MIT license. Use this module at your own risk.
# I'm an Erlang/Perl/C++ guy so please forgive my dirty ruby.
require 'rubygems'
require 'sequel'
require 'fileutils'
require 'yaml'
# NOTE: This converter requires Sequel and the MySQL gems.
# The MySQL gem can be difficult to install on OS X. Once you have MySQL
# installed, running the following commands should work:
# $ sudo gem install sequel
# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
module Jekyll
module MT
# This query will pull blog posts from all entries across all blogs. If
# you've got unpublished, deleted or otherwise hidden posts please sift
# through the created posts to make sure nothing is accidently published.
QUERY = "SELECT entry_id, \
entry_basename, \
entry_text, \
entry_text_more, \
entry_authored_on, \
entry_title, \
entry_convert_breaks \
FROM mt_entry"
def self.process(dbname, user, pass, host = 'localhost')
db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host, :encoding => 'utf8')
FileUtils.mkdir_p "_posts"
db[QUERY].each do |post|
title = post[:entry_title]
slug = post[:entry_basename].gsub(/_/, '-')
date = post[:entry_authored_on]
content = post[:entry_text]
more_content = post[:entry_text_more]
entry_convert_breaks = post[:entry_convert_breaks]
# Be sure to include the body and extended body.
if more_content != nil
content = content + " \n" + more_content
end
# Ideally, this script would determine the post format (markdown,
# html, etc) and create files with proper extensions. At this point
# it just assumes that markdown will be acceptable.
name = [date.year, date.month, date.day, slug].join('-') + '.' +
self.suffix(entry_convert_breaks)
data = {
'layout' => 'post',
'title' => title.to_s,
'mt_id' => post[:entry_id],
'date' => date
}.delete_if { |k,v| v.nil? || v == '' }.to_yaml
File.open("_posts/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
end
end
def self.suffix(entry_type)
if entry_type.nil? || entry_type.include?("markdown")
# The markdown plugin I have saves this as
# "markdown_with_smarty_pants", so I just look for "markdown".
"markdown"
elsif entry_type.include?("textile")
# This is saved as "textile_2" on my installation of MT 5.1.
"textile"
elsif entry_type == "0" || entry_type.include?("richtext")
# Richtext looks to me like it's saved as HTML, so I include it here.
"html"
else
# Other values might need custom work.
entry_type
end
end
end
end

67
lib/jekyll/migrators/posterous.rb
View File

@ -1,67 +0,0 @@
require 'rubygems'
require 'jekyll'
require 'fileutils'
require 'net/http'
require 'uri'
require "json"
# ruby -r './lib/jekyll/migrators/posterous.rb' -e 'Jekyll::Posterous.process(email, pass, api_key, blog)'
module Jekyll
module Posterous
def self.fetch(uri_str, limit = 10)
# You should choose better exception.
raise ArgumentError, 'Stuck in a redirect loop. Please double check your email and password' if limit == 0
response = nil
Net::HTTP.start('posterous.com') do |http|
req = Net::HTTP::Get.new(uri_str)
req.basic_auth @email, @pass
response = http.request(req)
end
case response
when Net::HTTPSuccess then response
when Net::HTTPRedirection then fetch(response['location'], limit - 1)
else response.error!
end
end
def self.process(email, pass, api_token, blog = 'primary')
@email, @pass, @api_token = email, pass, api_token
FileUtils.mkdir_p "_posts"
posts = JSON.parse(self.fetch("/api/v2/users/me/sites/#{blog}/posts?api_token=#{@api_token}").body)
page = 1
while posts.any?
posts.each do |post|
title = post["title"]
slug = title.gsub(/[^[:alnum:]]+/, '-').downcase
date = Date.parse(post["display_date"])
content = post["body_html"]
published = !post["is_private"]
name = "%02d-%02d-%02d-%s.html" % [date.year, date.month, date.day, slug]
# Get the relevant fields as a hash, delete empty fields and convert
# to YAML for the header
data = {
'layout' => 'post',
'title' => title.to_s,
'published' => published
}.delete_if { |k,v| v.nil? || v == ''}.to_yaml
# Write out the data and content to file
File.open("_posts/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
end
page += 1
posts = JSON.parse(self.fetch("/api/v2/users/me/sites/#{blog}/posts?api_token=#{@api_token}&page=#{page}").body)
end
end
end
end

47
lib/jekyll/migrators/rss.rb
View File

@ -1,47 +0,0 @@
# Created by Kendall Buchanan (https://github.com/kendagriff) on 2011-12-22.
# Use at your own risk. The end.
#
# Usage:
# (URL)
# ruby -r '_import/rss.rb' -e "Jekyll::MigrateRSS.process('http://yourdomain.com/your-favorite-feed.xml')"
#
# (Local file)
# ruby -r '_import/rss.rb' -e "Jekyll::MigrateRSS.process('./somefile/on/your/computer.xml')"
require 'rubygems'
require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
require 'fileutils'
require 'yaml'
module Jekyll
module MigrateRSS
# The `source` argument may be a URL or a local file.
def self.process(source)
content = ""
open(source) { |s| content = s.read }
rss = RSS::Parser.parse(content, false)
raise "There doesn't appear to be any RSS items at the source (#{source}) provided." unless rss
rss.items.each do |item|
formatted_date = item.date.strftime('%Y-%m-%d')
post_name = item.title.split(%r{ |!|/|:|&|-|$|,}).map { |i| i.downcase if i != '' }.compact.join('-')
name = "#{formatted_date}-#{post_name}"
header = {
'layout' => 'post',
'title' => item.title
}
File.open("_posts/#{name}.html", "w") do |f|
f.puts header.to_yaml
f.puts "---\n"
f.puts item.description
end
end
end
end
end

58
lib/jekyll/migrators/textpattern.rb
View File

@ -1,58 +0,0 @@
require 'rubygems'
require 'sequel'
require 'fileutils'
require 'yaml'
# NOTE: This converter requires Sequel and the MySQL gems.
# The MySQL gem can be difficult to install on OS X. Once you have MySQL
# installed, running the following commands should work:
# $ sudo gem install sequel
# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
module Jekyll
module TextPattern
# Reads a MySQL database via Sequel and creates a post file for each post.
# The only posts selected are those with a status of 4 or 5, which means
# "live" and "sticky" respectively.
# Other statuses are 1 => draft, 2 => hidden and 3 => pending.
QUERY = "SELECT Title, \
url_title, \
Posted, \
Body, \
Keywords \
FROM textpattern \
WHERE Status = '4' OR \
Status = '5'"
def self.process(dbname, user, pass, host = 'localhost')
db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host, :encoding => 'utf8')
FileUtils.mkdir_p "_posts"
db[QUERY].each do |post|
# Get required fields and construct Jekyll compatible name.
title = post[:Title]
slug = post[:url_title]
date = post[:Posted]
content = post[:Body]
name = [date.strftime("%Y-%m-%d"), slug].join('-') + ".textile"
# Get the relevant fields as a hash, delete empty fields and convert
# to YAML for the header.
data = {
'layout' => 'post',
'title' => title.to_s,
'tags' => post[:Keywords].split(',')
}.delete_if { |k,v| v.nil? || v == ''}.to_yaml
# Write out the data and content to file.
File.open("_posts/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
end
end
end
end

201
lib/jekyll/migrators/tumblr.rb
View File

@ -1,201 +0,0 @@
require 'rubygems'
require 'open-uri'
require 'fileutils'
require 'nokogiri'
require 'date'
require 'json'
require 'uri'
require 'jekyll'
module Jekyll
module Tumblr
def self.process(url, format = "html", grab_images = false,
add_highlights = false, rewrite_urls = true)
@grab_images = grab_images
FileUtils.mkdir_p "_posts/tumblr"
url += "/api/read/json/"
per_page = 50
posts = []
# Two passes are required so that we can rewrite URLs.
# First pass builds up an array of each post as a hash.
begin
current_page = (current_page || -1) + 1
feed = open(url + "?num=#{per_page}&start=#{current_page * per_page}")
json = feed.readlines.join("\n")[21...-2] # Strip Tumblr's JSONP chars.
blog = JSON.parse(json)
puts "Page: #{current_page + 1} - Posts: #{blog["posts"].size}"
posts += blog["posts"].map { |post| post_to_hash(post, format) }
end until blog["posts"].size < per_page
# Rewrite URLs and create redirects.
posts = rewrite_urls_and_redirects posts if rewrite_urls
# Second pass for writing post files.
posts.each do |post|
if format == "md"
post[:content] = html_to_markdown post[:content]
post[:content] = add_syntax_highlights post[:content] if add_highlights
end
post[:name] = truncate_post_name post[:name] if post[:name].size > 255
File.open("_posts/tumblr/#{post[:name]}", "w") do |f|
f.puts post[:header].to_yaml + "---\n" + post[:content]
end
end
end
private
def self.truncate_post_name name
post = name.match(/^(.+)\.(.+)$/).captures
post[0][0..(-1 - post[1].size)] + post[1].size
end
# Converts each type of Tumblr post to a hash with all required
# data for Jekyll.
def self.post_to_hash(post, format)
case post['type']
when "regular"
title = post["regular-title"]
content = post["regular-body"]
when "link"
title = post["link-text"] || post["link-url"]
content = "<a href=\"#{post["link-url"]}\">#{title}</a>"
unless post["link-description"].nil?
content << "<br/>" + post["link-description"]
end
when "photo"
title = post["photo-caption"]
max_size = post.keys.map{ |k| k.gsub("photo-url-", "").to_i }.max
url = post["photo-url"] || post["photo-url-#{max_size}"]
ext = "." + post[post.keys.select { |k|
k =~ /^photo-url-/ && post[k].split("/").last =~ /\./
}.first].split(".").last
content = "<img src=\"#{save_file(url, ext)}\"/>"
unless post["photo-link-url"].nil?
content = "<a href=\"#{post["photo-link-url"]}\">#{content}</a>"
end
when "audio"
if !post["id3-title"].nil?
title = post["id3-title"]
content = post.at["audio-player"] + "<br/>" + post["audio-caption"]
else
title = post["audio-caption"]
content = post.at["audio-player"]
end
when "quote"
title = post["quote-text"]
content = "<blockquote>#{post["quote-text"]}</blockquote>"
unless post["quote-source"].nil?
content << "&#8212;" + post["quote-source"]
end
when "conversation"
title = post["conversation-title"]
content = "<section><dialog>"
post["conversation"]["line"].each do |line|
content << "<dt>#{line['label']}</dt><dd>#{line}</dd>"
end
content << "</section></dialog>"
when "video"
title = post["video-title"]
content = post["video-player"]
unless post["video-caption"].nil?
content << "<br/>" + post["video-caption"]
end
end
date = Date.parse(post['date']).to_s
title = Nokogiri::HTML(title).text
slug = title.downcase.strip.gsub(' ', '-').gsub(/[^\w-]/, '')
{
:name => "#{date}-#{slug}.#{format}",
:header => {
"layout" => "post",
"title" => title,
"tags" => post["tags"],
},
:content => content,
:url => post["url"],
:slug => post["url-with-slug"],
}
end
# Create a Hash of old urls => new urls, for rewriting and
# redirects, and replace urls in each post. Instantiate Jekyll
# site/posts to get the correct permalink format.
def self.rewrite_urls_and_redirects(posts)
site = Jekyll::Site.new(Jekyll.configuration({}))
dir = File.join(File.dirname(__FILE__), "..")
urls = Hash[posts.map { |post|
# Create an initial empty file for the post so that
# we can instantiate a post object.
File.open("_posts/tumblr/#{post[:name]}", "w")
tumblr_url = URI.parse(post[:slug]).path
jekyll_url = Jekyll::Post.new(site, dir, "", "tumblr/" + post[:name]).url
redirect_dir = tumblr_url.sub(/\//, "") + "/"
FileUtils.mkdir_p redirect_dir
File.open(redirect_dir + "index.html", "w") do |f|
f.puts "<html><head><meta http-equiv='Refresh' content='0; " +
"url=#{jekyll_url}'></head><body></body></html>"
end
[tumblr_url, jekyll_url]
}]
posts.map { |post|
urls.each do |tumblr_url, jekyll_url|
post[:content].gsub!(/#{tumblr_url}/i, jekyll_url)
end
post
}
end
# Uses Python's html2text to convert a post's content to
# markdown. Preserve HTML tables as per the markdown docs.
def self.html_to_markdown(content)
preserve = ["table", "tr", "th", "td"]
preserve.each do |tag|
content.gsub!(/<#{tag}/i, "$$" + tag)
content.gsub!(/<\/#{tag}/i, "||" + tag)
end
content = %x[echo '#{content.gsub("'", "''")}' | html2text]
preserve.each do |tag|
content.gsub!("$$" + tag, "<" + tag)
content.gsub!("||" + tag, "</" + tag)
end
content
end
# Adds pygments highlight tags to code blocks in posts that use
# markdown format. This doesn't guess the language of the code
# block, so you should modify this to suit your own content.
# For example, my code block only contain Python and JavaScript,
# so I can assume the block is JavaScript if it contains a
# semi-colon.
def self.add_syntax_highlights(content)
lines = content.split("\n")
block, indent, lang, start = false, /^ /, nil, nil
lines.each_with_index do |line, i|
if !block && line =~ indent
block = true
lang = "python"
start = i
elsif block
lang = "javascript" if line =~ /;$/
block = line =~ indent && i < lines.size - 1 # Also handle EOF
if !block
lines[start] = "{% highlight #{lang} %}"
lines[i - 1] = "{% endhighlight %}"
end
lines[i] = lines[i].sub(indent, "")
end
end
lines.join("\n")
end
def self.save_file(url, ext)
if @grab_images
path = "tumblr_files/#{url.split('/').last}"
path += ext unless path =~ /#{ext}$/
FileUtils.mkdir_p "tumblr_files"
File.open(path, "w") { |f| f.write(open(url).read) }
url = "/" + path
end
url
end
end
end

51
lib/jekyll/migrators/typo.rb
View File

@ -1,51 +0,0 @@
# Author: Toby DiPasquale <toby@cbcg.net>
require 'fileutils'
require 'rubygems'
require 'sequel'
require 'yaml'
module Jekyll
module Typo
# This SQL *should* work for both MySQL and PostgreSQL, but I haven't
# tested PostgreSQL yet (as of 2008-12-16).
SQL = <<-EOS
SELECT c.id id,
c.title title,
c.permalink slug,
c.body body,
c.published_at date,
c.state state,
COALESCE(tf.name, 'html') filter
FROM contents c
LEFT OUTER JOIN text_filters tf
ON c.text_filter_id = tf.id
EOS
def self.process dbname, user, pass, host='localhost'
FileUtils.mkdir_p '_posts'
db = Sequel.mysql(dbname, :user => user, :password => pass, :host => host, :encoding => 'utf8')
db[SQL].each do |post|
next unless post[:state] =~ /published/
name = [ sprintf("%.04d", post[:date].year),
sprintf("%.02d", post[:date].month),
sprintf("%.02d", post[:date].day),
post[:slug].strip ].join('-')
# Can have more than one text filter in this field, but we just want
# the first one for this.
name += '.' + post[:filter].split(' ')[0]
File.open("_posts/#{name}", 'w') do |f|
f.puts({ 'layout' => 'post',
'title' => post[:title].to_s,
'typo_id' => post[:id]
}.delete_if { |k, v| v.nil? || v == '' }.to_yaml)
f.puts '---'
f.puts post[:body].delete("\r")
end
end
end
end
end

294
lib/jekyll/migrators/wordpress.rb
View File

@ -1,294 +0,0 @@
require 'rubygems'
require 'sequel'
require 'fileutils'
require 'yaml'
# NOTE: This converter requires Sequel and the MySQL gems.
# The MySQL gem can be difficult to install on OS X. Once you have MySQL
# installed, running the following commands should work:
# $ sudo gem install sequel
# $ sudo gem install mysql -- --with-mysql-config=/usr/local/mysql/bin/mysql_config
module Jekyll
module WordPress
# Main migrator function. Call this to perform the migration.
#
# dbname:: The name of the database
# user:: The database user name
# pass:: The database user's password
# host:: The address of the MySQL database host. Default: 'localhost'
# options:: A hash table of configuration options.
#
# Supported options are:
#
# :table_prefix:: Prefix of database tables used by WordPress.
# Default: 'wp_'
# :clean_entities:: If true, convert non-ASCII characters to HTML
# entities in the posts, comments, titles, and
# names. Requires the 'htmlentities' gem to
# work. Default: true.
# :comments:: If true, migrate post comments too. Comments
# are saved in the post's YAML front matter.
# Default: true.
# :categories:: If true, save the post's categories in its
# YAML front matter.
# :tags:: If true, save the post's tags in its
# YAML front matter.
# :more_excerpt:: If true, when a post has no excerpt but
# does have a <!-- more --> tag, use the
# preceding post content as the excerpt.
# Default: true.
# :more_anchor:: If true, convert a <!-- more --> tag into
# two HTML anchors with ids "more" and
# "more-NNN" (where NNN is the post number).
# Default: true.
# :status:: Array of allowed post statuses. Only
# posts with matching status will be migrated.
# Known statuses are :publish, :draft, :private,
# and :revision. If this is nil or an empty
# array, all posts are migrated regardless of
# status. Default: [:publish].
#
def self.process(dbname, user, pass, host='localhost', options={})
options = {
:table_prefix => 'wp_',
:clean_entities => true,
:comments => true,
:categories => true,
:tags => true,
:more_excerpt => true,
:more_anchor => true,
:status => [:publish] # :draft, :private, :revision
}.merge(options)
if options[:clean_entities]
begin
require 'htmlentities'
rescue LoadError
STDERR.puts "Could not require 'htmlentities', so the " +
":clean_entities option is now disabled."
options[:clean_entities] = false
end
end
FileUtils.mkdir_p("_posts")
db = Sequel.mysql(dbname, :user => user, :password => pass,
:host => host, :encoding => 'utf8')
px = options[:table_prefix]
posts_query = "
SELECT
posts.ID AS `id`,
posts.guid AS `guid`,
posts.post_type AS `type`,
posts.post_status AS `status`,
posts.post_title AS `title`,
posts.post_name AS `slug`,
posts.post_date AS `date`,
posts.post_content AS `content`,
posts.post_excerpt AS `excerpt`,
posts.comment_count AS `comment_count`,
users.display_name AS `author`,
users.user_login AS `author_login`,
users.user_email AS `author_email`,
users.user_url AS `author_url`
FROM #{px}posts AS `posts`
LEFT JOIN #{px}users AS `users`
ON posts.post_author = users.ID"
if options[:status] and not options[:status].empty?
status = options[:status][0]
posts_query << "
WHERE posts.post_status = '#{status.to_s}'"
options[:status][1..-1].each do |status|
posts_query << " OR
posts.post_status = '#{status.to_s}'"
end
end
db[posts_query].each do |post|
process_post(post, db, options)
end
end
def self.process_post(post, db, options)
px = options[:table_prefix]
title = post[:title]
if options[:clean_entities]
title = clean_entities(title)
end
slug = post[:slug]
if !slug or slug.empty?
slug = sluggify(title)
end
date = post[:date] || Time.now
name = "%02d-%02d-%02d-%s.markdown" % [date.year, date.month,
date.day, slug]
content = post[:content].to_s
if options[:clean_entities]
content = clean_entities(content)
end
excerpt = post[:excerpt].to_s
more_index = content.index(/<!-- *more *-->/)
more_anchor = nil
if more_index
if options[:more_excerpt] and
(post[:excerpt].nil? or post[:excerpt].empty?)
excerpt = content[0...more_index]
end
if options[:more_anchor]
more_link = "more"
content.sub!(/<!-- *more *-->/,
"<a id=\"more\"></a>" +
"<a id=\"more-#{post[:id]}\"></a>")
end
end
categories = []
tags = []
if options[:categories] or options[:tags]
cquery =
"SELECT
terms.name AS `name`,
ttax.taxonomy AS `type`
FROM
#{px}terms AS `terms`,
#{px}term_relationships AS `trels`,
#{px}term_taxonomy AS `ttax`
WHERE
trels.object_id = '#{post[:id]}' AND
trels.term_taxonomy_id = ttax.term_taxonomy_id AND
terms.term_id = ttax.term_id"
db[cquery].each do |term|
if options[:categories] and term[:type] == "category"
if options[:clean_entities]
categories << clean_entities(term[:name])
else
categories << term[:name]
end
elsif options[:tags] and term[:type] == "post_tag"
if options[:clean_entities]
tags << clean_entities(term[:name])
else
tags << term[:name]
end
end
end
end
comments = []
if options[:comments] and post[:comment_count].to_i > 0
cquery =
"SELECT
comment_ID AS `id`,
comment_author AS `author`,
comment_author_email AS `author_email`,
comment_author_url AS `author_url`,
comment_date AS `date`,
comment_date_gmt AS `date_gmt`,
comment_content AS `content`
FROM #{px}comments
WHERE
comment_post_ID = '#{post[:id]}' AND
comment_approved != 'spam'"
db[cquery].each do |comment|
comcontent = comment[:content].to_s
if comcontent.respond_to?(:force_encoding)
comcontent.force_encoding("UTF-8")
end
if options[:clean_entities]
comcontent = clean_entities(comcontent)
end
comauthor = comment[:author].to_s
if options[:clean_entities]
comauthor = clean_entities(comauthor)
end
comments << {
'id' => comment[:id].to_i,
'author' => comauthor,
'author_email' => comment[:author_email].to_s,
'author_url' => comment[:author_url].to_s,
'date' => comment[:date].to_s,
'date_gmt' => comment[:date_gmt].to_s,
'content' => comcontent,
}
end
comments.sort!{ |a,b| a['id'] <=> b['id'] }
end
# Get the relevant fields as a hash, delete empty fields and
# convert to YAML for the header.
data = {
'layout' => post[:type].to_s,
'status' => post[:status].to_s,
'published' => (post[:status].to_s == "publish"),
'title' => title.to_s,
'author' => post[:author].to_s,
'author_login' => post[:author_login].to_s,
'author_email' => post[:author_email].to_s,
'author_url' => post[:author_url].to_s,
'excerpt' => excerpt,
'more_anchor' => more_anchor,
'wordpress_id' => post[:id],
'wordpress_url' => post[:guid].to_s,
'date' => date,
'categories' => options[:categories] ? categories : nil,
'tags' => options[:tags] ? tags : nil,
'comments' => options[:comments] ? comments : nil,
}.delete_if { |k,v| v.nil? || v == '' }.to_yaml
# Write out the data and content to file
File.open("_posts/#{name}", "w") do |f|
f.puts data
f.puts "---"
f.puts content
end
end
def self.clean_entities( text )
if text.respond_to?(:force_encoding)
text.force_encoding("UTF-8")
end
text = HTMLEntities.new.encode(text, :named)
# We don't want to convert these, it would break all
# HTML tags in the post and comments.
text.gsub!("&amp;", "&")
text.gsub!("&lt;", "<")
text.gsub!("&gt;", ">")
text.gsub!("&quot;", '"')
text.gsub!("&apos;", "'")
text
end
def self.sluggify( title )
begin
require 'unidecode'
title = title.to_ascii
rescue LoadError
STDERR.puts "Could not require 'unidecode'. If your post titles have non-ASCII characters, you could get nicer permalinks by installing unidecode."
end
title.downcase.gsub(/[^0-9A-Za-z]+/, " ").strip.gsub(" ", "-")
end
end
end

70
lib/jekyll/migrators/wordpressdotcom.rb
View File

@ -1,70 +0,0 @@
# coding: utf-8
require 'rubygems'
require 'hpricot'
require 'fileutils'
require 'yaml'
require 'time'
module Jekyll
# This importer takes a wordpress.xml file, which can be exported from your
# wordpress.com blog (/wp-admin/export.php).
module WordpressDotCom
def self.process(filename = "wordpress.xml")
import_count = Hash.new(0)
doc = Hpricot::XML(File.read(filename))
(doc/:channel/:item).each do |item|
title = item.at(:title).inner_text.strip
permalink_title = item.at('wp:post_name').inner_text
# Fallback to "prettified" title if post_name is empty (can happen)
if permalink_title == ""
permalink_title = title.downcase.split.join('-')
end
date = Time.parse(item.at('wp:post_date').inner_text)
status = item.at('wp:status').inner_text
if status == "publish"
published = true
else
published = false
end
type = item.at('wp:post_type').inner_text
tags = (item/:category).map{|c| c.inner_text}.reject{|c| c == 'Uncategorized'}.uniq
metas = Hash.new
item.search("wp:postmeta").each do |meta|
key = meta.at('wp:meta_key').inner_text
value = meta.at('wp:meta_value').inner_text
metas[key] = value;
end
name = "#{date.strftime('%Y-%m-%d')}-#{permalink_title}.html"
header = {
'layout' => type,
'title' => title,
'tags' => tags,
'status' => status,
'type' => type,
'published' => published,
'meta' => metas
}
FileUtils.mkdir_p "_#{type}s"
File.open("_#{type}s/#{name}", "w") do |f|
f.puts header.to_yaml
f.puts '---'
f.puts item.at('content:encoded').inner_text
end
import_count[type] += 1
end
import_count.each do |key, value|
puts "Imported #{value} #{key}s"
end
end
end
end

3
lib/jekyll/page.rb
View File

@ -1,5 +1,4 @@
module Jekyll
class Page
include Convertible
@ -161,7 +160,5 @@ module Jekyll
def index?
basename == 'index'
end
end
end

2
lib/jekyll/plugin.rb
View File

@ -1,5 +1,4 @@
module Jekyll
class Plugin
PRIORITIES = { :lowest => -100,
:low => -10,
@ -73,5 +72,4 @@ module Jekyll
# no-op for default
end
end
end

4
lib/jekyll/post.rb
View File

@ -1,5 +1,4 @@
module Jekyll
class Post
include Comparable
include Convertible
@ -154,7 +153,7 @@ module Jekyll
"title" => CGI.escape(slug),
"i_day" => date.strftime("%d").to_i.to_s,
"i_month" => date.strftime("%m").to_i.to_s,
"categories" => categories.map { |c| URI.escape(c) }.join('/'),
"categories" => categories.map { |c| URI.escape(c.to_s) }.join('/'),
"output_ext" => self.output_ext
}.inject(template) { |result, token|
result.gsub(/:#{Regexp.escape token.first}/, token.last)
@ -281,5 +280,4 @@ module Jekyll
end
end
end
end

1
lib/jekyll/site.rb
View File

@ -1,7 +1,6 @@
require 'set'
module Jekyll
class Site
attr_accessor :config, :layouts, :posts, :pages, :static_files,
:categories, :exclude, :include, :source, :dest, :lsi, :pygments,

2
lib/jekyll/static_file.rb
View File

@ -1,5 +1,4 @@
module Jekyll
class StaticFile
# The cache of last modification times [path] -> mtime.
@@mtimes = Hash.new
@ -68,5 +67,4 @@ module Jekyll
nil
end
end
end

19
lib/jekyll/tags/gist.rb Normal file
View File

@ -0,0 +1,19 @@
# Gist Liquid Tag
#
# Example:
# {% gist 1234567 %}
module Jekyll
class GistTag < Liquid::Tag
def initialize(tag_name, gist, tokens)
super
@gist = gist.strip
end
def render(context)
"<script src=\"https://gist.github.com/#{@gist}.js\"> </script>"
end
end
end
Liquid::Template.register_tag('gist', Jekyll::GistTag)

126
lib/jekyll/tags/highlight.rb
View File

@ -1,77 +1,77 @@
module Jekyll
module Tags
class HighlightBlock < Liquid::Block
include Liquid::StandardFilters
class HighlightBlock < Liquid::Block
include Liquid::StandardFilters
# The regular expression syntax checker. Start with the language specifier.
# Follow that by zero or more space separated options that take one of two
# forms:
#
# 1. name
# 2. name=value
SYNTAX = /^([a-zA-Z0-9.+#-]+)((\s+\w+(=\w+)?)*)$/
# The regular expression syntax checker. Start with the language specifier.
# Follow that by zero or more space separated options that take one of two
# forms:
#
# 1. name
# 2. name=value
SYNTAX = /^([a-zA-Z0-9.+#-]+)((\s+\w+(=\w+)?)*)$/
def initialize(tag_name, markup, tokens)
super
if markup.strip =~ SYNTAX
@lang = $1
@options = {}
if defined?($2) && $2 != ''
$2.split.each do |opt|
key, value = opt.split('=')
if value.nil?
if key == 'linenos'
value = 'inline'
else
value = true
def initialize(tag_name, markup, tokens)
super
if markup.strip =~ SYNTAX
@lang = $1
@options = {}
if defined?($2) && $2 != ''
$2.split.each do |opt|
key, value = opt.split('=')
if value.nil?
if key == 'linenos'
value = 'inline'
else
value = true
end
end
@options[key] = value
end
@options[key] = value
end
else
raise SyntaxError.new("Syntax Error in 'highlight' - Valid syntax: highlight <lang> [linenos]")
end
else
raise SyntaxError.new("Syntax Error in 'highlight' - Valid syntax: highlight <lang> [linenos]")
end
end
def render(context)
if context.registers[:site].pygments
render_pygments(context, super)
else
render_codehighlighter(context, super)
def render(context)
if context.registers[:site].pygments
render_pygments(context, super)
else
render_codehighlighter(context, super)
end
end
def render_pygments(context, code)
@options[:encoding] = 'utf-8'
output = add_code_tags(
Pygments.highlight(code, :lexer => @lang, :options => @options),
@lang
)
output = context["pygments_prefix"] + output if context["pygments_prefix"]
output = output + context["pygments_suffix"] if context["pygments_suffix"]
output
end
def render_codehighlighter(context, code)
#The div is required because RDiscount blows ass
<<-HTML
<div>
<pre><code class='#{@lang}'>#{h(code).strip}</code></pre>
</div>
HTML
end
def add_code_tags(code, lang)
# Add nested <code> tags to code blocks
code = code.sub(/<pre>/,'<pre><code class="' + lang + '">')
code = code.sub(/<\/pre>/,"</code></pre>")
end
end
def render_pygments(context, code)
@options[:encoding] = 'utf-8'
output = add_code_tags(
Pygments.highlight(code, :lexer => @lang, :options => @options),
@lang
)
output = context["pygments_prefix"] + output if context["pygments_prefix"]
output = output + context["pygments_suffix"] if context["pygments_suffix"]
output
end
def render_codehighlighter(context, code)
#The div is required because RDiscount blows ass
<<-HTML
<div>
<pre><code class='#{@lang}'>#{h(code).strip}</code></pre>
</div>
HTML
end
def add_code_tags(code, lang)
# Add nested <code> tags to code blocks
code = code.sub(/<pre>/,'<pre><code class="' + lang + '">')
code = code.sub(/<\/pre>/,"</code></pre>")
end
end
end
Liquid::Template.register_tag('highlight', Jekyll::HighlightBlock)
Liquid::Template.register_tag('highlight', Jekyll::Tags::HighlightBlock)

52
lib/jekyll/tags/include.rb
View File

@ -1,37 +1,37 @@
module Jekyll
class IncludeTag < Liquid::Tag
def initialize(tag_name, file, tokens)
super
@file = file.strip
end
def render(context)
includes_dir = File.join(context.registers[:site].source, '_includes')
if File.symlink?(includes_dir)
return "Includes directory '#{includes_dir}' cannot be a symlink"
module Tags
class IncludeTag < Liquid::Tag
def initialize(tag_name, file, tokens)
super
@file = file.strip
end
if @file !~ /^[a-zA-Z0-9_\/\.-]+$/ || @file =~ /\.\// || @file =~ /\/\./
return "Include file '#{@file}' contains invalid characters or sequences"
end
def render(context)
includes_dir = File.join(context.registers[:site].source, '_includes')
Dir.chdir(includes_dir) do
choices = Dir['**/*'].reject { |x| File.symlink?(x) }
if choices.include?(@file)
source = File.read(@file)
partial = Liquid::Template.parse(source)
context.stack do
partial.render(context)
if File.symlink?(includes_dir)
return "Includes directory '#{includes_dir}' cannot be a symlink"
end
if @file !~ /^[a-zA-Z0-9_\/\.-]+$/ || @file =~ /\.\// || @file =~ /\/\./
return "Include file '#{@file}' contains invalid characters or sequences"
end
Dir.chdir(includes_dir) do
choices = Dir['**/*'].reject { |x| File.symlink?(x) }
if choices.include?(@file)
source = File.read(@file)
partial = Liquid::Template.parse(source)
context.stack do
partial.render(context)
end
else
"Included file '#{@file}' not found in _includes directory"
end
else
"Included file '#{@file}' not found in _includes directory"
end
end
end
end
end
Liquid::Template.register_tag('include', Jekyll::IncludeTag)
Liquid::Template.register_tag('include', Jekyll::Tags::IncludeTag)

53
lib/jekyll/tags/post_url.rb
View File

@ -1,38 +1,39 @@
module Jekyll
module Tags
class PostComparer
MATCHER = /^(.+\/)*(\d+-\d+-\d+)-(.*)$/
class PostComparer
MATCHER = /^(.+\/)*(\d+-\d+-\d+)-(.*)$/
attr_accessor :date, :slug
attr_accessor :date, :slug
def initialize(name)
who, cares, date, slug = *name.match(MATCHER)
@slug = slug
@date = Time.parse(date)
end
end
class PostUrl < Liquid::Tag
def initialize(tag_name, post, tokens)
super
@orig_post = post.strip
@post = PostComparer.new(@orig_post)
def initialize(name)
who, cares, date, slug = *name.match(MATCHER)
@slug = slug
@date = Time.parse(date)
end
end
def render(context)
site = context.registers[:site]
site.posts.each do |p|
if p == @post
return p.url
end
class PostUrl < Liquid::Tag
def initialize(tag_name, post, tokens)
super
@orig_post = post.strip
@post = PostComparer.new(@orig_post)
end
puts "ERROR: post_url: \"#{@orig_post}\" could not be found"
def render(context)
site = context.registers[:site]
return "#"
site.posts.each do |p|
if p == @post
return p.url
end
end
puts "ERROR: post_url: \"#{@orig_post}\" could not be found"
return "#"
end
end
end
end
Liquid::Template.register_tag('post_url', Jekyll::PostUrl)
Liquid::Template.register_tag('post_url', Jekyll::Tags::PostUrl)

2
script/bootstrap Executable file
View File

@ -0,0 +1,2 @@
#!/bin/sh
bundle install

4
test/fixtures/exploit_front_matter.erb vendored Normal file
View File

@ -0,0 +1,4 @@
---
test: !ruby/hash:DoesNotExist {}
---
Real content starts here

7
test/helper.rb
View File

@ -1,4 +1,11 @@
if RUBY_VERSION > '1.9' && ENV["COVERAGE"] == "true"
require 'simplecov'
require 'simplecov-gem-adapter'
SimpleCov.start('gem')
end
require 'rubygems'
require 'test/unit'
gem 'RedCloth', '>= 4.2.1'
require 'jekyll'

15
test/test_configuration.rb
View File

@ -7,22 +7,21 @@ class TestConfiguration < Test::Unit::TestCase
end
should "fire warning with no _config.yml" do
mock(YAML).load_file(@path) { raise "No such file or directory - #{@path}" }
mock($stderr).puts("WARNING: Could not read configuration. Using defaults (and options).")
mock($stderr).puts("\tNo such file or directory - #{@path}")
mock(YAML).safe_load_file(@path) { raise SystemCallError, "No such file or directory - #{@path}" }
mock($stderr).puts("Configuration file: none")
assert_equal Jekyll::DEFAULTS, Jekyll.configuration({})
end
should "load configuration as hash" do
mock(YAML).load_file(@path) { Hash.new }
mock($stdout).puts("Configuration from #{@path}")
mock(YAML).safe_load_file(@path) { Hash.new }
mock($stdout).puts("Configuration file: #{@path}")
assert_equal Jekyll::DEFAULTS, Jekyll.configuration({})
end
should "fire warning with bad config" do
mock(YAML).load_file(@path) { Array.new }
mock($stderr).puts("WARNING: Could not read configuration. Using defaults (and options).")
mock($stderr).puts("\tInvalid configuration - #{@path}")
mock(YAML).safe_load_file(@path) { Array.new }
mock($stderr).puts(" WARNING: Error reading configuration. Using defaults (and options).")
mock($stderr).puts("Configuration file: (INVALID) #{@path}")
assert_equal Jekyll::DEFAULTS, Jekyll.configuration({})
end
end

7
test/test_convertible.rb
View File

@ -29,6 +29,13 @@ class TestConvertible < Test::Unit::TestCase
assert_match(/#{File.join(@base, name)}/, out)
end
should "not allow ruby objects in yaml" do
out = capture_stdout do
@convertible.read_yaml(@base, 'exploit_front_matter.erb')
end
assert_no_match /undefined class\/module DoesNotExist/, out
end
if RUBY_VERSION >= '1.9.2'
should "not parse if there is encoding error in file" do
name = 'broken_front_matter3.erb'

10
test/test_kramdown.rb
View File

@ -17,17 +17,17 @@ class TestKramdown < Test::Unit::TestCase
# http://kramdown.rubyforge.org/converter/html.html#options
should "pass kramdown options" do
markdown = MarkdownConverter.new(@config)
markdown = Converters::Markdown.new(@config)
assert_equal "<h1>Some Header</h1>", markdown.convert('# Some Header #').strip
end
should "convert quotes to smart quotes" do
markdown = MarkdownConverter.new(@config)
assert_equal "<p>&ldquo;Pit&rsquo;hy&rdquo;</p>", markdown.convert(%{"Pit'hy"}).strip
markdown = Converters::Markdown.new(@config)
assert_equal "<p>&#8220;Pit&#8217;hy&#8221;</p>", markdown.convert(%{"Pit'hy"}).strip
override = { 'kramdown' => { 'smart_quotes' => 'lsaquo,rsaquo,laquo,raquo' } }
markdown = MarkdownConverter.new(@config.deep_merge(override))
assert_equal "<p>&laquo;Pit&rsaquo;hy&raquo;</p>", markdown.convert(%{"Pit'hy"}).strip
markdown = Converters::Markdown.new(@config.deep_merge(override))
assert_equal "<p>&#171;Pit&#8250;hy&#187;</p>", markdown.convert(%{"Pit'hy"}).strip
end
end
end

22
test/test_post.rb
View File

@ -128,6 +128,18 @@ class TestPost < Test::Unit::TestCase
end
end
context "with unspecified (date) style and a numeric category" do
setup do
@post.categories << 2013
@post.process(@fake_file)
end
should "process the url correctly" do
assert_equal "/:categories/:year/:month/:day/:title.html", @post.template
assert_equal "/2013/2008/09/09/foo-bar.html", @post.url
end
end
context "with unspecified layout" do
setup do
file = '2013-01-12-no-layout.textile'
@ -445,34 +457,34 @@ class TestPost < Test::Unit::TestCase
should "process .md as markdown under default configuration" do
post = setup_post '2011-04-12-md-extension.md'
conv = post.converter
assert conv.kind_of? Jekyll::MarkdownConverter
assert conv.kind_of? Jekyll::Converters::Markdown
end
should "process .text as indentity under default configuration" do
post = setup_post '2011-04-12-text-extension.text'
conv = post.converter
assert conv.kind_of? Jekyll::IdentityConverter
assert conv.kind_of? Jekyll::Converters::Identity
end
should "process .text as markdown under alternate configuration" do
@site.config['markdown_ext'] = 'markdown,mdw,mdwn,md,text'
post = setup_post '2011-04-12-text-extension.text'
conv = post.converter
assert conv.kind_of? Jekyll::MarkdownConverter
assert conv.kind_of? Jekyll::Converters::Markdown
end
should "process .md as markdown under alternate configuration" do
@site.config['markdown_ext'] = 'markdown,mkd,mkdn,md,text'
post = setup_post '2011-04-12-text-extension.text'
conv = post.converter
assert conv.kind_of? Jekyll::MarkdownConverter
assert conv.kind_of? Jekyll::Converters::Markdown
end
should "process .text as textile under alternate configuration" do
@site.config['textile_ext'] = 'textile,text'
post = setup_post '2011-04-12-text-extension.text'
conv = post.converter
assert conv.kind_of? Jekyll::TextileConverter
assert conv.kind_of? Jekyll::Converters::Textile
end
end

2
test/test_rdiscount.rb
View File

@ -8,7 +8,7 @@ class TestRdiscount < Test::Unit::TestCase
'markdown' => 'rdiscount',
'rdiscount' => { 'extensions' => ['smart', 'generate_toc'], 'toc_token' => '{:toc}' }
}
@markdown = MarkdownConverter.new config
@markdown = Converters::Markdown.new config
end
should "pass rdiscount extensions" do

2
test/test_redcarpet.rb
View File

@ -7,7 +7,7 @@ class TestRedcarpet < Test::Unit::TestCase
'redcarpet' => { 'extensions' => ['smart', 'strikethrough', 'filter_html'] },
'markdown' => 'redcarpet'
}
@markdown = MarkdownConverter.new config
@markdown = Converters::Markdown.new config
end
should "pass redcarpet options" do

12
test/test_redcloth.rb
View File

@ -4,7 +4,7 @@ class TestRedCloth < Test::Unit::TestCase
context "RedCloth default (no explicit config) hard_breaks enabled" do
setup do
@textile = TextileConverter.new
@textile = Converters::Textile.new
end
should "preserve single line breaks in HTML output" do
@ -17,7 +17,7 @@ class TestRedCloth < Test::Unit::TestCase
config = {
'redcloth' => {}
}
@textile = TextileConverter.new config
@textile = Converters::Textile.new config
end
should "preserve single line breaks in HTML output" do
@ -32,7 +32,7 @@ class TestRedCloth < Test::Unit::TestCase
'hard_breaks' => true # default
}
}
@textile = TextileConverter.new config
@textile = Converters::Textile.new config
end
should "preserve single line breaks in HTML output" do
@ -47,7 +47,7 @@ class TestRedCloth < Test::Unit::TestCase
'hard_breaks' => false
}
}
@textile = TextileConverter.new config
@textile = Converters::Textile.new config
end
should "not generate break tags in HTML output" do
@ -62,7 +62,7 @@ class TestRedCloth < Test::Unit::TestCase
'no_span_caps' => false
}
}
@textile = TextileConverter.new config
@textile = Converters::Textile.new config
end
should "generate span tags around capitalized words" do
assert_equal "<p><span class=\"caps\">NSC</span></p>", @textile.convert("NSC").strip
@ -76,7 +76,7 @@ class TestRedCloth < Test::Unit::TestCase
'no_span_caps' => true
}
}
@textile = TextileConverter.new config
@textile = Converters::Textile.new config
end
should "not generate span tags around capitalized words" do

34
test/test_tags.rb
View File

@ -4,7 +4,7 @@ require 'helper'
class TestTags < Test::Unit::TestCase
def create_post(content, override = {}, converter_class = Jekyll::MarkdownConverter)
def create_post(content, override = {}, converter_class = Jekyll::Converters::Markdown)
stub(Jekyll).configuration do
Jekyll::DEFAULTS.deep_merge({'pygments' => true}).deep_merge(override)
end
@ -39,7 +39,7 @@ CONTENT
context "language name" do
should "match only the required set of chars" do
r = Jekyll::HighlightBlock::SYNTAX
r = Jekyll::Tags::HighlightBlock::SYNTAX
assert_match r, "ruby"
assert_match r, "c#"
assert_match r, "xml+cheetah"
@ -55,19 +55,19 @@ CONTENT
context "initialized tag" do
should "work" do
tag = Jekyll::HighlightBlock.new('highlight', 'ruby ', ["test", "{% endhighlight %}", "\n"])
tag = Jekyll::Tags::HighlightBlock.new('highlight', 'ruby ', ["test", "{% endhighlight %}", "\n"])
assert_equal({}, tag.instance_variable_get(:@options))
tag = Jekyll::HighlightBlock.new('highlight', 'ruby linenos ', ["test", "{% endhighlight %}", "\n"])
tag = Jekyll::Tags::HighlightBlock.new('highlight', 'ruby linenos ', ["test", "{% endhighlight %}", "\n"])
assert_equal({ 'linenos' => 'inline' }, tag.instance_variable_get(:@options))
tag = Jekyll::HighlightBlock.new('highlight', 'ruby linenos=table ', ["test", "{% endhighlight %}", "\n"])
tag = Jekyll::Tags::HighlightBlock.new('highlight', 'ruby linenos=table ', ["test", "{% endhighlight %}", "\n"])
assert_equal({ 'linenos' => 'table' }, tag.instance_variable_get(:@options))
tag = Jekyll::HighlightBlock.new('highlight', 'ruby linenos=table nowrap', ["test", "{% endhighlight %}", "\n"])
tag = Jekyll::Tags::HighlightBlock.new('highlight', 'ruby linenos=table nowrap', ["test", "{% endhighlight %}", "\n"])
assert_equal({ 'linenos' => 'table', 'nowrap' => true }, tag.instance_variable_get(:@options))
tag = Jekyll::HighlightBlock.new('highlight', 'ruby linenos=table cssclass=hl', ["test", "{% endhighlight %}", "\n"])
tag = Jekyll::Tags::HighlightBlock.new('highlight', 'ruby linenos=table cssclass=hl', ["test", "{% endhighlight %}", "\n"])
assert_equal({ 'cssclass' => 'hl', 'linenos' => 'table' }, tag.instance_variable_get(:@options))
end
end
@ -129,7 +129,7 @@ CONTENT
context "using Textile" do
setup do
create_post(@content, {}, Jekyll::TextileConverter)
create_post(@content, {}, Jekyll::Converters::Textile)
end
# Broken in RedCloth 4.1.9
@ -203,4 +203,22 @@ CONTENT
assert_match %r{/2008/11/21/complex/}, @result
end
end
context "simple gist inclusion" do
setup do
@gist = 358471
content = <<CONTENT
---
title: My Cool Gist
---
{% gist #{@gist} %}
CONTENT
create_post(content, {'permalink' => 'pretty', 'source' => source_dir, 'destination' => dest_dir, 'read_posts' => true})
end
should "write script tag" do
assert_match %r{<script src='https://gist.github.com/#{@gist}.js'>\s</script>}, @result
end
end
end