Propagate _data folder from theme (#8815)

Merge pull request 8815

Gemfile

@@ -32,6 +32,7 @@ group :test do
   gem "test-theme", :path => File.expand_path("test/fixtures/test-theme", __dir__)
   gem "test-theme-skinny", :path => File.expand_path("test/fixtures/test-theme-skinny", __dir__)
   gem "test-theme-symlink", :path => File.expand_path("test/fixtures/test-theme-symlink", __dir__)
+  gem "test-theme-w-empty-data", :path => File.expand_path("test/fixtures/test-theme-w-empty-data", __dir__)
 
   if RUBY_ENGINE == "jruby"
     gem "http_parser.rb", "~> 0.6.0"

docs/_docs/themes.md

@@ -21,7 +21,7 @@ See also: [resources](/resources/).
 
 When you [create a new Jekyll site](/docs/) (by running the `jekyll new <PATH>` command), Jekyll installs a site that uses a gem-based theme called [Minima](https://github.com/jekyll/minima).
 
-With gem-based themes, some of the site's directories (such as the `assets`, `_layouts`, `_includes`, and `_sass` directories) are stored in the theme's gem, hidden from your immediate view. Yet all of the necessary directories will be read and processed during Jekyll's build process.
+With gem-based themes, some of the site's directories (such as the `assets`, `_data`, `_layouts`, `_includes`, and `_sass` directories) are stored in the theme's gem, hidden from your immediate view. Yet all of the necessary directories will be read and processed during Jekyll's build process.
 
 In the case of Minima, you see only the following files in your Jekyll site directory:
 
@@ -46,7 +46,7 @@ The goal of gem-based themes is to allow you to get all the benefits of a robust
 
 ## Overriding theme defaults
 
-Jekyll themes set default layouts, includes, and stylesheets. However, you can override any of the theme defaults with your own site content.
+Jekyll themes set default data, layouts, includes, and stylesheets. However, you can override any of the theme defaults with your own site content.
 
 To replace layouts or includes in your theme, make a copy in your `_layouts` or `_includes` directory of the specific file you wish to modify, or create the file from scratch giving it the same name as the file you wish to override.
 
@@ -117,6 +117,7 @@ To modify any stylesheet you must take the extra step of also copying the main s
 Jekyll will look first to your site's content before looking to the theme's defaults for any requested file in the following folders:
 
 - `/assets`
+- `/_data`
 - `/_layouts`
 - `/_includes`
 - `/_sass`
@@ -126,6 +127,49 @@ Note that making copies of theme files will prevent you from receiving any theme
 {: .note .info}
 Refer to your selected theme's documentation and source repository for more information on which files you can override.
 
+### Themes with `_data` directory {%- include docs_version_badge.html version="4.3.0" -%}
+{: #themes-with-data-directory }
+
+Starting with version 4.3.0, Jekyll also takes into account the `_data` directory of themes. This allows data to be distributed across themes. 
+
+A typical example is text used within design elements.
+
+Imagine a theme provides the include file `testimonials.html`. This design element creates a new section on the page, and puts a h3 heading over the list of testimonials.
+
+A theme developer will probably formulate the heading in English and put it directly into the HTML source code.
+
+Consumers of the theme can copy the included file into their project and replace the heading there.
+
+With the consideration of the `_data` directory there is another solution for this standard task.
+
+Instead of entering the text directly into the design template, the designer adds a reference to a text catalog (e.g. `site.data.i18n.testimonials.header`) and create a file `_data/i18n/testimonials.yml` in the data directory of the theme.
+
+In this file the header is put under the key `header` and Jekyll takes care of the rest.
+
+For theme developers, this, at first sight, is of course a bigger effort than before.
+
+However, for the consumers of the theme, the customization is greatly simplified.
+
+Imagine the theme is used by a customer from Germany. In order for her to get the translated header for the testimonials design element in, she just has to create a data file in her project directory with the key `site.data.i18n.testimonials.header`, put the German translation or a header of her choice on top of it and the design element is already customized.
+
+She no longer has to copy the included file into her project directory, customize it there and, what weighs heaviest, waiver all updates of the theme, simply because the theme developer offered her the possibility to make changes to text modules centrally via text files.
+
+{: .note .warning}
+Data files provide a high degree of flexibility. The place where theme developers put text modules may differ from that of the consumer of the theme which can cause unforeseen troubles!
+
+Related to above example the overriding key `site.data.i18n.testimonials.header` from the theme's `_data/i18n/testimonials.yml` file on the consumer site can be located in three different locations:
+
+- `_data/i18n.yml` with key `testimonials.header`
+- `_data/i18n/testimonials.yml` with key `header` (which mirrors the layout of the given example)
+- `_data/i18n/testimonials/header.yml` without any key, the headline can go straight into the file
+
+Theme developers should have this ambiguity in mind, when supporting consumers that feel lost in setting their text modules for the design elements the theme provides.
+
+{: .note .info}
+When using the data feature ask yourself, is the key that you introduce something that changes the behaviour of the theme when present or not, or is it just data that's displayed anyway. If it's changing the behaviour of the theme it should go into `site.config` otherwise it's fine to be provided via `site.data`.
+
+Bundling data that modifies the behavior of a theme is considered an **anti-pattern** whose use is strongly discouraged. It is solely up to the author of the theme to ensure that every provided data can be easily overridden by the consumer of the theme if they desire to.
+
 ## Converting gem-based themes to regular themes
 
 Suppose you want to get rid of the gem-based theme and convert it to a regular theme, where all files are present in your Jekyll site directory, with nothing stored in the theme gem.

features/theme.feature

@@ -41,6 +41,33 @@ Feature: Writing themes
     And I should see "I'm in the project." in "_site/index.html"
     And I should see "<span class=\"sample\">include.html from test-theme</span>" in "_site/index.html"
 
+  Scenario: A theme without data
+    Given I have a configuration file with "theme" set to "test-theme-skinny"
+    And I have a _data directory
+    And I have a "_data/greetings.yml" file with content:
+      """
+      foo: "Hello! I’m foo. And who are you?"
+      """
+    And I have an "index.html" page that contains "{{ site.data.greetings.foo }}"
+    When I run jekyll build
+    Then I should get a zero exit status
+    And the _site directory should exist
+    And I should see "Hello! I’m foo. And who are you?" in "_site/index.html"
+
+  Scenario: A theme with data overridden by data in source directory
+    Given I have a configuration file with "theme" set to "test-theme"
+    And I have a _data directory
+    And I have a "_data/greetings.yml" file with content:
+      """
+      foo: "Hello! I’m foo. And who are you?"
+      """
+    And I have an "index.html" page that contains "{{ site.data.greetings.foo }}"
+    When I run jekyll build
+    Then I should get a zero exit status
+    And the _site directory should exist
+    And I should see "Hello! I’m foo. And who are you?" in "_site/index.html"
+    And I should not see "Hello! I’m bar. What’s up so far?" in "_site/index.html"
+
   Scenario: A theme with a layout
     Given I have a configuration file with "theme" set to "test-theme"
     And I have an _layouts directory
@@ -106,3 +133,19 @@ Feature: Writing themes
     And I should see "default.html from test-theme:" in "_site/2016/04/21/entry1.html"
     And I should see "I am using a local layout." in "_site/2016/04/21/entry1.html"
     And I should see "I am a post layout!" in "_site/2016/04/21/entry1.html"
+
+  Scenario: Complicated site that puts it all together in respect to data folders
+    Given I have a configuration file with "theme" set to "test-theme"
+    And I have a _data directory
+    And I have a "_data/i18n.yml" file with content:
+      """
+      testimonials:
+        header: Kundenstimmen
+      """
+    And I have an "index.html" page that contains "{% include testimonials.html %}"
+    When I run jekyll build
+    Then I should get a zero exit status
+    And the _site directory should exist
+    And I should not see "Testimonials" in "_site/index.html"
+    And I should see "Kundenstimmen" in "_site/index.html"
+    And I should see "Design by FTC" in "_site/index.html"

lib/jekyll/reader.rb

@@ -16,9 +16,26 @@ module Jekyll
       read_directories
       read_included_excludes
       sort_files!
-      @site.data = DataReader.new(site).read(site.config["data_dir"])
       CollectionReader.new(site).read
       ThemeAssetsReader.new(site).read
+      read_data
+    end
+
+    # Read and merge the data files.
+    # If a theme is specified and it contains data, it will be read.
+    # Site data will overwrite theme data with the same key using the
+    # semantics of Utils.deep_merge_hashes.
+    #
+    # Returns nothing.
+    def read_data
+      @site.data = DataReader.new(site).read(site.config["data_dir"])
+      return unless site.theme&.data_path
+
+      theme_data = DataReader.new(
+        site,
+        :in_source_dir => site.method(:in_theme_dir)
+      ).read(site.theme.data_path)
+      @site.data = Jekyll::Utils.deep_merge_hashes(theme_data, @site.data)
     end
 
     # Sorts posts, pages, and static files.

lib/jekyll/readers/data_reader.rb

@@ -4,11 +4,12 @@ module Jekyll
   class DataReader
     attr_reader :site, :content
 
-    def initialize(site)
+    def initialize(site, in_source_dir: nil)
       @site = site
       @content = {}
       @entry_filter = EntryFilter.new(site)
-      @source_dir = site.in_source_dir("/")
+      @in_source_dir = in_source_dir || @site.method(:in_source_dir)
+      @source_dir = @in_source_dir.call("/")
     end
 
     # Read all the files in <dir> and adds them to @content
@@ -18,7 +19,7 @@ module Jekyll
     # Returns @content, a Hash of the .yaml, .yml,
     # .json, and .csv files in the base directory
     def read(dir)
-      base = site.in_source_dir(dir)
+      base = @in_source_dir.call(dir)
       read_data_to(base, @content)
       @content
     end
@@ -38,7 +39,7 @@ module Jekyll
       end
 
       entries.each do |entry|
-        path = @site.in_source_dir(dir, entry)
+        path = @in_source_dir.call(dir, entry)
         next if @entry_filter.symlink?(path)
 
         if File.directory?(path)

lib/jekyll/theme.rb

@@ -43,6 +43,10 @@ module Jekyll
       @assets_path ||= path_for "assets"
     end
 
+    def data_path
+      @data_path ||= path_for "_data"
+    end
+
     def runtime_dependencies
       gemspec.runtime_dependencies
     end

lib/jekyll/theme_builder.rb

@@ -3,7 +3,7 @@
 module Jekyll
   class ThemeBuilder
     SCAFFOLD_DIRECTORIES = %w(
-      assets _layouts _includes _sass
+      assets _data _layouts _includes _sass
     ).freeze
 
     attr_reader :name, :path, :code_of_conduct

test/fixtures/test-theme-w-empty-data/_layouts/default.html

@@ -0,0 +1,11 @@
+<!DOCTYPE html>
+<html>
+<head>
+  <meta charset="UTF-8">
+  <title>Skinny</title>
+</head>
+<body>
+  <h1>Hello World</h1>
+  {{ content }}
+</body>
+</html>

test/fixtures/test-theme-w-empty-data/test-theme-w-empty-data.gemspec

@@ -0,0 +1,11 @@
+# frozen_string_literal: true
+
+Gem::Specification.new do |s|
+  s.name     = "test-theme-w-empty-data"
+  s.version  = "0.1.0"
+  s.licenses = ["MIT"]
+  s.summary  = "This is a theme with just one layout and an empty _data folder used to test Jekyll"
+  s.authors  = ["Jekyll"]
+  s.files    = ["lib/example.rb"]
+  s.homepage = "https://github.com/jekyll/jekyll"
+end

test/fixtures/test-theme/_data/cars.yml

@@ -0,0 +1,6 @@
+manufacturer: Mercedes
+models:
+- model: A-Klasse
+  price: 32,000.00
+- model: B-Klasse
+  price: 35,000.00

test/fixtures/test-theme/_data/categories/dairy.yaml

@@ -0,0 +1,6 @@
+name: Cheese Dairy
+products:
+- name: spread cheese
+  price: 1.2
+- name: cheddar cheese
+  price: 4.5

test/fixtures/test-theme/_data/greetings.yml

@@ -0,0 +1 @@
+foo: "Hello! I’m bar. What’s up so far?"

test/fixtures/test-theme/_data/i18n/testimonials.yml

@@ -0,0 +1,2 @@
+header: Testimonials
+footer: Design by FTC

test/fixtures/test-theme/_includes/testimonials.html

@@ -0,0 +1,9 @@
+<section class="testimonials">
+  <h3>{{ site.data.i18n.testimonials.header }}</h3>
+  <!-- for testimonial in site.data.testimonial }} -->
+  …
+  <!-- endfor -->
+  <footer class="testimonials-footer">
+    {{ site.data.i18n.testimonials.footer }}
+  </footer>
+</section>

test/source/_data/greetings.yml

@@ -0,0 +1 @@
+foo: "Hello! I’m foo. And who are you?"

test/source/_data/i18n.yml

@@ -0,0 +1,3 @@
+testimonials:
+  header: Kundenstimmen
+  # footer omitted by design

test/test_theme.rb

@@ -29,7 +29,7 @@ class TestTheme < JekyllUnitTest
   end
 
   context "path generation" do
-    [:assets, :_layouts, :_includes, :_sass].each do |folder|
+    [:assets, :_data, :_layouts, :_includes, :_sass].each do |folder|
       should "know the #{folder} path" do
         expected = theme_dir(folder.to_s)
         assert_equal expected, @theme.public_send("#{folder.to_s.tr("_", "")}_path")

test/test_theme_data_reader.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+require "helper"
+
+class TestThemeDataReader < JekyllUnitTest
+  context "site without a theme" do
+    setup do
+      @site = fixture_site("theme" => nil)
+      @site.reader.read_data
+      assert @site.data["greetings"]
+      assert @site.data["categories"]["dairy"]
+    end
+
+    should "should read data from source" do
+      assert_equal "Hello! I’m foo. And who are you?", @site.data["greetings"]["foo"]
+      assert_equal "Dairy", @site.data["categories"]["dairy"]["name"]
+    end
+  end
+
+  context "site with a theme without _data" do
+    setup do
+      @site = fixture_site("theme" => "test-theme-skinny")
+      @site.reader.read_data
+      assert @site.data["greetings"]
+      assert @site.data["categories"]["dairy"]
+    end
+
+    should "should read data from source" do
+      assert_equal "Hello! I’m foo. And who are you?", @site.data["greetings"]["foo"]
+      assert_equal "Dairy", @site.data["categories"]["dairy"]["name"]
+    end
+  end
+
+  context "site with a theme with empty _data directory" do
+    setup do
+      @site = fixture_site("theme" => "test-theme-w-empty-data")
+      @site.reader.read_data
+      assert @site.data["greetings"]
+      assert @site.data["categories"]["dairy"]
+    end
+
+    should "should read data from source" do
+      assert_equal "Hello! I’m foo. And who are you?", @site.data["greetings"]["foo"]
+      assert_equal "Dairy", @site.data["categories"]["dairy"]["name"]
+    end
+  end
+
+  context "site with a theme with data at root of _data" do
+    setup do
+      @site = fixture_site("theme" => "test-theme")
+      @site.reader.read_data
+      assert @site.data["greetings"]
+      assert @site.data["categories"]["dairy"]
+      assert @site.data["cars"]
+    end
+
+    should "should merge nested keys" do
+      refute_equal "Hello! I’m bar. What’s up so far?", @site.data["greetings"]["foo"]
+      assert_equal "Hello! I’m foo. And who are you?", @site.data["greetings"]["foo"]
+      assert_equal "Mercedes", @site.data["cars"]["manufacturer"]
+    end
+  end
+
+  context "site with a theme with data at root of _data and in a subdirectory" do
+    setup do
+      @site = fixture_site("theme" => "test-theme")
+      @site.reader.read_data
+      assert @site.data["greetings"]
+      assert @site.data["categories"]["dairy"]
+      assert @site.data["cars"]
+    end
+
+    should "should merge nested keys" do
+      refute_equal "Cheese Dairy", @site.data["categories"]["dairy"]["name"]
+      expected_names = %w(cheese milk)
+      product_names  = @site.data["categories"]["dairy"]["products"].map do |product|
+        product["name"]
+      end
+      expected_names.each do |expected_name|
+        assert_includes product_names, expected_name
+      end
+      assert_equal "Dairy", @site.data["categories"]["dairy"]["name"]
+    end
+
+    should "should illustrate the documented sample" do
+      assert_equal "Kundenstimmen", @site.data["i18n"]["testimonials"]["header"]
+      assert_equal "Design by FTC", @site.data["i18n"]["testimonials"]["footer"]
+    end
+  end
+end