Ember 2.0 and the Indie Web with Yehuda Katz and Tom Dale

Link:

Tags:

Yehuda Katz and Tom Dale join us to talk about the road to Ember 2.0 and "Fast Boot". They share insight about why they stick to a 6 week release cycle, and why they think JS frameworks might be the future of all web apps (especially content sites). We also chat about what "indie open source" means, and exactly how much design goes into the Ember project and community.

Transcript:

FILE NAME: The Frontside 16 - Yehuda Katz & Tom Dale Talk About Javascript

DURATION: 55:59 minutes

CHARLES: Everybody, welcome to Frontside the Podcast, Episode 16. We've got Brandon and Stanley here with me on the podcast and some very special guests who need no introduction, so I'll let them introduce themselves.

TOM: Maybe we just start with Yehuda because he's the most famous. Are we starting by number of Twitter followers?

STANLEY: Actually, Cloud score.

TOM: Yeah, it's Cloud score based.

YEHUDA: I think Tom has a higher Cloud score than me.

BRANDON: All right, Tom. Go for it.

TOM: Hey, what's up? I'm Tom. I had the idea for Ember JS in the shower.

[Laughter]

YEHUDA: Hey. I'm Yehuda. I work on standard stuff and Ember a lot these days, and now Rust also.

TOM: Yehuda just joined the Rust core team. I don't know if you guys saw that.

BRANDON: I did see that. Rust is a programming language.

YEHUDA: Programming language.

STANLEY: Are we going to get to talk about that later?

BRANDON: Sure.

TOM: We can talk about whatever you want. I'm not going to have any insights on that.

YEHUDA: We'll have some insights.

TOM: Yeah. I don't know if you guys ever saw the Pokemon Movie, but basically Yehuda is reenacting that with core teams. You've got to catch them all.

[Laughter]

BRANDON: That's great.

STANLEY: That's not just the movie, Tom. That's literally everything around Pokemon.

TOM: Oh, okay.

STANLEY: That is the tagline.

TOM: I will definitely defer. You seem like an expert here, Stanley.

STANLEY: You know; I know the most important facts of all time.

BRANDON: Stanley is on the Pokemon core team actually.

STANLEY: I actually just made a new Pokemon that's like a guitar, a chair, and a microwave put together.

BRANDON: What's it called?

STANLEY: Rock-On.

TOM: That is the worst Pokemon name I have ever heard.

[Laughter]

TOM: Oh, my gosh.

BRANDON: All right, well, off to an auspicious start here. So the two of you and Leah formed the original core team for Ember. Is that fair to say or were there other people involved at that time when you kind of were switching SproutCore 2.0 to Ember?

YEHUDA: I don't think I would call the original group that switched SproutCore over to Ember necessarily the core team. I would say that there was a bunch of people that were working on what was called SproutCore 2 at the time at Strobe, and it doesn't -- what we were doing there doesn't really meet my requirements for a good core team or a good Indie Open Source project. But one of the things that we did after switching over to being Ember and announcing the separate project was to make the core team more like what I would want.

TOM: Well, I would say that there was never one moment where we were like, hey, let's create a core team. I think one thing that I learned from Yehuda about managing an open source project is that it is extremely important to start delegating way before you feel ready or comfortable. So there was a point early on where we were just totally overwhelmed as people started using it and people came along and were interested. And so we just gave them commit bit without really thinking about the bureaucracy of it or the structure of it. And then it definitely got to a point where it was like, "Why is there no core team yet?" because there's a ton of people with commit. So we should probably think about this a little bit more.

YEHUDA: Of the people who are on the core team now, Erik, Chris, and Steph were all involved extremely early. I think Erik Bryn was the second contributor. Well, the first contributor after people working at Strobe, and Chris and Steph got involved really early because they were building an app on Ember that was very, very mobile focused. Well, it's a mobile app, and so they needed heavy performance, and we were not necessarily focusing on that, so they got involved pretty early also.

BRANDON: And you gave a really awesome talk about this recently at Hack Summit. We'll throw the link to that in the show notes. I thought it was terrific, and I thought there was a lot of amazing ideas that were clearly born of painful experience. And I want to talk about that in a moment and kind of basically running and maintaining an open source project that keeps the open source ethos, I think, was kind of the thrust of the talk and keeping the Web and Open Source Indie.

But before we jump into that, I wanted to get kind of a -- without going into, like, a full state of the union, there's a really lot going on in the world of Ember right now. It can be actually kind of hard for individual developers to just keep up with the news of it. There are just so many cool things happening at once. And there are a few things in particular that I wanted to get an update about. You guys are doing some really interesting stuff right now, but some things that are shipping soon: HTMLBars is actually happening.

YEHUDA: It's in the beta.

BRANDON: Yeah, it's in the beta, so people -- we tested it in our app already, and with one exception with, I guess, Ember list view isn't quite ready for it yet, but.

YEHUDA: I think that's ready now too.

BRANDON: Oh, really?

TOM: Yeah, that just got updated.

BRANDON: So, yeah, it was phenomenal. I mean it just works. Like, it was pretty amazing. So the benefit to users for that has been kind of already gradually been implemented where the metamorph tags in the DOM were gone.

TOM: Right.

BRANDON: What else can people expect to see once HTMLBars is in place?

YEHUDA: So let me just reiterate a thing that you just said that maybe people weren't clear about before I let Tom a little bit about HTMLBars, which is, one of our major goals now and probably forever is to continue to update things incrementally and without breaking apps. And that's something that takes a lot of effort, so I want to reiterate it because it's probably the driving force of everything that we do, so like you said, there's a lot of news. We've been talking about a lot of stuff. We can talk for hours on it, probably.

But the key thing is that a lot of times you hear there's a lot of news in a project, and it feels overwhelming. It feels like, oh, my God, if there's that much news, it probably means I'm going to have to spend the next five years catching up with all the things that are happening. And with Ember, our major goal is to make sure that all that, all those new features don't affect your app. I mean there will be a 2.0, and at 2.0 there may be some breaking changes, but even with the 2.0, all the breaking changes will land before 2.0. They'll land on the 1x brands together with deprecation warnings so you'll learn about them as you upgrade.

TOM: Yeah.

YEHUDA: And so I think this is the driving force of everything we're doing now.

TOM: Yeah, I think, with 2.0, it's not like oh, my gosh, there's all this new stuff I have to learn. Instead, what it's going to be is us removing stuff that you probably never even learned about anyway.

YEHUDA: Or that we told you in 1.10 or 1.11 to switch away from and you had plenty of releases to remove.

TOM: Yeah, you'll have ample warning, and you'll definitely -- it's not going to blindside anyone. But I think this is exactly the point is we're on a six-week release cycle, and it is impossible to do big bang stuff in six weeks. Right? Think about any big software project anyone listening to this has worked on. It's hard to build a huge thing in six weeks, which maybe seems like a limitation, but actually I think we both see that as a huge strength, which is that it really forces you, as an engineer, to think about, okay, I want to move mountains. But I need to do it six weeks at a time, so how do I basically touch back down to reality as often as possible?

With HTMLBars, if you think about it, it's a pretty dramatic thing, right? We're basically entirely replacing the rendering engine of this pretty large JavaScript framework, which in some sense is like trying to change the engine on a 747 mid flight. And the only way that we can get away with doing that is to do it very incrementally. And the only way we can do it without breaking people's apps, I should say, is to do it incrementally.

Step one was metal-views, which removed metamorphs. But, fundamentally, all view rendering was still string concatenation. And then the next step after that was to get actual HTMLBars in with creating DOM instead of creating strings.

YEHUDA: There was actually another step in between, which was to change all the -- so the internal, whenever you use a curlies and handlebars, those curlies, in the old version of the templating engine, would go and they would have ad hoc code to observe something, observe paths, and there was all this complicated code at each point where a curlies was used anywhere in the templating engine, and the new system -- and this is again behind the scenes, so it's easy for even Tom to have forgotten about it. We refactored everything internally so that it used a stream-based system so that all the parts of the templating engine don't have to know exactly how that's all structured internally. And that makes it a lot easier to do performance optimizations, but also makes it a lot easier to avoid mistakes and bugs that cropped up from time-to-time. So that was another step in the direction that was necessary to get all the way to the end.

TOM: So now that's in, and I think the next step is to actually -- the next step will be now that we've got HTMLBars integrated in a backwards compatible way, the next step is introducing nice, new syntax. So just one example of this is this gives us the ability to remove the bind-attr helper. Most people that I've noticed when I'm teaching them Ember intuitively think that they should be able to do attrf equals curly, curly URL, and that doesn't work together. You have to do bind-attr. But because HTMLBars is a much smarter parser, we're able to have that context, and we can actually just dramatically simplify the whole model.

BRANDON: Okay. And you also answered another question I had, which was, there are a lot of people basically talking about how they should be building apps for 2.0 friendliness, but it sounds like if they stay with the point releases, there'll be very little work involved in moving to 2.0. So you don't have to kind of like today sit and architect your app in a certain way as long as you're staying up to date with the point releases. Would you say that's relatively accurate?

TOM: Yeah, I think so. 2.0 is not going to have any new features, and one feature that Teddy Zeenny is working on for the Ember Inspector, you know the Chrome and Firefox plugin for the developer tools, is adding a deprecations pane. So what we expect to happen is that people should just keep upgrading their apps on the 1x point releases and then, every upgrade, you may see one or more deprecation warnings pop up, either in the console or in this pane in the developer tools. And you just fix those at your leisure. Then when 2.0 comes out, all that will happen is that the payload size of Ember will shrink.

YEHUDA: Yeah. I think another way to put all that is, when we looked at React, so we looked at React a lot as part of the run-up to 2.0 for the past, like, six months. And when we looked at React, initially we saw a programming model that actually wasn't that different from how we thought people should build Ember apps, so things like you should have data flowing down from a single point and, in Ember, that single point is the model hook in your route, and then we always thought about data flowing down. And you should have events bubbling up, and you should use actions mostly for communication.

I remember Erik Bryn very, very early on said, "I think people are overusing data bindings. People should use events more." And that's when we started adding the evented system to a bunch of the parts of the framework.

TOM: Actions. Actions weren't there originally … two-way bindings.

YEHUDA: Well, we added actions, but we also added, like, Ember.evented.

TOM: Yeah.

YEHUDA: And I think we kind of knew all this, right? And so idiomatically the way that Ember 2.0 works is actually not that different from how we thought Ember 1.x should work, but I think one of the things that ended up happening is that because data bindings are so -- two-way data bindings can be very nice and clever, a lot of times people would reach for two-way data mining because it was the first thing that was sitting there. And then they would end up building somewhat complicated systems that rely on communication through two-way data mining.

TOM: The syntactic sugar pushed you in that direction.

YEHUDA: And so a lot of what Ember 2.0 is is about making syntactic sugar more honest about what is the right default, and that does mean that there may be applications that were heavily relying on communication, especially communication channels through two-way data bindings. And that will work much less nicely in Ember 2.0, and it might feel painful to upgrade if you're trying to be both idiomatic and upgrade to 2.0 at the same time. But I think, for most people who are using actions and were using data down through the model hook, I think a lot of it will feel very familiar, will feel very much like how you've been doing things all along.

CHARLES: Cool. I actually had a question about HTMLBars landing, and that's when we upgrade our apps, everything should just work seamless. Like Brandon said, we actually did a spike of that, and it mostly seems to be the case. You said that there are no new features, but are there more, like, newer development stories around? Like, given that the templating engine or the view layer understands the DOM, what power or APIs will users have to interact with it, like to do animations or bring things in and out and stuff like that?

YEHUDA: Oh, yeah.

CHARLES: Is there any of that stuff planned?

YEHUDA: I don't think we meant to suggest that there are no new features in number 2.0. Just that they will land in the 1x series, I think, is the point.

CHARLES: Ah, right, right. I see.

TOM: I think probably the biggest feature is just speed. And I think, also, what HTMLBars' architecture unlocks is the ability to better integrate with other libraries by adopting kind of a diffing approach similar to what React does with the virtual DOM. Basically, in my mind, HTMLBars is all about a bunch of infrastructure work that allows us to make the programming model feel more natural for people who are….

YEHUDA: One way to think about it is that it's the first templating engine that was ever built specifically for Ember. Handlebars templating engine before was built as a general-purpose templating engine, and we pushed it as far as it could go to be real useful for Ember. But things like bind-attr and the metamorph tags kind of crept in as necessary because the templating engine wasn't really built for this purpose, the exact purpose that Ember was designed for. And the HTMLBars engine, part of it is that it's DOM based, and part of it is that it supports diffing, and part of it is that it's faster. But I think, ultimately, it allowed us to look at a lot of areas that are annoying about using templates in Ember and make them nicer. And, yeah, so I think that it's --

TOM: I'd say it also unlocks things like we're working on server side rendering right now, and a lot of that is due to the power of HTMLBars because we can run it in so many different environments, and we can model all of these things as streams internally. It gives us a lot of flexibility about what we can do with your applications. You know, we can do things like server side render your applications even though, of course, you never designed your app for that case in mind. But because of how expressive the templating and, in fact, the entire application architecture is, and because we all agree as a community that this is how we architect our apps, it unlocks a lot more stuff. And I think there'll be more things like server side rendering in the future that we all benefit from by sharing this very declarative application structure and very declarative templating language.

YEHUDA: Yeah, I mean honestly, under the hood, the fact a lot of the innovations of the templating engine are not things that any user will ever see directly because they're just implementation. And if we wanted to go around pimping things like streams or the way that the diffing algorithm works internally and the way we clone fragments and all this stuff, we could probably spend a lot of time pimping it, and maybe that would even make a lot of people, some people happy. But I think you'll see, over the next year or so, these things will all lead towards better features or to more features that will be nice, and that's how I think we'd like to roll --

TOM: I think Web components integration is a big one.

YEHUDA: Ah, yeah, that's a good point.

TOM: I think HTMLBars makes it very easy. And so, in terms of actual improvements coming on top of HTMLBars, the component syntax, instead of being curly, curly to indicate that you want a component, you'll be able to use just regular angle brackets, so that'll be nice. And another thing that we're really keen to get rid of is: You know how today when you're building an Ember component if you want to customize the element associated with that component, you have to say, like, tag name, class name, bindings. You guys know what I'm talking about?

BRANDON: Mm-hmm.

TOM: So that's kind of annoying because all of those special properties on the component class that you used to customize the element are all duplicating features that already exist in the templating language. So it's just kind of this weird bifurcation of the programming model where, if you want to customize the element for this top level, do it in JavaScript. Everything else, do it in the template. So HTMLBars will allow us to allow you to customize your component element purely in the template, and you won't have to -- basically there are far more cases now where you'll be able to get away with a component that's just a template file, and you'll reduce the number of JavaScript classes in your app.

YEHUDA: Yeah, I think, from a high level, the biggest -- the high level of the internal technical improvements are largely that it allows for contextual work. So the old templating engine wouldn't necessarily know that you're inside of an attrf when you have curlies, so we would have to spit out a bunch of extra stuff and maybe make you use extra helpers. It wouldn't necessarily know that you're in image FRC or a video tag or a component. It wouldn't be able to tell if you were using a polymer component that implements the bind protocol, right? But the new templating engine basically lets us see exactly what's happening at every curly, and that just has a large number of positive effects.

BRANDON: So you said something else that I felt like was a good segue into the next part of the discussion that this basically allows you to do server side rendering, which you guys are really, like, in the thick of it right now. But for me, a lot of the talk I've seen a about server side rendering comes across as a little inside baseball. There's this sort of discussion between people who are really into React because they're suddenly doing a lot of stuff with server side rendering, and they're seeing some benefits out of it. And you see this stuff kind of pop up in the JavaScript community, but I'm curious to know if you guys can explain a little bit about the benefit of server side rendering that this is a major new feature coming to Ember now.

YEHUDA: I'll let Tom maybe give the full pitch, but I think one thing that I feel somewhat strongly about is that, for most people, if you don't have a system that actually gives you something that works pretty well out of the box, in other words it doesn't ask you to do a lot of the work yourself, the idea of isomorphic server side rendering where you run your same app on the client and the server, I don't think that that ends up being worth it. And if you look at a lot of apps that use Ember today, a lot of them have spent relatively little time building non-isomorphic solutions for specifically SEO that have been very, very cheap in terms of time and very, very effective.

But I think there's the: I want to not have to write that, even that little bit, and I think that that you get a lot of benefits out of if you just have your framework do it. In other words, if it's not just like your framework does 20% and you do 80%. If it's your framework does 95% and you do maybe a little bit, or you have some constraints, I think that is worth something. And I think the rehydration of fast boot is worth something, although that has even more issues and even a larger percentage that most people have to do on their own.

Basically, I think the TLDR for me is that I've always saw server side rendering as indeed somewhat inside baseball because, for most people historically, and I think this is even true largely about React, the solution that you're offered is the framework will do a little bit for you, and you have to go figure out a lot of the details about how to make this work for real. And I think most people just don't have the time to figure all those details out.

TOM: So it's been kind of interesting to watch this play out over the last year or two because I think there's been a big split between the JavaScript application community and old school people that create content for the Web who are really keen on this idea of progressive enhancement. And so there's kind been this split. And, for me, for a long time, I personally didn't really care about this case because, in my mind, JavaScript apps were really good for what I'll call workspace apps, which is most of them are behind a login. You log in. You have these very rich interactions. You're editing something. You know, I worked on the iCloud apps. It's a feature, not a bug, that Google can't index your calendar and your email, right? So to me that was the use case for JavaScript apps. But that was until I saw content sites.

Like, I remember the first time thinking, "Wow, maybe JavaScript apps are actually the future of all Web applications," was when I saw Bustle, actually. When I saw Bustle, it was just a content site. It was just news articles. And if you would've asked me, I would have said you should probably use Rails or some other server rendered framework for this. But then I saw it, and it felt just like a website. I couldn't actually tell the difference other than how damn fast it was. And I kind of had to step back and question all of my opinions about how people should be building these kinds of applications.

And especially for content sites, I think that the server rendering is really important, right, because historically your user has had to download all of this JavaScript and all that JavaScript had to be downloaded and evaluated and run before they saw anything. So having the ability to bootstrap that process on the server and have the first bits that the user starts downloading not be the JavaScript payload, but be HTML and CSS, so that the first thing that they see is useful, I think that's really going to change how people build applications because you get the benefits of server rendering while still retaining the kind of interactions that you can build with something like Ember that you just can't do with Rails.

YEHUDA: But again, I think people trying to do this on their own and taking maybe a half solution and then trying to figure out how to make this work reliably ends up producing things that are pretty buggy and feel pretty bad on first boot, or they end up requiring tremendous amounts of engineering resources. And it's possible that, like, huge companies can make this work. But I mostly think about startups, and startups simply do not have the engineering resources to take on the server side rendering task on their own, so this is why I think we care about it for Ember because, as Tom said before, Ember is already pushing you down a pretty conventional path, and we think -- our hope is that by having people do that conventional path and us taking charge of server side rendering will have something that works mostly out of the box for a lot of users. Again, assuming they follow some additional constraints about what you're allowed to do on the server.

TOM: And I think we should be clear here because there's, as always happens, there's ambiguity in the terminology. So first is the term isomorphic app, which I don't really love that term, but I guess we better get used to it, Yehuda.

YEHUDA: Yeah.

TOM: But there's really a spectrum. On the one side of the spectrum you have something like Airbnb has a library for Backbone called Rendr (with no e - well, one e, but not the second e). And that kind of lets you wire up some of the server side rendering. But again, it's very, very manual. And the whole purpose of this is just to make sure that the first thing that you deliver to the browser is HTML and not JavaScript so that the user, even if they don't have JavaScript enabled or they're on a slow connection or whatever, they get something useful.

But then on the other side, you have things like Meteor or Asana where the whole idea is -- and to me, I'll be honest with you. It strikes me as extremely silly, but the idea is that you're writing both your server and your client in the same code base, and then you're deploying them both. You describe it, and it sounds like this magic bullet, but it also just seems really silly to me.

YEHUDA: Well, I think the fact that even the first releases of Meteor had if Meteor.isClient and if Meteor.isServer, and even the first demos had large blocks of content….

TOM: Yeah. Conditionals. Yeah.

YEHUDA: Basically means that people hadn't really figured it out exactly right yet.

TOM: Yeah, the point is that even if you have a client side app, your server still has a lot of responsibilities, especially around data access to the database, authorization, authentication, and so on. And putting that in the code base with your UI and your drag and drop code just does not make any sense to me. So I want to be clear to everyone listening that that is not what we're going for. The idea is not that you're writing your json API server in Ember. The point is that you're writing the same old app. In fact, we hope that you don't have to make any changes to your app. And you can deploy it, and it will do a render using the same code. It's basically like your app running in a browser on the server, and then we'll have some way of --

YEHUDA: Except not actually a browser.

TOM: -- it's actually a transferring state.

YEHUDA: Except, importantly, it's much, much cheaper than being an actual browser. We're not using phantom JS or a zombie or anything.

TOM: Right. Conceptually a browser, but we don't want to pay the cost. Phantom JS is a source of great pain for many people, ourselves included, so we want a pure JavaScript environment. But conceptually it's a browser.

YEHUDA: I think the reason I hopped in there is I just want to be clear that the goal of conceptually a browser is actually not to be a browser, but to make Ember itself internally abstract enough so that the most expensive parts of being a browser can be replicated in a cheaper way on the server. Obviously the most important part of that is the DOM. And that's the part that React worked out for server side rendering is use the virtual DOM on the server and not a real DOM, right? And that means you don't need real phantom JS and the full scope of DOM.

But there's also other stuff like how your routing works and how the model hook runs and how it makes requests, how it makes "XHRs" to get the data in the first place. Right? There are a lot of details if you think about what it takes to boot up an app in the first place. For us, the goal is to go through that whole process of booting up an app all the way through, but not including the did insert element hooks in your DOM and make sure that all that stuff doesn't require -- it has really constrained and clear abstractions, right? So rendering has a render abstraction and routing has a location abstraction, and the DOM has the HTML bars, little dom, lower case dom abstraction, right? So instead of saying we're running it inside of something that pretends to be a browser, we're saying Ember doesn't actually care whether it's in a browser or not, but it has very, very clear abstractions for what it means to be a browser.

BRANDON: Okay, so is there--? It sounds like it could be a little like -- is this a little ways off? It sounds like there are a lot of benefits. Like, if you see the hand rolled stuff that Discourse has done, clearly this is something that Bustle cares enough about to sponsor, this is probably a little ways off for Ember developers. Is there anything that you want people doing Ember to know right now about server side rendering in terms of how it's going to affect them or some of the technical stuff that you're learning through this or anything like that?

TOM: I think we've been thinking about this particular problem for a very long time. And in fact, we've intentionally designed the architecture of Ember specifically to handle this case, even from the beginning, even like three years ago. We knew that this is something that we wanted or at least we didn't want to paint ourselves into a corner around. So I think there are two aspects of this, and one of them, I think, is pretty well bounded and pretty straightforward. The other one is definitely going to require a little bit more research.

The first step is simply getting rendering happening on the server. So because we designed Ember for this case, we were actually able to get Ember as a framework booting up in node in about a day's worth of work. So a couple things have crept in. There were areas where we had been a little bit sloppy and introduced dependencies on things like document.body, jQuery, things like that. So it was about a day to kind of encapsulate those, make sure that they weren't hard requirements for booting the framework, and that was about a day's worth of work.

And then, by the end of the week -- we've only been working on this for about a week now -- at the end of the week, we actually had an app booting in node and handling route requests. So in terms of progress, it's been really great. But I guess all that we're saying is that, in a week, we were able to capitalize on the last three years of work we've been doing. That's not as impressive as it sounds.

YEHUDA: It was nice that there were relatively few regressions, right?

TOM: Yeah.

YEHUDA: That the list of things where people were accidentally doing things that assumed the browser was actually relatively small.

TOM: Yeah. And the way that we did that is basically introducing an abstraction that provides your environment to you, so a node that's different than in a browser. So that's the first part is to get the app booting, and the second part is to get it rendering. I think what's really cool is that, even though HTMLBars, of course, is a DOM based rendering engine, or despite that, we still use an abstraction around DOM access.

What we're going to be able to do when we start, again, in earnest tomorrow, on Monday, is basically replace that DOM helper that we used to create DOM in the browser with something that we'll be able to build up strings so that you can build up your HTML on the server and serve that to the client. That's step one: rendering. I think we'll be able to make quite quick progress on that. I would guess probably about -- I don't know if I should give timelines here. I think we're ballparking around a month before we can at least beta it, the server rendering.

BRANDON: That's a little faster timeline than I had assumed.

TOM: That's purely rendering. I want to make clear that that is purely for things like SEO, for Web crawlers, for curl, things like that. Then the next phase after that, and this is where it gets into the tricky bit, is being able to --

YEHUDA: Rehydration.

TOM: -- is rehydration, what people call it. What we call fast boot. And with fast boot the idea is that whatever state that we use to build up your application on the server, we also transfer that state, not just the HTML, not just the output, but the state that we use to build that output is somehow seamlessly transferred from the server to the client. And the client basically just takes the HTML that we've given it and reconnects all these bindings and goes from there, so it's totally seamless to the user.

YEHUDA: And I think there's some complexity there. For example, there may be some part of your page that you can't actually render on the server because it says, like, "Hello, authenticated user," with the user's name. So thinking about ways to make sure that you can mark those as needing to be rendered on the client without causing jank. There's a bunch of stuff like that where, at first glance, it seems not too bad, but I guess the high level if there's a determinism issue, right? So the goal is to make sure that roughly what you did on the server is the same thing that you do on the client because if it's too far off, then you end up having to throw -- no matter how smart our algorithm is, we're going to end up having to throw away everything and replace it again.

The idea is how to structure your app, how to structure the way that you set up your app in Ember CLI so that you tend to be putting out output that's deterministic on both sides. And, like Tom said, I think state is maybe overbroad. It's mostly modeled batter, right? Making sure that the model batter that you got on the server is equivalent and transferred together with the HTML payload so that the model hooks that you have in your client will not have to go make another XHR. They'll just have the data that the server already collected, and then hopefully rerender an equivalent DOM on the clients with relatively low….

TOM: There are just a lot of tricky cases, like imagine you have a bound helper that shows relative dates, like 30 seconds ago. So you have a clock on the server, and then you have a clock on the client. How do you reconcile those two?

YEHUDA: Yeah, so the good news there -- the good news with all that, without getting too much into the weeds, is that the HTMLBars' engine is already broken up between rendering the parts of your DOM that are static, that are like the hello, the text hello inside of an H1, and updating the static parts with dynamic content. And today that's purely a performance optimization, and so that we could use the same document fragment over and over again after cloning, but that will also allow us to use server rendered content where, instead of expecting to have an empty text node that is to be filling in with dynamic content, we'll have a filled in text node. And we see, in that case, Tom, that the time that is in there already is not the correct time. It's not the equivalent time, and so we'll update it. So that's sort of like the React diffing strategy.

I'm less worried about, like, there's a single text node with the wrong content because I know we can deal with that. And I'm a lot -- although if it's too much changes, it will look very bad. It will look very janky. I'm more worried about, like, this entire conditional change. So you're looking at something and then, like, your sidebar swaps out for a different sidebar, which I think would be a pretty unacceptable UI.

CHARLES: Obviously there are a lot of different server side environments that people use. What requirements of the server is there going to be if I'm using a Rails app or something in Python or Java or whatnot? How am I going to interface to this?

TOM: Well, you definitely need a JavaScript runtime, right, because your Ember app is written in JavaScript. Unless you're planning on pouring it into Ruby or whatever, we're definitely going to require JavaScript runtime. And I think we're starting with just supporting node. But what I would like to see, and maybe you guys can write this, is a Rails gem that you can install that will install dependencies in everything and basically get set up in a production environment.

YEHUDA: I think one thing people often forget is you can definitely -- you could you have a node app running. People try to embed JavaScript. They try to use, like, The Ruby Racer, and embedding JavaScript is quite a disaster.

BRANDON: [Laughter] I'm sorry. I don't know if you know. Charles wrote The Ruby Racer, and it is a disaster.

STANLEY: It's cool. It's a disaster.

YEHUDA: So let me be clear. It's a great idea. I use it a lot, but it just fundamentally doesn't now work. Right? It fundamentally cannot -- you cannot embed two VGCs in the same process.

BRANDON: Right.

YEHUDA: Okay.

BRANDON: This is the greatest moment in the history of our podcast. I just want to say that.

CHARLES: No, I know. There's no way to collect….

YEHUDA: Yes, exactly.

CHARLES: -- for example.

YEHUDA: I was about to use cycles as an example.

CHARLES: The APIs just don't exist.

YEHUDA: Yeah, so basically cycles, so this is why. The reason why I feel strongly about this is that we use Rust for Skylight, which is our product. And we need to embed something, and I would never in a million years embed something with a garbage collector. And I think The Ruby Racer was a pretty good -- I think, for constrained cases, it works fine if you know what you're doing, but people basically tried to use it as, "Oh, I'm just going to write part of my program in JavaScript. And, by the way, both languages have closures, so good luck," basically. The Ruby Racer is cool, but I would not -- I don't think that that's the correct strategy for having long-running JavaScript programs like Ember, like complicated stuff like Ember. I think a better thing for people to do would be to figure out a simple IPC protocol so that they could run their Ember app and then have a simple way of talking over a socket or something with the node app, so booting up your Rails app will also boot up the node app. And, if necessary, and you're serving through your Rails app, you could communicate.

TOM: I think, to be clear, the Ember app, even when running on the server, still talks to the database server using json, right? So it's the exact same data marshalling flow. It's just presumably it'll be a lot faster because probably your API server and your application server are in the same data….

YEHUDA: Well, at a minimum, it'll be a lot more consistent, right?

TOM: Yes.

YEHUDA: I think when people -- when Twitter complained about somebody from some country connecting and getting a really slow connection, the issue there was that they were downloading the app shell and then who knows how long it took to download the json payload, right? But if the json payload is coming from the same data center, then it's, by definition, going to be downed within some range, reasonable range.

BRANDON: Okay. Awesome. I'd like to shift gears and spend a couple minutes talking about something that most of the questions that I had originally for this were actually answered in your talk. But I'd like to go over it just a little bit. You alluded earlier to the way that you're running Ember as an ideal, sort of your ideals, discovering your ideals about open source projects and the Indie Web. And I think it's a really important topic, and I want to ask a little bit about that because I don't think a lot of people understand this.

I think, especially I see in the JavaScript community, a lot more people establishing open source projects that are corporate run and that being considered a benefit rather than a drawback. And I've seen you push back on that a little bit, and I wanted to ask you, Yehuda, about what your definition of the Indie Web is and why you specifically run the Ember project the way that you guy run it.

YEHUDA: Sure. I think there are basically two parts of what it means to be Indie, and the second one sort of derives out of the first one, but I think it's -- you wouldn't necessarily arrive at it yourself, so I'll make it explicit. The first one is that you have a diversified core team. What I mean by that is essentially diverse in all the ways you could possibly think of, and things that I learned from other projects are, like, functionally diverse. So make sure that if there's a person on your team -- if there's a person around somewhere running your events or doing documentation or doing evangelism or running user groups, make sure that if there's a person who is very skilled at doing that that they are the top person on your team in charge of that.

The counter, the alternative that I've seen a lot in the open source community is that the person running events essentially reports to the core team, right? There's a person who is maybe a professional event runner, and they are not in charge of events. They're in charge of coming up with ideas for events that they run by the core team, and the core team decides yes or no. The problem is the core team has no skills in doing that, so this person ends up spending huge amounts of time being frustrated trying to explain to the core team something or other, right? That would be the equivalent of somebody on the core team writing some area of code, having to come to the rest to the core team and talking about really tiny, nitty-gritty implementation details.

Of course, unlike code where I think people have an intuitive sense that you're deep in the weeds of some performance thing, and I don't really understand that. In a lot of areas that are not code, code people have a very high sense of their own understanding, right? The core teams, I've seen a lot of core teams that people come and they say, well, I happen to know a lot about how people want to read docs in this country and so I'm going to help with the translation effort. All of a sudden the core team is an expert on, like, Indian documentation. And they have all these opinions that are totally unwarranted.

Step one is have a functionally diverse group of people, and have people that are not necessarily hardcore coders, but are talented and professionals in their area. Have them do that work at the top level. And I spent a little time on this just now because I feel strongly that this is an area that people miss, and it's just an area that code people are too high on their own supply. They're too impressed with their own skills. They cause a lot of pain for the people who are good at areas that are not hard-core code, so that's one.

Two is be diverse in terms of the set of people that own decision-making in your project, so this is what you were talking about with the company run open source, and this is something companies can do. I've seen, for example, I worked with the Rust Project, and the Rust Project originally started at Mozilla. But they've been spending tremendous amounts of effort to try to increase the set of people who are on the core team of Rust that are not working at Mozilla. And this is something that maybe takes a lot of effort once you have an established project at a company.

There's all the internal company politics you have to deal with. But the reality is that if you have a project that's at one company, you're kind of at the whims and the mercies of that one company's how they do resourcing, whether they think it's important, what their actual goals are. Maybe the thing that they built the project for doesn't necessarily match what the community is doing, right? So become diverse in the companies that own it.

I think projects like Rails, Postgres, Ember, increasingly Rust are good examples of this. And, of course, I mean the regular meaning of the word diverse here, just because if you become more diverse in all areas, you actually find yourself being more function diverse just because of who ends up being in what areas. You end up finding yourself having a lot better insight. You end up sitting in a room, and when there are people of diverse backgrounds, the kinds of questions that people ask.

And I can only say this. This is the thing that I've noticed personally. It's not a thing that I can empirically measure. I've just noticed that the kinds of questions that people ask, when you're diverse in all kinds of ways, ends up being -- they end up being stronger, better, and they end up pushing back on the kinds of decisions that you can make as a monoculture with group think, right? The harder it is to have group think, the harder it is for everyone to sit around and say, "You know what we're going to do? We're going to rewrite everything," right? That kind of thing is hard to do when you have a lot of people with a lot of different backgrounds with a lot of different interests. So that's, I think, the higher order bit of what in the open source means is be very diversified in a lot of different ways.

But there's a specific thing that I think comes out of it that is very important, which is to do the work that you're doing incrementally. Again, the reason I think that this is a derive is that if you have a lot of people with a lot of different interests with a lot of different projects, I think it becomes very difficult for you to do full on rewrites because everybody has interests, and maybe a few people are excited about doing a rewrite, but everyone else says, "Oh, my God. What about my app?" Then you have the docs guy saying, "Oh, my God. Now we have to maintain two sets of docs?" And you have the evangelism guy hearing all the pushback from the community about the rewrite. So it becomes very difficult for you to have this situation where you're not doing things incrementally.

But I think, in my talk, I spend some time on what it means to do things incrementally and what the benefits are and how to adopt the six-week release cycle and all that. I talked about that in more detail because, even though I think it's a natural consequence of being diversified, it's also something that you have to think about if you want to do it well.

BRANDON: Yeah. It seems to me that that would require, like it's kind of a chicken and egg deal, but to me it seems like there was a lot of discipline in switching to a six-week release cycle, and that's important. It seems, for us at least as consumers of Ember as an open source framework, that's benefited us greatly. We find the framework much, much easier to keep up with on the six-week release cycles than pretty much most other open source projects we've worked with.

YEHUDA: Which is kind of like a paradox, right, because you expect that six-week release cycle means you can never keep up with it because it's always changing. But in fact, six-week release cycle means you don't have time to ever change that much.

BRANDON: Well, and even for apps that go dormant for a while, we find that, okay; we'll bring it up to one. Bring it up to the next one. The upgrade path becomes extremely obvious.

YEHUDA: Yep, exactly, and there's deprecation warnings, and there's ... I think people should watch my talk on this specific area because it took me 15 minutes or something to lay out the whole thing. But I think the basic idea of just doing things incremental, and incrementally has this bad sense. It's like, oh, maybe you're hunting for the local maximum or something. All I mean by incrementally is the same way that the human body replaces its cells, right? You don't do it all at once.

Maybe over an extended period of time, the thing that you're looking at is completely different. But because you did it a little at a time, you were able to move the whole community together in the direction instead of people, many, many people getting left behind, which is what used to happen a lot with Rails. I think Rails has gotten better at this over time. But it used to happen, like Rails 3 came out and a lot of people were stuck on Rails 2.3.

BRANDON: I think when people hear incrementally, they can think about possibly incrementally be led in circles, right? You could incrementally be every day wake up and decide to change one degree or the other. What matters is if you have a compass that's pointing somewhere.

For the Ember project, that incremental stuff doesn't work unless it's pointed somewhere very clear. And you and Tom are basically the keepers of that vision. And I wanted to ask about that, what the vision of the framework is that keeps guiding everybody. Is it sort of implicit to the project? As you use it, you recognize it. Or is it something that you've explicitly outlined somewhere?

YEHUDA: I'll take this for a second and then Tom can jump in. I think, fundamentally, the main vision is just we ultimately wanted to have a full stack solution that solved the majority, the vast majority of the problems that a regular person would have writing a Web app, but we knew from the beginning that if we tried to take on that whole project, I mean even Meteor, who took on that whole project, is still struggling to have to complete the vision. And they tried to sort of boil the ocean.

And so, we knew from the beginning that we were going to get it wrong if we tried to do a CLI tool and a framework and a data library and all this stuff all at once. And so I think we started off with the V in MVC, basically, right at the beginning and added routing and other stuff over time. But the mission always has been to build the thing that's a full stack of what you need to build front-end tools. Tom, you can take it from there.

TOM: Yeah. I think, in the JavaScript world, I think, because JavaScript for so long was treated as a toy language that people didn't do serious stuff in, it attracted a certain type of developer who is -- which is a totally legitimate opinion, but they tended to be kind of hackers who would do these one-off experiments. And because of that, the notion of convention over configuration and having shared solutions is still a somewhat surprisingly controversial opinion in the JavaScript community.

And so I think the role, as you said, for Yehuda and I, is to basically be willing to stand up and take the tomatoes that get thrown at us and say, "You know what? No, there is benefit in having a shared solution, especially when it's not just one-off hackers in their basement, but when it's a team of engineers working at a company, and you have a product that you need to ship, and it needs to have good interaction. It needs to be done yesterday. Those people need tools too. People like that deserve tools."

And I think that's our goal is to have a framework that will last for at least the next ten years that is willing to incorporate good ideas as they come out and as they're embedded, move the community together, as Yehuda was saying, but without chasing the hype dragon where every six months: "Throw away everything you know because the next big thing is coming out. Rewrite your apps."

I see Ember as a way of kind of tempering that instinct for engineers to chase the new and shiny constantly in a way that basically we have a community that agrees together what the next big thing is, and then we start moving towards that. I think, right now, major things that we're thinking about are one server rendering, as we talked about. Getting the CLI tools in place, that's a thing that we've wanted for a long time. But as Yehuda said, we didn't want to try boiling the ocean. And then the new HTMLBars view layer, which unlocks a lot of the cool things that React is able to do around, like, DOM diffing and so on.

YEHUDA: Yeah. I usually tell people -- recently, I've come to a line, which is, if you want to tell me that there's not a place for shared solutions in some area or some abstraction, I think the burden is on you, actually. I think people in the JavaScript community, and there's a small group in the Rails community that feels the same way, they assume that the burden is on the person who is trying to abstract, right? If there's a common problem that a lot of people have, they think it's your job to prove that abstraction is a good idea.

I think it is your job to prove that a particular abstraction is a good idea. But I think my de facto, my default position is that if a lot of people are solving the same problem that there's a shared solution worth hunting for. And I would say the JavaScript community is really -- big chunks of the JavaScript community are pretty anti this approach. But I think it's really the only way that you could build -- like Tom said, the only way you could build projects with large teams is to have some sense of what the shared answer is and to not have it be some genius on the fourth floor somewhere that does everything. And if you want to make any changes, you have to go to them. I've seen a lot of companies that work like this, and it works fine.

Anther facet of this is that pretty much every company deciding to adopt, like, Ember, Angular, React, or Backbone, whatever, they do like this "taste test," right? A taste test is like a two-week project where they see which one is faster. By definition, the taste test doesn't successfully analyze what happens over a longer period or when you have a bigger set of developers, right? It's by definition optimized for short projects with a small number of developers, and so --

TOM: And usually it's the guy on the fourth floor conducting the taste test.

YEHUDA: He's either conducting it or he is actively attacking it, right? He is saying we should not -- for example, Firefox OS refused to adopt any framework for an extended period despite the complaints of many people on the Firefox OS team because of the fact that they had a religion against frameworks. They didn't like frameworks as a concept. I've heard this from large numbers of people working on the periphery of Firefox OS and many complaints, right?

And so I think we, Ember, one of the things we had to learn was that we can't get away with just saying that. We can't get away with saying, oh, you'll learn. If you just use Ember for a year, you'll figure it out. We had to really improve the getting started experience. But I think, on the flipside of that, there's no way that we could ever -- even if we get to be as nice as to optimal getting started framework, getting started tool, we're always going to have benefits that are not part of that that are very difficult to see when you're doing a quick analysis.

Actually, recently we've seen a lot more big companies come out and talk about the benefits of Ember for long-lived projects, and I think that helps a lot just having people testify that they used Ember for a year within a team that wasn't three people, and they found it to be productive in these specific ways. I think that's helped a lot of people feel comfortable.

BRANDON: That's been my experience, certainly, as well, just seeing that increase. I think everybody should -- honestly, I believe everybody listening to this should drop what they're doing and go watch the Hack Summit talk. I thought it was phenomenal, and I think it made me think a little differently because it's a little confusing out there, like what some of the tradeoffs are in these open source projects that are run in that kind of echo chamber. And the fact that you guys work so hard to pierce the echo chamber is really cool.

I know that there may be some technical questions. We don't get a chance to have you guys on the podcast very often, obviously, so I wonder if anybody has any more questions.

STANLEY: Before we get back to technical questions, I just want to cut in and say I really like Yehuda's talk from Rails Conf as well. It was really eye opening for me as somebody new to the Rails community, even though Rails has been around for a while, and kind of understanding the value of shared solutions and kind of the philosophy behind that.

YEHUDA: Yeah, that was definitely the first time I tried to formulate a general theory for what shared solutions look like and why they're good, and essentially why Rails hit the nail on the head with the right amount of shared solutions and where the experimentation is happening and all that stuff.

STANLEY: Cool. Back to you, Charles.

CHARLES: I just had a quick question I wanted, before we wrap up. I had another question about the server side rendering, kind of a general one. I know I've definitely been burned by server side rendering in the past, you know, because it's been something that people talk about on and off, it seems like, for the last five, six years. I remember when mustache first came out. The first time that I tried it, it's like, oh, I've got this templating thing that I can run inside on my Rail server, and I can run it. There's a JavaScript implementation too.

One of the things that I found was I was able to get up running very quickly, but then I felt like I was eaten alive by the edge cases. It was actually -- I think it was a blog post that you wrote, Tom, where you were talking about kind of the justification for resolving, always resolving RSVP promises asynchronously. Because, to not do so, have a different context, different stack sitting on top of the resolution was like releasing Zalgo into your application.

I actually, when I read that thing about promises, it actually made me harken back to my experience with server side rendering. I was like, oh, with server side rendering I was releasing Zalgo onto my client. I had a very different context.

TOM: Yeah, the thing you're describing is sharing templates across two different apps, right?

CHARLES: Right.

TOM: The data model diverges, and then the things that you need diverge. That specifically is something that we are going to avoid. The idea isn't, oh, you can reuse your model on the server, and you reuse your templates on the server. The point is it's your app, the same exact app, same code base that you would run in the user's browser just happens to be running on the server.

YEHUDA: And I think there's also -- I think there's another important point, which is that if you look at how people do SSR, I think historically people have said, "Well, I'm going to use a view layer that's very good at SSR." And then you would have this pile of hacks that was involved in booting up your app. Honestly, Ember, in the beginning, had piles of hacks used to booting up your app. I definitely remember that period. Now the view layer maybe is very good at running on the client server and, like you said, originally was just like the template. But now maybe the whole view layer is good at it. But now the process of actually booting things is a source of non-determinism. You're saying Zalgo. I'm saying non-determinism. Zalgo is a better word.

CHARLES: [Laughter]

YEHUDA: And Ember has definitely held off on tackling server side rendering seriously until we felt confident that we had the full stack handled. In other words that, as a framework, we had the whole lifecycle handled. Then actually, if you look at technically what we've been doing recently, a lot of it is like separating out. We have an application right now, an application only -- there's only ever one instance of it. Now we're saying, well, there could be multiple sessions.

And so we're really looking at the whole lifecycle of the application and, because we own the whole lifecycle of the application, we can actually feel confident that the path that we're going through is correct, so that's one part of it. The other part of it is essentially what React figured out at a high level. I think what we're doing is equivalent, which is you don't necessarily assume that you got exactly the same thing on the client server because probably in practice there's always going to be some thing or other, like the clock case that Tom talked about, or the hello Yehuda case that I talked about before, the authentication case.

What you do is you rerender the template, and you don't say if it's not exactly the same, throw it away and start over. You say parts that are the same you can keep, and parts that are different you replace. Therefore, it's not so much a whack-a-mole problem. It's more like how much replacement can I tolerate and have it not feel janky, right? So you go use it, and if you see that there's an area that's popping in and replacing, that's an area that you have to go and figure out so, first of all, it will work. Right? It will work. It will not be broken. It might feel a little bad, and that's an area for you to go and improve the Zalgoishness of your solution.

In practice, in Ember, it will almost always be something weird like you're relying on a non-deterministic DOM API or something like this, or you're relying on some XHR that even though we serialized it, you're getting a push notification, and it's different, and it happens quickly, so it's janked. Right? I think the basic point of try to control everything and also only replace things that are needed will get you to a much better starting place out of the gate with Ember than you will have if you try to do the old solutions.

It may turn out to be a lot of work. And if it turns out to be a lot of work regardless, then I think it will still, even with Ember, be a thing that is used by people who really need it and not so much by people who don't. But I'm hopeful that the fact that we own the whole lifecycle of your application will let it be useful for a bigger set of people than people who are desperately in need of it as a solution.

BRANDON: Awesome, so I think we're going to wrap up. Is there anything you guys want to give a shout-out to or anyone?

YEHUDA: Please sign up for Skylight to make your Rails apps faster.

TOM: Yeah, please. Make my Christmas a merry one.

YEHUDA: We didn't talk about this at all.

TOM: And sign up for Skylight.

YEHUDA: We didn't talk about this at all, but Tom and I, much of our day job is working on Skylight. And if you watched my talk at Hack Summit, one of the things that I advocate and I really feel it in my gut because of Skylight is I advocate spending, even if you're full time in open source, spending some time, even a day a week or two days a week, would help a lot working on something that you have to maintain over the long haul because, like I said before, maintenance over the long haul is very difficult for you to market. It's something that you have to feel in your heart.

If you're working on an open source project, work on -- use it for a real thing that you spend significant time on that you have to maintain over the long haul so you could feel, in a year, whether you're practice is holding up to being maintainable. And, yes, now that I've talked about Skylight for a second, please sign up. This is how we fund our ability to do any open source. Working on Skylight has definitely been the most enjoyable thing I've done in my career so far in the sense that I've had a lot of control over it, but it's also the most harrowing in the sense that we are responsible for getting all the revenue, so please sign up.

BRANDON: All right, well, thanks very much, you all, for coming on. Everybody, go sign up for Skylight. It's very cool, very beautiful, and very actionable insight for Rails apps. Tom, Yehuda, thank you so much for joining us on this. It's been super enlightening. Again, everybody, please go watch Yehuda's talk on keeping the Web Indie. And if you've got a little extra time, the Rails Conf talk on layers of abstraction is also, I found, something that changed my views on a lot of stuff as well. Thanks again, both of you all, for coming on.

YEHUDA: Thanks a lot.

TOM: Thank you, guys.

BRANDON: All right, talk to you later.

CHARLES: Thank you, guys.

You are here

Ember 2.0 and the Indie Web with Yehuda Katz and Tom Dale